Re: Review Request 36858: Patch for KAFKA-2120

[Guozhang Wang]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36858/#review101511
---



clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java 
(line 427)


One minor thing: could you remove this comment?


- Guozhang Wang


On Sept. 28, 2015, 11:16 p.m., Mayuresh Gharat wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/36858/
> ---
> 
> (Updated Sept. 28, 2015, 11:16 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-2120
> https://issues.apache.org/jira/browse/KAFKA-2120
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> Solved compile error
> 
> 
> Addressed Jason's comments for Kip-19
> 
> 
> Addressed Jun's comments
> 
> 
> Addressed Jason's comments about the default values for requestTimeout
> 
> 
> checkpoint
> 
> 
> Addressed Joel's concerns. Also tried to include Jun's feedback.
> 
> 
> Fixed a minor comment
> 
> 
> Solved unittest issue
> 
> 
> Addressed Jun's comments regarding NetworkClient
> 
> 
> Addressed Jun's comments about disconnect() in Selector
> 
> 
> changed logging level to debug
> 
> 
> Addressed Joels comments to break out early from the loop while aborting 
> expired batches
> 
> 
> Addressed Jun's comments
> 
> 
> Addressed Jason's concern about iterating over timeout request in 
> getNodesWithTimedOutRequest()
> 
> 
> Addressed Joel's comments
> 
> 
> Changed the default timeouts for requesttimeout in KafkaConfig and also made 
> requestrtimeout consistent in KafkaConsumer
> 
> 
> Fixed the forward referencing bug in KafkaConfig.
> 
> 
> Diffs
> -
> 
>   clients/src/main/java/org/apache/kafka/clients/ClientRequest.java 
> dc8f0f115bcda893c95d17c0a57be8d14518d034 
>   clients/src/main/java/org/apache/kafka/clients/CommonClientConfigs.java 
> 7d24c6f5dd2b63b96584f3aa8922a1d048dc1ae4 
>   clients/src/main/java/org/apache/kafka/clients/InFlightRequests.java 
> 15d00d4e484bb5d51a9ae6857ed6e024a2cc1820 
>   clients/src/main/java/org/apache/kafka/clients/KafkaClient.java 
> f46c0d9b5eb73887c62a0e09c96e9d8c964c709d 
>   clients/src/main/java/org/apache/kafka/clients/NetworkClient.java 
> 51a6c5dc47c4008e92b9fdd2ee359b573184e8ed 
>   clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerConfig.java 
> b9a2d4e2bc565f0ee72b27791afe5c894af262f1 
>   clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java 
> 3ac2be840c5d313a497ab67dc024d1578d4679fd 
>   
> clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerNetworkClient.java
>  9517d9d0cd480d5ba1d12f1fde7963e60528d2f8 
>   clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 
> 804d569498396d431880641041fc9292076452cb 
>   clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 
> 06f00a99a73a288df9afa8c1d4abe3580fa968a6 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/BufferPool.java
>  4cb1e50d6c4ed55241aeaef1d3af09def5274103 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java
>  a152bd7697dca55609a9ec4cfe0a82c10595fbc3 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java
>  06182db1c3a5da85648199b4c0c98b80ea7c6c0c 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 
> d2e64f7cd8bf56e433a210905b2874f71eee9ea0 
>   clients/src/main/java/org/apache/kafka/common/network/Selectable.java 
> 70e74bd6aa629c430b2850ca40c97df0b16e5d75 
>   clients/src/main/java/org/apache/kafka/common/network/Selector.java 
> 5a4909e4e5afe1b7ce2c33b9324d3947d2d91a38 
>   clients/src/test/java/org/apache/kafka/clients/MockClient.java 
> e5815f56bdf8e2d980f2bc36b831ed234c0ac781 
>   clients/src/test/java/org/apache/kafka/clients/NetworkClientTest.java 
> 69c93c3adf674b1640534c3d7410fcaafaf2232c 
>   
> clients/src/test/java/org/apache/kafka/clients/producer/internals/BufferPoolTest.java
>  2c693824fa53db1e38766b8c66a0ef42ef9d0f3a 
>   
> clients/src/test/java/org/apache/kafka/clients/producer/internals/RecordAccumulatorTest.java
>  5b2e4ffaeab7127648db608c179703b27b577414 
>   
> clients/src/test/java/org/apache/kafka/clients/producer/internals/SenderTest.java
>  aa44991777a855f4b7f4f7bf17107c69393ff8ff 
>   clients/src/test/java/org/apache/kafka/common/network/SSLSelectorTest.java 
> df1205c935bee9a30a50816dbade64d6014b1ef2 
>   clients/src/test/java/org/apache/kafka/common/network/SelectorTest.java 
> 3a684d98b05cadfb25c6f7f9a038ef1f6697edbf 
>   clients/src/test/java/org/apache/kafka/test/MockSelector.java 
> f83fd9b794a3bd191121a22bcb40fd6ec31d83b5 
>   

Re: Review Request 36858: Patch for KAFKA-2120

[Joel Koshy]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36858/#review101524
---



clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java 
(line 427)


Yeah I don't quite see why it needs more work - Mayuresh - can you comment?


- Joel Koshy


On Sept. 28, 2015, 11:16 p.m., Mayuresh Gharat wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/36858/
> ---
> 
> (Updated Sept. 28, 2015, 11:16 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-2120
> https://issues.apache.org/jira/browse/KAFKA-2120
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> Solved compile error
> 
> 
> Addressed Jason's comments for Kip-19
> 
> 
> Addressed Jun's comments
> 
> 
> Addressed Jason's comments about the default values for requestTimeout
> 
> 
> checkpoint
> 
> 
> Addressed Joel's concerns. Also tried to include Jun's feedback.
> 
> 
> Fixed a minor comment
> 
> 
> Solved unittest issue
> 
> 
> Addressed Jun's comments regarding NetworkClient
> 
> 
> Addressed Jun's comments about disconnect() in Selector
> 
> 
> changed logging level to debug
> 
> 
> Addressed Joels comments to break out early from the loop while aborting 
> expired batches
> 
> 
> Addressed Jun's comments
> 
> 
> Addressed Jason's concern about iterating over timeout request in 
> getNodesWithTimedOutRequest()
> 
> 
> Addressed Joel's comments
> 
> 
> Changed the default timeouts for requesttimeout in KafkaConfig and also made 
> requestrtimeout consistent in KafkaConsumer
> 
> 
> Fixed the forward referencing bug in KafkaConfig.
> 
> 
> Diffs
> -
> 
>   clients/src/main/java/org/apache/kafka/clients/ClientRequest.java 
> dc8f0f115bcda893c95d17c0a57be8d14518d034 
>   clients/src/main/java/org/apache/kafka/clients/CommonClientConfigs.java 
> 7d24c6f5dd2b63b96584f3aa8922a1d048dc1ae4 
>   clients/src/main/java/org/apache/kafka/clients/InFlightRequests.java 
> 15d00d4e484bb5d51a9ae6857ed6e024a2cc1820 
>   clients/src/main/java/org/apache/kafka/clients/KafkaClient.java 
> f46c0d9b5eb73887c62a0e09c96e9d8c964c709d 
>   clients/src/main/java/org/apache/kafka/clients/NetworkClient.java 
> 51a6c5dc47c4008e92b9fdd2ee359b573184e8ed 
>   clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerConfig.java 
> b9a2d4e2bc565f0ee72b27791afe5c894af262f1 
>   clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java 
> 3ac2be840c5d313a497ab67dc024d1578d4679fd 
>   
> clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerNetworkClient.java
>  9517d9d0cd480d5ba1d12f1fde7963e60528d2f8 
>   clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 
> 804d569498396d431880641041fc9292076452cb 
>   clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 
> 06f00a99a73a288df9afa8c1d4abe3580fa968a6 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/BufferPool.java
>  4cb1e50d6c4ed55241aeaef1d3af09def5274103 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java
>  a152bd7697dca55609a9ec4cfe0a82c10595fbc3 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java
>  06182db1c3a5da85648199b4c0c98b80ea7c6c0c 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 
> d2e64f7cd8bf56e433a210905b2874f71eee9ea0 
>   clients/src/main/java/org/apache/kafka/common/network/Selectable.java 
> 70e74bd6aa629c430b2850ca40c97df0b16e5d75 
>   clients/src/main/java/org/apache/kafka/common/network/Selector.java 
> 5a4909e4e5afe1b7ce2c33b9324d3947d2d91a38 
>   clients/src/test/java/org/apache/kafka/clients/MockClient.java 
> e5815f56bdf8e2d980f2bc36b831ed234c0ac781 
>   clients/src/test/java/org/apache/kafka/clients/NetworkClientTest.java 
> 69c93c3adf674b1640534c3d7410fcaafaf2232c 
>   
> clients/src/test/java/org/apache/kafka/clients/producer/internals/BufferPoolTest.java
>  2c693824fa53db1e38766b8c66a0ef42ef9d0f3a 
>   
> clients/src/test/java/org/apache/kafka/clients/producer/internals/RecordAccumulatorTest.java
>  5b2e4ffaeab7127648db608c179703b27b577414 
>   
> clients/src/test/java/org/apache/kafka/clients/producer/internals/SenderTest.java
>  aa44991777a855f4b7f4f7bf17107c69393ff8ff 
>   clients/src/test/java/org/apache/kafka/common/network/SSLSelectorTest.java 
> df1205c935bee9a30a50816dbade64d6014b1ef2 
>   clients/src/test/java/org/apache/kafka/common/network/SelectorTest.java 
> 3a684d98b05cadfb25c6f7f9a038ef1f6697edbf 
>   clients/src/test/java/org/apache/kafka/test/MockSelector.java 
> f83fd9b794a3bd191121a22bcb40fd6ec31d83b5 
>   

Re: Review Request 36858: Patch for KAFKA-2120

[Ismael Juma]

> On Oct. 5, 2015, 6:51 p.m., Guozhang Wang wrote:
> > clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java, 
> > line 427
> > 
> >
> > One minor thing: could you remove this comment?

I had asked for that to be removed too (you can see it in the discussion), but 
Mayuresh said that some more work needed to be done before it could be removed 
(I didn't quite understand it to be honest).


- Ismael


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36858/#review101511
---


On Sept. 28, 2015, 11:16 p.m., Mayuresh Gharat wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/36858/
> ---
> 
> (Updated Sept. 28, 2015, 11:16 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-2120
> https://issues.apache.org/jira/browse/KAFKA-2120
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> Solved compile error
> 
> 
> Addressed Jason's comments for Kip-19
> 
> 
> Addressed Jun's comments
> 
> 
> Addressed Jason's comments about the default values for requestTimeout
> 
> 
> checkpoint
> 
> 
> Addressed Joel's concerns. Also tried to include Jun's feedback.
> 
> 
> Fixed a minor comment
> 
> 
> Solved unittest issue
> 
> 
> Addressed Jun's comments regarding NetworkClient
> 
> 
> Addressed Jun's comments about disconnect() in Selector
> 
> 
> changed logging level to debug
> 
> 
> Addressed Joels comments to break out early from the loop while aborting 
> expired batches
> 
> 
> Addressed Jun's comments
> 
> 
> Addressed Jason's concern about iterating over timeout request in 
> getNodesWithTimedOutRequest()
> 
> 
> Addressed Joel's comments
> 
> 
> Changed the default timeouts for requesttimeout in KafkaConfig and also made 
> requestrtimeout consistent in KafkaConsumer
> 
> 
> Fixed the forward referencing bug in KafkaConfig.
> 
> 
> Diffs
> -
> 
>   clients/src/main/java/org/apache/kafka/clients/ClientRequest.java 
> dc8f0f115bcda893c95d17c0a57be8d14518d034 
>   clients/src/main/java/org/apache/kafka/clients/CommonClientConfigs.java 
> 7d24c6f5dd2b63b96584f3aa8922a1d048dc1ae4 
>   clients/src/main/java/org/apache/kafka/clients/InFlightRequests.java 
> 15d00d4e484bb5d51a9ae6857ed6e024a2cc1820 
>   clients/src/main/java/org/apache/kafka/clients/KafkaClient.java 
> f46c0d9b5eb73887c62a0e09c96e9d8c964c709d 
>   clients/src/main/java/org/apache/kafka/clients/NetworkClient.java 
> 51a6c5dc47c4008e92b9fdd2ee359b573184e8ed 
>   clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerConfig.java 
> b9a2d4e2bc565f0ee72b27791afe5c894af262f1 
>   clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java 
> 3ac2be840c5d313a497ab67dc024d1578d4679fd 
>   
> clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerNetworkClient.java
>  9517d9d0cd480d5ba1d12f1fde7963e60528d2f8 
>   clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 
> 804d569498396d431880641041fc9292076452cb 
>   clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 
> 06f00a99a73a288df9afa8c1d4abe3580fa968a6 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/BufferPool.java
>  4cb1e50d6c4ed55241aeaef1d3af09def5274103 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java
>  a152bd7697dca55609a9ec4cfe0a82c10595fbc3 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java
>  06182db1c3a5da85648199b4c0c98b80ea7c6c0c 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 
> d2e64f7cd8bf56e433a210905b2874f71eee9ea0 
>   clients/src/main/java/org/apache/kafka/common/network/Selectable.java 
> 70e74bd6aa629c430b2850ca40c97df0b16e5d75 
>   clients/src/main/java/org/apache/kafka/common/network/Selector.java 
> 5a4909e4e5afe1b7ce2c33b9324d3947d2d91a38 
>   clients/src/test/java/org/apache/kafka/clients/MockClient.java 
> e5815f56bdf8e2d980f2bc36b831ed234c0ac781 
>   clients/src/test/java/org/apache/kafka/clients/NetworkClientTest.java 
> 69c93c3adf674b1640534c3d7410fcaafaf2232c 
>   
> clients/src/test/java/org/apache/kafka/clients/producer/internals/BufferPoolTest.java
>  2c693824fa53db1e38766b8c66a0ef42ef9d0f3a 
>   
> clients/src/test/java/org/apache/kafka/clients/producer/internals/RecordAccumulatorTest.java
>  5b2e4ffaeab7127648db608c179703b27b577414 
>   
> clients/src/test/java/org/apache/kafka/clients/producer/internals/SenderTest.java
>  aa44991777a855f4b7f4f7bf17107c69393ff8ff 
>   clients/src/test/java/org/apache/kafka/common/network/SSLSelectorTest.java 
> df1205c935bee9a30a50816dbade64d6014b1ef2 
>   

Re: Review Request 36858: Patch for KAFKA-2120

[Guozhang Wang]

> On Oct. 5, 2015, 8:34 p.m., Joel Koshy wrote:
> > clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java, 
> > line 427
> > 
> >
> > LGTM - Guozhang, can you do a trivial commit to remove it? (Or I can do 
> > it later today/tomorrow)

Thanks Joel for looking, I can do the trival commit.


- Guozhang


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36858/#review101520
---


On Sept. 28, 2015, 11:16 p.m., Mayuresh Gharat wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/36858/
> ---
> 
> (Updated Sept. 28, 2015, 11:16 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-2120
> https://issues.apache.org/jira/browse/KAFKA-2120
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> Solved compile error
> 
> 
> Addressed Jason's comments for Kip-19
> 
> 
> Addressed Jun's comments
> 
> 
> Addressed Jason's comments about the default values for requestTimeout
> 
> 
> checkpoint
> 
> 
> Addressed Joel's concerns. Also tried to include Jun's feedback.
> 
> 
> Fixed a minor comment
> 
> 
> Solved unittest issue
> 
> 
> Addressed Jun's comments regarding NetworkClient
> 
> 
> Addressed Jun's comments about disconnect() in Selector
> 
> 
> changed logging level to debug
> 
> 
> Addressed Joels comments to break out early from the loop while aborting 
> expired batches
> 
> 
> Addressed Jun's comments
> 
> 
> Addressed Jason's concern about iterating over timeout request in 
> getNodesWithTimedOutRequest()
> 
> 
> Addressed Joel's comments
> 
> 
> Changed the default timeouts for requesttimeout in KafkaConfig and also made 
> requestrtimeout consistent in KafkaConsumer
> 
> 
> Fixed the forward referencing bug in KafkaConfig.
> 
> 
> Diffs
> -
> 
>   clients/src/main/java/org/apache/kafka/clients/ClientRequest.java 
> dc8f0f115bcda893c95d17c0a57be8d14518d034 
>   clients/src/main/java/org/apache/kafka/clients/CommonClientConfigs.java 
> 7d24c6f5dd2b63b96584f3aa8922a1d048dc1ae4 
>   clients/src/main/java/org/apache/kafka/clients/InFlightRequests.java 
> 15d00d4e484bb5d51a9ae6857ed6e024a2cc1820 
>   clients/src/main/java/org/apache/kafka/clients/KafkaClient.java 
> f46c0d9b5eb73887c62a0e09c96e9d8c964c709d 
>   clients/src/main/java/org/apache/kafka/clients/NetworkClient.java 
> 51a6c5dc47c4008e92b9fdd2ee359b573184e8ed 
>   clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerConfig.java 
> b9a2d4e2bc565f0ee72b27791afe5c894af262f1 
>   clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java 
> 3ac2be840c5d313a497ab67dc024d1578d4679fd 
>   
> clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerNetworkClient.java
>  9517d9d0cd480d5ba1d12f1fde7963e60528d2f8 
>   clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 
> 804d569498396d431880641041fc9292076452cb 
>   clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 
> 06f00a99a73a288df9afa8c1d4abe3580fa968a6 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/BufferPool.java
>  4cb1e50d6c4ed55241aeaef1d3af09def5274103 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java
>  a152bd7697dca55609a9ec4cfe0a82c10595fbc3 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java
>  06182db1c3a5da85648199b4c0c98b80ea7c6c0c 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 
> d2e64f7cd8bf56e433a210905b2874f71eee9ea0 
>   clients/src/main/java/org/apache/kafka/common/network/Selectable.java 
> 70e74bd6aa629c430b2850ca40c97df0b16e5d75 
>   clients/src/main/java/org/apache/kafka/common/network/Selector.java 
> 5a4909e4e5afe1b7ce2c33b9324d3947d2d91a38 
>   clients/src/test/java/org/apache/kafka/clients/MockClient.java 
> e5815f56bdf8e2d980f2bc36b831ed234c0ac781 
>   clients/src/test/java/org/apache/kafka/clients/NetworkClientTest.java 
> 69c93c3adf674b1640534c3d7410fcaafaf2232c 
>   
> clients/src/test/java/org/apache/kafka/clients/producer/internals/BufferPoolTest.java
>  2c693824fa53db1e38766b8c66a0ef42ef9d0f3a 
>   
> clients/src/test/java/org/apache/kafka/clients/producer/internals/RecordAccumulatorTest.java
>  5b2e4ffaeab7127648db608c179703b27b577414 
>   
> clients/src/test/java/org/apache/kafka/clients/producer/internals/SenderTest.java
>  aa44991777a855f4b7f4f7bf17107c69393ff8ff 
>   clients/src/test/java/org/apache/kafka/common/network/SSLSelectorTest.java 
> df1205c935bee9a30a50816dbade64d6014b1ef2 
>   clients/src/test/java/org/apache/kafka/common/network/SelectorTest.java 
> 

Re: Review Request 36858: Patch for KAFKA-2120

[Mayuresh Gharat]

> On Oct. 5, 2015, 6:51 p.m., Guozhang Wang wrote:
> > clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java, 
> > line 427
> > 
> >
> > One minor thing: could you remove this comment?
> 
> Ismael Juma wrote:
> I had asked for that to be removed too (you can see it in the 
> discussion), but Mayuresh said that some more work needed to be done before 
> it could be removed (I didn't quite understand it to be honest).

We have a ticket KAFKA-2428 which needs to be handled after KIP-19. 
The reason for that ticket is availble on KAFKA-2120 Jira. 

This was the brief discussion on the ticket KAFKA-2120:

The request time is for exception handling purpose, the request timeout should 
be the highest timeout across all the timeout. Can we just sanity check the 
configurations user provides? The rules should be:
Request timeout > session timeout.
Request timeout > fetch.max.wait.timeout
request timeout won't kick in before the other timeout reached.
The sanity check patch should be relatively simple. We can either do it in this 
patch or in a separate patch.

I created KAFKA-2428 to handle this.

I have explained about discrepency of long/int datatype of requestTimeout in 
kafkaConsumer and sanity checks on Ismael's comments above. Probably I should 
have listed the Jira ticket there for the sanity checks which would have made 
this more clear.

The TODO was left as it is to remind us that there is still work needed to be 
done after KIP-19 for KafkaConsumer on usage of requestTimeout.


- Mayuresh


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36858/#review101511
---


On Sept. 28, 2015, 11:16 p.m., Mayuresh Gharat wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/36858/
> ---
> 
> (Updated Sept. 28, 2015, 11:16 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-2120
> https://issues.apache.org/jira/browse/KAFKA-2120
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> Solved compile error
> 
> 
> Addressed Jason's comments for Kip-19
> 
> 
> Addressed Jun's comments
> 
> 
> Addressed Jason's comments about the default values for requestTimeout
> 
> 
> checkpoint
> 
> 
> Addressed Joel's concerns. Also tried to include Jun's feedback.
> 
> 
> Fixed a minor comment
> 
> 
> Solved unittest issue
> 
> 
> Addressed Jun's comments regarding NetworkClient
> 
> 
> Addressed Jun's comments about disconnect() in Selector
> 
> 
> changed logging level to debug
> 
> 
> Addressed Joels comments to break out early from the loop while aborting 
> expired batches
> 
> 
> Addressed Jun's comments
> 
> 
> Addressed Jason's concern about iterating over timeout request in 
> getNodesWithTimedOutRequest()
> 
> 
> Addressed Joel's comments
> 
> 
> Changed the default timeouts for requesttimeout in KafkaConfig and also made 
> requestrtimeout consistent in KafkaConsumer
> 
> 
> Fixed the forward referencing bug in KafkaConfig.
> 
> 
> Diffs
> -
> 
>   clients/src/main/java/org/apache/kafka/clients/ClientRequest.java 
> dc8f0f115bcda893c95d17c0a57be8d14518d034 
>   clients/src/main/java/org/apache/kafka/clients/CommonClientConfigs.java 
> 7d24c6f5dd2b63b96584f3aa8922a1d048dc1ae4 
>   clients/src/main/java/org/apache/kafka/clients/InFlightRequests.java 
> 15d00d4e484bb5d51a9ae6857ed6e024a2cc1820 
>   clients/src/main/java/org/apache/kafka/clients/KafkaClient.java 
> f46c0d9b5eb73887c62a0e09c96e9d8c964c709d 
>   clients/src/main/java/org/apache/kafka/clients/NetworkClient.java 
> 51a6c5dc47c4008e92b9fdd2ee359b573184e8ed 
>   clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerConfig.java 
> b9a2d4e2bc565f0ee72b27791afe5c894af262f1 
>   clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java 
> 3ac2be840c5d313a497ab67dc024d1578d4679fd 
>   
> clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerNetworkClient.java
>  9517d9d0cd480d5ba1d12f1fde7963e60528d2f8 
>   clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 
> 804d569498396d431880641041fc9292076452cb 
>   clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 
> 06f00a99a73a288df9afa8c1d4abe3580fa968a6 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/BufferPool.java
>  4cb1e50d6c4ed55241aeaef1d3af09def5274103 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java
>  a152bd7697dca55609a9ec4cfe0a82c10595fbc3 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java
>  06182db1c3a5da85648199b4c0c98b80ea7c6c0c 
>   
> 

Re: Review Request 36858: Patch for KAFKA-2120

[Joel Koshy]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36858/#review101520
---



clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java 
(line 427)


LGTM - Guozhang, can you do a trivial commit to remove it? (Or I can do it 
later today/tomorrow)


- Joel Koshy


On Sept. 28, 2015, 11:16 p.m., Mayuresh Gharat wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/36858/
> ---
> 
> (Updated Sept. 28, 2015, 11:16 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-2120
> https://issues.apache.org/jira/browse/KAFKA-2120
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> Solved compile error
> 
> 
> Addressed Jason's comments for Kip-19
> 
> 
> Addressed Jun's comments
> 
> 
> Addressed Jason's comments about the default values for requestTimeout
> 
> 
> checkpoint
> 
> 
> Addressed Joel's concerns. Also tried to include Jun's feedback.
> 
> 
> Fixed a minor comment
> 
> 
> Solved unittest issue
> 
> 
> Addressed Jun's comments regarding NetworkClient
> 
> 
> Addressed Jun's comments about disconnect() in Selector
> 
> 
> changed logging level to debug
> 
> 
> Addressed Joels comments to break out early from the loop while aborting 
> expired batches
> 
> 
> Addressed Jun's comments
> 
> 
> Addressed Jason's concern about iterating over timeout request in 
> getNodesWithTimedOutRequest()
> 
> 
> Addressed Joel's comments
> 
> 
> Changed the default timeouts for requesttimeout in KafkaConfig and also made 
> requestrtimeout consistent in KafkaConsumer
> 
> 
> Fixed the forward referencing bug in KafkaConfig.
> 
> 
> Diffs
> -
> 
>   clients/src/main/java/org/apache/kafka/clients/ClientRequest.java 
> dc8f0f115bcda893c95d17c0a57be8d14518d034 
>   clients/src/main/java/org/apache/kafka/clients/CommonClientConfigs.java 
> 7d24c6f5dd2b63b96584f3aa8922a1d048dc1ae4 
>   clients/src/main/java/org/apache/kafka/clients/InFlightRequests.java 
> 15d00d4e484bb5d51a9ae6857ed6e024a2cc1820 
>   clients/src/main/java/org/apache/kafka/clients/KafkaClient.java 
> f46c0d9b5eb73887c62a0e09c96e9d8c964c709d 
>   clients/src/main/java/org/apache/kafka/clients/NetworkClient.java 
> 51a6c5dc47c4008e92b9fdd2ee359b573184e8ed 
>   clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerConfig.java 
> b9a2d4e2bc565f0ee72b27791afe5c894af262f1 
>   clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java 
> 3ac2be840c5d313a497ab67dc024d1578d4679fd 
>   
> clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerNetworkClient.java
>  9517d9d0cd480d5ba1d12f1fde7963e60528d2f8 
>   clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 
> 804d569498396d431880641041fc9292076452cb 
>   clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 
> 06f00a99a73a288df9afa8c1d4abe3580fa968a6 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/BufferPool.java
>  4cb1e50d6c4ed55241aeaef1d3af09def5274103 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java
>  a152bd7697dca55609a9ec4cfe0a82c10595fbc3 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java
>  06182db1c3a5da85648199b4c0c98b80ea7c6c0c 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 
> d2e64f7cd8bf56e433a210905b2874f71eee9ea0 
>   clients/src/main/java/org/apache/kafka/common/network/Selectable.java 
> 70e74bd6aa629c430b2850ca40c97df0b16e5d75 
>   clients/src/main/java/org/apache/kafka/common/network/Selector.java 
> 5a4909e4e5afe1b7ce2c33b9324d3947d2d91a38 
>   clients/src/test/java/org/apache/kafka/clients/MockClient.java 
> e5815f56bdf8e2d980f2bc36b831ed234c0ac781 
>   clients/src/test/java/org/apache/kafka/clients/NetworkClientTest.java 
> 69c93c3adf674b1640534c3d7410fcaafaf2232c 
>   
> clients/src/test/java/org/apache/kafka/clients/producer/internals/BufferPoolTest.java
>  2c693824fa53db1e38766b8c66a0ef42ef9d0f3a 
>   
> clients/src/test/java/org/apache/kafka/clients/producer/internals/RecordAccumulatorTest.java
>  5b2e4ffaeab7127648db608c179703b27b577414 
>   
> clients/src/test/java/org/apache/kafka/clients/producer/internals/SenderTest.java
>  aa44991777a855f4b7f4f7bf17107c69393ff8ff 
>   clients/src/test/java/org/apache/kafka/common/network/SSLSelectorTest.java 
> df1205c935bee9a30a50816dbade64d6014b1ef2 
>   clients/src/test/java/org/apache/kafka/common/network/SelectorTest.java 
> 3a684d98b05cadfb25c6f7f9a038ef1f6697edbf 
>   clients/src/test/java/org/apache/kafka/test/MockSelector.java 
> f83fd9b794a3bd191121a22bcb40fd6ec31d83b5 
>   

Re: Review Request 36858: Patch for KAFKA-2120

[Mayuresh Gharat]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36858/#review100885
---



core/src/main/scala/kafka/controller/ControllerChannelManager.scala (line 109)


The request time outs in those classes are mainly used for Select operation 
in the Selector or timing out the blocking method. The requestTimeout in KIP-19 
is used for already in-flight requests (on the wire, and regardless of the 
select timeout). It should be possible to combine these but that will likely 
require a callback to handle the KIP-19 timeout and I think that should really 
be a separate jira.


- Mayuresh Gharat


On Sept. 19, 2015, 2:27 a.m., Mayuresh Gharat wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/36858/
> ---
> 
> (Updated Sept. 19, 2015, 2:27 a.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-2120
> https://issues.apache.org/jira/browse/KAFKA-2120
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> Solved compile error
> 
> 
> Addressed Jason's comments for Kip-19
> 
> 
> Addressed Jun's comments
> 
> 
> Addressed Jason's comments about the default values for requestTimeout
> 
> 
> checkpoint
> 
> 
> Addressed Joel's concerns. Also tried to include Jun's feedback.
> 
> 
> Fixed a minor comment
> 
> 
> Solved unittest issue
> 
> 
> Addressed Jun's comments regarding NetworkClient
> 
> 
> Addressed Jun's comments about disconnect() in Selector
> 
> 
> changed logging level to debug
> 
> 
> Addressed Joels comments to break out early from the loop while aborting 
> expired batches
> 
> 
> Addressed Jun's comments
> 
> 
> Addressed Jason's concern about iterating over timeout request in 
> getNodesWithTimedOutRequest()
> 
> 
> Addressed Joel's comments
> 
> 
> Changed the default timeouts for requesttimeout in KafkaConfig and also made 
> requestrtimeout consistent in KafkaConsumer
> 
> 
> Diffs
> -
> 
>   clients/src/main/java/org/apache/kafka/clients/ClientRequest.java 
> dc8f0f115bcda893c95d17c0a57be8d14518d034 
>   clients/src/main/java/org/apache/kafka/clients/CommonClientConfigs.java 
> 7d24c6f5dd2b63b96584f3aa8922a1d048dc1ae4 
>   clients/src/main/java/org/apache/kafka/clients/InFlightRequests.java 
> 15d00d4e484bb5d51a9ae6857ed6e024a2cc1820 
>   clients/src/main/java/org/apache/kafka/clients/KafkaClient.java 
> f46c0d9b5eb73887c62a0e09c96e9d8c964c709d 
>   clients/src/main/java/org/apache/kafka/clients/NetworkClient.java 
> 51a6c5dc47c4008e92b9fdd2ee359b573184e8ed 
>   clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerConfig.java 
> b9a2d4e2bc565f0ee72b27791afe5c894af262f1 
>   clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java 
> 3ac2be840c5d313a497ab67dc024d1578d4679fd 
>   
> clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerNetworkClient.java
>  9517d9d0cd480d5ba1d12f1fde7963e60528d2f8 
>   clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 
> 804d569498396d431880641041fc9292076452cb 
>   clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 
> 06f00a99a73a288df9afa8c1d4abe3580fa968a6 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/BufferPool.java
>  4cb1e50d6c4ed55241aeaef1d3af09def5274103 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java
>  a152bd7697dca55609a9ec4cfe0a82c10595fbc3 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java
>  06182db1c3a5da85648199b4c0c98b80ea7c6c0c 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 
> d2e64f7cd8bf56e433a210905b2874f71eee9ea0 
>   clients/src/main/java/org/apache/kafka/common/network/Selectable.java 
> 70e74bd6aa629c430b2850ca40c97df0b16e5d75 
>   clients/src/main/java/org/apache/kafka/common/network/Selector.java 
> 5a4909e4e5afe1b7ce2c33b9324d3947d2d91a38 
>   clients/src/test/java/org/apache/kafka/clients/MockClient.java 
> e5815f56bdf8e2d980f2bc36b831ed234c0ac781 
>   clients/src/test/java/org/apache/kafka/clients/NetworkClientTest.java 
> 69c93c3adf674b1640534c3d7410fcaafaf2232c 
>   
> clients/src/test/java/org/apache/kafka/clients/producer/internals/BufferPoolTest.java
>  2c693824fa53db1e38766b8c66a0ef42ef9d0f3a 
>   
> clients/src/test/java/org/apache/kafka/clients/producer/internals/RecordAccumulatorTest.java
>  5b2e4ffaeab7127648db608c179703b27b577414 
>   
> clients/src/test/java/org/apache/kafka/clients/producer/internals/SenderTest.java
>  aa44991777a855f4b7f4f7bf17107c69393ff8ff 
>   clients/src/test/java/org/apache/kafka/common/network/SSLSelectorTest.java 
> df1205c935bee9a30a50816dbade64d6014b1ef2 
>   

Re: Review Request 36858: Patch for KAFKA-2120

[Mayuresh Gharat]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36858/
---

(Updated Sept. 28, 2015, 11:13 p.m.)


Review request for kafka.


Bugs: KAFKA-2120
https://issues.apache.org/jira/browse/KAFKA-2120


Repository: kafka


Description (updated)
---

Solved compile error


Addressed Jason's comments for Kip-19


Addressed Jun's comments


Addressed Jason's comments about the default values for requestTimeout


checkpoint


Addressed Joel's concerns. Also tried to include Jun's feedback.


Fixed a minor comment


Solved unittest issue


Addressed Jun's comments regarding NetworkClient


Addressed Jun's comments about disconnect() in Selector


changed logging level to debug


Addressed Joels comments to break out early from the loop while aborting 
expired batches


Addressed Jun's comments


Addressed Jason's concern about iterating over timeout request in 
getNodesWithTimedOutRequest()


Addressed Joel's comments


Changed the default timeouts for requesttimeout in KafkaConfig and also made 
requestrtimeout consistent in KafkaConsumer


Fixed the forward referencing bug in KafkaConfig.


Diffs (updated)
-

  clients/src/main/java/org/apache/kafka/clients/ClientRequest.java 
dc8f0f115bcda893c95d17c0a57be8d14518d034 
  clients/src/main/java/org/apache/kafka/clients/CommonClientConfigs.java 
7d24c6f5dd2b63b96584f3aa8922a1d048dc1ae4 
  clients/src/main/java/org/apache/kafka/clients/InFlightRequests.java 
15d00d4e484bb5d51a9ae6857ed6e024a2cc1820 
  clients/src/main/java/org/apache/kafka/clients/KafkaClient.java 
f46c0d9b5eb73887c62a0e09c96e9d8c964c709d 
  clients/src/main/java/org/apache/kafka/clients/NetworkClient.java 
51a6c5dc47c4008e92b9fdd2ee359b573184e8ed 
  clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerConfig.java 
b9a2d4e2bc565f0ee72b27791afe5c894af262f1 
  clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java 
3ac2be840c5d313a497ab67dc024d1578d4679fd 
  
clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerNetworkClient.java
 9517d9d0cd480d5ba1d12f1fde7963e60528d2f8 
  clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 
804d569498396d431880641041fc9292076452cb 
  clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 
06f00a99a73a288df9afa8c1d4abe3580fa968a6 
  
clients/src/main/java/org/apache/kafka/clients/producer/internals/BufferPool.java
 4cb1e50d6c4ed55241aeaef1d3af09def5274103 
  
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java
 a152bd7697dca55609a9ec4cfe0a82c10595fbc3 
  
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java
 06182db1c3a5da85648199b4c0c98b80ea7c6c0c 
  clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 
d2e64f7cd8bf56e433a210905b2874f71eee9ea0 
  clients/src/main/java/org/apache/kafka/common/network/Selectable.java 
70e74bd6aa629c430b2850ca40c97df0b16e5d75 
  clients/src/main/java/org/apache/kafka/common/network/Selector.java 
5a4909e4e5afe1b7ce2c33b9324d3947d2d91a38 
  clients/src/test/java/org/apache/kafka/clients/MockClient.java 
e5815f56bdf8e2d980f2bc36b831ed234c0ac781 
  clients/src/test/java/org/apache/kafka/clients/NetworkClientTest.java 
69c93c3adf674b1640534c3d7410fcaafaf2232c 
  
clients/src/test/java/org/apache/kafka/clients/producer/internals/BufferPoolTest.java
 2c693824fa53db1e38766b8c66a0ef42ef9d0f3a 
  
clients/src/test/java/org/apache/kafka/clients/producer/internals/RecordAccumulatorTest.java
 5b2e4ffaeab7127648db608c179703b27b577414 
  
clients/src/test/java/org/apache/kafka/clients/producer/internals/SenderTest.java
 aa44991777a855f4b7f4f7bf17107c69393ff8ff 
  clients/src/test/java/org/apache/kafka/common/network/SSLSelectorTest.java 
df1205c935bee9a30a50816dbade64d6014b1ef2 
  clients/src/test/java/org/apache/kafka/common/network/SelectorTest.java 
3a684d98b05cadfb25c6f7f9a038ef1f6697edbf 
  clients/src/test/java/org/apache/kafka/test/MockSelector.java 
f83fd9b794a3bd191121a22bcb40fd6ec31d83b5 
  core/src/main/scala/kafka/controller/ControllerChannelManager.scala 
b1cf668dbb5bc8617d320fd3a307c6c8005b78a2 
  core/src/main/scala/kafka/server/KafkaConfig.scala 
1e8b2331486ffe55bfcc0919e48e12aad23b7d3c 
  core/src/main/scala/kafka/server/KafkaServer.scala 
f3f1fa653b9c85e729cbe32b63669011ada46547 
  core/src/main/scala/kafka/server/ReplicaFetcherThread.scala 
6c85e52e172bfc0278e724068a48a271bc15dc9f 
  core/src/main/scala/kafka/tools/ProducerPerformance.scala 
46a68e97b8bcc8821f21e4220ce9b3acedc5dafe 
  core/src/main/scala/kafka/utils/NetworkClientBlockingOps.scala 
ad10721de844725f27a116611209992cea61b088 
  core/src/test/scala/integration/kafka/api/ProducerFailureHandlingTest.scala 
1198df02ddd7727269e84a751ba99520f6d5584a 
  core/src/test/scala/unit/kafka/server/KafkaConfigTest.scala 

Re: Review Request 36858: Patch for KAFKA-2120

[Joel Koshy]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36858/#review100880
---



core/src/main/scala/kafka/server/KafkaConfig.scala (line 67)


This change is actually sort of unnecessary, but there is a bug in the 
earlier (pre-existing code). Basically, the default `RequestTimeoutMs` is 
always initialized to zero due to a forward reference to 
`ControllerSocketTimeoutMs` which leads to initialization issues - you can 
verify this by enabling logs in the unit test for `LogRecoveryTest`. You can 
address this by explicitly setting `RequestTimeoutMs` and then `val 
ControllerSocketTimeoutMs = RequestTimeoutMs`.


- Joel Koshy


On Sept. 19, 2015, 2:27 a.m., Mayuresh Gharat wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/36858/
> ---
> 
> (Updated Sept. 19, 2015, 2:27 a.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-2120
> https://issues.apache.org/jira/browse/KAFKA-2120
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> Solved compile error
> 
> 
> Addressed Jason's comments for Kip-19
> 
> 
> Addressed Jun's comments
> 
> 
> Addressed Jason's comments about the default values for requestTimeout
> 
> 
> checkpoint
> 
> 
> Addressed Joel's concerns. Also tried to include Jun's feedback.
> 
> 
> Fixed a minor comment
> 
> 
> Solved unittest issue
> 
> 
> Addressed Jun's comments regarding NetworkClient
> 
> 
> Addressed Jun's comments about disconnect() in Selector
> 
> 
> changed logging level to debug
> 
> 
> Addressed Joels comments to break out early from the loop while aborting 
> expired batches
> 
> 
> Addressed Jun's comments
> 
> 
> Addressed Jason's concern about iterating over timeout request in 
> getNodesWithTimedOutRequest()
> 
> 
> Addressed Joel's comments
> 
> 
> Changed the default timeouts for requesttimeout in KafkaConfig and also made 
> requestrtimeout consistent in KafkaConsumer
> 
> 
> Diffs
> -
> 
>   clients/src/main/java/org/apache/kafka/clients/ClientRequest.java 
> dc8f0f115bcda893c95d17c0a57be8d14518d034 
>   clients/src/main/java/org/apache/kafka/clients/CommonClientConfigs.java 
> 7d24c6f5dd2b63b96584f3aa8922a1d048dc1ae4 
>   clients/src/main/java/org/apache/kafka/clients/InFlightRequests.java 
> 15d00d4e484bb5d51a9ae6857ed6e024a2cc1820 
>   clients/src/main/java/org/apache/kafka/clients/KafkaClient.java 
> f46c0d9b5eb73887c62a0e09c96e9d8c964c709d 
>   clients/src/main/java/org/apache/kafka/clients/NetworkClient.java 
> 51a6c5dc47c4008e92b9fdd2ee359b573184e8ed 
>   clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerConfig.java 
> b9a2d4e2bc565f0ee72b27791afe5c894af262f1 
>   clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java 
> 3ac2be840c5d313a497ab67dc024d1578d4679fd 
>   
> clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerNetworkClient.java
>  9517d9d0cd480d5ba1d12f1fde7963e60528d2f8 
>   clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 
> 804d569498396d431880641041fc9292076452cb 
>   clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 
> 06f00a99a73a288df9afa8c1d4abe3580fa968a6 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/BufferPool.java
>  4cb1e50d6c4ed55241aeaef1d3af09def5274103 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java
>  a152bd7697dca55609a9ec4cfe0a82c10595fbc3 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java
>  06182db1c3a5da85648199b4c0c98b80ea7c6c0c 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 
> d2e64f7cd8bf56e433a210905b2874f71eee9ea0 
>   clients/src/main/java/org/apache/kafka/common/network/Selectable.java 
> 70e74bd6aa629c430b2850ca40c97df0b16e5d75 
>   clients/src/main/java/org/apache/kafka/common/network/Selector.java 
> 5a4909e4e5afe1b7ce2c33b9324d3947d2d91a38 
>   clients/src/test/java/org/apache/kafka/clients/MockClient.java 
> e5815f56bdf8e2d980f2bc36b831ed234c0ac781 
>   clients/src/test/java/org/apache/kafka/clients/NetworkClientTest.java 
> 69c93c3adf674b1640534c3d7410fcaafaf2232c 
>   
> clients/src/test/java/org/apache/kafka/clients/producer/internals/BufferPoolTest.java
>  2c693824fa53db1e38766b8c66a0ef42ef9d0f3a 
>   
> clients/src/test/java/org/apache/kafka/clients/producer/internals/RecordAccumulatorTest.java
>  5b2e4ffaeab7127648db608c179703b27b577414 
>   
> clients/src/test/java/org/apache/kafka/clients/producer/internals/SenderTest.java
>  aa44991777a855f4b7f4f7bf17107c69393ff8ff 
>   clients/src/test/java/org/apache/kafka/common/network/SSLSelectorTest.java 
> 

Re: Review Request 36858: Patch for KAFKA-2120

[Mayuresh Gharat]

> On Sept. 28, 2015, 11:04 p.m., Mayuresh Gharat wrote:
> > core/src/main/scala/kafka/controller/ControllerChannelManager.scala, line 
> > 109
> > 
> >
> > The request time outs in those classes are mainly used for Select 
> > operation in the Selector or timing out the blocking method. The 
> > requestTimeout in KIP-19 is used for already in-flight requests (on the 
> > wire, and regardless of the select timeout). It should be possible to 
> > combine these but that will likely require a callback to handle the KIP-19 
> > timeout and I think that should really be a separate jira.
> 
> Ismael Juma wrote:
> I am not sure I agree with the distinction. In those classes, we send a 
> request and then we poll until we receive a response or the time out expires 
> (the other use-case is when we wait for a connection to be established, but 
> that's different, yes). A follow-up JIRA sounds good (as it would be great to 
> get this patch in), but I think it would be good to look into it sooner 
> rather than later.

Hi Ismael,
Sure. I will create a ticket to follow this up.


- Mayuresh


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36858/#review100885
---


On Sept. 28, 2015, 11:16 p.m., Mayuresh Gharat wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/36858/
> ---
> 
> (Updated Sept. 28, 2015, 11:16 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-2120
> https://issues.apache.org/jira/browse/KAFKA-2120
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> Solved compile error
> 
> 
> Addressed Jason's comments for Kip-19
> 
> 
> Addressed Jun's comments
> 
> 
> Addressed Jason's comments about the default values for requestTimeout
> 
> 
> checkpoint
> 
> 
> Addressed Joel's concerns. Also tried to include Jun's feedback.
> 
> 
> Fixed a minor comment
> 
> 
> Solved unittest issue
> 
> 
> Addressed Jun's comments regarding NetworkClient
> 
> 
> Addressed Jun's comments about disconnect() in Selector
> 
> 
> changed logging level to debug
> 
> 
> Addressed Joels comments to break out early from the loop while aborting 
> expired batches
> 
> 
> Addressed Jun's comments
> 
> 
> Addressed Jason's concern about iterating over timeout request in 
> getNodesWithTimedOutRequest()
> 
> 
> Addressed Joel's comments
> 
> 
> Changed the default timeouts for requesttimeout in KafkaConfig and also made 
> requestrtimeout consistent in KafkaConsumer
> 
> 
> Fixed the forward referencing bug in KafkaConfig.
> 
> 
> Diffs
> -
> 
>   clients/src/main/java/org/apache/kafka/clients/ClientRequest.java 
> dc8f0f115bcda893c95d17c0a57be8d14518d034 
>   clients/src/main/java/org/apache/kafka/clients/CommonClientConfigs.java 
> 7d24c6f5dd2b63b96584f3aa8922a1d048dc1ae4 
>   clients/src/main/java/org/apache/kafka/clients/InFlightRequests.java 
> 15d00d4e484bb5d51a9ae6857ed6e024a2cc1820 
>   clients/src/main/java/org/apache/kafka/clients/KafkaClient.java 
> f46c0d9b5eb73887c62a0e09c96e9d8c964c709d 
>   clients/src/main/java/org/apache/kafka/clients/NetworkClient.java 
> 51a6c5dc47c4008e92b9fdd2ee359b573184e8ed 
>   clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerConfig.java 
> b9a2d4e2bc565f0ee72b27791afe5c894af262f1 
>   clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java 
> 3ac2be840c5d313a497ab67dc024d1578d4679fd 
>   
> clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerNetworkClient.java
>  9517d9d0cd480d5ba1d12f1fde7963e60528d2f8 
>   clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 
> 804d569498396d431880641041fc9292076452cb 
>   clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 
> 06f00a99a73a288df9afa8c1d4abe3580fa968a6 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/BufferPool.java
>  4cb1e50d6c4ed55241aeaef1d3af09def5274103 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java
>  a152bd7697dca55609a9ec4cfe0a82c10595fbc3 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java
>  06182db1c3a5da85648199b4c0c98b80ea7c6c0c 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 
> d2e64f7cd8bf56e433a210905b2874f71eee9ea0 
>   clients/src/main/java/org/apache/kafka/common/network/Selectable.java 
> 70e74bd6aa629c430b2850ca40c97df0b16e5d75 
>   clients/src/main/java/org/apache/kafka/common/network/Selector.java 
> 5a4909e4e5afe1b7ce2c33b9324d3947d2d91a38 
>   clients/src/test/java/org/apache/kafka/clients/MockClient.java 
> 

Re: Review Request 36858: Patch for KAFKA-2120

[Ismael Juma]

> On Sept. 28, 2015, 11:04 p.m., Mayuresh Gharat wrote:
> > core/src/main/scala/kafka/controller/ControllerChannelManager.scala, line 
> > 109
> > 
> >
> > The request time outs in those classes are mainly used for Select 
> > operation in the Selector or timing out the blocking method. The 
> > requestTimeout in KIP-19 is used for already in-flight requests (on the 
> > wire, and regardless of the select timeout). It should be possible to 
> > combine these but that will likely require a callback to handle the KIP-19 
> > timeout and I think that should really be a separate jira.

I am not sure I agree with the distinction. In those classes, we send a request 
and then we poll until we receive a response or the time out expires (the other 
use-case is when we wait for a connection to be established, but that's 
different, yes). A follow-up JIRA sounds good (as it would be great to get this 
patch in), but I think it would be good to look into it sooner rather than 
later.


- Ismael


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36858/#review100885
---


On Sept. 28, 2015, 11:16 p.m., Mayuresh Gharat wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/36858/
> ---
> 
> (Updated Sept. 28, 2015, 11:16 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-2120
> https://issues.apache.org/jira/browse/KAFKA-2120
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> Solved compile error
> 
> 
> Addressed Jason's comments for Kip-19
> 
> 
> Addressed Jun's comments
> 
> 
> Addressed Jason's comments about the default values for requestTimeout
> 
> 
> checkpoint
> 
> 
> Addressed Joel's concerns. Also tried to include Jun's feedback.
> 
> 
> Fixed a minor comment
> 
> 
> Solved unittest issue
> 
> 
> Addressed Jun's comments regarding NetworkClient
> 
> 
> Addressed Jun's comments about disconnect() in Selector
> 
> 
> changed logging level to debug
> 
> 
> Addressed Joels comments to break out early from the loop while aborting 
> expired batches
> 
> 
> Addressed Jun's comments
> 
> 
> Addressed Jason's concern about iterating over timeout request in 
> getNodesWithTimedOutRequest()
> 
> 
> Addressed Joel's comments
> 
> 
> Changed the default timeouts for requesttimeout in KafkaConfig and also made 
> requestrtimeout consistent in KafkaConsumer
> 
> 
> Fixed the forward referencing bug in KafkaConfig.
> 
> 
> Diffs
> -
> 
>   clients/src/main/java/org/apache/kafka/clients/ClientRequest.java 
> dc8f0f115bcda893c95d17c0a57be8d14518d034 
>   clients/src/main/java/org/apache/kafka/clients/CommonClientConfigs.java 
> 7d24c6f5dd2b63b96584f3aa8922a1d048dc1ae4 
>   clients/src/main/java/org/apache/kafka/clients/InFlightRequests.java 
> 15d00d4e484bb5d51a9ae6857ed6e024a2cc1820 
>   clients/src/main/java/org/apache/kafka/clients/KafkaClient.java 
> f46c0d9b5eb73887c62a0e09c96e9d8c964c709d 
>   clients/src/main/java/org/apache/kafka/clients/NetworkClient.java 
> 51a6c5dc47c4008e92b9fdd2ee359b573184e8ed 
>   clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerConfig.java 
> b9a2d4e2bc565f0ee72b27791afe5c894af262f1 
>   clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java 
> 3ac2be840c5d313a497ab67dc024d1578d4679fd 
>   
> clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerNetworkClient.java
>  9517d9d0cd480d5ba1d12f1fde7963e60528d2f8 
>   clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 
> 804d569498396d431880641041fc9292076452cb 
>   clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 
> 06f00a99a73a288df9afa8c1d4abe3580fa968a6 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/BufferPool.java
>  4cb1e50d6c4ed55241aeaef1d3af09def5274103 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java
>  a152bd7697dca55609a9ec4cfe0a82c10595fbc3 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java
>  06182db1c3a5da85648199b4c0c98b80ea7c6c0c 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 
> d2e64f7cd8bf56e433a210905b2874f71eee9ea0 
>   clients/src/main/java/org/apache/kafka/common/network/Selectable.java 
> 70e74bd6aa629c430b2850ca40c97df0b16e5d75 
>   clients/src/main/java/org/apache/kafka/common/network/Selector.java 
> 5a4909e4e5afe1b7ce2c33b9324d3947d2d91a38 
>   clients/src/test/java/org/apache/kafka/clients/MockClient.java 
> e5815f56bdf8e2d980f2bc36b831ed234c0ac781 
>   clients/src/test/java/org/apache/kafka/clients/NetworkClientTest.java 
> 

Re: Review Request 36858: Patch for KAFKA-2120

[Mayuresh Gharat]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36858/
---

(Updated Sept. 28, 2015, 11:16 p.m.)


Review request for kafka.


Bugs: KAFKA-2120
https://issues.apache.org/jira/browse/KAFKA-2120


Repository: kafka


Description
---

Solved compile error


Addressed Jason's comments for Kip-19


Addressed Jun's comments


Addressed Jason's comments about the default values for requestTimeout


checkpoint


Addressed Joel's concerns. Also tried to include Jun's feedback.


Fixed a minor comment


Solved unittest issue


Addressed Jun's comments regarding NetworkClient


Addressed Jun's comments about disconnect() in Selector


changed logging level to debug


Addressed Joels comments to break out early from the loop while aborting 
expired batches


Addressed Jun's comments


Addressed Jason's concern about iterating over timeout request in 
getNodesWithTimedOutRequest()


Addressed Joel's comments


Changed the default timeouts for requesttimeout in KafkaConfig and also made 
requestrtimeout consistent in KafkaConsumer


Fixed the forward referencing bug in KafkaConfig.


Diffs
-

  clients/src/main/java/org/apache/kafka/clients/ClientRequest.java 
dc8f0f115bcda893c95d17c0a57be8d14518d034 
  clients/src/main/java/org/apache/kafka/clients/CommonClientConfigs.java 
7d24c6f5dd2b63b96584f3aa8922a1d048dc1ae4 
  clients/src/main/java/org/apache/kafka/clients/InFlightRequests.java 
15d00d4e484bb5d51a9ae6857ed6e024a2cc1820 
  clients/src/main/java/org/apache/kafka/clients/KafkaClient.java 
f46c0d9b5eb73887c62a0e09c96e9d8c964c709d 
  clients/src/main/java/org/apache/kafka/clients/NetworkClient.java 
51a6c5dc47c4008e92b9fdd2ee359b573184e8ed 
  clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerConfig.java 
b9a2d4e2bc565f0ee72b27791afe5c894af262f1 
  clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java 
3ac2be840c5d313a497ab67dc024d1578d4679fd 
  
clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerNetworkClient.java
 9517d9d0cd480d5ba1d12f1fde7963e60528d2f8 
  clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 
804d569498396d431880641041fc9292076452cb 
  clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 
06f00a99a73a288df9afa8c1d4abe3580fa968a6 
  
clients/src/main/java/org/apache/kafka/clients/producer/internals/BufferPool.java
 4cb1e50d6c4ed55241aeaef1d3af09def5274103 
  
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java
 a152bd7697dca55609a9ec4cfe0a82c10595fbc3 
  
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java
 06182db1c3a5da85648199b4c0c98b80ea7c6c0c 
  clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 
d2e64f7cd8bf56e433a210905b2874f71eee9ea0 
  clients/src/main/java/org/apache/kafka/common/network/Selectable.java 
70e74bd6aa629c430b2850ca40c97df0b16e5d75 
  clients/src/main/java/org/apache/kafka/common/network/Selector.java 
5a4909e4e5afe1b7ce2c33b9324d3947d2d91a38 
  clients/src/test/java/org/apache/kafka/clients/MockClient.java 
e5815f56bdf8e2d980f2bc36b831ed234c0ac781 
  clients/src/test/java/org/apache/kafka/clients/NetworkClientTest.java 
69c93c3adf674b1640534c3d7410fcaafaf2232c 
  
clients/src/test/java/org/apache/kafka/clients/producer/internals/BufferPoolTest.java
 2c693824fa53db1e38766b8c66a0ef42ef9d0f3a 
  
clients/src/test/java/org/apache/kafka/clients/producer/internals/RecordAccumulatorTest.java
 5b2e4ffaeab7127648db608c179703b27b577414 
  
clients/src/test/java/org/apache/kafka/clients/producer/internals/SenderTest.java
 aa44991777a855f4b7f4f7bf17107c69393ff8ff 
  clients/src/test/java/org/apache/kafka/common/network/SSLSelectorTest.java 
df1205c935bee9a30a50816dbade64d6014b1ef2 
  clients/src/test/java/org/apache/kafka/common/network/SelectorTest.java 
3a684d98b05cadfb25c6f7f9a038ef1f6697edbf 
  clients/src/test/java/org/apache/kafka/test/MockSelector.java 
f83fd9b794a3bd191121a22bcb40fd6ec31d83b5 
  core/src/main/scala/kafka/controller/ControllerChannelManager.scala 
b1cf668dbb5bc8617d320fd3a307c6c8005b78a2 
  core/src/main/scala/kafka/server/KafkaConfig.scala 
1e8b2331486ffe55bfcc0919e48e12aad23b7d3c 
  core/src/main/scala/kafka/server/KafkaServer.scala 
f3f1fa653b9c85e729cbe32b63669011ada46547 
  core/src/main/scala/kafka/server/ReplicaFetcherThread.scala 
6c85e52e172bfc0278e724068a48a271bc15dc9f 
  core/src/main/scala/kafka/tools/ProducerPerformance.scala 
46a68e97b8bcc8821f21e4220ce9b3acedc5dafe 
  core/src/main/scala/kafka/utils/NetworkClientBlockingOps.scala 
ad10721de844725f27a116611209992cea61b088 
  core/src/test/scala/integration/kafka/api/ProducerFailureHandlingTest.scala 
1198df02ddd7727269e84a751ba99520f6d5584a 
  core/src/test/scala/unit/kafka/server/KafkaConfigTest.scala 

Re: Review Request 36858: Patch for KAFKA-2120

[Ismael Juma]

> On Sept. 19, 2015, 11:22 a.m., Ismael Juma wrote:
> > clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java, 
> > line 427
> > 
> >
> > Can this TODO be removed?
> 
> Mayuresh Gharat wrote:
> This wasn't a part of this patch and was existing before.
> 
> Mayuresh Gharat wrote:
> Sure we can remove it though
> 
> Ismael Juma wrote:
> I thought your patch fixed the issue in the TODO. Is that not true?
> 
> Mayuresh Gharat wrote:
> The motivation for KIP-19 was to actually to unblock the new producer 
> whenever it gets stuck because of unresponsive broker. NetworkClient is also 
> used by new Consumer now and hence this config becomes relevant ot the new 
> Consumer as well. If you check the new consumer has the requestimeout as long 
> but actually the requestimeout is suppopse to be int. I was going to handle 
> it in a separate patch and also have some sanity checks as per discussion 
> with Jason on the ticket. We can remove the TODO after that. Does that sound 
> ok?

Yes, sure.


- Ismael


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36858/#review99669
---


On Sept. 19, 2015, 2:27 a.m., Mayuresh Gharat wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/36858/
> ---
> 
> (Updated Sept. 19, 2015, 2:27 a.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-2120
> https://issues.apache.org/jira/browse/KAFKA-2120
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> Solved compile error
> 
> 
> Addressed Jason's comments for Kip-19
> 
> 
> Addressed Jun's comments
> 
> 
> Addressed Jason's comments about the default values for requestTimeout
> 
> 
> checkpoint
> 
> 
> Addressed Joel's concerns. Also tried to include Jun's feedback.
> 
> 
> Fixed a minor comment
> 
> 
> Solved unittest issue
> 
> 
> Addressed Jun's comments regarding NetworkClient
> 
> 
> Addressed Jun's comments about disconnect() in Selector
> 
> 
> changed logging level to debug
> 
> 
> Addressed Joels comments to break out early from the loop while aborting 
> expired batches
> 
> 
> Addressed Jun's comments
> 
> 
> Addressed Jason's concern about iterating over timeout request in 
> getNodesWithTimedOutRequest()
> 
> 
> Addressed Joel's comments
> 
> 
> Changed the default timeouts for requesttimeout in KafkaConfig and also made 
> requestrtimeout consistent in KafkaConsumer
> 
> 
> Diffs
> -
> 
>   clients/src/main/java/org/apache/kafka/clients/ClientRequest.java 
> dc8f0f115bcda893c95d17c0a57be8d14518d034 
>   clients/src/main/java/org/apache/kafka/clients/CommonClientConfigs.java 
> 7d24c6f5dd2b63b96584f3aa8922a1d048dc1ae4 
>   clients/src/main/java/org/apache/kafka/clients/InFlightRequests.java 
> 15d00d4e484bb5d51a9ae6857ed6e024a2cc1820 
>   clients/src/main/java/org/apache/kafka/clients/KafkaClient.java 
> f46c0d9b5eb73887c62a0e09c96e9d8c964c709d 
>   clients/src/main/java/org/apache/kafka/clients/NetworkClient.java 
> 51a6c5dc47c4008e92b9fdd2ee359b573184e8ed 
>   clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerConfig.java 
> b9a2d4e2bc565f0ee72b27791afe5c894af262f1 
>   clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java 
> 3ac2be840c5d313a497ab67dc024d1578d4679fd 
>   
> clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerNetworkClient.java
>  9517d9d0cd480d5ba1d12f1fde7963e60528d2f8 
>   clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 
> 804d569498396d431880641041fc9292076452cb 
>   clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 
> 06f00a99a73a288df9afa8c1d4abe3580fa968a6 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/BufferPool.java
>  4cb1e50d6c4ed55241aeaef1d3af09def5274103 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java
>  a152bd7697dca55609a9ec4cfe0a82c10595fbc3 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java
>  06182db1c3a5da85648199b4c0c98b80ea7c6c0c 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 
> d2e64f7cd8bf56e433a210905b2874f71eee9ea0 
>   clients/src/main/java/org/apache/kafka/common/network/Selectable.java 
> 70e74bd6aa629c430b2850ca40c97df0b16e5d75 
>   clients/src/main/java/org/apache/kafka/common/network/Selector.java 
> 5a4909e4e5afe1b7ce2c33b9324d3947d2d91a38 
>   clients/src/test/java/org/apache/kafka/clients/MockClient.java 
> e5815f56bdf8e2d980f2bc36b831ed234c0ac781 
>   clients/src/test/java/org/apache/kafka/clients/NetworkClientTest.java 
> 

Re: Review Request 36858: Patch for KAFKA-2120

[Mayuresh Gharat]

> On Sept. 19, 2015, 11:22 a.m., Ismael Juma wrote:
> > clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java, 
> > line 427
> > 
> >
> > Can this TODO be removed?
> 
> Mayuresh Gharat wrote:
> This wasn't a part of this patch and was existing before.
> 
> Mayuresh Gharat wrote:
> Sure we can remove it though
> 
> Ismael Juma wrote:
> I thought your patch fixed the issue in the TODO. Is that not true?

The motivation for KIP-19 was to actually to unblock the new producer whenever 
it gets stuck because of unresponsive broker. NetworkClient is also used by new 
Consumer now and hence this config becomes relevant ot the new Consumer as 
well. If you check the new consumer has the requestimeout as long but actually 
the requestimeout is suppopse to be int. I was going to handle it in a separate 
patch and also have some sanity checks as per discussion with Jason on the 
ticket. We can remove the TODO after that. Does that sound ok?


- Mayuresh


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36858/#review99669
---


On Sept. 19, 2015, 2:27 a.m., Mayuresh Gharat wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/36858/
> ---
> 
> (Updated Sept. 19, 2015, 2:27 a.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-2120
> https://issues.apache.org/jira/browse/KAFKA-2120
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> Solved compile error
> 
> 
> Addressed Jason's comments for Kip-19
> 
> 
> Addressed Jun's comments
> 
> 
> Addressed Jason's comments about the default values for requestTimeout
> 
> 
> checkpoint
> 
> 
> Addressed Joel's concerns. Also tried to include Jun's feedback.
> 
> 
> Fixed a minor comment
> 
> 
> Solved unittest issue
> 
> 
> Addressed Jun's comments regarding NetworkClient
> 
> 
> Addressed Jun's comments about disconnect() in Selector
> 
> 
> changed logging level to debug
> 
> 
> Addressed Joels comments to break out early from the loop while aborting 
> expired batches
> 
> 
> Addressed Jun's comments
> 
> 
> Addressed Jason's concern about iterating over timeout request in 
> getNodesWithTimedOutRequest()
> 
> 
> Addressed Joel's comments
> 
> 
> Changed the default timeouts for requesttimeout in KafkaConfig and also made 
> requestrtimeout consistent in KafkaConsumer
> 
> 
> Diffs
> -
> 
>   clients/src/main/java/org/apache/kafka/clients/ClientRequest.java 
> dc8f0f115bcda893c95d17c0a57be8d14518d034 
>   clients/src/main/java/org/apache/kafka/clients/CommonClientConfigs.java 
> 7d24c6f5dd2b63b96584f3aa8922a1d048dc1ae4 
>   clients/src/main/java/org/apache/kafka/clients/InFlightRequests.java 
> 15d00d4e484bb5d51a9ae6857ed6e024a2cc1820 
>   clients/src/main/java/org/apache/kafka/clients/KafkaClient.java 
> f46c0d9b5eb73887c62a0e09c96e9d8c964c709d 
>   clients/src/main/java/org/apache/kafka/clients/NetworkClient.java 
> 51a6c5dc47c4008e92b9fdd2ee359b573184e8ed 
>   clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerConfig.java 
> b9a2d4e2bc565f0ee72b27791afe5c894af262f1 
>   clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java 
> 3ac2be840c5d313a497ab67dc024d1578d4679fd 
>   
> clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerNetworkClient.java
>  9517d9d0cd480d5ba1d12f1fde7963e60528d2f8 
>   clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 
> 804d569498396d431880641041fc9292076452cb 
>   clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 
> 06f00a99a73a288df9afa8c1d4abe3580fa968a6 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/BufferPool.java
>  4cb1e50d6c4ed55241aeaef1d3af09def5274103 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java
>  a152bd7697dca55609a9ec4cfe0a82c10595fbc3 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java
>  06182db1c3a5da85648199b4c0c98b80ea7c6c0c 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 
> d2e64f7cd8bf56e433a210905b2874f71eee9ea0 
>   clients/src/main/java/org/apache/kafka/common/network/Selectable.java 
> 70e74bd6aa629c430b2850ca40c97df0b16e5d75 
>   clients/src/main/java/org/apache/kafka/common/network/Selector.java 
> 5a4909e4e5afe1b7ce2c33b9324d3947d2d91a38 
>   clients/src/test/java/org/apache/kafka/clients/MockClient.java 
> e5815f56bdf8e2d980f2bc36b831ed234c0ac781 
>   clients/src/test/java/org/apache/kafka/clients/NetworkClientTest.java 
> 69c93c3adf674b1640534c3d7410fcaafaf2232c 
>   
> 

Re: Review Request 36858: Patch for KAFKA-2120

[Ismael Juma]

> On Sept. 19, 2015, 11:22 a.m., Ismael Juma wrote:
> > clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java, 
> > line 427
> > 
> >
> > Can this TODO be removed?
> 
> Mayuresh Gharat wrote:
> This wasn't a part of this patch and was existing before.
> 
> Mayuresh Gharat wrote:
> Sure we can remove it though

I thought your patch fixed the issue in the TODO. Is that not true?


- Ismael


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36858/#review99669
---


On Sept. 19, 2015, 2:27 a.m., Mayuresh Gharat wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/36858/
> ---
> 
> (Updated Sept. 19, 2015, 2:27 a.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-2120
> https://issues.apache.org/jira/browse/KAFKA-2120
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> Solved compile error
> 
> 
> Addressed Jason's comments for Kip-19
> 
> 
> Addressed Jun's comments
> 
> 
> Addressed Jason's comments about the default values for requestTimeout
> 
> 
> checkpoint
> 
> 
> Addressed Joel's concerns. Also tried to include Jun's feedback.
> 
> 
> Fixed a minor comment
> 
> 
> Solved unittest issue
> 
> 
> Addressed Jun's comments regarding NetworkClient
> 
> 
> Addressed Jun's comments about disconnect() in Selector
> 
> 
> changed logging level to debug
> 
> 
> Addressed Joels comments to break out early from the loop while aborting 
> expired batches
> 
> 
> Addressed Jun's comments
> 
> 
> Addressed Jason's concern about iterating over timeout request in 
> getNodesWithTimedOutRequest()
> 
> 
> Addressed Joel's comments
> 
> 
> Changed the default timeouts for requesttimeout in KafkaConfig and also made 
> requestrtimeout consistent in KafkaConsumer
> 
> 
> Diffs
> -
> 
>   clients/src/main/java/org/apache/kafka/clients/ClientRequest.java 
> dc8f0f115bcda893c95d17c0a57be8d14518d034 
>   clients/src/main/java/org/apache/kafka/clients/CommonClientConfigs.java 
> 7d24c6f5dd2b63b96584f3aa8922a1d048dc1ae4 
>   clients/src/main/java/org/apache/kafka/clients/InFlightRequests.java 
> 15d00d4e484bb5d51a9ae6857ed6e024a2cc1820 
>   clients/src/main/java/org/apache/kafka/clients/KafkaClient.java 
> f46c0d9b5eb73887c62a0e09c96e9d8c964c709d 
>   clients/src/main/java/org/apache/kafka/clients/NetworkClient.java 
> 51a6c5dc47c4008e92b9fdd2ee359b573184e8ed 
>   clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerConfig.java 
> b9a2d4e2bc565f0ee72b27791afe5c894af262f1 
>   clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java 
> 3ac2be840c5d313a497ab67dc024d1578d4679fd 
>   
> clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerNetworkClient.java
>  9517d9d0cd480d5ba1d12f1fde7963e60528d2f8 
>   clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 
> 804d569498396d431880641041fc9292076452cb 
>   clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 
> 06f00a99a73a288df9afa8c1d4abe3580fa968a6 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/BufferPool.java
>  4cb1e50d6c4ed55241aeaef1d3af09def5274103 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java
>  a152bd7697dca55609a9ec4cfe0a82c10595fbc3 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java
>  06182db1c3a5da85648199b4c0c98b80ea7c6c0c 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 
> d2e64f7cd8bf56e433a210905b2874f71eee9ea0 
>   clients/src/main/java/org/apache/kafka/common/network/Selectable.java 
> 70e74bd6aa629c430b2850ca40c97df0b16e5d75 
>   clients/src/main/java/org/apache/kafka/common/network/Selector.java 
> 5a4909e4e5afe1b7ce2c33b9324d3947d2d91a38 
>   clients/src/test/java/org/apache/kafka/clients/MockClient.java 
> e5815f56bdf8e2d980f2bc36b831ed234c0ac781 
>   clients/src/test/java/org/apache/kafka/clients/NetworkClientTest.java 
> 69c93c3adf674b1640534c3d7410fcaafaf2232c 
>   
> clients/src/test/java/org/apache/kafka/clients/producer/internals/BufferPoolTest.java
>  2c693824fa53db1e38766b8c66a0ef42ef9d0f3a 
>   
> clients/src/test/java/org/apache/kafka/clients/producer/internals/RecordAccumulatorTest.java
>  5b2e4ffaeab7127648db608c179703b27b577414 
>   
> clients/src/test/java/org/apache/kafka/clients/producer/internals/SenderTest.java
>  aa44991777a855f4b7f4f7bf17107c69393ff8ff 
>   clients/src/test/java/org/apache/kafka/common/network/SSLSelectorTest.java 
> df1205c935bee9a30a50816dbade64d6014b1ef2 
>   

Re: Review Request 36858: Patch for KAFKA-2120

[Ismael Juma]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36858/#review99669
---



clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java 
(line 427)


Can this TODO be removed?


- Ismael Juma


On Sept. 19, 2015, 2:27 a.m., Mayuresh Gharat wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/36858/
> ---
> 
> (Updated Sept. 19, 2015, 2:27 a.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-2120
> https://issues.apache.org/jira/browse/KAFKA-2120
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> Solved compile error
> 
> 
> Addressed Jason's comments for Kip-19
> 
> 
> Addressed Jun's comments
> 
> 
> Addressed Jason's comments about the default values for requestTimeout
> 
> 
> checkpoint
> 
> 
> Addressed Joel's concerns. Also tried to include Jun's feedback.
> 
> 
> Fixed a minor comment
> 
> 
> Solved unittest issue
> 
> 
> Addressed Jun's comments regarding NetworkClient
> 
> 
> Addressed Jun's comments about disconnect() in Selector
> 
> 
> changed logging level to debug
> 
> 
> Addressed Joels comments to break out early from the loop while aborting 
> expired batches
> 
> 
> Addressed Jun's comments
> 
> 
> Addressed Jason's concern about iterating over timeout request in 
> getNodesWithTimedOutRequest()
> 
> 
> Addressed Joel's comments
> 
> 
> Changed the default timeouts for requesttimeout in KafkaConfig and also made 
> requestrtimeout consistent in KafkaConsumer
> 
> 
> Diffs
> -
> 
>   clients/src/main/java/org/apache/kafka/clients/ClientRequest.java 
> dc8f0f115bcda893c95d17c0a57be8d14518d034 
>   clients/src/main/java/org/apache/kafka/clients/CommonClientConfigs.java 
> 7d24c6f5dd2b63b96584f3aa8922a1d048dc1ae4 
>   clients/src/main/java/org/apache/kafka/clients/InFlightRequests.java 
> 15d00d4e484bb5d51a9ae6857ed6e024a2cc1820 
>   clients/src/main/java/org/apache/kafka/clients/KafkaClient.java 
> f46c0d9b5eb73887c62a0e09c96e9d8c964c709d 
>   clients/src/main/java/org/apache/kafka/clients/NetworkClient.java 
> 51a6c5dc47c4008e92b9fdd2ee359b573184e8ed 
>   clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerConfig.java 
> b9a2d4e2bc565f0ee72b27791afe5c894af262f1 
>   clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java 
> 3ac2be840c5d313a497ab67dc024d1578d4679fd 
>   
> clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerNetworkClient.java
>  9517d9d0cd480d5ba1d12f1fde7963e60528d2f8 
>   clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 
> 804d569498396d431880641041fc9292076452cb 
>   clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 
> 06f00a99a73a288df9afa8c1d4abe3580fa968a6 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/BufferPool.java
>  4cb1e50d6c4ed55241aeaef1d3af09def5274103 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java
>  a152bd7697dca55609a9ec4cfe0a82c10595fbc3 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java
>  06182db1c3a5da85648199b4c0c98b80ea7c6c0c 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 
> d2e64f7cd8bf56e433a210905b2874f71eee9ea0 
>   clients/src/main/java/org/apache/kafka/common/network/Selectable.java 
> 70e74bd6aa629c430b2850ca40c97df0b16e5d75 
>   clients/src/main/java/org/apache/kafka/common/network/Selector.java 
> 5a4909e4e5afe1b7ce2c33b9324d3947d2d91a38 
>   clients/src/test/java/org/apache/kafka/clients/MockClient.java 
> e5815f56bdf8e2d980f2bc36b831ed234c0ac781 
>   clients/src/test/java/org/apache/kafka/clients/NetworkClientTest.java 
> 69c93c3adf674b1640534c3d7410fcaafaf2232c 
>   
> clients/src/test/java/org/apache/kafka/clients/producer/internals/BufferPoolTest.java
>  2c693824fa53db1e38766b8c66a0ef42ef9d0f3a 
>   
> clients/src/test/java/org/apache/kafka/clients/producer/internals/RecordAccumulatorTest.java
>  5b2e4ffaeab7127648db608c179703b27b577414 
>   
> clients/src/test/java/org/apache/kafka/clients/producer/internals/SenderTest.java
>  aa44991777a855f4b7f4f7bf17107c69393ff8ff 
>   clients/src/test/java/org/apache/kafka/common/network/SSLSelectorTest.java 
> df1205c935bee9a30a50816dbade64d6014b1ef2 
>   clients/src/test/java/org/apache/kafka/common/network/SelectorTest.java 
> 3a684d98b05cadfb25c6f7f9a038ef1f6697edbf 
>   clients/src/test/java/org/apache/kafka/test/MockSelector.java 
> f83fd9b794a3bd191121a22bcb40fd6ec31d83b5 
>   core/src/main/scala/kafka/controller/ControllerChannelManager.scala 
> b1cf668dbb5bc8617d320fd3a307c6c8005b78a2 
>   

Re: Review Request 36858: Patch for KAFKA-2120

[Ismael Juma]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36858/#review99661
---



core/src/main/scala/kafka/controller/ControllerChannelManager.scala (line 109)


As I said in JIRA, the brokers already check for timeouts since they do the 
calls in a blocking way (see NetworkClientBlockingOps). It seems a bit weird to 
do it in two different ways. I think we should use one or the other. This also 
applies to KafkaServer and ReplicaFetcherThread.


- Ismael Juma


On Sept. 19, 2015, 2:27 a.m., Mayuresh Gharat wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/36858/
> ---
> 
> (Updated Sept. 19, 2015, 2:27 a.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-2120
> https://issues.apache.org/jira/browse/KAFKA-2120
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> Solved compile error
> 
> 
> Addressed Jason's comments for Kip-19
> 
> 
> Addressed Jun's comments
> 
> 
> Addressed Jason's comments about the default values for requestTimeout
> 
> 
> checkpoint
> 
> 
> Addressed Joel's concerns. Also tried to include Jun's feedback.
> 
> 
> Fixed a minor comment
> 
> 
> Solved unittest issue
> 
> 
> Addressed Jun's comments regarding NetworkClient
> 
> 
> Addressed Jun's comments about disconnect() in Selector
> 
> 
> changed logging level to debug
> 
> 
> Addressed Joels comments to break out early from the loop while aborting 
> expired batches
> 
> 
> Addressed Jun's comments
> 
> 
> Addressed Jason's concern about iterating over timeout request in 
> getNodesWithTimedOutRequest()
> 
> 
> Addressed Joel's comments
> 
> 
> Changed the default timeouts for requesttimeout in KafkaConfig and also made 
> requestrtimeout consistent in KafkaConsumer
> 
> 
> Diffs
> -
> 
>   clients/src/main/java/org/apache/kafka/clients/ClientRequest.java 
> dc8f0f115bcda893c95d17c0a57be8d14518d034 
>   clients/src/main/java/org/apache/kafka/clients/CommonClientConfigs.java 
> 7d24c6f5dd2b63b96584f3aa8922a1d048dc1ae4 
>   clients/src/main/java/org/apache/kafka/clients/InFlightRequests.java 
> 15d00d4e484bb5d51a9ae6857ed6e024a2cc1820 
>   clients/src/main/java/org/apache/kafka/clients/KafkaClient.java 
> f46c0d9b5eb73887c62a0e09c96e9d8c964c709d 
>   clients/src/main/java/org/apache/kafka/clients/NetworkClient.java 
> 51a6c5dc47c4008e92b9fdd2ee359b573184e8ed 
>   clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerConfig.java 
> b9a2d4e2bc565f0ee72b27791afe5c894af262f1 
>   clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java 
> 3ac2be840c5d313a497ab67dc024d1578d4679fd 
>   
> clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerNetworkClient.java
>  9517d9d0cd480d5ba1d12f1fde7963e60528d2f8 
>   clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 
> 804d569498396d431880641041fc9292076452cb 
>   clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 
> 06f00a99a73a288df9afa8c1d4abe3580fa968a6 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/BufferPool.java
>  4cb1e50d6c4ed55241aeaef1d3af09def5274103 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java
>  a152bd7697dca55609a9ec4cfe0a82c10595fbc3 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java
>  06182db1c3a5da85648199b4c0c98b80ea7c6c0c 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 
> d2e64f7cd8bf56e433a210905b2874f71eee9ea0 
>   clients/src/main/java/org/apache/kafka/common/network/Selectable.java 
> 70e74bd6aa629c430b2850ca40c97df0b16e5d75 
>   clients/src/main/java/org/apache/kafka/common/network/Selector.java 
> 5a4909e4e5afe1b7ce2c33b9324d3947d2d91a38 
>   clients/src/test/java/org/apache/kafka/clients/MockClient.java 
> e5815f56bdf8e2d980f2bc36b831ed234c0ac781 
>   clients/src/test/java/org/apache/kafka/clients/NetworkClientTest.java 
> 69c93c3adf674b1640534c3d7410fcaafaf2232c 
>   
> clients/src/test/java/org/apache/kafka/clients/producer/internals/BufferPoolTest.java
>  2c693824fa53db1e38766b8c66a0ef42ef9d0f3a 
>   
> clients/src/test/java/org/apache/kafka/clients/producer/internals/RecordAccumulatorTest.java
>  5b2e4ffaeab7127648db608c179703b27b577414 
>   
> clients/src/test/java/org/apache/kafka/clients/producer/internals/SenderTest.java
>  aa44991777a855f4b7f4f7bf17107c69393ff8ff 
>   clients/src/test/java/org/apache/kafka/common/network/SSLSelectorTest.java 
> df1205c935bee9a30a50816dbade64d6014b1ef2 
>   clients/src/test/java/org/apache/kafka/common/network/SelectorTest.java 
> 3a684d98b05cadfb25c6f7f9a038ef1f6697edbf 
>   

Re: Review Request 36858: Patch for KAFKA-2120

[Ismael Juma]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36858/#review99659
---



clients/src/main/java/org/apache/kafka/clients/KafkaClient.java (line 73)


One question: did we consider setting sendMs directly in ClientRequest? I 
don't see the benefit of passing it as a parameter here and then setting it in 
the request anyway.


- Ismael Juma


On Sept. 19, 2015, 2:27 a.m., Mayuresh Gharat wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/36858/
> ---
> 
> (Updated Sept. 19, 2015, 2:27 a.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-2120
> https://issues.apache.org/jira/browse/KAFKA-2120
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> Solved compile error
> 
> 
> Addressed Jason's comments for Kip-19
> 
> 
> Addressed Jun's comments
> 
> 
> Addressed Jason's comments about the default values for requestTimeout
> 
> 
> checkpoint
> 
> 
> Addressed Joel's concerns. Also tried to include Jun's feedback.
> 
> 
> Fixed a minor comment
> 
> 
> Solved unittest issue
> 
> 
> Addressed Jun's comments regarding NetworkClient
> 
> 
> Addressed Jun's comments about disconnect() in Selector
> 
> 
> changed logging level to debug
> 
> 
> Addressed Joels comments to break out early from the loop while aborting 
> expired batches
> 
> 
> Addressed Jun's comments
> 
> 
> Addressed Jason's concern about iterating over timeout request in 
> getNodesWithTimedOutRequest()
> 
> 
> Addressed Joel's comments
> 
> 
> Changed the default timeouts for requesttimeout in KafkaConfig and also made 
> requestrtimeout consistent in KafkaConsumer
> 
> 
> Diffs
> -
> 
>   clients/src/main/java/org/apache/kafka/clients/ClientRequest.java 
> dc8f0f115bcda893c95d17c0a57be8d14518d034 
>   clients/src/main/java/org/apache/kafka/clients/CommonClientConfigs.java 
> 7d24c6f5dd2b63b96584f3aa8922a1d048dc1ae4 
>   clients/src/main/java/org/apache/kafka/clients/InFlightRequests.java 
> 15d00d4e484bb5d51a9ae6857ed6e024a2cc1820 
>   clients/src/main/java/org/apache/kafka/clients/KafkaClient.java 
> f46c0d9b5eb73887c62a0e09c96e9d8c964c709d 
>   clients/src/main/java/org/apache/kafka/clients/NetworkClient.java 
> 51a6c5dc47c4008e92b9fdd2ee359b573184e8ed 
>   clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerConfig.java 
> b9a2d4e2bc565f0ee72b27791afe5c894af262f1 
>   clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java 
> 3ac2be840c5d313a497ab67dc024d1578d4679fd 
>   
> clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerNetworkClient.java
>  9517d9d0cd480d5ba1d12f1fde7963e60528d2f8 
>   clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 
> 804d569498396d431880641041fc9292076452cb 
>   clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 
> 06f00a99a73a288df9afa8c1d4abe3580fa968a6 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/BufferPool.java
>  4cb1e50d6c4ed55241aeaef1d3af09def5274103 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java
>  a152bd7697dca55609a9ec4cfe0a82c10595fbc3 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java
>  06182db1c3a5da85648199b4c0c98b80ea7c6c0c 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 
> d2e64f7cd8bf56e433a210905b2874f71eee9ea0 
>   clients/src/main/java/org/apache/kafka/common/network/Selectable.java 
> 70e74bd6aa629c430b2850ca40c97df0b16e5d75 
>   clients/src/main/java/org/apache/kafka/common/network/Selector.java 
> 5a4909e4e5afe1b7ce2c33b9324d3947d2d91a38 
>   clients/src/test/java/org/apache/kafka/clients/MockClient.java 
> e5815f56bdf8e2d980f2bc36b831ed234c0ac781 
>   clients/src/test/java/org/apache/kafka/clients/NetworkClientTest.java 
> 69c93c3adf674b1640534c3d7410fcaafaf2232c 
>   
> clients/src/test/java/org/apache/kafka/clients/producer/internals/BufferPoolTest.java
>  2c693824fa53db1e38766b8c66a0ef42ef9d0f3a 
>   
> clients/src/test/java/org/apache/kafka/clients/producer/internals/RecordAccumulatorTest.java
>  5b2e4ffaeab7127648db608c179703b27b577414 
>   
> clients/src/test/java/org/apache/kafka/clients/producer/internals/SenderTest.java
>  aa44991777a855f4b7f4f7bf17107c69393ff8ff 
>   clients/src/test/java/org/apache/kafka/common/network/SSLSelectorTest.java 
> df1205c935bee9a30a50816dbade64d6014b1ef2 
>   clients/src/test/java/org/apache/kafka/common/network/SelectorTest.java 
> 3a684d98b05cadfb25c6f7f9a038ef1f6697edbf 
>   clients/src/test/java/org/apache/kafka/test/MockSelector.java 
> f83fd9b794a3bd191121a22bcb40fd6ec31d83b5 
>   

Re: Review Request 36858: Patch for KAFKA-2120

[Mayuresh Gharat]

> On Sept. 19, 2015, 11:22 a.m., Ismael Juma wrote:
> > clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java, 
> > line 427
> > 
> >
> > Can this TODO be removed?

This wasn't a part of this patch and was existing before.


- Mayuresh


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36858/#review99669
---


On Sept. 19, 2015, 2:27 a.m., Mayuresh Gharat wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/36858/
> ---
> 
> (Updated Sept. 19, 2015, 2:27 a.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-2120
> https://issues.apache.org/jira/browse/KAFKA-2120
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> Solved compile error
> 
> 
> Addressed Jason's comments for Kip-19
> 
> 
> Addressed Jun's comments
> 
> 
> Addressed Jason's comments about the default values for requestTimeout
> 
> 
> checkpoint
> 
> 
> Addressed Joel's concerns. Also tried to include Jun's feedback.
> 
> 
> Fixed a minor comment
> 
> 
> Solved unittest issue
> 
> 
> Addressed Jun's comments regarding NetworkClient
> 
> 
> Addressed Jun's comments about disconnect() in Selector
> 
> 
> changed logging level to debug
> 
> 
> Addressed Joels comments to break out early from the loop while aborting 
> expired batches
> 
> 
> Addressed Jun's comments
> 
> 
> Addressed Jason's concern about iterating over timeout request in 
> getNodesWithTimedOutRequest()
> 
> 
> Addressed Joel's comments
> 
> 
> Changed the default timeouts for requesttimeout in KafkaConfig and also made 
> requestrtimeout consistent in KafkaConsumer
> 
> 
> Diffs
> -
> 
>   clients/src/main/java/org/apache/kafka/clients/ClientRequest.java 
> dc8f0f115bcda893c95d17c0a57be8d14518d034 
>   clients/src/main/java/org/apache/kafka/clients/CommonClientConfigs.java 
> 7d24c6f5dd2b63b96584f3aa8922a1d048dc1ae4 
>   clients/src/main/java/org/apache/kafka/clients/InFlightRequests.java 
> 15d00d4e484bb5d51a9ae6857ed6e024a2cc1820 
>   clients/src/main/java/org/apache/kafka/clients/KafkaClient.java 
> f46c0d9b5eb73887c62a0e09c96e9d8c964c709d 
>   clients/src/main/java/org/apache/kafka/clients/NetworkClient.java 
> 51a6c5dc47c4008e92b9fdd2ee359b573184e8ed 
>   clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerConfig.java 
> b9a2d4e2bc565f0ee72b27791afe5c894af262f1 
>   clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java 
> 3ac2be840c5d313a497ab67dc024d1578d4679fd 
>   
> clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerNetworkClient.java
>  9517d9d0cd480d5ba1d12f1fde7963e60528d2f8 
>   clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 
> 804d569498396d431880641041fc9292076452cb 
>   clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 
> 06f00a99a73a288df9afa8c1d4abe3580fa968a6 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/BufferPool.java
>  4cb1e50d6c4ed55241aeaef1d3af09def5274103 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java
>  a152bd7697dca55609a9ec4cfe0a82c10595fbc3 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java
>  06182db1c3a5da85648199b4c0c98b80ea7c6c0c 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 
> d2e64f7cd8bf56e433a210905b2874f71eee9ea0 
>   clients/src/main/java/org/apache/kafka/common/network/Selectable.java 
> 70e74bd6aa629c430b2850ca40c97df0b16e5d75 
>   clients/src/main/java/org/apache/kafka/common/network/Selector.java 
> 5a4909e4e5afe1b7ce2c33b9324d3947d2d91a38 
>   clients/src/test/java/org/apache/kafka/clients/MockClient.java 
> e5815f56bdf8e2d980f2bc36b831ed234c0ac781 
>   clients/src/test/java/org/apache/kafka/clients/NetworkClientTest.java 
> 69c93c3adf674b1640534c3d7410fcaafaf2232c 
>   
> clients/src/test/java/org/apache/kafka/clients/producer/internals/BufferPoolTest.java
>  2c693824fa53db1e38766b8c66a0ef42ef9d0f3a 
>   
> clients/src/test/java/org/apache/kafka/clients/producer/internals/RecordAccumulatorTest.java
>  5b2e4ffaeab7127648db608c179703b27b577414 
>   
> clients/src/test/java/org/apache/kafka/clients/producer/internals/SenderTest.java
>  aa44991777a855f4b7f4f7bf17107c69393ff8ff 
>   clients/src/test/java/org/apache/kafka/common/network/SSLSelectorTest.java 
> df1205c935bee9a30a50816dbade64d6014b1ef2 
>   clients/src/test/java/org/apache/kafka/common/network/SelectorTest.java 
> 3a684d98b05cadfb25c6f7f9a038ef1f6697edbf 
>   clients/src/test/java/org/apache/kafka/test/MockSelector.java 
> f83fd9b794a3bd191121a22bcb40fd6ec31d83b5 
>   

Re: Review Request 36858: Patch for KAFKA-2120

[Mayuresh Gharat]

> On Sept. 19, 2015, 11:22 a.m., Ismael Juma wrote:
> > clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java, 
> > line 427
> > 
> >
> > Can this TODO be removed?
> 
> Mayuresh Gharat wrote:
> This wasn't a part of this patch and was existing before.

Sure we can remove it though


- Mayuresh


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36858/#review99669
---


On Sept. 19, 2015, 2:27 a.m., Mayuresh Gharat wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/36858/
> ---
> 
> (Updated Sept. 19, 2015, 2:27 a.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-2120
> https://issues.apache.org/jira/browse/KAFKA-2120
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> Solved compile error
> 
> 
> Addressed Jason's comments for Kip-19
> 
> 
> Addressed Jun's comments
> 
> 
> Addressed Jason's comments about the default values for requestTimeout
> 
> 
> checkpoint
> 
> 
> Addressed Joel's concerns. Also tried to include Jun's feedback.
> 
> 
> Fixed a minor comment
> 
> 
> Solved unittest issue
> 
> 
> Addressed Jun's comments regarding NetworkClient
> 
> 
> Addressed Jun's comments about disconnect() in Selector
> 
> 
> changed logging level to debug
> 
> 
> Addressed Joels comments to break out early from the loop while aborting 
> expired batches
> 
> 
> Addressed Jun's comments
> 
> 
> Addressed Jason's concern about iterating over timeout request in 
> getNodesWithTimedOutRequest()
> 
> 
> Addressed Joel's comments
> 
> 
> Changed the default timeouts for requesttimeout in KafkaConfig and also made 
> requestrtimeout consistent in KafkaConsumer
> 
> 
> Diffs
> -
> 
>   clients/src/main/java/org/apache/kafka/clients/ClientRequest.java 
> dc8f0f115bcda893c95d17c0a57be8d14518d034 
>   clients/src/main/java/org/apache/kafka/clients/CommonClientConfigs.java 
> 7d24c6f5dd2b63b96584f3aa8922a1d048dc1ae4 
>   clients/src/main/java/org/apache/kafka/clients/InFlightRequests.java 
> 15d00d4e484bb5d51a9ae6857ed6e024a2cc1820 
>   clients/src/main/java/org/apache/kafka/clients/KafkaClient.java 
> f46c0d9b5eb73887c62a0e09c96e9d8c964c709d 
>   clients/src/main/java/org/apache/kafka/clients/NetworkClient.java 
> 51a6c5dc47c4008e92b9fdd2ee359b573184e8ed 
>   clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerConfig.java 
> b9a2d4e2bc565f0ee72b27791afe5c894af262f1 
>   clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java 
> 3ac2be840c5d313a497ab67dc024d1578d4679fd 
>   
> clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerNetworkClient.java
>  9517d9d0cd480d5ba1d12f1fde7963e60528d2f8 
>   clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 
> 804d569498396d431880641041fc9292076452cb 
>   clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 
> 06f00a99a73a288df9afa8c1d4abe3580fa968a6 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/BufferPool.java
>  4cb1e50d6c4ed55241aeaef1d3af09def5274103 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java
>  a152bd7697dca55609a9ec4cfe0a82c10595fbc3 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java
>  06182db1c3a5da85648199b4c0c98b80ea7c6c0c 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 
> d2e64f7cd8bf56e433a210905b2874f71eee9ea0 
>   clients/src/main/java/org/apache/kafka/common/network/Selectable.java 
> 70e74bd6aa629c430b2850ca40c97df0b16e5d75 
>   clients/src/main/java/org/apache/kafka/common/network/Selector.java 
> 5a4909e4e5afe1b7ce2c33b9324d3947d2d91a38 
>   clients/src/test/java/org/apache/kafka/clients/MockClient.java 
> e5815f56bdf8e2d980f2bc36b831ed234c0ac781 
>   clients/src/test/java/org/apache/kafka/clients/NetworkClientTest.java 
> 69c93c3adf674b1640534c3d7410fcaafaf2232c 
>   
> clients/src/test/java/org/apache/kafka/clients/producer/internals/BufferPoolTest.java
>  2c693824fa53db1e38766b8c66a0ef42ef9d0f3a 
>   
> clients/src/test/java/org/apache/kafka/clients/producer/internals/RecordAccumulatorTest.java
>  5b2e4ffaeab7127648db608c179703b27b577414 
>   
> clients/src/test/java/org/apache/kafka/clients/producer/internals/SenderTest.java
>  aa44991777a855f4b7f4f7bf17107c69393ff8ff 
>   clients/src/test/java/org/apache/kafka/common/network/SSLSelectorTest.java 
> df1205c935bee9a30a50816dbade64d6014b1ef2 
>   clients/src/test/java/org/apache/kafka/common/network/SelectorTest.java 
> 3a684d98b05cadfb25c6f7f9a038ef1f6697edbf 
>   

Re: Review Request 36858: Patch for KAFKA-2120

[Mayuresh Gharat]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36858/
---

(Updated Sept. 19, 2015, 2:27 a.m.)


Review request for kafka.


Bugs: KAFKA-2120
https://issues.apache.org/jira/browse/KAFKA-2120


Repository: kafka


Description (updated)
---

Solved compile error


Addressed Jason's comments for Kip-19


Addressed Jun's comments


Addressed Jason's comments about the default values for requestTimeout


checkpoint


Addressed Joel's concerns. Also tried to include Jun's feedback.


Fixed a minor comment


Solved unittest issue


Addressed Jun's comments regarding NetworkClient


Addressed Jun's comments about disconnect() in Selector


changed logging level to debug


Addressed Joels comments to break out early from the loop while aborting 
expired batches


Addressed Jun's comments


Addressed Jason's concern about iterating over timeout request in 
getNodesWithTimedOutRequest()


Addressed Joel's comments


Changed the default timeouts for requesttimeout in KafkaConfig and also made 
requestrtimeout consistent in KafkaConsumer


Diffs (updated)
-

  clients/src/main/java/org/apache/kafka/clients/ClientRequest.java 
dc8f0f115bcda893c95d17c0a57be8d14518d034 
  clients/src/main/java/org/apache/kafka/clients/CommonClientConfigs.java 
7d24c6f5dd2b63b96584f3aa8922a1d048dc1ae4 
  clients/src/main/java/org/apache/kafka/clients/InFlightRequests.java 
15d00d4e484bb5d51a9ae6857ed6e024a2cc1820 
  clients/src/main/java/org/apache/kafka/clients/KafkaClient.java 
f46c0d9b5eb73887c62a0e09c96e9d8c964c709d 
  clients/src/main/java/org/apache/kafka/clients/NetworkClient.java 
51a6c5dc47c4008e92b9fdd2ee359b573184e8ed 
  clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerConfig.java 
b9a2d4e2bc565f0ee72b27791afe5c894af262f1 
  clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java 
3ac2be840c5d313a497ab67dc024d1578d4679fd 
  
clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerNetworkClient.java
 9517d9d0cd480d5ba1d12f1fde7963e60528d2f8 
  clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 
804d569498396d431880641041fc9292076452cb 
  clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 
06f00a99a73a288df9afa8c1d4abe3580fa968a6 
  
clients/src/main/java/org/apache/kafka/clients/producer/internals/BufferPool.java
 4cb1e50d6c4ed55241aeaef1d3af09def5274103 
  
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java
 a152bd7697dca55609a9ec4cfe0a82c10595fbc3 
  
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java
 06182db1c3a5da85648199b4c0c98b80ea7c6c0c 
  clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 
d2e64f7cd8bf56e433a210905b2874f71eee9ea0 
  clients/src/main/java/org/apache/kafka/common/network/Selectable.java 
70e74bd6aa629c430b2850ca40c97df0b16e5d75 
  clients/src/main/java/org/apache/kafka/common/network/Selector.java 
5a4909e4e5afe1b7ce2c33b9324d3947d2d91a38 
  clients/src/test/java/org/apache/kafka/clients/MockClient.java 
e5815f56bdf8e2d980f2bc36b831ed234c0ac781 
  clients/src/test/java/org/apache/kafka/clients/NetworkClientTest.java 
69c93c3adf674b1640534c3d7410fcaafaf2232c 
  
clients/src/test/java/org/apache/kafka/clients/producer/internals/BufferPoolTest.java
 2c693824fa53db1e38766b8c66a0ef42ef9d0f3a 
  
clients/src/test/java/org/apache/kafka/clients/producer/internals/RecordAccumulatorTest.java
 5b2e4ffaeab7127648db608c179703b27b577414 
  
clients/src/test/java/org/apache/kafka/clients/producer/internals/SenderTest.java
 aa44991777a855f4b7f4f7bf17107c69393ff8ff 
  clients/src/test/java/org/apache/kafka/common/network/SSLSelectorTest.java 
df1205c935bee9a30a50816dbade64d6014b1ef2 
  clients/src/test/java/org/apache/kafka/common/network/SelectorTest.java 
3a684d98b05cadfb25c6f7f9a038ef1f6697edbf 
  clients/src/test/java/org/apache/kafka/test/MockSelector.java 
f83fd9b794a3bd191121a22bcb40fd6ec31d83b5 
  core/src/main/scala/kafka/controller/ControllerChannelManager.scala 
b1cf668dbb5bc8617d320fd3a307c6c8005b78a2 
  core/src/main/scala/kafka/server/KafkaConfig.scala 
1e8b2331486ffe55bfcc0919e48e12aad23b7d3c 
  core/src/main/scala/kafka/server/KafkaServer.scala 
f3f1fa653b9c85e729cbe32b63669011ada46547 
  core/src/main/scala/kafka/server/ReplicaFetcherThread.scala 
6c85e52e172bfc0278e724068a48a271bc15dc9f 
  core/src/main/scala/kafka/tools/ProducerPerformance.scala 
46a68e97b8bcc8821f21e4220ce9b3acedc5dafe 
  core/src/main/scala/kafka/utils/NetworkClientBlockingOps.scala 
ad10721de844725f27a116611209992cea61b088 
  core/src/test/scala/integration/kafka/api/ProducerFailureHandlingTest.scala 
1198df02ddd7727269e84a751ba99520f6d5584a 
  core/src/test/scala/unit/kafka/server/KafkaConfigTest.scala 
5b4f2db4607ae6d17696c1140f1a771ce75c80e0 
  

Re: Review Request 36858: Patch for KAFKA-2120

[Joel Koshy]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36858/#review99235
---

Ship it!


Thanks for the updated patch - these tests seem to fail consistently. (I 
haven't checked trunk though)
```
KafkaConsumerTest. testConstructorClose
KafkaProducerTest. testConstructorFailureCloseResource
SelectorTest. testNoRouteToHost
```

- Joel Koshy


On Sept. 16, 2015, 1:57 a.m., Mayuresh Gharat wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/36858/
> ---
> 
> (Updated Sept. 16, 2015, 1:57 a.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-2120
> https://issues.apache.org/jira/browse/KAFKA-2120
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> Solved compile error
> 
> 
> Addressed Jason's comments for Kip-19
> 
> 
> Addressed Jun's comments
> 
> 
> Addressed Jason's comments about the default values for requestTimeout
> 
> 
> checkpoint
> 
> 
> Addressed Joel's concerns. Also tried to include Jun's feedback.
> 
> 
> Fixed a minor comment
> 
> 
> Solved unittest issue
> 
> 
> Addressed Jun's comments regarding NetworkClient
> 
> 
> Addressed Jun's comments about disconnect() in Selector
> 
> 
> changed logging level to debug
> 
> 
> Addressed Joels comments to break out early from the loop while aborting 
> expired batches
> 
> 
> Addressed Jun's comments
> 
> 
> Addressed Jason's concern about iterating over timeout request in 
> getNodesWithTimedOutRequest()
> 
> 
> Addressed Joel's comments
> 
> 
> Diffs
> -
> 
>   clients/src/main/java/org/apache/kafka/clients/ClientRequest.java 
> dc8f0f115bcda893c95d17c0a57be8d14518d034 
>   clients/src/main/java/org/apache/kafka/clients/CommonClientConfigs.java 
> 7d24c6f5dd2b63b96584f3aa8922a1d048dc1ae4 
>   clients/src/main/java/org/apache/kafka/clients/InFlightRequests.java 
> 15d00d4e484bb5d51a9ae6857ed6e024a2cc1820 
>   clients/src/main/java/org/apache/kafka/clients/KafkaClient.java 
> f46c0d9b5eb73887c62a0e09c96e9d8c964c709d 
>   clients/src/main/java/org/apache/kafka/clients/NetworkClient.java 
> 51a6c5dc47c4008e92b9fdd2ee359b573184e8ed 
>   clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerConfig.java 
> b9a2d4e2bc565f0ee72b27791afe5c894af262f1 
>   clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java 
> 3ac2be840c5d313a497ab67dc024d1578d4679fd 
>   
> clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerNetworkClient.java
>  9517d9d0cd480d5ba1d12f1fde7963e60528d2f8 
>   clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 
> 804d569498396d431880641041fc9292076452cb 
>   clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 
> 06f00a99a73a288df9afa8c1d4abe3580fa968a6 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/BufferPool.java
>  4cb1e50d6c4ed55241aeaef1d3af09def5274103 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java
>  a152bd7697dca55609a9ec4cfe0a82c10595fbc3 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java
>  06182db1c3a5da85648199b4c0c98b80ea7c6c0c 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 
> d2e64f7cd8bf56e433a210905b2874f71eee9ea0 
>   clients/src/main/java/org/apache/kafka/common/network/Selectable.java 
> 70e74bd6aa629c430b2850ca40c97df0b16e5d75 
>   clients/src/main/java/org/apache/kafka/common/network/Selector.java 
> 5a4909e4e5afe1b7ce2c33b9324d3947d2d91a38 
>   clients/src/test/java/org/apache/kafka/clients/MockClient.java 
> e5815f56bdf8e2d980f2bc36b831ed234c0ac781 
>   clients/src/test/java/org/apache/kafka/clients/NetworkClientTest.java 
> 69c93c3adf674b1640534c3d7410fcaafaf2232c 
>   
> clients/src/test/java/org/apache/kafka/clients/producer/internals/BufferPoolTest.java
>  2c693824fa53db1e38766b8c66a0ef42ef9d0f3a 
>   
> clients/src/test/java/org/apache/kafka/clients/producer/internals/RecordAccumulatorTest.java
>  5b2e4ffaeab7127648db608c179703b27b577414 
>   
> clients/src/test/java/org/apache/kafka/clients/producer/internals/SenderTest.java
>  aa44991777a855f4b7f4f7bf17107c69393ff8ff 
>   clients/src/test/java/org/apache/kafka/common/network/SSLSelectorTest.java 
> df1205c935bee9a30a50816dbade64d6014b1ef2 
>   clients/src/test/java/org/apache/kafka/common/network/SelectorTest.java 
> 3a684d98b05cadfb25c6f7f9a038ef1f6697edbf 
>   clients/src/test/java/org/apache/kafka/test/MockSelector.java 
> f83fd9b794a3bd191121a22bcb40fd6ec31d83b5 
>   core/src/main/scala/kafka/controller/ControllerChannelManager.scala 
> b1cf668dbb5bc8617d320fd3a307c6c8005b78a2 
>   core/src/main/scala/kafka/server/KafkaConfig.scala 
> 

Re: Review Request 36858: Patch for KAFKA-2120

[Joel Koshy]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36858/#review99098
---


Looks good overall - only have minor comments.

Got this compilation error:
```
ReplicaFetcherThread.scala:79: overloaded method constructor NetworkClient with 
alternatives:
  (x$1: org.apache.kafka.common.network.Selectable,x$2: 
org.apache.kafka.clients.MetadataUpdater,x$3: String,x$4: Int,x$5: Long,x$6: 
Int,x$7: Int,x$8: Int)org.apache.kafka.clients.NetworkClient 
  (x$1: org.apache.kafka.common.network.Selectable,x$2: 
org.apache.kafka.clients.Metadata,x$3: String,x$4: Int,x$5: Long,x$6: Int,x$7: 
Int,x$8: Int)org.apache.kafka.clients.NetworkClient
 cannot be applied to (org.apache.kafka.common.network.Selector, 
org.apache.kafka.clients.ManualMetadataUpdater, String, Int, Int, Int, Integer)
new NetworkClient(
 ```


clients/src/main/java/org/apache/kafka/clients/NetworkClient.java (line 426)


`Disconnecting from node {} due to request timeout.` (i.e., to make it 
clear that the client is initiating the disconnect here)



clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 
(line 687)


`checkMaybeGetRemainingTime`?



clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 
(line 49)


`removed in a future release` (here and elsewhere) sounds a little less 
vague.



clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 
(line 141)


`Also, the {@link #METADATA_FETCH_TIMEOUT} is no longer honored when this 
property is set to true.`



clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java
 (line 132)


Perhaps add a comment:
`// we need to explicitly check if the record is full or linger time is met 
because the accumulator's partition may not be ready if the leader is 
unavailable`



clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java
 (line 142)


R



clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java
 (line 149)


Set...
is being retried (for send)



clients/src/test/java/org/apache/kafka/clients/producer/internals/RecordAccumulatorTest.java
 (line 314)


rename to `expiredBatches`



clients/src/test/java/org/apache/kafka/clients/producer/internals/RecordAccumulatorTest.java
 (line 320)


This is unrelated to expiration right?



core/src/main/scala/kafka/tools/ProducerPerformance.scala (line 126)


revert!


- Joel Koshy


On Sept. 11, 2015, 9:54 p.m., Mayuresh Gharat wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/36858/
> ---
> 
> (Updated Sept. 11, 2015, 9:54 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-2120
> https://issues.apache.org/jira/browse/KAFKA-2120
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> Solved compile error
> 
> 
> Addressed Jason's comments for Kip-19
> 
> 
> Addressed Jun's comments
> 
> 
> Addressed Jason's comments about the default values for requestTimeout
> 
> 
> checkpoint
> 
> 
> Addressed Joel's concerns. Also tried to include Jun's feedback.
> 
> 
> Fixed a minor comment
> 
> 
> Solved unittest issue
> 
> 
> Addressed Jun's comments regarding NetworkClient
> 
> 
> Addressed Jun's comments about disconnect() in Selector
> 
> 
> changed logging level to debug
> 
> 
> Addressed Joels comments to break out early from the loop while aborting 
> expired batches
> 
> 
> Addressed Jun's comments
> 
> 
> Addressed Jason's concern about iterating over timeout request in 
> getNodesWithTimedOutRequest()
> 
> 
> Diffs
> -
> 
>   clients/src/main/java/org/apache/kafka/clients/ClientRequest.java 
> dc8f0f115bcda893c95d17c0a57be8d14518d034 
>   clients/src/main/java/org/apache/kafka/clients/CommonClientConfigs.java 
> 7d24c6f5dd2b63b96584f3aa8922a1d048dc1ae4 
>   clients/src/main/java/org/apache/kafka/clients/InFlightRequests.java 
> 15d00d4e484bb5d51a9ae6857ed6e024a2cc1820 
>   clients/src/main/java/org/apache/kafka/clients/KafkaClient.java 
> f46c0d9b5eb73887c62a0e09c96e9d8c964c709d 
>   clients/src/main/java/org/apache/kafka/clients/NetworkClient.java 
> 1302f359791b657b6f7c1ca1bd419ded9b01c67d 
>   

Re: Review Request 36858: Patch for KAFKA-2120

[Mayuresh Gharat]

> On Aug. 26, 2015, 8:20 p.m., Joel Koshy wrote:
> > clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java, 
> > line 230
> > 
> >
> > This logic is a bit confusing. Is this block necessary here? i.e., vs 
> > being written once below?

Thought of doing it here since we will have to check the configs at some point 
to give warnings to users regarding the usage of the deprecated configs.


> On Aug. 26, 2015, 8:20 p.m., Joel Koshy wrote:
> > clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java, 
> > line 247
> > 
> >
> > I'm not sure we should remove the _replication timeout_ though. i.e., 
> > sure the replication timeout should not be used for request timeout going 
> > forward, but we still need a replication timeout in the producer request.

As per the KIP, requestTimeout will be used for replicationTimeout going 
forward.


> On Aug. 26, 2015, 8:20 p.m., Joel Koshy wrote:
> > clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java,
> >  line 208
> > 
> >
> > Maybe drop the "due to...". This would be just one possibility right? 
> > E.g., you could have a low request timeout and high linger time... possibly 
> > other scenarios as well.

We expire batches only if the metadata is unavailable, as per the KIP.


- Mayuresh


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36858/#review96095
---


On Sept. 16, 2015, 1:57 a.m., Mayuresh Gharat wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/36858/
> ---
> 
> (Updated Sept. 16, 2015, 1:57 a.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-2120
> https://issues.apache.org/jira/browse/KAFKA-2120
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> Solved compile error
> 
> 
> Addressed Jason's comments for Kip-19
> 
> 
> Addressed Jun's comments
> 
> 
> Addressed Jason's comments about the default values for requestTimeout
> 
> 
> checkpoint
> 
> 
> Addressed Joel's concerns. Also tried to include Jun's feedback.
> 
> 
> Fixed a minor comment
> 
> 
> Solved unittest issue
> 
> 
> Addressed Jun's comments regarding NetworkClient
> 
> 
> Addressed Jun's comments about disconnect() in Selector
> 
> 
> changed logging level to debug
> 
> 
> Addressed Joels comments to break out early from the loop while aborting 
> expired batches
> 
> 
> Addressed Jun's comments
> 
> 
> Addressed Jason's concern about iterating over timeout request in 
> getNodesWithTimedOutRequest()
> 
> 
> Addressed Joel's comments
> 
> 
> Diffs
> -
> 
>   clients/src/main/java/org/apache/kafka/clients/ClientRequest.java 
> dc8f0f115bcda893c95d17c0a57be8d14518d034 
>   clients/src/main/java/org/apache/kafka/clients/CommonClientConfigs.java 
> 7d24c6f5dd2b63b96584f3aa8922a1d048dc1ae4 
>   clients/src/main/java/org/apache/kafka/clients/InFlightRequests.java 
> 15d00d4e484bb5d51a9ae6857ed6e024a2cc1820 
>   clients/src/main/java/org/apache/kafka/clients/KafkaClient.java 
> f46c0d9b5eb73887c62a0e09c96e9d8c964c709d 
>   clients/src/main/java/org/apache/kafka/clients/NetworkClient.java 
> 51a6c5dc47c4008e92b9fdd2ee359b573184e8ed 
>   clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerConfig.java 
> b9a2d4e2bc565f0ee72b27791afe5c894af262f1 
>   clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java 
> 3ac2be840c5d313a497ab67dc024d1578d4679fd 
>   
> clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerNetworkClient.java
>  9517d9d0cd480d5ba1d12f1fde7963e60528d2f8 
>   clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 
> 804d569498396d431880641041fc9292076452cb 
>   clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 
> 06f00a99a73a288df9afa8c1d4abe3580fa968a6 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/BufferPool.java
>  4cb1e50d6c4ed55241aeaef1d3af09def5274103 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java
>  a152bd7697dca55609a9ec4cfe0a82c10595fbc3 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java
>  06182db1c3a5da85648199b4c0c98b80ea7c6c0c 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 
> d2e64f7cd8bf56e433a210905b2874f71eee9ea0 
>   clients/src/main/java/org/apache/kafka/common/network/Selectable.java 
> 70e74bd6aa629c430b2850ca40c97df0b16e5d75 
>   

Re: Review Request 36858: Patch for KAFKA-2120

[Mayuresh Gharat]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36858/
---

(Updated Sept. 16, 2015, 1:57 a.m.)


Review request for kafka.


Bugs: KAFKA-2120
https://issues.apache.org/jira/browse/KAFKA-2120


Repository: kafka


Description (updated)
---

Solved compile error


Addressed Jason's comments for Kip-19


Addressed Jun's comments


Addressed Jason's comments about the default values for requestTimeout


checkpoint


Addressed Joel's concerns. Also tried to include Jun's feedback.


Fixed a minor comment


Solved unittest issue


Addressed Jun's comments regarding NetworkClient


Addressed Jun's comments about disconnect() in Selector


changed logging level to debug


Addressed Joels comments to break out early from the loop while aborting 
expired batches


Addressed Jun's comments


Addressed Jason's concern about iterating over timeout request in 
getNodesWithTimedOutRequest()


Addressed Joel's comments


Diffs (updated)
-

  clients/src/main/java/org/apache/kafka/clients/ClientRequest.java 
dc8f0f115bcda893c95d17c0a57be8d14518d034 
  clients/src/main/java/org/apache/kafka/clients/CommonClientConfigs.java 
7d24c6f5dd2b63b96584f3aa8922a1d048dc1ae4 
  clients/src/main/java/org/apache/kafka/clients/InFlightRequests.java 
15d00d4e484bb5d51a9ae6857ed6e024a2cc1820 
  clients/src/main/java/org/apache/kafka/clients/KafkaClient.java 
f46c0d9b5eb73887c62a0e09c96e9d8c964c709d 
  clients/src/main/java/org/apache/kafka/clients/NetworkClient.java 
51a6c5dc47c4008e92b9fdd2ee359b573184e8ed 
  clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerConfig.java 
b9a2d4e2bc565f0ee72b27791afe5c894af262f1 
  clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java 
3ac2be840c5d313a497ab67dc024d1578d4679fd 
  
clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerNetworkClient.java
 9517d9d0cd480d5ba1d12f1fde7963e60528d2f8 
  clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 
804d569498396d431880641041fc9292076452cb 
  clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 
06f00a99a73a288df9afa8c1d4abe3580fa968a6 
  
clients/src/main/java/org/apache/kafka/clients/producer/internals/BufferPool.java
 4cb1e50d6c4ed55241aeaef1d3af09def5274103 
  
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java
 a152bd7697dca55609a9ec4cfe0a82c10595fbc3 
  
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java
 06182db1c3a5da85648199b4c0c98b80ea7c6c0c 
  clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 
d2e64f7cd8bf56e433a210905b2874f71eee9ea0 
  clients/src/main/java/org/apache/kafka/common/network/Selectable.java 
70e74bd6aa629c430b2850ca40c97df0b16e5d75 
  clients/src/main/java/org/apache/kafka/common/network/Selector.java 
5a4909e4e5afe1b7ce2c33b9324d3947d2d91a38 
  clients/src/test/java/org/apache/kafka/clients/MockClient.java 
e5815f56bdf8e2d980f2bc36b831ed234c0ac781 
  clients/src/test/java/org/apache/kafka/clients/NetworkClientTest.java 
69c93c3adf674b1640534c3d7410fcaafaf2232c 
  
clients/src/test/java/org/apache/kafka/clients/producer/internals/BufferPoolTest.java
 2c693824fa53db1e38766b8c66a0ef42ef9d0f3a 
  
clients/src/test/java/org/apache/kafka/clients/producer/internals/RecordAccumulatorTest.java
 5b2e4ffaeab7127648db608c179703b27b577414 
  
clients/src/test/java/org/apache/kafka/clients/producer/internals/SenderTest.java
 aa44991777a855f4b7f4f7bf17107c69393ff8ff 
  clients/src/test/java/org/apache/kafka/common/network/SSLSelectorTest.java 
df1205c935bee9a30a50816dbade64d6014b1ef2 
  clients/src/test/java/org/apache/kafka/common/network/SelectorTest.java 
3a684d98b05cadfb25c6f7f9a038ef1f6697edbf 
  clients/src/test/java/org/apache/kafka/test/MockSelector.java 
f83fd9b794a3bd191121a22bcb40fd6ec31d83b5 
  core/src/main/scala/kafka/controller/ControllerChannelManager.scala 
b1cf668dbb5bc8617d320fd3a307c6c8005b78a2 
  core/src/main/scala/kafka/server/KafkaConfig.scala 
1e8b2331486ffe55bfcc0919e48e12aad23b7d3c 
  core/src/main/scala/kafka/server/KafkaServer.scala 
f3f1fa653b9c85e729cbe32b63669011ada46547 
  core/src/main/scala/kafka/server/ReplicaFetcherThread.scala 
6c85e52e172bfc0278e724068a48a271bc15dc9f 
  core/src/main/scala/kafka/tools/ProducerPerformance.scala 
46a68e97b8bcc8821f21e4220ce9b3acedc5dafe 
  core/src/main/scala/kafka/utils/NetworkClientBlockingOps.scala 
ad10721de844725f27a116611209992cea61b088 
  core/src/test/scala/integration/kafka/api/ProducerFailureHandlingTest.scala 
1198df02ddd7727269e84a751ba99520f6d5584a 
  core/src/test/scala/unit/kafka/server/KafkaConfigTest.scala 
5b4f2db4607ae6d17696c1140f1a771ce75c80e0 
  core/src/test/scala/unit/kafka/utils/TestUtils.scala 
09b8444c2add87f0f70dbb182e892977a6b5c243 

Diff: https://reviews.apache.org/r/36858/diff/

Re: Review Request 36858: Patch for KAFKA-2120

[Jason Gustafson]

> On Sept. 11, 2015, 1:44 a.m., Jason Gustafson wrote:
> > clients/src/main/java/org/apache/kafka/clients/InFlightRequests.java, line 
> > 140
> > 
> >
> > This line is puzzling me a little bit. Wouldn't we want to use the 
> > oldest request for this check. If we use the last sent request, then we 
> > might miss older requests that had already timed out, which would tend to 
> > delay timeout detection. Maybe I'm missing something?
> 
> Mayuresh Gharat wrote:
> The inflight request are sent one after another. The reason we use 
> lastSent() is because if we know that if the last Sent is timed out, it means 
> that all older requests should have been timed out.
> 
> Jason Gustafson wrote:
> Ok, so the guarantee this provides is that each request will wait at 
> least the timeout to be fulfilled, but some could wait longer. This treats 
> the timeout setting as a minimum. The downside is that it will take longer to 
> detect a node failure since we always have to wait for the latest request to 
> timeout (which will delay the timeout for older requests). Is that right? 
> Alternatively, if you used the oldest request, the timeout would be treated 
> as a maximum: no request would wait longer than the timeout, but some could 
> be cancelled sooner. In general, this would lead to earlier failure 
> detection. I'm not sure I see the benefit of using the minimum approach, but 
> it might be worthwhile documenting the tradeoff if you haven't already done 
> so somewhere.
> 
> Mayuresh Gharat wrote:
> Hi Jason,
> 
> Now I understand what you are saying. Thats an excellent catch. Although 
> the time diference between the subsequent request will be minimum and my 
> current patch will detect the timeout and do metadata update, I completely 
> agree with you : We should be peeking the tail element from the queue rather 
> than the head element. Will upload a new patch for it. Thanks a lot for this.
> 
> 
> Actually thinking further we should be doing the same thing when we are 
> iterating over completedSends() in NetworkClient.handleCompletedSends(). 
> Although it works the current way, its kind of confusing if we don't keep the 
> iteration consistent. What do you think on this?

Seems like handleCompletedSends might be correct since the next Send will be at 
the front and the code doesn't allow more than one pending Send. It's a little 
confusing because completing the Send just means successfully transmitting it 
to the network layer. Does that seem right?


- Jason


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36858/#review98526
---


On Sept. 11, 2015, 9:54 p.m., Mayuresh Gharat wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/36858/
> ---
> 
> (Updated Sept. 11, 2015, 9:54 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-2120
> https://issues.apache.org/jira/browse/KAFKA-2120
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> Solved compile error
> 
> 
> Addressed Jason's comments for Kip-19
> 
> 
> Addressed Jun's comments
> 
> 
> Addressed Jason's comments about the default values for requestTimeout
> 
> 
> checkpoint
> 
> 
> Addressed Joel's concerns. Also tried to include Jun's feedback.
> 
> 
> Fixed a minor comment
> 
> 
> Solved unittest issue
> 
> 
> Addressed Jun's comments regarding NetworkClient
> 
> 
> Addressed Jun's comments about disconnect() in Selector
> 
> 
> changed logging level to debug
> 
> 
> Addressed Joels comments to break out early from the loop while aborting 
> expired batches
> 
> 
> Addressed Jun's comments
> 
> 
> Addressed Jason's concern about iterating over timeout request in 
> getNodesWithTimedOutRequest()
> 
> 
> Diffs
> -
> 
>   clients/src/main/java/org/apache/kafka/clients/ClientRequest.java 
> dc8f0f115bcda893c95d17c0a57be8d14518d034 
>   clients/src/main/java/org/apache/kafka/clients/CommonClientConfigs.java 
> 7d24c6f5dd2b63b96584f3aa8922a1d048dc1ae4 
>   clients/src/main/java/org/apache/kafka/clients/InFlightRequests.java 
> 15d00d4e484bb5d51a9ae6857ed6e024a2cc1820 
>   clients/src/main/java/org/apache/kafka/clients/KafkaClient.java 
> f46c0d9b5eb73887c62a0e09c96e9d8c964c709d 
>   clients/src/main/java/org/apache/kafka/clients/NetworkClient.java 
> 1302f359791b657b6f7c1ca1bd419ded9b01c67d 
>   clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerConfig.java 
> b9a2d4e2bc565f0ee72b27791afe5c894af262f1 
>   clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java 
> 5763bac6cfe667bfbabc5f160f35fb85f9b158e3 
>   
> 

Re: Review Request 36858: Patch for KAFKA-2120

[Jason Gustafson]

> On Sept. 11, 2015, 1:44 a.m., Jason Gustafson wrote:
> > clients/src/main/java/org/apache/kafka/clients/InFlightRequests.java, line 
> > 140
> > 
> >
> > This line is puzzling me a little bit. Wouldn't we want to use the 
> > oldest request for this check. If we use the last sent request, then we 
> > might miss older requests that had already timed out, which would tend to 
> > delay timeout detection. Maybe I'm missing something?
> 
> Mayuresh Gharat wrote:
> The inflight request are sent one after another. The reason we use 
> lastSent() is because if we know that if the last Sent is timed out, it means 
> that all older requests should have been timed out.

Ok, so the guarantee this provides is that each request will wait at least the 
timeout to be fulfilled, but some could wait longer. This treats the timeout 
setting as a minimum. The downside is that it will take longer to detect a node 
failure since we always have to wait for the latest request to timeout (which 
will delay the timeout for older requests). Is that right? Alternatively, if 
you used the oldest request, the timeout would be treated as a maximum: no 
request would wait longer than the timeout, but some could be cancelled sooner. 
In general, this would lead to earlier failure detection. I'm not sure I see 
the benefit of using the minimum approach, but it might be worthwhile 
documenting the tradeoff if you haven't already done so somewhere.


- Jason


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36858/#review98526
---


On Sept. 11, 2015, 4:39 a.m., Mayuresh Gharat wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/36858/
> ---
> 
> (Updated Sept. 11, 2015, 4:39 a.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-2120
> https://issues.apache.org/jira/browse/KAFKA-2120
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> Solved compile error
> 
> 
> Addressed Jason's comments for Kip-19
> 
> 
> Addressed Jun's comments
> 
> 
> Addressed Jason's comments about the default values for requestTimeout
> 
> 
> checkpoint
> 
> 
> Addressed Joel's concerns. Also tried to include Jun's feedback.
> 
> 
> Fixed a minor comment
> 
> 
> Solved unittest issue
> 
> 
> Addressed Jun's comments regarding NetworkClient
> 
> 
> Addressed Jun's comments about disconnect() in Selector
> 
> 
> changed logging level to debug
> 
> 
> Addressed Joels comments to break out early from the loop while aborting 
> expired batches
> 
> 
> Addressed Jun's comments
> 
> 
> Diffs
> -
> 
>   clients/src/main/java/org/apache/kafka/clients/ClientRequest.java 
> dc8f0f115bcda893c95d17c0a57be8d14518d034 
>   clients/src/main/java/org/apache/kafka/clients/CommonClientConfigs.java 
> 7d24c6f5dd2b63b96584f3aa8922a1d048dc1ae4 
>   clients/src/main/java/org/apache/kafka/clients/InFlightRequests.java 
> 15d00d4e484bb5d51a9ae6857ed6e024a2cc1820 
>   clients/src/main/java/org/apache/kafka/clients/KafkaClient.java 
> f46c0d9b5eb73887c62a0e09c96e9d8c964c709d 
>   clients/src/main/java/org/apache/kafka/clients/NetworkClient.java 
> 049b22eadd5496b70dfcfd9d821f67c62c68a052 
>   clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerConfig.java 
> b9a2d4e2bc565f0ee72b27791afe5c894af262f1 
>   clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java 
> 19ef6ebead3dc97d4912985e35b6b8d17503cb0e 
>   
> clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerNetworkClient.java
>  9517d9d0cd480d5ba1d12f1fde7963e60528d2f8 
>   clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 
> 804d569498396d431880641041fc9292076452cb 
>   clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 
> 06f00a99a73a288df9afa8c1d4abe3580fa968a6 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/BufferPool.java
>  4cb1e50d6c4ed55241aeaef1d3af09def5274103 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java
>  a152bd7697dca55609a9ec4cfe0a82c10595fbc3 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java
>  06182db1c3a5da85648199b4c0c98b80ea7c6c0c 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 
> d2e64f7cd8bf56e433a210905b2874f71eee9ea0 
>   clients/src/main/java/org/apache/kafka/common/network/Selectable.java 
> 70e74bd6aa629c430b2850ca40c97df0b16e5d75 
>   clients/src/main/java/org/apache/kafka/common/network/Selector.java 
> 4aa5cbb86ce6e1bf8f6769147ee2a6452c855c74 
>   clients/src/test/java/org/apache/kafka/clients/MockClient.java 
> 

Re: Review Request 36858: Patch for KAFKA-2120

[Ismael Juma]

> On Sept. 10, 2015, 6:19 p.m., Ismael Juma wrote:
> > clients/src/main/java/org/apache/kafka/clients/InFlightRequests.java, line 
> > 137
> > 
> >
> > Out of curiosity, why are we using `LinkedList` here? Generally 
> > `ArrayList` is a better default, but maybe there's a good reason to deviate 
> > in this case.
> 
> Mayuresh Gharat wrote:
> LinkedList add method gives O(1) performance while ArrayList gives O(n) 
> in worst case. Since we are just adding and not doing random accesses 
> LinkedList would be better in this case.

This analysis is a bit incomplete: ArrayList has amortised O(1) append and 
simply O(1) append if you know the expected size. Furthermore, it is much more 
cache-friendly and memory efficient than LinkedList. These factors mean that 
it's almost always a better data structure than LinkedList in modern hardware.


- Ismael


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36858/#review98438
---


On Sept. 11, 2015, 4:39 a.m., Mayuresh Gharat wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/36858/
> ---
> 
> (Updated Sept. 11, 2015, 4:39 a.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-2120
> https://issues.apache.org/jira/browse/KAFKA-2120
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> Solved compile error
> 
> 
> Addressed Jason's comments for Kip-19
> 
> 
> Addressed Jun's comments
> 
> 
> Addressed Jason's comments about the default values for requestTimeout
> 
> 
> checkpoint
> 
> 
> Addressed Joel's concerns. Also tried to include Jun's feedback.
> 
> 
> Fixed a minor comment
> 
> 
> Solved unittest issue
> 
> 
> Addressed Jun's comments regarding NetworkClient
> 
> 
> Addressed Jun's comments about disconnect() in Selector
> 
> 
> changed logging level to debug
> 
> 
> Addressed Joels comments to break out early from the loop while aborting 
> expired batches
> 
> 
> Addressed Jun's comments
> 
> 
> Diffs
> -
> 
>   clients/src/main/java/org/apache/kafka/clients/ClientRequest.java 
> dc8f0f115bcda893c95d17c0a57be8d14518d034 
>   clients/src/main/java/org/apache/kafka/clients/CommonClientConfigs.java 
> 7d24c6f5dd2b63b96584f3aa8922a1d048dc1ae4 
>   clients/src/main/java/org/apache/kafka/clients/InFlightRequests.java 
> 15d00d4e484bb5d51a9ae6857ed6e024a2cc1820 
>   clients/src/main/java/org/apache/kafka/clients/KafkaClient.java 
> f46c0d9b5eb73887c62a0e09c96e9d8c964c709d 
>   clients/src/main/java/org/apache/kafka/clients/NetworkClient.java 
> 049b22eadd5496b70dfcfd9d821f67c62c68a052 
>   clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerConfig.java 
> b9a2d4e2bc565f0ee72b27791afe5c894af262f1 
>   clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java 
> 19ef6ebead3dc97d4912985e35b6b8d17503cb0e 
>   
> clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerNetworkClient.java
>  9517d9d0cd480d5ba1d12f1fde7963e60528d2f8 
>   clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 
> 804d569498396d431880641041fc9292076452cb 
>   clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 
> 06f00a99a73a288df9afa8c1d4abe3580fa968a6 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/BufferPool.java
>  4cb1e50d6c4ed55241aeaef1d3af09def5274103 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java
>  a152bd7697dca55609a9ec4cfe0a82c10595fbc3 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java
>  06182db1c3a5da85648199b4c0c98b80ea7c6c0c 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 
> d2e64f7cd8bf56e433a210905b2874f71eee9ea0 
>   clients/src/main/java/org/apache/kafka/common/network/Selectable.java 
> 70e74bd6aa629c430b2850ca40c97df0b16e5d75 
>   clients/src/main/java/org/apache/kafka/common/network/Selector.java 
> 4aa5cbb86ce6e1bf8f6769147ee2a6452c855c74 
>   clients/src/test/java/org/apache/kafka/clients/MockClient.java 
> e5815f56bdf8e2d980f2bc36b831ed234c0ac781 
>   clients/src/test/java/org/apache/kafka/clients/NetworkClientTest.java 
> 69c93c3adf674b1640534c3d7410fcaafaf2232c 
>   
> clients/src/test/java/org/apache/kafka/clients/producer/internals/BufferPoolTest.java
>  2c693824fa53db1e38766b8c66a0ef42ef9d0f3a 
>   
> clients/src/test/java/org/apache/kafka/clients/producer/internals/RecordAccumulatorTest.java
>  5b2e4ffaeab7127648db608c179703b27b577414 
>   
> clients/src/test/java/org/apache/kafka/clients/producer/internals/SenderTest.java
>  aa44991777a855f4b7f4f7bf17107c69393ff8ff 
>   

Re: Review Request 36858: Patch for KAFKA-2120

[Mayuresh Gharat]

> On Sept. 11, 2015, 1:44 a.m., Jason Gustafson wrote:
> > clients/src/main/java/org/apache/kafka/clients/InFlightRequests.java, line 
> > 140
> > 
> >
> > This line is puzzling me a little bit. Wouldn't we want to use the 
> > oldest request for this check. If we use the last sent request, then we 
> > might miss older requests that had already timed out, which would tend to 
> > delay timeout detection. Maybe I'm missing something?
> 
> Mayuresh Gharat wrote:
> The inflight request are sent one after another. The reason we use 
> lastSent() is because if we know that if the last Sent is timed out, it means 
> that all older requests should have been timed out.
> 
> Jason Gustafson wrote:
> Ok, so the guarantee this provides is that each request will wait at 
> least the timeout to be fulfilled, but some could wait longer. This treats 
> the timeout setting as a minimum. The downside is that it will take longer to 
> detect a node failure since we always have to wait for the latest request to 
> timeout (which will delay the timeout for older requests). Is that right? 
> Alternatively, if you used the oldest request, the timeout would be treated 
> as a maximum: no request would wait longer than the timeout, but some could 
> be cancelled sooner. In general, this would lead to earlier failure 
> detection. I'm not sure I see the benefit of using the minimum approach, but 
> it might be worthwhile documenting the tradeoff if you haven't already done 
> so somewhere.

Hi Jason,

Now I understand what you are saying. Thats an excellent catch. Although the 
time diference between the subsequent request will be minimum and my current 
patch will detect the timeout and do metadata update, I completely agree with 
you : We should be peeking the tail element from the queue rather than the head 
element. Will upload a new patch for it. Thanks a lot for this.


Actually thinking further we should be doing the same thing when we are 
iterating over completedSends() in NetworkClient.handleCompletedSends(). 
Although it works the current way, its kind of confusing if we don't keep the 
iteration consistent. What do you think on this?


- Mayuresh


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36858/#review98526
---


On Sept. 11, 2015, 9:54 p.m., Mayuresh Gharat wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/36858/
> ---
> 
> (Updated Sept. 11, 2015, 9:54 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-2120
> https://issues.apache.org/jira/browse/KAFKA-2120
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> Solved compile error
> 
> 
> Addressed Jason's comments for Kip-19
> 
> 
> Addressed Jun's comments
> 
> 
> Addressed Jason's comments about the default values for requestTimeout
> 
> 
> checkpoint
> 
> 
> Addressed Joel's concerns. Also tried to include Jun's feedback.
> 
> 
> Fixed a minor comment
> 
> 
> Solved unittest issue
> 
> 
> Addressed Jun's comments regarding NetworkClient
> 
> 
> Addressed Jun's comments about disconnect() in Selector
> 
> 
> changed logging level to debug
> 
> 
> Addressed Joels comments to break out early from the loop while aborting 
> expired batches
> 
> 
> Addressed Jun's comments
> 
> 
> Addressed Jason's concern about iterating over timeout request in 
> getNodesWithTimedOutRequest()
> 
> 
> Diffs
> -
> 
>   clients/src/main/java/org/apache/kafka/clients/ClientRequest.java 
> dc8f0f115bcda893c95d17c0a57be8d14518d034 
>   clients/src/main/java/org/apache/kafka/clients/CommonClientConfigs.java 
> 7d24c6f5dd2b63b96584f3aa8922a1d048dc1ae4 
>   clients/src/main/java/org/apache/kafka/clients/InFlightRequests.java 
> 15d00d4e484bb5d51a9ae6857ed6e024a2cc1820 
>   clients/src/main/java/org/apache/kafka/clients/KafkaClient.java 
> f46c0d9b5eb73887c62a0e09c96e9d8c964c709d 
>   clients/src/main/java/org/apache/kafka/clients/NetworkClient.java 
> 1302f359791b657b6f7c1ca1bd419ded9b01c67d 
>   clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerConfig.java 
> b9a2d4e2bc565f0ee72b27791afe5c894af262f1 
>   clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java 
> 5763bac6cfe667bfbabc5f160f35fb85f9b158e3 
>   
> clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerNetworkClient.java
>  9517d9d0cd480d5ba1d12f1fde7963e60528d2f8 
>   clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 
> 804d569498396d431880641041fc9292076452cb 
>   clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 
> 06f00a99a73a288df9afa8c1d4abe3580fa968a6 
>   
> 

Re: Review Request 36858: Patch for KAFKA-2120

[Mayuresh Gharat]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36858/
---

(Updated Sept. 11, 2015, 9:54 p.m.)


Review request for kafka.


Bugs: KAFKA-2120
https://issues.apache.org/jira/browse/KAFKA-2120


Repository: kafka


Description (updated)
---

Solved compile error


Addressed Jason's comments for Kip-19


Addressed Jun's comments


Addressed Jason's comments about the default values for requestTimeout


checkpoint


Addressed Joel's concerns. Also tried to include Jun's feedback.


Fixed a minor comment


Solved unittest issue


Addressed Jun's comments regarding NetworkClient


Addressed Jun's comments about disconnect() in Selector


changed logging level to debug


Addressed Joels comments to break out early from the loop while aborting 
expired batches


Addressed Jun's comments


Addressed Jason's concern about iterating over timeout request in 
getNodesWithTimedOutRequest()


Diffs (updated)
-

  clients/src/main/java/org/apache/kafka/clients/ClientRequest.java 
dc8f0f115bcda893c95d17c0a57be8d14518d034 
  clients/src/main/java/org/apache/kafka/clients/CommonClientConfigs.java 
7d24c6f5dd2b63b96584f3aa8922a1d048dc1ae4 
  clients/src/main/java/org/apache/kafka/clients/InFlightRequests.java 
15d00d4e484bb5d51a9ae6857ed6e024a2cc1820 
  clients/src/main/java/org/apache/kafka/clients/KafkaClient.java 
f46c0d9b5eb73887c62a0e09c96e9d8c964c709d 
  clients/src/main/java/org/apache/kafka/clients/NetworkClient.java 
1302f359791b657b6f7c1ca1bd419ded9b01c67d 
  clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerConfig.java 
b9a2d4e2bc565f0ee72b27791afe5c894af262f1 
  clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java 
5763bac6cfe667bfbabc5f160f35fb85f9b158e3 
  
clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerNetworkClient.java
 9517d9d0cd480d5ba1d12f1fde7963e60528d2f8 
  clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 
804d569498396d431880641041fc9292076452cb 
  clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 
06f00a99a73a288df9afa8c1d4abe3580fa968a6 
  
clients/src/main/java/org/apache/kafka/clients/producer/internals/BufferPool.java
 4cb1e50d6c4ed55241aeaef1d3af09def5274103 
  
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java
 a152bd7697dca55609a9ec4cfe0a82c10595fbc3 
  
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java
 06182db1c3a5da85648199b4c0c98b80ea7c6c0c 
  clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 
d2e64f7cd8bf56e433a210905b2874f71eee9ea0 
  clients/src/main/java/org/apache/kafka/common/network/Selectable.java 
70e74bd6aa629c430b2850ca40c97df0b16e5d75 
  clients/src/main/java/org/apache/kafka/common/network/Selector.java 
4aa5cbb86ce6e1bf8f6769147ee2a6452c855c74 
  clients/src/test/java/org/apache/kafka/clients/MockClient.java 
e5815f56bdf8e2d980f2bc36b831ed234c0ac781 
  clients/src/test/java/org/apache/kafka/clients/NetworkClientTest.java 
69c93c3adf674b1640534c3d7410fcaafaf2232c 
  
clients/src/test/java/org/apache/kafka/clients/producer/internals/BufferPoolTest.java
 2c693824fa53db1e38766b8c66a0ef42ef9d0f3a 
  
clients/src/test/java/org/apache/kafka/clients/producer/internals/RecordAccumulatorTest.java
 5b2e4ffaeab7127648db608c179703b27b577414 
  
clients/src/test/java/org/apache/kafka/clients/producer/internals/SenderTest.java
 aa44991777a855f4b7f4f7bf17107c69393ff8ff 
  clients/src/test/java/org/apache/kafka/common/network/SSLSelectorTest.java 
df1205c935bee9a30a50816dbade64d6014b1ef2 
  clients/src/test/java/org/apache/kafka/common/network/SelectorTest.java 
3a684d98b05cadfb25c6f7f9a038ef1f6697edbf 
  clients/src/test/java/org/apache/kafka/test/MockSelector.java 
f83fd9b794a3bd191121a22bcb40fd6ec31d83b5 
  core/src/main/scala/kafka/controller/ControllerChannelManager.scala 
da1cff07f7f76dcfa5a805718febcccd4ed5f578 
  core/src/main/scala/kafka/server/KafkaConfig.scala 
1e8b2331486ffe55bfcc0919e48e12aad23b7d3c 
  core/src/main/scala/kafka/server/KafkaServer.scala 
30406ce809caaac56aca1f30c235b35962d55a50 
  core/src/main/scala/kafka/tools/ProducerPerformance.scala 
46a68e97b8bcc8821f21e4220ce9b3acedc5dafe 
  core/src/main/scala/kafka/utils/NetworkClientBlockingOps.scala 
ad10721de844725f27a116611209992cea61b088 
  core/src/test/scala/integration/kafka/api/ProducerFailureHandlingTest.scala 
1198df02ddd7727269e84a751ba99520f6d5584a 
  core/src/test/scala/unit/kafka/server/KafkaConfigTest.scala 
5b4f2db4607ae6d17696c1140f1a771ce75c80e0 
  core/src/test/scala/unit/kafka/utils/TestUtils.scala 
09b8444c2add87f0f70dbb182e892977a6b5c243 

Diff: https://reviews.apache.org/r/36858/diff/


Testing
---


Thanks,

Mayuresh Gharat

Re: Review Request 36858: Patch for KAFKA-2120

[Mayuresh Gharat]

> On Sept. 9, 2015, 4:43 a.m., Jun Rao wrote:
> > clients/src/main/java/org/apache/kafka/common/network/Selector.java, lines 
> > 185-190
> > 
> >
> > Do we still need disconnect(id)? It seems that we can just replace the 
> > usage in test with close(id) and remove disconnect from selectable and 
> > KafkaChannel? We have to implement close(id) in MockSelector().
> 
> Mayuresh Gharat wrote:
> Hi Jun,
> Thats a very good suggestion. I had a small orthogonal concern here. If 
> we want people to start using NetworkClient instead of BLockingChannel, for 
> example for writing a tool for fetching and commiting offsets, it would be 
> great if we can provide a disconnect() in NetworkClient that can either call 
> close(id) in Selector or disconnect in Selector() (In this case we need to 
> change how disconnect() in Selector functions). I can add disconnect API in 
> NetworkClient as a separate patch if you feel my concern is valid.
> 
> Ismael Juma wrote:
> NetworkClient has a `close(id)` method which I have used in a number of 
> places where I replaced `BlockingChannel` by `NetworkClient`. What would the 
> proposed `disconnect` method do differently?

Ahh. yes that is what I was looking for. Thanks.


- Mayuresh


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36858/#review98136
---


On Sept. 10, 2015, 1:56 a.m., Mayuresh Gharat wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/36858/
> ---
> 
> (Updated Sept. 10, 2015, 1:56 a.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-2120
> https://issues.apache.org/jira/browse/KAFKA-2120
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> Solved compile error
> 
> 
> Addressed Jason's comments for Kip-19
> 
> 
> Addressed Jun's comments
> 
> 
> Addressed Jason's comments about the default values for requestTimeout
> 
> 
> checkpoint
> 
> 
> Addressed Joel's concerns. Also tried to include Jun's feedback.
> 
> 
> Fixed a minor comment
> 
> 
> Solved unittest issue
> 
> 
> Addressed Jun's comments regarding NetworkClient
> 
> 
> Addressed Jun's comments about disconnect() in Selector
> 
> 
> changed logging level to debug
> 
> 
> Addressed Joels comments to break out early from the loop while aborting 
> expired batches
> 
> 
> Diffs
> -
> 
>   clients/src/main/java/org/apache/kafka/clients/ClientRequest.java 
> dc8f0f115bcda893c95d17c0a57be8d14518d034 
>   clients/src/main/java/org/apache/kafka/clients/CommonClientConfigs.java 
> 7d24c6f5dd2b63b96584f3aa8922a1d048dc1ae4 
>   clients/src/main/java/org/apache/kafka/clients/InFlightRequests.java 
> 15d00d4e484bb5d51a9ae6857ed6e024a2cc1820 
>   clients/src/main/java/org/apache/kafka/clients/KafkaClient.java 
> f46c0d9b5eb73887c62a0e09c96e9d8c964c709d 
>   clients/src/main/java/org/apache/kafka/clients/NetworkClient.java 
> 049b22eadd5496b70dfcfd9d821f67c62c68a052 
>   clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerConfig.java 
> b9a2d4e2bc565f0ee72b27791afe5c894af262f1 
>   clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java 
> 19ef6ebead3dc97d4912985e35b6b8d17503cb0e 
>   
> clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerNetworkClient.java
>  9517d9d0cd480d5ba1d12f1fde7963e60528d2f8 
>   clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 
> 804d569498396d431880641041fc9292076452cb 
>   clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 
> 06f00a99a73a288df9afa8c1d4abe3580fa968a6 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/BufferPool.java
>  4cb1e50d6c4ed55241aeaef1d3af09def5274103 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java
>  a152bd7697dca55609a9ec4cfe0a82c10595fbc3 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java
>  06182db1c3a5da85648199b4c0c98b80ea7c6c0c 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 
> d2e64f7cd8bf56e433a210905b2874f71eee9ea0 
>   clients/src/main/java/org/apache/kafka/common/network/Selectable.java 
> 70e74bd6aa629c430b2850ca40c97df0b16e5d75 
>   clients/src/main/java/org/apache/kafka/common/network/Selector.java 
> 4aa5cbb86ce6e1bf8f6769147ee2a6452c855c74 
>   clients/src/test/java/org/apache/kafka/clients/MockClient.java 
> e5815f56bdf8e2d980f2bc36b831ed234c0ac781 
>   clients/src/test/java/org/apache/kafka/clients/NetworkClientTest.java 
> 69c93c3adf674b1640534c3d7410fcaafaf2232c 
>   
> clients/src/test/java/org/apache/kafka/clients/producer/internals/BufferPoolTest.java
>  

Re: Review Request 36858: Patch for KAFKA-2120

[Ismael Juma]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36858/#review98438
---



clients/src/main/java/org/apache/kafka/clients/InFlightRequests.java (line 137)


Out of curiosity, why are we using `LinkedList` here? Generally `ArrayList` 
is a better default, but maybe there's a good reason to deviate in this case.



clients/src/main/java/org/apache/kafka/common/network/Selector.java (line 477)


Sorry if this has already been covered before, but why are we logging a 
warning in this case instead of throwing an exception? What should the user do 
if he sees this warning in the logs?


- Ismael Juma


On Sept. 10, 2015, 1:56 a.m., Mayuresh Gharat wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/36858/
> ---
> 
> (Updated Sept. 10, 2015, 1:56 a.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-2120
> https://issues.apache.org/jira/browse/KAFKA-2120
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> Solved compile error
> 
> 
> Addressed Jason's comments for Kip-19
> 
> 
> Addressed Jun's comments
> 
> 
> Addressed Jason's comments about the default values for requestTimeout
> 
> 
> checkpoint
> 
> 
> Addressed Joel's concerns. Also tried to include Jun's feedback.
> 
> 
> Fixed a minor comment
> 
> 
> Solved unittest issue
> 
> 
> Addressed Jun's comments regarding NetworkClient
> 
> 
> Addressed Jun's comments about disconnect() in Selector
> 
> 
> changed logging level to debug
> 
> 
> Addressed Joels comments to break out early from the loop while aborting 
> expired batches
> 
> 
> Diffs
> -
> 
>   clients/src/main/java/org/apache/kafka/clients/ClientRequest.java 
> dc8f0f115bcda893c95d17c0a57be8d14518d034 
>   clients/src/main/java/org/apache/kafka/clients/CommonClientConfigs.java 
> 7d24c6f5dd2b63b96584f3aa8922a1d048dc1ae4 
>   clients/src/main/java/org/apache/kafka/clients/InFlightRequests.java 
> 15d00d4e484bb5d51a9ae6857ed6e024a2cc1820 
>   clients/src/main/java/org/apache/kafka/clients/KafkaClient.java 
> f46c0d9b5eb73887c62a0e09c96e9d8c964c709d 
>   clients/src/main/java/org/apache/kafka/clients/NetworkClient.java 
> 049b22eadd5496b70dfcfd9d821f67c62c68a052 
>   clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerConfig.java 
> b9a2d4e2bc565f0ee72b27791afe5c894af262f1 
>   clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java 
> 19ef6ebead3dc97d4912985e35b6b8d17503cb0e 
>   
> clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerNetworkClient.java
>  9517d9d0cd480d5ba1d12f1fde7963e60528d2f8 
>   clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 
> 804d569498396d431880641041fc9292076452cb 
>   clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 
> 06f00a99a73a288df9afa8c1d4abe3580fa968a6 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/BufferPool.java
>  4cb1e50d6c4ed55241aeaef1d3af09def5274103 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java
>  a152bd7697dca55609a9ec4cfe0a82c10595fbc3 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java
>  06182db1c3a5da85648199b4c0c98b80ea7c6c0c 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 
> d2e64f7cd8bf56e433a210905b2874f71eee9ea0 
>   clients/src/main/java/org/apache/kafka/common/network/Selectable.java 
> 70e74bd6aa629c430b2850ca40c97df0b16e5d75 
>   clients/src/main/java/org/apache/kafka/common/network/Selector.java 
> 4aa5cbb86ce6e1bf8f6769147ee2a6452c855c74 
>   clients/src/test/java/org/apache/kafka/clients/MockClient.java 
> e5815f56bdf8e2d980f2bc36b831ed234c0ac781 
>   clients/src/test/java/org/apache/kafka/clients/NetworkClientTest.java 
> 69c93c3adf674b1640534c3d7410fcaafaf2232c 
>   
> clients/src/test/java/org/apache/kafka/clients/producer/internals/BufferPoolTest.java
>  2c693824fa53db1e38766b8c66a0ef42ef9d0f3a 
>   
> clients/src/test/java/org/apache/kafka/clients/producer/internals/RecordAccumulatorTest.java
>  5b2e4ffaeab7127648db608c179703b27b577414 
>   
> clients/src/test/java/org/apache/kafka/clients/producer/internals/SenderTest.java
>  aa44991777a855f4b7f4f7bf17107c69393ff8ff 
>   clients/src/test/java/org/apache/kafka/common/network/SSLSelectorTest.java 
> df1205c935bee9a30a50816dbade64d6014b1ef2 
>   clients/src/test/java/org/apache/kafka/common/network/SelectorTest.java 
> 3a684d98b05cadfb25c6f7f9a038ef1f6697edbf 
>   clients/src/test/java/org/apache/kafka/test/MockSelector.java 
> f83fd9b794a3bd191121a22bcb40fd6ec31d83b5 
>   

Re: Review Request 36858: Patch for KAFKA-2120

[Jun Rao]

> On Sept. 10, 2015, 6:19 p.m., Ismael Juma wrote:
> > clients/src/main/java/org/apache/kafka/common/network/Selector.java, line 
> > 488
> > 
> >
> > Sorry if this has already been covered before, but why are we logging a 
> > warning in this case instead of throwing an exception? What should the user 
> > do if he sees this warning in the logs?
> 
> Mayuresh Gharat wrote:
> This is just to tell the user what happened. Before it would throw 
> exception which would be propogated to the run() in sender. This will be 
> caught and neglected as such by logging an error level message which does not 
> print the actual error message and would only print "channel is not 
> connected". 
> 
> So IMO, we should atleast log this. Actually previously the send() in 
> Selector() would call channelForId(). If channel was null it would throw 
> exception. But still there is a check if (channel == null) in send()and it 
> throws an IllegalStateException, which will never be executed. I am thinking 
> of getting rid of it as well.
> Also we should be checking if channel is null wherever the channelForId() 
> is used.

Currently, all callers of channelForId() expect the channel to exist. So, it 
seems it's simpler to just throw IllegalStateException in channelForId() and 
propagate it to the caller. We can get rid of the null check in send().


- Jun


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36858/#review98438
---


On Sept. 10, 2015, 1:56 a.m., Mayuresh Gharat wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/36858/
> ---
> 
> (Updated Sept. 10, 2015, 1:56 a.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-2120
> https://issues.apache.org/jira/browse/KAFKA-2120
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> Solved compile error
> 
> 
> Addressed Jason's comments for Kip-19
> 
> 
> Addressed Jun's comments
> 
> 
> Addressed Jason's comments about the default values for requestTimeout
> 
> 
> checkpoint
> 
> 
> Addressed Joel's concerns. Also tried to include Jun's feedback.
> 
> 
> Fixed a minor comment
> 
> 
> Solved unittest issue
> 
> 
> Addressed Jun's comments regarding NetworkClient
> 
> 
> Addressed Jun's comments about disconnect() in Selector
> 
> 
> changed logging level to debug
> 
> 
> Addressed Joels comments to break out early from the loop while aborting 
> expired batches
> 
> 
> Diffs
> -
> 
>   clients/src/main/java/org/apache/kafka/clients/ClientRequest.java 
> dc8f0f115bcda893c95d17c0a57be8d14518d034 
>   clients/src/main/java/org/apache/kafka/clients/CommonClientConfigs.java 
> 7d24c6f5dd2b63b96584f3aa8922a1d048dc1ae4 
>   clients/src/main/java/org/apache/kafka/clients/InFlightRequests.java 
> 15d00d4e484bb5d51a9ae6857ed6e024a2cc1820 
>   clients/src/main/java/org/apache/kafka/clients/KafkaClient.java 
> f46c0d9b5eb73887c62a0e09c96e9d8c964c709d 
>   clients/src/main/java/org/apache/kafka/clients/NetworkClient.java 
> 049b22eadd5496b70dfcfd9d821f67c62c68a052 
>   clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerConfig.java 
> b9a2d4e2bc565f0ee72b27791afe5c894af262f1 
>   clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java 
> 19ef6ebead3dc97d4912985e35b6b8d17503cb0e 
>   
> clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerNetworkClient.java
>  9517d9d0cd480d5ba1d12f1fde7963e60528d2f8 
>   clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 
> 804d569498396d431880641041fc9292076452cb 
>   clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 
> 06f00a99a73a288df9afa8c1d4abe3580fa968a6 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/BufferPool.java
>  4cb1e50d6c4ed55241aeaef1d3af09def5274103 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java
>  a152bd7697dca55609a9ec4cfe0a82c10595fbc3 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java
>  06182db1c3a5da85648199b4c0c98b80ea7c6c0c 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 
> d2e64f7cd8bf56e433a210905b2874f71eee9ea0 
>   clients/src/main/java/org/apache/kafka/common/network/Selectable.java 
> 70e74bd6aa629c430b2850ca40c97df0b16e5d75 
>   clients/src/main/java/org/apache/kafka/common/network/Selector.java 
> 4aa5cbb86ce6e1bf8f6769147ee2a6452c855c74 
>   clients/src/test/java/org/apache/kafka/clients/MockClient.java 
> e5815f56bdf8e2d980f2bc36b831ed234c0ac781 
>   clients/src/test/java/org/apache/kafka/clients/NetworkClientTest.java 
> 

Re: Review Request 36858: Patch for KAFKA-2120

[Jason Gustafson]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36858/#review98526
---



clients/src/main/java/org/apache/kafka/clients/InFlightRequests.java (line 140)


This line is puzzling me a little bit. Wouldn't we want to use the oldest 
request for this check. If we use the last sent request, then we might miss 
older requests that had already timed out, which would tend to delay timeout 
detection. Maybe I'm missing something?


- Jason Gustafson


On Sept. 10, 2015, 1:56 a.m., Mayuresh Gharat wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/36858/
> ---
> 
> (Updated Sept. 10, 2015, 1:56 a.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-2120
> https://issues.apache.org/jira/browse/KAFKA-2120
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> Solved compile error
> 
> 
> Addressed Jason's comments for Kip-19
> 
> 
> Addressed Jun's comments
> 
> 
> Addressed Jason's comments about the default values for requestTimeout
> 
> 
> checkpoint
> 
> 
> Addressed Joel's concerns. Also tried to include Jun's feedback.
> 
> 
> Fixed a minor comment
> 
> 
> Solved unittest issue
> 
> 
> Addressed Jun's comments regarding NetworkClient
> 
> 
> Addressed Jun's comments about disconnect() in Selector
> 
> 
> changed logging level to debug
> 
> 
> Addressed Joels comments to break out early from the loop while aborting 
> expired batches
> 
> 
> Diffs
> -
> 
>   clients/src/main/java/org/apache/kafka/clients/ClientRequest.java 
> dc8f0f115bcda893c95d17c0a57be8d14518d034 
>   clients/src/main/java/org/apache/kafka/clients/CommonClientConfigs.java 
> 7d24c6f5dd2b63b96584f3aa8922a1d048dc1ae4 
>   clients/src/main/java/org/apache/kafka/clients/InFlightRequests.java 
> 15d00d4e484bb5d51a9ae6857ed6e024a2cc1820 
>   clients/src/main/java/org/apache/kafka/clients/KafkaClient.java 
> f46c0d9b5eb73887c62a0e09c96e9d8c964c709d 
>   clients/src/main/java/org/apache/kafka/clients/NetworkClient.java 
> 049b22eadd5496b70dfcfd9d821f67c62c68a052 
>   clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerConfig.java 
> b9a2d4e2bc565f0ee72b27791afe5c894af262f1 
>   clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java 
> 19ef6ebead3dc97d4912985e35b6b8d17503cb0e 
>   
> clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerNetworkClient.java
>  9517d9d0cd480d5ba1d12f1fde7963e60528d2f8 
>   clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 
> 804d569498396d431880641041fc9292076452cb 
>   clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 
> 06f00a99a73a288df9afa8c1d4abe3580fa968a6 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/BufferPool.java
>  4cb1e50d6c4ed55241aeaef1d3af09def5274103 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java
>  a152bd7697dca55609a9ec4cfe0a82c10595fbc3 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java
>  06182db1c3a5da85648199b4c0c98b80ea7c6c0c 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 
> d2e64f7cd8bf56e433a210905b2874f71eee9ea0 
>   clients/src/main/java/org/apache/kafka/common/network/Selectable.java 
> 70e74bd6aa629c430b2850ca40c97df0b16e5d75 
>   clients/src/main/java/org/apache/kafka/common/network/Selector.java 
> 4aa5cbb86ce6e1bf8f6769147ee2a6452c855c74 
>   clients/src/test/java/org/apache/kafka/clients/MockClient.java 
> e5815f56bdf8e2d980f2bc36b831ed234c0ac781 
>   clients/src/test/java/org/apache/kafka/clients/NetworkClientTest.java 
> 69c93c3adf674b1640534c3d7410fcaafaf2232c 
>   
> clients/src/test/java/org/apache/kafka/clients/producer/internals/BufferPoolTest.java
>  2c693824fa53db1e38766b8c66a0ef42ef9d0f3a 
>   
> clients/src/test/java/org/apache/kafka/clients/producer/internals/RecordAccumulatorTest.java
>  5b2e4ffaeab7127648db608c179703b27b577414 
>   
> clients/src/test/java/org/apache/kafka/clients/producer/internals/SenderTest.java
>  aa44991777a855f4b7f4f7bf17107c69393ff8ff 
>   clients/src/test/java/org/apache/kafka/common/network/SSLSelectorTest.java 
> df1205c935bee9a30a50816dbade64d6014b1ef2 
>   clients/src/test/java/org/apache/kafka/common/network/SelectorTest.java 
> 3a684d98b05cadfb25c6f7f9a038ef1f6697edbf 
>   clients/src/test/java/org/apache/kafka/test/MockSelector.java 
> f83fd9b794a3bd191121a22bcb40fd6ec31d83b5 
>   core/src/main/scala/kafka/controller/ControllerChannelManager.scala 
> da1cff07f7f76dcfa5a805718febcccd4ed5f578 
>   core/src/main/scala/kafka/server/KafkaConfig.scala 
> 1e8b2331486ffe55bfcc0919e48e12aad23b7d3c 
>   

Re: Review Request 36858: Patch for KAFKA-2120

[Jun Rao]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36858/#review98451
---

Ship it!


Thanks for the patch. Just a minor comment below.


clients/src/test/java/org/apache/kafka/clients/NetworkClientTest.java (lines 72 
- 74)


It seems that in this case, we should just do client.close(id) w/o doing 
another poll and test if the connection is no longer ready. Actually, this test 
is already covered in testClose(). So, we can probably just remove 
testReadyAndDisconnect().


- Jun Rao


On Sept. 10, 2015, 1:56 a.m., Mayuresh Gharat wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/36858/
> ---
> 
> (Updated Sept. 10, 2015, 1:56 a.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-2120
> https://issues.apache.org/jira/browse/KAFKA-2120
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> Solved compile error
> 
> 
> Addressed Jason's comments for Kip-19
> 
> 
> Addressed Jun's comments
> 
> 
> Addressed Jason's comments about the default values for requestTimeout
> 
> 
> checkpoint
> 
> 
> Addressed Joel's concerns. Also tried to include Jun's feedback.
> 
> 
> Fixed a minor comment
> 
> 
> Solved unittest issue
> 
> 
> Addressed Jun's comments regarding NetworkClient
> 
> 
> Addressed Jun's comments about disconnect() in Selector
> 
> 
> changed logging level to debug
> 
> 
> Addressed Joels comments to break out early from the loop while aborting 
> expired batches
> 
> 
> Diffs
> -
> 
>   clients/src/main/java/org/apache/kafka/clients/ClientRequest.java 
> dc8f0f115bcda893c95d17c0a57be8d14518d034 
>   clients/src/main/java/org/apache/kafka/clients/CommonClientConfigs.java 
> 7d24c6f5dd2b63b96584f3aa8922a1d048dc1ae4 
>   clients/src/main/java/org/apache/kafka/clients/InFlightRequests.java 
> 15d00d4e484bb5d51a9ae6857ed6e024a2cc1820 
>   clients/src/main/java/org/apache/kafka/clients/KafkaClient.java 
> f46c0d9b5eb73887c62a0e09c96e9d8c964c709d 
>   clients/src/main/java/org/apache/kafka/clients/NetworkClient.java 
> 049b22eadd5496b70dfcfd9d821f67c62c68a052 
>   clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerConfig.java 
> b9a2d4e2bc565f0ee72b27791afe5c894af262f1 
>   clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java 
> 19ef6ebead3dc97d4912985e35b6b8d17503cb0e 
>   
> clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerNetworkClient.java
>  9517d9d0cd480d5ba1d12f1fde7963e60528d2f8 
>   clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 
> 804d569498396d431880641041fc9292076452cb 
>   clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 
> 06f00a99a73a288df9afa8c1d4abe3580fa968a6 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/BufferPool.java
>  4cb1e50d6c4ed55241aeaef1d3af09def5274103 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java
>  a152bd7697dca55609a9ec4cfe0a82c10595fbc3 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java
>  06182db1c3a5da85648199b4c0c98b80ea7c6c0c 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 
> d2e64f7cd8bf56e433a210905b2874f71eee9ea0 
>   clients/src/main/java/org/apache/kafka/common/network/Selectable.java 
> 70e74bd6aa629c430b2850ca40c97df0b16e5d75 
>   clients/src/main/java/org/apache/kafka/common/network/Selector.java 
> 4aa5cbb86ce6e1bf8f6769147ee2a6452c855c74 
>   clients/src/test/java/org/apache/kafka/clients/MockClient.java 
> e5815f56bdf8e2d980f2bc36b831ed234c0ac781 
>   clients/src/test/java/org/apache/kafka/clients/NetworkClientTest.java 
> 69c93c3adf674b1640534c3d7410fcaafaf2232c 
>   
> clients/src/test/java/org/apache/kafka/clients/producer/internals/BufferPoolTest.java
>  2c693824fa53db1e38766b8c66a0ef42ef9d0f3a 
>   
> clients/src/test/java/org/apache/kafka/clients/producer/internals/RecordAccumulatorTest.java
>  5b2e4ffaeab7127648db608c179703b27b577414 
>   
> clients/src/test/java/org/apache/kafka/clients/producer/internals/SenderTest.java
>  aa44991777a855f4b7f4f7bf17107c69393ff8ff 
>   clients/src/test/java/org/apache/kafka/common/network/SSLSelectorTest.java 
> df1205c935bee9a30a50816dbade64d6014b1ef2 
>   clients/src/test/java/org/apache/kafka/common/network/SelectorTest.java 
> 3a684d98b05cadfb25c6f7f9a038ef1f6697edbf 
>   clients/src/test/java/org/apache/kafka/test/MockSelector.java 
> f83fd9b794a3bd191121a22bcb40fd6ec31d83b5 
>   core/src/main/scala/kafka/controller/ControllerChannelManager.scala 
> da1cff07f7f76dcfa5a805718febcccd4ed5f578 
>   core/src/main/scala/kafka/server/KafkaConfig.scala 
> 

Re: Review Request 36858: Patch for KAFKA-2120

[Mayuresh Gharat]

> On Sept. 10, 2015, 6:19 p.m., Ismael Juma wrote:
> > clients/src/main/java/org/apache/kafka/clients/InFlightRequests.java, line 
> > 137
> > 
> >
> > Out of curiosity, why are we using `LinkedList` here? Generally 
> > `ArrayList` is a better default, but maybe there's a good reason to deviate 
> > in this case.

LinkedList add method gives O(1) performance while ArrayList gives O(n) in 
worst case. Since we are just adding and not doing random accesses LinkedList 
would be better in this case.


- Mayuresh


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36858/#review98438
---


On Sept. 11, 2015, 4:39 a.m., Mayuresh Gharat wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/36858/
> ---
> 
> (Updated Sept. 11, 2015, 4:39 a.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-2120
> https://issues.apache.org/jira/browse/KAFKA-2120
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> Solved compile error
> 
> 
> Addressed Jason's comments for Kip-19
> 
> 
> Addressed Jun's comments
> 
> 
> Addressed Jason's comments about the default values for requestTimeout
> 
> 
> checkpoint
> 
> 
> Addressed Joel's concerns. Also tried to include Jun's feedback.
> 
> 
> Fixed a minor comment
> 
> 
> Solved unittest issue
> 
> 
> Addressed Jun's comments regarding NetworkClient
> 
> 
> Addressed Jun's comments about disconnect() in Selector
> 
> 
> changed logging level to debug
> 
> 
> Addressed Joels comments to break out early from the loop while aborting 
> expired batches
> 
> 
> Addressed Jun's comments
> 
> 
> Diffs
> -
> 
>   clients/src/main/java/org/apache/kafka/clients/ClientRequest.java 
> dc8f0f115bcda893c95d17c0a57be8d14518d034 
>   clients/src/main/java/org/apache/kafka/clients/CommonClientConfigs.java 
> 7d24c6f5dd2b63b96584f3aa8922a1d048dc1ae4 
>   clients/src/main/java/org/apache/kafka/clients/InFlightRequests.java 
> 15d00d4e484bb5d51a9ae6857ed6e024a2cc1820 
>   clients/src/main/java/org/apache/kafka/clients/KafkaClient.java 
> f46c0d9b5eb73887c62a0e09c96e9d8c964c709d 
>   clients/src/main/java/org/apache/kafka/clients/NetworkClient.java 
> 049b22eadd5496b70dfcfd9d821f67c62c68a052 
>   clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerConfig.java 
> b9a2d4e2bc565f0ee72b27791afe5c894af262f1 
>   clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java 
> 19ef6ebead3dc97d4912985e35b6b8d17503cb0e 
>   
> clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerNetworkClient.java
>  9517d9d0cd480d5ba1d12f1fde7963e60528d2f8 
>   clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 
> 804d569498396d431880641041fc9292076452cb 
>   clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 
> 06f00a99a73a288df9afa8c1d4abe3580fa968a6 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/BufferPool.java
>  4cb1e50d6c4ed55241aeaef1d3af09def5274103 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java
>  a152bd7697dca55609a9ec4cfe0a82c10595fbc3 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java
>  06182db1c3a5da85648199b4c0c98b80ea7c6c0c 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 
> d2e64f7cd8bf56e433a210905b2874f71eee9ea0 
>   clients/src/main/java/org/apache/kafka/common/network/Selectable.java 
> 70e74bd6aa629c430b2850ca40c97df0b16e5d75 
>   clients/src/main/java/org/apache/kafka/common/network/Selector.java 
> 4aa5cbb86ce6e1bf8f6769147ee2a6452c855c74 
>   clients/src/test/java/org/apache/kafka/clients/MockClient.java 
> e5815f56bdf8e2d980f2bc36b831ed234c0ac781 
>   clients/src/test/java/org/apache/kafka/clients/NetworkClientTest.java 
> 69c93c3adf674b1640534c3d7410fcaafaf2232c 
>   
> clients/src/test/java/org/apache/kafka/clients/producer/internals/BufferPoolTest.java
>  2c693824fa53db1e38766b8c66a0ef42ef9d0f3a 
>   
> clients/src/test/java/org/apache/kafka/clients/producer/internals/RecordAccumulatorTest.java
>  5b2e4ffaeab7127648db608c179703b27b577414 
>   
> clients/src/test/java/org/apache/kafka/clients/producer/internals/SenderTest.java
>  aa44991777a855f4b7f4f7bf17107c69393ff8ff 
>   clients/src/test/java/org/apache/kafka/common/network/SSLSelectorTest.java 
> df1205c935bee9a30a50816dbade64d6014b1ef2 
>   clients/src/test/java/org/apache/kafka/common/network/SelectorTest.java 
> 3a684d98b05cadfb25c6f7f9a038ef1f6697edbf 
>   clients/src/test/java/org/apache/kafka/test/MockSelector.java 
> f83fd9b794a3bd191121a22bcb40fd6ec31d83b5 
>   

Re: Review Request 36858: Patch for KAFKA-2120

[Mayuresh Gharat]

> On Sept. 11, 2015, 1:44 a.m., Jason Gustafson wrote:
> > clients/src/main/java/org/apache/kafka/clients/InFlightRequests.java, line 
> > 140
> > 
> >
> > This line is puzzling me a little bit. Wouldn't we want to use the 
> > oldest request for this check. If we use the last sent request, then we 
> > might miss older requests that had already timed out, which would tend to 
> > delay timeout detection. Maybe I'm missing something?

The inflight request are sent one after another. The reason we use lastSent() 
is because if we know that if the last Sent is timed out, it means that all 
older requests should have been timed out.


- Mayuresh


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36858/#review98526
---


On Sept. 10, 2015, 1:56 a.m., Mayuresh Gharat wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/36858/
> ---
> 
> (Updated Sept. 10, 2015, 1:56 a.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-2120
> https://issues.apache.org/jira/browse/KAFKA-2120
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> Solved compile error
> 
> 
> Addressed Jason's comments for Kip-19
> 
> 
> Addressed Jun's comments
> 
> 
> Addressed Jason's comments about the default values for requestTimeout
> 
> 
> checkpoint
> 
> 
> Addressed Joel's concerns. Also tried to include Jun's feedback.
> 
> 
> Fixed a minor comment
> 
> 
> Solved unittest issue
> 
> 
> Addressed Jun's comments regarding NetworkClient
> 
> 
> Addressed Jun's comments about disconnect() in Selector
> 
> 
> changed logging level to debug
> 
> 
> Addressed Joels comments to break out early from the loop while aborting 
> expired batches
> 
> 
> Diffs
> -
> 
>   clients/src/main/java/org/apache/kafka/clients/ClientRequest.java 
> dc8f0f115bcda893c95d17c0a57be8d14518d034 
>   clients/src/main/java/org/apache/kafka/clients/CommonClientConfigs.java 
> 7d24c6f5dd2b63b96584f3aa8922a1d048dc1ae4 
>   clients/src/main/java/org/apache/kafka/clients/InFlightRequests.java 
> 15d00d4e484bb5d51a9ae6857ed6e024a2cc1820 
>   clients/src/main/java/org/apache/kafka/clients/KafkaClient.java 
> f46c0d9b5eb73887c62a0e09c96e9d8c964c709d 
>   clients/src/main/java/org/apache/kafka/clients/NetworkClient.java 
> 049b22eadd5496b70dfcfd9d821f67c62c68a052 
>   clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerConfig.java 
> b9a2d4e2bc565f0ee72b27791afe5c894af262f1 
>   clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java 
> 19ef6ebead3dc97d4912985e35b6b8d17503cb0e 
>   
> clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerNetworkClient.java
>  9517d9d0cd480d5ba1d12f1fde7963e60528d2f8 
>   clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 
> 804d569498396d431880641041fc9292076452cb 
>   clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 
> 06f00a99a73a288df9afa8c1d4abe3580fa968a6 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/BufferPool.java
>  4cb1e50d6c4ed55241aeaef1d3af09def5274103 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java
>  a152bd7697dca55609a9ec4cfe0a82c10595fbc3 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java
>  06182db1c3a5da85648199b4c0c98b80ea7c6c0c 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 
> d2e64f7cd8bf56e433a210905b2874f71eee9ea0 
>   clients/src/main/java/org/apache/kafka/common/network/Selectable.java 
> 70e74bd6aa629c430b2850ca40c97df0b16e5d75 
>   clients/src/main/java/org/apache/kafka/common/network/Selector.java 
> 4aa5cbb86ce6e1bf8f6769147ee2a6452c855c74 
>   clients/src/test/java/org/apache/kafka/clients/MockClient.java 
> e5815f56bdf8e2d980f2bc36b831ed234c0ac781 
>   clients/src/test/java/org/apache/kafka/clients/NetworkClientTest.java 
> 69c93c3adf674b1640534c3d7410fcaafaf2232c 
>   
> clients/src/test/java/org/apache/kafka/clients/producer/internals/BufferPoolTest.java
>  2c693824fa53db1e38766b8c66a0ef42ef9d0f3a 
>   
> clients/src/test/java/org/apache/kafka/clients/producer/internals/RecordAccumulatorTest.java
>  5b2e4ffaeab7127648db608c179703b27b577414 
>   
> clients/src/test/java/org/apache/kafka/clients/producer/internals/SenderTest.java
>  aa44991777a855f4b7f4f7bf17107c69393ff8ff 
>   clients/src/test/java/org/apache/kafka/common/network/SSLSelectorTest.java 
> df1205c935bee9a30a50816dbade64d6014b1ef2 
>   clients/src/test/java/org/apache/kafka/common/network/SelectorTest.java 
> 3a684d98b05cadfb25c6f7f9a038ef1f6697edbf 
>   

Re: Review Request 36858: Patch for KAFKA-2120

[Mayuresh Gharat]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36858/
---

(Updated Sept. 11, 2015, 4:39 a.m.)


Review request for kafka.


Bugs: KAFKA-2120
https://issues.apache.org/jira/browse/KAFKA-2120


Repository: kafka


Description (updated)
---

Solved compile error


Addressed Jason's comments for Kip-19


Addressed Jun's comments


Addressed Jason's comments about the default values for requestTimeout


checkpoint


Addressed Joel's concerns. Also tried to include Jun's feedback.


Fixed a minor comment


Solved unittest issue


Addressed Jun's comments regarding NetworkClient


Addressed Jun's comments about disconnect() in Selector


changed logging level to debug


Addressed Joels comments to break out early from the loop while aborting 
expired batches


Addressed Jun's comments


Diffs (updated)
-

  clients/src/main/java/org/apache/kafka/clients/ClientRequest.java 
dc8f0f115bcda893c95d17c0a57be8d14518d034 
  clients/src/main/java/org/apache/kafka/clients/CommonClientConfigs.java 
7d24c6f5dd2b63b96584f3aa8922a1d048dc1ae4 
  clients/src/main/java/org/apache/kafka/clients/InFlightRequests.java 
15d00d4e484bb5d51a9ae6857ed6e024a2cc1820 
  clients/src/main/java/org/apache/kafka/clients/KafkaClient.java 
f46c0d9b5eb73887c62a0e09c96e9d8c964c709d 
  clients/src/main/java/org/apache/kafka/clients/NetworkClient.java 
049b22eadd5496b70dfcfd9d821f67c62c68a052 
  clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerConfig.java 
b9a2d4e2bc565f0ee72b27791afe5c894af262f1 
  clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java 
19ef6ebead3dc97d4912985e35b6b8d17503cb0e 
  
clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerNetworkClient.java
 9517d9d0cd480d5ba1d12f1fde7963e60528d2f8 
  clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 
804d569498396d431880641041fc9292076452cb 
  clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 
06f00a99a73a288df9afa8c1d4abe3580fa968a6 
  
clients/src/main/java/org/apache/kafka/clients/producer/internals/BufferPool.java
 4cb1e50d6c4ed55241aeaef1d3af09def5274103 
  
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java
 a152bd7697dca55609a9ec4cfe0a82c10595fbc3 
  
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java
 06182db1c3a5da85648199b4c0c98b80ea7c6c0c 
  clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 
d2e64f7cd8bf56e433a210905b2874f71eee9ea0 
  clients/src/main/java/org/apache/kafka/common/network/Selectable.java 
70e74bd6aa629c430b2850ca40c97df0b16e5d75 
  clients/src/main/java/org/apache/kafka/common/network/Selector.java 
4aa5cbb86ce6e1bf8f6769147ee2a6452c855c74 
  clients/src/test/java/org/apache/kafka/clients/MockClient.java 
e5815f56bdf8e2d980f2bc36b831ed234c0ac781 
  clients/src/test/java/org/apache/kafka/clients/NetworkClientTest.java 
69c93c3adf674b1640534c3d7410fcaafaf2232c 
  
clients/src/test/java/org/apache/kafka/clients/producer/internals/BufferPoolTest.java
 2c693824fa53db1e38766b8c66a0ef42ef9d0f3a 
  
clients/src/test/java/org/apache/kafka/clients/producer/internals/RecordAccumulatorTest.java
 5b2e4ffaeab7127648db608c179703b27b577414 
  
clients/src/test/java/org/apache/kafka/clients/producer/internals/SenderTest.java
 aa44991777a855f4b7f4f7bf17107c69393ff8ff 
  clients/src/test/java/org/apache/kafka/common/network/SSLSelectorTest.java 
df1205c935bee9a30a50816dbade64d6014b1ef2 
  clients/src/test/java/org/apache/kafka/common/network/SelectorTest.java 
3a684d98b05cadfb25c6f7f9a038ef1f6697edbf 
  clients/src/test/java/org/apache/kafka/test/MockSelector.java 
f83fd9b794a3bd191121a22bcb40fd6ec31d83b5 
  core/src/main/scala/kafka/controller/ControllerChannelManager.scala 
da1cff07f7f76dcfa5a805718febcccd4ed5f578 
  core/src/main/scala/kafka/server/KafkaConfig.scala 
1e8b2331486ffe55bfcc0919e48e12aad23b7d3c 
  core/src/main/scala/kafka/server/KafkaServer.scala 
30406ce809caaac56aca1f30c235b35962d55a50 
  core/src/main/scala/kafka/tools/ProducerPerformance.scala 
46a68e97b8bcc8821f21e4220ce9b3acedc5dafe 
  core/src/main/scala/kafka/utils/NetworkClientBlockingOps.scala 
ad10721de844725f27a116611209992cea61b088 
  core/src/test/scala/integration/kafka/api/ProducerFailureHandlingTest.scala 
1198df02ddd7727269e84a751ba99520f6d5584a 
  core/src/test/scala/unit/kafka/server/KafkaConfigTest.scala 
5b4f2db4607ae6d17696c1140f1a771ce75c80e0 
  core/src/test/scala/unit/kafka/utils/TestUtils.scala 
09b8444c2add87f0f70dbb182e892977a6b5c243 

Diff: https://reviews.apache.org/r/36858/diff/


Testing
---


Thanks,

Mayuresh Gharat

Re: Review Request 36858: Patch for KAFKA-2120

[Ismael Juma]

> On July 27, 2015, 10:55 p.m., Jason Gustafson wrote:
> > clients/src/main/java/org/apache/kafka/clients/InFlightRequests.java, line 
> > 48
> > 
> >
> > It seems like it might be more natural to set this in 
> > NetworkClient.send.
> 
> Mayuresh Gharat wrote:
> That was my original thinking, but the problem is that if we set this in 
> NetworkClient, we will have to set it in send() call and 
> maybeUpdateMetadata() as well. Basically it looks a little loose since if 
> there are new changes we might miss setting this. To avoid that we wanted to 
> set this in a single place.
> 
> Jason Gustafson wrote:
> Hmm.. It still sort of feels like it ought to be the responsibility of 
> NetworkClient to set that and not InFlightRequests. An easy way to address 
> the issue would be to create a NetworkClient.doSend method. For example:
> 
> 
> ```
> void doSend(ClientRequest req) {
>req.setSendMs(time.milliseconds());
>this.selector.send(req.request());
>this.inFlightRequests.add(metadataRequest);
> }
> ```
> 
> Then NetworkClient.send and NetworkClient.maybeUpdateMetdata could invoke 
> this method. Better or worse?

In fact, there is already a `doSend` method, which is called from `send` and 
`maybeUpdate` (I added that somewhat recently).


- Ismael


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36858/#review93189
---


On Sept. 10, 2015, 1:56 a.m., Mayuresh Gharat wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/36858/
> ---
> 
> (Updated Sept. 10, 2015, 1:56 a.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-2120
> https://issues.apache.org/jira/browse/KAFKA-2120
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> Solved compile error
> 
> 
> Addressed Jason's comments for Kip-19
> 
> 
> Addressed Jun's comments
> 
> 
> Addressed Jason's comments about the default values for requestTimeout
> 
> 
> checkpoint
> 
> 
> Addressed Joel's concerns. Also tried to include Jun's feedback.
> 
> 
> Fixed a minor comment
> 
> 
> Solved unittest issue
> 
> 
> Addressed Jun's comments regarding NetworkClient
> 
> 
> Addressed Jun's comments about disconnect() in Selector
> 
> 
> changed logging level to debug
> 
> 
> Addressed Joels comments to break out early from the loop while aborting 
> expired batches
> 
> 
> Diffs
> -
> 
>   clients/src/main/java/org/apache/kafka/clients/ClientRequest.java 
> dc8f0f115bcda893c95d17c0a57be8d14518d034 
>   clients/src/main/java/org/apache/kafka/clients/CommonClientConfigs.java 
> 7d24c6f5dd2b63b96584f3aa8922a1d048dc1ae4 
>   clients/src/main/java/org/apache/kafka/clients/InFlightRequests.java 
> 15d00d4e484bb5d51a9ae6857ed6e024a2cc1820 
>   clients/src/main/java/org/apache/kafka/clients/KafkaClient.java 
> f46c0d9b5eb73887c62a0e09c96e9d8c964c709d 
>   clients/src/main/java/org/apache/kafka/clients/NetworkClient.java 
> 049b22eadd5496b70dfcfd9d821f67c62c68a052 
>   clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerConfig.java 
> b9a2d4e2bc565f0ee72b27791afe5c894af262f1 
>   clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java 
> 19ef6ebead3dc97d4912985e35b6b8d17503cb0e 
>   
> clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerNetworkClient.java
>  9517d9d0cd480d5ba1d12f1fde7963e60528d2f8 
>   clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 
> 804d569498396d431880641041fc9292076452cb 
>   clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 
> 06f00a99a73a288df9afa8c1d4abe3580fa968a6 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/BufferPool.java
>  4cb1e50d6c4ed55241aeaef1d3af09def5274103 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java
>  a152bd7697dca55609a9ec4cfe0a82c10595fbc3 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java
>  06182db1c3a5da85648199b4c0c98b80ea7c6c0c 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 
> d2e64f7cd8bf56e433a210905b2874f71eee9ea0 
>   clients/src/main/java/org/apache/kafka/common/network/Selectable.java 
> 70e74bd6aa629c430b2850ca40c97df0b16e5d75 
>   clients/src/main/java/org/apache/kafka/common/network/Selector.java 
> 4aa5cbb86ce6e1bf8f6769147ee2a6452c855c74 
>   clients/src/test/java/org/apache/kafka/clients/MockClient.java 
> e5815f56bdf8e2d980f2bc36b831ed234c0ac781 
>   clients/src/test/java/org/apache/kafka/clients/NetworkClientTest.java 
> 69c93c3adf674b1640534c3d7410fcaafaf2232c 
>   
> 

Re: Review Request 36858: Patch for KAFKA-2120

[Mayuresh Gharat]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36858/
---

(Updated Sept. 9, 2015, 11:49 p.m.)


Review request for kafka.


Bugs: KAFKA-2120
https://issues.apache.org/jira/browse/KAFKA-2120


Repository: kafka


Description (updated)
---

Solved compile error


Addressed Jason's comments for Kip-19


Addressed Jun's comments


Addressed Jason's comments about the default values for requestTimeout


checkpoint


Addressed Joel's concerns. Also tried to include Jun's feedback.


Fixed a minor comment


Solved unittest issue


Addressed Jun's comments regarding NetworkClient


Addressed Jun's comments about disconnect() in Selector


Diffs
-

  clients/src/main/java/org/apache/kafka/clients/ClientRequest.java 
dc8f0f115bcda893c95d17c0a57be8d14518d034 
  clients/src/main/java/org/apache/kafka/clients/CommonClientConfigs.java 
7d24c6f5dd2b63b96584f3aa8922a1d048dc1ae4 
  clients/src/main/java/org/apache/kafka/clients/InFlightRequests.java 
15d00d4e484bb5d51a9ae6857ed6e024a2cc1820 
  clients/src/main/java/org/apache/kafka/clients/KafkaClient.java 
f46c0d9b5eb73887c62a0e09c96e9d8c964c709d 
  clients/src/main/java/org/apache/kafka/clients/NetworkClient.java 
049b22eadd5496b70dfcfd9d821f67c62c68a052 
  clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerConfig.java 
b9a2d4e2bc565f0ee72b27791afe5c894af262f1 
  clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java 
19ef6ebead3dc97d4912985e35b6b8d17503cb0e 
  
clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerNetworkClient.java
 9517d9d0cd480d5ba1d12f1fde7963e60528d2f8 
  clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 
804d569498396d431880641041fc9292076452cb 
  clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 
06f00a99a73a288df9afa8c1d4abe3580fa968a6 
  
clients/src/main/java/org/apache/kafka/clients/producer/internals/BufferPool.java
 4cb1e50d6c4ed55241aeaef1d3af09def5274103 
  
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java
 a152bd7697dca55609a9ec4cfe0a82c10595fbc3 
  
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java
 06182db1c3a5da85648199b4c0c98b80ea7c6c0c 
  clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 
d2e64f7cd8bf56e433a210905b2874f71eee9ea0 
  clients/src/main/java/org/apache/kafka/common/network/Selectable.java 
70e74bd6aa629c430b2850ca40c97df0b16e5d75 
  clients/src/main/java/org/apache/kafka/common/network/Selector.java 
4aa5cbb86ce6e1bf8f6769147ee2a6452c855c74 
  clients/src/test/java/org/apache/kafka/clients/MockClient.java 
e5815f56bdf8e2d980f2bc36b831ed234c0ac781 
  clients/src/test/java/org/apache/kafka/clients/NetworkClientTest.java 
69c93c3adf674b1640534c3d7410fcaafaf2232c 
  
clients/src/test/java/org/apache/kafka/clients/producer/internals/BufferPoolTest.java
 2c693824fa53db1e38766b8c66a0ef42ef9d0f3a 
  
clients/src/test/java/org/apache/kafka/clients/producer/internals/RecordAccumulatorTest.java
 5b2e4ffaeab7127648db608c179703b27b577414 
  
clients/src/test/java/org/apache/kafka/clients/producer/internals/SenderTest.java
 aa44991777a855f4b7f4f7bf17107c69393ff8ff 
  clients/src/test/java/org/apache/kafka/common/network/SSLSelectorTest.java 
df1205c935bee9a30a50816dbade64d6014b1ef2 
  clients/src/test/java/org/apache/kafka/common/network/SelectorTest.java 
3a684d98b05cadfb25c6f7f9a038ef1f6697edbf 
  clients/src/test/java/org/apache/kafka/test/MockSelector.java 
f83fd9b794a3bd191121a22bcb40fd6ec31d83b5 
  core/src/main/scala/kafka/controller/ControllerChannelManager.scala 
da1cff07f7f76dcfa5a805718febcccd4ed5f578 
  core/src/main/scala/kafka/server/KafkaConfig.scala 
1e8b2331486ffe55bfcc0919e48e12aad23b7d3c 
  core/src/main/scala/kafka/server/KafkaServer.scala 
30406ce809caaac56aca1f30c235b35962d55a50 
  core/src/main/scala/kafka/tools/ProducerPerformance.scala 
46a68e97b8bcc8821f21e4220ce9b3acedc5dafe 
  core/src/main/scala/kafka/utils/NetworkClientBlockingOps.scala 
ad10721de844725f27a116611209992cea61b088 
  core/src/test/scala/integration/kafka/api/ProducerFailureHandlingTest.scala 
1198df02ddd7727269e84a751ba99520f6d5584a 
  core/src/test/scala/unit/kafka/server/KafkaConfigTest.scala 
5b4f2db4607ae6d17696c1140f1a771ce75c80e0 
  core/src/test/scala/unit/kafka/utils/TestUtils.scala 
09b8444c2add87f0f70dbb182e892977a6b5c243 

Diff: https://reviews.apache.org/r/36858/diff/


Testing
---


Thanks,

Mayuresh Gharat

Re: Review Request 36858: Patch for KAFKA-2120

[Ismael Juma]

> On Sept. 9, 2015, 4:43 a.m., Jun Rao wrote:
> > clients/src/main/java/org/apache/kafka/common/network/Selector.java, lines 
> > 185-190
> > 
> >
> > Do we still need disconnect(id)? It seems that we can just replace the 
> > usage in test with close(id) and remove disconnect from selectable and 
> > KafkaChannel? We have to implement close(id) in MockSelector().
> 
> Mayuresh Gharat wrote:
> Hi Jun,
> Thats a very good suggestion. I had a small orthogonal concern here. If 
> we want people to start using NetworkClient instead of BLockingChannel, for 
> example for writing a tool for fetching and commiting offsets, it would be 
> great if we can provide a disconnect() in NetworkClient that can either call 
> close(id) in Selector or disconnect in Selector() (In this case we need to 
> change how disconnect() in Selector functions). I can add disconnect API in 
> NetworkClient as a separate patch if you feel my concern is valid.

NetworkClient has a `close(id)` method which I have used in a number of places 
where I replaced `BlockingChannel` by `NetworkClient`. What would the proposed 
`disconnect` method do differently?


- Ismael


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36858/#review98136
---


On Sept. 9, 2015, 11:49 p.m., Mayuresh Gharat wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/36858/
> ---
> 
> (Updated Sept. 9, 2015, 11:49 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-2120
> https://issues.apache.org/jira/browse/KAFKA-2120
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> Solved compile error
> 
> 
> Addressed Jason's comments for Kip-19
> 
> 
> Addressed Jun's comments
> 
> 
> Addressed Jason's comments about the default values for requestTimeout
> 
> 
> checkpoint
> 
> 
> Addressed Joel's concerns. Also tried to include Jun's feedback.
> 
> 
> Fixed a minor comment
> 
> 
> Solved unittest issue
> 
> 
> Addressed Jun's comments regarding NetworkClient
> 
> 
> Addressed Jun's comments about disconnect() in Selector
> 
> 
> Diffs
> -
> 
>   clients/src/main/java/org/apache/kafka/clients/ClientRequest.java 
> dc8f0f115bcda893c95d17c0a57be8d14518d034 
>   clients/src/main/java/org/apache/kafka/clients/CommonClientConfigs.java 
> 7d24c6f5dd2b63b96584f3aa8922a1d048dc1ae4 
>   clients/src/main/java/org/apache/kafka/clients/InFlightRequests.java 
> 15d00d4e484bb5d51a9ae6857ed6e024a2cc1820 
>   clients/src/main/java/org/apache/kafka/clients/KafkaClient.java 
> f46c0d9b5eb73887c62a0e09c96e9d8c964c709d 
>   clients/src/main/java/org/apache/kafka/clients/NetworkClient.java 
> 049b22eadd5496b70dfcfd9d821f67c62c68a052 
>   clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerConfig.java 
> b9a2d4e2bc565f0ee72b27791afe5c894af262f1 
>   clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java 
> 19ef6ebead3dc97d4912985e35b6b8d17503cb0e 
>   
> clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerNetworkClient.java
>  9517d9d0cd480d5ba1d12f1fde7963e60528d2f8 
>   clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 
> 804d569498396d431880641041fc9292076452cb 
>   clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 
> 06f00a99a73a288df9afa8c1d4abe3580fa968a6 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/BufferPool.java
>  4cb1e50d6c4ed55241aeaef1d3af09def5274103 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java
>  a152bd7697dca55609a9ec4cfe0a82c10595fbc3 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java
>  06182db1c3a5da85648199b4c0c98b80ea7c6c0c 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 
> d2e64f7cd8bf56e433a210905b2874f71eee9ea0 
>   clients/src/main/java/org/apache/kafka/common/network/Selectable.java 
> 70e74bd6aa629c430b2850ca40c97df0b16e5d75 
>   clients/src/main/java/org/apache/kafka/common/network/Selector.java 
> 4aa5cbb86ce6e1bf8f6769147ee2a6452c855c74 
>   clients/src/test/java/org/apache/kafka/clients/MockClient.java 
> e5815f56bdf8e2d980f2bc36b831ed234c0ac781 
>   clients/src/test/java/org/apache/kafka/clients/NetworkClientTest.java 
> 69c93c3adf674b1640534c3d7410fcaafaf2232c 
>   
> clients/src/test/java/org/apache/kafka/clients/producer/internals/BufferPoolTest.java
>  2c693824fa53db1e38766b8c66a0ef42ef9d0f3a 
>   
> clients/src/test/java/org/apache/kafka/clients/producer/internals/RecordAccumulatorTest.java
>  5b2e4ffaeab7127648db608c179703b27b577414 
>   
> 

Re: Review Request 36858: Patch for KAFKA-2120

[Mayuresh Gharat]

> On Sept. 9, 2015, 4:43 a.m., Jun Rao wrote:
> > clients/src/main/java/org/apache/kafka/common/network/Selector.java, lines 
> > 185-190
> > 
> >
> > Do we still need disconnect(id)? It seems that we can just replace the 
> > usage in test with close(id) and remove disconnect from selectable and 
> > KafkaChannel? We have to implement close(id) in MockSelector().

Hi Jun,
Thats a very good suggestion. I had a small orthogonal concern here. If we want 
people to start using NetworkClient instead of BLockingChannel, for example for 
writing a tool for fetching and commiting offsets, it would be great if we can 
provide a disconnect() in NetworkClient that can either call close(id) in 
Selector or disconnect in Selector() (In this case we need to change how 
disconnect() in Selector functions). I can add disconnect API in NetworkClient 
as a separate patch if you feel my concern is valid.


- Mayuresh


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36858/#review98136
---


On Sept. 9, 2015, 11:46 p.m., Mayuresh Gharat wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/36858/
> ---
> 
> (Updated Sept. 9, 2015, 11:46 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-2120
> https://issues.apache.org/jira/browse/KAFKA-2120
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> Solved compile error
> 
> 
> Addressed Jason's comments for Kip-19
> 
> 
> Addressed Jun's comments
> 
> 
> Addressed Jason's comments about the default values for requestTimeout
> 
> 
> checkpoint
> 
> 
> Addressed Joel's concerns. Also tried to include Jun's feedback.
> 
> 
> Fixed a minor comment
> 
> 
> Solved unittest issue
> 
> 
> Addressed Jun's comments regarding NetworkClient
> 
> 
> Addressed Jun's comments about disconnect() in Selector
> 
> 
> changed logging level to debug
> 
> 
> Diffs
> -
> 
>   clients/src/main/java/org/apache/kafka/clients/ClientRequest.java 
> dc8f0f115bcda893c95d17c0a57be8d14518d034 
>   clients/src/main/java/org/apache/kafka/clients/CommonClientConfigs.java 
> 7d24c6f5dd2b63b96584f3aa8922a1d048dc1ae4 
>   clients/src/main/java/org/apache/kafka/clients/InFlightRequests.java 
> 15d00d4e484bb5d51a9ae6857ed6e024a2cc1820 
>   clients/src/main/java/org/apache/kafka/clients/KafkaClient.java 
> f46c0d9b5eb73887c62a0e09c96e9d8c964c709d 
>   clients/src/main/java/org/apache/kafka/clients/NetworkClient.java 
> 049b22eadd5496b70dfcfd9d821f67c62c68a052 
>   clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerConfig.java 
> b9a2d4e2bc565f0ee72b27791afe5c894af262f1 
>   clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java 
> 19ef6ebead3dc97d4912985e35b6b8d17503cb0e 
>   
> clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerNetworkClient.java
>  9517d9d0cd480d5ba1d12f1fde7963e60528d2f8 
>   clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 
> 804d569498396d431880641041fc9292076452cb 
>   clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 
> 06f00a99a73a288df9afa8c1d4abe3580fa968a6 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/BufferPool.java
>  4cb1e50d6c4ed55241aeaef1d3af09def5274103 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java
>  a152bd7697dca55609a9ec4cfe0a82c10595fbc3 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java
>  06182db1c3a5da85648199b4c0c98b80ea7c6c0c 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 
> d2e64f7cd8bf56e433a210905b2874f71eee9ea0 
>   clients/src/main/java/org/apache/kafka/common/network/Selectable.java 
> 70e74bd6aa629c430b2850ca40c97df0b16e5d75 
>   clients/src/main/java/org/apache/kafka/common/network/Selector.java 
> 4aa5cbb86ce6e1bf8f6769147ee2a6452c855c74 
>   clients/src/test/java/org/apache/kafka/clients/MockClient.java 
> e5815f56bdf8e2d980f2bc36b831ed234c0ac781 
>   clients/src/test/java/org/apache/kafka/clients/NetworkClientTest.java 
> 69c93c3adf674b1640534c3d7410fcaafaf2232c 
>   
> clients/src/test/java/org/apache/kafka/clients/producer/internals/BufferPoolTest.java
>  2c693824fa53db1e38766b8c66a0ef42ef9d0f3a 
>   
> clients/src/test/java/org/apache/kafka/clients/producer/internals/RecordAccumulatorTest.java
>  5b2e4ffaeab7127648db608c179703b27b577414 
>   
> clients/src/test/java/org/apache/kafka/clients/producer/internals/SenderTest.java
>  aa44991777a855f4b7f4f7bf17107c69393ff8ff 
>   clients/src/test/java/org/apache/kafka/common/network/SSLSelectorTest.java 
> 

Re: Review Request 36858: Patch for KAFKA-2120

[Mayuresh Gharat]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36858/
---

(Updated Sept. 9, 2015, 11:46 p.m.)


Review request for kafka.


Bugs: KAFKA-2120
https://issues.apache.org/jira/browse/KAFKA-2120


Repository: kafka


Description (updated)
---

Solved compile error


Addressed Jason's comments for Kip-19


Addressed Jun's comments


Addressed Jason's comments about the default values for requestTimeout


checkpoint


Addressed Joel's concerns. Also tried to include Jun's feedback.


Fixed a minor comment


Solved unittest issue


Addressed Jun's comments regarding NetworkClient


Addressed Jun's comments about disconnect() in Selector


changed logging level to debug


Diffs (updated)
-

  clients/src/main/java/org/apache/kafka/clients/ClientRequest.java 
dc8f0f115bcda893c95d17c0a57be8d14518d034 
  clients/src/main/java/org/apache/kafka/clients/CommonClientConfigs.java 
7d24c6f5dd2b63b96584f3aa8922a1d048dc1ae4 
  clients/src/main/java/org/apache/kafka/clients/InFlightRequests.java 
15d00d4e484bb5d51a9ae6857ed6e024a2cc1820 
  clients/src/main/java/org/apache/kafka/clients/KafkaClient.java 
f46c0d9b5eb73887c62a0e09c96e9d8c964c709d 
  clients/src/main/java/org/apache/kafka/clients/NetworkClient.java 
049b22eadd5496b70dfcfd9d821f67c62c68a052 
  clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerConfig.java 
b9a2d4e2bc565f0ee72b27791afe5c894af262f1 
  clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java 
19ef6ebead3dc97d4912985e35b6b8d17503cb0e 
  
clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerNetworkClient.java
 9517d9d0cd480d5ba1d12f1fde7963e60528d2f8 
  clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 
804d569498396d431880641041fc9292076452cb 
  clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 
06f00a99a73a288df9afa8c1d4abe3580fa968a6 
  
clients/src/main/java/org/apache/kafka/clients/producer/internals/BufferPool.java
 4cb1e50d6c4ed55241aeaef1d3af09def5274103 
  
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java
 a152bd7697dca55609a9ec4cfe0a82c10595fbc3 
  
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java
 06182db1c3a5da85648199b4c0c98b80ea7c6c0c 
  clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 
d2e64f7cd8bf56e433a210905b2874f71eee9ea0 
  clients/src/main/java/org/apache/kafka/common/network/Selectable.java 
70e74bd6aa629c430b2850ca40c97df0b16e5d75 
  clients/src/main/java/org/apache/kafka/common/network/Selector.java 
4aa5cbb86ce6e1bf8f6769147ee2a6452c855c74 
  clients/src/test/java/org/apache/kafka/clients/MockClient.java 
e5815f56bdf8e2d980f2bc36b831ed234c0ac781 
  clients/src/test/java/org/apache/kafka/clients/NetworkClientTest.java 
69c93c3adf674b1640534c3d7410fcaafaf2232c 
  
clients/src/test/java/org/apache/kafka/clients/producer/internals/BufferPoolTest.java
 2c693824fa53db1e38766b8c66a0ef42ef9d0f3a 
  
clients/src/test/java/org/apache/kafka/clients/producer/internals/RecordAccumulatorTest.java
 5b2e4ffaeab7127648db608c179703b27b577414 
  
clients/src/test/java/org/apache/kafka/clients/producer/internals/SenderTest.java
 aa44991777a855f4b7f4f7bf17107c69393ff8ff 
  clients/src/test/java/org/apache/kafka/common/network/SSLSelectorTest.java 
df1205c935bee9a30a50816dbade64d6014b1ef2 
  clients/src/test/java/org/apache/kafka/common/network/SelectorTest.java 
3a684d98b05cadfb25c6f7f9a038ef1f6697edbf 
  clients/src/test/java/org/apache/kafka/test/MockSelector.java 
f83fd9b794a3bd191121a22bcb40fd6ec31d83b5 
  core/src/main/scala/kafka/controller/ControllerChannelManager.scala 
da1cff07f7f76dcfa5a805718febcccd4ed5f578 
  core/src/main/scala/kafka/server/KafkaConfig.scala 
1e8b2331486ffe55bfcc0919e48e12aad23b7d3c 
  core/src/main/scala/kafka/server/KafkaServer.scala 
30406ce809caaac56aca1f30c235b35962d55a50 
  core/src/main/scala/kafka/tools/ProducerPerformance.scala 
46a68e97b8bcc8821f21e4220ce9b3acedc5dafe 
  core/src/main/scala/kafka/utils/NetworkClientBlockingOps.scala 
ad10721de844725f27a116611209992cea61b088 
  core/src/test/scala/integration/kafka/api/ProducerFailureHandlingTest.scala 
1198df02ddd7727269e84a751ba99520f6d5584a 
  core/src/test/scala/unit/kafka/server/KafkaConfigTest.scala 
5b4f2db4607ae6d17696c1140f1a771ce75c80e0 
  core/src/test/scala/unit/kafka/utils/TestUtils.scala 
09b8444c2add87f0f70dbb182e892977a6b5c243 

Diff: https://reviews.apache.org/r/36858/diff/


Testing
---


Thanks,

Mayuresh Gharat

Re: Review Request 36858: Patch for KAFKA-2120

[Mayuresh Gharat]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36858/#review98327
---



clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java
 (line 208)


I am going to upload a new patch for breaking out of this iteration early 
if possible. I am going to test it out and upload a new patch if that works.


- Mayuresh Gharat


On Sept. 9, 2015, 11:49 p.m., Mayuresh Gharat wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/36858/
> ---
> 
> (Updated Sept. 9, 2015, 11:49 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-2120
> https://issues.apache.org/jira/browse/KAFKA-2120
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> Solved compile error
> 
> 
> Addressed Jason's comments for Kip-19
> 
> 
> Addressed Jun's comments
> 
> 
> Addressed Jason's comments about the default values for requestTimeout
> 
> 
> checkpoint
> 
> 
> Addressed Joel's concerns. Also tried to include Jun's feedback.
> 
> 
> Fixed a minor comment
> 
> 
> Solved unittest issue
> 
> 
> Addressed Jun's comments regarding NetworkClient
> 
> 
> Addressed Jun's comments about disconnect() in Selector
> 
> 
> Diffs
> -
> 
>   clients/src/main/java/org/apache/kafka/clients/ClientRequest.java 
> dc8f0f115bcda893c95d17c0a57be8d14518d034 
>   clients/src/main/java/org/apache/kafka/clients/CommonClientConfigs.java 
> 7d24c6f5dd2b63b96584f3aa8922a1d048dc1ae4 
>   clients/src/main/java/org/apache/kafka/clients/InFlightRequests.java 
> 15d00d4e484bb5d51a9ae6857ed6e024a2cc1820 
>   clients/src/main/java/org/apache/kafka/clients/KafkaClient.java 
> f46c0d9b5eb73887c62a0e09c96e9d8c964c709d 
>   clients/src/main/java/org/apache/kafka/clients/NetworkClient.java 
> 049b22eadd5496b70dfcfd9d821f67c62c68a052 
>   clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerConfig.java 
> b9a2d4e2bc565f0ee72b27791afe5c894af262f1 
>   clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java 
> 19ef6ebead3dc97d4912985e35b6b8d17503cb0e 
>   
> clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerNetworkClient.java
>  9517d9d0cd480d5ba1d12f1fde7963e60528d2f8 
>   clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 
> 804d569498396d431880641041fc9292076452cb 
>   clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 
> 06f00a99a73a288df9afa8c1d4abe3580fa968a6 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/BufferPool.java
>  4cb1e50d6c4ed55241aeaef1d3af09def5274103 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java
>  a152bd7697dca55609a9ec4cfe0a82c10595fbc3 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java
>  06182db1c3a5da85648199b4c0c98b80ea7c6c0c 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 
> d2e64f7cd8bf56e433a210905b2874f71eee9ea0 
>   clients/src/main/java/org/apache/kafka/common/network/Selectable.java 
> 70e74bd6aa629c430b2850ca40c97df0b16e5d75 
>   clients/src/main/java/org/apache/kafka/common/network/Selector.java 
> 4aa5cbb86ce6e1bf8f6769147ee2a6452c855c74 
>   clients/src/test/java/org/apache/kafka/clients/MockClient.java 
> e5815f56bdf8e2d980f2bc36b831ed234c0ac781 
>   clients/src/test/java/org/apache/kafka/clients/NetworkClientTest.java 
> 69c93c3adf674b1640534c3d7410fcaafaf2232c 
>   
> clients/src/test/java/org/apache/kafka/clients/producer/internals/BufferPoolTest.java
>  2c693824fa53db1e38766b8c66a0ef42ef9d0f3a 
>   
> clients/src/test/java/org/apache/kafka/clients/producer/internals/RecordAccumulatorTest.java
>  5b2e4ffaeab7127648db608c179703b27b577414 
>   
> clients/src/test/java/org/apache/kafka/clients/producer/internals/SenderTest.java
>  aa44991777a855f4b7f4f7bf17107c69393ff8ff 
>   clients/src/test/java/org/apache/kafka/common/network/SSLSelectorTest.java 
> df1205c935bee9a30a50816dbade64d6014b1ef2 
>   clients/src/test/java/org/apache/kafka/common/network/SelectorTest.java 
> 3a684d98b05cadfb25c6f7f9a038ef1f6697edbf 
>   clients/src/test/java/org/apache/kafka/test/MockSelector.java 
> f83fd9b794a3bd191121a22bcb40fd6ec31d83b5 
>   core/src/main/scala/kafka/controller/ControllerChannelManager.scala 
> da1cff07f7f76dcfa5a805718febcccd4ed5f578 
>   core/src/main/scala/kafka/server/KafkaConfig.scala 
> 1e8b2331486ffe55bfcc0919e48e12aad23b7d3c 
>   core/src/main/scala/kafka/server/KafkaServer.scala 
> 30406ce809caaac56aca1f30c235b35962d55a50 
>   core/src/main/scala/kafka/tools/ProducerPerformance.scala 
> 46a68e97b8bcc8821f21e4220ce9b3acedc5dafe 
>   

Re: Review Request 36858: Patch for KAFKA-2120

[Mayuresh Gharat]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36858/
---

(Updated Sept. 10, 2015, 1:56 a.m.)


Review request for kafka.


Bugs: KAFKA-2120
https://issues.apache.org/jira/browse/KAFKA-2120


Repository: kafka


Description (updated)
---

Solved compile error


Addressed Jason's comments for Kip-19


Addressed Jun's comments


Addressed Jason's comments about the default values for requestTimeout


checkpoint


Addressed Joel's concerns. Also tried to include Jun's feedback.


Fixed a minor comment


Solved unittest issue


Addressed Jun's comments regarding NetworkClient


Addressed Jun's comments about disconnect() in Selector


changed logging level to debug


Addressed Joels comments to break out early from the loop while aborting 
expired batches


Diffs (updated)
-

  clients/src/main/java/org/apache/kafka/clients/ClientRequest.java 
dc8f0f115bcda893c95d17c0a57be8d14518d034 
  clients/src/main/java/org/apache/kafka/clients/CommonClientConfigs.java 
7d24c6f5dd2b63b96584f3aa8922a1d048dc1ae4 
  clients/src/main/java/org/apache/kafka/clients/InFlightRequests.java 
15d00d4e484bb5d51a9ae6857ed6e024a2cc1820 
  clients/src/main/java/org/apache/kafka/clients/KafkaClient.java 
f46c0d9b5eb73887c62a0e09c96e9d8c964c709d 
  clients/src/main/java/org/apache/kafka/clients/NetworkClient.java 
049b22eadd5496b70dfcfd9d821f67c62c68a052 
  clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerConfig.java 
b9a2d4e2bc565f0ee72b27791afe5c894af262f1 
  clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java 
19ef6ebead3dc97d4912985e35b6b8d17503cb0e 
  
clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerNetworkClient.java
 9517d9d0cd480d5ba1d12f1fde7963e60528d2f8 
  clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 
804d569498396d431880641041fc9292076452cb 
  clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 
06f00a99a73a288df9afa8c1d4abe3580fa968a6 
  
clients/src/main/java/org/apache/kafka/clients/producer/internals/BufferPool.java
 4cb1e50d6c4ed55241aeaef1d3af09def5274103 
  
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java
 a152bd7697dca55609a9ec4cfe0a82c10595fbc3 
  
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java
 06182db1c3a5da85648199b4c0c98b80ea7c6c0c 
  clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 
d2e64f7cd8bf56e433a210905b2874f71eee9ea0 
  clients/src/main/java/org/apache/kafka/common/network/Selectable.java 
70e74bd6aa629c430b2850ca40c97df0b16e5d75 
  clients/src/main/java/org/apache/kafka/common/network/Selector.java 
4aa5cbb86ce6e1bf8f6769147ee2a6452c855c74 
  clients/src/test/java/org/apache/kafka/clients/MockClient.java 
e5815f56bdf8e2d980f2bc36b831ed234c0ac781 
  clients/src/test/java/org/apache/kafka/clients/NetworkClientTest.java 
69c93c3adf674b1640534c3d7410fcaafaf2232c 
  
clients/src/test/java/org/apache/kafka/clients/producer/internals/BufferPoolTest.java
 2c693824fa53db1e38766b8c66a0ef42ef9d0f3a 
  
clients/src/test/java/org/apache/kafka/clients/producer/internals/RecordAccumulatorTest.java
 5b2e4ffaeab7127648db608c179703b27b577414 
  
clients/src/test/java/org/apache/kafka/clients/producer/internals/SenderTest.java
 aa44991777a855f4b7f4f7bf17107c69393ff8ff 
  clients/src/test/java/org/apache/kafka/common/network/SSLSelectorTest.java 
df1205c935bee9a30a50816dbade64d6014b1ef2 
  clients/src/test/java/org/apache/kafka/common/network/SelectorTest.java 
3a684d98b05cadfb25c6f7f9a038ef1f6697edbf 
  clients/src/test/java/org/apache/kafka/test/MockSelector.java 
f83fd9b794a3bd191121a22bcb40fd6ec31d83b5 
  core/src/main/scala/kafka/controller/ControllerChannelManager.scala 
da1cff07f7f76dcfa5a805718febcccd4ed5f578 
  core/src/main/scala/kafka/server/KafkaConfig.scala 
1e8b2331486ffe55bfcc0919e48e12aad23b7d3c 
  core/src/main/scala/kafka/server/KafkaServer.scala 
30406ce809caaac56aca1f30c235b35962d55a50 
  core/src/main/scala/kafka/tools/ProducerPerformance.scala 
46a68e97b8bcc8821f21e4220ce9b3acedc5dafe 
  core/src/main/scala/kafka/utils/NetworkClientBlockingOps.scala 
ad10721de844725f27a116611209992cea61b088 
  core/src/test/scala/integration/kafka/api/ProducerFailureHandlingTest.scala 
1198df02ddd7727269e84a751ba99520f6d5584a 
  core/src/test/scala/unit/kafka/server/KafkaConfigTest.scala 
5b4f2db4607ae6d17696c1140f1a771ce75c80e0 
  core/src/test/scala/unit/kafka/utils/TestUtils.scala 
09b8444c2add87f0f70dbb182e892977a6b5c243 

Diff: https://reviews.apache.org/r/36858/diff/


Testing
---


Thanks,

Mayuresh Gharat

Re: Review Request 36858: Patch for KAFKA-2120

[Jun Rao]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36858/#review98136
---


Thanks for the patch. A few more minor comments below.


clients/src/main/java/org/apache/kafka/clients/NetworkClient.java (line 406)


Perhaps it's better to log the warning in the caller and distinguish 
whether the disconnect is due to the client timeout or not.



clients/src/main/java/org/apache/kafka/clients/NetworkClient.java (lines 415 - 
417)


The comment on handleDisconnections() is no longer accurate.



clients/src/main/java/org/apache/kafka/common/network/Selector.java (lines 184 
- 189)


Do we still need disconnect(id)? It seems that we can just replace the 
usage in test with close(id) and remove disconnect from selectable and 
KafkaChannel? We have to implement close(id) in MockSelector().



core/src/main/scala/kafka/server/KafkaConfig.scala (line 67)


Do we need this? Could we just use ControllerSocketTimeoutMs?


- Jun Rao


On Sept. 5, 2015, 12:49 a.m., Mayuresh Gharat wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/36858/
> ---
> 
> (Updated Sept. 5, 2015, 12:49 a.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-2120
> https://issues.apache.org/jira/browse/KAFKA-2120
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> Solved compile error
> 
> 
> Addressed Jason's comments for Kip-19
> 
> 
> Addressed Jun's comments
> 
> 
> Addressed Jason's comments about the default values for requestTimeout
> 
> 
> checkpoint
> 
> 
> Addressed Joel's concerns. Also tried to include Jun's feedback.
> 
> 
> Fixed a minor comment
> 
> 
> Solved unittest issue
> 
> 
> Addressed Jun's comments regarding NetworkClient
> 
> 
> Diffs
> -
> 
>   clients/src/main/java/org/apache/kafka/clients/ClientRequest.java 
> dc8f0f115bcda893c95d17c0a57be8d14518d034 
>   clients/src/main/java/org/apache/kafka/clients/CommonClientConfigs.java 
> 7d24c6f5dd2b63b96584f3aa8922a1d048dc1ae4 
>   clients/src/main/java/org/apache/kafka/clients/InFlightRequests.java 
> 15d00d4e484bb5d51a9ae6857ed6e024a2cc1820 
>   clients/src/main/java/org/apache/kafka/clients/KafkaClient.java 
> f46c0d9b5eb73887c62a0e09c96e9d8c964c709d 
>   clients/src/main/java/org/apache/kafka/clients/NetworkClient.java 
> 049b22eadd5496b70dfcfd9d821f67c62c68a052 
>   clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerConfig.java 
> b9a2d4e2bc565f0ee72b27791afe5c894af262f1 
>   clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java 
> 73237e455a9e5aa38672522cfd9e5fcdafbcef3b 
>   
> clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerNetworkClient.java
>  9517d9d0cd480d5ba1d12f1fde7963e60528d2f8 
>   clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 
> 804d569498396d431880641041fc9292076452cb 
>   clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 
> 06f00a99a73a288df9afa8c1d4abe3580fa968a6 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/BufferPool.java
>  4cb1e50d6c4ed55241aeaef1d3af09def5274103 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java
>  a152bd7697dca55609a9ec4cfe0a82c10595fbc3 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java
>  06182db1c3a5da85648199b4c0c98b80ea7c6c0c 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 
> d2e64f7cd8bf56e433a210905b2874f71eee9ea0 
>   clients/src/main/java/org/apache/kafka/common/network/Selector.java 
> 4aa5cbb86ce6e1bf8f6769147ee2a6452c855c74 
>   clients/src/test/java/org/apache/kafka/clients/MockClient.java 
> e5815f56bdf8e2d980f2bc36b831ed234c0ac781 
>   clients/src/test/java/org/apache/kafka/clients/NetworkClientTest.java 
> 69c93c3adf674b1640534c3d7410fcaafaf2232c 
>   
> clients/src/test/java/org/apache/kafka/clients/producer/internals/BufferPoolTest.java
>  2c693824fa53db1e38766b8c66a0ef42ef9d0f3a 
>   
> clients/src/test/java/org/apache/kafka/clients/producer/internals/RecordAccumulatorTest.java
>  5b2e4ffaeab7127648db608c179703b27b577414 
>   
> clients/src/test/java/org/apache/kafka/clients/producer/internals/SenderTest.java
>  aa44991777a855f4b7f4f7bf17107c69393ff8ff 
>   clients/src/test/java/org/apache/kafka/test/MockSelector.java 
> f83fd9b794a3bd191121a22bcb40fd6ec31d83b5 
>   core/src/main/scala/kafka/controller/ControllerChannelManager.scala 
> da1cff07f7f76dcfa5a805718febcccd4ed5f578 
>   

Re: Review Request 36858: Patch for KAFKA-2120

[Jun Rao]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36858/#review97710
---



clients/src/main/java/org/apache/kafka/clients/NetworkClient.java (lines 361 - 
366)


But those closed connections are only added to the disconnected list on the 
next selector.select() call right? So, you still have the issue that after a 
networkClient.poll() call, some socket connections are already cancelled but 
the connectionState is not reflecting that (need to wait for next poll() call). 
Also, it's a bit weird that the disconnect is initated in NetworkClient, but we 
have to push the info through selector and get it back.

I was thinking that we can call handleTimedOutRequests() after 
handleConnections(). In that call, we first figure out the nodeIds that need to 
be closed. Then call selector.close() and for each such node, call the 
following code.

connectionStates.disconnected(node);
log.debug("Node {} disconnected.", node);
for (ClientRequest request : 
this.inFlightRequests.clearAll(node)) {
log.trace("Cancelled request {} due to node {} being 
disconnected", request, node);
if (!metadataUpdater.maybeHandleDisconnection(request))
responses.add(new ClientResponse(request, now, true, 
null));
}

The above code can be put in a private method and be reused in 
handleDisconnected().

Then, we can get rid of selector.disconect and replace existing usage with 
selector.close instead.


- Jun Rao


On Sept. 3, 2015, 10:12 p.m., Mayuresh Gharat wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/36858/
> ---
> 
> (Updated Sept. 3, 2015, 10:12 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-2120
> https://issues.apache.org/jira/browse/KAFKA-2120
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> Solved compile error
> 
> 
> Addressed Jason's comments for Kip-19
> 
> 
> Addressed Jun's comments
> 
> 
> Addressed Jason's comments about the default values for requestTimeout
> 
> 
> checkpoint
> 
> 
> Addressed Joel's concerns. Also tried to include Jun's feedback.
> 
> 
> Diffs
> -
> 
>   clients/src/main/java/org/apache/kafka/clients/ClientRequest.java 
> dc8f0f115bcda893c95d17c0a57be8d14518d034 
>   clients/src/main/java/org/apache/kafka/clients/CommonClientConfigs.java 
> 7d24c6f5dd2b63b96584f3aa8922a1d048dc1ae4 
>   clients/src/main/java/org/apache/kafka/clients/InFlightRequests.java 
> 15d00d4e484bb5d51a9ae6857ed6e024a2cc1820 
>   clients/src/main/java/org/apache/kafka/clients/KafkaClient.java 
> 7ab2503794ff3aab39df881bd9fbae6547827d3b 
>   clients/src/main/java/org/apache/kafka/clients/NetworkClient.java 
> b31f7f1fbf93d29268b93811c9aad3e3c18e5312 
>   clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerConfig.java 
> b9a2d4e2bc565f0ee72b27791afe5c894af262f1 
>   clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java 
> 938981c23ec16dfaf81d1e647929a59e1572f40f 
>   
> clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerNetworkClient.java
>  9517d9d0cd480d5ba1d12f1fde7963e60528d2f8 
>   clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 
> 804d569498396d431880641041fc9292076452cb 
>   clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 
> 06f00a99a73a288df9afa8c1d4abe3580fa968a6 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/BufferPool.java
>  4cb1e50d6c4ed55241aeaef1d3af09def5274103 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java
>  a152bd7697dca55609a9ec4cfe0a82c10595fbc3 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java
>  06182db1c3a5da85648199b4c0c98b80ea7c6c0c 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 
> d2e64f7cd8bf56e433a210905b2874f71eee9ea0 
>   clients/src/main/java/org/apache/kafka/common/network/Selector.java 
> f49d54cbc1915ac686ff70ac657f08e4c96489c1 
>   clients/src/test/java/org/apache/kafka/clients/MockClient.java 
> 9133d85342b11ba2c9888d4d2804d181831e7a8e 
>   clients/src/test/java/org/apache/kafka/clients/NetworkClientTest.java 
> 43238ceaad0322e39802b615bb805b895336a009 
>   
> clients/src/test/java/org/apache/kafka/clients/producer/internals/BufferPoolTest.java
>  2c693824fa53db1e38766b8c66a0ef42ef9d0f3a 
>   
> clients/src/test/java/org/apache/kafka/clients/producer/internals/RecordAccumulatorTest.java
>  5b2e4ffaeab7127648db608c179703b27b577414 
>   
> 

Re: Review Request 36858: Patch for KAFKA-2120

[Mayuresh Gharat]

> On Sept. 4, 2015, 4:07 p.m., Jun Rao wrote:
> > clients/src/main/java/org/apache/kafka/clients/NetworkClient.java, lines 
> > 362-367
> > 
> >
> > But those closed connections are only added to the disconnected list on 
> > the next selector.select() call right? So, you still have the issue that 
> > after a networkClient.poll() call, some socket connections are already 
> > cancelled but the connectionState is not reflecting that (need to wait for 
> > next poll() call). Also, it's a bit weird that the disconnect is initated 
> > in NetworkClient, but we have to push the info through selector and get it 
> > back.
> > 
> > I was thinking that we can call handleTimedOutRequests() after 
> > handleConnections(). In that call, we first figure out the nodeIds that 
> > need to be closed. Then call selector.close() and for each such node, call 
> > the following code.
> > 
> > connectionStates.disconnected(node);
> > log.debug("Node {} disconnected.", node);
> > for (ClientRequest request : 
> > this.inFlightRequests.clearAll(node)) {
> > log.trace("Cancelled request {} due to node {} being 
> > disconnected", request, node);
> > if (!metadataUpdater.maybeHandleDisconnection(request))
> > responses.add(new ClientResponse(request, now, 
> > true, null));
> > }
> > 
> > The above code can be put in a private method and be reused in 
> > handleDisconnected().
> > 
> > Then, we can get rid of selector.disconect and replace existing usage 
> > with selector.close instead.

Hi Jun,

Thanks a lot for the comments.

For the concern :

"But those closed connections are only added to the disconnected list on the 
next selector.select() call right? So, you still have the issue that after a 
networkClient.poll() call, some socket connections are already cancelled but 
the connectionState is not reflecting that (need to wait for next poll() 
call)", I am adding the explicitly disconnected nodes to the list of 
disconnected nodes in the same NetworkClient.poll() itself. I add it to a list 
called clientDisconnects. When the  NetworkClient calls handleDisconnections(), 
it calls selector.disconnected(). Inside selector.disconnected(), I check if 
there are any entries in clientDisconnects and add thenm to the disconnected 
list and clear the clientDisconnects list. This means that every call to 
disconnected will get you all the disconnected nodes in that 
NetworkClient.poll(). So the explicitly disconnected nodes are added to the 
disconnected list in the same NetworkClient.poll().

But this appraoch kind of changes what selector.disconnect() javadoc says. I 
thought that if client is explicltiy calling disconnect() he would expect the 
disconnection to happen immidiately. The approach that you suggested above 
avoids making this change and I agree with that. 

I think Kafka-2411 kind of introduced some changes and I will need to rebase 
this patch with that. I am facing some issues when I rebased (test failures), 
so I will try to get those solved, include the approach that you have suggested 
above and upload a new patch.


- Mayuresh


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36858/#review97710
---


On Sept. 3, 2015, 10:12 p.m., Mayuresh Gharat wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/36858/
> ---
> 
> (Updated Sept. 3, 2015, 10:12 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-2120
> https://issues.apache.org/jira/browse/KAFKA-2120
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> Solved compile error
> 
> 
> Addressed Jason's comments for Kip-19
> 
> 
> Addressed Jun's comments
> 
> 
> Addressed Jason's comments about the default values for requestTimeout
> 
> 
> checkpoint
> 
> 
> Addressed Joel's concerns. Also tried to include Jun's feedback.
> 
> 
> Diffs
> -
> 
>   clients/src/main/java/org/apache/kafka/clients/ClientRequest.java 
> dc8f0f115bcda893c95d17c0a57be8d14518d034 
>   clients/src/main/java/org/apache/kafka/clients/CommonClientConfigs.java 
> 7d24c6f5dd2b63b96584f3aa8922a1d048dc1ae4 
>   clients/src/main/java/org/apache/kafka/clients/InFlightRequests.java 
> 15d00d4e484bb5d51a9ae6857ed6e024a2cc1820 
>   clients/src/main/java/org/apache/kafka/clients/KafkaClient.java 
> 7ab2503794ff3aab39df881bd9fbae6547827d3b 
>   clients/src/main/java/org/apache/kafka/clients/NetworkClient.java 
> b31f7f1fbf93d29268b93811c9aad3e3c18e5312 
>   

Re: Review Request 36858: Patch for KAFKA-2120

[Mayuresh Gharat]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36858/
---

(Updated Sept. 5, 2015, 12:49 a.m.)


Review request for kafka.


Bugs: KAFKA-2120
https://issues.apache.org/jira/browse/KAFKA-2120


Repository: kafka


Description (updated)
---

Solved compile error


Addressed Jason's comments for Kip-19


Addressed Jun's comments


Addressed Jason's comments about the default values for requestTimeout


checkpoint


Addressed Joel's concerns. Also tried to include Jun's feedback.


Fixed a minor comment


Solved unittest issue


Addressed Jun's comments regarding NetworkClient


Diffs (updated)
-

  clients/src/main/java/org/apache/kafka/clients/ClientRequest.java 
dc8f0f115bcda893c95d17c0a57be8d14518d034 
  clients/src/main/java/org/apache/kafka/clients/CommonClientConfigs.java 
7d24c6f5dd2b63b96584f3aa8922a1d048dc1ae4 
  clients/src/main/java/org/apache/kafka/clients/InFlightRequests.java 
15d00d4e484bb5d51a9ae6857ed6e024a2cc1820 
  clients/src/main/java/org/apache/kafka/clients/KafkaClient.java 
f46c0d9b5eb73887c62a0e09c96e9d8c964c709d 
  clients/src/main/java/org/apache/kafka/clients/NetworkClient.java 
049b22eadd5496b70dfcfd9d821f67c62c68a052 
  clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerConfig.java 
b9a2d4e2bc565f0ee72b27791afe5c894af262f1 
  clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java 
73237e455a9e5aa38672522cfd9e5fcdafbcef3b 
  
clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerNetworkClient.java
 9517d9d0cd480d5ba1d12f1fde7963e60528d2f8 
  clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 
804d569498396d431880641041fc9292076452cb 
  clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 
06f00a99a73a288df9afa8c1d4abe3580fa968a6 
  
clients/src/main/java/org/apache/kafka/clients/producer/internals/BufferPool.java
 4cb1e50d6c4ed55241aeaef1d3af09def5274103 
  
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java
 a152bd7697dca55609a9ec4cfe0a82c10595fbc3 
  
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java
 06182db1c3a5da85648199b4c0c98b80ea7c6c0c 
  clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 
d2e64f7cd8bf56e433a210905b2874f71eee9ea0 
  clients/src/main/java/org/apache/kafka/common/network/Selector.java 
4aa5cbb86ce6e1bf8f6769147ee2a6452c855c74 
  clients/src/test/java/org/apache/kafka/clients/MockClient.java 
e5815f56bdf8e2d980f2bc36b831ed234c0ac781 
  clients/src/test/java/org/apache/kafka/clients/NetworkClientTest.java 
69c93c3adf674b1640534c3d7410fcaafaf2232c 
  
clients/src/test/java/org/apache/kafka/clients/producer/internals/BufferPoolTest.java
 2c693824fa53db1e38766b8c66a0ef42ef9d0f3a 
  
clients/src/test/java/org/apache/kafka/clients/producer/internals/RecordAccumulatorTest.java
 5b2e4ffaeab7127648db608c179703b27b577414 
  
clients/src/test/java/org/apache/kafka/clients/producer/internals/SenderTest.java
 aa44991777a855f4b7f4f7bf17107c69393ff8ff 
  clients/src/test/java/org/apache/kafka/test/MockSelector.java 
f83fd9b794a3bd191121a22bcb40fd6ec31d83b5 
  core/src/main/scala/kafka/controller/ControllerChannelManager.scala 
da1cff07f7f76dcfa5a805718febcccd4ed5f578 
  core/src/main/scala/kafka/server/KafkaConfig.scala 
1e8b2331486ffe55bfcc0919e48e12aad23b7d3c 
  core/src/main/scala/kafka/server/KafkaServer.scala 
30406ce809caaac56aca1f30c235b35962d55a50 
  core/src/main/scala/kafka/tools/ProducerPerformance.scala 
46a68e97b8bcc8821f21e4220ce9b3acedc5dafe 
  core/src/main/scala/kafka/utils/NetworkClientBlockingOps.scala 
ad10721de844725f27a116611209992cea61b088 
  core/src/test/scala/integration/kafka/api/ProducerFailureHandlingTest.scala 
1198df02ddd7727269e84a751ba99520f6d5584a 
  core/src/test/scala/unit/kafka/server/KafkaConfigTest.scala 
5b4f2db4607ae6d17696c1140f1a771ce75c80e0 
  core/src/test/scala/unit/kafka/utils/TestUtils.scala 
09b8444c2add87f0f70dbb182e892977a6b5c243 

Diff: https://reviews.apache.org/r/36858/diff/


Testing
---


Thanks,

Mayuresh Gharat

Re: Review Request 36858: Patch for KAFKA-2120

[Mayuresh Gharat]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36858/#review96965
---



clients/src/main/java/org/apache/kafka/clients/InFlightRequests.java (line 141)


discussed offline



clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 
(line 245)


As per the KIP, the replicationTimeout will be replaced by requestTimeout. 

"request.timeout.ms - the maximum time to wait for the response of a 
message AFTER the batch is ready,  including: actual network RTT, server 
replication time"


- Mayuresh Gharat


On Sept. 3, 2015, 10:12 p.m., Mayuresh Gharat wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/36858/
> ---
> 
> (Updated Sept. 3, 2015, 10:12 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-2120
> https://issues.apache.org/jira/browse/KAFKA-2120
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> Solved compile error
> 
> 
> Addressed Jason's comments for Kip-19
> 
> 
> Addressed Jun's comments
> 
> 
> Addressed Jason's comments about the default values for requestTimeout
> 
> 
> checkpoint
> 
> 
> Addressed Joel's concerns. Also tried to include Jun's feedback.
> 
> 
> Diffs
> -
> 
>   clients/src/main/java/org/apache/kafka/clients/ClientRequest.java 
> dc8f0f115bcda893c95d17c0a57be8d14518d034 
>   clients/src/main/java/org/apache/kafka/clients/CommonClientConfigs.java 
> 7d24c6f5dd2b63b96584f3aa8922a1d048dc1ae4 
>   clients/src/main/java/org/apache/kafka/clients/InFlightRequests.java 
> 15d00d4e484bb5d51a9ae6857ed6e024a2cc1820 
>   clients/src/main/java/org/apache/kafka/clients/KafkaClient.java 
> 7ab2503794ff3aab39df881bd9fbae6547827d3b 
>   clients/src/main/java/org/apache/kafka/clients/NetworkClient.java 
> b31f7f1fbf93d29268b93811c9aad3e3c18e5312 
>   clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerConfig.java 
> b9a2d4e2bc565f0ee72b27791afe5c894af262f1 
>   clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java 
> 938981c23ec16dfaf81d1e647929a59e1572f40f 
>   
> clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerNetworkClient.java
>  9517d9d0cd480d5ba1d12f1fde7963e60528d2f8 
>   clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 
> 804d569498396d431880641041fc9292076452cb 
>   clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 
> 06f00a99a73a288df9afa8c1d4abe3580fa968a6 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/BufferPool.java
>  4cb1e50d6c4ed55241aeaef1d3af09def5274103 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java
>  a152bd7697dca55609a9ec4cfe0a82c10595fbc3 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java
>  06182db1c3a5da85648199b4c0c98b80ea7c6c0c 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 
> d2e64f7cd8bf56e433a210905b2874f71eee9ea0 
>   clients/src/main/java/org/apache/kafka/common/network/Selector.java 
> f49d54cbc1915ac686ff70ac657f08e4c96489c1 
>   clients/src/test/java/org/apache/kafka/clients/MockClient.java 
> 9133d85342b11ba2c9888d4d2804d181831e7a8e 
>   clients/src/test/java/org/apache/kafka/clients/NetworkClientTest.java 
> 43238ceaad0322e39802b615bb805b895336a009 
>   
> clients/src/test/java/org/apache/kafka/clients/producer/internals/BufferPoolTest.java
>  2c693824fa53db1e38766b8c66a0ef42ef9d0f3a 
>   
> clients/src/test/java/org/apache/kafka/clients/producer/internals/RecordAccumulatorTest.java
>  5b2e4ffaeab7127648db608c179703b27b577414 
>   
> clients/src/test/java/org/apache/kafka/clients/producer/internals/SenderTest.java
>  aa44991777a855f4b7f4f7bf17107c69393ff8ff 
>   clients/src/test/java/org/apache/kafka/common/network/SSLSelectorTest.java 
> df1205c935bee9a30a50816dbade64d6014b1ef2 
>   clients/src/test/java/org/apache/kafka/common/network/SelectorTest.java 
> 3a684d98b05cadfb25c6f7f9a038ef1f6697edbf 
>   core/src/main/scala/kafka/tools/ProducerPerformance.scala 
> 0335cc64013ffe2cdf1c4879e86e11ec8c526712 
>   core/src/test/scala/integration/kafka/api/ProducerFailureHandlingTest.scala 
> 1198df02ddd7727269e84a751ba99520f6d5584a 
>   core/src/test/scala/unit/kafka/utils/TestUtils.scala 
> 09b8444c2add87f0f70dbb182e892977a6b5c243 
> 
> Diff: https://reviews.apache.org/r/36858/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Mayuresh Gharat
> 
>

Re: Review Request 36858: Patch for KAFKA-2120

[Mayuresh Gharat]

> On Sept. 1, 2015, 1:02 a.m., Jun Rao wrote:
> > clients/src/main/java/org/apache/kafka/clients/NetworkClient.java, lines 
> > 362-367
> > 
> >
> > We need to think through this logic a bit more. The patch here 
> > disconnects the connection from the selector, but doesn't mark the 
> > connectionState as disconnected immediately. Only until the next 
> > networkClient.poll(), does the connectionState change to disconnected. The 
> > issue with this approach is that selector.disconnect actually cancels the 
> > socket key. So, at that moment, the connection is no longer usable. 
> > However, the connectionState is still connected. A client can check the 
> > connection as ready and then makes a send call. The send will then get a 
> > CancelledKeyException, which is weird.
> > 
> > We are dealing with a similar issue in KAFKA-2411. Our current thinking 
> > is to have a networkClient.disconnect() that closes the socket key as well 
> > as removes the client from connectionState. This will make the state in 
> > networkClient consistent after each poll() call. If we have that, we can 
> > just call networkClient.disconnect() in handleTimedOutRequests() and handle 
> > those disconnected connections immediately. Then, we don't need to maintain 
> > the clientDisconnects state in Selector.
> 
> Mayuresh Gharat wrote:
> Thanks a lot Jun.
> I was thinking about something similar when I was rebasing the patch 
> yesterday with latest trunk. So the initial code that I wrote, was taking the 
> disconnected nodes and adding them to disconnected list in disconnect() in 
> Selector. But that imposes dpendency that handleTimeout() should be called 
> before handledisconnections(), because every poll clears the disconnected 
> list. I will test the approach you suggested and upload a patch for this.

Thinking more on this I feel that we can add the explicitly disconnected node 
in disconnect() function of Selector to the List in selector and 
call the handleTimeOut() before handleDisconnections() in NetworkClient. That 
will handle all the above requirements in the same poll(). The only issue with 
this is that as per the javadoc for disconnect() ("*The disconnection is 
asynchronous and will not be processed until the next {@link #poll(long) 
poll()} call.*")
The reason we need to have the explicitly disconnected node in to the selectors 
disconnected list is we need to have all the functionality in 
handleDisconnections() (metadataUpdate, clearing the inflight request for that 
node) and don't want to duplicate code.

The other way is maintain a list of disconnected node from handleTimeOut. In 
handleDisconnections iterate over the merged list of disconnected nodes from 
this list and the selectors disconnected list. This takes into consideration 
the approach that you mentioned above. What do you think?


- Mayuresh


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36858/#review97213
---


On Aug. 12, 2015, 5:59 p.m., Mayuresh Gharat wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/36858/
> ---
> 
> (Updated Aug. 12, 2015, 5:59 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-2120
> https://issues.apache.org/jira/browse/KAFKA-2120
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> Solved compile error
> 
> 
> Addressed Jason's comments for Kip-19
> 
> 
> Addressed Jun's comments
> 
> 
> Addressed Jason's comments about the default values for requestTimeout
> 
> 
> Diffs
> -
> 
>   clients/src/main/java/org/apache/kafka/clients/ClientRequest.java 
> dc8f0f115bcda893c95d17c0a57be8d14518d034 
>   clients/src/main/java/org/apache/kafka/clients/CommonClientConfigs.java 
> 2c421f42ed3fc5d61cf9c87a7eaa7bb23e26f63b 
>   clients/src/main/java/org/apache/kafka/clients/InFlightRequests.java 
> 15d00d4e484bb5d51a9ae6857ed6e024a2cc1820 
>   clients/src/main/java/org/apache/kafka/clients/KafkaClient.java 
> 7ab2503794ff3aab39df881bd9fbae6547827d3b 
>   clients/src/main/java/org/apache/kafka/clients/NetworkClient.java 
> 0e51d7bd461d253f4396a5b6ca7cd391658807fa 
>   clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerConfig.java 
> d35b421a515074d964c7fccb73d260b847ea5f00 
>   clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java 
> ed99e9bdf7c4ea7a6d4555d4488cf8ed0b80641b 
>   
> clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerNetworkClient.java
>  9517d9d0cd480d5ba1d12f1fde7963e60528d2f8 
>   clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 
> 03b8dd23df63a8d8a117f02eabcce4a2d48c44f7 
>   

Re: Review Request 36858: Patch for KAFKA-2120

[Mayuresh Gharat]

> On Sept. 1, 2015, 1:02 a.m., Jun Rao wrote:
> > clients/src/main/java/org/apache/kafka/clients/NetworkClient.java, lines 
> > 362-367
> > 
> >
> > We need to think through this logic a bit more. The patch here 
> > disconnects the connection from the selector, but doesn't mark the 
> > connectionState as disconnected immediately. Only until the next 
> > networkClient.poll(), does the connectionState change to disconnected. The 
> > issue with this approach is that selector.disconnect actually cancels the 
> > socket key. So, at that moment, the connection is no longer usable. 
> > However, the connectionState is still connected. A client can check the 
> > connection as ready and then makes a send call. The send will then get a 
> > CancelledKeyException, which is weird.
> > 
> > We are dealing with a similar issue in KAFKA-2411. Our current thinking 
> > is to have a networkClient.disconnect() that closes the socket key as well 
> > as removes the client from connectionState. This will make the state in 
> > networkClient consistent after each poll() call. If we have that, we can 
> > just call networkClient.disconnect() in handleTimedOutRequests() and handle 
> > those disconnected connections immediately. Then, we don't need to maintain 
> > the clientDisconnects state in Selector.

Thanks a lot Jun.
I was thinking about something similar when I was rebasing the patch yesterday 
with latest trunk. So the initial code that I wrote, was taking the 
disconnected nodes and adding them to disconnected list in disconnect() in 
Selector. But that imposes dpendency that handleTimeout() should be called 
before handledisconnections(), because every poll clears the disconnected list. 
I will test the approach you suggested and upload a patch for this.


- Mayuresh


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36858/#review97213
---


On Aug. 12, 2015, 5:59 p.m., Mayuresh Gharat wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/36858/
> ---
> 
> (Updated Aug. 12, 2015, 5:59 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-2120
> https://issues.apache.org/jira/browse/KAFKA-2120
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> Solved compile error
> 
> 
> Addressed Jason's comments for Kip-19
> 
> 
> Addressed Jun's comments
> 
> 
> Addressed Jason's comments about the default values for requestTimeout
> 
> 
> Diffs
> -
> 
>   clients/src/main/java/org/apache/kafka/clients/ClientRequest.java 
> dc8f0f115bcda893c95d17c0a57be8d14518d034 
>   clients/src/main/java/org/apache/kafka/clients/CommonClientConfigs.java 
> 2c421f42ed3fc5d61cf9c87a7eaa7bb23e26f63b 
>   clients/src/main/java/org/apache/kafka/clients/InFlightRequests.java 
> 15d00d4e484bb5d51a9ae6857ed6e024a2cc1820 
>   clients/src/main/java/org/apache/kafka/clients/KafkaClient.java 
> 7ab2503794ff3aab39df881bd9fbae6547827d3b 
>   clients/src/main/java/org/apache/kafka/clients/NetworkClient.java 
> 0e51d7bd461d253f4396a5b6ca7cd391658807fa 
>   clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerConfig.java 
> d35b421a515074d964c7fccb73d260b847ea5f00 
>   clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java 
> ed99e9bdf7c4ea7a6d4555d4488cf8ed0b80641b 
>   
> clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerNetworkClient.java
>  9517d9d0cd480d5ba1d12f1fde7963e60528d2f8 
>   clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 
> 03b8dd23df63a8d8a117f02eabcce4a2d48c44f7 
>   clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 
> aa264202f2724907924985a5ecbe74afc4c6c04b 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/BufferPool.java
>  4cb1e50d6c4ed55241aeaef1d3af09def5274103 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java
>  a152bd7697dca55609a9ec4cfe0a82c10595fbc3 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java
>  06182db1c3a5da85648199b4c0c98b80ea7c6c0c 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 
> 0baf16e55046a2f49f6431e01d52c323c95eddf0 
>   clients/src/main/java/org/apache/kafka/common/network/Selector.java 
> ce20111ac434eb8c74585e9c63757bb9d60a832f 
>   clients/src/test/java/org/apache/kafka/clients/MockClient.java 
> 9133d85342b11ba2c9888d4d2804d181831e7a8e 
>   clients/src/test/java/org/apache/kafka/clients/NetworkClientTest.java 
> 43238ceaad0322e39802b615bb805b895336a009 
>   
> 

Re: Review Request 36858: Patch for KAFKA-2120

[Jun Rao]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36858/#review97213
---



clients/src/main/java/org/apache/kafka/clients/NetworkClient.java (lines 361 - 
366)


We need to think through this logic a bit more. The patch here disconnects 
the connection from the selector, but doesn't mark the connectionState as 
disconnected immediately. Only until the next networkClient.poll(), does the 
connectionState change to disconnected. The issue with this approach is that 
selector.disconnect actually cancels the socket key. So, at that moment, the 
connection is no longer usable. However, the connectionState is still 
connected. A client can check the connection as ready and then makes a send 
call. The send will then get a CancelledKeyException, which is weird.

We are dealing with a similar issue in KAFKA-2411. Our current thinking is 
to have a networkClient.disconnect() that closes the socket key as well as 
removes the client from connectionState. This will make the state in 
networkClient consistent after each poll() call. If we have that, we can just 
call networkClient.disconnect() in handleTimedOutRequests() and handle those 
disconnected connections immediately. Then, we don't need to maintain the 
clientDisconnects state in Selector.


- Jun Rao


On Aug. 12, 2015, 5:59 p.m., Mayuresh Gharat wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/36858/
> ---
> 
> (Updated Aug. 12, 2015, 5:59 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-2120
> https://issues.apache.org/jira/browse/KAFKA-2120
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> Solved compile error
> 
> 
> Addressed Jason's comments for Kip-19
> 
> 
> Addressed Jun's comments
> 
> 
> Addressed Jason's comments about the default values for requestTimeout
> 
> 
> Diffs
> -
> 
>   clients/src/main/java/org/apache/kafka/clients/ClientRequest.java 
> dc8f0f115bcda893c95d17c0a57be8d14518d034 
>   clients/src/main/java/org/apache/kafka/clients/CommonClientConfigs.java 
> 2c421f42ed3fc5d61cf9c87a7eaa7bb23e26f63b 
>   clients/src/main/java/org/apache/kafka/clients/InFlightRequests.java 
> 15d00d4e484bb5d51a9ae6857ed6e024a2cc1820 
>   clients/src/main/java/org/apache/kafka/clients/KafkaClient.java 
> 7ab2503794ff3aab39df881bd9fbae6547827d3b 
>   clients/src/main/java/org/apache/kafka/clients/NetworkClient.java 
> 0e51d7bd461d253f4396a5b6ca7cd391658807fa 
>   clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerConfig.java 
> d35b421a515074d964c7fccb73d260b847ea5f00 
>   clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java 
> ed99e9bdf7c4ea7a6d4555d4488cf8ed0b80641b 
>   
> clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerNetworkClient.java
>  9517d9d0cd480d5ba1d12f1fde7963e60528d2f8 
>   clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 
> 03b8dd23df63a8d8a117f02eabcce4a2d48c44f7 
>   clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 
> aa264202f2724907924985a5ecbe74afc4c6c04b 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/BufferPool.java
>  4cb1e50d6c4ed55241aeaef1d3af09def5274103 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java
>  a152bd7697dca55609a9ec4cfe0a82c10595fbc3 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java
>  06182db1c3a5da85648199b4c0c98b80ea7c6c0c 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 
> 0baf16e55046a2f49f6431e01d52c323c95eddf0 
>   clients/src/main/java/org/apache/kafka/common/network/Selector.java 
> ce20111ac434eb8c74585e9c63757bb9d60a832f 
>   clients/src/test/java/org/apache/kafka/clients/MockClient.java 
> 9133d85342b11ba2c9888d4d2804d181831e7a8e 
>   clients/src/test/java/org/apache/kafka/clients/NetworkClientTest.java 
> 43238ceaad0322e39802b615bb805b895336a009 
>   
> clients/src/test/java/org/apache/kafka/clients/producer/internals/BufferPoolTest.java
>  2c693824fa53db1e38766b8c66a0ef42ef9d0f3a 
>   
> clients/src/test/java/org/apache/kafka/clients/producer/internals/RecordAccumulatorTest.java
>  5b2e4ffaeab7127648db608c179703b27b577414 
>   
> clients/src/test/java/org/apache/kafka/clients/producer/internals/SenderTest.java
>  8b1805d3d2bcb9fe2bacb37d870c3236aa9532c4 
>   clients/src/test/java/org/apache/kafka/common/network/SelectorTest.java 
> 158f9829ff64a969008f699e40c51e918287859e 
>   core/src/main/scala/kafka/tools/ProducerPerformance.scala 
> 0335cc64013ffe2cdf1c4879e86e11ec8c526712 
>   

Re: Review Request 36858: Patch for KAFKA-2120

[Joel Koshy]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36858/#review96095
---


Can you rebase?


clients/src/main/java/org/apache/kafka/clients/InFlightRequests.java (line 130)
https://reviews.apache.org/r/36858/#comment151310

Returns a list of nodes with pending inflight requests that need to be 
timed out



clients/src/main/java/org/apache/kafka/clients/InFlightRequests.java (line 138)
https://reviews.apache.org/r/36858/#comment151311

Can do without this guard.



clients/src/main/java/org/apache/kafka/clients/InFlightRequests.java (line 141)
https://reviews.apache.org/r/36858/#comment151323

Is this right? i.e., `lastSent`?



clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 
(line 228)
https://reviews.apache.org/r/36858/#comment152086

This logic is a bit confusing. Is this block necessary here? i.e., vs being 
written once below?



clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 
(line 245)
https://reviews.apache.org/r/36858/#comment152089

I'm not sure we should remove the _replication timeout_ though. i.e., sure 
the replication timeout should not be used for request timeout going forward, 
but we still need a replication timeout in the producer request.



clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 
(line 428)
https://reviews.apache.org/r/36858/#comment152144

Minor point: given that there may be a custom serializer and custom 
partitioner, the elapsed check should probably be made after each step.



clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 
(line 100)
https://reviews.apache.org/r/36858/#comment152146

See comment above on the (continued) need for replication timeout. Also, 
can you fix whitespace on all your comments? i.e., `* @deprecated` instead of 
`*@deprecated`.



clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 
(line 196)
https://reviews.apache.org/r/36858/#comment152148

Let us make this doc string as exhaustive and clear as possible - i.e., 
drop the etc. and enumerate everything. You can also add something along the 
lines of - also see the request timeout config, with a doc link.



clients/src/main/java/org/apache/kafka/clients/producer/internals/BufferPool.java
 (line 94)
https://reviews.apache.org/r/36858/#comment152158

Can you add the new param to the javadoc?



clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java
 (line 155)
https://reviews.apache.org/r/36858/#comment152162

same here



clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java
 (line 205)
https://reviews.apache.org/r/36858/#comment152168

Maybe drop the due to This would be just one possibility right? E.g., 
you could have a low request timeout and high linger time... possibly other 
scenarios as well.



clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java
 (line 214)
https://reviews.apache.org/r/36858/#comment152170

typo in comment



clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java
 (line 221)
https://reviews.apache.org/r/36858/#comment152172

If you do a reverse iteration and a batch has _not_ expired, then we can 
break early right?


- Joel Koshy


On Aug. 12, 2015, 5:59 p.m., Mayuresh Gharat wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/36858/
 ---
 
 (Updated Aug. 12, 2015, 5:59 p.m.)
 
 
 Review request for kafka.
 
 
 Bugs: KAFKA-2120
 https://issues.apache.org/jira/browse/KAFKA-2120
 
 
 Repository: kafka
 
 
 Description
 ---
 
 Solved compile error
 
 
 Addressed Jason's comments for Kip-19
 
 
 Addressed Jun's comments
 
 
 Addressed Jason's comments about the default values for requestTimeout
 
 
 Diffs
 -
 
   clients/src/main/java/org/apache/kafka/clients/ClientRequest.java 
 dc8f0f115bcda893c95d17c0a57be8d14518d034 
   clients/src/main/java/org/apache/kafka/clients/CommonClientConfigs.java 
 2c421f42ed3fc5d61cf9c87a7eaa7bb23e26f63b 
   clients/src/main/java/org/apache/kafka/clients/InFlightRequests.java 
 15d00d4e484bb5d51a9ae6857ed6e024a2cc1820 
   clients/src/main/java/org/apache/kafka/clients/KafkaClient.java 
 7ab2503794ff3aab39df881bd9fbae6547827d3b 
   clients/src/main/java/org/apache/kafka/clients/NetworkClient.java 
 0e51d7bd461d253f4396a5b6ca7cd391658807fa 
   clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerConfig.java 
 d35b421a515074d964c7fccb73d260b847ea5f00 
   clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java 
 ed99e9bdf7c4ea7a6d4555d4488cf8ed0b80641b 
   
 

Re: Review Request 36858: Patch for KAFKA-2120

[Joel Koshy]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36858/#review96586
---



clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java
 (line 240)
https://reviews.apache.org/r/36858/#comment152173

Also, per the KIP agreement, the request timeout will reset for retries - 
are you handling that?


- Joel Koshy


On Aug. 12, 2015, 5:59 p.m., Mayuresh Gharat wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/36858/
 ---
 
 (Updated Aug. 12, 2015, 5:59 p.m.)
 
 
 Review request for kafka.
 
 
 Bugs: KAFKA-2120
 https://issues.apache.org/jira/browse/KAFKA-2120
 
 
 Repository: kafka
 
 
 Description
 ---
 
 Solved compile error
 
 
 Addressed Jason's comments for Kip-19
 
 
 Addressed Jun's comments
 
 
 Addressed Jason's comments about the default values for requestTimeout
 
 
 Diffs
 -
 
   clients/src/main/java/org/apache/kafka/clients/ClientRequest.java 
 dc8f0f115bcda893c95d17c0a57be8d14518d034 
   clients/src/main/java/org/apache/kafka/clients/CommonClientConfigs.java 
 2c421f42ed3fc5d61cf9c87a7eaa7bb23e26f63b 
   clients/src/main/java/org/apache/kafka/clients/InFlightRequests.java 
 15d00d4e484bb5d51a9ae6857ed6e024a2cc1820 
   clients/src/main/java/org/apache/kafka/clients/KafkaClient.java 
 7ab2503794ff3aab39df881bd9fbae6547827d3b 
   clients/src/main/java/org/apache/kafka/clients/NetworkClient.java 
 0e51d7bd461d253f4396a5b6ca7cd391658807fa 
   clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerConfig.java 
 d35b421a515074d964c7fccb73d260b847ea5f00 
   clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java 
 ed99e9bdf7c4ea7a6d4555d4488cf8ed0b80641b 
   
 clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerNetworkClient.java
  9517d9d0cd480d5ba1d12f1fde7963e60528d2f8 
   clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 
 03b8dd23df63a8d8a117f02eabcce4a2d48c44f7 
   clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 
 aa264202f2724907924985a5ecbe74afc4c6c04b 
   
 clients/src/main/java/org/apache/kafka/clients/producer/internals/BufferPool.java
  4cb1e50d6c4ed55241aeaef1d3af09def5274103 
   
 clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java
  a152bd7697dca55609a9ec4cfe0a82c10595fbc3 
   
 clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java
  06182db1c3a5da85648199b4c0c98b80ea7c6c0c 
   
 clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 
 0baf16e55046a2f49f6431e01d52c323c95eddf0 
   clients/src/main/java/org/apache/kafka/common/network/Selector.java 
 ce20111ac434eb8c74585e9c63757bb9d60a832f 
   clients/src/test/java/org/apache/kafka/clients/MockClient.java 
 9133d85342b11ba2c9888d4d2804d181831e7a8e 
   clients/src/test/java/org/apache/kafka/clients/NetworkClientTest.java 
 43238ceaad0322e39802b615bb805b895336a009 
   
 clients/src/test/java/org/apache/kafka/clients/producer/internals/BufferPoolTest.java
  2c693824fa53db1e38766b8c66a0ef42ef9d0f3a 
   
 clients/src/test/java/org/apache/kafka/clients/producer/internals/RecordAccumulatorTest.java
  5b2e4ffaeab7127648db608c179703b27b577414 
   
 clients/src/test/java/org/apache/kafka/clients/producer/internals/SenderTest.java
  8b1805d3d2bcb9fe2bacb37d870c3236aa9532c4 
   clients/src/test/java/org/apache/kafka/common/network/SelectorTest.java 
 158f9829ff64a969008f699e40c51e918287859e 
   core/src/main/scala/kafka/tools/ProducerPerformance.scala 
 0335cc64013ffe2cdf1c4879e86e11ec8c526712 
   core/src/test/scala/integration/kafka/api/ProducerFailureHandlingTest.scala 
 ee94011894b46864614b97bbd2a98375a7d3f20b 
   core/src/test/scala/unit/kafka/utils/TestUtils.scala 
 eb169d8b33c27d598cc24e5a2e5f78b789fa38d3 
 
 Diff: https://reviews.apache.org/r/36858/diff/
 
 
 Testing
 ---
 
 
 Thanks,
 
 Mayuresh Gharat
 

Re: Review Request 36858: Patch for KAFKA-2120

[Jason Gustafson]

 On Aug. 11, 2015, 8:49 p.m., Jason Gustafson wrote:
  clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerConfig.java,
   line 302
  https://reviews.apache.org/r/36858/diff/4/?file=1037078#file1037078line302
 
  Can we make this value greater than sessionTimeoutMs (which is 30s). 
  Even if we don't address the issue of sanity between the different timeouts 
  in this patch, it would be nice to have compatible defaults to keep the 
  consumer from breaking out of the box.
 
 Mayuresh Gharat wrote:
 Hi Jason,
 
 I will upload a new patch with the sanity test.
 
 Mayuresh Gharat wrote:
 Hi Jason,
 
 I am creating a new jira ticket for the sanity test. This is because the 
 requestTimeout in KafkaConsumer is long but it has to be int. This is because 
 it is used as timeout in produceRequest that is used as replicationTimeout on 
 the server side. The change might occur at multiple places and is completely 
 orthogonal to this KIP. Here is the jira ticket :
 https://issues.apache.org/jira/browse/KAFKA-2428

Cool, thanks!


- Jason


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36858/#review94999
---


On Aug. 12, 2015, 5:59 p.m., Mayuresh Gharat wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/36858/
 ---
 
 (Updated Aug. 12, 2015, 5:59 p.m.)
 
 
 Review request for kafka.
 
 
 Bugs: KAFKA-2120
 https://issues.apache.org/jira/browse/KAFKA-2120
 
 
 Repository: kafka
 
 
 Description
 ---
 
 Solved compile error
 
 
 Addressed Jason's comments for Kip-19
 
 
 Addressed Jun's comments
 
 
 Addressed Jason's comments about the default values for requestTimeout
 
 
 Diffs
 -
 
   clients/src/main/java/org/apache/kafka/clients/ClientRequest.java 
 dc8f0f115bcda893c95d17c0a57be8d14518d034 
   clients/src/main/java/org/apache/kafka/clients/CommonClientConfigs.java 
 2c421f42ed3fc5d61cf9c87a7eaa7bb23e26f63b 
   clients/src/main/java/org/apache/kafka/clients/InFlightRequests.java 
 15d00d4e484bb5d51a9ae6857ed6e024a2cc1820 
   clients/src/main/java/org/apache/kafka/clients/KafkaClient.java 
 7ab2503794ff3aab39df881bd9fbae6547827d3b 
   clients/src/main/java/org/apache/kafka/clients/NetworkClient.java 
 0e51d7bd461d253f4396a5b6ca7cd391658807fa 
   clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerConfig.java 
 d35b421a515074d964c7fccb73d260b847ea5f00 
   clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java 
 ed99e9bdf7c4ea7a6d4555d4488cf8ed0b80641b 
   
 clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerNetworkClient.java
  9517d9d0cd480d5ba1d12f1fde7963e60528d2f8 
   clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 
 03b8dd23df63a8d8a117f02eabcce4a2d48c44f7 
   clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 
 aa264202f2724907924985a5ecbe74afc4c6c04b 
   
 clients/src/main/java/org/apache/kafka/clients/producer/internals/BufferPool.java
  4cb1e50d6c4ed55241aeaef1d3af09def5274103 
   
 clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java
  a152bd7697dca55609a9ec4cfe0a82c10595fbc3 
   
 clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java
  06182db1c3a5da85648199b4c0c98b80ea7c6c0c 
   
 clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 
 0baf16e55046a2f49f6431e01d52c323c95eddf0 
   clients/src/main/java/org/apache/kafka/common/network/Selector.java 
 ce20111ac434eb8c74585e9c63757bb9d60a832f 
   clients/src/test/java/org/apache/kafka/clients/MockClient.java 
 9133d85342b11ba2c9888d4d2804d181831e7a8e 
   clients/src/test/java/org/apache/kafka/clients/NetworkClientTest.java 
 43238ceaad0322e39802b615bb805b895336a009 
   
 clients/src/test/java/org/apache/kafka/clients/producer/internals/BufferPoolTest.java
  2c693824fa53db1e38766b8c66a0ef42ef9d0f3a 
   
 clients/src/test/java/org/apache/kafka/clients/producer/internals/RecordAccumulatorTest.java
  5b2e4ffaeab7127648db608c179703b27b577414 
   
 clients/src/test/java/org/apache/kafka/clients/producer/internals/SenderTest.java
  8b1805d3d2bcb9fe2bacb37d870c3236aa9532c4 
   clients/src/test/java/org/apache/kafka/common/network/SelectorTest.java 
 158f9829ff64a969008f699e40c51e918287859e 
   core/src/main/scala/kafka/tools/ProducerPerformance.scala 
 0335cc64013ffe2cdf1c4879e86e11ec8c526712 
   core/src/test/scala/integration/kafka/api/ProducerFailureHandlingTest.scala 
 ee94011894b46864614b97bbd2a98375a7d3f20b 
   core/src/test/scala/unit/kafka/utils/TestUtils.scala 
 eb169d8b33c27d598cc24e5a2e5f78b789fa38d3 
 
 Diff: https://reviews.apache.org/r/36858/diff/
 
 
 Testing
 ---
 
 
 Thanks,
 

Re: Review Request 36858: Patch for KAFKA-2120

[Mayuresh Gharat]

 On Aug. 11, 2015, 8:49 p.m., Jason Gustafson wrote:
  clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerConfig.java,
   line 302
  https://reviews.apache.org/r/36858/diff/4/?file=1037078#file1037078line302
 
  Can we make this value greater than sessionTimeoutMs (which is 30s). 
  Even if we don't address the issue of sanity between the different timeouts 
  in this patch, it would be nice to have compatible defaults to keep the 
  consumer from breaking out of the box.
 
 Mayuresh Gharat wrote:
 Hi Jason,
 
 I will upload a new patch with the sanity test.

Hi Jason,

I am creating a new jira ticket for the sanity test. This is because the 
requestTimeout in KafkaConsumer is long but it has to be int. This is because 
it is used as timeout in produceRequest that is used as replicationTimeout on 
the server side. The change might occur at multiple places and is completely 
orthogonal to this KIP. Here is the jira ticket :
https://issues.apache.org/jira/browse/KAFKA-2428


- Mayuresh


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36858/#review94999
---


On Aug. 11, 2015, 2:55 a.m., Mayuresh Gharat wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/36858/
 ---
 
 (Updated Aug. 11, 2015, 2:55 a.m.)
 
 
 Review request for kafka.
 
 
 Bugs: KAFKA-2120
 https://issues.apache.org/jira/browse/KAFKA-2120
 
 
 Repository: kafka
 
 
 Description
 ---
 
 Solved compile error
 
 
 Addressed Jason's comments for Kip-19
 
 
 Addressed Jun's comments
 
 
 Diffs
 -
 
   clients/src/main/java/org/apache/kafka/clients/ClientRequest.java 
 dc8f0f115bcda893c95d17c0a57be8d14518d034 
   clients/src/main/java/org/apache/kafka/clients/CommonClientConfigs.java 
 2c421f42ed3fc5d61cf9c87a7eaa7bb23e26f63b 
   clients/src/main/java/org/apache/kafka/clients/InFlightRequests.java 
 15d00d4e484bb5d51a9ae6857ed6e024a2cc1820 
   clients/src/main/java/org/apache/kafka/clients/KafkaClient.java 
 7ab2503794ff3aab39df881bd9fbae6547827d3b 
   clients/src/main/java/org/apache/kafka/clients/NetworkClient.java 
 0e51d7bd461d253f4396a5b6ca7cd391658807fa 
   clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerConfig.java 
 d35b421a515074d964c7fccb73d260b847ea5f00 
   clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java 
 ed99e9bdf7c4ea7a6d4555d4488cf8ed0b80641b 
   
 clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerNetworkClient.java
  9517d9d0cd480d5ba1d12f1fde7963e60528d2f8 
   clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 
 03b8dd23df63a8d8a117f02eabcce4a2d48c44f7 
   clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 
 aa264202f2724907924985a5ecbe74afc4c6c04b 
   
 clients/src/main/java/org/apache/kafka/clients/producer/internals/BufferPool.java
  4cb1e50d6c4ed55241aeaef1d3af09def5274103 
   
 clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java
  a152bd7697dca55609a9ec4cfe0a82c10595fbc3 
   
 clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java
  06182db1c3a5da85648199b4c0c98b80ea7c6c0c 
   
 clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 
 0baf16e55046a2f49f6431e01d52c323c95eddf0 
   clients/src/main/java/org/apache/kafka/common/network/Selector.java 
 ce20111ac434eb8c74585e9c63757bb9d60a832f 
   clients/src/test/java/org/apache/kafka/clients/MockClient.java 
 9133d85342b11ba2c9888d4d2804d181831e7a8e 
   clients/src/test/java/org/apache/kafka/clients/NetworkClientTest.java 
 43238ceaad0322e39802b615bb805b895336a009 
   
 clients/src/test/java/org/apache/kafka/clients/producer/internals/BufferPoolTest.java
  2c693824fa53db1e38766b8c66a0ef42ef9d0f3a 
   
 clients/src/test/java/org/apache/kafka/clients/producer/internals/RecordAccumulatorTest.java
  5b2e4ffaeab7127648db608c179703b27b577414 
   
 clients/src/test/java/org/apache/kafka/clients/producer/internals/SenderTest.java
  8b1805d3d2bcb9fe2bacb37d870c3236aa9532c4 
   clients/src/test/java/org/apache/kafka/common/network/SelectorTest.java 
 158f9829ff64a969008f699e40c51e918287859e 
   core/src/main/scala/kafka/tools/ProducerPerformance.scala 
 0335cc64013ffe2cdf1c4879e86e11ec8c526712 
   core/src/test/scala/integration/kafka/api/ProducerFailureHandlingTest.scala 
 ee94011894b46864614b97bbd2a98375a7d3f20b 
   core/src/test/scala/unit/kafka/utils/TestUtils.scala 
 eb169d8b33c27d598cc24e5a2e5f78b789fa38d3 
 
 Diff: https://reviews.apache.org/r/36858/diff/
 
 
 Testing
 ---
 
 
 Thanks,
 
 Mayuresh Gharat
 

Re: Review Request 36858: Patch for KAFKA-2120

[Mayuresh Gharat]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36858/
---

(Updated Aug. 12, 2015, 5:59 p.m.)


Review request for kafka.


Bugs: KAFKA-2120
https://issues.apache.org/jira/browse/KAFKA-2120


Repository: kafka


Description (updated)
---

Solved compile error


Addressed Jason's comments for Kip-19


Addressed Jun's comments


Addressed Jason's comments about the default values for requestTimeout


Diffs (updated)
-

  clients/src/main/java/org/apache/kafka/clients/ClientRequest.java 
dc8f0f115bcda893c95d17c0a57be8d14518d034 
  clients/src/main/java/org/apache/kafka/clients/CommonClientConfigs.java 
2c421f42ed3fc5d61cf9c87a7eaa7bb23e26f63b 
  clients/src/main/java/org/apache/kafka/clients/InFlightRequests.java 
15d00d4e484bb5d51a9ae6857ed6e024a2cc1820 
  clients/src/main/java/org/apache/kafka/clients/KafkaClient.java 
7ab2503794ff3aab39df881bd9fbae6547827d3b 
  clients/src/main/java/org/apache/kafka/clients/NetworkClient.java 
0e51d7bd461d253f4396a5b6ca7cd391658807fa 
  clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerConfig.java 
d35b421a515074d964c7fccb73d260b847ea5f00 
  clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java 
ed99e9bdf7c4ea7a6d4555d4488cf8ed0b80641b 
  
clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerNetworkClient.java
 9517d9d0cd480d5ba1d12f1fde7963e60528d2f8 
  clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 
03b8dd23df63a8d8a117f02eabcce4a2d48c44f7 
  clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 
aa264202f2724907924985a5ecbe74afc4c6c04b 
  
clients/src/main/java/org/apache/kafka/clients/producer/internals/BufferPool.java
 4cb1e50d6c4ed55241aeaef1d3af09def5274103 
  
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java
 a152bd7697dca55609a9ec4cfe0a82c10595fbc3 
  
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java
 06182db1c3a5da85648199b4c0c98b80ea7c6c0c 
  clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 
0baf16e55046a2f49f6431e01d52c323c95eddf0 
  clients/src/main/java/org/apache/kafka/common/network/Selector.java 
ce20111ac434eb8c74585e9c63757bb9d60a832f 
  clients/src/test/java/org/apache/kafka/clients/MockClient.java 
9133d85342b11ba2c9888d4d2804d181831e7a8e 
  clients/src/test/java/org/apache/kafka/clients/NetworkClientTest.java 
43238ceaad0322e39802b615bb805b895336a009 
  
clients/src/test/java/org/apache/kafka/clients/producer/internals/BufferPoolTest.java
 2c693824fa53db1e38766b8c66a0ef42ef9d0f3a 
  
clients/src/test/java/org/apache/kafka/clients/producer/internals/RecordAccumulatorTest.java
 5b2e4ffaeab7127648db608c179703b27b577414 
  
clients/src/test/java/org/apache/kafka/clients/producer/internals/SenderTest.java
 8b1805d3d2bcb9fe2bacb37d870c3236aa9532c4 
  clients/src/test/java/org/apache/kafka/common/network/SelectorTest.java 
158f9829ff64a969008f699e40c51e918287859e 
  core/src/main/scala/kafka/tools/ProducerPerformance.scala 
0335cc64013ffe2cdf1c4879e86e11ec8c526712 
  core/src/test/scala/integration/kafka/api/ProducerFailureHandlingTest.scala 
ee94011894b46864614b97bbd2a98375a7d3f20b 
  core/src/test/scala/unit/kafka/utils/TestUtils.scala 
eb169d8b33c27d598cc24e5a2e5f78b789fa38d3 

Diff: https://reviews.apache.org/r/36858/diff/


Testing
---


Thanks,

Mayuresh Gharat

Re: Review Request 36858: Patch for KAFKA-2120

[Mayuresh Gharat]

 On Aug. 11, 2015, 8:49 p.m., Jason Gustafson wrote:
  clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerConfig.java,
   line 302
  https://reviews.apache.org/r/36858/diff/4/?file=1037078#file1037078line302
 
  Can we make this value greater than sessionTimeoutMs (which is 30s). 
  Even if we don't address the issue of sanity between the different timeouts 
  in this patch, it would be nice to have compatible defaults to keep the 
  consumer from breaking out of the box.

Hi Jason,

I will upload a new patch with the sanity test.


- Mayuresh


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36858/#review94999
---


On Aug. 11, 2015, 2:55 a.m., Mayuresh Gharat wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/36858/
 ---
 
 (Updated Aug. 11, 2015, 2:55 a.m.)
 
 
 Review request for kafka.
 
 
 Bugs: KAFKA-2120
 https://issues.apache.org/jira/browse/KAFKA-2120
 
 
 Repository: kafka
 
 
 Description
 ---
 
 Solved compile error
 
 
 Addressed Jason's comments for Kip-19
 
 
 Addressed Jun's comments
 
 
 Diffs
 -
 
   clients/src/main/java/org/apache/kafka/clients/ClientRequest.java 
 dc8f0f115bcda893c95d17c0a57be8d14518d034 
   clients/src/main/java/org/apache/kafka/clients/CommonClientConfigs.java 
 2c421f42ed3fc5d61cf9c87a7eaa7bb23e26f63b 
   clients/src/main/java/org/apache/kafka/clients/InFlightRequests.java 
 15d00d4e484bb5d51a9ae6857ed6e024a2cc1820 
   clients/src/main/java/org/apache/kafka/clients/KafkaClient.java 
 7ab2503794ff3aab39df881bd9fbae6547827d3b 
   clients/src/main/java/org/apache/kafka/clients/NetworkClient.java 
 0e51d7bd461d253f4396a5b6ca7cd391658807fa 
   clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerConfig.java 
 d35b421a515074d964c7fccb73d260b847ea5f00 
   clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java 
 ed99e9bdf7c4ea7a6d4555d4488cf8ed0b80641b 
   
 clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerNetworkClient.java
  9517d9d0cd480d5ba1d12f1fde7963e60528d2f8 
   clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 
 03b8dd23df63a8d8a117f02eabcce4a2d48c44f7 
   clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 
 aa264202f2724907924985a5ecbe74afc4c6c04b 
   
 clients/src/main/java/org/apache/kafka/clients/producer/internals/BufferPool.java
  4cb1e50d6c4ed55241aeaef1d3af09def5274103 
   
 clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java
  a152bd7697dca55609a9ec4cfe0a82c10595fbc3 
   
 clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java
  06182db1c3a5da85648199b4c0c98b80ea7c6c0c 
   
 clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 
 0baf16e55046a2f49f6431e01d52c323c95eddf0 
   clients/src/main/java/org/apache/kafka/common/network/Selector.java 
 ce20111ac434eb8c74585e9c63757bb9d60a832f 
   clients/src/test/java/org/apache/kafka/clients/MockClient.java 
 9133d85342b11ba2c9888d4d2804d181831e7a8e 
   clients/src/test/java/org/apache/kafka/clients/NetworkClientTest.java 
 43238ceaad0322e39802b615bb805b895336a009 
   
 clients/src/test/java/org/apache/kafka/clients/producer/internals/BufferPoolTest.java
  2c693824fa53db1e38766b8c66a0ef42ef9d0f3a 
   
 clients/src/test/java/org/apache/kafka/clients/producer/internals/RecordAccumulatorTest.java
  5b2e4ffaeab7127648db608c179703b27b577414 
   
 clients/src/test/java/org/apache/kafka/clients/producer/internals/SenderTest.java
  8b1805d3d2bcb9fe2bacb37d870c3236aa9532c4 
   clients/src/test/java/org/apache/kafka/common/network/SelectorTest.java 
 158f9829ff64a969008f699e40c51e918287859e 
   core/src/main/scala/kafka/tools/ProducerPerformance.scala 
 0335cc64013ffe2cdf1c4879e86e11ec8c526712 
   core/src/test/scala/integration/kafka/api/ProducerFailureHandlingTest.scala 
 ee94011894b46864614b97bbd2a98375a7d3f20b 
   core/src/test/scala/unit/kafka/utils/TestUtils.scala 
 eb169d8b33c27d598cc24e5a2e5f78b789fa38d3 
 
 Diff: https://reviews.apache.org/r/36858/diff/
 
 
 Testing
 ---
 
 
 Thanks,
 
 Mayuresh Gharat
 

Re: Review Request 36858: Patch for KAFKA-2120

[Jun Rao]

 On Aug. 7, 2015, 12:36 a.m., Jun Rao wrote:
  clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java,
   line 223
  https://reviews.apache.org/r/36858/diff/3/?file=1024852#file1024852line223
 
  Not sure if the test is needed. First, it seems that batch should never 
  will be null. Second, let's say the producer can't connect to any broker. 
  The producer can't refresh the metdata. So the leader will still be the old 
  one and may not be null. In this case, it seems that we should still expire 
  the records.
 
 Mayuresh Gharat wrote:
 In this case : Second, let's say the producer can't connect to any 
 broker. The producer can't refresh the metdata. So the leader will still be 
 the old one and may not be null. In this case, it seems that we should still 
 expire the records., the request will eventually fail due to requestTimeout 
 and retry exhaustion, when trying to send to broker.
 
  I was thinking on the same line of your suggestion, expiring the 
 batch if it has exceeded the threshold even if we have metadata available, 
 but the KIP said explicitly that Request timeout will also be used when the 
 batches in the accumulator that are ready but not drained due to metadata 
 missing.

Got it. Thanks for the explanation.


- Jun


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36858/#review94447
---


On Aug. 11, 2015, 2:55 a.m., Mayuresh Gharat wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/36858/
 ---
 
 (Updated Aug. 11, 2015, 2:55 a.m.)
 
 
 Review request for kafka.
 
 
 Bugs: KAFKA-2120
 https://issues.apache.org/jira/browse/KAFKA-2120
 
 
 Repository: kafka
 
 
 Description
 ---
 
 Solved compile error
 
 
 Addressed Jason's comments for Kip-19
 
 
 Addressed Jun's comments
 
 
 Diffs
 -
 
   clients/src/main/java/org/apache/kafka/clients/ClientRequest.java 
 dc8f0f115bcda893c95d17c0a57be8d14518d034 
   clients/src/main/java/org/apache/kafka/clients/CommonClientConfigs.java 
 2c421f42ed3fc5d61cf9c87a7eaa7bb23e26f63b 
   clients/src/main/java/org/apache/kafka/clients/InFlightRequests.java 
 15d00d4e484bb5d51a9ae6857ed6e024a2cc1820 
   clients/src/main/java/org/apache/kafka/clients/KafkaClient.java 
 7ab2503794ff3aab39df881bd9fbae6547827d3b 
   clients/src/main/java/org/apache/kafka/clients/NetworkClient.java 
 0e51d7bd461d253f4396a5b6ca7cd391658807fa 
   clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerConfig.java 
 d35b421a515074d964c7fccb73d260b847ea5f00 
   clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java 
 ed99e9bdf7c4ea7a6d4555d4488cf8ed0b80641b 
   
 clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerNetworkClient.java
  9517d9d0cd480d5ba1d12f1fde7963e60528d2f8 
   clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 
 03b8dd23df63a8d8a117f02eabcce4a2d48c44f7 
   clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 
 aa264202f2724907924985a5ecbe74afc4c6c04b 
   
 clients/src/main/java/org/apache/kafka/clients/producer/internals/BufferPool.java
  4cb1e50d6c4ed55241aeaef1d3af09def5274103 
   
 clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java
  a152bd7697dca55609a9ec4cfe0a82c10595fbc3 
   
 clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java
  06182db1c3a5da85648199b4c0c98b80ea7c6c0c 
   
 clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 
 0baf16e55046a2f49f6431e01d52c323c95eddf0 
   clients/src/main/java/org/apache/kafka/common/network/Selector.java 
 ce20111ac434eb8c74585e9c63757bb9d60a832f 
   clients/src/test/java/org/apache/kafka/clients/MockClient.java 
 9133d85342b11ba2c9888d4d2804d181831e7a8e 
   clients/src/test/java/org/apache/kafka/clients/NetworkClientTest.java 
 43238ceaad0322e39802b615bb805b895336a009 
   
 clients/src/test/java/org/apache/kafka/clients/producer/internals/BufferPoolTest.java
  2c693824fa53db1e38766b8c66a0ef42ef9d0f3a 
   
 clients/src/test/java/org/apache/kafka/clients/producer/internals/RecordAccumulatorTest.java
  5b2e4ffaeab7127648db608c179703b27b577414 
   
 clients/src/test/java/org/apache/kafka/clients/producer/internals/SenderTest.java
  8b1805d3d2bcb9fe2bacb37d870c3236aa9532c4 
   clients/src/test/java/org/apache/kafka/common/network/SelectorTest.java 
 158f9829ff64a969008f699e40c51e918287859e 
   core/src/main/scala/kafka/tools/ProducerPerformance.scala 
 0335cc64013ffe2cdf1c4879e86e11ec8c526712 
   core/src/test/scala/integration/kafka/api/ProducerFailureHandlingTest.scala 
 ee94011894b46864614b97bbd2a98375a7d3f20b 
   

Re: Review Request 36858: Patch for KAFKA-2120

[Jason Gustafson]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36858/#review94999
---

Ship it!


LGTM (other than the minor issue below). As discussed on the jira board, a more 
general approach would be to allow a timeout on the client request itself. My 
guess is that we'll need that in the long run, but the approach here is a good 
starting point.


clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerConfig.java 
(line 302)
https://reviews.apache.org/r/36858/#comment149747

Can we make this value greater than sessionTimeoutMs (which is 30s). Even 
if we don't address the issue of sanity between the different timeouts in this 
patch, it would be nice to have compatible defaults to keep the consumer from 
breaking out of the box.


- Jason Gustafson


On Aug. 11, 2015, 2:55 a.m., Mayuresh Gharat wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/36858/
 ---
 
 (Updated Aug. 11, 2015, 2:55 a.m.)
 
 
 Review request for kafka.
 
 
 Bugs: KAFKA-2120
 https://issues.apache.org/jira/browse/KAFKA-2120
 
 
 Repository: kafka
 
 
 Description
 ---
 
 Solved compile error
 
 
 Addressed Jason's comments for Kip-19
 
 
 Addressed Jun's comments
 
 
 Diffs
 -
 
   clients/src/main/java/org/apache/kafka/clients/ClientRequest.java 
 dc8f0f115bcda893c95d17c0a57be8d14518d034 
   clients/src/main/java/org/apache/kafka/clients/CommonClientConfigs.java 
 2c421f42ed3fc5d61cf9c87a7eaa7bb23e26f63b 
   clients/src/main/java/org/apache/kafka/clients/InFlightRequests.java 
 15d00d4e484bb5d51a9ae6857ed6e024a2cc1820 
   clients/src/main/java/org/apache/kafka/clients/KafkaClient.java 
 7ab2503794ff3aab39df881bd9fbae6547827d3b 
   clients/src/main/java/org/apache/kafka/clients/NetworkClient.java 
 0e51d7bd461d253f4396a5b6ca7cd391658807fa 
   clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerConfig.java 
 d35b421a515074d964c7fccb73d260b847ea5f00 
   clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java 
 ed99e9bdf7c4ea7a6d4555d4488cf8ed0b80641b 
   
 clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerNetworkClient.java
  9517d9d0cd480d5ba1d12f1fde7963e60528d2f8 
   clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 
 03b8dd23df63a8d8a117f02eabcce4a2d48c44f7 
   clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 
 aa264202f2724907924985a5ecbe74afc4c6c04b 
   
 clients/src/main/java/org/apache/kafka/clients/producer/internals/BufferPool.java
  4cb1e50d6c4ed55241aeaef1d3af09def5274103 
   
 clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java
  a152bd7697dca55609a9ec4cfe0a82c10595fbc3 
   
 clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java
  06182db1c3a5da85648199b4c0c98b80ea7c6c0c 
   
 clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 
 0baf16e55046a2f49f6431e01d52c323c95eddf0 
   clients/src/main/java/org/apache/kafka/common/network/Selector.java 
 ce20111ac434eb8c74585e9c63757bb9d60a832f 
   clients/src/test/java/org/apache/kafka/clients/MockClient.java 
 9133d85342b11ba2c9888d4d2804d181831e7a8e 
   clients/src/test/java/org/apache/kafka/clients/NetworkClientTest.java 
 43238ceaad0322e39802b615bb805b895336a009 
   
 clients/src/test/java/org/apache/kafka/clients/producer/internals/BufferPoolTest.java
  2c693824fa53db1e38766b8c66a0ef42ef9d0f3a 
   
 clients/src/test/java/org/apache/kafka/clients/producer/internals/RecordAccumulatorTest.java
  5b2e4ffaeab7127648db608c179703b27b577414 
   
 clients/src/test/java/org/apache/kafka/clients/producer/internals/SenderTest.java
  8b1805d3d2bcb9fe2bacb37d870c3236aa9532c4 
   clients/src/test/java/org/apache/kafka/common/network/SelectorTest.java 
 158f9829ff64a969008f699e40c51e918287859e 
   core/src/main/scala/kafka/tools/ProducerPerformance.scala 
 0335cc64013ffe2cdf1c4879e86e11ec8c526712 
   core/src/test/scala/integration/kafka/api/ProducerFailureHandlingTest.scala 
 ee94011894b46864614b97bbd2a98375a7d3f20b 
   core/src/test/scala/unit/kafka/utils/TestUtils.scala 
 eb169d8b33c27d598cc24e5a2e5f78b789fa38d3 
 
 Diff: https://reviews.apache.org/r/36858/diff/
 
 
 Testing
 ---
 
 
 Thanks,
 
 Mayuresh Gharat
 

Re: Review Request 36858: Patch for KAFKA-2120

[Edward Ribeiro]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36858/#review94824
---



clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java
 (line 135)
https://reviews.apache.org/r/36858/#comment149464

When we code a method that may or may not execute a given operation, it's 
usually a good practice to name it ``maybeExpire``. There's a couple of 
examples of maybe* in Kafka code base already. wdyt?


- Edward Ribeiro


On July 29, 2015, 10:58 p.m., Mayuresh Gharat wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/36858/
 ---
 
 (Updated July 29, 2015, 10:58 p.m.)
 
 
 Review request for kafka.
 
 
 Bugs: KAFKA-2120
 https://issues.apache.org/jira/browse/KAFKA-2120
 
 
 Repository: kafka
 
 
 Description
 ---
 
 Patch for Kip-19 : Added RequestTimeOut and MaxBlockTimeOut
 
 Solved compile error
 
 
 Addressed Jason's comments for Kip-19
 
 
 Diffs
 -
 
   clients/src/main/java/org/apache/kafka/clients/ClientRequest.java 
 dc8f0f115bcda893c95d17c0a57be8d14518d034 
   clients/src/main/java/org/apache/kafka/clients/CommonClientConfigs.java 
 2c421f42ed3fc5d61cf9c87a7eaa7bb23e26f63b 
   clients/src/main/java/org/apache/kafka/clients/InFlightRequests.java 
 15d00d4e484bb5d51a9ae6857ed6e024a2cc1820 
   clients/src/main/java/org/apache/kafka/clients/KafkaClient.java 
 7ab2503794ff3aab39df881bd9fbae6547827d3b 
   clients/src/main/java/org/apache/kafka/clients/NetworkClient.java 
 0e51d7bd461d253f4396a5b6ca7cd391658807fa 
   clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerConfig.java 
 70377ae2fa46deb381139d28590ce6d4115e1adc 
   clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java 
 923ff999d1b04718ddd9a9132668446525bf62f3 
   
 clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerNetworkClient.java
  9517d9d0cd480d5ba1d12f1fde7963e60528d2f8 
   clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 
 03b8dd23df63a8d8a117f02eabcce4a2d48c44f7 
   clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 
 aa264202f2724907924985a5ecbe74afc4c6c04b 
   
 clients/src/main/java/org/apache/kafka/clients/producer/internals/BufferPool.java
  4cb1e50d6c4ed55241aeaef1d3af09def5274103 
   
 clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java
  a152bd7697dca55609a9ec4cfe0a82c10595fbc3 
   
 clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java
  06182db1c3a5da85648199b4c0c98b80ea7c6c0c 
   
 clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 
 0baf16e55046a2f49f6431e01d52c323c95eddf0 
   clients/src/main/java/org/apache/kafka/common/network/Selector.java 
 aaf60c98c2c0f4513a8d65ee0db67953a529d598 
   clients/src/test/java/org/apache/kafka/clients/MockClient.java 
 d9c97e966c0e2fb605b67285f4275abb89f8813e 
   clients/src/test/java/org/apache/kafka/clients/NetworkClientTest.java 
 43238ceaad0322e39802b615bb805b895336a009 
   
 clients/src/test/java/org/apache/kafka/clients/producer/internals/BufferPoolTest.java
  2c693824fa53db1e38766b8c66a0ef42ef9d0f3a 
   
 clients/src/test/java/org/apache/kafka/clients/producer/internals/RecordAccumulatorTest.java
  5b2e4ffaeab7127648db608c179703b27b577414 
   
 clients/src/test/java/org/apache/kafka/clients/producer/internals/SenderTest.java
  8b1805d3d2bcb9fe2bacb37d870c3236aa9532c4 
 
 Diff: https://reviews.apache.org/r/36858/diff/
 
 
 Testing
 ---
 
 
 Thanks,
 
 Mayuresh Gharat
 

Re: Review Request 36858: Patch for KAFKA-2120

[Edward Ribeiro]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36858/#review94817
---



clients/src/main/java/org/apache/kafka/clients/InFlightRequests.java (line 140)
https://reviews.apache.org/r/36858/#comment149453

Just out of curiosity: why this relience on use ``Iterable`` instead of 
using ``Collection`` or even ``List``. Didn't get why to use the highest 
interface is preferrable here. Again, just out of curiosity.



clients/src/main/java/org/apache/kafka/clients/InFlightRequests.java (line 141)
https://reviews.apache.org/r/36858/#comment149450

As jdk 1.7 is the current version then this line can be simplified to use 
diamond operators:

ListString nodeIds = new LinkedList();



clients/src/main/java/org/apache/kafka/clients/InFlightRequests.java (line 142)
https://reviews.apache.org/r/36858/#comment149449

this if condition is unnecessary: if the map is empty the for-loop will NOT 
iterate a single time.



clients/src/main/java/org/apache/kafka/clients/InFlightRequests.java (line 154)
https://reviews.apache.org/r/36858/#comment149452

nit: I would wrap this on Collections.unmodifiableList() to make this List  
virtually immutable, in this particular case.



clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java
 (line 209)
https://reviews.apache.org/r/36858/#comment149454

Use diamond operators:

``ListRecordBatch expiredBatches = new ArrayList();``



clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java
 (line 216)
https://reviews.apache.org/r/36858/#comment149460

The lines can be simplified (i.e., more readable) a tidy bit. You can 
either write:

```
   for (IteratorRecordBatch it = dq.iterator; it.hasNext; ) {
   RecordBatch batch = batchIterator.next();
   Node leader = cluster.leaderFor(topicAndPartition);
   if (batch != null  leader == null) {
  // check if the batch is expired
  if (batch.expire(requestTimeout, now)) {
 expiredBatches.add(batch);
 count++;
 it.remove();
 deallocate(batch);
  }
   }
   }
```

Or even simplier:

```
   for (RecordBatch batch : dq) {
   Node leader = cluster.leaderFor(topicAndPartition);
   if (batch != null  leader == null) {
  // check if the batch is expired
  if (batch.expire(requestTimeout, now)) {
 expiredBatches.add(batch);
 count++;
 dq.remove(batch);
 deallocate(batch);
  }
   }
   }
```



clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java
 (line 236)
https://reviews.apache.org/r/36858/#comment149461

Again, we could wrap it in ``Collections.unmodifiableList``? (ps: I am 
asking, not stating).


- Edward Ribeiro


On July 29, 2015, 10:58 p.m., Mayuresh Gharat wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/36858/
 ---
 
 (Updated July 29, 2015, 10:58 p.m.)
 
 
 Review request for kafka.
 
 
 Bugs: KAFKA-2120
 https://issues.apache.org/jira/browse/KAFKA-2120
 
 
 Repository: kafka
 
 
 Description
 ---
 
 Patch for Kip-19 : Added RequestTimeOut and MaxBlockTimeOut
 
 Solved compile error
 
 
 Addressed Jason's comments for Kip-19
 
 
 Diffs
 -
 
   clients/src/main/java/org/apache/kafka/clients/ClientRequest.java 
 dc8f0f115bcda893c95d17c0a57be8d14518d034 
   clients/src/main/java/org/apache/kafka/clients/CommonClientConfigs.java 
 2c421f42ed3fc5d61cf9c87a7eaa7bb23e26f63b 
   clients/src/main/java/org/apache/kafka/clients/InFlightRequests.java 
 15d00d4e484bb5d51a9ae6857ed6e024a2cc1820 
   clients/src/main/java/org/apache/kafka/clients/KafkaClient.java 
 7ab2503794ff3aab39df881bd9fbae6547827d3b 
   clients/src/main/java/org/apache/kafka/clients/NetworkClient.java 
 0e51d7bd461d253f4396a5b6ca7cd391658807fa 
   clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerConfig.java 
 70377ae2fa46deb381139d28590ce6d4115e1adc 
   clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java 
 923ff999d1b04718ddd9a9132668446525bf62f3 
   
 clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerNetworkClient.java
  9517d9d0cd480d5ba1d12f1fde7963e60528d2f8 
   clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 
 03b8dd23df63a8d8a117f02eabcce4a2d48c44f7 
   clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 
 aa264202f2724907924985a5ecbe74afc4c6c04b 
   
 

Re: Review Request 36858: Patch for KAFKA-2120

[Mayuresh Gharat]

 On Aug. 7, 2015, 12:36 a.m., Jun Rao wrote:
  clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java,
   line 223
  https://reviews.apache.org/r/36858/diff/3/?file=1024852#file1024852line223
 
  Not sure if the test is needed. First, it seems that batch should never 
  will be null. Second, let's say the producer can't connect to any broker. 
  The producer can't refresh the metdata. So the leader will still be the old 
  one and may not be null. In this case, it seems that we should still expire 
  the records.

In this case : Second, let's say the producer can't connect to any broker. The 
producer can't refresh the metdata. So the leader will still be the old one and 
may not be null. In this case, it seems that we should still expire the 
records., the request will eventually fail due to requestTimeout and retry 
exhaustion, when trying to send to broker.

 I was thinking on the same line of your suggestion, expiring the batch if 
it has exceeded the threshold even if we have metadata available, but the KIP 
said explicitly that Request timeout will also be used when the batches in the 
accumulator that are ready but not drained due to metadata missing.


 On Aug. 7, 2015, 12:36 a.m., Jun Rao wrote:
  clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java,
   line 228
  https://reviews.apache.org/r/36858/diff/3/?file=1024852#file1024852line228
 
  We can't remove from the iterator this way. Will get a 
  ConcurrentModificationException. Need to call iterator.remove.

Ahh. My Bad.


- Mayuresh


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36858/#review94447
---


On Aug. 11, 2015, 2:55 a.m., Mayuresh Gharat wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/36858/
 ---
 
 (Updated Aug. 11, 2015, 2:55 a.m.)
 
 
 Review request for kafka.
 
 
 Bugs: KAFKA-2120
 https://issues.apache.org/jira/browse/KAFKA-2120
 
 
 Repository: kafka
 
 
 Description
 ---
 
 Solved compile error
 
 
 Addressed Jason's comments for Kip-19
 
 
 Addressed Jun's comments
 
 
 Diffs
 -
 
   clients/src/main/java/org/apache/kafka/clients/ClientRequest.java 
 dc8f0f115bcda893c95d17c0a57be8d14518d034 
   clients/src/main/java/org/apache/kafka/clients/CommonClientConfigs.java 
 2c421f42ed3fc5d61cf9c87a7eaa7bb23e26f63b 
   clients/src/main/java/org/apache/kafka/clients/InFlightRequests.java 
 15d00d4e484bb5d51a9ae6857ed6e024a2cc1820 
   clients/src/main/java/org/apache/kafka/clients/KafkaClient.java 
 7ab2503794ff3aab39df881bd9fbae6547827d3b 
   clients/src/main/java/org/apache/kafka/clients/NetworkClient.java 
 0e51d7bd461d253f4396a5b6ca7cd391658807fa 
   clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerConfig.java 
 d35b421a515074d964c7fccb73d260b847ea5f00 
   clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java 
 ed99e9bdf7c4ea7a6d4555d4488cf8ed0b80641b 
   
 clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerNetworkClient.java
  9517d9d0cd480d5ba1d12f1fde7963e60528d2f8 
   clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 
 03b8dd23df63a8d8a117f02eabcce4a2d48c44f7 
   clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 
 aa264202f2724907924985a5ecbe74afc4c6c04b 
   
 clients/src/main/java/org/apache/kafka/clients/producer/internals/BufferPool.java
  4cb1e50d6c4ed55241aeaef1d3af09def5274103 
   
 clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java
  a152bd7697dca55609a9ec4cfe0a82c10595fbc3 
   
 clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java
  06182db1c3a5da85648199b4c0c98b80ea7c6c0c 
   
 clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 
 0baf16e55046a2f49f6431e01d52c323c95eddf0 
   clients/src/main/java/org/apache/kafka/common/network/Selector.java 
 ce20111ac434eb8c74585e9c63757bb9d60a832f 
   clients/src/test/java/org/apache/kafka/clients/MockClient.java 
 9133d85342b11ba2c9888d4d2804d181831e7a8e 
   clients/src/test/java/org/apache/kafka/clients/NetworkClientTest.java 
 43238ceaad0322e39802b615bb805b895336a009 
   
 clients/src/test/java/org/apache/kafka/clients/producer/internals/BufferPoolTest.java
  2c693824fa53db1e38766b8c66a0ef42ef9d0f3a 
   
 clients/src/test/java/org/apache/kafka/clients/producer/internals/RecordAccumulatorTest.java
  5b2e4ffaeab7127648db608c179703b27b577414 
   
 clients/src/test/java/org/apache/kafka/clients/producer/internals/SenderTest.java
  8b1805d3d2bcb9fe2bacb37d870c3236aa9532c4 
   clients/src/test/java/org/apache/kafka/common/network/SelectorTest.java 
 

Re: Review Request 36858: Patch for KAFKA-2120

[Mayuresh Gharat]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36858/
---

(Updated Aug. 11, 2015, 2:55 a.m.)


Review request for kafka.


Bugs: KAFKA-2120
https://issues.apache.org/jira/browse/KAFKA-2120


Repository: kafka


Description (updated)
---

Solved compile error


Addressed Jason's comments for Kip-19


Addressed Jun's comments


Diffs (updated)
-

  clients/src/main/java/org/apache/kafka/clients/ClientRequest.java 
dc8f0f115bcda893c95d17c0a57be8d14518d034 
  clients/src/main/java/org/apache/kafka/clients/CommonClientConfigs.java 
2c421f42ed3fc5d61cf9c87a7eaa7bb23e26f63b 
  clients/src/main/java/org/apache/kafka/clients/InFlightRequests.java 
15d00d4e484bb5d51a9ae6857ed6e024a2cc1820 
  clients/src/main/java/org/apache/kafka/clients/KafkaClient.java 
7ab2503794ff3aab39df881bd9fbae6547827d3b 
  clients/src/main/java/org/apache/kafka/clients/NetworkClient.java 
0e51d7bd461d253f4396a5b6ca7cd391658807fa 
  clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerConfig.java 
d35b421a515074d964c7fccb73d260b847ea5f00 
  clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java 
ed99e9bdf7c4ea7a6d4555d4488cf8ed0b80641b 
  
clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerNetworkClient.java
 9517d9d0cd480d5ba1d12f1fde7963e60528d2f8 
  clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 
03b8dd23df63a8d8a117f02eabcce4a2d48c44f7 
  clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 
aa264202f2724907924985a5ecbe74afc4c6c04b 
  
clients/src/main/java/org/apache/kafka/clients/producer/internals/BufferPool.java
 4cb1e50d6c4ed55241aeaef1d3af09def5274103 
  
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java
 a152bd7697dca55609a9ec4cfe0a82c10595fbc3 
  
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java
 06182db1c3a5da85648199b4c0c98b80ea7c6c0c 
  clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 
0baf16e55046a2f49f6431e01d52c323c95eddf0 
  clients/src/main/java/org/apache/kafka/common/network/Selector.java 
ce20111ac434eb8c74585e9c63757bb9d60a832f 
  clients/src/test/java/org/apache/kafka/clients/MockClient.java 
9133d85342b11ba2c9888d4d2804d181831e7a8e 
  clients/src/test/java/org/apache/kafka/clients/NetworkClientTest.java 
43238ceaad0322e39802b615bb805b895336a009 
  
clients/src/test/java/org/apache/kafka/clients/producer/internals/BufferPoolTest.java
 2c693824fa53db1e38766b8c66a0ef42ef9d0f3a 
  
clients/src/test/java/org/apache/kafka/clients/producer/internals/RecordAccumulatorTest.java
 5b2e4ffaeab7127648db608c179703b27b577414 
  
clients/src/test/java/org/apache/kafka/clients/producer/internals/SenderTest.java
 8b1805d3d2bcb9fe2bacb37d870c3236aa9532c4 
  clients/src/test/java/org/apache/kafka/common/network/SelectorTest.java 
158f9829ff64a969008f699e40c51e918287859e 
  core/src/main/scala/kafka/tools/ProducerPerformance.scala 
0335cc64013ffe2cdf1c4879e86e11ec8c526712 
  core/src/test/scala/integration/kafka/api/ProducerFailureHandlingTest.scala 
ee94011894b46864614b97bbd2a98375a7d3f20b 
  core/src/test/scala/unit/kafka/utils/TestUtils.scala 
eb169d8b33c27d598cc24e5a2e5f78b789fa38d3 

Diff: https://reviews.apache.org/r/36858/diff/


Testing
---


Thanks,

Mayuresh Gharat

Re: Review Request 36858: Patch for KAFKA-2120

[Jun Rao]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36858/#review94447
---


Thanks for the patch. A few comments below. Also, there seems to be style check 
failure when running unit tests.

:clients:checkstyleMain[ant:checkstyle] 
/Users/junrao/intellij/kafka/clients/src/main/java/org/apache/kafka/clients/InFlightRequests.java:22:8:
 Unused import - java.util.Set.
[ant:checkstyle] 
/Users/junrao/intellij/kafka/clients/src/main/java/org/apache/kafka/clients/InFlightRequests.java:23:8:
 Unused import - org.apache.kafka.common.utils.SystemTime.
[ant:checkstyle] 
/Users/junrao/intellij/kafka/clients/src/main/java/org/apache/kafka/clients/InFlightRequests.java:24:8:
 Unused import - org.apache.kafka.common.utils.Time.
[ant:checkstyle] 
/Users/junrao/intellij/kafka/clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerConfig.java:164:100:
 ';' is preceded with whitespace.
[ant:checkstyle] 
/Users/junrao/intellij/kafka/clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java:223:
 if child at indentation level 14 not at correct indentation, 16
[ant:checkstyle] 
/Users/junrao/intellij/kafka/clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java:223:
 method call child at indentation level 14 not at correct indentation, 16
[ant:checkstyle] 
/Users/junrao/intellij/kafka/clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java:224:
 method call child at indentation level 18 not at correct indentation, 20
[ant:checkstyle] 
/Users/junrao/intellij/kafka/clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java:225:
 if child at indentation level 14 not at correct indentation, 16
[ant:checkstyle] 
/Users/junrao/intellij/kafka/clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java:226:
 if at indentation level 14 not at correct indentation, 16
[ant:checkstyle] 
/Users/junrao/intellij/kafka/clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java:227:
 if child at indentation level 16 not at correct indentation, 20
[ant:checkstyle] 
/Users/junrao/intellij/kafka/clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java:228:
 if rcurly at indentation level 14 not at correct indentation, 16
[ant:checkstyle] 
/Users/junrao/intellij/kafka/clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java:229:
 if child at indentation level 16 not at correct indentation, 20
[ant:checkstyle] 
/Users/junrao/intellij/kafka/clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java:229:
 method call child at indentation level 16 not at correct indentation, 20
[ant:checkstyle] 
/Users/junrao/intellij/kafka/clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java:230:
 method call child at indentation level 20 not at correct indentation, 24
[ant:checkstyle] 
/Users/junrao/intellij/kafka/clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java:231:
 if child at indentation level 16 not at correct indentation, 20
[ant:checkstyle] 
/Users/junrao/intellij/kafka/clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java:232:
 if rcurly at indentation level 14 not at correct indentation, 16
[ant:checkstyle] 
/Users/junrao/intellij/kafka/clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java:233:
 else child at indentation level 16 not at correct indentation, 20
[ant:checkstyle] 
/Users/junrao/intellij/kafka/clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java:234:
 else rcurly at indentation level 14 not at correct indentation, 16
[ant:checkstyle] 
/Users/junrao/intellij/kafka/clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java:236:
 if child at indentation level 14 not at correct indentation, 16
[ant:checkstyle] 
/Users/junrao/intellij/kafka/clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java:236:
 method call child at indentation level 14 not at correct indentation, 16
[ant:checkstyle] 
/Users/junrao/intellij/kafka/clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java:237:
 method call child at indentation level 18 not at correct indentation, 20
[ant:checkstyle] 
/Users/junrao/intellij/kafka/clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java:238:
 if child at indentation level 14 not at correct indentation, 16
[ant:checkstyle] 
/Users/junrao/intellij/kafka/clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java:240:
 else child at indentation level 14 not at correct indentation, 16
[ant:checkstyle] 
/Users/junrao/intellij/kafka/clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java:248:
 if child at indentation level 14 not at correct indentation, 16
[ant:checkstyle] 

Re: Review Request 36858: Patch for KAFKA-2120

[Jason Gustafson]

 On July 27, 2015, 10:55 p.m., Jason Gustafson wrote:
  clients/src/main/java/org/apache/kafka/clients/ClientRequest.java, line 26
  https://reviews.apache.org/r/36858/diff/1/?file=1022752#file1022752line26
 
  Should ClientResponse.requestLatencyMs be updated to use sendMs instead 
  of createdMs? Or perhaps a new method like wireLatencyMs() can record 
  responseMs - sendMs? Both seem like useful metrics.
 
 Mayuresh Gharat wrote:
 Agreed with the wireLatency(). I would like to put it into a separate 
 patch, since this is orthogonal to this KIP and we also need to think were we 
 can use it. What do you think?

Yeah, I think that's reasonable.


 On July 27, 2015, 10:55 p.m., Jason Gustafson wrote:
  clients/src/main/java/org/apache/kafka/clients/InFlightRequests.java, line 
  48
  https://reviews.apache.org/r/36858/diff/1/?file=1022754#file1022754line48
 
  It seems like it might be more natural to set this in 
  NetworkClient.send.
 
 Mayuresh Gharat wrote:
 That was my original thinking, but the problem is that if we set this in 
 NetworkClient, we will have to set it in send() call and 
 maybeUpdateMetadata() as well. Basically it looks a little loose since if 
 there are new changes we might miss setting this. To avoid that we wanted to 
 set this in a single place.

Hmm.. It still sort of feels like it ought to be the responsibility of 
NetworkClient to set that and not InFlightRequests. An easy way to address the 
issue would be to create a NetworkClient.doSend method. For example:


```
void doSend(ClientRequest req) {
   req.setSendMs(time.milliseconds());
   this.selector.send(req.request());
   this.inFlightRequests.add(metadataRequest);
}
```

Then NetworkClient.send and NetworkClient.maybeUpdateMetdata could invoke this 
method. Better or worse?


 On July 27, 2015, 10:55 p.m., Jason Gustafson wrote:
  clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java, 
  line 538
  https://reviews.apache.org/r/36858/diff/1/?file=1022758#file1022758line538
 
  You don't necessarily need to fix this in this patch, but this setting 
  could conflict with FETCH_MAX_WAIT_MS_CONFIG.
 
 Mayuresh Gharat wrote:
 Yup. It kind of does. I will create a separate jira for this. Also The 
 requestTimeout in KafkaConsumer is long. It should be int.

Sounds good. I'll drop this issue.


 On July 27, 2015, 10:55 p.m., Jason Gustafson wrote:
  clients/src/main/java/org/apache/kafka/common/network/Selector.java, line 86
  https://reviews.apache.org/r/36858/diff/1/?file=1022765#file1022765line86
 
  Would it make sense to try to consolidate clientDisconnects and 
  failedSends into the same collection? Unfortunately I can't think of a good 
  name. Maybe nextDisconnects?
 
 Mayuresh Gharat wrote:
 I was thinking that it would be better, to separate out the request that 
 are failed due to request timeout causing the client to close it explicitly 
 and request that are failed because of other error between broker and client. 
 Also clientDisconnects can be renamed to clientSideDisconnects to make it 
 more explicit. What do you think?

It would be nice to keep them separate if there was functional value in doing 
so, but it seems like they are used pretty much identically as a way to pass 
disconnects to the next round of poll. That's probably fine if it makes the 
code more understandable, and that might just be a matter of preference.

What about userDisconnects instead of clientSideDisconnects?


- Jason


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36858/#review93189
---


On July 29, 2015, 10:58 p.m., Mayuresh Gharat wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/36858/
 ---
 
 (Updated July 29, 2015, 10:58 p.m.)
 
 
 Review request for kafka.
 
 
 Bugs: KAFKA-2120
 https://issues.apache.org/jira/browse/KAFKA-2120
 
 
 Repository: kafka
 
 
 Description
 ---
 
 Patch for Kip-19 : Added RequestTimeOut and MaxBlockTimeOut
 
 Solved compile error
 
 
 Addressed Jason's comments for Kip-19
 
 
 Diffs
 -
 
   clients/src/main/java/org/apache/kafka/clients/ClientRequest.java 
 dc8f0f115bcda893c95d17c0a57be8d14518d034 
   clients/src/main/java/org/apache/kafka/clients/CommonClientConfigs.java 
 2c421f42ed3fc5d61cf9c87a7eaa7bb23e26f63b 
   clients/src/main/java/org/apache/kafka/clients/InFlightRequests.java 
 15d00d4e484bb5d51a9ae6857ed6e024a2cc1820 
   clients/src/main/java/org/apache/kafka/clients/KafkaClient.java 
 7ab2503794ff3aab39df881bd9fbae6547827d3b 
   clients/src/main/java/org/apache/kafka/clients/NetworkClient.java 
 0e51d7bd461d253f4396a5b6ca7cd391658807fa 
   

Re: Review Request 36858: Patch for KAFKA-2120

[Mayuresh Gharat]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36858/
---

(Updated July 29, 2015, 10:57 p.m.)


Review request for kafka.


Bugs: KAFKA-2120
https://issues.apache.org/jira/browse/KAFKA-2120


Repository: kafka


Description (updated)
---

Solved compile error


Addressed Jason's comments for Kip-19


Diffs (updated)
-

  clients/src/main/java/org/apache/kafka/clients/ClientRequest.java 
dc8f0f115bcda893c95d17c0a57be8d14518d034 
  clients/src/main/java/org/apache/kafka/clients/CommonClientConfigs.java 
2c421f42ed3fc5d61cf9c87a7eaa7bb23e26f63b 
  clients/src/main/java/org/apache/kafka/clients/InFlightRequests.java 
15d00d4e484bb5d51a9ae6857ed6e024a2cc1820 
  clients/src/main/java/org/apache/kafka/clients/KafkaClient.java 
7ab2503794ff3aab39df881bd9fbae6547827d3b 
  clients/src/main/java/org/apache/kafka/clients/NetworkClient.java 
0e51d7bd461d253f4396a5b6ca7cd391658807fa 
  clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerConfig.java 
70377ae2fa46deb381139d28590ce6d4115e1adc 
  clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java 
923ff999d1b04718ddd9a9132668446525bf62f3 
  
clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerNetworkClient.java
 9517d9d0cd480d5ba1d12f1fde7963e60528d2f8 
  clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 
03b8dd23df63a8d8a117f02eabcce4a2d48c44f7 
  clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 
aa264202f2724907924985a5ecbe74afc4c6c04b 
  
clients/src/main/java/org/apache/kafka/clients/producer/internals/BufferPool.java
 4cb1e50d6c4ed55241aeaef1d3af09def5274103 
  
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java
 a152bd7697dca55609a9ec4cfe0a82c10595fbc3 
  
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java
 06182db1c3a5da85648199b4c0c98b80ea7c6c0c 
  clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 
0baf16e55046a2f49f6431e01d52c323c95eddf0 
  clients/src/main/java/org/apache/kafka/common/network/Selector.java 
aaf60c98c2c0f4513a8d65ee0db67953a529d598 
  clients/src/test/java/org/apache/kafka/clients/MockClient.java 
d9c97e966c0e2fb605b67285f4275abb89f8813e 
  clients/src/test/java/org/apache/kafka/clients/NetworkClientTest.java 
43238ceaad0322e39802b615bb805b895336a009 
  
clients/src/test/java/org/apache/kafka/clients/producer/internals/BufferPoolTest.java
 2c693824fa53db1e38766b8c66a0ef42ef9d0f3a 
  
clients/src/test/java/org/apache/kafka/clients/producer/internals/RecordAccumulatorTest.java
 5b2e4ffaeab7127648db608c179703b27b577414 
  
clients/src/test/java/org/apache/kafka/clients/producer/internals/SenderTest.java
 8b1805d3d2bcb9fe2bacb37d870c3236aa9532c4 

Diff: https://reviews.apache.org/r/36858/diff/


Testing
---


Thanks,

Mayuresh Gharat

Re: Review Request 36858: Patch for KAFKA-2120

[Mayuresh Gharat]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36858/
---

(Updated July 29, 2015, 10:58 p.m.)


Review request for kafka.


Bugs: KAFKA-2120
https://issues.apache.org/jira/browse/KAFKA-2120


Repository: kafka


Description (updated)
---

Patch for Kip-19 : Added RequestTimeOut and MaxBlockTimeOut

Solved compile error


Addressed Jason's comments for Kip-19


Diffs
-

  clients/src/main/java/org/apache/kafka/clients/ClientRequest.java 
dc8f0f115bcda893c95d17c0a57be8d14518d034 
  clients/src/main/java/org/apache/kafka/clients/CommonClientConfigs.java 
2c421f42ed3fc5d61cf9c87a7eaa7bb23e26f63b 
  clients/src/main/java/org/apache/kafka/clients/InFlightRequests.java 
15d00d4e484bb5d51a9ae6857ed6e024a2cc1820 
  clients/src/main/java/org/apache/kafka/clients/KafkaClient.java 
7ab2503794ff3aab39df881bd9fbae6547827d3b 
  clients/src/main/java/org/apache/kafka/clients/NetworkClient.java 
0e51d7bd461d253f4396a5b6ca7cd391658807fa 
  clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerConfig.java 
70377ae2fa46deb381139d28590ce6d4115e1adc 
  clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java 
923ff999d1b04718ddd9a9132668446525bf62f3 
  
clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerNetworkClient.java
 9517d9d0cd480d5ba1d12f1fde7963e60528d2f8 
  clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 
03b8dd23df63a8d8a117f02eabcce4a2d48c44f7 
  clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 
aa264202f2724907924985a5ecbe74afc4c6c04b 
  
clients/src/main/java/org/apache/kafka/clients/producer/internals/BufferPool.java
 4cb1e50d6c4ed55241aeaef1d3af09def5274103 
  
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java
 a152bd7697dca55609a9ec4cfe0a82c10595fbc3 
  
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java
 06182db1c3a5da85648199b4c0c98b80ea7c6c0c 
  clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 
0baf16e55046a2f49f6431e01d52c323c95eddf0 
  clients/src/main/java/org/apache/kafka/common/network/Selector.java 
aaf60c98c2c0f4513a8d65ee0db67953a529d598 
  clients/src/test/java/org/apache/kafka/clients/MockClient.java 
d9c97e966c0e2fb605b67285f4275abb89f8813e 
  clients/src/test/java/org/apache/kafka/clients/NetworkClientTest.java 
43238ceaad0322e39802b615bb805b895336a009 
  
clients/src/test/java/org/apache/kafka/clients/producer/internals/BufferPoolTest.java
 2c693824fa53db1e38766b8c66a0ef42ef9d0f3a 
  
clients/src/test/java/org/apache/kafka/clients/producer/internals/RecordAccumulatorTest.java
 5b2e4ffaeab7127648db608c179703b27b577414 
  
clients/src/test/java/org/apache/kafka/clients/producer/internals/SenderTest.java
 8b1805d3d2bcb9fe2bacb37d870c3236aa9532c4 

Diff: https://reviews.apache.org/r/36858/diff/


Testing
---


Thanks,

Mayuresh Gharat

Re: Review Request 36858: Patch for KAFKA-2120

[Mayuresh Gharat]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36858/
---

(Updated July 27, 2015, 9:09 p.m.)


Review request for kafka.


Bugs: KAFKA-2120
https://issues.apache.org/jira/browse/KAFKA-2120


Repository: kafka


Description (updated)
---

Kip-19 : Added RequestTimeout and MaxBlockTimeout as per the KIP.


Diffs
-

  clients/src/main/java/org/apache/kafka/clients/ClientRequest.java 
ed4c0d98596cc294757f35df8c8cbc8e36ff42de 
  clients/src/main/java/org/apache/kafka/clients/CommonClientConfigs.java 
2c421f42ed3fc5d61cf9c87a7eaa7bb23e26f63b 
  clients/src/main/java/org/apache/kafka/clients/InFlightRequests.java 
15d00d4e484bb5d51a9ae6857ed6e024a2cc1820 
  clients/src/main/java/org/apache/kafka/clients/KafkaClient.java 
7ab2503794ff3aab39df881bd9fbae6547827d3b 
  clients/src/main/java/org/apache/kafka/clients/NetworkClient.java 
48fe7961e2215372d8033ece4af739ea06c6457b 
  clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerConfig.java 
70377ae2fa46deb381139d28590ce6d4115e1adc 
  clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java 
bea3d737c51be77d5b5293cdd944d33b905422ba 
  clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 
03b8dd23df63a8d8a117f02eabcce4a2d48c44f7 
  clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 
aa264202f2724907924985a5ecbe74afc4c6c04b 
  
clients/src/main/java/org/apache/kafka/clients/producer/internals/BufferPool.java
 4cb1e50d6c4ed55241aeaef1d3af09def5274103 
  
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java
 a152bd7697dca55609a9ec4cfe0a82c10595fbc3 
  
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java
 06182db1c3a5da85648199b4c0c98b80ea7c6c0c 
  clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 
0baf16e55046a2f49f6431e01d52c323c95eddf0 
  clients/src/main/java/org/apache/kafka/common/network/Selector.java 
aaf60c98c2c0f4513a8d65ee0db67953a529d598 
  clients/src/test/java/org/apache/kafka/clients/MockClient.java 
d9c97e966c0e2fb605b67285f4275abb89f8813e 
  clients/src/test/java/org/apache/kafka/clients/NetworkClientTest.java 
43238ceaad0322e39802b615bb805b895336a009 
  
clients/src/test/java/org/apache/kafka/clients/producer/internals/BufferPoolTest.java
 2c693824fa53db1e38766b8c66a0ef42ef9d0f3a 
  
clients/src/test/java/org/apache/kafka/clients/producer/internals/RecordAccumulatorTest.java
 5b2e4ffaeab7127648db608c179703b27b577414 
  
clients/src/test/java/org/apache/kafka/clients/producer/internals/SenderTest.java
 8b1805d3d2bcb9fe2bacb37d870c3236aa9532c4 

Diff: https://reviews.apache.org/r/36858/diff/


Testing
---


Thanks,

Mayuresh Gharat

Re: Review Request 36858: Patch for KAFKA-2120

[Mayuresh Gharat]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36858/
---

(Updated July 27, 2015, 10:31 p.m.)


Review request for kafka.


Bugs: KAFKA-2120
https://issues.apache.org/jira/browse/KAFKA-2120


Repository: kafka


Description (updated)
---

Solved compile error


Diffs (updated)
-

  clients/src/main/java/org/apache/kafka/clients/ClientRequest.java 
ed4c0d98596cc294757f35df8c8cbc8e36ff42de 
  clients/src/main/java/org/apache/kafka/clients/CommonClientConfigs.java 
2c421f42ed3fc5d61cf9c87a7eaa7bb23e26f63b 
  clients/src/main/java/org/apache/kafka/clients/InFlightRequests.java 
15d00d4e484bb5d51a9ae6857ed6e024a2cc1820 
  clients/src/main/java/org/apache/kafka/clients/KafkaClient.java 
7ab2503794ff3aab39df881bd9fbae6547827d3b 
  clients/src/main/java/org/apache/kafka/clients/NetworkClient.java 
48fe7961e2215372d8033ece4af739ea06c6457b 
  clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerConfig.java 
70377ae2fa46deb381139d28590ce6d4115e1adc 
  clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java 
bea3d737c51be77d5b5293cdd944d33b905422ba 
  
clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerNetworkClient.java
 9517d9d0cd480d5ba1d12f1fde7963e60528d2f8 
  clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 
03b8dd23df63a8d8a117f02eabcce4a2d48c44f7 
  clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 
aa264202f2724907924985a5ecbe74afc4c6c04b 
  
clients/src/main/java/org/apache/kafka/clients/producer/internals/BufferPool.java
 4cb1e50d6c4ed55241aeaef1d3af09def5274103 
  
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java
 a152bd7697dca55609a9ec4cfe0a82c10595fbc3 
  
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java
 06182db1c3a5da85648199b4c0c98b80ea7c6c0c 
  clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 
0baf16e55046a2f49f6431e01d52c323c95eddf0 
  clients/src/main/java/org/apache/kafka/common/network/Selector.java 
aaf60c98c2c0f4513a8d65ee0db67953a529d598 
  clients/src/test/java/org/apache/kafka/clients/MockClient.java 
d9c97e966c0e2fb605b67285f4275abb89f8813e 
  clients/src/test/java/org/apache/kafka/clients/NetworkClientTest.java 
43238ceaad0322e39802b615bb805b895336a009 
  
clients/src/test/java/org/apache/kafka/clients/producer/internals/BufferPoolTest.java
 2c693824fa53db1e38766b8c66a0ef42ef9d0f3a 
  
clients/src/test/java/org/apache/kafka/clients/producer/internals/RecordAccumulatorTest.java
 5b2e4ffaeab7127648db608c179703b27b577414 
  
clients/src/test/java/org/apache/kafka/clients/producer/internals/SenderTest.java
 8b1805d3d2bcb9fe2bacb37d870c3236aa9532c4 

Diff: https://reviews.apache.org/r/36858/diff/


Testing
---


Thanks,

Mayuresh Gharat

Re: Review Request 36858: Patch for KAFKA-2120

[Jason Gustafson]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36858/#review93189
---


Looks pretty good overall. Found mostly trivial stuff.


clients/src/main/java/org/apache/kafka/clients/ClientRequest.java (line 26)
https://reviews.apache.org/r/36858/#comment147486

Should ClientResponse.requestLatencyMs be updated to use sendMs instead of 
createdMs? Or perhaps a new method like wireLatencyMs() can record responseMs - 
sendMs? Both seem like useful metrics.



clients/src/main/java/org/apache/kafka/clients/InFlightRequests.java (line 48)
https://reviews.apache.org/r/36858/#comment147491

It seems like it might be more natural to set this in NetworkClient.send.



clients/src/main/java/org/apache/kafka/clients/InFlightRequests.java (line 142)
https://reviews.apache.org/r/36858/#comment147492

Probably better practice to use !requests.isEmpty().



clients/src/main/java/org/apache/kafka/clients/KafkaClient.java (line 72)
https://reviews.apache.org/r/36858/#comment147493

Missing @param for now.



clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java 
(line 538)
https://reviews.apache.org/r/36858/#comment147497

You don't necessarily need to fix this in this patch, but this setting 
could conflict with FETCH_MAX_WAIT_MS_CONFIG.



clients/src/main/java/org/apache/kafka/common/network/Selector.java (line 86)
https://reviews.apache.org/r/36858/#comment147510

Would it make sense to try to consolidate clientDisconnects and failedSends 
into the same collection? Unfortunately I can't think of a good name. Maybe 
nextDisconnects?



clients/src/test/java/org/apache/kafka/clients/MockClient.java (line 94)
https://reviews.apache.org/r/36858/#comment147498

Do we need to set sendMs in this method for consistency?


- Jason Gustafson


On July 27, 2015, 10:32 p.m., Mayuresh Gharat wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/36858/
 ---
 
 (Updated July 27, 2015, 10:32 p.m.)
 
 
 Review request for kafka.
 
 
 Bugs: KAFKA-2120
 https://issues.apache.org/jira/browse/KAFKA-2120
 
 
 Repository: kafka
 
 
 Description
 ---
 
 Kip-19 :  Added RequestTimeout and MaxBlockTimeout as per the Kip
 
 
 Diffs
 -
 
   clients/src/main/java/org/apache/kafka/clients/ClientRequest.java 
 ed4c0d98596cc294757f35df8c8cbc8e36ff42de 
   clients/src/main/java/org/apache/kafka/clients/CommonClientConfigs.java 
 2c421f42ed3fc5d61cf9c87a7eaa7bb23e26f63b 
   clients/src/main/java/org/apache/kafka/clients/InFlightRequests.java 
 15d00d4e484bb5d51a9ae6857ed6e024a2cc1820 
   clients/src/main/java/org/apache/kafka/clients/KafkaClient.java 
 7ab2503794ff3aab39df881bd9fbae6547827d3b 
   clients/src/main/java/org/apache/kafka/clients/NetworkClient.java 
 48fe7961e2215372d8033ece4af739ea06c6457b 
   clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerConfig.java 
 70377ae2fa46deb381139d28590ce6d4115e1adc 
   clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java 
 bea3d737c51be77d5b5293cdd944d33b905422ba 
   
 clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerNetworkClient.java
  9517d9d0cd480d5ba1d12f1fde7963e60528d2f8 
   clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 
 03b8dd23df63a8d8a117f02eabcce4a2d48c44f7 
   clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 
 aa264202f2724907924985a5ecbe74afc4c6c04b 
   
 clients/src/main/java/org/apache/kafka/clients/producer/internals/BufferPool.java
  4cb1e50d6c4ed55241aeaef1d3af09def5274103 
   
 clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java
  a152bd7697dca55609a9ec4cfe0a82c10595fbc3 
   
 clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java
  06182db1c3a5da85648199b4c0c98b80ea7c6c0c 
   
 clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 
 0baf16e55046a2f49f6431e01d52c323c95eddf0 
   clients/src/main/java/org/apache/kafka/common/network/Selector.java 
 aaf60c98c2c0f4513a8d65ee0db67953a529d598 
   clients/src/test/java/org/apache/kafka/clients/MockClient.java 
 d9c97e966c0e2fb605b67285f4275abb89f8813e 
   clients/src/test/java/org/apache/kafka/clients/NetworkClientTest.java 
 43238ceaad0322e39802b615bb805b895336a009 
   
 clients/src/test/java/org/apache/kafka/clients/producer/internals/BufferPoolTest.java
  2c693824fa53db1e38766b8c66a0ef42ef9d0f3a 
   
 clients/src/test/java/org/apache/kafka/clients/producer/internals/RecordAccumulatorTest.java
  5b2e4ffaeab7127648db608c179703b27b577414 
   
 clients/src/test/java/org/apache/kafka/clients/producer/internals/SenderTest.java
  8b1805d3d2bcb9fe2bacb37d870c3236aa9532c4