date:20150306

Review Request 31816: Fix for KAFKA-527

2015-03-06 Thread Guozhang Wang


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31816/
---

Review request for kafka.


Bugs: KAFKA-527
https://issues.apache.org/jira/browse/KAFKA-527


Repository: kafka


Description
---

Avoid double copying on decompress


Diffs
-

  core/src/main/scala/kafka/consumer/ConsumerIterator.scala 
ac491b4da2583ef7227c67f5b8bc0fd731d705c3 
  core/src/main/scala/kafka/message/ByteBufferMessageSet.scala 
788c7864bc881b935975ab4a4e877b690e65f1f1 
  core/src/test/scala/unit/kafka/message/MessageCompressionTest.scala 
6f0addcea64f1e78a4de50ec8135f4d02cebd305 
  core/src/test/scala/unit/kafka/producer/SyncProducerTest.scala 
24deea06753e5358aa341c589ca7a7704317e29c 

Diff: https://reviews.apache.org/r/31816/diff/


Testing
---

Unit tests


Thanks,

Guozhang Wang

[jira] [Commented] (KAFKA-1660) Ability to call close() with a timeout on the Java Kafka Producer.

2015-03-06 Thread Jiangjie Qin (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-1660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14351186#comment-14351186
 ] 

Jiangjie Qin commented on KAFKA-1660:
-

Thanks for the quick response [~parth.brahmbhatt], I'll take this over as 
mirror maker would need this close call for no data loss.

 Ability to call close() with a timeout on the Java Kafka Producer. 
 ---

 Key: KAFKA-1660
 URL: https://issues.apache.org/jira/browse/KAFKA-1660
 Project: Kafka
  Issue Type: Improvement
  Components: clients, producer 
Affects Versions: 0.8.2.0
Reporter: Andrew Stein
Assignee: Parth Brahmbhatt
 Fix For: 0.8.3

 Attachments: KAFKA-1660.patch, KAFKA-1660_2015-02-17_16:41:19.patch, 
 KAFKA-1660_2015-03-02_10:41:49.patch


 I would like the ability to call {{close}} with a timeout on the Java 
 Client's KafkaProducer.
 h6. Workaround
 Currently, it is possible to ensure that {{close}} will return quickly by 
 first doing a {{future.get(timeout)}} on the last future produced on each 
 partition, but this means that the user has to define the partitions up front 
 at the time of {{send}} and track the returned {{future}}'s



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 31650: Drag Coordinator and FetchManager out of KafkaConsumer, fix a bunch of consumer test issues

2015-03-06 Thread Guozhang Wang



 On March 5, 2015, 11:39 p.m., Onur Karaman wrote:
  clients/src/test/java/org/apache/kafka/clients/consumer/internals/CoordinatorTest.java,
   line 83
  https://reviews.apache.org/r/31650/diff/2/?file=886350#file886350line83
 
  I think these scenarios should be split up into separate tests.
 
 Guozhang Wang wrote:
 The general law of defining unit test cases is by functionality instead 
 of scenarios, hence I think it is OK to group them in one test.
 
 Onur Karaman wrote:
 Grouping them means that if one of the earlier scenarios fail, then the 
 later scenarios will not be tested. So we don't know if only that one 
 scenario failed or if multiple later scenarios would have failed.

Hmm.. I think it depends on whether we want to trouble shoot failed tests 
one-at-a-time or just check if one of the tests fail in that test case, fix 
that one and retry. For this case I agree that it may be better to get 
knowledge of all failed cases. Will split them and check-in.


- Guozhang


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31650/#review75406
---


On March 5, 2015, 10:57 p.m., Guozhang Wang wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/31650/
 ---
 
 (Updated March 5, 2015, 10:57 p.m.)
 
 
 Review request for kafka.
 
 
 Bugs: KAFKA-1910
 https://issues.apache.org/jira/browse/KAFKA-1910
 
 
 Repository: kafka
 
 
 Description
 ---
 
 See comments in KAFKA-1910;
 
 Updated RB includes unit test for Coordinator / FetchManager / Heartbeat and 
 a couple changes on MemoryRecords and test utils.
 
 
 Diffs
 -
 
   clients/src/main/java/org/apache/kafka/clients/CommonClientConfigs.java 
 06fcfe62cc1fe76f58540221698ef076fe150e96 
   clients/src/main/java/org/apache/kafka/clients/KafkaClient.java 
 8a3e55aaff7d8c26e56a8407166a4176c1da2644 
   clients/src/main/java/org/apache/kafka/clients/NetworkClient.java 
 a7fa4a9dfbcfbc4d9e9259630253cbcded158064 
   clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerConfig.java 
 5fb21001abd77cac839bd724afa04e377a3e82aa 
   clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java 
 67ceb754a52c07143c69b053fe128b9e24060b99 
   
 clients/src/main/java/org/apache/kafka/clients/consumer/internals/Coordinator.java
  PRE-CREATION 
   
 clients/src/main/java/org/apache/kafka/clients/consumer/internals/FetchManager.java
  PRE-CREATION 
   
 clients/src/main/java/org/apache/kafka/clients/consumer/internals/Heartbeat.java
  ee0751e4949120d114202c2299d49612a89b9d97 
   
 clients/src/main/java/org/apache/kafka/clients/consumer/internals/SubscriptionState.java
  d41d3068c11d4b5c640467dc0ae1b7c20a8d128c 
   clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 
 7397e565fd865214529ffccadd4222d835ac8110 
   clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 
 122375c473bf73caf05299b9f5174c6b226ca863 
   
 clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 
 ed9c63a6679e3aaf83d19fde19268553a4c107c2 
   clients/src/main/java/org/apache/kafka/common/network/Selector.java 
 6baad9366a1975dbaba1786da91efeaa38533319 
   clients/src/main/java/org/apache/kafka/common/protocol/Errors.java 
 ad2171f5417c93194f5f234bdc7fdd0b8d59a8a8 
   clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java 
 083e7a39249ab56a73a014b106876244d619f189 
   clients/src/main/java/org/apache/kafka/common/requests/FetchResponse.java 
 e67c4c8332cb1dd3d9cde5de687df7760045dfe6 
   
 clients/src/main/java/org/apache/kafka/common/requests/HeartbeatResponse.java 
 0057496228feeeccbc0c009a42f5268fa2cb8611 
   
 clients/src/main/java/org/apache/kafka/common/requests/JoinGroupRequest.java 
 8c50e9be534c61ecf56106bf2b68cf678ea50d66 
   
 clients/src/main/java/org/apache/kafka/common/requests/JoinGroupResponse.java 
 52b1803d8b558c1eeb978ba8821496c7d3c20a6b 
   
 clients/src/main/java/org/apache/kafka/common/requests/ListOffsetResponse.java
  cfac47a4a05dc8a535595542d93e55237b7d1e93 
   
 clients/src/main/java/org/apache/kafka/common/requests/MetadataResponse.java 
 90f31413d7d80a06c0af359009cc271aa0c67be3 
   
 clients/src/main/java/org/apache/kafka/common/requests/OffsetCommitResponse.java
  4d3b9ececee4b4c0b50ba99da2ddbbb15f9cc08d 
   
 clients/src/main/java/org/apache/kafka/common/requests/OffsetFetchResponse.java
  edbed5880dc44fc178737a5e298c106a00f38443 
   clients/src/main/java/org/apache/kafka/common/requests/ProduceResponse.java 
 a00dcdf15d1c7bac7228be140647bd7d849deb9b 
   clients/src/test/java/org/apache/kafka/clients/MockClient.java 
 8f1a7a625e4eeafa44bbf9e5cff987de86c949be

[jira] [Commented] (KAFKA-1660) Ability to call close() with a timeout on the Java Kafka Producer.

2015-03-06 Thread Jiangjie Qin (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-1660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14351159#comment-14351159
 ] 

Jiangjie Qin commented on KAFKA-1660:
-

[~parth.brahmbhatt] Have you got a chance to work on a KIP for this?

 Ability to call close() with a timeout on the Java Kafka Producer. 
 ---

 Key: KAFKA-1660
 URL: https://issues.apache.org/jira/browse/KAFKA-1660
 Project: Kafka
  Issue Type: Improvement
  Components: clients, producer 
Affects Versions: 0.8.2.0
Reporter: Andrew Stein
Assignee: Parth Brahmbhatt
 Fix For: 0.8.3

 Attachments: KAFKA-1660.patch, KAFKA-1660_2015-02-17_16:41:19.patch, 
 KAFKA-1660_2015-03-02_10:41:49.patch


 I would like the ability to call {{close}} with a timeout on the Java 
 Client's KafkaProducer.
 h6. Workaround
 Currently, it is possible to ensure that {{close}} will return quickly by 
 first doing a {{future.get(timeout)}} on the last future produced on each 
 partition, but this means that the user has to define the partitions up front 
 at the time of {{send}} and track the returned {{future}}'s



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2003) Add upgrade tests

2015-03-06 Thread Guozhang Wang (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-2003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14351184#comment-14351184
 ] 

Guozhang Wang commented on KAFKA-2003:
--

Are these three tickets concerning the same testing scenario? KAFKA-1888, 
KAFKA-1898, KAFKA-2003.

 Add upgrade tests
 -

 Key: KAFKA-2003
 URL: https://issues.apache.org/jira/browse/KAFKA-2003
 Project: Kafka
  Issue Type: Improvement
Reporter: Gwen Shapira
Assignee: Ashish K Singh

 To test protocol changes, compatibility and upgrade process, we need a good 
 way to test different versions of the product together and to test end-to-end 
 upgrade process.
 For example, for 0.8.2 to 0.8.3 test we want to check:
 * Can we start a cluster with a mix of 0.8.2 and 0.8.3 brokers?
 * Can a cluster of 0.8.3 brokers bump the protocol level one broker at a time?
 * Can 0.8.2 clients run against a cluster of 0.8.3 brokers?
 There are probably more questions. But an automated framework that can test 
 those and report results will be a good start.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1660) Ability to call close() with a timeout on the Java Kafka Producer.

2015-03-06 Thread Parth Brahmbhatt (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-1660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14351179#comment-14351179
 ] 

Parth Brahmbhatt commented on KAFKA-1660:
-

No and I wont be able to do it for probably next couple of weeks. If this is a 
pressing issue and someone else wants to take over the jira, please feel free 
to do so.

 Ability to call close() with a timeout on the Java Kafka Producer. 
 ---

 Key: KAFKA-1660
 URL: https://issues.apache.org/jira/browse/KAFKA-1660
 Project: Kafka
  Issue Type: Improvement
  Components: clients, producer 
Affects Versions: 0.8.2.0
Reporter: Andrew Stein
Assignee: Parth Brahmbhatt
 Fix For: 0.8.3

 Attachments: KAFKA-1660.patch, KAFKA-1660_2015-02-17_16:41:19.patch, 
 KAFKA-1660_2015-03-02_10:41:49.patch


 I would like the ability to call {{close}} with a timeout on the Java 
 Client's KafkaProducer.
 h6. Workaround
 Currently, it is possible to ensure that {{close}} will return quickly by 
 first doing a {{future.get(timeout)}} on the last future produced on each 
 partition, but this means that the user has to define the partitions up front 
 at the time of {{send}} and track the returned {{future}}'s



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-527) Compression support does numerous byte copies

2015-03-06 Thread Guozhang Wang (JIRA)

[
https://issues.apache.org/jira/browse/KAFKA-527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14351177#comment-14351177
]

Guozhang Wang commented on KAFKA-527:
-

Thanks for the patch, this is very promising.

There are a couple of issues we want to resolve here:

1. ByteArrayOutputStream copies data upon overflowing and resizing.

2. Compressed stream needs one extra copy upon finishing reading / writing.

This patch is mainly aimed at #1 above, and I have uploaded a patch for
optimizing decompressed iterator, just as an example for resolving #2. In
addition, I think in the end we will deprecate ByeBufferMessageSet and move to
o.a.k.c.r.MemoryRecords, which will resolve both points above. We can discuss
whether we want to incorporate these patches into ByeBufferMessageSet now or
just wait for the migration and improve on o.a.k.c.r.MemoryRecords.

For example, today MemoryRecords's write pattern is only for appending messages
with pre-defined records batch size, and try to close the batch when its size
is approached; in ByteBufferMessageSet.create() we are given a set of messages
without a predicated batch size, but it is still possible to get the value from
the estimated compression ratio as we do in Compressor, such that in the worst
case only one or two buffer expansions (i.e. data copies) are needed. Just is
just an alternative to the linked-list buffers as proposed in this patch.

Compression support does numerous byte copies
-

Key: KAFKA-527
URL: https://issues.apache.org/jira/browse/KAFKA-527
Project: Kafka
Issue Type: Bug
Components: compression
Reporter: Jay Kreps
Assignee: Yasuhiro Matsuda
Priority: Critical
Attachments: KAFKA-527.message-copy.history, KAFKA-527.patch,
java.hprof.no-compression.txt, java.hprof.snappy.text

The data path for compressing or decompressing messages is extremely
inefficient. We do something like 7 (?) complete copies of the data, often
for simple things like adding a 4 byte size to the front. I am not sure how
this went by unnoticed.
This is likely the root cause of the performance issues we saw in doing bulk
recompression of data in mirror maker.
The mismatch between the InputStream and OutputStream interfaces and the
Message/MessageSet interfaces which are based on byte buffers is the cause of
many of these.

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-527) Compression support does numerous byte copies

2015-03-06 Thread Guozhang Wang (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14351180#comment-14351180
 ] 

Guozhang Wang commented on KAFKA-527:
-

Created reviewboard https://reviews.apache.org/r/31816/diff/
against branch origin/trunk

 Compression support does numerous byte copies
 -

 Key: KAFKA-527
 URL: https://issues.apache.org/jira/browse/KAFKA-527
 Project: Kafka
  Issue Type: Bug
  Components: compression
Reporter: Jay Kreps
Assignee: Yasuhiro Matsuda
Priority: Critical
 Attachments: KAFKA-527.message-copy.history, KAFKA-527.patch, 
 java.hprof.no-compression.txt, java.hprof.snappy.text


 The data path for compressing or decompressing messages is extremely 
 inefficient. We do something like 7 (?) complete copies of the data, often 
 for simple things like adding a 4 byte size to the front. I am not sure how 
 this went by unnoticed.
 This is likely the root cause of the performance issues we saw in doing bulk 
 recompression of data in mirror maker.
 The mismatch between the InputStream and OutputStream interfaces and the 
 Message/MessageSet interfaces which are based on byte buffers is the cause of 
 many of these.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2003) Add upgrade tests

2015-03-06 Thread Gwen Shapira (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-2003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14351199#comment-14351199
 ] 

Gwen Shapira commented on KAFKA-2003:
-

I think this one is a dupe of KAFKA-1888, but that KAFKA-1898 is different. 
(I.e full wire and API compatibility is not the same as an upgrade).

I'll let [~singhashish] and [~anigam] figure out who's doing what and which 
JIRA we are keeping :)
IMO, there's more then enough work for two people here.

 Add upgrade tests
 -

 Key: KAFKA-2003
 URL: https://issues.apache.org/jira/browse/KAFKA-2003
 Project: Kafka
  Issue Type: Improvement
Reporter: Gwen Shapira
Assignee: Ashish K Singh

 To test protocol changes, compatibility and upgrade process, we need a good 
 way to test different versions of the product together and to test end-to-end 
 upgrade process.
 For example, for 0.8.2 to 0.8.3 test we want to check:
 * Can we start a cluster with a mix of 0.8.2 and 0.8.3 brokers?
 * Can a cluster of 0.8.3 brokers bump the protocol level one broker at a time?
 * Can 0.8.2 clients run against a cluster of 0.8.3 brokers?
 There are probably more questions. But an automated framework that can test 
 those and report results will be a good start.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (KAFKA-1660) Ability to call close() with a timeout on the Java Kafka Producer.

2015-03-06 Thread Jiangjie Qin (JIRA)


 [ 
https://issues.apache.org/jira/browse/KAFKA-1660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jiangjie Qin reassigned KAFKA-1660:
---

Assignee: Jiangjie Qin  (was: Parth Brahmbhatt)

 Ability to call close() with a timeout on the Java Kafka Producer. 
 ---

 Key: KAFKA-1660
 URL: https://issues.apache.org/jira/browse/KAFKA-1660
 Project: Kafka
  Issue Type: Improvement
  Components: clients, producer 
Affects Versions: 0.8.2.0
Reporter: Andrew Stein
Assignee: Jiangjie Qin
 Fix For: 0.8.3

 Attachments: KAFKA-1660.patch, KAFKA-1660_2015-02-17_16:41:19.patch, 
 KAFKA-1660_2015-03-02_10:41:49.patch


 I would like the ability to call {{close}} with a timeout on the Java 
 Client's KafkaProducer.
 h6. Workaround
 Currently, it is possible to ensure that {{close}} will return quickly by 
 first doing a {{future.get(timeout)}} on the last future produced on each 
 partition, but this means that the user has to define the partitions up front 
 at the time of {{send}} and track the returned {{future}}'s



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-2007) update offsetrequest for more useful information we have on broker about partition

2015-03-06 Thread Joe Stein (JIRA)


 [ 
https://issues.apache.org/jira/browse/KAFKA-2007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Stein updated KAFKA-2007:
-
Description: 
this will need a KIP

via [~jkreps] in KIP-6 discussion about KAFKA-1694

The other information that would be really useful to get would be
information about partitions--how much data is in the partition, what are
the segment offsets, what is the log-end offset (i.e. last offset), what is
the compaction point, etc. I think that done right this would be the
successor to the very awkward OffsetRequest we have today.

This is not really blocking that ticket and could happen before/after and has a 
lot of other useful purposes and is important to get done so tracking it here 
in this JIRA.

  was:
this will need a KIP

via [~jkreps] in KIP-6 discussion about KAFKA-

The other information that would be really useful to get would be
information about partitions--how much data is in the partition, what are
the segment offsets, what is the log-end offset (i.e. last offset), what is
the compaction point, etc. I think that done right this would be the
successor to the very awkward OffsetRequest we have today.

This is not really blocking that ticket and could happen before/after and has a 
lot of other useful purposes and is important to get done so tracking it here 
in this JIRA.


 update offsetrequest for more useful information we have on broker about 
 partition
 --

 Key: KAFKA-2007
 URL: https://issues.apache.org/jira/browse/KAFKA-2007
 Project: Kafka
  Issue Type: Bug
Reporter: Joe Stein
 Fix For: 0.8.3


 this will need a KIP
 via [~jkreps] in KIP-6 discussion about KAFKA-1694
 The other information that would be really useful to get would be
 information about partitions--how much data is in the partition, what are
 the segment offsets, what is the log-end offset (i.e. last offset), what is
 the compaction point, etc. I think that done right this would be the
 successor to the very awkward OffsetRequest we have today.
 This is not really blocking that ticket and could happen before/after and has 
 a lot of other useful purposes and is important to get done so tracking it 
 here in this JIRA.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-2006) switch the broker server over to the new java protocol definitions

2015-03-06 Thread Joe Stein (JIRA)

Joe Stein created KAFKA-2006:


 Summary: switch the broker server over to the new java protocol 
definitions
 Key: KAFKA-2006
 URL: https://issues.apache.org/jira/browse/KAFKA-2006
 Project: Kafka
  Issue Type: Bug
Reporter: Joe Stein
Assignee: Andrii Biletskyi
Priority: Blocker
 Fix For: 0.8.3






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-2007) update offsetrequest for more useful information we have on broker about partition

2015-03-06 Thread Joe Stein (JIRA)

Joe Stein created KAFKA-2007:


 Summary: update offsetrequest for more useful information we have 
on broker about partition
 Key: KAFKA-2007
 URL: https://issues.apache.org/jira/browse/KAFKA-2007
 Project: Kafka
  Issue Type: Bug
Reporter: Joe Stein
 Fix For: 0.8.3


this will need a KIP

via [~jkreps] in KIP-6 discussion about KAFKA-

The other information that would be really useful to get would be
information about partitions--how much data is in the partition, what are
the segment offsets, what is the log-end offset (i.e. last offset), what is
the compaction point, etc. I think that done right this would be the
successor to the very awkward OffsetRequest we have today.

This is not really blocking that ticket and could happen before/after and has a 
lot of other useful purposes and is important to get done so tracking it here 
in this JIRA.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

58 matches

Mail list logo