[jira] [Commented] (KAFKA-1790) Remote controlled shutdown was removed
[ https://issues.apache.org/jira/browse/KAFKA-1790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14234496#comment-14234496 ] James Oliver commented on KAFKA-1790: - Thanks for the input, Joel. We'll keep that in mind as a possible solution. Jun, we are using Apache Mesos which uses resource isolation instead of VMs to schedule distributed workloads (Kafka in this case). I'm not sure if it's considered the exception anymore - it is a different way of doing things. For example, Google uses isolation instead of VMs for its datacenter computing. In this context, there is no VM but simply a sandbox (path on some mesos-slave). To bring down an instance, the entire process tree is sent a signal to shutdown. However, the shutdown timeouts in our scheduling framework are mostly out of our control - there's no good way to guarantee controlled shutdown occurs before the process is forced to terminate. In order to make sure it happens, we instead thought we might send a request directly to the broker to perform the controlled shutdown and, once everything is jolly, send the kill signal (which causes a restart). Since the remote controlled shutdown went away, we now think we might need to defer that responsibility to a wrapper script with a simple remote interface. Thoughts? Remote controlled shutdown was removed -- Key: KAFKA-1790 URL: https://issues.apache.org/jira/browse/KAFKA-1790 Project: Kafka Issue Type: Bug Components: core Affects Versions: 0.8.2 Reporter: James Oliver Assignee: James Oliver Priority: Blocker Fix For: 0.8.2 In core: kafka.admin.ShutdownBroker was removed, rendering remote controlled shutdowns impossible. A Kafka administrator needs to be able to perform a controlled shutdown without issuing a SIGTERM/SIGKILL. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1790) Remote controlled shutdown was removed
[ https://issues.apache.org/jira/browse/KAFKA-1790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14233313#comment-14233313 ] James Oliver commented on KAFKA-1790: - Hi [~junrao], the particular use case we were trying to satisfy is this: A Kafka cluster is deployed on cloud infrastructure. Operations wants to modify configurations and bounce boxes (keeping replication in mind so we don't corrupt any topics), but the tool they use to interface with the cloud infrastructure does not, at present, allow for sending signals to the underlying processes. We are looking for a way to upgrade binaries / modify configurations and have those changes reflected in our cluster by doing a remote SIGTERM to do a controlled bounce. Thinking about this in a more general sense, I don't think it's a good idea to rely on Kafka for this ability when the root issue is the inability to send a remote SIGTERM. That and the possible security issue is enough for me to call this a no-go. Remote controlled shutdown was removed -- Key: KAFKA-1790 URL: https://issues.apache.org/jira/browse/KAFKA-1790 Project: Kafka Issue Type: Bug Components: core Affects Versions: 0.8.2 Reporter: James Oliver Assignee: James Oliver Priority: Blocker Fix For: 0.8.2 In core: kafka.admin.ShutdownBroker was removed, rendering remote controlled shutdowns impossible. A Kafka administrator needs to be able to perform a controlled shutdown without issuing a SIGTERM/SIGKILL. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (KAFKA-1790) Remote controlled shutdown was removed
James Oliver created KAFKA-1790: --- Summary: Remote controlled shutdown was removed Key: KAFKA-1790 URL: https://issues.apache.org/jira/browse/KAFKA-1790 Project: Kafka Issue Type: Bug Components: core Affects Versions: 0.8.2 Reporter: James Oliver Assignee: James Oliver Priority: Blocker In core: kafka.admin.ShutdownBroker was removed, rendering remote controlled shutdowns impossible. A Kafka administrator needs to be able to perform a controlled shutdown without issuing a SIGTERM/SIGKILL. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1790) Remote controlled shutdown was removed
[ https://issues.apache.org/jira/browse/KAFKA-1790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14219883#comment-14219883 ] James Oliver commented on KAFKA-1790: - https://github.com/apache/kafka/commit/d0df33df3df314ccd42f700a592e8efc31e80f87#diff-db89a14f2c85068b1f0076d52e590d05 Remote controlled shutdown was removed -- Key: KAFKA-1790 URL: https://issues.apache.org/jira/browse/KAFKA-1790 Project: Kafka Issue Type: Bug Components: core Affects Versions: 0.8.2 Reporter: James Oliver Assignee: James Oliver Priority: Blocker In core: kafka.admin.ShutdownBroker was removed, rendering remote controlled shutdowns impossible. A Kafka administrator needs to be able to perform a controlled shutdown without issuing a SIGTERM/SIGKILL. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-1493) Use a well-documented LZ4 compression format and remove redundant LZ4HC option
[ https://issues.apache.org/jira/browse/KAFKA-1493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Oliver updated KAFKA-1493: Attachment: KAFKA-1493_2014-10-16_13:49:34.patch Use a well-documented LZ4 compression format and remove redundant LZ4HC option -- Key: KAFKA-1493 URL: https://issues.apache.org/jira/browse/KAFKA-1493 Project: Kafka Issue Type: Improvement Affects Versions: 0.8.2 Reporter: James Oliver Assignee: Ivan Lyutov Priority: Blocker Fix For: 0.8.2 Attachments: KAFKA-1493.patch, KAFKA-1493.patch, KAFKA-1493_2014-10-16_13:49:34.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1493) Use a well-documented LZ4 compression format and remove redundant LZ4HC option
[ https://issues.apache.org/jira/browse/KAFKA-1493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14174217#comment-14174217 ] James Oliver commented on KAFKA-1493: - Updated reviewboard https://reviews.apache.org/r/26658/diff/ against branch origin/trunk Use a well-documented LZ4 compression format and remove redundant LZ4HC option -- Key: KAFKA-1493 URL: https://issues.apache.org/jira/browse/KAFKA-1493 Project: Kafka Issue Type: Improvement Affects Versions: 0.8.2 Reporter: James Oliver Assignee: Ivan Lyutov Priority: Blocker Fix For: 0.8.2 Attachments: KAFKA-1493.patch, KAFKA-1493.patch, KAFKA-1493_2014-10-16_13:49:34.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (KAFKA-1493) Use a well-documented LZ4 compression format and remove redundant LZ4HC option
[ https://issues.apache.org/jira/browse/KAFKA-1493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Oliver reassigned KAFKA-1493: --- Assignee: James Oliver (was: Ivan Lyutov) Use a well-documented LZ4 compression format and remove redundant LZ4HC option -- Key: KAFKA-1493 URL: https://issues.apache.org/jira/browse/KAFKA-1493 Project: Kafka Issue Type: Improvement Affects Versions: 0.8.2 Reporter: James Oliver Assignee: James Oliver Priority: Blocker Fix For: 0.8.2 Attachments: KAFKA-1493.patch, KAFKA-1493.patch, KAFKA-1493_2014-10-16_13:49:34.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1493) Use a well-documented LZ4 compression format and remove redundant LZ4HC option
[ https://issues.apache.org/jira/browse/KAFKA-1493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14174356#comment-14174356 ] James Oliver commented on KAFKA-1493: - Jun, My pleasure. The TODOs are parts of the specification that are unimplemented, but are not required. I left them in there as hints if/when the spec is contributed back to lz4-java. The validation routines will disallow the use of any portion of the spec that is unimplemented, but it's totally usable. What the spec can do - compress decompress messages using 64kb/256kb/1mb/4mb blockSize (64kb by default) with optional block checksums (disabled by default) What the spec cannot do - decompress messages compressed by an implementation supporting some of the missing features. If this were to occur, a RuntimeException with detailed information will be thrown. Use a well-documented LZ4 compression format and remove redundant LZ4HC option -- Key: KAFKA-1493 URL: https://issues.apache.org/jira/browse/KAFKA-1493 Project: Kafka Issue Type: Improvement Affects Versions: 0.8.2 Reporter: James Oliver Assignee: James Oliver Priority: Blocker Fix For: 0.8.2 Attachments: KAFKA-1493.patch, KAFKA-1493.patch, KAFKA-1493_2014-10-16_13:49:34.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (KAFKA-1493) Use a well-documented LZ4 compression format and remove redundant LZ4HC option
[ https://issues.apache.org/jira/browse/KAFKA-1493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14174356#comment-14174356 ] James Oliver edited comment on KAFKA-1493 at 10/16/14 10:23 PM: Jun, My pleasure. The TODOs are parts of the specification that are unimplemented, but are not required. I left them in there as hints if/when the spec is contributed back to lz4-java. The validation routines will disallow the use of any portion of the spec that is unimplemented, but it's totally usable. What the spec can do - compress decompress messages using 64kb/256kb/1mb/4mb blockSize (64kb by default) with optional block checksums (disabled by default) What the spec cannot do - decompress messages compressed by a more advanced implementation, using one or more of the missing features. If this were to occur, a RuntimeException with detailed information will be thrown. was (Author: joliver): Jun, My pleasure. The TODOs are parts of the specification that are unimplemented, but are not required. I left them in there as hints if/when the spec is contributed back to lz4-java. The validation routines will disallow the use of any portion of the spec that is unimplemented, but it's totally usable. What the spec can do - compress decompress messages using 64kb/256kb/1mb/4mb blockSize (64kb by default) with optional block checksums (disabled by default) What the spec cannot do - decompress messages compressed by an implementation supporting some of the missing features. If this were to occur, a RuntimeException with detailed information will be thrown. Use a well-documented LZ4 compression format and remove redundant LZ4HC option -- Key: KAFKA-1493 URL: https://issues.apache.org/jira/browse/KAFKA-1493 Project: Kafka Issue Type: Improvement Affects Versions: 0.8.2 Reporter: James Oliver Assignee: James Oliver Priority: Blocker Fix For: 0.8.2 Attachments: KAFKA-1493.patch, KAFKA-1493.patch, KAFKA-1493_2014-10-16_13:49:34.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (KAFKA-1493) Use a well-documented LZ4 compression format and remove redundant LZ4HC option
[ https://issues.apache.org/jira/browse/KAFKA-1493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14174356#comment-14174356 ] James Oliver edited comment on KAFKA-1493 at 10/16/14 10:30 PM: Jun, My pleasure. The TODOs are parts of the specification that are unimplemented, but are not required. I left them in there as hints if/when the spec is contributed back to lz4-java. The validation routines will disallow the use of any portion of the spec that is unimplemented, but it's totally usable. What the spec can do - compress decompress messages using 64kb/256kb/1mb/4mb blockSize (64kb by default) with optional block checksums (disabled by default) What the spec cannot do - decompress messages compressed by a more advanced implementation, using one or more of the missing features. If this were to occur, a RuntimeException with detailed information will be thrown. EDIT: I can take out the TODOs if you think it causes confusion was (Author: joliver): Jun, My pleasure. The TODOs are parts of the specification that are unimplemented, but are not required. I left them in there as hints if/when the spec is contributed back to lz4-java. The validation routines will disallow the use of any portion of the spec that is unimplemented, but it's totally usable. What the spec can do - compress decompress messages using 64kb/256kb/1mb/4mb blockSize (64kb by default) with optional block checksums (disabled by default) What the spec cannot do - decompress messages compressed by a more advanced implementation, using one or more of the missing features. If this were to occur, a RuntimeException with detailed information will be thrown. Use a well-documented LZ4 compression format and remove redundant LZ4HC option -- Key: KAFKA-1493 URL: https://issues.apache.org/jira/browse/KAFKA-1493 Project: Kafka Issue Type: Improvement Affects Versions: 0.8.2 Reporter: James Oliver Assignee: James Oliver Priority: Blocker Fix For: 0.8.2 Attachments: KAFKA-1493.patch, KAFKA-1493.patch, KAFKA-1493_2014-10-16_13:49:34.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1493) Use a well-documented LZ4 compression format and remove redundant LZ4HC option
[ https://issues.apache.org/jira/browse/KAFKA-1493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14174716#comment-14174716 ] James Oliver commented on KAFKA-1493: - Updated reviewboard https://reviews.apache.org/r/26658/diff/ against branch origin/trunk Use a well-documented LZ4 compression format and remove redundant LZ4HC option -- Key: KAFKA-1493 URL: https://issues.apache.org/jira/browse/KAFKA-1493 Project: Kafka Issue Type: Improvement Affects Versions: 0.8.2 Reporter: James Oliver Assignee: James Oliver Priority: Blocker Fix For: 0.8.2 Attachments: KAFKA-1493.patch, KAFKA-1493.patch, KAFKA-1493_2014-10-16_13:49:34.patch, KAFKA-1493_2014-10-16_21:25:23.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-1493) Use a well-documented LZ4 compression format and remove redundant LZ4HC option
[ https://issues.apache.org/jira/browse/KAFKA-1493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Oliver updated KAFKA-1493: Attachment: KAFKA-1493_2014-10-16_21:25:23.patch Use a well-documented LZ4 compression format and remove redundant LZ4HC option -- Key: KAFKA-1493 URL: https://issues.apache.org/jira/browse/KAFKA-1493 Project: Kafka Issue Type: Improvement Affects Versions: 0.8.2 Reporter: James Oliver Assignee: James Oliver Priority: Blocker Fix For: 0.8.2 Attachments: KAFKA-1493.patch, KAFKA-1493.patch, KAFKA-1493_2014-10-16_13:49:34.patch, KAFKA-1493_2014-10-16_21:25:23.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1493) Use a well-documented LZ4 compression format and remove redundant LZ4HC option
[ https://issues.apache.org/jira/browse/KAFKA-1493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14171082#comment-14171082 ] James Oliver commented on KAFKA-1493: - Sorry to not be more clear - I fixed a few spots related to the removal of the LZ4HC option, but left the I/O streams in Ivan's patch alone. Since I didn't have permissions to update Ivan's reviewboard, I created a new review. 1. This looks like Ivan's interpretation of the lz4-java block stream format. 2. We should use neither - the lz4-java impl was used previously (KAFKA-1456). Review by the community produced this issue. We need a real implementation of http://fastcompression.blogspot.com/2013/04/lz4-streaming-format-final.html Use a well-documented LZ4 compression format and remove redundant LZ4HC option -- Key: KAFKA-1493 URL: https://issues.apache.org/jira/browse/KAFKA-1493 Project: Kafka Issue Type: Improvement Affects Versions: 0.8.2 Reporter: James Oliver Assignee: Ivan Lyutov Priority: Blocker Fix For: 0.8.2 Attachments: KAFKA-1493.patch, KAFKA-1493.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1493) Use a well-documented LZ4 compression format and remove redundant LZ4HC option
[ https://issues.apache.org/jira/browse/KAFKA-1493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14171748#comment-14171748 ] James Oliver commented on KAFKA-1493: - I implemented the OutputStream today. If I can't get the InputStream done and tested before I leave for vacation Thursday, IMO we should take it out. Use a well-documented LZ4 compression format and remove redundant LZ4HC option -- Key: KAFKA-1493 URL: https://issues.apache.org/jira/browse/KAFKA-1493 Project: Kafka Issue Type: Improvement Affects Versions: 0.8.2 Reporter: James Oliver Assignee: Ivan Lyutov Priority: Blocker Fix For: 0.8.2 Attachments: KAFKA-1493.patch, KAFKA-1493.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1493) Use a well-documented LZ4 compression format and remove redundant LZ4HC option
[ https://issues.apache.org/jira/browse/KAFKA-1493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14169616#comment-14169616 ] James Oliver commented on KAFKA-1493: - Sure, I'll take a look at it now. Use a well-documented LZ4 compression format and remove redundant LZ4HC option -- Key: KAFKA-1493 URL: https://issues.apache.org/jira/browse/KAFKA-1493 Project: Kafka Issue Type: Improvement Affects Versions: 0.8.2 Reporter: James Oliver Assignee: Ivan Lyutov Priority: Blocker Fix For: 0.8.2 Attachments: KAFKA-1493.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (KAFKA-1493) Use a well-documented LZ4 compression format and remove redundant LZ4HC option
[ https://issues.apache.org/jira/browse/KAFKA-1493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14169616#comment-14169616 ] James Oliver edited comment on KAFKA-1493 at 10/13/14 7:41 PM: --- The i/o streams provided in the patch are nearly identical to the i/o streams in lz4-java. As it stands, I don't think it buys us much. Other than that, looks good, just a few spots still referring to lz4hc (I think KAFKA-1471 changes were applied to trunk after this patch was created). This details the formal specification http://fastcompression.blogspot.com/2013/04/lz4-streaming-format-final.html Here's an example C++ implementation of that format: https://github.com/t-mat/lz4mt/blob/master/src/lz4mt.cpp We still need a Java implementation was (Author: joliver): Sure, I'll take a look at it now. Use a well-documented LZ4 compression format and remove redundant LZ4HC option -- Key: KAFKA-1493 URL: https://issues.apache.org/jira/browse/KAFKA-1493 Project: Kafka Issue Type: Improvement Affects Versions: 0.8.2 Reporter: James Oliver Assignee: Ivan Lyutov Priority: Blocker Fix For: 0.8.2 Attachments: KAFKA-1493.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-1493) Use a well-documented LZ4 compression format and remove redundant LZ4HC option
[ https://issues.apache.org/jira/browse/KAFKA-1493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Oliver updated KAFKA-1493: Attachment: KAFKA-1493.patch Use a well-documented LZ4 compression format and remove redundant LZ4HC option -- Key: KAFKA-1493 URL: https://issues.apache.org/jira/browse/KAFKA-1493 Project: Kafka Issue Type: Improvement Affects Versions: 0.8.2 Reporter: James Oliver Assignee: Ivan Lyutov Priority: Blocker Fix For: 0.8.2 Attachments: KAFKA-1493.patch, KAFKA-1493.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1493) Use a well-documented LZ4 compression format and remove redundant LZ4HC option
[ https://issues.apache.org/jira/browse/KAFKA-1493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14131758#comment-14131758 ] James Oliver commented on KAFKA-1493: - I have today to work on this, I will see how far I can get. Use a well-documented LZ4 compression format and remove redundant LZ4HC option -- Key: KAFKA-1493 URL: https://issues.apache.org/jira/browse/KAFKA-1493 Project: Kafka Issue Type: Improvement Affects Versions: 0.8.2 Reporter: James Oliver Priority: Blocker Fix For: 0.8.2 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1489) Global threshold on data retention size
[ https://issues.apache.org/jira/browse/KAFKA-1489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14082503#comment-14082503 ] James Oliver commented on KAFKA-1489: - +1 for Steven's strategy - I could see this being immediately beneficial to end users and not overly difficult to configure. FYI, I won't get to this for some time. Global threshold on data retention size --- Key: KAFKA-1489 URL: https://issues.apache.org/jira/browse/KAFKA-1489 Project: Kafka Issue Type: New Feature Components: log Affects Versions: 0.8.1.1 Reporter: Andras Sereny Assignee: Jay Kreps Labels: newbie Currently, Kafka has per topic settings to control the size of one single log (log.retention.bytes). With lots of topics of different volume and as they grow in number, it could become tedious to maintain topic level settings applying to a single log. Often, a chunk of disk space is dedicated to Kafka that hosts all logs stored, so it'd make sense to have a configurable threshold to control how much space *all* data in Kafka can take up. See also: http://mail-archives.apache.org/mod_mbox/kafka-users/201406.mbox/browser http://mail-archives.apache.org/mod_mbox/kafka-users/201311.mbox/%3c20131107015125.gc9...@jkoshy-ld.linkedin.biz%3E -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (KAFKA-1493) Use a well-documented LZ4 compression format and remove redundant LZ4HC option
[ https://issues.apache.org/jira/browse/KAFKA-1493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14032832#comment-14032832 ] James Oliver commented on KAFKA-1493: - Snappy's block (default size 32kb) compression format is this: snappy codec header: 8-byte magic header, version [4-byte integer], min compatible version [4-byte integer] compressed block 1: compressed data size [4-byte integer], compressed data compressed block 2 ... Notable limitations: no checksum If I understand the proposed format correctly, this is what you're suggesting: uncompressed data size [n-byte varint], compressed data While I would expect compressing an entire message as a single block would provide a better compression ratio than compressing smaller chunks, doing so for larger messages is going to cause serious performance problems. Use a well-documented LZ4 compression format and remove redundant LZ4HC option -- Key: KAFKA-1493 URL: https://issues.apache.org/jira/browse/KAFKA-1493 Project: Kafka Issue Type: Improvement Reporter: James Oliver Fix For: 0.8.2 -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Comment Edited] (KAFKA-1493) Use a well-documented LZ4 compression format and remove redundant LZ4HC option
[ https://issues.apache.org/jira/browse/KAFKA-1493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14032832#comment-14032832 ] James Oliver edited comment on KAFKA-1493 at 6/16/14 9:23 PM: -- Snappy's block (default size 32kb) compression format is this: snappy codec header: 8-byte magic header, version [4-byte integer], min compatible version [4-byte integer] compressed block 1: compressed data size [4-byte integer], compressed data compressed block 2 ... Notable limitations: no checksum If I understand the proposed format correctly, this is what you're suggesting: uncompressed data size [n-byte varint], compressed data I would expect compressing an entire message as a single block would provide a better compression ratio than compressing smaller chunks. We would need to allocate enough heap to fit both the uncompressed and compressed message in memory. If this is ok then it would drastically simplify things. was (Author: joliver): Snappy's block (default size 32kb) compression format is this: snappy codec header: 8-byte magic header, version [4-byte integer], min compatible version [4-byte integer] compressed block 1: compressed data size [4-byte integer], compressed data compressed block 2 ... Notable limitations: no checksum If I understand the proposed format correctly, this is what you're suggesting: uncompressed data size [n-byte varint], compressed data While I would expect compressing an entire message as a single block would provide a better compression ratio than compressing smaller chunks, doing so for larger messages is going to cause serious performance problems. Use a well-documented LZ4 compression format and remove redundant LZ4HC option -- Key: KAFKA-1493 URL: https://issues.apache.org/jira/browse/KAFKA-1493 Project: Kafka Issue Type: Improvement Reporter: James Oliver Fix For: 0.8.2 -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (KAFKA-1493) Use a well-documented LZ4 compression format and remove redundant LZ4HC option
[ https://issues.apache.org/jira/browse/KAFKA-1493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14033274#comment-14033274 ] James Oliver commented on KAFKA-1493: - I agree that storing the uncompressed length as a varint makes logical sense for allocating the required heap space IFF the entire uncompressed message is destined for the heap. Otherwise, this strategy introduces unnecessary heap requirements. I also agree that the checksum doesn't buy us much... IMO LZ4 is mature enough to not worry about distortion, and as you mentioned we already checksum the compressed message to verify accurate transmission. Looks like the LZ4 Java path doesn't pass that default blockSize to the underlying stream, which should be changed (if we go with the LZ4Block streams). That being said, the ultra-small block size is robbing performance...we should consider bumping it up to something in the 32-64kb range to improve our compression ratio and reduce block overhead. We could just compress the entire message as [~alb...@stonethree.com] mentioned and document the heap requirements, but it doesn't look like any of the other compression codecs do so and I'm hesitant to change the way LZ4 would work... partially implementing https://docs.google.com/document/d/1gZbUoLw5hRzJ5Q71oPRN6TO4cRMTZur60qip-TE7BhQ/edit?pli=1 might still be our best option. Use a well-documented LZ4 compression format and remove redundant LZ4HC option -- Key: KAFKA-1493 URL: https://issues.apache.org/jira/browse/KAFKA-1493 Project: Kafka Issue Type: Improvement Reporter: James Oliver Fix For: 0.8.2 -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (KAFKA-1493) Use a well-documented LZ4 compression format and remove redundant LZ4HC option
James Oliver created KAFKA-1493: --- Summary: Use a well-documented LZ4 compression format and remove redundant LZ4HC option Key: KAFKA-1493 URL: https://issues.apache.org/jira/browse/KAFKA-1493 Project: Kafka Issue Type: Improvement Reporter: James Oliver Fix For: 0.8.2 -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (KAFKA-1456) Add LZ4 and LZ4C as a compression codec
[ https://issues.apache.org/jira/browse/KAFKA-1456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14030858#comment-14030858 ] James Oliver commented on KAFKA-1456: - Created KAFKA-1493 Add LZ4 and LZ4C as a compression codec --- Key: KAFKA-1456 URL: https://issues.apache.org/jira/browse/KAFKA-1456 Project: Kafka Issue Type: Improvement Reporter: Joe Stein Labels: newbie Fix For: 0.8.2 Attachments: KAFKA-1456.patch, KAFKA-1456_2014-05-19_15:01:10.patch, KAFKA-1456_2014-05-19_16:39:01.patch, KAFKA-1456_2014-05-19_18:19:32.patch, KAFKA-1456_2014-05-19_23:24:27.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (KAFKA-1456) Add LZ4 and LZ4C as a compression codec
[ https://issues.apache.org/jira/browse/KAFKA-1456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14029284#comment-14029284 ] James Oliver commented on KAFKA-1456: - Sure, I'll open another ticket to address the outstanding issues this afternoon. [~joestein] Did the KAFKA-1471 patch get committed to trunk? If not I can roll those changes into the next ticket as well. Add LZ4 and LZ4C as a compression codec --- Key: KAFKA-1456 URL: https://issues.apache.org/jira/browse/KAFKA-1456 Project: Kafka Issue Type: Improvement Reporter: Joe Stein Labels: newbie Fix For: 0.8.2 Attachments: KAFKA-1456.patch, KAFKA-1456_2014-05-19_15:01:10.patch, KAFKA-1456_2014-05-19_16:39:01.patch, KAFKA-1456_2014-05-19_18:19:32.patch, KAFKA-1456_2014-05-19_23:24:27.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (KAFKA-1489) Global threshold on data retention size
[ https://issues.apache.org/jira/browse/KAFKA-1489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14028159#comment-14028159 ] James Oliver commented on KAFKA-1489: - I'll work this one if there are no objections. Global threshold on data retention size --- Key: KAFKA-1489 URL: https://issues.apache.org/jira/browse/KAFKA-1489 Project: Kafka Issue Type: Bug Components: log Affects Versions: 0.8.1.1 Reporter: Andras Sereny Assignee: Jay Kreps Labels: newbie Currently, Kafka has per topic settings to control the size of one single log (log.retention.bytes). With lots of topics of different volume and as they grow in number, it could become tedious to maintain topic level settings applying to a single log. Often, a chunk of disk space is dedicated to Kafka that hosts all logs stored, so it'd make sense to have a configurable threshold to control how much space *all* data in Kafka can take up. See also: http://mail-archives.apache.org/mod_mbox/kafka-users/201406.mbox/browser http://mail-archives.apache.org/mod_mbox/kafka-users/201311.mbox/%3c20131107015125.gc9...@jkoshy-ld.linkedin.biz%3E -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (KAFKA-1456) Add LZ4 and LZ4C as a compression codec
[ https://issues.apache.org/jira/browse/KAFKA-1456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14017243#comment-14017243 ] James Oliver commented on KAFKA-1456: - The Java client uses reflection to invoke the codec so there's no runtime dependency, and the Scala code only imports the codec inside of a case clause. We should be OK on that note. [~stephanl] Yep, you're totally correct regarding the enum...there should only be 1 with the capability of passing custom configuration. Does a Java implementation for the well-documented streaming format exist? Or would we need to implement it ourselves? Add LZ4 and LZ4C as a compression codec --- Key: KAFKA-1456 URL: https://issues.apache.org/jira/browse/KAFKA-1456 Project: Kafka Issue Type: Improvement Reporter: Joe Stein Labels: newbie Fix For: 0.8.2 Attachments: KAFKA-1456.patch, KAFKA-1456_2014-05-19_15:01:10.patch, KAFKA-1456_2014-05-19_16:39:01.patch, KAFKA-1456_2014-05-19_18:19:32.patch, KAFKA-1456_2014-05-19_23:24:27.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (KAFKA-1471) Add Producer Unit Tests for LZ4 and LZ4HC compression
James Oliver created KAFKA-1471: --- Summary: Add Producer Unit Tests for LZ4 and LZ4HC compression Key: KAFKA-1471 URL: https://issues.apache.org/jira/browse/KAFKA-1471 Project: Kafka Issue Type: Sub-task Reporter: James Oliver Attachments: KAFKA-1471.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (KAFKA-1471) Add Producer Unit Tests for LZ4 and LZ4HC compression
[ https://issues.apache.org/jira/browse/KAFKA-1471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14010167#comment-14010167 ] James Oliver commented on KAFKA-1471: - Created reviewboard against branch trunk Add Producer Unit Tests for LZ4 and LZ4HC compression - Key: KAFKA-1471 URL: https://issues.apache.org/jira/browse/KAFKA-1471 Project: Kafka Issue Type: Sub-task Reporter: James Oliver Fix For: 0.8.2 Attachments: KAFKA-1471.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (KAFKA-1471) Add Producer Unit Tests for LZ4 and LZ4HC compression
[ https://issues.apache.org/jira/browse/KAFKA-1471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Oliver updated KAFKA-1471: Attachment: KAFKA-1471.patch Add Producer Unit Tests for LZ4 and LZ4HC compression - Key: KAFKA-1471 URL: https://issues.apache.org/jira/browse/KAFKA-1471 Project: Kafka Issue Type: Sub-task Reporter: James Oliver Fix For: 0.8.2 Attachments: KAFKA-1471.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (KAFKA-1471) Add Producer Unit Tests for LZ4 and LZ4HC compression
[ https://issues.apache.org/jira/browse/KAFKA-1471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14010168#comment-14010168 ] James Oliver commented on KAFKA-1471: - Created reviewboard https://reviews.apache.org/r/21938/diff/ against branch trunk Add Producer Unit Tests for LZ4 and LZ4HC compression - Key: KAFKA-1471 URL: https://issues.apache.org/jira/browse/KAFKA-1471 Project: Kafka Issue Type: Sub-task Reporter: James Oliver Fix For: 0.8.2 Attachments: KAFKA-1471.patch, KAFKA-1471.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (KAFKA-1471) Add Producer Unit Tests for LZ4 and LZ4HC compression
[ https://issues.apache.org/jira/browse/KAFKA-1471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Oliver updated KAFKA-1471: Attachment: KAFKA-1471.patch Add Producer Unit Tests for LZ4 and LZ4HC compression - Key: KAFKA-1471 URL: https://issues.apache.org/jira/browse/KAFKA-1471 Project: Kafka Issue Type: Sub-task Reporter: James Oliver Fix For: 0.8.2 Attachments: KAFKA-1471.patch, KAFKA-1471.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (KAFKA-1432) Make num.producerThreads configurable on new MirrrorMaker
[ https://issues.apache.org/jira/browse/KAFKA-1432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14002843#comment-14002843 ] James Oliver commented on KAFKA-1432: - Updated reviewboard https://reviews.apache.org/r/21663/diff/ against branch trunk Make num.producerThreads configurable on new MirrrorMaker - Key: KAFKA-1432 URL: https://issues.apache.org/jira/browse/KAFKA-1432 Project: Kafka Issue Type: Bug Reporter: Guozhang Wang Assignee: Guozhang Wang Fix For: 0.8.2 Attachments: KAFKA-1432.patch, KAFKA-1432.patch, KAFKA-1432.patch, KAFKA-1432.patch, KAFKA-1432_2014-05-05_10:31:42.patch, KAFKA-1432_2014-05-05_14:33:01.patch, KAFKA-1432_2014-05-05_15:24:08.patch, KAFKA-1432_2014-05-06_16:05:28.patch, KAFKA-1432_2014-05-19_23:13:01.patch Originally we make the num.producerThreads coupled with num.consumerThreads since we thought with the new producer, a single or couple of instances are sufficient for throughput. However, with compression turned on, which is now executed in the caller thread we still need configurable number of producer threads. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (KAFKA-1432) Make num.producerThreads configurable on new MirrrorMaker
[ https://issues.apache.org/jira/browse/KAFKA-1432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Oliver updated KAFKA-1432: Attachment: KAFKA-1432_2014-05-19_23:13:01.patch Make num.producerThreads configurable on new MirrrorMaker - Key: KAFKA-1432 URL: https://issues.apache.org/jira/browse/KAFKA-1432 Project: Kafka Issue Type: Bug Reporter: Guozhang Wang Assignee: Guozhang Wang Fix For: 0.8.2 Attachments: KAFKA-1432.patch, KAFKA-1432.patch, KAFKA-1432.patch, KAFKA-1432.patch, KAFKA-1432_2014-05-05_10:31:42.patch, KAFKA-1432_2014-05-05_14:33:01.patch, KAFKA-1432_2014-05-05_15:24:08.patch, KAFKA-1432_2014-05-06_16:05:28.patch, KAFKA-1432_2014-05-19_23:13:01.patch Originally we make the num.producerThreads coupled with num.consumerThreads since we thought with the new producer, a single or couple of instances are sufficient for throughput. However, with compression turned on, which is now executed in the caller thread we still need configurable number of producer threads. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (KAFKA-1432) Make num.producerThreads configurable on new MirrrorMaker
[ https://issues.apache.org/jira/browse/KAFKA-1432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Oliver updated KAFKA-1432: Attachment: (was: KAFKA-1432_2014-05-19_23:13:01.patch) Make num.producerThreads configurable on new MirrrorMaker - Key: KAFKA-1432 URL: https://issues.apache.org/jira/browse/KAFKA-1432 Project: Kafka Issue Type: Bug Reporter: Guozhang Wang Assignee: Guozhang Wang Fix For: 0.8.2 Attachments: KAFKA-1432.patch, KAFKA-1432.patch, KAFKA-1432.patch, KAFKA-1432.patch, KAFKA-1432_2014-05-05_10:31:42.patch, KAFKA-1432_2014-05-05_14:33:01.patch, KAFKA-1432_2014-05-05_15:24:08.patch, KAFKA-1432_2014-05-06_16:05:28.patch Originally we make the num.producerThreads coupled with num.consumerThreads since we thought with the new producer, a single or couple of instances are sufficient for throughput. However, with compression turned on, which is now executed in the caller thread we still need configurable number of producer threads. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Issue Comment Deleted] (KAFKA-1432) Make num.producerThreads configurable on new MirrrorMaker
[ https://issues.apache.org/jira/browse/KAFKA-1432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Oliver updated KAFKA-1432: Comment: was deleted (was: Updated reviewboard https://reviews.apache.org/r/21663/diff/ against branch trunk) Make num.producerThreads configurable on new MirrrorMaker - Key: KAFKA-1432 URL: https://issues.apache.org/jira/browse/KAFKA-1432 Project: Kafka Issue Type: Bug Reporter: Guozhang Wang Assignee: Guozhang Wang Fix For: 0.8.2 Attachments: KAFKA-1432.patch, KAFKA-1432.patch, KAFKA-1432.patch, KAFKA-1432.patch, KAFKA-1432_2014-05-05_10:31:42.patch, KAFKA-1432_2014-05-05_14:33:01.patch, KAFKA-1432_2014-05-05_15:24:08.patch, KAFKA-1432_2014-05-06_16:05:28.patch Originally we make the num.producerThreads coupled with num.consumerThreads since we thought with the new producer, a single or couple of instances are sufficient for throughput. However, with compression turned on, which is now executed in the caller thread we still need configurable number of producer threads. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (KAFKA-1456) Add LZ4 and LZ4C as a compression codec
[ https://issues.apache.org/jira/browse/KAFKA-1456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14002856#comment-14002856 ] James Oliver commented on KAFKA-1456: - Updated reviewboard https://reviews.apache.org/r/21663/diff/ against branch trunk Add LZ4 and LZ4C as a compression codec --- Key: KAFKA-1456 URL: https://issues.apache.org/jira/browse/KAFKA-1456 Project: Kafka Issue Type: Improvement Reporter: Joe Stein Labels: newbie Fix For: 0.8.2 Attachments: KAFKA-1456.patch, KAFKA-1456_2014-05-19_15:01:10.patch, KAFKA-1456_2014-05-19_16:39:01.patch, KAFKA-1456_2014-05-19_18:19:32.patch, KAFKA-1456_2014-05-19_23:24:27.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (KAFKA-1456) Add LZ4 and LZ4C as a compression codec
[ https://issues.apache.org/jira/browse/KAFKA-1456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Oliver updated KAFKA-1456: Attachment: KAFKA-1456_2014-05-19_23:24:27.patch Add LZ4 and LZ4C as a compression codec --- Key: KAFKA-1456 URL: https://issues.apache.org/jira/browse/KAFKA-1456 Project: Kafka Issue Type: Improvement Reporter: Joe Stein Labels: newbie Fix For: 0.8.2 Attachments: KAFKA-1456.patch, KAFKA-1456_2014-05-19_15:01:10.patch, KAFKA-1456_2014-05-19_16:39:01.patch, KAFKA-1456_2014-05-19_18:19:32.patch, KAFKA-1456_2014-05-19_23:24:27.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (KAFKA-1456) Add LZ4 and LZ4C as a compression codec
[ https://issues.apache.org/jira/browse/KAFKA-1456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Oliver updated KAFKA-1456: Attachment: KAFKA-1456.patch Add LZ4 and LZ4C as a compression codec --- Key: KAFKA-1456 URL: https://issues.apache.org/jira/browse/KAFKA-1456 Project: Kafka Issue Type: Improvement Reporter: Joe Stein Labels: newbie Fix For: 0.8.2 Attachments: KAFKA-1456.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (KAFKA-1456) Add LZ4 and LZ4C as a compression codec
[ https://issues.apache.org/jira/browse/KAFKA-1456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14002418#comment-14002418 ] James Oliver commented on KAFKA-1456: - Created reviewboard against branch origin/trunk Add LZ4 and LZ4C as a compression codec --- Key: KAFKA-1456 URL: https://issues.apache.org/jira/browse/KAFKA-1456 Project: Kafka Issue Type: Improvement Reporter: Joe Stein Labels: newbie Fix For: 0.8.2 Attachments: KAFKA-1456.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (KAFKA-1456) Add LZ4 and LZ4C as a compression codec
[ https://issues.apache.org/jira/browse/KAFKA-1456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14002424#comment-14002424 ] James Oliver commented on KAFKA-1456: - Created reviewboard https://reviews.apache.org/r/21663/diff/ against branch origin/trunk Add LZ4 and LZ4C as a compression codec --- Key: KAFKA-1456 URL: https://issues.apache.org/jira/browse/KAFKA-1456 Project: Kafka Issue Type: Improvement Reporter: Joe Stein Labels: newbie Fix For: 0.8.2 Attachments: KAFKA-1456.patch, KAFKA-1456.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (KAFKA-1456) Add LZ4 and LZ4C as a compression codec
[ https://issues.apache.org/jira/browse/KAFKA-1456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Oliver updated KAFKA-1456: Attachment: KAFKA-1456.patch Add LZ4 and LZ4C as a compression codec --- Key: KAFKA-1456 URL: https://issues.apache.org/jira/browse/KAFKA-1456 Project: Kafka Issue Type: Improvement Reporter: Joe Stein Labels: newbie Fix For: 0.8.2 Attachments: KAFKA-1456.patch, KAFKA-1456.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Issue Comment Deleted] (KAFKA-1456) Add LZ4 and LZ4C as a compression codec
[ https://issues.apache.org/jira/browse/KAFKA-1456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Oliver updated KAFKA-1456: Comment: was deleted (was: Created reviewboard against branch origin/trunk) Add LZ4 and LZ4C as a compression codec --- Key: KAFKA-1456 URL: https://issues.apache.org/jira/browse/KAFKA-1456 Project: Kafka Issue Type: Improvement Reporter: Joe Stein Labels: newbie Fix For: 0.8.2 Attachments: KAFKA-1456.patch, KAFKA-1456.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (KAFKA-1456) Add LZ4 and LZ4C as a compression codec
[ https://issues.apache.org/jira/browse/KAFKA-1456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Oliver updated KAFKA-1456: Attachment: (was: KAFKA-1456.patch) Add LZ4 and LZ4C as a compression codec --- Key: KAFKA-1456 URL: https://issues.apache.org/jira/browse/KAFKA-1456 Project: Kafka Issue Type: Improvement Reporter: Joe Stein Labels: newbie Fix For: 0.8.2 Attachments: KAFKA-1456.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (KAFKA-1456) Add LZ4 and LZ4C as a compression codec
[ https://issues.apache.org/jira/browse/KAFKA-1456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14002477#comment-14002477 ] James Oliver commented on KAFKA-1456: - Updated reviewboard https://reviews.apache.org/r/21663/diff/ against branch origin/trunk Add LZ4 and LZ4C as a compression codec --- Key: KAFKA-1456 URL: https://issues.apache.org/jira/browse/KAFKA-1456 Project: Kafka Issue Type: Improvement Reporter: Joe Stein Labels: newbie Fix For: 0.8.2 Attachments: KAFKA-1456.patch, KAFKA-1456_2014-05-19_15:01:10.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (KAFKA-1456) Add LZ4 and LZ4C as a compression codec
[ https://issues.apache.org/jira/browse/KAFKA-1456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Oliver updated KAFKA-1456: Attachment: KAFKA-1456_2014-05-19_15:01:10.patch Add LZ4 and LZ4C as a compression codec --- Key: KAFKA-1456 URL: https://issues.apache.org/jira/browse/KAFKA-1456 Project: Kafka Issue Type: Improvement Reporter: Joe Stein Labels: newbie Fix For: 0.8.2 Attachments: KAFKA-1456.patch, KAFKA-1456_2014-05-19_15:01:10.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (KAFKA-1456) Add LZ4 and LZ4C as a compression codec
[ https://issues.apache.org/jira/browse/KAFKA-1456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14002583#comment-14002583 ] James Oliver commented on KAFKA-1456: - Updated reviewboard https://reviews.apache.org/r/21663/diff/ against branch origin/trunk Add LZ4 and LZ4C as a compression codec --- Key: KAFKA-1456 URL: https://issues.apache.org/jira/browse/KAFKA-1456 Project: Kafka Issue Type: Improvement Reporter: Joe Stein Labels: newbie Fix For: 0.8.2 Attachments: KAFKA-1456.patch, KAFKA-1456_2014-05-19_15:01:10.patch, KAFKA-1456_2014-05-19_16:39:01.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (KAFKA-1456) Add LZ4 and LZ4C as a compression codec
[ https://issues.apache.org/jira/browse/KAFKA-1456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Oliver updated KAFKA-1456: Attachment: KAFKA-1456_2014-05-19_16:39:01.patch Add LZ4 and LZ4C as a compression codec --- Key: KAFKA-1456 URL: https://issues.apache.org/jira/browse/KAFKA-1456 Project: Kafka Issue Type: Improvement Reporter: Joe Stein Labels: newbie Fix For: 0.8.2 Attachments: KAFKA-1456.patch, KAFKA-1456_2014-05-19_15:01:10.patch, KAFKA-1456_2014-05-19_16:39:01.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (KAFKA-1456) Add LZ4 and LZ4C as a compression codec
[ https://issues.apache.org/jira/browse/KAFKA-1456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Oliver updated KAFKA-1456: Attachment: KAFKA-1456_2014-05-19_18:19:32.patch Add LZ4 and LZ4C as a compression codec --- Key: KAFKA-1456 URL: https://issues.apache.org/jira/browse/KAFKA-1456 Project: Kafka Issue Type: Improvement Reporter: Joe Stein Labels: newbie Fix For: 0.8.2 Attachments: KAFKA-1456.patch, KAFKA-1456_2014-05-19_15:01:10.patch, KAFKA-1456_2014-05-19_16:39:01.patch, KAFKA-1456_2014-05-19_18:19:32.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (KAFKA-1456) Add LZ4 and LZ4C as a compression codec
[ https://issues.apache.org/jira/browse/KAFKA-1456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14002677#comment-14002677 ] James Oliver commented on KAFKA-1456: - Updated reviewboard https://reviews.apache.org/r/21663/diff/ against branch trunk Add LZ4 and LZ4C as a compression codec --- Key: KAFKA-1456 URL: https://issues.apache.org/jira/browse/KAFKA-1456 Project: Kafka Issue Type: Improvement Reporter: Joe Stein Labels: newbie Fix For: 0.8.2 Attachments: KAFKA-1456.patch, KAFKA-1456_2014-05-19_15:01:10.patch, KAFKA-1456_2014-05-19_16:39:01.patch, KAFKA-1456_2014-05-19_18:19:32.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
