I have verified this functionality on kafka trunk using librdkafka lz4 branch and it works as intended.
2016-05-07 18:07 GMT+02:00 Dana Powers <dana.pow...@gmail.com>: > Vote Passed! I will update the wiki. > > -Dana > On May 7, 2016 3:48 AM, "Ismael Juma" <ism...@juma.me.uk> wrote: > > > Dana, a long time has passed since the vote started and there are enough > > binding votes, so maybe it's time to declare that the vote has passed? > > Please mark the KIP as adopted in the KIP page and move it to the adopted > > table in the KIPs page once you do that. > > > > Ismael > > On 6 May 2016 22:16, "Ismael Juma" <ism...@juma.me.uk> wrote: > > > > +1 (assuming the changes I mentioned in the discuss thread are > > incorporated) > > > > Ismael > > > > On Thu, May 5, 2016 at 1:13 AM, Jun Rao <j...@confluent.io> wrote: > > > > > Thanks for the response. +1 on the KIP. > > > > > > Jun > > > > > > On Thu, Apr 28, 2016 at 9:01 AM, Dana Powers <dana.pow...@gmail.com> > > > wrote: > > > > > > > Sure thing. Yes, the substantive change is fixing the HC checksum. > > > > > > > > But to further improve interoperability, the kafka LZ4 class would no > > > > longer reject messages that have these optional header flags set. The > > > > flags might get set if the client/user chooses to use a non-java lz4 > > > > compression library that includes them. In practice, naive support > for > > > > the flags just means reading a few extra bytes in the header and/or > > > > footer of the payload. The KIP does not intend to use or validate > this > > > > extra data. > > > > > > > > ContentSize is described as: "This field has no impact on decoding, > it > > > > just informs the decoder how much data the frame holds (for example, > > > > to display it during decoding process, or for verification purpose). > > > > It can be safely skipped by a conformant decoder." We skip it. > > > > > > > > ContentChecksum is "Content Checksum validates the result, that all > > > > blocks were fully transmitted in the correct order and without error, > > > > and also that the encoding/decoding process itself generated no > > > > distortion." We skip it. > > > > > > > > -Dana > > > > > > > > > > > > On Thu, Apr 28, 2016 at 7:43 AM, Jun Rao <j...@confluent.io> wrote: > > > > > Hi, Dana, > > > > > > > > > > Could you explain the following from the KIP a bit more? The KIP is > > > > > intended to just fix the HC checksum, but the following seems to > > > suggest > > > > > there are other format changes? > > > > > > > > > > KafkaLZ4* code: > > > > > > > > > > - add naive support for optional header flags (ContentSize, > > > > > ContentChecksum) to enable interoperability with off-the-shelf > lz4 > > > > libraries > > > > > - the only flag left unsupported is dependent-block compression, > > > which > > > > > our implementation does not currently support. > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > Jun > > > > > > > > > > On Mon, Apr 25, 2016 at 2:26 PM, Dana Powers < > dana.pow...@gmail.com> > > > > wrote: > > > > > > > > > >> Hi all, > > > > >> > > > > >> Initiating a vote thread because the KIP-57 proposal is specific > to > > > > >> the 0.10 release. > > > > >> > > > > >> KIP-57 can be accessed here: > > > > >> < > > > > >> > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-57+-+Interoperable+LZ4+Framing > > > > >> >. > > > > >> > > > > >> The related JIRA is > > https://issues.apache.org/jira/browse/KAFKA-3160 > > > > >> and working github PR at > https://github.com/apache/kafka/pull/1212 > > > > >> > > > > >> The vote will run for 72 hours. > > > > >> > > > > >> +1 (non-binding) > > > > >> > > > > > > > > > >
