*Hi Jun,
Soo sorry for the typo/mistake.
On 02/12/2016, 11:19, "Michael Pearce" <michael.pea...@ig.com> wrote:
Hi Jao
Thanks for the response. Sorry for slow reply, both with personal sickness
and also battling some critical issues encountered since upgrading to 0.10.1.0
1) Thans for spotting, Document error where we branched this KIP from
KIP-82, will get that removed.
2) Intent is to do this just at the record message level.
3) Thanks for spotting, Will ensure this is corrected.
4) As per discussion thread we will support tombstone + null value,
tombstone + non null value, no tombstone + null value.
5) I believe this was in the discussion thread, @Mayuresh is this something
we’ve overlooked? I thought we would down convert and remove the value so the
old consumer had existing behavior, or is there something we haven’t thought
about?
Cheers
Mike
On 30/11/2016, 18:12, "Jun Rao" <j...@confluent.io> wrote:
Hi, Michael,
Thanks for the KIP. A few comments below.
1. The message format change contains "HeadersLength Headers". Is that
intended?
2. For compressed messageset, is the tombstone bit only set at the
shallow
level? Do we always leave that bit in the wrapper message unset? An
alternative is to set the tombstone bit in the wrapper if at least one
inner message has the tombstone bit set. This makes things a bit more
complicated, but we could potentially exploit that for optimizing down
conversion. For example, we only need to convert messages with magic 2
to
magic 1 if the wrapper's tombstone bit is set (conversion is always
needed
from magic 2 to magic 0). Not sure if the optimization is worth the
complexity though.
3. The referencing of the new version of ProducerRequest/FetchRequest is
inconsistent (v4 vs v3). Since our convention starts at version at 0, I
think the new version would be 3.
4. "If the magic byte on message is 2, the broker should use the
tombstone
bit for log compaction." What about null value? My understanding is that
null value will be treated the same as setting the tombstone bit.
5. For the migration path, it would be useful to describe the down
conversion path to consumers (i.e., brokers on message format 0.10.2 and
consumers on older version).
Thanks,
Jun
On Tue, Nov 29, 2016 at 3:18 AM, Michael Pearce <michael.pea...@ig.com>
wrote:
> Hi All,
>
> We have been discussing in the below thread and final changes have
been
> made to the KIP wiki based on these discussions.
>
> We would now like to put to the vote the following KIP:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-87+-+
> Add+Compaction+Tombstone+Flag
>
> This kip is for having a distinct compaction attribute “tombstone”
flag
> instead of relying on null value, allowing non-null value delete
messages.
>
> Many thanks,
> Michael
>
>
>
> On 22/11/2016, 15:52, "Michael Pearce" <michael.pea...@ig.com> wrote:
>
> Hi Mayuresh,
>
> LGTM. Ive just made one small adjustment updating the wire
protocol to
> show the magic byte bump.
>
> Do we think we’re good to put to a vote? Is there any other bits
> needing discussion?
>
> Cheers
> Mike
>
> On 21/11/2016, 18:26, "Mayuresh Gharat"
<gharatmayures...@gmail.com>
> wrote:
>
> Hi Michael,
>
> I have updated the migration section of the KIP. Can you
please
> take a look?
>
> Thanks,
>
> Mayuresh
>
> On Fri, Nov 18, 2016 at 9:07 AM, Mayuresh Gharat <
> gharatmayures...@gmail.com
> > wrote:
>
> > Hi Michael,
> >
> > That whilst sending tombstone and non null value, the
consumer
> can expect
> > only to receive the non-null message only in step (3) is
this
> correct?
> > ---> I do agree with you here.
> >
> > Becket, Ismael : can you guys review the migration plan
listed
> above using
> > magic byte?
> >
> > Thanks,
> >
> > Mayuresh
> >
> > On Fri, Nov 18, 2016 at 8:58 AM, Michael Pearce <
> michael.pea...@ig.com>
> > wrote:
> >
> >> Many thanks for this Mayuresh. I don't have any objections.
> >>
> >> I assume we should state:
> >>
> >> That whilst sending tombstone and non null value, the
consumer
> can expect
> >> only to receive the non-null message only in step (3) is
this
> correct?
> >>
> >> Cheers
> >> Mike
> >>
> >>
> >>
> >> Sent using OWA for iPhone
> >> ________________________________________
> >> From: Mayuresh Gharat <gharatmayures...@gmail.com>
> >> Sent: Thursday, November 17, 2016 5:18:41 PM
> >> To: dev@kafka.apache.org
> >> Subject: Re: [DISCUSS] KIP-87 - Add Compaction Tombstone
Flag
> >>
> >> Hi Ismael,
> >>
> >> Thanks for the explanation.
> >> Specially I like this part where in you mentioned we can
get
> rid of the
> >> older null value support for log compaction later on, here
:
> >> We can't change semantics of the message format without
having
> a long
> >> transition period. And we can't rely
> >> on people reading documentation or acting on a warning for
> something so
> >> fundamental. As such, my take is that we need to bump the
magic
> byte. The
> >> good news is
> >> that we don't have to support all versions forever. We have
> said that we
> >> will support direct upgrades for 2 years. That means that
> message format
> >> version n could, in theory, be removed 2 years after the
it's
> introduced.
> >>
> >> Just a heads up, I would like to mention that even without
> bumping magic
> >> byte, we will *NOT* loose zero copy as in the client(x+1)
in my
> >> explanation
> >> above will convert internally a null value to have a
tombstone
> bit set and
> >> a tombstone bit set to have a null value automatically
> internally and by
> >> the time we move to version (x+2), the clients would have
> upgraded.
> >> Obviously if we support a request from consumer(x), we will
> loose zero
> >> copy
> >> but that is the same case with magic byte.
> >>
> >> But if magic byte bump makes life easier for transition
for the
> above
> >> reasons that you explained, I am OK with it since we are
going
> to meet the
> >> end goal down the road :)
> >>
> >> On a side note can we update the doc here on magic byte to
say
> that "*it
> >> should be bumped whenever the message format is changed or
the
> >> interpretation of message format (usage of the reserved
bits as
> well) is
> >> changed*".
> >>
> >>
> >> Hi Michael,
> >>
> >> Here is the update plan that we discussed offline
yesterday :
> >>
> >> Currently the magic-byte which corresponds to the
> "message.format.version"
> >> is set to 1.
> >>
> >> 1) On broker it will be set to 1 initially.
> >>
> >> 2) When a producer client sends a message with magic-byte
= 2,
> since the
> >> broker is on magic-byte = 1, we will down convert it, which
> means if the
> >> tombstone bit is set, the value will be set to null. A
consumer
> >> understanding magic-byte = 1, will still work with this. A
> consumer
> >> working
> >> with magic-byte =2 will also be able to understand this,
since
> it
> >> understands the tombstone.
> >> Now there is still the question of supporting a
non-tombstone
> and null
> >> value from producer client with magic-byte = 2.* (I am not
sure
> if we
> >> should support this. Ismael/Becket can comment here)*
> >>
> >> 3) When almost all the clients have upgraded, the
> message.format.version
> >> on
> >> the broker can be changed to 2, where in the down
conversion in
> the above
> >> step will not happen. If at this point we get a consumer
> request from a
> >> older consumer, we might have to down convert where in we
loose
> zero copy,
> >> but these cases should be rare.
> >>
> >> Becket can you review this plan and add more details if I
have
> >> missed/wronged something, before we put it on KIP.
> >>
> >> Thanks,
> >>
> >> Mayuresh
> >>
> >> On Wed, Nov 16, 2016 at 11:07 PM, Michael Pearce <
> michael.pea...@ig.com>
> >> wrote:
> >>
> >> > Thanks guys, for discussing this offline and getting some
> consensus.
> >> >
> >> > So its clear for myself and others what is proposed now
(i
> think i
> >> > understand, but want to make sure)
> >> >
> >> > Could i ask either directly update the kip to detail the
> migration
> >> > strategy, or (re-)state your offline discussed and agreed
> migration
> >> > strategy based on a magic byte is in this thread.
> >> >
> >> >
> >> > The main original driver for the KIP was to support
> compaction where
> >> value
> >> > isn't null, based off the discussions on KIP-82 thread.
> >> >
> >> > We should be able to support non-tombstone + null value
by the
> >> completion
> >> > of the KIP, as we noted when discussing this kip, having
> logic based on
> >> a
> >> > null value isn't very clean and also separates the
concerns.
> >> >
> >> > As discussed already though we can split this into
KIP-87a
> and KIP-87b
> >> >
> >> > Where we look to deliver KIP-87a on a compacted topic (to
> address the
> >> > immediate issues)
> >> > * tombstone + null value
> >> > * tombstone + non-null value
> >> > * non-tombstone + non-null value
> >> >
> >> > Then we can discuss once KIP-87a is completed options
later
> and how we
> >> > support the second part KIP-87b to deliver:
> >> > * non-tombstone + null value
> >> >
> >> > Cheers
> >> > Mike
> >> >
> >> >
> >> >
> >> > ________________________________________
> >> > From: Becket Qin <becket....@gmail.com>
> >> > Sent: Thursday, November 17, 2016 1:43 AM
> >> > To: dev@kafka.apache.org
> >> > Subject: Re: [DISCUSS] KIP-87 - Add Compaction Tombstone
Flag
> >> >
> >> > Renu, Mayuresh and I had an offline discussion, and
following
> is a brief
> >> > summary.
> >> >
> >> > 1. We agreed that not bumping up magic value may result
in
> losing zero
> >> copy
> >> > during migration.
> >> > 2. Given that bumping up magic value is almost free and
has
> benefit of
> >> > avoiding potential performance issue. It is probably
worth
> doing.
> >> >
> >> > One issue we still need to think about is whether we
want to
> support a
> >> > non-tombstone message with null value.
> >> > Currently it is not supported by Kafka. If we allow a
> non-tombstone null
> >> > value message to exist after KIP-87. The problem is that
such
> message
> >> will
> >> > not be supported by the consumers prior to KIP-87.
Because a
> null value
> >> > will always be interpreted to a tombstone.
> >> >
> >> > One option is that we keep the current way, i.e. do not
> support such
> >> > message. It would be good to know if there is a concrete
use
> case for
> >> such
> >> > message. If there is not, we can probably just not
support it.
> >> >
> >> > Thanks,
> >> >
> >> > JIangjie (Becket) Qin
> >> >
> >> >
> >> >
> >> > On Wed, Nov 16, 2016 at 1:28 PM, Mayuresh Gharat <
> >> > gharatmayures...@gmail.com
> >> > > wrote:
> >> >
> >> > > Hi Ismael,
> >> > >
> >> > > This is something I can think of for migration plan:
> >> > > So the migration plan can look something like this,
with up
> >> conversion :
> >> > >
> >> > > 1) Currently lets say we have Broker at version x.
> >> > > 2) Currently we have clients at version x.
> >> > > 3) a) We move the version to Broker(x+1) : supports
both
> tombstone and
> >> > null
> >> > > for log compaction.
> >> > > b) We upgrade the client to version client(x+1) :
if in
> the
> >> producer
> >> > > client(x+1) the value is set to null, we will
automatically
> set the
> >> > > Tombstone bit internally. If the producer client(x+1)
sets
> the
> >> tombstone
> >> > > itself, well and good. For producer client(x), the
broker
> will up
> >> convert
> >> > > to have the tombstone bit. Broker(x+1) is supporting
both.
> Consumer
> >> > > client(x+1) will be aware of this and should be able to
> handle this.
> >> For
> >> > > consumer client(x) we will down convert the message on
the
> broker
> >> side.
> >> > > c) At this point we will have to specify a warning
or
> clearly
> >> specify
> >> > > in docs that this behavior is about to be changed for
log
> compaction.
> >> > > 4) a) In next release of the Broker(x+2), we say that
only
> Tombstone
> >> is
> >> > > used for log compaction on the Broker side.
Clients(x+1)
> still is
> >> > > supported.
> >> > > b) We upgrade the client to version client(x+2) :
if
> value is set
> >> to
> >> > > null, tombstone will not be set automatically. The
client
> will have to
> >> > call
> >> > > setTombstone() to actually set the tombstone.
> >> > >
> >> > > We should compare this migration plan with the
migration
> plan for
> >> magic
> >> > > byte bump and do whatever looks good.
> >> > > I am just worried that if we go down magic byte route,
> unless I am
> >> > missing
> >> > > something, it sounds like kafka will be stuck with
> supporting both
> >> null
> >> > > value and tombstone bit for log compaction for life
long,
> which does
> >> not
> >> > > look like a good end state.
> >> > >
> >> > > Thanks,
> >> > >
> >> > > Mayuresh
> >> > >
> >> > >
> >> > >
> >> > >
> >> > > On Wed, Nov 16, 2016 at 9:32 AM, Mayuresh Gharat <
> >> > > gharatmayures...@gmail.com
> >> > > > wrote:
> >> > >
> >> > > > Hi Ismael,
> >> > > >
> >> > > > That's a very good point which I might have not
> considered earlier.
> >> > > >
> >> > > > Here is a plan that I can think of:
> >> > > >
> >> > > > Stage 1) The broker from now on, up converts the
message
> to have the
> >> > > > tombstone marker. The log compaction thread does log
> compaction
> >> based
> >> > on
> >> > > > both null and tombstone marker. This is our
transition
> period.
> >> > > > Stage 2) The next release we only say that log
compaction
> is based
> >> on
> >> > > > tombstone marker. (Open source kafka makes this as a
> policy). By
> >> this
> >> > > time,
> >> > > > the organization which is moving to this release
will be
> sure that
> >> they
> >> > > > have gone through the entire transition period.
> >> > > >
> >> > > > My only goal of doing this is that Kafka clearly
> specifies the end
> >> > state
> >> > > > about what log compaction means (is it null value or
a
> tombstone
> >> > marker,
> >> > > > but not both).
> >> > > >
> >> > > > What do you think?
> >> > > >
> >> > > > Thanks,
> >> > > >
> >> > > > Mayuresh
> >> > > > .
> >> > > >
> >> > > > On Wed, Nov 16, 2016 at 9:17 AM, Ismael Juma <
> ism...@juma.me.uk>
> >> > wrote:
> >> > > >
> >> > > >> One comment below.
> >> > > >>
> >> > > >> On Wed, Nov 16, 2016 at 5:08 PM, Mayuresh Gharat <
> >> > > >> gharatmayures...@gmail.com
> >> > > >> > wrote:
> >> > > >>
> >> > > >> > - If we don't bump up the magic byte, on the
broker
> side, the
> >> > > broker
> >> > > >> > will always have to look at both tombstone bit
and
> the value
> >> when
> >> > > do
> >> > > >> the
> >> > > >> > compaction. Assuming we do not bump up the
magic
> byte,
> >> > > >> > imagine the broker sees a message which does
not
> have a
> >> tombstone
> >> > > bit
> >> > > >> > set. The broker does not know when the message
was
> produced
> >> (i.e.
> >> > > >> > whether
> >> > > >> > the message has been up converted or not), it
has
> to take a
> >> > further
> >> > > >> > look at
> >> > > >> > the value to see if it is null or not in order
to
> determine
> >> if it
> >> > > is
> >> > > >> a
> >> > > >> > tombstone. The same logic has to be put on the
> consumer as
> >> well
> >> > > >> because
> >> > > >> > the
> >> > > >> > consumer does not know if the message has been
up
> converted or
> >> > not.
> >> > > >> > - If we upconvert while appending, this is
not
> the case,
> >> > right?
> >> > > >>
> >> > > >>
> >> > > >> If I understand you correctly, this is not
sufficient
> because the
> >> log
> >> > > may
> >> > > >> have messages appended before it was upgraded to
include
> KIP-87.
> >> > > >>
> >> > > >> Ismael
> >> > > >>
> >> > > >
> >> > > >
> >> > > >
> >> > > > --
> >> > > > -Regards,
> >> > > > Mayuresh R. Gharat
> >> > > > (862) 250-7125
> >> > > >
> >> > >
> >> > >
> >> > >
> >> > > --
> >> > > -Regards,
> >> > > Mayuresh R. Gharat
> >> > > (862) 250-7125
> >> > >
> >> > The information contained in this email is strictly
> confidential and for
> >> > the use of the addressee only, unless otherwise
indicated. If
> you are
> >> not
> >> > the intended recipient, please do not read, copy, use or
> disclose to
> >> others
> >> > this message or any attachment. Please also notify the
sender
> by
> >> replying
> >> > to this email or by telephone (+44(020 7896 0011) and
then
> delete the
> >> email
> >> > and any copies of it. Opinions, conclusion (etc) that do
not
> relate to
> >> the
> >> > official business of this company shall be understood as
> neither given
> >> nor
> >> > endorsed by it. IG is a trading name of IG Markets
Limited (a
> company
> >> > registered in England and Wales, company number
04008957) and
> IG Index
> >> > Limited (a company registered in England and Wales,
company
> number
> >> > 01190902). Registered address at Cannon Bridge House, 25
> Dowgate Hill,
> >> > London EC4R 2YA. Both IG Markets Limited (register number
> 195355) and IG
> >> > Index Limited (register number 114059) are authorised and
> regulated by
> >> the
> >> > Financial Conduct Authority.
> >> >
> >>
> >>
> >>
> >> --
> >> -Regards,
> >> Mayuresh R. Gharat
> >> (862) 250-7125
> >> The information contained in this email is strictly
> confidential and for
> >> the use of the addressee only, unless otherwise indicated.
If
> you are not
> >> the intended recipient, please do not read, copy, use or
> disclose to others
> >> this message or any attachment. Please also notify the
sender
> by replying
> >> to this email or by telephone (+44(020 7896 0011) and then
> delete the email
> >> and any copies of it. Opinions, conclusion (etc) that do
not
> relate to the
> >> official business of this company shall be understood as
> neither given nor
> >> endorsed by it. IG is a trading name of IG Markets Limited
(a
> company
> >> registered in England and Wales, company number 04008957)
and
> IG Index
> >> Limited (a company registered in England and Wales, company
> number
> >> 01190902). Registered address at Cannon Bridge House, 25
> Dowgate Hill,
> >> London EC4R 2YA. Both IG Markets Limited (register number
> 195355) and IG
> >> Index Limited (register number 114059) are authorised and
> regulated by the
> >> Financial Conduct Authority.
> >>
> >
> >
> >
> > --
> > -Regards,
> > Mayuresh R. Gharat
> > (862) 250-7125
> >
>
>
>
> --
> -Regards,
> Mayuresh R. Gharat
> (862) 250-7125
>
>
> The information contained in this email is strictly confidential
and
> for the use of the addressee only, unless otherwise indicated. If you
are
> not the intended recipient, please do not read, copy, use or disclose
to
> others this message or any attachment. Please also notify the sender
by
> replying to this email or by telephone (+44(020 7896 0011) and then
delete
> the email and any copies of it. Opinions, conclusion (etc) that do not
> relate to the official business of this company shall be understood as
> neither given nor endorsed by it. IG is a trading name of IG Markets
> Limited (a company registered in England and Wales, company number
> 04008957) and IG Index Limited (a company registered in England and
Wales,
> company number 01190902). Registered address at Cannon Bridge House,
25
> Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register
number
> 195355) and IG Index Limited (register number 114059) are authorised
and
> regulated by the Financial Conduct Authority.
>
>
>
The information contained in this email is strictly confidential and for
the use of the addressee only, unless otherwise indicated. If you are not the
intended recipient, please do not read, copy, use or disclose to others this
message or any attachment. Please also notify the sender by replying to this
email or by telephone (+44(020 7896 0011) and then delete the email and any
copies of it. Opinions, conclusion (etc) that do not relate to the official
business of this company shall be understood as neither given nor endorsed by
it. IG is a trading name of IG Markets Limited (a company registered in England
and Wales, company number 04008957) and IG Index Limited (a company registered
in England and Wales, company number 01190902). Registered address at Cannon
Bridge House, 25 Dowgate Hill, London EC4R 2YA. Both IG Markets Limited
(register number 195355) and IG Index Limited (register number 114059) are
authorised and regulated by the Financial Conduct Authority.
