Hey Michael, Sorry if it's frustrating. Definitely not a Confluent thing, I think you've seen both sides from different Confluent folks. I'm just trying to understand the compatibility implications to get on board myself. Given you're saying there is no compatibility implication I think maybe I'm confused with how the proposal works. Let me see if I can go back through an un-confuse myself.
-Jay On Fri, Dec 16, 2016 at 12:16 PM, Michael Pearce <michael.pea...@ig.com> wrote: > Hi Jay > > I disagree here that we are breaking any compatibility, we went through > this on the discussion thread in fact with the help of that thread is how > the kip came to the solution. > > Also on the supported combinations front you mention, we are not taking > anything away afaik. > > Currently supported are only are: > Null value = delete > Non-null value = non delete > > With this kip we would support > Null value + tombstone = delete > Non null value + tombstone = delete > Non null value + no tombstone = non delete > > As for the alternative idea, this is simply a new policy, how can there be > confusion here? For this policy it would be explicit that tombstone marker > would need to be set for a delete. > > I'm going to vent a little now as starting to get quite frustrated. > > We are going round in circles on kip-82 as per use cases there is now many > use cases, how many more are needed? just because confluent don't see these > doesn't mean they aren't real use cases other have, this is the point of > the Apache foundation, it shouldn't be the view of just one organisation. > It really is getting a feeling of the NIH syndrome. Rather than it being > constructive on discussion of the implementation detail. > > kip-87 spawned from as on the kip call we all agreed this was needed. And > would at least allow a custom wrapper be supported in a compacted topic, > allowing meta data. Which again now I feel we are spinning wheels, and > simply finding reasons not support it. > > Cheers > Mike > > > > Sent using OWA for iPad > ________________________________________ > From: Jay Kreps <j...@confluent.io> > Sent: Friday, December 16, 2016 7:09:23 PM > To: dev@kafka.apache.org > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag > > Hey Michael, > > I do think it might have been better had we started with a separate concept > of null vs delete. Given that we are where we are, I'm not sure that the > possible cures we explored so far are better than the disease. > > I apologize for coming to this late, but I didn't really understand the > implications of the KIP and that we'd be breaking compatibility with > existing apps until the vote had begun, and in my defense I'm not sure the > other folks voting did either. > > I think we all agree there are many existing apps that are built with the > assumption of "null value non-tombstone" and it isn't possible to > disambiguate these from tombstones on the producer. It isn't that anyone is > saying we have to support all four possibilities at once, it's that we > simply can't orphan one of the existing combinations or our users will eat > us! > > If I've understood your alternate solution of adding another setting for > compaction, I think this does fix the compatibility problem, but it adds an > odd mode the user has to add on all their topics. While the current state > is easily explainable, the resulting state where the meaning of tombstone > and null are overlapping and ambiguous and dependent on a compaction > setting that could change out of band or not be in sync with your code in > some environment seems worse to me then where we currently are. I think the > question is how would this combination be explained to users and does it > make sense? > > -Jay > > > > On Fri, Dec 16, 2016 at 9:25 AM, Michael Pearce <michael.pea...@ig.com> > wrote: > > > Hi Chaps, > > > > Can we either get one more +1 binding (we have 2 already) on the > existing? > > > > Or have some response on the below possible alternative. We are keen to > > get working on this, so we make next feature release. > > > > Cheers > > Mike > > > > > > On 13/12/2016, 16:32, "Michael Pearce" <michael.pea...@ig.com> wrote: > > > > Hi Ismael > > > > Did you see our email this morning, what's your thoughts on this > > approach to instead we simply have a brand new policy? > > > > Cheers > > Mike > > > > > > Sent using OWA for iPhone > > ________________________________________ > > From: isma...@gmail.com <isma...@gmail.com> on behalf of Ismael > Juma < > > ism...@juma.me.uk> > > Sent: Tuesday, December 13, 2016 11:30:05 AM > > To: dev@kafka.apache.org > > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag > > > > Yes, this is actually tricky to do in a way where we both have the > > desired > > semantics and maintain compatibility. When someone creates a > > `ProducerRecord` with a `null` value today, the producer doesn't know > > if > > it's meant to be a tombstone or not. For V3 messages, it's easy when > > the > > constructor that takes a tombstone is used. However, if any other > > constructor is used, it's not clear. A couple of options I can think > > of, > > none of them particularly nice: > > > > 1. Have a third state where tombstone = unknown and the broker would > > set > > the tombstone bit if the value was null and the topic was compacted. > > People > > that wanted to pass a non-null value for the tombstone would have to > > use > > the constructor that takes a tombstone. The drawbacks: third state > for > > tombstone in message format, message conversion at the broker for a > > common > > case. > > > > 2. Extend MetadataResponse to optionally include topic configs, which > > would > > make it possible for the producer to be smarter about setting the > > tombstone. It would only do it if a tombstone was not passed > > explicitly, > > the value was null and the topic was compacted. The main drawback is > > that > > the producer would be getting a bit more data for each topic even > > though it > > probably won't use it most of the time. Extending MetadataResponse to > > return topic configs would be useful for other reasons as well, so > that > > part seems OK. > > > > In addition, for both proposals, we could consider adding warnings to > > the > > documentation that the behaviour of the constructors that don't take > a > > tombstone would change in the next major release so that tombstone = > > false. > > Not sure if this would be worth it though. > > > > Ismael > > > > On Sun, Dec 11, 2016 at 11:15 PM, Ewen Cheslack-Postava < > > e...@confluent.io> > > wrote: > > > > > Michael, > > > > > > It kind of depends on how you want to interpret the tombstone flag. > > If it's > > > purely a producer-facing Kafka-level thing that we treat as > internal > > to the > > > broker and log cleaner once the record is sent, then your approach > > makes > > > sense. You're just moving copying the null-indicates-delete > behavior > > of the > > > old constructor into the tombstone flag. > > > > > > However, if you want this change to more generally decouple the > idea > > of > > > deletion and null values, then you are sometimes converting what > > might be a > > > completely valid null value that doesn't indicate deletion into a > > > tombstone. Downstream applications could potentially handle these > > cases > > > differently given the separation of deletion from value. > > > > > > I guess the question is if we want to try to support the latter > even > > for > > > topics where we have older produce requests. An example where this > > could > > > come up is in something like a CDC Connector. If we try to support > > the > > > semantic difference, a connector might write changes to Kafka using > > the > > > tombstone flag to indicate when a row was truly deleted (vs an > > update that > > > sets it to null but still present; this probably makes more sense > > for CDC > > > from document stores or extracting single columns). There are > various > > > reasons we might want to maintain the full log and not turn > > compaction on > > > (or just use a time-based retention policy), but downstream > > applications > > > might care to know the difference between a delete and a null > value. > > In > > > fact both versions of the same log (compacted and time-retention) > > could be > > > useful and I don't think it'll be uncommon to maintain both or use > > KIP-71 > > > to maintain a hybrid compacted/retention topic. > > > > > > -Ewen > > > > > > On Sun, Dec 11, 2016 at 1:18 PM, Michael Pearce < > > michael.pea...@ig.com> > > > wrote: > > > > > > > Hi Jay, > > > > > > > > Why wouldn't that work, the tombstone value is only looked at by > > the > > > > broker, on a topic configured for compaction as such is benign on > > non > > > > compacted topics. This is as much as sending a null value > currently > > > > > > > > > > > > Regards > > > > Mike > > > > > > > > > > > > > > > > Sent using OWA for iPhone > > > > ________________________________________ > > > > From: Jay Kreps <j...@confluent.io> > > > > Sent: Sunday, December 11, 2016 8:58:53 PM > > > > To: dev@kafka.apache.org > > > > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag > > > > > > > > Hey Michael, > > > > > > > > I'm not quite sure that works as that would translate ALL null > > values to > > > > tombstones, even for non-compacted topics that use null as an > > acceptable > > > > value sent by the producer and expected by the consumer. > > > > > > > > -Jay > > > > > > > > On Sun, Dec 11, 2016 at 3:26 AM, Michael Pearce < > > michael.pea...@ig.com> > > > > wrote: > > > > > > > > > Hi Ewen, > > > > > > > > > > I think the easiest way to show this is with code. > > > > > > > > > > As you can see we keep the existing behaviour for code/binaries > > calling > > > > > the pre-existing constructors, whereby if the value is null the > > > tombstone > > > > > is set to true. > > > > > > > > > > Regards > > > > > Mike > > > > > > > > > > > > > > > > > > > > /** > > > > > * Creates a record with a specified timestamp to be sent > to > > a > > > > > specified topic and partition > > > > > * > > > > > * @param topic The topic the record will be appended to > > > > > * @param partition The partition to which the record > should > > be > > > sent > > > > > * @param timestamp The timestamp of the record > > > > > * @param tombstone if the record should be treated as a > > tombstone > > > if > > > > > the topic is compacted > > > > > * @param key The key that will be included in the record > > > > > * @param value The record contents > > > > > */ > > > > > public ProducerRecord(String topic, Integer partition, > > Boolean > > > > > tombstone, Long timestamp, K key, V value) { > > > > > if (topic == null) > > > > > throw new IllegalArgumentException("Topic cannot > be > > > null."); > > > > > if (timestamp != null && timestamp < 0) > > > > > throw new IllegalArgumentException( > > > > > String.format("Invalid timestamp: %d. > > Timestamp > > > > should > > > > > always be non-negative or null.", timestamp)); > > > > > if (partition != null && partition < 0) > > > > > throw new IllegalArgumentException( > > > > > String.format("Invalid partition: %d. > > Partition > > > > number > > > > > should always be non-negative or null.", partition)); > > > > > if (!tombstone && value == null){ > > > > > throw new IllegalArgumentException( > > > > > String.format("Tombstone must be true if > null > > > > value")); > > > > > } > > > > > this.topic = topic; > > > > > this.partition = partition; > > > > > this.tombstone = tombstone; > > > > > this.key = key; > > > > > this.value = value; > > > > > this.timestamp = timestamp; > > > > > } > > > > > > > > > > public ProducerRecord(String topic, Integer partition, Long > > > > timestamp, > > > > > K key, V value) { > > > > > this(topic, partition, value == null, timestamp, key, > > value); > > > > > } > > > > > ________________________________________ > > > > > From: Ewen Cheslack-Postava <e...@confluent.io> > > > > > Sent: Sunday, December 11, 2016 5:45 AM > > > > > To: dev@kafka.apache.org > > > > > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag > > > > > > > > > > It seemed like this was addressed in the migration section, > > wasn't it? > > > > The > > > > > V2 vs V3 requests and the fact that the broker will downgrade > the > > > message > > > > > format (losing zero copy) if you issues a V2 request to a > broker > > using > > > V3 > > > > > format handles compatibility, does it not? The existing > > consumers won't > > > > see > > > > > the extra metadata in the value, but they will get a null > > instead and > > > > treat > > > > > it as a tombstone. Certainly there is a performance impact, but > > it > > > seemed > > > > > compatible. > > > > > > > > > > I'm worried about this though: > > > > > > > > > > > > > > > - *NOTE *: With the new version of producer client using > > > > ProduceRequest > > > > > V3 (magic byte = 2), a non tombstone (tombstone bit not set) > > and > > > null > > > > > value > > > > > should not be allowed as the older version of consumer using > > > > > FetchRequest > > > > > V2 will think of this as a tombstone when its actually not. > > > > > > > > > > > > > > > Unless I'm misunderstanding, this ends up breaking binary > > compatibility > > > > for > > > > > the Java API. It sounds like this suggests that passing a null > > value to > > > > the > > > > > existing ProducerRecord constructors would generate an > exception > > since > > > > you > > > > > didn't explicitly enable the tombstone (via whatever new > > constructor is > > > > > provided). But that means you can't swap in a newer client jar > > without > > > > > recompiling and get the same behavior if your app deletes keys > > using > > > the > > > > > current approach because your app will start throwing > > exceptions. Maybe > > > > > this is a tradeoff we're ok with, but we've tried pretty hard > to > > avoid > > > > > breaking compatibility like this so far -- it makes picking up > > bug > > > fixes > > > > in > > > > > the clients harder for users of frameworks that have to pin to > > earlier > > > > > default versions for compatibility. > > > > > > > > > > But then later the KIP says: > > > > > > > > > > > > > > > - The old constructors for ProducerRecord without this > > parameter > > > will > > > > be > > > > > kept but updated so that their default behaviour if setting > > null > > > value > > > > > will > > > > > be the tombstone will be set to true to keep existing > > behaviour. > > > > > > > > > > > > > > > So maybe I am misinterpreting something. > > > > > > > > > > And just a nit re: motivation section, item 6 would be clearer > > for a > > > > union > > > > > schema with null (or optional schemas in other formats), e.g. > > [null, > > > > > string], in which case losing the schema truly is losing > > information > > > > > (whereas null is already the only valid value for a pure null > > schema). > > > > > > > > > > -Ewen > > > > > > > > > > > > > > > On Sat, Dec 10, 2016 at 9:24 PM, Michael Pearce < > > michael.pea...@ig.com > > > > > > > > > wrote: > > > > > > > > > > > Hi Jay, > > > > > > > > > > > > Good point this detail is missing in the KIP write up. Ive > > added this > > > > > now. > > > > > > > > > > > > Essentially simply just upgrading the clients we do not > expect > > any > > > > client > > > > > > app code change needed. > > > > > > > > > > > > Cheers > > > > > > Mike > > > > > > ________________________________________ > > > > > > From: Jay Kreps <j...@confluent.io> > > > > > > Sent: Saturday, December 10, 2016 10:51 PM > > > > > > To: dev@kafka.apache.org > > > > > > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag > > > > > > > > > > > > Michael, > > > > > > > > > > > > The compatibility section goes through the migration path, > but > > isn't > > > > the > > > > > > bigger compatibility issue with existing apps? There are many > > > (probably > > > > > > thousands) of apps in production that use this feature and > > send null > > > to > > > > > > mean delete. It seems like this would break compatibility > with > > them, > > > > and > > > > > > they would have to be rewritten to right? > > > > > > > > > > > > -Jay > > > > > > > > > > > > On Thu, Dec 8, 2016 at 12:12 AM, Michael Pearce < > > > michael.pea...@ig.com > > > > > > > > > > > wrote: > > > > > > > > > > > > > Hi Jun, > > > > > > > > > > > > > > 4) On v3 we honour the tombstone. As such we expect it to > be > > set > > > > > > correctly > > > > > > > as per the KIP. > > > > > > > > > > > > > > 4.1) why would we want to produce an error when its v3? > This > > is the > > > > > exact > > > > > > > purpose to support non-null tombstone’s > > > > > > > 4.2) again here im unclear on the question on the v3, > produce > > > request > > > > > we > > > > > > > expect the tombstone flag to be set correctly. > > > > > > > > > > > > > > 4.4) compaction only occurs on compacted topics, the bit > > makes no > > > > > > > difference and not looked at on un-compacted (time/size > based > > > > > eviction). > > > > > > > > > > > > > > > > > > > > > On 06/12/2016, 20:08, "Jun Rao" <j...@confluent.io> wrote: > > > > > > > > > > > > > > Hi, Michael, > > > > > > > > > > > > > > 4. Then, I think I misunderstood this point. Could you > > document > > > > the > > > > > > > following points in the wiki? > > > > > > > 4.1 If producer V3 sets tombstone, but provides a > > non-null > > > value, > > > > > > does > > > > > > > the > > > > > > > send() get an error or does the producer automatically > > set the > > > > > value > > > > > > to > > > > > > > null? > > > > > > > 4.2 If producer V3 doesn't set tombstone, but provides > a > > null > > > > > value, > > > > > > > does > > > > > > > the send() get an error or does the producer > > automatically sets > > > > the > > > > > > > tombstone? > > > > > > > 4.3 Does the broker only expect messages that (a) have > no > > > > tombstone > > > > > > and > > > > > > > non-null value; (b) have tombstone and null value and > > reject > > > the > > > > > > > messages > > > > > > > with an error code otherwise? > > > > > > > 4.4 Do 4.1, 4.2, 4.3 depend on whether the topic is > > compacted > > > on > > > > > > not? > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > Jun > > > > > > > > > > > > > > On Tue, Dec 6, 2016 at 10:35 AM, Michael Pearce < > > > > > > michael.pea...@ig.com > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > Not at all. This only acts on compacted topics just > > as what > > > > > occurs > > > > > > > today > > > > > > > > > > > > > > > > Sent using OWA for iPhone > > > > > > > > ________________________________________ > > > > > > > > From: Jun Rao <j...@confluent.io> > > > > > > > > Sent: Tuesday, December 6, 2016 6:25:28 PM > > > > > > > > To: dev@kafka.apache.org > > > > > > > > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone > > Flag > > > > > > > > > > > > > > > > Hi, Michael, > > > > > > > > > > > > > > > > 4. Hmm, does that mean the new client library can > > never send > > > a > > > > > null > > > > > > > message > > > > > > > > even to a regular topic? This seems like a change of > > the > > > > existing > > > > > > > behavior. > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > Jun > > > > > > > > > > > > > > > > On Tue, Dec 6, 2016 at 9:51 AM, Michael Pearce < > > > > > > > michael.pea...@ig.com> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > Hi Jun, > > > > > > > > > > > > > > > > > > Re 4) That's because we expect the tombstone value > > to be > > > set > > > > > > > correctly if > > > > > > > > > message bit is 2, as such if an older client sends > > in on > > > old > > > > > > > message the > > > > > > > > > message is upcast and the bit is set correctly. And > > such no > > > > > > longer > > > > > > > need > > > > > > > > to > > > > > > > > > check the value. Mayuresh can you confirm my > > thinking and > > > > > > > understanding > > > > > > > > of > > > > > > > > > what we've discussed? > > > > > > > > > > > > > > > > > > The second point I understand what you're getting > at > > now my > > > > > > > apologies. > > > > > > > > Yes > > > > > > > > > this makes sense to save on touching the message, > if > > we're > > > > the > > > > > > > only kip > > > > > > > > > going in, in this release. > > > > > > > > > > > > > > > > > > Cheers > > > > > > > > > Mike > > > > > > > > > > > > > > > > > > Sent using OWA for iPhone > > > > > > > > > ________________________________________ > > > > > > > > > From: Jun Rao <j...@confluent.io> > > > > > > > > > Sent: Tuesday, December 6, 2016 5:22:13 PM > > > > > > > > > To: dev@kafka.apache.org > > > > > > > > > Subject: Re: [VOTE] KIP-87 - Add Compaction > > Tombstone Flag > > > > > > > > > > > > > > > > > > Hi, Michael, > > > > > > > > > > > > > > > > > > 4. Is this updated in the wiki? The text "If the > > magic byte > > > > on > > > > > > > message is > > > > > > > > > 2, the broker should use the tombstone bit for log > > > > compaction." > > > > > > > doesn't > > > > > > > > > seem to have changed. > > > > > > > > > > > > > > > > > > 2. My point is that if we change the message format > > just > > > for > > > > > this > > > > > > > KIP, we > > > > > > > > > should consider whether it's worth optimizing the > > down > > > > > conversion > > > > > > > path > > > > > > > > > (i.e., decide whether a conversion is needed by > just > > > looking > > > > at > > > > > > the > > > > > > > > > tombstone bit in the wrapper message) since > > tombstone will > > > be > > > > > > used > > > > > > > > rarely. > > > > > > > > > However, if the message format change here is > > combined with > > > > > other > > > > > > > KIPs, > > > > > > > > > then this optimization likely won't be needed. The > > latter > > > > > > probably > > > > > > > makes > > > > > > > > > the code simpler. Jiangjie, Mayuresh, what do you > > think? > > > > > > > > > > > > > > > > > > Other than those, +1 from me, > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > > Jun > > > > > > > > > > > > > > > > > > On Tue, Dec 6, 2016 at 8:54 AM, Michael Pearce < > > > > > > > michael.pea...@ig.com> > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > Hi Jun > > > > > > > > > > > > > > > > > > > > do we have your vote on this now? > > > > > > > > > > > > > > > > > > > > Any other concerns? > > > > > > > > > > > > > > > > > > > > Cheers > > > > > > > > > > Mike > > > > > > > > > > > > > > > > > > > > Sent using OWA for iPhone > > > > > > > > > > ________________________________________ > > > > > > > > > > From: Michael Pearce <michael.pea...@ig.com> > > > > > > > > > > Sent: Saturday, December 3, 2016 1:37:45 AM > > > > > > > > > > To: dev@kafka.apache.org > > > > > > > > > > Subject: Re: [VOTE] KIP-87 - Add Compaction > > Tombstone > > > Flag > > > > > > > > > > > > > > > > > > > > Hi Jun, > > > > > > > > > > > > > > > > > > > > Have updated. Thanks again for the feedback. > > > > > > > > > > > > > > > > > > > > Agree yes we should align up when it gets to > that, > > I > > > assume > > > > > > > you’ve > > > > > > > > > flagged > > > > > > > > > > the same in the other KIP? > > > > > > > > > > > > > > > > > > > > I think for now let’s get this KIP approved, then > > we can > > > > > start > > > > > > > the work > > > > > > > > > to > > > > > > > > > > get an initial PR, then we can discuss how to > > align the > > > two > > > > > if > > > > > > > needed. > > > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > > Mike > > > > > > > > > > > > > > > > > > > > On 02/12/2016, 18:26, "Jun Rao" < > j...@confluent.io> > > > wrote: > > > > > > > > > > > > > > > > > > > > Hi, Michael, > > > > > > > > > > > > > > > > > > > > For 2), this is fine. Could you update the > KIP > > wiki > > > to > > > > > make > > > > > > > this > > > > > > > > and > > > > > > > > > > other > > > > > > > > > > points clearer? Other than that, the KIP > looks > > good > > > to > > > > > me. > > > > > > > > > > > > > > > > > > > > An orthogonal thing is that there are other > > KIPs such > > > > as > > > > > > > KIP-98 > > > > > > > > that > > > > > > > > > > also > > > > > > > > > > intend to change the message format. If they > > all get > > > > > > > approved, we > > > > > > > > > > should > > > > > > > > > > think about whether it's better to just bump > > up the > > > > magic > > > > > > > byte once > > > > > > > > > to > > > > > > > > > > incorporate multiple format changes like we > > did in > > > > > > > KIP-31/KIP-32. > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > > > > Jun > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Dec 2, 2016 at 3:19 AM, Michael > Pearce > > < > > > > > > > > > michael.pea...@ig.com> > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > Hi Jao > > > > > > > > > > > > > > > > > > > > > > Thanks for the response. Sorry for slow > > reply, both > > > > > with > > > > > > > personal > > > > > > > > > > sickness > > > > > > > > > > > and also battling some critical issues > > encountered > > > > > since > > > > > > > > upgrading > > > > > > > > > to > > > > > > > > > > > 0.10.1.0 > > > > > > > > > > > > > > > > > > > > > > 1) Thans for spotting, Document error where > > we > > > > branched > > > > > > > this KIP > > > > > > > > > from > > > > > > > > > > > KIP-82, will get that removed. > > > > > > > > > > > 2) Intent is to do this just at the record > > message > > > > > level. > > > > > > > > > > > 3) Thanks for spotting, Will ensure this is > > > > corrected. > > > > > > > > > > > 4) As per discussion thread we will support > > > > tombstone + > > > > > > > null > > > > > > > > value, > > > > > > > > > > > tombstone + non null value, no tombstone + > > null > > > > value. > > > > > > > > > > > 5) I believe this was in the discussion > > thread, > > > > > @Mayuresh > > > > > > > is this > > > > > > > > > > > something we’ve overlooked? I thought we > > would down > > > > > > > convert and > > > > > > > > > > remove the > > > > > > > > > > > value so the old consumer had existing > > behavior, or > > > > is > > > > > > > there > > > > > > > > > > something we > > > > > > > > > > > haven’t thought about? > > > > > > > > > > > > > > > > > > > > > > Cheers > > > > > > > > > > > Mike > > > > > > > > > > > > > > > > > > > > > > On 30/11/2016, 18:12, "Jun Rao" < > > j...@confluent.io> > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > Hi, Michael, > > > > > > > > > > > > > > > > > > > > > > Thanks for the KIP. A few comments > below. > > > > > > > > > > > > > > > > > > > > > > 1. The message format change contains > > > > > "HeadersLength > > > > > > > > Headers". > > > > > > > > > > Is that > > > > > > > > > > > intended? > > > > > > > > > > > > > > > > > > > > > > 2. For compressed messageset, is the > > tombstone > > > > bit > > > > > > > only set > > > > > > > > at > > > > > > > > > > the > > > > > > > > > > > shallow > > > > > > > > > > > level? Do we always leave that bit in > the > > > wrapper > > > > > > > message > > > > > > > > > unset? > > > > > > > > > > An > > > > > > > > > > > alternative is to set the tombstone bit > > in the > > > > > > wrapper > > > > > > > if at > > > > > > > > > > least one > > > > > > > > > > > inner message has the tombstone bit > set. > > This > > > > makes > > > > > > > things a > > > > > > > > > bit > > > > > > > > > > more > > > > > > > > > > > complicated, but we could potentially > > exploit > > > > that > > > > > > for > > > > > > > > > > optimizing down > > > > > > > > > > > conversion. For example, we only need > to > > > convert > > > > > > > messages > > > > > > > > with > > > > > > > > > > magic 2 > > > > > > > > > > > to > > > > > > > > > > > magic 1 if the wrapper's tombstone bit > > is set > > > > > > > (conversion is > > > > > > > > > > always > > > > > > > > > > > needed > > > > > > > > > > > from magic 2 to magic 0). Not sure if > the > > > > > > optimization > > > > > > > is > > > > > > > > worth > > > > > > > > > > the > > > > > > > > > > > complexity though. > > > > > > > > > > > > > > > > > > > > > > 3. The referencing of the new version > of > > > > > > > > > > ProducerRequest/FetchRequest > > > > > > > > > > > is > > > > > > > > > > > inconsistent (v4 vs v3). Since our > > convention > > > > > starts > > > > > > at > > > > > > > > version > > > > > > > > > > at 0, I > > > > > > > > > > > think the new version would be 3. > > > > > > > > > > > > > > > > > > > > > > 4. "If the magic byte on message is 2, > > the > > > broker > > > > > > > should use > > > > > > > > > the > > > > > > > > > > > tombstone > > > > > > > > > > > bit for log compaction." What about > null > > value? > > > > My > > > > > > > > > understanding > > > > > > > > > > is > > > > > > > > > > > that > > > > > > > > > > > null value will be treated the same as > > setting > > > > the > > > > > > > tombstone > > > > > > > > > bit. > > > > > > > > > > > > > > > > > > > > > > 5. For the migration path, it would be > > useful > > > to > > > > > > > describe the > > > > > > > > > > down > > > > > > > > > > > conversion path to consumers (i.e., > > brokers on > > > > > > message > > > > > > > format > > > > > > > > > > 0.10.2 > > > > > > > > > > > and > > > > > > > > > > > consumers on older version). > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > > > > > > Jun > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Nov 29, 2016 at 3:18 AM, > Michael > > > Pearce < > > > > > > > > > > michael.pea...@ig.com > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > Hi All, > > > > > > > > > > > > > > > > > > > > > > > > We have been discussing in the below > > thread > > > and > > > > > > final > > > > > > > > changes > > > > > > > > > > have > > > > > > > > > > > been > > > > > > > > > > > > made to the KIP wiki based on these > > > > discussions. > > > > > > > > > > > > > > > > > > > > > > > > We would now like to put to the vote > > the > > > > > following > > > > > > > KIP: > > > > > > > > > > > > https://cwiki.apache.org/ > > > > > > > confluence/display/KAFKA/KIP- > > > > > > > > 87+-+ > > > > > > > > > > > > Add+Compaction+Tombstone+Flag > > > > > > > > > > > > > > > > > > > > > > > > This kip is for having a distinct > > compaction > > > > > > > attribute > > > > > > > > > > “tombstone” > > > > > > > > > > > flag > > > > > > > > > > > > instead of relying on null value, > > allowing > > > > > non-null > > > > > > > value > > > > > > > > > > delete > > > > > > > > > > > messages. > > > > > > > > > > > > > > > > > > > > > > > > Many thanks, > > > > > > > > > > > > Michael > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On 22/11/2016, 15:52, "Michael > Pearce" > > < > > > > > > > > > michael.pea...@ig.com> > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > Hi Mayuresh, > > > > > > > > > > > > > > > > > > > > > > > > LGTM. Ive just made one small > > adjustment > > > > > > > updating the > > > > > > > > > wire > > > > > > > > > > > protocol to > > > > > > > > > > > > show the magic byte bump. > > > > > > > > > > > > > > > > > > > > > > > > Do we think we’re good to put to > a > > vote? > > > Is > > > > > > > there any > > > > > > > > > > other bits > > > > > > > > > > > > needing discussion? > > > > > > > > > > > > > > > > > > > > > > > > Cheers > > > > > > > > > > > > Mike > > > > > > > > > > > > > > > > > > > > > > > > On 21/11/2016, 18:26, "Mayuresh > > Gharat" < > > > > > > > > > > > gharatmayures...@gmail.com> > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > Hi Michael, > > > > > > > > > > > > > > > > > > > > > > > > I have updated the migration > > section > > > of > > > > > the > > > > > > > KIP. > > > > > > > > Can > > > > > > > > > > you > > > > > > > > > > > please > > > > > > > > > > > > take a look? > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > > > > > > > > Mayuresh > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Nov 18, 2016 at 9:07 > > AM, > > > > Mayuresh > > > > > > > Gharat < > > > > > > > > > > > > gharatmayures...@gmail.com > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > Hi Michael, > > > > > > > > > > > > > > > > > > > > > > > > > > That whilst sending > > tombstone and > > > non > > > > > > null > > > > > > > value, > > > > > > > > > the > > > > > > > > > > > consumer > > > > > > > > > > > > can expect > > > > > > > > > > > > > only to receive the > non-null > > > message > > > > > only > > > > > > > in step > > > > > > > > > > (3) is > > > > > > > > > > > this > > > > > > > > > > > > correct? > > > > > > > > > > > > > ---> I do agree with you > > here. > > > > > > > > > > > > > > > > > > > > > > > > > > Becket, Ismael : can you > guys > > > review > > > > > the > > > > > > > > migration > > > > > > > > > > plan > > > > > > > > > > > listed > > > > > > > > > > > > above using > > > > > > > > > > > > > magic byte? > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > > > > > > > > > > Mayuresh > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Nov 18, 2016 at > 8:58 > > AM, > > > > > Michael > > > > > > > Pearce < > > > > > > > > > > > > michael.pea...@ig.com> > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > >> Many thanks for this > > Mayuresh. I > > > > don't > > > > > > > have any > > > > > > > > > > > objections. > > > > > > > > > > > > >> > > > > > > > > > > > > >> I assume we should state: > > > > > > > > > > > > >> > > > > > > > > > > > > >> That whilst sending > > tombstone and > > > > non > > > > > > null > > > > > > > > value, > > > > > > > > > > the > > > > > > > > > > > consumer > > > > > > > > > > > > can expect > > > > > > > > > > > > >> only to receive the > non-null > > > message > > > > > > only > > > > > > > in > > > > > > > > step > > > > > > > > > > (3) is > > > > > > > > > > > this > > > > > > > > > > > > correct? > > > > > > > > > > > > >> > > > > > > > > > > > > >> Cheers > > > > > > > > > > > > >> Mike > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> Sent using OWA for iPhone > > > > > > > > > > > > >> > > ______________________________ > > > > > > __________ > > > > > > > > > > > > >> From: Mayuresh Gharat < > > > > > > > > gharatmayures...@gmail.com > > > > > > > > > > > > > > > > > > > > > > >> Sent: Thursday, November > > 17, 2016 > > > > > > 5:18:41 > > > > > > > PM > > > > > > > > > > > > >> To: dev@kafka.apache.org > > > > > > > > > > > > >> Subject: Re: [DISCUSS] > > KIP-87 - > > > Add > > > > > > > Compaction > > > > > > > > > > Tombstone > > > > > > > > > > > Flag > > > > > > > > > > > > >> > > > > > > > > > > > > >> Hi Ismael, > > > > > > > > > > > > >> > > > > > > > > > > > > >> Thanks for the > explanation. > > > > > > > > > > > > >> Specially I like this part > > where > > > in > > > > > you > > > > > > > > mentioned > > > > > > > > > > we can > > > > > > > > > > > get > > > > > > > > > > > > rid of the > > > > > > > > > > > > >> older null value support > > for log > > > > > > > compaction > > > > > > > > later > > > > > > > > > > on, > > > > > > > > > > > here : > > > > > > > > > > > > >> We can't change semantics > > of the > > > > > message > > > > > > > format > > > > > > > > > > without > > > > > > > > > > > having > > > > > > > > > > > > a long > > > > > > > > > > > > >> transition period. And we > > can't > > > rely > > > > > > > > > > > > >> on people reading > > documentation or > > > > > > acting > > > > > > > on a > > > > > > > > > > warning for > > > > > > > > > > > > something so > > > > > > > > > > > > >> fundamental. As such, my > > take is > > > > that > > > > > we > > > > > > > need to > > > > > > > > > > bump the > > > > > > > > > > > magic > > > > > > > > > > > > byte. The > > > > > > > > > > > > >> good news is > > > > > > > > > > > > >> that we don't have to > > support all > > > > > > versions > > > > > > > > > forever. > > > > > > > > > > We > > > > > > > > > > > have > > > > > > > > > > > > said that we > > > > > > > > > > > > >> will support direct > > upgrades for 2 > > > > > > years. > > > > > > > That > > > > > > > > > > means that > > > > > > > > > > > > message format > > > > > > > > > > > > >> version n could, in > theory, > > be > > > > > removed 2 > > > > > > > years > > > > > > > > > > after the > > > > > > > > > > > it's > > > > > > > > > > > > introduced. > > > > > > > > > > > > >> > > > > > > > > > > > > >> Just a heads up, I would > > like to > > > > > mention > > > > > > > that > > > > > > > > even > > > > > > > > > > without > > > > > > > > > > > > bumping magic > > > > > > > > > > > > >> byte, we will *NOT* loose > > zero > > > copy > > > > as > > > > > > in > > > > > > > the > > > > > > > > > > client(x+1) > > > > > > > > > > > in my > > > > > > > > > > > > >> explanation > > > > > > > > > > > > >> above will convert > > internally a > > > null > > > > > > > value to > > > > > > > > > have a > > > > > > > > > > > tombstone > > > > > > > > > > > > bit set and > > > > > > > > > > > > >> a tombstone bit set to > have > > a null > > > > > value > > > > > > > > > > automatically > > > > > > > > > > > > internally and by > > > > > > > > > > > > >> the time we move to > version > > (x+2), > > > > the > > > > > > > clients > > > > > > > > > > would have > > > > > > > > > > > > upgraded. > > > > > > > > > > > > >> Obviously if we support a > > request > > > > from > > > > > > > > > consumer(x), > > > > > > > > > > we > > > > > > > > > > > will > > > > > > > > > > > > loose zero > > > > > > > > > > > > >> copy > > > > > > > > > > > > >> but that is the same case > > with > > > magic > > > > > > byte. > > > > > > > > > > > > >> > > > > > > > > > > > > >> But if magic byte bump > > makes life > > > > > easier > > > > > > > for > > > > > > > > > > transition > > > > > > > > > > > for the > > > > > > > > > > > > above > > > > > > > > > > > > >> reasons that you > explained, > > I am > > > OK > > > > > with > > > > > > > it > > > > > > > > since > > > > > > > > > > we are > > > > > > > > > > > going > > > > > > > > > > > > to meet the > > > > > > > > > > > > >> end goal down the road :) > > > > > > > > > > > > >> > > > > > > > > > > > > >> On a side note can we > > update the > > > doc > > > > > > here > > > > > > > on > > > > > > > > magic > > > > > > > > > > byte > > > > > > > > > > > to say > > > > > > > > > > > > that "*it > > > > > > > > > > > > >> should be bumped whenever > > the > > > > message > > > > > > > format is > > > > > > > > > > changed > > > > > > > > > > > or the > > > > > > > > > > > > >> interpretation of message > > format > > > > > (usage > > > > > > > of the > > > > > > > > > > reserved > > > > > > > > > > > bits as > > > > > > > > > > > > well) is > > > > > > > > > > > > >> changed*". > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> Hi Michael, > > > > > > > > > > > > >> > > > > > > > > > > > > >> Here is the update plan > > that we > > > > > > discussed > > > > > > > > offline > > > > > > > > > > > yesterday : > > > > > > > > > > > > >> > > > > > > > > > > > > >> Currently the magic-byte > > which > > > > > > > corresponds to > > > > > > > > the > > > > > > > > > > > > "message.format.version" > > > > > > > > > > > > >> is set to 1. > > > > > > > > > > > > >> > > > > > > > > > > > > >> 1) On broker it will be > set > > to 1 > > > > > > > initially. > > > > > > > > > > > > >> > > > > > > > > > > > > >> 2) When a producer client > > sends a > > > > > > message > > > > > > > with > > > > > > > > > > magic-byte > > > > > > > > > > > = 2, > > > > > > > > > > > > since the > > > > > > > > > > > > >> broker is on magic-byte = > > 1, we > > > will > > > > > > down > > > > > > > > convert > > > > > > > > > > it, > > > > > > > > > > > which > > > > > > > > > > > > means if the > > > > > > > > > > > > >> tombstone bit is set, the > > value > > > will > > > > > be > > > > > > > set to > > > > > > > > > > null. A > > > > > > > > > > > consumer > > > > > > > > > > > > >> understanding magic-byte = > > 1, will > > > > > still > > > > > > > work > > > > > > > > with > > > > > > > > > > this. A > > > > > > > > > > > > consumer > > > > > > > > > > > > >> working > > > > > > > > > > > > >> with magic-byte =2 will > > also be > > > able > > > > > to > > > > > > > > understand > > > > > > > > > > this, > > > > > > > > > > > since > > > > > > > > > > > > it > > > > > > > > > > > > >> understands the tombstone. > > > > > > > > > > > > >> Now there is still the > > question of > > > > > > > supporting a > > > > > > > > > > > non-tombstone > > > > > > > > > > > > and null > > > > > > > > > > > > >> value from producer client > > with > > > > > > > magic-byte = 2.* > > > > > > > > > (I > > > > > > > > > > am > > > > > > > > > > > not sure > > > > > > > > > > > > if we > > > > > > > > > > > > >> should support this. > > Ismael/Becket > > > > can > > > > > > > comment > > > > > > > > > > here)* > > > > > > > > > > > > >> > > > > > > > > > > > > >> 3) When almost all the > > clients > > > have > > > > > > > upgraded, > > > > > > > > the > > > > > > > > > > > > message.format.version > > > > > > > > > > > > >> on > > > > > > > > > > > > >> the broker can be changed > > to 2, > > > > where > > > > > in > > > > > > > the > > > > > > > > down > > > > > > > > > > > conversion in > > > > > > > > > > > > the above > > > > > > > > > > > > >> step will not happen. If > at > > this > > > > point > > > > > > we > > > > > > > get a > > > > > > > > > > consumer > > > > > > > > > > > > request from a > > > > > > > > > > > > >> older consumer, we might > > have to > > > > down > > > > > > > convert > > > > > > > > > where > > > > > > > > > > in we > > > > > > > > > > > loose > > > > > > > > > > > > zero copy, > > > > > > > > > > > > >> but these cases should be > > rare. > > > > > > > > > > > > >> > > > > > > > > > > > > >> Becket can you review this > > plan > > > and > > > > > add > > > > > > > more > > > > > > > > > > details if I > > > > > > > > > > > have > > > > > > > > > > > > >> missed/wronged something, > > before > > > we > > > > > put > > > > > > > it on > > > > > > > > KIP. > > > > > > > > > > > > >> > > > > > > > > > > > > >> Thanks, > > > > > > > > > > > > >> > > > > > > > > > > > > >> Mayuresh > > > > > > > > > > > > >> > > > > > > > > > > > > >> On Wed, Nov 16, 2016 at > > 11:07 PM, > > > > > > Michael > > > > > > > > Pearce < > > > > > > > > > > > > michael.pea...@ig.com> > > > > > > > > > > > > >> wrote: > > > > > > > > > > > > >> > > > > > > > > > > > > >> > Thanks guys, for > > discussing this > > > > > > > offline and > > > > > > > > > > getting > > > > > > > > > > > some > > > > > > > > > > > > consensus. > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > So its clear for myself > > and > > > others > > > > > > what > > > > > > > is > > > > > > > > > > proposed now > > > > > > > > > > > (i > > > > > > > > > > > > think i > > > > > > > > > > > > >> > understand, but want to > > make > > > sure) > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > Could i ask either > > directly > > > update > > > > > the > > > > > > > kip to > > > > > > > > > > detail the > > > > > > > > > > > > migration > > > > > > > > > > > > >> > strategy, or (re-)state > > your > > > > offline > > > > > > > discussed > > > > > > > > > and > > > > > > > > > > > agreed > > > > > > > > > > > > migration > > > > > > > > > > > > >> > strategy based on a > magic > > byte > > > is > > > > in > > > > > > > this > > > > > > > > > thread. > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > The main original driver > > for the > > > > KIP > > > > > > > was to > > > > > > > > > > support > > > > > > > > > > > > compaction where > > > > > > > > > > > > >> value > > > > > > > > > > > > >> > isn't null, based off > the > > > > > discussions > > > > > > on > > > > > > > > KIP-82 > > > > > > > > > > thread. > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > We should be able to > > support > > > > > > > non-tombstone + > > > > > > > > > null > > > > > > > > > > value > > > > > > > > > > > by the > > > > > > > > > > > > >> completion > > > > > > > > > > > > >> > of the KIP, as we noted > > when > > > > > > discussing > > > > > > > this > > > > > > > > > kip, > > > > > > > > > > having > > > > > > > > > > > > logic based on > > > > > > > > > > > > >> a > > > > > > > > > > > > >> > null value isn't very > > clean and > > > > also > > > > > > > separates > > > > > > > > > the > > > > > > > > > > > concerns. > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > As discussed already > > though we > > > can > > > > > > > split this > > > > > > > > > into > > > > > > > > > > > KIP-87a > > > > > > > > > > > > and KIP-87b > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > Where we look to deliver > > KIP-87a > > > > on > > > > > a > > > > > > > > compacted > > > > > > > > > > topic > > > > > > > > > > > (to > > > > > > > > > > > > address the > > > > > > > > > > > > >> > immediate issues) > > > > > > > > > > > > >> > * tombstone + null value > > > > > > > > > > > > >> > * tombstone + non-null > > value > > > > > > > > > > > > >> > * non-tombstone + > > non-null value > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > Then we can discuss once > > KIP-87a > > > > is > > > > > > > completed > > > > > > > > > > options > > > > > > > > > > > later > > > > > > > > > > > > and how we > > > > > > > > > > > > >> > support the second part > > KIP-87b > > > to > > > > > > > deliver: > > > > > > > > > > > > >> > * non-tombstone + null > > value > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > Cheers > > > > > > > > > > > > >> > Mike > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > ______________________________ > > > > > > > __________ > > > > > > > > > > > > >> > From: Becket Qin < > > > > > > becket....@gmail.com> > > > > > > > > > > > > >> > Sent: Thursday, November > > 17, > > > 2016 > > > > > 1:43 > > > > > > > AM > > > > > > > > > > > > >> > To: > dev@kafka.apache.org > > > > > > > > > > > > >> > Subject: Re: [DISCUSS] > > KIP-87 - > > > > Add > > > > > > > Compaction > > > > > > > > > > > Tombstone Flag > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > Renu, Mayuresh and I had > > an > > > > offline > > > > > > > > discussion, > > > > > > > > > > and > > > > > > > > > > > following > > > > > > > > > > > > is a brief > > > > > > > > > > > > >> > summary. > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > 1. We agreed that not > > bumping up > > > > > magic > > > > > > > value > > > > > > > > may > > > > > > > > > > result > > > > > > > > > > > in > > > > > > > > > > > > losing zero > > > > > > > > > > > > >> copy > > > > > > > > > > > > >> > during migration. > > > > > > > > > > > > >> > 2. Given that bumping up > > magic > > > > value > > > > > > is > > > > > > > almost > > > > > > > > > > free and > > > > > > > > > > > has > > > > > > > > > > > > benefit of > > > > > > > > > > > > >> > avoiding potential > > performance > > > > > issue. > > > > > > > It is > > > > > > > > > > probably > > > > > > > > > > > worth > > > > > > > > > > > > doing. > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > One issue we still need > > to think > > > > > about > > > > > > > is > > > > > > > > > whether > > > > > > > > > > we > > > > > > > > > > > want to > > > > > > > > > > > > support a > > > > > > > > > > > > >> > non-tombstone message > > with null > > > > > value. > > > > > > > > > > > > >> > Currently it is not > > supported by > > > > > > Kafka. > > > > > > > If we > > > > > > > > > > allow a > > > > > > > > > > > > non-tombstone null > > > > > > > > > > > > >> > value message to exist > > after > > > > KIP-87. > > > > > > The > > > > > > > > problem > > > > > > > > > > is > > > > > > > > > > > that such > > > > > > > > > > > > message > > > > > > > > > > > > >> will > > > > > > > > > > > > >> > not be supported by the > > > consumers > > > > > > prior > > > > > > > to > > > > > > > > > KIP-87. > > > > > > > > > > > Because a > > > > > > > > > > > > null value > > > > > > > > > > > > >> > will always be > > interpreted to a > > > > > > > tombstone. > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > One option is that we > > keep the > > > > > current > > > > > > > way, > > > > > > > > i.e. > > > > > > > > > > do not > > > > > > > > > > > > support such > > > > > > > > > > > > >> > message. It would be > good > > to > > > know > > > > if > > > > > > > there is > > > > > > > > a > > > > > > > > > > > concrete use > > > > > > > > > > > > case for > > > > > > > > > > > > >> such > > > > > > > > > > > > >> > message. If there is > not, > > we can > > > > > > > probably just > > > > > > > > > not > > > > > > > > > > > support it. > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > Thanks, > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > JIangjie (Becket) Qin > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > On Wed, Nov 16, 2016 at > > 1:28 PM, > > > > > > > Mayuresh > > > > > > > > > Gharat < > > > > > > > > > > > > >> > > > gharatmayures...@gmail.com > > > > > > > > > > > > >> > > wrote: > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > Hi Ismael, > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > This is something I > can > > think > > > of > > > > > for > > > > > > > > migration > > > > > > > > > > plan: > > > > > > > > > > > > >> > > So the migration plan > > can look > > > > > > > something > > > > > > > > like > > > > > > > > > > this, > > > > > > > > > > > with up > > > > > > > > > > > > >> conversion : > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > 1) Currently lets say > > we have > > > > > Broker > > > > > > > at > > > > > > > > > version > > > > > > > > > > x. > > > > > > > > > > > > >> > > 2) Currently we have > > clients > > > at > > > > > > > version x. > > > > > > > > > > > > >> > > 3) a) We move the > > version to > > > > > > > Broker(x+1) : > > > > > > > > > > supports > > > > > > > > > > > both > > > > > > > > > > > > tombstone and > > > > > > > > > > > > >> > null > > > > > > > > > > > > >> > > for log compaction. > > > > > > > > > > > > >> > > b) We upgrade the > > client > > > to > > > > > > > version > > > > > > > > > > client(x+1) : > > > > > > > > > > > if in > > > > > > > > > > > > the > > > > > > > > > > > > >> producer > > > > > > > > > > > > >> > > client(x+1) the value > > is set > > > to > > > > > > null, > > > > > > > we > > > > > > > > will > > > > > > > > > > > automatically > > > > > > > > > > > > set the > > > > > > > > > > > > >> > > Tombstone bit > > internally. If > > > the > > > > > > > producer > > > > > > > > > > client(x+1) > > > > > > > > > > > sets > > > > > > > > > > > > the > > > > > > > > > > > > >> tombstone > > > > > > > > > > > > >> > > itself, well and good. > > For > > > > > producer > > > > > > > > client(x), > > > > > > > > > > the > > > > > > > > > > > broker > > > > > > > > > > > > will up > > > > > > > > > > > > >> convert > > > > > > > > > > > > >> > > to have the tombstone > > bit. > > > > > > > Broker(x+1) is > > > > > > > > > > supporting > > > > > > > > > > > both. > > > > > > > > > > > > Consumer > > > > > > > > > > > > >> > > client(x+1) will be > > aware of > > > > this > > > > > > and > > > > > > > should > > > > > > > > > be > > > > > > > > > > able > > > > > > > > > > > to > > > > > > > > > > > > handle this. > > > > > > > > > > > > >> For > > > > > > > > > > > > >> > > consumer client(x) we > > will > > > down > > > > > > > convert the > > > > > > > > > > message > > > > > > > > > > > on the > > > > > > > > > > > > broker > > > > > > > > > > > > >> side. > > > > > > > > > > > > >> > > c) At this point > we > > will > > > > have > > > > > to > > > > > > > > specify a > > > > > > > > > > > warning or > > > > > > > > > > > > clearly > > > > > > > > > > > > >> specify > > > > > > > > > > > > >> > > in docs that this > > behavior is > > > > > about > > > > > > > to be > > > > > > > > > > changed for > > > > > > > > > > > log > > > > > > > > > > > > compaction. > > > > > > > > > > > > >> > > 4) a) In next release > > of the > > > > > > > Broker(x+2), we > > > > > > > > > > say that > > > > > > > > > > > only > > > > > > > > > > > > Tombstone > > > > > > > > > > > > >> is > > > > > > > > > > > > >> > > used for log > compaction > > on the > > > > > > Broker > > > > > > > side. > > > > > > > > > > > Clients(x+1) > > > > > > > > > > > > still is > > > > > > > > > > > > >> > > supported. > > > > > > > > > > > > >> > > b) We upgrade the > > client > > > to > > > > > > > version > > > > > > > > > > client(x+2) : > > > > > > > > > > > if > > > > > > > > > > > > value is set > > > > > > > > > > > > >> to > > > > > > > > > > > > >> > > null, tombstone will > > not be > > > set > > > > > > > > automatically. > > > > > > > > > > The > > > > > > > > > > > client > > > > > > > > > > > > will have to > > > > > > > > > > > > >> > call > > > > > > > > > > > > >> > > setTombstone() to > > actually set > > > > the > > > > > > > > tombstone. > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > We should compare this > > > migration > > > > > > plan > > > > > > > with > > > > > > > > the > > > > > > > > > > > migration > > > > > > > > > > > > plan for > > > > > > > > > > > > >> magic > > > > > > > > > > > > >> > > byte bump and do > > whatever > > > looks > > > > > > good. > > > > > > > > > > > > >> > > I am just worried that > > if we > > > go > > > > > down > > > > > > > magic > > > > > > > > > byte > > > > > > > > > > route, > > > > > > > > > > > > unless I am > > > > > > > > > > > > >> > missing > > > > > > > > > > > > >> > > something, it sounds > > like > > > kafka > > > > > will > > > > > > > be > > > > > > > > stuck > > > > > > > > > > with > > > > > > > > > > > > supporting both > > > > > > > > > > > > >> null > > > > > > > > > > > > >> > > value and tombstone > bit > > for > > > log > > > > > > > compaction > > > > > > > > for > > > > > > > > > > life > > > > > > > > > > > long, > > > > > > > > > > > > which does > > > > > > > > > > > > >> not > > > > > > > > > > > > >> > > look like a good end > > state. > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > Thanks, > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > Mayuresh > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > On Wed, Nov 16, 2016 > at > > 9:32 > > > AM, > > > > > > > Mayuresh > > > > > > > > > > Gharat < > > > > > > > > > > > > >> > > > > gharatmayures...@gmail.com > > > > > > > > > > > > >> > > > wrote: > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > Hi Ismael, > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > That's a very good > > point > > > > which I > > > > > > > might > > > > > > > > have > > > > > > > > > > not > > > > > > > > > > > > considered earlier. > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > Here is a plan that > I > > can > > > > think > > > > > > of: > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > Stage 1) The broker > > from now > > > > on, > > > > > > up > > > > > > > > converts > > > > > > > > > > the > > > > > > > > > > > message > > > > > > > > > > > > to have the > > > > > > > > > > > > >> > > > tombstone marker. > The > > log > > > > > > compaction > > > > > > > > thread > > > > > > > > > > does log > > > > > > > > > > > > compaction > > > > > > > > > > > > >> based > > > > > > > > > > > > >> > on > > > > > > > > > > > > >> > > > both null and > > tombstone > > > > marker. > > > > > > > This is > > > > > > > > our > > > > > > > > > > > transition > > > > > > > > > > > > period. > > > > > > > > > > > > >> > > > Stage 2) The next > > release we > > > > > only > > > > > > > say that > > > > > > > > > log > > > > > > > > > > > compaction > > > > > > > > > > > > is based > > > > > > > > > > > > >> on > > > > > > > > > > > > >> > > > tombstone marker. > > (Open > > > source > > > > > > > kafka makes > > > > > > > > > > this as a > > > > > > > > > > > > policy). By > > > > > > > > > > > > >> this > > > > > > > > > > > > >> > > time, > > > > > > > > > > > > >> > > > the organization > > which is > > > > moving > > > > > > to > > > > > > > this > > > > > > > > > > release > > > > > > > > > > > will be > > > > > > > > > > > > sure that > > > > > > > > > > > > >> they > > > > > > > > > > > > >> > > > have gone through > the > > entire > > > > > > > transition > > > > > > > > > > period. > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > My only goal of > doing > > this > > > is > > > > > that > > > > > > > Kafka > > > > > > > > > > clearly > > > > > > > > > > > > specifies the end > > > > > > > > > > > > >> > state > > > > > > > > > > > > >> > > > about what log > > compaction > > > > means > > > > > > (is > > > > > > > it > > > > > > > > null > > > > > > > > > > value > > > > > > > > > > > or a > > > > > > > > > > > > tombstone > > > > > > > > > > > > >> > marker, > > > > > > > > > > > > >> > > > but not both). > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > What do you think? > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > Thanks, > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > Mayuresh > > > > > > > > > > > > >> > > > . > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > On Wed, Nov 16, 2016 > > at 9:17 > > > > AM, > > > > > > > Ismael > > > > > > > > > Juma < > > > > > > > > > > > > ism...@juma.me.uk> > > > > > > > > > > > > >> > wrote: > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > >> One comment below. > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > >> > > >> On Wed, Nov 16, > 2016 > > at > > > 5:08 > > > > > PM, > > > > > > > Mayuresh > > > > > > > > > > Gharat < > > > > > > > > > > > > >> > > >> > > gharatmayures...@gmail.com > > > > > > > > > > > > >> > > >> > wrote: > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > >> > > >> > - If we don't > > bump up > > > > the > > > > > > > magic > > > > > > > > byte, > > > > > > > > > > on the > > > > > > > > > > > broker > > > > > > > > > > > > side, the > > > > > > > > > > > > >> > > broker > > > > > > > > > > > > >> > > >> > will always > > have to > > > look > > > > > at > > > > > > > both > > > > > > > > > > tombstone > > > > > > > > > > > bit and > > > > > > > > > > > > the value > > > > > > > > > > > > >> when > > > > > > > > > > > > >> > > do > > > > > > > > > > > > >> > > >> the > > > > > > > > > > > > >> > > >> > compaction. > > Assuming > > > we > > > > do > > > > > > > not bump > > > > > > > > up > > > > > > > > > > the > > > > > > > > > > > magic > > > > > > > > > > > > byte, > > > > > > > > > > > > >> > > >> > imagine the > > broker > > > sees > > > > a > > > > > > > message > > > > > > > > > which > > > > > > > > > > does > > > > > > > > > > > not > > > > > > > > > > > > have a > > > > > > > > > > > > >> tombstone > > > > > > > > > > > > >> > > bit > > > > > > > > > > > > >> > > >> > set. The > broker > > does > > > not > > > > > > know > > > > > > > when > > > > > > > > the > > > > > > > > > > > message was > > > > > > > > > > > > produced > > > > > > > > > > > > >> (i.e. > > > > > > > > > > > > >> > > >> > whether > > > > > > > > > > > > >> > > >> > the message > has > > been > > > up > > > > > > > converted or > > > > > > > > > > not), it > > > > > > > > > > > has > > > > > > > > > > > > to take a > > > > > > > > > > > > >> > further > > > > > > > > > > > > >> > > >> > look at > > > > > > > > > > > > >> > > >> > the value to > > see if it > > > > is > > > > > > > null or > > > > > > > > not > > > > > > > > > in > > > > > > > > > > > order to > > > > > > > > > > > > determine > > > > > > > > > > > > >> if it > > > > > > > > > > > > >> > > is > > > > > > > > > > > > >> > > >> a > > > > > > > > > > > > >> > > >> > tombstone. The > > same > > > > logic > > > > > > has > > > > > > > to be > > > > > > > > > put > > > > > > > > > > on the > > > > > > > > > > > > consumer as > > > > > > > > > > > > >> well > > > > > > > > > > > > >> > > >> because > > > > > > > > > > > > >> > > >> > the > > > > > > > > > > > > >> > > >> > consumer does > > not know > > > > if > > > > > > the > > > > > > > > message > > > > > > > > > > has > > > > > > > > > > > been up > > > > > > > > > > > > converted or > > > > > > > > > > > > >> > not. > > > > > > > > > > > > >> > > >> > - If we > > upconvert > > > > while > > > > > > > > appending, > > > > > > > > > > this is > > > > > > > > > > > not > > > > > > > > > > > > the case, > > > > > > > > > > > > >> > right? > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > >> > > >> If I understand you > > > > correctly, > > > > > > > this is > > > > > > > > not > > > > > > > > > > > sufficient > > > > > > > > > > > > because the > > > > > > > > > > > > >> log > > > > > > > > > > > > >> > > may > > > > > > > > > > > > >> > > >> have messages > > appended > > > before > > > > > it > > > > > > > was > > > > > > > > > > upgraded to > > > > > > > > > > > include > > > > > > > > > > > > KIP-87. > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > >> > > >> Ismael > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > -- > > > > > > > > > > > > >> > > > -Regards, > > > > > > > > > > > > >> > > > Mayuresh R. Gharat > > > > > > > > > > > > >> > > > (862) 250-7125 > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > -- > > > > > > > > > > > > >> > > -Regards, > > > > > > > > > > > > >> > > Mayuresh R. Gharat > > > > > > > > > > > > >> > > (862) 250-7125 > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > The information > contained > > in > > > this > > > > > > email > > > > > > > is > > > > > > > > > > strictly > > > > > > > > > > > > confidential and for > > > > > > > > > > > > >> > the use of the addressee > > only, > > > > > unless > > > > > > > > otherwise > > > > > > > > > > > indicated. If > > > > > > > > > > > > you are > > > > > > > > > > > > >> not > > > > > > > > > > > > >> > the intended recipient, > > please > > > do > > > > > not > > > > > > > read, > > > > > > > > > copy, > > > > > > > > > > use or > > > > > > > > > > > > disclose to > > > > > > > > > > > > >> others > > > > > > > > > > > > >> > this message or any > > attachment. > > > > > Please > > > > > > > also > > > > > > > > > > notify the > > > > > > > > > > > sender > > > > > > > > > > > > by > > > > > > > > > > > > >> replying > > > > > > > > > > > > >> > to this email or by > > telephone > > > > > (+44(020 > > > > > > > 7896 > > > > > > > > > 0011) > > > > > > > > > > and > > > > > > > > > > > then > > > > > > > > > > > > delete the > > > > > > > > > > > > >> email > > > > > > > > > > > > >> > and any copies of it. > > Opinions, > > > > > > > conclusion > > > > > > > > (etc) > > > > > > > > > > that > > > > > > > > > > > do not > > > > > > > > > > > > relate to > > > > > > > > > > > > >> the > > > > > > > > > > > > >> > official business of > this > > > company > > > > > > shall > > > > > > > be > > > > > > > > > > understood as > > > > > > > > > > > > neither given > > > > > > > > > > > > >> nor > > > > > > > > > > > > >> > endorsed by it. IG is a > > trading > > > > name > > > > > > of > > > > > > > IG > > > > > > > > > Markets > > > > > > > > > > > Limited (a > > > > > > > > > > > > company > > > > > > > > > > > > >> > registered in England > and > > Wales, > > > > > > company > > > > > > > > number > > > > > > > > > > > 04008957) and > > > > > > > > > > > > IG Index > > > > > > > > > > > > >> > Limited (a company > > registered in > > > > > > > England and > > > > > > > > > > Wales, > > > > > > > > > > > company > > > > > > > > > > > > number > > > > > > > > > > > > >> > 01190902). Registered > > address at > > > > > > Cannon > > > > > > > Bridge > > > > > > > > > > House, 25 > > > > > > > > > > > > Dowgate Hill, > > > > > > > > > > > > >> > London EC4R 2YA. Both IG > > Markets > > > > > > Limited > > > > > > > > > (register > > > > > > > > > > > number > > > > > > > > > > > > 195355) and IG > > > > > > > > > > > > >> > Index Limited (register > > number > > > > > 114059) > > > > > > > are > > > > > > > > > > authorised > > > > > > > > > > > and > > > > > > > > > > > > regulated by > > > > > > > > > > > > >> the > > > > > > > > > > > > >> > Financial Conduct > > Authority. > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> -- > > > > > > > > > > > > >> -Regards, > > > > > > > > > > > > >> Mayuresh R. Gharat > > > > > > > > > > > > >> (862) 250-7125 > > > > > > > > > > > > >> The information contained > > in this > > > > > email > > > > > > is > > > > > > > > > strictly > > > > > > > > > > > > confidential and for > > > > > > > > > > > > >> the use of the addressee > > only, > > > > unless > > > > > > > otherwise > > > > > > > > > > > indicated. If > > > > > > > > > > > > you are not > > > > > > > > > > > > >> the intended recipient, > > please do > > > > not > > > > > > > read, > > > > > > > > copy, > > > > > > > > > > use or > > > > > > > > > > > > disclose to others > > > > > > > > > > > > >> this message or any > > attachment. > > > > Please > > > > > > > also > > > > > > > > notify > > > > > > > > > > the > > > > > > > > > > > sender > > > > > > > > > > > > by replying > > > > > > > > > > > > >> to this email or by > > telephone > > > > (+44(020 > > > > > > > 7896 > > > > > > > > 0011) > > > > > > > > > > and then > > > > > > > > > > > > delete the email > > > > > > > > > > > > >> and any copies of it. > > Opinions, > > > > > > > conclusion (etc) > > > > > > > > > > that do > > > > > > > > > > > not > > > > > > > > > > > > relate to the > > > > > > > > > > > > >> official business of this > > company > > > > > shall > > > > > > be > > > > > > > > > > understood as > > > > > > > > > > > > neither given nor > > > > > > > > > > > > >> endorsed by it. IG is a > > trading > > > name > > > > > of > > > > > > IG > > > > > > > > Markets > > > > > > > > > > > Limited (a > > > > > > > > > > > > company > > > > > > > > > > > > >> registered in England and > > Wales, > > > > > company > > > > > > > number > > > > > > > > > > 04008957) > > > > > > > > > > > and > > > > > > > > > > > > IG Index > > > > > > > > > > > > >> Limited (a company > > registered in > > > > > England > > > > > > > and > > > > > > > > > Wales, > > > > > > > > > > > company > > > > > > > > > > > > number > > > > > > > > > > > > >> 01190902). Registered > > address at > > > > > Cannon > > > > > > > Bridge > > > > > > > > > > House, 25 > > > > > > > > > > > > Dowgate Hill, > > > > > > > > > > > > >> London EC4R 2YA. Both IG > > Markets > > > > > Limited > > > > > > > > (register > > > > > > > > > > number > > > > > > > > > > > > 195355) and IG > > > > > > > > > > > > >> Index Limited (register > > number > > > > 114059) > > > > > > are > > > > > > > > > > authorised and > > > > > > > > > > > > regulated by the > > > > > > > > > > > > >> Financial Conduct > Authority. > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > -Regards, > > > > > > > > > > > > > Mayuresh R. Gharat > > > > > > > > > > > > > (862) 250-7125 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > -Regards, > > > > > > > > > > > > Mayuresh R. Gharat > > > > > > > > > > > > (862) 250-7125 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > The information contained in this > > email > > > is > > > > > > > strictly > > > > > > > > > > confidential > > > > > > > > > > > and > > > > > > > > > > > > for the use of the addressee only, > > unless > > > > > otherwise > > > > > > > > > indicated. > > > > > > > > > > If > > > > > > > > > > > you are > > > > > > > > > > > > not the intended recipient, please do > > not > > > read, > > > > > > > copy, use > > > > > > > > or > > > > > > > > > > > disclose to > > > > > > > > > > > > others this message or any > attachment. > > Please > > > > > also > > > > > > > notify > > > > > > > > the > > > > > > > > > > sender > > > > > > > > > > > by > > > > > > > > > > > > replying to this email or by > telephone > > > (+44(020 > > > > > > 7896 > > > > > > > 0011) > > > > > > > > > and > > > > > > > > > > then > > > > > > > > > > > delete > > > > > > > > > > > > the email and any copies of it. > > Opinions, > > > > > > conclusion > > > > > > > (etc) > > > > > > > > > > that do > > > > > > > > > > > not > > > > > > > > > > > > relate to the official business of > this > > > company > > > > > > > shall be > > > > > > > > > > understood > > > > > > > > > > > as > > > > > > > > > > > > neither given nor endorsed by it. IG > > is a > > > > trading > > > > > > > name of > > > > > > > > IG > > > > > > > > > > Markets > > > > > > > > > > > > Limited (a company registered in > > England and > > > > > Wales, > > > > > > > company > > > > > > > > > > number > > > > > > > > > > > > 04008957) and IG Index Limited (a > > company > > > > > > registered > > > > > > > in > > > > > > > > > > England and > > > > > > > > > > > Wales, > > > > > > > > > > > > company number 01190902). Registered > > address > > > at > > > > > > > Cannon > > > > > > > > Bridge > > > > > > > > > > House, > > > > > > > > > > > 25 > > > > > > > > > > > > Dowgate Hill, London EC4R 2YA. Both > IG > > > Markets > > > > > > > Limited > > > > > > > > > > (register > > > > > > > > > > > number > > > > > > > > > > > > 195355) and IG Index Limited > (register > > number > > > > > > > 114059) are > > > > > > > > > > authorised > > > > > > > > > > > and > > > > > > > > > > > > regulated by the Financial Conduct > > Authority. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > The information contained in this email is > > strictly > > > > > > > confidential > > > > > > > > > and > > > > > > > > > > for > > > > > > > > > > > the use of the addressee only, unless > > otherwise > > > > > > indicated. > > > > > > > If you > > > > > > > > > > are not > > > > > > > > > > > the intended recipient, please do not read, > > copy, > > > use > > > > > or > > > > > > > disclose > > > > > > > > > to > > > > > > > > > > others > > > > > > > > > > > this message or any attachment. Please also > > notify > > > > the > > > > > > > sender by > > > > > > > > > > replying > > > > > > > > > > > to this email or by telephone (+44(020 7896 > > 0011 > > > <020%207896%200011>) and > > > > > > then > > > > > > > delete > > > > > > > > > > the email > > > > > > > > > > > and any copies of it. Opinions, conclusion > > (etc) > > > that > > > > > do > > > > > > > not > > > > > > > > relate > > > > > > > > > > to the > > > > > > > > > > > official business of this company shall be > > > understood > > > > > as > > > > > > > neither > > > > > > > > > > given nor > > > > > > > > > > > endorsed by it. IG is a trading name of IG > > Markets > > > > > > Limited > > > > > > > (a > > > > > > > > > company > > > > > > > > > > > registered in England and Wales, company > > number > > > > > 04008957) > > > > > > > and IG > > > > > > > > > > Index > > > > > > > > > > > Limited (a company registered in England > and > > Wales, > > > > > > company > > > > > > > > number > > > > > > > > > > > 01190902). Registered address at Cannon > > Bridge > > > House, > > > > > 25 > > > > > > > Dowgate > > > > > > > > > > Hill, > > > > > > > > > > > London EC4R 2YA. Both IG Markets Limited > > (register > > > > > number > > > > > > > 195355) > > > > > > > > > > and IG > > > > > > > > > > > Index Limited (register number 114059) are > > > authorised > > > > > and > > > > > > > > regulated > > > > > > > > > > by the > > > > > > > > > > > Financial Conduct Authority. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > The information contained in this email is > strictly > > > > > > confidential > > > > > > > and > > > > > > > > for > > > > > > > > > > the use of the addressee only, unless otherwise > > > indicated. > > > > If > > > > > > > you are > > > > > > > > not > > > > > > > > > > the intended recipient, please do not read, copy, > > use or > > > > > > > disclose to > > > > > > > > > others > > > > > > > > > > this message or any attachment. Please also > notify > > the > > > > sender > > > > > > by > > > > > > > > replying > > > > > > > > > > to this email or by telephone (+44(020 7896 0011 > > > <020%207896%200011>) and then > > > > > > > delete the > > > > > > > > > email > > > > > > > > > > and any copies of it. Opinions, conclusion (etc) > > that do > > > > not > > > > > > > relate to > > > > > > > > > the > > > > > > > > > > official business of this company shall be > > understood as > > > > > > neither > > > > > > > given > > > > > > > > > nor > > > > > > > > > > endorsed by it. IG is a trading name of IG > Markets > > > Limited > > > > (a > > > > > > > company > > > > > > > > > > registered in England and Wales, company number > > 04008957) > > > > and > > > > > > IG > > > > > > > Index > > > > > > > > > > Limited (a company registered in England and > Wales, > > > company > > > > > > > number > > > > > > > > > > 01190902). Registered address at Cannon Bridge > > House, 25 > > > > > > Dowgate > > > > > > > Hill, > > > > > > > > > > London EC4R 2YA. Both IG Markets Limited > (register > > number > > > > > > > 195355) and > > > > > > > > IG > > > > > > > > > > Index Limited (register number 114059) are > > authorised and > > > > > > > regulated by > > > > > > > > > the > > > > > > > > > > Financial Conduct Authority. > > > > > > > > > > > > > > > > > > > The information contained in this email is strictly > > > > > confidential > > > > > > > and for > > > > > > > > > the use of the addressee only, unless otherwise > > indicated. > > > If > > > > > you > > > > > > > are not > > > > > > > > > the intended recipient, please do not read, copy, > > use or > > > > > disclose > > > > > > > to > > > > > > > > others > > > > > > > > > this message or any attachment. Please also notify > > the > > > sender > > > > > by > > > > > > > replying > > > > > > > > > to this email or by telephone (+44(020 7896 0011 > > > <020%207896%200011>) and then > > > > > delete > > > > > > > the > > > > > > > > email > > > > > > > > > and any copies of it. Opinions, conclusion (etc) > > that do > > > not > > > > > > > relate to > > > > > > > > the > > > > > > > > > official business of this company shall be > > understood as > > > > > neither > > > > > > > given > > > > > > > > nor > > > > > > > > > endorsed by it. IG is a trading name of IG Markets > > Limited > > > (a > > > > > > > company > > > > > > > > > registered in England and Wales, company number > > 04008957) > > > and > > > > > IG > > > > > > > Index > > > > > > > > > Limited (a company registered in England and Wales, > > company > > > > > > number > > > > > > > > > 01190902). Registered address at Cannon Bridge > > House, 25 > > > > > Dowgate > > > > > > > Hill, > > > > > > > > > London EC4R 2YA. Both IG Markets Limited (register > > number > > > > > 195355) > > > > > > > and IG > > > > > > > > > Index Limited (register number 114059) are > > authorised and > > > > > > > regulated by > > > > > > > > the > > > > > > > > > Financial Conduct Authority. > > > > > > > > > > > > > > > > > > > > > > > > > > The information contained in this email is strictly > > > > confidential > > > > > > and > > > > > > > for > > > > > > > > the use of the addressee only, unless otherwise > > indicated. If > > > > you > > > > > > > are not > > > > > > > > the intended recipient, please do not read, copy, use > > or > > > > disclose > > > > > > to > > > > > > > others > > > > > > > > this message or any attachment. Please also notify > the > > sender > > > > by > > > > > > > replying > > > > > > > > to this email or by telephone (+44(020 7896 0011 > > > <020%207896%200011>) and then > > > > delete > > > > > > > the email > > > > > > > > and any copies of it. Opinions, conclusion (etc) that > > do not > > > > > relate > > > > > > > to the > > > > > > > > official business of this company shall be understood > > as > > > > neither > > > > > > > given nor > > > > > > > > endorsed by it. IG is a trading name of IG Markets > > Limited (a > > > > > > company > > > > > > > > registered in England and Wales, company number > > 04008957) and > > > > IG > > > > > > > Index > > > > > > > > Limited (a company registered in England and Wales, > > company > > > > > number > > > > > > > > 01190902). Registered address at Cannon Bridge House, > > 25 > > > > Dowgate > > > > > > > Hill, > > > > > > > > London EC4R 2YA. Both IG Markets Limited (register > > number > > > > 195355) > > > > > > > and IG > > > > > > > > Index Limited (register number 114059) are authorised > > and > > > > > regulated > > > > > > > by the > > > > > > > > Financial Conduct Authority. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > The information contained in this email is strictly > > confidential > > > and > > > > > for > > > > > > > the use of the addressee only, unless otherwise indicated. > > If you > > > are > > > > > not > > > > > > > the intended recipient, please do not read, copy, use or > > disclose > > > to > > > > > > others > > > > > > > this message or any attachment. Please also notify the > > sender by > > > > > replying > > > > > > > to this email or by telephone (+44(020 7896 0011 > > > <020%207896%200011>) and then delete the > > > > > > email > > > > > > > and any copies of it. Opinions, conclusion (etc) that do > not > > relate > > > > to > > > > > > the > > > > > > > official business of this company shall be understood as > > neither > > > > given > > > > > > nor > > > > > > > endorsed by it. IG is a trading name of IG Markets Limited > (a > > > company > > > > > > > registered in England and Wales, company number 04008957) > > and IG > > > > Index > > > > > > > Limited (a company registered in England and Wales, company > > number > > > > > > > 01190902). Registered address at Cannon Bridge House, 25 > > Dowgate > > > > Hill, > > > > > > > London EC4R 2YA. Both IG Markets Limited (register number > > 195355) > > > and > > > > > IG > > > > > > > Index Limited (register number 114059) are authorised and > > regulated > > > > by > > > > > > the > > > > > > > Financial Conduct Authority. > > > > > > > > > > > > > The information contained in this email is strictly > > confidential and > > > > for > > > > > > the use of the addressee only, unless otherwise indicated. If > > you are > > > > not > > > > > > the intended recipient, please do not read, copy, use or > > disclose to > > > > > others > > > > > > this message or any attachment. Please also notify the sender > > by > > > > replying > > > > > > to this email or by telephone (+44(020 7896 0011 > > <020%207896%200011>) > > > and then delete the > > > > > email > > > > > > and any copies of it. Opinions, conclusion (etc) that do not > > relate > > > to > > > > > the > > > > > > official business of this company shall be understood as > > neither > > > given > > > > > nor > > > > > > endorsed by it. IG is a trading name of IG Markets Limited (a > > company > > > > > > registered in England and Wales, company number 04008957) and > > IG > > > Index > > > > > > Limited (a company registered in England and Wales, company > > number > > > > > > 01190902). Registered address at Cannon Bridge House, 25 > > Dowgate > > > Hill, > > > > > > London EC4R 2YA. Both IG Markets Limited (register number > > 195355) and > > > > IG > > > > > > Index Limited (register number 114059) are authorised and > > regulated > > > by > > > > > the > > > > > > Financial Conduct Authority. > > > > > > > > > > > The information contained in this email is strictly > confidential > > and > > > for > > > > > the use of the addressee only, unless otherwise indicated. If > > you are > > > not > > > > > the intended recipient, please do not read, copy, use or > > disclose to > > > > others > > > > > this message or any attachment. Please also notify the sender > by > > > replying > > > > > to this email or by telephone (+44(020 7896 0011 > > <020%207896%200011>) > > > and then delete the > > > > email > > > > > and any copies of it. Opinions, conclusion (etc) that do not > > relate to > > > > the > > > > > official business of this company shall be understood as > neither > > given > > > > nor > > > > > endorsed by it. IG is a trading name of IG Markets Limited (a > > company > > > > > registered in England and Wales, company number 04008957) and > IG > > Index > > > > > Limited (a company registered in England and Wales, company > > number > > > > > 01190902). Registered address at Cannon Bridge House, 25 > Dowgate > > Hill, > > > > > London EC4R 2YA. Both IG Markets Limited (register number > > 195355) and > > > IG > > > > > Index Limited (register number 114059) are authorised and > > regulated by > > > > the > > > > > Financial Conduct Authority. > > > > > > > > > The information contained in this email is strictly confidential > > and for > > > > the use of the addressee only, unless otherwise indicated. If you > > are not > > > > the intended recipient, please do not read, copy, use or disclose > > to > > > others > > > > this message or any attachment. Please also notify the sender by > > replying > > > > to this email or by telephone (+44(020 7896 0011 > > <020%207896%200011>) > > > and then delete the email > > > > and any copies of it. Opinions, conclusion (etc) that do not > > relate to > > > the > > > > official business of this company shall be understood as neither > > given > > > nor > > > > endorsed by it. IG is a trading name of IG Markets Limited (a > > company > > > > registered in England and Wales, company number 04008957) and IG > > Index > > > > Limited (a company registered in England and Wales, company > number > > > > 01190902). Registered address at Cannon Bridge House, 25 Dowgate > > Hill, > > > > London EC4R 2YA. Both IG Markets Limited (register number 195355) > > and IG > > > > Index Limited (register number 114059) are authorised and > > regulated by > > > the > > > > Financial Conduct Authority. > > > > > > > > > > > > > The information contained in this email is strictly confidential and for > > the use of the addressee only, unless otherwise indicated. If you are not > > the intended recipient, please do not read, copy, use or disclose to > others > > this message or any attachment. Please also notify the sender by replying > > to this email or by telephone (+44(020 7896 0011) and then delete the > email > > and any copies of it. Opinions, conclusion (etc) that do not relate to > the > > official business of this company shall be understood as neither given > nor > > endorsed by it. IG is a trading name of IG Markets Limited (a company > > registered in England and Wales, company number 04008957) and IG Index > > Limited (a company registered in England and Wales, company number > > 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill, > > London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG > > Index Limited (register number 114059) are authorised and regulated by > the > > Financial Conduct Authority. > > > The information contained in this email is strictly confidential and for > the use of the addressee only, unless otherwise indicated. If you are not > the intended recipient, please do not read, copy, use or disclose to others > this message or any attachment. Please also notify the sender by replying > to this email or by telephone (+44(020 7896 0011) and then delete the email > and any copies of it. Opinions, conclusion (etc) that do not relate to the > official business of this company shall be understood as neither given nor > endorsed by it. IG is a trading name of IG Markets Limited (a company > registered in England and Wales, company number 04008957) and IG Index > Limited (a company registered in England and Wales, company number > 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill, > London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG > Index Limited (register number 114059) are authorised and regulated by the > Financial Conduct Authority. >