and we can start with {namespace, id} and no re-mapping support and always
add it later on if/when collisions actually happen (i dont think they'd be
a problem).
every interested party (so orgs or individuals) could then register a
prefix (0 = reserved, 1 = confluent ... 666 = me :-) ) and do whatever with
the 2nd ID - so once linkedin registers, say 3, then linkedin devs are free
to use {3, *} with a reasonable expectation to to collide with anything
else. further partitioning of that * becomes linkedin's problem, but the
"upstream registration" of a namespace only has to happen once.
On Tue, Nov 8, 2016 at 9:03 PM, James Cheng <wushuja...@gmail.com> wrote:
>
>
>
> > On Nov 8, 2016, at 5:54 PM, Gwen Shapira <g...@confluent.io> wrote:
> >
> > Thank you so much for this clear and fair summary of the arguments.
> >
> > I'm in favor of ints. Not a deal-breaker, but in favor.
> >
> > Even more in favor of Magnus's decentralized suggestion with Roger's
> > tweak: add a namespace for headers. This will allow each app to just
> > use whatever IDs it wants internally, and then let the admin deploying
> > the app figure out an available namespace ID for the app to live in.
> > So io.confluent.schema-registry can be namespace 0x01 on my deployment
> > and 0x57 on yours, and the poor guys developing the app don't need to
> > worry about that.
> >
>
> Gwen, if I understand your example right, an application deployer might
> decide to use 0x01 in one deployment, and that means that once the message
> is written into the broker, it will be saved on the broker with that
> specific namespace (0x01).
>
> If you were to mirror that message into another cluster, the 0x01 would
> accompany the message, right? What if the deployers of the same app in the
> other cluster uses 0x57? They won't understand each other?
>
> I'm not sure that's an avoidable problem. I think it simply means that in
> order to share data, you have to also have a shared (agreed upon)
> understanding of what the namespaces mean. Which I think makes sense,
> because the alternate (sharing *nothing* at all) would mean that there
> would be no way to understand each other.
>
> -James
>
> > Gwen
> >
> > On Tue, Nov 8, 2016 at 4:23 PM, radai <radai.rosenbl...@gmail.com>
> wrote:
> >> +1 for sean's document. it covers pretty much all the trade-offs and
> >> provides concrete figures to argue about :-)
> >> (nit-picking - used the same xkcd twice, also trove has been superceded
> for
> >> purposes of high performance collections: look at
> >> https://github.com/leventov/Koloboke)
> >>
> >> so to sum up the string vs int debate:
> >>
> >> performance - you can do 140k ops/sec _per thread_ with string headers.
> you
> >> could do x2-3 better with ints. there's no arguing the relative diff
> >> between the two, there's only the question of whether or not _the rest
> of
> >> kafka_ operates fast enough to care. if we want to make choices solely
> >> based on performance we need ints. if we are willing to
> settle/compromise
> >> for a nicer (to some) API than strings are good enough for the current
> >> state of affairs.
> >>
> >> message size - with batching and compression it comes down to a ~5%
> >> difference (internal testing, not in the doc. maybe would help adding if
> >> this becomes a point of contention?). this means it wont really affect
> >> kafka in "throughput mode" (large, compressed batches). in "low latency"
> >> mode (meaning less/no batching and compression) the difference can be
> >> extreme (it'll easily be an order of magnitude with small payloads like
> >> stock ticks and header keys of the form
> >> "com.acme.infraTeam.kafka.hiMom.auditPlugin"). we have a few such
> topics at
> >> linkedin where actual payloads are ~2 ints and are eclipsed by our
> in-house
> >> audit "header" which is why we liked ints to begin with.
> >>
> >> "ease of use" - strings would probably still require _some_ degree of
> >> partitioning by convention (imagine if everyone used the key "infra"...)
> >> but its very intuitive for java devs to do anyway (reverse-domain is
> >> ingrained into java developers at a young age :-) ). also most java devs
> >> find Map<String, whatever> more intuitive than Map<Integer, whatever> -
> >> probably because of other text-based protocols like http. ints would
> >> require a number registry. if you think number registries are hard just
> >> look at the wiki page for KIPs (specifically the number for next
> available
> >> KIP) and think again - we are probably talking about the same volume of
> >> requests. also this would only be "required" (good citizenship, more
> like)
> >> if you want to publish your plugin for others to use. within your org do
> >> whatever you want - just know that if you use [some "reserved" range]
> and a
> >> future kafka update breaks it its your problem. RTFM.
> >>
> >> personally im in favor of ints.
> >>
> >> having said that (and like nacho) I will settle if int vs string remains
> >> the only obstacle to this.
> >>
> >> On Tue, Nov 8, 2016 at 3:53 PM, Nacho Solis <nso...@linkedin.com.invalid
> >
> >> wrote:
> >>
> >>> I think it's well known I've been pushing for ints (and I could switch
> to
> >>> 16 bit shorts if pressed).
> >>>
> >>> - efficient (space)
> >>> - efficient (processing)
> >>> - easily partitionable
> >>>
> >>>
> >>> However, if the only thing that is keeping us from adopting headers is
> the
> >>> use of strings vs ints as keys, then I would cave in and accept
> strings. If
> >>> we do so, I would like to limit string keys to 128 bytes in length.
> This
> >>> way 1) I could use a 3 letter string if I wanted (effectively using 4
> total
> >>> bytes), 2) limit overall impact of possible keys (don't really want
> people
> >>> to send a 16K header string key).
> >>>
> >>> Nacho
> >>>
> >>>
> >>> On Tue, Nov 8, 2016 at 3:35 PM, Gwen Shapira <g...@confluent.io>
> wrote:
> >>>
> >>>> Forgot to mention: Thank you for quantifying the trade-off - it is
> >>>> helpful and important regardless of what we end up deciding.
> >>>>
> >>>> On Tue, Nov 8, 2016 at 3:12 PM, Sean McCauliff
> >>>> <smccaul...@linkedin.com.invalid> wrote:
> >>>>> On Tue, Nov 8, 2016 at 2:15 PM, Gwen Shapira <g...@confluent.io>
> >>> wrote:
> >>>>>
> >>>>>> Since Kafka specifically targets high-throughput, low-latency
> >>>>>> use-cases, I don't think we should trade them off that easily.
> >>>>>>
> >>>>>
> >>>>> I find these kind of design goals not to be really helpful unless
> it's
> >>>>> quantified in someway. Because it's always possible to argue against
> >>>>> something as either being not performant or just an implementation
> >>>> detail.
> >>>>>
> >>>>> This is a single threaded benchmarks so all the measurements are per
> >>>>> thread.
> >>>>>
> >>>>> For 1M messages/s/thread if header keys are int and you had even a
> >>>> single
> >>>>> header key, value pair then it's still about 2^-2 microseconds which
> >>>> means
> >>>>> you only have another 0.75 microseconds to do everything else you
> want
> >>> to
> >>>>> do with a message (1M messages/s means 1 micro second per message).
> >>> With
> >>>>> string header keys there is still 0.5 micro seconds to process a
> >>> message.
> >>>>>
> >>>>>
> >>>>>
> >>>>> I love strings as much as the next guy (we had them in Flume), but I
> >>>>>> was convinced by Magnus/Michael/Radai that strings don't actually
> have
> >>>>>> strong benefits as opposed to ints (you'll need a string registry
> >>>>>> anyway - otherwise, how will you know what does the "profile_id"
> >>>>>> header refers to?) and I want to keep closer to our original design
> >>>>>> goals for Kafka.
> >>>>>>
> >>>>>
> >>>>> "confluent.profile_id"
> >>>>>
> >>>>>
> >>>>>>
> >>>>>> If someone likes strings in the headers and doesn't do millions of
> >>>>>> messages a sec, they probably have lots of other systems they can
> use
> >>>>>> instead.
> >>>>>>
> >>>>>
> >>>>> None of them will scale like Kafka. Horizontal scaling is still
> good.
> >>>>>
> >>>>>
> >>>>>>
> >>>>>>
> >>>>>> On Tue, Nov 8, 2016 at 1:22 PM, Sean McCauliff
> >>>>>> <smccaul...@linkedin.com.invalid> wrote:
> >>>>>>> +1 for String keys.
> >>>>>>>
> >>>>>>> I've been doing some bechmarking and it seems like the speedup for
> >>>> using
> >>>>>>> integer keys is about 2-5 depending on the length of the strings
> and
> >>>> what
> >>>>>>> collections are being used. The overall amount of time spent
> >>> parsing
> >>>> a
> >>>>>> set
> >>>>>>> of header key, value pairs probably does not matter unless you are
> >>>>>> getting
> >>>>>>> close to 1M messages per consumer. In which case probably don't
> use
> >>>>>>> headers. There is also the option to use very short strings; some
> >>>> that
> >>>>>> are
> >>>>>>> even shorter than integers.
> >>>>>>>
> >>>>>>> Partitioning the string key space will be easier than partitioning
> >>> an
> >>>>>>> integer key space. We won't need a global registry. Kafka
> >>> internally
> >>>> can
> >>>>>>> reserve some prefix like "_" as its namespace. Everyone else can
> >>> use
> >>>>>> their
> >>>>>>> company or project name as namespace prefix and life should be
> good.
> >>>>>>>
> >>>>>>> Here's the link to some of the benchmarking info:
> >>>>>>> https://docs.google.com/document/d/1tfT-
> >>>> 6SZdnKOLyWGDH82kS30PnUkmgb7nPL
> >>>>>> dw6p65pAI/edit?usp=sharing
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> --
> >>>>>>> Sean McCauliff
> >>>>>>> Staff Software Engineer
> >>>>>>> Kafka
> >>>>>>>
> >>>>>>> smccaul...@linkedin.com
> >>>>>>> linkedin.com/in/sean-mccauliff-b563192
> >>>>>>>
> >>>>>>> On Mon, Nov 7, 2016 at 11:51 PM, Michael Pearce <
> >>>> michael.pea...@ig.com>
> >>>>>>> wrote:
> >>>>>>>
> >>>>>>>> +1 on this slimmer version of our proposal
> >>>>>>>>
> >>>>>>>> I def think the Id space we can reduce from the proposed
> >>>> int32(4bytes)
> >>>>>>>> down to int16(2bytes) it saves on space and as headers we wouldn't
> >>>>>> expect
> >>>>>>>> the number of headers being used concurrently being that high.
> >>>>>>>>
> >>>>>>>> I would wonder if we should make the value byte array length still
> >>>> int32
> >>>>>>>> though as This is the standard Max array length in Java saying
> that
> >>>> it
> >>>>>> is a
> >>>>>>>> header and I guess limiting the size is sensible and would work
> for
> >>>> all
> >>>>>> the
> >>>>>>>> use cases we have in mind so happy with limiting this.
> >>>>>>>>
> >>>>>>>> Do people generally concur on Magnus's slimmer version? Anyone see
> >>>> any
> >>>>>>>> issues if we moved from int32 to int16?
> >>>>>>>>
> >>>>>>>> Re configurable ids per plugin over a global registry also would
> >>> work
> >>>>>> for
> >>>>>>>> us. As such if this has better concensus over the proposed global
> >>>>>> registry
> >>>>>>>> I'd be happy to change that.
> >>>>>>>>
> >>>>>>>> I was already sold on ints over strings for keys ;)
> >>>>>>>>
> >>>>>>>> Cheers
> >>>>>>>> Mike
> >>>>>>>>
> >>>>>>>> ________________________________________
> >>>>>>>> From: Magnus Edenhill <mag...@edenhill.se>
> >>>>>>>> Sent: Monday, November 7, 2016 10:10:21 PM
> >>>>>>>> To: dev@kafka.apache.org
> >>>>>>>> Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
> >>>>>>>>
> >>>>>>>> Hi,
> >>>>>>>>
> >>>>>>>> I'm +1 for adding generic message headers, but I do share the
> >>>> concerns
> >>>>>>>> previously aired on this thread and during the KIP meeting.
> >>>>>>>>
> >>>>>>>> So let me propose a slimmer alternative that does not require any
> >>>> sort
> >>>>>> of
> >>>>>>>> global header registry, does not affect broker performance or
> >>>>>> operations,
> >>>>>>>> and adds as little overhead as possible.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> Message
> >>>>>>>> ------------
> >>>>>>>> The protocol Message type is extended with a Headers array
> consting
> >>>> of
> >>>>>>>> Tags, where a Tag is defined as:
> >>>>>>>> int16 Id
> >>>>>>>> int16 Len // binary_data length
> >>>>>>>> binary_data[Len] // opaque binary data
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> Ids
> >>>>>>>> ---
> >>>>>>>> The Id space is not centrally managed, so whenever an application
> >>>> needs
> >>>>>> to
> >>>>>>>> add headers, or use an eco-system plugin that does, its Id
> >>> allocation
> >>>>>> will
> >>>>>>>> need to be manually configured.
> >>>>>>>> This moves the allocation concern from the global space down to
> >>>>>>>> organization level and avoids the risk for id conflicts.
> >>>>>>>> Example pseudo-config for some app:
> >>>>>>>> sometrackerplugin.tag.sourcev3.id=1000
> >>>>>>>> dbthing.tag.tablename.id=1001
> >>>>>>>> myschemareg.tag.schemaname.id=1002
> >>>>>>>> myschemareg.tag.schemaversion.id=1003
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> Each header-writing or header-reading plugin must provide means
> >>>>>> (typically
> >>>>>>>> through configuration) to specify the tag for each header it uses.
> >>>>>> Defaults
> >>>>>>>> should be avoided.
> >>>>>>>> A consumer silently ignores tags it does not have a mapping for
> >>>> (since
> >>>>>> the
> >>>>>>>> binary_data can't be parsed without knowing what it is).
> >>>>>>>>
> >>>>>>>> Id range 0..999 is reserved for future use by the broker and must
> >>>> not be
> >>>>>>>> used by plugins.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> Broker
> >>>>>>>> ---------
> >>>>>>>> The broker does not process the tags (other than the standard
> >>>> protocol
> >>>>>>>> syntax verification), it simply stores and forwards them as opaque
> >>>> data.
> >>>>>>>>
> >>>>>>>> Standard message translation (removal of Headers) kicks in for
> >>> older
> >>>>>>>> clients.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> Why not string ids?
> >>>>>>>> -------------------------
> >>>>>>>> String ids might seem like a good idea, but:
> >>>>>>>> * does not really solve uniqueness
> >>>>>>>> * consumes a lot of space (2 byte string length + string, per
> >>>> header)
> >>>>>> to
> >>>>>>>> be meaningful
> >>>>>>>> * doesn't really say anything how to parse the tag's data, so it
> >>> is
> >>>> in
> >>>>>>>> effect useless on its own.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> Regards,
> >>>>>>>> Magnus
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> 2016-11-07 18:32 GMT+01:00 Michael Pearce <michael.pea...@ig.com
> >:
> >>>>>>>>
> >>>>>>>>> Hi Roger,
> >>>>>>>>>
> >>>>>>>>> Thanks for the support.
> >>>>>>>>>
> >>>>>>>>> I think the key thing is to have a common key space to make an
> >>>>>> ecosystem,
> >>>>>>>>> there does have to be some level of contract for people to play
> >>>>>> nicely.
> >>>>>>>>>
> >>>>>>>>> Having map<String, byte[]> or as per current proposed in kip of
> >>>>>> having a
> >>>>>>>>> numerical key space of map<int, byte[]> is a level of the
> >>> contract
> >>>>>> that
> >>>>>>>>> most people would expect.
> >>>>>>>>>
> >>>>>>>>> I think the example in a previous comment someone else made
> >>>> linking to
> >>>>>>>> AWS
> >>>>>>>>> blog and also implemented api where originally they didn’t have a
> >>>>>> header
> >>>>>>>>> space but not they do, where keys are uniform but the value can
> >>> be
> >>>>>>>> string,
> >>>>>>>>> int, anything is a good example.
> >>>>>>>>>
> >>>>>>>>> Having a custom MetadataSerializer is something we had played
> >>> with,
> >>>>>> but
> >>>>>>>>> discounted the idea, as if you wanted everyone to work the same
> >>>> way in
> >>>>>>>> the
> >>>>>>>>> ecosystem, having to have this also customizable makes it a bit
> >>>>>> harder.
> >>>>>>>>> Think about making the whole message record custom serializable,
> >>>> this
> >>>>>>>> would
> >>>>>>>>> make it fairly tricky (though it would not be impossible) to have
> >>>> made
> >>>>>>>> work
> >>>>>>>>> nicely. Having the value customizable we thought is a reasonable
> >>>>>> tradeoff
> >>>>>>>>> here of flexibility over contract of interaction between
> >>> different
> >>>>>>>> parties.
> >>>>>>>>>
> >>>>>>>>> Is there a particular case or benefit of having serialization
> >>>>>>>> customizable
> >>>>>>>>> that you have in mind?
> >>>>>>>>>
> >>>>>>>>> Saying this it is obviously something that could be implemented,
> >>> if
> >>>>>> there
> >>>>>>>>> is a need. If we did go this avenue I think a defaulted
> >>> serializer
> >>>>>>>>> implementation should exist so for the 80:20 rule, people can
> >>> just
> >>>>>> have
> >>>>>>>> the
> >>>>>>>>> broker and clients get default behavior.
> >>>>>>>>>
> >>>>>>>>> Cheers
> >>>>>>>>> Mike
> >>>>>>>>>
> >>>>>>>>> On 11/6/16, 5:25 PM, "radai" <radai.rosenbl...@gmail.com> wrote:
> >>>>>>>>>
> >>>>>>>>> making header _key_ serialization configurable potentially
> >>>>>> undermines
> >>>>>>>>> the
> >>>>>>>>> board usefulness of the feature (any point along the path
> >>> must
> >>>> be
> >>>>>>>> able
> >>>>>>>>> to
> >>>>>>>>> read the header keys. the values may be whatever and require
> >>>> more
> >>>>>>>>> intimate
> >>>>>>>>> knowledge of the code that produced specific headers, but
> >>> keys
> >>>>>> should
> >>>>>>>>> be
> >>>>>>>>> universally readable).
> >>>>>>>>>
> >>>>>>>>> it would also make it hard to write really portable plugins -
> >>>> say
> >>>>>> i
> >>>>>>>>> wrote a
> >>>>>>>>> large message splitter/combiner - if i rely on key
> >>>> "largeMessage"
> >>>>>> and
> >>>>>>>>> values of the form "1/20" someone who uses (contrived
> >>> example)
> >>>>>>>>> Map<Byte[],
> >>>>>>>>> Double> wouldnt be able to re-use my code.
> >>>>>>>>>
> >>>>>>>>> not the end of a the world within an organization, but
> >>>>>> problematic if
> >>>>>>>>> you
> >>>>>>>>> want to enable an ecosystem
> >>>>>>>>>
> >>>>>>>>> On Thu, Nov 3, 2016 at 2:04 PM, Roger Hoover <
> >>>>>> roger.hoo...@gmail.com
> >>>>>>>>>
> >>>>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>>> As others have laid out, I see strong reasons for a common
> >>>>>> message
> >>>>>>>>>> metadata structure for the Kafka ecosystem. In particular,
> >>>> I've
> >>>>>>>>> seen that
> >>>>>>>>>> even within a single organization, infrastructure teams
> >>> often
> >>>>>> own
> >>>>>>>> the
> >>>>>>>>>> message metadata while application teams own the
> >>>>>> application-level
> >>>>>>>>> data
> >>>>>>>>>> format. Allowing metadata and content to have different
> >>>>>> structure
> >>>>>>>>> and
> >>>>>>>>>> evolve separately is very helpful for this. Also, I think
> >>>>>> there's
> >>>>>>>> a
> >>>>>>>>> lot of
> >>>>>>>>>> value to having a common metadata structure shared across
> >>> the
> >>>>>> Kafka
> >>>>>>>>>> ecosystem so that tools which leverage metadata can more
> >>>> easily
> >>>>>> be
> >>>>>>>>> shared
> >>>>>>>>>> across organizations and integrated together.
> >>>>>>>>>>
> >>>>>>>>>> The question is, where does the metadata structure belong?
> >>>>>> Here's
> >>>>>>>>> my take:
> >>>>>>>>>>
> >>>>>>>>>> We change the Kafka wire and on-disk format to from a (key,
> >>>>>> value)
> >>>>>>>>> model to
> >>>>>>>>>> a (key, metadata, value) model where all three are byte
> >>>> arrays
> >>>>>> from
> >>>>>>>>> the
> >>>>>>>>>> brokers point of view. The primary reason for this is that
> >>>> it
> >>>>>>>>> provides a
> >>>>>>>>>> backward compatible migration path forward. Producers can
> >>>> start
> >>>>>>>>> populating
> >>>>>>>>>> metadata fields before all consumers understand the
> >>> metadata
> >>>>>>>>> structure.
> >>>>>>>>>> For people who already have custom envelope structures,
> >>> they
> >>>> can
> >>>>>>>>> populate
> >>>>>>>>>> their existing structure and the new structure for a while
> >>> as
> >>>>>> they
> >>>>>>>>> make the
> >>>>>>>>>> transition.
> >>>>>>>>>>
> >>>>>>>>>> We could stop there and let the clients plug in a
> >>>> KeySerializer,
> >>>>>>>>>> MetadataSerializer, and ValueSerializer but I think it is
> >>>> also
> >>>>>> be
> >>>>>>>>> useful to
> >>>>>>>>>> have a default MetadataSerializer that implements a
> >>> key-value
> >>>>>> model
> >>>>>>>>> similar
> >>>>>>>>>> to AMQP or HTTP headers. Or we could go even further and
> >>>>>>>> prescribe a
> >>>>>>>>>> Map<String, byte[]> or Map<String, String> data model for
> >>>>>> headers
> >>>>>>>> in
> >>>>>>>>> the
> >>>>>>>>>> clients (while still allowing custom serialization of the
> >>>> header
> >>>>>>>> data
> >>>>>>>>>> model).
> >>>>>>>>>>
> >>>>>>>>>> I think this would address Radai's concerns:
> >>>>>>>>>> 1. All client code would not need to be updated to know
> >>> about
> >>>>>> the
> >>>>>>>>>> container.
> >>>>>>>>>> 2. Middleware friendly clients would have a standard header
> >>>> data
> >>>>>>>>> model to
> >>>>>>>>>> work with.
> >>>>>>>>>> 3. KIP is required both b/c of broker changes and because
> >>> of
> >>>>>> client
> >>>>>>>>> API
> >>>>>>>>>> changes.
> >>>>>>>>>>
> >>>>>>>>>> Cheers,
> >>>>>>>>>>
> >>>>>>>>>> Roger
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> On Wed, Nov 2, 2016 at 4:38 PM, radai <
> >>>>>> radai.rosenbl...@gmail.com>
> >>>>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> my biggest issues with a "standard" wrapper format:
> >>>>>>>>>>>
> >>>>>>>>>>> 1. _ALL_ client _CODE_ (as opposed to kafka lib version)
> >>>> must
> >>>>>> be
> >>>>>>>>> updated
> >>>>>>>>>> to
> >>>>>>>>>>> know about the container, because any old naive code
> >>>> trying to
> >>>>>>>>> directly
> >>>>>>>>>>> deserialize its own payload would keel over and die (it
> >>>> needs
> >>>>>> to
> >>>>>>>>> know to
> >>>>>>>>>>> deserialize a container, and then dig in there for its
> >>>>>> payload).
> >>>>>>>>>>> 2. in order to write middleware-friendly clients that
> >>>> utilize
> >>>>>>>> such
> >>>>>>>>> a
> >>>>>>>>>>> container one would basically have to write their own
> >>>>>>>>> producer/consumer
> >>>>>>>>>> API
> >>>>>>>>>>> on top of the open source kafka one.
> >>>>>>>>>>> 3. if you were going to go with a wrapper format you
> >>> really
> >>>>>> dont
> >>>>>>>>> need to
> >>>>>>>>>>> bother with a kip (just open source your own client stack
> >>>>>> from #2
> >>>>>>>>> above
> >>>>>>>>>> so
> >>>>>>>>>>> others could stop re-inventing it)
> >>>>>>>>>>>
> >>>>>>>>>>> On Wed, Nov 2, 2016 at 4:25 PM, James Cheng <
> >>>>>>>> wushuja...@gmail.com>
> >>>>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> How exactly would this work? Or maybe that's out of
> >>> scope
> >>>>>> for
> >>>>>>>>> this
> >>>>>>>>>> email.
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> The information contained in this email is strictly confidential
> >>>> and
> >>>>>> for
> >>>>>>>>> the use of the addressee only, unless otherwise indicated. If you
> >>>> are
> >>>>>> not
> >>>>>>>>> the intended recipient, please do not read, copy, use or disclose
> >>>> to
> >>>>>>>> others
> >>>>>>>>> this message or any attachment. Please also notify the sender by
> >>>>>> replying
> >>>>>>>>> to this email or by telephone (+44(020 7896 0011) and then delete
> >>>> the
> >>>>>>>> email
> >>>>>>>>> and any copies of it. Opinions, conclusion (etc) that do not
> >>>> relate to
> >>>>>>>> the
> >>>>>>>>> official business of this company shall be understood as neither
> >>>> given
> >>>>>>>> nor
> >>>>>>>>> endorsed by it. IG is a trading name of IG Markets Limited (a
> >>>> company
> >>>>>>>>> registered in England and Wales, company number 04008957) and IG
> >>>> Index
> >>>>>>>>> Limited (a company registered in England and Wales, company
> >>> number
> >>>>>>>>> 01190902). Registered address at Cannon Bridge House, 25 Dowgate
> >>>> Hill,
> >>>>>>>>> London EC4R 2YA. Both IG Markets Limited (register number 195355)
> >>>> and
> >>>>>> IG
> >>>>>>>>> Index Limited (register number 114059) are authorised and
> >>>> regulated by
> >>>>>>>> the
> >>>>>>>>> Financial Conduct Authority.
> >>>>>>>>>
> >>>>>>>> The information contained in this email is strictly confidential
> >>> and
> >>>> for
> >>>>>>>> the use of the addressee only, unless otherwise indicated. If you
> >>> are
> >>>>>> not
> >>>>>>>> the intended recipient, please do not read, copy, use or disclose
> >>> to
> >>>>>> others
> >>>>>>>> this message or any attachment. Please also notify the sender by
> >>>>>> replying
> >>>>>>>> to this email or by telephone (+44(020 7896 0011) and then delete
> >>> the
> >>>>>> email
> >>>>>>>> and any copies of it. Opinions, conclusion (etc) that do not
> relate
> >>>> to
> >>>>>> the
> >>>>>>>> official business of this company shall be understood as neither
> >>>> given
> >>>>>> nor
> >>>>>>>> endorsed by it. IG is a trading name of IG Markets Limited (a
> >>> company
> >>>>>>>> registered in England and Wales, company number 04008957) and IG
> >>>> Index
> >>>>>>>> Limited (a company registered in England and Wales, company number
> >>>>>>>> 01190902). Registered address at Cannon Bridge House, 25 Dowgate
> >>>> Hill,
> >>>>>>>> London EC4R 2YA. Both IG Markets Limited (register number 195355)
> >>>> and IG
> >>>>>>>> Index Limited (register number 114059) are authorised and
> regulated
> >>>> by
> >>>>>> the
> >>>>>>>> Financial Conduct Authority.
> >>>>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> --
> >>>>>> Gwen Shapira
> >>>>>> Product Manager | Confluent
> >>>>>> 650.450.2760 | @gwenshap
> >>>>>> Follow us: Twitter | blog
> >>>>>>
> >>>>
> >>>>
> >>>>
> >>>> --
> >>>> Gwen Shapira
> >>>> Product Manager | Confluent
> >>>> 650.450.2760 | @gwenshap
> >>>> Follow us: Twitter | blog
> >>>>
> >>>
> >>>
> >>>
> >>> --
> >>> Nacho (Ignacio) Solis
> >>> Kafka
> >>> nso...@linkedin.com
> >>>
> >
> >
> >
> > --
> > Gwen Shapira
> > Product Manager | Confluent
> > 650.450.2760 | @gwenshap
> > Follow us: Twitter | blog
>
>
