Re: Different Schemas on same Kafka Topic

2017-08-18 Thread Stephen Durfey
ers@kafka.apache.org Subject: Re: Different Schemas on same Kafka Topic Thank you Stephen for a very detailed write up. Really helpful. I was stuck in a concept of one schema per topic. Let me try this in my use case. Thank you very much. And thank you Svante. Regards, Sreejith On 18-Aug-201

Re: Different Schemas on same Kafka Topic

2017-08-17 Thread Sreejith S
type the schema registry > will > > > allow any schema to be registered, even if they are not compatible > > because > > > you've informed the registry not to care. So, you should really choose > > > amongst backward, forward, and full. I use FULL in prod

Re: Different Schemas on same Kafka Topic

2017-08-17 Thread Stephen Durfey
and will have multiple readers and > > writers of the data, and the data needs to be passively evolved. Backward > > and forward can be fine too, just depending upon the needs of the data > > being produced and consumed. > > > > On Thu, Aug 17, 2017 at 12:22 PM, Tau

Re: Different Schemas on same Kafka Topic

2017-08-17 Thread Svante Karlsson
nd consumed. > > > > On Thu, Aug 17, 2017 at 12:22 PM, Tauzell, Dave < > > dave.tauz...@surescripts.com> wrote: > > > > > Hmm, I think you are right that you cannot have multiple schemas on the > > > same topic. > > > > > > -Dave >

Re: Different Schemas on same Kafka Topic

2017-08-17 Thread Sreejith S
wrapper that > > > contains type, version and data fields. Then you wrap everything. > > > Another option is to do what Kafka is doing and prepend some sort of > > > fixed length value to all messages that have the schema and version > > > you are using for that me

Re: Different Schemas on same Kafka Topic

2017-08-17 Thread Stephen Durfey
hat you cannot have multiple schemas on the > same topic. > > -Dave > > > -Original Message- > From: Sreejith S [mailto:srssreej...@gmail.com] > Sent: Thursday, August 17, 2017 11:42 AM > To: users@kafka.apache.org > Subject: RE: Different Schemas on same Kafka T

RE: Different Schemas on same Kafka Topic

2017-08-17 Thread Tauzell, Dave
Hmm, I think you are right that you cannot have multiple schemas on the same topic. -Dave -Original Message- From: Sreejith S [mailto:srssreej...@gmail.com] Sent: Thursday, August 17, 2017 11:42 AM To: users@kafka.apache.org Subject: RE: Different Schemas on same Kafka Topic Hi Dave

RE: Different Schemas on same Kafka Topic

2017-08-17 Thread Sreejith S
hat have the schema and version you are using > for that message. > > -Dave > > -Original Message- > From: Shajahan, Nishanth [mailto:nshaj...@visa.com] > Sent: Thursday, August 17, 2017 11:02 AM > To: users@kafka.apache.org > Subject: RE: Different Schemas on same

RE: Different Schemas on same Kafka Topic

2017-08-17 Thread Tauzell, Dave
.apache.org Subject: RE: Different Schemas on same Kafka Topic Thanks Dave. We may not want to start using schema registery immediately . We would have java producers and consumers . I might also go with using byte messages but when consumer de serilize how can they map the byte[] to the correct

RE: Different Schemas on same Kafka Topic

2017-08-17 Thread Shajahan, Nishanth
to:dave.tauz...@surescripts.com] Sent: Thursday, August 17, 2017 8:30 AM To: users@kafka.apache.org Subject: RE: Different Schemas on same Kafka Topic It does. The way it works is that the Avro serializer precedes each message with a two-byte integer that references a schema id in the confluent schema registry

RE: Different Schemas on same Kafka Topic

2017-08-17 Thread Tauzell, Dave
It does. The way it works is that the Avro serializer precedes each message with a two-byte integer that references a schema id in the confluent schema registry. The Avro deserializer looks at this value to determine which schema to de-serialize with. In order for this to work you need to u