Re: kafka 0.9 support
Kafka >>>>> 0.9 >>>>> we should leave a fork (renamed) of the IO in the tree for 0.9, but we can >>>>> definitely remove 0.9 support from the main IO if we want, especially if >>>>> it's complicated changes to that IO. If we do though, we should fail with >>>>> a >>>>> clear error message telling users to use the Kafka 0.9 IO. >>>>> >>>>> On Tue, Apr 2, 2019 at 9:34 AM Alexey Romanenko < >>>>> aromanenko@gmail.com> wrote: >>>>> >>>>>> > How are multiple versions of Kafka supported? Are they all in one >>>>>> client, or is there a case for forks like ElasticSearchIO? >>>>>> >>>>>> They are supported in one client but we have additional >>>>>> “ConsumerSpEL” adapter which unifies interface difference among different >>>>>> Kafka client versions (mostly to support old ones 0.9-0.10.0). >>>>>> >>>>>> On the other hand, we warn user in Javadoc of KafkaIO (which is >>>>>> Unstable, btw) by the following: >>>>>> *“KafkaIO relies on kafka-clients for all its interactions with the >>>>>> Kafka cluster.**kafka-clients versions 0.10.1 and newer are >>>>>> supported at runtime. The older versions 0.9.x **- 0.10.0.0 are also >>>>>> supported, but are deprecated and likely be removed in near future.”* >>>>>> >>>>>> Despite the fact that, personally, I’d prefer to have only one >>>>>> unified client interface but, since people still use Beam with old Kafka >>>>>> instances, we, likely, should stick with it till Beam 3.0. >>>>>> >>>>>> WDYT? >>>>>> >>>>>> On 2 Apr 2019, at 02:27, Austin Bennett >>>>>> wrote: >>>>>> >>>>>> FWIW -- >>>>>> >>>>>> On my (desired, not explicitly job-function) roadmap is to tap into a >>>>>> bunch of our corporate Kafka queues to ingest that data to places I can >>>>>> use. Those are 'stuck' 0.9, with no upgrade in sight (am told the >>>>>> upgrade >>>>>> path isn't trivial, is very critical flows, and they are scared for it to >>>>>> break, so it just sits behind firewalls, etc). But, I wouldn't begin >>>>>> that >>>>>> for probably at least another quarter. >>>>>> >>>>>> I don't contribute to nor understand the burden of maintaining the >>>>>> support for the older version, so can't reasonably lobby for that >>>>>> continued >>>>>> pain. >>>>>> >>>>>> Anecdotally, this could be a place many enterprises are at (though I >>>>>> also wonder whether many of the people that would be 'stuck' on such >>>>>> versions would also have Beam on their current radar). >>>>>> >>>>>> >>>>>> On Mon, Apr 1, 2019 at 2:29 PM Kenneth Knowles >>>>>> wrote: >>>>>> >>>>>>> This could be a backward-incompatible change, though that notion has >>>>>>> many interpretations. What matters is user pain. Technically if we don't >>>>>>> break the core SDK, users should be able to use Java SDK >=2.11.0 with >>>>>>> KafkaIO 2.11.0 forever. >>>>>>> >>>>>>> How are multiple versions of Kafka supported? Are they all in one >>>>>>> client, or is there a case for forks like ElasticSearchIO? >>>>>>> >>>>>>> Kenn >>>>>>> >>>>>>> On Mon, Apr 1, 2019 at 10:37 AM Jean-Baptiste Onofré < >>>>>>> j...@nanthrax.net> wrote: >>>>>>> >>>>>>>> +1 to remove 0.9 support. >>>>>>>> >>>>>>>> I think it's more interesting to test and verify Kafka 2.2.0 than >>>>>>>> 0.9 ;) >>>>>>>> >>>>>>>> Regards >>>>>>>> JB >>>>>>>> >>>>>>>> On 01/04/2019 19:36, David Morávek wrote: >>>>>>>> > Hello, >>>>>>>> > >>>>>>>> > is there still a reason to keep Kafka 0.9 support? This >>>>>>>> unfortunately >>>>>>>> > adds lot of complexity to KafkaIO implementation. >>>>>>>> > >>>>>>>> > Kafka 0.9 was released on Nov 2015. >>>>>>>> > >>>>>>>> > My first shot on removing Kafka 0.9 support would remove second >>>>>>>> > consumer, which is used for fetching offsets. >>>>>>>> > >>>>>>>> > WDYT? Is this support worth keeping? >>>>>>>> > >>>>>>>> > https://github.com/apache/beam/pull/8186 >>>>>>>> > >>>>>>>> > D. >>>>>>>> >>>>>>>> -- >>>>>>>> Jean-Baptiste Onofré >>>>>>>> jbono...@apache.org >>>>>>>> http://blog.nanthrax.net >>>>>>>> Talend - http://www.talend.com >>>>>>>> >>>>>>> >>>>>> >>>> >>>> -- >>>> >>>> Mingmin >>>> >>>
Re: kafka 0.9 support
we should fail with a clear error > message telling users to use the Kafka 0.9 IO. > > On Tue, Apr 2, 2019 at 9:34 AM Alexey Romanenko > <mailto:aromanenko@gmail.com>> wrote: > > > How are multiple versions of Kafka supported? Are > they all in one client, or is there a case for forks > like ElasticSearchIO? > > They are supported in one client but we have > additional “ConsumerSpEL” adapter which unifies > interface difference among different Kafka client > versions (mostly to support old ones 0.9-0.10.0). > > On the other hand, we warn user in Javadoc of > KafkaIO (which is Unstable, btw) by the following: > /“KafkaIO relies on kafka-clients for all its > interactions with the Kafka > cluster.//kafka-clients versions 0.10.1 and newer > are supported at runtime. The older versions > 0.9.x //- 0.10.0.0 are also supported, but are > deprecated and likely be removed in near future.”/ > / > / > Despite the fact that, personally, I’d prefer to > have only one unified client interface but, since > people still use Beam with old Kafka instances, we, > likely, should stick with it till Beam 3.0. > > WDYT? > >> On 2 Apr 2019, at 02:27, Austin Bennett >> > <mailto:whatwouldausti...@gmail.com>> wrote: >> >> FWIW -- >> >> On my (desired, not explicitly job-function) >> roadmap is to tap into a bunch of our corporate >> Kafka queues to ingest that data to places I can >> use. Those are 'stuck' 0.9, with no upgrade in >> sight (am told the upgrade path isn't trivial, is >> very critical flows, and they are scared for it to >> break, so it just sits behind firewalls, etc). >> But, I wouldn't begin that for probably at least >> another quarter. >> >> I don't contribute to nor understand the burden of >> maintaining the support for the older version, so >> can't reasonably lobby for that continued pain. >> >> Anecdotally, this could be a place many >> enterprises are at (though I also wonder whether >> many of the people that would be 'stuck' on such >> versions would also have Beam on their current >> radar). >> >> >> On Mon, Apr 1, 2019 at 2:29 PM Kenneth Knowles >> mailto:k...@apache.org>> wrote: >> >> This could be a backward-incompatible change, >> though that notion has many interpretations. >> What matters is user pain. Technically if we >> don't break the core SDK, users should be able >> to use Java SDK >=2.11.0 with KafkaIO 2.11.0 >> forever. >> >> How are multiple versions of Kafka supported? >> Are they all in one client, or is there a case >> for forks like ElasticSearchIO? >> >> Kenn >> >> On Mon, Apr 1, 2019 at 10:37 AM Jean-Baptiste >> Onofré > <mailto:j...@nanthrax.net>> wrote: >> >> +1 to remove 0.9 support. >> >> I think it's more interesting to test and >> verify Kafka 2.2.0 than 0.9 ;) >> >> Regards >> JB >> >> On 01/04/2019 19:36, David Morávek wrote: >> > Hello, >> > >> > is there still a reason to keep Kafka >> 0.9
Re: kafka 0.9 support
ll its interactions with the >>>>> Kafka cluster.**kafka-clients versions 0.10.1 and newer are supported >>>>> at runtime. The older versions 0.9.x **- 0.10.0.0 are also supported, >>>>> but are deprecated and likely be removed in near future.”* >>>>> >>>>> Despite the fact that, personally, I’d prefer to have only one unified >>>>> client interface but, since people still use Beam with old Kafka >>>>> instances, >>>>> we, likely, should stick with it till Beam 3.0. >>>>> >>>>> WDYT? >>>>> >>>>> On 2 Apr 2019, at 02:27, Austin Bennett >>>>> wrote: >>>>> >>>>> FWIW -- >>>>> >>>>> On my (desired, not explicitly job-function) roadmap is to tap into a >>>>> bunch of our corporate Kafka queues to ingest that data to places I can >>>>> use. Those are 'stuck' 0.9, with no upgrade in sight (am told the upgrade >>>>> path isn't trivial, is very critical flows, and they are scared for it to >>>>> break, so it just sits behind firewalls, etc). But, I wouldn't begin that >>>>> for probably at least another quarter. >>>>> >>>>> I don't contribute to nor understand the burden of maintaining the >>>>> support for the older version, so can't reasonably lobby for that >>>>> continued >>>>> pain. >>>>> >>>>> Anecdotally, this could be a place many enterprises are at (though I >>>>> also wonder whether many of the people that would be 'stuck' on such >>>>> versions would also have Beam on their current radar). >>>>> >>>>> >>>>> On Mon, Apr 1, 2019 at 2:29 PM Kenneth Knowles >>>>> wrote: >>>>> >>>>>> This could be a backward-incompatible change, though that notion has >>>>>> many interpretations. What matters is user pain. Technically if we don't >>>>>> break the core SDK, users should be able to use Java SDK >=2.11.0 with >>>>>> KafkaIO 2.11.0 forever. >>>>>> >>>>>> How are multiple versions of Kafka supported? Are they all in one >>>>>> client, or is there a case for forks like ElasticSearchIO? >>>>>> >>>>>> Kenn >>>>>> >>>>>> On Mon, Apr 1, 2019 at 10:37 AM Jean-Baptiste Onofré >>>>>> wrote: >>>>>> >>>>>>> +1 to remove 0.9 support. >>>>>>> >>>>>>> I think it's more interesting to test and verify Kafka 2.2.0 than >>>>>>> 0.9 ;) >>>>>>> >>>>>>> Regards >>>>>>> JB >>>>>>> >>>>>>> On 01/04/2019 19:36, David Morávek wrote: >>>>>>> > Hello, >>>>>>> > >>>>>>> > is there still a reason to keep Kafka 0.9 support? This >>>>>>> unfortunately >>>>>>> > adds lot of complexity to KafkaIO implementation. >>>>>>> > >>>>>>> > Kafka 0.9 was released on Nov 2015. >>>>>>> > >>>>>>> > My first shot on removing Kafka 0.9 support would remove second >>>>>>> > consumer, which is used for fetching offsets. >>>>>>> > >>>>>>> > WDYT? Is this support worth keeping? >>>>>>> > >>>>>>> > https://github.com/apache/beam/pull/8186 >>>>>>> > >>>>>>> > D. >>>>>>> >>>>>>> -- >>>>>>> Jean-Baptiste Onofré >>>>>>> jbono...@apache.org >>>>>>> http://blog.nanthrax.net >>>>>>> Talend - http://www.talend.com >>>>>>> >>>>>> >>>>> >>> >>> -- >>> >>> Mingmin >>> >>
Re: kafka 0.9 support
t; for probably at least another quarter. >>>> >>>> I don't contribute to nor understand the burden of maintaining the >>>> support for the older version, so can't reasonably lobby for that continued >>>> pain. >>>> >>>> Anecdotally, this could be a place many enterprises are at (though I >>>> also wonder whether many of the people that would be 'stuck' on such >>>> versions would also have Beam on their current radar). >>>> >>>> >>>> On Mon, Apr 1, 2019 at 2:29 PM Kenneth Knowles wrote: >>>> >>>>> This could be a backward-incompatible change, though that notion has >>>>> many interpretations. What matters is user pain. Technically if we don't >>>>> break the core SDK, users should be able to use Java SDK >=2.11.0 with >>>>> KafkaIO 2.11.0 forever. >>>>> >>>>> How are multiple versions of Kafka supported? Are they all in one >>>>> client, or is there a case for forks like ElasticSearchIO? >>>>> >>>>> Kenn >>>>> >>>>> On Mon, Apr 1, 2019 at 10:37 AM Jean-Baptiste Onofré >>>>> wrote: >>>>> >>>>>> +1 to remove 0.9 support. >>>>>> >>>>>> I think it's more interesting to test and verify Kafka 2.2.0 than 0.9 >>>>>> ;) >>>>>> >>>>>> Regards >>>>>> JB >>>>>> >>>>>> On 01/04/2019 19:36, David Morávek wrote: >>>>>> > Hello, >>>>>> > >>>>>> > is there still a reason to keep Kafka 0.9 support? This >>>>>> unfortunately >>>>>> > adds lot of complexity to KafkaIO implementation. >>>>>> > >>>>>> > Kafka 0.9 was released on Nov 2015. >>>>>> > >>>>>> > My first shot on removing Kafka 0.9 support would remove second >>>>>> > consumer, which is used for fetching offsets. >>>>>> > >>>>>> > WDYT? Is this support worth keeping? >>>>>> > >>>>>> > https://github.com/apache/beam/pull/8186 >>>>>> > >>>>>> > D. >>>>>> >>>>>> -- >>>>>> Jean-Baptiste Onofré >>>>>> jbono...@apache.org >>>>>> http://blog.nanthrax.net >>>>>> Talend - http://www.talend.com >>>>>> >>>>> >>>> >> >> -- >> >> Mingmin >> >
Re: kafka 0.9 support
his could be a backward-incompatible change, though that notion has >>>> many interpretations. What matters is user pain. Technically if we don't >>>> break the core SDK, users should be able to use Java SDK >=2.11.0 with >>>> KafkaIO 2.11.0 forever. >>>> >>>> How are multiple versions of Kafka supported? Are they all in one >>>> client, or is there a case for forks like ElasticSearchIO? >>>> >>>> Kenn >>>> >>>> On Mon, Apr 1, 2019 at 10:37 AM Jean-Baptiste Onofré >>>> wrote: >>>> >>>>> +1 to remove 0.9 support. >>>>> >>>>> I think it's more interesting to test and verify Kafka 2.2.0 than 0.9 >>>>> ;) >>>>> >>>>> Regards >>>>> JB >>>>> >>>>> On 01/04/2019 19:36, David Morávek wrote: >>>>> > Hello, >>>>> > >>>>> > is there still a reason to keep Kafka 0.9 support? This unfortunately >>>>> > adds lot of complexity to KafkaIO implementation. >>>>> > >>>>> > Kafka 0.9 was released on Nov 2015. >>>>> > >>>>> > My first shot on removing Kafka 0.9 support would remove second >>>>> > consumer, which is used for fetching offsets. >>>>> > >>>>> > WDYT? Is this support worth keeping? >>>>> > >>>>> > https://github.com/apache/beam/pull/8186 >>>>> > >>>>> > D. >>>>> >>>>> -- >>>>> Jean-Baptiste Onofré >>>>> jbono...@apache.org >>>>> http://blog.nanthrax.net >>>>> Talend - http://www.talend.com >>>>> >>>> >>> > > -- > > Mingmin >
Re: kafka 0.9 support
> How are multiple versions of Kafka supported? Are they all in one client, or > is there a case for forks like ElasticSearchIO? They are supported in one client but we have additional “ConsumerSpEL” adapter which unifies interface difference among different Kafka client versions (mostly to support old ones 0.9-0.10.0). On the other hand, we warn user in Javadoc of KafkaIO (which is Unstable, btw) by the following: “KafkaIO relies on kafka-clients for all its interactions with the Kafka cluster.kafka-clients versions 0.10.1 and newer are supported at runtime. The older versions 0.9.x - 0.10.0.0 are also supported, but are deprecated and likely be removed in near future.” Despite the fact that, personally, I’d prefer to have only one unified client interface but, since people still use Beam with old Kafka instances, we, likely, should stick with it till Beam 3.0. WDYT? > On 2 Apr 2019, at 02:27, Austin Bennett wrote: > > FWIW -- > > On my (desired, not explicitly job-function) roadmap is to tap into a bunch > of our corporate Kafka queues to ingest that data to places I can use. Those > are 'stuck' 0.9, with no upgrade in sight (am told the upgrade path isn't > trivial, is very critical flows, and they are scared for it to break, so it > just sits behind firewalls, etc). But, I wouldn't begin that for probably at > least another quarter. > > I don't contribute to nor understand the burden of maintaining the support > for the older version, so can't reasonably lobby for that continued pain. > > Anecdotally, this could be a place many enterprises are at (though I also > wonder whether many of the people that would be 'stuck' on such versions > would also have Beam on their current radar). > > > On Mon, Apr 1, 2019 at 2:29 PM Kenneth Knowles <mailto:k...@apache.org>> wrote: > This could be a backward-incompatible change, though that notion has many > interpretations. What matters is user pain. Technically if we don't break the > core SDK, users should be able to use Java SDK >=2.11.0 with KafkaIO 2.11.0 > forever. > > How are multiple versions of Kafka supported? Are they all in one client, or > is there a case for forks like ElasticSearchIO? > > Kenn > > On Mon, Apr 1, 2019 at 10:37 AM Jean-Baptiste Onofré <mailto:j...@nanthrax.net>> wrote: > +1 to remove 0.9 support. > > I think it's more interesting to test and verify Kafka 2.2.0 than 0.9 ;) > > Regards > JB > > On 01/04/2019 19:36, David Morávek wrote: > > Hello, > > > > is there still a reason to keep Kafka 0.9 support? This unfortunately > > adds lot of complexity to KafkaIO implementation. > > > > Kafka 0.9 was released on Nov 2015. > > > > My first shot on removing Kafka 0.9 support would remove second > > consumer, which is used for fetching offsets. > > > > WDYT? Is this support worth keeping? > > > > https://github.com/apache/beam/pull/8186 > > <https://github.com/apache/beam/pull/8186> > > > > D. > > -- > Jean-Baptiste Onofré > jbono...@apache.org <mailto:jbono...@apache.org> > http://blog.nanthrax.net <http://blog.nanthrax.net/> > Talend - http://www.talend.com <http://www.talend.com/>
Re: kafka 0.9 support
FWIW -- On my (desired, not explicitly job-function) roadmap is to tap into a bunch of our corporate Kafka queues to ingest that data to places I can use. Those are 'stuck' 0.9, with no upgrade in sight (am told the upgrade path isn't trivial, is very critical flows, and they are scared for it to break, so it just sits behind firewalls, etc). But, I wouldn't begin that for probably at least another quarter. I don't contribute to nor understand the burden of maintaining the support for the older version, so can't reasonably lobby for that continued pain. Anecdotally, this could be a place many enterprises are at (though I also wonder whether many of the people that would be 'stuck' on such versions would also have Beam on their current radar). On Mon, Apr 1, 2019 at 2:29 PM Kenneth Knowles wrote: > This could be a backward-incompatible change, though that notion has many > interpretations. What matters is user pain. Technically if we don't break > the core SDK, users should be able to use Java SDK >=2.11.0 with KafkaIO > 2.11.0 forever. > > How are multiple versions of Kafka supported? Are they all in one client, > or is there a case for forks like ElasticSearchIO? > > Kenn > > On Mon, Apr 1, 2019 at 10:37 AM Jean-Baptiste Onofré > wrote: > >> +1 to remove 0.9 support. >> >> I think it's more interesting to test and verify Kafka 2.2.0 than 0.9 ;) >> >> Regards >> JB >> >> On 01/04/2019 19:36, David Morávek wrote: >> > Hello, >> > >> > is there still a reason to keep Kafka 0.9 support? This unfortunately >> > adds lot of complexity to KafkaIO implementation. >> > >> > Kafka 0.9 was released on Nov 2015. >> > >> > My first shot on removing Kafka 0.9 support would remove second >> > consumer, which is used for fetching offsets. >> > >> > WDYT? Is this support worth keeping? >> > >> > https://github.com/apache/beam/pull/8186 >> > >> > D. >> >> -- >> Jean-Baptiste Onofré >> jbono...@apache.org >> http://blog.nanthrax.net >> Talend - http://www.talend.com >> >
Re: kafka 0.9 support
This could be a backward-incompatible change, though that notion has many interpretations. What matters is user pain. Technically if we don't break the core SDK, users should be able to use Java SDK >=2.11.0 with KafkaIO 2.11.0 forever. How are multiple versions of Kafka supported? Are they all in one client, or is there a case for forks like ElasticSearchIO? Kenn On Mon, Apr 1, 2019 at 10:37 AM Jean-Baptiste Onofré wrote: > +1 to remove 0.9 support. > > I think it's more interesting to test and verify Kafka 2.2.0 than 0.9 ;) > > Regards > JB > > On 01/04/2019 19:36, David Morávek wrote: > > Hello, > > > > is there still a reason to keep Kafka 0.9 support? This unfortunately > > adds lot of complexity to KafkaIO implementation. > > > > Kafka 0.9 was released on Nov 2015. > > > > My first shot on removing Kafka 0.9 support would remove second > > consumer, which is used for fetching offsets. > > > > WDYT? Is this support worth keeping? > > > > https://github.com/apache/beam/pull/8186 > > > > D. > > -- > Jean-Baptiste Onofré > jbono...@apache.org > http://blog.nanthrax.net > Talend - http://www.talend.com >
