We're still using Kafka 0.10 a lot, similar as 0.9 IMO. To expand multiple versions in KafkaIO is quite complex now, and it confuses users which is supported / which is not. I would prefer to support Kafka 2.0+ only in the latest version. For old versions, there're some options: 1). document Kafka-Beam support versions, like what we do in FlinkRunner; 2). maintain separated KafkaIOs for old versions;
1) would be easy to maintain, and I assume there should be no issue to use Beam-Core 3.0 together with KafkaIO 2.0. Any thoughts? Mingmin On Tue, Apr 2, 2019 at 9:56 AM Reuven Lax <[email protected]> wrote: > KafkaIO is marked as Experimental, and the comment already warns that 0.9 > support might be removed. I think that if users still rely on Kafka 0.9 we > should leave a fork (renamed) of the IO in the tree for 0.9, but we can > definitely remove 0.9 support from the main IO if we want, especially if > it's complicated changes to that IO. If we do though, we should fail with a > clear error message telling users to use the Kafka 0.9 IO. > > On Tue, Apr 2, 2019 at 9:34 AM Alexey Romanenko <[email protected]> > wrote: > >> > How are multiple versions of Kafka supported? Are they all in one >> client, or is there a case for forks like ElasticSearchIO? >> >> They are supported in one client but we have additional “ConsumerSpEL” >> adapter which unifies interface difference among different Kafka client >> versions (mostly to support old ones 0.9-0.10.0). >> >> On the other hand, we warn user in Javadoc of KafkaIO (which is Unstable, >> btw) by the following: >> *“KafkaIO relies on kafka-clients for all its interactions with the Kafka >> cluster.**kafka-clients versions 0.10.1 and newer are supported at >> runtime. The older versions 0.9.x **- 0.10.0.0 are also supported, but >> are deprecated and likely be removed in near future.”* >> >> Despite the fact that, personally, I’d prefer to have only one unified >> client interface but, since people still use Beam with old Kafka instances, >> we, likely, should stick with it till Beam 3.0. >> >> WDYT? >> >> On 2 Apr 2019, at 02:27, Austin Bennett <[email protected]> >> wrote: >> >> FWIW -- >> >> On my (desired, not explicitly job-function) roadmap is to tap into a >> bunch of our corporate Kafka queues to ingest that data to places I can >> use. Those are 'stuck' 0.9, with no upgrade in sight (am told the upgrade >> path isn't trivial, is very critical flows, and they are scared for it to >> break, so it just sits behind firewalls, etc). But, I wouldn't begin that >> for probably at least another quarter. >> >> I don't contribute to nor understand the burden of maintaining the >> support for the older version, so can't reasonably lobby for that continued >> pain. >> >> Anecdotally, this could be a place many enterprises are at (though I also >> wonder whether many of the people that would be 'stuck' on such versions >> would also have Beam on their current radar). >> >> >> On Mon, Apr 1, 2019 at 2:29 PM Kenneth Knowles <[email protected]> wrote: >> >>> This could be a backward-incompatible change, though that notion has >>> many interpretations. What matters is user pain. Technically if we don't >>> break the core SDK, users should be able to use Java SDK >=2.11.0 with >>> KafkaIO 2.11.0 forever. >>> >>> How are multiple versions of Kafka supported? Are they all in one >>> client, or is there a case for forks like ElasticSearchIO? >>> >>> Kenn >>> >>> On Mon, Apr 1, 2019 at 10:37 AM Jean-Baptiste Onofré <[email protected]> >>> wrote: >>> >>>> +1 to remove 0.9 support. >>>> >>>> I think it's more interesting to test and verify Kafka 2.2.0 than 0.9 ;) >>>> >>>> Regards >>>> JB >>>> >>>> On 01/04/2019 19:36, David Morávek wrote: >>>> > Hello, >>>> > >>>> > is there still a reason to keep Kafka 0.9 support? This unfortunately >>>> > adds lot of complexity to KafkaIO implementation. >>>> > >>>> > Kafka 0.9 was released on Nov 2015. >>>> > >>>> > My first shot on removing Kafka 0.9 support would remove second >>>> > consumer, which is used for fetching offsets. >>>> > >>>> > WDYT? Is this support worth keeping? >>>> > >>>> > https://github.com/apache/beam/pull/8186 >>>> > >>>> > D. >>>> >>>> -- >>>> Jean-Baptiste Onofré >>>> [email protected] >>>> http://blog.nanthrax.net >>>> Talend - http://www.talend.com >>>> >>> >> -- ---- Mingmin
