I withdraw my concern -- checked on info on the cluster I will eventually access. It is on 0.8, so I was speaking too soon. Can't speak to rest of user base.
On Tue, Apr 2, 2019 at 11:03 AM Raghu Angadi <[email protected]> wrote: > Thanks to David Morávek for pointing out possible improvement to KafkaIO > for dropping support for 0.9 since it avoids having a second consumer just > to fetch latest offsets for backlog. > > Ideally we should be dropping 0.9 support for next major release, in fact > better to drop versions before 0.10.1 at the same time. This would further > reduce reflection based calls for supporting multiple versions. If the > users still on 0.9 could stay on current stable release of Beam, dropping > would not affect them. Otherwise, it would be good to hear from them about > how long we need to keep support for old versions. > > I don't think it is good idea to have multiple forks of KafkaIO in the > same repo. If we do go that route, we should fork the entire kafka > directory and rename the main class KafkaIO_Unmaintained :). > > IMHO, so far, additional complexity for supporting these versions is not > that bad. Most of it is isolated to ConsumerSpEL.java & ProducerSpEL.java. > My first preference is dropping support for deprecated versions (and a > deprecate a few more versions, may be till the version that added > transactions around 0.11.x I think). > > I haven't looked into what's new in Kafka 2.x. Are there any features that > KafkaIO should take advantage of? I have not noticed our existing code > breaking. We should certainly certainly support latest releases of Kafka. > > Raghu. > > On Tue, Apr 2, 2019 at 10:27 AM Mingmin Xu <[email protected]> wrote: > >> >> We're still using Kafka 0.10 a lot, similar as 0.9 IMO. To expand >> multiple versions in KafkaIO is quite complex now, and it confuses users >> which is supported / which is not. I would prefer to support Kafka 2.0+ >> only in the latest version. For old versions, there're some options: >> 1). document Kafka-Beam support versions, like what we do in FlinkRunner; >> 2). maintain separated KafkaIOs for old versions; >> >> 1) would be easy to maintain, and I assume there should be no issue to >> use Beam-Core 3.0 together with KafkaIO 2.0. >> >> Any thoughts? >> >> Mingmin >> >> On Tue, Apr 2, 2019 at 9:56 AM Reuven Lax <[email protected]> wrote: >> >>> KafkaIO is marked as Experimental, and the comment already warns that >>> 0.9 support might be removed. I think that if users still rely on Kafka 0.9 >>> we should leave a fork (renamed) of the IO in the tree for 0.9, but we can >>> definitely remove 0.9 support from the main IO if we want, especially if >>> it's complicated changes to that IO. If we do though, we should fail with a >>> clear error message telling users to use the Kafka 0.9 IO. >>> >>> On Tue, Apr 2, 2019 at 9:34 AM Alexey Romanenko < >>> [email protected]> wrote: >>> >>>> > How are multiple versions of Kafka supported? Are they all in one >>>> client, or is there a case for forks like ElasticSearchIO? >>>> >>>> They are supported in one client but we have additional “ConsumerSpEL” >>>> adapter which unifies interface difference among different Kafka client >>>> versions (mostly to support old ones 0.9-0.10.0). >>>> >>>> On the other hand, we warn user in Javadoc of KafkaIO (which is >>>> Unstable, btw) by the following: >>>> *“KafkaIO relies on kafka-clients for all its interactions with the >>>> Kafka cluster.**kafka-clients versions 0.10.1 and newer are supported >>>> at runtime. The older versions 0.9.x **- 0.10.0.0 are also supported, >>>> but are deprecated and likely be removed in near future.”* >>>> >>>> Despite the fact that, personally, I’d prefer to have only one unified >>>> client interface but, since people still use Beam with old Kafka instances, >>>> we, likely, should stick with it till Beam 3.0. >>>> >>>> WDYT? >>>> >>>> On 2 Apr 2019, at 02:27, Austin Bennett <[email protected]> >>>> wrote: >>>> >>>> FWIW -- >>>> >>>> On my (desired, not explicitly job-function) roadmap is to tap into a >>>> bunch of our corporate Kafka queues to ingest that data to places I can >>>> use. Those are 'stuck' 0.9, with no upgrade in sight (am told the upgrade >>>> path isn't trivial, is very critical flows, and they are scared for it to >>>> break, so it just sits behind firewalls, etc). But, I wouldn't begin that >>>> for probably at least another quarter. >>>> >>>> I don't contribute to nor understand the burden of maintaining the >>>> support for the older version, so can't reasonably lobby for that continued >>>> pain. >>>> >>>> Anecdotally, this could be a place many enterprises are at (though I >>>> also wonder whether many of the people that would be 'stuck' on such >>>> versions would also have Beam on their current radar). >>>> >>>> >>>> On Mon, Apr 1, 2019 at 2:29 PM Kenneth Knowles <[email protected]> wrote: >>>> >>>>> This could be a backward-incompatible change, though that notion has >>>>> many interpretations. What matters is user pain. Technically if we don't >>>>> break the core SDK, users should be able to use Java SDK >=2.11.0 with >>>>> KafkaIO 2.11.0 forever. >>>>> >>>>> How are multiple versions of Kafka supported? Are they all in one >>>>> client, or is there a case for forks like ElasticSearchIO? >>>>> >>>>> Kenn >>>>> >>>>> On Mon, Apr 1, 2019 at 10:37 AM Jean-Baptiste Onofré <[email protected]> >>>>> wrote: >>>>> >>>>>> +1 to remove 0.9 support. >>>>>> >>>>>> I think it's more interesting to test and verify Kafka 2.2.0 than 0.9 >>>>>> ;) >>>>>> >>>>>> Regards >>>>>> JB >>>>>> >>>>>> On 01/04/2019 19:36, David Morávek wrote: >>>>>> > Hello, >>>>>> > >>>>>> > is there still a reason to keep Kafka 0.9 support? This >>>>>> unfortunately >>>>>> > adds lot of complexity to KafkaIO implementation. >>>>>> > >>>>>> > Kafka 0.9 was released on Nov 2015. >>>>>> > >>>>>> > My first shot on removing Kafka 0.9 support would remove second >>>>>> > consumer, which is used for fetching offsets. >>>>>> > >>>>>> > WDYT? Is this support worth keeping? >>>>>> > >>>>>> > https://github.com/apache/beam/pull/8186 >>>>>> > >>>>>> > D. >>>>>> >>>>>> -- >>>>>> Jean-Baptiste Onofré >>>>>> [email protected] >>>>>> http://blog.nanthrax.net >>>>>> Talend - http://www.talend.com >>>>>> >>>>> >>>> >> >> -- >> ---- >> Mingmin >> >
