October 2020, Beam Community Update

2020-10-30 Thread Brittany Hermann
Hi everyone, Attached is the October 2020 Beam Community Update. The purpose of this newsletter is purely community focused - giving folks the visibility to the insights of the health and growth of the community. This newsletter will not cover project releases or feature updates. Please let me kno

Re: [DISCUSS] Update Kafka dependencies in Beam Java SDK

2020-10-30 Thread Piotr Szuberski
Oh I see, I thought that our tests are testing different Kafka versions right now. There is a test suite for 2.1.0 version in addition to 1.0.0 if I understand that gradle.build file correctly. I don't mean to drop the support to earlier Kafka versions. I just mean to set the recent versions as

Re: [DISCUSS] Update Kafka dependencies in Beam Java SDK

2020-10-30 Thread Alexey Romanenko
We have a Jira task about testing KafkaIO against different Kafka versions [1]. Once it will be implemented, then I think it should solve a potential problem with new Kafka client versions. In the same time, imho, until there are Beam users, that use old Kafka client versions, we need to suppo

Re: Issues with python's external ReadFromPubSub

2020-10-30 Thread Maximilian Michels
We used to run legacy sources using the old-style Read translation. Changing it to SDF might have broken ReadFromPubSub. Could you check in the Flink jobs whether it uses the SDF code or the Read translation? For Read you should be seeing the UnboundedSourceWrapper. Looking at the code, there

Re: [DISCUSS] Update Kafka dependencies in Beam Java SDK

2020-10-30 Thread Piotr Szuberski
In my opinion it would be good to keep Beam's dependencies as close to the recent stable versions as possible and, if needed, keep the support for earlier versions. For now we keep the old dependency as the base and test whether it works for some newer versions. That way we may always ignore th

Re: Which Solr versions should be supported by Beam

2020-10-30 Thread Piotr Szuberski
Another response from Solr devlist: Based on the questions that we've seen over the past month on this list, there are still users with Solr on 6, 7, and 8. I suspect there are still Solr 5 users out there too, although they don't appear to be asking for help - likely they are in set it and forget