> On 30 Oct 2020, at 19:33, Piotr Szuberski <[email protected]> wrote:
>
> Oh I see, I thought that our tests are testing different Kafka versions right
> now. There is a test suite for 2.1.0 version in addition to 1.0.0 if I
> understand that gradle.build file correctly.
Yes and iirc, it was added to test Kafka headers deserialisation. Perhaps, we
could follow the similar approach with testing for other versions.
> I don't mean to drop the support to earlier Kafka versions. I just mean to
> set the recent versions as default and add tests verifying that earlier
> versions are also still supported.
Until we do test the old major versions, that should work, imo. Though, it
would be great to hear other opinions on this because KafkaIO is a very popular
IO connector and used by many users in different environments.
>
> On 2020/10/30 16:24:45, Alexey Romanenko <[email protected]> wrote:
>> We have a Jira task about testing KafkaIO against different Kafka versions
>> [1]. Once it will be implemented, then I think it should solve a potential
>> problem with new Kafka client versions.
>>
>> In the same time, imho, until there are Beam users, that use old Kafka
>> client versions, we need to support them.
>>
>> [1] https://issues.apache.org/jira/browse/BEAM-7003
>>
>>> On 30 Oct 2020, at 10:05, Piotr Szuberski <[email protected]>
>>> wrote:
>>>
>>> In my opinion it would be good to keep Beam's dependencies as close to the
>>> recent stable versions as possible and, if needed, keep the support for
>>> earlier versions.
>>>
>>> For now we keep the old dependency as the base and test whether it works
>>> for some newer versions. That way we may always ignore the dependency check
>>> report that will get resolved
>>> at 1.0.0 and miss out that some important change has been released because
>>> every dependency check points at the Kafka dependency's deprecation. If we
>>> used newer version as the base then the Dependency check will do its job in
>>> this case.
>>>
>>> I've run Kafka both local and performance tests using kafka-clients:2.6.0
>>> of Kafka clients and they passed - I'm not sure whether it answers your
>>> question about the implementation with the usage of ConsumerSpEL class.
>>>
>>>
>>> On 2020/10/28 17:06:56, Alexey Romanenko <[email protected]> wrote:
>>>> Piotr, thank you for tasing this question. Let me ask some questions
>>>> before.
>>>>
>>>> What will give us this dependencies update? What are the pros and cons?
>>>> Can users use recent versions of Kafka client with current implementation
>>>> based on ConsumerSpEL class?
>>>>
>>>>
>>>>> On 22 Oct 2020, at 10:47, Piotr Szuberski <[email protected]>
>>>>> wrote:
>>>>>
>>>>> Should we update Kafka dependencies to the recent ones (Kafka clients to
>>>>> 2.6.0 and Kafka_2.11 to 2.4.1)?
>>>>>
>>>>> What would have to be done to keep the backwards compatibility and which
>>>>> previous versions would we want to support?
>>>>>
>>>>> Kafka's backward compatibility is quite good so maybe there wouldn't be
>>>>> anything to do?
>>>>>
>>>>> Let's vote/discuss.
>>>>
>>>>
>>
>>
