Re: A new reworked Elasticsearch 7+ IO module

Hi Kenn, The user does not specify the backendVersion targeted (at least on the current version of the IO) it is transparent to him: the IO detects the version with a REST call and adapts its behavior. But, anyway, I agree, we need to put at least a WARN if detected version is 2. As the new

Re: A new reworked Elasticsearch 7+ IO module

Hi all, The survey regarding Elasticsearch support in Beam is now closed. Here are the results after 38 days: users using ESv2: 0 ESV5: 1 ESV6: 5 ESV7: 8 So, the new version of ElasticsearchIO after the refactoring discussed in this thread will no more support Elasticsearch v2. Regards

Re: A new reworked Elasticsearch 7+ IO module

Hi, I think WARN makes sense and the safest approach. It allows users to be notify and eventually update or back on previous Beam IO version. Regards JB > Le 6 mars 2020 à 18:49, Kenneth Knowles a écrit : > > Since the user provides backendVersion, here are some possible levels of > things

Re: A new reworked Elasticsearch 7+ IO module

Since the user provides backendVersion, here are some possible levels of things to add in expand() based on that (these are extra niceties beyond the agreed number of releases to remove) - WARN for backendVersion < n - reject for backendVersion < n with opt-in pipeline option to keep it working

Re: A new reworked Elasticsearch 7+ IO module

Hi all, it's been 3 weeks since the survey on ES versions the users use. The survey received very few responses: only 9 responses for now (multiple versions possible of course). The responses are the following: ES2: 0 clients, ES5: 1, ES6: 5, ES7: 8 It tends to go toward a drop of ES2

Re: A new reworked Elasticsearch 7+ IO module

Hi Cham, thanks for your comments ! I just sent an email to user ML with a survey link to count ES uses per version: https://lists.apache.org/thread.html/rc8185afb8af86a2a032909c13f569e18bd89e75a5839894d5b5d4082%40%3Cuser.beam.apache.org%3E Best Etienne On 10/02/2020 19:46, Chamikara

Re: A new reworked Elasticsearch 7+ IO module

On Thu, Feb 6, 2020 at 8:13 AM Etienne Chauchot wrote: > Hi, > > please see my comments inline > On 06/02/2020 16:24, Alexey Romanenko wrote: > > Please, see my comments inline. > > On 6 Feb 2020, at 10:50, Etienne Chauchot wrote: > > 1. regarding version support: ES v2 is no more maintained by

Re: A new reworked Elasticsearch 7+ IO module

Hi, please see my comments inline On 06/02/2020 16:24, Alexey Romanenko wrote: Please, see my comments inline. On 6 Feb 2020, at 10:50, Etienne Chauchot > wrote: 1. regarding version support: ES v2 is no more maintained by Elastic since 2018/02

Re: A new reworked Elasticsearch 7+ IO module

Please, see my comments inline. > On 6 Feb 2020, at 10:50, Etienne Chauchot wrote: 1. regarding version support: ES v2 is no more maintained by Elastic since 2018/02 so we plan to remove it from the IO. In the past we already retired versions (like spark 1.6 for instance). >>

Re: A new reworked Elasticsearch 7+ IO module

Hi, Let’s sync together about this IO. Regarding mock and IOs, and Etienne’s comment, there are two things: 1. Of course, it’s always preferable to use concrete backend, but several times it’s not possible. It’s there mock is required. 2. The mock can be smart enough to cover core IO behavior

Re: A new reworked Elasticsearch 7+ IO module

Hi, Thanks all for your comments, my comments are inline On 06/02/2020 09:47, Ludovic Boutros wrote: Hi all, First, thank you all for your answers and especially, Etienne for your time, advises and kindness :) @Jean-Baptiste, any help on this module is welcome of course. @Chamikara

Re: A new reworked Elasticsearch 7+ IO module

Hi all, First, thank you all for your answers and especially, Etienne for your time, advises and kindness :) @Jean-Baptiste, any help on this module is welcome of course. @Chamikara Jayalath, my aswers are inline. Have a good day ! Ludovic Le mer. 5 févr. 2020 à 20:15, Chamikara Jayalath a

Re: A new reworked Elasticsearch 7+ IO module

On Wed, Feb 5, 2020 at 6:35 AM Etienne Chauchot wrote: > Still there is something I don't agree with is that IOs can be tested on > mock. We don't really test IO behavior with mocks: there is always special > behaviors that cannot be reproduced in mocks (split, load, with corner > cases etc...).

Re: A new reworked Elasticsearch 7+ IO module

Still there is something I don't agree with is that IOs can be tested on mock. We don't really test IO behavior with mocks: there is always special behaviors that cannot be reproduced in mocks (split, load, with corner cases etc...). There was in the past IOs that were tested using mocks and

Re: A new reworked Elasticsearch 7+ IO module

Hi, We talked in the past about multiple/single module. IMHO the always preferred goal is to have a single module. However, it’s tricky when we have such difference, including on the user facing API. So, I would go with module per version, or use a specified version for a target Beam release.

Re: A new reworked Elasticsearch 7+ IO module

Hi all, We had a long discussion with Ludovic about this IO. I'd like to share with you to keep you informed and also gather your opinions 1. regarding version support: ES v2 is no more maintained by Elastic since 2018/02 so we plan to remove it from the IO. In the past we already retired

Re: A new reworked Elasticsearch 7+ IO module

I’m second for this question. We have a similar (maybe a bit less painful) issue for KafkaIO and it would be useful to have a general strategy for such cases about how to deal with that. > On 24 Jan 2020, at 21:54, Kenneth Knowles wrote: > > Would it make sense to have different

Re: A new reworked Elasticsearch 7+ IO module

Hi Ludovic, First of all thanks for your work. Then, please be aware that the current ES IO on master supports ES7 already and will be part of Beam 2.19. I understand that your approach enables many new features which is great ! For the record, the current ES module was designed to have

Re: A new reworked Elasticsearch 7+ IO module

Hi all, First, thank you for your great answers. I thank Zhong Chen and Etienne Chauchot for their great job on this too ! Alexey and Chamikara, I understand your point of view. Actually, I have the same as much as possible. But in this case, my goal was to be able to do all the following

Re: A new reworked Elasticsearch 7+ IO module

Would it make sense to have different version-specialized connectors with a common core library and common API package? On Fri, Jan 24, 2020 at 11:52 AM Chamikara Jayalath wrote: > Thanks for the contribution. I agree with Alexey that we should try to add > any new features brought in with the

Re: A new reworked Elasticsearch 7+ IO module

Thanks for the contribution. I agree with Alexey that we should try to add any new features brought in with the new PR into existing connector instead of trying to maintain two implementations. Thanks, Cham On Fri, Jan 24, 2020 at 9:01 AM Alexey Romanenko wrote: > Hi Ludovic, > > Thank you for

Re: A new reworked Elasticsearch 7+ IO module

Hi Ludovic, Thank you for working on this and sharing the details with us. This is really great job! As I recall, we already have some support of Elasticsearch7 in current ElasticsearchIO (afaik, at least they are compatible), thanks to Zhong Chen and Etienne Chauchot, who were working on

A new reworked Elasticsearch 7+ IO module

Dear all, I have written a completely reworked Elasticsearch 7+ IO module. It can be found here: https://github.com/ludovic-boutros/beam/tree/fresh-reworked-elasticsearch-io-v7/sdks/java/io/elasticsearch7 This is a quite advance WIP work but I'm a quite new user of Apache Beam and I would like