Re: ElasticSearch with RestHighLevelClient

Hi Etienne, That was just the hint I needed! Until Beam 2.2 is released, my solution is to keep a handle to the low level client, and close that in the teardown method. This seems to solve the threading issue. ry On Wed, Oct 25, 2017 at 6:09 AM, Etienne Chauchot wrote: >

Re: ElasticSearch with RestHighLevelClient

Hi, We have used the low level rest client for the ElasticsearchIO despite the fact that there is now the high level client for maintenance reasons. Indeed, the IO production code is common to all Elasticsearch versions because the low level Rest client is compatible with all ES versions.

Re: ElasticSearch with RestHighLevelClient

Thanks Tim, I believe I'm doing what Jean-Baptiste recommends, so I guess I'll have a look at the snapshot and see what's different. I don't mind waiting a bit if it means I don't have to duplicate working code. ry On Mon, Oct 23, 2017 at 3:15 PM, Tim Robertson wrote:

Re: ElasticSearch with RestHighLevelClient

Hi Ryan, I can confirm 2.2.0-SNAPSHOT works fine with an ES 5.6 cluster. I am told 2.2.0 is expected within a couple weeks. My work is only a proof of concept for now, but I put in 300M fairly small docs at around 100,000/sec on a 3 node cluster without any issue [1]. Hope this helps, Tim [1]

Re: ElasticSearch with RestHighLevelClient

Hi Ryan, the last version of ElasticsearchIO (that will be included in Beam 2.2.0) supports Elasticsearch 5.x. The client should be created in the @Setup (or @StartBundle) and release cleanly in @Teardown (or @FinishBundle). Then, it's used in @ProcessElement to actually store the elements

Re: ElasticSearch with RestHighLevelClient

Hi JB, Thanks for your input. I'm trying to update ElasticsearchIO, and hopefully learn a bit about Beam in the process. The documentation says ElasticsearchIO only works with ES 2.X, and I'm using ES 5.6. I'd prefer not to have two ES libs in my classpath if I can avoid it. I'm just getting

Re: ElasticSearch with RestHighLevelClient

Hi Ryan, Why don't you use the ElasticsearchIO for that ? Anyway, can you share your pipeline where you have the ParDo calling your DoFn ? Thanks, Regards JB On 10/23/2017 07:50 PM, r...@ostrich-emulators.com wrote: Hi List, I'm trying to write an updated ElasticSearch client using the