It looks more like a client issue where the stream is already read, maybe give a try to reproduce it in a unit test in beam ES module? This will enable us to help you more accurately.
Romain Manni-Bucau @rmannibucau <https://twitter.com/rmannibucau> | Blog <https://rmannibucau.metawerx.net/> | Old Blog <http://rmannibucau.wordpress.com> | Github <https://github.com/rmannibucau> | LinkedIn <https://www.linkedin.com/in/rmannibucau> | Book <https://www.packtpub.com/application-development/java-ee-8-high-performance> Le jeu. 11 oct. 2018 à 16:18, Wout Scheepers < [email protected]> a écrit : > Hey Romain, > > > > I’ve check and am using the same http client as beam 2.7.0. > > Just to be sure, I’ve created a minimal reproducible with a fresh project > with only the following dependencies in my build.gradle: > dependencies { > compile (*'org.apache.beam:beam-sdks-java-io-elasticsearch:2.7.0'*) > compile (*'org.apache.beam:beam-runners-direct-java:2.7.0'*) > compile > (*'org.apache.beam:beam-runners-google-cloud-dataflow-java:2.7.0'*) > compile (*'org.apache.beam:beam-sdks-java-extensions-protobuf:2.7.0'*) > compile > (*'org.apache.beam:beam-sdks-java-extensions-google-cloud-platform-core:2.7.0'*) > compile > (*'org.apache.beam:beam-sdks-java-io-google-cloud-platform:2.7.0'*) > compile (*'org.apache.beam:beam-sdks-java-io-common:2.7.0'*) > compile (*'org.apache.beam:beam-sdks-java-extensions-json-jackson:2.7.0'*) > compile (*'org.apache.beam:beam-sdks-java-io-jdbc:2.7.0'*) > > > testCompile > *'org.hamcrest:hamcrest-all:1.3' *testCompile > *'org.assertj:assertj-core:3.4.1' *testCompile > *'junit:junit:4.12'*} > > > > However, the problem still persists when writing a document to elastic > with the retryConfiguration set. > > I guess the problem lies at my elastic version, as JB implies? > > > > Anyway, thanks for the suggestion. > > > > Wout > > > > *From: *Romain Manni-Bucau <[email protected]> > *Reply-To: *"[email protected]" <[email protected]> > *Date: *Wednesday, 10 October 2018 at 16:53 > *To: *"[email protected]" <[email protected]> > *Subject: *Re: ElasticIO retry configuration exception > > > > Hi Wout, > > > > Maye check your classpath http client versions (against > https://github.com/apache/beam/blob/v2.7.0/sdks/java/io/elasticsearch/build.gradle > for instance). > > > Romain Manni-Bucau > @rmannibucau <https://twitter.com/rmannibucau> | Blog > <https://rmannibucau.metawerx.net/> | Old Blog > <http://rmannibucau.wordpress.com> | Github > <https://github.com/rmannibucau> | LinkedIn > <https://www.linkedin.com/in/rmannibucau> | Book > <https://www.packtpub.com/application-development/java-ee-8-high-performance> > > > > > > Le mer. 10 oct. 2018 à 15:37, Wout Scheepers < > [email protected]> a écrit : > > Hey JB, > > Thanks for your fast reply. > The elastic version we're using is 5.6.2. > > "version": { > "number": "5.6.2", > "build_hash": "57e20f3", > "build_date": "2017-09-23T13:16:45.703Z", > "build_snapshot": false, > "lucene_version": "6.6.1" > } > > > Wout > > > > On 10/10/2018, 15:34, "Jean-Baptiste Onofré" <[email protected]> wrote: > > Hi Wout, > > what's the elasticsearch version ? (just to try to reproduce) > > Thanks, > Regards > JB > > On 10/10/2018 15:31, Wout Scheepers wrote: > > Hey all, > > > > > > > > When using .withRetryConfiguration()for ElasticsearchIO, I get the > > following stacktrace: > > > > > > > > Caused by: > com.fasterxml.jackson.databind.exc.MismatchedInputException: > > No content to map due to end-of-input > > > > at [Source: (org.apache.http.nio.entity.ContentInputStream); line: 1, > > column: 0] > > > > at > > > com.fasterxml.jackson.databind.exc.MismatchedInputException.from(MismatchedInputException.java:59) > > > > at > > > com.fasterxml.jackson.databind.ObjectMapper._initForReading(ObjectMapper.java:4133) > > > > at > > > com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:3988) > > > > at > > > com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3058) > > > > at > > org.apache.beam.sdk.io > .elasticsearch.ElasticsearchIO.parseResponse(ElasticsearchIO.java:167) > > > > at > > org.apache.beam.sdk.io > .elasticsearch.ElasticsearchIO.checkForErrors(ElasticsearchIO.java:171) > > > > at > > org.apache.beam.sdk.io > .elasticsearch.ElasticsearchIO$Write$WriteFn.flushBatch(ElasticsearchIO.java:1213) > > > > at > > org.apache.beam.sdk.io > .elasticsearch.ElasticsearchIO$Write$WriteFn.finishBundle(ElasticsearchIO.java:1183) > > > > > > > > I’ve been breaking my head on this one. > > > > Apparently the elastic Response object can’t be parsed anymore in the > > checkForErrors() method. > > > > However, it is parsed successfully in the default RetryPredicate’s > test > > method, which is called in flushBatch() in the if clause related to > the > > retryConfig (ElasticsearchIO:1201). > > > > As far as I know, the Response object is not altered. > > > > > > > > Any clues why this doesn’t work for me? > > > > I really need this feature, as inserting 40M documents into elastic > > results in too many retry timeouts ☺. > > > > > > > > Thanks! > > Wout > > > > > > > > > > > > -- > Jean-Baptiste Onofré > [email protected] > http://blog.nanthrax.net > Talend - http://www.talend.com > >
