[
https://issues.apache.org/jira/browse/BEAM-6052?focusedWorklogId=176033&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-176033
]
ASF GitHub Bot logged work on BEAM-6052:
----------------------------------------
Author: ASF GitHub Bot
Created on: 17/Dec/18 14:12
Start Date: 17/Dec/18 14:12
Worklog Time Spent: 10m
Work Description: echauchot commented on a change in pull request #7206:
[BEAM-6052] Fix the bug for checkForErrors of ElasticsearchIO in case of updates
URL: https://github.com/apache/beam/pull/7206#discussion_r242145923
##########
File path:
sdks/java/io/elasticsearch/src/main/java/org/apache/beam/sdk/io/elasticsearch/ElasticsearchIO.java
##########
@@ -176,21 +176,27 @@ static JsonNode parseResponse(HttpEntity responseEntity)
throws IOException {
return mapper.readValue(responseEntity.getContent(), JsonNode.class);
}
- static void checkForErrors(HttpEntity responseEntity, int backendVersion)
throws IOException {
+ static void checkForErrors(HttpEntity responseEntity, int backendVersion,
boolean partialUpdate)
+ throws IOException {
JsonNode searchResult = parseResponse(responseEntity);
boolean errors = searchResult.path("errors").asBoolean();
if (errors) {
StringBuilder errorMessages =
new StringBuilder("Error writing to Elasticsearch, some elements
could not be inserted:");
JsonNode items = searchResult.path("items");
- //some items present in bulk might have errors, concatenate error
messages
+ // some items present in bulk might have errors, concatenate error
messages
for (JsonNode item : items) {
String errorRootName = "";
- if (backendVersion == 2) {
- errorRootName = "create";
- } else if (backendVersion == 5 || backendVersion == 6) {
- errorRootName = "index";
+ // when use partial update, the response items includes all is update.
Review comment:
ping?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 176033)
Time Spent: 4h 40m (was: 4.5h)
> elasticsearchIO checkForErrors method bug
> -----------------------------------------
>
> Key: BEAM-6052
> URL: https://issues.apache.org/jira/browse/BEAM-6052
> Project: Beam
> Issue Type: Bug
> Components: io-java-elasticsearch
> Environment: beam-sdk-java-io-elasticsearch-2.8.0
> Reporter: Fred k
> Assignee: Etienne Chauchot
> Priority: Minor
> Labels: easyfix
> Time Spent: 4h 40m
> Remaining Estimate: 0h
>
> When i use Write to write update bulk request to elasticsearch, it appear the
> exception below:
> {code:java}
> Caused by: java.io.IOException: Error writing to Elasticsearch, some elements
> could not be inserted:
> at
> org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO.checkForErrors(ElasticsearchIO.java:215)
> at
> org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO$Write$WriteFn.flushBatch(ElasticsearchIO.java:1235)
> at
> org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO$Write$WriteFn.processElement(ElasticsearchIO.java:1199)
> {code}
> I check the method of checkForErrors, found out that can not parse the
> response include update contents. So i add the logic for parse update, i can
> see the output like below:
> {code:java}
> Caused by: java.io.IOException: Error writing to Elasticsearch, some elements
> could not be inserted:
> Document id 1465285334751e039cc4883a8a270191: rejected execution of
> org.elasticsearch.transport.TransportService$7@6c8edc37 on
> EsThreadPoolExecutor[name = gjzx159-node2/write, queue capacity = 200,
> org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@56b9faa3[Running,
> pool size = 40, active threads = 40, queued tasks = 198, completed tasks =
> 10324166]] (es_rejected_execution_exception)
> Document id e2722c653c65a4cb119e9b8dc44e37ad: rejected execution of
> org.elasticsearch.transport.TransportService$7@6c8edc37 on
> EsThreadPoolExecutor[name = gjzx159-node2/write, queue capacity = 200,
> org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@56b9faa3[Running,
> pool size = 40, active threads = 40, queued tasks = 198, completed tasks =
> 10324166]] (es_rejected_execution_exception)
> Document id b25472e3665695c49861f6eceee5531a: rejected execution of
> org.elasticsearch.transport.TransportService$7@6c8edc37 on
> EsThreadPoolExecutor[name = gjzx159-node2/write, queue capacity = 200,
> org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@56b9faa3[Running,
> pool size = 40, active threads = 40, queued tasks = 198, completed tasks =
> 10324166]] (es_rejected_execution_exception)
> Document id 022c1accdae82f6fe4108ba7989732fc: rejected execution of
> org.elasticsearch.transport.TransportService$7@6c8edc37 on
> EsThreadPoolExecutor[name = gjzx159-node2/write, queue capacity = 200,
> org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@56b9faa3[Running,
> pool size = 40, active threads = 40, queued tasks = 198, completed tasks =
> 10324166]] (es_rejected_execution_exception)
> at
> org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO.checkForErrors(ElasticsearchIO.java:215)
> at
> org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO$Write$WriteFn.flushBatch(ElasticsearchIO.java:1235)
> at
> org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO$Write$WriteFn.processElement(ElasticsearchIO.java:1199)
> {code}
> the reponse content is like below:
> {
> "took": 293,
> "errors": true,
> "items": [
> {
> "update": {
> "_index": "test_kevin_2018-11",
> "_type": "kevin",
> "_id": "8d7664286c0887c637229166c7c613bc",
> "_version": 1,
> "result": "noop",
> "_shards": {
> "total": 1,
> "successful": 1,
> "failed": 0
> },
> "status": 200
> }
> },
> {
> "update": {
> "_index": "test_kevin_2018-11",
> "_type": "kevin",
> "_id": "49952be98f4fc160f56bcdb33b1dbf7e",
> "status": 429,
> "error": {
> "type": "es_rejected_execution_exception",
> "reason": "rejected execution of
> org.elasticsearch.transport.TransportService$7@3f70bbe7 on
> EsThreadPoolExecutor[name = gjzx159-node2/write, queue capacity = 200,
> org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@56b9faa3[Running,
> pool size = 40, active threads = 40, queued tasks = 200, completed tasks =
> 10034174]]"
> }
> }
> }
> }
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
