[
https://issues.apache.org/jira/browse/BEAM-12601?focusedWorklogId=633799&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-633799
]
ASF GitHub Bot logged work on BEAM-12601:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 04/Aug/21 20:04
Start Date: 04/Aug/21 20:04
Worklog Time Spent: 10m
Work Description: echauchot commented on pull request #15257:
URL: https://github.com/apache/beam/pull/15257#issuecomment-892937377
> Changes LGTM, with the exception of Dataflow pre commit tests failing. The
failing checks seem entirely unrelated, but I'm not certain.
>
> @echauchot Over to you for final approval or comment regarding
failing/flaky Dataflow precommit tests Test logs[1] seem to indicate an error
determining whether the Dataflow job was successful or not:
>
> ```
> WARNING: Dataflow job 2021-08-03_06_12_06-18435283618356752107 did not
output a success or failure metric.
> Aug 03, 2021 1:18:37 PM
org.apache.beam.runners.dataflow.DataflowPipelineJob logTerminalState
> INFO: Job 2021-08-03_06_12_06-18435283618356752107 finished with status
DONE.
> ```
>
> [1]
https://ci-beam.apache.org/job/beam_PreCommit_Java_Examples_Dataflow_Commit/15174/testReport/org.apache.beam.examples/WordCountIT/testE2EWordCount/
Thanks @andresrc for your work and thanks @egalpin for the review !
Dataflow examples tests are failing in my PRs also and on PRs I saw lately
so it seems to be totally unrelated. I'll merge either way but it is 10pm here
now, so I'll take a better look tomorrow and merge.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 633799)
Time Spent: 3h (was: 2h 50m)
> Support append-only indices in ES output
> -----------------------------------------
>
> Key: BEAM-12601
> URL: https://issues.apache.org/jira/browse/BEAM-12601
> Project: Beam
> Issue Type: Improvement
> Components: io-java-elasticsearch
> Reporter: Andres Rodriguez
> Priority: P2
> Time Spent: 3h
> Remaining Estimate: 0h
>
> Currently, the Apache Beam Elasticsearch sink is
> [using|https://github.com/apache/beam/blob/master/sdks/java/io/elasticsearch/src/main/java/org/apache/beam/sdk/io/elasticsearch/ElasticsearchIO.java#L1532]
> the
> [index|https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-bulk.html#bulk-api-request-body]
> bulk API operation to add data to the target index. When using append-only
> indices it is better to use the
> [create|https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-bulk.html#bulk-api-request-body]
> operation. This also applies to new append-only indexes, like [data
> streams|https://www.elastic.co/guide/en/elasticsearch/reference/7.x/use-a-data-stream.html#add-documents-to-a-data-stream].
> The scope of this improvement is to add a new boolean configuration option,
> {{append-only}}, to the Elasticsearch sink, with a default value of {{false}}
> (to keep the current behaivour) that when enabled makes it use the {{create}}
> operation instead of the {{index}} one when sending data.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
