[
https://issues.apache.org/jira/browse/BEAM-5757?focusedWorklogId=478439&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-478439
]
ASF GitHub Bot logged work on BEAM-5757:
----------------------------------------
Author: ASF GitHub Bot
Created on: 03/Sep/20 10:01
Start Date: 03/Sep/20 10:01
Worklog Time Spent: 10m
Work Description: jithin97 commented on a change in pull request #12670:
URL: https://github.com/apache/beam/pull/12670#discussion_r482860614
##########
File path:
sdks/java/io/elasticsearch/src/main/java/org/apache/beam/sdk/io/elasticsearch/ElasticsearchIO.java
##########
@@ -1346,17 +1359,36 @@ private static String lowerCaseOrNull(String input) {
@ProcessElement
public void processElement(ProcessContext context) throws Exception {
- String document = context.element();
- String documentMetadata = getDocumentMetadata(document);
-
- // index is an insert/upsert and update is a partial update (or insert
if not existing)
- if (spec.getUsePartialUpdate()) {
- batch.add(
- String.format(
- "{ \"update\" : %s }%n{ \"doc\" : %s, \"doc_as_upsert\" :
true }%n",
- documentMetadata, document));
+ String document = context.element(); // use configuration and
auto-generated document IDs
+ String documentMetadata = "{}";
+ boolean isDelete = false;
+ if (spec.getIndexFn() != null || spec.getTypeFn() != null ||
spec.getIdFn() != null) {
+ // parse once and reused for efficiency
+ JsonNode parsedDocument = OBJECT_MAPPER.readTree(document);
+ documentMetadata = getDocumentMetadata(parsedDocument);
+ if (spec.getIsDeleteFn() != null) {
+ isDelete = spec.getIsDeleteFn().apply(parsedDocument);
+ // if it is a delete opration, then it is mandatory to specify the
document id using
+ // getIdFn
+ checkArgument(
+ !(isDelete && spec.getIdFn() == null),
+ "Id needs to be specified by withIdFn for delete operation");
Review comment:
Yes, I think that is a better way to handle it. I fixed it.
Thanks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 478439)
Time Spent: 1h 40m (was: 1.5h)
> Elasticsearch IO provide delete function
> ----------------------------------------
>
> Key: BEAM-5757
> URL: https://issues.apache.org/jira/browse/BEAM-5757
> Project: Beam
> Issue Type: New Feature
> Components: io-java-elasticsearch
> Affects Versions: 2.7.0
> Reporter: Prasad Marne
> Priority: P3
> Labels: stale-assigned
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> Hello beam maintainers,
> I am trying to delete some index from Elasticsearch using beam pipeline. I
> couldn't find and delete function in ElasticsearchIO. It would be nice to
> have and would make sense for cleaning up indexes overtime.
> I also checked some other IO classes and they also don't support delete. Not
> sure if beam has some policy against supporting delete.
> Please guide me on this. I am willing to create pull request if this feature
> makes sense to project contributors.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
