[
https://issues.apache.org/jira/browse/BEAM-4389?focusedWorklogId=107049&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-107049
]
ASF GitHub Bot logged work on BEAM-4389:
----------------------------------------
Author: ASF GitHub Bot
Created on: 30/May/18 07:54
Start Date: 30/May/18 07:54
Worklog Time Spent: 10m
Work Description: echauchot commented on issue #5463: [BEAM-4389] Enable
partial updates in ElasticsearchIO
URL: https://github.com/apache/beam/pull/5463#issuecomment-393065779
LGTM as you also shared me your IT runs. BTW, A good CI improvement would be
to be able to run the IT from github so that we could publicly share the
results instead of privately run them.
Thanks for your work Tim, merging.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 107049)
Time Spent: 1.5h (was: 1h 20m)
> Enable partial updates Elasticsearch
> ------------------------------------
>
> Key: BEAM-4389
> URL: https://issues.apache.org/jira/browse/BEAM-4389
> Project: Beam
> Issue Type: New Feature
> Components: io-java-elasticsearch
> Affects Versions: 2.4.0
> Reporter: Tim Robertson
> Assignee: Tim Robertson
> Priority: Major
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> Expose a configuration option on the {{ElasticsearchIO}} to enable partial
> updates rather than full document inserts.
> Rationale: We have the case where different pipelines process different
> categories of information of the target entity (e.g. one for taxonomic
> processing, another for geospatial processing). A read and merge is not
> possible inside the batch call, meaning the only way to do it is through a
> join. The join approach is slow, and also stops the ability to run a single
> process in isolation (e.g. reprocess the geospatial component of all docs).
> Use of this configuration parameter has to be used in conjunction with
> controlling the document ID (possible since BEAM-3201) to make sense.
> The client API would include a {{withUseUpdate(...)}} such as:
> {code}
> source.apply(
> ElasticsearchIO.write()
> .withConnectionConfiguration(connectionConfiguration)
> .withIdFn(new ExtractValueFn("id"))
> .withUseUpdate(true)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
