[
https://issues.apache.org/jira/browse/BEAM-6168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16730468#comment-16730468
]
Mark Norkin commented on BEAM-6168:
-----------------------------------
[~echauchot] thank you for the reply.
The workaround you propose would probably work for me, it's just the downside
that this is some other job which should be maintained separately.
The point you made about behavior of other connectors, just to be clear, is it
the issue of breaking underlying connectors ? If so, I'm not sure how providing
mutation on beam level IO would break that as I'm assuming it can be performed
before actually going to underlying connector. Can you elaborate on that a bit ?
Thank you.
> Allow modification of JSON value before writing to ElasticSearch
> ----------------------------------------------------------------
>
> Key: BEAM-6168
> URL: https://issues.apache.org/jira/browse/BEAM-6168
> Project: Beam
> Issue Type: Improvement
> Components: io-java-elasticsearch
> Reporter: Mark Norkin
> Assignee: Etienne Chauchot
> Priority: Major
>
> I have an Apache Beam streaming job which reads data from Kafka and writes to
> ElasticSearch using ElasticSearchIO.
> The issue I'm having is that messages in Kafka already have _{{key}}_ field,
> and using {{ElasticSearchIO.Write.withIdFn()}} I'm mapping this field to
> document _{{_id}}_ field in ElasticSearch.
> Having a big volume of data I don't want the _{{key}}_ field to be also
> written to ElasticSearch as part of _{{_source}}_.
> Is there an option/workaround that would allow doing that?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
