[
https://issues.apache.org/jira/browse/NIFI-7990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17425451#comment-17425451
]
ASF subversion and git services commented on NIFI-7990:
-------------------------------------------------------
Commit 3892e50991ff803b762c26aec6067aecacda548e in nifi's branch
refs/heads/main from Chris Sampson
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=3892e50 ]
NIFI-7990: Add properties to map Record field as @timestamp in output to
Elasticsearch for PutElasticsearchRecord and PutElasticsearchHttpRecord
processors; NIFI-7474 allow mapped id field to be retained within the Record
for PutElasticsearchRecord
Signed-off-by: Joe Gresock <[email protected]>
This closes #4691.
> PutElasticsearch/RecordHttp processors should support Elasticsearch Data
> Streams
> --------------------------------------------------------------------------------
>
> Key: NIFI-7990
> URL: https://issues.apache.org/jira/browse/NIFI-7990
> Project: Apache NiFi
> Issue Type: Improvement
> Affects Versions: 1.11.4, 1.12.1
> Reporter: Chris Sampson
> Assignee: Chris Sampson
> Priority: Minor
> Attachments: NIFI-7990.json, NIFI-7990.xml
>
> Original Estimate: 4h
> Time Spent: 7h
> Remaining Estimate: 0h
>
> PUT Elasticsearch should support the new [Elasticsearch Data
> Streams|https://www.elastic.co/guide/en/elasticsearch/reference/current/use-a-data-stream.html#add-documents-to-a-data-stream]
> (new in Elasticsearch 7.9).
> NIFI-7474 will allow these processors to submit {{create}} operations via the
> _bulk API (which is a large part of the requirement).
> However, Data Streams require an {{@timestamp}} field to be provided in each
> document, but this field name is illegal in [Avro
> schemas|http://avro.apache.org/docs/1.8.2/spec.html#names] due to the leading
> {{@}}. The Record-based processors should therefore allow for the injection
> of this field into the JSON being sent to Elasticsearch - this could be based
> upon an existing field within the FlowFile and be identified by a property on
> the processor (e.g. like the {{_id}} field can be specified using Record
> Path).
> Optionally, the processor allow for the field used as the {{@timestamp}}
> field to be removed from the data being sent to Elasticsearch (i.e. rename
> the existing field *or* duplicate it depending upon property settings). Such
> field transformation should also take the timestamp format settings into
> account(e.g. if a {{Long}} epoch millisecond value is to be converted to a
> formatted date/time {{String}}).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
