[
https://issues.apache.org/jira/browse/NIFI-8002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Sampson updated NIFI-8002:
--------------------------------
Description:
The existing ScrollElasticsearch processor does not fully work with newer
versions of Elasticsearch (e.g. 7.x).
A new version of this processor should be created using the REST API approach,
allowing for a query to be passed as input FlowFile content. Appropriate
parameters should be set as processor properties (e.g. index, scroll size,
scroll time) and allow for additional query string parameters to be set as
dynamic properties (see NIFI-8001).
The processor should support both
[_scroll|https://www.elastic.co/guide/en/elasticsearch/reference/current/paginate-search-results.html#scroll-search-results]
and
[_pit/search_after|https://www.elastic.co/guide/en/elasticsearch/reference/current/paginate-search-results.html#search-after].
It may be possible to simply extend the existing REST API's Query processor to
page through results with the selected method, rather than create a new
processor.
was:
The existing ScrollElasticsearch processor does not fully work with newer
versions of Elasticsearch (e.g. 7.x).
A new version of this processor should be created using the REST API approach,
allowing for a query to be passed as input FlowFile content. Appropriate
parameters should be set as processor properties (e.g. index, scroll size,
scroll time) and allow for additional query string parameters to be set as
dynamic properties (see NIFI-8001).
> Create new Elasticsearch Scroll/Paginated Query processor using REST API
> ------------------------------------------------------------------------
>
> Key: NIFI-8002
> URL: https://issues.apache.org/jira/browse/NIFI-8002
> Project: Apache NiFi
> Issue Type: Improvement
> Affects Versions: 1.12.1
> Reporter: Chris Sampson
> Priority: Minor
> Original Estimate: 5h
> Remaining Estimate: 5h
>
> The existing ScrollElasticsearch processor does not fully work with newer
> versions of Elasticsearch (e.g. 7.x).
> A new version of this processor should be created using the REST API
> approach, allowing for a query to be passed as input FlowFile content.
> Appropriate parameters should be set as processor properties (e.g. index,
> scroll size, scroll time) and allow for additional query string parameters to
> be set as dynamic properties (see NIFI-8001).
> The processor should support both
> [_scroll|https://www.elastic.co/guide/en/elasticsearch/reference/current/paginate-search-results.html#scroll-search-results]
> and
> [_pit/search_after|https://www.elastic.co/guide/en/elasticsearch/reference/current/paginate-search-results.html#search-after].
> It may be possible to simply extend the existing REST API's Query processor
> to page through results with the selected method, rather than create a new
> processor.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
