[
https://issues.apache.org/jira/browse/NIFI-8002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Sampson updated NIFI-8002:
--------------------------------
Description:
The existing ScrollElasticsearch processor does not fully work with newer
versions of Elasticsearch (e.g. 7.x).
A new version of this processor should be created using the REST API approach,
allowing for a query to be passed as input FlowFile content. Appropriate
parameters should be set as processor properties (e.g. index, scroll size,
scroll time) and allow for additional query string parameters to be set as
dynamic properties (see NIFI-8001).
The processor should support both
[_scroll|https://www.elastic.co/guide/en/elasticsearch/reference/current/paginate-search-results.html#scroll-search-results]
and
[_pit/search_after|https://www.elastic.co/guide/en/elasticsearch/reference/current/paginate-search-results.html#search-after].
While scroll may be backward compatible, _pit was introduced in ES 7 (an
earlier version of search_before exists for <7, but suggest not trying to
support that as part of this ticket).
It may be possible to simply extend the existing REST API's Query processor to
page through results with the selected method, rather than create a new
processor.
was:
The existing ScrollElasticsearch processor does not fully work with newer
versions of Elasticsearch (e.g. 7.x).
A new version of this processor should be created using the REST API approach,
allowing for a query to be passed as input FlowFile content. Appropriate
parameters should be set as processor properties (e.g. index, scroll size,
scroll time) and allow for additional query string parameters to be set as
dynamic properties (see NIFI-8001).
The processor should support both
[_scroll|https://www.elastic.co/guide/en/elasticsearch/reference/current/paginate-search-results.html#scroll-search-results]
and
[_pit/search_after|https://www.elastic.co/guide/en/elasticsearch/reference/current/paginate-search-results.html#search-after].
It may be possible to simply extend the existing REST API's Query processor to
page through results with the selected method, rather than create a new
processor.
> Create new Elasticsearch Scroll/Paginated Query processor using REST API
> ------------------------------------------------------------------------
>
> Key: NIFI-8002
> URL: https://issues.apache.org/jira/browse/NIFI-8002
> Project: Apache NiFi
> Issue Type: Improvement
> Affects Versions: 1.12.1
> Reporter: Chris Sampson
> Priority: Minor
> Original Estimate: 5h
> Remaining Estimate: 5h
>
> The existing ScrollElasticsearch processor does not fully work with newer
> versions of Elasticsearch (e.g. 7.x).
> A new version of this processor should be created using the REST API
> approach, allowing for a query to be passed as input FlowFile content.
> Appropriate parameters should be set as processor properties (e.g. index,
> scroll size, scroll time) and allow for additional query string parameters to
> be set as dynamic properties (see NIFI-8001).
> The processor should support both
> [_scroll|https://www.elastic.co/guide/en/elasticsearch/reference/current/paginate-search-results.html#scroll-search-results]
> and
> [_pit/search_after|https://www.elastic.co/guide/en/elasticsearch/reference/current/paginate-search-results.html#search-after].
> While scroll may be backward compatible, _pit was introduced in ES 7 (an
> earlier version of search_before exists for <7, but suggest not trying to
> support that as part of this ticket).
> It may be possible to simply extend the existing REST API's Query processor
> to page through results with the selected method, rather than create a new
> processor.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
