[
https://issues.apache.org/jira/browse/NIFI-7449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aquilino Viveiros updated NIFI-7449:
------------------------------------
Description:
Using Elasticsearch 7.6 (probably true for any 7.x version) and using the
processor to scroll over the data Nifi receives a 400 error from ES with the
following content => *Field expansion matches too many fields, limit: 1024*
Precondition: Index documents and mapping must have fields over the limit above.
When initiating the scroll, the Nifi processor (1) uses a GET with a query
parameter as follow:
*[http://localhost:9200/index/_search?q=type:*]*
Which causes the error above. Removing the q parameter eliminates the error
Few ideas to fix it
1) Make query parameter optional (keep GET)
2) Refactor existing code to use POST instead of GET.
3) Introduce a different property for a Query to be used as a POST (does not
trigger the error above)
Tested on
Elasticsearch 7.6
Nifi 1.9.0 and 1.9.2
1)
[https://github.com/apache/nifi/blob/9608fe33fa4a3f39ee6ddca54184ccc85ef1a163/nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/ScrollElasticsearchHttp.java#L284]
was:
Using Elasticsearch 7.6 (probably true for any 7.x version) and using the
processor to scroll over the data Nifi receives a 400 error from ES with the
following content => *Field expansion matches too many fields, limit: 1024*
Precondition: Index documents and mapping must have fields over the limit above.
When initiating the scroll, the Nifi processor (1) uses a GET with a query
parameter as follow:
*[http://localhost:9200/index/_search?q=type:*]*
Which causes the error above. Removing the q parameter eliminates the error
Few ideas to fix it
1) Make query optional (keep GET)
2) Introduce a different property for Query to be used as a POST (does not
trigger the error above)
3) Refactor existing code to use POST instead of GET.
Tested on
Elasticsearch 7.6
Nifi 1.9.0 and 1.9.2
1)
https://github.com/apache/nifi/blob/9608fe33fa4a3f39ee6ddca54184ccc85ef1a163/nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/ScrollElasticsearchHttp.java#L284
> ScrollElasticsearchHttp processor fails against ES 7.6 with error 400 =>
> "Field expansion matches too many fields, limit: 1024"
> -------------------------------------------------------------------------------------------------------------------------------
>
> Key: NIFI-7449
> URL: https://issues.apache.org/jira/browse/NIFI-7449
> Project: Apache NiFi
> Issue Type: Bug
> Reporter: Aquilino Viveiros
> Priority: Major
>
> Using Elasticsearch 7.6 (probably true for any 7.x version) and using the
> processor to scroll over the data Nifi receives a 400 error from ES with the
> following content => *Field expansion matches too many fields, limit: 1024*
>
> Precondition: Index documents and mapping must have fields over the limit
> above.
>
> When initiating the scroll, the Nifi processor (1) uses a GET with a query
> parameter as follow:
> *[http://localhost:9200/index/_search?q=type:*]*
> Which causes the error above. Removing the q parameter eliminates the error
>
> Few ideas to fix it
> 1) Make query parameter optional (keep GET)
> 2) Refactor existing code to use POST instead of GET.
> 3) Introduce a different property for a Query to be used as a POST (does not
> trigger the error above)
>
> Tested on
> Elasticsearch 7.6
> Nifi 1.9.0 and 1.9.2
>
> 1)
> [https://github.com/apache/nifi/blob/9608fe33fa4a3f39ee6ddca54184ccc85ef1a163/nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/ScrollElasticsearchHttp.java#L284]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
