dpol1 opened a new pull request, #2237: URL: https://github.com/apache/stormcrawler/pull/2237
I hit this while comparing the two OpenSearch modules for #2212. With the `opensearch.status.bucket.sort.field` of the [sample configuration](https://github.com/apache/stormcrawler/blob/main/external/opensearch/opensearch-conf.yaml) (`nextFetchDate`, `url`), every query HybridSpout sends to reload a queue fails with `Exception with OpenSearch query`. The root cause in the OpenSearch response is: ``` failed to parse date field [org.opensearch.client.opensearch._types.FieldValue@c82c3cf8] with format [date_optional_time] ``` The spout then emits only the URLs returned by the aggregations. HybridSpout keeps the sort values as `FieldValue` objects and was sending their `toString()` in `search_after`. 3.6.0 and 3.7.0 are affected. The legacy `opensearch` module passes the sort values as they are and is not affected. The new `HybridSpoutTest` puts three URLs in one queue and reads one URL per query, so the second and third come from reloads. It expects all three; on main it gets only the first. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
