dpol1 commented on issue #2212: URL: https://github.com/apache/stormcrawler/issues/2212#issuecomment-6126893648
+1 on removing `opensearch` and renaming `opensearch-java`. Now that #2225 is in, I compared the two modules again, beyond the differences the README already lists. A few things I would sort out before the switch: - HybridSpout sends the sort values of the last hit back as `FieldValue@...` strings in `search_after`. With `nextFetchDate` in `bucket.sort.field`, as in the sample conf, every query sent when a queue empties fails with `failed to parse date field [...FieldValue@c82c3cf8]`, so only the aggregations bring URLs (already in the 3.6.0 and 3.7.0 releases). I have a fix with a test, will open a PR. - bulk items rejected with a 429 fail their tuples straight away, whereas the legacy BulkProcessor retries them with a backoff - bulks are not flushed on size, the legacy one flushes at 5 MB (our README says otherwise, I'll fix it) - smaller ones: a `null` metadata value fails the tuple in StatusUpdaterBolt, and a missing mapping file only logs a warning when the index has to be created, where the legacy module fails at startup For the 429s and the size, the client has a `BulkIngester` which does both. Would you rather switch to it or extend AsyncBulkProcessor? The module's tests run against OpenSearch 3.5.0. I also ran them against 2.19.6, the version the legacy module tests with, plus the new HybridSpout test, and they all pass. I can run your benchmarks (#2213, #2226) on both modules with TLS and basic auth (your run here had the security plugin disabled), and with HybridSpout. Do you see any of these as blockers? Happy to do the removal and the rename once these are sorted. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
