SteNicholas commented on issue #404: URL: https://github.com/apache/paimon-cpp/issues/404#issuecomment-6569752900
Follow-up after #461: since it was merged, the `tsan-x86_64` job of `Build and Test` runs past its 120-minute timeout ([run 41000500442](https://github.com/apache/paimon-cpp/actions/runs/41000500442/job/123883528750)). `paimon-global-index-test` grew from about 1400s to about 3500s on `tsan-x86_64`, and from about 1000s to about 3000s on `tsan-aarch64`. **Root cause:** every `LuceneGlobalIndexReader` and `LuceneGlobalIndexWriter` constructed its own `cppjieba::Jieba`, reloading about 11.6MB of Jieba dictionaries each time. A table-level full-text search creates one reader per shard, so every query paid this cost once per shard, and the table-level full-text search tests added in #461 run many such queries. The cost is not limited to tests: every table-level search on a `lucene-fts` index pays it in production too. **Fix:** #486 loads each dictionary directory once per process and shares a const segmenter with all Lucene index readers and writers. With it, `paimon-global-index-test` takes about 156s on `tsan-x86_64` and about 103s on `tsan-aarch64`, and the `tsan-x86_64` job passes in about an hour. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
