SteNicholas opened a new pull request, #486: URL: https://github.com/apache/paimon-cpp/pull/486
<!-- PR titles must follow Conventional Commits: <type>(<optional-scope>): <description> --> ### Purpose <!-- Linking this pull request to the issue --> Linked issue: #404 (follow-up of #461) <!-- What is the purpose of the change --> Since #461, the `tsan-x86_64` job of `Build and Test` runs past its 120-minute timeout ([run 41000500442](https://github.com/apache/paimon-cpp/actions/runs/41000500442/job/123883528750)): `paimon-global-index-test` grew from about 1400s to about 3500s. Every `LuceneGlobalIndexReader` and `LuceneGlobalIndexWriter` constructed its own `cppjieba::Jieba`, reloading about 11.6MB of dictionaries each time. A table-level full-text search creates one reader per shard, so every query paid this cost once per shard, and the table-level full-text search tests added in #461 run many such queries. This PR adds `LuceneUtils::GetJieba()`: - Each dictionary directory is loaded once per process, and a `std::shared_ptr<const cppjieba::Jieba>` is shared by all Lucene index readers and writers. The lock is held while loading, so concurrent first callers wait for a single load. - Segmentation only calls const `cppjieba` methods, which keep no mutable state and are safe to call concurrently. - The cache is keyed by the canonical absolute path. A relative `PAIMON_JIEBA_DICT_DIR` is resolved against the current working directory instead of reusing dictionaries loaded from another one, and a directory that cannot be resolved returns an `IOError` instead of aborting inside `cppjieba`. - Trade-off: the dictionaries stay in memory until the process exits, so changes to the dictionary files take effect only after a restart. Measured on the fork CI of the first revision of this change (`cc54c144`, before the path canonicalization): | Sanitizer job | Before #461 | After #461 | This PR | |---|---|---|---| | `tsan-x86_64`: `paimon-global-index-test` | 1382s | 3512s | 156s | | `tsan-x86_64`: whole job | passed | cancelled at 120 min | passed in 62 min | | `tsan-aarch64`: `paimon-global-index-test` | 1019s | 3045s | 103s | ### Tests <!-- List UT and IT cases to verify this change --> - `LuceneUtilsTest.TestGetJiebaSharesOneSegmenter`: concurrent first callers on a fresh dictionary directory get one segmenter and segment correctly with it; after all callers release their references, the cache still holds the segmenter and returns it again. - `LuceneUtilsTest.TestGetJiebaResolvesRelativeDictionaryDir`: an unresolvable relative directory returns an error; the same relative directory resolved from two working directories loads two isolated segmenters (a user word changes the `Cut` result of one of them); the absolute path returns the cached segmenter. - The existing `LuceneGlobalIndexTest` cases and the Lucene cases of `GlobalIndexTest` cover the reader and writer paths. Validation: `pre-commit` passed on the changed files. The change was not built locally. On the fork, the first revision passed every completed job of `Build and Test` (including `tsan-x86_64`, `tsan-aarch64`, `asan-ubsan-x86_64` and `asan-ubsan-aarch64`) and `Gcc 8 Test`; CI of the current revision is pending. ### API and Format <!-- Does this change affect API in include dir or storage format or protocol --> No. `LuceneUtils` is internal; there are no changes under `include/` and no storage format or protocol changes. ### Documentation <!-- Does this change introduce a new feature --> No new feature. The description of `PAIMON_JIEBA_DICT_DIR` in `docs/source/user_guide/global_index.rst` now explains how a relative directory is resolved and that dictionary changes take effect only after the process restarts. ### Generative AI tooling Generated-by: Claude Code (Claude Opus 5.5) 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
