neoremind opened a new pull request, #16534: URL: https://github.com/apache/lucene/pull/16534
FreqProxTermsEnum is the bridge between the in-memory hash+pool and the codec writer, it uses `termsPool` to retrieve real term bytes when iterating terms in lexicographical order during segment flush, but the constructor passes `terms.bytePool` where it should be `terms.termBytePool`. There is no problem now because we do pool sharing today, we use the same `ByteBlockPool` to store both term bytes and their corresponding postings data, so the two 1) `terms.bytePool` (major responsibility is to store terms' posting data in memory) and 2) `terms.termBytePool` (used to store terms' bytes) point to the same pool. Semantically speaking, FreqProxTermsEnum's constructor should use the right reference even though they refer to the same pool. Add a bit clarification of how the in-memory things before segment flush work based on my investigation. The below class diagram shows the relationship between `TermsHash`, `BytesRefHash` and the relevant pools. <img width="3192" height="3773" alt="termshash" src="https://github.com/user-attachments/assets/4865de7b-cc23-43a0-af22-909b599b13d9"; /> `TermsHash` maintains three pool references: - intPool (IntBlockPool): Stores per-term posting streams' write cursors. Each cursor tracks the offset in bytePool where the next postings write should go. For example, when we store one field's term M in doc N, the cursor directs to where term M's doc N's doc ID delta, freq, and pos should get persisted. - bytePool (ByteBlockPool): Today we do pool sharing, it stores both term bytes and postings streams (doc ID delta, freq, pos, offsets, payloads) together in the same pool. - termBytePool (ByteBlockPool): This points to the same bytePool above. `BytesRefHash`: For each term, it stores a bytesStart offset pointing into termBytePool, and supports fast hash-based find/add operations to dedup terms. `IntBlockPool`: Each 4-bytes represents a write cursor, the current position in bytePool where the next byte for that term's stream posting data should be written to. `ByteBlockPool`: We interleave term bytes and their respective postings data in this single pool now. The postings data for each term forms a linked list of growing slices. For frequent terms like "the", "a", "and", the linked list grows long, span multiple slices, they are not contiguous, usually hop. For rare terms, usually just the term length prefix + term bytes + a 5-byte first postings slice. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
