J-HowHuang opened a new pull request, #19212:
URL: https://github.com/apache/pinot/pull/19212
## Description
Currently, the implementation of `getColumnsWithIndex` for text index or
vector index involves iterations over columns, and for each column it checks if
a few certain files exist, because text index and vector index store their
index as separate files alongside `column.psf`.
This introduces unnecessary `stat()` system call coming from
`File.exists()`. And result in a poor performance while running `needReload`
API.
This PR changes the `getColumnsWithIndex` implementation for text index and
vector index so that it lists the directory once, and collect the columns that
has the corresponding index file in the directory.
## Performance
* Setup:
* single server
* 20-column offline table
* local segments.
* `needReload` measured with no config change pending
* Task: for each segment the check asked "which columns have a text or
vector index?"
| Segments per server | Before this PR | After this PR |
| --------------: | -----: | ----: |
| 10 000 | 8.6 s | 1.3 s |
| 50 000 | 52.3 s | 6.3 s |
Per segment: 1.05 ms → 0.13 ms (8×). Still O(N) — this changes the constant,
not the shape.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
