hristo-stripe opened a new pull request, #8496:
URL: https://github.com/apache/pinot/pull/8496
We are seeing ~50% of memory allocated in a given period from these arrays.
Our use case is such that we have heavy filtering from a given segment,
producing
fewer than 100 docs in the vast majority of cases.
This patch reduces the size of these arrays to be match the `_length` of each
data block. Given that we already have that length and each iteration over a
segment
will likely run many times for up to 10k docs, followed by some remaining
smaller
block, this should maintain the same properties of allocating arrays of 10k
size,
in the case of full table scans or similar workload where there are many
blocks
accessed.
However, it vastly reduces the memory churn on cases that fetch only a few
rows.
Instructions:
1. The PR has to be tagged with at least one of the following labels (*):
1. `feature`
2. `bugfix`
3. `performance`
4. `ui`
5. `backward-incompat`
6. `release-notes` (**)
2. Remove these instructions before publishing the PR.
(*) Other labels to consider:
- `testing`
- `dependencies`
- `docker`
- `kubernetes`
- `observability`
- `security`
- `code-style`
- `extension-point`
- `refactor`
- `cleanup`
(**) Use `release-notes` label for scenarios like:
- New configuration options
- Deprecation of configurations
- Signature changes to public methods/interfaces
- New plugins added or old plugins removed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
