jon-wei opened a new pull request #10689: URL: https://github.com/apache/druid/pull/10689
This PR introduces a new tuning config parameter, `maxColumnsToMerge`. This functions as a limit on how many segments can be merged at the same time by the IndexMerger. When the column limit is exceeded across a set of segments, the IndexMerger will break the segments to be merged into smaller phases, and merge the smaller phases in a tree. A minimum of 2 segments will be merged at once, regardless of the limit. If there is only 1 segment being merged, the limit does not apply. (A warning is logged in these cases, but merging is allowed to proceed). Currently only the native batch and parallel ingest task tuning config have this new parameter added, this PR does not add support for it to Kafka/Kinesis ingestion yet. This PR has: - [x] been self-reviewed. - [ ] added documentation for new or modified features or behaviors. - [ ] added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links. - [ ] added or updated version, license, or notice information in [licenses.yaml](https://github.com/apache/druid/blob/master/licenses.yaml) - [ ] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader. - [x] added unit tests or modified existing tests to cover new code paths, ensuring the threshold for [code coverage](https://github.com/apache/druid/blob/master/dev/code-review/code-coverage.md) is met. - [ ] added integration tests. - [ ] been tested in a test Druid cluster. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
