karthi07 opened a new pull request, #12681:
URL: https://github.com/apache/pinot/pull/12681
Adding new config "realtime.segment.flush.threshold.segment.rows" for
real-time table to configure the segment threshold independent of max partitions
**Assumptions:**
- If flush size > 0, create a new DefaultFlushThresholdUpdater with given
flush size.
- If flush size <= 0, create new SegmentRowsBasedFlushThresholdUpdater if
flushThresholdSegmentRows > 0.
- If flush size <= 0 AND segment.row <=0, create new
SegmentSizeBasedFlushThresholdUpdater if not already created.
**Default values**
- when value is not provided -> value is considered as 0
- when invalid config is provided -> 5000000 will be considered.
**Issue[12508]:**
https://github.com/apache/pinot/issues/12508
When realtime.segment.flush.threshold.rows is used, the flush threshold for
the new CONSUMING segment is determined by both this value and the max
partitions consumed by any server. This is not very straight forward, and
rebalancing a table could cause new consuming segment size to change.
In order to tackle this problem, we may add a new config (proposing
realtime.segment.flush.threshold.segment.rows) which works independent of
partitions consumed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
