justinborromeo commented on issue #7238: [PROPOSAL] Add segment limit for native and streaming index tasks URL: https://github.com/apache/incubator-druid/issues/7238#issuecomment-481041614 I think `maxRowsPerSegment`, `maxTotalRows`, and `maxTotalSegments` are all necessary since they protect against different cases. - `maxRowsPerSegment` exists so that a specific segment doesn't have too much data - `maxTotalRows` exists for the case where there is a moderately high number of segments each with a large amount of data - `maxTotalSegments` exists for the cases when there is an extremely high number of segments with a small amount of data As an example, if we use the `maxTotalRows = maxTotalSegments * maxRowsPerSegment` calculation with the existing 5M `maxRowsPerSegment` default and the proposed 1K `maxTotalSegments` default, our calculated `maxTotalRows` is 5B which seems a bit high.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
