ccaominh commented on issue #9168: "assumeGrouped" behaves differently in native batch and hadoop tasks URL: https://github.com/apache/druid/issues/9168#issuecomment-576457905 I believe that for hadoop ingestion, regardless of the value of `assumeGrouped`, `DeterminePartitionsJob.DeterminePartitionsDimSelectionReducer` always runs and examines the dimension value distribution to determine the partitions. When `assumeGrouped` is false, there is an earlier stage the data to group rows (`DeterminePartitionsGroupByMapper`/`DeterminePartitionsGroupByReducer`). For native batch ingestion range partitioning, instead of having an earlier stage to group rows, the grouping occurs in the same stage that determines the partitions (i.e., both grouping and partitioning are done with a single pass over the data instead of the two used for hadoop ingestion). Having `assumeGrouped` as `true` still benefits native batch ingestion range partitioning since it avoids the time/space overhead of using a bloom filter to group the rows. In short, the behavior of `assumeGrouped` is not identical between range partitioning for hadoop and native batch ingestion, but it does have a similar effect in improving ingestion performance when set to `true`.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
