ariel-miculas commented on PR #23777: URL: https://github.com/apache/datafusion/pull/23777#issuecomment-5277544549
> Even if we can't eliminate the extra overhead, we can still add a config option to enable or disable this hash-set-based deduplication, this shouldn't be a hard blocker. Instead of a config option, could we use a flag that gets set by select upstream operators, such as hash aggregate? Because for hash aggregate we already know that sliced RecordBatches will be produced from a huge initial RecordBatch. The issue is that we need to pass this information to the downstream consumer of these batches, since there's where the memory accounting is broken (in main). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
