rekbun commented on PR #59522: URL: https://github.com/apache/spark/pull/59522#issuecomment-6473104114
Thanks, @peter-toth and @ulysses-you. Our inputs don’t implement HasPartitionStatistics today. The connector already has file lengths, so we can expose them. For this PR, I’d take the Spark-side sizing approach using that existing interface. It would cover unkeyed V2 scans too. I’d start with the cases Peter listed where the plan doesn’t use the scan’s partitioning or ordering, and pack inside the scan so partial aggregation benefits as well. I’ll keep it default-off and add the 4.2 migration note. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
