PingLiuPing opened a new issue, #11023: URL: https://github.com/apache/incubator-gluten/issues/11023
### Description Currently, Iceberg partition transform is performed in the execution layer (within TableWrite operator). For better architecture and optimization opportunities, this computation should be moved to the optimizer/planner layer where it logically belongs. And this can be achieved by adding a projection node above TableWriter node and letting this node to perform the partition transform and send the pre-computed partition keys to TableWriter. Doing so would allow for this computation to be optimized further by, for example, pushing it below the exchange and letting it run with higher concurrency than write itself. It would also provide more transparency for debugging as we would get runtime stats just for this piece of computation. It would allow users to see more clearly in the query plan what is happening allowing for more efficient debugging. ### Gluten version None -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
