EmilyMatt commented on issue #1389: URL: https://github.com/apache/datafusion-comet/issues/1389#issuecomment-3647036600
> [@EmilyMatt](https://github.com/EmilyMatt), now that I have worked on this and become more familiar with the issue, could you confirm that I understand it correctly? > > If we have a Comet final and Spark partial agg, or a Spark final and Comet partial agg, we can experience runtime errors if the intermediate buffer representation isn't compatible. In those cases, we should either a) fall back to Spark for both, or b) add additional projections or take other measures to make it still work. Am I understanding this correctly? Indeed^ The better solution imo is to just separate the result expressions into a Spark ProjectExec, and have the conversion after the aggreagte(assuming that our aggregate is better than Spark ;) ) The conversion will also have less rows because of the group-by, which seems more optimal all-around. I can give this a go again if you'd like -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
