Kimahriman commented on issue #841: URL: https://github.com/apache/datafusion-comet/issues/841#issuecomment-2300007154
Ok I learned a little more about DataFusion so I think I understand what the options are now. ScalarUDFs support type coercion, which will automatically cast each expression to the right type, and that gets inserted in analysis. Because of this, there's nothing technically "wrong" or buggy in DataFusion. Possibly just sub-optimal of doing a cast instead of smartly handling mixed types. Obviously Comet isn't using the DataFusion analyzer, so that will never happen automatically here. And the type coercion theoretically handles the differences between dictionaries/non-dictionaries. Based on this, it seems like there's two options: - Add some form of support for handling ScalarUDF type coercion automatically - Don't use ScalarUDFs when type coercion may be needed and only use custom implemented expressions -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
