comphead commented on issue #5274: URL: https://github.com/apache/datafusion-comet/issues/5274#issuecomment-5198571780
It seemed like an easy issue to fix, however it brought me down to a rabbit hole. Before addressing it we need to answer other questions: - Are we treating `spark.comet.enabled` parameter as static, can be set up once on runtime and it is not supposed to be changed in runtime? - Or like runtime modifiable parameter similar to `spark.sql.adaptive.enabled`? This allows enable Comet on a query level and it is current behavior. For runtime parameter we need to make sure Comet code is preloaded, even if the user opted out. Potentially this leads to confusing Comet related messages/exceptions when Comet is disabled. So its up to the user knowledge to disable Comet completely they expected to remove Comet from `spark.plugins`. Static parameter allows to load Comet code once on demand but its not flexible enough to use on query level. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
