PHILO-HE commented on issue #10209: URL: https://github.com/apache/incubator-gluten/issues/10209#issuecomment-3209493085
We have set the following environment variable in CI to make Spark ANSI default to false. So in CI tests on Spark-4.0, ANSI is disabled by default unless it's explicitly enabled by test. https://github.com/apache/incubator-gluten/blob/677f3cab93e8e0cb6bb5f9fdcb978df99ea8b007/.github/workflows/velox_backend_x86.yml#L48 Setting the above environment variable in scala is not workable, because Spark read OS environment while scala `System.setProperty` only affects JVM process level environment. So to avoid ANSI-caused fallback on user side, maybe we have to temporarily set ANSI to false in `GlutenPlugin` init, regardless of user's explicit ANSI config, aligned with the proposal here. It needs to be well documented to eliminate confusion. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
