ArnavBalyan opened a new pull request, #8456:
URL: https://github.com/apache/incubator-gluten/pull/8456
- Currently Scan is offloaded to Velox, however Velox does not support
encrypted parquet files.
- This leads to a Velox side error and runtime failure. Added support to
fallback in such cases.
- We attempt to read parquet footer for the root paths in scan operator, if
decryption is detected, the scan fallsback
- This is behind a config and will not be enabled by default, users can
enable scan fallback by passing `spark.gluten.sql.fallbackEncryptedParquet`
true.
- Currently parquet 1.12 does not offer direct signal for checking
encryption, it's available in 1.13 which can be used in the future if spark
upgrades.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
