brijrajk commented on PR #12333:
URL: https://github.com/apache/gluten/pull/12333#issuecomment-4785142670
Thanks @marin-ma and @FelixYBW for the feedback. Updated in the latest push:
- Removed the silent fallback to `zstd` — unsupported codecs (including
`none`) now throw `IllegalArgumentException`
- Refined the error message to match the suggested format:
```
Gluten shuffle only supports lz4, zstd. snappy is not supported. You may
configure spark.gluten.sql.columnar.shuffle.codec to lz4 or zstd.
```
- Updated the regression test in `MiscOperatorSuite` to assert the exception
and message instead of the old fallback behaviour
Regarding `spark.io.compression.codec=none` (@marin-ma): the current
implementation throws for `none` as well, pointing users to
`spark.gluten.sql.columnar.shuffle.codec`. Supporting uncompressed native
shuffle would be a separate follow-up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
