LuciferYang opened a new issue, #12987:
URL: https://github.com/apache/gluten/issues/12987
### Backend
VL (Velox)
### Bug description
With `spark.sql.hive.convertMetastoreParquet=false`, the native writer
ignores the `parquet.compression` table property and uses
`spark.sql.parquet.compression.codec` instead.
Reproduce with a Hive table whose table property asks for zstd while the
session conf asks for gzip:
```sql
SET spark.sql.hive.convertMetastoreParquet=false;
SET spark.sql.parquet.compression.codec=gzip;
CREATE TABLE t STORED AS PARQUET TBLPROPERTIES
('parquet.compression'='zstd') AS SELECT 1 as c;
```
The Parquet footer reports GZIP. Vanilla Spark honours the table property
here, so the written file differs depending on whether the native writer is on.
### How this surfaced
`VeloxParquetWriteForHiveSuite."native writer should respect table
properties"` already compared the codec against zstd, but the comparison was
the last expression of a `Utils.tryWithResource` block and its value was
discarded, so the case passed regardless. #12981 wrapped it in `assert`, and it
fails on 3.4, 3.5, 4.0 and 4.1:
```
- native writer should respect table properties *** FAILED ***
"zstd".equalsIgnoreCase(column.getCodec().toString()) was false
expected zstd but got GZIP in
file:.../VeloxParquetWriteForHiveSuite/t/part-00000-...,
nativeWriteEnabled=true (VeloxParquetWriteForHiveSuite.scala:294)
```
The case is `ignore`d in #12981 until this is fixed, with a comment pointing
here.
One thing the test cannot tell us yet: the case loops over `Seq(true,
false)` for `enableNativeWrite`, and the assertion aborts on the first
iteration, so the vanilla writer path was never reached. Whether vanilla also
writes GZIP under these settings is unverified.
### Spark version
None
### Spark configurations
`spark.sql.hive.convertMetastoreParquet=false`,
`spark.sql.parquet.compression.codec=gzip`,
`spark.gluten.sql.native.writer.enabled=true`
### System information
N/A
### Relevant logs
```
- native writer should respect table properties *** FAILED ***
"zstd".equalsIgnoreCase(column.getCodec().toString()) was false
expected zstd but got GZIP in
file:/__w/gluten/gluten/backends-velox/spark-warehouse/org.apache.spark.sql.execution.VeloxParquetWriteForHiveSuite/t/part-00000-8619fbf4-c5f2-419f-8a61-1c9a19a9b3ca,
nativeWriteEnabled=true (VeloxParquetWriteForHiveSuite.scala:294)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
