cshuo opened a new issue, #20505:
URL: https://github.com/apache/hudi/issues/20505
### Describe the problem
A Spark SQL INSERT into a copy-on-write Hudi table fails when an array
contains a NULL element with the default Parquet Avro list encoding. An INSERT
of a non-null array succeeds. This reproduces with the native Spark SQL CLI,
without Spark Connect or ADBC.
### To Reproduce
Start `spark-sql` with the corresponding Hudi bundle on the classpath and
these settings:
```bash
spark-sql --master 'local[2]' \
--conf spark.serializer=org.apache.spark.serializer.KryoSerializer \
--conf
spark.sql.extensions=org.apache.spark.sql.hudi.HoodieSparkSessionExtension \
--conf
spark.sql.catalog.spark_catalog=org.apache.spark.sql.hudi.catalog.HoodieCatalog
```
Using a fresh local warehouse and no `parquet.avro.write-old-list-structure`
override:
```sql
CREATE TABLE orders (id INT, arr ARRAY<INT>, ts BIGINT) USING hudi
TBLPROPERTIES (type='cow', primaryKey='id', preCombineField='ts');
-- Succeeds; SELECT returns [1, 2, 3].
INSERT INTO orders VALUES (1, array(1, 2, 3), 1);
SELECT id, arr FROM orders;
-- Fails.
INSERT INTO orders VALUES (2, array(1, CAST(NULL AS INT), 3), 2);
```
The failed INSERT reports `HoodieException: Insert Into to Hudi table
failed`. The underlying writer log says:
```text
java.lang.NullPointerException: Array contains a null element at 1. Set
parquet.avro.write-old-list-structure=false to turn on support for arrays with
null elements.
```
### Expected behavior
A nullable array element accepted by Spark SQL should be written and read
back correctly from a Hudi table.
### Environment
- Hudi 1.2.0, Spark 4.1.3 / Scala 2.13 / Java 17
- Hudi 1.2.0, Spark 3.5.7 / Scala 2.12 / Java 17
- Both tested in disposable Linux Docker containers with local filesystem
storage, local Spark master and a CoW table.
### Additional context
A separate experiment with
`spark.hadoop.parquet.avro.write-old-list-structure=false` set at driver
startup succeeded for a nullable array, but this is a driver-wide setting.
Compatibility of files written with different list encodings by other writers,
readers, compaction and clustering was not validated; this report does not
recommend it as a general workaround.
Related: #11785 reports the same Parquet Avro exception in a Hudi 0.15.0
CDC/DataFrame write; #2265 is an older, closed report involving array nulls and
Parquet files.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
