umehrot2 commented on issue #2265: URL: https://github.com/apache/hudi/issues/2265#issuecomment-745043470
So, on the surface it seems like `parquet-avro` support which Hudi uses to write the data in parquet format no longer supports the scenario for Lists containing nulls: https://github.com/apache/parquet-mr/blob/0a4e3eea991f7588c9c5e056e9d7b32a76eed5da/parquet-avro/src/main/java/org/apache/parquet/avro/AvroWriteSupport.java#L583 Spark uses its own native writers to write parquet which are able to handle this scenario. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
