Re: [PR] [SPARK-41952][SQL] Fix Parquet zstd off-heap memory leak as a workaround for PARQUET-2160 [spark]

Tue, 10 Oct 2023 20:46:13 -0700 


yujhe commented on PR #40091:
URL: https://github.com/apache/spark/pull/40091#issuecomment-1756720258

   @pan3793 We found that in some cases this workaround did not work when 
reading Parquet files in zstd codec and outputting it to a new path.
   
   ```scala
   val df = spark.read.parquet("zstd_parquet_file")
   df.repartition(3).write.mode("overwrite").parquet("/tmp/test")
   ```
   
   ![Screenshot 2023-10-11 at 10 48 19 
AM](https://github.com/apache/spark/assets/6325926/577ecfe1-e9c0-41ea-ab63-bddb2142a3be)
   
   In debug mode, you can see it won't set ParquetCodecFactory when 
initializing `org.apache.parquet.hadoop.ParquetFileReader` which is called from 
[ParquetFileFormat.buildReaderWithPartitionValues](https://github.com/apache/spark/blob/branch-3.3/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala#L405C32-L405C32).
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to