zzzzming95 commented on PR #37416: URL: https://github.com/apache/spark/pull/37416#issuecomment-1207677419
@dongjoon-hyun Detailed discussion can be viewed in jira : https://issues.apache.org/jira/browse/SPARK-39743 The user wants to write a parquet file and set some compression configurations such as zstd level. In the documentation, user found that there is a `spark.io.compression.zstd.level` in the relevant configuration, which seems to be the desired configuration, but after using it, it is found that it does not take effect, which will make users confused. ``` sparkSession.sessionState.newHadoopConfWithOptions( "spark.io.compression.zstd.level" -> "10" ) df.coalesce(1).write.parquet("file:///home/test_data/nn_parq_10") ``` After adding the description, the user will look for the description of the `spark.io.compression.codec` configuration, and then can make it clear that the `spark.io.compression.zstd.level` related configuration is only valid for internal data. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
