dongjoon-hyun commented on pull request #32826: URL: https://github.com/apache/spark/pull/32826#issuecomment-1030711209
@mixermt . It's enabled by default since Apache Spark 3.2.0. - https://github.com/apache/spark/blob/74ebef243c18e7a8f32bf90ea75ab6afed9e3132/core/src/main/scala/org/apache/spark/internal/config/package.scala#L1778-L1783 Although ZSTD-JNI implementation is improved a lot recently, you will hit high memory usage if you turn off the configuration. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
