jingpan xiong created SPARK-36773: ------------------------------------- Summary: Fix the uts to check the parquet codec Key: SPARK-36773 URL: https://issues.apache.org/jira/browse/SPARK-36773 Project: Spark Issue Type: Bug Components: SQL Affects Versions: 3.3 Reporter: jingpan xiong Fix For: 3.3
In my own test env. I import a wrong jar version about zstd-jni-1.4.4-3.jar with the parquet version 1.12.0, when I pass the full uts and release a spark version to my env , using the parquet table with compression zstd, it throw a "NoClassDefFoundError" Exception. And I check the parquet source code, find the right zstd version for the 1.12.0 is 1.4.9-1. Once I upgrade the version about zstd-jni, it work well. !https://clouddevops.huawei.com/vision-file-storage/api/file/download/upload-v2/2021/8/14/xwx915262/ed1735e06a684bf19cf1d1a68a130b6a/image.png! But I think we need check for all plugin compression algorithm can work well when we upgrade the version about those compression jars and parquet version. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org