jingpan xiong created SPARK-36773:
-------------------------------------
Summary: Fix the uts to check the parquet codec
Key: SPARK-36773
URL: https://issues.apache.org/jira/browse/SPARK-36773
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 3.3
Reporter: jingpan xiong
Fix For: 3.3
In my own test env. I import a wrong jar version about zstd-jni-1.4.4-3.jar
with the parquet version 1.12.0, when I pass the full uts and release a spark
version to my env , using the parquet table with compression zstd, it throw a
"NoClassDefFoundError" Exception. And I check the parquet source code, find the
right zstd version for the 1.12.0 is 1.4.9-1. Once I upgrade the version about
zstd-jni, it work well.
!https://clouddevops.huawei.com/vision-file-storage/api/file/download/upload-v2/2021/8/14/xwx915262/ed1735e06a684bf19cf1d1a68a130b6a/image.png!
But I think we need check for all plugin compression algorithm can work well
when we upgrade the version about those compression jars and parquet version.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
