findepi edited a comment on issue #3621: URL: https://github.com/apache/iceberg/issues/3621#issuecomment-981646949
Iceberg 0.12 (`iceberg-spark3-runtime-0.12.jar`) bundles shaded version of Parquet 1.12. Since https://github.com/apache/parquet-mr/commit/279255df0c050aa95b5f5eb5963cf7eae5b8d180 (`apache-parquet-1.12.0-rc2` and newer), Parquet requires `com.github.luben.zstd.ZstdInputStream` to have 2-arg constructor (InputStream, BufferPool). This is available since https://github.com/luben/zstd-jni/commit/dd2588edd302823fa534de1516e4ae6d6dc6417e, i.e. zstd-jni `v1.4.5-8` and newer. - my spark runtime (https://archive.apache.org/dist/spark/spark-3.0.0/spark-3.0.0-bin-hadoop2.7.tgz) includes `zstd-jni-1.4.4-3.jar`, so too old for Parquet 1.12 to decode ZSTD-compressed parquet files. - iceberg seems to bundle zstd-jni too, not sure which version (might relate to https://github.com/apache/iceberg/pull/3058) i tried using newer spark versions, 3.1.2 and 3.2.0 (both seem to bundle sufficiently new `zstd-jni`), but was getting ``` java.lang.NoSuchMethodError: 'void org.apache.spark.sql.internal.VariableSubstitution.<init>(org.apache.spark.sql.internal.SQLConf) ``` (i am not sure whether i was testing this with Iceberg 0.12 or 0.11 though) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
