singhpk234 commented on code in PR #14733:
URL: https://github.com/apache/iceberg/pull/14733#discussion_r2596526304
##########
docs/docs/spark-configuration.md:
##########
@@ -197,6 +197,14 @@ val spark = SparkSession.builder()
| spark.sql.iceberg.merge-schema | false
| Enables modifying the table schema
to match the write schema. Only adds columns missing columns
|
| spark.sql.iceberg.report-column-stats | true
| Report Puffin Table Statistics if
available to Spark's Cost Based Optimizer. CBO must be enabled for this to be
effective |
+### Classpath order and Parquet version compatibility
+
+Spark distributions may bundle a different Parquet version than Iceberg’s
Spark runtime. For example, Spark 4.0.x uses Parquet 1.15.2 while Iceberg
1.10.x can use Parquet 1.16.0 for features like Variant/Geometry. With the
default classpath order, Spark’s older Parquet may be picked up first and can
cause NoSuchMethodError when Iceberg code invokes newer Parquet APIs.
Review Comment:
is 1.10.1 release notes a better place to document this ?
orthogonally can we not shade it and track it as a fix for 1.10.x, to avoid
user configuring these config as the same can be true for other engine ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
