wombatu-kun commented on code in PR #19235:
URL: https://github.com/apache/hudi/pull/19235#discussion_r3552255358
##########
packaging/hudi-spark-bundle/pom.xml:
##########
@@ -98,6 +98,9 @@
<include>com.github.davidmoten:hilbert-curve</include>
<include>com.github.ben-manes.caffeine:caffeine</include>
<include>org.apache.parquet:parquet-avro</include>
+ <!-- Parquet 1.16.0+ (Spark 4.1+) splits VariantConverters
into parquet-variant,
+ which parquet-avro needs at runtime. No-op for older
Spark. See apache/hudi#19234. -->
+ <include>org.apache.parquet:parquet-variant</include>
Review Comment:
`hudi-utilities-bundle` has the same gap: it shades `parquet-avro` at
compile scope without `parquet-variant`, does not relocate parquet, and unlike
the slim bundle it is a standalone spark-submit runtime that
`validate-bundle-spark4` builds under `-Dspark4.1`. A HoodieStreamer `--op
UPSERT` on Spark 4.1 will hit the same `NoClassDefFoundError`, and
`validate.sh` will not catch it because it runs the standalone utilities-bundle
check only on `spark-3.5`.
Worth closing here rather than in a follow-up: put the include once into the
`<!-- common to all bundles -->` `artifactSet` in the root `pom.xml` that every
bundle appends to via `combine.children="append"`. That also covers
`hudi-integ-test-bundle`, `hudi-hive-sync-bundle`, `hudi-gcp-bundle` and
`hudi-azure-bundle`, and stays a no-op for `hudi-hadoop-mr-bundle`,
`hudi-presto-bundle`, `hudi-trino-bundle` and `hudi-flink-bundle`, which pin
pre-1.16 parquet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
