voonhous commented on code in PR #19235:
URL: https://github.com/apache/hudi/pull/19235#discussion_r3552902498
##########
packaging/hudi-spark-bundle/pom.xml:
##########
@@ -98,6 +98,9 @@
<include>com.github.davidmoten:hilbert-curve</include>
<include>com.github.ben-manes.caffeine:caffeine</include>
<include>org.apache.parquet:parquet-avro</include>
+ <!-- Parquet 1.16.0+ (Spark 4.1+) splits VariantConverters
into parquet-variant,
+ which parquet-avro needs at runtime. No-op for older
Spark. See apache/hudi#19234. -->
+ <include>org.apache.parquet:parquet-variant</include>
Review Comment:
Good catch. Moved the include to the `<!-- common to all bundles -->`
artifactSet in the root pom instead of just the `spark-bundle`, so every bundle
picks it up via `combine.children="append"`.
Verified the rebuilt Spark 4.1 hoodie-spark-bundle and
hoodie-utilities-bundle both now contains `VariantConverters;` no-op for the
pre-1.16 bundles.
##########
packaging/hudi-spark-bundle/pom.xml:
##########
@@ -98,6 +98,9 @@
<include>com.github.davidmoten:hilbert-curve</include>
<include>com.github.ben-manes.caffeine:caffeine</include>
<include>org.apache.parquet:parquet-avro</include>
+ <!-- Parquet 1.16.0+ (Spark 4.1+) splits VariantConverters
into parquet-variant,
+ which parquet-avro needs at runtime. No-op for older
Spark. See apache/hudi#19234. -->
+ <include>org.apache.parquet:parquet-variant</include>
Review Comment:
Good catch. Moved the include to the `<!-- common to all bundles -->`
artifactSet in the root pom instead of just the `spark-bundle`, so every bundle
picks it up via `combine.children="append"`.
Verified the rebuilt Spark 4.1 `hoodie-spark-bundle` and
`hoodie-utilities-bundle` both now contains `VariantConverters;` no-op for the
pre-1.16 bundles.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
