yihua commented on code in PR #19674: URL: https://github.com/apache/hudi/pull/19674#discussion_r4343870662
########## README.md: ########## @@ -145,6 +145,25 @@ off-heap memory when reading ZSTD files ([PARQUET-2160](https://issues.apache.or Upgrade to Spark 3.5 or newer before using ZSTD. Keeping GZIP as the write default does not remove the risk when an older Spark runtime reads ZSTD files produced by another engine. +### Spark native bundle + +`hudi-spark-native-bundle` is `hudi-spark-bundle` plus [Apache DataFusion Comet](https://datafusion.apache.org/comet/) +for native vectorized execution. It is built alongside the Spark bundle for the Spark versions Comet +publishes artifacts for, and produces nothing for the others. + +| Maven build options | Expected native bundle jar name | Notes | +|:--------------------------|:--------------------------------|:-----------------------------------------------| +| `-Dspark3.4` | hudi-spark3.4-native-bundle_2.12 | For Spark 3.4.x and Scala 2.12 | Review Comment: Instead of adding a new table and section, could you add hudi-spark-native-bundle to the same table above and explain the native bundle after the table? ########## packaging/bundle-validation/README.md: ########## @@ -22,6 +22,14 @@ This directory contains scripts for running bundle validation in GitHub Actions (`validate-bundles` specified in `.github/workflows/bot.yml`) and build profile for Docker images used. +## Spark native bundle Review Comment: Add this at the end of this doc ########## .github/workflows/bot.yml: ########## @@ -1335,6 +1335,43 @@ jobs: HUDI_VERSION=$(mvn help:evaluate -Dexpression=project.version -q -DforceStdout) ./packaging/bundle-validation/ci_run.sh hudi_docker_java17 $HUDI_VERSION openjdk17 + # Every published native bundle coordinate is shaded here. Only the Spark 3.5 lane above runs + # it, because Comet's libcomet.so needs a glibc image; this catches shade and dependency breakage + # on the other coordinates rather than discovering it on release day. + build-spark-native-bundle: Review Comment: Instead of adding a new `build-spark-native-bundle`, could you re-enable the matrix for Spark 4.0 and 4.1 in validate-bundles so the native bundle on these two Spark versions is validated along with other dimensions? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
