rangareddy commented on issue #16946:
URL: https://github.com/apache/hudi/issues/16946#issuecomment-5410515896
This issue was reviewed as part of the JIRA-migrated backlog triage
(HUDI-9265).
**Findings: still open, and the exposure is wider than the ticket records.
`parquet-avro` is genuinely shaded into the bundles, and most build profiles
still resolve a version below the 1.15.1 named here.**
**1. The bundles do ship `parquet-avro`.** The root `pom.xml` declares it
`<scope>provided</scope>` (`:993-995`), which on its own would mean the runtime
supplies it - but the bundle poms override that and shade it in explicitly:
| bundle | shade include | dependency scope |
| --- | --- | --- |
| `packaging/hudi-spark-bundle/pom.xml` | `:100`
`<include>org.apache.parquet:parquet-avro</include>` | `:254`
`<scope>compile</scope>` |
| `packaging/hudi-utilities-bundle/pom.xml` | `:118` same | `:299` |
| `packaging/hudi-flink-bundle/pom.xml` | `:101` same (plus
`parquet-hadoop`, `-column`, `-common`, `-format-structures`, `-encoding`,
`-jackson`) | - |
So the premise in the title - "being pulled into our bundles" - is correct.
Note shading relocates the classes but does not remove them, so a relocated
copy is still a copy.
**2. Which profiles resolve a patched version.** Against the 1.15.1 this
ticket names as the fix:
| profile | `parquet.version` | ≥ 1.15.1 |
| --- | --- | --- |
| **(default)** | 1.13.1 | no |
| `spark3` | 1.13.1 | no |
| `spark3.3` | 1.12.2 | no |
| `spark3.4` | 1.12.3 | no |
| **`spark3.5`** (the default Spark line) | 1.13.1 | no |
| `spark4.0` | 1.15.2 | yes |
| `spark4.1` | 1.16.0 | yes |
| `spark4.2` | 1.17.0 | yes |
Since `spark3.version` defaults to `spark35.version`, a plain `mvn install`
produces Spark bundles on parquet **1.13.1**.
**3. Flink is partially patched** - worth adding to the ticket, since it is
not mentioned. `hudi-flink-bundle` uses `${flink.format.parquet.version}`:
| profile | version | ≥ 1.15.1 |
| --- | --- | --- |
| (default) / `flink2.1` | 1.15.2 | yes |
| `flink2.0` | 1.14.4 | no |
| `flink1.20` / `flink1.19` / `flink1.18` | 1.13.1 | no |
**4. Nothing has landed.** The linked PR #13102 is titled `[HUDI-9265][DNM]
Bumping parquet version to 1.15.1` - explicitly Do-Not-Merge - and is **closed
unmerged**. I am removing the `status:pr-available` label accordingly, since it
implies a patch is awaiting review when there is none.
**Scope note for whoever picks this up.** This is probably not a one-line
property bump, which may be why the DNM PR stalled: `spark3.3` and `spark3.4`
are pinned to parquet 1.12.x, and moving them to 1.15.x is a larger
compatibility step than moving 1.13.1 forward. It may be worth splitting - bump
the 1.13.1 lines (default, `spark3`, `spark3.5`, `flink1.18`-`1.20`) separately
from the 1.12.x lines (`spark3.3`, `spark3.4`), so the common case can be fixed
without waiting on the harder ones.
Also worth a separate cleanup: the checked-in `dependencies/*.txt` manifests
still list `parquet-avro-1.10.1` / `1.11.1` for Scala 2.11 and Spark 2 bundles
that no longer exist. Those are stale rather than current exposure, but they
are misleading to anyone auditing shipped versions.
Keeping this open - it is a `priority:blocker` and the default build is
still on 1.13.1.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
