rangareddy commented on issue #12902:
URL: https://github.com/apache/hudi/issues/12902#issuecomment-5164805832
@izu-x apologies for the long silence, and thank you for pushing back — you
were right to. **My earlier answer pointed you at the wrong thing.** The AWS
SDK version is not what is failing here, so migrating to SDK 2.18.40 was never
going to fix it. Here is what your stack trace actually says.
### Your error is a Hudi version mismatch, not an SDK one
```
java.lang.NoSuchMethodException:
org.apache.hudi.metrics.cloudwatch.CloudWatchMetricsReporter.<init>(
org.apache.hudi.config.metrics.HoodieMetricsConfig,
com.codahale.metrics.MetricRegistry)
at java.lang.Class.getConstructor0(Class.java:3110)
at
org.apache.hudi.common.util.ReflectionUtils.loadClass(ReflectionUtils.java:73)
```
This is `Class.getConstructor` failing on a **Hudi** class. It never touches
the AWS SDK — no SDK class is loaded at that point, so no SDK version can
affect it. If the SDK were the problem you would see a `NoClassDefFoundError`
on a `software.amazon.awssdk.*` class, or an `InvocationTargetException`, and
it would come from `newInstance` rather than `getConstructor`.
What it means is that the `CloudWatchMetricsReporter` class on your
classpath is **older than the code calling it**. That constructor was added in
0.15.0:
| release | constructors on `CloudWatchMetricsReporter` |
| --- | --- |
| 0.14.1 and earlier | `(HoodieWriteConfig, MetricRegistry)` only |
| 0.15.0+ | `(HoodieWriteConfig, MetricRegistry)` **and**
`(HoodieMetricsConfig, MetricRegistry)` |
0.15.0's `MetricsReporterFactory` asks for `(HoodieMetricsConfig,
MetricRegistry)`. A genuine 0.15.0 jar has it. So something on your test
classpath is supplying a pre-0.15.0 copy of that class.
One detail makes this especially likely in a unit test, and explains why the
same config can work on a cluster. In 0.15.0 `HoodieMetricsConfig` **moved
module**, from `hudi-client-common` to `hudi-common`, while keeping the package
`org.apache.hudi.config.metrics`. So a stale 0.14.x `hudi-client-common` on the
classpath carries both an old `CloudWatchMetricsReporter` *and* a second
`org.apache.hudi.config.metrics.HoodieMetricsConfig` in the same package as
0.15.0's. Nothing is shaded or relocated in a plain Gradle test, and jar
precedence is arbitrary, so whichever copy wins first decides whether the
signature matches.
To confirm on your side:
```bash
./gradlew dependencies --configuration testRuntimeClasspath | grep -i hudi
```
Look for more than one Hudi version, and for `hudi-client-common` or a
`hudi-spark*-bundle` arriving transitively at a version other than 0.15.0 (an
EMR-provided Hudi jar on the test classpath does this too). To see which jar
actually wins:
```bash
for j in $(find ~/.gradle -name 'hudi-*.jar'); do
unzip -l "$j" | grep -q
'metrics/cloudwatch/CloudWatchMetricsReporter.class' && echo "$j"
done
```
More than one hit is the bug. Pin every Hudi artifact to the same version.
### On AWS SDK v1 vs v2 — you do not have to migrate your project
This is the part I should have been clearer about, because it is the actual
worry in your comment.
`hudi-aws` is written against AWS SDK **v2**: `CloudWatchReporter` imports
`software.amazon.awssdk.services.cloudwatch.*`, and `hudi-aws/pom.xml` declares
`software.amazon.awssdk:cloudwatch`, `:auth`, `:regions` and friends. The
`1.12.772` you have is SDK **v1**, whose coordinates are
`com.amazonaws:aws-java-sdk-*` and whose packages are `com.amazonaws.*`. So v1
cannot satisfy `hudi-aws`, whatever the version number.
But v1 and v2 are **different artifacts in different packages, and they
coexist happily on one classpath**. That is exactly why the EMR page you linked
lists both 2.28.8 and 1.12.772. You can keep `1.12.772` for your own code and
add the v2 artifacts alongside it purely for `hudi-aws` — no migration of your
project, and no incompatibility, because nothing of yours has to import
`software.amazon.*`.
Note this is a *separate* requirement from the error above: you would meet
it at instantiation time, only after the version skew is sorted out. Also note
`hudi-spark-bundle` does not shade `hudi-aws`, so you need `hudi-aws-bundle` at
the matching Hudi version on the test classpath as well.
### The message is being fixed
That "Unable to instantiate class" told you nothing, which is why this took
a stack-trace reading to resolve:
- #19418 (merged) covers the case where `hudi-aws` is absent entirely — the
error now names the class, the bundle to add, and the config to change.
- #19477 (open) covers exactly your case — a constructor that does not match
is now reported as a probable version mismatch, with "use a `hudi-aws-bundle`
whose version matches the Hudi bundle in use" instead of the bare message you
got.
If you try the two checks above and still see the same failure, please paste
the `./gradlew dependencies` output for `testRuntimeClasspath` and I will take
another look.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
