lkemmerer commented on issue #433:
URL:
https://github.com/apache/incubator-xtable/issues/433#issuecomment-2123659805
Hi! My team and I have also run into the issue described here while
attempting to convert Hudi data stored in S3.
Using Java 11 and the SHA `d991e75339f2c564897828bf6d647fcccd986cc5`, and
using a config file similar to the original commenter (we're converting from
Hudi to Delta, but the dataset configuration uses `tableBasePath`, `tableName`,
and `partitionSpec` and we are also using S3 for our data), we get the
following:
```
➜ incubator-xtable git:(main) java -jar
utilities/target/utilities-0.1.0-SNAPSHOT-bundled.jar --datasetConfig
../config.yaml
WARNING: Runtime environment or build system does not support multi-release
JARs. This will impact location-based features.
2024-05-21 17:22:05 INFO org.apache.xtable.utilities.RunSync:147 - Running
sync for basePath s3://s3-bucket-XXX/x_table_prefix for following table formats
[DELTA]
2024-05-21 17:22:05 INFO
org.apache.hudi.common.table.HoodieTableMetaClient:133 - Loading
HoodieTableMetaClient from s3://s3-bucket-XXX/x_table_prefix
2024-05-21 17:22:05 WARN org.apache.hadoop.util.NativeCodeLoader:60 -
Unable to load native-hadoop library for your platform... using builtin-java
classes where applicable
2024-05-21 17:22:05 WARN org.apache.hadoop.metrics2.impl.MetricsConfig:136
- Cannot locate configuration: tried
hadoop-metrics2-s3a-file-system.properties,hadoop-metrics2.properties
2024-05-21 17:22:06 WARN org.apache.hadoop.fs.s3a.SDKV2Upgrade:39 -
Directly referencing AWS SDK V1 credential provider
com.amazonaws.auth.DefaultAWSCredentialsProviderChain. AWS SDK V1 credential
providers will be removed once S3A is upgraded to SDK V2
2024-05-21 17:22:07 INFO org.apache.hudi.common.table.HoodieTableConfig:276
- Loading table properties from
s3://s3-bucket-XXX/x_table_prefix/.hoodie/hoodie.properties
Exception in thread "main" java.lang.NoSuchMethodError: 'java.lang.Object
org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.invokeTrackingDuration(org.apache.hadoop.fs.statistics.DurationTracker,
org.apache.hadoop.util.functional.CallableRaisingIOE)'
at
org.apache.hadoop.fs.s3a.Invoker.onceTrackingDuration(Invoker.java:147)
at
org.apache.hadoop.fs.s3a.S3AInputStream.reopen(S3AInputStream.java:282)
at
org.apache.hadoop.fs.s3a.S3AInputStream.lambda$lazySeek$1(S3AInputStream.java:435)
at
org.apache.hadoop.fs.s3a.Invoker.lambda$maybeRetry$3(Invoker.java:284)
at org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:122)
at
org.apache.hadoop.fs.s3a.Invoker.lambda$maybeRetry$5(Invoker.java:408)
at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:468)
at org.apache.hadoop.fs.s3a.Invoker.maybeRetry(Invoker.java:404)
at org.apache.hadoop.fs.s3a.Invoker.maybeRetry(Invoker.java:282)
at org.apache.hadoop.fs.s3a.Invoker.maybeRetry(Invoker.java:326)
at
org.apache.hadoop.fs.s3a.S3AInputStream.lazySeek(S3AInputStream.java:427)
at org.apache.hadoop.fs.s3a.S3AInputStream.read(S3AInputStream.java:545)
at java.base/java.io.DataInputStream.read(DataInputStream.java:149)
at java.base/java.io.DataInputStream.read(DataInputStream.java:100)
at
java.base/java.util.Properties$LineReader.readLine(Properties.java:502)
at java.base/java.util.Properties.load0(Properties.java:418)
at java.base/java.util.Properties.load(Properties.java:407)
at
org.apache.hudi.common.table.HoodieTableConfig.fetchConfigs(HoodieTableConfig.java:352)
at
org.apache.hudi.common.table.HoodieTableConfig.<init>(HoodieTableConfig.java:278)
at
org.apache.hudi.common.table.HoodieTableMetaClient.<init>(HoodieTableMetaClient.java:141)
at
org.apache.hudi.common.table.HoodieTableMetaClient.newMetaClient(HoodieTableMetaClient.java:692)
at
org.apache.hudi.common.table.HoodieTableMetaClient.access$000(HoodieTableMetaClient.java:85)
at
org.apache.hudi.common.table.HoodieTableMetaClient$Builder.build(HoodieTableMetaClient.java:774)
at
org.apache.xtable.hudi.HudiSourceClientProvider.getSourceClientInstance(HudiSourceClientProvider.java:42)
at
org.apache.xtable.hudi.HudiSourceClientProvider.getSourceClientInstance(HudiSourceClientProvider.java:31)
at org.apache.xtable.client.OneTableClient.sync(OneTableClient.java:90)
at org.apache.xtable.utilities.RunSync.main(RunSync.java:168)
```
Looking at the output for `mvn install` and at the maven dependency graph,
it appears that Hudi may have a dependency that requires Hadoop 2.10. I've
attempted to cut down both outputs to show pertinent info, but I'll also
include the full output as file attachments.
**`mvn install` shaded output**
[install.txt](https://github.com/apache/incubator-xtable/files/15396690/install.txt)
```shell
[INFO] --- shade:3.5.1:shade (default) @ xtable-hudi-support-extensions ---
[INFO] Including
org.apache.xtable:xtable-hudi-support-utils:jar:0.1.0-SNAPSHOT in the shaded
jar.
[INFO] Including org.apache.hudi:hudi-common:jar:0.14.0 in the shaded jar.
[INFO] Including org.apache.hadoop:hadoop-distcp:jar:2.10.0 in the shaded
jar.
[INFO] Including org.apache.hadoop:hadoop-mapreduce-client-core:jar:2.10.0
in the shaded jar.
[INFO] Including org.apache.hadoop:hadoop-yarn-client:jar:2.10.0 in the
shaded jar.
[INFO] Including org.apache.hadoop:hadoop-yarn-api:jar:2.10.0 in the shaded
jar.
[INFO] Including org.apache.hadoop:hadoop-yarn-common:jar:2.10.0 in the
shaded jar.
[INFO] Including org.apache.hadoop:hadoop-hdfs:jar:2.10.0 in the shaded jar.
[INFO] Including org.apache.hadoop:hadoop-hdfs-client:jar:2.10.0 in the
shaded jar.
[INFO] Including org.apache.hadoop:hadoop-annotations:jar:3.3.6 in the
shaded jar.
[INFO] Including org.apache.hadoop:hadoop-auth:jar:3.3.6 in the shaded jar.
[INFO] --- shade:3.5.1:shade (default) @ xtable-utilities ---
[INFO] Including org.apache.hadoop:hadoop-distcp:jar:2.10.0 in the shaded
jar.
[INFO] Including org.apache.hadoop:hadoop-mapreduce-client-core:jar:2.10.0
in the shaded jar.
[INFO] Including org.apache.hadoop:hadoop-yarn-client:jar:2.10.0 in the
shaded jar.
[INFO] Including org.apache.hadoop:hadoop-yarn-common:jar:2.10.0 in the
shaded jar.
[INFO] Including org.apache.hadoop:hadoop-hdfs:jar:2.10.0 in the shaded jar.
[INFO] Including org.apache.hadoop:hadoop-hdfs-client:jar:2.10.0 in the
shaded jar.
[INFO] Including org.apache.hadoop:hadoop-client-api:jar:3.3.4 in the shaded
jar.
[INFO] Including org.apache.hadoop:hadoop-client-runtime:jar:3.3.4 in the
shaded jar.
[INFO] Including org.apache.hadoop:hadoop-common:jar:3.3.6 in the shaded jar.
[INFO] Including
org.apache.hadoop.thirdparty:hadoop-shaded-protobuf_3_7:jar:1.1.1 in the shaded
jar.
[INFO] Including org.apache.hadoop:hadoop-annotations:jar:3.3.6 in the
shaded jar.
[INFO] Including org.apache.hadoop.thirdparty:hadoop-shaded-guava:jar:1.1.1
in the shaded jar.
[INFO] Including org.apache.hadoop:hadoop-auth:jar:3.3.6 in the shaded jar.
[INFO] Including
org.apache.hadoop:hadoop-yarn-server-resourcemanager:jar:3.1.0 in the shaded
jar.
[INFO] Including org.apache.hadoop:hadoop-yarn-api:jar:3.1.0 in the shaded
jar.
[INFO] Including org.apache.hadoop:hadoop-yarn-server-common:jar:3.1.0 in
the shaded jar.
[INFO] Including org.apache.hadoop:hadoop-yarn-registry:jar:3.1.0 in the
shaded jar.
[INFO] Including
org.apache.hadoop:hadoop-yarn-server-applicationhistoryservice:jar:3.1.0 in the
shaded jar.
[INFO] Including org.apache.hadoop:hadoop-yarn-server-web-proxy:jar:3.1.0 in
the shaded jar.
[INFO] Including org.apache.hadoop:hadoop-aws:jar:3.3.6 in the shaded jar.
[INFO] Including org.apache.hadoop:hadoop-azure:jar:3.3.6 in the shaded jar.
```
**dependencies**
[dependencies.txt](https://github.com/apache/incubator-xtable/files/15396689/dependencies.txt)
```shell
[INFO] Scanning for projects...
[INFO]
------------------------------------------------------------------------
[INFO] Reactor Build Order:
[INFO]
[INFO] xtable
[pom]
[INFO] xtable-api
[jar]
[INFO] xtable-hudi-support
[pom]
[INFO] xtable-hudi-support-utils
[jar]
[INFO] xtable-core
[jar]
[INFO] xtable-utilities
[jar]
[INFO] xtable-hudi-support-extensions
[jar]
[INFO]
[INFO]
[INFO] --------------------< org.apache.xtable:xtable-api
>--------------------
[INFO] Building xtable-api 0.1.0-SNAPSHOT
[2/7]
[INFO] from xtable-api/pom.xml
[INFO] --------------------------------[ jar
]---------------------------------
[INFO]
[INFO] --- dependency:3.6.1:tree (default-cli) @ xtable-api ---
[INFO] org.apache.xtable:xtable-api:jar:0.1.0-SNAPSHOT
[INFO] +- org.apache.hadoop:hadoop-common:jar:3.3.6:provided
[INFO] | +-
org.apache.hadoop.thirdparty:hadoop-shaded-protobuf_3_7:jar:1.1.1:provided
[INFO] | +- org.apache.hadoop:hadoop-annotations:jar:3.3.6:provided
[INFO] | +-
org.apache.hadoop.thirdparty:hadoop-shaded-guava:jar:1.1.1:provided
[INFO] | +- org.apache.hadoop:hadoop-auth:jar:3.3.6:provided
[INFO] +- org.apache.hudi:hudi-common:jar:0.14.0:provided
[INFO] | +- org.apache.hbase:hbase-client:jar:2.4.9:provided
[INFO] | | +- org.apache.hadoop:hadoop-auth:jar:2.10.0:provided
[INFO] | | +- org.apache.hadoop:hadoop-common:jar:3.3.6:provided
[INFO] | +- org.apache.hbase:hbase-server:jar:2.4.9:provided
[INFO] | | +- org.apache.hadoop:hadoop-distcp:jar:2.10.0:provided
[INFO] | | +- org.apache.hadoop:hadoop-annotations:jar:2.10.0:provided
[INFO] | | +-
org.apache.hadoop:hadoop-mapreduce-client-core:jar:2.10.0:provided
[INFO] | | | +- org.apache.hadoop:hadoop-yarn-client:jar:2.10.0:provided
[INFO] | | | | \- org.apache.hadoop:hadoop-yarn-api:jar:2.10.0:provided
[INFO] | | | +- org.apache.hadoop:hadoop-yarn-common:jar:2.10.0:provided
[INFO] | | \- org.apache.hadoop:hadoop-hdfs:jar:2.10.0:provided
[INFO] | | +- org.apache.hadoop:hadoop-hdfs-client:jar:2.10.0:provided
[INFO]
[INFO] -------------------< org.apache.xtable:xtable-core
>--------------------
[INFO] Building xtable-core 0.1.0-SNAPSHOT
[5/7]
[INFO] from xtable-core/pom.xml
[INFO] --------------------------------[ jar
]---------------------------------
[INFO]
[INFO] --- dependency:3.6.1:tree (default-cli) @ xtable-core ---
[INFO] org.apache.xtable:xtable-core:jar:0.1.0-SNAPSHOT
[INFO] +- org.apache.hudi:hudi-spark3.4-bundle_2.12:jar:0.14.0:test
[INFO] +- org.apache.hudi:hudi-common:jar:0.14.0:compile
[INFO] | +- org.apache.hbase:hbase-client:jar:2.4.9:compile
[INFO] | | +-
org.apache.hbase.thirdparty:hbase-shaded-protobuf:jar:3.5.1:compile
[INFO] | | +- org.apache.hbase:hbase-common:jar:2.4.9:compile
[INFO] | | | +- org.apache.hbase:hbase-logging:jar:2.4.9:compile
[INFO] | | | \-
org.apache.hbase.thirdparty:hbase-shaded-gson:jar:3.5.1:compile
[INFO] | | +- org.apache.hbase:hbase-hadoop-compat:jar:2.4.9:compile
[INFO] | | +- org.apache.hbase:hbase-hadoop2-compat:jar:2.4.9:compile
[INFO] | +- org.apache.hbase:hbase-server:jar:2.4.9:compile
[INFO] | | +- org.apache.hadoop:hadoop-distcp:jar:2.10.0:compile
[INFO] | | +-
org.apache.hadoop:hadoop-mapreduce-client-core:jar:2.10.0:compile
[INFO] | | | +- org.apache.hadoop:hadoop-yarn-client:jar:2.10.0:compile
[INFO] | | | | \- org.apache.hadoop:hadoop-yarn-api:jar:2.10.0:compile
[INFO] | | | +- org.apache.hadoop:hadoop-yarn-common:jar:2.10.0:compile
[INFO] | | \- org.apache.hadoop:hadoop-hdfs:jar:2.10.0:compile
[INFO] | | +- org.apache.hadoop:hadoop-hdfs-client:jar:2.10.0:compile
[INFO] +- org.apache.hadoop:hadoop-common:jar:3.3.6:provided
[INFO] | +-
org.apache.hadoop.thirdparty:hadoop-shaded-protobuf_3_7:jar:1.1.1:provided
[INFO] | +- org.apache.hadoop:hadoop-annotations:jar:3.3.6:compile
[INFO] | +-
org.apache.hadoop.thirdparty:hadoop-shaded-guava:jar:1.1.1:compile
[INFO] | +- org.apache.hadoop:hadoop-auth:jar:3.3.6:compile
[INFO] | +- org.apache.hadoop:hadoop-client-api:jar:3.3.4:provided
[INFO] | +- org.apache.hadoop:hadoop-client-runtime:jar:3.3.4:provided
[INFO] -----------------< org.apache.xtable:xtable-utilities
>-----------------
[INFO] Building xtable-utilities 0.1.0-SNAPSHOT
[6/7]
[INFO] from xtable-utilities/pom.xml
[INFO] --------------------------------[ jar
]---------------------------------
[INFO]
[INFO] --- dependency:3.6.1:tree (default-cli) @ xtable-utilities ---
[INFO] org.apache.xtable:xtable-utilities:jar:0.1.0-SNAPSHOT
[INFO] +- org.apache.xtable:xtable-core:jar:0.1.0-SNAPSHOT:compile
[INFO] | +-
org.apache.xtable:xtable-hudi-support-utils:jar:0.1.0-SNAPSHOT:compile
[INFO] | +- org.apache.hudi:hudi-common:jar:0.14.0:compile
[INFO] | | +- org.apache.hbase:hbase-client:jar:2.4.9:compile
[INFO] | | | +- org.apache.hbase:hbase-hadoop-compat:jar:2.4.9:compile
[INFO] | | | +- org.apache.hbase:hbase-hadoop2-compat:jar:2.4.9:compile
[INFO] | | +- org.apache.hbase:hbase-server:jar:2.4.9:compile
[INFO] | | | +- org.apache.hadoop:hadoop-distcp:jar:2.10.0:compile
[INFO] | | | +-
org.apache.hadoop:hadoop-mapreduce-client-core:jar:2.10.0:compile
[INFO] | | | | +- org.apache.hadoop:hadoop-yarn-client:jar:2.10.0:compile
[INFO] | | | | +- org.apache.hadoop:hadoop-yarn-common:jar:2.10.0:compile
[INFO] | | | \- org.apache.hadoop:hadoop-hdfs:jar:2.10.0:compile
[INFO] | | | +- org.apache.hadoop:hadoop-hdfs-client:jar:2.10.0:compile
[INFO] | +- org.apache.hudi:hudi-java-client:jar:0.14.0:compile
[INFO] | | \- org.apache.hudi:hudi-client-common:jar:0.14.0:compile
[INFO] | | +- org.apache.hudi:hudi-timeline-service:jar:0.14.0:compile
[INFO] +- org.apache.hadoop:hadoop-common:jar:3.3.6:compile
[INFO] | +-
org.apache.hadoop.thirdparty:hadoop-shaded-protobuf_3_7:jar:1.1.1:compile
[INFO] | +- org.apache.hadoop:hadoop-annotations:jar:3.3.6:compile
[INFO] | +-
org.apache.hadoop.thirdparty:hadoop-shaded-guava:jar:1.1.1:compile
[INFO] | +- org.apache.hadoop:hadoop-auth:jar:3.3.6:compile
[INFO] +- org.apache.hive:hive-common:jar:3.1.3:compile
[INFO] | +- org.apache.hive:hive-classification:jar:3.1.3:compile
[INFO] | +- org.apache.hive:hive-shims:jar:3.1.3:compile
[INFO] | | +- org.apache.hive.shims:hive-shims-0.23:jar:3.1.3:runtime
[INFO] | | | \-
org.apache.hadoop:hadoop-yarn-server-resourcemanager:jar:3.1.0:runtime
[INFO] | | | +- org.apache.hadoop:hadoop-yarn-api:jar:3.1.0:compile
[INFO] | | | +-
org.apache.hadoop:hadoop-yarn-server-common:jar:3.1.0:runtime
[INFO] | | | | +-
org.apache.hadoop:hadoop-yarn-registry:jar:3.1.0:runtime
[INFO] | | | +-
org.apache.hadoop:hadoop-yarn-server-applicationhistoryservice:jar:3.1.0:runtime
[INFO] | | | \-
org.apache.hadoop:hadoop-yarn-server-web-proxy:jar:3.1.0:runtime
[INFO] +- org.apache.hadoop:hadoop-aws:jar:3.3.6:runtime
[INFO]
[INFO] ----------< org.apache.xtable:xtable-hudi-support-extensions
>----------
[INFO] Building xtable-hudi-support-extensions 0.1.0-SNAPSHOT
[7/7]
[INFO] from xtable-hudi-support/xtable-hudi-support-extensions/pom.xml
[INFO] --------------------------------[ jar
]---------------------------------
[INFO]
[INFO] --- dependency:3.6.1:tree (default-cli) @
xtable-hudi-support-extensions ---
[INFO] org.apache.xtable:xtable-hudi-support-extensions:jar:0.1.0-SNAPSHOT
[INFO] +-
org.apache.xtable:xtable-hudi-support-utils:jar:0.1.0-SNAPSHOT:compile
[INFO] +- org.apache.xtable:xtable-core:jar:0.1.0-SNAPSHOT:compile
[INFO] | +- org.apache.xtable:xtable-api:jar:0.1.0-SNAPSHOT:compile
[INFO] | +- org.apache.hudi:hudi-common:jar:0.14.0:compile
[INFO] | | +- org.apache.hbase:hbase-client:jar:2.4.9:compile
[INFO] | | | +- org.apache.hbase:hbase-hadoop-compat:jar:2.4.9:compile
[INFO] | | | +- org.apache.hbase:hbase-hadoop2-compat:jar:2.4.9:compile
[INFO] | | +- org.apache.hbase:hbase-server:jar:2.4.9:compile
[INFO] | | | +- org.apache.hadoop:hadoop-distcp:jar:2.10.0:compile
[INFO] | | | +-
org.apache.hadoop:hadoop-mapreduce-client-core:jar:2.10.0:compile
[INFO] | | | | +- org.apache.hadoop:hadoop-yarn-client:jar:2.10.0:compile
[INFO] | | | | | \- org.apache.hadoop:hadoop-yarn-api:jar:2.10.0:compile
[INFO] | | | | +- org.apache.hadoop:hadoop-yarn-common:jar:2.10.0:compile
[INFO] | | | \- org.apache.hadoop:hadoop-hdfs:jar:2.10.0:compile
[INFO] | | | +- org.apache.hadoop:hadoop-hdfs-client:jar:2.10.0:compile
[INFO] +- org.apache.hudi:hudi-client-common:jar:0.14.0:provided
[INFO] | +- org.apache.hudi:hudi-timeline-service:jar:0.14.0:provided
[INFO] +- org.apache.hudi:hudi-sync-common:jar:0.14.0:provided
[INFO] +- org.apache.hadoop:hadoop-common:jar:3.3.6:provided
[INFO] | +-
org.apache.hadoop.thirdparty:hadoop-shaded-protobuf_3_7:jar:1.1.1:provided
[INFO] | +- org.apache.hadoop:hadoop-annotations:jar:3.3.6:compile
[INFO] | +-
org.apache.hadoop.thirdparty:hadoop-shaded-guava:jar:1.1.1:compile
[INFO] | +- org.apache.hadoop:hadoop-auth:jar:3.3.6:compile
[INFO] +- org.apache.hudi:hudi-spark3.4-bundle_2.12:jar:0.14.0:test
[INFO] +- org.apache.hudi:hudi-java-client:jar:0.14.0:test
[INFO]
------------------------------------------------------------------------
[INFO] Reactor Summary for xtable 0.1.0-SNAPSHOT:
[INFO]
[INFO] xtable ............................................. SUCCESS [ 1.167
s]
[INFO] xtable-api ......................................... SUCCESS [ 0.270
s]
[INFO] xtable-hudi-support ................................ SUCCESS [ 0.002
s]
[INFO] xtable-hudi-support-utils .......................... SUCCESS [ 0.294
s]
[INFO] xtable-core ........................................ SUCCESS [ 0.391
s]
[INFO] xtable-utilities ................................... SUCCESS [ 0.529
s]
[INFO] xtable-hudi-support-extensions ..................... SUCCESS [ 0.044
s]
[INFO]
------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO]
------------------------------------------------------------------------
[INFO] Total time: 2.968 s
[INFO] Finished at: 2024-05-21T16:24:17-07:00
[INFO]
------------------------------------------------------------------------
```
I'm not familiar enough with Java to help very much here, but If there's any
other information that I can add to this, let me know. Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
