JiaDe-Wu opened a new issue #2108:
URL: https://github.com/apache/hudi/issues/2108
Hi
I used Hudi-CLI to roll back and found that yarn has no resources, and I
have been waiting for the task to start.
Then manually killed the queued yarn and returned to hudi to find that the
data to be rolled back was gone.
I think if you need to rely on Hudi's MVCC data protection, this is a
problem that needs to be solved.
**Expected behavior**
A clear and concise description of what you expected to happen.
**Environment Description**
* Hudi version : Hudi 0.5
* Spark version :Spark 2.4.4
* Hive version :Hive 2.3.6
* Hadoop version :
* Storage (HDFS/S3/GCS..) :S3
* Running on Docker? (yes/no) :no
log：
```bash
hudi:hudi_trips_table->commits show
20/09/24 03:05:10 INFO timeline.HoodieActiveTimeline: Loaded instants
java.util.stream.ReferencePipeline$Head@4c8c1c29
20/09/24 03:05:10 INFO s3n.S3NativeFileSystem: Opening
's3://jiade-hudi-demo/tmp/hudi/hudi_trips_table/.hoodie/20200924030219.commit'
for reading
20/09/24 03:05:10 INFO s3n.S3NativeFileSystem: Opening
's3://jiade-hudi-demo/tmp/hudi/hudi_trips_table/.hoodie/20200924030059.commit'
for reading
20/09/24 03:05:10 INFO s3n.S3NativeFileSystem: Opening
's3://jiade-hudi-demo/tmp/hudi/hudi_trips_table/.hoodie/20200924025851.commit'
for reading
20/09/24 03:05:10 INFO s3n.S3NativeFileSystem: Opening
's3://jiade-hudi-demo/tmp/hudi/hudi_trips_table/.hoodie/20200924025642.commit'
for reading
╔════════════════╤═════════════════════╤═══════════════════╤═════════════════════╤══════════════════════════╤═══════════════════════╤══════════════════════════════╤══════════════╗
║ CommitTime │ Total Bytes Written │ Total Files Added │ Total Files
Updated │ Total Partitions Written │ Total Records Written │ Total Update
Records Written │ Total Errors ║
╠════════════════╪═════════════════════╪═══════════════════╪═════════════════════╪══════════════════════════╪═══════════════════════╪══════════════════════════════╪══════════════╣
║ 20200924030219 │ 4.6 MB │ 0 │ 1
│ 1 │ 674046 │ 0
│ 0 ║
╟────────────────┼─────────────────────┼───────────────────┼─────────────────────┼──────────────────────────┼───────────────────────┼──────────────────────────────┼──────────────╢
║ 20200924030059 │ 4.6 MB │ 0 │ 1
│ 1 │ 674056 │ 0
│ 0 ║
╟────────────────┼─────────────────────┼───────────────────┼─────────────────────┼──────────────────────────┼───────────────────────┼──────────────────────────────┼──────────────╢
║ 20200924025851 │ 4.7 MB │ 0 │ 1
│ 1 │ 680187 │ 10
│ 0 ║
╟────────────────┼─────────────────────┼───────────────────┼─────────────────────┼──────────────────────────┼───────────────────────┼──────────────────────────────┼──────────────╢
║ 20200924025642 │ 13.8 MB │ 3 │ 0
│ 1 │ 2000000 │ 0
│ 0 ║
╚════════════════╧═════════════════════╧═══════════════════╧═════════════════════╧══════════════════════════╧═══════════════════════╧══════════════════════════════╧══════════════╝
hudi:hudi_trips_table->commit rollback --commit 20200924030219
20/09/24 03:06:28 INFO SparkContext: Running Spark version 2.4.4
20/09/24 03:06:28 INFO SparkContext: Submitted application:
hoodie-cli-ROLLBACK
20/09/24 03:06:28 INFO SecurityManager: Changing view acls to: hadoop
20/09/24 03:06:28 INFO SecurityManager: Changing modify acls to: hadoop
20/09/24 03:06:28 INFO SecurityManager: Changing view acls groups to:
20/09/24 03:06:28 INFO SecurityManager: Changing modify acls groups to:
20/09/24 03:06:28 INFO SecurityManager: SecurityManager: authentication
disabled; ui acls disabled; users with view permissions: Set(hadoop); groups
with view permissions: Set(); users with modify permissions: Set(hadoop);
groups with modify permissions: Set()
20/09/24 03:06:28 INFO deprecation: mapred.output.compression.codec is
deprecated. Instead, use mapreduce.output.fileoutputformat.compress.codec
20/09/24 03:06:28 INFO deprecation: mapred.output.compression.type is
deprecated. Instead, use mapreduce.output.fileoutputformat.compress.type
20/09/24 03:06:28 INFO deprecation: mapred.output.compress is deprecated.
Instead, use mapreduce.output.fileoutputformat.compress
20/09/24 03:06:28 INFO Utils: Successfully started service 'sparkDriver' on
port 33871.
20/09/24 03:06:28 INFO SparkEnv: Registering MapOutputTracker
20/09/24 03:06:28 INFO SparkEnv: Registering BlockManagerMaster
20/09/24 03:06:28 INFO BlockManagerMasterEndpoint: Using
org.apache.spark.storage.DefaultTopologyMapper for getting topology information
20/09/24 03:06:28 INFO BlockManagerMasterEndpoint:
BlockManagerMasterEndpoint up
20/09/24 03:06:28 INFO DiskBlockManager: Created local directory at
/mnt/tmp/blockmgr-4c566073-eb73-4989-ac0a-baa7ee687b75
20/09/24 03:06:28 INFO MemoryStore: MemoryStore started with capacity 1007.8
MB
20/09/24 03:06:28 INFO SparkEnv: Registering OutputCommitCoordinator
20/09/24 03:06:28 WARN Utils: Service 'SparkUI' could not bind on port 4040.
Attempting port 4041.
20/09/24 03:06:29 INFO Utils: Successfully started service 'SparkUI' on port
4041.
20/09/24 03:06:29 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at
http://ip-172-31-24-65.cn-northwest-1.compute.internal:4041
20/09/24 03:06:29 INFO SparkContext: Added JAR
file:/usr/lib/hudi/cli/hudi-cli-0.5.0-incubating.jar at
spark://ip-172-31-24-65.cn-northwest-1.compute.internal:33871/jars/hudi-cli-0.5.0-incubating.jar
with timestamp 1600916789065
20/09/24 03:06:29 INFO Utils: Using initial executors = 2, max of
spark.dynamicAllocation.initialExecutors, spark.dynamicAllocation.minExecutors
and spark.executor.instances
20/09/24 03:06:29 INFO RMProxy: Connecting to ResourceManager at
ip-172-31-24-65.cn-northwest-1.compute.internal/172.31.24.65:8032
20/09/24 03:06:29 INFO Client: Requesting a new application from cluster
with 1 NodeManagers
20/09/24 03:06:29 INFO Client: Verifying our application has not requested
more than the maximum memory capability of the cluster (57344 MB per container)
20/09/24 03:06:29 INFO Client: Will allocate AM container, with 896 MB
memory including 384 MB overhead
20/09/24 03:06:29 INFO Client: Setting up container launch context for our AM
20/09/24 03:06:29 INFO Client: Setting up the launch environment for our AM
container
20/09/24 03:06:29 INFO Client: Preparing resources for our AM container
20/09/24 03:06:29 WARN Client: Neither spark.yarn.jars nor
spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME.
20/09/24 03:06:31 INFO Client: Uploading resource
file:/mnt/tmp/spark-53e31f61-8f5d-49ff-b507-8089dcca8206/__spark_libs__5093892707637809748.zip
->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/__spark_libs__5093892707637809748.zip
20/09/24 03:06:32 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/json4s-core_2.11-3.5.3.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/json4s-core_2.11-3.5.3.jar
20/09/24 03:06:32 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/json4s-jackson_2.11-3.5.3.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/json4s-jackson_2.11-3.5.3.jar
20/09/24 03:06:32 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/json-smart-2.3.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/json-smart-2.3.jar
20/09/24 03:06:32 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/metrics-core-4.0.2.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/metrics-core-4.0.2.jar
20/09/24 03:06:32 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/scala-reflect-2.11.8.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/scala-reflect-2.11.8.jar
20/09/24 03:06:32 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/jackson-module-scala_2.11-2.6.7.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/jackson-module-scala_2.11-2.6.7.jar
20/09/24 03:06:32 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/spring-context-support-4.2.4.RELEASE.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/spring-context-support-4.2.4.RELEASE.jar
20/09/24 03:06:32 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/slf4j-log4j12-1.7.16.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/slf4j-log4j12-1.7.16.jar
20/09/24 03:06:32 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/jetty-server-9.4.15.v20190215.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/jetty-server-9.4.15.v20190215.jar
20/09/24 03:06:33 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/httpcore-4.3.2.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/httpcore-4.3.2.jar
20/09/24 03:06:33 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/stream-2.7.0.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/stream-2.7.0.jar
20/09/24 03:06:33 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/jaxb-impl-2.2.3-1.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/jaxb-impl-2.2.3-1.jar
20/09/24 03:06:33 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/junit-4.11.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/junit-4.11.jar
20/09/24 03:06:33 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/commons-crypto-1.0.0.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/commons-crypto-1.0.0.jar
20/09/24 03:06:33 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/stringtemplate-4.0.2.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/stringtemplate-4.0.2.jar
20/09/24 03:06:33 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/scala-xml_2.11-1.0.2.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/scala-xml_2.11-1.0.2.jar
20/09/24 03:06:33 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/json4s-scalap_2.11-3.5.3.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/json4s-scalap_2.11-3.5.3.jar
20/09/24 03:06:33 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/parquet-column-1.10.1.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/parquet-column-1.10.1.jar
20/09/24 03:06:33 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/metrics-core-2.2.0.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/metrics-core-2.2.0.jar
20/09/24 03:06:33 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/arrow-format-0.14.0.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/arrow-format-0.14.0.jar
20/09/24 03:06:33 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/lz4-java-1.4.0.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/lz4-java-1.4.0.jar
20/09/24 03:06:33 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/jsch-0.1.54.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/jsch-0.1.54.jar
20/09/24 03:06:33 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/curator-recipes-2.7.1.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/curator-recipes-2.7.1.jar
20/09/24 03:06:33 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/apacheds-i18n-2.0.0-M15.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/apacheds-i18n-2.0.0-M15.jar
20/09/24 03:06:33 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/arrow-memory-0.14.0.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/arrow-memory-0.14.0.jar
20/09/24 03:06:33 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/antlr-runtime-3.3.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/antlr-runtime-3.3.jar
20/09/24 03:06:33 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/javassist-3.18.1-GA.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/javassist-3.18.1-GA.jar
20/09/24 03:06:33 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/jetty-xml-9.4.15.v20190215.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/jetty-xml-9.4.15.v20190215.jar
20/09/24 03:06:33 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/scala-parser-combinators_2.11-1.1.0.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/scala-parser-combinators_2.11-1.1.0.jar
20/09/24 03:06:34 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/commons-daemon-1.0.13.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/commons-daemon-1.0.13.jar
20/09/24 03:06:34 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/objenesis-2.5.1.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/objenesis-2.5.1.jar
20/09/24 03:06:34 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/chill-java-0.9.3.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/chill-java-0.9.3.jar
20/09/24 03:06:34 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/jetty-6.1.26-emr.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/jetty-6.1.26-emr.jar
20/09/24 03:06:34 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/chill_2.11-0.9.3.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/chill_2.11-0.9.3.jar
20/09/24 03:06:34 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/pyrolite-4.13.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/pyrolite-4.13.jar
20/09/24 03:06:34 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/javax.annotation-api-1.2.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/javax.annotation-api-1.2.jar
20/09/24 03:06:34 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/leveldbjni-all-1.8.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/leveldbjni-all-1.8.jar
20/09/24 03:06:34 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/jetty-io-9.4.15.v20190215.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/jetty-io-9.4.15.v20190215.jar
20/09/24 03:06:34 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/jersey-client-2.22.2.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/jersey-client-2.22.2.jar
20/09/24 03:06:34 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/hudi-hive-0.5.0-incubating.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/hudi-hive-0.5.0-incubating.jar
20/09/24 03:06:34 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/jline-2.12.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/jline-2.12.jar
20/09/24 03:06:34 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/osgi-resource-locator-1.0.1.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/osgi-resource-locator-1.0.1.jar
20/09/24 03:06:34 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/avro-1.8.2.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/avro-1.8.2.jar
20/09/24 03:06:34 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/kotlin-stdlib-common-1.2.71.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/kotlin-stdlib-common-1.2.71.jar
20/09/24 03:06:34 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/curator-framework-2.7.1.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/curator-framework-2.7.1.jar
20/09/24 03:06:34 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/jcommander-1.72.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/jcommander-1.72.jar
20/09/24 03:06:34 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/jackson-xc-1.9.13.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/jackson-xc-1.9.13.jar
20/09/24 03:06:34 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/minlog-1.3.0.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/minlog-1.3.0.jar
20/09/24 03:06:34 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/common-config-3.0.0.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/common-config-3.0.0.jar
20/09/24 03:06:34 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/jul-to-slf4j-1.7.16.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/jul-to-slf4j-1.7.16.jar
20/09/24 03:06:34 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/jetty-util-9.4.15.v20190215.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/jetty-util-9.4.15.v20190215.jar
20/09/24 03:06:34 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/xmlenc-0.52.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/xmlenc-0.52.jar
20/09/24 03:06:34 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/accessors-smart-1.2.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/accessors-smart-1.2.jar
20/09/24 03:06:34 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/common-utils-3.0.0.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/common-utils-3.0.0.jar
20/09/24 03:06:34 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/parquet-encoding-1.10.1.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/parquet-encoding-1.10.1.jar
20/09/24 03:06:34 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/netty-all-4.1.17.Final.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/netty-all-4.1.17.Final.jar
20/09/24 03:06:34 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/parquet-avro-1.10.1.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/parquet-avro-1.10.1.jar
20/09/24 03:06:34 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/jackson-mapper-asl-1.9.13.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/jackson-mapper-asl-1.9.13.jar
20/09/24 03:06:34 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/aircompressor-0.10.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/aircompressor-0.10.jar
20/09/24 03:06:34 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/servlet-api-2.5.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/servlet-api-2.5.jar
20/09/24 03:06:34 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/jsp-api-2.1.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/jsp-api-2.1.jar
20/09/24 03:06:34 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/curator-client-2.7.1.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/curator-client-2.7.1.jar
20/09/24 03:06:34 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/metrics-graphite-4.0.2.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/metrics-graphite-4.0.2.jar
20/09/24 03:06:35 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/asm-5.0.4.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/asm-5.0.4.jar
20/09/24 03:06:35 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/json4s-ast_2.11-3.5.3.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/json4s-ast_2.11-3.5.3.jar
20/09/24 03:06:35 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/commons-lang-2.6.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/commons-lang-2.6.jar
20/09/24 03:06:35 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/websocket-servlet-9.4.15.v20190215.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/websocket-servlet-9.4.15.v20190215.jar
20/09/24 03:06:35 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/okio-1.4.0.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/okio-1.4.0.jar
20/09/24 03:06:35 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/jcl-over-slf4j-1.7.16.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/jcl-over-slf4j-1.7.16.jar
20/09/24 03:06:35 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/jackson-databind-2.6.7.1.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/jackson-databind-2.6.7.1.jar
20/09/24 03:06:35 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/guava-15.0.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/guava-15.0.jar
20/09/24 03:06:35 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/ivy-2.4.0.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/ivy-2.4.0.jar
20/09/24 03:06:35 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/commons-net-3.1.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/commons-net-3.1.jar
20/09/24 03:06:35 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/py4j-0.10.7.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/py4j-0.10.7.jar
20/09/24 03:06:35 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/fluent-hc-4.3.2.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/fluent-hc-4.3.2.jar
20/09/24 03:06:35 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/lz4-1.2.0.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/lz4-1.2.0.jar
20/09/24 03:06:35 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/jackson-core-asl-1.9.13.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/jackson-core-asl-1.9.13.jar
20/09/24 03:06:35 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/jsr305-1.3.9.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/jsr305-1.3.9.jar
20/09/24 03:06:35 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/parquet-format-2.4.0.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/parquet-format-2.4.0.jar
20/09/24 03:06:35 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/flatbuffers-java-1.9.0.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/flatbuffers-java-1.9.0.jar
20/09/24 03:06:35 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/bijection-core_2.11-0.9.2.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/bijection-core_2.11-0.9.2.jar
20/09/24 03:06:35 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/orc-mapreduce-1.5.5-nohive.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/orc-mapreduce-1.5.5-nohive.jar
20/09/24 03:06:35 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/parquet-common-1.10.1.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/parquet-common-1.10.1.jar
20/09/24 03:06:36 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/jersey-container-servlet-2.22.2.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/jersey-container-servlet-2.22.2.jar
20/09/24 03:06:36 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/websocket-client-9.4.15.v20190215.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/websocket-client-9.4.15.v20190215.jar
20/09/24 03:06:36 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/fliptables-1.0.2.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/fliptables-1.0.2.jar
20/09/24 03:06:37 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/zkclient-0.5.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/zkclient-0.5.jar
20/09/24 03:06:37 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/rocksdbjni-5.17.2.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/rocksdbjni-5.17.2.jar
20/09/24 03:06:37 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/log4j-1.2.17.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/log4j-1.2.17.jar
20/09/24 03:06:38 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/avro-ipc-1.8.2.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/avro-ipc-1.8.2.jar
20/09/24 03:06:38 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/jackson-annotations-2.6.7.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/jackson-annotations-2.6.7.jar
20/09/24 03:06:38 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/jersey-guava-2.22.2.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/jersey-guava-2.22.2.jar
20/09/24 03:06:38 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/antlr4-runtime-4.7.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/antlr4-runtime-4.7.jar
20/09/24 03:06:38 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/annotations-13.0.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/annotations-13.0.jar
20/09/24 03:06:38 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/commons-collections-3.2.2.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/commons-collections-3.2.2.jar
20/09/24 03:06:38 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/netty-3.9.9.Final.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/netty-3.9.9.Final.jar
20/09/24 03:06:38 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/gson-2.3.1.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/gson-2.3.1.jar
20/09/24 03:06:38 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/kafka_2.11-0.8.2.1.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/kafka_2.11-0.8.2.1.jar
20/09/24 03:06:38 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/bijection-avro_2.11-0.9.2.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/bijection-avro_2.11-0.9.2.jar
20/09/24 03:06:38 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/httpclient-4.3.2.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/httpclient-4.3.2.jar
20/09/24 03:06:38 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/api-util-1.0.0-M20.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/api-util-1.0.0-M20.jar
20/09/24 03:06:38 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/commons-codec-1.4.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/commons-codec-1.4.jar
20/09/24 03:06:38 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/commons-lang3-3.5.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/commons-lang3-3.5.jar
20/09/24 03:06:38 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/nimbus-jose-jwt-4.41.1.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/nimbus-jose-jwt-4.41.1.jar
20/09/24 03:06:38 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/jersey-core-1.9.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/jersey-core-1.9.jar
20/09/24 03:06:38 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/jcip-annotations-1.0-1.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/jcip-annotations-1.0-1.jar
20/09/24 03:06:38 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/commons-compress-1.4.1.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/commons-compress-1.4.1.jar
20/09/24 03:06:38 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/kryo-shaded-4.0.2.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/kryo-shaded-4.0.2.jar
20/09/24 03:06:38 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/jettison-1.1.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/jettison-1.1.jar
20/09/24 03:06:38 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/asm-3.1.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/asm-3.1.jar
20/09/24 03:06:38 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/websocket-common-9.4.15.v20190215.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/websocket-common-9.4.15.v20190215.jar
20/09/24 03:06:38 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/hudi-timeline-service-0.5.0-incubating.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/hudi-timeline-service-0.5.0-incubating.jar
20/09/24 03:06:38 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/commons-math3-3.4.1.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/commons-math3-3.4.1.jar
20/09/24 03:06:38 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/jetty-http-9.4.15.v20190215.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/jetty-http-9.4.15.v20190215.jar
20/09/24 03:06:38 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/commons-compiler-3.0.9.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/commons-compiler-3.0.9.jar
20/09/24 03:06:38 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/javax.servlet-api-3.1.0.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/javax.servlet-api-3.1.0.jar
20/09/24 03:06:38 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/jersey-common-2.22.2.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/jersey-common-2.22.2.jar
20/09/24 03:06:38 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/htrace-core4-4.0.1-incubating.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/htrace-core4-4.0.1-incubating.jar
20/09/24 03:06:38 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/commons-configuration-1.6.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/commons-configuration-1.6.jar
20/09/24 03:06:38 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/api-asn1-api-1.0.0-M20.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/api-asn1-api-1.0.0-M20.jar
20/09/24 03:06:38 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/websocket-api-9.4.15.v20190215.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/websocket-api-9.4.15.v20190215.jar
20/09/24 03:06:38 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/validation-api-1.1.0.Final.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/validation-api-1.1.0.Final.jar
20/09/24 03:06:38 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/zstd-jni-1.3.2-2.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/zstd-jni-1.3.2-2.jar
20/09/24 03:06:38 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/jackson-jaxrs-1.9.13.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/jackson-jaxrs-1.9.13.jar
20/09/24 03:06:38 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/jersey-server-1.9.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/jersey-server-1.9.jar
20/09/24 03:06:38 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/jetty-webapp-9.4.15.v20190215.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/jetty-webapp-9.4.15.v20190215.jar
20/09/24 03:06:38 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/jackson-dataformat-cbor-2.6.7.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/jackson-dataformat-cbor-2.6.7.jar
20/09/24 03:06:38 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/hudi-client-0.5.0-incubating.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/hudi-client-0.5.0-incubating.jar
20/09/24 03:06:38 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/metrics-jvm-3.1.5.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/metrics-jvm-3.1.5.jar
20/09/24 03:06:38 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/slf4j-api-1.7.16.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/slf4j-api-1.7.16.jar
20/09/24 03:06:38 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/joda-time-2.9.9.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/joda-time-2.9.9.jar
20/09/24 03:06:38 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/orc-shims-1.5.5.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/orc-shims-1.5.5.jar
20/09/24 03:06:38 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/kotlin-stdlib-1.2.71.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/kotlin-stdlib-1.2.71.jar
20/09/24 03:06:39 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/spring-expression-4.2.4.RELEASE.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/spring-expression-4.2.4.RELEASE.jar
20/09/24 03:06:39 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/spring-context-4.2.4.RELEASE.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/spring-context-4.2.4.RELEASE.jar
20/09/24 03:06:39 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/orc-core-1.5.5-nohive.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/orc-core-1.5.5-nohive.jar
20/09/24 03:06:39 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/commons-digester-1.8.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/commons-digester-1.8.jar
20/09/24 03:06:39 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/spring-shell-1.2.0.RELEASE.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/spring-shell-1.2.0.RELEASE.jar
20/09/24 03:06:39 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/hamcrest-core-1.3.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/hamcrest-core-1.3.jar
20/09/24 03:06:39 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/websocket-server-9.4.15.v20190215.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/websocket-server-9.4.15.v20190215.jar
20/09/24 03:06:39 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/kotlin-stdlib-jdk7-1.2.71.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/kotlin-stdlib-jdk7-1.2.71.jar
20/09/24 03:06:39 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/hk2-locator-2.4.0-b34.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/hk2-locator-2.4.0-b34.jar
20/09/24 03:06:39 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/commons-pool-1.6.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/commons-pool-1.6.jar
20/09/24 03:06:39 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/univocity-parsers-2.7.3.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/univocity-parsers-2.7.3.jar
20/09/24 03:06:39 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/javax.ws.rs-api-2.0.1.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/javax.ws.rs-api-2.0.1.jar
20/09/24 03:06:39 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/commons-beanutils-core-1.8.0.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/commons-beanutils-core-1.8.0.jar
20/09/24 03:06:39 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/jets3t-0.9.0.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/jets3t-0.9.0.jar
20/09/24 03:06:39 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/snappy-java-1.1.7.3.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/snappy-java-1.1.7.3.jar
20/09/24 03:06:39 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/java-xmlbuilder-0.4.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/java-xmlbuilder-0.4.jar
20/09/24 03:06:39 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/jersey-media-jaxb-2.17.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/jersey-media-jaxb-2.17.jar
20/09/24 03:06:39 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/arrow-vector-0.14.0.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/arrow-vector-0.14.0.jar
20/09/24 03:06:39 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/spring-core-4.2.4.RELEASE.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/spring-core-4.2.4.RELEASE.jar
20/09/24 03:06:39 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/hudi-utilities-0.5.0-incubating.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/hudi-utilities-0.5.0-incubating.jar
20/09/24 03:06:39 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/jackson-core-2.6.7.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/jackson-core-2.6.7.jar
20/09/24 03:06:39 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/jetty-client-9.4.15.v20190215.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/jetty-client-9.4.15.v20190215.jar
20/09/24 03:06:39 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/scala-library-2.11.8.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/scala-library-2.11.8.jar
20/09/24 03:06:39 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/xercesImpl-2.9.1.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/xercesImpl-2.9.1.jar
20/09/24 03:06:39 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/jetty-util-6.1.26-emr.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/jetty-util-6.1.26-emr.jar
20/09/24 03:06:40 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/hk2-utils-2.4.0-b34.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/hk2-utils-2.4.0-b34.jar
20/09/24 03:06:40 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/javax.inject-2.4.0-b34.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/javax.inject-2.4.0-b34.jar
20/09/24 03:06:41 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/jersey-server-2.17.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/jersey-server-2.17.jar
20/09/24 03:06:41 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/RoaringBitmap-0.7.45.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/RoaringBitmap-0.7.45.jar
20/09/24 03:06:41 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/jetty-servlet-9.4.15.v20190215.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/jetty-servlet-9.4.15.v20190215.jar
20/09/24 03:06:42 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/hppc-0.7.2.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/hppc-0.7.2.jar
20/09/24 03:06:42 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/kafka-avro-serializer-3.0.0.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/kafka-avro-serializer-3.0.0.jar
20/09/24 03:06:43 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/parquet-jackson-1.10.1.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/parquet-jackson-1.10.1.jar
20/09/24 03:06:43 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/okhttp-2.4.0.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/okhttp-2.4.0.jar
20/09/24 03:06:43 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/aopalliance-1.0.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/aopalliance-1.0.jar
20/09/24 03:06:44 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/htrace-core-3.1.0-incubating.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/htrace-core-3.1.0-incubating.jar
20/09/24 03:06:44 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/spring-beans-4.2.4.RELEASE.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/spring-beans-4.2.4.RELEASE.jar
20/09/24 03:06:45 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/jcodings-1.0.8.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/jcodings-1.0.8.jar
20/09/24 03:06:45 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/aopalliance-repackaged-2.4.0-b34.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/aopalliance-repackaged-2.4.0-b34.jar
20/09/24 03:06:45 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/jersey-json-1.9.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/jersey-json-1.9.jar
20/09/24 03:06:46 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/jetty-security-9.4.15.v20190215.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/jetty-security-9.4.15.v20190215.jar
20/09/24 03:06:46 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/protobuf-java-2.5.0.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/protobuf-java-2.5.0.jar
20/09/24 03:06:46 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/jetty-sslengine-6.1.26-emr.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/jetty-sslengine-6.1.26-emr.jar
20/09/24 03:06:47 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/kotlin-stdlib-jdk8-1.2.71.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/kotlin-stdlib-jdk8-1.2.71.jar
20/09/24 03:06:47 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/xbean-asm6-shaded-4.8.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/xbean-asm6-shaded-4.8.jar
20/09/24 03:06:48 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/hudi-common-0.5.0-incubating.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/hudi-common-0.5.0-incubating.jar
20/09/24 03:06:48 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/activation-1.1.1.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/activation-1.1.1.jar
20/09/24 03:06:49 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/oro-2.0.8.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/oro-2.0.8.jar
20/09/24 03:06:49 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/spring-aop-4.2.4.RELEASE.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/spring-aop-4.2.4.RELEASE.jar
20/09/24 03:06:49 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/kafka-clients-0.8.2.1.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/kafka-clients-0.8.2.1.jar
20/09/24 03:06:50 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/javalin-2.8.0.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/javalin-2.8.0.jar
20/09/24 03:06:50 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/findbugs-annotations-1.3.9-1.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/findbugs-annotations-1.3.9-1.jar
20/09/24 03:06:51 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/paranamer-2.8.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/paranamer-2.8.jar
20/09/24 03:06:51 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/joni-2.1.2.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/joni-2.1.2.jar
20/09/24 03:06:51 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/unused-1.0.0.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/unused-1.0.0.jar
20/09/24 03:06:52 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/xz-1.5.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/xz-1.5.jar
20/09/24 03:06:52 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/compress-lzf-1.0.3.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/compress-lzf-1.0.3.jar
20/09/24 03:06:53 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/commons-beanutils-1.7.0.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/commons-beanutils-1.7.0.jar
20/09/24 03:06:53 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/xml-apis-1.3.04.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/xml-apis-1.3.04.jar
20/09/24 03:06:54 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/jersey-container-servlet-core-2.17.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/jersey-container-servlet-core-2.17.jar
20/09/24 03:06:54 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/apacheds-kerberos-codec-2.0.0-M15.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/apacheds-kerberos-codec-2.0.0-M15.jar
20/09/24 03:06:54 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/metrics-json-3.1.5.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/metrics-json-3.1.5.jar
20/09/24 03:06:55 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/commons-logging-1.1.3.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/commons-logging-1.1.3.jar
20/09/24 03:06:55 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/shims-0.7.45.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/shims-0.7.45.jar
20/09/24 03:06:56 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/janino-3.0.9.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/janino-3.0.9.jar
20/09/24 03:06:56 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/kafka-schema-registry-client-3.0.0.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/kafka-schema-registry-client-3.0.0.jar
20/09/24 03:06:57 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/commons-cli-1.2.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/commons-cli-1.2.jar
20/09/24 03:06:57 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/hk2-api-2.4.0-b34.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/hk2-api-2.4.0-b34.jar
20/09/24 03:06:57 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/commons-io-2.4.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/commons-io-2.4.jar
20/09/24 03:06:57 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/fastutil-7.0.13.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/fastutil-7.0.13.jar
20/09/24 03:06:57 INFO Client: Uploading resource
file:/usr/lib/hudi/cli/lib/jackson-module-paranamer-2.6.7.jar ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/jackson-module-paranamer-2.6.7.jar
20/09/24 03:06:57 INFO Client: Uploading resource
file:/etc/spark/conf/hive-site.xml ->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/hive-site.xml
20/09/24 03:06:58 INFO Client: Uploading resource
file:/mnt/tmp/spark-53e31f61-8f5d-49ff-b507-8089dcca8206/__spark_conf__571277557544014864.zip
->
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002/__spark_conf__.zip
20/09/24 03:06:58 INFO SecurityManager: Changing view acls to: hadoop
20/09/24 03:06:58 INFO SecurityManager: Changing modify acls to: hadoop
20/09/24 03:06:58 INFO SecurityManager: Changing view acls groups to:
20/09/24 03:06:58 INFO SecurityManager: Changing modify acls groups to:
20/09/24 03:06:58 INFO SecurityManager: SecurityManager: authentication
disabled; ui acls disabled; users with view permissions: Set(hadoop); groups
with view permissions: Set(); users with modify permissions: Set(hadoop);
groups with modify permissions: Set()
20/09/24 03:06:59 INFO Client: Submitting application
application_1600906332593_0002 to ResourceManager
20/09/24 03:06:59 INFO YarnClientImpl: Submitted application
application_1600906332593_0002
20/09/24 03:06:59 INFO SchedulerExtensionServices: Starting Yarn extension
services with app application_1600906332593_0002 and attemptId None
20/09/24 03:07:00 INFO Client: Application report for
application_1600906332593_0002 (state: ACCEPTED)
20/09/24 03:07:00 INFO Client:
hudi:hud client token: N/A
hudi:hud diagnostics: AM container is launched, waiting for AM container to
Register with RM
hudi:hud ApplicationMaster host: N/A
hudi:hud ApplicationMaster RPC port: -1
hudi:hud queue: default
hudi:hud start time: 1600916819911
hudi:hud final status: UNDEFINED
hudi:hud tracking URL:
http://ip-172-31-24-65.cn-northwest-1.compute.internal:20888/proxy/application_1600906332593_0002/
hudi:hud user: hadoop->
20/09/24 03:07:01 INFO Client: Application report for
application_1600906332593_0002 (state: ACCEPTED)
20/09/24 03:07:02 INFO Client: Application report for
application_1600906332593_0002 (state: ACCEPTED)
20/09/24 03:07:03 INFO Client: Application report for
application_1600906332593_0002 (state: ACCEPTED)
20/09/24 03:07:04 INFO Client: Application report for
application_1600906332593_0002 (state: ACCEPTED)
20/09/24 03:07:05 INFO Client: Application report for
application_1600906332593_0002 (state: ACCEPTED)
20/09/24 03:07:06 INFO YarnClientSchedulerBackend: Add WebUI Filter.
org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter, Map(PROXY_HOSTS ->
ip-172-31-24-65.cn-northwest-1.compute.internal, PROXY_URI_BASES ->
http://ip-172-31-24-65.cn-northwest-1.compute.internal:20888/proxy/application_1600906332593_0002),
/proxy/application_1600906332593_0002
20/09/24 03:07:06 INFO JettyUtils: Adding filter
org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter to /jobs,
/jobs/json, /jobs/job, /jobs/job/json, /stages, /stages/json, /stages/stage,
/stages/stage/json, /stages/pool, /stages/pool/json, /storage, /storage/json,
/storage/rdd, /storage/rdd/json, /environment, /environment/json, /executors,
/executors/json, /executors/threadDump, /executors/threadDump/json, /static, /,
/api, /jobs/job/kill, /stages/stage/kill.
20/09/24 03:07:06 INFO YarnSchedulerBackend$YarnSchedulerEndpoint:
ApplicationMaster registered as NettyRpcEndpointRef(spark-client://YarnAM)
20/09/24 03:07:06 INFO Client: Application report for
application_1600906332593_0002 (state: RUNNING)
20/09/24 03:07:06 INFO Client:
hudi:hud client token: N/A
hudi:hud diagnostics: N/A
hudi:hud ApplicationMaster host: 172.31.24.65
hudi:hud ApplicationMaster RPC port: -1
hudi:hud queue: default
hudi:hud start time: 1600916819911
hudi:hud final status: UNDEFINED
hudi:hud tracking URL:
http://ip-172-31-24-65.cn-northwest-1.compute.internal:20888/proxy/application_1600906332593_0002/
hudi:hud user: hadoop->
20/09/24 03:07:06 INFO YarnClientSchedulerBackend: Application
application_1600906332593_0002 has started running.
20/09/24 03:07:06 INFO Utils: Successfully started service
'org.apache.spark.network.netty.NettyBlockTransferService' on port 35295.
20/09/24 03:07:06 INFO NettyBlockTransferService: Server created on
ip-172-31-24-65.cn-northwest-1.compute.internal:35295
20/09/24 03:07:06 INFO BlockManager: Using
org.apache.spark.storage.RandomBlockReplicationPolicy for block replication
policy
20/09/24 03:07:06 INFO BlockManagerMaster: Registering BlockManager
BlockManagerId(driver, ip-172-31-24-65.cn-northwest-1.compute.internal, 35295,
None)
20/09/24 03:07:06 INFO BlockManagerMasterEndpoint: Registering block manager
ip-172-31-24-65.cn-northwest-1.compute.internal:35295 with 1007.8 MB RAM,
BlockManagerId(driver, ip-172-31-24-65.cn-northwest-1.compute.internal, 35295,
None)
20/09/24 03:07:06 INFO BlockManagerMaster: Registered BlockManager
BlockManagerId(driver, ip-172-31-24-65.cn-northwest-1.compute.internal, 35295,
None)
20/09/24 03:07:06 INFO BlockManager: external shuffle service port = 7337
20/09/24 03:07:06 INFO BlockManager: Initialized BlockManager:
BlockManagerId(driver, ip-172-31-24-65.cn-northwest-1.compute.internal, 35295,
None)
20/09/24 03:07:07 INFO JettyUtils: Adding filter
org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter to /metrics/json.
20/09/24 03:07:07 INFO EventLoggingListener: Logging events to
hdfs:/var/log/spark/apps/application_1600906332593_0002
20/09/24 03:07:07 INFO Utils: Using initial executors = 2, max of
spark.dynamicAllocation.initialExecutors, spark.dynamicAllocation.minExecutors
and spark.executor.instances
20/09/24 03:07:07 INFO YarnClientSchedulerBackend: SchedulerBackend is ready
for scheduling beginning after reached minRegisteredResourcesRatio: 0.0
20/09/24 03:07:08 INFO S3NativeFileSystem: Opening
's3://jiade-hudi-demo/tmp/hudi/hudi_trips_table/.hoodie/hoodie.properties' for
reading
20/09/24 03:07:09 INFO S3NativeFileSystem: Opening
's3://jiade-hudi-demo/tmp/hudi/hudi_trips_table/.hoodie/hoodie.properties' for
reading
20/09/24 03:07:09 INFO S3NativeFileSystem: rename
s3://jiade-hudi-demo/tmp/hudi/hudi_trips_table/.hoodie/20200924030219.commit
s3://jiade-hudi-demo/tmp/hudi/hudi_trips_table/.hoodie/20200924030219.inflight
20/09/24 03:07:09 INFO SparkContext: Starting job: collect at
HoodieCopyOnWriteTable.java:359
20/09/24 03:07:09 INFO DAGScheduler: Got job 0 (collect at
HoodieCopyOnWriteTable.java:359) with 32 output partitions
20/09/24 03:07:09 INFO DAGScheduler: Final stage: ResultStage 0 (collect at
HoodieCopyOnWriteTable.java:359)
20/09/24 03:07:09 INFO DAGScheduler: Parents of final stage: List()
20/09/24 03:07:09 INFO DAGScheduler: Missing parents: List()
20/09/24 03:07:09 INFO DAGScheduler: Submitting ResultStage 0
(MapPartitionsRDD[1] at map at HoodieCopyOnWriteTable.java:353), which has no
missing parents
20/09/24 03:07:09 INFO MemoryStore: Block broadcast_0 stored as values in
memory (estimated size 147.9 KB, free 1007.7 MB)
20/09/24 03:07:09 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes
in memory (estimated size 55.0 KB, free 1007.7 MB)
20/09/24 03:07:09 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory
on ip-172-31-24-65.cn-northwest-1.compute.internal:35295 (size: 55.0 KB, free:
1007.8 MB)
20/09/24 03:07:09 INFO SparkContext: Created broadcast 0 from broadcast at
DAGScheduler.scala:1201
20/09/24 03:07:09 INFO DAGScheduler: Submitting 32 missing tasks from
ResultStage 0 (MapPartitionsRDD[1] at map at HoodieCopyOnWriteTable.java:353)
(first 15 tasks are for partitions Vector(0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11,
12, 13, 14))
20/09/24 03:07:09 INFO YarnScheduler: Adding task set 0.0 with 32 tasks
20/09/24 03:07:24 WARN YarnScheduler: Initial job has not accepted any
resources; check your cluster UI to ensure that workers are registered and have
sufficient resources
20/09/24 03:07:39 WARN YarnScheduler: Initial job has not accepted any
resources; check your cluster UI to ensure that workers are registered and have
sufficient resources
20/09/24 03:07:54 WARN YarnScheduler: Initial job has not accepted any
resources; check your cluster UI to ensure that workers are registered and have
sufficient resources
20/09/24 03:08:09 WARN YarnScheduler: Initial job has not accepted any
resources; check your cluster UI to ensure that workers are registered and have
sufficient resources
20/09/24 03:08:24 WARN YarnScheduler: Initial job has not accepted any
resources; check your cluster UI to ensure that workers are registered and have
sufficient resources
20/09/24 03:08:39 WARN YarnScheduler: Initial job has not accepted any
resources; check your cluster UI to ensure that workers are registered and have
sufficient resources
20/09/24 03:08:54 WARN YarnScheduler: Initial job has not accepted any
resources; check your cluster UI to ensure that workers are registered and have
sufficient resources
20/09/24 03:09:09 WARN YarnScheduler: Initial job has not accepted any
resources; check your cluster UI to ensure that workers are registered and have
sufficient resources
20/09/24 03:09:24 WARN YarnScheduler: Initial job has not accepted any
resources; check your cluster UI to ensure that workers are registered and have
sufficient resources
20/09/24 03:09:39 WARN YarnScheduler: Initial job has not accepted any
resources; check your cluster UI to ensure that workers are registered and have
sufficient resources
20/09/24 03:09:54 WARN YarnScheduler: Initial job has not accepted any
resources; check your cluster UI to ensure that workers are registered and have
sufficient resources
20/09/24 03:10:09 WARN YarnScheduler: Initial job has not accepted any
resources; check your cluster UI to ensure that workers are registered and have
sufficient resources
20/09/24 03:10:24 WARN YarnScheduler: Initial job has not accepted any
resources; check your cluster UI to ensure that workers are registered and have
sufficient resources
20/09/24 03:10:39 WARN YarnScheduler: Initial job has not accepted any
resources; check your cluster UI to ensure that workers are registered and have
sufficient resources
20/09/24 03:10:54 WARN YarnScheduler: Initial job has not accepted any
resources; check your cluster UI to ensure that workers are registered and have
sufficient resources
20/09/24 03:11:09 WARN YarnScheduler: Initial job has not accepted any
resources; check your cluster UI to ensure that workers are registered and have
sufficient resources
20/09/24 03:11:24 WARN YarnScheduler: Initial job has not accepted any
resources; check your cluster UI to ensure that workers are registered and have
sufficient resources
20/09/24 03:11:39 WARN YarnScheduler: Initial job has not accepted any
resources; check your cluster UI to ensure that workers are registered and have
sufficient resources
20/09/24 03:11:54 WARN YarnScheduler: Initial job has not accepted any
resources; check your cluster UI to ensure that workers are registered and have
sufficient resources
20/09/24 03:12:09 WARN YarnScheduler: Initial job has not accepted any
resources; check your cluster UI to ensure that workers are registered and have
sufficient resources
20/09/24 03:12:24 WARN YarnScheduler: Initial job has not accepted any
resources; check your cluster UI to ensure that workers are registered and have
sufficient resources
20/09/24 03:12:28 INFO Client: Deleted staging directory
hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1600906332593_0002
20/09/24 03:12:28 ERROR YarnClientSchedulerBackend: YARN application has
exited unexpectedly with state KILLED! Check the YARN application logs for more
details.
20/09/24 03:12:28 ERROR YarnClientSchedulerBackend: Diagnostics message:
Application application_1600906332593_0002 was killed by user hadoop at
172.31.24.65
20/09/24 03:12:28 INFO SparkUI: Stopped Spark web UI at
http://ip-172-31-24-65.cn-northwest-1.compute.internal:4041
20/09/24 03:12:28 INFO DAGScheduler: Job 0 failed: collect at
HoodieCopyOnWriteTable.java:359, took 318.580508 s
Exception in thread "main" org.apache.spark.SparkException: Job 0 cancelled
because SparkContext was shut down
hudi:hudat
org.apache.spark.scheduler.DAGScheduler$$anonfun$cleanUpAfterSchedulerStop$1.apply(DAGScheduler.scala:972)
hudi:hudat
org.apache.spark.scheduler.DAGScheduler$$anonfun$cleanUpAfterSchedulerStop$1.apply(DAGScheduler.scala:970)
hudi:hudat scala.collection.mutable.HashSet.foreach(HashSet.scala:78)
hudi:hudat
org.apache.spark.scheduler.DAGScheduler.cleanUpAfterSchedulerStop(DAGScheduler.scala:970)
hudi:hudat
org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onStop(DAGScheduler.scala:2286)
hudi:hudat org.apache.spark.util.EventLoop.stop(EventLoop.scala:84)
hudi:hudat
org.apache.spark.scheduler.DAGScheduler.stop(DAGScheduler.scala:2193)
hudi:hudat
org.apache.spark.SparkContext$$anonfun$stop$6.apply$mcV$sp(SparkContext.scala:1949)
hudi:hudat org.apache.spark.util.Utils$.tryLogNonFatalError(Utils.scala:1340)
hudi:hudat org.apache.spark.SparkContext.stop(SparkContext.scala:1948)
hudi:hudat
org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend$MonitorThread.run(YarnClientSchedulerBackend.scala:121)
hudi:hudat
org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:777)
hudi:hudat org.apache.spark.SparkContext.runJob(SparkContext.scala:2061)
hudi:hudat org.apache.spark.SparkContext.runJob(SparkContext.scala:2082)
hudi:hudat org.apache.spark.SparkContext.runJob(SparkContext.scala:2101)
hudi:hudat org.apache.spark.SparkContext.runJob(SparkContext.scala:2126)
hudi:hudat org.apache.spark.rdd.RDD$$anonfun$collect$1.apply(RDD.scala:945)
hudi:hudat
org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
hudi:hudat
org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
hudi:hudat org.apache.spark.rdd.RDD.withScope(RDD.scala:363)
hudi:hudat org.apache.spark.rdd.RDD.collect(RDD.scala:944)
hudi:hudat
org.apache.spark.api.java.JavaRDDLike$class.collect(JavaRDDLike.scala:361)
hudi:hudat
org.apache.spark.api.java.AbstractJavaRDDLike.collect(JavaRDDLike.scala:45)
hudi:hudat
org.apache.hudi.table.HoodieCopyOnWriteTable.rollback(HoodieCopyOnWriteTable.java:359)
hudi:hudat
org.apache.hudi.HoodieWriteClient.doRollbackAndGetStats(HoodieWriteClient.java:834)
hudi:hudat
org.apache.hudi.HoodieWriteClient.rollbackInternal(HoodieWriteClient.java:907)
hudi:hudat
org.apache.hudi.HoodieWriteClient.rollback(HoodieWriteClient.java:733)
hudi:hudat
org.apache.hudi.cli.commands.SparkMain.rollback(SparkMain.java:223)
hudi:hudat org.apache.hudi.cli.commands.SparkMain.main(SparkMain.java:60)
hudi:hudat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
hudi:hudat
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
hudi:hudat
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
hudi:hudat java.lang.reflect.Method.invoke(Method.java:498)
hudi:hudat
org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
hudi:hudat
org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$runMain(SparkSubmit.scala:853)
hudi:hudat
org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:161)
hudi:hudat org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:184)
hudi:hudat org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
hudi:hudat
org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:928)
hudi:hudat org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:937)
hudi:hudat org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
20/09/24 03:12:28 INFO DAGScheduler: ResultStage 0 (collect at
HoodieCopyOnWriteTable.java:359) failed in 318.530 s due to Stage cancelled
because SparkContext was shut down
20/09/24 03:12:28 ERROR TransportClient: Failed to send RPC RPC
8528272860118368920 to /172.31.24.65:33218:
java.nio.channels.ClosedChannelException
java.nio.channels.ClosedChannelException
hudi:hudat
io.netty.channel.AbstractChannel$AbstractUnsafe.write(...)(Unknown Source)
20/09/24 03:12:28 INFO DiskBlockManager: Shutdown hook called
20/09/24 03:12:28 ERROR YarnSchedulerBackend$YarnSchedulerEndpoint: Sending
RequestExecutors(0,0,Map(),Set()) to AM was unsuccessful
java.io.IOException: Failed to send RPC RPC 8528272860118368920 to
/172.31.24.65:33218: java.nio.channels.ClosedChannelException
hudi:hudat
org.apache.spark.network.client.TransportClient$RpcChannelListener.handleFailure(TransportClient.java:362)
hudi:hudat
org.apache.spark.network.client.TransportClient$StdChannelListener.operationComplete(TransportClient.java:339)
hudi:hudat
io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507)
hudi:hudat
io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:481)
hudi:hudat
io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:420)
hudi:hudat
io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:122)
hudi:hudat
io.netty.channel.AbstractChannel$AbstractUnsafe.safeSetFailure(AbstractChannel.java:987)
hudi:hudat
io.netty.channel.AbstractChannel$AbstractUnsafe.write(AbstractChannel.java:869)
hudi:hudat
io.netty.channel.DefaultChannelPipeline$HeadContext.write(DefaultChannelPipeline.java:1316)
hudi:hudat
io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:738)
hudi:hudat
io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:730)
hudi:hudat
io.netty.channel.AbstractChannelHandlerContext.access$1900(AbstractChannelHandlerContext.java:38)
hudi:hudat
io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.write(AbstractChannelHandlerContext.java:1081)
hudi:hudat
io.netty.channel.AbstractChannelHandlerContext$WriteAndFlushTask.write(AbstractChannelHandlerContext.java:1128)
hudi:hudat
io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.run(AbstractChannelHandlerContext.java:1070)
hudi:hudat
io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
hudi:hudat
io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:403)
hudi:hudat io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:463)
hudi:hudat
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
hudi:hudat
io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
hudi:hudat java.lang.Thread.run(Thread.java:748)
Caused by: java.nio.channels.ClosedChannelException
hudi:hudat
io.netty.channel.AbstractChannel$AbstractUnsafe.write(...)(Unknown Source)
20/09/24 03:12:28 INFO SchedulerExtensionServices: Stopping
SchedulerExtensionServices
(serviceOption=None,e->
services=List(),able->
started=false)_table->
20/09/24 03:12:28 ERROR Utils: Uncaught exception in thread YARN application
state monitor
org.apache.spark.SparkException: Exception thrown in awaitResult:
hudi:hudat
org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:226)
hudi:hudat org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:75)
hudi:hudat
org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend.requestTotalExecutors(CoarseGrainedSchedulerBackend.scala:599)
hudi:hudat
org.apache.spark.scheduler.cluster.YarnSchedulerBackend.stop(YarnSchedulerBackend.scala:103)
hudi:hudat
org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.stop(YarnClientSchedulerBackend.scala:164)
hudi:hudat
org.apache.spark.scheduler.TaskSchedulerImpl.stop(TaskSchedulerImpl.scala:653)
hudi:hudat
org.apache.spark.scheduler.DAGScheduler.stop(DAGScheduler.scala:2194)
hudi:hudat
org.apache.spark.SparkContext$$anonfun$stop$6.apply$mcV$sp(SparkContext.scala:1949)
hudi:hudat org.apache.spark.util.Utils$.tryLogNonFatalError(Utils.scala:1340)
hudi:hudat org.apache.spark.SparkContext.stop(SparkContext.scala:1948)
hudi:hudat
org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend$MonitorThread.run(YarnClientSchedulerBackend.scala:121)
Caused by: java.io.IOException: Failed to send RPC RPC 8528272860118368920
to /172.31.24.65:33218: java.nio.channels.ClosedChannelException
hudi:hudat
org.apache.spark.network.client.TransportClient$RpcChannelListener.handleFailure(TransportClient.java:362)
hudi:hudat
org.apache.spark.network.client.TransportClient$StdChannelListener.operationComplete(TransportClient.java:339)
hudi:hudat
io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507)
hudi:hudat
io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:481)
hudi:hudat
io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:420)
hudi:hudat
io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:122)
hudi:hudat
io.netty.channel.AbstractChannel$AbstractUnsafe.safeSetFailure(AbstractChannel.java:987)
hudi:hudat
io.netty.channel.AbstractChannel$AbstractUnsafe.write(AbstractChannel.java:869)
hudi:hudat
io.netty.channel.DefaultChannelPipeline$HeadContext.write(DefaultChannelPipeline.java:1316)
hudi:hudat
io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:738)
hudi:hudat
io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:730)
hudi:hudat
io.netty.channel.AbstractChannelHandlerContext.access$1900(AbstractChannelHandlerContext.java:38)
hudi:hudat
io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.write(AbstractChannelHandlerContext.java:1081)
hudi:hudat
io.netty.channel.AbstractChannelHandlerContext$WriteAndFlushTask.write(AbstractChannelHandlerContext.java:1128)
hudi:hudat
io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.run(AbstractChannelHandlerContext.java:1070)
hudi:hudat
io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
hudi:hudat
io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:403)
hudi:hudat io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:463)
hudi:hudat
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
hudi:hudat
io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
hudi:hudat java.lang.Thread.run(Thread.java:748)
Caused by: java.nio.channels.ClosedChannelException
hudi:hudat
io.netty.channel.AbstractChannel$AbstractUnsafe.write(...)(Unknown Source)
20/09/24 03:12:28 INFO ShutdownHookManager: Shutdown hook called
20/09/24 03:12:28 INFO ShutdownHookManager: Deleting directory
/mnt/tmp/spark-53e31f61-8f5d-49ff-b507-8089dcca8206
20/09/24 03:12:28 INFO ShutdownHookManager: Deleting directory
/mnt/tmp/spark-b6a65784-1347-4db0-b15e-53056dc7b336
20/09/24 03:12:28 INFO ShutdownHookManager: Deleting directory
/mnt/tmp/spark-53e31f61-8f5d-49ff-b507-8089dcca8206/userFiles-279321df-bb93-4739-b4a0-91cf9d88f6d5
hudi:hudi_trips_table->20/09/24 03:12:28 INFO table.HoodieTableMetaClient:
Loading HoodieTableMetaClient from
s3://jiade-hudi-demo/tmp/hudi/hudi_trips_table/
20/09/24 03:12:28 INFO util.FSUtils: Hadoop Configuration: fs.defaultFS:
[hdfs://ip-172-31-24-65.cn-northwest-1.compute.internal:8020],
Config:[Configuration: core-default.xml, core-site.xml, mapred-default.xml,
mapred-site.xml, yarn-default.xml, yarn-site.xml, hdfs-default.xml,
hdfs-site.xml, emrfs-site.xml], FileSystem:
[com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem@38bdd078]
20/09/24 03:12:28 INFO table.HoodieTableConfig: Loading dataset properties
from s3://jiade-hudi-demo/tmp/hudi/hudi_trips_table/.hoodie/hoodie.properties
20/09/24 03:12:28 INFO s3n.S3NativeFileSystem: Opening
's3://jiade-hudi-demo/tmp/hudi/hudi_trips_table/.hoodie/hoodie.properties' for
reading
20/09/24 03:12:28 INFO table.HoodieTableMetaClient: Finished Loading Table
of type COPY_ON_WRITE from s3://jiade-hudi-demo/tmp/hudi/hudi_trips_table/
Commit 20200924030219 failed to roll back
hudi:hudi_trips_table->commit rollback --commit 20200924030219
20/09/24 03:12:38 INFO timeline.HoodieActiveTimeline: Loaded instants
java.util.stream.ReferencePipeline$Head@3bb9fcbd
Commit 20200924030219 not found in Commits
org.apache.hudi.common.table.timeline.HoodieDefaultTimeline:
[20200924025642__commit__COMPLETED],[20200924025851__commit__COMPLETED],[20200924030059__commit__COMPLETED]
hudi:hudi_trips_table->commits show
20/09/24 03:12:50 INFO s3n.S3NativeFileSystem: Opening
's3://jiade-hudi-demo/tmp/hudi/hudi_trips_table/.hoodie/20200924030059.commit'
for reading
20/09/24 03:12:50 INFO s3n.S3NativeFileSystem: Opening
's3://jiade-hudi-demo/tmp/hudi/hudi_trips_table/.hoodie/20200924025851.commit'
for reading
20/09/24 03:12:50 INFO s3n.S3NativeFileSystem: Opening
's3://jiade-hudi-demo/tmp/hudi/hudi_trips_table/.hoodie/20200924025642.commit'
for reading
╔════════════════╤═════════════════════╤═══════════════════╤═════════════════════╤══════════════════════════╤═══════════════════════╤══════════════════════════════╤══════════════╗
║ CommitTime │ Total Bytes Written │ Total Files Added │ Total Files
Updated │ Total Partitions Written │ Total Records Written │ Total Update
Records Written │ Total Errors ║
╠════════════════╪═════════════════════╪═══════════════════╪═════════════════════╪══════════════════════════╪═══════════════════════╪══════════════════════════════╪══════════════╣
║ 20200924030059 │ 4.6 MB │ 0 │ 1
│ 1 │ 674056 │ 0
│ 0 ║
╟────────────────┼─────────────────────┼───────────────────┼─────────────────────┼──────────────────────────┼───────────────────────┼──────────────────────────────┼──────────────╢
║ 20200924025851 │ 4.7 MB │ 0 │ 1
│ 1 │ 680187 │ 10
│ 0 ║
╟────────────────┼─────────────────────┼───────────────────┼─────────────────────┼──────────────────────────┼───────────────────────┼──────────────────────────────┼──────────────╢
║ 20200924025642 │ 13.8 MB │ 3 │ 0
│ 1 │ 2000000 │ 0
│ 0 ║
╚════════════════╧═════════════════════╧═══════════════════╧═════════════════════╧══════════════════════════╧═══════════════════════╧══════════════════════════════╧══════════════╝
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
