alberttwong commented on issue #11797:
URL: https://github.com/apache/hudi/issues/11797#issuecomment-2311107717
confirmed.
```
albert@Alberts-MBP docker % docker exec -it adhoc-2 /bin/bash
root@adhoc-2:/opt# spark-submit \
> --class org.apache.hudi.utilities.streamer.HoodieStreamer
$HUDI_UTILITIES_BUNDLE \
> --table-type COPY_ON_WRITE \
> --source-class org.apache.hudi.utilities.sources.JsonKafkaSource \
> --source-ordering-field ts \
> --target-base-path /user/hive/warehouse/stock_ticks_cow \
> --target-table stock_ticks_cow --props
/var/demo/config/kafka-source.properties \
> --schemaprovider-class
org.apache.hudi.utilities.schema.FilebasedSchemaProvider
24/08/26 21:12:04 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
24/08/26 21:12:04 WARN streamer.SchedulerConfGenerator: Job Scheduling
Configs will not be in effect as spark.scheduler.mode is not set to FAIR at
instantiation time. Continuing without scheduling configs
24/08/26 21:12:04 INFO spark.SparkContext: Running Spark version 2.4.4
24/08/26 21:12:04 INFO spark.SparkContext: Submitted application:
streamer-stock_ticks_cow
24/08/26 21:12:04 INFO spark.SecurityManager: Changing view acls to: root
24/08/26 21:12:04 INFO spark.SecurityManager: Changing modify acls to: root
24/08/26 21:12:04 INFO spark.SecurityManager: Changing view acls groups to:
24/08/26 21:12:04 INFO spark.SecurityManager: Changing modify acls groups to:
24/08/26 21:12:04 INFO spark.SecurityManager: SecurityManager:
authentication disabled; ui acls disabled; users with view permissions:
Set(root); groups with view permissions: Set(); users with modify permissions:
Set(root); groups with modify permissions: Set()
24/08/26 21:12:04 INFO Configuration.deprecation:
mapred.output.compression.codec is deprecated. Instead, use
mapreduce.output.fileoutputformat.compress.codec
24/08/26 21:12:04 INFO Configuration.deprecation: mapred.output.compress is
deprecated. Instead, use mapreduce.output.fileoutputformat.compress
24/08/26 21:12:04 INFO Configuration.deprecation:
mapred.output.compression.type is deprecated. Instead, use
mapreduce.output.fileoutputformat.compress.type
24/08/26 21:12:04 INFO util.Utils: Successfully started service
'sparkDriver' on port 42985.
24/08/26 21:12:04 INFO spark.SparkEnv: Registering MapOutputTracker
24/08/26 21:12:04 INFO spark.SparkEnv: Registering BlockManagerMaster
24/08/26 21:12:04 INFO storage.BlockManagerMasterEndpoint: Using
org.apache.spark.storage.DefaultTopologyMapper for getting topology information
24/08/26 21:12:04 INFO storage.BlockManagerMasterEndpoint:
BlockManagerMasterEndpoint up
24/08/26 21:12:04 INFO storage.DiskBlockManager: Created local directory at
/tmp/blockmgr-f0cbf1d6-2863-462c-9d80-530876684a5a
24/08/26 21:12:04 INFO memory.MemoryStore: MemoryStore started with capacity
366.3 MB
24/08/26 21:12:04 INFO spark.SparkEnv: Registering OutputCommitCoordinator
24/08/26 21:12:04 INFO util.log: Logging initialized @736ms
24/08/26 21:12:04 INFO server.Server: jetty-9.3.z-SNAPSHOT, build timestamp:
unknown, git hash: unknown
24/08/26 21:12:04 INFO server.Server: Started @761ms
24/08/26 21:12:04 INFO server.AbstractConnector: Started
ServerConnector@2c1dc8e{HTTP/1.1,[http/1.1]}{0.0.0.0:8090}
24/08/26 21:12:04 INFO util.Utils: Successfully started service 'SparkUI' on
port 8090.
24/08/26 21:12:04 INFO handler.ContextHandler: Started
o.s.j.s.ServletContextHandler@5fa47fea{/jobs,null,AVAILABLE,@Spark}
24/08/26 21:12:04 INFO handler.ContextHandler: Started
o.s.j.s.ServletContextHandler@4e406694{/jobs/json,null,AVAILABLE,@Spark}
24/08/26 21:12:04 INFO handler.ContextHandler: Started
o.s.j.s.ServletContextHandler@5ab9b447{/jobs/job,null,AVAILABLE,@Spark}
24/08/26 21:12:04 INFO handler.ContextHandler: Started
o.s.j.s.ServletContextHandler@4f8caaf3{/jobs/job/json,null,AVAILABLE,@Spark}
24/08/26 21:12:04 INFO handler.ContextHandler: Started
o.s.j.s.ServletContextHandler@2b50150{/stages,null,AVAILABLE,@Spark}
24/08/26 21:12:04 INFO handler.ContextHandler: Started
o.s.j.s.ServletContextHandler@15b986cd{/stages/json,null,AVAILABLE,@Spark}
24/08/26 21:12:04 INFO handler.ContextHandler: Started
o.s.j.s.ServletContextHandler@6bb7cce7{/stages/stage,null,AVAILABLE,@Spark}
24/08/26 21:12:04 INFO handler.ContextHandler: Started
o.s.j.s.ServletContextHandler@328572f0{/stages/stage/json,null,AVAILABLE,@Spark}
24/08/26 21:12:04 INFO handler.ContextHandler: Started
o.s.j.s.ServletContextHandler@678040b3{/stages/pool,null,AVAILABLE,@Spark}
24/08/26 21:12:04 INFO handler.ContextHandler: Started
o.s.j.s.ServletContextHandler@17f460bb{/stages/pool/json,null,AVAILABLE,@Spark}
24/08/26 21:12:04 INFO handler.ContextHandler: Started
o.s.j.s.ServletContextHandler@64a1923a{/storage,null,AVAILABLE,@Spark}
24/08/26 21:12:04 INFO handler.ContextHandler: Started
o.s.j.s.ServletContextHandler@7d2a6eac{/storage/json,null,AVAILABLE,@Spark}
24/08/26 21:12:04 INFO handler.ContextHandler: Started
o.s.j.s.ServletContextHandler@18ca3c62{/storage/rdd,null,AVAILABLE,@Spark}
24/08/26 21:12:04 INFO handler.ContextHandler: Started
o.s.j.s.ServletContextHandler@2c0f7678{/storage/rdd/json,null,AVAILABLE,@Spark}
24/08/26 21:12:04 INFO handler.ContextHandler: Started
o.s.j.s.ServletContextHandler@44d70181{/environment,null,AVAILABLE,@Spark}
24/08/26 21:12:04 INFO handler.ContextHandler: Started
o.s.j.s.ServletContextHandler@6aa648b9{/environment/json,null,AVAILABLE,@Spark}
24/08/26 21:12:04 INFO handler.ContextHandler: Started
o.s.j.s.ServletContextHandler@23c650a3{/executors,null,AVAILABLE,@Spark}
24/08/26 21:12:04 INFO handler.ContextHandler: Started
o.s.j.s.ServletContextHandler@742d4e15{/executors/json,null,AVAILABLE,@Spark}
24/08/26 21:12:04 INFO handler.ContextHandler: Started
o.s.j.s.ServletContextHandler@88a8218{/executors/threadDump,null,AVAILABLE,@Spark}
24/08/26 21:12:04 INFO handler.ContextHandler: Started
o.s.j.s.ServletContextHandler@50b1f030{/executors/threadDump/json,null,AVAILABLE,@Spark}
24/08/26 21:12:04 INFO handler.ContextHandler: Started
o.s.j.s.ServletContextHandler@4163f1cd{/static,null,AVAILABLE,@Spark}
24/08/26 21:12:04 INFO handler.ContextHandler: Started
o.s.j.s.ServletContextHandler@1b1637e1{/,null,AVAILABLE,@Spark}
24/08/26 21:12:04 INFO handler.ContextHandler: Started
o.s.j.s.ServletContextHandler@18151a14{/api,null,AVAILABLE,@Spark}
24/08/26 21:12:04 INFO handler.ContextHandler: Started
o.s.j.s.ServletContextHandler@ceb4bd2{/jobs/job/kill,null,AVAILABLE,@Spark}
24/08/26 21:12:04 INFO handler.ContextHandler: Started
o.s.j.s.ServletContextHandler@60297f36{/stages/stage/kill,null,AVAILABLE,@Spark}
24/08/26 21:12:04 INFO ui.SparkUI: Bound SparkUI to 0.0.0.0, and started at
http://adhoc-2:8090
24/08/26 21:12:04 INFO spark.SparkContext: Added JAR
file:/var/hoodie/ws/docker/hoodie/hadoop/hive_base/target/hoodie-utilities.jar
at spark://adhoc-2:42985/jars/hoodie-utilities.jar with timestamp 1724706724533
24/08/26 21:12:04 INFO executor.Executor: Starting executor ID driver on
host localhost
24/08/26 21:12:04 INFO util.Utils: Successfully started service
'org.apache.spark.network.netty.NettyBlockTransferService' on port 42359.
24/08/26 21:12:04 INFO netty.NettyBlockTransferService: Server created on
adhoc-2:42359
24/08/26 21:12:04 INFO storage.BlockManager: Using
org.apache.spark.storage.RandomBlockReplicationPolicy for block replication
policy
24/08/26 21:12:04 INFO storage.BlockManagerMaster: Registering BlockManager
BlockManagerId(driver, adhoc-2, 42359, None)
24/08/26 21:12:04 INFO storage.BlockManagerMasterEndpoint: Registering block
manager adhoc-2:42359 with 366.3 MB RAM, BlockManagerId(driver, adhoc-2, 42359,
None)
24/08/26 21:12:04 INFO storage.BlockManagerMaster: Registered BlockManager
BlockManagerId(driver, adhoc-2, 42359, None)
24/08/26 21:12:04 INFO storage.BlockManager: Initialized BlockManager:
BlockManagerId(driver, adhoc-2, 42359, None)
24/08/26 21:12:04 INFO handler.ContextHandler: Started
o.s.j.s.ServletContextHandler@1dcca8d3{/metrics/json,null,AVAILABLE,@Spark}
24/08/26 21:12:04 WARN config.DFSPropertiesConfiguration: Cannot find
HUDI_CONF_DIR, please set it as the dir of hudi-defaults.conf
24/08/26 21:12:04 WARN config.DFSPropertiesConfiguration: Properties file
file:/etc/hudi/conf/hudi-defaults.conf not found. Ignoring to load props file
24/08/26 21:12:05 INFO server.AbstractConnector: Stopped
Spark@2c1dc8e{HTTP/1.1,[http/1.1]}{0.0.0.0:8090}
24/08/26 21:12:05 INFO ui.SparkUI: Stopped Spark web UI at
http://adhoc-2:8090
24/08/26 21:12:05 INFO spark.MapOutputTrackerMasterEndpoint:
MapOutputTrackerMasterEndpoint stopped!
24/08/26 21:12:05 INFO memory.MemoryStore: MemoryStore cleared
24/08/26 21:12:05 INFO storage.BlockManager: BlockManager stopped
24/08/26 21:12:05 INFO storage.BlockManagerMaster: BlockManagerMaster stopped
24/08/26 21:12:05 INFO
scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint:
OutputCommitCoordinator stopped!
24/08/26 21:12:05 INFO spark.SparkContext: Successfully stopped SparkContext
Exception in thread "main" java.lang.NoSuchMethodError:
scala.Function1.$init$(Lscala/Function1;)V
at
org.apache.spark.sql.hudi.HoodieSparkSessionExtension.<init>(HoodieSparkSessionExtension.scala:28)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at java.lang.Class.newInstance(Class.java:442)
at
org.apache.spark.sql.SparkSession$Builder.liftedTree1$1(SparkSession.scala:945)
at
org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:943)
at org.apache.spark.sql.SQLContext$.getOrCreate(SQLContext.scala:1066)
at org.apache.spark.sql.SQLContext.getOrCreate(SQLContext.scala)
at
org.apache.hudi.client.common.HoodieSparkEngineContext.<init>(HoodieSparkEngineContext.java:72)
at
org.apache.hudi.utilities.streamer.HoodieStreamer.<init>(HoodieStreamer.java:166)
at
org.apache.hudi.utilities.streamer.HoodieStreamer.<init>(HoodieStreamer.java:150)
at
org.apache.hudi.utilities.streamer.HoodieStreamer.<init>(HoodieStreamer.java:136)
at
org.apache.hudi.utilities.streamer.HoodieStreamer.main(HoodieStreamer.java:606)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at
org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:845)
at
org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:161)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:184)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
at
org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:920)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:929)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
24/08/26 21:12:05 INFO util.ShutdownHookManager: Shutdown hook called
24/08/26 21:12:05 INFO util.ShutdownHookManager: Deleting directory
/tmp/spark-015224c9-692d-47a5-b2e4-45d649ae189a
24/08/26 21:12:05 INFO util.ShutdownHookManager: Deleting directory
/tmp/spark-4159ae0c-5c40-4dc6-aaf1-b4d54f578ee4
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
