georgepap9808 commented on issue #10262:
URL: https://github.com/apache/hudi/issues/10262#issuecomment-1848430992
> @georgepap9808 I just ran this command `mvn clean package
-Pintegration-tests -DskipTests` on latest master and it worked fine for me.
Can you please try once. Thanks.
@ad1happy2go thanks for the reply. I tried it again but I had the same error
with the -Pintegration-tests. But as @CodyPin said, I switched to the latest
release-0.14.0 and it worked.
Regarding the issue with adhoc-2 we had, after running
```
spark-submit \
--class org.apache.hudi.utilities.streamer.HoodieStreamer
$HUDI_UTILITIES_BUNDLE \
--table-type COPY_ON_WRITE \
--source-class org.apache.hudi.utilities.sources.JsonKafkaSource \
--source-ordering-field ts \
--target-base-path /user/hive/warehouse/stock_ticks_cow \
--target-table stock_ticks_cow --props
/var/demo/config/kafka-source.properties \
--schemaprovider-class
org.apache.hudi.utilities.schema.FilebasedSchemaProvider
```
I don't have the error I had before
```
"java.lang.ClassNotFoundException:
org.apache.hudi.utilities.streamer.HoodieStreamer"
```
Now this is the error
```
root@adhoc-2:/opt# spark-submit \
> --class org.apache.hudi.utilities.streamer.HoodieStreamer
$HUDI_UTILITIES_BUNDLE \
> --table-type COPY_ON_WRITE \
> --source-class org.apache.hudi.utilities.sources.JsonKafkaSource \
> --source-ordering-field ts \
> --target-base-path /user/hive/warehouse/stock_ticks_cow \
> --target-table stock_ticks_cow --props
/var/demo/config/kafka-source.properties \
> --schemaprovider-class
org.apache.hudi.utilities.schema.FilebasedSchemaProvider
23/12/09 14:53:45 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
23/12/09 14:53:46 WARN streamer.SchedulerConfGenerator: Job Scheduling
Configs will not be in effect as spark.scheduler.mode is not set to FAIR at
instantiation time. Continuing without scheduling configs
23/12/09 14:53:46 INFO spark.SparkContext: Running Spark version 2.4.4
23/12/09 14:53:46 INFO spark.SparkContext: Submitted application:
streamer-stock_ticks_cow
23/12/09 14:53:46 INFO spark.SecurityManager: Changing view acls to: root
23/12/09 14:53:46 INFO spark.SecurityManager: Changing modify acls to: root
23/12/09 14:53:46 INFO spark.SecurityManager: Changing view acls groups to:
23/12/09 14:53:46 INFO spark.SecurityManager: Changing modify acls groups to:
23/12/09 14:53:46 INFO spark.SecurityManager: SecurityManager:
authentication disabled; ui acls disabled; users with view permissions:
Set(root); groups with view permissions: Set(); users with modify permissions:
Set(root); groups with modify permissions: Set()
23/12/09 14:53:46 INFO Configuration.deprecation:
mapred.output.compression.codec is deprecated. Instead, use
mapreduce.output.fileoutputformat.compress.codec
23/12/09 14:53:46 INFO Configuration.deprecation: mapred.output.compress is
deprecated. Instead, use mapreduce.output.fileoutputformat.compress
23/12/09 14:53:46 INFO Configuration.deprecation:
mapred.output.compression.type is deprecated. Instead, use
mapreduce.output.fileoutputformat.compress.type
23/12/09 14:53:46 INFO util.Utils: Successfully started service
'sparkDriver' on port 39795.
23/12/09 14:53:46 INFO spark.SparkEnv: Registering MapOutputTracker
23/12/09 14:53:46 INFO spark.SparkEnv: Registering BlockManagerMaster
23/12/09 14:53:46 INFO storage.BlockManagerMasterEndpoint: Using
org.apache.spark.storage.DefaultTopologyMapper for getting topology information
23/12/09 14:53:46 INFO storage.BlockManagerMasterEndpoint:
BlockManagerMasterEndpoint up
23/12/09 14:53:46 INFO storage.DiskBlockManager: Created local directory at
/tmp/blockmgr-fd13e125-9ce4-4bb1-b60d-0b2544e6606e
23/12/09 14:53:46 INFO memory.MemoryStore: MemoryStore started with capacity
366.3 MB
23/12/09 14:53:46 INFO spark.SparkEnv: Registering OutputCommitCoordinator
23/12/09 14:53:46 INFO util.log: Logging initialized @1747ms
23/12/09 14:53:46 INFO server.Server: jetty-9.3.z-SNAPSHOT, build timestamp:
unknown, git hash: unknown
23/12/09 14:53:46 INFO server.Server: Started @1806ms
23/12/09 14:53:46 INFO server.AbstractConnector: Started
ServerConnector@26f3d90c{HTTP/1.1,[http/1.1]}{0.0.0.0:8090}
23/12/09 14:53:46 INFO util.Utils: Successfully started service 'SparkUI' on
port 8090.
23/12/09 14:53:46 INFO handler.ContextHandler: Started
o.s.j.s.ServletContextHandler@6c2f1700{/jobs,null,AVAILABLE,@Spark}
23/12/09 14:53:46 INFO handler.ContextHandler: Started
o.s.j.s.ServletContextHandler@7d2a6eac{/jobs/json,null,AVAILABLE,@Spark}
23/12/09 14:53:46 INFO handler.ContextHandler: Started
o.s.j.s.ServletContextHandler@18ca3c62{/jobs/job,null,AVAILABLE,@Spark}
23/12/09 14:53:46 INFO handler.ContextHandler: Started
o.s.j.s.ServletContextHandler@44d70181{/jobs/job/json,null,AVAILABLE,@Spark}
23/12/09 14:53:46 INFO handler.ContextHandler: Started
o.s.j.s.ServletContextHandler@6aa648b9{/stages,null,AVAILABLE,@Spark}
23/12/09 14:53:46 INFO handler.ContextHandler: Started
o.s.j.s.ServletContextHandler@23c650a3{/stages/json,null,AVAILABLE,@Spark}
23/12/09 14:53:46 INFO handler.ContextHandler: Started
o.s.j.s.ServletContextHandler@742d4e15{/stages/stage,null,AVAILABLE,@Spark}
23/12/09 14:53:46 INFO handler.ContextHandler: Started
o.s.j.s.ServletContextHandler@4163f1cd{/stages/stage/json,null,AVAILABLE,@Spark}
23/12/09 14:53:46 INFO handler.ContextHandler: Started
o.s.j.s.ServletContextHandler@5fa05212{/stages/pool,null,AVAILABLE,@Spark}
23/12/09 14:53:46 INFO handler.ContextHandler: Started
o.s.j.s.ServletContextHandler@3e681bc{/stages/pool/json,null,AVAILABLE,@Spark}
23/12/09 14:53:46 INFO handler.ContextHandler: Started
o.s.j.s.ServletContextHandler@5c09d180{/storage,null,AVAILABLE,@Spark}
23/12/09 14:53:46 INFO handler.ContextHandler: Started
o.s.j.s.ServletContextHandler@23aae55{/storage/json,null,AVAILABLE,@Spark}
23/12/09 14:53:46 INFO handler.ContextHandler: Started
o.s.j.s.ServletContextHandler@5f574cc2{/storage/rdd,null,AVAILABLE,@Spark}
23/12/09 14:53:46 INFO handler.ContextHandler: Started
o.s.j.s.ServletContextHandler@680bddf5{/storage/rdd/json,null,AVAILABLE,@Spark}
23/12/09 14:53:46 INFO handler.ContextHandler: Started
o.s.j.s.ServletContextHandler@7a9c84a5{/environment,null,AVAILABLE,@Spark}
23/12/09 14:53:46 INFO handler.ContextHandler: Started
o.s.j.s.ServletContextHandler@2d83c5a5{/environment/json,null,AVAILABLE,@Spark}
23/12/09 14:53:46 INFO handler.ContextHandler: Started
o.s.j.s.ServletContextHandler@48d7ad8b{/executors,null,AVAILABLE,@Spark}
23/12/09 14:53:46 INFO handler.ContextHandler: Started
o.s.j.s.ServletContextHandler@7e053511{/executors/json,null,AVAILABLE,@Spark}
23/12/09 14:53:46 INFO handler.ContextHandler: Started
o.s.j.s.ServletContextHandler@60222fd8{/executors/threadDump,null,AVAILABLE,@Spark}
23/12/09 14:53:46 INFO handler.ContextHandler: Started
o.s.j.s.ServletContextHandler@53bf7094{/executors/threadDump/json,null,AVAILABLE,@Spark}
23/12/09 14:53:46 INFO handler.ContextHandler: Started
o.s.j.s.ServletContextHandler@26f1249d{/static,null,AVAILABLE,@Spark}
23/12/09 14:53:46 INFO handler.ContextHandler: Started
o.s.j.s.ServletContextHandler@7af707e0{/,null,AVAILABLE,@Spark}
23/12/09 14:53:46 INFO handler.ContextHandler: Started
o.s.j.s.ServletContextHandler@737edcfa{/api,null,AVAILABLE,@Spark}
23/12/09 14:53:46 INFO handler.ContextHandler: Started
o.s.j.s.ServletContextHandler@3359c978{/jobs/job/kill,null,AVAILABLE,@Spark}
23/12/09 14:53:46 INFO handler.ContextHandler: Started
o.s.j.s.ServletContextHandler@7ba63fe5{/stages/stage/kill,null,AVAILABLE,@Spark}
23/12/09 14:53:46 INFO ui.SparkUI: Bound SparkUI to 0.0.0.0, and started at
http://adhoc-2:8090
23/12/09 14:53:46 INFO spark.SparkContext: Added JAR
file:/var/hoodie/ws/docker/hoodie/hadoop/hive_base/target/hoodie-utilities.jar
at spark://adhoc-2:39795/jars/hoodie-utilities.jar with timestamp 1702133626668
23/12/09 14:53:46 INFO executor.Executor: Starting executor ID driver on
host localhost
23/12/09 14:53:46 INFO util.Utils: Successfully started service
'org.apache.spark.network.netty.NettyBlockTransferService' on port 44703.
23/12/09 14:53:46 INFO netty.NettyBlockTransferService: Server created on
adhoc-2:44703
23/12/09 14:53:46 INFO storage.BlockManager: Using
org.apache.spark.storage.RandomBlockReplicationPolicy for block replication
policy
23/12/09 14:53:46 INFO storage.BlockManagerMaster: Registering BlockManager
BlockManagerId(driver, adhoc-2, 44703, None)
23/12/09 14:53:46 INFO storage.BlockManagerMasterEndpoint: Registering block
manager adhoc-2:44703 with 366.3 MB RAM, BlockManagerId(driver, adhoc-2, 44703,
None)
23/12/09 14:53:46 INFO storage.BlockManagerMaster: Registered BlockManager
BlockManagerId(driver, adhoc-2, 44703, None)
23/12/09 14:53:46 INFO storage.BlockManager: Initialized BlockManager:
BlockManagerId(driver, adhoc-2, 44703, None)
23/12/09 14:53:46 INFO handler.ContextHandler: Started
o.s.j.s.ServletContextHandler@31ee96f4{/metrics/json,null,AVAILABLE,@Spark}
23/12/09 14:53:47 WARN config.DFSPropertiesConfiguration: Cannot find
HUDI_CONF_DIR, please set it as the dir of hudi-defaults.conf
23/12/09 14:53:47 WARN config.DFSPropertiesConfiguration: Properties file
file:/etc/hudi/conf/hudi-defaults.conf not found. Ignoring to load props file
23/12/09 14:53:47 INFO server.AbstractConnector: Stopped
Spark@26f3d90c{HTTP/1.1,[http/1.1]}{0.0.0.0:8090}
23/12/09 14:53:47 INFO ui.SparkUI: Stopped Spark web UI at
http://adhoc-2:8090
23/12/09 14:53:47 INFO spark.MapOutputTrackerMasterEndpoint:
MapOutputTrackerMasterEndpoint stopped!
23/12/09 14:53:47 INFO memory.MemoryStore: MemoryStore cleared
23/12/09 14:53:47 INFO storage.BlockManager: BlockManager stopped
23/12/09 14:53:47 INFO storage.BlockManagerMaster: BlockManagerMaster stopped
23/12/09 14:53:47 INFO
scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint:
OutputCommitCoordinator stopped!
23/12/09 14:53:47 INFO spark.SparkContext: Successfully stopped SparkContext
Exception in thread "main" java.lang.NoSuchMethodError:
scala.Function1.$init$(Lscala/Function1;)V
at
org.apache.spark.sql.hudi.HoodieSparkSessionExtension.<init>(HoodieSparkSessionExtension.scala:28)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at java.lang.Class.newInstance(Class.java:442)
at
org.apache.spark.sql.SparkSession$Builder.liftedTree1$1(SparkSession.scala:945)
at
org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:943)
at
org.apache.spark.sql.SQLContext$.getOrCreate(SQLContext.scala:1066)
at org.apache.spark.sql.SQLContext.getOrCreate(SQLContext.scala)
at
org.apache.hudi.client.common.HoodieSparkEngineContext.<init>(HoodieSparkEngineContext.java:72)
at
org.apache.hudi.utilities.streamer.HoodieStreamer.<init>(HoodieStreamer.java:157)
at
org.apache.hudi.utilities.streamer.HoodieStreamer.<init>(HoodieStreamer.java:131)
at
org.apache.hudi.utilities.streamer.HoodieStreamer.main(HoodieStreamer.java:584)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at
org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:845)
at
org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:161)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:184)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
at
org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:920)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:929)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
23/12/09 14:53:47 INFO util.ShutdownHookManager: Shutdown hook called
23/12/09 14:53:47 INFO util.ShutdownHookManager: Deleting directory
/tmp/spark-57b09e97-9565-46da-ad79-8d65a2267be1
23/12/09 14:53:47 INFO util.ShutdownHookManager: Deleting directory
/tmp/spark-6a67089d-1b9e-44f7-8230-6d6055d257c2
root@adhoc-2:/opt#
```
Any ideas where this comes from?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
