Raja10D commented on issue #4752:
URL: https://github.com/apache/hop/issues/4752#issuecomment-2572816889
**Thanks for the reply, I've changed the command, now also recieving error**
Command:
bin/spark-submit \
--master spark://decoders-Latitude-7480:7077 \
--class org.apache.hop.beam.run.MainBeam \
--driver-java-options
'-DPROJECT_HOME=/home/decoders/Downloads/apache-hop-client-2.11.0/hop/config/projects/default'
\
/home/decoders/Downloads/apache-hop-client-2.11.0/fatjar.jar \
/home/decoders/Downloads/apache-hop-client-2.11.0/hop/config/projects/default/pipelines/hop_trans2.hpl
\
/home/decoders/Downloads/apache-hop-client-2.11.0/hop/config/projects/default/metadata.json
\
Beam_Spark_pipeline_engine
Error:
decoders@decoders-Latitude-7480:~/Downloads/spark-3.5.4-bin-hadoop3$
bin/spark-submit \
--master spark://decoders-Latitude-7480:7077 \
--class org.apache.hop.beam.run.MainBeam \
--driver-java-options
'-DPROJECT_HOME=/home/decoders/Downloads/apache-hop-client-2.11.0/hop/config/projects/default'
\
/home/decoders/Downloads/apache-hop-client-2.11.0/fatjar.jar \
/home/decoders/Downloads/apache-hop-client-2.11.0/hop/config/projects/default/pipelines/hop_trans2.hpl
\
/home/decoders/Downloads/apache-hop-client-2.11.0/hop/config/projects/default/metadata.json
\
Beam_Spark_pipeline_engine
25/01/06 15:55:27 WARN Utils: Your hostname, decoders-Latitude-7480 resolves
to a loopback address: 127.0.1.1; using 192.125.1.140 instead (on interface
enp0s31f6)
25/01/06 15:55:27 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to
another address
>>>>>> Initializing Hop
Hop configuration file not found, not serializing:
/home/decoders/Downloads/spark-3.5.4-bin-hadoop3/config/hop-config.json
Argument 1 : Pipeline filename (.hpl) :
/home/decoders/Downloads/apache-hop-client-2.11.0/hop/config/projects/default/pipelines/hop_trans2.hpl
Argument 2 : Environment state filename: (.json) :
/home/decoders/Downloads/apache-hop-client-2.11.0/hop/config/projects/default/metadata.json
Argument 3 : Pipeline run configuration : Beam_Spark_pipeline_engine
>>>>>> Loading pipeline metadata
>>>>>> Building Apache Beam Pipeline...
>>>>>> Pipeline executing starting...
Exception in thread "main" java.lang.NoClassDefFoundError:
org/apache/beam/sdk/options/PipelineOptions
at java.base/java.lang.ClassLoader.defineClass1(Native Method)
at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1017)
at
java.base/java.security.SecureClassLoader.defineClass(SecureClassLoader.java:150)
at
java.base/jdk.internal.loader.BuiltinClassLoader.defineClass(BuiltinClassLoader.java:862)
at
java.base/jdk.internal.loader.BuiltinClassLoader.findClassOnClassPathOrNull(BuiltinClassLoader.java:760)
at
java.base/jdk.internal.loader.BuiltinClassLoader.loadClassOrNull(BuiltinClassLoader.java:681)
at
java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:639)
at
java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:525)
at java.base/java.lang.ClassLoader.defineClass1(Native Method)
at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1017)
at
java.base/java.security.SecureClassLoader.defineClass(SecureClassLoader.java:150)
at
java.base/jdk.internal.loader.BuiltinClassLoader.defineClass(BuiltinClassLoader.java:862)
at
java.base/jdk.internal.loader.BuiltinClassLoader.findClassOnClassPathOrNull(BuiltinClassLoader.java:760)
at
java.base/jdk.internal.loader.BuiltinClassLoader.loadClassOrNull(BuiltinClassLoader.java:681)
at
java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:639)
at
java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:579)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:525)
at
org.apache.hop.beam.engines.spark.BeamSparkPipelineRunConfiguration.getPipelineOptions(BeamSparkPipelineRunConfiguration.java:213)
at
org.apache.hop.beam.pipeline.HopPipelineMetaToBeamPipelineConverter.<init>(HopPipelineMetaToBeamPipelineConverter.java:119)
at
org.apache.hop.beam.engines.BeamPipelineEngine.prepareExecution(BeamPipelineEngine.java:263)
at
org.apache.hop.beam.engines.BeamPipelineEngine.execute(BeamPipelineEngine.java:633)
at org.apache.hop.beam.run.MainBeam.main(MainBeam.java:152)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:569)
at
org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at
org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:1034)
at
org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:199)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:222)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:91)
at
org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1125)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1134)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException:
org.apache.beam.sdk.options.PipelineOptions
at
java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641)
at
java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:525)
... 36 more
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
