Raja10D commented on issue #4752:
URL: https://github.com/apache/hop/issues/4752#issuecomment-2572391323
I have created a simple pipeline -> hop_trans2.hpl. After that I have
generated a fatjar using generate fatjar in hopgui and then I have exported a
metadata.json file using export metadata.json in hopgui.
After that I have used Spark submit to run my pipeline (In local runner it
is running fine)
My spark-submit command
spark-submit \
--master local[4] \
--class org.apache.hop.beam.run.MainBeam \
--driver-java-options
'-DPROJECT_HOME=/home/decoders/Downloads/apache-hop-client-2.11.0/hop/config/projects/default'
\
/home/decoders/Downloads/apache-hop-client-2.11.0/fatjar.jar \
--pipeline=/home/decoders/Downloads/apache-hop-client-2.11.0/hop/config/projects/default/pipelines/hop_trans2.hpl
\
--metadata=/home/decoders/Downloads/apache-hop-client-2.11.0/hop/config/projects/default/metadata.json
\
Spark
My versions:
My Hop version: 2.11.0
Apache Beam Version: 2.61.0
Apache Spark Version: 3.5.4
java 17
Error :
25/01/06 11:52:34 WARN Utils: Your hostname, decoders-Latitude-7480 resolves
to a loopback address: 127.0.1.1; using 192.125.1.140 instead (on interface
enp0s31f6)
25/01/06 11:52:34 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to
another address
>>>>>> Initializing Hop
Hop configuration file not found, not serializing:
/home/decoders/config/hop-config.json
Error running Beam pipeline:
Error reading metadata from JSON
Error find metadata class for key null
The metadata plugin for key null could not be found in the plugin registry
org.apache.hop.core.exception.HopException:
Error reading metadata from JSON
Error find metadata class for key null
The metadata plugin for key null could not be found in the plugin registry
at
org.apache.hop.core.metadata.SerializableMetadataProvider.<init>(SerializableMetadataProvider.java:143)
at org.apache.hop.beam.run.MainBeam.main(MainBeam.java:101)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:569)
at
org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at
org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:1034)
at
org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:199)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:222)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:91)
at
org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1125)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1134)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: org.apache.hop.core.exception.HopException:
Error find metadata class for key null
The metadata plugin for key null could not be found in the plugin registry
at
org.apache.hop.metadata.serializer.BaseMetadataProvider.getMetadataClassForKey(BaseMetadataProvider.java:69)
at
org.apache.hop.core.metadata.SerializableMetadataProvider.<init>(SerializableMetadataProvider.java:122)
... 13 more
Caused by: org.apache.hop.core.exception.HopException:
The metadata plugin for key null could not be found in the plugin registry
at
org.apache.hop.metadata.serializer.BaseMetadataProvider.getMetadataClassForKey(BaseMetadataProvider.java:61)
... 14 more
25/01/06 11:52:47 INFO ShutdownHookManager: Shutdown hook called
25/01/06 11:52:47 INFO ShutdownHookManager: Deleting directory
/tmp/spark-05fb3578-b82c-4c9a-9fd2-f549997cd73e
If You know any ways to rectify this error means say it otherwise if you can
run hop pipelines with Beam Spark Runner means tell me the versions you are
using, I will try in that version.
Our team is working on this for 10 days, we couldn't get it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
