arunkumarArunachalam99 opened a new issue, #4836: URL: https://github.com/apache/hop/issues/4836
I am writing to report an issue we are encountering when running a Hop pipeline on a Spark Standalone cluster. Our Spark-Submit command works perfectly in a local environment with the following configuration: **spark-submit --master local[4] --class org.apache.hop.beam.run.MainBeam --driver-java-options '-DPROJECT_HOME=/home/decoders/hop_test_2_8_0_backup/hop/config/projects/default' /home/decoders/hop_test_2_8_0_backup/hop/fat-jar.jar /home/decoders/hop_test_2_8_0_backup/hop/Creating_5_payoutAmt_source_Dec23.hpl /home/decoders/hop_test_2_8_0_backup/hop/metadata.json beam_runner_spark** However, when we replace local[4] with the Spark Standalone cluster URL (spark://10d154:7077), the pipeline fails to run in our real-time production environment. We have verified that all necessary dependencies are included in the fat-jar.jar and reviewed the configuration settings, but the issue persists. Could you please provide guidance on any additional files or settings that might be required to run the Hop pipeline successfully on a Spark Standalone cluster? Thank you for your assistance. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
