Hi Rajaganesh, Class-not-found exceptions are usually seen if you didn't build a fat jar or if there's a mismatch between your Spark version and the one included in Hop. It's best to follow the Getting Started With Beam page in the documentation. Good luck!
Matt Op do 2 jan 2025 om 12:12 schreef Raja Ganesh < rajaganesh18beemepco...@gmail.com>: > Dear Apache Hop Team, > > I am reaching out to seek your assistance with an issue I am encountering > while running a simple Hop pipeline using the Beam Spark pipeline engine. > The pipeline executes without any issues when using the default local > runner, but it fails when I switch to the Beam Spark engine. > > Below are the details of my setup: > > - *Apache Beam Version:* 2.61.0 > - *Apache Spark Version:* 3.5.1 > - *Apache Hop Version:* apache-hop-client-2.11.0 > > The error logs are as follows: > > yaml > Copy code > 2025/01/02 15:30:53 - Hop - Pipeline opened. 2025/01/02 15:30:53 - Hop - > Launching pipeline [hop_trans2]... 2025/01/02 15:30:53 - Hop - Started the > pipeline execution. java.lang.NoClassDefFoundError: > org/apache/spark/metrics/source/Source > at java.base/java.lang.Class.getDeclaredMethods0(Native Method) > ... 2025/01/02 15:30:53 - Hop - ERROR: hop_trans2: preparing pipeline > execution failed 2025/01/02 15:30:53 - Hop - ERROR: > org.apache.hop.core.exception.HopException: 2025/01/02 15:30:53 - Hop - > Error preparing remote pipeline 2025/01/02 15:30:53 - Hop - Error converting > Hop pipeline to Beam 2025/01/02 15:30:53 - Hop - 2025/01/02 15:30:53 - Hop > - Caused by: java.lang.NoClassDefFoundError: > org/apache/spark/metrics/source/Source 2025/01/02 15:30:53 - Hop - at > java.base/java.lang.Class.getDeclaredMethods0(Native Method) > ... > > It appears that the issue is related to the > org.apache.spark.metrics.source.Source class not being found. > > I have also attached the following: > > 1. *Pipeline Configuration Screenshot* > 2. *Pipeline Screenshot* > 3. *Complete Error Log* > > Could you please provide guidance on how to resolve this issue? Let me > know if there are any additional steps or configurations required to > successfully run the pipeline on the Beam Spark pipeline engine. > > Thank you for your support. > > Best regards, > Rajaganesh S >
