Dear Apache Hop Team,
I am reaching out to seek your assistance with an issue I am encountering
while running a simple Hop pipeline using the Beam Spark pipeline engine.
The pipeline executes without any issues when using the default local
runner, but it fails when I switch to the Beam Spark engine.
Below are the details of my setup:
- *Apache Beam Version:* 2.61.0
- *Apache Spark Version:* 3.5.1
- *Apache Hop Version:* apache-hop-client-2.11.0
The error logs are as follows:
yaml
Copy code
2025/01/02 15:30:53 - Hop - Pipeline opened. 2025/01/02 15:30:53 -
Hop - Launching pipeline [hop_trans2]... 2025/01/02 15:30:53 - Hop -
Started the pipeline execution. java.lang.NoClassDefFoundError:
org/apache/spark/metrics/source/Source
at java.base/java.lang.Class.getDeclaredMethods0(Native Method)
... 2025/01/02 15:30:53 - Hop - ERROR: hop_trans2: preparing
pipeline execution failed 2025/01/02 15:30:53 - Hop - ERROR:
org.apache.hop.core.exception.HopException: 2025/01/02 15:30:53 - Hop
- Error preparing remote pipeline 2025/01/02 15:30:53 - Hop - Error
converting Hop pipeline to Beam 2025/01/02 15:30:53 - Hop -
2025/01/02 15:30:53 - Hop - Caused by: java.lang.NoClassDefFoundError:
org/apache/spark/metrics/source/Source 2025/01/02 15:30:53 - Hop -
at java.base/java.lang.Class.getDeclaredMethods0(Native Method)
...
It appears that the issue is related to the
org.apache.spark.metrics.source.Source class not being found.
I have also attached the following:
1. *Pipeline Configuration Screenshot*
2. *Pipeline Screenshot*
3. *Complete Error Log*
Could you please provide guidance on how to resolve this issue? Let me know
if there are any additional steps or configurations required to
successfully run the pipeline on the Beam Spark pipeline engine.
Thank you for your support.
Best regards,
Rajaganesh S
