Thanks a lot for the quick response I have one more question please: I created a simple pipeline that generates fictitious data and filter values. i was able to create a far jar from HOP and i deployed it manually to my flink cluster (via flink run command) the problem i am getting now is "org.apache.flink.client.program.ProgramInvocationException: No manifest found in jar file '/opt/flink/bin/../lib/my-far.jar'. The manifest is need to point to the program's main class."
Is there any configuration/ definition I am missing before generating the jar? I know that flink requires a main class in order to deploy the job. In my java beam pipeline implementation i created those as part of my project How do I make sure it exists in the HOP jar? Thanks Sigalit On Mon, Jan 3, 2022 at 2:54 PM Matt Casters <matt.cast...@neo4j.com> wrote: > Hi Sigalit, > > Thanks for testing the Flink runner in Hop and thanks for bringing to our > attention that the documentation is not up to spec. > > https://issues.apache.org/jira/browse/HOP-3599 > > > Our plan is to modify the pipelines and implement them in HOP. I > understand I need to generate a 'far' jar and then deploy it to the flink > master. > > That's right! The process is the same as for Apache Spark documented > here: [1] > The caveat being that the fat jar can be anywhere that the Beam Flink > runner can find it, including in a hdfs:// location. > I think that's where the real problem is with the flink run command as > some library is forcing a really old hadoop-common library to be included > with which there is an issue. > > https://issues.apache.org/jira/browse/HOP-3600 > > Finally, it *is* weird that in the Hop GUI it's always running with the > embedded engine. I think something might have changed in the Flink runner > API. I'll have a look there as well. > > https://issues.apache.org/jira/browse/HOP-3601 > > These things shouldn't be hard to fix but I'll keep you in the loop. > > Cheers, > Matt > > [1] > https://hop.apache.org/manual/latest/pipeline/pipeline-run-configurations/beam-spark-pipeline-engine.html#_running_with_spark_submit > > -- > Neo4j Chief Solutions Architect > *✉ *matt.cast...@neo4j.com > > > >
