I just tried it with the Beam Quickstart form the website and a fresh Flink 1.2.1 download. This is the command I used for building the Quickstart:
mvn package -DskipTests -Pflink-runner (The important bit here is -Pflink-runner which brings in the Flink Runner and its dependencies) For running I did this: bin/start-cluster.sh bin/flink run -c org.apache.beam.examples.WordCount ../word-count-beam/target/word-count-beam-0.1.jar --inputFile=/Users/aljoscha/Downloads/word-count-beam/pom.xml --output=/tmp/counts --runner=FlinkRunner Best, Aljoscha > On 30. Jun 2017, at 01:02, Will Walters <[email protected]> wrote: > > As for creating a fat jar, I've encountered difficulty with this as well. > When attempting to compile (with Maven, using the package command) the Beam > master to a jar, it instead creates several separate jars in each subfolder. > And without messing with the pom file, it fails because of missing > dependencies. I've tried to get around this by altering the pom file to > include all possible dependencies, but even after doing this, the job > submission will still fail, citing some missing dependency. > > Any advice on how to create a working fat jar? > > Thanks, > Will. > > On Thursday, June 29, 2017 1:28 AM, Jean-Baptiste Onofré <[email protected]> > wrote: > > > Good point, fair enough. > > Regards > JB > > On 06/29/2017 10:26 AM, Aljoscha Krettek wrote: > > I think it’s a bug because if you start a Flink cluster on bare-metal it > > works, just when it’s started in YARN it doesn’t. And I feel that the way > > you start your cluster should not affect how you can submit jobs to it. > > > > Best, > > Aljoscha > > > >> On 29. Jun 2017, at 10:15, Jean-Baptiste Onofré <[email protected] > >> <mailto:[email protected]>> wrote: > >> > >> Yes, it's the same with the spark runner using bin/spark-submit. From my > >> standpoint, it's not a bug, it's a feature request. > >> > >> Regards > >> JB > >> > >> On 06/29/2017 10:12 AM, Aljoscha Krettek wrote: > >>> I also responded to a separate mail by Will. The problem is that > >>> currently we cannot submit a job using the remote client to a Flink > >>> cluster that was started on YARN. (It’s a bug or “feature” of how > >>> communication with a Flink cluster from a client works.) > >>> The workaround for that is to use the bin/flink command to submit a Beam > >>> fat-jar on a Flink YARN cluster. > >>> Best, > >>> Aljoscha > >>>> On 29. Jun 2017, at 07:23, Jean-Baptiste Onofré <[email protected] > >>>> <mailto:[email protected]>> wrote: > >>>> > >>>> Hi Will, > >>>> > >>>> assuming you are using Beam 2.0.0, the Flink runner uses Flink 1.2.1 by > >>>> default. So, I would recommend this version or 1.2.x. > >>>> > >>>> Regards > >>>> JB > >>>> > >>>> On 06/28/2017 10:39 PM, Will Walters wrote: > >>>>> Hello, > >>>>> I've been attempting to run Beam through Flink on a Yarn cluster and > >>>>> have run into trouble with getting a job to submit, partly because of > >>>>> incompatibility between versions. Does anyone know what versions of > >>>>> Beam and Flink I should be using to give myself the best chance of > >>>>> finding compatibility? > >>>>> Thank you, > >>>>> Will. > >>>> > >>>> -- > >>>> Jean-Baptiste Onofré > >>>> [email protected] <mailto:[email protected]> > >>>> http://blog.nanthrax.net <http://blog.nanthrax.net/> > >>>> Talend - http://www.talend.com <http://www.talend.com/> > > >> > >> -- > >> Jean-Baptiste Onofré > >> [email protected] <mailto:[email protected]> > >> http://blog.nanthrax.net <http://blog.nanthrax.net/> > >> Talend - http://www.talend.com <http://www.talend.com/> > > > > -- > Jean-Baptiste Onofré > [email protected] <mailto:[email protected]> > http://blog.nanthrax.net <http://blog.nanthrax.net/> > Talend - http://www.talend.com <http://www.talend.com/> > >
