As for creating a fat jar, I've encountered difficulty with this as well. When
attempting to compile (with Maven, using the package command) the Beam master
to a jar, it instead creates several separate jars in each subfolder. And
without messing with the pom file, it fails because of missing dependencies.
I've tried to get around this by altering the pom file to include all possible
dependencies, but even after doing this, the job submission will still fail,
citing some missing dependency.
Any advice on how to create a working fat jar?
Thanks,Will.
On Thursday, June 29, 2017 1:28 AM, Jean-Baptiste Onofré
<j...@nanthrax.net> wrote:
Good point, fair enough.
Regards
JB
On 06/29/2017 10:26 AM, Aljoscha Krettek wrote:
> I think it’s a bug because if you start a Flink cluster on bare-metal it
> works, just when it’s started in YARN it doesn’t. And I feel that the way you
> start your cluster should not affect how you can submit jobs to it.
>
> Best,
> Aljoscha
>
>> On 29. Jun 2017, at 10:15, Jean-Baptiste Onofré <j...@nanthrax.net> wrote:
>>
>> Yes, it's the same with the spark runner using bin/spark-submit. From my
>> standpoint, it's not a bug, it's a feature request.
>>
>> Regards
>> JB
>>
>> On 06/29/2017 10:12 AM, Aljoscha Krettek wrote:
>>> I also responded to a separate mail by Will. The problem is that currently
>>> we cannot submit a job using the remote client to a Flink cluster that was
>>> started on YARN. (It’s a bug or “feature” of how communication with a Flink
>>> cluster from a client works.)
>>> The workaround for that is to use the bin/flink command to submit a Beam
>>> fat-jar on a Flink YARN cluster.
>>> Best,
>>> Aljoscha
>>>> On 29. Jun 2017, at 07:23, Jean-Baptiste Onofré <j...@nanthrax.net> wrote:
>>>>
>>>> Hi Will,
>>>>
>>>> assuming you are using Beam 2.0.0, the Flink runner uses Flink 1.2.1 by
>>>> default. So, I would recommend this version or 1.2.x.
>>>>
>>>> Regards
>>>> JB
>>>>
>>>> On 06/28/2017 10:39 PM, Will Walters wrote:
>>>>> Hello,
>>>>> I've been attempting to run Beam through Flink on a Yarn cluster and have
>>>>> run into trouble with getting a job to submit, partly because of
>>>>> incompatibility between versions. Does anyone know what versions of Beam
>>>>> and Flink I should be using to give myself the best chance of finding
>>>>> compatibility?
>>>>> Thank you,
>>>>> Will.
>>>>
>>>> --
>>>> Jean-Baptiste Onofré
>>>> jbono...@apache.org
>>>> http://blog.nanthrax.net
>>>> Talend - http://www.talend.com
>>
>> --
>> Jean-Baptiste Onofré
>> jbono...@apache.org
>> http://blog.nanthrax.net
>> Talend - http://www.talend.com
>
--
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com
