unsubscribe

Re: Regression of (BEAM-2277) - IllegalArgumentException when using Hadoop file system for WordCount example.

Thanks for taking care. On Mon, Aug 20, 2018 at 5:02 PM Tim Robertson wrote: > Hi Juan, > > You are correct that BEAM-2277 seems to be recurring. I have today > stumbled upon that myself in my own pipeline (not word count). > I have just posted a workaround at the bottom of the issue, and will

unsubscribe

** Jacob Kroeze VPAFA jkro...@usc.edu 213-740-3487 From: Romain Manni-Bucau Sent: Monday, August 20, 2018 8:57:36 AM To: user@beam.apache.org Subject: Re: Launching a subprocess in DoFn It is surprising eclipse doesn't run with your user but yes, perms

Re: Launching a subprocess in DoFn

Hi Romain - OKay, so I was running it through eclipse. I believe the jobs were running as eclipse user (?) as oppose to me (authorized to launch docker). Once, I ran project on command line (irrespective of sudo or not), containers executed just fine. Let me know if that makes sense. Thanks

Re: Launching a subprocess in DoFn

Hi Romain - Got it. Apparently, I needed to run beam as sudo to launch the docker containers. I still need to figure out why that's the case. Thank you. *--* *Mahesh Vangala* *(Ph) 443-326-1957* *(web) mvangala.com * On Mon, Aug 20, 2018 at 11:05 AM Mahesh Vangala wrote:

Re: Launching a subprocess in DoFn

Hello Romain - So, I have added pb.inheritIO().start().waitFor(); and now I have an error /bin/bash: docker: command not found. But, I have docker installed on the system. /usr/local/bin/docker Any ideas why I'm seeing this error when launched from within DoFn? Thank you so much for your help.

Re: Regression of (BEAM-2277) - IllegalArgumentException when using Hadoop file system for WordCount example.

Hi Juan, You are correct that BEAM-2277 seems to be recurring. I have today stumbled upon that myself in my own pipeline (not word count). I have just posted a workaround at the bottom of the issue, and will reopen the issue. Thank you for reminding us on this, Tim On Mon, Aug 20, 2018 at 4:44

Re: Launching a subprocess in DoFn

Weird, this code works: https://gist.github.com/rmannibucau/4703f321bb1962d1303f8eccbd05df0e Are you sure your test_in.csv has some data (otherwise no DoFn processing will be triggered)? Romain Manni-Bucau @rmannibucau | Blog

Re: Regression of (BEAM-2277) - IllegalArgumentException when using Hadoop file system for WordCount example.

BUMP!!! There are people reporting the problem for BEAM 2.6.0 , any CORE dev out there? :) On Mon, Jul 30, 2018 at 3:25 PM Juan Carlos

Re: Launching a subprocess in DoFn

Hi Mahesh, Did you get the same error? This is quite unrelated to beam itself normally. If your engine (spark, dataflow etc) doesn't have a security manager active it should be enough, if it has you can be forbidden to use that. Romain Manni-Bucau @rmannibucau |

Re: Launching a subprocess in DoFn

Hello Romain - I did try that, still no luck. Also, when I put the process start logic into separate Test script, I do notice successful docker container when I do "docker ps". However, no such luck implementing that logic with in DoFn. Any thoughts? Thank you. Regards, Mahesh *--* *Mahesh