The issue was with java installation. It is not trivial to install it not from the site with cookies assigned to the download. I have copied the java .gz installation file to bucket and gsutil cp the .gz to the worker machine at the setup time (setup.py). I am facing an issue with exporting the Java folder to PATH. ['export', 'PATH=$PATH:/opt/userowned/jdk-14.0.1/bin'] error: [Errno 2] No such file or directory: 'export'
What is the right way to use export in setup time? Thanks, Eila On Tue, May 5, 2020 at 12:47 AM OrielResearch Eila Arich-Landkof < [email protected]> wrote: > I tried that. same error once the Jave installation is added. > CUSTOM_COMMANDS = [['echo', 'Custom command worked!'], > ['apt-get', 'update'], > ['apt-get', '--assume-yes', 'install', 'unzip'], > ['apt-get', '--assume-yes', 'install', 'perl'], > ['apt-get', '--assume-yes', 'install', 'default-jre']] > > and the update at the execution: > def RunCustomCommand(self, command_list): > print('Running command: %s' % command_list) > retVal = subprocess.check_call(command_list) > print("retVal from check_call is {}".format(retVal)) > > all the rest is identical to the apache beam recommended setup.py > > https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/complete/juliaset/setup.py > > Thanks, > Eila > subprocess.CalledProcessError: Command '['apt-get', '--assume-yes', > 'install', 'unzip']' returned non-zero exit status 100 > > > On Mon, May 4, 2020 at 1:43 PM OrielResearch Eila Arich-Landkof < > [email protected]> wrote: > >> Hi Robert, >> >> Thank you. My mistake. It is actually not working. >> testing it with suborcess.check_call now and will let you know >> Best, >> Eila >> >> On Mon, May 4, 2020 at 12:49 PM Robert Bradshaw <[email protected]> >> wrote: >> >>> Printing the result out shouldn't matter, but as mentioned in the doce >>> Popen.communicate is not intended to be used when the amount of output is >>> large. If you need to just run the process, I would recommend a simple >>> subprocess.check_output(). >>> >>> On Mon, May 4, 2020 at 9:00 AM OrielResearch Eila Arich-Landkof < >>> [email protected]> wrote: >>> >>>> I have updated the setup.py to print the error from the installation >>>> and it is working now. I have no idea why that would change anything. Happy >>>> to learn if someone knows the reason >>>> >>>> *The code before:* >>>> stdout_data, _ = p.communicate() >>>> print('Command output: %s' % stdout_data) >>>> *The code now:* >>>> stdout_data, *stderr_data* = p.communicate() >>>> print('Command output: %s' % stdout_data) >>>> *print('Command error data : %s' % stderr_data)* >>>> >>>> So that issue is resolved for me. >>>> Thanks, >>>> Eila >>>> >>>> On Sat, May 2, 2020 at 11:27 PM OrielResearch Eila Arich-Landkof < >>>> [email protected]> wrote: >>>> >>>>> Hi all, >>>>> >>>>> I have experience and very odd behaviour. >>>>> when executing the setup.py with the following CUSTOM COMMAND >>>>> CUSTOM_COMMANDS = [['echo', 'Custom command worked!'], >>>>> ['apt-get', 'update'], >>>>> ['apt-get', 'install', '-y', 'unzip']] >>>>> everything works great. >>>>> when executing: >>>>> CUSTOM_COMMANDS = [['echo', 'Custom command worked!'], >>>>> ['apt-get', 'update'], >>>>> ['apt-get', 'install', '-y', 'unzip'], >>>>> ['apt-get', 'install', '-y', 'perl'], >>>>> ['apt-get', 'install', '-y', 'default-jre']] >>>>> unzip install return error: >>>>> RuntimeError: Command ['apt-get', 'install', '-y', 'unzip'] failed: >>>>> exit code: 100 >>>>> >>>>> Any idea what is the issue? it is either something super simple or >>>>> workaround for some issue that I am not aware of . >>>>> I am using the format of >>>>> https://github.com/apache/beam/tree/master/sdks/python/apache_beam/examples/complete/juliaset >>>>> Thanks, >>>>> -- >>>>> Eila >>>>> <http://www.orielresearch.com> >>>>> Meetup <https://www.meetup.com/Deep-Learning-In-Production/> >>>>> >>>> >>>> >>>> -- >>>> Eila >>>> <http://www.orielresearch.com> >>>> Meetup <https://www.meetup.com/Deep-Learning-In-Production/> >>>> >>> >> >> -- >> Eila >> <http://www.orielresearch.com> >> Meetup <https://www.meetup.com/Deep-Learning-In-Production/> >> > > > -- > Eila > <http://www.orielresearch.com> > Meetup <https://www.meetup.com/Deep-Learning-In-Production/> > -- Eila <http://www.orielresearch.com> Meetup <https://www.meetup.com/Deep-Learning-In-Production/>
