Thank you. I should have realized this flag. Stay safe Best, Eila
On Mon, Apr 20, 2020 at 12:42 PM Luke Cwik <lc...@google.com> wrote: > It looks like anaconda assumes that the directory doesn't exist before > installation. I would recommend using a new directory such as > ["/opt/userowned/anaconda.sh", "-b","-p","/opt/userowned/anaconda/"] > but if you really want it to be in "/opt/userowned", you should add "-f": > ["/opt/userowned/anaconda.sh","-f","-b","-p","/opt/userowned/"] > > See the Anaconda silent install[1] instructions for more details. > > 1: https://docs.anaconda.com/anaconda/install/silent-mode/#linux-macos > > > On Fri, Apr 17, 2020 at 9:28 PM OrielResearch Eila Arich-Landkof < > e...@orielresearch.org> wrote: > >> Thank you. I was able to tar my libraries at the /opt/userowned fodler. >> I am using setup.py from this url (recommended your apache-beam >> documentation) >> >> https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/complete/juliaset/setup.py >> >> Last, I want to install anaconda. I have a library with dependencies that >> I rather have anaconda take care of. using the custom commands, I am >> executing the following commands. the first two are running fine, the last >> one, fires and error: >> ["wget", "-O", "/opt/userowned/anaconda.sh"," >> https://repo.anaconda.com/archive/Anaconda3-2020.02-Linux-x86_64.sh";], >> ['echo', 'Change conda permissions'], >> ["chmod", "777", "/opt/userowned/anaconda.sh"], >> ["/opt/userowned/anaconda.sh", "-b","-p","/opt/userowned/"] >> >> the last command failed with the following error: >> Command output: b"ERROR: File or directory already exists: >> '/opt/userowned/'\nIf you want to update an existing installation, use the >> -u option.\n" >> >> Please let me know if you have any recommendations on how to install >> anaconda >> >> Many thanks, >> Eila >> >> On Fri, Apr 17, 2020 at 12:12 PM Luke Cwik <lc...@google.com> wrote: >> >>> On Dataflow you should be able to use /opt/userowned >>> >>> On Fri, Apr 17, 2020 at 9:01 AM OrielResearch Eila Arich-Landkof < >>> e...@orielresearch.org> wrote: >>> >>>> See inline >>>> >>>> >>>> — >>>> Eila >>>> www.orielesearch.com >>>> https://www.meetu <https://www.meetup.com/Deep-Learning-In-Production/> >>>> p.co <https://www.meetup.com/Deep-Learning-In-Production/> >>>> m/Deep-Learning-In-Production >>>> <https://www.meetup.com/Deep-Learning-In-Production/> >>>> >>>> Sent from my iPhone >>>> >>>> On Apr 17, 2020, at 11:32 AM, Luke Cwik <lc...@google.com> wrote: >>>> >>>> >>>> When you said you checked '/usr/local/', did you check inside the >>>> docker container or on the VM itself? >>>> >>>> Yes. Checked as part of the DoFn function that runs on the Docker >>>> >>>> Have you tried adding an echo command before and after your script runs >>>> and looked for them in the stackdriver logs? >>>> >>>> No. I see the name of the setup.py that I have in the logs >>>> I can add the echo command as well >>>> >>>> It should help you locate any errors that might have happened when >>>> executing the custom commands. >>>> >>>> Will try that. Is it correct to target to this folder. Any other folder >>>> is ‘dedicated’ for custom downloads? >>>> >>>> Thanks >>>> Eila >>>> >>>> >>>> On Thu, Apr 16, 2020 at 6:30 PM OrielResearch Eila Arich-Landkof < >>>> e...@orielresearch.org> wrote: >>>> >>>>> Hi all, >>>>> >>>>> This is a question that I have post on user-h...@apache.org. in case, >>>>> the other one is not a valid address. posting here again. if you have >>>>> already received it, apologies for the spam. >>>>> >>>>> I hope that you are all well. I would like to copy a tools library >>>>> into the worker machines and uses the setup.py file. I have update the >>>>> CUSTOM_COMMANDS: >>>>> >>>>> CUSTOM_COMMANDS = [ >>>>> ["wget", "-O", >>>>> "/usr/local/sratoolkit.tar.gz","http://ftp-trace.ncbi.nlm.nih.gov/sra/sdk/current/sratoolkit.current-centos_linux64.tar.gz";], >>>>> ["tar", "-xzf", "/usr/local/sratoolkit.tar.gz","-C","/usr/local/"]] >>>>> >>>>> When I am looking for the execution files at the /user/local folder, I >>>>> can not find the tools that I have copied to the worker. What is the right >>>>> & easiest way to copy tools libraries into the worker machines? I was able >>>>> to get the expected behavior of the local runner and now it's a matter of >>>>> finding the right way to reproduce it for the dataflow runner. >>>>> >>>>> I am using python 3.5 with the latest apache-beam 2.20 and latest >>>>> dataflow Thanks a lot, eilalan >>>>> -- >>>>> Eila >>>>> <http://www.orielresearch.com> >>>>> Meetup <https://www.meetup.com/Deep-Learning-In-Production/> >>>>> >>>> >> >> -- >> Eila >> <http://www.orielresearch.com> >> Meetup <https://www.meetup.com/Deep-Learning-In-Production/> >> > -- Eila <http://www.orielresearch.com> Meetup <https://www.meetup.com/Deep-Learning-In-Production/>
