Thank you for the clarification. We will follow the folder creation recommendation. All the best, Eila
On Fri, Mar 25, 2022 at 1:49 PM Luke Cwik <[email protected]> wrote: > /opt/userowned was slightly more as it gave access to the VMs filesystem > outside of the container. This was done due to space limits within > Dataflows containers preventing arbitrarily large files. This was done > since custom containers were not supported for Dataflow runner v1. Now with > custom containers this is no longer an issue and yes if the users files fit > within the container they could create the /op/userowned directory > themselves. > > On Fri, Mar 25, 2022 at 10:43 AM Robert Bradshaw <[email protected]> > wrote: > >> There's no reason we couldn't create a /opt/userowned in the runner v2 >> (aka portable) container as well for backwards compatibility (though in >> some sense, the user owns the full container rather than just a >> subdirectory in this case). Any reason not to do that? Should we try to get >> this into the next release? >> >> A short term workaround would be to simply create /opt/userowned in your >> script. >> >> >> On Fri, Mar 25, 2022 at 10:03 AM Luke Cwik <[email protected]> wrote: >> >>> /opt/userowned was for Dataflow runner v1 to store extra data and a >>> short term solution which was replaced with custom containers that is >>> now GA for Python for runner v2 and it looks like your pipeline(s) have >>> been opted into runner v2. >>> >>> You can restore the existing behavior by opting out of using runner v2 >>> by passing in the experiment: >>> --experiments=disable_runner_v2 >>> >>> It would be worthwhile to swap to use custom containers instead of using >>> /opt/userowned because eventually runner v1 will be unsupported. >>> >>> 1: https://cloud.google.com/dataflow/docs/guides/using-custom-containers >>> >>> On Fri, Mar 25, 2022 at 5:19 AM Eila Oriel Research < >>> [email protected]> wrote: >>> >>>> Thank you. I am trying to figure it out too. most of my pipelines are >>>> relying on setup.py work on the workers >>>> Eila >>>> >>>> On Thu, Mar 24, 2022 at 10:31 PM Ahmet Altay <[email protected]> wrote: >>>> >>>>> Adding people who might have knowledge: @Reza Rokni <[email protected]> >>>>> @Valentyn >>>>> Tymofieiev <[email protected]> >>>>> >>>>> On Thu, Mar 24, 2022 at 7:08 PM Eila Oriel Research < >>>>> [email protected]> wrote: >>>>> >>>>>> Hello, >>>>>> >>>>>> We used /opt/userowned/ drive on the workers to copy files and >>>>>> installation vi setup.py >>>>>> >>>>>> We have noticed that /opt/userowned/ is not available anymore. The >>>>>> only folder under opt is google. >>>>>> >>>>>> Please let me know if there is any folder "dedicated" for >>>>>> installation / external files. >>>>>> >>>>>> Thanks, >>>>>> -- >>>>>> Eila Arich >>>>>> >>>>>> Founder, CEO >>>>>> >>>>>> Oriel Research Therapeutics >>>>>> >>>>>> https://www.orielresearch.com/blog >>>>>> [email protected] >>>>>> www.orielresearch.com >>>>>> Cambridge, MA >>>>>> [image: twitter] <https://twitter.com/eilalan1> >>>>>> [image: linkedin] <https://www.linkedin.com/in/eilalandkof/> >>>>>> [image: instagram] <https://www.instagram.com/eilalan/> >>>>>> >>>>>> >>>> >>>> -- >>>> Eila Arich >>>> >>>> Founder, CEO >>>> >>>> Oriel Research Therapeutics >>>> >>>> https://www.orielresearch.com/blog >>>> [email protected] >>>> www.orielresearch.com >>>> Cambridge, MA >>>> [image: twitter] <https://twitter.com/eilalan1> >>>> [image: linkedin] <https://www.linkedin.com/in/eilalandkof/> >>>> [image: instagram] <https://www.instagram.com/eilalan/> >>>> >>>> -- Eila Arich Founder, CEO Oriel Research Therapeutics https://www.orielresearch.com/blog [email protected] www.orielresearch.com Cambridge, MA [image: twitter] <https://twitter.com/eilalan1> [image: linkedin] <https://www.linkedin.com/in/eilalandkof/> [image: instagram] <https://www.instagram.com/eilalan/>
