Sébastien, what kind of an issue you had with using setup.py with installation_requires?
On Mon, Jun 5, 2017 at 4:44 PM, Morand, Sebastien < [email protected]> wrote: > Hi, > > I ran into trouble when using setup.py with installation_requires. So I > basically ended up with setup.py with no installation requirements inside + > requirements.txt : > > PIPELINE_OPTIONS = [ > '--project={}'.format(projectname), > '--runner=DataflowRunner', > '--temp_location=gs://dataflow-run/temp', > '--staging_location=gs://dataflow-run/staging', > '--requirements_file=requirements.txt', > '--save_main_session', > '--setup_file=./setup.py' > ] > > with setup.py: > setup( > name='MyProject', > version='1.0', > description='My Description', > author='myself', > author_email='[email protected]', > url='http://myurl.whatever.com', > package_dir={'': 'src'}, > packages=[ > 'package1', > 'package1.subpackage1', > 'package1.subpackage2', > 'package2' > ] > ) > > Regards > > *Sébastien MORAND* > Team Lead Solution Architect > Technology & Operations / Digital Factory > Veolia - Group Information Systems & Technology (IS&T) > Cell.: +33 7 52 66 20 81 / Direct: +33 1 85 57 71 08 > <+33%201%2085%2057%2071%2008> > Bureau 0144C (Ouest) > 30, rue Madeleine-Vionnet - 93300 Aubervilliers, France > *www.veolia.com <http://www.veolia.com>* > <http://www.veolia.com> > <https://www.facebook.com/veoliaenvironment/> > <https://www.youtube.com/user/veoliaenvironnement> > <https://www.linkedin.com/company/veolia-environnement> > <https://twitter.com/veolia> > > On 5 June 2017 at 22:56, Dmitry Demeshchuk <[email protected]> wrote: > >> Hi list, >> >> Suppose, you have a private Python package that contains some code people >> want to be sharing when writing their pipelines. >> >> So, typically, the installation process of the package would be either >> >> pip install git+ssh://[email protected]/mycompany/mypackage#egg=mypackage >> >> or >> >> git clone git://[email protected]/mycompany/mypackage >> python setup.py mypackage/setup.py >> >> Now, the problem starts when we want to get that package into Dataflow. >> Right now, to my understanding, DataflowRunner supports 3 approaches: >> >> 1. >> >> Specifying a requirements_file parameter in the pipeline options. >> This basically must be a requirements.txt file. >> 2. >> >> Specifying an extra_packages parameter in the pipeline options. This >> must be a list of tarballs, each of which contains a Python package >> packaged using distutils. >> 3. >> >> Specifying a setup_file parameter in the pipeline options. This will >> just run the python path/to/my/setup.py package command and then send >> the files over the wire. >> >> The best approach I could come up with was including an *additional* >> setup.py into the package itself, so that when we install that package, >> the setup.py file gets installed along with it. And then, I’d point the >> setup_file option to that file. >> >> This gist >> <https://gist.github.com/doubleyou/be01226352372491babda7602022c506> >> shows the basic approach in code. Both setup.py and options.py are >> supposed to be present in the installed package. >> >> It kind of works for me, with some caveats, but I just wanted to find out >> if it’s a more decent way to handle my situation. I’m not keen on >> specifying that private package as a git dependency, because of having to >> worry about git credentials, but maybe there are other ways? >> >> Thanks! >> >> -- >> Best regards, >> Dmitry Demeshchuk. >> > > > > ------------------------------------------------------------ > -------------------------------- > This e-mail transmission (message and any attached files) may contain > information that is proprietary, privileged and/or confidential to Veolia > Environnement and/or its affiliates and is intended exclusively for the > person(s) to whom it is addressed. If you are not the intended recipient, > please notify the sender by return e-mail and delete all copies of this > e-mail, including all attachments. Unless expressly authorized, any use, > disclosure, publication, retransmission or dissemination of this e-mail > and/or of its attachments is strictly prohibited. > > Ce message electronique et ses fichiers attaches sont strictement > confidentiels et peuvent contenir des elements dont Veolia Environnement > et/ou l'une de ses entites affiliees sont proprietaires. Ils sont donc > destines a l'usage de leurs seuls destinataires. Si vous avez recu ce > message par erreur, merci de le retourner a son emetteur et de le detruire > ainsi que toutes les pieces attachees. L'utilisation, la divulgation, la > publication, la distribution, ou la reproduction non expressement > autorisees de ce message et de ses pieces attachees sont interdites. > ------------------------------------------------------------ > -------------------------------- >
