Hi,
I ran into trouble when using setup.py with installation_requires. So I
basically ended up with setup.py with no installation requirements inside +
requirements.txt :
PIPELINE_OPTIONS = [
'--project={}'.format(projectname),
'--runner=DataflowRunner',
'--temp_location=gs://dataflow-run/temp',
'--staging_location=gs://dataflow-run/staging',
'--requirements_file=requirements.txt',
'--save_main_session',
'--setup_file=./setup.py'
]
with setup.py:
setup(
name='MyProject',
version='1.0',
description='My Description',
author='myself',
author_email='[email protected]',
url='http://myurl.whatever.com',
package_dir={'': 'src'},
packages=[
'package1',
'package1.subpackage1',
'package1.subpackage2',
'package2'
]
)
Regards
*Sébastien MORAND*
Team Lead Solution Architect
Technology & Operations / Digital Factory
Veolia - Group Information Systems & Technology (IS&T)
Cell.: +33 7 52 66 20 81 / Direct: +33 1 85 57 71 08
Bureau 0144C (Ouest)
30, rue Madeleine-Vionnet - 93300 Aubervilliers, France
*www.veolia.com <http://www.veolia.com>*
<http://www.veolia.com>
<https://www.facebook.com/veoliaenvironment/>
<https://www.youtube.com/user/veoliaenvironnement>
<https://www.linkedin.com/company/veolia-environnement>
<https://twitter.com/veolia>
On 5 June 2017 at 22:56, Dmitry Demeshchuk <[email protected]> wrote:
> Hi list,
>
> Suppose, you have a private Python package that contains some code people
> want to be sharing when writing their pipelines.
>
> So, typically, the installation process of the package would be either
>
> pip install git+ssh://[email protected]/mycompany/mypackage#egg=mypackage
>
> or
>
> git clone git://[email protected]/mycompany/mypackage
> python setup.py mypackage/setup.py
>
> Now, the problem starts when we want to get that package into Dataflow.
> Right now, to my understanding, DataflowRunner supports 3 approaches:
>
> 1.
>
> Specifying a requirements_file parameter in the pipeline options. This
> basically must be a requirements.txt file.
> 2.
>
> Specifying an extra_packages parameter in the pipeline options. This
> must be a list of tarballs, each of which contains a Python package
> packaged using distutils.
> 3.
>
> Specifying a setup_file parameter in the pipeline options. This will
> just run the python path/to/my/setup.py package command and then send
> the files over the wire.
>
> The best approach I could come up with was including an *additional*
> setup.py into the package itself, so that when we install that package,
> the setup.py file gets installed along with it. And then, I’d point the
> setup_file option to that file.
>
> This gist
> <https://gist.github.com/doubleyou/be01226352372491babda7602022c506>
> shows the basic approach in code. Both setup.py and options.py are
> supposed to be present in the installed package.
>
> It kind of works for me, with some caveats, but I just wanted to find out
> if it’s a more decent way to handle my situation. I’m not keen on
> specifying that private package as a git dependency, because of having to
> worry about git credentials, but maybe there are other ways?
>
> Thanks!
>
> --
> Best regards,
> Dmitry Demeshchuk.
>
--
--------------------------------------------------------------------------------------------
This e-mail transmission (message and any attached files) may contain
information that is proprietary, privileged and/or confidential to Veolia
Environnement and/or its affiliates and is intended exclusively for the
person(s) to whom it is addressed. If you are not the intended recipient,
please notify the sender by return e-mail and delete all copies of this
e-mail, including all attachments. Unless expressly authorized, any use,
disclosure, publication, retransmission or dissemination of this e-mail
and/or of its attachments is strictly prohibited.
Ce message electronique et ses fichiers attaches sont strictement
confidentiels et peuvent contenir des elements dont Veolia Environnement
et/ou l'une de ses entites affiliees sont proprietaires. Ils sont donc
destines a l'usage de leurs seuls destinataires. Si vous avez recu ce
message par erreur, merci de le retourner a son emetteur et de le detruire
ainsi que toutes les pieces attachees. L'utilisation, la divulgation, la
publication, la distribution, ou la reproduction non expressement
autorisees de ce message et de ses pieces attachees sont interdites.
--------------------------------------------------------------------------------------------
