Hi Ahmat,
I have received the version from the worker using the following commands: *from google.cloud import bigquery* *logging.info <http://logging.info>('bigquery.__version__ is %s ',bigquery.__version__)* I tried few time to install the google-cloud-bigquery on the workers using setup.py with no much success: *from setuptools import setup, find_packages* *setup(* * name='label-or',* * version='1.0.0',* * packages=find_packages(),* * keywords=[* * ],* * license="Apache Software License",* * install_requires=[* * 'google-cloud-bigquery==0.28.0',* * ],* * package_data={* * },* * data_files=[],* *)* on the job report UI, this message is being reported ( I dont know if it is relevant to the dependencies) SDK version Google Cloud Dataflow SDK for Python 2.0.0 A newer version of this SDK is available. <https://cloud.google.com/dataflow/support?hl=en_US> I was able to upgrade to bigquery.__version__ is 0.25.0 but not to 0.28.0 (which has different API) could you please advice what am I missing? Is it impossible to work with newer version? Many thanks, Eila On Thu, Jul 12, 2018 at 9:40 PM, Ahmet Altay <[email protected]> wrote: > Hi Eila, > > You can find a list of dependencies installed in Dataflow workers in [1]. > Dataflow workers will have a set of dependencies that will satisfy the > requirements from setup.py. > > Which bigquery library you are using? There is > a google-cloud-bigquery==0.25.0 dependency, I am not sure where the > 0.23.0 is coming from. > > Workers do not pick up libraries from the client environment as part of > the job submission. I am not sure how datalab UI integration works > however you have a few options for installing any set of dependencies in > the workers. Using requirements.txt is one of those options. > > Ahmet > > [1] https://cloud.google.com/dataflow/docs/concepts/sdk- > worker-dependencies#version-250_1 > > On Thu, Jul 12, 2018 at 8:51 AM, OrielResearch Eila Arich-Landkof < > [email protected]> wrote: > >> Hi all, >> >> I am running python pipeline with google.cloud.bigquery library. >> on the local runner, everything runs great >> bigquery.__version__ is 0.28.0 >> >> on the dataflow runner, the version is 0.23.0 bigquery.__version__ is >> 0.23.0 >> and there are many API changes between these versions. >> >> What will be the best way to change the installed version on the workers? >> I was assuming the the worker has all the master machine libraries >> installed when the execution is done from datalab - is that true? >> I am not generating any requirements.txt, the execution is done through >> the run button on the datalab UI. >> >> >> please help me solve that issue. >> Thanks, >> -- >> Eila >> www.orielresearch.org >> https://www.meetu <https://www.meetup.com/Deep-Learning-In-Production/> >> p.co <https://www.meetup.com/Deep-Learning-In-Production/>m/Deep-Le >> arning-In-Production/ >> <https://www.meetup.com/Deep-Learning-In-Production/> >> >> >> > -- Eila www.orielresearch.org https://www.meetu <https://www.meetup.com/Deep-Learning-In-Production/>p.co <https://www.meetup.com/Deep-Learning-In-Production/> m/Deep-Learning-In-Production/ <https://www.meetup.com/Deep-Learning-In-Production/>
