Re: Big Query source and Dataflow template doubt

[Pablo Estrada]

Hello Andre!
Unfortunately, the BigQuerySource does not support value providers.
There is a different transform to read from bigquery in
apache_beam.io.gcp.bigquery._ReadFromBigQuery. This one will soon support
it (hopefully by 2.21).
Unfortunately at the moment, it is not possible to change the query
depending on the value providers.
Best
-P.

On Fri, Mar 13, 2020 at 12:46 PM André Rocha Silva <
a.si...@portaltelemedicina.com.br> wrote:

> Hello everybody
>
> I have a query that I am trying to use in dataflow:
>
> ***
>  'QueryTable' >> beam.io.Read(
> beam.io.BigQuerySource(
> dataset='dataset',
> project='project',
> query=f''' SELECT * FROM dataset.table WHERE version =
> '{user_options.version} )
> ***
> When I run it from my terminal with Dataflow runner, it goes as it should
> be. No problem at all.
>
> But when I try to launch it from template the query does not get the
> option. That appears in the step info.
>
> [image: image.png]
>
> I have tried to change from {user_options.version}
> to {user_options.version.get()}. But got this error:
> apache_beam.error.RuntimeValueProviderError: RuntimeValueProvider(option:
> version, type: str, default_value: None).get() not called from a runtime
> context
>
> Does someone know how to correct this?
> Locally, I am using:
> SDK version
> Apache Beam Python 3.7 SDK 2.17.0
>
> Thank you a lot!
>
> --
>
>*ANDRÉ ROCHA SILVA*
>   * DATA ENGINEER*
>   (48) 3181-0611
>
>    /andre-rocha-silva/
> 
> 
> 
> 
>
>

Big Query source and Dataflow template doubt

[André Rocha Silva]

Hello everybody

I have a query that I am trying to use in dataflow:

***
 'QueryTable' >> beam.io.Read(
beam.io.BigQuerySource(
dataset='dataset',
project='project',
query=f''' SELECT * FROM dataset.table WHERE version =
'{user_options.version} )
***
When I run it from my terminal with Dataflow runner, it goes as it should
be. No problem at all.

But when I try to launch it from template the query does not get the
option. That appears in the step info.

[image: image.png]

I have tried to change from {user_options.version}
to {user_options.version.get()}. But got this error:
apache_beam.error.RuntimeValueProviderError: RuntimeValueProvider(option:
version, type: str, default_value: None).get() not called from a runtime
context

Does someone know how to correct this?
Locally, I am using:
SDK version
Apache Beam Python 3.7 SDK 2.17.0

Thank you a lot!

-- 

   *ANDRÉ ROCHA SILVA*
  * DATA ENGINEER*
  (48) 3181-0611

   /andre-rocha-silva/

Re: Hello Beam Community!

[Robert Bradshaw]

Welcome!

On Fri, Mar 13, 2020 at 7:41 AM Ismaël Mejía  wrote:

> Welcome !
>
> On Fri, Mar 13, 2020 at 3:00 PM Connell O'Callaghan 
> wrote:
>
>> Welcome Brittany
>>
>> On Fri, Mar 13, 2020 at 6:45 AM Rustam Mehmandarov 
>> wrote:
>>
>>> Welcome Brittany!
>>>
>>> Cheers,
>>> Rustam
>>> @rmehmandarov 
>>>
>>> On Fri, Mar 13, 2020 at 2:31 AM Brittany Hermann 
>>> wrote:
>>>
 Hello Beam Community!

 My name is Brittany Hermann and I recently joined the Open Source team
 in Data Analytics at Google. As a Program Manager, I will be focusing on
 community engagement while getting to work on Apache Beam and Airflow
 projects! I have always thrived on creating healthy, diverse, and overall
 happy communities and am excited to bring that to the team. For a fun fact,
 I am a big Wisconsin Badgers Football fan and have a goldendoodle puppy
 named Ollie!

 I look forward to collaborating with you all!

 Kind regards,

 Brittany Hermann

 --

 Brittany Hermann

 Open Source Program Manager (Provided by Adecco Staffing)

 1190 Bordeaux Drive , Building 4, Sunnyvale, CA 94089
 

Re: Hello Beam Community!

[Ismaël Mejía]

Welcome !

On Fri, Mar 13, 2020 at 3:00 PM Connell O'Callaghan 
wrote:

> Welcome Brittany
>
> On Fri, Mar 13, 2020 at 6:45 AM Rustam Mehmandarov 
> wrote:
>
>> Welcome Brittany!
>>
>> Cheers,
>> Rustam
>> @rmehmandarov 
>>
>> On Fri, Mar 13, 2020 at 2:31 AM Brittany Hermann 
>> wrote:
>>
>>> Hello Beam Community!
>>>
>>> My name is Brittany Hermann and I recently joined the Open Source team
>>> in Data Analytics at Google. As a Program Manager, I will be focusing on
>>> community engagement while getting to work on Apache Beam and Airflow
>>> projects! I have always thrived on creating healthy, diverse, and overall
>>> happy communities and am excited to bring that to the team. For a fun fact,
>>> I am a big Wisconsin Badgers Football fan and have a goldendoodle puppy
>>> named Ollie!
>>>
>>> I look forward to collaborating with you all!
>>>
>>> Kind regards,
>>>
>>> Brittany Hermann
>>>
>>> --
>>>
>>> Brittany Hermann
>>>
>>> Open Source Program Manager (Provided by Adecco Staffing)
>>>
>>> 1190 Bordeaux Drive , Building 4, Sunnyvale, CA 94089
>>> 
>>>
>>>
>>>

Re: Hello Beam Community!

[Connell O'Callaghan]

Welcome Brittany

On Fri, Mar 13, 2020 at 6:45 AM Rustam Mehmandarov 
wrote:

> Welcome Brittany!
>
> Cheers,
> Rustam
> @rmehmandarov 
>
> On Fri, Mar 13, 2020 at 2:31 AM Brittany Hermann 
> wrote:
>
>> Hello Beam Community!
>>
>> My name is Brittany Hermann and I recently joined the Open Source team in
>> Data Analytics at Google. As a Program Manager, I will be focusing on
>> community engagement while getting to work on Apache Beam and Airflow
>> projects! I have always thrived on creating healthy, diverse, and overall
>> happy communities and am excited to bring that to the team. For a fun fact,
>> I am a big Wisconsin Badgers Football fan and have a goldendoodle puppy
>> named Ollie!
>>
>> I look forward to collaborating with you all!
>>
>> Kind regards,
>>
>> Brittany Hermann
>>
>> --
>>
>> Brittany Hermann
>>
>> Open Source Program Manager (Provided by Adecco Staffing)
>>
>> 1190 Bordeaux Drive , Building 4, Sunnyvale, CA 94089
>> 
>>
>>
>>

Re: Hello Beam Community!

[Rustam Mehmandarov]

Welcome Brittany!

Cheers,
Rustam
@rmehmandarov 

On Fri, Mar 13, 2020 at 2:31 AM Brittany Hermann 
wrote:

> Hello Beam Community!
>
> My name is Brittany Hermann and I recently joined the Open Source team in
> Data Analytics at Google. As a Program Manager, I will be focusing on
> community engagement while getting to work on Apache Beam and Airflow
> projects! I have always thrived on creating healthy, diverse, and overall
> happy communities and am excited to bring that to the team. For a fun fact,
> I am a big Wisconsin Badgers Football fan and have a goldendoodle puppy
> named Ollie!
>
> I look forward to collaborating with you all!
>
> Kind regards,
>
> Brittany Hermann
>
> --
>
> Brittany Hermann
>
> Open Source Program Manager (Provided by Adecco Staffing)
>
> 1190 Bordeaux Drive , Building 4, Sunnyvale, CA 94089
> 
>
>
>