[jira] [Work logged] (BEAM-9305) Support ValueProvider for BigQuerySource query string
[ https://issues.apache.org/jira/browse/BEAM-9305?focusedWorklogId=409131=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-409131 ] ASF GitHub Bot logged work on BEAM-9305: Author: ASF GitHub Bot Created on: 24/Mar/20 22:27 Start Date: 24/Mar/20 22:27 Worklog Time Spent: 10m Work Description: TheNeuralBit commented on issue #11040: [BEAM-9305] Allow value provider query strings in _CustomBigQuerySource URL: https://github.com/apache/beam/pull/11040#issuecomment-603537323 Thank _you_ for catching it @pabloem :) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 409131) Time Spent: 4h 50m (was: 4h 40m) > Support ValueProvider for BigQuerySource query string > - > > Key: BEAM-9305 > URL: https://issues.apache.org/jira/browse/BEAM-9305 > Project: Beam > Issue Type: New Feature > Components: io-py-gcp >Reporter: Elias Djurfeldt >Assignee: Elias Djurfeldt >Priority: Minor > Time Spent: 4h 50m > Remaining Estimate: 0h > > Users should be able to use ValueProviders for the query string in > BigQuerySource. > Ref: > [https://stackoverflow.com/questions/60146887/expected-eta-to-avail-pipeline-i-o-and-runtime-parameters-in-apache-beam-gcp-dat/60170614?noredirect=1#comment106464448_60170614] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9305) Support ValueProvider for BigQuerySource query string
[ https://issues.apache.org/jira/browse/BEAM-9305?focusedWorklogId=409104=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-409104 ] ASF GitHub Bot logged work on BEAM-9305: Author: ASF GitHub Bot Created on: 24/Mar/20 21:22 Start Date: 24/Mar/20 21:22 Worklog Time Spent: 10m Work Description: pabloem commented on issue #11040: [BEAM-9305] Allow value provider query strings in _CustomBigQuerySource URL: https://github.com/apache/beam/pull/11040#issuecomment-603511803 Thanks for catching this @TheNeuralBit . I have https://github.com/apache/beam/pull/11206 out to fix this. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 409104) Time Spent: 4h 40m (was: 4.5h) > Support ValueProvider for BigQuerySource query string > - > > Key: BEAM-9305 > URL: https://issues.apache.org/jira/browse/BEAM-9305 > Project: Beam > Issue Type: New Feature > Components: io-py-gcp >Reporter: Elias Djurfeldt >Assignee: Elias Djurfeldt >Priority: Minor > Time Spent: 4h 40m > Remaining Estimate: 0h > > Users should be able to use ValueProviders for the query string in > BigQuerySource. > Ref: > [https://stackoverflow.com/questions/60146887/expected-eta-to-avail-pipeline-i-o-and-runtime-parameters-in-apache-beam-gcp-dat/60170614?noredirect=1#comment106464448_60170614] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9305) Support ValueProvider for BigQuerySource query string
[ https://issues.apache.org/jira/browse/BEAM-9305?focusedWorklogId=409103=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-409103 ] ASF GitHub Bot logged work on BEAM-9305: Author: ASF GitHub Bot Created on: 24/Mar/20 21:18 Start Date: 24/Mar/20 21:18 Worklog Time Spent: 10m Work Description: TheNeuralBit commented on issue #11040: [BEAM-9305] Allow value provider query strings in _CustomBigQuerySource URL: https://github.com/apache/beam/pull/11040#issuecomment-603510136 It looks like this change may have broken `apache_beam.io.gcp.bigquery_io_read_it_test.BigqueryIOReadIT.test_bigquery_read_1M_python` in Python postcommits, e.g. https://builds.apache.org/job/beam_PostCommit_Python2/2036/ Could you take a look @EDjur and/or @kamilwu? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 409103) Time Spent: 4.5h (was: 4h 20m) > Support ValueProvider for BigQuerySource query string > - > > Key: BEAM-9305 > URL: https://issues.apache.org/jira/browse/BEAM-9305 > Project: Beam > Issue Type: New Feature > Components: io-py-gcp >Reporter: Elias Djurfeldt >Assignee: Elias Djurfeldt >Priority: Minor > Time Spent: 4.5h > Remaining Estimate: 0h > > Users should be able to use ValueProviders for the query string in > BigQuerySource. > Ref: > [https://stackoverflow.com/questions/60146887/expected-eta-to-avail-pipeline-i-o-and-runtime-parameters-in-apache-beam-gcp-dat/60170614?noredirect=1#comment106464448_60170614] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9305) Support ValueProvider for BigQuerySource query string
[ https://issues.apache.org/jira/browse/BEAM-9305?focusedWorklogId=408295=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-408295 ] ASF GitHub Bot logged work on BEAM-9305: Author: ASF GitHub Bot Created on: 23/Mar/20 20:47 Start Date: 23/Mar/20 20:47 Worklog Time Spent: 10m Work Description: pabloem commented on issue #11040: [BEAM-9305] Allow value provider query strings in _CustomBigQuerySource URL: https://github.com/apache/beam/pull/11040#issuecomment-602847544 Thanks @EDjur for the contribution! Thanks @kamilwu for reviewing This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 408295) Time Spent: 4h 20m (was: 4h 10m) > Support ValueProvider for BigQuerySource query string > - > > Key: BEAM-9305 > URL: https://issues.apache.org/jira/browse/BEAM-9305 > Project: Beam > Issue Type: New Feature > Components: io-py-gcp >Reporter: Elias Djurfeldt >Assignee: Elias Djurfeldt >Priority: Minor > Time Spent: 4h 20m > Remaining Estimate: 0h > > Users should be able to use ValueProviders for the query string in > BigQuerySource. > Ref: > [https://stackoverflow.com/questions/60146887/expected-eta-to-avail-pipeline-i-o-and-runtime-parameters-in-apache-beam-gcp-dat/60170614?noredirect=1#comment106464448_60170614] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9305) Support ValueProvider for BigQuerySource query string
[ https://issues.apache.org/jira/browse/BEAM-9305?focusedWorklogId=408294=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-408294 ] ASF GitHub Bot logged work on BEAM-9305: Author: ASF GitHub Bot Created on: 23/Mar/20 20:47 Start Date: 23/Mar/20 20:47 Worklog Time Spent: 10m Work Description: pabloem commented on pull request #11040: [BEAM-9305] Allow value provider query strings in _CustomBigQuerySource URL: https://github.com/apache/beam/pull/11040 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 408294) Time Spent: 4h 10m (was: 4h) > Support ValueProvider for BigQuerySource query string > - > > Key: BEAM-9305 > URL: https://issues.apache.org/jira/browse/BEAM-9305 > Project: Beam > Issue Type: New Feature > Components: io-py-gcp >Reporter: Elias Djurfeldt >Assignee: Elias Djurfeldt >Priority: Minor > Time Spent: 4h 10m > Remaining Estimate: 0h > > Users should be able to use ValueProviders for the query string in > BigQuerySource. > Ref: > [https://stackoverflow.com/questions/60146887/expected-eta-to-avail-pipeline-i-o-and-runtime-parameters-in-apache-beam-gcp-dat/60170614?noredirect=1#comment106464448_60170614] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9305) Support ValueProvider for BigQuerySource query string
[ https://issues.apache.org/jira/browse/BEAM-9305?focusedWorklogId=403850=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-403850 ] ASF GitHub Bot logged work on BEAM-9305: Author: ASF GitHub Bot Created on: 16/Mar/20 10:30 Start Date: 16/Mar/20 10:30 Worklog Time Spent: 10m Work Description: kamilwu commented on issue #11040: [BEAM-9305] Allow value provider query strings in _CustomBigQuerySource URL: https://github.com/apache/beam/pull/11040#issuecomment-599459578 LGTM, let's wait for @pabloem's final review as well This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 403850) Time Spent: 4h (was: 3h 50m) > Support ValueProvider for BigQuerySource query string > - > > Key: BEAM-9305 > URL: https://issues.apache.org/jira/browse/BEAM-9305 > Project: Beam > Issue Type: New Feature > Components: io-py-gcp >Reporter: Elias Djurfeldt >Assignee: Elias Djurfeldt >Priority: Minor > Time Spent: 4h > Remaining Estimate: 0h > > Users should be able to use ValueProviders for the query string in > BigQuerySource. > Ref: > [https://stackoverflow.com/questions/60146887/expected-eta-to-avail-pipeline-i-o-and-runtime-parameters-in-apache-beam-gcp-dat/60170614?noredirect=1#comment106464448_60170614] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9305) Support ValueProvider for BigQuerySource query string
[ https://issues.apache.org/jira/browse/BEAM-9305?focusedWorklogId=403840=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-403840 ] ASF GitHub Bot logged work on BEAM-9305: Author: ASF GitHub Bot Created on: 16/Mar/20 10:09 Start Date: 16/Mar/20 10:09 Worklog Time Spent: 10m Work Description: EDjur commented on issue #11040: [BEAM-9305] Allow value provider query strings in _CustomBigQuerySource URL: https://github.com/apache/beam/pull/11040#issuecomment-599450124 Sure, JIRA ticket is at https://issues.apache.org/jira/browse/BEAM-9506 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 403840) Time Spent: 3h 50m (was: 3h 40m) > Support ValueProvider for BigQuerySource query string > - > > Key: BEAM-9305 > URL: https://issues.apache.org/jira/browse/BEAM-9305 > Project: Beam > Issue Type: New Feature > Components: io-py-gcp >Reporter: Elias Djurfeldt >Assignee: Elias Djurfeldt >Priority: Minor > Time Spent: 3h 50m > Remaining Estimate: 0h > > Users should be able to use ValueProviders for the query string in > BigQuerySource. > Ref: > [https://stackoverflow.com/questions/60146887/expected-eta-to-avail-pipeline-i-o-and-runtime-parameters-in-apache-beam-gcp-dat/60170614?noredirect=1#comment106464448_60170614] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9305) Support ValueProvider for BigQuerySource query string
[ https://issues.apache.org/jira/browse/BEAM-9305?focusedWorklogId=403837=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-403837 ] ASF GitHub Bot logged work on BEAM-9305: Author: ASF GitHub Bot Created on: 16/Mar/20 09:54 Start Date: 16/Mar/20 09:54 Worklog Time Spent: 10m Work Description: kamilwu commented on issue #11040: [BEAM-9305] Allow value provider query strings in _CustomBigQuerySource URL: https://github.com/apache/beam/pull/11040#issuecomment-599442967 @EDjur Could you create a JIRA ticket regarding `gcs_location` value provider evaluation and assign it to me afterwards? Thanks. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 403837) Time Spent: 3h 40m (was: 3.5h) > Support ValueProvider for BigQuerySource query string > - > > Key: BEAM-9305 > URL: https://issues.apache.org/jira/browse/BEAM-9305 > Project: Beam > Issue Type: New Feature > Components: io-py-gcp >Reporter: Elias Djurfeldt >Assignee: Elias Djurfeldt >Priority: Minor > Time Spent: 3h 40m > Remaining Estimate: 0h > > Users should be able to use ValueProviders for the query string in > BigQuerySource. > Ref: > [https://stackoverflow.com/questions/60146887/expected-eta-to-avail-pipeline-i-o-and-runtime-parameters-in-apache-beam-gcp-dat/60170614?noredirect=1#comment106464448_60170614] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9305) Support ValueProvider for BigQuerySource query string
[
https://issues.apache.org/jira/browse/BEAM-9305?focusedWorklogId=402759=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-402759
]
ASF GitHub Bot logged work on BEAM-9305:
Author: ASF GitHub Bot
Created on: 13/Mar/20 09:43
Start Date: 13/Mar/20 09:43
Worklog Time Spent: 10m
Work Description: EDjur commented on pull request #11040: [BEAM-9305]
Allow value provider query strings in _CustomBigQuerySource
URL: https://github.com/apache/beam/pull/11040#discussion_r392122769
##
File path: sdks/python/apache_beam/io/gcp/bigquery.py
##
@@ -694,14 +699,16 @@ def read(self, range_tracker):
raise NotImplementedError('BigQuery source must be split before being
read')
def _setup_temporary_dataset(self, bq):
+query = self.evaluate_query_valueprovider()
Review comment:
Thanks for the tip re: `check_accessible`, I didn't know about that. I've
updated the PR based on these suggestions.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 402759)
Time Spent: 3.5h (was: 3h 20m)
> Support ValueProvider for BigQuerySource query string
> -
>
> Key: BEAM-9305
> URL: https://issues.apache.org/jira/browse/BEAM-9305
> Project: Beam
> Issue Type: New Feature
> Components: io-py-gcp
>Reporter: Elias Djurfeldt
>Assignee: Elias Djurfeldt
>Priority: Minor
> Time Spent: 3.5h
> Remaining Estimate: 0h
>
> Users should be able to use ValueProviders for the query string in
> BigQuerySource.
> Ref:
> [https://stackoverflow.com/questions/60146887/expected-eta-to-avail-pipeline-i-o-and-runtime-parameters-in-apache-beam-gcp-dat/60170614?noredirect=1#comment106464448_60170614]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9305) Support ValueProvider for BigQuerySource query string
[ https://issues.apache.org/jira/browse/BEAM-9305?focusedWorklogId=402325=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-402325 ] ASF GitHub Bot logged work on BEAM-9305: Author: ASF GitHub Bot Created on: 12/Mar/20 17:15 Start Date: 12/Mar/20 17:15 Worklog Time Spent: 10m Work Description: kamilwu commented on issue #11040: [BEAM-9305] Allow value provider query strings in _CustomBigQuerySource URL: https://github.com/apache/beam/pull/11040#issuecomment-598308988 > Is that a bug then? Yes, this might be a bug. I don't know a good solution at the moment though... this is more complicated, because we have to fallback to temp_location from pipeline options if gcs_location is empty This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 402325) Time Spent: 3h 20m (was: 3h 10m) > Support ValueProvider for BigQuerySource query string > - > > Key: BEAM-9305 > URL: https://issues.apache.org/jira/browse/BEAM-9305 > Project: Beam > Issue Type: New Feature > Components: io-py-gcp >Reporter: Elias Djurfeldt >Assignee: Elias Djurfeldt >Priority: Minor > Time Spent: 3h 20m > Remaining Estimate: 0h > > Users should be able to use ValueProviders for the query string in > BigQuerySource. > Ref: > [https://stackoverflow.com/questions/60146887/expected-eta-to-avail-pipeline-i-o-and-runtime-parameters-in-apache-beam-gcp-dat/60170614?noredirect=1#comment106464448_60170614] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9305) Support ValueProvider for BigQuerySource query string
[
https://issues.apache.org/jira/browse/BEAM-9305?focusedWorklogId=402318=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-402318
]
ASF GitHub Bot logged work on BEAM-9305:
Author: ASF GitHub Bot
Created on: 12/Mar/20 16:58
Start Date: 12/Mar/20 16:58
Worklog Time Spent: 10m
Work Description: kamilwu commented on pull request #11040: [BEAM-9305]
Allow value provider query strings in _CustomBigQuerySource
URL: https://github.com/apache/beam/pull/11040#discussion_r391760791
##
File path: sdks/python/apache_beam/io/gcp/bigquery.py
##
@@ -694,14 +699,16 @@ def read(self, range_tracker):
raise NotImplementedError('BigQuery source must be split before being
read')
def _setup_temporary_dataset(self, bq):
+query = self.evaluate_query_valueprovider()
Review comment:
There's a `check_accessible` decorator that could be used for this function
(as well as for `estimate_size`).
I think you could also test if `self.query` is a ValueProvider or not in the
constructor. If it's not, you can then create a StaticValueProvider object.
`self.evaluate_query_valueprovider()` would be unnecessary.
Here'a en example:
https://github.com/apache/beam/blob/f75838f306fc1d851b0dea11c2cb861b43430568/sdks/python/apache_beam/io/filebasedsource.py#L115-L117
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 402318)
Time Spent: 3h 10m (was: 3h)
> Support ValueProvider for BigQuerySource query string
> -
>
> Key: BEAM-9305
> URL: https://issues.apache.org/jira/browse/BEAM-9305
> Project: Beam
> Issue Type: New Feature
> Components: io-py-gcp
>Reporter: Elias Djurfeldt
>Assignee: Elias Djurfeldt
>Priority: Minor
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> Users should be able to use ValueProviders for the query string in
> BigQuerySource.
> Ref:
> [https://stackoverflow.com/questions/60146887/expected-eta-to-avail-pipeline-i-o-and-runtime-parameters-in-apache-beam-gcp-dat/60170614?noredirect=1#comment106464448_60170614]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9305) Support ValueProvider for BigQuerySource query string
[ https://issues.apache.org/jira/browse/BEAM-9305?focusedWorklogId=401866=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-401866 ] ASF GitHub Bot logged work on BEAM-9305: Author: ASF GitHub Bot Created on: 11/Mar/20 23:31 Start Date: 11/Mar/20 23:31 Worklog Time Spent: 10m Work Description: pabloem commented on issue #11040: [BEAM-9305] Allow value provider query strings in _CustomBigQuerySource URL: https://github.com/apache/beam/pull/11040#issuecomment-597930882 ah good catch. Yes, that would make it a bug.. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 401866) Time Spent: 3h (was: 2h 50m) > Support ValueProvider for BigQuerySource query string > - > > Key: BEAM-9305 > URL: https://issues.apache.org/jira/browse/BEAM-9305 > Project: Beam > Issue Type: New Feature > Components: io-py-gcp >Reporter: Elias Djurfeldt >Assignee: Elias Djurfeldt >Priority: Minor > Time Spent: 3h > Remaining Estimate: 0h > > Users should be able to use ValueProviders for the query string in > BigQuerySource. > Ref: > [https://stackoverflow.com/questions/60146887/expected-eta-to-avail-pipeline-i-o-and-runtime-parameters-in-apache-beam-gcp-dat/60170614?noredirect=1#comment106464448_60170614] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9305) Support ValueProvider for BigQuerySource query string
[ https://issues.apache.org/jira/browse/BEAM-9305?focusedWorklogId=401778=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-401778 ] ASF GitHub Bot logged work on BEAM-9305: Author: ASF GitHub Bot Created on: 11/Mar/20 20:54 Start Date: 11/Mar/20 20:54 Worklog Time Spent: 10m Work Description: EDjur commented on issue #11040: [BEAM-9305] Allow value provider query strings in _CustomBigQuerySource URL: https://github.com/apache/beam/pull/11040#issuecomment-597872563 The `gcs_location` value provider is evaluated in the `expand` method: https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/gcp/bigquery.py#L1575. Is that a bug then? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 401778) Time Spent: 2h 50m (was: 2h 40m) > Support ValueProvider for BigQuerySource query string > - > > Key: BEAM-9305 > URL: https://issues.apache.org/jira/browse/BEAM-9305 > Project: Beam > Issue Type: New Feature > Components: io-py-gcp >Reporter: Elias Djurfeldt >Assignee: Elias Djurfeldt >Priority: Minor > Time Spent: 2h 50m > Remaining Estimate: 0h > > Users should be able to use ValueProviders for the query string in > BigQuerySource. > Ref: > [https://stackoverflow.com/questions/60146887/expected-eta-to-avail-pipeline-i-o-and-runtime-parameters-in-apache-beam-gcp-dat/60170614?noredirect=1#comment106464448_60170614] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9305) Support ValueProvider for BigQuerySource query string
[ https://issues.apache.org/jira/browse/BEAM-9305?focusedWorklogId=401777=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-401777 ] ASF GitHub Bot logged work on BEAM-9305: Author: ASF GitHub Bot Created on: 11/Mar/20 20:51 Start Date: 11/Mar/20 20:51 Worklog Time Spent: 10m Work Description: EDjur commented on issue #11040: [BEAM-9305] Allow value provider query strings in _CustomBigQuerySource URL: https://github.com/apache/beam/pull/11040#issuecomment-597872563 The `gcs_location` value provider is evaluated in the `expand` method: https://github.com/apache/beam/pull/11040/files#diff-683ba04a0e750cf0456b9b119393ad64R1587. Is that a bug then? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 401777) Time Spent: 2h 40m (was: 2.5h) > Support ValueProvider for BigQuerySource query string > - > > Key: BEAM-9305 > URL: https://issues.apache.org/jira/browse/BEAM-9305 > Project: Beam > Issue Type: New Feature > Components: io-py-gcp >Reporter: Elias Djurfeldt >Assignee: Elias Djurfeldt >Priority: Minor > Time Spent: 2h 40m > Remaining Estimate: 0h > > Users should be able to use ValueProviders for the query string in > BigQuerySource. > Ref: > [https://stackoverflow.com/questions/60146887/expected-eta-to-avail-pipeline-i-o-and-runtime-parameters-in-apache-beam-gcp-dat/60170614?noredirect=1#comment106464448_60170614] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9305) Support ValueProvider for BigQuerySource query string
[ https://issues.apache.org/jira/browse/BEAM-9305?focusedWorklogId=401767=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-401767 ] ASF GitHub Bot logged work on BEAM-9305: Author: ASF GitHub Bot Created on: 11/Mar/20 20:43 Start Date: 11/Mar/20 20:43 Worklog Time Spent: 10m Work Description: pabloem commented on issue #11040: [BEAM-9305] Allow value provider query strings in _CustomBigQuerySource URL: https://github.com/apache/beam/pull/11040#issuecomment-597869174 I'm srry @EDjur - the `expand` method runs at template construction, so the change currently will not work with parameters provided at execution time. You'd need to call `get` on the valueprovider in estimate_size/split/etc like we had discussed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 401767) Time Spent: 2.5h (was: 2h 20m) > Support ValueProvider for BigQuerySource query string > - > > Key: BEAM-9305 > URL: https://issues.apache.org/jira/browse/BEAM-9305 > Project: Beam > Issue Type: New Feature > Components: io-py-gcp >Reporter: Elias Djurfeldt >Assignee: Elias Djurfeldt >Priority: Minor > Time Spent: 2.5h > Remaining Estimate: 0h > > Users should be able to use ValueProviders for the query string in > BigQuerySource. > Ref: > [https://stackoverflow.com/questions/60146887/expected-eta-to-avail-pipeline-i-o-and-runtime-parameters-in-apache-beam-gcp-dat/60170614?noredirect=1#comment106464448_60170614] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9305) Support ValueProvider for BigQuerySource query string
[ https://issues.apache.org/jira/browse/BEAM-9305?focusedWorklogId=400705=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-400705 ] ASF GitHub Bot logged work on BEAM-9305: Author: ASF GitHub Bot Created on: 10/Mar/20 12:44 Start Date: 10/Mar/20 12:44 Worklog Time Spent: 10m Work Description: EDjur commented on issue #11040: [BEAM-9305] Allow value provider query strings in _CustomBigQuerySource URL: https://github.com/apache/beam/pull/11040#issuecomment-597065772 @kamilwu Looks like the `AnnotateText` PTransform mention in `CHANGES.md` was removed in https://github.com/apache/beam/commit/0cc577134b9e98b32020efab2ca07ccd1665f7cb#diff-8b1c3fd0d4a6765c16dfd18509182f9d. So I added it back again. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 400705) Time Spent: 2h 20m (was: 2h 10m) > Support ValueProvider for BigQuerySource query string > - > > Key: BEAM-9305 > URL: https://issues.apache.org/jira/browse/BEAM-9305 > Project: Beam > Issue Type: New Feature > Components: io-py-gcp >Reporter: Elias Djurfeldt >Assignee: Elias Djurfeldt >Priority: Minor > Time Spent: 2h 20m > Remaining Estimate: 0h > > Users should be able to use ValueProviders for the query string in > BigQuerySource. > Ref: > [https://stackoverflow.com/questions/60146887/expected-eta-to-avail-pipeline-i-o-and-runtime-parameters-in-apache-beam-gcp-dat/60170614?noredirect=1#comment106464448_60170614] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9305) Support ValueProvider for BigQuerySource query string
[
https://issues.apache.org/jira/browse/BEAM-9305?focusedWorklogId=400682=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-400682
]
ASF GitHub Bot logged work on BEAM-9305:
Author: ASF GitHub Bot
Created on: 10/Mar/20 11:29
Start Date: 10/Mar/20 11:29
Worklog Time Spent: 10m
Work Description: kamilwu commented on pull request #11040: [BEAM-9305]
Allow value provider query strings in _CustomBigQuerySource
URL: https://github.com/apache/beam/pull/11040#discussion_r390250330
##
File path: sdks/python/apache_beam/io/gcp/bigquery_read_it_test.py
##
@@ -156,12 +157,19 @@ def test_native_source(self):
@attr('IT')
def test_iobase_source(self):
+query = StaticValueProvider(str, self.query)
with beam.Pipeline(argv=self.args) as p:
result = (
p | 'read' >> beam.io._ReadFromBigQuery(
query=self.query, use_standard_sql=True, project=self.project))
assert_that(result, equal_to(self.TABLE_DATA))
+with beam.Pipeline(argv=self.args) as p:
Review comment:
@EDjur Thanks for explanation. That's right, if we want to check that query
is successfully evaluated to a string, we have to execute the pipeline.
@pabloem I think this is good idea. Let's do it this way.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 400682)
Time Spent: 2h 10m (was: 2h)
> Support ValueProvider for BigQuerySource query string
> -
>
> Key: BEAM-9305
> URL: https://issues.apache.org/jira/browse/BEAM-9305
> Project: Beam
> Issue Type: New Feature
> Components: io-py-gcp
>Reporter: Elias Djurfeldt
>Assignee: Elias Djurfeldt
>Priority: Minor
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> Users should be able to use ValueProviders for the query string in
> BigQuerySource.
> Ref:
> [https://stackoverflow.com/questions/60146887/expected-eta-to-avail-pipeline-i-o-and-runtime-parameters-in-apache-beam-gcp-dat/60170614?noredirect=1#comment106464448_60170614]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9305) Support ValueProvider for BigQuerySource query string
[
https://issues.apache.org/jira/browse/BEAM-9305?focusedWorklogId=400236=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-400236
]
ASF GitHub Bot logged work on BEAM-9305:
Author: ASF GitHub Bot
Created on: 09/Mar/20 17:14
Start Date: 09/Mar/20 17:14
Worklog Time Spent: 10m
Work Description: pabloem commented on pull request #11040: [BEAM-9305]
Allow value provider query strings in _CustomBigQuerySource
URL: https://github.com/apache/beam/pull/11040#discussion_r389835922
##
File path: sdks/python/apache_beam/io/gcp/bigquery_read_it_test.py
##
@@ -156,12 +157,19 @@ def test_native_source(self):
@attr('IT')
def test_iobase_source(self):
+query = StaticValueProvider(str, self.query)
with beam.Pipeline(argv=self.args) as p:
result = (
p | 'read' >> beam.io._ReadFromBigQuery(
query=self.query, use_standard_sql=True, project=self.project))
assert_that(result, equal_to(self.TABLE_DATA))
+with beam.Pipeline(argv=self.args) as p:
Review comment:
What I would say is use the value provider in the first pipeline of this
method, and don't add a second pipeline. I think that should be fine. WDYT
@kamilwu ?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 400236)
Time Spent: 2h (was: 1h 50m)
> Support ValueProvider for BigQuerySource query string
> -
>
> Key: BEAM-9305
> URL: https://issues.apache.org/jira/browse/BEAM-9305
> Project: Beam
> Issue Type: New Feature
> Components: io-py-gcp
>Reporter: Elias Djurfeldt
>Assignee: Elias Djurfeldt
>Priority: Minor
> Time Spent: 2h
> Remaining Estimate: 0h
>
> Users should be able to use ValueProviders for the query string in
> BigQuerySource.
> Ref:
> [https://stackoverflow.com/questions/60146887/expected-eta-to-avail-pipeline-i-o-and-runtime-parameters-in-apache-beam-gcp-dat/60170614?noredirect=1#comment106464448_60170614]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9305) Support ValueProvider for BigQuerySource query string
[
https://issues.apache.org/jira/browse/BEAM-9305?focusedWorklogId=400021=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-400021
]
ASF GitHub Bot logged work on BEAM-9305:
Author: ASF GitHub Bot
Created on: 09/Mar/20 09:06
Start Date: 09/Mar/20 09:06
Worklog Time Spent: 10m
Work Description: EDjur commented on pull request #11040: [BEAM-9305]
Allow value provider query strings in _CustomBigQuerySource
URL: https://github.com/apache/beam/pull/11040#discussion_r389532725
##
File path: sdks/python/apache_beam/io/gcp/bigquery_read_it_test.py
##
@@ -156,12 +157,19 @@ def test_native_source(self):
@attr('IT')
def test_iobase_source(self):
+query = StaticValueProvider(str, self.query)
with beam.Pipeline(argv=self.args) as p:
result = (
p | 'read' >> beam.io._ReadFromBigQuery(
query=self.query, use_standard_sql=True, project=self.project))
assert_that(result, equal_to(self.TABLE_DATA))
+with beam.Pipeline(argv=self.args) as p:
Review comment:
To actually evaluate the value providers, the pipeline needs to be executed
right? Cause these are realistically only 'known' at runtime. The tests in
`TestReadFromBigQuery` just validate the input data to the initialisation.
We can of course just validate that query is of type `ValueProvider` after
initialising a pipeline, but I figured we want to check that the value provider
query is actually evaluated to a string type on pipeline execution.
Am I perhaps missing something?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 400021)
Time Spent: 1h 50m (was: 1h 40m)
> Support ValueProvider for BigQuerySource query string
> -
>
> Key: BEAM-9305
> URL: https://issues.apache.org/jira/browse/BEAM-9305
> Project: Beam
> Issue Type: New Feature
> Components: io-py-gcp
>Reporter: Elias Djurfeldt
>Assignee: Elias Djurfeldt
>Priority: Minor
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> Users should be able to use ValueProviders for the query string in
> BigQuerySource.
> Ref:
> [https://stackoverflow.com/questions/60146887/expected-eta-to-avail-pipeline-i-o-and-runtime-parameters-in-apache-beam-gcp-dat/60170614?noredirect=1#comment106464448_60170614]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9305) Support ValueProvider for BigQuerySource query string
[
https://issues.apache.org/jira/browse/BEAM-9305?focusedWorklogId=399007=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-399007
]
ASF GitHub Bot logged work on BEAM-9305:
Author: ASF GitHub Bot
Created on: 06/Mar/20 09:13
Start Date: 06/Mar/20 09:13
Worklog Time Spent: 10m
Work Description: kamilwu commented on pull request #11040: [BEAM-9305]
Allow value provider query strings in _CustomBigQuerySource
URL: https://github.com/apache/beam/pull/11040#discussion_r388788211
##
File path: sdks/python/apache_beam/io/gcp/bigquery_read_it_test.py
##
@@ -156,12 +157,19 @@ def test_native_source(self):
@attr('IT')
def test_iobase_source(self):
+query = StaticValueProvider(str, self.query)
with beam.Pipeline(argv=self.args) as p:
result = (
p | 'read' >> beam.io._ReadFromBigQuery(
query=self.query, use_standard_sql=True, project=self.project))
assert_that(result, equal_to(self.TABLE_DATA))
+with beam.Pipeline(argv=self.args) as p:
Review comment:
Do we need an integration test to cover your value provider's logic? Take a
look at `TestReadFromBigQuery` test case in `bigquery_test.py`. Those are unit
tests for `_ReadFromBigQuery` that actually run very fast, because pipeline is
not executed.
Adding new tests in `bigquery_read_it_test.py` causes huge overhead, since
they are all executed by three runners (Direct, Dataflow and Flink) on every
supported Python version.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 399007)
Time Spent: 1h 40m (was: 1.5h)
> Support ValueProvider for BigQuerySource query string
> -
>
> Key: BEAM-9305
> URL: https://issues.apache.org/jira/browse/BEAM-9305
> Project: Beam
> Issue Type: New Feature
> Components: io-py-gcp
>Reporter: Elias Djurfeldt
>Assignee: Elias Djurfeldt
>Priority: Minor
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> Users should be able to use ValueProviders for the query string in
> BigQuerySource.
> Ref:
> [https://stackoverflow.com/questions/60146887/expected-eta-to-avail-pipeline-i-o-and-runtime-parameters-in-apache-beam-gcp-dat/60170614?noredirect=1#comment106464448_60170614]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9305) Support ValueProvider for BigQuerySource query string
[ https://issues.apache.org/jira/browse/BEAM-9305?focusedWorklogId=398288=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-398288 ] ASF GitHub Bot logged work on BEAM-9305: Author: ASF GitHub Bot Created on: 05/Mar/20 10:32 Start Date: 05/Mar/20 10:32 Worklog Time Spent: 10m Work Description: EDjur commented on issue #11040: [BEAM-9305] Allow value provider query strings in _CustomBigQuerySource URL: https://github.com/apache/beam/pull/11040#issuecomment-595155257 Adding @kamilwu as you were mentioned in the Jira ticket too. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 398288) Time Spent: 1.5h (was: 1h 20m) > Support ValueProvider for BigQuerySource query string > - > > Key: BEAM-9305 > URL: https://issues.apache.org/jira/browse/BEAM-9305 > Project: Beam > Issue Type: New Feature > Components: io-py-gcp >Reporter: Elias Djurfeldt >Assignee: Elias Djurfeldt >Priority: Minor > Time Spent: 1.5h > Remaining Estimate: 0h > > Users should be able to use ValueProviders for the query string in > BigQuerySource. > Ref: > [https://stackoverflow.com/questions/60146887/expected-eta-to-avail-pipeline-i-o-and-runtime-parameters-in-apache-beam-gcp-dat/60170614?noredirect=1#comment106464448_60170614] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9305) Support ValueProvider for BigQuerySource query string
[ https://issues.apache.org/jira/browse/BEAM-9305?focusedWorklogId=398283=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-398283 ] ASF GitHub Bot logged work on BEAM-9305: Author: ASF GitHub Bot Created on: 05/Mar/20 10:30 Start Date: 05/Mar/20 10:30 Worklog Time Spent: 10m Work Description: EDjur commented on issue #11040: [BEAM-9305] Allow value provider query strings in _CustomBigQuerySource URL: https://github.com/apache/beam/pull/11040#issuecomment-595154066 How about something like this instead @pabloem ? I noticed that `gcs_location`supports ValueProviders and is wrapped & initialised in the `_ReadFromBigQuery` transform. This reduces clutter in the `_CustomBigQuerySource`. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 398283) Time Spent: 1h 20m (was: 1h 10m) > Support ValueProvider for BigQuerySource query string > - > > Key: BEAM-9305 > URL: https://issues.apache.org/jira/browse/BEAM-9305 > Project: Beam > Issue Type: New Feature > Components: io-py-gcp >Reporter: Elias Djurfeldt >Assignee: Elias Djurfeldt >Priority: Minor > Time Spent: 1h 20m > Remaining Estimate: 0h > > Users should be able to use ValueProviders for the query string in > BigQuerySource. > Ref: > [https://stackoverflow.com/questions/60146887/expected-eta-to-avail-pipeline-i-o-and-runtime-parameters-in-apache-beam-gcp-dat/60170614?noredirect=1#comment106464448_60170614] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9305) Support ValueProvider for BigQuerySource query string
[
https://issues.apache.org/jira/browse/BEAM-9305?focusedWorklogId=397744=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-397744
]
ASF GitHub Bot logged work on BEAM-9305:
Author: ASF GitHub Bot
Created on: 04/Mar/20 18:25
Start Date: 04/Mar/20 18:25
Worklog Time Spent: 10m
Work Description: pabloem commented on pull request #11040: [BEAM-9305]
Allow value provider query strings in _CustomBigQuerySource
URL: https://github.com/apache/beam/pull/11040#discussion_r387847682
##
File path: sdks/python/apache_beam/io/gcp/bigquery_read_it_test.py
##
@@ -162,6 +163,15 @@ def test_iobase_source(self):
query=self.query, use_standard_sql=True, project=self.project))
assert_that(result, equal_to(self.TABLE_DATA))
+ @attr('IT')
+ def test_valueprovider_query_string(self):
+query = StaticValueProvider(str, self.query)
+with beam.Pipeline(argv=self.args) as p:
+ result = (
+ p | 'read' >> beam.io._ReadFromBigQuery(
+ query=query, use_standard_sql=True, project=self.project))
+ assert_that(result, equal_to(self.TABLE_DATA))
Review comment:
could you use the value provider in another one of the tests in this file?
Since each test in this file takes ~5min to run, I'd prefer to avoid adding a
new test.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 397744)
Time Spent: 1h (was: 50m)
> Support ValueProvider for BigQuerySource query string
> -
>
> Key: BEAM-9305
> URL: https://issues.apache.org/jira/browse/BEAM-9305
> Project: Beam
> Issue Type: New Feature
> Components: io-py-gcp
>Reporter: Elias Djurfeldt
>Assignee: Elias Djurfeldt
>Priority: Minor
> Time Spent: 1h
> Remaining Estimate: 0h
>
> Users should be able to use ValueProviders for the query string in
> BigQuerySource.
> Ref:
> [https://stackoverflow.com/questions/60146887/expected-eta-to-avail-pipeline-i-o-and-runtime-parameters-in-apache-beam-gcp-dat/60170614?noredirect=1#comment106464448_60170614]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9305) Support ValueProvider for BigQuerySource query string
[ https://issues.apache.org/jira/browse/BEAM-9305?focusedWorklogId=397745=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-397745 ] ASF GitHub Bot logged work on BEAM-9305: Author: ASF GitHub Bot Created on: 04/Mar/20 18:25 Start Date: 04/Mar/20 18:25 Worklog Time Spent: 10m Work Description: pabloem commented on pull request #11040: [BEAM-9305] Allow value provider query strings in _CustomBigQuerySource URL: https://github.com/apache/beam/pull/11040#discussion_r387850061 ## File path: sdks/python/apache_beam/io/gcp/bigquery.py ## @@ -644,6 +644,8 @@ def estimate_size(self): self.table_reference.tableId) return int(table.numBytes) else: + if isinstance(self.query, ValueProvider): +self.query = self.query.get() Review comment: I would probably do this without changing the class variable. I'd do a local variable: `query = self.query.get()`, and do it on every instance where `self.query` is accessed. The reason is that the Source objects may be recreated in different workers, and the `self.query` variable would contain the original ValueProvider in all of those. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 397745) Time Spent: 1h 10m (was: 1h) > Support ValueProvider for BigQuerySource query string > - > > Key: BEAM-9305 > URL: https://issues.apache.org/jira/browse/BEAM-9305 > Project: Beam > Issue Type: New Feature > Components: io-py-gcp >Reporter: Elias Djurfeldt >Assignee: Elias Djurfeldt >Priority: Minor > Time Spent: 1h 10m > Remaining Estimate: 0h > > Users should be able to use ValueProviders for the query string in > BigQuerySource. > Ref: > [https://stackoverflow.com/questions/60146887/expected-eta-to-avail-pipeline-i-o-and-runtime-parameters-in-apache-beam-gcp-dat/60170614?noredirect=1#comment106464448_60170614] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9305) Support ValueProvider for BigQuerySource query string
[ https://issues.apache.org/jira/browse/BEAM-9305?focusedWorklogId=397480=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-397480 ] ASF GitHub Bot logged work on BEAM-9305: Author: ASF GitHub Bot Created on: 04/Mar/20 10:49 Start Date: 04/Mar/20 10:49 Worklog Time Spent: 10m Work Description: EDjur commented on issue #11040: [BEAM-9305] Allow value provider query strings in _CustomBigQuerySource URL: https://github.com/apache/beam/pull/11040#issuecomment-594450634 Since `estimate_size` is called before `split` and the uses `self.query`, I realised the value provider evaluation actually needs to take place before `split` is called. Does this make sense @pabloem ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 397480) Time Spent: 40m (was: 0.5h) > Support ValueProvider for BigQuerySource query string > - > > Key: BEAM-9305 > URL: https://issues.apache.org/jira/browse/BEAM-9305 > Project: Beam > Issue Type: New Feature > Components: io-py-gcp >Reporter: Elias Djurfeldt >Assignee: Elias Djurfeldt >Priority: Minor > Time Spent: 40m > Remaining Estimate: 0h > > Users should be able to use ValueProviders for the query string in > BigQuerySource. > Ref: > [https://stackoverflow.com/questions/60146887/expected-eta-to-avail-pipeline-i-o-and-runtime-parameters-in-apache-beam-gcp-dat/60170614?noredirect=1#comment106464448_60170614] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9305) Support ValueProvider for BigQuerySource query string
[ https://issues.apache.org/jira/browse/BEAM-9305?focusedWorklogId=397481=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-397481 ] ASF GitHub Bot logged work on BEAM-9305: Author: ASF GitHub Bot Created on: 04/Mar/20 10:49 Start Date: 04/Mar/20 10:49 Worklog Time Spent: 10m Work Description: EDjur commented on issue #11040: [BEAM-9305] Allow value provider query strings in _CustomBigQuerySource URL: https://github.com/apache/beam/pull/11040#issuecomment-594450634 Since `estimate_size` is called before `split` and uses `self.query`, I realised the value provider evaluation actually needs to take place before `split` is called. Does this make sense @pabloem ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 397481) Time Spent: 50m (was: 40m) > Support ValueProvider for BigQuerySource query string > - > > Key: BEAM-9305 > URL: https://issues.apache.org/jira/browse/BEAM-9305 > Project: Beam > Issue Type: New Feature > Components: io-py-gcp >Reporter: Elias Djurfeldt >Assignee: Elias Djurfeldt >Priority: Minor > Time Spent: 50m > Remaining Estimate: 0h > > Users should be able to use ValueProviders for the query string in > BigQuerySource. > Ref: > [https://stackoverflow.com/questions/60146887/expected-eta-to-avail-pipeline-i-o-and-runtime-parameters-in-apache-beam-gcp-dat/60170614?noredirect=1#comment106464448_60170614] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9305) Support ValueProvider for BigQuerySource query string
[ https://issues.apache.org/jira/browse/BEAM-9305?focusedWorklogId=397479=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-397479 ] ASF GitHub Bot logged work on BEAM-9305: Author: ASF GitHub Bot Created on: 04/Mar/20 10:49 Start Date: 04/Mar/20 10:49 Worklog Time Spent: 10m Work Description: EDjur commented on issue #11040: [BEAM-9305] Allow value provider query strings in _CustomBigQuerySource URL: https://github.com/apache/beam/pull/11040#issuecomment-594450634 Since `estimate_size` is called before `split` and the uses `self.query`, I realised the value provider evaluation actually needs to take place before `split` is called. Does this make sense? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 397479) Time Spent: 0.5h (was: 20m) > Support ValueProvider for BigQuerySource query string > - > > Key: BEAM-9305 > URL: https://issues.apache.org/jira/browse/BEAM-9305 > Project: Beam > Issue Type: New Feature > Components: io-py-gcp >Reporter: Elias Djurfeldt >Assignee: Elias Djurfeldt >Priority: Minor > Time Spent: 0.5h > Remaining Estimate: 0h > > Users should be able to use ValueProviders for the query string in > BigQuerySource. > Ref: > [https://stackoverflow.com/questions/60146887/expected-eta-to-avail-pipeline-i-o-and-runtime-parameters-in-apache-beam-gcp-dat/60170614?noredirect=1#comment106464448_60170614] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9305) Support ValueProvider for BigQuerySource query string
[ https://issues.apache.org/jira/browse/BEAM-9305?focusedWorklogId=397467=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-397467 ] ASF GitHub Bot logged work on BEAM-9305: Author: ASF GitHub Bot Created on: 04/Mar/20 10:26 Start Date: 04/Mar/20 10:26 Worklog Time Spent: 10m Work Description: EDjur commented on issue #11040: [BEAM-9305] Allow value provider query strings in _CustomBigQuerySource URL: https://github.com/apache/beam/pull/11040#issuecomment-594440681 R: @pabloem As I have little experience with the BQ module, I'm a bit unsure if the test was placed in the correct module. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 397467) Time Spent: 20m (was: 10m) > Support ValueProvider for BigQuerySource query string > - > > Key: BEAM-9305 > URL: https://issues.apache.org/jira/browse/BEAM-9305 > Project: Beam > Issue Type: New Feature > Components: io-py-gcp >Reporter: Elias Djurfeldt >Assignee: Elias Djurfeldt >Priority: Minor > Time Spent: 20m > Remaining Estimate: 0h > > Users should be able to use ValueProviders for the query string in > BigQuerySource. > Ref: > [https://stackoverflow.com/questions/60146887/expected-eta-to-avail-pipeline-i-o-and-runtime-parameters-in-apache-beam-gcp-dat/60170614?noredirect=1#comment106464448_60170614] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9305) Support ValueProvider for BigQuerySource query string
[ https://issues.apache.org/jira/browse/BEAM-9305?focusedWorklogId=397464=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-397464 ] ASF GitHub Bot logged work on BEAM-9305: Author: ASF GitHub Bot Created on: 04/Mar/20 10:23 Start Date: 04/Mar/20 10:23 Worklog Time Spent: 10m Work Description: EDjur commented on pull request #11040: [BEAM-9305] Allow value provider query strings in _CustomBigQuerySource URL: https://github.com/apache/beam/pull/11040 This PR relates to https://issues.apache.org/jira/browse/BEAM-9305 and allows for the usage of ValueProvider query strings in _ReadFromBigQuery. Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] Update `CHANGES.md` with noteworthy changes. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[