[jira] [Commented] (BEAM-8023) Allow specifying BigQuery Storage API readOptions at runtime
[ https://issues.apache.org/jira/browse/BEAM-8023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17131266#comment-17131266 ] Beam JIRA Bot commented on BEAM-8023: - This issue was marked "stale-assigned" and has not received a public comment in 7 days. It is now automatically unassigned. If you are still working on it, you can assign it to yourself again. Please also give an update about the status of the work. > Allow specifying BigQuery Storage API readOptions at runtime > > > Key: BEAM-8023 > URL: https://issues.apache.org/jira/browse/BEAM-8023 > Project: Beam > Issue Type: Improvement > Components: io-java-gcp >Reporter: Jeff Klukas >Priority: P3 > Time Spent: 2h 50m > Remaining Estimate: 0h > > We have support in the Java SDK for using the BigQuery Storage API for reads, > but only the target query or table is supported as a ValueProvider to be > specified at runtime. AFAICT, there is no reason we can't delay specifying > readOptions until runtime as well. > The readOptions are accessed by BigQueryStorageTableSource in getTargetTable; > I believe that's occurring at runtime, but I'd love for someone with deeper > BoundedSource knowledge to confirm that. > I'd advocate for adding new methods > `TypedRead.withSelectedFields(ValueProvider> value)` and > `TypedRead.withRowRestriction(ValueProvider value)`. The existing > `withReadOptions` method would then populate the other two as > StaticValueProviders. Perhaps we'd want to deprecate `withReadOptions` in > favor or specifying individual read options as separate parameters. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-8023) Allow specifying BigQuery Storage API readOptions at runtime
[ https://issues.apache.org/jira/browse/BEAM-8023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17121926#comment-17121926 ] Kenneth Knowles commented on BEAM-8023: --- This issue is assigned but has not received an update in 30 days so it has been labeled "stale-assigned". If you are still working on the issue, please give an update and remove the label. If you are no longer working on the issue, please unassign so someone else may work on it. In 7 days the issue will be automatically unassigned. > Allow specifying BigQuery Storage API readOptions at runtime > > > Key: BEAM-8023 > URL: https://issues.apache.org/jira/browse/BEAM-8023 > Project: Beam > Issue Type: Improvement > Components: io-java-gcp >Reporter: Jeff Klukas >Assignee: Kenneth Jung >Priority: P3 > Labels: stale-assigned > Time Spent: 2h 50m > Remaining Estimate: 0h > > We have support in the Java SDK for using the BigQuery Storage API for reads, > but only the target query or table is supported as a ValueProvider to be > specified at runtime. AFAICT, there is no reason we can't delay specifying > readOptions until runtime as well. > The readOptions are accessed by BigQueryStorageTableSource in getTargetTable; > I believe that's occurring at runtime, but I'd love for someone with deeper > BoundedSource knowledge to confirm that. > I'd advocate for adding new methods > `TypedRead.withSelectedFields(ValueProvider> value)` and > `TypedRead.withRowRestriction(ValueProvider value)`. The existing > `withReadOptions` method would then populate the other two as > StaticValueProviders. Perhaps we'd want to deprecate `withReadOptions` in > favor or specifying individual read options as separate parameters. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-8023) Allow specifying BigQuery Storage API readOptions at runtime
[ https://issues.apache.org/jira/browse/BEAM-8023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16913647#comment-16913647 ] Chamikara Jayalath commented on BEAM-8023: -- Wow that was fast :) Thanks Ken. I'll take a look. > Allow specifying BigQuery Storage API readOptions at runtime > > > Key: BEAM-8023 > URL: https://issues.apache.org/jira/browse/BEAM-8023 > Project: Beam > Issue Type: Improvement > Components: io-java-gcp >Reporter: Jeff Klukas >Assignee: Kenneth Jung >Priority: Minor > Time Spent: 20m > Remaining Estimate: 0h > > We have support in the Java SDK for using the BigQuery Storage API for reads, > but only the target query or table is supported as a ValueProvider to be > specified at runtime. AFAICT, there is no reason we can't delay specifying > readOptions until runtime as well. > The readOptions are accessed by BigQueryStorageTableSource in getTargetTable; > I believe that's occurring at runtime, but I'd love for someone with deeper > BoundedSource knowledge to confirm that. > I'd advocate for adding new methods > `TypedRead.withSelectedFields(ValueProvider> value)` and > `TypedRead.withRowRestriction(ValueProvider value)`. The existing > `withReadOptions` method would then populate the other two as > StaticValueProviders. Perhaps we'd want to deprecate `withReadOptions` in > favor or specifying individual read options as separate parameters. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (BEAM-8023) Allow specifying BigQuery Storage API readOptions at runtime
[ https://issues.apache.org/jira/browse/BEAM-8023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16913610#comment-16913610 ] Kenneth Jung commented on BEAM-8023: This seems fairly straightforward. PR forthcoming. > Allow specifying BigQuery Storage API readOptions at runtime > > > Key: BEAM-8023 > URL: https://issues.apache.org/jira/browse/BEAM-8023 > Project: Beam > Issue Type: Improvement > Components: io-java-gcp >Reporter: Jeff Klukas >Priority: Minor > > We have support in the Java SDK for using the BigQuery Storage API for reads, > but only the target query or table is supported as a ValueProvider to be > specified at runtime. AFAICT, there is no reason we can't delay specifying > readOptions until runtime as well. > The readOptions are accessed by BigQueryStorageTableSource in getTargetTable; > I believe that's occurring at runtime, but I'd love for someone with deeper > BoundedSource knowledge to confirm that. > I'd advocate for adding new methods > `TypedRead.withSelectedFields(ValueProvider> value)` and > `TypedRead.withRowRestriction(ValueProvider value)`. The existing > `withReadOptions` method would then populate the other two as > StaticValueProviders. Perhaps we'd want to deprecate `withReadOptions` in > favor or specifying individual read options as separate parameters. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (BEAM-8023) Allow specifying BigQuery Storage API readOptions at runtime
[ https://issues.apache.org/jira/browse/BEAM-8023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16913498#comment-16913498 ] Jeff Klukas commented on BEAM-8023: --- I would send a PR for this _eventually_, but it may be several months out before I can prioritize it. I would be delighted if somebody could take it up earlier and I would be happy to be a reviewer. > Allow specifying BigQuery Storage API readOptions at runtime > > > Key: BEAM-8023 > URL: https://issues.apache.org/jira/browse/BEAM-8023 > Project: Beam > Issue Type: Improvement > Components: io-java-gcp >Reporter: Jeff Klukas >Priority: Minor > > We have support in the Java SDK for using the BigQuery Storage API for reads, > but only the target query or table is supported as a ValueProvider to be > specified at runtime. AFAICT, there is no reason we can't delay specifying > readOptions until runtime as well. > The readOptions are accessed by BigQueryStorageTableSource in getTargetTable; > I believe that's occurring at runtime, but I'd love for someone with deeper > BoundedSource knowledge to confirm that. > I'd advocate for adding new methods > `TypedRead.withSelectedFields(ValueProvider> value)` and > `TypedRead.withRowRestriction(ValueProvider value)`. The existing > `withReadOptions` method would then populate the other two as > StaticValueProviders. Perhaps we'd want to deprecate `withReadOptions` in > favor or specifying individual read options as separate parameters. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (BEAM-8023) Allow specifying BigQuery Storage API readOptions at runtime
[ https://issues.apache.org/jira/browse/BEAM-8023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16913488#comment-16913488 ] Chamikara Jayalath commented on BEAM-8023: -- Yeah, I believe this should be possible. [~jeff.klu...@gmail.com] will you be able to send a PR for this ? If not [~aryann] or [~kjung520] may be interested in adding this functionality. > Allow specifying BigQuery Storage API readOptions at runtime > > > Key: BEAM-8023 > URL: https://issues.apache.org/jira/browse/BEAM-8023 > Project: Beam > Issue Type: Improvement > Components: io-java-gcp >Reporter: Jeff Klukas >Priority: Minor > > We have support in the Java SDK for using the BigQuery Storage API for reads, > but only the target query or table is supported as a ValueProvider to be > specified at runtime. AFAICT, there is no reason we can't delay specifying > readOptions until runtime as well. > The readOptions are accessed by BigQueryStorageTableSource in getTargetTable; > I believe that's occurring at runtime, but I'd love for someone with deeper > BoundedSource knowledge to confirm that. > I'd advocate for adding new methods > `TypedRead.withSelectedFields(ValueProvider> value)` and > `TypedRead.withRowRestriction(ValueProvider value)`. The existing > `withReadOptions` method would then populate the other two as > StaticValueProviders. Perhaps we'd want to deprecate `withReadOptions` in > favor or specifying individual read options as separate parameters. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (BEAM-8023) Allow specifying BigQuery Storage API readOptions at runtime
[ https://issues.apache.org/jira/browse/BEAM-8023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16913066#comment-16913066 ] Ismaël Mejía commented on BEAM-8023: [~chamikara] or [~pabloem] any comment? > Allow specifying BigQuery Storage API readOptions at runtime > > > Key: BEAM-8023 > URL: https://issues.apache.org/jira/browse/BEAM-8023 > Project: Beam > Issue Type: Improvement > Components: io-java-gcp >Reporter: Jeff Klukas >Priority: Minor > > We have support in the Java SDK for using the BigQuery Storage API for reads, > but only the target query or table is supported as a ValueProvider to be > specified at runtime. AFAICT, there is no reason we can't delay specifying > readOptions until runtime as well. > The readOptions are accessed by BigQueryStorageTableSource in getTargetTable; > I believe that's occurring at runtime, but I'd love for someone with deeper > BoundedSource knowledge to confirm that. > I'd advocate for adding new methods > `TypedRead.withSelectedFields(ValueProvider> value)` and > `TypedRead.withRowRestriction(ValueProvider value)`. The existing > `withReadOptions` method would then populate the other two as > StaticValueProviders. Perhaps we'd want to deprecate `withReadOptions` in > favor or specifying individual read options as separate parameters. -- This message was sent by Atlassian Jira (v8.3.2#803003)
