[
https://issues.apache.org/jira/browse/BEAM-6841?focusedWorklogId=219698&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-219698
]
ASF GitHub Bot logged work on BEAM-6841:
----------------------------------------
Author: ASF GitHub Bot
Created on: 27/Mar/19 23:08
Start Date: 27/Mar/19 23:08
Worklog Time Spent: 10m
Work Description: chamikaramj commented on pull request #8061:
[BEAM-6841] Add support for reading query results using the BigQuery storage
API.
URL: https://github.com/apache/beam/pull/8061#discussion_r269300789
##########
File path:
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java
##########
@@ -914,6 +933,181 @@ void cleanup(PassThroughThenCleanup.ContextContainer c)
throws Exception {
return rows.apply(new PassThroughThenCleanup<>(cleanupOperation,
jobIdTokenView));
}
+ private PCollection<T> expandForDirectRead(PBegin input, Coder<T>
outputCoder) {
+ ValueProvider<TableReference> tableProvider = getTableProvider();
+ Pipeline p = input.getPipeline();
+ if (tableProvider != null) {
+ // No job ID is required. Read directly from BigQuery storage.
+ return p.apply(
+ org.apache.beam.sdk.io.Read.from(
+ BigQueryStorageTableSource.create(
+ tableProvider,
+ getReadOptions(),
+ getParseFn(),
+ outputCoder,
+ getBigQueryServices())));
+ }
+
+ checkArgument(
+ getReadOptions() == null,
+ "Invalid BigQueryIO.Read: Specifies table read options, "
+ + "which only applies when reading from a table");
+
+ //
+ // N.B. All of the code below exists because the BigQuery storage API
can't (yet) read from
+ // all anonymous tables, so we need the job ID to reason about the name
of the destination
+ // table for the query to read the data and subsequently delete the
table and dataset. Once
+ // the storage API can handle anonymous tables, the storage source
should be modified to use
Review comment:
Can you create a JIRA for this and mention it in a TODO here.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 219698)
Time Spent: 2h 10m (was: 2h)
> Support reading query results with the BigQuery storage API
> -----------------------------------------------------------
>
> Key: BEAM-6841
> URL: https://issues.apache.org/jira/browse/BEAM-6841
> Project: Beam
> Issue Type: New Feature
> Components: io-java-gcp
> Reporter: Kenneth Jung
> Assignee: Kenneth Jung
> Priority: Minor
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
