[jira] [Work logged] (BEAM-11587) Support pd.read_gbq and DataFrame.to_gbq

[ASF GitHub Bot (Jira)]

     [ 
https://issues.apache.org/jira/browse/BEAM-11587?focusedWorklogId=766963&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-766963
 ]

ASF GitHub Bot logged work on BEAM-11587:
-----------------------------------------

                Author: ASF GitHub Bot
            Created on: 06/May/22 00:13
            Start Date: 06/May/22 00:13
    Worklog Time Spent: 10m 
      Work Description: svetakvsundhar commented on code in PR #17159:
URL: https://github.com/apache/beam/pull/17159#discussion_r866406831


##########
sdks/python/apache_beam/io/gcp/bigquery.py:
##########
@@ -2525,6 +2526,12 @@ def _get_pipeline_details(unused_elm):
                 **self._kwargs))
         | _PassThroughThenCleanupTempDatasets(project_to_cleanup_pcoll))
 
+  def get_pcoll_from_schema(table_schema):
+    pcoll_val = apache_beam.io.gcp.bigquery_schema_tools.\
+        produce_pcoll_with_schema(table_schema)
+    return beam.Map(lambda values: pcoll_val(**values)).with_output_types(

Review Comment:
   @TheNeuralBit , tried this here but looks like the pickling still had an 
error. Something to note is that this is able to run fine on Colab notebook via 
DirectRunner 
(https://colab.sandbox.google.com/drive/1_BkzBwCLB7OjR1jGGzCkxXFHreUZgCZl#revisionId=0BzxzuriHpr5fcXNOWHcvcGlqYTBWWUZxRWdQaFIrVVFYV0Y4PQ&scrollTo=YITcZHkoZR59)
 
   
   Is there a way we can further see which coder is being used? 





Issue Time Tracking
-------------------

    Worklog Id:     (was: 766963)
    Time Spent: 6h 50m  (was: 6h 40m)

> Support pd.read_gbq and DataFrame.to_gbq
> ----------------------------------------
>
>                 Key: BEAM-11587
>                 URL: https://issues.apache.org/jira/browse/BEAM-11587
>             Project: Beam
>          Issue Type: New Feature
>          Components: dsl-dataframe, io-py-gcp, sdk-py-core
>            Reporter: Brian Hulette
>            Assignee: Svetak Vihaan Sundhar
>            Priority: P3
>              Labels: dataframe-api
>          Time Spent: 6h 50m
>  Remaining Estimate: 0h
>
> We should support 
> [read_gbq|https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_gbq.html]
>  and 
> [to_gbq|https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.to_gbq.html]
>  in the DataFrame API when gcp extras are installed.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)