[jira] [Commented] (BEAM-2390) allow user to use .setTimePartitioning in BigQueryIO.write
[ https://issues.apache.org/jira/browse/BEAM-2390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16032409#comment-16032409 ] Eric Johston commented on BEAM-2390: My initial commit had some errors in that a) TimePartitioning is not serializable, and b) CreateTables#possibleCreate fails when TableDestination includes a partition with $ I've made changes to these by storing TimePartitioning in Json format similar to how the schemas are propagated. I've also modified the table creation such that when creating a table Beam only looks at the part before $. It seems to be working now (running this from my own fork in production) > allow user to use .setTimePartitioning in BigQueryIO.write > -- > > Key: BEAM-2390 > URL: https://issues.apache.org/jira/browse/BEAM-2390 > Project: Beam > Issue Type: Improvement > Components: beam-model-runner-api >Affects Versions: 2.0.0 >Reporter: Eric Johston >Assignee: Kenneth Knowles > Labels: easyfix, features, newbie > Fix For: 2.0.0 > > Original Estimate: 2h > Remaining Estimate: 2h > > Currently when writing to a table with BigQueryIO sink, there is no way to > create a new table that is date partitioned. This would be very useful, since > currently the only way to do this is by manually creating a table ahead of > time. We should be able to leverage the automatic table creation > functionality for date partitioned tables. > The best way to do this would be to have a withTimePartitioning method in the > BigQueryIO class. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (BEAM-2390) allow user to use .setTimePartitioning in BigQueryIO.write
[ https://issues.apache.org/jira/browse/BEAM-2390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16030499#comment-16030499 ] Eric Johston commented on BEAM-2390: I've created a pull request with the functionality I believe is needed: https://github.com/apache/beam/pull/3266 > allow user to use .setTimePartitioning in BigQueryIO.write > -- > > Key: BEAM-2390 > URL: https://issues.apache.org/jira/browse/BEAM-2390 > Project: Beam > Issue Type: Improvement > Components: beam-model-runner-api >Affects Versions: 2.0.0 >Reporter: Eric Johston >Assignee: Kenneth Knowles > Labels: easyfix, features, newbie > Fix For: 2.0.0 > > Original Estimate: 2h > Remaining Estimate: 2h > > Currently when writing to a table with BigQueryIO sink, there is no way to > create a new table that is date partitioned. This would be very useful, since > currently the only way to do this is by manually creating a table ahead of > time. We should be able to leverage the automatic table creation > functionality for date partitioned tables. > The best way to do this would be to have a withTimePartitioning method in the > BigQueryIO class. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (BEAM-2390) allow user to use .setTimePartitioning in BigQueryIO.write
Eric Johston created BEAM-2390: -- Summary: allow user to use .setTimePartitioning in BigQueryIO.write Key: BEAM-2390 URL: https://issues.apache.org/jira/browse/BEAM-2390 Project: Beam Issue Type: Improvement Components: beam-model-runner-api Affects Versions: 2.0.0 Reporter: Eric Johston Assignee: Kenneth Knowles Fix For: 2.0.0 Currently when writing to a table with BigQueryIO sink, there is no way to create a new table that is date partitioned. This would be very useful, since currently the only way to do this is by manually creating a table ahead of time. We should be able to leverage the automatic table creation functionality for date partitioned tables. The best way to do this would be to have a withTimePartitioning method in the BigQueryIO class. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
