Hi Josh, You raise a good point. I think we had put this check in (long before partition tables existed) because we need schema to create a table and we assumed the number of tables would be unbounded. But now it's an outdated check, overly conservative, and probably should be removed.
Would you like to send a PR to fix this? Thanks, Dan On Fri, Apr 7, 2017 at 10:03 AM, Josh <[email protected]> wrote: > Hi all, > > I have a use case where I want to stream into BigQuery, using a tablespec > but with CreateDisposition.CREATE_NEVER.I want to partition/shard my data > by date, and use BigQuery's date partitioning feature within a single table > (rather than creating a new BigQuery table for every day). In this case > writes would be made to a partition in a single table, e.g. > `my-project:dataset.my_table$20170407`, and in my tablespec I would just > be choosing the partition decorator using the window. > > Unfortunately this doesn't seem possible with BigQueryIO at the moment, > because it requires me to use CreateDisposition.CREATE_IF_NEEDED. I can't > use CreateDisposition.CREATE_IF_NEEDED because it requires me to provide > a table schema and my BigQuery schema isn't available at compile time. > > Is there any good reason why CREATE_NEVER is not allowed when using a > tablespec? > > Thanks, > Josh >
