It's for a batch job. I'll think about possible workarounds then (perhaps I can have a prestage process that reads the necessary info from the files and passes as an argument). Thanks everyone!
On Wed, Aug 23, 2017 at 9:17 PM, Chamikara Jayalath <[email protected]> wrote: > Ah, actually for this feature I think we need a BigQuery custom sink. Which > also doesn't have an ETA currently. > > Thanks, > Cham > > On Wed, Aug 23, 2017 at 6:14 PM Chamikara Jayalath <[email protected]> > wrote: > > > We have to develop a BigQuery BoundedSource for this first. So I expect > > support for Python SDK to be at least few months away. > > > > Thanks, > > Cham > > > > > > On Wed, Aug 23, 2017 at 6:10 PM Asha Rostamianfar > > <[email protected]> wrote: > > > >> ah, ok..is there an existing JIRA entry with some approximate ETA? (i.e. > >> weeks, months, or quarters). > >> > >> Thanks again. > >> > >> On Wed, Aug 23, 2017 at 8:58 PM, Reuven Lax <[email protected]> > >> wrote: > >> > >> > Not yet, only in Java so far. > >> > > >> > On Wed, Aug 23, 2017 at 5:53 PM, Asha Rostamianfar < > >> > [email protected]> wrote: > >> > > >> > > Thanks, Eugene! Is this also supported in the python SDK? > >> > > > >> > > On Wed, Aug 23, 2017 at 8:39 PM, Eugene Kirpichov < > >> > > [email protected]> wrote: > >> > > > >> > > > Yes, this is possible using the BigQueryIO.write().to( > >> > > DynamicDestinations) > >> > > > API. It allows you to write different values to different tables > >> with > >> > > > different schemas. > >> > > > On Wed, Aug 23, 2017, 2:51 PM Asha Rostamianfar > >> > > > <[email protected]> > >> > > > wrote: > >> > > > > >> > > > > Hi, > >> > > > > > >> > > > > I'm wondering whether it's possible to dynamically generate a > >> > BigQuery > >> > > > > schema based on input. For instance, the fields would be > >> specified in > >> > > one > >> > > > > or more input files that are read and processed as part of the > >> > > pipeline. > >> > > > > > >> > > > > Thanks, > >> > > > > Asha > >> > > > > > >> > > > > >> > > > >> > > >> > > >
