Here are the JIRAs. BigQuery custom sink: https://issues.apache.org/jira/browse/BEAM-2801 BigQuery custom source: https://issues.apache.org/jira/browse/BEAM-1440
Volunteers are welcome :) Thanks, Cham On Wed, Aug 23, 2017 at 7:21 PM Chamikara Jayalath <[email protected]> wrote: > I don't think we have JIRAs. I'll go ahead and create them. > > Thanks, > Cham > > > On Wed, Aug 23, 2017 at 6:33 PM Reuven Lax <[email protected]> > wrote: > >> Cham, >> >> Do we have JIRAs tracking these features? >> >> Reuven >> >> On Wed, Aug 23, 2017 at 6:26 PM, Asha Rostamianfar < >> [email protected]> wrote: >> >> > It's for a batch job. I'll think about possible workarounds then >> (perhaps I >> > can have a prestage process that reads the necessary info from the files >> > and passes as an argument). Thanks everyone! >> > >> > On Wed, Aug 23, 2017 at 9:17 PM, Chamikara Jayalath < >> [email protected]> >> > wrote: >> > >> > > Ah, actually for this feature I think we need a BigQuery custom sink. >> > Which >> > > also doesn't have an ETA currently. >> > > >> > > Thanks, >> > > Cham >> > > >> > > On Wed, Aug 23, 2017 at 6:14 PM Chamikara Jayalath < >> [email protected] >> > > >> > > wrote: >> > > >> > > > We have to develop a BigQuery BoundedSource for this first. So I >> expect >> > > > support for Python SDK to be at least few months away. >> > > > >> > > > Thanks, >> > > > Cham >> > > > >> > > > >> > > > On Wed, Aug 23, 2017 at 6:10 PM Asha Rostamianfar >> > > > <[email protected]> wrote: >> > > > >> > > >> ah, ok..is there an existing JIRA entry with some approximate ETA? >> > (i.e. >> > > >> weeks, months, or quarters). >> > > >> >> > > >> Thanks again. >> > > >> >> > > >> On Wed, Aug 23, 2017 at 8:58 PM, Reuven Lax >> <[email protected] >> > > >> > > >> wrote: >> > > >> >> > > >> > Not yet, only in Java so far. >> > > >> > >> > > >> > On Wed, Aug 23, 2017 at 5:53 PM, Asha Rostamianfar < >> > > >> > [email protected]> wrote: >> > > >> > >> > > >> > > Thanks, Eugene! Is this also supported in the python SDK? >> > > >> > > >> > > >> > > On Wed, Aug 23, 2017 at 8:39 PM, Eugene Kirpichov < >> > > >> > > [email protected]> wrote: >> > > >> > > >> > > >> > > > Yes, this is possible using the BigQueryIO.write().to( >> > > >> > > DynamicDestinations) >> > > >> > > > API. It allows you to write different values to different >> tables >> > > >> with >> > > >> > > > different schemas. >> > > >> > > > On Wed, Aug 23, 2017, 2:51 PM Asha Rostamianfar >> > > >> > > > <[email protected]> >> > > >> > > > wrote: >> > > >> > > > >> > > >> > > > > Hi, >> > > >> > > > > >> > > >> > > > > I'm wondering whether it's possible to dynamically >> generate a >> > > >> > BigQuery >> > > >> > > > > schema based on input. For instance, the fields would be >> > > >> specified in >> > > >> > > one >> > > >> > > > > or more input files that are read and processed as part of >> the >> > > >> > > pipeline. >> > > >> > > > > >> > > >> > > > > Thanks, >> > > >> > > > > Asha >> > > >> > > > > >> > > >> > > > >> > > >> > > >> > > >> > >> > > >> >> > > > >> > > >> > >> >
