[jira] [Commented] (BEAM-4096) BigQueryIO ValueProvider support for Method and Triggering Frequency
[ https://issues.apache.org/jira/browse/BEAM-4096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16447936#comment-16447936 ] Jan Peuker commented on BEAM-4096: -- Thanks Eugene, very helpful, I suppose you mean the fact that in BigQueryIO.Write.expand different PTransforms are created per method. In my tests using a manually constructed graph (i.e. StaticValueProvider) it worked, but that's probably because I haven't done enough dynamic testing yet. > BigQueryIO ValueProvider support for Method and Triggering Frequency > > > Key: BEAM-4096 > URL: https://issues.apache.org/jira/browse/BEAM-4096 > Project: Beam > Issue Type: Improvement > Components: io-java-gcp >Affects Versions: 2.4.0 >Reporter: Ryan McDowell >Priority: Minor > Fix For: 2.5.0 > > > Enhance BigQueryIO to accept ValueProviders for: > * withMethod(..) > * withTriggeringFrequency(..) > * withNumFileShards(..) > It would allow Dataflow templates to accept these parameters at runtime > instead of being hardcoded. This opens up the ability to create Dataflow > templates which allow users to flip back-and-forth between batch and > streaming inserts. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-4096) BigQueryIO ValueProvider support for Method and Triggering Frequency
[ https://issues.apache.org/jira/browse/BEAM-4096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16447521#comment-16447521 ] Eugene Kirpichov commented on BEAM-4096: Quick comment: making Method a ValueProvider is currently impossible because ValueProvider's can not affect the shape of the graph (because their values are unknown at the time the graph is constructed). > BigQueryIO ValueProvider support for Method and Triggering Frequency > > > Key: BEAM-4096 > URL: https://issues.apache.org/jira/browse/BEAM-4096 > Project: Beam > Issue Type: Improvement > Components: io-java-gcp >Affects Versions: 2.4.0 >Reporter: Ryan McDowell >Priority: Minor > Fix For: 2.5.0 > > > Enhance BigQueryIO to accept ValueProviders for: > * withMethod(..) > * withTriggeringFrequency(..) > * withNumFileShards(..) > It would allow Dataflow templates to accept these parameters at runtime > instead of being hardcoded. This opens up the ability to create Dataflow > templates which allow users to flip back-and-forth between batch and > streaming inserts. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-4096) BigQueryIO ValueProvider support for Method and Triggering Frequency
[ https://issues.apache.org/jira/browse/BEAM-4096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440367#comment-16440367 ] Ryan McDowell commented on BEAM-4096: - Added withNumFileShards(..) to the description. > BigQueryIO ValueProvider support for Method and Triggering Frequency > > > Key: BEAM-4096 > URL: https://issues.apache.org/jira/browse/BEAM-4096 > Project: Beam > Issue Type: Improvement > Components: io-java-gcp >Affects Versions: 2.4.0 >Reporter: Ryan McDowell >Priority: Minor > Fix For: 2.5.0 > > > Enhance BigQueryIO to accept ValueProviders for: > * withMethod(..) > * withTriggeringFrequency(..) > * withNumFileShards(..) > It would allow Dataflow templates to accept these parameters at runtime > instead of being hardcoded. This opens up the ability to create Dataflow > templates which allow users to flip back-and-forth between batch and > streaming inserts. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-4096) BigQueryIO ValueProvider support for Method and Triggering Frequency
[ https://issues.apache.org/jira/browse/BEAM-4096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440364#comment-16440364 ] Jan Peuker commented on BEAM-4096: -- Hi this is Jan, all set up with Jira now. Small addition here: We also need to be change withNumFileShards to a ValueProviders which is a required option right now. The default 1000 mentioned in the JavaDoc is incorrect and tends to cause OutOfMemoryError in DataflowRunner. From my current, native, benchmarks it seems a more sensible suggestion for most cases seems to have 100 shards (easy to calculate shard on powers of 2 and reaches common chunk sizes earlier). > BigQueryIO ValueProvider support for Method and Triggering Frequency > > > Key: BEAM-4096 > URL: https://issues.apache.org/jira/browse/BEAM-4096 > Project: Beam > Issue Type: Improvement > Components: io-java-gcp >Affects Versions: 2.4.0 >Reporter: Ryan McDowell >Priority: Minor > Fix For: 2.5.0 > > > Enhance BigQueryIO to accept ValueProviders for: > * withMethod(..) > * withTriggeringFrequency(..) > It would allow Dataflow templates to accept these parameters at runtime > instead of being hardcoded. This opens up the ability to create Dataflow > templates which allow users to flip back-and-forth between batch and > streaming inserts. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-4096) BigQueryIO ValueProvider support for Method and Triggering Frequency
[ https://issues.apache.org/jira/browse/BEAM-4096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440217#comment-16440217 ] Chamikara Jayalath commented on BEAM-4096: -- Looks like you haven't been given the Jira contributors role yet. [~kenn] might be able to add you. > BigQueryIO ValueProvider support for Method and Triggering Frequency > > > Key: BEAM-4096 > URL: https://issues.apache.org/jira/browse/BEAM-4096 > Project: Beam > Issue Type: Improvement > Components: io-java-gcp >Affects Versions: 2.4.0 >Reporter: Ryan McDowell >Priority: Minor > Fix For: 2.5.0 > > > Enhance BigQueryIO to accept ValueProviders for: > * withMethod(..) > * withTriggeringFrequency(..) > It would allow Dataflow templates to accept these parameters at runtime > instead of being hardcoded. This opens up the ability to create Dataflow > templates which allow users to flip back-and-forth between batch and > streaming inserts. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-4096) BigQueryIO ValueProvider support for Method and Triggering Frequency
[ https://issues.apache.org/jira/browse/BEAM-4096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440215#comment-16440215 ] Ryan McDowell commented on BEAM-4096: - I have a member on my team who is interested in taking it on this week but I don't believe they're on the Jira yet. Feel free to assign to me and I'll update once they're setup. > BigQueryIO ValueProvider support for Method and Triggering Frequency > > > Key: BEAM-4096 > URL: https://issues.apache.org/jira/browse/BEAM-4096 > Project: Beam > Issue Type: Improvement > Components: io-java-gcp >Affects Versions: 2.4.0 >Reporter: Ryan McDowell >Priority: Minor > Fix For: 2.5.0 > > > Enhance BigQueryIO to accept ValueProviders for: > * withMethod(..) > * withTriggeringFrequency(..) > It would allow Dataflow templates to accept these parameters at runtime > instead of being hardcoded. This opens up the ability to create Dataflow > templates which allow users to flip back-and-forth between batch and > streaming inserts. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-4096) BigQueryIO ValueProvider support for Method and Triggering Frequency
[ https://issues.apache.org/jira/browse/BEAM-4096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440214#comment-16440214 ] Chamikara Jayalath commented on BEAM-4096: -- Ryan, thanks for filing the JIRA. Is this something you hope to work on ? > BigQueryIO ValueProvider support for Method and Triggering Frequency > > > Key: BEAM-4096 > URL: https://issues.apache.org/jira/browse/BEAM-4096 > Project: Beam > Issue Type: Improvement > Components: io-java-gcp >Affects Versions: 2.4.0 >Reporter: Ryan McDowell >Assignee: Chamikara Jayalath >Priority: Minor > Fix For: 2.5.0 > > > Enhance BigQueryIO to accept ValueProviders for: > * withMethod(..) > * withTriggeringFrequency(..) > It would allow Dataflow templates to accept these parameters at runtime > instead of being hardcoded. This opens up the ability to create Dataflow > templates which allow users to flip back-and-forth between batch and > streaming inserts. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
