[jira] [Comment Edited] (BEAM-4096) BigQueryIO ValueProvider support for Method and Triggering Frequency
[ https://issues.apache.org/jira/browse/BEAM-4096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440364#comment-16440364 ] Jan Peuker edited comment on BEAM-4096 at 4/17/18 5:40 AM: --- Hi this is Jan, all set up with Jira now. Small addition here: We also need to change withNumFileShards to a ValueProvider which is a required option right now. The default 1000 mentioned in the JavaDoc is incorrect and tends to cause OutOfMemoryError in DataflowRunner. From my current, naive, benchmarks it seems a more sensible suggestion for most cases seems to have 100 shards (easy to calculate shard on powers of 10 and reaches common chunk sizes earlier). was (Author: janpeuker): Hi this is Jan, all set up with Jira now. Small addition here: We also need to change withNumFileShards to a ValueProvider which is a required option right now. The default 1000 mentioned in the JavaDoc is incorrect and tends to cause OutOfMemoryError in DataflowRunner. From my current, native, benchmarks it seems a more sensible suggestion for most cases seems to have 100 shards (easy to calculate shard on powers of 2 and reaches common chunk sizes earlier). > BigQueryIO ValueProvider support for Method and Triggering Frequency > > > Key: BEAM-4096 > URL: https://issues.apache.org/jira/browse/BEAM-4096 > Project: Beam > Issue Type: Improvement > Components: io-java-gcp >Affects Versions: 2.4.0 >Reporter: Ryan McDowell >Priority: Minor > Fix For: 2.5.0 > > > Enhance BigQueryIO to accept ValueProviders for: > * withMethod(..) > * withTriggeringFrequency(..) > * withNumFileShards(..) > It would allow Dataflow templates to accept these parameters at runtime > instead of being hardcoded. This opens up the ability to create Dataflow > templates which allow users to flip back-and-forth between batch and > streaming inserts. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (BEAM-4096) BigQueryIO ValueProvider support for Method and Triggering Frequency
[ https://issues.apache.org/jira/browse/BEAM-4096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440364#comment-16440364 ] Jan Peuker edited comment on BEAM-4096 at 4/17/18 4:19 AM: --- Hi this is Jan, all set up with Jira now. Small addition here: We also need to change withNumFileShards to a ValueProvider which is a required option right now. The default 1000 mentioned in the JavaDoc is incorrect and tends to cause OutOfMemoryError in DataflowRunner. From my current, native, benchmarks it seems a more sensible suggestion for most cases seems to have 100 shards (easy to calculate shard on powers of 2 and reaches common chunk sizes earlier). was (Author: janpeuker): Hi this is Jan, all set up with Jira now. Small addition here: We also need to be change withNumFileShards to a ValueProviders which is a required option right now. The default 1000 mentioned in the JavaDoc is incorrect and tends to cause OutOfMemoryError in DataflowRunner. From my current, native, benchmarks it seems a more sensible suggestion for most cases seems to have 100 shards (easy to calculate shard on powers of 2 and reaches common chunk sizes earlier). > BigQueryIO ValueProvider support for Method and Triggering Frequency > > > Key: BEAM-4096 > URL: https://issues.apache.org/jira/browse/BEAM-4096 > Project: Beam > Issue Type: Improvement > Components: io-java-gcp >Affects Versions: 2.4.0 >Reporter: Ryan McDowell >Priority: Minor > Fix For: 2.5.0 > > > Enhance BigQueryIO to accept ValueProviders for: > * withMethod(..) > * withTriggeringFrequency(..) > It would allow Dataflow templates to accept these parameters at runtime > instead of being hardcoded. This opens up the ability to create Dataflow > templates which allow users to flip back-and-forth between batch and > streaming inserts. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
