[jira] [Assigned] (BEAM-3489) Expose the message id of received messages within PubsubMessage
[ https://issues.apache.org/jira/browse/BEAM-3489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Tumale reassigned BEAM-3489: --- Assignee: (was: Chamikara Jayalath) > Expose the message id of received messages within PubsubMessage > --- > > Key: BEAM-3489 > URL: https://issues.apache.org/jira/browse/BEAM-3489 > Project: Beam > Issue Type: New Feature > Components: io-java-gcp >Reporter: Luke Cwik >Priority: Minor > Labels: newbie, starter > > This task is about passing forward the message id from the pubsub proto to > the java PubsubMessage. > Add a message id field to PubsubMessage. > Update the coder for PubsubMessage to encode the message id. > Update the translation from the Pubsub proto message to the Dataflow message: > https://github.com/apache/beam/blob/2e275264b21db45787833502e5e42907b05e28b8/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubUnboundedSource.java#L976 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (BEAM-3489) Expose the message id of received messages within PubsubMessage
[ https://issues.apache.org/jira/browse/BEAM-3489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Tumale reassigned BEAM-3489: --- Assignee: Chamikara Jayalath (was: Justin Tumale) > Expose the message id of received messages within PubsubMessage > --- > > Key: BEAM-3489 > URL: https://issues.apache.org/jira/browse/BEAM-3489 > Project: Beam > Issue Type: New Feature > Components: io-java-gcp >Reporter: Luke Cwik >Assignee: Chamikara Jayalath >Priority: Minor > Labels: newbie, starter > > This task is about passing forward the message id from the pubsub proto to > the java PubsubMessage. > Add a message id field to PubsubMessage. > Update the coder for PubsubMessage to encode the message id. > Update the translation from the Pubsub proto message to the Dataflow message: > https://github.com/apache/beam/blob/2e275264b21db45787833502e5e42907b05e28b8/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubUnboundedSource.java#L976 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (BEAM-3489) Expose the message id of received messages within PubsubMessage
[ https://issues.apache.org/jira/browse/BEAM-3489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Tumale reassigned BEAM-3489: --- Assignee: Justin Tumale > Expose the message id of received messages within PubsubMessage > --- > > Key: BEAM-3489 > URL: https://issues.apache.org/jira/browse/BEAM-3489 > Project: Beam > Issue Type: New Feature > Components: io-java-gcp >Reporter: Luke Cwik >Assignee: Justin Tumale >Priority: Minor > Labels: newbie, starter > > This task is about passing forward the message id from the pubsub proto to > the java PubsubMessage. > Add a message id field to PubsubMessage. > Update the coder for PubsubMessage to encode the message id. > Update the translation from the Pubsub proto message to the Dataflow message: > https://github.com/apache/beam/blob/2e275264b21db45787833502e5e42907b05e28b8/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubUnboundedSource.java#L976 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (BEAM-3665) Port BigQueryIOWriteTest off DoFnTester
[ https://issues.apache.org/jira/browse/BEAM-3665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Tumale reassigned BEAM-3665: --- Assignee: (was: Chamikara Jayalath) > Port BigQueryIOWriteTest off DoFnTester > --- > > Key: BEAM-3665 > URL: https://issues.apache.org/jira/browse/BEAM-3665 > Project: Beam > Issue Type: Sub-task > Components: io-java-gcp >Reporter: Kenneth Knowles >Priority: Major > Labels: beginner, newbie, starter > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (BEAM-3665) Port BigQueryIOWriteTest off DoFnTester
[ https://issues.apache.org/jira/browse/BEAM-3665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Tumale reassigned BEAM-3665: --- Assignee: Chamikara Jayalath (was: Justin Tumale) > Port BigQueryIOWriteTest off DoFnTester > --- > > Key: BEAM-3665 > URL: https://issues.apache.org/jira/browse/BEAM-3665 > Project: Beam > Issue Type: Sub-task > Components: io-java-gcp >Reporter: Kenneth Knowles >Assignee: Chamikara Jayalath >Priority: Major > Labels: beginner, newbie, starter > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (BEAM-2817) Bigquery queries should allow options to run in batch mode or not
[ https://issues.apache.org/jira/browse/BEAM-2817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16394296#comment-16394296 ] Justin Tumale edited comment on BEAM-2817 at 3/10/18 6:42 PM: -- This has been completed and merged. I think it will be a part of the 2.4.0 release. PR #4606 was (Author: justintumale): This has been completed and merged. I think it will be a part of the 2.4.0 release. > Bigquery queries should allow options to run in batch mode or not > - > > Key: BEAM-2817 > URL: https://issues.apache.org/jira/browse/BEAM-2817 > Project: Beam > Issue Type: Bug > Components: io-java-gcp >Affects Versions: 2.0.0 >Reporter: Lara Schmidt >Assignee: Justin Tumale >Priority: Major > Labels: newbie, starter > Fix For: 2.4.0 > > Time Spent: 1h 50m > Remaining Estimate: 0h > > When bigquery read does a query it sets the mode to batch. A batch query can > be very slow to schedule as it batches it with other queries. However it > doesn't use batch quota which is better for some cases. However, in some > cases a fast query is better (especially in timed tests). It would be a good > idea to have a configuration to the bigquery source to set this per-read. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-2817) Bigquery queries should allow options to run in batch mode or not
[ https://issues.apache.org/jira/browse/BEAM-2817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16394296#comment-16394296 ] Justin Tumale commented on BEAM-2817: - This has been completed and merged. I think it will be a part of the 2.4.0 release. > Bigquery queries should allow options to run in batch mode or not > - > > Key: BEAM-2817 > URL: https://issues.apache.org/jira/browse/BEAM-2817 > Project: Beam > Issue Type: Bug > Components: io-java-gcp >Affects Versions: 2.0.0 >Reporter: Lara Schmidt >Assignee: Justin Tumale >Priority: Major > Labels: newbie, starter > Fix For: 2.4.0 > > Time Spent: 1h 50m > Remaining Estimate: 0h > > When bigquery read does a query it sets the mode to batch. A batch query can > be very slow to schedule as it batches it with other queries. However it > doesn't use batch quota which is better for some cases. However, in some > cases a fast query is better (especially in timed tests). It would be a good > idea to have a configuration to the bigquery source to set this per-read. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (BEAM-2817) Bigquery queries should allow options to run in batch mode or not
[ https://issues.apache.org/jira/browse/BEAM-2817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Tumale resolved BEAM-2817. - Resolution: Done Fix Version/s: 2.4.0 > Bigquery queries should allow options to run in batch mode or not > - > > Key: BEAM-2817 > URL: https://issues.apache.org/jira/browse/BEAM-2817 > Project: Beam > Issue Type: Bug > Components: io-java-gcp >Affects Versions: 2.0.0 >Reporter: Lara Schmidt >Assignee: Justin Tumale >Priority: Major > Labels: newbie, starter > Fix For: 2.4.0 > > Time Spent: 1h 50m > Remaining Estimate: 0h > > When bigquery read does a query it sets the mode to batch. A batch query can > be very slow to schedule as it batches it with other queries. However it > doesn't use batch quota which is better for some cases. However, in some > cases a fast query is better (especially in timed tests). It would be a good > idea to have a configuration to the bigquery source to set this per-read. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Issue Comment Deleted] (BEAM-3665) Port BigQueryIOWriteTest off DoFnTester
[ https://issues.apache.org/jira/browse/BEAM-3665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Tumale updated BEAM-3665: Comment: was deleted (was: Hi, please assign to me when you get a chance [~kenn]. I recently gained experience with some of the BigQuery code on my last pull request #4606 BEAM-2817. Thanks h1. ) > Port BigQueryIOWriteTest off DoFnTester > --- > > Key: BEAM-3665 > URL: https://issues.apache.org/jira/browse/BEAM-3665 > Project: Beam > Issue Type: Sub-task > Components: io-java-gcp >Reporter: Kenneth Knowles >Assignee: Justin Tumale >Priority: Major > Labels: beginner, newbie, starter > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (BEAM-3665) Port BigQueryIOWriteTest off DoFnTester
[ https://issues.apache.org/jira/browse/BEAM-3665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Tumale reassigned BEAM-3665: --- Assignee: Justin Tumale > Port BigQueryIOWriteTest off DoFnTester > --- > > Key: BEAM-3665 > URL: https://issues.apache.org/jira/browse/BEAM-3665 > Project: Beam > Issue Type: Sub-task > Components: io-java-gcp >Reporter: Kenneth Knowles >Assignee: Justin Tumale >Priority: Major > Labels: beginner, newbie, starter > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (BEAM-3665) Port BigQueryIOWriteTest off DoFnTester
[ https://issues.apache.org/jira/browse/BEAM-3665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16388421#comment-16388421 ] Justin Tumale edited comment on BEAM-3665 at 3/6/18 7:55 PM: - Hi, please assign to me when you get a chance [~kenn]. I recently gained experience with some of the BigQuery code on my last pull request #4606 BEAM-2817. Thanks h1. was (Author: justintumale): Hi, please assign to me when you get a chance [~kenn]. Thanks > Port BigQueryIOWriteTest off DoFnTester > --- > > Key: BEAM-3665 > URL: https://issues.apache.org/jira/browse/BEAM-3665 > Project: Beam > Issue Type: Sub-task > Components: io-java-gcp >Reporter: Kenneth Knowles >Priority: Major > Labels: beginner, newbie, starter > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-3665) Port BigQueryIOWriteTest off DoFnTester
[ https://issues.apache.org/jira/browse/BEAM-3665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16388421#comment-16388421 ] Justin Tumale commented on BEAM-3665: - Hi, please assign to me when you get a chance [~kenn]. Thanks > Port BigQueryIOWriteTest off DoFnTester > --- > > Key: BEAM-3665 > URL: https://issues.apache.org/jira/browse/BEAM-3665 > Project: Beam > Issue Type: Sub-task > Components: io-java-gcp >Reporter: Kenneth Knowles >Priority: Major > Labels: beginner, newbie, starter > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-2817) Bigquery queries should allow options to run in batch mode or not
[ https://issues.apache.org/jira/browse/BEAM-2817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16226259#comment-16226259 ] Justin Tumale commented on BEAM-2817: - Hi [~chamikara], I was wondering about what you think would be the best way to pass in this configuration. In the BigQueryIO class, there is a createSource method (line 573) which calls BigQueryQuerySource.create(...) which I can pass in the priority to. When executeQuery (line 133) is called from this BigQueryQuerySource object, the configuration will be based on the priority passed upon create. Another way I was thinking was to have the configuration set from the BigQueryOptions. Please advise when you get a chance. Thanks, -Justin cc [~laraschmidt] > Bigquery queries should allow options to run in batch mode or not > - > > Key: BEAM-2817 > URL: https://issues.apache.org/jira/browse/BEAM-2817 > Project: Beam > Issue Type: Bug > Components: sdk-java-gcp >Affects Versions: 2.0.0 >Reporter: Lara Schmidt >Assignee: Justin Tumale > Labels: newbie, starter > > When bigquery read does a query it sets the mode to batch. A batch query can > be very slow to schedule as it batches it with other queries. However it > doesn't use batch quota which is better for some cases. However, in some > cases a fast query is better (especially in timed tests). It would be a good > idea to have a configuration to the bigquery source to set this per-read. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (BEAM-2817) Bigquery queries should allow options to run in batch mode or not
[ https://issues.apache.org/jira/browse/BEAM-2817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Tumale reassigned BEAM-2817: --- Assignee: Justin Tumale > Bigquery queries should allow options to run in batch mode or not > - > > Key: BEAM-2817 > URL: https://issues.apache.org/jira/browse/BEAM-2817 > Project: Beam > Issue Type: Bug > Components: sdk-java-gcp >Affects Versions: 2.0.0 >Reporter: Lara Schmidt >Assignee: Justin Tumale > Labels: newbie, starter > > When bigquery read does a query it sets the mode to batch. A batch query can > be very slow to schedule as it batches it with other queries. However it > doesn't use batch quota which is better for some cases. However, in some > cases a fast query is better (especially in timed tests). It would be a good > idea to have a configuration to the bigquery source to set this per-read. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
