[jira] [Commented] (BEAM-3412) Update BigTable client version to 1.0
[
https://issues.apache.org/jira/browse/BEAM-3412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16338271#comment-16338271
]
ASF GitHub Bot commented on BEAM-3412:
--
lukecwik closed pull request #4462: [BEAM-3412] Upgrade Cloud Bigtable to 1.0.0
URL: https://github.com/apache/beam/pull/4462
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/build.gradle b/build.gradle
index e51ee28bd8e..a86efa9971c 100644
--- a/build.gradle
+++ b/build.gradle
@@ -27,7 +27,8 @@ apply from: project(":").file("build_rules.gradle")
// a dependency version which should match across multiple
// Maven artifacts.
def google_cloud_bigdataoss_version = "1.4.5"
-def bigtable_version = "1.0.0-pre3"
+def bigtable_version = "1.0.0"
+def bigtable_proto_version = "1.0.0-pre3"
def google_clients_version = "1.22.0"
def google_auth_version = "0.7.1"
def grpc_version = "1.2.0"
@@ -64,7 +65,7 @@ ext.library = [
bigdataoss_gcsio:
"com.google.cloud.bigdataoss:gcsio:$google_cloud_bigdataoss_version",
bigdataoss_util:
"com.google.cloud.bigdataoss:util:$google_cloud_bigdataoss_version",
bigtable_client_core:
"com.google.cloud.bigtable:bigtable-client-core:$bigtable_version",
-bigtable_protos:
"com.google.cloud.bigtable:bigtable-protos:$bigtable_version",
+bigtable_protos:
"com.google.cloud.bigtable:bigtable-protos:$bigtable_proto_version",
byte_buddy: "net.bytebuddy:byte-buddy:1.6.8",
commons_compress: "org.apache.commons:commons-compress:1.14",
commons_csv: "org.apache.commons:commons-csv:1.4",
diff --git a/pom.xml b/pom.xml
index fd65b2d9606..4a075dfbeec 100644
--- a/pom.xml
+++ b/pom.xml
@@ -110,7 +110,8 @@
2.33
1.8.2
v2-rev355-1.22.0
-1.0.0-pre3
+1.0.0
+1.0.0-pre3
v1-rev6-1.22.0
0.1.18
v2-rev8-1.22.0
@@ -990,7 +991,7 @@
com.google.cloud.bigtable
bigtable-protos
-${bigtable.version}
+${bigtable.proto.version}
diff --git
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableServiceImpl.java
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableServiceImpl.java
index af756a889e0..fe25e20b5d0 100644
---
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableServiceImpl.java
+++
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableServiceImpl.java
@@ -18,8 +18,8 @@
package org.apache.beam.sdk.io.gcp.bigtable;
import com.google.bigtable.admin.v2.GetTableRequest;
-import com.google.bigtable.v2.MutateRowRequest;
import com.google.bigtable.v2.MutateRowResponse;
+import com.google.bigtable.v2.MutateRowsRequest;
import com.google.bigtable.v2.Mutation;
import com.google.bigtable.v2.ReadRowsRequest;
import com.google.bigtable.v2.Row;
@@ -30,9 +30,9 @@
import com.google.cloud.bigtable.config.BigtableOptions;
import com.google.cloud.bigtable.grpc.BigtableSession;
import com.google.cloud.bigtable.grpc.BigtableTableName;
-import com.google.cloud.bigtable.grpc.async.AsyncExecutor;
import com.google.cloud.bigtable.grpc.async.BulkMutation;
import com.google.cloud.bigtable.grpc.scanner.ResultScanner;
+import com.google.common.annotations.VisibleForTesting;
import com.google.common.base.MoreObjects;
import com.google.common.io.Closer;
import com.google.common.util.concurrent.ListenableFuture;
@@ -94,13 +94,15 @@ public boolean tableExists(String tableId) throws
IOException {
}
}
- private class BigtableReaderImpl implements Reader {
+ @VisibleForTesting
+ static class BigtableReaderImpl implements Reader {
private BigtableSession session;
private final BigtableSource source;
private ResultScanner results;
private Row currentRow;
-public BigtableReaderImpl(BigtableSession session, BigtableSource source) {
+@VisibleForTesting
+BigtableReaderImpl(BigtableSession session, BigtableSource source) {
this.session = session;
this.source = source;
}
@@ -116,10 +118,13 @@ public boolean start() throws IOException {
}
RowSet rowSet = rowSetBuilder.build();
+ String tableNameSr =
+
session.getOptions().getInstanceName().toTableNameStr(source.getTableId().get());
+
ReadRowsRequest.Builder requestB =
ReadRowsRequest.newBuilder()
.setRows(rowSet)
-
.setTableName(options.getInstanceName().toTableNameStr(source.getTableId().get()));
+ .setTableName(tableNameSr);
if (source.getRowFilter() != null) {
requestB.setFilter(source.getRowFilter());
}
@@ -166,17 +171,14 @@ public
[jira] [Commented] (BEAM-3412) Update BigTable client version to 1.0
[ https://issues.apache.org/jira/browse/BEAM-3412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16337601#comment-16337601 ] ASF GitHub Bot commented on BEAM-3412: -- sduskis opened a new pull request #4462: [BEAM-3412] Upgrade Cloud Bigtable to 1.0.0 URL: https://github.com/apache/beam/pull/4462 The 1.0.0 Cloud Bigtable client upgrade included a non-backwards compatible change in protobuf generation. Specifically, addAll() methods no longer work correctly. `BigtableServiceImpl` used `addAllMutations()`, which caused run-time breakages. To get around that, add each Mutation independently to the protobuf object. This PR also adds unit tests for BigtableServiceImpl so that read/write interactions do not cause runtime problems. Follow this checklist to help us incorporate your contribution quickly and easily: - [x] Make sure there is a [JIRA issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the change (usually before you start working on it). Trivial changes like typos do not require a JIRA issue. Your pull request should address just this issue, without pulling in other changes. - [x] Each commit in the pull request should have a meaningful subject line and body. - [x] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue. - [x] Write a pull request description that is detailed enough to understand what the pull request does, how, and why. - [x] Run `mvn clean verify` to make sure basic checks pass. A more thorough check will be performed on your pull request automatically. - [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). --- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Update BigTable client version to 1.0 > - > > Key: BEAM-3412 > URL: https://issues.apache.org/jira/browse/BEAM-3412 > Project: Beam > Issue Type: Improvement > Components: sdk-java-gcp >Reporter: Chamikara Jayalath >Assignee: Solomon Duskis >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-3412) Update BigTable client version to 1.0
[
https://issues.apache.org/jira/browse/BEAM-3412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336828#comment-16336828
]
ASF GitHub Bot commented on BEAM-3412:
--
sduskis closed pull request #4462: [BEAM-3412] Upgrade Cloud Bigtable to 1.0.0
URL: https://github.com/apache/beam/pull/4462
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/build.gradle b/build.gradle
index e51ee28bd8e..10086b4824a 100644
--- a/build.gradle
+++ b/build.gradle
@@ -27,7 +27,7 @@ apply from: project(":").file("build_rules.gradle")
// a dependency version which should match across multiple
// Maven artifacts.
def google_cloud_bigdataoss_version = "1.4.5"
-def bigtable_version = "1.0.0-pre3"
+def bigtable_version = "1.0.0"
def google_clients_version = "1.22.0"
def google_auth_version = "0.7.1"
def grpc_version = "1.2.0"
diff --git a/pom.xml b/pom.xml
index fd65b2d9606..627551f68a8 100644
--- a/pom.xml
+++ b/pom.xml
@@ -110,7 +110,7 @@
2.33
1.8.2
v2-rev355-1.22.0
-1.0.0-pre3
+1.0.0
v1-rev6-1.22.0
0.1.18
v2-rev8-1.22.0
diff --git
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableServiceImpl.java
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableServiceImpl.java
index 06c459bb447..2206bdd86d4 100644
---
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableServiceImpl.java
+++
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableServiceImpl.java
@@ -18,8 +18,8 @@
package org.apache.beam.sdk.io.gcp.bigtable;
import com.google.bigtable.admin.v2.GetTableRequest;
-import com.google.bigtable.v2.MutateRowRequest;
import com.google.bigtable.v2.MutateRowResponse;
+import com.google.bigtable.v2.MutateRowsRequest;
import com.google.bigtable.v2.Mutation;
import com.google.bigtable.v2.ReadRowsRequest;
import com.google.bigtable.v2.Row;
@@ -30,9 +30,9 @@
import com.google.cloud.bigtable.config.BigtableOptions;
import com.google.cloud.bigtable.grpc.BigtableSession;
import com.google.cloud.bigtable.grpc.BigtableTableName;
-import com.google.cloud.bigtable.grpc.async.AsyncExecutor;
import com.google.cloud.bigtable.grpc.async.BulkMutation;
import com.google.cloud.bigtable.grpc.scanner.ResultScanner;
+import com.google.common.annotations.VisibleForTesting;
import com.google.common.base.MoreObjects;
import com.google.common.io.Closer;
import com.google.common.util.concurrent.ListenableFuture;
@@ -94,13 +94,15 @@ public boolean tableExists(String tableId) throws
IOException {
}
}
- private class BigtableReaderImpl implements Reader {
+ @VisibleForTesting
+ static class BigtableReaderImpl implements Reader {
private BigtableSession session;
private final BigtableSource source;
private ResultScanner results;
private Row currentRow;
-public BigtableReaderImpl(BigtableSession session, BigtableSource source) {
+@VisibleForTesting
+BigtableReaderImpl(BigtableSession session, BigtableSource source) {
this.session = session;
this.source = source;
}
@@ -119,7 +121,8 @@ public boolean start() throws IOException {
ReadRowsRequest.Builder requestB =
ReadRowsRequest.newBuilder()
.setRows(rowSet)
-
.setTableName(options.getInstanceName().toTableNameStr(source.getTableId()));
+ .setTableName(session.getOptions().getInstanceName()
+ .toTableNameStr(source.getTableId()));
if (source.getRowFilter() != null) {
requestB.setFilter(source.getRowFilter());
}
@@ -166,17 +169,14 @@ public Row getCurrentRow() throws NoSuchElementException {
}
}
- private static class BigtableWriterImpl implements Writer {
+ @VisibleForTesting
+ static class BigtableWriterImpl implements Writer {
private BigtableSession session;
-private AsyncExecutor executor;
private BulkMutation bulkMutation;
-private final String tableName;
-public BigtableWriterImpl(BigtableSession session, BigtableTableName
tableName) {
+BigtableWriterImpl(BigtableSession session, BigtableTableName tableName) {
this.session = session;
- executor = session.createAsyncExecutor();
- bulkMutation = session.createBulkMutation(tableName, executor);
- this.tableName = tableName.toString();
+ bulkMutation = session.createBulkMutation(tableName);
}
@Override
@@ -189,7 +189,6 @@ public void flush() throws IOException {
// We fail since flush() operation was interrupted.
throw new IOException(e);
}
-
[jira] [Commented] (BEAM-3412) Update BigTable client version to 1.0
[ https://issues.apache.org/jira/browse/BEAM-3412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16334746#comment-16334746 ] Chamikara Jayalath commented on BEAM-3412: -- Thanks. Will take a look. > Update BigTable client version to 1.0 > - > > Key: BEAM-3412 > URL: https://issues.apache.org/jira/browse/BEAM-3412 > Project: Beam > Issue Type: Improvement > Components: sdk-java-gcp >Reporter: Chamikara Jayalath >Assignee: Solomon Duskis >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-3412) Update BigTable client version to 1.0
[ https://issues.apache.org/jira/browse/BEAM-3412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16334635#comment-16334635 ] Solomon Duskis commented on BEAM-3412: -- I submitted [https://github.com/apache/beam/pull/4462]. Basically, I used a for loop + _addMutations()_ instead of _addAllMutations()_. I also added mock tests for _BigtableServiceImpl_ so that future upgrades won't cause problems like this one. > Update BigTable client version to 1.0 > - > > Key: BEAM-3412 > URL: https://issues.apache.org/jira/browse/BEAM-3412 > Project: Beam > Issue Type: Improvement > Components: sdk-java-gcp >Reporter: Chamikara Jayalath >Assignee: Solomon Duskis >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-3412) Update BigTable client version to 1.0
[ https://issues.apache.org/jira/browse/BEAM-3412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16332782#comment-16332782 ] ASF GitHub Bot commented on BEAM-3412: -- chamikaramj closed pull request #4347: [BEAM-3412] Updates BigTable client version to 1.0 URL: https://github.com/apache/beam/pull/4347 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a fork), the diff is supplied below (as it won't show otherwise due to GitHub magic): diff --git a/pom.xml b/pom.xml index 987a995fd71..d9378442964 100644 --- a/pom.xml +++ b/pom.xml @@ -110,7 +110,7 @@ 2.33 1.8.2 v2-rev355-1.22.0 -1.0.0-pre3 +1.0.0 v1-rev6-1.22.0 0.1.18 v2-rev8-1.22.0 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Update BigTable client version to 1.0 > - > > Key: BEAM-3412 > URL: https://issues.apache.org/jira/browse/BEAM-3412 > Project: Beam > Issue Type: Improvement > Components: sdk-java-gcp >Reporter: Chamikara Jayalath >Assignee: Solomon Duskis >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-3412) Update BigTable client version to 1.0
[ https://issues.apache.org/jira/browse/BEAM-3412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16331343#comment-16331343 ] Chamikara Jayalath commented on BEAM-3412: -- Thanks. It's great if we have a simple workaround for this and I agree with your long term fixes. > Update BigTable client version to 1.0 > - > > Key: BEAM-3412 > URL: https://issues.apache.org/jira/browse/BEAM-3412 > Project: Beam > Issue Type: Improvement > Components: sdk-java-gcp >Reporter: Chamikara Jayalath >Assignee: Solomon Duskis >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-3412) Update BigTable client version to 1.0
[ https://issues.apache.org/jira/browse/BEAM-3412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16314312#comment-16314312 ] Chamikara Jayalath commented on BEAM-3412: -- Solomon, is using already shaded dependency bigtable-hbase-1.x-shaded [1] an option ? [1] https://mvnrepository.com/artifact/com.google.cloud.bigtable/bigtable-hbase-1.x-shaded/1.0.0 > Update BigTable client version to 1.0 > - > > Key: BEAM-3412 > URL: https://issues.apache.org/jira/browse/BEAM-3412 > Project: Beam > Issue Type: Improvement > Components: sdk-java-gcp >Reporter: Chamikara Jayalath >Assignee: Chamikara Jayalath > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (BEAM-3412) Update BigTable client version to 1.0
[ https://issues.apache.org/jira/browse/BEAM-3412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16313766#comment-16313766 ] Chamikara Jayalath commented on BEAM-3412: -- Bumping up Bigtable client version requires bumping up GRPC dependency which require bumping up netty/jetty dependencies of Beam. Historically bumping up these dependencies has resulted in issues. So we'll have to shard away these dependencies. cc: [~lcwik] [~sduskis] > Update BigTable client version to 1.0 > - > > Key: BEAM-3412 > URL: https://issues.apache.org/jira/browse/BEAM-3412 > Project: Beam > Issue Type: Improvement > Components: sdk-java-gcp >Reporter: Chamikara Jayalath >Assignee: Chamikara Jayalath > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (BEAM-3412) Update BigTable client version to 1.0
[ https://issues.apache.org/jira/browse/BEAM-3412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16312263#comment-16312263 ] ASF GitHub Bot commented on BEAM-3412: -- chamikaramj opened a new pull request #4347: [BEAM-3412] Updates BigTable client version to 1.0 URL: https://github.com/apache/beam/pull/4347 Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] Make sure there is a [JIRA issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the change (usually before you start working on it). Trivial changes like typos do not require a JIRA issue. Your pull request should address just this issue, without pulling in other changes. - [ ] Each commit in the pull request should have a meaningful subject line and body. - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue. - [ ] Write a pull request description that is detailed enough to understand what the pull request does, how, and why. - [ ] Run `mvn clean verify` to make sure basic checks pass. A more thorough check will be performed on your pull request automatically. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). --- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Update BigTable client version to 1.0 > - > > Key: BEAM-3412 > URL: https://issues.apache.org/jira/browse/BEAM-3412 > Project: Beam > Issue Type: Improvement > Components: sdk-java-gcp >Reporter: Chamikara Jayalath >Assignee: Chamikara Jayalath > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
