[kudu-CR] webserver: update to latest squeasel build
Will Berkeley has posted comments on this change. ( http://gerrit.cloudera.org:8080/13728 ) Change subject: webserver: update to latest squeasel build .. Patch Set 1: Code-Review+2 (1 comment) http://gerrit.cloudera.org:8080/#/c/13728/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/13728/1//COMMIT_MSG@17 PS1, Line 17: d237491 Don't log EAGAIN when reading :) -- To view, visit http://gerrit.cloudera.org:8080/13728 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iab0ba2e4ae5fbd0e0c1ed688b9706c4e4423b179 Gerrit-Change-Number: 13728 Gerrit-PatchSet: 1 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Will Berkeley Gerrit-Comment-Date: Tue, 25 Jun 2019 22:15:32 + Gerrit-HasComments: Yes
[kudu-CR] [backup] Deflake TestKuduBackup.testRandomBackupAndRestore
Will Berkeley has submitted this change and it was merged. (
http://gerrit.cloudera.org:8080/13693 )
Change subject: [backup] Deflake TestKuduBackup.testRandomBackupAndRestore
..
[backup] Deflake TestKuduBackup.testRandomBackupAndRestore
TestKuduBackup.testRandomBackupAndRestore creates a random table with
random partitions. In a run with TSAN binaries, I saw the following
partition schema:
partition_schema {
hash_bucket_schemas {
columns {
name: "int160"
}
num_buckets: 8
seed: 395460932
}
hash_bucket_schemas {
columns {
name: "int321"
}
num_buckets: 6
seed: 2072428334
}
hash_bucket_schemas {
columns {
name: "int82"
}
num_buckets: 9
seed: 138096156
}
}
So there were 8 * 6 * 9 = 432 tablets being created on a 3-tserver
minicluster running TSAN. As a result, the test timed out waiting for
the table to finish creating.
To prevent creating tables with too many tablets, this patch reduces the
maximum number of hash buckets to 3 and it reduces the maximum nesting of
hash partitioning to 2. Combined with the existing maximum of 8 range
partitions, this means no table will consist of more than 9 * 8 = 72
tablets. Hopefully this is few enough that create table will always
finish in a timely manner.
Change-Id: Id937f4513bb0ebaf6dec345a141ed9aef2c44388
Reviewed-on: http://gerrit.cloudera.org:8080/13693
Reviewed-by: Grant Henke
Tested-by: Kudu Jenkins
---
M java/kudu-client/src/main/java/org/apache/kudu/util/SchemaGenerator.java
1 file changed, 2 insertions(+), 2 deletions(-)
Approvals:
Grant Henke: Looks good to me, approved
Kudu Jenkins: Verified
--
To view, visit http://gerrit.cloudera.org:8080/13693
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Id937f4513bb0ebaf6dec345a141ed9aef2c44388
Gerrit-Change-Number: 13693
Gerrit-PatchSet: 2
Gerrit-Owner: Will Berkeley
Gerrit-Reviewer: Grant Henke
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Will Berkeley
[kudu-CR] [backup] Deflake TestKuduBackup.testRandomBackupAndRestore
Will Berkeley has uploaded this change for review. (
http://gerrit.cloudera.org:8080/13693
Change subject: [backup] Deflake TestKuduBackup.testRandomBackupAndRestore
..
[backup] Deflake TestKuduBackup.testRandomBackupAndRestore
TestKuduBackup.testRandomBackupAndRestore creates a random table with
random partitions. In a run with TSAN binaries, I saw the following
partition schema:
partition_schema {
hash_bucket_schemas {
columns {
name: "int160"
}
num_buckets: 8
seed: 395460932
}
hash_bucket_schemas {
columns {
name: "int321"
}
num_buckets: 6
seed: 2072428334
}
hash_bucket_schemas {
columns {
name: "int82"
}
num_buckets: 9
seed: 138096156
}
}
So there were 8 * 6 * 9 = 432 tablets being created on a 3-tserver
minicluster running TSAN. As a result, the test timed out waiting for
the table to finish creating.
To prevent creating tables with too many tablets, this patch reduces the
maximum number of hash buckets to 3 and it reduces the maximum nesting of
hash partitioning to 2. Combined with the existing maximum of 8 range
partitions, this means no table will consist of more than 9 * 8 = 72
tablets. Hopefully this is few enough that create table will always
finish in a timely manner.
Change-Id: Id937f4513bb0ebaf6dec345a141ed9aef2c44388
---
M java/kudu-client/src/main/java/org/apache/kudu/util/SchemaGenerator.java
1 file changed, 2 insertions(+), 2 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/93/13693/1
--
To view, visit http://gerrit.cloudera.org:8080/13693
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Id937f4513bb0ebaf6dec345a141ed9aef2c44388
Gerrit-Change-Number: 13693
Gerrit-PatchSet: 1
Gerrit-Owner: Will Berkeley
[kudu-CR](branch-1.10.x) Optimize and sort imports
Will Berkeley has posted comments on this change. ( http://gerrit.cloudera.org:8080/13654 ) Change subject: Optimize and sort imports .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13654 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: branch-1.10.x Gerrit-MessageType: comment Gerrit-Change-Id: Ida7cbcee5b40888ee2f78ded06206a2c51c5eed7 Gerrit-Change-Number: 13654 Gerrit-PatchSet: 1 Gerrit-Owner: Grant Henke Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Will Berkeley Gerrit-Comment-Date: Fri, 14 Jun 2019 20:03:48 + Gerrit-HasComments: No
[kudu-CR] Optimize and sort imports
Will Berkeley has posted comments on this change. ( http://gerrit.cloudera.org:8080/13642 ) Change subject: Optimize and sort imports .. Patch Set 3: Verified+1 Unrelated failures in C++ tests. Something funny is going on with Jenkins. -- To view, visit http://gerrit.cloudera.org:8080/13642 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ida7cbcee5b40888ee2f78ded06206a2c51c5eed7 Gerrit-Change-Number: 13642 Gerrit-PatchSet: 3 Gerrit-Owner: Will Berkeley Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Will Berkeley Gerrit-Comment-Date: Fri, 14 Jun 2019 18:47:59 + Gerrit-HasComments: No
[kudu-CR] Optimize and sort imports
Will Berkeley has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/13642 ) Change subject: Optimize and sort imports .. Optimize and sort imports I ran IntelliJ's "Optimize imports" function on the kudu-client module after noticing how many violations of our import sorting rules were reported by checkstyle. Change-Id: Ida7cbcee5b40888ee2f78ded06206a2c51c5eed7 Reviewed-on: http://gerrit.cloudera.org:8080/13642 Reviewed-by: Grant Henke Tested-by: Will Berkeley --- M java/kudu-client/src/main/java/org/apache/kudu/Schema.java M java/kudu-client/src/main/java/org/apache/kudu/client/AlterTableOptions.java M java/kudu-client/src/main/java/org/apache/kudu/client/AsyncKuduClient.java M java/kudu-client/src/main/java/org/apache/kudu/client/AsyncKuduScanner.java M java/kudu-client/src/main/java/org/apache/kudu/client/AsyncKuduSession.java M java/kudu-client/src/main/java/org/apache/kudu/client/AuthzTokenCache.java M java/kudu-client/src/main/java/org/apache/kudu/client/Batch.java M java/kudu-client/src/main/java/org/apache/kudu/client/ConnectToCluster.java M java/kudu-client/src/main/java/org/apache/kudu/client/ConnectionCache.java M java/kudu-client/src/main/java/org/apache/kudu/client/GetTableSchemaRequest.java M java/kudu-client/src/main/java/org/apache/kudu/client/GetTableSchemaResponse.java M java/kudu-client/src/main/java/org/apache/kudu/client/HiveMetastoreConfig.java M java/kudu-client/src/main/java/org/apache/kudu/client/HostAndPort.java M java/kudu-client/src/main/java/org/apache/kudu/client/KuduClient.java M java/kudu-client/src/main/java/org/apache/kudu/client/KuduPartitioner.java M java/kudu-client/src/main/java/org/apache/kudu/client/KuduPredicate.java M java/kudu-client/src/main/java/org/apache/kudu/client/KuduRpc.java M java/kudu-client/src/main/java/org/apache/kudu/client/KuduScannerIterator.java M java/kudu-client/src/main/java/org/apache/kudu/client/ListTablesRequest.java M java/kudu-client/src/main/java/org/apache/kudu/client/Operation.java M java/kudu-client/src/main/java/org/apache/kudu/client/PartialRow.java M java/kudu-client/src/main/java/org/apache/kudu/client/RowResult.java M java/kudu-client/src/main/java/org/apache/kudu/client/RpcProxy.java M java/kudu-client/src/main/java/org/apache/kudu/client/SecurityContext.java M java/kudu-client/src/main/java/org/apache/kudu/client/SplitKeyRangeResponse.java M java/kudu-client/src/main/java/org/apache/kudu/util/BloomFilter.java M java/kudu-client/src/main/java/org/apache/kudu/util/DataGenerator.java M java/kudu-client/src/main/java/org/apache/kudu/util/DecimalUtil.java M java/kudu-client/src/main/java/org/apache/kudu/util/NetUtil.java M java/kudu-client/src/main/java/org/apache/kudu/util/SchemaGenerator.java M java/kudu-client/src/main/java/org/apache/kudu/util/SecurityUtil.java M java/kudu-client/src/main/java/org/apache/kudu/util/TimestampUtil.java M java/kudu-client/src/test/java/org/apache/kudu/TestColumnSchema.java M java/kudu-client/src/test/java/org/apache/kudu/client/ITClient.java M java/kudu-client/src/test/java/org/apache/kudu/client/ITClientStress.java M java/kudu-client/src/test/java/org/apache/kudu/client/ITScannerMultiTablet.java M java/kudu-client/src/test/java/org/apache/kudu/client/TestAlterTable.java M java/kudu-client/src/test/java/org/apache/kudu/client/TestAsyncKuduClient.java M java/kudu-client/src/test/java/org/apache/kudu/client/TestAsyncKuduSession.java M java/kudu-client/src/test/java/org/apache/kudu/client/TestAuthTokenReacquire.java M java/kudu-client/src/test/java/org/apache/kudu/client/TestAuthnTokenReacquireOpen.java M java/kudu-client/src/test/java/org/apache/kudu/client/TestAuthzTokenCache.java M java/kudu-client/src/test/java/org/apache/kudu/client/TestBloomFilter.java M java/kudu-client/src/test/java/org/apache/kudu/client/TestBytes.java M java/kudu-client/src/test/java/org/apache/kudu/client/TestClientFailoverSupport.java M java/kudu-client/src/test/java/org/apache/kudu/client/TestColumnRangePredicate.java M java/kudu-client/src/test/java/org/apache/kudu/client/TestConnectionCache.java M java/kudu-client/src/test/java/org/apache/kudu/client/TestFlexiblePartitioning.java M java/kudu-client/src/test/java/org/apache/kudu/client/TestHandleTooBusy.java M java/kudu-client/src/test/java/org/apache/kudu/client/TestHybridTime.java M java/kudu-client/src/test/java/org/apache/kudu/client/TestKeyEncoding.java M java/kudu-client/src/test/java/org/apache/kudu/client/TestKuduClient.java M java/kudu-client/src/test/java/org/apache/kudu/client/TestKuduPartitioner.java M java/kudu-client/src/test/java/org/apache/kudu/client/TestKuduPredicate.java M java/kudu-client/src/test/java/org/apache/kudu/client/TestKuduScanner.java M java/kudu-client/src/test/java/org/apache/kudu/client/TestKuduSession.java M java/kudu-client/src/test/java/org/apache/kudu/client/TestKuduTable.java M java/kudu-client/src/test/java/org/apache/kudu/cli
[kudu-CR] Optimize and sort imports
Will Berkeley has removed a vote on this change. Change subject: Optimize and sort imports .. Removed Verified-1 by Kudu Jenkins (120) -- To view, visit http://gerrit.cloudera.org:8080/13642 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: deleteVote Gerrit-Change-Id: Ida7cbcee5b40888ee2f78ded06206a2c51c5eed7 Gerrit-Change-Number: 13642 Gerrit-PatchSet: 3 Gerrit-Owner: Will Berkeley Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Kudu Jenkins (120)
[kudu-CR](branch-1.10.x) KUDU-2846 (part 2): optimize IS [NOT] NULL predicates
Will Berkeley has posted comments on this change. ( http://gerrit.cloudera.org:8080/13650 ) Change subject: KUDU-2846 (part 2): optimize IS [NOT] NULL predicates .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13650 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: branch-1.10.x Gerrit-MessageType: comment Gerrit-Change-Id: I14fb75c7c330e2415fd7a877186f7ed41a1accce Gerrit-Change-Number: 13650 Gerrit-PatchSet: 1 Gerrit-Owner: Grant Henke Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Will Berkeley Gerrit-Comment-Date: Fri, 14 Jun 2019 16:46:43 + Gerrit-HasComments: No
[kudu-CR](branch-1.10.x) [java] Add unit tests demonstrating good behavior of scan tokens when range partitions are dropped
Will Berkeley has posted comments on this change. ( http://gerrit.cloudera.org:8080/13643 ) Change subject: [java] Add unit tests demonstrating good behavior of scan tokens when range partitions are dropped .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13643 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: branch-1.10.x Gerrit-MessageType: comment Gerrit-Change-Id: Ib25ea2b62321934313c9c8852f0bc00b28bf7f2b Gerrit-Change-Number: 13643 Gerrit-PatchSet: 2 Gerrit-Owner: Grant Henke Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Will Berkeley Gerrit-Comment-Date: Fri, 14 Jun 2019 16:43:49 + Gerrit-HasComments: No
[kudu-CR](branch-1.10.x) [java] Favor column ids over column names in scan tokens
Will Berkeley has posted comments on this change. ( http://gerrit.cloudera.org:8080/13646 ) Change subject: [java] Favor column ids over column names in scan tokens .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13646 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: branch-1.10.x Gerrit-MessageType: comment Gerrit-Change-Id: Ib3f05a4175c7e7bfaec2cbd3586723e6de3823f0 Gerrit-Change-Number: 13646 Gerrit-PatchSet: 1 Gerrit-Owner: Mike Percy Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Mike Percy Gerrit-Reviewer: Will Berkeley Gerrit-Comment-Date: Fri, 14 Jun 2019 16:43:40 + Gerrit-HasComments: No
[kudu-CR] Optimize and sort imports
Hello Kudu Jenkins, Grant Henke, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13642 to look at the new patch set (#3). Change subject: Optimize and sort imports .. Optimize and sort imports I ran IntelliJ's "Optimize imports" function on the kudu-client module after noticing how many violations of our import sorting rules were reported by checkstyle. Change-Id: Ida7cbcee5b40888ee2f78ded06206a2c51c5eed7 --- M java/kudu-client/src/main/java/org/apache/kudu/Schema.java M java/kudu-client/src/main/java/org/apache/kudu/client/AlterTableOptions.java M java/kudu-client/src/main/java/org/apache/kudu/client/AsyncKuduClient.java M java/kudu-client/src/main/java/org/apache/kudu/client/AsyncKuduScanner.java M java/kudu-client/src/main/java/org/apache/kudu/client/AsyncKuduSession.java M java/kudu-client/src/main/java/org/apache/kudu/client/AuthzTokenCache.java M java/kudu-client/src/main/java/org/apache/kudu/client/Batch.java M java/kudu-client/src/main/java/org/apache/kudu/client/ConnectToCluster.java M java/kudu-client/src/main/java/org/apache/kudu/client/ConnectionCache.java M java/kudu-client/src/main/java/org/apache/kudu/client/GetTableSchemaRequest.java M java/kudu-client/src/main/java/org/apache/kudu/client/GetTableSchemaResponse.java M java/kudu-client/src/main/java/org/apache/kudu/client/HiveMetastoreConfig.java M java/kudu-client/src/main/java/org/apache/kudu/client/HostAndPort.java M java/kudu-client/src/main/java/org/apache/kudu/client/KuduClient.java M java/kudu-client/src/main/java/org/apache/kudu/client/KuduPartitioner.java M java/kudu-client/src/main/java/org/apache/kudu/client/KuduPredicate.java M java/kudu-client/src/main/java/org/apache/kudu/client/KuduRpc.java M java/kudu-client/src/main/java/org/apache/kudu/client/KuduScannerIterator.java M java/kudu-client/src/main/java/org/apache/kudu/client/ListTablesRequest.java M java/kudu-client/src/main/java/org/apache/kudu/client/Operation.java M java/kudu-client/src/main/java/org/apache/kudu/client/PartialRow.java M java/kudu-client/src/main/java/org/apache/kudu/client/RowResult.java M java/kudu-client/src/main/java/org/apache/kudu/client/RpcProxy.java M java/kudu-client/src/main/java/org/apache/kudu/client/SecurityContext.java M java/kudu-client/src/main/java/org/apache/kudu/client/SplitKeyRangeResponse.java M java/kudu-client/src/main/java/org/apache/kudu/util/BloomFilter.java M java/kudu-client/src/main/java/org/apache/kudu/util/DataGenerator.java M java/kudu-client/src/main/java/org/apache/kudu/util/DecimalUtil.java M java/kudu-client/src/main/java/org/apache/kudu/util/NetUtil.java M java/kudu-client/src/main/java/org/apache/kudu/util/SchemaGenerator.java M java/kudu-client/src/main/java/org/apache/kudu/util/SecurityUtil.java M java/kudu-client/src/main/java/org/apache/kudu/util/TimestampUtil.java M java/kudu-client/src/test/java/org/apache/kudu/TestColumnSchema.java M java/kudu-client/src/test/java/org/apache/kudu/client/ITClient.java M java/kudu-client/src/test/java/org/apache/kudu/client/ITClientStress.java M java/kudu-client/src/test/java/org/apache/kudu/client/ITScannerMultiTablet.java M java/kudu-client/src/test/java/org/apache/kudu/client/TestAlterTable.java M java/kudu-client/src/test/java/org/apache/kudu/client/TestAsyncKuduClient.java M java/kudu-client/src/test/java/org/apache/kudu/client/TestAsyncKuduSession.java M java/kudu-client/src/test/java/org/apache/kudu/client/TestAuthTokenReacquire.java M java/kudu-client/src/test/java/org/apache/kudu/client/TestAuthnTokenReacquireOpen.java M java/kudu-client/src/test/java/org/apache/kudu/client/TestAuthzTokenCache.java M java/kudu-client/src/test/java/org/apache/kudu/client/TestBloomFilter.java M java/kudu-client/src/test/java/org/apache/kudu/client/TestBytes.java M java/kudu-client/src/test/java/org/apache/kudu/client/TestClientFailoverSupport.java M java/kudu-client/src/test/java/org/apache/kudu/client/TestColumnRangePredicate.java M java/kudu-client/src/test/java/org/apache/kudu/client/TestConnectionCache.java M java/kudu-client/src/test/java/org/apache/kudu/client/TestFlexiblePartitioning.java M java/kudu-client/src/test/java/org/apache/kudu/client/TestHandleTooBusy.java M java/kudu-client/src/test/java/org/apache/kudu/client/TestHybridTime.java M java/kudu-client/src/test/java/org/apache/kudu/client/TestKeyEncoding.java M java/kudu-client/src/test/java/org/apache/kudu/client/TestKuduClient.java M java/kudu-client/src/test/java/org/apache/kudu/client/TestKuduPartitioner.java M java/kudu-client/src/test/java/org/apache/kudu/client/TestKuduPredicate.java M java/kudu-client/src/test/java/org/apache/kudu/client/TestKuduScanner.java M java/kudu-client/src/test/java/org/apache/kudu/client/TestKuduSession.java M java/kudu-client/src/test/java/org/apache/kudu/client/TestKuduTable.java M java/kudu-client/src/test/java/org/apache/kudu/client/TestLeaderFailover.java M java/kudu-cli
[kudu-CR] Optimize and sort imports
Hello Kudu Jenkins, Grant Henke, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13642 to look at the new patch set (#2). Change subject: Optimize and sort imports .. Optimize and sort imports I ran IntelliJ's "Optimize imports" function on the kudu-client module after noticing how many violations of our import sorting rules were reported by checkstyle. Change-Id: Ida7cbcee5b40888ee2f78ded06206a2c51c5eed7 --- M java/kudu-client/src/main/java/org/apache/kudu/Schema.java M java/kudu-client/src/main/java/org/apache/kudu/client/AlterTableOptions.java M java/kudu-client/src/main/java/org/apache/kudu/client/AsyncKuduClient.java M java/kudu-client/src/main/java/org/apache/kudu/client/AsyncKuduScanner.java M java/kudu-client/src/main/java/org/apache/kudu/client/AsyncKuduSession.java M java/kudu-client/src/main/java/org/apache/kudu/client/AuthzTokenCache.java M java/kudu-client/src/main/java/org/apache/kudu/client/Batch.java M java/kudu-client/src/main/java/org/apache/kudu/client/ConnectToCluster.java M java/kudu-client/src/main/java/org/apache/kudu/client/ConnectionCache.java M java/kudu-client/src/main/java/org/apache/kudu/client/GetTableSchemaRequest.java M java/kudu-client/src/main/java/org/apache/kudu/client/GetTableSchemaResponse.java M java/kudu-client/src/main/java/org/apache/kudu/client/HiveMetastoreConfig.java M java/kudu-client/src/main/java/org/apache/kudu/client/HostAndPort.java M java/kudu-client/src/main/java/org/apache/kudu/client/KuduClient.java M java/kudu-client/src/main/java/org/apache/kudu/client/KuduPartitioner.java M java/kudu-client/src/main/java/org/apache/kudu/client/KuduPredicate.java M java/kudu-client/src/main/java/org/apache/kudu/client/KuduRpc.java M java/kudu-client/src/main/java/org/apache/kudu/client/KuduScanToken.java M java/kudu-client/src/main/java/org/apache/kudu/client/KuduScannerIterator.java M java/kudu-client/src/main/java/org/apache/kudu/client/ListTablesRequest.java M java/kudu-client/src/main/java/org/apache/kudu/client/Operation.java M java/kudu-client/src/main/java/org/apache/kudu/client/PartialRow.java M java/kudu-client/src/main/java/org/apache/kudu/client/RowResult.java M java/kudu-client/src/main/java/org/apache/kudu/client/RpcProxy.java M java/kudu-client/src/main/java/org/apache/kudu/client/SecurityContext.java M java/kudu-client/src/main/java/org/apache/kudu/client/SplitKeyRangeResponse.java M java/kudu-client/src/main/java/org/apache/kudu/util/BloomFilter.java M java/kudu-client/src/main/java/org/apache/kudu/util/DataGenerator.java M java/kudu-client/src/main/java/org/apache/kudu/util/DecimalUtil.java M java/kudu-client/src/main/java/org/apache/kudu/util/NetUtil.java M java/kudu-client/src/main/java/org/apache/kudu/util/SchemaGenerator.java M java/kudu-client/src/main/java/org/apache/kudu/util/SecurityUtil.java M java/kudu-client/src/main/java/org/apache/kudu/util/TimestampUtil.java M java/kudu-client/src/test/java/org/apache/kudu/TestColumnSchema.java M java/kudu-client/src/test/java/org/apache/kudu/client/ITClient.java M java/kudu-client/src/test/java/org/apache/kudu/client/ITClientStress.java M java/kudu-client/src/test/java/org/apache/kudu/client/ITScannerMultiTablet.java M java/kudu-client/src/test/java/org/apache/kudu/client/TestAlterTable.java M java/kudu-client/src/test/java/org/apache/kudu/client/TestAsyncKuduClient.java M java/kudu-client/src/test/java/org/apache/kudu/client/TestAsyncKuduSession.java M java/kudu-client/src/test/java/org/apache/kudu/client/TestAuthTokenReacquire.java M java/kudu-client/src/test/java/org/apache/kudu/client/TestAuthnTokenReacquireOpen.java M java/kudu-client/src/test/java/org/apache/kudu/client/TestAuthzTokenCache.java M java/kudu-client/src/test/java/org/apache/kudu/client/TestBloomFilter.java M java/kudu-client/src/test/java/org/apache/kudu/client/TestBytes.java M java/kudu-client/src/test/java/org/apache/kudu/client/TestClientFailoverSupport.java M java/kudu-client/src/test/java/org/apache/kudu/client/TestColumnRangePredicate.java M java/kudu-client/src/test/java/org/apache/kudu/client/TestConnectionCache.java M java/kudu-client/src/test/java/org/apache/kudu/client/TestFlexiblePartitioning.java M java/kudu-client/src/test/java/org/apache/kudu/client/TestHandleTooBusy.java M java/kudu-client/src/test/java/org/apache/kudu/client/TestHybridTime.java M java/kudu-client/src/test/java/org/apache/kudu/client/TestKeyEncoding.java M java/kudu-client/src/test/java/org/apache/kudu/client/TestKuduClient.java M java/kudu-client/src/test/java/org/apache/kudu/client/TestKuduPartitioner.java M java/kudu-client/src/test/java/org/apache/kudu/client/TestKuduPredicate.java M java/kudu-client/src/test/java/org/apache/kudu/client/TestKuduScanner.java M java/kudu-client/src/test/java/org/apache/kudu/client/TestKuduSession.java M java/kudu-client/src/test/java/org/apache/kudu/client/TestKuduTable.java M java/kudu-client/s
[kudu-CR] Optimize and sort imports
Will Berkeley has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13642 Change subject: Optimize and sort imports .. Optimize and sort imports I ran IntelliJ's "Optimize imports" function on the kudu-client module after noticing how many violations of our import sorting rules were reported by checkstyle. Change-Id: Ida7cbcee5b40888ee2f78ded06206a2c51c5eed7 --- M java/kudu-client/src/main/java/org/apache/kudu/Schema.java M java/kudu-client/src/main/java/org/apache/kudu/client/AlterTableOptions.java M java/kudu-client/src/main/java/org/apache/kudu/client/AsyncKuduClient.java M java/kudu-client/src/main/java/org/apache/kudu/client/AsyncKuduScanner.java M java/kudu-client/src/main/java/org/apache/kudu/client/AsyncKuduSession.java M java/kudu-client/src/main/java/org/apache/kudu/client/AuthzTokenCache.java M java/kudu-client/src/main/java/org/apache/kudu/client/Batch.java M java/kudu-client/src/main/java/org/apache/kudu/client/ConnectToCluster.java M java/kudu-client/src/main/java/org/apache/kudu/client/ConnectionCache.java M java/kudu-client/src/main/java/org/apache/kudu/client/GetTableSchemaRequest.java M java/kudu-client/src/main/java/org/apache/kudu/client/GetTableSchemaResponse.java M java/kudu-client/src/main/java/org/apache/kudu/client/HiveMetastoreConfig.java M java/kudu-client/src/main/java/org/apache/kudu/client/HostAndPort.java M java/kudu-client/src/main/java/org/apache/kudu/client/KuduClient.java M java/kudu-client/src/main/java/org/apache/kudu/client/KuduPartitioner.java M java/kudu-client/src/main/java/org/apache/kudu/client/KuduPredicate.java M java/kudu-client/src/main/java/org/apache/kudu/client/KuduRpc.java M java/kudu-client/src/main/java/org/apache/kudu/client/KuduScanToken.java M java/kudu-client/src/main/java/org/apache/kudu/client/KuduScannerIterator.java M java/kudu-client/src/main/java/org/apache/kudu/client/ListTablesRequest.java M java/kudu-client/src/main/java/org/apache/kudu/client/Operation.java M java/kudu-client/src/main/java/org/apache/kudu/client/PartialRow.java M java/kudu-client/src/main/java/org/apache/kudu/client/RowResult.java M java/kudu-client/src/main/java/org/apache/kudu/client/RpcProxy.java M java/kudu-client/src/main/java/org/apache/kudu/client/SecurityContext.java M java/kudu-client/src/main/java/org/apache/kudu/client/SplitKeyRangeResponse.java M java/kudu-client/src/main/java/org/apache/kudu/util/BloomFilter.java M java/kudu-client/src/main/java/org/apache/kudu/util/DataGenerator.java M java/kudu-client/src/main/java/org/apache/kudu/util/DecimalUtil.java M java/kudu-client/src/main/java/org/apache/kudu/util/NetUtil.java M java/kudu-client/src/main/java/org/apache/kudu/util/SchemaGenerator.java M java/kudu-client/src/main/java/org/apache/kudu/util/SecurityUtil.java M java/kudu-client/src/main/java/org/apache/kudu/util/TimestampUtil.java M java/kudu-client/src/test/java/org/apache/kudu/TestColumnSchema.java M java/kudu-client/src/test/java/org/apache/kudu/client/ITClient.java M java/kudu-client/src/test/java/org/apache/kudu/client/ITClientStress.java M java/kudu-client/src/test/java/org/apache/kudu/client/ITScannerMultiTablet.java M java/kudu-client/src/test/java/org/apache/kudu/client/TestAlterTable.java M java/kudu-client/src/test/java/org/apache/kudu/client/TestAsyncKuduClient.java M java/kudu-client/src/test/java/org/apache/kudu/client/TestAsyncKuduSession.java M java/kudu-client/src/test/java/org/apache/kudu/client/TestAuthTokenReacquire.java M java/kudu-client/src/test/java/org/apache/kudu/client/TestAuthnTokenReacquireOpen.java M java/kudu-client/src/test/java/org/apache/kudu/client/TestAuthzTokenCache.java M java/kudu-client/src/test/java/org/apache/kudu/client/TestBloomFilter.java M java/kudu-client/src/test/java/org/apache/kudu/client/TestBytes.java M java/kudu-client/src/test/java/org/apache/kudu/client/TestClientFailoverSupport.java M java/kudu-client/src/test/java/org/apache/kudu/client/TestColumnRangePredicate.java M java/kudu-client/src/test/java/org/apache/kudu/client/TestConnectionCache.java M java/kudu-client/src/test/java/org/apache/kudu/client/TestFlexiblePartitioning.java M java/kudu-client/src/test/java/org/apache/kudu/client/TestHandleTooBusy.java M java/kudu-client/src/test/java/org/apache/kudu/client/TestHybridTime.java M java/kudu-client/src/test/java/org/apache/kudu/client/TestKeyEncoding.java M java/kudu-client/src/test/java/org/apache/kudu/client/TestKuduClient.java M java/kudu-client/src/test/java/org/apache/kudu/client/TestKuduPartitioner.java M java/kudu-client/src/test/java/org/apache/kudu/client/TestKuduPredicate.java M java/kudu-client/src/test/java/org/apache/kudu/client/TestKuduScanner.java M java/kudu-client/src/test/java/org/apache/kudu/client/TestKuduSession.java M java/kudu-client/src/test/java/org/apache/kudu/client/TestKuduTable.java M java/kudu-client/src/test/java/org/apache/kudu/client/TestLeaderFailover.java M java/ku
[kudu-CR] [java] Add unit tests demonstrating good behavior of scan tokens when range partitions are dropped
Will Berkeley has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/13565 ) Change subject: [java] Add unit tests demonstrating good behavior of scan tokens when range partitions are dropped .. [java] Add unit tests demonstrating good behavior of scan tokens when range partitions are dropped It turns out that scan tokens in the Java client behave nicely with respect to dropping range partitions- one can drop a range partition anytime after creating the tokens and the tokens can still be used. There is a case where some tablets in the dropped partition will still return rows, while others don't, but that can be looked at as a manifestation of the fact that tablet alters aren't linearizable in Kudu (yet?). Change-Id: Ib25ea2b62321934313c9c8852f0bc00b28bf7f2b Reviewed-on: http://gerrit.cloudera.org:8080/13565 Reviewed-by: Adar Dembo Reviewed-by: Grant Henke Tested-by: Kudu Jenkins --- M java/kudu-client/src/test/java/org/apache/kudu/client/TestScanToken.java 1 file changed, 67 insertions(+), 1 deletion(-) Approvals: Adar Dembo: Looks good to me, approved Grant Henke: Looks good to me, approved Kudu Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/13565 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ib25ea2b62321934313c9c8852f0bc00b28bf7f2b Gerrit-Change-Number: 13565 Gerrit-PatchSet: 4 Gerrit-Owner: Will Berkeley Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Will Berkeley
[kudu-CR] [java] Favor column ids over column names in scan tokens
Hello Mike Percy, Kudu Jenkins, Adar Dembo, Grant Henke, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13562 to look at the new patch set (#5). Change subject: [java] Favor column ids over column names in scan tokens .. [java] Favor column ids over column names in scan tokens Previously, a scan token would use column name to map a column in its projection to a column in the target table's current schema. Therefore, a scan token couldn't be used if a column were renamed between when the token is cut and when it is rehydrated into a scanner. This adjusts the Java client to prefer ids to names, to fix this behavior. Since this involves including column ids when serializing columns to PBs as part of scan tokens, but the server does not permit clients to send column ids in most cases, this patch adds a new serialization option that includes column ids. Note that this patch does not make _scanners_ resistant to column name changes. If a scanner is opened against a table and a column name changes on a replica before the scanner opens a server-side scanner on it, the scan will fail if the column is in the projection. A follow-up will add similar capability to the C++ client. Change-Id: Ib3f05a4175c7e7bfaec2cbd3586723e6de3823f0 --- M java/kudu-client/src/main/java/org/apache/kudu/client/AlterTableOptions.java M java/kudu-client/src/main/java/org/apache/kudu/client/CreateTableRequest.java M java/kudu-client/src/main/java/org/apache/kudu/client/KuduScanToken.java M java/kudu-client/src/main/java/org/apache/kudu/client/Operation.java M java/kudu-client/src/main/java/org/apache/kudu/client/ProtobufHelper.java M java/kudu-client/src/test/java/org/apache/kudu/client/TestKeyEncoding.java M java/kudu-client/src/test/java/org/apache/kudu/client/TestScanToken.java 7 files changed, 230 insertions(+), 67 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/62/13562/5 -- To view, visit http://gerrit.cloudera.org:8080/13562 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ib3f05a4175c7e7bfaec2cbd3586723e6de3823f0 Gerrit-Change-Number: 13562 Gerrit-PatchSet: 5 Gerrit-Owner: Will Berkeley Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Mike Percy Gerrit-Reviewer: Will Berkeley
[kudu-CR] [java] Favor column ids over column names in scan tokens
Will Berkeley has posted comments on this change. ( http://gerrit.cloudera.org:8080/13562 ) Change subject: [java] Favor column ids over column names in scan tokens .. Patch Set 4: (1 comment) http://gerrit.cloudera.org:8080/#/c/13562/4/java/kudu-client/src/main/java/org/apache/kudu/client/ProtobufHelper.java File java/kudu-client/src/main/java/org/apache/kudu/client/ProtobufHelper.java: http://gerrit.cloudera.org:8080/#/c/13562/4/java/kudu-client/src/main/java/org/apache/kudu/client/ProtobufHelper.java@50 PS4, Line 50: SCHEMA_PB_WITH_ID > My argument is one of consistency with the existing schema PB conversion fl Done -- To view, visit http://gerrit.cloudera.org:8080/13562 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib3f05a4175c7e7bfaec2cbd3586723e6de3823f0 Gerrit-Change-Number: 13562 Gerrit-PatchSet: 4 Gerrit-Owner: Will Berkeley Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Mike Percy Gerrit-Reviewer: Will Berkeley Gerrit-Comment-Date: Thu, 13 Jun 2019 18:06:32 + Gerrit-HasComments: Yes
[kudu-CR] [java] Add unit tests demonstrating good behavior of scan tokens when range partitions are dropped
Hello Kudu Jenkins, Adar Dembo, Grant Henke, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13565 to look at the new patch set (#3). Change subject: [java] Add unit tests demonstrating good behavior of scan tokens when range partitions are dropped .. [java] Add unit tests demonstrating good behavior of scan tokens when range partitions are dropped It turns out that scan tokens in the Java client behave nicely with respect to dropping range partitions- one can drop a range partition anytime after creating the tokens and the tokens can still be used. There is a case where some tablets in the dropped partition will still return rows, while others don't, but that can be looked at as a manifestation of the fact that tablet alters aren't linearizable in Kudu (yet?). Change-Id: Ib25ea2b62321934313c9c8852f0bc00b28bf7f2b --- M java/kudu-client/src/test/java/org/apache/kudu/client/TestScanToken.java 1 file changed, 67 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/65/13565/3 -- To view, visit http://gerrit.cloudera.org:8080/13565 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ib25ea2b62321934313c9c8852f0bc00b28bf7f2b Gerrit-Change-Number: 13565 Gerrit-PatchSet: 3 Gerrit-Owner: Will Berkeley Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Will Berkeley
[kudu-CR] [java] Favor column ids over column names in scan tokens
Will Berkeley has posted comments on this change. ( http://gerrit.cloudera.org:8080/13562 ) Change subject: [java] Favor column ids over column names in scan tokens .. Patch Set 4: (3 comments) http://gerrit.cloudera.org:8080/#/c/13562/4//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/13562/4//COMMIT_MSG@19 PS4, Line 19: Note that this patch does not make _scanners_ resistant to column name : changes. If a scanner is opened against a table and a column name : changes on a replica before the scanner opens a server-side scanner on : it, the scan will fail if the column is in the projection. > Do you think we should address that too (separately)? Seems like unexpected Yep. http://gerrit.cloudera.org:8080/#/c/13562/4/java/kudu-client/src/main/java/org/apache/kudu/client/KuduScanToken.java File java/kudu-client/src/main/java/org/apache/kudu/client/KuduScanToken.java: http://gerrit.cloudera.org:8080/#/c/13562/4/java/kudu-client/src/main/java/org/apache/kudu/client/KuduScanToken.java@373 PS4, Line 373: schema.hasColumnIds() ? schema.getColumnId(columnName) : -1, > Hmm, under what circumstances would the schema not have column IDs? At these callsite? Probably none because the schema comes from the table we just opened. I still think it's nice to handle it. http://gerrit.cloudera.org:8080/#/c/13562/4/java/kudu-client/src/main/java/org/apache/kudu/client/ProtobufHelper.java File java/kudu-client/src/main/java/org/apache/kudu/client/ProtobufHelper.java: http://gerrit.cloudera.org:8080/#/c/13562/4/java/kudu-client/src/main/java/org/apache/kudu/client/ProtobufHelper.java@50 PS4, Line 50: SCHEMA_PB_WITH_ID > This is confusing; could you also make this WITHOUT and update the "default I'm not sure how I feel about that. The name makes it clear what the flag does-- it's not as if true means include and false means exclude for comments and the opposite for IDs. This minimizing the extent of the changes and so minimizes the risk from them. Accidentally including the id can cause server-side RPC failures when the server rejects client schemas with ids. If you/others still feel strongly about it I can change it. -- To view, visit http://gerrit.cloudera.org:8080/13562 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib3f05a4175c7e7bfaec2cbd3586723e6de3823f0 Gerrit-Change-Number: 13562 Gerrit-PatchSet: 4 Gerrit-Owner: Will Berkeley Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Mike Percy Gerrit-Reviewer: Will Berkeley Gerrit-Comment-Date: Thu, 13 Jun 2019 16:53:59 + Gerrit-HasComments: Yes
[kudu-CR] Modernize AsyncKuduClient
Will Berkeley has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/13589 ) Change subject: Modernize AsyncKuduClient .. Modernize AsyncKuduClient Lambdas, etc. There are no functional changes in this patch. Change-Id: I046971bf84c48d380c950c988a64c2effcab6f5e Reviewed-on: http://gerrit.cloudera.org:8080/13589 Tested-by: Kudu Jenkins Reviewed-by: Adar Dembo --- M java/kudu-client/src/main/java/org/apache/kudu/client/AsyncKuduClient.java 1 file changed, 212 insertions(+), 289 deletions(-) Approvals: Kudu Jenkins: Verified Adar Dembo: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/13589 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I046971bf84c48d380c950c988a64c2effcab6f5e Gerrit-Change-Number: 13589 Gerrit-PatchSet: 2 Gerrit-Owner: Will Berkeley Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Will Berkeley
[kudu-CR] Modernize AsyncKuduClient
Will Berkeley has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13589 Change subject: Modernize AsyncKuduClient .. Modernize AsyncKuduClient Lambdas, etc. There are no functional changes in this patch. Change-Id: I046971bf84c48d380c950c988a64c2effcab6f5e --- M java/kudu-client/src/main/java/org/apache/kudu/client/AsyncKuduClient.java 1 file changed, 212 insertions(+), 289 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/89/13589/1 -- To view, visit http://gerrit.cloudera.org:8080/13589 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I046971bf84c48d380c950c988a64c2effcab6f5e Gerrit-Change-Number: 13589 Gerrit-PatchSet: 1 Gerrit-Owner: Will Berkeley
[kudu-CR] [java] Favor column ids over column names in scan tokens
Hello Mike Percy, Kudu Jenkins, Adar Dembo, Grant Henke, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13562 to look at the new patch set (#4). Change subject: [java] Favor column ids over column names in scan tokens .. [java] Favor column ids over column names in scan tokens Previously, a scan token would use column name to map a column in its projection to a column in the target table's current schema. Therefore, a scan token couldn't be used if a column were renamed between when the token is cut and when it is rehydrated into a scanner. This adjusts the Java client to prefer ids to names, to fix this behavior. Since this involves including column ids when serializing columns to PBs as part of scan tokens, but the server does not permit clients to send column ids in most cases, this patch adds a new serialization option that includes column ids. Note that this patch does not make _scanners_ resistant to column name changes. If a scanner is opened against a table and a column name changes on a replica before the scanner opens a server-side scanner on it, the scan will fail if the column is in the projection. A follow-up will add similar capability to the C++ client. Change-Id: Ib3f05a4175c7e7bfaec2cbd3586723e6de3823f0 --- M java/kudu-client/src/main/java/org/apache/kudu/client/KuduScanToken.java M java/kudu-client/src/main/java/org/apache/kudu/client/ProtobufHelper.java M java/kudu-client/src/test/java/org/apache/kudu/client/TestScanToken.java 3 files changed, 213 insertions(+), 60 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/62/13562/4 -- To view, visit http://gerrit.cloudera.org:8080/13562 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ib3f05a4175c7e7bfaec2cbd3586723e6de3823f0 Gerrit-Change-Number: 13562 Gerrit-PatchSet: 4 Gerrit-Owner: Will Berkeley Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Mike Percy Gerrit-Reviewer: Will Berkeley
[kudu-CR] [java] Favor column ids over column names in scan tokens
Hello Mike Percy, Kudu Jenkins, Adar Dembo, Grant Henke, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13562 to look at the new patch set (#3). Change subject: [java] Favor column ids over column names in scan tokens .. [java] Favor column ids over column names in scan tokens Previously, a scan token would use column name to map a column in its projection to a column in the target table's current schema. Therefore, a scan token couldn't be used if a column were renamed between when the token is cut and when it is rehydrated into a scanner. This adjusts the Java client to prefer ids to names, to fix this behavior. A follow-up will add similar capability to the C++ client. Change-Id: Ib3f05a4175c7e7bfaec2cbd3586723e6de3823f0 --- M java/kudu-client/src/main/java/org/apache/kudu/client/KuduScanToken.java M java/kudu-client/src/main/java/org/apache/kudu/client/ProtobufHelper.java M java/kudu-client/src/test/java/org/apache/kudu/client/TestScanToken.java 3 files changed, 101 insertions(+), 59 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/62/13562/3 -- To view, visit http://gerrit.cloudera.org:8080/13562 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ib3f05a4175c7e7bfaec2cbd3586723e6de3823f0 Gerrit-Change-Number: 13562 Gerrit-PatchSet: 3 Gerrit-Owner: Will Berkeley Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Mike Percy Gerrit-Reviewer: Will Berkeley
[kudu-CR] [java] Favor column ids over column names in scan tokens
Will Berkeley has posted comments on this change. ( http://gerrit.cloudera.org:8080/13562 ) Change subject: [java] Favor column ids over column names in scan tokens .. Patch Set 3: The current revision has a known defect that will be addressed in a follow-up revision. -- To view, visit http://gerrit.cloudera.org:8080/13562 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib3f05a4175c7e7bfaec2cbd3586723e6de3823f0 Gerrit-Change-Number: 13562 Gerrit-PatchSet: 3 Gerrit-Owner: Will Berkeley Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Mike Percy Gerrit-Reviewer: Will Berkeley Gerrit-Comment-Date: Tue, 11 Jun 2019 20:09:03 + Gerrit-HasComments: No
[kudu-CR] [java] Favor column ids over column names in scan tokens
Will Berkeley has posted comments on this change. (
http://gerrit.cloudera.org:8080/13562 )
Change subject: [java] Favor column ids over column names in scan tokens
..
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/13562/1/src/kudu/client/client.proto
File src/kudu/client/client.proto:
http://gerrit.cloudera.org:8080/#/c/13562/1/src/kudu/client/client.proto@58
PS1, Line 58: repeated uint32 projected_column_ids = 21;
> So the C++ and Java Schema/ColumnSchema code agree on this:
Yeah that's probably a better way to do it. Im gonna have to plumb through all
the SCHEMA_PB_{WITH|WITHOUT}_IDS stuff everywhere.
--
To view, visit http://gerrit.cloudera.org:8080/13562
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib3f05a4175c7e7bfaec2cbd3586723e6de3823f0
Gerrit-Change-Number: 13562
Gerrit-PatchSet: 1
Gerrit-Owner: Will Berkeley
Gerrit-Reviewer: Adar Dembo
Gerrit-Reviewer: Grant Henke
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Mike Percy
Gerrit-Reviewer: Will Berkeley
Gerrit-Comment-Date: Sat, 08 Jun 2019 02:11:20 +
Gerrit-HasComments: Yes
[kudu-CR] [java] Add unit tests demonstrating good behavior of scan tokens when range partitions are dropped
Will Berkeley has posted comments on this change. ( http://gerrit.cloudera.org:8080/13565 ) Change subject: [java] Add unit tests demonstrating good behavior of scan tokens when range partitions are dropped .. Patch Set 2: (2 comments) http://gerrit.cloudera.org:8080/#/c/13565/2/java/kudu-client/src/test/java/org/apache/kudu/client/TestScanToken.java File java/kudu-client/src/test/java/org/apache/kudu/client/TestScanToken.java: http://gerrit.cloudera.org:8080/#/c/13565/2/java/kudu-client/src/test/java/org/apache/kudu/client/TestScanToken.java@340 PS2, Line 340: Dehydrate > Technically you're hydrating them. Oops. http://gerrit.cloudera.org:8080/#/c/13565/2/java/kudu-client/src/test/java/org/apache/kudu/client/TestScanToken.java@351 PS2, Line 351: // Check the scanners work. The scanners for the tablets in the range [10, 20) definitely won't : // see any rows. The scanners for the tablets in the range [20, 30) might see rows. > Have you looked into why this is? I would have expected a TABLET_NOT_FOUND That's recoverable- it'll check for another replica. Once it checks for all replicas, it refreshes the cache from the master, and then realizes there's no tablet for that part of partition key space, and apparently returns 0 results for the scan. -- To view, visit http://gerrit.cloudera.org:8080/13565 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib25ea2b62321934313c9c8852f0bc00b28bf7f2b Gerrit-Change-Number: 13565 Gerrit-PatchSet: 2 Gerrit-Owner: Will Berkeley Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Will Berkeley Gerrit-Comment-Date: Sat, 08 Jun 2019 02:06:46 + Gerrit-HasComments: Yes
[kudu-CR] [c++] Support table rename between scan token creation and rehydration
Will Berkeley has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/13521 ) Change subject: [c++] Support table rename between scan token creation and rehydration .. [c++] Support table rename between scan token creation and rehydration Previously, if a scan token was created against table 'foo', and table 'foo' was renamed to table 'bar', rehydrating the scan token into a scanner would either - fail if there is no longer a table 'foo', or - attempt to scan another table if that table had been renamed to 'foo'. This patch alters how the C++ client manages scan tokens. It prefers to use the table id to identify a table, rather than a table name. This eliminates the above two problems. This is a follow up to the Java client patch for the same (and that patch includes the necessary protobuf changes). Change-Id: Ib4d48513dff67012f26a99877b168d777d3049fd Reviewed-on: http://gerrit.cloudera.org:8080/13521 Reviewed-by: Adar Dembo Tested-by: Kudu Jenkins Reviewed-by: Mike Percy --- M src/kudu/client/client-internal.cc M src/kudu/client/client-internal.h M src/kudu/client/client.cc M src/kudu/client/scan_token-internal.cc M src/kudu/client/scan_token-test.cc 5 files changed, 132 insertions(+), 31 deletions(-) Approvals: Adar Dembo: Looks good to me, approved Kudu Jenkins: Verified Mike Percy: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/13521 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ib4d48513dff67012f26a99877b168d777d3049fd Gerrit-Change-Number: 13521 Gerrit-PatchSet: 5 Gerrit-Owner: Will Berkeley Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Mike Percy Gerrit-Reviewer: Tidy Bot (241) Gerrit-Reviewer: Will Berkeley
[kudu-CR] [java] Add unit tests demonstrating good behavior of scan tokens when range partitions are dropped
Hello Kudu Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13565 to look at the new patch set (#2). Change subject: [java] Add unit tests demonstrating good behavior of scan tokens when range partitions are dropped .. [java] Add unit tests demonstrating good behavior of scan tokens when range partitions are dropped It turns out that scan tokens in the Java client behave nicely with respect to dropping range partitions- one can drop a range partition anytime after creating the tokens and the tokens can still be used. There is a case where some tablets in the dropped partition will still return rows, while others don't, but that can be looked at as a manifestation of the fact that tablet alters aren't linearizable in Kudu (yet?). Change-Id: Ib25ea2b62321934313c9c8852f0bc00b28bf7f2b --- M java/kudu-client/src/test/java/org/apache/kudu/client/TestScanToken.java 1 file changed, 67 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/65/13565/2 -- To view, visit http://gerrit.cloudera.org:8080/13565 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ib25ea2b62321934313c9c8852f0bc00b28bf7f2b Gerrit-Change-Number: 13565 Gerrit-PatchSet: 2 Gerrit-Owner: Will Berkeley Gerrit-Reviewer: Kudu Jenkins (120)
[kudu-CR] foo
Will Berkeley has abandoned this change. ( http://gerrit.cloudera.org:8080/13564 ) Change subject: foo .. Abandoned -- To view, visit http://gerrit.cloudera.org:8080/13564 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: abandon Gerrit-Change-Id: Icea1d2487210d54485715cdb059e5cd2f41561c2 Gerrit-Change-Number: 13564 Gerrit-PatchSet: 1 Gerrit-Owner: Will Berkeley
[kudu-CR] foo
Will Berkeley has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13564 Change subject: foo .. foo Change-Id: Icea1d2487210d54485715cdb059e5cd2f41561c2 --- M java/kudu-client/src/test/java/org/apache/kudu/client/TestScanToken.java 1 file changed, 51 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/64/13564/1 -- To view, visit http://gerrit.cloudera.org:8080/13564 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Icea1d2487210d54485715cdb059e5cd2f41561c2 Gerrit-Change-Number: 13564 Gerrit-PatchSet: 1 Gerrit-Owner: Will Berkeley
[kudu-CR] [java] Add unit tests demonstrating good behavior of scan tokens when range partitions are dropped
Will Berkeley has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13565 Change subject: [java] Add unit tests demonstrating good behavior of scan tokens when range partitions are dropped .. [java] Add unit tests demonstrating good behavior of scan tokens when range partitions are dropped It turns out that scan tokens in the Java client behave nicely with respect to dropping range partitions- one can drop a range partition anytime after creating the tokens and the tokens can still be used. There is a case where some tablets in the dropped partition will still return rows, while others don't, but that can be looked at as a manifestation of the fact that tablet alters aren't linearizable in Kudu (yet?). Change-Id: Ib25ea2b62321934313c9c8852f0bc00b28bf7f2b --- M java/kudu-client/src/test/java/org/apache/kudu/client/TestScanToken.java 1 file changed, 41 insertions(+), 26 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/65/13565/1 -- To view, visit http://gerrit.cloudera.org:8080/13565 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ib25ea2b62321934313c9c8852f0bc00b28bf7f2b Gerrit-Change-Number: 13565 Gerrit-PatchSet: 1 Gerrit-Owner: Will Berkeley
[kudu-CR] [backup] Make KuduBackupCLI executable
Will Berkeley has posted comments on this change. ( http://gerrit.cloudera.org:8080/13551 ) Change subject: [backup] Make KuduBackupCLI executable .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13551 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I15e0270a94844ffb7cb804a182dcf9699dd12ec7 Gerrit-Change-Number: 13551 Gerrit-PatchSet: 2 Gerrit-Owner: Grant Henke Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Mike Percy Gerrit-Reviewer: Will Berkeley Gerrit-Comment-Date: Fri, 07 Jun 2019 22:24:05 + Gerrit-HasComments: No
[kudu-CR] [java] Favor column ids over column names in scan tokens
Hello Mike Percy, Kudu Jenkins, Adar Dembo, Grant Henke, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13562 to look at the new patch set (#2). Change subject: [java] Favor column ids over column names in scan tokens .. [java] Favor column ids over column names in scan tokens Previously, a scan token would use column name to map a column in its projection to a column in the target table's current schema. Therefore, a scan token couldn't be used if a column were renamed between when the token is cut and when it is rehydrated into a scanner. This changes scan tokens in the Java client to hold column ids, if they are available, and to use them when being rehydrated into a scanner, if they are available. The original by-name behavior is the fallback if ids are not available. A follow-up will add similar capability to the C++ client. Change-Id: Ib3f05a4175c7e7bfaec2cbd3586723e6de3823f0 --- M java/kudu-client/src/main/java/org/apache/kudu/client/KuduScanToken.java M java/kudu-client/src/test/java/org/apache/kudu/client/TestScanToken.java M src/kudu/client/client.proto 3 files changed, 86 insertions(+), 47 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/62/13562/2 -- To view, visit http://gerrit.cloudera.org:8080/13562 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ib3f05a4175c7e7bfaec2cbd3586723e6de3823f0 Gerrit-Change-Number: 13562 Gerrit-PatchSet: 2 Gerrit-Owner: Will Berkeley Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Mike Percy
[kudu-CR] [java] Favor column ids over column names in scan tokens
Will Berkeley has posted comments on this change. ( http://gerrit.cloudera.org:8080/13562 ) Change subject: [java] Favor column ids over column names in scan tokens .. Patch Set 1: (2 comments) http://gerrit.cloudera.org:8080/#/c/13562/1/java/kudu-client/src/main/java/org/apache/kudu/client/KuduScanToken.java File java/kudu-client/src/main/java/org/apache/kudu/client/KuduScanToken.java: http://gerrit.cloudera.org:8080/#/c/13562/1/java/kudu-client/src/main/java/org/apache/kudu/client/KuduScanToken.java@162 PS1, Line 162: > Extra space here. Done http://gerrit.cloudera.org:8080/#/c/13562/1/src/kudu/client/client.proto File src/kudu/client/client.proto: http://gerrit.cloudera.org:8080/#/c/13562/1/src/kudu/client/client.proto@58 PS1, Line 58: repeated uint32 projected_column_ids = 21; > ColumnSchemaPB already has an id field; why can't we just fill it? Because a ColumnSchema doesn't have an id field, and the only purpose of adding an optional one would be its use in scan tokens (right now). ColumnSchemas don't always have ids associated with them- like when the Schema is built client side. Also, in the Schema class itself, the ids are a separate (possibly null) collection, so this mirrors that setup. -- To view, visit http://gerrit.cloudera.org:8080/13562 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib3f05a4175c7e7bfaec2cbd3586723e6de3823f0 Gerrit-Change-Number: 13562 Gerrit-PatchSet: 1 Gerrit-Owner: Will Berkeley Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Mike Percy Gerrit-Reviewer: Will Berkeley Gerrit-Comment-Date: Fri, 07 Jun 2019 22:14:11 + Gerrit-HasComments: Yes
[kudu-CR] [c++] Support table rename between scan token creation and rehydration
Will Berkeley has posted comments on this change. ( http://gerrit.cloudera.org:8080/13521 ) Change subject: [c++] Support table rename between scan token creation and rehydration .. Patch Set 4: (7 comments) http://gerrit.cloudera.org:8080/#/c/13521/3/src/kudu/client/client-internal.h File src/kudu/client/client-internal.h: http://gerrit.cloudera.org:8080/#/c/13521/3/src/kudu/client/client-internal.h@148 PS3, Line 148: // 'table_id_or_name' should contain a table id or name as 'identifier_type' : // is ID or NAME, respectively. : Status GetTableSchema(KuduClien > Update this. Done http://gerrit.cloudera.org:8080/#/c/13521/3/src/kudu/client/client-internal.h@156 PS3, Line 156: std::string* table_name, : int* num_replicas); > You could also make the contract such that 'table' is both IN and OUT, and Negatory. http://gerrit.cloudera.org:8080/#/c/13521/3/src/kudu/client/client-internal.cc File src/kudu/client/client-internal.cc: http://gerrit.cloudera.org:8080/#/c/13521/3/src/kudu/client/client-internal.cc@457 PS3, Line 457:const TableIdentifierPB& table_identifier, > warning: the parameter 'table_identifier' is copied for each invocation but Done http://gerrit.cloudera.org:8080/#/c/13521/3/src/kudu/client/client-internal.cc@457 PS3, Line 457:const TableIdentifierPB& table_identifier, > Agreed with clang-tidy here (and below). Done http://gerrit.cloudera.org:8080/#/c/13521/3/src/kudu/client/client-internal.cc@497 PS3, Line 497: const TableIdentifierPB& table, > warning: the parameter 'table' is copied for each invocation but only used Done http://gerrit.cloudera.org:8080/#/c/13521/3/src/kudu/client/scan_token-internal.cc File src/kudu/client/scan_token-internal.cc: http://gerrit.cloudera.org:8080/#/c/13521/3/src/kudu/client/scan_token-internal.cc@48 PS3, Line 48: #include "kudu/gutil/ref_counted.h" > And this too. Done http://gerrit.cloudera.org:8080/#/c/13521/3/src/kudu/client/scan_token-internal.cc@48 PS3, Line 48: #include "kudu/gutil/ref_counted.h" > warning: #includes are not sorted properly [llvm-include-order] Done -- To view, visit http://gerrit.cloudera.org:8080/13521 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib4d48513dff67012f26a99877b168d777d3049fd Gerrit-Change-Number: 13521 Gerrit-PatchSet: 4 Gerrit-Owner: Will Berkeley Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Mike Percy Gerrit-Reviewer: Tidy Bot (241) Gerrit-Reviewer: Will Berkeley Gerrit-Comment-Date: Fri, 07 Jun 2019 21:11:23 + Gerrit-HasComments: Yes
[kudu-CR] [java] Support table rename between scan token creation and rehydration
Will Berkeley has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/13518 ) Change subject: [java] Support table rename between scan token creation and rehydration .. [java] Support table rename between scan token creation and rehydration Previously, if a scan token was created against table 'foo', and table 'foo' was renamed to table 'bar', rehydrating the scan token into a scanner would either - fail if there is no longer a table 'foo', or - attempt to scan another table if that table had been renamed to 'foo'. This patch alters how the Java client manages scan tokens. It prefers to use the table id to identify a table, rather than a table name. This eliminates the above two problems. The table id has been added to the scan token PB to enable this. A follow-up will add support for the same in the C++ client. Change-Id: Iba5b0ac89b9cf98f4cf37c35d740eef744a85005 Reviewed-on: http://gerrit.cloudera.org:8080/13518 Reviewed-by: Mike Percy Tested-by: Kudu Jenkins Reviewed-by: Adar Dembo --- M java/kudu-client/src/main/java/org/apache/kudu/client/AsyncKuduClient.java M java/kudu-client/src/main/java/org/apache/kudu/client/GetTableSchemaRequest.java M java/kudu-client/src/main/java/org/apache/kudu/client/GetTableSchemaResponse.java M java/kudu-client/src/main/java/org/apache/kudu/client/KuduClient.java M java/kudu-client/src/main/java/org/apache/kudu/client/KuduScanToken.java M java/kudu-client/src/test/java/org/apache/kudu/client/TestScanToken.java M src/kudu/client/client.proto 7 files changed, 90 insertions(+), 13 deletions(-) Approvals: Mike Percy: Looks good to me, approved Kudu Jenkins: Verified Adar Dembo: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/13518 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Iba5b0ac89b9cf98f4cf37c35d740eef744a85005 Gerrit-Change-Number: 13518 Gerrit-PatchSet: 3 Gerrit-Owner: Will Berkeley Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Mike Percy Gerrit-Reviewer: Will Berkeley
[kudu-CR] [c++] Support table rename between scan token creation and rehydration
Hello Tidy Bot, Mike Percy, Kudu Jenkins, Adar Dembo, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13521 to look at the new patch set (#4). Change subject: [c++] Support table rename between scan token creation and rehydration .. [c++] Support table rename between scan token creation and rehydration Previously, if a scan token was created against table 'foo', and table 'foo' was renamed to table 'bar', rehydrating the scan token into a scanner would either - fail if there is no longer a table 'foo', or - attempt to scan another table if that table had been renamed to 'foo'. This patch alters how the C++ client manages scan tokens. It prefers to use the table id to identify a table, rather than a table name. This eliminates the above two problems. This is a follow up to the Java client patch for the same (and that patch includes the necessary protobuf changes). Change-Id: Ib4d48513dff67012f26a99877b168d777d3049fd --- M src/kudu/client/client-internal.cc M src/kudu/client/client-internal.h M src/kudu/client/client.cc M src/kudu/client/scan_token-internal.cc M src/kudu/client/scan_token-test.cc 5 files changed, 132 insertions(+), 31 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/21/13521/4 -- To view, visit http://gerrit.cloudera.org:8080/13521 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ib4d48513dff67012f26a99877b168d777d3049fd Gerrit-Change-Number: 13521 Gerrit-PatchSet: 4 Gerrit-Owner: Will Berkeley Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Mike Percy Gerrit-Reviewer: Tidy Bot (241) Gerrit-Reviewer: Will Berkeley
[kudu-CR] [java] Favor column ids over column names in scan tokens
Will Berkeley has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13562 Change subject: [java] Favor column ids over column names in scan tokens .. [java] Favor column ids over column names in scan tokens Previously, a scan token would use column name to map a column in its projection to a column in the target table's current schema. Therefore, a scan token couldn't be used if a column were renamed between when the token is cut and when it is rehydrated into a scanner. This changes scan tokens in the Java client to hold column ids, if they are available, and to use them when being rehydrated into a scanner, if they are available. The original by-name behavior is the fallback if ids are not available. A follow-up will add similar capability to the C++ client. Change-Id: Ib3f05a4175c7e7bfaec2cbd3586723e6de3823f0 --- M java/kudu-client/src/main/java/org/apache/kudu/client/KuduScanToken.java M java/kudu-client/src/test/java/org/apache/kudu/client/TestScanToken.java M src/kudu/client/client.proto 3 files changed, 86 insertions(+), 47 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/62/13562/1 -- To view, visit http://gerrit.cloudera.org:8080/13562 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ib3f05a4175c7e7bfaec2cbd3586723e6de3823f0 Gerrit-Change-Number: 13562 Gerrit-PatchSet: 1 Gerrit-Owner: Will Berkeley
[kudu-CR] [java] Support table rename between scan token creation and rehydration
Will Berkeley has posted comments on this change. ( http://gerrit.cloudera.org:8080/13518 ) Change subject: [java] Support table rename between scan token creation and rehydration .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/13518/1/java/kudu-client/src/main/java/org/apache/kudu/client/GetTableSchemaResponse.java File java/kudu-client/src/main/java/org/apache/kudu/client/GetTableSchemaResponse.java: http://gerrit.cloudera.org:8080/#/c/13518/1/java/kudu-client/src/main/java/org/apache/kudu/client/GetTableSchemaResponse.java@35 PS1, Line 35: /** > Update this. Done -- To view, visit http://gerrit.cloudera.org:8080/13518 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iba5b0ac89b9cf98f4cf37c35d740eef744a85005 Gerrit-Change-Number: 13518 Gerrit-PatchSet: 2 Gerrit-Owner: Will Berkeley Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Mike Percy Gerrit-Reviewer: Will Berkeley Gerrit-Comment-Date: Fri, 07 Jun 2019 17:56:32 + Gerrit-HasComments: Yes
[kudu-CR] [c++] Support table rename between scan token creation and rehydration
Will Berkeley has posted comments on this change. (
http://gerrit.cloudera.org:8080/13521 )
Change subject: [c++] Support table rename between scan token creation and
rehydration
..
Patch Set 2:
(5 comments)
http://gerrit.cloudera.org:8080/#/c/13521/2/src/kudu/client/client-internal.h
File src/kudu/client/client-internal.h:
http://gerrit.cloudera.org:8080/#/c/13521/2/src/kudu/client/client-internal.h@144
PS2, Line 144: enum class TableIdentifierType {
> Maybe just pass the TableIdentifierPB into these functions directly? They'r
Done
http://gerrit.cloudera.org:8080/#/c/13521/2/src/kudu/client/client-internal.h@151
PS2, Line 151: Status OpenTable(KuduClient* client,
> warning: function 'kudu::client::KuduClient::Data::OpenTable' has a definit
Done
http://gerrit.cloudera.org:8080/#/c/13521/2/src/kudu/client/client-internal.h@152
PS2, Line 152: identifer
> ah, it's complaining about a typo here: should be indentifier_type
Done
http://gerrit.cloudera.org:8080/#/c/13521/2/src/kudu/client/client-internal.h@169
PS2, Line 169: Status GetTableSchema(KuduClient* client,
: const std::string& table_name,
: const MonoTime& deadline,
: KuduSchema* schema,
: PartitionSchema* partition_schema,
: std::string* table_id,
: int* num_replicas);
> Can we eliminate this variant? Seems like it's only used in one or two plac
Done
http://gerrit.cloudera.org:8080/#/c/13521/2/src/kudu/client/client-internal.cc
File src/kudu/client/client-internal.cc:
http://gerrit.cloudera.org:8080/#/c/13521/2/src/kudu/client/client-internal.cc@563
PS2, Line 563: nullptr
> nit: /*table_name=*/nullptr
--
To view, visit http://gerrit.cloudera.org:8080/13521
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib4d48513dff67012f26a99877b168d777d3049fd
Gerrit-Change-Number: 13521
Gerrit-PatchSet: 2
Gerrit-Owner: Will Berkeley
Gerrit-Reviewer: Adar Dembo
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Mike Percy
Gerrit-Reviewer: Tidy Bot (241)
Gerrit-Reviewer: Will Berkeley
Gerrit-Comment-Date: Fri, 07 Jun 2019 17:56:28 +
Gerrit-HasComments: Yes
[kudu-CR] [c++] Support table rename between scan token creation and rehydration
Hello Tidy Bot, Mike Percy, Kudu Jenkins, Adar Dembo, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13521 to look at the new patch set (#3). Change subject: [c++] Support table rename between scan token creation and rehydration .. [c++] Support table rename between scan token creation and rehydration Previously, if a scan token was created against table 'foo', and table 'foo' was renamed to table 'bar', rehydrating the scan token into a scanner would either - fail if there is no longer a table 'foo', or - attempt to scan another table if that table had been renamed to 'foo'. This patch alters how the C++ client manages scan tokens. It prefers to use the table id to identify a table, rather than a table name. This eliminates the above two problems. This is a follow up to the Java client patch for the same (and that patch includes the necessary protobuf changes). Change-Id: Ib4d48513dff67012f26a99877b168d777d3049fd --- M src/kudu/client/client-internal.cc M src/kudu/client/client-internal.h M src/kudu/client/client.cc M src/kudu/client/scan_token-internal.cc M src/kudu/client/scan_token-test.cc 5 files changed, 133 insertions(+), 31 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/21/13521/3 -- To view, visit http://gerrit.cloudera.org:8080/13521 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ib4d48513dff67012f26a99877b168d777d3049fd Gerrit-Change-Number: 13521 Gerrit-PatchSet: 3 Gerrit-Owner: Will Berkeley Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Mike Percy Gerrit-Reviewer: Tidy Bot (241) Gerrit-Reviewer: Will Berkeley
[kudu-CR] [java] Support table rename between scan token creation and rehydration
Hello Mike Percy, Kudu Jenkins, Adar Dembo, Grant Henke, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13518 to look at the new patch set (#2). Change subject: [java] Support table rename between scan token creation and rehydration .. [java] Support table rename between scan token creation and rehydration Previously, if a scan token was created against table 'foo', and table 'foo' was renamed to table 'bar', rehydrating the scan token into a scanner would either - fail if there is no longer a table 'foo', or - attempt to scan another table if that table had been renamed to 'foo'. This patch alters how the Java client manages scan tokens. It prefers to use the table id to identify a table, rather than a table name. This eliminates the above two problems. The table id has been added to the scan token PB to enable this. A follow-up will add support for the same in the C++ client. Change-Id: Iba5b0ac89b9cf98f4cf37c35d740eef744a85005 --- M java/kudu-client/src/main/java/org/apache/kudu/client/AsyncKuduClient.java M java/kudu-client/src/main/java/org/apache/kudu/client/GetTableSchemaRequest.java M java/kudu-client/src/main/java/org/apache/kudu/client/GetTableSchemaResponse.java M java/kudu-client/src/main/java/org/apache/kudu/client/KuduClient.java M java/kudu-client/src/main/java/org/apache/kudu/client/KuduScanToken.java M java/kudu-client/src/test/java/org/apache/kudu/client/TestScanToken.java M src/kudu/client/client.proto 7 files changed, 90 insertions(+), 13 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/18/13518/2 -- To view, visit http://gerrit.cloudera.org:8080/13518 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iba5b0ac89b9cf98f4cf37c35d740eef744a85005 Gerrit-Change-Number: 13518 Gerrit-PatchSet: 2 Gerrit-Owner: Will Berkeley Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Mike Percy Gerrit-Reviewer: Will Berkeley
[kudu-CR] KUDU-2809 (2/6): skip unobservable rows when iterating an MRS
Will Berkeley has posted comments on this change. ( http://gerrit.cloudera.org:8080/13534 ) Change subject: KUDU-2809 (2/6): skip unobservable rows when iterating an MRS .. Patch Set 2: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/13534 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I801991260749d1f810540cb32ec84c3ea6a02160 Gerrit-Change-Number: 13534 Gerrit-PatchSet: 2 Gerrit-Owner: Adar Dembo Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Mike Percy Gerrit-Reviewer: Will Berkeley Gerrit-Comment-Date: Thu, 06 Jun 2019 22:18:02 + Gerrit-HasComments: No
[kudu-CR] KUDU-2809 (1/6): temporarily disable TestKuduScanner.testDiffScan
Will Berkeley has posted comments on this change. ( http://gerrit.cloudera.org:8080/13533 ) Change subject: KUDU-2809 (1/6): temporarily disable TestKuduScanner.testDiffScan .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13533 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I981aaa47a532dd0c1271008e27de052e3c5ad1a6 Gerrit-Change-Number: 13533 Gerrit-PatchSet: 2 Gerrit-Owner: Adar Dembo Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Mike Percy Gerrit-Reviewer: Will Berkeley Gerrit-Comment-Date: Thu, 06 Jun 2019 21:55:10 + Gerrit-HasComments: No
[kudu-CR] ksck remote-test: make checksum tests more robust
Will Berkeley has removed a vote on this change. Change subject: ksck_remote-test: make checksum tests more robust .. Removed Verified-1 by Kudu Jenkins (120) -- To view, visit http://gerrit.cloudera.org:8080/13529 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: deleteVote Gerrit-Change-Id: I1822e5b31656acf4dfbe1991cd59fa58bb574c81 Gerrit-Change-Number: 13529 Gerrit-PatchSet: 1 Gerrit-Owner: Adar Dembo Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Will Berkeley
[kudu-CR] ksck remote-test: make checksum tests more robust
Will Berkeley has posted comments on this change. ( http://gerrit.cloudera.org:8080/13529 ) Change subject: ksck_remote-test: make checksum tests more robust .. Patch Set 1: Verified+1 Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13529 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1822e5b31656acf4dfbe1991cd59fa58bb574c81 Gerrit-Change-Number: 13529 Gerrit-PatchSet: 1 Gerrit-Owner: Adar Dembo Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Will Berkeley Gerrit-Comment-Date: Thu, 06 Jun 2019 21:54:32 + Gerrit-HasComments: No
[kudu-CR] KUDU-2830: [java] Improve RPC traces on RPC timeout
Will Berkeley has posted comments on this change. ( http://gerrit.cloudera.org:8080/13509 ) Change subject: KUDU-2830: [java] Improve RPC traces on RPC timeout .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13509 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I725e3e8f89f5a433a0ffcb4b0facf82d9ffc4840 Gerrit-Change-Number: 13509 Gerrit-PatchSet: 2 Gerrit-Owner: Grant Henke Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Mike Percy Gerrit-Reviewer: Will Berkeley Gerrit-Comment-Date: Thu, 06 Jun 2019 21:49:33 + Gerrit-HasComments: No
[kudu-CR] [c++] Support table rename between scan token creation and rehydration
Will Berkeley has posted comments on this change. ( http://gerrit.cloudera.org:8080/13521 ) Change subject: [c++] Support table rename between scan token creation and rehydration .. Patch Set 1: (2 comments) http://gerrit.cloudera.org:8080/#/c/13521/1/src/kudu/client/client-internal.cc File src/kudu/client/client-internal.cc: http://gerrit.cloudera.org:8080/#/c/13521/1/src/kudu/client/client-internal.cc@456 PS1, Line 456: Status KuduClient::Data::OpenTable(KuduClient* client, > error: out-of-line definition of 'OpenTable' does not match any declaration Done http://gerrit.cloudera.org:8080/#/c/13521/1/src/kudu/client/client-internal.cc@476 PS1, Line 476: // TODO: in the future, probably will look up the table in some map to reuse > warning: missing username/bug in TODO [google-readability-todo] Done -- To view, visit http://gerrit.cloudera.org:8080/13521 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib4d48513dff67012f26a99877b168d777d3049fd Gerrit-Change-Number: 13521 Gerrit-PatchSet: 1 Gerrit-Owner: Will Berkeley Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Tidy Bot (241) Gerrit-Reviewer: Will Berkeley Gerrit-Comment-Date: Wed, 05 Jun 2019 20:51:33 + Gerrit-HasComments: Yes
[kudu-CR] [c++] Support table rename between scan token creation and rehydration
Hello Tidy Bot, Kudu Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13521 to look at the new patch set (#2). Change subject: [c++] Support table rename between scan token creation and rehydration .. [c++] Support table rename between scan token creation and rehydration Previously, if a scan token was created against table 'foo', and table 'foo' was renamed to table 'bar', rehydrating the scan token into a scanner would either - fail if there is no longer a table 'foo', or - attempt to scan another table if that table had been renamed to 'foo'. This patch alters how the C++ client manages scan tokens. It prefers to use the table id to identify a table, rather than a table name. This eliminates the above two problems. This is a follow up to the Java client patch for the same (and that patch includes the necessary protobuf changes). Change-Id: Ib4d48513dff67012f26a99877b168d777d3049fd --- M src/kudu/client/client-internal.cc M src/kudu/client/client-internal.h M src/kudu/client/client.cc M src/kudu/client/scan_token-internal.cc M src/kudu/client/scan_token-test.cc 5 files changed, 157 insertions(+), 29 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/21/13521/2 -- To view, visit http://gerrit.cloudera.org:8080/13521 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ib4d48513dff67012f26a99877b168d777d3049fd Gerrit-Change-Number: 13521 Gerrit-PatchSet: 2 Gerrit-Owner: Will Berkeley Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Tidy Bot (241)
[kudu-CR] [c++] Support table rename between scan token creation and rehydration
Will Berkeley has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13521 Change subject: [c++] Support table rename between scan token creation and rehydration .. [c++] Support table rename between scan token creation and rehydration Previously, if a scan token was created against table 'foo', and table 'foo' was renamed to table 'bar', rehydrating the scan token into a scanner would either - fail if there is no longer a table 'foo', or - attempt to scan another table if that table had been renamed to 'foo'. This patch alters how the C++ client manages scan tokens. It prefers to use the table id to identify a table, rather than a table name. This eliminates the above two problems. This is a follow up to the Java client patch for the same (and that patch includes the necessary protobuf changes). Change-Id: Ib4d48513dff67012f26a99877b168d777d3049fd --- M src/kudu/client/client-internal.cc M src/kudu/client/client-internal.h M src/kudu/client/client.cc M src/kudu/client/scan_token-internal.cc M src/kudu/client/scan_token-test.cc 5 files changed, 157 insertions(+), 29 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/21/13521/1 -- To view, visit http://gerrit.cloudera.org:8080/13521 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ib4d48513dff67012f26a99877b168d777d3049fd Gerrit-Change-Number: 13521 Gerrit-PatchSet: 1 Gerrit-Owner: Will Berkeley
[kudu-CR] [java] Support table rename between scan token creation and rehydration
Will Berkeley has posted comments on this change. (
http://gerrit.cloudera.org:8080/13518 )
Change subject: [java] Support table rename between scan token creation and
rehydration
..
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/13518/1/java/kudu-client/src/main/java/org/apache/kudu/client/KuduScanToken.java
File java/kudu-client/src/main/java/org/apache/kudu/client/KuduScanToken.java:
http://gerrit.cloudera.org:8080/#/c/13518/1/java/kudu-client/src/main/java/org/apache/kudu/client/KuduScanToken.java@283
PS1, Line 283: if (message.hasTableId() && other.message.hasTableId()) {
> Is comparing by table name safe across a rename? I suppose either way, a to
Safe like you could end up scanning a different table / comparing scan tokens
in the case where you meant to scan 'foo' but then ('foo' rename to 'bar') and
('baaz' rename to 'foo')? No, I wouldn't call that safe. I would call what's
being done here preserving unsafe legacy behavior :(
--
To view, visit http://gerrit.cloudera.org:8080/13518
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iba5b0ac89b9cf98f4cf37c35d740eef744a85005
Gerrit-Change-Number: 13518
Gerrit-PatchSet: 1
Gerrit-Owner: Will Berkeley
Gerrit-Reviewer: Grant Henke
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Will Berkeley
Gerrit-Comment-Date: Wed, 05 Jun 2019 20:13:43 +
Gerrit-HasComments: Yes
[kudu-CR] [java] Support table rename between scan token creation and rehydration
Will Berkeley has posted comments on this change. (
http://gerrit.cloudera.org:8080/13518 )
Change subject: [java] Support table rename between scan token creation and
rehydration
..
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/13518/1/java/kudu-client/src/main/java/org/apache/kudu/client/KuduScanToken.java
File java/kudu-client/src/main/java/org/apache/kudu/client/KuduScanToken.java:
http://gerrit.cloudera.org:8080/#/c/13518/1/java/kudu-client/src/main/java/org/apache/kudu/client/KuduScanToken.java@283
PS1, Line 283: if (message.hasTableId() && other.message.hasTableId()) {
> Should cases where message.hasTableId() != other.message.hasTableId() fail?
I am thinking no because one scan token may be old and only have table id and
the other may be new and have table id, while both are for the same table.
--
To view, visit http://gerrit.cloudera.org:8080/13518
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iba5b0ac89b9cf98f4cf37c35d740eef744a85005
Gerrit-Change-Number: 13518
Gerrit-PatchSet: 1
Gerrit-Owner: Will Berkeley
Gerrit-Reviewer: Grant Henke
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Will Berkeley
Gerrit-Comment-Date: Wed, 05 Jun 2019 19:53:04 +
Gerrit-HasComments: Yes
[kudu-CR] [java] Support table rename between scan token creation and rehydration
Will Berkeley has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13518 Change subject: [java] Support table rename between scan token creation and rehydration .. [java] Support table rename between scan token creation and rehydration Previously, if a scan token was created against table 'foo', and table 'foo' was renamed to table 'bar', rehydrating the scan token into a scanner would either - fail if there is no longer a table 'foo', or - attempt to scan another table if that table had been renamed to 'foo'. This patch alters how the Java client manages scan tokens. It prefers to use the table id to identify a table, rather than a table name. This eliminates the above two problems. The table id has been added to the scan token PB to enable this. A follow-up will add support for the same in the C++ client. Change-Id: Iba5b0ac89b9cf98f4cf37c35d740eef744a85005 --- M java/kudu-client/src/main/java/org/apache/kudu/client/AsyncKuduClient.java M java/kudu-client/src/main/java/org/apache/kudu/client/GetTableSchemaRequest.java M java/kudu-client/src/main/java/org/apache/kudu/client/GetTableSchemaResponse.java M java/kudu-client/src/main/java/org/apache/kudu/client/KuduClient.java M java/kudu-client/src/main/java/org/apache/kudu/client/KuduScanToken.java M java/kudu-client/src/test/java/org/apache/kudu/client/TestScanToken.java M src/kudu/client/client.proto 7 files changed, 89 insertions(+), 13 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/18/13518/1 -- To view, visit http://gerrit.cloudera.org:8080/13518 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Iba5b0ac89b9cf98f4cf37c35d740eef744a85005 Gerrit-Change-Number: 13518 Gerrit-PatchSet: 1 Gerrit-Owner: Will Berkeley
[kudu-CR] KUDU-2830: [java] Improve RPC traces on RPC timeout
Will Berkeley has posted comments on this change. (
http://gerrit.cloudera.org:8080/13509 )
Change subject: KUDU-2830: [java] Improve RPC traces on RPC timeout
..
Patch Set 1:
(4 comments)
http://gerrit.cloudera.org:8080/#/c/13509/1/java/kudu-client/src/main/java/org/apache/kudu/client/KuduRpc.java
File java/kudu-client/src/main/java/org/apache/kudu/client/KuduRpc.java:
http://gerrit.cloudera.org:8080/#/c/13509/1/java/kudu-client/src/main/java/org/apache/kudu/client/KuduRpc.java@403
PS1, Line 403: buf.append(", deferred=").append(deferred);
Is this intentionally omitted from 'buf' in the other branch?
http://gerrit.cloudera.org:8080/#/c/13509/1/java/kudu-client/src/main/java/org/apache/kudu/client/RpcTraceFrame.java
File java/kudu-client/src/main/java/org/apache/kudu/client/RpcTraceFrame.java:
http://gerrit.cloudera.org:8080/#/c/13509/1/java/kudu-client/src/main/java/org/apache/kudu/client/RpcTraceFrame.java@187
PS1, Line 187: UNKOWN
UNKNOWN here and below.
http://gerrit.cloudera.org:8080/#/c/13509/1/java/kudu-client/src/main/java/org/apache/kudu/client/RpcTraceFrame.java@213
PS1, Line 213: authWaitCount
I think 'authTokenRefresh' or 'authRefresh' is better. What do you think?
http://gerrit.cloudera.org:8080/#/c/13509/1/java/kudu-client/src/main/java/org/apache/kudu/client/RpcTraceFrame.java@222
PS1, Line 222: StringBuilder sb = new StringBuilder();
I wish there were a better alternative to this StringBuilder nonsense,
something like what's possible with strings::Substitute and multiline literals
in C++, or with the s"$foo" feature in Scala. Obviously no defects vs. no
obvious defects.
--
To view, visit http://gerrit.cloudera.org:8080/13509
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I725e3e8f89f5a433a0ffcb4b0facf82d9ffc4840
Gerrit-Change-Number: 13509
Gerrit-PatchSet: 1
Gerrit-Owner: Grant Henke
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Mike Percy
Gerrit-Reviewer: Will Berkeley
Gerrit-Comment-Date: Tue, 04 Jun 2019 15:15:16 +
Gerrit-HasComments: Yes
[kudu-CR] external mini cluster: bump start process timeout a bit more
Will Berkeley has posted comments on this change. ( http://gerrit.cloudera.org:8080/13484 ) Change subject: external_mini_cluster: bump start process timeout a bit more .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13484 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I95de5368298b9c7fd1ea0b7166dd6c3cd24e356b Gerrit-Change-Number: 13484 Gerrit-PatchSet: 1 Gerrit-Owner: Adar Dembo Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Will Berkeley Gerrit-Comment-Date: Mon, 03 Jun 2019 22:25:53 + Gerrit-HasComments: No
[kudu-CR] docs: update docs for heap sampling
Will Berkeley has posted comments on this change. ( http://gerrit.cloudera.org:8080/13459 ) Change subject: docs: update docs for heap sampling .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13459 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5bdf8b2175f1bc492e98dc2a815bde4ab8fbe626 Gerrit-Change-Number: 13459 Gerrit-PatchSet: 1 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Will Berkeley Gerrit-Comment-Date: Mon, 03 Jun 2019 17:25:01 + Gerrit-HasComments: No
[kudu-CR] [java] Micro-improvements to DataGenerator
Will Berkeley has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/13498 ) Change subject: [java] Micro-improvements to DataGenerator .. [java] Micro-improvements to DataGenerator Change-Id: I78eca7c6a4d8e0f52fe646a4157774b9123c8a4a Reviewed-on: http://gerrit.cloudera.org:8080/13498 Tested-by: Kudu Jenkins Reviewed-by: Grant Henke --- M java/kudu-client/src/main/java/org/apache/kudu/util/DataGenerator.java 1 file changed, 30 insertions(+), 28 deletions(-) Approvals: Kudu Jenkins: Verified Grant Henke: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/13498 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I78eca7c6a4d8e0f52fe646a4157774b9123c8a4a Gerrit-Change-Number: 13498 Gerrit-PatchSet: 3 Gerrit-Owner: Will Berkeley Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Mike Percy Gerrit-Reviewer: Will Berkeley
[kudu-CR] [backup] Deflake TestKuduBackup.testRandomBackupAndRestore
Will Berkeley has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/13499 ) Change subject: [backup] Deflake TestKuduBackup.testRandomBackupAndRestore .. [backup] Deflake TestKuduBackup.testRandomBackupAndRestore TestKuduBackup.testRandomBackupAndRestore was flaky because it the DataGenerator sometimes generated duplicate rows, which caused count mismatches between the number of rows upserted and the number of rows scanned from the table (particularly when the random schema had a single INT8 primary key). This fix works around the problem by eliminating the validation of backups against the generated rows, reducing the set of validations to just the validation of the final state of the table against the restored table. Change-Id: Ic8500f884b05d09a8679863c380dd89f1675519f Reviewed-on: http://gerrit.cloudera.org:8080/13499 Tested-by: Kudu Jenkins Reviewed-by: Grant Henke --- M java/kudu-backup/src/test/scala/org/apache/kudu/backup/TestKuduBackup.scala 1 file changed, 9 insertions(+), 5 deletions(-) Approvals: Kudu Jenkins: Verified Grant Henke: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/13499 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ic8500f884b05d09a8679863c380dd89f1675519f Gerrit-Change-Number: 13499 Gerrit-PatchSet: 4 Gerrit-Owner: Will Berkeley Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Mike Percy Gerrit-Reviewer: Will Berkeley
[kudu-CR] [java] Fix ordering bug in AsyncKuduSession
Will Berkeley has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/13496 ) Change subject: [java] Fix ordering bug in AsyncKuduSession .. [java] Fix ordering bug in AsyncKuduSession AsyncKuduSession.testBatchErrorCauseSessionStuck occasionally failed because AsyncKuduSession#hasPendingOperations returned true even though the test joined on the deferred results of its two inserts. I tracked this down to a small mis-ordering of lines where the AsyncKuduSession fires a flush notification (from a Netty worker thread) before it actually swaps the just-flushed buffer back to the inactive list, so there was a small window of time when the client thread could observe hasPendingOperations() as true even though there were no pending operations. I ran the test 100 times in dist-test with this fix and saw no failures. I didn't run the test pre-fix through dist-test but the flaky dashboard indicated it was 2-3% flaky. Change-Id: I498636aeb0de2673c15b90260e6acf56de1ead06 Reviewed-on: http://gerrit.cloudera.org:8080/13496 Tested-by: Kudu Jenkins Reviewed-by: Grant Henke --- M java/kudu-client/src/main/java/org/apache/kudu/client/AsyncKuduSession.java 1 file changed, 1 insertion(+), 1 deletion(-) Approvals: Kudu Jenkins: Verified Grant Henke: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/13496 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I498636aeb0de2673c15b90260e6acf56de1ead06 Gerrit-Change-Number: 13496 Gerrit-PatchSet: 2 Gerrit-Owner: Will Berkeley Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Will Berkeley
[kudu-CR] [java] Deflake TestKuduSession.testInsertAutoFlushBackgroundNonCoveredRange
Will Berkeley has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/13497 ) Change subject: [java] Deflake TestKuduSession.testInsertAutoFlushBackgroundNonCoveredRange .. [java] Deflake TestKuduSession.testInsertAutoFlushBackgroundNonCoveredRange The test uses an AsyncKuduSession in AUTO_FLUSH_BACKGROUND mode to send 20 inserts. 10 of them should fail because they are in a non-covered range. However, the test did not join on the deferred when it called AsyncKuduSession#flush. Therefore, there was no guarantee the operations had been completed. So, every now and then, the ops were still in flight when the errors were checked, causing the test to fail because it saw no error where it expected to see errors. Before this fix, the test failed because of this problem about 1/1000 times. I ran 1000 with the fix and saw no failures, but the rate was so low maybe I was lucky. Regardless, I'm confident the test is more correct with this fix. Change-Id: I40360e9c7979c2edf01b4a70b64f71af4d1a1e11 Reviewed-on: http://gerrit.cloudera.org:8080/13497 Tested-by: Will Berkeley Reviewed-by: Grant Henke --- M java/kudu-client/src/test/java/org/apache/kudu/client/TestKuduSession.java 1 file changed, 2 insertions(+), 2 deletions(-) Approvals: Will Berkeley: Verified Grant Henke: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/13497 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I40360e9c7979c2edf01b4a70b64f71af4d1a1e11 Gerrit-Change-Number: 13497 Gerrit-PatchSet: 2 Gerrit-Owner: Will Berkeley Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Will Berkeley
[kudu-CR] [java] Fix ordering bug in AsyncKuduSession
Will Berkeley has posted comments on this change. ( http://gerrit.cloudera.org:8080/13496 ) Change subject: [java] Fix ordering bug in AsyncKuduSession .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/13496/1/java/kudu-client/src/main/java/org/apache/kudu/client/AsyncKuduSession.java File java/kudu-client/src/main/java/org/apache/kudu/client/AsyncKuduSession.java: http://gerrit.cloudera.org:8080/#/c/13496/1/java/kudu-client/src/main/java/org/apache/kudu/client/AsyncKuduSession.java@306 PS1, Line 306:* Returns a buffer to the inactive queue after flushing. > nit: Can you update this comment since the order changed? Maybe: The buffer is already flushed at this point. This function resets state post-flush. I'm thinking about rewriting some of the internals of AsyncKuduSession to make buffer management easier to understand. -- To view, visit http://gerrit.cloudera.org:8080/13496 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I498636aeb0de2673c15b90260e6acf56de1ead06 Gerrit-Change-Number: 13496 Gerrit-PatchSet: 1 Gerrit-Owner: Will Berkeley Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Will Berkeley Gerrit-Comment-Date: Mon, 03 Jun 2019 16:47:42 + Gerrit-HasComments: Yes
[kudu-CR] [java] Micro-improvements to DataGenerator
Will Berkeley has posted comments on this change. ( http://gerrit.cloudera.org:8080/13498 ) Change subject: [java] Micro-improvements to DataGenerator .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/13498/2/java/kudu-client/src/main/java/org/apache/kudu/util/DataGenerator.java File java/kudu-client/src/main/java/org/apache/kudu/util/DataGenerator.java: http://gerrit.cloudera.org:8080/#/c/13498/2/java/kudu-client/src/main/java/org/apache/kudu/util/DataGenerator.java@85 PS2, Line 85: continue; > Really curious about the motivation for this. The following two general principles of readable code: 1. Minimize the amount of indentation, because each level of indentation is usually another level of control flow to keep in mind. 2. Early return (or break or continue) is good because it allows the reader to eliminate cases from consideration in the rest of the function or block. Also, I didn't actually mean to submit this patch. I just made this change to help me understand this code better, and then forgot to undo it when submitting the follow-up. -- To view, visit http://gerrit.cloudera.org:8080/13498 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I78eca7c6a4d8e0f52fe646a4157774b9123c8a4a Gerrit-Change-Number: 13498 Gerrit-PatchSet: 2 Gerrit-Owner: Will Berkeley Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Mike Percy Gerrit-Reviewer: Will Berkeley Gerrit-Comment-Date: Mon, 03 Jun 2019 16:44:17 + Gerrit-HasComments: Yes
[kudu-CR] [backup] Deflake TestKuduBackup.testRandomBackupAndRestore
Hello Mike Percy, Kudu Jenkins, Adar Dembo, Grant Henke, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13499 to look at the new patch set (#3). Change subject: [backup] Deflake TestKuduBackup.testRandomBackupAndRestore .. [backup] Deflake TestKuduBackup.testRandomBackupAndRestore TestKuduBackup.testRandomBackupAndRestore was flaky because it the DataGenerator sometimes generated duplicate rows, which caused count mismatches between the number of rows upserted and the number of rows scanned from the table (particularly when the random schema had a single INT8 primary key). This fix works around the problem by eliminating the validation of backups against the generated rows, reducing the set of validations to just the validation of the final state of the table against the restored table. Change-Id: Ic8500f884b05d09a8679863c380dd89f1675519f --- M java/kudu-backup/src/test/scala/org/apache/kudu/backup/TestKuduBackup.scala 1 file changed, 9 insertions(+), 5 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/99/13499/3 -- To view, visit http://gerrit.cloudera.org:8080/13499 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ic8500f884b05d09a8679863c380dd89f1675519f Gerrit-Change-Number: 13499 Gerrit-PatchSet: 3 Gerrit-Owner: Will Berkeley Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Mike Percy
[kudu-CR] [java] Micro-improvements to DataGenerator
Hello Mike Percy, Kudu Jenkins, Adar Dembo, Grant Henke, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13498 to look at the new patch set (#2). Change subject: [java] Micro-improvements to DataGenerator .. [java] Micro-improvements to DataGenerator Change-Id: I78eca7c6a4d8e0f52fe646a4157774b9123c8a4a --- M java/kudu-client/src/main/java/org/apache/kudu/util/DataGenerator.java 1 file changed, 30 insertions(+), 28 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/98/13498/2 -- To view, visit http://gerrit.cloudera.org:8080/13498 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I78eca7c6a4d8e0f52fe646a4157774b9123c8a4a Gerrit-Change-Number: 13498 Gerrit-PatchSet: 2 Gerrit-Owner: Will Berkeley Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Mike Percy
[kudu-CR] [java] Deflake TestKuduSession.testInsertAutoFlushBackgroundNonCoveredRange
Will Berkeley has posted comments on this change. ( http://gerrit.cloudera.org:8080/13497 ) Change subject: [java] Deflake TestKuduSession.testInsertAutoFlushBackgroundNonCoveredRange .. Patch Set 1: Verified+1 Looks like a (probably benign) race involving log-capturing code and the stack trace collector. Certainly unrelated to this patch. -- To view, visit http://gerrit.cloudera.org:8080/13497 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I40360e9c7979c2edf01b4a70b64f71af4d1a1e11 Gerrit-Change-Number: 13497 Gerrit-PatchSet: 1 Gerrit-Owner: Will Berkeley Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Will Berkeley Gerrit-Comment-Date: Mon, 03 Jun 2019 09:41:31 + Gerrit-HasComments: No
[kudu-CR] [java] Deflake TestKuduSession.testInsertAutoFlushBackgroundNonCoveredRange
Will Berkeley has removed a vote on this change. Change subject: [java] Deflake TestKuduSession.testInsertAutoFlushBackgroundNonCoveredRange .. Removed Verified-1 by Kudu Jenkins (120) -- To view, visit http://gerrit.cloudera.org:8080/13497 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: deleteVote Gerrit-Change-Id: I40360e9c7979c2edf01b4a70b64f71af4d1a1e11 Gerrit-Change-Number: 13497 Gerrit-PatchSet: 1 Gerrit-Owner: Will Berkeley Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Will Berkeley
[kudu-CR] [java] Micro-improvements to DataGenerator
Will Berkeley has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13498 Change subject: [java] Micro-improvements to DataGenerator .. [java] Micro-improvements to DataGenerator Change-Id: I78eca7c6a4d8e0f52fe646a4157774b9123c8a4a --- M java/kudu-client/src/main/java/org/apache/kudu/util/DataGenerator.java 1 file changed, 30 insertions(+), 28 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/98/13498/1 -- To view, visit http://gerrit.cloudera.org:8080/13498 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I78eca7c6a4d8e0f52fe646a4157774b9123c8a4a Gerrit-Change-Number: 13498 Gerrit-PatchSet: 1 Gerrit-Owner: Will Berkeley
[kudu-CR] [backup] Deflake TestKuduBackup.testRandomBackupAndRestore
Will Berkeley has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13499 Change subject: [backup] Deflake TestKuduBackup.testRandomBackupAndRestore .. [backup] Deflake TestKuduBackup.testRandomBackupAndRestore TestKuduBackup.testRandomBackupAndRestore was flaky because it the DataGenerator sometimes generated duplicate rows, which caused count mismatches between the number of rows upserted and the number of rows scanned from the table (particularly when the random schema had a single INT8 primary key). This fix works around the problem by eliminating the validation of backups against the generated rows, reducing the set of validations to just the validation of the final state of the table against the restored table. Change-Id: Ic8500f884b05d09a8679863c380dd89f1675519f --- M java/kudu-backup/src/test/scala/org/apache/kudu/backup/TestKuduBackup.scala 1 file changed, 8 insertions(+), 5 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/99/13499/1 -- To view, visit http://gerrit.cloudera.org:8080/13499 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ic8500f884b05d09a8679863c380dd89f1675519f Gerrit-Change-Number: 13499 Gerrit-PatchSet: 1 Gerrit-Owner: Will Berkeley
[kudu-CR] [java] Deflake TestKuduSession.testInsertAutoFlushBackgroundNonCoveredRange
Will Berkeley has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13497 Change subject: [java] Deflake TestKuduSession.testInsertAutoFlushBackgroundNonCoveredRange .. [java] Deflake TestKuduSession.testInsertAutoFlushBackgroundNonCoveredRange The test uses an AsyncKuduSession in AUTO_FLUSH_BACKGROUND mode to send 20 inserts. 10 of them should fail because they are in a non-covered range. However, the test did not join on the deferred when it called AsyncKuduSession#flush. Therefore, there was no guarantee the operations had been completed. So, every now and then, the ops were still in flight when the errors were checked, causing the test to fail because it saw no error where it expected to see errors. Before this fix, the test failed because of this problem about 1/1000 times. I ran 1000 with the fix and saw no failures, but the rate was so low maybe I was lucky. Regardless, I'm confident the test is more correct with this fix. Change-Id: I40360e9c7979c2edf01b4a70b64f71af4d1a1e11 --- M java/kudu-client/src/test/java/org/apache/kudu/client/TestKuduSession.java 1 file changed, 2 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/97/13497/1 -- To view, visit http://gerrit.cloudera.org:8080/13497 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I40360e9c7979c2edf01b4a70b64f71af4d1a1e11 Gerrit-Change-Number: 13497 Gerrit-PatchSet: 1 Gerrit-Owner: Will Berkeley
[kudu-CR] [java] Deflake TestKuduScanner.testDiffScan
Will Berkeley has submitted this change and it was merged. (
http://gerrit.cloudera.org:8080/13495 )
Change subject: [java] Deflake TestKuduScanner.testDiffScan
..
[java] Deflake TestKuduScanner.testDiffScan
I noticed a funny failure in TestKuduScanner.testDiffScan:
21:55:41.681 [DEBUG - New I/O worker #13] (AsyncKuduScanner.java:556) Can not
open scanner
org.apache.kudu.client.NonRecoverableException: snapshot scan start timestamp
is earlier than the ancient history mark. Consider increasing the value of the
configuration parameter --tablet_history_max_age_sec. Snapshot timestamp: P: 0
usec, L: 0 Ancient History Mark: P: 1558907741677535 usec, L: 0 Physical time
difference: -1558907741.678s
The client logs the request it sent, which indeed has a
'snap_start_timestamp' of 0. This timestamp is supposed to come from
incrementing the propagated timestamp received from the client after
some random inserts and mutations. Fortunately, this test has nice
logging: it logs the operations it did and the timestamp:
21:55:40.874 [INFO - Time-limited test] (TestKuduScanner.java:250) Before: {}
21:55:40.874 [INFO - Time-limited test] (TestKuduScanner.java:255) startHT: 0
'generateMutationOperations' could generate 0 mutations, causing this
test to fail. This requires the random number generator to generate 0 3
times in a row when sampling uniformly from {0, 1, 2, 3, 4}, so the odds
of failure due to this defect in the test were only 1/125. I fixed this
by adjusting the number of initial inserts generated so it is always at
least 1.
I ran the test 1250 times with the fix and saw zero failures. Without
the fix, I saw 9/1250 failures.
Change-Id: Iadb46e5dae71724aa8ff88d04c40ef4eaf1ddd2a
Reviewed-on: http://gerrit.cloudera.org:8080/13495
Tested-by: Will Berkeley
Reviewed-by: Alexey Serbin
---
M java/kudu-client/src/test/java/org/apache/kudu/client/TestKuduScanner.java
M java/kudu-test-utils/src/main/java/org/apache/kudu/test/RandomUtils.java
2 files changed, 32 insertions(+), 14 deletions(-)
Approvals:
Will Berkeley: Verified
Alexey Serbin: Looks good to me, approved
--
To view, visit http://gerrit.cloudera.org:8080/13495
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Iadb46e5dae71724aa8ff88d04c40ef4eaf1ddd2a
Gerrit-Change-Number: 13495
Gerrit-PatchSet: 3
Gerrit-Owner: Will Berkeley
Gerrit-Reviewer: Adar Dembo
Gerrit-Reviewer: Alexey Serbin
Gerrit-Reviewer: Grant Henke
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Will Berkeley
[kudu-CR] [java] Fix ordering bug in AsyncKuduSession
Will Berkeley has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13496 Change subject: [java] Fix ordering bug in AsyncKuduSession .. [java] Fix ordering bug in AsyncKuduSession AsyncKuduSession.testBatchErrorCauseSessionStuck occasionally failed because AsyncKuduSession#hasPendingOperations returned true even though the test joined on the deferred results of its two inserts. I tracked this down to a small mis-ordering of lines where the AsyncKuduSession fires a flush notification (from a Netty worker thread) before it actually swaps the just-flushed buffer back to the inactive list, so there was a small window of time when the client thread could observe hasPendingOperations() as true even though there were no pending operations. I ran the test 100 times in dist-test with this fix and saw no failures. I didn't run the test pre-fix through dist-test but the flaky dashboard indicated it was 2-3% flaky. Change-Id: I498636aeb0de2673c15b90260e6acf56de1ead06 --- M java/kudu-client/src/main/java/org/apache/kudu/client/AsyncKuduSession.java 1 file changed, 1 insertion(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/96/13496/1 -- To view, visit http://gerrit.cloudera.org:8080/13496 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I498636aeb0de2673c15b90260e6acf56de1ead06 Gerrit-Change-Number: 13496 Gerrit-PatchSet: 1 Gerrit-Owner: Will Berkeley
[kudu-CR] [java] Deflake TestKuduScanner.testDiffScan
Will Berkeley has posted comments on this change. ( http://gerrit.cloudera.org:8080/13495 ) Change subject: [java] Deflake TestKuduScanner.testDiffScan .. Patch Set 2: Verified+1 When will the flakes end :'( -- To view, visit http://gerrit.cloudera.org:8080/13495 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iadb46e5dae71724aa8ff88d04c40ef4eaf1ddd2a Gerrit-Change-Number: 13495 Gerrit-PatchSet: 2 Gerrit-Owner: Will Berkeley Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Will Berkeley Gerrit-Comment-Date: Mon, 03 Jun 2019 06:02:43 + Gerrit-HasComments: No
[kudu-CR] [java] Deflake TestKuduScanner.testDiffScan
Will Berkeley has removed a vote on this change. Change subject: [java] Deflake TestKuduScanner.testDiffScan .. Removed Verified-1 by Kudu Jenkins (120) -- To view, visit http://gerrit.cloudera.org:8080/13495 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: deleteVote Gerrit-Change-Id: Iadb46e5dae71724aa8ff88d04c40ef4eaf1ddd2a Gerrit-Change-Number: 13495 Gerrit-PatchSet: 2 Gerrit-Owner: Will Berkeley Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Kudu Jenkins (120)
[kudu-CR] [java] Deflake TestKuduScanner.testDiffScan
Hello Alexey Serbin, Kudu Jenkins, Adar Dembo, Grant Henke,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/13495
to look at the new patch set (#2).
Change subject: [java] Deflake TestKuduScanner.testDiffScan
..
[java] Deflake TestKuduScanner.testDiffScan
I noticed a funny failure in TestKuduScanner.testDiffScan:
21:55:41.681 [DEBUG - New I/O worker #13] (AsyncKuduScanner.java:556) Can not
open scanner
org.apache.kudu.client.NonRecoverableException: snapshot scan start timestamp
is earlier than the ancient history mark. Consider increasing the value of the
configuration parameter --tablet_history_max_age_sec. Snapshot timestamp: P: 0
usec, L: 0 Ancient History Mark: P: 1558907741677535 usec, L: 0 Physical time
difference: -1558907741.678s
The client logs the request it sent, which indeed has a
'snap_start_timestamp' of 0. This timestamp is supposed to come from
incrementing the propagated timestamp received from the client after
some random inserts and mutations. Fortunately, this test has nice
logging: it logs the operations it did and the timestamp:
21:55:40.874 [INFO - Time-limited test] (TestKuduScanner.java:250) Before: {}
21:55:40.874 [INFO - Time-limited test] (TestKuduScanner.java:255) startHT: 0
'generateMutationOperations' could generate 0 mutations, causing this
test to fail. This requires the random number generator to generate 0 3
times in a row when sampling uniformly from {0, 1, 2, 3, 4}, so the odds
of failure due to this defect in the test were only 1/125. I fixed this
by adjusting the number of initial inserts generated so it is always at
least 1.
I ran the test 1250 times with the fix and saw zero failures. Without
the fix, I saw 9/1250 failures.
Change-Id: Iadb46e5dae71724aa8ff88d04c40ef4eaf1ddd2a
---
M java/kudu-client/src/test/java/org/apache/kudu/client/TestKuduScanner.java
M java/kudu-test-utils/src/main/java/org/apache/kudu/test/RandomUtils.java
2 files changed, 32 insertions(+), 14 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/95/13495/2
--
To view, visit http://gerrit.cloudera.org:8080/13495
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Iadb46e5dae71724aa8ff88d04c40ef4eaf1ddd2a
Gerrit-Change-Number: 13495
Gerrit-PatchSet: 2
Gerrit-Owner: Will Berkeley
Gerrit-Reviewer: Adar Dembo
Gerrit-Reviewer: Alexey Serbin
Gerrit-Reviewer: Grant Henke
Gerrit-Reviewer: Kudu Jenkins (120)
[kudu-CR] [java] Deflake TestKuduScanner.testDiffScan
Will Berkeley has uploaded this change for review. (
http://gerrit.cloudera.org:8080/13495
Change subject: [java] Deflake TestKuduScanner.testDiffScan
..
[java] Deflake TestKuduScanner.testDiffScan
I noticed a funny failure in TestKuduScanner.testDiffScan:
21:55:41.681 [DEBUG - New I/O worker #13] (AsyncKuduScanner.java:556) Can not
open scanner
org.apache.kudu.client.NonRecoverableException: snapshot scan start timestamp
is earlier than the ancient history mark. Consider increasing the value of the
configuration parameter --tablet_history_max_age_sec. Snapshot timestamp: P: 0
usec, L: 0 Ancient History Mark: P: 1558907741677535 usec, L: 0 Physical time
difference: -1558907741.678s
The client logs the request it sent, which indeed has a 'snap_start_timestamp'
of 0. This timestamp is supposed to come from incrementing the propagated
timestamp received from the client after some random inserts and mutations.
Fortunately, this test has nice logging: it logs the operations it did and the
timestamp:
21:55:40.874 [INFO - Time-limited test] (TestKuduScanner.java:250) Before: {}
21:55:40.874 [INFO - Time-limited test] (TestKuduScanner.java:255) startHT: 0
'generateMutationOperations' could generate 0 mutations, causing this test to
fail. This requires the random number generate to generate 0 3 times in
a row when sample uniformly from {0, 1, 2, 3, 4}, so the odds of failure
due to this defect in the tests were only 1/125. I fixed this by
adjusting the number of inserts generated so it is always at least 1.
I ran the test 1250 times with the fix and saw zero failures. Without
the fix, I saw 9/1250 failures.
Change-Id: Iadb46e5dae71724aa8ff88d04c40ef4eaf1ddd2a
---
M java/kudu-client/src/test/java/org/apache/kudu/client/TestKuduScanner.java
M java/kudu-test-utils/src/main/java/org/apache/kudu/test/RandomUtils.java
2 files changed, 32 insertions(+), 14 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/95/13495/1
--
To view, visit http://gerrit.cloudera.org:8080/13495
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Iadb46e5dae71724aa8ff88d04c40ef4eaf1ddd2a
Gerrit-Change-Number: 13495
Gerrit-PatchSet: 1
Gerrit-Owner: Will Berkeley
[kudu-CR] [spark tools] KUDU-2831: Deflake DistributedDataGeneratorTest.testGenerateRandomData
Will Berkeley has posted comments on this change. ( http://gerrit.cloudera.org:8080/13477 ) Change subject: [spark tools] KUDU-2831: Deflake DistributedDataGeneratorTest.testGenerateRandomData .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/13477/2//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/13477/2//COMMIT_MSG@10 PS2, Line 10: an > a Hawkeye :( -- To view, visit http://gerrit.cloudera.org:8080/13477 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie2218328e400d76dbeab50cdb0d0241dfe56d20c Gerrit-Change-Number: 13477 Gerrit-PatchSet: 2 Gerrit-Owner: Will Berkeley Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Will Berkeley Gerrit-Comment-Date: Fri, 31 May 2019 16:39:27 + Gerrit-HasComments: Yes
[kudu-CR] [spark tools] KUDU-2831: Deflake DistributedDataGeneratorTest.testGenerateRandomData
Will Berkeley has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/13477 ) Change subject: [spark tools] KUDU-2831: Deflake DistributedDataGeneratorTest.testGenerateRandomData .. [spark tools] KUDU-2831: Deflake DistributedDataGeneratorTest.testGenerateRandomData Oops. It seems it's not possible to retrieve an accumulator by name. We were creating an new accumulator for collisions after the old one was cleaned up because the job finished and all references to it ended. This meant we never actually counted collisions, so when they happened our expectation of the number of rows written was off. This fixes the problem by plumbing a reference to the rows written and collisions accumulators back to the test code. Before this patch, I saw 12/1000 failures. After, 0/1000. Change-Id: Ie2218328e400d76dbeab50cdb0d0241dfe56d20c Reviewed-on: http://gerrit.cloudera.org:8080/13477 Reviewed-by: Grant Henke Tested-by: Kudu Jenkins Reviewed-by: Adar Dembo --- M java/kudu-spark-tools/src/main/scala/org/apache/kudu/spark/tools/DistributedDataGenerator.scala M java/kudu-spark-tools/src/test/scala/org/apache/kudu/spark/tools/DistributedDataGeneratorTest.scala 2 files changed, 22 insertions(+), 14 deletions(-) Approvals: Grant Henke: Looks good to me, but someone else must approve Kudu Jenkins: Verified Adar Dembo: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/13477 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ie2218328e400d76dbeab50cdb0d0241dfe56d20c Gerrit-Change-Number: 13477 Gerrit-PatchSet: 3 Gerrit-Owner: Will Berkeley Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Will Berkeley
[kudu-CR] [spark tools] KUDU-2831: Deflake DistributedDataGeneratorTest.testGenerateRandomData
Hello Kudu Jenkins, Adar Dembo, Grant Henke, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13477 to look at the new patch set (#2). Change subject: [spark tools] KUDU-2831: Deflake DistributedDataGeneratorTest.testGenerateRandomData .. [spark tools] KUDU-2831: Deflake DistributedDataGeneratorTest.testGenerateRandomData Oops. It seems it's not possible to retrieve an accumulator by name. We were creating an new accumulator for collisions after the old one was cleaned up because the job finished and all references to it ended. This meant we never actually counted collisions, so when they happened our expectation of the number of rows written was off. This fixes the problem by plumbing a reference to the rows written and collisions accumulators back to the test code. Before this patch, I saw 12/1000 failures. After, 0/1000. Change-Id: Ie2218328e400d76dbeab50cdb0d0241dfe56d20c --- M java/kudu-spark-tools/src/main/scala/org/apache/kudu/spark/tools/DistributedDataGenerator.scala M java/kudu-spark-tools/src/test/scala/org/apache/kudu/spark/tools/DistributedDataGeneratorTest.scala 2 files changed, 22 insertions(+), 14 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/77/13477/2 -- To view, visit http://gerrit.cloudera.org:8080/13477 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ie2218328e400d76dbeab50cdb0d0241dfe56d20c Gerrit-Change-Number: 13477 Gerrit-PatchSet: 2 Gerrit-Owner: Will Berkeley Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Kudu Jenkins (120)
[kudu-CR] [spark tools] KUDU-2831: Deflake DistributedDataGeneratorTest.testGenerateRandomData
Will Berkeley has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13477 Change subject: [spark tools] KUDU-2831: Deflake DistributedDataGeneratorTest.testGenerateRandomData .. [spark tools] KUDU-2831: Deflake DistributedDataGeneratorTest.testGenerateRandomData Oops. It seems it's not possible to retrieve an accumulator by name. We were creating an new accumulator for collisions after the old one was cleaned up because the job finished and all references to it ended. This meant we never actually counted collisions, so when they happened our expectation of the number of rows written was off. This fixes the problem by plumbing a reference to the rows written and collisions accumulators back to the test code. Before this patch, I saw 12/1000 failures. After, 0/1000. Change-Id: Ie2218328e400d76dbeab50cdb0d0241dfe56d20c --- M java/kudu-spark-tools/src/main/scala/org/apache/kudu/spark/tools/DistributedDataGenerator.scala M java/kudu-spark-tools/src/test/scala/org/apache/kudu/spark/tools/DistributedDataGeneratorTest.scala 2 files changed, 22 insertions(+), 14 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/77/13477/1 -- To view, visit http://gerrit.cloudera.org:8080/13477 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ie2218328e400d76dbeab50cdb0d0241dfe56d20c Gerrit-Change-Number: 13477 Gerrit-PatchSet: 1 Gerrit-Owner: Will Berkeley
[kudu-CR] [java] Deflake TestAuthTokenReacquire.testBasicMasterOperations
Will Berkeley has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/13461 ) Change subject: [java] Deflake TestAuthTokenReacquire.testBasicMasterOperations .. [java] Deflake TestAuthTokenReacquire.testBasicMasterOperations This test was flaky because DDL operations aren't exactly-once (KUDU-1527). The following could happen with table creation or deletion: 1. Thread A issues a CreateTable request for Table A. 2. The master receives the CreateTable request and gets to work creating Table A but has not responded to Thread A yet. 3. Thread B calls `dropConnectionsAndExpireTokens`, dropping connections to the master. 4. Thread A retries the CreateTable RPC, thinking it has failed because the connection was dropped. 5. The master responds with an error because the table exists. This works around the problem by catching and ignoring exceptions indicating that the table already exists in the case of table creation and catching and ignoring exceptions that the table is deleted or does not exist in the case of table deletion. Before this patch, I saw 16/100 runs fail in dist-test against TSAN binaries. With this patch, I saw 0/100 failures. Change-Id: Ifef0a254c6eb1cadeb54e9cef52f78c7ccc7b9c5 Reviewed-on: http://gerrit.cloudera.org:8080/13461 Tested-by: Kudu Jenkins Reviewed-by: Alexey Serbin --- M java/kudu-client/src/test/java/org/apache/kudu/client/TestAuthTokenReacquire.java 1 file changed, 22 insertions(+), 2 deletions(-) Approvals: Kudu Jenkins: Verified Alexey Serbin: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/13461 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ifef0a254c6eb1cadeb54e9cef52f78c7ccc7b9c5 Gerrit-Change-Number: 13461 Gerrit-PatchSet: 4 Gerrit-Owner: Will Berkeley Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Will Berkeley
[kudu-CR] [java] Deflake TestSecurity.testImportExportAuthenticationCredentials
Will Berkeley has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/13466 ) Change subject: [java] Deflake TestSecurity.testImportExportAuthenticationCredentials .. [java] Deflake TestSecurity.testImportExportAuthenticationCredentials This test was flaky for the same reason that SecureKuduSinkTest.testEventsWithShortTickets was flaky (as addressed in 3cbc025869494be2a1fa91582dfabc95d7bc88f9): the test was using short ticket lifetimes, and sometimes in TSAN the cluster would simply take longer than the ticket lifetime to start, causing the minicluster check to fail with "not authorized" errors. However, unlike the sink test, I don't see a reason why this test needs to use short lifetimes. It manually invalidates credentials and does not wait for expiration. So, I simply unset the option making it use short ticket lifetimes. Change-Id: Ifb9fd101deb82acc849ba5dd6dfe8a34a76eea03 Reviewed-on: http://gerrit.cloudera.org:8080/13466 Tested-by: Will Berkeley Reviewed-by: Alexey Serbin --- M java/kudu-client/src/test/java/org/apache/kudu/client/TestSecurity.java 1 file changed, 1 insertion(+), 2 deletions(-) Approvals: Will Berkeley: Verified Alexey Serbin: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/13466 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ifb9fd101deb82acc849ba5dd6dfe8a34a76eea03 Gerrit-Change-Number: 13466 Gerrit-PatchSet: 2 Gerrit-Owner: Will Berkeley Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Will Berkeley
[kudu-CR] [java] Deflake TestSecurity.testImportExportAuthenticationCredentials
Will Berkeley has posted comments on this change. ( http://gerrit.cloudera.org:8080/13466 ) Change subject: [java] Deflake TestSecurity.testImportExportAuthenticationCredentials .. Patch Set 1: Verified+1 Flakes -.- -- To view, visit http://gerrit.cloudera.org:8080/13466 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifb9fd101deb82acc849ba5dd6dfe8a34a76eea03 Gerrit-Change-Number: 13466 Gerrit-PatchSet: 1 Gerrit-Owner: Will Berkeley Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Will Berkeley Gerrit-Comment-Date: Thu, 30 May 2019 07:10:07 + Gerrit-HasComments: No
[kudu-CR] [java] Deflake TestSecurity.testImportExportAuthenticationCredentials
Will Berkeley has removed a vote on this change. Change subject: [java] Deflake TestSecurity.testImportExportAuthenticationCredentials .. Removed Verified-1 by Kudu Jenkins (120) -- To view, visit http://gerrit.cloudera.org:8080/13466 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: deleteVote Gerrit-Change-Id: Ifb9fd101deb82acc849ba5dd6dfe8a34a76eea03 Gerrit-Change-Number: 13466 Gerrit-PatchSet: 1 Gerrit-Owner: Will Berkeley Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Kudu Jenkins (120)
[kudu-CR] [java] Deflake TestSecurity.testImportExportAuthenticationCredentials
Will Berkeley has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13466 Change subject: [java] Deflake TestSecurity.testImportExportAuthenticationCredentials .. [java] Deflake TestSecurity.testImportExportAuthenticationCredentials This test was flaky for the same reason that SecureKuduSinkTest.testEventsWithShortTickets was flaky (as addressed in 3cbc025869494be2a1fa91582dfabc95d7bc88f9): the test was using short ticket lifetimes, and sometimes in TSAN the cluster would simply take longer than the ticket lifetime to start, causing the minicluster check to fail with "not authorized" errors. However, unlike the sink test, I don't see a reason why this test needs to use short lifetimes. It manually invalidates credentials and does not wait for expiration. So, I simply unset the option making it use short ticket lifetimes. Change-Id: Ifb9fd101deb82acc849ba5dd6dfe8a34a76eea03 --- M java/kudu-client/src/test/java/org/apache/kudu/client/TestSecurity.java 1 file changed, 1 insertion(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/66/13466/1 -- To view, visit http://gerrit.cloudera.org:8080/13466 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ifb9fd101deb82acc849ba5dd6dfe8a34a76eea03 Gerrit-Change-Number: 13466 Gerrit-PatchSet: 1 Gerrit-Owner: Will Berkeley
[kudu-CR] [java] Deflake TestAuthTokenReacquire.testBasicMasterOperations
Hello Kudu Jenkins, Andrew Wong, Adar Dembo, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13461 to look at the new patch set (#3). Change subject: [java] Deflake TestAuthTokenReacquire.testBasicMasterOperations .. [java] Deflake TestAuthTokenReacquire.testBasicMasterOperations This test was flaky because DDL operations aren't exactly-once (KUDU-1527). The following could happen with table creation or deletion: 1. Thread A issues a CreateTable request for Table A. 2. The master receives the CreateTable request and gets to work creating Table A but has not responded to Thread A yet. 3. Thread B calls `dropConnectionsAndExpireTokens`, dropping connections to the master. 4. Thread A retries the CreateTable RPC, thinking it has failed because the connection was dropped. 5. The master responds with an error because the table exists. This works around the problem by catching and ignoring exceptions indicating that the table already exists in the case of table creation and catching and ignoring exceptions that the table is deleted or does not exist in the case of table deletion. Before this patch, I saw 16/100 runs fail in dist-test against TSAN binaries. With this patch, I saw 0/100 failures. Change-Id: Ifef0a254c6eb1cadeb54e9cef52f78c7ccc7b9c5 --- M java/kudu-client/src/test/java/org/apache/kudu/client/TestAuthTokenReacquire.java 1 file changed, 22 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/61/13461/3 -- To view, visit http://gerrit.cloudera.org:8080/13461 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ifef0a254c6eb1cadeb54e9cef52f78c7ccc7b9c5 Gerrit-Change-Number: 13461 Gerrit-PatchSet: 3 Gerrit-Owner: Will Berkeley Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Will Berkeley
[kudu-CR] [java] Deflake TestAuthTokenReacquire.testBasicMasterOperations
Will Berkeley has posted comments on this change. ( http://gerrit.cloudera.org:8080/13461 ) Change subject: [java] Deflake TestAuthTokenReacquire.testBasicMasterOperations .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/13461/2/java/kudu-client/src/test/java/org/apache/kudu/client/TestAuthTokenReacquire.java File java/kudu-client/src/test/java/org/apache/kudu/client/TestAuthTokenReacquire.java: http://gerrit.cloudera.org:8080/#/c/13461/2/java/kudu-client/src/test/java/org/apache/kudu/client/TestAuthTokenReacquire.java@135 PS2, Line 135: // TODO(KUDU-1537): Remove this workaround when table creation is exactly-once. > master-stress-test also has this case: Done -- To view, visit http://gerrit.cloudera.org:8080/13461 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifef0a254c6eb1cadeb54e9cef52f78c7ccc7b9c5 Gerrit-Change-Number: 13461 Gerrit-PatchSet: 2 Gerrit-Owner: Will Berkeley Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Will Berkeley Gerrit-Comment-Date: Thu, 30 May 2019 06:25:59 + Gerrit-HasComments: Yes
[kudu-CR] [java] Attempt to deflake SecureKuduSinkTest
Will Berkeley has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/13454 ) Change subject: [java] Attempt to deflake SecureKuduSinkTest .. [java] Attempt to deflake SecureKuduSinkTest Occasionally, SecureKuduSinkTest fails when running with TSAN binaries because of the following sequence of operations: 1. The Kerberos ticket lifetime is set to 10s. 2. The test sets up a mini kudu cluster. This first sets up the KDC, which creates credentials for all of the Kudu servers and kinits using test user credentials for the test process. 3. The setup of the cluster takes > 10s. 4. At the end of the cluster setup, the test checks that setup succeeded in part by issuing a ListTabletServers RPC. This fails because the test user ticket has expired. 5. The test fails because it can't set up the cluster. The failure looks like 21:50:06.500 [ERROR - main] (RetryRule.java:217) org.apache.kudu.flume.sink.SecureKuduSinkTest.testEventsWithShortTickets: failed attempt 1 java.io.IOException: ListTabletServers RPC failed: Client connection negotiation failed: client connection to 127.12.111.60:36425: server requires authentication, but client does not have Kerberos credentials available at org.apache.kudu.test.cluster.MiniKuduCluster.sendRequestToCluster(MiniKuduCluster.java:169) ~[kudu-test-utils-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT] at org.apache.kudu.test.cluster.MiniKuduCluster.start(MiniKuduCluster.java:234) ~[kudu-test-utils-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT] at org.apache.kudu.test.cluster.MiniKuduCluster.access$300(MiniKuduCluster.java:71) ~[kudu-test-utils-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT] at org.apache.kudu.test.cluster.MiniKuduCluster$MiniKuduClusterBuilder.build(MiniKuduCluster.java:658) ~[kudu-test-utils-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT] at org.apache.kudu.test.KuduTestHarness.before(KuduTestHarness.java:140) ~[kudu-test-utils-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:46) ~[junit-4.12.jar:4.12] at org.apache.kudu.test.junit.RetryRule$RetryStatement.doOneAttempt(RetryRule.java:215) [kudu-test-utils-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT] at org.apache.kudu.test.junit.RetryRule$RetryStatement.evaluate(RetryRule.java:232) [kudu-test-utils-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT] ... This patch attempts to deflake the test a bit by doubling the ticket lifetime to 20s. It also raises the renewal lifetime to 35s from 30s, to provide a bit of extra time between the ticket expiring and when Flume needs to renew the ticket. Before, the test waited 2x the renewable ticket lifetime. I made it so the test waits until the renewable ticket lifetime plus one second has passed, including the time spent in the test so far. I tried to test this on dist-test using TSAN binaries. With the new patch I saw 0/1000 failures, but without it I saw 830/1000 failures. That's *way* flakier than any previous indication, so I don't trust those results. The failures I sampled did seem to be related to the same issue, but it was ConnectToCluster RPCs failing instead. Change-Id: Icc936878d7f1496905e83ddaf93b9b049f417f72 Reviewed-on: http://gerrit.cloudera.org:8080/13454 Reviewed-by: Adar Dembo Tested-by: Kudu Jenkins Reviewed-by: Alexey Serbin --- M java/kudu-flume-sink/src/test/java/org/apache/kudu/flume/sink/SecureKuduSinkTest.java 1 file changed, 9 insertions(+), 3 deletions(-) Approvals: Adar Dembo: Looks good to me, approved Kudu Jenkins: Verified Alexey Serbin: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/13454 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Icc936878d7f1496905e83ddaf93b9b049f417f72 Gerrit-Change-Number: 13454 Gerrit-PatchSet: 3 Gerrit-Owner: Will Berkeley Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Will Berkeley
[kudu-CR] [java] Deflake TestAuthTokenReacquire.testBasicMasterOperations
Will Berkeley has posted comments on this change. ( http://gerrit.cloudera.org:8080/13461 ) Change subject: [java] Deflake TestAuthTokenReacquire.testBasicMasterOperations .. Patch Set 1: (2 comments) http://gerrit.cloudera.org:8080/#/c/13461/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/13461/1//COMMIT_MSG@26 PS1, Line 26: bianries > binaries Done http://gerrit.cloudera.org:8080/#/c/13461/1/java/kudu-client/src/test/java/org/apache/kudu/client/TestAuthTokenReacquire.java File java/kudu-client/src/test/java/org/apache/kudu/client/TestAuthTokenReacquire.java: PS1: > Want to use the Status from inside the KuduException? Basically just need t Done -- To view, visit http://gerrit.cloudera.org:8080/13461 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifef0a254c6eb1cadeb54e9cef52f78c7ccc7b9c5 Gerrit-Change-Number: 13461 Gerrit-PatchSet: 1 Gerrit-Owner: Will Berkeley Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Will Berkeley Gerrit-Comment-Date: Thu, 30 May 2019 00:20:03 + Gerrit-HasComments: Yes
[kudu-CR] [java] Deflake TestAuthTokenReacquire.testBasicMasterOperations
Hello Kudu Jenkins, Andrew Wong, Adar Dembo, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13461 to look at the new patch set (#2). Change subject: [java] Deflake TestAuthTokenReacquire.testBasicMasterOperations .. [java] Deflake TestAuthTokenReacquire.testBasicMasterOperations This test was flaky because DDL operations aren't exactly-once (KUDU-1527). The following could happen with table creation or deletion: 1. Thread A issues a CreateTable request for Table A. 2. The master receives the CreateTable request and gets to work creating Table A but has not responded to Thread A yet. 3. Thread B calls `dropConnectionsAndExpireTokens`, dropping connections to the master. 4. Thread A retries the CreateTable RPC, thinking it has failed because the connection was dropped. 5. The master responds with an error because the table exists. This works around the problem by catching and ignoring exceptions indicating that the table already exists in the case of table creation and catching and ignoring exceptions that the table is deleted or does not exist in the case of table deletion. Before this patch, I saw 16/100 runs fail in dist-test against TSAN binaries. With this patch, I saw 0/100 failures. Change-Id: Ifef0a254c6eb1cadeb54e9cef52f78c7ccc7b9c5 --- M java/kudu-client/src/test/java/org/apache/kudu/client/TestAuthTokenReacquire.java 1 file changed, 21 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/61/13461/2 -- To view, visit http://gerrit.cloudera.org:8080/13461 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ifef0a254c6eb1cadeb54e9cef52f78c7ccc7b9c5 Gerrit-Change-Number: 13461 Gerrit-PatchSet: 2 Gerrit-Owner: Will Berkeley Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Kudu Jenkins (120)
[kudu-CR] [java] Attempt to deflake SecureKuduSinkTest
Hello Alexey Serbin, Kudu Jenkins, Adar Dembo, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13454 to look at the new patch set (#2). Change subject: [java] Attempt to deflake SecureKuduSinkTest .. [java] Attempt to deflake SecureKuduSinkTest Occasionally, SecureKuduSinkTest fails when running with TSAN binaries because of the following sequence of operations: 1. The Kerberos ticket lifetime is set to 10s. 2. The test sets up a mini kudu cluster. This first sets up the KDC, which creates credentials for all of the Kudu servers and kinits using test user credentials for the test process. 3. The setup of the cluster takes > 10s. 4. At the end of the cluster setup, the test checks that setup succeeded in part by issuing a ListTabletServers RPC. This fails because the test user ticket has expired. 5. The test fails because it can't set up the cluster. The failure looks like 21:50:06.500 [ERROR - main] (RetryRule.java:217) org.apache.kudu.flume.sink.SecureKuduSinkTest.testEventsWithShortTickets: failed attempt 1 java.io.IOException: ListTabletServers RPC failed: Client connection negotiation failed: client connection to 127.12.111.60:36425: server requires authentication, but client does not have Kerberos credentials available at org.apache.kudu.test.cluster.MiniKuduCluster.sendRequestToCluster(MiniKuduCluster.java:169) ~[kudu-test-utils-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT] at org.apache.kudu.test.cluster.MiniKuduCluster.start(MiniKuduCluster.java:234) ~[kudu-test-utils-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT] at org.apache.kudu.test.cluster.MiniKuduCluster.access$300(MiniKuduCluster.java:71) ~[kudu-test-utils-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT] at org.apache.kudu.test.cluster.MiniKuduCluster$MiniKuduClusterBuilder.build(MiniKuduCluster.java:658) ~[kudu-test-utils-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT] at org.apache.kudu.test.KuduTestHarness.before(KuduTestHarness.java:140) ~[kudu-test-utils-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:46) ~[junit-4.12.jar:4.12] at org.apache.kudu.test.junit.RetryRule$RetryStatement.doOneAttempt(RetryRule.java:215) [kudu-test-utils-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT] at org.apache.kudu.test.junit.RetryRule$RetryStatement.evaluate(RetryRule.java:232) [kudu-test-utils-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT] ... This patch attempts to deflake the test a bit by doubling the ticket lifetime to 20s. It also raises the renewal lifetime to 35s from 30s, to provide a bit of extra time between the ticket expiring and when Flume needs to renew the ticket. Before, the test waited 2x the renewable ticket lifetime. I made it so the test waits until the renewable ticket lifetime plus one second has passed, including the time spent in the test so far. I tried to test this on dist-test using TSAN binaries. With the new patch I saw 0/1000 failures, but without it I saw 830/1000 failures. That's *way* flakier than any previous indication, so I don't trust those results. The failures I sampled did seem to be related to the same issue, but it was ConnectToCluster RPCs failing instead. Change-Id: Icc936878d7f1496905e83ddaf93b9b049f417f72 --- M java/kudu-flume-sink/src/test/java/org/apache/kudu/flume/sink/SecureKuduSinkTest.java 1 file changed, 9 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/54/13454/2 -- To view, visit http://gerrit.cloudera.org:8080/13454 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Icc936878d7f1496905e83ddaf93b9b049f417f72 Gerrit-Change-Number: 13454 Gerrit-PatchSet: 2 Gerrit-Owner: Will Berkeley Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Will Berkeley
[kudu-CR] [java] Attempt to deflake SecureKuduSinkTest
Will Berkeley has posted comments on this change. (
http://gerrit.cloudera.org:8080/13454 )
Change subject: [java] Attempt to deflake SecureKuduSinkTest
..
Patch Set 1:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/13454/1//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/13454/1//COMMIT_MSG@25
PS1, Line 25: java.io.IOException: ListTabletServers RPC failed: Client
connection negotiation failed: client connection to 127.12.111.60:36425: server
requires authentication, but client does not have Kerberos credentials available
> Ah, ignore that: here we should be talking only C++ client/server negotiati
Alexey said he'd address this C++ logging improvement in another patch.
http://gerrit.cloudera.org:8080/#/c/13454/1/java/kudu-flume-sink/src/test/java/org/apache/kudu/flume/sink/SecureKuduSinkTest.java
File
java/kudu-flume-sink/src/test/java/org/apache/kudu/flume/sink/SecureKuduSinkTest.java:
http://gerrit.cloudera.org:8080/#/c/13454/1/java/kudu-flume-sink/src/test/java/org/apache/kudu/flume/sink/SecureKuduSinkTest.java@90
PS1, Line 90: LOG.info("Waiting for tickets to expire");
: TimeUnit.SECONDS.sleep(RENEWABLE_LIFETIME_SECONDS * 2);
> If my understanding is correct, this seems too conservative, indeed.
Done
--
To view, visit http://gerrit.cloudera.org:8080/13454
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icc936878d7f1496905e83ddaf93b9b049f417f72
Gerrit-Change-Number: 13454
Gerrit-PatchSet: 1
Gerrit-Owner: Will Berkeley
Gerrit-Reviewer: Adar Dembo
Gerrit-Reviewer: Alexey Serbin
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Will Berkeley
Gerrit-Comment-Date: Thu, 30 May 2019 00:14:13 +
Gerrit-HasComments: Yes
[kudu-CR] [java] Deflake TestAuthTokenReacquire.testBasicMasterOperations
Will Berkeley has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13461 Change subject: [java] Deflake TestAuthTokenReacquire.testBasicMasterOperations .. [java] Deflake TestAuthTokenReacquire.testBasicMasterOperations This test was flaky because DDL operations aren't exactly-once (KUDU-1527). The following could happen with table creation or deletion: 1. Thread A issues a CreateTable request for Table A. 2. The master receives the CreateTable request and gets to work creating Table A but has not responded to Thread A yet. 3. Thread B calls `dropConnectionsAndExpireTokens`, dropping connections to the master. 4. Thread A retries the CreateTable RPC, thinking it has failed because the connection was dropped. 5. The master responds with an error because the table exists. This works around the problem by catching and ignoring exceptions indicating that the table already exists in the case of table creation and catching and ignoring exceptions that the table is deleted or does not exist in the case of table deletion. Before this patch, I saw 16/100 runs fail in dist-test against TSAN bianries. With this patch, I saw 0/100 failures. Change-Id: Ifef0a254c6eb1cadeb54e9cef52f78c7ccc7b9c5 --- M java/kudu-client/src/test/java/org/apache/kudu/client/TestAuthTokenReacquire.java 1 file changed, 23 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/61/13461/1 -- To view, visit http://gerrit.cloudera.org:8080/13461 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ifef0a254c6eb1cadeb54e9cef52f78c7ccc7b9c5 Gerrit-Change-Number: 13461 Gerrit-PatchSet: 1 Gerrit-Owner: Will Berkeley
[kudu-CR] [java] Improve TestAsyncKuduSession: Refactor into more tests
Will Berkeley has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/13460 ) Change subject: [java] Improve TestAsyncKuduSession: Refactor into more tests .. [java] Improve TestAsyncKuduSession: Refactor into more tests TestAsyncKuduSession had a few tests, but one of them, ominously called `test`, was actually 7-8 tests in one, with each test shackled to the next by the shared state of the underlying table. The main improvement of this commit is to refactor mega-`test` into several targeted tests. There were some parts of the original mega test method for which I couldn't determine what precisely they were testing. Those parts have been dropped. I also made a few other small improvements which allowed me to drop almost all the helper code in favor of generic test utils. Change-Id: I73675605c04d16f23ce9cd61763a97486dfcb009 Reviewed-on: http://gerrit.cloudera.org:8080/13460 Reviewed-by: Adar Dembo Tested-by: Kudu Jenkins --- M java/kudu-client/src/test/java/org/apache/kudu/client/TestAsyncKuduSession.java 1 file changed, 150 insertions(+), 291 deletions(-) Approvals: Adar Dembo: Looks good to me, approved Kudu Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/13460 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I73675605c04d16f23ce9cd61763a97486dfcb009 Gerrit-Change-Number: 13460 Gerrit-PatchSet: 2 Gerrit-Owner: Will Berkeley Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Will Berkeley
[kudu-CR] [backup] KUDU-2786 Parallelize tables for backup and restore
Hello Mike Percy, Kudu Jenkins, Grant Henke, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13430 to look at the new patch set (#2). Change subject: [backup] KUDU-2786 Parallelize tables for backup and restore .. [backup] KUDU-2786 Parallelize tables for backup and restore This patch adds a hidden, experimental option to run backups and restores in parallel across tables. Managing resources across parallel backups and restores is very difficult: the sizes of tables in terms of number of tablets and size of tables can vary by orders of magnitude across a cluster, and there are many resources which may be constrained depending on many factors: CPU, memory, disk I/O, network, number of executors available. This patch doesn't do resource management. It will kick off the jobs in parallel, and it's up to Spark to manage the resources of parallel jobs. Maybe this will work well, maybe it won't... that's why this is just experimental. I tested manually on a Spark cluster to verify that jobs are actually run in parallel. Change-Id: I02f0a818a6fa372ab3c696c11882284877ce207e --- M java/kudu-backup/src/main/scala/org/apache/kudu/backup/KuduBackup.scala M java/kudu-backup/src/main/scala/org/apache/kudu/backup/KuduRestore.scala M java/kudu-backup/src/main/scala/org/apache/kudu/backup/Options.scala M java/kudu-backup/src/test/scala/org/apache/kudu/backup/TestKuduBackup.scala 4 files changed, 69 insertions(+), 14 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/30/13430/2 -- To view, visit http://gerrit.cloudera.org:8080/13430 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I02f0a818a6fa372ab3c696c11882284877ce207e Gerrit-Change-Number: 13430 Gerrit-PatchSet: 2 Gerrit-Owner: Will Berkeley Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Mike Percy
[kudu-CR] [backup] KUDU-2787 Allow single table failures for backup
Will Berkeley has posted comments on this change. (
http://gerrit.cloudera.org:8080/13415 )
Change subject: [backup] KUDU-2787 Allow single table failures for backup
..
Patch Set 5:
(6 comments)
http://gerrit.cloudera.org:8080/#/c/13415/4//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/13415/4//COMMIT_MSG@7
PS4, Line 7: [backup] KUDU-2787 Allow single table failures for backup
> s/and restore//
Done
http://gerrit.cloudera.org:8080/#/c/13415/4/java/kudu-backup/src/main/scala/org/apache/kudu/backup/KuduBackup.scala
File java/kudu-backup/src/main/scala/org/apache/kudu/backup/KuduBackup.scala:
http://gerrit.cloudera.org:8080/#/c/13415/4/java/kudu-backup/src/main/scala/org/apache/kudu/backup/KuduBackup.scala@140
PS4, Line 140: s"Failed to back up table $tableName: Look back in the
logs for the full exception. Error: ${ex.toString}")
> Is ex.toString not the full exception? You should be able to use `log.error
It doesn't include the stack trace. I don't want that here because it should be
printed above by the `log.error`.
http://gerrit.cloudera.org:8080/#/c/13415/4/java/kudu-backup/src/main/scala/org/apache/kudu/backup/KuduBackup.scala@142
PS4, Line 142: if (backupResults.exists(_._2.isFailure))
> Maybe return a boolean instead of an int here. That way this is just:
I like that it returns the exit code. It makes `main` very clean.
http://gerrit.cloudera.org:8080/#/c/13415/4/java/kudu-backup/src/main/scala/org/apache/kudu/backup/KuduBackup.scala@158
PS4, Line 158: System.exit(run(options, session))
> Instead of calling System.exit which is difficult to capture in testing, ca
Call `run` in tests.
http://gerrit.cloudera.org:8080/#/c/13415/4/java/kudu-backup/src/test/scala/org/apache/kudu/backup/TestKuduBackup.scala
File
java/kudu-backup/src/test/scala/org/apache/kudu/backup/TestKuduBackup.scala:
http://gerrit.cloudera.org:8080/#/c/13415/4/java/kudu-backup/src/test/scala/org/apache/kudu/backup/TestKuduBackup.scala@107
PS4, Line 107: // Check that a backup of a missing table fails fast with an
exception if the
> Is there a test for failOnFirstError = false?
Yes. See the second backup taken in `testFailedTableDoesNotFailOtherTables`.
http://gerrit.cloudera.org:8080/#/c/13415/4/java/kudu-backup/src/test/scala/org/apache/kudu/backup/TestKuduBackup.scala@596
PS4, Line 596: runBacku
> nit: Would you mind renaming this function to runBackup() to avoid confusio
Done
--
To view, visit http://gerrit.cloudera.org:8080/13415
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4f3731f8df04791727c44af9ade9989f2cc6ef95
Gerrit-Change-Number: 13415
Gerrit-PatchSet: 5
Gerrit-Owner: Will Berkeley
Gerrit-Reviewer: Grant Henke
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Mike Percy
Gerrit-Reviewer: Will Berkeley
Gerrit-Comment-Date: Wed, 29 May 2019 18:05:36 +
Gerrit-HasComments: Yes
[kudu-CR] [java] Attempt to deflake SecureKuduSinkTest
Will Berkeley has posted comments on this change. ( http://gerrit.cloudera.org:8080/13454 ) Change subject: [java] Attempt to deflake SecureKuduSinkTest .. Patch Set 1: Verified+1 Unrelated flake in DeleteTabletITest -- To view, visit http://gerrit.cloudera.org:8080/13454 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icc936878d7f1496905e83ddaf93b9b049f417f72 Gerrit-Change-Number: 13454 Gerrit-PatchSet: 1 Gerrit-Owner: Will Berkeley Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Will Berkeley Gerrit-Comment-Date: Wed, 29 May 2019 17:21:00 + Gerrit-HasComments: No
[kudu-CR] [backup] KUDU-2787 Pt 2 Allow single table failures for restore
Will Berkeley has posted comments on this change. ( http://gerrit.cloudera.org:8080/13416 ) Change subject: [backup] KUDU-2787 Pt 2 Allow single table failures for restore .. Patch Set 3: (6 comments) http://gerrit.cloudera.org:8080/#/c/13416/2//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/13416/2//COMMIT_MSG@7 PS2, Line 7: [backup] KUDU-2787 Pt 2 Allow single table failures for restore > s/backup and //; consider noting as part 2 of KUDU-2787 Done http://gerrit.cloudera.org:8080/#/c/13416/2/java/kudu-backup/src/main/scala/org/apache/kudu/backup/KuduRestore.scala File java/kudu-backup/src/main/scala/org/apache/kudu/backup/KuduRestore.scala: http://gerrit.cloudera.org:8080/#/c/13416/2/java/kudu-backup/src/main/scala/org/apache/kudu/backup/KuduRestore.scala@164 PS2, Line 164: TODO > mind filing a jira for this? Done http://gerrit.cloudera.org:8080/#/c/13416/2/java/kudu-backup/src/main/scala/org/apache/kudu/backup/KuduRestore.scala@166 PS2, Line 166: changed. We ought to try to clean up the mess when a failure > Are you thinking we should drop the table if we couldn't successfully resto Yeah, so a retry of the job wouldn't auto-fail. http://gerrit.cloudera.org:8080/#/c/13416/2/java/kudu-backup/src/main/scala/org/apache/kudu/backup/KuduRestore.scala@187 PS2, Line 187: if (restoreResults.exists(_._2.isFailure)) > See my comments on the backup patch. See my responses there. http://gerrit.cloudera.org:8080/#/c/13416/2/java/kudu-backup/src/main/scala/org/apache/kudu/backup/KuduRestore.scala@299 PS2, Line 299: System.exit(run(options, session)) > See my comments on the backup patch. Ditto. http://gerrit.cloudera.org:8080/#/c/13416/2/java/kudu-backup/src/test/scala/org/apache/kudu/backup/TestKuduBackup.scala File java/kudu-backup/src/test/scala/org/apache/kudu/backup/TestKuduBackup.scala: http://gerrit.cloudera.org:8080/#/c/13416/2/java/kudu-backup/src/test/scala/org/apache/kudu/backup/TestKuduBackup.scala@659 PS2, Line 659: runRestor > nit: can we rename this to runRestore() to avoid confusion with the method Done -- To view, visit http://gerrit.cloudera.org:8080/13416 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9dcc0be78e637b47f35e494dce4a6df274c8d559 Gerrit-Change-Number: 13416 Gerrit-PatchSet: 3 Gerrit-Owner: Will Berkeley Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Mike Percy Gerrit-Reviewer: Will Berkeley Gerrit-Comment-Date: Wed, 29 May 2019 18:05:32 + Gerrit-HasComments: Yes
[kudu-CR] [java] Deflake TestAsyncKuduSession with temporary workaround
Will Berkeley has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/13453 ) Change subject: [java] Deflake TestAsyncKuduSession with temporary workaround .. [java] Deflake TestAsyncKuduSession with temporary workaround TestAsyncKuduSession.test is kind of a mess. In particular, it has one section where it tries to insert 21 ops into a session with 20 ops of buffer space. It expects this to trigger a PleaseThrottleException. However, it is inherently racy because the session may manage to flush one or more buffers of ops to the server so no throttling is needed. It's always been an icky test but it wasn't flaky until 2bf80b5147f09c980c2f2268ee3467a5b3305a57 cleaned up the session code a bit. Now it's about 30% flaky in RELEASE mode according to the flaky test dashboard. Until I can spend some time to redo the test and understand exactly what about the previous change triggered the flakiness, I'm turning off the check for an exception. I also moved a debug log statement so there's less noise when calling flush in non-AUTO_FLUSH_BACKGROUND modes. Change-Id: I751f1ed3a151eee611a8b88c1527e12271024bf6 Reviewed-on: http://gerrit.cloudera.org:8080/13453 Tested-by: Kudu Jenkins Reviewed-by: Adar Dembo --- M java/kudu-client/src/main/java/org/apache/kudu/client/AsyncKuduSession.java M java/kudu-client/src/test/java/org/apache/kudu/client/TestAsyncKuduSession.java 2 files changed, 6 insertions(+), 10 deletions(-) Approvals: Kudu Jenkins: Verified Adar Dembo: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/13453 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I751f1ed3a151eee611a8b88c1527e12271024bf6 Gerrit-Change-Number: 13453 Gerrit-PatchSet: 2 Gerrit-Owner: Will Berkeley Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Will Berkeley
[kudu-CR] [backup] KUDU-2787 Pt 2 Allow single table failures for restore
Hello Mike Percy, Kudu Jenkins, Grant Henke, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13416 to look at the new patch set (#3). Change subject: [backup] KUDU-2787 Pt 2 Allow single table failures for restore .. [backup] KUDU-2787 Pt 2 Allow single table failures for restore This path enhances restore to be able to survive the failures of individual tables. It adds a new option `--failOnFirstError` to the restore job, which defaults to false, and which when true causes the restore job to fail as soon as any table has an error (the previous behavior). Change-Id: I9dcc0be78e637b47f35e494dce4a6df274c8d559 --- M java/kudu-backup/src/main/scala/org/apache/kudu/backup/KuduRestore.scala M java/kudu-backup/src/main/scala/org/apache/kudu/backup/Options.scala M java/kudu-backup/src/test/scala/org/apache/kudu/backup/TestKuduBackup.scala M java/kudu-backup/src/test/scala/org/apache/kudu/backup/TestOptions.scala 4 files changed, 169 insertions(+), 107 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/16/13416/3 -- To view, visit http://gerrit.cloudera.org:8080/13416 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I9dcc0be78e637b47f35e494dce4a6df274c8d559 Gerrit-Change-Number: 13416 Gerrit-PatchSet: 3 Gerrit-Owner: Will Berkeley Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Mike Percy
[kudu-CR] [backup] KUDU-2786 Parallelize tables for backup and restore
Will Berkeley has posted comments on this change. ( http://gerrit.cloudera.org:8080/13430 ) Change subject: [backup] KUDU-2786 Parallelize tables for backup and restore .. Patch Set 1: (4 comments) http://gerrit.cloudera.org:8080/#/c/13430/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/13430/1//COMMIT_MSG@10 PS1, Line 10: parallel > in parallel Done http://gerrit.cloudera.org:8080/#/c/13430/1//COMMIT_MSG@11 PS1, Line 11: is very difficult > How would queueing and resource management of this be any different than do I'm not sure, but I know there's a lot I don't know about Spark, and I know we haven't tested this very much. http://gerrit.cloudera.org:8080/#/c/13430/1//COMMIT_MSG@24 PS1, Line 24: : foo : : Change-Id: Ib02f26fbfd6a714ad0797f8b5ed1eeeb8fd6e371 : : b : : Change-Id: I02f0a818a6fa372ab3c696c11882284877ce207e > nit: remove these remnants of a git squash :( http://gerrit.cloudera.org:8080/#/c/13430/1/java/kudu-backup/src/main/scala/org/apache/kudu/backup/KuduBackup.scala File java/kudu-backup/src/main/scala/org/apache/kudu/backup/KuduBackup.scala: http://gerrit.cloudera.org:8080/#/c/13430/1/java/kudu-backup/src/main/scala/org/apache/kudu/backup/KuduBackup.scala@128 PS1, Line 128: val pool = new ForkJoinPool(options.numParallelBackups) // Need a clean-up reference. > Due to Data input and output format and layout assumptions it's easier to k +1 to what Grant said. Separate jobs can fail separately. Re: the ForkJoinPool, I don't think that the "Fork" here is the syscall fork. Of course the threads in the pool will be forked (or cloned) at some point, but I don't think the pool is forking for every task. See this for an explanation of why the pool is called a ForkJoinPool: http://tutorials.jenkov.com/java-util-concurrent/java-fork-and-join-forkjoinpool.html. Also, this is the default type of ExecutorService used, not a specific choice by me. I needed to configure the parallelism explicitly, else I could have dispensed with configuring my own pool. That would have resulted in parallelism equal to the number of processors on the driver node, which doesn't have much to do with the parallelism one might want in the restore job. -- To view, visit http://gerrit.cloudera.org:8080/13430 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I02f0a818a6fa372ab3c696c11882284877ce207e Gerrit-Change-Number: 13430 Gerrit-PatchSet: 1 Gerrit-Owner: Will Berkeley Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Mike Percy Gerrit-Reviewer: Will Berkeley Gerrit-Comment-Date: Wed, 29 May 2019 18:05:21 + Gerrit-HasComments: Yes
[kudu-CR] [java] Attempt to deflake SecureKuduSinkTest
Will Berkeley has removed a vote on this change. Change subject: [java] Attempt to deflake SecureKuduSinkTest .. Removed Verified-1 by Kudu Jenkins (120) -- To view, visit http://gerrit.cloudera.org:8080/13454 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: deleteVote Gerrit-Change-Id: Icc936878d7f1496905e83ddaf93b9b049f417f72 Gerrit-Change-Number: 13454 Gerrit-PatchSet: 1 Gerrit-Owner: Will Berkeley Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Will Berkeley
