[kudu-CR] preflight: add checks for openssl and zlib
Adar Dembo has removed Kudu Jenkins from this change. ( http://gerrit.cloudera.org:8080/10304 ) Change subject: preflight: add checks for openssl and zlib .. Removed reviewer Kudu Jenkins with the following votes: * Verified-1 by Kudu Jenkins (120) -- To view, visit http://gerrit.cloudera.org:8080/10304 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: deleteReviewer Gerrit-Change-Id: Ib9bd33ab489d742e9cdfc9af1aeedbd1905bd12a Gerrit-Change-Number: 10304 Gerrit-PatchSet: 1 Gerrit-Owner: Adar Dembo Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Todd Lipcon
[kudu-CR] preflight: add checks for openssl and zlib
Adar Dembo has posted comments on this change. ( http://gerrit.cloudera.org:8080/10304 ) Change subject: preflight: add checks for openssl and zlib .. Patch Set 1: Verified+1 Overriding Jenkins, unrelated failure. -- To view, visit http://gerrit.cloudera.org:8080/10304 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib9bd33ab489d742e9cdfc9af1aeedbd1905bd12a Gerrit-Change-Number: 10304 Gerrit-PatchSet: 1 Gerrit-Owner: Adar Dembo Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Fri, 04 May 2018 03:35:50 + Gerrit-HasComments: No
[kudu-CR] KUDU-2191: HMS inconsistent metadata fix tool
Hao Hao has posted comments on this change. ( http://gerrit.cloudera.org:8080/10303 ) Change subject: KUDU-2191: HMS inconsistent metadata fix tool .. Patch Set 2: Verified+1 Tests failed because of clock synchronization issue. -- To view, visit http://gerrit.cloudera.org:8080/10303 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I63c694b5d9877cfbd218139f2e9ecf05fd69c997 Gerrit-Change-Number: 10303 Gerrit-PatchSet: 2 Gerrit-Owner: Hao Hao Gerrit-Reviewer: Dan Burkert Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Tidy Bot Gerrit-Comment-Date: Fri, 04 May 2018 02:06:05 + Gerrit-HasComments: No
[kudu-CR] KUDU-2191: HMS inconsistent metadata fix tool
Hao Hao has removed a vote on this change. Change subject: KUDU-2191: HMS inconsistent metadata fix tool .. Removed Verified-1 by Kudu Jenkins (120) -- To view, visit http://gerrit.cloudera.org:8080/10303 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: deleteVote Gerrit-Change-Id: I63c694b5d9877cfbd218139f2e9ecf05fd69c997 Gerrit-Change-Number: 10303 Gerrit-PatchSet: 2 Gerrit-Owner: Hao Hao Gerrit-Reviewer: Dan Burkert Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Tidy Bot
[kudu-CR] KUDU-2191: HMS Metadata Consistency Check Tool
Hao Hao has posted comments on this change. (
http://gerrit.cloudera.org:8080/10217 )
Change subject: KUDU-2191: HMS Metadata Consistency Check Tool
..
Patch Set 5:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/10217/4/src/kudu/tools/kudu-tool-test.cc
File src/kudu/tools/kudu-tool-test.cc:
http://gerrit.cloudera.org:8080/#/c/10217/4/src/kudu/tools/kudu-tool-test.cc@2090
PS4, Line 2090: hive::EnvironmentContext env_ctx;
> error: no member named 'DropTableWithContext' in 'kudu::hms::HmsClient' [cl
Done
http://gerrit.cloudera.org:8080/#/c/10217/4/src/kudu/tools/kudu-tool-test.cc@2091
PS4, Line 2091: env_ctx.__set_properties({
make_pair(HmsClient::kKuduTableIdKey, hms_table_id) });
> error: no member named 'DropTableWithContext' in 'kudu::hms::HmsClient' [cl
Done
http://gerrit.cloudera.org:8080/#/c/10217/4/src/kudu/tools/tool_action_hms.cc
File src/kudu/tools/tool_action_hms.cc:
http://gerrit.cloudera.org:8080/#/c/10217/4/src/kudu/tools/tool_action_hms.cc@60
PS4, Line 60: using std::ostream;
> warning: using decl 'make_pair' is unused [misc-unused-using-decls]
Done
--
To view, visit http://gerrit.cloudera.org:8080/10217
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I3495e9c755571b85beaecca849f83457b592ee90
Gerrit-Change-Number: 10217
Gerrit-PatchSet: 5
Gerrit-Owner: Hao Hao
Gerrit-Reviewer: Dan Burkert
Gerrit-Reviewer: Hao Hao
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot
Gerrit-Comment-Date: Fri, 04 May 2018 00:32:58 +
Gerrit-HasComments: Yes
[kudu-CR] KUDU-2191: Metadata Upgrade Tool
Hello Tidy Bot, Alexey Serbin, Dan Burkert, Kudu Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/10075 to look at the new patch set (#12). Change subject: KUDU-2191: Metadata Upgrade Tool .. KUDU-2191: Metadata Upgrade Tool This commit introduces an upgrade tool to allows backward compatibility. It provides support to upgrade the metadata format of existing Impala managed/external tables in HMS entries, as well as rename the existing tables in Kudu to adapt to the new naming rules. It adds test case using external mini cluster to ensure the upgrade tool works as expected. Change-Id: I1c8488e1c7b04b8bedc31c83b1496b53bae49cb3 --- M java/kudu-hive/src/main/java/org/apache/kudu/hive/metastore/KuduMetastorePlugin.java M java/kudu-hive/src/test/java/org/apache/kudu/hive/metastore/TestKuduMetastorePlugin.java M src/kudu/client/client-test-util.cc M src/kudu/client/client-test-util.h M src/kudu/client/schema.h M src/kudu/hms/hms_catalog-test.cc M src/kudu/hms/hms_catalog.cc M src/kudu/hms/hms_catalog.h M src/kudu/hms/hms_client.cc M src/kudu/hms/hms_client.h M src/kudu/integration-tests/alter_table-randomized-test.cc M src/kudu/integration-tests/master_failover-itest.cc M src/kudu/integration-tests/master_hms-itest.cc M src/kudu/mini-cluster/external_mini_cluster.cc M src/kudu/mini-cluster/external_mini_cluster.h M src/kudu/tools/CMakeLists.txt M src/kudu/tools/kudu-tool-test.cc M src/kudu/tools/tool.proto M src/kudu/tools/tool_action.h A src/kudu/tools/tool_action_hms.cc M src/kudu/tools/tool_action_test.cc M src/kudu/tools/tool_main.cc 22 files changed, 653 insertions(+), 24 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/75/10075/12 -- To view, visit http://gerrit.cloudera.org:8080/10075 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I1c8488e1c7b04b8bedc31c83b1496b53bae49cb3 Gerrit-Change-Number: 10075 Gerrit-PatchSet: 12 Gerrit-Owner: Hao Hao Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Dan Burkert Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Tidy Bot
[kudu-CR] KUDU-2191: HMS Metadata Consistency Check Tool
Hello Tidy Bot, Dan Burkert, Kudu Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/10217 to look at the new patch set (#5). Change subject: KUDU-2191: HMS Metadata Consistency Check Tool .. KUDU-2191: HMS Metadata Consistency Check Tool This commit introduces a metadata consistency check CLI tool for Hive Metastore integration. It checks metadata (table ID, table name, Kudu master addresses, and colnum name and type) between Kudu and HMS, and report inconsistent metadata to the user if any. It adds test case using external mini cluster to ensure the tool works as expected. Change-Id: I3495e9c755571b85beaecca849f83457b592ee90 --- M src/kudu/hms/hms_catalog-test.cc M src/kudu/hms/hms_catalog.cc M src/kudu/hms/hms_catalog.h M src/kudu/tools/kudu-tool-test.cc M src/kudu/tools/tool_action_hms.cc 5 files changed, 304 insertions(+), 36 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/17/10217/5 -- To view, visit http://gerrit.cloudera.org:8080/10217 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I3495e9c755571b85beaecca849f83457b592ee90 Gerrit-Change-Number: 10217 Gerrit-PatchSet: 5 Gerrit-Owner: Hao Hao Gerrit-Reviewer: Dan Burkert Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Tidy Bot
[kudu-CR] bitmap-test: fix dangling-else warnings
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/10305 ) Change subject: bitmap-test: fix dangling-else warnings .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/10305 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0cf567b69c39958acb0c6fc7aa110d38f7b718ef Gerrit-Change-Number: 10305 Gerrit-PatchSet: 1 Gerrit-Owner: Adar Dembo Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Fri, 04 May 2018 00:23:49 + Gerrit-HasComments: No
[kudu-CR] preflight: add checks for openssl and zlib
Hello Todd Lipcon, I'd like you to do a code review. Please visit http://gerrit.cloudera.org:8080/10304 to review the following change. Change subject: preflight: add checks for openssl and zlib .. preflight: add checks for openssl and zlib The openssl devel package is required for Kudu, but also for squeasel, so we should check for its existence in preflight.py. Same goes for the zlib devel package, which is also required by IWYU. Change-Id: Ib9bd33ab489d742e9cdfc9af1aeedbd1905bd12a --- M thirdparty/preflight.py 1 file changed, 23 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/04/10304/1 -- To view, visit http://gerrit.cloudera.org:8080/10304 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ib9bd33ab489d742e9cdfc9af1aeedbd1905bd12a Gerrit-Change-Number: 10304 Gerrit-PatchSet: 1 Gerrit-Owner: Adar Dembo Gerrit-Reviewer: Todd Lipcon
[kudu-CR] bitmap-test: fix dangling-else warnings
Hello Todd Lipcon, I'd like you to do a code review. Please visit http://gerrit.cloudera.org:8080/10305 to review the following change. Change subject: bitmap-test: fix dangling-else warnings .. bitmap-test: fix dangling-else warnings This squelches the following two warnings: [396/750] Building CXX object src/kudu/util/CMakeFiles/bitmap-test.dir/bitmap-test.cc.o ../../src/kudu/util/bitmap-test.cc: In member function ‘virtual void kudu::TestBitMap_TestFindBit_Test::TestBody()’: ../../src/kudu/util/bitmap-test.cc:187:10: warning: suggest explicit braces to avoid ambiguous ‘else’ [-Wdangling-else] if (expect_set_found) ASSERT_EQ(expected_set_idx, idx); ^ ../../src/kudu/util/bitmap-test.cc:194:10: warning: suggest explicit braces to avoid ambiguous ‘else’ [-Wdangling-else] if (expect_zero_found) ASSERT_EQ(expected_zero_idx, idx); ^ Change-Id: I0cf567b69c39958acb0c6fc7aa110d38f7b718ef --- M src/kudu/util/bitmap-test.cc 1 file changed, 6 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/05/10305/1 -- To view, visit http://gerrit.cloudera.org:8080/10305 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I0cf567b69c39958acb0c6fc7aa110d38f7b718ef Gerrit-Change-Number: 10305 Gerrit-PatchSet: 1 Gerrit-Owner: Adar Dembo Gerrit-Reviewer: Todd Lipcon
[kudu-CR] KUDU-2191: HMS inconsistent metadata fix tool
Hao Hao has uploaded this change for review. ( http://gerrit.cloudera.org:8080/10303 Change subject: KUDU-2191: HMS inconsistent metadata fix tool .. KUDU-2191: HMS inconsistent metadata fix tool This commit introduces a CLI tool that fixes inconsistent metadata for Hive Metastore integration. It corrects the table name based on the users input. Ant it fixes the master addresses property, column name and type based on the values in Kudu. Test case is added using external mini cluster to ensure the tool works as expected. Change-Id: I63c694b5d9877cfbd218139f2e9ecf05fd69c997 --- M src/kudu/tools/kudu-tool-test.cc M src/kudu/tools/tool_action_hms.cc M src/kudu/tools/tool_test_util.cc M src/kudu/tools/tool_test_util.h 4 files changed, 232 insertions(+), 44 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/03/10303/1 -- To view, visit http://gerrit.cloudera.org:8080/10303 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I63c694b5d9877cfbd218139f2e9ecf05fd69c997 Gerrit-Change-Number: 10303 Gerrit-PatchSet: 1 Gerrit-Owner: Hao Hao
[kudu-CR] KUDU-2191: HMS Metadata Consistency Check Tool
Hello Tidy Bot, Dan Burkert, Kudu Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/10217 to look at the new patch set (#4). Change subject: KUDU-2191: HMS Metadata Consistency Check Tool .. KUDU-2191: HMS Metadata Consistency Check Tool This commit introduces a metadata consistency check CLI tool for Hive Metastore integration. It checks metadata (table ID, table name, Kudu master addresses, and colnum name and type) between Kudu and HMS, and report inconsistent metadata to the user if any. It adds test case using external mini cluster to ensure the tool works as expected. Change-Id: I3495e9c755571b85beaecca849f83457b592ee90 --- M src/kudu/hms/hms_catalog-test.cc M src/kudu/hms/hms_catalog.cc M src/kudu/hms/hms_catalog.h M src/kudu/tools/kudu-tool-test.cc M src/kudu/tools/tool_action_hms.cc 5 files changed, 302 insertions(+), 35 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/17/10217/4 -- To view, visit http://gerrit.cloudera.org:8080/10217 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I3495e9c755571b85beaecca849f83457b592ee90 Gerrit-Change-Number: 10217 Gerrit-PatchSet: 4 Gerrit-Owner: Hao Hao Gerrit-Reviewer: Dan Burkert Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Tidy Bot
[kudu-CR] KUDU-2191: Metadata Upgrade Tool
Hello Tidy Bot, Alexey Serbin, Dan Burkert, Kudu Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/10075 to look at the new patch set (#11). Change subject: KUDU-2191: Metadata Upgrade Tool .. KUDU-2191: Metadata Upgrade Tool This commit introduces an upgrade tool to allows backward compatibility. It provides support to upgrade the metadata format of existing Impala managed/external tables in HMS entries, as well as rename the existing tables in Kudu to adapt to the new naming rules. It adds test case using external mini cluster to ensure the upgrade tool works as expected. Change-Id: I1c8488e1c7b04b8bedc31c83b1496b53bae49cb3 --- M java/kudu-hive/src/main/java/org/apache/kudu/hive/metastore/KuduMetastorePlugin.java M java/kudu-hive/src/test/java/org/apache/kudu/hive/metastore/TestKuduMetastorePlugin.java M src/kudu/client/client-test-util.cc M src/kudu/client/client-test-util.h M src/kudu/client/schema.h M src/kudu/hms/hms_catalog-test.cc M src/kudu/hms/hms_catalog.cc M src/kudu/hms/hms_catalog.h M src/kudu/hms/hms_client.cc M src/kudu/hms/hms_client.h M src/kudu/integration-tests/alter_table-randomized-test.cc M src/kudu/integration-tests/master_failover-itest.cc M src/kudu/integration-tests/master_hms-itest.cc M src/kudu/mini-cluster/external_mini_cluster.cc M src/kudu/mini-cluster/external_mini_cluster.h M src/kudu/tools/CMakeLists.txt M src/kudu/tools/kudu-tool-test.cc M src/kudu/tools/tool.proto M src/kudu/tools/tool_action.h A src/kudu/tools/tool_action_hms.cc M src/kudu/tools/tool_action_test.cc M src/kudu/tools/tool_main.cc 22 files changed, 653 insertions(+), 24 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/75/10075/11 -- To view, visit http://gerrit.cloudera.org:8080/10075 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I1c8488e1c7b04b8bedc31c83b1496b53bae49cb3 Gerrit-Change-Number: 10075 Gerrit-PatchSet: 11 Gerrit-Owner: Hao Hao Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Dan Burkert Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Tidy Bot
[kudu-CR] Add validation and docs on setting block cache capacity mb too high
Will Berkeley has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/10266 ) Change subject: Add validation and docs on setting block_cache_capacity_mb too high .. Add validation and docs on setting block_cache_capacity_mb too high Raising --block_cache_capacity_mb can improve Kudu's performance, but raising it too high (above the memory pressure threshold, which is --memory_pressure_percentage of the hard memory limit) causes constant flushing even if write throughput is low. This patch adds validation of the capacity using a group flag validator, and does not permit Kudu to start if the block cache size can can cause Kudu to get stuck in aggressive flushing mode. It warns if the capacity is large enough (> 50%) of the pressure threshold. I chose 50% as the max recommended block cache size as a percentage of the memory pressure threshold somewhat arbitrarily. It seems safe, but also leaves room for allocating a large block cache. There's no automated tests but I did verify the expected messages appeared in the logs and the server fails to start when --block_cache_capacity_mb is too high. Change-Id: Ia99fa95c578235beb3a1c72a33a22b4d8ff4800c Reviewed-on: http://gerrit.cloudera.org:8080/10266 Reviewed-by: Alexey Serbin Tested-by: Kudu Jenkins --- M docs/troubleshooting.adoc M src/kudu/cfile/block_cache.cc M src/kudu/integration-tests/client-stress-test.cc M src/kudu/integration-tests/raft_consensus-itest.cc M src/kudu/util/process_memory.cc M src/kudu/util/process_memory.h 6 files changed, 76 insertions(+), 10 deletions(-) Approvals: Alexey Serbin: Looks good to me, approved Kudu Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/10266 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ia99fa95c578235beb3a1c72a33a22b4d8ff4800c Gerrit-Change-Number: 10266 Gerrit-PatchSet: 8 Gerrit-Owner: Will Berkeley Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Will Berkeley
[kudu-CR] Add validation and docs on setting block cache capacity mb too high
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/10266 ) Change subject: Add validation and docs on setting block_cache_capacity_mb too high .. Patch Set 7: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/10266 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia99fa95c578235beb3a1c72a33a22b4d8ff4800c Gerrit-Change-Number: 10266 Gerrit-PatchSet: 7 Gerrit-Owner: Will Berkeley Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Will Berkeley Gerrit-Comment-Date: Thu, 03 May 2018 22:09:56 + Gerrit-HasComments: No
[kudu-CR](branch-1.7.x) [Java] Fix TestSecurity tests on Mac
Grant Henke has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/10299 ) Change subject: [Java] Fix TestSecurity tests on Mac .. [Java] Fix TestSecurity tests on Mac TestSecurity tests were failing because InetAddress.getCanonicalHostName was returning localhost for the 127.0.0.1 IP address which results Kerberos rejecting as not kinit’d. Other tests don’t have this issue because they extend BaseKuduTest which installs the FakeDNS workaround for 127.* IP addresses. This patch adds the FakeDNS setup to the TestSecurity class. Change-Id: Ic61680f37c2d589ae4f5b553e69bc0b9eb15588d Reviewed-on: http://gerrit.cloudera.org:8080/10299 Reviewed-by: Dan Burkert Tested-by: Kudu Jenkins --- M java/kudu-client/src/test/java/org/apache/kudu/client/TestSecurity.java 1 file changed, 6 insertions(+), 0 deletions(-) Approvals: Dan Burkert: Looks good to me, approved Kudu Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/10299 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: branch-1.7.x Gerrit-MessageType: merged Gerrit-Change-Id: Ic61680f37c2d589ae4f5b553e69bc0b9eb15588d Gerrit-Change-Number: 10299 Gerrit-PatchSet: 2 Gerrit-Owner: Grant Henke Gerrit-Reviewer: Dan Burkert Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Kudu Jenkins
[kudu-CR] Add validation and docs on setting block cache capacity mb too high
Hello Alexey Serbin, Kudu Jenkins, Adar Dembo, Todd Lipcon, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/10266 to look at the new patch set (#7). Change subject: Add validation and docs on setting block_cache_capacity_mb too high .. Add validation and docs on setting block_cache_capacity_mb too high Raising --block_cache_capacity_mb can improve Kudu's performance, but raising it too high (above the memory pressure threshold, which is --memory_pressure_percentage of the hard memory limit) causes constant flushing even if write throughput is low. This patch adds validation of the capacity using a group flag validator, and does not permit Kudu to start if the block cache size can can cause Kudu to get stuck in aggressive flushing mode. It warns if the capacity is large enough (> 50%) of the pressure threshold. I chose 50% as the max recommended block cache size as a percentage of the memory pressure threshold somewhat arbitrarily. It seems safe, but also leaves room for allocating a large block cache. There's no automated tests but I did verify the expected messages appeared in the logs and the server fails to start when --block_cache_capacity_mb is too high. Change-Id: Ia99fa95c578235beb3a1c72a33a22b4d8ff4800c --- M docs/troubleshooting.adoc M src/kudu/cfile/block_cache.cc M src/kudu/integration-tests/client-stress-test.cc M src/kudu/integration-tests/raft_consensus-itest.cc M src/kudu/util/process_memory.cc M src/kudu/util/process_memory.h 6 files changed, 76 insertions(+), 10 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/66/10266/7 -- To view, visit http://gerrit.cloudera.org:8080/10266 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ia99fa95c578235beb3a1c72a33a22b4d8ff4800c Gerrit-Change-Number: 10266 Gerrit-PatchSet: 7 Gerrit-Owner: Will Berkeley Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Will Berkeley
[kudu-CR] [tools] ksck imp. [8/n]: Easy-to-see errors when running w/o admin
Will Berkeley has posted comments on this change. ( http://gerrit.cloudera.org:8080/10300 ) Change subject: [tools] ksck imp. [8/n]: Easy-to-see errors when running w/o admin .. Patch Set 1: Probably isn't worth reviewing until KUDU-2426 (https://gerrit.cloudera.org/#/c/10293/) is fixed. -- To view, visit http://gerrit.cloudera.org:8080/10300 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I644957103efefceeba4b82f4557376a603507423 Gerrit-Change-Number: 10300 Gerrit-PatchSet: 1 Gerrit-Owner: Will Berkeley Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Will Berkeley Gerrit-Comment-Date: Thu, 03 May 2018 21:49:14 + Gerrit-HasComments: No
[kudu-CR] [tools] ksck imp. [8/n]: Easy-to-see errors when running w/o admin
Will Berkeley has uploaded this change for review. ( http://gerrit.cloudera.org:8080/10300 Change subject: [tools] ksck imp. [8/n]: Easy-to-see errors when running w/o admin .. [tools] ksck imp. [8/n]: Easy-to-see errors when running w/o admin The consensus info gathering parts of ksck have clearly become central to its functionality, but occasionally administrators are fooled by the results when they run ksck without kudu admin privileges. This change surfaces unauthorized errors from the master and tablet servers clearly. This also removes the --consensus flag which could be used to turn off gathering consensus info. It shouldn't really be used as it can give misleading output where the cluster appears healthy but some tablets are actually unable to make progress. Change-Id: I644957103efefceeba4b82f4557376a603507423 --- M src/kudu/tools/ksck-test.cc M src/kudu/tools/ksck.cc M src/kudu/tools/tool_action_cluster.cc 3 files changed, 56 insertions(+), 12 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/00/10300/1 -- To view, visit http://gerrit.cloudera.org:8080/10300 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I644957103efefceeba4b82f4557376a603507423 Gerrit-Change-Number: 10300 Gerrit-PatchSet: 1 Gerrit-Owner: Will Berkeley
[kudu-CR] mini-cluster: support parallel multi-master clusters
Dan Burkert has removed Tidy Bot from this change. ( http://gerrit.cloudera.org:8080/8280 ) Change subject: mini-cluster: support parallel multi-master clusters .. Removed reviewer Tidy Bot. -- To view, visit http://gerrit.cloudera.org:8080/8280 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: deleteReviewer Gerrit-Change-Id: I0b0ff7bfc179d8fdb1ed306d1bbd12acddeb060c Gerrit-Change-Number: 8280 Gerrit-PatchSet: 9 Gerrit-Owner: Dan Burkert Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Dan Burkert Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Kudu Jenkins
[kudu-CR] mini-cluster: support parallel multi-master clusters
Dan Burkert has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/8280 ) Change subject: mini-cluster: support parallel multi-master clusters .. mini-cluster: support parallel multi-master clusters This commit refactors the mini-clusters to internally use reserved sockets for their child daemons. Reserved sockets are simply sockets bound to a random port with SO_REUSEPORT. As a result, master addresses for multi-master mini clusters no longer need to hard-coded master ports, and the associated ctest resource lock. This also significantly lessens the chances that port conflicts will occur, although it is still possible when masters are restarted during tests. To mitigate that on Linux, tests are still run in a unique subnet. Change-Id: I0b0ff7bfc179d8fdb1ed306d1bbd12acddeb060c Reviewed-on: http://gerrit.cloudera.org:8080/8280 Reviewed-by: Adar Dembo Tested-by: Dan Burkert --- M src/kudu/client/client-test.cc M src/kudu/integration-tests/CMakeLists.txt M src/kudu/integration-tests/catalog_manager_tsk-itest.cc M src/kudu/integration-tests/client-negotiation-failover-itest.cc M src/kudu/integration-tests/client-stress-test.cc M src/kudu/integration-tests/master-stress-test.cc M src/kudu/integration-tests/master_cert_authority-itest.cc M src/kudu/integration-tests/master_failover-itest.cc M src/kudu/integration-tests/master_migration-itest.cc M src/kudu/integration-tests/master_replication-itest.cc M src/kudu/integration-tests/security-faults-itest.cc M src/kudu/integration-tests/security-itest.cc M src/kudu/integration-tests/security-master-auth-itest.cc M src/kudu/integration-tests/token_signer-itest.cc M src/kudu/integration-tests/webserver-stress-itest.cc M src/kudu/master/mini_master.cc M src/kudu/mini-cluster/CMakeLists.txt M src/kudu/mini-cluster/external_mini_cluster-test.cc M src/kudu/mini-cluster/external_mini_cluster.cc M src/kudu/mini-cluster/external_mini_cluster.h M src/kudu/mini-cluster/internal_mini_cluster.cc M src/kudu/mini-cluster/internal_mini_cluster.h M src/kudu/mini-cluster/mini_cluster.cc M src/kudu/mini-cluster/mini_cluster.h M src/kudu/tools/CMakeLists.txt M src/kudu/tools/ksck_remote-test.cc M src/kudu/tools/tool_action_test.cc 27 files changed, 249 insertions(+), 326 deletions(-) Approvals: Adar Dembo: Looks good to me, approved Dan Burkert: Verified -- To view, visit http://gerrit.cloudera.org:8080/8280 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I0b0ff7bfc179d8fdb1ed306d1bbd12acddeb060c Gerrit-Change-Number: 8280 Gerrit-PatchSet: 10 Gerrit-Owner: Dan Burkert Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Dan Burkert Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Kudu Jenkins
[kudu-CR] mini-cluster: support parallel multi-master clusters
Dan Burkert has removed a vote on this change. Change subject: mini-cluster: support parallel multi-master clusters .. Removed Verified-1 by Kudu Jenkins (120) -- To view, visit http://gerrit.cloudera.org:8080/8280 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: deleteVote Gerrit-Change-Id: I0b0ff7bfc179d8fdb1ed306d1bbd12acddeb060c Gerrit-Change-Number: 8280 Gerrit-PatchSet: 9 Gerrit-Owner: Dan Burkert Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Dan Burkert Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Kudu Jenkins
[kudu-CR] mini-cluster: support parallel multi-master clusters
Dan Burkert has posted comments on this change. ( http://gerrit.cloudera.org:8080/8280 ) Change subject: mini-cluster: support parallel multi-master clusters .. Patch Set 9: Verified+1 This has failed three times in a row on non-intersecting flakes, so I'm going to override. -- To view, visit http://gerrit.cloudera.org:8080/8280 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0b0ff7bfc179d8fdb1ed306d1bbd12acddeb060c Gerrit-Change-Number: 8280 Gerrit-PatchSet: 9 Gerrit-Owner: Dan Burkert Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Dan Burkert Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Tidy Bot Gerrit-Comment-Date: Thu, 03 May 2018 21:42:50 + Gerrit-HasComments: No
[kudu-CR] [Java] Fix TestSecurity tests on Mac
Grant Henke has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/10298 ) Change subject: [Java] Fix TestSecurity tests on Mac .. [Java] Fix TestSecurity tests on Mac TestSecurity tests were failing because InetAddress.getCanonicalHostName was returning localhost for the 127.0.0.1 IP address which results Kerberos rejecting as not kinit’d. Other tests don’t have this issue because they extend BaseKuduTest which installs the FakeDNS workaround for 127.* IP addresses. This patch adds the FakeDNS setup to the TestSecurity class. Change-Id: Ic61680f37c2d589ae4f5b553e69bc0b9eb15588d Reviewed-on: http://gerrit.cloudera.org:8080/10298 Reviewed-by: Dan Burkert Tested-by: Kudu Jenkins --- M java/kudu-client/src/test/java/org/apache/kudu/client/TestSecurity.java 1 file changed, 6 insertions(+), 0 deletions(-) Approvals: Dan Burkert: Looks good to me, approved Kudu Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/10298 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ic61680f37c2d589ae4f5b553e69bc0b9eb15588d Gerrit-Change-Number: 10298 Gerrit-PatchSet: 2 Gerrit-Owner: Grant Henke Gerrit-Reviewer: Dan Burkert Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Kudu Jenkins
[kudu-CR] docs: workflow for backing up or restoring an entire node
Adar Dembo has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/10223 ) Change subject: docs: workflow for backing up or restoring an entire node .. docs: workflow for backing up or restoring an entire node I explicitly chose to be vague and not provide example shell commands because this is such a niche workflow. Change-Id: I638f4b169282d6688dec414eabfe4c81d0a3f5df Reviewed-on: http://gerrit.cloudera.org:8080/10223 Reviewed-by: Alexey Serbin Reviewed-by: Will Berkeley Tested-by: Kudu Jenkins --- M docs/administration.adoc 1 file changed, 29 insertions(+), 0 deletions(-) Approvals: Alexey Serbin: Looks good to me, approved Will Berkeley: Looks good to me, approved Kudu Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/10223 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I638f4b169282d6688dec414eabfe4c81d0a3f5df Gerrit-Change-Number: 10223 Gerrit-PatchSet: 4 Gerrit-Owner: Adar Dembo Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Will Berkeley
[kudu-CR] docs: workflow for backing up or restoring an entire node
Will Berkeley has posted comments on this change. ( http://gerrit.cloudera.org:8080/10223 ) Change subject: docs: workflow for backing up or restoring an entire node .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/10223 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I638f4b169282d6688dec414eabfe4c81d0a3f5df Gerrit-Change-Number: 10223 Gerrit-PatchSet: 3 Gerrit-Owner: Adar Dembo Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Will Berkeley Gerrit-Comment-Date: Thu, 03 May 2018 21:23:29 + Gerrit-HasComments: No
[kudu-CR](branch-1.7.x) [Java] Fix TestSecurity tests on Mac
Dan Burkert has posted comments on this change. ( http://gerrit.cloudera.org:8080/10299 ) Change subject: [Java] Fix TestSecurity tests on Mac .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/10299 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: branch-1.7.x Gerrit-MessageType: comment Gerrit-Change-Id: Ic61680f37c2d589ae4f5b553e69bc0b9eb15588d Gerrit-Change-Number: 10299 Gerrit-PatchSet: 1 Gerrit-Owner: Grant Henke Gerrit-Reviewer: Dan Burkert Gerrit-Reviewer: Kudu Jenkins Gerrit-Comment-Date: Thu, 03 May 2018 20:57:28 + Gerrit-HasComments: No
[kudu-CR] Add validation and docs on setting block cache capacity mb too high
Adar Dembo has posted comments on this change. ( http://gerrit.cloudera.org:8080/10266 ) Change subject: Add validation and docs on setting block_cache_capacity_mb too high .. Patch Set 6: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/10266 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia99fa95c578235beb3a1c72a33a22b4d8ff4800c Gerrit-Change-Number: 10266 Gerrit-PatchSet: 6 Gerrit-Owner: Will Berkeley Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Will Berkeley Gerrit-Comment-Date: Thu, 03 May 2018 20:55:04 + Gerrit-HasComments: No
[kudu-CR] Add validation and docs on setting block cache capacity mb too high
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/10266 ) Change subject: Add validation and docs on setting block_cache_capacity_mb too high .. Patch Set 6: Code-Review+2 LGTM; it's worth checking with Adar as well before submitting. -- To view, visit http://gerrit.cloudera.org:8080/10266 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia99fa95c578235beb3a1c72a33a22b4d8ff4800c Gerrit-Change-Number: 10266 Gerrit-PatchSet: 6 Gerrit-Owner: Will Berkeley Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Will Berkeley Gerrit-Comment-Date: Thu, 03 May 2018 20:54:29 + Gerrit-HasComments: No
[kudu-CR](branch-1.7.x) [Java] Fix TestSecurity tests on Mac
Grant Henke has uploaded this change for review. ( http://gerrit.cloudera.org:8080/10299 Change subject: [Java] Fix TestSecurity tests on Mac .. [Java] Fix TestSecurity tests on Mac TestSecurity tests were failing because InetAddress.getCanonicalHostName was returning localhost for the 127.0.0.1 IP address which results Kerberos rejecting as not kinit’d. Other tests don’t have this issue because they extend BaseKuduTest which installs the FakeDNS workaround for 127.* IP addresses. This patch adds the FakeDNS setup to the TestSecurity class. Change-Id: Ic61680f37c2d589ae4f5b553e69bc0b9eb15588d --- M java/kudu-client/src/test/java/org/apache/kudu/client/TestSecurity.java 1 file changed, 6 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/99/10299/1 -- To view, visit http://gerrit.cloudera.org:8080/10299 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: branch-1.7.x Gerrit-MessageType: newchange Gerrit-Change-Id: Ic61680f37c2d589ae4f5b553e69bc0b9eb15588d Gerrit-Change-Number: 10299 Gerrit-PatchSet: 1 Gerrit-Owner: Grant Henke
[kudu-CR] docs: workflow for backing up or restoring an entire node
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/10223 ) Change subject: docs: workflow for backing up or restoring an entire node .. Patch Set 3: Code-Review+2 LGTM, maybe Will or Grant might have some more feedback. -- To view, visit http://gerrit.cloudera.org:8080/10223 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I638f4b169282d6688dec414eabfe4c81d0a3f5df Gerrit-Change-Number: 10223 Gerrit-PatchSet: 3 Gerrit-Owner: Adar Dembo Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Will Berkeley Gerrit-Comment-Date: Thu, 03 May 2018 20:50:05 + Gerrit-HasComments: No
[kudu-CR] [Java] Fix TestSecurity tests on Mac
Dan Burkert has posted comments on this change. ( http://gerrit.cloudera.org:8080/10298 ) Change subject: [Java] Fix TestSecurity tests on Mac .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/10298 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic61680f37c2d589ae4f5b553e69bc0b9eb15588d Gerrit-Change-Number: 10298 Gerrit-PatchSet: 1 Gerrit-Owner: Grant Henke Gerrit-Reviewer: Dan Burkert Gerrit-Reviewer: Kudu Jenkins Gerrit-Comment-Date: Thu, 03 May 2018 20:48:04 + Gerrit-HasComments: No
[kudu-CR] mini-cluster: support parallel multi-master clusters
Adar Dembo has posted comments on this change. ( http://gerrit.cloudera.org:8080/8280 ) Change subject: mini-cluster: support parallel multi-master clusters .. Patch Set 9: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/8280 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0b0ff7bfc179d8fdb1ed306d1bbd12acddeb060c Gerrit-Change-Number: 8280 Gerrit-PatchSet: 9 Gerrit-Owner: Dan Burkert Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Dan Burkert Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Tidy Bot Gerrit-Comment-Date: Thu, 03 May 2018 20:47:51 + Gerrit-HasComments: No
[kudu-CR] docs: workflow for backing up or restoring an entire node
Hello Will Berkeley, Alexey Serbin, Kudu Jenkins, Grant Henke, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/10223 to look at the new patch set (#3). Change subject: docs: workflow for backing up or restoring an entire node .. docs: workflow for backing up or restoring an entire node I explicitly chose to be vague and not provide example shell commands because this is such a niche workflow. Change-Id: I638f4b169282d6688dec414eabfe4c81d0a3f5df --- M docs/administration.adoc 1 file changed, 29 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/23/10223/3 -- To view, visit http://gerrit.cloudera.org:8080/10223 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I638f4b169282d6688dec414eabfe4c81d0a3f5df Gerrit-Change-Number: 10223 Gerrit-PatchSet: 3 Gerrit-Owner: Adar Dembo Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Will Berkeley
[kudu-CR] docs: workflow for backing up or restoring an entire node
Adar Dembo has posted comments on this change. ( http://gerrit.cloudera.org:8080/10223 ) Change subject: docs: workflow for backing up or restoring an entire node .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/10223/2/docs/administration.adoc File docs/administration.adoc: http://gerrit.cloudera.org:8080/#/c/10223/2/docs/administration.adoc@996 PS2, Line 996: If creating a backup, make a file-attribute-preserving copy of the WAL, : metadata, and data directories on each node to be backed up. If restoring from : a backup, delete the existing WAL, metadata, and data directories, then : restore the backup via move or file-attribute-preserving copy. > Does it make sense to mention using backup/restore tools which are aware of Good point. Done -- To view, visit http://gerrit.cloudera.org:8080/10223 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I638f4b169282d6688dec414eabfe4c81d0a3f5df Gerrit-Change-Number: 10223 Gerrit-PatchSet: 2 Gerrit-Owner: Adar Dembo Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Will Berkeley Gerrit-Comment-Date: Thu, 03 May 2018 20:44:02 + Gerrit-HasComments: Yes
[kudu-CR] [Java] Fix TestSecurity tests on Mac
Grant Henke has uploaded this change for review. ( http://gerrit.cloudera.org:8080/10298 Change subject: [Java] Fix TestSecurity tests on Mac .. [Java] Fix TestSecurity tests on Mac TestSecurity tests were failing because InetAddress.getCanonicalHostName was returning localhost for the 127.0.0.1 IP address which results Kerberos rejecting as not kinit’d. Other tests don’t have this issue because they extend BaseKuduTest which installs the FakeDNS workaround for 127.* IP addresses. This patch adds the FakeDNS setup to the TestSecurity class. Change-Id: Ic61680f37c2d589ae4f5b553e69bc0b9eb15588d --- M java/kudu-client/src/test/java/org/apache/kudu/client/TestSecurity.java 1 file changed, 6 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/98/10298/1 -- To view, visit http://gerrit.cloudera.org:8080/10298 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ic61680f37c2d589ae4f5b553e69bc0b9eb15588d Gerrit-Change-Number: 10298 Gerrit-PatchSet: 1 Gerrit-Owner: Grant Henke
[kudu-CR] Add validation and docs on setting block cache capacity mb too high
Will Berkeley has posted comments on this change. ( http://gerrit.cloudera.org:8080/10266 ) Change subject: Add validation and docs on setting block_cache_capacity_mb too high .. Patch Set 5: (8 comments) http://gerrit.cloudera.org:8080/#/c/10266/5/src/kudu/cfile/block_cache.cc File src/kudu/cfile/block_cache.cc: http://gerrit.cloudera.org:8080/#/c/10266/5/src/kudu/cfile/block_cache.cc@41 PS5, Line 41: DEFINE_bool(force_block_cache_capacity_mb, false, > Nit: the _mb suffix makes me think this is a megabyte quantity. Done http://gerrit.cloudera.org:8080/#/c/10266/5/src/kudu/cfile/block_cache.cc@77 PS5, Line 77: LOG(ERROR) << "Block cache capacity exceeds the memory pressure threshold " > Nit: use strings::Substitute() here and below, makes the text a little more Done http://gerrit.cloudera.org:8080/#/c/10266/5/src/kudu/cfile/block_cache.cc@80 PS5, Line 80: > nit: extra space Done http://gerrit.cloudera.org:8080/#/c/10266/5/src/kudu/cfile/block_cache.cc@94 PS5, Line 94: // Group validation is necessary since the validator depends on : // --memory_limit_hard_bytes and other memory flags. > FWIW, I think this is implied by the content of the validator. Done http://gerrit.cloudera.org:8080/#/c/10266/5/src/kudu/integration-tests/client-stress-test.cc File src/kudu/integration-tests/client-stress-test.cc: http://gerrit.cloudera.org:8080/#/c/10266/5/src/kudu/integration-tests/client-stress-test.cc@241 PS5, Line 241: // The block cache capacity needs to be disabled. See the group validator > Hmm, this just disables the block cache outright, correct? Reword to say th Done http://gerrit.cloudera.org:8080/#/c/10266/5/src/kudu/integration-tests/client-stress-test.cc@241 PS5, Line 241: // The block cache capacity needs to be disabled. See the group validator > An alternative approach might be using --force_block_cache_capacity (since Done http://gerrit.cloudera.org:8080/#/c/10266/5/src/kudu/integration-tests/raft_consensus-itest.cc File src/kudu/integration-tests/raft_consensus-itest.cc: http://gerrit.cloudera.org:8080/#/c/10266/5/src/kudu/integration-tests/raft_consensus-itest.cc@1803 PS5, Line 1803: // We also need to limit the block cache capacity in tandem. See the group > Maybe, since the --force_block_cache_capacity is introduced, it would be po Done http://gerrit.cloudera.org:8080/#/c/10266/5/src/kudu/integration-tests/raft_consensus-itest.cc@1803 PS5, Line 1803: // We also need to limit the block cache capacity in tandem. See the group > Again, isn't this outright disabling? Done -- To view, visit http://gerrit.cloudera.org:8080/10266 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia99fa95c578235beb3a1c72a33a22b4d8ff4800c Gerrit-Change-Number: 10266 Gerrit-PatchSet: 5 Gerrit-Owner: Will Berkeley Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Will Berkeley Gerrit-Comment-Date: Thu, 03 May 2018 20:37:52 + Gerrit-HasComments: Yes
[kudu-CR] Add validation and docs on setting block cache capacity mb too high
Hello Alexey Serbin, Kudu Jenkins, Adar Dembo, Todd Lipcon, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/10266 to look at the new patch set (#6). Change subject: Add validation and docs on setting block_cache_capacity_mb too high .. Add validation and docs on setting block_cache_capacity_mb too high Raising --block_cache_capacity_mb can improve Kudu's performance, but raising it too high (above the memory pressure threshold, which is --memory_pressure_percentage of the hard memory limit) causes constant flushing even if write throughput is low. This patch adds validation of the capacity using a group flag validator, and does not permit Kudu to start if the block cache size can can cause Kudu to get stuck in aggressive flushing mode. It warns if the capacity is large enough (> 50%) of the pressure threshold. I chose 50% as the max recommended block cache size as a percentage of the memory pressure threshold somewhat arbitrarily. It seems safe, but also leaves room for allocating a large block cache. There's no automated tests but I did verify the expected messages appeared in the logs and the server fails to start when --block_cache_capacity_mb is too high. Change-Id: Ia99fa95c578235beb3a1c72a33a22b4d8ff4800c --- M docs/troubleshooting.adoc M src/kudu/cfile/block_cache.cc M src/kudu/integration-tests/client-stress-test.cc M src/kudu/integration-tests/raft_consensus-itest.cc M src/kudu/util/process_memory.cc M src/kudu/util/process_memory.h 6 files changed, 77 insertions(+), 10 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/66/10266/6 -- To view, visit http://gerrit.cloudera.org:8080/10266 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ia99fa95c578235beb3a1c72a33a22b4d8ff4800c Gerrit-Change-Number: 10266 Gerrit-PatchSet: 6 Gerrit-Owner: Will Berkeley Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Will Berkeley
[kudu-CR] docs: workflow for backing up or restoring an entire node
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/10223 ) Change subject: docs: workflow for backing up or restoring an entire node .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/10223/2/docs/administration.adoc File docs/administration.adoc: http://gerrit.cloudera.org:8080/#/c/10223/2/docs/administration.adoc@996 PS2, Line 996: If creating a backup, make a file-attribute-preserving copy of the WAL, : metadata, and data directories on each node to be backed up. If restoring from : a backup, delete the existing WAL, metadata, and data directories, then : restore the backup via move or file-attribute-preserving copy. Does it make sense to mention using backup/restore tools which are aware of sparse files (like using '-S' or '--sparse' option for tar)? -- To view, visit http://gerrit.cloudera.org:8080/10223 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I638f4b169282d6688dec414eabfe4c81d0a3f5df Gerrit-Change-Number: 10223 Gerrit-PatchSet: 2 Gerrit-Owner: Adar Dembo Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Will Berkeley Gerrit-Comment-Date: Thu, 03 May 2018 20:31:47 + Gerrit-HasComments: Yes
[kudu-CR] KUDU-2191 (10/n): PoC: disallow Kudu alter column, create table operations from Hive
Dan Burkert has posted comments on this change. ( http://gerrit.cloudera.org:8080/10111 ) Change subject: KUDU-2191 (10/n): PoC: disallow Kudu alter column, create table operations from Hive .. Patch Set 3: Code-Review+2 Carrying over Hao's +2 -- To view, visit http://gerrit.cloudera.org:8080/10111 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idc39b1df07952d37e9ce93f1673aad535767a8ac Gerrit-Change-Number: 10111 Gerrit-PatchSet: 3 Gerrit-Owner: Dan Burkert Gerrit-Reviewer: Dan Burkert Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Kudu Jenkins Gerrit-Comment-Date: Thu, 03 May 2018 20:31:01 + Gerrit-HasComments: No
[kudu-CR] KUDU-2191 (10/n): PoC: disallow Kudu alter column, create table operations from Hive
Dan Burkert has submitted this change and it was merged. (
http://gerrit.cloudera.org:8080/10111 )
Change subject: KUDU-2191 (10/n): PoC: disallow Kudu alter column, create table
operations from Hive
..
KUDU-2191 (10/n): PoC: disallow Kudu alter column, create table operations from
Hive
This commit makes it more difficult for processes or users who are not
the Kudu Master to alter the columns of Kudu table HMS entries through
Hive APIs.
The bulk of this patch is introducing a new environment context key/pair
('kudu.master_event=true') which is sent by the Kudu Master with every
HMS create/drop/alter table call. The KuduMetastorePlugin checks that
this property is set appropriately when operations modify a Kudu table
HMS entry.
The result is any attempt to create a table through Hive APIs or alter
the columns of a Kudu table through Hive APIs will fail. These
operations can not be supported because Hive lacks the DDL syntax and
HMS APIs to support necessary Kudu-specific table and column options.
It's possible for a user to spoof this environment context entry, but
it's a big enough roadblock that it should prevent accidental DDL
operations.
Change-Id: Idc39b1df07952d37e9ce93f1673aad535767a8ac
Reviewed-on: http://gerrit.cloudera.org:8080/10111
Tested-by: Kudu Jenkins
Reviewed-by: Dan Burkert
---
M
java/kudu-hive/src/main/java/org/apache/kudu/hive/metastore/KuduMetastorePlugin.java
M
java/kudu-hive/src/test/java/org/apache/kudu/hive/metastore/TestKuduMetastorePlugin.java
M src/kudu/hms/hms_catalog.cc
M src/kudu/hms/hms_catalog.h
M src/kudu/hms/hms_client-test.cc
M src/kudu/hms/hms_client.cc
M src/kudu/hms/hms_client.h
M src/kudu/integration-tests/master_hms-itest.cc
8 files changed, 125 insertions(+), 37 deletions(-)
Approvals:
Kudu Jenkins: Verified
Dan Burkert: Looks good to me, approved
--
To view, visit http://gerrit.cloudera.org:8080/10111
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Idc39b1df07952d37e9ce93f1673aad535767a8ac
Gerrit-Change-Number: 10111
Gerrit-PatchSet: 4
Gerrit-Owner: Dan Burkert
Gerrit-Reviewer: Dan Burkert
Gerrit-Reviewer: Hao Hao
Gerrit-Reviewer: Kudu Jenkins
[kudu-CR] Add validation and docs on setting block cache capacity mb too high
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/10266 ) Change subject: Add validation and docs on setting block_cache_capacity_mb too high .. Patch Set 5: (4 comments) http://gerrit.cloudera.org:8080/#/c/10266/5/src/kudu/cfile/block_cache.cc File src/kudu/cfile/block_cache.cc: http://gerrit.cloudera.org:8080/#/c/10266/5/src/kudu/cfile/block_cache.cc@80 PS5, Line 80: nit: extra space http://gerrit.cloudera.org:8080/#/c/10266/5/src/kudu/integration-tests/client-stress-test.cc File src/kudu/integration-tests/client-stress-test.cc: http://gerrit.cloudera.org:8080/#/c/10266/5/src/kudu/integration-tests/client-stress-test.cc@241 PS5, Line 241: // The block cache capacity needs to be disabled. See the group validator > Hmm, this just disables the block cache outright, correct? Reword to say th An alternative approach might be using --force_block_cache_capacity (since it's been introduced already) instead of disabling the block cache at all? http://gerrit.cloudera.org:8080/#/c/10266/5/src/kudu/integration-tests/raft_consensus-itest.cc File src/kudu/integration-tests/raft_consensus-itest.cc: http://gerrit.cloudera.org:8080/#/c/10266/5/src/kudu/integration-tests/raft_consensus-itest.cc@1803 PS5, Line 1803: // We also need to limit the block cache capacity in tandem. See the group > Again, isn't this outright disabling? Maybe, since the --force_block_cache_capacity is introduced, it would be possible to use it here and don't disable the block cache at all? http://gerrit.cloudera.org:8080/#/c/10266/4/src/kudu/util/process_memory.cc File src/kudu/util/process_memory.cc: http://gerrit.cloudera.org:8080/#/c/10266/4/src/kudu/util/process_memory.cc@a182 PS4, Line 182: : : : : : : : > It wasn't printed before this change AFAICT, and now the validator makes it Ah, I see -- that makes sense. -- To view, visit http://gerrit.cloudera.org:8080/10266 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia99fa95c578235beb3a1c72a33a22b4d8ff4800c Gerrit-Change-Number: 10266 Gerrit-PatchSet: 5 Gerrit-Owner: Will Berkeley Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Will Berkeley Gerrit-Comment-Date: Thu, 03 May 2018 20:23:51 + Gerrit-HasComments: Yes
[kudu-CR] KUDU-2214 Update logging to differentiate voting while copying/tombstoned
Mike Percy has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/10169 ) Change subject: KUDU-2214 Update logging to differentiate voting while copying/tombstoned .. KUDU-2214 Update logging to differentiate voting while copying/tombstoned Change-Id: I07007601d0a86d6161065629ba167121a33635d6 Reviewed-on: http://gerrit.cloudera.org:8080/10169 Tested-by: Kudu Jenkins Reviewed-by: Todd Lipcon Reviewed-by: Mike Percy --- M src/kudu/consensus/consensus-test-util.h M src/kudu/consensus/raft_consensus.cc M src/kudu/consensus/raft_consensus.h M src/kudu/consensus/raft_consensus_quorum-test.cc M src/kudu/tserver/tablet_service.cc 5 files changed, 53 insertions(+), 16 deletions(-) Approvals: Kudu Jenkins: Verified Todd Lipcon: Looks good to me, but someone else must approve Mike Percy: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/10169 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I07007601d0a86d6161065629ba167121a33635d6 Gerrit-Change-Number: 10169 Gerrit-PatchSet: 5 Gerrit-Owner: Fengling Wang Gerrit-Reviewer: Fengling Wang Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Mike Percy Gerrit-Reviewer: Tidy Bot Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Will Berkeley
[kudu-CR] KUDU-2214 Update logging to differentiate voting while copying/tombstoned
Mike Percy has posted comments on this change. ( http://gerrit.cloudera.org:8080/10169 ) Change subject: KUDU-2214 Update logging to differentiate voting while copying/tombstoned .. Patch Set 4: LGTM, I don't think it's critical to add an automated test to validate the log message change. -- To view, visit http://gerrit.cloudera.org:8080/10169 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I07007601d0a86d6161065629ba167121a33635d6 Gerrit-Change-Number: 10169 Gerrit-PatchSet: 4 Gerrit-Owner: Fengling Wang Gerrit-Reviewer: Fengling Wang Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Mike Percy Gerrit-Reviewer: Tidy Bot Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Will Berkeley Gerrit-Comment-Date: Thu, 03 May 2018 20:17:42 + Gerrit-HasComments: No
[kudu-CR] KUDU-2214 Update logging to differentiate voting while copying/tombstoned
Mike Percy has posted comments on this change. ( http://gerrit.cloudera.org:8080/10169 ) Change subject: KUDU-2214 Update logging to differentiate voting while copying/tombstoned .. Patch Set 4: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/10169 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I07007601d0a86d6161065629ba167121a33635d6 Gerrit-Change-Number: 10169 Gerrit-PatchSet: 4 Gerrit-Owner: Fengling Wang Gerrit-Reviewer: Fengling Wang Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Mike Percy Gerrit-Reviewer: Tidy Bot Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Will Berkeley Gerrit-Comment-Date: Thu, 03 May 2018 20:16:52 + Gerrit-HasComments: No
[kudu-CR] docs: workflow for backing up or restoring an entire node
Will Berkeley has posted comments on this change. ( http://gerrit.cloudera.org:8080/10223 ) Change subject: docs: workflow for backing up or restoring an entire node .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/10223 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I638f4b169282d6688dec414eabfe4c81d0a3f5df Gerrit-Change-Number: 10223 Gerrit-PatchSet: 2 Gerrit-Owner: Adar Dembo Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Will Berkeley Gerrit-Comment-Date: Thu, 03 May 2018 20:13:21 + Gerrit-HasComments: No
[kudu-CR] KUDU-2214 Update logging to differentiate voting while copying/tombstoned
Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/10169 ) Change subject: KUDU-2214 Update logging to differentiate voting while copying/tombstoned .. Patch Set 4: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/10169 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I07007601d0a86d6161065629ba167121a33635d6 Gerrit-Change-Number: 10169 Gerrit-PatchSet: 4 Gerrit-Owner: Fengling Wang Gerrit-Reviewer: Fengling Wang Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Mike Percy Gerrit-Reviewer: Tidy Bot Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Will Berkeley Gerrit-Comment-Date: Thu, 03 May 2018 19:26:06 + Gerrit-HasComments: No
[kudu-CR] mini-cluster: support parallel multi-master clusters
Hello Tidy Bot, Kudu Jenkins, Adar Dembo, Grant Henke, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/8280 to look at the new patch set (#9). Change subject: mini-cluster: support parallel multi-master clusters .. mini-cluster: support parallel multi-master clusters This commit refactors the mini-clusters to internally use reserved sockets for their child daemons. Reserved sockets are simply sockets bound to a random port with SO_REUSEPORT. As a result, master addresses for multi-master mini clusters no longer need to hard-coded master ports, and the associated ctest resource lock. This also significantly lessens the chances that port conflicts will occur, although it is still possible when masters are restarted during tests. To mitigate that on Linux, tests are still run in a unique subnet. Change-Id: I0b0ff7bfc179d8fdb1ed306d1bbd12acddeb060c --- M src/kudu/client/client-test.cc M src/kudu/integration-tests/CMakeLists.txt M src/kudu/integration-tests/catalog_manager_tsk-itest.cc M src/kudu/integration-tests/client-negotiation-failover-itest.cc M src/kudu/integration-tests/client-stress-test.cc M src/kudu/integration-tests/master-stress-test.cc M src/kudu/integration-tests/master_cert_authority-itest.cc M src/kudu/integration-tests/master_failover-itest.cc M src/kudu/integration-tests/master_migration-itest.cc M src/kudu/integration-tests/master_replication-itest.cc M src/kudu/integration-tests/security-faults-itest.cc M src/kudu/integration-tests/security-itest.cc M src/kudu/integration-tests/security-master-auth-itest.cc M src/kudu/integration-tests/token_signer-itest.cc M src/kudu/integration-tests/webserver-stress-itest.cc M src/kudu/master/mini_master.cc M src/kudu/mini-cluster/CMakeLists.txt M src/kudu/mini-cluster/external_mini_cluster-test.cc M src/kudu/mini-cluster/external_mini_cluster.cc M src/kudu/mini-cluster/external_mini_cluster.h M src/kudu/mini-cluster/internal_mini_cluster.cc M src/kudu/mini-cluster/internal_mini_cluster.h M src/kudu/mini-cluster/mini_cluster.cc M src/kudu/mini-cluster/mini_cluster.h M src/kudu/tools/CMakeLists.txt M src/kudu/tools/ksck_remote-test.cc M src/kudu/tools/tool_action_test.cc 27 files changed, 249 insertions(+), 326 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/80/8280/9 -- To view, visit http://gerrit.cloudera.org:8080/8280 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I0b0ff7bfc179d8fdb1ed306d1bbd12acddeb060c Gerrit-Change-Number: 8280 Gerrit-PatchSet: 9 Gerrit-Owner: Dan Burkert Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Dan Burkert Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Tidy Bot
[kudu-CR] mini-cluster: support parallel multi-master clusters
Dan Burkert has posted comments on this change. ( http://gerrit.cloudera.org:8080/8280 ) Change subject: mini-cluster: support parallel multi-master clusters .. Patch Set 9: (1 comment) http://gerrit.cloudera.org:8080/#/c/8280/8/src/kudu/integration-tests/registration-test.cc File src/kudu/integration-tests/registration-test.cc: PS8: > Could probably just revert the changes to this file. Done -- To view, visit http://gerrit.cloudera.org:8080/8280 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0b0ff7bfc179d8fdb1ed306d1bbd12acddeb060c Gerrit-Change-Number: 8280 Gerrit-PatchSet: 9 Gerrit-Owner: Dan Burkert Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Dan Burkert Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Tidy Bot Gerrit-Comment-Date: Thu, 03 May 2018 19:21:40 + Gerrit-HasComments: Yes
[kudu-CR] mini-cluster: support parallel multi-master clusters
Adar Dembo has posted comments on this change. ( http://gerrit.cloudera.org:8080/8280 ) Change subject: mini-cluster: support parallel multi-master clusters .. Patch Set 8: (1 comment) http://gerrit.cloudera.org:8080/#/c/8280/8/src/kudu/integration-tests/registration-test.cc File src/kudu/integration-tests/registration-test.cc: PS8: Could probably just revert the changes to this file. -- To view, visit http://gerrit.cloudera.org:8080/8280 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0b0ff7bfc179d8fdb1ed306d1bbd12acddeb060c Gerrit-Change-Number: 8280 Gerrit-PatchSet: 8 Gerrit-Owner: Dan Burkert Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Dan Burkert Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Tidy Bot Gerrit-Comment-Date: Thu, 03 May 2018 19:18:12 + Gerrit-HasComments: Yes
[kudu-CR] KUDU-2191 (10/n): PoC: disallow Kudu alter column, create table operations from Hive
Hello Kudu Jenkins, Hao Hao,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/10111
to look at the new patch set (#3).
Change subject: KUDU-2191 (10/n): PoC: disallow Kudu alter column, create table
operations from Hive
..
KUDU-2191 (10/n): PoC: disallow Kudu alter column, create table operations from
Hive
This commit makes it more difficult for processes or users who are not
the Kudu Master to alter the columns of Kudu table HMS entries through
Hive APIs.
The bulk of this patch is introducing a new environment context key/pair
('kudu.master_event=true') which is sent by the Kudu Master with every
HMS create/drop/alter table call. The KuduMetastorePlugin checks that
this property is set appropriately when operations modify a Kudu table
HMS entry.
The result is any attempt to create a table through Hive APIs or alter
the columns of a Kudu table through Hive APIs will fail. These
operations can not be supported because Hive lacks the DDL syntax and
HMS APIs to support necessary Kudu-specific table and column options.
It's possible for a user to spoof this environment context entry, but
it's a big enough roadblock that it should prevent accidental DDL
operations.
Change-Id: Idc39b1df07952d37e9ce93f1673aad535767a8ac
---
M
java/kudu-hive/src/main/java/org/apache/kudu/hive/metastore/KuduMetastorePlugin.java
M
java/kudu-hive/src/test/java/org/apache/kudu/hive/metastore/TestKuduMetastorePlugin.java
M src/kudu/hms/hms_catalog.cc
M src/kudu/hms/hms_catalog.h
M src/kudu/hms/hms_client-test.cc
M src/kudu/hms/hms_client.cc
M src/kudu/hms/hms_client.h
M src/kudu/integration-tests/master_hms-itest.cc
8 files changed, 125 insertions(+), 37 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/11/10111/3
--
To view, visit http://gerrit.cloudera.org:8080/10111
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Idc39b1df07952d37e9ce93f1673aad535767a8ac
Gerrit-Change-Number: 10111
Gerrit-PatchSet: 3
Gerrit-Owner: Dan Burkert
Gerrit-Reviewer: Hao Hao
Gerrit-Reviewer: Kudu Jenkins
[kudu-CR] mini-cluster: support parallel multi-master clusters
Dan Burkert has posted comments on this change. (
http://gerrit.cloudera.org:8080/8280 )
Change subject: mini-cluster: support parallel multi-master clusters
..
Patch Set 8:
(4 comments)
http://gerrit.cloudera.org:8080/#/c/8280/5/src/kudu/integration-tests/master_migration-itest.cc
File src/kudu/integration-tests/master_migration-itest.cc:
http://gerrit.cloudera.org:8080/#/c/8280/5/src/kudu/integration-tests/master_migration-itest.cc@89
PS5, Line 89: // Collect and keep alive the set of master sockets bound with
SO_REUSEPORT.
> Can you add some comments here explaining why we're doing this?
Done
http://gerrit.cloudera.org:8080/#/c/8280/7/src/kudu/integration-tests/master_migration-itest.cc
File src/kudu/integration-tests/master_migration-itest.cc:
http://gerrit.cloudera.org:8080/#/c/8280/7/src/kudu/integration-tests/master_migration-itest.cc@89
PS7, Line 89: SO_REUSEPOR
> SO_REUSEPORT
Done
http://gerrit.cloudera.org:8080/#/c/8280/7/src/kudu/integration-tests/registration-test.cc
File src/kudu/integration-tests/registration-test.cc:
http://gerrit.cloudera.org:8080/#/c/8280/7/src/kudu/integration-tests/registration-test.cc@272
PS7, Line 272: const int startup_rows_inserted =
GetCatalogMetric(METRIC_rows_inserted);
:
: // Add a table, make sure it reports itself.
> Hmm, the fact that you had to add this means the patch has changed some sem
Done
http://gerrit.cloudera.org:8080/#/c/8280/5/src/kudu/mini-cluster/internal_mini_cluster.cc
File src/kudu/mini-cluster/internal_mini_cluster.cc:
http://gerrit.cloudera.org:8080/#/c/8280/5/src/kudu/mini-cluster/internal_mini_cluster.cc@149
PS5, Line 149: if (num_masters > 1) {
: mini_master->SetMasterAddresses(master_rpc_addrs);
: }
> Not sure I understand the new condition; if num_masters was 0, won't this l
Good catch, this was supposed to be > 1. The Internal mini cluster already had
the equivalent.
--
To view, visit http://gerrit.cloudera.org:8080/8280
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0b0ff7bfc179d8fdb1ed306d1bbd12acddeb060c
Gerrit-Change-Number: 8280
Gerrit-PatchSet: 8
Gerrit-Owner: Dan Burkert
Gerrit-Reviewer: Adar Dembo
Gerrit-Reviewer: Dan Burkert
Gerrit-Reviewer: Grant Henke
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot
Gerrit-Comment-Date: Thu, 03 May 2018 19:13:30 +
Gerrit-HasComments: Yes
[kudu-CR] mini-cluster: support parallel multi-master clusters
Hello Tidy Bot, Kudu Jenkins, Adar Dembo, Grant Henke, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/8280 to look at the new patch set (#8). Change subject: mini-cluster: support parallel multi-master clusters .. mini-cluster: support parallel multi-master clusters This commit refactors the mini-clusters to internally use reserved sockets for their child daemons. Reserved sockets are simply sockets bound to a random port with SO_REUSEPORT. As a result, master addresses for multi-master mini clusters no longer need to hard-coded master ports, and the associated ctest resource lock. This also significantly lessens the chances that port conflicts will occur, although it is still possible when masters are restarted during tests. To mitigate that on Linux, tests are still run in a unique subnet. Change-Id: I0b0ff7bfc179d8fdb1ed306d1bbd12acddeb060c --- M src/kudu/client/client-test.cc M src/kudu/integration-tests/CMakeLists.txt M src/kudu/integration-tests/catalog_manager_tsk-itest.cc M src/kudu/integration-tests/client-negotiation-failover-itest.cc M src/kudu/integration-tests/client-stress-test.cc M src/kudu/integration-tests/master-stress-test.cc M src/kudu/integration-tests/master_cert_authority-itest.cc M src/kudu/integration-tests/master_failover-itest.cc M src/kudu/integration-tests/master_migration-itest.cc M src/kudu/integration-tests/master_replication-itest.cc M src/kudu/integration-tests/registration-test.cc M src/kudu/integration-tests/security-faults-itest.cc M src/kudu/integration-tests/security-itest.cc M src/kudu/integration-tests/security-master-auth-itest.cc M src/kudu/integration-tests/token_signer-itest.cc M src/kudu/integration-tests/webserver-stress-itest.cc M src/kudu/master/mini_master.cc M src/kudu/mini-cluster/CMakeLists.txt M src/kudu/mini-cluster/external_mini_cluster-test.cc M src/kudu/mini-cluster/external_mini_cluster.cc M src/kudu/mini-cluster/external_mini_cluster.h M src/kudu/mini-cluster/internal_mini_cluster.cc M src/kudu/mini-cluster/internal_mini_cluster.h M src/kudu/mini-cluster/mini_cluster.cc M src/kudu/mini-cluster/mini_cluster.h M src/kudu/tools/CMakeLists.txt M src/kudu/tools/ksck_remote-test.cc M src/kudu/tools/tool_action_test.cc 28 files changed, 251 insertions(+), 327 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/80/8280/8 -- To view, visit http://gerrit.cloudera.org:8080/8280 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I0b0ff7bfc179d8fdb1ed306d1bbd12acddeb060c Gerrit-Change-Number: 8280 Gerrit-PatchSet: 8 Gerrit-Owner: Dan Burkert Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Dan Burkert Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Tidy Bot
[kudu-CR] docs: workflow for backing up or restoring an entire node
Adar Dembo has posted comments on this change. ( http://gerrit.cloudera.org:8080/10223 ) Change subject: docs: workflow for backing up or restoring an entire node .. Patch Set 2: (4 comments) http://gerrit.cloudera.org:8080/#/c/10223/1/docs/administration.adoc File docs/administration.adoc: http://gerrit.cloudera.org:8080/#/c/10223/1/docs/administration.adoc@983 PS1, Line 983: tablet > nit: tablet server. Done http://gerrit.cloudera.org:8080/#/c/10223/1/docs/administration.adoc@993 PS1, Line 993: . Stop all Kudu processes in the cluster. This prevents the tablets on the > nit: I think it's wise to recommend stopping the whole cluster, but a reade Done http://gerrit.cloudera.org:8080/#/c/10223/1/docs/administration.adoc@995 PS1, Line 995: > nit: I think it's a mite clearer if you say "file-attribute-preserving". Done http://gerrit.cloudera.org:8080/#/c/10223/1/docs/administration.adoc@998 PS1, Line 998: e existing WAL, meta > Same nit. Done -- To view, visit http://gerrit.cloudera.org:8080/10223 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I638f4b169282d6688dec414eabfe4c81d0a3f5df Gerrit-Change-Number: 10223 Gerrit-PatchSet: 2 Gerrit-Owner: Adar Dembo Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Will Berkeley Gerrit-Comment-Date: Thu, 03 May 2018 18:58:43 + Gerrit-HasComments: Yes
[kudu-CR] docs: workflow for backing up or restoring an entire node
Hello Will Berkeley, Kudu Jenkins, Grant Henke, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/10223 to look at the new patch set (#2). Change subject: docs: workflow for backing up or restoring an entire node .. docs: workflow for backing up or restoring an entire node I explicitly chose to be vague and not provide example shell commands because this is such a niche workflow. Change-Id: I638f4b169282d6688dec414eabfe4c81d0a3f5df --- M docs/administration.adoc 1 file changed, 25 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/23/10223/2 -- To view, visit http://gerrit.cloudera.org:8080/10223 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I638f4b169282d6688dec414eabfe4c81d0a3f5df Gerrit-Change-Number: 10223 Gerrit-PatchSet: 2 Gerrit-Owner: Adar Dembo Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Will Berkeley
[kudu-CR] Add validation and docs on setting block cache capacity mb too high
Adar Dembo has posted comments on this change. ( http://gerrit.cloudera.org:8080/10266 ) Change subject: Add validation and docs on setting block_cache_capacity_mb too high .. Patch Set 5: (5 comments) http://gerrit.cloudera.org:8080/#/c/10266/5/src/kudu/cfile/block_cache.cc File src/kudu/cfile/block_cache.cc: http://gerrit.cloudera.org:8080/#/c/10266/5/src/kudu/cfile/block_cache.cc@41 PS5, Line 41: DEFINE_bool(force_block_cache_capacity_mb, false, Nit: the _mb suffix makes me think this is a megabyte quantity. http://gerrit.cloudera.org:8080/#/c/10266/5/src/kudu/cfile/block_cache.cc@77 PS5, Line 77: LOG(ERROR) << "Block cache capacity exceeds the memory pressure threshold " Nit: use strings::Substitute() here and below, makes the text a little more clear IMHO. http://gerrit.cloudera.org:8080/#/c/10266/5/src/kudu/cfile/block_cache.cc@94 PS5, Line 94: // Group validation is necessary since the validator depends on : // --memory_limit_hard_bytes and other memory flags. FWIW, I think this is implied by the content of the validator. http://gerrit.cloudera.org:8080/#/c/10266/5/src/kudu/integration-tests/client-stress-test.cc File src/kudu/integration-tests/client-stress-test.cc: http://gerrit.cloudera.org:8080/#/c/10266/5/src/kudu/integration-tests/client-stress-test.cc@241 PS5, Line 241: // The block cache capacity needs to be disabled. See the group validator Hmm, this just disables the block cache outright, correct? Reword to say that? Also, please add a few more words about _why_ it needs to be disabled. http://gerrit.cloudera.org:8080/#/c/10266/5/src/kudu/integration-tests/raft_consensus-itest.cc File src/kudu/integration-tests/raft_consensus-itest.cc: http://gerrit.cloudera.org:8080/#/c/10266/5/src/kudu/integration-tests/raft_consensus-itest.cc@1803 PS5, Line 1803: // We also need to limit the block cache capacity in tandem. See the group Again, isn't this outright disabling? -- To view, visit http://gerrit.cloudera.org:8080/10266 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia99fa95c578235beb3a1c72a33a22b4d8ff4800c Gerrit-Change-Number: 10266 Gerrit-PatchSet: 5 Gerrit-Owner: Will Berkeley Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Will Berkeley Gerrit-Comment-Date: Thu, 03 May 2018 18:49:58 + Gerrit-HasComments: Yes
[kudu-CR] mini-cluster: support parallel multi-master clusters
Adar Dembo has posted comments on this change. ( http://gerrit.cloudera.org:8080/8280 ) Change subject: mini-cluster: support parallel multi-master clusters .. Patch Set 7: (2 comments) http://gerrit.cloudera.org:8080/#/c/8280/7/src/kudu/integration-tests/master_migration-itest.cc File src/kudu/integration-tests/master_migration-itest.cc: http://gerrit.cloudera.org:8080/#/c/8280/7/src/kudu/integration-tests/master_migration-itest.cc@89 PS7, Line 89: SO_REUSPORT SO_REUSEPORT http://gerrit.cloudera.org:8080/#/c/8280/7/src/kudu/integration-tests/registration-test.cc File src/kudu/integration-tests/registration-test.cc: http://gerrit.cloudera.org:8080/#/c/8280/7/src/kudu/integration-tests/registration-test.cc@272 PS7, Line 272: ASSERT_OK(cluster_->mini_master() : ->master() : ->WaitUntilCatalogManagerIsLeaderAndReadyForTests(MonoDelta::FromSeconds(10))); Hmm, the fact that you had to add this means the patch has changed some semantics. My guess is that InternalMiniCluster::Start used to do this for single masters, and when you unified its single/master code paths, this got dropped. Could you restore it there? Alternatively, if you do want to make these semantics more consistent and clear, I would recommend taking this further. ExternalMiniCluster provides no guarantees about the state of the catalogs after starting a cluster, regardless of the number of masters. I think that's how InternalMiniCluster should behave too; any tests that need the catalog to be initialized, or for a leader master to be elected should wait for that explicitly. But I understand if you feel like that's out of scope for this patch, in which case my request is that you restore the old semantics as I wrote above. -- To view, visit http://gerrit.cloudera.org:8080/8280 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0b0ff7bfc179d8fdb1ed306d1bbd12acddeb060c Gerrit-Change-Number: 8280 Gerrit-PatchSet: 7 Gerrit-Owner: Dan Burkert Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Dan Burkert Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Tidy Bot Gerrit-Comment-Date: Thu, 03 May 2018 18:40:43 + Gerrit-HasComments: Yes
[kudu-CR] Add validation and docs on setting block cache capacity mb too high
Hello Alexey Serbin, Kudu Jenkins, Todd Lipcon, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/10266 to look at the new patch set (#5). Change subject: Add validation and docs on setting block_cache_capacity_mb too high .. Add validation and docs on setting block_cache_capacity_mb too high Raising --block_cache_capacity_mb can improve Kudu's performance, but raising it too high (above the memory pressure threshold, which is --memory_pressure_percentage of the hard memory limit) causes constant flushing even if write throughput is low. This patch adds validation of the capacity using a group flag validator, and does not permit Kudu to start if the block cache size can can cause Kudu to get stuck in aggressive flushing mode. It warns if the capacity is large enough (> 50%) of the pressure threshold. I chose 50% as the max recommended block cache size as a percentage of the memory pressure threshold somewhat arbitrarily. It seems safe, but also leaves room for allocating a large block cache. There's no automated tests but I did verify the expected messages appeared in the logs and the server fails to start when --block_cache_capacity_mb is too high. Change-Id: Ia99fa95c578235beb3a1c72a33a22b4d8ff4800c --- M docs/troubleshooting.adoc M src/kudu/cfile/block_cache.cc M src/kudu/integration-tests/client-stress-test.cc M src/kudu/integration-tests/raft_consensus-itest.cc M src/kudu/util/process_memory.cc M src/kudu/util/process_memory.h 6 files changed, 70 insertions(+), 10 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/66/10266/5 -- To view, visit http://gerrit.cloudera.org:8080/10266 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ia99fa95c578235beb3a1c72a33a22b4d8ff4800c Gerrit-Change-Number: 10266 Gerrit-PatchSet: 5 Gerrit-Owner: Will Berkeley Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Will Berkeley
[kudu-CR] Add validation and docs on setting block cache capacity mb too high
Will Berkeley has posted comments on this change. (
http://gerrit.cloudera.org:8080/10266 )
Change subject: Add validation and docs on setting block_cache_capacity_mb too
high
..
Patch Set 4:
(7 comments)
http://gerrit.cloudera.org:8080/#/c/10266/4/src/kudu/cfile/block_cache.cc
File src/kudu/cfile/block_cache.cc:
http://gerrit.cloudera.org:8080/#/c/10266/4/src/kudu/cfile/block_cache.cc@67
PS4, Line 67: int64
> int64_t ?
Done
http://gerrit.cloudera.org:8080/#/c/10266/4/src/kudu/cfile/block_cache.cc@72
PS4, Line 72:
> nit: extra space
Done
http://gerrit.cloudera.org:8080/#/c/10266/4/src/kudu/cfile/block_cache.cc@72
PS4, Line 72:
: return false;
> Do you think it's worth to provide an ability to override this and run with
Sure, as an unsafe flag, if you wanted to see the effects or were working on a
patch to mitigate them. Done.
http://gerrit.cloudera.org:8080/#/c/10266/4/src/kudu/cfile/block_cache.cc@77
PS4, Line 77:
> nit: extra space
Done
http://gerrit.cloudera.org:8080/#/c/10266/4/src/kudu/integration-tests/client-stress-test.cc
File src/kudu/integration-tests/client-stress-test.cc:
http://gerrit.cloudera.org:8080/#/c/10266/4/src/kudu/integration-tests/client-stress-test.cc@241
PS4, Line 241: // The block cache capacity needs to be disabled. See the
group validator
: // in cfile/block_cache.cc.
:
opts.extra_tserver_flags.push_back("--block_cache_capacity_mb=0");
> Does it induce some crucial changes in the behavior of the system while run
Don't think so. Since the soft limit percentage is 0, the servers are always
under memory pressure anyway.
http://gerrit.cloudera.org:8080/#/c/10266/4/src/kudu/integration-tests/raft_consensus-itest.cc
File src/kudu/integration-tests/raft_consensus-itest.cc:
http://gerrit.cloudera.org:8080/#/c/10266/4/src/kudu/integration-tests/raft_consensus-itest.cc@1810
PS4, Line 1810: "--block_cache_capacity_mb=0",
> Why not just move one "--block_cache_capacity_mb=0" out of the #ifdefs ?
Done
http://gerrit.cloudera.org:8080/#/c/10266/4/src/kudu/util/process_memory.cc
File src/kudu/util/process_memory.cc:
http://gerrit.cloudera.org:8080/#/c/10266/4/src/kudu/util/process_memory.cc@a182
PS4, Line 182:
:
:
:
:
:
:
:
> Wasn't it useful at all?
It wasn't printed before this change AFAICT, and now the validator makes it get
printed before logging is initialized, which is annoying. The info is available
from flag values (and arithmetic).
--
To view, visit http://gerrit.cloudera.org:8080/10266
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia99fa95c578235beb3a1c72a33a22b4d8ff4800c
Gerrit-Change-Number: 10266
Gerrit-PatchSet: 4
Gerrit-Owner: Will Berkeley
Gerrit-Reviewer: Alexey Serbin
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Todd Lipcon
Gerrit-Reviewer: Will Berkeley
Gerrit-Comment-Date: Thu, 03 May 2018 18:31:14 +
Gerrit-HasComments: Yes
[kudu-CR] KUDU-2191 (10/n): PoC: disallow Kudu alter column, create table operations from Hive
Hao Hao has posted comments on this change. ( http://gerrit.cloudera.org:8080/10111 ) Change subject: KUDU-2191 (10/n): PoC: disallow Kudu alter column, create table operations from Hive .. Patch Set 2: Code-Review+2 LGTM, but need to fix the IWYU build. -- To view, visit http://gerrit.cloudera.org:8080/10111 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idc39b1df07952d37e9ce93f1673aad535767a8ac Gerrit-Change-Number: 10111 Gerrit-PatchSet: 2 Gerrit-Owner: Dan Burkert Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Kudu Jenkins Gerrit-Comment-Date: Thu, 03 May 2018 18:22:14 + Gerrit-HasComments: No
[kudu-CR] docs: workflow for backing up or restoring an entire node
Will Berkeley has posted comments on this change. ( http://gerrit.cloudera.org:8080/10223 ) Change subject: docs: workflow for backing up or restoring an entire node .. Patch Set 1: (4 comments) http://gerrit.cloudera.org:8080/#/c/10223/1/docs/administration.adoc File docs/administration.adoc: http://gerrit.cloudera.org:8080/#/c/10223/1/docs/administration.adoc@983 PS1, Line 983: tserver nit: tablet server. http://gerrit.cloudera.org:8080/#/c/10223/1/docs/administration.adoc@993 PS1, Line 993: . Stop all Kudu processes in the entire cluster. nit: I think it's wise to recommend stopping the whole cluster, but a reader might think it sounds funny if they are backing up just one node. Would you consider a sentence of justification for why it's smart to stop the whole cluster? http://gerrit.cloudera.org:8080/#/c/10223/1/docs/administration.adoc@995 PS1, Line 995: attribute nit: I think it's a mite clearer if you say "file-attribute-preserving". http://gerrit.cloudera.org:8080/#/c/10223/1/docs/administration.adoc@998 PS1, Line 998: attribute-preserving Same nit. -- To view, visit http://gerrit.cloudera.org:8080/10223 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I638f4b169282d6688dec414eabfe4c81d0a3f5df Gerrit-Change-Number: 10223 Gerrit-PatchSet: 1 Gerrit-Owner: Adar Dembo Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Will Berkeley Gerrit-Comment-Date: Thu, 03 May 2018 18:20:47 + Gerrit-HasComments: Yes
[kudu-CR] Add validation and docs on setting block cache capacity mb too high
Alexey Serbin has posted comments on this change. (
http://gerrit.cloudera.org:8080/10266 )
Change subject: Add validation and docs on setting block_cache_capacity_mb too
high
..
Patch Set 4:
(7 comments)
http://gerrit.cloudera.org:8080/#/c/10266/4/src/kudu/cfile/block_cache.cc
File src/kudu/cfile/block_cache.cc:
http://gerrit.cloudera.org:8080/#/c/10266/4/src/kudu/cfile/block_cache.cc@67
PS4, Line 67: int64
int64_t ?
http://gerrit.cloudera.org:8080/#/c/10266/4/src/kudu/cfile/block_cache.cc@72
PS4, Line 72:
nit: extra space
http://gerrit.cloudera.org:8080/#/c/10266/4/src/kudu/cfile/block_cache.cc@72
PS4, Line 72:
: return false;
Do you think it's worth to provide an ability to override this and run with
those pernicious effects if a special --force_xxx was specified?
http://gerrit.cloudera.org:8080/#/c/10266/4/src/kudu/cfile/block_cache.cc@77
PS4, Line 77:
nit: extra space
http://gerrit.cloudera.org:8080/#/c/10266/4/src/kudu/integration-tests/client-stress-test.cc
File src/kudu/integration-tests/client-stress-test.cc:
http://gerrit.cloudera.org:8080/#/c/10266/4/src/kudu/integration-tests/client-stress-test.cc@241
PS4, Line 241: // The block cache capacity needs to be disabled. See the
group validator
: // in cfile/block_cache.cc.
:
opts.extra_tserver_flags.push_back("--block_cache_capacity_mb=0");
Does it induce some crucial changes in the behavior of the system while running
this test?
http://gerrit.cloudera.org:8080/#/c/10266/4/src/kudu/integration-tests/raft_consensus-itest.cc
File src/kudu/integration-tests/raft_consensus-itest.cc:
http://gerrit.cloudera.org:8080/#/c/10266/4/src/kudu/integration-tests/raft_consensus-itest.cc@1810
PS4, Line 1810: "--block_cache_capacity_mb=0",
Why not just move one "--block_cache_capacity_mb=0" out of the #ifdefs ?
http://gerrit.cloudera.org:8080/#/c/10266/4/src/kudu/util/process_memory.cc
File src/kudu/util/process_memory.cc:
http://gerrit.cloudera.org:8080/#/c/10266/4/src/kudu/util/process_memory.cc@a182
PS4, Line 182:
:
:
:
:
:
:
:
Wasn't it useful at all?
In other words, do we print that info from somewhere else?
--
To view, visit http://gerrit.cloudera.org:8080/10266
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia99fa95c578235beb3a1c72a33a22b4d8ff4800c
Gerrit-Change-Number: 10266
Gerrit-PatchSet: 4
Gerrit-Owner: Will Berkeley
Gerrit-Reviewer: Alexey Serbin
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Todd Lipcon
Gerrit-Reviewer: Will Berkeley
Gerrit-Comment-Date: Thu, 03 May 2018 18:20:36 +
Gerrit-HasComments: Yes
[kudu-CR] Add validation and docs on setting block cache capacity mb too high
Will Berkeley has posted comments on this change. ( http://gerrit.cloudera.org:8080/10266 ) Change subject: Add validation and docs on setting block_cache_capacity_mb too high .. Patch Set 4: Verified+1 TSAN failure is 2059. -- To view, visit http://gerrit.cloudera.org:8080/10266 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia99fa95c578235beb3a1c72a33a22b4d8ff4800c Gerrit-Change-Number: 10266 Gerrit-PatchSet: 4 Gerrit-Owner: Will Berkeley Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Will Berkeley Gerrit-Comment-Date: Thu, 03 May 2018 18:17:27 + Gerrit-HasComments: No
[kudu-CR] Add validation and docs on setting block cache capacity mb too high
Will Berkeley has removed a vote on this change. Change subject: Add validation and docs on setting block_cache_capacity_mb too high .. Removed Verified-1 by Kudu Jenkins (120) -- To view, visit http://gerrit.cloudera.org:8080/10266 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: deleteVote Gerrit-Change-Id: Ia99fa95c578235beb3a1c72a33a22b4d8ff4800c Gerrit-Change-Number: 10266 Gerrit-PatchSet: 4 Gerrit-Owner: Will Berkeley Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Will Berkeley
[kudu-CR] KUDU-2426 Fix WRONG SERVER UUID case in ksck
Todd Lipcon has posted comments on this change. (
http://gerrit.cloudera.org:8080/10293 )
Change subject: KUDU-2426 Fix WRONG_SERVER_UUID case in ksck
..
Patch Set 2:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/10293/2/src/kudu/tools/ksck_remote.cc
File src/kudu/tools/ksck_remote.cc:
http://gerrit.cloudera.org:8080/#/c/10293/2/src/kudu/tools/ksck_remote.cc@165
PS2, Line 165: return Status::UuidMismatch(Substitute("ID reported by
tablet server ($0) doesn't "
would rather introduce some new enum specific to ksck tablet server status
instead of a new general-purpose Status
--
To view, visit http://gerrit.cloudera.org:8080/10293
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2b4f50fe4dd94450b4f2e34dbad315bd761b071f
Gerrit-Change-Number: 10293
Gerrit-PatchSet: 2
Gerrit-Owner: Attila Bukor
Gerrit-Reviewer: Adar Dembo
Gerrit-Reviewer: Andrew Wong
Gerrit-Reviewer: Attila Bukor
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Todd Lipcon
Gerrit-Reviewer: Will Berkeley
Gerrit-Comment-Date: Thu, 03 May 2018 18:09:57 +
Gerrit-HasComments: Yes
[kudu-CR] KUDU-2426 Fix WRONG SERVER UUID case in ksck
Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/10293 ) Change subject: KUDU-2426 Fix WRONG_SERVER_UUID case in ksck .. Patch Set 2: Yea, I dont think adding a new Status is the right approach here from a compatibility perspective. -- To view, visit http://gerrit.cloudera.org:8080/10293 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2b4f50fe4dd94450b4f2e34dbad315bd761b071f Gerrit-Change-Number: 10293 Gerrit-PatchSet: 2 Gerrit-Owner: Attila Bukor Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Attila Bukor Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Will Berkeley Gerrit-Comment-Date: Thu, 03 May 2018 18:08:51 + Gerrit-HasComments: No
[kudu-CR] tool: improve format for dumping a rowset
Todd Lipcon has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/3946 ) Change subject: tool: improve format for dumping a rowset .. tool: improve format for dumping a rowset This changes the output for 'kudu local_replica dump rowset' to be more human-readable. The output now looks like this: RowIdxInBlock: 0; Base: (int32 key=0, int32 int_val=0, string string_val="HelloWorld"); Undo Mutations: [@1(DELETE)]; Redo Mutations: []; RowIdxInBlock: 1; Base: (int32 key=1, int32 int_val=10, string string_val="HelloWorld"); Undo Mutations: [@2(DELETE)]; Redo Mutations: []; RowIdxInBlock: 2; Base: (int32 key=2, int32 int_val=20, string string_val="HelloWorld"); Undo Mutations: [@3(DELETE)]; Redo Mutations: []; ... rather than separately dumping each column block. Dumping individual blocks is still possible by using the cfile dump commands. Change-Id: I0f1d08e08d2a3d20a87e49bb5338bf0585bd8e40 Reviewed-on: http://gerrit.cloudera.org:8080/3946 Tested-by: Todd Lipcon Reviewed-by: Alexey Serbin --- M src/kudu/tools/kudu-tool-test.cc M src/kudu/tools/tool_action_local_replica.cc 2 files changed, 23 insertions(+), 217 deletions(-) Approvals: Todd Lipcon: Verified Alexey Serbin: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/3946 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I0f1d08e08d2a3d20a87e49bb5338bf0585bd8e40 Gerrit-Change-Number: 3946 Gerrit-PatchSet: 5 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Dinesh Bhat Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Tidy Bot Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Will Berkeley
[kudu-CR] tool: improve format for dumping a rowset
Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/3946 ) Change subject: tool: improve format for dumping a rowset .. Patch Set 4: (1 comment) http://gerrit.cloudera.org:8080/#/c/3946/4/src/kudu/tools/tool_action_local_replica.cc File src/kudu/tools/tool_action_local_replica.cc: http://gerrit.cloudera.org:8080/#/c/3946/4/src/kudu/tools/tool_action_local_replica.cc@630 PS4, Line 630: scoped_refptr > nit: I saw std::unique_ptr is used in this file, so why not to use std::uni LogAnchorRegistry inherits from RefCountedThreadSafe so it has to be created in a scoped_refptr -- To view, visit http://gerrit.cloudera.org:8080/3946 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0f1d08e08d2a3d20a87e49bb5338bf0585bd8e40 Gerrit-Change-Number: 3946 Gerrit-PatchSet: 4 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Dinesh Bhat Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Tidy Bot Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Will Berkeley Gerrit-Comment-Date: Thu, 03 May 2018 18:05:22 + Gerrit-HasComments: Yes
[kudu-CR] mini-cluster: support parallel multi-master clusters
Hello Tidy Bot, Kudu Jenkins, Adar Dembo, Grant Henke, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/8280 to look at the new patch set (#7). Change subject: mini-cluster: support parallel multi-master clusters .. mini-cluster: support parallel multi-master clusters This commit refactors the mini-clusters to internally use reserved sockets for their child daemons. Reserved sockets are simply sockets bound to a random port with SO_REUSEPORT. As a result, master addresses for multi-master mini clusters no longer need to hard-coded master ports, and the associated ctest resource lock. This also significantly lessens the chances that port conflicts will occur, although it is still possible when masters are restarted during tests. To mitigate that on Linux, tests are still run in a unique subnet. Change-Id: I0b0ff7bfc179d8fdb1ed306d1bbd12acddeb060c --- M src/kudu/client/client-test.cc M src/kudu/integration-tests/CMakeLists.txt M src/kudu/integration-tests/catalog_manager_tsk-itest.cc M src/kudu/integration-tests/client-negotiation-failover-itest.cc M src/kudu/integration-tests/client-stress-test.cc M src/kudu/integration-tests/master-stress-test.cc M src/kudu/integration-tests/master_cert_authority-itest.cc M src/kudu/integration-tests/master_failover-itest.cc M src/kudu/integration-tests/master_migration-itest.cc M src/kudu/integration-tests/master_replication-itest.cc M src/kudu/integration-tests/registration-test.cc M src/kudu/integration-tests/security-faults-itest.cc M src/kudu/integration-tests/security-itest.cc M src/kudu/integration-tests/security-master-auth-itest.cc M src/kudu/integration-tests/token_signer-itest.cc M src/kudu/integration-tests/webserver-stress-itest.cc M src/kudu/master/mini_master.cc M src/kudu/mini-cluster/CMakeLists.txt M src/kudu/mini-cluster/external_mini_cluster-test.cc M src/kudu/mini-cluster/external_mini_cluster.cc M src/kudu/mini-cluster/external_mini_cluster.h M src/kudu/mini-cluster/internal_mini_cluster.cc M src/kudu/mini-cluster/internal_mini_cluster.h M src/kudu/mini-cluster/mini_cluster.cc M src/kudu/mini-cluster/mini_cluster.h M src/kudu/tools/CMakeLists.txt M src/kudu/tools/ksck_remote-test.cc M src/kudu/tools/tool_action_test.cc 28 files changed, 253 insertions(+), 328 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/80/8280/7 -- To view, visit http://gerrit.cloudera.org:8080/8280 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I0b0ff7bfc179d8fdb1ed306d1bbd12acddeb060c Gerrit-Change-Number: 8280 Gerrit-PatchSet: 7 Gerrit-Owner: Dan Burkert Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Dan Burkert Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Tidy Bot
[kudu-CR] mini-cluster: support parallel multi-master clusters
Hello Tidy Bot, Kudu Jenkins, Adar Dembo, Grant Henke, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/8280 to look at the new patch set (#6). Change subject: mini-cluster: support parallel multi-master clusters .. mini-cluster: support parallel multi-master clusters This commit refactors the mini-clusters to internally use reserved sockets for their child daemons. Reserved sockets are simply sockets bound to a random port with SO_REUSEPORT. As a result, master addresses for multi-master mini clusters no longer need to hard-coded master ports, and the associated ctest resource lock. This also significantly lessens the chances that port conflicts will occur, although it is still possible when masters are restarted during tests. To mitigate that on Linux, tests are still run in a unique subnet. Change-Id: I0b0ff7bfc179d8fdb1ed306d1bbd12acddeb060c --- M src/kudu/client/client-test.cc M src/kudu/integration-tests/CMakeLists.txt M src/kudu/integration-tests/catalog_manager_tsk-itest.cc M src/kudu/integration-tests/client-negotiation-failover-itest.cc M src/kudu/integration-tests/client-stress-test.cc M src/kudu/integration-tests/master-stress-test.cc M src/kudu/integration-tests/master_cert_authority-itest.cc M src/kudu/integration-tests/master_failover-itest.cc M src/kudu/integration-tests/master_migration-itest.cc M src/kudu/integration-tests/master_replication-itest.cc M src/kudu/integration-tests/security-faults-itest.cc M src/kudu/integration-tests/security-itest.cc M src/kudu/integration-tests/security-master-auth-itest.cc M src/kudu/integration-tests/token_signer-itest.cc M src/kudu/integration-tests/webserver-stress-itest.cc M src/kudu/master/mini_master.cc M src/kudu/mini-cluster/CMakeLists.txt M src/kudu/mini-cluster/external_mini_cluster-test.cc M src/kudu/mini-cluster/external_mini_cluster.cc M src/kudu/mini-cluster/external_mini_cluster.h M src/kudu/mini-cluster/internal_mini_cluster.cc M src/kudu/mini-cluster/internal_mini_cluster.h M src/kudu/mini-cluster/mini_cluster.cc M src/kudu/mini-cluster/mini_cluster.h M src/kudu/tools/CMakeLists.txt M src/kudu/tools/ksck_remote-test.cc M src/kudu/tools/tool_action_test.cc 27 files changed, 244 insertions(+), 326 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/80/8280/6 -- To view, visit http://gerrit.cloudera.org:8080/8280 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I0b0ff7bfc179d8fdb1ed306d1bbd12acddeb060c Gerrit-Change-Number: 8280 Gerrit-PatchSet: 6 Gerrit-Owner: Dan Burkert Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Dan Burkert Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Tidy Bot
[kudu-CR] rpc: add experimental rpc reuseport flag
Dan Burkert has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/8279 ) Change subject: rpc: add experimental rpc_reuseport flag .. rpc: add experimental rpc_reuseport flag This option sets the SO_REUSEPORT socket option on a server's bound socket. This socket option is supported on all platforms which Kudu supports, except RHEL 6.4 and below. The motivation is to use the option in the minicluster so that master ports can be reserved by the mini cluster control processes and reused by the master processes. When rpc_reuseport is set on RHEL 6.4 the server fails to startup with the following message (which is expected): $ cat /etc/redhat-release CentOS release 6.4 (Final) $ ./kudu-master --fs-wal-dir=/tmp/kudu-master --logtostderr --unlock-experimental-flags --rpc-reuseport=true ... F0501 14:56:30.672461 3271 master_main.cc:74] Check failed: _s.ok() Bad status: Network error: failed to set SO_REUSEPORT: Protocol not available (error 92) Change-Id: I5d8ce9faa646fa2be554f5cfdf8b6ed0c48b496e Reviewed-on: http://gerrit.cloudera.org:8080/8279 Reviewed-by: Alexey Serbin Tested-by: Kudu Jenkins --- M src/kudu/rpc/messenger.cc M src/kudu/rpc/messenger.h M src/kudu/rpc/rpc-test.cc M src/kudu/server/rpc_server.cc M src/kudu/server/rpc_server.h M src/kudu/server/server_base.cc M src/kudu/util/net/socket-test.cc M src/kudu/util/net/socket.cc M src/kudu/util/net/socket.h 9 files changed, 88 insertions(+), 70 deletions(-) Approvals: Alexey Serbin: Looks good to me, approved Kudu Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/8279 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I5d8ce9faa646fa2be554f5cfdf8b6ed0c48b496e Gerrit-Change-Number: 8279 Gerrit-PatchSet: 7 Gerrit-Owner: Dan Burkert Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Dan Burkert Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Tidy Bot
[kudu-CR] docs: workflow for backing up or restoring an entire node
Grant Henke has posted comments on this change. ( http://gerrit.cloudera.org:8080/10223 ) Change subject: docs: workflow for backing up or restoring an entire node .. Patch Set 1: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/10223 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I638f4b169282d6688dec414eabfe4c81d0a3f5df Gerrit-Change-Number: 10223 Gerrit-PatchSet: 1 Gerrit-Owner: Adar Dembo Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Will Berkeley Gerrit-Comment-Date: Thu, 03 May 2018 16:50:31 + Gerrit-HasComments: No
[kudu-CR] java: fix error-prone DefaultCharset[1] issues
Grant Henke has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/10197 ) Change subject: java: fix error-prone DefaultCharset[1] issues .. java: fix error-prone DefaultCharset[1] issues This should be a non-functional change on all systems where UTF8 is the default character set. It may fix issues on other systems which are exceedingly rare these days. [1] https://github.com/google/error-prone/blob/master/docs/bugpattern/DefaultCharset.md Change-Id: Ic92985b5d9466c8629475f3645c872914e3f73a1 Reviewed-on: http://gerrit.cloudera.org:8080/10197 Reviewed-by: Grant Henke Tested-by: Kudu Jenkins --- M java/kudu-client-tools/src/test/java/org/apache/kudu/mapreduce/tools/ITImportCsv.java M java/kudu-client/src/main/java/org/apache/kudu/client/AsyncKuduScanner.java M java/kudu-client/src/main/java/org/apache/kudu/client/ColumnRangePredicate.java M java/kudu-client/src/main/java/org/apache/kudu/client/ProtobufHelper.java M java/kudu-client/src/main/java/org/apache/kudu/client/RemoteTablet.java M java/kudu-client/src/test/java/org/apache/kudu/client/MiniKuduCluster.java M java/kudu-client/src/test/java/org/apache/kudu/client/TestAsyncKuduClient.java M java/kudu-client/src/test/java/org/apache/kudu/client/TestKeyEncoding.java M java/kudu-client/src/test/java/org/apache/kudu/client/TestKuduClient.java M java/kudu-client/src/test/java/org/apache/kudu/client/TestKuduPredicate.java M java/kudu-client/src/test/java/org/apache/kudu/client/TestRowResult.java M java/kudu-client/src/test/java/org/apache/kudu/client/TestTestUtils.java M java/kudu-flume-sink/src/test/java/org/apache/kudu/flume/sink/AvroKuduOperationsProducerTest.java M java/kudu-flume-sink/src/test/java/org/apache/kudu/flume/sink/KeyedKuduOperationsProducerTest.java M java/kudu-flume-sink/src/test/java/org/apache/kudu/flume/sink/KuduSinkTest.java M java/kudu-flume-sink/src/test/java/org/apache/kudu/flume/sink/RegexpKuduOperationsProducerTest.java M java/kudu-mapreduce/src/test/java/org/apache/kudu/mapreduce/ITOutputFormatJob.java M java/kudu-mapreduce/src/test/java/org/apache/kudu/mapreduce/TestJarFinder.java 18 files changed, 68 insertions(+), 47 deletions(-) Approvals: Grant Henke: Looks good to me, approved Kudu Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/10197 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ic92985b5d9466c8629475f3645c872914e3f73a1 Gerrit-Change-Number: 10197 Gerrit-PatchSet: 5 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Todd Lipcon
[kudu-CR] KUDU-2426 Fix WRONG SERVER UUID case in ksck
Attila Bukor has posted comments on this change. ( http://gerrit.cloudera.org:8080/10293 ) Change subject: KUDU-2426 Fix WRONG_SERVER_UUID case in ksck .. Patch Set 2: I wasn't sure adding a new Status for this is the best idea from a forward-compatibility standpoint. For now the server doesn't respond with this status though, the Status is returned on the client side in ksck_remote.cc. Adar, could you chime in on the possible compatibility issue? -- To view, visit http://gerrit.cloudera.org:8080/10293 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2b4f50fe4dd94450b4f2e34dbad315bd761b071f Gerrit-Change-Number: 10293 Gerrit-PatchSet: 2 Gerrit-Owner: Attila Bukor Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Attila Bukor Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Will Berkeley Gerrit-Comment-Date: Thu, 03 May 2018 16:14:16 + Gerrit-HasComments: No
[kudu-CR] KUDU-2426 Fix WRONG SERVER UUID case in ksck
Hello Will Berkeley, Kudu Jenkins, Adar Dembo, Todd Lipcon, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/10293 to look at the new patch set (#2). Change subject: KUDU-2426 Fix WRONG_SERVER_UUID case in ksck .. KUDU-2426 Fix WRONG_SERVER_UUID case in ksck RemoteKsckTabletServer::FetchInfo() returned a Status::RemoteError since KUDU-2364 to indicate a UUID mismatch. Ksck::FetchInfoFromTabletServers() checked for this Status, and this resulted in any RemoteError showing as a WRONG_SERVER_UUID in the tablet server health list. This commit introduces Status::UuidMismatch() which is used for this purpose instead and can be used for similar UUID mismatch errors in the future. Change-Id: I2b4f50fe4dd94450b4f2e34dbad315bd761b071f --- M src/kudu/common/wire_protocol.cc M src/kudu/common/wire_protocol.proto M src/kudu/tools/ksck-test.cc M src/kudu/tools/ksck.cc M src/kudu/tools/ksck_remote-test.cc M src/kudu/tools/ksck_remote.cc M src/kudu/util/status.cc M src/kudu/util/status.h 8 files changed, 20 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/93/10293/2 -- To view, visit http://gerrit.cloudera.org:8080/10293 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I2b4f50fe4dd94450b4f2e34dbad315bd761b071f Gerrit-Change-Number: 10293 Gerrit-PatchSet: 2 Gerrit-Owner: Attila Bukor Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Attila Bukor Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Will Berkeley
[kudu-CR] [jepsen] make kudu-master-see-tservers more robust
Alexey Serbin has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/10290 ) Change subject: [jepsen] make kudu-master-see-tservers more robust .. [jepsen] make kudu-master-see-tservers more robust Updated kudu-master-see-tservers function to be less fragile. The prior implementation depended on the format of the output of the 'kudu cluster ksck' sub-command. The updated implementation uses 'kudu tserver list' and relies only on the number of output lines by the latter sub-command. Also, dropped redundant 'node' parameter of the kudu-cli function. Change-Id: I8b9ff0b7c145e5235a94de27bbedea1fe36f5e60 Reviewed-on: http://gerrit.cloudera.org:8080/10290 Reviewed-by: Will Berkeley Tested-by: Alexey Serbin --- M java/kudu-jepsen/src/main/clojure/jepsen/kudu/util.clj 1 file changed, 11 insertions(+), 9 deletions(-) Approvals: Will Berkeley: Looks good to me, approved Alexey Serbin: Verified -- To view, visit http://gerrit.cloudera.org:8080/10290 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I8b9ff0b7c145e5235a94de27bbedea1fe36f5e60 Gerrit-Change-Number: 10290 Gerrit-PatchSet: 3 Gerrit-Owner: Alexey Serbin Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Will Berkeley
[kudu-CR] [jepsen] make kudu-master-see-tservers more robust
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/10290 ) Change subject: [jepsen] make kudu-master-see-tservers more robust .. Patch Set 2: Verified+1 Unrelated dist-test failure: failed to download task files: WARNING 102 isolateserver(1484): Adding unknown file be1c45bc2f9 -- To view, visit http://gerrit.cloudera.org:8080/10290 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8b9ff0b7c145e5235a94de27bbedea1fe36f5e60 Gerrit-Change-Number: 10290 Gerrit-PatchSet: 2 Gerrit-Owner: Alexey Serbin Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Will Berkeley Gerrit-Comment-Date: Thu, 03 May 2018 16:11:47 + Gerrit-HasComments: No
[kudu-CR] [jepsen] make kudu-master-see-tservers more robust
Alexey Serbin has removed Kudu Jenkins from this change. ( http://gerrit.cloudera.org:8080/10290 ) Change subject: [jepsen] make kudu-master-see-tservers more robust .. Removed reviewer Kudu Jenkins with the following votes: * Verified-1 by Kudu Jenkins (120) -- To view, visit http://gerrit.cloudera.org:8080/10290 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: deleteReviewer Gerrit-Change-Id: I8b9ff0b7c145e5235a94de27bbedea1fe36f5e60 Gerrit-Change-Number: 10290 Gerrit-PatchSet: 2 Gerrit-Owner: Alexey Serbin Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Will Berkeley
[kudu-CR] KUDU-2426 Fix WRONG SERVER UUID case in ksck
Attila Bukor has uploaded this change for review. ( http://gerrit.cloudera.org:8080/10293 Change subject: KUDU-2426 Fix WRONG_SERVER_UUID case in ksck .. KUDU-2426 Fix WRONG_SERVER_UUID case in ksck RemoteKsckTabletServer::FetchInfo() returned a Status::RemoteError since KUDU-2364 to indicate a UUID mismatch. Ksck::FetcvhInfoFromTabletServers() checked for this Status, and this resulted in any RemoteError showing as a WRONG_SERVER_UUID in the tablet server health list. This commit introduces Status::UuidMismatch() which is used for this purpose instead and can be used for similar UUID mismatch errors in the future. Change-Id: I2b4f50fe4dd94450b4f2e34dbad315bd761b071f --- M src/kudu/common/wire_protocol.cc M src/kudu/common/wire_protocol.proto M src/kudu/tools/ksck-test.cc M src/kudu/tools/ksck.cc M src/kudu/tools/ksck_remote-test.cc M src/kudu/tools/ksck_remote.cc M src/kudu/util/status.cc M src/kudu/util/status.h 8 files changed, 20 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/93/10293/1 -- To view, visit http://gerrit.cloudera.org:8080/10293 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I2b4f50fe4dd94450b4f2e34dbad315bd761b071f Gerrit-Change-Number: 10293 Gerrit-PatchSet: 1 Gerrit-Owner: Attila Bukor
[kudu-CR] Add validation and docs on setting block cache capacity mb too high
Hello Alexey Serbin, Kudu Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/10266 to look at the new patch set (#4). Change subject: Add validation and docs on setting block_cache_capacity_mb too high .. Add validation and docs on setting block_cache_capacity_mb too high Raising --block_cache_capacity_mb can improve Kudu's performance, but raising it too high (above the memory pressure threshold, which is --memory_pressure_percentage of the hard memory limit) causes constant flushing even if write throughput is low. This patch adds validation of the capacity using a group flag validator, and does not permit Kudu to start if the block cache size can can cause Kudu to get stuck in aggressive flushing mode. It warns if the capacity is large enough (> 50%) of the pressure threshold. I chose 50% as the max recommended block cache size as a percentage of the memory pressure threshold somewhat arbitrarily. It seems safe, but also leaves room for allocating a large block cache. There's no automated tests but I did verify the expected messages appeared in the logs and the server fails to start when --block_cache_capacity_mb is too high. Change-Id: Ia99fa95c578235beb3a1c72a33a22b4d8ff4800c --- M docs/troubleshooting.adoc M src/kudu/cfile/block_cache.cc M src/kudu/integration-tests/client-stress-test.cc M src/kudu/integration-tests/raft_consensus-itest.cc M src/kudu/util/process_memory.cc M src/kudu/util/process_memory.h 6 files changed, 63 insertions(+), 10 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/66/10266/4 -- To view, visit http://gerrit.cloudera.org:8080/10266 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ia99fa95c578235beb3a1c72a33a22b4d8ff4800c Gerrit-Change-Number: 10266 Gerrit-PatchSet: 4 Gerrit-Owner: Will Berkeley Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Will Berkeley
[kudu-CR] [jepsen] make kudu-master-see-tservers more robust
Will Berkeley has posted comments on this change. ( http://gerrit.cloudera.org:8080/10290 ) Change subject: [jepsen] make kudu-master-see-tservers more robust .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/10290 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8b9ff0b7c145e5235a94de27bbedea1fe36f5e60 Gerrit-Change-Number: 10290 Gerrit-PatchSet: 2 Gerrit-Owner: Alexey Serbin Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Will Berkeley Gerrit-Comment-Date: Thu, 03 May 2018 15:24:00 + Gerrit-HasComments: No
[kudu-CR] [jepsen] make kudu-master-see-tservers more robust
Hello Will Berkeley, Kudu Jenkins, Hao Hao, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/10290 to look at the new patch set (#2). Change subject: [jepsen] make kudu-master-see-tservers more robust .. [jepsen] make kudu-master-see-tservers more robust Updated kudu-master-see-tservers function to be less fragile. The prior implementation depended on the format of the output of the 'kudu cluster ksck' sub-command. The updated implementation uses 'kudu tserver list' and relies only on the number of output lines by the latter sub-command. Also, dropped redundant 'node' parameter of the kudu-cli function. Change-Id: I8b9ff0b7c145e5235a94de27bbedea1fe36f5e60 --- M java/kudu-jepsen/src/main/clojure/jepsen/kudu/util.clj 1 file changed, 11 insertions(+), 9 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/90/10290/2 -- To view, visit http://gerrit.cloudera.org:8080/10290 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I8b9ff0b7c145e5235a94de27bbedea1fe36f5e60 Gerrit-Change-Number: 10290 Gerrit-PatchSet: 2 Gerrit-Owner: Alexey Serbin Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Will Berkeley
[kudu-CR] [jepsen] make kudu-master-see-tservers more robust
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/10290 ) Change subject: [jepsen] make kudu-master-see-tservers more robust .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/10290/1/java/kudu-jepsen/src/main/clojure/jepsen/kudu/util.clj File java/kudu-jepsen/src/main/clojure/jepsen/kudu/util.clj: http://gerrit.cloudera.org:8080/#/c/10290/1/java/kudu-jepsen/src/main/clojure/jepsen/kudu/util.clj@146 PS1, Line 146: is > While you're here, delete this extra 'is'. Done -- To view, visit http://gerrit.cloudera.org:8080/10290 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8b9ff0b7c145e5235a94de27bbedea1fe36f5e60 Gerrit-Change-Number: 10290 Gerrit-PatchSet: 1 Gerrit-Owner: Alexey Serbin Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Will Berkeley Gerrit-Comment-Date: Thu, 03 May 2018 15:21:12 + Gerrit-HasComments: Yes
