kudu git commit: [java] fixed kudu-jepsen build
Repository: kudu Updated Branches: refs/heads/master 31e3d4616 -> 89206ed91 [java] fixed kudu-jepsen build The code in client.clj and table.clj in the kudu-jepsen package were importing BaseKuduTest but didn't use it. After the recent changes, the Clojure build cannot import BaseKuduTest anymore. Since it was unused anyway, I decided to stop importing the useless stuff without digging in further. That fixed the build error. This is a follow-up to dc8ae79961f71b8bdc344781fc89d38d94152fc4. Change-Id: I2d88b55a5f0b3ef4bf683d1de1bddc090d351b56 Reviewed-on: http://gerrit.cloudera.org:8080/11622 Tested-by: Alexey Serbin Reviewed-by: Grant Henke Project: http://git-wip-us.apache.org/repos/asf/kudu/repo Commit: http://git-wip-us.apache.org/repos/asf/kudu/commit/89206ed9 Tree: http://git-wip-us.apache.org/repos/asf/kudu/tree/89206ed9 Diff: http://git-wip-us.apache.org/repos/asf/kudu/diff/89206ed9 Branch: refs/heads/master Commit: 89206ed9195bd37a405658856b3463611382810a Parents: 31e3d46 Author: Alexey Serbin Authored: Mon Oct 8 17:54:35 2018 -0700 Committer: Alexey Serbin Committed: Tue Oct 9 02:33:13 2018 + -- java/kudu-jepsen/src/main/clojure/jepsen/kudu/client.clj | 1 - java/kudu-jepsen/src/main/clojure/jepsen/kudu/table.clj | 1 - 2 files changed, 2 deletions(-) -- http://git-wip-us.apache.org/repos/asf/kudu/blob/89206ed9/java/kudu-jepsen/src/main/clojure/jepsen/kudu/client.clj -- diff --git a/java/kudu-jepsen/src/main/clojure/jepsen/kudu/client.clj b/java/kudu-jepsen/src/main/clojure/jepsen/kudu/client.clj index c50201c..f24804f 100644 --- a/java/kudu-jepsen/src/main/clojure/jepsen/kudu/client.clj +++ b/java/kudu-jepsen/src/main/clojure/jepsen/kudu/client.clj @@ -24,7 +24,6 @@ Type]) (:import [org.apache.kudu.client AbstractKuduScannerBuilder AsyncKuduScanner$ReadMode - BaseKuduTest CreateTableOptions KuduClient KuduClient$KuduClientBuilder http://git-wip-us.apache.org/repos/asf/kudu/blob/89206ed9/java/kudu-jepsen/src/main/clojure/jepsen/kudu/table.clj -- diff --git a/java/kudu-jepsen/src/main/clojure/jepsen/kudu/table.clj b/java/kudu-jepsen/src/main/clojure/jepsen/kudu/table.clj index 19c76a6..a58acb3 100644 --- a/java/kudu-jepsen/src/main/clojure/jepsen/kudu/table.clj +++ b/java/kudu-jepsen/src/main/clojure/jepsen/kudu/table.clj @@ -25,7 +25,6 @@ Type]) (:import [org.apache.kudu.client AbstractKuduScannerBuilder AsyncKuduScanner$ReadMode - BaseKuduTest CreateTableOptions KuduClient KuduClient$KuduClientBuilder
kudu git commit: KUDU-2601: correctly print newly created files
Repository: kudu
Updated Branches:
refs/heads/master 4de4347a3 -> 31e3d4616
KUDU-2601: correctly print newly created files
I verified this manually by removing the contents of a data root from a
a cluster and running the `kudu fs update_dirs` tool, checking that it
correctly output files that were created when no new directories are
created (whereas before, it would omit printing the new files).
Change-Id: I2ab56f2516a97d4a5aa07451c37a4c39dfa965a4
Reviewed-on: http://gerrit.cloudera.org:8080/11614
Reviewed-by: Alexey Serbin
Tested-by: Kudu Jenkins
Project: http://git-wip-us.apache.org/repos/asf/kudu/repo
Commit: http://git-wip-us.apache.org/repos/asf/kudu/commit/31e3d461
Tree: http://git-wip-us.apache.org/repos/asf/kudu/tree/31e3d461
Diff: http://git-wip-us.apache.org/repos/asf/kudu/diff/31e3d461
Branch: refs/heads/master
Commit: 31e3d4616042093d1a621e6de820ba76f1cc767a
Parents: 4de4347
Author: Andrew Wong
Authored: Mon Oct 8 08:36:50 2018 -0700
Committer: Andrew Wong
Committed: Tue Oct 9 01:24:23 2018 +
--
src/kudu/fs/fs_manager.cc | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/kudu/blob/31e3d461/src/kudu/fs/fs_manager.cc
--
diff --git a/src/kudu/fs/fs_manager.cc b/src/kudu/fs/fs_manager.cc
index 1255888..b3b2503 100644
--- a/src/kudu/fs/fs_manager.cc
+++ b/src/kudu/fs/fs_manager.cc
@@ -441,7 +441,7 @@ Status FsManager::Open(FsReport* report) {
LOG(INFO) << "New directories created while opening local filesystem: " <<
JoinStrings(created_dirs, ", ");
}
- if (!created_dirs.empty()) {
+ if (!created_files.empty()) {
LOG(INFO) << "New files created while opening local filesystem: " <<
JoinStrings(created_files, ", ");
}
[2/2] kudu git commit: KUDU-2598 skip test_scanner_to_pandas_index appropriately
KUDU-2598 skip test_scanner_to_pandas_index appropriately We should skip the test if either pandas is not available or if decimal is not available. Change-Id: Ie9e3f0c2f6884bd5388c9b918703e5b4b2e543d7 Reviewed-on: http://gerrit.cloudera.org:8080/11620 Reviewed-by: Hao Hao Tested-by: Kudu Jenkins Project: http://git-wip-us.apache.org/repos/asf/kudu/repo Commit: http://git-wip-us.apache.org/repos/asf/kudu/commit/4de4347a Tree: http://git-wip-us.apache.org/repos/asf/kudu/tree/4de4347a Diff: http://git-wip-us.apache.org/repos/asf/kudu/diff/4de4347a Branch: refs/heads/master Commit: 4de4347a3fd28a8bad3d481fdfa90acecd584520 Parents: 0d4740b Author: Andrew Wong Authored: Mon Oct 8 16:30:38 2018 -0700 Committer: Andrew Wong Committed: Tue Oct 9 00:35:30 2018 + -- python/kudu/tests/test_scanner.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) -- http://git-wip-us.apache.org/repos/asf/kudu/blob/4de4347a/python/kudu/tests/test_scanner.py -- diff --git a/python/kudu/tests/test_scanner.py b/python/kudu/tests/test_scanner.py index 3c7fe57..4bedf96 100644 --- a/python/kudu/tests/test_scanner.py +++ b/python/kudu/tests/test_scanner.py @@ -382,7 +382,7 @@ class TestScanner(TestScanBase): self.assertEqual(df.index.name, 'key') self.assertEqual(list(df.index), [1, 2]) -@pytest.mark.skipif((not(kudu.CLIENT_SUPPORTS_PANDAS) and +@pytest.mark.skipif((not(kudu.CLIENT_SUPPORTS_PANDAS) or (not(kudu.CLIENT_SUPPORTS_DECIMAL))), reason="Pandas and Decimal support required to run this test.") def test_scanner_to_pandas_index(self):
[1/2] kudu git commit: [tools] KUDU-2179: Have ksck not use a single snapshot for all tablets
Repository: kudu Updated Branches: refs/heads/master 8d6cfe10d -> 4de4347a3 [tools] KUDU-2179: Have ksck not use a single snapshot for all tablets ksck checksum scans allow the user to checksum with snapshot scans, so that a checksum can be done even as tablets are mutated. It also allows users to omit a snapshot timestamp. Previously, in this case, the snapshot timestamp would be retrieved from some healthy tablet server at the beginning of the checksum process, and used for every replica. This didn't work well for checksumming large tables, because eventually the snapshot timestamp fell before the ancient history mark, and subsequent checksums scans would not be accepted by the tablet servers. This changes how checksum scans work to address this problem: 1. A background process periodically updates timestamps from tablet servers. 2. The checksum process is reorganized so the replicas of one tablet are checksummed together. 3. When a tablet is about to be checksummed, and the checksum scan is a snapshot scan with no user-provided timestamp, the tablet is assigned an up-to-date timestamp from one of the tablet servers that hosts a replica. Every replica is then checksummed using this snapshot timestamp. 4. The original default timeout of 3600 seconds for a checksum scan is too low, but it didn't really matter because the default tablet history max age was 900 seconds. Now that checksum scans can continue for many hours, the default timeout is raised to 86400 seconds (1 day), and a new idle timeout is added. If a checksum process does not checksum an additional row for this idle timeout (default 10 minutes), it will idle time out. Note that there is a new scheduling problem given #2: each tablet server has a fixed number of slots for checksum scans, but every tablet server hosting a replica must have a slot available before any replica's checksum can start, so deciding in which order to checksum tablets and how to find which are available to schedule is important. Given that the bulk of the time in checksums is occupied waiting for tablet servers to read lots of data off disk, materialize it as rows, and checksum it, it's worth spending a lot of effort to make sure the cluster is fully utilized given the scan concurrency constraints. So, the tool uses a brute force approach and simply checks all tablets to see which can be checksummed, any time a replica checksum finishes and frees a slot. Tablets are considered in tablet id order. Since tablet ids are UUIDs, there should be no correlation between a tablet's id and how its replicas are distributed across tablet servers. There are several tests added: 1. For the KUDU-2179 fix itself. 2. For the idle timeout. 3. For when a checksum finds mismatches. Yes, we didn't have a test for this before. After adding this test I saw that the output is a little confusing since it reported the number of replicas with mismatches rather than the number of tablets, so I altered the output to fix that. 4. A couple of tests exercising situations when all tablet servers are unavailable and when all peers of a tablet are unavailable. I also checksummed a very large cluster with 500TB of data or so, across about 37000 replicas. The checksum scan completed successfully after more than 12 hours. Change-Id: Iff0905c2099e6f56ed1cb651611918acbaf75476 Reviewed-on: http://gerrit.cloudera.org:8080/11554 Tested-by: Kudu Jenkins Reviewed-by: Attila Bukor Reviewed-by: Andrew Wong Project: http://git-wip-us.apache.org/repos/asf/kudu/repo Commit: http://git-wip-us.apache.org/repos/asf/kudu/commit/0d4740b7 Tree: http://git-wip-us.apache.org/repos/asf/kudu/tree/0d4740b7 Diff: http://git-wip-us.apache.org/repos/asf/kudu/diff/0d4740b7 Branch: refs/heads/master Commit: 0d4740b7692c0cb5c39cca73276b395921bd3ae4 Parents: 8d6cfe1 Author: Will Berkeley Authored: Tue Sep 25 10:26:15 2018 -0700 Committer: Will Berkeley Committed: Mon Oct 8 22:23:04 2018 + -- src/kudu/tools/ksck-test.cc| 106 +- src/kudu/tools/ksck.h | 21 +- src/kudu/tools/ksck_checksum.cc| 548 src/kudu/tools/ksck_checksum.h | 253 ++- src/kudu/tools/ksck_remote-test.cc | 89 +- src/kudu/tools/ksck_remote.cc | 77 +++-- src/kudu/tools/ksck_remote.h | 39 ++- 7 files changed, 863 insertions(+), 270 deletions(-) -- http://git-wip-us.apache.org/repos/asf/kudu/blob/0d4740b7/src/kudu/tools/ksck-test.cc -- diff --git a/src/kudu/tools/ksck-test.cc b/src/kudu/tools/ksck-test.cc index a07554e..558eff1 100644 --- a/src/kudu/tools/ksck-test.cc +++ b/src/kudu/tools/ksck-test.cc @@ -18,6 +18,7 @@ #include "kudu/tools/ksck.h" #include +#include #include #include
kudu git commit: Bump Sentry and Hadoop versions
Repository: kudu
Updated Branches:
refs/heads/master 2d6f8ffa9 -> 8d6cfe10d
Bump Sentry and Hadoop versions
The motivation is to get access to SENTRY-2371, which introduces a new
Thrift interface that will be used by Kudu to retrieve user privileges.
SENTRY-2371 has not been released yet, so the new Sentry version is
based on the latest commit of the master branch. The new Sentry version
appears to have an incompatibility with Hadoop 2.8.2, so this also
includes a bump to the latest Hadoop 2.8.x version. I suspect the fix
between 2.8.2 and 2.8.5 was introduced in [1], but the commit message is
vague and I haven't dug any further. The incompatiblity manifests as an
exception during Sentry startup:
java.lang.NoSuchMethodError:
org.apache.hadoop.conf.Configuration.addResource(Ljava/net/URL;Z)V
at
org.apache.sentry.service.thrift.SentryService.loadConfig(SentryService.java:576)
at
org.apache.sentry.service.thrift.SentryService$CommandImpl.run(SentryService.java:600)
at org.apache.sentry.SentryMain.main(SentryMain.java:120)
[1]:
https://github.com/apache/hadoop/commit/7af9b8ad1e993ef791aa38740b6aabc4c233a30f
Change-Id: I8bcc4ff6fac0435b037b984f45da75bed6ff4be5
Reviewed-on: http://gerrit.cloudera.org:8080/11601
Reviewed-by: Hao Hao
Tested-by: Kudu Jenkins
Project: http://git-wip-us.apache.org/repos/asf/kudu/repo
Commit: http://git-wip-us.apache.org/repos/asf/kudu/commit/8d6cfe10
Tree: http://git-wip-us.apache.org/repos/asf/kudu/tree/8d6cfe10
Diff: http://git-wip-us.apache.org/repos/asf/kudu/diff/8d6cfe10
Branch: refs/heads/master
Commit: 8d6cfe10d47bdb422c6cc0e770afbd70d6ec7adf
Parents: 2d6f8ff
Author: Dan Burkert
Authored: Fri Oct 5 14:10:32 2018 -0700
Committer: Dan Burkert
Committed: Mon Oct 8 17:59:17 2018 +
--
src/kudu/sentry/sentry_common_service.thrift | 3 +-
src/kudu/sentry/sentry_policy_service.thrift | 100 --
thirdparty/vars.sh | 6 +-
3 files changed, 99 insertions(+), 10 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/kudu/blob/8d6cfe10/src/kudu/sentry/sentry_common_service.thrift
--
diff --git a/src/kudu/sentry/sentry_common_service.thrift
b/src/kudu/sentry/sentry_common_service.thrift
index e37f4f6..b8a730e 100644
--- a/src/kudu/sentry/sentry_common_service.thrift
+++ b/src/kudu/sentry/sentry_common_service.thrift
@@ -19,7 +19,7 @@
*/
# DO NOT MODIFY! Copied from
-#
https://raw.githubusercontent.com/apache/sentry/release-2.0.1/sentry-provider/sentry-provider-db/src/main/resources/sentry_common_service.thrift
+#
https://raw.githubusercontent.com/apache/sentry/2c9a927a9e87cba0e4c0f34fc0b55887c6636927/sentry-service/sentry-service-api/src/main/resources/sentry_common_service.thrift
#
# With edits:
# - Change cpp namespace to 'sentry' to match the Kudu codebase style.
@@ -47,4 +47,3 @@ struct TSentryResponseStatus {
2: required string message
3: optional string stack
}
-
http://git-wip-us.apache.org/repos/asf/kudu/blob/8d6cfe10/src/kudu/sentry/sentry_policy_service.thrift
--
diff --git a/src/kudu/sentry/sentry_policy_service.thrift
b/src/kudu/sentry/sentry_policy_service.thrift
index a11872b..d32e39b 100644
--- a/src/kudu/sentry/sentry_policy_service.thrift
+++ b/src/kudu/sentry/sentry_policy_service.thrift
@@ -19,7 +19,7 @@
*/
# DO NOT MODIFY! Copied from
-#
https://raw.githubusercontent.com/apache/sentry/release-2.0.1/sentry-provider/sentry-provider-db/src/main/resources/sentry_policy_service.thrift
+#
https://raw.githubusercontent.com/apache/sentry/2c9a927a9e87cba0e4c0f34fc0b55887c6636927/sentry-service/sentry-service-api/src/main/resources/sentry_policy_service.thrift
#
# With edits:
# - Change cpp namespace to 'sentry' to match the Kudu codebase style.
@@ -30,8 +30,8 @@
include "sentry_common_service.thrift"
-namespace java org.apache.sentry.provider.db.service.thrift
-namespace php sentry.provider.db.service.thrift
+namespace java org.apache.sentry.api.service.thrift
+namespace php sentry.api.service.thrift
namespace cpp sentry
enum TSentryGrantOption {
@@ -44,6 +44,12 @@ enum TSentryGrantOption {
UNSET = -1
}
+enum TSentryPrincipalType {
+ NONE = 0,
+ ROLE = 1,
+ USER = 2
+}
+
# Represents a Privilege in transport from the client to the server
struct TSentryPrivilege {
1: required string privilegeScope, # Valid values are SERVER, DATABASE, TABLE,
COLUMN, URI
@@ -190,9 +196,18 @@ struct TSentryAuthorizable {
struct TListSentryPrivilegesRequest {
1: required i32 protocol_version = sentry_common_service.TSENTRY_SERVICE_V2,
2: required string requestorUserName, # user on whose behalf the request is
issued
+
+# @Deprecated Use principalName instead to set role names or
[2/2] kudu git commit: [mac] Add support for XCode 10
[mac] Add support for XCode 10
Change-Id: I442a1cd4605858b0800fc7d7686817936efd155e
Reviewed-on: http://gerrit.cloudera.org:8080/11602
Reviewed-by: Alexey Serbin
Tested-by: Kudu Jenkins
Reviewed-by: Attila Bukor
Tested-by: Attila Bukor
Project: http://git-wip-us.apache.org/repos/asf/kudu/repo
Commit: http://git-wip-us.apache.org/repos/asf/kudu/commit/2d6f8ffa
Tree: http://git-wip-us.apache.org/repos/asf/kudu/tree/2d6f8ffa
Diff: http://git-wip-us.apache.org/repos/asf/kudu/diff/2d6f8ffa
Branch: refs/heads/master
Commit: 2d6f8ffa9d14aacb0096d9d62fe4f5d71741fb4f
Parents: 88229a5
Author: Will Berkeley
Authored: Fri Oct 5 16:20:07 2018 -0700
Committer: Will Berkeley
Committed: Mon Oct 8 15:46:28 2018 +
--
cmake_modules/CompilerInfo.cmake | 10 ++
1 file changed, 10 insertions(+)
--
http://git-wip-us.apache.org/repos/asf/kudu/blob/2d6f8ffa/cmake_modules/CompilerInfo.cmake
--
diff --git a/cmake_modules/CompilerInfo.cmake b/cmake_modules/CompilerInfo.cmake
index ff5f789..8422e99 100644
--- a/cmake_modules/CompilerInfo.cmake
+++ b/cmake_modules/CompilerInfo.cmake
@@ -55,6 +55,15 @@ elseif("${COMPILER_VERSION_FULL}" MATCHES ".*clang-9")
set(COMPILER_FAMILY "clang")
set(COMPILER_VERSION "4.0.0svn")
+# clang on macOS, XCode 10.
+#
+# TODO(wdberkeley): 6.0.0svn is also just a guess, for the same reason, and
+# generated from the same sources, particularly the comment
+# https://gist.github.com/yamaya/2924292#gistcomment-801888.
+elseif("${COMPILER_VERSION_FULL}" MATCHES ".*clang-10")
+ set(COMPILER_FAMILY "clang")
+ set(COMPILER_VERSION "6.0.0svn")
+
# gcc
elseif("${COMPILER_VERSION_FULL}" MATCHES ".*gcc version.*")
set(COMPILER_FAMILY "gcc")
@@ -63,6 +72,7 @@ elseif("${COMPILER_VERSION_FULL}" MATCHES ".*gcc version.*")
else()
message(FATAL_ERROR "Unknown compiler. Version
info:\n${COMPILER_VERSION_FULL}")
endif()
+
message("Selected compiler ${COMPILER_FAMILY} ${COMPILER_VERSION}")
# gcc (and some varieties of clang) mention the path prefix where system
headers
