[jira] [Assigned] (KUDU-2375) Can't parse message of type "kudu.master.SysTablesEntryPB" because it is missing required fields: schema.columns[5].type
[
https://issues.apache.org/jira/browse/KUDU-2375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Grant Henke reassigned KUDU-2375:
-
Assignee: (was: Grant Henke)
> Can't parse message of type "kudu.master.SysTablesEntryPB" because it is
> missing required fields: schema.columns[5].type
>
>
> Key: KUDU-2375
> URL: https://issues.apache.org/jira/browse/KUDU-2375
> Project: Kudu
> Issue Type: Bug
> Components: master
>Affects Versions: 1.7.0
>Reporter: Michael Brown
>Priority: Major
>
> When tables with decimals are added in 1.7.0, a downgrade from 1.7.0 to 1.6
> results in a dcheck when 1.6 starts and Kudu isn't usable in its downgraded
> version.
> {noformat}
> F0324 17:45:10.681808 105716 catalog_manager.cc:935] Loading table and tablet
> metadata into memory failed: Corruption: Failed while visiting tables in sys
> catalog: unable to parse metadata field for row
> 467d365fffbe4485a3249079c48f42a9: Error parsing msg: Can't parse message of
> type "kudu.master.SysTablesEntryPB" because it is missing required fields:
> schema.columns[5].type
> {noformat}
> {noformat}
> #0 0x003355e32625 in raise () from /lib64/libc.so.6
> #1 0x003355e33e05 in abort () from /lib64/libc.so.6
> #2 0x01cea129 in ?? ()
> #3 0x009268cd in google::LogMessage::Fail() ()
> #4 0x0092878d in google::LogMessage::SendToLog() ()
> #5 0x00926409 in google::LogMessage::Flush() ()
> #6 0x0092922f in google::LogMessageFatal::~LogMessageFatal() ()
> #7 0x008f05de in ?? ()
> #8 0x008f6039 in
> kudu::master::CatalogManager::PrepareForLeadershipTask() ()
> #9 0x01d297d7 in kudu::ThreadPool::DispatchThread() ()
> #10 0x01d20151 in kudu::Thread::SuperviseThread(void*) ()
> #11 0x003356207aa1 in start_thread () from /lib64/libpthread.so.0
> #12 0x003355ee893d in clone () from /lib64/libc.so.6
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (KUDU-3135) Add Client Metadata Tokens
[ https://issues.apache.org/jira/browse/KUDU-3135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Henke reassigned KUDU-3135: - Assignee: (was: Grant Henke) > Add Client Metadata Tokens > -- > > Key: KUDU-3135 > URL: https://issues.apache.org/jira/browse/KUDU-3135 > Project: Kudu > Issue Type: Improvement > Components: client >Affects Versions: 1.12.0 >Reporter: Grant Henke >Priority: Major > Labels: roadmap-candidate, scalability > > Currently when a distributed task is done using the Kudu client, the > driver/coordinator client needs to open the table to request its current > metadata and locations. Then it can distribute the work to tasks/executors on > remote nodes. In the case of reading data, often ScanTokens are used to > distribute the work, and in the case of writing data perhaps just the table > name is required. > The problem is that each parallel task then also needs to open the table to > request the metadata for the table. Using Spark as an example, this happens > when deserializing the scan tokens in KuduRDD > ([here|https://github.com/apache/kudu/blob/master/java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/KuduRDD.scala#L107-L108]) > or when writing rows using the KuduContext > ([here|https://github.com/apache/kudu/blob/master/java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/KuduContext.scala#L466]). > This results in a large burst of metadata requests to the leader Kudu master > all at once. Given the Kudu master is only a single server and requests can't > be served from the follower masters, this effectively limits the amount of > parallel tasks that can run in a large Kudu deployment. Even if the follower > masters could service the requests, that still limits scalability in very > large clusters given most deployments would only have 3-5 masters. > Adding a metadata token, similar to a scan token, would be a useful way to > allow the single driver to fetch all the metadata required for the parallel > tasks. The tokens can be serialized and then passed to each task in a similar > fashion to scan tokens. > Of course in a pessimistic case, something may change between generation of > the token and the start of the task. In that case a request would need to be > sent to get the updated metadata. However, that scenario should be rare and > likely would not result in all of the requests happening at the same time. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (KUDU-3245) Provide Client API to set verbose logging filtered by vmodule
[
https://issues.apache.org/jira/browse/KUDU-3245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Grant Henke reassigned KUDU-3245:
-
Assignee: (was: Grant Henke)
> Provide Client API to set verbose logging filtered by vmodule
> --
>
> Key: KUDU-3245
> URL: https://issues.apache.org/jira/browse/KUDU-3245
> Project: Kudu
> Issue Type: Improvement
> Components: client
>Reporter: Hao Hao
>Priority: Major
>
> Similar to
> [{{client::SetVerboseLogLevel}}|https://github.com/apache/kudu/blob/master/src/kudu/client/client.h#L164]
> API, it will be nice to add another API to allow enabling verbose logging
> filtered by module.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (KUDU-2619) Track Java test failures in the flaky test dashboard
[ https://issues.apache.org/jira/browse/KUDU-2619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Henke reassigned KUDU-2619: - Assignee: (was: Grant Henke) > Track Java test failures in the flaky test dashboard > > > Key: KUDU-2619 > URL: https://issues.apache.org/jira/browse/KUDU-2619 > Project: Kudu > Issue Type: Improvement > Components: java, test >Affects Versions: n/a >Reporter: Adar Dembo >Priority: Major > > Right now our flaky test tracking infrastructure only incorporates C++ tests > using GTest; we should extend it to include Java tests too. > I spent some time on this recently and I wanted to collect my notes in one > place. > For reference, here's how C++ test reporting works: > # The build-and-test.sh script rebuilds thirdparty dependencies, builds Kudu, > and invokes all test suites, optionally using dist-test. After all tests have > been run, it also collects any dist-test artifacts and logs so that all of > the test results are available in one place. > # The run-test.sh script is effectively the C++ "test runner". It is > responsible for running a test binary, retrying it if it fails, and calling > report-test.sh after each test run (success or failure). Importantly, > report-test.sh is invoked once per test binary (not individual test), and on > test success we don't wait for the script to finish, because we don't care as > much about collecting successes. > # The report-test.sh collects some basic information about the test > environment (such as the git hash used, whether ASAN or TSAN was enabled, > etc.), then uses curl to send the information to the test result server. > # The test result server will store the test run result in a database, and > will query that database to produce a dashboard. > There are several problems to solve if we're going to replicate this for Java: > # There's no equivalent to run-test.sh. The entry point for running the Java > test suite is Gradle, but in dist-test, the test invocation is actually done > via 'java org.junit.runner.JUnitCore'. Note that C++ test reporting is > currently also incompatible with dist-test, so the Java tests aren't unique > in that respect. > # It'd be some work to replace report-test.sh with a Java equivalent. > My thinking is that we should move test reporting from run-test.sh into > build-and-test.sh: > # It's a good separation of concerns. The test "runners" are responsible for > running and maybe retrying tests, while the test "aggregator" > (build-and-test.sh) is responsible for reporting. > # It's more performant. You can imagine building a test_result_server.py > endpoint for reporting en masse, which would cut down on the number of round > trips. That's especially important if we start reporting individual test > results (as opposed to test _suite_ results). > # It means the reporting logic need only be written once. > # It was always a bit unexpected to find reporting logic buried in > run-test.sh. I mean, it made sense for rapid prototyping but it never really > made that much sense to me. > So then the problem is ensuring that, after all tests have run, we have the > right JUnit XML and log files for every test that ran, including retries, > which is more tractable, and doable for dist-test environments too. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (KUDU-2283) Improve KuduPartialRow::ToString() decimal output
[ https://issues.apache.org/jira/browse/KUDU-2283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Henke reassigned KUDU-2283: - Assignee: (was: Grant Henke) > Improve KuduPartialRow::ToString() decimal output > - > > Key: KUDU-2283 > URL: https://issues.apache.org/jira/browse/KUDU-2283 > Project: Kudu > Issue Type: Improvement >Affects Versions: 1.7.0 >Reporter: Grant Henke >Priority: Minor > > Currently KuduPartialRow::ToString() uses "AppendDebugStringForValue" to > print decimal values. However we could use the ColumnTypeAttributes to better > "pretty print" the decimal values. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (KUDU-1261) Support nested data types
[ https://issues.apache.org/jira/browse/KUDU-1261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Henke reassigned KUDU-1261: - Assignee: (was: Grant Henke) > Support nested data types > - > > Key: KUDU-1261 > URL: https://issues.apache.org/jira/browse/KUDU-1261 > Project: Kudu > Issue Type: New Feature >Reporter: Jean-Daniel Cryans >Priority: Major > Labels: limitations, roadmap-candidate > > AKA complex data types. > This is a common ask. I'm creating this jira so that we can at least start > tracking how people want to use it. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (KUDU-2860) Sign docker images
[ https://issues.apache.org/jira/browse/KUDU-2860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Henke reassigned KUDU-2860: - Assignee: (was: Grant Henke) > Sign docker images > -- > > Key: KUDU-2860 > URL: https://issues.apache.org/jira/browse/KUDU-2860 > Project: Kudu > Issue Type: Improvement > Components: docker >Reporter: Grant Henke >Priority: Major > Labels: docker > > We should sign the Apache docker images following the instructions here: > [https://docs.docker.com/ee/dtr/user/manage-images/sign-images/] > > Ideally this would be handled by the build script. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (KUDU-2524) scalafmt incompatible with jdk8 older than u25
[
https://issues.apache.org/jira/browse/KUDU-2524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Grant Henke reassigned KUDU-2524:
-
Assignee: (was: Grant Henke)
> scalafmt incompatible with jdk8 older than u25
> --
>
> Key: KUDU-2524
> URL: https://issues.apache.org/jira/browse/KUDU-2524
> Project: Kudu
> Issue Type: Bug
> Components: java
>Affects Versions: 1.8.0
>Reporter: Adar Dembo
>Priority: Major
>
> We're seeing a fair number of Gradle build failures in scalafmt with the
> following output:
> {noformat}
> 1: Task failed with an exception.
> ---
> * What went wrong:
> Execution failed for task ':kudu-spark:scalafmt'.
> > Uninitialized object exists on backward branch 209
> Exception Details:
> Location:
>
> scala/collection/immutable/HashMap$HashTrieMap.split()Lscala/collection/immutable/Seq;
> @249: goto
> Reason:
> Error exists in the bytecode
> Bytecode:
> 000: 2ab6 005b 04a0 001e b200 b3b2 00b8 04bd
> 010: 0002 5903 2a53 c000 bab6 00be b600 c2c0
> 020: 00c4 b02a b600 31b8 003b 3c1b 04a4 015e
> 030: 1b05 6c3d 2a1b 056c 2ab6 0031 b700 c63e
> 040: 2ab6 0031 021d 787e 3604 2ab6 0031 0210
> 050: 201d 647c 7e36 05bb 0014 59b2 00b8 2ab6
> 060: 0033 c000 bab6 00ca b700 cd1c b600 d13a
> 070: 0619 06c6 001a 1906 b600 d5c0 0081 3a07
> 080: 1906 b600 d8c0 0081 3a08 a700 0dbb 00da
> 090: 5919 06b7 00dd bf19 073a 0919 083a 0abb
> 0a0: 0002 5915 0419 09bb 0014 59b2 00b8 1909
> 0b0: c000 bab6 00ca b700 cd03 b800 e33a 0e3a
> 0c0: 0d03 190d b900 e701 0019 0e3a 1136 1036
> 0d0: 0f15 0f15 109f 0027 150f 0460 1510 190d
> 0e0: 150f b900 ea02 00c0 0005 3a17 1911 1917
> 0f0: b800 ee3a 1136 1036 0fa7 ffd8 1911 b800
> 100: f2b7 0060 3a0b bb00 0259 1505 190a bb00
> 110: 1459 b200 b819 0ac0 00ba b600 cab7 00cd
> 120: 03b8 00e3 3a13 3a12 0319 12b9 00e7 0100
> 130: 1913 3a16 3615 3614 1514 1515 9f00 2715
> 140: 1404 6015 1519 1215 14b9 00ea 0200 c000
> 150: 053a 1819 1619 18b8 00f5 3a16 3615 3614
> 160: a7ff d819 16b8 00f2 b700 603a 0cb2 00fa
> 170: b200 b805 bd00 0259 0319 0b53 5904 190c
> 180: 53c0 00ba b600 beb6 00fd b02a b600 3303
> 190: 32b6 00ff b0
> Stackmap Table:
> same_frame(@35)
>
> full_frame(@141,{Object[#2],Integer,Integer,Integer,Integer,Integer,Object[#109]},{})
> append_frame(@151,Object[#129],Object[#129])
>
> full_frame(@209,{Object[#2],Integer,Integer,Integer,Integer,Integer,Object[#109],Object[#129],Object[#129],Object[#129],Object[#129],Top,Top,Object[#20],Object[#55],Integer,Integer,Object[#107]},{Uninitialized[#159],Uninitialized[#159],Integer,Object[#129]})
>
> full_frame(@252,{Object[#2],Integer,Integer,Integer,Integer,Integer,Object[#109],Object[#129],Object[#129],Object[#129],Object[#129],Top,Top,Object[#20],Object[#55],Integer,Integer,Object[#107]},{Uninitialized[#159],Uninitialized[#159],Integer,Object[#129]})
>
> full_frame(@312,{Object[#2],Integer,Integer,Integer,Integer,Integer,Object[#109],Object[#129],Object[#129],Object[#129],Object[#129],Object[#2],Top,Object[#20],Object[#55],Integer,Integer,Object[#107],Object[#20],Object[#55],Integer,Integer,Object[#107]},{Uninitialized[#262],Uninitialized[#262],Integer,Object[#129]})
>
> full_frame(@355,{Object[#2],Integer,Integer,Integer,Integer,Integer,Object[#109],Object[#129],Object[#129],Object[#129],Object[#129],Object[#2],Top,Object[#20],Object[#55],Integer,Integer,Object[#107],Object[#20],Object[#55],Integer,Integer,Object[#107]},{Uninitialized[#262],Uninitialized[#262],Integer,Object[#129]})
> full_frame(@395,{Object[#2],Integer},{}){noformat}
> This appears to be due to [this JDK
> issue|https://stackoverflow.com/questions/24061672/verifyerror-uninitialized-object-exists-on-backward-branch-jvm-spec-4-10-2-4],
> which was fixed in JDK 8u25.
> And sure enough, here's the JDK version for failing builds:
> {noformat}
> -- Found Java: /opt/toolchain/sun-jdk-64bit-1.8.0.05/bin/java (found suitable
> version "1.8.0.05", minimum required is "1.7")
> {noformat}
> And here it is for successful builds:
> {noformat}
> 19:06:12 -- Found Java: /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java (found
> suitable version "1.8.0.111", minimum required is "1.7")
> {noformat}
> We either need to blacklist JDK8 versions older than u25, or we need to
> condition the scalafmt step on the JDK version.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (KUDU-3287) Threads can linger for some time after calling close on the Java KuduClient
[ https://issues.apache.org/jira/browse/KUDU-3287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Henke reassigned KUDU-3287: - Assignee: (was: Grant Henke) > Threads can linger for some time after calling close on the Java KuduClient > --- > > Key: KUDU-3287 > URL: https://issues.apache.org/jira/browse/KUDU-3287 > Project: Kudu > Issue Type: Bug >Affects Versions: 1.12.0, 1.13.0, 1.14.0 >Reporter: Grant Henke >Priority: Major > > After the upgrade to Netty 4 in Kudu 1.12 the close/shutdown behavior of the > Java client changed where threads and resources could linger for some time > after the call to close() returned. This looks the be because > `bootstrap.config().group().shutdownGracefully` is called with the default of > 15s and returns asynchronously. Additionally, the default ExecutorService was > not shutdown on close. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (KUDU-3218) client_symbol-test fails on Centos 7 with devtoolset-8
[
https://issues.apache.org/jira/browse/KUDU-3218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Grant Henke reassigned KUDU-3218:
-
Assignee: (was: Grant Henke)
> client_symbol-test fails on Centos 7 with devtoolset-8
> --
>
> Key: KUDU-3218
> URL: https://issues.apache.org/jira/browse/KUDU-3218
> Project: Kudu
> Issue Type: Bug
> Components: client
>Affects Versions: 1.14.0
>Reporter: Grant Henke
>Priority: Major
>
> When running the client_symbol-test on Centos 7 with devtoolset-8 the test
> fails with the following bad symbols:
> {code:java}
> Found bad symbol 'operator delete[](void*, unsigned long)'
> Found bad symbol 'operator delete(void*, unsigned long)'
> Found bad symbol 'transaction clone for std::logic_error::what() const'
> Found bad symbol 'transaction clone for std::runtime_error::what() const'
> Found bad symbol 'transaction clone for std::logic_error::logic_error(char
> const*)'
> Found bad symbol 'transaction clone for
> std::logic_error::logic_error(std::__cxx11::basic_string std::char_traits, std::allocator > const&)'
> Found bad symbol 'transaction clone for std::logic_error::logic_error(char
> const*)'
> Found bad symbol 'transaction clone for
> std::logic_error::logic_error(std::__cxx11::basic_string std::char_traits, std::allocator > const&)'
> Found bad symbol 'transaction clone for std::logic_error::~logic_error()'
> Found bad symbol 'transaction clone for std::logic_error::~logic_error()'
> Found bad symbol 'transaction clone for std::logic_error::~logic_error()'
> Found bad symbol 'transaction clone for std::range_error::range_error(char
> const*)'
> Found bad symbol 'transaction clone for
> std::range_error::range_error(std::__cxx11::basic_string std::char_traits, std::allocator > const&)'
> Found bad symbol 'transaction clone for std::range_error::range_error(char
> const*)'
> Found bad symbol 'transaction clone for
> std::range_error::range_error(std::__cxx11::basic_string std::char_traits, std::allocator > const&)'
> Found bad symbol 'transaction clone for std::range_error::~range_error()'
> Found bad symbol 'transaction clone for std::range_error::~range_error()'
> Found bad symbol 'transaction clone for std::range_error::~range_error()'
> Found bad symbol 'transaction clone for std::domain_error::domain_error(char
> const*)'
> Found bad symbol 'transaction clone for
> std::domain_error::domain_error(std::__cxx11::basic_string std::char_traits, std::allocator > const&)'
> Found bad symbol 'transaction clone for std::domain_error::domain_error(char
> const*)'
> Found bad symbol 'transaction clone for
> std::domain_error::domain_error(std::__cxx11::basic_string std::char_traits, std::allocator > const&)'
> Found bad symbol 'transaction clone for std::domain_error::~domain_error()'
> Found bad symbol 'transaction clone for std::domain_error::~domain_error()'
> Found bad symbol 'transaction clone for std::domain_error::~domain_error()'
> Found bad symbol 'transaction clone for std::length_error::length_error(char
> const*)'
> Found bad symbol 'transaction clone for
> std::length_error::length_error(std::__cxx11::basic_string std::char_traits, std::allocator > const&)'
> Found bad symbol 'transaction clone for std::length_error::length_error(char
> const*)'
> Found bad symbol 'transaction clone for
> std::length_error::length_error(std::__cxx11::basic_string std::char_traits, std::allocator > const&)'
> Found bad symbol 'transaction clone for std::length_error::~length_error()'
> Found bad symbol 'transaction clone for std::length_error::~length_error()'
> Found bad symbol 'transaction clone for std::length_error::~length_error()'
> Found bad symbol 'transaction clone for std::out_of_range::out_of_range(char
> const*)'
> Found bad symbol 'transaction clone for
> std::out_of_range::out_of_range(std::__cxx11::basic_string std::char_traits, std::allocator > const&)'
> Found bad symbol 'transaction clone for std::out_of_range::out_of_range(char
> const*)'
> Found bad symbol 'transaction clone for
> std::out_of_range::out_of_range(std::__cxx11::basic_string std::char_traits, std::allocator > const&)'
> Found bad symbol 'transaction clone for std::out_of_range::~out_of_range()'
> Found bad symbol 'transaction clone for std::out_of_range::~out_of_range()'
> Found bad symbol 'transaction clone for std::out_of_range::~out_of_range()'
> Found bad symbol 'transaction clone for
> std::runtime_error::runtime_error(char const*)'
> Found bad symbol 'transaction clone for
> std::runtime_error::runtime_error(std::__cxx11::basic_string std::char_traits, std::allocator > const&)'
> Found bad symbol 'transaction clone for
> std::runtime_error::runtime_error(char const*)'
> Found bad symbol 'transaction clone for
>
[jira] [Assigned] (KUDU-3132) Support RPC compression
[ https://issues.apache.org/jira/browse/KUDU-3132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Henke reassigned KUDU-3132: - Assignee: (was: Grant Henke) > Support RPC compression > --- > > Key: KUDU-3132 > URL: https://issues.apache.org/jira/browse/KUDU-3132 > Project: Kudu > Issue Type: New Feature > Components: perf, rpc >Reporter: Grant Henke >Priority: Major > Labels: performance, roadmap-candidate > > I have seen more and more deployments of Kudu where the tablet servers are > not co-located with the compute resources such as Impala or Spark. In > deployments like this, there could be significant network savings by > compressing the RPC messages (especially those that write or scan data). > Adding simple LZ4 or Snappy compression support to the RPC messages when not > on a loopback/local connection should be a great improvement for network > bound applications. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (KUDU-2282) Support coercion of Decimal values
[ https://issues.apache.org/jira/browse/KUDU-2282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Henke reassigned KUDU-2282: - Assignee: (was: Grant Henke) > Support coercion of Decimal values > --- > > Key: KUDU-2282 > URL: https://issues.apache.org/jira/browse/KUDU-2282 > Project: Kudu > Issue Type: Improvement >Affects Versions: 1.7.0 >Reporter: Grant Henke >Priority: Major > > Currently when decimal values are used in KuduValue.cc or PartialRow.cc we > enforce that the scale matches the expected scale. Instead we should support > basic coercion where no value rounding or truncating is required. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (KUDU-3134) Adjust default value for --raft_heartbeat_interval
[ https://issues.apache.org/jira/browse/KUDU-3134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Henke reassigned KUDU-3134: - Assignee: (was: Grant Henke) > Adjust default value for --raft_heartbeat_interval > -- > > Key: KUDU-3134 > URL: https://issues.apache.org/jira/browse/KUDU-3134 > Project: Kudu > Issue Type: Improvement >Affects Versions: 1.12.0 >Reporter: Grant Henke >Priority: Major > > Users often increase the `--raft_heartbeat_interval` on larger clusters or on > clusters with high replica counts. This helps avoid the servers flooding each > other with heartbeat RPCs causing queue overflows and using too much idle > CPU. Users have adjusted the values from 1.5 seconds to as high as 10s and we > have never seen people complain about problems after doing so. > Anecdotally, I recently saw a cluster with 4k tablets per tablet server using > ~150% cpu usage while idle. By increasing the `--raft_heartbeat_interval` > from 500ms to 1500ms the cpu usage dropped to ~50%. > Generally speaking users often care about Kudu stability and scalability over > an extremely short MTTR. Additionally our default client RPC timeouts of 30s > also seem to indicate slightly longer failover/retry times are tolerable in > the default case. > We should consider adjusting the default value of `--raft_heartbeat_interval` > to a higher value to support larger and more efficient clusters by default. > Users who need a low MTTR can always adjust the value lower while also > adjusting other related timeouts. We may also want to consider adjusting the > default `--heartbeat_interval_ms` accordingly. > Note: Batching the RPCs like mentioned in KUDU-1973 or providing a server to > server proxy for heartbeating may be a way to solve the issues without > adjusting the default configuration. However, adjusting the configuration is > easy and has proven effective in production deployments. Additionally > adjusting the defaults along with a KUDU-1973 like approach could lead to > even lower idle resource usage. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (KUDU-2530) Add kudu pbc replace tool
[ https://issues.apache.org/jira/browse/KUDU-2530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Henke reassigned KUDU-2530: - Assignee: (was: Grant Henke) > Add kudu pbc replace tool > - > > Key: KUDU-2530 > URL: https://issues.apache.org/jira/browse/KUDU-2530 > Project: Kudu > Issue Type: Improvement > Components: CLI >Reporter: Grant Henke >Priority: Minor > > We currently have a _kudu pbc dump_ and a _kudu pbc edit_ tool. However, it > could be nice to edit the dumped file elsewhere and be able to load/replace > the dumped pbc with it. Adding _kudu pbc replace_ would make this easier. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (KUDU-2500) Kudu Spark InterfaceStability class not found
[
https://issues.apache.org/jira/browse/KUDU-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Grant Henke reassigned KUDU-2500:
-
Assignee: (was: Grant Henke)
> Kudu Spark InterfaceStability class not found
> -
>
> Key: KUDU-2500
> URL: https://issues.apache.org/jira/browse/KUDU-2500
> Project: Kudu
> Issue Type: Bug
> Components: spark
>Affects Versions: 1.7.0
>Reporter: Grant Henke
>Priority: Major
>
> We recently marked the Yetus annotation library as optional because the
> annotations are not used at runtime and therefore should not be needed. Here
> is a good summary of why the annotations are not required at runtime:
> https://stackoverflow.com/questions/3567413/why-doesnt-a-missing-annotation-cause-a-classnotfoundexception-at-runtime/3568041#3568041
> However, for some reason Spark is requiring the annotation when performing
> some reflection. See the sample stacktrace below:
> {code}
> Driver stacktrace:
> at
> org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1602)
> at
> org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1590)
> at
> org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1589)
> at
> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
> at
> org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1589)
> at
> org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:831)
> at
> org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:831)
> at scala.Option.foreach(Option.scala:257)
> at
> org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:831)
> at
> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1823)
> at
> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1772)
> at
> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1761)
> at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
> at
> org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:642)
> at org.apache.spark.SparkContext.runJob(SparkContext.scala:2034)
> at org.apache.spark.SparkContext.runJob(SparkContext.scala:2055)
> at org.apache.spark.SparkContext.runJob(SparkContext.scala:2074)
> at org.apache.spark.SparkContext.runJob(SparkContext.scala:2099)
> at
> org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1.apply(RDD.scala:929)
> at
> org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1.apply(RDD.scala:927)
> at
> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
> at
> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
> at org.apache.spark.rdd.RDD.withScope(RDD.scala:363)
> at org.apache.spark.rdd.RDD.foreachPartition(RDD.scala:927)
> at
> org.apache.spark.sql.Dataset$$anonfun$foreachPartition$1.apply$mcV$sp(Dataset.scala:2675)
> at
> org.apache.spark.sql.Dataset$$anonfun$foreachPartition$1.apply(Dataset.scala:2675)
> at
> org.apache.spark.sql.Dataset$$anonfun$foreachPartition$1.apply(Dataset.scala:2675)
> at
> org.apache.spark.sql.Dataset$$anonfun$withNewRDDExecutionId$1.apply(Dataset.scala:3239)
> at
> org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:77)
> at
> org.apache.spark.sql.Dataset.withNewRDDExecutionId(Dataset.scala:3235)
> at org.apache.spark.sql.Dataset.foreachPartition(Dataset.scala:2674)
> at
> org.apache.kudu.spark.kudu.KuduContext.writeRows(KuduContext.scala:276)
> at
> org.apache.kudu.spark.kudu.KuduContext.insertRows(KuduContext.scala:206)
> at
> org.apache.kudu.backup.KuduRestore$$anonfun$run$1.apply(KuduRestore.scala:65)
> at
> org.apache.kudu.backup.KuduRestore$$anonfun$run$1.apply(KuduRestore.scala:44)
> at scala.collection.immutable.List.foreach(List.scala:392)
> at org.apache.kudu.backup.KuduRestore$.run(KuduRestore.scala:44)
> at
> org.apache.kudu.backup.TestKuduBackup.backupAndRestore(TestKuduBackup.scala:310)
> at
> org.apache.kudu.backup.TestKuduBackup$$anonfun$2.apply$mcV$sp(TestKuduBackup.scala:83)
> at
> org.apache.kudu.backup.TestKuduBackup$$anonfun$2.apply(TestKuduBackup.scala:76)
> at
> org.apache.kudu.backup.TestKuduBackup$$anonfun$2.apply(TestKuduBackup.scala:76)
> at
[jira] [Assigned] (KUDU-982) nullable columns should support DEFAULT NULL
[ https://issues.apache.org/jira/browse/KUDU-982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Henke reassigned KUDU-982: Assignee: (was: Grant Henke) > nullable columns should support DEFAULT NULL > > > Key: KUDU-982 > URL: https://issues.apache.org/jira/browse/KUDU-982 > Project: Kudu > Issue Type: Improvement > Components: api, client, master >Affects Versions: Private Beta >Reporter: Todd Lipcon >Priority: Major > > I don't think we have APIs which work for setting the default to NULL in > Alter/Create. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (KUDU-3172) Enable hybrid clock and built-in NTP client in Docker by default
[ https://issues.apache.org/jira/browse/KUDU-3172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Henke reassigned KUDU-3172: - Assignee: (was: Grant Henke) > Enable hybrid clock and built-in NTP client in Docker by default > > > Key: KUDU-3172 > URL: https://issues.apache.org/jira/browse/KUDU-3172 > Project: Kudu > Issue Type: Improvement >Affects Versions: 1.12.0 >Reporter: Grant Henke >Priority: Minor > > Currently the docker entrypoint sets `--use_hybrid_clock=false` by default. > This can cause unusual issues when snapshot scans are needed. Now that the > built-in client is available we should switch to use that by default in the > docker image by setting `--time_source=auto`. > For the quickstart cluster we can use `--time_source=system_unsync` given we > expect all nodes will be on the same machine. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (KUDU-2788) Validate metadata across backup and restore jobs
[ https://issues.apache.org/jira/browse/KUDU-2788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Henke reassigned KUDU-2788: - Assignee: (was: Grant Henke) > Validate metadata across backup and restore jobs > > > Key: KUDU-2788 > URL: https://issues.apache.org/jira/browse/KUDU-2788 > Project: Kudu > Issue Type: Improvement >Affects Versions: 1.9.0 >Reporter: Grant Henke >Priority: Critical > Labels: backup > > Currently the backup and restore jobs assume the metadata hasn't changed or > has changed in a compatible way across runs. We should validate that this is > true when building the backup graph and handle as many metadata changes as > possible. > The metadata changes that can't be handled should be clearly documented and a > follow up Jira filed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (KUDU-3211) Add a cluster supported features request
[ https://issues.apache.org/jira/browse/KUDU-3211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Henke reassigned KUDU-3211: - Assignee: (was: Grant Henke) > Add a cluster supported features request > > > Key: KUDU-3211 > URL: https://issues.apache.org/jira/browse/KUDU-3211 > Project: Kudu > Issue Type: Improvement > Components: master, supportability >Affects Versions: 1.13.0 >Reporter: Grant Henke >Priority: Major > > Recently we have come across a few scenarios where it would be useful to make > decisions in client integrations (Backup/Restore, Spark, NiFi, Impala) based > on the supported features of the target Kudu cluster. This can especially > helpful when we want to use new features by default if available but using > the new feature requires client/integration logic changes. > Some recent examples: > - Push bloomfilter predicates only if supported > - Use insert ignore operations (vs session based ignore) only if supported > It is technically possible to be optimistic about the support of a feature > and try to handle errors in a clever way using the required feature > capabilities of the RPCs. However, that can be difficult to express and near > impossible if you want to make a decision for multiple requests or based on > what all tablet servers support instead of based on a single request to a > single tablet server. > Additionally now that we support rolling restart, we can't assume that > because a single master or tablet server supports a feature that all servers > in the cluster support the feature. > Some thoughts on the feature/implementation: > - This should be a master request in order to prevent needing to talk to all > the tablet servers. > - We could leverage server registration requests or heartbeats to aggregate > the current state on the leader master. > - We could represent these features as "cluster" level features and indicate > that some (union) or all (intersect) of the servers support a given feature. > - If this request/response is not available in a cluster the response would > indicate that feature support is unknown and the user can decide how to > proceed. > - If we want to support disabling features via runtime flags we will need to > ensure we update the master, maybe via heartbeat, with changed support for a > running server. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (KUDU-2696) libgmock is linked into the kudu cli binary
[
https://issues.apache.org/jira/browse/KUDU-2696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Grant Henke reassigned KUDU-2696:
-
Assignee: (was: Grant Henke)
> libgmock is linked into the kudu cli binary
> ---
>
> Key: KUDU-2696
> URL: https://issues.apache.org/jira/browse/KUDU-2696
> Project: Kudu
> Issue Type: Bug
> Components: build
>Affects Versions: 1.8.0
>Reporter: Mike Percy
>Priority: Minor
>
> libgmock is linked into the kudu cli binary, even though we consider it a
> test-only dependency. Possibly a configuration problem in our cmake files?
> {code:java}
> $ ldd build/dynclang/bin/kudu | grep mock
> libgmock.so =>
> /home/mpercy/src/kudu/thirdparty/installed/uninstrumented/lib/libgmock.so
> (0x7f01f1495000)
> {code}
> The gmock dependency does not appear in the server binaries, as expected.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (KUDU-2820) No JUnit XMLs when running Java dist-test makes for frustrating precommit experience
[ https://issues.apache.org/jira/browse/KUDU-2820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Henke reassigned KUDU-2820: - Assignee: (was: Grant Henke) > No JUnit XMLs when running Java dist-test makes for frustrating precommit > experience > > > Key: KUDU-2820 > URL: https://issues.apache.org/jira/browse/KUDU-2820 > Project: Kudu > Issue Type: Improvement > Components: test >Affects Versions: 1.10.0 >Reporter: Adar Dembo >Priority: Major > > When running Java tests in dist-test (as the precommit job does), JUnit XML > files aren't generated. That's because normally they're generated by Gradle, > but we don't run Gradle in the dist-test slaves; we run JUnit directly. > As a result, test failures don't propagate back to the Jenkins job, and you > have to click through a few links (console output --> link to dist-test job > --> filter failures only --> download the artifacts) to figure out what went > wrong. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (KUDU-2858) Update docker readme to be more user focused
[ https://issues.apache.org/jira/browse/KUDU-2858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Henke reassigned KUDU-2858: - Assignee: (was: Grant Henke) > Update docker readme to be more user focused > > > Key: KUDU-2858 > URL: https://issues.apache.org/jira/browse/KUDU-2858 > Project: Kudu > Issue Type: Improvement > Components: docker, documentation >Reporter: Grant Henke >Priority: Major > Labels: docker > > Now that the docker images are being published, we should update the readme > to focus less on building the images and more on using the already built > images. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (KUDU-3244) Build and publish kudu-binary via Gradle
[ https://issues.apache.org/jira/browse/KUDU-3244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Henke reassigned KUDU-3244: - Assignee: (was: Grant Henke) > Build and publish kudu-binary via Gradle > > > Key: KUDU-3244 > URL: https://issues.apache.org/jira/browse/KUDU-3244 > Project: Kudu > Issue Type: Improvement > Components: build, test >Affects Versions: 1.14.0 >Reporter: Grant Henke >Priority: Major > > Now that the kudu-binary jar only uses the `kudu` binary > ([here|https://gerrit.cloudera.org/#/c/12523/]), we should be able to > simplify the build and release process of that jar, and build that jar inside > the Gradle build. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (KUDU-3148) Add Java client metrics
[ https://issues.apache.org/jira/browse/KUDU-3148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Henke reassigned KUDU-3148: - Assignee: (was: Grant Henke) > Add Java client metrics > --- > > Key: KUDU-3148 > URL: https://issues.apache.org/jira/browse/KUDU-3148 > Project: Kudu > Issue Type: Improvement > Components: client >Affects Versions: 1.12.0 >Reporter: Grant Henke >Priority: Major > Labels: roadmap-candidate, supportability > > This Jira is to track adding complete metrics to the Java client. There are > many cases where applications using the client have issues that are difficult > to debug. The primary reason is that it's hard to reason about what the > application is doing with the Kudu client without inspecting the code, and > even then it can be easy to miss an issue in the code as well. > For example we have seen many cases where an application creates a Kudu > client, sends a few messages, and then closes the client in a loop. Creating > many clients over an over not only impacts performance/stability of the > application but can also put unwelcome load on the servers. If we had request > metrics with a client id tag periodically logged, then it would be easy to > grep the application logs for unique client ids and spot the issue and the > offending application. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (KUDU-1945) Support generation of surrogate primary keys (or tables with no PK)
[ https://issues.apache.org/jira/browse/KUDU-1945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Henke reassigned KUDU-1945: - Assignee: (was: Grant Henke) > Support generation of surrogate primary keys (or tables with no PK) > --- > > Key: KUDU-1945 > URL: https://issues.apache.org/jira/browse/KUDU-1945 > Project: Kudu > Issue Type: New Feature > Components: client, master, tablet >Reporter: Todd Lipcon >Priority: Major > Labels: roadmap-candidate > > Many use cases have data where there is no "natural" primary key. For > example, a web log use case mostly cares about partitioning and not about > precise sorting by timestamp, and timestamps themselves are not necessarily > unique. Rather than forcing users to come up with their own surrogate primary > keys, Kudu should support some kind of "auto_increment" equivalent which > generates primary keys on insertion. Alternatively, Kudu could support tables > which are partitioned but not internally sorted. > The advantages would be: > - Kudu can pick primary keys on insertion to guarantee that there is no > compaction required on the table (eg always assign a new key higher than any > existing key in the local tablet). This can improve write throughput > substantially, especially compared to naive PK generation schemes that a user > might pick such as UUID, which would generate a uniform random-insert > workload (worst case for performance) > - Make Kudu easier to use for such use cases (no extra client code necessary) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KUDU-3292) Show non-default flags on varz Web UI
Grant Henke created KUDU-3292: - Summary: Show non-default flags on varz Web UI Key: KUDU-3292 URL: https://issues.apache.org/jira/browse/KUDU-3292 Project: Kudu Issue Type: Improvement Components: ui Reporter: Grant Henke Currently each Kudu server has a /varz webpage (the Flags tab) showing all of the flags set on the server. It would be a nice usability change to include a seperate section showing only the non-default flags. This should be super straigtforward given we have the ability to get all the non-default flags via GetNonDefaultFlags or GetNonDefaultFlagsMap in flags.cc. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KUDU-3255) get-pip.py URL updates which breaks docker-build.py
[ https://issues.apache.org/jira/browse/KUDU-3255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Henke resolved KUDU-3255. --- Fix Version/s: 1.15.0 Resolution: Fixed > get-pip.py URL updates which breaks docker-build.py > --- > > Key: KUDU-3255 > URL: https://issues.apache.org/jira/browse/KUDU-3255 > Project: Kudu > Issue Type: Bug >Reporter: Redriver >Priority: Major > Fix For: 1.15.0 > > > The error message: > #17 177.5 + curl > [https://bootstrap.pypa.io/2.7/get-pip.py] > #17 177.5 % Total% Received % Xferd Average Speed TimeTime > Time Current > #17 177.5 Dload Upload Total Spent > Left Speed > #17 177.5 > 0 00 00 0 0 0 --:--:-- --:--:-- --:--:-- 0 > 100 936 100 9360 0 3620 0 --:--:-- --:--:-- --:--:-- 3627 > #17 177.8 > #17 177.8 Hi there! > #17 177.8 > #17 177.8 The URL you are using to fetch this script has changed, and this > one will no > #17 177.8 longer work. Please use get-pip.py from the following URL instead: > #17 177.8 > #17 177.8 > [https://bootstrap.pypa.io/pip/2.7/get-pip.py] > #17 177.8 > #17 177.8 Sorry if this change causes any inconvenience for you! > #17 177.8 > #17 177.8 We don't have a good mechanism to make more gradual changes here, > and this > #17 177.8 renaming is a part of an effort to make it easier to us to update > these > #17 177.8 scripts, when there's a pip release. It's also essential for > improving how we > #17 177.8 handle the `get-pip.py` scripts, when pip drops support for a > Python minor > #17 177.8 version. > #17 177.8 > #17 177.8 There are no more renames/URL changes planned, and we don't expect > that a need > #17 177.8 would arise to do this again in the near future. > #17 177.8 > #17 177.8 Thanks for understanding! > #17 177.8 > #17 177.8 - Pradyun, on behalf of the volunteers who maintain pip. > #17 177.8 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KUDU-3287) Threads can linger for some time after calling close on the Java KuduClient
Grant Henke created KUDU-3287: - Summary: Threads can linger for some time after calling close on the Java KuduClient Key: KUDU-3287 URL: https://issues.apache.org/jira/browse/KUDU-3287 Project: Kudu Issue Type: Bug Affects Versions: 1.14.0, 1.13.0, 1.12.0 Reporter: Grant Henke Assignee: Grant Henke After the upgrade to Netty 4 in Kudu 1.12 the close/shutdown behavior of the Java client changed where threads and resources could linger for some time after the call to close() returned. This looks the be because `bootstrap.config().group().shutdownGracefully` is called with the default of 15s and returns asynchronously. Additionally, the default ExecutorService was not shutdown on close. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KUDU-1440) API for scanning rows in order
[
https://issues.apache.org/jira/browse/KUDU-1440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Grant Henke updated KUDU-1440:
--
Labels: roadmap-candidate (was: )
> API for scanning rows in order
> --
>
> Key: KUDU-1440
> URL: https://issues.apache.org/jira/browse/KUDU-1440
> Project: Kudu
> Issue Type: New Feature
> Components: client, master, tablet
>Affects Versions: 0.8.0
> Environment: CentOS 7
>Reporter: Martin Weindel
>Priority: Major
> Labels: roadmap-candidate
> Attachments: CreateTableTimeSeriesBug.java
>
>
> I have following simple table with two columns:
> {code}
> time TIMESTAMP,
> value FLOAT
> {code}
> The time column is used as range partition key.
> If I have understand the architecture of Kudu correctly, the rows should then
> be returned in ascending order for the time column.
> This works as long as not more than about 60 rows are inserted.
> If the number of inserted rows is above 1 mio, the order is messed up
> globally. On a microlevel it is still correct 99.9% if you look on successive
> rows.
> My setup is single master / single tablet server on a linux server. The table
> is created, filled and read with the Kudu Java client version 0.8.0.
> See attached Java code to reproduce the problem.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (KUDU-3164) Support table comment in Hive Metastore Integration
[ https://issues.apache.org/jira/browse/KUDU-3164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Henke resolved KUDU-3164. --- Fix Version/s: 1.15.0 Resolution: Fixed > Support table comment in Hive Metastore Integration > --- > > Key: KUDU-3164 > URL: https://issues.apache.org/jira/browse/KUDU-3164 > Project: Kudu > Issue Type: New Feature >Reporter: Quanlong Huang >Assignee: Grant Henke >Priority: Critical > Fix For: 1.15.0 > > > When Hive Metastore Integration is enabled, tables are created by Kudu. > However, Impala can't pass the table comment since the CreateTable API > doesn't have arguments for table comments, which results in lossing the table > comments. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KUDU-3279) Limit the number of diagnostic files kept
[
https://issues.apache.org/jira/browse/KUDU-3279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Grant Henke resolved KUDU-3279.
---
Fix Version/s: 1.7.0
Resolution: Duplicate
> Limit the number of diagnostic files kept
> -
>
> Key: KUDU-3279
> URL: https://issues.apache.org/jira/browse/KUDU-3279
> Project: Kudu
> Issue Type: Improvement
> Components: log
>Reporter: Alexey Serbin
>Priority: Major
> Fix For: 1.7.0
>
>
> When enabled, {{kudu-master}} and {{kudu-tserver}} processes generate
> diagnostics files in the log directory along with INFO/WARNING/ERROR/FATAL
> log files. To prevent issues running out of logging space, it would be nice
> if a user could configure the maximum number of diagnostics files to keep.
> Older diagnostics files should be removed if there are too many of those
> stored already.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KUDU-3279) Limit the number of diagnostic files kept
[
https://issues.apache.org/jira/browse/KUDU-3279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17338151#comment-17338151
]
Grant Henke commented on KUDU-3279:
---
I looked into implementing this, but it turns out it's already been done in
Kudu 1.7.0+ via KUDU-2279 here:
https://github.com/apache/kudu/commit/3e49fe2f7ca9afa603da600872cf3649346a0c22
It looks like the diagnostic log is implemented using a `RollingLog`:
https://github.com/apache/kudu/blob/master/src/kudu/server/diagnostics_log.cc#L139
The RollingLog respects `--max_log_files`:
https://github.com/apache/kudu/blob/master/src/kudu/util/rolling_log.cc#L63
This functionality is ensure via RollingLogTest.TestFileCountLimit:
https://github.com/apache/kudu/blob/master/src/kudu/util/rolling_log-test.cc#L133
It looks like this cleanup in done when closing the log which is called in the
destructor and when rolling a new file due to `roll_threshold_bytes_`:
https://github.com/apache/kudu/blob/master/src/kudu/util/rolling_log.cc#L184
> Limit the number of diagnostic files kept
> -
>
> Key: KUDU-3279
> URL: https://issues.apache.org/jira/browse/KUDU-3279
> Project: Kudu
> Issue Type: Improvement
> Components: log
>Reporter: Alexey Serbin
>Priority: Major
>
> When enabled, {{kudu-master}} and {{kudu-tserver}} processes generate
> diagnostics files in the log directory along with INFO/WARNING/ERROR/FATAL
> log files. To prevent issues running out of logging space, it would be nice
> if a user could configure the maximum number of diagnostics files to keep.
> Older diagnostics files should be removed if there are too many of those
> stored already.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (KUDU-3279) Limit the number of diagnostic files kept
[
https://issues.apache.org/jira/browse/KUDU-3279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Grant Henke reassigned KUDU-3279:
-
Assignee: Grant Henke
> Limit the number of diagnostic files kept
> -
>
> Key: KUDU-3279
> URL: https://issues.apache.org/jira/browse/KUDU-3279
> Project: Kudu
> Issue Type: Improvement
> Components: log
>Reporter: Alexey Serbin
>Assignee: Grant Henke
>Priority: Major
> Fix For: 1.7.0
>
>
> When enabled, {{kudu-master}} and {{kudu-tserver}} processes generate
> diagnostics files in the log directory along with INFO/WARNING/ERROR/FATAL
> log files. To prevent issues running out of logging space, it would be nice
> if a user could configure the maximum number of diagnostics files to keep.
> Older diagnostics files should be removed if there are too many of those
> stored already.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (KUDU-2754) Keep a maximum number of old log files
[ https://issues.apache.org/jira/browse/KUDU-2754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Henke resolved KUDU-2754. --- Fix Version/s: 1.15.0 Resolution: Fixed > Keep a maximum number of old log files > -- > > Key: KUDU-2754 > URL: https://issues.apache.org/jira/browse/KUDU-2754 > Project: Kudu > Issue Type: Improvement >Reporter: Grant Henke >Assignee: Grant Henke >Priority: Major > Labels: beginner, trivial > Fix For: 1.15.0 > > > Kudu generates various different log files > (INFO,WARNING,ERROR,diagnostic,minidumps,etc). To prevent issues running out > of logging space, it would be nice if a user could configure the maximum > number of each log file type to keep. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (KUDU-2754) Keep a maximum number of old log files
[ https://issues.apache.org/jira/browse/KUDU-2754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Henke reassigned KUDU-2754: - Assignee: Grant Henke > Keep a maximum number of old log files > -- > > Key: KUDU-2754 > URL: https://issues.apache.org/jira/browse/KUDU-2754 > Project: Kudu > Issue Type: Improvement >Reporter: Grant Henke >Assignee: Grant Henke >Priority: Major > Labels: beginner, trivial > > Kudu generates various different log files > (INFO,WARNING,ERROR,diagnostic,minidumps,etc). To prevent issues running out > of logging space, it would be nice if a user could configure the maximum > number of each log file type to keep. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (KUDU-3245) Provide Client API to set verbose logging filtered by vmodule
[
https://issues.apache.org/jira/browse/KUDU-3245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Grant Henke reassigned KUDU-3245:
-
Assignee: Grant Henke
> Provide Client API to set verbose logging filtered by vmodule
> --
>
> Key: KUDU-3245
> URL: https://issues.apache.org/jira/browse/KUDU-3245
> Project: Kudu
> Issue Type: Improvement
> Components: client
>Reporter: Hao Hao
>Assignee: Grant Henke
>Priority: Major
>
> Similar to
> [{{client::SetVerboseLogLevel}}|https://github.com/apache/kudu/blob/master/src/kudu/client/client.h#L164]
> API, it will be nice to add another API to allow enabling verbose logging
> filtered by module.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (KUDU-3269) Add UUID of the server into the metrics output
[
https://issues.apache.org/jira/browse/KUDU-3269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Grant Henke updated KUDU-3269:
--
Labels: newbie++ (was: )
> Add UUID of the server into the metrics output
> --
>
> Key: KUDU-3269
> URL: https://issues.apache.org/jira/browse/KUDU-3269
> Project: Kudu
> Issue Type: Improvement
>Reporter: Alexey Serbin
>Priority: Minor
> Labels: newbie++
>
> The server's section in the metrics output looks like the following:
> {noformat}
> {
> "type": "server",
>
> "id": "kudu.tabletserver",
>
> "attributes": {},
>
> "metrics": [ ... ]
> }
> {noformat}
> It would be great to add the UUID of the server into the output along with
> {{"type"}}, node/host name might be helpful as well.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (KUDU-3270) Add metrics for DNS resolution times and DNS cache-related metrics
[ https://issues.apache.org/jira/browse/KUDU-3270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Henke updated KUDU-3270: -- Labels: roadmap-candidate (was: ) > Add metrics for DNS resolution times and DNS cache-related metrics > -- > > Key: KUDU-3270 > URL: https://issues.apache.org/jira/browse/KUDU-3270 > Project: Kudu > Issue Type: Improvement >Reporter: Alexey Serbin >Priority: Major > Labels: roadmap-candidate > > When constructing a proxy for remote peer to perform a RPC, DNS resolution is > performed. It would be great to add metrics to measure how much time the DNS > resolution takes. That information could be provided by : > * a histogram for DNS resolver times > * 'standard' cache metrics for the DNS resolver cache -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KUDU-3275) RaftConsensusITest.TestSlowFollower is Flaky
[
https://issues.apache.org/jira/browse/KUDU-3275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Grant Henke updated KUDU-3275:
--
Attachment: raft_consensus-itest.2.txt.gz
> RaftConsensusITest.TestSlowFollower is Flaky
>
>
> Key: KUDU-3275
> URL: https://issues.apache.org/jira/browse/KUDU-3275
> Project: Kudu
> Issue Type: Bug
>Affects Versions: 1.14.0
>Reporter: Grant Henke
>Priority: Major
> Attachments: raft_consensus-itest.2.txt.gz
>
>
> I have seen RaftConsensusITest.TestSlowFollower fail quite a few times with
> the following:
> {code:java}
> I0411 07:15:34.772778 3404 raft_consensus.cc:1200] T
> a82c506675e74d62a75e53acd05e86e8 P 8110dc17824943a68d3051e0df96ae5d [term 1
> FOLLOWER]: Deduplicated request from leader. Original:
> 1.18320->[1.18321-1.18746] Dedup: 1.18746->[]I0411 07:15:34.772778 3404
> raft_consensus.cc:1200] T a82c506675e74d62a75e53acd05e86e8 P
> 8110dc17824943a68d3051e0df96ae5d [term 1 FOLLOWER]: Deduplicated request from
> leader. Original: 1.18320->[1.18321-1.18746] Dedup: 1.18746->[]W0411
> 07:15:34.798368 3632 log.cc:502] T a82c506675e74d62a75e53acd05e86e8 P
> 8110dc17824943a68d3051e0df96ae5d: Injecting 1096ms of latency in
> SegmentAllocator::Sync()W0411 07:15:34.834003 3206 consensus_peers.cc:480] T
> a82c506675e74d62a75e53acd05e86e8 P 5698d4de566c4adbae64b6f234c0561d -> Peer
> 8110dc17824943a68d3051e0df96ae5d (127.28.159.66:42271): Couldn't send request
> to peer 8110dc17824943a68d3051e0df96ae5d. Status: Timed out: UpdateConsensus
> RPC to 127.28.159.66:42271 timed out after 0.050s (SENT). This is attempt 1:
> this message will repeat every 5th retry.W0411 07:15:35.333277 3366
> tablet_service.cc:2874] Error setting up scanner with request: Service
> unavailable: Timed out: could not wait for desired snapshot timestamp to be
> consistent: Tablet is lagging too much to be able to serve snapshot scan.
> Lagging by: 45366 ms, (max is 3 ms):: new_scan_request { tablet_id:
> "a82c506675e74d62a75e53acd05e86e8" projected_columns { name: "key" type:
> INT32 is_key: true is_nullable: false } projected_columns { name: "int_val"
> type: INT32 is_key: false is_nullable: false } projected_columns { name:
> "string_val" type: STRING is_key: false is_nullable: true } read_mode:
> READ_AT_SNAPSHOT propagated_timestamp: 6627841138429693952 cache_blocks: true
> order_mode: ORDERED row_format_flags: 0 authz_token { token_data:
> "\010\210\306\312\203\006\"3\n\005slave\022*\n
> 4b6d5a12476d4dfcb5a4ab8a6f1190cb\020\001\030\001 \001(\001" signature:
> """\036\312x4\323\300\350\177k\357\226a signing_key_seq_num: 0 } } call_seq_id: 0W0411 07:15:35.635779 3366
> tablet_service.cc:2874] Error setting up scanner with request: Service
> unavailable: Timed out: could not wait for desired snapshot timestamp to be
> consistent: Tablet is lagging too much to be able to serve snapshot scan.
> Lagging by: 45668 ms, (max is 3 ms):: new_scan_request { tablet_id:
> "a82c506675e74d62a75e53acd05e86e8" projected_columns { name: "key" type:
> INT32 is_key: true is_nullable: false } projected_columns { name: "int_val"
> type: INT32 is_key: false is_nullable: false } projected_columns { name:
> "string_val" type: STRING is_key: false is_nullable: true } read_mode:
> READ_AT_SNAPSHOT propagated_timestamp: 6627841138425311232 cache_blocks: true
> order_mode: ORDERED row_format_flags: 0 authz_token { token_data:
> "\010\210\306\312\203\006\"3\n\005slave\022*\n
> 4b6d5a12476d4dfcb5a4ab8a6f1190cb\020\001\030\001 \001(\001" signature:
> """\036\312x4\323\300\350\177k\357\226a signing_key_seq_num: 0 } } call_seq_id: 0W0411 07:15:35.636209 3626
> scanner-internal.cc:406] Time spent opening tablet: real 57.698s user 0.002s
> sys 0.000sF0411 07:15:35.636267 3626 test_workload.cc:255] Check failed:
> _s.ok() Bad status: Timed out: exceeded configured scan timeout of 60.000s:
> after 20 scan attempts: unable to retry before timeout: Remote error: Service
> unavailable: Timed out: could not wait for desired snapshot timestamp to be
> consistent: Timed out waiting for ts: P: 1618125277942117 usec, L: 0 to be
> safe (mode: NON-LEADER). Current safe time: P: 1618125277252906 usec, L: 0
> Physical time difference: 0.689s*** Check failure stack trace: ** Aborted
> at 1618125335 (unix time) try "date -d @1618125335" if you are using GNU date
> ***PC: @ 0x7fe2e72bafb7 gsignal*** SIGABRT (@0x3e8727d) received by
> PID 29309 (TID 0x7fe2d979a700) from PID 29309; stack trace: *** @
> 0x7fe2e97881f1 google::(anonymous namespace)::FailureSignalHandler() at ??:0
> @ 0x7fe2eb20e980 (unknown) at ??:0 @ 0x7fe2e72bafb7 gsignal at
> ??:0 @ 0x7fe2e72bc921 abort at ??:0 @ 0x7fe2e9778439
> google::logging_fail() at ??:0 @
[jira] [Created] (KUDU-3275) RaftConsensusITest.TestSlowFollower is Flaky
Grant Henke created KUDU-3275:
-
Summary: RaftConsensusITest.TestSlowFollower is Flaky
Key: KUDU-3275
URL: https://issues.apache.org/jira/browse/KUDU-3275
Project: Kudu
Issue Type: Bug
Affects Versions: 1.14.0
Reporter: Grant Henke
I have seen RaftConsensusITest.TestSlowFollower fail quite a few times with the
following:
{code:java}
I0411 07:15:34.772778 3404 raft_consensus.cc:1200] T
a82c506675e74d62a75e53acd05e86e8 P 8110dc17824943a68d3051e0df96ae5d [term 1
FOLLOWER]: Deduplicated request from leader. Original:
1.18320->[1.18321-1.18746] Dedup: 1.18746->[]I0411 07:15:34.772778 3404
raft_consensus.cc:1200] T a82c506675e74d62a75e53acd05e86e8 P
8110dc17824943a68d3051e0df96ae5d [term 1 FOLLOWER]: Deduplicated request from
leader. Original: 1.18320->[1.18321-1.18746] Dedup: 1.18746->[]W0411
07:15:34.798368 3632 log.cc:502] T a82c506675e74d62a75e53acd05e86e8 P
8110dc17824943a68d3051e0df96ae5d: Injecting 1096ms of latency in
SegmentAllocator::Sync()W0411 07:15:34.834003 3206 consensus_peers.cc:480] T
a82c506675e74d62a75e53acd05e86e8 P 5698d4de566c4adbae64b6f234c0561d -> Peer
8110dc17824943a68d3051e0df96ae5d (127.28.159.66:42271): Couldn't send request
to peer 8110dc17824943a68d3051e0df96ae5d. Status: Timed out: UpdateConsensus
RPC to 127.28.159.66:42271 timed out after 0.050s (SENT). This is attempt 1:
this message will repeat every 5th retry.W0411 07:15:35.333277 3366
tablet_service.cc:2874] Error setting up scanner with request: Service
unavailable: Timed out: could not wait for desired snapshot timestamp to be
consistent: Tablet is lagging too much to be able to serve snapshot scan.
Lagging by: 45366 ms, (max is 3 ms):: new_scan_request { tablet_id:
"a82c506675e74d62a75e53acd05e86e8" projected_columns { name: "key" type: INT32
is_key: true is_nullable: false } projected_columns { name: "int_val" type:
INT32 is_key: false is_nullable: false } projected_columns { name: "string_val"
type: STRING is_key: false is_nullable: true } read_mode: READ_AT_SNAPSHOT
propagated_timestamp: 6627841138429693952 cache_blocks: true order_mode:
ORDERED row_format_flags: 0 authz_token { token_data:
"\010\210\306\312\203\006\"3\n\005slave\022*\n
4b6d5a12476d4dfcb5a4ab8a6f1190cb\020\001\030\001 \001(\001" signature:
"""\036\312x4\323\300\350\177k\357\226a""\036\312x4\323\300\350\177k\357\226a
[jira] [Resolved] (KUDU-3248) Expose replica selection option that provides better affinity for remote scans
[ https://issues.apache.org/jira/browse/KUDU-3248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Henke resolved KUDU-3248. --- Fix Version/s: 1.15.0 Resolution: Fixed > Expose replica selection option that provides better affinity for remote scans > -- > > Key: KUDU-3248 > URL: https://issues.apache.org/jira/browse/KUDU-3248 > Project: Kudu > Issue Type: Improvement > Components: client >Reporter: David Rorke >Assignee: Grant Henke >Priority: Major > Fix For: 1.15.0 > > > Kudu clients like Impala that use the CLOSEST_REPLICA selection option can > suffer from bad tserver buffer cache warmup/locality during remote scans due > to lack of tserver affinity as described in IMPALA-10481. > The Kudu client should expose a replica selection option that provides better > affinity when doing remote scans. The comments in IMPALA-10481 suggest a > new LOCAL_OR_LEADER option that would select a local replica if one exists, > otherwise fallback to the leader. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KUDU-3142) Deprecate and remove kudu-mapreduce integration
[ https://issues.apache.org/jira/browse/KUDU-3142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Henke resolved KUDU-3142. --- Fix Version/s: 1.15.0 Resolution: Fixed > Deprecate and remove kudu-mapreduce integration > --- > > Key: KUDU-3142 > URL: https://issues.apache.org/jira/browse/KUDU-3142 > Project: Kudu > Issue Type: Task >Affects Versions: 1.12.0 >Reporter: Grant Henke >Assignee: Grant Henke >Priority: Major > Fix For: 1.15.0 > > > The kudu-mapreduce integration has not been improved in a couple of years > even though there are known outstanding issues and feature gaps. Additionally > similar functionality and capabilities now exist via Spark, Hive, Impala, and > NiFi integrations. We should document the deprecation of the kudu-mapreduce > integration to notify users and so that we can remove it if maintenance > becomes an issue. > Even if eventually removed, users can still use the jars published in older > versions as needed. > Additionally if a Hadoop InputFormat and OutputFormat is required, a much > more modern and maintained version is available in the Apache Hive project: > https://github.com/apache/hive/tree/master/kudu-handler/src/java/org/apache/hadoop/hive/kudu -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KUDU-3142) Deprecate and remove kudu-mapreduce integration
[ https://issues.apache.org/jira/browse/KUDU-3142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Henke updated KUDU-3142: -- Summary: Deprecate and remove kudu-mapreduce integration (was: Deprecate kudu-mapreduce integration) > Deprecate and remove kudu-mapreduce integration > --- > > Key: KUDU-3142 > URL: https://issues.apache.org/jira/browse/KUDU-3142 > Project: Kudu > Issue Type: Task >Affects Versions: 1.12.0 >Reporter: Grant Henke >Assignee: Grant Henke >Priority: Major > > The kudu-mapreduce integration has not been improved in a couple of years > even though there are known outstanding issues and feature gaps. Additionally > similar functionality and capabilities now exist via Spark, Hive, Impala, and > NiFi integrations. We should document the deprecation of the kudu-mapreduce > integration to notify users and so that we can remove it if maintenance > becomes an issue. > Even if eventually removed, users can still use the jars published in older > versions as needed. > Additionally if a Hadoop InputFormat and OutputFormat is required, a much > more modern and maintained version is available in the Apache Hive project: > https://github.com/apache/hive/tree/master/kudu-handler/src/java/org/apache/hadoop/hive/kudu -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (KUDU-2696) libgmock is linked into the kudu cli binary
[
https://issues.apache.org/jira/browse/KUDU-2696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Grant Henke reassigned KUDU-2696:
-
Assignee: Grant Henke
> libgmock is linked into the kudu cli binary
> ---
>
> Key: KUDU-2696
> URL: https://issues.apache.org/jira/browse/KUDU-2696
> Project: Kudu
> Issue Type: Bug
> Components: build
>Affects Versions: 1.8.0
>Reporter: Mike Percy
>Assignee: Grant Henke
>Priority: Minor
>
> libgmock is linked into the kudu cli binary, even though we consider it a
> test-only dependency. Possibly a configuration problem in our cmake files?
> {code:java}
> $ ldd build/dynclang/bin/kudu | grep mock
> libgmock.so =>
> /home/mpercy/src/kudu/thirdparty/installed/uninstrumented/lib/libgmock.so
> (0x7f01f1495000)
> {code}
> The gmock dependency does not appear in the server binaries, as expected.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (KUDU-3248) Expose replica selection option that provides better affinity for remote scans
[ https://issues.apache.org/jira/browse/KUDU-3248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Henke reassigned KUDU-3248: - Assignee: Grant Henke > Expose replica selection option that provides better affinity for remote scans > -- > > Key: KUDU-3248 > URL: https://issues.apache.org/jira/browse/KUDU-3248 > Project: Kudu > Issue Type: Improvement > Components: client >Reporter: David Rorke >Assignee: Grant Henke >Priority: Major > > Kudu clients like Impala that use the CLOSEST_REPLICA selection option can > suffer from bad tserver buffer cache warmup/locality during remote scans due > to lack of tserver affinity as described in IMPALA-10481. > The Kudu client should expose a replica selection option that provides better > affinity when doing remote scans. The comments in IMPALA-10481 suggest a > new LOCAL_OR_LEADER option that would select a local replica if one exists, > otherwise fallback to the leader. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KUDU-3247) UUID column type
[ https://issues.apache.org/jira/browse/KUDU-3247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Henke updated KUDU-3247: -- Labels: roadmap-candidate (was: ) > UUID column type > > > Key: KUDU-3247 > URL: https://issues.apache.org/jira/browse/KUDU-3247 > Project: Kudu > Issue Type: Improvement >Reporter: Grant Henke >Priority: Major > Labels: roadmap-candidate > > It could be useful to have a UUID column type in Kudu given often when users > can't find a natural key, they look to a UUID as an alternate option. The > problem with this is that the UUID value is often stored in a STRING or > BINARY column resulting a random ordered write workload which puts a lot of > strain on Kudu compaction for high volumes. > If Kudu has a native UUID type it can leverage the underlying structure of > the UUID to benefit from the mostly ordered properties (version 1 and the > proposed version 6 UUIDs have a date-time component). > The implementation could be a logical type built on top of BYTES or INT128. > The client should have methods to write the UUID type using encoded UUID > strings or UUID objects. > Here are some references/examples of a UUID type: > * https://docs.yugabyte.com/latest/api/ycql/type_uuid/ > * https://cloud.google.com/spanner/docs/schema-design#uuid_primary_key > * https://www.percona.com/blog/2014/12/19/store-uuid-optimized-way/ > * http://gh.peabody.io/uuidv6/ > * https://github.com/f4b6a3/uuid-creator#version-6-time-ordered-proposed -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KUDU-3247) UUID column type
Grant Henke created KUDU-3247: - Summary: UUID column type Key: KUDU-3247 URL: https://issues.apache.org/jira/browse/KUDU-3247 Project: Kudu Issue Type: Improvement Reporter: Grant Henke It could be useful to have a UUID column type in Kudu given often when users can't find a natural key, they look to a UUID as an alternate option. The problem with this is that the UUID value is often stored in a STRING or BINARY column resulting a random ordered write workload which puts a lot of strain on Kudu compaction for high volumes. If Kudu has a native UUID type it can leverage the underlying structure of the UUID to benefit from the mostly ordered properties (version 1 and the proposed version 6 UUIDs have a date-time component). The implementation could be a logical type built on top of BYTES or INT128. The client should have methods to write the UUID type using encoded UUID strings or UUID objects. Here are some references/examples of a UUID type: * https://docs.yugabyte.com/latest/api/ycql/type_uuid/ * https://cloud.google.com/spanner/docs/schema-design#uuid_primary_key * https://www.percona.com/blog/2014/12/19/store-uuid-optimized-way/ * http://gh.peabody.io/uuidv6/ * https://github.com/f4b6a3/uuid-creator#version-6-time-ordered-proposed -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (KUDU-3132) Support RPC compression
[ https://issues.apache.org/jira/browse/KUDU-3132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Henke reassigned KUDU-3132: - Assignee: Grant Henke > Support RPC compression > --- > > Key: KUDU-3132 > URL: https://issues.apache.org/jira/browse/KUDU-3132 > Project: Kudu > Issue Type: New Feature > Components: perf, rpc >Reporter: Grant Henke >Assignee: Grant Henke >Priority: Major > Labels: performance, roadmap-candidate > > I have seen more and more deployments of Kudu where the tablet servers are > not co-located with the compute resources such as Impala or Spark. In > deployments like this, there could be significant network savings by > compressing the RPC messages (especially those that write or scan data). > Adding simple LZ4 or Snappy compression support to the RPC messages when not > on a loopback/local connection should be a great improvement for network > bound applications. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KUDU-3243) Fix Cmake warnings | `find_package` result variables (e.g., `_FOUND`) to follow a certain pattern.
[
https://issues.apache.org/jira/browse/KUDU-3243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Grant Henke updated KUDU-3243:
--
Fix Version/s: (was: 1.15)
1.15.0
> Fix Cmake warnings | `find_package` result variables (e.g., `_FOUND`) to
> follow a certain pattern.
> -
>
> Key: KUDU-3243
> URL: https://issues.apache.org/jira/browse/KUDU-3243
> Project: Kudu
> Issue Type: Improvement
> Components: build
>Affects Versions: 1.14.0
>Reporter: Bankim Bhavsar
>Assignee: Bankim Bhavsar
>Priority: Major
> Fix For: 1.15.0
>
>
> {noformat}
> Added shared library dependency yaml:
> /home/bankim/src/kudu/thirdparty/installed/uninstrumented/lib/libyaml-cpp.so
> CMake Warning (dev) at
> /home/bankim/Desktop/clion-2018.3.3/bin/cmake/linux/share/cmake-3.17/Modules/FindPackageHandleStandardArgs.cmake:272
> (message):
> The package name passed to `find_package_handle_standard_args`
> (GUMBO_PARSER) does not match the name of the calling package
> (GumboParser). This can lead to problems in calling code that expects
> `find_package` result variables (e.g., `_FOUND`) to follow a certain
> pattern.
> Call Stack (most recent call first):
> cmake_modules/FindGumboParser.cmake:40 (find_package_handle_standard_args)
> CMakeLists.txt:1205 (find_package)
> This warning is for project developers. Use -Wno-dev to suppress it.
> Added shared library dependency gumbo-parser:
> /home/bankim/src/kudu/thirdparty/installed/uninstrumented/lib/libgumbo.so
> CMake Warning (dev) at
> /home/bankim/Desktop/clion-2018.3.3/bin/cmake/linux/share/cmake-3.17/Modules/FindPackageHandleStandardArgs.cmake:272
> (message):
> The package name passed to `find_package_handle_standard_args`
> (GUMBO_QUERY) does not match the name of the calling package (GumboQuery).
> This can lead to problems in calling code that expects `find_package`
> result variables (e.g., `_FOUND`) to follow a certain pattern.
> Call Stack (most recent call first):
> cmake_modules/FindGumboQuery.cmake:40 (find_package_handle_standard_args)
> CMakeLists.txt:1212 (find_package)
> This warning is for project developers. Use -Wno-dev to suppress it.
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (KUDU-3237) MaintenanceManagerTest.TestCompletedOpsHistory is flaky
[
https://issues.apache.org/jira/browse/KUDU-3237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Grant Henke updated KUDU-3237:
--
Fix Version/s: (was: 1.15)
1.15.0
> MaintenanceManagerTest.TestCompletedOpsHistory is flaky
> ---
>
> Key: KUDU-3237
> URL: https://issues.apache.org/jira/browse/KUDU-3237
> Project: Kudu
> Issue Type: Test
>Reporter: Hao Hao
>Assignee: Alexey Serbin
>Priority: Major
> Fix For: 1.15.0
>
>
> Came across test failure in MaintenanceManagerTest.TestCompletedOpsHistory as
> the following:
> {noformat}
> I0125 19:55:10.782884 24454 maintenance_manager.cc:594] P 12345: op5
> complete. Timing: real 0.000suser 0.000s sys 0.000s Metrics: {}
> /data/1/hao/Repositories/kudu/src/kudu/util/maintenance_manager-test.cc:525:
> Failure
> Expected: std::min(kHistorySize, i + 1)
> Which is: 6
> To be equal to: status_pb.completed_operations_size()
> Which is: 5
> I0125 19:55:10.783524 24420 test_util.cc:148]
> ---
> I0125 19:55:10.783561 24420 test_util.cc:149] Had fatal failures, leaving
> test files at
> /tmp/dist-test-task1ofSWE/test-tmp/maintenance_manager-test.0.MaintenanceManagerTest.TestCompletedOpsHistory.1611604508702756-24420
> {noformat}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (KUDU-3244) Build and publish kudu-binary via Gradle
Grant Henke created KUDU-3244: - Summary: Build and publish kudu-binary via Gradle Key: KUDU-3244 URL: https://issues.apache.org/jira/browse/KUDU-3244 Project: Kudu Issue Type: Improvement Components: build, test Affects Versions: 1.14.0 Reporter: Grant Henke Assignee: Grant Henke Now that the kudu-binary jar only uses the `kudu` binary ([here|https://gerrit.cloudera.org/#/c/12523/]), we should be able to simplify the build and release process of that jar, and build that jar inside the Gradle build. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KUDU-3207) Standardize RSA private key format
[
https://issues.apache.org/jira/browse/KUDU-3207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17281507#comment-17281507
]
Grant Henke commented on KUDU-3207:
---
I don't have a lot of background or context on the formats and encodings and
their purpose or benefits. I found this post to be super helpful for context
and sharing here in case anyone else could benefit from it:
https://stackoverflow.com/a/48960291
> Standardize RSA private key format
> --
>
> Key: KUDU-3207
> URL: https://issues.apache.org/jira/browse/KUDU-3207
> Project: Kudu
> Issue Type: Improvement
>Reporter: Attila Bukor
>Priority: Minor
> Labels: OpenSSL, security
>
> Currently, Kudu stores RSA private keys in PEM format using
> PEM_write_bio_RSAPrivateKey(), which doesn't specify the format in which the
> key is stored. It expects it to be PKCS #1 (BEGIN/END RSA PRIVATE KEY), but
> it seems there are some OpenSSL versions (CryptoComply) that use PKCS #8
> instead (BEGIN/END PRIVATE KEY). {{CryptoTest.RsaPrivateKeyInputOutputPEM}}
> fails due to this, as it compares the private key to an expected string,
> which is in PKCS #1 format. The read functions are explicitly said to handle
> any known format, so this shouldn't cause any issues, but it would still be
> nice to standardize on a single format (probably PKCS #8).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (KUDU-2490) implement Kudu DataSourceV2 and related classes
[ https://issues.apache.org/jira/browse/KUDU-2490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Henke reassigned KUDU-2490: - Assignee: Mayank Asthana (was: Grant Henke) > implement Kudu DataSourceV2 and related classes > --- > > Key: KUDU-2490 > URL: https://issues.apache.org/jira/browse/KUDU-2490 > Project: Kudu > Issue Type: Improvement > Components: spark >Reporter: Andrew Wong >Assignee: Mayank Asthana >Priority: Major > Labels: roadmap-candidate > > The current Kudu-Spark bindings implement a DefaultSource that extends a > RelationProvider, which provides BaseRelations to Spark, which, as I > understand it, are physical units of query execution and represent sets of > rows. The Kudu BaseRelation (the KuduRelation) implements a couple of traits > to fit into Spark: PrunedFilteredScan, which allows predicates to be pushed > into Kudu, and InsertableRelation, which allows writes to be pushed into > Kudu. An issue with these bindings is that, while they provide interfaces to > insert/get data, they do not provide interfaces to push details to Spark that > might be useful to optimizing a Kudu query. > Among other things, this is inconvenient for all datasources that might want > to take such optimizations into their own hands, and the Spark community > appears to be making efforts in revamping their DataSource APIs in the form > of DataSourceV2, and as it pertains to read support, the v2 DataSourceReader. > This new world order provides a clear path towards implementing various > optimizations that are currently unavailable with the current Spark bindings, > without pushing changes to Spark itself. > Of note, the v2 DataSourceReader can be extended with > SupportsReportStatistics, which could allow Kudu to expose statistics to Kudu > without having to rely on HMS (although pushing stats to HMS isn't an > unreasonable approach either). More traits and details about the API can be > found > [here|https://spark.apache.org/docs/2.3.0/api/java/org/apache/spark/sql/sources/v2/reader/DataSourceReader.html]. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (KUDU-2490) implement Kudu DataSourceV2 and related classes
[ https://issues.apache.org/jira/browse/KUDU-2490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Henke reassigned KUDU-2490: - Assignee: Grant Henke > implement Kudu DataSourceV2 and related classes > --- > > Key: KUDU-2490 > URL: https://issues.apache.org/jira/browse/KUDU-2490 > Project: Kudu > Issue Type: Improvement > Components: spark >Reporter: Andrew Wong >Assignee: Grant Henke >Priority: Major > Labels: roadmap-candidate > > The current Kudu-Spark bindings implement a DefaultSource that extends a > RelationProvider, which provides BaseRelations to Spark, which, as I > understand it, are physical units of query execution and represent sets of > rows. The Kudu BaseRelation (the KuduRelation) implements a couple of traits > to fit into Spark: PrunedFilteredScan, which allows predicates to be pushed > into Kudu, and InsertableRelation, which allows writes to be pushed into > Kudu. An issue with these bindings is that, while they provide interfaces to > insert/get data, they do not provide interfaces to push details to Spark that > might be useful to optimizing a Kudu query. > Among other things, this is inconvenient for all datasources that might want > to take such optimizations into their own hands, and the Spark community > appears to be making efforts in revamping their DataSource APIs in the form > of DataSourceV2, and as it pertains to read support, the v2 DataSourceReader. > This new world order provides a clear path towards implementing various > optimizations that are currently unavailable with the current Spark bindings, > without pushing changes to Spark itself. > Of note, the v2 DataSourceReader can be extended with > SupportsReportStatistics, which could allow Kudu to expose statistics to Kudu > without having to rely on HMS (although pushing stats to HMS isn't an > unreasonable approach either). More traits and details about the API can be > found > [here|https://spark.apache.org/docs/2.3.0/api/java/org/apache/spark/sql/sources/v2/reader/DataSourceReader.html]. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KUDU-3205) NPE in KuduScanTokenBuilder#build after a tablet server goes down
[
https://issues.apache.org/jira/browse/KUDU-3205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Grant Henke resolved KUDU-3205.
---
Fix Version/s: 1.14.0
Resolution: Fixed
> NPE in KuduScanTokenBuilder#build after a tablet server goes down
> -
>
> Key: KUDU-3205
> URL: https://issues.apache.org/jira/browse/KUDU-3205
> Project: Kudu
> Issue Type: Bug
> Components: spark
>Affects Versions: 1.13.0
>Reporter: Junegunn Choi
>Assignee: Grant Henke
>Priority: Major
> Fix For: 1.14.0
>
>
> When a tablet server goes down while running a query on Spark, the connection
> becomes unusable due to the cached tablet locations that have become stale.
> h2. Steps to reproduce
> h3. Start spark-shell with kudu-spark2 1.13.0
> The problem is not reproducible with kudu-spark2 1.12.0 or below, because it
> was introduced in [KUDU-1802
> |https://github.com/apache/kudu/commit/d23ee5d38ddc4317f431dd65df0c825c00cc968a].
> h3. Run a scan query
> {code:scala}
> import org.apache.kudu.spark.kudu._
> val dummy = spark.read.options(Map("kudu.master" -> kuduMasters, "kudu.table"
> -> "dummy")).kudu
> dummy.createOrReplaceTempView("dummy")
> spark.sql("select sum(id), min(val2), max(val2), count(*) from dummy").show
> {code}
> h3. Kill a tablet server
> Kill one of the tablet servers that are serving data for the query. The query
> should fail immediately.
> {noformat}
> org.apache.spark.SparkException: Job aborted due to stage failure: Task 2 in
> stage 0.0 failed 1 times, most recent failure: Lost task 2.0 in stage 0.0
> (TID 2, localhost, executor driver): java.lang.RuntimeException:
> org.apache.kudu.client.NonRecoverableException: Scanner *** not found (it may
> have expired)
> {noformat}
> h3. Re-run the query
> {code:scala}
> spark.sql("select sum(id), min(val2), max(val2), count(*) from dummy").show
> {code}
> Doesn't work, fails with an NPE.
> {noformat}
> Caused by: java.lang.RuntimeException: java.lang.NullPointerException
> at
> org.apache.kudu.client.KuduScanToken$KuduScanTokenBuilder.build(KuduScanToken.java:697)
> at org.apache.kudu.spark.kudu.KuduRDD.getPartitions(KuduRDD.scala:95)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:273)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:269)
> at scala.Option.getOrElse(Option.scala:121)
> at org.apache.spark.rdd.RDD.partitions(RDD.scala:269)
> at
> org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:49)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:273)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:269)
> at scala.Option.getOrElse(Option.scala:121)
> at org.apache.spark.rdd.RDD.partitions(RDD.scala:269)
> at
> org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:49)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:273)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:269)
> at scala.Option.getOrElse(Option.scala:121)
> at org.apache.spark.rdd.RDD.partitions(RDD.scala:269)
> at
> org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:49)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:273)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:269)
> at scala.Option.getOrElse(Option.scala:121)
> at org.apache.spark.rdd.RDD.partitions(RDD.scala:269)
> at
> org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:49)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:273)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:269)
> at scala.Option.getOrElse(Option.scala:121)
> at org.apache.spark.rdd.RDD.partitions(RDD.scala:269)
> at org.apache.spark.ShuffleDependency.(Dependency.scala:94)
> at
> org.apache.spark.sql.execution.exchange.ShuffleExchangeExec$.prepareShuffleDependency(ShuffleExchangeExec.scala:323)
> at
> org.apache.spark.sql.execution.exchange.ShuffleExchangeExec.prepareShuffleDependency(ShuffleExchangeExec.scala:91)
> at
> org.apache.spark.sql.execution.exchange.ShuffleExchangeExec$$anonfun$doExecute$1.apply(ShuffleExchangeExec.scala:128)
> at
> org.apache.spark.sql.execution.exchange.ShuffleExchangeExec$$anonfun$doExecute$1.apply(ShuffleExchangeExec.scala:119)
> at
> org.apache.spark.sql.catalyst.errors.package$.attachTree(package.scala:52)
> ... 86 more
> Caused by: java.lang.NullPointerException
> at
> org.apache.kudu.client.KuduScanToken$KuduScanTokenBuilder.build(KuduScanToken.java:674)
> ... 117 more
> {noformat}
> Re-creating the DataFrame doesn't help:
> {code:scala}
> val dummy = spark.read.options(Map("kudu.master" -> kuduMasters, "kudu.table"
> ->
[jira] [Updated] (KUDU-331) Allow evicting already-timed-out RPCs from inbound queue when new RPC arrives
[ https://issues.apache.org/jira/browse/KUDU-331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Henke updated KUDU-331: - Labels: performance scalability (was: ) > Allow evicting already-timed-out RPCs from inbound queue when new RPC arrives > - > > Key: KUDU-331 > URL: https://issues.apache.org/jira/browse/KUDU-331 > Project: Kudu > Issue Type: Improvement > Components: rpc >Affects Versions: M4 >Reporter: Todd Lipcon >Priority: Major > Labels: performance, scalability > > If the queue is really backed up, there might be RPCs in the queue which are > already timed out. Currently, we skip those when we get to processing them. > We could do a little better, though, by checking for timed-out calls when a > new call arrives. If a call arrives and the queue is full, just iterate > through the queue and clean out any that have already timed out to make room > for the new call. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KUDU-2981) Push predicate evaluation into more CFile decoders
[ https://issues.apache.org/jira/browse/KUDU-2981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Henke updated KUDU-2981: -- Labels: newbie performance (was: newbie) > Push predicate evaluation into more CFile decoders > -- > > Key: KUDU-2981 > URL: https://issues.apache.org/jira/browse/KUDU-2981 > Project: Kudu > Issue Type: Improvement > Components: cfile, perf >Reporter: Bankim Bhavsar >Priority: Major > Labels: newbie, performance > > Commit c0f3727 added an optimization to push predicate evaluation into the > CFile decoders without fully materializing the contents of each cblock. It > did this with dictionary-encoded blocks, but the optimization can be applied > to any other encoding types too. > KUDU-736 also notes that we may be able to apply some predicates on > bitshuffled data. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KUDU-1235) Add Get API
[
https://issues.apache.org/jira/browse/KUDU-1235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Grant Henke updated KUDU-1235:
--
Labels: performance roadmap-candidate (was: roadmap-candidate)
> Add Get API
> ---
>
> Key: KUDU-1235
> URL: https://issues.apache.org/jira/browse/KUDU-1235
> Project: Kudu
> Issue Type: New Feature
> Components: client, perf, tablet, tserver
>Reporter: Binglin Chang
>Priority: Major
> Labels: performance, roadmap-candidate
> Attachments: perf-get.svg, perf-scan-opt.svg, perf-scan.svg
>
>
> Get API is more user friendly and efficient if use just want primary key
> lookup.
> I setup a cluster and test get/scan single row using ycsb, initial test shows
> better performance for get.
> {noformat}
> kudu_workload:
> recordcount=100
> operationcount=100
> workload=com.yahoo.ycsb.workloads.CoreWorkload
> readallfields=false
> readproportion=1
> updateproportion=0
> scanproportion=0
> insertproportion=0
> requestdistribution=uniform
> use_get_api=false
> load:
> ./bin/ycsb load kudu -P workloads/kudu_workload -p sync_ops=false -p
> pre_split_num_tablets=1 -p table_name=ycsb_wiki_example -p
> masterQuorum='c3-kudu-tst-st01.bj:32600' -threads 100
> read test:
> ./bin/ycsb run kudu -P workloads/kudu_workload -p
> masterQuorum='c3-kudu-tst-st01.bj:32600' -threads 100
> {noformat}
> Get API:
> [OVERALL], RunTime(ms), 21304.0
> [OVERALL], Throughput(ops/sec), 46939.54187007135
> [CLEANUP], Operations, 100.0
> [CLEANUP], AverageLatency(us), 423.57
> [CLEANUP], MinLatency(us), 24.0
> [CLEANUP], MaxLatency(us), 19327.0
> [CLEANUP], 95thPercentileLatency(us), 52.0
> [CLEANUP], 99thPercentileLatency(us), 18815.0
> [READ], Operations, 100.0
> [READ], AverageLatency(us), 2065.185152
> [READ], MinLatency(us), 134.0
> [READ], MaxLatency(us), 92159.0
> [READ], 95thPercentileLatency(us), 2391.0
> [READ], 99thPercentileLatency(us), 6359.0
> [READ], Return=0, 100
> Scan API:
> [OVERALL], RunTime(ms), 38259.0
> [OVERALL], Throughput(ops/sec), 26137.6408165399
> [CLEANUP], Operations, 100.0
> [CLEANUP], AverageLatency(us), 47.32
> [CLEANUP], MinLatency(us), 16.0
> [CLEANUP], MaxLatency(us), 1837.0
> [CLEANUP], 95thPercentileLatency(us), 41.0
> [CLEANUP], 99thPercentileLatency(us), 158.0
> [READ], Operations, 100.0
> [READ], AverageLatency(us), 3595.825249
> [READ], MinLatency(us), 139.0
> [READ], MaxLatency(us), 3139583.0
> [READ], 95thPercentileLatency(us), 3775.0
> [READ], 99thPercentileLatency(us), 7659.0
> [READ], Return=0, 100
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (KUDU-2935) Implement built-in NTP client
[
https://issues.apache.org/jira/browse/KUDU-2935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Grant Henke updated KUDU-2935:
--
Labels: clock roadmap-candidate (was: clock)
> Implement built-in NTP client
> -
>
> Key: KUDU-2935
> URL: https://issues.apache.org/jira/browse/KUDU-2935
> Project: Kudu
> Issue Type: New Feature
> Components: clock, master, tserver
>Affects Versions: 1.11.0
>Reporter: Alexey Serbin
>Priority: Major
> Labels: clock, roadmap-candidate
>
> It would be nice to add a stripped-down implementation of built-in NTP client
> without any reliance on the kernel NTP discipline. The built-in client
> should maintain wall clock synchronized with NTP servers, and calling
> {{WalltimeWithError()}} should return wall clock timestamp with the
> estimation of error/offset from true time. Having built-in NTP client would
> provide more control over acceptable clock error and jitter acceptable for
> HybridTime timestamp generation.
> From the operability perspective, it would make it easier to run Kudu in
> containerized environments and overall make it easier for users to configure
> NTP even if they don't have superuser privileges at a node.
> The very first implementation should be good enough to work with properly
> configured and well behaving NTP servers, not necessarily being full-featured
> and 100% RFC-compliant NTP client. Later on, we can add more features and
> constraints to protect against misbehaving and rogue NTP servers.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (KUDU-3132) Support RPC compression
[ https://issues.apache.org/jira/browse/KUDU-3132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Henke updated KUDU-3132: -- Labels: performance roadmap-candidate (was: roadmap-candidate) > Support RPC compression > --- > > Key: KUDU-3132 > URL: https://issues.apache.org/jira/browse/KUDU-3132 > Project: Kudu > Issue Type: New Feature > Components: rpc >Reporter: Grant Henke >Priority: Major > Labels: performance, roadmap-candidate > > I have seen more and more deployments of Kudu where the tablet servers are > not co-located with the compute resources such as Impala or Spark. In > deployments like this, there could be significant network savings by > compressing the RPC messages (especially those that write or scan data). > Adding simple LZ4 or Snappy compression support to the RPC messages when not > on a loopback/local connection should be a great improvement for network > bound applications. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KUDU-3132) Support RPC compression
[ https://issues.apache.org/jira/browse/KUDU-3132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Henke updated KUDU-3132: -- Component/s: perf > Support RPC compression > --- > > Key: KUDU-3132 > URL: https://issues.apache.org/jira/browse/KUDU-3132 > Project: Kudu > Issue Type: New Feature > Components: perf, rpc >Reporter: Grant Henke >Priority: Major > Labels: performance, roadmap-candidate > > I have seen more and more deployments of Kudu where the tablet servers are > not co-located with the compute resources such as Impala or Spark. In > deployments like this, there could be significant network savings by > compressing the RPC messages (especially those that write or scan data). > Adding simple LZ4 or Snappy compression support to the RPC messages when not > on a loopback/local connection should be a great improvement for network > bound applications. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KUDU-1639) Improve predicate pushdown
[ https://issues.apache.org/jira/browse/KUDU-1639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Henke updated KUDU-1639: -- Component/s: perf > Improve predicate pushdown > -- > > Key: KUDU-1639 > URL: https://issues.apache.org/jira/browse/KUDU-1639 > Project: Kudu > Issue Type: Improvement > Components: client, perf, tablet >Reporter: Dan Burkert >Priority: Major > Labels: perf, roadmap-candidate > > Umbrella ticket for proposed improvements to predicates, scan optimization > based on predicates, and partition pruning. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KUDU-1639) Improve predicate pushdown
[ https://issues.apache.org/jira/browse/KUDU-1639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Henke updated KUDU-1639: -- Labels: performance roadmap-candidate (was: perf roadmap-candidate) > Improve predicate pushdown > -- > > Key: KUDU-1639 > URL: https://issues.apache.org/jira/browse/KUDU-1639 > Project: Kudu > Issue Type: Improvement > Components: client, perf, tablet >Reporter: Dan Burkert >Priority: Major > Labels: performance, roadmap-candidate > > Umbrella ticket for proposed improvements to predicates, scan optimization > based on predicates, and partition pruning. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KUDU-1439) Optimization for batch inserts into empty key ranges
[ https://issues.apache.org/jira/browse/KUDU-1439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Henke updated KUDU-1439: -- Labels: performance (was: ) > Optimization for batch inserts into empty key ranges > > > Key: KUDU-1439 > URL: https://issues.apache.org/jira/browse/KUDU-1439 > Project: Kudu > Issue Type: Improvement > Components: perf, tablet >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Major > Labels: performance > > Got this idea from a CockroachDB optimization: > https://github.com/cockroachdb/cockroach/pull/6375 > The short version is that if we have a moderately large batch of inserts > which are sorted, we can do the following pseudocode: > - sort the inserts by primary key > - instead of using bloom filter checks, use SeekAtOrAfter on the first > primary key in the batch. This yields the next higher primary key that might > exist in the table (_nextKey_). > - for each of the keys in the sorted batch, if it's less than _nextKey_, we > don't need to do an existence check for it. > In the common case where clients are writing non-overlapping batches of rows > (eg importing from parquet) this should reduce the number of seeks and bloom > checks dramatically (order of batch size). Plus, it doesn't require much new > code to be written, so worth a shot. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KUDU-1865) Create fast path for RespondSuccess() in KRPC
[
https://issues.apache.org/jira/browse/KUDU-1865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Grant Henke updated KUDU-1865:
--
Component/s: perf
> Create fast path for RespondSuccess() in KRPC
> -
>
> Key: KUDU-1865
> URL: https://issues.apache.org/jira/browse/KUDU-1865
> Project: Kudu
> Issue Type: Improvement
> Components: perf, rpc
>Reporter: Sailesh Mukil
>Priority: Major
> Labels: perfomance, rpc, scalability
> Attachments: alloc-pattern.py, cross-thread.txt
>
>
> A lot of RPCs just respond with RespondSuccess() which returns the exact
> payload every time. This takes the same path as any other response by
> ultimately calling Connection::QueueResponseForCall() which has a few small
> allocations. These small allocations (and their corresponding deallocations)
> are called quite frequently (once for every IncomingCall) and end up taking
> quite some time in the kernel (traversing the free list, spin locks etc.)
> This was found when [~mmokhtar] ran some profiles on Impala over KRPC on a 20
> node cluster and found the following:
> The exact % of time spent is hard to quantify from the profiles, but these
> were the among the top 5 of the slowest stacks:
> {code:java}
> impalad ! tcmalloc::CentralFreeList::ReleaseToSpans - [unknown source file]
> impalad ! tcmalloc::CentralFreeList::ReleaseListToSpans + 0x1a - [unknown
> source file]
> impalad ! tcmalloc::CentralFreeList::InsertRange + 0x3b - [unknown source
> file]
> impalad ! tcmalloc::ThreadCache::ReleaseToCentralCache + 0x103 - [unknown
> source file]
> impalad ! tcmalloc::ThreadCache::Scavenge + 0x3e - [unknown source file]
> impalad ! operator delete + 0x329 - [unknown source file]
> impalad ! __gnu_cxx::new_allocator::deallocate + 0x4 -
> new_allocator.h:110
> impalad ! std::_Vector_base std::allocator>::_M_deallocate + 0x5 - stl_vector.h:178
> impalad ! ~_Vector_base + 0x4 - stl_vector.h:160
> impalad ! ~vector - stl_vector.h:425 'slices' vector
> impalad ! kudu::rpc::Connection::QueueResponseForCall + 0xac -
> connection.cc:433
> impalad ! kudu::rpc::InboundCall::Respond + 0xfa - inbound_call.cc:133
> impalad ! kudu::rpc::InboundCall::RespondSuccess + 0x43 - inbound_call.cc:77
> impalad ! kudu::rpc::RpcContext::RespondSuccess + 0x1f7 - rpc_context.cc:66
> ..
> {code}
> {code:java}
> impalad ! tcmalloc::CentralFreeList::FetchFromOneSpans - [unknown source file]
> impalad ! tcmalloc::CentralFreeList::RemoveRange + 0xc0 - [unknown source
> file]
> impalad ! tcmalloc::ThreadCache::FetchFromCentralCache + 0x62 - [unknown
> source file]
> impalad ! operator new + 0x297 - [unknown source file]<--- Creating
> new 'OutboundTransferTask' object.
> impalad ! kudu::rpc::Connection::QueueResponseForCall + 0x76 -
> connection.cc:432
> impalad ! kudu::rpc::InboundCall::Respond + 0xfa - inbound_call.cc:133
> impalad ! kudu::rpc::InboundCall::RespondSuccess + 0x43 - inbound_call.cc:77
> impalad ! kudu::rpc::RpcContext::RespondSuccess + 0x1f7 - rpc_context.cc:66
> ...
> {code}
> Even creating and deleting the 'RpcContext' takes a lot of time:
> {code:java}
> impalad ! tcmalloc::CentralFreeList::ReleaseToSpans - [unknown source file]
> impalad ! tcmalloc::CentralFreeList::ReleaseListToSpans + 0x1a - [unknown
> source file]
> impalad ! tcmalloc::CentralFreeList::InsertRange + 0x3b - [unknown source
> file]
> impalad ! tcmalloc::ThreadCache::ReleaseToCentralCache + 0x103 - [unknown
> source file]
> impalad ! tcmalloc::ThreadCache::Scavenge + 0x3e - [unknown source file]
> impalad ! operator delete + 0x329 - [unknown source file]
> impalad ! impala::TransmitDataResponsePb::~TransmitDataResponsePb + 0x16 -
> impala_internal_service.pb.cc:1221
> impalad ! impala::TransmitDataResponsePb::~TransmitDataResponsePb + 0x8 -
> impala_internal_service.pb.cc:1222
> impalad ! kudu::DefaultDeleter::operator() + 0x5 -
> gscoped_ptr.h:145
> impalad ! ~gscoped_ptr_impl + 0x9 - gscoped_ptr.h:228
> impalad ! ~gscoped_ptr - gscoped_ptr.h:318
> impalad ! kudu::rpc::RpcContext::~RpcContext + 0x1e - rpc_context.cc:53
> <-
> impalad ! kudu::rpc::RpcContext::RespondSuccess + 0x1ff - rpc_context.cc:67
> {code}
> The above show that creating these small objects under moderately heavy load
> results in heavy contention in the kernel. We will benefit a lot if we create
> a fast path for 'RespondSuccess'.
> My suggestion is to create all these small objects at once along with the
> 'InboundCall' object while it is being created, in a 'RespondSuccess'
> structure, and just use that structure when we want to send 'success' back to
> the sender. This would already contain the 'OutboundTransferTask', a 'Slice'
> with 'success', etc. We would expect that most RPCs respond with 'success' a
>
[jira] [Updated] (KUDU-2244) spinlock contention in raft_consensus
[
https://issues.apache.org/jira/browse/KUDU-2244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Grant Henke updated KUDU-2244:
--
Labels: performance (was: )
> spinlock contention in raft_consensus
> -
>
> Key: KUDU-2244
> URL: https://issues.apache.org/jira/browse/KUDU-2244
> Project: Kudu
> Issue Type: Improvement
> Components: consensus
>Reporter: Andrew Wong
>Priority: Major
> Labels: performance
>
> I was going through the logs of a cluster that was seeing a bunch of
> kernel_stack_watchdog traces, and the slowness seemed to be caused by a lot
> of activity in consensus requests. E.g.
> W1214 18:57:29.514219 36138 kernel_stack_watchdog.cc:145] Thread 36317 stuck
> at
> /data/jenkins/workspace/generic-package-centos64-7-0-impala/topdir/BUILD/kudu-1.3.0-cdh5.11.0/src/kudu/rpc/outbound_call.cc:192
> for 123ms:
> Kernel stack:
> [] sys_sched_yield+0x65/0xd0
> [] system_call_fastpath+0x16/0x1b
> [] 0x
> User stack:
> @ 0x7f72fab92057 __GI___sched_yield
> @ 0x19498bf kudu::Thread::StartThread()
> @ 0x1952e7d kudu::ThreadPool::CreateThreadUnlocked()
> @ 0x19534d3 kudu::ThreadPool::Submit()
> @ 0x1953a27 kudu::ThreadPool::SubmitFunc()
> @ 0x1953ecb kudu::ThreadPool::SubmitClosure()
> @ 0x9c94ec kudu::consensus::RaftConsensus::ElectionCallback()
> @ 0x9e6032 kudu::consensus::LeaderElection::CheckForDecision()
> @ 0x9e78c3
> kudu::consensus::LeaderElection::VoteResponseRpcCallback()
> @ 0xa8b137 kudu::rpc::OutboundCall::CallCallback()
> @ 0xa8c2bc kudu::rpc::OutboundCall::SetResponse()
> @ 0xa822c0 kudu::rpc::Connection::HandleCallResponse()
> @ 0xa83ffc ev::base<>::method_thunk<>()
> @ 0x198a07f ev_invoke_pending
> @ 0x198af71 ev_run
> @ 0xa5e049 kudu::rpc::ReactorThread::RunThread()
> So it seemed to be cause by some slowness in getting threads. Upon perusing
> the logs a bit more, there were a sizable number of spinlock profiling traces:
> W1214 18:54:27.897955 36379 rpcz_store.cc:238] Trace:
> 1214 18:54:26.766922 (+ 0us) service_pool.cc:143] Inserting onto call
> queue
> 1214 18:54:26.771135 (+ 4213us) service_pool.cc:202] Handling call
> 1214 18:54:26.771138 (+ 3us) raft_consensus.cc:1126] Updating replica for
> 0 ops
> 1214 18:54:27.897699 (+1126561us) raft_consensus.cc:1165] Early marking
> committed up to index 0
> 1214 18:54:27.897700 (+ 1us) raft_consensus.cc:1170] Triggering prepare
> for 0 ops
> 1214 18:54:27.897701 (+ 1us) raft_consensus.cc:1282] Marking committed up
> to 1766
> 1214 18:54:27.897702 (+ 1us) raft_consensus.cc:1332] Filling consensus
> response to leader.
> 1214 18:54:27.897736 (+34us) spinlock_profiling.cc:255] Waited 991 ms on
> lock 0x120b3540. stack: 019406c5 009c60d7 009c75f7
> 007dc628 00a7adfc 00a7b9cd 0194d059
> 7f72fbcc2dc4 7f72fabad1cc
> 1214 18:54:27.897737 (+ 1us) raft_consensus.cc:1327] UpdateReplicas()
> finished
> 1214 18:54:27.897741 (+ 4us) inbound_call.cc:130] Queueing success
> response
> Metrics: {"spinlock_wait_cycles":2478395136}
> Each of the traces noted on the order of 500-1000ms of waiting on spinlocks.
> Upon looking at raft_consensus.cc, it seems we're holding a spinlock
> (update_lock_) while we call RaftConsensus::UpdateReplica(), which according
> to its header, "won't return until all operations have been stored in the log
> and all Prepares() have been completed". While locking may be necessary, it's
> worth considering using a different kind of lock here.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (KUDU-2076) Deleting/updating is slow on single numeric row key tables
[ https://issues.apache.org/jira/browse/KUDU-2076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Henke updated KUDU-2076: -- Labels: performance (was: ) > Deleting/updating is slow on single numeric row key tables > -- > > Key: KUDU-2076 > URL: https://issues.apache.org/jira/browse/KUDU-2076 > Project: Kudu > Issue Type: Bug > Components: perf, tablet >Affects Versions: 1.4.0 >Reporter: Jean-Daniel Cryans >Priority: Major > Labels: performance > > A user reported that deleting 50M rows on the simplest of tables, (id INT, > text STRING, PRIMARY KEY (id)), doesn't complete. > It reproduces locally and I was able to see that we're deleting 1.4 rows / ms > which is awful considering that everything is cached. > Todd found that we're spending most of our time decoding big blocks of > bit-shuffled keys. Intuitively he though that having a composite row key > would perform better and indeed adding a column set to 0 in front makes it > 10x faster. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KUDU-1954) Improve maintenance manager behavior in heavy write workload
[ https://issues.apache.org/jira/browse/KUDU-1954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Henke updated KUDU-1954: -- Labels: performance roadmap-candidate scalability (was: roadmap-candidate scalability) > Improve maintenance manager behavior in heavy write workload > > > Key: KUDU-1954 > URL: https://issues.apache.org/jira/browse/KUDU-1954 > Project: Kudu > Issue Type: Improvement > Components: compaction, perf, tserver >Affects Versions: 1.3.0 >Reporter: Todd Lipcon >Priority: Major > Labels: performance, roadmap-candidate, scalability > Attachments: mm-trace.png > > > During the investigation in [this > doc|https://docs.google.com/document/d/1U1IXS1XD2erZyq8_qG81A1gZaCeHcq2i0unea_eEf5c/edit] > I found a few maintenance-manager-related issues during heavy writes: > - we don't schedule flushes until we are already in "backpressure" realm, so > we spent most of our time doing backpressure > - even if we configure N maintenance threads, we typically are only using > ~50% of those threads due to the scheduling granularity > - when we do hit the "memory-pressure flush" threshold, all threads quickly > switch to flushing, which then brings us far beneath the threshold > - long running compactions can temporarily starve flushes > - high volume of writes can starve compactions -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KUDU-2482) Add "delete ignore" capability to spark
[
https://issues.apache.org/jira/browse/KUDU-2482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Grant Henke resolved KUDU-2482.
---
Fix Version/s: 1.14.0
Resolution: Fixed
This is resolved in KUDU-1563
> Add "delete ignore" capability to spark
> ---
>
> Key: KUDU-2482
> URL: https://issues.apache.org/jira/browse/KUDU-2482
> Project: Kudu
> Issue Type: Improvement
> Components: spark
>Affects Versions: 1.7.1
>Reporter: William Berkeley
>Priority: Major
> Fix For: 1.14.0
>
>
> If a user tries to delete a row from a Kudu table using kudu-spark but the
> row is not present, they receive an exception like
> {{java.lang.RuntimeException: failed to write N rows from DataFrame to Kudu;
> sample errors: Not found: key not found (error 0)}}
> This is the expected behavior, but it might also be nice to allow users to
> ignore PK errors so they can achieve clean semantics of "delete if present,
> else ignore".
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (KUDU-2887) Expose the tablet statistics in Client API
[ https://issues.apache.org/jira/browse/KUDU-2887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Henke updated KUDU-2887: -- Labels: impala performance roadmap-candidate (was: impala roadmap-candidate) > Expose the tablet statistics in Client API > -- > > Key: KUDU-2887 > URL: https://issues.apache.org/jira/browse/KUDU-2887 > Project: Kudu > Issue Type: Improvement > Components: client >Reporter: LiFu He >Priority: Minor > Labels: impala, performance, roadmap-candidate > > The patch about aggregating tablet statistics on the kudu-master is on the > way. And I think it's important to expose these statistics in client api by > which the query engine can optimize their query plan. For example: (1) adjust > the order of scanning tables, (2) Split a big tablet into multiple range > pieces(KUDU-2437) to improve concurrency automatically, (3) speed up the > query like "select count( *) from table". -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KUDU-523) Add comprehensive cfile read/write benchmarks to dashboard
[ https://issues.apache.org/jira/browse/KUDU-523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Henke updated KUDU-523: - Labels: benchmarks performance (was: benchmarks) > Add comprehensive cfile read/write benchmarks to dashboard > -- > > Key: KUDU-523 > URL: https://issues.apache.org/jira/browse/KUDU-523 > Project: Kudu > Issue Type: Improvement > Components: perf, test >Affects Versions: Backlog >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Major > Labels: benchmarks, performance > > We have a couple of benchmarks in cfile-test, but we don't expose them as > part of the benchmarks dashboard. > We should set up a templated test that runs through all valid pairs of > type/encoding and times sequential write, sequential read, and random read of > a cfile with a few million rows. We should also test each including some > NULLs (eg 50/50 alternating nulls). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KUDU-3033) Add min/max values for the non-primary key columns in the metadata of rowsets/datablocks
[ https://issues.apache.org/jira/browse/KUDU-3033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Henke updated KUDU-3033: -- Labels: performance roadmap-candidate (was: ) > Add min/max values for the non-primary key columns in the metadata of > rowsets/datablocks > > > Key: KUDU-3033 > URL: https://issues.apache.org/jira/browse/KUDU-3033 > Project: Kudu > Issue Type: New Feature > Components: cfile, perf, tablet >Reporter: LiFu He >Priority: Major > Labels: performance, roadmap-candidate > > It's possible to add min/max values for the non-primary key columns in the > metadata of diskrowset/datablock, and then we can skip decoding/evaluating > the unnecessary diskrowset/datablock while scanning. Just like the "compute > stats" feature on impala, and the only difference is that kudu supports > updates. So, the min/max values should be invalid if the columns that have > deltas while scanning. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KUDU-2975) Spread WAL across multiple data directories
[ https://issues.apache.org/jira/browse/KUDU-2975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Henke updated KUDU-2975: -- Labels: performance roadmap-candidate scalability (was: roadmap-candidate scalability) > Spread WAL across multiple data directories > --- > > Key: KUDU-2975 > URL: https://issues.apache.org/jira/browse/KUDU-2975 > Project: Kudu > Issue Type: New Feature > Components: fs, perf, tablet, tserver >Reporter: LiFu He >Assignee: YangSong >Priority: Major > Labels: performance, roadmap-candidate, scalability > Attachments: network.png, tserver-WARNING.png, util.png > > > Recently, we deployed a new kudu cluster and every node has 12 SSD. Then, we > created a big table and loaded data to it through flink. We noticed that the > util of one SSD which is used to store WAL is 100% but others are free. So, > we suggest to spread WAL across multiple data directories. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KUDU-2019) Expose table/column statistics
[ https://issues.apache.org/jira/browse/KUDU-2019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Henke updated KUDU-2019: -- Labels: impala performance roadmap-candidate (was: impala roadmap-candidate) > Expose table/column statistics > -- > > Key: KUDU-2019 > URL: https://issues.apache.org/jira/browse/KUDU-2019 > Project: Kudu > Issue Type: New Feature > Components: client >Affects Versions: 1.0.0 >Reporter: Matthew Jacobs >Priority: Major > Labels: impala, performance, roadmap-candidate > > It would be helpful for query engines such as Impala to get table/column > statistics from Kudu. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KUDU-3160) Upgrade devtoolset version for el6
[ https://issues.apache.org/jira/browse/KUDU-3160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Henke resolved KUDU-3160. --- Fix Version/s: 1.14.0 Resolution: Fixed This was completed in https://github.com/apache/kudu/commit/5aacd6c64ddd2669005d0b3ca57801a7a7124033 > Upgrade devtoolset version for el6 > -- > > Key: KUDU-3160 > URL: https://issues.apache.org/jira/browse/KUDU-3160 > Project: Kudu > Issue Type: Task > Components: build >Affects Versions: 1.12.0 >Reporter: Bankim Bhavsar >Priority: Major > Fix For: 1.14.0 > > Attachments: client_symbol-test.txt > > > Currently for building RHEL6 based distributions, kudu uses devtoolset3. > However devtoolset3 is EOL as of Nov 2016 and it's repo location has been > removed from the official software collections repository. See related > KUDU-3159. > So upgrade devtoolset to a newer version for building kudu on RHEL6 based > distributions. > Apart from the fact that devtoolset3 is EOL and not available officially, > newer version of devtoolset will also lead to generation of better code. > See comment from [~tlipcon] > https://gerrit.cloudera.org/c/16114/#message-86ea8228_e84eeb60 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KUDU-1291) Efficiently support predicates on non-prefix key components
[ https://issues.apache.org/jira/browse/KUDU-1291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Henke updated KUDU-1291: -- Labels: performance roadmap-candidate (was: roadmap-candidate) > Efficiently support predicates on non-prefix key components > --- > > Key: KUDU-1291 > URL: https://issues.apache.org/jira/browse/KUDU-1291 > Project: Kudu > Issue Type: Sub-task > Components: perf, tablet >Reporter: Todd Lipcon >Priority: Major > Labels: performance, roadmap-candidate > > In a lot of workloads, users have a compound primary key where the first > component (or few components) is low cardinality. For example, a time series > workload may have (year, month, day, entity_id, timestamp) as a primary key. > A metrics or log storage workload might have (hostname, timestamp). > It's common to want to do cross-user or cross-date analytics like 'WHERE > timestamp BETWEEN and ' without specifying any predicate for the first > column(s) of the PK. Currently, we do not execute this efficiently, but > rather scan the whole table evaluating the predicate. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KUDU-636) optimization: we spend a lot of time in alloc/free
[ https://issues.apache.org/jira/browse/KUDU-636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Henke updated KUDU-636: - Labels: performance (was: ) > optimization: we spend a lot of time in alloc/free > -- > > Key: KUDU-636 > URL: https://issues.apache.org/jira/browse/KUDU-636 > Project: Kudu > Issue Type: Improvement > Components: perf >Affects Versions: Public beta >Reporter: Todd Lipcon >Priority: Major > Labels: performance > > Looking at a workload in the cluster, several of the top 10 lines of perf > report are tcmalloc-related. It seems like we don't do a good job of making > use of the per-thread free-lists, and we end up in a lot of contention on the > central free list. There are a few low-hanging fruit things we could do to > improve this for a likely perf boost. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KUDU-3186) Translate columnar RPC to row based API in C++ client
[ https://issues.apache.org/jira/browse/KUDU-3186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Henke updated KUDU-3186: -- Component/s: perf > Translate columnar RPC to row based API in C++ client > -- > > Key: KUDU-3186 > URL: https://issues.apache.org/jira/browse/KUDU-3186 > Project: Kudu > Issue Type: Improvement > Components: client, perf >Affects Versions: 1.12.0 >Reporter: Grant Henke >Priority: Major > Labels: impala, performance, roadmap-candidate > > The Java client is capable of receiving the columnar based RPC row format > response and then translating it for use in the row based API. This is useful > because it allows the more efficient (wire size and server CPU usage) format > to be use without code changes in client applications. > This would be especially beneficial for Apache Impala in cases where an > external cluster is accessing Kudu data remotely. > The Java client commit for reference: > https://github.com/apache/kudu/commit/34e85622f37f88b0436d9ba51df7592bf0e159de -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KUDU-3186) Translate columnar RPC to row based API in C++ client
[ https://issues.apache.org/jira/browse/KUDU-3186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Henke updated KUDU-3186: -- Labels: impala performance roadmap-candidate (was: impala perf roadmap-candidate) > Translate columnar RPC to row based API in C++ client > -- > > Key: KUDU-3186 > URL: https://issues.apache.org/jira/browse/KUDU-3186 > Project: Kudu > Issue Type: Improvement > Components: client >Affects Versions: 1.12.0 >Reporter: Grant Henke >Priority: Major > Labels: impala, performance, roadmap-candidate > > The Java client is capable of receiving the columnar based RPC row format > response and then translating it for use in the row based API. This is useful > because it allows the more efficient (wire size and server CPU usage) format > to be use without code changes in client applications. > This would be especially beneficial for Apache Impala in cases where an > external cluster is accessing Kudu data remotely. > The Java client commit for reference: > https://github.com/apache/kudu/commit/34e85622f37f88b0436d9ba51df7592bf0e159de -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KUDU-3210) Support FIPS approved mode
[ https://issues.apache.org/jira/browse/KUDU-3210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Henke resolved KUDU-3210. --- Fix Version/s: 1.14.0 Resolution: Fixed > Support FIPS approved mode > -- > > Key: KUDU-3210 > URL: https://issues.apache.org/jira/browse/KUDU-3210 > Project: Kudu > Issue Type: Improvement >Reporter: Attila Bukor >Assignee: Attila Bukor >Priority: Major > Fix For: 1.14.0 > > > FIPS 140-2 is a standard used to approve cryptographic modules. Some versions > of OpenSSL support a "FIPS mode" where only approved algorithms and key sizes > are enabled. Kudu should be able to run when FIPS mode is enabled and should > provide a way for admins to require that FIPS mode is enabled. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KUDU-2817) C++ Upgrades for before Kudu 1.13 release
[ https://issues.apache.org/jira/browse/KUDU-2817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Henke updated KUDU-2817: -- Description: We should consider reviewing and upgrading our dependencies before the next release. Below is a list of current dependencies and their latest release. * -gflags: 2.2.0 (Nov 2016) -> 2.2.2 (Nov 2018)- * glog: 0.3.5 (May 2017) -> 0.4.0 (Mar 2019) * gmock: 1.8.0 -> 1.8.1 * -gperftools: 2.6.90 -> 2.8 * -protobuf: 3.12.3 -> 3.14.0 * -cmake: 3.16.4 -> 3.19.0 * -snappy: 1.1.4 (Jan 2017) -> 1.1.8- * -lz4: 1.9.2 -> 1.9.3 * -bitshuffle: 55f9b4c (patched, 2016) -> 0.3.5 (Nov 2018)- * -zlib: 1.2.8 (Apr 2013) -> 1.2.11 (Jan 2017)- * libev: 4.20 -> 4.22 * -rapidjson: 1.1.0 (current)- * -squeasel: current- * mustache: 87a592e8aa04497764c533acd6e887618ca7b8a8 (Feb 2017) -> cf5c3dd499ea2bc9eb5c2072fb551dc7af75aa57 (Jun 2017) ** Consider using official mustach c++ support? * -curl: 7.59.0 (Mar 2018) -> 7.68.0- * -crcutil: current- * libunwind: 1.3-rc1 (patched, Nov 2017) -> 1.5.0 * llvm: 9.0.0 -> 11.0.0 * iwyu: 0.13 -> 0.14 * -nvml: 1.1 (2016) -> 1.6 (now called pmdk, Mar 2019)- ** -Patch to replace with memkind is posted- * -boost: 1.61.0 (patched, 2016) -> 1.74.0- * breakpad: 9eac2058b70615519b2c4d8c6bdbfca1bd079e39 (Apr 2013) -> 21b48a72aa50dde84149267f6b7402522b846b24 (Apr 2019) * sparsepp: 47a55825ca3b35eab1ca22b7ab82b9544e32a9af (Nov 2016) -> 5ca6de766db32b3fb08a040636423cd3988d2d4f (Jun 2018) * thrift: 0.11 (Dec 2017) -> 0.13 * -bison: 3.0.4 (patched, 2015) -> 3.5.4- * -hive: 498021fa15186aee8b282d3c032fbd2cede6bec4 (commit in Hive 2) -> 3.1.1 (Oct 2018)- * -hadoop: 2.8.5 (Sept 2018) -> 3.2.0 * -sentry: 505b42e81a9d85c4ebe8db3f48ad7a6e824a5db5 (commit in Master)- * -ranger: f37f5407eee8d2627a4306a25938b151f8e2ba31 -> 2.1.0 * python: 2.7.13 -> (a lot of choices here) * chrony: 3.5 -> 4.0 A quick risk/reward review should be done and we should upgrade the dependencies that are expected to be beneficial. was: We should consider reviewing and upgrading our dependencies before the next release. Below is a list of current dependencies and their latest release. * -gflags: 2.2.0 (Nov 2016) -> 2.2.2 (Nov 2018)- * glog: 0.3.5 (May 2017) -> 0.4.0 (Mar 2019) * gmock: 1.8.0 -> 1.8.1 * gperftools: 2.6.90 -> 2.8 * protobuf: 3.12.3 -> 3.14.0 * cmake: 3.16.4 -> 3.19.0 * -snappy: 1.1.4 (Jan 2017) -> 1.1.8- * lz4: 1.9.2 -> 1.9.3 * -bitshuffle: 55f9b4c (patched, 2016) -> 0.3.5 (Nov 2018)- * -zlib: 1.2.8 (Apr 2013) -> 1.2.11 (Jan 2017)- * libev: 4.20 -> 4.22 * -rapidjson: 1.1.0 (current)- * -squeasel: current- * mustache: 87a592e8aa04497764c533acd6e887618ca7b8a8 (Feb 2017) -> cf5c3dd499ea2bc9eb5c2072fb551dc7af75aa57 (Jun 2017) ** Consider using official mustach c++ support? * -curl: 7.59.0 (Mar 2018) -> 7.68.0- * -crcutil: current- * libunwind: 1.3-rc1 (patched, Nov 2017) -> 1.5.0 * llvm: 9.0.0 -> 11.0.0 * iwyu: 0.13 -> 0.14 * -nvml: 1.1 (2016) -> 1.6 (now called pmdk, Mar 2019)- ** -Patch to replace with memkind is posted- * -boost: 1.61.0 (patched, 2016) -> 1.74.0- * breakpad: 9eac2058b70615519b2c4d8c6bdbfca1bd079e39 (Apr 2013) -> 21b48a72aa50dde84149267f6b7402522b846b24 (Apr 2019) * sparsepp: 47a55825ca3b35eab1ca22b7ab82b9544e32a9af (Nov 2016) -> 5ca6de766db32b3fb08a040636423cd3988d2d4f (Jun 2018) * thrift: 0.11 (Dec 2017) -> 0.13 * -bison: 3.0.4 (patched, 2015) -> 3.5.4- * -hive: 498021fa15186aee8b282d3c032fbd2cede6bec4 (commit in Hive 2) -> 3.1.1 (Oct 2018)- * hadoop: 2.8.5 (Sept 2018) -> 3.2.0 * -sentry: 505b42e81a9d85c4ebe8db3f48ad7a6e824a5db5 (commit in Master)- * ranger: f37f5407eee8d2627a4306a25938b151f8e2ba31 -> 2.1.0 * python: 2.7.13 -> (a lot of choices here) * chrony: 3.5 -> 4.0 A quick risk/reward review should be done and we should upgrade the dependencies that are expected to be beneficial. > C++ Upgrades for before Kudu 1.13 release > - > > Key: KUDU-2817 > URL: https://issues.apache.org/jira/browse/KUDU-2817 > Project: Kudu > Issue Type: Improvement > Components: build >Affects Versions: 1.10.0 >Reporter: Grant Henke >Priority: Major > Fix For: 1.14.0 > > > We should consider reviewing and upgrading our dependencies before the next > release. Below is a list of current dependencies and their latest release. > * -gflags: 2.2.0 (Nov 2016) -> 2.2.2 (Nov 2018)- > * glog: 0.3.5 (May 2017) -> 0.4.0 (Mar 2019) > * gmock: 1.8.0 -> 1.8.1 > * -gperftools: 2.6.90 -> 2.8 > * -protobuf: 3.12.3 -> 3.14.0 > * -cmake: 3.16.4 -> 3.19.0 > * -snappy: 1.1.4 (Jan 2017) -> 1.1.8- > * -lz4: 1.9.2 -> 1.9.3 > * -bitshuffle: 55f9b4c (patched, 2016) -> 0.3.5 (Nov 2018)- > * -zlib: 1.2.8 (Apr 2013) -> 1.2.11 (Jan 2017)- > * libev: 4.20 -> 4.22 > * -rapidjson: 1.1.0 (current)- > * -squeasel: current- >
[jira] [Updated] (KUDU-2817) C++ Upgrades for before Kudu 1.13 release
[ https://issues.apache.org/jira/browse/KUDU-2817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Henke updated KUDU-2817: -- Description: We should consider reviewing and upgrading our dependencies before the next release. Below is a list of current dependencies and their latest release. * -gflags: 2.2.0 (Nov 2016) -> 2.2.2 (Nov 2018)- * glog: 0.3.5 (May 2017) -> 0.4.0 (Mar 2019) * gmock: 1.8.0 -> 1.8.1 * -gperftools: 2.6.90 -> 2.8- * -protobuf: 3.12.3 -> 3.14.0- * -cmake: 3.16.4 -> 3.19.0- * -snappy: 1.1.4 (Jan 2017) -> 1.1.8- * -lz4: 1.9.2 -> 1.9.3- * -bitshuffle: 55f9b4c (patched, 2016) -> 0.3.5 (Nov 2018)- * -zlib: 1.2.8 (Apr 2013) -> 1.2.11 (Jan 2017)- * libev: 4.20 -> 4.22 * -rapidjson: 1.1.0 (current)- * -squeasel: current- * mustache: 87a592e8aa04497764c533acd6e887618ca7b8a8 (Feb 2017) -> cf5c3dd499ea2bc9eb5c2072fb551dc7af75aa57 (Jun 2017) ** Consider using official mustach c++ support? * -curl: 7.59.0 (Mar 2018) -> 7.68.0- * -crcutil: current- * libunwind: 1.3-rc1 (patched, Nov 2017) -> 1.5.0 * llvm: 9.0.0 -> 11.0.0 * iwyu: 0.13 -> 0.14 * -nvml: 1.1 (2016) -> 1.6 (now called pmdk, Mar 2019)- ** -Patch to replace with memkind is posted- * -boost: 1.61.0 (patched, 2016) -> 1.74.0- * breakpad: 9eac2058b70615519b2c4d8c6bdbfca1bd079e39 (Apr 2013) -> 21b48a72aa50dde84149267f6b7402522b846b24 (Apr 2019) * sparsepp: 47a55825ca3b35eab1ca22b7ab82b9544e32a9af (Nov 2016) -> 5ca6de766db32b3fb08a040636423cd3988d2d4f (Jun 2018) * thrift: 0.11 (Dec 2017) -> 0.13 * -bison: 3.0.4 (patched, 2015) -> 3.5.4- * -hive: 498021fa15186aee8b282d3c032fbd2cede6bec4 (commit in Hive 2) -> 3.1.1 (Oct 2018)- * -hadoop: 2.8.5 (Sept 2018) -> 3.2.0- * -sentry: 505b42e81a9d85c4ebe8db3f48ad7a6e824a5db5 (commit in Master)- * -ranger: f37f5407eee8d2627a4306a25938b151f8e2ba31 -> 2.1.0- * python: 2.7.13 -> (a lot of choices here) * chrony: 3.5 -> 4.0 A quick risk/reward review should be done and we should upgrade the dependencies that are expected to be beneficial. was: We should consider reviewing and upgrading our dependencies before the next release. Below is a list of current dependencies and their latest release. * -gflags: 2.2.0 (Nov 2016) -> 2.2.2 (Nov 2018)- * glog: 0.3.5 (May 2017) -> 0.4.0 (Mar 2019) * gmock: 1.8.0 -> 1.8.1 * -gperftools: 2.6.90 -> 2.8 * -protobuf: 3.12.3 -> 3.14.0 * -cmake: 3.16.4 -> 3.19.0 * -snappy: 1.1.4 (Jan 2017) -> 1.1.8- * -lz4: 1.9.2 -> 1.9.3 * -bitshuffle: 55f9b4c (patched, 2016) -> 0.3.5 (Nov 2018)- * -zlib: 1.2.8 (Apr 2013) -> 1.2.11 (Jan 2017)- * libev: 4.20 -> 4.22 * -rapidjson: 1.1.0 (current)- * -squeasel: current- * mustache: 87a592e8aa04497764c533acd6e887618ca7b8a8 (Feb 2017) -> cf5c3dd499ea2bc9eb5c2072fb551dc7af75aa57 (Jun 2017) ** Consider using official mustach c++ support? * -curl: 7.59.0 (Mar 2018) -> 7.68.0- * -crcutil: current- * libunwind: 1.3-rc1 (patched, Nov 2017) -> 1.5.0 * llvm: 9.0.0 -> 11.0.0 * iwyu: 0.13 -> 0.14 * -nvml: 1.1 (2016) -> 1.6 (now called pmdk, Mar 2019)- ** -Patch to replace with memkind is posted- * -boost: 1.61.0 (patched, 2016) -> 1.74.0- * breakpad: 9eac2058b70615519b2c4d8c6bdbfca1bd079e39 (Apr 2013) -> 21b48a72aa50dde84149267f6b7402522b846b24 (Apr 2019) * sparsepp: 47a55825ca3b35eab1ca22b7ab82b9544e32a9af (Nov 2016) -> 5ca6de766db32b3fb08a040636423cd3988d2d4f (Jun 2018) * thrift: 0.11 (Dec 2017) -> 0.13 * -bison: 3.0.4 (patched, 2015) -> 3.5.4- * -hive: 498021fa15186aee8b282d3c032fbd2cede6bec4 (commit in Hive 2) -> 3.1.1 (Oct 2018)- * -hadoop: 2.8.5 (Sept 2018) -> 3.2.0 * -sentry: 505b42e81a9d85c4ebe8db3f48ad7a6e824a5db5 (commit in Master)- * -ranger: f37f5407eee8d2627a4306a25938b151f8e2ba31 -> 2.1.0 * python: 2.7.13 -> (a lot of choices here) * chrony: 3.5 -> 4.0 A quick risk/reward review should be done and we should upgrade the dependencies that are expected to be beneficial. > C++ Upgrades for before Kudu 1.13 release > - > > Key: KUDU-2817 > URL: https://issues.apache.org/jira/browse/KUDU-2817 > Project: Kudu > Issue Type: Improvement > Components: build >Affects Versions: 1.10.0 >Reporter: Grant Henke >Priority: Major > Fix For: 1.14.0 > > > We should consider reviewing and upgrading our dependencies before the next > release. Below is a list of current dependencies and their latest release. > * -gflags: 2.2.0 (Nov 2016) -> 2.2.2 (Nov 2018)- > * glog: 0.3.5 (May 2017) -> 0.4.0 (Mar 2019) > * gmock: 1.8.0 -> 1.8.1 > * -gperftools: 2.6.90 -> 2.8- > * -protobuf: 3.12.3 -> 3.14.0- > * -cmake: 3.16.4 -> 3.19.0- > * -snappy: 1.1.4 (Jan 2017) -> 1.1.8- > * -lz4: 1.9.2 -> 1.9.3- > * -bitshuffle: 55f9b4c (patched, 2016) -> 0.3.5 (Nov 2018)- > * -zlib: 1.2.8 (Apr 2013) -> 1.2.11 (Jan 2017)- > * libev: 4.20 -> 4.22 > * -rapidjson: 1.1.0 (current)- > *
[jira] [Resolved] (KUDU-2817) C++ Upgrades for before Kudu 1.13 release
[ https://issues.apache.org/jira/browse/KUDU-2817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Henke resolved KUDU-2817. --- Fix Version/s: 1.14.0 Resolution: Fixed Resolving as many of these upgrades have occurred across the 1.12, 1.13, and 1.14 releases. > C++ Upgrades for before Kudu 1.13 release > - > > Key: KUDU-2817 > URL: https://issues.apache.org/jira/browse/KUDU-2817 > Project: Kudu > Issue Type: Improvement > Components: build >Affects Versions: 1.10.0 >Reporter: Grant Henke >Priority: Major > Fix For: 1.14.0 > > > We should consider reviewing and upgrading our dependencies before the next > release. Below is a list of current dependencies and their latest release. > * -gflags: 2.2.0 (Nov 2016) -> 2.2.2 (Nov 2018)- > * glog: 0.3.5 (May 2017) -> 0.4.0 (Mar 2019) > * gmock: 1.8.0 -> 1.8.1 > * gperftools: 2.6.90 -> 2.8 > * protobuf: 3.12.3 -> 3.14.0 > * cmake: 3.16.4 -> 3.19.0 > * -snappy: 1.1.4 (Jan 2017) -> 1.1.8- > * lz4: 1.9.2 -> 1.9.3 > * -bitshuffle: 55f9b4c (patched, 2016) -> 0.3.5 (Nov 2018)- > * -zlib: 1.2.8 (Apr 2013) -> 1.2.11 (Jan 2017)- > * libev: 4.20 -> 4.22 > * -rapidjson: 1.1.0 (current)- > * -squeasel: current- > * mustache: 87a592e8aa04497764c533acd6e887618ca7b8a8 (Feb 2017) -> > cf5c3dd499ea2bc9eb5c2072fb551dc7af75aa57 (Jun 2017) > ** Consider using official mustach c++ support? > * -curl: 7.59.0 (Mar 2018) -> 7.68.0- > * -crcutil: current- > * libunwind: 1.3-rc1 (patched, Nov 2017) -> 1.5.0 > * llvm: 9.0.0 -> 11.0.0 > * iwyu: 0.13 -> 0.14 > * -nvml: 1.1 (2016) -> 1.6 (now called pmdk, Mar 2019)- > ** -Patch to replace with memkind is posted- > * -boost: 1.61.0 (patched, 2016) -> 1.74.0- > * breakpad: 9eac2058b70615519b2c4d8c6bdbfca1bd079e39 (Apr 2013) -> > 21b48a72aa50dde84149267f6b7402522b846b24 (Apr 2019) > * sparsepp: 47a55825ca3b35eab1ca22b7ab82b9544e32a9af (Nov 2016) -> > 5ca6de766db32b3fb08a040636423cd3988d2d4f (Jun 2018) > * thrift: 0.11 (Dec 2017) -> 0.13 > * -bison: 3.0.4 (patched, 2015) -> 3.5.4- > * -hive: 498021fa15186aee8b282d3c032fbd2cede6bec4 (commit in Hive 2) -> > 3.1.1 (Oct 2018)- > * hadoop: 2.8.5 (Sept 2018) -> 3.2.0 > * -sentry: 505b42e81a9d85c4ebe8db3f48ad7a6e824a5db5 (commit in Master)- > * ranger: f37f5407eee8d2627a4306a25938b151f8e2ba31 -> 2.1.0 > * python: 2.7.13 -> (a lot of choices here) > * chrony: 3.5 -> 4.0 > A quick risk/reward review should be done and we should upgrade the > dependencies that are expected to be beneficial. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KUDU-2038) Add b-tree or inverted index on value field
[ https://issues.apache.org/jira/browse/KUDU-2038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Henke updated KUDU-2038: -- Labels: performance roadmap-candidate (was: roadmap-candidate) > Add b-tree or inverted index on value field > --- > > Key: KUDU-2038 > URL: https://issues.apache.org/jira/browse/KUDU-2038 > Project: Kudu > Issue Type: Task >Reporter: Yi Guolei >Priority: Major > Labels: performance, roadmap-candidate > > Do we have a plan to add index on any column [not primary column] ? Currently > kudu does not have btree or inverted index on columns. In this case if a > query wants to filter a column then kudu has to scan all datas in all > rowsets. > For example, select * from table where salary > 1 and age < 40, the bloom > filter or min max index will have no effect, kudu has to scan all datas in > all row sets. But if kudu has inverted index, then it will be much faster. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KUDU-3147) Balance tablets based on range hash buckets
[ https://issues.apache.org/jira/browse/KUDU-3147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Henke updated KUDU-3147: -- Labels: balancer performance roadmap-candidate (was: balancer perf roadmap-candidate) > Balance tablets based on range hash buckets > --- > > Key: KUDU-3147 > URL: https://issues.apache.org/jira/browse/KUDU-3147 > Project: Kudu > Issue Type: Improvement > Components: master, perf >Affects Versions: 1.12.0 >Reporter: Grant Henke >Assignee: Ravi Bhanot >Priority: Major > Labels: balancer, performance, roadmap-candidate > > When a user defines a schema that uses range + hash partitioning its is often > the case that the tablets in the latest range, based on time or any > semi-sequential data, are the only tablets that receive writes. Or even if > not the latest, it is common for a single range to receive a burst of writes > if backloading. > This is so common, that the default Kudu balancing scheme should consider > placing/rebalancing the tablets for the hash buckets within each range on as > many servers as possible in order to support the maximum write throughput. In > that case, `min(#buckets, #total-cluster-tservers)` tservers will be used to > handle the writes if the cluster is perfectly balanced. Today, even if > perfectly balanced, it is possible for all the hash buckets to be on a single > tserver. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KUDU-3133) Poor TLS cypher performance on Java 8
[
https://issues.apache.org/jira/browse/KUDU-3133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Grant Henke updated KUDU-3133:
--
Labels: performance (was: )
> Poor TLS cypher performance on Java 8
> -
>
> Key: KUDU-3133
> URL: https://issues.apache.org/jira/browse/KUDU-3133
> Project: Kudu
> Issue Type: Bug
> Components: perf, security
>Affects Versions: 1.7.0
>Reporter: Grant Henke
>Assignee: wangningito
>Priority: Major
> Labels: performance
>
> It was reported a while back that Kudu TLS doesn't perform well on Java 8 due
> to a potential GCM cypher bug or bad selection via `PREFERRED_CIPHER_SUITES`.
> It would be good to get to the bottom of this and fix it or document the
> recommendation to use Java 11.
> Here was the observed impact:
> {code}
> ./bin/ycsb run kudu -P workloads/workloadc -p operationcount=1
> -threads 64 -p kudu_num_clients=16 -s -p fieldlength=1 -p
> kudu_table_num_replicas=1
> java 7u75 with master:
> 0205 11:18:48.647920 (+28us) server_negotiation.cc:581] Negotiated
> TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1
> ~12k rows/sec
> java 8_141 with master:
> 0205 11:17:45.977107 (+31us) server_negotiation.cc:581] Negotiated
> TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 Kx=ECDH Au=RSA
> Enc=AESGCM(256) Mac=AEAD
> 6k rows/sec
> java 8_141 with GCM-based codecs removed from Negotiator.java
> 0205 11:25:33.268689 (+40us) server_negotiation.cc:581] Negotiated
> TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 TLSv1.2 Kx=ECDH Au=RSA
> Enc=AES(256) Mac=SHA384
> ~6k rows/sec
> java 8_141 with only AES256-SHA remaining in Negotiator.java:
> "TLS_RSA_WITH_AES_256_CBC_SHA" )
> 0205 11:32:07.674860 (+44us) server_negotiation.cc:581] Negotiated
> TLSv1.2 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1
> ~9.5k rows/sec
> java 11 with master:
> 0205 11:17:01.416066 (+41us) server_negotiation.cc:581] Negotiated
> TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 Kx=ECDH Au=RSA
> Enc=AESGCM(256) Mac=AEAD
> ~19k rows/sec
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (KUDU-3133) Poor TLS cypher performance on Java 8
[
https://issues.apache.org/jira/browse/KUDU-3133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Grant Henke updated KUDU-3133:
--
Affects Version/s: 1.7.0
> Poor TLS cypher performance on Java 8
> -
>
> Key: KUDU-3133
> URL: https://issues.apache.org/jira/browse/KUDU-3133
> Project: Kudu
> Issue Type: Bug
> Components: perf, security
>Affects Versions: 1.7.0
>Reporter: Grant Henke
>Assignee: wangningito
>Priority: Major
>
> It was reported a while back that Kudu TLS doesn't perform well on Java 8 due
> to a potential GCM cypher bug or bad selection via `PREFERRED_CIPHER_SUITES`.
> It would be good to get to the bottom of this and fix it or document the
> recommendation to use Java 11.
> Here was the observed impact:
> {code}
> ./bin/ycsb run kudu -P workloads/workloadc -p operationcount=1
> -threads 64 -p kudu_num_clients=16 -s -p fieldlength=1 -p
> kudu_table_num_replicas=1
> java 7u75 with master:
> 0205 11:18:48.647920 (+28us) server_negotiation.cc:581] Negotiated
> TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1
> ~12k rows/sec
> java 8_141 with master:
> 0205 11:17:45.977107 (+31us) server_negotiation.cc:581] Negotiated
> TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 Kx=ECDH Au=RSA
> Enc=AESGCM(256) Mac=AEAD
> 6k rows/sec
> java 8_141 with GCM-based codecs removed from Negotiator.java
> 0205 11:25:33.268689 (+40us) server_negotiation.cc:581] Negotiated
> TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 TLSv1.2 Kx=ECDH Au=RSA
> Enc=AES(256) Mac=SHA384
> ~6k rows/sec
> java 8_141 with only AES256-SHA remaining in Negotiator.java:
> "TLS_RSA_WITH_AES_256_CBC_SHA" )
> 0205 11:32:07.674860 (+44us) server_negotiation.cc:581] Negotiated
> TLSv1.2 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1
> ~9.5k rows/sec
> java 11 with master:
> 0205 11:17:01.416066 (+41us) server_negotiation.cc:581] Negotiated
> TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 Kx=ECDH Au=RSA
> Enc=AESGCM(256) Mac=AEAD
> ~19k rows/sec
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (KUDU-613) Scan-resistant cache replacement algorithm for the block cache
[ https://issues.apache.org/jira/browse/KUDU-613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Henke updated KUDU-613: - Labels: performance roadmap-candidate (was: roadmap-candidate) > Scan-resistant cache replacement algorithm for the block cache > -- > > Key: KUDU-613 > URL: https://issues.apache.org/jira/browse/KUDU-613 > Project: Kudu > Issue Type: Improvement > Components: perf >Affects Versions: M4.5 >Reporter: Andrew Wang >Priority: Major > Labels: performance, roadmap-candidate > > The block cache currently uses LRU, which is vulnerable to large scan > workloads. It'd be good to implement something like 2Q. > ARC (patent encumbered, but good for ideas): > https://www.usenix.org/conference/fast-03/arc-self-tuning-low-overhead-replacement-cache > HBase (2Q like): > https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KUDU-3221) Add Bloom filter predicate push down support in Kudu Java client
[ https://issues.apache.org/jira/browse/KUDU-3221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Henke updated KUDU-3221: -- Labels: roadmap-candidate (was: ) > Add Bloom filter predicate push down support in Kudu Java client > > > Key: KUDU-3221 > URL: https://issues.apache.org/jira/browse/KUDU-3221 > Project: Kudu > Issue Type: Task > Components: client, java >Affects Versions: 1.13.0 >Reporter: Bankim Bhavsar >Priority: Major > Labels: roadmap-candidate > > We added support for predicate pushdown with Bloom filter predicate in 1.13. > See KUDU-2483. > However instead of using the existing Bloom filter in Kudu which is used > internally for index look ups, we decided to import the Cache, Hash and Space > efficient Bloom filter from Impala which was written in C++. > We need to create a binary compatible Bloom filter in Java to be push down > the Bloom filter predicate to Kudu servers from Java clients. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KUDU-3222) std::bad_alloc in full_stack-insert-scan-test.cc
[
https://issues.apache.org/jira/browse/KUDU-3222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Grant Henke resolved KUDU-3222.
---
Fix Version/s: 1.14.0
Resolution: Fixed
I think this has been resolved by the follow on gperftools upgrade.
> std::bad_alloc in full_stack-insert-scan-test.cc
>
>
> Key: KUDU-3222
> URL: https://issues.apache.org/jira/browse/KUDU-3222
> Project: Kudu
> Issue Type: Bug
> Components: test
>Reporter: Andrew Wong
>Priority: Major
> Fix For: 1.14.0
>
> Attachments: FullStackScanInsertMRSOnly3.log
>
>
> Recently we've been starting to see the following in our runs of
> full_stack-insert-scan-test:
> {code:java}
> I1214 13:30:32.995853 39072 full_stack-insert-scan-test.cc:271] Insertion
> thread 7 of 50 is 69% done.
> terminate called after throwing an instance of 'std::bad_alloc'
> what(): std::bad_alloc
> *** Aborted at 1607981433 (unix time) try "date -d @1607981433" if you are
> using GNU date ***
> PC: @ 0x3f85032625 __GI_raise
> *** SIGABRT (@0x11569802) received by PID 38914 (TID 0x7f81b4a02700) from
> PID 38914; stack trace: ***
> @ 0xcf8a21 google::(anonymous namespace)::FailureSignalHandler()
> @ 0x3f8540f710 (unknown)
> @ 0x3f85032625 __GI_raise
> @ 0x3f85033e05 __GI_abort
> @ 0x3f884bea7d __gnu_cxx::__verbose_terminate_handler()
> @ 0x3f884bcbd6 (unknown)
> @ 0x3f884bcc03 std::terminate()
> @ 0x3f884bcd22 __cxa_throw
> @ 0xd14bd5 (anonymous namespace)::handle_oom()
> @ 0x2ff3872 tcmalloc::allocate_full_cpp_throw_oom()
> @ 0x2cd4c1a
> _ZNSt6vectorIN4kudu19DecodedRowOperationESaIS1_EE17_M_realloc_insertIJRKS1_EEEvN9__gnu_cxx17__normal_iteratorIPS1_S3_EEDpOT_
> @ 0x2cd535a kudu::RowOperationsPBDecoder::DecodeOperations<>()
> @ 0x131c2a6 kudu::tablet::Tablet::DecodeWriteOperations()
> I1214 13:30:33.075912 39094 full_stack-insert-scan-test.cc:271] Insertion
> thread 29 of 50 is 69% done.
> @ 0x135bcb6 kudu::tablet::WriteOp::Prepare()
> @ 0x13514ac kudu::tablet::OpDriver::Prepare()
> @ 0x13520ed
> _ZNSt17_Function_handlerIFvvEZN4kudu6tablet8OpDriver12ExecuteAsyncEvEUlvE_E9_M_invokeERKSt9_Any_data
> @ 0x2e2409e kudu::ThreadPool::DispatchThread()
> @ 0x2e1b2c5 kudu::Thread::SuperviseThread()
> @ 0x3f854079d1 start_thread
> @ 0x3f850e88fd clone {code}
> This runs as a part of a suite of several tests in {{scripts/benchmarks.sh}}.
> This started happening fairly consistently starting around commit
> 2943aa701ee092158c2084c614a91f92513ef7c4, when we bumped glog and gperftools,
> though I'm not sure they are directly related here.
> The attached logs are a run on CentOS 6.6, with around 100GB of memory.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (KUDU-1563) Add support for INSERT/UPDATE/DELETE IGNORE
[ https://issues.apache.org/jira/browse/KUDU-1563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Henke resolved KUDU-1563. --- Fix Version/s: 1.14.0 Resolution: Fixed > Add support for INSERT/UPDATE/DELETE IGNORE > --- > > Key: KUDU-1563 > URL: https://issues.apache.org/jira/browse/KUDU-1563 > Project: Kudu > Issue Type: New Feature >Reporter: Dan Burkert >Assignee: Grant Henke >Priority: Major > Labels: backup, impala, roadmap-candidate > Fix For: 1.14.0 > > > The Java client currently has an [option to ignore duplicate row key errors| > https://kudu.apache.org/apidocs/org/kududb/client/AsyncKuduSession.html#setIgnoreAllDuplicateRows-boolean-], > which is implemented by filtering the errors on the client side. If we are > going to continue to support this feature (and the consensus seems to be that > we probably should), we should promote it to a first class operation type > that is handled on the server side. This would have a modest perf. > improvement since less errors are returned, and it would allow INSERT IGNORE > ops to be mixed in the same batch as other INSERT, DELETE, UPSERT, etc. ops. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KUDU-3202) Add Spark 3 Support
[ https://issues.apache.org/jira/browse/KUDU-3202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Henke resolved KUDU-3202. --- Fix Version/s: 1.14.0 Resolution: Fixed > Add Spark 3 Support > --- > > Key: KUDU-3202 > URL: https://issues.apache.org/jira/browse/KUDU-3202 > Project: Kudu > Issue Type: Improvement > Components: spark >Affects Versions: 1.13.0 >Reporter: Grant Henke >Assignee: Grant Henke >Priority: Major > Fix For: 1.14.0 > > > Now that Spark 3 is released we should build and publish the Spark related > jars using Spark 3 while maintaining cross-build support for Spark 2. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KUDU-3219) Incremental Restore Feature
Grant Henke created KUDU-3219: - Summary: Incremental Restore Feature Key: KUDU-3219 URL: https://issues.apache.org/jira/browse/KUDU-3219 Project: Kudu Issue Type: Improvement Components: backup Affects Versions: 1.13.0 Reporter: Grant Henke Currently Kudu supports incremental backup but it doesn't support incremental restore. This is because we can't validate the state of the target tables has gone unchanged since the full backup was applied so the only "safe" thing to do is to apply a full restore plus all the incrementals. It would be nice to add some validation of table state, but a good first step might be to provide the mechanism for incremental restore and document the limitation. Today in order to restore only new incrementals you would need to set `createTables=false` and a `rootPath` that only contains the incrementals you want to apply. That is more difficult than necessary because it requires alteration to the backup job or manually dissecting the backup root directory structure. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KUDU-3218) client_symbol-test fails on Centos 7 with devtoolset-8
Grant Henke created KUDU-3218:
-
Summary: client_symbol-test fails on Centos 7 with devtoolset-8
Key: KUDU-3218
URL: https://issues.apache.org/jira/browse/KUDU-3218
Project: Kudu
Issue Type: Bug
Components: client
Affects Versions: 1.14.0
Reporter: Grant Henke
Assignee: Grant Henke
When running the client_symbol-test on Centos 7 with devtoolset-8 the test
fails with the following bad symbols:
{code:java}
Found bad symbol 'operator delete[](void*, unsigned long)'
Found bad symbol 'operator delete(void*, unsigned long)'
Found bad symbol 'transaction clone for std::logic_error::what() const'
Found bad symbol 'transaction clone for std::runtime_error::what() const'
Found bad symbol 'transaction clone for std::logic_error::logic_error(char
const*)'
Found bad symbol 'transaction clone for
std::logic_error::logic_error(std::__cxx11::basic_string, std::allocator > const&)'
Found bad symbol 'transaction clone for std::logic_error::logic_error(char
const*)'
Found bad symbol 'transaction clone for
std::logic_error::logic_error(std::__cxx11::basic_string, std::allocator > const&)'
Found bad symbol 'transaction clone for std::logic_error::~logic_error()'
Found bad symbol 'transaction clone for std::logic_error::~logic_error()'
Found bad symbol 'transaction clone for std::logic_error::~logic_error()'
Found bad symbol 'transaction clone for std::range_error::range_error(char
const*)'
Found bad symbol 'transaction clone for
std::range_error::range_error(std::__cxx11::basic_string, std::allocator > const&)'
Found bad symbol 'transaction clone for std::range_error::range_error(char
const*)'
Found bad symbol 'transaction clone for
std::range_error::range_error(std::__cxx11::basic_string, std::allocator > const&)'
Found bad symbol 'transaction clone for std::range_error::~range_error()'
Found bad symbol 'transaction clone for std::range_error::~range_error()'
Found bad symbol 'transaction clone for std::range_error::~range_error()'
Found bad symbol 'transaction clone for std::domain_error::domain_error(char
const*)'
Found bad symbol 'transaction clone for
std::domain_error::domain_error(std::__cxx11::basic_string, std::allocator > const&)'
Found bad symbol 'transaction clone for std::domain_error::domain_error(char
const*)'
Found bad symbol 'transaction clone for
std::domain_error::domain_error(std::__cxx11::basic_string, std::allocator > const&)'
Found bad symbol 'transaction clone for std::domain_error::~domain_error()'
Found bad symbol 'transaction clone for std::domain_error::~domain_error()'
Found bad symbol 'transaction clone for std::domain_error::~domain_error()'
Found bad symbol 'transaction clone for std::length_error::length_error(char
const*)'
Found bad symbol 'transaction clone for
std::length_error::length_error(std::__cxx11::basic_string, std::allocator > const&)'
Found bad symbol 'transaction clone for std::length_error::length_error(char
const*)'
Found bad symbol 'transaction clone for
std::length_error::length_error(std::__cxx11::basic_string, std::allocator > const&)'
Found bad symbol 'transaction clone for std::length_error::~length_error()'
Found bad symbol 'transaction clone for std::length_error::~length_error()'
Found bad symbol 'transaction clone for std::length_error::~length_error()'
Found bad symbol 'transaction clone for std::out_of_range::out_of_range(char
const*)'
Found bad symbol 'transaction clone for
std::out_of_range::out_of_range(std::__cxx11::basic_string, std::allocator > const&)'
Found bad symbol 'transaction clone for std::out_of_range::out_of_range(char
const*)'
Found bad symbol 'transaction clone for
std::out_of_range::out_of_range(std::__cxx11::basic_string, std::allocator > const&)'
Found bad symbol 'transaction clone for std::out_of_range::~out_of_range()'
Found bad symbol 'transaction clone for std::out_of_range::~out_of_range()'
Found bad symbol 'transaction clone for std::out_of_range::~out_of_range()'
Found bad symbol 'transaction clone for std::runtime_error::runtime_error(char
const*)'
Found bad symbol 'transaction clone for
std::runtime_error::runtime_error(std::__cxx11::basic_string, std::allocator > const&)'
Found bad symbol 'transaction clone for std::runtime_error::runtime_error(char
const*)'
Found bad symbol 'transaction clone for
std::runtime_error::runtime_error(std::__cxx11::basic_string, std::allocator > const&)'
Found bad symbol 'transaction clone for std::runtime_error::~runtime_error()'
Found bad symbol 'transaction clone for std::runtime_error::~runtime_error()'
Found bad symbol 'transaction clone for std::runtime_error::~runtime_error()'
Found bad symbol 'transaction clone for
std::overflow_error::overflow_error(char const*)'
Found bad symbol 'transaction clone for
std::overflow_error::overflow_error(std::__cxx11::basic_string, std::allocator > const&)'
Found bad symbol 'transaction clone
[jira] [Assigned] (KUDU-3147) Balance tablets based on range hash buckets
[ https://issues.apache.org/jira/browse/KUDU-3147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Henke reassigned KUDU-3147: - Assignee: Ravi Bhanot > Balance tablets based on range hash buckets > --- > > Key: KUDU-3147 > URL: https://issues.apache.org/jira/browse/KUDU-3147 > Project: Kudu > Issue Type: Improvement > Components: master, perf >Affects Versions: 1.12.0 >Reporter: Grant Henke >Assignee: Ravi Bhanot >Priority: Major > Labels: balancer, perf, roadmap-candidate > > When a user defines a schema that uses range + hash partitioning its is often > the case that the tablets in the latest range, based on time or any > semi-sequential data, are the only tablets that receive writes. Or even if > not the latest, it is common for a single range to receive a burst of writes > if backloading. > This is so common, that the default Kudu balancing scheme should consider > placing/rebalancing the tablets for the hash buckets within each range on as > many servers as possible in order to support the maximum write throughput. In > that case, `min(#buckets, #total-cluster-tservers)` tservers will be used to > handle the writes if the cluster is perfectly balanced. Today, even if > perfectly balanced, it is possible for all the hash buckets to be on a single > tserver. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KUDU-3147) Balance tablets based on range hash buckets
[ https://issues.apache.org/jira/browse/KUDU-3147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17241951#comment-17241951 ] Grant Henke commented on KUDU-3147: --- Absolutely. Please do pick it up. > Balance tablets based on range hash buckets > --- > > Key: KUDU-3147 > URL: https://issues.apache.org/jira/browse/KUDU-3147 > Project: Kudu > Issue Type: Improvement > Components: master, perf >Affects Versions: 1.12.0 >Reporter: Grant Henke >Priority: Major > Labels: balancer, perf, roadmap-candidate > > When a user defines a schema that uses range + hash partitioning its is often > the case that the tablets in the latest range, based on time or any > semi-sequential data, are the only tablets that receive writes. Or even if > not the latest, it is common for a single range to receive a burst of writes > if backloading. > This is so common, that the default Kudu balancing scheme should consider > placing/rebalancing the tablets for the hash buckets within each range on as > many servers as possible in order to support the maximum write throughput. In > that case, `min(#buckets, #total-cluster-tservers)` tservers will be used to > handle the writes if the cluster is perfectly balanced. Today, even if > perfectly balanced, it is possible for all the hash buckets to be on a single > tserver. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KUDU-2817) C++ Upgrades for before Kudu 1.13 release
[ https://issues.apache.org/jira/browse/KUDU-2817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Henke updated KUDU-2817: -- Description: We should consider reviewing and upgrading our dependencies before the next release. Below is a list of current dependencies and their latest release. * -gflags: 2.2.0 (Nov 2016) -> 2.2.2 (Nov 2018)- * glog: 0.3.5 (May 2017) -> 0.4.0 (Mar 2019) * gmock: 1.8.0 -> 1.8.1 * gperftools: 2.6.90 -> 2.8 * protobuf: 3.12.3 -> 3.14.0 * cmake: 3.16.4 -> 3.19.0 * -snappy: 1.1.4 (Jan 2017) -> 1.1.8- * lz4: 1.9.2 -> 1.9.3 * -bitshuffle: 55f9b4c (patched, 2016) -> 0.3.5 (Nov 2018)- * -zlib: 1.2.8 (Apr 2013) -> 1.2.11 (Jan 2017)- * libev: 4.20 -> 4.22 * -rapidjson: 1.1.0 (current)- * -squeasel: current- * mustache: 87a592e8aa04497764c533acd6e887618ca7b8a8 (Feb 2017) -> cf5c3dd499ea2bc9eb5c2072fb551dc7af75aa57 (Jun 2017) ** Consider using official mustach c++ support? * -curl: 7.59.0 (Mar 2018) -> 7.68.0- * -crcutil: current- * libunwind: 1.3-rc1 (patched, Nov 2017) -> 1.5.0 * llvm: 9.0.0 -> 11.0.0 * iwyu: 0.13 -> 0.14 * -nvml: 1.1 (2016) -> 1.6 (now called pmdk, Mar 2019)- ** -Patch to replace with memkind is posted- * -boost: 1.61.0 (patched, 2016) -> 1.74.0- * breakpad: 9eac2058b70615519b2c4d8c6bdbfca1bd079e39 (Apr 2013) -> 21b48a72aa50dde84149267f6b7402522b846b24 (Apr 2019) * sparsepp: 47a55825ca3b35eab1ca22b7ab82b9544e32a9af (Nov 2016) -> 5ca6de766db32b3fb08a040636423cd3988d2d4f (Jun 2018) * thrift: 0.11 (Dec 2017) -> 0.13 * -bison: 3.0.4 (patched, 2015) -> 3.5.4- * -hive: 498021fa15186aee8b282d3c032fbd2cede6bec4 (commit in Hive 2) -> 3.1.1 (Oct 2018)- * hadoop: 2.8.5 (Sept 2018) -> 3.2.0 * -sentry: 505b42e81a9d85c4ebe8db3f48ad7a6e824a5db5 (commit in Master)- * ranger: f37f5407eee8d2627a4306a25938b151f8e2ba31 -> 2.1.0 * python: 2.7.13 -> (a lot of choices here) * chrony: 3.5 -> 4.0 A quick risk/reward review should be done and we should upgrade the dependencies that are expected to be beneficial. was: We should consider reviewing and upgrading our dependencies before the next release. Below is a list of current dependencies and their latest release. * -gflags: 2.2.0 (Nov 2016) -> 2.2.2 (Nov 2018)- * glog: 0.3.5 (May 2017) -> 0.4.0 (Mar 2019) * gmock: 1.8.0 -> 1.8.1 * gperftools: 2.6.90 -> 2.8 * protobuf: 3.12.3 -> 3.14.0 * cmake: 3.16.4 -> 3.19.0 * -snappy: 1.1.4 (Jan 2017) -> 1.1.8- * -lz4: r130 (patched, 2015) -> 1.9.2- * -bitshuffle: 55f9b4c (patched, 2016) -> 0.3.5 (Nov 2018)- * -zlib: 1.2.8 (Apr 2013) -> 1.2.11 (Jan 2017)- * libev: 4.20 -> 4.22 * rapidjson: 1.1.0 (current) * squeasel: current * mustache: 87a592e8aa04497764c533acd6e887618ca7b8a8 (Feb 2017) -> cf5c3dd499ea2bc9eb5c2072fb551dc7af75aa57 (Jun 2017) ** Consider using official mustach c++ support? * curl: 7.59.0 (Mar 2018) -> 7.64.1 (Mar 2019) * crcutil: current * libunwind: 1.3-rc1 (patched, Nov 2017) -> 1.3.1 (Jan 2019) * llvm: 6.0.0 (Mar 2018) -> 8.0.0 (Mar 2019) * iwyu: 0.9 -> 0.12 (May 2019) * nvml: 1.1 (2016) -> 1.6 (now called pmdk, Mar 2019) ** Patch to replace with memkind is posted * -boost: 1.61.0 (patched, 2016) -> 1.74.0- * breakpad: 9eac2058b70615519b2c4d8c6bdbfca1bd079e39 (Apr 2013) -> 21b48a72aa50dde84149267f6b7402522b846b24 (Apr 2019) * sparsepp: 47a55825ca3b35eab1ca22b7ab82b9544e32a9af (Nov 2016) -> 5ca6de766db32b3fb08a040636423cd3988d2d4f (Jun 2018) * thrift: 0.11 (Dec 2017) -> 0.12 (Dec 2018) * -bison: 3.0.4 (patched, 2015) -> 3.5.4- * hive: 498021fa15186aee8b282d3c032fbd2cede6bec4 (commit in Hive 2) -> 3.1.1 (Oct 2018) * hadoop: 2.8.5 (Sept 2018) -> 3.1.2 (Feb 2019) * sentry: 505b42e81a9d85c4ebe8db3f48ad7a6e824a5db5 (commit in Master) * python: 2.7.13 -> (a lot of choices here) A quick risk/reward review should be done and we should upgrade the dependencies that are expected to be beneficial. > C++ Upgrades for before Kudu 1.13 release > - > > Key: KUDU-2817 > URL: https://issues.apache.org/jira/browse/KUDU-2817 > Project: Kudu > Issue Type: Improvement > Components: build >Affects Versions: 1.10.0 >Reporter: Grant Henke >Priority: Major > > We should consider reviewing and upgrading our dependencies before the next > release. Below is a list of current dependencies and their latest release. > * -gflags: 2.2.0 (Nov 2016) -> 2.2.2 (Nov 2018)- > * glog: 0.3.5 (May 2017) -> 0.4.0 (Mar 2019) > * gmock: 1.8.0 -> 1.8.1 > * gperftools: 2.6.90 -> 2.8 > * protobuf: 3.12.3 -> 3.14.0 > * cmake: 3.16.4 -> 3.19.0 > * -snappy: 1.1.4 (Jan 2017) -> 1.1.8- > * lz4: 1.9.2 -> 1.9.3 > * -bitshuffle: 55f9b4c (patched, 2016) -> 0.3.5 (Nov 2018)- > * -zlib: 1.2.8 (Apr 2013) -> 1.2.11 (Jan 2017)- > * libev: 4.20 -> 4.22 > * -rapidjson: 1.1.0 (current)- > * -squeasel: current- > * mustache:
[jira] [Updated] (KUDU-2817) C++ Upgrades for before Kudu 1.13 release
[ https://issues.apache.org/jira/browse/KUDU-2817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Henke updated KUDU-2817: -- Description: We should consider reviewing and upgrading our dependencies before the next release. Below is a list of current dependencies and their latest release. * -gflags: 2.2.0 (Nov 2016) -> 2.2.2 (Nov 2018)- * glog: 0.3.5 (May 2017) -> 0.4.0 (Mar 2019) * gmock: 1.8.0 -> 1.8.1 * gperftools: 2.6.90 -> 2.8 * protobuf: 3.12.3 -> 3.14.0 * cmake: 3.16.4 -> 3.19.0 * -snappy: 1.1.4 (Jan 2017) -> 1.1.8- * -lz4: r130 (patched, 2015) -> 1.9.2- * -bitshuffle: 55f9b4c (patched, 2016) -> 0.3.5 (Nov 2018)- * -zlib: 1.2.8 (Apr 2013) -> 1.2.11 (Jan 2017)- * libev: 4.20 -> 4.22 * rapidjson: 1.1.0 (current) * squeasel: current * mustache: 87a592e8aa04497764c533acd6e887618ca7b8a8 (Feb 2017) -> cf5c3dd499ea2bc9eb5c2072fb551dc7af75aa57 (Jun 2017) ** Consider using official mustach c++ support? * curl: 7.59.0 (Mar 2018) -> 7.64.1 (Mar 2019) * crcutil: current * libunwind: 1.3-rc1 (patched, Nov 2017) -> 1.3.1 (Jan 2019) * llvm: 6.0.0 (Mar 2018) -> 8.0.0 (Mar 2019) * iwyu: 0.9 -> 0.12 (May 2019) * nvml: 1.1 (2016) -> 1.6 (now called pmdk, Mar 2019) ** Patch to replace with memkind is posted * -boost: 1.61.0 (patched, 2016) -> 1.74.0- * breakpad: 9eac2058b70615519b2c4d8c6bdbfca1bd079e39 (Apr 2013) -> 21b48a72aa50dde84149267f6b7402522b846b24 (Apr 2019) * sparsepp: 47a55825ca3b35eab1ca22b7ab82b9544e32a9af (Nov 2016) -> 5ca6de766db32b3fb08a040636423cd3988d2d4f (Jun 2018) * thrift: 0.11 (Dec 2017) -> 0.12 (Dec 2018) * -bison: 3.0.4 (patched, 2015) -> 3.5.4- * hive: 498021fa15186aee8b282d3c032fbd2cede6bec4 (commit in Hive 2) -> 3.1.1 (Oct 2018) * hadoop: 2.8.5 (Sept 2018) -> 3.1.2 (Feb 2019) * sentry: 505b42e81a9d85c4ebe8db3f48ad7a6e824a5db5 (commit in Master) * python: 2.7.13 -> (a lot of choices here) A quick risk/reward review should be done and we should upgrade the dependencies that are expected to be beneficial. was: We should consider reviewing and upgrading our dependencies before the next release. Below is a list of current dependencies and their latest release. * gflags: 2.2.0 (Nov 2016) -> 2.2.2 (Nov 2018) * glog: 0.3.5 (May 2017) -> 0.4.0 (Mar 2019) * gmock: 1.8.0 -> 1.8.1 * gperftools: 2.6.90 -> 2.7 * protobuf: 3.4.1 -> 3.7.1 (3.8.0 soon) * cmake: 3.9.0 (Nov 2018) -> 3.14.3 (May 2019) * snappy: 1.1.4 (Jan 2017) -> 1.1.7 (Aug 2017) * lz4: r130 (patched, 2015) -> 1.9.1 (May 2019, expected perf gains) * bitshuffle: 55f9b4c (patched, 2016) -> 0.3.5 (Nov 2018) * zlib: 1.2.8 (Apr 2013) -> 1.2.11 (Jan 2017) * libev: 4.20 -> 4.22 * rapidjson: 1.1.0 (current) * squeasel: current * mustache: 87a592e8aa04497764c533acd6e887618ca7b8a8 (Feb 2017) -> cf5c3dd499ea2bc9eb5c2072fb551dc7af75aa57 (Jun 2017) ** Consider using official mustach c++ support? * curl: 7.59.0 (Mar 2018) -> 7.64.1 (Mar 2019) * crcutil: current * libunwind: 1.3-rc1 (patched, Nov 2017) -> 1.3.1 (Jan 2019) * llvm: 6.0.0 (Mar 2018) -> 8.0.0 (Mar 2019) * iwyu: 0.9 -> 0.12 (May 2019) * nvml: 1.1 (2016) -> 1.6 (now called pmdk, Mar 2019) ** Patch to replace with memkind is posted * -boost: 1.61.0 (patched, 2016) -> 1.74.0- * breakpad: 9eac2058b70615519b2c4d8c6bdbfca1bd079e39 (Apr 2013) -> 21b48a72aa50dde84149267f6b7402522b846b24 (Apr 2019) * sparsepp: 47a55825ca3b35eab1ca22b7ab82b9544e32a9af (Nov 2016) -> 5ca6de766db32b3fb08a040636423cd3988d2d4f (Jun 2018) * thrift: 0.11 (Dec 2017) -> 0.12 (Dec 2018) * bison: 3.0.4 (patched, 2015) -> 3.3 (Jan 2019) * hive: 498021fa15186aee8b282d3c032fbd2cede6bec4 (commit in Hive 2) -> 3.1.1 (Oct 2018) * hadoop: 2.8.5 (Sept 2018) -> 3.1.2 (Feb 2019) * sentry: 505b42e81a9d85c4ebe8db3f48ad7a6e824a5db5 (commit in Master) * python: 2.7.13 -> (a lot of choices here) A quick risk/reward review should be done and we should upgrade the dependencies that are expected to be beneficial. > C++ Upgrades for before Kudu 1.13 release > - > > Key: KUDU-2817 > URL: https://issues.apache.org/jira/browse/KUDU-2817 > Project: Kudu > Issue Type: Improvement > Components: build >Affects Versions: 1.10.0 >Reporter: Grant Henke >Priority: Major > > We should consider reviewing and upgrading our dependencies before the next > release. Below is a list of current dependencies and their latest release. > * -gflags: 2.2.0 (Nov 2016) -> 2.2.2 (Nov 2018)- > * glog: 0.3.5 (May 2017) -> 0.4.0 (Mar 2019) > * gmock: 1.8.0 -> 1.8.1 > * gperftools: 2.6.90 -> 2.8 > * protobuf: 3.12.3 -> 3.14.0 > * cmake: 3.16.4 -> 3.19.0 > * -snappy: 1.1.4 (Jan 2017) -> 1.1.8- > * -lz4: r130 (patched, 2015) -> 1.9.2- > * -bitshuffle: 55f9b4c (patched, 2016) -> 0.3.5 (Nov 2018)- > * -zlib: 1.2.8 (Apr 2013) -> 1.2.11 (Jan 2017)- > * libev: 4.20 -> 4.22 > * rapidjson: 1.1.0
[jira] [Updated] (KUDU-2817) C++ Upgrades for before Kudu 1.13 release
[ https://issues.apache.org/jira/browse/KUDU-2817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Henke updated KUDU-2817: -- Description: We should consider reviewing and upgrading our dependencies before the next release. Below is a list of current dependencies and their latest release. * gflags: 2.2.0 (Nov 2016) -> 2.2.2 (Nov 2018) * glog: 0.3.5 (May 2017) -> 0.4.0 (Mar 2019) * gmock: 1.8.0 -> 1.8.1 * gperftools: 2.6.90 -> 2.7 * protobuf: 3.4.1 -> 3.7.1 (3.8.0 soon) * cmake: 3.9.0 (Nov 2018) -> 3.14.3 (May 2019) * snappy: 1.1.4 (Jan 2017) -> 1.1.7 (Aug 2017) * lz4: r130 (patched, 2015) -> 1.9.1 (May 2019, expected perf gains) * bitshuffle: 55f9b4c (patched, 2016) -> 0.3.5 (Nov 2018) * zlib: 1.2.8 (Apr 2013) -> 1.2.11 (Jan 2017) * libev: 4.20 -> 4.22 * rapidjson: 1.1.0 (current) * squeasel: current * mustache: 87a592e8aa04497764c533acd6e887618ca7b8a8 (Feb 2017) -> cf5c3dd499ea2bc9eb5c2072fb551dc7af75aa57 (Jun 2017) ** Consider using official mustach c++ support? * curl: 7.59.0 (Mar 2018) -> 7.64.1 (Mar 2019) * crcutil: current * libunwind: 1.3-rc1 (patched, Nov 2017) -> 1.3.1 (Jan 2019) * llvm: 6.0.0 (Mar 2018) -> 8.0.0 (Mar 2019) * iwyu: 0.9 -> 0.12 (May 2019) * nvml: 1.1 (2016) -> 1.6 (now called pmdk, Mar 2019) ** Patch to replace with memkind is posted * -boost: 1.61.0 (patched, 2016) -> 1.74.0- * breakpad: 9eac2058b70615519b2c4d8c6bdbfca1bd079e39 (Apr 2013) -> 21b48a72aa50dde84149267f6b7402522b846b24 (Apr 2019) * sparsepp: 47a55825ca3b35eab1ca22b7ab82b9544e32a9af (Nov 2016) -> 5ca6de766db32b3fb08a040636423cd3988d2d4f (Jun 2018) * thrift: 0.11 (Dec 2017) -> 0.12 (Dec 2018) * bison: 3.0.4 (patched, 2015) -> 3.3 (Jan 2019) * hive: 498021fa15186aee8b282d3c032fbd2cede6bec4 (commit in Hive 2) -> 3.1.1 (Oct 2018) * hadoop: 2.8.5 (Sept 2018) -> 3.1.2 (Feb 2019) * sentry: 505b42e81a9d85c4ebe8db3f48ad7a6e824a5db5 (commit in Master) * python: 2.7.13 -> (a lot of choices here) A quick risk/reward review should be done and we should upgrade the dependencies that are expected to be beneficial. was: We should consider reviewing and upgrading our dependencies before the next release. Below is a list of current dependencies and their latest release. * gflags: 2.2.0 (Nov 2016) -> 2.2.2 (Nov 2018) * glog: 0.3.5 (May 2017) -> 0.4.0 (Mar 2019) * gmock: 1.8.0 -> 1.8.1 * gperftools: 2.6.90 -> 2.7 * protobuf: 3.4.1 -> 3.7.1 (3.8.0 soon) * cmake: 3.9.0 (Nov 2018) -> 3.14.3 (May 2019) * snappy: 1.1.4 (Jan 2017) -> 1.1.7 (Aug 2017) * lz4: r130 (patched, 2015) -> 1.9.1 (May 2019, expected perf gains) * bitshuffle: 55f9b4c (patched, 2016) -> 0.3.5 (Nov 2018) * zlib: 1.2.8 (Apr 2013) -> 1.2.11 (Jan 2017) * libev: 4.20 -> 4.22 * rapidjson: 1.1.0 (current) * squeasel: current * mustache: 87a592e8aa04497764c533acd6e887618ca7b8a8 (Feb 2017) -> cf5c3dd499ea2bc9eb5c2072fb551dc7af75aa57 (Jun 2017) ** Consider using official mustach c++ support? * curl: 7.59.0 (Mar 2018) -> 7.64.1 (Mar 2019) * crcutil: current * libunwind: 1.3-rc1 (patched, Nov 2017) -> 1.3.1 (Jan 2019) * llvm: 6.0.0 (Mar 2018) -> 8.0.0 (Mar 2019) * iwyu: 0.9 -> 0.12 (May 2019) * nvml: 1.1 (2016) -> 1.6 (now called pmdk, Mar 2019) ** Patch to replace with memkind is posted * boost: 1.61.0 (patched, 2016) -> 1.70.0 (Apr 2019) * breakpad: 9eac2058b70615519b2c4d8c6bdbfca1bd079e39 (Apr 2013) -> 21b48a72aa50dde84149267f6b7402522b846b24 (Apr 2019) * sparsepp: 47a55825ca3b35eab1ca22b7ab82b9544e32a9af (Nov 2016) -> 5ca6de766db32b3fb08a040636423cd3988d2d4f (Jun 2018) * thrift: 0.11 (Dec 2017) -> 0.12 (Dec 2018) * bison: 3.0.4 (patched, 2015) -> 3.3 (Jan 2019) * hive: 498021fa15186aee8b282d3c032fbd2cede6bec4 (commit in Hive 2) -> 3.1.1 (Oct 2018) * hadoop: 2.8.5 (Sept 2018) -> 3.1.2 (Feb 2019) * sentry: 505b42e81a9d85c4ebe8db3f48ad7a6e824a5db5 (commit in Master) * python: 2.7.13 -> (a lot of choices here) A quick risk/reward review should be done and we should upgrade the dependencies that are expected to be beneficial. > C++ Upgrades for before Kudu 1.13 release > - > > Key: KUDU-2817 > URL: https://issues.apache.org/jira/browse/KUDU-2817 > Project: Kudu > Issue Type: Improvement > Components: build >Affects Versions: 1.10.0 >Reporter: Grant Henke >Priority: Major > > We should consider reviewing and upgrading our dependencies before the next > release. Below is a list of current dependencies and their latest release. > * gflags: 2.2.0 (Nov 2016) -> 2.2.2 (Nov 2018) > * glog: 0.3.5 (May 2017) -> 0.4.0 (Mar 2019) > * gmock: 1.8.0 -> 1.8.1 > * gperftools: 2.6.90 -> 2.7 > * protobuf: 3.4.1 -> 3.7.1 (3.8.0 soon) > * cmake: 3.9.0 (Nov 2018) -> 3.14.3 (May 2019) > * snappy: 1.1.4 (Jan 2017) -> 1.1.7 (Aug 2017) > * lz4: r130 (patched, 2015) -> 1.9.1 (May 2019, expected perf gains) > * bitshuffle:
[jira] [Resolved] (KUDU-2997) Can't build spark-examples on MacOS
[
https://issues.apache.org/jira/browse/KUDU-2997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Grant Henke resolved KUDU-2997.
---
Fix Version/s: n/a
Resolution: Cannot Reproduce
> Can't build spark-examples on MacOS
> ---
>
> Key: KUDU-2997
> URL: https://issues.apache.org/jira/browse/KUDU-2997
> Project: Kudu
> Issue Type: Task
> Environment: OS:
> macOS Mojave 10.14.6
> Java:
> openjdk 11.0.2 2019-01-15
> OpenJDK Runtime Environment 18.9 (build 11.0.2+9)
> OpenJDK 64-Bit Server VM 18.9 (build 11.0.2+9, mixed mode)
> Maven:
> Apache Maven 3.6.0 (97c98ec64a1fdfee7767ce5ffb20918da4f719f3;
> 2018-10-24T20:41:47+02:00)
> Maven home: /usr/local/Cellar/maven/3.6.0/libexec
> Java version: 11.0.2, vendor: Oracle Corporation, runtime:
> /Library/Java/JavaVirtualMachines/jdk-11.0.2.jdk/Contents/Home
> Default locale: en_HU, platform encoding: UTF-8
> OS name: "mac os x", version: "10.14.6", arch: "x86_64", family: "mac"
>Reporter: Attila Bukor
>Assignee: Grant Henke
>Priority: Major
> Fix For: n/a
>
>
> When trying to build spark-examples on macOS Mojave it fails while it works
> fine on CentOS. Could be environmental as well, someone pease double check.
> {code}
> [INFO] Compiling 1 source files to
> /Users/charger/src/kudu/examples/scala/spark-example/target/classes at
> 1573937163020
> [ERROR] error: scala.reflect.internal.MissingRequirementError: object
> java.lang.Object in compiler mirror not found.
> [ERROR] at
> scala.reflect.internal.MissingRequirementError$.signal(MissingRequirementError.scala:17)
> [ERROR] at
> scala.reflect.internal.MissingRequirementError$.notFound(MissingRequirementError.scala:18)
> [ERROR] at
> scala.reflect.internal.Mirrors$RootsBase.getModuleOrClass(Mirrors.scala:53)
> [ERROR] at
> scala.reflect.internal.Mirrors$RootsBase.getModuleOrClass(Mirrors.scala:45)
> [ERROR] at
> scala.reflect.internal.Mirrors$RootsBase.getModuleOrClass(Mirrors.scala:45)
> [ERROR] at
> scala.reflect.internal.Mirrors$RootsBase.getModuleOrClass(Mirrors.scala:66)
> [ERROR] at
> scala.reflect.internal.Mirrors$RootsBase.getClassByName(Mirrors.scala:102)
> [ERROR] at
> scala.reflect.internal.Mirrors$RootsBase.getRequiredClass(Mirrors.scala:105)
> [ERROR] at
> scala.reflect.internal.Definitions$DefinitionsClass.ObjectClass$lzycompute(Definitions.scala:257)
> [ERROR] at
> scala.reflect.internal.Definitions$DefinitionsClass.ObjectClass(Definitions.scala:257)
> [ERROR] at
> scala.reflect.internal.Definitions$DefinitionsClass.init(Definitions.scala:1394)
> [ERROR] at scala.tools.nsc.Global$Run.(Global.scala:1215)
> [ERROR] at scala.tools.nsc.Driver.doCompile(Driver.scala:31)
> [ERROR] at scala.tools.nsc.MainClass.doCompile(Main.scala:23)
> [ERROR] at scala.tools.nsc.Driver.process(Driver.scala:51)
> [ERROR] at scala.tools.nsc.Driver.main(Driver.scala:64)
> [ERROR] at scala.tools.nsc.Main.main(Main.scala)
> [ERROR] at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> [ERROR] at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> [ERROR] at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> [ERROR] at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> [ERROR] at
> scala_maven_executions.MainHelper.runMain(MainHelper.java:164)
> [ERROR] at
> scala_maven_executions.MainWithArgsInFile.main(MainWithArgsInFile.java:26)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
