[jira] [Updated] (KUDU-3097) whether master load deleted entries into memory could be configuable
[ https://issues.apache.org/jira/browse/KUDU-3097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wangningito updated KUDU-3097: -- Description: The tablet of master is not under control of MVCC. The deleted entries like table structure, deleted tablet ids would be load into memory. For those who use the massive columns or lots of tablets and frequently switch table, it may result in some unnecessary memory usage. By the way, the memory usage is different between leader and follower in master. It may result in imbalance among master cluster. was:The > whether master load deleted entries into memory could be configuable > > > Key: KUDU-3097 > URL: https://issues.apache.org/jira/browse/KUDU-3097 > Project: Kudu > Issue Type: New Feature >Reporter: wangningito >Assignee: wangningito >Priority: Major > Attachments: image-2020-05-28-19-41-05-485.png, screenshot-1.png > > > The tablet of master is not under control of MVCC. > The deleted entries like table structure, deleted tablet ids would be load > into memory. > For those who use the massive columns or lots of tablets and frequently > switch table, it may result in some unnecessary memory usage. > By the way, the memory usage is different between leader and follower in > master. It may result in imbalance among master cluster. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KUDU-3097) whether master load deleted entries into memory could be configuable
[ https://issues.apache.org/jira/browse/KUDU-3097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wangningito updated KUDU-3097: -- Description: The (was: Recently I addressed a memory problem occurred by kudu master. Related info: Master memory report by system , 8.6g Master memory report by kudu web page, 1g The master keep running from 2018 to now, and still running. During two years, I create over 3 tables and delete them for test requirement. And we keep only 20 tables running during those time. I found the memory usage strange, so I tried to dump all the data of master via "kudu lcoal_replica dump rowset 0 ..." I found over 330k records in the environment. And the records are just mark as deleted. Guess this could cause be the burden for master, and some untracked memory.) > whether master load deleted entries into memory could be configuable > > > Key: KUDU-3097 > URL: https://issues.apache.org/jira/browse/KUDU-3097 > Project: Kudu > Issue Type: New Feature >Reporter: wangningito >Assignee: wangningito >Priority: Major > Attachments: image-2020-05-28-19-41-05-485.png, screenshot-1.png > > > The -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KUDU-3097) whether master load deleted entries into memory could be configuable
[ https://issues.apache.org/jira/browse/KUDU-3097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wangningito updated KUDU-3097: -- Summary: whether master load deleted entries into memory could be configuable (was: master should delete table info if table deleted, instead of keeping them forever) > whether master load deleted entries into memory could be configuable > > > Key: KUDU-3097 > URL: https://issues.apache.org/jira/browse/KUDU-3097 > Project: Kudu > Issue Type: New Feature >Reporter: wangningito >Assignee: wangningito >Priority: Major > Attachments: image-2020-05-28-19-41-05-485.png, screenshot-1.png > > > Recently I addressed a memory problem occurred by kudu master. > Related info: > Master memory report by system , 8.6g > Master memory report by kudu web page, 1g > The master keep running from 2018 to now, and still running. > During two years, I create over 3 tables and delete them for test > requirement. And we keep only 20 tables running during those time. > I found the memory usage strange, so I tried to dump all the data of master > via > "kudu lcoal_replica dump rowset 0 ..." > I found over 330k records in the environment. And the records are just mark > as deleted. > Guess this could cause be the burden for master, and some untracked memory. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KUDU-2612) Implement multi-row transactions
[ https://issues.apache.org/jira/browse/KUDU-2612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17119169#comment-17119169 ] Andrew Wong commented on KUDU-2612: --- I posted this design doc to the dev mailing list and intend on working on it in the near future: [https://docs.google.com/document/d/1qv7Zejpfzg-HvF5azRL49g5lRLQ4437EmJ53GiupcWQ/edit#] > Implement multi-row transactions > > > Key: KUDU-2612 > URL: https://issues.apache.org/jira/browse/KUDU-2612 > Project: Kudu > Issue Type: Task >Reporter: Mike Percy >Priority: Major > Labels: roadmap-candidate > > Tracking Jira to implement multi-row / multi-table transactions in Kudu. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KUDU-2632) Add DATE data type
[
https://issues.apache.org/jira/browse/KUDU-2632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Grant Henke resolved KUDU-2632.
---
Fix Version/s: 1.12.0
Resolution: Fixed
> Add DATE data type
> --
>
> Key: KUDU-2632
> URL: https://issues.apache.org/jira/browse/KUDU-2632
> Project: Kudu
> Issue Type: New Feature
> Components: tablet
>Affects Versions: 1.8.0
>Reporter: Adar Dembo
>Assignee: Vladimir Verjovkin
>Priority: Major
> Labels: roadmap-candidate
> Fix For: 1.12.0
>
> Attachments: Screenshot 2019-10-14 11.18.12.png
>
>
> {{DATE}} is a timezone-agnostic measure of the current date (without time of
> day) in -MM-DD form. The range of values typically supported is
> -01-01 to -12-31. [Parquet has
> implemented|https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#date]
> {{DATE}} as a logical type mapped to int32, represented as the number of
> days since the start of the UNIX epoch (i.e. January 1, 1970).
> Support for {{DATE}} is currently being added to Impala; we should also add
> support for it to Kudu.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (KUDU-368) Run local benchmarks under perf-stat
[ https://issues.apache.org/jira/browse/KUDU-368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Serbin reassigned KUDU-368: -- Assignee: Alexey Serbin > Run local benchmarks under perf-stat > > > Key: KUDU-368 > URL: https://issues.apache.org/jira/browse/KUDU-368 > Project: Kudu > Issue Type: Improvement > Components: test >Affects Versions: M4.5 >Reporter: Todd Lipcon >Assignee: Alexey Serbin >Priority: Minor > Labels: benchmarks, perf > > Would be nice to run a lot of our nightly benchmarks under perf-stat so we > can see on regression what factors changed (eg instruction count, cycles, > stalled cycles, cache misses, etc) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KUDU-477) Implement block storage microbenchmark
[ https://issues.apache.org/jira/browse/KUDU-477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Henke updated KUDU-477: - Labels: benchmarks (was: ) > Implement block storage microbenchmark > -- > > Key: KUDU-477 > URL: https://issues.apache.org/jira/browse/KUDU-477 > Project: Kudu > Issue Type: Sub-task > Components: fs >Affects Versions: M4.5 >Reporter: Adar Dembo >Priority: Major > Labels: benchmarks > > With two block storage allocation strategies implemented, we ought to develop > a synthetic microbenchmark to evaluate the two (and future allocation > strategies). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KUDU-523) Add comprehensive cfile read/write benchmarks to dashboard
[ https://issues.apache.org/jira/browse/KUDU-523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Henke updated KUDU-523: - Labels: benchmarks (was: ) > Add comprehensive cfile read/write benchmarks to dashboard > -- > > Key: KUDU-523 > URL: https://issues.apache.org/jira/browse/KUDU-523 > Project: Kudu > Issue Type: Bug > Components: perf, test >Affects Versions: Backlog >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Major > Labels: benchmarks > > We have a couple of benchmarks in cfile-test, but we don't expose them as > part of the benchmarks dashboard. > We should set up a templated test that runs through all valid pairs of > type/encoding and times sequential write, sequential read, and random read of > a cfile with a few million rows. We should also test each including some > NULLs (eg 50/50 alternating nulls). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KUDU-368) Run local benchmarks under perf-stat
[ https://issues.apache.org/jira/browse/KUDU-368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Henke updated KUDU-368: - Labels: benchmarks perf (was: perf) > Run local benchmarks under perf-stat > > > Key: KUDU-368 > URL: https://issues.apache.org/jira/browse/KUDU-368 > Project: Kudu > Issue Type: Improvement > Components: test >Affects Versions: M4.5 >Reporter: Todd Lipcon >Priority: Minor > Labels: benchmarks, perf > > Would be nice to run a lot of our nightly benchmarks under perf-stat so we > can see on regression what factors changed (eg instruction count, cycles, > stalled cycles, cache misses, etc) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KUDU-545) TSAN failure in master_replication-itest
[
https://issues.apache.org/jira/browse/KUDU-545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Grant Henke resolved KUDU-545.
--
Fix Version/s: NA
Resolution: Fixed
> TSAN failure in master_replication-itest
>
>
> Key: KUDU-545
> URL: https://issues.apache.org/jira/browse/KUDU-545
> Project: Kudu
> Issue Type: Bug
> Components: rpc
>Affects Versions: M4.5
>Reporter: Adar Dembo
>Priority: Minor
> Fix For: NA
>
>
> {noformat}
> ==
> WARNING: ThreadSanitizer: data race (pid=18993)
> Read of size 4 at 0x7d1c000345d0 by thread T9:
> #0 std::string* google::Check_EQImpl kudu::RpcServer::ServerState>(kudu::RpcServer::ServerState const&,
> kudu::RpcServer::ServerState const&, char const*)
> /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/thirdparty/installed/include/glog/logging.h:694
> (libserver_process.so+0x0005b5f7)
> #1 kudu::RpcServer::GetBoundAddresses(std::vector std::allocator >*) const
> /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/src/kudu/server/rpc_server.cc:117
> (libserver_process.so+0x0005af68)
> #2
> kudu::master::Master::GetMasterRegistration(kudu::master::MasterRegistrationPB*)
> const
> /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/src/kudu/master/master.cc:133
> (libmaster.so+0x000f523f)
> #3
> kudu::master::Master::ListMasters(std::vector std::allocator >*) const
> /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/src/kudu/master/master.cc:178
> (libmaster.so+0x000f56c7)
> #4
> kudu::master::MasterServiceImpl::ListMasters(kudu::master::ListMastersRequestPB
> const*, kudu::master::ListMastersResponsePB*, kudu::rpc::RpcContext*)
> /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/src/kudu/master/master_service.cc:352
> (libmaster.so+0x000fe267)
> #5 kudu::master::MasterServiceIf::Handle(kudu::rpc::InboundCall*)
> /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/src/kudu/master/master.service.cc:259
> (libmaster_proto.so+0x000a21b2)
> #6 kudu::rpc::ServicePool::RunThread()
> /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/src/kudu/rpc/service_pool.cc:155
> (libkrpc.so+0x000c91aa)
> #7 boost::_mfi::mf0 kudu::rpc::ServicePool>::operator()(kudu::rpc::ServicePool*) const
> /usr/include/boost/bind/mem_fn_template.hpp:49 (libkrpc.so+0x000cbe9d)
> #8 void boost::_bi::list1
> >::operator(),
> boost::_bi::list0>(boost::_bi::type, boost::_mfi::mf0 kudu::rpc::ServicePool>&, boost::_bi::list0&, int)
> /usr/include/boost/bind/bind.hpp:246 (libkrpc.so+0x000cbe0a)
> #9 boost::_bi::bind_t kudu::rpc::ServicePool>,
> boost::_bi::list1 >
> >::operator()() /usr/include/boost/bind/bind_template.hpp:20
> (libkrpc.so+0x000cbdb3)
> #10
> boost::detail::function::void_function_obj_invoker0 boost::_mfi::mf0,
> boost::_bi::list1 > >,
> void>::invoke(boost::detail::function::function_buffer&)
> /usr/include/boost/function/function_template.hpp:153
> (libkrpc.so+0x000cbbb9)
> #11 boost::function0::operator()() const
> /usr/include/boost/function/function_template.hpp:1012
> (libtablet.so+0x002145f1)
> #12 kudu::Thread::SuperviseThread(void*)
> /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/src/kudu/util/thread.cc:435
> (libkudu_util.so+0x00151fdb)
>
> Previous write of size 4 at 0x7d1c000345d0 by thread T198:
> #0 kudu::RpcServer::Start()
> /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/src/kudu/server/rpc_server.cc:101
> (libserver_process.so+0x0005ae40)
> #1 kudu::server::ServerBase::Start()
> /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/src/kudu/server/server_base.cc:176
> (libserver_process.so+0x0006079e)
> #2 kudu::master::Master::StartAsync()
> /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/src/kudu/master/master.cc:90
> (libmaster.so+0x000f4b46)
> #3 kudu::master::MiniMaster::StartOnPorts(unsigned short, unsigned short,
> kudu::master::MasterOptions*)
> /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/src/kudu/master/mini_master.cc:71
> (libmaster.so+0x0010a6b6)
> #4 kudu::master::MiniMaster::StartLeaderOnPorts(unsigned short, unsigned
> short, std::vector > const&)
> /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/src/kudu/master/mini_master.cc:94
> (libmaster.so+0x00109f12)
> #5 kudu::master::MiniMaster::StartLeader(std::vector std::allocator > const&)
> /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE
[jira] [Resolved] (KUDU-515) Log-dump, and anything else that acts on logs must handle schema changes mid segment.
[ https://issues.apache.org/jira/browse/KUDU-515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Henke resolved KUDU-515. -- Fix Version/s: 0.7.0 Resolution: Fixed > Log-dump, and anything else that acts on logs must handle schema changes mid > segment. > - > > Key: KUDU-515 > URL: https://issues.apache.org/jira/browse/KUDU-515 > Project: Kudu > Issue Type: Sub-task > Components: log, tablet >Affects Versions: M4.5 >Reporter: Alex Feinberg >Assignee: Todd Lipcon >Priority: Minor > Fix For: 0.7.0 > > > Currently (see KUDU-508) we log the tablet's schema to the log segment's > header. However, a schema may change mid-flight. We need to be able to handle > this gracefully. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KUDU-515) Log-dump, and anything else that acts on logs must handle schema changes mid segment.
[ https://issues.apache.org/jira/browse/KUDU-515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17118853#comment-17118853 ] Grant Henke commented on KUDU-515: -- It looks like this was resolved with the following commits: * https://github.com/apache/kudu/commit/b0f5bfd9a3f81a7e9a2305b7a3a63e898026a11f * https://github.com/apache/kudu/commit/6d47a476d1286bc9f2b9fa7546cd8a53d84b14ee * https://github.com/apache/kudu/commit/a1e44477ce10b34ca8b8a2f8bb4f9c17f75c2ad2 * https://github.com/apache/kudu/commit/36fd7e05ab0926bdd18a5fb42b601d9dd7a970b8 > Log-dump, and anything else that acts on logs must handle schema changes mid > segment. > - > > Key: KUDU-515 > URL: https://issues.apache.org/jira/browse/KUDU-515 > Project: Kudu > Issue Type: Sub-task > Components: log, tablet >Affects Versions: M4.5 >Reporter: Alex Feinberg >Assignee: Todd Lipcon >Priority: Minor > > Currently (see KUDU-508) we log the tablet's schema to the log segment's > header. However, a schema may change mid-flight. We need to be able to handle > this gracefully. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KUDU-521) Consider splitting the WAL in two, one for replicated operations and one for local operations
[ https://issues.apache.org/jira/browse/KUDU-521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Henke updated KUDU-521: - Labels: roadmap-candidate scalability (was: ) > Consider splitting the WAL in two, one for replicated operations and one for > local operations > - > > Key: KUDU-521 > URL: https://issues.apache.org/jira/browse/KUDU-521 > Project: Kudu > Issue Type: Sub-task > Components: consensus, log >Affects Versions: Backlog >Reporter: David Alves >Priority: Minor > Labels: roadmap-candidate, scalability > > Currently we rewrite the wal on replay because it has both local (the > commits) and replicated operations (the replicates). But we could potentially > split this in two separate WALs, one for the consensus replicated stuff and > one for the local stuff. > Keeping 2 different wals would have the following advantages: > - Only replicates would be fsynced for writes. > - On bootstrap the replicates WAL would not have to be rewritten > - Replicate WAL segments could be checksummed across machines to make sure > they are the same. > - The order of commits w.r.t. replicates would not matter. > - Flushing the WAL queue before we write the meta would only concern the > "local" WAL. > ... and the following disadvantages: > - On bootstrap replaying would be slightly more complex. One would have to > read from the replicated WAL and then advance the local wal until we find the > commits we want. > - GC would be more complex (but not as complex as dealing with commits before > replicates in the current version though). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KUDU-500) Changes to SysTables should be queryable on follower leaders upon replication
[ https://issues.apache.org/jira/browse/KUDU-500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Henke updated KUDU-500: - Labels: roadmap-candidate (was: ) > Changes to SysTables should be queryable on follower leaders upon replication > - > > Key: KUDU-500 > URL: https://issues.apache.org/jira/browse/KUDU-500 > Project: Kudu > Issue Type: Sub-task > Components: master >Affects Versions: M5 >Reporter: Alex Feinberg >Priority: Major > Labels: roadmap-candidate > > Currently CatalogManager serves requests from its in-memory objects. However, > these objects are not updated on the replicas. There are two approaches for > this: one is to serve CatalogManager from the underlying tablet (accepting > potential latency hits, but making the implementation simpler), another is to > implement callbacks on transaction completion on replicas (like MarkDirty but > called whenever writes are committed on a replica). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KUDU-437) Support tablet splits
[ https://issues.apache.org/jira/browse/KUDU-437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Henke updated KUDU-437: - Target Version/s: (was: 1.8.0) > Support tablet splits > - > > Key: KUDU-437 > URL: https://issues.apache.org/jira/browse/KUDU-437 > Project: Kudu > Issue Type: Task > Components: consensus, tablet >Affects Versions: M4.5 >Reporter: Jean-Daniel Cryans >Priority: Minor > Labels: roadmap-candidate > > Pre-splitting can get us pretty far but at some point we'll need to have > manual and automatic tablet splitting. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KUDU-423) Implement scalable and performant on-disk storage
[ https://issues.apache.org/jira/browse/KUDU-423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Henke updated KUDU-423: - Target Version/s: (was: 1.8.0) > Implement scalable and performant on-disk storage > - > > Key: KUDU-423 > URL: https://issues.apache.org/jira/browse/KUDU-423 > Project: Kudu > Issue Type: New Feature > Components: fs >Affects Versions: M4 >Reporter: Todd Lipcon >Priority: Major > Labels: roadmap-candidate, scalability > > Our current on-disk storage has a number of issues: > - only writes to one local disk (need to scale to at least 12) > - exhausts file descriptors quickly (see KUDU-56) > - causes a ton of seeks at startup (KUDU-374) > - has big latency stalls on write due to ext4 journal-related chunky writeback > - doesn't support features we'll eventually want to support such as tiered > storage -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KUDU-388) Simplify C++ KuduScanner iteration API
[
https://issues.apache.org/jira/browse/KUDU-388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Grant Henke updated KUDU-388:
-
Summary: Simplify C++ KuduScanner iteration API (was: Simplify KuduScanner
iteration API)
> Simplify C++ KuduScanner iteration API
> --
>
> Key: KUDU-388
> URL: https://issues.apache.org/jira/browse/KUDU-388
> Project: Kudu
> Issue Type: Improvement
> Components: api, client
>Affects Versions: M5
>Reporter: Vladimir Feinberg
>Priority: Major
>
> Currently, the {{KuduScanner}} for the C++ client has an iteration API that
> always results in the following (verbose) code pattern:
> {code}
> vector rows;
> while (scanner.HasMoreRows()) {
> CHECK_OK(scanner.NextBatch(&rows));
> BOOST_FOREACH(const KuduRowResult& row, rows) {
> DoStuffWithRow(row);
> }
> rows.clear();
> }
> {code}
> On top of the extra avoidable LOCs, the client has to manage individual
> batches and clear the vector. This batching should be done internally by the
> scanner, probably in the same manner above, but only exposing a {{bool
> HasNext()}} and {{const KuduRowResult& GetNext()}} interface. The simpler
> interface has another advantage in that it would allow us to change the way
> we gather rows while keeping the implementation details away from the client.
> See discussion below for further API considerations.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (KUDU-351) Tag sessions with a descriptor
[ https://issues.apache.org/jira/browse/KUDU-351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Henke updated KUDU-351: - Labels: supportability (was: ) > Tag sessions with a descriptor > -- > > Key: KUDU-351 > URL: https://issues.apache.org/jira/browse/KUDU-351 > Project: Kudu > Issue Type: New Feature > Components: client >Affects Versions: Backlog >Reporter: Jean-Daniel Cryans >Priority: Major > Labels: supportability > > Todd and I just discussed the concept of having a potentially user-provided > descriptor for sessions. We could pass this down to the servers and log it > when things happen. This isn't something we would rely on for authentication, > since it could be faked, but really just to help with tracing. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KUDU-1768) Tablet server cannot recover after run out of disk space
[
https://issues.apache.org/jira/browse/KUDU-1768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Grant Henke updated KUDU-1768:
--
Labels: roadmap-candidate (was: )
> Tablet server cannot recover after run out of disk space
>
>
> Key: KUDU-1768
> URL: https://issues.apache.org/jira/browse/KUDU-1768
> Project: Kudu
> Issue Type: Bug
>Affects Versions: 1.0.0
>Reporter: Juan Yu
>Priority: Major
> Labels: roadmap-candidate
> Attachments: tserver.log.tar.gz
>
>
> My node has small disk. after upsert into a large table several times, it ran
> out of disk space. I dropped the table. all nodes dropped the tablet
> successfully except one. it stuck in a bad state and tablet data are not
> removed even after restart.
> For dropped table, it shows
> {code}
> tmp_store_sales d88f8a7eba974cba8d9c547f69168c6bhash buckets:
> (0) FAILED (TABLET_DATA_DELETED): Invalid argument: Unable to delete
> on-disk data from tablet d88f8a7eba974cba8d9c547f69168c6b: The metadata for
> tablet d88f8a7eba974cba8d9c547f69168c6b still references orphaned blocks.
> Call DeleteTabletData() firstInvalid argument: Unable to
> delete on-disk data from tablet d88f8a7eba974cba8d9c547f69168c6b: The
> metadata for tablet d88f8a7eba974cba8d9c547f69168c6b still references
> orphaned blocks. Call DeleteTabletData() first
> {code}
> for other tables, it shows
> {code}
> kudu_store_sales f539ce77657e431aa5ce4ab7f7375f84hash buckets:
> (0, 7)FAILED (TABLET_DATA_READY): IO error: Failed log replay. Reason:
> Failed to open new log: Insufficient disk space to allocate 67108864 bytes
> under path
> /dataroot/dataroot/tserver/wal/wals/f539ce77657e431aa5ce4ab7f7375f84/.tmp.newsegmentFdOSTp
> (30613504 bytes free vs 0 bytes reserved) (error 28) IO
> error: Failed log replay. Reason: Failed to open new log: Insufficient disk
> space to allocate 67108864 bytes under path
> /dataroot/dataroot/tserver/wal/wals/f539ce77657e431aa5ce4ab7f7375f84/.tmp.newsegmentFdOSTp
> (30613504 bytes free vs 0 bytes reserved) (error 28)
> {code}
> attached tablet server log from this node.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (KUDU-320) Running out of space should be handled more gracefully
[ https://issues.apache.org/jira/browse/KUDU-320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Henke updated KUDU-320: - Labels: roadmap-candidate (was: ) > Running out of space should be handled more gracefully > -- > > Key: KUDU-320 > URL: https://issues.apache.org/jira/browse/KUDU-320 > Project: Kudu > Issue Type: Bug > Components: tablet >Affects Versions: M4 >Reporter: Alex Feinberg >Priority: Trivial > Labels: roadmap-candidate > > We could be more graceful when handling situations when we run out of space: > it can be deadlock (when happens during unit test) or a bus error. > This is low priority and we shouldn't do much (it is the operator's > responsibility to monitor disk space), but shutting down with "out of disk > space" would be more graceful. > clien-test logs when my desktop ran out of space (for an unrelated reason): > W0531 15:49:12.632431 13187 rpc_server.cc:102] Unable to unregister service > kudu.tserver.TabletServerService: Service unavailable: Service is not > registered > ... > W0531 15:49:46.814546 13187 thread.cc:360] Waited for 18000ms trying to join > with rpc worker -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KUDU-272) [java client] Investigate using FastTuple
[ https://issues.apache.org/jira/browse/KUDU-272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Henke resolved KUDU-272. -- Fix Version/s: n/a Resolution: Won't Fix > [java client] Investigate using FastTuple > - > > Key: KUDU-272 > URL: https://issues.apache.org/jira/browse/KUDU-272 > Project: Kudu > Issue Type: Improvement > Components: client >Affects Versions: M4 >Reporter: Jean-Daniel Cryans >Priority: Major > Fix For: n/a > > > https://github.com/boundary/fasttuple -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KUDU-245) Add ability to limit size of threadpool queue
[ https://issues.apache.org/jira/browse/KUDU-245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Henke resolved KUDU-245. -- Fix Version/s: (was: Backlog) NA Resolution: Fixed > Add ability to limit size of threadpool queue > - > > Key: KUDU-245 > URL: https://issues.apache.org/jira/browse/KUDU-245 > Project: Kudu > Issue Type: Improvement > Components: supportability >Affects Versions: M3 >Reporter: Mike Percy >Assignee: Mike Percy >Priority: Major > Fix For: NA > > > Currently, the threadpool queue has no limit. This can result in a lot of > wasted work and lack of backpressure for RPC services, including consensus, > that simply queue up work when it may time out later. > We should parameterize the threadpool queue and fail to Submit to the > threadpool if the queue is beyond the specified threshold. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KUDU-80) Facility to dump master metadata
[ https://issues.apache.org/jira/browse/KUDU-80?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17118817#comment-17118817 ] Grant Henke commented on KUDU-80: - This can be done via some of the more manual tools like `kudu local_replica dump rowset 0 ...` but perhaps a more public api type tool should exist. > Facility to dump master metadata > > > Key: KUDU-80 > URL: https://issues.apache.org/jira/browse/KUDU-80 > Project: Kudu > Issue Type: New Feature > Components: master, supportability >Affects Versions: M4 >Reporter: Todd Lipcon >Priority: Minor > > Would be good to have a utility which can dump the contents of the master > metadata tables, eg in protobuf TextFormat. This would be useful both at > runtime and in an "offline tool" fashion. Tool to force-modify these tables > offline would be handy too (again probably from PB TextFormat) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KUDU-39) RPC system metrics
[ https://issues.apache.org/jira/browse/KUDU-39?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17118782#comment-17118782 ] Grant Henke commented on KUDU-39: - Resolving as we have a few RPC metrics: * rpcs_queue_overflow * rpc_incoming_queue_time * rpc_connections_accepted * rpcs_timed_out_in_queue * handler_latency_ metrics for each RPC request handling method in the service Adding more specific metrics can be addressed in follow on jiras. > RPC system metrics > -- > > Key: KUDU-39 > URL: https://issues.apache.org/jira/browse/KUDU-39 > Project: Kudu > Issue Type: Improvement > Components: rpc >Affects Versions: M3 >Reporter: Todd Lipcon >Priority: Major > Labels: newbie > > would be useful to have some basic RPC system metrics: > - number of calls currently being processed > - number of handler threads busy > - call queue length > - number of connected sessions > ... -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KUDU-39) RPC system metrics
[ https://issues.apache.org/jira/browse/KUDU-39?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Henke resolved KUDU-39. - Fix Version/s: n/a Target Version/s: (was: Backlog) Resolution: Fixed > RPC system metrics > -- > > Key: KUDU-39 > URL: https://issues.apache.org/jira/browse/KUDU-39 > Project: Kudu > Issue Type: Improvement > Components: rpc >Affects Versions: M3 >Reporter: Todd Lipcon >Priority: Major > Labels: newbie > Fix For: n/a > > > would be useful to have some basic RPC system metrics: > - number of calls currently being processed > - number of handler threads busy > - call queue length > - number of connected sessions > ... -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KUDU-2604) Add label for tserver
[
https://issues.apache.org/jira/browse/KUDU-2604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17118788#comment-17118788
]
Alexey Serbin commented on KUDU-2604:
-
[~granthenke], yes, I think the remaining functionality can be broken down into
smaller JIRA items. At the higher level, I see the following pieces:
* Define and assign tags to tablet servers.
* Update master's placement policies to take into account tags when
adding/distributing replicas of tablets.
* Add support for C++ and Java clients: clients can specify set of tags when
creating tables.
* The {{kudu cluster rebalance}} tool and the auto-rebalancer honors the tags
when rebalancing corresponding tables. The tools is also able to report on
tablet replicas which are placed in a non-conforming way w.r.t. tags specified
for tables (those con-conformant-placed replicas might appear during automatic
re-replication: this is something similar that we have with current placement
policies).
* The {{kudu cluster ksck}} CLI tool provides information on tags for tablet
servers.
We can create sub-tasks for this if we decide to implement this.
> Add label for tserver
> -
>
> Key: KUDU-2604
> URL: https://issues.apache.org/jira/browse/KUDU-2604
> Project: Kudu
> Issue Type: New Feature
>Reporter: Hong Shen
>Priority: Major
> Labels: location-awareness, rack-awareness
> Fix For: n/a
>
> Attachments: image-2018-10-15-21-52-21-426.png
>
>
> When the cluster is bigger and bigger, big table with a lot of tablets will
> be distributed in almost all the tservers, when client write batch to the big
> table, it may cache connections to lots of tservers, the scalability may
> constrained.
> If the tablets in one table or partition only in a part of tservers, client
> will only have to cache connections to the part's tservers. So we propose to
> add label to tservers, each tserver belongs to a unique label. Client
> specified label when create table or add partition, the tablets will only be
> created on the tservers in specified label, if not specified, defalut label
> will be used.
> It will also benefit for:
> 1 Tserver across data center.
> 2 Heterogeneous tserver, like different disk, cpu or memory.
> 3 Physical isolating, especially IO, isolate some tables with others.
> 4 Gated Launch, upgrade tservers one by one label.
> In our product cluster, we have encounter the above issues and need to be
> resolved.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (KUDU-65) TS should refuse to re-register to different cluster
[ https://issues.apache.org/jira/browse/KUDU-65?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Henke updated KUDU-65: Summary: TS should refuse to re-register to different cluster (was: TS should remember master UUID and refuse to re-register to different cluster) > TS should refuse to re-register to different cluster > > > Key: KUDU-65 > URL: https://issues.apache.org/jira/browse/KUDU-65 > Project: Kudu > Issue Type: Improvement > Components: master, tserver >Affects Versions: M5 >Reporter: Todd Lipcon >Priority: Major > > prevent accidental dataloss if the tserver is pointed at the wrong cluster's > master -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KUDU-18) Handle "slow readers" not holding too much memory
[ https://issues.apache.org/jira/browse/KUDU-18?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Henke updated KUDU-18: Target Version/s: (was: 1.8.0) > Handle "slow readers" not holding too much memory > - > > Key: KUDU-18 > URL: https://issues.apache.org/jira/browse/KUDU-18 > Project: Kudu > Issue Type: Improvement > Components: tablet >Affects Versions: M3 >Reporter: Todd Lipcon >Priority: Major > > Currently if a scanner is held open, it can hold on to resources from the > tablet indefinitely. This includes memory resources like > MemRowSet/DeltaMemStore which can be quite large. We should figure out a way > to forceably expire slow scanners or otherwise migrate them to the flushed > copies. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KUDU-5) Shared memory transport for local scanners
[ https://issues.apache.org/jira/browse/KUDU-5?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17118774#comment-17118774 ] Grant Henke commented on KUDU-5: Some related work is in progress here: https://gerrit.cloudera.org/#/c/15706/ > Shared memory transport for local scanners > -- > > Key: KUDU-5 > URL: https://issues.apache.org/jira/browse/KUDU-5 > Project: Kudu > Issue Type: Improvement > Components: client, impala, perf, tserver >Affects Versions: Backlog >Reporter: Todd Lipcon >Priority: Major > > For Impala in particular, it would be nice to be able to set up the RowBlock > in a shared memory region, so that Impala scanners could avoid copying data > over the network stack. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KUDU-2604) Add label for tserver
[ https://issues.apache.org/jira/browse/KUDU-2604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17118772#comment-17118772 ] Grant Henke commented on KUDU-2604: --- [~aserbin] Do you think the remaining functionality not addressed by location awareness can/should be broken into new or smaller jiras? At a minimum I think the subject needs to be updated to reflect the outstanding work given tservers now have labels. > Add label for tserver > - > > Key: KUDU-2604 > URL: https://issues.apache.org/jira/browse/KUDU-2604 > Project: Kudu > Issue Type: New Feature >Reporter: Hong Shen >Priority: Major > Labels: location-awareness, rack-awareness > Fix For: n/a > > Attachments: image-2018-10-15-21-52-21-426.png > > > When the cluster is bigger and bigger, big table with a lot of tablets will > be distributed in almost all the tservers, when client write batch to the big > table, it may cache connections to lots of tservers, the scalability may > constrained. > If the tablets in one table or partition only in a part of tservers, client > will only have to cache connections to the part's tservers. So we propose to > add label to tservers, each tserver belongs to a unique label. Client > specified label when create table or add partition, the tablets will only be > created on the tservers in specified label, if not specified, defalut label > will be used. > It will also benefit for: > 1 Tserver across data center. > 2 Heterogeneous tserver, like different disk, cpu or memory. > 3 Physical isolating, especially IO, isolate some tables with others. > 4 Gated Launch, upgrade tservers one by one label. > In our product cluster, we have encounter the above issues and need to be > resolved. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Reopened] (KUDU-2604) Add label for tserver
[ https://issues.apache.org/jira/browse/KUDU-2604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Serbin reopened KUDU-2604: - It seems this JIRA item contains some useful ideas and details which are orthogonal to current implementation of the rack awareness feature. If implemented, they might complement the overall functionality of the placement policies in Kudu. I'm removing the duplicate of KUDU-1535 resolution. > Add label for tserver > - > > Key: KUDU-2604 > URL: https://issues.apache.org/jira/browse/KUDU-2604 > Project: Kudu > Issue Type: New Feature >Reporter: Hong Shen >Priority: Major > Labels: location-awareness, rack-awareness > Fix For: n/a > > Attachments: image-2018-10-15-21-52-21-426.png > > > When the cluster is bigger and bigger, big table with a lot of tablets will > be distributed in almost all the tservers, when client write batch to the big > table, it may cache connections to lots of tservers, the scalability may > constrained. > If the tablets in one table or partition only in a part of tservers, client > will only have to cache connections to the part's tservers. So we propose to > add label to tservers, each tserver belongs to a unique label. Client > specified label when create table or add partition, the tablets will only be > created on the tservers in specified label, if not specified, defalut label > will be used. > It will also benefit for: > 1 Tserver across data center. > 2 Heterogeneous tserver, like different disk, cpu or memory. > 3 Physical isolating, especially IO, isolate some tables with others. > 4 Gated Launch, upgrade tservers one by one label. > In our product cluster, we have encounter the above issues and need to be > resolved. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (KUDU-2604) Add label for tserver
[ https://issues.apache.org/jira/browse/KUDU-2604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17118729#comment-17118729 ] Grant Henke edited comment on KUDU-2604 at 5/28/20, 2:36 PM: - I think a similar goal was achieved via the rack awareness feature (KUDU-1535). https://kudu.apache.org/2019/04/30/location-awareness.html Please do re-open this or file a new jira with the remaining features that are not addressed. was (Author: granthenke): I think a similar goal was achieved via the rack awareness feature (KUDU-1535). https://kudu.apache.org/2019/04/30/location-awareness.html > Add label for tserver > - > > Key: KUDU-2604 > URL: https://issues.apache.org/jira/browse/KUDU-2604 > Project: Kudu > Issue Type: New Feature >Reporter: Hong Shen >Priority: Major > Labels: location-awareness, rack-awareness > Fix For: n/a > > Attachments: image-2018-10-15-21-52-21-426.png > > > When the cluster is bigger and bigger, big table with a lot of tablets will > be distributed in almost all the tservers, when client write batch to the big > table, it may cache connections to lots of tservers, the scalability may > constrained. > If the tablets in one table or partition only in a part of tservers, client > will only have to cache connections to the part's tservers. So we propose to > add label to tservers, each tserver belongs to a unique label. Client > specified label when create table or add partition, the tablets will only be > created on the tservers in specified label, if not specified, defalut label > will be used. > It will also benefit for: > 1 Tserver across data center. > 2 Heterogeneous tserver, like different disk, cpu or memory. > 3 Physical isolating, especially IO, isolate some tables with others. > 4 Gated Launch, upgrade tservers one by one label. > In our product cluster, we have encounter the above issues and need to be > resolved. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KUDU-1013) MR input format read from any replica
[ https://issues.apache.org/jira/browse/KUDU-1013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Henke resolved KUDU-1013. --- Fix Version/s: NA Resolution: Won't Fix Resolving as wont fix given we don't really focus on the MR code anymore. > MR input format read from any replica > - > > Key: KUDU-1013 > URL: https://issues.apache.org/jira/browse/KUDU-1013 > Project: Kudu > Issue Type: Improvement > Components: client, perf >Affects Versions: Private Beta >Reporter: Todd Lipcon >Priority: Major > Fix For: NA > > > right now, the MR input format only lists the leader replica as a local > split. I guess this is because it also reads from the leader by default. > For the case of MR or even Spark, it makes sense to read from followers and > pass a snapshot generated at the start of the job. This would give better > distribution of reads even if we dont balance the leaders well. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KUDU-2577) Support rebalancing data allocation across directories when adding a new data dir
[ https://issues.apache.org/jira/browse/KUDU-2577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Henke updated KUDU-2577: -- Labels: roadmap-candidate (was: ) > Support rebalancing data allocation across directories when adding a new data > dir > - > > Key: KUDU-2577 > URL: https://issues.apache.org/jira/browse/KUDU-2577 > Project: Kudu > Issue Type: Improvement > Components: ops-tooling, tablet >Affects Versions: 1.7.0 >Reporter: Mike Percy >Priority: Major > Labels: roadmap-candidate > > I got a request for a tool to rebalance data usage across a single server's > data directories when adding a data dir. There is no such tool, but I wanted > to document that request because it's a reasonable feature to have. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KUDU-577) Specification for expected semantics and client modes
[ https://issues.apache.org/jira/browse/KUDU-577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Henke resolved KUDU-577. -- Fix Version/s: NA Resolution: Fixed > Specification for expected semantics and client modes > - > > Key: KUDU-577 > URL: https://issues.apache.org/jira/browse/KUDU-577 > Project: Kudu > Issue Type: Sub-task > Components: api, client >Affects Versions: M4.5 >Reporter: Jean-Daniel Cryans >Assignee: David Alves >Priority: Critical > Fix For: NA > > > We need a detailed description of what the different client modes are and > what it means the clients should do. This is to ensure that both terminology > and behavior match between languages. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KUDU-3006) RebalanceIgnoredTserversTest.Basic is flaky
[
https://issues.apache.org/jira/browse/KUDU-3006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Grant Henke resolved KUDU-3006.
---
Fix Version/s: 1.12.0
Resolution: Fixed
> RebalanceIgnoredTserversTest.Basic is flaky
> ---
>
> Key: KUDU-3006
> URL: https://issues.apache.org/jira/browse/KUDU-3006
> Project: Kudu
> Issue Type: Bug
>Reporter: Hao Hao
>Assignee: YifanZhang
>Priority: Minor
> Fix For: 1.12.0
>
> Attachments: rebalancer_tool-test.1.txt
>
>
> RebalanceIgnoredTserversTest.Basic of the rebalancer_tool-test sometimes
> fails with an error like below. I attached full test log.
> {noformat}
> /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/tools/rebalancer_tool-test.cc:350:
> Failure
> Value of: out
> Expected: has substring "2dd9365c71c54e5d83294b31046c5478 | 0"
> Actual: "Per-server replica distribution summary for tservers_to_empty:\n
> Server UUID| Replica
> Count\n--+---\n
> 2dd9365c71c54e5d83294b31046c5478 | 1\n\nPer-server replica distribution
> summary:\n Statistic |
> Value\n---+--\n Minimum Replica Count | 0\n
> Maximum Replica Count | 1\n Average Replica Count | 0.50\n\nPer-table
> replica distribution summary:\n Replica Skew |
> Value\n--+--\n Minimum | 1\n Maximum | 1\n
> Average | 1.00\n\n\nrebalancing is complete: cluster is balanced
> (moved 0 replicas)\n" (of type std::string)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (KUDU-3127) Have replica placement and rebalancing consider non-uniform hardware
[ https://issues.apache.org/jira/browse/KUDU-3127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Henke updated KUDU-3127: -- Labels: roadmap-candidate (was: ) > Have replica placement and rebalancing consider non-uniform hardware > > > Key: KUDU-3127 > URL: https://issues.apache.org/jira/browse/KUDU-3127 > Project: Kudu > Issue Type: Improvement > Components: CLI, master >Reporter: Andrew Wong >Priority: Major > Labels: roadmap-candidate > > We've seen multiple deployments suffer from the fact of life that data > centers don't always have uniform hardware. Often times, racks are comprised > of whatever hardware we can salvage from other projects. As such, Kudu's > assumption that all tablet servers should be treated equally (sans location > awareness) can be a bad one. > There are a few pieces to making this better: > * Having Kudu determine the relative capacities of each tablet servers > (either automatically, or as input by an operator) > * Updating the replica placement policy to account for capacity across > tablet servers > * Bonus: have Kudu account for the current size used on each tablet server > Some things that might be worth considering: > * It seems reasonable to assume that each data directory is independent of > one another, so we should be able to determine with relative ease the total > capacity of a server by aggregating the total capacities of its data > directories. This doesn't account for colocated WAL directories, but that > might be a fine limitation, since we expect WAL usage to vary wildly as > ingest workloads vary. The capacity could be heartbeated to masters > periodically, or fetched via RPC by rebalancer tooling. > * Updating the placement policy seems trickier, since there are a lot of > nice properties with using the PO2C algorithm (e.g. eventual fixing of skew), > and with assuming that all tablets have equal weight (e.g. it's harder to > fall into the trap of moving a replica, only to move it back). Some variant > of PO2C, but based on _available space_ instead of replica count might be > worth considering for initial placement and for defining balance. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KUDU-2604) Add label for tserver
[ https://issues.apache.org/jira/browse/KUDU-2604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Henke resolved KUDU-2604. --- Fix Version/s: n/a Resolution: Duplicate > Add label for tserver > - > > Key: KUDU-2604 > URL: https://issues.apache.org/jira/browse/KUDU-2604 > Project: Kudu > Issue Type: New Feature >Reporter: Hong Shen >Priority: Major > Labels: location-awareness, rack-awareness > Fix For: n/a > > Attachments: image-2018-10-15-21-52-21-426.png > > > When the cluster is bigger and bigger, big table with a lot of tablets will > be distributed in almost all the tservers, when client write batch to the big > table, it may cache connections to lots of tservers, the scalability may > constrained. > If the tablets in one table or partition only in a part of tservers, client > will only have to cache connections to the part's tservers. So we propose to > add label to tservers, each tserver belongs to a unique label. Client > specified label when create table or add partition, the tablets will only be > created on the tservers in specified label, if not specified, defalut label > will be used. > It will also benefit for: > 1 Tserver across data center. > 2 Heterogeneous tserver, like different disk, cpu or memory. > 3 Physical isolating, especially IO, isolate some tables with others. > 4 Gated Launch, upgrade tservers one by one label. > In our product cluster, we have encounter the above issues and need to be > resolved. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KUDU-2604) Add label for tserver
[ https://issues.apache.org/jira/browse/KUDU-2604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17118729#comment-17118729 ] Grant Henke commented on KUDU-2604: --- I think a similar goal was achieved via the rack awareness feature (KUDU-1535). https://kudu.apache.org/2019/04/30/location-awareness.html > Add label for tserver > - > > Key: KUDU-2604 > URL: https://issues.apache.org/jira/browse/KUDU-2604 > Project: Kudu > Issue Type: New Feature >Reporter: Hong Shen >Priority: Major > Labels: location-awareness, rack-awareness > Attachments: image-2018-10-15-21-52-21-426.png > > > When the cluster is bigger and bigger, big table with a lot of tablets will > be distributed in almost all the tservers, when client write batch to the big > table, it may cache connections to lots of tservers, the scalability may > constrained. > If the tablets in one table or partition only in a part of tservers, client > will only have to cache connections to the part's tservers. So we propose to > add label to tservers, each tserver belongs to a unique label. Client > specified label when create table or add partition, the tablets will only be > created on the tservers in specified label, if not specified, defalut label > will be used. > It will also benefit for: > 1 Tserver across data center. > 2 Heterogeneous tserver, like different disk, cpu or memory. > 3 Physical isolating, especially IO, isolate some tables with others. > 4 Gated Launch, upgrade tservers one by one label. > In our product cluster, we have encounter the above issues and need to be > resolved. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KUDU-2726) Very large tablets defeat budgeted compaction
[ https://issues.apache.org/jira/browse/KUDU-2726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Henke updated KUDU-2726: -- Labels: density roadmap-candidate (was: ) > Very large tablets defeat budgeted compaction > - > > Key: KUDU-2726 > URL: https://issues.apache.org/jira/browse/KUDU-2726 > Project: Kudu > Issue Type: Improvement >Affects Versions: 1.9.0 >Reporter: William Berkeley >Priority: Major > Labels: density, roadmap-candidate > > On very large tablets (50GB+), despite being very uncompacted with a large > average rowset height, a default budget (128MB) worth of compaction may not > reduce average rowset height enough to pass the minimum threshold. Thus the > tablet stays uncompacted forever. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KUDU-886) Cluster load balancing
[ https://issues.apache.org/jira/browse/KUDU-886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Henke resolved KUDU-886. -- Fix Version/s: 1.9.0 Resolution: Fixed Kudu has the ability to rebalance load using the rebalancer tool. The commits that enabled this initially are: * https://github.com/apache/kudu/commit/81d0109b461fc29c427c1fc81598c286318a95b9 * https://github.com/apache/kudu/commit/fd507235b531419f6bc2447263278eee315177ab * https://github.com/apache/kudu/commit/43161e5aa75764a7f1748f85e0500fc454a301d8 * https://github.com/apache/kudu/commit/34bb7f93b4bcb8d39803d0f0718148f84f5cca22 * https://github.com/apache/kudu/commit/87084c108e1836ebb7811eace93836ee872a253e * https://github.com/apache/kudu/commit/f731ea004590217fe21b133ed093a7e9d21e7d42 * https://github.com/apache/kudu/commit/81bba2472b469996c3a0d8ed9f6412fe29cd4771 * https://github.com/apache/kudu/commit/4ec2598a355bbbd1c00c4bdc03d37bafe04d0d5f * https://github.com/apache/kudu/commit/8e9345a79849ed3e96a85dc5240e0a4e709b2055 A shortlist of the resolved jiras to support this also include KUDU-2780, KUDU-2823, KUDU-2901. Of course there will be ongoing improvements tracked by other jiras (ex: KUDU-3061). > Cluster load balancing > -- > > Key: KUDU-886 > URL: https://issues.apache.org/jira/browse/KUDU-886 > Project: Kudu > Issue Type: New Feature > Components: master >Affects Versions: 1.2.0 >Reporter: Todd Lipcon >Assignee: William Berkeley >Priority: Major > Labels: roadmap-candidate > Fix For: 1.9.0 > > > We should add some load balancing support for GA: > - move leaders to evenly spread RPC load. > - eventually move tablets to even out disk space or load. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KUDU-3061) Balance tablet leaders across TServers
[ https://issues.apache.org/jira/browse/KUDU-3061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Henke updated KUDU-3061: -- Labels: performance roadmap-candidate scalability (was: balance features performance) > Balance tablet leaders across TServers > -- > > Key: KUDU-3061 > URL: https://issues.apache.org/jira/browse/KUDU-3061 > Project: Kudu > Issue Type: New Feature > Components: api, tablet >Affects Versions: 1.11.1 >Reporter: Adam Voga >Priority: Major > Labels: performance, roadmap-candidate, scalability > > The number of leader tablets per tablet server can become imbalanced over > time, putting additional pressure on a few nodes. > A CLI tool or an extension to the existing balancer should be added to take > care of this. > Currently the only option is running leader_step_down manually. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (KUDU-3097) master should delete table info if table deleted, instead of keeping them forever
[ https://issues.apache.org/jira/browse/KUDU-3097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17118571#comment-17118571 ] wangningito edited comment on KUDU-3097 at 5/28/20, 11:42 AM: -- Loading all data into memory. !image-2020-05-28-19-41-05-485.png! With minor change of code, just skip loading of deleted entries. !screenshot-1.png! was (Author: wangning): Loading all data into memory. !image-2020-05-28-19-41-05-485.png! With minor change of code, just skip loading of them. !screenshot-1.png! > master should delete table info if table deleted, instead of keeping them > forever > - > > Key: KUDU-3097 > URL: https://issues.apache.org/jira/browse/KUDU-3097 > Project: Kudu > Issue Type: New Feature >Reporter: wangningito >Assignee: wangningito >Priority: Major > Attachments: image-2020-05-28-19-41-05-485.png, screenshot-1.png > > > Recently I addressed a memory problem occurred by kudu master. > Related info: > Master memory report by system , 8.6g > Master memory report by kudu web page, 1g > The master keep running from 2018 to now, and still running. > During two years, I create over 3 tables and delete them for test > requirement. And we keep only 20 tables running during those time. > I found the memory usage strange, so I tried to dump all the data of master > via > "kudu lcoal_replica dump rowset 0 ..." > I found over 330k records in the environment. And the records are just mark > as deleted. > Guess this could cause be the burden for master, and some untracked memory. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KUDU-3097) master should delete table info if table deleted, instead of keeping them forever
[ https://issues.apache.org/jira/browse/KUDU-3097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17118571#comment-17118571 ] wangningito commented on KUDU-3097: --- Loading all data into memory. !image-2020-05-28-19-41-05-485.png! With minor change of code, just skip loading of them. !screenshot-1.png! > master should delete table info if table deleted, instead of keeping them > forever > - > > Key: KUDU-3097 > URL: https://issues.apache.org/jira/browse/KUDU-3097 > Project: Kudu > Issue Type: New Feature >Reporter: wangningito >Assignee: wangningito >Priority: Major > Attachments: image-2020-05-28-19-41-05-485.png, screenshot-1.png, > screenshot-2.png > > > Recently I addressed a memory problem occurred by kudu master. > Related info: > Master memory report by system , 8.6g > Master memory report by kudu web page, 1g > The master keep running from 2018 to now, and still running. > During two years, I create over 3 tables and delete them for test > requirement. And we keep only 20 tables running during those time. > I found the memory usage strange, so I tried to dump all the data of master > via > "kudu lcoal_replica dump rowset 0 ..." > I found over 330k records in the environment. And the records are just mark > as deleted. > Guess this could cause be the burden for master, and some untracked memory. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KUDU-3097) master should delete table info if table deleted, instead of keeping them forever
[ https://issues.apache.org/jira/browse/KUDU-3097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wangningito updated KUDU-3097: -- Attachment: image-2020-05-28-19-41-05-485.png > master should delete table info if table deleted, instead of keeping them > forever > - > > Key: KUDU-3097 > URL: https://issues.apache.org/jira/browse/KUDU-3097 > Project: Kudu > Issue Type: New Feature >Reporter: wangningito >Assignee: wangningito >Priority: Major > Attachments: image-2020-05-28-19-41-05-485.png, screenshot-1.png > > > Recently I addressed a memory problem occurred by kudu master. > Related info: > Master memory report by system , 8.6g > Master memory report by kudu web page, 1g > The master keep running from 2018 to now, and still running. > During two years, I create over 3 tables and delete them for test > requirement. And we keep only 20 tables running during those time. > I found the memory usage strange, so I tried to dump all the data of master > via > "kudu lcoal_replica dump rowset 0 ..." > I found over 330k records in the environment. And the records are just mark > as deleted. > Guess this could cause be the burden for master, and some untracked memory. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KUDU-3097) master should delete table info if table deleted, instead of keeping them forever
[ https://issues.apache.org/jira/browse/KUDU-3097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wangningito updated KUDU-3097: -- Attachment: (was: screenshot-2.png) > master should delete table info if table deleted, instead of keeping them > forever > - > > Key: KUDU-3097 > URL: https://issues.apache.org/jira/browse/KUDU-3097 > Project: Kudu > Issue Type: New Feature >Reporter: wangningito >Assignee: wangningito >Priority: Major > Attachments: image-2020-05-28-19-41-05-485.png, screenshot-1.png > > > Recently I addressed a memory problem occurred by kudu master. > Related info: > Master memory report by system , 8.6g > Master memory report by kudu web page, 1g > The master keep running from 2018 to now, and still running. > During two years, I create over 3 tables and delete them for test > requirement. And we keep only 20 tables running during those time. > I found the memory usage strange, so I tried to dump all the data of master > via > "kudu lcoal_replica dump rowset 0 ..." > I found over 330k records in the environment. And the records are just mark > as deleted. > Guess this could cause be the burden for master, and some untracked memory. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KUDU-3097) master should delete table info if table deleted, instead of keeping them forever
[ https://issues.apache.org/jira/browse/KUDU-3097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wangningito updated KUDU-3097: -- Attachment: screenshot-2.png > master should delete table info if table deleted, instead of keeping them > forever > - > > Key: KUDU-3097 > URL: https://issues.apache.org/jira/browse/KUDU-3097 > Project: Kudu > Issue Type: New Feature >Reporter: wangningito >Assignee: wangningito >Priority: Major > Attachments: screenshot-1.png, screenshot-2.png > > > Recently I addressed a memory problem occurred by kudu master. > Related info: > Master memory report by system , 8.6g > Master memory report by kudu web page, 1g > The master keep running from 2018 to now, and still running. > During two years, I create over 3 tables and delete them for test > requirement. And we keep only 20 tables running during those time. > I found the memory usage strange, so I tried to dump all the data of master > via > "kudu lcoal_replica dump rowset 0 ..." > I found over 330k records in the environment. And the records are just mark > as deleted. > Guess this could cause be the burden for master, and some untracked memory. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KUDU-3097) master should delete table info if table deleted, instead of keeping them forever
[ https://issues.apache.org/jira/browse/KUDU-3097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wangningito updated KUDU-3097: -- Attachment: screenshot-1.png > master should delete table info if table deleted, instead of keeping them > forever > - > > Key: KUDU-3097 > URL: https://issues.apache.org/jira/browse/KUDU-3097 > Project: Kudu > Issue Type: New Feature >Reporter: wangningito >Assignee: wangningito >Priority: Major > Attachments: screenshot-1.png > > > Recently I addressed a memory problem occurred by kudu master. > Related info: > Master memory report by system , 8.6g > Master memory report by kudu web page, 1g > The master keep running from 2018 to now, and still running. > During two years, I create over 3 tables and delete them for test > requirement. And we keep only 20 tables running during those time. > I found the memory usage strange, so I tried to dump all the data of master > via > "kudu lcoal_replica dump rowset 0 ..." > I found over 330k records in the environment. And the records are just mark > as deleted. > Guess this could cause be the burden for master, and some untracked memory. -- This message was sent by Atlassian Jira (v8.3.4#803005)
