[jira] [Assigned] (KUDU-1721) Expose ability to force an unsafe Raft configuration change
[ https://issues.apache.org/jira/browse/KUDU-1721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhangsong reassigned KUDU-1721: --- Assignee: zhangsong > Expose ability to force an unsafe Raft configuration change > --- > > Key: KUDU-1721 > URL: https://issues.apache.org/jira/browse/KUDU-1721 > Project: Kudu > Issue Type: Bug > Components: consensus >Reporter: Mike Percy >Assignee: zhangsong > > Add the ability to force an “unsafe” configuration change via RPC to the > leader. Instead of specifying AddServer() or RemoveServer(), which enforce > one-by-one configuration changes due to their inherent contract, we could > specify an unsafe configuration change API for administrative / emergency > cases only that the leader would process without going through the normal > "one by one" safety checks required by the Raft dissertation config change > protocol. > The leader should still reject configuration changes in which is it not a > member of the requested new configuration. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KUDU-1622) ResultTracker exhibits high contention under non-batched write workload
[ https://issues.apache.org/jira/browse/KUDU-1622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated KUDU-1622: -- Summary: ResultTracker exhibits high contention under non-batched write workload (was: ResultCache exhibits high contention under non-batched write workload) > ResultTracker exhibits high contention under non-batched write workload > --- > > Key: KUDU-1622 > URL: https://issues.apache.org/jira/browse/KUDU-1622 > Project: Kudu > Issue Type: Improvement > Components: perf, rpc >Affects Versions: 1.0.0 >Reporter: Todd Lipcon > Attachments: fg.svg, pprof13384.0.svg > > > I am running a YCSB workload with non-batched writes, and I see ResultCache > among the highest CPU consumers (>15% of the CPU). There's a lot of lock > contention on the result cache map. We should consider a more concurrent data > structure and/or finer-grained locking. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KUDU-1779) Consensus "stuck" with all transaction trackers are at limit
[ https://issues.apache.org/jira/browse/KUDU-1779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15713726#comment-15713726 ] Todd Lipcon commented on KUDU-1779: --- fwiw, restarting the servers in question unstuck the tablet by virtue of aborting all of the pending leader operations. > Consensus "stuck" with all transaction trackers are at limit > > > Key: KUDU-1779 > URL: https://issues.apache.org/jira/browse/KUDU-1779 > Project: Kudu > Issue Type: Bug > Components: consensus >Affects Versions: 1.1.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Critical > > In a stress cluster, I saw one tablet get "stuck" in the following state: > - the transaction_tracker on all three replicas is "full" (no more can be > submitted) > - leader elections proceed just fine, but no leader is able to advance the > commit index > The issue seems to be that a replica will respond with 'CANNOT_PREPARE' when > its transaction tracker is full. The leader then ignores this response, and > doesn't advance the majority-replicated watermark. The transaction tracker > stays full forever because the in-flight transactions can't get committed. > Notes to follow. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (KUDU-1773) Kudu client DCHECK fails
[
https://issues.apache.org/jira/browse/KUDU-1773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Todd Lipcon resolved KUDU-1773.
---
Resolution: Fixed
Assignee: Adar Dembo
Fix Version/s: 1.2.0
> Kudu client DCHECK fails
>
>
> Key: KUDU-1773
> URL: https://issues.apache.org/jira/browse/KUDU-1773
> Project: Kudu
> Issue Type: Bug
> Components: client
>Affects Versions: 1.1.0
>Reporter: Matthew Jacobs
>Assignee: Adar Dembo
>Priority: Critical
> Labels: impala
> Fix For: 1.2.0
>
>
> Impalad running stress tests w/ debug libkudu_client.so gets the following
> DCHECK failure:
> F1129 19:32:42.075459 185566 meta_cache.cc:267] Check failed: found Tablet
> e0eea4076a6843749389956abfed3172: Specified server not found:
> 520ffc6d07ad42baaf49e59f7129b080 (ve1121.halxg.cloudera.com:7050). Replicas:
> 275ece6d98a14be9b7dfcee3bec8d7a8 (FOLLOWER, OK),
> 2d52a82cb62c4064842cf628d01083fe (FOLLOWER, OK),
> c101883d3e82496989a5f9f667c30e38 (FOLLOWER, OK)
> stack from gdb dump:
> {code}
> #0 0x003fdd232625 in raise () from /lib64/libc.so.6
> #1 0x003fdd233e05 in abort () from /lib64/libc.so.6
> #2 0x0281f3b4 in ?? ()
> #3 0x0281881d in google::LogMessage::Fail() ()
> #4 0x0281b146 in google::LogMessage::SendToLog() ()
> #5 0x0281833d in google::LogMessage::Flush() ()
> #6 0x02818619 in google::LogMessage::~LogMessage() ()
> #7 0x011c902f in impala::LogKuduMessage(void*,
> kudu::client::KuduLogSeverity, char const*, int, tm const*, char const*,
> unsigned long) ()
> #8 0x011c9423 in
> kudu::client::KuduLoggingFunctionCallback::Run(kudu::client::KuduLogSeverity,
> char const*, int, tm const*, char const*, unsigned long) ()
> #9 0x7eff1b24073e in
> kudu::client::LoggingAdapterCB(kudu::client::KuduLoggingCallback*,
> kudu::LogSeverity, char const*, int, tm const*, char const*, unsigned long) ()
> at
> /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/kudu/kudu-f2aeba6c059ea61f9cf8984d3e84d6c27b64d463/src/kudu/client/client.cc:163
> #10 0x7eff1b25863c in kudu::internal::RunnableAdapter (*)(kudu::client::KuduLoggingCallback*, kudu::LogSeverity, char const*, int,
> tm const*, char const*, unsigned
> long)>::Run(kudu::client::KuduLoggingCallback* const&, kudu::LogSeverity
> const&, char const* const&, int const&, tm const* const&, char const* const&,
> unsigned long const&) ()
> at
> /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/kudu/kudu-f2aeba6c059ea61f9cf8984d3e84d6c27b64d463/src/kudu/gutil/bind_internal.h:584
> #11 0x7eff1b2568fe in kudu::internal::InvokeHelper kudu::internal::RunnableAdapter kudu::LogSeverity, char const*, int, tm const*, char const*, unsigned long)>,
> void ()(kudu::client::KuduLoggingCallback*, kudu::LogSeverity const&, char
> const* const&, int const&, tm const* const&, char const* const&, unsigned
> long const&)>::MakeItSo(kudu::internal::RunnableAdapter (*)(kudu::client::KuduLoggingCallback*, kudu::LogSeverity, char const*, int,
> tm const*, char const*, unsigned long)>, kudu::client::KuduLoggingCallback*,
> kudu::LogSeverity const&, char const* const&, int const&, tm const* const&,
> char const* const&, unsigned long const&) ()
> at
> /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/kudu/kudu-f2aeba6c059ea61f9cf8984d3e84d6c27b64d463/src/kudu/gutil/bind_internal.h:990
> #12 0x7eff1b2540e3 in kudu::internal::Invoker<1,
> kudu::internal::BindState tm const*, char const*, unsigned long)>, void
> ()(kudu::client::KuduLoggingCallback*, kudu::LogSeverity, char const*, int,
> tm const*, char const*, unsigned long), void
> ()(kudu::internal::UnretainedWrapper)>,
> void ()(kudu::client::KuduLoggingCallback*, kudu::LogSeverity, char const*,
> int, tm const*, char const*, unsigned
> long)>::Run(kudu::internal::BindStateBase*, kudu::LogSeverity const&, char
> const* const&, int const&, tm const* const&, char const* const&, unsigned
> long const&) ()
> at
> /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/kudu/kudu-f2aeba6c059ea61f9cf8984d3e84d6c27b64d463/src/kudu/gutil/bind_internal.h:2114
> #13 0x7eff1b3d1f53 in kudu::Callback const*, int, tm const*, char const*, unsigned long)>::Run(kudu::LogSeverity
> const&, char const* const&, int const&, tm const* const&, char const* const&,
> unsigned long const&) const ()
> at
>
[jira] [Commented] (KUDU-1779) Consensus "stuck" with all transaction trackers are at limit
[
https://issues.apache.org/jira/browse/KUDU-1779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15713618#comment-15713618
]
Todd Lipcon commented on KUDU-1779:
---
Here's some notes on the state of the tablet:
- on all three replicas, the transaction tracker is at the limit
-- the tracker has 5-10 transactions which have been assigned OpIds and many
other LEADER transactions that have no OpId assigned yet (i.e they are still in
the prepare queue)
--- these were submitted to the prepare queue at some past time when the
replica was LEADER, even though in many cases the replica is no longer LEADER
and we know that the operations will eventually fail once they make it out of
the prepare queue
-- the ops themselves have some row key conflicts: one op has been prepared and
is waiting to be replicated, and another op is now blocked waiting to obtain
the same row key lock. This has "stopped" the prepare thread from making any
progress
- the leader's queue state looks like this:
{code}
Peer: 2d52a82cb62c4064842cf628d01083fe
Is new: false
Last received: 151.174779
Next index: 174780
Last known committed idx: 174762
Last exchange result: ERROR
Needs tablet copy: false
Peer: 1d2e1b44012e419392279a8e674638cb
Is new: false
Last received: 145.174776
Next index: 174777
Last known committed idx: 174762
Last exchange result: ERROR
Needs tablet copy: false
(LEADER) Peer: c101883d3e82496989a5f9f667c30e38
Is new: false
Last received: 153.174786
Next index: 174787
Last known committed idx: 174762
Last exchange result: SUCCESS
Needs tablet copy: false
Consensus queue metrics:
Only Majority Done Ops: 174762
In Progress Ops: 24
Cache: LogCacheStats(num_ops=1115, bytes=54426033)
{code}
... its two followers are in 'ERROR' state because every attempt to replicate
to them is getting CANNOT_PREPARE errors.
I captured an RPC exchange on a replica using /tracing.html:
{code}
request:
preceding: 151.174779
committed_index: 174762
ops: 151.174780 - 153.174786
response:
code: CANNOT_PREPARE:
last_received: 151.174779
last_received_current_leader: 0.0
last_committed: 174762
{code}
the follower is logging:
{code}
I1201 17:06:01.941049 111261 raft_consensus.cc:866] T
9f7e54d8b37645c084a46fc6b27ddd34 P 2d52a82cb62c4064842cf628d01083fe [term 161
FOLLOWER]: Deduplicated request from leader. Original:
144.174762->[144.174763-161.174790] Dedup: 151.174779->[151.174780-161.174790]
W1201 17:06:01.941187 111261 raft_consensus.cc:1218] T
9f7e54d8b37645c084a46fc6b27ddd34 P 2d52a82cb62c4064842cf628d01083fe [term 161
FOLLOWER]: Could not prepare transaction for op: 151.174780. Suppressed 10
other warnings. Status for this op: Service unavailable: Transaction failed,
tablet 9f7e54d8b37645c084a46fc6b27ddd34 transaction memory consumption
(67108557) has exceeded its limit (67108864) or the limit of an ancestral
tracker
I1201 17:06:01.941339 111261 raft_consensus.cc:1232] T
9f7e54d8b37645c084a46fc6b27ddd34 P 2d52a82cb62c4064842cf628d01083fe [term 161
FOLLOWER]: Rejecting Update request from peer c101883d3e82496989a5f9f667c30e38
for term 161. Could not prepare a single transaction due to: Service
unavailable: Transaction failed, tablet 9f7e54d8b37645c084a46fc6b27ddd34
transaction memory consumption (67108557) has exceeded its limit (67108864) or
the limit of an ancestral tracker
{code}
the leader is logging:
{code}
W1201 17:06:34.255569 182530 wire_protocol.cc:150] Unknown error code in
status:
W1201 17:06:34.255590 182530 consensus_peers.cc:328] T
9f7e54d8b37645c084a46fc6b27ddd34 P c101883d3e82496989a5f9f667c30e38 -> Peer
2d52a82cb62c4064842cf628d01083fe (ve1120.halxg.cloudera.com:7050): Couldn't
send request to peer 2d52a82cb62c4064842cf628d01083fe for tablet
9f7e54d8b37645c084a46fc6b27ddd34. Status: Runtime error: (unknown error code).
Retrying in the next heartbeat period. Already trie
{code}
So it seems like there are a few things going wrong here:
- when the follower responds with 'CANNOT_PREPARE', the leader should still
take into account that follower's 'last_received' index in order to update the
majority-replicated watermark. Note here that the majority 'last_received' is
id 174779, but the queue watermark is calculated at 174762.
- the leader is not understanding the error response being sent back and
logging that odd 'Unknown error code in Status' message. This may contribute to
the above
- we seem to be lacking appropriate test coverage of cases where the
transaction tracker gets full
> Consensus "stuck" with all transaction trackers are at limit
>
>
> Key: KUDU-1779
> URL: https://issues.apache.org/jira/browse/KUDU-1779
> Project: Kudu
> Issue Type: Bug
> Components: consensus
>Affects Versions: 1.1.0
>Reporter: Todd Lipcon
>Assignee: Todd Lipcon
[jira] [Resolved] (KUDU-1774) add dirs use flag fs_data_dirs, but restart tserver failed
[ https://issues.apache.org/jira/browse/KUDU-1774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon resolved KUDU-1774. --- Resolution: Invalid Fix Version/s: n/a > add dirs use flag fs_data_dirs, but restart tserver failed > -- > > Key: KUDU-1774 > URL: https://issues.apache.org/jira/browse/KUDU-1774 > Project: Kudu > Issue Type: Bug >Affects Versions: 1.0.1 > Environment: CentOS 6.x > Kudu 1.0.1 >Reporter: zhangguangqiang > Fix For: n/a > > > before restart tserver, the flags is: > -fs_data_dirs= /data1/kudu/tserver_data/ > -fs_wal_dir=/data/kudu/tserver_wal > when I update the flag fs_data_dirs to > -fs_data_dirs= /data1/kudu/tserver_data/, /data2/kudu/tserver_data/, > /data3/kudu/tserver_data/ > and restart the tserver. the error messages shows below: > I1130 11:29:35.860208 56435 hybrid_clock.cc:178] HybridClock initialized. > Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current > error: 334145 > I1130 11:29:35.861098 56435 server_base.cc:164] Could not load existing FS > layout: Not found: /data2/kudu/tserver_data/instance: No such file or > directory (error 2) > I1130 11:29:35.861104 56435 server_base.cc:165] Creating new FS layout > F1130 11:29:35.861150 56435 tablet_server_main.cc:55] Check failed: _s.ok() > Bad status: Already present: Could not create new FS layout: FSManager root > is not empty: /data/kudu/tserver_wal > So I want to know is this a bug or not supported yet ？ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KUDU-1774) add dirs use flag fs_data_dirs, but restart tserver failed
[ https://issues.apache.org/jira/browse/KUDU-1774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15713599#comment-15713599 ] Todd Lipcon commented on KUDU-1774: --- That's the most safe way, for sure. If you want to try something that is more risky, I _think_ it would work to do the following: - copy the 'instance' file from one of your existing directories to the new one - move some of the '.data' and corresponding '.metadata' files from the existing disk into the new directory I've never tried this, though, so if it's important data, I would try the procedure first in a test environment. > add dirs use flag fs_data_dirs, but restart tserver failed > -- > > Key: KUDU-1774 > URL: https://issues.apache.org/jira/browse/KUDU-1774 > Project: Kudu > Issue Type: Bug >Affects Versions: 1.0.1 > Environment: CentOS 6.x > Kudu 1.0.1 >Reporter: zhangguangqiang > > before restart tserver, the flags is: > -fs_data_dirs= /data1/kudu/tserver_data/ > -fs_wal_dir=/data/kudu/tserver_wal > when I update the flag fs_data_dirs to > -fs_data_dirs= /data1/kudu/tserver_data/, /data2/kudu/tserver_data/, > /data3/kudu/tserver_data/ > and restart the tserver. the error messages shows below: > I1130 11:29:35.860208 56435 hybrid_clock.cc:178] HybridClock initialized. > Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current > error: 334145 > I1130 11:29:35.861098 56435 server_base.cc:164] Could not load existing FS > layout: Not found: /data2/kudu/tserver_data/instance: No such file or > directory (error 2) > I1130 11:29:35.861104 56435 server_base.cc:165] Creating new FS layout > F1130 11:29:35.861150 56435 tablet_server_main.cc:55] Check failed: _s.ok() > Bad status: Already present: Could not create new FS layout: FSManager root > is not empty: /data/kudu/tserver_wal > So I want to know is this a bug or not supported yet ？ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (KUDU-1779) Consensus "stuck" with all transaction trackers are at limit
Todd Lipcon created KUDU-1779: - Summary: Consensus "stuck" with all transaction trackers are at limit Key: KUDU-1779 URL: https://issues.apache.org/jira/browse/KUDU-1779 Project: Kudu Issue Type: Bug Components: consensus Affects Versions: 1.1.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Priority: Critical In a stress cluster, I saw one tablet get "stuck" in the following state: - the transaction_tracker on all three replicas is "full" (no more can be submitted) - leader elections proceed just fine, but no leader is able to advance the commit index The issue seems to be that a replica will respond with 'CANNOT_PREPARE' when its transaction tracker is full. The leader then ignores this response, and doesn't advance the majority-replicated watermark. The transaction tracker stays full forever because the in-flight transactions can't get committed. Notes to follow. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KUDU-1778) Consensus "stuck" after a leader election when both peers were divergent
[ https://issues.apache.org/jira/browse/KUDU-1778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated KUDU-1778: -- Status: In Review (was: Open) > Consensus "stuck" after a leader election when both peers were divergent > > > Key: KUDU-1778 > URL: https://issues.apache.org/jira/browse/KUDU-1778 > Project: Kudu > Issue Type: Bug > Components: consensus >Affects Versions: 1.1.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Critical > > On a stress cluster we saw the following sequence of events following a > service restart while under load: > - a peer is elected leader successfully > - both of its followers have divergent logs > - when it connects to a new peer with a divergent log, it decides to fall > back to index 0 rather than falling back to the proper committed index of > that peer > - upon falling back to index 0, will never succeed since the first segment of > the log was already GCed long ago. > Thus, the leader thinks that it needs to evict both of the followers and > can't replicate to them, and the tablet gets "stuck". -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KUDU-424) Kudu CLI/shell for DDL and basic DML
[ https://issues.apache.org/jira/browse/KUDU-424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15713582#comment-15713582 ] Todd Lipcon commented on KUDU-424: -- Yea, I think it's reasonable to close this. Another point is that the Python integration is now pretty full-featured and is pretty useful when you just want to "poke at" a table and dont have any other software installed. > Kudu CLI/shell for DDL and basic DML > > > Key: KUDU-424 > URL: https://issues.apache.org/jira/browse/KUDU-424 > Project: Kudu > Issue Type: New Feature > Components: client, ops-tooling >Affects Versions: M5 >Reporter: Todd Lipcon >Priority: Critical > Labels: kudu-roadmap > > Currently there is basically no way to access Kudu aside from programmatic > APIs. We need to have some simple ability to do basic tasks from the command > line: > - create/drop/alter table > - scan a table/key range > - stretch goal: insert/delete/update -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (KUDU-766) Handle log file deletion after a time/size
[ https://issues.apache.org/jira/browse/KUDU-766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dan Burkert reassigned KUDU-766: Assignee: Dan Burkert > Handle log file deletion after a time/size > -- > > Key: KUDU-766 > URL: https://issues.apache.org/jira/browse/KUDU-766 > Project: Kudu > Issue Type: Improvement > Components: ops-tooling >Affects Versions: Private Beta >Reporter: Todd Lipcon >Assignee: Dan Burkert >Priority: Critical > > According to Phil Z, CM doesn't do anything about log retention. We need to > add some code which will periodically scan our log dir and remove logs older > than a certain date (or maybe enforce a total size?). Otherwise it's likely > users are going to run out of space on their log volume. (after running a > workload for a few days with metrics logging enabled, I have 3.5GB of metrics > logs on one host) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KUDU-1778) Consensus "stuck" after a leader election when both peers were divergent
[
https://issues.apache.org/jira/browse/KUDU-1778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15712812#comment-15712812
]
Todd Lipcon commented on KUDU-1778:
---
Actually noticed this in the follower logs:
{code}
I1201 03:05:52.344377 168075 consensus_queue.cc:167] T
07b3624f00864ab18f984364ed6e2d11 P a1a2d4b5585a4ac2a4d6e4d9a02fce6b
[NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0,
Majority replicated index: 0, Committed index: 0, Last appended: 111.11350,
Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER
{code}
so I guess that if a replica starts up and has a divergent log, it will return
committed_index=0 unless it gets elected leader first?
> Consensus "stuck" after a leader election when both peers were divergent
>
>
> Key: KUDU-1778
> URL: https://issues.apache.org/jira/browse/KUDU-1778
> Project: Kudu
> Issue Type: Bug
> Components: consensus
>Affects Versions: 1.1.0
>Reporter: Todd Lipcon
>Assignee: Todd Lipcon
>Priority: Critical
>
> On a stress cluster we saw the following sequence of events following a
> service restart while under load:
> - a peer is elected leader successfully
> - both of its followers have divergent logs
> - when it connects to a new peer with a divergent log, it decides to fall
> back to index 0 rather than falling back to the proper committed index of
> that peer
> - upon falling back to index 0, will never succeed since the first segment of
> the log was already GCed long ago.
> Thus, the leader thinks that it needs to evict both of the followers and
> can't replicate to them, and the tablet gets "stuck".
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KUDU-1752) C++ client error memory should be bounded
[ https://issues.apache.org/jira/browse/KUDU-1752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15712799#comment-15712799 ] Alexey Serbin commented on KUDU-1752: - In review: https://gerrit.cloudera.org/#/c/5308/ > C++ client error memory should be bounded > - > > Key: KUDU-1752 > URL: https://issues.apache.org/jira/browse/KUDU-1752 > Project: Kudu > Issue Type: Bug > Components: client >Affects Versions: 1.0.0 >Reporter: Matthew Jacobs >Assignee: Alexey Serbin >Priority: Critical > Labels: client, impala > > The Kudu client can allocate unbounded amounts of memory for error messages > today, which is a potential problem for processes using the client. The API > to get errors does have a way to indicate the errors overflowed the error > buffer, but this isn't set yet. > Given that error messages can vary significantly in size (as they include a > var length string which will soon also include tracing output) and the row, > the client should have a way to indicate the max buffer memory size rather > than just some number of rows. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KUDU-1752) C++ client error memory should be bounded
[ https://issues.apache.org/jira/browse/KUDU-1752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Serbin updated KUDU-1752: Status: In Review (was: Open) > C++ client error memory should be bounded > - > > Key: KUDU-1752 > URL: https://issues.apache.org/jira/browse/KUDU-1752 > Project: Kudu > Issue Type: Bug > Components: client >Affects Versions: 1.0.0 >Reporter: Matthew Jacobs >Assignee: Alexey Serbin >Priority: Critical > Labels: client, impala > > The Kudu client can allocate unbounded amounts of memory for error messages > today, which is a potential problem for processes using the client. The API > to get errors does have a way to indicate the errors overflowed the error > buffer, but this isn't set yet. > Given that error messages can vary significantly in size (as they include a > var length string which will soon also include tracing output) and the row, > the client should have a way to indicate the max buffer memory size rather > than just some number of rows. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KUDU-1778) Consensus "stuck" after a leader election when both peers were divergent
[
https://issues.apache.org/jira/browse/KUDU-1778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15712794#comment-15712794
]
Todd Lipcon commented on KUDU-1778:
---
Here are the interesting logs from the leader:
{code}
[LEADER]: Peer 275ece log is divergent from this leader: its last log entry
110.11279 is not in this leader's log and it has not received anything from
this leader yet. Falling back to committed index 0
[LEADER]: Connected to new peer: Peer: 275ece, Is new: false, Last received:
0.0, Next index: 1, Last known committed idx: 0, Last exchange result: ERROR,
Needs tablet copy: false
[LEADER]: Peer a1a2d4 log is divergent from this leader: its last log entry
111.11350 is not in this leader's log and it has not received anything from
this leader yet. Falling back to committed index 0
[LEADER]: Connected to new peer: Peer: a1a2d4, Is new: false, Last received:
0.0, Next index: 1, Last known committed idx: 0, Last exchange result: ERROR,
Needs tablet copy: false
{code}
On the followers I see reasonable status from their bootstrap logs:
{code}
I1201 03:03:39.364948 147662 tablet_bootstrap.cc:1019] T
07b3624f00864ab18f984364ed6e2d11 P 275ece6d98a14be9b7dfcee3bec8d7a8:
ReplayState: Previous OpId: 110.11279, Committed OpId: 108.11277, Pending
Replicates: 2, Pending Commits: 0
... on the other node:
I1201 03:05:52.102141 168075 tablet_bootstrap.cc:1019] T
07b3624f00864ab18f984364ed6e2d11 P a1a2d4b5585a4ac2a4d6e4d9a02fce6b:
ReplayState: Previous OpId: 111.11350, Committed OpId: 111.11347, Pending
Replicates: 3, Pending Commits: 0
{code}
It seems like perhaps the follower didn't send its committed index properly in
the LMP mismatch error?
> Consensus "stuck" after a leader election when both peers were divergent
>
>
> Key: KUDU-1778
> URL: https://issues.apache.org/jira/browse/KUDU-1778
> Project: Kudu
> Issue Type: Bug
> Components: consensus
>Affects Versions: 1.1.0
>Reporter: Todd Lipcon
>Assignee: Todd Lipcon
>Priority: Critical
>
> On a stress cluster we saw the following sequence of events following a
> service restart while under load:
> - a peer is elected leader successfully
> - both of its followers have divergent logs
> - when it connects to a new peer with a divergent log, it decides to fall
> back to index 0 rather than falling back to the proper committed index of
> that peer
> - upon falling back to index 0, will never succeed since the first segment of
> the log was already GCed long ago.
> Thus, the leader thinks that it needs to evict both of the followers and
> can't replicate to them, and the tablet gets "stuck".
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (KUDU-1778) Consensus "stuck" after a leader election when both peers were divergent
Todd Lipcon created KUDU-1778: - Summary: Consensus "stuck" after a leader election when both peers were divergent Key: KUDU-1778 URL: https://issues.apache.org/jira/browse/KUDU-1778 Project: Kudu Issue Type: Bug Components: consensus Affects Versions: 1.1.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Priority: Critical On a stress cluster we saw the following sequence of events following a service restart while under load: - a peer is elected leader successfully - both of its followers have divergent logs - when it connects to a new peer with a divergent log, it decides to fall back to index 0 rather than falling back to the proper committed index of that peer - upon falling back to index 0, will never succeed since the first segment of the log was already GCed long ago. Thus, the leader thinks that it needs to evict both of the followers and can't replicate to them, and the tablet gets "stuck". -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (KUDU-1422) java client: Error buffer size is hard-coded
[
https://issues.apache.org/jira/browse/KUDU-1422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15709967#comment-15709967
]
Eric Maynard edited comment on KUDU-1422 at 12/1/16 6:59 PM:
-
Ah. In that case, I noticed that {{AsyncKuduSession}} already helpfully
provides a mutator for that variable. I don't see why {{maxCapacity}} needs to
be final, so we can access it in that mutator perhaps:
https://gerrit.cloudera.org/#/c/5291/
was (Author: emaynard):
Ah. In that case, I noticed that {{AsyncKuduSession}} already helpfully
provides a mutator for that variable. I don't see why {{maxCapacity}} needs to
be final, so we can access it in that mutator perhaps:
https://gerrit.cloudera.org/#/c/5286/
https://gerrit.cloudera.org/#/c/5287/
> java client: Error buffer size is hard-coded
>
>
> Key: KUDU-1422
> URL: https://issues.apache.org/jira/browse/KUDU-1422
> Project: Kudu
> Issue Type: Bug
> Components: client
>Affects Versions: 0.8.0
>Reporter: Mike Percy
>
> The java client has an error buffer for auto flush. The error buffer size is
> hard coded to 1000 errors; this should be tunable.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (KUDU-981) Validate table and column names
[ https://issues.apache.org/jira/browse/KUDU-981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Will Berkeley updated KUDU-981: --- Assignee: Todd Lipcon (was: Will Berkeley) > Validate table and column names > --- > > Key: KUDU-981 > URL: https://issues.apache.org/jira/browse/KUDU-981 > Project: Kudu > Issue Type: Bug > Components: master >Affects Versions: Private Beta >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Critical > Attachments: naming-freedom.png > > > Currently I believe we allow table names such as "", " ", etc. We should > probably figure out a reasonable subset of characters and validate names. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
