[ 
https://issues.apache.org/jira/browse/KUDU-1839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15869723#comment-15869723
 ] 

Adar Dembo commented on KUDU-1839:
----------------------------------

KUDU-1099 describes a similar crash in consensus when DNS became temporarily 
unavailable.

> DNS failure during tablet creation lead to undeletable tablet
> -------------------------------------------------------------
>
>                 Key: KUDU-1839
>                 URL: https://issues.apache.org/jira/browse/KUDU-1839
>             Project: Kudu
>          Issue Type: Bug
>          Components: master, tablet
>    Affects Versions: 1.2.0
>            Reporter: Adar Dembo
>            Priority: Critical
>
> During a YCSB workload, two tservers died due to DNS resolution timeouts. For 
> example: 
> {noformat}
> F0117 09:21:14.952937  8392 raft_consensus.cc:1985] Check failed: _s.ok() Bad 
> status: Network error: Could not obtain a remote proxy to the peer.: Unable 
> to resolve address 've0130.halxg.cloudera.com': Name or service not known
> {noformat}
> It's not clear why this happened; perhaps table creation places an inordinate 
> strain on DNS due to concurrent resolution load from all the bootstrapping 
> peers.
> In any case, when these tservers were restarted, two tablets failed to 
> bootstrap, both for the same reason. I'll focus on just one tablet from here 
> on out to simplify troubleshooting:
> {noformat}
> E0117 15:35:45.567312 85124 ts_tablet_manager.cc:749] T 
> 8c167c441a7d44b8add737d13797e694 P 7425c65d80f54f2da0a85494a5eb3e68: Tablet 
> failed to bootstrap: Not found: Unable to load Consensus metadata: 
> /data/2/kudu/consensus-meta/8c167c441a7d44b8add737d13797e694: No such file or 
> directory (error 2)
> {noformat}
> Eventually, the master decided to delete this tablet:
> {noformat}
> I0117 15:42:32.119601 85166 tablet_service.cc:672] Processing DeleteTablet 
> for tablet 8c167c441a7d44b8add737d13797e694 with delete_type 
> TABLET_DATA_TOMBSTONED (TS 7425c65d80f54f2da0a85494a5eb3e68 not found in new 
> config with opid_index 29) from {real_user=kudu} at 10.17.236.18:42153
> I0117 15:42:32.139128 85166 tablet_service.cc:672] Processing DeleteTablet 
> for tablet 8c167c441a7d44b8add737d13797e694 with delete_type 
> TABLET_DATA_TOMBSTONED (TS 7425c65d80f54f2da0a85494a5eb3e68 not found in new 
> config with opid_index 29) from {real_user=kudu} at 10.17.236.18:42153
> I0117 15:42:32.181843 85166 tablet_service.cc:672] Processing DeleteTablet 
> for tablet 8c167c441a7d44b8add737d13797e694 with delete_type 
> TABLET_DATA_TOMBSTONED (TS 7425c65d80f54f2da0a85494a5eb3e68 not found in new 
> config with opid_index 29) from {real_user=kudu} at 10.17.236.18:42153
> I0117 15:42:32.276289 85166 tablet_service.cc:672] Processing DeleteTablet 
> for tablet 8c167c441a7d44b8add737d13797e694 with delete_type 
> TABLET_DATA_TOMBSTONED (TS 7425c65d80f54f2da0a85494a5eb3e68 not found in new 
> config with opid_index 29) from {real_user=kudu} at 10.17.236.18:42153
> {noformat}
> As can be seen by the presence of multiple deletion requests, each one 
> failed. It's annoying that the tserver didn't log why. But the master did:
> {noformat}
> I0117 15:42:32.117022 33903 catalog_manager.cc:2758] Sending 
> DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet 
> 8c167c441a7d44b8add737d13797e694 on 7425c65d80f54f2da0a85494a5eb3e68 
> (ve0122.halxg.cloudera.com:7050) (TS 7425c65d80f54f2da0a85494a5eb3e68 not 
> found in new config with opid_index 29)
> W0117 15:42:32.117463 33890 catalog_manager.cc:2725] TS 
> 7425c65d80f54f2da0a85494a5eb3e68 (ve0122.halxg.cloudera.com:7050): delete 
> failed for tablet 8c167c441a7d44b8add737d13797e694 with error code 
> TABLET_NOT_RUNNING: Illegal state: Consensus not available. Tablet shutting 
> down
> I0117 15:42:32.117491 33890 catalog_manager.cc:2522] Scheduling retry of 
> 8c167c441a7d44b8add737d13797e694 Delete Tablet RPC for 
> TS=7425c65d80f54f2da0a85494a5eb3e68 with a delay of 19ms (attempt = 1)...
> {noformat}
> This isn't a fatal error as far as the master is concerned, so it retries the 
> deletion forever.
> Meanwhile, the broken replica of this tablet still appears to be part of the 
> replication group. At least, that's true as far as both the master web UI and 
> the tserver web UI are concerned. The leader tserver is logging this error 
> repeatedly:
> {noformat}
> W0117 16:38:04.797828 81809 consensus_peers.cc:329] T 
> 8c167c441a7d44b8add737d13797e694 P 335d132897de4bdb9b87443f2c487a42 -> Peer 
> 7425c65d80f54f2da0a85494a5eb3e68 (ve0122.halxg.cloudera.com:7050): Couldn't 
> send request to peer 7425c65d80f54f2da0a85494a5eb3e68 for tablet 
> 8c167c441a7d44b8add737d13797e694. Error code: TABLET_NOT_RUNNING (12). 
> Status: Illegal state: Tablet not RUNNING: FAILED: Not found: Unable to load 
> Consensus metadata: 
> /data/2/kudu/consensus-meta/8c167c441a7d44b8add737d13797e694: No such file or 
> directory (error 2). Retrying in the next heartbeat period. Already tried 
> 6666 times.
> {noformat}
> It's not clear to me exactly what state the replication group is in. The 
> master did issue an AddServer request:
> {noformat}
> I0117 15:42:32.117065 33903 catalog_manager.cc:3069] Started AddServer task 
> for tablet 8c167c441a7d44b8add737d13797e694
> {noformat}
> But the leader of the tablet still thinks the broken replica is in the 
> replication group. So is it a tablet with two healthy replicas and one broken 
> one, that can't recover? Maybe.
> So a couple things are broken here:
> # Table creation probably created a DNS resolution storm.
> # Failure in DNS resolution is not retried, and led to tserver death.
> # On bootstrap, this replica was detected as having a tablet-meta file but no 
> consensus-meta, and was set aside as corrupt (good). But the lack of a 
> consensus-meta means there's no consensus state and so the tserver cannot 
> perform an "atomic delete" as requested by the master. Must we manually 
> delete this replica? Or should the master be able to force the issue?
> # The tserver did not log the tablet deletion failure.
> # The master retried the deletion in perpetuity.
> # Re-replication of this tablet by the leader appears to be broken.
> I think at least some of these issues are tracked in other JIRAs.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Reply via email to