Abhishek Chennaka created KUDU-3770:
---------------------------------------
Summary: Add guard rails during new Kudu master addition
Key: KUDU-3770
URL: https://issues.apache.org/jira/browse/KUDU-3770
Project: Kudu
Issue Type: Improvement
Reporter: Abhishek Chennaka
When a new blank (empty data and wal directory) Kudu master is started by
providing the addresses of existing Kudu masters, the following happens in the
new Kudu master:
1. Formats the file system
2. Creates a new syscatalog
3. Talks to the existing Kudu master(/s) and realizes it has to copy over the
system catalog from them
4. Deletes the existing system catalog and copies it from the existing Kudu
masters.
5. Loads the copied over system catalog and heartbeats to the other Kudu
masters to eventually become a FOLLOWER.
We should have a guard rail when deleting the system catalog in step 4 as there
have been cases (due to user error) where the system catalog in step 4 was not
empty and lead to dataloss of the Kudu master.
There was also the below error seen in this case when the new Kudu master was
trying to establish and secure rpc connection to the running Kudu master:
{code:java}
W20260331 00:13:07.996311 790211 negotiation.cc:337] Failed RPC negotiation.
Trace:
0331 00:13:07.990432 (+ 0us) reactor.cc:625] Submitting negotiation task
for client connection to 10.140.222.1:7051 (local address 10.140.157.65:58380)
0331 00:13:07.990669 (+ 237us) negotiation.cc:107] Waiting for socket to
connect
0331 00:13:07.990699 (+ 30us) client_negotiation.cc:174] Beginning
negotiation
0331 00:13:07.990770 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE
NegotiatePB request
0331 00:13:07.991182 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE
NegotiatePB response
0331 00:13:07.991183 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE
response from server
0331 00:13:07.991187 (+ 4us) client_negotiation.cc:189] Negotiated
authn=CERTIFICATE
0331 00:13:07.991612 (+ 425us) client_negotiation.cc:514] Sending
TLS_HANDSHAKE message to server
0331 00:13:07.991615 (+ 3us) client_negotiation.cc:252] Sending
TLS_HANDSHAKE NegotiatePB request
0331 00:13:07.995412 (+ 3797us) client_negotiation.cc:271] Received
TLS_HANDSHAKE NegotiatePB response
0331 00:13:07.995413 (+ 1us) client_negotiation.cc:527] Received
TLS_HANDSHAKE response from server
0331 00:13:07.996217 (+ 804us) negotiation.cc:327] Negotiation complete:
Runtime error: Client connection negotiation failed: client connection to
10.140.222.1:7051: TLS Handshake error: error:0407008A:rsa
routines:RSA_padding_check_PKCS1_type_1:invalid padding:crypto/rsa/rsa_pk1.c:70
error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check
failed:crypto/rsa/rsa_ossl.c:769 error:0D0C5006:asn1 encoding
routines:ASN1_item_verify:EVP lib:crypto/asn1/a_verify.c:179 error:1416F086:SSL
routines:tls_process_server_certificate:certificate verify
failed:ssl/statem/statem_clnt.c:1915{code}
This seems to be due to a mismatch of the secret key and the certificate in the
new Kudu master. I am not sure what exactly is the reason behind this issue but
the higher priority item would be to ensure a Kudu master should not delete its
sys catalog if it has any cluster data which would ensure we won't hit the
above issue.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
