DNS Configs may completely break HBase cluster
----------------------------------------------
Key: HBASE-3413
URL: https://issues.apache.org/jira/browse/HBASE-3413
Project: HBase
Issue Type: Bug
Affects Versions: 0.90.0
Environment: all
Reporter: Mathias Herberts
I recently experienced a cluster malfunction which was caused by a change in
DNS config for services co-hosted on the machines running region servers.
The RS are specified using IP addresses in the 'regionservers' file. Those
machines are 1.example.com to N.example.com (there are A RRs for those names to
each of the N IP addresses in 'regionservers').
Until recently, the PTR RRs for the RS IPs were those x.example.com names.
Then a service was deployed on some of the x.example.com machines, and new A
RRs were added for svc.example.com which point to each of the IPs used for the
service.
Jointly new PTR records were added too for the given IPs. Those PTR records
have 'svc.example.com' as their PTRDATA, and this is causing the HBase cluster
to get completely confused.
Since it is perfectly legal to have multiple PTR records, it seems important to
make the canonicalization of RS more robust to DNS tweaks.
Maybe generating a UUID when a RS is started would help, this UUID could be
used to register the RS in ZK and we would not rely on DNS for obtaining a
stable canonical name (which may not even exist...).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
