Hi Team, I have a issue with HBase cluster.
I have hosted an HBase cluster with Phoenix on EMR emr-5.8.0. I have 1 master and 5 slaves 4.x large nodes. I’m losing the data while querying a table after a region server dies. Here are the steps I followed. 1. Launched the cluster with hfs replication factory as 3. 2. Created the tables and loaded the data using Phoenix. 3. Cross checked the data I loaded into tables and I see the data. 4. Wantedly terminated a EC2 machine which is part of cluster, meaning killing region server. 5. I could see EMR resizing and bringing up the new node. 6. When I query the table after the whole cluster is stable, which usually took 5-10 minutes, I see losing some data which is on dead RS. I believe HBase replays the WAL once new node is brought up and I could also see the WAL file on HDFS new RS’s directory. But somehow I don’t see the complete data from the table. Could you please let me know what possibly could go wrong. Also please let me know if I have to set any properties. I would be happy to provide more details if you need. Thanks, Nandakishore.
