Nandakishore, Are the edits persisted to the WAL? The RS will only be able to recover edits that were persisted to the WAL. Therefore, if writes happened that had not been synced/flushed to the WAL when the node was terminated, it is expected that there could be a loss of that data. However, from the client side, I don't believe the write would have succeeded (someone can correct me on this as I don't have concrete evidence) if the data hadn't been fsynced/flushed to the WAL.
Thanks, Zach On Sun, May 20, 2018 at 11:27 PM, Nandakishore Arvapaly < [email protected]> wrote: > Hi Team, > > I have a issue with HBase cluster. > > I have hosted an HBase cluster with Phoenix on EMR emr-5.8.0. I have 1 > master and 5 slaves 4.x large nodes. I’m losing the data while querying a > table after a region server dies. Here are the steps I followed. > > 1. Launched the cluster with hfs replication factory as 3. > 2. Created the tables and loaded the data using Phoenix. > 3. Cross checked the data I loaded into tables and I see the data. > 4. Wantedly terminated a EC2 machine which is part of cluster, meaning > killing region server. > 5. I could see EMR resizing and bringing up the new node. > 6. When I query the table after the whole cluster is stable, which usually > took 5-10 minutes, I see losing some data which is on dead RS. > > I believe HBase replays the WAL once new node is brought up and I could > also see the WAL file on HDFS new RS’s directory. But somehow I don’t see > the complete data from the table. > > Could you please let me know what possibly could go wrong. Also please let > me know if I have to set any properties. > > I would be happy to provide more details if you need. > > Thanks, > Nandakishore. >
