Hi All, We have been experiencing severe data loss issues from few hours. There are some wierd things going on in the cluster. We were unable to locate the data even in hdfs
Hbase version 0.94.1 Here is the wierd things that are going on: 1) Table which was once 1TB has now become 170GB with many of the regions which we once 7gb are now becoming few MB's. We are no clue what is happening at all 2) Table is splitting (or what ever) (100 regions have become 200 regions) and ours is constantregionsplitpolicy with region size 20gb. I don't know why it is even spltting 3) HDFS namenode dump size which we periodically backup is decreasing 4) And there is a region chain with start keys and end keys as, I can't copy paste the exact thing. For example K1.xxx K2.xyz K2.xyz K3.xyz,138798010000.xyp K3.xyz,138798010000.xyp K4.xyq I have never seen a wierd start key and end key like this. We also suspect a failed split of a region around 20GB. We looked at logs many times but unable to get any sense out of it. Please help us out and we can't afford data loss. Yesterday, There was an cluster crash of root region but we thought we sucessfully restored that.But things did n't go that way.... There was a consitent data loss after that. -- Thank you Kiran Sarvabhotla -----Even a correct decision is wrong when it is taken late
