Hi devs, [Had a quick chat with Lars G. about this and before opening a Jira I thought I'd raise it here first]
We have just experienced data loss in HBase 1.0.0-cdh5.4.10. Before I dig into this further, I'd like to just ask if anyone has seen this before? The initial state was a table (tim_test) built with MOB support and a few 10's million rows and 10's billions of cells. I wanted to rename the table to get this into production and did so as follows: snapshot 'tim_test', 'tim_test-snapshot' clone_snapshot 'tim_test-snapshot', 'prod_b_map' At this stage the application all looked good, and so I continued with: delete_snapshot 'tim_test-snapshot' disable 'tim_test' drop ‘tim_test’ Then things went... awry and data just started dropping out in the app. Before long, all MOB data seemingly is gone. The references in the new table MOB folder appear to point to the source table (e.g. /hbase/mobdir/data/default/prod_b_map/ba42a2e8e9b669d9fc85bdfeed2f5f2a/EPSG_4326/tim_test=14bf5f1737ac65c34615ed97c0b7de06-d41d8cd98f00b204e9800998ecf8427e20161006ff8baa70d21f408caefe8ae6318dfba2). The RS logs full of ERROR like: 2016-10-12 15:19:14,640 ERROR org.apache.hadoop.hbase.regionserver.HStore: The mob file d41d8cd98f00b204e9800998ecf8427e20161006b59865f80e604781a79ebfa2ddd66b48 could not be found in the locations [hdfs://ha-nn/hbase/mobdir/data/default/tim_test/14bf5f1737ac65c34615ed97c0b7de06/EPSG_4326, hdfs://ha-nn/hbase/archive/data/default/tim_test/14bf5f1737ac65c34615ed97c0b7de06/EPSG_4326] What I don't know is: 1) was this running a background task to copy the MOB data when the snapshot was cloned and I just deleted the source before the copy was complete? - or 2) when running "snapshot and clone" it just references the source MOB data until a (?) change? 3) snapshot and clone just doesn't support MOB? Can anyone shed some light on this easily before I dig into it please? While this situation exists (at least in 1.0.0) might it be good to get info about data loss for MOB tables into the snapshot clone docs? Thanks, Tim
