Thanks Ted, Huaxiang I'll move this to a Cloudera forum and comment back here if it appears unrelated.
On Wed, Oct 12, 2016 at 7:24 PM, Huaxiang Sun <h...@cloudera.com> wrote: > By the way, I forgot the forum link: http://community.cloudera.com < > http://community.cloudera.com/> > > Thanks, > Huaxiang > > > On Oct 12, 2016, at 10:10 AM, Huaxiang Sun <h...@cloudera.com> wrote: > > > > Hi Tim, > > > > I believe that it runs into an issue which is specific to cloudera > release we fixed recently. For details, could you discuss it in cdh forum? > > Copy me(h...@cloudera.com <mailto:h...@cloudera.com>) in the forum so I > can explain more there. > > > > Thanks, > > Huaxiang > > > >> On Oct 12, 2016, at 8:13 AM, Ted Yu <yuzhih...@gmail.com <mailto: > yuzhih...@gmail.com>> wrote: > >> > >> Have you looked at HBASE-16578 ? > >> > >> Cheers > >> > >>> On Oct 12, 2016, at 8:02 AM, Tim Robertson <timrobertson...@gmail.com > <mailto:timrobertson...@gmail.com>> wrote: > >>> > >>> Hi devs, > >>> [Had a quick chat with Lars G. about this and before opening a Jira I > >>> thought I'd raise it here first] > >>> > >>> We have just experienced data loss in HBase 1.0.0-cdh5.4.10. > >>> > >>> Before I dig into this further, I'd like to just ask if anyone has seen > >>> this before? > >>> > >>> The initial state was a table (tim_test) built with MOB support and a > few > >>> 10's million rows and 10's billions of cells. > >>> > >>> I wanted to rename the table to get this into production and did so as > >>> follows: > >>> > >>> snapshot 'tim_test', 'tim_test-snapshot' > >>> clone_snapshot 'tim_test-snapshot', 'prod_b_map' > >>> > >>> At this stage the application all looked good, and so I continued with: > >>> > >>> delete_snapshot 'tim_test-snapshot' > >>> disable 'tim_test' > >>> drop ‘tim_test’ > >>> > >>> Then things went... awry and data just started dropping out in the app. > >>> Before long, all MOB data seemingly is gone. > >>> > >>> The references in the new table MOB folder appear to point to the > source > >>> table (e.g. > >>> /hbase/mobdir/data/default/prod_b_map/ba42a2e8e9b669d9fc85bdfeed2f5f > 2a/EPSG_4326/tim_test=14bf5f1737ac65c34615ed97c0b7de06- > d41d8cd98f00b204e9800998ecf8427e20161006ff8baa70d21f408caefe8ae6318dfba2). > >>> > >>> The RS logs full of ERROR like: > >>> > >>> 2016-10-12 15:19:14,640 ERROR org.apache.hadoop.hbase. > regionserver.HStore: > >>> The mob file > >>> d41d8cd98f00b204e9800998ecf8427e20161006b59865f80e604781a79e > bfa2ddd66b48 > >>> could not be found in the locations > >>> [hdfs://ha-nn/hbase/mobdir/data/default/tim_test/ > 14bf5f1737ac65c34615ed97c0b7de06/EPSG_4326 <hdfs://ha-nn/hbase/mobdir/ > data/default/tim_test/14bf5f1737ac65c34615ed97c0b7de06/EPSG_4326>, > >>> hdfs://ha-nn/hbase/archive/data/default/tim_test/ > 14bf5f1737ac65c34615ed97c0b7de06/EPSG_4326] <hdfs://ha-nn/hbase/archive/ > data/default/tim_test/14bf5f1737ac65c34615ed97c0b7de06/EPSG_4326]> > >>> > >>> What I don't know is: > >>> 1) was this running a background task to copy the MOB data when the > >>> snapshot was cloned and I just deleted the source before the copy was > >>> complete? > >>> - or > >>> 2) when running "snapshot and clone" it just references the source MOB > >>> data until a (?) change? > >>> 3) snapshot and clone just doesn't support MOB? > >>> > >>> Can anyone shed some light on this easily before I dig into it please? > >>> > >>> While this situation exists (at least in 1.0.0) might it be good to get > >>> info about data loss for MOB tables into the snapshot clone docs? > >>> > >>> Thanks, > >>> Tim > > > >
