While you continue on the snapshot approach, have you tried to Export the table in 0.94 to hdfs, and then Import the data from hdfs to 0.98? On Sep 15, 2014 10:19 PM, "Matteo Bertozzi" <[email protected]> wrote:
> can you post the full exception and the file path ? > maybe there is a bug in looking up the reference file. > It seems to not be able to find enough data in the file... > > Matteo > > > On Mon, Sep 15, 2014 at 10:08 PM, Gautam <[email protected]> wrote: > > > Thanks for the reply Matteo. > > > > This is exactly what I did. I modified the source cluster's dir structure > > to mimic that of the 98 cluster. I even got as far as it trying to look > > through the reference files. > > > > I end up with this exception : > > > > 14/09/15 23:34:59 ERROR snapshot.ExportSnapshot: Snapshot export failed > > java.io.IOException > > at > > > > > org.apache.hadoop.hbase.snapshot.SnapshotManifestV1.loadRegionManifests(SnapshotManifestV1.java:145) > > at > > > > > org.apache.hadoop.hbase.snapshot.SnapshotManifest.load(SnapshotManifest.java:265) > > at > > > > > org.apache.hadoop.hbase.snapshot.SnapshotManifest.open(SnapshotManifest.java:119) > > at > > > > > org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil.visitTableStoreFiles(SnapshotReferenceUtil.java:125) > > ... > > .. > > Caused by: java.io.IOException: read=-1, wanted=4 > > at org.apache.hadoop.hbase.io.Reference.read(Reference.java:175) > > at > > > > > org.apache.hadoop.hbase.regionserver.StoreFileInfo.<init>(StoreFileInfo.java:115) > > at > > > > > org.apache.hadoop.hbase.regionserver.HRegionFileSystem.getStoreFiles(HRegionFileSystem.java > > .. > > > > > > This and Ted's reply about HBASE-7987 leads me to believe that the export > > tool from my distro is in capable of working around the regionManifest > file > > requirement. I'm now left with the option of downgrading my dest cluster > to > > 94, copying data and then upgrading using the upgrade migration tool. > > Wanted to know if others have tried this or there are other things I can > > do. If not, i'l just go ahead and do this :-) > > > > > > Cheers, > > -Gautam. > > > > > > > > On Mon, Sep 15, 2014 at 8:10 PM, Matteo Bertozzi < > [email protected]> > > wrote: > > > > > 94 and 98 differs in directory layout > > > so 98 is not able to read 94 layout unless you run the migration tool > > > which is basically moving all the data in a "default" namespace > directory > > > e.g. > > > /hbase/table -> /hbase/data/default/table > > > /hbase/.archive/table -> /hbase/archive/default/table > > > > > > Matteo > > > > > > > > > On Mon, Sep 15, 2014 at 6:17 PM, Gautam <[email protected]> > wrote: > > > > > > > Yep, looks like the CDH distro backports HBASE-7987. Having said > that, > > is > > > > there a transition path for us or are we hosed :-) ? In general, > what's > > > the > > > > recommended way to achieve this, at this point I feel i'm going > around > > > the > > > > system to achieve what I want. If nothing else works with export > > snapshot > > > > should I just downgrade to 94, export snapshot and then upgrade to > 98? > > Is > > > > the upgrade migration path different from what export snapshot does > > (i'd > > > > imagine yes)? > > > > > > > > Cheers, > > > > -Gautam. > > > > > > > > > > > > > > > > > > > > On Mon, Sep 15, 2014 at 5:14 PM, Ted Yu <[email protected]> wrote: > > > > > > > > > bq. 98.1 on dest cluster > > > > > > > > > > Looking at the history for SnapshotManifestV1, it came with > > HBASE-7987 > > > > > which went to 0.99.0 > > > > > > > > > > Perhaps you're using a distro with HBASE-7987 ? > > > > > > > > > > On Mon, Sep 15, 2014 at 4:58 PM, Gautam <[email protected]> > > > wrote: > > > > > > > > > > > Hello, > > > > > > I'm trying to copy data between Hbase clusters on > > different > > > > > > versions. I am using : > > > > > > > > > > > > /usr/bin/hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot > > > > > > -chuser hbase > > > > > > -chgroup hadoop > > > > > > -snapshot msg_snapshot > > > > > > -mappers 50 > > > > > > -copy-from hftp://src-cluster:50070/hbase > > > > > > -copy-to hdfs:/dest-cluster:8020/hbase > > > > > > > > > > > > > > > > > > Till now, based on various tips from the mailing list, I have > > > modified > > > > > the > > > > > > source cluster data dir paths to mimic the 98 convention > (archive, > > > > table > > > > > > data paths, etc). This helped in jumping some roadblocks but not > > all. > > > > > > > > > > > > This is what I see now : > > > > > > > > > > > > > > > > > > 14/09/15 23:34:59 ERROR snapshot.ExportSnapshot: Snapshot export > > > failed > > > > > > java.io.IOException > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.hadoop.hbase.snapshot.SnapshotManifestV1.loadRegionManifests(SnapshotManifestV1.java:145) > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.hadoop.hbase.snapshot.SnapshotManifest.load(SnapshotManifest.java:265) > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.hadoop.hbase.snapshot.SnapshotManifest.open(SnapshotManifest.java:119) > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil.visitTableStoreFiles(SnapshotReferenceUtil.java:125) > > > > > > ... > > > > > > .. > > > > > > Caused by: java.io.IOException: read=-1, wanted=4 > > > > > > at org.apache.hadoop.hbase.io.Reference.read(Reference.java:175) > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.hadoop.hbase.regionserver.StoreFileInfo.<init>(StoreFileInfo.java:115) > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.hadoop.hbase.regionserver.HRegionFileSystem.getStoreFiles(HRegionFileSystem.java > > > > > > .. > > > > > > > > > > > > > > > > > > Fails while trying to read refernce hfile. Is this something > folks > > > have > > > > > > done before and/or is possible to do? I'd really like to do this > > > > without > > > > > > having to upgrade my source cluster or downgrade my dest cluster. > > > > > > > > > > > > I'm using 94.6 on source cluster and 98.1 on dest cluster. > > > > > > > > > > > > > > > > > > Cheers, > > > > > > -Gautam. > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > "If you really want something in this life, you have to work for it. > > Now, > > > > quiet! They're about to announce the lottery numbers..." > > > > > > > > > > > > > > > -- > > "If you really want something in this life, you have to work for it. Now, > > quiet! They're about to announce the lottery numbers..." > > >
