Hi James/Lars, I did another run of bulk load, backup & restore on a new test cluster but did not encounter the issue this time. It appears to be some specific issue with that version of backup. I will pass on the steps, as soon as I can, to replicate it with a small dataset.
Having said that, it is still beneficial to have some kind of an option in CREATE TABLE & CREATE INDEX clauses to indicate that underlying table is already a phoenix table. Thanks, Krishna On Monday, September 29, 2014, lars hofhansl <la...@apache.org> wrote: > Not offhand. > > A few guesses/questions for Krihna: > - does the restore tool restore the exact timestamps? In fact how exactly > do you backup and restore the tables? > - is it possible that you updated the CATALOG (via some DDL) while the > backup was running? (although I think that should be OK) > > - does it restore into a clean(ed) cluster? If cleaned, was the ZK state > removed? > - after you restore the two tables, can you scan SYSTEM.CATALOG from an > HBase shell? Or does that part hang? > - in the HBase UI, are there any regions in transition? (might be some > outdated coprocessors, etc, although unlikely) > - exact same version of Phoenix? (between the Phoenix that wrote the data > and the one that restored it) > - same for HBase. Exact same version used for the restore? > > As James said, we need some logs from both the client and the region > server(s). > Also if possible, do you have some precise steps to reproduce with a small > table? > > Lots of questions :) > > Thanks. > > -- Lars > > > ----- Original Message ----- > From: James Taylor <jamestay...@apache.org <javascript:;>> > To: user <user@phoenix.apache.org <javascript:;>>; lars hofhansl < > la...@apache.org <javascript:;>> > Cc: > Sent: Monday, September 29, 2014 4:26 PM > Subject: Re: Recreating SYSTEM.CATALOG metadata > > @Lars - any idea why Krishna may run into issues using Phoenix after a > restore from an HBase backup? > > > > > On Sun, Sep 28, 2014 at 9:00 PM, James Taylor <jamestay...@apache.org > <javascript:;>> wrote: > > Hi Krishna, > > I think that's what we need to figure out - why is Phoenix having > > trouble when you restore the SYSTEM.CATALOG table. Any client or > > server-side exceptions in the logs when it hangs? > > Thanks, > > James > > > > On Sun, Sep 28, 2014 at 6:18 PM, Krishna <research...@gmail.com > <javascript:;>> wrote: > >> Hi James, > >> > >> I did include SYSTEM.CATALOG table in the backup & restore process. I > dont > >> know why sqlline is having trouble using the backup version - it just > hangs, > >> unless catalog table is dropped from hbase shell. I will see if there > are > >> any errors in logs that's causing the behavior. > >> > >> Regards > >> Krishna > >> > >> > >> On Sunday, September 28, 2014, James Taylor <jamestay...@apache.org > <javascript:;>> wrote: > >>> > >>> Hi Krishna, > >>> Any reason why the SYSTEM.CATALOG hbase table isn't restored as well > >>> from backup? Yes, if you try to re-create the SYSTEM.CATALOG by > >>> re-issuing your DDL statement, Phoenix won't know that the tables were > >>> Phoenix tables before, so will try to add the empty key value to each > >>> row. It's possible that an option could be made to avoid this, but > >>> it'd be good to understand a bit more why this is needed. > >>> Thanks, > >>> James > >>> > >>> On Sun, Sep 28, 2014 at 1:56 PM, Krishna <research...@gmail.com > <javascript:;>> wrote: > >>> > Hi, > >>> > > >>> > When I restore hbase from a backup, sqlline gets stuck unless > >>> > SYSTEM.CATALOG > >>> > table is dropped. It is automatically re-created via sqlline. > However, > >>> > metadata of previously created phoenix tables is lost. > >>> > > >>> > So, to restore the metadata, when a 'CREATE TABLE' statement is > >>> > re-issued, > >>> > Phoenix takes very, very long time to execute which I think, is > because > >>> > Phoenix is upserting null value for _0 column qualifier again for > every > >>> > row > >>> > (billions of rows in the table). > >>> > > >>> > Is there a work-around for this by somehow telling Phoenix that the > >>> > underlying table is already a Phoenix table? Using views is not an > >>> > option > >>> > here. If my understanding is wrong, any pointers as to why it takes > so > >>> > long > >>> > for executing 'CREATE TABLE' statement is appreciated. > >>> > > >>> > Thanks > >>> > > >>> > > >
