Not offhand. A few guesses/questions for Krihna: - does the restore tool restore the exact timestamps? In fact how exactly do you backup and restore the tables? - is it possible that you updated the CATALOG (via some DDL) while the backup was running? (although I think that should be OK)
- does it restore into a clean(ed) cluster? If cleaned, was the ZK state removed? - after you restore the two tables, can you scan SYSTEM.CATALOG from an HBase shell? Or does that part hang? - in the HBase UI, are there any regions in transition? (might be some outdated coprocessors, etc, although unlikely) - exact same version of Phoenix? (between the Phoenix that wrote the data and the one that restored it) - same for HBase. Exact same version used for the restore? As James said, we need some logs from both the client and the region server(s). Also if possible, do you have some precise steps to reproduce with a small table? Lots of questions :) Thanks. -- Lars ----- Original Message ----- From: James Taylor <jamestay...@apache.org> To: user <user@phoenix.apache.org>; lars hofhansl <la...@apache.org> Cc: Sent: Monday, September 29, 2014 4:26 PM Subject: Re: Recreating SYSTEM.CATALOG metadata @Lars - any idea why Krishna may run into issues using Phoenix after a restore from an HBase backup? On Sun, Sep 28, 2014 at 9:00 PM, James Taylor <jamestay...@apache.org> wrote: > Hi Krishna, > I think that's what we need to figure out - why is Phoenix having > trouble when you restore the SYSTEM.CATALOG table. Any client or > server-side exceptions in the logs when it hangs? > Thanks, > James > > On Sun, Sep 28, 2014 at 6:18 PM, Krishna <research...@gmail.com> wrote: >> Hi James, >> >> I did include SYSTEM.CATALOG table in the backup & restore process. I dont >> know why sqlline is having trouble using the backup version - it just hangs, >> unless catalog table is dropped from hbase shell. I will see if there are >> any errors in logs that's causing the behavior. >> >> Regards >> Krishna >> >> >> On Sunday, September 28, 2014, James Taylor <jamestay...@apache.org> wrote: >>> >>> Hi Krishna, >>> Any reason why the SYSTEM.CATALOG hbase table isn't restored as well >>> from backup? Yes, if you try to re-create the SYSTEM.CATALOG by >>> re-issuing your DDL statement, Phoenix won't know that the tables were >>> Phoenix tables before, so will try to add the empty key value to each >>> row. It's possible that an option could be made to avoid this, but >>> it'd be good to understand a bit more why this is needed. >>> Thanks, >>> James >>> >>> On Sun, Sep 28, 2014 at 1:56 PM, Krishna <research...@gmail.com> wrote: >>> > Hi, >>> > >>> > When I restore hbase from a backup, sqlline gets stuck unless >>> > SYSTEM.CATALOG >>> > table is dropped. It is automatically re-created via sqlline. However, >>> > metadata of previously created phoenix tables is lost. >>> > >>> > So, to restore the metadata, when a 'CREATE TABLE' statement is >>> > re-issued, >>> > Phoenix takes very, very long time to execute which I think, is because >>> > Phoenix is upserting null value for _0 column qualifier again for every >>> > row >>> > (billions of rows in the table). >>> > >>> > Is there a work-around for this by somehow telling Phoenix that the >>> > underlying table is already a Phoenix table? Using views is not an >>> > option >>> > here. If my understanding is wrong, any pointers as to why it takes so >>> > long >>> > for executing 'CREATE TABLE' statement is appreciated. >>> > >>> > Thanks >>> > >>> >
