Re: Error: Cluster is being concurrently upgraded from 4.13.x to 5.0.x. Please retry establishing connection.

[John West]

I had to add, we had one corrupt old hbase table during the upgrade on the
second cluster, which prevent hbase from starting correctly.
That corrupt table had been dropped completely


Earlier:
2019-09-24 16:24:36,469 ERROR
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Failed open
of
region=old_metrics,8701203$lamp,1400225340230.551495036e5680480ed00c2045438676.
java.io.IOException: java.io.IOException:
org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem reading
HFile Trailer from file
hdfs://cluster/hbase/data/default/old_metrics/551495036e5680480ed00c2045438676/d/b5ef4e30ce6c4bd09fa6279a4edc68ec
at
org.apache.hadoop.hbase.regionserver.HRegion.initializeStores(HRegion.java:1081)
at
org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionInternals(HRegion.java:942)
at
org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:898)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7241)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7200)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7172)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7130)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7081)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:283)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:108)
at
org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)


On Tue, Sep 24, 2019 at 11:14 PM John West  wrote:

> We are in the process of upgrading two of our CDH clusters from version
> CDH 5.12  (Hbase 1.2.0) to CDH 6.3 (hbase 2.1.0)
> The first cluster went without any hiccup, and Phoenix 5.0.X worked
> without any issue with hbase.
>
> But on the second cluster, which is slightly larger than the first
> cluster,  we keep getting the following error, which I think is related to
> https://issues.apache.org/jira/browse/PHOENIX-4653
>
>  Error: Cluster is being concurrently upgraded from 4.13.x to 5.0.x.
> Please retry establishing connection. (state=INT12,code=2010)
> org.apache.phoenix.exception.UpgradeInProgressException: Cluster is being
> concurrently upgraded from 4.13.x to 5.0.x. Please retry establishing
> connection.
> at
> org.apache.phoenix.query.ConnectionQueryServicesImpl.acquireUpgradeMutex(ConnectionQueryServicesImpl.java:3500)
> at
> org.apache.phoenix.query.ConnectionQueryServicesImpl.upgradeSystemTables(ConnectionQueryServicesImpl.java:3083)
> at
> org.apache.phoenix.query.ConnectionQueryServicesImpl$12.call(ConnectionQueryServicesImpl.java:2625)
> at
> org.apache.phoenix.query.ConnectionQueryServicesImpl$12.call(ConnectionQueryServicesImpl.java:2532)
> at
> org.apache.phoenix.util.PhoenixContextExecutor.call(PhoenixContextExecutor.java:76)
> at
> org.apache.phoenix.query.ConnectionQueryServicesImpl.init(ConnectionQueryServicesImpl.java:2532)
> at
> org.apache.phoenix.jdbc.PhoenixDriver.getConnectionQueryServices(PhoenixDriver.java:255)
> at
> org.apache.phoenix.jdbc.PhoenixEmbeddedDriver.createConnection(PhoenixEmbeddedDriver.java:150)
> at
> org.apache.phoenix.jdbc.PhoenixDriver.connect(PhoenixDriver.java:221)
> at sqlline.DatabaseConnection.connect(DatabaseConnection.java:157)
> at
> sqlline.DatabaseConnection.getConnection(DatabaseConnection.java:203)
> at sqlline.Commands.close(Commands.java:906)
> at sqlline.Commands.closeall(Commands.java:880)
> at sqlline.SqlLine.begin(SqlLine.java:714)
> at sqlline.SqlLine.start(SqlLine.java:398)
> at sqlline.SqlLine.main(SqlLine.java:291)
>
> On hbase shell , list, I noticed there is a SYSTEM.MUTEX table, with
> content as follow:
> hbase(main):002:0> scan 'SYSTEM.MUTEX'
> ROW
>COLUMN+CELL
>  \x00SYSTEM\x00CATALOG
>   column=0:UPGRADE_MUTEX, timestamp=1569387929428,
> value=UPGRADE_MUTEX_LOCKED
> 1 row(s)
> Took 0.1329 seconds
>
> Is it safe to just disable, and drop the SYSTEM.MUTEX table from hbase,
> and try rerunning Phoenix sqlline.py again ?
>
> Thanks
>

Error: Cluster is being concurrently upgraded from 4.13.x to 5.0.x. Please retry establishing connection.

[John West]

We are in the process of upgrading two of our CDH clusters from version CDH
5.12  (Hbase 1.2.0) to CDH 6.3 (hbase 2.1.0)
The first cluster went without any hiccup, and Phoenix 5.0.X worked without
any issue with hbase.

But on the second cluster, which is slightly larger than the first
cluster,  we keep getting the following error, which I think is related to
https://issues.apache.org/jira/browse/PHOENIX-4653

 Error: Cluster is being concurrently upgraded from 4.13.x to 5.0.x. Please
retry establishing connection. (state=INT12,code=2010)
org.apache.phoenix.exception.UpgradeInProgressException: Cluster is being
concurrently upgraded from 4.13.x to 5.0.x. Please retry establishing
connection.
at
org.apache.phoenix.query.ConnectionQueryServicesImpl.acquireUpgradeMutex(ConnectionQueryServicesImpl.java:3500)
at
org.apache.phoenix.query.ConnectionQueryServicesImpl.upgradeSystemTables(ConnectionQueryServicesImpl.java:3083)
at
org.apache.phoenix.query.ConnectionQueryServicesImpl$12.call(ConnectionQueryServicesImpl.java:2625)
at
org.apache.phoenix.query.ConnectionQueryServicesImpl$12.call(ConnectionQueryServicesImpl.java:2532)
at
org.apache.phoenix.util.PhoenixContextExecutor.call(PhoenixContextExecutor.java:76)
at
org.apache.phoenix.query.ConnectionQueryServicesImpl.init(ConnectionQueryServicesImpl.java:2532)
at
org.apache.phoenix.jdbc.PhoenixDriver.getConnectionQueryServices(PhoenixDriver.java:255)
at
org.apache.phoenix.jdbc.PhoenixEmbeddedDriver.createConnection(PhoenixEmbeddedDriver.java:150)
at
org.apache.phoenix.jdbc.PhoenixDriver.connect(PhoenixDriver.java:221)
at sqlline.DatabaseConnection.connect(DatabaseConnection.java:157)
at
sqlline.DatabaseConnection.getConnection(DatabaseConnection.java:203)
at sqlline.Commands.close(Commands.java:906)
at sqlline.Commands.closeall(Commands.java:880)
at sqlline.SqlLine.begin(SqlLine.java:714)
at sqlline.SqlLine.start(SqlLine.java:398)
at sqlline.SqlLine.main(SqlLine.java:291)

On hbase shell , list, I noticed there is a SYSTEM.MUTEX table, with
content as follow:
hbase(main):002:0> scan 'SYSTEM.MUTEX'
ROW
 COLUMN+CELL
 \x00SYSTEM\x00CATALOG
  column=0:UPGRADE_MUTEX, timestamp=1569387929428,
value=UPGRADE_MUTEX_LOCKED
1 row(s)
Took 0.1329 seconds

Is it safe to just disable, and drop the SYSTEM.MUTEX table from hbase, and
try rerunning Phoenix sqlline.py again ?

Thanks

Re: Performance degradation on query analysis

[Josh Elser]

Did you change your configuration to prevent compactions from regularly 
happening, Stepan?


By default, you should have a major compaction run weekly which would 
have fixed this for you, although minor compactions would have run 
automatically as well to rewrite small hfiles as you are creating new 
one (generating new stats).


On 9/19/19 4:50 PM, Ankit Singhal wrote:

Please schedule compaction on SYSTEM.STATS table to clear the old entries.

On Thu, Sep 19, 2019 at 1:48 PM Stepan Migunov 
> wrote:


Thanks, Josh. The problem was really related to reading the SYSTEM.STATS
table.
There were only 8,000 rows in the table, but COUNT took more than 10
minutes. I noticed that the storage files (34) had a total size of
10 GB.

DELETE FROM SYSTEM.STATS did not help - the storage files are still
10 GB,
and COUNT took a long time.
Then I truncated the table from the hbase shell. And this fixed the
problem - after UPDATE STATS for each table, everything works fine.

Are there any known issues with SYSTEM.STATS table? Apache Phoenix
4.13.1
with 15 Region Servers.

-Original Message-
From: Josh Elser [mailto:els...@apache.org ]
Sent: Tuesday, September 17, 2019 5:16 PM
To: user@phoenix.apache.org 
Subject: Re: Performance degradation on query analysis

Can you share the output you see from the EXPLAIN? Does it differ
between
times it's "fast" and times it's "slow"?

Sharing the table(s) DDL statements would also help, along with the
shape
and version of your cluster (e.g. Apache Phoenix 4.14.2 with 8
RegionServers).

Spit-balling ideas:

Could be reads over the SYSTEM.CATALOG table or the SYSTEM.STATS table.

Have you looked more coarsely at the RegionServer logs/metrics? Any
obvious
saturation issues (e.g. handlers consumed, JVM GC pauses, host CPU
saturation)?

Turn on DEBUG log4j client side (beware of chatty ZK logging) and see if
there's something obvious from when the EXPLAIN is slow.

On 9/17/19 3:58 AM, Stepan Migunov wrote:
 > Hi
 > We have an issue with our production environment - from time to
time we
 > notice a significant performance degradation for some queries.
The strange
 > thing is that the EXPLAIN operator for these queries takes the
same time
 > as queries execution (5 minutes or more). So, I guess, the issue is
 > related to query's analysis but not data extraction. Is it
possible that
 > issue is related to SYSTEM.STATS access problem? Any other ideas?
 >

Re: What is the phoenix-queryserver-client-4.14.2-HBase-1.4.jar?

[Josh Elser]

Please be aware that you're referencing code that hasn't been updated in 
3 years. Take it with a grain of salt.


On 9/19/19 6:42 PM, jesse wrote:
Josh, in your sample project pom.xml file, the following build 
dependence is not needed:



org.apache.phoenix
phoenix-server-client
4.7.0-HBase-1.1-SNAPSHOT



On Thu, Sep 19, 2019, 10:53 AM jesse > wrote:


A) phoenix-4.14.2-HBase-1.4-thin-client.jar

Just A) is good enough, Josh, you had a sample program here:
https://github.com/joshelser/phoenix-queryserver-jdbc-client

And the phoenix-4.14.2-HBase-1.4-thin-client.jar already contains
the org.apache.phoenix.queryserver.client.Driver






On Thu, Sep 19, 2019, 8:54 AM jesse mailto:chat2je...@gmail.com>> wrote:

You confused me more, if I write a Java program with http
endpoint to PQS for Phoenix read/write functions, should I depend on

A) phoenix-4.14.2-HBase-1.4-thin-client.jar

B) phoenix-queryserver-client-4.14.2-HBase-1.4.jar

C) both



On Thu, Sep 19, 2019, 4:12 AM Josh Elser mailto:els...@apache.org>> wrote:

"phoenix-queryserver-client" is the name of the Maven module
which holds
the required code for the "JDBC thin client", aka PQS
client, aka
"queryserver client".

Maven convention is that a jar with the name of the Maven
module is created.

However, the majority of the code for the thin client is
pulled from
another Apache project. In fact, we only have one piece of
code that we
maintain client-side to connect to PQS.

That third party code _does_ need to be included on the
classpath for
you to use the client. Thus, a shaded jar is created, with the
human-readable name "thin-client" to make it very clear to
you that this
is the jar the use.

The Maven build shows how all of this work.

On 9/18/19 8:04 PM, jesse wrote:
 > It seems it is just the sqllinewrapper client, so
confusing name...
 >
 >
 >
 > On Wed, Sep 18, 2019, 4:46 PM jesse mailto:chat2je...@gmail.com>
 > >> wrote:
 >
 >     For query via PQS, we are using
phoenix-4.14.2-HBase-1.4-thin-client.jar
 >
 >     Then what is purpose and usage
 >     of phoenix-queryserver-client-4.14.2-HBase-1.4.jar?
 >
 >     Thanks
 >

Re: What is the phoenix-queryserver-client-4.14.2-HBase-1.4.jar?

[Josh Elser]

Yes. Also, this is what I said originally:

> the human-readable name "thin-client" to make it very clear to you 
that this is the jar the use.


We try to be consistent everywhere with the phrase "thin client" to 
indicate what you use to interact with PQS.


On 9/19/19 1:53 PM, jesse wrote:

A) phoenix-4.14.2-HBase-1.4-thin-client.jar

Just A) is good enough, Josh, you had a sample program here:
https://github.com/joshelser/phoenix-queryserver-jdbc-client

And the phoenix-4.14.2-HBase-1.4-thin-client.jar already contains the 
org.apache.phoenix.queryserver.client.Driver







On Thu, Sep 19, 2019, 8:54 AM jesse > wrote:


You confused me more, if I write a Java program with http endpoint
to PQS for Phoenix read/write functions, should I depend on

A) phoenix-4.14.2-HBase-1.4-thin-client.jar

B) phoenix-queryserver-client-4.14.2-HBase-1.4.jar

C) both



On Thu, Sep 19, 2019, 4:12 AM Josh Elser mailto:els...@apache.org>> wrote:

"phoenix-queryserver-client" is the name of the Maven module
which holds
the required code for the "JDBC thin client", aka PQS client, aka
"queryserver client".

Maven convention is that a jar with the name of the Maven module
is created.

However, the majority of the code for the thin client is pulled
from
another Apache project. In fact, we only have one piece of code
that we
maintain client-side to connect to PQS.

That third party code _does_ need to be included on the
classpath for
you to use the client. Thus, a shaded jar is created, with the
human-readable name "thin-client" to make it very clear to you
that this
is the jar the use.

The Maven build shows how all of this work.

On 9/18/19 8:04 PM, jesse wrote:
 > It seems it is just the sqllinewrapper client, so confusing
name...
 >
 >
 >
 > On Wed, Sep 18, 2019, 4:46 PM jesse mailto:chat2je...@gmail.com>
 > >>
wrote:
 >
 >     For query via PQS, we are using
phoenix-4.14.2-HBase-1.4-thin-client.jar
 >
 >     Then what is purpose and usage
 >     of phoenix-queryserver-client-4.14.2-HBase-1.4.jar?
 >
 >     Thanks
 >