In my case, region server reboot does NOT fix the problem. Otherwise, behavior is identical to Justin. I can't talk to phoenix, period, across reboots, unless I remove SYSTEM.TABLE and all others and then reboot RSs.
Identical behavior on 2.1.2 and 2.2.2 regarding working for a while (hours to one day) and then becoming unreachable. Ditto on hbase she'll working fine. On Monday, February 17, 2014, Justin Workman <[email protected]> wrote: > Yes. That is the case in our situation. Cluster has been idle for the > weekend. I'll check shortly and see if it is in that state again > today. > > Sent from my iPhone > > > On Feb 17, 2014, at 3:16 PM, lars hofhansl > > <[email protected]<javascript:;>> > wrote: > > > > Interesting. So only Phoenix clients are at issues (normal HBase shell > connection is fine), yet an RS restart rectifies the issue? > > > > ________________________________ > > From: Justin Workman <[email protected] <javascript:;>> > > To: "[email protected] <javascript:;>" < > [email protected] <javascript:;>> > > Sent: Monday, February 17, 2014 10:20 AM > > Subject: Re: Phoenix becomes unreachable > > > > > > > > Specifically with a 2.2.2 installation. However, I believe it has also > happened on our 2.2.3 instance as well. > > > > These are not clients that are always connected. Mostly users adhoc > connections with sqlline. But we have experienced the issue with a long > running java client as well. This effects clients from all machines until > we restart the region servers. > > > > Sent from my iPhone > > > > On Feb 17, 2014, at 11:10 AM, James Taylor > > <[email protected]<javascript:;>> > wrote: > > > > > > We'll try to repro on our end. Is this on 2.2.3 or on 3.0.0-SNAPSHOT? > Does it happen after the client is in use for a day or only after it's > idle? Next time it happens, can you try doing a > PhoenixDriver.INSTANCE.close() on the client to see if it helps? Are you > able to connect through Phoenix to the server from a new client or one on a > different machine? > >> > >> > >> Thanks, > >> James > >> > >> On Monday, February 17, 2014, Justin Workman > >> <[email protected]<javascript:;>> > wrote: > >> > >> We experience the same type of issue. After a day or so clients stop > being able to connect via jdbc, however connecting to hbase using the hbase > shell works fine. > >>> > >>> > >>> Likewise, there doesn't seem to be any log output indicating why, > although I haven't spent much time trying to investigate. A restart of the > hbase region servers seems to resolve the issue. > >>> > >>> > >>> 5 region servers 16gb heap each. No specific memory setting configured > on the client, and effects clients from all machines. > >>> > >>> Sent from my iPhone > >>> > >>> On Feb 17, 2014, at 10:24 AM, James Taylor > >>> <[email protected]<javascript:;>> > wrote: > >>> > >>> > >>> How big a heap have you configured on the client and region server? > Just a wild guess, but it sounds like it could be an OOM error. > >>>> > >>>> > >>>> James > >>>> > >>>> On Monday, February 17, 2014, Russell Jurney < > [email protected] <javascript:;>> wrote: > >>>> > >>>> Has anyone observed this behavior? After running for hours or a > couple days, Phoenix becomes unreachable by JDBC, and sqlline.sh just > freezes. > >>>>> > >>>>> > >>>>> There is no debug output indicating an error. Anyone know how to > debug this? > >>>>> > >>>>> > >>>>> -- > >>>>> Russell Jurney twitter.com/rjurney > >>>>> [email protected]<javascript:;> > datasyndrome.com > -- Russell Jurney twitter.com/rjurney [email protected] datasyndrome.com
