We have found and fixed the problem now. It was related faulty upgrade script where the server started before the phoenix jar was put in the lib directory.
I suspect range queries and inserts that spanned rows belonging to this particular region server failed, while other queries succeeded. Thanks for your help James! Cheers, -Kristoffer On Tue, Sep 30, 2014 at 1:20 AM, James Taylor <jamestay...@apache.org> wrote: > Hi Kristoffer, > Did something change on your cluster prior to the breaking of your > SELECT queries? Did it ever work? > Thanks, > James > > On Mon, Sep 29, 2014 at 4:06 PM, Kristoffer Sjögren <sto...@gmail.com> > wrote: > > Thank for quick reply. Yes, we will try to upgrade as soon as we can get > the > > servers up on 2.2.3. If we can't then we go straight to upgrade. > > > > The region servers are running phoenix-2.2.3-incubating.jar found in [1]. > > Are the clients suppose to use the exact same jar? The jar files have > same > > name but are indeed slightly different in size between client (1542522 > > bytes) and server (1540202 bytes). Bummer that the jar MANIFEST does not > > provide enough build information. > > > > I'm still puzzled why sqlline.sh [1] have the same problem though? Maybe > > something with phoenix-2.2.3-incubating-client.jar? > > > > [1] > > > http://apache.mirrors.spacedump.net/incubator/phoenix/phoenix-2.2.3-incubating/bin/phoenix-2.2.3-incubating.tar.gz > > > > > > On Tue, Sep 30, 2014 at 12:34 AM, James Taylor <jamestay...@apache.org> > > wrote: > >> > >> Hi Kristoffer, > >> Did something change on your cluster prior to the breaking of your > >> SELECT queries? An upgrade of something? > >> > >> From the look of the exceptions, it seems like the Phoenix client jar > >> and server jar are out-of-sync, because the client is sending > >> information in a format that's no longer understood by the server. > >> > >> I do encourage you to upgrade, though, once you get your production > >> servers functioning again. > >> > >> Thanks, > >> James > >> > >> On Mon, Sep 29, 2014 at 3:17 PM, Kristoffer Sjögren <sto...@gmail.com> > >> wrote: > >> > Hi > >> > > >> > Our SELECT queries went broken just now and we're seeing the following > >> > exception [1] using Apache Phoenix 2.2.3-incubating with HBase 0.94.6 > >> > (0.94.6-cdh4.4.0). > >> > > >> > After a some testing I realized that sqlline.sh from the > >> > 2.2.3-incubating > >> > tarball is also broken all of a sudden (with same error)? > >> > > >> > Seems like SELECT ... WHERE queries are problematic whereas basic > >> > queries > >> > like "SELECT * FROM COUNTERS LIMIT 1;" works fine. Simple INSERT > >> > statements > >> > also seems to be working. > >> > > >> > Is upgrading to phoenix-3.0.0-incubating an option? Or is there a way > to > >> > debug this further? I tried to enable DEBUG in log4j.properties using > >> > the > >> > 'console' in sqlline but without success. > >> > > >> > Our production servers are pretty much down at the moment... > >> > > >> > Cheers, > >> > -Kristoffer > >> > > >> > [1] http://pastebin.com/H2a6xpBX > >> > > > > > >
