Thanks for confirming the workaround, Ashish. I filed this JIRA:
https://issues.apache.org/jira/browse/PHOENIX-1159.
James
On Sun, Aug 10, 2014 at 3:35 PM, ashish tapdiya <ashishtapd...@gmail.com> wrote:
> Gabriel,
>
> Suggested work around of disabling secondary works like a charm. Thanks for
> your help.
>
> ~Ashish
>
>
> On Thu, Aug 7, 2014 at 2:59 PM, ashish tapdiya <ashishtapd...@gmail.com>
> wrote:
>>
>> Gabriel,
>>
>> Thanks for taking the pain to create the test case. I appreciate it. I am
>> currently not using secondary indexing. I will try out the work around you
>> suggested and post result here. If not I will upgrade.
>>
>> ~Ashish
>>
>>
>> On Thu, Aug 7, 2014 at 2:07 PM, Gabriel Reid <gabriel.r...@gmail.com>
>> wrote:
>>>
>>> Hi Ashish,
>>>
>>> Thanks for the additional info. I just made an actual test case of
>>> this, and I can reproduce the issue locally. It appears to be related
>>> to the secondary indexing coprocessor, and only seems to be an issue
>>> with HBase 0.94.x and Phoenix 3.x.
>>>
>>> A temporary work-around for this is to disable the secondary indexing
>>> coprocessor (if you're not relying on secondary indexing). You can do
>>> this programmatically as follows:
>>>
>>> HBaseAdmin hbaseAdmin = new HBaseAdmin(getTestClusterConfig());
>>>
>>> HTableDescriptor htd = hbaseAdmin.getTableDescriptor(tableName);
>>> htd.removeCoprocessor("org.apache.phoenix.hbase.index.Indexer");
>>>
>>> hBaseAdmin.disableTable(tableName);
>>> hBaseAdmin.modifyTable(tableName, htd);
>>> hBaseAdmin.enableTable(tableName);
>>>
>>> Another option is to upgrade to HBase 0.98 and Phoenix 4, but that is
>>> obviously a bigger issue.
>>>
>>> Are you currently making use of secondary indexing?
>>>
>>> - Gabriel
>>>
>>> On Wed, Aug 6, 2014 at 8:57 PM, ashish tapdiya <ashishtapd...@gmail.com>
>>> wrote:
>>> > Gabriel,
>>> >
>>> > Nothing in region server's or master logs. However, i do see some stuff
>>> > in
>>> > zookeeper's log,
>>> >
>>> > 2014-08-06 12:15:30,621 [myid:] - INFO
>>> > [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] -
>>> > Accepted socket connection from /192.168.1.102:38727
>>> > 2014-08-06 12:15:30,623 [myid:] - INFO
>>> > [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@868] -
>>> > Client
>>> > attempting to establish new session at /192.168.1.102:38$
>>> > 2014-08-06 12:15:30,645 [myid:] - INFO
>>> > [SyncThread:0:ZooKeeperServer@617] -
>>> > Established session 0x14658117d544452 with negotiated timeout 40000 for
>>> > client /192.168.1.10$
>>> > 2014-08-06 12:20:58,741 [myid:] - WARN
>>> > [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@357] - caught
>>> > end
>>> > of stream exception
>>> > EndOfStreamException: Unable to read additional data from client
>>> > sessionid
>>> > 0x14658117d544452, likely client has closed socket
>>> > at
>>> > org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
>>> > at
>>> >
>>> > org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
>>> > at java.lang.Thread.run(Thread.java:744
>>> >
>>> >
>>> >
>>> > But everything works fine if i do scan,get.
>>> >
>>> > Thanks,
>>> > ~Ashish
>>> >
>>> >
>>> > On Wed, Aug 6, 2014 at 1:24 AM, Gabriel Reid <gabriel.r...@gmail.com>
>>> > wrote:
>>> >>
>>> >> Is there anything in the regionserver logs that gives any additional
>>> >> information (e.g. an additional stack trace?)
>>> >>
>>> >> On Tue, Aug 5, 2014 at 9:23 PM, ashish tapdiya
>>> >> <ashishtapd...@gmail.com>
>>> >> wrote:
>>> >> > Hi Gabriel,
>>> >> >
>>> >> > Thanks for your response. Below is the stack trace
>>> >> >
>>> >> > org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed
>>> >> > after
>>> >> > attempts=14, exceptions:
>>> >> > Tue Aug 05 14:11:49 CDT 2014,
>>> >> > org.apache.hadoop.hbase.client.HTable$9@30a1e97a,
>>> >> > java.io.IOException:
>>> >> > java.io.IOException
>>> >> > : java.lang.NullPointerException
>>> >> > Tue Aug 05 14:11:50 CDT 2014,
>>> >> > org.apache.hadoop.hbase.client.HTable$9@30a1e97a,
>>> >> > java.io.IOException:
>>> >> > java.io.IOException
>>> >> > : java.lang.NullPointerException
>>> >> > Tue Aug 05 14:11:51 CDT 2014,
>>> >> > org.apache.hadoop.hbase.client.HTable$9@30a1e97a,
>>> >> > java.io.IOException:
>>> >> > java.io.IOException
>>> >> > : java.lang.NullPointerException
>>> >> > Tue Aug 05 14:11:53 CDT 2014,
>>> >> > org.apache.hadoop.hbase.client.HTable$9@30a1e97a,
>>> >> > java.io.IOException:
>>> >> > java.io.IOException
>>> >> > : java.lang.NullPointerException
>>> >> > Tue Aug 05 14:11:55 CDT 2014,
>>> >> > org.apache.hadoop.hbase.client.HTable$9@30a1e97a,
>>> >> > java.io.IOException:
>>> >> > java.io.IOException
>>> >> > : java.lang.NullPointerException
>>> >> > Tue Aug 05 14:11:59 CDT 2014,
>>> >> > org.apache.hadoop.hbase.client.HTable$9@30a1e97a,
>>> >> > java.io.IOException:
>>> >> > java.io.IOException
>>> >> > : java.lang.NullPointerException
>>> >> > Tue Aug 05 14:12:03 CDT 2014,
>>> >> > org.apache.hadoop.hbase.client.HTable$9@30a1e97a,
>>> >> > java.io.IOException:
>>> >> > java.io.IOException
>>> >> > : java.lang.NullPointerException
>>> >> > Tue Aug 05 14:12:11 CDT 2014,
>>> >> > org.apache.hadoop.hbase.client.HTable$9@30a1e97a,
>>> >> > java.io.IOException:
>>> >> > java.io.IOException
>>> >> > : java.lang.NullPointerException
>>> >> > Tue Aug 05 14:12:27 CDT 2014,
>>> >> > org.apache.hadoop.hbase.client.HTable$9@30a1e97a,
>>> >> > java.io.IOException:
>>> >> > java.io.IOException
>>> >> > : java.lang.NullPointerException
>>> >> > Tue Aug 05 14:12:59 CDT 2014,
>>> >> > org.apache.hadoop.hbase.client.HTable$9@30a1e97a,
>>> >> > java.io.IOException:
>>> >> > java.io.IOException
>>> >> > : java.lang.NullPointerException
>>> >> > Tue Aug 05 14:14:03 CDT 2014,
>>> >> > org.apache.hadoop.hbase.client.HTable$9@30a1e97a,
>>> >> > java.io.IOException:
>>> >> > java.io.IOException
>>> >> > : java.lang.NullPointerException
>>> >> > Tue Aug 05 14:15:07 CDT 2014,
>>> >> > org.apache.hadoop.hbase.client.HTable$9@30a1e97a,
>>> >> > java.io.IOException:
>>> >> > java.io.IOException
>>> >> > : java.lang.NullPointerException
>>> >> > Tue Aug 05 14:16:12 CDT 2014,
>>> >> > org.apache.hadoop.hbase.client.HTable$9@30a1e97a,
>>> >> > java.io.IOException:
>>> >> > java.io.IOException
>>> >> > : java.lang.NullPointerException
>>> >> > Tue Aug 05 14:17:16 CDT 2014,
>>> >> > org.apache.hadoop.hbase.client.HTable$9@30a1e97a,
>>> >> > java.io.IOException:
>>> >> > java.io.IOException
>>> >> > : java.lang.NullPointerException
>>> >> >
>>> >> > at
>>> >> >
>>> >> >
>>> >> > org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:187)
>>> >> > at
>>> >> > org.apache.hadoop.hbase.client.HTable.checkAndPut(HTable.java:946)
>>> >> > at HAdminTest.testCheckNPut(HAdminTest.java:150)
>>> >> > at HAdminTest.main(HAdminTest.java:257)
>>> >> >
>>> >> >
>>> >> > Thanks,
>>> >> > ~Ashish
>>> >> >
>>> >> >
>>> >> > On Tue, Aug 5, 2014 at 1:21 PM, Gabriel Reid
>>> >> > <gabriel.r...@gmail.com>
>>> >> > wrote:
>>> >> >>
>>> >> >> Hi Ashish,
>>> >> >>
>>> >> >> Could you post the full stack trace you're getting when the
>>> >> >> checkAndPut fails? No immediate reason I can think of as to why
>>> >> >> this
>>> >> >> would happen.
>>> >> >>
>>> >> >> - Gabriel
>>> >> >>
>>> >> >> On Tue, Aug 5, 2014 at 7:57 PM, ashish tapdiya
>>> >> >> <ashishtapd...@gmail.com>
>>> >> >> wrote:
>>> >> >> > Folks,
>>> >> >> >
>>> >> >> > any intuition why this is happening.
>>> >> >> >
>>> >> >> > Thanks,
>>> >> >> > ~Ashish
>>> >> >> >
>>> >> >> >
>>> >> >> > On Thu, Jul 31, 2014 at 8:29 PM, ashish tapdiya
>>> >> >> > <ashishtapd...@gmail.com>
>>> >> >> > wrote:
>>> >> >> >>
>>> >> >> >> I am using checkandput operation through HBase API on a table
>>> >> >> >> created
>>> >> >> >> in
>>> >> >> >> Phoenix. However, it fails after 14 client retries. Any insight
>>> >> >> >> on
>>> >> >> >> why
>>> >> >> >> it is
>>> >> >> >> happening and if there is a work around?
>>> >> >> >>
>>> >> >> >> I tried checkandput on a table created in HBase and it works.
>>> >> >> >> Also,
>>> >> >> >> scan,
>>> >> >> >> get using HBase API work fine with tables created in Phoenix.
>>> >> >> >>
>>> >> >> >> HBase version: 0.94
>>> >> >> >> Phoenix version: 3.0
>>> >> >> >>
>>> >> >> >> Thanks,
>>> >> >> >> ~Ashish
>>> >> >> >
>>> >> >> >
>>> >> >
>>> >> >
>>> >
>>> >
>>
>>
>
