Re: Table dead lock: ERROR 1120 (XCL20): Writes to table blocked until index can be updated

Is there any plan to port the fix to the CDH branch? The latest is still 4.14.0-cdh5.14.2 (09/jun/2018). Thanks, Miles -- Sent from: http://apache-phoenix-user-list.1124778.n5.nabble.com/

Re: Table dead lock: ERROR 1120 (XCL20): Writes to table blocked until index can be updated

I've created bug with reproduce steps: https://issues.apache.org/jira/browse/PHOENIX-4960 > On 3 Oct 2018, at 21:06, Batyrshin Alexander <0x62...@gmail.com> wrote: > > But we see that Phoenix commit() in our cases fails with "ERROR 1120 (XCL20): > Writes to table blocked until index can be

Re: Table dead lock: ERROR 1120 (XCL20): Writes to table blocked until index can be updated

But we see that Phoenix commit() in our cases fails with "ERROR 1120 (XCL20): Writes to table blocked until index can be updated" because of org.apache.hadoop.hbase.NotServingRegionException. Expected that there must be retry and success for commit() > On 2 Oct 2018, at 22:02, Josh Elser

Re: Table dead lock: ERROR 1120 (XCL20): Writes to table blocked until index can be updated

HBase will invalidate the location of a Region on seeing certain exceptions (including NotServingRegionException). After it sees the exception you have copied below, it should re-fetch the location of the Region. If HBase keeps trying to access a Region on a RS that isn't hosting it, either

Re: Table dead lock: ERROR 1120 (XCL20): Writes to table blocked until index can be updated

We tried branch 4.14-HBase-1.4 at commit https://github.com/apache/phoenix/commit/52893c240e4f24e2bfac0834d35205f866c16ed8 Is there any way to invalidate meta-cache on event of index regions split? Maybe there

Re: Table dead lock: ERROR 1120 (XCL20): Writes to table blocked until index can be updated

Yep, we tried restarting Master + all Region Servers. hbase.hregion.max.filesize is just 10GB. Our CPU/Disk busy at 5% or even lesser Just got another split: Oct 01 17:24:09 prod001 hbase[18135]: 2018-10-01 17:24:09,964 INFO [prod001,6,1538342034697_ChoreService_6]

Re: Table dead lock: ERROR 1120 (XCL20): Writes to table blocked until index can be updated

Did you restart the cluster and you should set 'hbase.hregion.max.filesize' to a safeguard value which less than RS's capabilities. Jaanai Zhang Best regards! Batyrshin Alexander <0x62...@gmail.com> 于2018年9月29日周六 下午5:28写道： > Meanwhile we tried to

Re: Table dead lock: ERROR 1120 (XCL20): Writes to table blocked until index can be updated

Meanwhile we tried to disable regions split via per index table options 'SPLIT_POLICY' => 'org.apache.hadoop.hbase.regionserver.ConstantSizeRegionSplitPolicy' and hbase.hregion.max.filesize = 10737418240 Looks like this options set doesn't. Some regions splits at size < 2GB Then we tried to

Re: Table dead lock: ERROR 1120 (XCL20): Writes to table blocked until index can be updated

We are planning a Phoenix 4.14.1 release which will have this fix On Wed, Sep 26, 2018 at 3:36 PM Batyrshin Alexander <0x62...@gmail.com> wrote: > Thank you. We will try somehow... > Is there any chance that this fix will be included in next release for > HBASE-1.4 (not 2.0)? > > On 27 Sep 2018,

Re: Table dead lock: ERROR 1120 (XCL20): Writes to table blocked until index can be updated

Thank you. We will try somehow... Is there any chance that this fix will be included in next release for HBASE-1.4 (not 2.0)? > On 27 Sep 2018, at 01:04, Ankit Singhal wrote: > > You might be hitting PHOENIX-4785 > , you can apply the patch

Re: Table dead lock: ERROR 1120 (XCL20): Writes to table blocked until index can be updated

You might be hitting PHOENIX-4785 , you can apply the patch on top of 4.14 and see if it fixes your problem. Regards, Ankit Singhal On Wed, Sep 26, 2018 at 2:33 PM Batyrshin Alexander <0x62...@gmail.com> wrote: > Any advices? Helps? > I can

Re: Table dead lock: ERROR 1120 (XCL20): Writes to table blocked until index can be updated

Any advices? Helps? I can reproduce problem and capture more logs if needed. > On 21 Sep 2018, at 02:13, Batyrshin Alexander <0x62...@gmail.com> wrote: > > Looks like lock goes away 30 minutes after index region split. > So i can assume that this issue comes from cache that configured by this >

Re: Table dead lock: ERROR 1120 (XCL20): Writes to table blocked until index can be updated

Looks like lock goes away 30 minutes after index region split. So i can assume that this issue comes from cache that configured by this option: phoenix.coprocessor.maxMetaDataCacheTimeToLiveMs > On 21 Sep 2018, at 00:15, Batyrshin Alexander <0x62...@gmail.com> wrote: > > And how this split

Re: Table dead lock: ERROR 1120 (XCL20): Writes to table blocked until index can be updated

And how this split looks at Master logs: Sep 20 19:45:04 prod001 hbase[10838]: 2018-09-20 19:45:04,888 INFO [AM.ZK.Worker-pool5-t282] master.RegionStates: Transition {3e44b85ddf407da831dbb9a871496986 state=OPEN, ts=1537304859509, server=prod013,60020,1537304282885} to

Re: Table dead lock: ERROR 1120 (XCL20): Writes to table blocked until index can be updated

Looks like problem was because of index region split Index region split at prod013: Sep 20 19:45:05 prod013 hbase[193055]: 2018-09-20 19:45:05,441 INFO [regionserver/prod013/10.0.0.13:60020-splits-1537400010677] regionserver.SplitRequest: Region split, hbase:meta updated, and report to

Re: Table dead lock: ERROR 1120 (XCL20): Writes to table blocked until index can be updated

Our setup: HBase-1.4.7 Phoenix-4.14-hbase-1.4 > On 20 Sep 2018, at 20:19, Batyrshin Alexander <0x62...@gmail.com> wrote: > > Hello, > Looks live we got dead lock with repeating "ERROR 1120 (XCL20)" exception. At > this time all indexes is ACTIVE. > Can you help to make deeper diagnose? > >

Table dead lock: ERROR 1120 (XCL20): Writes to table blocked until index can be updated

Hello, Looks live we got dead lock with repeating "ERROR 1120 (XCL20)" exception. At this time all indexes is ACTIVE. Can you help to make deeper diagnose? java.sql.SQLException: ERROR 1120 (XCL20): Writes to table blocked until index can be updated. tableName=TBL_MARK at