Re: StaleRegionBoundaryCacheException with Phoenix 4.5.1-HBase 1.0 and Spark 1.4.1

2016-01-16 Thread Josh Mahonin 
Hi Li,

This should be resolved in Phoenix 4.7.0. You can also apply the patch
yourself from here:
https://issues.apache.org/jira/browse/PHOENIX-2599

Please let us know if it fixes the problem for you.

Thanks!

Josh

On Thu, Jan 14, 2016 at 6:49 PM, James Taylor <jamestay...@apache.org>
wrote:

> That was my first thought too, Alicia. However, our MR and Spark
> integration uses a different code path. See my comment on PHOENIX-2599 for
> a potential fix.
>
> Thanks,
> James
>
> On Thu, Jan 14, 2016 at 2:55 PM, Alicia Shu <a...@hortonworks.com> wrote:
>
>> With the fix of PHOENIX-2447
>> <https://issues.apache.org/jira/browse/PHOENIX-2447> in 4.7.0, now
>> Phoenix will try multiple times for StaleRegionBoundaryCacheException until
>> query timeout.
>>
>> Alicia
>>
>> From: Josh Mahonin <jmaho...@gmail.com>
>> Reply-To: "user@phoenix.apache.org" <user@phoenix.apache.org>
>> Date: Thursday, January 14, 2016 at 12:33 PM
>> To: "user@phoenix.apache.org" <user@phoenix.apache.org>
>> Subject: Re: StaleRegionBoundaryCacheException with Phoenix 4.5.1-HBase
>> 1.0 and Spark 1.4.1
>>
>> Hi Li,
>>
>> I've not seen this error myself, though some searching returns a possible
>> root cause:
>>
>> http://mail-archives.apache.org/mod_mbox/incubator-phoenix-user/201507.mbox/%3CCAAF1JdjNW98dAnxf3kx=ndkswyorpt1redb9dqwjbhvvlfn...@mail.gmail.com%3E
>>
>> Could you file a JIRA ticket for this please? It's possible the MapReduce
>> (and by extension, Spark) integration isn't handling this gracefully.
>>
>> Also, if possible, any information you have about your environment, table
>> sizes, read/write workload, etc. would be helpful as well.
>>
>> Thanks!
>>
>> Josh
>>
>> On Thu, Jan 14, 2016 at 3:14 PM, Li Gao <g...@marinsoftware.com> wrote:
>>
>>> Hi Phoenix users,
>>>
>>>
>>> We are seeing occasionally (maybe 30 ~ 50% of time) this
>>> StaleRegionBoundaryCacheException when running Spark 1.4.1 and Phoenix
>>> 4.5.1-HBase 1.0.
>>>
>>> Not sure how to troubleshoot such issue. Any hints and insights are
>>> greatly appreciated!
>>>
>>> Thanks,
>>>
>>> Li
>>>
>>> PS: The following are the exception stack trace:
>>>
>>> 16/01/14 19:40:16 ERROR yarn.ApplicationMaster: User class threw
>>> exception: org.apache.spark.SparkException: Job aborted due to stage
>>> failure: Task 5 in stage 110.0 failed 4 times, most recent failure: Lost
>>> task 5.3 in stage 110.0 (TID 35526, datanode-123.somewhere):
>>> java.lang.RuntimeException:
>>> org.apache.phoenix.schema.StaleRegionBoundaryCacheException: ERROR 1108
>>> (XCL08): Cache of region boundaries are out of date.
>>>
>>> at com.google.common.base.Throwables.propagate(Throwables.java:156)
>>>
>>> at
>>> org.apache.phoenix.mapreduce.PhoenixRecordReader.initialize(PhoenixRecordReader.java:126)
>>>
>>> at
>>> org.apache.spark.rdd.NewHadoopRDD$$anon$1.(NewHadoopRDD.scala:133)
>>>
>>> at org.apache.spark.rdd.NewHadoopRDD.compute(NewHadoopRDD.scala:104)
>>>
>>> at org.apache.spark.rdd.NewHadoopRDD.compute(NewHadoopRDD.scala:66)
>>>
>>> at org.apache.phoenix.spark.PhoenixRDD.compute(PhoenixRDD.scala:52)
>>>
>>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>>
>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>>
>>> at
>>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>>
>>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>>
>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>>
>>> at
>>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>>
>>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>>
>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>>
>>> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:63)
>>>
>>> at org.apache.spark.scheduler.Task.run(Task.scala:70)
>>>
>>> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213)
>>>
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor

Re: StaleRegionBoundaryCacheException with Phoenix 4.5.1-HBase 1.0 and Spark 1.4.1

2016-01-14 Thread Alicia Shu 
With the fix of 
PHOENIX-2447<https://issues.apache.org/jira/browse/PHOENIX-2447> in 4.7.0, now 
Phoenix will try multiple times for StaleRegionBoundaryCacheException until 
query timeout.

Alicia

From: Josh Mahonin <jmaho...@gmail.com<mailto:jmaho...@gmail.com>>
Reply-To: "user@phoenix.apache.org<mailto:user@phoenix.apache.org>" 
<user@phoenix.apache.org<mailto:user@phoenix.apache.org>>
Date: Thursday, January 14, 2016 at 12:33 PM
To: "user@phoenix.apache.org<mailto:user@phoenix.apache.org>" 
<user@phoenix.apache.org<mailto:user@phoenix.apache.org>>
Subject: Re: StaleRegionBoundaryCacheException with Phoenix 4.5.1-HBase 1.0 and 
Spark 1.4.1

Hi Li,

I've not seen this error myself, though some searching returns a possible root 
cause:
http://mail-archives.apache.org/mod_mbox/incubator-phoenix-user/201507.mbox/%3CCAAF1JdjNW98dAnxf3kx=ndkswyorpt1redb9dqwjbhvvlfn...@mail.gmail.com%3E

Could you file a JIRA ticket for this please? It's possible the MapReduce (and 
by extension, Spark) integration isn't handling this gracefully.

Also, if possible, any information you have about your environment, table 
sizes, read/write workload, etc. would be helpful as well.

Thanks!

Josh

On Thu, Jan 14, 2016 at 3:14 PM, Li Gao 
<g...@marinsoftware.com<mailto:g...@marinsoftware.com>> wrote:

Hi Phoenix users,


We are seeing occasionally (maybe 30 ~ 50% of time) this 
StaleRegionBoundaryCacheException when running Spark 1.4.1 and Phoenix 
4.5.1-HBase 1.0.

Not sure how to troubleshoot such issue. Any hints and insights are greatly 
appreciated!

Thanks,

Li

PS: The following are the exception stack trace:

16/01/14 19:40:16 ERROR yarn.ApplicationMaster: User class threw exception: 
org.apache.spark.SparkException: Job aborted due to stage failure: Task 5 in 
stage 110.0 failed 4 times, most recent failure: Lost task 5.3 in stage 110.0 
(TID 35526, datanode-123.somewhere): java.lang.RuntimeException: 
org.apache.phoenix.schema.StaleRegionBoundaryCacheException: ERROR 1108 
(XCL08): Cache of region boundaries are out of date.

at com.google.common.base.Throwables.propagate(Throwables.java:156)

at 
org.apache.phoenix.mapreduce.PhoenixRecordReader.initialize(PhoenixRecordReader.java:126)

at org.apache.spark.rdd.NewHadoopRDD$$anon$1.(NewHadoopRDD.scala:133)

at org.apache.spark.rdd.NewHadoopRDD.compute(NewHadoopRDD.scala:104)

at org.apache.spark.rdd.NewHadoopRDD.compute(NewHadoopRDD.scala:66)

at org.apache.phoenix.spark.PhoenixRDD.compute(PhoenixRDD.scala:52)

at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)

at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)

at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)

at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)

at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)

at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)

at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)

at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)

at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:63)

at org.apache.spark.scheduler.Task.run(Task.scala:70)

at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213)

at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

at java.lang.Thread.run(Thread.java:745)

Caused by: org.apache.phoenix.schema.StaleRegionBoundaryCacheException: ERROR 
1108 (XCL08): Cache of region boundaries are out of date.

at 
org.apache.phoenix.exception.SQLExceptionCode$13.newException(SQLExceptionCode.java:304)

at 
org.apache.phoenix.exception.SQLExceptionInfo.buildException(SQLExceptionInfo.java:145)

at org.apache.phoenix.util.ServerUtil.parseRemoteException(ServerUtil.java:131)

at 
org.apache.phoenix.util.ServerUtil.parseServerExceptionOrNull(ServerUtil.java:115)

at org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:104)

at 
org.apache.phoenix.iterate.TableResultIterator.getDelegate(TableResultIterator.java:70)

at 
org.apache.phoenix.iterate.TableResultIterator.(TableResultIterator.java:88)

at 
org.apache.phoenix.iterate.TableResultIterator.(TableResultIterator.java:79)

at 
org.apache.phoenix.mapreduce.PhoenixRecordReader.initialize(PhoenixRecordReader.java:111)

... 18 more

Re: StaleRegionBoundaryCacheException with Phoenix 4.5.1-HBase 1.0 and Spark 1.4.1

2016-01-14 Thread James Taylor 
That was my first thought too, Alicia. However, our MR and Spark
integration uses a different code path. See my comment on PHOENIX-2599 for
a potential fix.

Thanks,
James

On Thu, Jan 14, 2016 at 2:55 PM, Alicia Shu <a...@hortonworks.com> wrote:

> With the fix of PHOENIX-2447
> <https://issues.apache.org/jira/browse/PHOENIX-2447> in 4.7.0, now
> Phoenix will try multiple times for StaleRegionBoundaryCacheException until
> query timeout.
>
> Alicia
>
> From: Josh Mahonin <jmaho...@gmail.com>
> Reply-To: "user@phoenix.apache.org" <user@phoenix.apache.org>
> Date: Thursday, January 14, 2016 at 12:33 PM
> To: "user@phoenix.apache.org" <user@phoenix.apache.org>
> Subject: Re: StaleRegionBoundaryCacheException with Phoenix 4.5.1-HBase
> 1.0 and Spark 1.4.1
>
> Hi Li,
>
> I've not seen this error myself, though some searching returns a possible
> root cause:
>
> http://mail-archives.apache.org/mod_mbox/incubator-phoenix-user/201507.mbox/%3CCAAF1JdjNW98dAnxf3kx=ndkswyorpt1redb9dqwjbhvvlfn...@mail.gmail.com%3E
>
> Could you file a JIRA ticket for this please? It's possible the MapReduce
> (and by extension, Spark) integration isn't handling this gracefully.
>
> Also, if possible, any information you have about your environment, table
> sizes, read/write workload, etc. would be helpful as well.
>
> Thanks!
>
> Josh
>
> On Thu, Jan 14, 2016 at 3:14 PM, Li Gao <g...@marinsoftware.com> wrote:
>
>> Hi Phoenix users,
>>
>>
>> We are seeing occasionally (maybe 30 ~ 50% of time) this
>> StaleRegionBoundaryCacheException when running Spark 1.4.1 and Phoenix
>> 4.5.1-HBase 1.0.
>>
>> Not sure how to troubleshoot such issue. Any hints and insights are
>> greatly appreciated!
>>
>> Thanks,
>>
>> Li
>>
>> PS: The following are the exception stack trace:
>>
>> 16/01/14 19:40:16 ERROR yarn.ApplicationMaster: User class threw
>> exception: org.apache.spark.SparkException: Job aborted due to stage
>> failure: Task 5 in stage 110.0 failed 4 times, most recent failure: Lost
>> task 5.3 in stage 110.0 (TID 35526, datanode-123.somewhere):
>> java.lang.RuntimeException:
>> org.apache.phoenix.schema.StaleRegionBoundaryCacheException: ERROR 1108
>> (XCL08): Cache of region boundaries are out of date.
>>
>> at com.google.common.base.Throwables.propagate(Throwables.java:156)
>>
>> at
>> org.apache.phoenix.mapreduce.PhoenixRecordReader.initialize(PhoenixRecordReader.java:126)
>>
>> at
>> org.apache.spark.rdd.NewHadoopRDD$$anon$1.(NewHadoopRDD.scala:133)
>>
>> at org.apache.spark.rdd.NewHadoopRDD.compute(NewHadoopRDD.scala:104)
>>
>> at org.apache.spark.rdd.NewHadoopRDD.compute(NewHadoopRDD.scala:66)
>>
>> at org.apache.phoenix.spark.PhoenixRDD.compute(PhoenixRDD.scala:52)
>>
>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>
>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>
>> at
>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>
>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>
>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>
>> at
>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>
>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>
>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>
>> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:63)
>>
>> at org.apache.spark.scheduler.Task.run(Task.scala:70)
>>
>> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213)
>>
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>
>> at java.lang.Thread.run(Thread.java:745)
>>
>> Caused by: org.apache.phoenix.schema.StaleRegionBoundaryCacheException:
>> ERROR 1108 (XCL08): Cache of region boundaries are out of date.
>>
>> at
>> org.apache.phoenix.exception.SQLExceptionCode$13.newException(SQLExceptionCode.java:304)
>>
>> at
>> org.apache.phoenix.exception.SQLExceptionInfo.buildException(SQLExceptionInfo.java:145)
>>
>> at
>> org.apache.phoenix.util.ServerUtil.parseRemoteException(ServerUtil.java:131)
>>
>> at
>> org.apache.phoenix.util.ServerUtil.parseServerExceptionOrNull(ServerUtil.java:115)
>>
>> at
>> org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:104)
>>
>> at
>> org.apache.phoenix.iterate.TableResultIterator.getDelegate(TableResultIterator.java:70)
>>
>> at
>> org.apache.phoenix.iterate.TableResultIterator.(TableResultIterator.java:88)
>>
>> at
>> org.apache.phoenix.iterate.TableResultIterator.(TableResultIterator.java:79)
>>
>> at
>> org.apache.phoenix.mapreduce.PhoenixRecordReader.initialize(PhoenixRecordReader.java:111)
>>
>> ... 18 more
>>
>
>

Re: StaleRegionBoundaryCacheException with Phoenix 4.5.1-HBase 1.0 and Spark 1.4.1

2016-01-14 Thread Josh Mahonin 
Hi Li,

I've not seen this error myself, though some searching returns a possible
root cause:
http://mail-archives.apache.org/mod_mbox/incubator-phoenix-user/201507.mbox/%3CCAAF1JdjNW98dAnxf3kx=ndkswyorpt1redb9dqwjbhvvlfn...@mail.gmail.com%3E

Could you file a JIRA ticket for this please? It's possible the MapReduce
(and by extension, Spark) integration isn't handling this gracefully.

Also, if possible, any information you have about your environment, table
sizes, read/write workload, etc. would be helpful as well.

Thanks!

Josh

On Thu, Jan 14, 2016 at 3:14 PM, Li Gao  wrote:

> Hi Phoenix users,
>
>
> We are seeing occasionally (maybe 30 ~ 50% of time) this
> StaleRegionBoundaryCacheException when running Spark 1.4.1 and Phoenix
> 4.5.1-HBase 1.0.
>
> Not sure how to troubleshoot such issue. Any hints and insights are
> greatly appreciated!
>
> Thanks,
>
> Li
>
> PS: The following are the exception stack trace:
>
> 16/01/14 19:40:16 ERROR yarn.ApplicationMaster: User class threw
> exception: org.apache.spark.SparkException: Job aborted due to stage
> failure: Task 5 in stage 110.0 failed 4 times, most recent failure: Lost
> task 5.3 in stage 110.0 (TID 35526, datanode-123.somewhere):
> java.lang.RuntimeException:
> org.apache.phoenix.schema.StaleRegionBoundaryCacheException: ERROR 1108
> (XCL08): Cache of region boundaries are out of date.
>
> at com.google.common.base.Throwables.propagate(Throwables.java:156)
>
> at
> org.apache.phoenix.mapreduce.PhoenixRecordReader.initialize(PhoenixRecordReader.java:126)
>
> at org.apache.spark.rdd.NewHadoopRDD$$anon$1.(NewHadoopRDD.scala:133)
>
> at org.apache.spark.rdd.NewHadoopRDD.compute(NewHadoopRDD.scala:104)
>
> at org.apache.spark.rdd.NewHadoopRDD.compute(NewHadoopRDD.scala:66)
>
> at org.apache.phoenix.spark.PhoenixRDD.compute(PhoenixRDD.scala:52)
>
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:63)
>
> at org.apache.spark.scheduler.Task.run(Task.scala:70)
>
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213)
>
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>
> at java.lang.Thread.run(Thread.java:745)
>
> Caused by: org.apache.phoenix.schema.StaleRegionBoundaryCacheException:
> ERROR 1108 (XCL08): Cache of region boundaries are out of date.
>
> at
> org.apache.phoenix.exception.SQLExceptionCode$13.newException(SQLExceptionCode.java:304)
>
> at
> org.apache.phoenix.exception.SQLExceptionInfo.buildException(SQLExceptionInfo.java:145)
>
> at
> org.apache.phoenix.util.ServerUtil.parseRemoteException(ServerUtil.java:131)
>
> at
> org.apache.phoenix.util.ServerUtil.parseServerExceptionOrNull(ServerUtil.java:115)
>
> at
> org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:104)
>
> at
> org.apache.phoenix.iterate.TableResultIterator.getDelegate(TableResultIterator.java:70)
>
> at
> org.apache.phoenix.iterate.TableResultIterator.(TableResultIterator.java:88)
>
> at
> org.apache.phoenix.iterate.TableResultIterator.(TableResultIterator.java:79)
>
> at
> org.apache.phoenix.mapreduce.PhoenixRecordReader.initialize(PhoenixRecordReader.java:111)
>
> ... 18 more
>