With the fix of PHOENIX-2447<https://issues.apache.org/jira/browse/PHOENIX-2447> in 4.7.0, now Phoenix will try multiple times for StaleRegionBoundaryCacheException until query timeout.
Alicia From: Josh Mahonin <jmaho...@gmail.com<mailto:jmaho...@gmail.com>> Reply-To: "user@phoenix.apache.org<mailto:user@phoenix.apache.org>" <user@phoenix.apache.org<mailto:user@phoenix.apache.org>> Date: Thursday, January 14, 2016 at 12:33 PM To: "user@phoenix.apache.org<mailto:user@phoenix.apache.org>" <user@phoenix.apache.org<mailto:user@phoenix.apache.org>> Subject: Re: StaleRegionBoundaryCacheException with Phoenix 4.5.1-HBase 1.0 and Spark 1.4.1 Hi Li, I've not seen this error myself, though some searching returns a possible root cause: http://mail-archives.apache.org/mod_mbox/incubator-phoenix-user/201507.mbox/%3CCAAF1JdjNW98dAnxf3kx=ndkswyorpt1redb9dqwjbhvvlfn...@mail.gmail.com%3E Could you file a JIRA ticket for this please? It's possible the MapReduce (and by extension, Spark) integration isn't handling this gracefully. Also, if possible, any information you have about your environment, table sizes, read/write workload, etc. would be helpful as well. Thanks! Josh On Thu, Jan 14, 2016 at 3:14 PM, Li Gao <g...@marinsoftware.com<mailto:g...@marinsoftware.com>> wrote: Hi Phoenix users, We are seeing occasionally (maybe 30 ~ 50% of time) this StaleRegionBoundaryCacheException when running Spark 1.4.1 and Phoenix 4.5.1-HBase 1.0. Not sure how to troubleshoot such issue. Any hints and insights are greatly appreciated! Thanks, Li PS: The following are the exception stack trace: 16/01/14 19:40:16 ERROR yarn.ApplicationMaster: User class threw exception: org.apache.spark.SparkException: Job aborted due to stage failure: Task 5 in stage 110.0 failed 4 times, most recent failure: Lost task 5.3 in stage 110.0 (TID 35526, datanode-123.somewhere): java.lang.RuntimeException: org.apache.phoenix.schema.StaleRegionBoundaryCacheException: ERROR 1108 (XCL08): Cache of region boundaries are out of date. at com.google.common.base.Throwables.propagate(Throwables.java:156) at org.apache.phoenix.mapreduce.PhoenixRecordReader.initialize(PhoenixRecordReader.java:126) at org.apache.spark.rdd.NewHadoopRDD$$anon$1.<init>(NewHadoopRDD.scala:133) at org.apache.spark.rdd.NewHadoopRDD.compute(NewHadoopRDD.scala:104) at org.apache.spark.rdd.NewHadoopRDD.compute(NewHadoopRDD.scala:66) at org.apache.phoenix.spark.PhoenixRDD.compute(PhoenixRDD.scala:52) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277) at org.apache.spark.rdd.RDD.iterator(RDD.scala:244) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277) at org.apache.spark.rdd.RDD.iterator(RDD.scala:244) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277) at org.apache.spark.rdd.RDD.iterator(RDD.scala:244) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:63) at org.apache.spark.scheduler.Task.run(Task.scala:70) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.phoenix.schema.StaleRegionBoundaryCacheException: ERROR 1108 (XCL08): Cache of region boundaries are out of date. at org.apache.phoenix.exception.SQLExceptionCode$13.newException(SQLExceptionCode.java:304) at org.apache.phoenix.exception.SQLExceptionInfo.buildException(SQLExceptionInfo.java:145) at org.apache.phoenix.util.ServerUtil.parseRemoteException(ServerUtil.java:131) at org.apache.phoenix.util.ServerUtil.parseServerExceptionOrNull(ServerUtil.java:115) at org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:104) at org.apache.phoenix.iterate.TableResultIterator.getDelegate(TableResultIterator.java:70) at org.apache.phoenix.iterate.TableResultIterator.<init>(TableResultIterator.java:88) at org.apache.phoenix.iterate.TableResultIterator.<init>(TableResultIterator.java:79) at org.apache.phoenix.mapreduce.PhoenixRecordReader.initialize(PhoenixRecordReader.java:111) ... 18 more
