That was my first thought too, Alicia. However, our MR and Spark integration uses a different code path. See my comment on PHOENIX-2599 for a potential fix.
Thanks, James On Thu, Jan 14, 2016 at 2:55 PM, Alicia Shu <a...@hortonworks.com> wrote: > With the fix of PHOENIX-2447 > <https://issues.apache.org/jira/browse/PHOENIX-2447> in 4.7.0, now > Phoenix will try multiple times for StaleRegionBoundaryCacheException until > query timeout. > > Alicia > > From: Josh Mahonin <jmaho...@gmail.com> > Reply-To: "user@phoenix.apache.org" <user@phoenix.apache.org> > Date: Thursday, January 14, 2016 at 12:33 PM > To: "user@phoenix.apache.org" <user@phoenix.apache.org> > Subject: Re: StaleRegionBoundaryCacheException with Phoenix 4.5.1-HBase > 1.0 and Spark 1.4.1 > > Hi Li, > > I've not seen this error myself, though some searching returns a possible > root cause: > > http://mail-archives.apache.org/mod_mbox/incubator-phoenix-user/201507.mbox/%3CCAAF1JdjNW98dAnxf3kx=ndkswyorpt1redb9dqwjbhvvlfn...@mail.gmail.com%3E > > Could you file a JIRA ticket for this please? It's possible the MapReduce > (and by extension, Spark) integration isn't handling this gracefully. > > Also, if possible, any information you have about your environment, table > sizes, read/write workload, etc. would be helpful as well. > > Thanks! > > Josh > > On Thu, Jan 14, 2016 at 3:14 PM, Li Gao <g...@marinsoftware.com> wrote: > >> Hi Phoenix users, >> >> >> We are seeing occasionally (maybe 30 ~ 50% of time) this >> StaleRegionBoundaryCacheException when running Spark 1.4.1 and Phoenix >> 4.5.1-HBase 1.0. >> >> Not sure how to troubleshoot such issue. Any hints and insights are >> greatly appreciated! >> >> Thanks, >> >> Li >> >> PS: The following are the exception stack trace: >> >> 16/01/14 19:40:16 ERROR yarn.ApplicationMaster: User class threw >> exception: org.apache.spark.SparkException: Job aborted due to stage >> failure: Task 5 in stage 110.0 failed 4 times, most recent failure: Lost >> task 5.3 in stage 110.0 (TID 35526, datanode-123.somewhere): >> java.lang.RuntimeException: >> org.apache.phoenix.schema.StaleRegionBoundaryCacheException: ERROR 1108 >> (XCL08): Cache of region boundaries are out of date. >> >> at com.google.common.base.Throwables.propagate(Throwables.java:156) >> >> at >> org.apache.phoenix.mapreduce.PhoenixRecordReader.initialize(PhoenixRecordReader.java:126) >> >> at >> org.apache.spark.rdd.NewHadoopRDD$$anon$1.<init>(NewHadoopRDD.scala:133) >> >> at org.apache.spark.rdd.NewHadoopRDD.compute(NewHadoopRDD.scala:104) >> >> at org.apache.spark.rdd.NewHadoopRDD.compute(NewHadoopRDD.scala:66) >> >> at org.apache.phoenix.spark.PhoenixRDD.compute(PhoenixRDD.scala:52) >> >> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277) >> >> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244) >> >> at >> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35) >> >> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277) >> >> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244) >> >> at >> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35) >> >> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277) >> >> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244) >> >> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:63) >> >> at org.apache.spark.scheduler.Task.run(Task.scala:70) >> >> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213) >> >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) >> >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) >> >> at java.lang.Thread.run(Thread.java:745) >> >> Caused by: org.apache.phoenix.schema.StaleRegionBoundaryCacheException: >> ERROR 1108 (XCL08): Cache of region boundaries are out of date. >> >> at >> org.apache.phoenix.exception.SQLExceptionCode$13.newException(SQLExceptionCode.java:304) >> >> at >> org.apache.phoenix.exception.SQLExceptionInfo.buildException(SQLExceptionInfo.java:145) >> >> at >> org.apache.phoenix.util.ServerUtil.parseRemoteException(ServerUtil.java:131) >> >> at >> org.apache.phoenix.util.ServerUtil.parseServerExceptionOrNull(ServerUtil.java:115) >> >> at >> org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:104) >> >> at >> org.apache.phoenix.iterate.TableResultIterator.getDelegate(TableResultIterator.java:70) >> >> at >> org.apache.phoenix.iterate.TableResultIterator.<init>(TableResultIterator.java:88) >> >> at >> org.apache.phoenix.iterate.TableResultIterator.<init>(TableResultIterator.java:79) >> >> at >> org.apache.phoenix.mapreduce.PhoenixRecordReader.initialize(PhoenixRecordReader.java:111) >> >> ... 18 more >> > >
