[
https://issues.apache.org/jira/browse/PHOENIX-2599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15102155#comment-15102155
]
Li Gao commented on PHOENIX-2599:
---------------------------------
Hi [~jmahonin] thanks for the reply. Some background information on how and
when this error happens.
1) We are running spark on yarn
2) the spark workflow (driver) has many spark jobs (200+ spark jobs) that all
share the same spark context and execute the jobs in sequence
3) At some point during the workflow, when right after a spark job processed a
large Phoenix table (i.e. say 50 million rows), the exception would be thrown
4) In the workflow, most of the spark jobs are reading data (using
phoenixTableAsRDD call) and do transform/shuffle/filter in spark and then save
the data back to phoenix
Let me know if you need more information about this issue.
> PhoenixRecordReader does not handle StaleRegionBoundaryCacheException
> ---------------------------------------------------------------------
>
> Key: PHOENIX-2599
> URL: https://issues.apache.org/jira/browse/PHOENIX-2599
> Project: Phoenix
> Issue Type: Bug
> Affects Versions: 4.5.1
> Environment: HBase 1.0 + Linux
> Reporter: Li Gao
> Assignee: Josh Mahonin
>
> When running Spark 1.4.1 and Phoenix 4.5.1 via Phoenix-Spark connector. We
> notice sometimes (30~50%) time the following error would appear and kill the
> running spark job:
> 16/01/14 19:40:16 ERROR yarn.ApplicationMaster: User class threw exception:
> org.apache.spark.SparkException: Job aborted due to stage failure: Task 5 in
> stage 110.0 failed 4 times, most recent failure: Lost task 5.3 in stage 110.0
> (TID 35526, datanode-123.somewhere): java.lang.RuntimeException:
> org.apache.phoenix.schema.StaleRegionBoundaryCacheException: ERROR 1108
> (XCL08): Cache of region boundaries are out of date.
> at com.google.common.base.Throwables.propagate(Throwables.java:156)
> at
> org.apache.phoenix.mapreduce.PhoenixRecordReader.initialize(PhoenixRecordReader.java:126)
> at org.apache.spark.rdd.NewHadoopRDD$$anon$1.<init>(NewHadoopRDD.scala:133)
> at org.apache.spark.rdd.NewHadoopRDD.compute(NewHadoopRDD.scala:104)
> at org.apache.spark.rdd.NewHadoopRDD.compute(NewHadoopRDD.scala:66)
> at org.apache.phoenix.spark.PhoenixRDD.compute(PhoenixRDD.scala:52)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:63)
> at org.apache.spark.scheduler.Task.run(Task.scala:70)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.phoenix.schema.StaleRegionBoundaryCacheException: ERROR
> 1108 (XCL08): Cache of region boundaries are out of date.
> at
> org.apache.phoenix.exception.SQLExceptionCode$13.newException(SQLExceptionCode.java:304)
> at
> org.apache.phoenix.exception.SQLExceptionInfo.buildException(SQLExceptionInfo.java:145)
> at
> org.apache.phoenix.util.ServerUtil.parseRemoteException(ServerUtil.java:131)
> at
> org.apache.phoenix.util.ServerUtil.parseServerExceptionOrNull(ServerUtil.java:115)
> at
> org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:104)
> at
> org.apache.phoenix.iterate.TableResultIterator.getDelegate(TableResultIterator.java:70)
> at
> org.apache.phoenix.iterate.TableResultIterator.<init>(TableResultIterator.java:88)
> at
> org.apache.phoenix.iterate.TableResultIterator.<init>(TableResultIterator.java:79)
> at
> org.apache.phoenix.mapreduce.PhoenixRecordReader.initialize(PhoenixRecordReader.java:111)
> ... 18 more
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
