Liu Shaohui created KYLIN-4099:
----------------------------------
Summary: Using no blocking unpersist in spark cubing job
Key: KYLIN-4099
URL: https://issues.apache.org/jira/browse/KYLIN-4099
Project: Kylin
Issue Type: Improvement
Reporter: Liu Shaohui
Assignee: Liu Shaohui
By default, the unpersist operation of RDD in spark is blocking which may cost
a lot time and
some times it may failed for some spark executors lost.
We can set blocking false to improve it.
{code:java}
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
scala.concurrent.impl.Promise$DefaultPromise.tryAwait(Promise.scala:208)
scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:218)
scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223)
scala.concurrent.Await$$anonfun$result$1.apply(package.scala:190)
scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
scala.concurrent.Await$.result(package.scala:190)
org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:81)
org.apache.spark.storage.BlockManagerMaster.removeRdd(BlockManagerMaster.scala:127)
org.apache.spark.SparkContext.unpersistRDD(SparkContext.scala:1709)
org.apache.spark.rdd.RDD.unpersist(RDD.scala:216)
org.apache.spark.api.java.JavaPairRDD.unpersist(JavaPairRDD.scala:73)
org.apache.kylin.engine.spark.SparkCubingByLayer.execute(SparkCubingByLayer.java:204)
org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:37)
org.apache.kylin.common.util.SparkEntry.main(SparkEntry.java:44)
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(Method.java:498)
org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:653){code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
