zhoukang created SPARK-26914:
--------------------------------
Summary: ThriftServer scheduler pool may be unpredictably when
using fair schedule mode
Key: SPARK-26914
URL: https://issues.apache.org/jira/browse/SPARK-26914
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 2.4.0
Reporter: zhoukang
When using fair scheduler mode for thrift server, we may have unpredictable
result.
{code:java}
val pool = sessionToActivePool.get(parentSession.getSessionHandle)
if (pool != null) {
sqlContext.sparkContext.setLocalProperty(SparkContext.SPARK_SCHEDULER_POOL,
pool)
}
{code}
Here is an example:
We have some query will use default pool, however it submit to 'normal' pool
I changed code and add some log.Got some strange result.
Then i found out that the localProperties of SparkContext may has unpredictable
result when call setLocalProperty. And since thriftserver use thread pool to
execute queries, it will trigger this bug sometimes.
{code:java}
/**
* Set a local property that affects jobs submitted from this thread, such as
the Spark fair
* scheduler pool. User-defined properties may also be set here. These
properties are propagated
* through to worker tasks and can be accessed there via
* [[org.apache.spark.TaskContext#getLocalProperty]].
*
* These properties are inherited by child threads spawned from this thread.
This
* may have unexpected consequences when working with thread pools. The
standard java
* implementation of thread pools have worker threads spawn other worker
threads.
* As a result, local properties may propagate unpredictably.
*/
def setLocalProperty(key: String, value: String) {
if (value == null) {
localProperties.get.remove(key)
} else {
localProperties.get.setProperty(key, value)
}
}
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
