[
https://issues.apache.org/jira/browse/SPARK-26914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Owen resolved SPARK-26914.
-------------------------------
Resolution: Fixed
Assignee: zhoukang
Fix Version/s: 3.0.0
Resolved by https://github.com/apache/spark
> ThriftServer scheduler pool may be unpredictably when using fair schedule mode
> ------------------------------------------------------------------------------
>
> Key: SPARK-26914
> URL: https://issues.apache.org/jira/browse/SPARK-26914
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.4.0
> Reporter: zhoukang
> Assignee: zhoukang
> Priority: Major
> Fix For: 3.0.0
>
> Attachments: 26914-01.png, 26914-02.png, 26914-03.png
>
>
> When using fair scheduler mode for thrift server, we may have unpredictable
> result.
> {code:java}
> val pool = sessionToActivePool.get(parentSession.getSessionHandle)
> if (pool != null) {
>
> sqlContext.sparkContext.setLocalProperty(SparkContext.SPARK_SCHEDULER_POOL,
> pool)
> }
> {code}
> Here is an example:
> We have some query will use default pool, however it submit to 'normal' pool.
> !26914-02.png!
> I changed code and add some log.Got some strange result.
> !26914-01.png!
> !26914-03.png!
> Then i found out that the localProperties of SparkContext may has
> unpredictable result when call setLocalProperty. And since thriftserver use
> thread pool to execute queries, it will trigger this bug sometimes.
> {code:java}
> /**
> * Set a local property that affects jobs submitted from this thread, such
> as the Spark fair
> * scheduler pool. User-defined properties may also be set here. These
> properties are propagated
> * through to worker tasks and can be accessed there via
> * [[org.apache.spark.TaskContext#getLocalProperty]].
> *
> * These properties are inherited by child threads spawned from this
> thread. This
> * may have unexpected consequences when working with thread pools. The
> standard java
> * implementation of thread pools have worker threads spawn other worker
> threads.
> * As a result, local properties may propagate unpredictably.
> */
> def setLocalProperty(key: String, value: String) {
> if (value == null) {
> localProperties.get.remove(key)
> } else {
> localProperties.get.setProperty(key, value)
> }
> }
> {code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
