GitHub user holdenk opened a pull request:
https://github.com/apache/spark/pull/11310
[SPARK-13398][STREAMING] move away from thread pool task support
## What changes were proposed in this pull request?
Remove old deprecated ThreadPoolExecutor and replace with ExecutionContext
using a ForkJoinPool. The downside of this is that scala's ForkJoinPool doesn't
give us a way to specify the thread pool name. One other implicit change that
happens is the old ExecutionContext would have reported a different default
parallelism since it used system parallelism rather than threadpool parallelism
(this was likely not intended but also likely not a huge difference).
The previous version of this PR attempted to use an execution context
constructed on the ThreadPool (but not the deprecated ThreadPoolExecutor class)
so as to keep the ability to have human readable named threads but this
reported system parallelism.
## How was the this patch tested?
unit tests: streaming/testOnly org.apache.spark.streaming.util.*
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/holdenk/spark
SPARK-13398-move-away-from-ThreadPoolTaskSupport
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/11310.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #11310
----
commit 68df46a1093132ad3d5eef3e7a3d669105083236
Author: Holden Karau <[email protected]>
Date: 2016-02-21T07:08:45Z
Change FileWriteAheadLogger to not depend on deprecated ThreadPoolContext
commit 9991ebce07b6fabe122719ba1c4690e733297b18
Author: Holden Karau <[email protected]>
Date: 2016-02-21T07:13:40Z
Remove now unused import
commit f2e6a25ae2581e9b6cc43950c1302c26a8602c2a
Author: Holden Karau <[email protected]>
Date: 2016-02-21T07:19:27Z
Update the comment to point out if the thread pool is smaller than 8, that
is the number of objects kept in memory.
commit ed9ff855871bfef69b39b1b9b791d1ce487093bd
Author: Holden Karau <[email protected]>
Date: 2016-02-22T01:07:56Z
Use a ForkJoinPool as the base
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
