GitHub user holdenk opened a pull request:
https://github.com/apache/spark/pull/11423
[SPARK-13398][STREAMING] Move away from thread pool task support java
forkjoin
## What changes were proposed in this pull request?
Remove old deprecated ThreadPoolExecutor and replace with ExecutionContext
using a ForkJoinPool. The downside of this is that scala's ForkJoinPool doesn't
give us a way to specify the thread pool name (and is also deprecated in 2.12).
We can use the Java version of ForkJoinPool instead, although we need supply
the custom factory instead. One other implicit change that happens is the old
ExecutionContext would have reported a different default parallelism since it
used system parallelism rather than threadpool parallelism (this was likely not
intended but also likely not a huge difference).
The previous version of this PR attempted to use an execution context
constructed on the ThreadPool (but not the deprecated ThreadPoolExecutor class)
so as to keep the ability to have human readable named threads but this
reported system parallelism.
## How was this patch tested?
unit tests: streaming/testOnly org.apache.spark.streaming.util.*
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/holdenk/spark
SPARK-13398-move-away-from-ThreadPoolTaskSupport-java-forkjoin
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/11423.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #11423
----
commit 68df46a1093132ad3d5eef3e7a3d669105083236
Author: Holden Karau <[email protected]>
Date: 2016-02-21T07:08:45Z
Change FileWriteAheadLogger to not depend on deprecated ThreadPoolContext
commit 9991ebce07b6fabe122719ba1c4690e733297b18
Author: Holden Karau <[email protected]>
Date: 2016-02-21T07:13:40Z
Remove now unused import
commit f2e6a25ae2581e9b6cc43950c1302c26a8602c2a
Author: Holden Karau <[email protected]>
Date: 2016-02-21T07:19:27Z
Update the comment to point out if the thread pool is smaller than 8, that
is the number of objects kept in memory.
commit ed9ff855871bfef69b39b1b9b791d1ce487093bd
Author: Holden Karau <[email protected]>
Date: 2016-02-22T01:07:56Z
Use a ForkJoinPool as the base
commit 296130cd262a3314349ced5b310a1b99130992e9
Author: Holden Karau <[email protected]>
Date: 2016-02-29T00:32:10Z
Merge branch 'master' into
SPARK-13398-move-away-from-ThreadPoolTaskSupport-java-forkjoin
commit 7218567c146126b12a213117e6cf8069e33db1ea
Author: Holden Karau <[email protected]>
Date: 2016-02-29T02:50:06Z
Use a Java based ForkJoinPool
commit d6b3fb0c229d576fb1382c4356bd60b3f9ecfcac
Author: Holden Karau <[email protected]>
Date: 2016-02-29T02:59:23Z
Style fixes.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
