[GitHub] spark pull request: [SPARK-13398][STREAMING] Move away from thread...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/11423


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13398][STREAMING] Move away from thread...

[srowen]

Github user srowen commented on the pull request:

https://github.com/apache/spark/pull/11423#issuecomment-192232248
  
Merged to master


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13398][STREAMING] Move away from thread...

[srowen]

Github user srowen commented on the pull request:

https://github.com/apache/spark/pull/11423#issuecomment-19222
  
LGTM


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13398][STREAMING] Move away from thread...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11423#issuecomment-192064925
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13398][STREAMING] Move away from thread...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11423#issuecomment-192064927
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52420/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13398][STREAMING] Move away from thread...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11423#issuecomment-192064713
  
**[Test build #52420 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52420/consoleFull)**
 for PR 11423 at commit 
[`6e8e7e0`](https://github.com/apache/spark/commit/6e8e7e0d484184030a6d519dad1dbef665758331).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13398][STREAMING] Move away from thread...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11423#issuecomment-192027067
  
**[Test build #52420 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52420/consoleFull)**
 for PR 11423 at commit 
[`6e8e7e0`](https://github.com/apache/spark/commit/6e8e7e0d484184030a6d519dad1dbef665758331).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13398][STREAMING] Move away from thread...

[holdenk]

Github user holdenk commented on the pull request:

https://github.com/apache/spark/pull/11423#issuecomment-190952064
  
I'll just switch this to use the Scala one for now (this code path does the 
correct thing in 2.12 but in 2.11 using the Java pool also reports system 
parallelism).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13398][STREAMING] Move away from thread...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11423#issuecomment-190728042
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13398][STREAMING] Move away from thread...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11423#issuecomment-190728050
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52239/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13398][STREAMING] Move away from thread...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11423#issuecomment-190727837
  
**[Test build #52239 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52239/consoleFull)**
 for PR 11423 at commit 
[`28d7d38`](https://github.com/apache/spark/commit/28d7d38c97353153925e216d4bd6599982fdba3d).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13398][STREAMING] Move away from thread...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11423#issuecomment-190684198
  
**[Test build #52239 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52239/consoleFull)**
 for PR 11423 at commit 
[`28d7d38`](https://github.com/apache/spark/commit/28d7d38c97353153925e216d4bd6599982fdba3d).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13398][STREAMING] Move away from thread...

[srowen]

Github user srowen commented on the pull request:

https://github.com/apache/spark/pull/11423#issuecomment-190683519
  
Jenkins, retest this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13398][STREAMING] Move away from thread...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11423#issuecomment-190476768
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13398][STREAMING] Move away from thread...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11423#issuecomment-190476770
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52201/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13398][STREAMING] Move away from thread...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11423#issuecomment-190476600
  
**[Test build #52201 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52201/consoleFull)**
 for PR 11423 at commit 
[`28d7d38`](https://github.com/apache/spark/commit/28d7d38c97353153925e216d4bd6599982fdba3d).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13398][STREAMING] Move away from thread...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11423#issuecomment-190442571
  
**[Test build #52201 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52201/consoleFull)**
 for PR 11423 at commit 
[`28d7d38`](https://github.com/apache/spark/commit/28d7d38c97353153925e216d4bd6599982fdba3d).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13398][STREAMING] Move away from thread...

[srowen]

Github user srowen commented on a diff in the pull request:

https://github.com/apache/spark/pull/11423#discussion_r54387309
  
--- Diff: core/src/main/scala/org/apache/spark/util/ThreadUtils.scala ---
@@ -156,4 +156,30 @@ private[spark] object ThreadUtils {
 result
 }
   }
+
+  /**
+   * Construct a new Java ForkJoinPool with a specified max parallelism 
and name prefix.
+   */
+  def newForkJoinPool(prefix: String, maxThreadNumber: Int): ForkJoinPool 
= {
+/**
+ * Extend ForkJoinWorkerThread so we can instantiate it.
+ */
+class SparkForkJoinWorkerThread(pool: ForkJoinPool) extends 
ForkJoinWorkerThread(pool) {
--- End diff --

Comments are great of course -- javadoc won't do anything here. It could be 
a line comment.

I think you might be able to collapse most of this with anonymous classes:

```
val factory = new ForkJoinWorkerThreadFactory {
  override def newThread(pool: ForkJoinPool) =
new ForkJoinWorkerThread(pool) {
  setName(prefix + "-" + super.getName)
}
}
```



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13398][STREAMING] Move away from thread...

[holdenk]

Github user holdenk commented on the pull request:

https://github.com/apache/spark/pull/11423#issuecomment-190072364
  
The HiveComptability test seems likely unrelated (happened in another 
unrelated PR at the same time). Will double check in the morning.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13398][STREAMING] Move away from thread...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11423#issuecomment-190072167
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13398][STREAMING] Move away from thread...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11423#issuecomment-190072170
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52165/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13398][STREAMING] Move away from thread...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11423#issuecomment-190072019
  
**[Test build #52165 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52165/consoleFull)**
 for PR 11423 at commit 
[`d6b3fb0`](https://github.com/apache/spark/commit/d6b3fb0c229d576fb1382c4356bd60b3f9ecfcac).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13398][STREAMING] Move away from thread...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11423#issuecomment-190040613
  
**[Test build #52165 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52165/consoleFull)**
 for PR 11423 at commit 
[`d6b3fb0`](https://github.com/apache/spark/commit/d6b3fb0c229d576fb1382c4356bd60b3f9ecfcac).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13398][STREAMING] Move away from thread...

[holdenk]

GitHub user holdenk opened a pull request:

https://github.com/apache/spark/pull/11423

[SPARK-13398][STREAMING] Move away from thread pool task support java 
forkjoin

## What changes were proposed in this pull request?

Remove old deprecated ThreadPoolExecutor and replace with ExecutionContext 
using a ForkJoinPool. The downside of this is that scala's ForkJoinPool doesn't 
give us a way to specify the thread pool name (and is also deprecated in 2.12). 
We can use the Java version of ForkJoinPool instead, although we need supply 
the custom factory instead. One other implicit change that happens is the old 
ExecutionContext would have reported a different default parallelism since it 
used system parallelism rather than threadpool parallelism (this was likely not 
intended but also likely not a huge difference).

The previous version of this PR attempted to use an execution context 
constructed on the ThreadPool (but not the deprecated ThreadPoolExecutor class) 
so as to keep the ability to have human readable named threads but this 
reported system parallelism.

## How was this patch tested?

unit tests: streaming/testOnly org.apache.spark.streaming.util.*


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/holdenk/spark 
SPARK-13398-move-away-from-ThreadPoolTaskSupport-java-forkjoin

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/11423.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #11423


commit 68df46a1093132ad3d5eef3e7a3d669105083236
Author: Holden Karau 
Date:   2016-02-21T07:08:45Z

Change FileWriteAheadLogger to not depend on deprecated ThreadPoolContext

commit 9991ebce07b6fabe122719ba1c4690e733297b18
Author: Holden Karau 
Date:   2016-02-21T07:13:40Z

Remove now unused import

commit f2e6a25ae2581e9b6cc43950c1302c26a8602c2a
Author: Holden Karau 
Date:   2016-02-21T07:19:27Z

Update the comment to point out if the thread pool is smaller than 8, that 
is the number of objects kept in memory.

commit ed9ff855871bfef69b39b1b9b791d1ce487093bd
Author: Holden Karau 
Date:   2016-02-22T01:07:56Z

Use a ForkJoinPool as the base

commit 296130cd262a3314349ced5b310a1b99130992e9
Author: Holden Karau 
Date:   2016-02-29T00:32:10Z

Merge branch 'master' into 
SPARK-13398-move-away-from-ThreadPoolTaskSupport-java-forkjoin

commit 7218567c146126b12a213117e6cf8069e33db1ea
Author: Holden Karau 
Date:   2016-02-29T02:50:06Z

Use a Java based ForkJoinPool

commit d6b3fb0c229d576fb1382c4356bd60b3f9ecfcac
Author: Holden Karau 
Date:   2016-02-29T02:59:23Z

Style fixes.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13398][STREAMING] move away from thread...

[srowen]

Github user srowen commented on the pull request:

https://github.com/apache/spark/pull/11310#issuecomment-189680177
  
This seemed like a good change to get rid of the deprecation. The thread 
name is kind of secondary to me.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13398][STREAMING] move away from thread...

[srowen]

Github user srowen commented on a diff in the pull request:

https://github.com/apache/spark/pull/11310#discussion_r54335512
  
--- Diff: core/src/main/scala/org/apache/spark/util/ThreadUtils.scala ---
@@ -156,4 +156,11 @@ private[spark] object ThreadUtils {
 result
 }
   }
+
+  /**
+   * Construct a new ForkJoinPool with a specified max parallelism.
+   */
+  def newForkJoinPool(maxThreadNumber: Int): 
scala.concurrent.forkjoin.ForkJoinPool = {
--- End diff --

Yes, if it's more than calling a constructor it may be worth it but not to 
wrap up "new ForkJoinPool" as "newForkJoinPool = new ForkJoinPool"


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13398][STREAMING] move away from thread...

[holdenk]

Github user holdenk commented on the pull request:

https://github.com/apache/spark/pull/11310#issuecomment-189573732
  
If we are going to go through the effort to do a custom factory for the 
thread names, looking at the future (e.g. scala 2.12) might be worth it to use 
the Java interface rather than the scala wrappers provided that we don't run 
into the same reporting incorrect parallelism problem. I'm going to close this 
for now and investigate that option mid next week once I'm back on my feet.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13398][STREAMING] move away from thread...

[holdenk]

Github user holdenk closed the pull request at:

https://github.com/apache/spark/pull/11310


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13398][STREAMING] move away from thread...

[holdenk]

Github user holdenk commented on a diff in the pull request:

https://github.com/apache/spark/pull/11310#discussion_r54327631
  
--- Diff: core/src/main/scala/org/apache/spark/util/ThreadUtils.scala ---
@@ -156,4 +156,11 @@ private[spark] object ThreadUtils {
 result
 }
   }
+
+  /**
+   * Construct a new ForkJoinPool with a specified max parallelism.
+   */
+  def newForkJoinPool(maxThreadNumber: Int): 
scala.concurrent.forkjoin.ForkJoinPool = {
--- End diff --

I figured it would be best to put it here with the rest of the ThreadUtils 
stuff. If we add the custom factory for the name as @zsxwing asked gets 
complicated enough probably worth keeping here in a utility function.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13398][STREAMING] move away from thread...

[holdenk]

Github user holdenk commented on a diff in the pull request:

https://github.com/apache/spark/pull/11310#discussion_r54327633
  
--- Diff: core/src/main/scala/org/apache/spark/util/ThreadUtils.scala ---
@@ -156,4 +156,11 @@ private[spark] object ThreadUtils {
 result
 }
   }
+
+  /**
+   * Construct a new ForkJoinPool with a specified max parallelism.
+   */
+  def newForkJoinPool(maxThreadNumber: Int): 
scala.concurrent.forkjoin.ForkJoinPool = {
+new scala.concurrent.forkjoin.ForkJoinPool(maxThreadNumber)
--- End diff --

So, with the scala ForkJoinPool even when provide a thread factory, it sets 
the thread names based on the "workerNamePrefix" (which is final) inside of 
"registerWorker" which is called when constructing a new ForkJoinWorkerThread - 
although with a custom factory we can change the name back after the thread has 
been constructed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13398][STREAMING] move away from thread...

[zsxwing]

Github user zsxwing commented on a diff in the pull request:

https://github.com/apache/spark/pull/11310#discussion_r54321079
  
--- Diff: core/src/main/scala/org/apache/spark/util/ThreadUtils.scala ---
@@ -156,4 +156,11 @@ private[spark] object ThreadUtils {
 result
 }
   }
+
+  /**
+   * Construct a new ForkJoinPool with a specified max parallelism.
+   */
+  def newForkJoinPool(maxThreadNumber: Int): 
scala.concurrent.forkjoin.ForkJoinPool = {
+new scala.concurrent.forkjoin.ForkJoinPool(maxThreadNumber)
--- End diff --

Please add a thread factory to set the thread name


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13398][STREAMING] move away from thread...

[srowen]

Github user srowen commented on the pull request:

https://github.com/apache/spark/pull/11310#issuecomment-187661137
  
That seems OK. It's weird that the first approach didn't work and that 
fork-join works as intended, but hey.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13398][STREAMING] move away from thread...

[srowen]

Github user srowen commented on a diff in the pull request:

https://github.com/apache/spark/pull/11310#discussion_r53767796
  
--- Diff: core/src/main/scala/org/apache/spark/util/ThreadUtils.scala ---
@@ -156,4 +156,11 @@ private[spark] object ThreadUtils {
 result
 }
   }
+
+  /**
+   * Construct a new ForkJoinPool with a specified max parallelism.
+   */
+  def newForkJoinPool(maxThreadNumber: Int): 
scala.concurrent.forkjoin.ForkJoinPool = {
--- End diff --

Do we need a method for this, really, vs just calling that constructor?
can it be imported as a class name?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13398][STREAMING] move away from thread...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11310#issuecomment-187446826
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/51685/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13398][STREAMING] move away from thread...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11310#issuecomment-187446823
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13398][STREAMING] move away from thread...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11310#issuecomment-187446529
  
**[Test build #51685 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51685/consoleFull)**
 for PR 11310 at commit 
[`ed9ff85`](https://github.com/apache/spark/commit/ed9ff855871bfef69b39b1b9b791d1ce487093bd).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13398][STREAMING] move away from thread...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11310#issuecomment-187400943
  
**[Test build #51685 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51685/consoleFull)**
 for PR 11310 at commit 
[`ed9ff85`](https://github.com/apache/spark/commit/ed9ff855871bfef69b39b1b9b791d1ce487093bd).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13398][STREAMING] move away from thread...

[holdenk]

GitHub user holdenk opened a pull request:

https://github.com/apache/spark/pull/11310

[SPARK-13398][STREAMING] move away from thread pool task support

## What changes were proposed in this pull request?

Remove old deprecated ThreadPoolExecutor and replace with ExecutionContext 
using a ForkJoinPool. The downside of this is that scala's ForkJoinPool doesn't 
give us a way to specify the thread pool name. One other implicit change that 
happens is the old ExecutionContext would have reported a different default 
parallelism since it used system parallelism rather than threadpool parallelism 
(this was likely not intended but also likely not a huge difference).

The previous version of this PR attempted to use an execution context 
constructed on the ThreadPool (but not the deprecated ThreadPoolExecutor class) 
so as to keep the ability to have human readable named threads but this 
reported system parallelism.

## How was the this patch tested?

unit tests: streaming/testOnly org.apache.spark.streaming.util.*

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/holdenk/spark 
SPARK-13398-move-away-from-ThreadPoolTaskSupport

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/11310.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #11310


commit 68df46a1093132ad3d5eef3e7a3d669105083236
Author: Holden Karau 
Date:   2016-02-21T07:08:45Z

Change FileWriteAheadLogger to not depend on deprecated ThreadPoolContext

commit 9991ebce07b6fabe122719ba1c4690e733297b18
Author: Holden Karau 
Date:   2016-02-21T07:13:40Z

Remove now unused import

commit f2e6a25ae2581e9b6cc43950c1302c26a8602c2a
Author: Holden Karau 
Date:   2016-02-21T07:19:27Z

Update the comment to point out if the thread pool is smaller than 8, that 
is the number of objects kept in memory.

commit ed9ff855871bfef69b39b1b9b791d1ce487093bd
Author: Holden Karau 
Date:   2016-02-22T01:07:56Z

Use a ForkJoinPool as the base




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org