[GitHub] spark issue #17833: [SPARK-20558][CORE] clear InheritableThreadLocal variabl...
Github user sameeragarwal commented on the issue: https://github.com/apache/spark/pull/17833 Are we still investigating the root cause? If not, perhaps try re-enabling the test in 2.2 to see if it's failing? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17833: [SPARK-20558][CORE] clear InheritableThreadLocal variabl...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/17833 merging to master/2.2/2.1/2.0 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17833: [SPARK-20558][CORE] clear InheritableThreadLocal variabl...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/17833 > I think it only cleans localProperties in the current thread. localProperties overrides childValue and always clones a new Properties for child threads. Yea, that's true. If some child threads are already there and cloned the local properties, we can't clean them. But we can avoid future child threads to inherit this local properties, which can reduce the memory footprint a lot if users create new `SparkContext` and stop it, and repeat this many times. Anyway, I'll merge this PR and see if it can fix the flaky test. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17833: [SPARK-20558][CORE] clear InheritableThreadLocal variabl...
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/17833 In a second thought, for https://issues.apache.org/jira/browse/SPARK-20548 , we can just combine some tests into one test to reduce the SparkContexts and REPLs. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17833: [SPARK-20558][CORE] clear InheritableThreadLocal variabl...
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/17833 By the way, this PR is good to me since it does reduce a little memory footprint. But we still cannot close https://issues.apache.org/jira/browse/SPARK-20548 though. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17833: [SPARK-20558][CORE] clear InheritableThreadLocal variabl...
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/17833 I think it only cleans `localProperties` in the current thread. `localProperties` overrides `childValue` and always clones a new Properties for child threads. In addition, I think it doesn't fix the flaky REPL tests. Last time I checked the head dump, I observed most of memory usage comes from REPL and it's class loader referred by `SparkContext`s. And the GC roots of `SparkContext`s are JVM internal threads in this thread pool: http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/8u40-b25/java/lang/UNIXProcess.java#220 It's a cache thread pool. I could not figure out a fix except adding `Thread.sleep` to wait for these threads being killed automatically. That's why we can only observe OOM in REPL tests. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17833: [SPARK-20558][CORE] clear InheritableThreadLocal variabl...
Github user sameeragarwal commented on the issue: https://github.com/apache/spark/pull/17833 LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17833: [SPARK-20558][CORE] clear InheritableThreadLocal variabl...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17833 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76384/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17833: [SPARK-20558][CORE] clear InheritableThreadLocal variabl...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17833 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17833: [SPARK-20558][CORE] clear InheritableThreadLocal variabl...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17833 **[Test build #76384 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76384/testReport)** for PR 17833 at commit [`310ad75`](https://github.com/apache/spark/commit/310ad753c594e272266689d760049cdc3b53e04c). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17833: [SPARK-20558][CORE] clear InheritableThreadLocal variabl...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17833 **[Test build #76384 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76384/testReport)** for PR 17833 at commit [`310ad75`](https://github.com/apache/spark/commit/310ad753c594e272266689d760049cdc3b53e04c). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17833: [SPARK-20558][CORE] clear InheritableThreadLocal variabl...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/17833 cc @JoshRosen @rxin @zsxwing --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
