[GitHub] spark issue #10846: [SPARK-12920][SQL] Fix high CPU usage in spark thrift se...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/10846 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/63402/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #10846: [SPARK-12920][SQL] Fix high CPU usage in spark thrift se...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/10846 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #10846: [SPARK-12920][SQL] Fix high CPU usage in spark thrift se...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/10846 **[Test build #63402 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63402/consoleFull)** for PR 10846 at commit [`b23ddc5`](https://github.com/apache/spark/commit/b23ddc51d89bf1a263ce92eb53107ea9589453e2). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #10846: [SPARK-12920][SQL] Fix high CPU usage in spark thrift se...
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/10846 @rajeshbalamohan can you also update the PR title and summary? thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #10846: [SPARK-12920][SQL] Fix high CPU usage in spark thrift se...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/10846 **[Test build #63402 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63402/consoleFull)** for PR 10846 at commit [`b23ddc5`](https://github.com/apache/spark/commit/b23ddc51d89bf1a263ce92eb53107ea9589453e2). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #10846: [SPARK-12920][SQL] Fix high CPU usage in spark thrift se...
Github user rajeshbalamohan commented on the issue: https://github.com/apache/spark/pull/10846 They take longer to clean up. If queries are executed continuously, major portion of thrift server wastes time in GC-ing. IAC, I have removed the HadoopRDD in the recent commit and can be tracked in separate JIRA. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #10846: [SPARK-12920][SQL] Fix high CPU usage in spark thrift se...
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/10846 I'm not saying we should fix just one of them. I'm saying we should treat them as separate issues. I am a little concerned about the workaround for the soft refs, for example, and that doesn't need to block the fix for the other issue. As for the soft ref issue, I'm not sure I understand what you mean: "Due to softRef they reach till GC threshold and gets cleared up". Are you saying soft refs don't get cleaned up when the HadoopRDD instances are collected? Or that they take longer? Can you clarify what you mean? If there's a problem with using soft refs here, then maybe a more explicit collection approach (e.g. a new method in `ContextCleaner` to track these) could be a better work around. But that assumes that your HadoopRDD instances are being collected, and if they're not, maybe *that's* the problem. Or if the caching is not bringing any benefits, maybe just remove the cache altogether. But with the little information you have provided, it's hard to know what's the case here. So, as you see, it's better to keep these two as separate issues. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #10846: [SPARK-12920][SQL] Fix high CPU usage in spark thrift se...
Github user rajeshbalamohan commented on the issue: https://github.com/apache/spark/pull/10846 SoftRef causes lots of mem-pressure on thrift server. To be precise, when executing query with large dataset, it can very soon run at 1200% CPU and all threads carrying out just GC activities. That is for the HadoopRDD conf caching. Due to softRef they reach till GC threshold and gets cleared up. It does not OOM, but runs at very high CPU due to GC. JobProgress* does not cleanup the data fast enough in some cases (e.g too many queries are executed continuously) and in such cases the memory pressure on thrift server increases. Both of them contribute to the high CPU usage. I am afraid that fixing one of them would still have the high-CPU usage issue. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #10846: [SPARK-12920][SQL] Fix high CPU usage in spark thrift se...
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/10846 @rajeshbalamohan could you break this into two separate bugs? The JobProgressListener issue is clear and the fix looks fine. But the cache issue is less clear - it would be better to understand why those instances are not being garbage collected instead of blindly turning off caching as a workaround. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #10846: [SPARK-12920][SQL] Fix high CPU usage in spark thrift se...
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/10846 So, `SparkEnv.hadoopJobMetadata` actually keeps soft refs to the conf objects, so they eventually should be garbage collected when the `HadoopRDD` instances go away. So isn't your problem just a matter of giving the thrift server more memory? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #10846: [SPARK-12920][SQL] Fix high CPU usage in spark thrift se...
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/10846 Ok, so this is actually two bugs: - more forcefully respecting the "retainedStages" config; now the code might actually stop showing active stages on the web ui, right? Not sure how big of an issue that is. I guess that was a possibility in the previous code too, so probably ok. - adding a new conf to stop caching of hadoop conf objects; it seems like there's a more fundamental problem in that after something is cached with `HadoopRDD.putCachedMetadata`, it never goes away. So your new config avoids that issue, but it seems like there should be code that "uncaches" those values when the HadoopRDD instance is collected. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #10846: [SPARK-12920][SQL] Fix high CPU usage in spark thrift se...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/10846 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/63367/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #10846: [SPARK-12920][SQL] Fix high CPU usage in spark thrift se...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/10846 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #10846: [SPARK-12920][SQL] Fix high CPU usage in spark thrift se...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/10846 **[Test build #63367 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63367/consoleFull)** for PR 10846 at commit [`096ce46`](https://github.com/apache/spark/commit/096ce46a7bd38a011ec67713431c6813a95fe79b). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #10846: [SPARK-12920][SQL] Fix high CPU usage in spark thrift se...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/10846 **[Test build #63367 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63367/consoleFull)** for PR 10846 at commit [`096ce46`](https://github.com/apache/spark/commit/096ce46a7bd38a011ec67713431c6813a95fe79b). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #10846: [SPARK-12920][SQL] Fix high CPU usage in spark thrift se...
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/10846 ok to test --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #10846: [SPARK-12920][SQL] Fix high CPU usage in spark thrift se...
Github user rajeshbalamohan commented on the issue: https://github.com/apache/spark/pull/10846 - Rebased to master and changed title. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
