[GitHub] spark issue #18723: [SPARK-21517][CORE] Avoid copying memory when transfer c...
Github user caneGuy commented on the issue: https://github.com/apache/spark/pull/18723 Thanks @kiszk @zsxwing --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18723: [SPARK-21517][CORE] Avoid copying memory when transfer c...
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/18723 Thanks! Merging to master. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18723: [SPARK-21517][CORE] Avoid copying memory when transfer c...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18723 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79943/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18723: [SPARK-21517][CORE] Avoid copying memory when transfer c...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18723 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18723: [SPARK-21517][CORE] Avoid copying memory when transfer c...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18723 **[Test build #79943 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79943/testReport)** for PR 18723 at commit [`4c576a4`](https://github.com/apache/spark/commit/4c576a4d81c6af744fab28a9363061885a8aeb91). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18723: [SPARK-21517][CORE] Avoid copying memory when transfer c...
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/18723 LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18723: [SPARK-21517][CORE] Avoid copying memory when transfer c...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18723 **[Test build #79943 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79943/testReport)** for PR 18723 at commit [`4c576a4`](https://github.com/apache/spark/commit/4c576a4d81c6af744fab28a9363061885a8aeb91). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18723: [SPARK-21517][CORE] Avoid copying memory when transfer c...
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/18723 ok to test --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18723: [SPARK-21517][CORE] Avoid copying memory when transfer c...
Github user kiszk commented on the issue: https://github.com/apache/spark/pull/18723 cc @JoshRosen --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18723: [SPARK-21517][CORE] Avoid copying memory when transfer c...
Github user caneGuy commented on the issue: https://github.com/apache/spark/pull/18723 @kiszk Actually, i am confused with default value 16 too. Yes, it occurs in specific scenario.In our case, it was large size block data which caused this issue. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18723: [SPARK-21517][CORE] Avoid copying memory when transfer c...
Github user kiszk commented on the issue: https://github.com/apache/spark/pull/18723 Thank you. 1. makes sense since memory is allocated by 2x at that point. While I looked at a [PR](https://github.com/netty/netty/pull/464) for netty, I cannot understand why 16 was used. I have another question to understand why it happens. Does this OOM occurs when any OpenBlocks message are received. Or, any specific scenario (e.g. receive a large message, a lot of multiple messages, or so on.) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18723: [SPARK-21517][CORE] Avoid copying memory when transfer c...
Github user caneGuy commented on the issue: https://github.com/apache/spark/pull/18723 @kiszk Thanks for your time. Yes,you are right. 1.OOM happens at io.netty.buffer.CompositeByteBuf.allocBuffer when one executor(client) fetch blocks from an other(server) , oom happens at server side before response to client. 2.Components number depends on chunks' length.Low-level is a list,since we do not need to remove or update component in the list, i can not think of any downside in our usage.However, if we do not consolidate we can avoid copy,which will improve the performance. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18723: [SPARK-21517][CORE] Avoid copying memory when transfer c...
Github user kiszk commented on the issue: https://github.com/apache/spark/pull/18723 IIUC, this PR avoid data copy for consolidation in `netty.buffer.CompositeByteBuf.consolidateIfNeeded`. I think that it is reasonable. I have two questions to understand this change. 1. Where OOM happens at `consoolidateIfNeeded` in the original code? 2. Is there any downside (e.g. performance degradation) to keep many components in `CompositeByteBuf`? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18723: [SPARK-21517][CORE] Avoid copying memory when transfer c...
Github user caneGuy commented on the issue: https://github.com/apache/spark/pull/18723 Can anyone help verify this?Thank too much. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18723: [SPARK-21517][CORE] Avoid copying memory when transfer c...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18723 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
