[GitHub] spark issue #15407: [SPARK-17841][STREAMING][KAFKA] drain commitQueue
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/15407 LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15407: [SPARK-17841][STREAMING][KAFKA] drain commitQueue
Github user rxin commented on the issue: https://github.com/apache/spark/pull/15407 Thanks for the explanation. Merging in master/branch-2.0. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15407: [SPARK-17841][STREAMING][KAFKA] drain commitQueue
Github user koeninger commented on the issue: https://github.com/apache/spark/pull/15407 @rxin @tdas right now, items to be committed can be added to the queue, but they will never actually be removed from the queue. poll() removes, iterator() does not. I updated the description of the PR. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15407: [SPARK-17841][STREAMING][KAFKA] drain commitQueue
Github user rxin commented on the issue: https://github.com/apache/spark/pull/15407 @koeninger can you put more information into the description of the pull request? At the very least we should talk about the current implementation causes memory leaks. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15407: [SPARK-17841][STREAMING][KAFKA] drain commitQueue
Github user tdas commented on the issue: https://github.com/apache/spark/pull/15407 Can you explain what is the memory that can currently leak with the iterator? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15407: [SPARK-17841][STREAMING][KAFKA] drain commitQueue
Github user koeninger commented on the issue: https://github.com/apache/spark/pull/15407 This doesn't affect correctness (only the highest offset for a given partition is used in any case), just memory leaks. I'm not sure what a good way to unit test memory leaks is, short of exposing the internal state of that queue, which seems less than optimal. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15407: [SPARK-17841][STREAMING][KAFKA] drain commitQueue
Github user rxin commented on the issue: https://github.com/apache/spark/pull/15407 Is there a test we can add to cover this? I realized "How was this patch tested?" was confusing. That was meant to ask how is the change tested itself, and a bug fix obviously can't be tested by existing unit tests. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15407: [SPARK-17841][STREAMING][KAFKA] drain commitQueue
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15407 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15407: [SPARK-17841][STREAMING][KAFKA] drain commitQueue
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15407 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66609/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15407: [SPARK-17841][STREAMING][KAFKA] drain commitQueue
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15407 **[Test build #66609 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66609/consoleFull)** for PR 15407 at commit [`8810a79`](https://github.com/apache/spark/commit/8810a79fa40cd777b5d68de1fed62e7ff8f9b9e7). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15407: [SPARK-17841][STREAMING][KAFKA] drain commitQueue
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15407 **[Test build #66609 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66609/consoleFull)** for PR 15407 at commit [`8810a79`](https://github.com/apache/spark/commit/8810a79fa40cd777b5d68de1fed62e7ff8f9b9e7). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
