[GitHub] spark issue #16714: [SPARK-16333][Core] Enable EventLoggingListener to log l...
Github user jisookim0513 commented on the issue: https://github.com/apache/spark/pull/16714 Ok, not including the updated blocks in task metrics reduced the size of our event logs. But I am closing this PR as the current implementation doesn't seem to be in the right way. Thanks for the inputs. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16714: [SPARK-16333][Core] Enable EventLoggingListener to log l...
Github user ajbozarth commented on the issue: https://github.com/apache/spark/pull/16714 Unless there's still an issue with file size I think I'm good without this, but I'll defer to @vanzin --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16714: [SPARK-16333][Core] Enable EventLoggingListener to log l...
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/16714 Options put the burden on the user to figure things out (do I need to set this or not?). If you want to investigate whether you can trim more data (e.g. internal metrics that just mirror stuff already in TaskMetrics) then there would be value. But adding an option doesn't really help users, since aside from you, probably nobody will ever even look at those. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16714: [SPARK-16333][Core] Enable EventLoggingListener to log l...
Github user jisookim0513 commented on the issue: https://github.com/apache/spark/pull/16714 @vanzin @ajbozarth if you guys think having an option to skip logging internal accumulators (in my case I don't use the SQL UI) and completely getting rid of updated block statues are not needed, I can close this PR. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16714: [SPARK-16333][Core] Enable EventLoggingListener to log l...
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/16714 I think so. Just replying to the question. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16714: [SPARK-16333][Core] Enable EventLoggingListener to log l...
Github user ajbozarth commented on the issue: https://github.com/apache/spark/pull/16714 Didn't #17412 already get rid of block statuses though? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16714: [SPARK-16333][Core] Enable EventLoggingListener to log l...
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/16714 After having played with some of this code for other reasons, at least some of the internal accumulators are needed to rebuild the SQL UI. As far as logging updated block statuses, I still don't like the idea of an option. It's either needed or it's not needed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16714: [SPARK-16333][Core] Enable EventLoggingListener to log l...
Github user jisookim0513 commented on the issue: https://github.com/apache/spark/pull/16714 I would still like not to have internal accumulators in the event logs, as well as updated block statuses. @vanzin would you be ok with eliminating all internal accumulators and have an option to skip logging updated block statues? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16714: [SPARK-16333][Core] Enable EventLoggingListener to log l...
Github user ajbozarth commented on the issue: https://github.com/apache/spark/pull/16714 Is this still an issue or did #17412 fix this? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16714: [SPARK-16333][Core] Enable EventLoggingListener to log l...
Github user jasonmoore2k commented on the issue: https://github.com/apache/spark/pull/16714 Would some of the other recent contributors to this area (e.g. @zsxwing or @JoshRosen) be able to comment on any use for these internal accumulables / block status updates, and whether they can be removed from the event log? I couldn't see anything that goes wrong, and my file went from 22 GB to 91 MB so it makes a big difference. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16714: [SPARK-16333][Core] Enable EventLoggingListener to log l...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16714 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16714: [SPARK-16333][Core] Enable EventLoggingListener to log l...
Github user drcrallen commented on the issue: https://github.com/apache/spark/pull/16714 @vanzin can you check this out please? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16714: [SPARK-16333][Core] Enable EventLoggingListener to log l...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16714 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16714: [SPARK-16333][Core] Enable EventLoggingListener to log l...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16714 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/72058/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16714: [SPARK-16333][Core] Enable EventLoggingListener to log l...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16714 **[Test build #72058 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72058/testReport)** for PR 16714 at commit [`f146121`](https://github.com/apache/spark/commit/f1461215c8210a19022d384a1ca95566da1406a8). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16714: [SPARK-16333][Core] Enable EventLoggingListener to log l...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16714 **[Test build #72058 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72058/testReport)** for PR 16714 at commit [`f146121`](https://github.com/apache/spark/commit/f1461215c8210a19022d384a1ca95566da1406a8). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16714: [SPARK-16333][Core] Enable EventLoggingListener to log l...
Github user jisookim0513 commented on the issue: https://github.com/apache/spark/pull/16714 Not sure why the second test build failed at PySpark unit tests. I only changed the comments. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16714: [SPARK-16333][Core] Enable EventLoggingListener to log l...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16714 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/72048/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16714: [SPARK-16333][Core] Enable EventLoggingListener to log l...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16714 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16714: [SPARK-16333][Core] Enable EventLoggingListener to log l...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16714 **[Test build #72048 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72048/testReport)** for PR 16714 at commit [`5d6cf56`](https://github.com/apache/spark/commit/5d6cf56b17dfa7151b81bab8c8a2044232c725c1). * This patch **fails PySpark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16714: [SPARK-16333][Core] Enable EventLoggingListener to log l...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16714 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/72043/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16714: [SPARK-16333][Core] Enable EventLoggingListener to log l...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16714 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16714: [SPARK-16333][Core] Enable EventLoggingListener to log l...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16714 **[Test build #72043 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72043/testReport)** for PR 16714 at commit [`b0bebcc`](https://github.com/apache/spark/commit/b0bebccb265aff4941a0356a5d7776632c2806fd). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16714: [SPARK-16333][Core] Enable EventLoggingListener to log l...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16714 **[Test build #72048 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72048/testReport)** for PR 16714 at commit [`5d6cf56`](https://github.com/apache/spark/commit/5d6cf56b17dfa7151b81bab8c8a2044232c725c1). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16714: [SPARK-16333][Core] Enable EventLoggingListener to log l...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16714 **[Test build #72043 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72043/testReport)** for PR 16714 at commit [`b0bebcc`](https://github.com/apache/spark/commit/b0bebccb265aff4941a0356a5d7776632c2806fd). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16714: [SPARK-16333][Core] Enable EventLoggingListener to log l...
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/16714 ok to test --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16714: [SPARK-16333][Core] Enable EventLoggingListener to log l...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16714 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
