Github user jisookim0513 commented on the issue:
https://github.com/apache/spark/pull/16714
Ok, not including the updated blocks in task metrics reduced the size of
our event logs. But I am closing this PR as the current implementation doesn't
seem to be in the right way. Thanks for
Github user ajbozarth commented on the issue:
https://github.com/apache/spark/pull/16714
Unless there's still an issue with file size I think I'm good without this,
but I'll defer to @vanzin
---
If your project is set up for it, you can reply to this email and have your
reply
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/16714
Options put the burden on the user to figure things out (do I need to set
this or not?). If you want to investigate whether you can trim more data (e.g.
internal metrics that just mirror stuff
Github user jisookim0513 commented on the issue:
https://github.com/apache/spark/pull/16714
@vanzin @ajbozarth if you guys think having an option to skip logging
internal accumulators (in my case I don't use the SQL UI) and completely
getting rid of updated block statues are not
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/16714
I think so. Just replying to the question.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user ajbozarth commented on the issue:
https://github.com/apache/spark/pull/16714
Didn't #17412 already get rid of block statuses though?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/16714
After having played with some of this code for other reasons, at least some
of the internal accumulators are needed to rebuild the SQL UI.
As far as logging updated block statuses, I still
Github user jisookim0513 commented on the issue:
https://github.com/apache/spark/pull/16714
I would still like not to have internal accumulators in the event logs, as
well as updated block statuses. @vanzin would you be ok with eliminating all
internal accumulators and have an option
Github user ajbozarth commented on the issue:
https://github.com/apache/spark/pull/16714
Is this still an issue or did #17412 fix this?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this
Github user jasonmoore2k commented on the issue:
https://github.com/apache/spark/pull/16714
Would some of the other recent contributors to this area (e.g. @zsxwing or
@JoshRosen) be able to comment on any use for these internal accumulables /
block status updates, and whether they
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16714
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this
Github user drcrallen commented on the issue:
https://github.com/apache/spark/pull/16714
@vanzin can you check this out please?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16714
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16714
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/72058/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16714
**[Test build #72058 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72058/testReport)**
for PR 16714 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16714
**[Test build #72058 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72058/testReport)**
for PR 16714 at commit
Github user jisookim0513 commented on the issue:
https://github.com/apache/spark/pull/16714
Not sure why the second test build failed at PySpark unit tests. I only
changed the comments.
---
If your project is set up for it, you can reply to this email and have your
reply appear on
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16714
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/72048/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16714
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16714
**[Test build #72048 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72048/testReport)**
for PR 16714 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16714
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/72043/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16714
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16714
**[Test build #72043 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72043/testReport)**
for PR 16714 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16714
**[Test build #72048 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72048/testReport)**
for PR 16714 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16714
**[Test build #72043 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72043/testReport)**
for PR 16714 at commit
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/16714
ok to test
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16714
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this
27 matches
Mail list logo
