Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19170
Can one of the admins verify this patch?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19170
Can one of the admins verify this patch?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19170
Can one of the admins verify this patch?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19170
Can one of the admins verify this patch?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19170
Can one of the admins verify this patch?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19170
Can one of the admins verify this patch?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/19170
> Maybe we are using SHS too aggressively, but the GC issue is one of the
major issues we met.
Can you describe what this issue is? That is not what the bug is showing.
The bug shows a heap
Github user zhouyejoe commented on the issue:
https://github.com/apache/spark/pull/19170
@vanzin Yes, I agree with you that the latest listener will not write these
data into logs. But here is the story. We deployed SHS(Spark History Server)
with LevelDB months ago in our clusters
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/19170
(BTW, you could argue this could be useful in 2.2 and 2.1, because they
still use the old listener code. But this is just dead code in master and we
shouldn't merge it there.)
---
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/19170
The filtering on write is to reduce the size of the event log file. What is
the filtering on read achieving? Especially since any recent event logs won't
even have that data?
---
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/19170
It's not a big improvement but makes the code base more consistent.
---
-
To unsubscribe, e-mail:
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/19170
if SHS is the only user of `JsonProtocol`, then we should ignore
BlockStatus update events in `JsonProtocol` as SHS doesn't need it at all.
---
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/19170
And then the event will be processed and garbage collected and the objects
will go away?
---
-
To unsubscribe, e-mail:
Github user zhouyejoe commented on the issue:
https://github.com/apache/spark/pull/19170
@vanzin The problem still exists with your new changes to Spark History
Server. Once you use ListenerBus to replay the
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/19170
I'm not asking whether it changes anything else, I'm asking whether it does
anything anymore.
The bug shows a heap dump with a bunch of `BlockStatus` objects, but the
SHS does not create
Github user zhouyejoe commented on the issue:
https://github.com/apache/spark/pull/19170
Hi, @vanzin. No, this doesn't change anything else. It only changes how the
JSON data gets transferred into Events. I was a little bit busy with other
stuffs. I will fix the unit test.
---
Github user zsxwing commented on the issue:
https://github.com/apache/spark/pull/19170
@vanzin I think it just changes to not load BlockStatuses generated by old
Spark versions.
---
-
To unsubscribe, e-mail:
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/19170
Does this change do anything anymore? I don't think the SHS (nor the UI)
uses `BlockStatus` anymore for anything.
---
-
To
Github user zhouyejoe commented on the issue:
https://github.com/apache/spark/pull/19170
I will work on it. Thanks for review.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/19170
the hight level idea LGTM, just make sure history sever is the only
consumer for this even logs json parser.
---
-
To
Github user jiangxb1987 commented on the issue:
https://github.com/apache/spark/pull/19170
Please feel free to fix the test case failures.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For
Github user jiangxb1987 commented on the issue:
https://github.com/apache/spark/pull/19170
The change should be safe as long as the extracted BlockStatus accumulable
info is not used in the web UI. Also cc @cloud-fan
---
Github user zhouyejoe commented on the issue:
https://github.com/apache/spark/pull/19170
@jiangxb1987 Hi, I was waiting for the response from Ryan Blue about the
ticket SPARK-20084. The fix for the unit test should be pretty straight
forward. I just need a confirmation on the
Github user jiangxb1987 commented on the issue:
https://github.com/apache/spark/pull/19170
ping @zhouyejoe
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user zhouyejoe commented on the issue:
https://github.com/apache/spark/pull/19170
I will fix the unit test failure.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands,
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19170
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19170
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81951/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19170
**[Test build #81951 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81951/testReport)**
for PR 19170 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19170
**[Test build #81951 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81951/testReport)**
for PR 19170 at commit
Github user zsxwing commented on the issue:
https://github.com/apache/spark/pull/19170
cc @vanzin
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user zsxwing commented on the issue:
https://github.com/apache/spark/pull/19170
ok to test
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19170
Can one of the admins verify this patch?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
32 matches
Mail list logo
