Github user mridulm commented on the issue:
https://github.com/apache/spark/pull/22609
I will close the PR and document in our tests to use spark.ui.retainedTasks
= 1 (or some very low value) : which will, in effect, be equivalent to this PR
- and loose information anyway; but will
Github user mridulm commented on the issue:
https://github.com/apache/spark/pull/22609
> I tried that a lot while writing this code, but the disk store is just
too slow
That is unfortunate ... I was actually hoping that would be a good
compromise if expanded api
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/22609
Task events still might need to be processed - even for live applications
with the UI disabled. See the API that has been mentioned before
(`SparkStatusTracker`) for why. Processing task events is
Github user mridulm commented on the issue:
https://github.com/apache/spark/pull/22609
@vanzin any reason why liveStore is hardcoded to be in-memory ? Any
implications of making it disk backed ? That might be another option to unblock
- requires a config change to existing
Github user mridulm commented on the issue:
https://github.com/apache/spark/pull/22609
@vanzin , @gengliangwang To clarify earlier behavior - if UI is disabled,
any task related metrics were reported as zero: granularity of update was at
the stage level.
UI is typically disabled
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/22609
@gengliangwang is right; there are still things that need to be done when
the UI is disabled; job, stage and executor info are all still exposed even if
the UI is disabled, and the task even
Github user gengliangwang commented on the issue:
https://github.com/apache/spark/pull/22609
Hi @mridulm ,
it seems that the changes will affect the metrics in `SparkStatusTracker`,
e.g. number of active tasks in `getExecutorInfos` or `getStageInfo` .
Maybe we
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22609
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22609
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96846/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22609
**[Test build #96846 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96846/testReport)**
for PR 22609 at commit
Github user mridulm commented on the issue:
https://github.com/apache/spark/pull/22609
This is a regression introduced in 2.3; unfortunately we did not notice it
until now.
On Tue, Oct 2, 2018 at 4:56 AM Shahid wrote:
> Hi @mridulm
Github user shahidki31 commented on the issue:
https://github.com/apache/spark/pull/22609
Hi @mridulm , Can't we limit the task information by setting
'spark.ui.retainedTasks' lesser, to avoid OOM? correct me if I am wrong.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22609
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22609
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22609
**[Test build #96846 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96846/testReport)**
for PR 22609 at commit
Github user mridulm commented on the issue:
https://github.com/apache/spark/pull/22609
ok to test
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user mridulm commented on the issue:
https://github.com/apache/spark/pull/22609
+CC @vanzin, @tgravescs
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
17 matches
Mail list logo
