[
https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16470789#comment-16470789
]
Edwina Lu commented on SPARK-23206:
---
[~irashid], I do not have the rest of the changes
[
https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16472405#comment-16472405
]
Edwina Lu commented on SPARK-23206:
---
The design discussion for SPARK-23206 is scheduled
[
https://issues.apache.org/jira/browse/SPARK-24356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16492960#comment-16492960
]
Edwina Lu commented on SPARK-24356:
---
Thanks! This is interesting, and could help with
[
https://issues.apache.org/jira/browse/SPARK-26329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16745362#comment-16745362
]
Edwina Lu commented on SPARK-26329:
---
[~wypoon], this sounds like a good solution. Send
[
https://issues.apache.org/jira/browse/SPARK-23429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Edwina Lu updated SPARK-23429:
--
Fix Version/s: 3.0.0
> Add executor memory metrics to heartbeat and expose in executors REST API
> ---
[
https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16665628#comment-16665628
]
Edwina Lu commented on SPARK-23206:
---
[~irashid], yes, I am planning to work on the oth
[
https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16355994#comment-16355994
]
Edwina Lu commented on SPARK-23206:
---
We ([~jerryshao], [~zhz] and I) are planning a con
[
https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16357841#comment-16357841
]
Edwina Lu commented on SPARK-23206:
---
[~jerryshao], thanks for your help and advice. [~c
[
https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16357846#comment-16357846
]
Edwina Lu commented on SPARK-23206:
---
[~cltlfcjin], thanks for uploading the screenshot
[
https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16357846#comment-16357846
]
Edwina Lu edited comment on SPARK-23206 at 2/9/18 2:30 AM:
---
[~c
[
https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16358866#comment-16358866
]
Edwina Lu commented on SPARK-23206:
---
[~irashid], total memory and off heap memory is al
[
https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Edwina Lu updated SPARK-23206:
--
Issue Type: Umbrella (was: Improvement)
> Additional Memory Tuning Metrics
> -
Edwina Lu created SPARK-23429:
-
Summary: Add executor memory metrics to heartbeat and expose in
executors REST API
Key: SPARK-23429
URL: https://issues.apache.org/jira/browse/SPARK-23429
Project: Spark
[
https://issues.apache.org/jira/browse/SPARK-23429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16364967#comment-16364967
]
Edwina Lu commented on SPARK-23429:
---
Subtask of SPARK-23206
> Add executor memory metr
[
https://issues.apache.org/jira/browse/SPARK-23429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Edwina Lu updated SPARK-23429:
--
Issue Type: Sub-task (was: Improvement)
Parent: SPARK-23206
> Add executor memory metrics to h
Edwina Lu created SPARK-23431:
-
Summary: Expose the new executor memory metrics at the stage level
Key: SPARK-23431
URL: https://issues.apache.org/jira/browse/SPARK-23431
Project: Spark
Issue Typ
Edwina Lu created SPARK-23432:
-
Summary: Expose executor memory metrics in the web UI for
executors and stages
Key: SPARK-23432
URL: https://issues.apache.org/jira/browse/SPARK-23432
Project: Spark
[
https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16364979#comment-16364979
]
Edwina Lu commented on SPARK-23206:
---
After discussion with [~cltlfcjin] and the eBay Ha
[
https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16378879#comment-16378879
]
Edwina Lu commented on SPARK-23206:
---
[~assia6], I haven't submitted the PR yet – the co
[
https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16387083#comment-16387083
]
Edwina Lu commented on SPARK-23206:
---
[~assia6], sorry for the delay. I'm planning to su
[
https://issues.apache.org/jira/browse/SPARK-23429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16409775#comment-16409775
]
Edwina Lu commented on SPARK-23429:
---
[~assia6], it's being reviewed internally right no
[
https://issues.apache.org/jira/browse/SPARK-23429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Edwina Lu updated SPARK-23429:
--
Description:
Add new executor level memory metrics ( jvmUsedMemory, onHeapExecutionMemory,
offHeapExec
[
https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16430765#comment-16430765
]
Edwina Lu commented on SPARK-23206:
---
[~irashid], this would be quantile values for peak
[
https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16430922#comment-16430922
]
Edwina Lu commented on SPARK-23206:
---
The doc wasn't very clear what the quantile values
[
https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16438879#comment-16438879
]
Edwina Lu commented on SPARK-23206:
---
After discussion with [~irashid] on the PR, we've
[
https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Edwina Lu updated SPARK-23206:
--
Attachment: SPARK-23206 Design Doc.pdf
> Additional Memory Tuning Metrics
> ---
[
https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Edwina Lu updated SPARK-23206:
--
Attachment: SPARK-23206 Design Doc.pdf
> Additional Memory Tuning Metrics
> ---
[
https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Edwina Lu updated SPARK-23206:
--
Attachment: (was: SPARK-23206 Design Doc.pdf)
> Additional Memory Tuning Metrics
>
[
https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16438879#comment-16438879
]
Edwina Lu edited comment on SPARK-23206 at 4/17/18 3:14 AM:
A
[
https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16438879#comment-16438879
]
Edwina Lu edited comment on SPARK-23206 at 4/17/18 3:15 AM:
A
[
https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16440326#comment-16440326
]
Edwina Lu commented on SPARK-23206:
---
[~smilegator], please try:
https://docs.google.co
[
https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16441006#comment-16441006
]
Edwina Lu commented on SPARK-23206:
---
[~assia6], could you please try the new link,
[ht
[
https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16442928#comment-16442928
]
Edwina Lu commented on SPARK-23206:
---
[~assia6], there is an initial PR: [[Github] Pull
[
https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16443025#comment-16443025
]
Edwina Lu commented on SPARK-23206:
---
We are planning a design discussion for early next
[
https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16446247#comment-16446247
]
Edwina Lu commented on SPARK-23206:
---
The design discussion is scheduled for Monday 4/23
[
https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16446247#comment-16446247
]
Edwina Lu edited comment on SPARK-23206 at 4/22/18 3:36 AM:
C
Edwina Lu created SPARK-23206:
-
Summary: Additional Memory Tuning Metrics
Key: SPARK-23206
URL: https://issues.apache.org/jira/browse/SPARK-23206
Project: Spark
Issue Type: Improvement
[
https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Edwina Lu updated SPARK-23206:
--
Attachment: MemoryTuningMetricsDesignDoc.pdf
> Additional Memory Tuning Metrics
> -
[
https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Edwina Lu updated SPARK-23206:
--
Description:
At LinkedIn, we have multiple clusters, running thousands of Spark
applications, and thes
[
https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16338559#comment-16338559
]
Edwina Lu commented on SPARK-23206:
---
Thanks, [~zsxwing]. Making the new metrics availab
[
https://issues.apache.org/jira/browse/SPARK-9103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16340449#comment-16340449
]
Edwina Lu commented on SPARK-9103:
--
We (at LinkedIn) are interested in gathering more mem
[
https://issues.apache.org/jira/browse/SPARK-9103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16340449#comment-16340449
]
Edwina Lu edited comment on SPARK-9103 at 1/26/18 2:08 AM:
---
We (
[
https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16340474#comment-16340474
]
Edwina Lu commented on SPARK-23206:
---
[~jerryshao], yes SPARK-9103 is similar, and this
[
https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Edwina Lu updated SPARK-23206:
--
Attachment: StageTab.png
> Additional Memory Tuning Metrics
>
>
>
[
https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Edwina Lu updated SPARK-23206:
--
Attachment: ExecutorsTab.png
> Additional Memory Tuning Metrics
>
>
>
[
https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Edwina Lu updated SPARK-23206:
--
Attachment: (was: ExecutorsTab.png)
> Additional Memory Tuning Metrics
> --
[
https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Edwina Lu updated SPARK-23206:
--
Attachment: ExecutorsTab.png
> Additional Memory Tuning Metrics
>
>
>
[
https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16341526#comment-16341526
]
Edwina Lu commented on SPARK-23206:
---
We'd like to monitor the following executor level
[
https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Edwina Lu updated SPARK-23206:
--
Attachment: ExecutorsTab.png
> Additional Memory Tuning Metrics
>
>
>
[
https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Edwina Lu updated SPARK-23206:
--
Attachment: (was: ExecutorsTab.png)
> Additional Memory Tuning Metrics
> --
[
https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Edwina Lu updated SPARK-23206:
--
Attachment: (was: StageTab.png)
> Additional Memory Tuning Metrics
> --
[
https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Edwina Lu updated SPARK-23206:
--
Attachment: StageTab.png
> Additional Memory Tuning Metrics
>
>
>
[
https://issues.apache.org/jira/browse/SPARK-26329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16715491#comment-16715491
]
Edwina Lu commented on SPARK-26329:
---
More frequent polling (and more accurate metrics)
Edwina Lu created SPARK-26341:
-
Summary: Expose executor memory metrics at the stage level, in the
Stages tab
Key: SPARK-26341
URL: https://issues.apache.org/jira/browse/SPARK-26341
Project: Spark
[
https://issues.apache.org/jira/browse/SPARK-23432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Edwina Lu updated SPARK-23432:
--
Description:
Add the new memory metrics (jvmUsedMemory, executionMemory, storageMemory, and
unifiedMe
[
https://issues.apache.org/jira/browse/SPARK-23432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Edwina Lu updated SPARK-23432:
--
Description:
Add the new memory metrics (jvmUsedMemory, executionMemory, storageMemory, and
unifiedMe
[
https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16717988#comment-16717988
]
Edwina Lu commented on SPARK-23206:
---
[~irashid], [~rezasafi], [~wypoon], as discussed,
[
https://issues.apache.org/jira/browse/SPARK-23432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Edwina Lu updated SPARK-23432:
--
Summary: Expose executor memory metrics in the web UI for executors (was:
Expose executor memory metr
[
https://issues.apache.org/jira/browse/SPARK-23431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16724472#comment-16724472
]
Edwina Lu commented on SPARK-23431:
---
Splitting off the new REST APIs into a new subtas
[
https://issues.apache.org/jira/browse/SPARK-23431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Edwina Lu updated SPARK-23431:
--
Description:
Collect and show the new executor memory metrics for each stage, to provide
more informa
Edwina Lu created SPARK-26399:
-
Summary: Add new stage-level REST APIs and parameters
Key: SPARK-26399
URL: https://issues.apache.org/jira/browse/SPARK-26399
Project: Spark
Issue Type: Sub-task
[
https://issues.apache.org/jira/browse/SPARK-26399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Edwina Lu updated SPARK-26399:
--
Description:
Add the peak values for the metrics to the stages REST API. Also add a new
executorSumma
62 matches
Mail list logo