[jira] [Updated] (FLINK-14422) Add metric for shuffle memory
[
https://issues.apache.org/jira/browse/FLINK-14422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias updated FLINK-14422:
-
Description: This issue refers to Step 2 in the implementation proposal of
[FLIP-102|https://cwiki.apache.org/confluence/display/FLINK/FLIP-102%3A+Add+More+Metrics+to+TaskManager]
(was: * add getTotalMemorySize and in getAvaliableMemorySize
NetworkBufferPool
{code:java}
public long getTotalMemorySize() {
return 1L * getTotalNumberOfMemorySegments() * memorySegmentSize;
}
public long getAvaliableMemorySize() {
return 1L * getNumberOfAvailableMemorySegments() * memorySegmentSize;
}
{code}
* update NettyShuffleMetricFactory#registerShuffleMetrics
{code:java}
private static final String METRIC_TOTAL_MEMORY_SEGMENT_TOTALCAPACITY =
"TotalMemoryCapacity";
private static final String METRIC_TOTAL_MEMORY_SEGMENT_AVALIABLEMEMORY =
"AvaliableMemory";
private static void registerShuffleMetrics(
String groupName,
MetricGroup metricGroup,
NetworkBufferPool networkBufferPool) {
MetricGroup networkGroup = metricGroup.addGroup(groupName);
networkGroup.>gauge(METRIC_TOTAL_MEMORY_SEGMENT,
networkBufferPool::getTotalNumberOfMemorySegments);
networkGroup.>gauge(METRIC_AVAILABLE_MEMORY_SEGMENT,
networkBufferPool::getNumberOfAvailableMemorySegments);
networkGroup.>gauge(METRIC_TOTAL_MEMORY_SEGMENT_TOTALCAPACITY,
networkBufferPool::getTotalMemorySize);
networkGroup.>gauge(METRIC_TOTAL_MEMORY_SEGMENT_AVALIABLEMEMORY,
networkBufferPool::getAvaliableMemorySize);
}
{code})
> Add metric for shuffle memory
> -
>
> Key: FLINK-14422
> URL: https://issues.apache.org/jira/browse/FLINK-14422
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Task
>Reporter: lining
>Priority: Major
>
> This issue refers to Step 2 in the implementation proposal of
> [FLIP-102|https://cwiki.apache.org/confluence/display/FLINK/FLIP-102%3A+Add+More+Metrics+to+TaskManager]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-14422) Add metric for shuffle memory
[
https://issues.apache.org/jira/browse/FLINK-14422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
lining updated FLINK-14422:
---
Description:
* add getTotalMemorySize and in getAvaliableMemorySize NetworkBufferPool
{code:java}
public long getTotalMemorySize() {
return 1L * getTotalNumberOfMemorySegments() * memorySegmentSize;
}
public long getAvaliableMemorySize() {
return 1L * getNumberOfAvailableMemorySegments() * memorySegmentSize;
}
{code}
* update NettyShuffleMetricFactory#registerShuffleMetrics
{code:java}
private static final String METRIC_TOTAL_MEMORY_SEGMENT_TOTALCAPACITY =
"TotalMemoryCapacity";
private static final String METRIC_TOTAL_MEMORY_SEGMENT_AVALIABLEMEMORY =
"AvaliableMemory";
private static void registerShuffleMetrics(
String groupName,
MetricGroup metricGroup,
NetworkBufferPool networkBufferPool) {
MetricGroup networkGroup = metricGroup.addGroup(groupName);
networkGroup.>gauge(METRIC_TOTAL_MEMORY_SEGMENT,
networkBufferPool::getTotalNumberOfMemorySegments);
networkGroup.>gauge(METRIC_AVAILABLE_MEMORY_SEGMENT,
networkBufferPool::getNumberOfAvailableMemorySegments);
networkGroup.>gauge(METRIC_TOTAL_MEMORY_SEGMENT_TOTALCAPACITY,
networkBufferPool::getTotalMemorySize);
networkGroup.>gauge(METRIC_TOTAL_MEMORY_SEGMENT_AVALIABLEMEMORY,
networkBufferPool::getAvaliableMemorySize);
}
{code}
was:
* add getTotalMemorySize and in getAvaliableMemorySize NetworkBufferPool
{code:java}
public long getTotalMemorySize() {
return 1L * getTotalNumberOfMemorySegments() *
memorySegmentSize;
}
public long getAvaliableMemorySize() {
return 1L * getNumberOfAvailableMemorySegments() *
memorySegmentSize;
}
{code}
> Add metric for shuffle memory
> -
>
> Key: FLINK-14422
> URL: https://issues.apache.org/jira/browse/FLINK-14422
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Task
>Reporter: lining
>Priority: Major
>
> * add getTotalMemorySize and in getAvaliableMemorySize NetworkBufferPool
> {code:java}
> public long getTotalMemorySize() {
> return 1L * getTotalNumberOfMemorySegments() * memorySegmentSize;
> }
> public long getAvaliableMemorySize() {
> return 1L * getNumberOfAvailableMemorySegments() * memorySegmentSize;
> }
> {code}
>
> * update NettyShuffleMetricFactory#registerShuffleMetrics
> {code:java}
> private static final String METRIC_TOTAL_MEMORY_SEGMENT_TOTALCAPACITY =
> "TotalMemoryCapacity";
> private static final String METRIC_TOTAL_MEMORY_SEGMENT_AVALIABLEMEMORY =
> "AvaliableMemory";
> private static void registerShuffleMetrics(
> String groupName,
> MetricGroup metricGroup,
> NetworkBufferPool networkBufferPool) {
> MetricGroup networkGroup = metricGroup.addGroup(groupName);
> networkGroup.>gauge(METRIC_TOTAL_MEMORY_SEGMENT,
>
> networkBufferPool::getTotalNumberOfMemorySegments);
> networkGroup. Gauge>gauge(METRIC_AVAILABLE_MEMORY_SEGMENT,
>
> networkBufferPool::getNumberOfAvailableMemorySegments);
> networkGroup. Gauge>gauge(METRIC_TOTAL_MEMORY_SEGMENT_TOTALCAPACITY,
>
> networkBufferPool::getTotalMemorySize);
> networkGroup. Gauge>gauge(METRIC_TOTAL_MEMORY_SEGMENT_AVALIABLEMEMORY,
>
> networkBufferPool::getAvaliableMemorySize);
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-14422) Add metric for shuffle memory
[
https://issues.apache.org/jira/browse/FLINK-14422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
lining updated FLINK-14422:
---
Description:
* add getTotalMemorySize and in getAvaliableMemorySize NetworkBufferPool
{code:java}
public long getTotalMemorySize() {
return 1L * getTotalNumberOfMemorySegments() *
memorySegmentSize;
}
public long getAvaliableMemorySize() {
return 1L * getNumberOfAvailableMemorySegments() *
memorySegmentSize;
}
{code}
was:
TaskExecutors in the same Flink cluster might be configured with different
resources. It would be helpful to expose TaskExecutors' resource configs to
RESTful API, so it can be displayed on Flink web ui or queried by external
systems.
Currently, resource configurations are used for creating TaskExecutorServices,
which will be passed into the TaskExecutor. The TaskExecutor, after created, is
not aware of its resource configuration.
We propose to also pass the original resource configs into the TaskExecutor,
and report it to ResourceManager on registration. In that way, the rest server
can acquire the resource configs from ResourceManager, as part of
TaskManagerInfo.
> Add metric for shuffle memory
> -
>
> Key: FLINK-14422
> URL: https://issues.apache.org/jira/browse/FLINK-14422
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Task
>Reporter: lining
>Priority: Major
>
> * add getTotalMemorySize and in getAvaliableMemorySize NetworkBufferPool
> {code:java}
> public long getTotalMemorySize() {
> return 1L * getTotalNumberOfMemorySegments() *
> memorySegmentSize;
> }
> public long getAvaliableMemorySize() {
> return 1L * getNumberOfAvailableMemorySegments() *
> memorySegmentSize;
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-14422) Add metric for shuffle memory
[ https://issues.apache.org/jira/browse/FLINK-14422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lining updated FLINK-14422: --- Summary: Add metric for shuffle memory (was: Expose TaskExecutor resource configurations to rest api) > Add metric for shuffle memory > - > > Key: FLINK-14422 > URL: https://issues.apache.org/jira/browse/FLINK-14422 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Task >Reporter: lining >Priority: Major > > TaskExecutors in the same Flink cluster might be configured with different > resources. It would be helpful to expose TaskExecutors' resource configs to > RESTful API, so it can be displayed on Flink web ui or queried by external > systems. > Currently, resource configurations are used for creating > TaskExecutorServices, which will be passed into the TaskExecutor. The > TaskExecutor, after created, is not aware of its resource configuration. > We propose to also pass the original resource configs into the TaskExecutor, > and report it to ResourceManager on registration. In that way, the rest > server can acquire the resource configs from ResourceManager, as part of > TaskManagerInfo. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
