[
https://issues.apache.org/jira/browse/FLINK-35557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Roman Khachatryan updated FLINK-35557:
--------------------------------------
Description:
# In {{MemoryManager.getSharedMemoryResourceForManagedMemory}} we
[create|https://github.com/apache/flink/blob/57869c11687e0053a242c90623779c0c7336cd33/flink-runtime/src/main/java/org/apache/flink/runtime/memory/MemoryManager.java#L526]
a reserve function
# The function
[decrements|https://github.com/apache/flink/blob/57869c11687e0053a242c90623779c0c7336cd33/flink-runtime/src/main/java/org/apache/flink/runtime/memory/UnsafeMemoryBudget.java#L61]
the available Slot memory and fails if there's not enough memory
# We pass it to {{SharedResources.getOrAllocateSharedResource}}
# In {{SharedResources.getOrAllocateSharedResource}} , we check if the
resource (memory) was already reserved by some key (e.g.
{{{}state-rocks-managed-memory{}}})
# If not, we create a new one and call the reserve function
# If the resource was already reserved (not null), we do NOT reserve the
memory again:
[https://github.com/apache/flink/blob/57869c11687e0053a242c90623779c0c7336cd33/flin[…]/main/java/org/apache/flink/runtime/memory/SharedResources.java|https://github.com/apache/flink/blob/57869c11687e0053a242c90623779c0c7336cd33/flink-runtime/src/main/java/org/apache/flink/runtime/memory/SharedResources.java#L71]
So there will be only one (first) memory reservation for rocksdb for example,
no matter how many state backends in a slot are created. Meaning that managed
memory limits are not applied
was:
# In {{MemoryManager.getSharedMemoryResourceForManagedMemory}} we
[create|https://github.com/apache/flink/blob/57869c11687e0053a242c90623779c0c7336cd33/flink-runtime/src/main/java/org/apache/flink/runtime/memory/MemoryManager.java#L526]
a reserve function
# The function
[decrements|https://github.com/apache/flink/blob/57869c11687e0053a242c90623779c0c7336cd33/flink-runtime/src/main/java/org/apache/flink/runtime/memory/UnsafeMemoryBudget.java#L61]
the available Slot memory and fails if there's not enough memory
# We pass it to {{SharedResources.getOrAllocateSharedResource}}
# In {{SharedResources.getOrAllocateSharedResource}} , we check if the
resource (memory) was already reserved by some key (e.g.
{{{}state-rocks-managed-memory{}}})
# If not, we create a new one and call the reserve function
# If the resource was already reserved (not null), we do NOT reserve the
memory again:
[https://github.com/apache/flink/blob/57869c11687e0053a242c90623779c0c7336cd33/flin[…]/main/java/org/apache/flink/runtime/memory/SharedResources.java|https://github.com/apache/flink/blob/57869c11687e0053a242c90623779c0c7336cd33/flink-runtime/src/main/java/org/apache/flink/runtime/memory/SharedResources.java#L71]
So there will be only one (first) memory reservation for rocksdb for example,
no matter how many state backends are created. Meaning that managed memory
limits are not followed (edited)
> MemoryManager only reserves memory per consumer type once
> ---------------------------------------------------------
>
> Key: FLINK-35557
> URL: https://issues.apache.org/jira/browse/FLINK-35557
> Project: Flink
> Issue Type: Bug
> Components: Runtime / State Backends, Runtime / Task
> Affects Versions: 1.16.3, 1.17.2, 1.19.0, 1.18.1, 1.20.0
> Reporter: Roman Khachatryan
> Assignee: Roman Khachatryan
> Priority: Major
> Fix For: 1.20.0, 1.19.1
>
>
> # In {{MemoryManager.getSharedMemoryResourceForManagedMemory}} we
> [create|https://github.com/apache/flink/blob/57869c11687e0053a242c90623779c0c7336cd33/flink-runtime/src/main/java/org/apache/flink/runtime/memory/MemoryManager.java#L526]
> a reserve function
> # The function
> [decrements|https://github.com/apache/flink/blob/57869c11687e0053a242c90623779c0c7336cd33/flink-runtime/src/main/java/org/apache/flink/runtime/memory/UnsafeMemoryBudget.java#L61]
> the available Slot memory and fails if there's not enough memory
> # We pass it to {{SharedResources.getOrAllocateSharedResource}}
> # In {{SharedResources.getOrAllocateSharedResource}} , we check if the
> resource (memory) was already reserved by some key (e.g.
> {{{}state-rocks-managed-memory{}}})
> # If not, we create a new one and call the reserve function
> # If the resource was already reserved (not null), we do NOT reserve the
> memory again:
> [https://github.com/apache/flink/blob/57869c11687e0053a242c90623779c0c7336cd33/flin[…]/main/java/org/apache/flink/runtime/memory/SharedResources.java|https://github.com/apache/flink/blob/57869c11687e0053a242c90623779c0c7336cd33/flink-runtime/src/main/java/org/apache/flink/runtime/memory/SharedResources.java#L71]
> So there will be only one (first) memory reservation for rocksdb for example,
> no matter how many state backends in a slot are created. Meaning that managed
> memory limits are not applied
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
