[
https://issues.apache.org/jira/browse/FLINK-30073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17635294#comment-17635294
]
Xintong Song commented on FLINK-30073:
--------------------------------------
I'd consider this more a design limitation, rather than a bug.
IIRC, the purpose of this configuration option is for backwards compatibility.
The problem was that, rocksdb state backend used an estimation-based approach
for managing its memory, which means it cannot guarantee not use more memory
than it is expected to use. When running into corner cases, one can manually
set rocksdb to use slightly less memory than calculated from weights, while
Flink will still reserver as many memory calculated from weights for it.
I'm not sure whether this workaround is still needed. I think people have been
discussing about fixing the problem that rocksdb uses more memory than
expected, by upgrading the rocksdb version. I'm not aware how that effort ends
up. Maybe [~yunta] can share some more details.
> Managed memory can be wasted if RocksDB memory is fixed-per-slot
> ----------------------------------------------------------------
>
> Key: FLINK-30073
> URL: https://issues.apache.org/jira/browse/FLINK-30073
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Task
> Affects Versions: 1.16.0, 1.17.0, 1.15.2
> Reporter: Roman Khachatryan
> Priority: Major
>
> When
> [state.backend.rocksdb.memory.fixed-per-slot|https://github.com/apache/flink/blob/ba4b182955867fedfa9891bf0bf430e92eeab41a/flink-runtime/src/main/java/org/apache/flink/runtime/util/config/memory/ManagedMemoryUtils.java#L75]
> is set, RocksDB does not use managed memory (this option overrides the
> 'state.backend.rocksdb.memory.managed').
> However, the runtime [doesn't take this into
> account|https://github.com/apache/flink/blob/ba4b182955867fedfa9891bf0bf430e92eeab41a/flink-runtime/src/main/java/org/apache/flink/runtime/util/config/memory/ManagedMemoryUtils.java#L75]
> and still reserves the managed memory according to the configured weigths.
> cc: [~yunta]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
