[
https://issues.apache.org/jira/browse/FLINK-18738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17176176#comment-17176176
]
Stephan Ewen commented on FLINK-18738:
--------------------------------------
If we go with the approach to use the managed memory (or a fraction thereof,
like 50%), then we may not even need another component like an
"ExternalExecutionBackend".
If you look at the way this is implemented for RocksDB, it only has an
"external resource initializer" which allocates the write buffers and the block
cache. In a similar way, the Python code could have a "Python Process External
Resource" which starts the Python process with the given size and returns the
RPC endpoint to talk to it.
The logic to start the resource only one per slot and return a reference to the
resource if it was already started before is already there (it is part of the
Memory Managers external resource tracking). The logic can be purely part of
the Python Operators, nothing needed inside the TaskExecutor or any additional
state backend or manager.
What we would need in that case is a config option to say how much of a slot's
memory RocksDB and Python should try to grab.
> Revisit resource management model for python processes.
> -------------------------------------------------------
>
> Key: FLINK-18738
> URL: https://issues.apache.org/jira/browse/FLINK-18738
> Project: Flink
> Issue Type: Task
> Components: API / Python, Runtime / Coordination
> Reporter: Xintong Song
> Assignee: Xintong Song
> Priority: Major
> Fix For: 1.12.0
>
>
> This ticket is for tracking the effort towards a proper long-term resource
> management model for python processes.
> In FLINK-17923, we run into problems due to python processes are not well
> integrate with the task manager resource management mechanism. A temporal
> workaround has been merged for release-1.11.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
