[
https://issues.apache.org/jira/browse/FLINK-18738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17176207#comment-17176207
]
Xintong Song commented on FLINK-18738:
--------------------------------------
[~sewen],
I see your point.
For the moment, I think introducing such a PythonProcessExternalResource should
satisfies our needs, while requiring minimal efforts.
For long term, if we want to provide a general external execution service, I
think it would be more flexible to abstract with an ExternalExecutionBackend
layer. E.g., we may have a backend that connects to an external service for
execution and does not use managed memory.
I would be ok with the PythonProcessExternalResource approach as a first step,
and postpone the ExternalExecutionBackend since we don't really need it at the
moment. IIUC, it should introduce not much (if not none) efforts that need to
be reworked if later we want to switch to the long term approach.
> Revisit resource management model for python processes.
> -------------------------------------------------------
>
> Key: FLINK-18738
> URL: https://issues.apache.org/jira/browse/FLINK-18738
> Project: Flink
> Issue Type: Task
> Components: API / Python, Runtime / Coordination
> Reporter: Xintong Song
> Assignee: Xintong Song
> Priority: Major
> Fix For: 1.12.0
>
>
> This ticket is for tracking the effort towards a proper long-term resource
> management model for python processes.
> In FLINK-17923, we run into problems due to python processes are not well
> integrate with the task manager resource management mechanism. A temporal
> workaround has been merged for release-1.11.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
