[
https://issues.apache.org/jira/browse/FLINK-22871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17357193#comment-17357193
]
konwu commented on FLINK-22871:
-------------------------------
ok [~dian.fu] , the
# upload venv.zip to hdfs
# add flink config
{code:java}
python.files=hdfs://fs/venv.zip,hdfs://fs/foo/main.py {code}
# update PythonEnvUtils.addToPythonPath method ,unzip the venv.zip, and add
targetpath to Env
{code:java}
FileUtils.copy(pythonFile, targetPath, true);
if (sourceFileName.endsWith(".zip")) {
ZipUtils.extractZipFileWithPermissions(targetPath.getPath(),
targetPath.getParent().getPath());
env.systemEnv.put("FLINK_USER_ZIP_" + targetPath .getName()
.replace(".zip", "")
.toUpperCase(),
targetPath.getParent().getPath());
}{code}
# edit PythonEnvUtils.preparePythonEnvironment ,move the first step( 1. set
the path of python interpreter) to the end of this method, and set
env.pythonExec from user config
{code:java}
// 1. set the path of python interpreter.
String pythonExec =
config.getOptional(PYTHON_CLIENT_EXECUTABLE)
.orElse(System.getenv(PYFLINK_CLIENT_EXECUTABLE));
// get path by upload venv
Optional<String> configuredExeEnv = config.getOptional(ConfigOptions
.key("venv.python.client.executable")
.stringType()
.noDefaultValue());
Optional<String> configuredExePath = config.getOptional(ConfigOptions
.key("venv.python.client.executable.path")
.stringType()
.noDefaultValue());
if (configuredExeEnv.isPresent() && configuredExePath.isPresent()) {
pythonExec = env.systemEnv.get(configuredExeEnv.get()) +
File.separator
+ configuredExePath.get();
}
{code}
# finnaly , set mainClass: org.apache.flink.client.python.PythonDriver,
mainArgs: ["-pym", "main"], and pipeline.jars: flink-python_2.12-1.12.1.jar ,
it will running success
> pyflink deploy support yarn application mode
> --------------------------------------------
>
> Key: FLINK-22871
> URL: https://issues.apache.org/jira/browse/FLINK-22871
> Project: Flink
> Issue Type: New Feature
> Components: API / Python, Deployment / YARN
> Affects Versions: 1.12.1
> Reporter: konwu
> Priority: Minor
>
> for now pyflink is not support hadoop yarn application mode, cause of yarn
> nodemanager may not have suitable python version
> after test for use venv(python virtual environment) that uploaded by
> 'python.files' properties, then change 'env.pythonExec' path, it also works
> so,is there any possiable to support this in a suitable way
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
