Aleksei Perminov created FLINK-34641:
----------------------------------------
Summary: Possibility to add Python Files from http/https sources
in PythonDriver
Key: FLINK-34641
URL: https://issues.apache.org/jira/browse/FLINK-34641
Project: Flink
Issue Type: New Feature
Components: flink-contrib
Affects Versions: 1.18.1, 1.18.0, 1.18.2
Reporter: Aleksei Perminov
Problem: Now it is impossible to add python files from the external http
sources, only fs sources are available. It is not very convenient to add Python
packages to the Flink Docker image or external fs every time the new version of
the Python job is built, especially when using the Session mode.
Proposal: Add possibility to handle the http/https protocols in the
PythonDriver so that it is possible to add packages from the external Python
repositories (e.g. PyPi).
Example of the solution in the PythonEnvUtils.java
{code:java}
private static void addToPythonPath(PythonEnvironment env, List<Path>
pythonFiles) {
...
if (isValidURL(pythonFile.toString()) && new
URL(pythonFile.toString()).getProtocol().startsWith("http")) {
getFileFromHTTP(pythonFile, targetPath);
} else if (!pythonFile.getFileSystem().isDistributedFS()) {
...
}{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
