dianfu commented on a change in pull request #11682: [FLINK-16668][python]
Refactor PythonDriver to read python dependency info both from command line and
environment configuration.
URL: https://github.com/apache/flink/pull/11682#discussion_r406600893
##########
File path:
flink-python/src/main/java/org/apache/flink/client/python/PythonDriverEnvUtils.java
##########
@@ -226,6 +186,48 @@ public FileVisitResult visitFile(java.nio.file.Path file,
BasicFileAttributes at
return libFiles;
}
+ private static void appendUserFiles(PythonEnvironment env, List<Path>
userFileList) throws IOException {
+ List<String> pythonPathList = new ArrayList<>();
+ Path tmpDirPath = new Path(env.tempDirectory);
+
+ for (Path pythonFile : userFileList) {
+ String sourceFileName = pythonFile.getName();
+ // add random UUID parent directory to avoid name
conflict.
+ Path targetPath = new Path(
+ tmpDirPath,
+ String.join(File.separator,
UUID.randomUUID().toString(), sourceFileName));
+ try {
+ if
(!pythonFile.getFileSystem().isDistributedFS()) {
+ if
(!pythonFile.getFileSystem().exists(pythonFile)) {
+ LOG.error(pythonFile + " not
exists, skipping...");
+ continue;
+ }
+ // if the path is local file, try to
create symbolic link.
+ new
File(targetPath.getParent().toString()).mkdir();
+ createSymbolicLinkForPyflinkLib(
+ Paths.get(new
File(pythonFile.getPath()).getAbsolutePath()),
+
Paths.get(targetPath.toString()));
+ } else {
+ FileUtils.copy(pythonFile, targetPath,
true);
+ }
+ } catch (Exception e) {
+ LOG.error("Error occured when copying user
python files, skipping...", e);
Review comment:
It seems that we should throw out the exception in this case.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
