[
https://issues.apache.org/jira/browse/FLINK-13127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16880494#comment-16880494
]
David Moravek commented on FLINK-13127:
---------------------------------------
Hello Yang, yes you can definitely do this. Problem is when you're using
existing libraries that rely on classloading. Also yarnship behavior is now
inconsistent, because classloading works for archives, but not for resources
(*which are added to classpath too, but in incorrect format*).
I think proper behavior would be:
* We traverse ship directory recursively
** If we find an archive, we add it to a archives list
** If we find a resource, we resolve its parent directory and add it to
resourceDirectories set
* We construct classpath for shipped files as follows:
sorted(resourceDirectories) + sorted(archives)
I'll provide a patch for this today.
> "--yarnship" doesn't support resource classloading
> --------------------------------------------------
>
> Key: FLINK-13127
> URL: https://issues.apache.org/jira/browse/FLINK-13127
> Project: Flink
> Issue Type: Bug
> Components: Deployment / YARN
> Affects Versions: 1.8.1
> Reporter: David Moravek
> Priority: Major
>
> Currently yarnship works as follows:
> * user specifies directory to ship with the job
> * yarn ships it with the container
> * org.apache.flink.yarn.AbstractYarnClusterDescriptor#uploadAndRegisterFiles
> traverses directory recursively and adds each file to the classpath
> This works well for shipping jars, but doesn't work correctly with shipping
> resources that we want to load using java.lang.ClassLoader#getResource method.
> In order to make resource classloading work, we need to register it's
> directory instead of the file itself (java classpath expects directories or
> archives).
> CLASSPATH="shipped/custom.conf:${CLASSPATH}" needs to become
> CLASSPATH="shipped:${CLASSPATH}"
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
