When we transitioned from 1.3 to 1.4, we ran into some class loader issues.
Though we weren’t using any sophisticated class loader helicopter stunts :)
Specifically…
1. Re-worked our pom.xml to set up shading to better mirror what the 1.4
example pom was doing.
2. Enabled child-first classloading
3. Ensured “hadoop classpath” command returned nothing (or failed), to avoid
loading Hadoop jars before our jars (even with #2 above)
— Ken
> On Apr 13, 2018, at 2:28 AM, eSKa <eska...@gmail.com> wrote:
>
> Hello,
> I still have problem after upgrading from flink 1.3.1 to 1.4.2
> Our scenario looks like that:
> we have container running on top of yarn. Machine that starts it has
> installed flink and also loading some classpath libraries (e.g. hadoop) into
> container.
> there is seperate rest service that gets requests to run export job - it
> uses YarnClusterClient and submitting Packaged program. Jars with actual
> flink jobs are located in lib/ directory of service. On the machine where
> Spring service is deployed we don't have flink installed.
> For version 1.3 we had some libraries also loaded to container so that they
> wont have to be loaded dynamically every time. If I understand it correctly
> in strategy child-first it should not be needed any more, right?
>
> Now our actual problems started to come up linked with class loading. After
> restarting rest service first trigger of job is working fine, but next ones
> are complaining on class versions that are loaded. We found out that
> PackagedProgram is creating new classLoader on every creation. So we
> overriden that behaviour so that we have static map holding one classloader
> per jarFileName:
>
>
> /private static final Map<String, ClassLoader> classLoaders =
> Maps.newHashMap();
> ...
> (constructor) {
> ...
> classLoaders.computeIfAbsent(jarFile.getName(),
> s -> getUserClassLoaderChildFirst(getAllLibraries(),
> classPaths, getClass().getClassLoader()));
>
> userCodeClassLoader = classLoaders.get(jarFile.getName());
> this.mainClass = loadMainClass(entryPointClassName,
> userCodeClassLoader);
>
> }
> /
>
> I don't know if that is a good direction, but seems to solve an issue for
> now. We are just not sure about stability of this solution - still tesing on
> our internal environment but I'm affraid for now to proceed on production.
> Can you give us any other things we could try out to deal with loading?
>
>
> Also PackagedProgram is still using parentFirst strategy, in JobWithJars you
> have method:
> /
>
> public static ClassLoader buildUserCodeClassLoader(List<URL> jars,
> List<URL> classpaths, ClassLoader parent) {
> URL[] urls = new URL[jars.size() + classpaths.size()];
> for (int i = 0; i < jars.size(); i++) {
> urls[i] = jars.get(i);
> }
> for (int i = 0; i < classpaths.size(); i++) {
> urls[i + jars.size()] = classpaths.get(i);
> }
> return FlinkUserCodeClassLoaders.parentFirst(urls, parent);
> }
> /
>
> is that correct to still point to parent?
>
> In some of issues I found in mailing list, you suggest to set up container
> to parent-first as a solving issue. We would like to find proper solution
> working on supported child-first path and don't use workaround fix.
>
>
>
>
>
> --
> Sent from:
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
--------------------------------------------
http://about.me/kkrugler
+1 530-210-6378
