Hi, I had a closer look into this and opened an PR to fix the issue: https://github.com/apache/flink/pull/1457
As I am afraid of side effects I am not aware of, please give feedback if this fix can be merged or not... Thx. -Matthias On 12/11/2015 06:26 PM, Nick Dimiduk wrote: > Ah I see. This explains the issues I had with submitting streaming jobs > that package JDBC drivers. Is there a second in the guide/docs about > classloader considerations with Flink? > > On Thu, Dec 10, 2015 at 11:53 PM, Stephan Ewen <se...@apache.org> wrote: > >> Flink's classloading is different from Hadoop's. >> >> In Hadoop, the entire JVM is started with all classes (including the user >> jar) in the classpath already. In Flink, jars are added dymanically, to >> running JVMs with custom class loaders. That way, running worker/master >> processes can accept new jars without restarts. Important for low-latency, >> shells, etc >> >> Flink itself respects these classloaders whenever dynamically looking up a >> class. It may be that Closure is written such that it can only dynamically >> instantiate what is the original classpath. >> >> >> >> On Fri, Dec 11, 2015 at 1:31 AM, Nick Dimiduk <ndimi...@apache.org> wrote: >> >>> As far as the jvm is concerned, clojure is just another library. You >> should >>> be able to package it up like any other dependency and submit the job. >>> That's always how it worked in Hadoop/MR anyway... >>> >>> On Thu, Dec 10, 2015 at 3:22 PM, Matthias J. Sax <mj...@apache.org> >> wrote: >>> >>>> Thanks for this idea. >>>> >>>> I extended my pom to include clojure-1.5.1.jar in my program jar. >>>> However, the problem is still there... I did some research on the >>>> Internet, and it seems I need to mess around with Clojure's class >>>> loading strategy... >>>> >>>> -Matthias >>>> >>>> On 12/10/2015 06:47 PM, Nick Dimiduk wrote: >>>>> I think Mattias's project is using maven though -- there's a pom in >> the >>>>> project that doesn't look generated. If you want to do it from lein, >>>> maybe >>>>> my old lein-hadoop [0] plugin can help? >>>>> >>>>> [0]: https://github.com/ndimiduk/lein-hadoop >>>>> >>>>> On Thu, Dec 10, 2015 at 8:54 AM, Robert Metzger <rmetz...@apache.org >>> >>>> wrote: >>>>> >>>>>> I had the same though as Nick. Maybe Leiningen allows to somehow >>> build a >>>>>> fat-jar containing the clojure standard library. >>>>>> >>>>>> On Thu, Dec 10, 2015 at 5:51 PM, Nick Dimiduk <ndimi...@apache.org> >>>> wrote: >>>>>> >>>>>>> What happens when you follow the packaging examples provided in the >>>> flink >>>>>>> quick start archetypes? These have the maven-foo required to >> package >>> an >>>>>>> uberjar suitable for flink submission. Can you try adding that step >>> to >>>>>> your >>>>>>> pom.xml? >>>>>>> >>>>>>> On Thursday, December 10, 2015, Stephan Ewen <se...@apache.org> >>> wrote: >>>>>>> >>>>>>>> This is a problem in Java. >>>>>>>> I think you cannot dynamically modify the initial system class >>> loader. >>>>>>>> >>>>>>>> What most apps do is check for the thread context class loader >> when >>>>>>>> dynamically loading classes. We can check and make sure that one >> is >>>>>> set, >>>>>>>> but if Closure does not respect that, we have a problem. >>>>>>>> Then Closure is not built for dynamic class loading. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Thu, Dec 10, 2015 at 5:15 PM, Matthias J. Sax < >> mj...@apache.org >>>>>>>> <javascript:;>> wrote: >>>>>>>> >>>>>>>>> Would it make sense (if possible?) for Flink to add the user jar >>>>>>>>> dynamically to it's own classpath so Clojure can find it? Or >>> somehow >>>>>>>>> modify Clojure's class loader? >>>>>>>>> >>>>>>>>> The jars in lib are added to the classpath at startup. This makes >>> it >>>>>>>>> practically impossible to execute a Flink program that is written >>> in >>>>>>>>> Clojure right now... >>>>>>>>> >>>>>>>>> >>>>>>>>> On 12/10/2015 05:09 PM, Aljoscha Krettek wrote: >>>>>>>>>> Clojure is not considering the user-jar when trying to load the >>>>>>> class. >>>>>>>>>> >>>>>>>>>>> On 10 Dec 2015, at 17:05, Matthias J. Sax <mj...@apache.org >>>>>>>> <javascript:;>> wrote: >>>>>>>>>>> >>>>>>>>>>> Hi Squirrels, >>>>>>>>>>> >>>>>>>>>>> I was playing with a Flink Clojure WordCount example today. >>>>>>>>>>> >>> https://github.com/mjsax/flink-external/tree/master/flink-clojure >>>>>>>>>>> >>>>>>>>>>> After building the project with "mvn package" I tried to submit >>> it >>>>>>> to >>>>>>>> a >>>>>>>>>>> local cluster. Before I started the cluster, I manually copied >>>>>>>>>>> "clojure-1.5.1.jar" into Flink's lib folder. >>>>>>>>>>> >>>>>>>>>>>> cp >> ~/.m2/repository/org/clojure/clojure/1.5.1/clojure-1.5.1.jar >>>>>>> lib/ >>>>>>>>>>>> bin/start-local.sh >>>>>>>>>>> >>>>>>>>>>> However, when submitting the jar, I get an exception: >>>>>>>>>>> >>>>>>>>>>>> bin/flink run -c org.apache.flink.clojure.WordCount >>>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>> >>> >> ~/workspace_flink/flink-external/flink-clojure/target/flink-clojure-0.10.0.jar >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> ------------------------------------------------------------ >>>>>>>>>>>> The program finished with the following exception: >>>>>>>>>>>> >>>>>>>>>>>> org.apache.flink.client.program.ProgramInvocationException: >> The >>>>>>>>> program's entry point class 'org.apache.flink.clojure.WordCount' >>>>>> threw >>>>>>> an >>>>>>>>> error during initialization. >>>>>>>>>>>> at >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>> >>> >> org.apache.flink.client.program.PackagedProgram.loadMainClass(PackagedProgram.java:585) >>>>>>>>>>>> at >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>> >>> >> org.apache.flink.client.program.PackagedProgram.<init>(PackagedProgram.java:195) >>>>>>>>>>>> at >>>>>>>>> >>>>>> >> org.apache.flink.client.CliFrontend.buildProgram(CliFrontend.java:784) >>>>>>>>>>>> at >> org.apache.flink.client.CliFrontend.run(CliFrontend.java:288) >>>>>>>>>>>> at >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>> >>> >> org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1050) >>>>>>>>>>>> at >>>>>> org.apache.flink.client.CliFrontend.main(CliFrontend.java:1103) >>>>>>>>>>>> Caused by: java.lang.ExceptionInInitializerError >>>>>>>>>>>> at java.lang.Class.forName0(Native Method) >>>>>>>>>>>> at java.lang.Class.forName(Class.java:278) >>>>>>>>>>>> at >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>> >>> >> org.apache.flink.client.program.PackagedProgram.loadMainClass(PackagedProgram.java:578) >>>>>>>>>>>> ... 5 more >>>>>>>>>>>> Caused by: java.io.FileNotFoundException: Could not locate >>>>>>>>> org/apache/flink/clojure/WordCount__init.class or >>>>>>>>> org/apache/flink/clojure/WordCount.clj on classpath: >>>>>>>>>>>> at clojure.lang.RT.load(RT.java:443) >>>>>>>>>>>> at clojure.lang.RT.load(RT.java:411) >>>>>>>>>>>> at clojure.core$load$fn__5018.invoke(core.clj:5530) >>>>>>>>>>>> at clojure.core$load.doInvoke(core.clj:5529) >>>>>>>>>>>> at clojure.lang.RestFn.invoke(RestFn.java:408) >>>>>>>>>>>> at clojure.lang.Var.invoke(Var.java:415) >>>>>>>>>>>> at org.apache.flink.clojure.WordCount.<clinit>(Unknown Source) >>>>>>>>>>>> ... 8 more >>>>>>>>>>> >>>>>>>>>>> I am not sure why the class is not found. It is contained in >> the >>>>>> jar >>>>>>>>>>> file. I can fix this error by copying the user jar >>>>>>>>>>> (flink-clojure-0.10.0.jar) into Flink's lib-folder. >>>>>>>>>>> >>>>>>>>>>> It seems, that Flink is not looking into the user-jar when >>> loading >>>>>>>> this >>>>>>>>>>> class. Can anybody explain why? >>>>>>>>>>> >>>>>>>>>>> Thx. >>>>>>>>>>> >>>>>>>>>>> -Matthias >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>>> >>> >> >
signature.asc
Description: OpenPGP digital signature
