On Thursday, January 29, 2015 at 4:18:15 PM UTC-8, shlomi...@gmail.com wrote: > > Hey, > Just to be sure, are you loading the full uberjar to hadoop? >
Yes. The issue isn't too few classes being found, it's too many :). If I start an nrepl server inside the job and futz around with loading and reloading a few times I can get all the code to load, but haven't figured out yet if it's possible to turn this into a klutzy workaround... > > > On Wednesday, January 28, 2015 at 11:39:54 PM UTC-8, Jason Wolfe wrote: >> >> First off, I apologize in advance for not having a reduced test case, and >> express my sincere gratitude in advance for any assistance. I've been >> tearing my hair out for a day or so and not making headway, and figured >> someone here might recognize some keywords and have a pointer in the right >> direction. (I'm admittedly pretty green when it comes to class loading, and >> have largely exhausted my google fu). >> >> *Problem: * >> >> I'm submitting a hadoop job using clojure-hadoop. All is well with a >> simple job, but once I require something that transitively depends on >> Schema, I end up with: >> >> clojure.lang.Compiler$CompilerException: >> java.lang.IllegalArgumentException: No implementation of method: :walker of >> protocol: #'schema.core/Schema found for class: clojure.core$long, >> compiling:(crane/config.clj:33:4) >> >> It works fine when run in-process with hadoop-mapreduce-client-jobclient, >> but not with bin/hadoop -jar. This stunk of a classloader issue, and after >> digging in it seems that there are multiple versions of clojure.core$long >> floating around. The version on which the protocol is extended is not the >> same class for the fn that the symbol 'long resolves to in client code. >> >> *Context: * >> >> clojure-hadoop is AOT-compiled, and after being loaded by hadoop it >> dynamically loads the target namespace (not AOT-compiled, nor any other of >> the code in question) using >> https://github.com/alexott/clojure-hadoop/blob/master/src/clojure_hadoop/load.clj#L3 >> >> From here, schema is transitively required, and then client namespaces >> attempt to use the Schema protocol to generate validators, and when the >> schema 'long is used (which resolves to the fn with class >> clojure.core$long), it fails to find the appropriate method. >> >> After repeated head-bashing, I've determined that there are (at least >> two) versions of the clojure.core$long class floating around -- the one >> used to extend the protocol, which stems from a DynamicClassLoader, and the >> one that 'long resolves to in client code, which stems from a >> URLClassLoader. The URLClassLoader is the loader of the current thread and >> Compiler, but not @(clojure.lang.Compiler/LOADER). >> >> *Attempts:* >> >> I've tried wrapping the clojure-hadoop loading code with >> .setContextClassLoader on some obvious candidates and binding >> *use-context-classloader* around the code doing the loading, with no avail. >> I've tried changing the schema code to reference the class in different >> ways (class (resolve 'long)), (class 'long), etc and that hasn't made a >> difference. I've checked and the clojure-hadoop jar doesn't contain any >> .class files for clojure, schema, or other offending code. >> >> *Plea:* >> >> I suspect there's something obvious I'm missing. (In retrospect it seems >> like the design of Schema may be suboptimal in light of this, but if >> possible I'd like to figure out a workaround without changing that >> substantially). Thanks in advance for your help -- any and all pointers are >> welcome. >> >> -Jason >> >> >> >> >> > -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
