We will release some open source when it's ready but the basic idea is to define a single task that implements all the interfaces, and have that delegate to clojure handlers. On Feb 10, 2016 4:39 PM, "Jacob Maes" <jacob.m...@gmail.com> wrote:
> Hey Andy, > > That's good news! Do you mind sharing the revelation for any other Samza > users in Clojure-land? > > -Jake > > On Wed, Feb 10, 2016 at 4:07 PM, Andy Chambers < > andy.chamb...@fundingcircle.com> wrote: > > > My co-worker has figured out how to do this without any changes required > in > > samza. > > Andy Chambers | Full-Stack Software Developer > > > > andy.chamb...@fundingcircle.com | 707-205-6502 (m) > > > > > > > > > > 747 Front St, 4th Fl | San Francisco, CA 94111 > > > > Our Mission: T o build a better financial world > > > > > > > > > > > > > > Unless specifically indicated, this e-mail is not an offer to sell or a > > solicitation of any investment products or other financial product or > > service, > > an official confirmation of any transaction, or an official statement of > > Funding > > Circle USA. This e-mail is meant only for the intended recipient of this > > transmission, and > > contains trade secret and strictly confidential information belonging to > > the > > sender. It is unlawful for unauthorized individuals to review, use, copy, > > disclose, or disseminate confidential information. If you have received > > this > > e-mail in error, please notify the sender immediately by telephone at > > 857.285.1263 or by return email and promptly delete this message from > > your system. > > > > > > > > On 9 February 2016 at 23:47, Andy Chambers < > > andy.chamb...@fundingcircle.com > wrote: > > No Problem. I'm new to samza so it is entirely possible there's already a > > way to > > do this that I'm missing. > > What I'm aiming for, is to implement the features described in the README > > of my > > project https://github.com/ cddr/samza-config . For discussion purposes, > > I'll copy the example here... > > (ns example.word-counter > > (:require > > [samza-config.core :refer [stateful-task key-value-store]])) > > > > (defn count-words [store sentence output] > > (doseq [word (split sentence)] > > (update-in store word inc) > > (output :word-count {:word word > > :count (get store word)}))) > > > > (defjob word-counter > > {:inputs [(topic “words”)] > > :outputs [(topic “word-counts”)] > > :storage (key-value-store :word-counts) > > :task (stateful-task > > (fn [store input output] > > (count-words store input output)))}) > > > > I thought the easiest way to implement this would be for defjob to setup > > some > > metadata that points to implementations of StreamTask/InitableTask and > > have a > > custom ConfigFactory that can return the config for a named job. > > Here is the (in development) implementation of ConfigFactory > > (defrecord JobConfigFactory [] ConfigFactory (getConfig [this uri] (let > > [job (find-job uri)] (println “found job:” job) job))) > > Unfortunately when I run the job runner, passing the name of this class > as > > the > > --config-factory, I get the following stacktrace. I believe this is > because > > Clojure uses a special “DynamicClassLoader” to find these classes. That > is > > what > > I'd set the classloader to if there was the option to do so. > > $ lein do run -m samza-config.job example.jobs.hello-world Running samza > > job: notifications.jobs.send-email Config Factory: samza_config.job. > > JobConfigFactory Class Loader: #object[sun.misc.Launcher$ AppClassLoader > > 0x4aa298b7 sun.misc.Launcher$ AppClassLoader@4aa298b7] Exception in > > thread “main” java.lang. ClassNotFoundException: samza_config.job. > > JobConfigFactory, compiling:(/private/var/ folders/yt/ > > ch58t4q565g3vjvyj5wjfkt00000gp /T/form-init23293643677066296. clj:1:123) > at > > clojure.lang.Compiler.load( Compiler.java:7391) at clojure.lang.Compiler. > > loadFile(Compiler.java:7317) at clojure.main$load_script. > > invokeStatic(main.clj:275) at clojure.main$init_opt. > > invokeStatic(main.clj:277) at clojure.main$init_opt.invoke( main.clj:277) > > at clojure.main$initialize. invokeStatic(main.clj:308) at > > clojure.main$null_opt. invokeStatic(main.clj:342) at > > clojure.main$null_opt.invoke( main.clj:339) at clojure.main$main. > > invokeStatic(main.clj:421) at clojure.main$main.doInvoke( main.clj:384) > at > > clojure.lang.RestFn.invoke( RestFn.java:421) at > > clojure.lang.Var.invoke(Var. java:383) at clojure.lang.AFn. > > applyToHelper(AFn.java:156) at clojure.lang.Var.applyTo(Var. java:700) at > > clojure.main.main(main.java: 37) Caused by: java.lang. > > ClassNotFoundException: samza_config.job. JobConfigFactory at > > java.net.URLClassLoader. findClass(URLClassLoader.java: 381) at > > java.lang.ClassLoader. loadClass(ClassLoader.java: 424) at > > sun.misc.Launcher$ AppClassLoader.loadClass( Launcher.java:331) at > > java.lang.ClassLoader. loadClass(ClassLoader.java: 357) at > > java.lang.Class.forName0( Native Method) at > java.lang.Class.forName(Class. > > java:264) at org.apache.samza.util. CommandLine.loadConfig( > > CommandLine.scala:66) at org.apache.samza.job. JobRunner$.main(JobRunner. > > scala:65) at org.apache.samza.job. JobRunner.main(JobRunner. scala) at > > samza_config.job$_main. invokeStatic(job.clj:102) at > > samza_config.job$_main. doInvoke(job.clj:96) at > clojure.lang.RestFn.invoke( > > RestFn.java:408) at clojure.lang.Var.invoke(Var. java:379) at > > user$eval63.invokeStatic(form- init23293643677066296.clj:1) at > > user$eval63.invoke(form- init23293643677066296.clj:1) at > > clojure.lang.Compiler.eval( Compiler.java:6927) at > > clojure.lang.Compiler.eval( Compiler.java:6917) at > > clojure.lang.Compiler.load( Compiler.java:7379) ... 14 more > > That repo also has similarly defined implementations of serde factories > > that > > serialize/deserialize avro messages and check them against the confluent > > schema > > registry. > > The biggest pain point IMO is the requirement to build a jar to run a job > > during > > development. I think it would be possible to define these classes in Java > > and > > have them call into a Clojure API but that's basically what I'm trying to > > avoid > > as I'm more likely to get the Java wrong. > > Thanks for taking the time to understand this request. > > Cheers, AndyAndy Chambers | Full-Stack Software Developer > > > > andy.chambers@fundingcircle. com | 707-205-6502 (m) > > > > > > > > > > 747 Front St, 4th Fl | San Francisco, CA 94111 > > > > Our Mission: T o build a better financial world > > > > > > > > > > > > > > Unless specifically indicated, this e-mail is not an offer to sell or a > > solicitation of any investment products or other financial product or > > service, > > an official confirmation of any transaction, or an official statement of > > Funding > > Circle USA. This e-mail is meant only for the intended recipient of this > > transmission, and > > contains trade secret and strictly confidential information belonging to > > the > > sender. It is unlawful for unauthorized individuals to review, use, copy, > > disclose, or disseminate confidential information. If you have received > > this > > e-mail in error, please notify the sender immediately by telephone at > > 857.285.1263 or by return email and promptly delete this message from > > your system. > > > > > > On 9 February 2016 at 22:46, Yi Pan < nickpa...@gmail.com > wrote: > > Hi, Andy, > > > > Forgive me for my ignorance on the topic of Clojure. Could you give some > > simple example that Config and Serde Factories are in "Clojure-land" and > > how would a customized ClassLoader help in this case? > > > > Thanks a lot! > > > > -Yi > > > > On Tue, Feb 9, 2016 at 10:36 PM, Andy Chambers < > > andy.chambers@fundingcircle. com > wrote: > > > > > That would help but if I understand it correctly, it would still be > > > necessary to implement things like Config and Serde Factories in Java. > It > > > would be nice to stay in "Clojure-land" to implement those too. > > > > > > *Andy Chambers* | Full-Stack Software Developer > > > > > > * andy.chambers@fundingcircle. com < andy.chambers@fundingcircle. com > > >* | > > > 707-205-6502 (m) > > > > > > > > > 747 Front St, 4th Fl | San Francisco, CA 94111 > > > > > > *Our Mission: **T**o build a better financial world* > > > > > > > > > Unless specifically indicated, this e-mail is not an offer to sell or a > > > solicitation of any investment products or other financial product or > > > service, an official confirmation of any transaction, or an official > > > statement of Funding Circle USA. This e-mail is meant only for the > > > intended recipient of this transmission, and contains trade secret and > > > strictly confidential information belonging to the sender. It is > unlawful > > > for unauthorized individuals to review, use, copy, disclose, or > > disseminate > > > confidential information. If you have received this e-mail in error, > > please > > > notify the sender immediately by telephone at 857.285.1263 or by > return > > > email and promptly delete this message from your system. > > > > > > On 9 February 2016 at 22:25, Yi Pan < nickpa...@gmail.com > wrote: > > > > > > > Hi, Andy, > > > > > > > > I think that you are looking for the feature in SAMZA-697. Or are you > > > > looking for something even more specific? > > > > > > > > On Tue, Feb 9, 2016 at 10:22 PM, Andy Chambers < > > > > andy.chambers@fundingcircle. com > wrote: > > > > > > > > > Hey Folks, > > > > > I'm trying to build some tooling to make writing jobs in Clojure a > > > little > > > > > bit > > > > > more interactive. One feature that I think would help a lot is to > > allow > > > > > specification of a ClassLoader that is able to find classes defined > > > > > dynamically > > > > > in Clojure. > > > > > Would you consider a feature/patch that adds a config parameter for > > > this > > > > > that > > > > > was respected everywhere Class/forName is called? > > > > > Is there anything else I should consider? > > > > > Thanks, > > > > > Andy > > > > > Andy Chambers | Full-Stack Software Developer > > > > > > > > > > andy.chambers@fundingcircle. com | 707-205-6502 (m) > > > > > > > > > > > > > > > > > > > > > > > > > 747 Front St, 4th Fl | San Francisco, CA 94111 > > > > > > > > > > Our Mission: T o build a better financial world > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Unless specifically indicated, this e-mail is not an offer to sell > > or a > > > > > solicitation of any investment products or other financial product > or > > > > > service, > > > > > an official confirmation of any transaction, or an official > statement > > > of > > > > > Funding > > > > > Circle USA. This e-mail is meant only for the intended recipient of > > > this > > > > > transmission, and > > > > > contains trade secret and strictly confidential information > belonging > > > to > > > > > the > > > > > sender. It is unlawful for unauthorized individuals to review, use, > > > copy, > > > > > disclose, or disseminate confidential information. If you have > > received > > > > > this > > > > > e-mail in error, please notify the sender immediately by telephone > at > > > > > 857.285.1263 or by return email and promptly delete this message > > from > > > > > your system. > > > > > > > > > >
