If I'm not mistaken, PigPen allows you to execute queries locally, without a hadoop cluster etc. Does that solve your problem?
https://github.com/Netflix/PigPen/wiki/Design-and-Features#testing-local-execution-and-debugging On Mon, May 26, 2014 at 3:26 AM, Hesen Peng <hesen.p...@gmail.com> wrote: > Hi everybody, > > I have been doing some random machine learning doodling and have always > been haunted with this problem. > > When I just wanna set out and try if my methodology would work, I would > program in Clojure aiming at a small dataset available within one single > host memory (or even just doodle R or Python. I know what's blasphemy). > > When I wanna implement the results, I would usually have to re-write the > whole process in Cascalog just in order to apply them in Hadoop data. I > know I can still just start up to write Hadoop functions. But that's a bit > overshoot since I have to run simulation study to verify the performance, > correctness etc before even going into production development. Besides, > having hadoop in memory to sit between my program and the computer just > slows down computation. > > I've day-dreamed a lot some smart ways to do prototyping and, with the > flip of a switch, my functions just become easily applicable to Hadoop > data. Wondering if there's any ways to do that? > > I actually asked around and some one who I admire a lot suggested writing > a Hadoop emulator-like wrapper for my input data. Wondering if there's an > easier way? I would appreciate any input. Thanks. > > Hesen > > -- > You received this message because you are subscribed to the Google > Groups "Clojure" group. > To post to this group, send email to clojure@googlegroups.com > Note that posts from new members are moderated - please be patient with > your first post. > To unsubscribe from this group, send email to > clojure+unsubscr...@googlegroups.com > For more options, visit this group at > http://groups.google.com/group/clojure?hl=en > --- > You received this message because you are subscribed to the Google Groups > "Clojure" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to clojure+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -- Kind Regards, Atamert Ölçgen -+- --+ +++ www.muhuk.com -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
