Sam Ritchie <sritchi...@gmail.com> writes: > Great stuff!
Thanks! > Just as a note, Cascalog 2.0 has a lower-level DSL that lets you > write Cascading in idiomatic clojure. Here are some test examples: > > https://github.com/nathanmarz/cascalog/blob/develop/cascalog-core/test/cascalog/cascading/operations_test.clj Cool. I did not know about that part of the API, which does look nifty. I’m working on a blog post digging into this some, and I’m hoping to snag one of the lightning talk spots at the Conj, but – I do think there’s a big difference between writing job-flows which use a `map`-like `map*` function and literally calling `map` in a literal plain function[1]. Want a state-bearing sequence-mapping transformation? With Parkour, you can just grab bbloom’s `transduce` library[2] and it works just as well in a remote task as it does in local code, because it does in fact do literally the same thing in both scenarios. You can get similar results in Cascalog/Cascading, but need to first re-express the functionality in terms of Cascalog/Cascading’s abstractions vs just leaning directly on Clojure’s. The algebraic execution planners backing Cascading- and FlumeJava-likes allow powerful optimization of cross-task operations, but do require all transformations to be expressed in terms of primitives the planners understand. Parkour loses the cross-task awareness, but allows MapReduce tasks to do anything which can be expressed as operations on a Clojure reducible collection. This can include repeated partial reductions (even map-side), full task-partition reductions, and arbitrary numbers of disjoint task outputs. It’s not a perfect example of what I’m talking about, but Parkour does include an example implementation of the MapReduce algorithm for transforming a graph into a sparse matrix of absolute-indexed cells: https://github.atl.damballa/rnd/parkour/blob/master/examples/parkour/examples/matrixify.clj I’ll see if I can distill out a more compelling example from some real jobs prior to the Conj :-). [1] It admittedly hurts this point a bit that Parkour exclusively uses reducers instead of lazy sequences, but I’m hoping shortly to add the necessary glue to allow tasks to work via seqs too when desired. [2] https://github.com/brandonbloom/transduce -- Marshall Bockrath-Vandegrift <llas...@damballa.com> Principal Software Engineer, Damballa R&D -- -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
