I’m pleased to announce the first public release of Parkour, a library
for writing Hadoop MapReduce applications in idiomatic Clojure. Parkour
takes your Clojure code’s functional gymnastics and sends it
free-running across the urban environment of your Hadoop cluster.
https://github.com/damballa/parkour/
Parkour aims to provide deep Clojure integration for Hadoop. Programs
using Parkour are normal Clojure programs, using standard Clojure
functions instead of new framework abstractions. Programs using Parkour
are also full Hadoop programs, with complete access to absolutely
everything possible in raw Java Hadoop MapReduce. If you know Clojure,
and you know Hadoop, then you’re most of the way to knowing Parkour.
Here is the core of the obligatory “word count” MapReduce program,
written using Parkour:
(defn mapper
[conf]
(fn [context input]
(->> (mr/vals input)
(r/mapcat #(str/split % #"\s+"))
(r/map #(-> [% 1])))))
(defn reducer
[conf]
(fn [context input]
(->> (mr/keyvalgroups input)
(r/map (fn [[word counts]]
[word (r/reduce + 0 counts)])))))
(defn word-count
[dseq dsink]
(-> (pg/input dseq)
(pg/map #'mapper)
(pg/partition [Text LongWritable])
(pg/combine #'reducer)
(pg/reduce #'reducer)
(pg/output dsink)))
Parkour includes detailed documentation, ranging from a quickstart
introduction through detailed discussions of several specific aspects:
https://github.com/damballa/parkour/#documentation
Although this is the first public release of Parkour, the Damballa R&D
team has been using it extensively since beginning serious development
earlier this year. We do also use and will continue to use Cascalog,
but we’ve found that Parkour’s simpler model and more direct Hadoop
integration is a better fit for many problems.
I am personally incredibly excited about this release. I will be at
this year’s Clojure/conj, and will be more than happy to discuss Parkour
in detail with those interested.
Questions and pull requests welcome!
--
Marshall Bockrath-Vandegrift <llas...@damballa.com>
Principal Software Engineer, Damballa R&D
--
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
