At Factual we get a lot of data thrown at us, and often don't have control over the rate at which it comes in. As such, it's preferable that our buffer isn't bounded by the process' memory, since a temporary blip in throughput may cause GC pauses, OOM exceptions, and other things that will only exacerbate the problem. It's also preferable that if the process dies, we won't lose any data which hasn't yet escaped the process. A disk-backed queue satisfies both of these requirements.
As such, I'm happy to announce that we're open sourcing 'durable-queue': https://github.com/Factual/durable-queue. It's a small, fast, pure-Clojure implementation that in our production systems is responsible for processing billions of entries daily. We believe it has broad applications, and are excited to see how others will use it. Zach P.S. If this sort of work is interesting to you, Factual is hiring: https://groups.google.com/forum/#!searchin/clojure/factual/clojure/8bPIEnNpfyQ/lvv-9gkVozAJ -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
