I added the above-described features a few weeks back, but only got around to marking 0.1.1 today. Fsync batching is described at the end of the README, let me know if you have any questions.
On Friday, February 7, 2014 11:52:11 AM UTC-8, Zach Tellman wrote: > > Hi Bob, > > Right now the API only allows for single puts, and fsyncing is > all-or-nothing. However, this is just an artifact of my major use case for > the library, which relies on upstream batching of tasks. I'm planning an > 0.1.1 release which has an explicit `sync` method, and support for > sync-intervals (i.e. sync twice a second) and sync-thresholds (i.e. sync > every ten puts or takes). The use case you describe could be achieved by > disabling automatic syncing, and doing a series of puts and takes followed > by a call to `sync`. > > If you have thoughts or suggestions on how this can be more useful for > you, please let me know. > > Zach > > > On Fri, Feb 7, 2014 at 5:26 AM, Bob Hutchison > <hutch...@recursive.ca<javascript:> > > wrote: > >> >> On Feb 6, 2014, at 6:45 PM, Zach Tellman <za...@factual.com <javascript:>> >> wrote: >> >> At Factual we get a lot of data thrown at us, and often don't have >> control over the rate at which it comes in. As such, it's preferable that >> our buffer isn't bounded by the process' memory, since a temporary blip in >> throughput may cause GC pauses, OOM exceptions, and other things that will >> only exacerbate the problem. It's also preferable that if the process >> dies, we won't lose any data which hasn't yet escaped the process. A >> disk-backed queue satisfies both of these requirements. >> >> As such, I'm happy to announce that we're open sourcing 'durable-queue': >> https://github.com/Factual/durable-queue. It's a small, fast, >> pure-Clojure implementation that in our production systems is responsible >> for processing billions of entries daily. We believe it has broad >> applications, and are excited to see how others will use it. >> >> >> What excellent timing! I’ve been looking at ZeroMQ, RabbitMQ, and Kafka >> for the last week or so. ZMQ is awfully attractive for what I’m trying to >> do, but there are a few things it doesn’t do that I need done. I had begun >> thinking of building something similar on top of Redis. >> >> You mention the idea of batching to reduce the impact of fsync. Is there >> an API for batching puts? Is there a way to batch a complete! and put! new >> tasks to the queue? >> >> One pattern that keeps coming up is: >> - take a single task from the queue >> - execute the task, which might generate a set of new tasks to be >> queued on the same queue (and likely on other queues too) >> - signal completion, and put the new tasks >> >> Cheers, >> Bob >> >> >> Zach >> >> P.S. If this sort of work is interesting to you, Factual is hiring: >> https://groups.google.com/forum/#!searchin/clojure/factual/clojure/8bPIEnNpfyQ/lvv-9gkVozAJ >> >> -- >> You received this message because you are subscribed to the Google >> Groups "Clojure" group. >> To post to this group, send email to clo...@googlegroups.com<javascript:> >> Note that posts from new members are moderated - please be patient with >> your first post. >> To unsubscribe from this group, send email to >> clojure+u...@googlegroups.com <javascript:> >> For more options, visit this group at >> http://groups.google.com/group/clojure?hl=en >> --- >> You received this message because you are subscribed to the Google Groups >> "Clojure" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to clojure+u...@googlegroups.com <javascript:>. >> >> For more options, visit https://groups.google.com/groups/opt_out. >> >> >> -- >> You received this message because you are subscribed to the Google >> Groups "Clojure" group. >> To post to this group, send email to clo...@googlegroups.com<javascript:> >> Note that posts from new members are moderated - please be patient with >> your first post. >> To unsubscribe from this group, send email to >> clojure+u...@googlegroups.com <javascript:> >> For more options, visit this group at >> http://groups.google.com/group/clojure?hl=en >> --- >> You received this message because you are subscribed to a topic in the >> Google Groups "Clojure" group. >> To unsubscribe from this topic, visit >> https://groups.google.com/d/topic/clojure/4tZFWdMKvjw/unsubscribe. >> To unsubscribe from this group and all its topics, send an email to >> clojure+u...@googlegroups.com <javascript:>. >> For more options, visit https://groups.google.com/groups/opt_out. >> > > -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
