I've got some code that's using Twitter's HoseBirdClient to pull tweets from the public stream, which I then preprocess and store with CouchDB. Right now, my HBC client is being forced to reconnect more than I'd like, which occasionally causes my app to hang, for reasons I'm not entirely clear on. Regardless, some preliminary research on HBC suggests that the reconnections are being caused by my code failing to keep up with the endpoint, which in turn suggests that my processing+uploading is taking too long. I tried wrapping the processing+uploading part in futures, which definitely sped things up, but caused 409 errors when uploading to CouchDB--briefly, Couch requires any update operation to include a git-style "rev" string, and if the rev you provide isn't the most recent one, it throws a 409 at you. I'm organizing things by hashtag, so tweets with multiple copies of the same hashtag, or series of tweets with the same hashtag are the culprit--future A gets the current doc from Couch, processes it, and uses the rev it got from the currently-existing doc, while future B does the same thing, but finishes first, so now future A has an outdated rev, and that causes the 409.
The vague solution I've come up with involves using a map to store the rev values, with the last step of the processing/uploading function being to store the rev number Clutch helpfully returns to you after a successful update. From what I can tell, refs are the way to go, since each future is effectively a separate thread. My questions are as follows: 1) Would I have to store the map-of-refs in a ref? 2) Is this even feasible? Would the timing work out? 3) With the addition of all this dereferencing and `dosync`+`alter`-ing, would this actually end up speeding things up all that much? -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
