loop problems
I'm using Twitter's HBC library to read from Twitter's public stream. HBC stores results in a LinkedBlockingQueue, from which you can then `.take` tweets and do stuff to them (in my case, doing some processing/normalization, then storing them in CouchDB). I've been struggling with how exactly best to do this, though. I tried `doseq`, but it stops when the queue is empty, which isn't what I want. Since my code is basically entirely IO, `map` and other lazy stuff causes me problems. Next, I reached for `loop`: (defn process-stream-nores ([in-queue] (process-stream-nores in-queue identity)) ([in-queue process-fn] (loop [res (.take in-queue)] (process-fn (parse-string res true)) (recur (.take in-queue) (where `in-queue` is a LinkedBlockingQueue). Unfortunately, this just hangs, even when the LinkedBlockingQueue isn't empty (I'd expect it to hang when the queue is empty, since the queue blocks until it gets something). I've also tried ... (while true (process-fn (parse-string (.take in-queue) true)) ... but that 1) also hangs, and 2) seems profoundly un-idiomatic. I want to continuously take from the queue, process the tweet, and then put it in CouchDB. (I'm planning on putting this in a Thread that I can stop when I have enough tweets.) I feel like loop is the right way to go, but I don't understand it very well, and can't get it to work. Any help would be greatly appreciated. -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: loop problems
Your loop seems to work as expected: user= (def in-queue (java.util.concurrent.LinkedBlockingQueue. (range 1 11))) #'user/in-queue user= (future (loop [res (.take in-queue)] (prn res) (recur (.take in-queue #core$future_call$reify__6320@2b106ce4: :pending1 2 3 4 5 6 7 8 9 10 user= (.put in-queue 11) nil11 user= What happens when you just (.take in-queue)? -- Henrik On Wed, Nov 12, 2014 at 5:43 PM, Sam Raker sam.ra...@gmail.com wrote: I'm using Twitter's HBC library to read from Twitter's public stream. HBC stores results in a LinkedBlockingQueue, from which you can then `.take` tweets and do stuff to them (in my case, doing some processing/normalization, then storing them in CouchDB). I've been struggling with how exactly best to do this, though. I tried `doseq`, but it stops when the queue is empty, which isn't what I want. Since my code is basically entirely IO, `map` and other lazy stuff causes me problems. Next, I reached for `loop`: (defn process-stream-nores ([in-queue] (process-stream-nores in-queue identity)) ([in-queue process-fn] (loop [res (.take in-queue)] (process-fn (parse-string res true)) (recur (.take in-queue) (where `in-queue` is a LinkedBlockingQueue). Unfortunately, this just hangs, even when the LinkedBlockingQueue isn't empty (I'd expect it to hang when the queue is empty, since the queue blocks until it gets something). I've also tried ... (while true (process-fn (parse-string (.take in-queue) true)) ... but that 1) also hangs, and 2) seems profoundly un-idiomatic. I want to continuously take from the queue, process the tweet, and then put it in CouchDB. (I'm planning on putting this in a Thread that I can stop when I have enough tweets.) I feel like loop is the right way to go, but I don't understand it very well, and can't get it to work. Any help would be greatly appreciated. -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: loop problems
Your loop pattern should work. (I've used this pattern before.) Just a sanity check: you *are* running this function in a different thread, right? Because whatever thread calls this function *will* block forever, whether the queue is empty or not. And unless you provide some side-effecting process-fn there will be no evidence that the thread is doing anything. (I.e. your first function will hang forever and appear to do nothing.) On Wednesday, November 12, 2014 10:43:57 AM UTC-6, Sam Raker wrote: I'm using Twitter's HBC library to read from Twitter's public stream. HBC stores results in a LinkedBlockingQueue, from which you can then `.take` tweets and do stuff to them (in my case, doing some processing/normalization, then storing them in CouchDB). I've been struggling with how exactly best to do this, though. I tried `doseq`, but it stops when the queue is empty, which isn't what I want. Since my code is basically entirely IO, `map` and other lazy stuff causes me problems. Next, I reached for `loop`: (defn process-stream-nores ([in-queue] (process-stream-nores in-queue identity)) ([in-queue process-fn] (loop [res (.take in-queue)] (process-fn (parse-string res true)) (recur (.take in-queue) (where `in-queue` is a LinkedBlockingQueue). Unfortunately, this just hangs, even when the LinkedBlockingQueue isn't empty (I'd expect it to hang when the queue is empty, since the queue blocks until it gets something). I've also tried ... (while true (process-fn (parse-string (.take in-queue) true)) ... but that 1) also hangs, and 2) seems profoundly un-idiomatic. I want to continuously take from the queue, process the tweet, and then put it in CouchDB. (I'm planning on putting this in a Thread that I can stop when I have enough tweets.) I feel like loop is the right way to go, but I don't understand it very well, and can't get it to work. Any help would be greatly appreciated. -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: loop problems
The plan is to run it in a thread, yeah. The process-fn I'm planning on running it with does some stuff to the tweet and then uploads the results to a local couchdb instance. I've been periodically checking /var/log/couchdb/couch.log to verify it's actually doing stuff. I *think* this could benefit from some core.async magic, but I don't understand that lib yet, and haven't had the time to watch Rich's talk on it. Someday... On Wednesday, November 12, 2014 5:34:14 PM UTC-5, Francis Avila wrote: Your loop pattern should work. (I've used this pattern before.) Just a sanity check: you *are* running this function in a different thread, right? Because whatever thread calls this function *will* block forever, whether the queue is empty or not. And unless you provide some side-effecting process-fn there will be no evidence that the thread is doing anything. (I.e. your first function will hang forever and appear to do nothing.) On Wednesday, November 12, 2014 10:43:57 AM UTC-6, Sam Raker wrote: I'm using Twitter's HBC library to read from Twitter's public stream. HBC stores results in a LinkedBlockingQueue, from which you can then `.take` tweets and do stuff to them (in my case, doing some processing/normalization, then storing them in CouchDB). I've been struggling with how exactly best to do this, though. I tried `doseq`, but it stops when the queue is empty, which isn't what I want. Since my code is basically entirely IO, `map` and other lazy stuff causes me problems. Next, I reached for `loop`: (defn process-stream-nores ([in-queue] (process-stream-nores in-queue identity)) ([in-queue process-fn] (loop [res (.take in-queue)] (process-fn (parse-string res true)) (recur (.take in-queue) (where `in-queue` is a LinkedBlockingQueue). Unfortunately, this just hangs, even when the LinkedBlockingQueue isn't empty (I'd expect it to hang when the queue is empty, since the queue blocks until it gets something). I've also tried ... (while true (process-fn (parse-string (.take in-queue) true)) ... but that 1) also hangs, and 2) seems profoundly un-idiomatic. I want to continuously take from the queue, process the tweet, and then put it in CouchDB. (I'm planning on putting this in a Thread that I can stop when I have enough tweets.) I feel like loop is the right way to go, but I don't understand it very well, and can't get it to work. Any help would be greatly appreciated. -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
