Re: Streaming a large file onto a channel

2015-03-18 Thread Erick Pintor
Hi Adrian, What is exactly the issue that you're facing? I did my own version and it seems to be working fine. Please, take a look and I hope it helps. (defn process-file [ch file] (async/thread (with-open [input (io/reader file)] (doseq [line (line-seq input)] (async/!! ch

Re: Streaming a large file onto a channel

2015-03-18 Thread Adrian Mowat
Hi Erick Thanks for getting back to me. On my system, I wasn't seeing the contents of my file being listed in the REPL. Your code is working fine though and I can't see anything significantly different so I wonder if I had managed to corrupt my session in some way. Anyway, it's good to know

Re: Streaming a large file onto a channel

2015-03-18 Thread Adam Clements
It's possible you are simply not seeing the println output from a background thread, depending on how your repl etc is set up. On Wed, 18 Mar 2015 3:19 pm Adrian Mowat adrian.mo...@gmail.com wrote: Hi Erick Thanks for getting back to me. On my system, I wasn't seeing the contents of my file

Re: Streaming a large file onto a channel

2015-03-18 Thread Adrian Mowat
Hi Adam I'm using the latest version on cider + cider-nrepl but it's a possibility. I suspect it's more of a case that I tried so many different combinations I polluted my repl beyond repair. My fault for not just using components from the outset :-( Thanks Adrian Sent from my iPhone

Re: Streaming a large file onto a channel

2015-03-18 Thread Christopher Small
It seems like you're generally on the right track here (though Erick Pintor's code has some nice cleanup, like removal of necessary do, etc). The one thing I'd recommend is testing what happens with a larger channel buffer; if the file io isn't the bottleneck, but rather the processing, this

Streaming a large file onto a channel

2015-03-17 Thread Adrian Mowat
Hi, I've played around with core.async a bit but now I'm trying to use it for a real project and I'm running into a problem getting data off a file and into a channel on the JVM (i.e. as opposed to ClojureScript) I have around 1GB of data sitting in a file. Each line of the file contains a