Chris, Thanks this was in fact it. I had read that sets had a near O[1] lookup, but apparently I was not achieving this properly with (some). Thank you the execution time is about 25x faster now!
Jarrod On Tuesday, January 21, 2014 8:11:09 AM UTC-5, Chris Perkins wrote: > > On Monday, January 20, 2014 11:55:00 PM UTC-7, Jarrod Swart wrote: >> >> I'm processing a large csv with Clojure, honestly not even that big (~18k >> rows, 11mb). I have a list of exported data from a client and I am >> de-duplicating URLs within the list. My final output is a series of >> vectors: [url url-hash]. >> >> The odd thing is how slow it seems to be going. I have tried >> implementing this as a reduce, and finally I thought to speed things up I >> might try a "with-open and a loop-recur". It doesn't seem to have done >> much in my case. I know I am doing something wrong I'm just not sure what >> yet. The best I can do is about 4 seconds, which may only seem slow >> because I implemented it in python first and it takes a half second to >> finish. Still this is one of the smaller files I will likely deal with so >> I'm worried that as the files grow it may get too slow. >> >> The code is here on ref-heap for easy viewing: >> https://www.refheap.com/26098 >> >> Any advice is appreciated. >> > > This part: (some #{hashed} already-seen) is doing a linear lookup in > `already-seen`. Try (contains? already-seen hashed) instead. > > - Chris > > -- -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
