On Tue, Jan 8, 2013 at 12:18 AM, Wolodja Wentland <babi...@gmail.com> wrote:
> On Thu, Nov 01, 2012 at 22:34 +0900, Alan Busby wrote: > > On Thu, Nov 1, 2012 at 8:27 PM, Wolodja Wentland <babi...@gmail.com> > wrote: > > > > Oh, fold-into-map and fold-into-map-with would be wonderful and I > tried to > > > implement the former along the lines of fold-into-vec, but the > performance was > > > abysmal. I am now using fold-into-vec + r/map with zipmap which is > better, but > > > I wouldn't consider that optimal. > > > > The bottleneck with fold-into-map is that merge just doesn't scale up > > well and you end up doing a significant amount of merging. > > Currently a merge operation takes elements one at a time from one map, > > and adds them one at a time to another. > > > > This is incredibly wasteful for large maps as you already have two > > "pre-sorted" tries, and can pair each branch together recursively. If > > each branch node stored the number of child nodes, then you can assign > > different threads to work on different branches as well. This would be > > perfect for reducers, but from a quick look it didn't appear that any > > of the key internals were exposed to be taken advantage of. > > > > Unfortunately I haven't discovered a way to facilitate more efficient > > merging without modifying the actual Clojure core source. I guess this > > is my next step. > > Just stumbled over this old thread again and wondered if you found time to > look into this again? > I've done a good bit of work here but haven't finished anything yet unfortunately. In the meantime I've been working on other reducer utility functions/libraries like, 1. fold-into-lazy-seq ( https://gist.github.com/4479877 ) 2. Iota, for treating large text files as input for a reducer ( https://github.com/thebusby/iota ) Unfortunately some changes in reducers between alpha2 and beta1 have slowed performance down a bit, so I've been waiting for 1.5 to be released before spending too much time on it. I guess all of the above is motivation for me to finally send in my Clojure license stuff... -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
