We have been busy here working with our little dataset library and I wanted to talk about some of the upgrades that I think are important/interesting.
We have first class support of Apache Arrow now which means I took the time to actually understand, byte-by-byte, the binary on-disk format. I also found a memory mapping library that I think is great, larray <https://github.com/xerial/larray>. Arrow files are really sequences of datasets and we have a brand new namespace <https://github.com/techascent/tech.ml.dataset/blob/master/src/tech/ml/dataset/reductions.clj> which will grow over time devoted to really large (multiple GB, out of memory) reductions over sequences of datasets which have competitive performance characteristics with anything out there. Lastly, we have a (great!) blog post exploring memory mapping, apache arrow, and the tech.datatype bindings <https://github.com/techascent/tech.datatype/blob/master/src/tech/v2/datatype/mmap.clj> to larray. It specifically highlights how nice Clojure is for exploring binary file formats — doing this brought back memories of doing similar things in C++ and wow, with Clojure I get the same performance and I can actually see what I am working with! Much appreciation to Alex, Rich, and the team! https://techascent.com/blog/memory-mapping-arrow.html Enjoy! Chris -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/clojure/CADbpEJsz1T%3D2Gu%2BVpxdy02QK8Ap2n1PxkmRy%2BCAJqWKepKwmVQ%40mail.gmail.com.
