Thanks for the response. 1. I don't think NRRD is as substantially used as you might think. I've > worked in imaging science for years on the data processing/file format end, > and I've never seen anyone use it, and I've never even heard of it. (Pity, > because it looks nice enough. :-\)
I think I can make a solid argument that the install-base of software supporting NRRD is on the order of 30-50k -- probably larger than anything but NIfTI (which is substantial, to me, as far as obscure, non-DICOM medical imaging formats go...) But that's neither here nor there. I'm not going to change your mind, but perhaps someone else will see this thread and think twice before creating `RawArraysWithSlightlyMoreMetadata.jl`. > JuliaIO/Images.jl relies on having ImageMagick installed, whereas > RawArray.jl is a pure Julia solution without any dependencies. > Indeed, that was a major pain point, especially for Windows users. However, Images.jl has been modularized, and no longer requires ImageMagick. NRRD.jl does have more dependencies (~10) than might be expected, in order to support Images inter-op, but from what I can tell they are all pure-Julia except for Rmath (the binary portions of which are distributed as part of base Julia). I'll take my other, minor, comments off-list since this curmudgeonly pet peeve probably isn't of much general interest! On Monday, September 26, 2016 at 8:59:15 AM UTC-5, David Smith wrote: >> >> Hi, Isaiah. This is a valid question. >> >> 0. As a preface, I'd like to say I'm not trying to replace anything. I >> wrote RawArray to solve a problem we have in magnetic resonance imaging >> (quickly saving and loading large complex float arrays), and then I decided >> to share it so if other people like it and find it useful, then cool beans. >> >> Now for the mild stumping... >> >> 1. I don't think NRRD is as substantially used as you might think. I've >> worked in imaging science for years on the data processing/file format end, >> and I've never seen anyone use it, and I've never even heard of it. (Pity, >> because it looks nice enough. :-\) >> >> 2. RawArray is simpler to handle and trivial to understand. I believe all >> you need from an I/O library is I/O.* I don't want my file I/O library >> performing transformations on my data. >> >> I also don't need it to read image formats. Part of the reason behind >> RawArray is to avoid standard image formats because they are not optimized >> for large complex-float arrays. I just want to save multi-GB data arrays to >> disk quickly and read them back quickly on a different machine, five years >> later. >> >> I have other implementations (https://github.com/davidssmith/ra), and >> all are super short and platform agnostic. >> >> 3. RawArray is surely faster. All it does is read. It doesn't perform any >> transformations or encoding, so it can't possibly be slower than NRRD. >> There is a C library at (https://github.com/davidssmith/ra) if you think >> a pure Julia implementation isn't fast enough. >> >> Cheers, >> Dave >> >> [*] That said, I'm not completely ruling out having transformations >> available in RawArray between the RAM and disk. For example, when I first >> wrote it, I had included Blosc compression as an option, signaled by a flag >> in the header. But in general most transformations are best made in RAM >> after reading or on disk with already existing, battle-proven tools, such >> as gzip, uunencode, tar, etc. >> >> >> On Sunday, September 25, 2016 at 9:59:45 PM UTC-5, Isaiah wrote: >>> >>> Is there a reason to use this file format over NRRD [1]? To borrow a >>> wise phrasing: I wonder if the world needs another lightweight raw data >>> format ;) >>> >>> For what it's worth, NRRD is already supported by JuliaIO/Images.jl, and >>> I believe addresses the use-cases identified in your readme, but with a >>> number of technical and non-technical advantages (not least: a number of >>> independent implementations, and a substantial user base, at least as far >>> as these things go). >>> >>> I say this -- very selfishly I admit -- as someone who has been on the >>> receiving end of far too many files in home-brewed formats. >>> >>> [1] http://teem.sourceforge.net/nrrd/descformat.html >>> >>> On Sunday, September 25, 2016, David Smith <david...@gmail.com> wrote: >>> >>>> Hi, all: >>>> >>>> I finally pushed this out, and it might satisfy some of your needs for >>>> a simple way to store N-d arrays to disk. Hope you enjoy it. >>>> >>>> RawArray (.ra) is a simple file format for storing n-dimensional >>>> arrays. RawArray was designed to be portable, fast, storage efficient, and >>>> future proof. Basically it writes the binary array data directly to disk >>>> with a short header that is used to recreate type and dimension >>>> information. >>>> >>>> RawArray is faster than HDF5 and supports complex numbers out of the >>>> box, which HDF5 does not. RawArray supports all basic `Int`, `UInt`, >>>> `Float`, and `Complex{}` types, and more can be easily added in the future, >>>> such as Rational or Big*. It can also handle derived types, but the >>>> serialization of them is currently left up to the user. >>>> >>>> A system of version numbers and flags are implemented to future-proof >>>> the data files as well, in case the implementation needs to change for some >>>> reason. >>>> >>>> You can grab it with `Pkg.add("RawArray")`. A minimum of Julia 0.4 is >>>> required. >>>> >>>> Repository: https://github.com/davidssmith/RawArray.jl >>>> >>>> Cheers, >>>> Dave >>>> >>>
