It turns out Vector{Bool} does not work well with JLD. So I played around
with BitArray, and I figure out that it would be pretty easy to use for my
purposes. It seems to me that BitArray could be made a bit more useful by
exporting a reinterpret method. That would certainly make my use case use
less code, but it could also replace the current implementation of bits. I
think it makes more sense for bits to return a BitArray than a String
anyway, since it would be much faster for uses like bits(4)[1]. Would it
be worth a making a pull request adding something like this in base?
(Clearly redefining bits would change behavior and break things, so I'm not
sure how to approach that)
I wrote up some simple example code and it works fine, however it isn't
actually any faster than the current bits implementation which I found
surprising. Maybe it would be a bit faster if BitArray were able to be
constructed from v directly, instead of allocating then immediately
replacing b.chunks.
function reinterpret2(::Type{BitArray}, v::Vector{Uint64}, dims=(64,-1))
dims[2] == -1 && (dims=(64,length(v)))
# check to make sure the dims are appopriate for length of v
b = BitArray(dims...)
b.chunks = v
b
end
function reinterpret2(::Type{BitArray}, i::Uint64, dims=64)
assert(dims <= 64)
b = BitArray(dims)
b.chunks = [i]
b
end
bits2(i::Uint64) = reinterpret2(BitArray, i)
bits2(x) = reinterpret2(BitArray,reinterpret(Uint64, x))
testbits(n) = [bits(i)[1] for i=1:n]
testbits2(n) = Bool[bits2(i)[1] for i=1:n]
testbits(1);
@time testbits(100000);
testbits2(1);
@time testbits2(100000);
On Tuesday, August 5, 2014 11:26:39 PM UTC-6, Simon Kornblith wrote:
>
> Assuming you have enough memory to write a BitArray to the JLD file
> initially, if you later open the JLD file with mmaparrays=true and read
> it, JLD will mmap the underlying Vector{Uint64} so that pieces are read
> from the disk as they are accessed. (The actual specifics of how this works
> is up to the OS, but generally it works well.) In principle you can also
> modify the BitArray the changes will be saved to the disk, although I'm not
> sure how well that works since I don't do it in my own code. There is no
> easy way to resize the BitArray if you do this, though.
>
> Simon
>
> On Tuesday, August 5, 2014 5:06:16 PM UTC-4, Tim Holy wrote:
>>
>> To me it sounds like you've come up with the main options: BitArray or
>> Array{Bool}. Since a BitArray is, underneath, a Vector{Uint64} with
>> different
>> indexing semantics, it seems you could probably come up with a reasonable
>> way
>> to update just part of it. But even if you use Array{Bool}, you're "only"
>> talking a few hundred megabytes, which is not a catastrophically large.
>> Also
>> consider keeping everything in memory; with 100GB of RAM you could store
>> an
>> awful lot of selections.
>>
>> --Tim
>>
>> On Tuesday, August 05, 2014 12:01:58 PM ggggg wrote:
>> > Hello,
>> >
>> > I have an application where I have a few hundred million events, and
>> I'd
>> > like to make and work with different selections of sets of those
>> events.
>> > The events each have various values associated with them, say for
>> > simplicity color, timestamp, and loudness. Say one selection includes
>> all
>> > the events within 5 minutes after a blue event. Or I want to select
>> all
>> > events that aren't above some loudness threshold. I'd like to be able
>> to
>> > save these selections in a JLD file for later use on some or all
>> events. I
>> > also need to be able update the selections as I observe more events.
>> >
>> > My baseline plane it to have an integer associated with each event and
>> each
>> > bit in the integer i corresponds to a selection. So bit 1 is true for
>> > events within 5 minutes and bit 2 is true for events above the loudness
>> > threshold. Then for each event's integer I can do bits(i)[1] or
>> bits(i)[2]
>> > to figure out if it is included in each selection. That seems like it
>> would
>> > be inefficient since bits() returns a string, so I would have to call
>> > bool(bits(i)[1]). I could use a bitwise mask of some sort like 1&i==0
>> for
>> > the first bit and 2&i==0 for the second bit.
>> >
>> > A BitArray seems like a decent choice, except that you can only
>> read/write
>> > the entire array from a JLD file, rather than just a part of it. That
>> will
>> > be inefficient since I'll often want to look at only a small subset of
>> the
>> > total events. And every time I want to update for new events, I would
>> need
>> > to read the entire BitArray, extend it in memory, then delete the old
>> JLD
>> > object and replace it with a new JLD object. It seems plausible I
>> could
>> > figure out how to read/write part of a BitArray from a JLD as I've
>> already
>> > done some hacking on HDF5.jl, but that could be a large amount of work.
>> >
>> > An Array{Bool} works well with JLD, and seems just as well suited as a
>> > BitArray. I think it's 8 times bigger than BitArray, and has a similar
>> > space ratio to an integer (depending on how many selections I actually
>> use)
>> > because bools are stored as 1 byte? I can probably live with that,
>> although
>> > again it seems sort of inefficient.
>> >
>> > Any advice on how I should go about deciding, or options I hadn't
>> > considered? Also why does bits() return a string, instead of say
>> > Vector{Bool} or BitArray?
>>
>>
