Hi, Rakesh, Sounds like your primary access pattern is to read or set an individual bit. This task is better suited for a hash function, not a compressed bitmap data structure. If you only set a small fraction of the bits to 1, then, a good hash function should able to keep the memory usage quite low.
A compressed bitmap data structure could probably keep the storage requirement smaller, however, the compression is typically based on run-length encoding, which require decoding of at least some part of the compressed block containing the designed bit position. This generally means the compressed bitmap would be slower in reading or setting an individual bit than hash function. Hope this helps. John On 10/12/13 3:16 AM, Rakesh Pandit wrote: > On Sat, Oct 12, 2013 at 1:01 PM, Rakesh Pandit wrote: >> On Sat, Oct 12, 2013 at 12:58 PM, Rakesh Pandit wrote: >>> >>> Hi, >>> >>> I had a question before I can start trying out fastbit for my experiment. >>> I need to do an operation billion times and save 0 or 1 in a bitmap which >>> will take as much as 128 MB of RAM in a handheld machine. >>> >>> The amount of free ram in this handheld machine after loading kernel and >>> other very important application is less and I don't know about entropy of >>> bits or whether they follow any pattern. They can be anything and my aim is >>> to create this bitmap and keep it compressed all the time while creating. >>> >>> Does WAH allows to do that (and fastbit library) ? >>> >>> As an example I ran this experiment with 1073741824 bits (128 MB) of >>> uncompressed bitmap on handheld device with plenty of RAM and it runs fine, >>> but once everything important is loaded then there isn't much left. My >>> target is to reduce the memory usage as much as possible. >>> >> >> One important aspect which I forgot to mention is that while creating bitmap >> I want to set bits randomly and they wouldn't be increasing order. They can >> be anyplace depending upon result of operation e.g. sequence could be >> 3000th, 1999th, 888888th, 1st etc >> > > Apologies for posting in HTML earlier. Hopefully this will now reach more > folks. > > Regards, > _______________________________________________ > FastBit-users mailing list > [email protected] > https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users > _______________________________________________ FastBit-users mailing list [email protected] https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users
