Hi Rakesh, As John said, it will be costly to randomly set bit in compressed bitmap. You may need postgresql bit string data type. It have set_bit and get_bit function, to set or get in a bit position.
You can save memory usage, by packing "fix length" of bit in a row (eg. 8 bit). Not saving all bits in a row. Another alternative, you can use fix-length data structure, for example: BDB's recno or TC's fixed-length database. On 10/12/13, K. John Wu <[email protected]> wrote: > Hi, Rakesh, > > Sounds like your primary access pattern is to read or set an > individual bit. This task is better suited for a hash function, not a > compressed bitmap data structure. If you only set a small fraction of > the bits to 1, then, a good hash function should able to keep the > memory usage quite low. > > A compressed bitmap data structure could probably keep the storage > requirement smaller, however, the compression is typically based on > run-length encoding, which require decoding of at least some part of > the compressed block containing the designed bit position. This > generally means the compressed bitmap would be slower in reading or > setting an individual bit than hash function. > > Hope this helps. > > John > > > On 10/12/13 3:16 AM, Rakesh Pandit wrote: >> On Sat, Oct 12, 2013 at 1:01 PM, Rakesh Pandit wrote: >>> On Sat, Oct 12, 2013 at 12:58 PM, Rakesh Pandit wrote: >>>> >>>> Hi, >>>> >>>> I had a question before I can start trying out fastbit for my >>>> experiment. >>>> I need to do an operation billion times and save 0 or 1 in a bitmap >>>> which >>>> will take as much as 128 MB of RAM in a handheld machine. >>>> >>>> The amount of free ram in this handheld machine after loading kernel >>>> and >>>> other very important application is less and I don't know about entropy >>>> of >>>> bits or whether they follow any pattern. They can be anything and my aim >>>> is >>>> to create this bitmap and keep it compressed all the time while >>>> creating. >>>> >>>> Does WAH allows to do that (and fastbit library) ? >>>> >>>> As an example I ran this experiment with 1073741824 bits (128 MB) of >>>> uncompressed bitmap on handheld device with plenty of RAM and it runs >>>> fine, >>>> but once everything important is loaded then there isn't much left. My >>>> target is to reduce the memory usage as much as possible. >>>> >>> >>> One important aspect which I forgot to mention is that while creating >>> bitmap >>> I want to set bits randomly and they wouldn't be increasing order. They >>> can >>> be anyplace depending upon result of operation e.g. sequence could be >>> 3000th, 1999th, 888888th, 1st etc >>> >> >> Apologies for posting in HTML earlier. Hopefully this will now reach more >> folks. >> >> Regards, >> _______________________________________________ >> FastBit-users mailing list >> [email protected] >> https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users >> > _______________________________________________ > FastBit-users mailing list > [email protected] > https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users > _______________________________________________ FastBit-users mailing list [email protected] https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users
