Hi Ian, I've used fastbit with IP's a fair bit and using primitive data types such as 8,16,32 or 64 bit integers is the way to go, as fastbit can query on them very quickly. The size of the integer should really be determined by what kinds of queries you are trying to do on the data. For example, are you going to be doing a lot of range queries over subnets of a certain size? I would probably first try using 4 32 bit integers and maybe adjust the size depending on your needs. A couple things to consider while you do this:
1.) you don't have to load up all the data columns of the IP address to do searches, you can leverage the indices to do the searches, and then only load up the data columns that you need. This would be handy if you are doing a lot of subnet searches. 2.) Performance for column oriented stores will deteriorate as you load up more columns. So if you are going to load up the entire IP address each time, you probably don't want to divide it into 16 columns, because it will likely be slower. If you only need to load a portion of the IP address with each query, then 16 columns may be a good choice. 3.) You can use partitioning to your advantage. For example, if you are only storing IP address for let's say 3 /64 subnets, then you could encode the subnet in the directory name of the partition and have 3 partitions while storing only the lower 64 bits of each individual IP address. Then your query would be a two step process, find the partition that refers to a subnet, and then query for a particular IP or IPs in that subnet. This can help speed things up a lot, and allow you to store a lot less data! Hope that helps, Teryl On Tue, Feb 16, 2016 at 10:00 PM, Ian Pye <[email protected]> wrote: > Right. Does make application side math a bit more complex. But thanks > for the idea. > > On Tue, Feb 16, 2016 at 6:17 PM, Justin Swanhart <[email protected]> > wrote: > > Hi, > > > > The traditional way would to be to store the high/low 64 bits into two 64 > > bit ints, or to use four 32 bit ints. That will make math more complex > for > > ranges but it is workable, might need some app-side math too. John may > be > > able to provide other solutions. > > > > --Justin > > > > On Tue, Feb 16, 2016 at 9:11 PM, Ian Pye <[email protected]> wrote: > >> > >> All, > >> > >> I'm looking for a way to store IPv6 addresses in fastbit. The catch is > >> that they are 128 bits wide. I could use a blob type, but then nice > >> operations like > and < don't work. I'm wondering if anyone has done > >> this, and if so what techniques worked. > >> > >> Best, > >> > >> Ian > >> _______________________________________________ > >> FastBit-users mailing list > >> [email protected] > >> https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users > > > > > > > > _______________________________________________ > > FastBit-users mailing list > > [email protected] > > https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users > > > _______________________________________________ > FastBit-users mailing list > [email protected] > https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users >
_______________________________________________ FastBit-users mailing list [email protected] https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users
