Frank, what do you mean when you write that you hav begun distibuting the files...still on hard disk, or p2p?
-----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of Frank A. Stevenson Sent: Wednesday, June 23, 2010 1:17 PM To: [email protected] Subject: [A51] Some words on Table formats I would like to try and clarify the current situation with the "Berlin A5/1 Rainbow Tables", and the data formats that are being used for exchange. There are a few steps in generating the tables. The first step of generation produces unsorted "multifiles" - all output is coarsely sorted into 256 buckets at 11 bytes pr chain. An unsorted table is ~90GB - and after sorting (using the sort2 program) merges have been removed, and you are left with 66.7GB of multifiles. For the test lookups, I introduced a slighlty more compact format "SSD" @49.5GB - which only uses 8 bytes pr chain with an accompanying 97Mbyte index file. For practical reasons this more compact format was also used in some initial distribution. Originally 2 programs were made; SSDwriter & SSDexploder to convert between SSD and MultiFile formats. But when we started adding more formats, having a single program for each conversion would not be a good idea. More recently an even more compact format had to be created in order to fit more tables in our 2TB limit. This format is a "raw delta" format, and does not currently have an index, meaning it can only be used for transportation. Also with this new format, I made a the conversion tool "TableConvert" - that is able to do any conversion required. But our intentions are to use a format derived from "raw delta" for the actual lookup. Raw delta compresses a chain to 54.085 bits, @41.9Gb allowing us a total of 47 tables on a 2TB disk, but is not the most efficient for lookup, in part because chains can span the disk block boundaries. This means that yet another format (at least) will be added, which may displace the "raw delta". For this reason "raw delta" should be considered a transitional format until we have figured out the best way to create an "indexed delta" format. The SSD format will also most likely go away when we get the new lookup code working. To convert from "raw delta" to SSD for test lookups, the following command can be used: TableConvert ds 100.dlt /dev/blockdevice index.dat "ds" are input / output format codes, where legal codes are: d - raw delta m - multifile s - SSD I have started distribution of tables in the "raw delta" format, and the MD5 checksums for those files are: 15768f7f5af4e5467304c7162688425d ./100-172/140.dlt 55b1bed783fd94a1feb06e70af5cd789 ./100-172/148.dlt cec43159aa1d7ca4982f1c7417b8dd09 ./100-172/156.dlt 9558f5a7152ab8dddc6e5ff819a3fa5f ./100-172/164.dlt 307cc99d1724bd1b460d7f39d6121048 ./100-172/172.dlt f0dbac482c9f6d6626a70bf8115515b4 ./100-172/132.dlt 9db534d5866cc6c8b18eef0bbf6aceae ./100-172/116.dlt d89f2f62a471734fc86844261e709719 ./100-172/108.dlt 6d36bca865ad5f5b3b95e50be5555f0a ./100-172/100.dlt 0c93864ad638b531dedf10d85880ee79 ./100-172/124.dlt 0cbd427fb98dc9cec05bc8cd9eec53c5 ./180-260/180.dlt 8771eba46785c636e606560265efcb6f ./180-260/188.dlt 7f4fcaec2da609dd794ecff07f1ad890 ./180-260/196.dlt 69b40dc68cd9d26006f9b1f70ab676ad ./180-260/204.dlt 032414372c864417a502ef1eaaa42454 ./180-260/220.dlt 6afc1ca6c0005bf2df6782371c56700d ./180-260/230.dlt c6a95cda3e55079295cf97706dc9de2e ./180-260/238.dlt f54c106d3ea9ef442330c5156c14c832 ./180-260/250.dlt 2cf1377b93139fb2a7ec9f687b272d3c ./180-260/260.dlt 9e9861bd2272b58737629b7c34eb1064 ./180-260/212.dlt af8167abdfd178a01b2bb3a2ee4f59c8 ./268-364/268.dlt d422218e33cb6c421108469940c03d41 ./268-364/276.dlt 6995a5445330b78380b3ba16642318e9 ./268-364/292.dlt 1cc82ad35788afc4acab95f481e3de9d ./268-364/340.dlt 183a3427843105942e9b0ab0a852c7db ./268-364/348.dlt 771ec801ac043291b9f134f75f82a31d ./268-364/356.dlt 59913ab26509afaecd8861a6877b06c6 ./268-364/364.dlt 75340dda19dc1b1ada8108d47461562a ./268-364/332.dlt 2ecc836a993a3f71cd710c4455e9780e ./268-364/324.dlt 5bc5814694b412cfb4ba52198492ba1b ./372-500/404.dlt 44f06b97410bda5b029754de9cc87db4 ./372-500/428.dlt 435e2d2755aa5ce9357889bef5ee534f ./372-500/380.dlt 98de5b58c98fafc5ada3897eddf7f31a ./372-500/396.dlt 4518baa2a1156451cbc0952eb216cb5f ./372-500/412.dlt 1bbceb9afda870b76b7e9928409183de ./372-500/372.dlt 3985446a74105d15f6c4b8bd0467771b ./372-500/420.dlt fabeaa1792045a0ab5471aeac7271481 ./372-500/388.dlt a0006b7b58e01b97d6bce1b43b015343 ./372-500/492.dlt fa47f2242230b69c5ae7a1196ea0311f ./372-500/500.dlt c3c48ba123a1ebd9ea6f0abaa56ac475 ./extra/436.dlt -- Frank _______________________________________________ A51 mailing list [email protected] http://lists.lists.reflextor.com/cgi-bin/mailman/listinfo/a51 _______________________________________________ A51 mailing list [email protected] http://lists.lists.reflextor.com/cgi-bin/mailman/listinfo/a51
