yes. the tables must be sorted before lookups can be done and should be sorted before they are distributed, because the receiver does not have to do it then. this leaves the possibility open to distribute incomplete tables at the expense of a sorting run at every receiver. since the table compression is not ready for production use, you either do not compress or do the sorting only to find out how many merges you have (and do it again when the table is finished). here is a quick summary how to sort with the current codebase:
sorting can be done with the stxxl work consumer: $ a51table --work file:prefix=data --consume stxxl:ram=3072m:complete=X:merge=Y:incomplete=Z copy where X, Y, Z is a work consumer like: complete=file::prefix=sorted_complete incomplete=null (note the doubling of the colon, this is necessary since it is a nested option). merged chains are in both the complete tables (only the first chain), and the merge output table (all but the first chain of the merge) sorting several hundred million chains can take hours and the output tables are only filled with data after most of the work has been done. the last thing to make sure is you have enough free space in /var/tmp for stxxl temporary files plus free space for the output files. stxxl temp files require about double the space of your source table files and output tables will be the same size as your source files. if you do not have enough space in /var/tmp you can create an stxxl config file as explained at http://stxxl.sf.net. i suggest you do an $ strace a51table --consume stxxl:... copy to find out where stxxl looks for the configuration file. (should be config.stxxl or .stxxl both in the current directory) > do we need to Sort the tables after generation is finished ? > > > <hr> > _______________________________________________ > A51 mailing list > A51@lists.reflextor.com > http://lists.lists.reflextor.com/cgi-bin/mailman/listinfo/a51 > > ______________________________________________________ GRATIS für alle WEB.DE-Nutzer: Die maxdome Movie-FLAT! Jetzt freischalten unter http://movieflat.web.de _______________________________________________ A51 mailing list A51@lists.reflextor.com http://lists.lists.reflextor.com/cgi-bin/mailman/listinfo/a51
