sd shields: > I am trying to grab a huge amount of data (100 meg?) > with plkr. The site is very straight forward, so it > should import well, when I am getting it from the > internet, the desktop freezes up after getting several > hundred pages.
Try splitting it into several smaller channels. As to why it happens: (1) The desktop uses the python parser. The python parser keeps everything in memory. In theory, the dictionaries could be replaced with shelves (disk), but at the moment - plucking a 100meg site will require at least 100meg of memory (and probably several times that). (2) Even if you solved the memory problem, the plucker format is limited to about 64K records because of PalmOS limitations, and perhaps 16K in practice. Individual records are limited to about 32K before they get split. You may be hitting these limits. (3) Even if you do have enough memory and few enough records, you are probably putting a severe strain on both that network (which can cause errors, which are not always handled gracefully) and on your memory (which can cause garbage collection, which looks like a freeze). -jJ _______________________________________________ plucker-list mailing list [EMAIL PROTECTED] http://lists.rubberchicken.org/mailman/listinfo/plucker-list
