Hi, If I want to just crawl webpages and store the least amount of data required to generate the next fetchlist, what must the segment folder contain? Does it have to contain parse_data/ and parse_text/? How about crawl_parse/? What is the minimal data I will need?
Thanks. Michael
