Amruta Purandare has been doing some experiments with the Ngram Statistics Package and the English GigaWord corpus lately. As a by-product of this work she found herself with files that contain bigram counts for the 4 individual corpora that make up the GigaWord corpus, as well as counts for the combined corpus. (The total number of words in the GigaWord corpus is well over 1 billion.)
Since it took her a little while to create these files (about 3 days for all the data on a 3GhZ Pentium) we thought it might be useful to make this data available for download. To be clear, we aren't providing the GigaWord corpus here, just the counts of bigrams that occur in the corpus. You can find the bigram counts of the English GigaWord data at: http://www.d.umn.edu/~tpederse/nsp.html Below you'll find the README for this data. Let us know if you have any questions or comments. Enjoy, Ted and Amruta ========================================================================= ========================================================================= README file for bigram counts of English GigaWord corpus =============================== This package contains the counts of bigrams that occur more than 1 time in the English GigaWord Corpus. These counts were found using the huge-count.pl utility program that is a part of the Ngram Statistics Package (version 0.69). The English GigaWord corpus is distributed by LDC, and consists of 4 sources of news text, which are identified as nyt, xie, apw, and afe. - Agence France Press English Service (afe) - Associated Press Worldstream English Service (apw) - The New York Times Newswire Service (nyt) - The Xinhua News Agency English Service (xie) The sum total of these files is over 1 billion words. You can find out more about GigaWord at: http://wave.ldc.upenn.edu/Catalog/CatalogEntry.jsp?catalogId=LDC2003T05 Four input files were created (one for each of the corpora above, plus the concatenation of these four files which we call "giga"). This data was created as follows: 1) Text between <p> and </p> tags was extracted for each corpora and saved into a file named as apw, afe, xie, or nyt. 2) The extracted text was converted to lower case. 3) The separate files (apw, afe, xie, nyt) were concatenated into a single file (giga). 4) Bigram counts are obtained by huge-count.pl as follows: Each of the five files (apw, afe, xie, nyt, giga) were processed by huge-count.pl as follows: huge-count.pl --newLine --stop stoplist.new --nontoken nontoken.regex --token token.regex --remove 2 --split 500 $input-count $input The file name is the value of $input. The value of --split was 500 for giga, and 100 for apw, afe, xie, and nyt. The stoplist, token, and nontoken files used are provided in this distribution. The output of huge-count are the bigram counts found in the files : apw-count, nyt-count, afe-count, xie-count, and giga-count. 5) Note that a user could create co-occurrence counts from this data using the NSP utility program combig.pl as follows: combig.pl $input-count > $input-cocs Bigrams are two word sequences that occur in a given order, while co-occurrences are two word sequences that occur next to each other without regard to any ordering. -- Ted Pedersen http://www.d.umn.edu/~tpederse ------------------------ Yahoo! Groups Sponsor --------------------~--> Make a clean sweep of pop-up ads. Yahoo! Companion Toolbar. Now with Pop-Up Blocker. Get it for free! http://us.click.yahoo.com/L5YrjA/eSIIAA/yQLSAA/x3XolB/TM --------------------------------------------------------------------~-> Yahoo! Groups Links <*> To visit your group on the web, go to: http://groups.yahoo.com/group/nlpatumd/ <*> To unsubscribe from this group, send an email to: [EMAIL PROTECTED] <*> Your use of Yahoo! Groups is subject to: http://docs.yahoo.com/info/terms/
