Hi Ying,
I've run the following job twice now, and both times it has ended with
a spare merge file being left behind (and hugecount.output looking
like it might be incomplete). Can you take a look at this and see if
there's a problem either in hugecount.pl or how I'm running it?
Details below, and do let me know if I can provide you with any
additional information.
Thanks!
Ted
---------------------------
Here's the job I've been running (this is a bash shell script I submit)
export INPUT=/home/ted/Corpora/TDP-XIE
export SPLIT=8000000
export OUTPUT=/home/ted/mycount
rm -fr $OUTPUT
mkdir $OUTPUT
huge-count.pl --tokenlist --stop /home/ted/vector/stoplist --split
$SPLIT --uremove 100 --remove 5 --window 4 $OUTPUT $INPUT
------------------------------------
And this is the output that is left behind...
t...@maraca:~/myhugecount$ ls
huge-count.output merge.82
t...@maraca:~/myhugecount$ wc *
5048833 15146497 154092132 huge-count.output
33467287 100401859 1027141307 merge.82
38516120 115548356 1181233439 total
t...@maraca:~/myhugecount$ ls
huge-count.output merge.82
--------------------------------------
My input is just the XIE portion of the GigaWord data...
t...@maraca:~/Corpora/TDP-XIE$ ls
xie199501.txt xie199606.txt xie199711.txt xie199904.txt xie200009.txt
xie199502.txt xie199607.txt xie199712.txt xie199905.txt xie200010.txt
xie199503.txt xie199608.txt xie199801.txt xie199906.txt xie200011.txt
xie199504.txt xie199609.txt xie199802.txt xie199907.txt xie200012.txt
xie199505.txt xie199610.txt xie199803.txt xie199908.txt xie200101.txt
xie199506.txt xie199611.txt xie199804.txt xie199909.txt xie200102.txt
xie199507.txt xie199612.txt xie199805.txt xie199910.txt xie200103.txt
xie199508.txt xie199701.txt xie199806.txt xie199911.txt xie200104.txt
xie199509.txt xie199702.txt xie199807.txt xie199912.txt xie200105.txt
xie199510.txt xie199703.txt xie199808.txt xie200001.txt xie200106.txt
xie199511.txt xie199704.txt xie199809.txt xie200002.txt xie200107.txt
xie199512.txt xie199705.txt xie199810.txt xie200003.txt xie200108.txt
xie199601.txt xie199706.txt xie199811.txt xie200004.txt xie200109.txt
xie199602.txt xie199707.txt xie199812.txt xie200005.txt xie200110.txt
xie199603.txt xie199708.txt xie199901.txt xie200006.txt xie200111.txt
xie199604.txt xie199709.txt xie199902.txt xie200007.txt
xie199605.txt xie199710.txt xie199903.txt xie200008.txt
t...@maraca:~/Corpora/TDP-XIE$ wc *
6202 1184507 7024023 xie199501.txt
6010 1134793 6715677 xie199502.txt
7400 1444049 8550899 xie199503.txt
6524 1255501 7446336 xie199504.txt
7402 1450923 8568172 xie199505.txt
7170 1409804 8332601 xie199506.txt
6927 1303696 7719045 xie199507.txt
7275 1413761 8299012 xie199508.txt
6876 1372918 8128653 xie199509.txt
7286 1405115 8364952 xie199510.txt
7450 1388922 8224930 xie199511.txt
6672 1300072 7715936 xie199512.txt
7188 1334015 7862867 xie199601.txt
6696 1215131 7156749 xie199602.txt
8033 1519423 9013013 xie199603.txt
7788 1473207 8730995 xie199604.txt
8129 1556154 9218247 xie199605.txt
8032 1549764 9194048 xie199606.txt
8514 1640081 9639336 xie199607.txt
8004 1531682 9038373 xie199608.txt
7828 1524511 9058350 xie199609.txt
8060 1549633 9218619 xie199610.txt
7734 1506007 8968417 xie199611.txt
7452 1465136 8689718 xie199612.txt
7794 1499399 8837231 xie199701.txt
6928 1333855 7875190 xie199702.txt
8525 1721433 10204831 xie199703.txt
7840 1546811 9144221 xie199704.txt
8240 1637605 9657260 xie199705.txt
7603 1511550 8911029 xie199706.txt
7658 1501173 8862111 xie199707.txt
7825 1495545 8777642 xie199708.txt
8025 1639203 9736899 xie199709.txt
8334 1661903 9854493 xie199710.txt
8509 1663965 9882142 xie199711.txt
8282 1657365 9812754 xie199712.txt
8053 1550193 9154341 xie199801.txt
7857 1555125 9163722 xie199802.txt
8848 1742820 10342905 xie199803.txt
8148 1581470 9393017 xie199804.txt
8965 1752483 10382755 xie199805.txt
8882 1749295 10302189 xie199806.txt
9051 1788730 10542299 xie199807.txt
8467 1690532 9940580 xie199808.txt
8284 1728637 10189171 xie199809.txt
8539 1739594 10310588 xie199810.txt
8649 1707732 10109928 xie199811.txt
9727 1867443 10940369 xie199812.txt
8203 1603090 9426368 xie199901.txt
7350 1461607 8615055 xie199902.txt
9442 1882454 11126748 xie199903.txt
9039 1788982 10560225 xie199904.txt
8839 1759592 10389989 xie199905.txt
8653 1732687 10220635 xie199906.txt
8698 1742722 10224635 xie199907.txt
8978 1751058 10251998 xie199908.txt
9175 1850108 10899941 xie199909.txt
8854 1733381 10273283 xie199910.txt
8679 1658967 9789022 xie199911.txt
8788 1716177 10116139 xie199912.txt
8516 1606427 9434389 xie200001.txt
8051 1571315 9239155 xie200002.txt
9717 1895946 11166496 xie200003.txt
9196 1830029 10819900 xie200004.txt
9392 1805885 10689714 xie200005.txt
9434 1826577 10834233 xie200006.txt
9100 1790377 10553950 xie200007.txt
9267 1818165 10695151 xie200008.txt
9571 1779519 10427577 xie200009.txt
8864 1796484 10646671 xie200010.txt
8841 1731864 10225318 xie200011.txt
8007 1623146 9549503 xie200012.txt
7880 1480773 8681644 xie200101.txt
8235 1581014 9272288 xie200102.txt
9643 1937289 11393670 xie200103.txt
8859 1748990 10261162 xie200104.txt
8924 1758391 10328510 xie200105.txt
8620 1716646 10131853 xie200106.txt
8581 1709051 10080829 xie200107.txt
9882 1983160 11583688 xie200108.txt
8867 1728337 10179257 xie200109.txt
9437 1793786 10614699 xie200110.txt
1740 339343 1995533 xie200111.txt
679007 132786005 783905863 total
--
Ted Pedersen
http://www.d.umn.edu/~tpederse
