Hi CC
Thanks for your response. I've checked everything you have mentioned and it all looks fine. I dont think its any problem with the html-parse. Cos i've been able to successfully crawl so many websites like www.discovery.com etc. Its only few websites that give this problem. Two that i have noticed so far are: http://java.sun.com and http://www.nih.gov.
I'm enclosing the log below: Thanks
iln
050113 143847 fetching http://www.nih.gov/
050113 143847 http.proxy.host = null
050113 143847 http.proxy.port = 8080
050113 143847 http.timeout = 10000
050113 143847 http.content.limit = 65536
050113 143847 http.agent = Nutch/0.06 (Nutch; http://www.nutch.org/docs/en/bot.html; [EMAIL PROTECTED])
050113 143847 fetcher.server.delay = 1000
050113 143847 http.max.delays = 100
050113 143848 fetch okay, but can't parse http://www.nih.gov/, reason: No extern
al command defined for contentType:
050113 143848 status: segment 20050113143846, 1 pages, 0 errors, 23098 bytes, 10
00 ms
050113 143848 status: 1.0 pag es/s, 180.45312 kb/s, 23098.0 bytes/page
050113 143849 Updating D:\nutch\nutch-2005-01-10\nutch-nightly\nih\db
050113 143849 Updating for D:\nutch\nutch-2005-01-10\nutch-nightly\nih\segments\
20050113143846
050113 143849 Processing document 0
050113 143849 Finishing update
050113 143847 http.proxy.host = null
050113 143847 http.proxy.port = 8080
050113 143847 http.timeout = 10000
050113 143847 http.content.limit = 65536
050113 143847 http.agent = Nutch/0.06 (Nutch; http://www.nutch.org/docs/en/bot.html; [EMAIL PROTECTED])
050113 143847 fetcher.server.delay = 1000
050113 143847 http.max.delays = 100
050113 143848 fetch okay, but can't parse http://www.nih.gov/, reason: No extern
al command defined for contentType:
050113 143848 status: segment 20050113143846, 1 pages, 0 errors, 23098 bytes, 10
00 ms
050113 143848 status: 1.0 pag es/s, 180.45312 kb/s, 23098.0 bytes/page
050113 143849 Updating D:\nutch\nutch-2005-01-10\nutch-nightly\nih\db
050113 143849 Updating for D:\nutch\nutch-2005-01-10\nutch-nightly\nih\segments\
20050113143846
050113 143849 Processing document 0
050113 143849 Finishing update
050113 143849 Processing pagesByURL: Sorted 1 instructions in 0.015 seconds.
050113 143849 Processing pagesByURL: Sorted 66.66666666666667 instructions/secon
d
050113 143849 Processing pagesByURL: Merged to new DB containing 1 records in 0.
0 seconds
050113 143849 Processing pagesByURL: Merged Infinity records/second
050113 143849 Processing pagesByMD5: Sorted 1 instructions in 0.0 seconds.
050113 143849 Processing pagesByMD5: Sorted Infinity instructions/second
050113 143849 Processing pagesByMD5: Merged to new DB containing 1 records in 0.
0 seconds
050113 143849 Processing pagesByMD5: Merged Infinity records/second
050113 143849 Processing linksByMD5: Copied file (0 bytes) in 0.0 secs.
050113 143849 Processing linksByURL: Copied file (0 bytes) in 0.015 secs.
050113 143850 Update finished
050113 143850 FetchListTool started
050113 143850 Overall processing: Sorted 0 entries in 0.0 seconds.
050113 143850 Overall processing : Sorted NaN entries/second
050113 143850 FetchListTool completed
050113 143850 logging at INFO
050113 143851 Updating D:\nutch\nutch-2005-01-10\nutch-nightly\nih\db
050113 143851 Updating for D:\nutch\nutch-2005-01-10\nutch-nightly\nih\segments\
20050113143850
050113 143851 Finishing update
050113 143851 Update finished
050113 143851 FetchListTool started
050113 143851 Overall processing: Sorted 0 entries in 0.0 seconds.
050113 143851 Overall processing: Sorted NaN entries/second
050113 143851 FetchListTool completed
050113 143851 logging at INFO
050113 143852 Updating D:\nutch\nutch-2005-01-10\nutch-nightly\nih\db
050113 143857 Updating for D:\nutch\nutch-2005-01-10\nutch-nightly\nih\segments\
20050113143851
050113 143857 Finishing update
050113 143857 Update finished
050113 143857 FetchListTool started
050113 143857 Overall processing: Sorted 0 entries in 0.0 seconds.
050113 143857 Overall processing: Sorted NaN entries/second
050113 143857 FetchListTool completed
050113 143858 logging at INFO
050113 143859 Updating D:\nutch\nutch-2005-01-10\nutch-nightly\nih\db
050113 143859 Updating for D:\nutch\nutch-2005-01-10\nutch-nightly\nih\segments\
20050113143857
050113 143859 Finishing update
050113 143859 Update finished
050113 143859 FetchListTool started
050113 143859 Overall processing: Sorted 0 entries in 0.0 seconds.
050113 143859 Overall processing: Sorted NaN entries/second
050113 143859 FetchListTool completed
050113 143859 logging at INFO
050113 143900 Updating D:\nutch\nutch-2005-01-10\nutch-nightly\nih\db
050113 143900 Updating for D:\nutch\nutch-2005-01-10\nutch-nightly\nih\segments\
20050113143859
050113 143900 Finishing update
050113 143900 Update finished
050113 143900 FetchListTool started
050113 143900 Overall processing: Sorted 0 entries in 0.0 seconds.
050113 143900 Overall processing: Sorted NaN entries/second
050113 143900 FetchListTool completed
050113 143901 logging at INFO
050113 143902 Updating D:\nutch\nutch-2005-01-10\nutch-nightly\nih\db
050113 143902 Updating for D:\nutch\nutch-2005-01-10\nutch-nightly\nih\segments\
20050113143900
050113 143902 Finishing update
050113 143902 Update finished
050113 143902 FetchListTool started
050113 143902 Overall processing: Sorted 0 entries in 0.0 seconds.
050113 143902 Overall processing: Sorted NaN entries/second
050113 143902 FetchListTool completed
050113 143902 logging at INFO
050113 143903 Updating D:\nutch\nutch-2005-01-10\nutch-nightly\nih\db
050113 143903 Updating for D:\nutch\nutch-2005-01-10\nutch-nightly\nih\segments\
20050113143902
050113 143903 Finishing update
050113 143903 Update finished
050113 143903 FetchListTool started
050113 143903 Overall processing: Sorted 0 entries in 0.0 seconds.
050113 143903 Overall processing: Sorted NaN entries/second
050113 143903 FetchListTool completed
050113 143903 logging at INFO
050113 143904 Updating D:\nutch\nutch-2005-01-10\nutch-nightly\nih\db
050113 143905 Updating for D:\nutch\nutch-2005-01-10\nutch-nightly\nih\segments\
20050113143903
050113 143905 Finishing update
050113 143905 Update finished
050113 143905 FetchListTool started
050113 143905 Overall processing: Sorted 0 entries in 0.0 seconds.
050113 143905 Overall processing: Sorted NaN entries/second
050113 143905 FetchListTool completed
050113 143905 logging at INFO
050113 143906 Updating D:\nutch\nutch-2005-01-10\nutch-nightly\nih\db
050113 143906 Updating for D:\nutch\nutch-2005-01-10\nutch-nightly\nih\segments\
20050113143905
050113 143906 Finishing update
050113 143906 Update finished
050113 143906 FetchListTool started
050113 143906 Overall processing: Sorted 0 entries in 0.0 seconds.
050113 143906 Overall processing: Sorted NaN entries/second
050113 143906 FetchListTool completed
050113 143906 logging at INFO
050113 143907 Updating D:\nutch\nutch-2005-01-10\nutch-nightly\nih\db
050113 143907 Updating for D:\nutch\nutch-2005-01-10\nutch-nightly\nih\segments\
20050113143906
050113 143907 Finishing update
050113 143907 Update finished
050113 143908 FetchListTool started
050113 143908 Overall processing: Sorted 0 entries in 0.0 seconds.
050113 143908 Overall processing: Sorted NaN entries/second
050113 143908 FetchListTool completed
050113 143908 logging at INFO
050113 143909 indexing segment: D:\nutch\nutch-2005-01-10\nutch-nightly\nih\segm
ents\20050113143908
050113 143909 * Opening segment 20050113143908
050113 143909 * Indexing segment 20050113143908
050113 143909 * Optimizing index...
050113 143909 * Moving index to NFS if needed...
050113 143909 DONE indexing segment 20050113143908: total 1 records in 0.047 s (
Infinity rec/s).
050113 143909 done indexing
050113 143909 Reading url hashes...
050113 143909 Sorting url hashes...
050113 143909 Deleting url duplicates...
050113 143909 Deleted 0 url duplicates.
050113 143909 Reading content hashes...
050113 143909 Sorting content hashes...
050113 143909 Deleting content duplicates...
050113 143909 Deleted 0 content duplicates.
050113 143909 Duplicate deletion complete locally. Now returning to NFS...
050113 143909 DeleteDuplicates complete
050113 143909 merging segment indexes to: D:\nutch\nutch-2005-01-10\nutch-nightl
y\nih\index
050113 143909 done merging
050113 143909 crawl finished: nih
050113 143849 Processing pagesByURL: Sorted 66.66666666666667 instructions/secon
d
050113 143849 Processing pagesByURL: Merged to new DB containing 1 records in 0.
0 seconds
050113 143849 Processing pagesByURL: Merged Infinity records/second
050113 143849 Processing pagesByMD5: Sorted 1 instructions in 0.0 seconds.
050113 143849 Processing pagesByMD5: Sorted Infinity instructions/second
050113 143849 Processing pagesByMD5: Merged to new DB containing 1 records in 0.
0 seconds
050113 143849 Processing pagesByMD5: Merged Infinity records/second
050113 143849 Processing linksByMD5: Copied file (0 bytes) in 0.0 secs.
050113 143849 Processing linksByURL: Copied file (0 bytes) in 0.015 secs.
050113 143850 Update finished
050113 143850 FetchListTool started
050113 143850 Overall processing: Sorted 0 entries in 0.0 seconds.
050113 143850 Overall processing : Sorted NaN entries/second
050113 143850 FetchListTool completed
050113 143850 logging at INFO
050113 143851 Updating D:\nutch\nutch-2005-01-10\nutch-nightly\nih\db
050113 143851 Updating for D:\nutch\nutch-2005-01-10\nutch-nightly\nih\segments\
20050113143850
050113 143851 Finishing update
050113 143851 Update finished
050113 143851 FetchListTool started
050113 143851 Overall processing: Sorted 0 entries in 0.0 seconds.
050113 143851 Overall processing: Sorted NaN entries/second
050113 143851 FetchListTool completed
050113 143851 logging at INFO
050113 143852 Updating D:\nutch\nutch-2005-01-10\nutch-nightly\nih\db
050113 143857 Updating for D:\nutch\nutch-2005-01-10\nutch-nightly\nih\segments\
20050113143851
050113 143857 Finishing update
050113 143857 Update finished
050113 143857 FetchListTool started
050113 143857 Overall processing: Sorted 0 entries in 0.0 seconds.
050113 143857 Overall processing: Sorted NaN entries/second
050113 143857 FetchListTool completed
050113 143858 logging at INFO
050113 143859 Updating D:\nutch\nutch-2005-01-10\nutch-nightly\nih\db
050113 143859 Updating for D:\nutch\nutch-2005-01-10\nutch-nightly\nih\segments\
20050113143857
050113 143859 Finishing update
050113 143859 Update finished
050113 143859 FetchListTool started
050113 143859 Overall processing: Sorted 0 entries in 0.0 seconds.
050113 143859 Overall processing: Sorted NaN entries/second
050113 143859 FetchListTool completed
050113 143859 logging at INFO
050113 143900 Updating D:\nutch\nutch-2005-01-10\nutch-nightly\nih\db
050113 143900 Updating for D:\nutch\nutch-2005-01-10\nutch-nightly\nih\segments\
20050113143859
050113 143900 Finishing update
050113 143900 Update finished
050113 143900 FetchListTool started
050113 143900 Overall processing: Sorted 0 entries in 0.0 seconds.
050113 143900 Overall processing: Sorted NaN entries/second
050113 143900 FetchListTool completed
050113 143901 logging at INFO
050113 143902 Updating D:\nutch\nutch-2005-01-10\nutch-nightly\nih\db
050113 143902 Updating for D:\nutch\nutch-2005-01-10\nutch-nightly\nih\segments\
20050113143900
050113 143902 Finishing update
050113 143902 Update finished
050113 143902 FetchListTool started
050113 143902 Overall processing: Sorted 0 entries in 0.0 seconds.
050113 143902 Overall processing: Sorted NaN entries/second
050113 143902 FetchListTool completed
050113 143902 logging at INFO
050113 143903 Updating D:\nutch\nutch-2005-01-10\nutch-nightly\nih\db
050113 143903 Updating for D:\nutch\nutch-2005-01-10\nutch-nightly\nih\segments\
20050113143902
050113 143903 Finishing update
050113 143903 Update finished
050113 143903 FetchListTool started
050113 143903 Overall processing: Sorted 0 entries in 0.0 seconds.
050113 143903 Overall processing: Sorted NaN entries/second
050113 143903 FetchListTool completed
050113 143903 logging at INFO
050113 143904 Updating D:\nutch\nutch-2005-01-10\nutch-nightly\nih\db
050113 143905 Updating for D:\nutch\nutch-2005-01-10\nutch-nightly\nih\segments\
20050113143903
050113 143905 Finishing update
050113 143905 Update finished
050113 143905 FetchListTool started
050113 143905 Overall processing: Sorted 0 entries in 0.0 seconds.
050113 143905 Overall processing: Sorted NaN entries/second
050113 143905 FetchListTool completed
050113 143905 logging at INFO
050113 143906 Updating D:\nutch\nutch-2005-01-10\nutch-nightly\nih\db
050113 143906 Updating for D:\nutch\nutch-2005-01-10\nutch-nightly\nih\segments\
20050113143905
050113 143906 Finishing update
050113 143906 Update finished
050113 143906 FetchListTool started
050113 143906 Overall processing: Sorted 0 entries in 0.0 seconds.
050113 143906 Overall processing: Sorted NaN entries/second
050113 143906 FetchListTool completed
050113 143906 logging at INFO
050113 143907 Updating D:\nutch\nutch-2005-01-10\nutch-nightly\nih\db
050113 143907 Updating for D:\nutch\nutch-2005-01-10\nutch-nightly\nih\segments\
20050113143906
050113 143907 Finishing update
050113 143907 Update finished
050113 143908 FetchListTool started
050113 143908 Overall processing: Sorted 0 entries in 0.0 seconds.
050113 143908 Overall processing: Sorted NaN entries/second
050113 143908 FetchListTool completed
050113 143908 logging at INFO
050113 143909 indexing segment: D:\nutch\nutch-2005-01-10\nutch-nightly\nih\segm
ents\20050113143908
050113 143909 * Opening segment 20050113143908
050113 143909 * Indexing segment 20050113143908
050113 143909 * Optimizing index...
050113 143909 * Moving index to NFS if needed...
050113 143909 DONE indexing segment 20050113143908: total 1 records in 0.047 s (
Infinity rec/s).
050113 143909 done indexing
050113 143909 Reading url hashes...
050113 143909 Sorting url hashes...
050113 143909 Deleting url duplicates...
050113 143909 Deleted 0 url duplicates.
050113 143909 Reading content hashes...
050113 143909 Sorting content hashes...
050113 143909 Deleting content duplicates...
050113 143909 Deleted 0 content duplicates.
050113 143909 Duplicate deletion complete locally. Now returning to NFS...
050113 143909 DeleteDuplicates complete
050113 143909 merging segment indexes to: D:\nutch\nutch-2005-01-10\nutch-nightl
y\nih\index
050113 143909 done merging
050113 143909 crawl finished: nih
Do you Yahoo!?
Yahoo! Mail - Easier than ever with enhanced search. Learn more.
