[
https://issues.apache.org/jira/browse/NUTCH-1634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
kaveh minooie updated NUTCH-1634:
---------------------------------
Attachment: NUTCH-1634-2.x.patch
I'll check the trunk as well and will post a patch for it if needed.
> readdb -stats show the result twice
> -----------------------------------
>
> Key: NUTCH-1634
> URL: https://issues.apache.org/jira/browse/NUTCH-1634
> Project: Nutch
> Issue Type: Bug
> Components: crawldb
> Affects Versions: 2.2.1
> Reporter: kaveh minooie
> Priority: Minor
> Attachments: NUTCH-1634-2.x.patch
>
>
> right now this is the ouput:
> WebTable statistics start
> Statistics for WebTable:
> status 2 (status_fetched): 1115
> min score: 0.0
> retry 2: 2
> retry 0: 11369
> jobs: {db_stats-job_local1037462743_0001={jobID=job_local1037462743_0001,
> jobName=db_stats, counters={File Input Format Counters ={BYTES_READ=0},
> Map-Reduce Framework={MAP_OUTPUT_MATERIALIZED_BYTES=209,
> MAP_INPUT_RECORDS=11376, REDUCE_SHUFFLE_BYTES=0, SPILLED_RECORDS=24,
> MAP_OUTPUT_BYTES=602928, COMMITTED_HEAP_BYTES=1181220864, CPU_MILLISECONDS=0,
> SPLIT_RAW_BYTES=757, COMBINE_INPUT_RECORDS=45504, REDUCE_INPUT_RECORDS=12,
> REDUCE_INPUT_GROUPS=12, COMBINE_OUTPUT_RECORDS=12, PHYSICAL_MEMORY_BYTES=0,
> REDUCE_OUTPUT_RECORDS=12, VIRTUAL_MEMORY_BYTES=0, MAP_OUTPUT_RECORDS=45504},
> FileSystemCounters={FILE_BYTES_READ=1819, FILE_BYTES_WRITTEN=166199}, File
> Output Format Counters ={BYTES_WRITTEN=373}}}}
> retry 1: 5
> status 5 (status_redir_perm): 69
> max score: 1.0
> TOTAL urls: 11376
> status 3 (status_gone): 3
> status 4 (status_redir_temp): 3
> status 1 (status_unfetched): 10186
> avg score: 0.00342827
> WebTable statistics: done
> status 2 (status_fetched): 1115
> min score: 0.0
> retry 2: 2
> retry 0: 11369
> jobs: {db_stats-job_local1037462743_0001={jobID=job_local1037462743_0001,
> jobName=db_stats, counters={File Input Format Counters ={BYTES_READ=0},
> Map-Reduce Framework={MAP_OUTPUT_MATERIALIZED_BYTES=209,
> MAP_INPUT_RECORDS=11376, REDUCE_SHUFFLE_BYTES=0, SPILLED_RECORDS=24,
> MAP_OUTPUT_BYTES=602928, COMMITTED_HEAP_BYTES=1181220864, CPU_MILLISECONDS=0,
> SPLIT_RAW_BYTES=757, COMBINE_INPUT_RECORDS=45504, REDUCE_INPUT_RECORDS=12,
> REDUCE_INPUT_GROUPS=12, COMBINE_OUTPUT_RECORDS=12, PHYSICAL_MEMORY_BYTES=0,
> REDUCE_OUTPUT_RECORDS=12, VIRTUAL_MEMORY_BYTES=0, MAP_OUTPUT_RECORDS=45504},
> FileSystemCounters={FILE_BYTES_READ=1819, FILE_BYTES_WRITTEN=166199}, File
> Output Format Counters ={BYTES_WRITTEN=373}}}}
> retry 1: 5
> status 5 (status_redir_perm): 69
> max score: 1.0
> TOTAL urls: 11376
> status 3 (status_gone): 3
> status 4 (status_redir_temp): 3
> status 1 (status_unfetched): 10186
> avg score: 0.00342827
> imho, it should be this:
> WebTable statistics start
> Statistics for WebTable:
> status 2 (status_fetched): 1115
> min score: 0.0
> retry 2: 2
> retry 0: 11369
> jobs: {db_stats-job_local801282144_0001={jobID=job_local801282144_0001,
> jobName=db_stats, counters={File Input Format Counters ={BYTES_READ=0},
> Map-Reduce Framework={MAP_OUTPUT_MATERIALIZED_BYTES=209,
> MAP_INPUT_RECORDS=11376, REDUCE_SHUFFLE_BYTES=0, SPILLED_RECORDS=24,
> MAP_OUTPUT_BYTES=602928, COMMITTED_HEAP_BYTES=1122631680, CPU_MILLISECONDS=0,
> SPLIT_RAW_BYTES=757, COMBINE_INPUT_RECORDS=45504, REDUCE_INPUT_RECORDS=12,
> REDUCE_INPUT_GROUPS=12, COMBINE_OUTPUT_RECORDS=12, PHYSICAL_MEMORY_BYTES=0,
> REDUCE_OUTPUT_RECORDS=12, VIRTUAL_MEMORY_BYTES=0, MAP_OUTPUT_RECORDS=45504},
> FileSystemCounters={FILE_BYTES_READ=1819, FILE_BYTES_WRITTEN=166191}, File
> Output Format Counters ={BYTES_WRITTEN=373}}}}
> retry 1: 5
> status 5 (status_redir_perm): 69
> max score: 1.0
> TOTAL urls: 11376
> status 3 (status_gone): 3
> status 4 (status_redir_temp): 3
> status 1 (status_unfetched): 10186
> avg score: 0.00342827
> WebTable statistics: done
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
