Hello All
Is this a bug?If Yes then should I move to nutch 2.3.1?Still waiting for
response.
Thanks
On Thursday 21 January 2016 12:45 PM, harsh wrote:
I had configured nutch 2.3 in local enviornment with
gora-mongoDB(0.6.1).Exceution flow was given below.
bin/nutch generate -topN 100 -crawlId nutch_crawler
GeneratorJob: starting at 2016-01-21 12:40:19
GeneratorJob: Selecting best-scoring urls due for fetch.
GeneratorJob: starting
GeneratorJob: filtering: true
GeneratorJob: normalizing: true
GeneratorJob: topN: 100
GeneratorJob: finished at 2016-01-21 12:40:22, time elapsed: 00:00:02
GeneratorJob: generated batch id: 1453360219-812835806 containing 0 URLs
bin/nutch fetch 1453359290-25044495
FetcherJob: starting at 2016-01-21 12:41:34
FetcherJob: batchId: 1453359290-25044495
FetcherJob: threads: 10
FetcherJob: parsing: false
FetcherJob: resuming: false
FetcherJob : timelimit set for : -1
Using queue mode : byHost
Fetcher: threads: 10
QueueFeeder finished: total 0 records. Hit by time limit :0
-finishing thread FetcherThread0, activeThreads=0
-finishing thread FetcherThread1, activeThreads=0
-finishing thread FetcherThread2, activeThreads=0
-finishing thread FetcherThread4, activeThreads=0
-finishing thread FetcherThread5, activeThreads=0
-finishing thread FetcherThread3, activeThreads=0
-finishing thread FetcherThread6, activeThreads=0
Fetcher: throughput threshold: -1
Fetcher: throughput threshold sequence: 5
-finishing thread FetcherThread7, activeThreads=0
-finishing thread FetcherThread8, activeThreads=0
-finishing thread FetcherThread9, activeThreads=0
0/0 spinwaiting/active, 0 pages, 0 errors, 0.0 0 pages/s, 0 0 kb/s, 0
URLs in 0 queues
-activeThreads=0
Using queue mode : byHost
Fetcher: threads: 10
*QueueFeeder finished: total 0 records. Hit by time limit :0*
-finishing thread FetcherThread0, activeThreads=0
-finishing thread FetcherThread2, activeThreads=0
-finishing thread FetcherThread1, activeThreads=0
-finishing thread FetcherThread3, activeThreads=0
-finishing thread FetcherThread5, activeThreads=1
-finishing thread FetcherThread4, activeThreads=0
-finishing thread FetcherThread6, activeThreads=0
-finishing thread FetcherThread7, activeThreads=0
-finishing thread FetcherThread8, activeThreads=0
Fetcher: throughput threshold: -1
-finishing thread FetcherThread9, activeThreads=0
Fetcher: throughput threshold sequence: 5
0/0 spinwaiting/active, 0 pages, 0 errors, 0.0 0 pages/s, 0 0 kb/s, 0
URLs in 0 queues
-activeThreads=0
FetcherJob: finished at 2016-01-21 12:41:47, time elapsed: 00:00:13
While there was only one document in MongoDB
db.nutch_crawler_webpage.findOne()
{
"_id" : "com.thehindu.www:http/",
"fetchTime" : NumberLong("1453359144150"),
"fetchInterval" : 2592000,
"score" : 1,
"markers" : {
"dist" : "0",
"_injmrk_" : "y"
},
"metadata" : {
"_csh_" : BinData(0,"P4AAAA==")
}
}
Internal links were not found.Why?Where they should be stored.
The configuration of nutch-site.xml is
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<name>storage.data.store.class</name>
<value>org.apache.gora.mongodb.store.MongoStore</value>
<description>Default class for storing data</description>
</property>
<property>
<name>http.agent.name</name>
<value>Orkash Crawler</value>
</property>
<property>
<name>plugin.folders</name>
<value>/home/user/Downloads/nutch_jars/mongo_nnutch_2/apache-nutch-2.3/build/plugins</value>
</property>
<property>
<name>http.content.limit</name>
<value>-1</value>
</property>
</configuration>
