Hi, I have build a plugin to do some POST request using HttpClient 4.1 and I have allowed to follow redirection. Everything goes well but I don't have any outlink from the content.
Where should I look for to solve this problem? Here is some stack trace 2014-04-25 14:32:28,342 INFO crawl.LinkDb - LinkDb: starting at 2014-04-25 14:32:28 2014-04-25 14:32:28,345 INFO crawl.LinkDb - LinkDb: linkdb: redirectCrawlTest4/linkdb 2014-04-25 14:32:28,345 INFO crawl.LinkDb - LinkDb: URL normalize: true 2014-04-25 14:32:28,346 INFO crawl.LinkDb - LinkDb: URL filter: true 2014-04-25 14:32:28,346 INFO crawl.LinkDb - LinkDb: internal links will be ignored. 2014-04-25 14:32:28,346 INFO crawl.LinkDb - LinkDb: adding segment: redirectCrawlTest4/segments/20140425143157 2014-04-25 14:32:30,889 INFO regex.RegexURLNormalizer - can't find rules for scope 'linkdb', using default 2014-04-25 14:32:31,801 INFO crawl.LinkDb - LinkDb: finished at 2014-04-25 14:32:31, elapsed: 00:00:03 Here is the dump from readdb Recno:: 0 URL:: http://www.cadremploi.fr/emploi/fr.cadremploi.publi.page.recherche_offres.RechercheOffresCtrl/1571601239 CrawlDatum:: Version: 7 Status: 1 (db_unfetched) Fetch time: Fri Apr 25 14:31:43 CEST 2014 Modified time: Thu Jan 01 01:00:00 CET 1970 Retries since fetch: 0 Retry interval: 2592000 seconds (30 days) Score: 1.0 Signature: null Metadata: method=post _ngt_=1398429111854 param=chk_fct:20500,mth:Rechercher,redirect:%2Femploi%2Frecherche_offres,provenance:2 ParseData:: Version: 5 Status: success(1,0) Title: Outlinks: 0 Content Metadata: Expires=Sat, 26 Jul 1997 05:00:00 GMT _fst_=33 nutch.segment.name=20140425143157 Connection=close Server=Apache-Coyote/1.1 X-Cache=MISS from dumbledore-1 Cache-Control=no-store,no-cache Pragma=no-cache nutch.content.digest=a74e846cc14dbd73369183d3def67e9c Date=Fri, 25 Apr 2014 12:32:05 GMT Vary=Accept-Encoding nutch.crawl.score=1.0 Via=1.0 www.cadremploi.fr Content-Type=text/html Parse Metadata: CharEncodingForConversion=windows-1252 OriginalCharEncoding=windows-1252 this line worried me: 2014-04-25 14:32:30,889 INFO regex.RegexURLNormalizer - can't find rules for scope 'linkdb', using default But I don't know how to solve it Thanks for the help, Zabini -- View this message in context: http://lucene.472066.n3.nabble.com/No-outlink-after-a-redirect-tp4133122.html Sent from the Nutch - User mailing list archive at Nabble.com.
