Hi Arthur, On Mon, Feb 15, 2016 at 1:02 AM, <user-digest-h...@nutch.apache.org> wrote:
> From: Arthur Yarwood <art...@fubaby.com> > To: "user@nutch.apache.org" <user@nutch.apache.org> > Cc: > Date: Sun, 14 Feb 2016 22:08:13 +0000 > Subject: Nutch/Tika failed to parse text/html content > I'm using nutch 1.10 and one site (http://www.choice-hifi.com) is failing > to be parsed. It doesn't seem to correctly ascertain the pages are mimetype > text/html. It then passes it to Tika parser, which also fails and returns > it as a octet stream and no further processing of page occurs. > > [snip] > > I can't work out what's going wrong here. The page in question is a little > old school html wise, but does have: > > OK I can replicate your results using the parsechecker tool. If however I use the parsechecker tool with the -forceAs text/html filag, I get the following parse result. It would seem that it is definitely a detection issue with the mimetype being detected. By way, I verified this with both parse-tika and parse-html. Can you please log an issue in Jira and we can try to address this issue? Thanks Lewis lmcgibbn@LMC-032857 /usr/local/trunk_sync $ ./runtime/local/bin/nutch parsechecker -dumpText -forceAs text/html http://www.choice-hifi.com fetching: http://www.choice-hifi.com robots.txt whitelist not configured. parsing: http://www.choice-hifi.com contentType: text/html signature: a1655a06344a546dfb563d2c395a5033 --------- Url --------------- http://www.choice-hifi.com --------- ParseData --------- Version: 5 Status: success(1,0) Title: Choice Hi-fi UK - High End Audio Equipment - Choice HiFi (Home Page) Outlinks: 46 outlink: toUrl: http://www.choice-hifi.com/includes/cms_functions.js anchor: outlink: toUrl: http://www.choice-hifi.com/choicehifi.css anchor: outlink: toUrl: http://www.choice-hifi.com/choicehifi_print.css anchor: outlink: toUrl: http://www.choice-hifi.com/includes/dreamweaver.js anchor: outlink: toUrl: http://www.choicehifi.com anchor: Choice Hi Fi Logo outlink: toUrl: http://www.choice-hifi.com/images/index_01.gif anchor: Choice Hi Fi Logo outlink: toUrl: http://www.choice-hifi.com/images/index_02.gif anchor: outlink: toUrl: http://www.choice-hifi.com/images/index_03.gif anchor: outlink: toUrl: http://www.choice-hifi.com/images/index_04.gif anchor: outlink: toUrl: http://www.choice-hifi.com/images/index_05.gif anchor: outlink: toUrl: http://www.choice-hifi.com/images/index_06.gif anchor: outlink: toUrl: http://www.choicehifi.com anchor: Choice Hi Fi Logo outlink: toUrl: http://www.choice-hifi.com/images/index_07.gif anchor: Choice Hi Fi Logo outlink: toUrl: http://www.choice-hifi.com/Hi-Fi-New/page/53 anchor: outlink: toUrl: http://www.choice-hifi.com/images/index_08.gif anchor: outlink: toUrl: http://www.choice-hifi.com/Hi-Fi-Exchange-FREE-ADS-Used-Second-Hand-HiFi-Equipment/page/27 anchor: outlink: toUrl: http://www.choice-hifi.com/images/index_09.gif anchor: outlink: toUrl: http://www.choice-hifi.com/Custom-installation-Smart-Home-Home-Automation/page/33 anchor: outlink: toUrl: http://www.choice-hifi.com/images/index_10.gif anchor: outlink: toUrl: http://www.choice-hifi.com/Home-Cinema-AV/page/29 anchor: outlink: toUrl: http://www.choice-hifi.com/images/index_11.gif anchor: outlink: toUrl: http://www.choice-hifi.com/images/index_12.gif anchor: outlink: toUrl: http://www.choicehifi.com anchor: Choice Hi Fi Logo outlink: toUrl: http://www.choice-hifi.com/images/index_13.gif anchor: Choice Hi Fi Logo outlink: toUrl: http://www.choice-hifi.com/images/index_18.gif anchor: outlink: toUrl: http://www.choice-hifi.com/images/gallery/thumb_ML no33h(1).jpg anchor: ML 33H outlink: toUrl: http://www.choice-hifi.com/galleries/photos.asp?folder=generalexamples&title=generalexamples anchor: Photo Gallery outlink: toUrl: http://www.choice-hifi.com/Hi-Fi-Exchange-FREE-ADS-Used-Second-Hand-HiFi-Equipment/page/34 anchor: HiFi FOR SALE VIEW all items currently available outlink: toUrl: http://www.choice-hifi.com/contact_us.asp anchor: contact us outlink: toUrl: http://www.choice-hifi.com/Hi-Fi-Exchange-FREE-ADS-Used-Second-Hand-HiFi-Equipment/page/27 anchor: Hi-Fi Exchange outlink: toUrl: http://www.jscomputing.co.uk anchor: JSComputing outlink: toUrl: http://www.choice-hifi.com/home/page/25 anchor: Home outlink: toUrl: http://www.choice-hifi.com/forum/index.php anchor: Forum outlink: toUrl: http://www.choice-hifi.com/items_search.asp anchor: Search Hi-Fi Exchange outlink: toUrl: http://www.choice-hifi.com/../index.asp?page=32 anchor: outlink: toUrl: http://www.choice-hifi.com/brands-stocked-available/page/32 anchor: Brands Stocked / Available outlink: toUrl: http://www.choice-hifi.com/Unique-Products/page/69 anchor: Unique Products outlink: toUrl: http://www.choice-hifi.com/Demonstrations/page/36 anchor: Demonstrations outlink: toUrl: http://www.choice-hifi.com/forum/index.php anchor: Advice and Help outlink: toUrl: http://www.choice-hifi.com/Warranty/page/38 anchor: Warranty outlink: toUrl: http://www.choice-hifi.com/Terms-and-Conditions/page/39 anchor: Terms & Conditions outlink: toUrl: http://www.choice-hifi.com/contact_us.asp anchor: Contact Us outlink: toUrl: http://www.choice-hifi.com/register_user.asp anchor: Register outlink: toUrl: http://www.choice-hifi.com/Benefits-of-registering/page/79 anchor: Benefits of registering outlink: toUrl: http://www.choice-hifi.com/forgotten_pass.asp anchor: Retrieve your login details outlink: toUrl: http://www.choice-hifi.com/images/cards.gif anchor: Accepted Credit and Debit Cards Content Metadata: nutch.fetch.time=1456279073140 Server=Microsoft-IIS/6.0 Filter=www.choice-hifi.com/=/identificador.asp?ids=/ Connection=close Date=Wed, 24 Feb 2016 01:54:04 GMT nutch.crawl.score=0.0 X-Powered-By= ASP.NET Parse Metadata: CharEncodingForConversion=windows-1252 OriginalCharEncoding=windows-1252 --------- ParseText --------- Content-Type: text/html Set-Cookie: ASPSESSIONIDCQCQDCRQ=KCNJHPHCGEBFNNFMNBLJHCJO; path=/ Cache-control: private Choice Hi-fi UK - High End Audio Equipment - Choice HiFi (Home Page) Choice HiFi (Home Page) Photo Gallery A Passion for Music, A love of HiFi always Exceeding Expectations New, X-Dem and Second Hand hifi Equipment + Your HiFi Adverts HiFi FOR SALE VIEW all items currently available Click Here For more than two decades Choice HiFi has provided a unique and unrivalled service to the dedicated audiophile. Born from the demand for clear concise and unbiased advice, together with the best price and service, we have been successful in marrying New, X-Dem and Second Hand hi-end audio equipment to fulfill our customers' needs and expectations. At Choice HiFi you can choose from hundreds of hi-end hifi equipment manufacturers, from the well established to the emerging; Krell, Linn, Naim, Audio research, Mark Levinson, Focal JMLab, Halcro, Wilson Benesch, Michell Engineering, McIntosh, Sonus Faber, B&W, Chord Electronics, SME, Clearaudio, Boulder Amplifiers, Musical Fidelity, Ayre, EMM, YG, Stillpoints, Magico, MFA… to mention just a few. HiFi reviews from HiFi News, HiFi World, HiFi Choice, HiFi+, Stereophile and other respected hifi press can only go so far. Our 20+ year, hands on experience with all leading high end audio equipment brands, in all conceivable combinations, has meant that we have built up vast knowledge and experience; with no political axes to grind- we give it to you straight. We welcome Part Exchange At Choice HiFi we recognize that: -For every new piece of hi-end audio equipment purchased, there is likely to be a trade in or part-exchange of a used/ second hand piece of hi-end hifi equipment. This trade in, contrary to recognized wisdom, is in fact a benefit to us not a hindrance. It gives us the opportunity to fulfill someone else's requirements. Uniquely we have also in recent years added the facility of allowing you to add your own items for sale to the HiFi Exchange. This site lists our current stock. However if you are unable to find the specific item of new or used hi-end hifi equipment you require please contact us and we will track it down for you. Home Cinema A natural progression from hi-end audio is hi-end Home Cinema and film systems. Either projector or Plasma (LED/LCD) based Home Cinema systems can re create the passion scale and emotion only previously achievable in the best movie houses. Home Automation Integrated systems Or Custom Installation uses the latest digital and wireless technologies to distribute audio, control lighting and provide data storage and distribution whether it is music and film or computer networking a home automation infrastructure can do it all. For further information on any subject of interest on this site please call Alain on Tel: +44(0) 208 392 1959 Office Tel: +44(0) 7768 720 456 Mobile email: i...@choice-hifi.com Go To The HiFi Exchange BROWSE/ BUY/ SELL USED HiFi -Browse All Second Hand HiFi Items FOR SALE Hi-Fi Exchange FREE ADS Produced by JSComputing Home Forum Search Hi-Fi Exchange Brands Stocked / Available Unique Products Demonstrations Advice and Help Warranty Terms & Conditions Contact Us Register Benefits of registering login username: password: Retrieve your login details
