I think this is fantastic Mohammad! Can you update the patch on NUTCH-1933 with this improvement, so we can get it into the sources?
Cheers, Chris ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Chris Mattmann, Ph.D. Chief Architect Instrument Software and Science Data Systems Section (398) NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA Office: 168-519, Mailstop: 168-527 Email: [email protected] WWW: http://sunset.usc.edu/~mattmann/ ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Adjunct Associate Professor, Computer Science Department University of Southern California, Los Angeles, CA 90089 USA ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ -----Original Message----- From: Mohammad Al-Mohsin <[email protected]> Reply-To: "[email protected]" <[email protected]> Date: Saturday, February 21, 2015 at 6:03 AM To: "[email protected]" <[email protected]> Cc: Mohammad Al-Mohsin <[email protected]> Subject: Nutch-Selenium Plugin Truncates Binary Data >I am using >nutch-selenium <https://github.com/momer/nutch-selenium> plugin and I >also have >Tesseract <https://wiki.apache.org/tika/TikaOCR> installed for parsing >text off images. > > >While crawling with Nutch & selenium, I noticed that binary data (e.g. >images, pdf) are always truncated and thus skip/fail parsing. Here is a >sample of the log: >Content of size 800750 was truncated to 368. Content is truncated, parse >may fail! > >When I turn selenium off, parsing works fine and the content is not >truncated. > > >I found that nutch-selenium gets the html body of whatever Firefox >displays. So even though you're fetching an image, selenium will just >give you the image html tag instead of the image itself. >e.g. <img src='xyz.png' height="400" width="600"> > > >To get around this, I modified selenium plugin to handle the fetch only >if the Content-Type header starts with 'text', i.e. to catch 'text/html'. >Otherwise, if the content is not textual, it just returns the content as >protocol-httpclient does. > > >Now, I am getting binary data properly parsed and also getting selenium >handle page rendering with javascript. > > >Is this is the proper way to tackle this? what do you think? > > > > >Best regards, >Mohammad Al-Mohsin > >
