Hi Semyon,

> # Logging Threshold
> log4j.threshold=ALL

Ok, I get similar messages with
   log4j.logger.org.apache.nutch=TRACE

  [Error] :24:21: Missing attribute name.
  [Warning] :27:16: Start element <DIV> automatically closes element <P>.

I think they can be ignored unless there is some missing content not contained 
in the output of
parse-html.

Best,
Sebastian

On 11/19/18 5:04 PM, Semyon Semyonov wrote:
> Upd. I finally managed to find out why I got these kind of messages in my 
> version
> "Missing attribute name and as a result I have an empty page back" 
> 
> It is not because of code but because of logs properties. Now, I managed to 
> reproduce it with master branch.
> 
> Having this log settings
> # Logging Threshold
> log4j.threshold=ALL
> 
> I receive
> [Error] :23:70: Missing attribute name.
> [Error] :24:68: Missing attribute name.
> [Error] :25:108: Missing attribute name.
> 
> etc...
> 
> Are these errors important?
> 
> 
> 
> 
> 
> 
> Sent: Thursday, November 15, 2018 at 3:33 PM
> From: "Semyon Semyonov" <semyon.semyo...@mail.com>
> To: user@nutch.apache.org
> Subject: Re: Quality problems of crawling. Parsing(Missing attribute name), 
> fetching(empty body) and javascript.
> Everyone, we need some kind of commercial support(maybe extra tools) for 
> improving the quality of crawling and fixing similar issues. If you are 
> interested please contact me.
> 
> Sebastian,
> My bad, I had another version(modified 1.14).
> In addition it is easy to misunderstand the results.
> 
> bin/nutch parsechecker -Dplugin.includes='protocol-okhttp|parse-tika' 
> -dumpText http://www.vialucy.nl/ return
> Parse Metadata: dc:title=Vialucy | nieuws
> 
> bin/nutch parsechecker -dumpText 
> http://www.vialucy.nl/[http://www.vialucy.nl/][http://www.vialucy.nl/[http://www.vialucy.nl/]]
> Parse Metadata:
> 
> So, default one provides empty metadata and no error messages. This is a bit 
> confusing.
> 
> Thanks.
> 
> 
> Sent: Thursday, November 15, 2018 at 3:05 PM
> From: "Sebastian Nagel" <wastl.na...@googlemail.com.INVALID>
> To: user@nutch.apache.org
> Subject: Re: Quality problems of crawling. Parsing(Missing attribute name), 
> fetching(empty body) and javascript.
> Hi Semyon,
> 
>> Is there any reasons to keep the default HTML plugin there? only for 
>> maintenance ?
> 
> Are there really HTML pages where parse-html fails?
> 
> From my experience it still does a good job and parses almost every HTML page,
> including HTML5. But I've never run any large scale comparison.
> 
> One argument pro: it's much smaller. While parse-tika including dependencies 
> uses around 60 MB,
> parse-html ships with only few 100 kB.
> 
> Regarding 
> http://www.vialucy.nl/[http://www.vialucy.nl/][http://www.vialucy.nl/[http://www.vialucy.nl/]]
>  : if the noindex is removed the page
> is parsed well by parse-tika and parse-html and the outputs only differ
> in white space in the parsed text.
> 
> Of course, for the long term parse-html should be either actively maintained
> or needs to be skipped.
> 
> Best,
> Sebastian
> 
> On 11/15/18 2:39 PM, Semyon Semyonov wrote:
>> Hi Sebastian,
>>  
>> Thanks for the detailed response.
>> I will try to migrate to Tika.
>>
>> Is there any reasons to keep the default HTML plugin there? only for 
>> maintenance ?
>>  
>> Semyon. 
>>
>> Sent: Thursday, November 15, 2018 at 2:23 PM
>> From: "Sebastian Nagel" <wastl.na...@googlemail.com.INVALID>
>> To: user@nutch.apache.org
>> Subject: Re: Quality problems of crawling. Parsing(Missing attribute name), 
>> fetching(empty body) and javascript.
>> Hi Semyon,
>>
>> I've tried to reproduce your problems using the recent Nutch master 
>> (upcoming 1.16).
>> I cannot see any issues, except that Javascript is not executed but that's 
>> clear.
>> Of course, you are free to use parse-tika instead of parse-html which is 
>> legacy.
>> See results below.
>>
>> Best,
>> Sebastian
>>
>>> http://www.vialucy.nl/[http://www.vialucy.nl/][http://www.vialucy.nl/[http://www.vialucy.nl/]][http://www.vialucy.nl/[http://www.vialucy.nl/][http://www.vialucy.nl/[http://www.vialucy.nl/]]][http://www.vialucy.nl/[http://www.vialucy.nl/][http://www.vialucy.nl/[http://www.vialucy.nl/]][http://www.vialucy.nl/[http://www.vialucy.nl/][http://www.vialucy.nl/[http://www.vialucy.nl/]]][http://www.vialucy.nl/[http://www.vialucy.nl/][http://www.vialucy.nl/[http://www.vialucy.nl/]][http://www.vialucy.nl/[http://www.vialucy.nl/][http://www.vialucy.nl/[http://www.vialucy.nl/]]]]]
>>
>> Successfully fetched and parsed (no errors). Of course, there is no content 
>> kept
>> because of robots=noindex. Here the output of parsechecker:
>>
>> % bin/nutch parsechecker -Dplugin.includes='protocol-okhttp|parse-tika' 
>> -dumpText 
>> http://www.vialucy.nl/[http://www.vialucy.nl/][http://www.vialucy.nl/[http://www.vialucy.nl/]][http://www.vialucy.nl/[http://www.vialucy.nl/][http://www.vialucy.nl/[http://www.vialucy.nl/]]][http://www.vialucy.nl/[http://www.vialucy.nl/][http://www.vialucy.nl/[http://www.vialucy.nl/]][http://www.vialucy.nl/[http://www.vialucy.nl/][http://www.vialucy.nl/[http://www.vialucy.nl/]]]]
>> ...
>> Parse Metadata:
>> dc:title=Vialucy | nieuws uit Les Vans – Ardêche – France
>> Content-Encoding=UTF-8
>> generator=WordPress 3.1
>> robots=noindex,nofollow
>> Content-Language=en-US
>> Content-Type=text/html; charset=UTF-8
>>
>>
>>> https://www.vishandelbunschoten.nl/[https://www.vishandelbunschoten.nl/][https://www.vishandelbunschoten.nl/[https://www.vishandelbunschoten.nl/]][https://www.vishandelbunschoten.nl/[https://www.vishandelbunschoten.nl/][https://www.vishandelbunschoten.nl/[https://www.vishandelbunschoten.nl/]]][https://www.vishandelbunschoten.nl/[https://www.vishandelbunschoten.nl/][https://www.vishandelbunschoten.nl/[https://www.vishandelbunschoten.nl/]][https://www.vishandelbunschoten.nl/[https://www.vishandelbunschoten.nl/][https://www.vishandelbunschoten.nl/[https://www.vishandelbunschoten.nl/]]]]
>> Succeeds if you can trick the anti-bot software, otherwise the server sends
>> empty content back. Recently discussed on this list.
>>
>>
>>> 3) Javascipt problems
>>>
>>> http://www.amphar.com/Home.html[http://www.amphar.com/Home.html][http://www.amphar.com/Home.html[http://www.amphar.com/Home.html]][http://www.amphar.com/Home.html[http://www.amphar.com/Home.html][http://www.amphar.com/Home.html[http://www.amphar.com/Home.html]]][http://www.amphar.com/Home.html[http://www.amphar.com/Home.html][http://www.amphar.com/Home.html[http://www.amphar.com/Home.html]][http://www.amphar.com/Home.html[http://www.amphar.com/Home.html][http://www.amphar.com/Home.html[http://www.amphar.com/Home.html]]]]
>>
>> Yes, Javascript is not executed. But fetching and parsing works pretty fine
>> for the HTML page as such:
>>
>> % bin/nutch parsechecker -Dplugin.includes='protocol-okhttp|parse-tika' \
>> -dumpText 
>> http://www.amphar.com/Home.html[http://www.amphar.com/Home.html][http://www.amphar.com/Home.html[http://www.amphar.com/Home.html]][http://www.amphar.com/Home.html[http://www.amphar.com/Home.html][http://www.amphar.com/Home.html[http://www.amphar.com/Home.html]]][http://www.amphar.com/Home.html[http://www.amphar.com/Home.html][http://www.amphar.com/Home.html[http://www.amphar.com/Home.html]][http://www.amphar.com/Home.html[http://www.amphar.com/Home.html][http://www.amphar.com/Home.html[http://www.amphar.com/Home.html]]]]
>> fetching: 
>> http://www.amphar.com/Home.html[http://www.amphar.com/Home.html][http://www.amphar.com/Home.html[http://www.amphar.com/Home.html]][http://www.amphar.com/Home.html[http://www.amphar.com/Home.html][http://www.amphar.com/Home.html[http://www.amphar.com/Home.html]]][http://www.amphar.com/Home.html[http://www.amphar.com/Home.html][http://www.amphar.com/Home.html[http://www.amphar.com/Home.html]][http://www.amphar.com/Home.html[http://www.amphar.com/Home.html][http://www.amphar.com/Home.html[http://www.amphar.com/Home.html]]]]
>> ...
>> Status: success(1,0)
>> Title: Home
>> Outlinks: 19
>> ...
>> Parse Metadata: iWeb-Build=local-build-20140815 
>> X-UA-Compatible=IE=EmulateIE7 viewport=width=700
>> dc:title=Home Content-Encoding=UTF-8 Content-Type-Hint=text/html; 
>> charset=UTF-8 Content-Language=en
>> Content-Type=application/xhtml+xml; charset=UTF-8 Generator=iWeb 3.0.4
>>
>> Founded in 1975, Amphar B.V. provides solutions, services and support to the 
>> generic pharmaceutical
>> industry.
>> Headquartered in Amsterdam, The Netherlands, we assist our customers in 
>> identifying and developing
>> new products, carefully select or initiate appropriate sources for Active 
>> Pharmaceutical Ingredients
>> (APIs), develop and test formulations as well as compilation and submission 
>> of the required
>> regulatory documentation and data.
>> With our dedicated staff of experienced professionals and our logistics 
>> centre at Amsterdam Schiphol
>> International Airport, we are well positioned to anticipate and react 
>> swiftly to the dynamic
>> requirements of our customers.
>> Amphar B.V.
>>  
>>
>>
>> On 11/15/18 1:30 PM, Semyon Semyonov wrote:
>>> Ok, with parsing it is more or less clear(in theory) - Nutch uses some kind 
>>> of legacy of the ancients for parsing.
>>>
>>> The error comes from both parsers available for html
>>>
>>> private DocumentFragment parse(InputSource input) throws Exception {
>>> if (parserImpl.equalsIgnoreCase("tagsoup"))
>>> return parseTagSoup(input);
>>> else
>>> return parseNeko(input);
>>> }
>>>  
>>> Neko and TagSoup both are dead for 4+ 
>>> years(https://en.wikipedia.org/wiki/Comparison_of_HTML_parsers#cite_note-1[https://en.wikipedia.org/wiki/Comparison_of_HTML_parsers#cite_note-1][https://en.wikipedia.org/wiki/Comparison_of_HTML_parsers#cite_note-1[https://en.wikipedia.org/wiki/Comparison_of_HTML_parsers#cite_note-1]][https://en.wikipedia.org/wiki/Comparison_of_HTML_parsers#cite_note-1[https://en.wikipedia.org/wiki/Comparison_of_HTML_parsers#cite_note-1][https://en.wikipedia.org/wiki/Comparison_of_HTML_parsers#cite_note-1[https://en.wikipedia.org/wiki/Comparison_of_HTML_parsers#cite_note-1]]][https://en.wikipedia.org/wiki/Comparison_of_HTML_parsers#cite_note-1[https://en.wikipedia.org/wiki/Comparison_of_HTML_parsers#cite_note-1][https://en.wikipedia.org/wiki/Comparison_of_HTML_parsers#cite_note-1[https://en.wikipedia.org/wiki/Comparison_of_HTML_parsers#cite_note-1]][https://en.wikipedia.org/wiki/Comparison_of_HTML_parsers#cite_note-1[https://en.wikipedia.org/wiki/Comparison_of_HTML_parsers#cite_note-1][https://en.wikipedia.org/wiki/Comparison_of_HTML_parsers#cite_note-1[https://en.wikipedia.org/wiki/Comparison_of_HTML_parsers#cite_note-1]]]]).
>>> If I try to parse it online with one of the modern plugin such as 
>>> https://jsoup.org/[https://jsoup.org/][https://jsoup.org/[https://jsoup.org/]][https://jsoup.org/[https://jsoup.org/][https://jsoup.org/[https://jsoup.org/]]][https://jsoup.org/[https://jsoup.org/][https://jsoup.org/[https://jsoup.org/]][https://jsoup.org/[https://jsoup.org/][https://jsoup.org/[https://jsoup.org/]]]]
>>>  it works fine.
>>>
>>> Very amazing considering the fact that it is THE core part of any parser.
>>>  
>>>
>>> Sent: Wednesday, November 14, 2018 at 3:32 PM
>>> From: "Semyon Semyonov" <semyon.semyo...@mail.com>
>>> To: user@nutch.apache.org
>>> Subject: Quality problems of crawling. Parsing(Missing attribute name), 
>>> fetching(empty body) and javascript.
>>> Hi everyone,
>>>
>>>
>>> We are testing the quality of our crawl for one of our domain countries 
>>> against the other public crawling tool( 
>>> http://tools.seochat.com/tools/online-crawl-google-sitemap-generator/#sthash.ZgdDhqwy.dpbs[http://tools.seochat.com/tools/online-crawl-google-sitemap-generator/#sthash.ZgdDhqwy.dpbs][http://tools.seochat.com/tools/online-crawl-google-sitemap-generator/#sthash.ZgdDhqwy.dpbs[http://tools.seochat.com/tools/online-crawl-google-sitemap-generator/#sthash.ZgdDhqwy.dpbs]][http://tools.seochat.com/tools/online-crawl-google-sitemap-generator/#sthash.ZgdDhqwy.dpbs[http://tools.seochat.com/tools/online-crawl-google-sitemap-generator/#sthash.ZgdDhqwy.dpbs][http://tools.seochat.com/tools/online-crawl-google-sitemap-generator/#sthash.ZgdDhqwy.dpbs[http://tools.seochat.com/tools/online-crawl-google-sitemap-generator/#sthash.ZgdDhqwy.dpbs]]][http://tools.seochat.com/tools/online-crawl-google-sitemap-generator/#sthash.ZgdDhqwy.dpbs[http://tools.seochat.com/tools/online-crawl-google-sitemap-generator/#sthash.ZgdDhqwy.dpbs][http://tools.seochat.com/tools/online-crawl-google-sitemap-generator/#sthash.ZgdDhqwy.dpbs[http://tools.seochat.com/tools/online-crawl-google-sitemap-generator/#sthash.ZgdDhqwy.dpbs]][http://tools.seochat.com/tools/online-crawl-google-sitemap-generator/#sthash.ZgdDhqwy.dpbs[http://tools.seochat.com/tools/online-crawl-google-sitemap-generator/#sthash.ZgdDhqwy.dpbs][http://tools.seochat.com/tools/online-crawl-google-sitemap-generator/#sthash.ZgdDhqwy.dpbs[http://tools.seochat.com/tools/online-crawl-google-sitemap-generator/#sthash.ZgdDhqwy.dpbs]]]]
>>>  ).
>>> All the webpages tested via both crawl script and the parsechecker tool for 
>>> both Tika and default HTML plugin. 
>>>  
>>> The results are not very good comparing to the tool, I would appreciate if 
>>> you give me a hint. 
>>>
>>>
>>> I classify several types of problems:
>>>  
>>> 1) Parsing problems.
>>>  
>>> http://www.vialucy.nl/[http://www.vialucy.nl/][http://www.vialucy.nl/[http://www.vialucy.nl/]][http://www.vialucy.nl/[http://www.vialucy.nl/][http://www.vialucy.nl/[http://www.vialucy.nl/]]][http://www.vialucy.nl/[http://www.vialucy.nl/][http://www.vialucy.nl/[http://www.vialucy.nl/]][http://www.vialucy.nl/[http://www.vialucy.nl/][http://www.vialucy.nl/[http://www.vialucy.nl/]]]][http://www.vialucy.nl/[http://www.vialucy.nl/][http://www.vialucy.nl/[http://www.vialucy.nl/]][http://www.vialucy.nl/[http://www.vialucy.nl/][http://www.vialucy.nl/[http://www.vialucy.nl/]]][http://www.vialucy.nl/[http://www.vialucy.nl/][http://www.vialucy.nl/[http://www.vialucy.nl/]][http://www.vialucy.nl/[http://www.vialucy.nl/][http://www.vialucy.nl/[http://www.vialucy.nl/]]]]]
>>> During the parsing I got a bunch of messages such as [Error] :4:23: Missing 
>>> attribute name and as a result I have an empty page back.   
>>>  
>>>  
>>> 2) Fetching problems 
>>>
>>> https://www.vishandelbunschoten.nl/[https://www.vishandelbunschoten.nl/][https://www.vishandelbunschoten.nl/[https://www.vishandelbunschoten.nl/]][https://www.vishandelbunschoten.nl/[https://www.vishandelbunschoten.nl/][https://www.vishandelbunschoten.nl/[https://www.vishandelbunschoten.nl/]]][https://www.vishandelbunschoten.nl/[https://www.vishandelbunschoten.nl/][https://www.vishandelbunschoten.nl/[https://www.vishandelbunschoten.nl/]][https://www.vishandelbunschoten.nl/[https://www.vishandelbunschoten.nl/][https://www.vishandelbunschoten.nl/[https://www.vishandelbunschoten.nl/]]]][https://www.vishandelbunschoten.nl/[https://www.vishandelbunschoten.nl/][https://www.vishandelbunschoten.nl/[https://www.vishandelbunschoten.nl/]][https://www.vishandelbunschoten.nl/[https://www.vishandelbunschoten.nl/][https://www.vishandelbunschoten.nl/[https://www.vishandelbunschoten.nl/]]][https://www.vishandelbunschoten.nl/[https://www.vishandelbunschoten.nl/][https://www.vishandelbunschoten.nl/[https://www.vishandelbunschoten.nl/]][https://www.vishandelbunschoten.nl/[https://www.vishandelbunschoten.nl/][https://www.vishandelbunschoten.nl/[https://www.vishandelbunschoten.nl/]]]]]
>>> Fetch returns HTTP/1.1 200 OK for header but empty body
>>>  
>>>  
>>> 3) Javascipt problems
>>>  
>>> http://www.amphar.com/Home.html[http://www.amphar.com/Home.html][http://www.amphar.com/Home.html[http://www.amphar.com/Home.html]][http://www.amphar.com/Home.html[http://www.amphar.com/Home.html][http://www.amphar.com/Home.html[http://www.amphar.com/Home.html]]][http://www.amphar.com/Home.html[http://www.amphar.com/Home.html][http://www.amphar.com/Home.html[http://www.amphar.com/Home.html]][http://www.amphar.com/Home.html[http://www.amphar.com/Home.html][http://www.amphar.com/Home.html[http://www.amphar.com/Home.html]]]][http://www.amphar.com/Home.html[http://www.amphar.com/Home.html][http://www.amphar.com/Home.html[http://www.amphar.com/Home.html]][http://www.amphar.com/Home.html[http://www.amphar.com/Home.html][http://www.amphar.com/Home.html[http://www.amphar.com/Home.html]]][http://www.amphar.com/Home.html[http://www.amphar.com/Home.html][http://www.amphar.com/Home.html[http://www.amphar.com/Home.html]][http://www.amphar.com/Home.html[http://www.amphar.com/Home.html][http://www.amphar.com/Home.html[http://www.amphar.com/Home.html]]]]]
>>>  
>>> Returns an empty body because of javasciprt
>>>  
>>>
>>> <?xml version="1.0" encoding="UTF-8"?><!DOCTYPE html PUBLIC "-//W3C//DTD 
>>> XHTML 1.0 Transitional//EN" 
>>> "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd[http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd][http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd[http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd]][http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd[http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd][http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd[http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd]]][http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd[http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd][http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd[http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd]][http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd[http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd][http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd[http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd]]]][http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd[http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd][http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd[http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd]][http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd[http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd][http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd[http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd]]][http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd[http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd][http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd[http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd]][http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd[http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd][http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd[http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd]]]]]";><html
>>>  xmlns="http://www.w3.org/1999/xhtml";><head><title></title><meta 
>>> http-equiv="refresh" content="0;url= Home.html" 
>>> /></head><body></body></html>
>>>  
>>> Another example ,
>>> https://www.sizo.com/[https://www.sizo.com/][https://www.sizo.com/[https://www.sizo.com/]][https://www.sizo.com/[https://www.sizo.com/][https://www.sizo.com/[https://www.sizo.com/]]][https://www.sizo.com/[https://www.sizo.com/][https://www.sizo.com/[https://www.sizo.com/]][https://www.sizo.com/[https://www.sizo.com/][https://www.sizo.com/[https://www.sizo.com/]]]][https://www.sizo.com/[https://www.sizo.com/][https://www.sizo.com/[https://www.sizo.com/]][https://www.sizo.com/[https://www.sizo.com/][https://www.sizo.com/[https://www.sizo.com/]]][https://www.sizo.com/[https://www.sizo.com/][https://www.sizo.com/[https://www.sizo.com/]][https://www.sizo.com/[https://www.sizo.com/][https://www.sizo.com/[https://www.sizo.com/]]]]]
>>>
>>> How to crawl these JavaScript websites? An activation of tika javascipt 
>>> doesnt help.
>>>
>>>
>>>
>>> Thanks.
>>>
>>> Semyon.
>>>
>>>  
>>>
>>  
>>
>  
> 

Reply via email to