[ https://issues.apache.org/jira/browse/NUTCH-2673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16679848#comment-16679848 ]
Sebastian Nagel commented on NUTCH-2673: ---------------------------------------- Hi [~markus17], I'm able to reproduce this issue with 1.14 - but with 1.15 and protocol-http it works: {noformat} % apache-nutch-1.14/bin/nutch parsechecker -Dplugin.includes='protocol-http|parse-tika' https://www.misdaadjournalist.nl/2018/11/politie-kraakt-server-van-blackbox-265-000-criminele-berichten-onderschept/ fetching: https://www.misdaadjournalist.nl/2018/11/politie-kraakt-server-van-blackbox-265-000-criminele-berichten-onderschept/ Fetch failed with protocol status: exception(16), lastModified=0: java.io.EOFException % apache-nutch-1.15/bin/nutch parsechecker -Dplugin.includes='protocol-http|parse-tika' https://www.misdaadjournalist.nl/2018/11/politie-kraakt-server-van-blackbox-265-000-criminele-berichten-onderschept/ fetching: https://www.misdaadjournalist.nl/2018/11/politie-kraakt-server-van-blackbox-265-000-criminele-berichten-onderschept/ parsing: https://www.misdaadjournalist.nl/2018/11/politie-kraakt-server-van-blackbox-265-000-criminele-berichten-onderschept/ contentType: application/octet-stream signature: 4a35b93aac1c2368b2c37a60375f8fb5 https://www.misdaadjournalist.nl/2018/11/politie-kraakt-server-van-blackbox-265-000-criminele-berichten-onderschept/ Version: 5 Status: failed(2,0): Can't retrieve Tika parser for mime-type application/octet-stream {noformat} There have been multiple fixes to the protocol-http plugin around NUTCH-2549. Interestingly, the new protocol-okhttp also fails: {noformat} % apache-nutch-1.15/bin/nutch parsechecker -Dplugin.includes='protocol-okhttp|parse-tika' https://www.misdaadjournalist.nl/2018/11/politie-kraakt-server-van-blackbox-265-000-criminele-berichten-onderschept/ fetching: https://www.misdaadjournalist.nl/2018/11/politie-kraakt-server-van-blackbox-265-000-criminele-berichten-onderschept/ Fetch failed with protocol status: exception(16), lastModified=0: java.io.IOException: unexpected end of stream on Connection{www.misdaadjournalist.nl:443, proxy=DIRECT hostAddress=www.misdaadjournalist.nl/84.22.108.7:443 cipherSuite=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 protocol=http/1.1} {noformat} Also wget and curl fail: {noformat} % wget https://www.misdaadjournalist.nl/2018/11/politie-kraakt-server-van-blackbox-265-000-criminele-berichten-onderschept/ ... HTTP request sent, awaiting response... No data received. Retrying. ... % curl https://www.misdaadjournalist.nl/2018/11/politie-kraakt-server-van-blackbox-265-000-criminele-berichten-onderschept/ curl: (92) HTTP/2 stream 1 was not closed cleanly: INTERNAL_ERROR (err 2) {noformat} So, the exception is either correct or does not really matter because empty content cannot be parsed anyway. > EOFException protocol-http > -------------------------- > > Key: NUTCH-2673 > URL: https://issues.apache.org/jira/browse/NUTCH-2673 > Project: Nutch > Issue Type: Bug > Affects Versions: 1.14 > Reporter: Markus Jelsma > Priority: Major > Fix For: 1.16 > > > Got an EOFException for some URL: > {code} > 2018-11-07 12:23:18,463 INFO indexer.IndexingFiltersChecker - fetching: > https://www.misdaadjournalist.nl/2018/11/politie-kraakt-server-van-blackbox-265-000-criminele-berichten-onderschept/ > 2018-11-07 12:23:18,704 INFO protocol.RobotRulesParser - robots.txt > whitelist not configured. > 2018-11-07 12:23:18,704 INFO http.Http - http.proxy.host = null > 2018-11-07 12:23:18,704 INFO http.Http - http.proxy.port = 8080 > 2018-11-07 12:23:18,704 INFO http.Http - http.proxy.exception.list = false > 2018-11-07 12:23:18,704 INFO http.Http - http.timeout = 30000 > 2018-11-07 12:23:18,704 INFO http.Http - http.content.limit = 32554432 > 2018-11-07 12:23:18,704 INFO http.Http - http.agent = Mozilla/5.0 > (compatible; OpenindexSpider; > +https://www.openindex.io/saas/about-our-spider/) > 2018-11-07 12:23:18,704 INFO http.Http - http.accept.language = > en-us,en-gb,en;q=0.7,*;q=0.3 > 2018-11-07 12:23:18,704 INFO http.Http - http.accept = > text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 > 2018-11-07 12:23:18,704 INFO http.Http - http.enable.cookie.header = false > 2018-11-07 12:23:18,911 ERROR http.Http - Failed to get protocol output > java.io.EOFException > at > org.apache.nutch.protocol.http.HttpResponse.readLine(HttpResponse.java:591) > at > org.apache.nutch.protocol.http.HttpResponse.parseStatusLine(HttpResponse.java:482) > at > org.apache.nutch.protocol.http.HttpResponse.<init>(HttpResponse.java:249) > at org.apache.nutch.protocol.http.Http.getResponse(Http.java:72) > at > org.apache.nutch.protocol.http.api.HttpBase.getProtocolOutput(HttpBase.java:276) > at > org.apache.nutch.indexer.IndexingFiltersChecker.getProtocolOutput(IndexingFiltersChecker.java:270) > at > org.apache.nutch.indexer.IndexingFiltersChecker.process(IndexingFiltersChecker.java:141) > at > org.apache.nutch.util.AbstractChecker.processSingle(AbstractChecker.java:86) > at > org.apache.nutch.indexer.IndexingFiltersChecker.run(IndexingFiltersChecker.java:111) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) > at > org.apache.nutch.indexer.IndexingFiltersChecker.main(IndexingFiltersChecker.java:275) > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)