[
https://issues.apache.org/jira/browse/NUTCH-2673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16679848#comment-16679848
]
Sebastian Nagel commented on NUTCH-2673:
----------------------------------------
Hi [~markus17], I'm able to reproduce this issue with 1.14 - but with 1.15 and
protocol-http it works:
{noformat}
% apache-nutch-1.14/bin/nutch parsechecker
-Dplugin.includes='protocol-http|parse-tika'
https://www.misdaadjournalist.nl/2018/11/politie-kraakt-server-van-blackbox-265-000-criminele-berichten-onderschept/
fetching:
https://www.misdaadjournalist.nl/2018/11/politie-kraakt-server-van-blackbox-265-000-criminele-berichten-onderschept/
Fetch failed with protocol status: exception(16), lastModified=0:
java.io.EOFException
% apache-nutch-1.15/bin/nutch parsechecker
-Dplugin.includes='protocol-http|parse-tika'
https://www.misdaadjournalist.nl/2018/11/politie-kraakt-server-van-blackbox-265-000-criminele-berichten-onderschept/
fetching:
https://www.misdaadjournalist.nl/2018/11/politie-kraakt-server-van-blackbox-265-000-criminele-berichten-onderschept/
parsing:
https://www.misdaadjournalist.nl/2018/11/politie-kraakt-server-van-blackbox-265-000-criminele-berichten-onderschept/
contentType: application/octet-stream
signature: 4a35b93aac1c2368b2c37a60375f8fb5
https://www.misdaadjournalist.nl/2018/11/politie-kraakt-server-van-blackbox-265-000-criminele-berichten-onderschept/
Version: 5
Status: failed(2,0): Can't retrieve Tika parser for mime-type
application/octet-stream
{noformat}
There have been multiple fixes to the protocol-http plugin around NUTCH-2549.
Interestingly, the new protocol-okhttp also fails:
{noformat}
% apache-nutch-1.15/bin/nutch parsechecker
-Dplugin.includes='protocol-okhttp|parse-tika'
https://www.misdaadjournalist.nl/2018/11/politie-kraakt-server-van-blackbox-265-000-criminele-berichten-onderschept/
fetching:
https://www.misdaadjournalist.nl/2018/11/politie-kraakt-server-van-blackbox-265-000-criminele-berichten-onderschept/
Fetch failed with protocol status: exception(16), lastModified=0:
java.io.IOException: unexpected end of stream on
Connection{www.misdaadjournalist.nl:443, proxy=DIRECT
hostAddress=www.misdaadjournalist.nl/84.22.108.7:443
cipherSuite=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 protocol=http/1.1}
{noformat}
Also wget and curl fail:
{noformat}
% wget
https://www.misdaadjournalist.nl/2018/11/politie-kraakt-server-van-blackbox-265-000-criminele-berichten-onderschept/
...
HTTP request sent, awaiting response... No data received.
Retrying.
...
% curl
https://www.misdaadjournalist.nl/2018/11/politie-kraakt-server-van-blackbox-265-000-criminele-berichten-onderschept/
curl: (92) HTTP/2 stream 1 was not closed cleanly: INTERNAL_ERROR (err 2)
{noformat}
So, the exception is either correct or does not really matter because empty
content cannot be parsed anyway.
> EOFException protocol-http
> --------------------------
>
> Key: NUTCH-2673
> URL: https://issues.apache.org/jira/browse/NUTCH-2673
> Project: Nutch
> Issue Type: Bug
> Affects Versions: 1.14
> Reporter: Markus Jelsma
> Priority: Major
> Fix For: 1.16
>
>
> Got an EOFException for some URL:
> {code}
> 2018-11-07 12:23:18,463 INFO indexer.IndexingFiltersChecker - fetching:
> https://www.misdaadjournalist.nl/2018/11/politie-kraakt-server-van-blackbox-265-000-criminele-berichten-onderschept/
> 2018-11-07 12:23:18,704 INFO protocol.RobotRulesParser - robots.txt
> whitelist not configured.
> 2018-11-07 12:23:18,704 INFO http.Http - http.proxy.host = null
> 2018-11-07 12:23:18,704 INFO http.Http - http.proxy.port = 8080
> 2018-11-07 12:23:18,704 INFO http.Http - http.proxy.exception.list = false
> 2018-11-07 12:23:18,704 INFO http.Http - http.timeout = 30000
> 2018-11-07 12:23:18,704 INFO http.Http - http.content.limit = 32554432
> 2018-11-07 12:23:18,704 INFO http.Http - http.agent = Mozilla/5.0
> (compatible; OpenindexSpider;
> +https://www.openindex.io/saas/about-our-spider/)
> 2018-11-07 12:23:18,704 INFO http.Http - http.accept.language =
> en-us,en-gb,en;q=0.7,*;q=0.3
> 2018-11-07 12:23:18,704 INFO http.Http - http.accept =
> text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
> 2018-11-07 12:23:18,704 INFO http.Http - http.enable.cookie.header = false
> 2018-11-07 12:23:18,911 ERROR http.Http - Failed to get protocol output
> java.io.EOFException
> at
> org.apache.nutch.protocol.http.HttpResponse.readLine(HttpResponse.java:591)
> at
> org.apache.nutch.protocol.http.HttpResponse.parseStatusLine(HttpResponse.java:482)
> at
> org.apache.nutch.protocol.http.HttpResponse.<init>(HttpResponse.java:249)
> at org.apache.nutch.protocol.http.Http.getResponse(Http.java:72)
> at
> org.apache.nutch.protocol.http.api.HttpBase.getProtocolOutput(HttpBase.java:276)
> at
> org.apache.nutch.indexer.IndexingFiltersChecker.getProtocolOutput(IndexingFiltersChecker.java:270)
> at
> org.apache.nutch.indexer.IndexingFiltersChecker.process(IndexingFiltersChecker.java:141)
> at
> org.apache.nutch.util.AbstractChecker.processSingle(AbstractChecker.java:86)
> at
> org.apache.nutch.indexer.IndexingFiltersChecker.run(IndexingFiltersChecker.java:111)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at
> org.apache.nutch.indexer.IndexingFiltersChecker.main(IndexingFiltersChecker.java:275)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
