[
https://issues.apache.org/jira/browse/NUTCH-3020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17781925#comment-17781925
]
ASF GitHub Bot commented on NUTCH-3020:
---------------------------------------
lewismc commented on PR #794:
URL: https://github.com/apache/nutch/pull/794#issuecomment-1789810071
We have no tests for `ParseSegment` right now. I think it would be excellent
if this PR could include a test for `ParseSegment.isTruncated`.
> ParseSegment should check for protocol's flags for truncation
> -------------------------------------------------------------
>
> Key: NUTCH-3020
> URL: https://issues.apache.org/jira/browse/NUTCH-3020
> Project: Nutch
> Issue Type: Task
> Reporter: Tim Allison
> Priority: Minor
>
> As discussed on the user list, several protocols can identify when a fetch
> has been truncated. ParseSegment only checks for the number of bytes fetched
> vs the http length header (if it exists). We should modify ParseSegment to
> check for notification of truncation from the protocols.
> I noticed this specifically with okhttp, but other protocols may flag
> truncation as well.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
