[jira] [Commented] (NUTCH-2704) Upgrade crawler-commons dependency to 1.0

2019-04-12 Thread Hudson (JIRA) 


[ 
https://issues.apache.org/jira/browse/NUTCH-2704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16816202#comment-16816202
 ] 

Hudson commented on NUTCH-2704:
---

SUCCESS: Integrated in Jenkins build Nutch-trunk #3622 (See 
[https://builds.apache.org/job/Nutch-trunk/3622/])
NUTCH-2704 Upgrade crawler-commons dependency to 1.0 (snagel: 
[https://github.com/apache/nutch/commit/84cca6c97f43d82a4e7461fe0da20659ada6af54])
* (edit) ivy/ivy.xml


> Upgrade crawler-commons dependency to 1.0
> -
>
> Key: NUTCH-2704
> URL: https://issues.apache.org/jira/browse/NUTCH-2704
> Project: Nutch
>  Issue Type: Improvement
>Affects Versions: 1.15
>Reporter: Sebastian Nagel
>Assignee: Sebastian Nagel
>Priority: Minor
> Fix For: 1.16
>
>
> [Crawler-commons 
> 1.0|https://github.com/crawler-commons/crawler-commons/#21st-march-2018crawler-commons-10-released]
>  has been released. We should upgrade.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NUTCH-2699) Protocol-okhttp: needless loops to increment requested bytes counter when more content is already buffered

2019-04-12 Thread Hudson (JIRA) 


[ 
https://issues.apache.org/jira/browse/NUTCH-2699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16816201#comment-16816201
 ] 

Hudson commented on NUTCH-2699:
---

SUCCESS: Integrated in Jenkins build Nutch-trunk #3622 (See 
[https://builds.apache.org/job/Nutch-trunk/3622/])
NUTCH-2699 Protocol-okhttp: needless loops to increment requested bytes 
(snagel: 
[https://github.com/apache/nutch/commit/3ba0622bced082115918e5113d05bab65bf07754])
* (edit) 
src/plugin/protocol-okhttp/src/java/org/apache/nutch/protocol/okhttp/OkHttpResponse.java
NUTCH-2699 Protocol-okhttp: needless loops to increment requested bytes 
(snagel: 
[https://github.com/apache/nutch/commit/f51a276d619eca2a9dc3171340061f3789bf7440])
* (edit) 
src/plugin/protocol-okhttp/src/java/org/apache/nutch/protocol/okhttp/OkHttpResponse.java


> Protocol-okhttp: needless loops to increment requested bytes counter when 
> more content is already buffered
> --
>
> Key: NUTCH-2699
> URL: https://issues.apache.org/jira/browse/NUTCH-2699
> Project: Nutch
>  Issue Type: Bug
>  Components: protocol
>Affects Versions: 1.15
>Reporter: Sebastian Nagel
>Assignee: Sebastian Nagel
>Priority: Minor
> Fix For: 1.16
>
>
> The okhttp library used by the plugin protocol-okhttp buffers content 
> internal and often has already buffered more content than has been requested. 
> The plugin should immediately set the request count to the size of the 
> buffered content to avoid needless loops when the buffered size comes close 
> to the content limit (the increment steps are too small):
> {noformat}
> 2019-03-11 14:56:36,642 DEBUG okhttp.OkHttpResponse - 
> http://localhost/large.pdf - http/1.1 200 OK
> 2019-03-11 14:56:36,643 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 8192, buffered = 16088
> 2019-03-11 14:56:36,643 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 16384, buffered = 24280
> 2019-03-11 14:56:36,643 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 24576, buffered = 32472
> 2019-03-11 14:56:36,643 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 32768, buffered = 40664
> 2019-03-11 14:56:36,643 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 40960, buffered = 48856
> 2019-03-11 14:56:36,643 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 49152, buffered = 57048
> 2019-03-11 14:56:36,643 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 57344, buffered = 65240
> 2019-03-11 14:56:36,643 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 57638, buffered = 65240
> 2019-03-11 14:56:36,643 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 57932, buffered = 65240
> 2019-03-11 14:56:36,643 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 58226, buffered = 65240
> 2019-03-11 14:56:36,643 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 58520, buffered = 65240
> 2019-03-11 14:56:36,643 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 58814, buffered = 65240
> 2019-03-11 14:56:36,643 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 59108, buffered = 65240
> 2019-03-11 14:56:36,643 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 59402, buffered = 65240
> 2019-03-11 14:56:36,643 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 59696, buffered = 65240
> 2019-03-11 14:56:36,643 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 59990, buffered = 65240
> 2019-03-11 14:56:36,644 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 60284, buffered = 65240
> 2019-03-11 14:56:36,644 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 60578, buffered = 65240
> 2019-03-11 14:56:36,644 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 60872, buffered = 65240
> 2019-03-11 14:56:36,644 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 61166, buffered = 65240
> 2019-03-11 14:56:36,644 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 61460, buffered = 65240
> 2019-03-11 14:56:36,644 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 61754, buffered = 65240
> 2019-03-11 14:56:36,644 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 62048, buffered = 65240
> 2019-03-11 14:56:36,644 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 62342, buffered = 65240
> 2019-03-11 14:56:36,644 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 62636, buffered = 65240
> 2019-03-11 14:56:36,644 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 62930, buffered = 65240
> 2019-03-11 14:56:36,644 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 63224, buffered = 65240
> 2019-03-11 14:56:36,644 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 63518, buffered = 65240
> 2019-03-11 14:56:36,644 DEBUG okhttp.OkHttpRespon

Jenkins build is back to normal : Nutch-trunk #3622

2019-04-12 Thread Apache Jenkins Server 
See

[jira] [Commented] (NUTCH-2704) Upgrade crawler-commons dependency to 1.0

2019-04-12 Thread Markus Jelsma (JIRA) 


[ 
https://issues.apache.org/jira/browse/NUTCH-2704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16816191#comment-16816191
 ] 

Markus Jelsma commented on NUTCH-2704:
--

+1

> Upgrade crawler-commons dependency to 1.0
> -
>
> Key: NUTCH-2704
> URL: https://issues.apache.org/jira/browse/NUTCH-2704
> Project: Nutch
>  Issue Type: Improvement
>Affects Versions: 1.15
>Reporter: Sebastian Nagel
>Assignee: Sebastian Nagel
>Priority: Minor
> Fix For: 1.16
>
>
> [Crawler-commons 
> 1.0|https://github.com/crawler-commons/crawler-commons/#21st-march-2018crawler-commons-10-released]
>  has been released. We should upgrade.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (NUTCH-2704) Upgrade crawler-commons dependency to 1.0

2019-04-12 Thread Sebastian Nagel (JIRA) 


 [ 
https://issues.apache.org/jira/browse/NUTCH-2704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sebastian Nagel resolved NUTCH-2704.

Resolution: Implemented

> Upgrade crawler-commons dependency to 1.0
> -
>
> Key: NUTCH-2704
> URL: https://issues.apache.org/jira/browse/NUTCH-2704
> Project: Nutch
>  Issue Type: Improvement
>Affects Versions: 1.15
>Reporter: Sebastian Nagel
>Assignee: Sebastian Nagel
>Priority: Minor
> Fix For: 1.16
>
>
> [Crawler-commons 
> 1.0|https://github.com/crawler-commons/crawler-commons/#21st-march-2018crawler-commons-10-released]
>  has been released. We should upgrade.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (NUTCH-2704) Upgrade crawler-commons dependency to 1.0

2019-04-12 Thread Sebastian Nagel (JIRA) 


 [ 
https://issues.apache.org/jira/browse/NUTCH-2704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sebastian Nagel reassigned NUTCH-2704:
--

Assignee: Sebastian Nagel

> Upgrade crawler-commons dependency to 1.0
> -
>
> Key: NUTCH-2704
> URL: https://issues.apache.org/jira/browse/NUTCH-2704
> Project: Nutch
>  Issue Type: Improvement
>Affects Versions: 1.15
>Reporter: Sebastian Nagel
>Assignee: Sebastian Nagel
>Priority: Minor
> Fix For: 1.16
>
>
> [Crawler-commons 
> 1.0|https://github.com/crawler-commons/crawler-commons/#21st-march-2018crawler-commons-10-released]
>  has been released. We should upgrade.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NUTCH-2704) Upgrade crawler-commons dependency to 1.0

2019-04-12 Thread ASF GitHub Bot (JIRA) 


[ 
https://issues.apache.org/jira/browse/NUTCH-2704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16816189#comment-16816189
 ] 

ASF GitHub Bot commented on NUTCH-2704:
---

sebastian-nagel commented on pull request #448: NUTCH-2704 Upgrade 
crawler-commons dependency to 1.0
URL: https://github.com/apache/nutch/pull/448
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Upgrade crawler-commons dependency to 1.0
> -
>
> Key: NUTCH-2704
> URL: https://issues.apache.org/jira/browse/NUTCH-2704
> Project: Nutch
>  Issue Type: Improvement
>Affects Versions: 1.15
>Reporter: Sebastian Nagel
>Assignee: Sebastian Nagel
>Priority: Minor
> Fix For: 1.16
>
>
> [Crawler-commons 
> 1.0|https://github.com/crawler-commons/crawler-commons/#21st-march-2018crawler-commons-10-released]
>  has been released. We should upgrade.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (NUTCH-2699) Protocol-okhttp: needless loops to increment requested bytes counter when more content is already buffered

2019-04-12 Thread Sebastian Nagel (JIRA) 


 [ 
https://issues.apache.org/jira/browse/NUTCH-2699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sebastian Nagel reassigned NUTCH-2699:
--

Assignee: Sebastian Nagel

> Protocol-okhttp: needless loops to increment requested bytes counter when 
> more content is already buffered
> --
>
> Key: NUTCH-2699
> URL: https://issues.apache.org/jira/browse/NUTCH-2699
> Project: Nutch
>  Issue Type: Bug
>  Components: protocol
>Affects Versions: 1.15
>Reporter: Sebastian Nagel
>Assignee: Sebastian Nagel
>Priority: Minor
> Fix For: 1.16
>
>
> The okhttp library used by the plugin protocol-okhttp buffers content 
> internal and often has already buffered more content than has been requested. 
> The plugin should immediately set the request count to the size of the 
> buffered content to avoid needless loops when the buffered size comes close 
> to the content limit (the increment steps are too small):
> {noformat}
> 2019-03-11 14:56:36,642 DEBUG okhttp.OkHttpResponse - 
> http://localhost/large.pdf - http/1.1 200 OK
> 2019-03-11 14:56:36,643 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 8192, buffered = 16088
> 2019-03-11 14:56:36,643 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 16384, buffered = 24280
> 2019-03-11 14:56:36,643 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 24576, buffered = 32472
> 2019-03-11 14:56:36,643 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 32768, buffered = 40664
> 2019-03-11 14:56:36,643 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 40960, buffered = 48856
> 2019-03-11 14:56:36,643 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 49152, buffered = 57048
> 2019-03-11 14:56:36,643 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 57344, buffered = 65240
> 2019-03-11 14:56:36,643 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 57638, buffered = 65240
> 2019-03-11 14:56:36,643 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 57932, buffered = 65240
> 2019-03-11 14:56:36,643 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 58226, buffered = 65240
> 2019-03-11 14:56:36,643 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 58520, buffered = 65240
> 2019-03-11 14:56:36,643 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 58814, buffered = 65240
> 2019-03-11 14:56:36,643 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 59108, buffered = 65240
> 2019-03-11 14:56:36,643 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 59402, buffered = 65240
> 2019-03-11 14:56:36,643 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 59696, buffered = 65240
> 2019-03-11 14:56:36,643 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 59990, buffered = 65240
> 2019-03-11 14:56:36,644 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 60284, buffered = 65240
> 2019-03-11 14:56:36,644 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 60578, buffered = 65240
> 2019-03-11 14:56:36,644 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 60872, buffered = 65240
> 2019-03-11 14:56:36,644 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 61166, buffered = 65240
> 2019-03-11 14:56:36,644 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 61460, buffered = 65240
> 2019-03-11 14:56:36,644 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 61754, buffered = 65240
> 2019-03-11 14:56:36,644 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 62048, buffered = 65240
> 2019-03-11 14:56:36,644 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 62342, buffered = 65240
> 2019-03-11 14:56:36,644 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 62636, buffered = 65240
> 2019-03-11 14:56:36,644 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 62930, buffered = 65240
> 2019-03-11 14:56:36,644 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 63224, buffered = 65240
> 2019-03-11 14:56:36,644 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 63518, buffered = 65240
> 2019-03-11 14:56:36,644 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 63812, buffered = 65240
> 2019-03-11 14:56:36,644 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 64106, buffered = 65240
> 2019-03-11 14:56:36,644 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 64400, buffered = 65240
> 2019-03-11 14:56:36,644 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 64694, buffered = 65240
> 2019-03-11 14:56:36,644 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 64988, buffered = 65240
> 2019-03-11 14:56:36,644 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 65282, buffered = 73432
> 2019-03-11 14:56:36,644 DEBUG okhttp.OkHttpResponse -

[jira] [Commented] (NUTCH-2699) Protocol-okhttp: needless loops to increment requested bytes counter when more content is already buffered

2019-04-12 Thread ASF GitHub Bot (JIRA) 


[ 
https://issues.apache.org/jira/browse/NUTCH-2699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16816187#comment-16816187
 ] 

ASF GitHub Bot commented on NUTCH-2699:
---

sebastian-nagel commented on pull request #445: NUTCH-2699 Protocol-okhttp: 
needless loops to increment requested bytes counter when more content is 
already buffered
URL: https://github.com/apache/nutch/pull/445
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Protocol-okhttp: needless loops to increment requested bytes counter when 
> more content is already buffered
> --
>
> Key: NUTCH-2699
> URL: https://issues.apache.org/jira/browse/NUTCH-2699
> Project: Nutch
>  Issue Type: Bug
>  Components: protocol
>Affects Versions: 1.15
>Reporter: Sebastian Nagel
>Priority: Minor
> Fix For: 1.16
>
>
> The okhttp library used by the plugin protocol-okhttp buffers content 
> internal and often has already buffered more content than has been requested. 
> The plugin should immediately set the request count to the size of the 
> buffered content to avoid needless loops when the buffered size comes close 
> to the content limit (the increment steps are too small):
> {noformat}
> 2019-03-11 14:56:36,642 DEBUG okhttp.OkHttpResponse - 
> http://localhost/large.pdf - http/1.1 200 OK
> 2019-03-11 14:56:36,643 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 8192, buffered = 16088
> 2019-03-11 14:56:36,643 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 16384, buffered = 24280
> 2019-03-11 14:56:36,643 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 24576, buffered = 32472
> 2019-03-11 14:56:36,643 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 32768, buffered = 40664
> 2019-03-11 14:56:36,643 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 40960, buffered = 48856
> 2019-03-11 14:56:36,643 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 49152, buffered = 57048
> 2019-03-11 14:56:36,643 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 57344, buffered = 65240
> 2019-03-11 14:56:36,643 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 57638, buffered = 65240
> 2019-03-11 14:56:36,643 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 57932, buffered = 65240
> 2019-03-11 14:56:36,643 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 58226, buffered = 65240
> 2019-03-11 14:56:36,643 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 58520, buffered = 65240
> 2019-03-11 14:56:36,643 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 58814, buffered = 65240
> 2019-03-11 14:56:36,643 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 59108, buffered = 65240
> 2019-03-11 14:56:36,643 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 59402, buffered = 65240
> 2019-03-11 14:56:36,643 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 59696, buffered = 65240
> 2019-03-11 14:56:36,643 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 59990, buffered = 65240
> 2019-03-11 14:56:36,644 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 60284, buffered = 65240
> 2019-03-11 14:56:36,644 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 60578, buffered = 65240
> 2019-03-11 14:56:36,644 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 60872, buffered = 65240
> 2019-03-11 14:56:36,644 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 61166, buffered = 65240
> 2019-03-11 14:56:36,644 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 61460, buffered = 65240
> 2019-03-11 14:56:36,644 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 61754, buffered = 65240
> 2019-03-11 14:56:36,644 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 62048, buffered = 65240
> 2019-03-11 14:56:36,644 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 62342, buffered = 65240
> 2019-03-11 14:56:36,644 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 62636, buffered = 65240
> 2019-03-11 14:56:36,644 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 62930, buffered = 65240
> 2019-03-11 14:56:36,644 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 63224, buffered = 65240
> 2019-03-11 14:56:36,644 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 63518, buffered = 65240
> 2019-03-11 14:56:36,644 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 63812, buffered = 65240
> 2019-03-11 14:56:36,644 DEBUG okhttp.OkHttpResponse - total bytes

[jira] [Resolved] (NUTCH-2699) Protocol-okhttp: needless loops to increment requested bytes counter when more content is already buffered

2019-04-12 Thread Sebastian Nagel (JIRA) 


 [ 
https://issues.apache.org/jira/browse/NUTCH-2699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sebastian Nagel resolved NUTCH-2699.

Resolution: Fixed

> Protocol-okhttp: needless loops to increment requested bytes counter when 
> more content is already buffered
> --
>
> Key: NUTCH-2699
> URL: https://issues.apache.org/jira/browse/NUTCH-2699
> Project: Nutch
>  Issue Type: Bug
>  Components: protocol
>Affects Versions: 1.15
>Reporter: Sebastian Nagel
>Assignee: Sebastian Nagel
>Priority: Minor
> Fix For: 1.16
>
>
> The okhttp library used by the plugin protocol-okhttp buffers content 
> internal and often has already buffered more content than has been requested. 
> The plugin should immediately set the request count to the size of the 
> buffered content to avoid needless loops when the buffered size comes close 
> to the content limit (the increment steps are too small):
> {noformat}
> 2019-03-11 14:56:36,642 DEBUG okhttp.OkHttpResponse - 
> http://localhost/large.pdf - http/1.1 200 OK
> 2019-03-11 14:56:36,643 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 8192, buffered = 16088
> 2019-03-11 14:56:36,643 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 16384, buffered = 24280
> 2019-03-11 14:56:36,643 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 24576, buffered = 32472
> 2019-03-11 14:56:36,643 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 32768, buffered = 40664
> 2019-03-11 14:56:36,643 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 40960, buffered = 48856
> 2019-03-11 14:56:36,643 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 49152, buffered = 57048
> 2019-03-11 14:56:36,643 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 57344, buffered = 65240
> 2019-03-11 14:56:36,643 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 57638, buffered = 65240
> 2019-03-11 14:56:36,643 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 57932, buffered = 65240
> 2019-03-11 14:56:36,643 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 58226, buffered = 65240
> 2019-03-11 14:56:36,643 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 58520, buffered = 65240
> 2019-03-11 14:56:36,643 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 58814, buffered = 65240
> 2019-03-11 14:56:36,643 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 59108, buffered = 65240
> 2019-03-11 14:56:36,643 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 59402, buffered = 65240
> 2019-03-11 14:56:36,643 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 59696, buffered = 65240
> 2019-03-11 14:56:36,643 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 59990, buffered = 65240
> 2019-03-11 14:56:36,644 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 60284, buffered = 65240
> 2019-03-11 14:56:36,644 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 60578, buffered = 65240
> 2019-03-11 14:56:36,644 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 60872, buffered = 65240
> 2019-03-11 14:56:36,644 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 61166, buffered = 65240
> 2019-03-11 14:56:36,644 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 61460, buffered = 65240
> 2019-03-11 14:56:36,644 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 61754, buffered = 65240
> 2019-03-11 14:56:36,644 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 62048, buffered = 65240
> 2019-03-11 14:56:36,644 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 62342, buffered = 65240
> 2019-03-11 14:56:36,644 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 62636, buffered = 65240
> 2019-03-11 14:56:36,644 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 62930, buffered = 65240
> 2019-03-11 14:56:36,644 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 63224, buffered = 65240
> 2019-03-11 14:56:36,644 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 63518, buffered = 65240
> 2019-03-11 14:56:36,644 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 63812, buffered = 65240
> 2019-03-11 14:56:36,644 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 64106, buffered = 65240
> 2019-03-11 14:56:36,644 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 64400, buffered = 65240
> 2019-03-11 14:56:36,644 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 64694, buffered = 65240
> 2019-03-11 14:56:36,644 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 64988, buffered = 65240
> 2019-03-11 14:56:36,644 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 65282, buffered = 73432
> 2019-03-11 14:56:36,644 DEBUG okhttp.OkHttpResponse - content limi

[jira] [Work started] (NUTCH-2699) Protocol-okhttp: needless loops to increment requested bytes counter when more content is already buffered

2019-04-12 Thread Sebastian Nagel (JIRA) 


 [ 
https://issues.apache.org/jira/browse/NUTCH-2699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on NUTCH-2699 started by Sebastian Nagel.
--
> Protocol-okhttp: needless loops to increment requested bytes counter when 
> more content is already buffered
> --
>
> Key: NUTCH-2699
> URL: https://issues.apache.org/jira/browse/NUTCH-2699
> Project: Nutch
>  Issue Type: Bug
>  Components: protocol
>Affects Versions: 1.15
>Reporter: Sebastian Nagel
>Assignee: Sebastian Nagel
>Priority: Minor
> Fix For: 1.16
>
>
> The okhttp library used by the plugin protocol-okhttp buffers content 
> internal and often has already buffered more content than has been requested. 
> The plugin should immediately set the request count to the size of the 
> buffered content to avoid needless loops when the buffered size comes close 
> to the content limit (the increment steps are too small):
> {noformat}
> 2019-03-11 14:56:36,642 DEBUG okhttp.OkHttpResponse - 
> http://localhost/large.pdf - http/1.1 200 OK
> 2019-03-11 14:56:36,643 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 8192, buffered = 16088
> 2019-03-11 14:56:36,643 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 16384, buffered = 24280
> 2019-03-11 14:56:36,643 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 24576, buffered = 32472
> 2019-03-11 14:56:36,643 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 32768, buffered = 40664
> 2019-03-11 14:56:36,643 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 40960, buffered = 48856
> 2019-03-11 14:56:36,643 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 49152, buffered = 57048
> 2019-03-11 14:56:36,643 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 57344, buffered = 65240
> 2019-03-11 14:56:36,643 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 57638, buffered = 65240
> 2019-03-11 14:56:36,643 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 57932, buffered = 65240
> 2019-03-11 14:56:36,643 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 58226, buffered = 65240
> 2019-03-11 14:56:36,643 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 58520, buffered = 65240
> 2019-03-11 14:56:36,643 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 58814, buffered = 65240
> 2019-03-11 14:56:36,643 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 59108, buffered = 65240
> 2019-03-11 14:56:36,643 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 59402, buffered = 65240
> 2019-03-11 14:56:36,643 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 59696, buffered = 65240
> 2019-03-11 14:56:36,643 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 59990, buffered = 65240
> 2019-03-11 14:56:36,644 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 60284, buffered = 65240
> 2019-03-11 14:56:36,644 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 60578, buffered = 65240
> 2019-03-11 14:56:36,644 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 60872, buffered = 65240
> 2019-03-11 14:56:36,644 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 61166, buffered = 65240
> 2019-03-11 14:56:36,644 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 61460, buffered = 65240
> 2019-03-11 14:56:36,644 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 61754, buffered = 65240
> 2019-03-11 14:56:36,644 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 62048, buffered = 65240
> 2019-03-11 14:56:36,644 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 62342, buffered = 65240
> 2019-03-11 14:56:36,644 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 62636, buffered = 65240
> 2019-03-11 14:56:36,644 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 62930, buffered = 65240
> 2019-03-11 14:56:36,644 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 63224, buffered = 65240
> 2019-03-11 14:56:36,644 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 63518, buffered = 65240
> 2019-03-11 14:56:36,644 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 63812, buffered = 65240
> 2019-03-11 14:56:36,644 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 64106, buffered = 65240
> 2019-03-11 14:56:36,644 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 64400, buffered = 65240
> 2019-03-11 14:56:36,644 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 64694, buffered = 65240
> 2019-03-11 14:56:36,644 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 64988, buffered = 65240
> 2019-03-11 14:56:36,644 DEBUG okhttp.OkHttpResponse - total bytes requested = 
> 65282, buffered = 73432
> 2019-03-11 14:56:36,644 DEBUG okhttp.OkHttpResponse - content limit r