[
https://issues.apache.org/jira/browse/NUTCH-2896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17418130#comment-17418130
]
ASF GitHub Bot commented on NUTCH-2896:
---------------------------------------
sebastian-nagel opened a new pull request #697:
URL: https://github.com/apache/nutch/pull/697
- add configuration property `http.connection.pool.okhttp` to configure the
number of connection pools, their size and the keep-alive time of the pooled
connections
- create as many clients as pools are configured, each client holding one
pool
- distribute connections by target host name over clients
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
> Protocol-okhttp: make connection pool configurable
> --------------------------------------------------
>
> Key: NUTCH-2896
> URL: https://issues.apache.org/jira/browse/NUTCH-2896
> Project: Nutch
> Issue Type: Improvement
> Components: plugin, protocol
> Affects Versions: 1.18
> Reporter: Sebastian Nagel
> Priority: Major
> Fix For: 1.19
>
>
> [OkHttp's
> ConnectionPool|https://square.github.io/okhttp/3.x/okhttp/okhttp3/ConnectionPool.html]
> "holds up to 5 idle connections which will be evicted after 5 minutes of
> inactivity." A pool of this size is suitable for site crawls but not for
> larger crawls over many different sites / hosts.
> Note: in the current version (4.9.1) the connection pool is implemented as a
> [linked
> queue|https://github.com/square/okhttp/blob/parent-4.9.1/okhttp/src/main/kotlin/okhttp3/internal/connection/RealConnectionPool.kt#L83].
> In order to scale beyond pool sizes exceeding 1000 we need to use a set of
> clients each with its own connection pool.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
