[
https://issues.apache.org/jira/browse/NUTCH-2896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17423276#comment-17423276
]
ASF GitHub Bot commented on NUTCH-2896:
---------------------------------------
sebastian-nagel commented on pull request #697:
URL: https://github.com/apache/nutch/pull/697#issuecomment-932208067
@lewismc: yes, I'll add first a page about HTTP protocol plugins to the
wiki. It should include a feature matrix what's supported - authentication,
http/2, proxies, connection pool, etc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscr...@nutch.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Protocol-okhttp: make connection pool configurable
> --------------------------------------------------
>
> Key: NUTCH-2896
> URL: https://issues.apache.org/jira/browse/NUTCH-2896
> Project: Nutch
> Issue Type: Improvement
> Components: plugin, protocol
> Affects Versions: 1.18
> Reporter: Sebastian Nagel
> Priority: Major
> Fix For: 1.19
>
>
> [OkHttp's
> ConnectionPool|https://square.github.io/okhttp/3.x/okhttp/okhttp3/ConnectionPool.html]
> "holds up to 5 idle connections which will be evicted after 5 minutes of
> inactivity." A pool of this size is suitable for site crawls but not for
> larger crawls over many different sites / hosts.
> Note: in the current version (4.9.1) the connection pool is implemented as a
> [linked
> queue|https://github.com/square/okhttp/blob/parent-4.9.1/okhttp/src/main/kotlin/okhttp3/internal/connection/RealConnectionPool.kt#L83].
> In order to scale beyond pool sizes exceeding 1000 we need to use a set of
> clients each with its own connection pool.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
