[
https://issues.apache.org/jira/browse/NUTCH-2929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17472762#comment-17472762
]
ASF GitHub Bot commented on NUTCH-2929:
---------------------------------------
sebastian-nagel opened a new pull request #722:
URL: https://github.com/apache/nutch/pull/722
Sleep for a configurable delay (fetcher.threads.start.delay) before starting
the next Fetcher thread to avoid that resources (DNS, Tika XML parser pools)
are temporarily exhausted when Fetcher threads fetch the first pages
simultaneously
The default delay (10 milliseconds) practically does not impact the
behavior/performance of Fetcher - 100 threads are started per second. By
choosing a longer delay (500 milliseconds) all Tika warnings about the
SAXParser pool size disappeared.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscr...@nutch.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Fetcher: start threads slowly to avoid that resources are temporarily
> exhausted
> -------------------------------------------------------------------------------
>
> Key: NUTCH-2929
> URL: https://issues.apache.org/jira/browse/NUTCH-2929
> Project: Nutch
> Issue Type: Improvement
> Affects Versions: 1.18
> Reporter: Sebastian Nagel
> Priority: Minor
> Fix For: 1.19
>
>
> Fetcher spins all threads without any delay. This may cause that certain
> resources are temporarily exhausted if all threads start fetching the first
> pages simultaneously.
> The issue has been observed by Tika warnings about overuse of the SAXParser
> pool which appeared only during the first 2-5 minutes of fetching a segment.
> See https://lists.apache.org/thread/lo6b9wdlxy2lz12wmosldgl9x9ov1cks - adding
> a short delay between thread launches makes the warnings disappear.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
