[
https://issues.apache.org/jira/browse/NUTCH-3018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17781483#comment-17781483
]
Tim Allison edited comment on NUTCH-3018 at 10/31/23 6:46 PM:
--------------------------------------------------------------
It looks like we cannot create more web drivers than the number of worker nodes
X {{SE_NODE_MAX_SESSIONS}}.
I think it would still be useful to reuse the webdriver(s) if we can. We could
reconnect on exception, etc...
This may be a horribly misguided approach. Let me know. :D
was (Author: talli...@mitre.org):
It looks like we cannot create more web drivers than the
{{SE_NODE_MAX_SESSIONS}} which defaults to 1.
I think it would still be useful to reuse the webdriver(s) if we can. We could
reconnect on exception, etc...
This may be a horribly misguided approach. Let me know. :D
> Consider pooling remote webdrivers for Selenium?
> ------------------------------------------------
>
> Key: NUTCH-3018
> URL: https://issues.apache.org/jira/browse/NUTCH-3018
> Project: Nutch
> Issue Type: Task
> Reporter: Tim Allison
> Priority: Minor
>
> It looks like it takes between 2x and 4x of the time to initialize the remote
> webdriver in selenium than it does to render/fetch a couple of test pages I'm
> working with.
> On a mac with a chrome driver, ~1.5 seconds to load the driver and then .5 of
> a second to fetch/render the page. On a mac, ~1.2 seconds to load and then .5
> of a second to fetch/render.
> On a mac with firefox driver, ~3.7 seconds to load the driver and ~1 second
> to fetch/render a page.
> Is it worth pooling webdrivers or does that add too much complexity/overhead?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
