[jira] [Commented] (NUTCH-2310) Protocol-Selenium does not support HTTPS protocol
[
https://issues.apache.org/jira/browse/NUTCH-2310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16385853#comment-16385853
]
Sebastian Nagel commented on NUTCH-2310:
The
[plugin.xml|https://github.com/apache/nutch/blob/master/src/plugin/protocol-selenium/plugin.xml]
must also list https as supported protocol. That's done by adding:
{noformat}
{noformat}
But it's likely that more points need to be done to support https.
> Protocol-Selenium does not support HTTPS protocol
> -
>
> Key: NUTCH-2310
> URL: https://issues.apache.org/jira/browse/NUTCH-2310
> Project: Nutch
> Issue Type: Bug
> Components: protocol
>Affects Versions: 1.12
>Reporter: Joey Hong
>Priority: Major
> Labels: easyfix
> Fix For: 1.15
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> The protocol-selenium and protocol-interactiveselenium plugins raise errors
> whenever there is a URL with the HTTPS protocol.
> From the source code for those plugins, we can see that HTTP is the only
> scheme currently accepted, which makes Nutch unable to crawl HTTPS sites with
> JS using Selenium Webdrivers.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NUTCH-2310) Protocol-Selenium does not support HTTPS protocol
[ https://issues.apache.org/jira/browse/NUTCH-2310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16375055#comment-16375055 ] Irinel commented on NUTCH-2310: --- Hi, I think we need to change more things, not just HttpResponse.java. Now the error is on ProtocolFactory: -activeThreads=1, spinWaiting=0, fetchQueues.totalSize=0, fetchQueues.getQueueCount=1 FetcherThread 41 fetching https://www.***.com (queue crawl delay=2000ms) FetcherThread 41 fetch of https://www.***.com failed with: org.apache.nutch.protocol.ProtocolNotFound: protocol not found for url=https atorg.apache.nutch.protocol.ProtocolFactory.getProtocol(ProtocolFactory.java:85) at org.apache.nutch.fetcher.FetcherThread.run(FetcherThread.java:300) > Protocol-Selenium does not support HTTPS protocol > - > > Key: NUTCH-2310 > URL: https://issues.apache.org/jira/browse/NUTCH-2310 > Project: Nutch > Issue Type: Bug > Components: protocol >Affects Versions: 1.12 >Reporter: Joey Hong >Priority: Major > Labels: easyfix > Fix For: 1.15 > > Original Estimate: 48h > Remaining Estimate: 48h > > The protocol-selenium and protocol-interactiveselenium plugins raise errors > whenever there is a URL with the HTTPS protocol. > From the source code for those plugins, we can see that HTTP is the only > scheme currently accepted, which makes Nutch unable to crawl HTTPS sites with > JS using Selenium Webdrivers. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NUTCH-2310) Protocol-Selenium does not support HTTPS protocol
[
https://issues.apache.org/jira/browse/NUTCH-2310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16142692#comment-16142692
]
Sebastian Nagel commented on NUTCH-2310:
Hi, ev. it helps to remove from
[HttpResponse.java|https://github.com/apache/nutch/blob/master/src/plugin/protocol-selenium/src/java/org/apache/nutch/protocol/selenium/HttpResponse.java#L63]
the lines
{code}
if (!"http".equals(url.getProtocol()))
throw new HttpException("Not an HTTP url:" + url);
{code}
Could try it, rebuild Nutch and report whether it works? Thanks!
> Protocol-Selenium does not support HTTPS protocol
> -
>
> Key: NUTCH-2310
> URL: https://issues.apache.org/jira/browse/NUTCH-2310
> Project: Nutch
> Issue Type: Bug
> Components: protocol
>Affects Versions: 1.12
>Reporter: Joey Hong
> Labels: easyfix
> Fix For: 1.14
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> The protocol-selenium and protocol-interactiveselenium plugins raise errors
> whenever there is a URL with the HTTPS protocol.
> From the source code for those plugins, we can see that HTTP is the only
> scheme currently accepted, which makes Nutch unable to crawl HTTPS sites with
> JS using Selenium Webdrivers.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (NUTCH-2310) Protocol-Selenium does not support HTTPS protocol
[ https://issues.apache.org/jira/browse/NUTCH-2310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16141837#comment-16141837 ] Antoine DELMOTTE commented on NUTCH-2310: - Any news on this issue yet ? Facing the same problem atm. > Protocol-Selenium does not support HTTPS protocol > - > > Key: NUTCH-2310 > URL: https://issues.apache.org/jira/browse/NUTCH-2310 > Project: Nutch > Issue Type: Bug > Components: protocol >Affects Versions: 1.12 >Reporter: Joey Hong > Labels: easyfix > Fix For: 1.14 > > Original Estimate: 48h > Remaining Estimate: 48h > > The protocol-selenium and protocol-interactiveselenium plugins raise errors > whenever there is a URL with the HTTPS protocol. > From the source code for those plugins, we can see that HTTP is the only > scheme currently accepted, which makes Nutch unable to crawl HTTPS sites with > JS using Selenium Webdrivers. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (NUTCH-2310) Protocol-Selenium does not support HTTPS protocol
[ https://issues.apache.org/jira/browse/NUTCH-2310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15437787#comment-15437787 ] Joey Hong commented on NUTCH-2310: -- It is a pretty simple fix from comparing with the other protocol-* plugins. I will have those changes committed ASAP. > Protocol-Selenium does not support HTTPS protocol > - > > Key: NUTCH-2310 > URL: https://issues.apache.org/jira/browse/NUTCH-2310 > Project: Nutch > Issue Type: Bug > Components: protocol >Affects Versions: 1.12 >Reporter: Joey Hong > Labels: easyfix > Fix For: 1.13 > > Original Estimate: 48h > Remaining Estimate: 48h > > The protocol-selenium and protocol-interactiveselenium plugins raise errors > whenever there is a URL with the HTTPS protocol. > From the source code for those plugins, we can see that HTTP is the only > scheme currently accepted, which makes Nutch unable to crawl HTTPS sites with > JS using Selenium Webdrivers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
