GitHub user eivindveg opened a pull request:
https://github.com/apache/nutch/pull/50
[DO NOT MERGE/DISCUSSION] add cleaned up version of momer's
protocol-selenium plugin
Hi,
For some time, I have been using @momer's Selenium plugin for Nutch 2.3,
which has worked wonders in AJAX crawling. Lately however, I've noticed the
following issues with it:
1. It does not support HTTPS. While adding support for HTTPS into it, I
realized that:
2. It does a lot of non-Selenium stuff that makes it inefficient, including
making every request twice.
I've cleaned up the code(unfortunately, the update history pointing to him
is gone because of the lazy copy), making sure it uses Selenium and only
selenium.
So far it has the following weaknesses:
* Selenium's Wait does not appear to be properly used(also an issue in
@momer's code)
* It does not fill out the WebPage fields according to snuff. I do think
someone more experienced with Selenium could get this done, or I might add
support in the future.
Due to these issues, I think merging is a bit premature. However, I'd like
to point out that I do not think @momer's code should be merged either.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/BusinessMonitor/nutch 2.x-selenium
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/nutch/pull/50.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #50
----
commit dfc6214f281d9f39fbe911ff062a6a27b9ef932e
Author: Eivind Vegsundvåg <[email protected]>
Date: 2015-08-24T13:29:18Z
add cleaned up version of momer's protocol-selenium plugin
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
