This is an automated email from the ASF dual-hosted git repository.
rzo1 pushed a change to branch storm-3.0.0-upgrade
in repository https://gitbox.apache.org/repos/asf/stormcrawler.git
omit 93a7e4b8 Upgrade to Apache Storm 3.0.0 and Java 25
add e87846ac Mark Selenium module as deprecated (#2019)
add 5eb99d0a Mark CloudSearch as deprecated. Fixes #2016 (#2018)
add 54c4a6d3 Allow indexer bolts to use a metadata value as the document
ID (#2012)
add 3c06a126 Mark HTTPClient as deprecated (#2020)
add 8667217f build(deps): bump org.apache.maven.plugins:maven-jar-plugin
(#2025)
add ee983434 build(deps): bump tika.version from 3.3.1 to 3.3.2 (#2023)
add f1e35b21 build(deps): bump software.amazon.awssdk:bom from 2.48.3 to
2.49.3 (#2022)
add cd5f59c5 Minor: Regenerated License File for
ee983434cdd00cfbf44d0c6b3d79f124942bf682 (#2028)
add b13ed812 Minor: Regenerated License File for
f1e35b21261fc43b6990bae7f633c071b7947d57 (#2029)
add 725781f1 Minor: Regenerated License File for
cd5f59c51bdae43202c2c0f408a98fe01b99ae83 (#2030)
add d193e6cd #2026 - http.agent.name is dropped from the robots.txt match
set when http.robots.agents is set (#2027)
add dc1f70bb fix: emit anchored tuples #2021 (#2031)
add 9304adac Enforce robots.txt Crawl-delay in URLFrontier queues (#2000)
add 6e690830 Upgrade to Storm 2.8.9 (#2032)
add e3297e48 Regenerated License file after dependency upgrades (#2033)
add 0dc2ab5e [maven-release-plugin] prepare release stormcrawler-3.7.0
add 883b8546 [maven-release-plugin] prepare for next development iteration
add 214b23e1 Fix typo in OpenSearch configuration comment
add 747f5d0a Fix broken link to Guava's CacheBuilderSpec
add 8f94a75f Fix broken anchor in Asciidoc
add 6aba5a5b Upgrade to Apache Storm 3.0.0 and Java 25
add 6976bc9a Move to StormCrawler 4
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (93a7e4b8)
\
N -- N -- N refs/heads/storm-3.0.0-upgrade (6976bc9a)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
THIRD-PARTY.txt | 168 +++++++--------
archetype/pom.xml | 2 +-
core/pom.xml | 2 +-
.../java/org/apache/stormcrawler/Constants.java | 7 +
.../org/apache/stormcrawler/bolt/FetcherBolt.java | 17 ++
.../stormcrawler/bolt/SimpleFetcherBolt.java | 16 +-
.../apache/stormcrawler/bolt/URLFilterBolt.java | 4 +-
.../stormcrawler/indexing/AbstractIndexerBolt.java | 48 ++++-
.../persistence/AbstractStatusUpdaterBolt.java | 6 +-
.../stormcrawler/protocol/DelegatorProtocol.java | 4 +-
.../stormcrawler/protocol/ProtocolFactory.java | 6 +-
.../stormcrawler/protocol/RobotRulesParser.java | 15 +-
.../protocol/httpclient/HttpProtocol.java | 9 +-
.../stormcrawler/proxy/MultiProxyManager.java | 3 +-
core/src/main/resources/crawler-default.yaml | 15 +-
.../stormcrawler/bolt/AbstractFetcherBoltTest.java | 54 ++++-
.../apache/stormcrawler/bolt/FetcherBoltTest.java | 113 ++++++++++
.../stormcrawler/bolt/SimpleFetcherBoltTest.java | 106 ++++++++++
.../stormcrawler/bolt/URLFilterBoltTest.java | 134 ++++++++++++
.../stormcrawler/indexer/BasicIndexingTest.java | 39 ++++
.../apache/stormcrawler/indexer/DummyIndexer.java | 4 +
.../protocol/RobotRulesParserAgentNamesTest.java | 94 +++++++++
.../HttpClientProtocolProxyManagerTest.java | 1 +
core/src/test/resources/test.urlfilterbolt.json | 11 +
docs/pom.xml | 2 +-
docs/src/main/asciidoc/configuration.adoc | 44 +++-
docs/src/main/asciidoc/extending.adoc | 2 +-
docs/src/main/asciidoc/internals.adoc | 8 +-
external/ai/pom.xml | 2 +-
external/aws/README.md | 4 +-
external/aws/aws-conf.yaml | 3 +
external/aws/pom.xml | 4 +-
.../aws/bolt/CloudSearchConstants.java | 5 +
.../aws/bolt/CloudSearchIndexerBolt.java | 12 +-
.../stormcrawler/aws/bolt/CloudSearchUtils.java | 5 +
external/langid/pom.xml | 2 +-
external/opensearch-java/pom.xml | 2 +-
.../stormcrawler/opensearch/bolt/DeletionBolt.java | 24 ++-
external/opensearch/archetype/pom.xml | 2 +-
.../archetype-resources/opensearch-conf.yaml | 2 +-
external/opensearch/opensearch-conf.yaml | 2 +-
external/opensearch/pom.xml | 2 +-
.../stormcrawler/opensearch/bolt/DeletionBolt.java | 24 ++-
external/playwright/pom.xml | 2 +-
external/pom.xml | 2 +-
external/selenium/README.md | 2 +
external/selenium/pom.xml | 4 +-
.../protocol/selenium/NavigationFilter.java | 5 +
.../protocol/selenium/NavigationFilters.java | 3 +
.../protocol/selenium/RemoteDriverProtocol.java | 4 +
.../protocol/selenium/SeleniumProtocol.java | 5 +
external/solr/archetype/pom.xml | 2 +-
external/solr/pom.xml | 2 +-
external/sql/pom.xml | 2 +-
external/tika/pom.xml | 2 +-
external/urlfrontier/README.md | 61 ++++++
external/urlfrontier/pom.xml | 2 +-
.../apache/stormcrawler/urlfrontier/Constants.java | 22 ++
.../stormcrawler/urlfrontier/CrawlDelayPolicy.java | 216 +++++++++++++++++++
.../urlfrontier/QueueRegulatorBolt.java | 218 +++++++++++++++++--
.../org/apache/stormcrawler/urlfrontier/Spout.java | 7 +-
.../urlfrontier/CrawlDelayPolicyTest.java | 232 +++++++++++++++++++++
.../QueueRegulatorBoltDecisionTest.java | 144 ++++++++++++-
.../urlfrontier/QueueRegulatorBoltTest.java | 97 ++++++++-
external/warc/pom.xml | 2 +-
pom.xml | 26 +--
66 files changed, 1899 insertions(+), 192 deletions(-)
create mode 100644
core/src/test/java/org/apache/stormcrawler/bolt/URLFilterBoltTest.java
create mode 100644
core/src/test/java/org/apache/stormcrawler/protocol/RobotRulesParserAgentNamesTest.java
create mode 100644 core/src/test/resources/test.urlfilterbolt.json
create mode 100644
external/urlfrontier/src/main/java/org/apache/stormcrawler/urlfrontier/CrawlDelayPolicy.java
create mode 100644
external/urlfrontier/src/test/java/org/apache/stormcrawler/urlfrontier/CrawlDelayPolicyTest.java
