[
https://issues.apache.org/jira/browse/CONNECTORS-781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karl Wright updated CONNECTORS-781:
-----------------------------------
Affects Version/s: ManifoldCF 1.5
Fix Version/s: ManifoldCF 1.5
Assignee: Karl Wright
Completion of CONNECTORS-13 should allow multiple agents to be worked on.
There are several issues:
(1) For performance, some database operations are constructed knowing that only
one possible thread can be performing the operation at a time, notably all
stuffing operations. This can be dealt with a simple LockManager lock.
(2) Throttling in Web and RSS connectors is performed locally and will thus be
insufficient if multiple JVMs are actively doing the same thing at the same
time. A lockmanager based solution will need to be constructed for these
connectors instead.
(3) Connection pool sizes are also per-JVM. Since these are a combination of
connection caching and connection count throttling, we will need a better
cross-JVM construction for handling this.
Nevertheless I think it is conceivable that this work could be completed for
the 1.5 release.
> Fault-Tolerant Setup for ManifoldCF Agent.
> ------------------------------------------
>
> Key: CONNECTORS-781
> URL: https://issues.apache.org/jira/browse/CONNECTORS-781
> Project: ManifoldCF
> Issue Type: Improvement
> Components: Framework agents process, Framework core, Framework
> crawler agent
> Affects Versions: ManifoldCF 1.5
> Reporter: Swami Rajamohan
> Assignee: Karl Wright
> Labels: agents, crawler, fault-tolerance
> Fix For: ManifoldCF 1.5
>
>
> It should be possible to setup ManifoldCF as a Fault-Tolerant infrastructure.
> The Agent component of ManifoldCF should support multiple instances of an
> agent crawling against a single crawl store, to be able to both distribute
> (share) the crawl load as well as to be able to pick up a request that gets
> abruptly terminated due to either partitioning of the instance/failure of the
> instance itself.
> Since there is a proposal to move to a store like Voldemort, it would be nice
> to be able to have a fault tolerant infrastructure.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
