[
https://issues.apache.org/jira/browse/CONNECTORS-781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13825017#comment-13825017
]
Karl Wright commented on CONNECTORS-781:
----------------------------------------
So it looks like we have quite a major amount of work to do to complete this
ticket.
I intend therefore to tackle it one piece at a time. The first piece (locks
around all single-threaded places) is "done" (unless I find other places that
need similar treatment). The rest, in order, looks like this:
(1) Schema change to jobqueue table to label records in transient states with a
process ID of some kind, and separation of global reset logic from local reset
logic;
(2) Reworking of QueueTracker logic to work off a jobqueue child table that
tracks bin information;
(3) Official support for multiple agents processes;
(4) Cross-process throttling;
(5) Cross-process connection-pool limiting.
This is not going to be completed in the 1.5 time frame after all. I think
therefore we should work towards getting (1) and (2) done, which is going to be
challenging enough.
> Fault-Tolerant Setup for ManifoldCF Agent.
> ------------------------------------------
>
> Key: CONNECTORS-781
> URL: https://issues.apache.org/jira/browse/CONNECTORS-781
> Project: ManifoldCF
> Issue Type: Improvement
> Components: Framework agents process, Framework core, Framework
> crawler agent
> Affects Versions: ManifoldCF 1.5
> Reporter: Swami Rajamohan
> Assignee: Karl Wright
> Labels: agents, crawler, fault-tolerance
> Fix For: ManifoldCF 1.5
>
>
> It should be possible to setup ManifoldCF as a Fault-Tolerant infrastructure.
> The Agent component of ManifoldCF should support multiple instances of an
> agent crawling against a single crawl store, to be able to both distribute
> (share) the crawl load as well as to be able to pick up a request that gets
> abruptly terminated due to either partitioning of the instance/failure of the
> instance itself.
> Since there is a proposal to move to a store like Voldemort, it would be nice
> to be able to have a fault tolerant infrastructure.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
