[
https://issues.apache.org/jira/browse/CONNECTORS-781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13837148#comment-13837148
]
Karl Wright commented on CONNECTORS-781:
----------------------------------------
So, for the issue of apportioning connector instances, it occurs to me that the
following is true:
(1) There is a solid "fixed" number of connections per process, and a number (<
the number of processes) that represents the non-fixed number of connections
available;
(2) Each process can determine the fixed number by inspecting the number of
registered processes;
(3) Periodically, the fixed number of connections for a given process can be
updated.
Thus, there can indeed be a global pool counter for each connection pool, with
a hard local limit of (global_limit/<registered_processes> + 1), and a minimum
pool limit of (global_limit/<registered_processes>). I'll need to look
carefully at how the code is structured to work with this, which is the next
step. Since the pooling logic is all done at the factory level, we may need to
create a base factory that does this job for all pools there are.
> Fault-Tolerant Setup for ManifoldCF Agent.
> ------------------------------------------
>
> Key: CONNECTORS-781
> URL: https://issues.apache.org/jira/browse/CONNECTORS-781
> Project: ManifoldCF
> Issue Type: Improvement
> Components: Framework agents process, Framework core, Framework
> crawler agent
> Affects Versions: ManifoldCF 1.5
> Reporter: Swami Rajamohan
> Assignee: Karl Wright
> Labels: agents, crawler, fault-tolerance
> Fix For: ManifoldCF 1.5
>
>
> It should be possible to setup ManifoldCF as a Fault-Tolerant infrastructure.
> The Agent component of ManifoldCF should support multiple instances of an
> agent crawling against a single crawl store, to be able to both distribute
> (share) the crawl load as well as to be able to pick up a request that gets
> abruptly terminated due to either partitioning of the instance/failure of the
> instance itself.
> Since there is a proposal to move to a store like Voldemort, it would be nice
> to be able to have a fault tolerant infrastructure.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
