[
https://issues.apache.org/jira/browse/CONNECTORS-13?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13774219#comment-13774219
]
Karl Wright commented on CONNECTORS-13:
---------------------------------------
Hi Graeme,
At least initially, the point of using zookeeper would be to handle clusters
that would have only one instance of each type of node. For example, there
would be a crawler-UI webapp node, a REST API node, and an agents node. There
would NOT be multiple agents nodes yet, under this proposal. More changes
(some of them significant) would be required to make that level of change.
As such, any service - such as a database - that can be described via machine
name and port number should require no difference in configuration from one
node to another. That's why I'm more concerned about file system paths, which
definitely have a local-node affinity. These should probably remain within the
properties.xml file for that reason.
> We should move to eliminate process synchronization via shared file system,
> and use a process/service instead
> -------------------------------------------------------------------------------------------------------------
>
> Key: CONNECTORS-13
> URL: https://issues.apache.org/jira/browse/CONNECTORS-13
> Project: ManifoldCF
> Issue Type: Improvement
> Components: Framework core
> Affects Versions: ManifoldCF 0.1, ManifoldCF 0.2
> Reporter: Karl Wright
> Fix For: ManifoldCF next
>
>
> The current implementation relies on the file system to synchronize activity
> between various LCF processes. This has several downsides: first, it is
> possible to get the file system into a state that is corrupted (by killing
> processes); second, this limits the future ability to spread crawler workload
> over multiple machines.
> It should be reasonably straightforward, and probably more resilient, to
> introduce a "synchronization process", which all other LCF processes talk to
> in order to manage locks, shared data, and other synchronization activities.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
