[
https://issues.apache.org/jira/browse/CONNECTORS-13?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13772650#comment-13772650
]
Graeme Seaton commented on CONNECTORS-13:
-----------------------------------------
I think the obvious low-hanging fruit (as has already been pointed out) is the
replacement of the shared file-system synchronisation with Zookeeper. Think we
could use the existing code within Solr as a template for the implementation.
This would provide experience with working with ZK from a development and
administrative perspective.
Once that is complete then the next step would be to migrate the properties
settings.
One concern I have is how node-specific settings would be supported i.e. I'm
using a Postgres cluster and want the agent to talk to the local instance.
Could carry out host file magic but that can also be problematic.
> We should move to eliminate process synchronization via shared file system,
> and use a process/service instead
> -------------------------------------------------------------------------------------------------------------
>
> Key: CONNECTORS-13
> URL: https://issues.apache.org/jira/browse/CONNECTORS-13
> Project: ManifoldCF
> Issue Type: Improvement
> Components: Framework core
> Affects Versions: ManifoldCF 0.1, ManifoldCF 0.2
> Reporter: Karl Wright
> Fix For: ManifoldCF next
>
>
> The current implementation relies on the file system to synchronize activity
> between various LCF processes. This has several downsides: first, it is
> possible to get the file system into a state that is corrupted (by killing
> processes); second, this limits the future ability to spread crawler workload
> over multiple machines.
> It should be reasonably straightforward, and probably more resilient, to
> introduce a "synchronization process", which all other LCF processes talk to
> in order to manage locks, shared data, and other synchronization activities.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
