[
https://issues.apache.org/jira/browse/CONNECTORS-13?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13776231#comment-13776231
]
Karl Wright commented on CONNECTORS-13:
---------------------------------------
Answer to (a) above: See
https://cwiki.apache.org/confluence/display/solr/Command+Line+Utilities . It
appears that Solr provides a zookeeper utility for uploading parameter files.
It appears that in most cases the entire FILE is loaded into Zookeeper,
although I'd want to confirm that. The class to look at is:
org.apache.solr.cloud.ZkCLI .
We'd probably want to borrow heavily from this class in any case, since we'd
need similar functionality for MCF. In fact, this might be the first
zookeeper-related thing we do.
> We should move to eliminate process synchronization via shared file system,
> and use a process/service instead
> -------------------------------------------------------------------------------------------------------------
>
> Key: CONNECTORS-13
> URL: https://issues.apache.org/jira/browse/CONNECTORS-13
> Project: ManifoldCF
> Issue Type: Improvement
> Components: Framework core
> Affects Versions: ManifoldCF 0.1, ManifoldCF 0.2
> Reporter: Karl Wright
> Fix For: ManifoldCF next
>
>
> The current implementation relies on the file system to synchronize activity
> between various LCF processes. This has several downsides: first, it is
> possible to get the file system into a state that is corrupted (by killing
> processes); second, this limits the future ability to spread crawler workload
> over multiple machines.
> It should be reasonably straightforward, and probably more resilient, to
> introduce a "synchronization process", which all other LCF processes talk to
> in order to manage locks, shared data, and other synchronization activities.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
