Hi Abe-san, This sounds like a workable solution to the cluster problem. Can you also write your lucene searcher to use the same technology?
Sent from my Windows Phone From: Shinichiro Abe (JIRA) Sent: 7/17/2015 1:18 AM To: [email protected] Subject: [jira] [Commented] (CONNECTORS-1219) Lucene Output Connector [ https://issues.apache.org/jira/browse/CONNECTORS-1219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14630787#comment-14630787 ] Shinichiro Abe commented on CONNECTORS-1219: -------------------------------------------- Thanks [~apillaiz], I'd like to collect not only web content but also manifold repositories content. [~DaddyWri], I discovered the [OakDirectory|https://github.com/apache/jackrabbit-oak/blob/trunk/oak-lucene/src/main/java/org/apache/jackrabbit/oak/plugins/index/lucene/LuceneIndexEditorContext.java#L89] which extends Lucene Directory class. I saw the below comment, they also had multi process(cluster) problem as to Lucene index, and they put the index to Blob object that means mongodb or rdb storage. From that, I come to switching Directory impl, for instance, we use FSDirectory on mcf single process, and use [HdfsDirectory|http://lucene.apache.org/solr/5_2_1/solr-core/org/apache/solr/store/hdfs/HdfsDirectory.html] on mcf multi process. The writes to Hdfs was [slow|https://github.com/ouava/lclient/blob/master/lclient-hdfs/src/main/java/org/apache/lucene/lclient/util/HdfsUtils.java#L47] when I tried to use before. But this will be expected to improve. I don't want to use RMI because... first: to avoid complexable operation or increase 2 steps for bootstrap on single process mode, second: I don't know how to write the test code, third: around me, only one user uses multi process and everyone will hope to run mcf as OOTB as possible, fourth: jackrabbit 2 has RMI api but oak doesn't have one. I think RMI is not cool as well as CMIS rather than JCR , fifth: I want to make mcf easy to use. These are not technical reason, but HdfsDirectory will help us. > Lucene Output Connector > ----------------------- > > Key: CONNECTORS-1219 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1219 > Project: ManifoldCF > Issue Type: New Feature > Reporter: Shinichiro Abe > Assignee: Shinichiro Abe > Attachments: CONNECTORS-1219-v0.1patch.patch, > CONNECTORS-1219-v0.2.patch, CONNECTORS-1219-v0.3.patch > > > A output connector for Lucene local index directly, not via remote search > engine. It would be nice if we could use Lucene various API to the index > directly, even though we could do the same thing to the Solr or Elasticsearch > index. I assume we can do something to classification, categorization, and > tagging, using e.g lucene-classification package. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
