Hi Rafa,

Thank you for your valuable suggestions.

On Tue, Aug 13, 2019 at 5:25 PM Rafa Haro <rh...@apache.org> wrote:

> Hi Dileepa,
> IMHO, Furkan's approach makes the most sense here. As Olivier pointed out,
> to retrieve the original content from a Lucene based index, all the fields
> you are interested in must be stored. If it is your case, you can probably
> implement a Repository connector then. You can enable incremental crawling
> by querying for all the Solr documents (q=*:*), using pagination and using
> one of the fields as a filter to locate only new or modified documents at
> each crawl.
> But, it seems to make more sense if you include your Solr index as a new
> distributed index along with the other index (ES or Solr) that you plan to
> populate using ManifoldCF. Typical resources you are going to need for
> achieving that is 1) a query adapter to convert the user query to a query
> language supported for all your indexes (easy in this case, because both
> can talk Lucene query syntax) and 2) a module to normalize the scores of
> the results from all your indexes. You can use a min-max approach for
> normalising, for example.
Are you referring to a query-time-merge approach instead of following a
index-time-merge approach (getting all the content in a central index
first) as a solution here? If so, yes, I am considering that approach as
well, however the concern is some data-sources are rdf data stores, and
querying rdf stores are typically slow in POV and the whole search would
then be slow with query-federation approach. WDYT?


> This is a quite typical scenario, so I'm sure you can easily find good
> literature about how to architecture a distributed federated search engine
> Cheers,
> Rafa
> On Tue, Aug 6, 2019 at 2:52 PM Dileepa Jayakody <dileepajayak...@gmail.com>
> wrote:
>> Hi All,
>> Thank you for your replies.
>> @Furkan, Olivier, thanks for the pointers. I will check the approach of
>> the Solr repository connector as per given references.
>> @Olivier if you can contribute the Solr repo-connector you are working
>> on, to MCF that will be awesome! Will be looking forward to an update on
>> that.
>> Regards,
>> Dileepa
>> On Mon, Aug 5, 2019 at 5:01 PM Olivier Tavard <
>> olivier.tav...@francelabs.com> wrote:
>>> Hello,
>>> We are currently working on this kind of repository connector for a
>>> customer. We plan to give the code to the MCF project if the customer lets
>>> us do it legally. We will know it at the end of the month or at the
>>> beginning of next month.
>>> In order to have this working, all the fields of the target Solr need to
>>> be stored, this condition is mandatory. You can give a look to the Solr
>>> entity processor of Data Import Handler component :
>>> https://lucene.apache.org/solr/guide/8_0/uploading-structured-data-store-data-with-the-data-import-handler.html#entity-processors.
>>>  We
>>> were inspired by that for the development of the connector.
>>> Best regards,
>>> Olivier
>>> Le 5 août 2019 à 16:38, Furkan KAMACI <furkankam...@gmail.com> a écrit :
>>> Hi Dileepa,
>>> Writing a custom repository connector can let you achieve your goal.
>>> Read and directly write to an output connector.
>>> You should check your requirements i.e. which data sources you will
>>> connect. MCF may rid of huge integration pains compared to many other ETL
>>> tools in your case.
>>> On the other hand, if you wanna achieve a federated search, you could
>>> search across distributed indexes. Otherwise, it is a heteregous sourced
>>> indexing architecture. You can federate your search query into Solr without
>>> ingesting it to any other place. By the way, MCF will let you make document
>>> level security, you should handle it manually in such a case.
>>> Kind Regards,
>>> Furkan KAMACI
>>> 5 Ağu 2019 Pzt, saat 17:11 tarihinde Dileepa Jayakody <
>>> dileepajayak...@gmail.com> şunu yazdı:
>>>> Hi Karl and all,
>>>> In my use-case, one of the data-sources is an already populated Solr
>>>> index which is an e-commerce web-site data index (customers, products &
>>>> services).
>>>> Apart from the Solr Index, I need to ingest several other heterogeneous
>>>> data-sources such as PostgresSQL databases, CRM data etc into the federated
>>>> search index (the output index will either be a Solr, Elastic-search. We
>>>> haven't yet finalized on the output index, but I know that both of these
>>>> are supported in MCF as output connectors.).
>>>> @Karl based on your comments, I would appreciate your opinion on below
>>>> ingestion flow.
>>>> Solr repository/data-source > Solr schema transformations >
>>>> Solr/Elastic-search search-index
>>>> For such a scenario, do you think MCF is not the ideal option as the
>>>> ETL/ingestion tool?  Should I go for a lower-level ETL tool such as Apache
>>>> Nifi ?
>>>> Or will writing a MCF Solr repository connector be useful to achieve
>>>> this?
>>>> WDYT?
>>>> Thanks a lot.
>>>> Regards,
>>>> Dileepa
>>>> On Mon, Aug 5, 2019 at 3:40 PM Karl Wright <daddy...@gmail.com> wrote:
>>>>> If you are trying to extract data from a Solr index, I know of no way
>>>>> to do that.
>>>>> Karl
>>>>> On Mon, Aug 5, 2019 at 9:08 AM Dileepa Jayakody <
>>>>> dileepajayak...@gmail.com> wrote:
>>>>>> Hi All,
>>>>>> Thanks for your replies.
>>>>>> I'm looking for a repository connector. I've used the Solr output
>>>>>> connector before. But now what I need is to connect to a solr index as a
>>>>>> repository and retrieve the documents from there. So I need a Solr
>>>>>> repository connector.
>>>>>> @Karl
>>>>>> I will look at the Solr connector, but this is an output connect,
>>>>>> isn't it? Can use this as a repository connector to retrieve docs?
>>>>>> Thanks,
>>>>>> Dileepa
>>>>>> On Mon, Aug 5, 2019 at 12:45 PM Cihad Guzel <cguz...@gmail.com>
>>>>>> wrote:
>>>>>>> Hi Dileepa,
>>>>>>> You can check all MFC Connectors list from
>>>>>>> https://manifoldcf.apache.org/release/release-2.13/en_US/included-connectors.html
>>>>>>> MFC have a Solr Output Connector. It is not a repository connector.
>>>>>>> if you want to use as repository connector, you should write a new
>>>>>>> repository connector.
>>>>>>> Regards,
>>>>>>> Cihad Guzel
>>>>>>> Dileepa Jayakody <dileepajayak...@gmail.com>, 5 Ağu 2019 Pzt, 13:18
>>>>>>> tarihinde şunu yazdı:
>>>>>>>> Hi All,
>>>>>>>> I'm working on a project which needs to implement a federated
>>>>>>>> search solution with heterogeneous data repositories. One repository 
>>>>>>>> is a
>>>>>>>> Solr index. I would like to use ManifoldCF as the data ingestion 
>>>>>>>> engine in
>>>>>>>> this project as I have worked with MCF before.
>>>>>>>> Does ManifoldCF has a Solr repository connector which I can use
>>>>>>>> here? Or will I need to implement a new repository connector for Solr?
>>>>>>>> Any guidance here is much appreciated.
>>>>>>>> Thanks,
>>>>>>>> Dileepa

Reply via email to