Re: Solr Kafka DIH

[Susheel Kumar]

Hello Joel,

This definitely seems like a good feature to add.  In fact I was also
looking to push data into Solr from Kafka and this can be good feature to
have.

I have created JIRA https://issues.apache.org/jira/browse/SOLR-10086 and
can contribute as well.

Thanks,
Susheel

On Tue, Jan 31, 2017 at 6:28 PM, John Bickerstaff 
wrote:

> I wrote a simple java microservice that did this about a year ago...  It
> was pretty simple - and the kafka topic served as a way to re-create my
> collection from scratch without hitting the database again in the event of
> the Solr servers going down.
>
> The code just read from Kafka topic one by one and shipped to Solr in
> batches of 500 (between commits)
>
> (It was a small data set, I was lucky that way)
>
> On Tue, Jan 31, 2017 at 3:41 PM, Joel Bernstein 
> wrote:
>
> > This would make a great Streaming Expression as well. If you're
> interested
> > in working on this I'll help out along the way. Here is an example Stream
> > that connects to a JDBC data source:
> >
> > https://github.com/apache/lucene-solr/blob/master/solr/
> > solrj/src/java/org/apache/solr/client/solrj/io/stream/JDBCStream.java
> >
> > Joel Bernstein
> > http://joelsolr.blogspot.com/
> >
> > On Tue, Jan 31, 2017 at 12:07 PM, Mike Thomsen 
> > wrote:
> >
> > > Probably not, but writing your own little Java process to do it would
> be
> > > trivial with Kafka 0.9.X or 0.10.X. You can also look at the Confluent
> > > Platform as they have tons of connectors for Kafka to directly feed
> into
> > > other systems.
> > >
> > > On Mon, Jan 30, 2017 at 3:05 AM, Mahmoud Almokadem <
> > prog.mahm...@gmail.com
> > > >
> > > wrote:
> > >
> > > > Hello,
> > > >
> > > > Is there a way to get SolrCloud to pull data from a topic in Kafak
> > > > periodically using Dataimport Handler?
> > > >
> > > > Thanks
> > > > Mahmoud
> > >
> >
>

Re: Solr Kafka DIH

[John Bickerstaff]

I wrote a simple java microservice that did this about a year ago...  It
was pretty simple - and the kafka topic served as a way to re-create my
collection from scratch without hitting the database again in the event of
the Solr servers going down.

The code just read from Kafka topic one by one and shipped to Solr in
batches of 500 (between commits)

(It was a small data set, I was lucky that way)

On Tue, Jan 31, 2017 at 3:41 PM, Joel Bernstein  wrote:

> This would make a great Streaming Expression as well. If you're interested
> in working on this I'll help out along the way. Here is an example Stream
> that connects to a JDBC data source:
>
> https://github.com/apache/lucene-solr/blob/master/solr/
> solrj/src/java/org/apache/solr/client/solrj/io/stream/JDBCStream.java
>
> Joel Bernstein
> http://joelsolr.blogspot.com/
>
> On Tue, Jan 31, 2017 at 12:07 PM, Mike Thomsen 
> wrote:
>
> > Probably not, but writing your own little Java process to do it would be
> > trivial with Kafka 0.9.X or 0.10.X. You can also look at the Confluent
> > Platform as they have tons of connectors for Kafka to directly feed into
> > other systems.
> >
> > On Mon, Jan 30, 2017 at 3:05 AM, Mahmoud Almokadem <
> prog.mahm...@gmail.com
> > >
> > wrote:
> >
> > > Hello,
> > >
> > > Is there a way to get SolrCloud to pull data from a topic in Kafak
> > > periodically using Dataimport Handler?
> > >
> > > Thanks
> > > Mahmoud
> >
>

Re: Solr Kafka DIH

[Joel Bernstein]

This would make a great Streaming Expression as well. If you're interested
in working on this I'll help out along the way. Here is an example Stream
that connects to a JDBC data source:

https://github.com/apache/lucene-solr/blob/master/solr/solrj/src/java/org/apache/solr/client/solrj/io/stream/JDBCStream.java

Joel Bernstein
http://joelsolr.blogspot.com/

On Tue, Jan 31, 2017 at 12:07 PM, Mike Thomsen 
wrote:

> Probably not, but writing your own little Java process to do it would be
> trivial with Kafka 0.9.X or 0.10.X. You can also look at the Confluent
> Platform as they have tons of connectors for Kafka to directly feed into
> other systems.
>
> On Mon, Jan 30, 2017 at 3:05 AM, Mahmoud Almokadem  >
> wrote:
>
> > Hello,
> >
> > Is there a way to get SolrCloud to pull data from a topic in Kafak
> > periodically using Dataimport Handler?
> >
> > Thanks
> > Mahmoud
>

Re: Solr Kafka DIH

[Mike Thomsen]

Probably not, but writing your own little Java process to do it would be
trivial with Kafka 0.9.X or 0.10.X. You can also look at the Confluent
Platform as they have tons of connectors for Kafka to directly feed into
other systems.

On Mon, Jan 30, 2017 at 3:05 AM, Mahmoud Almokadem 
wrote:

> Hello,
>
> Is there a way to get SolrCloud to pull data from a topic in Kafak
> periodically using Dataimport Handler?
>
> Thanks
> Mahmoud

Solr Kafka DIH

[Mahmoud Almokadem]

Hello,

Is there a way to get SolrCloud to pull data from a topic in Kafak periodically 
using Dataimport Handler?

Thanks
Mahmoud