[
https://issues.apache.org/jira/browse/SOLR-8588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joel Bernstein updated SOLR-8588:
---------------------------------
Attachment: SOLR-8588.patch
Added tests for checkpointing during read() iteration.
> Add TopicStream to the streaming API to support publish/subscribe messaging
> ---------------------------------------------------------------------------
>
> Key: SOLR-8588
> URL: https://issues.apache.org/jira/browse/SOLR-8588
> Project: Solr
> Issue Type: New Feature
> Reporter: Joel Bernstein
> Assignee: Joel Bernstein
> Attachments: SOLR-8588.patch, SOLR-8588.patch, SOLR-8588.patch,
> SOLR-8588.patch, SOLR-8588.patch
>
>
> The TopicStream is a *publish/subscribe messaging service* built on top of
> SolrCloud. The TopicStream returns all *new* documents for a specific query.
> Version numbers will be used as checkpoints for Topics to ensure single
> delivery of each document. When combined with the DaemonStream (SOLR-8550),
> Topics can provide continuous streaming. Sample syntax:
> {code}
> topic(checkpointCollection, dataCollection, id="topicA", q="awesome stuff"
> checkpointEvery="1000")
> {code}
> The checkpoint collection will be used to persist the topic checkpoints.
> Example combined with the DaemonStream:
> {code}
> daemon(topic(...)...)
> {code}
> When combined with SOLR-7739 this allows for messaging based on *machine
> learned* classifications.
> The TopicStream supports 3 models of publish/subscribe messaging:
> 1) *Request & response*: In this model a topic(...) expression can be saved
> and executed at any time. In this scenario the TopicStream will always
> retrieve it's checkpoints and start from where it left off.
> 2) *Continuous pull streaming*: In this model you would wrap the TopicStream
> in a DaemonStream and call read() in a loop inside a java program. This
> would provide a continuous stream of new content as it arrives in the index.
> 3) *Continuous push streaming*: In this model you would send an expression
> like this to the /stream handler: *daemon(update(topic(...)...)...)*. This
> daemon process would run inside Solr and continuously stream new documents
> from the topic and push them to another SolrCloud collection. Other pushing
> expressions can be created to push documents in different ways or take other
> types of actions.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
