Hey Charlie, Thanks for your private message. I am replying here as well for posterity .. I think these links you shared may be helpful for future searchers:
hackathon feature branch luwak-server https://github.com/flaxsearch/luwak/tree/luwak-server Scott's blog post on the matter https://opensourceconnections.com/blog/2016/02/05/luwak/ I'll try to follow up on this thread with anything more concrete as we will be discussing internally. Regards, Luke From: users@solr.apache.org At: 12/13/23 13:03:35 UTC-5:00To: users@solr.apache.org Subject: Re: Solr Query Alerting Hi Luke, We built Luwak, the basis of Solcolator, at my previous company Flax, and donated it to Lucene. I've sent you a separate private email. I don't know who built Solcolator sadly, it would indeed be awesome to get this feature properly integrated into Solr. Best Charlie On 13/12/2023 15:53, Luke Kot-Zaniewski (BLOOMBERG/ 919 3RD A) wrote: > Hi All, > > My team and I are trying to match documents against a set of queries. Both need to be updated in real-time (at least on the order of seconds) and the system needs to scale to potentially millions of queries. Although lucene monitor greatly optimizes this kind of alerting logic, as far as we know, it won't handle scaling as it isn't wrapped by a sophisticated runtime. > > Reading through this thread https://lists.apache.org/list.html?dev@find_parent=true pointed me at https://github.com/SOLR4189/solcolator. Overall, this project looks promising but I'm concerned that it was contributed by an anonymous user and is lacking basic maintenance, at the very least it needs dependency upgrades. My second concern with it is that it ingests _documents_ via AddUpdateCommand and then writes the output to a configurable sink. In my team's use case, it would be better to invert the flow and process documents in the SearchHandler (so that the output can be written to SolrQueryResponse) and AddUpdate the queries to take better advantage of solr's sharding. The second concern is not as strong as the first as I am still thinking about how to implement the "solcolator" within our constraints. Perhaps the document-as-query approach has its own drawbacks that I am not seeing. > > In either case I have two questions: > > 1. Has anyone here used solcolator and with what effect? I would be grateful to learn specifics about the scale of your solution. > 2. What do you think about forking or riffing off this project and adding a document-as-query control flow? > > As it stands now we are designing a system that coordinates lucene monitor by directly interacting with zookeeper and it feels like we are reinventing some of solr's capabilities. Any feedback anyone might have would be greatly appreciated. > > Many thanks, > Luke -- Charlie Hull - Managing Consultant at OpenSource Connections Limited Founding member of The Search Network and co-author of Searching the Enterprise tel/fax: +44 (0)8700 118334 mobile: +44 (0)7767 825828 OpenSource Connections Europe GmbH | Pappelallee 78/79 | 10437 Berlin Amtsgericht Charlottenburg | HRB 230712 B Geschäftsführer: John M. Woodell | David E. Pugh Finanzamt: Berlin Finanzamt für Körperschaften II
