There’s something enticing when thinking of Lucene and Solr as independent codebases. I’ve always thought of Lucene as core search (indexing, analysis, tokenization, etc…) and Solr as a search experience. Lucene is more a library (or set of libraries) used by applications providing search experiences. Solr is just one of those applications - it provides the experience of search as a service, and feels focused on making search approachable and palatable to search novices.
The work I put into streaming expressions was born out of a desire to more widely expose search functionality. The streaming API drew me in because it exposed a new way to interact with core search functionality, and expressions came out of wanting to make it all easy to use for end users. I didn’t, and don’t, give a whole lot of thought to the internals of Lucene. I like a good user experience and I see Solr as an application trying to provide that. I do, however, have concerns about the long-term impact of a split. Lucene is able to set a very explicit N-1 backward compatibility policy because it can have less immediate concern for the downstream user. And this is not to denigrate Lucene at all - in fact I agree with that policy for core search functionality. If and when incompatible changes lead to significant gains they can and are made. Inefficient older ways are not brought forward further than necessary. Solr has to be concerned with their end users, who may be relative search novices, when considering backward incompatible changes. More thought is given to the experience and impact of upgrades. How are those issues dealt with across replicas and shards? What will happen in a cloud made up of lucene indexes of varying versions? My concern revolves around what happens if (when) Solr falls behind Lucene. Will it ever be able to catch up? There’s an argument to be made that Solr being a consistent N versions behind Lucene has some value to the Solr project. But, what happens if Solr gets a slower release cadence? Will it fall further and further behind? Will its inability to use the latest and greatest in Lucene be the impetus for a community splitting fork? Will a new search application come along without the legacy concerns of Solr and become a more enticing option? Perhaps, to all of that. I can’t really say. What I can say is I don’t think it’s appropriate to stifle the growth, or in this case the change, of a community because of fear of the unknown. Yes, I am worried that a project split will lead to trouble and issues for Solr, and some of those fears are born out of how I know my company uses Solr. But I also think a lot of good could come out of a split. It’d be exciting to see how a Lucene community advances the state of the art of core search, and how a Solr community provides a clean and easily digestible search experience to end users. Will Lucene become more embeddable? Will Solr become more plug-n-play? I’m a fan of Christine’s suggestion of first executing a code and release split and later, after seeing the impact of such a split, decide on a project split. Full disclosure, Christine and I work at the same company. I think independent codebases will in the end benefit both, though I do agree there is more inherent and immediate risk to Solr. - Dennis On Fri, May 15, 2020 at 4:03 AM Dawid Weiss <dawid.we...@gmail.com> wrote: > Hi Christine! > > > * After a while (perhaps with Lucene 10.0 or perhaps at some other > natural point) we re-arrive at the "together or separate" question. If > splitting worked well then Solr promotion to TLP could be a natural next > step > > My whole point is that I think the split is by large already there: > the mailing lists, the issues, the codebase (git constitutes common > storage but the build system and nearly anything else pretty much > independent with Solr consuming Lucene artifacts). I also believe the > will to separate the projects has been with (some of) us for a long > time and postponing this decision won't change anything. > > Dawid > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org > For additional commands, e-mail: dev-h...@lucene.apache.org > >
