Thanks for your feedback Mikhail. Your comment makes a lot of sense. When discussing these (pretty cool) architectures I'm missing the point of > implementing transaction log in Solr codebase. > I think Kafka is the best fit for such a pre-indexer buffer. WYDT?
Starting from scratch with a design for such a write ahead log, there likely are multiple options. But here we're considering an integration into an existing system (SolrCloud) that already has existing logic and implementation around transaction logs and that currently does not have a dependency on Kafka. I think mandating the use of Kafka to switch to this mode of separating compute and storage would make adoption harder. One would also need a deployment of Kafka that is resilient to an AZ going down. We get this "for free" by using S3/GCS or similar. Moreover, the transaction log will be per shard but we can't afford a Kafka topic per shard (in our use case we have thousands of shards on the cluster). Addressing all this will likely complexify a design that will lose its initial beauty and appeal. In any case, work on the transaction log side of things has not started, and our (Salesforce) plan is to start that work once the existing state of the separation of compute and storage has been shared upstream (in a branch). There will be plenty of opportunities to discuss and decide on the best approach (and work on it if anybody is interested). Maybe even decide to not decide: use the abstraction internally and allow multiple implementations: existing local file system, S3/GCS based as suggested, Kafka or other based. I hope to be able to present this topic using 5 minutes during the upcoming community virtual meetup <https://cwiki.apache.org/confluence/display/SOLR/2023-07-19+Meeting+notes> (July 19th). Ilan On Wed, Jul 12, 2023 at 10:54 AM Mikhail Khludnev <m...@apache.org> wrote: > Hello Ilan, > Late comment, though. > > On Fri, Apr 28, 2023 at 8:33 PM Ilan Ginzburg <ilans...@gmail.com> wrote: > > > ... > > We're considering improving this approach by making the transaction log a > > shard level abstraction (rather than a replica/node abstraction), and > store > > it in S3 as well with a transaction log per shard, not per replica. > > This would allow indexing to not commit on every batch, speed up /update > > requests, push the constructed segments asynchronously to S3, guarantee > > data durability while still allowing nodes to be stateless (so can be > shut > > down at any time in any number without data loss and without having to > > restart these nodes to recover data only they can access). > > ... > > Thanks, > > Ilan > > > > When discussing these (pretty cool) architectures I'm missing the point of > implementing transaction log in Solr codebase. > I think Kafka is the best fit for such a pre-indexer buffer. WYDT? > > -- > Sincerely yours > Mikhail Khludnev >
