Hello Ilan, Late comment, though. On Fri, Apr 28, 2023 at 8:33 PM Ilan Ginzburg <ilans...@gmail.com> wrote:
> ... > We're considering improving this approach by making the transaction log a > shard level abstraction (rather than a replica/node abstraction), and store > it in S3 as well with a transaction log per shard, not per replica. > This would allow indexing to not commit on every batch, speed up /update > requests, push the constructed segments asynchronously to S3, guarantee > data durability while still allowing nodes to be stateless (so can be shut > down at any time in any number without data loss and without having to > restart these nodes to recover data only they can access). > ... > Thanks, > Ilan > When discussing these (pretty cool) architectures I'm missing the point of implementing transaction log in Solr codebase. I think Kafka is the best fit for such a pre-indexer buffer. WYDT? -- Sincerely yours Mikhail Khludnev
