As far as a Lucene/Solr directory on cloud storage. Performance on the write has a lot of overhead per file, hundreds of millis. The read overhead is about half as much. I believe the write is so expensive due to the strong consistency of both gcs and s3. So I think the main bottleneck would be indexing and merging lots of small segments etc ...
Joel Bernstein http://joelsolr.blogspot.com/ On Fri, Apr 21, 2023 at 3:27 AM Ishan Chattopadhyaya < ichattopadhy...@gmail.com> wrote: > My colleague at SearchScale has tried S3FS, and running Solr indexes off > S3. We can chat about it, if you're interested. > > On Fri, 21 Apr, 2023, 10:38 am David Smiley, <dsmi...@apache.org> wrote: > > > Cool! > > I wonder if anyone has tried such things for a Lucene/Solr "Directory" as > > well? > > > > ~ David Smiley > > Apache Lucene/Solr Search Developer > > http://www.linkedin.com/in/davidwsmiley > > > > > > On Mon, Apr 17, 2023 at 1:14 PM Joel Bernstein <joels...@gmail.com> > wrote: > > > > > I've been testing Java NIO providers for cloud storage. These two in > > > particular worked for our use cases: > > > > > > https://github.com/googleapis/java-storage-nio > > > https://github.com/carlspring/s3fs-nio > > > > > > I believe an Azure provider is available. > > > > > > We've been working on sponsoring getting the s3 provider into a public > > > maven repo and I can update this thread when that's done. > > > > > > > > > > > > Joel Bernstein > > > http://joelsolr.blogspot.com/ > > > > > > > > > On Mon, Apr 10, 2023 at 6:51 PM Ishan Chattopadhyaya < > > > ichattopadhy...@gmail.com> wrote: > > > > > > > Oh thanks, Jan. I had missed it. It is a shame because it looks like > a > > > very > > > > neat project. > > > > > > > > On Mon, 10 Apr, 2023, 23:53 Jan Høydahl, <jan....@cominvent.com> > > wrote: > > > > > > > > > Looks like a nice project. With the promise of low-hanging support > > for > > > > > more providers than those three for free. > > > > > > > > > > However, > > > > https://lists.apache.org/thread/w61gzk2ohjtshbwcb5gy6wb2htv7fo0x > > > > > does not look promising - they plan to move the project to the > Attic, > > > and > > > > > no new releases has happened during the 6 months since the > > proposal... > > > > > > > > > > Jan > > > > > > > > > > > 10. apr. 2023 kl. 19:08 skrev Ishan Chattopadhyaya < > > > > > ichattopadhy...@gmail.com>: > > > > > > > > > > > > I think we should deprecate both the modules for S3 and GCS, and > > > > > > adopt Apache JCloud project that supports all three. > > > > > > > > > > > > > > > > > > > >
