I've opened the PR https://github.com/apache/solr/pull/1440
Some tests are still failing. I'll address them later On Tue, Mar 7, 2023 at 1:56 PM David Smiley <dsmi...@apache.org> wrote: > Fantastic! > I really appreciate you working with the community on this one. > > ~ David Smiley > Apache Lucene/Solr Search Developer > http://www.linkedin.com/in/davidwsmiley > > > On Mon, Mar 6, 2023 at 6:32 PM Fikavec F <fika...@yandex.ru> wrote: > > > Thank you, you are very kind. I took measurements on two physical servers > > with a 10 Gigabit link, the speed and time of full fetching 10 Gb > > collection (one shard; empty "Accept-Encoding: " header; collection with > > only id and string stored fields) are as follows: > > original wt=json - 419 Mb/s fetching time: 3m 26s > > original wt=csv - 463 Mb/s fetching time: 3m 6s > > my wt=myfastjson - 2.33 Gb/s fetching time: 37s > > original wt=smile - 2.38 Gb/s fetching time: 36s > > my wt=myfastcbor - 2.55 Gb/s fetching time: 34s > > original wt=javabin - 2.76 Gb/s fetching time: 31s > > > > With "Accept-Encoding: gzip" header: > > original wt=json - 81 Mb/s fetching time: 12m 3s > > my wt=myfastjson - 114 Mb/s fetching time: 8m 32s > > > > When I placed the test collection on 4 shards, wt=json worked at > speeds > > of about 430 Mb/s, and javabin, myfastcbor, myfastjson worked at speeds > of > > 5.2-6.7 Gb/s, however, before the start of data transfer, there was a > > significant delay (possibly for data collection from individual shards > and > > sorting) and the final full fetching time did not exceed the above. > > Downloading a 10 GB file from Jetty took place at a speed of 9.87 Gb/s. > > The time to move the cursor through the collection documents and get > > the field values in my python application has been reduced from 4m 24s to > > 1m 4s, moreover, half of this minute was spent to deserialization in > python > > from json and cbor. No failures were noticed, everything seemed to work > as > > before, only very quickly (4х+ faster). > > I dreamed of seeing Solr work at 5Gigabit+ speeds and with the help of > > your support, everything worked out, thank you all. > > > > Best Regards, > > > > > > > -- ----------------------------------------------------- Noble Paul
