As being said above, the speed of streaming file to socket, via OS internals is hardly achievable with java code crunching bytes through heap. Using RamDirectory might push on GC ,it's rather better to stick with the default one and leave enough RAM for file cache. Regarding the actual params: q=*%3A*&rows=100000 I'm not sure there's a shortcut bypassing ordering results through heap (even *:* score is constant), quite often sort=_docid_ asc allows to avoid sorting, try it. Also, there is https://solr.apache.org/guide/8_6/exporting-result-sets.html which might be better suites for straightforward download, don't forget about sort=_docid_ asc.
On Sun, Feb 26, 2023 at 1:58 AM Fikavec F <fika...@yandex.ru> wrote: > Thanks for the patch for testing. I could not see significant improvements > on virtual machines, I will try again this week on servers. > I tried the following values for buffers: 65536 - 64Kb, 262144 - 256Kb, > 524288 - 512Kb, 1048576 - 1MB, 4194304 - 4MB, 16777216 - 16MB, 33554432 - > 32Mb, 67108864 - 64Mb, 134217728 - 128MB. I changed the buffer size of Solr > and Jetty. It was visible in the logs: > > 2023-02-25 19:01:42.201 DEBUG (qtp1812823171-22) [ ] > o.a.s.c.u.FastWriter checking OS env for BUFSIZE => > java.lang.NumberFormatException: null > 2023-02-25 19:01:42.201 INFO (qtp1812823171-22) [ ] > o.a.s.c.u.FastWriter FastWriter.BUFSIZE=4194304 > > I noticed that increasing the buffer reduces %wait on the core down to 0 > and also with 100% loaded core, the speed sometimes increased to 520 > megabits (I haven't seen such numbers before, but It's still far from > Gigabit+). Adding ident=false and/or wt=csv increases the speed a bit more > (+30/50 Mbit and wt=xml slow down -80 Mbit). > > What else in the data chain can be a bottleneck? OS and Network (network > interface and kernel tuned for 10-Gigabit, tested by iperf - ok), disk > (ramdisk), processor (except that 4.3 GHz core is not enough to transfer > data from Solr in single thread faster than 0.5 Gigabit) are not a > bottleneck, jetty is able to distribute a file at high speeds, with large > buffers I have now received by wget: 2023-02-25 21:51:37 (6.25 Gb/s), > FastWriter.BUFSIZE now it's big too, what is the next possible bottleneck > in Solr software architecture to explore and search further? > > Thank you for your help, I hope if these are not natural algorithmic > limitations, we will be able to figure out and make Solr even better, > especially since with the advent of PCIe 5.0, NVME, DDR5 and Wi-Fi 7 speeds > close to 10 Gigabit are already commonplacem but many end-user needs still > dependent on single-threaded/core performance and do not get significant > benefits from new hardware speeds... > > Best Regards, > -- Sincerely yours Mikhail Khludnev https://t.me/MUST_SEARCH A caveat: Cyrillic!
