Thank you for helping me figure out the performance of "Response Writers". If you wish, you can close the topic or move it to issues under the title performance of non-binary types of "Response Writers" (if in your opinion the difference of 4-8 times between wt=javabin and wt=json is not normal). Maybe it is worth adding to the documentation a mention of the order of the difference in the speeds of "Response Writers" in /select and /export handlers? I would really like wt=json to work as fast as possible, as it is popular and compatible, moreover, Solr itself works much faster than wt=json.
I have read and tested the specified wt=smile format (it is slower than java bin, but also significantly faster than wt=json). Does the JDBC driver (when working from python) offered in Solr 9.1.1 work internally using javabin for client-server interaction (that is, data is received on it at javabin speeds or as in wt=json)?
As I wrote earlier - jetty is not a bottleneck in this issue and in nginx activated { io threads; sendfile on; } as in the article on configuring nginx. I haven't thoroughly tested jetty specifically, just quickly checked that jetty is able to transfer data (files) faster than when they are fetched from Solr. I saw that jetty can work to transmit data at very high speeds of the order of 6-9 gigabytes, and this was enough for me, I did not think to compare and demand from it the speed of work as nginx optimized for file transfer.
Michael Gibney - without using sorting in my tests on Solr 9.1.1, fetching looks a bit faster, but not so much that it is significant:
- 409 Mb/s sort=id asc wt=javabin
- 411 Mb/s without sort wt=javabin
- 1.40 Gb/s sort=id asc wt=javabin
- 1.48 Gb/s without sort wt=javabin
- 2.38 Gb/s sort=id asc wt=smile
- 2.59 Gb/s without sort wt=smile
All measurement results:
<<<<----- SOLR 9.1.1 tests 28.02.2023 field with stored=true docValues=false ------>>>>>!!![sort=id asc]/select HANDLER wt=json sort=id ascwget --report-speed=bits --server-response -O /dev/null --header='Accept-Encoding: ' --post-data 'indent=false&wt=json&q=*%3A*&rows=1000000&sort=id%20asc' http://192.168.220.135:8983/solr/test_collection/select10.03G in 3m 30s2023-02-27 21:55:53 (409 Mb/s) - ‘/dev/null’ saved [10772687921]/select HANDLER wt=javabin sort=id ascwget --report-speed=bits --server-response -O /dev/null --header='Accept-Encoding: ' --post-data 'indent=false&wt=javabin&q=*%3A*&rows=1000000&sort=id%20asc' http://192.168.220.135:8983/solr/test_collection/select10.01G in 61s2023-02-27 22:01:32 (1.40 Gb/s) - ‘/dev/null’ saved [10749142324]/select HANDLER wt=smile sort=id ascwget --report-speed=bits --server-response -O /dev/null --header='Accept-Encoding: ' --post-data 'indent=false&wt=smile&q=*%3A*&rows=1000000&sort=id%20asc' http://192.168.220.135:8983/solr/test_collection/select10.01G in 36s2023-02-27 22:31:40 (2.38 Gb/s) - ‘/dev/null’ saved [10748488299]!!![without sort]/select HANDLER wt=json without sortwget --report-speed=bits --server-response -O /dev/null --header='Accept-Encoding: ' --post-data 'indent=false&wt=json&q=*%3A*&rows=1000000' http://192.168.220.135:8983/solr/test_collection/select10.03G in 3m 30s2023-02-27 22:06:31 (411 Mb/s) - ‘/dev/null’ saved [10772687903]/select HANDLER wt=javabin without sortwget --report-speed=bits --server-response -O /dev/null --header='Accept-Encoding: ' --post-data 'indent=false&wt=javabin&q=*%3A*&rows=1000000' http://192.168.220.135:8983/solr/test_collection/select10.01G in 58s2023-02-27 22:13:36 (1.48 Gb/s) - ‘/dev/null’ saved [10749142310]/select HANDLER wt=smile without sortwget --report-speed=bits --server-response -O /dev/null --header='Accept-Encoding: ' --post-data 'indent=false&wt=smile&q=*%3A*&rows=1000000' http://192.168.220.135:8983/solr/test_collection/select10.01G in 33s2023-02-27 22:29:38 (2.59 Gb/s) - ‘/dev/null’ saved [10748488286]
Ishan Chattopadhyaya - yes, regarding my tests, it looks like javabean in Solr 9.1.1 significantly degraded in speed relative to version 8.11.2. Since I was taking measurements in a 10 Gigabit network infrastructure (emulated on Vms and real) with special optimization of the OS for it (enable Jumbo Frames mtu 9000; tcp_window_scaling; 128MB tcp_wmem and tcp_rmem) and disabled gzip, then the difference in performance could become noticeable, which under normal conditions was not noticeable at all due to the impossibility of achieving such network speeds with standard settings. Therefore, my aspirations and measurements at speeds over gigabit are not yet a mass need and phenomenon and may not be worth close attention. The most interesting thing is that in today's measurement for Michael Gibney in Solr 9.1.1 (presented above) javabin was slower than wt=smile, but in version 8.11.2 wt=smile was fast, but noticeably slower than javabin. If necessary, I will provide you with everything you may need to reproduce. Performance is a very complicated matter, it is affected by a lot of things, it is possible that the slowdown occurred due to the peculiarities of my test data or test queries. Does Solr have a project to track basic performance metrics from version to version, similar to Rally (the macro-benchmarking framework for Elasticsearch)?
Best Regards,
