Why solr 4.0 use FSIndexOutput to write file, otherwise MMap/NIO
I have checked the FSDirectory, it will create MMapDirectory or NIOFSDirectory for Directory. This two directory only supply IndexInput extend for read file (MMapIndexInput extends ByteBufferIndexInput), why not there is not MMap/NIO IndexOutput extend for file write. It only use FSIndexOutput for file write(FSIndexOutput extends BufferedIndexOutput). Does FSIndexOutput wirte file very slow than MMap/NIO? How to improve the IO write performance. Thanks, __ Jeffery Wang Application Service - Backend Morningstar (Shenzhen) Ltd. Morningstar. Illuminating investing worldwide. +86 755 3311 0220 Office +86 130 7782 2813 Mobile jeffery.w...@morningstar.commailto:jeffery.w...@morningstar.com This e-mail contains privileged and confidential information and is intended only for the use of the person(s) named above. Any dissemination, distribution or duplication of this communication without prior written consent from Morningstar is strictly prohibited. If you received this message in error please contact the sender immediately and delete the materials from any computer.
RE: Solr 4.0 Optimize query very slow before the optimize end of a few minutes
Yes, I used the same query url for each curl-call, it is very simple http://...q=OS01W:sina*fl=SecId,OS01Wrows=1wt=xmlindent=true;. -Original Message- From: Toke Eskildsen [mailto:t...@statsbiblioteket.dk] Sent: 2013年6月14日 16:20 To: solr-user@lucene.apache.org Subject: RE: Solr 4.0 Optimize query very slow before the optimize end of a few minutes On Fri, 2013-06-14 at 06:59 +0200, Jeffery Wang wrote: Time queryTime(ms), CPU % r/s w/s rMB/s wMB/s IO % ... 7:30:52 16594 26 36 0 0.140 99.3 7:30:53 31 80 368 0 42.43 0 94.3 7:31:23 28575 41 35 21 0.372.3695.9 7:32:22 53399 31 81 39 0.742.6399.5!!! 7:32:23 11 54 155 0 16.46 0 99.6 7:33:28 60199 28 30 2 0.120.0199.8!! Having a single query that is slow is expected behaviour as the reader will have opened the merged segment and caches needs to be filled. But I do not know why you have more than one query that is slow. Do you use the same query for each curl-call? - Toke Eskildsen, State and University Library, Denmark
Solr 4.0 Optimize query very slow before the optimize end of a few minutes
Do someone known Why the query is very slow before the optimize end of a few minutes. When the solr optimize, I have a loop query( curl query url and sleep one second) every one second to check the query speed. It is normal, the query time can be accept. But it always very slow before the optimize end of a few minutes. The solr index size is about 22G after optimized. The follows is the query time cost, CPU and IO usage. The whole optimize process, the IO is high, it can be understand. time query time(ms) CPU % r/s w/s rMB/s wMB/s IO % 7:30:24 12 89 156.44 0 16.4 0 94.06 7:30:25 18 91 157 0 15.35 0 98.1 7:30:26 9 91 194 0 19.62 0 96.1 7:30:27 14 38 352 0 38.17 0 100.1 7:30:28 30 77 205.94 16.83 20.17 4.02 98.51 7:30:30 101 88 396 0 45.99 0 90.7 7:30:31 11 90 120 0 11.34 0 97.5 7:30:32 38 89 262.38 0 28.03 0 96.24 7:30:33 11 78 68 17 4.89 4.93 99.9 7:30:34 9 29 201 0 20.16 0 100.3 7:30:35 9 87 181 0 17.27 0 94.3 7:30:52 16594 26 36 0 0.14 0 99.3 7:30:53 31 80 368 0 42.43 0 94.3 7:31:23 28575 41 35 21 0.37 2.36 95.9 7:31:27 2676 60 127 0 13.76 0 83.5 7:31:28 8 59 279 0 30.99 0 99.4 7:32:22 53399 31 81 39 0.74 2.63 99.5 7:32:23 11 54 155 0 16.46 0 99.6 7:32:24 9 47 63.37 4.95 4.18 0.02 98.42 7:32:25 9 25 34 0 0.13 0 98.8 7:32:26 8 27 30 0 0.12 0 99.9 7:33:28 60199 28 30 2 0.12 0.01 99.8 Thanks, __ Jeffery Wang
RE: Solr 4.0 Optimize query very slow before the optimize end of a few minutes
Hi Otis, Sorry, it does not formatted. TimequeryTime(ms), CPU % r/s w/s rMB/s wMB/s IO % ... 7:30:24 12 89 156.44 0 16.40 94.06 7:30:25 18 91 157 0 15.35 0 98.1 7:30:26 9 91 194 0 19.62 0 96.1 7:30:27 14 38 352 0 38.17 0 100.1 7:30:28 30 77 205.94 16.83 20.17 4.0298.51 7:30:30 101 88 396 0 45.99 0 90.7 7:30:31 11 90 120 0 11.34 0 97.5 7:30:32 38 89 262.38 0 28.03 0 96.24 7:30:33 11 78 68 17 4.894.9399.9 7:30:34 9 29 201 0 20.16 0 100.3 7:30:35 9 87 181 0 17.27 0 94.3 7:30:52 16594 26 36 0 0.140 99.3 7:30:53 31 80 368 0 42.43 0 94.3 7:31:23 28575 41 35 21 0.372.3695.9 7:31:27 267660 127 0 13.76 0 83.5 7:31:28 8 59 279 0 30.99 0 99.4 7:32:22 53399 31 81 39 0.742.6399.5!!! 7:32:23 11 54 155 0 16.46 0 99.6 7:32:24 9 47 63.37 4.954.180.0298.42 7:32:25 9 25 34 0 0.130 98.8 7:32:26 8 27 30 0 0.120 99.9 7:33:28 60199 28 30 2 0.120.0199.8!! But why it always query slow at the last few minutes. I have tested it many times the optimize will last for 2 hours , almost every time, the query is quick enough(query cost about 30ms) in the 2 hours, only slow at the last few minutes(query will cost 6ms). Thanks, Jeffery -Original Message- From: Otis Gospodnetic [mailto:otis.gospodne...@gmail.com] Sent: 2013年6月14日 12:20 To: solr-user@lucene.apache.org Subject: Re: Solr 4.0 Optimize query very slow before the optimize end of a few minutes Hi, What you pasted from console didn't come across well. Yes, optimizing a static index is OK and yes, if your index is very unoptimized then yes, it will be slower than when it is optimized not sure if that addresses your concerns... Otis -- Solr ElasticSearch Support -- http://sematext.com/ On Fri, Jun 14, 2013 at 12:04 AM, Jeffery Wang jeffery.w...@morningstar.com wrote: Do someone known Why the query is very slow before the optimize end of a few minutes. When the solr optimize, I have a loop query( curl query url and sleep one second) every one second to check the query speed. It is normal, the query time can be accept. But it always very slow before the optimize end of a few minutes. The solr index size is about 22G after optimized. The follows is the query time cost, CPU and IO usage. The whole optimize process, the IO is high, it can be understand. time query time(ms) CPU % r/s w/s rMB/s wMB/s IO % 7:30:24 12 89 156.44 0 16.4 0 94.06 7:30:25 18 91 157 0 15.35 0 98.1 7:30:26 9 91 194 0 19.62 0 96.1 7:30:27 14 38 352 0 38.17 0 100.1 7:30:28 30 77 205.94 16.83 20.17 4.02 98.51 7:30:30 101 88 396 0 45.99 0 90.7 7:30:31 11 90 120 0 11.34 0 97.5 7:30:32 38 89 262.38 0 28.03 0 96.24 7:30:33 11 78 68 17 4.89 4.93 99.9 7:30:34 9 29 201 0 20.16 0 100.3 7:30:35 9 87 181 0 17.27 0 94.3 7:30:52 16594 26 36 0 0.14 0 99.3 7:30:53 31 80 368 0 42.43 0 94.3 7:31:23 28575 41 35 21 0.37 2.36 95.9 7:31:27 2676 60 127 0 13.76 0 83.5 7:31:28 8 59 279 0 30.99 0 99.4 7:32:22 53399 31 81 39 0.74 2.63 99.5 7:32:23 11 54 155 0 16.46 0 99.6 7:32:24 9 47 63.37 4.95 4.18 0.02 98.42 7:32:25 9 25 34 0 0.13 0 98.8 7:32:26 8 27 30 0 0.12 0 99.9 7:33:28 60199 28 30 2 0.12 0.01 99.8 Thanks, __ Jeffery Wang
