Otis Gospodnetic wrote:
Perhaps the container logs explain what happened?
How about just throttling to the point where the failure rate is 0%?
Too slow?
Otis's questions regarding dropped inserts sent me back to the drawing
board. The system had been tuned to a slower database to optimize speed
and accept a few drops. When I migrated to a faster DB I didn't
retune. Here are results of testing indexing performance for Tomcat and
Jetty. The DB speedup apparently moved the bottleneck from getting
records from the database (around 400 rps) to cramming records into the
servlet container.
System: 16 processor, 2.5 ghz, 64G memory
Index: 33 Gig, freshly optimized, avg recordsize 1.4k
Insert load: 250,000 records
I calculate records/sec by dividing the number of successful inserts by
the time. The adjusted time is the estimated time it would take to
insert the full 250,000 records with no errors, which is raw time plus
the additional time required to insert those dropped records, ie, raw
time * (1 + error-rate * 0.01). Judging from processor/memory/io
utilization, it appears the write speed of a single java thread is
dominating the solr indexing speed. Which makes sense.
Takehome lessons:
The speed limit is about 450 records per second in our environment.
Three or four threads posting inserts max out speed. More threads
don't help.
Jetty is significantly faster than Tomcat at sane thread counts in
our environment
I hope this is useful.
-Jim
PS: If you have formatting issues with this table, try viewing with a
fixed width font
Tomcat
Jetty
_____________________________________________________________
____________________________________________________________
# threads Raw time # Drops % Error Records/sec
Adj. time Raw time # Drops % Error Records/sec Adj. time
16 533 17131 6.85
436.9 569.51 594 24222 9.69 380.1 651.55
15 520 16878 6.75 448.31
555.1 518 28581 11.43 427.45 577.22
14 547 16378 6.55
427.1 582.83 496 30047 12.02 443.45 555.61
13 540 16638 6.65
432.15 575.91 495 27076 10.83 450.35 548.61
12 545 15920 6.36
429.5 579.66 494 28785 11.51 447.8 550.88
11 523 16192 6.47 447.05
556.84 484 26495 10.6 461.79 535.29
10 540 15643 6.26 433.99
573.8 497 27190 10.88 448.31 551.05
9 553 15543 6.21
423.97 587.34 494 25862 10.34 453.72 545.1
8 541 14095 5.64
436.05 571.51 501 23482 9.39 452.13 548.06
7 549 10735 4.29 435.82
572.55 499 24657 9.86 451.59 548.22
6 566 9468 3.79
424.97 587.45 502 23074 9.23 452.04 548.33
5 588 7754 3.10
411.98 606.23 527 20779 8.31 434.95 570.8
4 577 4201 1.68
425.99 586.69 513 16608 6.64 454.96 547.08
3 613 0 0
407.83 613 537 9503 3.8 447.85 557.41
2 801 0 0
312.11 801 633 0 0 394.94 633
1 1365 0 0
183.15 1365 1122 0 0 222.82 1122
