Here's the webrev for the new search backend:
http://cr.opensolaris.org/~bpytlik/ips-search/
983: pkg search returns just one action per package/token-type combo
1949: pkg search is inordinately slow
2122: catalog file update needs to be atomic with respect to both other
updates and reads
This is a complete changes the search backend as well.
It also adds support of Unix/file glob style matching. (ie, *, ?, [char]
and [!char] are supported by using fnmatch [1])
Descriptions are now searchable, along with any "set" action in the
manifest.
The performance data is below. There is an increase in RAM requirements,
but the search performance is better, especially on local search.
Also, the large, multi-terabyte sparse file has been removed.
Future work:
Possibly offloading sorting work on main dictionary to Unix sort both
for possible speed gain and better control over RAM requirements.
Support for case insensitive matching (probably requires revving
communication protocol between server and client)
Support for boolean search (applied at both the package and action
level, ie package-and and action-and) (requires revving protocol)
Support for phrase search (probably requires revving protocol)
Support for structured searching (searching for "office" in a
description only, for example)
Revving the communication protocol to allow for more flexibility in
search criteria and results returned
Returning the entire action to the client, instead of the four pieces of
data currently returned.
Performance Info (if the columns are all over the place,try using a
fixed width font):
Wall time
server index from scratch 12h 40m 9:22.370
client index (91) from scratch 38.746s
install [EMAIL PROTECTED] real: 12:49.888 11:22.948
user: 2:52.374 3:17.613
sys: 1:12.638 1:04.843
upgrade to 91 real: 6:10.144 7:02.448
user: 2:07.217 2:52.159
sys: 21.975 24.088
local search suite pass one tot 58.579s 13.654s
local search suite pass one max 16.117s 1.630s
local search suite pass one min 1.262s 1.173s
local search suite pass two tot 13.389s 12.565s
local search suite pass two max 1.397s 1.279s
local search suite pass two min 1.268s 1.163s
remote search suite pass one tot 3.826s 4.892s
remote search suite pass one max 0.601s 1.423s
remote search suite pass one min 0.263s 0.253s
remote search suite pass two tot 2.959s 2.771s
remote search suite pass two max 0.564s 0.268s
remote search suite pass two min 0.239s 0.246s
Index time (according to dtrace)
gate search
install [EMAIL PROTECTED] 0 42s
upgrade to 91 0 121s
Server RAM footprint
gate search
install [EMAIL PROTECTED] 16M 118M
upgrade to 91 16M 118M
server index from scratch 414M
Client RAM footprint
gate search
install [EMAIL PROTECTED] 117M 198M (grows
from about 118M prior to indexing)
upgrade to 91 75M 148M (grows
from about 79M prior to indexing)
index from scratch, 91 107M
Disk footprint Index
gate search
server 246M + 3.9T 121M
client @86 19M 29M
client @91 38M 29M
Client side pkg rebuild-index (29M)
Cap Observed Memory Usage Passes Time
50 66M 4 1:02.488s (46.073) (1.988)
75 73M 3 56.335s (42.832) (1.968)
100 85M 2 47.187s (33.721) (1.663)
150 107M 1 34.635s (25.738) (1.226)
300 107M 1 35.796s (25.750) (1.247)
Sever side pkg index from scratch
Cap Observed Memory Usage Passes Time
100 159M 15 17:42.486s (16:08.920)
(28.461)
250 246M 5 11:22.274s (10:06.505)
(13.011)
500 414M 1 9:11.228s ( 8:05.560) (
5.792)
1000 414M 1 9:47.286s ( 8:10.930) (
6.745)
[1] http://docs.python.org/lib/module-fnmatch.html
_______________________________________________
pkg-discuss mailing list
[email protected]
http://mail.opensolaris.org/mailman/listinfo/pkg-discuss
