Greetings,
The following webrev contains general changes to improve the overall
performance of the package system:
7145683 explore general pkg performance improvements
webrev:
https://cr.opensolaris.org/action/browse/pkg/swalker/pkg-7145863/webrev/
On an Ultra 27 system (nehalem), several call profile analysis were
performed on various operations such as "pkg update -nf --no-refresh",
"pkg list -af --no-refresh", and "pkg info".
The resulting call graphs indicated that action parsing and
variant/facet inclusion evaluation are the most frequent (and
time-consuming) operations that the system performs. Version and FMRI
parsing and evaluation also figured into the call graph, but in smaller
percentages depending on the operation.
Initially, I approached performance issues by focusing on simply
rewriting existing python code using 'ptime pkg -R / update -nf
--no-refresh' and 'actionbench.py' and 'fmribench.py' as a guide. This
yielded improvements that ranged from tenths of a second to a few
seconds for each set of changes.
At that point, I had raised action processing performance from ~81K
actions per second on my Ultra 27 to ~83K actions per second.
I then started pursuing conversion of existing Python code into CPython
code. I started by merging the logic found in fromstr() into _fromstr()
in pkg._actions. This improved action parsing perforamnce from ~83K
actions per second to ~105K actions per second.
I then rewrote Action.__init__ in CPython, which improved action parsing
perfromance from ~105K actions/s to ~126K actions/s. Going further, I
rewrote FileAction.__init__ in CPython, and this increased it to ~185K
actions/s (by far the largest win).
Further work then focused on overall planning performance by reducing
the amount of work the solver was performing (by rewriting the
algorithms ued for version trimming and publisher filtering), which
reduced planning times modestly (about ~8%).
The final set of work was to rewrite allow_action() for Facets and
Variants in CPython, which yielded another ~10% reduction in planning times.
After this work was completed, it was discovered that general packaging
operation execution time had been reduced by ~10-35% (info, list, etc.),
with the biggest wins seen in planning (~26-38%) depending on platform
(sparc, x86).
Before and after call profile graphs (from my system):
pkg update -nf
https://cr.opensolaris.org/action/browse/pkg/swalker/pkg-7145863/webrev/update-nf-before.ps
https://cr.opensolaris.org/action/browse/pkg/swalker/pkg-7145863/webrev/update-nf-after.ps
pkg list -af
https://cr.opensolaris.org/action/browse/pkg/swalker/pkg-7145863/webrev/list-af-before.ps
https://cr.opensolaris.org/action/browse/pkg/swalker/pkg-7145863/webrev/list-af-after.ps
pkg info
https://cr.opensolaris.org/action/browse/pkg/swalker/pkg-7145863/webrev/info-before.ps
https://cr.opensolaris.org/action/browse/pkg/swalker/pkg-7145863/webrev/info-after.ps
As an example, on my system:
pkg -R / update -nf --no-refresh (484 updates available)
Before: 27.2s
After: 17.74s
Diff: -34.78%
pkg -R / list -af --no-refresh > /dev/null
Before: 20.72s
After: 13.63s
Diff: -34.22%
pkg -R / info > /dev/null
Before: 9.79s
After: 4.94s
Diff: -49.54%
pkg -R / info core-os > /dev/null
Before: 0.54s
After: 0.47s
Diff: -12.96%
pkg -R / uninstall -n '*emacs*' > /dev/null
Before: 12.40s
After: 9.09s
Diff: -26.69%
pkg -R / update -n --no-refresh vim > /dev/null (no update available)
Before: 9.04s
After: 7.02s
Diff: -22.35%
Likewise, action benchmark performance also greatly improved:
Items Per Second
Before After Diff
action creation:
82,125 184,634 +124.82%
action comparison:
473,962 581,362 +22.66%
minimialist comparison:
1,267,925 1,252,675 -1.20%
action to string conversion:
108,827 175,079 +60.88%
manifest contents loading:
787 1,878 +138.63%
...and fmri/version benchmark performance improved (key items only shown):
dotsequence creation:
726,943 756,900 +4.12%
dotsequence is_subsequence (true):
653,957 782,903 +19.72%
fmri is_successor (true):
640,404 655,991 +2.43%
T4 (new kodiak):
pkg -R / update -nf --no-refresh (454 updates available)
Before: 1m1s
After: 38.17s
Diff: -37.43%
pkg -R / list -af --no-refresh > /dev/null
Before: 37.34s
After: 24.14s
Diff: -35.35%
pkg -R / info > /dev/null
Before: 12.38s
After: 6.52s
Diff: -47.33%
pkg -R / info core-os > /dev/null
Before: 0.97s
After: 0.84s
Diff: -13.40%
pkg -R / uninstall -n '*emacs*' > /dev/null
Before: 18.36s
After: 13.16s
Diff: -28.32%
pkg -R / update -n --no-refresh vim > /dev/null (no update available)
Before: 14.15s
After: 10.68s
Diff: -24.52%
Action benchmark performance:
Items Per Second
Before After Diff
action creation:
41,393 89,283 +115.70%
action comparison:
198,366 248,144 +25.09%
minimialist comparison:
503,193 492,592 -2.11%
action to string conversion:
50,760 77,424 +52.53%
manifest contents loading:
419 918 +119.09%
...select fmri/version benchmark performance:
dotsequence creation:
272,811 275,935 +1.15%
dotsequence is_subsequence (true):
355,982 377,954 +6.17%
fmri is_successor (true):
287,683 288,657 +0.34%
M5000 @ 2150 MHz (kodiak-old):
ptime pkg -R / update -nf --no-refresh (no updates available)
Before: 15.61s
After: 11.97s
Diff: -23.32%
pkg -R / list -af --no-refresh > /dev/null
Before: 35.58s
After: 24.54s
Diff: -31.03%
pkg -R / info > /dev/null
Before: 22.25s
After: 12.20s
Diff: -45.17%
pkg -R / info core-os > /dev/null
Before: 1.07s
After: 0.93s
Diff: -13.08%
pkg -R / uninstall -n '*emacs*' > /dev/null
Before: 26.80s
After: 19.70s
Diff: -26.49%
pkg -R / update -n --no-refresh vim > /dev/null (no update available)
Before: 19.13s
After: 14.67s
Diff: -23.31%
Action benchmark performance:
Items Per Second
Before After Diff
action creation:
33,196 65,146 +96.25%
action comparison:
142,104 170,475 +19.96%
minimialist comparison:
379,812 371,666 -2.14%
action to string conversion:
38,616 63,730 +65.04%
manifest contents loading:
382 818 +114.14%
...select fmri/version benchmark performance:
dotsequence creation:
207,733 207,747 +0.01%
dotsequence is_subsequence (true):
257,880 286,440 +11.07%
fmri is_successor (true):
215,273 214,936 -0.16%
-Shawn
_______________________________________________
pkg-discuss mailing list
[email protected]
http://mail.opensolaris.org/mailman/listinfo/pkg-discuss
