On 03/10/10 08:45 PM, Shawn Walker wrote: > On 03/10/10 02:21 AM, Jan Damborsky wrote: > ... >> As you pointed out above, we know what the problem is in #1- there is a >> scalability >> regression in pkg CLI which slows down installations in AI and DC (as >> Keith reported >> in comment #30 for bug 13321), since transfer module calls 'pkg install' >> for each package. > > I believe it's logical to assume, regardless of what command you're > executing, that executing a single command with all expected input > once instead of many times is going to be more efficient.
I understand, but I also believe this should be perceived as standard performance versus consumed resources trade off. For instance, running 'pkg install babel_install' on build 134 consumes 170 MB of heap and ~850MiB of filesystem space (/var/pkg/download cache). On the other hand, it is pretty fast comparing to 'per package' pkg install. Based on this, I am not sure that non-batched approach should be completely omitted as far as scalability aspect is concerned - I can see that there might be cases (e.g. bug 9076) where installation time is lower priority than consumed resources. That assumes that the loss of performance is acceptable. Since AI and DC have been using this approach since beginning, I am not sure about is why this issue is perceived as a blocker right now. Might it be because performance of that approach visibly degraded in some recent build ? In order to obtain some data, I have tried to examine pkg transfer phase (pkg client - pkg server interaction) while trying to eliminate impact of networking and AI environment as well as avoid potential running out of resources (CPU, memory). I tried to run x86 DC AI im-pop step which is equivalent of AI pkg install phase. Package list contains ~200 packages. IPS repository and DC build were running on the same machine (thus eliminating impact of networking) and with sufficient resources (x86, 8 core CPU, 12GiB memory). Tests were done on build 128, 130, 134. Local IPS repositories containing build 128 and 130 (they were built by means of 'make redist_import') were used. 128 on host running build 128 and 130 on host running build 130 and 134. Tests were done twice (round 1, round 2). Machine was rebooted before each test and it was made sure it is in idle state before test itself was started. For comparison, time of batched 'pkg install' was measured as well. Results (times in MM:SS): host | build |DC im-pop|DC im-pop| batched running|installed| test 1 | test 2 | 'pkg install' ---------------------------------------------------- 128 | 128 | 6:30 | 6:31 | 1:30 130 | 130 | 11:04 | 10:50 | 1:40 134 | 130 | 11:08 | 11:18 | 1:10 ---------------------------------------------------- From this it seems that performance of pkg client-server chain improved in build 134 comparing to previous builds in case of batched 'pkg install' while the opposite is true for non-batched 'pkg install'. Thank you, Jan
