Re: UPDATING 20110730
On 08/03/2011 03:39, Andriy Gapon wrote: > on 03/08/2011 00:09 Peter Jeremy said the following: >> An alternative viewpoint is that this is wasteful because data is then >> double-buffered. > > If you stop accessing data on disk after putting it into an application cache, > then there would not be double-buffering, the OS is free to evict it from its > cache. I agree with Andriy that the OS is smart enough to manage the disk buffer cache all on its own. Meanwhile, another detail that I didn't include in previous messages that I probably should have is that portmaster is already caching a whole bunch of stuff, which is one of the reasons it's as fast as it is. In fact, because it does a lot of this caching through environment variables I have run into problems where attempting to run external commands with long command lines sometimes fails because there is not enough memory available. As a result I've been attempting to streamline both my command lines, and what I cache. I hadn't had this complaint for a while but I got one the other day, so it looks like I still have some more work to do in this area. Doug -- Nothin' ever doesn't change, but nothin' changes much. -- OK Go Breadth of IT experience, and depth of knowledge in the DNS. Yours for the right price. :) http://SupersetSolutions.com/ ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"
Re: UPDATING 20110730
on 03/08/2011 00:09 Peter Jeremy said the following: > An alternative viewpoint is that this is wasteful because data is then > double-buffered. If you stop accessing data on disk after putting it into an application cache, then there would not be double-buffering, the OS is free to evict it from its cache. > An alternative view is that the default ZFS configuration is sub-optimal and > should be fixed I agree with this. > - rather than insisting that every tool that accesses more than a handful of > files should do its own caching. But not for this reason. > (And, reading zfs-discuss, avg@ is far from the only person to have been > bitten by the ZFS metadata limit). But for this reason :-) -- Andriy Gapon ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"
Re: UPDATING 20110730
On 2011-Aug-01 19:21:21 +0200, Michel Talon wrote: >This is unfortunately impossible because the ports system is organized >around a make logic and the relevant dependency variables are only >obtained through running make on each ports Makefile *in the context* of >the gigantic makefiles (bsd.port.mk, etc) which are included. We've had this discussion before but there is plenty of scope for someone with copious free time to optimise this. Options include a new tool that handles the "easy" cases without needing to fully parse all of bsd.*.mk (and knows to punt the cases it can't handle to make) and/or pre-precessing bsd.*.mk to speed up their loading. Note that about 1/3 of bsd.*.mk is comments. On 2011-Aug-02 22:12:48 +0300, Andriy Gapon wrote: >I will repeat myself: currently portmaster's performance relies on >the fact that certain often used data originating from disk is >actually cached in memory by the OS. Typically performance-conscious >applications explicitly pull such data into an application cache. An alternative viewpoint is that this is wasteful because data is then double-buffered. An alternative view is that the default ZFS configuration is sub-optimal and should be fixed - rather than insisting that every tool that accesses more than a handful of files should do its own caching. (And, reading zfs-discuss, avg@ is far from the only person to have been bitten by the ZFS metadata limit). -- Peter Jeremy pgp4AvKy1tNvA.pgp Description: PGP signature
Re: UPDATING 20110730
On 08/02/2011 13:39, Andriy Gapon wrote: > Just a few small corrections: I was using your description of the situation. Sorry if I misunderstood. -- Nothin' ever doesn't change, but nothin' changes much. -- OK Go Breadth of IT experience, and depth of knowledge in the DNS. Yours for the right price. :) http://SupersetSolutions.com/ ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"
Re: UPDATING 20110730
on 02/08/2011 23:08 Doug Barton said the following: > On 08/02/2011 12:12, Andriy Gapon wrote: >> on 02/08/2011 21:26 Doug Barton said the following: >>> On 08/02/2011 06:14, Andriy Gapon wrote: Second, I think that portmaster could cache the origin => pkg mapping that it builds while working on port A, so that it can be readily re-used for port B. That could also include "negative" mapping where there is no installed pkg for a given origin. >>> >>> That's a reasonable idea, but moderately complex to do. I'll put it on >>> "the list" but it's not going to be a priority since in non-worst-case >>> scenarios it's generally quite fast as it is. >>> >>> Meanwhile thanks for digging further into your situation and confirming >>> that it's a local problem. >> >> Well, yes with a little bit of no. >> I will repeat myself: currently portmaster's performance relies on the fact >> that >> certain often used data originating from disk is actually cached in memory by >> the OS. > > And I will repeat myself, one last time. The assumptions that portmaster > makes are reasonable under typical conditions, but can be improved which > I will do in due course. However, your conditions are very non-typical, > including but not limited to: slow disk, small amount of ram, zfs on a > slow disk with a small amount of ram, poorly tuned zfs on a slow disk > with a small amount of ram, and a large'ish number of ports installed. Just a few small corrections: - the disks are not that slow (not sure how you deduced that): Seagate Barracuda 7200.12 - the RAM is not that small: 4GB - ZFS is not in such a bad environment as you portrayed it - ZFS is not so poorly tuned, ARC size above 1GB should be sufficient for a desktop-ish machine - the number of ports is not so large-ish for a desktop-ish machine > I get that you're disappointed with portmaster's performance under these > circumstances, and I've already said that I will do what I can, when I > can, to improve that. Hopefully I won't need to repeat myself again. :) I got this. -- Andriy Gapon ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"
Re: UPDATING 20110730
On 08/02/2011 12:12, Andriy Gapon wrote: > on 02/08/2011 21:26 Doug Barton said the following: >> On 08/02/2011 06:14, Andriy Gapon wrote: >>> Second, I think that portmaster could cache the origin => pkg mapping that >>> it >>> builds while working on port A, so that it can be readily re-used for port >>> B. >>> That could also include "negative" mapping where there is no installed pkg >>> for a >>> given origin. >> >> That's a reasonable idea, but moderately complex to do. I'll put it on >> "the list" but it's not going to be a priority since in non-worst-case >> scenarios it's generally quite fast as it is. >> >> Meanwhile thanks for digging further into your situation and confirming >> that it's a local problem. > > Well, yes with a little bit of no. > I will repeat myself: currently portmaster's performance relies on the fact > that > certain often used data originating from disk is actually cached in memory by > the OS. And I will repeat myself, one last time. The assumptions that portmaster makes are reasonable under typical conditions, but can be improved which I will do in due course. However, your conditions are very non-typical, including but not limited to: slow disk, small amount of ram, zfs on a slow disk with a small amount of ram, poorly tuned zfs on a slow disk with a small amount of ram, and a large'ish number of ports installed. I get that you're disappointed with portmaster's performance under these circumstances, and I've already said that I will do what I can, when I can, to improve that. Hopefully I won't need to repeat myself again. :) Doug -- Nothin' ever doesn't change, but nothin' changes much. -- OK Go Breadth of IT experience, and depth of knowledge in the DNS. Yours for the right price. :) http://SupersetSolutions.com/ ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"
Re: UPDATING 20110730
on 02/08/2011 21:26 Doug Barton said the following: > On 08/02/2011 06:14, Andriy Gapon wrote: >> Second, I think that portmaster could cache the origin => pkg mapping that it >> builds while working on port A, so that it can be readily re-used for port B. >> That could also include "negative" mapping where there is no installed pkg >> for a >> given origin. > > That's a reasonable idea, but moderately complex to do. I'll put it on > "the list" but it's not going to be a priority since in non-worst-case > scenarios it's generally quite fast as it is. > > Meanwhile thanks for digging further into your situation and confirming > that it's a local problem. Well, yes with a little bit of no. I will repeat myself: currently portmaster's performance relies on the fact that certain often used data originating from disk is actually cached in memory by the OS. Typically performance-conscious applications explicitly pull such data into an application cache. But practice is the main criterion. -- Andriy Gapon ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"
Re: UPDATING 20110730
On 08/02/2011 06:14, Andriy Gapon wrote: > Second, I think that portmaster could cache the origin => pkg mapping that it > builds while working on port A, so that it can be readily re-used for port B. > That could also include "negative" mapping where there is no installed pkg > for a > given origin. That's a reasonable idea, but moderately complex to do. I'll put it on "the list" but it's not going to be a priority since in non-worst-case scenarios it's generally quite fast as it is. Meanwhile thanks for digging further into your situation and confirming that it's a local problem. Doug -- Nothin' ever doesn't change, but nothin' changes much. -- OK Go Breadth of IT experience, and depth of knowledge in the DNS. Yours for the right price. :) http://SupersetSolutions.com/ ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"
portmaster, zfs metadata caching [Was: UPDATING 20110730]
on 02/08/2011 16:14 Andriy Gapon said the following: > And now to my side of the problem. > While "profiling" pkg_info with ktrace I see getdirentries(2) calls sometimes > take quite a while. And since I have > 1000 ports all those calls do add up. > DTrace shows that the calls are quite fast (~0.3 ms) when there is no actual > disk access, but if it occurs then it introduces a delay on the orders of 1 - > 100 milliseconds. > I am really in doubts about what is happening here. It seems that all the > directory data isn't kept in ZFS ARC for long enough or is squeezed out of it > by > some other data (without additional pressure it should easily fit into the > ARC). > And also that somehow disk accesses have quite large latency. Although > svc_t > according to iostat is smaller (5 - 10 ms). DTrace shows that the thread > spends > the time in cv_wait. So it's possible that the scheduler is also involved > here > as its decisions also may add a delay to when the thread becomes runnable > again. Reporting further, just in case anyone follows this. (You may want to scroll down to my conclusions at the end of the message). I tracked my ZFS problem to my experiments with ZFS tuning. I limited my ARC size at some value that I considered to be large enough to cache my working sets of data and _metadata_. Little did I know that by default ZFS sets aside only 1/4th of ARC size for metadata. So this is already significantly smaller than I expected. Then it seems that a large piece of that metadata portion is permanently occupied by some non-evict-able data (not sure what it actually is, haven't tracked yet). In the end only a small portion of my ARC was available for holding the metadata which included the directory contents data. So this is what I had with the old settings: vfs.zfs.arc_meta_limit: 314572800 vfs.zfs.arc_meta_used: 401064272 and $ for i in $(jot 5) ; do /usr/bin/time -p pkg_info -O print/printme ; done The following installed package(s) has print/printme origin: real 12.55 user 0.02 sys 2.51 The following installed package(s) has print/printme origin: real 12.65 user 0.03 sys 1.99 The following installed package(s) has print/printme origin: real 10.57 user 0.02 sys 1.57 The following installed package(s) has print/printme origin: real 8.85 user 0.03 sys 0.17 The following installed package(s) has print/printme origin: real 9.28 user 0.02 sys 0.20 I think that you should get the picture. Now I have bumped the limit and this is what I had just right after doing it: vfs.zfs.arc_meta_limit: 717225984 vfs.zfs.arc_meta_used: 414439800 and $ for i in $(jot 5) ; do /usr/bin/time -p pkg_info -O print/printme ; done The following installed package(s) has print/printme origin: real 9.08 user 0.01 sys 0.18 The following installed package(s) has print/printme origin: real 7.48 user 0.04 sys 0.14 The following installed package(s) has print/printme origin: real 0.08 user 0.00 sys 0.07 The following installed package(s) has print/printme origin: real 0.95 user 0.03 sys 0.04 The following installed package(s) has print/printme origin: real 0.08 user 0.00 sys 0.07 Two runs to "warm up" the ARC and then everything works just perfect. I think that this is an important discovery for two reason: 1. I learned a new thing about ZFS ARC. 2. This problem demonstrates that portmaster currently does depend on a filesystem cache being able to hold a significant amount of ports/packages (meta)data. -- Andriy Gapon ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"
Re: UPDATING 20110730
on 31/07/2011 22:14 Doug Barton said the following: > Understood, but I'm not sure what I can do about it unless I can > reproduce it. I reproduced the problem by doing a similar kind of upgrade: $ pkg_delete -f gtk-2.\* $ portmaster -v atk-2.0.1 gio-fam-backend-2.28.8 glib-2.28.8 gobject-introspection-0.10.8 x11-toolkits/gtk20 Now I see some duality in the problem. First, let's consider the portmaster side. Suppose we have two ports A and B that both depend on both of two other ports C and D. When portmaster upgrades port C it updates dependency information in ports A and B, and while doing that it also checks (and potentially updates) dependency information for port D. First of all, it's not clear if that check for port D is really required. Second, I think that portmaster could cache the origin => pkg mapping that it builds while working on port A, so that it can be readily re-used for port B. That could also include "negative" mapping where there is no installed pkg for a given origin. Why this caching could be useful becomes more evident if there are hundreds of ports in the A, B, ... class and hundreds of ports in the C, D, ... class. As it stands now, the above invocation of portmaster searches for an installed package with an origin of x11-toolkits/gtk20 for 714 times. Each search includes grepping */+CONTENTS in /var/db/pkg directory and also executing pkg_info -q -O x11-toolkits/gtk20. As far as I can see, each such pkg_info invocation also performs something very similar to the grep action: it calls getdirentries(2) on each port subdirectory and then reads +CONTENTS from it. What I further observe is those pkg_info calls are sometimes quite fast, sometimes slow and sometimes are very slow like couple dozen seconds. Given that there are ~700 pkg_info runs all those delays add up to quite a long period. And now to my side of the problem. While "profiling" pkg_info with ktrace I see getdirentries(2) calls sometimes take quite a while. And since I have > 1000 ports all those calls do add up. DTrace shows that the calls are quite fast (~0.3 ms) when there is no actual disk access, but if it occurs then it introduces a delay on the orders of 1 - 100 milliseconds. I am really in doubts about what is happening here. It seems that all the directory data isn't kept in ZFS ARC for long enough or is squeezed out of it by some other data (without additional pressure it should easily fit into the ARC). And also that somehow disk accesses have quite large latency. Although svc_t according to iostat is smaller (5 - 10 ms). DTrace shows that the thread spends the time in cv_wait. So it's possible that the scheduler is also involved here as its decisions also may add a delay to when the thread becomes runnable again. -- Andriy Gapon ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"
Re: UPDATING 20110730
On Mon, Aug 01, 2011 at 09:39:05AM -0700, Jos Backus wrote: > On Aug 1, 2011 7:10 AM, "Michel Talon" wrote: > [snip] > > This being said if an upgrade tool needs to compute (partially) the INDEX, > > most of the time is spent in running make -V in each port, > > because make has to read and interpret enormous files. I don't see any way > to > > cut on that, or one should need to develop a special purpose version of > make > > to evaluate these variables, perhaps which should keep persistent > > computations between ports (but this is dangerous). > > Or don't store lots of data in files in Makefile format. The make language > is a poor data storage format that doesn't allow access to that data from > other tools easily or efficiently. I'm struggling with a similar problem at > $WORK. > > Jos This is unfortunately impossible because the ports system is organized around a make logic and the relevant dependency variables are only obtained through running make on each ports Makefile *in the context* of the gigantic makefiles (bsd.port.mk, etc) which are included. -- Michel TALON ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"
Re: UPDATING 20110730
On Mon, Aug 01, 2011 at 01:59:08PM -0300, Sergio de Almeida Lenzi wrote: > Hello > > Even I use portmaster (a very good piece of software), > it becomes very slow when you have 1550 ports installed in your > system. > > As only a few ports (about 100, in my case) changes in a week time, > I build a database (postgres) that contains all the ports installed, > de depencies and a flag that tells me if that port needs updating > (pkg_version) > a shell script scans the ports (pkg_info | cut -d ' ' -f 1) and builds > the database once a week (can take several hours... > > Once the database is built, an sql query (only ms...) tells me what to > do... > it then executes pkg_delete, cd /usr/ports/..., make clean all package.. > and after doing all the job, it updates the postgresql database > (seconds... ). > > In my case I use a central server with all the 1550 ports... and all I > do > is to install them on the slaves, (again, using the postgres database > data)... > > Hope this can give someone some ideas > > Sergio Some years ago the idea floated around to use a sqlite database to keep a fast access copy of the important data in /var/db/pkg, but this idea was dismissed for "various" reasons, in particular the fact that the base system has the Berkeley database, or that using the filesystem as a poor man's database was a better idea. -- Michel TALON ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"
Re: UPDATING 20110730
On Aug 1, 2011 7:10 AM, "Michel Talon" wrote: [snip] > This being said if an upgrade tool needs to compute (partially) the INDEX, > most of the time is spent in running make -V in each port, > because make has to read and interpret enormous files. I don't see any way to > cut on that, or one should need to develop a special purpose version of make > to evaluate these variables, perhaps which should keep persistent > computations between ports (but this is dangerous). Or don't store lots of data in files in Makefile format. The make language is a poor data storage format that doesn't allow access to that data from other tools easily or efficiently. I'm struggling with a similar problem at $WORK. Jos ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"
Re: UPDATING 20110730
Hello Even I use portmaster (a very good piece of software), it becomes very slow when you have 1550 ports installed in your system. As only a few ports (about 100, in my case) changes in a week time, I build a database (postgres) that contains all the ports installed, de depencies and a flag that tells me if that port needs updating (pkg_version) a shell script scans the ports (pkg_info | cut -d ' ' -f 1) and builds the database once a week (can take several hours... Once the database is built, an sql query (only ms...) tells me what to do... it then executes pkg_delete, cd /usr/ports/..., make clean all package.. and after doing all the job, it updates the postgresql database (seconds... ). In my case I use a central server with all the 1550 ports... and all I do is to install them on the slaves, (again, using the postgres database data)... Hope this can give someone some ideas Sergio ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"
Re: UPDATING 20110730
Le Monday 01 August 2011, Doug wrote: > > A lot of people say that, but I'll stack it up against just about any > interpreted language. Some of my routines are actually faster than the > equivalents in pkg_info (which is why I use them). > Yes, i have seen that portmaster is quite fast. I was meaning that shell scripting is not the clearest tool to program complex stuff, but of course this is dependant on each person. As for the pkg* stuff they are written in C, but this is irrelevant enough if they do a lot of IO, or use poorly performing algos. I remember that Marc Espie said that, after having rewritten the OpenBSD equivalents in perl, they were both clearer and more powerful, and much faster. The slowness gripe i have is about portupgrade. This is particularly obvious when running portupgrade -PP, which may take hours to upgrade a machine without spending any time in compilation. As far as i have understood the pkg* tools are presently being rewritten by a FreeBSD team, i hope the new tools will be much better. This being said if an upgrade tool needs to compute (partially) the INDEX, most of the time is spent in running make -V in each port, because make has to read and interpret enormous files. I don't see any way to cut on that, or one should need to develop a special purpose version of make to evaluate these variables, perhaps which should keep persistent computations between ports (but this is dangerous). ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"
Re: UPDATING 20110730
On 08/01/2011 01:51, Michel Talon wrote: > Doug wrote: >> Unfortunately the only way to improve on this would be to not do the >> checks on a port-by-port basis, and do them all together at the end. >> While that sounds appealing, it would dramatically increase the code >> complexity, and also dramatically increase the chances of leaving the >> pkg files in an inconsistent state if the process gets interrupted. I >> don't like either one of those options. > > In here i have a program which checks the +REQUIRED_BY files and > fixes the origins in +CONTENTS > http://www.lpthe.jussieu.fr/~talon/check_pkg.py > proceeding globally as you describe above. It would not make a big > difference to completely fix the +CONTENTS. FYI, what portmaster is doing is fixing the pkgdep lines in +CONTENTS, and updating +REQUIRED_BY as needed. > On an old machine with around +1000 ports installed it takes of the > order of 10-30 seconds to run. Of course this is very fast because it > uses the information in the INDEX file. 2 problems, obviously portmaster is doing more work, and it's not using the INDEX file by default. Without using INDEX with 600 ports installed portmaster --check-depends takes less than a minute. Using INDEX actually takes about 1:20, but that's because the way that portmaster accesses the INDEX file isn't really optimized for hundreds of reads by the same process. > If one accepts to download the > INDEX (like portupgrade does) this is no problem. If one wants to rebuild > the index from the ports, it takes time, but one can build a partial > index for the installed ports (and dependencies). This is done in > http://www.lpthe.jussieu.fr/~talon/pkgupgrade > and takes someting like 1-2mn on the same machine. Either of which takes more time. :) > So there are ways to speed up the bookeeping done by programs like > portupgrade, portmaster, but, as you are saying, doing this job between > *each* port upgrade is far more time consuming. Just to be clear, what portmaster does after installing a port is to take care of +CONTENTS and +REQUIRED_BY only for the relevant files. --check-depends does everything. > Of course the complexity > is also increased, perhaps shell scripting is not the good tool to do that A lot of people say that, but I'll stack it up against just about any interpreted language. Some of my routines are actually faster than the equivalents in pkg_info (which is why I use them). Doug -- Nothin' ever doesn't change, but nothin' changes much. -- OK Go Breadth of IT experience, and depth of knowledge in the DNS. Yours for the right price. :) http://SupersetSolutions.com/ ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"
Re: UPDATING 20110730
Doug wrote: > Unfortunately the only way to improve on this would be to not do the > checks on a port-by-port basis, and do them all together at the end. > While that sounds appealing, it would dramatically increase the code > complexity, and also dramatically increase the chances of leaving the > pkg files in an inconsistent state if the process gets interrupted. I > don't like either one of those options. In here i have a program which checks the +REQUIRED_BY files and fixes the origins in +CONTENTS http://www.lpthe.jussieu.fr/~talon/check_pkg.py proceeding globally as you describe above. It would not make a big difference to completely fix the +CONTENTS. On an old machine with around +1000 ports installed it takes of the order of 10-30 seconds to run. Of course this is very fast because it uses the information in the INDEX file. If one accepts to download the INDEX (like portupgrade does) this is no problem. If one wants to rebuild the index from the ports, it takes time, but one can build a partial index for the installed ports (and dependencies). This is done in http://www.lpthe.jussieu.fr/~talon/pkgupgrade and takes someting like 1-2mn on the same machine. So there are ways to speed up the bookeeping done by programs like portupgrade, portmaster, but, as you are saying, doing this job between *each* port upgrade is far more time consuming. Of course the complexity is also increased, perhaps shell scripting is not the good tool to do that i don't know. Moreover i am not convinced that continually forking tons of programs can be very fast, and it would be nice to be able to exploit parallelism on modern multiproc machines. -- Michel TALON ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"
Re: UPDATING 20110730
On 08/01/2011 05:09, per...@pluto.rain.com wrote: > Andriy Gapon wrote: > >> If for X ports all the relevant data under /var/db/pkg fits into >> fs cache, then the performance may be blazing, but once you exceed >> the cache size the performance might become totally different. > > and/or the poorly-performing system may have enough less memory than > the other for paging/swapping to be a factor. I may not be the smartest guy in the project, but I think that you(pl.) can reasonably assume that I understand something as fundamental as "data set fits in RAM means it goes fast." :) You can also safely assume that I understand that if portmaster's work causes the cache to overflow, or the user gets stuck behind a combination of slow disk and/or a small amount of RAM that performance is going to suck. My point was simply that there isn't anything I can do about it. Making sure that the +CONTENTS and +REQUIRED_BY files are up to date is an important part of portmaster's core functionality. Unfortunately the only way to improve on this would be to not do the checks on a port-by-port basis, and do them all together at the end. While that sounds appealing, it would dramatically increase the code complexity, and also dramatically increase the chances of leaving the pkg files in an inconsistent state if the process gets interrupted. I don't like either one of those options. Doug -- Nothin' ever doesn't change, but nothin' changes much. -- OK Go Breadth of IT experience, and depth of knowledge in the DNS. Yours for the right price. :) http://SupersetSolutions.com/ ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"
Re: UPDATING 20110730
Andriy Gapon wrote: > If for X ports all the relevant data under /var/db/pkg fits into > fs cache, then the performance may be blazing, but once you exceed > the cache size the performance might become totally different. and/or the poorly-performing system may have enough less memory than the other for paging/swapping to be a factor. ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"
Re: UPDATING 20110730
on 31/07/2011 22:14 Doug Barton said the following: > On 07/31/2011 04:19, Andriy Gapon wrote: >> on 31/07/2011 04:32 Doug Barton said the following: > >>> I am not sure what you mean by "inside portmaster" is quite accurate. I >>> followed the instructions and everything worked according to plan. The >>> vast majority of the wall clock time spent following these instructions >>> is in the compilation of the various ports. >> >> Well, then we have different experiences and maybe environments. >> I am quite sure that more than 1 hour of wall time was spent in portmaster >> proper after portmaster performed upgrade of gio-fam-backend-2.28.8 and >> before >> portmaster had a a chance to proceed to the next port > > That's very odd. It was moments for me between each port on my middle of > the road laptop. > >> Some stats about my ports: >> $ l /var/db/pkg/ | wc -l >> 1088 >> $ pkg_info -R gio-fam-backend-2.28.8| wc -l >> 27 >> $ pkg_info -R gtk-2.24.5 | wc -l >> 132 >> $ pkg_info -R gobject-introspection-0.10.8| wc -l >> 219 > > You have about twice as many ports installed as I do, and that's a lot > depending on gobject-introspection. But still an hour is very disturbing. > >> I am not complaining about the messages. >> My complaint is about the performance of handling this case. > > Understood, but I'm not sure what I can do about it unless I can > reproduce it. I understand. Another thing to keep in mind is that the performance might not scale linearly with a number of relevant ports. If for X ports all the relevant data under /var/db/pkg fits into fs cache, then the performance may be blazing, but once you exceed the cache size the performance might become totally different. -- Andriy Gapon ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"
Re: UPDATING 20110730
On 07/31/2011 04:19, Andriy Gapon wrote: > on 31/07/2011 04:32 Doug Barton said the following: >> I am not sure what you mean by "inside portmaster" is quite accurate. I >> followed the instructions and everything worked according to plan. The >> vast majority of the wall clock time spent following these instructions >> is in the compilation of the various ports. > > Well, then we have different experiences and maybe environments. > I am quite sure that more than 1 hour of wall time was spent in portmaster > proper after portmaster performed upgrade of gio-fam-backend-2.28.8 and before > portmaster had a a chance to proceed to the next port That's very odd. It was moments for me between each port on my middle of the road laptop. > Some stats about my ports: > $ l /var/db/pkg/ | wc -l > 1088 > $ pkg_info -R gio-fam-backend-2.28.8| wc -l > 27 > $ pkg_info -R gtk-2.24.5 | wc -l > 132 > $ pkg_info -R gobject-introspection-0.10.8| wc -l > 219 You have about twice as many ports installed as I do, and that's a lot depending on gobject-introspection. But still an hour is very disturbing. > I am not complaining about the messages. > My complaint is about the performance of handling this case. Understood, but I'm not sure what I can do about it unless I can reproduce it. Doug -- Nothin' ever doesn't change, but nothin' changes much. -- OK Go Breadth of IT experience, and depth of knowledge in the DNS. Yours for the right price. :) http://SupersetSolutions.com/ ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"
Re: UPDATING 20110730
on 31/07/2011 04:32 Doug Barton said the following: > On 07/30/2011 12:38, Andriy Gapon wrote: >>> 20110730: >>> AFFECTS: users of x11-toolkits/gtk20 >>> AUTHOR: gn...@freebsd.org >>> >>> The gtk-update-icon-cache utility has been slipt out of the gtk20 port. >> >> A minor typo in the above line - slipt -> split. >> >>> Use the following instructions to update your system. >>> >>> # pkg_delete -f gtk-2.\* >>> # portmaster x11-toolkits/gtk20 >> >> I would like to warn other users and at the same time ask for alternative >> instructions. > > I don't think that's necessary. Well, I decided to report what I experienced firsthand. >> I have a regular (consumer) HDD and not so huge amount of RAM. >> I also have many (maybe very many) ports depending on gtk20 and some other >> loosely related ports like glib and gobject-introspection. >> Execution of the above command already takes more than an hour which is spent >> inside portmaster. > > I am not sure what you mean by "inside portmaster" is quite accurate. I > followed the instructions and everything worked according to plan. The > vast majority of the wall clock time spent following these instructions > is in the compilation of the various ports. Well, then we have different experiences and maybe environments. I am quite sure that more than 1 hour of wall time was spent in portmaster proper after portmaster performed upgrade of gio-fam-backend-2.28.8 and before portmaster had a a chance to proceed to the next port - I pressed ^C at that moment. I pressed ^T quite a few times during that hour and portmaster was actually running grep or pkg_info at those times, mostly pkg_info. E.g.: load: 0.25 cmd: pkg_info 49079 [zio->io_cv)] 10.83r 0.01u 0.25s 1% 2856k load: 0.23 cmd: pkg_info 49093 [zio->io_cv)] 4.13r 0.00u 0.09s 0% 2660k load: 0.29 cmd: grep 49324 [zio->io_cv)] 1.37r 0.00u 0.03s 0% 1756k Some stats about my ports: $ l /var/db/pkg/ | wc -l 1088 $ pkg_info -R gio-fam-backend-2.28.8| wc -l 27 $ pkg_info -R gtk-2.24.5 | wc -l 132 $ pkg_info -R gobject-introspection-0.10.8| wc -l 219 > Very very little is actually > spent "in portmaster" in the sense of time spent by portmaster > performing its functions. > >> I have hundreds of lines like the following in portmaster output: >> >> ===>>> x11-toolkits/gtk20 is listed as a dependency >> ===>>> but there is no installed version > > Yes, that's to be expected since it's accurate. :) Every time > portmaster installs a port it updates the +CONTENTS and +REQUIRED_BY > files as appropriate. If the condition described above exists it lets > you know, but since this is almost always a transient error it continues > processing which is usually what is necessary to correct the problem > anyway. > > I've considered adding code to avoid the spurious warnings but they are > harmless and don't happen very often so I haven't bothered. > > In short, I don't think there is anything wrong here. I am not complaining about the messages. My complaint is about the performance of handling this case. Here's what I did after ^C. $ cd /usr/ports/devel/gobject-introspection $ make install $ cd /usr/ports/x11-toolkits/gtk20 $ make install $ portmaster x11-toolkits/gtk20 devel/gobject-introspection The last portmaster command decided to upgrade almost the same set of ports and did that much much faster in my environment. Here's what the original portmaster run intended to do: ===>>> The following actions will be taken if you choose to proceed: Install x11-toolkits/gtk20 Upgrade atk-1.32.0 to atk-2.0.1 Upgrade gio-fam-backend-2.26.1 to gio-fam-backend-2.28.8 Upgrade glib-2.26.1_1 to glib-2.28.8 Upgrade gobject-introspection-0.9.12_1 to gobject-introspection-0.10.8 Upgrade gdk-pixbuf-2.22.1 to gdk-pixbuf-2.23.5 Install graphics/gtk-update-icon-cache Upgrade pango-1.28.3 to pango-1.28.4 Here's what portmaster intended to do on the second run: ===>>> The following actions will be taken if you choose to proceed: Re-install gtk-2.24.5 Upgrade atk-1.32.0 to atk-2.0.1 Re-install gobject-introspection-0.10.8 Upgrade gdk-pixbuf-2.22.1 to gdk-pixbuf-2.23.5 Upgrade pango-1.28.3 to pango-1.28.4 I am not sure how to distill the above into a minimalistic test-case. But it seems to me that portmaster spends too much time churning away when there are a lot of installed ports/packages with a missing dependency. I guess it's something in update_contents() or iport_from_origin() or in code that calls update_contents(). -- Andriy Gapon ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"
Re: UPDATING 20110730
On 07/30/2011 12:38, Andriy Gapon wrote: >> 20110730: >> AFFECTS: users of x11-toolkits/gtk20 >> AUTHOR: gn...@freebsd.org >> >> The gtk-update-icon-cache utility has been slipt out of the gtk20 port. > > A minor typo in the above line - slipt -> split. > >> Use the following instructions to update your system. >> >> # pkg_delete -f gtk-2.\* >> # portmaster x11-toolkits/gtk20 > > I would like to warn other users and at the same time ask for alternative > instructions. I don't think that's necessary. > I have a regular (consumer) HDD and not so huge amount of RAM. > I also have many (maybe very many) ports depending on gtk20 and some other > loosely related ports like glib and gobject-introspection. > Execution of the above command already takes more than an hour which is spent > inside portmaster. I am not sure what you mean by "inside portmaster" is quite accurate. I followed the instructions and everything worked according to plan. The vast majority of the wall clock time spent following these instructions is in the compilation of the various ports. Very very little is actually spent "in portmaster" in the sense of time spent by portmaster performing its functions. > I have hundreds of lines like the following in portmaster output: > > ===>>> x11-toolkits/gtk20 is listed as a dependency > ===>>> but there is no installed version Yes, that's to be expected since it's accurate. :) Every time portmaster installs a port it updates the +CONTENTS and +REQUIRED_BY files as appropriate. If the condition described above exists it lets you know, but since this is almost always a transient error it continues processing which is usually what is necessary to correct the problem anyway. I've considered adding code to avoid the spurious warnings but they are harmless and don't happen very often so I haven't bothered. In short, I don't think there is anything wrong here. Doug -- Nothin' ever doesn't change, but nothin' changes much. -- OK Go Breadth of IT experience, and depth of knowledge in the DNS. Yours for the right price. :) http://SupersetSolutions.com/ ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"
UPDATING 20110730
> 20110730: > AFFECTS: users of x11-toolkits/gtk20 > AUTHOR: gn...@freebsd.org > > The gtk-update-icon-cache utility has been slipt out of the gtk20 port. A minor typo in the above line - slipt -> split. > Use the following instructions to update your system. > > # pkg_delete -f gtk-2.\* > # portmaster x11-toolkits/gtk20 I would like to warn other users and at the same time ask for alternative instructions. I have a regular (consumer) HDD and not so huge amount of RAM. I also have many (maybe very many) ports depending on gtk20 and some other loosely related ports like glib and gobject-introspection. Execution of the above command already takes more than an hour which is spent inside portmaster. I have hundreds of lines like the following in portmaster output: ===>>> x11-toolkits/gtk20 is listed as a dependency ===>>> but there is no installed version ===>>> Try portmaster --check-depends ===>>> devel/gobject-introspection is listed as a dependency ===>>> but there is no installed version ===>>> Try portmaster --check-depends ===>>> devel/gobject-introspection is listed as a dependency ===>>> but there is no installed version ===>>> Try portmaster --check-depends This carpet of output starts with: ===>>> Updating dependency entry for gio-fam-backend-2.28.8 in each dependent port > # portmaster -a -- Andriy Gapon ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"
