Re: A new and better way to do make readmes?
On Fri, 27 Jan 2012 20:03:25 -0600 Conrad J. Sabatier wrote: I've been thinking for a long time that we need a better way to do make readmes, one that would be properly integrated into our ports Mk infrastructure, to take advantage of make's ability to recognize which files are up-to-date and which really do need rebuilding. This wont help and I think there's a better way that will make it up to 700 times faster. When a make readmes is done at the top-level, the top-level and category READMEs are created by make targets and the per port READMEs are created by a perl script in one go from the INDEX-n file. I once timed this and the 64 category READMEs took 2 hours, but the ~20,000 port READMEs only took about 9 seconds. Selective updating isn't going to help because 99.9% of the time is spent in the categories and it only takes a single port update to make a category file obsolete. I think the way to speed this up is to have the script generate the category files too. There's no point in bringing in the top-level README since that's already fast. I've been toying with the idea of doing this, but have never got around to it. If anyone wants to have a go I think it would be sensible to write it in awk, since perl is no longer in the base system and the existing perl script isn't really complex enough to be worth hanging-on to. ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org
Re: A new and better way to do make readmes?
On Sat, 28 Jan 2012 14:37:34 + RW rwmailli...@googlemail.com wrote: On Fri, 27 Jan 2012 20:03:25 -0600 Conrad J. Sabatier wrote: I've been thinking for a long time that we need a better way to do make readmes, one that would be properly integrated into our ports Mk infrastructure, to take advantage of make's ability to recognize which files are up-to-date and which really do need rebuilding. This wont help and I think there's a better way that will make it up to 700 times faster. When a make readmes is done at the top-level, the top-level and category READMEs are created by make targets and the per port READMEs are created by a perl script in one go from the INDEX-n file. I once timed this and the 64 category READMEs took 2 hours, but the ~20,000 port READMEs only took about 9 seconds. rubbing eyes in disbelief Am I understanding you correctly? Are you saying you built 20,000+ port READMEs in only 9 seconds?! How is that possible? Or do you mean 9 seconds for each one? Selective updating isn't going to help because 99.9% of the time is spent in the categories and it only takes a single port update to make a category file obsolete. This is the part I find troubling. It would seem that it should be more work to create an individual port README, with its plucking the appropriate line out of the INDEX-* file and then parsing it into its respective pieces and filling in a template, than to simply string together a list of references to a bunch of already built port READMEs into a category README. What am I not getting here? I think the way to speed this up is to have the script generate the category files too. There's no point in bringing in the top-level README since that's already fast. So what's making the category READMEs so slow then? I've been toying with the idea of doing this, but have never got around to it. If anyone wants to have a go I think it would be sensible to write it in awk, since perl is no longer in the base system and the existing perl script isn't really complex enough to be worth hanging-on to. Oooo, awk! Been a while since I wrote any sizeable bit of code in it, but I do remember it was rather fun to work with. :-) I'm still not sure I read that paragraph above correctly, though (re: the times). :-) -- Conrad J. Sabatier conr...@cox.net ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org
Re: A new and better way to do make readmes?
On 28/01/2012 16:28, Conrad J. Sabatier wrote: rubbing eyes in disbelief Am I understanding you correctly? Are you saying you built 20,000+ port READMEs in only 9 seconds?! How is that possible? Or do you mean 9 seconds for each one? 9 seconds sounds quite reasonable for generating 23000 or so files. Selective updating isn't going to help because 99.9% of the time is spent in the categories and it only takes a single port update to make a category file obsolete. This is the part I find troubling. It would seem that it should be more work to create an individual port README, with its plucking the appropriate line out of the INDEX-* file and then parsing it into its respective pieces and filling in a template, than to simply string together a list of references to a bunch of already built port READMEs into a category README. What am I not getting here? No -- you're quite right. You could generate the category README.html files entirely from the data in the INDEX. It's not quite as easy as all that, because there aren't entries for each category separately, so you'll have to parse the structure out of all of the paths in the INDEX. I think the way to speed this up is to have the script generate the category files too. There's no point in bringing in the top-level README since that's already fast. So what's making the category READMEs so slow then? The big problem with performance in all this INDEX and README.html building is that it takes quite a long time relatively to run make(1) within any port or category directory. make(1) has to read in a lot of other files and stat(2) many more[*] -- all of which involves a lot of random-access disk IO, and that's always going to take quite a lot of time. Now, doing 'make readme' in a category directory doesn't just run make in that directory, but also in every port in that category. Popular categories can contain many hundreds of ports. Maybe I should add README.html generation to my FreeBSD::Portindex stuff. Should be pretty simple -- all the necessary bits are readily available and it is just a matter of formatting it as HTML and printing it out. Cheers, Matthew [*] Running 'make -dA' with maximum debug output is quite enlightening, as is running make under truss(1) -- Dr Matthew J Seaman MA, D.Phil. 7 Priory Courtyard Flat 3 PGP: http://www.infracaninophile.co.uk/pgpkey Ramsgate JID: matt...@infracaninophile.co.uk Kent, CT11 9PW signature.asc Description: OpenPGP digital signature
Re: A new and better way to do make readmes?
Hello, On Sat, Jan 28, 2012 at 3:03 AM, Conrad J. Sabatier conr...@cox.net wrote: I've been thinking for a long time that we need a better way to do make readmes, one that would be properly integrated into our ports Mk infrastructure, to take advantage of make's ability to recognize which files are up-to-date and which really do need rebuilding. I like to make sure my README.html files are all up-to-date after my nightly ports tree update, but with the current scheme, that means either rebuilding *all* of the files in the tree, or (as I'm doing at present) using some sort of kludgey (kludgy?) workaround. So people are actually using the readme files? Are many people using them? I ask because I *never* use them (unless they are used by 'make search'?), I always use freshports.org (BTW, thanks for an excellent service!) when I need to find out anything about a port. -- Regards, Torfinn Ingolfsen ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org
Re: A new and better way to do make readmes?
Matthew Seaman said The big problem with performance in all this INDEX and README.html building is that it takes quite a long time relatively to run make(1) within any port or category directory. make(1) has to read in a lot of other files and stat(2) many more[*] -- all of which involves a lot of random-access disk IO, and that's always going to take quite a lot of time. Now, doing 'make readme' in a category directory doesn't just run make in that directory, but also in every port in that category. Popular categories can contain many hundreds of ports. Maybe I should add README.html generation to my FreeBSD::Portindex stuff. Should be pretty simple -- all the necessary bits are readily available and it is just a matter of formatting it as HTML and printing it out. Indeed, the following python script http://www.lpthe.jussieu.fr/~talon/show_index.py parses the index in a few seconds and can display exactly the same information as the readme.html on demand in a web browser, which is far cleaner than polluting the ports tree with the readmes. Alternatively i have a fcgi version that can be coupled to web servers supporting fcgi like lighttpd. http://www.lpthe.jussieu.fr/~talon/show_index.fcgi Already 5 years this was done ... -- Michel Talon ta...@lpthe.jussieu.fr ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org
Distfiles with conflicting size/sha256 distinfo
Hmmm... Something is definitely wrong here. Ports are either referring to the same distfile, but the sha256 and size data are not the same in both distinfo files, or they are downloading distinct files with an unfortunate conflict of names and not using DIST_SUBDIR to prevent themselves stomping on the other port's distfile. distfile_path here is the path relative to /usr/ports/distfiles that the file would be downloaded to. distfile_path| distfile_size | distfile_sha256 -+---+-- android-9-arm.tar.bz2 | 1924233 | 682cee5219f171a8bf4d0b35a33bc00d265a8c5a0b1d81ddd04389b402783190 android-9-arm.tar.bz2 | 1923856 | c27063b829b4b49ea604c1b524399c7672ecb0f869478b18d37b9d4d32841a6b eclipse/org.hamcrest.core_1.1.0.v20090501071000.jar | 27828 | db7f2cd14b9e263d791631960a0cd5a9ff4c8257939b551a5b86dd2ac6d23345 eclipse/org.hamcrest.core_1.1.0.v20090501071000.jar | 27827 | 3990293a31c98e079be5c24410fcaefb06f2770547879b27796651f8df092a0e pfpro_freebsd.tar.gz|322411 | 1afeedc6b67be35b372ccedc717516eabe8a4850ab7ecc5e605a44d14d721036 pfpro_freebsd.tar.gz|322410 | 396943299424066d23ef1307d02a044b2d7b45758a16e1dca5f47c907caa5606 (6 rows) port |distfile_path -+- lang/gnatdroid-binutils | android-9-arm.tar.bz2 lang/gnatdroid-sysroot | android-9-arm.tar.bz2 java/eclipse| eclipse/org.hamcrest.core_1.1.0.v20090501071000.jar java/eclipse-devel | eclipse/org.hamcrest.core_1.1.0.v20090501071000.jar finance/p5-PFProAPI | pfpro_freebsd.tar.gz finance/pfpro | pfpro_freebsd.tar.gz (6 rows) After checking the ports in question, it looks like the first case: incorrect size and sha256 data in at least one of those distfiles. Cheers, Matthew -- Dr Matthew J Seaman MA, D.Phil. 7 Priory Courtyard Flat 3 PGP: http://www.infracaninophile.co.uk/pgpkey Ramsgate JID: matt...@infracaninophile.co.uk Kent, CT11 9PW signature.asc Description: OpenPGP digital signature
Portmaster quickly fails to update xcb-utils
When I try to follow the instructions in UPDATING for upgrading xcb-util, using portmaster it fails very early in the process: % portmaster -bDw -R -r xcb-util-0 === Working on: xcb-util-0.3.6,1 === Port directory: /usr/ports/x11/xcb-util === Launching 'make checksum' for x11/xcb-util in background === Gathering dependency list for x11/xcb-util from ports === Update failed === Aborting update === Update for xcb-util-0.3.6,1 failed === Aborting update Terminated Terminated I am seeing this on FreeBSD 7.4-RELEASE-p5 and FreeBSD 8.2-RELEASE-p3. I updated the ports tree again just now to be sure I have the latest version and there is no change after doing that. Any hints? I'm not having much luck figuring out the problem myself, since I don't know why a dependency check would fail here... ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org
Re: databases/mysql-workbench52
On Monday 23 January 2012 17:08:44 you wrote: On 23 Jan 2012 15:31, t...@diogunix.com t...@diogunix.com wrote: Hm, well the page at [1] claims that 5+ should be supported, and it's from the 5.5 docs. I would try asking the mailing list at http://lists.mysql.com/internals . Did so, awaiting reply there. Ok, disappoingly, I did not get any reply on http://lists.mysql.com/internals As a solution (regarding practical terms) I just deinstalled the MySQL ports coming with KDE and instead installed MySQL 5.1. Then, the Workbench 5.2 port compiled without any issues. So far, the Makefile of the Workbench52 port was correct. Of course, some parts of KDE will not work now :-( As doesn't appear to be too easy to make the Workbench Port work with mySQL 5.5 I'll propose the KDE folks to ask for a decision on whether to build the KDE parts in question with MySQL 5.5 OR with MySQL 5.1. This at least should give us the option to work with KDE (including Akonadi / KDE PIM) AND with Workbench as long as Workbench won't work with 5.5 - better than nothing ... Tom ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org
