Yoshihiro Ota wrote:
On Thu, 26 Jul 2007 02:17:50 -0700
Garrett Cooper <[EMAIL PROTECTED]> wrote:
Yoshihiro Ota wrote:
To Whom Slowness of Portupgrade Concerns a Lot:
As I got tired of long waiting of portupgrade trying to resolve dependencies, I
came up with
yet another tool for upgrading FreeBSD ports system. Unlink other tools, it
tries to maximize
existing resource to maximize its performance. This program attempts to wrap
around with
another 'make' and expand use of FreeBSD ports system. The heart of ports+ is
parsing INDEX
and +CONTENTS files. The rest is handed to GNU make. I think it comes to a
point where I seek
wider audiences to test with it.
Hiro,
You're correct when you say that reading INDEX* does take a long time
for portupgrade, but the problem is partly with how portupgrade formats
its version of the INDEX database in BDB format, as well as how it
doesn't buffer up proper information (pkg database information for
instance), and guesses at port dependencies (origins, deporigins, etc).
Portupgrade is not only slow reading INDEX file but also on dependency resolving and
updating +CONTENTS files, too. I think portmaster is also one tries to read and do the
same things but with shell script. I personally didn't have good luck with portmaster
and haven't really used to evaluate. However, "portmaster -a -n" wasn't not
fast, neither. By the way, it builds ports in background, doesn't it?
Yes, but a lot of time is spent reading the INDEX file, especially since
portupgrade / portinstall don't keep that information in memory
(probably for the better though).
Like Doug said, portmaster doesn't built ports in the background, and
the way that the package system is currently setup I don't advise doing
that unless you plan everything out properly (I see potential for
duplicated installs, race conditions, and other unwanted possibilities).
I'm working on getting away from that at the moment.
1. Can your solution be made bsdmake and bsd awk compatible?
I am not as familiar in newer BSD awk and make extensions. I can point out
what kinds of GNU extensions are being used so that someone can tell it is also
available in BSD make/awk.
For awk, this feature is required, quoted from "man gawk". I am not sure if
BSD awk implements this feature.
The book indicates that command line variable assignment happens when
awk would otherwise open the argument as a file, which is after the
BEGIN block is executed. However, in earlier implementations, when
such an assignment appeared before any file names, the assignment would
happen before the BEGIN block was run. Applications came to depend on
this "feature." When awk was changed to match its documentation, the
-v option for assigning variables before program execution was added to
accommodate applications that depended upon the old behavior. (This
feature was agreed upon by both the Bell Laboratories and the GNU
developers.)
For make, 1.1 and 1.2 below are in the same topic. I use GNU extension which
keeps generating makefiles until dependency rule saticifies and use it part of
the make rule. GNU extension reloads newer makefiles and reconstruct
dependencies. I am not sure if it is possible in BSD make; however, this is
not a heart of this program. We can work around with bootstrap script or
something.
1.3 This is recursive expansion of variables. If you take a look, every port
needs upgrade uses $(UPGRADE). $(UPGRADE) is expanded multiple times using the
target variable, $@, such that it backups up correct old package names and so
on. This is very important and unless this works in BSD make, my tool really
needed full redesign from scratch.
1.4 is just how GNU make does shell invocations. It is available in most of
Make tools.
1.5 is something I am not sure which makes support these. I don't think GUN make recommends
using special characters as variable names. However, when I tried, it was too difficult to
normalize to alphabets and underscore only. If we cannot use variable like PORTS+IGNORE as a
variable name, it will be very difficult and personally I don't even think about trying that
myself. If you take a look at compat.gmk which ports+ generates as a part of its build rule, you
can find something like "COMPAT_$(PKG_DIR)Hermes-1.3.3_2 = cp -f
/usr/local/lib/libHermes.so.1 /usr/local/lib/compat/pkg&&" being variable.
1.1 How makefiles get remade:
http://uw713doc.sco.com/cgi-bin/info2html?(make.info)Remaking%2520Makefiles&lang=en
1.2 Including Other Makefiles:
http://uw713doc.sco.com/cgi-bin/info2html?(make.info)Include&lang=en
1.3 Computed Variable Names or recursive expandsion of variables:
http://uw713doc.sco.com/cgi-bin/info2html?(make.info)Computed%2520Names&lang=en
1.4 The `shell' Function
http://uw713doc.sco.com/cgi-bin/info2html?(make.info)Shell%2520Function&lang=en
1.5 Special characters in variable names such as dot, ., comma, ,, slash, /,
and so on.
Ah, good ole SCO.. I can't believe they're posting docs like that. Lol.
I do see your variable naming scheme as a cause for concern though,
unfortunately. I steer clear of shell-like configurations like this when
you can potentially get malicious text inside a variable name / declaration.
2. Does your solution account for cases when you're trying to install
package a, which depends on package b, but because you built package b
while trying to build a, and are at an intermediate package c (between a
and b), the dependencies for a are only partially complete. Thus when
you try to install direct or indirect dependences, the install fails?
Some of answers are explain in the GNUmakefile. Anyway, ports+ categorizes all ports
into three, NEW as ones not installed, UPGRADE as installed and newer version is
available, NOOP as installed an no newer version is available. I only started with
upgrading only and haven't really though thought installing new ones. I am not ready to
talk about installing new ports via ports+. Short answer for upgrade is, "Yes, it
does." Here are some examples. Let's say 'a' depends on 'b'. Then, it builds 'b'
first and than 'a'. If new version of 'b' has new requirement of 'c'. Making 'a' will
result installing 'c' and than upgrading 'b.' If 'e' requires 'f', 'g', and 'h' and
these are independent from each other, upgrading 'e' with -j 3 will start upgrading 'f',
'g', and 'h' at the same time and once all three are upgraded, it will start on 'e'.
That's really good thinking sir for single installs. Props to you for
that :).
3. How is this different from pkg_version combined with similar scripts?
I wasn't aware of pkg_version, but it looks like it only tells if port is up to
date or not. It doesn't tell anything about dependencies among ports, does it?
If so, it doesn't provide enough information to upgrade.
Every time you run make install, portmaster, or portupgrade, there's
a lot of stuff going on behind the scenes. One of the things involved is
installing package information in /var/db/pkg. That's where pkg_version
gets its information from, along with portmaster (AFAIK) and the
makefiles for ports (for sure).
It does provide enough info to tell whether or not something needs
to be upgraded, based on the output (<, =, or >). It doesn't tell you
the official target you're upgrading to, but it doesn't really need to
do that though.
The difference from other two is the following. Portupgrade and portmaster "reads" what's in ports, INDEX,
and what's installed, +CONTENTS, "solves" dependencies itself, and "manages" upgrading port itself.
Instead, ports+ "converts" INDEX and +CONTENTS to (GNU) Makefile and let the dependency expert solve the
rest.
Hiro
I do like the ideas of simplified makefiles, but I see dependencies
changing frequency, unfortunately, and if dependencies are changed or
touch(1)'ed, then things have to be rebuilt to make gmake happy :). Your
exercise in doing this is interesting though, nonetheless.
-Garrett
_______________________________________________
freebsd-ports@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "[EMAIL PROTECTED]"
