Re: [PATCHES] Faster install-sh in C
On Sat, Jun 04, 2005 at 08:42:29PM -0400, Bruce Momjian wrote: I am using 'cp' in current CVS for this. I assume it is as fast as the C implementation, but if not, please let me know. Yes, your change did make the installation considerably faster so I abandoned the binary install idea. Thanks. -- Alvaro Herrera ([EMAIL PROTECTED]) Lo esencial es invisible para los ojos (A. de Saint Exúpery) ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: [PATCHES] Faster install-sh in C
I am using 'cp' in current CVS for this. I assume it is as fast as the C implementation, but if not, please let me know. --- Alvaro Herrera wrote: Patchers, I wrote an install program in C. It's supposed to replace the config/install-sh script, limited to the functionality we need, i.e. what is in Makefiles in the Pg main source tree. The main objective of this exercise is to reduce make install execution time; a part of that is being able to install multiple files with one command. Portability testing right now is limited to my machine, which is Linux with glibc 2.3.2 (Debian Sid). With this in place, make install in src/include takes 17 seconds on this machine, where the script version takes more than a minute. I think this is a useful improvement. Right now I'm missing a Makefile rule for it. It needs the pg_progname symbol from src/port, and the includes from $(srcdir)/src/include and $(builddir)/src/include. From the config directory, this works: $(CC) $(CFLAGS) -I$(top_srcdir)/src/include -I$(top_builddir)/src/include ../src/port/path.o ../src/port/exec.o $ -o $@ Also, I don't know how to force it to build before executing install, short of putting it as a dependency in every makefile (which sounds silly to me). I attach the source code, and a src/include/Makefile modification to use the multi-file install feature. -- Alvaro Herrera (alvherre[a]dcc.uchile.cl) Hay quien adquiere la mala costumbre de ser infeliz (M. A. Evans) [ Attachment, skipping... ] [ Attachment, skipping... ] ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster -- Bruce Momjian| http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 9: the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [PATCHES] Faster install-sh in C
Tom Lane wrote: Bruce Momjian pgman@candle.pha.pa.us writes: OK, what is 'install' doing for us that 'cp' and 'chmod' would not already do? Quite a lot of things, such as coping with busy target files --- not too important for headers, but very important for executables and shlibs. We might be able to get away with this for just the headers, though, and that's certainly the bulk of the install work now. Here is my next version of the patch that uses 'cp' and 'chmod' to install multiple header files rather than 'install'. I moved the file modes into variables so any changes are propogated to src/include/Makefile. This is 20 times faster than what we have now, 8 seconds vs 0.40 seconds. -- Bruce Momjian| http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 Index: src/Makefile.global.in === RCS file: /cvsroot/pgsql/src/Makefile.global.in,v retrieving revision 1.212 diff -c -c -r1.212 Makefile.global.in *** src/Makefile.global.in 25 Mar 2005 18:17:12 - 1.212 --- src/Makefile.global.in 25 Mar 2005 22:58:33 - *** *** 232,240 INSTALL = $(SHELL) $(top_srcdir)/config/install-sh -c INSTALL_PROGRAM = $(INSTALL_PROGRAM_ENV) $(INSTALL) $(INSTALL_STRIP_FLAG) ! INSTALL_SCRIPT= $(INSTALL) -m 755 ! INSTALL_DATA = $(INSTALL) -m 644 INSTALL_STLIB = $(INSTALL_STLIB_ENV) $(INSTALL_DATA) $(INSTALL_STRIP_FLAG) INSTALL_SHLIB = $(INSTALL_SHLIB_ENV) $(INSTALL) $(INSTALL_SHLIB_OPTS) $(INSTALL_STRIP_FLAG) # Override in Makefile.port if necessary --- 232,242 INSTALL = $(SHELL) $(top_srcdir)/config/install-sh -c + INSTALL_SCRIPT_MODE = 755 + INSTALL_DATA_MODE = 644 INSTALL_PROGRAM = $(INSTALL_PROGRAM_ENV) $(INSTALL) $(INSTALL_STRIP_FLAG) ! INSTALL_SCRIPT= $(INSTALL) -m $(INSTALL_SCRIPT_MODE) ! INSTALL_DATA = $(INSTALL) -m $(INSTALL_DATA_MODE) INSTALL_STLIB = $(INSTALL_STLIB_ENV) $(INSTALL_DATA) $(INSTALL_STRIP_FLAG) INSTALL_SHLIB = $(INSTALL_SHLIB_ENV) $(INSTALL) $(INSTALL_SHLIB_OPTS) $(INSTALL_STRIP_FLAG) # Override in Makefile.port if necessary Index: src/include/Makefile === RCS file: /cvsroot/pgsql/src/include/Makefile,v retrieving revision 1.19 diff -c -c -r1.19 Makefile *** src/include/Makefile6 Jan 2005 21:00:24 - 1.19 --- src/include/Makefile25 Mar 2005 22:58:37 - *** *** 37,49 # These headers are needed for server-side development $(INSTALL_DATA) pg_config.h$(DESTDIR)$(includedir_server) $(INSTALL_DATA) pg_config_os.h $(DESTDIR)$(includedir_server) ! for file in $(srcdir)/*.h; do \ ! $(INSTALL_DATA) $$file $(DESTDIR)$(includedir_server)/`basename $$file` || exit; \ ! done for dir in $(SUBDIRS); do \ ! for file in $(srcdir)/$$dir/*.h; do \ ! $(INSTALL_DATA) $$file $(DESTDIR)$(includedir_server)/$$dir/`basename $$file` || exit; \ ! done \ done installdirs: --- 37,48 # These headers are needed for server-side development $(INSTALL_DATA) pg_config.h$(DESTDIR)$(includedir_server) $(INSTALL_DATA) pg_config_os.h $(DESTDIR)$(includedir_server) ! # We don't use INSTALL_DATA for performance reasons --- there are a lot of files ! cp $(srcdir)/*.h $(DESTDIR)$(includedir_server)/ || exit; \ ! chmod $(INSTALL_DATA_MODE) $(DESTDIR)$(includedir_server)/*.h || exit; \ for dir in $(SUBDIRS); do \ ! cp $(srcdir)/$$dir/*.h $(DESTDIR)$(includedir_server)/$$dir/ || exit; \ ! chmod $(INSTALL_DATA_MODE) $(DESTDIR)$(includedir_server)/$$dir/*.h || exit; \ done installdirs: ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: [PATCHES] Faster install-sh in C
Bruce Momjian pgman@candle.pha.pa.us writes: Tom Lane wrote: We might be able to get away with this for just the headers, though, and that's certainly the bulk of the install work now. Here is my next version of the patch that uses 'cp' and 'chmod' to install multiple header files rather than 'install'. I moved the file modes into variables so any changes are propogated to src/include/Makefile. This is 20 times faster than what we have now, 8 seconds vs 0.40 seconds. I like it. Given that we are installing into directories we'd have just created ourselves, which should have no extraneous files in them, a lot of the objections one might raise go away. Why don't you go ahead and apply that to HEAD for the time being, even if we end up replacing it later? Might as well start saving developers' time immediately. regards, tom lane ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [PATCHES] Faster install-sh in C
Tom Lane wrote: Bruce Momjian pgman@candle.pha.pa.us writes: Tom Lane wrote: We might be able to get away with this for just the headers, though, and that's certainly the bulk of the install work now. Here is my next version of the patch that uses 'cp' and 'chmod' to install multiple header files rather than 'install'. I moved the file modes into variables so any changes are propogated to src/include/Makefile. This is 20 times faster than what we have now, 8 seconds vs 0.40 seconds. I like it. Given that we are installing into directories we'd have just created ourselves, which should have no extraneous files in them, a lot of the objections one might raise go away. Why don't you go ahead and apply that to HEAD for the time being, even if we end up replacing it later? Might as well start saving developers' time immediately. Done. -- Bruce Momjian| http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [PATCHES] Faster install-sh in C
On Mon, Mar 07, 2005 at 07:50:23PM +0100, Peter Eisentraut wrote: Alvaro Herrera wrote: Well, apparently everyone says the system install is not portable and the discussion stops there. Well, who actually says that? I know that I was the one who actually coded up the current avoid-system-install-at-all-cost behavior, but only because we were too annoyed at the time to do more research. Now, as long as someone's spending time on it, it may be worth investigating what is wrong and provide an explanation for the benefit of the entire autotools-using community. Well, the problem with this approach is that while I did have time to write the C replacement, I didn't (and still don't) have access to a lot of systems in order to check what the differences between the install programs are. Also, keep in my that this C install program has the extra feature of being able to install multiple files on one invocation, per suggestion from Tom Lane. This allows us to save the nested for-loop in src/include/Makefile. GNU install (available on my system) also has this capability, but would we be able to use the trick if we had to cater for the lowest common denominator found on other, non-GNU-enabled systems? -- Alvaro Herrera ([EMAIL PROTECTED]) Llegará una época en la que una investigación diligente y prolongada sacará a la luz cosas que hoy están ocultas (Séneca, siglo I) ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [PATCHES] Faster install-sh in C
Alvaro Herrera [EMAIL PROTECTED] writes: Also, keep in my that this C install program has the extra feature of being able to install multiple files on one invocation, per suggestion from Tom Lane. This allows us to save the nested for-loop in src/include/Makefile. GNU install (available on my system) also has this capability, but would we be able to use the trick if we had to cater for the lowest common denominator found on other, non-GNU-enabled systems? We would definitely not ... and my suspicion is that that is the largest single component of the available speedup. regards, tom lane ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://archives.postgresql.org
Re: [PATCHES] Faster install-sh in C
Tom Lane wrote: Alvaro Herrera [EMAIL PROTECTED] writes: Also, keep in my that this C install program has the extra feature of being able to install multiple files on one invocation, per suggestion from Tom Lane. This allows us to save the nested for-loop in src/include/Makefile. GNU install (available on my system) also has this capability, but would we be able to use the trick if we had to cater for the lowest common denominator found on other, non-GNU-enabled systems? We would definitely not ... and my suspicion is that that is the largest single component of the available speedup. OK, what is 'install' doing for us that 'cp' and 'chmod' would not already do? The following patch reduces the 'gmake install' in /src/include from 8 seconds to 0.8 seconds. Oh, and for testing I use a sourceforge shell farm account, and a HP testdrive shell acount that gives me access to almost every operating system. -- Bruce Momjian| http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 Index: src/include/Makefile === RCS file: /cvsroot/pgsql/src/include/Makefile,v retrieving revision 1.19 diff -c -c -r1.19 Makefile *** src/include/Makefile6 Jan 2005 21:00:24 - 1.19 --- src/include/Makefile23 Mar 2005 20:22:56 - *** *** 41,49 $(INSTALL_DATA) $$file $(DESTDIR)$(includedir_server)/`basename $$file` || exit; \ done for dir in $(SUBDIRS); do \ ! for file in $(srcdir)/$$dir/*.h; do \ ! $(INSTALL_DATA) $$file $(DESTDIR)$(includedir_server)/$$dir/`basename $$file` || exit; \ ! done \ done installdirs: --- 41,48 $(INSTALL_DATA) $$file $(DESTDIR)$(includedir_server)/`basename $$file` || exit; \ done for dir in $(SUBDIRS); do \ ! cp $(srcdir)/$$dir/*.h $(DESTDIR)$(includedir_server)/$$dir/ || exit; \ ! chmod 644 $(DESTDIR)$(includedir_server)/$$dir/*.h || exit; \ done installdirs: ---(end of broadcast)--- TIP 8: explain analyze is your friend
Re: [PATCHES] Faster install-sh in C
Bruce Momjian pgman@candle.pha.pa.us writes: OK, what is 'install' doing for us that 'cp' and 'chmod' would not already do? Quite a lot of things, such as coping with busy target files --- not too important for headers, but very important for executables and shlibs. We might be able to get away with this for just the headers, though, and that's certainly the bulk of the install work now. regards, tom lane ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
Re: [PATCHES] Faster install-sh in C
Alvaro Herrera wrote: Also, keep in my that this C install program has the extra feature of being able to install multiple files on one invocation, per suggestion from Tom Lane. This allows us to save the nested for-loop in src/include/Makefile. GNU install (available on my system) also has this capability, but would we be able to use the trick if we had to cater for the lowest common denominator found on other, non-GNU-enabled systems? Why not check for GNU install in configure, and only enable the optimization if it is available? -Neil ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send unregister YourEmailAddressHere to [EMAIL PROTECTED])
Re: [PATCHES] Faster install-sh in C
Alvaro Herrera wrote: Well, apparently everyone says the system install is not portable and the discussion stops there. Well, who actually says that? I know that I was the one who actually coded up the current avoid-system-install-at-all-cost behavior, but only because we were too annoyed at the time to do more research. Now, as long as someone's spending time on it, it may be worth investigating what is wrong and provide an explanation for the benefit of the entire autotools-using community. -- Peter Eisentraut http://developer.postgresql.org/~petere/ ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://archives.postgresql.org
Re: [PATCHES] Faster install-sh in C
On Fri, Mar 04, 2005 at 11:20:07AM +0100, Peter Eisentraut wrote: Am Freitag, 4. März 2005 01:11 schrieb Neil Conway: Alvaro Herrera wrote: I wrote an install program in C. It's supposed to replace the config/install-sh script, limited to the functionality we need, i.e. what is in Makefiles in the Pg main source tree. The main objective of this exercise is to reduce make install execution time; a part of that is being able to install multiple files with one command. What's the status of this patch? I was playing a little with Make just before leaving, but since I'm hardly an expert I was having difficulty making it all work. Also, because I need to include bits from both the source dir and the build dir (to get get_progname from the port dir and signal handling from libpq), it was getting somewhat ugly. With a little more effort it can be made to work cleanly. I'm not convinced that this is the best approach. Apparently, PostgreSQL is about the only project that is somehow bothered by this. It would be more productive to find out why we can't use the system install and find fixes for that. Well, apparently everyone says the system install is not portable and the discussion stops there. Probably some projects need a sh install because they don't require a compiler, but we do; we won't require anything beyond what we already do to have a C install. Maybe most projects don't install a lot of files like we do, or maybe people just don't care. (Neither did I care back when installing all headers wasn't the default behaviour.) I do care now. And given that the work is mostly already done, I don't see why we couldn't use it. Anyway, if you want to speed up your installation today, just override the INSTALL variable by hand. This doesn't solve the fact that the INSTALL program is called once for each file in the include directories. -- Alvaro Herrera ([EMAIL PROTECTED]) Y una voz del caos me habló y me dijo Sonríe y sé feliz, podría ser peor. Y sonreí. Y fui feliz. Y fue peor. ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [PATCHES] Faster install-sh in C
Am Freitag, 4. März 2005 01:11 schrieb Neil Conway: Alvaro Herrera wrote: I wrote an install program in C. It's supposed to replace the config/install-sh script, limited to the functionality we need, i.e. what is in Makefiles in the Pg main source tree. The main objective of this exercise is to reduce make install execution time; a part of that is being able to install multiple files with one command. What's the status of this patch? I'm not convinced that this is the best approach. Apparently, PostgreSQL is about the only project that is somehow bothered by this. It would be more productive to find out why we can't use the system install and find fixes for that. Anyway, if you want to speed up your installation today, just override the INSTALL variable by hand. -- Peter Eisentraut http://developer.postgresql.org/~petere/ ---(end of broadcast)--- TIP 7: don't forget to increase your free space map settings
Re: [PATCHES] Faster install-sh in C
Alvaro Herrera wrote: I wrote an install program in C. It's supposed to replace the config/install-sh script, limited to the functionality we need, i.e. what is in Makefiles in the Pg main source tree. The main objective of this exercise is to reduce make install execution time; a part of that is being able to install multiple files with one command. What's the status of this patch? -Neil ---(end of broadcast)--- TIP 7: don't forget to increase your free space map settings
Re: [PATCHES] Faster install-sh in C
This has been saved for the 8.1 release: http:/momjian.postgresql.org/cgi-bin/pgpatches2 --- Alvaro Herrera wrote: Patchers, I wrote an install program in C. It's supposed to replace the config/install-sh script, limited to the functionality we need, i.e. what is in Makefiles in the Pg main source tree. The main objective of this exercise is to reduce make install execution time; a part of that is being able to install multiple files with one command. Portability testing right now is limited to my machine, which is Linux with glibc 2.3.2 (Debian Sid). With this in place, make install in src/include takes 17 seconds on this machine, where the script version takes more than a minute. I think this is a useful improvement. Right now I'm missing a Makefile rule for it. It needs the pg_progname symbol from src/port, and the includes from $(srcdir)/src/include and $(builddir)/src/include. From the config directory, this works: $(CC) $(CFLAGS) -I$(top_srcdir)/src/include -I$(top_builddir)/src/include ../src/port/path.o ../src/port/exec.o $ -o $@ Also, I don't know how to force it to build before executing install, short of putting it as a dependency in every makefile (which sounds silly to me). I attach the source code, and a src/include/Makefile modification to use the multi-file install feature. -- Alvaro Herrera (alvherre[a]dcc.uchile.cl) Hay quien adquiere la mala costumbre de ser infeliz (M. A. Evans) [ Attachment, skipping... ] [ Attachment, skipping... ] ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster -- Bruce Momjian| http://candle.pha.pa.us [EMAIL PROTECTED] | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faqs/FAQ.html
