Re: [HACKERS] Log rotation
On Friday 12 March 2004 03:21 pm, Fernando Nasser wrote: Lamar Owen wrote: Uh, we have many many many different ways to use syslog. So my other message on the topic. Which other message? The one I didn't post. :-) Anyway, Syslog is not an option for us. We need flat files. Ok, riddle me this: If I have PostgreSQL set to log to syslog facility LOCAL0, and a local0.none on /var/log/messages and local0.* to /var/log/pgsql (assuming only one postmaster, unfortunately) then you get a flat file. I can see that in a multipostmaster setting how you might want some differentiation between postmasters, but ISTM that the tool reading these logs should be trained in how to separate loglines out. I use a product called SmoothWall as a firewall/VPN solution, and its log reporting modules split out loglines in this way, so that I can have the ipsec logs in one browser page and the l2tp logs elsewhere, and the ppp logs elsewhere, and the kernel logs elsewhere Or you'll have to include some other log rotator. Uh, upgrading? I'm sure we have more reports about upgrading than logging. Yeah, but that only comes with a full version upgrade :-) Which is quite often. And people tend to upgrade as part of the OS upgrade, which could easily be every other version (in the case of Fedora). Upgrading from RHAS 2.1 to RHEL3 (I know it's not supported) should prove to be an interesting one. The logging one keeps popping up as people try and use the server for production. Yes, and for the vast majority of cases syslog is enough solution for the job. -- Lamar Owen Director of Information Technology Pisgah Astronomical Research Institute 1 PARI Drive Rosman, NC 28772 (828)862-5554 www.pari.edu ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
Re: [HACKERS] Log rotation
On Saturday 13 March 2004 09:36, Lamar Owen wrote:
On Friday 12 March 2004 03:21 pm, Fernando Nasser wrote:
Lamar Owen wrote:
Uh, we have many many many different ways to use syslog. So my other
message on the topic.
Which other message?
The one I didn't post. :-)
Anyway, Syslog is not an option for us. We need flat files.
Ok, riddle me this:
If I have PostgreSQL set to log to syslog facility LOCAL0, and a
local0.none on /var/log/messages and local0.* to /var/log/pgsql (assuming
only one postmaster, unfortunately) then you get a flat file.
I can see that in a multipostmaster setting how you might want some
differentiation between postmasters, but ISTM that the tool reading these
logs should be trained in how to separate loglines out. snip
Different postmasters = different conf files. Just set your syslog_facility
and/or your syslog_ident differently and it should be pretty easy to seperate
the logs. Actually, now that I have started using syslog fairly regularly, I
find it hard to believe it would be worth the effort to try to recreate a
logging facility as feature complete as syslog when we have syslog available.
ranks right up there with recreate cron, another feature many people think an
enterprise database has to have. Personally I think in place upgrades are
far, far more important than either of those two, to both end users and to
the development community.
Robert Treat
--
Build A Brighter Lamp :: Linux Apache {middleware} PostgreSQL
---(end of broadcast)---
TIP 5: Have you checked our extensive FAQ?
http://www.postgresql.org/docs/faqs/FAQ.html
Re: [HACKERS] Log rotation
Bruno Wolff III wrote: On Fri, Mar 12, 2004 at 15:19:29 -0500, Fernando Nasser [EMAIL PROTECTED] wrote: Bruno Wolff III wrote: I can see their problem with making a dependency to all of apache or including multilog in their distribution. But they probably could include something that is only a logger either using some project that is only a logger or splitting out the logger that is bundled with apache. Then it wouldn't be unreasonable to make a dependency for postgres requiring that logging rpm. Other services could also make use of this logging package as well. Yes that would be nice. I have no idea how difficult it would be to extricate the logrotate program from Apache. I also don't know if there would be any license restrictions, would we be able to redistribute it as an independently package? I don't know the answer. I was suggesting this as something a distro maintainer (such as Redhat) could do. I think that the postgres developers shouldn't be spending time doing this. They should be just suggesting some possibilities in the admin part of the documentation. The distro maintainers should only pack proven solutions created by the community. When they add man power to a project they must do it in the community process, not in house. A decent logging mechanism (that allows log rotation) is important to postgres and should not require one to depend on a distro. They should be able to get it from pgsql land. It would be nice if there was a community recommended and community tested logging solution, even if the instructions to set it up required one to download and install some apache package. But adding prodution quality logging to PostgreSQL is still a postgresql community TODO and should not be delegated to the distros and thus restricted to a distro only. Regards, Fernando ---(end of broadcast)--- TIP 8: explain analyze is your friend
Re: [HACKERS] Log rotation
Lamar Owen wrote: Anyway, Syslog is not an option for us. We need flat files. Ok, riddle me this: If I have PostgreSQL set to log to syslog facility LOCAL0, and a local0.none on /var/log/messages and local0.* to /var/log/pgsql (assuming only one postmaster, unfortunately) then you get a flat file. The problem is that sysloging has more overhead than a plain append to a file. There are some very strict response time AppServer applications where we want to keep this things out of the picture. The other problem is that we have some nice graphical tools for configuring logging but the /etc/syslog.conf is a very hard one to automate dur to the pattern matching. We can add some lines there, like one to get local0 to a specific file, but it will probably be guesswork and require human inspection anyway. I can see that in a multipostmaster setting how you might want some differentiation between postmasters, but ISTM that the tool reading these logs should be trained in how to separate loglines out. I use a product called SmoothWall as a firewall/VPN solution, and its log reporting modules split out loglines in this way, so that I can have the ipsec logs in one browser page and the l2tp logs elsewhere, and the ppp logs elsewhere, and the kernel logs elsewhere It may be desirable to logrotate them at different times as well, so they would have to be in different files. Or you'll have to include some other log rotator. That is what Tom is proposing. -- Fernando Nasser Red Hat Canada Ltd. E-Mail: [EMAIL PROTECTED] 2323 Yonge Street, Suite #300 Toronto, Ontario M4P 2C9 ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faqs/FAQ.html
Re: [HACKERS] Log rotation
I can see that in a multipostmaster setting how you might want some differentiation between postmasters, but ISTM that the tool reading these logs should be trained in how to separate loglines out. snip Different postmasters = different conf files. Just set your syslog_facility and/or your syslog_ident differently and it should be pretty easy to seperate the logs. Actually, now that I have started using syslog fairly regularly, I Not that I'm volunteering, but I think the biggest issue is many users simply don't know how to approach the problem. Some docs on using syslog, cron, etc. with PostgreSQL to accomplish maintenace jobs would probably be enough. The only mention we have of syslog is related to the postgresql.conf entries and their appropriate values. http://www.postgresql.org/docs/7.4/static/runtime-config.html#RUNTIME-CONFIG-LOGGING-SYSLOG ---(end of broadcast)--- TIP 8: explain analyze is your friend
Re: [HACKERS] Log rotation
Robert Treat wrote: Different postmasters = different conf files. Just set your syslog_facility and/or your syslog_ident differently and it should be pretty easy to seperate the logs. Actually, now that I have started using syslog fairly regularly, I find it hard to believe it would be worth the effort to try to recreate a logging facility as feature complete as syslog when we have syslog available. ranks right up there with recreate cron, another feature many people think an enterprise database has to have. Personally I think in place upgrades are far, far more important than either of those two, to both end users and to the development community. The concerns with syslog were related to performance (additional load on the server). Perhaps these concerns were unfounded. I am not saying that upgrades are not important. I have been asking for it and have even tried to come up with a process to keep a continuously functional pg_upgrade program in synch with the cvs head development, even adding bits to the catalog for one release to aloow the conversion to be done. Ask Bruce and Tom, I've mentioned this to them 2 or 3 years ago. But the log rotation is a relatively small task in comparison, and it is at least as equaly visible in terms of production users (at least in the enterprise). I am talking about benefit/effort ratio. Regards, Fernando ---(end of broadcast)--- TIP 8: explain analyze is your friend
Re: [HACKERS] Log rotation
Rod Taylor wrote: I can see that in a multipostmaster setting how you might want some differentiation between postmasters, but ISTM that the tool reading these logs should be trained in how to separate loglines out. snip Different postmasters = different conf files. Just set your syslog_facility and/or your syslog_ident differently and it should be pretty easy to seperate the logs. Actually, now that I have started using syslog fairly regularly, I Not that I'm volunteering, but I think the biggest issue is many users simply don't know how to approach the problem. Some docs on using syslog, cron, etc. with PostgreSQL to accomplish maintenace jobs would probably be enough. It is very easy to setup logging with the Control Center tool. Except that it does not handle the /etc/syslog.conf bit (you have to add the entry to get a specific LOCAL to a specific file by hand. Maybe someone could make some measurements to desmistify the performance impact of syslog. There is this generalized belief in IS departments that should minimize it. Perhaps is an old fud that is not anylonger true. -- Fernando Nasser Red Hat Canada Ltd. E-Mail: [EMAIL PROTECTED] 2323 Yonge Street, Suite #300 Toronto, Ontario M4P 2C9 ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send unregister YourEmailAddressHere to [EMAIL PROTECTED])
Re: [HACKERS] Log rotation
On Sat, Mar 13, 2004 at 10:36:23AM -0500, Fernando Nasser wrote: Lamar Owen wrote: Ok, riddle me this: If I have PostgreSQL set to log to syslog facility LOCAL0, and a local0.none on /var/log/messages and local0.* to /var/log/pgsql (assuming only one postmaster, unfortunately) then you get a flat file. The problem is that sysloging has more overhead than a plain append to a file. There are some very strict response time AppServer applications where we want to keep this things out of the picture. I thought it was an advantage to say log to that box running syslog over there and leave my disk alone - what do you have in mind with AppServer applications ? It may be desirable to logrotate them at different times as well, so they would have to be in different files. syslogd with newsyslog, just like any other log file? I must be missing something.. I don't see why postgresql is different.. Cheers, Patrick ---(end of broadcast)--- TIP 8: explain analyze is your friend
Re: [HACKERS] Log rotation
Patrick Welche wrote: On Sat, Mar 13, 2004 at 10:36:23AM -0500, Fernando Nasser wrote: Lamar Owen wrote: Ok, riddle me this: If I have PostgreSQL set to log to syslog facility LOCAL0, and a local0.none on /var/log/messages and local0.* to /var/log/pgsql (assuming only one postmaster, unfortunately) then you get a flat file. The problem is that sysloging has more overhead than a plain append to a file. There are some very strict response time AppServer applications where we want to keep this things out of the picture. I thought it was an advantage to say log to that box running syslog over there and leave my disk alone - what do you have in mind with AppServer applications ? And add more packets to the network, buffering etc.? I don't think so. There are some applicatons which run in servers with very strict response times and any syscall, network packet that can be saved counts. It may be desirable to logrotate them at different times as well, so they would have to be in different files. syslogd with newsyslog, just like any other log file? I must be missing something.. I don't see why postgresql is different.. The number of generated messgaes. Maybe that is an area that can be worked on, i.e. reducing log verbosity. Is 7.4.x much better than 7.3.x in that respect? -- Fernando Nasser Red Hat Canada Ltd. E-Mail: [EMAIL PROTECTED] 2323 Yonge Street, Suite #300 Toronto, Ontario M4P 2C9 ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faqs/FAQ.html
Re: [HACKERS] Log rotation
On Saturday 13 March 2004 10:36 am, Fernando Nasser wrote: The problem is that sysloging has more overhead than a plain append to a file. There are some very strict response time AppServer applications where we want to keep this things out of the picture. Well, I have a couple of ideas on that. First, a better syslog. SDSC secure syslog has been available for some time and is BSD licensed, and is specifically designed for high-performance RFC3195 compliant secure syslog. Second, if the syslog server is separate, then you might get better performance as the size of the logs grow, since appending a very large file is slower than generating an IP datagram. Third, it seems that you don't have enough profiling data to support a 'syslog is bad' position. Java is bad too, from a performance standpoint (at this time, at least, you always get a performance hit of some kind using any portable code). But if this AppServer is based on the ArsDigita Java codebase, you have other performance issues already (the Tcl codebase, OTOH, is very fast, partly because AOLserver is a faster appserver than most Java appservers).). The other problem is that we have some nice graphical tools for configuring logging but the /etc/syslog.conf is a very hard one to automate dur to the pattern matching. We can add some lines there, like one to get local0 to a specific file, but it will probably be guesswork and require human inspection anyway. Control Center looks promising. But I much prefer having a central syslog server than having each server in a cluster having separate logs. It may be desirable to logrotate them at different times as well, so they would have to be in different files. Different facilities can then go to different files. The problem, of course, is syslog's limited number of facilities. Or you'll have to include some other log rotator. That is what Tom is proposing. I am not opposed to including a small logrotator for stderr logging. I just think it is redundant when a good highly configurable logging facility already exists. But, if Red Hat wants to pay Tom to do it... :-) -- Lamar Owen Director of Information Technology Pisgah Astronomical Research Institute 1 PARI Drive Rosman, NC 28772 (828)862-5554 www.pari.edu ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] Log rotation
On Saturday 13 March 2004 01:00 pm, Fernando Nasser wrote: There are some applicatons which run in servers with very strict response times and any syscall, network packet that can be saved counts. Ok, what about pipe overhead? If we're gong to split hairs, let's split all of them. The design of the pipeline in this logrotator filter will have to be such that minimal overhead occurs, because, at the real-time level, every concurrent process also counts. The number of generated messgaes. Maybe that is an area that can be worked on, i.e. reducing log verbosity. Is 7.4.x much better than 7.3.x in that respect? There are several levels documented in postgresql.conf. Try the terse level and see what happens. -- Lamar Owen Director of Information Technology Pisgah Astronomical Research Institute 1 PARI Drive Rosman, NC 28772 (828)862-5554 www.pari.edu ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: [HACKERS] Log rotation
Tom Lane wrote: Andrew Dunstan [EMAIL PROTECTED] writes: Did anything ever come from this thread? http://archives.postgresql.org/pgsql-hackers/2003-05/msg00603.php (Heading: Plan B for log rotation support: borrow Apache code) Only an entry on my depressingly long personal to-do list :-( I did take a look at the Apache rotator program, and found that it was probably more trouble to adopt than it's worth. It seemed to depend on a lot of configuration and library-routine infrastructure that we don't share. (No big surprise; I suppose someone trying to pull out a random bit of our backend code would be at least as unhappy.) I suspect it would be less trouble, as well as legalistically cleaner, to write our own from scratch. Andrew Sullivan offered Afilias' rotator script awhile back also. I think that works fine if you like a Perl script. FWIW, in less than 30 minutes I took the log rotator from apache 1.3.29 (i.e. the latest non-APR version) and imported it into a fresh postgreql tree. With very little massaging it built happily (see below). If it will advance matters, I can submit this as a patch filling the currently empty contrib/apache_logging directory. You could be right about the legal stuff - worth talking to the apache folks? cheers andrew [EMAIL PROTECTED] apache_logging]$ make gcc -O2 -fno-strict-aliasing -Wall -Wmissing-prototypes -Wmissing-declarations rotatelogs.o -L../../src/port -Wl,-rpath,/home/andrew/tpg/inst//lib -o rotatelogs [EMAIL PROTECTED] apache_logging]$ rm rotatelogs [EMAIL PROTECTED] apache_logging]$ rm rotatelogs.o [EMAIL PROTECTED] apache_logging]$ make gcc -O2 -fno-strict-aliasing -Wall -Wmissing-prototypes -Wmissing-declarations -I. -I../../src/include -D_GNU_SOURCE -c -o rotatelogs.o rotatelogs.c -MMD gcc -O2 -fno-strict-aliasing -Wall -Wmissing-prototypes -Wmissing-declarations rotatelogs.o -L../../src/port -Wl,-rpath,/home/andrew/tpg/inst//lib -o rotatelogs [EMAIL PROTECTED] apache_logging]$ ldd rotatelogs libc.so.6 = /lib/tls/libc.so.6 (0x0065e000) /lib/ld-linux.so.2 = /lib/ld-linux.so.2 (0x0056a000) [EMAIL PROTECTED] apache_logging]$ diff -cw ~/apache_1.3.29/src/support/rotatelogs.c rotatelogs.c *** /home/andrew/apache_1.3.29/src/support/rotatelogs.c Mon Jul 14 14:31:26 2003 --- rotatelogs.cSat Mar 13 13:47:41 2004 *** *** 7,16 */ ! #include ap_config.h #include time.h #include errno.h #include fcntl.h #if defined(WIN32) || defined(OS2) #include io.h --- 7,18 */ ! #include postgres.h #include time.h #include errno.h #include fcntl.h + #include unistd.h + #include sys/types.h #if defined(WIN32) || defined(OS2) #include io.h [EMAIL PROTECTED] apache_logging]$ cat Makefile # $PostgreSQL$ subdir = contrib/apache_logging top_builddir = ../.. include $(top_builddir)/src/Makefile.global PROGRAM = rotatelogs OBJS= rotatelogs.o include $(top_srcdir)/contrib/contrib-global.mk #LIBS:=$(filter-out -lpgport, $(LIBS)) LIBS:= [EMAIL PROTECTED] apache_logging]$ ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] Log rotation
Tom Lane wrote: I did take a look at the Apache rotator program, and found that it was probably more trouble to adopt than it's worth. It seemed to depend on a lot of configuration and library-routine infrastructure that we don't share. Here's a log rotation program that does not share those problems, I hope: http://developer.postgresql.org/~petere/pg_logpipe/ I haven't compiled it recently, so beware. ---(end of broadcast)--- TIP 8: explain analyze is your friend
[HACKERS] 7.4.2 Build broken on (Sparc) Solaris 7 and 8
Hi, Environment: SunOS 5.8 Generic_108528-27 sun4u sparc SUNW,Ultra-250 gcc version 3.3.1 PostgreSQL-7.4.2 ./configure --with-java --enable-thread-safety make results in: gcc -O2 -fno-strict-aliasing -Wall -Wmissing-prototypes -Wmissing-declarations -fPIC -I. -I../../../src/include -DFRONTEND -DSYSCONFDIR='/usr/local/pgsql/etc' -c -o thread.o thread.c thread.c: In function `pqGetpwuid': thread.c:116: error: too many arguments to function `getpwuid_r' Environment: SunOS 5.7 Generic_106541-29 sun4u sparc SUNW,UltraSPARC-IIi-Engine gcc version 3.3.1 PostgreSQL-7.4.2 ./configure --with-java --enable-thread-safety make' results in: gcc -O2 -fno-strict-aliasing -Wall -Wmissing-prototypes -Wmissing-declarations -fPIC -I. -I../../../src/include -DFRONTEND -DSYSCONFDIR='/usr/local/pgsql/etc' -c -o thread.o thread.c thread.c: In function `pqGetpwuid': thread.c:116: error: too many arguments to function `getpwuid_r' thread.c: In function `pqGethostbyname': thread.c:189: error: `resbuf' undeclared (first use in this function) thread.c:189: error: (Each undeclared identifier is reported only once thread.c:189: error: for each function it appears in.) Diff'ing thread.c between 7.4.1 and 7.4.2, it *looks* like, at first blush, nothing changed that should affect the relevant code. Anybody got any idea what's broken? Regards, Jim -- Jim Seymour | PGP Public Key available at: [EMAIL PROTECTED] | http://www.uk.pgp.net/pgpnet/pks-commands.html http://jimsun.LinxNet.com| ---(end of broadcast)--- TIP 8: explain analyze is your friend
[HACKERS] Further thoughts about warning for costly FK checks
While reviewing Fabien Coelho's patch for emitting warnings for slow foreign-key checks, it occurred to me that we aren't covering all bases. The patch as committed makes sure that there is a usable index on the referenced table, but it does not look for one on the referencing table. Failure to provide such an index will lead to slow DELETEs on the referenced table. And that's a mistake plenty of people make, even without bringing datatype incompatibilities into it. I am tempted to add some more code that issues a WARNING about slow deletes if there's no matching index on the referencing table, or if that index has type-compatibility problems. (It turns out that this is not necessarily the same check as whether the PK index has compatibility problems.) The main problem with doing this is that in the common case of doing CREATE TABLE foo (f1 int REFERENCES bar); there will normally not be any matching index available yet. Unless you want a UNIQUE index, which you often wouldn't, there isn't any way to make the required index during CREATE TABLE; you have to add it later. So I'm worried that adding such a warning would create useless noise during CREATE TABLE. A possible compromise is to issue warnings only during ALTER TABLE ADD CONSTRAINT. I'm not sure how useful that would really be though. Comments, ideas? regards, tom lane ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faqs/FAQ.html
Re: [HACKERS] Further thoughts about warning for costly FK checks
Tom Lane wrote: While reviewing Fabien Coelho's patch for emitting warnings for slow foreign-key checks, it occurred to me that we aren't covering all bases. The patch as committed makes sure that there is a usable index on the referenced table, but it does not look for one on the referencing table. Failure to provide such an index will lead to slow DELETEs on the referenced table. And that's a mistake plenty of people make, even without bringing datatype incompatibilities into it. I am tempted to add some more code that issues a WARNING about slow deletes if there's no matching index on the referencing table, or if that index has type-compatibility problems. (It turns out that this is not necessarily the same check as whether the PK index has compatibility problems.) The main problem with doing this is that in the common case of doing CREATE TABLE foo (f1 int REFERENCES bar); there will normally not be any matching index available yet. Unless you want a UNIQUE index, which you often wouldn't, there isn't any way to make the required index during CREATE TABLE; you have to add it later. So I'm worried that adding such a warning would create useless noise during CREATE TABLE. A possible compromise is to issue warnings only during ALTER TABLE ADD CONSTRAINT. I'm not sure how useful that would really be though. Yes, I was worried about this too, and mentioned it in relation to the pg_statistic bucket size discussion we had. Agreed, there seems to be no good way to emit the warning during table creation. Isn't the ALTER TABLE ADD CONSTRAINT used by pg_dump? Looking at what we have, we know every table will get some inserts, and we know every insert will have to check the primary key. What we don't know is if there will be any modifications or deletes to the primary key. Call me crazy, but maybe we have to throw a message for primary key lookups on foreign key tables without indexes. I hate to throw a message on update/delete rather than create table, but I don't see another way. -- Bruce Momjian| http://candle.pha.pa.us [EMAIL PROTECTED] | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 9: the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [HACKERS] Further thoughts about warning for costly FK checks
Bruce Momjian [EMAIL PROTECTED] writes: Looking at what we have, we know every table will get some inserts, and we know every insert will have to check the primary key. What we don't know is if there will be any modifications or deletes to the primary key. Yeah. It's possible that the user has deliberately omitted the index because he knows that PK deletions will be nonexistent, or at least so infrequent that it's not worth maintaining an index on the FK column. So there are use-cases for it, it's just not the normal case. (This is also the reason not to automatically create the supporting FK index during ADD CONSTRAINT, which is an idea I'd toyed with briefly.) Call me crazy, but maybe we have to throw a message for primary key lookups on foreign key tables without indexes. I hate to throw a message on update/delete rather than create table, but I don't see another way. I don't think that will fly. It's too noisy/repetitive, and it's complaining at the wrong time (in production rather than when you're setting up the DB schema). Imagine how annoying it would be to get such warnings if you were a user without the privileges needed to create the requested index. regards, tom lane ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] 7.4.2 Build broken on (Sparc) Solaris 7 and 8
[EMAIL PROTECTED] (Jim Seymour) writes: Diff'ing thread.c between 7.4.1 and 7.4.2, it *looks* like, at first blush, nothing changed that should affect the relevant code. Anybody got any idea what's broken? The relevant change is probably this one: 2004-02-11 12:32 momjian * configure, configure.in, src/include/pg_config.h.in (REL7_4_STABLE): Properly set NEED_REENTRANT_FUNCS for threaded libpq/ecpg. Without this patch, no thread locking or *_r functions were being used. (I also see some post-7.4.1 changes in src/template/solaris, so you possibly should look there too.) I'm betting that thread.c never worked on your platform, but wasn't compiled before. Easiest answer is probably to not do --enable-thread-safety. If you really want that, you'll need to dig in and figure out how to make that code work on your platform. Send a patch if you figure it out. regards, tom lane ---(end of broadcast)--- TIP 8: explain analyze is your friend
Re: [HACKERS] Timing of 'SELECT 1'
Merlin Moncure wrote: The problem with gprof is that I am going to see all the backend startup stuff too, no? Is there a way to get a dump just the run of the query? I was sort of lurking on this thread, waiting to see what became of it. Did nobody actually come to a conclusion on what that last msec was from? I think the consensus was it was coming from the network layer somehow. If that's the case (it probably is), there isn't a whole lot that can be done about it except to bypass it using server side functions and such. I did a little more research and found an unusual cause. It turns out enabling log_parser/executor_status itself makes SELECT 1's log_duration go from 0.296 to 1.156 so the slowness I was seeing for SELECT 1 was from the measurement, not slowness in the normal code path. -- Bruce Momjian| http://candle.pha.pa.us [EMAIL PROTECTED] | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
Re: [HACKERS] libpq thread safety
Manfred Spraul wrote: Bruce Momjian wrote: What killed the idea of doing ssl or kerberos locking inside libpq was that there was no way to be sure that outside code didn't also access those routines. A callback based implementation can handle that: libpq has a default implementation for apps that do not use openssl or kerberos themself. If the app wants to use the libraries, too, then it must replace the hooks with their own locks. I've attached a simple proposal, just for kerberos 4. If you agree on the general approach, I'll add it to all functions that are not thread safe. I have documented that SSL and Kerberos are not thread-safe in the libpq docs. Let's wait and see If we need additional work in this area. It means that multithreading is not usable: As Tom explained, the connect string is often set directly by the end user. Setting sslmode would result is races - impossible to support. In the very least, sslmode and Kerberos would have to fail if the app is multithreaded. I think it is even worse than you state. SSL and Kerberos is mostly controlled by pg_hba.conf, not the client connection string. Also, while we create a thread-aware libpq, we don't actually have any way to test if threads are being used by the application. Let's go in this direction for Kerberos and SSL, and I can modify the libpq docs on threading. -- Bruce Momjian| http://candle.pha.pa.us [EMAIL PROTECTED] | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: [pgsql-hackers-win32] [HACKERS] What's left?
Greg Stark wrote: Jan Wieck [EMAIL PROTECTED] writes: the point is that PostgreSQL is no GNU product, never has been and if someone intends to he shall do so after yanking out the contributions I made. Note that when you released your contributions you did so under a license that imposed no such conditions. If Microsoft wanted to release a Microsoft Postgresql under a completely proprietary license they would be free to do so. Likewise if someone wanted to release a GPL'd GNU Postgresql they could do it. And nobody could force either to yank anyone's code. I released my contributions under the BSD license. A license change is only possible when accepted by the Copyright holder. I might have missed something, but when did Microsoft get the Copyright of my code? Jan -- #==# # It's easier to get forgiveness for being wrong than for being right. # # Let's break this rule - forgive me. # #== [EMAIL PROTECTED] # ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: [HACKERS] libpq thread safety
Your patch has been added to the PostgreSQL unapplied patches list at:
http://momjian.postgresql.org/cgi-bin/pgpatches
I will try to apply it within the next 48 hours.
---
Manfred Spraul wrote:
Bruce Momjian wrote:
What killed the idea of doing ssl or kerberos locking inside libpq was
that there was no way to be sure that outside code didn't also access
those routines.
A callback based implementation can handle that: libpq has a default
implementation for apps that do not use openssl or kerberos themself. If
the app wants to use the libraries, too, then it must replace the hooks
with their own locks.
I've attached a simple proposal, just for kerberos 4. If you agree on
the general approach, I'll add it to all functions that are not thread safe.
I have documented that SSL and Kerberos are not
thread-safe in the libpq docs. Let's wait and see If we need additional
work in this area.
It means that multithreading is not usable: As Tom explained, the
connect string is often set directly by the end user. Setting sslmode
would result is races - impossible to support. In the very least,
sslmode and Kerberos would have to fail if the app is multithreaded.
--
Manfred
Index: src/interfaces/libpq/fe-auth.c
===
RCS file: /projects/cvsroot/pgsql-server/src/interfaces/libpq/fe-auth.c,v
retrieving revision 1.89
diff -u -r1.89 fe-auth.c
--- src/interfaces/libpq/fe-auth.c7 Jan 2004 18:56:29 - 1.89
+++ src/interfaces/libpq/fe-auth.c12 Mar 2004 20:07:02 -
@@ -590,6 +590,7 @@
case AUTH_REQ_KRB4:
#ifdef KRB4
+ pglock_thread();
if (pg_krb4_sendauth(PQerrormsg, conn-sock,
(struct sockaddr_in *)
conn-laddr.addr,
(struct sockaddr_in *)
conn-raddr.addr,
@@ -597,8 +598,10 @@
{
snprintf(PQerrormsg, PQERRORMSG_LENGTH,
libpq_gettext(Kerberos 4 authentication
failed\n));
+ pgunlock_thread();
return STATUS_ERROR;
}
+ pgunlock_thread();
break;
#else
snprintf(PQerrormsg, PQERRORMSG_LENGTH,
@@ -722,6 +725,7 @@
if (authsvc == 0)
return NULL;/* leave original error message in
place */
+ pglock_thread();
#ifdef KRB4
if (authsvc == STARTUP_KRB4_MSG)
name = pg_krb4_authname(PQerrormsg);
@@ -759,5 +763,6 @@
if (name (authn = (char *) malloc(strlen(name) + 1)))
strcpy(authn, name);
+ pgunlock_thread();
return authn;
}
Index: src/interfaces/libpq/fe-connect.c
===
RCS file: /projects/cvsroot/pgsql-server/src/interfaces/libpq/fe-connect.c,v
retrieving revision 1.268
diff -u -r1.268 fe-connect.c
--- src/interfaces/libpq/fe-connect.c 10 Mar 2004 21:12:47 - 1.268
+++ src/interfaces/libpq/fe-connect.c 12 Mar 2004 20:07:03 -
@@ -3163,4 +3163,34 @@
#undef LINELEN
}
+/*
+ * To keep the API consistent, the locking stubs are always provided, even
+ * if they are not required.
+ */
+pgthreadlock_t *g_threadlock;
+static pgthreadlock_t default_threadlock;
+static void
+default_threadlock(bool acquire)
+{
+#if defined(ENABLE_THREAD_SAFETY)
+ static pthread_mutex_t singlethread_lock = PTHREAD_MUTEX_INITIALIZER;
+ if (acquire)
+ pthread_mutex_lock(singlethread_lock);
+ else
+ pthread_mutex_unlock(singlethread_lock);
+#endif
+}
+
+pgthreadlock_t *
+PQregisterThreadLock(pgthreadlock_t *newhandler)
+{
+ pgthreadlock_t *prev;
+
+ prev = g_threadlock;
+ if (newhandler)
+ g_threadlock = newhandler;
+ else
+ g_threadlock = default_threadlock;
+ return prev;
+}
Index: src/interfaces/libpq/libpq-fe.h
===
RCS file: /projects/cvsroot/pgsql-server/src/interfaces/libpq/libpq-fe.h,v
retrieving revision 1.102
diff -u -r1.102 libpq-fe.h
--- src/interfaces/libpq/libpq-fe.h 9 Jan 2004 02:02:43 - 1.102
+++ src/interfaces/libpq/libpq-fe.h 12 Mar 2004 20:07:03 -
@@ -274,6 +274,22 @@
PQnoticeProcessor proc,
void *arg);
+typedef void (pgsigpipehandler_t)(bool enable, void **state);
+
+extern pgsigpipehandler_t *
+PQregisterSigpipeCallback(pgsigpipehandler_t *newhandler);
+
+/*
+ * Used to set callback that prevents
Re: [pgsql-hackers-win32] [HACKERS] What's left?
Jan Wieck wrote: Greg Stark wrote: Jan Wieck [EMAIL PROTECTED] writes: the point is that PostgreSQL is no GNU product, never has been and if someone intends to he shall do so after yanking out the contributions I made. Note that when you released your contributions you did so under a license that imposed no such conditions. If Microsoft wanted to release a Microsoft Postgresql under a completely proprietary license they would be free to do so. Likewise if someone wanted to release a GPL'd GNU Postgresql they could do it. And nobody could force either to yank anyone's code. I released my contributions under the BSD license. A license change is only possible when accepted by the Copyright holder. I might have missed something, but when did Microsoft get the Copyright of my code? We allow companies to make commercial versions of PostgreSQL that use a proprietary license, so I don't see you could prevent Microsoft from doing the same. -- Bruce Momjian| http://candle.pha.pa.us [EMAIL PROTECTED] | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
[HACKERS] Regression failure for floats
I am seeing the following regression failure for current CVS. On my OS, BSD/OS 4.3, it seems once you hit Infinity, you can't negate it. /usr/include/math.h has: /* Generate an overflow to create +Inf; the multiply shuts up gcc 1 */ #define HUGE_VAL(1e250*1e250) /* IEEE: positive infinity */ and our float4in code has: else if (strcasecmp(num, -Infinity) == 0) val = -HUGE_VAL; Seems that doesn't work for me. --- *** ./expected/float4.out Sat Mar 13 23:07:10 2004 --- ./results/float4.outSat Mar 13 23:10:30 2004 *** *** 58,65 SELECT ' -INFINiTY '::float4; float4 ! --- ! -Infinity (1 row) -- bad special inputs --- 58,65 SELECT ' -INFINiTY '::float4; float4 ! -- ! Infinity (1 row) -- bad special inputs == *** ./expected/float8.out Sat Mar 13 23:07:10 2004 --- ./results/float8.outSat Mar 13 23:10:30 2004 *** *** 58,65 SELECT ' -INFINiTY '::float8; float8 ! --- ! -Infinity (1 row) -- bad special inputs --- 58,65 SELECT ' -INFINiTY '::float8; float8 ! -- ! Infinity (1 row) -- bad special inputs -- Bruce Momjian| http://candle.pha.pa.us [EMAIL PROTECTED] | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
Re: [pgsql-hackers-win32] [HACKERS] What's left?
Bruce Momjian wrote: Jan Wieck wrote: Greg Stark wrote: Jan Wieck [EMAIL PROTECTED] writes: the point is that PostgreSQL is no GNU product, never has been and if someone intends to he shall do so after yanking out the contributions I made. Note that when you released your contributions you did so under a license that imposed no such conditions. If Microsoft wanted to release a Microsoft Postgresql under a completely proprietary license they would be free to do so. Likewise if someone wanted to release a GPL'd GNU Postgresql they could do it. And nobody could force either to yank anyone's code. I released my contributions under the BSD license. A license change is only possible when accepted by the Copyright holder. I might have missed something, but when did Microsoft get the Copyright of my code? We allow companies to make commercial versions of PostgreSQL that use a proprietary license, so I don't see you could prevent Microsoft from doing the same. The BSD license allows everyone to use the code in proprietary software. But that doesn't mean that you can relicense THAT code. I seem to remember that one of our arguments against license changes was that we'd need written agreement from all former contributors. Is that wrong? Jan -- #==# # It's easier to get forgiveness for being wrong than for being right. # # Let's break this rule - forgive me. # #== [EMAIL PROTECTED] # ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: [HACKERS] Regression failure for floats
Bruce Momjian [EMAIL PROTECTED] writes:
I am seeing the following regression failure for current CVS. On my
OS, BSD/OS 4.3, it seems once you hit Infinity, you can't negate it.
Actually, I suspect the problem is that isinf() on your platform
returns 1 for any infinity (rather than -1 for negative infinity and 1
for positive infinity). Some existing code in float4out() and
float8out() assumed that a positive return from isinf() indicated a
positive infinity, which is not per C99.
Anyway, Tom and I worked through this issue, and a couple other
portability problems with the recent float changes, via private
email. The current patch is attached -- Tom hasn't yet gotten back to
me on whether this fixes the problem for him on HPUX, but it fixes my
OS X box.
-Neil
Index: src/backend/utils/adt/float.c
===
RCS file: /var/lib/cvs/pgsql-server/src/backend/utils/adt/float.c,v
retrieving revision 1.99
diff -c -r1.99 float.c
*** a/src/backend/utils/adt/float.c 12 Mar 2004 00:25:40 - 1.99
--- b/src/backend/utils/adt/float.c 14 Mar 2004 01:07:21 -
***
*** 109,117
--- 109,138
static void CheckFloat4Val(double val);
static void CheckFloat8Val(double val);
+ static int is_infinite(double val);
static int float4_cmp_internal(float4 a, float4 b);
static int float8_cmp_internal(float8 a, float8 b);
+ /*
+ * Returns -1 if 'val' represents negative infinity, 1 if 'val'
+ * represents (positive) infinity, and 0 otherwise. On same platforms,
+ * this is equivalent to the isinf() macro, but not everywhere: C99
+ * does not specify that isinf() needs to distinguish between positive
+ * and negative infinity.
+ */
+ static int
+ is_infinite(double val)
+ {
+ int inf = isinf(val);
+
+ if (inf == 0)
+ return 0;
+
+ if (val 0)
+ return 1;
+
+ return -1;
+ }
/*
* check to see if a float4 val is outside of the FLOAT4_MIN,
***
*** 154,160
/*
* float4in - converts num to float
* restricted syntax:
! * {sp} [+|-] {digit} [.{digit}] [exp]
* where sp is a space, digit is 0-9,
* exp is e or E followed by an integer.
*/
--- 175,182
/*
* float4in - converts num to float
* restricted syntax:
! *
! {sp} [+|-] {digit} [.{digit}] [exp]
* where sp is a space, digit is 0-9,
* exp is e or E followed by an integer.
*/
***
*** 162,209
float4in(PG_FUNCTION_ARGS)
{
char *num = PG_GETARG_CSTRING(0);
double val;
char *endptr;
errno = 0;
val = strtod(num, endptr);
if (errno == ERANGE)
ereport(ERROR,
(errcode(ERRCODE_NUMERIC_VALUE_OUT_OF_RANGE),
! errmsg(\%s\ is out of range for type real, num)));
if (num == endptr)
{
/*
! * We didn't find anything that looks like a float in the input
! *
! * In releases prior to 7.5, we accepted an empty string as
! * valid input (yielding a float8 of 0). In 7.5, we accept
! * empty strings, but emit a warning noting that the feature
! * is deprecated. In 7.6+, the warning should be replaced by
! * an error.
*/
! if (*num == '\0')
{
- ereport(WARNING,
- (errcode(ERRCODE_WARNING_DEPRECATED_FEATURE),
- errmsg(deprecated input syntax for type real: \\),
- errdetail(This input will be rejected in
- a future release of PostgreSQL.)));
- Assert(val == 0.0);
- }
- else if (strcasecmp(num, NaN) == 0)
val = NAN;
! else if (strcasecmp(num, Infinity) == 0)
val = HUGE_VAL;
! else if (strcasecmp(num, -Infinity) == 0)
val = -HUGE_VAL;
else
ereport(ERROR,
(errcode(ERRCODE_INVALID_TEXT_REPRESENTATION),
errmsg(invalid input syntax for type real: \%s\,
! num)));
}
/* skip trailing whitespace */
--- 184,261
float4in(PG_FUNCTION_ARGS)
{
char *num = PG_GETARG_CSTRING(0);
+ char *orig_num;
double val;
char *endptr;
+ /*
+ * endptr points to the first character _after_ the sequence we
+ * recognized as a valid floating point number. orig_num points to
+ * the original input string.
+ */
+ orig_num = num;
+
+ /*
+ * Check for an empty-string input to begin with, to avoid
+ * the vagaries of strtod() on different platforms.
+ *
+ * In releases prior to 7.5, we accepted an empty string as valid
+ * input (yielding a float4 of 0). In 7.5, we accept empty
+ * strings, but emit a warning noting that the feature is
+ * deprecated. In 7.6+, the warning should be replaced by an
+ * error.
+ */
+ if (*num == '\0')
+ {
+ ereport(WARNING,
+ (errcode(ERRCODE_WARNING_DEPRECATED_FEATURE),
+ errmsg(deprecated input syntax for type real: \\),
+ errdetail(This input will be rejected in
+ a future release of PostgreSQL.)));
+ PG_RETURN_FLOAT4((float4) 0.0);
+ }
+
+ /* skip leading whitespace */
+ while (*num !=
Re: [pgsql-hackers-win32] [HACKERS] What's left?
Jan Wieck wrote: Bruce Momjian wrote: Jan Wieck wrote: Greg Stark wrote: Jan Wieck [EMAIL PROTECTED] writes: the point is that PostgreSQL is no GNU product, never has been and if someone intends to he shall do so after yanking out the contributions I made. Note that when you released your contributions you did so under a license that imposed no such conditions. If Microsoft wanted to release a Microsoft Postgresql under a completely proprietary license they would be free to do so. Likewise if someone wanted to release a GPL'd GNU Postgresql they could do it. And nobody could force either to yank anyone's code. I released my contributions under the BSD license. A license change is only possible when accepted by the Copyright holder. I might have missed something, but when did Microsoft get the Copyright of my code? We allow companies to make commercial versions of PostgreSQL that use a proprietary license, so I don't see you could prevent Microsoft from doing the same. The BSD license allows everyone to use the code in proprietary software. But that doesn't mean that you can relicense THAT code. I seem to remember that one of our arguments against license changes was that we'd need written agreement from all former contributors. Is that wrong? You know, that is a good point. When someone makes a proprietary version of PostgreSQL, what are they licensing as proprietary? The binary or our source code? When someone takes our code, modifies it, then makes a propriety version, are their additions only proprietary? -- Bruce Momjian| http://candle.pha.pa.us [EMAIL PROTECTED] | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] Regression failure for floats
Bruce Momjian [EMAIL PROTECTED] writes: I am seeing the following regression failure for current CVS. On my OS, BSD/OS 4.3, it seems once you hit Infinity, you can't negate it. /usr/include/math.h has: /* Generate an overflow to create +Inf; the multiply shuts up gcc 1 */ #define HUGE_VAL(1e250*1e250) /* IEEE: positive infinity */ and our float4in code has: else if (strcasecmp(num, -Infinity) == 0) val = -HUGE_VAL; Seems that doesn't work for me. No, the bug is that Neil assumed isinf() would tell him the sign of an infinity. I believe he's about to commit a patch for this and some other bogosities. regards, tom lane ---(end of broadcast)--- TIP 8: explain analyze is your friend
[HACKERS] try to find out the checkpoint record?
Currently we need to read pg_control to know the location(LSN) of the checkpoint record. This means if pg_control is lost or corrupted, we have to give up the database recovery. I think we could start from the first WAL segment and read through entire WAL logs to find out the latest valid checkpoint record. This may take considerable amount of time, but still better than giving up recovery IMO. Any reason we cannot do this? -- Tatsuo Ishii ---(end of broadcast)--- TIP 7: don't forget to increase your free space map settings
Re: [HACKERS] 7.4.2 Build broken on (Sparc) Solaris 7 and 8
Tom Lane [EMAIL PROTECTED] wrote: [snip] (I also see some post-7.4.1 changes in src/template/solaris, so you possibly should look there too.) [snip] I think I have the fix for part of it, but this remains... gcc -O2 -fno-strict-aliasing -Wall -Wmissing-prototypes -Wmissing-declarations -fPIC -I. -I../../../src/include -D_POSIX_PTHREAD_SEMANTICS -DFRONTEND -DSYSCONFDIR='/usr/local/pgsql/etc' -c -o thread.o thread.c thread.c: In function `pqGethostbyname': thread.c:189: error: `resbuf' undeclared (first use in this function) Looking at src/port/thread.c, line 189, it looks like somebody typo'd. Looks like that second parameter should be resultbuf, not resbuf? The above compile error happens on Solaris 7, but not Solaris 8. So next I have to see why the difference and if there *should* be a difference. But I still suspect the above is a coding error. Let me know and, if so, I'll fix it here, look into the other part, and submit a patch for it all. Regards, Jim ---(end of broadcast)--- TIP 7: don't forget to increase your free space map settings
Re: [HACKERS] Regression failure for floats
email. The current patch is attached -- Tom hasn't yet gotten back to me on whether this fixes the problem for him on HPUX, but it fixes my OS X box. Fixes issues under win32. Thanks. Claudio --- Certain disclaimers and policies apply to all email sent from Memetrics. For the full text of these disclaimers and policies see a href=http://www.memetrics.com/emailpolicy.html;http://www.memetrics.com/em ailpolicy.html/a ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faqs/FAQ.html
