Re: [HACKERS] trivial DoS on char recoding
On Tue, Jun 20, 2006 at 06:10:38PM -0400, Tom Lane wrote: > > Should we get firmer in rejecting invalid configurations? > > The question is how sure are we whether a configuration is "invalid". > AFAIK there's not a really portable way to determine which encoding > matches a locale. initdb has a kluge that seems to work most of the > time, but do we want the database to refuse to start when it doesn't? Well, this "kludge" is the recommended and documented way to do it on glibc based systems as well as many others. It turns out however that there is a libcharset[1] for portably determining the charset for your current locale. What's most interesting about it is that it has tables for various OSes and mappings from their names to standard names (the ones used by Glibc). It's LGPL so we can't include the stuff verbatim, but it's not a lot of code. I'm not sure why we persist in beleiving this test is so unreliable we won't even emit a warning... [1] http://www.haible.de/bruno/packages-libcharset.html Have a nice day, -- Martijn van Oosterhout http://svana.org/kleptog/ > From each according to his ability. To each according to his ability to > litigate. signature.asc Description: Digital signature
Re: [HACKERS] shall we have a TRACE_MEMORY mode
"Alvaro Herrera" <[EMAIL PROTECTED]> wrote > > But the problem (or at last a part of the problem) is not what context > each chunk is allocated in, but where did a given chunk come from (where > was it allocated), Which is why saving __FILE__/__LINE__ is useful. > Agreed. Maybe we should not clutter these trace info in the AllocChunkData. We save them in a separe memory context which is only activated when TRACE_MEMORY is on. Also, recording every __FILE__/__LINE__ seems not neccessary, we merge them and record the count of calls. Once a leak is happened, the usual suspect is the high-count one. So the output of memory context dump will be looks like this: execQual.c 1953123456 execHash.c 208 12 ... > > #ifdef TRACE_MEMORY > #define lappend(_list_, _elt_) \ > lappend_tracemem(_list_, _elt_, __FILE__, __LINE__) > #endif > This might be the only portable way I could think of. We don't want to redefine all of the functions calling palloc()/MemoryContextAlloc(), we redefine the most suspectable ones like those in heaptuple.c. Regards, Qingqing ---(end of broadcast)--- TIP 9: In versions below 8.0, the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
[HACKERS] My new email address
After twelve years of using the domain candle.pha.pa.us, I have moved to a new domain that is easier to remember, momjian.us. New email address: [EMAIL PROTECTED] New web site: http://momjian.us The domain candle.pha.pa.us will continue to function indefinitely, but if you have me as in your email address book, you might want to update your records. Thanks. -- Bruce Momjian [EMAIL PROTECTED] EnterpriseDBhttp://www.enterprisedb.com + If your life is a hard drive, Christ can be your backup. + ---(end of broadcast)--- TIP 4: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] trivial DoS on char recoding
Alvaro Herrera <[EMAIL PROTECTED]> writes: > Note that the PO file for the spanish translation is written in Latin1, > not UTF8. So I can adventure that the server is trying to recode a > string which is originally in Latin1, but assuming it is UTF-8, to > Win1250. Yeah, this is a known problem --- basically it seems a shortcoming of the gettext() API. You can find details in the archives. > Should we get firmer in rejecting invalid configurations? The question is how sure are we whether a configuration is "invalid". AFAIK there's not a really portable way to determine which encoding matches a locale. initdb has a kluge that seems to work most of the time, but do we want the database to refuse to start when it doesn't? regards, tom lane ---(end of broadcast)--- TIP 1: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] union all bug?
Tom Lane wrote: Joe Conway <[EMAIL PROTECTED]> writes: I was trying to work around limitations with "partitioning" of tables using constraint exclusion, when I ran across this little oddity: I think you're under a misimpression about the syntax behavior of ORDER BY and UNION. Per spec, ORDER BY binds less tightly than UNION, thus Thanks for the explanation. Now it makes sense :-) Joe ---(end of broadcast)--- TIP 2: Don't 'kill -9' the postmaster
Re: [HACKERS] trivial DoS on char recoding
Alvaro Herrera wrote: > To reproduce, you using a non-C locale is (es_ES works for me). *blush* Sorry, I rewrote this phrase and obviously didn't reread it very carefully :-) It means that you must use a non-C locale. -- Alvaro Herrerahttp://www.CommandPrompt.com/ PostgreSQL Replication, Consulting, Custom Development, 24x7 support ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [HACKERS] UPDATE crash in HEAD and 8.1
Tom Lane wrote: > Alvaro Herrera <[EMAIL PROTECTED]> writes: > > Tom Lane wrote: > >> I'm fairly sure this query is illegal per spec. There are ancient > >> discussions in the archives about whether aggregates in an UPDATE target > >> list can have a consistent interpretation or not. We never found one, > >> but never got around to disallowing it either. Maybe it's time. If you > >> try it with something like sum() you don't get a crash, but you do get > >> rather bizarre behavior. > > > Yeah, I agree we should disallow it. For the curious, the bizarre behavior > > is > > > alvherre=# update pk set id = count(id) ; > > ERROR: ctid is NULL > > Hmm, what version are you testing? What I see is that it updates a > single one of the table rows :-( The trick seems to be that the table must be empty. I'm doing this in 8.1.3. -- Alvaro Herrerahttp://www.CommandPrompt.com/ The PostgreSQL Company - Command Prompt, Inc. ---(end of broadcast)--- TIP 9: In versions below 8.0, the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [HACKERS] UPDATE crash in HEAD and 8.1
Alvaro Herrera <[EMAIL PROTECTED]> writes: > Tom Lane wrote: >> I'm fairly sure this query is illegal per spec. There are ancient >> discussions in the archives about whether aggregates in an UPDATE target >> list can have a consistent interpretation or not. We never found one, >> but never got around to disallowing it either. Maybe it's time. If you >> try it with something like sum() you don't get a crash, but you do get >> rather bizarre behavior. > Yeah, I agree we should disallow it. For the curious, the bizarre behavior > is > alvherre=# update pk set id = count(id) ; > ERROR: ctid is NULL Hmm, what version are you testing? What I see is that it updates a single one of the table rows :-( I found the previous discussion (or one such, anyway): http://archives.postgresql.org/pgsql-bugs/2000-07/msg00046.php That message mentions "ctid is NULL" in the context of a join update, but for the single-table case, all the versions I've tried seem to do the other thing. It's pretty broken either way of course ... regards, tom lane ---(end of broadcast)--- TIP 2: Don't 'kill -9' the postmaster
[HACKERS] trivial DoS on char recoding
Oswaldo Hernandez just reported this in the pgsql-es-ayuda list. Basically, a conversion between UTF8 and windows_1250 can crash the server. I recall a bug around this general code but I don't recall it being able to provoke a PANIC. To reproduce, create a cluster with UTF-8 encoding and locale es_ES (I'm actually using es_CL but it should be the same). Note that the es_ES locale is declared to use Latin1 encoding, not UTF-8. In a psql session, template1=# copy foo from '/tmp/foo' ; ERROR: no existe la relación «foo» template1=# \encoding latin1 template1=# copy foo from '/tmp/foo' ; ERROR: could not convert UTF8 character 0x00f3 to ISO8859-1 template1=# \encoding windows_1250 template1=# copy foo from '/tmp/foo' ; PANIC: ERRORDATA_STACK_SIZE exceeded Table "foo" nor the /tmp/foo file need to exist. In the server logs, I set "log_line_prefix" to %x (Xid) to make it obvious that these reports are in processing the same message. When the PANIC occurs, the server logs this: 574 ERROR: no existe la relación «foo» 574 WARNING: ignorando el carácter UTF-8 no convertible 0xf36e20ab 574 WARNING: ignorando el carácter UTF-8 no convertible 0xe16374 574 WARNING: ignorando el carácter UTF-8 no convertible 0xe16374 574 WARNING: ignorando el carácter UTF-8 no convertible 0xe16374 574 PANIC: ERRORDATA_STACK_SIZE exceeded 574 SENTENCIA: copy foo from '/tmp/datoscopy' ; To reproduce, you using a non-C locale is (es_ES works for me). If I start the postmaster with -C lc_messages=C, the problem does not occur. Note that the PO file for the spanish translation is written in Latin1, not UTF8. So I can adventure that the server is trying to recode a string which is originally in Latin1, but assuming it is UTF-8, to Win1250. Now, it can be argued that this is really operator error -- because I can't crash the server if I correctly initdb with es_CL.UTF8. Should we get firmer in rejecting invalid configurations? I'm not sure up to what point this affects other translations, collates, encodings -- right now I only have "es" (spanish) compiled and my system is not configured to accept anything else. -- Alvaro Herrerahttp://www.CommandPrompt.com/ PostgreSQL Replication, Consulting, Custom Development, 24x7 support ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [HACKERS] Slightly bogus regression test for contrib/dblink
Joe Conway wrote: ...the "BEGIN" statement returned successfully as usual, but for some reason left (PQtransactionStatus(conn) != PQTRANS_IDLE), causing dblink_open() to start a transaction and later complete it on line 454. Oops, I meant "... some reason left (PQtransactionStatus(conn) == PQTRANS_IDLE), causing ..." Joe ---(end of broadcast)--- TIP 4: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] Slightly bogus regression test for contrib/dblink
Tom Lane wrote:
What's even more interesting is that there are now three later runs of
HEAD on osprey, and none of them failed. So unless Remi's been fooling
with the environment on that machine, this was a one-shot irreproducible
failure. That's disturbing in a different way ...
http://www.pgbuildfarm.org/cgi-bin/show_history.pl?nm=osprey&br=HEAD
I'm trying to imagine how this diff could come about, and all I can
think is that somehow in this sequence...
--[next line is line 453]--
-- this should not commit the transaction because the client opened it
SELECT dblink_close('myconn','rmt_foo_cursor');
dblink_close
--
OK
(1 row)
-- this should succeed because we have an open transaction
SELECT dblink_exec('myconn','DECLARE xact_test CURSOR FOR SELECT * FROM
foo');
dblink_exec
DECLARE CURSOR
(1 row)
--[last line is line 465]--
... if dblink_close() actually (incorrectly) committed the transaction,
I think you would get exactly the diff we got (both hunks).
Now, why that would happen just once, I'm not sure.
That would imply that rconn->newXactForCursor was somehow TRUE, which in
turn implies that in this previous sequence...
--[next line is line 439]--
-- test opening cursor in a transaction
SELECT dblink_exec('myconn','BEGIN');
dblink_exec
-
BEGIN
(1 row)
-- an open transaction will prevent dblink_open() from opening its own
SELECT dblink_open('myconn','rmt_foo_cursor','SELECT * FROM foo');
dblink_open
-
OK
(1 row)
--[last line is line 451]--
...the "BEGIN" statement returned successfully as usual, but for some
reason left (PQtransactionStatus(conn) != PQTRANS_IDLE), causing
dblink_open() to start a transaction and later complete it on line 454.
Is that somehow possible?
Joe
---(end of broadcast)---
TIP 9: In versions below 8.0, the planner will ignore your desire to
choose an index scan if your joining column's datatypes do not
match
Re: [HACKERS] Slightly bogus regression test for contrib/dblink
Tom Lane <[EMAIL PROTECTED]> writes: > Joe Conway <[EMAIL PROTECTED]> writes: >> Tom Lane wrote: >>> It looks to me like the diffs are consistent with the idea that the >>> test is using a copy of dblink that predates this patch ... >> I would think that the diffs would be significantly larger if that were >> the case. In fact, when was PG_MODULE_MAGIC first made mandatory? > Good point. So then why the failure? What's even more interesting is that there are now three later runs of HEAD on osprey, and none of them failed. So unless Remi's been fooling with the environment on that machine, this was a one-shot irreproducible failure. That's disturbing in a different way ... http://www.pgbuildfarm.org/cgi-bin/show_history.pl?nm=osprey&br=HEAD regards, tom lane ---(end of broadcast)--- TIP 6: explain analyze is your friend
Re: [HACKERS] Slightly bogus regression test for contrib/dblink
Joe Conway <[EMAIL PROTECTED]> writes: > Tom Lane wrote: >>> It looks to me like the diffs are consistent with the idea that the >>> test is using a copy of dblink that predates this patch ... > I would think that the diffs would be significantly larger if that were > the case. In fact, when was PG_MODULE_MAGIC first made mandatory? Good point. So then why the failure? regards, tom lane ---(end of broadcast)--- TIP 5: don't forget to increase your free space map settings
Re: [HACKERS] Slightly bogus regression test for contrib/dblink
Tom Lane wrote: Joe Conway <[EMAIL PROTECTED]> writes: I haven't really looked at the buildfarm before -- I might be blind, but I couldn't figure out how to see the regression.diff file. It's on the cited page, if you scroll down far enough. OK, I'm officially blind (so much for that lasik work I had done ;-)), I had scrolled right on past the diff the first time. > It looks to me like the diffs are consistent with the idea that the > test is using a copy of dblink that predates this patch ... do you > agree? If so, anyone have an idea how that could happen? I thought > we'd fixed all the rpath problems, and anyway osprey wasn't failing > like this before today. I would think that the diffs would be significantly larger if that were the case. In fact, when was PG_MODULE_MAGIC first made mandatory? Joe ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [HACKERS] Some small code-restructuring issues
Alvaro Herrera <[EMAIL PROTECTED]> writes: > A workaround of sorts would be to mention the origin of the files being > moved, so that an interested person can look it up via the Attic. Yeah, that should be sufficient. The history is actually still there, just attached to the old file location. regards, tom lane ---(end of broadcast)--- TIP 5: don't forget to increase your free space map settings
Re: [HACKERS] Some small code-restructuring issues
Stephen Frost wrote: > * Tom Lane ([EMAIL PROTECTED]) wrote: > > libpgport (ie, move 'em to src/port). Moving them would lose some CVS > > history but would probably be the cleanest thing in the long run. > > Comments? > > Time to consider something other than CVS...? In the end, personally > I'd rather have it be cleaner than the history. There are ways to > preserve the history though, if it's a huge concern, even with CVS... > It's just ugly... Every hack to the repository makes it more difficult to get it to migrate cleanly to something else. A workaround of sorts would be to mention the origin of the files being moved, so that an interested person can look it up via the Attic. -- Alvaro Herrerahttp://www.CommandPrompt.com/ The PostgreSQL Company - Command Prompt, Inc. ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [HACKERS] UPDATE crash in HEAD and 8.1
Tom Lane wrote: Alvaro Herrera <[EMAIL PROTECTED]> writes: update pk set id = max(id) + 2; I'm fairly sure this query is illegal per spec. There are ancient discussions in the archives about whether aggregates in an UPDATE target list can have a consistent interpretation or not. We never found one, but never got around to disallowing it either. Maybe it's time. If you try it with something like sum() you don't get a crash, but you do get rather bizarre behavior. On 8.x (7.4 and 7.3 as well) it will update "1" row :). On 8.1 and HEAD it crashes. This has been verified on 32bit, 64bit x86 and PPC. Having said that, this may well expose a bug in the MAX-optimization code that has consequences for more useful queries. I'll take a look later today if no one beats me to it. Sincerely, Joshua D. Drake regards, tom lane ---(end of broadcast)--- TIP 4: Have you searched our list archives? http://archives.postgresql.org -- === The PostgreSQL Company: Command Prompt, Inc. === Sales/Support: +1.503.667.4564 || 24x7/Emergency: +1.800.492.2240 Providing the most comprehensive PostgreSQL solutions since 1997 http://www.commandprompt.com/ ---(end of broadcast)--- TIP 5: don't forget to increase your free space map settings
Re: [HACKERS] Initdb segfaults during "initializing pg_authid"
Looks like the distclean may have done it. I thought I had already, but who knows. Thanks. -Wade On Tuesday 20 June 2006 09:51, Tom Lane wrote: > Wade Klaver <[EMAIL PROTECTED]> writes: > > Initdb seems to barf on me during the pg_authid bit. Below are the > > specifics. Please ask if you need anything else. The build is CVS -HEAD. > > Are you sure it's a clean build? "make distclean" and trying again is > often the first thing to try when seeing unexpected problems with a CVS > pull. > > If that doesn't help, please try it with --enable-debug --enable-cassert > so we can get more info. > > regards, tom lane -- Wade Klaver Wavefire Technologies Corporation GPG Public Key at http://archeron.wavefire.com /"\ ASCII Ribbon Campaign . \ / - NO HTML/RTF in e-mail . X - NO Word docs in e-mail . / \ - ---(end of broadcast)--- TIP 2: Don't 'kill -9' the postmaster
Re: [HACKERS] UPDATE crash in HEAD and 8.1
Tom Lane wrote: > Alvaro Herrera <[EMAIL PROTECTED]> writes: > > update pk set id = max(id) + 2; > > I'm fairly sure this query is illegal per spec. There are ancient > discussions in the archives about whether aggregates in an UPDATE target > list can have a consistent interpretation or not. We never found one, > but never got around to disallowing it either. Maybe it's time. If you > try it with something like sum() you don't get a crash, but you do get > rather bizarre behavior. Yeah, I agree we should disallow it. For the curious, the bizarre behavior is alvherre=# update pk set id = count(id) ; ERROR: ctid is NULL alvherre=# update pk set id = sum(id) ; ERROR: ctid is NULL Clearly not a very useful error message. > Having said that, this may well expose a bug in the MAX-optimization > code that has consequences for more useful queries. I'll take a look > later today if no one beats me to it. I refrain -- tried following it but I don't know that code at all. -- Alvaro Herrerahttp://www.CommandPrompt.com/ The PostgreSQL Company - Command Prompt, Inc. ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [HACKERS] UPDATE crash in HEAD and 8.1
Tom Lane wrote: Alvaro Herrera <[EMAIL PROTECTED]> writes: update pk set id = max(id) + 2; I'm fairly sure this query is illegal per spec. There are ancient discussions in the archives about whether aggregates in an UPDATE target list can have a consistent interpretation or not. We never found one, but never got around to disallowing it either. Maybe it's time. If you try it with something like sum() you don't get a crash, but you do get rather bizarre behavior. Having said that, this may well expose a bug in the MAX-optimization code that has consequences for more useful queries. I'll take a look later today if no one beats me to it. If you try the query on 8.0 and before you don't get a crash either, but the result is unexpected. Try this version: create table pk (id bigserial primary key, orig bigint); insert into pk (id) values (DEFAULT); insert into pk (id) values (DEFAULT); insert into pk (id) values (DEFAULT); update pk set orig = id; select * from pk; update pk set id = max(id) + 2; select * from pk; One could almost argue that crashing would be better ;-) cheers andrew ---(end of broadcast)--- TIP 5: don't forget to increase your free space map settings
Re: [HACKERS] Generic Monitoring Framework Proposal
Greg Stark wrote: It seems pointless to me to expose things like lwlock_acuire that map 1-1 to C function calls like LWLockAcquire. They're useless except to people who understand what's going on and if people know the low level implementation details of Postgres they can already trace those calls with dtrace without any help. lwlock_acquire is just an example. I think once we decided to down this path, we can solicit ideas for interesting probes and put them up for discussion on this alias whether or not they are needed. I think we need to have two categories of probes for admins and developers. Perhaps the probes for admins are more important since, as you said, the developers already know which function does what, but I think the low-level probes are still useful for new developers as there behavior will be documented. What would be useful is instrumenting high level calls that can't be traced without application guidance. For example, inserting a dtrace probe for each SQL and each plan node. That way someone could get the same info as EXPLAIN ANALYZE from a production server without having to make application modifications (or suffer the gettimeofday overhead). It's one thing to know "I seem to be acquiring a lot of locks" or "i'm spending all my time in sorting". It's another to be able to ask dtrace "what query am I running when doing all this sorting?" or "what kind of plan node am I running when I'm acquiring all these locks?" Completely agree. Regards, Robert ---(end of broadcast)--- TIP 9: In versions below 8.0, the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [HACKERS] Some small code-restructuring issues
* Tom Lane ([EMAIL PROTECTED]) wrote: > libpgport (ie, move 'em to src/port). Moving them would lose some CVS > history but would probably be the cleanest thing in the long run. > Comments? Time to consider something other than CVS...? In the end, personally I'd rather have it be cleaner than the history. There are ways to preserve the history though, if it's a huge concern, even with CVS... It's just ugly... Enjoy, Stephen signature.asc Description: Digital signature
Re: [HACKERS] UPDATE crash in HEAD and 8.1
Alvaro Herrera <[EMAIL PROTECTED]> writes: > update pk set id = max(id) + 2; I'm fairly sure this query is illegal per spec. There are ancient discussions in the archives about whether aggregates in an UPDATE target list can have a consistent interpretation or not. We never found one, but never got around to disallowing it either. Maybe it's time. If you try it with something like sum() you don't get a crash, but you do get rather bizarre behavior. Having said that, this may well expose a bug in the MAX-optimization code that has consequences for more useful queries. I'll take a look later today if no one beats me to it. regards, tom lane ---(end of broadcast)--- TIP 4: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] Slightly bogus regression test for contrib/dblink
Tom Lane wrote: Note to Andrew: would it make sense for the larger log files to be split out as linked pages in a buildfarm report? regards, tom lane I will put it on the TODO list. cheers andrew ---(end of broadcast)--- TIP 6: explain analyze is your friend
Re: [HACKERS] Generic Monitoring Framework Proposal
Simon Riggs wrote: This needs to work on Linux and Windows, minimum, also. The proposed solution will work on Linux & Windows if they similar facility that the macros can map to. Otherwise, the macros stay as no-ops and will not affect those platforms at all. It's obviously impossible to move a production system to a different OS just to use a cool tracing tool. So the architecture must intelligently handle the needs of multiple OS - even if the underlying facilities on them do not yet provide what we'd like. So I'm OK with Solaris being the best, just as long as its not the only one that benefits. The way it's proposed now, any OS can use the same interfaces and map to their underlying facilities. Does it look reasonable? Regards, Robert ---(end of broadcast)--- TIP 4: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] Slightly bogus regression test for contrib/dblink
Joe Conway <[EMAIL PROTECTED]> writes: > Tom Lane wrote: >> BTW, I was led to notice this while examining the current buildfarm >> failure report from osprey, >> http://www.pgbuildfarm.org/cgi-bin/show_log.pl?nm=osprey&dt=2006-06-17%2004:00:16 > I haven't really looked at the buildfarm before -- I might be blind, but > I couldn't figure out how to see the regression.diff file. It's on the cited page, if you scroll down far enough. Note to Andrew: would it make sense for the larger log files to be split out as linked pages in a buildfarm report? regards, tom lane ---(end of broadcast)--- TIP 6: explain analyze is your friend
[HACKERS] Some small code-restructuring issues
In pursuit of eliminating some redundant gettimeofday() calls, I just tried to change struct Port's session_start field to TimestampTz, which necessitated including utils/timestamp.h in libpq/libpq-be.h. That caused things to blow up real good :-(. The problem is that backend/libpq/md5.c includes c.h (not postgres.h) and then libpq/crypt.h, and then the latter includes libpq/libpq-be.h, and that fails because c.h doesn't define such things as Datum. This is all pretty horrid code really, and I think we need to clean it up. A relatively minimal fix would be to separate out the declarations for md5.c into its own header file include/libpq/md5.h, which would not need to include any backend-only definitions. More aggressively, we could decide that since md5.c and ip.c are supposed to be both backend and frontend code, they don't belong here at all, but should be in libpgport (ie, move 'em to src/port). Moving them would lose some CVS history but would probably be the cleanest thing in the long run. Comments? regards, tom lane ---(end of broadcast)--- TIP 6: explain analyze is your friend
Re: [HACKERS] pltcl -- solved
Hi Tom, On Tue, 20 Jun 2006, Tom Lane wrote: > Date: Tue, 20 Jun 2006 12:42:24 -0400 > From: Tom Lane <[EMAIL PROTECTED]> > To: ohp@pyrenet.fr > Cc: pgsql-hackers list > Subject: Re: [HACKERS] pltcl -- solved > > ohp@pyrenet.fr writes: > > Just a note that pltcl is now passing regression tests on Unixware. > > For some unexplained reason, it did'nt pass with tcl-8.5 but is ok with > > tcl -8.4.13. > > AFAICT there is no "tcl 8.5" yet; there is an alpha release tcl8.5a4 > which is stated to still be under active feature development. Yes > I installed it locally and pltcl seems to work OK with it, so I'm not > sure what problem you saw. One possibility is that the Tcl folk > recommend against building extensions with a different compiler than > was used to build Tcl ... so if you tried to build pltcl with gcc > against a Tcl compiled with Unixware cc, or vice versa, that might be > the core of the problem. It's not the case. > > Anyway I'm not too concerned about misbehavior with an alpha Tcl release > --- it's as likely to be their bug as ours. > I agree, I'll test it again when tcl 8.5 is out... > regards, tom lane > Best regards -- Olivier PRENANT Tel: +33-5-61-50-97-00 (Work) 15, Chemin des Monges+33-5-61-50-97-01 (Fax) 31190 AUTERIVE +33-6-07-63-80-64 (GSM) FRANCE Email: ohp@pyrenet.fr -- Make your life a dream, make your dream a reality. (St Exupery) ---(end of broadcast)--- TIP 9: In versions below 8.0, the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [HACKERS] Slightly bogus regression test for contrib/dblink
Tom Lane wrote:
Lines 509-512 of contrib/dblink/expected/dblink.out read:
-- this should fail because there is no open transaction
SELECT dblink_exec('myconn','DECLARE xact_test CURSOR FOR SELECT * FROM foo');
ERROR: sql error
DETAIL: ERROR: cursor "xact_test" already exists
The error message is not consistent with what the comment claims.
Looking at the test case in total, I think the code is responding
Actually the comment was correct, but there was a bug which caused the
automatically opened transaction to not get closed.
I was really expecting a "DECLARE CURSOR may only be used in transaction
blocks" error there, but didn't notice that I was actually getting a
different error message :-(.
The bug can be reproduced by using dblink_open to automatically open a
transaction, and then using dblink_exec to manually ABORT it:
-- open a test connection
SELECT dblink_connect('myconn','dbname=contrib_regression');
dblink_connect
OK
(1 row)
-- open a cursor incorrectly; this bumps up the refcount
contrib_regression=# SELECT
dblink_open('myconn','rmt_foo_cursor','SELECT * FROM foobar',false);
NOTICE: sql error
DETAIL: ERROR: relation "foobar" does not exist
dblink_open
-
ERROR
(1 row)
-- manually abort remote transaction; does not maintain refcount
SELECT dblink_exec('myconn','ABORT');
dblink_exec
-
ROLLBACK
(1 row)
-- test automatic transaction; this bumps up the refcount
SELECT dblink_open('myconn','rmt_foo_cursor','SELECT * FROM foo');
dblink_open
-
OK
(1 row)
-- this should commit the automatically opened transaction
-- but it doesn't because the refcount is wrong
SELECT dblink_close('myconn','rmt_foo_cursor');
dblink_close
--
OK
(1 row)
-- this should fail because there is no open transaction
-- but it doesn't because dblink_close did not commit
SELECT dblink_exec('myconn','DECLARE xact_test2 CURSOR FOR SELECT * FROM
foo');
dblink_exec
DECLARE CURSOR
(1 row)
I think the attached patch does the trick in a minimally invasive way.
If there are no objections I'll commit to head and 8.1 stable branches.
The problem doesn't exist before 8.1.
BTW, I was led to notice this while examining the current buildfarm
failure report from osprey,
http://www.pgbuildfarm.org/cgi-bin/show_log.pl?nm=osprey&dt=2006-06-17%2004:00:16
It looks to me like the diffs are consistent with the idea that the test
is using a copy of dblink that predates this patch ... do you agree?
If so, anyone have an idea how that could happen? I thought we'd fixed
all the rpath problems, and anyway osprey wasn't failing like this
before today.
I haven't really looked at the buildfarm before -- I might be blind, but
I couldn't figure out how to see the regression.diff file.
Joe
? .deps
? .regression.diffs.swp
? current.diff
? dblink.sql
? libdblink.so.0.0
? results
Index: dblink.c
===
RCS file: /cvsroot/pgsql/contrib/dblink/dblink.c,v
retrieving revision 1.55
diff -c -r1.55 dblink.c
*** dblink.c 30 May 2006 22:12:12 - 1.55
--- dblink.c 20 Jun 2006 16:28:01 -
***
*** 361,366
--- 361,373
DBLINK_RES_INTERNALERROR("begin error");
PQclear(res);
rconn->newXactForCursor = TRUE;
+ /*
+ * Since transaction state was IDLE, we force cursor count to
+ * initially be 0. This is needed as a previous ABORT might
+ * have wiped out our transaction without maintaining the
+ * cursor count for us.
+ */
+ rconn->openCursorCount = 0;
}
/* if we started a transaction, increment cursor count */
Index: expected/dblink.out
===
RCS file: /cvsroot/pgsql/contrib/dblink/expected/dblink.out,v
retrieving revision 1.16
diff -c -r1.16 dblink.out
*** expected/dblink.out 18 Oct 2005 02:55:49 - 1.16
--- expected/dblink.out 20 Jun 2006 16:28:01 -
***
*** 509,515
-- this should fail because there is no open transaction
SELECT dblink_exec('myconn','DECLARE xact_test CURSOR FOR SELECT * FROM foo');
ERROR: sql error
! DETAIL: ERROR: cursor "xact_test" already exists
-- reset remote transaction state
SELECT dblink_exec('myconn','ABORT');
--- 509,515
-- this should fail because there is no open transaction
SELECT dblink_exec('myconn','DECLARE xact_test CURSOR FOR SELECT * FROM foo');
ERROR: sql error
! DETAIL: ERROR: DECLARE CURSOR may only be used in transaction blocks
-- reset remote transaction state
SELECT dblink_exec('myconn','ABORT');
---(end of broadcast)---
TIP 9: In versions below 8.0, the planner will ignore your desire to
choose an index scan if your joining column's datatypes do not
match
[HACKERS] UPDATE crash in HEAD and 8.1
A bug reported by Josh Drake, crashes 8.1 and CVS HEAD: Test case is: create table pk (id bigserial primary key); insert into pk values (DEFAULT); insert into pk values (DEFAULT); insert into pk values (DEFAULT); update pk set id = max(id) + 2; -- SEGV simple eh? :-) The backtrace on HEAD looks like this: (gdb) bt #0 slot_getattr (slot=0x0, attnum=-1, isnull=0x83fc3f9 "\177~\177\177\177\177\177¬\005A\b@") at /pgsql/source/00orig/src/backend/access/common/heaptuple.c:1288 #1 0x08168ae1 in ExecProject (projInfo=0x83fc40c, isDone=0xafecda2c) at /pgsql/source/00orig/src/backend/executor/execQual.c:3847 #2 0x08176c9d in ExecResult (node=0x83fbce0) at /pgsql/source/00orig/src/backend/executor/nodeResult.c:157 #3 0x081647e5 in ExecProcNode (node=0x83fbce0) at /pgsql/source/00orig/src/backend/executor/execProcnode.c:329 #4 0x0816315b in ExecutorRun (queryDesc=0x8404698, direction=ForwardScanDirection, count=0) at /pgsql/source/00orig/src/backend/executor/execMain.c:1139 #5 0x081f8298 in ProcessQuery (parsetree=, plan=0x8412670, params=0x0, dest=0x8412754, completionTag=0xafecdcb8 "") at /pgsql/source/00orig/src/backend/tcop/pquery.c:174 #6 0x081f94c2 in PortalRun (portal=0x84024bc, count=2147483647, dest=0x8412754, altdest=0x8412754, completionTag=0xafecdcb8 "") at /pgsql/source/00orig/src/backend/tcop/pquery.c:1079 #7 0x081f484f in exec_simple_query ( query_string=0x83f0ffc "update pk set id = max(id) + 2;") at /pgsql/source/00orig/src/backend/tcop/postgres.c:1025 NULL slot!? -- Alvaro Herrerahttp://www.CommandPrompt.com/ The PostgreSQL Company - Command Prompt, Inc. ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [HACKERS] Initdb segfaults during "initializing pg_authid"
Wade Klaver <[EMAIL PROTECTED]> writes: > Initdb seems to barf on me during the pg_authid bit. Below are the > specifics. > Please ask if you need anything else. The build is CVS -HEAD. Are you sure it's a clean build? "make distclean" and trying again is often the first thing to try when seeing unexpected problems with a CVS pull. If that doesn't help, please try it with --enable-debug --enable-cassert so we can get more info. regards, tom lane ---(end of broadcast)--- TIP 2: Don't 'kill -9' the postmaster
Re: [HACKERS] shall we have a TRACE_MEMORY mode
Alvaro Herrera <[EMAIL PROTECTED]> writes: > Simon Riggs wrote: >> Could we set that as an option for each memory context when we create >> it? All or nothing seems too extreme for me for most cases. > What most cases? There is only one case -- there is a big leak and you > want to find out where. There's a more significant reason why not, which is that all AllocChunks must have the same header, else pfree doesn't know what to do. >> That seems mostly the hard way to me, because our memory management >> scheme is *not* based around "thou shalt free() what thou malloc()ed". >> You'd need a tool that understood about resetting memory contexts >> (recursively) to get anywhere at all in analyzing such a trace. > Of course. It's not difficult to do that; just tedious. I wrote such a > tool to debug a Mammoth Replicator problem (I don't think I've kept it > though). The logging code must emit messages about context creation, > destruction and reset, and have the alloc message indicate what context > is the chunk being created on. Well, the logging approach would definitely be less intrusive to the system's runtime behavior, and would (maybe) not require gdb to use. If you can resurrect that tool it'd be interesting to look at. Maybe it's on a backup tape somewhere? regards, tom lane ---(end of broadcast)--- TIP 4: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] pltcl -- solved
ohp@pyrenet.fr writes: > Just a note that pltcl is now passing regression tests on Unixware. > For some unexplained reason, it did'nt pass with tcl-8.5 but is ok with > tcl -8.4.13. AFAICT there is no "tcl 8.5" yet; there is an alpha release tcl8.5a4 which is stated to still be under active feature development. I installed it locally and pltcl seems to work OK with it, so I'm not sure what problem you saw. One possibility is that the Tcl folk recommend against building extensions with a different compiler than was used to build Tcl ... so if you tried to build pltcl with gcc against a Tcl compiled with Unixware cc, or vice versa, that might be the core of the problem. Anyway I'm not too concerned about misbehavior with an alpha Tcl release --- it's as likely to be their bug as ours. regards, tom lane ---(end of broadcast)--- TIP 5: don't forget to increase your free space map settings
[HACKERS] Initdb segfaults during "initializing pg_authid"
Hello folks, Initdb seems to barf on me during the pg_authid bit. Below are the specifics. Please ask if you need anything else. The build is CVS -HEAD. Initdb output: [EMAIL PROTECTED]:bin/initdb The files belonging to this database system will be owned by user "pgsql". This user must also own the server process. The database cluster will be initialized with locale C. creating directory /usr/local/pgsql/data ... ok creating subdirectories ... ok selecting default max_connections ... 100 selecting default shared_buffers/max_fsm_pages ... 3500/175000 creating configuration files ... ok creating template1 database in /usr/local/pgsql/data/base/1 ... ok initializing pg_authid ... Segmentation fault (core dumped) child process exited with exit code 139 initdb: removing data directory "/usr/local/pgsql/data" [EMAIL PROTECTED]:uname -a FreeBSD arch.wavefire.com 6.0-STABLE FreeBSD 6.0-STABLE #0: Thu Nov 3 10:59:55 PST 2005 [EMAIL PROTECTED]:/usr/obj/usr/src/sys/WORKSTATION-5.0-SMP i386 (gdb) bt #0 0x080bb086 in FuncnameGetCandidates () #1 0x080e08d6 in LookupFuncName () #2 0x0810ebf3 in CreateTrigger () #3 0x08197c57 in PortalRunUtility () #4 0x081981ad in PortalRun () #5 0x08194306 in exec_simple_query () #6 0x08196394 in PostgresMain () #7 0x0813c908 in main () -- Wade Klaver Wavefire Technologies Corporation GPG Public Key at http://archeron.wavefire.com /"\ ASCII Ribbon Campaign . \ / - NO HTML/RTF in e-mail . X - NO Word docs in e-mail . / \ - ---(end of broadcast)--- TIP 1: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] Generic Monitoring Framework Proposal
Tom Lane <[EMAIL PROTECTED]> writes: > Greg Stark <[EMAIL PROTECTED]> writes: > > What would be useful is instrumenting high level calls that can't be traced > > without application guidance. For example, inserting a dtrace probe for each > > SQL and each plan node. That way someone could get the same info as EXPLAIN > > ANALYZE from a production server without having to make application > > modifications (or suffer the gettimeofday overhead). > > My bogometer just went off again. How is something like dtrace going to > magically get realtime information without reading the clock? Sorry, I meant get the same info as EXPLAIN ANALYZE minus the timing. I'm not familiar with DTrace first-hand but I did have the impression it was possible to get timing information though. I don't know how much overhead it has but I wouldn't be surprised if it was lower for a kernel-based profiling elapsed time counter on Sun hardware than a general purpose gettimeofday call on commodity PC hardware. For example it could use a cpu instruction counter and have hooks in the scheduler for saving and restoring the counter to avoid the familiar gotchas with being rescheduled across processors. -- greg ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [HACKERS] Generic Monitoring Framework Proposal
Greg Stark <[EMAIL PROTECTED]> writes: > What would be useful is instrumenting high level calls that can't be traced > without application guidance. For example, inserting a dtrace probe for each > SQL and each plan node. That way someone could get the same info as EXPLAIN > ANALYZE from a production server without having to make application > modifications (or suffer the gettimeofday overhead). My bogometer just went off again. How is something like dtrace going to magically get realtime information without reading the clock? regards, tom lane ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [HACKERS] sync_file_range()
"Tom Lane" <[EMAIL PROTECTED]> writes: > > Indeed, I've been wondering lately if we shouldn't resurrect > > LET_OS_MANAGE_FILESIZE and make that the default on systems with > > largefile support. If nothing else it would cut down on open/close > > overhead on very large relations. > > > I'd still put some limit on the filesize, else you cannot manually > > distribute a table across spindles anymore. Also some > backup solutions > > are not too happy with too large files eighter (they have > trouble with > > staging the backup). I would suggest something like 32 Gb. > > Well, some people would find those arguments compelling and > some wouldn't. We already have a manually configurable > RELSEG_SIZE, so people who want a 32Gb or whatever segment > size can have it. > But if you're dealing with terabyte-sized tables that's still > a lot of segments. > > What I'd be inclined to do is allow people to set RELSEG_SIZE > = 0 in pg_config_manual.h to select the unsegmented option. > That way we already have the infrastructure in pg_control etc > to ensure that the database layout matches the backend. That sounds perfect. Still leaves the question of what to default to ? Another issue is, that we would probably need to detect large file support of the underlying filesystem, else we might fail at runtime :-( Andreas ---(end of broadcast)--- TIP 1: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] Generic Monitoring Framework Proposal
Robert Lor <[EMAIL PROTECTED]> writes: > Yes, I'm proposing user-space probes (aka User Statically-Defined Tracing - > USDT). USDT provides a high-level abstraction so the application can expose > well defined probes without the user having to know the detailed > implementation. For example, instead of having to know the function > LWLockAcquire(), a well documented probe called lwlock_acquire with the > appropriate args is much more usable. It seems pointless to me to expose things like lwlock_acuire that map 1-1 to C function calls like LWLockAcquire. They're useless except to people who understand what's going on and if people know the low level implementation details of Postgres they can already trace those calls with dtrace without any help. What would be useful is instrumenting high level calls that can't be traced without application guidance. For example, inserting a dtrace probe for each SQL and each plan node. That way someone could get the same info as EXPLAIN ANALYZE from a production server without having to make application modifications (or suffer the gettimeofday overhead). It's one thing to know "I seem to be acquiring a lot of locks" or "i'm spending all my time in sorting". It's another to be able to ask dtrace "what query am I running when doing all this sorting?" or "what kind of plan node am I running when I'm acquiring all these locks?" -- greg ---(end of broadcast)--- TIP 6: explain analyze is your friend
Re: [HACKERS] shall we have a TRACE_MEMORY mode
Tom Lane <[EMAIL PROTECTED]> writes: > Another thing to consider is that the proximate location of the palloc > is frequently *not* very useful. For instance, if your memory is > getting eaten by lists, all the palloc traces will point at > new_tail_cell(). Not much help. I don't know what to do about that > ... any ideas? Well the traditional thing to do is store a backtrace a la Dmalloc, one of the better debugging malloc libraries out there. It has mostly been superceded by Purify/Valgrind type tools but it still has a place for handling memory leaks. It's unfortunate that's impossible to use Dmalloc with Postgres. It would probably be nigh impossible to merge in Dmalloc code into Postgres's allocator given the different models. But perhaps it would be possible to steal specific pieces of it like the backtrace grabbing code. -- greg ---(end of broadcast)--- TIP 9: In versions below 8.0, the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [HACKERS] sync_file_range()
"Zeugswetter Andreas DCP SD" <[EMAIL PROTECTED]> writes: > Indeed, I've been wondering lately if we shouldn't resurrect > LET_OS_MANAGE_FILESIZE and make that the default on systems with > largefile support. If nothing else it would cut down on open/close > overhead on very large relations. > I'd still put some limit on the filesize, else you cannot manually > distribute a table across spindles anymore. Also some backup solutions > are not too happy with too large files eighter (they have trouble > with staging the backup). I would suggest something like 32 Gb. Well, some people would find those arguments compelling and some wouldn't. We already have a manually configurable RELSEG_SIZE, so people who want a 32Gb or whatever segment size can have it. But if you're dealing with terabyte-sized tables that's still a lot of segments. What I'd be inclined to do is allow people to set RELSEG_SIZE = 0 in pg_config_manual.h to select the unsegmented option. That way we already have the infrastructure in pg_control etc to ensure that the database layout matches the backend. regards, tom lane ---(end of broadcast)--- TIP 4: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] sync_file_range()
Simon Riggs <[EMAIL PROTECTED]> writes: > So we would use the async properties of sync, but not the file range > support? That's the part of it that looked potentially useful to me, anyway. I don't see any value for us in syncing just part of a file, because we don't have enough disk layout knowledge to make intelligent choices of what to sync. I think the OP had some idea of having the bgwriter write and then force-sync individual pages, but what good is that? Once we've done the write() the page is exposed to the kernel's write scheduler and should be written at an intelligent time. Trying to force sync in advance of our own real need for it to be synced (ie the next checkpoint) doesn't seem to me to offer any benefit. regards, tom lane ---(end of broadcast)--- TIP 1: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] shall we have a TRACE_MEMORY mode
On Tue, Jun 20, 2006 at 12:18:32AM -0400, Tom Lane wrote: > Another thing to consider is that the proximate location of the palloc > is frequently *not* very useful. For instance, if your memory is > getting eaten by lists, all the palloc traces will point at > new_tail_cell(). Not much help. I don't know what to do about that > ... any ideas? GCC has __builtin_return_address (LEVEL) which returns the frame address of the LEVELth caller (on systems where this is possible). You could perhaps track the caller and the callers caller. glibc comes with a function called backtrace(0 which can be used to grab several levels simultaneously. You can use dladdr() to turn these into useful addresses. These are probably not portable, on the whole. As for overhead, maybe you can deal with that by only tracing blocks that exceed a megabyte or more. Perhaps a small test: if( increasing size of context over 10 MB ) dump_stack_trace() Ofcourse, you might just miss the allocations you need to look at... The backtrace_symbols_fd() function dumps straight to a file, if you want to avoid cluttering up the logs. Have a nice day, -- Martijn van Oosterhout http://svana.org/kleptog/ > From each according to his ability. To each according to his ability to > litigate. signature.asc Description: Digital signature
Re: [HACKERS] PAM auth
Satoshi Nagayasu wrote: > Andrew Dunstan wrote: > >don't use system auth. PAM can authenticate from many sources, not just > >the system password files. LDAP is a commonly used source. > > The reason why I'm trying to use PAM, is I need a feature > to account lock-out after N-times login failures on PG, > like pam_tally module. I think Andrew is suggesting using LDAP atop PAM, so you'd be able to use pam_tally anyway -- just switch away from "system-auth". -- Alvaro Herrerahttp://www.CommandPrompt.com/ PostgreSQL Replication, Consulting, Custom Development, 24x7 support ---(end of broadcast)--- TIP 6: explain analyze is your friend
Re: [HACKERS] shall we have a TRACE_MEMORY mode
Andrew Dunstan wrote: > Alvaro Herrera said: > > > > >> That seems mostly the hard way to me, because our memory management > >> scheme is *not* based around "thou shalt free() what thou malloc()ed". > >> You'd need a tool that understood about resetting memory contexts > >> (recursively) to get anywhere at all in analyzing such a trace. > > > > Of course. It's not difficult to do that; just tedious. I wrote such > > a tool to debug a Mammoth Replicator problem (I don't think I've kept > > it though). The logging code must emit messages about context > > creation, destruction and reset, and have the alloc message indicate > > what context is the chunk being created on. > > Could we tag each context with its name? Then we could centralise a lot of > this, ISTM, and the overhead involved in setting the tag at context creation > doesn't seem like a heavy price to pay. Each context already keeps track of its own name. But the problem (or at last a part of the problem) is not what context each chunk is allocated in, but where did a given chunk come from (where was it allocated), Which is why saving __FILE__/__LINE__ is useful. Regarding stuff allocated by lappend(), makeNode() or other functions which centralizedly allocate in the name of the caller, maybe we could enhance the prototypes to get __FILE__ and __LINE__ from their caller. That would help pinpoint the true source of allocation. Something like #ifdef TRACE_MEMORY #define lappend(_list_, _elt_) \ lappend_tracemem(_list_, _elt_, __FILE__, __LINE__) #endif etc. -- Alvaro Herrerahttp://www.CommandPrompt.com/ The PostgreSQL Company - Command Prompt, Inc. ---(end of broadcast)--- TIP 5: don't forget to increase your free space map settings
Re: [HACKERS] Generic Monitoring Framework Proposal
On Mon, Jun 19, 2006 at 05:14:15PM -0400, Chris Browne wrote: > [EMAIL PROTECTED] (Robert Lor) writes: > > For DTrace, probes can be enabled using a D script. When the probes > > are not enabled, there is absolutely no performance hit whatsoever. > > That seems inconceivable. > > In order to have a way of deciding whether or not the probes are > enabled, there has *got* to be at least one instruction executed, and > that can't be costless. I think the trick is that the probe are enabled by overwriting bits of code. So by default you might put a No-Op instruction and if you want to trace you replace that with an illegal instruction or the special one-byte INT3 instruction x86 system have for this purpose. With a 17-stage pipelined processor I imagine the cost of a no-op would indeed be almost unmeasurable (increase code size I suppose). Have a nice day, -- Martijn van Oosterhout http://svana.org/kleptog/ > From each according to his ability. To each according to his ability to > litigate. signature.asc Description: Digital signature
Re: [HACKERS] shall we have a TRACE_MEMORY mode
Alvaro Herrera said: > >> That seems mostly the hard way to me, because our memory management >> scheme is *not* based around "thou shalt free() what thou malloc()ed". >> You'd need a tool that understood about resetting memory contexts >> (recursively) to get anywhere at all in analyzing such a trace. > > Of course. It's not difficult to do that; just tedious. I wrote such > a tool to debug a Mammoth Replicator problem (I don't think I've kept > it though). The logging code must emit messages about context > creation, destruction and reset, and have the alloc message indicate > what context is the chunk being created on. > Could we tag each context with its name? Then we could centralise a lot of this, ISTM, and the overhead involved in setting the tag at context creation doesn't seem like a heavy price to pay. cheers andrew ---(end of broadcast)--- TIP 5: don't forget to increase your free space map settings
Re: [HACKERS] shall we have a TRACE_MEMORY mode
Simon Riggs wrote: > On Tue, 2006-06-20 at 00:18 -0400, Tom Lane wrote: > > > One idea that comes to mind is to have a compile time option to record > > the palloc __FILE__ and _LINE__ in every AllocChunk header. Then it > > would not be so hard to identify the culprit while trawling through > > memory. The overhead costs would be so high that you'd never turn it on > > by default though :-( > > Could we set that as an option for each memory context when we create > it? All or nothing seems too extreme for me for most cases. What most cases? There is only one case -- there is a big leak and you want to find out where. You don't have this code turned on all the time, you must enable it at compile time, so we want it to be as simple as possible. On Tue, 2006-06-20 at 00:18 -0400, Tom Lane wrote: > That seems mostly the hard way to me, because our memory management > scheme is *not* based around "thou shalt free() what thou malloc()ed". > You'd need a tool that understood about resetting memory contexts > (recursively) to get anywhere at all in analyzing such a trace. Of course. It's not difficult to do that; just tedious. I wrote such a tool to debug a Mammoth Replicator problem (I don't think I've kept it though). The logging code must emit messages about context creation, destruction and reset, and have the alloc message indicate what context is the chunk being created on. -- Alvaro Herrerahttp://www.CommandPrompt.com/ PostgreSQL Replication, Consulting, Custom Development, 24x7 support ---(end of broadcast)--- TIP 4: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] CVS HEAD busted on Windows?
> -Original Message- > From: Andrew Dunstan [mailto:[EMAIL PROTECTED] > Sent: 20 June 2006 12:12 > To: Dave Page > Cc: Tom Lane; Peter Eisentraut; pgsql-hackers@postgresql.org > Subject: Re: [HACKERS] CVS HEAD busted on Windows? > > > 1. it's nothing to do with upgrading - I saw this also and on > a machine > that hasn't upgraded the buildfarm script. Nope, didn't think it did - just happened at the same time. > 2. It looks like you got past initdb, so Tom's theory about that is > looking unlikely. Yeah - I've been seeing both issues though - Tom's one where it complains that postgres.exe doesn't exist then bombs out of initdb (it does exist btw), and the one above where make check hangs and I end up killing it off. Regards, Dave ---(end of broadcast)--- TIP 4: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] PAM auth
Andrew Dunstan wrote: don't use system auth. PAM can authenticate from many sources, not just the system password files. LDAP is a commonly used source. The reason why I'm trying to use PAM, is I need a feature to account lock-out after N-times login failures on PG, like pam_tally module. I'm going to try LDAP next. Thanks. -- NAGAYASU Satoshi <[EMAIL PROTECTED]> Phone: +81-3-3523-8122 ---(end of broadcast)--- TIP 2: Don't 'kill -9' the postmaster
Re: [HACKERS] CVS HEAD busted on Windows?
Dave Page wrote: -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Tom Lane Sent: 20 June 2006 00:02 To: Peter Eisentraut; pgsql-hackers@postgresql.org Subject: [HACKERS] CVS HEAD busted on Windows? I notice buildfarm member snake is unhappy: The program "postgres" is needed by initdb but was not found in the same directory as "C:/msys/1.0/local/build-farm/HEAD/pgsql.696/src/test/regress/ tmp_check/install/usr/local/build-farm/HEAD/inst/bin/initdb.exe". Check your installation. I'm betting Peter's recent patch to merge postmaster and postgres missed some little thing or other ... Yeah, I had guessed at that, but as the breakage coincided with a 'make check' hang and a buildfarm script upgrade I was waiting until after tonights run (following a cleanup yesterday) before reporting. What I'm seeing this morning is another hang in 'make check'. After killing it off, the status can be seen at: http://www.pgbuildfarm.org/cgi-bin/show_log.pl?nm=snake&dt=2006-06-20%20 01:00:01 1. it's nothing to do with upgrading - I saw this also and on a machine that hasn't upgraded the buildfarm script. 2. It looks like you got past initdb, so Tom's theory about that is looking unlikely. cheers andrew ---(end of broadcast)--- TIP 1: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] PAM auth
Satoshi Nagayasu wrote: >Albe, > >Albe Laurenz wrote: > > >>/etc/pam.d/system-auth probably uses pam_unix.so to authenticate. >> >>Does the user exist on the machine and have the password you try? >> >> > >Yes, I have same user name on my linux box and postgresql, >and they have same password (now). > > > >>You could add 'debug' to the pam_unix.so lines in /etc/pam.d/system-auth >>and capture what PAM logs to syslog, maybe that will help. >> >> > >Finally, by my small program, I found the PAM module is attempting >to read /etc/shadow to authenticate, but /etc/shadow can't be read >by non-superuser privilege. > >I know, the postmaster is running under "postgres" user privilege, >so PAM auth will always cause 'permission denied' around /etc/shadow. > >How can I solve this? Any ideas? > don't use system auth. PAM can authenticate from many sources, not just the system password files. LDAP is a commonly used source. cheers andrew ---(end of broadcast)--- TIP 6: explain analyze is your friend
[HACKERS] pltcl -- solved
Hi all Just a note that pltcl is now passing regression tests on Unixware. For some unexplained reason, it did'nt pass with tcl-8.5 but is ok with tcl -8.4.13. build farm build script updated accordingly. My next try will be python. Regards -- Olivier PRENANT Tel: +33-5-61-50-97-00 (Work) 15, Chemin des Monges+33-5-61-50-97-01 (Fax) 31190 AUTERIVE +33-6-07-63-80-64 (GSM) FRANCE Email: ohp@pyrenet.fr -- Make your life a dream, make your dream a reality. (St Exupery) ---(end of broadcast)--- TIP 9: In versions below 8.0, the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [HACKERS] sync_file_range()
> > Indeed, I've been wondering lately if we shouldn't resurrect > > LET_OS_MANAGE_FILESIZE and make that the default on systems with > > largefile support. If nothing else it would cut down on open/close > > overhead on very large relations. I'd still put some limit on the filesize, else you cannot manually distribute a table across spindles anymore. Also some backup solutions are not too happy with too large files eighter (they have trouble with staging the backup). I would suggest something like 32 Gb. Andreas ---(end of broadcast)--- TIP 9: In versions below 8.0, the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [HACKERS] Generic Monitoring Framework Proposal
On Mon, 2006-06-19 at 19:36 -0400, Theo Schlossnagle wrote: > The idea of having intelligently placed dtrace probes in Postrgres > would allow us ... > to troubleshoot[ing] obtuse production > problems. That, to me, is exciting stuff. [paraphrased by SR] I very much agree with the requirement here. This needs to work on Linux and Windows, minimum, also. It's obviously impossible to move a production system to a different OS just to use a cool tracing tool. So the architecture must intelligently handle the needs of multiple OS - even if the underlying facilities on them do not yet provide what we'd like. So I'm OK with Solaris being the best, just as long as its not the only one that benefits. -- Simon Riggs EnterpriseDB http://www.enterprisedb.com ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [HACKERS] shall we have a TRACE_MEMORY mode
On Tue, 2006-06-20 at 00:18 -0400, Tom Lane wrote: > One idea that comes to mind is to have a compile time option to record > the palloc __FILE__ and _LINE__ in every AllocChunk header. Then it > would not be so hard to identify the culprit while trawling through > memory. The overhead costs would be so high that you'd never turn it on > by default though :-( Could we set that as an option for each memory context when we create it? All or nothing seems too extreme for me for most cases. -- Simon Riggs EnterpriseDB http://www.enterprisedb.com ---(end of broadcast)--- TIP 2: Don't 'kill -9' the postmaster
Re: [HACKERS] sync_file_range()
On Mon, 2006-06-19 at 21:35 -0400, Tom Lane wrote: > Greg Stark <[EMAIL PROTECTED]> writes: > > Come to think of it I wonder whether there's anything to be gained by using > > smaller files for tables. Instead of 1G files maybe 256M files or something > > like that to reduce the hit of fsyncing a file. > sync_file_range() is not that exactly, but since it lets you request > syncing and then go back and wait for the syncs later, we could get the > desired effect with two passes over the file list. (If the file list > is longer than our allowed number of open files, though, the extra > opens/closes could hurt.) So we would use the async properties of sync, but not the file range support? Sounds like it could help with multiple filesystems. > Indeed, I've been wondering lately if we shouldn't resurrect > LET_OS_MANAGE_FILESIZE and make that the default on systems with > largefile support. If nothing else it would cut down on open/close > overhead on very large relations. Agreed. -- Simon Riggs EnterpriseDB http://www.enterprisedb.com ---(end of broadcast)--- TIP 1: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] PAM auth
Albe, Albe Laurenz wrote: > /etc/pam.d/system-auth probably uses pam_unix.so to authenticate. > > Does the user exist on the machine and have the password you try? Yes, I have same user name on my linux box and postgresql, and they have same password (now). > You could add 'debug' to the pam_unix.so lines in /etc/pam.d/system-auth > and capture what PAM logs to syslog, maybe that will help. Finally, by my small program, I found the PAM module is attempting to read /etc/shadow to authenticate, but /etc/shadow can't be read by non-superuser privilege. I know, the postmaster is running under "postgres" user privilege, so PAM auth will always cause 'permission denied' around /etc/shadow. How can I solve this? Any ideas? Thanks. -- NAGAYASU Satoshi <[EMAIL PROTECTED]> Phone: +81-3-3523-8122 ---(end of broadcast)--- TIP 9: In versions below 8.0, the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [HACKERS] CVS HEAD busted on Windows?
> -Original Message- > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED] On Behalf Of Tom Lane > Sent: 20 June 2006 00:02 > To: Peter Eisentraut; pgsql-hackers@postgresql.org > Subject: [HACKERS] CVS HEAD busted on Windows? > > I notice buildfarm member snake is unhappy: > > The program "postgres" is needed by initdb but was not found in the > same directory as > "C:/msys/1.0/local/build-farm/HEAD/pgsql.696/src/test/regress/ > tmp_check/install/usr/local/build-farm/HEAD/inst/bin/initdb.exe". > Check your installation. > > I'm betting Peter's recent patch to merge postmaster and > postgres missed > some little thing or other ... Yeah, I had guessed at that, but as the breakage coincided with a 'make check' hang and a buildfarm script upgrade I was waiting until after tonights run (following a cleanup yesterday) before reporting. What I'm seeing this morning is another hang in 'make check'. After killing it off, the status can be seen at: http://www.pgbuildfarm.org/cgi-bin/show_log.pl?nm=snake&dt=2006-06-20%20 01:00:01 As I'm currently running all three builds from one task, I'll go separate them out now... Regards, Dave. ---(end of broadcast)--- TIP 6: explain analyze is your friend
Re: [HACKERS] shall we have a TRACE_MEMORY mode
On 6/20/06, Tom Lane <[EMAIL PROTECTED]> wrote: One idea that comes to mind is to have a compile time option to record the palloc __FILE__ and _LINE__ in every AllocChunk header. Then it would not be so hard to identify the culprit while trawling through memory. The overhead costs would be so high that you'd never turn it on by default though :-( Will adding 8 bytes, that too as a compile-time option, be a big overhead? 4 for __FILE__'s char* and 4 for __LINE__'s int; this, assuming 32 bit arch, and that no duplicates of __FILE__ string for each file are stored in the binary by the compiler, also called 'Enable string Pooling' in VS.Net (http://msdn2.microsoft.com/en-us/library/s0s0asdt.aspx). Another thing to consider is that the proximate location of the palloc is frequently *not* very useful. For instance, if your memory is getting eaten by lists, all the palloc traces will point at new_tail_cell(). Not much help. I don't know what to do about that ... any ideas? We can consider such utility functions equivalent to palloc, hence the caller's __FILE__ and __LINE__ will passed in to these functions, and these functions will use the same to call the palloc (or the palloc's #define expanded). So, in effect, in the log files, allocation will seem to have been done from the location which called the utility function. Regards, Gurjeet. ---(end of broadcast)--- TIP 6: explain analyze is your friend
Re: [HACKERS] PAM auth
satoshi nagayasu wrote: > I'm trying to use PAM auth on PostgreSQL, but I still cannot > get success on PAM auth (with PG813 and RHEL3). > > pg_hba.conf has > > hostpamtest all 0.0.0.0/0 pam > > /etc/pam.d/postgresql is > > #%PAM-1.0 > > auth required pam_stack.so service=system-auth > > accountrequired pam_stack.so service=system-auth > > password required pam_stack.so service=system-auth > > And I've changed user password with "ALTER USER ... PASSWORD". > > However, my postmaster always denies my login. /etc/pam.d/system-auth probably uses pam_unix.so to authenticate. Does the user exist on the machine and have the password you try? You could add 'debug' to the pam_unix.so lines in /etc/pam.d/system-auth and capture what PAM logs to syslog, maybe that will help. Yours, Laurenz Albe ---(end of broadcast)--- TIP 4: Have you searched our list archives? http://archives.postgresql.org
