[HACKERS] pg_dump
I know that new pg_dump can dump out large objects. But what about pg_dumpall? Do we have to dump out a whole database cluster by using pg_dumpall then run pg_dump separetly to dump large objects? That seems pain... -- Tatsuo Ishii ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
[HACKERS] beta6 pg_restore core dumps
pg_restore crushes if dump data includes large objects...
--
Tatsuo Ishii
[t-ishii@srapc1474 7.1]$ createdb test
CREATE DATABASE
[t-ishii@srapc1474 7.1]$ psql -c "select lo_import('/boot/vmlinuz')" test
lo_import
---
20736
(1 row)
[t-ishii@srapc1474 7.1]$ pg_dump -F c -b test test.db
[t-ishii@srapc1474 7.1]$ createdb test2
CREATE DATABASE
[t-ishii@srapc1474 7.1]$ pg_restore -d test2 test.db
Segmentation fault (core dumped)
[t-ishii@srapc1474 7.1]$ gdb pg_restore core
GNU gdb 5.0
Copyright 2000 Free Software Foundation, Inc.
[snip]
#0 0x804abd4 in _enableTriggersIfNecessary (AH=0x8057d30, te=0x0,
ropt=0x8057c90) at pg_backup_archiver.c:474
474 ahprintf(AH, "UPDATE pg_class SET reltriggers = "
(gdb) where
#0 0x804abd4 in _enableTriggersIfNecessary (AH=0x8057d30, te=0x0,
ropt=0x8057c90) at pg_backup_archiver.c:474
#1 0x804a8c0 in RestoreArchive (AHX=0x8057d30, ropt=0x8057c90)
at pg_backup_archiver.c:336
#2 0x804a03e in main (argc=4, argv=0x7864) at pg_restore.c:312
#3 0x2ab9796b in __libc_start_main (main=0x8049a40 main, argc=4,
argv=0x7864, init=0x8049394 _init, fini=0x8052d2c _fini,
rtld_fini=0x2aab5d00 _dl_fini, stack_end=0x785c)
at ../sysdeps/generic/libc-start.c:92
(gdb)
---(end of broadcast)---
TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: [HACKERS] Performance monitor signal handler
At 13:49 16/03/01 -0500, Jan Wieck wrote: Similar problem as with shared memory - size. If a long running backend of a multithousand table database needs to send access stats per table - and had accessed them all up to now - it'll be alot of wasted bandwidth. Not if you only send totals for individual counters when they change; some stats may never be resynced, but for the most part it will work. Also, does Unix allow interrupts to occur as a result of data arrivibg in a pipe? If so, how about: - All backends to do *blocking* IO to collector. - Collector to receive an interrupt when a message arrives; while in the interrupt it reads the buffer into a local queue, and returns from the interrupt. - Main line code processes the queue and writes it to a memory mapped file for durability. - If collector dies, postmaster starts another immediately, which slears the backlog of data in the pipe and then remaps the file. - Each backend has its own local copy of it's counters which *possibly* to collector can ask for when it restarts. Philip Warner| __---_ Albatross Consulting Pty. Ltd. |/ - \ (A.B.N. 75 008 659 498) | /(@) __---_ Tel: (+61) 0500 83 82 81 | _ \ Fax: (+61) 0500 83 82 82 | ___ | Http://www.rhyme.com.au |/ \| |---- PGP key available upon request, | / and from pgp5.ai.mit.edu:11371 |/ ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://www.postgresql.org/search.mpl
Re: [HACKERS] pg_dump
At 17:36 17/03/01 +0900, Tatsuo Ishii wrote: I know that new pg_dump can dump out large objects. But what about pg_dumpall? Do we have to dump out a whole database cluster by using pg_dumpall then run pg_dump separetly to dump large objects? That won't even work, since pg_dump won't dump BLOBs without dumping all the tables in the database. That seems pain... It is if you do not have individual database backup procedures; but pg_dumpall uses the plain text dump format, which, without changes to lo_import, can not restore binary data. If lo_import could load UUENCODED data from STDIN, then maybe we could get it to work. Alternatively, we may be able to put an option on pg_dumpall that will dump to one long script file with embedded TAR archives, but I have not really looked at the option. Philip Warner| __---_ Albatross Consulting Pty. Ltd. |/ - \ (A.B.N. 75 008 659 498) | /(@) __---_ Tel: (+61) 0500 83 82 81 | _ \ Fax: (+61) 0500 83 82 82 | ___ | Http://www.rhyme.com.au |/ \| |---- PGP key available upon request, | / and from pgp5.ai.mit.edu:11371 |/ ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/users-lounge/docs/faq.html
Re: [HACKERS] Performance monitor signal handler
Philip Warner wrote: At 13:49 16/03/01 -0500, Jan Wieck wrote: Similar problem as with shared memory - size. If a long running backend of a multithousand table database needs to send access stats per table - and had accessed them all up to now - it'll be alot of wasted bandwidth. Not if you only send totals for individual counters when they change; some stats may never be resynced, but for the most part it will work. Also, does Unix allow interrupts to occur as a result of data arrivibg in a pipe? If so, how about: - All backends to do *blocking* IO to collector. The general problem remains. We only have one central collector with a limited receive capacity. The more load is on the machine, the smaller it's capacity gets. The more complex the DB schemas get and the more load is on the system, the more interesting accurate statistics get. Both factors are contraproductive. More complex schema means more tables and thus bigger messages. More load means more messages. Having good statistics on a toy system while they get worse for a web backend server that's really under pressure is braindead from the start. We don't want the backends to block, so that they can do THEIR work. That's to process queries, nothing else. Pipes seem to be inappropriate because their buffer is limited to 4K on Linux and most BSD flavours. Message queues are too because they are limited to 2K on most BSD's. So only sockets remain. If we have multiple processes that try to receive from the UDP socket, condense the received packets into summary messages and send them to the central collector, this might solve the problem. Jan -- #==# # It's easier to get forgiveness for being wrong than for being right. # # Let's break this rule - forgive me. # #== [EMAIL PROTECTED] # _ Do You Yahoo!? Get your free @yahoo.com address at http://mail.yahoo.com ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://www.postgresql.org/search.mpl
Re: [HACKERS] Performance monitor signal handler
On Sat, Mar 17, 2001 at 09:33:03AM -0500, Jan Wieck wrote: The general problem remains. We only have one central collector with a limited receive capacity. The more load is on the machine, the smaller it's capacity gets. The more complex the DB schemas get and the more load is on the system, the more interesting accurate statistics get. Both factors are contraproductive. More complex schema means more tables and thus bigger messages. More load means more messages. Having good statistics on a toy system while they get worse for a web backend server that's really under pressure is braindead from the start. Just as another suggestion, what about sending the data to a different computer, so instead of tying up the database server with processing the statistics, you have another computer that has some free time to do the processing. Some drawbacks are that you can't automatically start/restart it from the postmaster and it will put a little more load on the network, but it seems to mostly solve the issues of blocked pipes and using too much cpu time on the database server. ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/users-lounge/docs/faq.html
Re: [HACKERS] beta6 pg_restore core dumps
Tatsuo Ishii [EMAIL PROTECTED] writes:
pg_restore crushes if dump data includes large objects...
This is probably the same problem that Martin Renters reported
yesterday. I have a patch that seems to fix it on my machine,
but I haven't heard back from Martin whether it solves his case
completely. In particular, he said something about memory leaks...
regards, tom lane
*** pg_backup_custom.c.orig Fri Feb 9 17:32:26 2001
--- pg_backup_custom.c Fri Mar 16 17:24:59 2001
***
*** 521,531
if (blkLen (ctx-inSize - 1)) {
free(ctx-zlibIn);
ctx-zlibIn = NULL;
! ctx-zlibIn = (char*)malloc(blkLen);
if (!ctx-zlibIn)
die_horribly(AH, "%s: failed to allocate decompression
buffer\n", progname);
! ctx-inSize = blkLen;
in = ctx-zlibIn;
}
--- 521,531
if (blkLen (ctx-inSize - 1)) {
free(ctx-zlibIn);
ctx-zlibIn = NULL;
! ctx-zlibIn = (char*)malloc(blkLen+1);
if (!ctx-zlibIn)
die_horribly(AH, "%s: failed to allocate decompression
buffer\n", progname);
! ctx-inSize = blkLen+1;
in = ctx-zlibIn;
}
---(end of broadcast)---
TIP 3: if posting/reading through Usenet, please send an appropriate
subscribe-nomail command to [EMAIL PROTECTED] so that your
message can get through to the mailing list cleanly
Re: [HACKERS] Performance monitor signal handler
Samuel Sieb [EMAIL PROTECTED] writes: Just as another suggestion, what about sending the data to a different computer, so instead of tying up the database server with processing the statistics, you have another computer that has some free time to do the processing. Some drawbacks are that you can't automatically start/restart it from the postmaster and it will put a little more load on the network, ... and a lot more load on the CPU. Same-machine "network" connections are much cheaper (on most kernels, anyway) than real network connections. I think all of this discussion is vast overkill. No one has yet demonstrated that it's not sufficient to have *one* collector process and a lossy transmission method. Let's try that first, and if it really proves to be unworkable then we can get out the lily-gilding equipment. But there is tons more stuff to do before we have useful stats at all, and I don't think that this aspect is the most critical part of the problem. regards, tom lane ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send "unregister YourEmailAddressHere" to [EMAIL PROTECTED])
Re: [HACKERS] Stuck spins in current
"Mikheev, Vadim" [EMAIL PROTECTED] writes: xlog.c revision 1.55 from Feb 26 already had log file zero-filling, so ... Oh, you're right, I didn't study the CVS log carefully enough. Hmm, maybe the control file lock isn't the problem. The abort() in s_lock_stuck should have left a core file --- what is the backtrace? After 10 times increasing DEFAULT_TIMEOUT in s_lock.c I got abort in xlog.c:626 - waiting for insert_lck. But problem is near new log file creation code: system goes sleep just after new one is created. Have you learned any more about this? Or can you send your test program so other people can try it? In the meantime, even if it turns out there's a different problem here, it seems clear to me that it's a bad idea to use a plain spinlock to interlock xlog segment creation. The spinlock timeouts are not set high enough to be safe for something that could take several seconds. Unless someone objects, I will go ahead and work on the change I suggested yesterday to not hold the ControlFileLockId spinlock while we are zero-filling the new segment. regards, tom lane ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://www.postgresql.org/search.mpl
Re: [HACKERS] Performance monitor signal handler
... and a lot more load on the CPU. Same-machine "network" connections are much cheaper (on most kernels, anyway) than real network connections. I think all of this discussion is vast overkill. No one has yet demonstrated that it's not sufficient to have *one* collector process and a lossy transmission method. Let's try that first, and if it really proves to be unworkable then we can get out the lily-gilding equipment. But there is tons more stuff to do before we have useful stats at all, and I don't think that this aspect is the most critical part of the problem. Agreed. Sounds like overkill. How about a per-backend shared memory area for stats, plus a global shared memory area that each backend can add to when it exists. That meets most of our problem. The only open issue is per-table stuff, and I would like to see some circular buffer implemented to handle that, with a collection process that has access to shared memory. Even better, have an SQL table updated with the per-table stats periodically. How about a collector process that periodically reads though the shared memory and UPDATE's SQL tables with the information. -- Bruce Momjian| http://candle.pha.pa.us [EMAIL PROTECTED] | (610) 853-3000 + If your life is a hard drive, | 830 Blythe Avenue + Christ can be your backup.| Drexel Hill, Pennsylvania 19026 ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/users-lounge/docs/faq.html
Re: [HACKERS] Performance monitor signal handler
Bruce Momjian [EMAIL PROTECTED] writes: The only open issue is per-table stuff, and I would like to see some circular buffer implemented to handle that, with a collection process that has access to shared memory. That will get us into locking/contention issues. OTOH, frequent trips to the kernel to send stats messages --- regardless of the transport mechanism chosen --- don't seem all that cheap either. Even better, have an SQL table updated with the per-table stats periodically. That will be horribly expensive, if it's a real table. I think you missed the point that somebody made a little while ago about waiting for functions that can return tuple sets. Once we have that, the stats tables can be *virtual* tables, ie tables that are computed on-demand by some function. That will be a lot less overhead than physically updating an actual table. regards, tom lane ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://www.postgresql.org/search.mpl
Re: [HACKERS] Performance monitor signal handler
Bruce Momjian [EMAIL PROTECTED] writes: The only open issue is per-table stuff, and I would like to see some circular buffer implemented to handle that, with a collection process that has access to shared memory. That will get us into locking/contention issues. OTOH, frequent trips to the kernel to send stats messages --- regardless of the transport mechanism chosen --- don't seem all that cheap either. I am confused. Reading/writing shared memory is not a kernel call, right? I agree on the locking contention problems of a circular buffer. Even better, have an SQL table updated with the per-table stats periodically. That will be horribly expensive, if it's a real table. But per-table stats aren't something that people will look at often, right? They can sit in the collector's memory for quite a while. See people wanting to look at per-backend stuff frequently, and that is why I thought share memory should be good, and a global area for aggregate stats for all backends. I think you missed the point that somebody made a little while ago about waiting for functions that can return tuple sets. Once we have that, the stats tables can be *virtual* tables, ie tables that are computed on-demand by some function. That will be a lot less overhead than physically updating an actual table. Yes, but do we want to keep these stats between postmaster restarts? And what about writing them to tables when our storage of table stats gets too big? -- Bruce Momjian| http://candle.pha.pa.us [EMAIL PROTECTED] | (610) 853-3000 + If your life is a hard drive, | 830 Blythe Avenue + Christ can be your backup.| Drexel Hill, Pennsylvania 19026 ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
Re: [HACKERS] Performance monitor signal handler
Bruce Momjian [EMAIL PROTECTED] writes: Even better, have an SQL table updated with the per-table stats periodically. That will be horribly expensive, if it's a real table. But per-table stats aren't something that people will look at often, right? They can sit in the collector's memory for quite a while. See people wanting to look at per-backend stuff frequently, and that is why I thought share memory should be good, and a global area for aggregate stats for all backends. I think you missed the point that somebody made a little while ago about waiting for functions that can return tuple sets. Once we have that, the stats tables can be *virtual* tables, ie tables that are computed on-demand by some function. That will be a lot less overhead than physically updating an actual table. Yes, but do we want to keep these stats between postmaster restarts? And what about writing them to tables when our storage of table stats gets too big? All those points seem to me to be arguments in *favor* of a virtual- table approach, not arguments against it. Or are you confusing the method of collecting stats with the method of making the collected stats available for use? regards, tom lane ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send "unregister YourEmailAddressHere" to [EMAIL PROTECTED])
Re: [HACKERS] Performance monitor signal handler
But per-table stats aren't something that people will look at often, right? They can sit in the collector's memory for quite a while. See people wanting to look at per-backend stuff frequently, and that is why I thought share memory should be good, and a global area for aggregate stats for all backends. I think you missed the point that somebody made a little while ago about waiting for functions that can return tuple sets. Once we have that, the stats tables can be *virtual* tables, ie tables that are computed on-demand by some function. That will be a lot less overhead than physically updating an actual table. Yes, but do we want to keep these stats between postmaster restarts? And what about writing them to tables when our storage of table stats gets too big? All those points seem to me to be arguments in *favor* of a virtual- table approach, not arguments against it. Or are you confusing the method of collecting stats with the method of making the collected stats available for use? Maybe I am confusing them. I didn't see a distinction in the discussion. I assumed the UDP/message passing of information to the collector was the way statistics were collected, and I don't understand why a per-backend area and global area, with some kind of cicular buffer for per-table stuff isn't the cheapest, cleanest solution. -- Bruce Momjian| http://candle.pha.pa.us [EMAIL PROTECTED] | (610) 853-3000 + If your life is a hard drive, | 830 Blythe Avenue + Christ can be your backup.| Drexel Hill, Pennsylvania 19026 ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: [HACKERS] new version of contrib-intarray
Seems we have an older version in CVS. I will update it now. I assume /contrib is available for changes up until release, as usual. Mark, we prepared new version of contrib-intarray - index support for 1-D integer arrays using GiST. Changes: - Improved regression test - Current implementation provides index support for one-dimensional array of int4's - gist__int_ops, suitable for small and medium size of arrays, and gist__intbig_ops for indexing large arrays (we use superimposed signature with length of 4096 bits to represent sets, see Sven Helmer,1997). Archive is available from http://www.sai.msu.su/~megera/postgres/gist/code/7.1/contrib-intarray.tar.gz Regards, Oleg _ Oleg Bartunov, sci.researcher, hostmaster of AstroNet, Sternberg Astronomical Institute, Moscow University (Russia) Internet: [EMAIL PROTECTED], http://www.sai.msu.su/~megera/ phone: +007(095)939-16-83, +007(095)939-23-83 -- Bruce Momjian| http://candle.pha.pa.us [EMAIL PROTECTED] | (610) 853-3000 + If your life is a hard drive, | 830 Blythe Avenue + Christ can be your backup.| Drexel Hill, Pennsylvania 19026 ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://www.postgresql.org/search.mpl
Re: [HACKERS] new version of contrib-intarray
Installed in CVS. Thanks. Mark, we prepared new version of contrib-intarray - index support for 1-D integer arrays using GiST. Changes: - Improved regression test - Current implementation provides index support for one-dimensional array of int4's - gist__int_ops, suitable for small and medium size of arrays, and gist__intbig_ops for indexing large arrays (we use superimposed signature with length of 4096 bits to represent sets, see Sven Helmer,1997). Archive is available from http://www.sai.msu.su/~megera/postgres/gist/code/7.1/contrib-intarray.tar.gz Regards, Oleg _ Oleg Bartunov, sci.researcher, hostmaster of AstroNet, Sternberg Astronomical Institute, Moscow University (Russia) Internet: [EMAIL PROTECTED], http://www.sai.msu.su/~megera/ phone: +007(095)939-16-83, +007(095)939-23-83 -- Bruce Momjian| http://candle.pha.pa.us [EMAIL PROTECTED] | (610) 853-3000 + If your life is a hard drive, | 830 Blythe Avenue + Christ can be your backup.| Drexel Hill, Pennsylvania 19026 ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: [HACKERS] new version of contrib-intarray
Bruce Momjian writes: Installed in CVS. Thanks. You overwrote the changes that other people have made meanwhile. Mark, we prepared new version of contrib-intarray - index support for 1-D integer arrays using GiST. Changes: - Improved regression test - Current implementation provides index support for one-dimensional array of int4's - gist__int_ops, suitable for small and medium size of arrays, and gist__intbig_ops for indexing large arrays (we use superimposed signature with length of 4096 bits to represent sets, see Sven Helmer,1997). Archive is available from http://www.sai.msu.su/~megera/postgres/gist/code/7.1/contrib-intarray.tar.gz Regards, Oleg _ Oleg Bartunov, sci.researcher, hostmaster of AstroNet, Sternberg Astronomical Institute, Moscow University (Russia) Internet: [EMAIL PROTECTED], http://www.sai.msu.su/~megera/ phone: +007(095)939-16-83, +007(095)939-23-83 -- Peter Eisentraut [EMAIL PROTECTED] http://yi.org/peter-e/ ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/users-lounge/docs/faq.html
Re: [HACKERS] new version of contrib-intarray
I checked README.intarray to see what the most recent date was, and it
was Jan 10, so I knew that this version was newer. I then did a diff -c
against the current CVS and I didn't see anything unusual in the
changes.
Attached is the CVS diff command line showing me all the changes made:
cvs diff -c -D '2001-01-13 00:00:00' -D'2001-03-16 00:00:00' .
I see change of += in CFLAGS (harmless), movement of #include
postgres.h, and removal of // comments, which don't appear anymore in
the code.
Do you see anything else? This one was easy to track because it was
installed only recently, but other /contrib stuff is much tougher
because you never know what the original install date was.
I usually only look at Makefile changes in /contrib because that is
where most of the customization is, and I don't see any changes made to
Makefile by the patch. It doesn't even touch the CFLAGS += change.
--
Bruce Momjian| http://candle.pha.pa.us
[EMAIL PROTECTED] | (610) 853-3000
+ If your life is a hard drive, | 830 Blythe Avenue
+ Christ can be your backup.| Drexel Hill, Pennsylvania 19026
? Makefile.703
Index: Makefile
===
RCS file: /home/projects/pgsql/cvsroot/pgsql/contrib/intarray/Makefile,v
retrieving revision 1.2
retrieving revision 1.3
diff -c -r1.2 -r1.3
*** Makefile2001/01/13 02:18:31 1.2
--- Makefile2001/02/20 19:20:27 1.3
***
*** 1,4
! # $Header: /home/projects/pgsql/cvsroot/pgsql/contrib/intarray/Makefile,v 1.2
2001/01/13 02:18:31 petere Exp $
subdir = contrib/intarray
top_builddir = ../..
--- 1,4
! # $Header: /home/projects/pgsql/cvsroot/pgsql/contrib/intarray/Makefile,v 1.3
2001/02/20 19:20:27 petere Exp $
subdir = contrib/intarray
top_builddir = ../..
***
*** 12,18
SO_MAJOR_VERSION= 1
SO_MINOR_VERSION= 0
! override CPPFLAGS += -I$(srcdir) -DPGSQL71
OBJS= _int.o
--- 12,18
SO_MAJOR_VERSION= 1
SO_MINOR_VERSION= 0
! override CPPFLAGS := -I$(srcdir) $(CPPFLAGS) -DPGSQL71
OBJS= _int.o
Index: _int.c
===
RCS file: /home/projects/pgsql/cvsroot/pgsql/contrib/intarray/_int.c,v
retrieving revision 1.1
retrieving revision 1.3
diff -c -r1.1 -r1.3
*** _int.c 2001/01/12 00:16:23 1.1
--- _int.c 2001/02/12 18:30:52 1.3
***
*** 4,14
format for these routines is dictated by Postgres architecture.
**/
! #include stdio.h
#include float.h
#include string.h
- #include "postgres.h"
#include "access/gist.h"
#include "access/itup.h"
#include "access/rtree.h"
--- 4,14
format for these routines is dictated by Postgres architecture.
**/
! #include "postgres.h"
!
#include float.h
#include string.h
#include "access/gist.h"
#include "access/itup.h"
#include "access/rtree.h"
***
*** 194,200
#ifdef GIST_DEBUG
elog(NOTICE, "COMP IN: %d leaf; %d rel; %d page; %d offset; %d bytes; %d elems",
entry-leafkey, (int)entry-rel, (int)entry-page, (int)entry-offset,
(int)entry-bytes, len);
! //printarr( r, len );
#endif
if ( len = 2*MAXNUMRANGE ) { /*compress*/
--- 194,200
#ifdef GIST_DEBUG
elog(NOTICE, "COMP IN: %d leaf; %d rel; %d page; %d offset; %d bytes; %d elems",
entry-leafkey, (int)entry-rel, (int)entry-page, (int)entry-offset,
(int)entry-bytes, len);
! /* printarr( r, len ); */
#endif
if ( len = 2*MAXNUMRANGE ) { /*compress*/
***
*** 260,266
#ifdef GIST_DEBUG
elog(NOTICE, "DECOMP IN: %d leaf; %d rel; %d page; %d offset; %d bytes; %d
elems", entry-leafkey, (int)entry-rel, (int)entry-page, (int)entry-offset,
(int)entry-bytes, lenin);
! //printarr( in, lenin );
#endif
lenr = internal_size(din, lenin);
--- 260,266
#ifdef GIST_DEBUG
elog(NOTICE, "DECOMP IN: %d leaf; %d rel; %d page; %d offset; %d bytes; %d
elems", entry-leafkey, (int)entry-rel, (int)entry-page, (int)entry-offset,
(int)entry-bytes, lenin);
! /* printarr( in, lenin ); */
#endif
lenr = internal_size(din, lenin);
***
*** 653,659
int i,j;
#ifdef GIST_DEBUG
! //elog(NOTICE, "inner_union %d %d", ARRISNULL( a ) , ARRISNULL( b ) );
#endif
if ( ARRISNULL( a ) ARRISNULL( b ) ) return new_intArrayType(0);
--- 653,659
int i,j;
#ifdef GIST_DEBUG
! /* elog(NOTICE, "inner_union %d %d", ARRISNULL( a ) , ARRISNULL( b ) ); */
#endif
if ( ARRISNULL( a ) ARRISNULL( b ) ) return new_intArrayType(0);
***
*** 709,715
int i,j;
#ifdef GIST_DEBUG
! //elog(NOTICE, "inner_inter %d %d", ARRISNULL( a ),
Re: [HACKERS] beta6 pg_restore core dumps
Thanks, at least the problem I have reported seems gone after I
applied your patch.
--
Tatsuo Ishii
After looking more closely I see that pg_restore has two different
buffer overrun conditions in this one routine. Attached is take two
of my patch.
This would be a lot simpler and cleaner if _PrintData() simply didn't
append a zero byte to the buffer contents. Philip, is it actually
necessary for it to do that?
regards, tom lane
*** pg_backup_custom.c.orig Fri Feb 9 17:32:26 2001
--- pg_backup_custom.cSat Mar 17 12:25:17 2001
***
*** 150,156
if (ctx-zp == NULL)
die_horribly(AH, "%s: unable to allocate zlib stream archive
context",progname);
! ctx-zlibOut = (char*)malloc(zlibOutSize);
ctx-zlibIn = (char*)malloc(zlibInSize);
ctx-inSize = zlibInSize;
ctx-filePos = 0;
--- 150,163
if (ctx-zp == NULL)
die_horribly(AH, "%s: unable to allocate zlib stream archive
context",progname);
! /*
! * zlibOutSize is the buffer size we tell zlib it can output to. We
! * actually allocate one extra byte because some routines want to append
! * a trailing zero byte to the zlib output. The input buffer is expansible
! * and is always of size ctx-inSize; zlibInSize is just the initial
! * default size for it.
! */
! ctx-zlibOut = (char*)malloc(zlibOutSize+1);
ctx-zlibIn = (char*)malloc(zlibInSize);
ctx-inSize = zlibInSize;
ctx-filePos = 0;
***
*** 518,531
blkLen = ReadInt(AH);
while (blkLen != 0) {
! if (blkLen (ctx-inSize - 1)) {
free(ctx-zlibIn);
ctx-zlibIn = NULL;
! ctx-zlibIn = (char*)malloc(blkLen);
if (!ctx-zlibIn)
die_horribly(AH, "%s: failed to allocate decompression
buffer\n", progname);
! ctx-inSize = blkLen;
in = ctx-zlibIn;
}
--- 525,538
blkLen = ReadInt(AH);
while (blkLen != 0) {
! if (blkLen+1 ctx-inSize) {
free(ctx-zlibIn);
ctx-zlibIn = NULL;
! ctx-zlibIn = (char*)malloc(blkLen+1);
if (!ctx-zlibIn)
die_horribly(AH, "%s: failed to allocate decompression
buffer\n", progname);
! ctx-inSize = blkLen+1;
in = ctx-zlibIn;
}
---(end of broadcast)---
TIP 3: if posting/reading through Usenet, please send an appropriate
subscribe-nomail command to [EMAIL PROTECTED] so that your
message can get through to the mailing list cleanly
Re: [HACKERS] beta6 pg_restore core dumps
At 12:31 17/03/01 -0500, Tom Lane wrote: This would be a lot simpler and cleaner if _PrintData() simply didn't append a zero byte to the buffer contents. Philip, is it actually necessary for it to do that? Strictly, I think the answer is that it is not necessary. The output of the uncompress may be a string, which could be passed to one of the str* functions by a downstream call. AFAICT, this is not the case, and the code should work without it, but it's probably safer in the long run to leave it there. If you have strong feelings about removing it, I'll have a closer look at the code, but my guess is that it was just me being paranoid (and stuffing up). Philip Warner| __---_ Albatross Consulting Pty. Ltd. |/ - \ (A.B.N. 75 008 659 498) | /(@) __---_ Tel: (+61) 0500 83 82 81 | _ \ Fax: (+61) 0500 83 82 82 | ___ | Http://www.rhyme.com.au |/ \| |---- PGP key available upon request, | / and from pgp5.ai.mit.edu:11371 |/ ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/users-lounge/docs/faq.html
Re: [HACKERS] beta6 pg_restore core dumps
Philip Warner [EMAIL PROTECTED] writes: At 12:31 17/03/01 -0500, Tom Lane wrote: This would be a lot simpler and cleaner if _PrintData() simply didn't append a zero byte to the buffer contents. Philip, is it actually necessary for it to do that? Strictly, I think the answer is that it is not necessary. The output of the uncompress may be a string, which could be passed to one of the str* functions by a downstream call. AFAICT, this is not the case, and the code should work without it, but it's probably safer in the long run to leave it there. Considering that the data we are working with is binary, and may contain nulls, any code that insisted on null-termination would probably be ipso facto broken. regards, tom lane ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/users-lounge/docs/faq.html
Re: [HACKERS] beta6 pg_restore core dumps
Philip Warner [EMAIL PROTECTED] writes: Considering that the data we are working with is binary, and may contain nulls, any code that insisted on null-termination would probably be ipso facto broken. But we're not; this is the same code that sends the COPY output back to PG. Oh, isn't this the code that pushes large-object bodies around? I should think the problem would've been noticed much sooner if not... regards, tom lane ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] Performance monitor signal handler
Tom Lane wrote: Samuel Sieb [EMAIL PROTECTED] writes: Just as another suggestion, what about sending the data to a different computer, so instead of tying up the database server with processing the statistics, you have another computer that has some free time to do the processing. Some drawbacks are that you can't automatically start/restart it from the postmaster and it will put a little more load on the network, ... and a lot more load on the CPU. Same-machine "network" connections are much cheaper (on most kernels, anyway) than real network connections. I think all of this discussion is vast overkill. No one has yet demonstrated that it's not sufficient to have *one* collector process and a lossy transmission method. Let's try that first, and if it really proves to be unworkable then we can get out the lily-gilding equipment. But there is tons more stuff to do before we have useful stats at all, and I don't think that this aspect is the most critical part of the problem. Well, back to my initial approach with the UDP socket collector. I now have a collector simply reading all messages from the socket. It doesn't do anything useful except for counting their number. Every backend sends a couple of 1K junk messages at the beginning of the main loop. Up to 16 messages, there is no time(1) measurable delay in the execution of the "make runcheck". The dummy collector can keep up during the parallel regression test until the backends send 64 messages each time, at that number he lost 1.25% of the messages. That is an amount of statistics data of 256MB to be collected. Most of the test queries will never generate 1K of message, so that there should be some space here. My plan now is to add some real functionality to the collector and the backend, to see if that has an impact. Jan -- #==# # It's easier to get forgiveness for being wrong than for being right. # # Let's break this rule - forgive me. # #== [EMAIL PROTECTED] # _ Do You Yahoo!? Get your free @yahoo.com address at http://mail.yahoo.com ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://www.postgresql.org/search.mpl
Re: [HACKERS] beta6 pg_restore core dumps
Philip Warner [EMAIL PROTECTED] writes: Oh, isn't this the code that pushes large-object bodies around? I should think the problem would've been noticed much sooner if not... It does both, which is why I was also surprised. Hmm ... digging through the code, it does look like one of the possible destinations is ExecuteSqlCommandBuf, which is a bit schizo about whether it's dealing with a null-terminated string or not, but is likely to get ill if handed one that isn't. Okay, I'll commit what I have then. regards, tom lane ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
Re: [HACKERS] beta6 pg_restore core dumps
Philip Warner [EMAIL PROTECTED] writes: Looking at Tatsuos original message, it looks like the lowest level call was: #0 0x804abd4 in _enableTriggersIfNecessary (AH=0x8057d30, te=0x0, ropt=0x8057c90) at pg_backup_archiver.c:474 which probably has nothing to do with BLOBs. Oh ... I had assumed it was just dying there because of collateral damage from the buffer overrun stomp, but if you see an actual bug there then by all means fix it ;-) regards, tom lane ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
Re: [HACKERS] beta6 pg_restore core dumps
At 21:08 17/03/01 -0500, Tom Lane wrote: Philip Warner [EMAIL PROTECTED] writes: Considering that the data we are working with is binary, and may contain nulls, any code that insisted on null-termination would probably be ipso facto broken. But we're not; this is the same code that sends the COPY output back to PG. Oh, isn't this the code that pushes large-object bodies around? I should think the problem would've been noticed much sooner if not... It does both, which is why I was also surprised. Philip Warner| __---_ Albatross Consulting Pty. Ltd. |/ - \ (A.B.N. 75 008 659 498) | /(@) __---_ Tel: (+61) 0500 83 82 81 | _ \ Fax: (+61) 0500 83 82 82 | ___ | Http://www.rhyme.com.au |/ \| |---- PGP key available upon request, | / and from pgp5.ai.mit.edu:11371 |/ ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] beta6 pg_restore core dumps
Looking at Tatsuos original message, it looks like the lowest level call was: #0 0x804abd4 in _enableTriggersIfNecessary (AH=0x8057d30, te=0x0, ropt=0x8057c90) at pg_backup_archiver.c:474 which probably has nothing to do with BLOBs. I think it's a different problem entirely, caused by a mistake in my recent trigger enable/disable code that only become apparent if BLOBs are being restored. I'll fix it soon... Philip Warner| __---_ Albatross Consulting Pty. Ltd. |/ - \ (A.B.N. 75 008 659 498) | /(@) __---_ Tel: (+61) 0500 83 82 81 | _ \ Fax: (+61) 0500 83 82 82 | ___ | Http://www.rhyme.com.au |/ \| |---- PGP key available upon request, | / and from pgp5.ai.mit.edu:11371 |/ ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send "unregister YourEmailAddressHere" to [EMAIL PROTECTED])
Re: [HACKERS] Stuck spins in current
"Mikheev, Vadim" [EMAIL PROTECTED] writes: And you know - I've run same tests on ~ Mar 9 snapshot without any problems. Oh, I see it: Process A is doing GetSnapShotData. It holds SInvalLock and calls ReadNewTransactionId, which wants XidGenLockId. Process B is doing GetNewTransactionId. It holds XidGenLockId and has run out of XIDs, so it needs to write a NEXTXID log record. Therefore, it calls XLogInsert which wants the insert_lck. Process C is inside XLogInsert on its first xlog entry of a transaction. It holds the insert_lck and wants to put its XID into MyProc-logRec, for which it needs SInvalLock. Ooops. At this point I must humbly say "yes, you told me so", because if I hadn't insisted that we needed NEXTXID records then we wouldn't have this deadlock. It looks to me like the simplest answer is to take NEXTXID records out again. (Fortunately, there doesn't seem to be any comparable cycle involving OidGenLock, or we'd need to think of a better answer.) I shall retire to lick my wounds, and make the changes tomorrow ... regards, tom lane ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/users-lounge/docs/faq.html
Re: [HACKERS] beta6 pg_restore core dumps
At 21:24 17/03/01 -0500, Tom Lane wrote: Philip Warner [EMAIL PROTECTED] writes: Looking at Tatsuos original message, it looks like the lowest level call was: #0 0x804abd4 in _enableTriggersIfNecessary (AH=0x8057d30, te=0x0, ropt=0x8057c90) at pg_backup_archiver.c:474 which probably has nothing to do with BLOBs. Oh ... I had assumed it was just dying there because of collateral damage from the buffer overrun stomp, but if you see an actual bug there then by all means fix it ;-) Fixed. It happened for Tatsuo because of the test case he used. Any real, full, database dump would have worked. It's just data-only ones that failed, and the test case he cited was an implied data-only restore (there were no tables or other metadata). Philip Warner| __---_ Albatross Consulting Pty. Ltd. |/ - \ (A.B.N. 75 008 659 498) | /(@) __---_ Tel: (+61) 0500 83 82 81 | _ \ Fax: (+61) 0500 83 82 82 | ___ | Http://www.rhyme.com.au |/ \| |---- PGP key available upon request, | / and from pgp5.ai.mit.edu:11371 |/ ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: [HACKERS] Stuck spins in current
At this point I must humbly say "yes, you told me so", because if I No, I didn't - I must humbly say that I didn't foresee this deadlock, so "I didn't tell you so" -:) Anyway, deadlock in my tests are very correlated with new log file creation - something probably is still wrong... Vadim ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://www.postgresql.org/search.mpl
Re: [HACKERS] Stuck spins in current
"Vadim Mikheev" [EMAIL PROTECTED] writes: Anyway, deadlock in my tests are very correlated with new log file creation - something probably is still wrong... Well, if you can reproduce it easily, seems like you could get in there and verify or disprove my theory about where the deadlock is. Or send the testbed and I'll try ... regards, tom lane ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/users-lounge/docs/faq.html
