Re: [HACKERS] bgwriter changes
(2) Remove bgwriter_percent. I have yet to hear anyone argue that there's an actual need for bgwriter_percent in tuning bgwriter behavior, One argument for it is to avoid writing very hot pages. (3) Change the meaning of bgwriter_percent, per Simon's proposal. Make it mean the percentage of the buffer pool to scan, at most, to look for dirty buffers. I don't think this is workable, at least not at this a la long I think we want to avoid that checkpoint needs to do a lot of writing, without writing hot pages too often. This can only reasonably be defined with a max number of pages we want to allow dirty at checkpoint time. bgwriter_percent comes close to this meaning, although in this sense the value would need to be high, like 80%. I think we do want 2 settings. Think of one as a short time value (so bgwriter does not write everything in one run) and one a long term target over multiple runs. Is it possible to do a patch that produces a dirty buffer list in LRU order and stops early when eighter maxpages is reached or bgwriter_percent pages are scanned ? Andreas ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
Re: [HACKERS] V8.0rc1 On AIX.
Peter Eisentraut wrote: Bruce Momjian wrote: Huh, isn't this port testing? Do we not want to fix port bugs at this stage? We are not really fixing anything, because it was never expected to work before. We are adding new functionality. It is, of course, a borderline case. But for example, do we have any information about whether what is being implemented actually works? I don't want to find out in two weeks, that yes, we managed to compile with thread support, but no, threaded applications still have issues on that platform. I assume threads are supposed to work on all platforms that support it. We have new threading detection code in 8.0 so I expected some adjustments. Also, someone pointed out that the problem was reported during beta but I missed it. As far as testing we have to have the users do any testing. The thread testing script already tests threading but the actual compile success is what we are adjusting now. -- Bruce Momjian| http://candle.pha.pa.us [EMAIL PROTECTED] | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 8: explain analyze is your friend
Re: [HACKERS] possible wierd boolean bug?
I confirmed the problem on a linux server running beta3...so this problem is quite reproducible by running the attached scripts on a freshly loaded database. To reproduce the problem [adjust host,etc as necessary]: 1. type/cat test_boolean.sql | psql template1 (this will create a database called 'test', connect to it, and load a few things.) 2. bzip -cd poline.bzip | psql test (this will load a table into test that was dumped via pg_dump) 3. try the following query: select 1::int4, * from data1.po_line_file wherepol_po_no = '0002' and (pol_po_no = '0002' and pol_po_rel_no = 0) and (pol_po_no = '0002' and pol_po_rel_no = 0 and pol_item_no = '1570'); it should return 0 rows. Try it with explain/analyze which reports 4 rows. Try it a third time as: select 1::int4, * from data1.po_line_file where -- pol_po_no = '0002' and -- (pol_po_no = '0002' and pol_po_rel_no = 0) and (pol_po_no = '0002' and pol_po_rel_no = 0 and pol_item_no = '1570'); which is logically equivalent to the first form (isn't it?) and this returns 1 row (the correct answer). Merlin test_boolean.sql Description: test_boolean.sql poline.bzip Description: poline.bzip ---(end of broadcast)--- TIP 7: don't forget to increase your free space map settings
Re: [HACKERS] cant write to file within call handler interface
On Tue, Dec 14, 2004 at 10:31:41AM -0800, Sibtay Abbas wrote: i am not able to write to file until the pl call handler interface. What are you expecting to happen and what actually does happen? Saying only it doesn't work doesn't give us much to go on. this is the template which i am following PG_FUNCTION_INFO_V1(my_call_handler); Datum my_call_handler(PG_FUNCTION_ARGS) { ...my code... int fd = open(filename,O_WRONLY); write(fd,buffer,strlen(buffer) + 1); //fsync(fd)...i tried this as well but did'nt work } A simple but complete example would be helpful so we can see everything you're doing; the parts of the code that you don't show could be relevant to the problem. Reducing the problem to the smallest possible example can also help you find where the mistake is by eliminating where it isn't. You show no error checking. Check the return value of all functions that can fail and use ereport() or elog() to report any failures. You also don't show where buffer comes from, so we don't know if its contents are a problem or not. And is filename the actual name in your code? Where are you expecting that file to be? Some programming mistakes can be spotted by the compiler. Turn on as many compiler warnings as you can. -- Michael Fuhr http://www.fuhr.org/~mfuhr/ ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] dropdb/contrib-regression
Andrew Dunstan [EMAIL PROTECTED] writes: Tom Lane said: At the moment I think a sleep in the contrib makefile is sufficient (though note I intended to apply it only to installcheck not the other actions ;-)) If you're going to make a separate rule for installcheck (which I agree is a good idea), please also consider making it fault tolerant as I suggested here:http://archives.postgresql.org/pgsql-patches/2004-09/msg00337.php Done. regards, tom lane ---(end of broadcast)--- TIP 9: the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [HACKERS] V8.0rc1 On AIX.
On Dec 14, 2004, at 12:51 PM, Bruce Momjian wrote: Peter Eisentraut wrote: Bruce Momjian wrote: Huh, isn't this port testing? Do we not want to fix port bugs at this stage? We are not really fixing anything, because it was never expected to work before. We are adding new functionality. It is, of course, a borderline case. But for example, do we have any information about whether what is being implemented actually works? I don't want to find out in two weeks, that yes, we managed to compile with thread support, but no, threaded applications still have issues on that platform. I assume threads are supposed to work on all platforms that support it. We have new threading detection code in 8.0 so I expected some adjustments. Also, someone pointed out that the problem was reported during beta but I missed it. As far as testing we have to have the users do any testing. The thread testing script already tests threading but the actual compile success is what we are adjusting now. And that failure was, as I understand it, to accommodate using /usr/bin/cc because as I mentioned in my port report on an AIX 5.1 system, /usr/bin/cc_r already works as-is. I believe that cc_r just adds -DTHREAD_SAFETY, adds -lpthead, sets a LIBPATH such that /usr/lib/threads preceeds /usr/lib/ so you pick up a thread-safe libc.a (which you don't even necessarily need unless you are assuming some libc functions are thread-safe -- if you are doing higher-level sync yourself, then that might not even be necessary), and cc_r does whatever other default defines, etc are best for a safe multithreaded compilation. As I recall, it used to just be a shell-script wrapper in AIX 4 days. Now, on my AIX 5.1 system, /usr/bin/ cc and cc_r are both just symlinks back to /usr/vac/bin/xlc. The different program names just invoke the different settings. Brad: can you make such a symlink and compile with that?, for example: /home/u/brad/cc_r - /usr/bin/xlc ./configure CC=/home/u/brad/cc_r Certainly, pthreads do generally work on AIX. I've been using them for years. However, I always use cc_r (or xlC_r) for multithreaded apps. Peter has a point that the patch makes it compile, but does it make all necessary adjustments such that everything will always work? I can't answer that. -Travis ---(end of broadcast)--- TIP 9: the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [HACKERS] V8.0rc1 On AIX.
That's a good point. Should we just use cc_r for all thread compiles on AIX if they are using cc and not gcc? I am not ready to have the backend compiled with cc_r at this stage so I think this level of support has to wait until 8.1. --- Travis P wrote: On Dec 14, 2004, at 12:51 PM, Bruce Momjian wrote: Peter Eisentraut wrote: Bruce Momjian wrote: Huh, isn't this port testing? Do we not want to fix port bugs at this stage? We are not really fixing anything, because it was never expected to work before. We are adding new functionality. It is, of course, a borderline case. But for example, do we have any information about whether what is being implemented actually works? I don't want to find out in two weeks, that yes, we managed to compile with thread support, but no, threaded applications still have issues on that platform. I assume threads are supposed to work on all platforms that support it. We have new threading detection code in 8.0 so I expected some adjustments. Also, someone pointed out that the problem was reported during beta but I missed it. As far as testing we have to have the users do any testing. The thread testing script already tests threading but the actual compile success is what we are adjusting now. And that failure was, as I understand it, to accommodate using /usr/bin/cc because as I mentioned in my port report on an AIX 5.1 system, /usr/bin/cc_r already works as-is. I believe that cc_r just adds -DTHREAD_SAFETY, adds -lpthead, sets a LIBPATH such that /usr/lib/threads preceeds /usr/lib/ so you pick up a thread-safe libc.a (which you don't even necessarily need unless you are assuming some libc functions are thread-safe -- if you are doing higher-level sync yourself, then that might not even be necessary), and cc_r does whatever other default defines, etc are best for a safe multithreaded compilation. As I recall, it used to just be a shell-script wrapper in AIX 4 days. Now, on my AIX 5.1 system, /usr/bin/ cc and cc_r are both just symlinks back to /usr/vac/bin/xlc. The different program names just invoke the different settings. Brad: can you make such a symlink and compile with that?, for example: /home/u/brad/cc_r - /usr/bin/xlc ./configure CC=/home/u/brad/cc_r Certainly, pthreads do generally work on AIX. I've been using them for years. However, I always use cc_r (or xlC_r) for multithreaded apps. Peter has a point that the patch makes it compile, but does it make all necessary adjustments such that everything will always work? I can't answer that. -Travis ---(end of broadcast)--- TIP 9: the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match -- Bruce Momjian| http://candle.pha.pa.us [EMAIL PROTECTED] | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faqs/FAQ.html
Re: [HACKERS] [Testperf-general] BufferSync and bgwriter
Sorry for the delay; here are results with the bg3.patch with database parameters that should match run 207. I haven't been able to take the time too look over the results myself, but I tried to make sure this run was the same as 207: http://www.osdl.org/projects/dbt2dev/results/dev4-010/207 Mark ---(end of broadcast)--- TIP 9: the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [HACKERS] cant write to file within call handler interface
Sibtay Abbas [EMAIL PROTECTED] writes: i am not able to write to file until the pl call handler interface. this is the template which i am following PG_FUNCTION_INFO_V1(my_call_handler); Datum my_call_handler(PG_FUNCTION_ARGS) { ...my code... int fd = open(filename,O_WRONLY); write(fd,buffer,strlen(buffer) + 1); Perhaps a little bit of checking for error returns would reveal the problem. Other theories are (a) you forgot to close the file so no write occurred; (b) the file was written but not where you think because the filename is relative to the backend's working directory. regards, tom lane ---(end of broadcast)--- TIP 7: don't forget to increase your free space map settings
Re: [HACKERS] [Testperf-general] BufferSync and bgwriter
Sorry, wrong link, right one here: http://www.osdl.org/projects/dbt2dev/results/dev4-010/211 Mark ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] [Testperf-general] BufferSync and bgwriter
On Wed, 2004-12-15 at 00:00, Mark Wong wrote: http://www.osdl.org/projects/dbt2dev/results/dev4-010/211 Thanks Mark for turning that around so quickly. Looks good... Results performed to compare test 207 http://www.osdl.org/projects/dbt2dev/results/dev4-010/207 test 211 with bg3.patch which matches Neil/my option (3) http://www.osdl.org/projects/dbt2dev/results/dev4-010/211 The overall results show 3% throughput gain. The negative effects of checkpointing are significantly reduced and this shows up in the New Order Transaction response time max dropping from 37s to 25s, which looks like a significant user-visible performance gain. Similar reduction in max response times is shown for all transaction types: consistent removal of the longest wait times. The gains come from greater effectiveness of the bgwriter, which reduces I/O wait time spikes to almost zero once the shared_buffers are completely full. (see Processor Utilization graph: wait) It looks to me that reducing the bgwriter_delay slightly might yield additional gains, say to 180 or 160. That should now be possible since the cost of doing so has been greatly reduced. StrategyDirtyBufferList has now dropped way down the list in oprofile results. Neil very kindly points out privately that the patch has a missing sanity check bug in it, which has shown up in Neil's testing. That wouldn't effect these performance results, however. I leave it to Neil to post a corrected version as a result of his efforts. I leave it to the consensus to decide whether these results represent significant gains and whether to add to 8.0, or defer. Neil's suggestion (2) should also needs to be considered - test results could still show that as the better option, so I keep an open mind. -- Best Regards, Simon Riggs ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
[HACKERS] production server down
I've got a down production server (will not restart) with the following tail to its log file: 2004-12-13 15:05:52 LOG: recycled transaction log file 0165004C 2004-12-13 15:26:01 LOG: recycled transaction log file 0165004D 2004-12-13 16:39:55 LOG: database system was shut down at 2004-11-02 17:05:33 PST 2004-12-13 16:39:55 LOG: checkpoint record is at 0/9B0B8C 2004-12-13 16:39:55 LOG: redo record is at 0/9B0B8C; undo record is at 0/0; shutdown TRUE 2004-12-13 16:39:55 LOG: next transaction ID: 536; next OID: 17142 2004-12-13 16:39:55 LOG: database system is ready 2004-12-14 15:36:20 FATAL: IDENT authentication failed for user colprod 2004-12-14 15:36:58 FATAL: IDENT authentication failed for user colprod 2004-12-14 15:39:26 LOG: received smart shutdown request 2004-12-14 15:39:26 LOG: shutting down 2004-12-14 15:39:28 PANIC: could not open file /replica/pgdata/pg_xlog/ (log file 0, segment 0): No such file or directory 2004-12-14 15:39:28 LOG: shutdown process (PID 23202) was terminated by signal 6 2004-12-14 15:39:39 LOG: database system shutdown was interrupted at 2004-12-14 15:39:26 PST 2004-12-14 15:39:39 LOG: could not open file /replica/pgdata/pg_xlog/ (log file 0, segment 0): No such file or directory 2004-12-14 15:39:39 LOG: invalid primary checkpoint record 2004-12-14 15:39:39 LOG: could not open file /replica/pgdata/pg_xlog/ (log file 0, segment 0): No such file or directory 2004-12-14 15:39:39 LOG: invalid secondary checkpoint record 2004-12-14 15:39:39 PANIC: could not locate a valid checkpoint record 2004-12-14 15:39:39 LOG: startup process (PID 23298) was terminated by signal 6 2004-12-14 15:39:39 LOG: aborting startup due to startup process failure This is a SuSE 9, 8-way Xeon IBM x445, with nfs mounted Network Appliance for database storage, postgresql-7.4.5-36.4. The server experienced a hang (as yet unexplained) yesterday and was restarted at 2004-12-13 16:38:49 according to syslog. I'm told by the network admin that there was a problem with the network card on restart, so the nfs mount most probably disappeared and then reappeared underneath a quiescent postgresql at some point between 2004-12-13 16:39:55 and 2004-12-14 15:36:20 (but much closer to the former than the latter). Any help would be much appreciated. Is our only option pg_resetxlog? Thanks, Joe ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send unregister YourEmailAddressHere to [EMAIL PROTECTED])
Re: [HACKERS] production server down
Joe Conway wrote: This is a SuSE 9, 8-way Xeon IBM x445, with nfs mounted Network Appliance for database storage, postgresql-7.4.5-36.4. The server experienced a hang (as yet unexplained) yesterday and was restarted at 2004-12-13 16:38:49 according to syslog. I'm told by the network admin that there was a problem with the network card on restart, so the nfs mount most probably disappeared and then reappeared underneath a quiescent postgresql at some point between 2004-12-13 16:39:55 and 2004-12-14 15:36:20 (but much closer to the former than the latter). Well, my first reaction is that if the file system storage was not always 100% reliable, then there is no way to know the data is correct except by restoring from backup. The startup failure indicates that there were surely storage problems in the past. There is no way to know how far that corrupt goes. You can use pg_resetxlog to clear it out and look to see how accurate it is, but there is no way to be sure. I would back up the file system with the server down in case you want to do some more serious recovery attempts later though. The Freenode IRC channel can probably walk you through more details of the recovery process. -- Bruce Momjian| http://candle.pha.pa.us [EMAIL PROTECTED] | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
[HACKERS] libpq *.def files built for non-Win32
Why is non-Win32 building the *.def files? Seems it should be only Win32. I realize distprep has to build them of course. -- Bruce Momjian| http://candle.pha.pa.us [EMAIL PROTECTED] | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 8: explain analyze is your friend
Re: [HACKERS] production server down
Tom Lane wrote: Joe Conway [EMAIL PROTECTED] writes: I've got a down production server (will not restart) with the following tail to its log file: Please show the output of pg_controldata, or a hex dump of pg_control if pg_controldata fails. OK, here it is: # pg_controldata /replica/pgdata pg_control version number:72 Catalog version number: 200310211 Database cluster state: shutting down pg_control last modified: Tue Dec 14 15:39:26 2004 Current log file ID: 0 Next log file segment:1 Latest checkpoint location: 0/9B0B8C Prior checkpoint location:0/9AA1B4 Latest checkpoint's REDO location:0/9B0B8C Latest checkpoint's UNDO location:0/0 Latest checkpoint's StartUpID:12 Latest checkpoint's NextXID: 536 Latest checkpoint's NextOID: 17142 Time of latest checkpoint:Tue Nov 2 17:05:32 2004 Database block size: 8192 Blocks per segment of large relation: 131072 Maximum length of identifiers:64 Maximum number of function arguments: 32 Date/time type storage: 64-bit integers Maximum length of locale name:128 LC_COLLATE: C LC_CTYPE: C Joe ---(end of broadcast)--- TIP 8: explain analyze is your friend
[HACKERS] dropdb - still occasional failures
We still seem to have occasional problems with dropdb while running contrib installcheck. The symptoms look like this: == dropping database regression == dropdb: database removal failed: ERROR: database regression is being accessed by other users == creating database regression == createdb: database creation failed: ERROR: database regression already exists pg_regress: createdb failed Example: (FBSD again) http://www.pgbuildfarm.org/cgi-bin/show_log.pl?nm=cockatoodt=2004-12-13%2008:10:01 I guess we could put a small sleep before dropdb in pg_regress.sh to make sure a backend from a previous client had enough time to clean up after itself, but I am wondering if this is a symptom of a larger problem? cheers andrew ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faqs/FAQ.html
Re: [HACKERS] [postgis-devel] RE: join selectivity
On Mon, Dec 13, 2004 at 03:04:01PM -, Mark Cave-Ayland wrote: Hi strk, -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] Sent: 13 December 2004 14:05 To: Mark Cave-Ayland Cc: [EMAIL PROTECTED] Subject: Re: [postgis-devel] RE: join selectivity On Mon, Dec 13, 2004 at 12:16:15PM -, Mark Cave-Ayland wrote: Hi strk, (cut) Taking a look at join selectivity... For a query like this: SELECT id FROM table1, table2 WHERE table1.geom table2.geom; RESTRICT selectivity is invoked twice and JOIN selectivity is invoked once. The RESTRICT code is not able to find a costant part and thus returns the default value (0.05), JOIN selectivity so far returns an hard-wired 0.1. Questions: (1) What should RESTRICT selectivity do in this case ?! Maybe that's how the planner decide what to do: 1) sequencially scan table1 and use index for each row (RESTRICT) 2) sequencially scan table2 and use index for each row (RESTRICT) 3) ... some other magic I'm missing .. (JOIN) Indeed, you could be on the right lines here in thinking the planner considers some form of individual scan on each first before finalising on a plan type (although unless the tables are small I would have thought this would not have been an option). Does this change if you do a SET ENABLE_SEQSCAN = 'f' before the query? Bingo. Both ENABLE_SEQSCAN = 'f' or unavailability of an index make the selectivity estimator calls go away. The join selectivity is called nonetheless (also in absence of indexes). Right. So what you're saying is that if there is *no* GiST index on *one* of the geom columns, or sequential scans are disabled, then the calls to RESTRICT go away? Index on a single table makes 2 calls to RESTRICT: strk=# select * from test1, test2 where test1.geom test2.geom; NOTICE: LWGEOM_gist_joinsel called (returning 0.05) NOTICE: LWGEOM_gist_sel called NOTICE: no constant arguments - returning default selectivity NOTICE: LWGEOM_gist_sel called NOTICE: no constant arguments - returning default selectivity Index on a both table makes 4 calls to RESTRICT: strk=# select * from test1, test2 where test1.geom test2.geom; NOTICE: LWGEOM_gist_joinsel called (returning 0.05) NOTICE: LWGEOM_gist_sel called NOTICE: no constant arguments - returning default selectivity NOTICE: LWGEOM_gist_sel called NOTICE: no constant arguments - returning default selectivity NOTICE: LWGEOM_gist_sel called NOTICE: no constant arguments - returning default selectivity NOTICE: LWGEOM_gist_sel called NOTICE: no constant arguments - returning default selectivity No index makes no calls to RESTRICT: strk=# select * from test1, test2 where test1.geom test2.geom; NOTICE: LWGEOM_gist_joinsel called (returning 0.05) It just seems strange for a column operator column clause to call a function involving a constant. Again, I'd probably ask on pgsql-hackers just to clarify - I think Tom Lane was involved with the planner, so will be able to answer this one fairly quickly. CCd to pgsql-hackers for clarification. BTW which version are you developing against - 7.4 or 8.0? 8.0.0RC1 --strk; Kind regards, Mark. WebBased Ltd South West Technology Centre Tamar Science Park Plymouth PL6 8BT T: +44 (0)1752 791021 F: +44 (0)1752 791023 W: http://www.webbased.co.uk ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
Re: [HACKERS] production server down
Tom Lane wrote: ... pg_control last modified: Tue Dec 14 15:39:26 2004 ... Time of latest checkpoint:Tue Nov 2 17:05:32 2004 [ blink... ] That seems like an unreasonable gap between checkpoints, especially for a production server. Can you see an explanation? Hmmm, this is even more scary. We have two database clusters on this server, one on /replica/pgdata, and one on /production/pgdata (ignore the names -- /replica is actually the production instance at the moment). # pg_controldata /replica/pgdata pg_control version number:72 Catalog version number: 200310211 Database cluster state: shutting down pg_control last modified: Tue Dec 14 15:39:26 2004 Current log file ID: 0 Next log file segment:1 Latest checkpoint location: 0/9B0B8C Prior checkpoint location:0/9AA1B4 Latest checkpoint's REDO location:0/9B0B8C Latest checkpoint's UNDO location:0/0 Latest checkpoint's StartUpID:12 Latest checkpoint's NextXID: 536 Latest checkpoint's NextOID: 17142 Time of latest checkpoint:Tue Nov 2 17:05:32 2004 Database block size: 8192 Blocks per segment of large relation: 131072 Maximum length of identifiers:64 Maximum number of function arguments: 32 Date/time type storage: 64-bit integers Maximum length of locale name:128 LC_COLLATE: C LC_CTYPE: C # pg_controldata /production/pgdata pg_control version number:72 Catalog version number: 200310211 Database cluster state: shutting down pg_control last modified: Tue Nov 2 21:57:49 2004 Current log file ID: 0 Next log file segment:1 Latest checkpoint location: 0/9B0B8C Prior checkpoint location:0/9AA1B4 Latest checkpoint's REDO location:0/9B0B8C Latest checkpoint's UNDO location:0/0 Latest checkpoint's StartUpID:12 Latest checkpoint's NextXID: 536 Latest checkpoint's NextOID: 17142 Time of latest checkpoint:Tue Nov 2 17:05:32 2004 Database block size: 8192 Blocks per segment of large relation: 131072 Maximum length of identifiers:64 Maximum number of function arguments: 32 Date/time type storage: 64-bit integers Maximum length of locale name:128 LC_COLLATE: C LC_CTYPE: C I have no idea how this happened, but those look too similar except for the last modified date. The space used is quite what I'd expect: # du -h --max-depth=1 /replica 403G/replica/pgdata # du -h --max-depth=1 /production 201G/production/pgdata The /production/pgdata cluster has not been in use since Nov 2. But we've been loading data aggressively into /replica/pgdata. Any theories on how we screwed up? Joe ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
Re: [HACKERS] production server down
Joe Conway [EMAIL PROTECTED] writes: Any theories on how we screwed up? I hesitate to suggest this, but maybe a cron job blindly copying data from point A to point B? I'm not sure that that could entirely explain the facts. My recollection of the xlog.c logic is that the pg_control file is read into shared memory during postmaster boot, and after that it's write-only: at checkpoint times we update the file image in shared memory and then write it out to pg_control. Offhand my bets would revolve around (a) multiple postmasters trying to run the same PGDATA directory (we have interlocks to protect against this, but I have no faith that they work against an NFS-mounted data directory), or (b) you somehow wiped a PGDATA directory and restored it from backup tapes underneath a running postmaster. regards, tom lane ---(end of broadcast)--- TIP 7: don't forget to increase your free space map settings
Re: [HACKERS] production server down
Tom Lane wrote: Joe Conway [EMAIL PROTECTED] writes: Any theories on how we screwed up? I hesitate to suggest this, but maybe a cron job blindly copying data from point A to point B? Not likely, but I'll check. Offhand my bets would revolve around (a) multiple postmasters trying to run the same PGDATA directory (we have interlocks to protect against this, but I have no faith that they work against an NFS-mounted data directory) This might be possible I suppose. I know we have two init scripts. Perhaps there is an error in them that caused both postmasters to point to the same place when the server was rebooted. I'll look them over. or (b) you somehow wiped a PGDATA directory and restored it from backup tapes underneath a running postmaster. This seems highly unlikely because our *nix admin would have had to deliberately do it, and I don't think he'd fail to tell me about something like that. But all the same, I'll ask him tomorrow. Assuming the only real problem here is the control data (long shot, I know), and the actual database files and transaction logs are OK, is there any reasonable way to reconstruct the correct contol data? Or is that the point at which you use pg_resetxlog? Joe ---(end of broadcast)--- TIP 9: the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [HACKERS] production server down
Tom Lane wrote: My advice is to backup the $PGDATA tree (which you said was in progress), then pg_resetxlog, then cross-check the hell out of the data you see. Only if you can detect some data problems can we guess at something else to do ... OK. I plan to gather the usual suspects and try to get an accurate picture of the chain of events first thing tomorrow. Then we'll likely proceed as you suggest. Thinking about your comments and reading xlog.c, it almost seems as though the mount points were momentarily reversed between /replica and /production. I.e. that the /production mount point was used near the beginning of StartupXLOG() for ReadControlFile(), and the /replica mount point was used at the end of StartupXLOG() for UpdateControlFile(). But I have no idea how that could happen. Thanks, Joe ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faqs/FAQ.html
Re: [HACKERS] production server down
Joe Conway [EMAIL PROTECTED] writes: Assuming the only real problem here is the control data (long shot, I know), and the actual database files and transaction logs are OK, is there any reasonable way to reconstruct the correct contol data? Or is that the point at which you use pg_resetxlog? Well, the problem is that if you can't trust pg_control you don't know what you can trust. My advice is to backup the $PGDATA tree (which you said was in progress), then pg_resetxlog, then cross-check the hell out of the data you see. Only if you can detect some data problems can we guess at something else to do ... regards, tom lane ---(end of broadcast)--- TIP 7: don't forget to increase your free space map settings
[HACKERS] 800RC1 valgrind-detected bug ?
Hi all. I'm getting error reports from valgrind while debugging postgis. It seems that the error only shows up when I build a GiST index AND vacuum analyze. If I drop the index the error goes away. If I create the index the error still doesn't show. If I vacuum analyze, the error is back, but not always. The query does not involve indexed operators. Any hint about what can couse this ? - versions: valgrind-2.1.2.CVS psql 8.0.0rc1 ==15489== Syscall param write(buf) contains uninitialised or unaddressable byte( s) ==15489==at 0x3C1AF9B8: write (in /lib/libc-2.3.2.so) ==15489==by 0x3C14F67D: (within /lib/libc-2.3.2.so) ==15489==by 0x3C14F615: _IO_do_write (in /lib/libc-2.3.2.so) ==15489==by 0x3C150138: _IO_file_overflow (in /lib/libc-2.3.2.so) ==15489==by 0x3C150D1F: _IO_file_xsputn (in /lib/libc-2.3.2.so) ==15489==by 0x3C145F8E: _IO_fwrite (in /lib/libc-2.3.2.so) ==15489==by 0x82062CE: write_relcache_init_file (in /pgroot-800/bin/postgres ) ==15489==by 0x82130AC: InitPostgres (in /pgroot-800/bin/postgres) ==15489==by 0x819B16A: PostgresMain (in /pgroot-800/bin/postgres) ==15489==by 0x813B2E2: main (in /pgroot-800/bin/postgres) ==15489== Address 0x3C0220F8 is not stack'd, malloc'd or (recently) free'd ==15489== ==15489== Syscall param write(buf) contains uninitialised or unaddressable byte( s) ==15489==at 0x3C1AF9B8: write (in /lib/libc-2.3.2.so) ==15489==by 0x3C14F67D: (within /lib/libc-2.3.2.so) ==15489==by 0x3C14F615: _IO_do_write (in /lib/libc-2.3.2.so) ==15489==by 0x3C14EF67: _IO_file_close_it (in /lib/libc-2.3.2.so) ==15489==by 0x3C144CE9: _IO_fclose (in /lib/libc-2.3.2.so) ==15489==by 0x8189F4E: FreeDesc (in /pgroot-800/bin/postgres) ==15489==by 0x820677E: write_relcache_init_file (in /pgroot-800/bin/postgres ) ==15489==by 0x82130AC: InitPostgres (in /pgroot-800/bin/postgres) ==15489==by 0x819B16A: PostgresMain (in /pgroot-800/bin/postgres) ==15489==by 0x813B2E2: main (in /pgroot-800/bin/postgres) ==15489== Address 0x3C022200 is not stack'd, malloc'd or (recently) free'd TIA --strk; ---(end of broadcast)--- TIP 8: explain analyze is your friend
Re: [HACKERS] 800RC1 valgrind-detected bug ?
strk [EMAIL PROTECTED] writes: ==15489== Syscall param write(buf) contains uninitialised or unaddressable byte(s) Valgrind is fairly useless for debugging postgres, because it doesn't know the difference between alignment-pad bytes in a struct and real data. What you've got here is a gripe arising from writing out a struct containing padding. regards, tom lane ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
Re: [HACKERS] possible wierd boolean bug?
That is bizarre. Does EXPLAIN show any difference? Uh oh. esp=# reindex table data1.parts_order_line_file; REINDEX esp=# explain analyze select 1::int4, * from data1.po_line_file esp-# where pol_po_no = '0002' and esp-# (pol_po_no = '0002' and pol_po_rel_no = 0) and esp-# (pol_po_no = '0002' and pol_po_rel_no = 0 and pol_item_no = '1570') esp-# limit 1 ; QUER Y PLAN -- Limit (cost=0.00..5.76 rows=1 width=313) (actual time=0.000..0.000 rows=0 loops=1) - Index Scan using po_line_file_pkey on po_line_file (cost=0.00..5.76 rows=1 width=313) (actua l time=0.000..0.000 rows=0 loops=1) Index Cond: ((pol_po_no = '0002'::bpchar) AND (pol_po_no = '0002'::bpchar) AND (pol _po_no = '0002'::bpchar) AND ((pol_po_rel_no)::smallint = 0) AND ((pol_po_rel_no)::smallint = 0) ) Filter: ((pol_item_no)::text = '1570'::text) Total runtime: 0.000 ms (5 rows) ---(end of broadcast)--- TIP 7: don't forget to increase your free space map settings
Re: [HACKERS] possible wierd boolean bug?
Bruce wrote: That is bizarre. Does EXPLAIN show any difference? -- esp=# explain analyze select 1::int4, * from data1.po_line_file esp-# where pol_po_no = '0002' and esp-# (pol_po_no = '0002' and pol_po_rel_no = 0) and esp-# (pol_po_no = '0002' and pol_po_rel_no = 0 and pol_item_no = '1570') esp-# limit 1 ; QUER Y PLAN -- Limit (cost=0.00..5.76 rows=1 width=313) (actual time=0.000..0.000 rows=0 loops=1) - Index Scan using po_line_file_pkey on po_line_file (cost=0.00..5.76 rows=1 width=313) (actua l time=0.000..0.000 rows=0 loops=1) Index Cond: ((pol_po_no = '0002'::bpchar) AND (pol_po_no = '0002'::bpchar) AND (pol _po_no = '0002'::bpchar) AND ((pol_po_rel_no)::smallint = 0) AND ((pol_po_rel_no)::smallint = 0) ) Filter: ((pol_item_no)::text = '1570'::text) Total runtime: 0.000 ms (5 rows) esp=# explain analyze select 1::int4, * from data1.po_line_file esp-# where -- pol_po_no = '0002' and esp-# -- (pol_po_no = '0002' and pol_po_rel_no = 0) and esp-# (pol_po_no = '0002' and pol_po_rel_no = 0 and pol_item_no = '1570') esp-# limit 1 ; QUERY PLAN Limit (cost=0.00..5.75 rows=1 width=313) (actual time=0.000..0.000 rows=1 loops=1) - Index Scan using po_line_file_pkey on po_line_file (cost=0.00..5.75 rows=1 width=313) (actua l time=0.000..0.000 rows=1 loops=1) Index Cond: ((pol_po_no = '0002'::bpchar) AND ((pol_po_rel_no)::smallint = 0)) Filter: ((pol_item_no)::text = '1570'::text) Total runtime: 0.000 ms (5 rows) ---(end of broadcast)--- TIP 7: don't forget to increase your free space map settings
[HACKERS] V8.0rc1 On AIX.
AIX 5.1 I applied Bruce's patch, configured with --enable-thread-safety and everything went smoothly. == All 96 tests passed. == Brad. ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: [HACKERS] V8.0rc1 On AIX.
Peter Eisentraut wrote: Am Dienstag, 14. Dezember 2004 16:51 schrieb Brad Nicholson: AIX 5.1 I applied Bruce's patch, configured with --enable-thread-safety and everything went smoothly. Nonetheless, threading support on AIX being a new feature, I don't think this should go into 8.0.0. Huh, isn't this port testing? Do we not want to fix port bugs at this stage? -- Bruce Momjian| http://candle.pha.pa.us [EMAIL PROTECTED] | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] V8.0rc1 On AIX.
Bruce Momjian wrote: Huh, isn't this port testing? Do we not want to fix port bugs at this stage? We are not really fixing anything, because it was never expected to work before. We are adding new functionality. It is, of course, a borderline case. But for example, do we have any information about whether what is being implemented actually works? I don't want to find out in two weeks, that yes, we managed to compile with thread support, but no, threaded applications still have issues on that platform. -- Peter Eisentraut http://developer.postgresql.org/~petere/ ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://archives.postgresql.org
[HACKERS] cant write to file within call handler interface
hello i am not able to write to file until the pl call handler interface. this is the template which i am following PG_FUNCTION_INFO_V1(my_call_handler); Datum my_call_handler(PG_FUNCTION_ARGS) { ...my code... int fd = open(filename,O_WRONLY); write(fd,buffer,strlen(buffer) + 1); //fsync(fd)...i tried this as well but did'nt work } __ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com ---(end of broadcast)--- TIP 8: explain analyze is your friend
Re: [HACKERS] bgwriter changes
Zeugswetter Andreas DAZ SD [EMAIL PROTECTED] writes: Is it possible to do a patch that produces a dirty buffer list in LRU order and stops early when eighter maxpages is reached or bgwriter_percent pages are scanned ? Only if you redefine the meaning of bgwriter_percent. At present it's defined by reference to the total number of dirty pages, and that can't be known without collecting them all. If it were, say, a percentage of the total length of the T1/T2 lists, then we'd have some chance of stopping the scan early. regards, tom lane ---(end of broadcast)--- TIP 8: explain analyze is your friend
[HACKERS] 8.0 release schedule
It's becoming pretty obvious that PG 8.0 won't be released on Dec 15. That was an optimistic target to begin with, and last week's network problems have put it out of reach. After some discussion, the core committee has decided to go week-to-week on this: once an RC release survives for a week with no significant code changes, we'll consider it ready for final release. (As usual, docs changes and translation fixes won't count.) There are several open issues standing between us and RC2, but we hope to put out RC2 within the next few days. Taking these things together, final release isn't going to happen before Christmas, but perhaps soon after. regards, tom lane ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send unregister YourEmailAddressHere to [EMAIL PROTECTED])
Re: [HACKERS] possible wierd boolean bug?
Merlin Moncure [EMAIL PROTECTED] writes: Try it with explain/analyze which reports 4 rows. I don't see four rows. I do see different results when I add the third redundant WHERE clause: it switches to a different index and fails to find the row it should find. I suspect the problem is located in the btree index scan setup code that I changed to support cross-data-type comparisons. (7.4 does not fail, but it doesn't try to use the index fully either.) I'm on it... thanks for the test case! regards, tom lane ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send unregister YourEmailAddressHere to [EMAIL PROTECTED])
Re: [HACKERS] bgwriter changes
On Tue, 2004-12-14 at 19:40, Tom Lane wrote: Zeugswetter Andreas DAZ SD [EMAIL PROTECTED] writes: Is it possible to do a patch that produces a dirty buffer list in LRU order and stops early when eighter maxpages is reached or bgwriter_percent pages are scanned ? Only if you redefine the meaning of bgwriter_percent. At present it's defined by reference to the total number of dirty pages, and that can't be known without collecting them all. If it were, say, a percentage of the total length of the T1/T2 lists, then we'd have some chance of stopping the scan early. ...which was exactly what was proposed for option (3). -- Best Regards, Simon Riggs ---(end of broadcast)--- TIP 8: explain analyze is your friend
Re: [HACKERS] possible wierd boolean bug?
Got it: _bt_preprocess_keys is setting keys_are_unique in cases where it shouldn't. The test at the bottom of that routine used to be correct, but no longer is, because the number of keys returned could be more than the number of attributes being tested when redundant cross-data-type quals are provided (see the comments above the routine). A fix will be forthcoming. regards, tom lane ---(end of broadcast)--- TIP 7: don't forget to increase your free space map settings
Re: [HACKERS] 800RC1 valgrind-detected bug ?
Tom Lane wrote: strk [EMAIL PROTECTED] writes: ==15489== Syscall param write(buf) contains uninitialised or unaddressable byte(s) Valgrind is fairly useless for debugging postgres, because it doesn't know the difference between alignment-pad bytes in a struct and real data. What you've got here is a gripe arising from writing out a struct containing padding. Is there any risk of leaking sensitive data to a file or the network via those uninitialized alignment padding bytes? -O ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: [HACKERS] bgwriter changes
On Tue, 2004-12-14 at 09:23 -0500, Tom Lane wrote: At this point in the release cycle I'm not sure we should be making any significant changes for anything less than a crashing bug. Yes, that's true, and I am definitely hesitant to make changes during RC. That said, adjust bgwriter defaults has been on the open items list for quite some time -- in some sense #2 is just a variant on that idea. I'd want to see some pretty impressive benchmark results before we consider making a change now. http://archives.postgresql.org/pgsql-hackers/2004-12/msg00426.php is with a patch from Simon that implements #3. While that's not exactly the same as #2, it does seem to suggest that the performance difference is rather noticeable. If the problem does indeed exacerbate BufMgrLock contention, it might be more noticeable still on an SMP machine. I'm going to try and get some more benchmark data; if anyone else wants to try the patch and contribute results they are welcome to. -Neil ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faqs/FAQ.html
Re: [HACKERS] production server down
Joe Conway [EMAIL PROTECTED] writes: I've got a down production server (will not restart) with the following tail to its log file: Please show the output of pg_controldata, or a hex dump of pg_control if pg_controldata fails. The server experienced a hang (as yet unexplained) yesterday and was restarted at 2004-12-13 16:38:49 according to syslog. I'm told by the network admin that there was a problem with the network card on restart, so the nfs mount most probably disappeared and then reappeared underneath a quiescent postgresql at some point between 2004-12-13 16:39:55 and 2004-12-14 15:36:20 (but much closer to the former than the latter). I've always felt that running a database across NFS was a Bad Idea ;-) Any help would be much appreciated. Is our only option pg_resetxlog? Possibly, but let's try to dig first. I suppose the DB is too large to save an image aside for forensics later? regards, tom lane ---(end of broadcast)--- TIP 7: don't forget to increase your free space map settings
Re: [HACKERS] production server down
Tom Lane wrote: Joe Conway [EMAIL PROTECTED] writes: I've got a down production server (will not restart) with the following tail to its log file: Please show the output of pg_controldata, or a hex dump of pg_control if pg_controldata fails. OK, will do shortly. The server experienced a hang (as yet unexplained) yesterday and was restarted at 2004-12-13 16:38:49 according to syslog. I'm told by the network admin that there was a problem with the network card on restart, so the nfs mount most probably disappeared and then reappeared underneath a quiescent postgresql at some point between 2004-12-13 16:39:55 and 2004-12-14 15:36:20 (but much closer to the former than the latter). I've always felt that running a database across NFS was a Bad Idea ;-) Yeah, I knew I had that coming :-) Any help would be much appreciated. Is our only option pg_resetxlog? Possibly, but let's try to dig first. I suppose the DB is too large to save an image aside for forensics later? Actually, although the database is about 400 GB, we do have room and are in the process of saving an image now. Joe ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://archives.postgresql.org
[HACKERS] Port report: NetBSD 2.0 mac68k
Hi, Here is a port report for NetBSD 2.0 mac68k, with sources of postgresql8.0.0rc1. Here is the configure line used : ./configure --prefix=/data/postgresql/pgsql-8.0.0rc1 --with-openssl --with-python --with-perl --with-tcl --with-krb5 --with-pam But some tweaking was necessary to make it work: * krb5.h is in /usr/include/krb5 on netbsd (set via CPPFLAGS) * krb5_encrypt is to be found in -lkrb5 -ldes -lasn1 -lroken -lcrypto --enable-thread-safety does not work because the thread safety test fails (src/tools/thread/thread_test) configure:18831: ./conftest conftest in free(): error: freelist is destroyed. [1] Abort trap (core dumped) ./conftest${ac_e... Then the tas code in src/backend/storage/lmgr/s_lock.c cannot be compiled and linked on this system without modification: the '_' in front of the tas symbol should be removes, and '%' added in front of register names. I've attached a diff that makes these modifications only for NetBSD mac68k ELF. With these modifications, make and make install are OK ! template1=# SELECT version(); version - PostgreSQL 8.0.0rc1 on m68k-unknown-netbsdelf2.0, compiled by GCC gcc (GCC) 3.3.3 (NetBSD nb3 20040520) (1 row) in make check, two tests fail: float8 and misc. I've attached the regression.diffs file. Regards, Rémi Zara -- Rémi Zara http://www.remi-zara.net/ regression.diffs Description: Binary data --- src/backend/storage/lmgr/s_lock.c.orig 2004-12-14 20:50:08.0 + +++ src/backend/storage/lmgr/s_lock.c 2004-12-14 20:59:28.0 + @@ -136,6 +136,26 @@ #if defined(__m68k__) +#if defined(__NetBSD__) defined(__ELF__) +static void +tas_dummy() /* really means: extern int tas(slock_t + * **lock); */ +{ + __asm__ __volatile__( + \ +.global tas\n\ +tas: \n\ + movel %sp@(0x4),%a0 \n\ + tas %a0@ \n\ + beq _success\n\ + moveq #-128,%d0\n\ + rts \n\ +_success: \n\ + moveq #0,%d0 \n\ + rts \n\ +); +} +#else static void tas_dummy()/* really means: extern int tas(slock_t * **lock); */ @@ -154,6 +174,7 @@ rts \n\ ); } +#endif /$ __NetBSD__ __ELF__ */ #endif /* __m68k__ */ smime.p7s Description: S/MIME cryptographic signature
Re: [HACKERS] production server down
Joe Conway [EMAIL PROTECTED] writes: Tom Lane wrote: Please show the output of pg_controldata, or a hex dump of pg_control if pg_controldata fails. OK, here it is: ... pg_control last modified: Tue Dec 14 15:39:26 2004 ... Time of latest checkpoint:Tue Nov 2 17:05:32 2004 [ blink... ] That seems like an unreasonable gap between checkpoints, especially for a production server. Can you see an explanation? regards, tom lane ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly