Re: [HACKERS] TOAST
On Fri, Mar 09, 2001 at 01:54:40PM +1100, Pam Withnall wrote: hi, I an using postgresql-7.1beta4 and am trying to use the large text fields. I have heard of TOAST. There is little documentation. I found one section about creating a data type, then creating two functions to convert the data types. Is this how TOAST is implemented? Am I on the right track?. If so, what do the conversion functions look like. I am using plpgsql. Thanks, Pam TOAST works transparently for the user. It just means that the old pre-7.1 8k (actually 32) row length limit is obsolete. In order to use large text fields all you need to do is install 7.1 (which you already have, as you say). No function creation etc. is required. Regards, Frank ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/users-lounge/docs/faq.html
Re: [HACKERS] Do we still need PowerPC-specifictimestamp_is_current/epoch?
Tatsuo Ishii [EMAIL PROTECTED] writes: After further research, I remembered that we used to have "DB_MIN check" in configure back to 6.4.2: I don't know wht it was removed, Hmm. Digging in the CVS logs shows that it was removed by Bruce in configure.in version 1.262, 1999/07/18, with the unedifying log message "configure cleanup". A guess is that he took it out because it wasn't being used anywhere. but I think we'd better to revive the checking and replace #if defined(linux) defined(__powerpc__) with #ifdef HAVE_DBL_MIN_PROBLEM What do you think? I think that is a bad idea, since that code is guaranteed to fail on any machine where the representation of double is at all different from a PPC's. (Even if you are willing to assume that the entire world uses IEEE floats these days, what of endianness?) We could revive the configure test and do #if defined(HAVE_DBL_MIN_PROBLEM) defined(__powerpc__) However, I really wonder whether there is any point. It may be worth noting that the original version of the patch read "#if ... defined(PPC)". It's quite likely that the current test, "... defined(__powerpc__)", doesn't even fire on the old compiler that the patch is intended for. If so, this is dead code and has been since release 6.5. Ok, let's remove the code in datetime.c and see anybody would come up and complain... -- Tatsuo Ishii ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send "unregister YourEmailAddressHere" to [EMAIL PROTECTED])
Re: [HACKERS] Performance monitor
Small question... Will it work in console? Or it will be X only? It will be tck/tk, so I guess X only. That's bad. Cause it will be unuseful for people having databases far away... Like me... :-((( Another point is that it is a little bit strange to have X-Window on machine with database server... At least if it is not for play, but production one... Also there should be a possibility of remote monitoring of the database. But that's just dream... :-))) -- Sincerely Yours, Denis Perchine -- E-Mail: [EMAIL PROTECTED] HomePage: http://www.perchine.com/dyp/ FidoNet: 2:5000/120.5 -- ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
AW: [HACKERS] RE: xlog loose ends, continued
Anyway I like idea of StartUpID in page headers - this will help Can you please describe StartUpID for me ? Ideal would be a stamp that has the last (smallest open) XID, or something else that has more or less timestamp characteristics (without the actual need of wallclock) in regard to the WAL. This could then also be used to scan all pages for modification since last backup, to make incremental backups possible. (note, that incremental backup is not WAL backup) Andreas ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
Re: [HACKERS] RE: xlog checkpoint depends on sync() ... seems unsafe
On Tue, 13 Mar 2001, Tom Lane wrote: I was told the same a long ago about FreeBSD. How much can we count on this undocumented sync() feature? Sounds quite unreliable to me. Unless there's some interlock ... like, say, the second sync not being able to advance past a buffer page that's as yet unwritten by the first sync. But would all Unixen share such a strange detail of implementation? The Linux manpage says: NAME sync - commit buffer cache to disk. [..] DESCRIPTION sync first commits inodes to buffers, and then buffers to disk. [..] CONFORMING TO SVr4, SVID, X/OPEN, BSD 4.3 BUGS According to the standard specification (e.g., SVID), sync() schedules the writes, but may return before the actual writing is done. However, since version 1.3.20 Linux does actually wait. (This still does not guarantee data integrity: modern disks have large caches.) And it's still true. On a fast system, if you do: $ cp /dev/zero /tmp sleep 1; sync the sync will often never finish. (Of course, that's just an implementation detail really.) Matthew. ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
RE: [HACKERS] Performance monitor
I don't want to look a gift horse in the mouth, but it seems to me that the performance monitor should wait until the now-famous query tree redesign which will allow for sets from functions. I realize that the shared memory requirements might be a bit large, but somehow Oracle accomplishes this nicely, with some 50 views (V$ACCESS through V$WAITSTAT) which can be queried, usually via SQL*DBA, for performance statistics. More then 50 performance views may be over-kill, but having the ability to fetch the performance statistics with normal queries sure is nice. Perhaps a postmaster option which would enable/disable the use of accumulating performance statistics in shared memory might ease the hesitation against it? Mike Mascari [EMAIL PROTECTED] -Original Message- From: Denis Perchine [SMTP:[EMAIL PROTECTED]] That's bad. Cause it will be unuseful for people having databases far away... Like me... :-((( Another point is that it is a little bit strange to have X-Window on machine with database server... At least if it is not for play, but production one... Also there should be a possibility of remote monitoring of the database. But that's just dream... :-))) -- Sincerely Yours, Denis Perchine ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
[HACKERS] Re: Performance monitor
Denis Perchine [EMAIL PROTECTED] writes: Small question... Will it work in console? Or it will be X only? It will be tck/tk, so I guess X only. That's bad. Cause it will be unuseful for people having databases far away... Like me... :-((( Another point is that it is a little bit strange to have X-Window on machine with database server... At least if it is not for play, but production one... Well, I use X all the time over far distances - it depends on your connection... And you are not running an X server on the database server, only an X client. But I agree that it would be nice to have monitoring architecture that allowed a multitude of clients... regards, Gunnar ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
Re: [HACKERS] Logical operators don't work
On Tue, Mar 13, 2001 at 07:18:36AM -0500, D'Arcy J.M. Cain wrote: The logical operators '', '|', '' and '' as documented on the page http://www.postgresql.org/devel-corner/docs/postgres/functions.html don't appear to work as advertised. darcy=# SELECT 91 15; ERROR: Unable to identify an operator '' for types 'int4' and 'int4' You will have to retype this query using an explicit cast Should this be fixed or should the documentation be changed? When did you do initdb? If it was more than couple of months ago you may not have them in your system catalogs? marko=# SELECT 4 4; ?column? -- 4 (1 row) marko=# SELECT 4 4; ?column? -- 64 (1 row) -- marko ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] WAL SHM principles
When you mmap, you don't use write() ! mlock actualy locks page in memory and as long as the page is locked the OS doesn't attempt to store the dirty page. It is intended also for security app to ensure that sensitive data are not written to unsecure storage (hdd). It is definition of mlock so that you can be probably sure with it. News to me ... can you please point to such a definition? I see no reference to such semantics in the mlock() manual page in UNIX98 (Single Unix Standard, version 2). mlock() guarantees that the locked address space is in memory. This doesn't imply that updates are not written to the backing file. I would expect an OS that doesn't have a unified buffer cache but tries to keep a consistent view for mmap() and read()/write() to update the file. Regards, Giles ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://www.postgresql.org/search.mpl
[HACKERS] pgtop version 0.1
I have completed version 0.1 of pgtop, a PostgreSQL session monitor. Screenshot attached. I show the currently running query by using gdb to attach to a running backend. The backend must have debug symbols and the 'postgres' binary must be in the current patch. I check for both of these in the program. I implement Cancel using SIGINT and Terminate using SIGTERM. I haven't done SORT yet. You can get it from: ftp://candle.pha.pa.us/pub/postgresql/pgtop.tcl -- Bruce Momjian| http://candle.pha.pa.us [EMAIL PROTECTED] | (610) 853-3000 + If your life is a hard drive, | 830 Blythe Avenue + Christ can be your backup.| Drexel Hill, Pennsylvania 19026 ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/users-lounge/docs/faq.html
[HACKERS] createdb and template0?
Are there plans to make 'createdb' support template0 before 7.1? If so, I'll amend backup.sgml... Philip Warner| __---_ Albatross Consulting Pty. Ltd. |/ - \ (A.B.N. 75 008 659 498) | /(@) __---_ Tel: (+61) 0500 83 82 81 | _ \ Fax: (+61) 0500 83 82 82 | ___ | Http://www.rhyme.com.au |/ \| |---- PGP key available upon request, | / and from pgp5.ai.mit.edu:11371 |/ ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: [HACKERS] createdb and template0?
At 00:32 14/03/01 +1100, Philip Warner wrote: Are there plans to make 'createdb' support template0 before 7.1? If so, I'll amend backup.sgml... Just looked into CVS, and support seems to be there... Philip Warner| __---_ Albatross Consulting Pty. Ltd. |/ - \ (A.B.N. 75 008 659 498) | /(@) __---_ Tel: (+61) 0500 83 82 81 | _ \ Fax: (+61) 0500 83 82 82 | ___ | Http://www.rhyme.com.au |/ \| |---- PGP key available upon request, | / and from pgp5.ai.mit.edu:11371 |/ ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://www.postgresql.org/search.mpl
[HACKERS] Re: Small bug in pg_dump
At 18:41 12/03/01 -0500, Tom Lane wrote: UPDATE pg_class SET reltriggers = (SELECT count(*) FROM pg_trigger where pg_class.oid = tgrelid) WHERE relname = 'Table'; Fixed done... Philip Warner| __---_ Albatross Consulting Pty. Ltd. |/ - \ (A.B.N. 75 008 659 498) | /(@) __---_ Tel: (+61) 0500 83 82 81 | _ \ Fax: (+61) 0500 83 82 82 | ___ | Http://www.rhyme.com.au |/ \| |---- PGP key available upon request, | / and from pgp5.ai.mit.edu:11371 |/ ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: [HACKERS] SIGTERM/FATAL error
Bruce Momjian [EMAIL PROTECTED] writes: Not sure. My admin tool is more proof of concept at this point. I think ultimately we will need to allow administrators to such individual backend terminations. I hope the tool is set up to encourage them to try something safer (ie, CANCEL QUERY) first... Yes, the CANCEL button appears before the TERMINATE button. On SIGTERM, I think we are fooling ourselves if we think people aren't SIGTERM'ing individual backends. Terminating individual db connections is a very common job for an administrator. If SIGTERM doesn't cause proper shutdown for individual backends, I think it should. -- Bruce Momjian| http://candle.pha.pa.us [EMAIL PROTECTED] | (610) 853-3000 + If your life is a hard drive, | 830 Blythe Avenue + Christ can be your backup.| Drexel Hill, Pennsylvania 19026 ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/users-lounge/docs/faq.html
Re: [HACKERS] SIGTERM/FATAL error
Bruce Momjian [EMAIL PROTECTED] writes: Not sure. My admin tool is more proof of concept at this point. I think ultimately we will need to allow administrators to such individual backend terminations. I hope the tool is set up to encourage them to try something safer (ie, CANCEL QUERY) first... regards, tom lane ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] Internationalized error messages
On Fri, Mar 09, 2001 at 03:48:33PM -0500, Tom Lane wrote: Peter Eisentraut [EMAIL PROTECTED] writes: Well, SQL defines these. Do we want to make our own list? However, numeric codes also have the advantage that some hierarchy is possible. E.g., the "22" in "2200G" is actually the category code "data exception". Personally, I would stick to the SQL codes but make some readable macro name for backend internal use. We will probably find cases where we need codes not defined by SQL (since we have non-SQL features). If there is room to invent our own codes then I have no objection to this. I am not sure we can/should use gettext (possible license problems?), Gettext is an open standard, invented at Sun IIRC. There is also an independent implementation for BSDs in the works. On GNU/Linux system it's in the C library. I don't see any license problems that way. Unless that BSD implementation is ready to go, I think we'd be talking about relying on GPL'd (not LGPL'd) code for an essential component of the system functionality. Given RMS' recent antics I am much less comfortable with that than I might once have been. cf. http://citrus.bsdclub.org/ and the libintl in NetBSD, at least NetBSD-current, works. The hard part was eg convincing gmake's configure to use it as there are bits like #if __USE_GNU_GETTEXT rather than just checking for the existence of the functions (as well as the internal symbol _nl_msg_cat_cntr). So yes it's ready to go, but please don't use the same m4 in configure.in as for GNU gettext. Cheers, Patrick ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send "unregister YourEmailAddressHere" to [EMAIL PROTECTED])
[HACKERS] problem with fe/be protocol and large objects
I am trying to debug my socket-level interface to the backend, which implements the 6.4 protocol. It works for general queries, but I have a problem with large objects. lo_create and lo_unlink seem to work OK; I get an oid which looks ok and there is a corresponding xinv??? file in the base/ directory. lo_open returns 0 as a file descriptor. However, following up with one of the other lo functions which take descriptor arguments (such as lo_write or lo_tell) fails with ERROR: lo_tell: invalid large object descriptor (0) Looking at be-fsstubs.c it seems that this arises when cookies[fd] is NULL. I don't know what this might come from: the lo_tell is sent right after the lo_open, on the same connection. Running the sample lo program in C works, so I suppose the problem must come from the bytes I'm sending. Any ideas what could cause this? PostgreSQL 7.0.3 on sparc-sun-solaris2.5.1, compiled by gcc 2.95.2 -- Eric Marsden URL:http://www.laas.fr/~emarsden/ ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://www.postgresql.org/search.mpl
Re: [HACKERS] Performance monitor signal handler
* Philip Warner [EMAIL PROTECTED] [010312 18:56] wrote: At 13:34 12/03/01 -0800, Alfred Perlstein wrote: Is it possible to have a spinlock over it so that an external utility can take a snapshot of it with the spinlock held? I'd suggest that locking the stats area might be a bad idea; there is only one writer for each backend-specific chunk, and it won't matter a hell of a lot if a reader gets inconsistent views (since I assume they will be re-reading every second or so). All the stats area should contain would be a bunch of counters with timestamps, I think, and the cost up writing to it should be kept to an absolute minimum. just some ideas.. Unfortunatley, based on prior discussions, Bruce seems quite opposed to a shared memory solution. Ok, here's another nifty idea. On reciept of the info signal, the backends collaborate to piece together a status file. The status file is given a temporay name. When complete the status file is rename(2)'d over a well known file. This ought to always give a consistant snapshot of the file to whomever opens it. -- -Alfred Perlstein - [[EMAIL PROTECTED]|[EMAIL PROTECTED]] Daemon News Magazine in your snail-mail! http://magazine.daemonnews.org/ ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
Re: [HACKERS] PostgreSQL on multi-CPU systems
Tom Lane [EMAIL PROTECTED] wrote: Tatsuo Ishii [EMAIL PROTECTED] writes: I have tested PostgreSQL with 2-4 CPU linux boxes. In summary, 2 CPU was a big win, but 4 was not. I'm not sure where the bottle neck is though. Our not-very-good implementation of spin locking (using select() to wait) might have something to do with this. Sometime soon I'd like to look at using POSIX semaphores where available, instead of spinlocks. did anyone from here follow the discussion about postgresql on smp machines on the linux kernel malinglist in the last days? (just as an info) t -- [EMAIL PROTECTED] innominate AG the linux architects tel: +49-30-308806-13 fax: -77 http://www.innominate.com ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: [HACKERS] Performance monitor signal handler
This ought to always give a consistant snapshot of the file to whomever opens it. I think Tom has previously stated that there are technical reasons not to do IO in signal handlers, and I have philosophical problems with performance monitors that ask 50 backends to do file IO. I really do think shared memory is TWTG. The good news is that right now pgmonitor gets all its information from 'ps', and only shows the query when the user asks for it. -- Bruce Momjian| http://candle.pha.pa.us [EMAIL PROTECTED] | (610) 853-3000 + If your life is a hard drive, | 830 Blythe Avenue + Christ can be your backup.| Drexel Hill, Pennsylvania 19026 ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/users-lounge/docs/faq.html
Re: [HACKERS] Performance monitor signal handler
This ought to always give a consistant snapshot of the file to whomever opens it. I think Tom has previously stated that there are technical reasons not to do IO in signal handlers, and I have philosophical problems with performance monitors that ask 50 backends to do file IO. I really do think shared memory is TWTG. Philip Warner| __---_ Albatross Consulting Pty. Ltd. |/ - \ (A.B.N. 75 008 659 498) | /(@) __---_ Tel: (+61) 0500 83 82 81 | _ \ Fax: (+61) 0500 83 82 82 | ___ | Http://www.rhyme.com.au |/ \| |---- PGP key available upon request, | / and from pgp5.ai.mit.edu:11371 |/ ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: [HACKERS] Performance monitor signal handler
* Philip Warner [EMAIL PROTECTED] [010313 06:42] wrote: This ought to always give a consistant snapshot of the file to whomever opens it. I think Tom has previously stated that there are technical reasons not to do IO in signal handlers, and I have philosophical problems with performance monitors that ask 50 backends to do file IO. I really do think shared memory is TWTG. I wasn't really suggesting any of those courses of action, all I suggested was using rename(2) to give a seperate appilcation a consistant snapshot of the stats. Actually, what makes the most sense (although it may be a performance killer) is to have the backends update a system table that the external app can query. -- -Alfred Perlstein - [[EMAIL PROTECTED]|[EMAIL PROTECTED]] Daemon News Magazine in your snail-mail! http://magazine.daemonnews.org/ ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: [HACKERS] Re: Performance monitor'
Denis Perchine [EMAIL PROTECTED] writes: Small question... Will it work in console? Or it will be X only? It will be tck/tk, so I guess X only. That's bad. Cause it will be unuseful for people having databases far away... Like me... :-((( Another point is that it is a little bit strange to have X-Window on machine with database server... At least if it is not for play, but production one... Well, I use X all the time over far distances - it depends on your connection... And you are not running an X server on the database server, only an X client. Yes, works fine. But I agree that it would be nice to have monitoring architecture that allowed a multitude of clients... Right now, pgmonitor is just 'ps' with some gdb/kill actions. If I add anything to the backend, you can be sure I will make it so any app can access it. -- Bruce Momjian| http://candle.pha.pa.us [EMAIL PROTECTED] | (610) 853-3000 + If your life is a hard drive, | 830 Blythe Avenue + Christ can be your backup.| Drexel Hill, Pennsylvania 19026 ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/users-lounge/docs/faq.html
Re: [HACKERS] problem with fe/be protocol and large objects
On Monday 12 March 2001 03:24, Eric Marsden wrote: I am trying to debug my socket-level interface to the backend, which implements the 6.4 protocol. It works for general queries, but I have a problem with large objects. lo_create and lo_unlink seem to work OK; I get an oid which looks ok and there is a corresponding xinv??? file in the base/ directory. lo_open returns 0 as a file descriptor. However, following up with one of the other lo functions which take descriptor arguments (such as lo_write or lo_tell) fails with ERROR: lo_tell: invalid large object descriptor (0) You should do ANY operations with LOs in transaction. Looking at be-fsstubs.c it seems that this arises when cookies[fd] is NULL. I don't know what this might come from: the lo_tell is sent right after the lo_open, on the same connection. Running the sample lo program in C works, so I suppose the problem must come from the bytes I'm sending. Any ideas what could cause this? PostgreSQL 7.0.3 on sparc-sun-solaris2.5.1, compiled by gcc 2.95.2 -- Sincerely Yours, Denis Perchine -- E-Mail: [EMAIL PROTECTED] HomePage: http://www.perchine.com/dyp/ FidoNet: 2:5000/120.5 -- ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send "unregister YourEmailAddressHere" to [EMAIL PROTECTED])
[HACKERS] Re: PostgreSQL on multi-CPU systems
did anyone from here follow the discussion about postgresql on smp machines on the linux kernel malinglist in the last days? (just as an info) I didn't. Do you have a synopsis or references? - Thomas ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: [HACKERS] Performance monitor signal handler
At 13:34 12/03/01 -0800, Alfred Perlstein wrote: Is it possible to have a spinlock over it so that an external utility can take a snapshot of it with the spinlock held? I'd suggest that locking the stats area might be a bad idea; there is only one writer for each backend-specific chunk, and it won't matter a hell of a lot if a reader gets inconsistent views (since I assume they will be re-reading every second or so). All the stats area should contain would be a bunch of counters with timestamps, I think, and the cost up writing to it should be kept to an absolute minimum. just some ideas.. Unfortunatley, based on prior discussions, Bruce seems quite opposed to a shared memory solution. No, I like the shared memory idea. Such an idea will have to wait for 7.2, and second, there are limits to how much shared memory I can use. Eventually, I think shared memory will be the way to go. -- Bruce Momjian| http://candle.pha.pa.us [EMAIL PROTECTED] | (610) 853-3000 + If your life is a hard drive, | 830 Blythe Avenue + Christ can be your backup.| Drexel Hill, Pennsylvania 19026 ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] Performance monitor
I don't want to look a gift horse in the mouth, but it seems to me that the performance monitor should wait until the now-famous query tree redesign which will allow for sets from functions. I realize that the shared memory requirements might be a bit large, but somehow Oracle accomplishes this nicely, with some 50 views (V$ACCESS through V$WAITSTAT) which can be queried, usually via SQL*DBA, for performance statistics. More then 50 performance views may be over-kill, but having the ability to fetch the performance statistics with normal queries sure is nice. Perhaps a postmaster option which would enable/disable the use of accumulating performance statistics in shared memory might ease the hesitation against it? I don't think query design is an issue here. We can already create views to do such things. Right now, pgmonitor simply uses 'ps'. and uses gdb to attach to the running process and show the query being executed. For 7.2, I hope to improve it. I like the shared memory ideas, and the ability to use a query rather than accessing shared memory directly. Seems we should have each backend store query/stat information in shared memory, and create special views to access that information. We can restrict such views to the postgres super-user. -- Bruce Momjian| http://candle.pha.pa.us [EMAIL PROTECTED] | (610) 853-3000 + If your life is a hard drive, | 830 Blythe Avenue + Christ can be your backup.| Drexel Hill, Pennsylvania 19026 ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
Re: [HACKERS] Do we still need PowerPC-specific timestamp_is_current/epoch?
Tatsuo Ishii [EMAIL PROTECTED] writes: After further research, I remembered that we used to have "DB_MIN check" in configure back to 6.4.2: I don't know wht it was removed, Hmm. Digging in the CVS logs shows that it was removed by Bruce in configure.in version 1.262, 1999/07/18, with the unedifying log message "configure cleanup". A guess is that he took it out because it wasn't being used anywhere. Yes, I checked all configure flags and removed the ones not being used. -- Bruce Momjian| http://candle.pha.pa.us [EMAIL PROTECTED] | (610) 853-3000 + If your life is a hard drive, | 830 Blythe Avenue + Christ can be your backup.| Drexel Hill, Pennsylvania 19026 ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://www.postgresql.org/search.mpl
Re: [HACKERS] Performance monitor
[ Charset KOI8-R unsupported, converting... ] Small question... Will it work in console? Or it will be X only? It will be tck/tk, so I guess X only. That's bad. Cause it will be unuseful for people having databases far away... Like me... :-((( Another point is that it is a little bit strange to have X-Window on machine with database server... At least if it is not for play, but production one... Also there should be a possibility of remote monitoring of the database. But that's just dream... :-))) What about remote-X using the DISPLAY variable? -- Bruce Momjian| http://candle.pha.pa.us [EMAIL PROTECTED] | (610) 853-3000 + If your life is a hard drive, | 830 Blythe Avenue + Christ can be your backup.| Drexel Hill, Pennsylvania 19026 ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/users-lounge/docs/faq.html
AW: [HACKERS] RE: xlog loose ends, continued
Anyway I like idea of StartUpID in page headers - this will help Can you please describe StartUpID for me ? Ideal would be a stamp that has the last (smallest open) XID, or something else that has more or less timestamp characteristics (without the actual need of wallclock) in regard to the WAL. StartUpID counts database startups and so has timestamp characteristics. Actually, idea is to use SUI in future to allow reusing XIDs after startup: seeing old SUI in data pages we'll know that all transaction on this page was committed "long ago" (ie visible from MVCC POV). This requires UNDO, of course. First thanx for the description, but db startup would only count to 5-7 per year :-), is that sufficient ? It hardly sounds like anything useful to include in page header. What about the xlog id, that is also used for xlog file name, but I still think a xid would be the best candidate. Andreas ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
Re: [HACKERS] RE: xlog loose ends, continued
Anyway I like idea of StartUpID in page headers - this will help Can you please describe StartUpID for me ? Ideal would be a stamp that has the last (smallest open) XID, or something else that has more or less timestamp characteristics (without the actual need of wallclock) in regard to the WAL. StartUpID counts database startups and so has timestamp characteristics. Actually, idea is to use SUI in future to allow reusing XIDs after startup: seeing old SUI in data pages we'll know that all transaction on this page was committed "long ago" (ie visible from MVCC POV). This requires UNDO, of course. This could then also be used to scan all pages for modification since last backup, to make incremental backups possible. (note, that incremental backup is not WAL backup) We can scan log itself to get all pages modified since last backup or whatever point we want - thanks to your idea about data pages backup. Vadim ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/users-lounge/docs/faq.html
Re: [HACKERS] RE: xlog loose ends, continued
StartUpID counts database startups and so has timestamp characteristics. Actually, idea is to use SUI in future to allow reusing XIDs after startup: seeing old SUI in data pages we'll know that all transaction on this page was committed "long ago" (ie visible from MVCC POV). This requires UNDO, of course. First thanx for the description, but db startup would only count to 5-7 per year :-), is that sufficient ? It hardly sounds like anything useful to include in page header. It will be sufficient if DB will not use all 2^32 XIDs without shutdown. Removing pg_log *segments* for old XIDs is another story. What about the xlog id, that is also used for xlog file name, but I still think a xid would be the best candidate. logid would be ok too, xid is not - we have to shorten xids lifetime in near future. Vadim ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] Performance monitor
Hard to say. 'ps' gives some great information about cpu/memory usage that may be hard/costly to put in shared memory. One idea should be to issue periodic 'ps/kill' commands though a telnet/ssh pipe to the remote machine, or just to the remote X display option. Of course, getrusage() gives us much of that information. Forget getrusage(). Only works on current process, so each backend would have to update its own statistics. Sounds expensive. -- Bruce Momjian| http://candle.pha.pa.us [EMAIL PROTECTED] | (610) 853-3000 + If your life is a hard drive, | 830 Blythe Avenue + Christ can be your backup.| Drexel Hill, Pennsylvania 19026 ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/users-lounge/docs/faq.html
AW: [HACKERS] RE: xlog loose ends, continued
StartUpID counts database startups and so has timestamp characteristics. Actually, idea is to use SUI in future to allow reusing XIDs after startup: seeing old SUI in data pages we'll know that all transaction on this page was committed "long ago" (ie visible from MVCC POV). This requires UNDO, of course. First thanx for the description, but db startup would only count to 5-7 per year :-), is that sufficient ? It hardly sounds like anything useful to include in page header. It will be sufficient if DB will not use all 2^32 XIDs without shutdown. I liked the xid wraparound idea, won't that be sufficient here too ? I don't like the idea to reuse a xid sooner than absolutely necessary. This would complicate the search for potentially inconsistent pages after crash. Andreas ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
[HACKERS] Re: xlog loose ends, continued
"Mikheev, Vadim" [EMAIL PROTECTED] writes: That doesn't seem like a good idea --- once we fail to read an XLOG record, it's probably best to stop there rather than continue on. I think we want to try for a consistent recovery to a past point in time (ie, wherever the xlog gap is) not a partial recovery to a later time. No way for consistent recovery if there is gap in log due to disk write re-ordering anyway (and we can't know what was the reason of the gap). I thought that you wanted apply as much of log as we can. If you don't then I missed your point in first message: xlog.c's ReadRecord formerly contained code that would zero out the rest of the log segment (and delete the next log segment, if any) upon detecting a missing or corrupted xlog record. I removed that code because I considered it horribly dangerous where it was. If there is anything wrong with either the xlog or pg_control's pointers to it, that code was quite capable of wiping out all hope of recovery *and* all evidence of what went wrong. What I was thinking about in that last paragraph was manual analysis and recovery. I don't think it's a good idea for automatic system startup to skip over gaps in the log. So, if we are not going to re-apply as much valid records as we can read from log then zeroing is no more dangerous than SUI in headers. But I totaly agreed that SUI is much better. Okay, I will change the page headers to include SUI (or the low-order bits of it anyway), and make ReadRecord stop if it notices a backwards jump in SUI. I don't think a few additional bytes in header is a problem. BTW, why not use CRC32 in header instead of magic? There is so little changeable information in a page header that a CRC wouldn't be much more than an eight-byte magic number. And we don't need eight bytes worth of magic number (even four is more than enough, really). So I'd just as soon keep the headers simple and small. regards, tom lane ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/users-lounge/docs/faq.html
Re: [HACKERS] SIGTERM/FATAL error
Tom is there new wording we can agree on? Bruce Momjian [EMAIL PROTECTED] writes: Not sure. My admin tool is more proof of concept at this point. I think ultimately we will need to allow administrators to such individual backend terminations. I hope the tool is set up to encourage them to try something safer (ie, CANCEL QUERY) first... regards, tom lane ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly -- Bruce Momjian| http://candle.pha.pa.us [EMAIL PROTECTED] | (610) 853-3000 + If your life is a hard drive, | 830 Blythe Avenue + Christ can be your backup.| Drexel Hill, Pennsylvania 19026 ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/users-lounge/docs/faq.html
Re: [HACKERS] WAL SHM principles
Giles Lean [EMAIL PROTECTED] wrote: When you mmap, you don't use write() ! mlock actualy locks page in memory and as long as the page is locked the OS doesn't attempt to store the dirty page. It is intended also for security app to ensure that sensitive data are not written to unsecure storage (hdd). It is definition of mlock so that you can be probably sure with it. News to me ... can you please point to such a definition? I see no reference to such semantics in the mlock() manual page in UNIX98 (Single Unix Standard, version 2). mlock() guarantees that the locked address space is in memory. This doesn't imply that updates are not written to the backing file. I've wondered about this myself. It _is_ true on Linux that mlock prevents writes to the backing store, and this is used as a security feature for cryptography software. The code for gnupg assumes that if you have mlock() on any operating system, it does mean this--which doesn't mean it's true, but perhaps whoever wrote it does have good reason to think so. But I don't know about other systems. Does anybody know what the POSIX.1b standard says? It was even suggested to me on the linux-fsdev mailing list that mlock() was a good way to insure the write-ahead condition. Ken Hirsch ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
[HACKERS] Another little xlog change idea
As long as I'm about to change the xlog page headers, I have another little idea. Wouldn't it be a good idea to allow three backup pages per xlog record, not only two? It seems like three pages would be a natural requirement for logging operations like index page splits. We could support as many as four pages per record, but that would mean having no free global bits in the info byte, which might be a bad idea. I think three page bits and one free bit is a good compromise. Thoughts? regards, tom lane ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
[HACKERS] RE: xlog loose ends, continued
xlog.c's ReadRecord formerly contained code that would zero out the rest of the log segment (and delete the next log segment, if any) upon detecting a missing or corrupted xlog record. I removed that code because I considered it horribly dangerous where it was. If there is anything wrong with either the xlog or pg_control's pointers to it, that code was quite capable of wiping out all hope of recovery *and* all evidence of what went wrong. What I was thinking about in that last paragraph was manual analysis and recovery. I don't think it's a good idea for automatic system startup to skip over gaps in the log. But if we'll not try to read after gap then after restart system will not notice gap and valid records after it and just rewrite log space with new records. Not much chance for manual analysis - ppl will not report any problems. Vadim ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://www.postgresql.org/search.mpl
[HACKERS] RE: Another little xlog change idea
As long as I'm about to change the xlog page headers, I have another little idea. Wouldn't it be a good idea to allow three backup pages per xlog record, not only two? It seems like three pages would be a natural requirement for logging operations like index page splits. On index splits we don't backup left and right whole pages - just part of pages filled by index tuples. This saves us ~ 8K for indices with small keys. We could support as many as four pages per record, but that would mean having no free global bits in the info byte, which might be a bad idea. I think three page bits and one free bit is a good compromise. Thoughts? You almost forget that we are in beta and should fix bugs, not implement features which useless in current version -:) Add comments and that will be enough, no? Vadim ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
[HACKERS] RE: xlog loose ends, continued
What I was thinking about in that last paragraph was manual analysis and recovery. I don't think it's a good idea for automatic system startup to skip over gaps in the log. But if we'll not try to read after gap then after restart system will not notice gap and valid records after it and just rewrite log space with new records. Not much chance for manual analysis - ppl will not report any problems. That'll be true in any case, unless we refuse to start up at all upon detecting xlog corruption (which doesn't seem like the way to fly). Not sure what we can do about that. What I would refuse in the event of log corruption is continuing normal database operations. It's ok to dump such database for manual recovery, but continuing to use it is VERY BAD THING. The fact that users will use inconsistent DB may become big headache for us - just imagine compains about index scans returning incorrect results (index tuples pointing to free heap space was left and then that space was used for tuple with different keys). Failing to restart was bad but silent restart in the event of log corruption is bad too. In first case we had at least chance to discover original problem. Vadim ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
RE: [HACKERS] RE: xlog loose ends, continued
It will be sufficient if DB will not use all 2^32 XIDs without shutdown. I liked the xid wraparound idea, won't that be sufficient here too ? I don't like the idea to reuse a xid sooner than absolutely necessary. We need it to reduce pg_log size requirements. This would complicate the search for potentially inconsistent pages after crash. There is no such search currently and I can't imagine how/when/for what to do such search. ? Vadim ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
[HACKERS] Re: xlog loose ends, continued
"Mikheev, Vadim" [EMAIL PROTECTED] writes: That'll be true in any case, unless we refuse to start up at all upon detecting xlog corruption (which doesn't seem like the way to fly). Not sure what we can do about that. What I would refuse in the event of log corruption is continuing normal database operations. Hmm. We could do that if we had some notion of a read-only operating mode, perhaps. But we don't have one now and I don't want to add it for 7.1. Can we agree to look at this more for 7.2? If we did have that, it would make sense to scan the rest of the log (after the last valid XLOG record) to see if we find any more records. If we do then --- whether they're valid or not --- we have a corrupted DB and we should go into the read-only state. But for the moment I think it's best not to make such a scan. regards, tom lane ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
[HACKERS] RE: xlog loose ends, continued
What I would refuse in the event of log corruption is continuing normal database operations. Hmm. We could do that if we had some notion of a read-only operating mode, perhaps. But we don't have one now and I don't want to add it for 7.1. Can we agree to look at this more for 7.2? We need not in full support of read-only mode - just set some flag in shmem and disallow write ops. I think 7.1.1 or so is good for that. Vadim ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
[HACKERS] Shutdown term
Can somone improve the wording? The system is shutting down. when the backend receives a SIGTERM. Seems we need some wording that can apply to db shutdown and individual backend termination by administrators. -- Bruce Momjian| http://candle.pha.pa.us [EMAIL PROTECTED] | (610) 853-3000 + If your life is a hard drive, | 830 Blythe Avenue + Christ can be your backup.| Drexel Hill, Pennsylvania 19026 ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
Re: [HACKERS] xlog patches reviewed
The Hermit Hacker writes: if this is the case, can we look at applying that patch tonight, give ppl till Friday to test and put out a RC1 depending on the results? This should probably be called beta6, given that there is a lot of new code in it. -- Peter Eisentraut [EMAIL PROTECTED] http://yi.org/peter-e/ ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/users-lounge/docs/faq.html
Re: [HACKERS] Shutdown term
Bruce Momjian writes: Can somone improve the wording? The system is shutting down. when the backend receives a SIGTERM. Seems we need some wording that can apply to db shutdown and individual backend termination by administrators. The connection was terminated. And make the postmaster print out The system is shutting down. before it sends out the SIGTERM's. -- Peter Eisentraut [EMAIL PROTECTED] http://yi.org/peter-e/ ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://www.postgresql.org/search.mpl
Re: [HACKERS] problem with fe/be protocol and large objects
"tl" == Tom Lane [EMAIL PROTECTED] writes: ecm ERROR: lo_tell: invalid large object descriptor (0) tl Are you remembering to wrap this sequence in a transaction block tl (begin/end)? LO descriptors are only valid till end of tl transaction. that was it, thanks. The code used to work with PostgreSQL 6.3, and I hadn't seen the relevant warning in the programmer's guide. -- Eric Marsden URL:http://www.laas.fr/~emarsden/ ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send "unregister YourEmailAddressHere" to [EMAIL PROTECTED])
Re: [HACKERS] WAL SHM principles
Giles Lean [EMAIL PROTECTED] wrote: When you mmap, you don't use write() ! mlock actualy locks page in memory and as long as the page is locked the OS doesn't attempt to store the dirty page. It is intended also for security app to ensure that sensitive data are not written to unsecure storage (hdd). It is definition of mlock so that you can be probably sure with it. News to me ... can you please point to such a definition? I see no reference to such semantics in the mlock() manual page in UNIX98 (Single Unix Standard, version 2). mlock() guarantees that the locked address space is in memory. This doesn't imply that updates are not written to the backing file. I've wondered about this myself. It _is_ true on Linux that mlock prevents writes to the backing store, and this is used as a security feature for cryptography software. The code for gnupg assumes that if you have mlock() on any operating system, it does mean this--which doesn't mean it's true, but perhaps whoever wrote it does have good reason to think so. But I don't know about other systems. Does anybody know what the POSIX.1b standard says? It was even suggested to me on the linux-fsdev mailing list that mlock() was a good way to insure the write-ahead condition. Ken Hirsch ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send "unregister YourEmailAddressHere" to [EMAIL PROTECTED])
Re: [HACKERS] Shutdown term
Peter Eisentraut [EMAIL PROTECTED] writes: The connection was terminated. And make the postmaster print out The system is shutting down. before it sends out the SIGTERM's. Unfortunately the postmaster is in no position to send any message to the individual clients. Maybe we should forget the idea of having a single message to cover both cases, and instead provide some flag in shared memory that the postmaster can set before it sends out SIGTERMs. Then the backends would actually know why they got a SIGTERM and could emit more-appropriate messages. Seems like overkill to me. We could have the postmaster use SIGQUIT for db shutdown and leave SIGKILL for admin shutdown of individual backends. -- Bruce Momjian| http://candle.pha.pa.us [EMAIL PROTECTED] | (610) 853-3000 + If your life is a hard drive, | 830 Blythe Avenue + Christ can be your backup.| Drexel Hill, Pennsylvania 19026 ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: [HACKERS] Shutdown term
Bruce Momjian [EMAIL PROTECTED] writes: Seems like overkill to me. We could have the postmaster use SIGQUIT for db shutdown and leave SIGKILL for admin shutdown of individual backends. Wrong... at least not with the current definitions of those signals! regards, tom lane ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://www.postgresql.org/search.mpl
Re: [HACKERS] WAL SHM principles
On Tue, 13 Mar 2001, Ken Hirsch wrote: mlock() guarantees that the locked address space is in memory. This doesn't imply that updates are not written to the backing file. I've wondered about this myself. It _is_ true on Linux that mlock prevents writes to the backing store, I don't believe that this is true. The manpage offers no such promises, and the semantics are not useful. and this is used as a security feature for cryptography software. mlock() is used to prevent pages being swapped out. Its use for crypto software is essentially restricted to anon memory (allocated via brk() or mmap() of /dev/zero). If my understanding is accurate, before 2.4 Linux would never swap out pages which had a backing store. It would simply write them back or drop them (if clean). (This is why you need around twice as much swap with 2.4.) The code for gnupg assumes that if you have mlock() on any operating system, it does mean this--which doesn't mean it's true, but perhaps whoever wrote it does have good reason to think so. strace on gpg startup says: mmap(0, 16384, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x40015000 getuid()= 500 mlock(0x40015000) = -1 EPERM (Operation not permitted) so whatever the authors think, it does not require this semantic. Matthew. ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://www.postgresql.org/search.mpl
Re: [HACKERS] WAL SHM principles
* Matthew Kirkwood [EMAIL PROTECTED] [010313 13:12] wrote: On Tue, 13 Mar 2001, Ken Hirsch wrote: mlock() guarantees that the locked address space is in memory. This doesn't imply that updates are not written to the backing file. I've wondered about this myself. It _is_ true on Linux that mlock prevents writes to the backing store, I don't believe that this is true. The manpage offers no such promises, and the semantics are not useful. Afaik FreeBSD's Linux emulator: revision 1.13 date: 2001/02/28 04:30:27; author: dillon; state: Exp; lines: +3 -1 Linux does not filesystem-sync file-backed writable mmap pages on a regular basis. Adjust our linux emulation to conform. This will cause more dirty pages to be left for the pagedaemon to deal with, but our new low-memory handling code can deal with it. The linux way appears to be a trend, and we may very well make MAP_NOSYNC the default for FreeBSD as well (once we have reasonable sequential write-behind heuristics for random faults). (will be MFC'd prior to 4.3 freeze) Suggested by: Andrew Gallatin Basically any mmap'd data doesn't seem to get sync()'d out on a regular basis. and this is used as a security feature for cryptography software. mlock() is used to prevent pages being swapped out. Its use for crypto software is essentially restricted to anon memory (allocated via brk() or mmap() of /dev/zero). What about userland device drivers that want to send parts of a disk backed file to a driver's dma routine? -- -Alfred Perlstein - [[EMAIL PROTECTED]|[EMAIL PROTECTED]] Daemon News Magazine in your snail-mail! http://magazine.daemonnews.org/ ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/users-lounge/docs/faq.html
[HACKERS] Re: Performance monitor signal handler
On reciept of the info signal, the backends collaborate to piece together a status file. The status file is given a temporay name. When complete the status file is rename(2)'d over a well known file. Reporting to files, particularly well known ones, could lead to race conditions. All in all, I think your better off passing messages through pipes or a similar communication method. I really liked the idea of a "server" that could parse/analyze data from multiple backends. My 2/100 worth... ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
[HACKERS] Re: Performance monitor signal handler
* Thomas Swan [EMAIL PROTECTED] [010313 13:37] wrote: On reciept of the info signal, the backends collaborate to piece together a status file. The status file is given a temporay name. When complete the status file is rename(2)'d over a well known file. Reporting to files, particularly well known ones, could lead to race conditions. All in all, I think your better off passing messages through pipes or a similar communication method. I really liked the idea of a "server" that could parse/analyze data from multiple backends. My 2/100 worth... Take a few moments to think about the semantics of rename(2). Yes, you would still need syncronization between the backend processes to do this correctly, but not any external app. The external app can just open the file, assuming it exists it will always have a complete and consistant snapshot of whatever the backends agreed on. -- -Alfred Perlstein - [[EMAIL PROTECTED]|[EMAIL PROTECTED]] Daemon News Magazine in your snail-mail! http://magazine.daemonnews.org/ ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] WAL SHM principles
On Tue, 13 Mar 2001, Alfred Perlstein wrote: [..] Linux does not filesystem-sync file-backed writable mmap pages on a regular basis. Very intersting. I'm not sure that is necessarily the case in 2.4, though -- my understanding is that the new all-singing, all-dancing page cache makes very little distinction between mapped and unmapped dirty pages. Basically any mmap'd data doesn't seem to get sync()'d out on a regular basis. Hmm.. I'd call that a bug, anyway. and this is used as a security feature for cryptography software. mlock() is used to prevent pages being swapped out. Its use for crypto software is essentially restricted to anon memory (allocated via brk() or mmap() of /dev/zero). What about userland device drivers that want to send parts of a disk backed file to a driver's dma routine? And realtime software. I'm not disputing that mlock is useful, but what it can do be security software is not that huge. The Linux manpage says: Memory locking has two main applications: real-time algo rithms and high-security data processing. Matthew. ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
[HACKERS] Re: xlog loose ends, continued
That'll be true in any case, unless we refuse to start up at all upon detecting xlog corruption (which doesn't seem like the way to fly). Not sure what we can do about that. What I would refuse in the event of log corruption is continuing normal database operations. Hmm. We could do that if we had some notion of a read-only operating mode, perhaps. But we don't have one now and I don't want to add it for 7.1. Can we agree to look at this more for 7.2? I'd like to have a readonly mode driven by integrity requirements for corruption recovery for database tables, for replication, and (in the future) for distributed databases, so perhaps we can do a trial implementation fairly soon. Not sure how it would impact the backend(s), but istm that we might be able to do a first implementation for 7.1.x. I'll bring it up again when appropriate... - Thomas ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
Re: [HACKERS] Performance monitor
Hi Guys, I'd just like to point out that for most secure installations, X is removed from servers as part of the "remove all software which isn't absolutely needed." I know of Solaris machines which perform as servers with a total of 19 OS packages installed, and then precompiled binaries of the server programs are loaded onto these machines. Removal of all not-absolutely-necessary software iss also the recommended procedure by Sun for setting up server platforms. Having something based on X will be useable by lots of people, just not by those who make the effort to take correct security precautions. Regards and best wishes, Justin Clift Bruce Momjian wrote: It will be tck/tk, so I guess X only. Good point. A typical DB server -- where is performance important -- has install Xwin? BTW, I hate Oracle 8.x.x because has X+java based installer, but some my servers hasn't monitor and keyboard let alone to Xwin. What implement performance monitor as client/server application where client is some shared lib? This solution allows to create more clients for more differents GUI. I know... it's easy planning, but the other thing is programming it :-) My idea is that they can telnet into the server machine and do remote-X with the application. Just set the DISPLAY variable and it should work. -- Bruce Momjian| http://candle.pha.pa.us [EMAIL PROTECTED] | (610) 853-3000 + If your life is a hard drive, | 830 Blythe Avenue + Christ can be your backup.| Drexel Hill, Pennsylvania 19026 ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED] ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: [HACKERS] RE: xlog loose ends, continued
"Mikheev, Vadim" wrote: What I was thinking about in that last paragraph was manual analysis and recovery. I don't think it's a good idea for automatic system startup to skip over gaps in the log. But if we'll not try to read after gap then after restart system will not notice gap and valid records after it and just rewrite log space with new records. Not much chance for manual analysis - ppl will not report any problems. That'll be true in any case, unless we refuse to start up at all upon detecting xlog corruption (which doesn't seem like the way to fly). Not sure what we can do about that. What I would refuse in the event of log corruption is continuing normal database operations. Log corruption is never an unique cause of a recovery failure. If there's a bug in redo stuff the result would also be a recovery failure. Currently the redo stuff has to accomplish redo operations completely. No matter how trivial the bug may be, it's always serious unfortunately. It's ok to dump such database for manual recovery, but continuing to use it is VERY BAD THING. The fact that users will use inconsistent DB may become big headache for us - just imagine compains about index scans returning incorrect results (index tuples pointing to free heap space was left and then that space was used for tuple with different keys). Hmm this seems nothing worse than 7.0. I would complain if postmaster couldn't restart due to this reason. IIRC few ppl mind the (even system) index corruption. Failing to restart was bad but silent restart in the event of log corruption is bad too. Agreed. regards, Hiroshi Inoue ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://www.postgresql.org/search.mpl
Re: [HACKERS] Re: xlog loose ends, continued
Tom Lane wrote: "Mikheev, Vadim" [EMAIL PROTECTED] writes: That'll be true in any case, unless we refuse to start up at all upon detecting xlog corruption (which doesn't seem like the way to fly). Not sure what we can do about that. What I would refuse in the event of log corruption is continuing normal database operations. Hmm. We could do that if we had some notion of a read-only operating mode, perhaps. But we don't have one now and I don't want to add it for 7.1. Can we agree to look at this more for 7.2? I'd love to see PostgreSQL have a read-only mode of some kind that let enquiry function against a possibly otherwise corrupted database, without the stress of worrying that you might be making things worse. I know other DB servers that have this sort of thing, and it has been a life-saver for me on occasion to allow critical information to be extracted before you nuke it all and start over. Cheers, Andrew. -- _ Andrew McMillan, e-mail: [EMAIL PROTECTED] Catalyst IT Ltd, PO Box 10-225, Level 22, 105 The Terrace, Wellington Me: +64 (21) 635 694, Fax: +64 (4) 499 5596, Office: +64 (4) 499 2267 ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] Performance monitor
It will be tck/tk, so I guess X only. Good point. A typical DB server -- where is performance important -- has install Xwin? BTW, I hate Oracle 8.x.x because has X+java based installer, but some my servers hasn't monitor and keyboard let alone to Xwin. What implement performance monitor as client/server application where client is some shared lib? This solution allows to create more clients for more differents GUI. I know... it's easy planning, but the other thing is programming it :-) My idea is that they can telnet into the server machine and do remote-X with the application. Just set the DISPLAY variable and it should work. -- Bruce Momjian| http://candle.pha.pa.us [EMAIL PROTECTED] | (610) 853-3000 + If your life is a hard drive, | 830 Blythe Avenue + Christ can be your backup.| Drexel Hill, Pennsylvania 19026 ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
[HACKERS] Re: Performance monitor
Bruce Momjian [EMAIL PROTECTED] writes: My idea is that they can telnet into the server machine and do remote-X with the application. Just set the DISPLAY variable and it should work. Well, actually you would want to tunnel your X session through ssh if security of the database server is of any importance... But this works fine on high bandwidth connections, but X is a real pain if you are sitting with low bandwidth(e.g. cellphone connection when you're in the middle of nowhere on your favorite vacation ;-) regards, Gunnar ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/users-lounge/docs/faq.html
Re: [HACKERS] Performance monitor
Bruce Momjian [EMAIL PROTECTED] writes: My idea is that they can telnet into the server machine and do remote-X with the application. Just set the DISPLAY variable and it should work. Remote X pretty well sucks in the real world. Aside from speed issues there is the little problem of firewalls filtering out X connections. If you've got ssh running then you can tunnel the X connection through the ssh connection, which fixes the firewall problem, but it makes the speed problem worse. And getting ssh plus X forwarding working is not something I want to have to hassle with when my remote database is down. If you are thinking of telnet-based remote admin then I suggest you get out your curses man page and do up a curses GUI. (No smiley... I'd seriously prefer that to something that depends on remote X.) regards, tom lane ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send "unregister YourEmailAddressHere" to [EMAIL PROTECTED])
