Re: [HACKERS] Issues for named/mixed function notation patch
2009/9/30 Pavel Stehule pavel.steh...@gmail.com: So I dropped variadic functions from mixed/named notation and little bit modified documentation. Please, can some native English speaker look on documentation? Hi Pavel, I've had a look through the documentation and cleaned up a few things. I changed the chapter heading from Positional and named notation to Calling Functions. I felt that the original heading would not give a clear hint about the subject matter to anyone who wasn't already familiar with the terminology. The rest of my changes are rephrasing sentences, fixing spelling and grammar, and cleaning up indentation and wrapping. The code examples are not changed and I haven't meddled with the structure of the chapter. I hope you find this helpful. Cheers, BJ named-notation-doc-fixes.diff Description: Binary data -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] navigation menu for documents
Peter Eisentraut wrote: On Fri, 2009-07-17 at 13:58 +0100, Richard Huxton wrote: 2. Titles on navigation links. Run ./STYLING/title_links.pl and it should add title attributes to the navigation links. This means hovering over the top links gives the title of the page they will go to. Presumably we could do this directly from the sgml source, and I think it's probably worthwhile. I have updated the stylesheet to add a title attribute to the header links. That has about the same effect as your script. Ah, good. My script was only ever intended to demonstrate. That's one item we can tick off. -- Richard Huxton Archonet Ltd -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] navigation menu for documents
David E. Wheeler wrote: On Sep 29, 2009, at 8:55 AM, Richard Huxton wrote: For the browser, does the following match what you're after, Andrew? - clicking chapter title opens the browser panel - panel stays open until you click close icon - panel contains collapsable tree of chapter/section headings Alternatively, could just auto-open the browser panel if javascript is enabled and window is wider than N pixels. Why wouldn't the entire TOC be in a collapsed list? Permanently on-screen? My only concern there would be for people viewing on phones etc. In addition we'll presumably want to meet: - no external js libraries (or do we care, if we just reference it from google?) Save yourself the hassle and just bundle jQuery. That's what I've done for Pod::Site (module that builds the Bricolage API browser). It's MIT licensed (well MIT+GPL) which is BSD compatible, but I don't know if that's acceptable. It would be easier for me if it could be bundled and presumably make it easier for other contributors in the future too. - navigation is optional, disabling js leaves docs as at present As long as there's a way to get the nav back from a link on each doc page. - works on all reasonable browsers (anything not IE6) +1 (IE6--) - works online and in downloaded docs (except Windows .chm of course) That'd be nice, too. Offline is crucial as far as I'm concerned. -- Richard Huxton Archonet Ltd -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Use samehost by default in pg_hba.conf?
Hi, Tom Lane t...@sss.pgh.pa.us writes: Peter Eisentraut pete...@gmx.net writes: On Wed, 2009-09-30 at 22:08 -0400, Tom Lane wrote: (Note that you would still need a non-default setting of listen_addresses for -h machine_name to actually work.) Which makes this proposal kind of uninteresting. As already said, it's one less step to have it working. This hba file is hard to get at for a lot of newbies we see on IRC. +1 for default configuration using samehost. Although come to think of it ... is there any reason besides sheer conservatism to not make the default listen_addresses value '*'? It won't result in letting in any outside connections unless you also add pg_hba.conf entries. Everywhere possible I have listen_addresses set to '127.0.0.1' a pgbouncer instance for clients to connect to, on the non loopback interface. That allows me to be sure that developers won't accidently bypass pgbouncer. But as we're only talking about default setup, Magnus argument weights much more (no DOS or portscan). Regards, -- dim -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Use samehost by default in pg_hba.conf?
At 2009-09-30 22:08:12 -0400, t...@sss.pgh.pa.us wrote: # local connections via TCP/IP: hostall all samehost @authmethod@ I think that's an excellent idea. On the other hand, I tend to be slightly against the idea of changing the default listen_addresses from localhost to '*', for a combination of the reasons mentioned by others; but I don't have strong feelings about it. -- ams -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] Limit allocated memory per session
I'd like to propose adding a new GUC to limit the amount of memory a backend
can allocate for its own use. The problem this addresses is that sometimes
one needs to set work_mem fairly high to get good query plans for large joins.
However, some complex queries will then use huge amounts of memory so that
one or a few of them will consume all the memory on the host and run it deep
into swap or trigger the oom killer or worse.
I've attached a patch based on 8.4.1. It works by keeping a track of the
total memory allocated via malloc to AllocBlocks (aset.c). If this is not
shot down/up too badly I will rebase it on CVS and submit it for the next
commit fest.
I would also like to propose a similar limit on temp space use. It is quite
easy for an unintended cartesion product to use hundreds of gigabytes of
scratch space and cause other processes to fail due to lack of disk space.
If this is not objectionable, I'll work on it too.
-dg
--
David Gould da...@sonic.net 510 536 1443510 282 0869
If simplicity worked, the world would be overrun with insects.
*** ./src/include/utils/memutils.h.orig 2009-09-30 01:54:36.0 -0700
--- ./src/include/utils/memutils.h 2009-09-30 03:33:44.0 -0700
***
*** 114,119
--- 114,122
*/
/* aset.c */
+
+ extern int max_allocated_mem;
+
extern MemoryContext AllocSetContextCreate(MemoryContext parent,
const char *name,
Size minContextSize,
*** ./src/backend/utils/mmgr/aset.c.orig2009-09-29 16:14:23.0
-0700
--- ./src/backend/utils/mmgr/aset.c 2009-10-01 03:07:34.0 -0700
***
*** 168,173
--- 168,187
} AllocBlockData;
/*
+ * AllocBlock accounting maintains total allocated memory to enforce the
memory use limit.
+ */
+ int max_allocated_mem = 0;
+ Size AllocBlockAccountingMemUsed = 0;
+
+ #define AllocBlockAccountingFree(block) \
+ (AllocBlockAccountingMemUsed -= block-endptr - (char
*) (block))
+ #define AllocBlockAccountingAlloc(block) \
+(AllocBlockAccountingMemUsed += block-endptr - (char
*) (block))
+ #define AllocBlockAccountingOverLimit() \
+ (max_allocated_mem != 0 \
+ AllocBlockAccountingMemUsed / 1024
max_allocated_mem)
+
+ /*
* AllocChunk
*The prefix of each piece of memory in an AllocBlock
*
***
*** 393,398
--- 407,423
context-blocks = block;
/* Mark block as not to be released at reset time */
context-keeper = block;
+
+ AllocBlockAccountingAlloc(block);
+ if (AllocBlockAccountingOverLimit())
+ {
+ MemoryContextStats(TopMemoryContext);
+ ereport(ERROR,
+ (errcode(ERRCODE_OUT_OF_MEMORY),
+errmsg(memory limit exceeded),
+errdetail(Failed while creating
memory context \%s\.,
+ name)));
+ }
}
context-isReset = true;
***
*** 476,481
--- 501,507
else
{
/* Normal case, release the block */
+ AllocBlockAccountingFree(block);
#ifdef CLOBBER_FREED_MEMORY
/* Wipe freed memory for debugging purposes */
memset(block, 0x7F, block-freeptr - ((char *) block));
***
*** 521,526
--- 547,553
{
AllocBlock next = block-next;
+ AllocBlockAccountingFree(block);
#ifdef CLOBBER_FREED_MEMORY
/* Wipe freed memory for debugging purposes */
memset(block, 0x7F, block-freeptr - ((char *) block));
***
*** 597,602
--- 624,640
set-blocks = block;
}
+ AllocBlockAccountingAlloc(block);
+ if (AllocBlockAccountingOverLimit())
+ {
+ MemoryContextStats(TopMemoryContext);
+ ereport(ERROR,
+ (errcode(ERRCODE_OUT_OF_MEMORY),
+errmsg(memory limit exceeded),
+errdetail(Failed on request of size
%lu.,
+ (unsigned long)
size)));
+ }
+
set-isReset = false;
AllocAllocInfo(set, chunk);
***
*** 767,772
--- 805,821
block-next = set-blocks;
set-blocks = block;
+
+ AllocBlockAccountingAlloc(block);
+ if
Re: [HACKERS] Hot Standby on git
+ /*
+* If our initial RunningXactData had an overflowed snapshot then we
+* knew we were missing some subxids from our snapshot. We can use
+* this data as an initial snapshot, but we cannot yet mark it valid.
+* We know that the missing subxids are equal to or earlier than
+* LatestRunningXid. After we initialise we continue to apply changes
+* during recovery, so once the oldestRunningXid is later than the
+* initLatestRunningXid we can now prove that we no longer have
+* missing information and can mark the snapshot as valid.
+*/
+ if (initRunningXactData !recoverySnapshotValid)
+ {
+ if (TransactionIdPrecedes(initLatestRunningXid,
xlrec-oldestRunningXid)
+ {
+ recoverySnapshotValid = true;
+ elog(trace_recovery(DEBUG2),
+ running xact data now proven complete);
+ elog(trace_recovery(DEBUG2),
+ recovery snapshots are now enabled);
+ }
+ return;
+ }
+
When GetRunningXactData() calculates latestRunningXid in the master,
which is stored in initLatestRunningXid in the standby, it only looks at
xids and subxids present in the procarray. It doesn't take into account
overflowed subxids. I think we could declare a recovery snapshot proven
complete too early because of that.
--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] CommitFest 2009-09, two weeks on
On Thu, Oct 1, 2009 at 04:11, Itagaki Takahiro itagaki.takah...@oss.ntt.co.jp wrote: Magnus Hagander mag...@hagander.net wrote: I can certainly review the win32 encoding patch, but I was rather hoping for some comments from others on if we're interested in a win32 only solution, or if we want something more generic. Should we just go with the win32-only one for now? Yes, because Windows is only platform that supports UTF-16 encoding natively. I believe my patch is the best solution for Windows even if we have another approach for other platforms. Actually, I think a better argument is that since Windows will *never* accept UTF8 logging, and that's what most databases will be in, much of this patch will be required anyway. So I should probably review and get this part in while we think about other solutions *as well* for other platforms. -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Use samehost by default in pg_hba.conf?
* Jaime Casanova (jcasa...@systemguards.com.ec) wrote: just my 2 cents. but i always wondered about the existence of listen_addresses at all... to me the only reason it exists is to force me to change 'localhost' to '*' after installing, something i always do almost automaticaly =) Try running two postmasters on the same box using the same port (but different IPs) w/o being able to control what IPs it listens on. Thanks, Stephen signature.asc Description: Digital signature
Re: [HACKERS] CommitFest 2009-09, two weeks on
On Wed, Sep 30, 2009 at 12:34:23PM -0400, Tom Lane wrote: qualified to review them. (I don't actually think we have anybody except Michael who's really familiar with ecpg.) I'm afraid I'm simply not able to spend much time on this in the near future as I'm simply too busy atm. I spend some time on these the last time, but wasn't even able to see how Zoltan changed the points we mentioned back then, but I'm sure he has. As already noted the patches stack on each other. There doesn't seem to be a technical reason for this at least not for some of those dependencies. With the first patch changing the grammar file and thus taking quite some reviewing effort this slows things down even more because one needs more effort to review for instance the sqlda addition, although that one seems to be quite easy to review. All of this is written from the top of my head, so please bear with me if I missed any changes in the patches. Michael -- Michael Meskes Michael at Fam-Meskes dot De, Michael at Meskes dot (De|Com|Net|Org) Michael at BorussiaFan dot De, Meskes at (Debian|Postgresql) dot Org ICQ: 179140304, AIM/Yahoo/Skype: michaelmeskes, Jabber: mes...@jabber.org Go VfL Borussia! Go SF 49ers! Use Debian GNU/Linux! Use PostgreSQL! -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Use samehost by default in pg_hba.conf?
Stephen Frost wrote: * Jaime Casanova (jcasa...@systemguards.com.ec) wrote: just my 2 cents. but i always wondered about the existence of listen_addresses at all... to me the only reason it exists is to force me to change 'localhost' to '*' after installing, something i always do almost automaticaly =) Try running two postmasters on the same box using the same port (but different IPs) w/o being able to control what IPs it listens on. Right. listen_addresses replaced two previous settings: tcpip_socket and virtual_host. And back before 8.0 the default setting was not to listen on *any* IP address. I'm inclined to think our default setting is about right. It is very common to run an application on the same machine as the database server it uses, and then there is no often need to adjust listen_addresses at all. cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Rejecting weak passwords
In the discussion following http://archives.postgresql.org/pgsql-hackers/2009-09/msg01766.php the consensus was that a hook that allows you to implement a password checking routine as a module would not hurt. So here's the patch. I don't think there is documentation required; correct me if I am wrong. Yours, Laurenz Albe passwd-check-hook.patch Description: passwd-check-hook.patch -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Rejecting weak passwords
On Thu, Oct 1, 2009 at 1:53 PM, Albe Laurenz laurenz.a...@wien.gv.at wrote: In the discussion following http://archives.postgresql.org/pgsql-hackers/2009-09/msg01766.php the consensus was that a hook that allows you to implement a password checking routine as a module would not hurt. So here's the patch. I don't think there is documentation required; correct me if I am wrong. How will people know how to use it, or that it's even there without at least a note in the docs somewhere? -- Dave Page EnterpriseDB UK: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Rejecting weak passwords
Dave Page wrote: On Thu, Oct 1, 2009 at 1:53 PM, Albe Laurenz laurenz.a...@wien.gv.at wrote: In the discussion following http://archives.postgresql.org/pgsql-hackers/2009-09/msg01766.php the consensus was that a hook that allows you to implement a password checking routine as a module would not hurt. So here's the patch. I don't think there is documentation required; correct me if I am wrong. How will people know how to use it, or that it's even there without at least a note in the docs somewhere? I'd prefer to have an example as a contrib module, as well as docs. Quite apart from anything else, how the heck would we test it without such a thing? cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Rejecting weak passwords
Andrew Dunstan wrote: So here's the patch. I don't think there is documentation required; correct me if I am wrong. How will people know how to use it, or that it's even there without at least a note in the docs somewhere? I'd prefer to have an example as a contrib module, as well as docs. Quite apart from anything else, how the heck would we test it without such a thing? I was not sure because no other hooks were documented anywhere else than in the code. I could add a paragraph in the auth-password section of client-auth.sgml. Or is there a better place? I could easily write a simple contrib that adds a check for username = password if there is interest. Yours, Laurenz Albe -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] CommitFest 2009-09, two weeks on
Michael Meskes írta: On Wed, Sep 30, 2009 at 12:34:23PM -0400, Tom Lane wrote: qualified to review them. (I don't actually think we have anybody except Michael who's really familiar with ecpg.) I'm afraid I'm simply not able to spend much time on this in the near future as I'm simply too busy atm. I spend some time on these the last time, but wasn't even able to see how Zoltan changed the points we mentioned back then, but I'm sure he has. As already noted the patches stack on each other. There doesn't seem to be a technical reason for this at least not for some of those dependencies. With the first patch changing the grammar file and thus taking quite some reviewing effort this slows things down even more because one needs more effort to review for instance the sqlda addition, although that one seems to be quite easy to review. You're not being fair with me. The dependencies are quite technical. First, Tom Lane suggested to unify core and ecpg FETCH syntaxes so both will accept optional FROM/IN, which I did. SQLDA support adds new FETCH forms (Informix-specific ones) so naturally these patches clash. There's no simple way to make they separately applicable. With the first version, the same technical dependency were also there, because of the (already explained) grammar problem, I got 2 shift/reduce problems in the FETCH/MOVE stmts unless I de-factorized FORWARD and BACKWARD out of fetch_direction. The new FETCH forms with SQLDA touched the same areas in ecpg.addon. Second, DESCRIBE support and SQLDA support also overlap, because SQLDA is a new descriptor form, and DESCRIBE has to support both SQL descriptors and SQLDA. Well, I can split the DESCRIBE patch in half, so it will be usable on SQL descriptors but the other part would depend on both SQLDA and basic DESCRIBE support. Technical dependency again. What non-technical dependencies are you talking about? Please explain, so I may fix them. Saying it vaguely doesn't help. When I first posted the split patchset, you didn't tell me that the split is no good. I tried everything help I could to explain why I did what. Also, the current reviewer (Dan Colish) haven't contacted me despite I offered help privately. I can't review my own patches, that's clear. But I can't do anything else to speed review up but to offer my help and wait for the help/explanation request that didn't arrive. Also, I have sent some independent very small patches, that don't even need much review. One of them (the typo in pgc.l) was already applied and I thank you Michael, but the memory leak fix for two improperly freed numerics is still behind. Please, look at that patch, it should be really obvious. All of this is written from the top of my head, so please bear with me if I missed any changes in the patches. Michael Best regards, Zoltán Böszörményi -- Bible has answers for everything. Proof: But let your communication be, Yea, yea; Nay, nay: for whatsoever is more than these cometh of evil. (Matthew 5:37) - basics of digital technology. May your kingdom come - superficial description of plate tectonics -- Zoltán Böszörményi Cybertec Schönig Schönig GmbH http://www.postgresql.at/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Use samehost by default in pg_hba.conf?
Tom Lane wrote: Now that the samehost/samenet patch is in, I wonder if it wouldn't be a good idea to replace this part of the default pg_hba.conf file: You're probably not suggesting this, but I would be against a default setting of 'samehost' used with 'trust'. Essentially that would be the same as rlogin rsh, where if the user can spoof a TCP connection, he can connect to postgresql. Depending on the platform, an interface may have to be down for this to work. Cheers, Stef -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Use samehost by default in pg_hba.conf?
Tom Lane wrote: Having looked at the code, I think that samehost is pretty safe. I'm still worried about samenet picking up a bogusly broad netmask --- but samehost hard-wires the netmask at all-ones. Even if your network configuration is really screwed up, the kernel isn't going to send that traffic off-machine. So I think it will act as advertised. But will it accept traffic from off machine? If so, then essentially the only line of defense is the security of the TCP stack. Or am I missing something? Cheers, Stef -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Rejecting weak passwords
On Thu, Oct 1, 2009 at 15:26, Albe Laurenz laurenz.a...@wien.gv.at wrote: Andrew Dunstan wrote: So here's the patch. I don't think there is documentation required; correct me if I am wrong. How will people know how to use it, or that it's even there without at least a note in the docs somewhere? I'd prefer to have an example as a contrib module, as well as docs. Quite apart from anything else, how the heck would we test it without such a thing? I was not sure because no other hooks were documented anywhere else than in the code. I could add a paragraph in the auth-password section of client-auth.sgml. Or is there a better place? I could easily write a simple contrib that adds a check for username = password if there is interest. I think it's better to have an actually *useful* contrib module for it, if there is one. Meaning perhaps something that links to that cracklib thing mentioned upthread. -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Hot Standby on git
On Wed, 2009-09-30 at 18:45 +0300, Heikki Linnakangas wrote: Regarding this in InitStandbyDelayTimers: + /* +* If replication delay is enormously huge, just treat that as +* zero and work up from there. This prevents us from acting +* foolishly when replaying old log files. +*/ + if (*currentDelay_ms 0) + *currentDelay_ms = 0; + So we're treating restoring from an old backup the same as an up-to-date standby server. If you're restoring from say a month old base backup with WAL archive up to present day, and have max_standby_delay set to say 5 seconds, the server will wait for that 5 seconds on each conflicting query before killing it. Until it reaches the point in the archive where the delay is less than INT_MAX/1000 seconds old: at that point it switches into oh my goodness, we've fallen badly behind, let's try to catch up ASAP and kill any queries that get into the way mode. That's pretty surprising behavior, and not documented either. I propose we simply remove the above check (fixing the rest of the code so that you don't hit integer overflows), and always respect max_standby_delay. Agreed. I will docuemnt the recommendation to set max_standby_delay = 0 if performing an archive recovery (and explain why). BTW, I wonder if should warn or something if we find that the timestamps in the archive are in the future? IOW, if either the master's or the standby's clock is not set correctly. Something similar was just spotted by a client. You can set a recovery_target_timestamp that is before the pg_stop_recovery() timestamp and it doesn't complain. Will fix. Not sure if I like the sound of a system moaning at me about the clock settings. Perhaps just once when it starts, when we read control file. -- Simon Riggs www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Rejecting weak passwords
On Thu, Oct 01, 2009 at 03:54:37PM +0200, Magnus Hagander wrote: On Thu, Oct 1, 2009 at 15:26, Albe Laurenz laurenz.a...@wien.gv.at wrote: Andrew Dunstan wrote: So here's the patch. I don't think there is documentation required; correct me if I am wrong. How will people know how to use it, or that it's even there without at least a note in the docs somewhere? I'd prefer to have an example as a contrib module, as well as docs. Quite apart from anything else, how the heck would we test it without such a thing? I was not sure because no other hooks were documented anywhere else than in the code. I could add a paragraph in the auth-password section of client-auth.sgml. Or is there a better place? I could easily write a simple contrib that adds a check for username = password if there is interest. I think it's better to have an actually *useful* contrib module for it, if there is one. Meaning perhaps something that links to that cracklib thing mentioned upthread. +1 for a sample module that will allow cracklib to drop in. Cheers, Ken -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Rejecting weak passwords
* Albe Laurenz laurenz.a...@wien.gv.at [091001 08:54]: In the discussion following http://archives.postgresql.org/pgsql-hackers/2009-09/msg01766.php the consensus was that a hook that allows you to implement a password checking routine as a module would not hurt. So here's the patch. I don't think there is documentation required; correct me if I am wrong. Can we have the check password hook function called in both the plaintext/encrypted case, with a flag as to whether it's encrypted or not? a. -- Aidan Van Dyk Create like a god, ai...@highrise.ca command like a king, http://www.highrise.ca/ work like a slave. signature.asc Description: Digital signature
Re: [HACKERS] Issues for named/mixed function notation patch
2009/10/1 Brendan Jurd dire...@gmail.com: 2009/9/30 Pavel Stehule pavel.steh...@gmail.com: So I dropped variadic functions from mixed/named notation and little bit modified documentation. Please, can some native English speaker look on documentation? Hi Pavel, I've had a look through the documentation and cleaned up a few things. I changed the chapter heading from Positional and named notation to Calling Functions. I felt that the original heading would not give a clear hint about the subject matter to anyone who wasn't already familiar with the terminology. The rest of my changes are rephrasing sentences, fixing spelling and grammar, and cleaning up indentation and wrapping. The code examples are not changed and I haven't meddled with the structure of the chapter. I hope you find this helpful. yes, it's very helpful. I thing, so there is all what is important. thank you Pavel Cheers, BJ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Hot Standby 0.2.1
On Sun, 2009-09-27 at 15:35 +0300, Heikki Linnakangas wrote: TransactionIdIsInProgress() doesn't consult the known-assigned-xids structure. That's a problem: in the standby, TransactionIdIsInProgress() can return false for a transaction that is still running in the master. HeapTupleSatisfies* functions can incorrectly set HEAP_XMIN/XMAX_INVALID hint bits for xids that commit later on. Agreed. Will fix. -- Simon Riggs www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Use samehost by default in pg_hba.conf?
Jaime Casanova jcasa...@systemguards.com.ec wrote: Tom Lane t...@sss.pgh.pa.us wrote: Although come to think of it ... is there any reason besides sheer conservatism to not make the default listen_addresses value '*'? just my 2 cents. but i always wondered about the existence of listen_addresses at all... to me the only reason it exists is to force me to change 'localhost' to '*' after installing, something i always do almost automaticaly =) For all the reasons already given, -1 from me for changing the default of listen_addresses or eliminating it. Trust authentication has a few valid use cases, but it does tend to worry me that people may leave it enabled in inappropriate situations on production clusters. I don't see how we could get rid of it, but I'd be OK with a warning in the log when a pg_hba.conf file is processed which contains any trust entries. -Kevin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] navigation menu for documents
On Oct 1, 2009, at 1:12 AM, Richard Huxton wrote: Why wouldn't the entire TOC be in a collapsed list? Permanently on-screen? My only concern there would be for people viewing on phones etc. I have to admit that I'm never looking at the Pg docs on my iPhone. This is mainly because I use them as a reference while hacking, and I'm not (yet) hacking PostgreSQL on my phone. It's MIT licensed (well MIT+GPL) which is BSD compatible, but I don't know if that's acceptable. It would be easier for me if it could be bundled and presumably make it easier for other contributors in the future too. Agreed. IIUC, the MIT license is compatible. Best, David -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] navigation menu for documents
David E. Wheeler wrote: On Oct 1, 2009, at 1:12 AM, Richard Huxton wrote: Why wouldn't the entire TOC be in a collapsed list? Permanently on-screen? My only concern there would be for people viewing on phones etc. I have to admit that I'm never looking at the Pg docs on my iPhone. This is mainly because I use them as a reference while hacking, and I'm not (yet) hacking PostgreSQL on my phone. Ah, I _do_ look at them on my Nokia N810 when I'm on the train etc. -- Richard Huxton Archonet Ltd -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Use samehost by default in pg_hba.conf?
On 10/01/2009 10:24 AM, Kevin Grittner wrote: Trust authentication has a few valid use cases, but it does tend to worry me that people may leave it enabled in inappropriate situations on production clusters. I don't see how we could get rid of it, but I'd be OK with a warning in the log when a pg_hba.conf file is processed which contains any trust entries. I don't think trust needs to be removed entirely - it is a valid option for demos or training sessions perhaps. By using the word abolishing, I might have created the wrong impression. I just meant the default pg_hba.conf having trust has always seemed to be a really bad thing to me. If people already have pg_hba.conf with trust, I see no reason to stop them. If a new user tries using PostgreSQL for the first time - I think the default configuration they encounter should be conservative and usable out of the box. I can see how samehost fits into this picture. I don't see how trust fits into this picture. Does anybody seriously recommend trust to newbies for production use? Shouldn't the default pg_hba.conf represent a conservative recommendation from the pgsql developers? Cheers, mark -- Mark Mielkem...@mielke.cc -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Hot Standby on git
On Thu, 2009-10-01 at 14:29 +0300, Heikki Linnakangas wrote:
+ /*
+* If our initial RunningXactData had an overflowed snapshot then we
+* knew we were missing some subxids from our snapshot. We can use
+* this data as an initial snapshot, but we cannot yet mark it valid.
+* We know that the missing subxids are equal to or earlier than
+* LatestRunningXid. After we initialise we continue to apply changes
+* during recovery, so once the oldestRunningXid is later than the
+* initLatestRunningXid we can now prove that we no longer have
+* missing information and can mark the snapshot as valid.
+*/
+ if (initRunningXactData !recoverySnapshotValid)
+ {
+ if (TransactionIdPrecedes(initLatestRunningXid,
xlrec-oldestRunningXid)
+ {
+ recoverySnapshotValid = true;
+ elog(trace_recovery(DEBUG2),
+ running xact data now proven complete);
+ elog(trace_recovery(DEBUG2),
+ recovery snapshots are now enabled);
+ }
+ return;
+ }
+
When GetRunningXactData() calculates latestRunningXid in the master,
which is stored in initLatestRunningXid in the standby, it only looks at
xids and subxids present in the procarray. It doesn't take into account
overflowed subxids. I think we could declare a recovery snapshot proven
complete too early because of that.
Hmm, yes. ISTM that I'm still calculating latestRunningXid the old way
while assuming it is calculated the new way. The new way is just to grab
nextXid since we have XidGenLock and do TransactionIdRetreat() on it.
--
Simon Riggs www.2ndQuadrant.com
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Limit allocated memory per session
daveg da...@sonic.net writes: I'd like to propose adding a new GUC to limit the amount of memory a backend can allocate for its own use. Use ulimit. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Hot Standby on git
On Wed, 2009-09-30 at 09:33 +0300, Heikki Linnakangas wrote: Looking at the changes to StartupMultiXact, you're changing the locking so that both MultiXactOffsetControlLock and MultiXactMemberControlLock are acquire first before changing anything. Why? Looking at the other functions in that file, all others that access both files are happy to acquire one lock at a time, like StartupMultiXact does without the patch. I think those changes are just paranoia from early versions of patch. We now know for certain that MultiXact isn't used at the time StartupMultiXact() is called. The same isn't true for StartupClog() and StartupSubtrans() which need to cope with concurrent callers. Will remove changes and document that nothing touching it when it runs. -- Simon Riggs www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] navigation menu for documents
On Thu, Oct 1, 2009 at 10:24 AM, David E. Wheeler da...@kineticode.com wrote: On Oct 1, 2009, at 1:12 AM, Richard Huxton wrote: Why wouldn't the entire TOC be in a collapsed list? Permanently on-screen? My only concern there would be for people viewing on phones etc. I have to admit that I'm never looking at the Pg docs on my iPhone. This is mainly because I use them as a reference while hacking, and I'm not (yet) hacking PostgreSQL on my phone. I probably shouldn't admit this, but I've been known to browse the *source code* on my iPhone while bored and daydream about some patch I'm interested in writing. So +1 from me on anything that is good for mobile users. There's no limit to what crazy things people will try to do with a phone. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Limit allocated memory per session
daveg wrote: I'd like to propose adding a new GUC to limit the amount of memory a backend can allocate for its own use. The problem this addresses is that sometimes one needs to set work_mem fairly high to get good query plans for large joins. However, some complex queries will then use huge amounts of memory so that one or a few of them will consume all the memory on the host and run it deep into swap or trigger the oom killer or worse. Oh, BTW, did anyone get interested in adding the bits to disable the OOM killer for postmaster on the various Linux initscripts? It needs some games with /proc/pid/oom_adj and requires root privileges, but I think an initscript is in an excellent position to do it. -- Alvaro Herrerahttp://www.CommandPrompt.com/ The PostgreSQL Company - Command Prompt, Inc. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] CommitFest 2009-09, two weeks on
Boszormenyi Zoltan escribió: First, Tom Lane suggested to unify core and ecpg FETCH syntaxes so both will accept optional FROM/IN, which I did. SQLDA support adds new FETCH forms (Informix-specific ones) so naturally these patches clash. There's no simple way to make they separately applicable. With the first version, the same technical dependency were also there, because of the (already explained) grammar problem, I got 2 shift/reduce problems in the FETCH/MOVE stmts unless I de-factorized FORWARD and BACKWARD out of fetch_direction. The new FETCH forms with SQLDA touched the same areas in ecpg.addon. Probably the parts that touch the core grammar can be reviewed and applied separately. -- Alvaro Herrerahttp://www.CommandPrompt.com/ The PostgreSQL Company - Command Prompt, Inc. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] hstore crasesh on 64bit Sparc
I'm looking why cometh_month fails and it is problem with last hstore putback: http://www.pgbuildfarm.org/cgi-bin/show_log.pl?nm=comet_mothdt=2009-09-30%2021:06:00 Stack trace is following: 6f013828 hstore_hash (7fffa050, 1395298, 1, 41, 7370616365414143, 0) + c0 0001005f396c FunctionCall1 (100b9ca10, 1395238, 7fffa4cf, 100ba0b48, 43, 100ba0b44) + 4c 0001002e8264 TupleHashTableHash (7fffa6f0, 8, 100b9f9e8, 1, 7fffa6ef, 0) + 15c 0001005fa734 hash_search (100ba20f8, 7fffa6f0, 1, 7fffa6ef, 0, 100ba4191) + 2c 0001002e7e74 LookupTupleHashEntry (100ba00e8, 100b9ae90, 7fffa7ef, 0, 1, 100b96b40) + 114 000100308420 lookup_hash_entry (100b9a978, 100b9b588, 0, 0, 0, 0) + 118 status: process terminated by SIGSEGV (Segmentation Fault), addr=500ba410c It is not assert problem but Segmentation Fault. 32bit compilation works fine (ghost moth). Any idea? Zdenek -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Rejecting weak passwords
Aidan Van Dyk wrote: Can we have the check password hook function called in both the plaintext/encrypted case, with a flag as to whether it's encrypted or not? It will be called in both cases, and I figured that you can check yourself the same way that PostgreSQL does: If isMD5(password), then it is treated as an encrypted password. Yours, Laurenz Albe -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Rejecting weak passwords
Dave Page dp...@pgadmin.org writes: How will people know how to use it, or that it's even there without at least a note in the docs somewhere? We don't normally document function hooks anywhere but the source code. That's one of the things that makes them low overhead ;-) I agree with the subsequent comments suggesting a sample module that actually does something useful --- although if it's going to link to external code like cracklib, it probably is going to have to be on pgfoundry not in contrib. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] FSM search modes
On Sep 30, 2009, at 5:13 PM, Kevin Grittner wrote: decibel deci...@decibel.org wrote: *any* step that improves dealing with table bloat is extremely welcome, as right now you're basically stuck rebuilding the table. +1 Although, possibly more irritating than actually rebuilding it is evaluating borderline bloat situations to determine if they will work themselves out over time or whether you need to bite the bullet to do aggressive maintenance. Having some way for routine vacuums (or any other routine process, currently available or that could be scheduled) to nibble away at moderate bloat without significant performance or usability impact would make life easier for at least *some* DBAs. Kevin, do you have tools that allow you to clear out the end of a table? That part is at least mostly possible from userland (get list of ctids from end of table, update those records to move them, rinse, repeat) but even if you do all that there's no guarantee that a vacuum will get the exclusive lock it needs to truncate the table. So while something that makes it easier to clean out the end of a table would be good, I think the critical need is a way to make vacuum more aggressive about obtaining the exclusive lock. -- Decibel!, aka Jim C. Nasby, Database Architect deci...@decibel.org Give your computer some brain candy! www.distributed.net Team #1828 -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Use samehost by default in pg_hba.conf?
Stef Walter stef-l...@memberwebs.com writes: Tom Lane wrote: Now that the samehost/samenet patch is in, I wonder if it wouldn't be a good idea to replace this part of the default pg_hba.conf file: You're probably not suggesting this, but I would be against a default setting of 'samehost' used with 'trust'. Essentially that would be the same as rlogin rsh, where if the user can spoof a TCP connection, he can connect to postgresql. Depending on the platform, an interface may have to be down for this to work. Is there any actual risk here that we aren't taking already just by allowing 127.0.0.1? regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Limit allocated memory per session
Tom Lane escreveu: daveg da...@sonic.net writes: I'd like to propose adding a new GUC to limit the amount of memory a backend can allocate for its own use. Use ulimit. What about plataforms (Windows) that don't have ulimit? -- Euler Taveira de Oliveira http://www.timbira.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Use samehost by default in pg_hba.conf?
On Thu, Oct 1, 2009 at 11:35 AM, Tom Lane t...@sss.pgh.pa.us wrote: Stef Walter stef-l...@memberwebs.com writes: Tom Lane wrote: Now that the samehost/samenet patch is in, I wonder if it wouldn't be a good idea to replace this part of the default pg_hba.conf file: You're probably not suggesting this, but I would be against a default setting of 'samehost' used with 'trust'. Essentially that would be the same as rlogin rsh, where if the user can spoof a TCP connection, he can connect to postgresql. Depending on the platform, an interface may have to be down for this to work. Is there any actual risk here that we aren't taking already just by allowing 127.0.0.1? I wouldn't bet that there isn't. I don't really think there's any need for our default configuration to be at the mercy of every half- baked TCP/IP implementation out there. A socket file in /tmp can't be remotely hacked (well, not directly anyway); anything else is further from a sure thing. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Limit allocated memory per session
Euler Taveira de Oliveira eu...@timbira.com writes: Tom Lane escreveu: daveg da...@sonic.net writes: I'd like to propose adding a new GUC to limit the amount of memory a backend can allocate for its own use. Use ulimit. What about plataforms (Windows) that don't have ulimit? Get a real operating system ;-) Seriously, the proposed patch introduces overhead into a place that is already a known hot spot, in return for not much of anything. It will *not* bound backend memory use very accurately, because there is no way to track raw malloc() calls. And I think that 99% of users will not find it useful. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Hot Standby on git
Simon Riggs wrote: On Wed, 2009-09-30 at 09:33 +0300, Heikki Linnakangas wrote: Looking at the changes to StartupMultiXact, you're changing the locking so that both MultiXactOffsetControlLock and MultiXactMemberControlLock are acquire first before changing anything. Why? Looking at the other functions in that file, all others that access both files are happy to acquire one lock at a time, like StartupMultiXact does without the patch. I think those changes are just paranoia from early versions of patch. We now know for certain that MultiXact isn't used at the time StartupMultiXact() is called. The same isn't true for StartupClog() and StartupSubtrans() which need to cope with concurrent callers. Will remove changes and document that nothing touching it when it runs. Thanks, I reverted that in my working version already. Comment patch welcome if you feel it's needed. Attached is a new batch of changes I've been doing since last batch. These are again extracted from my git repository. It includes some of the add-on patches from your repository, the rest I believe have I had already did myself earlier, or are not necessary anymore for other reasons. Could you look into these two TODO items you listed on the wiki page: - Correct SET default_transaction_read_only and SET transaction_read_only (Heikki 21/9 Hackers) - Shutdown checkpoints must not clear locks for prepared transactions (Heikki 23/9) And if you could please review the changes I've been doing, just to make sure I haven't inadvertently introduced new bugs. That has happened before, as you've rightfully reminded me :-). There's also the issue that you can't go into hot standby mode after a shutdown checkpoint. I think that really should be fixed, it's just weird from a usability point of view if it doesn't work. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com hs-riggs-branch-20091001.tar.gz Description: GNU Zip compressed data -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Hot Standby on git
Simon Riggs wrote: Hmm, yes. ISTM that I'm still calculating latestRunningXid the old way while assuming it is calculated the new way. The new way is just to grab nextXid since we have XidGenLock and do TransactionIdRetreat() on it. Ok, good, that's what I thought too. I'll fix that. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Limit allocated memory per session
In response to Euler Taveira de Oliveira eu...@timbira.com: Tom Lane escreveu: daveg da...@sonic.net writes: I'd like to propose adding a new GUC to limit the amount of memory a backend can allocate for its own use. Use ulimit. What about plataforms (Windows) that don't have ulimit? I have a hard time believing that Windows doesn't have a ulimit equivalent. I don't want to degrade this thread into MS-bashing, but if that were the case, it would make Windows a pretty crippled OS. -- Bill Moran http://www.potentialtech.com http://people.collaborativefusion.com/~wmoran/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Limit allocated memory per session
On Thu, Oct 1, 2009 at 11:47 AM, Tom Lane t...@sss.pgh.pa.us wrote: Euler Taveira de Oliveira eu...@timbira.com writes: Tom Lane escreveu: daveg da...@sonic.net writes: I'd like to propose adding a new GUC to limit the amount of memory a backend can allocate for its own use. Use ulimit. What about plataforms (Windows) that don't have ulimit? Get a real operating system ;-) Seriously, the proposed patch introduces overhead into a place that is already a known hot spot, in return for not much of anything. It will *not* bound backend memory use very accurately, because there is no way to track raw malloc() calls. And I think that 99% of users will not find it useful. What WOULD be useful is to find a way to provide a way to configure work_mem per backend rather than per executor node. But that's a much harder problem. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] FSM search modes
decibel wrote: So while something that makes it easier to clean out the end of a table would be good, I think the critical need is a way to make vacuum more aggressive about obtaining the exclusive lock. I wonder if we should have a different mode of operation that only attempted the truncate (say VACUUM TRUNCATE), optionally being non-conditional about obtaining the required lock. That said, I wonder even more whether any such hacks are still needed after the visilibity map that changed the landscape for vacuum so dramatically. -- Alvaro Herrerahttp://www.CommandPrompt.com/ The PostgreSQL Company - Command Prompt, Inc. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] FSM search modes
Alvaro Herrera alvhe...@commandprompt.com writes: I wonder if we should have a different mode of operation that only attempted the truncate (say VACUUM TRUNCATE), optionally being non-conditional about obtaining the required lock. That said, I wonder even more whether any such hacks are still needed after the visilibity map that changed the landscape for vacuum so dramatically. Yeah. The one thing in this thread that seemed like a good idea to me was to bias the FSM code a little bit towards returning space near the start of the relation, rather than its current behavior of treating all the free space equally. The current arrangement pretty much guarantees that you'll never recover from a bloating episode without taking special manual action. (This is not new to 8.4, the previous FSM code behaved the same.) The analogy in the back of my mind is the btree code that decides whether to split the current page or move to the next page when it has a full page and a new key that could go to either page. We found out that randomizing that choice made a *huge* improvement in the average behavior, even with the probabilities set up as 99-to-1. I'm thinking that just a small chance of resetting the search to the start of the relation might do the trick for FSM. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Hot Standby on git
On Thu, 2009-10-01 at 18:47 +0300, Heikki Linnakangas wrote: Could you look into these two TODO items you listed on the wiki page: Unless we agree otherwise, if its listed on the Wiki page then I will work on it. Maybe not as when you might like it, but I am working through the list. 5 new changes pushed just minutes ago, sans full testing. Yes, will review your changes also. -- Simon Riggs www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] CommitFest 2009-09, two weeks on
On Thu, Oct 01, 2009 at 03:47:07PM +0200, Boszormenyi Zoltan wrote: You're not being fair with me. The dependencies are quite technical. I'm sorry that you interpreted my email this way, it wasn't at all meant to offend you. First, Tom Lane suggested to unify core and ecpg FETCH syntaxes so both will accept optional FROM/IN, which I did. SQLDA support adds new FETCH forms (Informix-specific This is actually one I was talking about. Adding SQLDA *only* seems like an easy one, as soon as it also hits the parser it becomes more complicated. This is not to say that it is not doable, but it wasn't for me with my time constraints. I was just explaining why I didn't delve into these any more so far. When I first posted the split patchset, you didn't tell me that the split is no good. I tried everything help I could to explain why I did what. Well actually I did, but that's not the problem here. I have no idea why you are ranting like this just because of me excusing myself for a lack of time. Also, the current reviewer (Dan Colish) haven't contacted me despite I offered help privately. I can't review my own patches, that's clear. But I can't do anything else to speed review up but to offer my help and wait for the help/explanation request that didn't arrive. Okay, so it's all my fault. Do you feel better now? Also, I have sent some independent very small patches, that don't even need much review. One of them (the typo in pgc.l) was already applied and I thank you Michael, but the memory leak fix for two improperly freed numerics is still behind. Please, look at that patch, it should be really obvious. My last commit was one of your patches, obviously I didn't do anything on ecpg since. I'm not sure what you are trying to tell me, you might want to explain the last two sentences. But keep in mind you are *not* the one to decide how I spend my spare time. Michael -- Michael Meskes Michael at Fam-Meskes dot De, Michael at Meskes dot (De|Com|Net|Org) Michael at BorussiaFan dot De, Meskes at (Debian|Postgresql) dot Org ICQ: 179140304, AIM/Yahoo/Skype: michaelmeskes, Jabber: mes...@jabber.org Go VfL Borussia! Go SF 49ers! Use Debian GNU/Linux! Use PostgreSQL! -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Hot Standby on git
On Thu, 2009-10-01 at 18:48 +0300, Heikki Linnakangas wrote: Simon Riggs wrote: Hmm, yes. ISTM that I'm still calculating latestRunningXid the old way while assuming it is calculated the new way. The new way is just to grab nextXid since we have XidGenLock and do TransactionIdRetreat() on it. Ok, good, that's what I thought too. I'll fix that. OK, not working on that, so go ahead. Thanks. -- Simon Riggs www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Limit allocated memory per session
Robert Haas robertmh...@gmail.com writes: What WOULD be useful is to find a way to provide a way to configure work_mem per backend rather than per executor node. But that's a much harder problem. I think it's mostly a planner problem: how do you deal with the fact that that would make cost estimates for different sub-problems interrelated? The cost of a hash, for instance, depends a lot on how much memory you assume it can use. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Limit allocated memory per session
Robert Haas escreveu: On Thu, Oct 1, 2009 at 11:47 AM, Tom Lane t...@sss.pgh.pa.us wrote: Euler Taveira de Oliveira eu...@timbira.com writes: Tom Lane escreveu: daveg da...@sonic.net writes: I'd like to propose adding a new GUC to limit the amount of memory a backend can allocate for its own use. Use ulimit. What about plataforms (Windows) that don't have ulimit? Get a real operating system ;-) Seriously, the proposed patch introduces overhead into a place that is already a known hot spot, in return for not much of anything. It will *not* bound backend memory use very accurately, because there is no way to track raw malloc() calls. And I think that 99% of users will not find it useful. What WOULD be useful is to find a way to provide a way to configure work_mem per backend rather than per executor node. But that's a much harder problem. I see. Tough problem is: how do we get per backend memory usage accurately? Is it relying on OS specific API the only way? -- Euler Taveira de Oliveira http://www.timbira.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] FSM search modes
On Thu, 2009-10-01 at 12:05 -0400, Tom Lane wrote: Alvaro Herrera alvhe...@commandprompt.com writes: I wonder if we should have a different mode of operation that only attempted the truncate (say VACUUM TRUNCATE), optionally being non-conditional about obtaining the required lock. That said, I wonder even more whether any such hacks are still needed after the visilibity map that changed the landscape for vacuum so dramatically. Yeah. The one thing in this thread that seemed like a good idea to me was to bias the FSM code a little bit towards returning space near the start of the relation, rather than its current behavior of treating all the free space equally. The current arrangement pretty much guarantees that you'll never recover from a bloating episode without taking special manual action. (This is not new to 8.4, the previous FSM code behaved the same.) The analogy in the back of my mind is the btree code that decides whether to split the current page or move to the next page when it has a full page and a new key that could go to either page. We found out that randomizing that choice made a *huge* improvement in the average behavior, even with the probabilities set up as 99-to-1. I'm thinking that just a small chance of resetting the search to the start of the relation might do the trick for FSM. No real need to be random is there? In the bloated space scenario, VACUUM will be triggered but will be unable to remove the empty blocks. So in that case VACUUM can hint the FSM to perform start from beginning of relation behaviour. When a searcher does that and can't usefully find space quickly, then we can reset the hint back to normal. So it's an automated mode change in both directions. I think we can use the N-to-1 idea to define what we mean by quickly, but that should be in proportion to exactly how many free blocks there were in table, not a fixed N. It would be a shame to make this work poorly for very large tables. It would be more useful to think of this as look for huge chunks of space and fill them rather than start at beginning, since space won't always be right at start. -- Simon Riggs www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] hstore crasesh on 64bit Sparc
Zdenek Kotala zdenek.kot...@sun.com writes: I'm looking why cometh_month fails and it is problem with last hstore putback: I think this is the same 64-bit problem we fixed last night --- wait for the next rebuild before worrying. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] FSM search modes
Simon Riggs si...@2ndquadrant.com writes: No real need to be random is there? In the bloated space scenario, VACUUM will be triggered but will be unable to remove the empty blocks. So in that case VACUUM can hint the FSM to perform start from beginning of relation behaviour. No, that's an entirely unrelated issue. My point is that there won't *be* any empty end blocks unless we discourage FSM from handing out those pages as insertion targets. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] FSM search modes
Simon Riggs si...@2ndquadrant.com wrote: It would be more useful to think of this as look for huge chunks of space and fill them rather than start at beginning, since space won't always be right at start. Either I misunderstand you or I disagree. If there's a huge chunk of space near the end, and many many smaller spaces spread throughout, what I'd like is for rows to be placed in those small ones. This would minimize the number of pages to read for queries, and would present some possibility that the rows past the huge chunk might eventually be deleted or non-HOT updated, allowing the bloat to eventually be cleaned up with minimal pain. -Kevin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Limit allocated memory per session
Euler Taveira de Oliveira eu...@timbira.com writes: I see. Tough problem is: how do we get per backend memory usage accurately? Is it relying on OS specific API the only way? Given all the third-party libraries (perl, python, libxml2, yadda yadda) that can be in use and won't go through palloc, I think that this would have to be done at the OS level to be very meaningful. The other problem is the one Robert touched on: what you actually *want* is something entirely different, namely for the backend to actively try to meet an overall target for its memory usage, rather than having queries fail ungracefully when they hit an arbitrary limit that the planner didn't even know about. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] FSM search modes
On Thu, 2009-10-01 at 11:32 -0500, Kevin Grittner wrote: Either I misunderstand you or I disagree. That does seem to be a common stance, though I will read on. :-) -- Simon Riggs www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] FSM search modes
On Thu, 2009-10-01 at 11:32 -0500, Kevin Grittner wrote: If there's a huge chunk of space near the end, and many many smaller spaces spread throughout, what I'd like is for rows to be placed in those small ones. This would minimize the number of pages to read for queries, and would present some possibility that the rows past the huge chunk might eventually be deleted or non-HOT updated, allowing the bloat to eventually be cleaned up with minimal pain. Yes, as Tom points out, this must be done with bias away from the very end of the table. I meant that we should start from the beginning of large spaces and that we shouldn't assume that all space worth filling is at start of relation. -- Simon Riggs www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Limit allocated memory per session
Tom Lane escreveu: Robert Haas robertmh...@gmail.com writes: What WOULD be useful is to find a way to provide a way to configure work_mem per backend rather than per executor node. But that's a much harder problem. I think it's mostly a planner problem: how do you deal with the fact that that would make cost estimates for different sub-problems interrelated? The cost of a hash, for instance, depends a lot on how much memory you assume it can use. It could introduce some complexity but you could track (subtract) the memory usage as you're walking up the tree. Also, you need to decide what to do when you have more than one node per level. :( How do you deal with priority in this case? -- Euler Taveira de Oliveira http://www.timbira.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] FSM search modes
Simon Riggs si...@2ndquadrant.com writes: Yes, as Tom points out, this must be done with bias away from the very end of the table. I meant that we should start from the beginning of large spaces and that we shouldn't assume that all space worth filling is at start of relation. Right. One of the goals that FSM is trying to meet is ensuring that different backends aren't trying to insert into the same page concurrently, so as to reduce page-level contention. So for example it'd be a bad idea to adopt the really-dumb strategy of searching from the start of the table for each request --- it'd funnel all the backends to the same target page. What we want is a behavior that is randomized but tends to prefer pages towards the start rather than hitting all free pages equally. The btree precedent makes me suspect that quite a weak preference would be sufficient. So for example we might try resetting the search to the start of the relation with probability 0.01. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Rejecting weak passwords
On Thu, Oct 1, 2009 at 17:24, Tom Lane t...@sss.pgh.pa.us wrote: Dave Page dp...@pgadmin.org writes: How will people know how to use it, or that it's even there without at least a note in the docs somewhere? We don't normally document function hooks anywhere but the source code. That's one of the things that makes them low overhead ;-) I agree with the subsequent comments suggesting a sample module that actually does something useful --- although if it's going to link to external code like cracklib, it probably is going to have to be on pgfoundry not in contrib. Why is that? we have plenty of other things in contrib that rely on external code, for example the uuid, xml or ssl stuff. -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Rejecting weak passwords
Magnus Hagander escreveu: On Thu, Oct 1, 2009 at 17:24, Tom Lane t...@sss.pgh.pa.us wrote: Dave Page dp...@pgadmin.org writes: How will people know how to use it, or that it's even there without at least a note in the docs somewhere? We don't normally document function hooks anywhere but the source code. That's one of the things that makes them low overhead ;-) I agree with the subsequent comments suggesting a sample module that actually does something useful --- although if it's going to link to external code like cracklib, it probably is going to have to be on pgfoundry not in contrib. Why is that? we have plenty of other things in contrib that rely on external code, for example the uuid, xml or ssl stuff. Because cracklib is GPL'ed. -- Euler Taveira de Oliveira http://www.timbira.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Rejecting weak passwords
Magnus Hagander mag...@hagander.net writes: On Thu, Oct 1, 2009 at 17:24, Tom Lane t...@sss.pgh.pa.us wrote: I agree with the subsequent comments suggesting a sample module that actually does something useful --- although if it's going to link to external code like cracklib, it probably is going to have to be on pgfoundry not in contrib. Why is that? we have plenty of other things in contrib that rely on external code, for example the uuid, xml or ssl stuff. Well, maybe. I was concerned about availability, portability, license compatibility, and so on. The bar's a lot lower for pgfoundry projects on all those points ... regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Limit allocated memory per session
Alvaro Herrera alvhe...@commandprompt.com writes: Oh, BTW, did anyone get interested in adding the bits to disable the OOM killer for postmaster on the various Linux initscripts? It needs some games with /proc/pid/oom_adj and requires root privileges, but I think an initscript is in an excellent position to do it. I was imagining that this would be something for individual distros to tackle. It's probably not portable enough to go into the contrib/start-scripts examples. On the other hand, it'd make lots of sense to have the Fedora or Debian or whatever scripts do this, since they know what kernel version they're targeting. (If anyone wants to send me the fixes to make Fedora's script do this ...) regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Limit allocated memory per session
* Tom Lane (t...@sss.pgh.pa.us) wrote: I was imagining that this would be something for individual distros to tackle. It's probably not portable enough to go into the contrib/start-scripts examples. On the other hand, it'd make lots of sense to have the Fedora or Debian or whatever scripts do this, since they know what kernel version they're targeting. (If anyone wants to send me the fixes to make Fedora's script do this ...) I'm not exactly keen on Debian init scripts hacking kernel settings. Should it hack up the shared memory numbers too? This is not what I would consider 'init script' material for specific applications. Thanks, Stephen signature.asc Description: Digital signature
Re: [HACKERS] Rejecting weak passwords
On Thu, Oct 1, 2009 at 19:07, Tom Lane t...@sss.pgh.pa.us wrote: Magnus Hagander mag...@hagander.net writes: On Thu, Oct 1, 2009 at 17:24, Tom Lane t...@sss.pgh.pa.us wrote: I agree with the subsequent comments suggesting a sample module that actually does something useful --- although if it's going to link to external code like cracklib, it probably is going to have to be on pgfoundry not in contrib. Why is that? we have plenty of other things in contrib that rely on external code, for example the uuid, xml or ssl stuff. Well, maybe. I was concerned about availability, portability, license compatibility, and so on. The bar's a lot lower for pgfoundry projects on all those points ... Fair enough. And on the licensing issue. That said, it would still be good to have something actually *useful* in contrib. A bit more than just comparing userid and password. Perhaps at least being able to set the min length, and the requirement on having 1 character class? -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Rejecting weak passwords
On Thu, Oct 01, 2009 at 01:07:04PM -0400, Tom Lane wrote: Magnus Hagander mag...@hagander.net writes: On Thu, Oct 1, 2009 at 17:24, Tom Lane t...@sss.pgh.pa.us wrote: I agree with the subsequent comments suggesting a sample module that actually does something useful --- although if it's going to link to external code like cracklib, it probably is going to have to be on pgfoundry not in contrib. Why is that? we have plenty of other things in contrib that rely on external code, for example the uuid, xml or ssl stuff. Well, maybe. I was concerned about availability, portability, license compatibility, and so on. The bar's a lot lower for pgfoundry projects on all those points ... regards, tom lane It has been a while since I last used cracklib, but the interface is generic enough that the sample we would ship in contrib could be trivially adapted to use cracklib. The version we include could just have the (username = password) check or something similar. Regards, Ken -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Limit allocated memory per session
Tom Lane wrote: Alvaro Herrera alvhe...@commandprompt.com writes: Oh, BTW, did anyone get interested in adding the bits to disable the OOM killer for postmaster on the various Linux initscripts? It needs some games with /proc/pid/oom_adj and requires root privileges, but I think an initscript is in an excellent position to do it. I was imagining that this would be something for individual distros to tackle. It's probably not portable enough to go into the contrib/start-scripts examples. Hmm? I think it should be just (as root) if [ -f /proc/$pid_of_postmaster/oom_adj ]; then echo -17 /proc/$pid_of_postmaster/oom_adj fi This is supported from 2.6.11 onwards AFAIK. If the kernel is older than that, the file would not exist and this would be a noop. -- Alvaro Herrerahttp://www.CommandPrompt.com/ The PostgreSQL Company - Command Prompt, Inc. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] CommitFest 2009-09, two weeks on
Michael Meskes írta: On Thu, Oct 01, 2009 at 03:47:07PM +0200, Boszormenyi Zoltan wrote: You're not being fair with me. The dependencies are quite technical. I'm sorry that you interpreted my email this way, it wasn't at all meant to offend you. Please, accept my apologies, I only tried to express my frustration, this is not a good situation for either of us. You were busy with your job and other occupations. We have a serious project going on that depend on ECPG being more compatible to Informix. First, Tom Lane suggested to unify core and ecpg FETCH syntaxes so both will accept optional FROM/IN, which I did. SQLDA support adds new FETCH forms (Informix-specific This is actually one I was talking about. Adding SQLDA *only* seems like an easy one, as soon as it also hits the parser it becomes more complicated. This is not to say that it is not doable, but it wasn't for me with my time constraints. I was just explaining why I didn't delve into these any more so far. This is now clear, I will separate that part out and post it. But this means that the same core grammar parts will be touched twice because two basic things are done there: - optional FROM/IN in core grammar, ECPG grammar updated - name - cursor_name transition in cursor-related statements in the core grammar and the dynamic cursorname patch So, technically dependant patches again, but maybe easier on the reviewer. Would this be accepted this way? Or the two modification washed into one? When I first posted the split patchset, you didn't tell me that the split is no good. I tried everything help I could to explain why I did what. Well actually I did, but that's not the problem here. I have no idea why you are ranting like this just because of me excusing myself for a lack of time. I am very sorry, under stress sometimes I get like a steam engine overloaded. Not a good behaviour at 37, I know. I am very sorry. Also, the current reviewer (Dan Colish) haven't contacted me despite I offered help privately. I can't review my own patches, that's clear. But I can't do anything else to speed review up but to offer my help and wait for the help/explanation request that didn't arrive. Okay, so it's all my fault. Do you feel better now? It's not your fault at all, it's a difficult situation and I had to express it somehow. My tone was not proper. BTW, thanks for adding my small collected knowledge about the ECPG grammar as official documentation. Also, I have sent some independent very small patches, that don't even need much review. One of them (the typo in pgc.l) was already applied and I thank you Michael, but the memory leak fix for two improperly freed numerics is still behind. Please, look at that patch, it should be really obvious. My last commit was one of your patches, Yes, the pgc.l one-liner. Thanks again for committing that. obviously I didn't do anything on ecpg since. I'm not sure what you are trying to tell me, you might want to explain the last two sentences. I thought you forgot that patch, the please, look at that patch was an (I thought) polite request, it's really two one-liners. But keep in mind you are *not* the one to decide how I spend my spare time. Obviously. Please, accept my apologies. Best regards, Zoltán Böszörményi -- Bible has answers for everything. Proof: But let your communication be, Yea, yea; Nay, nay: for whatsoever is more than these cometh of evil. (Matthew 5:37) - basics of digital technology. May your kingdom come - superficial description of plate tectonics -- Zoltán Böszörményi Cybertec Schönig Schönig GmbH http://www.postgresql.at/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Rejecting weak passwords
Tom Lane wrote: Magnus Hagander mag...@hagander.net writes: On Thu, Oct 1, 2009 at 17:24, Tom Lane t...@sss.pgh.pa.us wrote: I agree with the subsequent comments suggesting a sample module that actually does something useful --- although if it's going to link to external code like cracklib, it probably is going to have to be on pgfoundry not in contrib. Why is that? we have plenty of other things in contrib that rely on external code, for example the uuid, xml or ssl stuff. Well, maybe. I was concerned about availability, portability, license compatibility, and so on. The bar's a lot lower for pgfoundry projects on all those points ... Yeah. I don't want to add another external dependency if we can avoid it. A module that depends on another library needs configure support etc. Surely we could provide something at least mildly useful without it getting too complex or depending on an external library. Remember, this is an example, not meant to have all the bells and whistles anyone could want. cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [PATCH] DefaultACLs
Petr Jelinek pjmo...@pjmodos.net writes: because it seems like merging privileges seems to be acceptable for most (although I am not sure I like it, but I don't have better solution for managing conflicts), I changed the patch to do just that. It's not clear to me whether we have consensus on this approach. Last chance for objections, anyone? The main argument I can see against doing it this way is that it doesn't provide a means for overriding the hard-wired public grants for object types that have such (principally functions). I think that a reasonable way to address that issue would be for a follow-on patch that allows changing the hard-wired default privileges for object types. It might well be that no one cares enough for it to matter, though. I think that in most simple cases what's needed is a way to add privileges, not subtract them --- and we're already agreed that this mechanism is only meant to simplify simple cases. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Limit allocated memory per session
Stephen Frost sfr...@snowman.net writes: I'm not exactly keen on Debian init scripts hacking kernel settings. Should it hack up the shared memory numbers too? This is not what I would consider 'init script' material for specific applications. What was suggested was tweaking the oom_adj setting for the postmaster process only, not messing with any system-wide settings. Do you really find that unreasonable? The default OOM killer behavior is just about as unreasonable as can be :-( regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Rejecting weak passwords
Magnus Hagander mag...@hagander.net writes: That said, it would still be good to have something actually *useful* in contrib. A bit more than just comparing userid and password. Perhaps at least being able to set the min length, and the requirement on having 1 character class? +1. There's still the issue of not being able to do much with a pre-MD5'd password, though. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [PATCH] DefaultACLs
* Tom Lane (t...@sss.pgh.pa.us) wrote: Petr Jelinek pjmo...@pjmodos.net writes: because it seems like merging privileges seems to be acceptable for most (although I am not sure I like it, but I don't have better solution for managing conflicts), I changed the patch to do just that. It's not clear to me whether we have consensus on this approach. Last chance for objections, anyone? I don't like it, but at the same time I'd rather have it with this than not have anything. The main argument I can see against doing it this way is that it doesn't provide a means for overriding the hard-wired public grants for object types that have such (principally functions). I think that a reasonable way to address that issue would be for a follow-on patch that allows changing the hard-wired default privileges for object types. It might well be that no one cares enough for it to matter, though. I think that in most simple cases what's needed is a way to add privileges, not subtract them --- and we're already agreed that this mechanism is only meant to simplify simple cases. This doesn't actually address the entire problem. How about schema-level default grants which you want to override with per-role default grants? Or the other way around? Is it always only more permissive with more defaults? Even when the grantee is the same? I dunno, I'll probably just ignore the per-role stuff, personally, but it seems more complex without sufficient definition about what's going to happen in each case. Thanks, Stephen signature.asc Description: Digital signature
Re: [HACKERS] [PATCH] DefaultACLs
Stephen Frost sfr...@snowman.net writes: This doesn't actually address the entire problem. How about schema-level default grants which you want to override with per-role default grants? Or the other way around? Is it always only more permissive with more defaults? Even when the grantee is the same? Well, bear in mind that we're *only* going to allow these things per-role, so as to avoid the problem of translating ACLs to a different grantor. So the main case that's not being solved is I'd like to grant privileges XYZ everywhere except in this schema. I'm willing to write that off as not being within the scope of a simple mechanism. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Use samehost by default in pg_hba.conf?
Tom Lane wrote: Stef Walter stef-l...@memberwebs.com writes: Tom Lane wrote: Now that the samehost/samenet patch is in, I wonder if it wouldn't be a good idea to replace this part of the default pg_hba.conf file: You're probably not suggesting this, but I would be against a default setting of 'samehost' used with 'trust'. Essentially that would be the same as rlogin rsh, where if the user can spoof a TCP connection, he can connect to postgresql. Depending on the platform, an interface may have to be down for this to work. Is there any actual risk here that we aren't taking already just by allowing 127.0.0.1? Yes, RFC 1122 specifies that 127.0.0.0/8 must not appear outside of a single host. Checks for this are implemented. For example in FreeBSD: http://fxr.watson.org/fxr/source/netinet/ip_input.c#L440 http://fxr.watson.org/fxr/search?string=IN_LOOPBACK Obviously I can't guarantee this for any given OS. However it does seem like a pretty standard security feature of the IP stack. In my opinion, adding samehost into the default pg_hba.conf with the method set to md5 (attached patch) would be both usable and secure. Cheers, Stef diff --git a/src/backend/libpq/pg_hba.conf.sample b/src/backend/libpq/pg_hba.conf.sample index cfcd246..ea83d4d 100644 *** a/src/backend/libpq/pg_hba.conf.sample --- b/src/backend/libpq/pg_hba.conf.sample *** *** 75,77 --- 75,79 hostall all 127.0.0.1/32 @authmethod@ # IPv6 local connections: hostall all ::1/128 @authmethod@ + # Host's IPv4 and IPv6 connections: + hostall all samehost md5 -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] CommitFest 2009-09, two weeks on
On Thu, Oct 01, 2009 at 07:21:54PM +0200, Boszormenyi Zoltan wrote: Please, accept my apologies, I only tried to express my frustration, this is not a good situation for either of us. Apologies accepted, email is a difficult means of communication anyway. It leads to misunderstanding IMO. You were busy with your job and other occupations. We have a serious project going on that depend on ECPG being more compatible to Informix. Please keep in mind that the needs of your business project cannot and will not influence the way PostgreSQL as on OSS project will work. Would this be accepted this way? Or the two modification washed into one? It is accepted either way. I was just pointing out that it might be easier to review/commit at least parts of your patches if they can be applied seperately. I thought you forgot that patch, the please, look at that patch was an (I thought) polite request, it's really two one-liners. Well, this is true as the patch was buried in the long thread containing all the other ones. And yes, now that you brought it into my memory again, I already committed it. Sorry for missing it. Michael -- Michael Meskes Michael at Fam-Meskes dot De, Michael at Meskes dot (De|Com|Net|Org) Michael at BorussiaFan dot De, Meskes at (Debian|Postgresql) dot Org ICQ: 179140304, AIM/Yahoo/Skype: michaelmeskes, Jabber: mes...@jabber.org Go VfL Borussia! Go SF 49ers! Use Debian GNU/Linux! Use PostgreSQL! -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [PATCH] DefaultACLs
* Tom Lane (t...@sss.pgh.pa.us) wrote: Stephen Frost sfr...@snowman.net writes: This doesn't actually address the entire problem. How about schema-level default grants which you want to override with per-role default grants? Or the other way around? Is it always only more permissive with more defaults? Even when the grantee is the same? Well, bear in mind that we're *only* going to allow these things per-role, so as to avoid the problem of translating ACLs to a different grantor. So the main case that's not being solved is I'd like to grant privileges XYZ everywhere except in this schema. I'm willing to write that off as not being within the scope of a simple mechanism. Erm, wait, we're going to drop the only piece of this that outside folks have actually been asking for? Specifically, having per-schema default ACLs? Big -1 for even bothering to add the complexity if we're not going to address what end users are actually looking for. Perhaps I missed where the issue with assigning the right grantor was, but that feels very much like an implementation detail we can certainly solve and document which way we decided to solve it (either schema owner, which would be my preference, or object creator, which would be acceptable as well). That's my thoughts on it. Thanks, Stephen signature.asc Description: Digital signature
Re: [HACKERS] [PATCH] DefaultACLs
Stephen Frost sfr...@snowman.net writes: Erm, wait, we're going to drop the only piece of this that outside folks have actually been asking for? Specifically, having per-schema default ACLs? They are per-schema for the objects belonging to the granting user. Otherwise you have a bunch of nasty issues, including the prospect of non-superusers being able to control the privileges granted on objects that don't belong to them. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [PATCH] DefaultACLs
Tom Lane wrote: Stephen Frost sfr...@snowman.net writes: Erm, wait, we're going to drop the only piece of this that outside folks have actually been asking for? Specifically, having per-schema default ACLs? They are per-schema for the objects belonging to the granting user. Otherwise you have a bunch of nasty issues, including the prospect of non-superusers being able to control the privileges granted on objects that don't belong to them. Also it's not really a problem since superuser or role admin can set these for other roles. -- Regards Petr Jelinek (PJMODOS)
Re: [HACKERS] FSM search modes
Tom Lane t...@sss.pgh.pa.us wrote: Simon Riggs si...@2ndquadrant.com writes: Yes, as Tom points out, this must be done with bias away from the very end of the table. I meant that we should start from the beginning of large spaces and that we shouldn't assume that all space worth filling is at start of relation. OK, so I did misunderstand you; we agree after all. :-) So for example we might try resetting the search to the start of the relation with probability 0.01. If I understand the heuristic you propose, and my math skill haven't eroded too badly from lack of use, every 229 spots considered would cause a 90% chance of reset. That means that the odds of getting past 50,000 spots (the number of pages with available free space at which I generally start to get worried) without resetting is about 1 in 10^218 -- which is a risk I'm willing to accept. ;-) -Kevin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] FSM search modes
Kevin Grittner kevin.gritt...@wicourts.gov writes: Tom Lane t...@sss.pgh.pa.us wrote: So for example we might try resetting the search to the start of the relation with probability 0.01. If I understand the heuristic you propose, and my math skill haven't eroded too badly from lack of use, every 229 spots considered would cause a 90% chance of reset. Sorry, I wasn't clear. What I was thinking of was that we'd consider resetting the search position once, upon entry to fsm_search, and then search normally thereafter. Some experimentation would be needed to choose the right probability of course. A number like 0.01 might seem too small to affect the behavior at all, but that's what we thought about the btree case too. A very light thumb upon the scales may be sufficient. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] CommitFest 2009-09, two weeks on
Michael Meskes írta:
On Thu, Oct 01, 2009 at 07:21:54PM +0200, Boszormenyi Zoltan wrote:
Please, accept my apologies, I only tried to express my
frustration, this is not a good situation for either of us.
Apologies accepted, email is a difficult means of communication anyway. It
leads to misunderstanding IMO.
You were busy with your job and other occupations.
We have a serious project going on that depend on
ECPG being more compatible to Informix.
Please keep in mind that the needs of your business project cannot and will
not
influence the way PostgreSQL as on OSS project will work.
Yes, but technical problems and solutions do. ECPG claims
to be ESQL/C compatible, but at places it's only half compatible.
For our project to succeed, we need more compatibility in ECPG.
It's easier to solve these problems in ECPG than to code around it
in literally thousands of little programs.
BTW, a thought about the comment in ecpg.header about adjust_informix():
/* Informix accepts DECLARE with variables that are out of scope
when OPEN is called.
* for instance you can declare variables in a function, and
then subsequently use them
* {
* declare_vars();
* exec sql ... which uses vars declared in the above function
*
* This breaks standard and leads to some very dangerous
programming.
This comment is misleading and reflects quite a narrow POV.
Not only OPEN and DECLARE may be out of scope,
but FETCH and CLOSE as well. The reason why ESQL/C
allows this construct is that this ultimately allows using
embedded SQL in event-driven code in a straightforward way.
For this purpose, native ECPG code is not usable currently,
or you need programming tricks, like tracking whether the
cursor is open and protecting DECLARE and OPEN under
some conditional branch to avoid double open, etc. A straight
DECLARE, OPEN, FETCH(s) and CLOSE series in
the same function is only good for batch programming.
Would this be accepted this way? Or the two modification washed into one?
It is accepted either way. I was just pointing out that it might be easier to
review/commit at least parts of your patches if they can be applied
seperately.
Okay, I will split the remaining patches into more little pieces
that can reviewed more easily. Some patches will still build
on earlier ones in the series, that's unavoidable.
I thought you forgot that patch, the please, look at that patch
was an (I thought) polite request, it's really two one-liners.
Well, this is true as the patch was buried in the long thread containing all
the other ones. And yes, now that you brought it into my memory again, I
already committed it. Sorry for missing it.
Thank you very much for committing it.
Best regards,
Zoltán Böszörményi
--
Bible has answers for everything. Proof:
But let your communication be, Yea, yea; Nay, nay: for whatsoever is more
than these cometh of evil. (Matthew 5:37) - basics of digital technology.
May your kingdom come - superficial description of plate tectonics
--
Zoltán Böszörményi
Cybertec Schönig Schönig GmbH
http://www.postgresql.at/
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] make install now tries to build the documentation
Peter Eisentraut wrote: I'm not exactly sure what the goal is, however. You built the documentation at some point. Then it gets updated when necessary. You can delete the documentation by running make -C doc maintainer-clean. Then it's gone forever and never reappears unless you explicitly build it again. I think this bit is missing something; the html-stamp file is created on the builddir, but maintainer-clean is trying to delete it from the source dir. -- Alvaro Herrerahttp://www.CommandPrompt.com/ The PostgreSQL Company - Command Prompt, Inc. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] FSM search modes
Kevin Grittner wrote: Tom Lane t...@sss.pgh.pa.us wrote: Simon Riggs si...@2ndquadrant.com writes: Yes, as Tom points out, this must be done with bias away from the very end of the table. I meant that we should start from the beginning of large spaces and that we shouldn't assume that all space worth filling is at start of relation. OK, so I did misunderstand you; we agree after all. :-) So for example we might try resetting the search to the start of the relation with probability 0.01. If I understand the heuristic you propose, and my math skill haven't eroded too badly from lack of use, every 229 spots considered would cause a 90% chance of reset. That means that the odds of getting past 50,000 spots (the number of pages with available free space at which I generally start to get worried) without resetting is about 1 in 10^218 -- which is a risk I'm willing to accept. ;-) The FSM currently uses a clock sweep kind of algorithm to hand out pages, and the clock hand is reset to 0 at every vacuum. The purpose of resetting the clock hand is precisely to bias towards lower-numbered pages, to allow truncation later on. That's a bit of an over-simplification, though, there's actually a separate clock hand on each FSM page (next_slot pointer). We probably could do with more bias. For example, we still prefer pages close to the page we last inserted to, by searching for free space in the same FSM page first, before starting the search from the top of the tree. And of course, each backend tries first to insert to the last page it inserted to before even consulting the FSM. A mode to disable those behaviors might indeed be useful, along with the random reset. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Limit allocated memory per session
On Thu, Oct 01, 2009 at 10:35:55AM -0400, Tom Lane wrote: daveg da...@sonic.net writes: I'd like to propose adding a new GUC to limit the amount of memory a backend can allocate for its own use. Use ulimit. That was my initial thought too. However, ulimit() is documented as superceded by setrlimit(). Which has the option RLIMIT_DATA to limit the size of the data segment. Perfect! Except, RLIMIT_DATA does not appear to work on linux. The call succeeds and the new value can even be read back with getrlimit(), but it does not seem to do anything to actually limit the memory allocated. I tested this on SuSE 11: kernel 2.6.25, and Ubuntu Intrepid: kernel 2.6.28. Setting RLIMIT_AS to limit the total address space for a process works as expected. However this seems undesireable for postgresql as it can also cause stack expansion to fail, which would then force a general restart. Also, this limit would interact with the buffercache size setting as it includes the shared address space as well. -dg -- David Gould da...@sonic.net 510 536 1443510 282 0869 If simplicity worked, the world would be overrun with insects. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] FSM search modes
Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: Kevin Grittner wrote: Tom Lane t...@sss.pgh.pa.us wrote: So for example we might try resetting the search to the start of the relation with probability 0.01. If I understand the heuristic you propose, and my math skill haven't eroded too badly from lack of use, every 229 spots considered would cause a 90% chance of reset. That means that the odds of getting past 50,000 spots (the number of pages with available free space at which I generally start to get worried) without resetting is about 1 in 10^218 -- which is a risk I'm willing to accept. ;-) The FSM currently uses a clock sweep kind of algorithm to hand out pages, and the clock hand is reset to 0 at every vacuum. The purpose of resetting the clock hand is precisely to bias towards lower-numbered pages, to allow truncation later on. That's a bit of an over-simplification, though, there's actually a separate clock hand on each FSM page (next_slot pointer). We probably could do with more bias. For example, we still prefer pages close to the page we last inserted to, by searching for free space in the same FSM page first, before starting the search from the top of the tree. And of course, each backend tries first to insert to the last page it inserted to before even consulting the FSM. A mode to disable those behaviors might indeed be useful, along with the random reset. That flushes out the descriptions given by Tom, but doesn't really make me think I misunderstood the heuristic or miscalculated the odds of getting far from the start with a 1% probability of a reset on each advance of the sweep. (Of course, it's possible I'm being dense and taking statements to mean what I expect, despite evidence to the contrary.) The way I figure it, if there is a 0.01 chance to reset the sweep, then there's a 0.99 percent chance to continue the sweep from the last position. 0.99^229 is about 0.1, which means there is a 10% chance not to have reset after that many attempts to advance. Every 229 attempts after that multiplies by 0.1, too. So, after 229*6 attempts (for example) the odds of not having reset are about one in a million. Am I saying something stupid here? (I used to be good at these sorts of calculations) -Kevin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Limit allocated memory per session
On Thu, Oct 01, 2009 at 11:47:43AM -0400, Tom Lane wrote: Euler Taveira de Oliveira eu...@timbira.com writes: Tom Lane escreveu: daveg da...@sonic.net writes: I'd like to propose adding a new GUC to limit the amount of memory a backend can allocate for its own use. Use ulimit. Seriously, the proposed patch introduces overhead into a place that is already a known hot spot, in return for not much of anything. It will The overhead is simply an integer addition and compare with values that are likely already in processor caches. And this only occurs when we actually call malloc() to get a new block, not on every palloc. So I suspect it will not be noticable. However, I welcome any suggestion on how to test this and actually measure the overhead if any. pg_bench? Something else? *not* bound backend memory use very accurately, because there is no way to track raw malloc() calls. And I think that 99% of users will not find it useful. The use case that motivated is a client that runs many postgresql instances with a mostly batch/large query workload. Some of the queries are code generated by an application and can be very complex. A few times a month one of these will run through 64GB of memory and oom the host. So it seriously hurts production. Setting work_mem low enough to prevent this results in poor query performance. This client does not use any outside libraries that call malloc() directly. -dg -- David Gould da...@sonic.net 510 536 1443510 282 0869 If simplicity worked, the world would be overrun with insects. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] FSM search modes
Kevin Grittner kevin.gritt...@wicourts.gov wrote: 0.99 percent chance to continue the sweep Make that a 99% chance, or a 0.99 chance (in case the typo was not apparent). Am I saying something stupid here? Well, besides that line? -Kevin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] FSM search modes
On Thu, Oct 1, 2009 at 3:23 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: We probably could do with more bias. For example, we still prefer pages close to the page we last inserted to, by searching for free space in the same FSM page first, before starting the search from the top of the tree. And of course, each backend tries first to insert to the last page it inserted to before even consulting the FSM. A mode to disable those behaviors might indeed be useful, along with the random reset. The elephant in the room here is that if the relation is a million pages of which 1-100,000 and 1,000,000 are in use, no amount of bias is going to help us truncate the relation unless every tuple on page 1,000,000 gets updated or deleted. If the user doesn't have any interest in doing anything to those tuples, nothing will ever happen. And this is likely to come up a lot, because many tables have a core of data that is rarely or never updated. For example, imagine that 90% of pages contain data that is updated regularly and 10% of pages contain data that is rarely or never updated. If these are distributed uniformly, we figure to hit one of the latter type within 10 pages or so, and then we're stymied. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] FSM search modes
Kevin Grittner kevin.gritt...@wicourts.gov writes: The way I figure it, if there is a 0.01 chance to reset the sweep, then there's a 0.99 percent chance to continue the sweep from the last position. 0.99^229 is about 0.1, which means there is a 10% chance not to have reset after that many attempts to advance. Right, so the odds would be that a backend will confine its insertion attempts to the first 229 pages containing a usable amount of free space. As long as the number of backends concurrently inserting into the relation is well under 229, this seems perfectly fine. (Hm, so we might want to make the probability depend on max_connections?) A possible downside of keeping things compact this way is that you'd probably get a longer average search distance because of all the early pages tending to remain full. Maybe what we want is some bias against inserting in the last half or quarter of the table, or some such rule, rather than necessarily heading for the start of the relation. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] TODO item: Allow more complex user/database default GUC settings
--On 30. September 2009 13:19:53 -0400 Alvaro Herrera alvhe...@commandprompt.com wrote: I think it would be helpful if you could post ONE patch with all the changes and all the new files in the diff. AIUI, the patch is now split across three separate emails. :-( That's correct, here it is. Some additional notes: - ALTER ROLE ... IN DATABASE is missing some documentation. If you want, i can work on this. - The patch as is has still some locking problems (AlterRoleSet() has a XXX about that): I've managed to create dead entries for a role or a database in pg_db_role_setting while altering and dropping a role/database in two concurrent sessions. -- Thanks Bernd -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] FSM search modes
Robert Haas robertmh...@gmail.com writes: The elephant in the room here is that if the relation is a million pages of which 1-100,000 and 1,000,000 are in use, no amount of bias is going to help us truncate the relation unless every tuple on page 1,000,000 gets updated or deleted. Well, there is no way to move a tuple across pages in a user-invisible, non-blocking fashion, so our ability to do something automatic about the above scenario is limited. The discussion at the moment is about ways of reducing the probability of getting into that situation in the first place. That doesn't preclude also providing some more-invasive tools that people can use when they do get into that situation; but let's not let I-want-a-magic-pony syndrome prevent us from doing anything at all. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] FSM search modes
On Thu, Oct 1, 2009 at 4:47 PM, Tom Lane t...@sss.pgh.pa.us wrote: A possible downside of keeping things compact this way is that you'd probably get a longer average search distance because of all the early pages tending to remain full. Maybe what we want is some bias against inserting in the last half or quarter of the table, or some such rule, rather than necessarily heading for the start of the relation. Yeah, that seems better. Avoiding the last quarter seems like it would likely be sufficient. If it's not too hard, it might make sense to only apply this bias when we know that the table is already somewhat bloated. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] FSM search modes
On Thu, Oct 1, 2009 at 5:08 PM, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: The elephant in the room here is that if the relation is a million pages of which 1-100,000 and 1,000,000 are in use, no amount of bias is going to help us truncate the relation unless every tuple on page 1,000,000 gets updated or deleted. Well, there is no way to move a tuple across pages in a user-invisible, non-blocking fashion, so our ability to do something automatic about the above scenario is limited. The discussion at the moment is about ways of reducing the probability of getting into that situation in the first place. That doesn't preclude also providing some more-invasive tools that people can use when they do get into that situation; but let's not let I-want-a-magic-pony syndrome prevent us from doing anything at all. That's fair enough, but it's our usual practice to consider, before implementing a feature or code change, what fraction of the people it will actually help and by how much. If there's a way that we can improve the behavior of the system in this area, I am all in favor of it, but I have pretty modest expectations for how much real-world benefit will ensue. I suspect that it's pretty common for large tables to contain a core of infrequently-updated records, and even a very light smattering of those, distributed randomly, will be enough to stop table shrinkage before it can get very far. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] TODO item: Allow more complex user/database default GUC settings
Bernd Helmle escribió: --On 30. September 2009 13:19:53 -0400 Alvaro Herrera alvhe...@commandprompt.com wrote: I think it would be helpful if you could post ONE patch with all the changes and all the new files in the diff. AIUI, the patch is now split across three separate emails. :-( That's correct, here it is. Some additional notes: - ALTER ROLE ... IN DATABASE is missing some documentation. If you want, i can work on this. Please. - The patch as is has still some locking problems (AlterRoleSet() has a XXX about that): I've managed to create dead entries for a role or a database in pg_db_role_setting while altering and dropping a role/database in two concurrent sessions. Yeah, I was playing with that too. I think we need a few extra LockSharedObject calls, and not only in the new code :-( (This troubles me in the case of databases, because we already grab a lock on it during connection establishing, so this could cause extra contention there.) -- Alvaro Herrerahttp://www.CommandPrompt.com/ PostgreSQL Replication, Consulting, Custom Development, 24x7 support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] FSM search modes
Robert Haas robertmh...@gmail.com wrote: The elephant in the room here is that if the relation is a million pages of which 1-100,000 and 1,000,000 are in use, no amount of bias is going to help us truncate the relation unless every tuple on page 1,000,000 gets updated or deleted. Perhaps bias, combined with a client utility to force non-HOT updates of some rows at the end of the table would provide the desired behavior. (It'd be nice if that could be built in to vacuum, but if it's not feasible, a separate utility is workable.) Off the top of my head, I might set up a routine crontab job for most databases to do that to the lesser of 1% of the rows or a number of rows which matches how far the pages with free space exceed 10% of total pages. That seems like it should contain things for most circumstances without getting too extreme. One could always run the utility manually to correct more extreme bloat. Some sort of delay (similar to what vacuum can do) to prevent tanking performance would be good. We wouldn't care about the occasional update conflict -- we expect that using a relational database means dealing with serialization failures. We'd be more than happy to accept a few of these in exchange for keeping performance optimal. If your software is designed to reschedule transactions which are rolled back for serialization failures, they are just another performance cost, so it's pretty easy to balance one cost against another. -Kevin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] FSM search modes
Tom Lane t...@sss.pgh.pa.us writes: That doesn't preclude also providing some more-invasive tools that people can use when they do get into that situation; About this side of things, what about having any query which asks for system columns (any of them) take a specific (new) exclusive row level lock. The tuple moving utility would try to take the same lock before moving any tuple, thus allowing those who need it to continue relying on the ctid / xmin queries trick. If that idea has any chance to fly, the tuple moving facility could even be integrated into autovacuum for actively self-healing PostgreSQL. Regards, -- dim Hoping this won't be some more hand waving. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
