Re: [HACKERS] Partitioning syntax
Robert Haas robertmh...@gmail.com wrote: I've taken a little bit more of a look at this patch and I guess I'm not too happy with the design. Thanks. I was thinking about only syntax for partitioning in the patch, but I need more consideration about insert-aware catalog design. 5. The use of the term partition is not very consistent. For example, we use CREATE PARTITION to create a partition, but we use DROP TABLE to get rid of it (there is no DROP PARTITION). I think that the right syntax to use here is ALTER TABLE ... ADD/DROP PARTITION; both Oracle and MySQL do it that way. And meanwhile OCLASS_PARTITION means the partitioning information associated with the parent table, not a partition of a parent table. ALTER TABLE ... ADD/DROP PARTITION was discussed many times, but I cannot solve syntax confict with ALTER TABLE ... ADD [COLUMN]. Since we can omit COLUMN, parser treats ADD PARTITION as adding a column named PARTITION. We need to add PARTITION into the reserved keyword list to avoid shift/reduce errors. Do you have any better idea? Regards, --- Takahiro Itagaki NTT Open Source Software Center -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] Invitation to connect on LinkedIn
LinkedIn Gourish Singbal souhaite se connecter à vous sur LinkedIn : -- Amine, I'd like to add you to my professional network on LinkedIn. - Gourish Accepter l'invitation de Gourish Singbal http://www.linkedin.com/e/cm7uxn-gbbsl8em-2n/VWCdriNHrcXMfYpwEG6pcCKmodzjIoHmhgnzwrRR5NC/blk/I70409762_6/6lColZJrmZznQNdhjRQnOpBtn9QfmhBt71BoSd1p65Lr6lOfPpvczoTej0Qc3t9bQh5h5Z4ck5pbPkVc3cOejgQd34LrCBxbOYWrSlI/EML_comm_afe/ Voir l'invitation de Gourish Singbal http://www.linkedin.com/e/cm7uxn-gbbsl8em-2n/VWCdriNHrcXMfYpwEG6pcCKmodzjIoHmhgnzwrRR5NC/blk/I70409762_6/dBYOdzsVc3gMdQALqnpPbOYWrSlI/svi/ -- SAVEZ-VOUS que vous pouvez utiliser votre profil LinkedIn comme votre page web personnelle ? Personnalisez l'adresse web de votre profil et utilisez-la sur vos cartes de visite, comme signature d'e-mail, sur votre site, etc. http://www.linkedin.com/e/cm7uxn-gbbsl8em-2n/ewp/inv-21/ -- (c) 2010, LinkedIn Corporation
[HACKERS] Python Interface Hacking
Hi all, I joined this list under the mis-impression that it was about hacking the Python interfaces to pgsql. Is there possibly another list for that? Or is the Python stuff just mixed in with all the rest? Sorry for the meta-question... Cheers, Peter -- Peter H. Froehlich http://www.cs.jhu.edu/~phf/ Senior Lecturer | Director, Johns Hopkins Gaming Lab -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: I: [HACKERS] About Our CLUSTER implementation is pessimal patch
Leonardo F m_li...@yahoo.it wrote:
I saw that you also changed the writing:
(snip)
Are we sure it's 100% equivalent?
I think writetup_rawheap() and readtup_rawheap() are a little complex,
but should work as long as there are no padding between t_len and t_self
in HeapTupleData struct.
- It might be cleaner if you write the total item length
and tuple data separately.
- (char *) tuple + sizeof(tuplen) might be more robust
than tuple-t_self.
Here is a sample code. writetup() and readtup() will be alike.
BTW, we could have LogicalTapeReadExact() as an alias of
LogicalTapeRead() and checking the result because we have
many duplicated codes for unexpected end of data errors.
static void
writetup_rawheap(Tuplesortstate *state, int tapenum, SortTuple *stup)
{
HeapTuple tuple = (HeapTuple) stup-tuple;
int tuplen = tuple-t_len + HEAPTUPLESIZE;
LogicalTapeWrite(state-tapeset, tapenum,
tuplen, sizeof(tuplen));
LogicalTapeWrite(state-tapeset, tapenum,
(char *) tuple + sizeof(tuplen),
HEAPTUPLESIZE - sizeof(tuplen);
LogicalTapeWrite(state-tapeset, tapenum, tuple-t_data, tuple-t_len);
if (state-randomAccess)/* need trailing length word? */
LogicalTapeWrite(state-tapeset, tapenum, tuplen,
sizeof(tuplen));
FREEMEM(state, GetMemoryChunkSpace(tuple));
heap_freetuple(tuple);
}
static void
readtup_rawheap(Tuplesortstate *state, SortTuple *stup,
int tapenum, unsigned int tuplen)
{
HeapTuple tuple = (HeapTuple) palloc(tuplen);
USEMEM(state, GetMemoryChunkSpace(tuple));
tuple-t_len = tuplen - HEAPTUPLESIZE;
if (LogicalTapeRead(state-tapeset, tapenum,
(char *) tuple +
sizeof(tuplen),
HEAPTUPLESIZE - sizeof(tuplen)) != HEAPTUPLESIZE -
sizeof(tuplen))
elog(ERROR, unexpected end of data);
tuple-t_data = (HeapTupleHeader) ((char *) tuple + HEAPTUPLESIZE);
if (LogicalTapeRead(state-tapeset, tapenum,
tuple-t_data, tuple-t_len) !=
tuple-t_len)
elog(ERROR, unexpected end of data);
if (state-randomAccess)/* need trailing length word? */
if (LogicalTapeRead(state-tapeset, tapenum, tuplen,
sizeof(tuplen)) !=
sizeof(tuplen))
elog(ERROR, unexpected end of data);
Regards,
---
Takahiro Itagaki
NTT Open Source Software Center
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Proposal for 9.1: WAL streaming from WAL buffers
Robert Haas robertmh...@gmail.com writes: If it's unsafe to send written but unflushed WAL to the standby, then for the same reasons we can't send unwritten WAL either. [...] Having said that, I do think we urgently need some high-level design discussion on how sync rep is actually going to handle this issue Stop me if I'm all wrong already, but I though we said that we should handle this case by decoupling what we can send to the standby and what it can apply. We could do this by sending the current WAL fsync'ed position on the master in the WAL sender protocol, either in the WAL itself or as out-of-bound messages, I guess. Now, this can be made safe, how to make it fast (low-latency) is yet to be addressed. Regards, -- dim -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] cvs to git migration - keywords
In the previous discussions of how to migrate from cvs to git, we've all agreed we should kill the keyword expansion that we have now. I don't think, however, that we ever decided what to do with the *old* keywords. We did say we want to be able to reproduce backbranches/tags *identically* to what they are now, which indicates we need to leave the keywords in for those. That has other drawbacks, though. The way I see it, we have two ways to do it: 1) We can migrate the repository with the keywords, and then make one big commit just after (or before, that doesn't make a difference) removing them. In this case, backbranches and tags look exactly like they do now, but it also means if you do git diff between old versions, the keywords will show up there. 2) We can filter out that row during the conversion, so they look like they never existed.That means that if you check out 7.4.3 or whatever fro git, it will look like the keyword lines never existed. Since they're in comments it shouldn''t affect functionality, but it does mean that we are *not* keeping history unmodified. The advantage is that git diff on and between old revision won't include the keyword changes, of course. #1 is most likely the easiest one. It really comes down to which is most important - being able to get easy to use diffs between old revisions, or keeping history intact. Obviously, for all *new* commits, either one of these two methods will make the diffs readable. And if they are new commits, well, they are by definition not history that needs to be kept :-) Thoughts? -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] cvs to git migration - keywords
On Wed, Jul 7, 2010 at 10:01 AM, Magnus Hagander mag...@hagander.net wrote: 1) We can migrate the repository with the keywords, and then make one big commit just after (or before, that doesn't make a difference) removing them. In this case, backbranches and tags look exactly like they do now, but it also means if you do git diff between old versions, the keywords will show up there. 2) We can filter out that row during the conversion, so they look like they never existed.That means that if you check out 7.4.3 or whatever fro git, it will look like the keyword lines never existed. Since they're in comments it shouldn''t affect functionality, but it does mean that we are *not* keeping history unmodified. The advantage is that git diff on and between old revision won't include the keyword changes, of course. #1 is most likely the easiest one. +1 for #1. Changing history and the resulting possibility of becoming one's own grandfather always makes me nervous. -- Dave Page EnterpriseDB UK: http://www.enterprisedb.com The Enterprise Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Proposal for 9.1: WAL streaming from WAL buffers
On Wed, Jul 7, 2010 at 4:40 AM, Dimitri Fontaine dfonta...@hi-media.com wrote: Stop me if I'm all wrong already, but I though we said that we should handle this case by decoupling what we can send to the standby and what it can apply. We could do this by sending the current WAL fsync'ed position on the master in the WAL sender protocol, either in the WAL itself or as out-of-bound messages, I guess. Now, this can be made safe, how to make it fast (low-latency) is yet to be addressed. Yeah, that's the trick, isn't it? -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Python Interface Hacking
On Wed, Jul 7, 2010 at 3:07 AM, Peter Froehlich peter.hans.froehl...@gmail.com wrote: I joined this list under the mis-impression that it was about hacking the Python interfaces to pgsql. Is there possibly another list for that? Or is the Python stuff just mixed in with all the rest? Sorry for the meta-question... Well, PL/python (using Python within the database) would be on-topic for this list, but the connectors like psycopg are not discussed here. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Python Interface Hacking
On ons, 2010-07-07 at 03:07 -0400, Peter Froehlich wrote: I joined this list under the mis-impression that it was about hacking the Python interfaces to pgsql. Is there possibly another list for that? Or is the Python stuff just mixed in with all the rest? Sorry for the meta-question... If you want to hack PL/Python, which is a Python interpreter embedded into the PostgreSQL server, then this is the right place. (Yes, it's mixed with all the rest.) If you want to hack a Python client driver, then go to http://initd.org/psycopg/. (There are others, but that's the most popular one.) -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Partitioning syntax
On Wed, Jul 7, 2010 at 2:14 AM, Takahiro Itagaki itagaki.takah...@oss.ntt.co.jp wrote: 5. The use of the term partition is not very consistent. For example, we use CREATE PARTITION to create a partition, but we use DROP TABLE to get rid of it (there is no DROP PARTITION). I think that the right syntax to use here is ALTER TABLE ... ADD/DROP PARTITION; both Oracle and MySQL do it that way. And meanwhile OCLASS_PARTITION means the partitioning information associated with the parent table, not a partition of a parent table. ALTER TABLE ... ADD/DROP PARTITION was discussed many times, but I cannot solve syntax confict with ALTER TABLE ... ADD [COLUMN]. Since we can omit COLUMN, parser treats ADD PARTITION as adding a column named PARTITION. We need to add PARTITION into the reserved keyword list to avoid shift/reduce errors. Do you have any better idea? No, I think we're going to need to at least partially reserve that keyword. However, SQL:2003 and SQL:2008 apparently have it as a reserved keyword, so I'm hoping we can get away with that. I don't think it's worth inventing a totally different (and, IMHO, not very appealing) syntax just to avoid reserving a keyword that is reserved in the standard. http://www.postgresql.org/docs/current/static/sql-keywords-appendix.html -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] keepalive in libpq using
On Tue, Jul 6, 2010 at 1:08 PM, Pavel Golub pa...@microolap.com wrote: While I'm very excited about enabling keepalives in libpq, I want to know how can I use this functionality in my application? Let's imagine that I connect to a server with keepalives option, other options (keepalives_idle, keepalives_interval, keepalives_count) are used either. Then network goes down. So, how will I know that connection is dead? Any callback function? Or should I check PQstatus periodically? I'm not sure, exactly. I think what'll happen is that if you're trying to read data from the remote server, the connection will eventually break instead of hanging forever, but I'm not exactly sure what that'll look like at the libpq level. I'm not sure what effect it'll have on an idle connection. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Keepalives win32
On Wed, Jun 30, 2010 at 17:46, Tom Lane t...@sss.pgh.pa.us wrote: Kevin Grittner kevin.gritt...@wicourts.gov writes: I also think we may want to suggest that for most environments, people may want to change these settings to something more aggressive, like a 30 to 120 second initial delay, with a 10 or 20 second retry interval. The RFC defaults seem approximately right for a TCP connection to a colony on the surface of the moon, where besides the round trip latency of 2.5 seconds they might have to pay by the byte. Well, the RFCs were definitely written at a time when bandwidth was a lot more expensive than it is today. In other words, it is *so* conservative that I have trouble seeing it ever causing a problem compared to not having keepalive enabled, but it will eventually clean things up. Yes. This is a large part of the reason why I think it's okay for us to turn libpq keepalive on by default in 9.0 --- the default parameters for it are so conservative as to be unlikely to cause trouble. If Windows isn't using RFC-equivalent default parameters, that seems like a good reason to disregard the system settings and force use of the RFC values as defaults. Here's an updated version of the patch, which includes server side functionality. I took out the code that tried tobe smart. It'll now set them to 2 hours/1 second by default. I looked quickly at the RFC and didn't find the exact values there, so those values are the documented out-of-the-box defaults on Windows. I can easily change them to RFC values if someone can find them for me :) It's also merged with roberts macos patch, since they were conflicting. Doc changes not included, but I'll get those in before commit. Comments? -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/ win32keepalive.patch Description: Binary data -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Keepalives win32
On Wed, Jul 7, 2010 at 9:20 AM, Magnus Hagander mag...@hagander.net wrote: On Wed, Jun 30, 2010 at 17:46, Tom Lane t...@sss.pgh.pa.us wrote: Kevin Grittner kevin.gritt...@wicourts.gov writes: I also think we may want to suggest that for most environments, people may want to change these settings to something more aggressive, like a 30 to 120 second initial delay, with a 10 or 20 second retry interval. The RFC defaults seem approximately right for a TCP connection to a colony on the surface of the moon, where besides the round trip latency of 2.5 seconds they might have to pay by the byte. Well, the RFCs were definitely written at a time when bandwidth was a lot more expensive than it is today. In other words, it is *so* conservative that I have trouble seeing it ever causing a problem compared to not having keepalive enabled, but it will eventually clean things up. Yes. This is a large part of the reason why I think it's okay for us to turn libpq keepalive on by default in 9.0 --- the default parameters for it are so conservative as to be unlikely to cause trouble. If Windows isn't using RFC-equivalent default parameters, that seems like a good reason to disregard the system settings and force use of the RFC values as defaults. Here's an updated version of the patch, which includes server side functionality. I took out the code that tried tobe smart. It'll now set them to 2 hours/1 second by default. I looked quickly at the RFC and didn't find the exact values there, so those values are the documented out-of-the-box defaults on Windows. I can easily change them to RFC values if someone can find them for me :) It's also merged with roberts macos patch, since they were conflicting. Doc changes not included, but I'll get those in before commit. Comments? Looks generally OK, though my knowledge of Windows is pretty limited. We'd better get this committed PDQ if it's going into beta3, else there won't be a full buildfarm cycle before we wrap. (BTW, there are two buildfarm machines - wigeon and orangutan - that are consistently failing with rather bizarre error messages. Are these real failures or are those machines just messed up?) -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Keepalives win32
Magnus Hagander mag...@hagander.net wrote: It'll now set them to 2 hours/1 second by default. I looked quickly at the RFC and didn't find the exact values there, so those values are the documented out-of-the-box defaults on Windows. I can easily change them to RFC values if someone can find them for me :) The RFC specifies 2 hours/75 seconds/9 tries. Even though we can't reasonably adjust the number of tries up from 5 in Windows, I'd be inclined to keep the 75 interval, rather than doubling it. -Kevin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Keepalives win32
Robert Haas wrote: (BTW, there are two buildfarm machines - wigeon and orangutan - that are consistently failing with rather bizarre error messages. Are these real failures or are those machines just messed up?) Dave and Scott, please investigate these errors in your buildfarm members. cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] multibyte-character aware support for function downcase_truncate_identifier()
Rajanikant Chirmade rajanikant.chirm...@enterprisedb.com writes: Every identifier is downcase truncated by function downcase_truncate_identifier() before using it. But since the function downcase_truncate_identifier() is not multibyte-charecter aware, it is not able to downcase some of special charecters in identifier like my_SchemÄ. IIRC this is intentional. Please consult the archives for previous discussions. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Proposal for 9.1: WAL streaming from WAL buffers
Dimitri Fontaine dfonta...@hi-media.com writes: Stop me if I'm all wrong already, but I though we said that we should handle this case by decoupling what we can send to the standby and what it can apply. What's the point of that? It won't make the standby apply any faster. What it will do is make the protocol more complicated, hence slower (more messages) and more at risk of bugs. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Proposal for 9.1: WAL streaming from WAL buffers
Tom Lane t...@sss.pgh.pa.us writes: Dimitri Fontaine dfonta...@hi-media.com writes: Stop me if I'm all wrong already, but I though we said that we should handle this case by decoupling what we can send to the standby and what it can apply. What's the point of that? It won't make the standby apply any faster. True, but it allows to send the WAL content before to ack its fsync. Regards. -- dim -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] cvs to git migration - keywords
Magnus Hagander wrote: In the previous discussions of how to migrate from cvs to git, we've all agreed we should kill the keyword expansion that we have now. I don't think, however, that we ever decided what to do with the *old* keywords. We did say we want to be able to reproduce backbranches/tags *identically* to what they are now, which indicates we need to leave the keywords in for those. That has other drawbacks, though. The way I see it, we have two ways to do it: 1) We can migrate the repository with the keywords, and then make one big commit just after (or before, that doesn't make a difference) removing them. In this case, backbranches and tags look exactly like they do now, but it also means if you do git diff between old versions, the keywords will show up there. I don't think this would be a terrible tragedy. Import, remove keyword lines on live branches, commit. That's what I'd do. cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] cvs to git migration - keywords
Dave Page dp...@pgadmin.org writes: On Wed, Jul 7, 2010 at 10:01 AM, Magnus Hagander mag...@hagander.net wrote: 1) We can migrate the repository with the keywords, and then make one big commit just after (or before, that doesn't make a difference) removing them. In this case, backbranches and tags look exactly like they do now, but it also means if you do git diff between old versions, the keywords will show up there. +1 for #1. Changing history and the resulting possibility of becoming one's own grandfather always makes me nervous. Yeah. One concrete problem with removing the $PostgreSQL$ lines is it will affect line numbering everywhere. Yeah, it's only off-by-one, but there could still be confusion. One point that isn't completely clear from Magnus' description is whether we should remove the $PostgreSQL$ lines from the HEAD branch only, or from the still-active back branches as well. I vote for the latter --- that is, if you pull a historical version of some file from the archives, you should see the appropriate $PostgreSQL$ line, but we won't have them in the source files for any future minor release. The reason for this is that otherwise there will be files floating around that claim to be CVS version x.y.z, but actually are different from that, because of back-patching activity after the git transition. That seems like a recipe for huge confusion in itself. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] cvs to git migration - keywords
On Wed, Jul 7, 2010 at 3:40 PM, Tom Lane t...@sss.pgh.pa.us wrote: One point that isn't completely clear from Magnus' description is whether we should remove the $PostgreSQL$ lines from the HEAD branch only, or from the still-active back branches as well. I vote for the latter --- that is, if you pull a historical version of some file from the archives, you should see the appropriate $PostgreSQL$ line, but we won't have them in the source files for any future minor release. The reason for this is that otherwise there will be files floating around that claim to be CVS version x.y.z, but actually are different from that, because of back-patching activity after the git transition. That seems like a recipe for huge confusion in itself. Agreed. They should be removed from the active back branches. -- Dave Page EnterpriseDB UK: http://www.enterprisedb.com The Enterprise Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] cvs to git migration - keywords
On Wed, Jul 7, 2010 at 5:01 AM, Magnus Hagander mag...@hagander.net wrote: In the previous discussions of how to migrate from cvs to git, we've all agreed we should kill the keyword expansion that we have now. I don't think, however, that we ever decided what to do with the *old* keywords. We did say we want to be able to reproduce backbranches/tags *identically* to what they are now, which indicates we need to leave the keywords in for those. That has other drawbacks, though. The way I see it, we have two ways to do it: 1) We can migrate the repository with the keywords, and then make one big commit just after (or before, that doesn't make a difference) removing them. In this case, backbranches and tags look exactly like they do now, but it also means if you do git diff between old versions, the keywords will show up there. 2) We can filter out that row during the conversion, so they look like they never existed.That means that if you check out 7.4.3 or whatever fro git, it will look like the keyword lines never existed. Since they're in comments it shouldn''t affect functionality, but it does mean that we are *not* keeping history unmodified. The advantage is that git diff on and between old revision won't include the keyword changes, of course. #1 is most likely the easiest one. It really comes down to which is most important - being able to get easy to use diffs between old revisions, or keeping history intact. Obviously, for all *new* commits, either one of these two methods will make the diffs readable. And if they are new commits, well, they are by definition not history that needs to be kept :-) Thoughts? So what happens right now using the existing git repository is that the $PostgeSQL$ tags are there, but they're unexpanded. They just say $PostgreSQL$ rather than $PostgreSQL: tgl blah blah$. I'm all in favor of removing them, but it would be nice if we could avoid cluttering the old changesets with useless changes to the keyword expansions. Maybe I'm smoking crack, though... -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] cvs to git migration - keywords
* Dave Page dp...@pgadmin.org [100707 05:05]: +1 for #1. Changing history and the resulting possibility of becoming one's own grandfather always makes me nervous. But, since we're already using CVS, our grandfather is already our granddaughter... I'll just point out that if you expand the CVS keywords in the conversion, then your git will differe from every CVS branch/date/tag checkout I do... Remember... Keywords don't *need* to be expanded... And yes, Magnus, I found that old cvs-pg stuff, I'm trying to get that info to you today... a. -- Aidan Van Dyk Create like a god, ai...@highrise.ca command like a king, http://www.highrise.ca/ work like a slave. signature.asc Description: Digital signature
Re: [HACKERS] cvs to git migration - keywords
Robert Haas robertmh...@gmail.com writes: So what happens right now using the existing git repository is that the $PostgeSQL$ tags are there, but they're unexpanded. They just say $PostgreSQL$ rather than $PostgreSQL: tgl blah blah$. Really? All of them? Seems like that would have taken some intentional processing somewhere. If we could make the conversion work like that (rather than removing the whole line) it would negate my line-number-change argument, which might mean that files pulled from the repository would be close enough to their actual historical form that no one would mind. It's still a judgment call though. On balance I think I'd rather adopt the simple rule that historical file states in the git repository should match what you would have gotten from the cvs repository. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Does mbutils.c really need to use L'\0' ?
Takahiro Itagaki itagaki.takah...@oss.ntt.co.jp writes: I think all of the following codes work in the same way at least on Windows, where the codes are actually used. utf16[dstlen] = L'\0'; utf16[dstlen] = '\0'; utf16[dstlen] = 0; utf16[dstlen] = (WCHAR) 0; The last one seems like the best choice, since it makes the intent visible. Committed that way --- thanks for the suggestion! regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Python Interface Hacking
On Wed, Jul 7, 2010 at 8:49 AM, Peter Eisentraut pete...@gmx.net wrote: If you want to hack PL/Python, which is a Python interpreter embedded into the PostgreSQL server, then this is the right place. (Yes, it's mixed with all the rest.) If you want to hack a Python client driver, then go to http://initd.org/psycopg/. (There are others, but that's the most popular one.) I was referring to this stuff: http://it.toolbox.com/blogs/database-soup/postgres-needs-a-new-python-driver-36815 http://wiki.postgresql.org/wiki/Python_PostgreSQL_Driver_TODO Anyone know where these guys hang out and discuss what to do? :-D -- Peter H. Froehlich http://www.cs.jhu.edu/~phf/ Senior Lecturer | Director, Johns Hopkins Gaming Lab -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] cvs to git migration - keywords
Tom Lane t...@sss.pgh.pa.us wrote: 1) We can migrate the repository with the keywords, and then make one big commit just after (or before, that doesn't make a difference) removing them. In this case, backbranches and tags look exactly like they do now, but it also means if you do git diff between old versions, the keywords will show up there. +1 for #1. Changing history and the resulting possibility of becoming one's own grandfather always makes me nervous. Yeah. One concrete problem with removing the $PostgreSQL$ lines is it will affect line numbering everywhere. Yeah, it's only off-by-one, but there could still be confusion. [...] If not the whole line was removed, but only the $PostgreSQL$ part, the numbering should stay the same. I guess it would otherwise be challenging to automatically not only delete the $PostgreSQL$ line, but also leading and/or trailing empty (comment) lines, and not mess up. Tim -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[PATCH] Re: [HACKERS] Issue: Deprecation of the XML2 module 'xml_is_well_formed' function
Peter Eisentraut wrote:
On lör, 2010-07-03 at 09:26 +0100, Mike Fowler wrote:
What I will do
instead is implement the xml_is_well_formed function and get a patch
out in the next day or two.
That sounds very useful.
Here's the patch to add the 'xml_is_well_formed' function. Paraphrasing
the SGML the syntax is:
|xml_is_well_formed|(/text/)
The function |xml_is_well_formed| evaluates whether the /text/ is well
formed XML content, returning a boolean. I've done some tests (included
in the patch) with tables containing a mixture of well formed documents
and content and the function is happily returning the expected result.
Combining with IS (NOT) DOCUMENT is working nicely for pulling out
content or documents from a table of text.
Unless I missed something in the original correspondence, I think this
patch will solve the issue.
Regards,
--
Mike Fowler
Registered Linux user: 379787
*** a/doc/src/sgml/func.sgml
--- b/doc/src/sgml/func.sgml
***
*** 8554,8562 SELECT xmlagg(x) FROM (SELECT * FROM test ORDER BY y DESC) AS tab;
]]/screen
/para
/sect3
sect3
! titleXML Predicates/title
indexterm
primaryIS DOCUMENT/primary
--- 8554,8566
]]/screen
/para
/sect3
+ /sect2
+
+ sect2
+titleXML Predicates/title
sect3
! titleIS DOCUMENT/title
indexterm
primaryIS DOCUMENT/primary
***
*** 8574,8579 SELECT xmlagg(x) FROM (SELECT * FROM test ORDER BY y DESC) AS tab;
--- 8578,8653
between documents and content fragments.
/para
/sect3
+
+sect3
+ titlexml_is_well_formed/title
+
+ indexterm
+ primaryxml_is_well_formed/primary
+ secondarywell formed/secondary
+ /indexterm
+
+ synopsis
+ functionxml_is_well_formed/function(replaceabletext/replaceable)
+ /synopsis
+
+ para
+ The function functionxml_is_well_formed/function evaluates whether
+ the replaceabletext/replaceable is well formed XML content, returning
+ a boolean.
+ /para
+ para
+ Example:
+ screen![CDATA[
+ SELECT xml_is_well_formed('foobar/foo');
+ xml_is_well_formed
+
+ t
+ (1 row)
+
+ SELECT xml_is_well_formed('foobar/foo');
+ xml_is_well_formed
+
+ f
+ (1 row)
+ ]]/screen
+ /para
+ para
+ This function can be combined with the IS DOCUMENT predicate to prevent
+ invalid XML content errors from occuring in queries. For example, given a
+ table that may have rows with invalid XML mixed in with rows of valid
+ XML, functionxml_is_well_formed/function can be used to filter out all
+ the invalid rows.
+ /para
+ para
+ Example:
+ screen![CDATA[
+ SELECT * FROM mixed;
+ data
+ --
+ foobar/foo
+ foobar/foo
+ foobar/foobarfoo/bar
+ foobar/foobarfoo/bar
+ (4 rows)
+
+ SELECT COUNT(data) FROM mixed WHERE data::xml IS DOCUMENT;
+ ERROR: invalid XML content
+ DETAIL: Entity: line 1: parser error : expected ''
+ foobar/foo
+ ^
+ Entity: line 1: parser error : chunk is not well balanced
+ foobar/foo
+ ^
+
+ SELECT COUNT(data) FROM mixed WHERE xml_is_well_formed(data) AND data::xml IS DOCUMENT;
+ count
+ ---
+ 1
+ (1 row)
+ ]]/screen
+ /para
+/sect3
/sect2
sect2 id=functions-xml-processing
*** a/src/backend/utils/adt/xml.c
--- b/src/backend/utils/adt/xml.c
***
*** 3293,3298 xml_xmlnodetoxmltype(xmlNodePtr cur)
--- 3293,3365
}
#endif
+ Datum
+ xml_is_well_formed(PG_FUNCTION_ARGS)
+ {
+ #ifdef USE_LIBXML
+ text*data = PG_GETARG_TEXT_P(0);
+ boolresult;
+ int res_code;
+ int32len;
+ const xmlChar *string;
+ xmlParserCtxtPtr ctxt;
+ xmlDocPtr doc = NULL;
+
+ len = VARSIZE(data) - VARHDRSZ;
+ string = xml_text2xmlChar(data);
+
+ /* Start up libxml and its parser (no-ops if already done) */
+ pg_xml_init();
+ xmlInitParser();
+
+ ctxt = xmlNewParserCtxt();
+ if (ctxt == NULL)
+ xml_ereport(ERROR, ERRCODE_OUT_OF_MEMORY,
+ could not allocate parser context);
+
+ PG_TRY();
+ {
+ size_t count;
+ xmlChar*version = NULL;
+ int standalone = -1;
+
+ res_code = parse_xml_decl(string, count, version, NULL, standalone);
+ if (res_code != 0)
+ xml_ereport_by_code(ERROR, ERRCODE_INVALID_XML_CONTENT,
+ invalid XML content: invalid XML declaration,
+ res_code);
+
+ doc = xmlNewDoc(version);
+ doc-encoding = xmlStrdup((const xmlChar *) UTF-8);
+ doc-standalone = 1;
+
+ res_code = xmlParseBalancedChunkMemory(doc, NULL, NULL, 0, string + count, NULL);
+
+ result = !res_code;
+ }
+ PG_CATCH();
+ {
+ if (doc)
+ xmlFreeDoc(doc);
+ if (ctxt)
+ xmlFreeParserCtxt(ctxt);
+
+ PG_RE_THROW();
+ }
+ PG_END_TRY();
+
+ if (doc)
+ xmlFreeDoc(doc);
+ if (ctxt)
+ xmlFreeParserCtxt(ctxt);
+
+ return result;
+ #else
+ NO_XML_SUPPORT();
+ return 0;
+
Re: [HACKERS] Python Interface Hacking
On 07/07/10 17:19, Peter Froehlich wrote: On Wed, Jul 7, 2010 at 8:49 AM, Peter Eisentrautpete...@gmx.net wrote: If you want to hack PL/Python, which is a Python interpreter embedded into the PostgreSQL server, then this is the right place. (Yes, it's mixed with all the rest.) If you want to hack a Python client driver, then go to http://initd.org/psycopg/. (There are others, but that's the most popular one.) I was referring to this stuff: http://it.toolbox.com/blogs/database-soup/postgres-needs-a-new-python-driver-36815 http://wiki.postgresql.org/wiki/Python_PostgreSQL_Driver_TODO Anyone know where these guys hang out and discuss what to do? :-D The (well, we) hang on http://lists.initd.org/mailman/listinfo/psycopg. Cheers, Jan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: I: [HACKERS] About Our CLUSTER implementation is pessimal patch
Excerpts from Takahiro Itagaki's message of mié jul 07 04:39:38 -0400 2010: BTW, we could have LogicalTapeReadExact() as an alias of LogicalTapeRead() and checking the result because we have many duplicated codes for unexpected end of data errors. I'd just add a boolean exact required to the existing functions. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] cvs to git migration - keywords
On Wed, Jul 7, 2010 at 10:57 AM, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: So what happens right now using the existing git repository is that the $PostgeSQL$ tags are there, but they're unexpanded. They just say $PostgreSQL$ rather than $PostgreSQL: tgl blah blah$. Really? All of them? Seems like that would have taken some intentional processing somewhere. I'm sure it did... -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] t_self as system column
Excerpts from Robert Haas's message of mar jul 06 17:24:21 -0400 2010: On Tue, Jul 6, 2010 at 5:18 PM, Andrew Dunstan and...@dunslane.net wrote: In any case, having a mutable logical column position is the feature that's been most requested. I think that's true. But the physical storage position would give us a performance benefit, by allowing us to try to avoid useless alignment padding. That's true too. I intend to look at both problems simultaneously, i.e. decoupling the current attnum in three columns as previously discussed; as Tom says, I think it'll end up being less work than attacking them separately. However, I will not attempt to include optimizations such as avoiding padding, in the first patch, just the possibility that it is added later. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] cvs to git migration - keywords
Robert Haas wrote: So what happens right now using the existing git repository is that the $PostgeSQL$ tags are there, but they're unexpanded. They just say $PostgreSQL$ rather than $PostgreSQL: tgl blah blah$. I'm all in favor of removing them, but it would be nice if we could avoid cluttering the old changesets with useless changes to the keyword expansions. Personally I favor leaving the expanded keywords in what we import, so that there's an exact mapping between what's in the final CVS repo and what's in the inital git repo, and then removing them entirely. I don't see that having old keyword expansions in the historical changesets is a bid deal. Nobody is going to base patches on them (I hope). cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] cvs to git migration - keywords
On Wed, Jul 7, 2010 at 20:31, Andrew Dunstan and...@dunslane.net wrote: Robert Haas wrote: So what happens right now using the existing git repository is that the $PostgeSQL$ tags are there, but they're unexpanded. They just say $PostgreSQL$ rather than $PostgreSQL: tgl blah blah$. I'm all in favor of removing them, but it would be nice if we could avoid cluttering the old changesets with useless changes to the keyword expansions. Personally I favor leaving the expanded keywords in what we import, so that there's an exact mapping between what's in the final CVS repo and what's in the inital git repo, and then removing them entirely. I don't see that having old keyword expansions in the historical changesets is a bid deal. Nobody is going to base patches on them (I hope). This is my general feeling as well. If there are outstanding patches they will need to be merged, but actually getting a conflict there would require that someone is working off their own cvs repository which expands the same tags - which would cause the conflicts today anyway. other than that, just rebasing across a HEAD that no longer has the keywords should be a very straightforward operation. Given that we generally *backpatch* fixes (rather than make them on backbranches and merge back into head), it shouldn't affect that at all. -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] cvs to git migration - keywords
On Wed, Jul 7, 2010 at 16:40, Tom Lane t...@sss.pgh.pa.us wrote: Dave Page dp...@pgadmin.org writes: On Wed, Jul 7, 2010 at 10:01 AM, Magnus Hagander mag...@hagander.net wrote: 1) We can migrate the repository with the keywords, and then make one big commit just after (or before, that doesn't make a difference) removing them. In this case, backbranches and tags look exactly like they do now, but it also means if you do git diff between old versions, the keywords will show up there. +1 for #1. Changing history and the resulting possibility of becoming one's own grandfather always makes me nervous. Yeah. One concrete problem with removing the $PostgreSQL$ lines is it will affect line numbering everywhere. Yeah, it's only off-by-one, but there could still be confusion. Uh, wouldn't that simply be dealt with by replacing them with an empty line instead of removing it? One point that isn't completely clear from Magnus' description is whether we should remove the $PostgreSQL$ lines from the HEAD branch only, or from the still-active back branches as well. I vote for the latter --- that is, if you pull a historical version of some file from the archives, you should see the appropriate $PostgreSQL$ line, but we won't have them in the source files for any future minor release. The reason for this is that otherwise there will be files floating around that claim to be CVS version x.y.z, but actually are different from that, because of back-patching activity after the git transition. That seems like a recipe for huge confusion in itself. Yeah, clearly I didn't say that :-) My intention was for them to be removed from head and all active back-branches at the time (e.g. we don't bother with 6.x, just the platforms that are currently being used). -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Python Interface Hacking
On Jul 7, 2010, at 12:07 AM, Peter Froehlich wrote: I joined this list under the mis-impression that it was about hacking the Python interfaces to pgsql. Is there possibly another list for that? Or is the Python stuff just mixed in with all the rest? Sorry for the meta-question... For BSD/MIT licensed python3 fun, be sure to check out: http://python.projects.postgresql.org/ http://python.projects.postgresql.org/backend/ http://pgfoundry.org/mailman/listinfo/python-general/ The project currently enjoys a status as an alternative, but with help from people like you that could change. =) -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] t_self as system column
On Wed, Jul 7, 2010 at 1:29 PM, Alvaro Herrera alvhe...@commandprompt.com wrote: Excerpts from Robert Haas's message of mar jul 06 17:24:21 -0400 2010: On Tue, Jul 6, 2010 at 5:18 PM, Andrew Dunstan and...@dunslane.net wrote: In any case, having a mutable logical column position is the feature that's been most requested. I think that's true. But the physical storage position would give us a performance benefit, by allowing us to try to avoid useless alignment padding. That's true too. I intend to look at both problems simultaneously, i.e. decoupling the current attnum in three columns as previously discussed; as Tom says, I think it'll end up being less work than attacking them separately. However, I will not attempt to include optimizations such as avoiding padding, in the first patch, just the possibility that it is added later. Sounds great. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [COMMITTERS] pgsql: Add note that using PL/Python 2 and 3 in the same session will
On tis, 2010-07-06 at 18:15 -0400, Tom Lane wrote:
At this point it seems clear to me that we've not adequately thought
through the implications of having two python versions in one
application namespace, and I'm not sure the Python people have either.
I think we need to do something to block that from happening, at least
until we have a plausible way to make it work.
How about this?
Index: src/pl/plpython/plpython.c
===
RCS file: /cvsroot/pgsql/src/pl/plpython/plpython.c,v
retrieving revision 1.145
diff -u -3 -p -r1.145 plpython.c
--- src/pl/plpython/plpython.c 29 Jun 2010 00:18:11 - 1.145
+++ src/pl/plpython/plpython.c 7 Jul 2010 21:04:33 -
@@ -3206,6 +3206,8 @@ PyInit_plpy(void)
#endif
+static const int plpython_python_version = PY_MAJOR_VERSION;
+
/*
* _PG_init() - library load-time initialization
*
@@ -3216,6 +3218,21 @@ _PG_init(void)
{
/* Be sure we do initialization only once (should be redundant now) */
static bool inited = false;
+ const int **version_ptr;
+
+ /* Be sure we don't run Python 2 and 3 in the same session (might crash) */
+ version_ptr = (const int **) find_rendezvous_variable(plpython_python_version);
+ if (!(*version_ptr))
+ *version_ptr = plpython_python_version;
+ else
+ {
+ if (**version_ptr != plpython_python_version)
+ ereport(ERROR,
+ (errmsg(Python major version mismatch in session),
+ errdetail(This session had previously used Python major version %d, and it is now attempting to use Python major version %d.,
+ **version_ptr, plpython_python_version),
+ errhint(Start a new session to use a different Python major version.)));
+ }
if (inited)
return;
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Why is vacuum_defer_cleanup_age PGC_USERSET?
On 7/3/10 10:07 AM, Tom Lane wrote: It seems to me it ought to be PGC_SIGHUP. +1 I believe it's userset because the other vacuum parameters are, and that nobody thought about it much. -- -- Josh Berkus PostgreSQL Experts Inc. http://www.pgexperts.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [COMMITTERS] pgsql: Add note that using PL/Python 2 and 3 in the same session will
Peter Eisentraut pete...@gmx.net writes: On tis, 2010-07-06 at 18:15 -0400, Tom Lane wrote: At this point it seems clear to me that we've not adequately thought through the implications of having two python versions in one application namespace, and I'm not sure the Python people have either. I think we need to do something to block that from happening, at least until we have a plausible way to make it work. How about this? Yeah, I was going to suggest something involving find_rendezvous_variable to let the two versions of plpython check for each other. But doesn't the error need to be elog(FATAL)? If you just elog(ERROR) then the conflicting version of python.so is already loaded and able to cause problems. elog(FATAL) isn't very desirable maybe but it beats crashing. Minor grammatical nit: I think session has previously used would read better in the errdetail. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Proposal for 9.1: WAL streaming from WAL buffers
On 7/6/10 4:44 PM, Robert Haas wrote: To recap the previous discussion on this thread, we ended up changing the behavior of 9.0 so that it only sends WAL which has been written to the OS *and flushed*, because sending unflushed WAL to the standby is unsafe. The standby can get ahead of the master while still believing that the databases are in sync, due to the fact that after an SR reconnect we rewind to the start of the current WAL segment. This results in a silently corrupt standby database. What was the final decision on behavior if fsync=off? -- -- Josh Berkus PostgreSQL Experts Inc. http://www.pgexperts.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Re: [COMMITTERS] pgsql: Fix log_temp_files docs and comments to say bytes not kilobytes.
On 7/6/10 8:06 AM, Tom Lane wrote: It might be that nobody's using any values other than 0 and -1 ... in which case it wouldn't matter anyway. I agree that the lack of bug reports is notable. But still, don't we try to avoid behavioral changes in stable branches? I think most people are doing what I was doing: looking at the values in the logs, and writing math appropriately. Most of the other log output isn't documented well, and the output values are obviously bytes, so frankly it never occurred to me to check the docs. Agreed that backporting the fix to 8.3 and 8.4 is infeasible. -- -- Josh Berkus PostgreSQL Experts Inc. http://www.pgexperts.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Bug? Concurrent COMMENT ON and DROP object
On Tue, Jul 6, 2010 at 10:59 PM, Alvaro Herrera alvhe...@commandprompt.com wrote: Excerpts from Robert Haas's message of mar jul 06 22:31:40 -0400 2010: On Tue, Jul 6, 2010 at 10:18 PM, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: Obviously not. We don't need to acquire an AccessExclusiveLock to comment on an object - just something that will CONFLICT WITH an AccessExclusiveLock. So, use the same locking rules, perhaps, but take a much weaker lock, like AccessShareLock. Well, it probably needs to be a self-conflicting lock type, so that two COMMENTs on the same object can't run concurrently. But I agree AccessExclusiveLock is too strong: that implies locking out read-only examination of the object, which we don't want. Hmm... so, maybe ShareUpdateExclusiveLock? So COMMENT ON will conflict with (auto)vacuum? Seems a bit weird ... Well, I'm open to suggestions... I doubt we want to create a new lock level just for this. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Proposal for 9.1: WAL streaming from WAL buffers
On Wed, Jul 7, 2010 at 6:44 PM, Josh Berkus j...@agliodbs.com wrote: On 7/6/10 4:44 PM, Robert Haas wrote: To recap the previous discussion on this thread, we ended up changing the behavior of 9.0 so that it only sends WAL which has been written to the OS *and flushed*, because sending unflushed WAL to the standby is unsafe. The standby can get ahead of the master while still believing that the databases are in sync, due to the fact that after an SR reconnect we rewind to the start of the current WAL segment. This results in a silently corrupt standby database. What was the final decision on behavior if fsync=off? I'm not sure we made any decision, per se, but if you use fsync=off in combination with SR and experience an unexpected crash-and-reboot on the master, you will be sad. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] pg_dump and join aliases (was Re: [BUGS] ERROR: cannot handle unplanned sub-select)
Merlin sent me a test case off-list for the problem mentioned here: http://archives.postgresql.org/pgsql-bugs/2010-07/msg00025.php After some investigation I was able to simplify it to the following example using the regression database: select (select sq1) as qq1 from (select exists(select 1 from int4_tbl where f1 = q2) as sq1, 42 as dummy from int8_tbl) sq0 join int4_tbl i4 on dummy = i4.f1; The problem is that flatten_join_alias_vars() can push SubLink expressions down into sub-selects, as in this example when it replaces the sq1 reference with the EXISTS() subexpression that was previously pulled up by flattening sq0. But it fails to set the hasSubLinks flag in the sub-Query, so subsequent processing doesn't think it needs to do SS_process_sublinks within the sub-Query, and eventually we fail when we come across the unprocessed SubLink. This bug goes clear back to 7.4. Fortunately it's simple to fix. What seems more interesting is that I initially had a hard time reproducing the bug under different conditions, and didn't figure out what was going on until I realized that I had used pg_dump to consolidate the multiple files Merlin sent ... and *reloading pg_dump's version of the views didn't exhibit the bug*. This is because pg_dump, or more accurately ruleutils.c, has a habit of qualifying variable references whether or not they were qualified in the original query. If you turn the above example into a view and then dump it, you'll get ... (select sq0.sq1) as qq1 ... and that doesn't tickle this bug. (That's because sq0.sq1 isn't a join alias Var, whereas unqualified sq1 is.) So the question that seems worth discussing is whether this difference ought to be considered a bug in ruleutils. In theory it shouldn't matter if pg_dump adds an unnecessary qualification, but here's an example where it did matter. Do we care? People tend to assume that dumping and reloading will make no change in the behavior of their views, so this seems kind of scary to me. On the other hand, the extra qualifications make view definitions a bit more robust in the face of column additions, renamings, etc. So there's certainly a case to be made that the dump behavior is preferable as-is. Thoughts? regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_dump and join aliases (was Re: [BUGS] ERROR: cannot handle unplanned sub-select)
On Wed, Jul 7, 2010 at 6:55 PM, Tom Lane t...@sss.pgh.pa.us wrote: After some investigation I was able to simplify it to the following example using the regression database: select (select sq1) as qq1 from (select exists(select 1 from int4_tbl where f1 = q2) as sq1, 42 as dummy from int8_tbl) sq0 join int4_tbl i4 on dummy = i4.f1; [discussion of bug] What seems more interesting is that I initially had a hard time reproducing the bug under different conditions, and didn't figure out what was going on until I realized that I had used pg_dump to consolidate the multiple files Merlin sent ... and *reloading pg_dump's version of the views didn't exhibit the bug*. This is because pg_dump, or more accurately ruleutils.c, has a habit of qualifying variable references whether or not they were qualified in the original query. If you turn the above example into a view and then dump it, you'll get ... (select sq0.sq1) as qq1 ... and that doesn't tickle this bug. (That's because sq0.sq1 isn't a join alias Var, whereas unqualified sq1 is.) I'm lost. What's a join alias var? -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [RRR] Reviewfest 2010-06 Plans and Call for Reviewers
On Mon, Jun 14, 2010 at 12:25 PM, Kevin Grittner kevin.gritt...@wicourts.gov wrote: The PostgreSQL 9.1 Development Plan: http://wiki.postgresql.org/wiki/PostgreSQL_9.1_Development_Plan calls for a ReviewFest to run from the 15th of June (tomorrow) until the start of the first CommitFest for the 9.1 release. The idea is that those with time available to contribute beyond what they can usefully contribute to getting 9.0 released can help provide feedback on patches submitted so far, to lighten the load of the CF proper when it starts. I have agreed to manage this RF. Who is going to manage the actual CF? -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_dump and join aliases (was Re: [BUGS] ERROR: cannot handle unplanned sub-select)
Robert Haas robertmh...@gmail.com writes: I'm lost. What's a join alias var? Suppose we have t1 with columns a,b,c and t2 with columns d,e,f, then consider select a from t1 join t2 on (whatever) select t1.a from t1 join t2 on (whatever) In the first case the parser generates a Var that references a column of the unnamed join's RTE; in the second case you get a Var that references t1 directly. These particular cases are semantically equivalent, but there are lots of other cases where it's important to draw the distinction. One interesting example is select x from (t1 join t2 on (whatever)) as j(x,y,z,xx,yy,zz) where per SQL spec it'd actually be illegal to write a (or t1.a) because the named join hides its components. But I think what forced us to have different representations is FULL JOIN USING. If you have select id from taba full join tabb using (id) then taba.id and tabb.id and the join's output variable id are all semantically different and *must* be given different representations at the Var level. Anyway, the way it works is that the parser generates alias Vars that refer to the join RTE, mainly because this makes life simpler for ruleutils. But the planner prefers to work with the real underlying columns whenever those are semantically equivalent, so it has a pass that does the replacement, and that's what's broken ... regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Bug? Concurrent COMMENT ON and DROP object
Robert Haas robertmh...@gmail.com writes: On Tue, Jul 6, 2010 at 10:59 PM, Alvaro Herrera alvhe...@commandprompt.com wrote: Excerpts from Robert Haas's message of mar jul 06 22:31:40 -0400 2010: Hmm... so, maybe ShareUpdateExclusiveLock? So COMMENT ON will conflict with (auto)vacuum? Seems a bit weird ... Well, I'm open to suggestions... I doubt we want to create a new lock level just for this. [ shrug... ] COMMENT ON is DDL, and most forms of DDL will conflict with vacuum. I can't get excited about that complaint. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Re: [COMMITTERS] pgsql: Fix log_temp_files docs and comments to say bytes not kilobytes.
2010/7/8 Josh Berkus j...@agliodbs.com: On 7/6/10 8:06 AM, Tom Lane wrote: It might be that nobody's using any values other than 0 and -1 ... in which case it wouldn't matter anyway. I agree that the lack of bug reports is notable. But still, don't we try to avoid behavioral changes in stable branches? I think most people are doing what I was doing: looking at the values in the logs, and writing math appropriately. Most of the other log output isn't documented well, and the output values are obviously bytes, so frankly it never occurred to me to check the docs. Samething here Agreed that backporting the fix to 8.3 and 8.4 is infeasible. +1 But I don't understand why not backport a documentation patch. Thing is identified, clearly boring for one just trusting the docs. -- -- Josh Berkus PostgreSQL Experts Inc. http://www.pgexperts.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers -- Cédric Villemain 2ndQuadrant http://2ndQuadrant.fr/ PostgreSQL : Expertise, Formation et Support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_dump and join aliases (was Re: [BUGS] ERROR: cannot handle unplanned sub-select)
On Wed, Jul 7, 2010 at 7:43 PM, Tom Lane t...@sss.pgh.pa.us wrote: Anyway, the way it works is that the parser generates alias Vars that refer to the join RTE, mainly because this makes life simpler for ruleutils. But the planner prefers to work with the real underlying columns whenever those are semantically equivalent, so it has a pass that does the replacement, and that's what's broken ... Well, +1 from me for leaving the ruleutils as-is. I don't think we should go out of our way to generate join alias vars just on the off chance that there's a bug in the translation from join alias vars to plain ol' vars, and I agree with your statement upthread that qualification makes things more robust. I like robust. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] ALTER TABLE SET STATISTICS requires AccessExclusiveLock
2010/3/3 Bruce Momjian br...@momjian.us: Peter Eisentraut wrote: On m?n, 2010-02-22 at 10:32 -0500, Bruce Momjian wrote: Simon Riggs wrote: On Mon, 2009-10-19 at 12:56 -0300, Alvaro Herrera wrote: Simon Riggs wrote: On Fri, 2009-08-07 at 15:58 -0400, Alvaro Herrera wrote: Tom Lane wrote: Peter Eisentraut pete...@gmx.net writes: Is there a good reason for $subject, other than that the code is entangled with other ALTER TABLE code? I think it could be lower, but it would take nontrivial restructuring of the ALTER TABLE support. In particular, consider what happens when you have a list of subcommands that don't all require the same lock level. I think you'd need to scan the list and find the highest required lock level before starting ... IIRC there was a patch from Simon to address this issue, but it had some holes which he didn't have time to close, so it sank. Maybe this can be resurrected and fixed. I was intending to finish that patch in this release cycle. Since you're busy with Hot Standby, any chance you could pass it on? If you'd like. It's mostly finished, just one last thing to finish: atomic changes to pg_class via an already agreed API. I assume this did not get done for 9.0. Do we want a TODO item? Yes. Added: Reduce locking required for ALTER commands I just faced production issue where it is impossible to alter table to adjust autovacuum settings in a pg8.4. (5K tps, 260M rows table, lock too much) Can we add some mechanism to prevent that situation also in the TODO item ? (alternative is actualy to alter other tables and adjust the postgresql.conf for biggest tables, but not an ideal solution anyway) * http://archives.postgresql.org/pgsql-hackers/2009-08/msg00533.php * http://archives.postgresql.org/pgsql-hackers/2009-10/msg01083.php * http://archives.postgresql.org/pgsql-hackers/2010-01/msg02349.php -- Bruce Momjian br...@momjian.us http://momjian.us EnterpriseDB http://enterprisedb.com PG East: http://www.enterprisedb.com/community/nav-pg-east-2010.do -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers -- Cédric Villemain 2ndQuadrant http://2ndQuadrant.fr/ PostgreSQL : Expertise, Formation et Support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] ALTER TABLE SET STATISTICS requires AccessExclusiveLock
On Wed, Jul 7, 2010 at 9:04 PM, Cédric Villemain
cedric.villemain.deb...@gmail.com wrote:
I assume this did not get done for 9.0. Do we want a TODO item?
Yes.
Added:
Reduce locking required for ALTER commands
I just faced production issue where it is impossible to alter table to
adjust autovacuum settings in a pg8.4. (5K tps, 260M rows table, lock
too much)
Can we add some mechanism to prevent that situation also in the TODO item ?
(alternative is actualy to alter other tables and adjust the
postgresql.conf for biggest tables, but not an ideal solution anyway)
*
http://archives.postgresql.org/pgsql-hackers/2009-08/msg00533.php
*
http://archives.postgresql.org/pgsql-hackers/2009-10/msg01083.php
*
http://archives.postgresql.org/pgsql-hackers/2010-01/msg02349.php
Bruce, that last link is about something else completely. Here are
some better ones:
http://archives.postgresql.org/pgsql-hackers/2008-10/msg01248.php
http://archives.postgresql.org/pgsql-hackers/2008-10/msg00242.php
All,
Rereading the thread, I'm a bit confused by why we're proposing to use
a SHARE lock; it seems to me that a self-conflicting lock type would
simplify things. There's a bunch of discussion on the thread about
how to handle pg_class updates atomically, but doesn't using a
self-conflicting lock type eliminate that problem?
It strikes me that for the following operations, which don't affect
queries at all, we could use a SHARE UPDATE EXCLUSIVE, which is likely
superior to SHARE for this purpose because it wouldn't lock out
concurrent DML write operations:
ALTER [ COLUMN ] column SET STATISTICS integer
ALTER [ COLUMN ] column SET STORAGE { PLAIN | EXTERNAL | EXTENDED | MAIN }
ALTER [ COLUMN ] column SET ( attribute_option = value [, ... ] )
ALTER [ COLUMN ] column RESET ( attribute_option [, ... ] )
CLUSTER ON index_name
SET WITHOUT CLUSTER
SET ( storage_parameter = value [, ... ] )
RESET ( storage_parameter [, ... ] )
(Of the above list, arguably SET STORAGE and [RE]SET (fillfactor) do
in fact affect DML writes, but it seems like changing them on the fly
should still be safe.)
The remaining commands which Simon proposed to downgrade to share-locks were:
ALTER [ COLUMN ] column SET DEFAULT expression
CREATE RULE (only non-ON SELECT rules)
CREATE TRIGGER
ALTER [ COLUMN ] column SET NOT NULL (but not DROP NOT NULL)
ADD table_constraint (but not DROP CONSTRAINT)
DISABLE TRIGGER [ trigger_name | ALL | USER ]
ENABLE TRIGGER [ trigger_name | ALL | USER ]
ENABLE REPLICA TRIGGER trigger_name
ENABLE ALWAYS TRIGGER trigger_name
Setting a column default, creating a non-select RULE, and
creating/disabling a trigger shouldn't affect SELECT statements, so as
long as we lock out all updates we should be OK. For these it seems
we could use SHARE ROW EXCLUSIVE, which will conflict with any other
DML command and with any data change, but not with SELECTs.
I am somewhat fuzzy on what the correct locking is for SET NOT NULL
and ADD table_constraint. I believe that the idea here is that a
query plan might rely on the existence of a constraint for
correctness, so we must lock out all queries when dropping one; but a
query plan can't rely on the absence of a constraint for correctness
(since the constraint could be true anyway), so it's safe to allow one
to be added even when there are queries in flight. If that's correct
then it seems like we could use SHARE ROW EXCLUSIVE for these command
types as well. However, these two particular commands have another
distinguishing characteristic also: they might run for a while, so it
would be useful to be able to do more than one at once. So maybe it's
worth thinking a little harder about how to weaken those two in
particular to some non-self-conflicting lock type. Then again, even
SHARE ROW EXCLUSIVE is a big improvement over ACCESS EXCLUSIVE, so
maybe that would be enough for a first go at the problem.
Thoughts?
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise Postgres Company
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] patch: preload dictionary new version
Pavel Stehule pavel.steh...@gmail.com wrote: this version has enhanced AllocSet allocator - it can use a mmap API. I review your patch and will report some comments. However, I don't have test cases for the patch because there is no large dictionaries in the default postgres installation. I'd like to ask you to supply test data for the patch. This patch allocates memory with non-file-based mmap() to preload text search dictionary files at the server start. Note that dist files are not mmap'ed directly in the patch; mmap() is used for reallocatable shared memory. The dictinary loader is also modified a bit to use simple_alloc() instead of palloc() for long-lived cache. It can reduce calls of AllocSetAlloc(), that have some overheads to support pfree(). Since the cache is never released, simple_alloc() seems to have better performance than palloc(). Note that the optimization will also work for non-preloaded dicts. === Questions === - How do backends share the dict cache? You might expect postmaster's catalog is inherited to backends with fork(), but we don't use fork() on Windows. - Why are SQL functions dpreloaddict_init() and dpreloaddict_lexize() defined but not used? === Design === - You added 3 custom parameters (dict_preload.dictfile/afffile/stopwords), but I think text search configuration names is better than file names. However, it requires system catalog access but we cannot access any catalog at the moment of preloading. If config-name-based setting is difficult, we need to write docs about where we can get the dict names to be preloaded instead. (from \dFd+ ?) - Do we need to support multiple preloaded dicts? I think dict_preload.* should accept a list of items to be loaded. GUC_LIST_INPUT will be a help. - Server doesn't start when I added dict_preload to shared_preload_libraries and didn't add any custom parameters. FATAL: missing AffFile parameter But server should start with no effects or print WARNING messages for no dicts are preloaded in such case. - We could replace simple_alloc() to a new MemoryContextMethods that doesn't support pfree() but has better performance. It doesn't look ideal for me to implement simple_alloc() on the top of palloc(). === Implementation === I'm sure that your patch is WIP, but I'll note some issues just in case. - We need Makefile for contrib/dict_preload. - mmap() is not always portable. We should check the availability in configure, and also have an alternative implementation for Win32. Regards, --- Takahiro Itagaki NTT Open Source Software Center -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Proposal for 9.1: WAL streaming from WAL buffers
Having said that, I do think we urgently need some high-level design discussion on how sync rep is actually going to handle this issue (perhaps on a new thread). If we can't resolve this issue, sync rep is going to be really slow; but there are no easy solutions to this problem in sight, so if we want to have sync rep for 9.1 we'd better agree on one of the difficult solutions soon so that work can begin. When standbys reconnect after a crash, they could send the ahead-of-the-master WAL to the master. This is an alternative to choosing the most-ahead standby as the new master, as suggested elsewhere. Greetings Marcin Mańk -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] bitmap indexes - performance
Are these improvements (index creation time, index size) worth enough to keep on working on this? I mean: given that bitmaps don't give any benefits in query times, but only benefits related to disk size and bulk index creation times, and will have horrible performance for insertions/deletions: would this job be worthed? In case it is: I will try to clean up the patch and post it... Well, if you can fix the more basic missing stuff, I think we could live with the performance issues. Bitmaps would still be a huge win for relatively static tables with lots of low-cardinality columns (basic data warehouse case). If I recall correctly, the old patch was still missing both WAL and VACUUM support. These would be required before tradeoffs of space vs. update performance would be worth talking about. As a side note: I guess that most of the bitmap indexes performance improvements in the SELECT area are already implemented in postgres in the bitmapand/or and bitmap scan stuff? I couldn't find any docs that say that bitmap indexes are faster for selects, unless of course they are ANDed/ORed together (which is something postgres already does for regular btree indexes) Have you tested this? The bitmap AND/OR for btrees in current postgres isn't exactly cost-free, especially the recheck. It seems like there could be room for better performance with bitmap indexes. -- -- Josh Berkus PostgreSQL Experts Inc. http://www.pgexperts.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
