Re: [HACKERS] WAL Info messages
On tis, 2009-12-15 at 03:34 +, Simon Riggs wrote: > Anybody know the latest on in-memory NOTIFY? "Returned with Feedback" -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Range types
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Mon, Dec 14, 2009 at 11:09:16AM -0800, Jeff Davis wrote: [...] > I think "countable" is a more accurate word than "discrete". Strings are > discrete but not countable. Oh, no -- strings (of finite, but arbitrary length) are not discrete -- you can always squeeze one more between two given strings. In this sense there are quite similar to rational numbers. Can we call them continuous? -- it depends, it seems that the terminology here isn't consistent: sometimes the rationals are considered continuous (as per the property above mentioned), sometimes the reals (which are a much more monstrous construct!) are referred to as "the continuum". As Robert points out, they are countable; you'd need infinite length for them to be more than that (then they would behave a bit like the reals, Cantor diagonal and all that ;-) All that said, it's moot: in computers, we can't represent strings of arbitrary length (PostgreSQL has an upper limit of about 1GB, right?). The situation is even more restricted with floats (they are much smaller; thus one could say that they're more "discrete" than strings, even). Problem with floats is -- the granule is not the "same size" over the whole range (nasty), and it's all implementation-dependent (nastier). But given an implementation, the concept of "next" and "previous" on floats is (if you give me some slack with NANs) mostly well-defined. Same with strings (up-to) some fixed length. Still, it seems non-discrete is a useful abstraction? Regards - -- tomás -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) iD8DBQFLJzqSBcgs9XrR2kYRAoydAJ9uUYt4aTj+BjuJv9XtDIU7UAAFjwCbBBSv gkw3a6oTqGOoQBHiuZjcJvQ= =l9YV -END PGP SIGNATURE- -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Hot Standby, release candidate?
Simon Riggs wrote: On Mon, 2009-12-14 at 11:44 +0200, Peter Eisentraut wrote: On mån, 2009-12-14 at 08:54 +, Simon Riggs wrote: Wednesday because that seemed a good delay to allow review. Josh and I had discussed the value of getting patch into Alpha3, so that was my wish and aim. I'm not aware of any particular date for end of commitfest, though possibly you are about to update me on that? Commit fests for 8.5 have usually run from the 15th to the 15th of next month, but the CF manager may choose to vary that. FWIW, the alpha release manager may also vary the release timeline of alpha3. ;-) I'm hoping that the alpha release manager can wait until Wednesday, please. At this point we've got 5 small to medium sized patches in the "Ready for Committer" queue. Maybe that gets all done on Tuesday, maybe it slips to Wednesday or later. It's not like a bell goes off tomorrow and we're done; there's probably going to be just a little slip here. In any case, it's certainly not the case that this is all done right now such that the alpha gets packed on Tuesday just because it's the 15th. It sounds like the worst case is that alpha would have to wait a day for Hot Standby to be finally committed, which seems well worth doing if it means HS gets that much more testing on it. It would be a help to eliminate the merge conflict issues for the Streaming Replication team by giving them only one code base to worry about merges against. And on the PR side, announcing HS as hitting core and available in the alpha is huge. -- Greg Smith2ndQuadrant Baltimore, MD PostgreSQL Training, Services and Support g...@2ndquadrant.com www.2ndQuadrant.com
Re: [HACKERS] pgbench: new feature allowing to launch shell commands
Greg Smith wrote: > How about this: the version you updated at > http://archives.postgresql.org/pgsql-hackers/2009-12/msg00847.php , your > pgbenchshell_20091210.patch, worked perfectly for me except for one > complaint I've since dropped. How about we move toward committing that > one, and maybe we look at this whole variable name handling improvement > as a separate patch later if you think it's worth pursuing? It's fine idea. I'll commit the previous lite version, and discuss whether we need the difference patch for fool proof. Regards, --- Takahiro Itagaki NTT Open Source Software Center -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] XLogInsert
Jaime Casanova wrote: So in this extreme case avg tps is just 6 transactions better Great job trying to find the spot where the code worked better. I'm not so sure I trust pgbench results where the TPS was so low though. Which leads us right back to exactly how Jeff measured his original results. As I said already, I think we need more insight into Jeff's performance report, a way to replicate that test, to look a bit at the latency as reported by the updated LWLock patch that Pierre submitted. Tweaking your test to give more useful results is a nice second opinion on top of that. But we're out of time for now, so this patch is getting returned with feedback. I encourage Jeff to resubmit the same patch or a better one with a little more data on performance measurements to our final 8.5 CommitFest in hopes we can confirm this an improvement worth committing. -- Greg Smith2ndQuadrant Baltimore, MD PostgreSQL Training, Services and Support g...@2ndquadrant.com www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pgbench: new feature allowing to launch shell commands
Takahiro Itagaki wrote: Michael Paquier wrote: Yeah the grammar ":variable" is used to set a parameter, the user will feel disorientated if there is no support. The attached patch supports this new case, based on Itagaki-san's last version. What is the spec of "\setshell :variable" ? Literally interpreted, it declares a variable with name ":variable". But pgbench cannot use such variables because only variables named with alphabets, numbers, and _ can be accepted. Should we forbid to assign variables that name contain invalid characters instead? After reviewing this a bit more I've realized my idea wasn't very good here. If this is what we're already accepting: \set nbranches :scale It's completely reasonable to say that *only* this works: \setshell aid expr 1 + :naccounts While this does not: \setshell :aid expr 1 + :naccounts And I was wrong to ask that it should. Sorry to have lead you both around over this, my bad. Proposed patch attached. It checks for variable names whether they consist of isalnum or _. The check is only performed when a new variable is assigned to avoid overhead. In normal workload, variables with the same names are re-used repeatedly. I don't think the additinal check would decrease performance of pgbench. This is interesting, but we're already past where this should have wrapped up and I'm not sure if it's worth fiddling with this code path right now. I think your idea is good, just outside of what we should fool with right now and I'm out of time to work on testing it further too. How about this: the version you updated at http://archives.postgresql.org/pgsql-hackers/2009-12/msg00847.php , your pgbenchshell_20091210.patch, worked perfectly for me except for one complaint I've since dropped. How about we move toward committing that one, and maybe we look at this whole variable name handling improvement as a separate patch later if you think it's worth pursuing? At this point I'd just like to have a version of the shell/setshell feature go into the pgbench code without dragging things out further into new territory. -- Greg Smith2ndQuadrant Baltimore, MD PostgreSQL Training, Services and Support g...@2ndquadrant.com www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Streaming replication and non-blocking I/O
On Tue, Dec 15, 2009 at 4:11 AM, Tom Lane wrote: > Hm. Perhaps it should be a loadable plugin and not hard-linked into the > backend? Compare dblink. You mean that such plugin is supplied in shared_preload_libraries, a new process is forked and the shared-memory related to walreceiver is created by using shmem_startup_hook? Since this approach would solve the problem discussed previously, ISTM this makes sense. http://archives.postgresql.org/pgsql-hackers/2009-11/msg00031.php Some additional code might be required to control the termination of walreceiver. Regards, -- Fujii Masao NIPPON TELEGRAPH AND TELEPHONE CORPORATION NTT Open Source Software Center -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Range types
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Mon, Dec 14, 2009 at 01:32:08PM -0500, Tom Lane wrote: [...] > (Also, stuff like strings simply doesn't have any sane concept of a > unique next or previous value. If you are willing to limit the length, then yes, you could consider them discrete too, but... > I think the number of types that are > really discrete in this sense is very small, like maybe just ints and > enums.) ...I do agree that ranges over continuous types are the more "interesting"[1] (and possibly more useful) beast. - - [11] Unfortunaltel they could turn out to be "interesting" in the sense of "may you live in interresting times" ;-) Regards - -- tomás -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) iD8DBQFLJx3xBcgs9XrR2kYRAn10AJ9f/MQiz45LS7ogsRmMXpawcOWSfgCggkWG gFev/SS09O+IOO+FB3shav0= =KwxB -END PGP SIGNATURE- -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pgAdmin III: timestamp displayed in what time zone?
Fred Janon wrote: Thanks Greg, at least among the socially unacceptable insults I got the the right list to post my question in. Aww, is somebody having a case of the Monday's? Frankly that was all useful advice. I was pointing out that despite what you thought, you might actually be decreasing your chances of getting an answer to your question in the future if you cross-post the way you did. Here's an equally helpful hint: when you have a question you'd like answered, don't insult someone in the group who spends their time every day answering questions in that area, particularly if they've actually given you a useful suggestion. Next time, I might be the best qualified to help you out, and you haven't really endeared yourself to me with your comments here. BTW, this list is listed as the list for tech questions in the pgAdmin tips, therefore if you don't want to be disturb, you might want to remove it from the pgAdmin tips. When I look at http://www.pgadmin.org/support/ for example it suggests the right list. I only see this one listed in the Translation section, as the place to ask to get added to the translators list. Does anyone know where the tips section suggesting people send tech questions to pgsql-hackers he's referring to is at? That seems like it should be cleaned up to point to the pgAdmin list instead if that's floating around there somewhere. Fred, if you could recall exactly what path you followed to end up here and let us know, we can try to keep someone else from being confused and instead directed to the right place more directly. It would be a nice gesture on your part to end our conversation here having done something useful, rather than with you just venting at me. -- Greg Smith2ndQuadrant Baltimore, MD PostgreSQL Training, Services and Support g...@2ndquadrant.com www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] EXPLAIN BUFFERS
On Sun, Dec 13, 2009 at 11:49 PM, Takahiro Itagaki wrote: > The attached patch [...] Committed. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pgAdmin III: timestamp displayed in what time zone?
On Mon, Dec 14, 2009 at 9:08 PM, Fred Janon wrote: > Thanks Greg, at least among the socially unacceptable insults I got the the > right list to post my question in. I am trying to get some help from > (supposedly) helping and knowledgeable people and I get insults in return. I don't want to make what's obviously a bad situation for you worse, but I think you might want to reconsider your position on this point. I can't find anything in Greg's email that sounds to me like an insult. I can see why you might think it's rude, but I have met Greg and he is a very nice guy and I am pretty sure he's just trying to help you understand your actions are likely to be interpreted. I actually spend a lot of time, for which I don't get paid, trying to make sure that questions posted to the mailing lists which I follow get answered. I have to admit that, like Greg, I find it annoying when people post a question that is off-topic for the list, or when they cross-post the same question to many different lists. That might not speak well of me, but it seems to me when someone cross-posts something to multiple mailing lists, they're (a) sending it to many people who are not interested in or knowledgeable about the topic of the question and (b) implying that they are entitled to free support from the community. Well, OK, we do try to provide good support, but since it is something people do on a volunteer basis, they like to be asked rather than told what they "have" to do. That probably wasn't your intention, of course. I actually thought about trying to answer your question myself, but then realized that as a non-pgadmin user I had no chance of telling you anything useful. Had Greg not answered, I probably would have looked up the correct mailing list for you myself, but that's really about all I would be able to do since it's outside my area of knowledge. A few more resources you might want to look at: http://www.postgresql.org/community/lists/ http://wiki.postgresql.org/wiki/Guide_to_reporting_problems > BTW, this list is listed as the list for tech questions in the pgAdmin tips, > therefore if you don't want to be disturb, you might want to remove it from > the pgAdmin tips. If there is a tip there that pointed you here in error, we should probably try to get that fixed, although I certainly have no ability to change the pgadmin tips myself. I've never even used it, let alone have access to change the source code. However, I don't think we get more than one or two pgadmin questions here a year, so I suspect that whatever the tip says includes some caveats about the circumstances under which posting to pgsql-hackers is recommended. Feel free to tell me if I'm wrong. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] WAL Info messages
On Mon, 2009-12-14 at 11:06 -0500, Tom Lane wrote: > Simon Riggs writes: > > I definitely wouldn't presume that anybody using Hot Standby would > > necessarily want NOTIFY to reach the standby, especially if there was an > > overhead to doing so. If using NOTIFY is the favoured approach, I would > > add a separate parameter for it and/or an explicit option on NOTIFY. > > Yeah, I had just come to the same conclusion: you'd want a separate > spigot handle on sending NOTIFY to WAL, regardless of any general flag > about HS. Anybody know the latest on in-memory NOTIFY? -- Simon Riggs www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Row-Level Security
2009/12/14 KaiGai Kohei : > IIRC, one headache issue is that user may provide well indexable conditions, > such as "SELECT * FROM view_x WHERE id = 1234". In this case, if we strictly > keep the order of evaluation between inside and outside of the view, its > performance penalty will over reasonable tradeoff to the better security. If you don't allow the indexable qual to be pushed down into the view in this situation, performance will be wretched. I think we need to distinguish between trusted and untrusted operations. Everything in the view definition is trusted. And some other things... perhaps access methods and some/most/all system catalog functions... are trusted. Other stuff is untrusted, and can't be pushed down. I think there was a previous discussion of this when Heikki first posted the issue to -hackers. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Adding support for SE-Linux security
Bruce, * Bruce Momjian (br...@momjian.us) wrote: > You are fine. I was just saying that at a time I was one of the few > loud voices on this, and if this is going to happen, it will be because > we have a team that wants to do this, not because I am being loud. I > see the team forming nicely. Not to rain down on the parade too much here, but I have to disagree about a team forming nicely. That's, unfortunately, what it looks like from the 10k-foot level. Indeed, it looks like we're making good headway to get some kind of support into core from that level. The reality is that we've barely started and really have still got quite a ways to go and it would really be useful to bring in additional resources on this. I wouldn't consider myself to be that "additional resource" unless and until I can get funding for dedicated time (either my own or someone else's). I've got a few action items that I'm planning to resolve in the next few weeks, but I've been involved in this for over a year now and it hasn't made much progress, overall, in that time. So, for anyone else who's interested in label-based security happening for PostgreSQL (for whatever reason, masochisim perfectly acceptable), please speak up and offer to help. We could use it. Thanks, Stephen signature.asc Description: Digital signature
Re: [HACKERS] Syntax for partitioning
On Mon, Dec 14, 2009 at 7:29 PM, Simon Riggs wrote: > On Fri, 2009-12-04 at 09:00 +, Simon Riggs wrote: >> On Fri, 2009-12-04 at 11:54 +0900, Itagaki Takahiro wrote: >> > Here is an update partitioning syntax patch. >> > >> > A bug reported by Marko is fixed. >> >> I will review and eventually commit this, if appropriate, though it is >> 3rd in my queue and will probably not be done for at least 2 weeks, >> possibly 4 weeks. > > I'll have to go back on this unfortunately, sorry about that. > the next patch for this will arrive in the next commitfest so maybe you have more time then -- Atentamente, Jaime Casanova Soporte y capacitación de PostgreSQL Asesoría y desarrollo de sistemas Guayaquil - Ecuador Cel. +59387171157 -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] New VACUUM FULL still needed?
On Tue, 2009-12-15 at 11:17 +0900, Takahiro Itagaki wrote: > Simon Riggs wrote: > > > I have enough items emerging from HS to keep me busy much longer than I > > thought. I'll run with VF if that's OK, since I have some other related > > changes in that area and it makes sense to understand that code also, if > > OK with you. > > Sure. Many users want to see HS. > > BTW, New VACUUM FULL patch is waiting for being applied. > https://commitfest.postgresql.org/action/patch_view?id=202 > But I heard HS is attempting to modify VFI in another way or remove it > completely. Do we still need the patch, or reject it and fix VFI in HS? Plan is to apply patch for new VF, then for me to write another patch to allow new VF to work with system relations also. VACUUM FULL INPLACE would then be prohibited if recovery_connections = on, which given that is the default will pretty much reduce VFI to not working at all in 8.5. But it remains an option if problems occur. My intention is to keep all of the code there for 8.5 and then begin removing old VF code at beginning of 8.6dev. It's been there too long and is in far too deep to rip it out quickly. There's no mileage in spending time on removing a non-feature when there is feature work to be done. -- Simon Riggs www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] New VACUUM FULL still needed?
Simon Riggs wrote: > I have enough items emerging from HS to keep me busy much longer than I > thought. I'll run with VF if that's OK, since I have some other related > changes in that area and it makes sense to understand that code also, if > OK with you. Sure. Many users want to see HS. BTW, New VACUUM FULL patch is waiting for being applied. https://commitfest.postgresql.org/action/patch_view?id=202 But I heard HS is attempting to modify VFI in another way or remove it completely. Do we still need the patch, or reject it and fix VFI in HS? Regards, --- Takahiro Itagaki NTT Open Source Software Center -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] 答复: [HACKERS] 答复: questions about concurrency control in Po stgresql
You are right. I never consider the SELECT FOR UPDATE/SHARE type queries, so I got the wrong conclusion. I have seen the content in the comment of heap_lock_tuple(). Thank you, Best Regards, --Huang Xiaocheng --Database & Information System Lab, Nankai University -邮件原件- 发件人: Alvaro Herrera [mailto:alvhe...@commandprompt.com] 发送时间: 2009年12月10日 22:54 收件人: 黄晓骋 抄送: 'Greg Stark'; pgsql-hackers@postgresql.org 主题: Re: [HACKERS] 答复: questions about concurrency control in Postgresql 黄晓骋 escribió: > I think I know why we need tuple lock. > Though we have tuple's infomask shows whether the tuple is being updated, > before we set the tuple's infomask, there may be two transaction coming and > updating the tuple. They both think the tuple is ok to be updated, and then > it's wrong. > In PostgreSQL, we can use buffer lock to solve the problem , but its > granularity is not proper. So we must use tuple lock to solve the problem. > Thank you, Greg. You prompt me to think clearly about it. Actually it's the buffer lock that's used to protect most of infomask. Tuple locks are only used while XMAX and some infomask bits are set for SELECT FOR UPDATE/SHARE type queries. That can take a while because it may need I/O in pg_multixact, so the buffer lock is not appropriate to hold for so long. -- Alvaro Herrerahttp://www.CommandPrompt.com/ The PostgreSQL Company - Command Prompt, Inc. __ Information from ESET NOD32 Antivirus, version of virus signature database 4677 (20091210) __ The message was checked by ESET NOD32 Antivirus. http://www.eset.com __ Information from ESET NOD32 Antivirus, version of virus signature database 4687 (20091214) __ The message was checked by ESET NOD32 Antivirus. http://www.eset.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pgAdmin III: timestamp displayed in what time zone?
Thanks Greg, at least among the socially unacceptable insults I got the the right list to post my question in. I am trying to get some help from (supposedly) helping and knowledgeable people and I get insults in return. BTW, this list is listed as the list for tech questions in the pgAdmin tips, therefore if you don't want to be disturb, you might want to remove it from the pgAdmin tips. Has someone heard of cyber bullying? Fred On Tue, Dec 15, 2009 at 00:44, Greg Smith wrote: > Fred Janon wrote: > >> Sorry if if's a double post, but I thought that it would be more likely I >> would get an answer on the hackers list. >> > In that case just posting here would have been better than hitting both. I > usually ignore any request for help that is posted on more than one list > just to draw attention to itself; I wouldn't be surprised that are other > people who are similarly annoyed by the behavior and do likewise. > In your case, both lists you picked were the wrong ones, since you're > asking a pgAdmin specific question which isn't the direct topic of any of > the pgsql-* lists; pgadmin-support was the right place for it: > http://archives.postgresql.org/pgadmin-support/ > > -- > Greg Smith2ndQuadrant Baltimore, MD > PostgreSQL Training, Services and Support > g...@2ndquadrant.com www.2ndQuadrant.com > >
Re: [HACKERS] Row-Level Security
KaiGai, * KaiGai Kohei (kai...@ak.jp.nec.com) wrote: > IIRC, one headache issue is that user may provide well indexable conditions, > such as "SELECT * FROM view_x WHERE id = 1234". In this case, if we strictly > keep the order of evaluation between inside and outside of the view, its > performance penalty will over reasonable tradeoff to the better security. > > Someone pointed out user given conditions which can be replaced by index scan > are "trusted", so all we need to focus on are conditions which need to check > for each tuples. I also think it is a right direction, as long as we can > restrict who can define index access method in appropriate way. It sounds like that might help, but I feel that a whole solution will be more complex than just differentiating between seq scan nodes and index scan ones. > If we can focus on the order of evaluation on the non-indexed conditions, > the point is order_qual_clauses() which sort the given qual list based on > the cost evaluation. If we can mark condition node a flag which means this > node come from inside of view or row-level policy, it is not difficult to > keep evaluation order. Identifying where this matters is important. Anyone have suggestions on how to do that? There was some discussion on IRC about that but it didn't really go anywhere. I don't like the idea of presuming the user will always want to limit the planner in this way. Perhaps we can convince ourselves, once we have an implementation, that it doesn't poorly affect performance (the primary reason to avoid constraining the planner), or that it's what our users would really want (I might be able to buy off on this..), but I doubt it. A couple of options about how the user could ask us to constrain the planning to eliminate this issue are, off-hand: Global GUC which enables/disables Attribute of the view, perhaps indicated as 'CREATE SECURITY VIEW' or similar Something in the definition of the WHERE clause, eg: select * from x where security(q = 50); Anyone have thoughts about this? Perhaps it's too early to discuss this anyway, just trying to keep the discussion moving in some way. > However, it is just my quick idea. It might miss something. > We need to consider the matter for more details... I agree, this needs more thought and input from others who are very familiar with the planner, executor, etc. Additionally, this needs to be done before we can really go anywhere with row-level security. Thanks, Stephen signature.asc Description: Digital signature
Re: [HACKERS] pgbench: new feature allowing to launch shell commands
Michael Paquier wrote: > Yeah the grammar ":variable" is used to set a parameter, the user will feel > disorientated if there is no support. > The attached patch supports this new case, based on Itagaki-san's last > version. What is the spec of "\setshell :variable" ? Literally interpreted, it declares a variable with name ":variable". But pgbench cannot use such variables because only variables named with alphabets, numbers, and _ can be accepted. Should we forbid to assign variables that name contain invalid characters instead? > I also added a warning to tell the user that pgbench is slowing > down when using this feature. This change seems to be nonsense. Do you want to fill your terminal with such messages? Proposed patch attached. It checks for variable names whether they consist of isalnum or _. The check is only performed when a new variable is assigned to avoid overhead. In normal workload, variables with the same names are re-used repeatedly. I don't think the additinal check would decrease performance of pgbench. Regards, --- Takahiro Itagaki NTT Open Source Software Center pgbenchshell_20091215.patch Description: Binary data pgbenchshell_test.sql Description: Binary data -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Row-Level Security
Stephen Frost wrote: > KaiGai, > > * KaiGai Kohei (kai...@kaigai.gr.jp) wrote: >> The reason why I put on the security hook in ExecScan() is to avoid the >> problem that row-cost user defined function can be evaluated earlier >> than row-level security policy. (I believed it was a well-known problem >> at that time yet.) So, I didn't want to append it before optimization. > > This is a problem which needs to be addressed and fixed independently. > >> I also believe this matter should be resolved when we provide row-level >> security stuff, because it is a security feature. > > This issue should be fixed first, not as part of some large-scale patch. > > If you have thoughts or ideas about how to address this problem as it > relates to views, I think you would find alot of people willing to > listen and to discuss it. This must be independent of SELinux, > independent of row-level security, and isn't something based on any of > the patches which have been submitted so far. None of them that I've > seen resolve this problem in a way that the community is willing to > accept. Sorry, I don't have something good idea at the moment. IIRC, one headache issue is that user may provide well indexable conditions, such as "SELECT * FROM view_x WHERE id = 1234". In this case, if we strictly keep the order of evaluation between inside and outside of the view, its performance penalty will over reasonable tradeoff to the better security. Someone pointed out user given conditions which can be replaced by index scan are "trusted", so all we need to focus on are conditions which need to check for each tuples. I also think it is a right direction, as long as we can restrict who can define index access method in appropriate way. If we can focus on the order of evaluation on the non-indexed conditions, the point is order_qual_clauses() which sort the given qual list based on the cost evaluation. If we can mark condition node a flag which means this node come from inside of view or row-level policy, it is not difficult to keep evaluation order. However, it is just my quick idea. It might miss something. We need to consider the matter for more details... -- OSS Platform Development Division, NEC KaiGai Kohei -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pgbench: new feature allowing to launch shell commands
Hi,
Yeah the grammar ":variable" is used to set a parameter, the user will feel
disorientated if there is no support.
The attached patch supports this new case, based on Itagaki-san's last
version. I also added a warning to tell the user that pgbench is slowing
down when using this feature.
If nobody is against it, I will mark it as ready to commit.
--
Regards,
Michael Paquier
NIPPON TELEGRAPH AND
TELEPHONE CORPORATION
NTT Open Source Software Center
diff --git a/contrib/pgbench/pgbench.c b/contrib/pgbench/pgbench.c
index 8a6437f..a5b8aeb 100644
--- a/contrib/pgbench/pgbench.c
+++ b/contrib/pgbench/pgbench.c
@@ -159,6 +159,7 @@ typedef struct
} Variable;
#define MAX_FILES 128 /* max number of SQL script files allowed */
+#define SHELL_COMMAND_SIZE 256 /* maximum size allowed for shell command */
/*
* structures used in custom query mode
@@ -590,6 +591,118 @@ getQueryParams(CState *st, const Command *command, const char **params)
params[i] = getVariable(st, command->argv[i + 1]);
}
+/*
+ * Run a shell command. The result is assigned to the variable if not NULL.
+ * Return true if succeeded, or false on error.
+ */
+static bool
+runShellCommand(CState *st, char *variable, char **argv, int argc)
+{
+ char command[SHELL_COMMAND_SIZE];
+ int i,
+ len = 0;
+ FILE *fp;
+ char res[64];
+ char *endptr;
+ int retval;
+
+ /*
+ * Join arguments separated with a whilespace. Arguments starting with
+ * just one colon are treated as variables:
+ * name - append a string "name"
+ * :var - append a variable named 'var'.
+ * ::name - append a string ":name"
+ */
+ for (i = 0; i < argc; i++)
+ {
+ char *arg;
+ int arglen;
+
+ if (argv[i][0] != ':')
+ {
+ arg = argv[i]; /* a string literal */
+ }
+ else if (argv[i][1] == ':')
+ {
+ arg = argv[i] + 1; /* a string literal starting with colons */
+ }
+ else if ((arg = getVariable(st, argv[i] + 1)) == NULL)
+ {
+ fprintf(stderr, "%s: undefined variable %s\n", argv[0], argv[i]);
+ return false;
+ }
+
+ arglen = strlen(arg);
+ if (len + arglen + (i > 0 ? 1 : 0) >= SHELL_COMMAND_SIZE)
+ {
+ fprintf(stderr, "%s: too long shell command\n", argv[0]);
+ return false;
+ }
+
+ if (i > 0)
+ command[len++] = ' ';
+ memcpy(command + len, arg, arglen);
+ len += arglen;
+ }
+
+ command[len] = '\0';
+
+ /* Fast path for non-assignment case */
+ if (variable == NULL)
+ {
+ fprintf(stderr, "Warning: slowing down due to %s command overhead\n", argv[0]);
+ if (system(command))
+ {
+ fprintf(stderr, "%s: cannot launch shell command\n", argv[0]);
+ return false;
+ }
+ return true;
+ }
+
+ /* Both variable and :variable grammars are supported */
+ if (variable[0] == ':')
+ variable++;
+
+ /* Execute the command with pipe and read the standard output. */
+ if ((fp = popen(command, "r")) == NULL)
+ {
+ fprintf(stderr, "%s: cannot launch shell command\n", argv[0]);
+ return false;
+ }
+ if (fgets(res, sizeof(res), fp) == NULL)
+ {
+ if (!timer_exceeded)
+ fprintf(stderr, "%s: cannot read the result\n", argv[0]);
+ return false;
+ }
+ if (pclose(fp) < 0)
+ {
+ fprintf(stderr, "%s: cannot close shell command\n", argv[0]);
+ return false;
+ }
+
+ /* Check whether the result is an integer and assign it to the variable */
+ retval = (int) strtol(res, &endptr, 10);
+ while (*endptr != '\0' && isspace((unsigned char) *endptr))
+ endptr++;
+ if (*res == '\0' || *endptr != '\0')
+ {
+ fprintf(stderr, "%s: must return an integer ('%s' returned)\n", argv[0], res);
+ return false;
+ }
+ snprintf(res, sizeof(res), "%d", retval);
+ if (!putVariable(st, variable, res))
+ {
+ fprintf(stderr, "%s: out of memory\n", argv[0]);
+ return false;
+ }
+
+#ifdef DEBUG
+ printf("shell parameter name: %s, value: %s\n", argv[1], res);
+#endif
+ return true;
+}
+
#define MAX_PREPARE_NAME 32
static void
preparedStatementName(char *buffer, int file, int state)
@@ -992,7 +1105,34 @@ top:
st->listen = 1;
}
+ else if (pg_strcasecmp(argv[0], "setshell") == 0)
+ {
+ bool ret = runShellCommand(st, argv[1], argv + 2, argc - 2);
+ if (timer_exceeded) /* timeout */
+return clientDone(st, true);
+ else if (!ret) /* on error */
+ {
+st->ecnt++;
+return true;
+ }
+ else /* succeeded */
+st->listen = 1;
+ }
+ else if (pg_strcasecmp(argv[0], "shell") == 0)
+ {
+ bool ret = runShellCommand(st, NULL, argv + 1, argc - 1);
+
+ if (timer_exceeded) /* timeout */
+return clientDone(st, true);
+ else if (!ret) /* on error */
+ {
+st->ecnt++;
+return true;
+ }
+ else /* succeeded */
+st->listen = 1;
+ }
goto top;
}
@@ -1313,6 +1453,22 @@ process_commands(char *buf)
fprintf(stderr, "%s: extra argument \"%s\" ignored\n",
my_commands->argv[0], my_commands->argv[j]);
}
+ else if (pg_strcasecmp(my_commands->argv[0], "setshell") == 0)
+ {
+ if (my_commands->argc < 3)
+ {
+fprintf(stderr, "%s: missing argument\n", my_commands->argv[0]);
+return NULL;
+ }
+
Re: [HACKERS] Syntax for partitioning
On Fri, 2009-12-04 at 09:00 +, Simon Riggs wrote: > On Fri, 2009-12-04 at 11:54 +0900, Itagaki Takahiro wrote: > > Here is an update partitioning syntax patch. > > > > A bug reported by Marko is fixed. > > I will review and eventually commit this, if appropriate, though it is > 3rd in my queue and will probably not be done for at least 2 weeks, > possibly 4 weeks. I'll have to go back on this unfortunately, sorry about that. I have enough items emerging from HS to keep me busy much longer than I thought. I'll run with VF if that's OK, since I have some other related changes in that area and it makes sense to understand that code also, if OK with you. -- Simon Riggs www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Hot Standby, release candidate?
On Mon, 2009-12-14 at 11:44 +0200, Peter Eisentraut wrote: > On mån, 2009-12-14 at 08:54 +, Simon Riggs wrote: > > Wednesday because that seemed a good delay to allow review. Josh and I > > had discussed the value of getting patch into Alpha3, so that was my > > wish and aim. > > > > I'm not aware of any particular date for end of commitfest, though > > possibly you are about to update me on that? > > Commit fests for 8.5 have usually run from the 15th to the 15th of next > month, but the CF manager may choose to vary that. > > FWIW, the alpha release manager may also vary the release timeline of > alpha3. ;-) I'm hoping that the alpha release manager can wait until Wednesday, please. -- Simon Riggs www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Range types
Scott Bailey writes: > Tom Lane wrote: >> If the only interesting use-cases are ints and enums, maybe we could >> just hard-wire it. > I think dates could be added to that list as well. Good point. Network addresses too probably. > But any implementation that doesn't do ranges of timestamptz are > non-starters as far as I'm concerned. Oh, I quite agree --- I'm just complaining about trying to force timestamps into a discrete model that they don't fit. What I was trying to suggest was that we could hard-wire a mechanism that says ints and a few other predetermined cases are discrete while everything else is treated as continuous. > Personally, I'd rather just see float timestamps go away. That's more or less irrelevant to my point. A large fraction of the datatypes in Postgres do not have discrete behavior. Reals, numerics, timestamps, strings, bitstrings, geometrics. Not to mention arrays and composites. Creating an artificial granularity is simply the wrong way to approach it, even for those types where there's an implementation artifact that allows you to make it sort-of-work. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Hot Standby, release candidate?
On Mon, 2009-12-14 at 16:39 -0500, Tom Lane wrote: > Simon Riggs writes: > > What is the best way of restricting the hash table to a maximum size? > > There is nothing in dynahash that will enforce a maximum size against > calling code that's not cooperating; and I'll resist any attempt to > add such a thing, because it would create a serialization point across > the whole hashtable. No problem, just checking with you where you'd like stuff put. > If you know that you need at most N entries in the hash table, you can > preallocate that many at startup (note the second arg to ShmemInitHash) > and be safe. If your calling code might go past that, you'll need to > fix the calling code. It's easy enough to count em on the way in and count em on the way out. -- Simon Riggs www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Hot Standby, release candidate?
Simon Riggs writes: > What is the best way of restricting the hash table to a maximum size? There is nothing in dynahash that will enforce a maximum size against calling code that's not cooperating; and I'll resist any attempt to add such a thing, because it would create a serialization point across the whole hashtable. If you know that you need at most N entries in the hash table, you can preallocate that many at startup (note the second arg to ShmemInitHash) and be safe. If your calling code might go past that, you'll need to fix the calling code. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Hot Standby, release candidate?
On Mon, 2009-12-14 at 15:24 -0500, Tom Lane wrote: > Simon Riggs writes: > > On Mon, 2009-12-14 at 20:32 +0200, Heikki Linnakangas wrote: > >>> I have ensured that they are always the same size, by definition, so no > >>> need to check. > >> > >> How did you ensure that? The hash table has no hard size limit. > > > The hash table is in shared memory and the entry size is fixed. My > > understanding was that this meant the hash table was fixed in size and > > could not grow beyond the allocation. If that assumption was wrong, then > > yes we could get an error. Is it? > > Entirely. The only thing the hash table size enters into is the sizing > of overall shared memory --- different hash tables then consume space > from the common pool, which includes not only the computed space > requirements but a pretty hefty slop overhead. You can go beyond the > original requested space if there is any slop left. OK, thanks. > For a number of shared hashtables that actually have a fixed set of > entries, we avoid the risk of unexpected out-of-memory by forcing all > the entries to come into existence during startup. If your table > doesn't work that way then you cannot be sure of the exact point where > it will get an out-of-memory failure. The data structure was originally a list of fixed size, though is now a shared hash table. What is the best way of restricting the hash table to a maximum size? Your last para makes me think there is a way, but I can't see it directly. If there isn't a facility to do this and I need to add code, should I add optional code into the dynahash.c to track size, or should I add that in the data structure code that uses the hash functions (so, internally or externally). -- Simon Riggs www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Range types
Tom Lane wrote: Jeff Davis writes: On Mon, 2009-12-14 at 14:23 -0500, Tom Lane wrote: I'd prefer not to leave it to the user to decide whether a type is discrete or not. I don't know how we can decide such a thing. Do you have any ideas? If the only interesting use-cases are ints and enums, maybe we could just hard-wire it. I think dates could be added to that list as well. But any implementation that doesn't do ranges of timestamptz are non-starters as far as I'm concerned. Certainly int64 timestamps and numeric are doable. And Jeff's period implementation supports float timestamps. I never use float timestamps so I can only assume that he made it work. Personally, I'd rather just see float timestamps go away. And if the range types never supported float or float timestamps, I'd be ok with that. Scott -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pgbench: new feature allowing to launch shell commands
Takahiro Itagaki wrote: Heavily cleaned up patch attached. Please review it. This is almost there. The refactoring work you both did is exactly how I was hoping this would end up looking, code-wise, and the feature set is basically feature complete except for one small UI concern I have. Attached is an updated version of my skewed-select.sql test program, adjusted to take advantage of the now supported syntax. Note that in order to run this properly, you need to pass the database scale into pgbench when you run it like this: pgbench -T 10 -j 4 -c 8 -s 10 -f skewed-select.sql pgbench Because when you use a custom "-f" script, :scale isn't set otherwise. In the current version of the patch, you can do this: \setshell aid skewed-acct.pl :naccounts But you can't do this: \setshell :aid skewed-acct.pl :naccounts What's worse is that you don't get an error in that second case--it just doesn't set the variable. I thought this was fixed by one of Michael's versions here, so maybe this is a regression from the other clean-up? Since all the other ways you interact with this type of variable in pgbench require the ":", not supporting it in this context seems like a recipe for weird problems for those trying to use it. If we can get an updated version where you can use either syntax for the variable name passed to setshell, this one will be ready for commit I think. I was tempted to fix the bug myself, but I seem to be doing better as a tester on this patch rather than working on its development. Everything else--docs, code--looks pretty good now. Only thing we might add is a warning that you're going to completely tank your pgbench results by using the high-overhead shell command if your test would otherwise be limited by CPU. This feature is only really useful as far as performance testing goes if you're using it on an I/O bound test. I have some more pgbench hints to apply at some point in the future, so it wouldn't be problem to skip this for now; I can bundle it into that section later. -- Greg Smith2ndQuadrant Baltimore, MD PostgreSQL Training, Services and Support g...@2ndquadrant.com www.2ndQuadrant.com \set naccounts 10 * :scale -- This works \setshell aid skewed-acct.pl :naccounts -- But this doesn't and should: -- \setshell :aid skewed-acct.pl :naccounts SELECT abalance FROM pgbench_accounts WHERE aid = :aid; \shell echo ::aid :aid selected from :naccounts accounts -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Range types
Scott Bailey writes: > I was referring to the syntax for how the user actually defined an enum > not about it's implementation. Basically what I was hoping to get out of > this thread was whether it was better to allow the user to define their > own range types by specifying the base type and possibly the granularity > and default inclusiveness of the end points, or if we should just > provide the types like period and intrange? If 99% of the usefulness will come from ranges over a fixed set of datatypes, it might be best to just do that. That last 1% would be very expensive to get, when you consider all the infrastructure that would be involved with an extensible definition. If we think there's a lot of usefulness for ranges over user-defined types, then this argument doesn't help ... regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Hot Standby, release candidate?
Simon Riggs writes: > On Mon, 2009-12-14 at 20:32 +0200, Heikki Linnakangas wrote: >>> I have ensured that they are always the same size, by definition, so no >>> need to check. >> >> How did you ensure that? The hash table has no hard size limit. > The hash table is in shared memory and the entry size is fixed. My > understanding was that this meant the hash table was fixed in size and > could not grow beyond the allocation. If that assumption was wrong, then > yes we could get an error. Is it? Entirely. The only thing the hash table size enters into is the sizing of overall shared memory --- different hash tables then consume space from the common pool, which includes not only the computed space requirements but a pretty hefty slop overhead. You can go beyond the original requested space if there is any slop left. For a number of shared hashtables that actually have a fixed set of entries, we avoid the risk of unexpected out-of-memory by forcing all the entries to come into existence during startup. If your table doesn't work that way then you cannot be sure of the exact point where it will get an out-of-memory failure. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Range types
> Right, I don't think strings are any more or less countable than integers. > (and yes, it's a bit OT). Well, if I remember my algebra, I think the issue is that integers are locally finite whereas strings are not ( assuming standard orderings, of course ). -Nathan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Aggregate ORDER BY patch
Andrew Gierth writes: > "Tom" == Tom Lane writes: > Tom> and I would also expect there to be a nonzero performance hit > Tom> from the extra TargetEntry expression nodes, even when the > Tom> feature is not in use. > I tested for that and couldn't reliably detect any (certainly <2% > on count(i) on generated data, and under the circumstances I was > testing that's not necessarily outside the error margin). Hm. Now that I look closer I see we already have a hack to avoid executing an extra expression node when a targetlist item is evaluated, so the extra cost should indeed be insignificant. There would be nonzero associated costs during planning and executor startup, but we seldom bother optimizing for that. So nevermind... regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Range types
Andrew Dunstan writes: > Surely the issue from our POV is whether, given two distinct members of > a class, we can ever say there is not any intervening member of the > class according to some ordering. If we can't then next() and prior() > make no sense for that class. We would also like to be sure that the answer is not machine-dependent. For example, you can make such an assertion for two normalized floats that differ by 1 unit in the last place --- but with a different float implementation the answer could be different, and anyway a lot of the really serious problems with float ranges would stem from the range boundary value probably not being exactly what the user thinks it is. Also, it's not just "some ordering" that's of interest, it's the natural one for the datatype. The reason the countability of strings isn't relevant to us here is that diagonalization or dovetailing counts them in an ordering that people wouldn't want in practice. But that brings up something that actually is interesting: should the range mechanism have a way to identify which btree opclass is considered to define the type's sort order? Or is it enough to provide ranges in the type's default ordering? regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [patch] executor and slru dtrace probes
Bernd Helmle píše v po 14. 12. 2009 v 20:42 +0100: > > --On 14. Dezember 2009 20:33:12 +0100 Bernd Helmle > wrote: > > >> Since the author has pretty much admitted he didn't fix any of the > >> issues that were raised by the last committer review, I'm a little > >> confused about why you're asking for another one. > > > > It wasn't clear to me what Zdenek meant with "If I think about it". > > Oh, and i was under the opinion the last discussions were about executor > probes only (note the patch is split up into two parts now, SLRU and > executor probes). The latter won't be fixed, but it seems the SLRU part at > least is ready. > I would like to add SimpleLruInit probe. I'm busy with PG packaging, but I hope that I will send updated version tomorrow. Zdenek -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Range types
Jeff Davis writes: > On Mon, 2009-12-14 at 14:23 -0500, Tom Lane wrote: >> I'd prefer not to leave it to the user to decide whether a type is >> discrete or not. > I don't know how we can decide such a thing. Do you have any ideas? If the only interesting use-cases are ints and enums, maybe we could just hard-wire it. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Range types
Tom Lane wrote: We can ask the user to provide a prior() and next() function, and if they aren't provided, we assume it's continuous. Well, that still leaves us with the problem that Joe Schmo will file a bug when "create function next(float4) returns float4 as $$ select $1 + 0.1 $$" doesn't behave sanely for him. I'd prefer not to leave it to the user to decide whether a type is discrete or not. The traffic on pgsql-bugs is convincing evidence that a very large fraction of our user-base doesn't understand that floats are inexact :-( Indeed. I think "countable" is a more accurate word than "discrete". Strings are discrete but not countable. It's been too long since college math classes for me to be sure whether "discrete" is really the exact term here. But I'm even more suspicious of "countable". I think a suitable diagonalization argument might show that strings are countable. That's getting a bit off-topic though... Right, I don't think strings are any more or less countable than integers. (and yes, it's a bit OT). Surely the issue from our POV is whether, given two distinct members of a class, we can ever say there is not any intervening member of the class according to some ordering. If we can't then next() and prior() make no sense for that class. cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [patch] executor and slru dtrace probes
--On 14. Dezember 2009 20:33:12 +0100 Bernd Helmle wrote: Since the author has pretty much admitted he didn't fix any of the issues that were raised by the last committer review, I'm a little confused about why you're asking for another one. It wasn't clear to me what Zdenek meant with "If I think about it". Oh, and i was under the opinion the last discussions were about executor probes only (note the patch is split up into two parts now, SLRU and executor probes). The latter won't be fixed, but it seems the SLRU part at least is ready. -- Thanks Bernd -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Range types
On Mon, 2009-12-14 at 14:23 -0500, Tom Lane wrote: > > We can ask the user to provide a prior() and next() function, and if > > they aren't provided, we assume it's continuous. > > Well, that still leaves us with the problem that Joe Schmo will file > a bug when "create function next(float4) returns float4 as > $$ select $1 + 0.1 $$" doesn't behave sanely for him. I'd prefer > not to leave it to the user to decide whether a type is discrete or > not. The traffic on pgsql-bugs is convincing evidence that a very > large fraction of our user-base doesn't understand that floats are > inexact :-( I don't know how we can decide such a thing. Do you have any ideas? Perhaps we can compromise and restrict the support functions to C? That might be a high-enough wall, and of course it would keep non-superusers from confusing the underlying mechanism. Regards, Jeff Davis -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Range types
On Mon, Dec 14, 2009 at 2:23 PM, Tom Lane wrote: > It's been too long since college math classes for me to be sure whether > "discrete" is really the exact term here. But I'm even more suspicious > of "countable". I think a suitable diagonalization argument might show > that strings are countable. That's getting a bit off-topic though... It's actually a dovetailing argument, not a diagonalization argument, but yes, the set of strings is most certainly countable. ...Robert (former CS theory teaching assistant) -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [patch] executor and slru dtrace probes
--On 14. Dezember 2009 07:49:34 -0500 Robert Haas wrote: Since the author has pretty much admitted he didn't fix any of the issues that were raised by the last committer review, I'm a little confused about why you're asking for another one. It wasn't clear to me what Zdenek meant with "If I think about it". -- Thanks Bernd -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Range types
On Mon, 2009-12-14 at 11:10 -0800, Scott Bailey wrote: > I was referring to the syntax for how the user actually defined an enum > not about it's implementation. Basically what I was hoping to get out of > this thread was whether it was better to allow the user to define their > own range types by specifying the base type and possibly the granularity > and default inclusiveness of the end points, or if we should just > provide the types like period and intrange? To be a little bit more specific about the alternatives: 1. Make a contrib module that creates a variety of range types like PERIOD, PERIODTZ, INT4RANGE, NUMRANGE, etc. In order to support some of the other features I intend to work on, such as a range join (e.g. temporal join), we would need to mark an operator family to know that it conforms to a certain strategy number convention. See: http://archives.postgresql.org/pgsql-hackers/2009-10/msg01769.php http://archives.postgresql.org/pgsql-hackers/2009-10/msg01441.php 2. Implement some kind of ANYRANGE type and associated operators; and provide a way to define new range types by providing the base type, difference type (e.g. for TIMESTAMP, the difference type would be INTERVAL) and a couple support functions. This might be more flexible, and it would obviate the need for marking operator families. If the second approach seems promising, we can hammer out a real proposal and see if it's viable. Regards, Jeff Davis -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Range types
Jeff Davis writes: > On Mon, 2009-12-14 at 13:32 -0500, Tom Lane wrote: >> The main question I would have is how to tell whether the underlying >> type is really discrete. > We can ask the user to provide a prior() and next() function, and if > they aren't provided, we assume it's continuous. Well, that still leaves us with the problem that Joe Schmo will file a bug when "create function next(float4) returns float4 as $$ select $1 + 0.1 $$" doesn't behave sanely for him. I'd prefer not to leave it to the user to decide whether a type is discrete or not. The traffic on pgsql-bugs is convincing evidence that a very large fraction of our user-base doesn't understand that floats are inexact :-( > I think "countable" is a more accurate word than "discrete". Strings are > discrete but not countable. It's been too long since college math classes for me to be sure whether "discrete" is really the exact term here. But I'm even more suspicious of "countable". I think a suitable diagonalization argument might show that strings are countable. That's getting a bit off-topic though... regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Hot Standby, release candidate?
On Mon, 2009-12-14 at 14:14 -0500, Tom Lane wrote: > Simon Riggs writes: > > On Mon, 2009-12-14 at 13:48 -0500, Tom Lane wrote: > >> Simon Riggs writes: > >>> * Disallow clustering system relations. This will definitely NOT work > >>> * for shared relations (we have no way to update pg_class rows in other > >>> * databases), nor for nailed-in-cache relations (the relfilenode values > >>> * for those are hardwired, see relcache.c). It might work for other > >>> * system relations, but I ain't gonna risk it. > >> > >>> I would presume we would not want to relax the restriction on CLUSTER > >>> working on these tables, even if new VACUUM FULL does. > >> > >> Why not? If we solve the problem of allowing these relations to change > >> relfilenodes, then CLUSTER would work just fine on them. Whether it's > >> particularly useful is not ours to decide I think. > > > I think you are probably right, but my wish to prove Schrodinger's Bug > > does not exist is not high enough for me personally to open that box > > this side of 8.6, especially when the previous code author saw it as a > > risk worth documenting. > > You're talking to the "previous code author" ... or at least I'm pretty > sure that comment is mine. Yeh, I figured, but I'm just as scared now as you were back then. This might allow CLUSTER to work, but it is definitely not something that I will enabling, testing and committing to fix *when* it breaks because my time is already allocated on other stuff. -- Simon Riggs www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Range types
On Mon, 2009-12-14 at 13:32 -0500, Tom Lane wrote: > Well, if the intention is to invent two different kinds of ranges, with > different operators, for continuous and discrete data types, then fine. Originally I thought most of the use cases were workable as discrete ranges. If people want continuous ranges, that's certainly doable by using a slightly different API. > But the original post suggested no such thing, and provided (unworkable) > examples suggesting that the intent was to force every type to be > treated as discrete whether that makes any sense or not. I agree, we shouldn't say we support continuous types, but force them to be treated like discrete types. > The main question I would have is how to tell whether the underlying > type is really discrete. We can ask the user to provide a prior() and next() function, and if they aren't provided, we assume it's continuous. Also, this range mechanism may be useful to get the meaningful operators for a range type. For instance, for a range join (e.g. a temporal join), we could recognize the && as "overlaps" and then find the "strictly left of" operator if we wanted to do a range merge join. This might be cleaner than the previous idea to mark operator families as conforming to a certain convention for strategy numbers. > (Also, stuff like strings simply doesn't have any sane concept of a > unique next or previous value. I think the number of types that are > really discrete in this sense is very small, like maybe just ints and > enums.) I think "countable" is a more accurate word than "discrete". Strings are discrete but not countable. Regards, Jeff Davis -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Hot Standby, release candidate?
Simon Riggs writes: > On Mon, 2009-12-14 at 13:48 -0500, Tom Lane wrote: >> Simon Riggs writes: >>> * Disallow clustering system relations. This will definitely NOT work >>> * for shared relations (we have no way to update pg_class rows in other >>> * databases), nor for nailed-in-cache relations (the relfilenode values >>> * for those are hardwired, see relcache.c). It might work for other >>> * system relations, but I ain't gonna risk it. >> >>> I would presume we would not want to relax the restriction on CLUSTER >>> working on these tables, even if new VACUUM FULL does. >> >> Why not? If we solve the problem of allowing these relations to change >> relfilenodes, then CLUSTER would work just fine on them. Whether it's >> particularly useful is not ours to decide I think. > I think you are probably right, but my wish to prove Schrodinger's Bug > does not exist is not high enough for me personally to open that box > this side of 8.6, especially when the previous code author saw it as a > risk worth documenting. You're talking to the "previous code author" ... or at least I'm pretty sure that comment is mine. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Streaming replication and non-blocking I/O
Tom Lane wrote: > Heikki Linnakangas writes: >> BTW, something that's been bothering me a bit with this patch is that we >> now have to link the backend with libpq. I don't see an immediate >> problem with that, but I'm not a packager. Does anyone see a problem >> with that? > > Yeah, I have a problem with that. What's the backend doing with libpq? > It's not receiving this data, it's sending it. walreceiver is a postmaster subprocess too. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Streaming replication and non-blocking I/O
Heikki Linnakangas writes: > Tom Lane wrote: >> Yeah, I have a problem with that. What's the backend doing with libpq? >> It's not receiving this data, it's sending it. > walreceiver is a postmaster subprocess too. Hm. Perhaps it should be a loadable plugin and not hard-linked into the backend? Compare dblink. The main concern I have with hard-linking libpq is that it has a lot of symbol conflicts with the backend --- and at least the ones from src/port/ aren't easily removed. I foresee problems that will be very difficult to fix on platforms where we can't filter the set of link symbols exposed by libpq. Linking a thread-enabled libpq into the backend could also create problems on some platforms --- it would likely cause a thread-capable libc to get linked, which is not what we want. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Range types
Tom Lane wrote: Scott Bailey writes: So basically I have an anyrange pseudo type with the functions prev, next, last, etc defined. So instead of hard coding range types, we would allow the user to define their own range types. Basically if we are able to determine the previous and next values of the base types we'd be able to define a range type. I'm envisioning in a manner much like defining an enum type. I think array types, not enums, would be a better model. I was referring to the syntax for how the user actually defined an enum not about it's implementation. Basically what I was hoping to get out of this thread was whether it was better to allow the user to define their own range types by specifying the base type and possibly the granularity and default inclusiveness of the end points, or if we should just provide the types like period and intrange? Scott -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Hot Standby, release candidate?
On Mon, 2009-12-14 at 13:48 -0500, Tom Lane wrote: > Simon Riggs writes: > > * Disallow clustering system relations. This will definitely NOT work > > * for shared relations (we have no way to update pg_class rows in other > > * databases), nor for nailed-in-cache relations (the relfilenode values > > * for those are hardwired, see relcache.c). It might work for other > > * system relations, but I ain't gonna risk it. > > > I would presume we would not want to relax the restriction on CLUSTER > > working on these tables, even if new VACUUM FULL does. > > Why not? If we solve the problem of allowing these relations to change > relfilenodes, then CLUSTER would work just fine on them. Whether it's > particularly useful is not ours to decide I think. I think you are probably right, but my wish to prove Schrodinger's Bug does not exist is not high enough for me personally to open that box this side of 8.6, especially when the previous code author saw it as a risk worth documenting. -- Simon Riggs www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Streaming replication and non-blocking I/O
Heikki Linnakangas writes: > It's going to be a bit more complicated in walsender/walreceiver to work > with the libpq COPY API. We're going to need a WAL sending/receiving > protocol on top of it, defined in terms of rows and columns passed > through the COPY protocol. AFAIR, libpq knows essentially nothing of the data being passed through COPY --- it just treats that as a byte stream. I think you can define any data format you want, it doesn't need to look exactly like a COPY of a table would. In fact it's probably a lot better if it DOESN'T look like COPY data once it gets past libpq, so that you can check that it is WAL and not COPY data. > One problem is the the standby is supposed to send back acknowledgments > to the master, telling it how far it has received/replayed the WAL. Is > there any way to send information back to the server, while a COPY OUT > is in progress? That's not absolutely necessary with asynchronous > replication, but will be with synchronous. Well, a real COPY would of course not stop to look for incoming messages, but I don't think that's inherent in the protocol. You would likely need some libpq adjustments so it didn't throw error when you tried that, but it would be a small and one-time adjustment. > BTW, something that's been bothering me a bit with this patch is that we > now have to link the backend with libpq. I don't see an immediate > problem with that, but I'm not a packager. Does anyone see a problem > with that? Yeah, I have a problem with that. What's the backend doing with libpq? It's not receiving this data, it's sending it. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Hot Standby, release candidate?
On Mon, 2009-12-14 at 20:32 +0200, Heikki Linnakangas wrote: > >> * You removed this comment from KnownAssignedXidsInit: > >> > >> - /* > >> -* XXX: We should check that we don't exceed maxKnownAssignedXids. > >> -* Even though the hash table might hold a few more entries than that, > >> -* we use fixed-size arrays of that size elsewhere and expected all > >> -* entries in the hash table to fit. > >> -*/ > >> > >> but AFAICS you didn't address the issue. It's referring to the 'xids' > >> array in TransactionIdIsInProgress(), which KnownAssignedXidsGet() fills > >> in without checking that it fits. > > > > I have ensured that they are always the same size, by definition, so no > > need to check. > > How did you ensure that? The hash table has no hard size limit. The hash table is in shared memory and the entry size is fixed. My understanding was that this meant the hash table was fixed in size and could not grow beyond the allocation. If that assumption was wrong, then yes we could get an error. Is it? Do you know from experience, or from code comments? Incidentally just picked up two much easier issues in that stretch of code. Thanks for making me look again! -- Simon Riggs www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Range types
Tom Lane wrote:
Scott Bailey writes:
Because intervals (mathematical not SQL) can be open or closed at each
end point we need to know what the next an previous value would be at
the specified granularity. And while you can do some operations without
knowing this, there are many you can't. For instance you could not tell
whether two [] or () ranges were adjacent, or be able to coalesce an
array of ranges.
This statement seems to me to demonstrate that you don't actually
understand the concept of open and closed ranges. It has nothing
whatsoever to do with assuming that the data type is discrete;
these concepts are perfectly well defined for the reals, for example.
What it is about is whether the inclusion conditions are "< bound"
or "<= bound".
I won't address how you draw your conclusions here. But I find it
'interesting' that you assume that I don't know what I'm talking about
rather than assume you don't fully understand what I'm talking about.
Anyhow. For any given range you may be 4 combinations of values. Either
the first value included in the range '[' or the last value preceding
the start of the range '('; and the last value included in the range ']'
or the first value following the end of the range ')'. We aren't going
to store all four data points so we need to normalize into the most
common form, a half-open interval [) and store just those two values.
The first value included in the range and the first value after the end
of our range.
So lets say you are using a numeric range to model the high and low
values of stocks trading on a given day. Now imagine doing this with no
concept of granularity. You will most likely be given a range [low,
high] with inclusive end points. So how do you convert that to a
closed-open interval so you can store it? Is 20.4201 the
next value after 20.42? Probably not. You are going to want to define
0.01 as the granularity for this (either type or column) so that 20.43 is.
Or again are the ranges [14.0, 22.0] and [22.1, 29.0] adjacent? Maybe,
maybe not. There is no way to tell w/o knowing the granularity. Perhaps
the granularity is 0.1 and there are a billion values that are
not included. Or perhaps the granularity is 0.1 and the are adjacent.
Scott
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Python 3.1 support
I wrote:
> On tor, 2009-11-12 at 16:06 -0500, Tom Lane wrote:
> > There was considerable debate earlier about whether we wanted to treat
> > Python 3 as a separate PL so it could be available in parallel with
> > plpython 2, because of the user-level coding incompatibilities. It
> > looks like this patch simply ignores that problem. What is going to
> > happen to plpython functions that depend on 2.x behavior?
>
> I have a proposal for how to handle this, and a prototype patch
> attached. This follows essentially what the CPython distribution itself
> does, which will make this tolerably easy to follow for users.
>
> We install plpython as plpython2.so or plpython3.so, depending on the
> version used to build it. Then, plpython.so is a symlink to
> plpython2.so.
So here is the potentially final patch for this, including the original
port of plpython.c itself, build system adjustments, and documentation.
Really attached this time.
diff --git a/config/python.m4 b/config/python.m4
index 9160a2b..32fff43 100644
--- a/config/python.m4
+++ b/config/python.m4
@@ -30,10 +30,12 @@ else
AC_MSG_ERROR([distutils module not found])
fi
AC_MSG_CHECKING([Python configuration directory])
+python_majorversion=`${PYTHON} -c "import sys; print(sys.version[[0]])"`
python_version=`${PYTHON} -c "import sys; print(sys.version[[:3]])"`
python_configdir=`${PYTHON} -c "from distutils.sysconfig import get_python_lib as f; import os; print(os.path.join(f(plat_specific=1,standard_lib=1),'config'))"`
python_includespec=`${PYTHON} -c "import distutils.sysconfig; print('-I'+distutils.sysconfig.get_python_inc())"`
+AC_SUBST(python_majorversion)[]dnl
AC_SUBST(python_version)[]dnl
AC_SUBST(python_configdir)[]dnl
AC_SUBST(python_includespec)[]dnl
diff --git a/configure b/configure
index 009a177..be51281 100755
--- a/configure
+++ b/configure
@@ -677,6 +677,7 @@ python_libdir
python_includespec
python_configdir
python_version
+python_majorversion
PYTHON
perl_embed_ldflags
perl_useshrplib
@@ -6964,6 +6965,7 @@ $as_echo "$as_me: error: distutils module not found" >&2;}
fi
{ $as_echo "$as_me:$LINENO: checking Python configuration directory" >&5
$as_echo_n "checking Python configuration directory... " >&6; }
+python_majorversion=`${PYTHON} -c "import sys; print(sys.version[0])"`
python_version=`${PYTHON} -c "import sys; print(sys.version[:3])"`
python_configdir=`${PYTHON} -c "from distutils.sysconfig import get_python_lib as f; import os; print(os.path.join(f(plat_specific=1,standard_lib=1),'config'))"`
python_includespec=`${PYTHON} -c "import distutils.sysconfig; print('-I'+distutils.sysconfig.get_python_inc())"`
diff --git a/doc/src/sgml/installation.sgml b/doc/src/sgml/installation.sgml
index 4746e68..f9221c9 100644
--- a/doc/src/sgml/installation.sgml
+++ b/doc/src/sgml/installation.sgml
@@ -195,8 +195,12 @@ su - postgres
To build the PL/Python server programming
language, you need a Python
- installation with the header files and the distutils module.
- The minimum required version is Python 2.2.
+ installation with the header files and
+ the distutils module. The minimum
+ required version is Python
+ 2.2. Python 3 is supported with
+ version 3.1 or later; but see
+ when using Python 3.
diff --git a/doc/src/sgml/plpython.sgml b/doc/src/sgml/plpython.sgml
index 502a5bc..e6c998a 100644
--- a/doc/src/sgml/plpython.sgml
+++ b/doc/src/sgml/plpython.sgml
@@ -14,7 +14,8 @@
To install PL/Python in a particular database, use
- createlang plpythonu dbname.
+ createlang plpythonu dbname (but
+ see also ).
@@ -42,6 +43,112 @@
+
+ Python 2 vs. Python 3
+
+
+ PL/Python supports both the Python 2 and Python 3 language
+ variants. (The PostgreSQL installation instructions might contain
+ more precise information about the exact supported minor versions
+ of Python.) Because the Python 2 and Python 3 language variants
+ are incompatible in some important aspects, the following naming
+ and transitioning scheme is used by PL/Python to avoid mixing them:
+
+
+
+
+ The PostgreSQL language named plpython2u
+ implements PL/Python based on the Python 2 language variant.
+
+
+
+
+
+ The PostgreSQL language named plpython3u
+ implements PL/Python based on the Python 3 language variant.
+
+
+
+
+
+ The language named plpythonu implements
+ PL/Python based on the default Python language variant, which is
+ currently Python 2. (This default is independent of what any
+ local Python installations might consider to be
+ their default, for example,
+ what /usr/bin/python might be.) The
+ default will probably be changed to Python 3 in a distant future
+ release of PostgreSQL, depending on the progress of the
+ migration to Python 3 in the Python community.
+
+
+
+
+ It depends on the bu
Re: [HACKERS] Hot Standby, release candidate?
Simon Riggs writes: > * Disallow clustering system relations. This will definitely NOT work > * for shared relations (we have no way to update pg_class rows in other > * databases), nor for nailed-in-cache relations (the relfilenode values > * for those are hardwired, see relcache.c). It might work for other > * system relations, but I ain't gonna risk it. > I would presume we would not want to relax the restriction on CLUSTER > working on these tables, even if new VACUUM FULL does. Why not? If we solve the problem of allowing these relations to change relfilenodes, then CLUSTER would work just fine on them. Whether it's particularly useful is not ours to decide I think. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Python 3.1 support
On Mon, Dec 14, 2009 at 1:42 PM, Peter Eisentraut wrote: > I wrote: >> On tor, 2009-11-12 at 16:06 -0500, Tom Lane wrote: >> > There was considerable debate earlier about whether we wanted to treat >> > Python 3 as a separate PL so it could be available in parallel with >> > plpython 2, because of the user-level coding incompatibilities. It >> > looks like this patch simply ignores that problem. What is going to >> > happen to plpython functions that depend on 2.x behavior? >> >> I have a proposal for how to handle this, and a prototype patch >> attached. This follows essentially what the CPython distribution itself >> does, which will make this tolerably easy to follow for users. >> >> We install plpython as plpython2.so or plpython3.so, depending on the >> version used to build it. Then, plpython.so is a symlink to >> plpython2.so. > > So here is the potentially final patch for this, including the original > port of plpython.c itself, build system adjustments, and documentation. I think you forgot to actually attach it... ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Streaming replication and non-blocking I/O
Tom Lane wrote: > The very, very large practical problem with this is that if you decide > to change the behavior at any time, the only way to be sure that the WAL > receiver is using the right libpq version is to perform a soname major > version bump. The transformations done by libpq will essentially become > part of its ABI, and not a very visible part at that. Not having to change the libpq API would certainly be a big advantage. It's going to be a bit more complicated in walsender/walreceiver to work with the libpq COPY API. We're going to need a WAL sending/receiving protocol on top of it, defined in terms of rows and columns passed through the COPY protocol. One problem is the the standby is supposed to send back acknowledgments to the master, telling it how far it has received/replayed the WAL. Is there any way to send information back to the server, while a COPY OUT is in progress? That's not absolutely necessary with asynchronous replication, but will be with synchronous. One idea is to stop/start the COPY between every batch of WAL records sent, giving the client (= walreceiver) a chance to send messages back. But that will lead to extra round trips. BTW, something that's been bothering me a bit with this patch is that we now have to link the backend with libpq. I don't see an immediate problem with that, but I'm not a packager. Does anyone see a problem with that? -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Aggregate ORDER BY patch
> "Tom" == Tom Lane writes: >> Updated version of the aggregate order by patch. Tom> I'm starting to look at this now. I find it rather bizarre to Tom> merge both the actual arguments of an aggregate and the optional Tom> ORDER BY expressions into a single targetlist. It doesn't seem Tom> like that would be an especially convenient representation to Tom> work with, It's extremely convenient, since you need the arguments and the ordering expressions together in a slot in order to feed them to tuplesort (in the general case where there is more than one expression); you need a tupledesc to pass to tuplesort; and there are existing functions to construct all of these things from the tlist. Also, you want to merge the argument expressions and ordering expressions where possible, and this is exactly what the existing transformSortClause stuff expects. Tom> and I would also expect there to be a nonzero performance hit Tom> from the extra TargetEntry expression nodes, even when the Tom> feature is not in use. I tested for that and couldn't reliably detect any (certainly <2% on count(i) on generated data, and under the circumstances I was testing that's not necessarily outside the error margin). -- Andrew (irc:RhodiumToad) -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Python 3.1 support
I wrote: > On tor, 2009-11-12 at 16:06 -0500, Tom Lane wrote: > > There was considerable debate earlier about whether we wanted to treat > > Python 3 as a separate PL so it could be available in parallel with > > plpython 2, because of the user-level coding incompatibilities. It > > looks like this patch simply ignores that problem. What is going to > > happen to plpython functions that depend on 2.x behavior? > > I have a proposal for how to handle this, and a prototype patch > attached. This follows essentially what the CPython distribution itself > does, which will make this tolerably easy to follow for users. > > We install plpython as plpython2.so or plpython3.so, depending on the > version used to build it. Then, plpython.so is a symlink to > plpython2.so. So here is the potentially final patch for this, including the original port of plpython.c itself, build system adjustments, and documentation. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Hot Standby, release candidate?
On Sun, 2009-12-13 at 22:25 +, Simon Riggs wrote: > * Which exact tables are we talking about: just pg_class and the shared > catalogs? Everything else is in pg_class, so if we can find it we're OK? > formrdesc() tells me the list of nailed relations is: pg_database, > pg_class, pg_attribute, pg_proc, and pg_type. Are the nailed relations > the ones we care about, or are they just a subset? Comments in cluster.c's check_index_is_clusterable() suggest that the list of tables to which this applies is nailed relations *and* shared relations, plus their indexes. /* * Disallow clustering system relations. This will definitely NOT work * for shared relations (we have no way to update pg_class rows in other * databases), nor for nailed-in-cache relations (the relfilenode values * for those are hardwired, see relcache.c). It might work for other * system relations, but I ain't gonna risk it. */ So that means we need to handle 3 cases: nailed-local, nailed-shared and non-nailed-shared. I would presume we would not want to relax the restriction on CLUSTER working on these tables, even if new VACUUM FULL does. Anyway, not going to be done for Alpha3, but seems fairly doable now. -- Simon Riggs www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Hot Standby, release candidate?
On Mon, 2009-12-14 at 18:02 +, Greg Stark wrote: > >> It doesn't seem any more wrong than using hash indexes right after > >> recovery, crash recovery or otherwise. It's certainly broken, but I > >> don't see much value in a partial fix. The bottom line is that hash > >> indexes should be WAL-logged. > > > > I know that's your thought; I'm just checking its everyone else's as > > well. We go to great lengths elsewhere in the patch to avoid queries > > returning incorrect results and there is a loss of capability as a > > result. I don't want Hash index users to view this as a feature. I > don't > > feel too strongly, but it can be argued both ways, at least. > > This goes back to your pluggable rmgr point. Someone could add a new > index method and get bogus results on their standby. And unlike hash > indexes where there's some hope of addressing the problem there's > nothing they can do to fix this. > > It does seem like having a flag in the catalog to mark nonrecoverable > indexes and make them unavailable to query plans on the standby would > be worth its weight in code. pg_am.amalmostworks or perhaps pg_am.amhalffinished... :-) I wouldn't wish to literally persist that situation, especially since if we had it we couldn't update it during recovery. We need to allow pluggable indexes in full, not just partially. I think we should extend pg_am so that rmgr routines can be defined for them also, with dynamic assignment of rmgrids, recorded in file so we can use them during recovery. What we also need is a mechanism to identify and mark indexes as corrupt while they are being rebuilt, so recovery can complete without them and then rebuild automatically when recovery finishes. And so they can be skipped during hot standby. -- Simon Riggs www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Hot Standby, release candidate?
Simon Riggs wrote: > On Mon, 2009-12-14 at 11:54 +0200, Heikki Linnakangas wrote: >> Simon Riggs wrote: >> * Are you planning to remove the recovery_connections setting before >> release? The documentation makes it sound like it's a temporary hack >> that we're not really sure is needed at all. That's not very comforting. > > It has been requested and I agree, so its there. Saying it might be > removed in future is no more than we do elsewhere and AFAIK we all hope > it will be. Not sure why that is or isn't comforting. Now that recovery_connections has a double-role, and does in the master what the wal_standby_info used to do, the documentation probably should be clarified that the whole parameter is not going to go away, just the role in the master. >> * You removed this comment from KnownAssignedXidsInit: >> >> - /* >> -* XXX: We should check that we don't exceed maxKnownAssignedXids. >> -* Even though the hash table might hold a few more entries than that, >> -* we use fixed-size arrays of that size elsewhere and expected all >> -* entries in the hash table to fit. >> -*/ >> >> but AFAICS you didn't address the issue. It's referring to the 'xids' >> array in TransactionIdIsInProgress(), which KnownAssignedXidsGet() fills >> in without checking that it fits. > > I have ensured that they are always the same size, by definition, so no > need to check. How did you ensure that? The hash table has no hard size limit. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Range types
Jeff Davis writes: > On Mon, 2009-12-14 at 11:25 -0500, Tom Lane wrote: >> This statement seems to me to demonstrate that you don't actually >> understand the concept of open and closed ranges. > Of course you can still define the obvious "contains" and "overlaps" > operators for a continuous range. But there are specific differences in > the API between a discrete range and a continuous range, which is what > Scott was talking about. Well, if the intention is to invent two different kinds of ranges, with different operators, for continuous and discrete data types, then fine. But the original post suggested no such thing, and provided (unworkable) examples suggesting that the intent was to force every type to be treated as discrete whether that makes any sense or not. The main question I would have is how to tell whether the underlying type is really discrete. If we allow people to define things like "range over float4 with 0.01 step", then somebody will try to do it --- and file bugs when it doesn't work sanely. I don't think there is anything in our API for user-defined types that really tells you whether it's an exact or approximate type. (Also, stuff like strings simply doesn't have any sane concept of a unique next or previous value. I think the number of types that are really discrete in this sense is very small, like maybe just ints and enums.) regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Range types
Jeff Davis wrote: > So there are certainly some user-visible API differences between the > two, and I don't think those differences can be hidden. ISTM you are saying we should have two types of range types. -- Alvaro Herrerahttp://www.CommandPrompt.com/ The PostgreSQL Company - Command Prompt, Inc. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Range types
On Mon, 2009-12-14 at 10:00 -0800, Nathan Boley wrote: > IMHO the first question is whether, for integers, [1,2] UNION [3,5] > should be equal to [1,5]. In math this is certainly true, and defining > 'next' seems like a reasonable way to establish this in postgres. [ you say "yes" ] Agreed. > The next question is whether, for floats, [1,3-FLT_EPSILON] UNION > [3,5] should be [1,5]. [ you say "no" ] I think this should be true, because all floats between 1 and 5 are contained. I don't feel too strongly about this, so I would not complain if floats were treated as continuous. > And the next question is whether, for numeric(6,2), [1,2.99] UNION > [3,5] should be [1,5]. [ you say "yes" ] I almost agree. Unfortunately, typmod isn't really a part of the type, it just affects input/output. So, we can't really use it that way -- as Tom points out, typmod is not passed along to functions that take the value. But if it were a part of the type, then I would certainly agree. Regards, Jeff Davis -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Range types
Nathan Boley writes: >> This statement seems to me to demonstrate that you don't actually >> understand the concept of open and closed ranges. > IMHO the first question is whether, for integers, [1,2] UNION [3,5] > should be equal to [1,5]. In math this is certainly true, and defining > 'next' seems like a reasonable way to establish this in postgres. Well, that's nice to have (not essential) for data types that actually are discrete. It's not a sufficient argument for creating a definition that is flat out broken for non-discrete datatypes. It might be worth pointing out here that the concept of an open interval was only invented in the first place for dealing with a continuum. If you could assume the underlying set is discrete, every open interval could be replaced with a closed one, using the next or prior value as the bound instead. There would be no need for two kinds of interval. If you are intending to support UNION on intervals, you are going to need some more-general representation anyway. (I trust you don't think we will accept a datatype for which [1,2] UNION [3,5] works but [1,2] UNION [4,5] throws an error.) So whether the code can reduce the result of a union to a single range or has to keep it as two ranges is only an internal optimization issue anyhow, not something that should drive an artificial (and infeasible) attempt to force every domain to be discrete. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Range types
On Mon, 2009-12-14 at 09:58 -0500, Tom Lane wrote: > In particular, the granularity examples you give seem to assume that > the underlying datatype is exact not approximate --- which among other > things will mean that it fails to work for float timestamps. Since > timestamps are supposedly the main use-case, that's pretty troubling. Additionally, granularity for timestamps is not particularly useful when you get to things like "days" and "months" which don't have a clean algebra. Is the granule there only to try to support continuous ranges? If so, I don't think it's necessary if we expose the API differences I outlined in another email in this thread. Also, that would mean that we don't need a granule for float, because we can already treat it as discrete*. Regards, Jeff Davis *: nextafter() allows you to increment or decrement a double (loosely speaking), and according to the man page it's part of C99 and POSIX.1-2001. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Hot Standby, release candidate?
Greg Smith wrote: > Heikki Linnakangas wrote: >> I note that if it was easy to run pgindent yourself on a patch before >> committing/submitting, we wouldn't need to have this discussion. I don't >> know hard it is to get it working right, but I recall I tried once and >> gave up. >> > What sort of problems did you run into? I don't remember, unfortunately. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Range types
On Mon, 2009-12-14 at 11:25 -0500, Tom Lane wrote: > Scott Bailey writes: > > Because intervals (mathematical not SQL) can be open or closed at each > > end point we need to know what the next an previous value would be at > > the specified granularity. And while you can do some operations without > > knowing this, there are many you can't. For instance you could not tell > > whether two [] or () ranges were adjacent, or be able to coalesce an > > array of ranges. > > This statement seems to me to demonstrate that you don't actually > understand the concept of open and closed ranges. It has nothing > whatsoever to do with assuming that the data type is discrete; > these concepts are perfectly well defined for the reals, for example. > What it is about is whether the inclusion conditions are "< bound" > or "<= bound". Of course you can still define the obvious "contains" and "overlaps" operators for a continuous range. But there are specific differences in the API between a discrete range and a continuous range, which is what Scott was talking about. 1. With discrete ranges, you can change the displayed inclusivity/exclusivity without changing the value. For instance in the integer domain, [ 5, 10 ] is the same value as [ 5, 11 ). This works on both input and output. It is not possible to change the display for continuous ranges. 2. With discrete ranges, you can get the last point before the range, the first point in the range, the last point in the range, and the first point after the range. These are more than enough to describe the range completely. For continuous ranges, those functions will fail depending on the inclusivity/exclusivity of the range. Practically speaking, you would want to have a different set of accessors: start_inclusive(), start_point(), end_point(), and end_inclusive(). However, those functions can't be usefully defined on a discrete range. We can't choose the API for continuous ranges as the API for discrete ranges as well. If we did, then we would think that [5, 10] and [5, 11) were not equal, but they are. Similarly, we would think that [5, 10] and [11, 12] were not adjacent, but they are. So there are certainly some user-visible API differences between the two, and I don't think those differences can be hidden. Regards, Jeff Davis -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Aggregate ORDER BY patch
Andrew Gierth writes: > Updated version of the aggregate order by patch. I'm starting to look at this now. I find it rather bizarre to merge both the actual arguments of an aggregate and the optional ORDER BY expressions into a single targetlist. It doesn't seem like that would be an especially convenient representation to work with, and I would also expect there to be a nonzero performance hit from the extra TargetEntry expression nodes, even when the feature is not in use. Why didn't you use separate lists? regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Hot Standby, release candidate?
On Mon, Dec 14, 2009 at 11:07 AM, Simon Riggs wrote: >> * vacuum_defer_cleanup_age is very hard to tune. You'll need an estimate >> on your transaction rate to begin with. Do we really want this setting >> in its current form? Does it make sense as PGC_USERSET, as if one >> backend uses a lower setting than others, that's the one that really >> determines when transactions are killed in the standby? I think this >> should be discussed and implemented as a separate patch. > > All the vacuum_*_age parameters have this characteristic, yet they > exist. I think it makes sense to have it be userset because someone could run vacuum on one table with a different defer_cleanup_age for some reason. I admit I'm having trouble coming up with a good use case. Personally I'm fine with having parameters like this in alphas that end up being renamed, or changed to different semantics, or even removed by the time it's launched. >> It doesn't seem any more wrong than using hash indexes right after >> recovery, crash recovery or otherwise. It's certainly broken, but I >> don't see much value in a partial fix. The bottom line is that hash >> indexes should be WAL-logged. > > I know that's your thought; I'm just checking its everyone else's as > well. We go to great lengths elsewhere in the patch to avoid queries > returning incorrect results and there is a loss of capability as a > result. I don't want Hash index users to view this as a feature. I don't > feel too strongly, but it can be argued both ways, at least. This goes back to your pluggable rmgr point. Someone could add a new index method and get bogus results on their standby. And unlike hash indexes where there's some hope of addressing the problem there's nothing they can do to fix this. It does seem like having a flag in the catalog to mark nonrecoverable indexes and make them unavailable to query plans on the standby would be worth its weight in code. -- greg -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Range types
>> Because intervals (mathematical not SQL) can be open or closed at each >> end point we need to know what the next an previous value would be at >> the specified granularity. And while you can do some operations without >> knowing this, there are many you can't. For instance you could not tell >> whether two [] or () ranges were adjacent, or be able to coalesce an >> array of ranges. > > This statement seems to me to demonstrate that you don't actually > understand the concept of open and closed ranges. It has nothing > whatsoever to do with assuming that the data type is discrete; > these concepts are perfectly well defined for the reals, for example. > What it is about is whether the inclusion conditions are "< bound" > or "<= bound". IMHO the first question is whether, for integers, [1,2] UNION [3,5] should be equal to [1,5]. In math this is certainly true, and defining 'next' seems like a reasonable way to establish this in postgres. The next question is whether, for floats, [1,3-FLT_EPSILON] UNION [3,5] should be [1,5]. And the next question is whether, for numeric(6,2), [1,2.99] UNION [3,5] should be [1,5]. FWIW, I would answer yes, no, yes to those three questions. -Nathan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] new CommitFest states
On Mon, Dec 14, 2009 at 12:38 PM, Greg Smith wrote: > Robert Haas wrote: >> I don't think there should be a transition from Returned with Feedback >> back to Waiting for review. Granted we might allow that occasionally >> as an exceptional case, but normally Returned with Feedback is a final >> state. > > I did throw some disclaimers in the notes about this particular subject at > the bottom of the table. The main reason I put that in there is that > sometimes a reviewer or even the CF manager (I did this myself once this > time) will mark something "Returned with feedback", thinking there's no way > the issues pointed out can be addressed right now. And then, a day or two > later, in comes a patch that does just that; surprise! Since it seems to > happen anyway, and I'd prefer not to get in the position where people are > screaming "you threw me out with 'RWF' unfairly", I thought it was better to > accept that possibility so long as the whole thing is tightly bounded as far > as how much time the author has to do it. Hmm, I'm not aware of any actual cases of this. I'm usually pretty conservative about jumping to RWF unless there's been lag or we're near the end of the CommitFest, so it doesn't come up. >> (Also, Waiting for review is actually the wrong name for the state >> it's trying to talk about.) > > Uh, what are you talking about here? Well, we have Needs Review and Waiting on Author, but not Waiting for Review. I assume you mean Needs Review. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] new CommitFest states
Robert Haas wrote: I don't think there should be a transition from Returned with Feedback back to Waiting for review. Granted we might allow that occasionally as an exceptional case, but normally Returned with Feedback is a final state. I did throw some disclaimers in the notes about this particular subject at the bottom of the table. The main reason I put that in there is that sometimes a reviewer or even the CF manager (I did this myself once this time) will mark something "Returned with feedback", thinking there's no way the issues pointed out can be addressed right now. And then, a day or two later, in comes a patch that does just that; surprise! Since it seems to happen anyway, and I'd prefer not to get in the position where people are screaming "you threw me out with 'RWF' unfairly", I thought it was better to accept that possibility so long as the whole thing is tightly bounded as far as how much time the author has to do it. (Also, Waiting for review is actually the wrong name for the state it's trying to talk about.) Uh, what are you talking about here? -- Greg Smith2ndQuadrant Baltimore, MD PostgreSQL Training, Services and Support g...@2ndquadrant.com www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] new CommitFest states
On Mon, Dec 14, 2009 at 12:01 PM, Greg Smith wrote: > Kevin Grittner wrote: > > http://wiki.postgresql.org/wiki/Running_a_CommitFest > > > > It seems to me that a patch could move from "Discussing review" to > "Needs review" -- if the reviewer decided to discuss the approach > before continuing the review process and the discussion confirms the > approach as viable. > > > In that case, the patch would be in "Needs review" the whole time. > "Discussing review" is intended to be a "I'm done but not sure of the next > step for this patch" state the reviewer can use. In the situation you > described, the patch would never have left "Needs review". I just made that > more clear by documenting that it's shorthand for "discussing review > results". > > I also added a transition path for a similar situation though, where the > discussion concludes the reviewer didn't do the right thing in the first > place (even though they thought they did) and they return to reviewing after > realizing what was missing. I don't think there should be a transition from Returned with Feedback back to Waiting for review. Granted we might allow that occasionally as an exceptional case, but normally Returned with Feedback is a final state. (Also, Waiting for review is actually the wrong name for the state it's trying to talk about.) ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Hot Standby, release candidate?
Heikki Linnakangas wrote: I note that if it was easy to run pgindent yourself on a patch before committing/submitting, we wouldn't need to have this discussion. I don't know hard it is to get it working right, but I recall I tried once and gave up. What sort of problems did you run into? -- Greg Smith2ndQuadrant Baltimore, MD PostgreSQL Training, Services and Support g...@2ndquadrant.com www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] new CommitFest states
Kevin Grittner wrote: http://wiki.postgresql.org/wiki/Running_a_CommitFest It seems to me that a patch could move from "Discussing review" to "Needs review" -- if the reviewer decided to discuss the approach before continuing the review process and the discussion confirms the approach as viable. In that case, the patch would be in "Needs review" the whole time. "Discussing review" is intended to be a "I'm done but not sure of the next step for this patch" state the reviewer can use. In the situation you described, the patch would never have left "Needs review". I just made that more clear by documenting that it's shorthand for "discussing review results". I also added a transition path for a similar situation though, where the discussion concludes the reviewer didn't do the right thing in the first place (even though they thought they did) and they return to reviewing after realizing what was missing. -- Greg Smith2ndQuadrant Baltimore, MD PostgreSQL Training, Services and Support g...@2ndquadrant.com www.2ndQuadrant.com
Re: [HACKERS] pgAdmin III: timestamp displayed in what time zone?
Fred Janon wrote: Sorry if if's a double post, but I thought that it would be more likely I would get an answer on the hackers list. In that case just posting here would have been better than hitting both. I usually ignore any request for help that is posted on more than one list just to draw attention to itself; I wouldn't be surprised that are other people who are similarly annoyed by the behavior and do likewise. In your case, both lists you picked were the wrong ones, since you're asking a pgAdmin specific question which isn't the direct topic of any of the pgsql-* lists; pgadmin-support was the right place for it: http://archives.postgresql.org/pgadmin-support/ -- Greg Smith2ndQuadrant Baltimore, MD PostgreSQL Training, Services and Support g...@2ndquadrant.com www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Range types
Scott Bailey writes: > Because intervals (mathematical not SQL) can be open or closed at each > end point we need to know what the next an previous value would be at > the specified granularity. And while you can do some operations without > knowing this, there are many you can't. For instance you could not tell > whether two [] or () ranges were adjacent, or be able to coalesce an > array of ranges. This statement seems to me to demonstrate that you don't actually understand the concept of open and closed ranges. It has nothing whatsoever to do with assuming that the data type is discrete; these concepts are perfectly well defined for the reals, for example. What it is about is whether the inclusion conditions are "< bound" or "<= bound". regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Hot Standby, release candidate?
Tom Lane wrote: > Robert Haas writes: > > On Mon, Dec 14, 2009 at 6:35 AM, Simon Riggs wrote: > >> Why is (1) important, and if it is important, why is it being mentioned > >> only now? Are we saying that all previous reviewers of my work (and > >> others') removed these without ever mentioning they had done so? > > > pgident will remove such white spaces and create merge conflicts for > > everyone working on those areas of the code. > > What I try really hard to remove from committed patches is spurious > whitespace changes to pre-existing code. Whether new code blocks > exactly match pgindent's rules is less of a concern, but changing > code you don't have to in a way that pgindent will undo later anyway > is just useless creation of potential conflicts. > > The whole thing would be a lot easier if someone would put together an > easily-installable version of pgindent. Bruce has posted the patches he > uses but I don't know what version of indent they're against... The entire indent tarball with patches is on our ftp site. -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.com + If your life is a hard drive, Christ can be your backup. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Range types
Martijn van Oosterhout wrote: On Sun, Dec 13, 2009 at 11:49:53PM -0800, Scott Bailey wrote: So basically I have an anyrange pseudo type with the functions prev, next, last, etc defined. So instead of hard coding range types, we would allow the user to define their own range types. Basically if we are able to determine the previous and next values of the base types we'd be able to define a range type. I'm envisioning in a manner much like defining an enum type. I find it odd that you could define functions next() and prev() since that assumes some kind of dicretisation which simply does not exist for most types I can think of. Because intervals (mathematical not SQL) can be open or closed at each end point we need to know what the next an previous value would be at the specified granularity. And while you can do some operations without knowing this, there are many you can't. For instance you could not tell whether two [] or () ranges were adjacent, or be able to coalesce an array of ranges. It would seem to me the real useful uses of ranges would be the operations overlaps, disjoint, proceeds, follows, etc, which could all be defined on any well-ordered type (wherever greater-than and less-than are well defined). No need to discretise anything. Do you have any actual usecase for a distretized range type for timestamp? -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] WAL Info messages
Simon Riggs writes: > I definitely wouldn't presume that anybody using Hot Standby would > necessarily want NOTIFY to reach the standby, especially if there was an > overhead to doing so. If using NOTIFY is the favoured approach, I would > add a separate parameter for it and/or an explicit option on NOTIFY. Yeah, I had just come to the same conclusion: you'd want a separate spigot handle on sending NOTIFY to WAL, regardless of any general flag about HS. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Streaming replication and non-blocking I/O
Fujii Masao writes: > I'm going to set the new function calling ereport as the current notice > processor by using PQsetNoticeProcessor. But the problem is that only the > completed message like "NOTICE: xxx" is passed to such notice processor, > i.e., the error level itself is not passed. Use PQsetNoticeReceiver. The other one is just there for backwards compatibility. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Hot Standby, release candidate?
On Mon, Dec 14, 2009 at 10:49 AM, Simon Riggs wrote: > On Mon, 2009-12-14 at 09:14 -0600, Kevin Grittner wrote: >> Simon Riggs wrote: >> >> > If we are going to run pgindent anyway, what is the point? >> >> Perhaps it would take less time to run this than to argue the point?: >> >> sed -e 's/[ \t][ \t]*$//' -e 's/ *\t/\t/g' * > > Not certain that is exactly correct, plus it doesn't only work against a > current patch since there are already many examples of whitespace in CVS > already. I do appreciate your attempts at resolution and an easy > tool-based approach for the future, though I'll let someone else run > with it. Yeah, that would actually be a disaster, because it would actually add deltas to the patch in the short term. Seems to me that we would be better off figuring out which exact ident Bruce is running and checking the typedef list into CVS. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Hot Standby, release candidate?
On Mon, Dec 14, 2009 at 11:09:49AM +0100, Magnus Hagander wrote: > On Mon, Dec 14, 2009 at 10:54, Heikki Linnakangas > wrote: > > * Please remove any spurious whitespace. "git diff --color" makes > > them stand out like a sore thumb, in red. (pgindent will fix them > > but always better to fix them before committing, IMO). > > +1 in general, not particularly for this patch (haven't checked that > in this patch). > > Actually, how about we add that to the page at > http://wiki.postgresql.org/wiki/Submitting_a_Patch? Done. Cheers, David. -- David Fetter http://fetter.org/ Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter Skype: davidfetter XMPP: david.fet...@gmail.com iCal: webcal://www.tripit.com/feed/ical/people/david74/tripit.ics Remember to vote! Consider donating to Postgres: http://www.postgresql.org/about/donate -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Hot Standby, release candidate?
On Mon, 2009-12-14 at 09:14 -0600, Kevin Grittner wrote: > Simon Riggs wrote: > > > If we are going to run pgindent anyway, what is the point? > > Perhaps it would take less time to run this than to argue the point?: > > sed -e 's/[ \t][ \t]*$//' -e 's/ *\t/\t/g' * Not certain that is exactly correct, plus it doesn't only work against a current patch since there are already many examples of whitespace in CVS already. I do appreciate your attempts at resolution and an easy tool-based approach for the future, though I'll let someone else run with it. -- Simon Riggs www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] new CommitFest states
Greg Smith wrote: > I didn't really get any feedback on my proposal to add a new > "Discussing review" state It seems like a good idea to me; it better models the reality. > http://wiki.postgresql.org/wiki/Running_a_CommitFest It seems to me that a patch could move from "Discussing review" to "Needs review" -- if the reviewer decided to discuss the approach before continuing the review process and the discussion confirms the approach as viable. -Kevin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Hot Standby, release candidate?
Simon Riggs wrote: > On Mon, 2009-12-14 at 11:09 +0100, Magnus Hagander wrote: >> On Mon, Dec 14, 2009 at 10:54, Heikki Linnakangas >> wrote: >>> * Please remove any spurious whitespace. "git diff --color" makes them >>> stand out like a sore thumb, in red. (pgindent will fix them but always >>> better to fix them before committing, IMO). >> +1 in general, not particularly for this patch (haven't checked that >> in this patch). >> >> Actually, how about we add that to the page at >> http://wiki.postgresql.org/wiki/Submitting_a_Patch? > > If we can define "spurious whitespace" it would help decide whether > there is any action to take, and when. There's two definitions that are useful: - Anything that "git diff --color" shows up as glaring red. Important to remove simply because the red whitespace is distracting when I review a patch. - Anything that pgindent would/will eventually fix. Because you might as well fix them before committing and save the extra churn and potential (although trivial) merge conflicts in people's outstanding patches when pgindent is run. I don't run pgindent myself, so I wouldn't notice most stuff, but I tend to fix things that I do notice. > Why is (1) important, and if it is important, why is it being mentioned > only now? Are we saying that all previous reviewers of my work (and > others') removed these without ever mentioning they had done so? I did it in one pass, Oct 15th according to git log. I tend to silently fix whitespace issues like that in all patches I commit. It's generally trivial enough to fix that it's easier to just fix it myself than complain, explain, and look like a real nitpick. I note that if it was easy to run pgindent yourself on a patch before committing/submitting, we wouldn't need to have this discussion. I don't know hard it is to get it working right, but I recall I tried once and gave up. Any volunteers? -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Hot Standby, release candidate?
Tom Lane escribió: > The whole thing would be a lot easier if someone would put together an > easily-installable version of pgindent. Bruce has posted the patches he > uses but I don't know what version of indent they're against... And we're still unclear on the typedef list that's going to be used. -- Alvaro Herrerahttp://www.CommandPrompt.com/ The PostgreSQL Company - Command Prompt, Inc. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Hot Standby, release candidate?
On Mon, Dec 14, 2009 at 10:08 AM, Simon Riggs wrote: > On Mon, 2009-12-14 at 09:32 -0500, Robert Haas wrote: >> If every patch perfectly matched the pgident style, then the pgindent >> run would change nothing and we would all be VERY happy. > > I've made all whitespace changes to the patch. > > I understand the reason for acting as you suggest, but we either police > it, or we don't. If we don't police in all cases then I'm not personally > happy to be policed. I am doing my best to police it, and certainly will police it for anything that I review or commit. Heikki was the one who originally pointed it on this thread, Magnus gave a +1, and I am pretty sure Tom tries to keep an eye out for it, too, so generally I would say it is project practice. Obviously I am not able to control the actions of all the other committers, and it does appear that some crap has crept in since the last pgindent run. :-( At any rate I don't think you're being singled out for special treatment. > About 90% of the whitespace in the patch were in docs, README or test > output files. A great many of that type of file have numerous line > ending whitespace, not introduced by me, so it seems to me that this has > never been adequately policed in the way you say. If we really do care > about this issue then I expect people to look a little further than just > my patch. pgindent won't reindent the docs or the README, but git diff --check picks up on trailing whitespace, so it may be that Tom (who doesn't use git but does commit a lot of patches) is less finicky about trailing whitespace in those places. If we move to git across the board, I expect this to get more standardized handling, but I think we have a ways to go on that yet. > Portability of patches across releases isn't a huge issue for me, nor > for the project, I think, but I am willing to commit to cleaning future > patches in this way. It's a pretty significant issue for me personally, and anyone who is maintaining a fork of the main source base that has to be merged when the pgindent run hits. It would be nice if someone wanted to build a better mousetrap here, but so far no volunteers. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Hot Standby, release candidate?
Robert Haas writes: > On Mon, Dec 14, 2009 at 6:35 AM, Simon Riggs wrote: >> Why is (1) important, and if it is important, why is it being mentioned >> only now? Are we saying that all previous reviewers of my work (and >> others') removed these without ever mentioning they had done so? > pgident will remove such white spaces and create merge conflicts for > everyone working on those areas of the code. What I try really hard to remove from committed patches is spurious whitespace changes to pre-existing code. Whether new code blocks exactly match pgindent's rules is less of a concern, but changing code you don't have to in a way that pgindent will undo later anyway is just useless creation of potential conflicts. The whole thing would be a lot easier if someone would put together an easily-installable version of pgindent. Bruce has posted the patches he uses but I don't know what version of indent they're against... regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] WAL Info messages
On Mon, 2009-12-14 at 09:51 -0500, Tom Lane wrote: > Greg Stark writes: > > What happens on the slave when normal NOTIFYs are generated on the > > master? IIRC NOTIFYs are wal-logged so I imagine LISTEN on the slave > > would just work and a plain old NOTIFY/LISTEN would suffice for this > > use case? > > Nothing happens: NOTIFYs are *not* WAL-logged, and it would be quite > against the intended use-case to have them be. > > If you're hoping that processes on the slave could LISTEN for events > happening on the master, we'd have a bit of a conflict. What I've proposed is essentially a variant of NOTIFY on master, LISTEN on slave, as Greg suggests. Almost identical if we have NOTIFY-with-payload. I definitely wouldn't presume that anybody using Hot Standby would necessarily want NOTIFY to reach the standby, especially if there was an overhead to doing so. If using NOTIFY is the favoured approach, I would add a separate parameter for it and/or an explicit option on NOTIFY. -- Simon Riggs www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Range types
Robert Haas writes: > On Mon, Dec 14, 2009 at 4:06 AM, wrote: >>> As another approach, what about storing typeid in typmod? >>> (Oid can be assumed to be stored in int32.) > It 's very different than the way we've traditionally used typmod, Aside from the problems already pointed out, there's another: this definition usurps the ability to attach a typmod to the range's base type. Again, you should be looking at arrays not enums for a reference example. The typmod of an array feeds through to its element type. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Hot Standby, release candidate?
On Mon, 2009-12-14 at 13:56 +0100, Magnus Hagander wrote: > Same issue can be it in git - did you do a "git pull" before? You may > need merging with what's on there, and for that to work you must pull > before you can push. Found some merge conflicts and resolved them. I did fetch and merge at different times, so that seems to be the source. I've resolved the other git issues, so latest version on git now. -- Simon Riggs www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Hot Standby, release candidate?
Simon Riggs wrote: > If we are going to run pgindent anyway, what is the point? Perhaps it would take less time to run this than to argue the point?: sed -e 's/[ \t][ \t]*$//' -e 's/ *\t/\t/g' * -Kevin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Hot Standby, release candidate?
On Mon, 2009-12-14 at 09:32 -0500, Robert Haas wrote: > If every patch perfectly matched the pgident style, then the pgindent > run would change nothing and we would all be VERY happy. I've made all whitespace changes to the patch. I understand the reason for acting as you suggest, but we either police it, or we don't. If we don't police in all cases then I'm not personally happy to be policed. About 90% of the whitespace in the patch were in docs, README or test output files. A great many of that type of file have numerous line ending whitespace, not introduced by me, so it seems to me that this has never been adequately policed in the way you say. If we really do care about this issue then I expect people to look a little further than just my patch. Portability of patches across releases isn't a huge issue for me, nor for the project, I think, but I am willing to commit to cleaning future patches in this way. -- Simon Riggs www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] WAL Info messages
On Mon, 2009-12-14 at 09:51 -0500, Tom Lane wrote: > This might be > an example of a case where we need that "are we doing HS" flag that > Simon insists we don't need. Simon has put it there at your request, in the place you suggested. Others did speak against it, not me. -- Simon Riggs www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Row-Level Security
KaiGai Kohei writes: > One more issue I found. > What row-level policy should be applied on inherited tables? Yup, that seems like an interesting problem. > Even if the inherited table has multiple parents, all the row-level > policies shall be applied, so here is no inconsistency. > (Needless to say, child table have same columns, so we can apply > same row-level policies.) I don't think I believe either of those statements. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
