Re: [HACKERS] Checksums by default?
-BEGIN PGP SIGNED MESSAGE- Hash: RIPEMD160 Tom Lane points out: > Yeah, and there's a bunch of usability tooling that we don't have, > centered around "what do you do after you get a checksum error?". I've asked myself this as well, and came up with a proof of conecpt repair tool called pg_healer: http://blog.endpoint.com/2016/09/pghealer-repairing-postgres-problems.html It's very rough, but my vision is that someday Postgres will have a background process akin to autovacuum that constantly sniffs out corruption problems and (optionally) repairs them. The ability to self-repair is very limited unless checksums are enabled. I agree that there is work needed and problems to be solved with our checksum implementation (e.g. what if cosmic ray hits the checksum itself!?), but I would love to see what we do have enabled by default so we dramatically increase the pool of people with checksums enabled. - -- Greg Sabino Mullane g...@turnstep.com End Point Corporation http://www.endpoint.com/ PGP Key: 0x14964AC8 201701211522 http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8 -BEGIN PGP SIGNATURE- iEYEAREDAAYFAliDw5oACgkQvJuQZxSWSshy4QCfXokvagoishfTUnmujjpBNTUT q7IAn0dR74bFy0mj0EMoTU7Taj0db3Sh =qBEJ -END PGP SIGNATURE- -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Checksums by default?
-BEGIN PGP SIGNED MESSAGE- Hash: RIPEMD160 tl;dr +1 from me for changing the default, it is worth it. Tom Lane wrote: > Have we seen *even one* report of checksums catching > problems in a usefuld way? Sort of chicken-and-egg, as most places don't have it enabled. Which leads us to: Stephen Frost replies: > This isn't the right question. > > The right question is "have we seen reports of corruption which > checksums *would* have caught?" Well, I've seen corruption that almost certainly would have got caught much earlier than stumbling upon it later on when the corruption happened to finally trigger an error. I don't normally report such things to the list: it's almost always a hardware bug or bad RAM. I would only post if it were caused by a Postgres bug. Tom Lane wrote: > I think this will be making the average user pay X% for nothing. I think you mean "the average user who doesn't check what initdb options are available". And we can certainly post a big notice about this in the release notes, so people can use the initdb option - --disable-data-checksums if they want. > ... pay X% for nothing. It is not for nothing, it is for increasing reliability by detecting (and pinpointing!) corruption as early as possible. - -- Greg Sabino Mullane g...@turnstep.com End Point Corporation http://www.endpoint.com/ PGP Key: 0x14964AC8 201701211513 http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8 -BEGIN PGP SIGNATURE- iEYEAREDAAYFAliDwU4ACgkQvJuQZxSWSsi06QCgpPUg4SljERHMWP9tTJnoIRic U2cAoLZINh2rSECNYOwjldlD4dK00FiV =pYQ/ -END PGP SIGNATURE- -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] RustgreSQL
-BEGIN PGP SIGNED MESSAGE- Hash: RIPEMD160 Joel Jacobson asked: > Is anyone working on porting PostgreSQL to Rust? No; extremely unlikely. > My motivation is primarily I don't want to learn all the > over-complicated details of C Well that's going to be a show-stopper right there. For a proper port, a deep understanding of the current source code is necessary. You'd need a team expert in both C and Rust to pull it off. > Porting PostgreSQL to Rust would be a multi-year project, > and it could only be done if the process could be fully automated, > by supporting all the coding patterns used by the project, > otherwise a Rust-port would quickly fall behind the master branch. > But if all git commits could be automatically converted to Rust, Developing such a system is bordering on AI and likely more complex than Postgres itself. :) > Is this completely unrealistic or is it carved in stone PostgreSQL will > always be a C project forever and ever? It's unrealistic, but there is nothing to say Postgres will stay in C forever. Right now, however, there is no compelling reason to move away from it, and the porting effort to any language would be immense. C++ would be the least painful option, probably. - -- Greg Sabino Mullane g...@turnstep.com End Point Corporation http://www.endpoint.com/ PGP Key: 0x14964AC8 201701080905 http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8 -BEGIN PGP SIGNATURE- iEYEAREDAAYFAlhyR44ACgkQvJuQZxSWSsimzgCg97QZZ47BfNtema5aoN2QIpY9 wTUAn3B042YDH82GPLDwXmDSgJMzsoGD =PH10 -END PGP SIGNATURE- -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] No longer possible to query catalogs for index capabilities?
-BEGIN PGP SIGNED MESSAGE- Hash: RIPEMD160 Robert Haas wrote: > But I'm neither willing to commit a patch to fix the day before rc1 > nor to argue that the whole release cycle should be put back by > several weeks on account of this issue. Seriously? First, not sure why this would put the whole release cycle back by 'several weeks'. Second, this is removing functionality, so what are apps supposed to do - have a three-choice case in the code to handle pg_am for < 9.6, do some ugly parsing for 9.6, and use the new functions when 10.0 comes out?! This issue was raised on July 25th, and the OP has gone out of his way to present the case and provide patches. It's hardly fair to discard it now. - -- Greg Sabino Mullane g...@turnstep.com End Point Corporation http://www.endpoint.com/ PGP Key: 0x14964AC8 201608071606 http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8 -BEGIN PGP SIGNATURE- iEYEAREDAAYFAlenlWUACgkQvJuQZxSWSsjjeACfVrThYGx+4DnBwO2ZAOYGoK7s wdgAoOoxdVo0RM7smSr3CJg8J4dM3YMo =+m9i -END PGP SIGNATURE- -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 10.0
-BEGIN PGP SIGNED MESSAGE- Hash: RIPEMD160 > Wasn't there some controversy about switching to major.minor versioning > this in -advocacy? > > http://www.postgresql.org/message-id/ee13fd2bb44cb086b457be34e81d5...@biglumber.com I proposed in that thread that we always increment the first number, never increment the second number, and increment the third exactly as we do now for bugfix releases. I think moving to a two-number format is a mistake: what exactly will PQserverVersion() return in that case? But I understand people have a hard time swallowing the "never change the middle number" portion of this idea. Thus, here's a slight variation on that theme: what if we simply reversed the expectations of bumping the first number, and put the onus on people to change the *middle* number? Thus, the next release by default will be 10.0.0, the one after that will be by default 11.0.0, and so on. We can reserve the middle number for "lesser" releases - which may never happen - but at least we will have a mechanism to provide for them. So rather than the current spate of messages like this: "This should be called 12.0 because of cool feature X and reason Y" we would get the rare message like this: "We don't really have much for this release, maybe it should just be 11.1?" - -- Greg Sabino Mullane g...@turnstep.com End Point Corporation http://www.endpoint.com/ PGP Key: 0x14964AC8 201605142247 http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8 -BEGIN PGP SIGNATURE- iEYEAREDAAYFAlc34/UACgkQvJuQZxSWSsgQLgCeJS9v69R5C3BJxNy2ih1P2Tk8 xngAn0UQoSn6y3iOwMr5aHSKzuBh+3Xn =wzw4 -END PGP SIGNATURE- -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Template for commit messages
-BEGIN PGP SIGNED MESSAGE- Hash: RIPEMD160 FWIW, I read the git logs quite a bit, especially after a release to gather some stats, and I *love* the commits that have some nice standard, easy to read fields (Alvaro for one does a great job at this). I don't think we need to mandate it, or even ensure they are machine-parseable, but I would like to see a few fields encouraged. I think it also helps the committers to not forget some important things, the way the free-form text can. tl;dr be like Alvaro, please - -- Greg Sabino Mullane g...@turnstep.com End Point Corporation http://www.endpoint.com/ PGP Key: 0x14964AC8 201602011037 http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8 -BEGIN PGP SIGNATURE- iEYEAREDAAYFAlavfNwACgkQvJuQZxSWSsg4YgCgmWoL38qljypgUn082aWVp4y8 WDsAn24IwFKwqaudYRF1e3rvd0yz5btw =2Jc2 -END PGP SIGNATURE- -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] Release of CVEs
The release notes for the new version reference some CVEs that have not been publically released yet. Are they slow, or is this something that needs to be added to the release process checklist? For example, see the CVE hyperlink for json parsing at: https://bucardo.org/postgres_all_versions.html#version_9.4.5 which leads to: http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2015-5289 It's also possible the wrong CVE was entered, but I don't see one that seems to pertain to the issue described (and CVE-2015-5288, -3166, -3167, -0243, -0244 are in the same boat). -- Greg Sabino Mullane g...@endpoint.com End Point Corporation PGP Key: 0x14964AC8 signature.asc Description: Digital signature
Re: [HACKERS] No Issue Tracker - Say it Ain`t So!]
-BEGIN PGP SIGNED MESSAGE- Hash: RIPEMD160 > Comments are welcome, and no, I don't really expect that this will be what > gets > adopted, mainly I wanted to show that we can probably just build something > rather effective off our existing infrastructure +1, good job. > The bugs have 3.5 messages each on average, with 2 being the most common > number, and 113 at the most, for bug 12990. 1284 bugs have only one message > associated with them. For anyone who is dying to know, as I was, what the winning bug report was: "Missing pg_multixact/members files (appears to have wrapped, then truncated)" http://www.postgresql.org/message-id/flat/20150406192130.2573.22...@wrigleys.postgresql.org#20150406192130.2573.22...@wrigleys.postgresql.org or: http://goo.gl/4lKYOC - -- Greg Sabino Mullane g...@turnstep.com End Point Corporation http://www.endpoint.com/ PGP Key: 0x14964AC8 201510041854 http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8 -BEGIN PGP SIGNATURE- iEYEAREDAAYFAlYRriIACgkQvJuQZxSWSsgJkwCgsROux3esaDxHbitNhHs17Thk rKIAoNMD6NnKRAvguuvxkg4hiJOfPDH6 =5kJJ -END PGP SIGNATURE- -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Problems with question marks in operators (JDBC, ECPG, ...)
-BEGIN PGP SIGNED MESSAGE- Hash: RIPEMD160 Jan de Visser wrote: Well, one could argue that it *is* their problem, as they should be using the standard Postgres way for placeholders, which is $1, $2, $3... Shirley you are joking: Many products use JDBC as an abstraction layer facilitating (mostly) seamless switching between databases. I know the product I worked on did. Are you advocating that every single statement should use SELECT * FROM foo WHERE bar = $1 on pg and SELECT * FROM foo WHERE bar = ? on every other database? I'm not joking, and don't call me Shirley. If you are running into situations where you have question mark operators in your queries, you have already lost the query abstraction battle. There will be no seamless switching if you are using jsonb, hstore, ltree, etc. My statement was more about pointing out that Postgres already offers a complete placeholder system, which drivers are free to implement if they want. A database is only as valuable as the the part of the outside world it can interact with. Large parts of the data-consuming world are developed in java using JDBC. If your opinion is that JDBC developers should adapt themselves to pg then you instantaneously diminish the value of pg. Well, they will have to adapt to one way or another: using ?? or \? is doing so, and the other solution (Postgres adapting itself to the driver by deprecating the ? operator) is not realistically likely to happen. - -- Greg Sabino Mullane g...@turnstep.com End Point Corporation http://www.endpoint.com/ PGP Key: 0x14964AC8 201505191718 http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8 -BEGIN PGP SIGNATURE- iEYEAREDAAYFAlVbq4AACgkQvJuQZxSWSsgrXgCaA6MTvbDeg2aMf+/HFnxutrqH P1sAoLZB1w5+UXHMxXqW/Ex0q7GwoFds =IOpS -END PGP SIGNATURE- -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Problems with question marks in operators (JDBC, ECPG, ...)
-BEGIN PGP SIGNED MESSAGE-
Hash: RIPEMD160
I did find some alternatives discussed a couple of years back, like
{postgres qm} and operator(?); the later simply being to allow the
operator to be quoted inside operator()
Yes, we (DBD::Pg) looked at using at some of the JDBC-ish alternatives
like the (very verbose) vendor escape clauses, but settled on the simplicity of
a single backslash in the end. See part of the discussion here:
http://www.nntp.perl.org/group/perl.dbi.users/2014/12/msg37057.html
- --
Greg Sabino Mullane g...@turnstep.com
End Point Corporation http://www.endpoint.com/
PGP Key: 0x14964AC8 201505191520
http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8
-BEGIN PGP SIGNATURE-
iEYEAREDAAYFAlVbjQQACgkQvJuQZxSWSsgYhACfUfztfxZBQEwESqRYkfRco29M
pAUAoO9qA5IWN96UXsh9iASspiEYfAfF
=k8Gl
-END PGP SIGNATURE-
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] collations in shared catalogs?
-BEGIN PGP SIGNED MESSAGE- Hash: RIPEMD160 It's maybe not absolutely strictly necessary. In fact in earlier versions of the patch it was name. But replication solutions like bdr, slony, whatever will have to store a bunch of values identifying a node in there. And that's much easier if you're not constrained by 63 chars. That's silly. We (third-party tools) already have to work around lots of things constrained by namedatalen. - -- Greg Sabino Mullane g...@turnstep.com End Point Corporation http://www.endpoint.com/ PGP Key: 0x14964AC8 201505182138 http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8 -BEGIN PGP SIGNATURE- iEYEAREDAAYFAlValBYACgkQvJuQZxSWSsiODwCfRDTNsEHKsp7rbK24lT4lApwa X1sAn0QL33wJyn/AWT2aLL9u+Ybt+aNb =VjvO -END PGP SIGNATURE- -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Problems with question marks in operators (JDBC, ECPG, ...)
-BEGIN PGP SIGNED MESSAGE-
Hash: RIPEMD160
Andrew Dunstan wrote:
FTR, Perl's DBD::Pg lets you do this:
$dbh-{pg_placeholder_dollaronly} = 1; # disable ? placeholders
You can also simply escape placeholders in DBD::Pg with a backslash:
$dbh-prepare(q{SELECT * FROM mytable WHERE lseg1 \?# lseg2 AND name = ?});
Dave Cramer wrote:
Well our solution was to use ?? but that does mean we have to do some
extra parsing which in a perfect world wouldn't be necessary.
That's not a good solution as '??' is a perfectly valid operator. ISTR
seeing it used somewhere in the wild, but I could be wrong.
In that case my vote is new operators. This has been a sore point for the
JDBC driver
Um, no, new operators is a bad idea. Question marks are used by hstore,
json, geometry, and who knows what else. I think the onus is solely on
JDBC to solve this problem. DBD::Pg solved it in 2008 with
the pg_placeholder_dollaronly solution, and earlier this year by allowing
backslashes before the question mark (because other parts of the stack were
not able to smoothly implement pg_placeholder_dollaronly.) I recommend
all drivers implement \? as a semi-standard workaround.
See also:
http://blog.endpoint.com/2015/01/dbdpg-escaping-placeholders-with.html
- --
Greg Sabino Mullane g...@turnstep.com
End Point Corporation http://www.endpoint.com/
PGP Key: 0x14964AC8 201505171212
http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8
-BEGIN PGP SIGNATURE-
iEYEAREDAAYFAlVYvmQACgkQvJuQZxSWSsj8SwCdEL3f0JvSlVQERpn+KJIaILzj
GqAAni9qcZ8PLixSLmGoXEQr8tnVZ2RI
=YJfa
-END PGP SIGNATURE-
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] Better error message on pg_upgrade checksum mismatches
Just a little thing that's been bugging me. If one side of the
pg_upgrade has checksums and the other does not, give a less
cryptic error message.
--
Greg Sabino Mullane g...@endpoint.com
End Point Corporation
PGP Key: 0x14964AC8
diff --git a/contrib/pg_upgrade/controldata.c b/contrib/pg_upgrade/controldata.c
index a02a8ec..8a7b976 100644
--- a/contrib/pg_upgrade/controldata.c
+++ b/contrib/pg_upgrade/controldata.c
@@ -572,9 +572,17 @@ check_control_data(ControlData *oldctrl,
* We might eventually allow upgrades from checksum to no-checksum
* clusters.
*/
+ if (! oldctrl-data_checksum_version newctrl-data_checksum_version)
+ {
+ pg_fatal(old version does not use data checksums but new one does\n);
+ }
+ if (oldctrl-data_checksum_version ! newctrl-data_checksum_version)
+ {
+ pg_fatal(old version uses data checksums but new one does not\n);
+ }
if (oldctrl-data_checksum_version != newctrl-data_checksum_version)
{
- pg_fatal(old and new pg_controldata checksum versions are invalid or do not match\n);
+ pg_fatal(old and new pg_controldata checksum versions do not match\n);
}
}
signature.asc
Description: Digital signature
Re: [HACKERS] Release note bloat is getting out of hand
-BEGIN PGP SIGNED MESSAGE- Hash: RIPEMD160 Robert Haas wrote: but there are times when it's easier to find out what release introduced a feature by looking at the release notes, and it's certainly more useful if you want to send a link to someone who is not git-aware illustrating the results of your search. Well, maybe I'm the only one who is doing this and it's not worth worrying about it just for me. But I do it, all the same. I do this *all the time*. Please don't mess with the release notes. Except to put them all on one page for easy searching. That would be awesome. - -- Greg Sabino Mullane g...@turnstep.com End Point Corporation http://www.endpoint.com/ PGP Key: 0x14964AC8 201502021555 http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8 -BEGIN PGP SIGNATURE- iEYEAREDAAYFAlTP5EQACgkQvJuQZxSWSsj13QCfTrKBKDlOm0E5K4+2ib7F8Tjl w5QAoOY3vX9tUb1KUxk3VaW+k71vrW7m =y+SU -END PGP SIGNATURE- -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Detecting backend failures via libpq / DBD::Pg
-BEGIN PGP SIGNED MESSAGE- Hash: RIPEMD160 Andrew Gierth asked: this is to send a simple SELECT via PQexec Why not PQexec(conn, ) ? Because I want to leave a good clue for debugging; so DBAs are better able to figure out where a mystery slew of queries is coming from. The query is: SELECT 'DBD::Pg ping test' Which also means the inverse is true: simple blank queries are guaranteed to *not* be coming from DBD::Pg. - -- Greg Sabino Mullane g...@turnstep.com End Point Corporation http://www.endpoint.com/ PGP Key: 0x14964AC8 201412301041 http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8 -BEGIN PGP SIGNATURE- iEYEAREDAAYFAlSix/YACgkQvJuQZxSWSsjILwCdHnkhYC1i+LJZkNUWjfTi5yG+ FHwAn007+arJIw62gIUO20+SxnzRT4ub =9Rym -END PGP SIGNATURE- -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] Detecting backend failures via libpq / DBD::Pg
-BEGIN PGP SIGNED MESSAGE- Hash: RIPEMD160 I am working on enhancing the ping() method of DBD::Pg. The goal of that is for a user to be able to determine if the connection to the database is still valid. The basic way we do this is to send a simple SELECT via PQexec and then check for a valid return value (and when in doubt, we check PQstatus). This works fine for most transaction statuses, including idle, active, and idle in transaction. It even works for copy in and copy out, although it obviously invalidates the current COPY (caveat emptor!). The problem comes when ping() is called and we are in a failed transaction. After some experimenting, the best solution I found is to send the PQexec, and then check if PQresultErrorField(result, 'C') is '25P02'. If it is, then all is well, in that the server is still alive. If it is not, then we can assume the backend is bad (for example, PQerrorMessage gives a could not receive data from server: Bad file descriptor). Being that we cannot do a rollback before calling the PQexec, is this a decent solution? Can we depend on really serious errors always trumping the expected 25P02? - -- Greg Sabino Mullane g...@turnstep.com End Point Corporation http://www.endpoint.com/ PGP Key: 0x14964AC8 201412291942 http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8 -BEGIN PGP SIGNATURE- iEYEAREDAAYFAlSh9QEACgkQvJuQZxSWSsjDMQCg3CO1eyrFXNUnfRbk/rRJmrCl PEoAnRl+M67kTkuZDi+3zMyVyblLvl9I =uW6Q -END PGP SIGNATURE- -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Commitfest problems
-BEGIN PGP SIGNED MESSAGE- Hash: RIPEMD160 #2 is solved by my previous comments about giving the CFM/C the authority. -Core could do that, they are in charge of release. I don't think authority is the solution. Or certainly not one that would work with an open source project like ours. What *would* work is to identify and fix the friction points that prevent people from joining, make the work harder than it needs to be, and makes people stop reviewing? I could quickly identify a handful of things, primarily among them the awful link-to-the-archives to gather up all the patches process. We have git, let's use it as it was intended. - -- Greg Sabino Mullane g...@turnstep.com End Point Corporation http://www.endpoint.com/ PGP Key: 0x14964AC8 201412141011 http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8 -BEGIN PGP SIGNATURE- iEYEAREDAAYFAlSNqK8ACgkQvJuQZxSWSsiewwCffAxv8xSZEyLWFz/b2+PxXOXS xB4An2ubr7ovELtFMKZOZCsFHQAyVca4 =S6ZQ -END PGP SIGNATURE- -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Repeatable read and serializable transactions see data committed after tx start
-BEGIN PGP SIGNED MESSAGE- Hash: RIPEMD160 Kevin Grittner wrote: I think most people have always assumed that BEGIN starts the transaction and that is the point at which the snapshot is obtained. But there is so much evidence to the contrary. Not only does the *name* of the command (BEGIN or START) imply a start, but pg_stat_activity shows the connection idle in transaction after the command (and before a snapshot is acquired) Er...I think we are arguing the same thing here. So no contrary needed? :) Why? This fix might not deal with the bigger issues that I discussed, like that the later-to-start and later-to-acquire-a-snapshot transaction might logically be first in the apparent order of execution. You can't fix that without a lot of blocking -- that most of us don't want. Right, which is why the suggestion of a user-controllable switch, that defaults to the current behavior, seems an excellent compromise. Depending on *why* they think this is important, they might need to be acquiring various locks to prevent behavior they don't want, in which case having acquired a snapshot at BEGIN would be exactly the *wrong* thing to do. The exact nature of the problem we're trying to solve here does matter. I cannot speak to the OP, but I also do not think we should try and figure out every possible scenario people may have. Certainly the long-standing documentation bug may have caused some unexpected or unwanted behavior, so let's start by fixing that. Tom Lane wrote: Another thing that I think hasn't been mentioned in this thread is that we used to have severe problems with client libraries that like to issue BEGIN and then go idle until they have something to do. Which, for some reason, is a prevalent behavior. I'm not advocating changing the default behavior, but I would not want to see bad client libraries used a reason for any change we make. Clients should not be doing this, period, and there is no reason for us to support that. - -- Greg Sabino Mullane g...@turnstep.com End Point Corporation http://www.endpoint.com/ PGP Key: 0x14964AC8 201411071600 http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8 -BEGIN PGP SIGNATURE- iEYEAREDAAYFAlRdMwwACgkQvJuQZxSWSsh/mgCeMdrj15bNVtzBhecG+QT2SlKh jboAnAjctUcrlA2aCCQmIsSM87ulmFEn =U5ld -END PGP SIGNATURE- -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Repeatable read and serializable transactions see data committed after tx start
-BEGIN PGP SIGNED MESSAGE- Hash: RIPEMD160 Kevin Grittner wrote: (wording change suggestion) | sees a snapshot as of the start of the first query within the | transaction, not as of the start of the current query within the | transaction. Would that have prevented the confusion here? I think it may have, but I also think the wording should be much stronger and clearer, as this is unintuitive behavior. Consider this snippet from Bruce's excellent MVCC Unmasked presentation: A snapshot is recorded at the start of each SQL statement in READ COMMITTED transaction isolation mode, and at transaction start in SERIALIZABLE transaction isolation mode. This is both correct and incorrect, depending on whether you consider a transaction to start with BEGIN; or with the first statement after the BEGIN. :) I think most people have always assumed that BEGIN starts the transaction and that is the point at which the snapshot is obtained. But what about creating a flag to BEGIN and SET TRANSACTION commands, called IMMEDIATE FREEZE (or something similar), which applies only to REPEATABLE READ and SERIALIZABLE? If this flag is set (and may be off by default, but of course the default may be configurable via a guc parameter), freeze happens when it is present (BEGIN or SET TRANSACTION) time. This would be a backwards-compatible change, while would provide the option of freezing without the nasty hack of having to do a SELECT 1 prior to your real queries, and everything will of course be well documented. What is the use case where you are having a problem? This seems like an odd solution, so it would be helpful to know what problem it is attempting to solve. Seems like a decent solution to me. The problem it that having to execute a dummy SQL statement to start a serializable transaction, rather than simply a BEGIN, is ugly.and error prone. Perhaps their app assumes (or even requires) that BEGIN starts the snapshot. - -- Greg Sabino Mullane g...@turnstep.com End Point Corporation http://www.endpoint.com/ PGP Key: 0x14964AC8 201411060922 http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8 -BEGIN PGP SIGNATURE- iEYEAREDAAYFAlRbhD4ACgkQvJuQZxSWSsg/kwCdE9E+d3jDDpLOo4+08wCOMMxE EHkAnj4uMO8cY6Jl0R19C/6lE6n3bae5 =syg9 -END PGP SIGNATURE- -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Feasibility of supporting bind params for all command types
-BEGIN PGP SIGNED MESSAGE- Hash: RIPEMD160 Tom Lane said: ... Craig Ringer cr...@2ndquadrant.com writes: While looking at an unrelated issue in PgJDBC I noticed that it's difficult for users and the driver to tell in advance if a given statement will support bind parameters. It's not that hard ;-) ... if it ain't SELECT/INSERT/UPDATE/DELETE, it won't accept parameters. Yes, it is as easy as that. That's exactly what DBD::Pg does - looks at the first word of the statement. Although you also need to add VALUES and WITH to that list. :) As a result, some code that worked with PgJDBC using the v2 protocol will fail with the v3 protocol, e.g. It'd be nice not to force users to do their own escaping of literals in non-plannable statements. Before embarking on anything like this I thought I'd check and see if anyone's looked into supporting bind parameters in utility statements, or if not, if anyone has any ideas about the feasibility of adding such support. I don't think that's a hill you want to conquer. Let that code relying on v2 behavior get rewritten, or make the driver smart enough to handle it automagically the best it can. - -- Greg Sabino Mullane g...@turnstep.com End Point Corporation http://www.endpoint.com/ PGP Key: 0x14964AC8 201410060710 http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8 -BEGIN PGP SIGNATURE- iEYEAREDAAYFAlQyeNIACgkQvJuQZxSWSshYewCgg/EmgTbPp5KnfUpYfga8nsee GVMAniXC+FxHFsiuT07idP8Tw70gCoBe =a20X -END PGP SIGNATURE- -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Similar to csvlog but not really, json logs?
-BEGIN PGP SIGNED MESSAGE- Hash: RIPEMD160 Stephen Frost wrote: To try to clarify that a bit, as it comes across as rather opaque even on my re-reading, consider a case where you can't have the credit_card_number field ever exported to an audit or log file, but you're required to log all other changes to a table. Then consider that such a situation extends to individual INSERT or UPDATE commands- you need the command logged, but you can't have the contents of that column in the log file. Perhaps you need a better example. Storing raw credit cards in the database is a bad idea (and potential PCI violation); audit/log files are only one of the many ways things can leak out. Encrypting sensitive columns is a solution that solves your auditing problem, and works on all current versions of Postgres. :) Our current capabilities around logging and auditing are dismal No arguments there. - -- Greg Sabino Mullane g...@turnstep.com End Point Corporation http://www.endpoint.com/ PGP Key: 0x14964AC8 201408271200 http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8 -BEGIN PGP SIGNATURE- iEYEAREDAAYFAlP+AKgACgkQvJuQZxSWSsjf7gCg00BwRbwRi/UPrHBs1RdfWX/I TRsAn2CDrG/ycetKOQFbn/4rnSSYPz9j =Ju0B -END PGP SIGNATURE- -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] In-core regression tests for replication, cascading, archiving, PITR, etc. Michael Paquier
-BEGIN PGP SIGNED MESSAGE- Hash: RIPEMD160 I thought the goal here was to have a testing framework that (a) is portable to every platform we support and (b) doesn't require root privileges to run. None of those options sound like they'll help meet those requirements. FWIW, I hacked up a Perl-based testing system as a proof of concept some time ago. I can dust it off if anyone is interested. Perl has a very nice testing ecosystem and is probably the most portable language we support, other than C. My quick goals for the project were: * allow granular testing (ala Andrew's recent email, which reminded me of this) * allow stackable methods and dependencies * make it very easy to write new tests * test various features that are way too diificult in our existing system (e.g. PITR, fdws) * get some automated code coverage metrics (this one was tricky) * allow future git integration based on subsytems - -- Greg Sabino Mullane g...@turnstep.com End Point Corporation http://www.endpoint.com/ PGP Key: 0x14964AC8 201401261211 http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8 -BEGIN PGP SIGNATURE- iEYEAREDAAYFAlLlQeMACgkQvJuQZxSWSsiYhACggHJgQWB/Q2HEfjGZCwR3yEZg zMsAnAssOStAmMuaJEScCGHGKWYNow1v =zi0Y -END PGP SIGNATURE- -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [9.4 CF 1] The Commitfest Slacker List
-BEGIN PGP SIGNED MESSAGE- Hash: RIPEMD160 Josh Berkus replied: I won't go into details here because frankly why I have no time for reviewing a patch is none of your business. Then just send an email saying Sorry, I don't have any time for patch review this time. Maybe next time. It's pretty simple. Hope about you not publically shame people in a volunteer project? That's pretty simple. I'm not going to apologize for expecting *committers* to participate in patch review and commit. I must have missed the page where patch review is defined as part of a committer's job. Possibly slacker was a poor choice of word given translations; in colloquial American English it's a casual term, even affectionate under some conditions. I'll make sure to use different words if I ever end up doing a list again. Please, don't ever do a list again. And yes, slacker was an extremely poor choice of word. This American English speaker certainly has a hard time viewing it as affectionate. I think the whole thread would have been better received with a subject line of Commitfest needs help. - -- Greg Sabino Mullane g...@turnstep.com PGP Key: 0x14964AC8 201307032150 http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8 -BEGIN PGP SIGNATURE- iEYEAREDAAYFAlHU1QQACgkQvJuQZxSWSsgoMgCfcUm/MnYzsUaqVWq3DvTh2kAi sYwAoLAijh3SkCbv2c7visToyqPAOWMG =xoKV -END PGP SIGNATURE- -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Kudos for Reviewers -- straw poll
-BEGIN PGP SIGNED MESSAGE- Hash: RIPEMD160 Josh Berkus wrote: I wasn't thinking about doing it every year -- just for 9.3, in order to encourage more reviewers, and encourage reviewers to do more reviews. - -1. It's not cool to set it up and then stop it the next go round. You want more reviewers? Start by streamlining the process as much as possible. I pretended I was new to the project and tried to figure out how to review something. The homepage has no mention of reviewers, not even if you drill down on some subpages. A Google search does lead one to: http://wiki.postgresql.org/wiki/Reviewing_a_Patch It has some good you can do it wordage. However, there is no clear path on how to actually start reviewing. There is this paragraph with two links in it: The current commitfest is here[1] and has plenty of room for you to help. You can sign up to become a Round Robin Reviewer here[2]. Once you have, write a mail to the list introducing yourself. [1] Leads to the commitfest, with a nice summary, but no way for new people to know what to do. [2] This link is even worse (http://www.postgresql.org/list/pgsql-rrreviewers/) It's an archive list for pgsql-rrreviewers, with no way to subscribe and certainly no indication on it or the previous page that sign up means (one might guess) join the mailing list. Anyway, just food for thought as far as attracting new people. It should be much easier and more intuitive. As far as rewarding current reviewers, put the names in the release notes, after each item. Full stop. - -- Greg Sabino Mullane g...@turnstep.com End Point Corporation http://www.endpoint.com/ PGP Key: 0x14964AC8 201306271636 http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8 -BEGIN PGP SIGNATURE- iEYEAREDAAYFAlHMoqIACgkQvJuQZxSWSsgCPACgovKYtxJV59Xro0MlxPDEHIy6 pmAAoOLOAlpO/dPlJbyHypdcY4ZxLCit =RwMh -END PGP SIGNATURE- -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Considering Gerrit for CFs
-BEGIN PGP SIGNED MESSAGE- Hash: RIPEMD160 The problem with doing it in-house is that the folks who can work on it and maintain it will be taking time away from developing PostgreSQL. Not sure that using Gerrit solves this. Someone will need to install it, maintain it, document, and hack it. Yes, hack it, as it is not a drop-in solution. ... I think one of them has, now: Gerrit. http://code.google.com/p/gerrit/ I use Gerrit in the MediaWiki project, and it ain't pretty. The interface is confusing, the workflow is more complex, and the MediaWiki folks have had to do a lot of work to make things usable, despite their having a non-email-centric workflow already. Maybe we can identify specific issues with our current app instead? - -- Greg Sabino Mullane g...@turnstep.com PGP Key: 0x14964AC8 201302081106 http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8 -BEGIN PGP SIGNATURE- iEYEAREDAAYFAlEVIqcACgkQvJuQZxSWSsh04gCfaK80dbuL8NnAVuViGR5sFQXN GzwAoM+2fcI6+zFZPqkslZrWjkZ05AOo =azLj -END PGP SIGNATURE- -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Considering Gerrit for CFs
-BEGIN PGP SIGNED MESSAGE- Hash: RIPEMD160 How would this go with PostgreSQL? You can use the bug form on the web site, but you can't attach any code, so the bug will just linger and ultimately put more burden on a core contributor to deal with the minutiae of developing, testing, and committing a trivial fix and sending feedback to the submitter. Well, they could attach a link to a github patch... Or the user could take the high road and develop and patch and submit it. Just make sure it's in context diff format! Search the wiki if you don't know how to do that! Send it to -hackers, your email will be held for moderation. We won't actually do anything with your patch, but we will tell you to add it to that commitfest app over there. You need to sign up for an account to use that. We will deal with your patch in one or two months. But only if you review another patch. And you should sign up for that other mailing list, to make sure you're doing it right. Chances are, the first review you're going to get is that your patch doesn't apply anymore, but which time you will have lost interest in the patch anyway. +1 to all that. Especially the signing up for the commitfest app. Of course, Gerrit doesn't actually address most of the issues above, but it could be part of a step forward. More of a step sideways. It doesn't address the bigger problems. - -- Greg Sabino Mullane g...@turnstep.com PGP Key: 0x14964AC8 201302081124 http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8 -BEGIN PGP SIGNATURE- iEYEAREDAAYFAlEVJrkACgkQvJuQZxSWSshh3gCgz+XHwAbk5rryttYPi68j4EJi 7DcAnjEdxDD4Rm2/oDBaqHbOzQLwR6zR =0lnp -END PGP SIGNATURE- 2~ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Tablespaces in the data directory
-BEGIN PGP SIGNED MESSAGE- Hash: RIPEMD160 As there isn't (as far as I know at least) any actual *point* in creating a tablespace inside the main data directory, should we perhaps disallow this in CREATE TABLESPACE? Or at least throw a WARNING if one does it? Sure there is a point - emulating some other system. Could be replication, QA box, disaster recovery, etc. I'd be cool with a warning, but do not think we should disallow it. - -- Greg Sabino Mullane g...@turnstep.com End Point Corporation http://www.endpoint.com/ PGP Key: 0x14964AC8 201212022133 http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8 -BEGIN PGP SIGNATURE- iEYEAREDAAYFAlC8D7kACgkQvJuQZxSWSsj+5gCgsmi6NXue+Hp0gycVOL/JEGUT anYAoIqwo24JeLfliRHLvwPbdK4F4TXa =EwgC -END PGP SIGNATURE- -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] My first patch! (to \df output)
-BEGIN PGP SIGNED MESSAGE- Hash: RIPEMD160 This was actually kind of anti-climactic, since it only took about 5 minutes to make the change and get it working. Didn't really feel the way I expected it to ;) Well, we can reject your patch and start bike-shedding it for the next four months, if that makes you feel better! :) Congrats! - -- Greg Sabino Mullane g...@turnstep.com PGP Key: 0x14964AC8 201210271914 http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8 -BEGIN PGP SIGNATURE- iEYEAREDAAYFAlCMau4ACgkQvJuQZxSWSshdoQCg6eJ14LLcJrn04rN2/efO14iz swgAoPbBSv8PAre6qtVrRH3LL/iNQqeD =m/ns -END PGP SIGNATURE- -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Deprecating RULES
-BEGIN PGP SIGNED MESSAGE- Hash: RIPEMD160 Tom and Simon wrote: If you want to get rid of rules, build the replacement; don't just try to be a pain in the ass to users. Supporting broken and non-standard features *is* a pain in the ass to users, since they are sometimes persuaded to use them and then regret it. Or if they do, hit later problems. Broken? That's a strong word. Tricky perhaps. Best avoided by novices, yes. But broken? Do they not work exactly as described in the fine manual? FWIW, I still see them a lot in the wild. Anyway, lets start with a discussion of what rules give us that SQL standard features do not? Even if the answer is nothing, if we do not implement the SQL standard feature yet (exhibit A: updateable views), it's a moot point unless the goal is to spur development of those features just so we can deprecate rules. - -- Greg Sabino Mullane g...@turnstep.com End Point Corporation http://www.endpoint.com/ PGP Key: 0x14964AC8 201210112251 http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8 -BEGIN PGP SIGNATURE- iEYEAREDAAYFAlB3hx8ACgkQvJuQZxSWSshhwQCfdtKc7R2i0kz7eDUTXtik93k3 KyEAoK0dQVZsfcAD3OlHYDVhWMjst8QZ =xY2L -END PGP SIGNATURE- -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] PQping command line tool
-BEGIN PGP SIGNED MESSAGE- Hash: RIPEMD160 Jim Nasby pointed out: It'd be useful to us to have a utility that could cleanly validate the server was up and communicating, without having to actually login. Well sure, but wouldn't it be even more useful to validate at the same time that logins are working? :) - -- Greg Sabino Mullane g...@turnstep.com End Point Corporation http://www.endpoint.com/ PGP Key: 0x14964AC8 201210101310 http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8 -BEGIN PGP SIGNATURE- iEYEAREDAAYFAlB1rDsACgkQvJuQZxSWSsibngCg61f1ldN2wZOT4FHOFvbiIfO6 3EIAoJyFfd4T2NXv/jH5zGD9pSypykXi =WP+s -END PGP SIGNATURE- -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] PQping command line tool
-BEGIN PGP SIGNED MESSAGE- Hash: RIPEMD160 I was wondering recently if there was any command line tool that utilized PQping() or PQpingParams(). I searched the code and couldn't find anything and was wondering if there was any interest to have something like this included? I wrote something for my purposes of performing a health check that also supports nagios style status output. It's probably convenient for scripting purposes as well. I'm not sure how useful this information would be. Most health checks (Nagios or otherwise) really only care if things are working all the up to point A or not, where point A is usually a simple query such as SELECT 1. Knowing various failure states as returned by PQping* does not seem to fit into such tools - any failure needs to be handled manually. - -- Greg Sabino Mullane g...@turnstep.com End Point Corporation http://www.endpoint.com/ PGP Key: 0x14964AC8 201210041146 http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8 -BEGIN PGP SIGNATURE- iEYEAREDAAYFAlBtukQACgkQvJuQZxSWSsiCbACePHFhTefoQnLwVuvIONH0JcSD jq8AoIPusD88fX1rBcse5IreaADH7wkZ =IRgc -END PGP SIGNATURE- -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_reorg in core?
-BEGIN PGP SIGNED MESSAGE- Hash: RIPEMD160 I think it's time to consider some *umbrella project* for maintaining several small projects outside the core. Well, that was pgfoundry, and it didn't work out. I'm not sure that is quite analogous to what was being proposed. I read it as more of let's package a bunch of these small utilities together into a single project, such that installing one installs them all (e.g. aptitude install pg_tools), and they all have a single bug tracker, etc. That tracker could be github, of course. I'm not convinced of the merit of that plan, but that's an alternative interpretation that doesn't involve our beloved pgfoundry. :) Oh, and -1 for putting it in core. Way too early, and not important enough. - -- Greg Sabino Mullane g...@turnstep.com PGP Key: 0x14964AC8 201209222334 http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8 -BEGIN PGP SIGNATURE- iEYEAREDAAYFAlBeg/AACgkQvJuQZxSWSsjL5ACgimT71B4lSb1ELhgMw5EBzAKs xHIAn08vxGzmM6eSmDfZfxlJDTousq7h =KgXW -END PGP SIGNATURE- -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] MySQL search query is not executing in Postgres DB
-BEGIN PGP SIGNED MESSAGE- Hash: RIPEMD160 Tom Lane replied: Come on, really? Note that the above example works without casts if you use int *or* bigint *or* numeric, but not smallint. That could be fixed by causing sufficiently-small integers to lex as smallints, Is there any general interest in adjusting smallint casting? ... It's conceivable that a change in the lexer behavior combined with a massive reorganization of the integer-related operators would bring us to a nicer place than where we are now. But it'd be a lot of work for dubious reward, and it would almost certainly generate a pile of application compatibility problems. Okay, but what about a more targeted solution to the original poster's problem? That seems doable without causing major breakage elsewhere - -- Greg Sabino Mullane g...@turnstep.com End Point Corporation http://www.endpoint.com/ PGP Key: 0x14964AC8 201208271818 http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8 -BEGIN PGP SIGNATURE- iEYEAREDAAYFAlA78m0ACgkQvJuQZxSWSshW2gCg1Xcx5zLORMIDQo2yE6QTLVuD P88AniE9rh4Dojg0o416cWK7cYHWaq0b =NOAR -END PGP SIGNATURE- -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] multi-master pgbench?
-BEGIN PGP SIGNED MESSAGE- Hash: RIPEMD160 The point of this functionality is to test some cluster software which have a capability to create multi-master configuration. As the maintainer of software that does multi-master, I'm a little confused as to why we would extend pg_bench to do this. The software in question should be doing the testing itself, ideally via it's test suite (i.e. make test). Having pg_bench do any of this would be at best a very poor subset of the tests the software should be performing. I suppose if the software *uses* pg_bench for its tests already, once could argue a limited test case - but it seems difficult to design some pg_bench options generic and powerful enough to handle other cases outside of the one software this change is aimed at. - -- Greg Sabino Mullane g...@turnstep.com End Point Corporation http://www.endpoint.com/ PGP Key: 0x14964AC8 201208212330 http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8 -BEGIN PGP SIGNATURE- iEYEAREDAAYFAlA0UvsACgkQvJuQZxSWSsjALgCgw2cGI3eWR5fBGkoX9hqV1N39 OSEAn2ZIxrNRCdkDfKVrMmx2PsQTs8ZS =Xhqb -END PGP SIGNATURE- -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] sha1, sha2 functions into core?
-BEGIN PGP SIGNED MESSAGE- Hash: RIPEMD160 If the hacker has access to the salt, then it will only slow him/her down somewhat because the search will be have to be restarted for each password. This. Further, anyone using MD5 or SHA* or any hash function for any serious storage of passwords is nuts, in this day and age. GPUs and rentable cloud computers means the ability to test billions of passwords per second is easy for anyone, salted or not. The issue is not Postgres' internal use of MD5 for passwords - that's a red herring, as it is basically no more relatively secure/insecure versus any other hashing algorithm that is not designed to be slow (e.g. bcrypt, scrypt, PBKDF2). The issue is simply exposing a more useful day to day algorithm by default. Much of the world uses SHA instead of MD5 these days for all sorts of purposes. So I am torn on this. On the one hand, having a few more things in core would be very nice, as it seems silly we have md5() as a builtin but sha256() requires a special module. But once you add sha* in, why not AES? Blowfish? Why not go the whole way and include some extremely useful ones such as bcrypt? At that point, we've deprecated pg_crypto and moved everything to core. Why I personally would love to see that someday (then we can boast built-in crypto :), I recognize that will be a very tough sell. So I will take the addition of whatever we can, including just a sha() as this thread asked for. 3) use a purposefully slow hashing function like bcrypt. but I disagree: I don't like any scheme that encourages use of low entropy passwords. Perhaps off-topic, but how to do you figure that? - -- Greg Sabino Mullane g...@turnstep.com End Point Corporation http://www.endpoint.com/ PGP Key: 0x14964AC8 201208201849 http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8 -BEGIN PGP SIGNATURE- iEYEAREDAAYFAlAywBwACgkQvJuQZxSWSsiS4QCbBC7X9MyQgVKC3DTKgjv0aj7D ik0AoNh1YBmhuaMXEKOP7z/GEBUR+EHe =54A2 -END PGP SIGNATURE- -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Btree or not btree? That is the question
On Mon, Jul 09, 2012 at 04:02:13PM -0400, Tom Lane wrote: ... Could you crank up the log verbosity so we can get file and line number, at least? Here is what the increased verbosity reveals in aggregate. This is about an 18-hour span, covering 12.5M transactions, on version 8.3.18: (13 times) Various OIDs that do exist cannot be found: ERROR: XX000: could not open relation with OID 1554847444 LOCATION: relation_open, heapam.c:879 (21 times) Relations that do exist cannot be found: ERROR: 42P01: relation foobar does not exist LOCATION: RangeVarGetRelid, namespace.c:273 (1 time) Qualified relation that exists cannot be found: ERROR: 42P01: relation public.foobar does not exist LOCATION: RangeVarGetRelid, namespace.c:268 (5 times) Failure to read a block: XX001: could not read block 3 of relation 1663/1554846571/4184054438: read only 0 of 8192 bytes LOCATION: mdread, md.c:631 (5 times) Cache lookup failure: XX000: cache lookup failed for relation 1554847255 LOCATION: has_subclass, plancat.c:921 -- Greg Sabino Mullane g...@endpoint.com End Point Corporation PGP Key: 0x14964AC8 pgpBH51gd5sgI.pgp Description: PGP signature
Re: [HACKERS] Btree or not btree? That is the question
I've expanded my searching a bit, to see if I can find any other correlations. One thing that seems to happen about 10 times a day is an error of this sort: ERROR: could not open relation with OID 1554847326 In this case, the OID in question always exists, and corresponds to one of a handful of particularly busy tables. Sometimes the query does not even touch the OID mentioned directly: in the above example, the SQL was an update to table A that had a FK to table B, and the OID above is for table B. The queries themselves vary: I've not found any common factor yet. These errors have been happening a long time, and obviously don't cause the same database-hosed-must-restart issue the btree does, but it is still a little disconcerting. Although 10 times out of 20 million transactions per day is at least an extremely rare event :) It is definitely NOT correlated to system table reindexing, but does seem to be roughly correlated to how busy things are in general. We've not been able to duplicate on a non-prod test system yet either, which points to either hardware or (more likely) a failure to completely simulate the high activity level of prod. No idea if this related to the relatively recent btree errors, but figured I would get it out there. There is also an even rarer sprinkling of: ERROR: relation with OID 3924107573 does not exist but I figured that was probably a variant of the first error. -- Greg Sabino Mullane g...@endpoint.com End Point Corporation PGP Key: 0x14964AC8 pgpmh4I30bBvW.pgp Description: PGP signature
Re: [HACKERS] Btree or not btree? That is the question
ERROR: could not open relation with OID 1554847326 Is that the *entire* message? No details? Could you crank up the log verbosity so we can get file and line number, at least? Yes, that's the entire thing, other than the statement line after it. In this particular case: STATEMENT: UPDATE mytable SET foobar = 'T' WHERE id IN ($1,$2,$3,$4) I'll start the wheels rolling for upping the verbosity. -- Greg Sabino Mullane g...@endpoint.com End Point Corporation PGP Key: 0x14964AC8 pgp1V3duXPAQS.pgp Description: PGP signature
Re: [HACKERS] Btree or not btree? That is the question
Could you crank up the log verbosity so we can get file and line number, at least? First hit since the change: ERROR: XX000: could not open relation with OID 1554847444 LOCATION: relation_open, heapam.c:879 STATEMENT: SELECT ... Will leave the verbosity up and see if it occurs in the same place. -- Greg Sabino Mullane g...@endpoint.com End Point Corporation PGP Key: 0x14964AC8 pgpSmpP1G2bZT.pgp Description: PGP signature
Re: [HACKERS] Btree or not btree? That is the question
ERROR: index pg_class_oid_index is not a btree That means you got bogus data while reading the metapage. I'm beginning to wonder about the hardware on this server ... This happened again, and this time I went back through the logs and found that it is always the exact same query causing the issue. I also found it occuring on different servers, which rules out RAM anyway (still shared disk, so those are suspect). This query also sometimes gives errors like this: ERROR: could not read block 3 of relation 1663/1554846571/3925298284: read only 0 of 8192 bytes However, the final number changes: these are invariably temporary relations. The query itself is a GROUP BY over a large view and the explain plan is 107 rows, with nothing esoteric about it. Most of the tables used are fairly common ones. I'm trying to duplicate on a non-production box, without success so far, and I'm loath to run it on production as it sometimes causes multiple backends to freeze up and requires a forceful restart. Any ideas on how to carefully debug this? There are a couple of quicksorts when I explain analyze on a non-prod system, which I am guessing where the temp tables come from (work_mem is 24MB). I'm not sure I understand what could be causing both the 'read 0' and btree errors for the same query - bad blocks on disk for one of the underlying tables? I'll work next on checking each of the tables the view is using. -- Greg Sabino Mullane g...@endpoint.com End Point Corporation PGP Key: 0x14964AC8 pgpCBcRgxlYYF.pgp Description: PGP signature
Re: [HACKERS] Btree or not btree? That is the question
I dug through the logs and found some other occurances of the could not read block errors. Some on dirt simple SELECT queries. Nothing else has generated the btree error yet. About 35 found in the last month. This theory would be more plausible if you're wrong about the second-case tables being temp, though, because if they are temp then their indexes would be kept in local buffers not shared buffers, making it much harder to believe in a single bug causing both symptoms. I grepped the last month of logs and found about 20 instances of that error: none of the relfilenodes given shows up in pg_class, even for that dirt simple SELECT. One possible mechanism for confusion of that sort would be if the spinlock code wasn't quite right, or the compiler was incorrectly moving loads/stores into or out of locked sections. So it might be time to ask exactly what kind of hardware this is, which compiler PG was built with, etc. Quad core AMD Opteron. RHEL. Compiled with gcc with all the options (basically the standard compilation e.g. --build=x86_64-redhat-linux-gnu) I can give you more details offlist if it will help. On the other hand, if the issue were of that sort then it ought to affect all buffers more or less at random; so if you're consistently seeing exactly these symptoms (in particular, if it's always pg_class_oid_index that's complained of), then I'm not sure I believe this theory either. I've never seen any other index for the btree error, but it has only happened a grand total of 3 times ever. The other error appears to be fairly random, except that the one particular query that gives the btree error always seems to give one version or the other. Which PG version again? Are you in the habit of doing VACUUM FULLs on system catalogs, and if so do these glitches correlate at all with such activities? Heh. 8.3.18. Yes, very heavy vac fulls (and reindexes) of the system catalogs. Cron-driven, and depends on the time of day and if any DDL is running (if so, it does not run), but probably on average pg_class is vacfulled and reindexed twice an hour during the times this happens (which is, during normal business hours). There is a lot in churn in pg_class, pg_attribute, and pg_depend in particular from all the temp stuff being created and torn down all day, as well as some Bucardo pg_class updating. -- Greg Sabino Mullane g...@endpoint.com End Point Corporation PGP Key: 0x14964AC8 pgpLQFTbOf8Tw.pgp Description: PGP signature
Re: [HACKERS] Btree or not btree? That is the question
On Mon, Jun 04, 2012 at 02:09:44PM -0400, Tom Lane wrote: Greg Sabino Mullane g...@endpoint.com writes: We have a 8.3.18 system (yes, the same one from the previous thread, finally upgraded!) that gave us this error yesterday: ERROR: index pg_class_oid_index is not a btree That means you got bogus data while reading the metapage. I'm beginning to wonder about the hardware on this server ... Thanks for the reply. Me too. This is on a cluster[1], so we do have the luxury of testing the hardware on each box and shuffling things around. I'm also thinking we need to get away from the near-constant updating of pg_class, just as a good measure. [1] Bruce, yet another usage! :) -- Greg Sabino Mullane g...@endpoint.com End Point Corporation PGP Key: 0x14964AC8 pgpbXwlXRWU0v.pgp Description: PGP signature
[HACKERS] Btree or not btree? That is the question
We have a 8.3.18 system (yes, the same one from the previous thread, finally upgraded!) that gave us this error yesterday: ERROR: index pg_class_oid_index is not a btree It seems like this happened from just a normal SELECT query, and did not occur again (and this is a very busy system) The pg_class table does gets vacuumed and reindexed often. All appeared fine when I did some manual investigation. Since this only happened this one time, can I chalk it up to some random rare race condition having to do with a reindex? Or is there anything else I should be looking for or can proactively do? No other system catalog problems have been seen before it or since. -- Greg Sabino Mullane g...@endpoint.com End Point Corporation PGP Key: 0x14964AC8 pgp83Kexp6hqX.pgp Description: PGP signature
Re: [HACKERS] Backends stalled in 'startup' state: index corruption
On Sun, May 27, 2012 at 05:44:15PM -0700, Jeff Frost wrote: On May 27, 2012, at 12:53 PM, Tom Lane wrote: occurring, they'd take long enough to expose the process to sinval overrun even with not-very-high DDL rates. As it turns out, there are quite a few temporary tables created. For the record, same here. We do *lots* of DDL (hence the cronjobs to vac/reindex system catalogs). -- Greg Sabino Mullane g...@endpoint.com End Point Corporation PGP Key: 0x14964AC8 pgpuQRprn1huB.pgp Description: PGP signature
Re: [HACKERS] Backends stalled in 'startup' state: index corruption
On Fri, May 25, 2012 at 07:02:42PM -0400, Tom Lane wrote: However, the remaining processes trying to compute new init files would still have to complete the process, so I'd expect there to be a diminishing effect --- the ones that were stalling shouldn't all release exactly together. Unless there is some additional effect that's syncing them all. (I wonder for instance if the syncscan logic is kicking in here.) How fast would you expect that to happen? As far as I could tell, they all released at once, or at least within probably 15 seconds of each other; I wasn't running ps constantly. I could check the logs and get a better figure if you think it's an important data point. One interesting question is why there's a thundering herd of new arrivals in the first place. IIRC you said you were using a connection pooler. I wonder if it has a bug^H^H^Hdesign infelicity that makes it drop and reopen all its connections simultaneously. No, we are not. Or rather, there is some pooling, but there is also a fairly large influx of new connections. As far as I could tell, the few existing connections were not affected. 1. Somebody decides to update one of those rows, and it gets dropped in some remote region of the table. The only really plausible reason for this is deciding to fool with the column-specific stats target (attstattarget) of a system catalog. Does that sound like something either of you might have done? No, zero chance of this, barring some rogue intruder on the network with a strange sense of humor. pg_attribute just enough smaller to avoid the scenario. Not sure about Greg's case, but he should be able to tell us the size of pg_attribute and his shared_buffers setting ... pg_attribute around 5 MB (+6MB indexes), shared_buffers 4GB. However, there is a *lot* of churn in pg_attribute and pg_class, mostly due to lots of temporary tables. P.S. Hmmm that's weird, I just double-checked the above and pg_attribute is now 52MB/70MB (the original figures were from yesterday). At any rate, nowhere near 1/4 shared buffers. -- Greg Sabino Mullane g...@endpoint.com End Point Corporation PGP Key: 0x14964AC8 pgpGtYKGLr70y.pgp Description: PGP signature
Re: [HACKERS] Backends stalled in 'startup' state: index corruption
On Sat, May 26, 2012 at 12:17:04PM -0400, Tom Lane wrote: If you see any block numbers above about 20 then maybe the triggering condition is a row relocation after all. Highest was 13. -- Greg Sabino Mullane g...@endpoint.com End Point Corporation PGP Key: 0x14964AC8 pgpa6XGTGTEIZ.pgp Description: PGP signature
Re: [HACKERS] Backends stalled in 'startup' state: index corruption
On Sat, May 26, 2012 at 01:25:29PM -0400, Tom Lane wrote: Greg Sabino Mullane g...@endpoint.com writes: On Sat, May 26, 2012 at 12:17:04PM -0400, Tom Lane wrote: If you see any block numbers above about 20 then maybe the triggering condition is a row relocation after all. Highest was 13. Hm ... but wait, you said you'd done a VACUUM FULL on the catalogs. So it's not clear whether this is reflective of the state at the time the problem was happening. True. I'll try to get a high water mark when (er...if!) it happens again. -- Greg Sabino Mullane g...@endpoint.com End Point Corporation PGP Key: 0x14964AC8 pgpUK3N5QYoTd.pgp Description: PGP signature
Re: [HACKERS] Backends stalled in 'startup' state: index corruption
Yeah, this is proof that what it was doing is the same as what we saw in Jeff's backtrace, ie loading up the system catalog relcache entries the hard way via seqscans on the core catalogs. So the question to be answered is why that's suddenly a big performance bottleneck. It's not a cheap operation of course (that's why we cache the results ;-)) but it shouldn't take minutes either. And, because they are seqscans, it doesn't seem like messed-up indexes should matter. FWIW, this appeared to be an all-or-nothing event: either every new backend was suffering through this, or none were. They all seemed to clear up at the same time as well. The theory I have in mind about Jeff's case is that it was basically an I/O storm, but it's not clear whether the same explanation works for your case. There may be some other contributing factor that we haven't identified yet. Let me know if you think of anything particular I can test while it is happening again. I'll try to arrange a (netapp) snapshot the next time it happens as well (this system is too busy and too large to do anything else). -- Greg Sabino Mullane g...@endpoint.com End Point Corporation PGP Key: 0x14964AC8 pgpYJDovfAM7L.pgp Description: PGP signature
[HACKERS] Backends stalled in 'startup' state: index corruption
Yesterday I had a client that experienced a sudden high load on one of their servers (8.3.5 - yes, I know. Those of you with clients will understand). When I checked, almost all connections were in a startup state, very similar to this thread: http://postgresql.1045698.n5.nabble.com/9-1-3-backends-getting-stuck-in-startup-td5670712.html Running a strace showed a lot of semop activity, and the logs showed a successful connection, then a 5 minute plus wait before a query was issued. So obviously, blocking on something. Unlike the thread above, I *did* find problems in the system catalogs. For example, both pg_class and pg_index gave warnings like this for every index during a VACUUM FULL VERBOSE tablename: WARNING: index pg_class_relname_nsp_index contains 7712 row versions, but table contains 9471 row versions HINT: Rebuild the index with REINDEX. A REINDEX did not solve the problem (initially), as a REINDEX followed by a VAC showed the same warning and hint. The next step was dropping to standalone mode, but before that could be done, the REINDEXes fixed the problem (no warnings, no stalled connections). So my questions are: * Why would a REINDEX not fix the problem as the hint suggested? Why would it then start working? * What exactly is the relationship between bad indexes and shared memory locks? * Is there some other emergency fix when it happens, such as killing all backends and hoping you kill the one that is actually holding the lock (if any was). * Did anything in the 8.3 series fix this? I saw nothing relevant in the release notes for everything up to 8.3.18 (which it will be on soon). -- Greg Sabino Mullane g...@endpoint.com End Point Corporation PGP Key: 0x14964AC8 pgpwbxRekkZQf.pgp Description: PGP signature
Re: [HACKERS] Backends stalled in 'startup' state: index corruption
On Thu, May 24, 2012 at 03:54:54PM -0400, Tom Lane wrote:
Did you check I/O activity? I looked again at Jeff Frost's report and
now think that what he saw was probably a lot of seqscans on bloated
system catalogs, cf
http://archives.postgresql.org/message-id/28484.1337887...@sss.pgh.pa.us
Thank you for the response. Yes, we did look at I/O, but nothing unusual
was seen that would explain the load. If it happens again I'll see if the
system catalogs are getting a lot of seqscans.
That's fairly interesting, but if it was a bloat situation then it
would've been the VAC FULL that fixed it rather than the REINDEX.
Did you happen to save the VERBOSE output? It'd be really useful to
know whether there was any major shrinkage of the core catalogs
(esp. pg_class, pg_attribute).
I did have them in screen, but the home screen box just craashed a
few hours ago (after weeks of uptime: Murphy's law). It certainly could
have been VAC FULL as we have processes that do both VAC FULL and REINDEX
periodically; I simply assumed based on the HINT it was the REINDEX that
cleared it up. The only snippet I have is:
# vacuum full verbose pg_class;
INFO: vacuuming pg_catalog.pg_class
INFO: pg_class: found 43 removable, 10376 nonremovable row versions in 518
pages
DETAIL: 6078 dead row versions cannot be removed yet.
Nonremovable row versions range from 160 to 628 bytes long.
There were 7367 unused item pointers.
Total free space (including removable row versions) is 157000 bytes.
0 pages are or will become empty, including 0 at the end of the table.
0 pages containing 0 free bytes are potential move destinations.
CPU 0.00s/0.00u sec elapsed 0.00 sec.
INFO: index pg_class_oid_index now contains 7712 row versions in 32 pages
DETAIL: 6 index row versions were removed.
0 index pages have been deleted, 0 are currently reusable.
CPU 0.00s/0.00u sec elapsed 0.00 sec.
WARNING: index pg_class_oid_index contains 7712 row versions, but table
contains 9471 row versions
HINT: Rebuild the index with REINDEX.
INFO: index pg_class_relname_nsp_index now contains 7712 row versions in 113
pages
DETAIL: 6 index row versions were removed.
0 index pages have been deleted, 0 are currently reusable.
CPU 0.00s/0.00u sec elapsed 0.00 sec.
WARNING: index pg_class_relname_nsp_index contains 7712 row versions, but
table contains 9471 row versions
HINT: Rebuild the index with REINDEX.
VACUUM
Time: 65.635 ms
Actually, since we have full logging, those warnings should still be
there... (checks) ... yep. Hints elided, the last one seen was:
WARNING: index pg_class_oid_index contains 14877 row versions,
but table contains 66648 row versions
WARNING: index pg_class_relname_nsp_index contains 14877 row versions,
but table contains 66648 row versions
WARNING: index pg_class_oid_index contains 60943 row versions,
but table contains 112714 row versions
WARNING: index pg_class_relname_nsp_index contains 60943 row versions,
but table contains 112714 row versions
Looks like there are some more going back a ways: I'll gather together
and send those offlist after this.
I think there are probably two independent issues here. The missing
index entries are clearly bad but it's not clear that they had anything
to do with the startup stall. There are a couple of fixes in recent
8.3.x releases that might possibly explain the index corruption,
especially if you're in the habit of reindexing the system catalogs
frequently.
Yes, we are in that habit. I hope they are not independent: I'd rather
have one problem to worry about than two. :) I'll certainly report if I
see either problem pop up again.
Oh, almost forgot: reading your reply to the old thread reminded me of
something I saw in one of the straces right as it woke up and left
the startup state to do some work. Here's a summary:
12:18:39 semop(4390981, 0x7fff66c4ec10, 1) = 0
12:18:39 semop(4390981, 0x7fff66c4ec10, 1) = 0
12:18:39 semop(4390981, 0x7fff66c4ec10, 1) = 0
(x a gazillion)
...
12:18:40 brk(0x1c0af000)= 0x1c0af000
...(some more semops)...
12:18:40 mmap(NULL, 266240, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS,
-1, 0) = 0x2ac062c98000
...(handful of semops)...
12:18:40 unlink(base/1554846571/pg_internal.init.11803) = -1 ENOENT (No such
file or directory)
12:18:40 open(base/1554846571/pg_internal.init.11803,
O_WRONLY|O_CREAT|O_TRUNC, 0666) = 13
12:18:40 fstat(13, {st_mode=S_IFREG|0600, st_size=0, ...}) = 0
12:18:40 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
0) = 0x2ac062cd9000
12:18:40 write(13, ...
...(normalish looking strace output after this)...
--
Greg Sabino Mullane g...@endpoint.com
End Point Corporation
PGP Key: 0x14964AC8
pgpgvtyJ9p6Fs.pgp
Description: PGP signature
Re: [HACKERS] Backends stalled in 'startup' state: index corruption
I think there are probably two independent issues here. The missing index entries are clearly bad but it's not clear that they had anything to do with the startup stall. On further log digging, I think you are correct, as those index warnings go back many days before the startup problems appeared. Let me know if you'd like any of those warnings from the logs. -- Greg Sabino Mullane g...@endpoint.com End Point Corporation PGP Key: 0x14964AC8 pgp9P2W8oU7Y8.pgp Description: PGP signature
Re: [HACKERS] Draft release notes complete
-BEGIN PGP SIGNED MESSAGE- Hash: RIPEMD160 Bruce wrote: In summary, names on release note items potentially have the following beneficial effects: * Encouraging new developers/reviewers * Encouraging long-established developers * Showing appreciation to developers * Assisting future employment for developers * Helping developers get future funding * Assigning responsibility for features * Showing Postgres's increased developer base The only important ones are: * Assisting future employment for developers * Helping developers get future funding * Assigning responsibility for features * Assigning blame for feature problems That last one is not very important either. If there is a bug, you report it. The original author may or may not handle it. A better way to state some of the above is: * Quick cross-reference of a person to a feature. If I claim to have written ON_ERROR_ROLLBACK, nobody should have to scroll back through git logs to confirm or deny. (For that matter, we should do everything possible to prevent anyone from using git log, especially non-developers, for any meta-information.) +1 to keep things they way they are. If you were significantly invested in [re]writing the patch, you get a name. Reviewers, I love you dearly, but you don't belong next to the patch. Group them all at the bottom if we must have them there. - -- Greg Sabino Mullane g...@turnstep.com End Point Corporation http://www.endpoint.com/ PGP Key: 0x14964AC8 201205151259 http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8 -BEGIN PGP SIGNATURE- iEYEAREDAAYFAk+yi3cACgkQvJuQZxSWSsiAcACfYC1HCxbMor/c0EJF6kn+XKc9 kOcAoMn0vnOJLa8+HVz5oWKAZxjkOtQi =eiUT -END PGP SIGNATURE- -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Draft release notes complete
-BEGIN PGP SIGNED MESSAGE- Hash: RIPEMD160 I'd vote for starting a separate thread to solicit people's opinions on whether we need names in the release notes. Is there anybody on -hackers who would be offended, or would have a harder time persuading $BOSS to let them spend time on Postgres if they weren't mentioned in the release notes? There'd still be a policy of crediting people in commit messages of course, but it's not clear to me whether the release note mentions are important to anybody. Looks like this is mostly answered, and we obviously don't need another thread, but the answer to the above is yes. Release notes are very public, plain text, easy to read, very archived and searchable. Commit messages might as well be a black hole as far as visibility to anyone not a developer in the project. - -- Greg Sabino Mullane g...@turnstep.com End Point Corporation http://www.endpoint.com/ PGP Key: 0x14964AC8 201205151301 http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8 -BEGIN PGP SIGNATURE- iEYEAREDAAYFAk+yjFEACgkQvJuQZxSWSsi3gACgmikPzvshZPftTuEdmcB8/Ply 4vMAn1DxvG6hntfxJzWRDdPyWlP5X7WM =pUbl -END PGP SIGNATURE- -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Bug tracker tool we need
-BEGIN PGP SIGNED MESSAGE- Hash: RIPEMD160 So I think Greg has exactly the right idea: we shouldn't try to incorporate one of these systems that aims to manage workflow; we should just design something really simple that tracks what happened and lets people who wish to volunteer to do so help keep that tracking information up to date. Note: the above is the other Greg :) If we are serious about building this ourselves, and we feel it is important, maybe we should sponsor it via our group funds or some other means? Seems like everyone here has lots of ideas but little free time. - -- Greg Sabino Mullane g...@turnstep.com End Point Corporation http://www.endpoint.com/ PGP Key: 0x14964AC8 201204191031 http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8 -BEGIN PGP SIGNATURE- iEYEAREDAAYFAk+QIeUACgkQvJuQZxSWSsi5NACg4ruX3jvuQ5zKnxbBPu2Kc9wW C+EAoPsIt2n0bbYau/aPhPbVdm+JPHj3 =j1XN -END PGP SIGNATURE- -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Bug tracker tool we need
-BEGIN PGP SIGNED MESSAGE- Hash: RIPEMD160 My github.com account currently has 4264 notifications in the inbox. Almost all of those are spam, growing constantly. �Because of that, the platform is currently fairly useless to me for actually communicating or collaborating on code. That's about the same amount that I have. I have no spam at all, despite being a fairly early github adopter. Wonder what the difference is? - -- Greg Sabino Mullane g...@turnstep.com PGP Key: 0x14964AC8 201204191044 http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8 -BEGIN PGP SIGNATURE- iEYEAREDAAYFAk+QJO4ACgkQvJuQZxSWSsg7OgCggq2MVw10W2+XxCyoDSdbjTYP JOAAoLVJeX/V5j1h8r0dpvyJAw9/O+BU =puT/ -END PGP SIGNATURE- -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Last gasp
-BEGIN PGP SIGNED MESSAGE- Hash: RIPEMD160 If the feature set is desirable, though, I wonder if Postgres is big/high profile enough for them to figure out some sort of better arrangement. They *love* it when big open-source projects use GitHub as their public repo - they'll email and blog announcements about it - and if there's interest I'd be happy to open a conversation with them. No need to wonder, we've been in contact with them before and they are very pro Postgres. I've looked at it in conjunction with Jenkins CI; it looked nice but was way too heavy-weight for a four-person startup (what's code review?). It's probably much more suitable for this sized project. Gerrit's a full-featured code review app with a tolerable UI; MediaWiki is just starting to jump into git/Gerrit and there are definitely a lot of rough edges in that workflow still. ... Someone mentioned upthread that github spam was a problem. I'm not sure I see the issue here - wouldn't mail from them still go through our lists and out current anti-spam measures anyway? - -- Greg Sabino Mullane g...@turnstep.com End Point Corporation http://www.endpoint.com/ PGP Key: 0x14964AC8 201204170623 http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8 -BEGIN PGP SIGNATURE- iEYEAREDAAYFAk+NRO4ACgkQvJuQZxSWSshPpACg9+ZB6NzCsvnkQwuoD/BzIHgL yMkAn3zwksbKxaSDt3k/YzKY7UVLmUZb =igZu -END PGP SIGNATURE- -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Bug tracker tool we need
-BEGIN PGP SIGNED MESSAGE- Hash: RIPEMD160 But for any given ABC there are also people who will tell you that it's got significant problems. We don't need to change anything to get a system that's got significant problems; we already have one. Let's not let perfect be the enemy of good. In this case, *anything* that actually tracks bugs (and they are all quite good at that, if nothing else) is an improvement over what we have now, and thus, quite good. :) Personally, I'm okay with, and have extensively hacked on, Bugzilla and RT, but anything should be fine as long as we have someone to take ownership. - -- Greg Sabino Mullane g...@turnstep.com PGP Key: 0x14964AC8 201204172131 http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8 -BEGIN PGP SIGNATURE- iEYEAREDAAYFAk+OL/0ACgkQvJuQZxSWSshMxACeJdr+WO4ttA2mkrGLv98PTTSH jSoAniKwQNPzokA3f0GYN8gB+hAOc0Hy =oPn6 -END PGP SIGNATURE- -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Last gasp
-BEGIN PGP SIGNED MESSAGE- Hash: RIPEMD160 I want to caution against adjusting things to improve funding possibilities. There is nothing wrong with increasing funding possibilities, per say, but such changes often distort behavior in unforeseen ways that adversely affect our community process. I don't see this as much of a problem. If somewhat arbitrary labels and powers allow the project to succeed, we should think long and hard before rejecting the idea. It's not like we are going to make anyone who asks a committer, like MediaWiki does. Indeed, we have been super cautious about handing out both commit bits, and labels (e.g. Major Developer). One wrinkle is the subsystems: there are some people who only work on certain parts, yet have a commit bit (with the understanding that they won't start editing core or other parts). From an outside perspective however, a Postgres committer [of certain subsystems] is a Postgres committer. One thing I think would help potential and current developers, and act as a further code review and safety valve, is to have a mailing list that actually shows the committed diffs. Links to a webpage showing the diff is just not the same. pgsql-commit-di...@postgresql.org, anyone? - -- Greg Sabino Mullane g...@turnstep.com End Point Corporation http://www.endpoint.com/ PGP Key: 0x14964AC8 201204121121 http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8 -BEGIN PGP SIGNATURE- iEYEAREDAAYFAk+G8/sACgkQvJuQZxSWSsh7HACgn7Wf/AQyUJwtvxgjYSHSIHkJ hq4AnjMgPlDakupg4mo204+N1p4C0mMZ =z+cR -END PGP SIGNATURE- -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Revisiting extract(epoch from timestamp)
-BEGIN PGP SIGNED MESSAGE- Hash: RIPEMD160 so that we could mark it immutable. On the other hand, it's not entirely apparent why people would need to create indexes on the epoch value rather than just indexing the timestamp itself Well, it makes for smaller indexes if you don't really care about sub-second resolutions. - -- Greg Sabino Mullane g...@turnstep.com End Point Corporation http://www.endpoint.com/ PGP Key: 0x14964AC8 201204091345 http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8 -BEGIN PGP SIGNATURE- iEYEAREDAAYFAk+DIJcACgkQvJuQZxSWSsiLsQCgrA8Sxcljm+HPJ1jQY7l0u3UZ UTwAnjBGM7SstLCnihtRkxDJrMax2Ikl =Kjic -END PGP SIGNATURE- -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] libxml related crash on git head
-BEGIN PGP SIGNED MESSAGE-
Hash: RIPEMD160
I'm getting HEAD errors on my build farm animal fennec.
I've narrowed it down to this test case:
greg=# CREATE TEMP TABLE boom AS SELECT 'ABC'::bytea;
greg=# SELECT table_to_xml('boom',false,false,'');
server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
The connection to the server was lost. Attempting reset: Failed.
Specifically, it's only the bytea data type, with that function. The
boolean arguments do not make a difference. I cannot reproduce this on
other boxes, so I suspect it may be a libxml bug. This server is running
libxml2 version 2.7.2: we require = 2.6.23. I've tried manually installing
a newer version of libxml, but make fails with:
ld: crtbeginS.o: No such file: No such file or directory
libtool: install: error: relink `libxml2mod.la' with the
above command before installing it
I don't have time/inclination to track down why the make is failing, but
may have some time to run any Postgres-specific tests, if anyone wishes
me to.
- --
Greg Sabino Mullane g...@turnstep.com
PGP Key: 0x14964AC8 201204021528
http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8
-BEGIN PGP SIGNATURE-
iEYEAREDAAYFAk95/iMACgkQvJuQZxSWSshVdACfQN+2EAgPP8LRq1wSAa33OWMm
oz8AoIaU91+JcFMdpb0ecWPPJyLRyRgq
=ZNWe
-END PGP SIGNATURE-
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] libxml related crash on git head
-BEGIN PGP SIGNED MESSAGE- Hash: RIPEMD160 I'm getting HEAD errors on my build farm animal fennec. Oh, I looked at that the other day. The animal started failing after you installed a new libxml in /usr/local. Ah, okay, that makes sense. So MediaWiki wanted a new version but libxml was being *really* problematic so I abandoned the install, but I guess it left some pieces around. I will see if I can clean it up. BTW, right at the moment you seem to have git issues, too. Thanks, I will check on that. - -- Greg Sabino Mullane g...@turnstep.com PGP Key: 0x14964AC8 201204021608 http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8 -BEGIN PGP SIGNATURE- iEYEAREDAAYFAk96B2IACgkQvJuQZxSWSsg9RQCg0DaEGVldBl6eI7ajZ+4DMcWx wtAAoO9h/9+isE/X/Y+T7xwcgCb1ZCxL =JAZN -END PGP SIGNATURE- -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] MySQL search query is not executing in Postgres DB
-BEGIN PGP SIGNED MESSAGE- Hash: RIPEMD160 The time I got bitten by this was actually with LPAD(), rather than LIKE. +1. This is one of the functions that gave some of our clients real trouble when 8.3 came out. If we really believed that implicit casts any form were evil, we would have removed them entirely instead of trimming them back. I don't see why it's heretical to suggest that the 8.3 casting changes brought us to exactly that point in the universe where everything is perfect and nothing can be further improved; does anyone seriously believe that? Agreed (although the last bit is a bit of a straw man). The idea in this thread of putting some implicit casts into an extension or other external package is not a very good one, either. Let's apply some common sense instead, and stick to our guns on the ones where we feel there could honestly be serious app consequences and thus we encourage^H^Hforce people to change their code (or write all sorts of custom casts and functions). I think the actual number of such app circumstances is rather small, but my clients are not your* clients, so who knows? In other words, I'll concede int==text, but really need a strong argument for conceding things like LPAD. * Your = everyone else, not just M. Haas. - -- Greg Sabino Mullane g...@turnstep.com End Point Corporation http://www.endpoint.com/ PGP Key: 0x14964AC8 201202181145 http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8 -BEGIN PGP SIGNATURE- iEYEAREDAAYFAk8/1usACgkQvJuQZxSWSsjE6ACdHy31jpHUsXo5juvXcCkzKpGH RQAAoM/uTbM/JBkDiDjrsI1Blyg3DsWf =7CA4 -END PGP SIGNATURE- -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] psql tab completion for SELECT
-BEGIN PGP SIGNED MESSAGE- Hash: RIPEMD160 Robert Haas wrote: One thing that's been bugging me for a while is that the tab completion code all works by looking backward up to n words. What we really want to know is what kind of statement we're in and where we are in it. Absent other information, if we're in the target list of a SELECT statement (nested arbitrarily) that behavior is reasonable. If we're someplace in a GRANT statement, or someplace in a CREATE STATEMENT where, say, a column name is expected, it's really not. I played with this years ago, but readline does not really offer a good way to easily get what we want (the whole statement, chunked into nice bits to analyze). Of course at this point we should think about making things more generic so we can drop in whatever readline-alternative comes along in the future. Unfortunately, making the tab completion something other than incredibly stupid is likely to be an insane amount of work. Insane amount of work? Check. Inredibly stupid? No, I think we've done pretty good given the limitations we have. You want to see incredibly stupid, see some of the *other* CLIs out there (hi, mysql! :) - -- Greg Sabino Mullane g...@turnstep.com End Point Corporation http://www.endpoint.com/ PGP Key: 0x14964AC8 201202101157 http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8 -BEGIN PGP SIGNATURE- iEYEAREDAAYFAk81TI4ACgkQvJuQZxSWSsivRQCfcze1WMq81rE+mtrOReHBQ6eV SzEAn2JySDAoCokFkY/gtz//GqolVVm5 =d2LG -END PGP SIGNATURE- -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Overhead cost of Serializable Snapshot Isolation
On Wed, Oct 12, 2011 at 10:50:13AM +0300, Peter Eisentraut wrote: Actually, I'm currently personally more concerned about the breakage we introduce in minor releases. We'd need to solve that problem before we can even begin to think about dealing with the major release issue. +1 This bit me the other day. -- Greg Sabino Mullane g...@endpoint.com End Point Corporation PGP Key: 0x14964AC8 pgprM9aFgot2o.pgp Description: PGP signature
Re: [HACKERS] Overhead cost of Serializable Snapshot Isolation
Robert Haas: Serializable mode is much slower on this test, though. On REL9_1_STABLE, it's about 8% slower with a single client. At 8 clients, the difference rises to 43%, and at 32 clients, it's 51% slower. Bummer. Thanks for putting some numbers out there; glad I was able to jump start a deeper look at this. Based on this thread so far, I am probably going to avoid serializable in this particular case, and stick to repeatable read. Once things are in place, perhaps I'll be able to try switching to serializable and get some measurements, but I wanted to see if the impact was minor enough to safely start with serializable. Seems not. :) Keep in mind this is not even a formal proposal yet for our client, so any benchmarks from me may be quite a while. Kevin Grittner: Did these transactions write anything? If not, were they declared to be READ ONLY? If they were, in fact, only reading, it would be interesting to see what the performance looks like if the recommendation to use the READ ONLY attribute is followed. Yes, I'll definitely look into that, but the great majority of the things done in this case are read/write. Simon Riggs: Most apps use mixed mode serializable/repeatable read and therefore can't be changed by simple parameter. Rewriting the application isn't a sensible solution. I think it's clear that SSI should have had and still needs an off switch for cases that cause performance problems. Eh? It has an off switch: repeatable read. Thanks for all replying to this thread, it's been very helpful. -- Greg Sabino Mullane g...@endpoint.com End Point Corporation PGP Key: 0x14964AC8 pgpkFVkl3Xl3T.pgp Description: PGP signature
Re: [HACKERS] Overhead cost of Serializable Snapshot Isolation
If the normal default_transaction_isolation = read committed and all transactions that require serializable are explicitly marked in the application then there is no way to turn off SSI without altering the application. That is not acceptable, since it causes changes in application behaviour and possibly also performance issues. Performance, perhaps. What application behavior changes? Less serialization conflicts? We should provide a mechanism to allow people to upgrade to 9.1+ without needing to change the meaning and/or performance of their apps. That ship has sailed. -- Greg Sabino Mullane g...@endpoint.com End Point Corporation PGP Key: 0x14964AC8 pgpsMUli41Pnm.pgp Description: PGP signature
Re: [HACKERS] Overhead cost of Serializable Snapshot Isolation
On Tue, Oct 11, 2011 at 04:32:45PM -0400, Bruce Momjian wrote: ... Simon seems to value backward-compatibility more than the average hackers poster. The lack of complaints about 9.1 I think means that the hackers decision of _not_ providing a swich was the right one. I wouldn't go that far: 9.1 is very new. Certainly the release notes do not explain the change enough: part of the reason I wrote: http://blog.endpoint.com/2011/09/postgresql-allows-for-different.html Simon has a point, but I think that having applications switch from serializable to repeatable read is a pain point people should pay when going to 9.1, rather than adding some switch now. -- Greg Sabino Mullane g...@endpoint.com End Point Corporation PGP Key: 0x14964AC8 pgp7jN6DSMohw.pgp Description: PGP signature
[HACKERS] Overhead cost of Serializable Snapshot Isolation
I'm looking into upgrading a fairly busy system to 9.1. They use serializable mode for a few certain things, and suffer through some serialization errors as a result. While looking over the new serializable/SSI documentation, one thing that stood out is: http://www.postgresql.org/docs/current/interactive/transaction-iso.html The monitoring of read/write dependencies has a cost, as does the restart of transactions which are terminated with a serialization failure, but balanced against the cost and blocking involved in use of explicit locks and SELECT FOR UPDATE or SELECT FOR SHARE, Serializable transactions are the best performance choice for some environments. I agree it is better versus SELECT FOR, but what about repeatable read versus the new serializable? How much overhead is there in the 'monitoring of read/write dependencies'? This is my only concern at the moment. Are we talking insignificant overhead? Minor? Is it measurable? Hard to say without knowing the number of txns, number of locks, etc.? -- Greg Sabino Mullane g...@endpoint.com End Point Corporation PGP Key: 0x14964AC8 pgpXfFQOk4fgH.pgp Description: PGP signature
Re: [HACKERS] Generating a query that never returns
-BEGIN PGP SIGNED MESSAGE-
Hash: RIPEMD160
I have a need to test timeouts in JDBC, is there a query that is
guaranteed not to return ?
Not *never*, but close enough:
select pg_sleep();
Or if you want to be strict:
CREATE FUNCTION noreturn()
RETURNS VOID
LANGUAGE plperl
AS $$ while (1) { select (undef,undef,undef,0.1) } $$;
- --
Greg Sabino Mullane g...@turnstep.com
End Point Corporation http://www.endpoint.com/
PGP Key: 0x14964AC8 201109191104
http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8
-BEGIN PGP SIGNATURE-
iEYEAREDAAYFAk53WvYACgkQvJuQZxSWSsiItACg+BXmjoR9ecJWuU/AOka+/CBX
rAcAoOQi0MhHk0cWp2aFc87yvZOyY5T1
=wnlW
-END PGP SIGNATURE-
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] A little pg_dump patch
-BEGIN PGP SIGNED MESSAGE- Hash: RIPEMD160 I wrote this little patch, becuse my customer doesn't want to have comments on the production db. It's not my choice Then use pg_restore --use-list to filter them out, and you won't need a Or just strip them out after the fact with a little bit of SQL, e.g. DELETE FROM pg_description WHERE objoid 5; (test first, your system may vary and 5 may not work) - -- Greg Sabino Mullane g...@turnstep.com End Point Corporation http://www.endpoint.com/ PGP Key: 0x14964AC8 201109191143 http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8 -BEGIN PGP SIGNATURE- iEYEAREDAAYFAk53YzsACgkQvJuQZxSWSsgA0wCguiURJI4Hneu7Dn+vBtcEOLsR /OcAn35ujJCWhf2tM3a1J2yek0uddF1X =wc3x -END PGP SIGNATURE- -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_restore --no-post-data and --post-data-only
-BEGIN PGP SIGNED MESSAGE- Hash: RIPEMD160 It's off topic. But I think custom format would require a major mangling to be able to handle a complete cluster. This isn't just a simple matter of programming, IMNSHO. Oh, I meant just having it create separate custom format files for each database. As shell scripts all over the world have been doing for years, but it would be nice if it was simply built in. - -- Greg Sabino Mullane g...@turnstep.com End Point Corporation http://www.endpoint.com/ PGP Key: 0x14964AC8 201109012139 http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8 -BEGIN PGP SIGNATURE- iEYEAREDAAYFAk5gM+oACgkQvJuQZxSWSsi+xgCfbr0q+Ilbw0JRsORLZN2pSz1r JtcAoJaleZvW/wWtU83d9MVeOes4I6+0 =VqFQ -END PGP SIGNATURE- -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Comparing two PostgreSQL databases -- order of pg_dump output
-BEGIN PGP SIGNED MESSAGE- Hash: RIPEMD160 Note that what I'm looking for is something to compare just about EVERYTHING DDL under the PostgreSQL sun: tables, types, functions, operators, etc. The description of same_schema appears to imply only a subset of objects are compared (in fact, looking at the code, I can confirm that limitation). You should try the latest version in git (which will soon be released as 2.18.0). The same_schema check has been overhauled, and now can also store a copy of a databases state to allow checking the same database over time to see what has changed. It doesn't check *everything* yet, but the only things missing are some of the more obscure items such as custom conversions. It should be pretty easy to add in anything that is not already covered, even for someone not versed in Perl. BTW, I tried installing check_postgres, but not being much into Perl and not knowing what dependencies it has, make test failed 38/42 tests. That's not much to worry about. It's a pretty straightforward script, in that it is very easy to determine if it is working for you or not, even if some of the tests fail. :) I'm not exactly sure how it does it check_postgres queries the system catalogs, normalizes some things based on the version, and creates a Perl object representation of the database. It then compares that to the same thing from a different database/server, or to a frozen version of an earlier scan. - -- Greg Sabino Mullane g...@turnstep.com End Point Corporation http://www.endpoint.com/ PGP Key: 0x14964AC8 201108302203 http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8 -BEGIN PGP SIGNATURE- iEYEAREDAAYFAk5dl28ACgkQvJuQZxSWSsidhwCeMGEx8eVeaPlyRALuh8VuQ+rN ynYAoLDGLOFNVbj3+NnRvZpLfgmh6Mgu =w1eI -END PGP SIGNATURE- -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_restore --no-post-data and --post-data-only
-BEGIN PGP SIGNED MESSAGE- Hash: RIPEMD160 Well, notwithstanding my well known love of perl, that strikes me as spending a pound to save a penny. And custom format dumps rock ;-) Also, your recipe above is buggy, BTW. A CREATE INDEX statement might well not be the first item in the post-data section. But we could also add these switches to pg_dump too if people feel it's worthwhile. I haven't looked but the logic should not be terribly hard. A big +1 to --pre-data and --post-data, but until we get there, or if you have an existing dump file (schema *or* schema+data) that needs parsing, there is an existing tool: http://blog.endpoint.com/2010/01/splitting-postgres-pgdump-into-pre-and.html Once these new flags and the ability to custom format dump pg_dumpall is done, I'll have very little left to complain about with pg_dump :) - -- Greg Sabino Mullane g...@turnstep.com End Point Corporation http://www.endpoint.com/ PGP Key: 0x14964AC8 201108271855 http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8 -BEGIN PGP SIGNATURE- iEYEAREDAAYFAk5ZdfwACgkQvJuQZxSWSsipDQCgpmNtD/I/2gfAzm2b3jouD8nS qhgAn33t5VLiF8HeslBwCqyMzQJy6VN5 =PfK7 -END PGP SIGNATURE- -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] index-only scans
-BEGIN PGP SIGNED MESSAGE- Hash: RIPEMD160 1. The way that nodeIndexscan.c builds up the faux heap tuple is perhaps susceptible to improvement. I thought about building a virtual tuple, but then what do I do with an OID column, if I have one? Or maybe this should be done some other way altogether. Maybe it's time to finally remove the been-deprecated-for-a-while OIDs? - -- Greg Sabino Mullane g...@turnstep.com End Point Corporation http://www.endpoint.com/ PGP Key: 0x14964AC8 201108111654 http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8 -BEGIN PGP SIGNATURE- iEYEAREDAAYFAk5EQiEACgkQvJuQZxSWSsglcQCeKsLRvd958M5QJ8YC8aNqr/Ku 11QAn1Iwaz9GuGVOB28orAITCsSX4MOo =JMag -END PGP SIGNATURE- -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] index-only scans
-BEGIN PGP SIGNED MESSAGE- Hash: RIPEMD160 Maybe it's time to finally remove the been-deprecated-for-a-while OIDs? I thought about just not supporting that for index-only scans, but system catalogs use them pretty extensively, and it doesn't seem out of the question that that could matter to people who have lots of SQL objects floating around. Right - when I said remove, I meant for all but system catalogs. I would think those are generally small enough that for most people the lack of index-only scans on those would not matter. Heck, the system catalogs are already special in lots of ways other than having OIDs (most anyway), so it's not as though we'd be breaking sacred ground with an index-only exception. :) I guess the question that should be asked is we are going to finally remove OIDs someday, right?. If so, and if it's potentially blocking a major new feature, why not now? - -- Greg Sabino Mullane g...@turnstep.com End Point Corporation http://www.endpoint.com/ PGP Key: 0x14964AC8 201108112140 http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8 -BEGIN PGP SIGNATURE- iEYEAREDAAYFAk5EhYEACgkQvJuQZxSWSsjnYQCgne81uKjiABVU3X3X+5cM/oFx 74YAoNX97hsOIxBx4Y1hcQHf/bWR813U =hEl2 -END PGP SIGNATURE- -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Indication of db-shared tables
-BEGIN PGP SIGNED MESSAGE- Hash: RIPEMD160 Do we do enough to show which tables are db shared, e.g. pg_database? I don't see any indication from psql \dS. Are our docs clear enough? I don't think \dS should be indicating such a thing. I think it's documented well enough: if you are doing something that it matters enough which tables are shared, you really oughtta know about them anyway. - -- Greg Sabino Mullane g...@turnstep.com End Point Corporation http://www.endpoint.com/ PGP Key: 0x14964AC8 201106212323 http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8 -BEGIN PGP SIGNATURE- iEYEAREDAAYFAk4BYF8ACgkQvJuQZxSWSsjOYACgnDq27MbRCg4Dr7QL/p6tq1kj 3EwAoPnJCqazL+akS1Au5WoxB5RvceDu =RpBk -END PGP SIGNATURE- -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] procpid?
-BEGIN PGP SIGNED MESSAGE- Hash: RIPEMD160 Or perhaps pg_connections. Yes, +1 to making things fully backwards compatible by keeping pg_stat_activity around but making a better designed and better named table (view/SRF/whatever). I thought about that too when reading the thread the first time, but pg_stat_sessions sounds better. Our documentation also primarily refers to a database connection as a session, i think. No, this is clearly connections, not sessions. At least based on the items in the postgresql.conf file, especially max_connections (probably one of the items most closely associated with pg_stat_activity) - -- Greg Sabino Mullane g...@turnstep.com End Point Corporation http://www.endpoint.com/ PGP Key: 0x14964AC8 201106161132 http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8 -BEGIN PGP SIGNATURE- iEYEAREDAAYFAk36IjYACgkQvJuQZxSWSsg8MgCgkMNw1o37cgmtJdYBAsGl7kz6 Q8sAoISFra0LyQjyKw3zcapWBdCLh2RV =EYAc -END PGP SIGNATURE- -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] procpid?
-BEGIN PGP SIGNED MESSAGE- Hash: RIPEMD160 At any rate, I like sessions. That's what it is, after all. But I will note that we had better be darn sure to make all the changes we want to make in one go, because I dowanna have to create pg_sessions2 (or pg_tessions?) in a year or three. Or perhaps pg_connections. Yes, +1 to making things fully backwards compatible by keeping pg_stat_activity around but making a better designed and better named table (view/SRF/whatever). Sounds like perhaps a wiki page to start documenting some of our monitoring shortcomings? Might as well fix as much as we can in one swoop. - -- Greg Sabino Mullane g...@turnstep.com End Point Corporation http://www.endpoint.com/ PGP Key: 0x14964AC8 201106151246 http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8 -BEGIN PGP SIGNATURE- iEYEAREDAAYFAk344ioACgkQvJuQZxSWSshy9wCgnrj4lQkaomsgS55yq9KI0HBl P2UAoI62Tkt9/U62l0Bxv/KfQUUlL/NF =aaTL -END PGP SIGNATURE- -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] procpid?
-BEGIN PGP SIGNED MESSAGE- Hash: RIPEMD160 For me, the litmus test is whether the change provides enough improvement that it outweighs the disruption when the user runs into it. For the procpid that started all of this, the clear answer is no. I'm surprised people seriously considered making this change. It's a historical accident: document and move on. And if we are going to talk about changing misnamed things, I've got a whole bunch of others I could throw at you (such as abbreviation rules: blks_read on the one extreme, and autovacuum_analyze_scale_factor on the other) :) This is why I suggested a specific, useful, and commonly requested (to me at least) change to pg_stat_activity go along with this. +1. The procpid change is silly, but fixing the current_query field would be very useful. You don't know how many times my fingers have typed WHERE current_query 'IDLE' - -- Greg Sabino Mullane g...@turnstep.com End Point Corporation http://www.endpoint.com/ PGP Key: 0x14964AC8 201106142300 http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8 -BEGIN PGP SIGNATURE- iEYEAREDAAYFAk34IRoACgkQvJuQZxSWSsi0dgCgi37mrLYbD6G3dS99GPbSFhHW EjYAniZNpRUXxYmhBHfb1k1LsMSoOHE7 =61nA -END PGP SIGNATURE- -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_listener in 9.0
-BEGIN PGP SIGNED MESSAGE- Hash: RIPEMD160 The pg_listener table was removed in 9.0 in the revamp of LISTEN/NOTIFY. In pgAdmin we used to perform a number of selects from the table to get information about Slony clusters - for example, the PID of the slon process or to check if a process is listening for a specific notification. This allows the app to indicate to the user if there is something wrong with their replication cluster. I can't find any way to get that information now - any ideas? Nope, you are out of luck: the information is locked away and cannot be seen by other processes. I'm sure of this because Bucardo went through the same questioning some time ago. We basically rewrote the app a bit to use the on-disk PID files to replace some of the lost functionality, and sucked up the rest. :) - -- Greg Sabino Mullane g...@turnstep.com End Point Corporation http://www.endpoint.com/ PGP Key: 0x14964AC8 201106010838 http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8 -BEGIN PGP SIGNATURE- iEYEAREDAAYFAk3mNEAACgkQvJuQZxSWSsh8LQCeKD/ot4mvXXd5Lgk4sIHwV0D2 CKsAn3Ub9Bdh0Fuyc0rDZr/OiSD8tkXq =cdCn -END PGP SIGNATURE- -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] Getting a bug tracker for the Postgres project
-BEGIN PGP SIGNED MESSAGE- Hash: RIPEMD160 well that is rather basic functionality of a tracker software and i would expect those to be a given, but I don't think that is where the problems are with implementing a tracker for postgresql.org... Right, the problem has been the lukewarm response from the hackers who would be using it every day, and without whose buy-in using a bug tracker would be possible, but much more difficult. Bug tracking software is definitely religious war territory; most people have a bug tracker they use and tolerate, and pretty much everyone has a bug tracker that they absolutely despise (hi JIRA!). Therefore, I suggest we adopt the first one that someone takes the time to build and implement, along with a plan for keeping it up to date. My own bare bones wish list for such a tracker is: * Runs on Postgres * Has an email interface Make no mistake, whichever we choose, the care of feeding of such a beast will require some precious resources in time from at least two people, probably more. If there is anyone in the community that wants to help the project but hasn't found a way, this is your chance to step up! :) - -- Greg Sabino Mullane g...@turnstep.com End Point Corporation http://www.endpoint.com/ PGP Key: 0x14964AC8 201105282322 http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8 -BEGIN PGP SIGNATURE- iEYEAREDAAYFAk3hvCgACgkQvJuQZxSWSsi8gwCfQq/2WRhtnN8HJKoup5KxTrI6 S6QAn1rhm5QIr5cLplhz6U67ZSv6njK8 =oU4a -END PGP SIGNATURE- -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Prefered Types
-BEGIN PGP SIGNED MESSAGE- Hash: RIPEMD160 Tom Lane wrote: BTW, not to rain on the parade or anything, but I'll bet that rejiggering anything at all here will result in whining that puts the 8.3-era removal of a few implicit casts to shame. I'll take that bet, as it's really hard to imagine anything being worse than the pain caused by 8.3 to many people using Postgres. But if this is anything at all like that (e.g. requiring rewriting tons of SQL queries or modifying system catalogs), then a big fat -1. I know, probably a moot point by now, but 8.3 is a sore spot for me. :) - -- Greg Sabino Mullane g...@turnstep.com PGP Key: 0x14964AC8 201105082230 http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8 -BEGIN PGP SIGNATURE- iEYEAREDAAYFAk3HU0IACgkQvJuQZxSWSshp2gCeLzjdXPQ0NkwDzby0f8DzUErz FUEAoLNkIzJ5jWxVP2Ck3BZgxhd6HUhq =yALY -END PGP SIGNATURE- -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Prefered Types
-BEGIN PGP SIGNED MESSAGE- Hash: RIPEMD160 Tom and I: BTW, not to rain on the parade or anything, but I'll bet that rejiggering anything at all here will result in whining that puts the 8.3-era removal of a few implicit casts to shame. I'll take that bet, as it's really hard to imagine anything being worse than the pain caused by 8.3 to many people using Postgres. You think? At least the 8.3 changes resulted in easily-diagnosed parser errors. The folks who complained about it were complaining because they couldn't be bothered to fix anything about their applications, not because it was difficult to understand or to fix. Those of us in the trenches saw things a little differently. There's a difference between couldn't be bothered and the sometimes herculean task of changing an existing complicated code base, including finding all the problems, fixing, writing tests, doing QA, etc. It was also difficult to explain all this to clients: why their code worked just fine on all previous versions, what the exact theoretical dangers involved are (and agreeing that, yes, it doesn't really apply to their particular code), and the sheer man-hours it was going to take to get their application over the 8.3 hump. (Granted, there's the system catalog hacks, but a) they introduce other problems and b) it's dangerous to reapply constantly when pg_dumping or moving across versions) It seems likely to me that any changes in function resolution behavior will result in failures that are *much* harder to diagnose. The actual fix might be the same (ie, insert an explicit cast or two) but back-tracking from the observed problem to that fix could be an order of magnitude more difficult. For example, if you start noticing an occasional integer overflow that didn't happen before, it might be pretty darn difficult to figure out that the problem is that an operation that was formerly resolved as int4 + int4 is now resolved as int2 + int2. Have I mentioned I'm already a big -1 on the whole idea? :) Yes, this will be a more subtle problem to diagnose, but I also think it will affect less code and thus not elicit as much whining. Besides, I never recommend clients use SMALLINT anyway. (That type you are using: I do not think it's as efficient as you think it is) - -- Greg Sabino Mullane g...@turnstep.com End Point Corporation http://www.endpoint.com/ PGP Key: 0x14964AC8 201105082312 http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8 -BEGIN PGP SIGNATURE- iEYEAREDAAYFAk3HYk4ACgkQvJuQZxSWSshQ+ACePUFS++9q4lhsdWSolIqDuI+r LY4AoOBsEszt1goBe73GBuSW+dt0DfWF =gycE -END PGP SIGNATURE- -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] increasing collapse_limits?
-BEGIN PGP SIGNED MESSAGE- Hash: RIPEMD160 Joshua Berkus wrote: I'm not comfortable with increasing the default, yet. While folks on dedicated good hardware can handle a collapse of 10-12 joins, a lot of people are running PostgreSQL on VMs these days whose real CPU power is no better than a Pentium IV. Really? First, I don't think that's true, the average CPU power is much higher than that. Second, this sounds like the 'ol tune it for a toaster trap where we never make improvements to the defaults because someone, somewhere, might *gasp* use Postgres on an underpowered server. Also, if you're doing OLTP queries on small tables, spending 20ms planning a query is unreasonably slow in a way it is not for a DW query. Again, seriously? Do you have numbers to back that up? I could see not going to 16 right away, but who would honestly have a problem with going to 10? I agree with Tom, let's bump this up a little bit and see what happens. My guess is that we won't see a single post in which we advise people to drop it down from 10 to 8. Personally, I'd like to see them go to 12, as that's the best sweet spot I've seen in the field, but I'll take 10 first. :) Tom Lane asked re setting to 10: Don't know how much difference that would make in the real world though. I've seen a handful of cases that have benefitted from 10, but many more* that benefitted from 12 (*okay, a larger handful anyway, it's not like I have to adjust it too often). - -- Greg Sabino Mullane g...@turnstep.com End Point Corporation http://www.endpoint.com/ PGP Key: 0x14964AC8 201105012153 http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8 -BEGIN PGP SIGNATURE- iEYEAREDAAYFAk2+DqsACgkQvJuQZxSWSshRfQCgzX5JlnCmKTndA7WcF/mt0Kpk b30AoLKrVKMm0rbZNNhgVjt/Xne4NDpj =0deF -END PGP SIGNATURE- -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Alignment padding bytes in arrays vs the planner
-BEGIN PGP SIGNED MESSAGE- Hash: RIPEMD160 Any ideas about better answers? Seems like you covered it - anything other than memcmp() is going to require a lot of brainz and have lots of sharp edges. But this example shows that we'd really have to enforce the rule of no ill-defined bytes for just about every user-callable function's results, which is a pretty ugly prospect. Why is that so ugly? Seems the most logical route. And even if we don't get all of them right away (e.g. not 'enforced' right away), we're no worse off than we are now, but we don't have to dive into retraining equal() or touch any other parts of the code. - -- Greg Sabino Mullane g...@turnstep.com End Point Corporation http://www.endpoint.com/ PGP Key: 0x14964AC8 201104262139 http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8 -BEGIN PGP SIGNATURE- iEYEAREDAAYFAk23dGEACgkQvJuQZxSWSsidwQCgrIc1I85P6a1jF5Xwq1vRbzwF v/wAoImYBZZo930+IGgL61BEQ+1YCMaN =9fkS -END PGP SIGNATURE- -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] branching for 9.2devel
-BEGIN PGP SIGNED MESSAGE- Hash: RIPEMD160 Sounds good to me ... who's volunteering? (Andrew) I will as well. Github perhaps, Andrew? I'll be happy to get some unit tests written. - -- Greg Sabino Mullane g...@turnstep.com End Point Corporation http://www.endpoint.com/ PGP Key: 0x14964AC8 201104252157 http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8 -BEGIN PGP SIGNATURE- iEYEAREDAAYFAk22JnMACgkQvJuQZxSWSsj4SgCg9k2HHBfAVXeZx7CwxDPuUTCX ZkYAnRCalvoKB4yhIeHaZywwBtzcz+93 =JptO -END PGP SIGNATURE- -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] clang and LLVM
-BEGIN PGP SIGNED MESSAGE- Hash: RIPEMD160 I was wondering if there has been anyone experimenting to compile PG using LLVM/clang compiler tools. I got it working on Linux but it required a Postgres src file change to work properly (see previous thread by me). Supposedly the clang bug that caused this was fixed in llvm's HEAD, but HEAD will not compile for me yet, so I cannot verify it yet. There's a separate bug concerning usage of plperl, but that's for another day. - -- Greg Sabino Mullane g...@turnstep.com End Point Corporation http://www.endpoint.com/ PGP Key: 0x14964AC8 201012171144 http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8 -BEGIN PGP SIGNATURE- iEYEAREDAAYFAk0Lk4UACgkQvJuQZxSWSsigEwCdG9aNk50eK1EzelewzqGMhgyO 8fYAoPt+emmdaxEd7lmeYidYgpqIdfjK =6ksq -END PGP SIGNATURE- -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] GCC vs clang
I've been trying to get clang working enough that I can at least get HEAD going for a build farm client, and the attached patch is the bare minimum to get it working. There may be a better way to do this, but as indicated in a past thread, the GNU_SOURCE variable does not play nicely with clang. Getting that removed does allow me to do a working make and make check. The make takes orders of magnitude longer than gcc does, but that's an issue for another day. -- Greg Sabino Mullane g...@turnstep.com PGP Key: 0x14964AC8 201011160940 http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8 diff --git a/src/template/linux b/src/template/linux index 3eb5ad2..527a559 100644 --- a/src/template/linux +++ b/src/template/linux @@ -1,7 +1,9 @@ # src/template/linux # Force _GNU_SOURCE on; plperl is broken with Perl 5.8.0 otherwise -CPPFLAGS=$CPPFLAGS -D_GNU_SOURCE +if $CC = gcc ; then + CPPFLAGS=$CPPFLAGS -D_GNU_SOURCE +fi # If --enable-profiling is specified, we need -DLINUX_PROFILE PLATFORM_PROFILE_FLAGS=-DLINUX_PROFILE pgpfXYEuhNStr.pgp Description: PGP signature
Re: [HACKERS] GCC vs clang
-BEGIN PGP SIGNED MESSAGE- Hash: RIPEMD160 Tom asked: What happens to plperl? It still doesn't work. I was going to leave it out via --without-perl, and save fixing that for another day. There's a handful of other warnings when making, but --with-perl is the only showstopper (once the GNU_SOURCE problem is solved). Peter chimed in: The underlying issue in clang has been resolved, so the next release should work out of the box. I suggest we wait for that. http://llvm.org/bugs/show_bug.cgi?id=5365 Sweet, good to know. I'll update my clang repo and see if it does the trick. - -- Greg Sabino Mullane g...@turnstep.com PGP Key: 0x14964AC8 201011161302 http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8 -BEGIN PGP SIGNATURE- iEYEAREDAAYFAkzix3IACgkQvJuQZxSWSsgKxQCgl2vbUGS9plb1Zm7Sg+sdKR+5 oIUAn0CI9Dky2bQsYkoPhV6yZrQWosvQ =f0+q -END PGP SIGNATURE- -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Version Numbering
-BEGIN PGP SIGNED MESSAGE- Hash: RIPEMD160 David Wheeler: No idea what you mean by that, but generally it's a bad idea to switch from dotted-integer version numbers and numeric version numbers. See Perl (Quel dsastre!). Yeah, I think Perl is a prime example of how NOT to handle version numbering. :) I think we got it right the first time. David Fetter: We're using Postgre 8 See also all the flocks of tools that claim to support Postgres 8 Flocks? Handful at best, and no reason we should be catering to their inaccuracies. - -- Greg Sabino Mullane g...@turnstep.com PGP Key: 0x14964AC8 201008201713 http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8 -BEGIN PGP SIGNATURE- iEYEAREDAAYFAkxu8NkACgkQvJuQZxSWSsgNVACfYko/YC7SOlMXpavO7JXWSZhp i7QAoKmPKvNlASLAYfimtnrpg0lk82vh =aWSL -END PGP SIGNATURE- -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Version Numbering
-BEGIN PGP SIGNED MESSAGE- Hash: RIPEMD160 Then why are we discussing it on -hackers? Because you will need buy in from the hackers if you ever want to do something as radical as change to a two-number, one dot system (or some the slightly less radical earlier suggestions). For the record, I'm with Tom on this: -1 to any changes. I do like the Ubuntu/Debian way of naming the releases with some sort of non-numeric name though. :) - -- Greg Sabino Mullane g...@turnstep.com PGP Key: 0x14964AC8 201008202036 http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8 -BEGIN PGP SIGNATURE- iEYEAREDAAYFAkxvH+EACgkQvJuQZxSWSshfdwCgxutLw7s2o225qvhKRXeJzvwo xVgAnAoptFyCTKljX52q7RTsGElDHswE =yS2/ -END PGP SIGNATURE- -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Version Numbering
-BEGIN PGP SIGNED MESSAGE- Hash: RIPEMD160 Flocks? Handful at best, and no reason we should be catering to their inaccuracies. Depends on the goal. If our goal is to continue to add confusion to the masses of users we have, you are correct. If our goal is to simplify the ability for a user to accurately understand the version of PostgreSQL they are running, then you are wrong. Are we adding confusion? Do you have any proof to back up that assertion? I'm pretty sure the masses can handle the fact that 9.1.x is going to come after 9.0.x, and that 9.0.1 is an bug fix for 9.0.0. True, we don't always have the best track record for bumping major releases. (ponders) Hmmm...I'm rethinking my immediate rejection of the idea now. 7.3 to 7.4 should have been 7.3 to 8.0. Certainly it was more major than 8.0 to 8.1 was, for example. Consider me a very weak -1 and open to persuasion. :) - -- Greg Sabino Mullane g...@turnstep.com PGP Key: 0x14964AC8 201008202130 http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8 -BEGIN PGP SIGNATURE- iEYEAREDAAYFAkxvLGQACgkQvJuQZxSWSsjIoQCfY4ANKov5TV/PDV+mc0Rhda5O wskAoMjZ4y9t+VOlP+84NMfz7Ws1aNVV =qRMV -END PGP SIGNATURE- -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Version Numbering
-BEGIN PGP SIGNED MESSAGE- Hash: RIPEMD160 Look at other DBMSes: Oracle: 8i, 9i, 10g, 11g Informix 9, 10, 11 MS SQL Server 7, 2000, 2005, 2008 is not only confusing but make people think we are somehow behind the others... someone actually told me that Oracle is in version 11 we only in version 8 so Oracle should have more features... no that i follow that reasoning but... Well by that reasoning SQL Server 2008 is a quantum leap ahead of Oracle! Frankly, that 'someone' should be hit hard with a clue stick and be forced to keep 50 feet away from all computers. - -- Greg Sabino Mullane g...@turnstep.com PGP Key: 0x14964AC8 201008202135 http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8 -BEGIN PGP SIGNATURE- iEYEAREDAAYFAkxvLZAACgkQvJuQZxSWSsjFcQCeMQX9fQcLZVv6q1wssFIsIMQE INAAoJPEsMRsezdT2bAWP8xLZ7wSpxvh =yKn1 -END PGP SIGNATURE- -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Version Numbering
-BEGIN PGP SIGNED MESSAGE- Hash: RIPEMD160 It's possible that we're arguing for the sake of arguing No it's not! ;) It's nice to be able to keep track of the major version number without running out of fingers (at least for a few more years) and it's nice to be able to bump the major version number when we do something to totally destabilize the tree^W^W^W^W^Wreally cool. Or at least, I think it's nice. Again, YMMV, IMHO, etc. If the Windows port was the primary justification for the 8.0 designation, and HS/SR are the justification for the 9.0 designation, what will 10.0 be? Therein lies the problem: our decision to do a major bump is inconsistent at best, and wildy confusing at worst. Does a new feature really constitute a major bump? Perhaps so, as with 9.0 SR/HS, but in that case there have been other times we should have bumped the major for some new feature and did not. What about major internal changes and libpq version bumps? You might think those would always be a major change, but they are not. We went from 7.2 to 7.3 without considering how major it is (hello, schemas!). What about end-user compatiblity? I sometimes suspect few hackers on this list realize how completely disruptive, annoying, and painful the removal of implicit casts was in 8.3. That would have been a major bump in my book at least. I think in the future we should consider lowering the bar for a major release, as it's better to err on that side. - -- Greg Sabino Mullane g...@turnstep.com PGP Key: 0x14964AC8 201008202330 http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8 -BEGIN PGP SIGNATURE- iEYEAREDAAYFAkxvSS0ACgkQvJuQZxSWSsjQ0QCfW/2l065L0XEO6kmnARpjgqJ5 t2EAn3xM8w5f5xmHl3EZAmXhxXFpEREo =/CYr -END PGP SIGNATURE- -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] remove upsert example from docs
-BEGIN PGP SIGNED MESSAGE- Hash: RIPEMD160 Attached is a patch to remove the upsert example from the pl/pgsql documentation. It has a serious bug (see: http://www.spinics.net/lists/pgsql/msg112560.html) which is nontrivial to fix. IMNSHO, our code examples should encourage good practices and style. No, removing is a bad idea, as it's referenced from here to the North Pole and back. Better would simply be a warning about the non uniqueness of the unique constraint message. The 'correct' way to do race free upsert is to take a table lock first -- you don't have to loop or open a subtransaction. A high concurrency version is nice but is more of a special case solution (it looks like concurrent MERGE might render the issue moot anyways). I think anything doing table locks should be the special case solution as production systems generally avoid full table locks like the plague. The existing solution works fine as long as we explain that caveat (which is a little bit of a corner case, else we'd have heard more complaints before now). - -- Greg Sabino Mullane g...@turnstep.com End Point Corporation http://www.endpoint.com/ PGP Key: 0x14964AC8 201008051402 http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8 -BEGIN PGP SIGNATURE- iEYEAREDAAYFAkxa/XgACgkQvJuQZxSWSsjTbACfcjrsBVXCOGUb6foARfNIztSo AswAn0bNttP8XOs/2tw6jFsSa0cZkq7e =HUcq -END PGP SIGNATURE- -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] SHOW TABLES
-BEGIN PGP SIGNED MESSAGE- Hash: RIPEMD160 1. \d isn't exactly the most intuitive thing ever Seems fairly mnemomic to me (d=describe) and it packs a *lot* of information into a single letter (see below). Things that are done often should have short keystrokes, and not require learning Yet Another Meta-Language. And it's pretty clear that we have been heading into some increasingly cryptic bits of fruit salad of \dfzb+-meta-bucky-alt-foo No arguments there, but that's the nature of the beast. I don't think it's as bad as is made out, however, as \d covers 99% of everyday usage and certainly the show tables that started this thread. Having SHOW THIS and SHOW THAT which are a bit more readily guessed would be somewhat nice. I'm not sure why easily guessed is thrown out in this thread as such a great thing. To achieve that goal, we simply need the help system that has been proposed many times: entering in SHOW anything gives you a quick rundown of the backslash system. As far as SHOW THIS, there is a big difference from a plain \dt and \d tablename. The former could be emulated quite easily with a SHOW command (although even our \dt prints out more information than mysql's SHOW TABLES), but the latter includes a crazy amount of information that would lead to quite a large SHOW... statement. Also, if it were made a server-side thing, how would you return things like indexes on a table in a SRF? Have a meta-column describing what the other columns represent? Ugly. information_schema doesn't have some useful things that we'd like ait to have ... Alas, I don't see a good way to improve on this :-( newsysviews seems the way out of that particular mess. I'm also not particularly opposed to adding new views or columns to information_schema. We would still support the standard by having all the required views and columns. The \? commands are *solely* for psql, and it would be nice to have the Improvement work on server side so it's not only usable with the one client. Agreed, but is there some other command-line client? If it's not command-line, free-form SQL typing, it inevitably already has support for querying the catalogs built in. At least, every GUI, app, and driver I can think of does. I've seen too many QA scripts that do awk parsing of output of psql \d commands that are vulnerable to all kinds of awfulness. They should be querying information_schema. I'd sure like to be able to write queries that *don't* involve array smashing or using grep on \z output to analyze object permissions. Yeah, that would be a better information_schema. :) - -- Greg Sabino Mullane g...@turnstep.com End Point Corporation http://www.endpoint.com/ PGP Key: 0x14964AC8 201007191011 http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8 -BEGIN PGP SIGNATURE- iEYEAREDAAYFAkxEXS0ACgkQvJuQZxSWSshLKwCffkfe0T3tELInxRqG7yCDS5Vr Ku8AoLUtOu7tTplGZZLPOEuDfKHt+EEm =Oubu -END PGP SIGNATURE- -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] SHOW TABLES
-BEGIN PGP SIGNED MESSAGE- Hash: RIPEMD160 Robert Haas (robertmh...@gmail.com) wrote: I think LIST COMMENTS ON SYSTEM AGGREGATES would be an epic step forward in usability. Perhaps. But it would behoove you to come up with a less er... arcane example. I've been using Postgres a long time, and I can count the number of times I've needed to see comments on system aggregates on my hand. With at least four fingers left over. ... in the alphabet soup paragraph above. I don't think there's anything WRONG with letting \dFp show text search dictionaries and \dfwS+ list system window functions with additional detail - but I'd like an alternative that emphasizes ease of remembering over brevity, works in every client, and can be extended in whatever reasonable ways the community decides are worth having. ... I don't know that I'd necessarily remember all those any better, and would certainly not enjoy typing out: LIST TEST SEARCH DICTIONARIES I don't have to remember \dFp - all I have to remember is \?. For the more common ones that I use day to day and don't have to look up (\d \dt \df \l etc.) the advantage of a two or three character string is strong. (There is some devil's advocate in there - a standard cross client (and dare I say it, cross RDBMS?) way would be nice) ... being powerful rings totally hollow for me. For ordinary, day to day tasks like listing all my tables, or looking at the details of a particular table, they're great. I use them all the time and would still use them even if some other syntax were available. But there is no reasonable way to pass options to them, and that to me is a pretty major drawback. Well, there's the rub. You're arguing this from a hacker's persepective, while the SHOW syntax seems to be overwhelmingly agreed upon to be either helpful for clueless noobs, or some nice syntactic sugar for average users. I'm not sure where to draw the line but implementing a proper shortcut interface for cammands is something taht should be done on the client side because not every client is the same and the needs of psql might be radically different from any other client (like pgadmin or a fancy Web 2.0 AJAX thingy - those will likely always use custom catalog queries). Maybe a differnet way to look at the whole thing is to reconsider our own catalogs (anyone remember newsysview?) and add a bunch of views to abstract away most of the current complexity for these usecases? Yep, agreed. Now, if we can just agree to put information_schema in the default search_path, because nobody enjoys having to type out information_schema... - -- Greg Sabino Mullane g...@turnstep.com End Point Corporation http://www.endpoint.com/ PGP Key: 0x14964AC8 201007191021 http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8 -BEGIN PGP SIGNATURE- iEYEAREDAAYFAkxEYDgACgkQvJuQZxSWSsikFwCdGo88Ehdcm8OHi2+VxISTG60Y b9sAoLsetxcpdMSconsCwj+3Xa1fCCzo =3aM1 -END PGP SIGNATURE- -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] SHOW TABLES
-BEGIN PGP SIGNED MESSAGE- Hash: RIPEMD160 Kevin Grittner wrote: Any solution which only works within psql isn't a solution for a large part of the problem space people are trying to address. One important goal is that if someone spends a day to whip up a GUI query tool (as I did when I first started working in Java), it's easy to get displays like we get from the psql backslash commands (as it was in Sybase, which is what we were using at the time, through sp_help and related stored procedures). I don't agree that this is an important goal. Certainly someone writing a GUI (or a new driver) should be expected to be familiar with the system catalogs. Moreover, a GUI relies on an underlying driver, and every driver should already be providing things like a list of tables natively. - -- Greg Sabino Mullane g...@turnstep.com End Point Corporation http://www.endpoint.com/ PGP Key: 0x14964AC8 201007191030 http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8 -BEGIN PGP SIGNATURE- iEYEAREDAAYFAkxEYZ0ACgkQvJuQZxSWSsiIlQCfdXDgTqletVez/r+pKHY4EcW6 QAsAoPLUmblzN2aNEw5DveHEav3XyB/K =TGq1 -END PGP SIGNATURE- -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] SHOW TABLES
-BEGIN PGP SIGNED MESSAGE- Hash: RIPEMD160 David Fetter wrote: No arguments there, but that's the nature of the beast. I don't think it's as bad as is made out, however, as \d covers 99% of everyday usage and certainly the show tables that started this thread. It covers 0% of cases where people are not using psql. Yes, and everything else already has a show tables. See for example, PPA: http://phppgadmin.sourceforge.net/images/4.png - -- Greg Sabino Mullane g...@turnstep.com End Point Corporation http://www.endpoint.com/ PGP Key: 0x14964AC8 201007191342 http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8 -BEGIN PGP SIGNATURE- iEYEAREDAAYFAkxEj4kACgkQvJuQZxSWSshrwgCg65eIziE2SW8XhdTSHwVMzxnm ynIAoLPOc0yuKyrE2kaaJFq5UiDb45Nd =veva -END PGP SIGNATURE- -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] psql auto-completion for multiple where clauses
-BEGIN PGP SIGNED MESSAGE- Hash: RIPEMD160 Someone highlighed on IRC that after the first WHERE clause, autocomplete no longer works. ... SELECT * FROM tab_completion WHERE id = 2 AND stabtab ... Is there any chance of improving this so it would work for more than 1 WHERE clause? I notice it also doesn't work for GROUP BY or HAVING at all, but seems to be fine for ORDER BY. No: there is only a small number of words that we go back through, so the above will not work as we cannot get back to the name of the table from the right side of the AND. The way to fix that is to redesign our tab-completion system such that it knows about a greater number of words, perhaps even the complete statement. SELECT * FROM tab_completion WHERE id = 2 AND btab Since there is no column beginning with b, it might be an idea to get it to match bark bark instead. It might help alleviate what may be a gotcha for some. This one is more doable, assuming we are really talking about: SELECT * FROM tab_completion WHERE btab Keep in mind it will show up in a list if you do the following: SELECT * FROM tab_completion WHERE tab - -- Greg Sabino Mullane g...@turnstep.com End Point Corporation http://www.endpoint.com/ PGP Key: 0x14964AC8 201007161102 http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8 -BEGIN PGP SIGNATURE- iEYEAREDAAYFAkxAdOkACgkQvJuQZxSWSsiMpACgvheNYe35eXugYQrR3fZ7AYl2 ZWoAnAwzDPREKuxrJzZK45TpInUCh03w =E6eG -END PGP SIGNATURE- -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
