Re: [HACKERS] Roadmaps 'n all that
Josh Berkus josh@agliodbs.com writes: I propose a modest experiment: for the 8.3 development cycle, let's try to agree (in the next month or so) on a roadmap of what major features should be in 8.3 and who will make each one happen. Well, I think the what is more important that the who -- we can switch whos if that's what it takes to make the what happen. Likely, we will have to. No doubt --- but if there's not someone specific who's agreed to take on each item on the roadmap, then what is it but pie-in-the-sky wishing? I'm entirely comfortable with the idea that we put some things on the roadmap that end up not being done when 8.3 release rolls around. We've been disappointed that way a few times before ;-) ... we won't be worse off if it happens again, and I'd rather aim high than low. I think the important point to try for in this iteration is that someone signs up for each thing the community thinks is important to get done, and that we all believe no one has taken on anything they can't credibly get done. regards, tom lane ---(end of broadcast)--- TIP 5: don't forget to increase your free space map settings
Re: [HACKERS] Prepared statements considered harmful
On Thu, Aug 31, 2006 at 07:04:07PM -0400, Gregory Stark wrote: The server has to prepare the query sometime. The v3 protocol just gives you control over when that happens, but it doesn't force you to do it at any particular time. Not really. All named prepares are planned straight away, all unnamed ones are planned at bind time. Therefore you cannot have more than one parsed-but-not-planned prepared query at a time. In a connection pool scenario there's no way to share such plans since you can't tell which query has been prepared. That's not forcing, but it's an asymmetry we could do with out. Have a nice day, -- Martijn van Oosterhout kleptog@svana.org http://svana.org/kleptog/ From each according to his ability. To each according to his ability to litigate. signature.asc Description: Digital signature
Re: [HACKERS] Prepared statements considered harmful
Martijn van Oosterhout wrote: On Thu, Aug 31, 2006 at 07:04:07PM -0400, Gregory Stark wrote: The server has to prepare the query sometime. The v3 protocol just gives you control over when that happens, but it doesn't force you to do it at any particular time. Not really. All named prepares are planned straight away, all unnamed ones are planned at bind time. Therefore you cannot have more than one parsed-but-not-planned prepared query at a time. In a connection pool scenario there's no way to share such plans since you can't tell which query has been prepared. That's not forcing, but it's an asymmetry we could do with out. AFAIK since Oracle 9i planning is always deferred until the first execution. This way they hope to get a better plan, which would obviously not be possible if the selectivity varies greatly. So are the plans generated without taking any bound values into account more stable in performance (albeit at times slower than what would have been produced if the value would have been known)? Either way mixing the question of when to prepare the plan with the prepared statement being named or unnamed seems unexpected. regards, Lukas ---(end of broadcast)--- TIP 2: Don't 'kill -9' the postmaster
Re: [HACKERS] Prepared statements considered harmful
I'd wish that we reconsider when and how prepared statements are
used. The JDBC interface and PL/pgSQL are frequently noticed
perpetrators, but the problem is really all over the place.
AFAIK those are the only two places where preparation is
the default
... what else were you thinking of?
Perl DBI (DBD::Pg) defaults to prepared plans when connecting to a
version 8.0 or higher server.
Or at least, that's the way I read the documentation.
Yea, but if you close the statement or leave the scope of the statement
variable the plan is gone. So it is doing exactly what I would expect.
It is written $stmt-prepare('select 1') what else would you expect ?
There are enough other functions to get a result without a plan sticking
around, like $db-selectrow_array
Andreas
---(end of broadcast)---
TIP 5: don't forget to increase your free space map settings
Re: [HACKERS] Prepared statements considered harmful
On Fri, Sep 01, 2006 at 09:26:24AM +0200, Lukas Kahwe Smith wrote: AFAIK since Oracle 9i planning is always deferred until the first execution. This way they hope to get a better plan, which would obviously not be possible if the selectivity varies greatly. Interesting. So are the plans generated without taking any bound values into account more stable in performance (albeit at times slower than what would have been produced if the value would have been known)? Possibly, though they might uniformly suck or be uniformly good... Either way mixing the question of when to prepare the plan with the prepared statement being named or unnamed seems unexpected. Yes, well, when the V3 protocol designed it wasn't considered to seperate the parse and plan phases. The exact commit that got us to where we are now is here: http://developer.postgresql.org/cvsweb.cgi/pgsql/src/backend/tcop/postgres.c#rev1.420 That was over two years ago, I'm not sure whether any of the ideas mentioned in there have been implemented since. Have a nice day, -- Martijn van Oosterhout kleptog@svana.org http://svana.org/kleptog/ From each according to his ability. To each according to his ability to litigate. signature.asc Description: Digital signature
Re: [HACKERS] Win32 hard crash problem
My bet is something depending on GetTickCount to measure elapsed time (and no, it's not used in the core Postgres code, but you've got plenty of other possible culprits in that stack). This doesn't quite make sense. The only reason we have to reboot is because PostgreSQL no longer responds. The system itself is fine. The Windows kernel may still work, but that doesn't mean that everything Postgres depends on still works. I'm wondering about (a) the TCP stack (and that includes 3rd party firewalls and such, not only the core Windows code); (b) timing or threading stuff inside the application that's using libpq, which the only thing we know about so far is that it's *not* JDBC/Hibernate. How about getting a simple backtrace from a couple of the stuck postgres processes? And from the postmaster which should be accepting new connections... Or does that also hang completely? How to get one? Well, since we don't have the MSVC build yet (yeah, yeah, eventually), you can only get a semi-backtrace that only looks at exported symbols. You can get this using process explorer (thread tab, click stack), using WinDBG or using Visual Studio (you'll need VS 2005, and you need to check the option for Load DLL exports in options-debugging-native). Oh, btw, if there is a 3rd firewall on the box the standard recommendation of uninstalling it definitely sounds like a good plan :-) //Magnus ---(end of broadcast)--- TIP 2: Don't 'kill -9' the postmaster
Re: [HACKERS] Win32 hard crash problem
Oops, going backwards through the mails it seems :) Subsequent connections to the database will fail (such as pgAdmin) and Windows must be completely rebooted. Fail in what way. Hang, not connect, or get an error msg? PostgreSQL will also not recover on its own (e.g; auto restart and roll through the logs). What do you mean by this? It doesn't start upon reboot? What is needed to make it start? //Magnus ---(end of broadcast)--- TIP 2: Don't 'kill -9' the postmaster
Re: [HACKERS] Win32 hard crash problem
My bet is something depending on GetTickCount to measure elapsed time (and no, it's not used in the core Postgres code, but you've got plenty of other possible culprits in that stack). This doesn't quite make sense. The only reason we have to reboot is because PostgreSQL no longer responds. The system itself is fine. The Windows kernel may still work, but that doesn't mean that everything Postgres depends on still works. It may be a not reacting listen socket. This may be because of a handle leak. Next time it blocks look at the handle counts (e.g. with handle.exe from sysinternals). You could also look for handle count now with Task Manager and see if it increases constantly. (handle.exe shows you the details) Andreas ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [HACKERS] Prepared statements considered harmful
On Thu, August 31, 2006 21:41, Phil Frost wrote: Is there any kind of pattern at all to this problem? Anything recognizable? A few typical pitfalls? Frequently I have found preplanning will result in a horrible plan because it is assumed parameters may be volatile while in practice they are literals. Here is a function from my database: That's a very common thing in processor design as well, and there's a standard trick for it: the saturating two-bit counter. It tends to work pretty well for branch prediction, value prediction etc. Usually it's the first thing you reach for, so of course somebody may already have tried it here and found it didn't work. In this particular case it might be applied something like this: for each parameter in a prepared statement you cache a predictor value, plus a confidence counter saying (more or less--see below) how many times in succession that value has repeated. Let's say each of the counters count from 0 to 3 inclusive, with its confidence threshold right in the middle, between 1 and 2. On every invocation, you check each parameter value against the corresponding predictor value. If it's identical, you increment its counter (provided it can be incremented any further). If it isn't, you decrement its counter, and if the counter ends up below its confidence threshold, you replace the predictor value with the new parameter value. Then, whenever any new planning needs to be done (I'll get to that in a moment), you see which counters are above their confidence thresholds. In your new planning you assume that all parameters with confident predictions will remain pseudo-constant for the next few invocations. Of course there's a problem when parameters do not match predicted values. That's where having one or two backup plans could come in handy. You could keep your original, fully-generalized plan around. If plans are cheap enough to store, you could try to keep a cache of old plans for the same query. The great thing about keeping some backup plans around is that a pseudo-constant parameter can have a different value once in a while, then flick back to its old habits without invalidating all your efforts. Your usually-unused search fields are a good example. You may also have two stable parameter patterns with different sets of pseudo-constants competing for your attention. It's not perfect, and it clearly has its pathological cases--but if it works well enough overall, the bad cases could be caught and handled as exceptions. Confidence counters can be tweaked to lose confidence more easily than they gain it, or vice versa. Some meta-confidence scheme may catch the worst offenders. I won't go into that now--first I'll shut up and wait for others to point out what I've missed. :) Jeroen ---(end of broadcast)--- TIP 2: Don't 'kill -9' the postmaster
Re: [HACKERS] GIN FailedAssertions on Itanium2 with Intel compiler
Huh, it's a over-optimization by icc on Itanium. With -00 or -02 there is no any problem, only -O2 produces such effect. The problem is in code Sorry, right With -00 or -01 there is no any problem -- Teodor Sigaev E-mail: [EMAIL PROTECTED] WWW: http://www.sigaev.ru/ ---(end of broadcast)--- TIP 6: explain analyze is your friend
Re: [HACKERS] Prepared statements considered harmful
On Fri, Sep 01, 2006 at 03:56:19PM +0700, Jeroen T. Vermeulen wrote: That's a very common thing in processor design as well, and there's a standard trick for it: the saturating two-bit counter. It tends to work pretty well for branch prediction, value prediction etc. Usually it's the first thing you reach for, so of course somebody may already have tried it here and found it didn't work. Interesting thought. It might be worth trying. But my big question: is all this testing and counting actually going to be faster than just replanning? Postgresql's planner is not that slow. Of course there's a problem when parameters do not match predicted values. That's where having one or two backup plans could come in handy. You could keep your original, fully-generalized plan around. If plans are cheap enough to store, you could try to keep a cache of old plans for the same query. The great thing about keeping some backup plans around is that a pseudo-constant parameter can have a different value once in a while, then flick back to its old habits without invalidating all your efforts. Your usually-unused search fields are a good example. You may also have two stable parameter patterns with different sets of pseudo-constants competing for your attention. The thing is that number of possible plans is going to be proportional to factorial(number of tables). Once you have 3 tables you're going to have at least a dozen possible plans, probably more. What the best plan is depends strongly on what the parameters are. Anyway, your plan assumes that you have information to work with. The current system plans prepared queries with no information at all about parameters and people are advocating to keep it that way. I think a good first step would be the plan on first execution, like Oracle does. Have a nice day, -- Martijn van Oosterhout kleptog@svana.org http://svana.org/kleptog/ From each according to his ability. To each according to his ability to litigate. signature.asc Description: Digital signature
Re: [HACKERS] Prepared statements considered harmful
Anyway, your plan assumes that you have information to work with. The current system plans prepared queries with no information at all about parameters and people are advocating to keep it that way. I think a good first step would be the plan on first execution, like Oracle does. Yup, it is also possible to try to find an obvious plan and only delay planning (or part of the plan) when different inputs make a big difference (like MaxDB and Informix). Andreas ---(end of broadcast)--- TIP 4: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] Prepared statements considered harmful
I don't chime in very often, but I do think the refusal to incorporate hints into the planner system is fantastically stubborn and nonsensical. What is actually fantastically nonsensical about this is that the issues I outlined about prepared statements would merely become worse if planner hints were used. Then, you wouldn't only have to worry about plans that were created earlier during the session, you would be faced with plans that were created earlier during the application's development. In general, the solutions to the prepared statement issues need to effect that the plans are created more often, not less often. I have yet to see one of our partial Informix hints (where the planner does it's usual job only with one path with lowered/elevated costs) fall foul on not anticipated change of underlying data. Thus I don't buy the argument that hints are always bad. Of course their use should be extremely rare and well thought out. Most of the time sql tuning involves a concerted effort between the programmer and a db performance expert, usually resulting in rewritten sql or program logic without adding hints. I can see arguments for hints the dba can set himself centrally on the server, but in my experience chances for substantial improvement are very limited in that case. Andreas ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [HACKERS] GIN FailedAssertions on Itanium2 with Intel compiler
% icc -O2 -o 1 1.c ./1 CALL cmp SET SI = 1 BUG: SI==0 % icc -O1 -o 1 1.c ./1 CALL cmp SET SI = 1 OK BTW, this example works correct with -nolib-inline (http://www.intel.com/software/products/compilers/clin/docs/main_cls/mergedprojects/copts_cls/ccpp_options/option_nolib_inline.htm): % icc -O2 -nolib_inline -o 1 1.c ./1 CALL cmp SET SI = 1 OK PS Have anybody any thoughts about workaround? -- Teodor Sigaev E-mail: [EMAIL PROTECTED] WWW: http://www.sigaev.ru/ ---(end of broadcast)--- TIP 9: In versions below 8.0, the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [PATCHES] [HACKERS] Interval aggregate regression failure
On Sep 1, 2006, at 11:31 , Bruce Momjian wrote: Tom Lane wrote: Bruce Momjian [EMAIL PROTECTED] writes: I am unclear about this report. The patch was not meant to fix every interval issue, but merely to improve multiplication and division computations. Does it do that? According to Michael's last report, your patch fails under --enable-integer-datetimes. Where does it fail? Here? select interval '41 mon 12 days 360:00' * 0.3 as product_a , interval '-41 mon -12 days +360:00' * 0.3 as product_b , interval '-41 mon 12 days 360:00' * 0.3 as product_c , interval '-41 mon -12 days -360:00' * 0.3 as product_d; product_a | product_b | product_c |product_d --+- ++- 1 year 11 days 146:24:00 | -1 years -11 days +69:36:00 | -1 years -5 days +98:24:00 | -1 years -11 days -146:23:60.00 - That is wrong, but I think we need another fix for that. Notice the problem is in minutes/seconds, not hours. I was sure it was more wrong than that the first time I saw it, but looks like I can't be sure of anything today :(. I need more sleep. Sorry for the noise on this one. Off work now, so I'm back at it. Michael Glaesemann grzm seespotcode net ---(end of broadcast)--- TIP 1: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [PATCHES] [HACKERS] Interval aggregate regression failure
On Sep 1, 2006, at 11:03 , Bruce Momjian wrote: I am unclear about this report. The patch was not meant to fix every interval issue, but merely to improve multiplication and division computations. Does it do that? I think the 23:60 is a time rounding issue that isn't covered in this patch. I am not against fixing it, but does the submitted patch improve things or not? Given we are post-feature freeze, we don't have time to fix all the interval issues. Your patch doesn't fix the things Tom referenced (nor did you intend it to). I just wanted to to collect examples of all the known issues with the interval code in one place. Probably too ambitious for September 1. Is it worth looking into the overflow and subtraction issues for 8.2? It seems to me they're bugs rather than features. Or are these 8.3 since it's so late? Michael Glaesemann grzm seespotcode net ---(end of broadcast)--- TIP 1: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] GIN FailedAssertions on Itanium2 with Intel compiler
On Fri, Sep 01, 2006 at 03:10:44PM +0400, Teodor Sigaev wrote: BTW, this example works correct with -nolib-inline (http://www.intel.com/software/products/compilers/clin/docs/main_cls/mergedprojects/copts_cls/ccpp_options/option_nolib_inline.htm): % icc -O2 -nolib_inline -o 1 1.c ./1 CALL cmp SET SI = 1 OK PS Have anybody any thoughts about workaround? What does that option do? Is it practical to enable it for the entire backend? And isn't this a straightforward compiler bug they should be notified about? Sounds like the -fno-strict-aliasing option for gcc, the compiler is making some assumptions about the code that arn't true, so we tell the compiler not to do that. Have a nice day, -- Martijn van Oosterhout kleptog@svana.org http://svana.org/kleptog/ From each according to his ability. To each according to his ability to litigate. signature.asc Description: Digital signature
Re: [HACKERS] Status on Fedora Core Extras packaging
On Wednesday 30 August 2006 05:12, Devrim GUNDUZ wrote:
Hello,
Yesterday night, I was approved (sponsored) for Fedora Core packaging
(please note that this is not something financial).
For the beginning, I officially submitted python-psycopg2 to FC6. More
are on the way; like pgpool, Slony-I, pgadmin3-adminpacks, phpPgAdmin,
postgresql_autodoc, pgfouine. Also they are in the queue for FC5. I'm
waiting for approval for them.
More will be in FC Extras. Please let me know that if you want to see
any PostgreSQL related software in the repository. I do have time to
package all related stuff.
Any chance at getting some of the non-core pl langs in ?
--
Robert Treat
Build A Brighter LAMP :: Linux Apache {middleware} PostgreSQL
---(end of broadcast)---
TIP 5: don't forget to increase your free space map settings
[HACKERS] about how to un-describe from pgsql-hackers
Hi, guys i have already send a undescribe request to [EMAIL PROTECTED], but i receive a reject reply. who can tell me how to undescribe it ? thanks. From: [EMAIL PROTECTED] Reply-To: [EMAIL PROTECTED] To: zhou bo [EMAIL PROTECTED] Subject: Majordomo results: unsubscribe pgsql-hackers Date: Fri, 01 Sep 2006 08:27:30 -0300 unsubscribe pgsql-hackers The unsubscribe command did not succeed. No e-mail addresses matching zhou bo [EMAIL PROTECTED] are subscribed to the pgsql-hackers mailing list. Valid commands processed: 1 0 succeeded, 0 stalled, and 1 failed. Use the following command: sessioninfo 15453d661d9e06ac044850d282fbe5935edba4bd to see technical information about this session. ---(end of broadcast)--- TIP 6: explain analyze is your friend
Re: [HACKERS] GIN FailedAssertions on Itanium2 with Intel compiler
What does that option do? Is it practical to enable it for the entire backend? From docs: Disables inline expansion of standard library or intrinsic functions. And isn't this a straightforward compiler bug they should be notified about? What's a choice? Now I see 3: 1) -O1 2) volatile 3) -nolib_inline IMHO, only -O1 is guarantee for other possible places... But I'm not familiar enough with such kinds of bugs. -- Teodor Sigaev E-mail: [EMAIL PROTECTED] WWW: http://www.sigaev.ru/ ---(end of broadcast)--- TIP 9: In versions below 8.0, the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [PATCHES] [HACKERS] Interval aggregate regression failure
Here's a patch that appears to work. Gives the same output with and without --enable-integer-datetimes. Answers look like they're correct. I'm basically treating the components as three different intervals (with the other two components zero), rounding them each to usecs, and adding them together. While it might be nice to carry a little extra precision around, it doesn't seem to be needed in these cases. If errors do arise, they should be at most 3 usec, which is pretty much noise for the floating point case, I suspect. Bruce, how's it look on your machine? If it looks good, I'll add the examples we've been using to the regression tests. Michael Glaesemann grzm seespotcode net Index: src/backend/utils/adt/timestamp.c === RCS file: /projects/cvsroot/pgsql/src/backend/utils/adt/timestamp.c,v retrieving revision 1.165 diff -c -r1.165 timestamp.c *** src/backend/utils/adt/timestamp.c 13 Jul 2006 16:49:16 - 1.165 --- src/backend/utils/adt/timestamp.c 1 Sep 2006 11:26:12 - *** *** 2494,2511 float8 factor = PG_GETARG_FLOAT8(1); double month_remainder, day_remainder, ! month_remainder_days; Interval *result; result = (Interval *) palloc(sizeof(Interval)); ! month_remainder = span-month * factor; ! day_remainder = span-day * factor; result-month = (int32) month_remainder; result-day = (int32) day_remainder; month_remainder -= result-month; day_remainder -= result-day; /* * The above correctly handles the whole-number part of the month and day * products, but we have to do something with any fractional part --- 2494,2553 float8 factor = PG_GETARG_FLOAT8(1); double month_remainder, day_remainder, ! month_remainder_days, ! month_remainder_time, ! day_remainder_time; Interval *result; result = (Interval *) palloc(sizeof(Interval)); ! ! month_remainder = span-month / factor; ! day_remainder = span-day / factor; result-month = (int32) month_remainder; result-day = (int32) day_remainder; month_remainder -= result-month; day_remainder -= result-day; + month_remainder_days = month_remainder * DAYS_PER_MONTH; + + /* + if month_remainder_days is not an integer, check to see if it's an + integer when converted to SECS or USECS. + If it is, round month_remainder_days to the nearest integer + +*/ + + if (month_remainder_days != (int32)month_remainder_days + TSROUND(month_remainder_days * SECS_PER_DAY) == + rint(month_remainder_days * SECS_PER_DAY)) + month_remainder_days = rint(month_remainder_days); + + result-day += (int32)month_remainder_days; + + #ifdef HAVE_INT64_TIMESTAMP + month_remainder_time = rint((month_remainder_days - + (int32)month_remainder_days) * USECS_PER_DAY); + + day_remainder_time = rint(day_remainder * USECS_PER_DAY); + + + result-time = rint(rint(span-time * factor) + day_remainder_time + + month_remainder_time); + #else + month_remainder_time = rint((month_remainder_days - + (int32)month_remainder_days) * SECS_PER_DAY); + day_remainder_time = rint(day_remainder * SECS_PER_DAY); + + result-time = span-time * factor + day_remainder_time + + month_remainder_time; + #endif + + + day_remainder = span-day * factor; + result-day = (int32) day_remainder; + day_remainder -= result-day; + /* * The above correctly handles the whole-number part of the month and day * products, but we have to do something with any fractional part *** *** 2518,2531 /* fractional months full days into days */ month_remainder_days = month_remainder * DAYS_PER_MONTH; ! result-day += (int32) month_remainder_days; ! /* fractional months partial days into time */ ! day_remainder += month_remainder_days - (int32) month_remainder_days; #ifdef HAVE_INT64_TIMESTAMP ! result-time = rint(span-time * factor + day_remainder * USECS_PER_DAY); #else ! result-time = span-time * factor + day_remainder * SECS_PER_DAY; #endif PG_RETURN_INTERVAL_P(result); --- 2560,2599 /* fractional months full days into days */ month_remainder_days = month_remainder * DAYS_PER_MONTH; ! /* !* The remainders suffer from float
Re: [PATCHES] [HACKERS] Interval aggregate regression failure
Please ignore the patch I just sent. Much too quick with the send button. Michael Glaesemann grzm seespotcode net ---(end of broadcast)--- TIP 9: In versions below 8.0, the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [HACKERS] Roadmaps 'n all that
Joshua D. Drake wrote: we will know whether this is a great thing we should continue, or we should stick to our traditional laissez-faire style of project management. I figure that even if it really sucks, it wouldn't kill us to try it for one release cycle --- at the very worst, we'd make up lost time in future by no longer needing to waste bandwidth arguing about it. Would this be a core postgresql code roadmap or something a bit broader (contrib, custom types, GUI-ish stuff, utilities and what have you)? I think that we could at this time only expect things that would be submitted for core inclusion. The less cats to herd the better :) yeah I would agree that it makes sense to do that for core postgresql code only for now - most of the others stuff is not even directly tied to the 8.3 development cycle either(GUIs will have to support more than a single release probably as would maybe special custom types or such). Stefan ---(end of broadcast)--- TIP 6: explain analyze is your friend
Re: [PATCHES] [HACKERS] Interval aggregate regression failure
Michael Glaesemann [EMAIL PROTECTED] writes: Is it worth looking into the overflow and subtraction issues for 8.2? It seems to me they're bugs rather than features. Or are these 8.3 since it's so late? IMHO they're bugs not new features, and therefore perfectly fair game to work on during beta. But for the moment I'd suggest staying focused on the interval_mul/interval_div roundoff issue. regards, tom lane ---(end of broadcast)--- TIP 1: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] [PATCHES] Updatable views
Bernd Helmle [EMAIL PROTECTED] writes: I would like to try to grab your idea to push down the CHECK OPTION logic down to the executor as a (table/view?) constraint. Would that be an idea worth to consider and, most important, is this doable? I don't have that much experience in the executor, so i fear this isn't something that will be done within a week or so.:( You're certainly welcome to work on it --- I don't have time at the moment. But I agree there's little chance of getting it done in time for 8.2. I have not read the spec's definition of WITH CHECK OPTION lately, so there may be something fundamentally wrong in what I'm about to say, but the way I'm envisioning this working is that W.C.O. is embodied as a check constraint (or something pretty similar --- a pg_constraint entry certainly) attached to the view. The rewriter would then be responsible for collecting all the check options that need to be enforced for a given rewritten query. This'd require some rework of the rewriter/planner/executor API: right now the executor is solely responsible for collecting check constraints to apply in an updating query, and we'd want to change that. My thought is we might as well move the collection responsibility over to the rewriter 100%: the rewriter decorates a Query with the list of constraint expressions to apply, and the executor just checks what it's told to. The planner probably need not do much with the constraint expressions beyond what it normally does with, say, targetlist expressions. Some thoughts: * It's too early in the morning for me to be clear about the difference between CASCADED and LOCAL CHECK OPTION --- I think that this would merely alter the set of check constraints collected for a particular query, but if there's something more fundamental there, this scheme might not work at all. So look into that first. * The reason we currently collect constraints at executor start is that ALTER TABLE ADD CONSTRAINT has no way to invalidate cached plans, so it's unsafe to store lists of constraints in plans. So this scheme *requires* that we have a plan invalidation mechanism in place (at least before we release, maybe not before the patch can go in). This doesn't bother me because I intend anyway to see to it that there's plan inval in 8.3. * With check constraints now passing through the planner, it'd become trivial to allow subqueries in check constraints. Just sayin'. * We'd probably eliminate the idea of storing constraints in TupleDescs, which would be a good simplification anyway (eg, eliminate the bogus distinction between CreateTupleDescCopy and CreateTupleDescCopyConstr). OTOH that might make it harder to allow rowtypes to have associated constraints? Needs thought. regards, tom lane ---(end of broadcast)--- TIP 2: Don't 'kill -9' the postmaster
Re: [PATCHES] [HACKERS] Interval aggregate regression failure
On Sep 1, 2006, at 20:39 , Michael Glaesemann wrote: Here's a patch that appears to work. Gives the same output with and without --enable-integer-datetimes. Answers look like they're correct. I'm basically treating the components as three different intervals (with the other two components zero), rounding them each to usecs, and adding them together. While it might be nice to carry a little extra precision around, it doesn't seem to be needed in these cases. If errors do arise, they should be at most 3 usec, which is pretty much noise for the floating point case, I suspect. Okay. Here's the patch. Bruce, does it work for you? Michael Glaesemann grzm seespotcode net Index: src/backend/utils/adt/timestamp.c === RCS file: /projects/cvsroot/pgsql/src/backend/utils/adt/timestamp.c,v retrieving revision 1.165 diff -c -r1.165 timestamp.c *** src/backend/utils/adt/timestamp.c 13 Jul 2006 16:49:16 - 1.165 --- src/backend/utils/adt/timestamp.c 1 Sep 2006 12:28:23 - *** *** 2494,2504 float8 factor = PG_GETARG_FLOAT8(1); double month_remainder, day_remainder, ! month_remainder_days; Interval *result; result = (Interval *) palloc(sizeof(Interval)); month_remainder = span-month * factor; day_remainder = span-day * factor; result-month = (int32) month_remainder; --- 2494,2507 float8 factor = PG_GETARG_FLOAT8(1); double month_remainder, day_remainder, ! month_remainder_days, ! month_remainder_time, ! day_remainder_time; Interval *result; result = (Interval *) palloc(sizeof(Interval)); + month_remainder = span-month * factor; day_remainder = span-day * factor; result-month = (int32) month_remainder; *** *** 2506,2511 --- 2509,2553 month_remainder -= result-month; day_remainder -= result-day; + month_remainder_days = month_remainder * DAYS_PER_MONTH; + + /* + if month_remainder_days is not an integer, check to see if it's an + integer when converted to SECS or USECS. + If it is, round month_remainder_days to the nearest integer + +*/ + + if (month_remainder_days != (int32)month_remainder_days + TSROUND(month_remainder_days * SECS_PER_DAY) == + rint(month_remainder_days * SECS_PER_DAY)) + month_remainder_days = rint(month_remainder_days); + + result-day += (int32)month_remainder_days; + + #ifdef HAVE_INT64_TIMESTAMP + month_remainder_time = rint((month_remainder_days - + (int32)month_remainder_days) * USECS_PER_DAY); + + day_remainder_time = rint(day_remainder * USECS_PER_DAY); + + + result-time = rint(rint(span-time * factor) + day_remainder_time + + month_remainder_time); + #else + month_remainder_time = rint((month_remainder_days - + (int32)month_remainder_days) * SECS_PER_DAY); + day_remainder_time = rint(day_remainder * SECS_PER_DAY); + + result-time = span-time * factor + day_remainder_time + + month_remainder_time; + #endif + + + day_remainder = span-day * factor; + result-day = (int32) day_remainder; + day_remainder -= result-day; + /* * The above correctly handles the whole-number part of the month and day * products, but we have to do something with any fractional part *** *** 2518,2531 /* fractional months full days into days */ month_remainder_days = month_remainder * DAYS_PER_MONTH; ! result-day += (int32) month_remainder_days; ! /* fractional months partial days into time */ ! day_remainder += month_remainder_days - (int32) month_remainder_days; #ifdef HAVE_INT64_TIMESTAMP ! result-time = rint(span-time * factor + day_remainder * USECS_PER_DAY); #else ! result-time = span-time * factor + day_remainder * SECS_PER_DAY; #endif PG_RETURN_INTERVAL_P(result); --- 2560,2599 /* fractional months full days into days */ month_remainder_days = month_remainder * DAYS_PER_MONTH; ! /* !* The remainders suffer from float rounding, so if they are !* within 1us of an integer, we round them to integers. !* It seems doing two multiplies is causing the imprecision. !*/ #ifdef HAVE_INT64_TIMESTAMP ! if (month_remainder_days != (int32)month_remainder_days !
Re: [HACKERS] [PATCHES] Updatable views
--On Donnerstag, August 31, 2006 11:10:47 -0400 Tom Lane
[EMAIL PROTECTED] wrote:
The problem is not with the view condition. Consider
CREATE TABLE data (id serial primary key, ...);
CREATE VIEW only_new_data AS SELECT * FROM data WHERE id 12345
WITH CHECK OPTION;
INSERT INTO only_new_data VALUES(nextval('data_id_seq'), ...);
The proposed implementation will execute nextval twice (bad), and will
apply the WITH CHECK OPTION test to the value that isn't the one stored
(much worse). It doesn't help if the id is defaulted.
*scratches head*i don't see a shortcoming solution for this in my
current implementation,
indeed. I admit that this is a serious containment of updatable views in
its current
incarnation (at least for the check option).
*thinking*
I would like to try to grab your idea to push down the CHECK OPTION logic
down to the executor as a (table/view?) constraint. Would that be an idea
worth to consider and,
most important, is this doable? I don't have that much experience in the
executor, so i fear
this isn't something that will be done within a week or so.:(
--
Thanks
Bernd
---(end of broadcast)---
TIP 2: Don't 'kill -9' the postmaster
Re: [HACKERS] Prepared statements considered harmful
On Fri, September 1, 2006 16:53, Martijn van Oosterhout wrote: Interesting thought. It might be worth trying. But my big question: is all this testing and counting actually going to be faster than just replanning? Postgresql's planner is not that slow. In the best case (which of course would have to be very frequent for any of this to matter in the first place) it's mainly just a short loop comparing the call's parameter values to their counterparts stored with the plan and update those two-bit confidence counters. You wouldn't *believe* how simple you have to keep these things in processor architecture. :-) The thing is that number of possible plans is going to be proportional to factorial(number of tables). Once you have 3 tables you're going to have at least a dozen possible plans, probably more. What the best plan is depends strongly on what the parameters are. Of course. That's the whole point: to end up with a small but effective subset of all those possible plans. I'd guess that you could cover even most of the nasty cases with a maximum of three plans or so per prepared statement, including the original fully-generalized one. The plans could be replaced on an LRU basis, which isn't very costly for three or so entries. Anyway, your plan assumes that you have information to work with. The current system plans prepared queries with no information at all about parameters and people are advocating to keep it that way. I think a good first step would be the plan on first execution, like Oracle does. Yes, delaying things a bit can help a lot sometimes. That's also what JIT compilers in JVMs do, for instance. FWIW, libpqxx doesn't prepare statements until they're first called anyway. But if this choice to discard parameter information is exactly what causes a lot of the bad plans in the first place, as Peter says, what's wrong with putting it to use instead? For those cases, you're pretty much screwed by definition as long as you fail to do so. And it's not like what I'm suggesting is very difficult! The real question is whether it's worthwhile. To find that out, we'd need to estimate four factors: coverage (how often you'd get a useful prediction), accuracy (how often that prediction would be accurate), cost of misprediction (near-zero compared to current situation, assuming we keep the generalized plans handy), and savings for correct prediction (in our case, benefit of planning for a constant instead of a variable minus the cost of re-planning which you say isn't very expensive). Based on what Peter and you tell me about cost, the main worries here are coverage and accuracy. Coverage and accuracy can be extracted (and tweaked!) relatively easily if we have logs of prepared-statement executions in a wide variety of real-life applications. Listings of consecutive prepared-statement invocations (statement name plus parameter values) are all that's needed. Do we have any logs like that? If we do, I'll be more than happy to run some simulations and see if the idea shows any promise. Like I said, there's every chance that it doesn't. It was just an off-the-cuff suggestion and if it's no good I'll have no problems saying so. But there's not much point sitting around arguing over theoretical merits if they're that easy to quantify! Jeroen ---(end of broadcast)--- TIP 4: Have you searched our list archives? http://archives.postgresql.org
[HACKERS] Sort performance
I'm not sure if this is good news or bad news. Either some kudos are due to the gang that worked on the external sort performance or something's very wrong with the qsort implementation in glibc because I'm seeing Postgres's external sort perform better than qsort. This is despite Postgres external sorts having to execute filesystem calls pushing buffers back and forth between user-space and kernel-space, which seems hard to believe. I feel like something's got to be pretty far wrong with the qsort call here for this to be possible. At first I chalked this up to qsort having O(n^2) behaviour occasionally but a) This is glibc where qsort is actually mergesort which should behave pretty similarly to Postgres's mergesort and b) the input data is randomized pretty well so it really ought be a problem even were it qsort. Mem RunsTime 1MB 18 8.25s 10MB3 5.6s 100MB qsort 6.1s The input is a table with one column, a text field. It contains /usr/share/dict/words ordered by random() and then repeated a bunch of times. (Sorry about the imprecision, I set this table up a while ago and don't remember exactly what I did). a The machine has plenty of RAM and isn't swapping or running any other services. -- Gregory Stark EnterpriseDB http://www.enterprisedb.com ---(end of broadcast)--- TIP 5: don't forget to increase your free space map settings
Re: [HACKERS] Prepared statements considered harmful
Martijn van Oosterhout kleptog@svana.org writes: The server has to prepare the query sometime. The v3 protocol just gives you control over when that happens, but it doesn't force you to do it at any particular time. Not really. All named prepares are planned straight away, all unnamed ones are planned at bind time. Therefore you cannot have more than one parsed-but-not-planned prepared query at a time. In a connection pool scenario there's no way to share such plans since you can't tell which query has been prepared. That's not forcing, but it's an asymmetry we could do with out. Sure, but how much does it really matter? If you don't want the plan saved for reuse, merely avoiding retransmission of the query text does not seem like a major win. Having had time to think about it, I no longer think the protocol design is a blocking bug for this problem area. It's something we could improve when we are ready to design protocol V4, but it does not seem in itself enough reason to make a new protocol (with all the pain that entails). regards, tom lane ---(end of broadcast)--- TIP 9: In versions below 8.0, the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [HACKERS] Prepared statements considered harmful
On Fri, Sep 01, 2006 at 11:53:11AM +0200, Martijn van Oosterhout wrote: On Fri, Sep 01, 2006 at 03:56:19PM +0700, Jeroen T. Vermeulen wrote: That's a very common thing in processor design as well, and there's a standard trick for it: the saturating two-bit counter. It tends to work pretty well for branch prediction, value prediction etc. Usually it's the first thing you reach for, so of course somebody may already have tried it here and found it didn't work. Interesting thought. It might be worth trying. But my big question: is all this testing and counting actually going to be faster than just replanning? Postgresql's planner is not that slow. The difference between a pre-planned query, and a plan each time query, for me, seems to be a minimum of around 0.3 - 0.5 ms. This is on a fairly modern AMD X2 3800+. If the tests and counting are kept simple - I don't see why they would take anywhere near that long. Cheers, mark -- [EMAIL PROTECTED] / [EMAIL PROTECTED] / [EMAIL PROTECTED] __ . . _ ._ . . .__. . ._. .__ . . . .__ | Neighbourhood Coder |\/| |_| |_| |/|_ |\/| | |_ | |/ |_ | | | | | | \ | \ |__ . | | .|. |__ |__ | \ |__ | Ottawa, Ontario, Canada One ring to rule them all, one ring to find them, one ring to bring them all and in the darkness bind them... http://mark.mielke.cc/ ---(end of broadcast)--- TIP 9: In versions below 8.0, the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [HACKERS] Prepared statements considered harmful
Martijn van Oosterhout kleptog@svana.org writes: Interesting thought. It might be worth trying. But my big question: is all this testing and counting actually going to be faster than just replanning? Postgresql's planner is not that slow. ... The thing is that number of possible plans is going to be proportional to factorial(number of tables). Yeah. One of the reasons the planner is acceptably fast is that it is aggressive about discarding candidate plans as soon as they are clearly inferior to other plans. Tracking multiple plans that might be optimal under varying assumptions about the query parameters would make things exponentially slower. regards, tom lane ---(end of broadcast)--- TIP 1: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] [PATCHES] DOC: catalog.sgml
Zdenek Kotala [EMAIL PROTECTED] writes: I little bit enhanced overview catalog tables. I added two new columns. First one is OID of catalog table and second one contains attributes which determine if the table is bootstrap, with oid and global. Why is this a good idea? It seems like mere clutter. regards, tom lane ---(end of broadcast)--- TIP 9: In versions below 8.0, the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [HACKERS] [PATCHES] Backend SSL configuration enhancement
On Thu, Aug 31, 2006 at 12:11:46 +0400, Victor B. Wagner [EMAIL PROTECTED] wrote: It contains !MD5 element, because MD5 digest algorithm was broken about year ago, and PostgreSQL expected to work with versions of OpenSSL which still consider it strong. MD5 wasn't completely broken and I believe it is still considered safe for the way it is used in SSL. It looks like SHA-1 is pretty much in the same boat now. (See http://www.heise-security.co.uk/news/77244) ---(end of broadcast)--- TIP 9: In versions below 8.0, the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
[HACKERS] Sort performance
I'm not sure if this is good news or bad news. Either some kudos are due to the gang that worked on the external sort performance or something's very wrong with the qsort implementation in glibc because I'm seeing Postgres's external sort perform better than qsort. And here's a really perverse case. The external sort runs in 740 milliseconds but qsort takes over 2 seconds: postgres=# select count(*) from (select * from (select * from w5 limit 20) as x order by w ) as x; count 20 (1 row) Time: 740.324 ms postgres=# set work_mem = '12MB'; SET Time: 0.145 ms postgres=# select count(*) from (select * from (select * from w5 limit 20) as x order by w ) as x; count 20 (1 row) Time: 2051.317 ms LOG: statement: set work_mem = '11MB'; LOG: statement: select count(*) from (select * from (select * from w5 limit 20) as x order by w ) as x; LOG: begin tuple sort: nkeys = 1, workMem = 11264, randomAccess = f LOG: switching to external sort with 41 tapes: CPU 0.01s/0.04u sec elapsed 0.05 sec LOG: performsort starting: CPU 0.01s/0.34u sec elapsed 0.35 sec LOG: finished writing run 1 to tape 0: CPU 0.01s/0.52u sec elapsed 0.54 sec LOG: finished writing final run 2 to tape 1: CPU 0.01s/0.60u sec elapsed 0.62 sec LOG: performsort done (except 2-way final merge): CPU 0.01s/0.63u sec elapsed 0.65 sec LOG: external sort ended, 593 disk blocks used: CPU 0.02s/0.71u sec elapsed 0.73 sec LOG: statement: set work_mem = '12MB'; LOG: statement: select count(*) from (select * from (select * from w5 limit 20) as x order by w ) as x; LOG: begin tuple sort: nkeys = 1, workMem = 12288, randomAccess = f LOG: performsort starting: CPU 0.00s/0.06u sec elapsed 0.06 sec LOG: doing qsort of 20 tuples LOG: performsort done: CPU 0.00s/1.99u sec elapsed 2.00 sec LOG: internal sort ended, 11919 KB used: CPU 0.00s/2.03u sec elapsed 2.04 sec -- Gregory Stark EnterpriseDB http://www.enterprisedb.com ---(end of broadcast)--- TIP 5: don't forget to increase your free space map settings
Re: [HACKERS] Sort performance
What version of pgsql? Recent changes stripped the sort set down considerably in size in external sort, I'm not sure the same is done if the data doesn't spill to disk. - Luke Sent by GoodLink (www.good.com) -Original Message- From: Gregory Stark [mailto:[EMAIL PROTECTED] Sent: Friday, September 01, 2006 11:03 AM Eastern Standard Time To: pgsql-hackers Subject:[HACKERS] Sort performance I'm not sure if this is good news or bad news. Either some kudos are due to the gang that worked on the external sort performance or something's very wrong with the qsort implementation in glibc because I'm seeing Postgres's external sort perform better than qsort. This is despite Postgres external sorts having to execute filesystem calls pushing buffers back and forth between user-space and kernel-space, which seems hard to believe. I feel like something's got to be pretty far wrong with the qsort call here for this to be possible. At first I chalked this up to qsort having O(n^2) behaviour occasionally but a) This is glibc where qsort is actually mergesort which should behave pretty similarly to Postgres's mergesort and b) the input data is randomized pretty well so it really ought be a problem even were it qsort. Mem RunsTime 1MB 18 8.25s 10MB3 5.6s 100MB qsort 6.1s The input is a table with one column, a text field. It contains /usr/share/dict/words ordered by random() and then repeated a bunch of times. (Sorry about the imprecision, I set this table up a while ago and don't remember exactly what I did). a The machine has plenty of RAM and isn't swapping or running any other services. -- Gregory Stark EnterpriseDB http://www.enterprisedb.com ---(end of broadcast)--- TIP 5: don't forget to increase your free space map settings ---(end of broadcast)--- TIP 2: Don't 'kill -9' the postmaster
Re: [HACKERS] Prepared statements considered harmful
Jeroen T. Vermeulen [EMAIL PROTECTED] writes: On Fri, September 1, 2006 16:53, Martijn van Oosterhout wrote: Interesting thought. It might be worth trying. But my big question: is all this testing and counting actually going to be faster than just replanning? Postgresql's planner is not that slow. In the best case (which of course would have to be very frequent for any of this to matter in the first place) it's mainly just a short loop comparing the call's parameter values to their counterparts stored with the plan and update those two-bit confidence counters. You wouldn't *believe* how simple you have to keep these things in processor architecture. :-) I think the slow part is trying to figure out whether to count the current call as a hit or a miss. How do you determine whether the plan you're running is the best plan without replanning the query? -- Gregory Stark EnterpriseDB http://www.enterprisedb.com ---(end of broadcast)--- TIP 5: don't forget to increase your free space map settings
Re: [HACKERS] Prepared statements considered harmful
On Fri, September 1, 2006 21:30, Tom Lane wrote: Yeah. One of the reasons the planner is acceptably fast is that it is aggressive about discarding candidate plans as soon as they are clearly inferior to other plans. Tracking multiple plans that might be optimal under varying assumptions about the query parameters would make things exponentially slower. AFAICS the planner shouldn't be affected at all--it'd just be invoked more often as and when the need for new plans became apparent. Not exponentially (that's an overused word anyway) but proportionally to that. I've been assuming that once you have a plan, storing it is not very expensive. If, say, doubling the number of plans stored with a session's prepared statements incurs some serious cost (apart from the planning itself, of course) then that changes things. Jeroen ---(end of broadcast)--- TIP 1: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] Prepared statements considered harmful
On Fri, Sep 01, 2006 at 10:18:37AM -0400, Tom Lane wrote: Sure, but how much does it really matter? If you don't want the plan saved for reuse, merely avoiding retransmission of the query text does not seem like a major win. Having had time to think about it, I no longer think the protocol design is a blocking bug for this problem area. It's something we could improve when we are ready to design protocol V4, but it does not seem in itself enough reason to make a new protocol (with all the pain that entails). Well, I see the documentation does actually describe the situation fairly well, so perhaps the right approach is to get interface writers to appreciate the difference and indicate if named or unnamed prepares are used. Have a nice day, -- Martijn van Oosterhout kleptog@svana.org http://svana.org/kleptog/ From each according to his ability. To each according to his ability to litigate. signature.asc Description: Digital signature
Re: [HACKERS] insert/update/delete returning and rules
Jaime Casanova [EMAIL PROTECTED] writes: On 8/15/06, Tom Lane [EMAIL PROTECTED] wrote: I'm tempted to suggest that the RETURNING commands might need to be separate rule events, and that to support this you'd need to write an additional rule: CREATE RULE r1 AS ON INSERT RETURNING TO myview DO INSTEAD INSERT ... RETURNING ... This is something for 8.3? Well, if we put it off till 8.3 we are going to have to write something pretty lame in the documentation about views not working with RETURNING. After some further thought, I think we could make it work if we treat XXX RETURNING as a distinct rule event type and make the following restrictions (which are exactly analogous to the restrictions for ON SELECT rules) for ON XXX RETURNING rules: * there can be only one ON XXX RETURNING rule per relation; * it has to be an unconditional DO INSTEAD rule; * it has to have a single action that is the same type of operation it's replacing (or maybe we could allow any RETURNING command?); * the RETURNING list has to match the column datatypes of the view. Perhaps later we could support more stuff, but this particular case would cover simple needs and it doesn't seem like something we'd regret supporting later. The main thing we'd be setting in stone is that the RETURNING commands require a different rule type, which is a bit tedious but I don't really see a good way around it. (Hopefully the updatable-views patch will soon save people from having to write all these rules out by hand, anyway.) I don't have a patch yet, but preliminary experimentation suggests that the rewriter will Just Work, and all we'll need is straightforward boilerplate code to support the additional possible values of pg_rewrite.ev_type --- so probably less than a day's work. Thoughts, objections? regards, tom lane ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [HACKERS] [PATCHES] Updatable views
Bernd Helmle wrote: [EMAIL PROTECTED] wrote: * It's too early in the morning for me to be clear about the difference between CASCADED and LOCAL CHECK OPTION --- I think that this would merely alter the set of check constraints collected for a particular query, but if there's something more fundamental there, this scheme might not work at all. So look into that first. LOCAL checks the data to be updated against its own view WHERE condition only, where CASCADED involves all WHERE conditions of all underlying views. I don't understand this part very well. Say if you have a view WITH CHECK OPTION whose condition is foo 5, and then define a view WITH LOCAL CHECK OPTION on top of that, whose condition is bar 5. Does the local check option on the second view that I can insert a row with foo=4, bar=6? That doesn't violate the condition of bar 5, so it seems fine to me. But it also seems quite idiotic because it violated the original foo5 condition. -- Alvaro Herrerahttp://www.CommandPrompt.com/ The PostgreSQL Company - Command Prompt, Inc. ---(end of broadcast)--- TIP 2: Don't 'kill -9' the postmaster
Re: [HACKERS] [PATCHES] Updatable views
Bernd Helmle [EMAIL PROTECTED] writes: What we can do is to restrict view updates that involves a volatile function completely. As soon as the rewriter wants to apply an implicit system rule to a current query which holds volatile functions, we could treat this as an error. So in other words, views on serial columns don't work? I don't think that's going to be acceptable. In order you want to do a CASCADED CHECK OPTION, you need to collect all expressions out of underlying relations and rewrite them to match the table you are selecting...that looks like a very expensive operation. Not really worse than what the rewriter is doing already --- in fact, I think it's isomorphic to what would happen to the rule qual expressions in your existing patch. So we need to stall this idea unless we have something workable in this area. So what's the plan for 8.2? Should we reject updatable views completely or is there some interest to apply this without CHECK OPTION? I'm about to propose that we should try to go beta next week (see forthcoming message). If you can strip down your patch to avoid the multi-eval problems in the next couple of days, I'm still willing to consider it, but at the moment I'm assuming that it needs to be held for 8.3. regards, tom lane ---(end of broadcast)--- TIP 5: don't forget to increase your free space map settings
Re: [HACKERS] Sort performance
Luke Lonergan [EMAIL PROTECTED] writes: What version of pgsql? Recent changes stripped the sort set down considerably in size in external sort, I'm not sure the same is done if the data doesn't spill to disk. This is a recent CVS checkout. If you're referring to MinimalTuples I think that's done before tuplesort ever sees the tuples. Besides when swapping things around in memory only the first datum and a pointer to the rest of the object actually gets moved around. I think. Now that I've investigated further I'm even more confused though. The cases where I'm seeing external sorts outperform internal sorts are when it just barely exceeds work_mem which means it's only doing one merge pass between initial tapes generated using inittapes. That means most of the work is actually being done using in-memory sorts. Guess what algorithm we use to generate initial tapes: heap sort! * See Knuth, volume 3, for more than you want to know about the external * sorting algorithm. We divide the input into sorted runs using replacement * selection, in the form of a priority tree implemented as a heap * (essentially his Algorithm 5.2.3H), So basically our heap sort implementation is 3x as fast a glibc's qsort implementation?! Is that believable? Certainly I don't get results like that if I just change the code to do a heap sort instead of qsort. I see it being substantially slower. [aside, that said that may be a useful feature to have: a user option to use our internal heap sort instead of qsort. I'm thinking of users on platforms where libc's qsort either performs poorly or is buggy. Since we have all the code for heap sort there already anyways...] I feel like I'm missing some extra work tuplesort is doing (possibly needlessly) in addition to the qsort. Now I'm getting paranoid that perhaps this is just a bug in my hacked up copy of this code. I can't see how that could be but I'll try reproducing it with stock CVS Postgres. -- Gregory Stark EnterpriseDB http://www.enterprisedb.com ---(end of broadcast)--- TIP 6: explain analyze is your friend
Re: [HACKERS] [PATCHES] Updatable views
[Quick note: sorry if you received this mail multiple times, i've moved to a new workstation and my MUA gots hosed up with its identities (all of them has the same adress, suddenly) and I recognized that too late.i'm sorry] --On Freitag, September 01, 2006 10:03:42 -0400 Tom Lane [EMAIL PROTECTED] wrote: Bernd Helmle [EMAIL PROTECTED] writes: You're certainly welcome to work on it --- I don't have time at the moment. But I agree there's little chance of getting it done in time for 8.2. What we can do is to restrict view updates that involves a volatile function completely. As soon as the rewriter wants to apply an implicit system rule to a current query which holds volatile functions, we could treat this as an error. However, i haven't looked into that right now how doable that would be, but it seems correct in terms of data reliability (treat it as volatile in view update is evil ;) Maybe it's worth to look how other database systems solve this problem. I have not read the spec's definition of WITH CHECK OPTION lately, so there may be something fundamentally wrong in what I'm about to say, but the way I'm envisioning this working is that W.C.O. is embodied as a check constraint (or something pretty similar --- a pg_constraint entry certainly) attached to the view. The rewriter would then be responsible for collecting all the check options that need to be enforced for a given rewritten query. This'd require some rework of the rewriter/planner/executor API: right now the executor is solely responsible for collecting check constraints to apply in an updating query, and we'd want to change that. My thought is we might as well move the collection responsibility over to the rewriter 100%: the rewriter decorates a Query with the list of constraint expressions to apply, and the executor just checks what it's told to. The planner probably need not do much with the constraint expressions beyond what it normally does with, say, targetlist expressions. In order you want to do a CASCADED CHECK OPTION, you need to collect all expressions out of underlying relations and rewrite them to match the table you are selecting...that looks like a very expensive operation. Some thoughts: * It's too early in the morning for me to be clear about the difference between CASCADED and LOCAL CHECK OPTION --- I think that this would merely alter the set of check constraints collected for a particular query, but if there's something more fundamental there, this scheme might not work at all. So look into that first. LOCAL checks the data to be updated against its own view WHERE condition only, where CASCADED involves all WHERE conditions of all underlying views. That said, it's necessary to grep out all WHERE conditions of all relations involved in an update operation and apply them to the current query as a constraint expression. The current implementation passes this recursively via a conditional rule through the rewriter. It looked to me as an attractive implementation, but it has this annoying multiple evaluation side effects:( * The reason we currently collect constraints at executor start is that ALTER TABLE ADD CONSTRAINT has no way to invalidate cached plans, so it's unsafe to store lists of constraints in plans. So this scheme *requires* that we have a plan invalidation mechanism in place (at least before we release, maybe not before the patch can go in). This doesn't bother me because I intend anyway to see to it that there's plan inval in 8.3. So we need to stall this idea unless we have something workable in this area. So what's the plan for 8.2? Should we reject updatable views completely or is there some interest to apply this without CHECK OPTION? Some basic functionality could be simulated with table constraints, however, it's not what users out there would expect * With check constraints now passing through the planner, it'd become trivial to allow subqueries in check constraints. Just sayin'. That would be a nice feature, indeed ;) * We'd probably eliminate the idea of storing constraints in TupleDescs, which would be a good simplification anyway (eg, eliminate the bogus distinction between CreateTupleDescCopy and CreateTupleDescCopyConstr). OTOH that might make it harder to allow rowtypes to have associated constraints? Needs thought. regards, tom lane -- Thanks Bernd ---(end of broadcast)--- TIP 2: Don't 'kill -9' the postmaster
[HACKERS] Getting a move on for 8.2 beta
(I've already bounced this off the core committee, but it's time for wider discussion.) September is upon us and it doesn't seem like we are ready to ship a beta. I think it's time to start making some hard choices. In the main code I see these outstanding features/patches: * bitmap indexes * updatable views * GUC variable reload + refactoring (previously applied and reverted) * plpython improvements (needs review by someone who knows plpython) * plpgsql improvements for returning record types * patch to build on VC * make libpq default client_encoding setting from locale? In contrib we've got: * new ISBN/etc module * hstore (finally proposed for inclusion) * new sslinfo module * pgstattuple changes * removing the deadwood These are not all the open issues, for sure, but these are what I think have to be resolved before we can go beta. Everything else is in the category of bug fixes, and none of it seems like a showstopper (in fact a lot of the small stuff on my todo list is bugs reported against 8.1, so those certainly are not beta-stoppers). My feeling is that we ought to bounce bitmap indexes and updatable views as not being ready, accept all the contrib stuff, and try to get the other items done in time for a beta at, say, the end of next week. I'm not thrilled about postponing those two large items till 8.3, but we are a month past feature freeze and we do not have credible patches in hand for either. The alternative is to be willing to slip the schedule until whenever they are ready, and we have learned the hard way that that's not good project management. Comments? regards, tom lane ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [HACKERS] Sort performance
Gregory Stark [EMAIL PROTECTED] writes: Mem RunsTime 1MB 18 8.25s 10MB 3 5.6s 100MB qsort 6.1s I'm confused what this means exactly? Are you saying that in the first two cases, there were 18 and 3 sorted runs generated in the initial pass, and in the third case we just did the sort in memory using qsort? How many items are being sorted, exactly? Since it's text, it probably also makes a big difference what LC_COLLATE setting you are using. Non-C sort locale could mean that the strcoll() calls swamp all else. How long does it take sort(1) to do the same task? regards, tom lane ---(end of broadcast)--- TIP 4: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] [COMMITTERS] pgsql: Revert change to turn autovacuum on by default.
[EMAIL PROTECTED] (Peter Eisentraut) writes: Revert change to turn autovacuum on by default. It looks like you reverted that whole patch including the changes to the default autovacuum threshold/scale parameters. I'd be inclined to keep those. regards, tom lane ---(end of broadcast)--- TIP 2: Don't 'kill -9' the postmaster
Re: [HACKERS] Prepared statements considered harmful
On Fri, September 1, 2006 22:14, Gregory Stark wrote: I think the slow part is trying to figure out whether to count the current call as a hit or a miss. How do you determine whether the plan you're running is the best plan without replanning the query? The question of knowing which plan is best _based on what's in the actual tables_ would be unsolved just as it always was. The scheme addresses only the opportunity to optimize for pseudo-constant parameters. It treats the existing planner as a black box. If you find a solution to the problem of inaccurate statistics, it'll probably be more or less orthogonal to what I'm describing: you could have one or the other, but combining them shouldn't be much harder. I don't think telling hits from misses would be all that hard. Let's say you're having a prepared statement called, and you're evaluating a candidate plan. Each parameter is in one of two sets: those predicted by the plan to have certain values (let's call them P), and those not predicted by the plan because their confidence counters were below the threshold (I'm tempted to call this set NP, but let's make it Q instead). Whether a parameter is in P or in Q can be derived from its confidence counter. In my previous example, you just take its most-significant bit. * For any parameter in P, if the actual value does not match the plan's prediction, you have a miss. Can't use this plan. Use another if you have one that applies (such as your regular old non-specialized plan--that always applies), or if not, write a new one! If you get through this without finding a mismatch, congratulations: you have a hit. The plan you're looking at is applicable to your call. But now we see if we can do better: * For any parameter in Q, if its value would have been predicted correctly but its counter was below the confidence threshold, you increment the counter. If that lifts the counter above the threshold, you have room for improving on this plan. It means there's a good chance you can re-plan for the case that this parameter is also a pseudo-constant, without the effort being wasted. Of course you could also set a minimum number of invocations between re-plannings to get a more long-term view (e.g. different parameters being recognized as pseudo-constants in subsequent calls--you may not want to re-plan for each of those calls). So which plan do you execute if you have more than one applicable candidate? We can see what works well. As a starter I would definitely pick the one with the larger P (smaller Q), breaking ties in favour of the most recently generated plan. I'm assuming we only want one plan for a given P. We'd probably want to limit the number of candidate plans per statement to some very small, fixed number--somewhere between one and four, I'd say; or maybe one generalized plan plus up to two specialized ones. With numbers like that, none of this should be very expensive. A simple linear match against 1-4 candidates may be more effective than any attempt to be clever. I must admit I haven't thought through all of the consequences of caching more than one specialized plan per statement. For example, we could give every cached plan its own set of confidence counters, and match an incoming invocation against each of those; or we could keep just one most likely plan with its associated predictor state, and only consider previously generated plans if we either miss or find room for improvement in the predictor. Jeroen ---(end of broadcast)--- TIP 1: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] [PATCHES] DOC: catalog.sgml
Tom Lane wrote: Zdenek Kotala [EMAIL PROTECTED] writes: I little bit enhanced overview catalog tables. I added two new columns. First one is OID of catalog table and second one contains attributes which determine if the table is bootstrap, with oid and global. Why is this a good idea? It seems like mere clutter. What's global? A maybe-useful flag would be telling that a table is shared. Is that it? Mind you, it's not useful to me because I know which tables are shared, but I guess for someone not so familiar with the catalogs it could have some use. The OIDs may be useful to people inspecting pg_depend, for example; but then, it's foolish not to be using regclass in that case. Whether a table is bootstrap or not doesn't seem useful to me. -- Alvaro Herrerahttp://www.CommandPrompt.com/ The PostgreSQL Company - Command Prompt, Inc. ---(end of broadcast)--- TIP 2: Don't 'kill -9' the postmaster
Re: [HACKERS] insert/update/delete returning and rules
I wrote: After some further thought, I think we could make it work if we treat XXX RETURNING as a distinct rule event type and make the following restrictions (which are exactly analogous to the restrictions for ON SELECT rules) for ON XXX RETURNING rules: After working on this for a bit, I realized that there's a serious, probably fatal objection to this approach: it's arguably a security hole. Suppose that you have a regular table on which you've defined rules that you consider security-critical --- maybe an ON INSERT DO ALSO that logs the action in a log table, for example. Now you migrate your database to 8.2. If we treat INSERT RETURNING as a separate rule event type, then any DB user can bypass your security-critical rules simply by using INSERT RETURNING instead of INSERT. Yeah, you can fix that by adding more rules, but it's not comfy-making to think that DB schemas will be insecure as soon as they are ported to 8.2 until they are fixed. In any case this thought blows out of the water the assumption that we can disallow auxiliary rules for RETURNING events --- on a plain table, that's an important feature to have. So here's my Plan B: the set of rule event types stays the same, and we give the rewriter a little bit of smarts about how to handle RETURNING, while still putting the burden on the rule author to say exactly what to return. Specifically, I suggest: * A rule can have a command with a RETURNING clause only if it's an unconditional DO INSTEAD rule, and there can be only one RETURNING clause among a table's rules for a particular event type. The clause must match the datatypes of the relation's columns. * When rewriting a query that does not have a RETURNING clause, the rewriter simply throws away any RETURNING clause in the rule. * When rewriting a query that does have a RETURNING clause, the rewriter rewrites the rule's RETURNING clause to generate the data required by the query RETURNING clause (same transformation as we do on a view SELECT targetlist). If there's no RETURNING in the rules, throw an error. With this approach, you still have to update your rules if you want to support RETURNING on your views --- but if you don't update them, you don't have a security hole. Basically the standard setup for an updatable view would use ON INSERT DO INSTEAD INSERT INTO ... RETURNING ... where today you don't write any RETURNING. Again, this is something we might want to generalize later, but it seems to be a reasonable basic capability. Thoughts? Have I missed something (again)? regards, tom lane ---(end of broadcast)--- TIP 4: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] [PATCHES] Updatable views
On 9/1/06, Alvaro Herrera [EMAIL PROTECTED] wrote: Bernd Helmle wrote: [EMAIL PROTECTED] wrote: * It's too early in the morning for me to be clear about the difference between CASCADED and LOCAL CHECK OPTION --- I think that this would merely alter the set of check constraints collected for a particular query, but if there's something more fundamental there, this scheme might not work at all. So look into that first. LOCAL checks the data to be updated against its own view WHERE condition only, where CASCADED involves all WHERE conditions of all underlying views. I don't understand this part very well. Say if you have a view WITH CHECK OPTION whose condition is foo 5, and then define a view WITH LOCAL CHECK OPTION on top of that, whose condition is bar 5. Does the local check option on the second view that I can insert a row with foo=4, bar=6? That doesn't violate the condition of bar 5, so it yes. or at least that's the way i read that... seems fine to me. But it also seems quite idiotic because it violated the original foo5 condition. and that means that without the CHECK OPTION constraint you can insert anything into the base table no matter what the WHERE condition in the view is... again, this is the way informix implements it... ahhh, the great members of the SQL COMITTEE... -- regards, Jaime Casanova Programming today is a race between software engineers striving to build bigger and better idiot-proof programs and the universe trying to produce bigger and better idiots. So far, the universe is winning. Richard Cook ---(end of broadcast)--- TIP 1: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] [PATCHES] Updatable views
--On Freitag, September 01, 2006 11:34:49 -0400 Alvaro Herrera [EMAIL PROTECTED] wrote: I don't understand this part very well. Say if you have a view WITH CHECK OPTION whose condition is foo 5, and then define a view WITH LOCAL CHECK OPTION on top of that, whose condition is bar 5. Does the local check option on the second view that I can insert a row with foo=4, bar=6? That doesn't violate the condition of bar 5, so it seems fine to me. But it also seems quite idiotic because it violated the original foo5 condition. That's exactly what i'm reading out there, too. If such a view definition is useful or not depends on its use case. Correct me if i'm wrong -- Thanks Bernd ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [HACKERS] Getting a move on for 8.2 beta
My feeling is that we ought to bounce bitmap indexes and updatable views as not being ready, accept all the contrib stuff, and try to get the other items done in time for a beta at, say, the end of next week. I'm not thrilled about postponing those two large items till 8.3, but we are a month past feature freeze and we do not have credible patches in hand for either. I know there is a lot of backend work that has been done for 8.2 but the two features you are suggesting we bounce, are probably the most visible of marketable features we will have for this release. Especially updateable views. The alternative is to be willing to slip the schedule until whenever they are ready, and we have learned the hard way that that's not good project management. Well I don't like the idea either, but if it isn't done, it isn't done. How often do we hear the complaint about software that is shipped before it is done? Outside of feature freeze, we really haven't had AFAICS a solid release schedule. Is it going to be push Beta for a month? Does that push us till the end of the year? I don't know which is the better solution, but you asked for comments :) Sincerely, Joshua D. Drake Comments? regards, tom lane ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq -- === The PostgreSQL Company: Command Prompt, Inc. === Sales/Support: +1.503.667.4564 || 24x7/Emergency: +1.800.492.2240 Providing the most comprehensive PostgreSQL solutions since 1997 http://www.commandprompt.com/ ---(end of broadcast)--- TIP 9: In versions below 8.0, the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [HACKERS] [PATCHES] Updatable views
--On Freitag, September 01, 2006 11:41:16 -0400 Tom Lane [EMAIL PROTECTED] wrote: So in other words, views on serial columns don't work? I don't think that's going to be acceptable. They work in such a case that someone isn't allowed to put a volatile function in an update query Not really worse than what the rewriter is doing already --- in fact, I think it's isomorphic to what would happen to the rule qual expressions in your existing patch. Currently you don't have to rewrite the rule conditions itself every time you apply them to the query tree since they are stored in pg_rewrite matching all various (reversed) varattno's, resno's and whatever. If i understand correctly you need to do that with constraints every time you fire an update query on a view for each underlying relation I'm about to propose that we should try to go beta next week (see forthcoming message). If you can strip down your patch to avoid the multi-eval problems in the next couple of days, I'm still willing to Depends on how many days couple of days are.if you mean the next three days then definitely not, since i'm afk for the whole upcoming weekend. Bad timing, but it's not deferrable :( consider it, but at the moment I'm assuming that it needs to be held for 8.3. Well, i'll see what i can do next weekit's a little bit disappointing that these problems raises so late, but that's no one's fault since there are many side effects of the rewriting system involved Many thanks for your comments. Bernd ---(end of broadcast)--- TIP 1: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] Getting a move on for 8.2 beta
Tom Lane wrote: September is upon us and it doesn't seem like we are ready to ship a beta. I think it's time to start making some hard choices. In the main code I see these outstanding features/patches: * bitmap indexes * updatable views IMHO these two should be bounced. They are big features that still seem to need a lot of work and will probably take long before they are ready for inclusion. Interested parties should continue working on them with an eye to be included *at the start* of the 8.3 development cycle. * GUC variable reload + refactoring (previously applied and reverted) * plpython improvements (needs review by someone who knows plpython) * plpgsql improvements for returning record types * patch to build on VC * make libpq default client_encoding setting from locale? In contrib we've got: * new ISBN/etc module * hstore (finally proposed for inclusion) * new sslinfo module * pgstattuple changes * removing the deadwood These all seem to be manageable within reasonable time. I can help with the contrib stuff. My feeling is that we ought to bounce bitmap indexes and updatable views as not being ready, accept all the contrib stuff, and try to get the other items done in time for a beta at, say, the end of next week. +1. -- Alvaro Herrerahttp://www.CommandPrompt.com/ The PostgreSQL Company - Command Prompt, Inc. ---(end of broadcast)--- TIP 6: explain analyze is your friend
Re: [HACKERS] Getting a move on for 8.2 beta
Tom, September is upon us and it doesn't seem like we are ready to ship a beta. I think it's time to start making some hard choices. In the main code I see these outstanding features/patches: * bitmap indexes * updatable views * GUC variable reload + refactoring (previously applied and reverted) * plpython improvements (needs review by someone who knows plpython) * plpgsql improvements for returning record types * patch to build on VC What's VC? * make libpq default client_encoding setting from locale? In contrib we've got: * new ISBN/etc module * hstore (finally proposed for inclusion) * new sslinfo module * pgstattuple changes * removing the deadwood This is just a delete. I've fixed the pgFoundry permissions issues and will be loading the last CVS snapshot today. At that point, Bruce can delete stuff. Can you be a little clearer on which things are non-working patches, and which things are pending due to lack of review? I'm all for bouncing stuff that's not ready to go, but bouncing stuff because we haven't recruited enough code reviewers is just bad practice. -- --Josh Josh Berkus PostgreSQL @ Sun San Francisco ---(end of broadcast)--- TIP 1: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] Getting a move on for 8.2 beta
* Tom Lane ([EMAIL PROTECTED]) wrote: My feeling is that we ought to bounce bitmap indexes and updatable views as not being ready, accept all the contrib stuff, and try to get the other items done in time for a beta at, say, the end of next week. I agree with bouncing these to get the beta out, though I hate having to wait another year for them. If things change in some way and we have the opportunity to fit one in this time, I'd rather see updatable views than bitmap indexes, personally. Overall, I really think 8.2 is going to be an excellent release. I wish autovacuum could have been enabled by default and I'd just like to ask, now and I'll try to remember again once 8.2 is out, please let's turn it on by default for 8.3 (and early on so we get some good testing of it). Thanks, Stephen signature.asc Description: Digital signature
Re: [HACKERS] Getting a move on for 8.2 beta
Josh Berkus josh@agliodbs.com writes: Can you be a little clearer on which things are non-working patches, and which things are pending due to lack of review? The only things I currently want to reject as not having provided a timely patch are bitmap indexes and updatable views. The other stuff I mentioned is in need of review, but I don't currently have a reason to think it can't get in. I'm all for bouncing stuff that's not ready to go, but bouncing stuff because we haven't recruited enough code reviewers is just bad practice. Well, it's taken us the full month to get this far through the queue, so I'd sure like to have more people on board reviewing next time. Alvaro helped but I miss Neil Conway, and some other people have helped in the past. However --- the present situation has nothing to do with lack of reviewers, it's lack of time to finish the patches. regards, tom lane ---(end of broadcast)--- TIP 2: Don't 'kill -9' the postmaster
Re: [HACKERS] Getting a move on for 8.2 beta
Joshua D. Drake [EMAIL PROTECTED] writes: My feeling is that we ought to bounce bitmap indexes and updatable views as not being ready, accept all the contrib stuff, and try to get the other items done in time for a beta at, say, the end of next week. I know there is a lot of backend work that has been done for 8.2 but the two features you are suggesting we bounce, are probably the most visible of marketable features we will have for this release. Especially updateable views. Josh said nearly the same in core's discussion, but come now. The updateable-view patch (in its current form) does not offer *one single thing* you can't do today, indeed for many PG releases past; it just saves writing out some tedious rules. If that's the most marketable feature in 8.2 we're already in trouble. I don't think it is anyway. Looking at the CVS logs I see INSERT/UPDATE/DELETE RETURNING VALUES lists CREATE INDEX CONCURRENTLY pl/pgsql debugger (OK, not in core, but available) restartable WAL recovery constraint exclusion works on UPDATE/DELETE queries multiple-argument aggregate functions SQL2003-standard statistical aggregates COPY (SELECT ...) TO ... pg_dump selectivity options customizable timezone names GIN indexes FILLFACTOR for indexes and tables and that's just back to the beginning of July, so it probably represents about a quarter of the work that's gone into 8.2, but I got tired of reading the logs at that point. We have this discussion every single release cycle: hey, if we wait X amount more time then we can have cool features Y and Z. But there's always another X, Y, and Z. More importantly, this view ignores the very real disadvantages of not getting already-finished cool features A through W out to our users sooner. regards, tom lane ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [HACKERS] Getting a move on for 8.2 beta
Josh said nearly the same in core's discussion, but come now. The updateable-view patch (in its current form) does not offer *one single thing* you can't do today, indeed for many PG releases past; it just saves writing out some tedious rules. Sure, I can write the rules, you can write the rules but we can do rollup queries using unions too, but we still don't have rollup :) Perception is everything and we don't have Updateable views until we don't have to write those rules (yes, its stupid)... If that's the most marketable feature in 8.2 we're already in trouble. I don't think it is anyway. Looking at the CVS logs I see INSERT/UPDATE/DELETE RETURNING VALUES lists CREATE INDEX CONCURRENTLY constraint exclusion works on UPDATE/DELETE queries multiple-argument aggregate functions SQL2003-standard statistical aggregates COPY (SELECT ...) TO ... pg_dump selectivity options customizable timezone names GIN indexes FILLFACTOR for indexes and tables Let's take multiple-argument aggregate functions for example... The marketable portion of that is limited. 85% of the people we are marketing too don't even know what that means. Heck, I haven't even run into the pervious limitation :) Versus SQL2003-standard statistical aggregates. That wins hands down just because of the SQL2003-standard part of the string :) COPY (SELECT...) TO, is only marketable to current PostgreSQL users. Those who are not users don't know the limitation and it is likely not something they would look for when evaluating PostgreSQL. For my current customers the things that are the most marketable (note current customers) is probably: * CREATE INDEX CONCURRENTLY (this is a huge in production boon) * GIN indexes (after explanation) * constraint exclusion works on UPDATE/DELETE queries (We are using alot of tp now, so this is great) * Bitmap Indexes It does not mean all those features are useful, they definitely are. I am just trying to look at it from at: WHIZ* BANG* POW* perspective. What makes the analyst (and I know this isn't the communities problem) say, Holy crap batman, we need to be talking to more people about this database. I think those would be Updateable views, Bitmap Indexes and the SQL2003-standard statistical aggregates. Sincerely, Joshua D. Drake and that's just back to the beginning of July, so it probably represents about a quarter of the work that's gone into 8.2, but I got tired of reading the logs at that point. We have this discussion every single release cycle: hey, if we wait X amount more time then we can have cool features Y and Z. But there's always another X, Y, and Z. More importantly, this view ignores the very real disadvantages of not getting already-finished cool features A through W out to our users sooner. regards, tom lane -- === The PostgreSQL Company: Command Prompt, Inc. === Sales/Support: +1.503.667.4564 || 24x7/Emergency: +1.800.492.2240 Providing the most comprehensive PostgreSQL solutions since 1997 http://www.commandprompt.com/ ---(end of broadcast)--- TIP 1: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] Getting a move on for 8.2 beta
Josh Berkus wrote: Tom, September is upon us and it doesn't seem like we are ready to ship a beta. I think it's time to start making some hard choices. In the main code I see these outstanding features/patches: * bitmap indexes * updatable views * GUC variable reload + refactoring (previously applied and reverted) * plpython improvements (needs review by someone who knows plpython) * plpgsql improvements for returning record types * patch to build on VC What's VC? MS Visual C++ -- Bruce Momjian [EMAIL PROTECTED] EnterpriseDBhttp://www.enterprisedb.com + If your life is a hard drive, Christ can be your backup. + ---(end of broadcast)--- TIP 2: Don't 'kill -9' the postmaster
Re: [HACKERS] Getting a move on for 8.2 beta
Tom Lane wrote: I'm all for bouncing stuff that's not ready to go, but bouncing stuff because we haven't recruited enough code reviewers is just bad practice. Well, it's taken us the full month to get this far through the queue, so I'd sure like to have more people on board reviewing next time. Alvaro helped but I miss Neil Conway, and some other people have helped in the past. However --- the present situation has nothing to do with lack of reviewers, it's lack of time to finish the patches. Also, for next time it would be good to have the reviewers got through the patches *before* feature freeze, so that developers get early feedback. (Which in turn means more time to finish the patches). -- Alvaro Herrerahttp://www.CommandPrompt.com/ PostgreSQL Replication, Consulting, Custom Development, 24x7 support ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [HACKERS] Getting a move on for 8.2 beta
Bruce, What's VC? MS Visual C++ Given that this could lead to us recruiting more developers out of our Windows user base, I'd prioritize it. Overall, I think this whole process is pointing up that there's a problem with our historic Feature Freeze+1 month|beta|RCs cycle. Given the number of developers and companies involved and the increasing size of the code, we need to develop a new schedule that allows us to get draft patches (or at least full specifications) eariler than the month before feature freeze. In short,the issue is that Feature Freeze isn't just Feature Freeze, it's final patch submission. If you look at the two incomplete patches, and the misfired one (Bitmaps, Updatable Views, and WITH RECURSIVE) all of them are patches where the submitter had been working on them months ago, and might have made the release (or let us know they weren't on schedule) if we'd held them to an earlier deadline. As I recall, it was the same situation last year: getting large patches dropped on us out of the blue the week before feature freeze, which then didnt' make it in. Therefore I propose that we adopt the following schedule for 8.3, assuming an September 1 Beta date: June 1: Specification Freeze: specifications for all new features due July 1: Feature Freeze: Draft patches and any minor tweaks due August 1: Final (completed, mostly debugged) patches due September 1: First Beta -- --Josh Josh Berkus PostgreSQL @ Sun San Francisco ---(end of broadcast)--- TIP 4: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] Getting a move on for 8.2 beta
Joshua D. Drake wrote: It does not mean all those features are useful, they definitely are. I am just trying to look at it from at: WHIZ* BANG* POW* perspective. Holy crap, Batman! This database can do INSERT INTO foo VALUES (1,1, 'so long'), (42, 2, 'and thanks'), (142857, 3, 'for all the fish') now! We should be talking to more people about that! That will make some MySQL users happy at least a bit :-) -- Alvaro Herrerahttp://www.CommandPrompt.com/ The PostgreSQL Company - Command Prompt, Inc. ---(end of broadcast)--- TIP 6: explain analyze is your friend
Re: [HACKERS] Getting a move on for 8.2 beta
Tom Lane wrote: Josh Berkus josh@agliodbs.com writes: Can you be a little clearer on which things are non-working patches, and which things are pending due to lack of review? The only things I currently want to reject as not having provided a timely patch are bitmap indexes and updatable views. The other stuff I mentioned is in need of review, but I don't currently have a reason to think it can't get in. I'm all for bouncing stuff that's not ready to go, but bouncing stuff because we haven't recruited enough code reviewers is just bad practice. Well, it's taken us the full month to get this far through the queue, so I'd sure like to have more people on board reviewing next time. Alvaro helped but I miss Neil Conway, and some other people have helped in the past. However --- the present situation has nothing to do with lack of reviewers, it's lack of time to finish the patches. I did try to get us additional help in reviewing. Neil was unavailable, and Alvaro could only give part of his time. -- Bruce Momjian [EMAIL PROTECTED] EnterpriseDBhttp://www.enterprisedb.com + If your life is a hard drive, Christ can be your backup. + ---(end of broadcast)--- TIP 5: don't forget to increase your free space map settings
Re: [HACKERS] Getting a move on for 8.2 beta
Josh Berkus wrote: Bruce, What's VC? MS Visual C++ Given that this could lead to us recruiting more developers out of our Windows user base, I'd prioritize it. Overall, I think this whole process is pointing up that there's a problem with our historic Feature Freeze+1 month|beta|RCs cycle. Given the number of developers and companies involved and the increasing size of the code, we need to develop a new schedule that allows us to get draft patches (or at least full specifications) eariler than the month before feature freeze. In short,the issue is that Feature Freeze isn't just Feature Freeze, it's final patch submission. If you look at the two incomplete patches, and the misfired one (Bitmaps, Updatable Views, and WITH RECURSIVE) all of them are patches where the submitter had been working on them months ago, and might have made the release (or let us know they weren't on schedule) if we'd held them to an earlier deadline. As I recall, it was the same situation last year: getting large patches dropped on us out of the blue the week before feature freeze, which then didnt' make it in. Therefore I propose that we adopt the following schedule for 8.3, assuming an September 1 Beta date: June 1: Specification Freeze: specifications for all new features due July 1: Feature Freeze: Draft patches and any minor tweaks due August 1: Final (completed, mostly debugged) patches due September 1: First Beta No rejiggering is going to get people to complete things they didn't complete under the old system. The plan you list above is what we did for this release. -- Bruce Momjian [EMAIL PROTECTED] EnterpriseDBhttp://www.enterprisedb.com + If your life is a hard drive, Christ can be your backup. + ---(end of broadcast)--- TIP 5: don't forget to increase your free space map settings
Re: [HACKERS] Getting a move on for 8.2 beta
Tom Lane wrote: Well, it's taken us the full month to get this far through the queue, so I'd sure like to have more people on board reviewing next time. Alvaro helped but I miss Neil Conway, and some other people have helped in the past. However --- the present situation has nothing to do with lack of reviewers, it's lack of time to finish the patches. Not that I've had particularly plenty of time to give, but what's concerning me is that while we're moving toward an increasingly rigid process, we don't really have the process support to go with it. How would anyone even know what patches are pending review? Something to think about ... -- Peter Eisentraut http://developer.postgresql.org/~petere/ ---(end of broadcast)--- TIP 1: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] Getting a move on for 8.2 beta
Joshua D. Drake wrote: Let's take multiple-argument aggregate functions for example... The marketable portion of that is limited. 85% of the people we are marketing too don't even know what that means. Heck, I haven't even run into the pervious limitation :) As far as market analysis goes, I'd guess that 100% of the people who already use PostgreSQL by definition don't need any new features and would rather see incremental improvements across the board, which is what this or any release would give them. -- Peter Eisentraut http://developer.postgresql.org/~petere/ ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [HACKERS] Getting a move on for 8.2 beta
Bruce Momjian [EMAIL PROTECTED] writes: Tom Lane wrote: Well, it's taken us the full month to get this far through the queue, so I'd sure like to have more people on board reviewing next time. Alvaro helped but I miss Neil Conway, and some other people have helped in the past. However --- the present situation has nothing to do with lack of reviewers, it's lack of time to finish the patches. I did try to get us additional help in reviewing. Neil was unavailable, and Alvaro could only give part of his time. It strikes me that setting feature freeze in midsummer might not be the best strategy for having manpower available to review --- people tend to be on vacation in August. Maybe the answer is just to move the dates a bit one way or the other. regards, tom lane ---(end of broadcast)--- TIP 9: In versions below 8.0, the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [HACKERS] Getting a move on for 8.2 beta
Josh Berkus josh@agliodbs.com writes: Bruce, What's VC? MS Visual C++ Given that this could lead to us recruiting more developers out of our Windows user base, I'd prioritize it. Yeah, that's why I listed it as a major item. I haven't had a chance to look at the patch, but if it's not too ugly I would like to get it in this time rather than next --- that could lead directly to having more people available next time. If you look at the two incomplete patches, and the misfired one (Bitmaps, Updatable Views, and WITH RECURSIVE) all of them are patches where the submitter had been working on them months ago, and might have made the release (or let us know they weren't on schedule) if we'd held them to an earlier deadline. Perhaps, but I'm not sure what we can or should do about it. Moving deadlines up will either create a dead zone where we all sit around twiddling our thumbs, or people will keep on coding till the last minute anyway. I think having a few patches that don't make the deadline isn't a bad thing: it means we didn't have people sitting idle. It's not like the work will go to waste --- those things can still get in in the next devel cycle. regards, tom lane ---(end of broadcast)--- TIP 2: Don't 'kill -9' the postmaster
Re: [HACKERS] Getting a move on for 8.2 beta
Therefore I propose that we adopt the following schedule for 8.3, assuming an September 1 Beta date: June 1: Specification Freeze: specifications for all new features due I would suggest the above be: June 1: Specification Freeze: specifications for all new features due approved. The last thing we want it to spend all of june arguing about how something should be implemented and thus no real coding gets done. July 1: Feature Freeze: Draft patches and any minor tweaks due August 1: Final (completed, mostly debugged) patches due September 1: First Beta other then that w00t! Joshua D. Drake -- === The PostgreSQL Company: Command Prompt, Inc. === Sales/Support: +1.503.667.4564 || 24x7/Emergency: +1.800.492.2240 Providing the most comprehensive PostgreSQL solutions since 1997 http://www.commandprompt.com/ ---(end of broadcast)--- TIP 9: In versions below 8.0, the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [HACKERS] Getting a move on for 8.2 beta
Alvaro Herrera wrote: Joshua D. Drake wrote: It does not mean all those features are useful, they definitely are. I am just trying to look at it from at: WHIZ* BANG* POW* perspective. Holy crap, Batman! This database can do INSERT INTO foo VALUES (1,1, 'so long'), (42, 2, 'and thanks'), (142857, 3, 'for all the fish') now! We should be talking to more people about that! That will make some MySQL users happy at least a bit :-) To be fair, it will make some of our customers happy to.. they can now get rid of the weird union hack they had to do instead :) Joshua D. Drake -- === The PostgreSQL Company: Command Prompt, Inc. === Sales/Support: +1.503.667.4564 || 24x7/Emergency: +1.800.492.2240 Providing the most comprehensive PostgreSQL solutions since 1997 http://www.commandprompt.com/ ---(end of broadcast)--- TIP 1: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] Getting a move on for 8.2 beta
On Fri, 1 Sep 2006, Tom Lane wrote: My feeling is that we ought to bounce bitmap indexes and updatable views as not being ready, accept all the contrib stuff, and try to get the other items done in time for a beta at, say, the end of next week. For what it's worth, Jie and I hope to have finished the bitmap streaming this weekend which is the only outstanding issue with the code as it currently stands. I now realise (nothing like hindsight) that we should have posted a patch before we looked at streaming bitmaps because they've proved more fiddly to implement than I thought :-(. At that point, we'd more or less addressed the issues I'd raised when I posted at feature freeze time. Thanks, Gavin ---(end of broadcast)--- TIP 4: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] Getting a move on for 8.2 beta
Bruce, Tom, all: No rejiggering is going to get people to complete things they didn't complete under the old system. It'll help the new people. A lot of people -- if not most -- submitting their first major patch to PostgreSQL dramatically underestimate the amount of fix-up that's going to be required, and assume that there won't be a spec discussion, which there often is. By getting them to submit a little at a time, *earlier*, we can avoid doing those things at the last minute. Alternately, we can just make sure that first-time patchers have mentors who check progress well before feature freeze. The plan you list above is what we did for this release. No, it's not. There's a bunch of patches which we had nothing on -- not spec, not design draft, not anything -- until we got them on July 20th. Our current system is to have only one deadline, at which point you're expected to have 85% of the patch done and up to PostgreSQL standards. That's quite a bit of jumping in with both feet for a newbie. I did try to get us additional help in reviewing. Neil was unavailable, and Alvaro could only give part of his time Asking two people is not exactly an all-out effort to get reviewers. It strikes me that setting feature freeze in midsummer might not be the best strategy for having manpower available to review --- people tend to be on vacation in August. Maybe the answer is just to move the dates a bit one way or the other. We've discussed that issue before, yes. Since we're proposing a new roadmap process for 8.3, and will likely be dealing with a lot of major patches, maybe that's the release to delay? However, as PR maven I do want to point out that doing the final release in December would be a bad idea. Hard to get news coverage. Also, we'd have the same issue with people being gone. -- --Josh Josh Berkus PostgreSQL @ Sun San Francisco ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [HACKERS] Getting a move on for 8.2 beta
Gavin Sherry [EMAIL PROTECTED] writes: On Fri, 1 Sep 2006, Tom Lane wrote: My feeling is that we ought to bounce bitmap indexes and updatable views as not being ready, accept all the contrib stuff, and try to get the other items done in time for a beta at, say, the end of next week. For what it's worth, Jie and I hope to have finished the bitmap streaming this weekend which is the only outstanding issue with the code as it currently stands. AFAIR, you told me it'll be done this weekend last week. And the week before. In any case, it's going to be a large complicated patch, and if I spend all next week reviewing it then I won't make any progress on the other items on my to-do list. At some point we have to say this has slipped too far, we have to hold it over for 8.3. regards, tom lane ---(end of broadcast)--- TIP 4: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] Getting a move on for 8.2 beta
Tom Lane [EMAIL PROTECTED] writes: If you look at the two incomplete patches, and the misfired one (Bitmaps, Updatable Views, and WITH RECURSIVE) all of them are patches where the submitter had been working on them months ago, and might have made the release (or let us know they weren't on schedule) if we'd held them to an earlier deadline. Perhaps, but I'm not sure what we can or should do about it. Moving deadlines up will either create a dead zone where we all sit around twiddling our thumbs, or people will keep on coding till the last minute anyway. I don't think that's what would happen at all. In fact what I think would happen is that it would free up people like you to work on stuff you're interested in instead of reviewing patches. As an extreme example consider the new Linux release cycle. They have a non-freeze period of a couple weeks, followed by months of frozen time. Users who want to try out new features on different hardware can do so and often turn up problems developers reviewing the patches missed. Developers spend the months of frozen time working on new patches which, if they're ready on time, all go into the queue in the first few weeks of a new release. If they miss the window they'll make the next one. As a result they don't get things like what we saw with 8.0 where major new features like PITR, nested transactions and so on all hit the tree in the final days before the freeze. Instead they go in early and if they turn out to be immature they get pulled long before the actual release. I would love to see the bitmap indexes and updatable views get merged into the tree just weeks after the release comes out rather than disappear again until just before the next release. If I were a user depending on these new features it would give me a hell of a lot more confidence if people could say they've been in the CVS version for months and nobody's had problems with them than to be told they were reviewed by one person three weeks ago and nobody else has had much of a chance to use it. -- Gregory Stark EnterpriseDB http://www.enterprisedb.com ---(end of broadcast)--- TIP 2: Don't 'kill -9' the postmaster
Re: [HACKERS] Getting a move on for 8.2 beta
Josh Berkus josh@agliodbs.com writes: No, it's not. There's a bunch of patches which we had nothing on -- not spec, not design draft, not anything -- until we got them on July 20th. Our current system is to have only one deadline, Well, no, it's not. We have told people till we're blue in the face post early, post often. Now I will plead guilty to not always having spent as much time giving feedback on draft patches as I should've, but the process is pretty clear. As I see it the main problem is people undertaking patches off in corners somewhere rather than discussing their work on the mailing lists while they do it. If you want more process in this we could institute rules like feature patches will be rejected out of hand if there wasn't previously a spec proposal posted to pghackers. I don't think this is really a good idea, but I'm not sure what else to do. regards, tom lane ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [HACKERS] Getting a move on for 8.2 beta
Tom Lane wrote: Well, no, it's not. We have told people till we're blue in the face post early, post often. Now I will plead guilty to not always having spent as much time giving feedback on draft patches as I should've, but the process is pretty clear. As I see it the main problem is people undertaking patches off in corners somewhere rather than discussing their work on the mailing lists while they do it. Again, process support. If all we can offer people is to post multi-megabyte patches to the mailing list every month, that totally doesn't help. We'd need ways to track the progress on these things: what was the specification for that patch, where was the discussion on it, what has changed in the patch since the last time, since the time before last time, what is left to be done, who has worked on it, etc. Figuring out the answer to those questions from a mailing list archive is tedious to the point that no one wants to do it. -- Peter Eisentraut http://developer.postgresql.org/~petere/ ---(end of broadcast)--- TIP 2: Don't 'kill -9' the postmaster
Re: [HACKERS] Getting a move on for 8.2 beta
Peter Eisentraut wrote: Tom Lane wrote: Well, it's taken us the full month to get this far through the queue, so I'd sure like to have more people on board reviewing next time. Alvaro helped but I miss Neil Conway, and some other people have helped in the past. However --- the present situation has nothing to do with lack of reviewers, it's lack of time to finish the patches. Not that I've had particularly plenty of time to give, but what's concerning me is that while we're moving toward an increasingly rigid process, we don't really have the process support to go with it. How would anyone even know what patches are pending review? Something to think about ... They are in the patches queue? http://momjian.postgresql.org/cgi-bin/pgpatches -- Bruce Momjian [EMAIL PROTECTED] EnterpriseDBhttp://www.enterprisedb.com + If your life is a hard drive, Christ can be your backup. + ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [HACKERS] Getting a move on for 8.2 beta
Josh Berkus wrote: Bruce, Tom, all: No rejiggering is going to get people to complete things they didn't complete under the old system. It'll help the new people. A lot of people -- if not most -- submitting their first major patch to PostgreSQL dramatically underestimate the amount of fix-up that's going to be required, and assume that there won't be a spec discussion, which there often is. By getting them to submit a little at a time, *earlier*, we can avoid doing those things at the last minute. That is in the developer's FAQ. Alternately, we can just make sure that first-time patchers have mentors who check progress well before feature freeze. The plan you list above is what we did for this release. No, it's not. There's a bunch of patches which we had nothing on -- not spec, not design draft, not anything -- until we got them on July 20th. Our current system is to have only one deadline, at which point you're expected to have 85% of the patch done and up to PostgreSQL standards. That's quite a bit of jumping in with both feet for a newbie. Right. The developer's FAQ says they should follow a process. Making another process doesn't mean they will follow that either. I did try to get us additional help in reviewing. Neil was unavailable, and Alvaro could only give part of his time Asking two people is not exactly an all-out effort to get reviewers. Well, not sure what else I can do. Those are the people who used to help out a lot. It strikes me that setting feature freeze in midsummer might not be the best strategy for having manpower available to review --- people tend to be on vacation in August. Maybe the answer is just to move the dates a bit one way or the other. We've discussed that issue before, yes. Since we're proposing a new roadmap process for 8.3, and will likely be dealing with a lot of major patches, maybe that's the release to delay? Moving it away from summer might help, yea. -- Bruce Momjian [EMAIL PROTECTED] EnterpriseDBhttp://www.enterprisedb.com + If your life is a hard drive, Christ can be your backup. + ---(end of broadcast)--- TIP 6: explain analyze is your friend
Re: [HACKERS] Getting a move on for 8.2 beta
Tom Lane wrote: Josh Berkus josh@agliodbs.com writes: No, it's not. There's a bunch of patches which we had nothing on -- not spec, not design draft, not anything -- until we got them on July 20th. Our current system is to have only one deadline, Well, no, it's not. We have told people till we're blue in the face post early, post often. Now I will plead guilty to not always having spent as much time giving feedback on draft patches as I should've, but the process is pretty clear. As I see it the main problem is people undertaking patches off in corners somewhere rather than discussing their work on the mailing lists while they do it. If you want more process in this we could institute rules like feature patches will be rejected out of hand if there wasn't previously a spec proposal posted to pghackers. I don't think this is really a good idea, but I'm not sure what else to do. Well stated. -- Bruce Momjian [EMAIL PROTECTED] EnterpriseDBhttp://www.enterprisedb.com + If your life is a hard drive, Christ can be your backup. + ---(end of broadcast)--- TIP 9: In versions below 8.0, the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [HACKERS] Getting a move on for 8.2 beta
Bruce Momjian wrote: Peter Eisentraut wrote: Tom Lane wrote: Well, it's taken us the full month to get this far through the queue, so I'd sure like to have more people on board reviewing next time. Alvaro helped but I miss Neil Conway, and some other people have helped in the past. However --- the present situation has nothing to do with lack of reviewers, it's lack of time to finish the patches. Not that I've had particularly plenty of time to give, but what's concerning me is that while we're moving toward an increasingly rigid process, we don't really have the process support to go with it. How would anyone even know what patches are pending review? Something to think about ... They are in the patches queue? http://momjian.postgresql.org/cgi-bin/pgpatches Side note:: Perhaps this would be better served from the main website under the developer and community sections. Joshua D. Drake -- === The PostgreSQL Company: Command Prompt, Inc. === Sales/Support: +1.503.667.4564 || 24x7/Emergency: +1.800.492.2240 Providing the most comprehensive PostgreSQL solutions since 1997 http://www.commandprompt.com/ ---(end of broadcast)--- TIP 1: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] Getting a move on for 8.2 beta
Peter Eisentraut wrote: Tom Lane wrote: Well, no, it's not. We have told people till we're blue in the face post early, post often. Now I will plead guilty to not always having spent as much time giving feedback on draft patches as I should've, but the process is pretty clear. As I see it the main problem is people undertaking patches off in corners somewhere rather than discussing their work on the mailing lists while they do it. Again, process support. If all we can offer people is to post multi-megabyte patches to the mailing list every month, that totally doesn't help. We'd need ways to track the progress on these things: what was the specification for that patch, where was the discussion on it, what has changed in the patch since the last time, since the time before last time, what is left to be done, who has worked on it, etc. Figuring out the answer to those questions from a mailing list archive is tedious to the point that no one wants to do it. Uh, Tom has been tracking Gavin on the bitmap patch every week for weeks, and I pummelled EnterpriseDB/Jonah over the recursive query patch. Neither effort was very fruitful, but tracking wasn't what made them fail. I am not saying tracking is wrong, but rather tracking would not have helped make these things happen faster. -- Bruce Momjian [EMAIL PROTECTED] EnterpriseDBhttp://www.enterprisedb.com + If your life is a hard drive, Christ can be your backup. + ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [HACKERS] Getting a move on for 8.2 beta
Joshua D. Drake wrote: Bruce Momjian wrote: Peter Eisentraut wrote: Tom Lane wrote: Well, it's taken us the full month to get this far through the queue, so I'd sure like to have more people on board reviewing next time. Alvaro helped but I miss Neil Conway, and some other people have helped in the past. However --- the present situation has nothing to do with lack of reviewers, it's lack of time to finish the patches. Not that I've had particularly plenty of time to give, but what's concerning me is that while we're moving toward an increasingly rigid process, we don't really have the process support to go with it. How would anyone even know what patches are pending review? Something to think about ... They are in the patches queue? http://momjian.postgresql.org/cgi-bin/pgpatches Side note:: Perhaps this would be better served from the main website under the developer and community sections. It pulls my a mailbox file I use, and it does instant updates as soon as I change it. It is a URL. Why do people care where it is? -- Bruce Momjian [EMAIL PROTECTED] EnterpriseDBhttp://www.enterprisedb.com + If your life is a hard drive, Christ can be your backup. + ---(end of broadcast)--- TIP 5: don't forget to increase your free space map settings
Re: [HACKERS] Getting a move on for 8.2 beta
They are in the patches queue? http://momjian.postgresql.org/cgi-bin/pgpatches Side note:: Perhaps this would be better served from the main website under the developer and community sections. It pulls my a mailbox file I use, and it does instant updates as soon as I change it. It is a URL. Why do people care where it is? I would think that it would be easier to find :) Joshua D. Drake -- === The PostgreSQL Company: Command Prompt, Inc. === Sales/Support: +1.503.667.4564 || 24x7/Emergency: +1.800.492.2240 Providing the most comprehensive PostgreSQL solutions since 1997 http://www.commandprompt.com/ ---(end of broadcast)--- TIP 5: don't forget to increase your free space map settings
Re: [HACKERS] Getting a move on for 8.2 beta
Bruce Momjian wrote: Uh, Tom has been tracking Gavin on the bitmap patch every week for weeks, and I pummelled EnterpriseDB/Jonah over the recursive query patch. Neither effort was very fruitful, but tracking wasn't what made them fail. I am not saying tracking is wrong, but rather tracking would not have helped make these things happen faster. It would make the process more transparent, which is something several people have expressed a desire for. cheers andrew ---(end of broadcast)--- TIP 5: don't forget to increase your free space map settings
Re: [HACKERS] Getting a move on for 8.2 beta
Gregory Stark [EMAIL PROTECTED] writes: As an extreme example consider the new Linux release cycle. They have a non-freeze period of a couple weeks, followed by months of frozen time. Users who want to try out new features on different hardware can do so and often turn up problems developers reviewing the patches missed. Developers spend the months of frozen time working on new patches which, if they're ready on time, all go into the queue in the first few weeks of a new release. If they miss the window they'll make the next one. That doesn't seem particularly appealing for our purposes. What it sounds like is that all the development happens somewhere outside the main tree, and every so often everybody tries to land a bunch of large patches at once. That's going to be a mess if the patches interact at all, and it certainly isn't going to address my primary concern of encouraging a publicly-visible development process instead of having people hiding in a corner until they can drop a large patch on us. I would love to see the bitmap indexes and updatable views get merged into the tree just weeks after the release comes out rather than disappear again until just before the next release. Here I agree - partly. With some continuing effort these patches could be in fine shape to apply by the time we branch CVS for 8.3 development. However, our traditional approach to beta period is that it's supposed to be a quiet time where people focus on testing, debugging, and documentation. Shepherding people who are doing major development during that period is likely to be a serious distraction from making the release ready and high-quality. regards, tom lane ---(end of broadcast)--- TIP 5: don't forget to increase your free space map settings
Re: [HACKERS] Getting a move on for 8.2 beta
Bruce Momjian wrote: It pulls my a mailbox file I use, and it does instant updates as soon as I change it. It is a URL. Why do people care where it is? The complaint has been that not enough people help with reviewing patches. That would indicate that the problem is not location but scalability. If everything has to go through your private mailbox, then it's not a very obvious process to outsiders. -- Peter Eisentraut http://developer.postgresql.org/~petere/ ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [HACKERS] Getting a move on for 8.2 beta
Joshua D. Drake wrote: They are in the patches queue? http://momjian.postgresql.org/cgi-bin/pgpatches Side note:: Perhaps this would be better served from the main website under the developer and community sections. It pulls my a mailbox file I use, and it does instant updates as soon as I change it. It is a URL. Why do people care where it is? I would think that it would be easier to find :) We have a link from the developer's page: http://www.postgresql.org/developer/roadmap -- Bruce Momjian [EMAIL PROTECTED] EnterpriseDBhttp://www.enterprisedb.com + If your life is a hard drive, Christ can be your backup. + ---(end of broadcast)--- TIP 4: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] Getting a move on for 8.2 beta
Peter Eisentraut wrote: Bruce Momjian wrote: It pulls my a mailbox file I use, and it does instant updates as soon as I change it. It is a URL. Why do people care where it is? The complaint has been that not enough people help with reviewing patches. That would indicate that the problem is not location but scalability. If everything has to go through your private mailbox, then it's not a very obvious process to outsiders. Well, you can grab items from there and apply them. I will remove them from the mailbox when I see the commit. You have another system that will not be significantly more work? Any you can apply them right from the email lists so they never get to the queue. -- Bruce Momjian [EMAIL PROTECTED] EnterpriseDBhttp://www.enterprisedb.com + If your life is a hard drive, Christ can be your backup. + ---(end of broadcast)--- TIP 5: don't forget to increase your free space map settings
Re: [HACKERS] Getting a move on for 8.2 beta
Bruce Momjian wrote: Peter Eisentraut wrote: Bruce Momjian wrote: It pulls my a mailbox file I use, and it does instant updates as soon as I change it. It is a URL. Why do people care where it is? The complaint has been that not enough people help with reviewing patches. That would indicate that the problem is not location but scalability. If everything has to go through your private mailbox, then it's not a very obvious process to outsiders. Well, you can grab items from there and apply them. I will remove them from the mailbox when I see the commit. You have another system that will not be significantly more work? Any you can apply them right from the email lists so they never get to the queue. Another novel idea: have a bugtracker of sorts where selected people can add feature requests and append patches to them. There, people could add fake feature requests describing the stuff they are working on, and publish the patches they have for them. -- Alvaro Herrerahttp://www.CommandPrompt.com/ PostgreSQL Replication, Consulting, Custom Development, 24x7 support ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [HACKERS] Getting a move on for 8.2 beta
Bruce Momjian wrote: Uh, Tom has been tracking Gavin on the bitmap patch every week for weeks, and I pummelled EnterpriseDB/Jonah over the recursive query patch. Great, but where is this documented, so others know about this? Neither effort was very fruitful, but tracking wasn't what made them fail. I am not saying tracking is wrong, but rather tracking would not have helped make these things happen faster. The fallacy here is assuming that all these things should be single-person tasks. As long as we only have one coder and one manager, we don't need much process support, but then we're pretty nearly at the point we're now, where two or three people review patches while the rest just sits around and wonders what this feature freeze thing is supposed to be about. I can tell you plenty of stories about the updatable views patch. One month after feature freeze, we notice that we didn't even have an accepted design specification. I'm sure it was posted sometime, but how do we find it now? People complain unjustly that the patch was posted at the last minute, but in fact updated patches and information have been posted regularly for more than one year. But it's impossible to tie these things together unless you are mailing list crawling software with artificial intelligence capabilities. And during the last two weeks, no make that six months, Bernd has spent half his time analyzing and reverting breakage that well-meaning reviewers had injected into his patch, with the other half possibly spent keeping the patch up to date with the moving development tree. There is, of course, no silver bullet. But more successful involvement of people who are not in the inner circle needs more support in many ways. -- Peter Eisentraut http://developer.postgresql.org/~petere/ ---(end of broadcast)--- TIP 9: In versions below 8.0, the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [HACKERS] Getting a move on for 8.2 beta
Bruce Momjian wrote: Well, you can grab items from there and apply them. I will remove them from the mailbox when I see the commit. You have another system that will not be significantly more work? Any you can apply them right from the email lists so they never get to the queue. What the current system lacks is documentation, which makes the process behind those lists known to others, and the ability to interact with the system. If I can't see any feedback in the system for the actions I do, then the system is not very useful to me. -- Peter Eisentraut http://developer.postgresql.org/~petere/ ---(end of broadcast)--- TIP 4: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] Getting a move on for 8.2 beta
Alvaro Herrera wrote: I've been bitten by having stuff rejected because there was no immediate use for them; so I had to maintain them in a private tree, where they were subsequently broken by someone else's parallel development. A better version control system would surely seem to help here and in other instances. -- Peter Eisentraut http://developer.postgresql.org/~petere/ ---(end of broadcast)--- TIP 1: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] Getting a move on for 8.2 beta
Alvaro Herrera wrote: Another novel idea: have a bugtracker of sorts where selected people can add feature requests and append patches to them. There, people could add fake feature requests describing the stuff they are working on, and publish the patches they have for them. Certainly, much of this discussion is trying to yell tracking software. Track patches, track comments, track open items, and so on. This doesn't have to be the same as a bug tracker, though. -- Peter Eisentraut http://developer.postgresql.org/~petere/ ---(end of broadcast)--- TIP 6: explain analyze is your friend
Re: [HACKERS] Getting a move on for 8.2 beta
Peter Eisentraut wrote: Bruce Momjian wrote: Uh, Tom has been tracking Gavin on the bitmap patch every week for weeks, and I pummelled EnterpriseDB/Jonah over the recursive query patch. Great, but where is this documented, so others know about this? I see no value in documenting it. Neither effort was very fruitful, but tracking wasn't what made them fail. I am not saying tracking is wrong, but rather tracking would not have helped make these things happen faster. The fallacy here is assuming that all these things should be single-person tasks. As long as we only have one coder and one manager, we don't need much process support, but then we're pretty nearly at the point we're now, where two or three people review patches while the rest just sits around and wonders what this feature freeze thing is supposed to be about. I can tell you plenty of stories about the updatable views patch. One month after feature freeze, we notice that we didn't even have an accepted design specification. I'm sure it was posted sometime, but how do we find it now? People complain unjustly that the patch was posted at the last minute, but in fact updated patches and information have been posted regularly for more than one year. But it's impossible to tie these things together unless you are mailing list crawling software with artificial intelligence capabilities. And during the last two weeks, no make that six months, Bernd has spent half his time analyzing and reverting breakage that well-meaning reviewers had injected into his patch, with the other half possibly spent keeping the patch up to date with the moving development tree. There is, of course, no silver bullet. But more successful involvement of people who are not in the inner circle needs more support in many ways. I do things only if others do not. If committers applied patches as they came in, the patch queue would be empty, and if others tracked open issues, I wouldn't have to. -- Bruce Momjian [EMAIL PROTECTED] EnterpriseDBhttp://www.enterprisedb.com + If your life is a hard drive, Christ can be your backup. + ---(end of broadcast)--- TIP 9: In versions below 8.0, the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [HACKERS] Getting a move on for 8.2 beta
Peter Eisentraut wrote: Bruce Momjian wrote: Well, you can grab items from there and apply them. I will remove them from the mailbox when I see the commit. You have another system that will not be significantly more work? Any you can apply them right from the email lists so they never get to the queue. What the current system lacks is documentation, which makes the process behind those lists known to others, and the ability to interact with the system. If I can't see any feedback in the system for the actions I do, then the system is not very useful to me. What do you want to happen? Just apply the patch? You want to log into a system to say you applied it? If people are not doing applications when they have no overhead, I doubt they will do it when there are additional steps. -- Bruce Momjian [EMAIL PROTECTED] EnterpriseDBhttp://www.enterprisedb.com + If your life is a hard drive, Christ can be your backup. + ---(end of broadcast)--- TIP 6: explain analyze is your friend
Re: [HACKERS] Getting a move on for 8.2 beta
Peter Eisentraut wrote: Alvaro Herrera wrote: Another novel idea: have a bugtracker of sorts where selected people can add feature requests and append patches to them. There, people could add fake feature requests describing the stuff they are working on, and publish the patches they have for them. Certainly, much of this discussion is trying to yell tracking software. Track patches, track comments, track open items, and so on. This doesn't have to be the same as a bug tracker, though. OK, why do we use email then? Move it all into a tracker? If not, who is going to move info from email to the tracking system? -- Bruce Momjian [EMAIL PROTECTED] EnterpriseDBhttp://www.enterprisedb.com + If your life is a hard drive, Christ can be your backup. + ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [HACKERS] Getting a move on for 8.2 beta
Bruce Momjian wrote: I see no value in documenting it. The value in documenting it is that the general awareness about the progress of development is raised, so more people know where and how to get involved, as opposed to only three people knowing and the rest finding out one month after feature freeze that the patch isn't ready. I do things only if others do not. If committers applied patches as they came in, the patch queue would be empty, and if others tracked open issues, I wouldn't have to. Fair enough. Perhaps we should find others with ideas about better tracking and the willingness to implement them. -- Peter Eisentraut http://developer.postgresql.org/~petere/ ---(end of broadcast)--- TIP 9: In versions below 8.0, the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
[HACKERS] GetSnapshotData never needs exclusive lock on ProcArrayLock
I was thinking about the business of capturing subtransaction XIDs during GetSnapshotData so as to reduce the need for visits to pg_subtrans, as implemented by Takahiro-san here: http://archives.postgresql.org/pgsql-patches/2006-08/msg00401.php The principal objection to this, I think, is the longer time spent holding the ProcArrayLock while grabbing a snapshot. That wouldn't be so bad if the lock were shared, but we hold it exclusive while taking the first serializable snap of a transaction. It strikes me though that this is excessive paranoia: it should be safe to take the lock shared even while setting a serializable snap. I believe the reasoning for using an exclusive lock was simply we are changing a shared data structure (by setting MyProc-xmin), therefore it oughta be exclusive. However: * No backend ever writes another's MyProc-xmin, so there is no danger of write conflicts. * We already assume that fetching/storing of TransactionId is atomic, so there shouldn't be any problem with reading a partially updated value. Onlooker backends should always read either zero or the correct xmin, and they are already coded to do the right thing with zero --- it just means this xact hasn't set its xmin yet. The only other assumption that I can see here is the one documented in GetOldestXmin: if another backend has not yet set its xmin, then when it does, it cannot set xmin less than the one we compute. This is trivially true if backends are forced to set their serializable snapshots one at a time, but if we allow two backends to execute GetSnapshotData at the same time, ISTM that they must in fact compute the same value of xmin, so there's no problem. Two backends concurrently running GetSnapshotData must already have stored their own XIDs into their PGPROC structures, so the xmins they compute will certainly be no larger than the older of these XIDs. The only way they could compute different xmins is if some even-older XID leaves the set of running transactions meanwhile --- but this is impossible because we require backends to take ProcArrayLock exclusively when removing an XID from the proc array. The two will therefore compute identical xmins, whether or not either one is slow enough to see the other's computed value appear in the proc array. I propose therefore that we make GetSnapshotData use a shared lock always for ProcArrayLock. Anyone see a hole in this analysis? regards, tom lane ---(end of broadcast)--- TIP 4: Have you searched our list archives? http://archives.postgresql.org
