[HACKERS] small pg_basebackup display bug
from 9.3devel (this morning): There is a small bug in pg_basebackup: it displays a truncated part of what seems to be the new $PGDATA (the source code has 'filename' there, but I don't see what's supposed to be in there). The truncated name in parentheses only shows up during the filling of the new PGDATA; when 100% is reached that part of the displayed name is deleted. That's another bug in itself, I think: shouldn't the end-state of that line also display the parenthesized name? Below is output that shows what I mean: I got that by entering ENTER e few times while pg_basebackup was running $ pg_basebackup -v -P -D /home/aardvark/pg_stuff/pg_installations/pgsql.93_3pulls2/data 19377/55092 kB (35%), 0/1 tablespace (/home/aardvark/pg_stuff/pg_ins) 45859/55092 kB (83%), 0/1 tablespace (/home/aardvark/pg_stuff/pg_ins) 55101/55101 kB (100%), 1/1 tablespace pg_basebackup: base backup completed Thanks, Erik Rijkers -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Makefiles don't seem to remember to rebuild everything anymore
Le vendredi 14 décembre 2012 23:02:11, Tom Lane a écrit : In a fully-built source tree: $ cd pgsql/src/backend/parser $ make make: Nothing to be done for `all'. ... okay so far ... $ rm gram.o rm: remove regular file `gram.o'? y $ make make: Nothing to be done for `all'. WTF? A previous patch changed the .SECONDARY from an if() section to the main section of src/Makefile.global.in, thus this target is triggered by each 'make' call. === commit 1eb1dde (HEAD, refs/bisect/bad) Author: Peter Eisentraut pete...@gmx.net Date: Wed Oct 31 23:33:35 2012 -0400 Have make never delete intermediate files automatically Several hacks in certain modes already thought this was a bad idea, so just disable it globally. === So 'one' fix for this case of gram.o is to add gram.c to the .SECONDARY, like this in the parser/Makefile # scan is compiled as part of gram +.SECONDARY: gram.c gram.o: scan.c While it fixes the example, I wonder if we have to maintain each intermediate files in Makefiles like that I also am not sure why the .SECONDARY has been moved up in the Makefile.global.in (removing this target completely also fix the issue) Hope it helps you and Peter find the good fix. -- Cédric Villemain +33 (0)6 20 30 22 52 http://2ndQuadrant.fr/ PostgreSQL: Support 24x7 - Développement, Expertise et Formation signature.asc Description: This is a digitally signed message part.
[HACKERS] Set visibility map bit after HOT prune
We discussed this idea in the past [1] and Robert recently again mentioned this in another thread [2]. Please see a rebased/revised patch attached with the mail. This is mostly similar to what I's submitted in [1] except some additions to also compute visibility cut-off XID. I also removed a warning that I'd added to the previous patch to report the case when the page's all-visible bit is already set, but our HOT prune scan finds it otherwise. This is not to hide the warning the earlier reviewer had reported, but I think its not required because we do those consistency checks at other places anyways. I've run several rounds of pgbench (-s 10 -c 10 -T 300) and did not find any issues. I don't see noticeable performance again/drop. But again pgbench may not be the most suitable test benchmark to test this. I think we will see positive differences in vacuum scans and also queries that are benefited from index-only scans. An early setting of the visibility map bit can help those two scenarios. Thanks, Pavan 1. http://archives.postgresql.org/pgsql-hackers/2010-02/msg02344.php 2. http://archives.postgresql.org/message-id/CA+TgmoZHWN1+N_CGD3hx=DJPHkd69c8x2r2EiQ5=c8yxnrc...@mail.gmail.com -- Pavan Deolasee http://www.linkedin.com/in/pavandeolasee hot-prune-set-all-visible-v2.patch Description: Binary data -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] small pg_basebackup display bug
On Sat, Dec 15, 2012 at 11:39 AM, Erik Rijkers e...@xs4all.nl wrote: from 9.3devel (this morning): Is this different from 9.2 and earlier? There is a small bug in pg_basebackup: it displays a truncated part of what seems to be the new $PGDATA (the source code has 'filename' there, but I don't see what's supposed to be in there). It is the full name of the file. If the path is too long it gets truncated before it gets to the actual filename, of course, so it looks to be the directory. The truncated name in parentheses only shows up during the filling of the new PGDATA; when 100% is reached that part of the displayed name is deleted. That's another bug in itself, I think: shouldn't the end-state of that line also display the parenthesized name? Those are both intentional. It may be argued that the intention shoud be changed :), but it's not a bug in itself - it's acting like intended. -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] is allow_nonpic_in_shlib still useful?
On Sat, Dec 15, 2012 at 01:23:38AM -0500, Peter Eisentraut wrote: In the plperl and plpython makefiles we look for a shared library of libperl or libpython, and if it's not found, we check for allow_nonpic_in_shlib, and if that's yes, then we proceed anyway. Apparently, and IIRC, this was set up in a time when those shared libraries were not available through standard builds, but I think that hasn't been the case for quite a while. The only platforms where we set allow_nonpick_in_shlib is linux and freebsd/i386 (presumably an obsolescent combination). Are there any Linux builds that don't supply the required shared libraries? I can't recall such a system. On x86_64, GNU ld would reject the resulting text relocations anyway. I suspend this hack isn't useful anymore and ought to be removed. Agreed. On !allow_nonpic_in_shlib systems, the effect appears to be that we quietly skip the plperl build, despite --with-perl, when the Perl build lacks a shlib. This seems unhelpful; I think --with-perl should result in either an error or a built plperl. Likewise for plpython. The coincidentally-better build behavior under allow_nonpic_in_shlib should become unconditional. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] small pg_basebackup display bug
On Sat, December 15, 2012 14:10, Magnus Hagander wrote: On Sat, Dec 15, 2012 at 11:39 AM, Erik Rijkers e...@xs4all.nl wrote: from 9.3devel (this morning): The truncated name in parentheses only shows up during the filling of the new PGDATA; when 100% is reached that part of the displayed name is deleted. That's another bug in itself, I think: shouldn't the end-state of that line also display the parenthesized name? Those are both intentional. It may be argued that the intention shoud be changed :), but it's not a bug in itself - it's acting like intended. Fair enough. How about just changing the possible length to enable display somewhat longer names like I used? It truncates to 30 characters; perhaps something like 100 would be more reasonable? That would make such a truncation less frequent, and after all a truncated display is not particular useful. Thanks, Erik Rijkers -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Makefiles don't seem to remember to rebuild everything anymore
=?iso-8859-15?q?C=E9dric_Villemain?= ced...@2ndquadrant.com writes: Le vendredi 14 décembre 2012 23:02:11, Tom Lane a écrit : $ rm gram.o rm: remove regular file `gram.o'? y $ make make: Nothing to be done for `all'. WTF? A previous patch changed the .SECONDARY from an if() section to the main section of src/Makefile.global.in, Hm. I should have made clear that this isn't specific to gram.o --- I also tried rm analyze.o and that didn't trigger any action either. So it doesn't seem to be a case of the bison rule chain being specifically at fault. I do suspect that this was triggered by some fairly recent makefile change, though, because I don't recall having seen anything as odd as this until recently. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Set visibility map bit after HOT prune
On 15 December 2012 12:42, Pavan Deolasee pavan.deola...@gmail.com wrote: We discussed this idea in the past [1] and Robert recently again mentioned this in another thread [2]. Please see a rebased/revised patch attached with the mail. This is mostly similar to what I's submitted in [1] except some additions to also compute visibility cut-off XID. I also removed a warning that I'd added to the previous patch to report the case when the page's all-visible bit is already set, but our HOT prune scan finds it otherwise. This is not to hide the warning the earlier reviewer had reported, but I think its not required because we do those consistency checks at other places anyways. Doing that only makes sense when we're running a SELECT. Setting the all visible bit immediately prior to an UPDATE that clears it again is pointless effort, generating extra work for no reason. -- Simon Riggs http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Parser Cruft in gram.y
Kevin Grittner kgri...@mail.com writes: Tom Lane wrote: the parser tables are basically number-of-tokens wide by number-of-states high. (In HEAD there are 433 tokens known to the grammar, all but 30 of which are keywords, and 4367 states.) Splitting the grammar into multiple grammars is unlikely to do much to improve this --- in fact, it could easily make matters worse due to duplication. Of course if they were both at 80% it would be a higher total than combined, but unless you have a handle on the percentages, it doesn't seem like a foregone conclusion. Do you have any feel for what the split would be? I don't really, but I will note that the scalar-expression subgrammar is a pretty sizable part of the whole, and it's difficult to see how you'd make a useful split that didn't duplicate it. I guess you could push CREATE TABLE, ALTER TABLE, CREATE DOMAIN, ALTER DOMAIN, COPY, and anything else that included expression arguments over into the main grammar. But that path leads to more and more stuff getting moved to the main grammar over time, making the whole thing more and more questionable. The whole concept seems ugly and unmaintainable in any case. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] is allow_nonpic_in_shlib still useful?
Peter Eisentraut pete...@gmx.net writes: In the plperl and plpython makefiles we look for a shared library of libperl or libpython, and if it's not found, we check for allow_nonpic_in_shlib, and if that's yes, then we proceed anyway. Apparently, and IIRC, this was set up in a time when those shared libraries were not available through standard builds, but I think that hasn't been the case for quite a while. The only platforms where we set allow_nonpick_in_shlib is linux and freebsd/i386 (presumably an obsolescent combination). Are there any Linux builds that don't supply the required shared libraries? Doubt it. I'm pretty sure that every Linux distro would strongly discourage linking static libraries as large as perl or python into another package anyway, because of the difficulty of applying security updates for said libraries if this has been done. In Red Hat's distros, static libraries aren't shipped *at all* without a damn good package-specific reason --- and neither the perl nor python packages provide such a library AFAICT. FreeBSD might be a different story though. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Makefiles don't seem to remember to rebuild everything anymore
On Sat, Dec 15, 2012 at 9:53 PM, Tom Lane t...@sss.pgh.pa.us wrote: =?iso-8859-15?q?C=E9dric_Villemain?= ced...@2ndquadrant.com writes: Le vendredi 14 décembre 2012 23:02:11, Tom Lane a écrit : $ rm gram.o rm: remove regular file `gram.o'? y $ make make: Nothing to be done for `all'. WTF? A previous patch changed the .SECONDARY from an if() section to the main section of src/Makefile.global.in, Hm. I should have made clear that this isn't specific to gram.o --- I also tried rm analyze.o and that didn't trigger any action either. So it doesn't seem to be a case of the bison rule chain being specifically at fault. I do suspect that this was triggered by some fairly recent makefile change, though, because I don't recall having seen anything as odd as this until recently. I can confirm that this happens on Mac OSX as well. I tried a lot of things in the Makefiles to see why the object file is not automatically built even though objfiles.txt is clearly marked as dependent on the OBJS. What I found though that the change Cédric mentioned above might have indeed introduced the problem. So if I comment out .SECONDARY directive in Makefile.global, .o files are rebuilt properly. I saw following in the documentation When a file is secondary, make will not create the file merely because it does not already exist, but make does not automatically delete the file. (link: ftp://ftp.gnu.org/old-gnu/Manuals/make-3.79.1/html_chapter/make_10.html#SEC97) Not sure if the .o files are treated as secondary files though. Thanks, Pavan -- Pavan Deolasee http://www.linkedin.com/in/pavandeolasee -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Set visibility map bit after HOT prune
Simon Riggs si...@2ndquadrant.com writes: Doing that only makes sense when we're running a SELECT. Setting the all visible bit immediately prior to an UPDATE that clears it again is pointless effort, generating extra work for no reason. On the other hand, the HOT prune operation itself is worthless when we're running a SELECT. The only reason we do it that way is that we have to prune before the query starts to use the page, else pruning might invalidate pointers-to-tuples that are being held within the query plan tree. Maybe it's time to look at what it'd take for the low-level scan operations to know whether they're scanning the target relation of an UPDATE query, so that we could skip pruning altogether except when a HOT update could conceivably ensue. I think this was discussed back when HOT went in, but nobody wanted to make the patch more invasive than it had to be. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] LATERAL and VOLATILE functions
Hello I tested some usage of LATERAL clause, and I found so LATERAL doesn't respects difference between VOLATILE and IMMUTABLE functions. Is this behave expected? -- unexpected postgres=# select * from generate_series(1,3) g(v), LATERAL (SELECT random()) x; ; v │ random ───┼── 1 │ 0.63025646051392 2 │ 0.63025646051392 3 │ 0.63025646051392 (3 rows) -- expected postgres=# select * from generate_series(1,3) g(v), LATERAL (SELECT random() - v + v) x; v │ ?column? ───┼─── 1 │ 0.381548477802426 2 │ 0.762988060247153 3 │ 0.181648664642125 (3 rows) Regards Pavel Stehule -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] LATERAL and VOLATILE functions
Pavel Stehule pavel.steh...@gmail.com writes: Is this behave expected? -- unexpected postgres=# select * from generate_series(1,3) g(v), LATERAL (SELECT random()) x; ; vrandom ---+-- 1 0.63025646051392 2 0.63025646051392 3 0.63025646051392 (3 rows) The LATERAL keyword is a no-op since x doesn't contain any side-reference to g(v). So you get a plain join between g and a single-row relation x. If the SQL standard actually specified what LATERAL means, we could argue about whether that's a correct interpretation or not. I haven't been able to find anyplace where the spec defines the semantics though. And I'm fairly certain that we *don't* want it to mean recompute for every row generated to the left of the keyword, whether there is a variable reference or not. Consider for example select ... from a, b, c join lateral d on ... If the D item only contains references to C, it's unlikely that the programmer wants it to be re-evaluated again for each possible row in A*B. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] LATERAL and VOLATILE functions
2012/12/15 Tom Lane t...@sss.pgh.pa.us: Pavel Stehule pavel.steh...@gmail.com writes: Is this behave expected? -- unexpected postgres=# select * from generate_series(1,3) g(v), LATERAL (SELECT random()) x; ; vrandom ---+-- 1 0.63025646051392 2 0.63025646051392 3 0.63025646051392 (3 rows) The LATERAL keyword is a no-op since x doesn't contain any side-reference to g(v). So you get a plain join between g and a single-row relation x. If the SQL standard actually specified what LATERAL means, we could argue about whether that's a correct interpretation or not. I haven't been able to find anyplace where the spec defines the semantics though. And I'm fairly certain that we *don't* want it to mean recompute for every row generated to the left of the keyword, whether there is a variable reference or not. Consider for example select ... from a, b, c join lateral d on ... If the D item only contains references to C, it's unlikely that the programmer wants it to be re-evaluated again for each possible row in A*B. Stable and immutable functions should be recalculated once time, but for volatile functions is recalculation probably more natural (expected). Every time is strange, when function random() returns same numbers. I am not sure if this behave can be problem in real usage - probably it can be a surprise for someone who use random() for some testing. Regards Pavel regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Assert for frontend programs?
On Fri, 2012-12-14 at 17:03 -0500, Tom Lane wrote: Having the layer is a good thing, eg so that USE_ASSERT_CHECKING can control it, or so that somebody can inject a different behavior if they want. You could also (or at least additionally) map !USE_ASSERT_CHECKING to NDEBUG. This would also help with imported code that calls assert() directly. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Assert for frontend programs?
On Fri, 2012-12-14 at 15:32 -0500, Andrew Dunstan wrote: Here's a patch for that. It appears that your change has caused new compiler warnings: encnames.c:9:1: warning: Assert redefined In file included from encnames.c:8: ../../../src/include/postgres_fe.h:36:1: warning: this is the location of the previous definition wchar.c:10:1: warning: Assert redefined In file included from wchar.c:9: ../../../src/include/postgres_fe.h:36:1: warning: this is the location of the previous definition encnames.c:9:1: warning: Assert redefined In file included from encnames.c:8: ../../../src/include/postgres_fe.h:36:1: warning: this is the location of the previous definition I changed some of the psql assertions so they all have explicit boolean expressions - I think that's better style for use of assert. I think not, but I probably wrote most of that originally. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] multiple CREATE FUNCTION AS items for PLs
I'm going to use PL/Python as an example, but I would also like to know if this could be applicable to other languages. When you do CREATE FUNCTION foo(...) ... LANGUAGE plpythonu AS $$ source code here $$; it internally creates a source file that contains --- def __plpython_procedure_foo_12345(): source code here --- It would be useful to be able to do something like this instead: --- some code here def __plpython_procedure_foo_12345(): some more code here --- This would especially be useful for placing imports into the first part. While you can have them in the function definition, that means they are executed every time the function is called, which makes it much slower. Also, future imports are not possible this way. CREATE FUNCTION already supports multiple AS items. Currently, multiple AS items are rejected for all languages but C. I'd imagine lifting that restriction and leaving it up to the validator to check it. Then any language can accept two AS items if it wants and paste them together in whichever way it needs. (The probin/prosrc naming will then become more obsolete, but it's perhaps not worth changing anything about that.) So in practice this might look like this: CREATE FUNCTION foo(...) ... LANGUAGE plpythonu AS $$ import x import y $$, $$ real code here $$; Comments? -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Add big fat caution to pg_restore docs regards partial db restores
On Sat, 2012-11-17 at 12:02 -0600, Karl O. Pinc wrote: Feel free to reject. I sent in the patch to try out the idea. It's a bit crazy, but I didn't think too crazy to share with the list. I'm going to set this patch as returned with feedback for now. I'm all for adding more help to the documentation, but I think in this case it should be more concrete, in the form of, if you have an incomplete restore, do these steps to verify the integrity of various things. (I suspect, if users actually encounter the situation you were referring to, they are pretty much hosed. Transactions are your friend.) -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Doc patch, further describe and-mask nature of the permission system
On Mon, 2012-12-10 at 20:48 -0600, Karl O. Pinc wrote: On 11/14/2012 02:35:54 PM, Karl O. Pinc wrote: On 11/13/2012 08:50:55 PM, Peter Eisentraut wrote: On Sat, 2012-09-29 at 01:16 -0500, Karl O. Pinc wrote: This patch makes some sweeping statements. Unfortunately, they are wrong. I will see if anything can be salvaged. Here's another try. (I bundled changes to both paragraphs into a single patch.) grants-of-roles-are-additive_v3.patch I don't get the point of this change, especially why you are trying to liken the roles system to the object hierarchy, when they are clearly different and unrelated. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Set visibility map bit after HOT prune
On Sun, Dec 16, 2012 at 3:18 AM, Tom Lane t...@sss.pgh.pa.us wrote: Simon Riggs si...@2ndquadrant.com writes: Doing that only makes sense when we're running a SELECT. Setting the all visible bit immediately prior to an UPDATE that clears it again is pointless effort, generating extra work for no reason. On the other hand, the HOT prune operation itself is worthless when we're running a SELECT. The only reason we do it that way is that we have to prune before the query starts to use the page, else pruning might invalidate pointers-to-tuples that are being held within the query plan tree. Right. HOT prune may and often would be called in the SELECT path or heap/index scan leading to UPDATE/DELETE. But whenever its called, it looks like a good idea to set the visibility map bit. There is some additional overhad to check if a LIVE tuple is all-visible or not, but that doesn't look too much. I did run some pgbench tests for fully cached tables and did not see any difference in tps. Another idea could have been to NOT clear the visibility bit when a HOT update happens. Such tuple can get pruned by HOT prune, so we don't need vacuum per se, and the index-only scans are not affected because the update was a HOT update, so the index keys did not change either. So index-only scans would continue to return the same result. Don't know if this would work with hot standby, probably not. Maybe it's time to look at what it'd take for the low-level scan operations to know whether they're scanning the target relation of an UPDATE query, so that we could skip pruning altogether except when a HOT update could conceivably ensue. Yeah, we could do that. We may not be able to predict whether the coming update is HOT or not, but I don't think that matters. OTOH its probably not too bad to prune in any scan (like we do today) because there is fairly high chance that the page will be dirtied for hint bit updates. So may be its better to just prune as well. We have already put in several optimisations to do so only when required and without any unnecessary contention. Of course, benchmarks can prove me wrong. Thanks, Pavan -- Pavan Deolasee http://www.linkedin.com/in/pavandeolasee -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
