== PostgreSQL Weekly News - November 23 2014 == PostgreSQL 9.4RC1 released. Test! https://wiki.postgresql.org/wiki/What%27s_new_in_PostgreSQL_9.4
The CfP for FOSDEM PGDay and Devroom 2015 is open. http://fosdem2015.pgconf.eu/callforpapers/ PostgreSQL Conference 2014 Tokyo will be held December 5 in Shinagawa. http://www.postgresql.jp/events/jpug-pgcon2014 pgDaySF 2015 will be held March 10, 2015 in Burlingame, California. http://sfpostgres.org/pgday-sf-2015-call-for-speakers-and-sponsors/ == PostgreSQL Product News == Ora2PG 14.1, a migration system for moving from Oracle to PostgreSQL, released. http://ora2pg.darold.net/ == PostgreSQL Jobs for November == http://archives.postgresql.org/pgsql-jobs/2014-11/threads.php == PostgreSQL Local == PG-Cuba will be November 25 and 26, 2014 http://postgresql.uci.cu/?p=1087#more-1087 The CfP for FOSDEM PGDay (January 30, 2015) and the FOSDEM PostgreSQL Dev Room (January 31-February 1, 2015) are open until November 24th, 2014. http://fosdem2015.pgconf.eu/callforpapers/ The CfP is open for Nordic PostgreSQL Day 2015, which will be held March 11, 2015 in Copenhagen, Denmark. http://2015.nordicpgday.org/cfp/ The CfP for PGConf US 2015 is open through December 17th, 2014 Notifications will go out on January 10, 2014. The event takes place March 25-27, 2015 in NYC. http://nyc.pgconf.us/2015/ == PostgreSQL in the News == Planet PostgreSQL: http://planet.postgresql.org/ PostgreSQL Weekly News is brought to you this week by David Fetter Submit news and announcements by Sunday at 3:00pm Pacific time. Please send English language ones to da...@fetter.org, German language to p...@pgug.de, Italian language to p...@itpug.org. Spanish language to p...@arpug.com.ar. == Applied Patches == Álvaro Herrera pushed: - Fix relpersistence setting in reindex_index. Buildfarm members with CLOBBER_CACHE_ALWAYS advised us that commit 85b506bbfc2937 was mistaken in setting the relpersistence value of the index directly in the relcache entry, within reindex_index. The reason for the failure is that an invalidation message that comes after mucking with the relcache entry directly, but before writing it to the catalogs, would cause the entry to become rebuilt in place from catalogs with the old contents, losing the update. Fix by passing the correct persistence value to RelationSetNewRelfilenode instead; this routine also writes the updated tuple to pg_class, avoiding the problem. Suggested by Tom Lane. http://git.postgresql.org/pg/commitdiff/0f9692b40d1292f1b2640f026561908fd37b7407 Heikki Linnakangas pushed: - Fix WAL-logging of B-tree "unlink halfdead page" operation. There was some confusion on how to record the case that the operation unlinks the last non-leaf page in the branch being deleted. _bt_unlink_halfdead_page set the "topdead" field in the WAL record to the leaf page, but the redo routine assumed that it would be an invalid block number in that case. This commit fixes _bt_unlink_halfdead_page to do what the redo routine expected. This code is new in 9.4, so backpatch there. http://git.postgresql.org/pg/commitdiff/c73669c0e0168923e3f9e787beec980f55af2bd8 - Remove obsolete debugging option, RTDEBUG. The r-tree AM that used it was removed back in 2005. Peter Geoghegan http://git.postgresql.org/pg/commitdiff/dedae6c211415290cdf5faca4ef874f803af204c - Add test cases for indexam operations not currently covered. That includes VACUUM on GIN, GiST and SP-GiST indexes, and B-tree indexes large enough to cause page deletions in B-tree. Plus some other special cases. After this patch, the regression tests generate all different WAL record types. Not all branches within the redo functions are covered, but it's a step forward. http://git.postgresql.org/pg/commitdiff/88fc71926392115cdc3672807f3903ce43d0ebcf - Silence compiler warning about variable being used uninitialized. It's a false positive - the variable is only used when 'onleft' is true, and it is initialized in that case. But the compiler doesn't necessarily see that. http://git.postgresql.org/pg/commitdiff/f464042161fb980578c33fedc2742c5a4d941fa2 - Revamp the WAL record format. Each WAL record now carries information about the modified relation and block(s) in a standardized format. That makes it easier to write tools that need that information, like pg_rewind, prefetching the blocks to speed up recovery, etc. There's a whole new API for building WAL records, replacing the XLogRecData chains used previously. The new API consists of XLogRegister* functions, which are called for each buffer and chunk of data that is added to the record. The new API also gives more control over when a full-page image is written, by passing flags to the XLogRegisterBuffer function. This also simplifies the XLogReadBufferForRedo() calls. The function can dig the relation and block number from the WAL record, so they no longer need to be passed as arguments. For the convenience of redo routines, XLogReader now disects each WAL record after reading it, copying the main data part and the per-block data into MAXALIGNed buffers. The data chunks are not aligned within the WAL record, but the redo routines can assume that the pointers returned by XLogRecGet* functions are. Redo routines are now passed the XLogReaderState, which contains the record in the already-disected format, instead of the plain XLogRecord. The new record format also makes the fixed size XLogRecord header smaller, by removing the xl_len field. The length of the "main data" portion is now stored at the end of the WAL record, and there's a separate header after XLogRecord for it. The alignment padding at the end of XLogRecord is also removed. This compansates for the fact that the new format would otherwise be more bulky than the old format. Reviewed by Andres Freund, Amit Kapila, Michael Paquier, Alvaro Herrera, Fujii Masao. http://git.postgresql.org/pg/commitdiff/2c03216d831160bedd72d45f712601b6f7d03f1c - Add pageinspect functions for inspecting GIN indexes. Patch by me, Peter Geoghegan and Michael Paquier, reviewed by Amit Kapila. http://git.postgresql.org/pg/commitdiff/3a82bc6f8ab38be3ed095f1c86866900b145f0f6 - Fix bogus comments in XLogRecordAssemble. Pointed out by Michael Paquier http://git.postgresql.org/pg/commitdiff/8f5dcb56cb1877f7ed6b0f3f72cc26c4e060aa61 - Add a comment to regress.c explaining what it contains. Ian Barwick http://git.postgresql.org/pg/commitdiff/b10a97b819a2594c97188bac02d87b550908a692 - No need to call XLogEnsureRecordSpace when the relation is unlogged. Amit Kapila http://git.postgresql.org/pg/commitdiff/622983ea6964fc55f1ad3063d3d7b8f5de8e8816 Tom Lane pushed: - Update time zone data files to tzdata release 2014j. DST law changes in the Turks & Caicos Islands (America/Grand_Turk) and in Fiji. New zone Pacific/Bougainville for portions of Papua New Guinea. Historical changes for Korea and Vietnam. http://git.postgresql.org/pg/commitdiff/bc241488b0f9bde3e14094bfaf0d7987fafb6600 - Update 9.4 release notes for commits through today. http://git.postgresql.org/pg/commitdiff/832054044f68080eaebccd771e21fdd56824db20 - Fix some bogus direct uses of realloc(). pg_dump/parallel.c was using realloc() directly with no error check. While the odds of an actual failure here seem pretty low, Coverity complains about it, so fix by using pg_realloc() instead. While looking for other instances, I noticed a couple of places in psql that hadn't gotten the memo about the availability of pg_realloc. These aren't bugs, since they did have error checks, but verbosely inconsistent code is not a good thing. Back-patch as far as 9.3. 9.2 did not have pg_dump/parallel.c, nor did it have pg_realloc available in all frontend code. http://git.postgresql.org/pg/commitdiff/8b13e5c6c0e8a6b797370fb91d207031df5e784a - Update comments in find_typedef. These comments don't seem to have been touched in a long time. Make them describe the current implementation rather than what was here last century, and be a bit more explicit about the unreferenced-typedefs issue. http://git.postgresql.org/pg/commitdiff/7aa8d9e56c18b1df9d924f144c06d921131a137e - Don't require bleeding-edge timezone data in timestamptz regression test. The regression test cases added in commits b2cbced9e et al depended in part on the Russian timezone offset changes of Oct 2014. While this is of no particular concern for a default Postgres build, it was possible for a build using --with-system-tzdata to fail the tests if the system tzdata database wasn't au courant. Bjorn Munch and Christoph Berg both complained about this while packaging 9.4rc1, so we probably shouldn't insist on the system tzdata being up-to-date. Instead, make an equivalent test using a zone change that occurred in Venezuela in 2007. With this patch, the regression tests should pass using any tzdata set from 2012 or later. (I can't muster much sympathy for somebody using --with-system-tzdata on a machine whose system tzdata is more than three years out-of-date.) http://git.postgresql.org/pg/commitdiff/8d7af8fbe7349cbebb576459c9b5f54dfcc6216a - Improve documentation's description of JOIN clauses. In bug #12000, Andreas Kunert complained that the documentation was misleading in saying "FROM T1 CROSS JOIN T2 is equivalent to FROM T1, T2". That's correct as far as it goes, but the equivalence doesn't hold when you consider three or more tables, since JOIN binds more tightly than comma. I added a <note> to explain this, and ended up rearranging some of the existing text so that the note would make sense in context. In passing, rewrite the description of JOIN USING, which was unnecessarily vague, and hadn't been helped any by somebody's reliance on markup as a substitute for clear writing. (Mostly this involved reintroducing a concrete example that was unaccountably removed by commit 032f3b7e166cfa28.) Back-patch to all supported branches. http://git.postgresql.org/pg/commitdiff/8372304e3594a1d67737df779f098d9ae9937603 - Add missing case for CustomScan. Per KaiGai Kohei. In passing improve formatting of some code added in commit 30d7ae3c, because otherwise pgindent will make a mess of it. http://git.postgresql.org/pg/commitdiff/f9e0255c6fbedda50aae6aa7479f0c5a8f90b831 - Remove no-longer-needed phony typedefs in genbki.h. Now that we have a policy of hiding varlena catalog fields behind "#ifdef CATALOG_VARLEN", there is no need for their type names to be acceptable to the C compiler. And experimentation shows that it does not matter to pgindent either. (If it did, we'd have problems anyway, since these typedefs are unreferenced so far as the C compiler is concerned, and find_typedef fails to identify such typedefs.) Hence, remove the phony typedefs that genbki.h provided to make some varlena field definitions compilable. In passing, rearrange #define's into what seemed a more logical order. http://git.postgresql.org/pg/commitdiff/c5111ea9ca049a650b63cdcb2551c330c2f998b1 - Fix another oversight in CustomScan patch. disuse_physical_tlist() must work for all plan types handled by create_scan_plan(). http://git.postgresql.org/pg/commitdiff/03e574af5f5d4682ce3ae6dc401ba126c70ce2ea - Fix another oversight in CustomScan patch. execCurrent.c's search_plan_tree() must recognize a CustomScan on the target relation. This would only be helpful for custom providers that support CurrentOfExpr quals, which is probably a bit far-fetched, but it's not impossible I think. But even without assuming that, we need to recognize a scanned-relation match so that we will properly throw error if the desired relation is being scanned with both a CustomScan and a regular scan (ie, self-join). Also recognize ForeignScanState for similar reasons. Supporting WHERE CURRENT OF on a foreign table is probably even more far-fetched than it is for custom scans, but I think in principle you could do it with postgres_fdw (or another FDW that supports the ctid column). This would be a back-patchable bug fix if existing FDWs handled CurrentOfExpr, but I doubt any do so I won't bother back-patching. http://git.postgresql.org/pg/commitdiff/081a6048cff07a83591ebcb08b676a771ae58d2b - Initial code review for CustomScan patch. Get rid of the pernicious entanglement between planner and executor headers introduced by commit 0b03e5951bf0a1a8868db13f02049cf686a82165. Also, rearrange the CustomFoo struct/typedef definitions so that all the typedef names are seen as used by the compiler. Without this pgindent will mess things up a bit, which is not so important perhaps, but it also removes a bizarre discrepancy between the declaration arrangement used for CustomExecMethods and that used for CustomScanMethods and CustomPathMethods. Clean up the commentary around ExecSupportsMarkRestore to reflect the rather large change in its API. Const-ify register_custom_path_provider's argument. This necessitates casting away const in the function, but that seems better than forcing callers of the function to do so (or else not const-ify their method pointer structs, which was sort of the whole point). De-export fix_expr_common. I don't like the exporting of fix_scan_expr or replace_nestloop_params either, but this one surely has got little excuse. http://git.postgresql.org/pg/commitdiff/a34fa8ee7cc757671632dc4dcae4f21e8f2e2357 - Remove dead code supporting mark/restore in SeqScan, TidScan, ValuesScan. There seems no prospect that any of this will ever be useful, and indeed it's questionable whether some of it would work if it ever got called; it's certainly not been exercised in a very long time, if ever. So let's get rid of it, and make the comments about mark/restore in execAmi.c less wishy-washy. The mark/restore support for Result nodes is also currently dead code, but that's due to planner limitations not because it's impossible that it could be useful. So I left it in. http://git.postgresql.org/pg/commitdiff/adbfab119b308a7e0e6b1305de9be222cfd5c85b - Simplify API for initially hooking custom-path providers into the planner. Instead of register_custom_path_provider and a CreateCustomScanPath callback, let's just provide a standard function hook in set_rel_pathlist. This is more flexible than what was previously committed, is more like the usual conventions for planner hooks, and requires less support code in the core. We had discussed this design (including centralizing the set_cheapest() calls) back in March or so, so I'm not sure why it wasn't done like this already. http://git.postgresql.org/pg/commitdiff/c2ea2285e978d9289084846a3343cef7d261d880 - Rearrange CustomScan API. Make it work more like FDW plans do: instead of assuming that there are expressions in a CustomScan plan node that the core code doesn't know about, insist that all subexpressions that need planner attention be in a "custom_exprs" list in the Plan representation. (Of course, the custom plugin can break the list apart again at executor initialization.) This lets us revert the parts of the patch that exposed setrefs.c and subselect.c processing to the outside world. Also revert the GetSpecialCustomVar stuff in ruleutils.c; that concept may work in future, but it's far from fully baked right now. http://git.postgresql.org/pg/commitdiff/447770404cce5ce86174fa4809784c4e5d0a0a76 - Fix mishandling of system columns in FDW queries. postgres_fdw would send query conditions involving system columns to the remote server, even though it makes no effort to ensure that system columns other than CTID match what the remote side thinks. tableoid, in particular, probably won't match and might have some use in queries. Hence, prevent sending conditions that include non-CTID system columns. Also, create_foreignscan_plan neglected to check local restriction conditions while determining whether to set fsSystemCol for a foreign scan plan node. This again would bollix the results for queries that test a foreign table's tableoid. Back-patch the first fix to 9.3 where postgres_fdw was introduced. Back-patch the second to 9.2. The code is probably broken in 9.1 as well, but the patch doesn't apply cleanly there; given the weak state of support for FDWs in 9.1, it doesn't seem worth fixing. Etsuro Fujita, reviewed by Ashutosh Bapat, and somewhat modified by me http://git.postgresql.org/pg/commitdiff/9c58101117d25f174c8d8013befdc33c632922d0 - Get rid of redundant production in plpgsql grammar. There may once have been a reason for the intermediate proc_stmts production in the plpgsql grammar, but it isn't doing anything useful anymore, so let's collapse it into proc_sect. Saves some code and probably a small number of nanoseconds per statement list. In passing, correctly alphabetize keyword lists to match pl_scanner.c; note that for "rowtype" vs "row_count", pl_scanner.c must sort on the basis of the lower-case spelling. Noted while fooling with a patch to de-reserve more plpgsql keywords. http://git.postgresql.org/pg/commitdiff/e2dc3f57726199408902ba87eaaf7b95d1223744 Fujii Masao pushed: - Add --synchronous option to pg_receivexlog, for more reliable WAL writing. Previously pg_receivexlog flushed WAL data only when WAL file was switched. Then 3dad73e added -F option to pg_receivexlog so that users could control how frequently sync commands were issued to WAL files. It also allowed users to make pg_receivexlog flush WAL data immediately after writing by specifying 0 in -F option. However feedback messages were not sent back immediately even after a flush location was updated. So even if WAL data was flushed in real time, the server could not see that for a while. This commit removes -F option from and adds --synchronous to pg_receivexlog. If --synchronous is specified, like the standby's wal receiver, pg_receivexlog flushes WAL data as soon as there is WAL data which has not been flushed yet. Then it sends back the feedback message identifying the latest flush location to the server. This option is useful to make pg_receivexlog behave as sync standby by using replication slot, for example. Original patch by Furuya Osamu, heavily rewritten by me. Reviewed by Heikki Linnakangas, Alvaro Herrera and Sawada Masahiko. http://git.postgresql.org/pg/commitdiff/c4f99d20294950576d552dcaf9ce5b9bdc4233a3 - Fix pg_receivexlog --slot so that it doesn't prevent the server shutdown. When pg_receivexlog --slot is connecting to the server, at the shutdown of the server, walsender keeps waiting for the last WAL record to be replicated and flushed in pg_receivexlog. But previously pg_receivexlog issued sync command only when WAL file was switched. So there was the case where the last WAL was never flushed and walsender had to keep waiting infinitely. This caused the server shutdown to get stuck. pg_recvlogical handles this problem by calling fsync() when it receives the request of immediate reply from the server. That is, at shutdown, walsender sends the request, pg_recvlogical receives it, flushes the last WAL record, and sends the flush location back to the server. Since walsender can see that the last WAL record is successfully flushed, it can exit cleanly. This commit introduces the same logic as pg_recvlogical has, to pg_receivexlog. Back-patch to 9.4 where pg_receivexlog was changed so that it can use the replication slot. Original patch by Michael Paquier, rewritten by me. Bug report by Furuya Osamu. http://git.postgresql.org/pg/commitdiff/f66c20b317578838a39a1de8014c4363bdc98b9a - Fix bug in the test of file descriptor of current WAL file in pg_receivexlog. In pg_receivexlog, in order to check whether the current WAL file is being opened or not, its file descriptor has to be checked against -1 as an invalid value. But, oops, 7900e94 added the incorrect test checking the descriptor against 1. This commit fixes that bug. Back-patch to 9.4 where the bug was added. Spotted by Magnus Hagander http://git.postgresql.org/pg/commitdiff/d5f4df7264ec73abda41eb78b81cf430b33f7606 Simon Riggs pushed: - Add pg_dump --snapshot option. Allows pg_dump to use a snapshot previously defined by a concurrent session that has either used pg_export_snapshot() or obtained a snapshot when creating a logical slot. When this option is used with parallel pg_dump, the snapshot defined by this option is used and no new snapshot is taken. Simon Riggs and Michael Paquier http://git.postgresql.org/pg/commitdiff/be1cc8f46f57a04e69d9e4dd268d34da885fe6eb - Reduce btree scan overhead for < and > strategies. For <, <=, > and >= strategies, mark the first scan key as already matched if scanning in an appropriate direction. If index tuple contains no nulls we can skip the first Author: Rajeev Rastogi. Reviewer: Haribabu Kommi. Rework of the code and comments by Simon Riggs http://git.postgresql.org/pg/commitdiff/606c0123d627b37d5ac3f7c2c97cd715dde7842f Robert Haas pushed: - Avoid file descriptor leak in pg_test_fsync. This can cause problems on Windows, where files that are still open can't be unlinked. Jeff Janes http://git.postgresql.org/pg/commitdiff/a0165553619107f990cb7872f4a9573598319581 Peter Eisentraut pushed: - Fix suggested layout for PGXS makefile. Custom rules must come after pgxs inclusion, not before, because any rule added before pgxs will break the default 'all' target. Author: Cédric Villemain <ced...@2ndquadrant.fr> http://git.postgresql.org/pg/commitdiff/8dc626defec23016dd5988208d8704b858b9d21d Andrew Dunstan pushed: - Make psql's \sf and \ef honor ECHO_HIDDEN. These commands were calling the database direct rather than calling PSQLexec like other slash commands that needed database data. The code is also changed not to pass the connection as a parameter to the helper functions. It's available in a global variable, and that's what PSQLexec uses. http://git.postgresql.org/pg/commitdiff/e4d28175a13d47a747b33e87c0f808b2ef57c0cd - Fix an error in psql that overcounted output lines. This error counted the first line of a cell as "extra". The effect was to cause far too frequent invocation of the pager. In most cases this can be worked around (for example, by using the "less" pager with the -F flag), so don't backpatch. http://git.postgresql.org/pg/commitdiff/4077fb4d1d34ad04dfb95ba676c2b43ea1f1da53 - Rework echo_hidden for \sf and \ef from commit e4d2817. PSQLexec's error reporting turns out to be too verbose for this case, so revert to using PQexec instead with minimal error reporting. Prior to calling PQexec, we call a function that mimics just the echo_hidden piece of PSQLexec. http://git.postgresql.org/pg/commitdiff/eca2b9ba3e1e65c2ac880578bfb3cbeed1cb1d26 - Fix memory leaks introduced by commit eca2b9b http://git.postgresql.org/pg/commitdiff/02d5ab6a86aa941109cecd9e6ebe73a67e8bf545 Noah Misch pushed: - Detect PG_PRINTF_ATTRIBUTE automatically. This eliminates gobs of "unrecognized format function type" warnings under MinGW compilers predating GCC 4.4. http://git.postgresql.org/pg/commitdiff/b779168ffe330d137b8b4329a7710c8bf1f7e661 == Rejected Patches (for now) == No one was disappointed this week :-) == Pending Patches == Etsuro Fujita sent in another revision of a patch to allow foreign tables to participate in table inheritance hierarchies. Jeff Davis and Tomas Vondra traded patches to do better memory accounting, in part as infrastructure for a memory-bounded HashAgg. Michael Paquier sent in a patch to fix CLOBBER_CACHE_ALWAYS. Michael Paquier sent in another revision of a patch to track number of files marked as ready for archiving in pg_stat_archiver. SAWADA Masahiko sent in another revision of a patch to implement REINDEX SCHEMA, in this case as an option to reindexdb. Heikki Linnakangas sent in two PoC patches to deal with a performance problem in PageRepairFragmentation. Petr (PJMODOS) Jelinek and Simon Riggs traded patches to add a shutdown_at_recovery_target option to recovery.conf. Dimitri Fontaine sent in three more revisions of a patch to add "table rewrite" as an event which can fire an event trigger. Adam Brightwell sent in a patch to add int64-to-int8 mapping in Catalog.pm. Magnus Hagander sent in another revision of a patch to add a pg_stat_ssl view. Aaron W. Swenson sent in a patch to replace configure.in with configure dot ac. Heikki Linnakangas sent in a patch to add some more tests for WALs. Peter Geoghegan sent in two revisions of a patch to create an amcheck utility, a generalization of his previous btreecheck. Michael Paquier sent in a patch to uppercase some SQL keywords in the BRIN documents. Amit Kapila sent in another revision of a patch to fix some infelicities between pg_basebackup and Windows. Peter Eisentraut sent in a series of patches which: Fix SHLIB_PREREQS use in contrib, allowing PGXS builds, allow the user to set VPATH directly, and clean up some makefile changes that weren't needed. Abhijit Menon-Sen sent in a patch to use slice-by-8 for CRC, which increased the performance of that operation by a factor of 4+. Adam Brightwell sent in another revision of a patch to fix Catalog.pm for role attributes. Laurenz Albe sent in a patch to disallow changing functions which are part of an index definition. Michael Paquier sent in a patch to enable GIN code managing entry insertion not to distinguish fresh from old indexes. Robert Haas sent in another revision of a patch to add infrastructure to save and restore GUC values. Michael Paquier sent in a patch to add fillfactor as storage parameter for GIN indexes. Petr (PJMODOS) Jelinek sent in two more revisions of a patch to track commit timestamps. Adam Brightwell sent in another revision of a patch to clean up superuser shortcuts. Alexander Shulgin sent in another revision of a patch to turn recovery.conf into GUCs. Álvaro Herrera added some opfamilies for BRIN indexes which should help make them more useful. Marko (johto) Tiikkaja sent in two revisions of a patch to make the HINTs more useful for certain types common mistakes in UPDATE. Amit Kapila sent in another revision of a patch to parallelize vacuumdb. David Rowley sent in another revision of a patch to support SEMI and ANTI join removal in certain cases. Tom Lane sent in a patch to remove 17 of PL/pgsql's reserved words. -- Sent via pgsql-announce mailing list (pgsql-announce@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-announce
