Re: pgsql: Report progress of CREATE INDEX operations
On 2019-Apr-02, Alvaro Herrera wrote: > Report progress of CREATE INDEX operations I just noticed that valgrind is also complaining about IndexVacuumInfo.report_progress being uninitialized in a couple of callsites. Will fix early tomorrow. -- Álvaro Herrerahttps://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Re: pgsql: Add support for multivariate MCV lists
On Fri, Mar 29, 2019 at 08:25:10PM +0100, Tomas Vondra wrote: On Fri, Mar 29, 2019 at 12:06:26PM -0700, Peter Geoghegan wrote: On Fri, Mar 29, 2019 at 11:20 AM Tomas Vondra wrote: I've pushed a fix for this. The short version is that the serialized representation was not respecting memory alignment requirements, which was causing issues in machines sensitive to this (ia64, sparc, hppa). It's a blind attempt, as I currently don't have access to any such machine. Looks like gharial still has problems: https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=gharial=2019-03-29%2018%3A30%3A47 Yeah, I saw that :-( Unfortunately fixing this without access to any ia64/sparc/hppa machine will be hard. The only thing I can do is wait for gaur/snapper to report the failure with a backtrace. OK, so the second memory-alignment fix I pushed on Saturday seems to have worked. There's one aspect of it that I don't like though - the MCV is first serialized into a buffer, which is then copied into the varlena buffer. Deserialization works in reverse, i.e. it copies the data from the varlena, then accessed the copy. For the emergency fix that seemed acceptable, as it somewhat reduced the amount of modified code, but I don't want to leave it like that because of efficiency and memory consumption. So barring objections I'll push the attached patch, which ensures proper memory alignment within the raw varlena buffer. Thus the copies are not needed. (The second part of the patch merely adds some alignment-enforcing asserts - I don't plan to commit that, but if someone plans decides to test this, it might be useful.) regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services >From 65c5613aeec46c425c6636d181c91631abfab05d Mon Sep 17 00:00:00 2001 From: Tomas Vondra Date: Tue, 2 Apr 2019 23:29:35 +0200 Subject: [PATCH 1/2] rework --- src/backend/statistics/mcv.c | 114 +++ 1 file changed, 75 insertions(+), 39 deletions(-) diff --git a/src/backend/statistics/mcv.c b/src/backend/statistics/mcv.c index d1af5d84d0..633f73e1be 100644 --- a/src/backend/statistics/mcv.c +++ b/src/backend/statistics/mcv.c @@ -491,7 +491,6 @@ statext_mcv_serialize(MCVList * mcvlist, VacAttrStats **stats) char *item = palloc0(itemsize); /* serialized items (indexes into arrays, etc.) */ - bytea *output; char *raw; char *ptr; @@ -625,15 +624,25 @@ statext_mcv_serialize(MCVList * mcvlist, VacAttrStats **stats) * Now we can finally compute how much space we'll actually need for the * whole serialized MCV list (varlena header, MCV header, dimension info * for each attribute, deduplicated values and items). +* +* The header fields are copied one by one, so that we don't need any +* explicit alignment (we copy them while deserializing). All fields +* after this need to be properly aligned, for direct access. */ - total_length = offsetof(MCVList, items) - + MAXALIGN(ndims * sizeof(DimensionInfo)); + total_length = MAXALIGN(VARHDRSZ + (3 * sizeof(uint32)) + + sizeof(AttrNumber) + (ndims * sizeof(Oid))); + + /* dimension info */ + total_length += MAXALIGN(ndims * sizeof(DimensionInfo)); /* add space for the arrays of deduplicated values */ for (i = 0; i < ndims; i++) total_length += MAXALIGN(info[i].nbytes); - /* and finally the items (no additional alignment needed) */ + /* +* And finally the items (no additional alignment needed, we start +* at proper alignment and the itemsize formula uses MAXALIGN) +*/ total_length += mcvlist->nitems * itemsize; /* @@ -641,11 +650,27 @@ statext_mcv_serialize(MCVList * mcvlist, VacAttrStats **stats) * so we set them to zero to make the result more compressible). */ raw = palloc0(total_length); - ptr = raw; + SET_VARSIZE(raw, total_length); + ptr = VARDATA(raw); + + /* copy the MCV list header fields, one by one */ + memcpy(ptr, >magic, sizeof(uint32)); + ptr += sizeof(uint32); + + memcpy(ptr, >type, sizeof(uint32)); + ptr += sizeof(uint32); - /* copy the MCV list header */ - memcpy(ptr, mcvlist, offsetof(MCVList, items)); - ptr += offsetof(MCVList, items); + memcpy(ptr, >nitems, sizeof(uint32)); + ptr += sizeof(uint32); + + memcpy(ptr, >ndimensions, sizeof(AttrNumber)); + ptr += sizeof(AttrNumber); + + memcpy(ptr, mcvlist->types, sizeof(Oid) * ndims); + ptr += (sizeof(Oid) * ndims); + + /* the header may not be exactly aligned, so make sure it is */ + ptr = raw + MAXALIGN(ptr - raw); /* store information about the attributes */ memcpy(ptr, info,
Re: pgsql: Report progress of CREATE INDEX operations
On 2019-Apr-02, Andres Freund wrote: > Note that longfin complains about something different than the output > files: > > https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=longfin=2019-04-02%2018%3A49%3A03 > > lmgr.c:882:14: error: variable 'count' is used uninitialized whenever '?:' > condition is false [-Werror,-Wsometimes-uninitialized] > > progress ? : NULL)); > > ^~~~ > lmgr.c:883:12: note: uninitialized use occurs here > total += count; > ^ > lmgr.c:882:14: note: remove the '?:' if its condition is always true > > progress ? : NULL)); > > ^~~ ~~~ > lmgr.c:878:14: note: initialize the variable 'count' to silence this warning > int count; > ^ Had noticed, thanks. Just pushed a blind fix for that ... -- Álvaro Herrerahttps://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
pgsql: Prevent use of uninitialized variable
Prevent use of uninitialized variable Per buildfarm member longfin. Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/e8abf97af770401934a2fc4887940b76403520f0 Modified Files -- src/backend/storage/lmgr/lmgr.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
Re: pgsql: Report progress of CREATE INDEX operations
Hi, On 2019-04-02 15:41:46 -0300, Alvaro Herrera wrote: > On 2019-Apr-02, Alvaro Herrera wrote: > > > Report progress of CREATE INDEX operations > > Argh, I pushed an older version of the commit. Note that longfin complains about something different than the output files: https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=longfin=2019-04-02%2018%3A49%3A03 lmgr.c:882:14: error: variable 'count' is used uninitialized whenever '?:' condition is false [-Werror,-Wsometimes-uninitialized] progress ? : NULL)); ^~~~ lmgr.c:883:12: note: uninitialized use occurs here total += count; ^ lmgr.c:882:14: note: remove the '?:' if its condition is always true progress ? : NULL)); ^~~ ~~~ lmgr.c:878:14: note: initialize the variable 'count' to silence this warning int count; ^ = 0 1 error generated. make[4]: *** [lmgr.o] Error 1 make[4]: *** Waiting for unfinished jobs Greetings, Andres Freund
pgsql: Update expected output for modified catalog definition
Update expected output for modified catalog definition Pilot error in previous commit Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/11074f26bc072334ab91e5646d113f95f884bd07 Modified Files -- src/test/regress/expected/rules.out | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
Re: pgsql: Report progress of CREATE INDEX operations
On 2019-Apr-02, Alvaro Herrera wrote: > Report progress of CREATE INDEX operations Argh, I pushed an older version of the commit. -- Álvaro Herrerahttps://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
pgsql: Report progress of CREATE INDEX operations
Report progress of CREATE INDEX operations This uses the progress reporting infrastructure added by c16dc1aca5e0, adding support for CREATE INDEX and CREATE INDEX CONCURRENTLY. There are two pieces to this: one is index-AM-agnostic, and the other is AM-specific. The latter is fairly elaborate for btrees, including reportage for parallel index builds and the separate phases that btree index creation uses; other index AMs, which are much simpler in their building procedures, have simplistic reporting only, but that seems sufficient, at least for non-concurrent builds. The index-AM-agnostic part is fairly complete, providing insight into the CONCURRENTLY wait phases as well as block-based progress during the index validation table scan. (The index validation index scan requires patching each AM, which has not been included here.) Reviewers: Rahila Syed, Pavan Deolasee, Tatsuro Yamada Discussion: https://postgr.es/m/20181220220022.mg63bhk26zdpvmcj@alvherre.pgsql Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/ab0dfc961b6a821f23d9c40c723d11380ce195a6 Modified Files -- contrib/amcheck/verify_nbtree.c | 2 +- contrib/bloom/blinsert.c | 2 +- contrib/bloom/blutils.c | 1 + doc/src/sgml/indexam.sgml| 13 ++ doc/src/sgml/monitoring.sgml | 224 ++- src/backend/access/brin/brin.c | 5 +- src/backend/access/gin/gininsert.c | 2 +- src/backend/access/gin/ginutil.c | 1 + src/backend/access/gist/gist.c | 1 + src/backend/access/gist/gistbuild.c | 2 +- src/backend/access/hash/hash.c | 7 +- src/backend/access/hash/hashsort.c | 6 + src/backend/access/heap/heapam_handler.c | 107 +++ src/backend/access/nbtree/nbtree.c | 9 ++ src/backend/access/nbtree/nbtsort.c | 56 +++- src/backend/access/nbtree/nbtutils.c | 24 src/backend/access/spgist/spginsert.c| 2 +- src/backend/access/spgist/spgutils.c | 1 + src/backend/catalog/index.c | 57 +++- src/backend/catalog/system_views.sql | 27 src/backend/commands/indexcmds.c | 72 -- src/backend/storage/ipc/standby.c| 2 +- src/backend/storage/lmgr/lmgr.c | 46 ++- src/backend/storage/lmgr/lock.c | 7 +- src/backend/utils/adt/amutils.c | 23 src/backend/utils/adt/pgstatfuncs.c | 2 + src/include/access/amapi.h | 4 + src/include/access/genam.h | 1 + src/include/access/nbtree.h | 11 ++ src/include/access/tableam.h | 7 + src/include/catalog/catversion.h | 2 +- src/include/catalog/pg_proc.dat | 10 +- src/include/commands/progress.h | 37 - src/include/pgstat.h | 5 +- src/include/storage/lmgr.h | 4 +- src/include/storage/lock.h | 2 +- src/test/regress/expected/rules.out | 30 - 37 files changed, 768 insertions(+), 46 deletions(-)
pgsql: Add support for partial TOAST decompression
Add support for partial TOAST decompression When asked for a slice of a TOAST entry, decompress enough to return the slice instead of decompressing the entire object. For use cases where the slice is at, or near, the beginning of the entry, this avoids a lot of unnecessary decompression work. This changes the signature of pglz_decompress() by adding a boolean to indicate if it's ok for the call to finish before consuming all of the source or destination buffers. Author: Paul Ramsey Reviewed-By: Rafia Sabih, Darafei Praliaskouski, Regina Obe Discussion: https://postgr.es/m/CACowWR07EDm7Y4m2kbhN_jnys%3DBBf9A6768RyQdKm_%3DNpkcaWg%40mail.gmail.com Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/4d0e994eed83c845a05da6e9a417b4efec67efaf Modified Files -- src/backend/access/heap/tuptoaster.c| 38 +++-- src/backend/access/transam/xlogreader.c | 2 +- src/backend/utils/adt/varlena.c | 22 ++--- src/common/pg_lzcompress.c | 42 +++-- src/include/common/pg_lzcompress.h | 2 +- 5 files changed, 70 insertions(+), 36 deletions(-)
pgsql: postgres_fdw: Perform the (FINAL, NULL) upperrel operations remo
postgres_fdw: Perform the (FINAL, NULL) upperrel operations remotely. The upper-planner pathification allows FDWs to arrange to push down different types of upper-stage operations to the remote side. This commit teaches postgres_fdw to do it for the (FINAL, NULL) upperrel, which is responsible for doing LockRows, LIMIT, and/or ModifyTable. This provides the ability for postgres_fdw to handle SELECT commands so that it 1) skips the LockRows step (if any) (note that this is safe since it performs early locking) and 2) pushes down the LIMIT and/or OFFSET restrictions (if any) to the remote side. This doesn't handle the INSERT/UPDATE/DELETE cases. Author: Etsuro Fujita Reviewed-By: Antonin Houska and Jeff Janes Discussion: https://postgr.es/m/87pnz1aby9@news-spur.riddles.org.uk Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/d50d172e517c1d2aabff3ceb3ad3113b909c5017 Modified Files -- contrib/postgres_fdw/deparse.c | 37 +- contrib/postgres_fdw/expected/postgres_fdw.out | 620 ++--- contrib/postgres_fdw/postgres_fdw.c| 342 +- contrib/postgres_fdw/postgres_fdw.h| 3 +- contrib/postgres_fdw/sql/postgres_fdw.sql | 16 +- src/backend/optimizer/plan/planner.c | 10 +- src/include/nodes/pathnodes.h | 18 + 7 files changed, 656 insertions(+), 390 deletions(-)
pgsql: Refactor create_limit_path() to share cost adjustment code with
Refactor create_limit_path() to share cost adjustment code with FDWs. This is in preparation for an upcoming commit. Author: Etsuro Fujita Reviewed-By: Antonin Houska and Jeff Janes Discussion: https://postgr.es/m/87pnz1aby9@news-spur.riddles.org.uk Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/aef65db6769e3f2c855dd89edcf95a536a9ab74f Modified Files -- src/backend/optimizer/util/pathnode.c | 87 ++- src/include/optimizer/pathnode.h | 3 ++ 2 files changed, 58 insertions(+), 32 deletions(-)
pgsql: postgres_fdw: Modify regression tests for EPQ-related planning p
postgres_fdw: Modify regression tests for EPQ-related planning problems. This prevents the tests added by commit 4bbf6edfbd and adjusted by commit 99f6a17dd6 from being useless by plan changes created by an upcoming commit. Author: Etsuro Fujita Discussion: https://postgr.es/m/87pnz1aby9@news-spur.riddles.org.uk Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/0269edefacb78700c95572c41a5a4c8633b631b6 Modified Files -- contrib/postgres_fdw/expected/postgres_fdw.out | 118 ++--- contrib/postgres_fdw/sql/postgres_fdw.sql | 12 ++- 2 files changed, 72 insertions(+), 58 deletions(-)
pgsql: postgres_fdw: Perform the (ORDERED, NULL) upperrel operations re
postgres_fdw: Perform the (ORDERED, NULL) upperrel operations remotely. The upper-planner pathification allows FDWs to arrange to push down different types of upper-stage operations to the remote side. This commit teaches postgres_fdw to do it for the (ORDERED, NULL) upperrel, which is responsible for evaluating the query's ORDER BY ordering. Since postgres_fdw is already able to evaluate that ordering remotely for foreign baserels and foreign joinrels (see commit aa09cd242f et al.), this adds support for that for foreign grouping relations. Author: Etsuro Fujita Reviewed-By: Antonin Houska and Jeff Janes Discussion: https://postgr.es/m/87pnz1aby9@news-spur.riddles.org.uk Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/ffab494a4d4676a71e830b2d7285eeeb5f2b53d7 Modified Files -- contrib/postgres_fdw/deparse.c | 28 +- contrib/postgres_fdw/expected/postgres_fdw.out | 167 +-- contrib/postgres_fdw/postgres_fdw.c| 388 +++-- contrib/postgres_fdw/postgres_fdw.h| 12 +- 4 files changed, 457 insertions(+), 138 deletions(-)
pgsql: Perform RLS subquery checks as the right user when going via a v
Perform RLS subquery checks as the right user when going via a view. When accessing a table with RLS via a view, the RLS checks are performed as the view owner. However, the code neglected to propagate that to any subqueries in the RLS checks. Fix that by calling setRuleCheckAsUser() for all RLS policy quals and withCheckOption checks for RTEs with RLS. Back-patch to 9.5 where RLS was added. Per bug #15708 from daurnimator. Discussion: https://postgr.es/m/15708-d65cab2ce9b17...@postgresql.org Branch -- REL_11_STABLE Details --- https://git.postgresql.org/pg/commitdiff/157dcf534f8e12486d425d6c0d111c065fbbb841 Modified Files -- src/backend/rewrite/rowsecurity.c | 8 src/test/regress/expected/rowsecurity.out | 27 +++ src/test/regress/sql/rowsecurity.sql | 26 ++ 3 files changed, 61 insertions(+)
pgsql: Perform RLS subquery checks as the right user when going via a v
Perform RLS subquery checks as the right user when going via a view. When accessing a table with RLS via a view, the RLS checks are performed as the view owner. However, the code neglected to propagate that to any subqueries in the RLS checks. Fix that by calling setRuleCheckAsUser() for all RLS policy quals and withCheckOption checks for RTEs with RLS. Back-patch to 9.5 where RLS was added. Per bug #15708 from daurnimator. Discussion: https://postgr.es/m/15708-d65cab2ce9b17...@postgresql.org Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/e2d28c0f404713f564dc2250646551c75172f17b Modified Files -- src/backend/rewrite/rowsecurity.c | 8 src/test/regress/expected/rowsecurity.out | 27 +++ src/test/regress/sql/rowsecurity.sql | 26 ++ 3 files changed, 61 insertions(+)
pgsql: Perform RLS subquery checks as the right user when going via a v
Perform RLS subquery checks as the right user when going via a view. When accessing a table with RLS via a view, the RLS checks are performed as the view owner. However, the code neglected to propagate that to any subqueries in the RLS checks. Fix that by calling setRuleCheckAsUser() for all RLS policy quals and withCheckOption checks for RTEs with RLS. Back-patch to 9.5 where RLS was added. Per bug #15708 from daurnimator. Discussion: https://postgr.es/m/15708-d65cab2ce9b17...@postgresql.org Branch -- REL_10_STABLE Details --- https://git.postgresql.org/pg/commitdiff/db4bc99948bf746c7035b49cc581eb1c44a30788 Modified Files -- src/backend/rewrite/rowsecurity.c | 8 src/test/regress/expected/rowsecurity.out | 27 +++ src/test/regress/sql/rowsecurity.sql | 26 ++ 3 files changed, 61 insertions(+)
pgsql: Perform RLS subquery checks as the right user when going via a v
Perform RLS subquery checks as the right user when going via a view. When accessing a table with RLS via a view, the RLS checks are performed as the view owner. However, the code neglected to propagate that to any subqueries in the RLS checks. Fix that by calling setRuleCheckAsUser() for all RLS policy quals and withCheckOption checks for RTEs with RLS. Back-patch to 9.5 where RLS was added. Per bug #15708 from daurnimator. Discussion: https://postgr.es/m/15708-d65cab2ce9b17...@postgresql.org Branch -- REL9_5_STABLE Details --- https://git.postgresql.org/pg/commitdiff/2e606d0ad7984c15f323eacfc333990b93683cc7 Modified Files -- src/backend/rewrite/rowsecurity.c | 8 src/test/regress/expected/rowsecurity.out | 31 +++ src/test/regress/sql/rowsecurity.sql | 31 +++ 3 files changed, 70 insertions(+)
pgsql: Perform RLS subquery checks as the right user when going via a v
Perform RLS subquery checks as the right user when going via a view. When accessing a table with RLS via a view, the RLS checks are performed as the view owner. However, the code neglected to propagate that to any subqueries in the RLS checks. Fix that by calling setRuleCheckAsUser() for all RLS policy quals and withCheckOption checks for RTEs with RLS. Back-patch to 9.5 where RLS was added. Per bug #15708 from daurnimator. Discussion: https://postgr.es/m/15708-d65cab2ce9b17...@postgresql.org Branch -- REL9_6_STABLE Details --- https://git.postgresql.org/pg/commitdiff/1b9a2f458b36747db07aff6631ade56c24cfda03 Modified Files -- src/backend/rewrite/rowsecurity.c | 8 src/test/regress/expected/rowsecurity.out | 27 +++ src/test/regress/sql/rowsecurity.sql | 26 ++ 3 files changed, 61 insertions(+)
