Re: vacuum verbose detail logs are unclear; log at *start* of each stage

2020-01-25 Thread Justin Pryzby
they're done in the text logfile. I looked, that's append_with_tabs() in elog.c. So that's a different possible implementation, which would apply to any message with newlines (or possibly just DETAIL). I'll also fork the allvisible/frozen/hintbits patches to a separate thread. Thanks, Justi

Re: explain HashAggregate to report bucket and memory stats

2020-01-26 Thread Justin Pryzby
On Sun, Jan 26, 2020 at 08:14:25AM -0600, Justin Pryzby wrote: > On Fri, Jan 03, 2020 at 10:19:25AM -0600, Justin Pryzby wrote: > > On Sun, Feb 17, 2019 at 11:29:56AM -0500, Jeff Janes wrote: > > https://www.postgresql.org/message-id/CAMkU%3D1zBJNVo2DGYBgLJqpu8fyjCE_ys%2Bms

vacuum verbose: show pages marked allvisible/frozen/hintbits

2020-01-26 Thread Justin Pryzby
d enough job of). The first patch seems simple enough but the 2nd could use critical review. >From 57eede7d1158904d6b66532c7d0ce6a59803210f Mon Sep 17 00:00:00 2001 From: Justin Pryzby Date: Sun, 29 Dec 2019 14:56:02 -0600 Subject: [PATCH v1 1/2] Report number of pages marked allvisible/frozen..

Re: explain HashAggregate to report bucket and memory stats

2020-01-26 Thread Justin Pryzby
On Fri, Jan 03, 2020 at 10:19:25AM -0600, Justin Pryzby wrote: > On Sun, Feb 17, 2019 at 11:29:56AM -0500, Jeff Janes wrote: > https://www.postgresql.org/message-id/CAMkU%3D1zBJNVo2DGYBgLJqpu8fyjCE_ys%2Bmsr6pOEoiwA7y5jrA%40mail.gmail.com > > What would I find

Re: error context for vacuum to include block number

2020-01-26 Thread Justin Pryzby
://www.postgresql.org/message-id/20200120054159.GT26045%40telsasoft.com On Sun, Jan 26, 2020 at 11:38:13PM -0600, Justin Pryzby wrote: > From 592a77554f99b5ff9035c55bf19a79a1443ae59e Mon Sep 17 00:00:00 2001 > From: Justin Pryzby > Date: Thu, 12 Dec 2019 20:54:37 -0600 > Subject: [PATCH v14

Re: error context for vacuum to include block number

2020-01-26 Thread Justin Pryzby
tablecmds.c: errdetail("The index \"%s\" belongs to a constraint in table \"%s\" but no constraint exists for index \"%s\".", src/backend/commands/cluster.c: errmsg("

Re: error context for vacuum to include block number

2020-01-27 Thread Justin Pryzby
On Mon, Jan 27, 2020 at 03:59:58PM +0900, Masahiko Sawada wrote: > On Mon, 27 Jan 2020 at 14:38, Justin Pryzby wrote: > > On Sun, Jan 26, 2020 at 12:29:38PM -0800, Andres Freund wrote: > > > > CONTEXT: while vacuuming relation "public.t_a_idx" > > > &g

typos in comments and user docs

2020-02-05 Thread Justin Pryzby
>From cb5842724330dfcfc914f2e3effdbfe4843be565 Mon Sep 17 00:00:00 2001 From: Justin Pryzby Date: Thu, 9 May 2019 21:13:55 -0500 Subject: [PATCH] spelling and typos --- doc/src/sgml/bloom.sgml| 2 +- doc/src/sgml/config.sgml | 2 +- doc/src/sgml/

Re: ALTER tbl rewrite loses CLUSTER ON index

2020-02-06 Thread Justin Pryzby
I wondered if it wouldn't be better if CLUSTER ON was stored in pg_class as the Oid of a clustered index, rather than a boolean in pg_index. That likely would've avoided (or at least exposed) this issue. And avoids the possibility of having two indices marked as "clustered". These would be more

Re: typos in comments and user docs

2020-02-06 Thread Justin Pryzby
n this branch, which shouldn't have been in this patch; fixed in the attached. >From a1780229e024e2e4b9a0549bcd516bb80b2d5a8d Mon Sep 17 00:00:00 2001 From: Justin Pryzby Date: Thu, 9 May 2019 21:13:55 -0500 Subject: [PATCH] spelling and typos --- doc/src/sgml/bloom.sgml

Re: error context for vacuum to include block number

2020-02-01 Thread Justin Pryzby
ch as LVRelStats > and LVShared. See attached Also, I think we shouldn't show a block number if it's "Invalid", to avoid saying "while vacuuming block 4294967295 of relation ..." For now, I made it not show any errcontext at all in that case. >From 94f715818dcdf3225a3e74

ALTER tbl rewrite loses CLUSTER ON index

2020-02-02 Thread Justin Pryzby
e | Default ++---+--+- i | bigint | | | Indexes: "t_i_idx" btree (i) WITH (fillfactor='11', vacuum_cleanup_index_scale_factor='12') >From f235a691722a464059358cd6b1d744f75d7bf92f Mon Sep 17 00:00:00 2001 From: Justin Pryz

Re: typos in comments and user docs

2020-02-06 Thread Justin Pryzby
On Fri, Feb 07, 2020 at 08:33:40AM +0530, Amit Kapila wrote: > On Thu, Feb 6, 2020 at 7:26 PM Justin Pryzby wrote: > > > > On Thu, Feb 06, 2020 at 04:43:18PM +0530, Amit Kapila wrote: > > > On Thu, Feb 6, 2020 at 10:45 AM Michael Paquier > > > wrote: > >

Re: typos in comments and user docs

2020-02-06 Thread Justin Pryzby
On Fri, Feb 07, 2020 at 09:26:04AM +0530, Amit Kapila wrote: > On Fri, Feb 7, 2020 at 8:41 AM Justin Pryzby wrote: > > > > On Fri, Feb 07, 2020 at 08:33:40AM +0530, Amit Kapila wrote: > > > On Thu, Feb 6, 2020 at 7:26 PM Justin Pryzby wrote: > > > > > &

Re: ALTER tbl rewrite loses CLUSTER ON index (consider moving indisclustered to pg_class)

2020-02-07 Thread Justin Pryzby
On Thu, Feb 06, 2020 at 02:24:47PM -0300, Alvaro Herrera wrote: > On 2020-Feb-06, Justin Pryzby wrote: > > > I wondered if it wouldn't be better if CLUSTER ON was stored in pg_class as > > the > > Oid of a clustered index, rather than a boolean in pg_index. > &

Re: error context for vacuum to include block number

2020-02-07 Thread Justin Pryzby
, so error handling is out of the way and minimally distract from the rest of vacuum. >From 95265412c56f3b308eed16531d7c83243e278f4f Mon Sep 17 00:00:00 2001 From: Justin Pryzby Date: Thu, 12 Dec 2019 20:54:37 -0600 Subject: [PATCH v17 1/2] vacuum errcontext to show block being processed As reques

ALTER TABLE rewrite to use clustered order

2020-02-08 Thread Justin Pryzby
Note, this doesn't use clustered order when rewriting only due to tablespace change. Alter currently does an AM specific block copy without looking at tuples. But I think it'd be possible to use tuplesort and copy if desired. >From f93bbd6c30e883068f46ff86def28d0e66aea4f5 Mon Sep 17 00:00:00 200

Re: error context for vacuum to include block number

2020-01-24 Thread Justin Pryzby
ot;while vacuuming index \"%s.%s\""? Yes. I'm still unclear if this is useful without a block number, or why it wouldn't be better to write DEBUG1 log with index name before vacuuming each. Justin >From 6332127178e29967dfeb12577eb9a61e813a33a8 Mon Sep 17 00:00:00 2001 From: Ju

tableam options for pg_dump/ALTER/LIKE

2020-01-28 Thread Justin Pryzby
I made these casual comments. If there's any agreement on their merit, it'd be nice to implement at least the first for v13. In <20190818193533.gl11...@telsasoft.com>, I wrote: > . What do you think about pg_restore --no-tableam; similar to >--no-tablespaces, it would allow restoring a

Re: ALTER tbl rewrite loses CLUSTER ON index

2020-02-05 Thread Justin Pryzby
On Wed, Feb 05, 2020 at 03:53:45PM +0900, Amit Langote wrote: > Hi Justin, > > On Mon, Feb 3, 2020 at 1:17 AM Justin Pryzby wrote: > > Other options are preserved by ALTER (and CLUSTER ON is and most obviously > > should be preserved by CLUSTER's rewrite), so I think

Re: error context for vacuum to include block number

2020-02-14 Thread Justin Pryzby
vious due to the switch statement you proposed. Thanks for continued reviews. -- Justin >From 94768a134118d30853b75a96b90166363f0fef5b Mon Sep 17 00:00:00 2001 From: Justin Pryzby Date: Thu, 12 Dec 2019 20:54:37 -0600 Subject: [PATCH v19] vacuum errcontext to show block being processed Discus

Re: explain HashAggregate to report bucket and memory stats

2020-02-19 Thread Justin Pryzby
On Sun, Feb 16, 2020 at 11:53:07AM -0600, Justin Pryzby wrote: > Updated: > > . remove from explain analyze those tests which would display sort >Memory/Disk. Oops. . Rebased on top of 5b618e1f48aecc66e3a9f60289491da520faae19 . Updated to avoid sort's Disk output, for re

Re: error context for vacuum to include block number

2020-02-19 Thread Justin Pryzby
# SET statement_timeout=0; SET maintenance_work_mem='1MB'; DROP TABLE tt; CREATE UNLOGGED TABLE tt(i int); INSERT INTO tt SELECT generate_series(1,39); CREATE INDEX ON tt(i); UPDATE tt SET i=i-1; SET statement_timeout=1222; VACUUM VERBOSE tt; >From 91158171f75cd20e69b18843dd3b6525961e4e8b Mon Sep 17

Re: subplan resets wrong hashtable

2020-02-09 Thread Justin Pryzby
On Sun, Feb 09, 2020 at 08:01:26PM -0800, Andres Freund wrote: > Ugh, that indeed looks wrong. Did you check whether it can actively > cause wrong query results? If so, did you do theoretically, or got to a > query returning wrong results? Actually .. I can "theoretically" prove that there's no

subplan resets wrong hashtable

2020-02-09 Thread Justin Pryzby
I believe the 2nd hunk should reset node->hashnulls, rather than reset ->hashtable a 2nd time: @@ -505,7 +505,10 @@ buildSubPlanHash(SubPlanState *node, ExprContext *econtext) if (nbuckets < 1) nbuckets = 1; - node->hashtable = BuildTupleHashTable(node->parent, +

Re: subplan resets wrong hashtable

2020-02-09 Thread Justin Pryzby
On Sun, Feb 09, 2020 at 08:01:26PM -0800, Andres Freund wrote: > Ugh, that indeed looks wrong. Did you check whether it can actively > cause wrong query results? If so, did you do theoretically, or got to a > query returning wrong results? No, I only noticed while reading code. I tried briefly

Re: Allow CLUSTER, VACUUM FULL and REINDEX to change tablespace on the fly

2020-02-11 Thread Justin Pryzby
For your v7 patch, which handles REINDEX to a new tablespace, I have a few minor comments: + * the relation will be rebuilt. If InvalidOid is used, the default => should say "currrent", not default ? +++ b/doc/src/sgml/ref/reindex.sgml +TABLESPACE ... +new_tablespace => I saw you

Re: explain HashAggregate to report bucket and memory stats

2020-02-16 Thread Justin Pryzby
table size is now redundant with nbuckets (if you know sizeof(TupleHashEntryData)); -- Justin >From c989b75f820dbda0540b3d2cd092eaf1f8629baa Mon Sep 17 00:00:00 2001 From: Justin Pryzby Date: Sat, 15 Feb 2020 12:03:11 -0600 Subject: [PATCH v3 1/7] Run some existing tests with explain (ANA

reindex concurrently and two toast indexes

2020-02-16 Thread Justin Pryzby
Forking old, long thread: https://www.postgresql.org/message-id/36712441546604286%40sas1-890ba5c2334a.qloud-c.yandex.net On Fri, Jan 04, 2019 at 03:18:06PM +0300, Sergei Kornilov wrote: > About reindex invalid indexes - i found one good question in archives [1]: > how about toast indexes? > I

Re: error context for vacuum to include block number

2020-02-16 Thread Justin Pryzby
which is more likely to hit any lowlevel error, so I added callback there, too. BTW, for the index cases, I didn't like repeating the namespace here, but WDYT ? |CONTEXT: while vacuuming index "public.t_i_idx3" of relation "t" Thanks for rerere-reviewing. -- Justin >From 82952

Re: error context for vacuum to include block number

2020-02-16 Thread Justin Pryzby
. Also, I just added this to the TRUNCATE case, even though that should never happen: if (BlockNumberIsValid(cbarg->blkno))... -- Justin >From 977b1b5e00ce522bd775cf91f7a9c7a9345d3171 Mon Sep 17 00:00:00 2001 From: Justin Pryzby Date: Thu, 12 Dec 2019 20:54:37 -0600 Subject: [PATCH v20 1/2]

Re: assert pg_class.relnatts is consistent

2020-02-16 Thread Justin Pryzby
On Mon, Feb 17, 2020 at 01:25:05PM +0900, Amit Langote wrote: > > Pushed both of those. > > Thank you. > > It's amazing to see how simple bootstrapping has now become thanks to > the work you guys have done recently. On Fri, Feb 14, 2020 at 06:00:05PM +0900, Amit Langote wrote: > > I can't

Re: ALTER tbl rewrite loses CLUSTER ON index (consider moving indisclustered to pg_class)

2020-02-16 Thread Justin Pryzby
On Mon, Feb 17, 2020 at 02:31:42PM +0900, Amit Langote wrote: > Hi Justin, > > On Fri, Feb 7, 2020 at 11:39 PM Justin Pryzby wrote: > > On Thu, Feb 06, 2020 at 02:24:47PM -0300, Alvaro Herrera wrote: > > > On 2020-Feb-06, Justin Pryzby wrote: > > > > >

Re: reindex concurrently and two toast indexes

2020-02-22 Thread Justin Pryzby
On Tue, Feb 18, 2020 at 02:29:33PM +0900, Michael Paquier wrote: > On Sun, Feb 16, 2020 at 01:08:35PM -0600, Justin Pryzby wrote: > > Forking old, long thread: > > https://www.postgresql.org/message-id/36712441546604286%40sas1-890ba5c2334a.qloud-c.yandex.net > > On Fri, Jan 0

Re: explain HashAggregate to report bucket and memory stats

2020-02-22 Thread Justin Pryzby
xplain_parallel_sort_stats. Actually, I wish there were a way to control Sort nodes' Memory/Disk output, too. I'm sure most of regression tests were meant to be run as explain(analyze NO), but it'd be much better if analyze YES were reasonably easy in the general case that might include Sort. If someone seconds that, I will start a separate thread. -- Justin Pryzby

v12 "won't fix" item regarding memory leak in "ATTACH PARTITION without AEL"; (or, relcache ref counting)

2020-02-23 Thread Justin Pryzby
This links to a long thread, from which I've tried to quote some of the most important mails, below. https://wiki.postgresql.org/wiki/PostgreSQL_12_Open_Items#Won.27t_Fix I wondered if there's an effort to pursue a resolution for v13 ? On Fri, Apr 12, 2019 at 11:42:24AM -0400, Tom Lane wrote in

Re: error context for vacuum to include block number

2020-02-13 Thread Justin Pryzby
UPDATE t SET i=i-1; SET statement_timeout=111; SET vacuum_cost_delay=3; SET vacuum_cost_page_dirty=0; SET vacuum_cost_page_hit=11; SET vacuum_cost_limit=33; SET statement_timeout=; VACUUM VERBOSE t; Thanks for re-reviewing. -- Justin >From 5b8cad37244cdc310d78719b64ff44a464910598 Mon Sep 17 00:00

Re: explain HashAggregate to report bucket and memory stats

2020-02-15 Thread Justin Pryzby
On Mon, Feb 03, 2020 at 06:53:01AM -0800, Andres Freund wrote: > On 2020-01-03 10:19:26 -0600, Justin Pryzby wrote: > > On Sun, Feb 17, 2019 at 11:29:56AM -0500, Jeff Janes wrote: > > https://www.postgresql.org/message-id/CAMkU%3D1zBJNVo2DGYBgLJqpu8fyjCE_ys%2Bmsr6pOEoiwA7y5jrA%4

Re: Make ringbuffer threshold and ringbuffer sizes configurable?

2020-02-19 Thread Justin Pryzby
On Wed, Feb 05, 2020 at 08:00:26PM -0800, Andres Freund wrote: > I think it would make sense to have seqscan_ringbuffer_threshold, > {bulkread,bulkwrite,vacuum}_ringbuffer_size. I suggest the possibility of somehow forcing a ringbuffer for nonbulk writes for the current session. In our use-case,

Re: assert pg_class.relnatts is consistent

2020-02-13 Thread Justin Pryzby
On Thu, Feb 13, 2020 at 04:51:01PM +0900, Amit Langote wrote: > On Thu, Feb 13, 2020 at 3:23 AM Justin Pryzby wrote: > > Forking this thread for two tangential patches which I think are more > > worthwhile than the original topic's patch. > > https://www.postgresql.org/mess

Re: bitmaps and correlation

2020-01-12 Thread Justin Pryzby
On Mon, Jan 06, 2020 at 11:26:06PM -0600, Justin Pryzby wrote: > As Jeff has pointed out, high correlation has two effects in cost_index(): > 1) the number of pages read will be less; > 2) the pages will be read more sequentially; > > cost_index reuses the pages_fetched variabl

Re: vacuum verbose detail logs are unclear; log at *start* of each stage; show allvisible/frozen/hintbits

2020-01-12 Thread Justin Pryzby
On Sun, Dec 29, 2019 at 01:15:24PM -0500, Jeff Janes wrote: > On Fri, Dec 20, 2019 at 12:11 PM Justin Pryzby wrote: > > > This is a usability complaint. If one knows enough about vacuum and/or > > logging, I'm sure there's no issue. > > > | 11 DEBUG: &q

Re: [PATCH v1] pg_ls_tmpdir to show directories

2020-01-15 Thread Justin Pryzby
On Wed, Jan 15, 2020 at 11:21:36AM +0100, Fabien COELHO wrote: > I'm trying to think about how to get rid of the strange structure and hacks, > and the arbitrary looking size 2 array. > > Also the recursion is one step, but I'm not sure why, ISTM it could/should > go on always? Because tmpfiles

Re: [PATCH v1] pg_ls_tmpdir to show directories

2020-01-16 Thread Justin Pryzby
On Thu, Jan 16, 2020 at 09:34:32AM +0100, Fabien COELHO wrote: > Also, I'm not fully sure why ".*" files should be skipped, maybe it should > be an option? Or the user can filter it with SQL if it does not want them? I think if someone wants the full generality, they can do this: postgres=#

Re: progress report for ANALYZE

2020-01-16 Thread Justin Pryzby
existing instances of "preparing to begin". >From de108e69b5d33c881074b0a04697d7061684f823 Mon Sep 17 00:00:00 2001 From: Justin Pryzby Date: Wed, 15 Jan 2020 23:10:29 -0600 Subject: [PATCH v1] Doc review for ANALYZE progress (a166d408) --- doc/src/sgml/monitoring.sgml | 10 +--

should crash recovery ignore checkpoint_flush_after ?

2020-01-18 Thread Justin Pryzby
One of our PG12 instances was in crash recovery for an embarassingly long time after hitting ENOSPC. (Note, I first started wroting this mail 10 months ago while running PG11 after having same experience after OOM). Running linux. As I understand, the first thing that happens syncing every file

Re: should crash recovery ignore checkpoint_flush_after ?

2020-01-18 Thread Justin Pryzby
On Sat, Jan 18, 2020 at 10:48:22AM -0800, Andres Freund wrote: > Hi, > > On 2020-01-18 08:08:07 -0600, Justin Pryzby wrote: > > One of our PG12 instances was in crash recovery for an embarassingly long > > time > > after hitting ENOSPC. (Note, I first started

Re: should crash recovery ignore checkpoint_flush_after ?

2020-01-19 Thread Justin Pryzby
On Sat, Jan 18, 2020 at 03:32:02PM -0800, Andres Freund wrote: > On 2020-01-19 09:52:21 +1300, Thomas Munro wrote: > > On Sun, Jan 19, 2020 at 3:08 AM Justin Pryzby wrote: > > Does sync_file_range() even do anything for non-mmap'd files on ZFS? > > Good point. Next time i

Re: doc: vacuum full, fillfactor, and "extra space"

2020-01-19 Thread Justin Pryzby
Rebased against 40d964ec997f64227bc0ff5e058dc4a5770a70a9 >From b9f10d21de62354d953e388642fcdfc6d97a4a47 Mon Sep 17 00:00:00 2001 From: Justin Pryzby Date: Thu, 26 Dec 2019 18:54:28 -0600 Subject: [PATCH v2] doc: VACUUM FULL: separate paragraph; fillfactor FILLFACTOR seems to apply here. A

Re: error context for vacuum to include block number

2020-01-19 Thread Justin Pryzby
Rebased against 40d964ec997f64227bc0ff5e058dc4a5770a70a9 I moved some unrelated patches to a separate thread ("vacuum verbose detail logs are unclear") >From 33c7166e3c8f056a8eb6295ec92fed8c85eda7d6 Mon Sep 17 00:00:00 2001 From: Justin Pryzby Date: Mon, 23 Dec 2019 14:38:01

Re: doc: vacuum full, fillfactor, and "extra space"

2020-01-14 Thread Justin Pryzby
On Fri, Dec 27, 2019 at 11:58:18AM +0100, Fabien COELHO wrote: >> I started writing this patch to avoid the possibly-misleading phrase: "with >> no >> extra space" (since it's expected to typically take ~2x space, or 1x "extra" >> space). >> >> But the original phrase "with no extra space" seems

Re: error context for vacuum to include block number

2020-01-21 Thread Justin Pryzby
>From c16989c79fe331a3280bbbfb2dd9c040948cff53 Mon Sep 17 00:00:00 2001 From: Justin Pryzby Date: Thu, 12 Dec 2019 20:54:37 -0600 Subject: [PATCH v12 1/5] vacuum errcontext to show block being processed As requested here. https://www.postgresql.org/message-id/20190807235154.erbmr4o4bo6vgn

Re: doc: alter table references bogus table-specific planner parameters

2020-01-21 Thread Justin Pryzby
On Mon, Jan 06, 2020 at 04:33:46AM +, Simon Riggs wrote: > On Mon, 6 Jan 2020 at 04:13, Justin Pryzby wrote: > > > I agree with the sentiment of the third doc change, but your patch removes > > > the mention of n_distinct, which isn't appropriate. > > > &g

Re: error context for vacuum to include block number

2020-01-22 Thread Justin Pryzby
On Mon, Jan 20, 2020 at 11:11:20AM -0800, Andres Freund wrote: > > @@ -966,8 +986,11 @@ lazy_scan_heap(Relation onerel, VacuumParams *params, > > LVRelStats *vacrelstats, > > /* Work on all the indexes, then the heap */ > > + /* Don't use the errcontext

Re: error context for vacuum to include block number

2020-01-21 Thread Justin Pryzby
med to be useful, I'll make a separate branch for the others. >From 9a28281a8f9c82634b263ce9b66b6c0cdfd01b2d Mon Sep 17 00:00:00 2001 From: Justin Pryzby Date: Thu, 12 Dec 2019 20:54:37 -0600 Subject: [PATCH v11 1/6] vacuum errcontext to show block being processed As requested here.

Re: error context for vacuum to include block number

2020-01-20 Thread Justin Pryzby
(VERBOSE, PARALLEL 0) t; ERROR: canceling statement due to statement timeout CONTEXT: while vacuuming relation "public.t_a_idx" I haven't found a good way of exercizing the "vacuuming heap" path, though. >From 44f5eaaef66c570395a9af2bdbe74943c9163c4d Mon Sep 17 00:

avoid some calls to memset with array initializer

2020-01-02 Thread Justin Pryzby
s.S1 >From 5117e66043b6c8c66c2f98fcd99fdaefec66f90e Mon Sep 17 00:00:00 2001 From: Justin Pryzby Date: Fri, 27 Dec 2019 17:30:36 -0600 Subject: [PATCH v1 1/2] Avoid some calls to memset.. ..in cases where that saves a couple lines of code. Note that gcc has builtin for memset, but inlined function is still not same as

infinite histogram bounds and nan (Re: comment regarding double timestamps; and, infinite timestamps and NaN)

2020-01-02 Thread Justin Pryzby
it's intended to be hit for an infinite histogram bound, but that doesn't work for timestamps (convert_to_scalar() should return (double)INFINITY and not (double)INT64_MIN/MAX). On Mon, Dec 30, 2019 at 02:18:17PM -0500, Tom Lane wrote: > Justin Pryzby writes: > > On Mon, Dec 30, 2019 at 09:05:2

Re: error context for vacuum to include block number

2020-01-02 Thread Justin Pryzby
On Thu, Dec 26, 2019 at 09:57:04AM -0600, Justin Pryzby wrote: > So rebasified against your patch. Rebased against your patch in master this time. >From dadb8dff6ea929d78f3695f606de9ade7674b7a1 Mon Sep 17 00:00:00 2001 From: Justin Pryzby Date: Wed, 27 Nov 2019 20:07:10 -0600 Subject: [PA

explain HashAggregate to report bucket and memory stats

2020-01-03 Thread Justin Pryzby
emory and bucket stats; and if the Aggregate > node would report...anything. Find attached my WIP attempt to implement this. Jeff: can you suggest what details Aggregate should show ? Justin >From 5d0afe5d92649f575d9b09ae19b31d2bfd5bfd12 Mon Sep 17 00:00:00 2001 From: Justin Pryzby Date: We

allow disabling indexscans without disabling bitmapscans

2020-01-04 Thread Justin Pryzby
to bitmap heapscan is topic for the other patch. Justin >From 6ad506879d8a754013b971197592fc9617850b7e Mon Sep 17 00:00:00 2001 From: Justin Pryzby Date: Sat, 4 Jan 2020 10:25:12 -0600 Subject: [PATCH v1] allow disabling indexscans but not bitmap scans --- src/backend/optimizer/path/costsize

Re: error context for vacuum to include block number

2019-12-23 Thread Justin Pryzby
On Mon, Dec 16, 2019 at 11:49:56AM +0900, Michael Paquier wrote: > On Sun, Dec 15, 2019 at 10:27:12AM -0600, Justin Pryzby wrote: > > I named it so because it calls both lazy_vacuum_index > > ("PROGRESS_VACUUM_PHASE_VACUUM_INDEX") and > > lazy_vacuum_heap(&qu

Re: error context for vacuum to include block number

2019-12-26 Thread Justin Pryzby
On Tue, Dec 24, 2019 at 01:19:09PM +0900, Michael Paquier wrote: > On Mon, Dec 23, 2019 at 07:24:28PM -0600, Justin Pryzby wrote: > > I renamed. > > Hmm. I have found what was partially itching me for patch 0002, and > that's actually the fact that we don't do the error

Re: ALTER INDEX fails on partitioned index

2019-12-26 Thread Justin Pryzby
On Mon, Jan 07, 2019 at 04:23:30PM -0300, Alvaro Herrera wrote: > On 2019-Jan-05, Justin Pryzby wrote: > > postgres=# CREATE TABLE t(i int)PARTITION BY RANGE(i); > > postgres=# CREATE INDEX ON t(i) WITH(fillfactor=11); > > postgres=# ALTER INDEX t_i_idx SET (fillfactor=1

doc: vacuum full, fillfactor, and "extra space"

2019-12-26 Thread Justin Pryzby
I started writing this patch to avoid the possibly-misleading phrase: "with no extra space" (since it's expected to typically take ~2x space, or 1x "extra" space). But the original phrase "with no extra space" seems to be wrong anyway, since it actually follows fillfactor, so say that. Possibly

Re: planner support functions: handle GROUP BY estimates ?

2019-12-26 Thread Justin Pryzby
On Sun, Dec 22, 2019 at 06:16:48PM -0600, Justin Pryzby wrote: > On Tue, Nov 19, 2019 at 01:34:21PM -0600, Justin Pryzby wrote: > > Tom implemented "Planner support functions": > > https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=a391ff3c3d418e404a2c6e4ff

Re: error context for vacuum to include block number (atomic progress update)

2019-12-29 Thread Justin Pryzby
On Sat, Dec 28, 2019 at 07:21:31PM -0500, Robert Haas wrote: > On Thu, Dec 26, 2019 at 10:57 AM Justin Pryzby wrote: > > I agree that's better. > > I don't see any reason why the progress params need to be updated > > atomically. > > So rebasified against your patch.

Re: [PATCH v1] pg_ls_tmpdir to show directories

2019-12-27 Thread Justin Pryzby
Re-added -hackers. Thanks for reviewing. On Fri, Dec 27, 2019 at 05:22:47PM +0100, Fabien COELHO wrote: > The implementation simply extends an existing functions with a boolean to > allow for sub-directories. However, the function does not seem to show > subdir contents recursively. Should it be

comment regarding double timestamps; and, infinite timestamps and NaN

2019-12-29 Thread Justin Pryzby
t; CREATE TABLE t(t) AS SELECT generate_series(now(), now()+'1 day', '5 minutes'); INSERT INTO t VALUES('-infinity'); ALTER TABLE t ALTER t SET STATISTICS 1; ANALYZE t; explain SELECT * FROM t WHERE t>='2010-12-29'; >From b0151e24819499607eb2894dd920d4d8ef74b57d Mon Sep 17 00:00:00 2001 From: Justin Pry

Re: comment regarding double timestamps; and, infinite timestamps and NaN

2019-12-30 Thread Justin Pryzby
On Mon, Dec 30, 2019 at 09:05:24AM -0500, Tom Lane wrote: > Justin Pryzby writes: > > That seems to be only used for ineq_histogram_selectivity() interpolation of > > histogram bins. It looks to me that at least isn't working for "special > > values", and needs to

Re: allow disabling indexscans without disabling bitmapscans

2020-01-04 Thread Justin Pryzby
On Sat, Jan 04, 2020 at 10:50:47AM -0600, Justin Pryzby wrote: > > Doesn't enable_indexscan=off accomplish this already? It is possible but > > not terribly likely to switch from index to seq, rather than from index to > > bitmap. (Unless the index scan was being used to obtain

doc: alter table references bogus table-specific planner parameters

2020-01-05 Thread Justin Pryzby
Justin >From 64699ee90ef6ebe9459e3b2b1f603f30ec2c49c8 Mon Sep 17 00:00:00 2001 From: Justin Pryzby Date: Sun, 5 Jan 2020 19:39:29 -0600 Subject: [PATCH v1] Fixes for commit 6f3a13ff Should backpatch to v10. --- doc/src/sgml/ref/alter_table.sgml | 8 +++- 1 file changed, 3 insertions(+)

Re: doc: alter table references bogus table-specific planner parameters

2020-01-05 Thread Justin Pryzby
On Mon, Jan 06, 2020 at 03:48:52AM +, Simon Riggs wrote: > On Mon, 6 Jan 2020 at 02:56, Justin Pryzby wrote: > > > commit 6f3a13ff058f15d565a30c16c0c2cb14cc994e42 Enhance docs for ALTER > > TABLE lock levels of storage parms > > Author: Simon Riggs > > Date:

Re: bitmaps and correlation

2020-01-06 Thread Justin Pryzby
Find attached cleaned up patch. For now, I updated the regress/expected/, but I think the test maybe has to be updated to do what it was written to do. >From 36f547d69b8fee25869d6ef3ef26d327a8ba1205 Mon Sep 17 00:00:00 2001 From: Justin Pryzby Date: Tue, 1 Jan 2019 16:17:28 -0600 Subject: [PA

Re: bitmaps and correlation

2020-01-06 Thread Justin Pryzby
On Tue, Jan 07, 2020 at 09:21:03AM +0530, Dilip Kumar wrote: > On Tue, Jan 7, 2020 at 1:29 AM Justin Pryzby wrote: > > > > Find attached cleaned up patch. > > For now, I updated the regress/expected/, but I think the test maybe has to > > be > > updated to do

Re: [PATCH v1] pg_ls_tmpdir to show directories

2019-12-27 Thread Justin Pryzby
On Fri, Dec 27, 2019 at 06:50:24PM +0100, Fabien COELHO wrote: > >On Fri, Dec 27, 2019 at 05:22:47PM +0100, Fabien COELHO wrote: > >>The implementation simply extends an existing functions with a boolean to > >>allow for sub-directories. However, the function does not seem to show > >>subdir

Re: [PATCH v1] pg_ls_tmpdir to show directories

2019-12-28 Thread Justin Pryzby
\"%s\": %m", dir))); +errmsg("could not stat file \"%s\": %m", path))); >From fd88be5f1687354d9990fb1838adc0db36bc6dde Mon Sep 17 00:00:00 2001 From: Justin Pryzby Date: Fri, 27 Dec 2019 23:34:14 -0600 Subject: [

Re: [PATCH v1] pg_ls_tmpdir to show directories

2019-12-28 Thread Justin Pryzby
, too. And maybe the is_dir flag should be re-introduced (although someone could call pg_stat_file if needed). I'm interested to hear feedback on that, although this patch still isn't great. >From dd3b2779939fc1b396fed1fba2f7cefc9a6b1ad5 Mon Sep 17 00:00:00 2001 From: Justin Pryzby Date: Fri,

Re: Allow CLUSTER, VACUUM FULL and REINDEX to change tablespace on the fly

2020-03-09 Thread Justin Pryzby
On Sat, Feb 29, 2020 at 08:53:04AM -0600, Justin Pryzby wrote: > On Sat, Feb 29, 2020 at 03:35:27PM +0300, Alexey Kondratov wrote: > > Anyway, new version is attached. It is rebased in order to resolve conflicts > > with a recent fix of REINDEX CONCURRENTLY + temp relation

Re: pg11+: pg_ls_*dir LIMIT 1: temporary files .. not closed at end-of-transaction

2020-03-12 Thread Justin Pryzby
On Sun, Mar 08, 2020 at 04:30:44PM -0400, Tom Lane wrote: > BTW, another thing I noticed while looking around is that some of > the functions using SRF_RETURN_DONE() think they should clean up > memory beforehand. This is a waste of code/cycles, as long as the > memory was properly allocated in

Re: pg11+: pg_ls_*dir LIMIT 1: temporary files .. not closed at end-of-transaction

2020-03-12 Thread Justin Pryzby
08b248df2bc3365b6336e4 Mon Sep 17 00:00:00 2001 From: Justin Pryzby Date: Wed, 11 Mar 2020 10:09:18 -0500 Subject: [PATCH] SRF: avoid leaking resources if not run to completion Change to return a tuplestore populated immediately and returned in full. Discussion: https://www.postgresql.org/me

Re: Berserk Autovacuum (let's save next Mandrill)

2020-03-09 Thread Justin Pryzby
> +++ b/src/backend/utils/misc/postgresql.conf.sample > +#autovacuum_vacuum_insert_threshold = 1000 # min number of row > inserts > + # before vacuum Similar to a previous comment [0] about reloptions or GUC: Can we say "threshold number of

Re: pg11+: pg_ls_*dir LIMIT 1: temporary files .. not closed at end-of-transaction

2020-03-11 Thread Justin Pryzby
On Sun, Mar 08, 2020 at 03:40:09PM -0400, Tom Lane wrote: > Justin Pryzby writes: > > On Sun, Mar 08, 2020 at 02:37:49PM -0400, Tom Lane wrote: > >> I guess we ought to change that function to use returns-a-tuplestore > >> protocol instead of thinking it can hold a

Re: explain HashAggregate to report bucket and memory stats

2020-03-06 Thread Justin Pryzby
eate a new explain(MACHINE) option to allow more stable output, by avoiding Memory/Disk. That doesn't attempt to make all "explain analyze" output stable - there's other issues, I think mostly related to parallel workers (see 4ea03f3f, 13e8b2ee). But does allow retiring explain_sq_limit and e

Re: pg_ls_tmpdir to show directories and shared filesets

2020-03-06 Thread Justin Pryzby
On Thu, Mar 05, 2020 at 10:18:38AM -0600, Justin Pryzby wrote: > I'm not sure if prefer the 0002 patch alone (which recurses into dirs all at > once during the initial call), or 0002+3+4, which incrementally reads the dirs > on each call (but requires keeping dirs opened). I fixed

pg11+: pg_ls_*dir LIMIT 1: temporary files .. not closed at end-of-transaction

2020-03-08 Thread Justin Pryzby
While working on a patch, I noticed this pre-existing behavior, which seems to be new since v11, maybe due to changes to SRF. |postgres=# SELECT pg_ls_dir('.') LIMIT 1; |WARNING: 1 temporary files and directories not closed at end-of-transaction |pg_ls_dir | pg_dynshmem |postgres=# SELECT

Re: pg11+: pg_ls_*dir LIMIT 1: temporary files .. not closed at end-of-transaction

2020-03-08 Thread Justin Pryzby
On Sun, Mar 08, 2020 at 02:37:49PM -0400, Tom Lane wrote: > Justin Pryzby writes: > > While working on a patch, I noticed this pre-existing behavior, which seems > > to > > be new since v11, maybe due to changes to SRF. > > > |postgres=# SELECT pg_ls_dir('.') LIMIT

Re: pg_ls_tmpdir to show directories and shared filesets

2020-03-07 Thread Justin Pryzby
On Sat, Mar 07, 2020 at 03:14:37PM +0100, Fabien COELHO wrote: > Some feedback about the v7 patch set. Thanks for looking again > About v7.1, seems ok. > > About v7.2 & v7.3 seems ok, altought the two could be merged. These are separate since I proprose that one should be backpatched to v12

Re: Memory-Bounded Hash Aggregation

2020-03-12 Thread Justin Pryzby
On Wed, Mar 11, 2020 at 11:55:35PM -0700, Jeff Davis wrote: > * tweaked EXPLAIN output some more > Unless I (or someone else) finds something significant, this is close > to commit. Thanks for working on this ; I finally made a pass over the patch. +++ b/doc/src/sgml/config.sgml +

Re: Additional size of hash table is alway zero for hash aggregates

2020-03-12 Thread Justin Pryzby
On Thu, Mar 12, 2020 at 12:16:26PM -0700, Andres Freund wrote: > On 2020-03-12 16:35:15 +0800, Pengzhou Tang wrote: > > When reading the grouping sets codes, I find that the additional size of > > the hash table for hash aggregates is always zero, this seems to be > > incorrect to me, attached a

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-03-12 Thread Justin Pryzby
Thanks for working on this. I have some minor comments. In 0005: + /* Restore the input path (we might have addes Sort on top). */ => added? There's at least two more of the same typo. + /* also ignore already sorted paths */ =>

Re: backend type in log_line_prefix?

2020-03-14 Thread Justin Pryzby
On Fri, Feb 21, 2020 at 10:09:38AM +0100, Peter Eisentraut wrote: > From 75ac8ed0c47801712eb2aa300d9cb29767d2e121 Mon Sep 17 00:00:00 2001 > From: Peter Eisentraut > Date: Thu, 20 Feb 2020 18:16:39 +0100 > Subject: [PATCH v2 3/4] Add backend type to csvlog and optionally > log_line_prefix >

Re: backend type in log_line_prefix?

2020-03-10 Thread Justin Pryzby
On Thu, Feb 13, 2020 at 06:43:32PM +0900, Fujii Masao wrote: > If we do this, backend type should be also included in csvlog? +1, I've been missing that Note, this patch seems to correspond to: b025f32e0b Add leader_pid to pg_stat_activity I had mentioned privately to Julien missing this info

Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*)

2020-03-10 Thread Justin Pryzby
rom 2c4b2c408490ecde3cfb4e336a78942f7a6f8197 Mon Sep 17 00:00:00 2001 From: Justin Pryzby Date: Fri, 27 Dec 2019 23:34:14 -0600 Subject: [PATCH v9 01/11] BUG: in errmsg Note there's two changes here. Should backpatch to v12, where pg_ls_tmpdir was added. --- src/backend/utils/adt/genfile.c | 2 +- 1 f

Re: backend type in log_line_prefix?

2020-03-11 Thread Justin Pryzby
On Tue, Mar 10, 2020 at 02:01:42PM -0500, Justin Pryzby wrote: > On Thu, Feb 13, 2020 at 06:43:32PM +0900, Fujii Masao wrote: > > If we do this, backend type should be also included in csvlog? > > +1, I've been missing that > > Note, this patch seems to correspond to: > b

Re: DETACH PARTITION and FOR EACH ROW triggers on partitioned tables

2020-04-08 Thread Justin Pryzby
On Wed, Apr 08, 2020 at 12:02:39PM -0400, Alvaro Herrera wrote: > On 2020-Apr-08, Justin Pryzby wrote: > > > This seems to be a bug in master, v12, and (probably) v11, where "FOR EACH > > FOR" > > was first allowed on partition tables (86f575948). >

Re: Vacuum o/p with (full 1, parallel 0) option throwing an error

2020-04-08 Thread Justin Pryzby
cases, we should through error. > > Oh, yeah, good point. Somebody must not've been careful enough with > the options-checking code. Actually I think someone was too careful. >From 9256cdb0a77fb33194727e265a346407921055ef Mon Sep 17 00:00:00 2001 From: Justin Pryzby Date: Wed, 8 Apr 20

DETACH PARTITION and FOR EACH ROW triggers on partitioned tables

2020-04-08 Thread Justin Pryzby
is, so I've stopped after implementing a partial fix. >From 2c31cac22178d904ee108b77f316886d1e2f6288 Mon Sep 17 00:00:00 2001 From: Justin Pryzby Date: Fri, 3 Apr 2020 22:43:26 -0500 Subject: [PATCH] WIP: fix detaching tables with inherited triggers --- src/backend/commands/tablecmds.c | 33 +

doc review for v13

2020-04-08 Thread Justin Pryzby
rom 482b590355cd7df327602dd36e91721b827f9c37 Mon Sep 17 00:00:00 2001 From: Justin Pryzby Date: Sun, 29 Mar 2020 19:31:04 -0500 Subject: [PATCH v1 01/19] docs: pg_statistic_ext.stxstattarget commit c31132d87c6315bbbe4b4aa383705aaae2348c0e Author: Tomas Vondra Date: Wed Mar 18 16:48:12 2020 +0100 ---

Re: debian bugrept involving fast default crash in pg11.7

2020-04-09 Thread Justin Pryzby
On Thu, Apr 09, 2020 at 02:36:26PM -0400, Tim Bishop wrote: > SELECT attrelid::regclass, * FROM pg_attribute WHERE atthasmissing; > -[ RECORD 1 ]-+- > attrelid | download > attrelid | 22749 > attname | filetype But that table isn't involved in the crashing query, right ?

<    1   2   3   4   5   6   7   8   9   10   >