Re: pg11+: pg_ls_*dir LIMIT 1: temporary files .. not closed at end-of-transaction

2020-03-29 Thread Justin Pryzby
On Sun, Mar 29, 2020 at 12:37:05PM -0400, Tom Lane wrote: > I wrote: > > Justin Pryzby writes: > >> Maybe we should lstat() the file to determine if it's a dangling link; if > >> lstat() fails, then skip it. Currently, we use stat(), which shows > >> metda

Re: error context for vacuum to include block number

2020-03-30 Thread Justin Pryzby
argument to two functions? -- Justin >From 85672d7f071c91f3ec9190be7feb293f0e49cf8a Mon Sep 17 00:00:00 2001 From: Justin Pryzby Date: Wed, 26 Feb 2020 19:22:55 -0600 Subject: [PATCH v39 1/2] Avoid some calls to RelationGetRelationName --- src/backend/access/heap/vacuumlazy.c | 20 ++-- 1 file c

Re: Allow CLUSTER, VACUUM FULL and REINDEX to change tablespace on the fly

2020-03-30 Thread Justin Pryzby
On Mon, Mar 30, 2020 at 09:02:22PM +0300, Alexey Kondratov wrote: > Hmm, I went through the well known to me SQL commands in Postgres and a bit > more. Parenthesized options list is mostly used in two common cases: There's also ANALYZE(VERBOSE), REINDEX(VERBOSE). There was debate a year ago [0]

Re: Add A Glossary

2020-04-01 Thread Justin Pryzby
On Tue, Mar 31, 2020 at 03:26:02PM -0400, Corey Huinker wrote: > Just so I can prioritize my work, which of these things, along with your > suggestions in previous emails, would you say is a barrier to considering > this ready for a committer? To answer your off-list inquiry, I'm not likely to

Re: control max length of parameter values logged

2020-04-01 Thread Justin Pryzby
Thanks for updating the patch. On Thu, Apr 02, 2020 at 01:29:04AM +0100, Alexey Bashtanov wrote: > +If greater than zero, bind parameter values reported in non-error > +statement-logging messages are trimmed to no more than this many > bytes. > +If this value is specified

Re: Allow continuations in "pg_hba.conf" files

2020-04-01 Thread Justin Pryzby
On Thu, Apr 02, 2020 at 07:25:36AM +0200, Fabien COELHO wrote: > > Hello, > > > FWIW, I don't think so. Generally a trailing backspace is an escape > > character for the following newline. And '\ ' is a escaped space, > > which is usualy menas a space itself. > > > > In this case escape

Re: ALTER tbl rewrite loses CLUSTER ON index

2020-04-02 Thread Justin Pryzby
On Thu, Apr 02, 2020 at 03:14:21PM +0900, Michael Paquier wrote: > Now, regarding patch 0002, note that you have a problem for this part: > -tuple = SearchSysCache1(INDEXRELID, ObjectIdGetDatum(indexOid)); > -if (!HeapTupleIsValid(tuple))/* probably can't happen */ > -

Re: User Interface for WAL usage data

2020-04-01 Thread Justin Pryzby
On Thu, Apr 02, 2020 at 10:13:18AM +0530, Amit Kapila wrote: > In thread [1], we are discussing to expose WAL usage data for each > statement in a way quite similar to how we expose BufferUsage data. > The way it exposes seems reasonable to me and no one else raises any > objection. It could be

Re: User Interface for WAL usage data

2020-04-02 Thread Justin Pryzby
On Thu, Apr 02, 2020 at 11:32:16AM +0530, Amit Kapila wrote: > On Thu, Apr 2, 2020 at 11:28 AM Kyotaro Horiguchi > wrote: > > > > At Thu, 2 Apr 2020 00:41:20 -0500, Justin Pryzby > > wrote in > > > Regarding v10-0004-Add-option-to-report-WAL-usage-in-EX

Re: error context for vacuum to include block number

2020-03-24 Thread Justin Pryzby
On Wed, Mar 25, 2020 at 10:22:21AM +0530, Amit Kapila wrote: > On Wed, Mar 25, 2020 at 10:05 AM Masahiko Sawada > wrote: > > > > On Wed, 25 Mar 2020 at 12:44, Amit Kapila wrote: > > > > > > On Tue, Mar 24, 2020 at 7:51 PM Masahiko Sawada > > > wrote: > > > > > > > > > > > > I got the point. But

Re: error context for vacuum to include block number

2020-03-24 Thread Justin Pryzby
On Wed, Mar 25, 2020 at 01:34:43PM +0900, Masahiko Sawada wrote: > I meant that with the patch, suppose that the table has 100 blocks and > we're truncating it to 50 blocks in RelationTruncate(), the error > context message will be "while truncating relation "aaa.bbb" to 100 > blocks", which is

Re: error context for vacuum to include block number

2020-03-24 Thread Justin Pryzby
- Justin >From 26c57039135896ebf29b96c172d35d869ed1ce69 Mon Sep 17 00:00:00 2001 From: Justin Pryzby Date: Thu, 12 Dec 2019 20:54:37 -0600 Subject: [PATCH v32 1/3] Introduce vacuum errcontext to display additional information. The additional information displayed will be block number for error occ

Re: error context for vacuum to include block number

2020-03-25 Thread Justin Pryzby
es instead of > > > new_rel_pages to indicate the remaining pages after truncation? > > > > Yea, I think that addresses the issue. Attached patch addressing these. -- Justin >From b3e112c3d02982b4c050a43c53e47a868879c561 Mon Sep 17 00:00:00 2001 From: Justin Pryzby Date:

Re: error context for vacuum to include block number

2020-03-25 Thread Justin Pryzby
On Wed, Mar 25, 2020 at 04:54:41PM +0530, Amit Kapila wrote: > On Wed, Mar 25, 2020 at 3:42 PM Justin Pryzby wrote: > > > > Attached patch addressing these. > > > > Thanks, you forgot to remove the below declaration which I have > removed in attached. Yes I saw..

Re: error context for vacuum to include block number

2020-03-26 Thread Justin Pryzby
On Thu, Mar 26, 2020 at 08:56:54PM +0900, Masahiko Sawada wrote: > 1. > @@ -1844,9 +1914,15 @@ lazy_vacuum_page(Relation onerel, BlockNumber > blkno, Buffer buffer, > int uncnt = 0; > TransactionId visibility_cutoff_xid; > boolall_frozen; > + LVRelStats olderrcbarg;

Re: Autovacuum on partitioned table (autoanalyze)

2020-04-25 Thread Justin Pryzby
On Wed, Mar 18, 2020 at 11:30:39AM -0500, Justin Pryzby wrote: > In the past, I think there's was talk that maybe someone would invent a clever > way to dynamically combine all the partitions' statistics, so analyzing the > parent wasn't needed. [...] I happened across the thread I was

Re: doc review for v13

2020-04-27 Thread Justin Pryzby
On Mon, Apr 27, 2020 at 03:03:05PM +0900, Michael Paquier wrote: > Hm, okay. There are still pieces in those patches about which I am > not sure, so I have let that aside for the time being. > > Anyway, I have applied patch 12, and reported the typos from imath.c Thank you. I will leave this

Re: weird hash plan cost, starting with pg10

2020-04-27 Thread Justin Pryzby
On Mon, Apr 27, 2020 at 12:26:03PM -0400, Tom Lane wrote: > Justin Pryzby writes: > > Checking if you're planning to backpatch this ? > > Are you speaking of 5c27bce7f et al? Oops, yes, thanks. I updated wiki/PostgreSQL_13_Open_Items just now. -- Justin

Re: weird hash plan cost, starting with pg10

2020-04-27 Thread Justin Pryzby
On Fri, Apr 10, 2020 at 04:11:27PM -0400, Tom Lane wrote: > I'm not sure it's worth any risk though. A much simpler > fix is to make sure we clear the dangling hashtable pointer, as in > 0002 below (a simplified form of Konstantin's patch). The net > effect of that is that in the case where a

Re: [PATCH'] Variables assigned with values that is never used.

2020-04-24 Thread Justin Pryzby
On Sat, Mar 28, 2020 at 10:33:23AM -0300, Ranier Vilela wrote: > Theses variables, are assigned with values that never is used and, can > safely have their values removed. I came across this one recently. commit ccf85a5512fe7cfd76c6586b67fe06d911428d34 Author: Justin Pryzby Date: Thu

Re: doc review for v13

2020-04-26 Thread Justin Pryzby
On Tue, Apr 14, 2020 at 02:47:54PM +0900, Michael Paquier wrote: > On Sun, Apr 12, 2020 at 04:35:45PM -0500, Justin Pryzby wrote: > > Added a few more. > > And rebased on top of dbc60c5593f26dc777a3be032bff4fb4eab1ddd1 > > Thanks for the patch set, I have applied the mos

Re: Allow CLUSTER, VACUUM FULL and REINDEX to change tablespace on the fly

2020-04-26 Thread Justin Pryzby
On Sat, Apr 11, 2020 at 08:33:52PM -0500, Justin Pryzby wrote: > > That's the last known issue with the patch. I doubt anyone will elect to > > pick > > it up in the next 8 hours, but I think it's in very good shape for v14 :) > > I tweaked some comments and docs a

Re: WAL usage calculation patch

2020-04-21 Thread Justin Pryzby
On Wed, Apr 22, 2020 at 09:15:08AM +0530, Amit Kapila wrote: > > > > And add the acronym to the docs: > > > > > > > > $ git grep 'full page' '*/explain.sgml' > > > > doc/src/sgml/ref/explain.sgml: number of records, number of full > > > > page writes and amount of WAL bytes > > > > > > > >

Re: DETACH PARTITION and FOR EACH ROW triggers on partitioned tables

2020-04-21 Thread Justin Pryzby
On Tue, Apr 21, 2020 at 07:03:30PM -0400, Alvaro Herrera wrote: > On 2020-Apr-20, Justin Pryzby wrote: > > > On Mon, Apr 20, 2020 at 06:35:44PM +0900, Amit Langote wrote: > > > > Also, how about, for consistency, making the parent table labeling of > &

Re: Cleanup/remove/update references to OID column

2020-04-29 Thread Justin Pryzby
Few comments seem to have dangling references to the behavior from pre-12 "WITH OIDS". Maybe varsup.c should get a wider change? diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 1e743d7d86..ce84e22cbd 100644 --- a/src/backend/access/common/tupdesc.c

Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*)

2020-05-02 Thread Justin Pryzby
nts to pg_ls_* functions (including pg_ls_tmpdir but not |pg_ls_dir). | - maybe add pg_ls_dir_recurse, which satisfies the original need; | - retire pg_ls_dir (does this work with tuplestore?) | - profit -- Justin >From 6175cecd312296bbb9099834d91ffaa50e059f6c Mon Sep 17 00:00:00 2001 From: Ju

Re: PG 13 release notes, first draft

2020-05-11 Thread Justin Pryzby
On Mon, May 11, 2020 at 10:52:41AM +0530, Amit Kapila wrote: > > 1. We have allowed an (auto)vacuum to display additional information > > about heap or index in case of an error in commit b61d161c14 [1]. > > Now, in general, it might not be worth saying much about error > > information but I

Re: PG 13 release notes, first draft

2020-05-11 Thread Justin Pryzby
|Allow function call backtraces of errors to be logged (Peter Eisentraut, Álvaro Herrera) |Server variable backtrace_functions specifies which C functions should generate backtraces on error. I think the details in the description are eclipsing the most important thing: backtraces on Assert().

Re: PG 13 release notes, first draft

2020-05-05 Thread Justin Pryzby
Do you want to include any of these? 5823677acc Provide pgbench --show-script to dump built-in scripts. ce8f946764 Report the time taken by pgbench initialization steps. 751c63cea0 Remove pg_regress' --load-language option. 33753ac9d7 Add object names to partition integrity violations. 246f136e76

Re: PG 13 release notes, first draft

2020-05-05 Thread Justin Pryzby
On Tue, May 05, 2020 at 01:18:09PM -0400, Bruce Momjian wrote: > > |Release date: 2020-05-03 > > => Should say 2020-XX-XX, before someone like me goes and installs it > > everywhere in sight. > > Agreed! > > > |These triggers cannot change the destination partition. > > => Maybe say "cannot

Re: PG 13 release notes, first draft

2020-05-05 Thread Justin Pryzby
|Release date: 2020-05-03 => Should say 2020-XX-XX, before someone like me goes and installs it everywhere in sight. |These triggers cannot change the destination partition. => Maybe say "cannot change which partition is the destination" |Allow incremental sorting (James Coleman, Alexander

Re: PG 13 release notes, first draft

2020-05-05 Thread Justin Pryzby
On Tue, May 05, 2020 at 02:10:24PM -0400, Bruce Momjian wrote: > > > > | This is controlled by GUC wal_skip_threshold. > > > > I think you should say that's a size threshold which determines which > > > > strategy > > > > to use (WAL or fsync). > > > > > > I went with: > > > The WAL write

Re: PG 13 release notes, first draft

2020-05-06 Thread Justin Pryzby
On Wed, May 06, 2020 at 07:35:34PM -0400, Bruce Momjian wrote: > On Wed, May 6, 2020 at 11:17:54AM +0500, Andrey M. Borodin wrote: > > I'm not sure, but probably it worth mentioning in "General performance" > > section that TOAST (and everything pglz-compressed) decompression should be > >

Re: PG 13 release notes, first draft

2020-05-05 Thread Justin Pryzby
On Tue, May 05, 2020 at 05:34:26PM -0400, Bruce Momjian wrote: > On Tue, May 5, 2020 at 02:37:16PM -0400, Bruce Momjian wrote: > > On Tue, May 5, 2020 at 12:50:11PM -0500, Justin Pryzby wrote: > > > Do you want to include any of these? > > > > > > 5823

Re: PG 13 release notes, first draft

2020-05-05 Thread Justin Pryzby
On Tue, May 05, 2020 at 03:44:37PM -0400, Alvaro Herrera wrote: > On 2020-May-05, Bruce Momjian wrote: > > On Tue, May 5, 2020 at 12:50:11PM -0500, Justin Pryzby wrote: > > > Do you want to include any of these? > > > > > > 5823677acc Provide pgbench --

Re: should INSERT SELECT use a BulkInsertState?

2020-05-10 Thread Justin Pryzby
On Fri, May 08, 2020 at 02:25:45AM -0500, Justin Pryzby wrote: > Seems to me it should, at least conditionally. At least if there's a function > scan or a relation or .. > > I mentioned a bit about our use-case here: > https://www.postgresql.org/message-id/20200219173742.GA30939%

Re: PG 13 release notes, first draft (ltree dot star)

2020-05-10 Thread Justin Pryzby
> In ltree, when using adjacent asterisks with braces, e.g. "*{2}.*{3}", > properly interpret that as "*{5}" (Nikita Glukhov) I think that should say ".*" not "*", as in: > In ltree, when using adjacent asterisks with braces, e.g. ".*{2}.*{3}", > properly interpret that as "*{5}" (Nikita

Re: BUG #16147: postgresql 12.1 (from homebrew) - pg_restore -h localhost --jobs=2 crashes

2020-05-18 Thread Justin Pryzby
On Thu, Mar 05, 2020 at 07:53:35PM -0800, David Zhang wrote: > I can reproduce this pg_restore crash issue (pg_dump crash too when running > with multiple jobs) on MacOS 10.14 Mojave and MacOS 10.15 Catalina using > following steps. Isn't this the same as here?

Re: Warn when parallel restoring a custom dump without data offsets

2020-05-19 Thread Justin Pryzby
On Sat, May 16, 2020 at 04:57:46PM -0400, David Gilman wrote: > If pg_dump can't seek on its output stream when writing a dump in the > custom archive format (possibly because you piped its stdout to a file) > it can't update that file with data offsets. These files will often > break parallel

Re: PG 13 release notes, first draft

2020-05-14 Thread Justin Pryzby
On Thu, May 14, 2020 at 11:01:51PM +0200, Peter Eisentraut wrote: > On 2020-05-12 02:41, Justin Pryzby wrote: > > I'm not opposed to including it, but I think it's still true that the user > > doesn't need to know in advance that the error message will be additionally > >

Re: Add A Glossary

2020-05-14 Thread Justin Pryzby
On Thu, May 14, 2020 at 08:00:17PM -0400, Alvaro Herrera wrote: > + ACID > + > + > + Atomicity, > + consistency, > + isolation, and > + durability. > + A set of properties of database transactions intended to guarantee > validity > + in concurrent operation and

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-05-09 Thread Justin Pryzby
Apr 18, 2020 at 10:36 PM Justin Pryzby wrote: > > > > On Tue, Apr 07, 2020 at 10:53:05AM -0500, Justin Pryzby wrote: > > > On Tue, Apr 07, 2020 at 08:40:30AM -0400, James Coleman wrote: > > > > > And, should it use two spaces before "Sort Method", "

Re: 2020-05-14 Press Release Draft

2020-05-11 Thread Justin Pryzby
On Sun, May 10, 2020 at 10:08:46PM -0400, Jonathan S. Katz wrote: > * Ensure that a detatched partition has triggers that come from its former > parent removed. I would have said: "fix for issue which prevented/precluded detaching partitions which have inherited ROW triggers" > * Several fixes

Re: Auxiliary Processes and MyAuxProc

2020-03-18 Thread Justin Pryzby
On Wed, Mar 18, 2020 at 09:22:58AM -0400, Mike Palmiotto wrote: > On Tue, Mar 17, 2020 at 9:04 PM Alvaro Herrera > wrote: > > > > On 2020-Mar-17, Justin Pryzby wrote: > > > > > +static PgSubprocess process_types[] = { > > > +

Re: Autovacuum on partitioned table (autoanalyze)

2020-03-18 Thread Justin Pryzby
Regarding this patch: +* the ANALYZE message as it resets the partition's changes_since_analze => analyze +* If the relation is a partitioned table, we must add up children's childrens' The approach in general: I see an issue for timeseries data, where only the most recent

Re: expose parallel leader in CSV and log_line_prefix

2020-03-18 Thread Justin Pryzby
On Sun, Mar 15, 2020 at 12:49:33PM +0100, Julien Rouhaud wrote: > On Sun, Mar 15, 2020 at 06:18:31AM -0500, Justin Pryzby wrote: > > See also: > > https://commitfest.postgresql.org/27/2390/ > > https://www.postgresql.org/message-id/flat/caobau_yy5bt0vtpz2_lum6cucg

Re: Berserk Autovacuum (let's save next Mandrill)

2020-03-19 Thread Justin Pryzby
On Fri, Mar 13, 2020 at 02:38:51PM -0700, Andres Freund wrote: > On 2020-03-13 13:44:42 -0500, Justin Pryzby wrote: > > Having now played with the patch, I'll suggest that 1000 is too high a > > threshold. If autovacuum runs without FREEZE, I don't see why it couldn't > &

Re: error context for vacuum to include block number

2020-03-18 Thread Justin Pryzby
phase information correctly so that it doesn't display the wrong info > > in the context in an error message. > > Justin, are you planning to work on the pending comments? If you > want, I can try to fix some of these. We have less time left for this > CF, so we need to do

Re: Memory-Bounded Hash Aggregation

2020-03-19 Thread Justin Pryzby
On Sun, Mar 15, 2020 at 04:05:37PM -0700, Jeff Davis wrote: > > + if (from_tape) > > + partition_mem += HASHAGG_READ_BUFFER_SIZE; > > + partition_mem = npartitions * HASHAGG_WRITE_BUFFER_SIZE; > > > > => That looks wrong ; should say += ? > > Good catch! Fixed. > +++

Re: Berserk Autovacuum (let's save next Mandrill)

2020-03-19 Thread Justin Pryzby
On Thu, Mar 19, 2020 at 09:52:11PM +1300, David Rowley wrote: > On Thu, 19 Mar 2020 at 19:07, Justin Pryzby wrote: > > > > On Fri, Mar 13, 2020 at 02:38:51PM -0700, Andres Freund wrote: > > > On 2020-03-13 13:44:42 -0500, Justin Pryzby wrote: > > > > Having now

Re: ALTER tbl rewrite loses CLUSTER ON index

2020-03-17 Thread Justin Pryzby
On Tue, Mar 17, 2020 at 02:33:32PM +0900, Michael Paquier wrote: > > Yeah, in cluster(), mark_index_clustered(). > > Patch 0002 from Justin does that, I would keep this refactoring as > HEAD-only material though, and I don't spot any other code paths in > need of patching. > > The commit message

Re: Berserk Autovacuum (let's save next Mandrill)

2020-03-17 Thread Justin Pryzby
On Tue, Mar 17, 2020 at 01:14:02AM +0100, Laurenz Albe wrote: > lazy_check_needs_freeze() is only called for an aggressive vacuum, which > this isn't. > --- a/src/backend/access/heap/vacuumlazy.c > +++ b/src/backend/access/heap/vacuumlazy.c > @@ -1388,17 +1388,26 @@ lazy_scan_heap(Relation

Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*)

2020-03-17 Thread Justin Pryzby
_ls". I think that's a good idea, except it doesn't fit with what the code does: AllocDir() and ReadDir(). Instead, use pg_stat_file() for that. Hm, I realized that the existing pg_ls_dir_metadata was skipping links to dirs, since !ISREG(). So changed to use both stat() and lstat(). -- J

Re: Berserk Autovacuum (let's save next Mandrill)

2020-03-17 Thread Justin Pryzby
On Mon, Mar 16, 2020 at 07:47:13AM -0500, Justin Pryzby wrote: > Normally, when someone complains about bad plan related to no index-onlyscan, > we tell them to run vacuum, and if that helps, then ALTER TABLE .. SET > (autovacuum_vacuum_scale_factor=0.005). > > If there's two thr

Re: Berserk Autovacuum (let's save next Mandrill)

2020-03-17 Thread Justin Pryzby
On Tue, Mar 17, 2020 at 08:42:07PM +0100, Laurenz Albe wrote: > Also, since aggressive^H^H^H^H^H^H^H^H^H^Hproactive freezing seems to be a > performance problem in some cases (pages with UPDATEs and DELETEs in otherwise > INSERT-mostly tables), I have done away with the whole freezing thing, >

Re: Auxiliary Processes and MyAuxProc

2020-03-17 Thread Justin Pryzby
On Tue, Mar 17, 2020 at 02:50:19PM -0400, Mike Palmiotto wrote: > The patchset is now split out. I've just noticed that Peter Eisentraut > included some changes for a generic MyBackendType, which I should have > been aware of. I was unable to rebase due to these changes, but can > fold these

Re: Berserk Autovacuum (let's save next Mandrill)

2020-03-17 Thread Justin Pryzby
On Tue, Mar 17, 2020 at 10:01:15PM +0100, Laurenz Albe wrote: > On Tue, 2020-03-17 at 14:56 -0500, Justin Pryzby wrote: > > I still suggest scale_factor maximum of 1e10, like > > 4d54543efa5eb074ead4d0fadb2af4161c943044 > > > > Which alows more effectively disabling it

Re: Berserk Autovacuum (let's save next Mandrill)

2020-03-17 Thread Justin Pryzby
On Tue, Mar 17, 2020 at 10:22:44PM +0100, Laurenz Albe wrote: > On Tue, 2020-03-17 at 16:07 -0500, Justin Pryzby wrote: > > > Assume a scale factor >= 1, for example 2, and n live tuples. > > > The table has just been vacuumed. > > > > > > Now

Re: control max length of parameter values logged

2020-03-18 Thread Justin Pryzby
On Sun, Mar 15, 2020 at 08:48:33PM -0300, Alvaro Herrera wrote: > On 2020-Mar-14, Tom Lane wrote: > > > Bruce Momjian writes: > > > I am sorry --- I am confused. Why are we truncating or allowing control > > > of truncation of BIND parameter values, but have no such facility for > > > queries.

Re: Berserk Autovacuum (let's save next Mandrill)

2020-03-17 Thread Justin Pryzby
On Tue, Mar 17, 2020 at 09:58:53PM -0400, James Coleman wrote: > On Tue, Mar 17, 2020 at 9:03 PM Andres Freund wrote: > > > > On 2020-03-17 20:42:07 +0100, Laurenz Albe wrote: > > > > I think Andres was thinking this would maybe be an optimization > > > > independent of > > > > is_insert_only

Re: error context for vacuum to include block number

2020-03-19 Thread Justin Pryzby
g, which is in the two routines handling indexes. It's probably a good idea to pass the indname rather than the relation in any case. I rebased the rest of my patches on top of yours. -- Justin >From a1ef4498cf93a9971be5c1683ceb62879ab9bd17 Mon Sep 17 00:00:00 2001 From: Justin Pryzby Date: Thu,

Re: error context for vacuum to include block number

2020-03-19 Thread Justin Pryzby
On Thu, Mar 19, 2020 at 03:29:31PM -0500, Justin Pryzby wrote: > I was going to suggest that we could do that by passing in a pointer to a > local > variable "LVRelStats olderrcbarg", like: > |update_vacuum_error_cbarg(vacrelstats, VACUUM

Re: explain HashAggregate to report bucket and memory stats

2020-03-20 Thread Justin Pryzby
: 262144 Batches: 1 Memory Usage: 9080kB | -> Seq Scan on t b (cost=0.00..3769.99 rows=199999 width=4) (actual time=3.273..40.163 rows=19 loops=1) -- Justin >From e593c119c97ea31edac4c9f08a39eee451964a16 Mon Sep 17 00:00:00 2001 From: Justin Pryzby Date: Thu, 19 Mar 20

Re: error context for vacuum to include block number

2020-03-20 Thread Justin Pryzby
On Fri, Mar 20, 2020 at 11:24:25AM +0530, Amit Kapila wrote: > On Fri, Mar 20, 2020 at 5:59 AM Justin Pryzby wrote: > That makes sense. I have a few more comments: > > 1. > + VACUUM_ERRCB_PHASE_INDEX_CLEANUP, > +} errcb_phase; > > Why do you need a comma after the l

Re: improve transparency of bitmap-only heap scans

2020-03-19 Thread Justin Pryzby
On Mon, Mar 16, 2020 at 09:08:36AM -0400, James Coleman wrote: > Does the original optimization cover parallel bitmap heap scans like this? It works for parallel bitmap only scans. template1=# explain analyze select count(*) from exp where a between 25 and 35 and d between 5 and 10; Finalize

Re: error context for vacuum to include block number

2020-03-20 Thread Justin Pryzby
On Fri, Mar 20, 2020 at 04:58:08PM +0530, Amit Kapila wrote: > See, how the attached looks? I have written a commit message as well, > see if I have missed anyone is from the credit list? Thanks for looking again. Couple tweaks: +/* Phases of vacuum during which an error can occur. */ Can you

Re: Add A Glossary

2020-03-20 Thread Justin Pryzby
On Thu, Mar 19, 2020 at 09:11:22PM -0300, Alvaro Herrera wrote: > +Aggregate > + > + > + To combine a collection of data values into a single value, whose > + value may not be of the same type as the original values. > + Aggregate Functions > + combine multiple

Re: Add A Glossary

2020-03-20 Thread Justin Pryzby
On Fri, Mar 20, 2020 at 11:32:25PM +0100, Jürgen Purtz wrote: > > > + > > > +File Segment > > > + > > > + > > > + If a heap or index file grows in size over 1 GB, it will be split > > 1GB is the default "segment size", which you should define. > > ??? "A <> or other

Re: error context for vacuum to include block number

2020-03-21 Thread Justin Pryzby
On Sat, Mar 21, 2020 at 01:00:03PM +0530, Amit Kapila wrote: > I have addressed your comments in the attached patch. Today, while > testing error messages from various phases, I noticed that the patch > fails to display error context if the error occurs during the truncate > phase. The reason

Re: Why does [auto-]vacuum delay not report a wait event?

2020-03-20 Thread Justin Pryzby
feb0c68afad310e3f52c21c3cdbaf Mon Sep 17 00:00:00 2001 From: Justin Pryzby Date: Fri, 20 Mar 2020 20:47:30 -0500 Subject: [PATCH v1 1/2] Report wait event for cost-based vacuum delay --- doc/src/sgml/monitoring.sgml| 2 ++ src/backend/commands/vacuum.c | 2 ++ src/backend/postmaster/pgstat.

Re: Add A Glossary

2020-03-21 Thread Justin Pryzby
On Sat, Mar 21, 2020 at 03:08:30PM +0100, Jürgen Purtz wrote: > On 21.03.20 00:03, Justin Pryzby wrote: > > > > > + > > > > > +Host > > > > > + > > > > > + > > > > > + See Server. > > >

doc review for parallel vacuum

2020-03-21 Thread Justin Pryzby
ndexes. -1 indicates parallel vacuum is * disabled. */ int nworkers; -- 2.17.0 >From aec387f1c5e405d504ade077a20db7b6ff6c3835 Mon Sep 17 00:00:00 2001 From: Justin Pryzby Date: Sun, 19 Jan 2020 22:35:01 -0600 Subject: [PATCH v1 1/2] docs review for

Re: kill_prior_tuple and index scan costing

2020-03-21 Thread Justin Pryzby
On Sat, Mar 21, 2020 at 07:33:02PM -0700, Andres Freund wrote: > While your recent btree work ensures that we get the heap tids for an > equality lookup in heap order (right?), I think when I tested the TID tiebreaker patch, it didn't help for our case, which is for inequality: (timestamptz >=

Re: Why does [auto-]vacuum delay not report a wait event?

2020-03-22 Thread Justin Pryzby
On Sat, Mar 21, 2020 at 05:24:57PM -0700, Andres Freund wrote: > > Also, I noticed that SLEEP_ON_ASSERT comment (31338352b) wants to use > > pg_usleep > > "which seems too short.". Surely it should use pg_sleep, added at > > 782eefc58 a > > few years later ? > > I don't see problem with using

Re: Database recovery from tablespace only

2020-03-22 Thread Justin Pryzby
On Sat, Mar 21, 2020 at 05:48:03PM -0600, Phillip Black wrote: > Hey Hackers, Hi, This list is for development and bug reports. I think you'll want a professional support contract, for postgres or for generic data recovery. https://www.postgresql.org/support/professional_support/ -- Justin

Re: error context for vacuum to include block number

2020-03-23 Thread Justin Pryzby
On Mon, Mar 23, 2020 at 04:39:54PM +0900, Masahiko Sawada wrote: > I've already commented on earlier patch but I personally think we'd be > better to report freespace map vacuum as a separate phase. The > progress report of vacuum command is used to know the progress but > this error context would

Re: RFC: Add 'taint' field to pg_control.

2020-03-23 Thread Justin Pryzby
On Wed, Feb 28, 2018 at 04:16:53PM -0600, Justin Pryzby wrote: > On Wed, Feb 28, 2018 at 01:43:11PM -0800, Andres Freund wrote: > > a significant number of times during investigations of bugs I wondered > > whether running the cluster with various settings, or various tools >

Re: ALTER INDEX fails on partitioned index

2020-03-23 Thread Justin Pryzby
On Thu, Feb 27, 2020 at 09:11:14PM -0300, Alvaro Herrera wrote: > On 2020-Feb-27, Justin Pryzby wrote: > > The attached allows CREATE/ALTER to specify reloptions on a partitioned > > table > > which are used as defaults for future children. > > > > I think

Re: improve transparency of bitmap-only heap scans

2020-03-24 Thread Justin Pryzby
On Tue, Mar 24, 2020 at 10:54:05AM +0530, Amit Kapila wrote: > On Fri, Mar 20, 2020 at 7:09 AM James Coleman wrote: > > > > Awesome, thanks for confirming with an actual plan. > > > > > I don't think it matters in nontext mode, but at least in text mode, I > > > think > > > maybe the Unfetched

Re: weird hash plan cost, starting with pg10

2020-03-24 Thread Justin Pryzby
On Mon, Mar 23, 2020 at 01:50:59PM -0300, Alvaro Herrera wrote: > While messing with EXPLAIN on a query emitted by pg_dump, I noticed that > current Postgres 10 emits weird bucket/batch/memory values for certain > hash nodes: > > -> Hash (cost=0.11..0.11 rows=10

Re: error context for vacuum to include block number

2020-03-24 Thread Justin Pryzby
On Tue, Mar 24, 2020 at 07:07:03PM +0530, Amit Kapila wrote: > On Tue, Mar 24, 2020 at 6:18 PM Masahiko Sawada > wrote: > > 1. > > +/* Update error traceback information */ > > +olderrcbarg = *vacrelstats; > > +update_vacuum_error_cbarg(vacrelstats, > > +

Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*)

2020-03-07 Thread Justin Pryzby
dir_files(..., true, DIR_HIDE);". > Alas, ISTM that there are no tests on any of these functions:-( Addressed these. And reordered the last two commits to demonstrate and exercize the behavior change in regress test. -- Justin >From a5b9a03445d1c768662cafebd8ab3bd7a62890aa Mon Sep 17 00

Re: pg11+: pg_ls_*dir LIMIT 1: temporary files .. not closed at end-of-transaction

2020-03-16 Thread Justin Pryzby
On Mon, Mar 16, 2020 at 09:38:50PM -0400, Tom Lane wrote: > Justin Pryzby writes: > > v2 attached - I will add to next CF in case you want to defer it until > > later. > > Thanks, reviewed and pushed. Since this is a bug fix (at least in part) > I didn't want to wai

Re: error context for vacuum to include block number

2020-03-16 Thread Justin Pryzby
On Tue, Mar 03, 2020 at 10:05:42PM +0900, Masahiko Sawada wrote: > I was concerned about fsm vacuum; vacuum error context might show heap > scan while actually doing fsm vacuum. But perhaps we can update > callback args for that. That would be helpful for user to distinguish > that the problem

Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*)

2020-03-16 Thread Justin Pryzby
use lstat ... and squished. -- Justin >From d8294c4747c5ba1f3bec858c137cc2d31e5a0425 Mon Sep 17 00:00:00 2001 From: Justin Pryzby Date: Mon, 16 Mar 2020 14:12:55 -0500 Subject: [PATCH v13 1/8] Document historic behavior of links to directories.. Backpatch to 9.5: pg_stat_file --- doc/src/sgml/f

Re: Berserk Autovacuum (let's save next Mandrill)

2020-03-16 Thread Justin Pryzby
On Mon, Mar 16, 2020 at 08:49:43PM +0100, Laurenz Albe wrote: > On Mon, 2020-03-16 at 07:47 -0500, Justin Pryzby wrote: > > It seems to me that the easy thing to do is to implement this initially > > without > > FREEZE (which is controlled by vacuum_freeze_table_age), and def

Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*)

2020-03-16 Thread Justin Pryzby
- Justin >From 1bb8e0efb4f14fa344cd5ee66c3138184a9fa9e2 Mon Sep 17 00:00:00 2001 From: Justin Pryzby Date: Fri, 6 Mar 2020 16:50:07 -0600 Subject: [PATCH v12 01/11] Document historic behavior about hiding directories and special files Should backpatch to v10: tmpdir, waldir and archive_stat

Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*)

2020-05-07 Thread Justin Pryzby
Rebased onto 1ad23335f36b07f4574906a8dc66a3d62af7c40c >From 698026f6365a324bf52c5985d549f71b52ada567 Mon Sep 17 00:00:00 2001 From: Justin Pryzby Date: Mon, 16 Mar 2020 14:12:55 -0500 Subject: [PATCH v17 01/10] Document historic behavior of links to directories.. Backpatch to 9.5: pg_stat_f

should INSERT SELECT use a BulkInsertState?

2020-05-08 Thread Justin Pryzby
ing large fractions of buffer cache and leaving it around for other backends to clean up. commit 7f9e061363e58f30eee08a0e46f637bf137b Author: Justin Pryzby Date: Fri May 8 02:17:32 2020 -0500 Make INSERT SELECT use a BulkInsertState diff --git a/src/backend/executor/nodeModifyTable.c b/

Re: PostgreSQL 13 Beta 1 Release Announcement Draft

2020-05-20 Thread Justin Pryzby
On Wed, May 20, 2020 at 06:11:08PM -0400, Jonathan S. Katz wrote: > This release includes more ways to monitor actibity within a PostgreSQL activity > partition its "accounts" table, making it easier to benchmark workloads that > contains partitions. contain No need to respond :) Thanks,

Re: Warn when parallel restoring a custom dump without data offsets

2020-05-19 Thread Justin Pryzby
I started fooling with this at home while our ISP is broke (pardon my brevity). Maybe you also saw commit b779ea8a9a2dc3a089b3ac152b1ec4568bfeb26f "Fix pg_restore so parallel restore doesn't fail when the input file doesn't contain data offsets (which it won't, if pg_dump thought its output

Re: Default gucs for EXPLAIN

2020-05-23 Thread Justin Pryzby
On Sat, May 23, 2020 at 11:14:05AM +0200, Vik Fearing wrote: > Here is a patch to provide default gucs for EXPLAIN options. > > I have two goals with this patch. The first is that I personally > *always* want BUFFERS turned on, so this would allow me to do it without > typing it every time. > >

Re: Warn when parallel restoring a custom dump without data offsets

2020-05-23 Thread Justin Pryzby
On Sat, May 23, 2020 at 03:54:30PM -0400, David Gilman wrote: > I've rounded this patch out with a test and I've set up the commitfest > website for this thread. The latest patches are attached and I think > they are ready for review. Thanks. https://commitfest.postgresql.org/28/2568/ I'm not

Re: v13: show extended stats target in \d

2020-09-05 Thread Justin Pryzby
On Tue, Sep 01, 2020 at 05:08:25PM -0400, Alvaro Herrera wrote: > +1 on fixing this, since the ability to change stats target is new in > pg13. > > On 2020-Aug-31, Justin Pryzby wrote: > > > Maybe it should have a comma, like ", STATISTICS %s"? > > It doe

Re: proposal: possibility to read dumped table's name from file

2020-09-03 Thread Justin Pryzby
On Sun, Jul 05, 2020 at 10:08:09PM +0200, Pavel Stehule wrote: > st 1. 7. 2020 v 23:24 odesílatel Justin Pryzby napsal: > > > On Thu, Jun 11, 2020 at 09:36:18AM +0200, Pavel Stehule wrote: > > > st 10. 6. 2020 v 0:30 odesílatel Justin Pryzby > > > > napsal: >

Re: Multivariate MCV list vs. statistics target

2020-09-05 Thread Justin Pryzby
I think the docs are inconsistent with the commit message and the code (d06215d03) and docs should be corrected, soemthing like so: diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b135c89005..cd10a6a6fc 100644 --- a/doc/src/sgml/catalogs.sgml +++

Re: v13: CLUSTER segv with wal_level=minimal and parallel index creation

2020-09-06 Thread Justin Pryzby
This is easily reproduced, at least on pg_attribute [pryzbyj@localhost ~]$ /usr/pgsql-13/bin/initdb -D pgsql13.dat [pryzbyj@localhost ~]$ /usr/pgsql-13/bin/postgres -D pgsql13.dat -c logging_collector=off -c port=5678 -c unix_socket_directories=/tmp -c wal-level=minimal -c max_wal_senders=0&

v13: CLUSTER segv with wal_level=minimal and parallel index creation

2020-09-06 Thread Justin Pryzby
Following a bulk load, a CLUSTER command run by a maintenance script crashed. This is currently reproducible on that instance, so please suggest if I can provide more info. < 2020-09-06 15:44:16.369 MDT >LOG: background worker "parallel worker" (PID 2576) was terminated by signal 6: Aborted <

Re: Allow CLUSTER, VACUUM FULL and REINDEX to change tablespace on the fly

2020-09-03 Thread Justin Pryzby
On Wed, Sep 02, 2020 at 06:07:06PM -0500, Justin Pryzby wrote: > On my side, I've also rearranged function parameters to make the diff more > readable. And squishes your changes into the respective patches. This resolves a breakage I failed to notice from a last-minute edit. And squish

Re: v13: show extended stats target in \d

2020-09-06 Thread Justin Pryzby
On Sun, Sep 06, 2020 at 01:06:05PM -0700, Peter Geoghegan wrote: > On Tue, Sep 1, 2020 at 2:08 PM Alvaro Herrera > wrote: > > It does need some separator. Maybe a comma is sufficient, but I'm not > > sure: that will fail when we add cross-relation stats, because the > > FROM clause will have

<    1   2   3   4   5   6   7   8   9   10   >