Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?)

2021-10-01 Thread Melanie Plageman
tializing backend activity state in the bootstrap process and by doing so, I don't have to do the check if (beentry) in pgstat_inc_ioop() --which happens on most buffer accesses. From ee11056ad25a095593ba2acc2dc8ff31f4ceb9ab Mon Sep 17 00:00:00 2001 From: Melanie Plageman Date: Wed, 29 Sep 2021

Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?)

2021-09-30 Thread Melanie Plageman
On Wed, Sep 29, 2021 at 4:46 PM Melanie Plageman wrote: > > On Mon, Sep 27, 2021 at 2:58 PM Melanie Plageman > wrote: > > > > On Fri, Sep 24, 2021 at 5:58 PM Melanie Plageman > > wrote: > > > > > > On Thu, Sep 23, 2021 at 5:05 PM Melanie Plageman

Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?)

2021-09-29 Thread Melanie Plageman
On Mon, Sep 27, 2021 at 2:58 PM Melanie Plageman wrote: > > On Fri, Sep 24, 2021 at 5:58 PM Melanie Plageman > wrote: > > > > On Thu, Sep 23, 2021 at 5:05 PM Melanie Plageman > > wrote: > > The only remaining TODOs are described in the commit message. > &

Re: Avoiding smgrimmedsync() during nbtree index builds

2021-09-29 Thread Melanie Plageman
On Mon, May 3, 2021 at 5:24 PM Melanie Plageman wrote: > On Thu, Jan 21, 2021 at 5:51 PM Andres Freund wrote: > > On 2021-01-21 23:54:04 +0200, Heikki Linnakangas wrote: > > > On 21/01/2021 22:36, Andres Freund wrote: > > > > > > > > Thinking through

Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?)

2021-09-27 Thread Melanie Plageman
On Fri, Sep 24, 2021 at 5:58 PM Melanie Plageman wrote: > > On Thu, Sep 23, 2021 at 5:05 PM Melanie Plageman > wrote: > The only remaining TODOs are described in the commit message. > most critical one is that the reset message doesn't work. v10 is attached with updated co

Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?)

2021-09-24 Thread Melanie Plageman
On Thu, Sep 23, 2021 at 5:05 PM Melanie Plageman wrote: > > The attached v8 patchset is rewritten to add in an additional dimension > -- buffer type. Now, a backend keeps track of how many buffers of a > particular type (e.g. shared, local) it has accessed in a particular way > (e.

Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?)

2021-09-23 Thread Melanie Plageman
On Tue, Sep 14, 2021 at 9:30 PM Alvaro Herrera wrote: > > On 2021-Sep-13, Melanie Plageman wrote: > > > I also think it makes sense to rename the pg_stat_buffer_actions view to > > pg_stat_buffers and to name the columns using both the buffer action > > type and

Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?)

2021-09-13 Thread Melanie Plageman
17bf27ad0a6ae54a6a898e96c630d36867e9d943 Mon Sep 17 00:00:00 2001 From: Melanie Plageman Date: Thu, 2 Sep 2021 11:33:59 -0400 Subject: [PATCH v7 1/3] Add system view tracking shared buffer actions Add a system view which tracks - number of shared buffers the checkpointer and bgwriter write out

Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?)

2021-09-10 Thread Melanie Plageman
On Wed, Sep 8, 2021 at 9:28 PM Melanie Plageman wrote: > > On Fri, Aug 13, 2021 at 3:08 AM Andres Freund wrote: > > > > Hi, > > > > On 2021-08-11 16:11:34 -0400, Melanie Plageman wrote: > > > On Tue, Aug 3, 2021 at 2:13 PM Andres Freund wrote: > >

Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?)

2021-09-08 Thread Melanie Plageman
On Fri, Aug 13, 2021 at 3:08 AM Andres Freund wrote: > > Hi, > > On 2021-08-11 16:11:34 -0400, Melanie Plageman wrote: > > On Tue, Aug 3, 2021 at 2:13 PM Andres Freund wrote: > > > > Also, I'm unsure how writing the buffer action stats out in > > > > pg

Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?)

2021-09-07 Thread Melanie Plageman
On Fri, Aug 13, 2021 at 3:08 AM Andres Freund wrote: > > Hi, > > On 2021-08-11 16:11:34 -0400, Melanie Plageman wrote: > > On Tue, Aug 3, 2021 at 2:13 PM Andres Freund wrote: > > > > diff --git a/src/backend/catalog/system_views.sql > > > > b/src/back

Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?)

2021-08-11 Thread Melanie Plageman
On Wed, Aug 11, 2021 at 4:11 PM Melanie Plageman wrote: > > On Tue, Aug 3, 2021 at 2:13 PM Andres Freund wrote: > > > > > @@ -2895,6 +2948,20 @@ FlushBuffer(BufferDesc *buf, SMgrRelation reln) > > > /* > > >* bufToWrite is either the

Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?)

2021-08-11 Thread Melanie Plageman
On Tue, Aug 3, 2021 at 2:13 PM Andres Freund wrote: > > Hi, > > On 2021-08-02 18:25:56 -0400, Melanie Plageman wrote: > > Thanks for the feedback! > > > > I agree it makes sense to count strategy writes separately. > > > > I thought about this som

Re: Asynchronous and "direct" IO support for PostgreSQL.

2021-08-09 Thread Melanie Plageman
On Wed, Jul 28, 2021 at 1:37 PM Melanie Plageman wrote: > > On Tue, Feb 23, 2021 at 5:04 AM Andres Freund wrote: > > > > ## AIO API overview > > > > The main steps to use AIO (without higher level helpers) are: > > > > 1) acquire an "unuse

Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?)

2021-08-02 Thread Melanie Plageman
On Fri, Jun 4, 2021 at 5:52 PM Alvaro Herrera wrote: > > On 2021-Apr-12, Melanie Plageman wrote: > > > As for the way I have recorded strategy writes -- it is quite inelegant, > > but, I wanted to make sure that I only counted a strategy write as one > > in which the

Re: Parallel Full Hash Join

2021-07-30 Thread Melanie Plageman
m" finger painting, to show how it looks > > > with these three patches. "scan*" is new. > > > > Patches 0002, 0003 no longer apply to the master branch, seemingly > > because of subsequent changes to pgstat, so need rebasing. > > I am changing the stat

Re: Asynchronous and "direct" IO support for PostgreSQL.

2021-07-30 Thread Melanie Plageman
On Wed, Jul 28, 2021 at 2:10 PM Andres Freund wrote: > On 2021-07-28 13:37:48 -0400, Melanie Plageman wrote: > > > Each IO will have its own TBMIterateResult allocated and returned by the > > PgStreamingRead helper and freed later by > > heapam_scan_bitmap_next_block() be

Re: Asynchronous and "direct" IO support for PostgreSQL.

2021-07-28 Thread Melanie Plageman
failing partition_prune regression test because I haven't had a chance to look at the EXPLAIN code which adds the text which is not being produced to see if it is actually a bug in my code or not. Oh, and I haven't done testing to see how effective the prefetching is -- that is a larger project that I hav

Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?)

2021-06-04 Thread Melanie Plageman
On Thu, Apr 15, 2021 at 7:59 PM Andres Freund wrote: > Hi, > > On 2021-04-12 19:49:36 -0700, Melanie Plageman wrote: > > So, I took a stab at implementing this in PgBackendStatus. > > Cool! > Just a note on v2 of the patch -- the diff for the changes I made to pgstatfun

Re: Avoiding smgrimmedsync() during nbtree index builds

2021-05-03 Thread Melanie Plageman
fered_extend(). I figured I would do so after ensuring the correctness of the logic in this patch. Then I will work on a patch which implements the unbuffered_write() API and demonstrates its utility with at least a few of the most compelling most compelling use cases in the code. - Melanie From 59837

Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?)

2021-04-12 Thread Melanie Plageman
On Sun, Jan 26, 2020 at 11:21 PM Kyotaro Horiguchi wrote: > At Sun, 26 Jan 2020 12:22:03 -0800, Andres Freund wrote > in > > On 2020-01-26 16:20:03 +0100, Magnus Hagander wrote: > > > On Sun, Jan 26, 2020 at 1:44 AM Andres Freund wrote: > > > > On 2020-01-25 15:43:41 +0100, Magnus Hagander

Re: Parallel Full Hash Join

2021-04-06 Thread Melanie Plageman
On Fri, Apr 2, 2021 at 3:06 PM Zhihong Yu wrote: > > Hi, > For v6-0003-Parallel-Hash-Full-Right-Outer-Join.patch > > +* current_chunk_idx: index in current HashMemoryChunk > > The above comment seems to be better fit for ExecScanHashTableForUnmatched(), > instead of

Re: Parallel Full Hash Join

2021-04-02 Thread Melanie Plageman
On Fri, Mar 5, 2021 at 8:31 PM Thomas Munro wrote: > > On Tue, Mar 2, 2021 at 11:27 PM Thomas Munro wrote: > > On Fri, Feb 12, 2021 at 11:02 AM Melanie Plageman > > wrote: > > > I just attached the diff. > > > > Squashed into one patch for the cfbot to ch

Re: Assertion failure with barriers in parallel hash join

2021-03-31 Thread Melanie Plageman
On Wed, Mar 17, 2021 at 8:18 AM Thomas Munro wrote: > > On Wed, Mar 17, 2021 at 6:58 PM Thomas Munro wrote: > > According to BF animal elver there is something wrong with this > > commit. Looking into it. > > Assertion failure reproduced here and understood, but unfortunately > it'll take some

Re: Assertion failure with barriers in parallel hash join

2021-03-31 Thread Melanie Plageman
On Wed, Mar 17, 2021 at 8:18 AM Thomas Munro wrote: > > On Wed, Mar 17, 2021 at 6:58 PM Thomas Munro wrote: > > According to BF animal elver there is something wrong with this > > commit. Looking into it. > > Assertion failure reproduced here and understood, but unfortunately > it'll take some

Re: Parallel Full Hash Join

2021-02-11 Thread Melanie Plageman
+bool > +ExecParallelPrepHashTableForUnmatched(HashJoinState *hjstate) > > Comment name doesn't match function name. Updated -- and a few other comment updates too. I just attached the diff. >From 213c36f9e125f52eb6731005d5dcdadca73a Mon Sep 17 00:00:00 2001 From: Melanie Plagem

Re: Parallel Full Hash Join

2020-11-04 Thread Melanie Plageman
for this concept, since it is a function with the purpose of terminating parallelism. Regards, Melanie (Microsoft) From 0b13b62a8aac071393ac65b19dc1ec86daa967f2 Mon Sep 17 00:00:00 2001 From: Melanie Plageman Date: Wed, 4 Nov 2020 14:25:33 -0800 Subject: [PATCH v3] Support Parallel FOJ and ROJ

Re: Assertion failure with barriers in parallel hash join

2020-10-12 Thread Melanie Plageman
On Thu, Oct 1, 2020 at 8:08 PM Thomas Munro wrote: > On Tue, Sep 29, 2020 at 9:12 PM Thomas Munro > wrote: > > On Tue, Sep 29, 2020 at 7:11 PM Michael Paquier > wrote: > > > #2 0x009027d2 in ExceptionalCondition > > > (conditionName=conditionName@entry=0xa80846

Re: Parallel Full Hash Join

2020-09-29 Thread Melanie Plageman
On Mon, Sep 21, 2020 at 8:34 PM Thomas Munro wrote: > On Tue, Sep 22, 2020 at 8:49 AM Melanie Plageman > wrote: > > On Wed, Sep 11, 2019 at 11:23 PM Thomas Munro > wrote: > > I took it for a very quick spin and saw simple cases working nicely, > but TPC-DS queries 51 an

Re: Parallel Full Hash Join

2020-09-21 Thread Melanie Plageman
On Wed, Sep 11, 2019 at 11:23 PM Thomas Munro wrote: > > While thinking about looping hash joins (an alternative strategy for > limiting hash join memory usage currently being investigated by > Melanie Plageman in a nearby thread[1]), the topic of parallel query > deadlock ha

Re: Reigning in ExecParallelHashRepartitionFirst

2020-07-08 Thread Melanie Plageman
s/reign/rein/ in $subject https://www.merriam-webster.com/words-at-play/do-you-rein-in-or-reign-in-something

Reigning in ExecParallelHashRepartitionFirst

2020-07-08 Thread Melanie Plageman
tgresql.org/message-id/flat/CA%2BhUKGJvYFCcF8vTHFSQQB_F8oGRsBp3JdZAPWbORZgfAPk5Sw%40mail.gmail.com#1156516651bb2587da3909cf1db29952 -- Melanie Plageman From 22b01b7e514cf975bb70d14918dcb6611a09bbd4 Mon Sep 17 00:00:00 2001 From: Melanie Plageman Date: Thu, 2 Jul 2020 17:02:48 -0700 Subject: [PATCH v1] Bail out of rep

Re: hashagg slowdown due to spill changes

2020-06-24 Thread Melanie Plageman
On Tue, Jun 23, 2020 at 10:06 AM Andres Freund wrote: > Hi, > > On 2020-06-23 09:23:57 -0700, Melanie Plageman wrote: > > On Mon, Jun 22, 2020 at 9:02 PM Andres Freund > wrote: > > > It's not this patch's fault, but none, really none, of this stuff > s

Re: Extracting only the columns needed for a query

2020-06-23 Thread Melanie Plageman
On Fri, Mar 13, 2020 at 12:09 PM Dmitry Dolgov <9erthali...@gmail.com> wrote: > I've also noticed that for partitioned tables every partition is > evaluated separately. IIRC they structure cannot differ, does it > makes sense then? Good point. At some point, we had discussed only collecting

Re: hashagg slowdown due to spill changes

2020-06-23 Thread Melanie Plageman
**unaggregated) > > { > > It's not this patch's fault, but none, really none, of this stuff should > be in the executor. > > Were you thinking it could be done in grouping_planner() and then the bitmaps could be saved in the PlannedStmt? Or would you have to wait until query_planner()? Or are you imagining somewhere else entirely? -- Melanie Plageman

Extra target list entries for child partitions in DELETE...USING...RETURNING

2020-06-22 Thread Melanie Plageman
ll correctly update * subroot->partColsUpdated.) */ [1] https://www.postgresql.org/message-id/flat/CAAKRu_ZQ0Jy7LfZDCY0JdxChdpja9rf-S8Y5%2BU4vX7cYJd62dA%40mail.gmail.com#f16fb3bdf33519c0d547a4b7ae2fc3c3 [2] https://www.postgresql.org/message-id/CAAKRu_ZQ0Jy7LfZDCY0JdxChdpja9rf-S8Y5%2BU4vX7cYJd62dA%40mail.gmail.com -- Melanie Plageman

Re: Extracting only the columns needed for a query

2020-06-18 Thread Melanie Plageman
On Fri, Mar 13, 2020 at 12:09 PM Dmitry Dolgov <9erthali...@gmail.com> wrote: > In general implemented functionality looks good. I've checked how it > works on the existing tests, almost everywhere required columns were not > missing in scanCols (which is probably the most important part). >

Re: Improve planner cost estimations for alternative subplans

2020-06-17 Thread Melanie Plageman
nt because instead of comparing alternatives you are blending them. I don't have any academic basis for saying that the alternatives costs shouldn't be averaged together for use in the rest of the plan, so I could definitely be wrong. -- Melanie Plageman

Re: Improve planner cost estimations for alternative subplans

2020-06-16 Thread Melanie Plageman
onek a group by grouping sets((ten,four),(ten)) having exists (select 1 from onek b where sum(distinct a.four) = b.four); But, the chosen plan for this query stays the same. It would be helpful to see a query where a different plan is chosen because of this change that is not from updatable

Re: Find query characters in respect of optimizer for develop purpose

2020-06-11 Thread Melanie Plageman
hat it seems like this struct would have to be updated throughout planning and that it would be easy to break it with the addition of new code. Couldn't every new optimization added to planner potentially affect the accuracy of the information in the struct? -- Melanie Plageman

Re: Default setting for enable_hashagg_disk

2020-06-10 Thread Melanie Plageman
On Wed, Jun 10, 2020 at 10:39 AM Jeff Davis wrote: > On Tue, 2020-06-09 at 18:20 -0700, Melanie Plageman wrote: > > So, I was catching up on email and noticed the last email in this > > thread. > > > > I think I am not fully understanding what > > enabl

Re: Default setting for enable_hashagg_disk

2020-06-10 Thread Melanie Plageman
On Tue, Jun 9, 2020 at 7:15 PM Justin Pryzby wrote: > On Tue, Jun 09, 2020 at 06:20:13PM -0700, Melanie Plageman wrote: > > On Thu, Apr 9, 2020 at 1:02 PM Jeff Davis wrote: > > > > > 2. enable_groupingsets_hash_disk (default false): > > > > > > This

Re: Default setting for enable_hashagg_disk

2020-06-09 Thread Melanie Plageman
sh Key: g10, g1000 Hash Key: g10 Group Key: () Peak Memory Usage: 405 kB Disk Usage: 59712 kB HashAgg Batches: 4209 -> Seq Scan on gs_hash_1 (actual rows=20 loops=1) I'm not sure if this is more what you were looking for--or maybe I am misunderstanding the guc. -- Melanie Plageman

Re: Trouble with hashagg spill I/O pattern and costing

2020-05-26 Thread Melanie Plageman
for all the cases I tried. Is that where you made the change? And then are you proposing to set it based on the aggstrategy to either CP_LABEL_TLIST or CP_SMALL_TLIST here? -- Melanie Plageman

Re: Avoiding hash join batch explosions with extreme skew and weird stats

2020-04-30 Thread Melanie Plageman
On Tue, Apr 28, 2020 at 11:50 PM Heikki Linnakangas wrote: > On 29/04/2020 05:03, Melanie Plageman wrote: > > I've attached a patch which should address some of the previous feedback > > about code complexity. Two of my co-workers and I wrote what is > > essentially a new

Re: Memory-Bounded Hash Aggregation

2020-02-18 Thread Melanie Plageman
I'll let Jeff decide what he wants to do about the patch at all (e.g. include it in his overall patch or exclude it for now). Anyway it is trivial to move those declarations up, were he to decide to include it. -- Melanie Plageman

Re: Extracting only the columns needed for a query

2020-02-18 Thread Melanie Plageman
On Fri, Jan 31, 2020 at 9:52 AM Melanie Plageman wrote: > I'm bumping this to the next commitfest because I haven't had a chance > to address the questions posed by Dmitry. I'm sure I'll get to it in > the next few weeks. > > > I believe it would be beneficial to add this poten

Re: Memory-Bounded Hash Aggregation

2020-02-13 Thread Melanie Plageman
s.com [2] https://www.postgresql.org/message-id/flat/CAAKRu_Yj%3DQ_ZxiGX%2BpgstNWMbUJApEJX-imvAEwryCk5SLUebg%40mail.gmail.com -- Melanie Plageman v1-0001-aggregated-unaggregated-cols-together.patch Description: Binary data

Re: Adding a test for speculative insert abort case

2020-02-11 Thread Melanie Plageman
> > Thanks so much for finishing the patch and checking for race conditions! -- Melanie Plageman

Re: Extracting only the columns needed for a query

2020-01-31 Thread Melanie Plageman
le of how a table AM API user like Zedstore uses the columns during planning. [1] https://www.postgresql.org/message-id/e5566f7def33a9e9fdff337cca32d07155d7b635.camel%40j-davis.com -- Melanie Plageman

Re: BufFileRead() error signalling

2020-01-30 Thread Melanie Plageman
ker reading one SharedTuplestoreChunk encounters an error and another worker on a different SharedTuplstoreChunk of the same file does not, I would find it useful to know more about which block it was on when the error was encountered. -- Melanie Plageman

Re: Parallel leader process info in EXPLAIN

2020-01-24 Thread Melanie Plageman
So, I think from a code review perspective the code in the patches LGTM. As for the EXPLAIN ANALYZE tests--I don't see that many of them in regress, so maybe that's because they aren't normally very useful. In this case, it would only be to protect against regressions in printing the leader

Re: Parallel leader process info in EXPLAIN

2020-01-17 Thread Melanie Plageman
with xxx the actual numbers, I would have thought that there would be an EXPLAIN VERBOSE with leader participation somewhere in regress). -- Melanie Plageman 0001-Show-parallel-leader-stats-in-EXPLAIN-output-v4.patch Description: Binary data

Re: Avoiding hash join batch explosions with extreme skew and weird stats

2020-01-09 Thread Melanie Plageman
t; that, blocks are also super overloaded). > Hmmm. I think loop is kinda confusing. "fragment" has potential. I also thought of "piece". That is actually where I am leaning now. What do you think? [1] https://www.postgresql.org/message-id/flat/CA%2BhUKG%2BA6ftXPz4oe92%2Bx8Er%2BxpGZqto70-Q_ERwRaSyA%3DafNg%40mail.gmail.com -- Melanie Plageman

Re: Extracting only the columns needed for a query

2020-01-07 Thread Melanie Plageman
I just wanted to address a question we got about making scanCols instead of using RelOptInfo->attr_needed. David Rowley had asked: > For planning, isn't this information already available via either > targetlists or from the RelOptInfo->attr_needed array combined with > min_attr and max_attr?

Re: accounting for memory used for BufFile during hash joins

2020-01-03 Thread Melanie Plageman
tps://www.postgresql.org/message-id/CAAKRu_YsWm7gc_b2nBGWFPE6wuhdOLfc1LBZ786DUzaCPUDXCA%40mail.gmail.com -- Melanie Plageman

Re: Extracting only the columns needed for a query

2020-01-02 Thread Melanie Plageman
when IS_DUMMY_REL is true for a relation, do we reference the associated RTE later? It seems like if it is a dummy rel, we wouldn't scan it. It still makes sense to add it to extract_used_columns(), I think, to avoid any wasted loops through the rel's expressions. Thanks for the idea! -- Melanie Plageman v2-0001-Plan-time-extraction-of-scan-cols.patch Description: Binary data

Re: Memory-Bounded Hash Aggregation

2019-12-04 Thread Melanie Plageman
On Thu, Nov 28, 2019 at 9:47 AM Tomas Vondra wrote: > On Wed, Nov 27, 2019 at 02:58:04PM -0800, Jeff Davis wrote: > >On Wed, 2019-08-28 at 12:52 -0700, Taylor Vesely wrote: > >> Right now the patch always initializes 32 spill partitions. Have you > >> given > >> any thought into how to

Re: Parallel leader process info in EXPLAIN

2019-10-30 Thread Melanie Plageman
me think about other explain wishlist items. For parallel hashjoin, I would find it useful to know which batches each worker participated in (maybe just probing to start with, but loading would be great too). I'm not sure anyone else (especially users) would care about this, though. -- Melanie Plageman

Re: Memory Accounting

2019-09-24 Thread Melanie Plageman
On Thu, Sep 19, 2019 at 11:00 AM Jeff Davis wrote: > On Wed, 2019-09-18 at 13:50 -0700, Soumyadeep Chakraborty wrote: > > Hi Jeff, > > Hi Soumyadeep and Melanie, > > Thank you for the review! > > > max_stack_depth max level lazy (ms) eager (ms) > (eage > > r/lazy) > > 2MB 82

Re: Memory Accounting

2019-09-18 Thread Melanie Plageman
I wanted to address a couple of questions Jeff asked me off-list about Greenplum's implementations of Memory Accounting. Greenplum has two memory accounting sub-systems -- one is the MemoryContext-based system proposed here. The other memory accounting system tracks "logical" memory owners in

Re: Avoiding hash join batch explosions with extreme skew and weird stats

2019-09-06 Thread Melanie Plageman
ker has scanned their outer match status files. All the probe loops would be done, and the worker that is emitting tuples is not referencing the inner side hashtable at all and only the outer batch file and the combined bitmap. -- Melanie Plageman

Re: accounting for memory used for BufFile during hash joins

2019-09-05 Thread Melanie Plageman
On Tue, Sep 3, 2019 at 9:36 AM Alvaro Herrera wrote: > On 2019-Jul-11, Tomas Vondra wrote: > > > On Wed, Jul 10, 2019 at 04:51:02PM -0700, Melanie Plageman wrote: > > > > I think implementing support for parallel hashjoin or explicitly > > > disabling it would b

Re: Cleanup isolation specs from unused steps

2019-08-22 Thread Melanie Plageman
On Thu, Aug 22, 2019 at 6:53 PM Michael Paquier wrote: > On Thu, Aug 22, 2019 at 10:20:48AM -0700, Melanie Plageman wrote: > > So, there is some historical context as to why it is a separate test > suite. > > And some of the differences are specific to Greenplum -- e.g. need

Re: Cleanup isolation specs from unused steps

2019-08-22 Thread Melanie Plageman
On Wed, Aug 21, 2019 at 12:16 PM Alvaro Herrera wrote: > On 2019-Aug-21, Melanie Plageman wrote: > > > In Greenplum, we mainly add new tests to a separate isolation > > framework (called isolation2) which uses a completely different > > syntax. It doesn't use isola

Re: Adding a test for speculative insert abort case

2019-08-21 Thread Melanie Plageman
On Wed, Aug 7, 2019 at 1:47 PM Melanie Plageman wrote: > > > On Wed, Jul 24, 2019 at 11:48 AM Andres Freund wrote: > >> > diff --git a/src/test/isolation/specs/insert-conflict-specconflict.spec >> b/src/test/isolation/specs/insert-conflict-specconflict.spec >&g

Re: Cleanup isolation specs from unused steps

2019-08-21 Thread Melanie Plageman
ce really much extra maintenance cost. > So, I think I completely misunderstood the purpose of 'dry-run'. If no one is using it, having a check for unused steps in dry-run may not be useful. +1 to renaming it to --print-permutations and, potentially, adding --print-all-permutations -- Melanie Plageman

Re: Cleanup isolation specs from unused steps

2019-08-21 Thread Melanie Plageman
fferent syntax. It doesn't use isolationtester at all. So, I haven't had a use case to add long options to isolationtester yet :) -- Melanie Plageman

Re: Cleanup isolation specs from unused steps

2019-08-19 Thread Melanie Plageman
o the check that all steps have been used in dry_run mode instead of when running the tests for real? -- Melanie Plageman

Re: Adding a test for speculative insert abort case

2019-08-07 Thread Melanie Plageman
On Wed, Jul 24, 2019 at 11:48 AM Andres Freund wrote: > > diff --git a/src/test/isolation/specs/insert-conflict-specconflict.spec > b/src/test/isolation/specs/insert-conflict-specconflict.spec > > index 3a70484fc2..7f29fb9d02 100644 > > ---

Re: Avoiding hash join batch explosions with extreme skew and weird stats

2019-07-30 Thread Melanie Plageman
On Tue, Jul 30, 2019 at 4:36 PM Robert Haas wrote: > On Tue, Jul 30, 2019 at 2:47 PM Melanie Plageman > wrote: > > I did the "needlessly dumb implementation" Robert mentioned, though, > > I thought about it and couldn't come up with a much smarter way to > > w

Re: Avoiding hash join batch explosions with extreme skew and weird stats

2019-07-30 Thread Melanie Plageman
22:09PM -0700, Melanie Plageman wrote: > >On Tue, Jun 18, 2019 at 3:24 PM Melanie Plageman < > melanieplage...@gmail.com> > > > >Before doing that, I wanted to ask what a desirable fallback condition > >would be. In this patch, fallback to hashloop join happens only whe

Re: Memory Accounting

2019-07-23 Thread Melanie Plageman
uld be good for memory intensive operators which use a large, representative context. I think the HashTableContext for HashJoin might be one? -- Melanie Plageman

Re: Extracting only the columns needed for a query

2019-07-17 Thread Melanie Plageman
On Sat, Jun 15, 2019 at 10:01 AM Tom Lane wrote: > Melanie Plageman writes: > > While hacking on zedstore, we needed to get a list of the columns to > > be projected--basically all of the columns needed to satisfy the > > query. The two use cases we have for this is >

Re: Extracting only the columns needed for a query

2019-07-16 Thread Melanie Plageman
We implemented Approach B in the attached patch set (patch 0001) and then implemented Approach A (patch 0002) to sanity check the pruned list of columns to scan we were getting at plan-time. We emit a notice in SeqNext() if the two column sets differ. Currently, for all of the queries in the

Re: PGOPTIONS="-fh" make check gets stuck since Postgres 11

2019-07-10 Thread Melanie Plageman
ule called atmsort [2] to deal with the specific ORDER BY case discussed here. [1] https://github.com/greenplum-db/gpdb/blob/master/src/test/regress/gpdiff.pl [2] https://github.com/greenplum-db/gpdb/blob/master/src/test/regress/atmsort.pl -- Melanie Plageman

Re: PGOPTIONS="-fh" make check gets stuck since Postgres 11

2019-07-09 Thread Melanie Plageman
results files to ignore row ordering in some cases to allow for easier comparison. Has this been proposed in the past? -- Melanie Plageman

Re: Avoiding hash join batch explosions with extreme skew and weird stats

2019-07-03 Thread Melanie Plageman
On Tue, Jun 18, 2019 at 3:24 PM Melanie Plageman wrote: > > These questions will probably make a lot more sense with corresponding > code, so I will follow up with the second version of the state machine > patch once I finish it. > > I have changed the state machine and resolv

Re: Avoiding hash join batch explosions with extreme skew and weird stats

2019-06-18 Thread Melanie Plageman
On Thu, Jun 13, 2019 at 7:10 AM Robert Haas wrote: > On Tue, Jun 11, 2019 at 2:35 PM Melanie Plageman > wrote: > > Let me try to articulate what I think the bitmap implementation would > look > > like: > > > > Before doing chunked hashloop join for any batch, w

Extracting only the columns needed for a query

2019-06-14 Thread Melanie Plageman
While hacking on zedstore, we needed to get a list of the columns to be projected--basically all of the columns needed to satisfy the query. The two use cases we have for this is 1) to pass this column list down to the AM layer for the AM to leverage it 2) for use during planning to improving

Re: Avoiding hash join batch explosions with extreme skew and weird stats

2019-06-11 Thread Melanie Plageman
On Fri, Jun 7, 2019 at 7:30 AM Robert Haas wrote: > On Thu, Jun 6, 2019 at 7:31 PM Melanie Plageman > wrote: > > I'm not sure I understand why you would need to compare the original > > tuples to the unmatched tuples file. > > I think I was confused. Actually, I'm sti

Re: Avoiding hash join batch explosions with extreme skew and weird stats

2019-06-06 Thread Melanie Plageman
do that was kind of like mmap'ing the outer side file to give the workers in parallel hashjoin the ability to update a match bit in the tuple in place and avoid writing the whole outer side out each time. -- Melanie Plageman

Re: Avoiding hash join batch explosions with extreme skew and weird stats

2019-06-06 Thread Melanie Plageman
On Tue, Jun 4, 2019 at 12:15 PM Robert Haas wrote: > On Tue, Jun 4, 2019 at 3:09 PM Melanie Plageman > wrote: > > One question I have is, how would the OR'd together bitmap be > > propagated to workers after the first chunk? That is, when there are > > no tuples

Re: Avoiding hash join batch explosions with extreme skew and weird stats

2019-06-06 Thread Melanie Plageman
On Tue, Jun 4, 2019 at 12:08 PM Robert Haas wrote: > On Tue, Jun 4, 2019 at 2:47 PM Melanie Plageman > wrote: > > Oops! You are totally right. > > I will amend the idea: > > For each chunk on the inner side, loop through both the original batch > > file and t

Re: Adding a test for speculative insert abort case

2019-06-05 Thread Melanie Plageman
On Thu, May 16, 2019 at 8:46 PM Melanie Plageman wrote: > > Good idea. > I squashed the changes I suggested in previous emails, Ashwin's patch, my > suggested updates to that patch, and the index order check all into one > updated > patch attached. > > I've updated thi

Re: Sort support for macaddr8

2019-06-05 Thread Melanie Plageman
ope, Melanie? > > I can take on making macaddr8 pass-by-value I tinkered a bit last night and got in/out mostly working (I think). I'm not sure about macaddr, TID, and user-defined types. -- Melanie Plageman

Re: Avoiding hash join batch explosions with extreme skew and weird stats

2019-06-04 Thread Melanie Plageman
given inner chunk, would you load the bitmaps from each worker into memory, OR them together, and then write the updated bitmap back out so that each worker starts with the updated bitmap? -- Melanie Plageman

Re: Avoiding hash join batch explosions with extreme skew and weird stats

2019-06-04 Thread Melanie Plageman
On Tue, Jun 4, 2019 at 5:43 AM Robert Haas wrote: > On Mon, Jun 3, 2019 at 5:10 PM Melanie Plageman > wrote: > > I was talking to Jeff Davis about this on Saturday, and, he felt that > > there might be a way to solve the problem differently if we thought of > > the left

Re: Sort support for macaddr8

2019-06-04 Thread Melanie Plageman
p_internal() is just specified as an int. I was wondering why. I also noticed that the prototype for macaddr8_cmp_internal() was not at the top of the file with the other static function prototypes. I added it there, but I wasn't sure if there was some reason that it was like tha

Re: Avoiding hash join batch explosions with extreme skew and weird stats

2019-06-03 Thread Melanie Plageman
On Sun, May 19, 2019 at 4:07 PM Thomas Munro wrote: > On Sat, May 18, 2019 at 12:15 PM Melanie Plageman > wrote: > > On Thu, May 16, 2019 at 3:22 PM Thomas Munro > wrote: > >> Admittedly I don't have a patch, just a bunch of handwaving. One > >> re

Re: accounting for memory used for BufFile during hash joins

2019-05-21 Thread Melanie Plageman
On Wed, May 8, 2019 at 8:08 AM Tomas Vondra wrote: > On Tue, May 07, 2019 at 05:30:27PM -0700, Melanie Plageman wrote: > > One thing I was a little confused by was the nbatch_inmemory member > > of the hashtable. The comment in ExecChooseHashTableSize says that > >

Re: describe working as intended?

2019-05-21 Thread Melanie Plageman
to display tables with such name in all schemas. > > regards, Sergei > Thanks! I suppose it would behoove me to check the documentation before resorting to looking at the source code :) -- Melanie Plageman

Re: Avoiding hash join batch explosions with extreme skew and weird stats

2019-05-20 Thread Melanie Plageman
On Sun, May 19, 2019 at 4:07 PM Thomas Munro wrote: > On Sat, May 18, 2019 at 12:15 PM Melanie Plageman > wrote: > > On Thu, May 16, 2019 at 3:22 PM Thomas Munro > wrote: > >> Admittedly I don't have a patch, just a bunch of handwaving. One > >> re

describe working as intended?

2019-05-17 Thread Melanie Plageman
---+---+- 23609 | public| t1 23612 | pg_temp_4 | t1 (2 rows) So, without much more digging, is the current behavior of describe intended? I couldn't find an email thread discussing this with the search terms I tried. (I noticed it on master and checked 11 as well and got the same behavior.) -- Melanie

Re: Avoiding hash join batch explosions with extreme skew and weird stats

2019-05-17 Thread Melanie Plageman
hes for the outer side, make one "batch" that has all the tuples from the outer side which the inner side batch which was flagged will do NLJ with. -- Melanie Plageman

Re: Adding a test for speculative insert abort case

2019-05-17 Thread Melanie Plageman
On Thu, May 16, 2019 at 8:46 PM Melanie Plageman wrote: > > I squashed the changes I suggested in previous emails, Ashwin's patch, my > suggested updates to that patch, and the index order check all into one > updated > patch attached. > > I realized that the numbers

Re: Adding a test for speculative insert abort case

2019-05-16 Thread Melanie Plageman
On Thu, May 16, 2019 at 2:03 PM Andres Freund wrote: > Hi, > > On 2019-05-16 13:59:47 -0700, Melanie Plageman wrote: > > On Wed, May 15, 2019 at 10:32 PM Ashwin Agrawal > wrote: > > > > > > > > The second index would help to hold the session after i

Re: Adding a test for speculative insert abort case

2019-05-16 Thread Melanie Plageman
the wording of the comments overall in the above email I sent). -- Melanie Plageman 0002-Add-test-to-validate-speculative-wait-is-performed.patch Description: Binary data

Re: Adding a test for speculative insert abort case

2019-05-15 Thread Melanie Plageman
t;s2_upsert" "controller_show" "controller_unlock_1_1" "controller_unlock_2_1" "controller_unlock_1_3" "controller_unlock_2_3" "controller_unlock_1_2" "s1_magically_pause_before_complete_speculative" # put s2 in speculative wait "controller_unlock_2_2" "s1_magically_unpause_before_complete_speculative" So, how would another lock on another index keep s1 from clearing the speculative token after it has updated the index? -- Melanie Plageman specconflict_permutation_comment_update.patch Description: Binary data

Re: Adding a test for speculative insert abort case

2019-05-15 Thread Melanie Plageman
ot;controller_show" I was also wondering: Is it possible that one of the "controller_unlock_*" functions will get called before the session with the upsert has had a chance to move forward in its progress and be waiting on that lock? That is, given that we don't check that the sessions are waiting on the locks before unlocking them, is there a race condition? I noticed that there is not a test case which would cover the speculative wait codepath. This seems much more challenging, however, it does seem like a worthwhile test to have. -- Melanie Plageman

<    1   2   3   4   5   6   7   >