Re: [HACKERS] Parallel bitmap heap scan

2017-02-19 Thread Thomas Munro
ngs a bit better. I don't know anywhere near enough about query planners to say whether such ideas about planning are reasonable, and am quite aware that it's difficult terrain, and I have other fish to fry, so I'm going to put down the can opener and back away. -- Thomas Munro http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] UPDATE of partition key

2017-02-20 Thread Thomas Munro
/static/transaction-iso.html#XACT-REPEATABLE-READ -- Thomas Munro http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] UPDATE of partition key

2017-02-20 Thread Thomas Munro
On Mon, Feb 20, 2017 at 3:36 PM, Thomas Munro wrote: > On Thu, Feb 16, 2017 at 8:53 PM, Robert Haas wrote: >> Generally speaking, we don't throw >> serialization errors today at READ COMMITTED, so if we do so here, >> that's going to be a noticeable and perhaps

Re: [HACKERS] SERIALIZABLE with parallel query

2017-02-20 Thread Thomas Munro
On Thu, Feb 16, 2017 at 6:19 PM, Thomas Munro wrote: > Specifically, DeleteChildTargetLocks() assumes it can walk > MySerializableXact->predicateLocks and throw away locks that are > covered by a new lock (ie throw away tuple locks because a covering > page lock has been acquired)

Re: [HACKERS] delta relations in AFTER triggers

2017-02-20 Thread Thomas Munro
= trigdata->tg_trigger->tgnewtable; + enr->md.tupdesc = trigdata->tg_relation->rd_att; + enr->md.enrtuples = tuplestore_tuple_count(trigdata->tg_newtable); + enr->reldata = trigdata->tg_newtable; + register_enr(estate.queryEnv, enr); + SPI_register_relation(enr); + } W

Re: [HACKERS] delta relations in AFTER triggers

2017-02-20 Thread Thomas Munro
On Tue, Feb 21, 2017 at 7:14 AM, Thomas Munro wrote: > On Fri, Jan 20, 2017 at 2:49 AM, Kevin Grittner wrote: >> Attached is v9 which fixes bitrot from v8. No other changes. >> >> Still needs review. Based on a suggestion from Robert off-list I tried inserting into a

Re: [HACKERS] Measuring replay lag

2017-02-21 Thread Thomas Munro
ere is still now new WAL so we keep showing the last measured lag. If new WAL is flushed it will pop back to 0ish, but until then its last known measurement is ~14 seconds, which I don't think is technically wrong. > Some minor points on code... > Why are things defined in walsender.c and not in .h? Because

Re: [HACKERS] Measuring replay lag

2017-02-22 Thread Thomas Munro
IF EXISTS foo; CREATE TABLE foo AS SELECT generate_series(1, 1000); SELECT pg_sleep(10); While testing with a small buffer I found a thinko when write_head is moved back, fixed in the attached. -- Thomas Munro http://www.enterprisedb.com replication-lag-v4.patch Description: Binar

Re: [HACKERS] Measuring replay lag

2017-02-22 Thread Thomas Munro
On Thu, Feb 23, 2017 at 11:52 AM, Thomas Munro wrote: > The overall graph looks pretty similar, but it is more likely to short > hiccups caused by occasional slow WAL fsyncs in walreceiver. See the I meant to write "more likely to *miss* short hiccups". -- T

Re: [HACKERS] A typo in mcxt.c

2017-02-23 Thread Thomas Munro
ny bit like a famous line from Hamlet, "Whether 'tis nobler in the mind to suffer...". https://en.wikipedia.org/wiki/To_be,_or_not_to_be -- Thomas Munro http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Poor memory context performance in large hash joins

2017-02-23 Thread Thomas Munro
On Fri, Feb 24, 2017 at 12:17 PM, Andres Freund wrote: > Jeff, do you have a handy demonstrator? If you want to hit ExecHashIncreaseNumBatches a bunch of times, see the "ugly" example in the attached. -- Thomas Munro http://www.enterprisedb.com hj-test-queries.sql Description

Re: [HACKERS] SerializedSnapshotData alignment

2017-02-26 Thread Thomas Munro
s to be building with -m64, so that's not the explanation. Could it be correctly aligned by coincidence? -- Thomas Munro http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] pg_serial early wraparound

2017-02-26 Thread Thomas Munro
On Wed, Nov 9, 2016 at 11:07 AM, Thomas Munro wrote: > The SLRU managed by predicate.c can wrap around and overwrite data if > you have more than 1 billion active XIDs. That's because when SSI was > implemented, slru.c was limited to four digit segment names, which > implied

Re: [HACKERS] pg_serial early wraparound

2017-02-26 Thread Thomas Munro
On Mon, Feb 27, 2017 at 7:28 PM, Thomas Munro wrote: > On Wed, Nov 9, 2016 at 11:07 AM, Thomas Munro > wrote: >> The SLRU managed by predicate.c can wrap around and overwrite data if >> you have more than 1 billion active XIDs. That's because when SSI was >> imple

Re: [HACKERS] make async slave to wait for lsn to be replayed

2017-02-27 Thread Thomas Munro
On Thu, Feb 23, 2017 at 3:08 AM, Thom Brown wrote: > On 23 January 2017 at 11:56, Ivan Kartyshov > wrote: >> >> Thank you for reading, will be glad to get your feedback. > > Could you please rebase your patch as it no longer applies cleanly. +1 -- Thomas Munro htt

Re: [HACKERS] GSoC 2017

2017-02-27 Thread Thomas Munro
t page claims that PostgreSQL runs on Irix and Tru64, which hasn't been true for a few years. -- Thomas Munro http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

[HACKERS] Unhelpful typesetting of callouts in example queries in the docs

2017-02-27 Thread Thomas Munro
) FROM weather WHERE city LIKE 'S%' -- (1) GROUP BY city HAVING max(temp_lo) < 40; See attached which does that and there and a couple of other places. -- Thomas Munro http://www.enterprisedb.com put-callouts-in-sql-comments.patch Description: Binary data

Re: [HACKERS] brin autosummarization -- autovacuum "work items"

2017-02-28 Thread Thomas Munro
clearing the DSA at an unwelcome > time, or perhaps there's a mistake in my handling of DSA "relative > pointers" stuff. Ok, I'll take a look. It's set up for ease of use in short lifespan situations like parallel query, and there are a few extra hoops to jump through

Re: [HACKERS] Parallel tuplesort (for parallel B-Tree index creation)

2017-03-01 Thread Thomas Munro
On Sat, Feb 11, 2017 at 1:52 AM, Robert Haas wrote: > On Thu, Feb 9, 2017 at 6:38 PM, Thomas Munro > wrote: >> Yes, potentially unbounded in rare case. If we plan for N batches, >> and then run out of work_mem because our estimates were just wrong or >> the distributions

Re: [HACKERS] WIP: [[Parallel] Shared] Hash

2017-03-01 Thread Thomas Munro
On Thu, Feb 16, 2017 at 9:08 PM, Thomas Munro wrote: > On Thu, Feb 16, 2017 at 3:36 PM, Andres Freund wrote: >> That's it for now... > > Thanks! Plenty for me to go away and think about. I will post a new > version soon. I'm testing a new version which incorpora

Re: [HACKERS] Measuring replay lag

2017-03-01 Thread Thomas Munro
On Fri, Feb 24, 2017 at 9:05 AM, Simon Riggs wrote: > On 21 February 2017 at 21:38, Thomas Munro > wrote: >> However, I think a call like LagTrackerWrite(SendRqstPtr, >> GetCurrentTimestamp()) needs to go into XLogSendLogical, to mirror >> what happens in XLogSendPhysi

Re: [HACKERS] 2017-03 Commitfest In Progress

2017-03-01 Thread Thomas Munro
On Thu, Mar 2, 2017 at 4:42 AM, David Steele wrote: > The 2017-03 commitfest is now in progress. Here's the breakdown: > > Needs review: 128 > Waiting on Author: 26 > Ready for Committer: 26 > Total: 180 Not quite a record haul but close. -- Thomas Munro http

Re: [HACKERS] brin autosummarization -- autovacuum "work items"

2017-03-02 Thread Thomas Munro
On Wed, Mar 1, 2017 at 6:06 PM, Thomas Munro wrote: > On Wed, Mar 1, 2017 at 5:58 PM, Alvaro Herrera > wrote: >> I think one of the most serious issues with BRIN indexes is how they >> don't get updated automatically as the table is filled. This patch >> attempt

Re: [HACKERS] brin autosummarization -- autovacuum "work items"

2017-03-02 Thread Thomas Munro
On Wed, Mar 1, 2017 at 6:06 PM, Thomas Munro wrote: > On Wed, Mar 1, 2017 at 5:58 PM, Alvaro Herrera > wrote: >> I think one of the most serious issues with BRIN indexes is how they >> don't get updated automatically as the table is filled. This patch >> attempt

Re: [HACKERS] WIP: [[Parallel] Shared] Hash

2017-03-06 Thread Thomas Munro
On Wed, Mar 1, 2017 at 10:40 PM, Thomas Munro wrote: > I'm testing a new version which incorporates feedback from Andres and > Ashutosh, and is refactored to use a new SharedBufFileSet component to > handle batch files, replacing the straw-man implementation from the v5 > patch

Re: [HACKERS] Parallel tuplesort (for parallel B-Tree index creation)

2017-03-06 Thread Thomas Munro
On Wed, Mar 1, 2017 at 10:29 PM, Thomas Munro wrote: > I'm testing a patch that lets you set up a fixed sized > SharedBufFileSet object in a DSM segment, with its own refcount for > the reason you explained. It supports a dynamically expandable set of > numbered files, so each

Re: [HACKERS] make async slave to wait for lsn to be replayed

2017-03-07 Thread Thomas Munro
aitlsn(some_lsn) waits for the LSN to be applied, the next statement will run with the snapshot from before and never see the transaction you were waiting for. -- Thomas Munro http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

[HACKERS] Removing #include "postgres.h" from a couple of headers

2017-03-08 Thread Thomas Munro
stom.h and snowball's header.h -- are those special cases? -- Thomas Munro http://www.enterprisedb.com remove-postgres.h-from-headers.patch Description: Binary data -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www

Re: [HACKERS] Supporting huge pages on Windows

2017-03-08 Thread Thomas Munro
is enabled, the normal +command prompt revokes the user right Lock Pages in Memory. The line beginning 'Huge pages are known as...' has been accidentally duplicated. -- Thomas Munro http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql

Re: [HACKERS] WIP: [[Parallel] Shared] Hash

2017-03-08 Thread Thomas Munro
tion about > alphabetizing #include references for all headers other than those magic > ones. Thanks for the explanation. Will post a new series addressing this and other complaints from Andres shortly. -- Thomas Munro http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] WIP: [[Parallel] Shared] Hash

2017-03-09 Thread Thomas Munro
return last; > +} > > Doesn't this, again, run into danger of leading to an assert failure in > the loop in BarrierWait? I believe this code is correct. The assertion in BarrierWait can't fail, because waiters know that there is no way for the phase to get any further a

Re: [HACKERS] on_dsm_detach() callback and parallel tuplesort BufFile resource management

2017-03-09 Thread Thomas Munro
it can support a dynamic set of N partitions from fixed set of M participants. I'm fairly sure that people won't be happy with two separate ways to manage shared temporary files. I also don't want patch A to be derailed by patch B, but even if these things finish up in d

Re: [HACKERS] GSOC Introduction / Eliminate O(N^2) scaling from rw-conflict tracking in serializable transactions

2017-03-09 Thread Thomas Munro
on. I suspect the latter may be quite hard to fix and the former quite easy. If you're currently studying the SERIALIZABLE internal data structures then you might like to review that patch and tell me if my optimism is entirely misplaced, and figure out how to test and break it... That said,

Re: [HACKERS] REINDEX CONCURRENTLY 2.0

2017-03-10 Thread Thomas Munro
he tag. -- Thomas Munro http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] REINDEX CONCURRENTLY 2.0

2017-03-10 Thread Thomas Munro
On Fri, Mar 10, 2017 at 9:36 PM, Thomas Munro wrote: > On Wed, Mar 8, 2017 at 4:12 PM, Andres Freund wrote: >> Can you come up with an halfway realistic scenario why an index oid, not >> a table, constraint, sequence oid, would be relied upon? > > Is there an implica

Re: [HACKERS] SERIALIZABLE with parallel query

2017-03-10 Thread Thomas Munro
lly good idea to get parallel SSI support (whether with this or some other approach) into the tree before people start showing up with parallel write patches. -- Thomas Munro http://www.enterprisedb.com ssi-parallel-v4.patch Description: Binary data -- Sent via pgsql-hackers mai

Re: [HACKERS] delta relations in AFTER triggers

2017-03-12 Thread Thomas Munro
ng it about functional dependencies as required by the spec; if you can SELECT id, name FROM GROUP BY id, I believe you should be able to SELECT id, name FROM GROUP BY id, but currently you can't. [1] https://www.postgresql.org/message-id/CAEepm=3wvmpmz3bkftk2kcnd9kr7hxpz2skj8sfzx_vsute...

Re: [HACKERS] make async slave to wait for lsn to be replayed

2017-03-12 Thread Thomas Munro
oice would be to create a custom wait list which actually holds the LSNs waited for in sorted order, so that we wake up exactly the right processes, cheaply, or in arbitrary order which makes insertion cheaper but search more expensive. -- Thomas Munro http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

[HACKERS] Bogus tab completion tweak for UPDATE ... SET ... =

2017-03-13 Thread Thomas Munro
Hi, Even though the following is coincidentally meaningful, I don't think it was intentional or is useful: postgres=# update foo set x = DEFAULT Shouldn't that completion should be suppressed, like in the attached? -- Thomas Munro http://www.enterprisedb.com fix-update-set-comple

Re: [HACKERS] WIP: [[Parallel] Shared] Hash

2017-03-13 Thread Thomas Munro
e in here: -> Parallel Hash Semi Join (cost=1006599.34..1719227.30 rows=0 width=24) (actual time=38716.488..100933.250 rows=7315896 loops=5) Will investigate, thanks. > Q8: 8 secs on HEAD to 14 secs with patch Also looking into this one. -- Thomas Munro http://www.enterprisedb.com --

Re: [HACKERS] Measuring replay lag

2017-03-13 Thread Thomas Munro
Hi, Please see separate replies to Simon and Craig below. On Sun, Mar 5, 2017 at 8:38 PM, Simon Riggs wrote: > On 1 March 2017 at 10:47, Thomas Munro wrote: >> I do see why a new user trying this feature for the first time might >> expect it to show a lag of 0 just as s

[HACKERS] Updating the "tool sets" documentation for modern FreeBSD

2017-03-13 Thread Thomas Munro
mmand. Also, a brief note about gmake vs make in the doc subdir. -- Thomas Munro http://www.enterprisedb.com freebsd-pkg-openjade-docs.patch Description: Binary data -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresq

Re: [HACKERS] Measuring replay lag

2017-03-15 Thread Thomas Munro
get from your definition to mine * lag numbers reported using my definition tell you how long each of the synchronous replication levels take, but with your definition they only do that if you catch them during times when they aren't showing the special case 00:00:00; a fast standby running any w

Re: [HACKERS] pg_ls_waldir() & pg_ls_logdir()

2017-03-16 Thread Thomas Munro
On Thu, Mar 16, 2017 at 10:40 PM, Dave Page wrote: >> +const int n = snprintf(NULL, 0, "%lld", attrib.st_size); I wonder what the portable printf directive is for off_t. Maybe better to use INT64_FORMAT and cast to int64? -- Thomas Munro http://www.enterprisedb.co

[HACKERS] Size vs size_t

2017-03-16 Thread Thomas Munro
Hi, Noticing that the assembled hackers don't seem to agree on $SUBJECT in new patches, I decided to plot counts of lines matching \ and \ over time. After a very long run in the lead, size_t has recently been left in the dust by Size. -- Thomas Munro http://www.enterprisedb.com -- Sen

Re: [HACKERS] Size vs size_t

2017-03-16 Thread Thomas Munro
| 6 +- src/include/replication/logicallauncher.h | 2 +- src/include/utils/backend_random.h | 2 +- src/include/utils/dsa.h| 10 +- src/include/utils/freepage.h | 24 ++-- src/include/utils/relptr.h | 4 +- 12 files changed, 206 in

[HACKERS] <> join selectivity estimate question

2017-03-16 Thread Thomas Munro
for reading. [1] https://www.postgresql.org/message-id/CAEepm%3D3Og-7-b3WOkiT%3Dc%2B6y3eZ0VVSyb1K%2BSOvF17BO5KAt0A%40mail.gmail.com -- Thomas Munro http://www.enterprisedb.com

Re: [HACKERS] <> join selectivity estimate question

2017-03-17 Thread Thomas Munro
nts, but slightly better because the aggregation has been pushed under the Gather node. See attached. -- Thomas Munro http://www.enterprisedb.com

Re: [HACKERS] <> join selectivity estimate question

2017-03-17 Thread Thomas Munro
On Sat, Mar 18, 2017 at 11:49 AM, Thomas Munro wrote: > On Sat, Mar 18, 2017 at 6:14 AM, Tom Lane wrote: >> After a bit more thought, it seems like the bug here is that "the >> fraction of the LHS that has a non-matching row" is not one minus >> "the fractio

Re: [HACKERS] WIP: [[Parallel] Shared] Hash

2017-03-17 Thread Thomas Munro
On Tue, Mar 14, 2017 at 8:03 AM, Thomas Munro wrote: > On Mon, Mar 13, 2017 at 8:40 PM, Rafia Sabih > wrote: >> In an attempt to test v7 of this patch on TPC-H 20 scale factor I found a >> few regressions, >> Q21: 52 secs on HEAD and 400 secs with this patch > &

Re: [HACKERS] Cluster name in ps output

2014-06-26 Thread Thomas Munro
bly incorrect static initialization with a string literal was intended to make sure logging would work before the GUC machinery has finished setting up default values, but that no longer applies.) Regards, Thomas Munro diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml index e3d1c62

Re: [HACKERS] Cluster name in ps output

2014-06-29 Thread Thomas Munro
k you. > I also think, but haven't done so, we should add a double colon after > the cluster name, so it's not: > > postgres: server1 stats collector process > but > postgres: server1: stats collector process +1 Best regards, Thomas Munro -- Sent via pgsql-hackers maili

[HACKERS] DISTINCT with btree skip scan

2014-07-04 Thread Thomas Munro
tinct to send such half-baked early hacking phase code to the list, but thought it would make sense to demo the concept and then seek advice, warnings, cease and desist notices etc before pressing on down that route!) I would be most grateful for any feedback you might have. Best regards, Thomas

Re: [HACKERS] DISTINCT with btree skip scan

2014-07-05 Thread Thomas Munro
On 5 July 2014 02:03, Vik Fearing wrote: > [1] http://wiki.postgresql.org/wiki/Loose_indexscan Thanks. It talks about DISTINCT, and also about using index when you don't have the leading column in your WHERE clause (as well as MySQL ("loose"), at least Oracle ("skip"), SQLite ("skip"), DB2 ("jump

Re: [HACKERS] SKIP LOCKED DATA (work in progress)

2014-07-24 Thread Thomas Munro
On 24 July 2014 00:52, Thomas Munro wrote: > On 23 July 2014 13:15, David Rowley wrote: >> I'm also wondering about this block of code in general: >> >> if (erm->waitPolicy == RWP_WAIT) >> wait_policy = LockWaitBlock; >> else if (erm->waitPolicy

Re: [HACKERS] SKIP LOCKED DATA (work in progress)

2014-07-26 Thread Thomas Munro
On 24 July 2014 00:52, Thomas Munro wrote: > On 23 July 2014 13:15, David Rowley wrote: >> I'm also wondering about this block of code in general: >> >> if (erm->waitPolicy == RWP_WAIT) >> wait_policy = LockWaitBlock; >> else if (erm->waitPolicy

Re: [HACKERS] SKIP LOCKED DATA (work in progress)

2014-07-26 Thread Thomas Munro
On 26 July 2014 15:43, Tom Lane wrote: > Thomas Munro writes: >> I couldn't find an existing reasonable place to share a single wait >> policy enumeration between parser/planner/executor and the heap access >> module, and I get the feeling that it would be unacce

Re: [HACKERS] SKIP LOCKED DATA (work in progress)

2014-07-27 Thread Thomas Munro
On 27 July 2014 14:31, David Rowley wrote: > On Sun, Jul 27, 2014 at 4:49 AM, Thomas Munro wrote: >> >> Here is a new version of the patch with a single enum LockWaitPolicy >> defined in utils/lockwaitpolicy.h. >> > > That seems much cleaner > > A few mor

Re: [HACKERS] SKIP LOCKED DATA (work in progress)

2014-07-28 Thread Thomas Munro
On 27 July 2014 23:19, Thomas Munro wrote: > On the subject of isolation tests, I think skip-locked.spec is only > producing schedules that reach third of the three 'return > HeapTupleWouldBlock' statements in heap_lock_tuple. I will follow up > with some more thorough

Re: [HACKERS] SKIP LOCKED DATA (work in progress)

2014-07-30 Thread Thomas Munro
de paths that trigger the error, even though you can't see from the output why the error was raised in each case without extra instrumentation (though it did cross my mind that it could be interesting at the very least for testing if the error message were different in each case). If there are no

Re: [HACKERS] SKIP LOCKED DATA (work in progress)

2014-08-01 Thread Thomas Munro
#x27;d be > very inclined, once the above are fixed up to mark the patch ready for > commiter. > > Good work Thanks for all the guidance, I appreciate it! My review karma account is now well overdrawn. Best regards, Thomas Munro diff --git a/doc/src/sgml/ref/select.sgml b/doc/src

[HACKERS] Problem in Parallel Bitmap Heap Scan?

2017-03-21 Thread Thomas Munro
i <> '192.168.1.0/24'::inet) -> Bitmap Index Scan on inet_idx3 (cost=0.00..12.26 rows=17 width=0) Index Cond: (i <> '192.168.1.0/24'::inet) (8 rows) -- Thomas Munro http://www.enterprisedb.com -- Sent via pgsql-hacke

Re: [HACKERS] WIP: [[Parallel] Shared] Hash

2017-03-21 Thread Thomas Munro
d the name to load_(private|shared)_tuple, and made it return NULL to indicate that work_mem would be exceeded. The caller needs to handle that by trying to shrink the hash table. Is this better? On Fri, Mar 10, 2017 at 3:02 PM, Peter Geoghegan wrote: > On Thu, Mar 9, 2017 at 4:29 PM, Thomas

Re: [HACKERS] Parallel tuplesort (for parallel B-Tree index creation)

2017-03-21 Thread Thomas Munro
'm going to experiment with refactoring the v10 parallel CREATE INDEX patch to use the SharedBufFileSet interface from hj-shared-buf-file-v8.patch today and see what problems I run into. -- Thomas Munro http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgre

[HACKERS] Aggregates and row types

2017-03-21 Thread Thomas Munro
it's something I noticed and thought I'd ask about, when I was trying (and failing) to find a query that would get transient types into a hash table in a parallel worker for a test case yesterday. -- Thomas Munro http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list

Re: [HACKERS] Problem in Parallel Bitmap Heap Scan?

2017-03-21 Thread Thomas Munro
#x27;t that one row short? What happened to this one? 10.0.0.0/8 | 10::/8 -- Thomas Munro http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] WIP: [[Parallel] Shared] Hash

2017-03-22 Thread Thomas Munro
Hi, Here is a new version addressing feedback from Peter and Andres. Please see below. On Wed, Mar 22, 2017 at 3:18 PM, Peter Geoghegan wrote: > On Tue, Mar 21, 2017 at 5:07 AM, Thomas Munro > wrote: >>> buffile.c should stop pretending to care about anything other than >

Re: [HACKERS] Measuring replay lag

2017-03-22 Thread Thomas Munro
next. > > Not sure whether this a 6 day lag, or we should show NULL because we > are up to date. Hah. Apologies for the delay -- I will post a patch with documentation as requested within 24 hours. -- Thomas Munro http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Measuring replay lag

2017-03-22 Thread Thomas Munro
On Thu, Mar 23, 2017 at 12:12 AM, Simon Riggs wrote: > On 22 March 2017 at 11:03, Thomas Munro wrote: > >> Hah. Apologies for the delay -- I will post a patch with >> documentation as requested within 24 hours. > > Thanks very much. I'll reserve time to commi

Re: [HACKERS] Measuring replay lag

2017-03-22 Thread Thomas Munro
ms seeing a stale value looks plain > wrong (and will cause no end of questions from people asking why lag > is still showing when their system isn't doing anything). Cool, and now Simon has agreed to this too. > I suggest the documentation of these columns needs to be extended to menti

Re: [HACKERS] WIP: [[Parallel] Shared] Hash

2017-03-23 Thread Thomas Munro
7;t. Thanks! > Unfortunately, that's about the only useful piece of feedback that I > can think of right now -- be more explicit about what is permissible > and not permissible in this area, and do something with > pgstat_report_tempfile(). This is a bit like the > unlink()-

Re: [HACKERS] Measuring replay lag

2017-03-23 Thread Thomas Munro
o call it. +system. Monitoring systems should choose whether the represent this +as missing data, zero or continue to display the last known value. s/whether the/whether to/ -- Thomas Munro http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To

Re: [HACKERS] ICU integration

2017-03-23 Thread Thomas Munro
e CFLAGS-like variables. Or am I misunderstanding what pkg-config is supposed to be doing for us here? -- Thomas Munro http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Measuring replay lag

2017-03-23 Thread Thomas Munro
g write and flush lag too, and to Abhijit, Sawada-san, Ian, Craig and Robert for valuable feedback. -- Thomas Munro http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] delta relations in AFTER triggers

2017-03-23 Thread Thomas Munro
On Tue, Mar 14, 2017 at 7:51 AM, Kevin Grittner wrote: > On Sun, Mar 12, 2017 at 4:08 PM, Thomas Munro > wrote: >> I found a new way to break it: run the trigger function so >> that the plan is cached by plpgsql, then ALTER TABLE incompatibly, >> then run the trig

Re: [HACKERS] delta relations in AFTER triggers

2017-03-23 Thread Thomas Munro
On Fri, Mar 24, 2017 at 1:14 PM, Thomas Munro wrote: > If that's fixed and the permissions question can be waved away by > saying it's the same as the per-row situation, my only other comment > would be a bikeshed issue: Enr isn't a great name for a struct. One more

Re: [HACKERS] pg_serial early wraparound

2017-03-24 Thread Thomas Munro
uickly to test wraparound scenarios no longer works, since this new assertion was added in ea42cc18. That was committed just a few hours before you tested this. Bad luck for me! > The new status of this patch is: Waiting on Author It's not urgent, it's just cleanup work, so I've n

Re: [HACKERS] delta relations in AFTER triggers

2017-03-24 Thread Thomas Munro
On Fri, Mar 24, 2017 at 5:36 PM, Thomas Munro wrote: > On Fri, Mar 24, 2017 at 1:14 PM, Thomas Munro > wrote: >> If that's fixed and the permissions question can be waved away by >> saying it's the same as the per-row situation, my only other comment >> woul

Re: [HACKERS] dsm.c API tweak

2017-03-24 Thread Thomas Munro
il explicitly detached. Creating or attaching with a NULL CurrentResourceOwner is equivalent to creating or attaching with a non-NULL CurrentResourceOwner and then calling dsm_pin_mapping()." Then dsm_attach() needs to say "See the note above dsm_create() about the CurrentResourceOwner.&qu

Re: [HACKERS] dsm.c API tweak

2017-03-24 Thread Thomas Munro
On Sat, Mar 25, 2017 at 1:59 PM, Thomas Munro wrote: > On Sat, Mar 25, 2017 at 12:35 PM, Alvaro Herrera > wrote: >> Alvaro Herrera wrote: >>> Per >>> https://postgr.es/m/CAEepm=11ma_Z1HoPxPcSCANnh5ykHORa=hca1u1v1+5s_jw...@mail.gmail.com >>> it seems that th

Re: [HACKERS] Problem in Parallel Bitmap Heap Scan?

2017-03-25 Thread Thomas Munro
On Sat, Mar 25, 2017 at 6:04 PM, Amit Kapila wrote: > On Tue, Mar 21, 2017 at 5:51 PM, Dilip Kumar wrote: >> On Tue, Mar 21, 2017 at 4:47 PM, Thomas Munro >> wrote: >>> I noticed a failure in the inet.sql test while running the regression >>> tests with parallel

Re: [HACKERS] WIP: [[Parallel] Shared] Hash

2017-03-25 Thread Thomas Munro
for Windows. I believe it doesn't allow files to be unlinked if they are open, and I see that DSM segments are cleaned up in resowner's phase == RESOURCE_RELEASE_BEFORE_LOCKS and files are closed in phase == RESOURCE_RELEASE_AFTER_LOCKS. Hmm. -- Thomas Munro http://www.enterprised

Re: [HACKERS] delta relations in AFTER triggers

2017-03-26 Thread Thomas Munro
On Fri, Mar 24, 2017 at 1:14 PM, Thomas Munro wrote: > On Tue, Mar 14, 2017 at 7:51 AM, Kevin Grittner wrote: >> On Sun, Mar 12, 2017 at 4:08 PM, Thomas Munro >> wrote: >>> I found a new way to break it: run the trigger function so >>> that the plan is ca

Re: [HACKERS] [sqlsmith] Unpinning error in parallel worker

2017-03-26 Thread Thomas Munro
On Mon, Mar 27, 2017 at 4:18 AM, Andreas Seltenreich wrote: > Hi, > > today's testing with master as of d253b0f6e3 yielded two clusters that > stopped processing queries. Symptoms: > > [...] Thanks Andreas. Investigating. -- Thomas Munro http://www.enterprisedb.c

Re: [HACKERS] WIP: [[Parallel] Shared] Hash

2017-03-26 Thread Thomas Munro
there are two kinds of partitioning going on at the same time. Partitioning the keyspace into batch numbers, and then the arbitrary partitioning that comes from each participant processing partial plans. This is how SharedBufFileSet finishes up managing a 2D matrix of BufFiles. You might argu

Re: [HACKERS] WIP: [[Parallel] Shared] Hash

2017-03-26 Thread Thomas Munro
On Mon, Mar 27, 2017 at 11:03 AM, Thomas Munro >> Is there a risk that this ends up running afoul of filename length >> limits on some platforms? > > Hmm. I didn't think so. Do we have a project guideline on maximum > path lengths based on some kind of survey?

Re: [HACKERS] WIP: [[Parallel] Shared] Hash

2017-03-26 Thread Thomas Munro
On Mon, Mar 27, 2017 at 12:12 PM, Peter Geoghegan wrote: > On Sun, Mar 26, 2017 at 3:41 PM, Thomas Munro > wrote: >>> 1. Segments are what buffile.c already calls the individual >>> capped-at-1GB files that it manages. They are an implementation >>> detail tha

Re: [HACKERS] [sqlsmith] Unpinning error in parallel worker

2017-03-26 Thread Thomas Munro
On Mon, Mar 27, 2017 at 8:38 AM, Thomas Munro wrote: > On Mon, Mar 27, 2017 at 4:18 AM, Andreas Seltenreich > wrote: >> Hi, >> >> today's testing with master as of d253b0f6e3 yielded two clusters that >> stopped processing queries. Symptoms: >> &g

Re: [HACKERS] WIP: [[Parallel] Shared] Hash

2017-03-26 Thread Thomas Munro
On Sun, Mar 26, 2017 at 3:56 PM, Thomas Munro wrote: > But... what you said above must be a problem for Windows. I believe > it doesn't allow files to be unlinked if they are open, and I see that > DSM segments are cleaned up in resowner's phase == > RESOURCE_RELEASE_BEFO

[HACKERS] O(1) DSM handle operations

2017-03-27 Thread Thomas Munro
ure to fit into key_t, and also the use of the special key_t sentinel values IPC_PRIVATE which somehow needs to be avoided. Do any systems really have sizeof(key_t) < 4? Point (2) seems to be implementable independently of (1). -- Thomas Munro http://www.enterprisedb.com -- Sent via pgsql-h

Re: [HACKERS] O(1) DSM handle operations

2017-03-27 Thread Thomas Munro
On Tue, Mar 28, 2017 at 3:52 PM, Robert Haas wrote: > On Mon, Mar 27, 2017 at 5:13 PM, Thomas Munro > wrote: >> This is just a thought for discussion, no patch attached... >> >> DSM operations dsm_create(), dsm_attach(), dsm_unpin_segment() perform >> linear sea

Re: [HACKERS] Getting server crash after running sqlsmith

2017-03-28 Thread Thomas Munro
to deliver the signal on an overloaded machine, maybe there is always a new SIGUSR1 and PMSIGNAL_ROTATE_LOGFILE waiting once the signal handler reaches PG_SETMASK(&UnBlockSig), at which point it immediately recurses into the signal handler until it blows the stack. -- Thomas Munro http://ww

Re: [HACKERS] [sqlsmith] Unpinning error in parallel worker

2017-03-28 Thread Thomas Munro
On Mon, Mar 27, 2017 at 6:53 PM, Thomas Munro wrote: > On Mon, Mar 27, 2017 at 8:38 AM, Thomas Munro > wrote: >> On Mon, Mar 27, 2017 at 4:18 AM, Andreas Seltenreich >> wrote: >>> Hi, >>> >>> today's testing with master as of d253b0f6e3 yielde

Re: [HACKERS] strange parallel query behavior after OOM crashes

2017-03-30 Thread Thomas Munro
code that crashed. I wonder if the DSM_CREATE_NULL_IF_MAXSEGMENTS case could be being triggered because the DSM control is somehow confused? -- Thomas Munro http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] WIP: [[Parallel] Shared] Hash

2017-03-30 Thread Thomas Munro
misation implemented. Some smaller changes and miles of feedback inline below: On Mon, Mar 27, 2017 at 11:03 AM, Thomas Munro wrote: > On Mon, Mar 27, 2017 at 9:41 AM, Andres Freund wrote: >> SharedBufFile allows temporary files to be created by one backend and >> then exporte

Re: [HACKERS] delta relations in AFTER triggers

2017-04-02 Thread Thomas Munro
showing what I mean. Please also find attached a rebased patch to add pl/python support, and new equivalent patches for pl/perl and pl/tcl. I am planning to add these to PG11 CF1, unless you think we should be more aggressive given the extra time? -- Thomas Munro http://www.enterprisedb.com s

Re: [HACKERS] delta relations in AFTER triggers

2017-04-02 Thread Thomas Munro
On Mon, Apr 3, 2017 at 3:50 PM, Thomas Munro wrote: > Please also find attached a rebased patch to add pl/python support, > and new equivalent patches for pl/perl and pl/tcl. I am planning to > add these to PG11 CF1, unless you think we should be more aggressive > given the ext

Re: [HACKERS] SERIALIZABLE with parallel query

2017-04-03 Thread Thomas Munro
On Tue, Apr 4, 2017 at 6:41 AM, Andres Freund wrote: > Hi, > > On 2017-03-11 15:19:23 +1300, Thomas Munro wrote: >> Here is a rebased patch. > > It seems that this patch is still undergoing development, review and > performance evaluation. Therefore it seems like it

Re: [HACKERS] WIP: [[Parallel] Shared] Hash

2017-04-03 Thread Thomas Munro
On Tue, Apr 4, 2017 at 9:11 AM, Andres Freund wrote: > Hi, > > On 2017-03-31 17:53:12 +1300, Thomas Munro wrote: >> Thanks very much to Rafia for testing, and to Andres for his copious >> review feedback. Here's a new version. Changes: > > I unfortunately thin

Re: [HACKERS] delta relations in AFTER triggers

2017-04-03 Thread Thomas Munro
On Tue, Apr 4, 2017 at 3:41 AM, Kevin Grittner wrote: > On Mon, Apr 3, 2017 at 8:59 AM, Tom Lane wrote: >> Thomas Munro writes: >>> Or perhaps the code to inject trigger data transition tables into SPI >>> (a near identical code block these three patches) should be s

<    1   2   3   4   5   6   7   8   9   >