Re: [HACKERS] PoC: Make it possible to disallow WHERE-less UPDATE and DELETE

2016-09-25 Thread Thomas Munro
On Mon, Sep 26, 2016 at 1:18 PM, Thomas Munro wrote: > underscore to a minus in various places. That fixes these errors: Correction: s/these errors:/the above errors./ -- Thomas Munro http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To m

Re: [HACKERS] PoC: Make it possible to disallow WHERE-less UPDATE and DELETE

2016-09-25 Thread Thomas Munro
at a large company with many database users. I think experienced users probably initially felt mollycoddled when they first encountered the error but I'm sure that some were secretly glad of its existence from time to time... I think it's a useful feature for users who want it, and

Re: [HACKERS] [GENERAL] C++ port of Postgres

2016-09-25 Thread Thomas Munro
* from a transaction that is not yet visible to snapshots; * compare the comments at the head of tqual.c. */ - *current = thisentry; + COPY_QUEUE_POS(*current, thisentry); reachedStop = true; break; } -- Thomas Munro http://www.enterprisedb.com

Re: [HACKERS] ICU integration

2016-09-24 Thread Thomas Munro
On Sat, Sep 24, 2016 at 10:13 PM, Peter Geoghegan wrote: > On Fri, Sep 23, 2016 at 7:27 AM, Thomas Munro > wrote: >> It looks like varstr_abbrev_convert calls strxfrm unconditionally >> (assuming TRUST_STRXFRM is defined). This needs to >> use ucol_getSortKey instead whe

Re: [HACKERS] Refactor StartupXLOG?

2016-09-24 Thread Thomas Munro
On Sat, Sep 24, 2016 at 8:24 PM, Heikki Linnakangas wrote: > On 09/24/2016 05:01 AM, Thomas Munro wrote: >> >> What would the appetite be for that kind of refactoring work, >> considering the increased burden on committers who have to backpatch >> bug fixes? Is it a

[HACKERS] Complete LOCK TABLE ... IN ACCESS|ROW|SHARE

2016-09-23 Thread Thomas Munro
Hi After LOCK TABLE ... IN ACCESS|ROW|SHARE we run out of completions. Here's a patch to improve that, for November. -- Thomas Munro http://www.enterprisedb.com complete-lock-table.patch Description: Binary data -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To

[HACKERS] Refactor StartupXLOG?

2016-09-23 Thread Thomas Munro
itters who have to backpatch bug fixes? Is it a project goal to reduce the size of large complicated functions like StartupXLOG and heap_update? It seems like a good way for new players to learn how they work. -- Thomas Munro http://www.enterprisedb.com refactor-startupxlog-sketch.patch Descri

Re: [HACKERS] ICU integration

2016-09-22 Thread Thomas Munro
.out @@ -0,0 +1,1076 @@ +/* + * This test is for Linux/glibc systems and assumes that a full set of + * locales is installed. It must be run in a database with UTF-8 encoding, + * because other encodings don't support all the characters used. + */ Should say ICU. [1] https://www.postgresql.

Re: [HACKERS] Tracking wait event for latches

2016-09-22 Thread Thomas Munro
On Fri, Sep 23, 2016 at 1:49 AM, Robert Haas wrote: > On Wed, Sep 21, 2016 at 5:49 PM, Thomas Munro > wrote: >>> Moreover, it's pretty confusing that we have this general concept of >>> wait events in pg_stat_activity, and then here the specific type of >>&g

Re: [HACKERS] Tracking wait event for latches

2016-09-21 Thread Thomas Munro
On Thu, Sep 22, 2016 at 1:23 AM, Robert Haas wrote: > On Tue, Sep 20, 2016 at 7:13 PM, Thomas Munro > wrote: >> This paragraph seems a bit confused. I suggest something more like >> this: "The server process is waiting for one or more sockets, a timer >> or an

Re: [HACKERS] Tracking wait event for latches

2016-09-20 Thread Thomas Munro
On Wed, Sep 21, 2016 at 3:40 PM, Michael Paquier wrote: > On Wed, Sep 21, 2016 at 8:13 AM, Thomas Munro > wrote: >> It looks like this array wants to be in alphabetical order, but it >> isn't quite. Also, perhaps a compile time assertion about the size of >> the a

Re: [HACKERS] Tracking wait event for latches

2016-09-20 Thread Thomas Munro
aps an extra boolean would be needed to record whether postmaster death is in there so we could deduce whether there are any sockets). It would be interesting specifically for the case of FDWs where it would be nice to be able to see clearly that it's waiting for a remote server ("Socket"). It may also be interesting to know if there is a timeout. Postmaster death doesn't seem newsworthy, we're nearly always also waiting for that exceptional event so it'd just be clutter to report it. [1] http://stackoverflow.com/questions/147267/easy-way-to-use-variables-of-enum-types-as-string-in-c -- Thomas Munro http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] PATCH: Keep one postmaster monitoring pipe per process

2016-09-19 Thread Thomas Munro
code in basebackup.c also waits for WL_POSTMASTER_DEATH but doesn't check for it in the return value *or* call PostmasterIsAlive(). I'm not sure what to make of that. I didn't test it but it looks like maybe it would continue running after postmaster death but not honour the thr

Re: [HACKERS] recovery_min_apply-delay and remote_apply

2016-09-16 Thread Thomas Munro
dangerous > if someone doesn't take care enough. Yes, I missed that sentence. Thanks. > I think we need a doc patch for that at least, see attached patch against > master, but 9.6 should have a corrected one, too. +1 -- Thomas Munro http://www.enterprisedb.com -- Sent via pgsql

Re: [HACKERS] kqueue

2016-09-15 Thread Thomas Munro
On Thu, Sep 15, 2016 at 11:04 AM, Thomas Munro wrote: > On Thu, Sep 15, 2016 at 10:48 AM, Keith Fiske wrote: >> Thomas Munro brought up in #postgresql on freenode needing someone to test a >> patch on a larger FreeBSD server. I've got a pretty decent machine (3.1Ghz >>

Re: [HACKERS] PATCH: Keep one postmaster monitoring pipe per process

2016-09-15 Thread Thomas Munro
essage-id/flat/CAEepm%3D2i78TOJeV4O0-0meiihiRfVQ29ur7%3DMBHxsUKaPSWeAg%40mail.gmail.com -- Thomas Munro http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] make async slave to wait for lsn to be replayed

2016-09-14 Thread Thomas Munro
ng 'wait forever', and another meaning 'don't wait at all: if it's not applied yet, then timeout immediately'. In any case I'd consider using names for special wait times and using those for clarity: WAITLSN_INFINITE_WAIT, WAITLSN_NO_WAIT. Later I'll have feedba

Re: [HACKERS] kqueue

2016-09-14 Thread Thomas Munro
On Thu, Sep 15, 2016 at 10:48 AM, Keith Fiske wrote: > Thomas Munro brought up in #postgresql on freenode needing someone to test a > patch on a larger FreeBSD server. I've got a pretty decent machine (3.1Ghz > Quad Core Xeon E3-1220V3, 16GB ECC RAM, ZFS mirror on WD Red HDD) so off

Re: [HACKERS] kqueue

2016-09-13 Thread Thomas Munro
51654 TPS -> 55739 TPS = 7.9% improvement GCC 6.1.0 from MacPorts: 52552 TPS -> 55143 TPS = 4.9% improvement I reran the tests under FreeBSD 10.3 on a 4 core laptop and again saw absolutely no measurable difference at 1, 4 or 24 clients. Maybe a big enough server could be made to contend on the p

Re: [HACKERS] Merge Join with an Index Optimization

2016-09-11 Thread Thomas Munro
trictly necessary, I don't know). There are a whole lot of interesting execution tricks that could be enabled by btree skipping (see Oracle, DB2, MySQL, SQLite). DISTINCT was the simplest thing I could think of where literally every other RDBMS beats us because we don't have it. But t

Re: [HACKERS] kqueue

2016-09-09 Thread Thomas Munro
bsd-src/blob/master/sys/kern/kern_event.c [5] https://github.com/openbsd/src/blob/master/sys/kern/kern_event.c [6] https://github.com/opensource-apple/xnu/blob/master/bsd/kern/kern_event.c [7] http://marc.info/?l=freebsd-arch&m=98147346707952&w=2 -- Thomas Munro http://www.enterprisedb.com

Re: [HACKERS] [RFC] Change the default of update_process_title to off

2016-08-24 Thread Thomas Munro
ing several GB of buffer pool. I wonder if that might help Postgres on Windows. This could be useful as a starting point to test that theory: https://www.postgresql.org/message-id/CAEepm%3D075-bgHi_VDt4SCAmt%2Bo_%2B1XaRap2zh7XwfZvT294oHA%40mail.gmail.com -- Thomas Munro http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Server crash due to SIGBUS(Bus Error) when trying to access the memory created using dsm_create().

2016-08-22 Thread Thomas Munro
0774us base + memset = 602925us base + fallocate + memset = 655433us -- Thomas Munro http://www.enterprisedb.com #include #include #include #include #include #include #include #define SEGMENT_NAME "/my_test_segment" int main(int argc, char *argv[]) { int loops, i; off_t size; bool

Re: [HACKERS] Server crash due to SIGBUS(Bus Error) when trying to access the memory created using dsm_create().

2016-08-22 Thread Thomas Munro
On Tue, Aug 23, 2016 at 8:41 AM, Robert Haas wrote: > On Tue, Aug 16, 2016 at 7:41 PM, Thomas Munro > wrote: >> I still think it's worth thinking about something along these lines on >> Linux only, where holey Swiss tmpfs files can bite you. Otherwise >> disablin

Re: [HACKERS] dsm_unpin_segment

2016-08-22 Thread Thomas Munro
andle count of postmaster is incremented > by 1 and then by using dsm_demo_unpin_segment() unpinned the segment > which decrements the Handle count in Postmaster. Thanks! -- Thomas Munro http://www.enterprisedb.com dsm-unpin-segment-v4.patch Description: Binary data -- Sent via pgsql-ha

Re: [HACKERS] dsm_unpin_segment

2016-08-21 Thread Thomas Munro
On Sat, Aug 20, 2016 at 11:37 PM, Amit Kapila wrote: > On Tue, Aug 9, 2016 at 10:07 AM, Thomas Munro > wrote: >> On Tue, Aug 9, 2016 at 12:53 PM, Tom Lane wrote: >>> The larger picture here is that Robert is exhibiting a touching but >>> unfounded faith that exte

Re: [HACKERS] synchronous_commit = remote_flush

2016-08-21 Thread Thomas Munro
re likely you'd want some combination: 2-safe or group-safe on some subset of servers to satisfy your durability requirements, and applied on some other perhaps larger subset of servers for consistency. But this is just water cooler handwaving. [1] https://infoscience.epfl.ch/rec

Re: [HACKERS] synchronous_commit = remote_flush

2016-08-21 Thread Thomas Munro
On Fri, Aug 19, 2016 at 7:32 PM, Masahiko Sawada wrote: > On Fri, Aug 19, 2016 at 5:25 AM, Robert Haas wrote: >> On Thu, Aug 18, 2016 at 12:22 AM, Thomas Munro >> wrote: >>> To do something about the confusion I keep seeing about what exactly >>> &qu

[HACKERS] synchronous_commit = remote_flush

2016-08-17 Thread Thomas Munro
t;on" to control syncrep, and also the people who use "off" vs "on" to control asynchronous commit on single-node systems. Is there any sensible way to do that, or is it not broken and I should pipe down, or is it just far too entrenched and never going to change? --

Re: [HACKERS] WIP: Barriers

2016-08-17 Thread Thomas Munro
On Tue, Aug 16, 2016 at 1:55 AM, Robert Haas wrote: > On Sat, Aug 13, 2016 at 7:18 PM, Thomas Munro > wrote: >> My use case for this is coordinating the phases of parallel hash >> joins, but I strongly suspect there are other cases. Parallel sort >> springs to mind,

Re: [HACKERS] support for NEXT VALUE FOR expression

2016-08-16 Thread Thomas Munro
ttps://www.postgresql.org/message-id/flat/CADLWmXUY2oo4XObQWF3yPUSK%3D5uEiSV%3DeTyLrnu%3DRzteOy%2B3Lg%40mail.gmail.com > I remain of the opinion that using spec-compliant syntax for > non-spec-compliant behavior isn't a great advance. -- Thomas Munro http://www.enterprisedb.com -- Sent via pgsql-hackers

Re: [HACKERS] Server crash due to SIGBUS(Bus Error) when trying to access the memory created using dsm_create().

2016-08-16 Thread Thomas Munro
On Wed, Aug 17, 2016 at 4:50 AM, Robert Haas wrote: > On Fri, Aug 12, 2016 at 9:22 PM, Thomas Munro > wrote: >> On Sat, Aug 13, 2016 at 8:26 AM, Thomas Munro >> wrote: >>> On Sat, Aug 13, 2016 at 2:08 AM, Tom Lane wrote: >>>> amul sul writes: >>&g

Re: [HACKERS] Anyone want to update our Windows timezone map?

2016-08-15 Thread Thomas Munro
of thing: http://www.unicode.org/cldr/charts/29/supplemental/zone_tzid.html Could that be a better source than dumping stuff from arbitrary Windows versions? -- Thomas Munro http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To

Re: [HACKERS] WIP: Barriers

2016-08-15 Thread Thomas Munro
. I realise that using signals for this sort of thing is a bit unusual outside the Postgres universe, but won't a semaphore-based implementation require just as many system calls, context switches and scheduling operations? -- Thomas Munro http://www.enterprisedb.com barrier-test.patc

Re: [HACKERS] condition variables

2016-08-14 Thread Thomas Munro
On Mon, Aug 15, 2016 at 5:58 PM, Thomas Munro wrote: > Also, I have attached a v2->v3 diff ... Ugh. I meant a v1->v2 diff. -- Thomas Munro http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscript

Re: [HACKERS] condition variables

2016-08-14 Thread Thomas Munro
On Sun, Aug 14, 2016 at 9:04 AM, Thomas Munro wrote: > On Fri, Aug 12, 2016 at 9:47 AM, Robert Haas wrote: >> [condition-variable-v1.patch] > > Don't you need to set proc->cvSleeping = false in ConditionVariableSignal? I poked at this a bit... OK, a lot... and have some

[HACKERS] WIP: Barriers

2016-08-13 Thread Thomas Munro
ostgresql.org/message-id/flat/CAEepm%3D0Vvr9zgwHt67RwuTfwMEby1GiGptBk3xFPDbbgEtZgMg%40mail.gmail.com [3] https://www.postgresql.org/message-id/flat/cab7npqtghfouhag1ejrvskn8-e5fpqvhm7al0tafsdzjqg_...@mail.gmail.com -- Thomas Munro http://www.enterprisedb.com barrier-v1.patch Description: Binary d

Re: [HACKERS] condition variables

2016-08-13 Thread Thomas Munro
On Fri, Aug 12, 2016 at 9:47 AM, Robert Haas wrote: > [condition-variable-v1.patch] Don't you need to set proc->cvSleeping = false in ConditionVariableSignal? -- Thomas Munro http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make

Re: [HACKERS] Server crash due to SIGBUS(Bus Error) when trying to access the memory created using dsm_create().

2016-08-12 Thread Thomas Munro
On Sat, Aug 13, 2016 at 8:26 AM, Thomas Munro wrote: > On Sat, Aug 13, 2016 at 2:08 AM, Tom Lane wrote: >> amul sul writes: >>> When I am calling dsm_create on Linux using the POSIX DSM implementation >>> can succeed, but result in SIGBUS when later try to access the

Re: [HACKERS] Server crash due to SIGBUS(Bus Error) when trying to access the memory created using dsm_create().

2016-08-12 Thread Thomas Munro
t looks like if we used fallocate or posix_fallocate in the dsm_impl_posix case we'd get a nice ESPC error, instead of success-but-later-SIGBUS-on-access. Whether there is *also* the possibility of overcommit biting you later I don't know, but I suspect that's an independent pr

Re: [HACKERS] condition variables

2016-08-11 Thread Thomas Munro
do similar extra work on top for my join points concept, because although I do need waiters to be poked at the time worker aborts or dies, one goodbye prod isn't enough: I'd also need to adjust the join point's set of workers, or put it into error state. -- Thomas Munro http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] dsm_unpin_segment

2016-08-09 Thread Thomas Munro
On Wed, Aug 10, 2016 at 2:43 PM, Jim Nasby wrote: > On 8/9/16 6:14 PM, Thomas Munro wrote: >> The can't be static, they need to be in shared memory, because we also >> want to protect against two *different* backends pinning it. > > Right, this would strictly protect

Re: [HACKERS] dsm_unpin_segment

2016-08-09 Thread Thomas Munro
ot pinned"); Those checks could arguably be assertions rather than errors, but I don't think that pin/unpin operations will ever be high frequency enough for it to be worth avoiding those instructions in production builds. The whole reason for pinning segments is likely to be able to reu

Re: [HACKERS] dsm_unpin_segment

2016-08-08 Thread Thomas Munro
f the existing cleanup mechanism for the control segment, while making sure that the auxiliary segments get pinned and unpinned exactly once. I'll have more to say about that when I post that patch... -- Thomas Munro http://www.enterprisedb.com dsm-unpin-segment-v2.patch Descr

Re: [HACKERS] dsm_unpin_segment

2016-08-08 Thread Thomas Munro
On Tue, Aug 9, 2016 at 12:22 PM, Robert Haas wrote: > On Mon, Aug 8, 2016 at 7:33 PM, Tom Lane wrote: >> Thomas Munro writes: >>> Please find attached a patch to add a corresponding operation >>> 'dsm_unpin_segment'. This gives you a way to ask for the

[HACKERS] dsm_unpin_segment

2016-08-08 Thread Thomas Munro
to keep the segment alive. This patch needs to close that handle when unpinning. Amazingly, that can be done without any cooperation from the postmaster. I'd be grateful for any feedback or thoughts, and will add this to the commitfest. -- Thomas Munro http://www.enterprisedb.com dsm-unpin-seg

[HACKERS] Consolidate 'unique array values' logic into a reusable function?

2016-08-06 Thread Thomas Munro
posted a while back that consolidates popcount and ffs/fls implementations. I don't like code duplication :-) -- Thomas Munro http://www.enterprisedb.com array-unique.patch Description: Binary data -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes t

Re: [HACKERS] [sqlsmith] FailedAssertion("!(k == indices_count)", File: "tsvector_op.c", Line: 511)

2016-08-03 Thread Thomas Munro
* to the same index. */ - if (k < indices_count && i == indices_to_delete[k]) + while (k < indices_count && i == indices_to_delete[k]) { + drop_lexeme = true; k++; - continue

Re: [HACKERS] Implementing full UTF-8 support (aka supporting 0x00)

2016-08-03 Thread Thomas Munro
https://wiki.postgresql.org/wiki/Todo:ICU ucoll_strcoll takes explicit lengths (though optionally accepts -1 for null terminated mode). http://userguide.icu-project.org/strings#TOC-Using-C-Strings:-NUL-Terminated-vs.-Length-Parameters -- Thomas Munro http://www.enterprisedb.com -- Sent via

Re: [HACKERS] PostmasterContext survives into parallel workers!?

2016-08-01 Thread Thomas Munro
1_handler. That's likely excessive, but it's what > we've got at the moment. I found this apparently unresolved bug report about glibc fork() inside a signal handler deadlocking: https://sourceware.org/bugzilla/show_bug.cgi?id=4737 I wonder if that could bite postmaster. It's

Re: [HACKERS] Combining hash values

2016-07-31 Thread Thomas Munro
On Mon, Aug 1, 2016 at 4:34 AM, Tom Lane wrote: > Thomas Munro writes: >> 2. I suspect that this algorithm for combining hashes is weak, and >> could amplify weaknesses in the hash functions feeding it. > > Very possibly, but ... Concrete example: suppose a clever data

[HACKERS] Combining hash values

2016-07-30 Thread Thomas Munro
knuth-derive-a [4] http://community.haskell.org/~simonmar/base/src/Data-HashTable.html -- Thomas Munro http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

[HACKERS] Double invocation of InitPostmasterChild in bgworker with -DEXEC_BACKEND

2016-07-28 Thread Thomas Munro
d is called twice. I can successfully use regular parallel query workers and bgworkers created by extensions if I apply the attached patch. -- Thomas Munro http://www.enterprisedb.com fix.patch Description: Binary data -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make cha

Re: [HACKERS] LWLocks in DSM memory

2016-07-28 Thread Thomas Munro
On Fri, Jul 29, 2016 at 2:12 AM, Robert Haas wrote: > On Thu, Jul 28, 2016 at 12:12 AM, Thomas Munro > wrote: >> On Tue, Jul 26, 2016 at 6:00 AM, Andres Freund wrote: >>> I think the better fix here would acutally be to get rid of a pointer >>> based list here,

Re: [HACKERS] LWLocks in DSM memory

2016-07-27 Thread Thomas Munro
might require a wider CAS than we have?). I wonder if a proclist with an optional lock-free interface would also be interesting for syncrep queues and others. -- Thomas Munro http://www.enterprisedb.com lwlocks-in-dsm-v2.patch Description: Binary data -- Sent via pgsql-hackers ma

Re: [HACKERS] Proposal: revert behavior of IS NULL on row types

2016-07-26 Thread Thomas Munro
ays, but in fact our NOT NULL constraints are equivalent to CHECK (column_name IS DISTINCT FROM NULL). Should we update the documentation with something like the attached? -- Thomas Munro http://www.enterprisedb.com not-null-does-not-mean-check-is-not-null.patch Description: Binary data -- Se

Re: [HACKERS] LWLocks in DSM memory

2016-07-24 Thread Thomas Munro
On Mon, Jul 25, 2016 at 3:02 PM, Thomas Munro wrote: > I measured the following times for unpatched master, on my 4 core laptop: > > 16 workers = 73.067s, 74.869s, 75.338s > 8 workers = 65.846s, 67.622s, 68.039s > 4 workers = 68.763s, 68.980s, 69.035s <-- curiously slowe

Re: [HACKERS] LWLocks in DSM memory

2016-07-24 Thread Thomas Munro
On Sun, Jul 24, 2016 at 1:10 AM, Thomas Munro wrote: > One solution could be to provide a non-circular variant of the dlist > interface that uses NULL list termination. I've attached a quick > sketch of something like that which seems to work correctly. It is > only lightly

[HACKERS] LWLocks in DSM memory

2016-07-23 Thread Thomas Munro
sage-id/ca+tgmobjia49ccj0zazbwavv7nkgyt+1zo5cwxkh9aahgn2...@mail.gmail.com -- Thomas Munro http://www.enterprisedb.com lwlocks-in-dsm.patch Description: Binary data -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] SELECT DISTINCT never uses an index?

2016-07-07 Thread Thomas Munro
mail.com I hope to return to that and some related ideas eventually as I learn more about the relevant areas of the source code, if someone doesn't beat me to it. https://wiki.postgresql.org/wiki/Loose_indexscan shows a recursive CTE that does the same thing at a higher level. -- Thomas Munro

Re: [HACKERS] Forthcoming SQL standards about JSON and Multi-Dimensional Arrays (FYI)

2016-07-03 Thread Thomas Munro
eferred back to Working Group". Is that how they say "returned with feedback"? ISO/IEC PDTR 19075-5 (Row Pattern Recognition) has also reached stage 30.60. Does anyone know what that one is about? Maybe something like MATCH_RECOGNIZE in Oracle? [1] http://www.iso.org/iso/home/s

Re: [HACKERS] [sqlsmith] crashes in RestoreSnapshot on hot standby

2016-06-30 Thread Thomas Munro
On Fri, Jul 1, 2016 at 3:25 PM, Amit Kapila wrote: > On Fri, Jul 1, 2016 at 8:48 AM, Thomas Munro > wrote: >> If serialized_snapshot->xcnt == 0, then snapshot->xip never gets >> initialized to a non-NULL value. Then if serialized_snapshot->subxcnt >> > 0, we

Re: [HACKERS] [sqlsmith] crashes in RestoreSnapshot on hot standby

2016-06-30 Thread Thomas Munro
thing like the attached. Theory 2: The DSM segment was deleted underneath us. We can see that it was not mapped by the time GDB dumped that (start_address is not accessible). Theory 3: Somehow the xcnt or xsubcnt was wrong or the serialized snapshot was truncated, and we read past the end of

Re: [HACKERS] Reviewing freeze map code

2016-06-20 Thread Thomas Munro
a badly timed crash was ever implemented. -- Thomas Munro http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Reviewing freeze map code

2016-06-20 Thread Thomas Munro
origin page. Then after locking the origin page, if it turns out you need a page but didn't get it earlier, asking for a free page with a higher block number than the origin page. -- Thomas Munro http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Reviewing freeze map code

2016-06-20 Thread Thomas Munro
commit f2bfe8a2 said that tqual routines would see the HEAP_XMAX_UNLOGGED flag in the event of a crash before logging (though I'm not sure if the tqual routines ever actually did that). -- Thomas Munro http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To

Re: [HACKERS] Assert(LWLockHeldByMeInMode(lock, LW_EXCLUSIVE))

2016-06-19 Thread Thomas Munro
On Mon, Jun 20, 2016 at 3:43 PM, Craig Ringer wrote: > On 18 June 2016 at 11:28, Thomas Munro > wrote: >> Several times now when reading, debugging and writing code I've wished >> that LWLockHeldByMe assertions specified the expected mode, especially >> where

[HACKERS] Assert(LWLockHeldByMeInMode(lock, LW_EXCLUSIVE))

2016-06-17 Thread Thomas Munro
o the next commitfest. -- Thomas Munro http://www.enterprisedb.com bufmgr-assert-lwlock-mode.patch Description: Binary data test-lwlock-mode.patch Description: Binary data -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription:

Re: [HACKERS] Typo in parallel comment of heap_delete()

2016-06-15 Thread Thomas Munro
l form where I would expect a present subjunctive: "... lest the new path *would* kick ..." /me ducks -- Thomas Munro http://www.enterprisedb.com lest.patch Description: Binary data -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make chan

Re: [HACKERS] Reviewing freeze map code

2016-06-14 Thread Thomas Munro
On Wed, Jun 15, 2016 at 11:43 AM, Thomas Munro wrote: > On Wed, Jun 15, 2016 at 12:44 AM, Robert Haas wrote: >> On Tue, Jun 14, 2016 at 8:11 AM, Robert Haas wrote: >>>>> I noticed that the tuples that it reported were always offset 1 in a >>>>> page, and t

Re: [HACKERS] Reviewing freeze map code

2016-06-14 Thread Thomas Munro
need to protect xmax in the > same way. With this version of the patch, I can no longer get any > TIDs to pop up out of pg_check_visible in my testing. (I haven't run > your test script for lack of the proper Python environment...) I can still reproduce the problem with this

Re: [HACKERS] Reviewing freeze map code

2016-06-13 Thread Thomas Munro
and that we called record_corrupt_item because VM_ALL_VISIBLE returned true but HeapTupleSatisfiesVacuum on the first tuple returned HEAPTUPLE_DELETE_IN_PROGRESS instead of the expected HEAPTUPLE_LIVE. It did that because HEAP_XMAX_COMMITTED was not set and TransactionIdIsInProgress returned true for

Re: [HACKERS] ERROR: ORDER/GROUP BY expression not found in targetlist

2016-06-12 Thread Thomas Munro
allel_workers_per_gather parameter. > The result of Q1 is bellow. Is this bug in the Open items on wiki? I don't see it on the Open Issues list. -- Thomas Munro http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscri

[HACKERS] ERROR: ORDER/GROUP BY expression not found in targetlist

2016-06-12 Thread Thomas Munro
ession not found in targetlist -- Thomas Munro http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

[HACKERS] Reference to UT1

2016-06-05 Thread Thomas Munro
ezone-names.html [2] https://en.wikipedia.org/wiki/Unix_time -- Thomas Munro http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] kqueue

2016-06-02 Thread Thomas Munro
533ac22, which sprinkled some MAXALIGN macros nearby. I've now done the same thing with the kevent struct because it's cheap, uniform with the other cases and could matter on some platforms for the same reason. It's in the September commitfest here: https://commitfest.postgresql.org/10/

Re: [HACKERS] Tracking wait event for latches

2016-06-01 Thread Thomas Munro
very useful for extensions to be able to name their wait points. For example, I'd rather see 'postgres_fdw.pgfdw_get_result' or similar than a vague 'Extension' string which appears not only for all wait points in an extension but also for all extensions. I hope we can fig

[HACKERS] Typo in comment in nbtree.h

2016-06-01 Thread Thomas Munro
ray */ + BlockNumber currPage; /* page referenced by items array */ BlockNumber nextPage; /* page's right link when we scanned it */ /* -- Thomas Munro http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To

Re: [HACKERS] Academic help for Postgres

2016-05-13 Thread Thomas Munro
et me > know if it needs more additions/changes. Thanks. Maybe slide 7 (NoSQL Sacrifices) should have a bullet point for "transaction isolation"? -- Thomas Munro http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make cha

Re: [HACKERS] A population of population counts

2016-05-08 Thread Thomas Munro
ion of popcount into a reusable API (without trying to get hardware support) could be worth polishing for the next CF? Annoyingly, it seems Windows doesn't have POSIX/SUSv2 ffs, though apparently it can reach that instruction with MSVC intrinsic _BitScanReverse or MingW __builtin_ffs. -- T

[HACKERS] A population of population counts

2016-05-06 Thread Thomas Munro
veral modules need to do. -- Thomas Munro http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Reviewing freeze map code

2016-05-06 Thread Thomas Munro
;> >> Did somebody verify the new contents are correct? > > I admit that I didn't. It seemed like an unlikely place for a goof, > but I guess we should verify. Looks correct. The tables match the output of the attached script. -- Thomas Munro http://www.enterpri

Re: [HACKERS] Is pg_control file crashsafe?

2016-05-04 Thread Thomas Munro
elevant either unless someone with expert knowledge of NTFS could explain how a crash could lead to truncation in the first place, and how rename would help. -- Thomas Munro http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Proposal: "Causal reads" mode for load balancing reads without stale data

2016-05-04 Thread Thomas Munro
On Tue, Mar 29, 2016 at 8:17 PM, Michael Paquier wrote: > On Tue, Mar 29, 2016 at 1:11 PM, Thomas Munro > wrote: >> (BTW, isn't the select call in libpq_select >> lacking an exceptfds set, and can't it therefore block forever when >> there is an error cond

Re: [HACKERS] About subxact and xact nesting level...

2016-05-01 Thread Thomas Munro
and much appreciated. Subtransactions are used to implement SAVEPOINT, and also BEGIN blocks with EXCEPTION clauses in plpgsql. http://www.postgresql.org/docs/9.5/static/sql-savepoint.html http://www.postgresql.org/docs/9.5/static/plpgsql-control-structures.html#PLPGSQL-ERROR-TRAPPING --

[HACKERS] Typo

2016-04-29 Thread Thomas Munro
Hi, Here is a patch to fix a typo in dsm_impl.h. -- Thomas Munro http://www.enterprisedb.com typo.patch Description: Binary data -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Shared memory and processes

2016-04-27 Thread Thomas Munro
a problem I'm working to fix by providing a general purpose allocator backed by a bunch of DSM segments -- watch this space). LWLocks (our usual lock primitive for cases where spinlocks are inappropriate) currently don't work correctly inside DSM segments (this too will be fixed). --

Re: [HACKERS] Re: [COMMITTERS] pgsql: Modify the isolation tester so that multiple sessions can wait.

2016-04-27 Thread Thomas Munro
On Thu, Apr 28, 2016 at 8:46 AM, Thomas Munro wrote: > On Thu, Apr 28, 2016 at 7:02 AM, Alvaro Herrera > wrote: >> Robert Haas wrote: >> >>> That looks like malloc() returned NULL. I noticed when writing that >>> patch that isolationtester has never had any

Re: [HACKERS] Re: [COMMITTERS] pgsql: Modify the isolation tester so that multiple sessions can wait.

2016-04-27 Thread Thomas Munro
, but at first glance the other two instances of memcpy in run_permutation should also be changed to memmove, no? -- Thomas Munro http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] kqueue

2016-04-22 Thread Thomas Munro
On Sat, Apr 23, 2016 at 4:36 AM, Andres Freund wrote: > On 2016-04-22 20:39:27 +1200, Thomas Munro wrote: >> While doing that I discovered that unpatched master doesn't actually >> build on recent NetBSD systems because our static function strtoi >> clashes with a non

Re: [HACKERS] kqueue

2016-04-22 Thread Thomas Munro
On Fri, Apr 22, 2016 at 12:21 PM, Andres Freund wrote: > On 2016-04-21 14:25:06 -0400, Robert Haas wrote: >> On Thu, Apr 21, 2016 at 2:22 PM, Andres Freund wrote: >> > On 2016-04-21 14:15:53 -0400, Robert Haas wrote: >> >> On Tue, Mar 29, 2016 at 7:53 PM, Thomas

Re: [HACKERS] Broken API specification for walrcv_receive

2016-04-14 Thread Thomas Munro
wait_fd needs to be of type pgsocket, > which is a different width from "int" on Windows. Right, I see. Thanks for fixing that. -- Thomas Munro http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription

Re: [HACKERS] Support for N synchronous standby servers - take 2

2016-04-07 Thread Thomas Munro
gested fixes attached. -- Thomas Munro http://www.enterprisedb.com doc.patch Description: Binary data -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] WIP: Detecting SSI conflicts before reporting constraint violations

2016-04-07 Thread Thomas Munro
escribe the situation in the documentation. The first three sentences highlight the general problem and apply to all versions since 9.1. Then the rest of the paragraph beginning "This can be avoided ..." describes the improved situation in 9.6 if this patch is committed. -- Thomas Munro ht

Re: [HACKERS] Proposal: "Causal reads" mode for load balancing reads without stale data

2016-04-05 Thread Thomas Munro
On Tue, Apr 5, 2016 at 4:17 AM, Robert Haas wrote: > On Wed, Mar 30, 2016 at 2:22 AM, Thomas Munro > wrote: >> On Wed, Mar 30, 2016 at 2:36 PM, Robert Haas wrote: >>> OK, I committed this, with a few tweaks. In particular, I added a >>> flag variable instead of r

Re: [HACKERS] Support for N synchronous standby servers - take 2

2016-04-01 Thread Thomas Munro
On Thu, Mar 31, 2016 at 5:11 PM, Thomas Munro wrote: > On Thu, Mar 31, 2016 at 3:55 AM, Masahiko Sawada > wrote: >> On Wed, Mar 30, 2016 at 11:43 PM, Masahiko Sawada >> wrote: >>> On Tue, Mar 29, 2016 at 5:36 PM, Kyotaro HORIGUCHI >>> wrote: >>&

Re: [HACKERS] So, can we stop supporting Windows native now?

2016-03-31 Thread Thomas Munro
GS_SUPERPAGE_SIZE_ANY but such mappings are not inherited by child processes. [1] https://lists.freebsd.org/pipermail/freebsd-hackers/2012-June/039018.html [2] https://www.youtube.com/watch?v=0wIxny-n_Mg <-- a really good talk about this stuff! -- Thomas Munro http://www.enterprisedb.com --

Re: [HACKERS] So, can we stop supporting Windows native now?

2016-03-31 Thread Thomas Munro
On Fri, Apr 1, 2016 at 11:44 AM, Alvaro Herrera wrote: > Thomas Munro wrote: >> On Fri, Apr 1, 2016 at 2:53 AM, Andrew Dunstan wrote: >> > It would also be nice to find out why we can't usefully scale shared >> > buffers >> > higher like we can on *nix.

Re: [HACKERS] So, can we stop supporting Windows native now?

2016-03-31 Thread Thomas Munro
eLockMemoryPrivilege. https://msdn.microsoft.com/en-us/library/windows/desktop/aa366543(v=vs.85).aspx Just a thought. -- Thomas Munro http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Support for N synchronous standby servers - take 2

2016-03-30 Thread Thomas Munro
andby \"%s\" is now the synchronous standby with priority %u", + application_name, MyWalSnd->sync_standby_priority))); s/ the / a / offered by a transaction commit. This level of protection is referred -to as 2-safe replication in computer science theory. +t

<    2   3   4   5   6   7   8   9   >