Re: [HACKERS] Template for commit messages

2016-01-29 Thread Heikki Linnakangas
On 28 January 2016 15:57:15 CET, Robert Haas wrote: >On Thu, Jan 28, 2016 at 9:52 AM, Tom Lane wrote: >> Robert Haas writes: >>> On Thu, Jan 28, 2016 at 8:04 AM, Tomas Vondra >>> wrote: Why can't we do both? That is, have a free-form text with the >nuances, and then Reviewed-By list

Re: [HACKERS] Template for commit messages

2016-02-01 Thread Heikki Linnakangas
On 01/02/16 13:36, Andres Freund wrote: On 2016-01-28 06:03:02 -0500, Bruce Momjian wrote: Reported-by: Bug: Author: Reviewed-by: Tested-by: Backpatch-through: I personally, and I realize that I'm likely alone on that, would really like to see re

Re: [HACKERS] WAL logging problem in 9.4.3?

2016-02-04 Thread Heikki Linnakangas
On 22/10/15 03:56, Michael Paquier wrote: On Wed, Oct 21, 2015 at 11:53 PM, Alvaro Herrera wrote: Heikki Linnakangas wrote: Thanks. For comparison, I wrote a patch to implement what I had in mind. When a WAL-skipping COPY begins, we add an entry for that relation in a "pending-fsyncs&

Re: [HACKERS] [Proposal] Improvement of GiST page layout

2016-02-08 Thread Heikki Linnakangas
On 09/02/16 07:15, Andrew Borodin wrote: Hi, hackers! I want to propose improvement of GiST page layout. GiST is optimized to reduce disk operations on search queries, for example, windows search queries in case of R-tree. I expect that more complicated page layout will help to tradeoff some o

Re: [HACKERS] custom function for converting human readable sizes to bytes

2016-02-12 Thread Heikki Linnakangas
On 12/02/16 10:19, Dean Rasheed wrote: On 12 February 2016 at 06:25, Pavel Stehule wrote: Thank you for review and other work This seems like a reasonable first patch for me as a committer, so I'll take it unless anyone else was planning to do so. Great! You'll want to copy-edit the commen

Re: [HACKERS] about google summer of code 2016

2016-02-21 Thread Heikki Linnakangas
On 19/02/16 10:10, Álvaro Hernández Tortosa wrote: Oleg and I discussed recently that a really good addition to a GSoC item would be to study whether it's convenient to have a binary serialization format for jsonb over the wire. Some argue this should be benchmarked first. So the scope for

Re: [HACKERS] Missing checks when malloc returns NULL...

2016-08-18 Thread Heikki Linnakangas
On 06/22/2016 04:41 AM, Michael Paquier wrote: On Tue, Jun 21, 2016 at 10:46 PM, Tom Lane wrote: Michael Paquier writes: - mcxt.c uses that, which is surprising: @@ -704,7 +704,8 @@ MemoryContextCreate(NodeTag tag, Size size, { /* Special case for startup: use good ol' malloc */

Re: [HACKERS] Password identifiers, protocol aging and SCRAM protocol

2016-08-18 Thread Heikki Linnakangas
On 07/22/2016 03:02 AM, Tom Lane wrote: Michael Paquier writes: On Fri, Jul 22, 2016 at 8:48 AM, Tom Lane wrote: I'm confused. We need that code in both libpq and backend, no? src/common is the place for stuff of that description. Not necessarily. src/interfaces/libpq/Makefile uses a set

Re: [HACKERS] Password identifiers, protocol aging and SCRAM protocol

2016-08-18 Thread Heikki Linnakangas
On 08/18/2016 03:45 PM, Michael Paquier wrote: On Thu, Aug 18, 2016 at 9:28 PM, Heikki Linnakangas wrote: Let's take the opportunity and also move src/backend/libpq/ip.c and md5.c into src/common. It would be weird to have sha.c in src/common, but md5.c in src/backend/libpq. Looking at

[HACKERS] Re: PROPOSAL: make PostgreSQL sanitizers-friendly (and prevent information disclosure)

2016-08-18 Thread Heikki Linnakangas
On 03/22/2016 03:27 PM, Aleksander Alekseev wrote: diff --git a/src/backend/commands/tablespace.c b/src/backend/commands/tablespace.c index 1ff5728..a10c078 100644 --- a/src/backend/commands/tablespace.c +++ b/src/backend/commands/tablespace.c @@ -669,6 +669,11 @@ destroy_tablespace_directories(

[HACKERS] Race condition in GetOldestActiveTransactionId()

2016-08-22 Thread Heikki Linnakangas
While hacking on the CSN patch, I spotted a race condition between GetOldestActiveTransactionId() and GetNewTransactionId(). GetOldestActiveTransactionId() calculates the oldest XID that's still running, by doing: 1. Read nextXid, without a lock. This is the upper bound, if there are no activ

Re: [HACKERS] Proposal for CSN based snapshots

2016-08-22 Thread Heikki Linnakangas
And here's a new patch version. Still lots of work to do, especially in performance testing, and minimizing the worst-case performance hit. On 08/09/2016 03:16 PM, Heikki Linnakangas wrote: Next steps: * Hot standby feedback is broken, now that CSN != LSN again. Will have to switch this

Re: [HACKERS] Proposal for CSN based snapshots

2016-08-22 Thread Heikki Linnakangas
On 08/22/2016 07:35 PM, Heikki Linnakangas wrote: And here's a new patch version... And here's the attachment I forgot, *sigh*.. - Heikki csn-4.patch.gz Description: application/gzip -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make chang

Re: [HACKERS] Proposal for CSN based snapshots

2016-08-22 Thread Heikki Linnakangas
On 08/22/2016 07:49 PM, Robert Haas wrote: Nice to see you working on this again. On Mon, Aug 22, 2016 at 12:35 PM, Heikki Linnakangas wrote: A sequential scan of a table like that with 10 million rows took about 700 ms on my laptop, when the hint bits are set, without this patch. With this

Re: [HACKERS] Proposal for CSN based snapshots

2016-08-23 Thread Heikki Linnakangas
On 08/22/2016 08:38 PM, Andres Freund wrote: On 2016-08-22 20:32:42 +0300, Heikki Linnakangas wrote: I remember seeing ProcArrayLock contention very visible earlier, but I can't hit that now. I suspect you'd still see contention on bigger hardware, though, my laptop has oly 4 cores.

Re: [HACKERS] Proposal for CSN based snapshots

2016-08-24 Thread Heikki Linnakangas
On 08/23/2016 06:18 PM, Heikki Linnakangas wrote: On 08/22/2016 08:38 PM, Andres Freund wrote: On 2016-08-22 20:32:42 +0300, Heikki Linnakangas wrote: I remember seeing ProcArrayLock contention very visible earlier, but I can't hit that now. I suspect you'd still see contention

Re: [HACKERS] OpenSSL 1.1 breaks configure and more

2016-08-26 Thread Heikki Linnakangas
On 07/05/2016 04:46 PM, Andreas Karlsson wrote: @@ -280,8 +287,9 @@ px_find_digest(const char *name, PX_MD **res) digest = px_alloc(sizeof(*digest)); digest->algo = md; - EVP_MD_CTX_init(&digest->ctx); - if (EVP_DigestInit_ex(&digest->ctx, digest->algo, NULL) == 0) +

Re: [HACKERS] Simplifying the interface of UpdateMinRecoveryPoint

2016-08-26 Thread Heikki Linnakangas
On 07/13/2016 04:25 PM, Michael Paquier wrote: On Wed, Jul 13, 2016 at 8:27 PM, Amit Kapila wrote: On Wed, Jul 13, 2016 at 9:19 AM, Michael Paquier wrote: Hence why not simplifying its interface and remove the force flag? One point to note is that the signature and usage of UpdateMinRecover

Re: [HACKERS] [Patch] RBTree iteration interface improvement

2016-08-26 Thread Heikki Linnakangas
On 08/22/2016 01:00 PM, Aleksander Alekseev wrote: It seems clear to me that the existing arrangement is hazardous for any RBTree that hasn't got exactly one consumer. I think Aleksander's plan to decouple the iteration state is probably a good one (NB: I've not read the patch, so this is not an

Re: [HACKERS] Odd oid-system-column handling in postgres_fdw

2016-08-26 Thread Heikki Linnakangas
On 04/05/2016 11:15 AM, Etsuro Fujita wrote: On 2016/03/16 16:25, Etsuro Fujita wrote: PG9.5 allows us to add an oid system column to foreign tables, using ALTER FOREIGN TABLE SET WITH OIDS, but currently, that column reads as zeroes in postgres_fdw. That seems to me like a bug. So, I'd like t

Re: [HACKERS] OpenSSL 1.1 breaks configure and more

2016-08-26 Thread Heikki Linnakangas
On 08/26/2016 07:44 PM, Tom Lane wrote: Peter Eisentraut writes: On 8/26/16 5:31 AM, Heikki Linnakangas wrote: I think now would be a good time to drop support for OpenSSL versions older than 0.9.8. OpenSSL don't even support 0.9.8 anymore, although there are probably distributions out

Re: [HACKERS] Implement targetlist SRFs using ROWS FROM() (was Changed SRF in targetlist handling)

2016-08-29 Thread Heikki Linnakangas
On 08/28/2016 12:48 AM, Andres Freund wrote: Attached is a significantly updated patch series (see the mail one up for details about what this is, I don't want to quote it in its entirety). There's still some corner cases (DISTINCT + SRF, UNION/INTERSECT with SRF) to test / implement and a good

Re: [HACKERS] OpenSSL 1.1 breaks configure and more

2016-08-29 Thread Heikki Linnakangas
On 08/27/2016 05:15 PM, Peter Eisentraut wrote: On 8/26/16 9:26 PM, Andreas Karlsson wrote: I have attached a patch which removes the < 0.9.8 compatibility code. Should we also add a version check to configure? We do not have any such check currently. I think that is not necessary. I was goi

Re: [HACKERS] OpenSSL 1.1 breaks configure and more

2016-08-29 Thread Heikki Linnakangas
On 08/29/2016 08:22 PM, Heikki Linnakangas wrote: On 08/27/2016 05:15 PM, Peter Eisentraut wrote: On 8/26/16 9:26 PM, Andreas Karlsson wrote: I have attached a patch which removes the < 0.9.8 compatibility code. Should we also add a version check to configure? We do not have any such ch

Re: [HACKERS] OpenSSL 1.1 breaks configure and more

2016-08-29 Thread Heikki Linnakangas
On 08/30/2016 03:26 AM, Andreas Karlsson wrote: On 08/26/2016 11:31 AM, Heikki Linnakangas wrote: On 07/05/2016 04:46 PM, Andreas Karlsson wrote: @@ -280,8 +287,9 @@ px_find_digest(const char *name, PX_MD **res) digest = px_alloc(sizeof(*digest)); digest->algo =

[HACKERS] Pinning a buffer in TupleTableSlot is unnecessary

2016-08-30 Thread Heikki Linnakangas
ExecMaterializeSlot() in its current form, would that be OK? - Heikki commit 44ba53760e83921817b4878e48a2e4ad2fd67493 Author: Heikki Linnakangas Date: Tue Aug 30 13:06:59 2016 +0300 Remove support for holding a buffer pin in a TupleTableSlot. diff --git a/contrib/postgres_fdw/postgre

Re: [HACKERS] Pinning a buffer in TupleTableSlot is unnecessary

2016-08-30 Thread Heikki Linnakangas
On 08/30/2016 02:38 PM, Tom Lane wrote: Heikki Linnakangas writes: While profiling some queries and looking at executor overhead, I realized that we're not making much use of TupleTableSlot's ability to hold a buffer pin. In a SeqScan, the buffer is held pinned by the underlying

[HACKERS] Use static inline functions for Float <-> Datum conversions

2016-08-31 Thread Heikki Linnakangas
out 4% faster on my laptop with this patch. - Heikki From 8ed0f619093b16d113a0f23ee903268799eb2212 Mon Sep 17 00:00:00 2001 From: Heikki Linnakangas Date: Wed, 31 Aug 2016 13:15:06 +0300 Subject: [PATCH] Use static inline functions for float <-> Datum conversions. Now that we are OK with u

Re: [HACKERS] Use static inline functions for Float <-> Datum conversions

2016-08-31 Thread Heikki Linnakangas
On 08/31/2016 02:38 PM, Tom Lane wrote: Heikki Linnakangas writes: Now that we are OK with static inline functions, we can save some cycles from floating-point functions, by turning Float4GetDatum, Float8GetDatum, and DatumGetFloat8 into static inlines. Looks good to me. Ok, will push. I

[HACKERS] Optimizing aggregates

2016-08-31 Thread Heikki Linnakangas
e trivial cases. - Heikki From 106c5742fde2ec83576323db74a7249d7d85101f Mon Sep 17 00:00:00 2001 From: Heikki Linnakangas Date: Wed, 31 Aug 2016 17:27:52 +0300 Subject: [PATCH] Skip ExecProject for non-ordered aggregates. When support for ordered aggregates was added, we started using ExecProject

Re: [HACKERS] Optimizing aggregates

2016-08-31 Thread Heikki Linnakangas
On 08/31/2016 06:51 PM, Andres Freund wrote: On 2016-08-31 17:47:18 +0300, Heikki Linnakangas wrote: We actually used to call ExecEvalExpr() directly for each argument, but that was changed by the patch that added support for ordered set aggregates. It looks like that was a bad idea, from a

[HACKERS] Re: [COMMITTERS] pgsql: Use static inline functions for float <-> Datum conversions.

2016-08-31 Thread Heikki Linnakangas
On 08/31/2016 08:27 PM, Tom Lane wrote: Heikki Linnakangas writes: Use static inline functions for float <-> Datum conversions. Hmm, it looks like narwhal for one is not happy with this: http://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=narwhal&dt=2016-08-31%2016%3A00%3A03

Re: [HACKERS] [Patch] RBTree iteration interface improvement

2016-09-01 Thread Heikki Linnakangas
On 08/26/2016 04:07 PM, Aleksander Alekseev wrote: Another unrelated change in this patch is the addition of rb_rightmost(). It's not used for anything, so I'm not sure what the point is. Then again, there don't seem to be any callers of rb_leftmost() either. It's just something I needed in tes

Re: [HACKERS] Optimizing numeric SUM() aggregate

2016-09-02 Thread Heikki Linnakangas
On 08/03/2016 08:47 PM, Andrey Borodin wrote: 1. Currenlty overflow is carried every addition. I suggest that it is possibe to do it only every (INT32_MAX - INT32_MAX / NBASE) / (NBASE - 1) addition. Please note that with this overflow count it will be neccesary to check whether two finishin

Re: [HACKERS] Password identifiers, protocol aging and SCRAM protocol

2016-09-02 Thread Heikki Linnakangas
On 08/19/2016 09:46 AM, Michael Paquier wrote: On Fri, Aug 19, 2016 at 1:51 AM, Heikki Linnakangas wrote: On 08/18/2016 03:45 PM, Michael Paquier wrote: On Thu, Aug 18, 2016 at 9:28 PM, Heikki Linnakangas wrote: For the current ip.c, I don't have a better idea than putting in src/c

Re: [HACKERS] Re: PROPOSAL: make PostgreSQL sanitizers-friendly (and prevent information disclosure)

2016-09-02 Thread Heikki Linnakangas
On 08/20/2016 10:46 PM, Tom Lane wrote: Noah Misch writes: On Fri, Aug 19, 2016 at 07:22:02PM -0400, Tom Lane wrote: So maybe we ought to work towards taking those out? Maybe. It's a policy question boiling down to our willingness to spend cycles zeroing memory in order to limit trust in t

Re: [HACKERS] [sqlsmith] Failed assertion in numeric aggregate

2016-09-04 Thread Heikki Linnakangas
On 09/03/2016 08:59 PM, Tom Lane wrote: Andreas Seltenreich writes: Digging in the coredumps, it looks like set_var_from_num() is invoked on an uninitialized NumericVar. Sample gdb session below. Hm, yeah, looks like numeric_poly_deserialize is missing a required init_var() step. Slightly a

Re: [HACKERS] Better locale-specific-character-class handling for regexps

2016-09-04 Thread Heikki Linnakangas
On 08/23/2016 03:54 AM, Tom Lane wrote: ! That's still not quite enough, though, because of locale-dependent ! character classes such as [[:alpha:]]. In Unicode locales these classes ! may have thousands of entries that are above MAX_SIMPLE_CHR, and we ! certainly don't want to be searching larg

Re: [HACKERS] Better locale-specific-character-class handling for regexps

2016-09-05 Thread Heikki Linnakangas
On 09/04/2016 08:44 PM, Tom Lane wrote: Heikki Linnakangas writes: On 08/23/2016 03:54 AM, Tom Lane wrote: +1 for this patch in general. Some regression test cases would be nice. I'm not sure how to write such tests without introducing insurmountable platform dependencies --- particular

Re: [HACKERS] OpenSSL 1.1 breaks configure and more

2016-09-05 Thread Heikki Linnakangas
On 09/05/2016 03:12 AM, Andreas Karlsson wrote: On 08/30/2016 08:42 AM, Heikki Linnakangas wrote: There's the ResourceOwner mechanism, see src/backend/utils/resowner/. That would be the proper way to do this. Call RegisterResourceReleaseCallback() when the context is allocated, and hav

Re: [HACKERS] Supporting SJIS as a database encoding

2016-09-05 Thread Heikki Linnakangas
On 09/05/2016 05:47 PM, Tom Lane wrote: "Tsunakawa, Takayuki" writes: Before digging into the problem, could you share your impression on whether PostgreSQL can support SJIS? Would it be hopeless? I think it's pretty much hopeless. Agreed. But one thing that would help a little, would be

Re: [HACKERS] Better locale-specific-character-class handling for regexps

2016-09-05 Thread Heikki Linnakangas
On 09/05/2016 07:10 PM, Tom Lane wrote: Heikki Linnakangas writes: On 09/04/2016 08:44 PM, Tom Lane wrote: I guess I could follow the lead of collate.linux.utf8.sql and produce a test that's only promised to pass on one platform with one encoding, but I'm not terribly excited by th

Re: [HACKERS] Parallel tuplesort (for parallel B-Tree index creation)

2016-09-06 Thread Heikki Linnakangas
On 08/16/2016 03:33 AM, Peter Geoghegan wrote: I attach a patch that changes how we maintain the heap invariant during tuplesort merging. I already mentioned this over on the "Parallel tuplesort, partitioning, merging, and the future" thread. As noted already on that thread, this patch makes merg

Re: [HACKERS] Parallel tuplesort (for parallel B-Tree index creation)

2016-09-06 Thread Heikki Linnakangas
I'm reviewing patches 1-3 in this series, i.e. those patches that are not directly related to parallelism, but are independent improvements to merging. Let's begin with patch 1: On 08/02/2016 01:18 AM, Peter Geoghegan wrote: Cap the number of tapes used by external sorts Commit df700e6b set

[HACKERS] Tuplesort merge pre-reading

2016-09-06 Thread Heikki Linnakangas
perhaps we should teach LogicalTape to use larger buffers, if we can't rely on the OS to do the buffering for us. But overall, this seems to make the code both simpler and faster. Am I missing something? - Heikki From ea4ce25a33d0dec370a1b5e45cbc6f794e377a90 Mon Sep 17 00:00:00 2001 Fr

Re: [HACKERS] GiST penalty functions [PoC]

2016-09-06 Thread Heikki Linnakangas
On 08/29/2016 09:04 AM, Andrew Borodin wrote: In this message I address only first problem. I want to construct a penalty function that will choose: 1.Entry with a zero volume and smallest margin, that can accommodate insertion tuple without extension, if one exists; 2.Entry with smallest

Re: [HACKERS] [GENERAL] C++ port of Postgres

2016-09-06 Thread Heikki Linnakangas
On 08/31/2016 04:41 PM, Peter Eisentraut wrote: I developed a minimally invasive patch for C++ support a few years ago shortly after I wrote that blog post. Since there appears to have been some interest here now, I have updated that and split it up into logical chunks. So here you go. Lookin

Re: [HACKERS] Parallel tuplesort (for parallel B-Tree index creation)

2016-09-06 Thread Heikki Linnakangas
On 09/06/2016 10:42 PM, Peter Geoghegan wrote: On Tue, Sep 6, 2016 at 12:39 PM, Peter Geoghegan wrote: On Tue, Sep 6, 2016 at 12:08 AM, Heikki Linnakangas wrote: I attach a patch that changes how we maintain the heap invariant during tuplesort merging. Nice! Thanks! BTW, the way that

Re: [HACKERS] Parallel tuplesort (for parallel B-Tree index creation)

2016-09-06 Thread Heikki Linnakangas
On 09/07/2016 12:46 AM, Peter Geoghegan wrote: On Tue, Sep 6, 2016 at 12:34 AM, Heikki Linnakangas wrote: Why do we reserve the buffer space for all the tapes right at the beginning? Instead of the single USEMEM(maxTapes * TAPE_BUFFER_OVERHEAD) callin inittapes(), couldn't we call U

Re: [HACKERS] Parallel tuplesort (for parallel B-Tree index creation)

2016-09-06 Thread Heikki Linnakangas
On 09/07/2016 09:01 AM, Peter Geoghegan wrote: On Tue, Sep 6, 2016 at 10:57 PM, Peter Geoghegan wrote: There isn't much point in that, because those buffers are never physically allocated in the first place when there are thousands. They are, however, entered into the tuplesort.c accounting as

Re: [HACKERS] GiST penalty functions [PoC]

2016-09-06 Thread Heikki Linnakangas
On 09/07/2016 09:42 AM, Andrew Borodin wrote: 2. Your algorithm, among loosing some bits of precision (which is absolutely acceptable - we have like 31 of them and that’s a lot) rely on false assumption. We compare tuples on page not by MBR of inserted tuple, but by MBR of tuple on page, which is

Re: [HACKERS] GiST penalty functions [PoC]

2016-09-07 Thread Heikki Linnakangas
On 09/07/2016 09:20 PM, Andrew Borodin wrote: Well, arithmetics is too fragile. This version of float packing with arithmetical packaging static float pack_float(float actualValue, int realm) { double max,min; max = FLT_MAX / ( 8 >> realm ); min = FLT_MAX / ( 16 >> realm ); if( realm == 0 ) min

Re: [HACKERS] Parallel tuplesort (for parallel B-Tree index creation)

2016-09-07 Thread Heikki Linnakangas
On 09/07/2016 09:17 AM, Peter Geoghegan wrote: On Tue, Sep 6, 2016 at 11:09 PM, Heikki Linnakangas wrote: The big picture here is that you can't only USEMEM() for tapes as the need arises for new tapes as new runs are created. You'll just run a massive availMem deficit, that you have

Re: [HACKERS] Is tuplesort_heap_siftup() a misnomer?

2016-09-08 Thread Heikki Linnakangas
On 09/08/2016 03:36 AM, Peter Geoghegan wrote: On Wed, Sep 7, 2016 at 2:42 PM, Tom Lane wrote: The reason it's called siftup is that that's what Knuth calls it. See Algorithm 5.2.3H (Heapsort), pp 146-147 in the first edition of Volume 3; tuplesort_heap_siftup corresponds directly to steps H3-H

Re: [HACKERS] GiST penalty functions [PoC]

2016-09-08 Thread Heikki Linnakangas
On 09/08/2016 09:39 AM, Михаил Бахтерев wrote: Excuse me for intervention. It depends. For instance, i run PostgreSQL on the modern MIPS CPU, which does not have sqrt support. But you are right, it is supported in most cases. And if execution speed of this very fuction is of concern, sqrtf(x) s

Re: [HACKERS] Tuplesort merge pre-reading

2016-09-08 Thread Heikki Linnakangas
er processes that could make use of it. FWIW, I did try an increase in the buffer size in LogicalTape at one time several months back, and so no benefit there (at least, with no other changes). Yeah, unless you get rid of the pre-reading in tuplesort.c, you're just double-buffering. -

Re: [HACKERS] Tuplesort merge pre-reading

2016-09-09 Thread Heikki Linnakangas
On 09/08/2016 09:59 PM, Heikki Linnakangas wrote: On 09/06/2016 10:26 PM, Peter Geoghegan wrote: On Tue, Sep 6, 2016 at 12:08 PM, Peter Geoghegan wrote: Offhand, I would think that taken together this is very important. I'd certainly want to see cases in the hundreds of megabytes or giga

Re: [HACKERS] Tuplesort merge pre-reading

2016-09-09 Thread Heikki Linnakangas
On 09/09/2016 03:25 PM, Greg Stark wrote: On Fri, Sep 9, 2016 at 1:01 PM, Heikki Linnakangas wrote: I'm happy with what it looks like. We are in fact getting a more sequential access pattern with these patches, because we're not expanding the pre-read tuples into SortTuples. Keepi

[HACKERS] More ideas for speeding up sorting

2016-09-09 Thread Heikki Linnakangas
A few things caught my eye while hacking on the latest round of sorting patches. Starting a new thread because these are orthogonal to the things discussed on the other threads: 1. SortTuple.tupindex is not used when the sort fits in memory. If we also got rid of the isnull1 flag, we could shr

Re: [HACKERS] Tuplesort merge pre-reading

2016-09-10 Thread Heikki Linnakangas
On 09/10/2016 04:21 AM, Claudio Freire wrote: On Fri, Sep 9, 2016 at 9:51 PM, Claudio Freire wrote: On Fri, Sep 9, 2016 at 8:13 AM, Heikki Linnakangas wrote: Claudio, if you could also repeat the tests you ran on Peter's patch set on the other thread, with these patches, that'

Re: [HACKERS] Parallel tuplesort (for parallel B-Tree index creation)

2016-09-11 Thread Heikki Linnakangas
On 09/10/2016 03:22 AM, Claudio Freire wrote: Overall, however, I believe the patch is in good shape. Only minor form issues need to be changed, the functionality seems both desirable and ready. Pushed this "displace root" patch, with some changes: * I renamed "tuplesort_heap_siftup()" to "tup

Re: [HACKERS] [GENERAL] C++ port of Postgres

2016-09-11 Thread Heikki Linnakangas
On 09/11/2016 01:20 AM, Christian Convey wrote: Hi Heikki, Could I ask you a newbie-reviewer question about something I'm seeing here? https://commitfest.postgresql.org/10/776/ From some reading I've done (e.g., Stephen Frost's PGCon 2011 slides), I got the impression that a successful patch w

Re: [HACKERS] [GENERAL] C++ port of Postgres

2016-09-11 Thread Heikki Linnakangas
On 09/11/2016 01:20 AM, Christian Convey wrote: Hi Heikki, Could I ask you a newbie-reviewer question about something I'm seeing here? https://commitfest.postgresql.org/10/776/ From some reading I've done (e.g., Stephen Frost's PGCon 2011 slides), I got the impression that a successful patch w

Re: [HACKERS] Tuplesort merge pre-reading

2016-09-11 Thread Heikki Linnakangas
Here's a new version of these patches, rebased over current master. I squashed the two patches into one, there's not much point to keep them separate. - Heikki >From 6e3813d876cf3efbe5f1b80c45f44ed5494304ab Mon Sep 17 00:00:00 2001 From: Heikki Linnakangas Date: Sun, 11 Sep

Re: [HACKERS] OpenSSL 1.1 breaks configure and more

2016-09-12 Thread Heikki Linnakangas
On 09/05/2016 02:52 PM, Heikki Linnakangas wrote: On 09/05/2016 03:23 AM, Tom Lane wrote: Judging by the number of people who have popped up recently with their own OpenSSL 1.1 patches, I think there is going to be a lot of demand for back-patching some sort of 1.1 support into our back

Re: [HACKERS] Tuplesort merge pre-reading

2016-09-12 Thread Heikki Linnakangas
On 09/12/2016 06:47 PM, Claudio Freire wrote: On Mon, Sep 12, 2016 at 12:02 PM, Claudio Freire wrote: On Sun, Sep 11, 2016 at 12:47 PM, Heikki Linnakangas wrote: Here's a new version of these patches, rebased over current master. I squashed the two patches into one, there's not muc

Re: [HACKERS] Supporting SJIS as a database encoding

2016-09-13 Thread Heikki Linnakangas
On 09/08/2016 09:35 AM, Kyotaro HORIGUCHI wrote: Returning in UTF-8 bloats the result string by about 1.5 times so it doesn't seem to make sense comparing with it. But it takes real = 47.35s. Nice! I was hoping that this would also make the binaries smaller. A few dozen kB of storage is perha

Re: [HACKERS] kqueue

2016-09-13 Thread Heikki Linnakangas
So, if I've understood correctly, the purpose of this patch is to improve performance on a multi-CPU system, which has the kqueue() function. Most notably, FreeBSD? I launched a FreeBSD 10.3 instance on Amazon EC2 (ami-e0682b80), on a m4.10xlarge instance. That's a 40 core system, biggest avai

Re: [HACKERS] kqueue

2016-09-13 Thread Heikki Linnakangas
On 09/13/2016 04:33 PM, Tom Lane wrote: Heikki Linnakangas writes: So, if I've understood correctly, the purpose of this patch is to improve performance on a multi-CPU system, which has the kqueue() function. Most notably, FreeBSD? OS X also has this, so it might be worth trying on a

Re: [HACKERS] Proposal: speeding up GIN build with parallel workers

2016-09-13 Thread Heikki Linnakangas
On 01/17/2016 10:03 PM, Jeff Janes wrote: On Fri, Jan 15, 2016 at 3:29 PM, Peter Geoghegan wrote: On Fri, Jan 15, 2016 at 2:38 PM, Constantin S. Pan wrote: I have a draft implementation which divides the whole process between N parallel workers, see the patch attached. Instead of a full scan

Re: [HACKERS] Tuplesort merge pre-reading

2016-09-14 Thread Heikki Linnakangas
-final merge). Is that right? I hope that you don't give the typically unused destTape tape a full share of batch memory all the time (the same applies to any other inactive-at-final-merge tapes). Ah, no, the "+ 1" comes from the need to hold the tuple that we last returned to the caller in tuplesort_get

Re: [HACKERS] OpenSSL 1.1 breaks configure and more

2016-09-15 Thread Heikki Linnakangas
On 09/15/2016 03:16 AM, Andreas Karlsson wrote: Now for a review: It looks generally good but I think I saw one error. In fe-secure-openssl.c your code still calls SSL_library_init() in OpenSSL 1.1. I think it should be enough to just call OPENSSL_init_ssl(OPENSSL_INIT_LOAD_CONFIG, NULL) like yo

Re: [HACKERS] [COMMITTERS] pgsql: Support OpenSSL 1.1.0.

2016-09-15 Thread Heikki Linnakangas
On 09/15/2016 07:41 PM, Tom Lane wrote: Heikki Linnakangas writes: Support OpenSSL 1.1.0. Buildfarm member curculio doesn't like this patch. I suspect the reason is it's got some slightly-too-old version of OpenSSL, but if so, we ought to try to fix configure's probe so t

Re: [HACKERS] [COMMITTERS] pgsql: Support OpenSSL 1.1.0.

2016-09-15 Thread Heikki Linnakangas
On 09/15/2016 07:51 PM, Heikki Linnakangas wrote: Wild guess: curculio is building with LibreSSL, which claims to be OpenSSL >= 1.1.0, but it doesn't actually implement all the functions that OpenSSL 1.1.0 does. Looks like we need some more autoconf scripting to detect LibreSSL. Or s

Re: [HACKERS] OpenSSL 1.1 breaks configure and more

2016-09-15 Thread Heikki Linnakangas
On 09/15/2016 05:33 PM, Christoph Berg wrote: Re: Michael Paquier 2016-09-15 On Thu, Sep 15, 2016 at 8:57 PM, Heikki Linnakangas wrote: I backpatched this to 9.5, but not further than that. The functions this modified were moved around in 9.5, so the patch wouldn't apply as is. It wou

Re: [HACKERS] Tuplesort merge pre-reading

2016-09-15 Thread Heikki Linnakangas
On 09/15/2016 10:12 PM, Peter Geoghegan wrote: On Wed, Sep 14, 2016 at 10:43 AM, Heikki Linnakangas wrote: Spotted another issue with this code just now. Shouldn't it be based on state->tapeRange? You don't want the destTape to get memory, since you don't use batch memory

Re: [HACKERS] Tuplesort merge pre-reading

2016-09-17 Thread Heikki Linnakangas
On 09/17/2016 07:27 PM, Peter Geoghegan wrote: On Wed, Sep 14, 2016 at 10:43 AM, Heikki Linnakangas wrote: Addressed all your comments one way or another, new patch attached. So, it's clear that this isn't ready today. As I mentioned, I'm going away for a week now. I ask that

Re: [HACKERS] ecpg -? option doesn't work in windows

2016-09-18 Thread Heikki Linnakangas
On 08/29/2016 09:10 AM, Haribabu Kommi wrote: ecpg option --help alternative -? doesn't work in windows. In windows, the PG provided getopt_long function is used for reading the provided options. The getopt_long function returns '?' for invalid characters also but it sets optopt option to 0 in c

Re: [HACKERS] pgbench - fix stats when using \sleep

2016-09-19 Thread Heikki Linnakangas
On 08/23/2016 05:47 PM, Fabien COELHO wrote: When \sleep is used within a pgbench script it resets txn_scheduled which is used for computing stats about the transaction, resulting in absurd statistics: latency average = 0.649 ms *** ??? *** ... script statistics: - statement latencies i

Re: [HACKERS] Refactoring speculative insertion with unique indexes a little

2016-09-20 Thread Heikki Linnakangas
On 03/17/2016 01:43 AM, Peter Geoghegan wrote: I attach a revision, that makes all the changes that Heikki suggested, except one. As already noted several times, following this suggestion would have added a bug. Alvaro preferred my original approach here in any case. I refer to my original approa

Re: [HACKERS] GiST penalty functions [PoC]

2016-09-21 Thread Heikki Linnakangas
On 09/14/2016 07:57 PM, Andrew Borodin wrote: Here is v6 of cube penalty patch. There was a warning about unused edge function under systems without __STDC_IEC_559__ defined, this patch fixes it. Thanks! Reading through this thread again, I think we got a bit side-tracked with this float bit

Re: [HACKERS] pgbench - compute & show latency consistently

2016-09-21 Thread Heikki Linnakangas
On 07/13/2016 11:39 AM, Fabien COELHO wrote: number of transactions per client: 1000 -latency average = 15.844 ms +latency average: 15.844 ms tps = 618.764555 (including connections establishing) I think what you have here is that colons separate input parameters and equal signs separate resu

Re: [HACKERS] pgbench - compute & show latency consistently

2016-09-21 Thread Heikki Linnakangas
(I just pushed the patch, didn't see your post until after that) On 09/21/2016 01:07 PM, Kuntal Ghosh wrote: On Thus, July 7,2016 at 08:39 PM, Fabien COELHO wrote: Also there is still the bug under -t which displays a 0 latency. Your patch clearly fixed the issue. The attached patch still f

Re: [HACKERS] Parallel tuplesort (for parallel B-Tree index creation)

2016-09-21 Thread Heikki Linnakangas
ould be mixed together, though, which would hurt the sequential pattern of the tapes, so each workers would need to allocate larger chunks to avoid that. - Heikki >From a1aa45c22cd13a2059880154e30f48d884a849ef Mon Sep 17 00:00:00 2001 From: Heikki Linnakangas Date: Wed, 21 Sep 2016 19:3

Re: [HACKERS] Tuplesort merge pre-reading

2016-09-22 Thread Heikki Linnakangas
On 09/22/2016 03:40 AM, Claudio Freire wrote: On Tue, Sep 20, 2016 at 3:34 PM, Claudio Freire wrote: The results seem all over the map. Some regressions seem significant (both in the amount of performance lost and their significance, since all 4 runs show a similar regression). The worst being

Re: [HACKERS] Parallel tuplesort (for parallel B-Tree index creation)

2016-09-22 Thread Heikki Linnakangas
On 08/02/2016 01:18 AM, Peter Geoghegan wrote: No merging in parallel -- Currently, merging worker *output* runs may only occur in the leader process. In other words, we always keep n worker processes busy with scanning-and-sorting (and maybe some merging), but then all proce

Re: [HACKERS] Typo in libpq-int.h

2016-09-22 Thread Heikki Linnakangas
On 09/22/2016 04:35 PM, Daniel Gustafsson wrote: Ran into a typo in libpq-int.h while reading/hacking: - char *gsslib; /* What GSS librart to use ("gssapi" or + char *gsslib; /* What GSS library to use ("gssapi” or Patch attached

Re: [HACKERS] PL/Python adding support for multi-dimensional arrays

2016-09-23 Thread Heikki Linnakangas
On 09/22/2016 10:28 AM, Pavel Stehule wrote: Now, the tests are enough - so I'll mark this patch as ready for commiters. I had to fix tests - there was lot of white spaces, and the result for python3 was missing Thanks Pavel! This crashes with arrays with non-default lower bounds: postgres=#

Re: [HACKERS] MSVC pl-perl error message is not verbose enough

2016-09-23 Thread Heikki Linnakangas
On 08/02/2016 08:18 PM, John Harvey wrote: On Mon, Aug 1, 2016 at 9:39 PM, Michael Paquier wrote: On Tue, Aug 2, 2016 at 2:08 AM, Robert Haas wrote: Did you add this to the next CommitFest? I have added it here: https://commitfest.postgresql.org/10/691/ John, it would be good if you could

Re: [HACKERS] Refactor StartupXLOG?

2016-09-24 Thread Heikki Linnakangas
On 09/24/2016 05:01 AM, Thomas Munro wrote: What would the appetite be for that kind of refactoring work, considering the increased burden on committers who have to backpatch bug fixes? Is it a project goal to reduce the size of large complicated functions like StartupXLOG and heap_update? It s

Re: [HACKERS] pgbench - minor fix for meta command only scripts

2016-09-26 Thread Heikki Linnakangas
On 09/24/2016 12:45 PM, Fabien COELHO wrote: Although I cannot be absolutely sure that the refactoring does not introduce any new bug, I'm convinced that it will be much easier to find them:-) :-) Attached are some small changes to your version: I have added the sleep_until fix. I have fixe

Re: [HACKERS] Password identifiers, protocol aging and SCRAM protocol

2016-09-26 Thread Heikki Linnakangas
On 09/26/2016 09:02 AM, Michael Paquier wrote: On Mon, Sep 26, 2016 at 2:15 AM, David Steele wrote: However, it doesn't look like they can be used in conjunction since the pg_hba.conf entry must specify either m5 or scram (though the database can easily contain a mixture). This would probably

Re: [HACKERS] Refactoring speculative insertion with unique indexes a little

2016-09-27 Thread Heikki Linnakangas
On 09/24/2016 02:34 PM, Peter Geoghegan wrote: On Tue, Sep 20, 2016 at 8:55 AM, Heikki Linnakangas wrote: Likewise, in ExecInsertIndexTuples(), this makes the deferred-index case work slightly differently from speculative insertion. It's not a big difference, but it again forces you to

Re: [HACKERS] Refactoring speculative insertion with unique indexes a little

2016-09-27 Thread Heikki Linnakangas
On 09/27/2016 11:22 AM, Peter Geoghegan wrote: On Tue, Sep 27, 2016 at 9:10 AM, Heikki Linnakangas wrote: Hmm, yeah, that'd be nice to fix. I'd like to see a patch specifically to fix that. I'm not convinced that we need all the complications of this patch, to get that fixed.

Re: [HACKERS] Refactoring speculative insertion with unique indexes a little

2016-09-27 Thread Heikki Linnakangas
On 09/27/2016 11:38 AM, Peter Geoghegan wrote: On Tue, Sep 27, 2016 at 9:25 AM, Heikki Linnakangas wrote: I didn't think very hard about it, but yeah, think so.. Do you think it's worth a backpatch? Or, too early to tell? Too early to tell.. - Heikki -- Sent via pgsql-hacke

Re: [HACKERS] Password identifiers, protocol aging and SCRAM protocol

2016-09-27 Thread Heikki Linnakangas
On 09/27/2016 04:19 PM, Michael Paquier wrote: On Tue, Sep 27, 2016 at 9:01 PM, Heikki Linnakangas wrote: * A source of random values. This currently uses PostmasterRandom() similarly to how the MD5 salt is generated, in the server, but plain old random() in the client. If built with OpenSSL

Re: [HACKERS] PL/Python adding support for multi-dimensional arrays

2016-09-27 Thread Heikki Linnakangas
18a3ecbb71d1ef7561a5bfd Mon Sep 17 00:00:00 2001 From: Heikki Linnakangas Date: Tue, 27 Sep 2016 21:53:17 +0300 Subject: [PATCH 1/1] WIP: Multi-dimensional arrays in PL/python --- src/pl/plpython/expected/plpython_types.out | 151 +- src/pl/plpython/expected/plpython_types_3.ou

Re: [HACKERS] should xlog_outdesc modify its argument?

2016-09-27 Thread Heikki Linnakangas
On 09/28/2016 02:35 AM, Mark Dilger wrote: The function static void xlog_outdesc(StringInfo buf, XLogReaderState *record); in src/backend/access/transam/xlog.c is called by XLogInsertRecord, and after returning a string describing an XLogRecord, it clears the state data in its XLogReaderState

Re: [HACKERS] Password identifiers, protocol aging and SCRAM protocol

2016-09-28 Thread Heikki Linnakangas
On 09/26/2016 09:02 AM, Michael Paquier wrote: On Mon, Sep 26, 2016 at 2:15 AM, David Steele wrote: * [PATCH 3/8] Switch password_encryption to a enum Does not apply on HEAD (98c2d3332): Interesting, it works for me on da6c4f6. For here on I used 39b691f251 for review and testing. I seems

Re: [HACKERS] Password identifiers, protocol aging and SCRAM protocol

2016-09-28 Thread Heikki Linnakangas
On 09/26/2016 09:02 AM, Michael Paquier wrote: * [PATCH 2/8] Move encoding routines to src/common/ > > I wonder if it is confusing to have two of encode.h/encode.c. Perhaps > they should be renamed to make them distinct? Yes it may be a good idea to rename that, like encode_utils.[c|h] for the

  1   2   3   4   5   6   7   8   9   10   >