Re: [HACKERS] Macro customizable hashtable / bitmapscan & aggregation perf

On Sat, Oct 1, 2016 at 2:44 AM, Andres Freund wrote: > Hi, > > On 2016-07-26 17:43:33 -0700, Andres Freund wrote: > > In the attached patch I've attached simplehash.h, which can be > > customized by a bunch of macros, before being inlined. There's also a > > patch using this

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

On Sep 14, 2016 5:18 PM, "Robert Haas" wrote: > > On Wed, Sep 14, 2016 at 8:16 AM, Pavan Deolasee > wrote: > > Ah, thanks. So MaxHeapTuplesPerPage sets the upper boundary for the per page > > bitmap size. Thats about 36 bytes for 8K page. IOW if

Re: [HACKERS] Status of 64 bit atomics

On May 27, 2016 5:01 PM, "John Gorman" wrote: > > Hi All > > Someone recently told me that the postgresql atomics library was incomplete > for 64 bit operations such as pg_atomic_fetch_add_u64() and should not be used. > > Can someone definitively confirm whether it is okay

Re: [HACKERS] 9.5 release notes

is a worthy inclusion in the release notes. -- Arthur Silva

Re: [HACKERS] 64-bit XIDs again

On Thu, Jul 30, 2015 at 5:31 PM, Gavin Flower gavinflo...@archidevsys.co.nz wrote: On 31/07/15 02:24, Heikki Linnakangas wrote: On 07/30/2015 04:26 PM, Alexander Korotkov wrote: Also, I think it's possible to migrate to 64-bit XIDs without breaking pg_upgrade. Old tuples can be leaved with

Re: [HACKERS] GSoC 2015: SP-GIST for geometrical objects

On Mar 27, 2015 11:08 AM, Dima Ivanovskiy dima...@mail.ru wrote: Hello, I am Dmitrii, student of Moscow Institute of Physics and Technology Abstract: I chose project Indexing prolonged geometrical objects (i.e. boxes, circles, polygons, not points) with SP-GiST by mapping to 4d-space.

Re: [HACKERS] GSoC 2015: SP-GIST for geometrical objects

On Mar 27, 2015 6:41 PM, Dima Ivanovskiy dima...@mail.ru wrote: On Mar 27, 2015 11:08 AM, Dima Ivanovskiy dima...@mail.ru wrote: Hello, I am Dmitrii, student of Moscow Institute of Physics and Technology Abstract: I chose project Indexing prolonged geometrical objects (i.e. boxes,

Re: [HACKERS] pg_rewind in contrib

On Mar 26, 2015 4:20 AM, Vladimir Borodin r...@simply.name wrote: 26 марта 2015 г., в 7:32, Michael Paquier michael.paqu...@gmail.com написал(а): On Thu, Mar 26, 2015 at 12:23 PM, Venkata Balaji N nag1...@gmail.com wrote: Test 1 : [...] If the master is crashed or killed abruptly, it

[HACKERS] Paper from IBM: Memory-Efficient Hash Joins

I've come across this paper today, I thought I'd share it here too. http://www.vldb.org/pvldb/vol8/p353-barber.pdf -- Arthur Silva

Re: [HACKERS] logical column ordering

On Fri, Feb 27, 2015 at 4:44 PM, Tomas Vondra tomas.von...@2ndquadrant.com wrote: On 27.2.2015 20:34, Alvaro Herrera wrote: Tomas Vondra wrote: I think we could calls to the randomization functions into some of the regression tests (say 'create_tables.sql'), but that makes regression

Re: [HACKERS] logical column ordering

On Feb 27, 2015 5:02 PM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: Arthur Silva wrote: Sorry to intrude, I've been following this post and I was wondering if it would allow (in the currently planed form or in the future) a wider set of non-rewriting DDLs to Postgres. For example

Re: [HACKERS] reducing our reliance on MD5

On Tue, Feb 10, 2015 at 11:19 PM, Peter Geoghegan p...@heroku.com wrote: On Tue, Feb 10, 2015 at 5:14 PM, Arthur Silva arthur...@gmail.com wrote: I don't think the password storing best practices apply to db connection authentication. Why not? -- Peter Geoghegan I assume

Re: [HACKERS] reducing our reliance on MD5

On Tue, Feb 10, 2015 at 11:25 PM, Peter Geoghegan p...@heroku.com wrote: On Tue, Feb 10, 2015 at 5:22 PM, Arthur Silva arthur...@gmail.com wrote: I assume if the hacker can intercept the server unencrypted traffic and/or has access to its hard-drive the database is compromised anyway

Re: [HACKERS] reducing our reliance on MD5

On Tue, Feb 10, 2015 at 10:32 PM, Peter Geoghegan p...@heroku.com wrote: On Tue, Feb 10, 2015 at 4:21 PM, Robert Haas robertmh...@gmail.com wrote: Although the patch was described as relatively easy to write, it never went anywhere, because it *replaced* MD5 authentication with bcrypt,

Re: [HACKERS] VODKA?

On Jan 6, 2015 7:14 PM, Josh Berkus j...@agliodbs.com wrote: Oleg, Teodor: I take it VODKA is sliding to version 9.6? -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your

Re: [HACKERS] Commitfest problems

everywhere and I find hard to believe that PG can't benefit from any of them. -- Arthur Silva

Re: [HACKERS] [REVIEW] Re: Compression of full-page-writes

On Wed, Dec 10, 2014 at 12:10 PM, Rahila Syed rahilasye...@gmail.com wrote: What I would suggest is instrument the backend with getrusage() at startup and shutdown and have it print the difference in user time and system time. Then, run tests for a fixed number of transactions and see how

Re: [HACKERS] GiST kNN search queue (Re: KNN-GiST with recheck)

On Wed, Dec 10, 2014 at 1:50 PM, Heikki Linnakangas hlinnakan...@vmware.com wrote: On 01/28/2014 04:12 PM, Alexander Korotkov wrote: 3. A binary heap would be a better data structure to buffer the rechecked values. A Red-Black tree allows random insertions and deletions, but in this case

[HACKERS] Faster relevance ranking didn't make it into PostgreSQL 9.4

to locate the relevant discussion. Can someone point me to to it? -- Arthur Silva

Re: [HACKERS] [WIP Patch] Using 128-bit integers for sum, avg and statistics aggregates

On Sat, Oct 25, 2014 at 12:38 PM, Andreas Karlsson andr...@proxel.se wrote: Hi, There was recently talk about if we should start using 128-bit integers (where available) to speed up the aggregate functions over integers which uses numeric for their internal state. So I hacked together a

Re: [HACKERS] Fixed xloginsert_locks for 9.4

On Fri, Oct 3, 2014 at 1:40 PM, Bruce Momjian br...@momjian.us wrote: On Fri, Oct 3, 2014 at 04:11:30PM +0200, Andres Freund wrote: On 2014-10-03 10:07:39 -0400, Gregory Smith wrote: On 10/3/14, 8:26 AM, Andres Freund wrote: #define NUM_XLOGINSERT_LOCKS 1 tps = 52.711939 (including

Re: [HACKERS] Fixed xloginsert_locks for 9.4

On Fri, Oct 3, 2014 at 3:10 PM, Bruce Momjian br...@momjian.us wrote: On Fri, Oct 3, 2014 at 02:07:45PM -0400, Bruce Momjian wrote: On Fri, Oct 3, 2014 at 03:00:56PM -0300, Arthur Silva wrote: I remember Informix had a setting that had no description except try different

Re: [HACKERS] jsonb format is pessimal for toast compression

On Tue, Sep 16, 2014 at 4:20 PM, Robert Haas robertmh...@gmail.com wrote: On Tue, Sep 16, 2014 at 1:11 PM, Josh Berkus j...@agliodbs.com wrote: Well, I can only judge from the use cases I personally have, none of which involve more than 100 keys at any level for most rows. So far I've

Re: [HACKERS] jsonb format is pessimal for toast compression

I couldn't get my hands on the twitter data but I'm generating my own. The json template is http://paste2.org/wJ1dfcjw and data was generated with http://www.json-generator.com/. It has 35 top level keys, just in case someone is wondering. I generated 1 random objects and I'm inserting them

Re: [HACKERS] [REVIEW] Re: Compression of full-page-writes

Em 14/09/2014 12:21, Andres Freund and...@2ndquadrant.com escreveu: On 2014-09-13 20:27:51 -0500, k...@rice.edu wrote: What we are looking for here is uniqueness thus better error detection. Not avalanche effect, nor cryptographically secure, nor bit distribution. As far as I'm aware

Re: [HACKERS] [REVIEW] Re: Compression of full-page-writes

On Sat, Sep 13, 2014 at 10:27 PM, k...@rice.edu k...@rice.edu wrote: On Sat, Sep 13, 2014 at 09:50:55PM -0300, Arthur Silva wrote: On Sat, Sep 13, 2014 at 1:55 PM, Tom Lane t...@sss.pgh.pa.us wrote: Andres Freund and...@2ndquadrant.com writes: On 2014-09-13 08:52:33 +0300, Ants Aasma

Re: [HACKERS] [REVIEW] Re: Compression of full-page-writes

That's not entirely true. CRC-32C beats pretty much everything with the same length quality-wise and has both hardware implementations and highly optimized software versions. Em 12/09/2014 17:18, Ants Aasma a...@cybertec.at escreveu: On Fri, Sep 12, 2014 at 10:38 PM, Heikki Linnakangas

Re: CRC algorithm (was Re: [HACKERS] [REVIEW] Re: Compression of full-page-writes)

Em 12/09/2014 17:23, Andres Freund and...@2ndquadrant.com escreveu: On 2014-09-12 23:03:00 +0300, Heikki Linnakangas wrote: On 09/12/2014 10:54 PM, Abhijit Menon-Sen wrote: At 2014-09-12 22:38:01 +0300, hlinnakan...@vmware.com wrote: We probably should consider switching to a faster CRC

Re: [HACKERS] [REVIEW] Re: Compression of full-page-writes

I agree that there's no reason to fix an algorithm to it, unless maybe it's pglz. There's some initial talk about implementing pluggable compression algorithms for TOAST and I guess the same must be taken into consideration for the WAL. -- Arthur Silva On Thu, Sep 11, 2014 at 2:46 AM, Rahila

Re: [HACKERS] Memory Alignment in Postgres

On Wed, Sep 10, 2014 at 12:43 PM, Robert Haas robertmh...@gmail.com wrote: On Tue, Sep 9, 2014 at 10:08 AM, Arthur Silva arthur...@gmail.com wrote: I'm continuously studying Postgres codebase. Hopefully I'll be able to make some contributions in the future. For now I'm intrigued about

Re: [HACKERS] Memory Alignment in Postgres

On Thu, Sep 11, 2014 at 11:27 AM, Andres Freund and...@2ndquadrant.com wrote: On 2014-09-11 10:32:24 -0300, Arthur Silva wrote: Unaligned memory access received a lot attention in Intel post-Nehalen era. So it may very well pay off on Intel servers. You might find this blog post and it's

Re: [HACKERS] Memory Alignment in Postgres

On Thu, Sep 11, 2014 at 12:39 PM, Tom Lane t...@sss.pgh.pa.us wrote: Merlin Moncure mmonc...@gmail.com writes: Be advised of the difficulties you are going to face here. Assuming for a second there is no reason not to go unaligned on Intel and there are material benefits to justify the

Re: [HACKERS] jsonb format is pessimal for toast compression

On Thu, Sep 11, 2014 at 10:01 PM, Josh Berkus j...@agliodbs.com wrote: So, I finally got time to test Tom's latest patch on this. TLDR: we want to go with Tom's latest patch and release beta3. Figures: So I tested HEAD against the latest lengths patch. Per Arthur Silva, I checked

[HACKERS] Memory Alignment in Postgres

but so far I wasn't able to get any conclusive results. My guess is that I'm missing something in the code or pg_bench doesn't stress the difference enough. -- Arthur Silva

Re: [HACKERS] [REVIEW] Re: Compression of full-page-writes

the results. Thank you, On Wed, Aug 27, 2014 at 9:16 PM, Arthur Silva arthur...@gmail.com wrote: Em 26/08/2014 09:16, Fujii Masao masao.fu...@gmail.com escreveu: On Tue, Aug 19, 2014 at 6:37 PM, Rahila Syed rahilasye...@gmail.com wrote: Hello, Thank you for comments. Could

Re: [HACKERS] [REVIEW] Re: Compression of full-page-writes

Em 26/08/2014 09:16, Fujii Masao masao.fu...@gmail.com escreveu: On Tue, Aug 19, 2014 at 6:37 PM, Rahila Syed rahilasye...@gmail.com wrote: Hello, Thank you for comments. Could you tell me where the patch for single block in one run is? Please find attached patch for single block

Re: [HACKERS] jsonb format is pessimal for toast compression

On Wed, Aug 27, 2014 at 1:09 AM, Arthur Silva arthur...@gmail.com wrote: It won't be faster by any means, but it should definitely be incorporated if any format changes are made (like Tom already suggested). I think it's important we gather at least 2 more things before making any calls

Re: [HACKERS] jsonb format is pessimal for toast compression

Lane patch) EXTERNAL Test query 1 runtime: 525ms Test query 2 runtime: 355ms -- Arthur Silva On Tue, Aug 26, 2014 at 7:11 PM, Tom Lane t...@sss.pgh.pa.us wrote: I wrote: I wish it were cache-friendly too, per the upthread tangent about having to fetch keys from all over the place within

Re: [HACKERS] jsonb format is pessimal for toast compression

prefered * Tests with toast hacked to use lz4 instead, which might ease any decisions -- Arthur Silva On Wed, Aug 27, 2014 at 12:53 AM, Peter Geoghegan p...@heroku.com wrote: On Tue, Aug 26, 2014 at 8:41 PM, Arthur Silva arthur...@gmail.com wrote: The difference is small but I's definitely

Re: [HACKERS] jsonb format is pessimal for toast compression

On Thu, Aug 21, 2014 at 6:20 PM, Josh Berkus j...@agliodbs.com wrote: On 08/20/2014 03:42 PM, Arthur Silva wrote: What data are you using right now Josh? The same data as upthread. Can you test the three patches (9.4 head, 9.4 with Tom's cleanup of Heikki's patch, and 9.4 with Tom's

Re: [HACKERS] jsonb format is pessimal for toast compression

What data are you using right now Josh? There's the github archive http://www.githubarchive.org/ Here's some sample data https://gist.github.com/igrigorik/2017462 -- Arthur Silva On Wed, Aug 20, 2014 at 6:09 PM, Josh Berkus j...@agliodbs.com wrote: On 08/20/2014 08:29 AM, Tom Lane wrote

Re: [HACKERS] [PATCH] Incremental backup: add backup profile to base backup

On Mon, Aug 18, 2014 at 10:05 AM, Heikki Linnakangas hlinnakan...@vmware.com wrote: On 08/18/2014 08:05 AM, Alvaro Herrera wrote: Marco Nenciarini wrote: To calculate the md5 checksum I've used the md5 code present in pgcrypto contrib as the code in src/include/libpq/md5.h is not suitable

Re: [HACKERS] jsonb format is pessimal for toast compression

On Fri, Aug 15, 2014 at 8:19 PM, Tom Lane t...@sss.pgh.pa.us wrote: Arthur Silva arthur...@gmail.com writes: We should add some sort of versionning to the jsonb format. This can be explored in the future in many ways. If we end up making an incompatible change to the jsonb format, I would

Re: [HACKERS] jsonb format is pessimal for toast compression

I'm still getting up to speed on postgres development but I'd like to leave an opinion. We should add some sort of versionning to the jsonb format. This can be explored in the future in many ways. As for the current problem, we should explore the directory at the end option. It should improve