Re: [HACKERS] Macro customizable hashtable / bitmapscan & aggregation perf

2016-10-03 Thread Arthur Silva
On Sat, Oct 1, 2016 at 2:44 AM, Andres Freund wrote: > Hi, > > On 2016-07-26 17:43:33 -0700, Andres Freund wrote: > > In the attached patch I've attached simplehash.h, which can be > > customized by a bunch of macros, before being inlined. There's also a > > patch using this for tidbitmap.c and

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

2016-09-14 Thread Arthur Silva
On Sep 14, 2016 5:18 PM, "Robert Haas" wrote: > > On Wed, Sep 14, 2016 at 8:16 AM, Pavan Deolasee > wrote: > > Ah, thanks. So MaxHeapTuplesPerPage sets the upper boundary for the per page > > bitmap size. Thats about 36 bytes for 8K page. IOW if on an average there > > are 6 or more dead tuples p

Re: [HACKERS] Status of 64 bit atomics

2016-05-27 Thread Arthur Silva
On May 27, 2016 5:01 PM, "John Gorman" wrote: > > Hi All > > Someone recently told me that the postgresql atomics library was incomplete > for 64 bit operations such as pg_atomic_fetch_add_u64() and should not be used. > > Can someone definitively confirm whether it is okay to rely on the 64 bit a

Re: [HACKERS] 9.5 release notes

2015-08-20 Thread Arthur Silva
http://momjian.us > EnterpriseDB http://enterprisedb.com > > + Everyone has their own god. + > > > -- > Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) > To make changes to your subscription: > http://www.postgresql.org/mailpref/pgsql-hackers > Are we landing pg_tgrm 1.2 in pg 9.5? If yes (we should), up to an order of magnitude improvements is a worthy inclusion in the release notes. -- Arthur Silva

Re: [HACKERS] 64-bit XIDs again

2015-07-30 Thread Arthur Silva
On Thu, Jul 30, 2015 at 5:31 PM, Gavin Flower wrote: > On 31/07/15 02:24, Heikki Linnakangas wrote: > >> On 07/30/2015 04:26 PM, Alexander Korotkov wrote: >> >>> Also, I think it's possible to migrate to 64-bit XIDs without breaking >>> pg_upgrade. Old tuples can be leaved with 32-bit XIDs while

Re: [HACKERS] GSoC 2015: SP-GIST for geometrical objects

2015-03-27 Thread Arthur Silva
On Mar 27, 2015 6:41 PM, "Dima Ivanovskiy" wrote: > > >> On Mar 27, 2015 11:08 AM, "Dima Ivanovskiy" wrote: >> > >> > Hello, I am Dmitrii, student of Moscow Institute of Physics and Technology >> > >> > Abstract: >> > >> > I chose project "Indexing prolonged geometrical objects (i.e. boxes, circl

Re: [HACKERS] GSoC 2015: SP-GIST for geometrical objects

2015-03-27 Thread Arthur Silva
On Mar 27, 2015 11:08 AM, "Dima Ivanovskiy" wrote: > > Hello, I am Dmitrii, student of Moscow Institute of Physics and Technology > > Abstract: > > I chose project "Indexing prolonged geometrical objects (i.e. boxes, circles, polygons, not points) with SP-GiST by mapping to 4d-space". > According

Re: [HACKERS] pg_rewind in contrib

2015-03-26 Thread Arthur Silva
On Mar 26, 2015 4:20 AM, "Vladimir Borodin" wrote: > > >> 26 марта 2015 г., в 7:32, Michael Paquier написал(а): >> >> On Thu, Mar 26, 2015 at 12:23 PM, Venkata Balaji N wrote: >>> >>> Test 1 : >>> >>> [...] >>> >>> If the master is crashed or killed abruptly, it may not be possible to do a >>> r

[HACKERS] Paper from IBM: Memory-Efficient Hash Joins

2015-03-13 Thread Arthur Silva
I've come across this paper today, I thought I'd share it here too. http://www.vldb.org/pvldb/vol8/p353-barber.pdf -- Arthur Silva

Re: [HACKERS] Abbreviated keys for text cost model fix

2015-03-02 Thread Arthur Silva
On Mon, Mar 2, 2015 at 8:04 PM, Jeremy Harris wrote: > On 25/02/15 00:32, Jeremy Harris wrote: > > On 23/02/15 16:40, Tomas Vondra wrote: > >> On 22.2.2015 22:30, Peter Geoghegan wrote: > >>> You should try it with the data fully sorted like this, but with one > >>> tiny difference: The very last

Re: [HACKERS] logical column ordering

2015-02-27 Thread Arthur Silva
On Feb 27, 2015 5:02 PM, "Alvaro Herrera" wrote: > > Arthur Silva wrote: > > > Sorry to intrude, I've been following this post and I was wondering if it > > would allow (in the currently planed form or in the future) a wider set of > > non-rewriting DD

Re: [HACKERS] logical column ordering

2015-02-27 Thread Arthur Silva
On Fri, Feb 27, 2015 at 4:44 PM, Tomas Vondra wrote: > On 27.2.2015 20:34, Alvaro Herrera wrote: > > Tomas Vondra wrote: > > > >> I think we could calls to the randomization functions into some of the > >> regression tests (say 'create_tables.sql'), but that makes regression > >> tests ... well,

Re: [HACKERS] reducing our reliance on MD5

2015-02-10 Thread Arthur Silva
On Tue, Feb 10, 2015 at 11:25 PM, Peter Geoghegan wrote: > On Tue, Feb 10, 2015 at 5:22 PM, Arthur Silva wrote: > > I assume if the hacker can intercept the server unencrypted traffic > and/or > > has access to its hard-drive the database is compromised anyway. > > That

Re: [HACKERS] reducing our reliance on MD5

2015-02-10 Thread Arthur Silva
On Tue, Feb 10, 2015 at 11:19 PM, Peter Geoghegan wrote: > On Tue, Feb 10, 2015 at 5:14 PM, Arthur Silva wrote: > > I don't think the "password storing best practices" apply to db > connection > > authentication. > > Why not? > > -- > Peter Geogh

Re: [HACKERS] reducing our reliance on MD5

2015-02-10 Thread Arthur Silva
On Tue, Feb 10, 2015 at 10:32 PM, Peter Geoghegan wrote: > On Tue, Feb 10, 2015 at 4:21 PM, Robert Haas > wrote: > > Although the patch was described as relatively easy to write, it never > > went anywhere, because it *replaced* MD5 authentication with bcrypt, > > which would be a big problem fo

Re: [HACKERS] VODKA?

2015-01-07 Thread Arthur Silva
On Jan 6, 2015 7:14 PM, "Josh Berkus" wrote: > > Oleg, Teodor: > > I take it VODKA is sliding to version 9.6? > > -- > Josh Berkus > PostgreSQL Experts Inc. > http://pgexperts.com > > > -- > Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) > To make changes to your subscription:

Re: [HACKERS] Commitfest problems

2014-12-19 Thread Arthur Silva
elp Postgres right now is a more streamlined/modern (the only words I could come up with) development process. Using the mailing list for everything is alright, it works. But there is lot of tools that could be used along it: gerrit/gitlab/github/bitbucket/jira and other tools that do: pull requests, code review and bugs (or any combination of them). That'd reduce friction for new contributors and further development. These tools are being used everywhere and I find hard to believe that PG can't benefit from any of them. -- Arthur Silva

Re: [HACKERS] GiST kNN search queue (Re: KNN-GiST with recheck)

2014-12-10 Thread Arthur Silva
On Wed, Dec 10, 2014 at 1:50 PM, Heikki Linnakangas wrote: > On 01/28/2014 04:12 PM, Alexander Korotkov wrote: > >> >3. A binary heap would be a better data structure to buffer the rechecked >>> >values. A Red-Black tree allows random insertions and deletions, but in >>> >this case you need to in

Re: [HACKERS] [REVIEW] Re: Compression of full-page-writes

2014-12-10 Thread Arthur Silva
On Wed, Dec 10, 2014 at 12:10 PM, Rahila Syed wrote: > >What I would suggest is instrument the backend with getrusage() at > >startup and shutdown and have it print the difference in user time and > >system time. Then, run tests for a fixed number of transactions and > >see how the total CPU usa

[HACKERS] Faster relevance ranking didn't make it into PostgreSQL 9.4

2014-10-30 Thread Arthur Silva
le to locate the relevant discussion. Can someone point me to to it? -- Arthur Silva

Re: [HACKERS] [WIP Patch] Using 128-bit integers for sum, avg and statistics aggregates

2014-10-28 Thread Arthur Silva
On Sat, Oct 25, 2014 at 12:38 PM, Andreas Karlsson wrote: > Hi, > > There was recently talk about if we should start using 128-bit integers > (where available) to speed up the aggregate functions over integers which > uses numeric for their internal state. So I hacked together a patch for > this

Re: [HACKERS] Fixed xloginsert_locks for 9.4

2014-10-03 Thread Arthur Silva
On Fri, Oct 3, 2014 at 3:10 PM, Bruce Momjian wrote: > On Fri, Oct 3, 2014 at 02:07:45PM -0400, Bruce Momjian wrote: > > On Fri, Oct 3, 2014 at 03:00:56PM -0300, Arthur Silva wrote: > > > I remember Informix had a setting that had no description except > "try >

Re: [HACKERS] Fixed xloginsert_locks for 9.4

2014-10-03 Thread Arthur Silva
On Fri, Oct 3, 2014 at 1:40 PM, Bruce Momjian wrote: > On Fri, Oct 3, 2014 at 04:11:30PM +0200, Andres Freund wrote: > > On 2014-10-03 10:07:39 -0400, Gregory Smith wrote: > > > On 10/3/14, 8:26 AM, Andres Freund wrote: > > > >#define NUM_XLOGINSERT_LOCKS 1 > > > >tps = 52.711939 (including con

Re: [HACKERS] jsonb format is pessimal for toast compression

2014-09-29 Thread Arthur Silva
On Mon, Sep 29, 2014 at 12:19 AM, Josh Berkus wrote: > On 09/26/2014 06:20 PM, Josh Berkus wrote: > > Overall, I'm satisfied with the performance of the length-and-offset > > patch. > > Oh, also ... no bugs found. > > So, can we get Beta3 out now? > > -- > Josh Berkus > PostgreSQL Experts Inc. >

Re: [HACKERS] jsonb format is pessimal for toast compression

2014-09-16 Thread Arthur Silva
On Tue, Sep 16, 2014 at 4:20 PM, Robert Haas wrote: > On Tue, Sep 16, 2014 at 1:11 PM, Josh Berkus wrote: > >>> Well, I can only judge from the use cases I personally have, none of > >>> which involve more than 100 keys at any level for most rows. So far > >>> I've seen some people argue hypote

Re: [HACKERS] jsonb format is pessimal for toast compression

2014-09-15 Thread Arthur Silva
I couldn't get my hands on the twitter data but I'm generating my own. The json template is http://paste2.org/wJ1dfcjw and data was generated with http://www.json-generator.com/. It has 35 top level keys, just in case someone is wondering. I generated 1 random objects and I'm inserting them rep

Re: [HACKERS] [REVIEW] Re: Compression of full-page-writes

2014-09-14 Thread Arthur Silva
Em 14/09/2014 12:21, "Andres Freund" escreveu: > > On 2014-09-13 20:27:51 -0500, k...@rice.edu wrote: > > > > > > What we are looking for here is uniqueness thus better error detection. Not > > > avalanche effect, nor cryptographically secure, nor bit distribution. > > > As far as I'm aware CRC32C

Re: [HACKERS] [REVIEW] Re: Compression of full-page-writes

2014-09-13 Thread Arthur Silva
On Sat, Sep 13, 2014 at 10:27 PM, k...@rice.edu wrote: > On Sat, Sep 13, 2014 at 09:50:55PM -0300, Arthur Silva wrote: > > On Sat, Sep 13, 2014 at 1:55 PM, Tom Lane wrote: > > > > > Andres Freund writes: > > > > On 2014-09-13 08:52:33 +0300, Ants Aasma wrote

Re: CRC algorithm (was Re: [HACKERS] [REVIEW] Re: Compression of full-page-writes)

2014-09-12 Thread Arthur Silva
Em 12/09/2014 17:23, "Andres Freund" escreveu: > > On 2014-09-12 23:03:00 +0300, Heikki Linnakangas wrote: > > On 09/12/2014 10:54 PM, Abhijit Menon-Sen wrote: > > >At 2014-09-12 22:38:01 +0300, hlinnakan...@vmware.com wrote: > > >> > > >>We probably should consider switching to a faster CRC algor

Re: [HACKERS] [REVIEW] Re: Compression of full-page-writes

2014-09-12 Thread Arthur Silva
That's not entirely true. CRC-32C beats pretty much everything with the same length quality-wise and has both hardware implementations and highly optimized software versions. Em 12/09/2014 17:18, "Ants Aasma" escreveu: > On Fri, Sep 12, 2014 at 10:38 PM, Heikki Linnakangas > wrote: > > I don't m

Re: [HACKERS] jsonb format is pessimal for toast compression

2014-09-11 Thread Arthur Silva
On Thu, Sep 11, 2014 at 10:01 PM, Josh Berkus wrote: > So, I finally got time to test Tom's latest patch on this. > > TLDR: we want to go with Tom's latest patch and release beta3. > > Figures: > > So I tested HEAD against the latest lengths patch. Per Arthur

Re: [HACKERS] Memory Alignment in Postgres

2014-09-11 Thread Arthur Silva
On Thu, Sep 11, 2014 at 12:39 PM, Tom Lane wrote: > Merlin Moncure writes: > > Be advised of the difficulties you are going to face here. Assuming > > for a second there is no reason not to go unaligned on Intel and there > > are material benefits to justify the effort, that doesn't necessarily

Re: [HACKERS] Memory Alignment in Postgres

2014-09-11 Thread Arthur Silva
On Thu, Sep 11, 2014 at 11:27 AM, Andres Freund wrote: > On 2014-09-11 10:32:24 -0300, Arthur Silva wrote: > > Unaligned memory access received a lot attention in Intel post-Nehalen > era. > > So it may very well pay off on Intel servers. You might find this blog > post

Re: [HACKERS] Memory Alignment in Postgres

2014-09-11 Thread Arthur Silva
On Wed, Sep 10, 2014 at 12:43 PM, Robert Haas wrote: > On Tue, Sep 9, 2014 at 10:08 AM, Arthur Silva wrote: > > I'm continuously studying Postgres codebase. Hopefully I'll be able to > make > > some contributions in the future. > > > > For now I'

Re: [HACKERS] [REVIEW] Re: Compression of full-page-writes

2014-09-11 Thread Arthur Silva
I agree that there's no reason to fix an algorithm to it, unless maybe it's pglz. There's some initial talk about implementing pluggable compression algorithms for TOAST and I guess the same must be taken into consideration for the WAL. -- Arthur Silva On Thu, Sep 11, 2014 at

[HACKERS] Memory Alignment in Postgres

2014-09-09 Thread Arthur Silva
I'm trying to messing with the *ALIGN macros but so far I wasn't able to get any conclusive results. My guess is that I'm missing something in the code or pg_bench doesn't stress the difference enough. -- Arthur Silva

Re: [HACKERS] [REVIEW] Re: Compression of full-page-writes

2014-09-02 Thread Arthur Silva
ring tests is very low i.e below 10% . > CPU remained idle for large(~90) percentage of time. I will repeat the > above tests with high load on CPU and using the benchmark given by > Fujii-san and post the results. > > > Thank you, > > > > On Wed, Aug 27, 2014 at 9:16

Re: [HACKERS] jsonb format is pessimal for toast compression

2014-08-27 Thread Arthur Silva
On Wed, Aug 27, 2014 at 1:09 AM, Arthur Silva wrote: > It won't be faster by any means, but it should definitely be incorporated > if any format changes are made (like Tom already suggested). > > I think it's important we gather at least 2 more things before making any >

Re: [HACKERS] [REVIEW] Re: Compression of full-page-writes

2014-08-27 Thread Arthur Silva
Em 26/08/2014 09:16, "Fujii Masao" escreveu: > > On Tue, Aug 19, 2014 at 6:37 PM, Rahila Syed wrote: > > Hello, > > Thank you for comments. > > > >>Could you tell me where the patch for "single block in one run" is? > > Please find attached patch for single block compression in one run. > > Thank

Re: [HACKERS] jsonb format is pessimal for toast compression

2014-08-26 Thread Arthur Silva
is indeed prefered * Tests with toast hacked to use lz4 instead, which might ease any decisions -- Arthur Silva On Wed, Aug 27, 2014 at 12:53 AM, Peter Geoghegan wrote: > On Tue, Aug 26, 2014 at 8:41 PM, Arthur Silva wrote: > > The difference is small but I's definitely faster, w

Re: [HACKERS] jsonb format is pessimal for toast compression

2014-08-26 Thread Arthur Silva
ll Lengths (Tom Lane patch) EXTERNAL Test query 1 runtime: 525ms Test query 2 runtime: 355ms -- Arthur Silva On Tue, Aug 26, 2014 at 7:11 PM, Tom Lane wrote: > I wrote: > > I wish it were cache-friendly too, per the upthread tangent about having > > to fetch keys from all over

Re: [HACKERS] jsonb format is pessimal for toast compression

2014-08-24 Thread Arthur Silva
On Thu, Aug 21, 2014 at 6:20 PM, Josh Berkus wrote: > On 08/20/2014 03:42 PM, Arthur Silva wrote: > > What data are you using right now Josh? > > The same data as upthread. > > Can you test the three patches (9.4 head, 9.4 with Tom's cleanup of > Heikki's patch,

Re: [HACKERS] jsonb format is pessimal for toast compression

2014-08-20 Thread Arthur Silva
What data are you using right now Josh? There's the github archive http://www.githubarchive.org/ Here's some sample data https://gist.github.com/igrigorik/2017462 -- Arthur Silva On Wed, Aug 20, 2014 at 6:09 PM, Josh Berkus wrote: > On 08/20/2014 08:29 AM, Tom Lane wrote: &g

Re: [HACKERS] [PATCH] Incremental backup: add backup profile to base backup

2014-08-18 Thread Arthur Silva
On Mon, Aug 18, 2014 at 10:05 AM, Heikki Linnakangas < hlinnakan...@vmware.com> wrote: > On 08/18/2014 08:05 AM, Alvaro Herrera wrote: > >> Marco Nenciarini wrote: >> >> To calculate the md5 checksum I've used the md5 code present in pgcrypto >>> contrib as the code in src/include/libpq/md5.h is

Re: [HACKERS] jsonb format is pessimal for toast compression

2014-08-15 Thread Arthur Silva
On Fri, Aug 15, 2014 at 8:19 PM, Tom Lane wrote: > Arthur Silva writes: > > We should add some sort of versionning to the jsonb format. This can be > > explored in the future in many ways. > > If we end up making an incompatible change to the jsonb format, I would

Re: [HACKERS] jsonb format is pessimal for toast compression

2014-08-15 Thread Arthur Silva
I'm still getting up to speed on postgres development but I'd like to leave an opinion. We should add some sort of versionning to the jsonb format. This can be explored in the future in many ways. As for the current problem, we should explore the directory at the end option. It should improve com