Re: [PERFORM] More cores or higer frequency ?

2017-05-23 Thread Sven R. Kunze
On 23.05.2017 22:14, Jarek wrote: I have pool of clients (~30) inserting to database about 50 records per second (in total from all clients) and small numer (<10) clients querying database for those records once per 10s. Other queries are rare and irregular. The biggest table has ~ 100mln

Re: [PERFORM] Speeding up JSON + TSQUERY + GIN

2017-03-06 Thread Sven R. Kunze
On 06.03.2017 05:25, Jeff Janes wrote: Bitmaps can overflow and drop the row-level information, tracking only the blocks which need to be inspected. So it has to have a recheck in case that happens (although in your case it is not actually overflowing--but it still needs to be prepared for

Re: [PERFORM] Speeding up JSON + TSQUERY + GIN

2017-03-03 Thread Sven R. Kunze
On 01.03.2017 18:04, Jeff Janes wrote: On Wed, Mar 1, 2017 at 6:02 AM, Sven R. Kunze <srku...@mail.de <mailto:srku...@mail.de>> wrote: On 28.02.2017 17:49, Jeff Janes wrote: Oh. In my hands, it works very well. I get 70 seconds to do the {age: 20} query from pure

Re: [PERFORM] Speeding up JSON + TSQUERY + GIN

2017-03-01 Thread Sven R. Kunze
On 28.02.2017 17:49, Jeff Janes wrote: Oh. In my hands, it works very well. I get 70 seconds to do the {age: 20} query from pure cold caches, versus 1.4 seconds from cold caches which was followed by pg_prewarm('docs','prefetch'). How much RAM do you have? Maybe you don't have enough to

Re: [PERFORM] Speeding up JSON + TSQUERY + GIN

2017-02-28 Thread Sven R. Kunze
On 27.02.2017 19:22, Jeff Janes wrote: If by 'permanently', you mean even when you intentionally break things, then no. You will always be able to intentionally break things. There is on-going discussion of an auto-prewarm feature. But that doesn't yet exist; and once it does, a super user

Re: [PERFORM] Speeding up JSON + TSQUERY + GIN

2017-02-27 Thread Sven R. Kunze
Thanks Oleg for your reply. On 26.02.2017 21:13, Oleg Bartunov wrote: On Sun, Feb 26, 2017 at 4:28 PM, Sven R. Kunze <srku...@mail.de <mailto:srku...@mail.de>>wrote: create index docs_meta_idx ON docs using gin (meta jsonb_path_ops); create index docs_name_idx ON do

[PERFORM] Speeding up JSON + TSQUERY + GIN

2017-02-26 Thread Sven R. Kunze
Hello everyone, I am currently evaluating the possibility of using PostgreSQL for storing and querying jsonb+tsvector queries. Let's consider this setup: create table docs (id serial primary key, meta jsonb); # generate 10M entries, cf. appendix create index docs_meta_idx ON docs using gin

Re: [PERFORM] Multiple-Table-Spanning Joins with ORs in WHERE Clause

2016-09-30 Thread Sven R. Kunze
Now I found time to investigate all proposed queries side by side. Here are the results (warmup + multiple executions). TL;DR - Jeff's proposed answer performs significantly faster with our data than any other solution (both planning and execution time). I have no real idea how PostgreSQL

Re: [PERFORM] Multiple-Table-Spanning Joins with ORs in WHERE Clause

2016-09-30 Thread Sven R. Kunze
On 29.09.2016 22:26, Jeff Janes wrote: Well, I don't recall seeing this issue on this list before (or a few other forums I read) while I see several other issues over and over again. So that is why I think it is a niche issue. Perhaps I've have seen it before and just forgotten, or have not

Re: [PERFORM] Multiple-Table-Spanning Joins with ORs in WHERE Clause

2016-09-29 Thread Sven R. Kunze
On 29.09.2016 20:12, Pavel Stehule wrote: In ideal world then plan should be independent on used form. The most difficult is safe estimation of OR predicates. With correct estimation the transformation to UNION form should not be necessary I am think. Ah, okay. That's interesting. So how can

Re: [PERFORM] Multiple-Table-Spanning Joins with ORs in WHERE Clause

2016-09-29 Thread Sven R. Kunze
Hi Jeff, On 29.09.2016 20:03, Jeff Janes wrote: I don't know what the subquery plan is, I don't see references to that in the email chain. Lutz posted the following solution: SELECT * FROM big_table WHERE id in (SELECT big_table_id FROM table_a WHERE "table_a"."item_id" IN ()) OR

Re: [PERFORM] Multiple-Table-Spanning Joins with ORs in WHERE Clause

2016-09-29 Thread Sven R. Kunze
On 23.09.2016 11:00, Pavel Stehule wrote: 2016-09-23 8:35 GMT+02:00 Sven R. Kunze <srku...@mail.de <mailto:srku...@mail.de>>: I was wondering: would it be possible for PostgreSQL to rewrite the query to generate the UNION (or subquery plan if it's also fast)

Re: [PERFORM] Multiple-Table-Spanning Joins with ORs in WHERE Clause

2016-09-23 Thread Sven R. Kunze
ble_a"."item_id" IN ()) OR id in (SELECT big_table_id FROM table_a WHERE "table_b"."item_id" IN ()) that way you don't need the "distinct" and therefore there should be less comparison going on. Lutz On 22/09/16 14:24, Sven

[PERFORM] Multiple-Table-Spanning Joins with ORs in WHERE Clause

2016-09-22 Thread Sven R. Kunze
Hi pgsql-performance list, what is the recommended way of doing **multiple-table-spanning joins with ORs in the WHERE-clause**? Until now, we've used the LEFT OUTER JOIN to filter big_table like so: SELECT DISTINCT FROM "big_table" LEFT OUTER JOIN "table_a" ON ("big_table"."id" =

Re: [PERFORM] Reverse Key Index

2015-03-05 Thread Sven R. Kunze
On 26.02.2015 13:37, Heikki Linnakangas wrote: On 02/26/2015 12:31 AM, Josh Berkus wrote: On 02/14/2015 10:35 AM, Sven R. Kunze wrote: Thanks for the immediate reply. I understand the use case is quite limited. On the other hand, I see potential when it comes to applications which use

Re: [PERFORM] Reverse Key Index

2015-02-26 Thread Sven R. Kunze
On 26.02.2015 13:48, Thomas Kellerer wrote: Sven R. Kunze schrieb am 26.02.2015 um 13:23: If you think Reverse Key Indexes have no usage here in PostgreSQL, you should not support convenience features for easily improving performance without breaking the querying API Sorry for my bad English

Re: [PERFORM] Reverse Key Index

2015-02-26 Thread Sven R. Kunze
On 25.02.2015 23:31, Josh Berkus wrote: On 02/14/2015 10:35 AM, Sven R. Kunze wrote: Thanks for the immediate reply. I understand the use case is quite limited. On the other hand, I see potential when it comes to applications which use PostgreSQL. There, programmers would have to change a lot

Re: [PERFORM] Reverse Key Index

2015-02-26 Thread Sven R. Kunze
On 26.02.2015 12:45, Thomas Kellerer wrote: Sven R. Kunze schrieb am 26.02.2015 um 12:04: I just thought about btree indexes here mainly because they well-known and well-used in ORM frameworks. If your ORM framework needs to know about the internals of an index definition or even requires

[PERFORM] Reverse Key Index

2015-02-14 Thread Sven R. Kunze
Hi, does PostgreSQL support the concept of reverse key indexing as described here? I couldn't find any documentation on this yet. http://www.toadworld.com/platforms/oracle/w/wiki/11075.reverse-key-index-from-the-concept-to-internals.aspx Regards, -- Sven R. Kunze TBZ-PARIV GmbH, Bernsdorfer

Re: [PERFORM] Reverse Key Index

2015-02-14 Thread Sven R. Kunze
. On 14.02.2015 19:18, Tom Lane wrote: Sven R. Kunze srku...@tbz-pariv.de writes: does PostgreSQL support the concept of reverse key indexing as described here? I couldn't find any documentation on this yet. http://www.toadworld.com/platforms/oracle/w/wiki/11075.reverse-key-index-from-the-concept

[PERFORM] When does PostgreSQL collapse subqueries to join?

2014-12-10 Thread Sven R. Kunze
and achieve the performance of the first one? Best regards, -- Sven R. Kunze TBZ-PARIV GmbH, Bernsdorfer Str. 210-212, 09130 Chemnitz Tel: +49 (0)371 5347916, Fax: +49 (0)371 5347920 e-mail: srku...@tbz-pariv.de web: www.tbz-pariv.de Geschäftsführer: Dr. Reiner Wohlgemuth Sitz der Gesellschaft