Re: [HACKERS] Block level parallel vacuum

2019-10-24 Thread Dilip Kumar
On Fri, Oct 25, 2019 at 10:22 AM Masahiko Sawada wrote: > > On Fri, Oct 25, 2019 at 12:44 PM Dilip Kumar wrote: > > > > On Thu, Oct 24, 2019 at 8:12 PM Masahiko Sawada > > wrote: > > > > > > On Thu, Oct 24, 2019 at 3:21 PM Dilip Kumar wrote: > > > > > > > > On Fri, Oct 18, 2019 at 12:18 PM

Re: pglz performance

2019-10-24 Thread Andrey Borodin
> 21 окт. 2019 г., в 14:09, Andrey Borodin написал(а): > > With Silesian corpus pglz_decompress_hacked is actually decreasing > performance on high-entropy data. > Meanwhile pglz_decompress_hacked8 is still faster than usual pglz_decompress. > In spite of this benchmarks, I think that

Re: [HACKERS] Block level parallel vacuum

2019-10-24 Thread Masahiko Sawada
On Fri, Oct 25, 2019 at 12:44 PM Dilip Kumar wrote: > > On Thu, Oct 24, 2019 at 8:12 PM Masahiko Sawada wrote: > > > > On Thu, Oct 24, 2019 at 3:21 PM Dilip Kumar wrote: > > > > > > On Fri, Oct 18, 2019 at 12:18 PM Dilip Kumar > > > wrote: > > > > > > > > On Fri, Oct 18, 2019 at 11:25 AM Amit

Re: [HACKERS] WAL logging problem in 9.4.3?

2019-10-24 Thread Kyotaro Horiguchi
Ugh! 2019年10月25日(金) 13:13 Kyotaro Horiguchi : > that. Instead, In the attached, MarkBufferDirtyHint() asks storage.c > for sync-pending state of the relfilenode for the buffer. In the > attached patch (0003) > regards. > It's wrong that it also skips chnging flags. I"ll fix it soon -- Kyotaro

Re: [HACKERS] WAL logging problem in 9.4.3?

2019-10-24 Thread Kyotaro Horiguchi
Hello. Thanks for the comment. # Sorry in advance for possilbe breaking the thread. > MarkBufferDirtyHint() writes WAL even when rd_firstRelfilenodeSubid or > rd_createSubid is set; see attached test case. It needs to skip WAL whenever > RelationNeedsWAL() returns false. Thanks for pointing

Re: [HACKERS] Block level parallel vacuum

2019-10-24 Thread Dilip Kumar
On Thu, Oct 24, 2019 at 8:12 PM Masahiko Sawada wrote: > > On Thu, Oct 24, 2019 at 3:21 PM Dilip Kumar wrote: > > > > On Fri, Oct 18, 2019 at 12:18 PM Dilip Kumar wrote: > > > > > > On Fri, Oct 18, 2019 at 11:25 AM Amit Kapila > > > wrote: > > > > > > > > On Fri, Oct 18, 2019 at 8:45 AM Dilip

Re: dropdb --force

2019-10-24 Thread Amit Kapila
On Thu, Oct 24, 2019 at 8:22 PM Pavel Stehule wrote: > > čt 24. 10. 2019 v 11:10 odesílatel Amit Kapila > napsal: >> >> While making some changes in the patch, I noticed that in >> TerminateOtherDBBackends, there is a race condition where after we >> release the ProcArrayLock, the backend

Re: EXPLAIN BUFFERS and I/O timing accounting questions

2019-10-24 Thread Andres Freund
Hi, Migrating to -hackers, this seems clearly suboptimal. and confusing. The original thread is at https://www.postgresql.org/message-id/20191025003834.2rswu7smheaddag3%40alap3.anarazel.de On 2019-10-24 17:38:34 -0700, Andres Freund wrote: > Perhaps you could send me the full plan and query

Re: [HACKERS] Block level parallel vacuum

2019-10-24 Thread Amit Kapila
On Fri, Oct 25, 2019 at 7:37 AM Amit Kapila wrote: > > On Thu, Oct 24, 2019 at 8:12 PM Masahiko Sawada wrote: > > > > On Thu, Oct 24, 2019 at 3:21 PM Dilip Kumar wrote: > > > > > > I have come up with the POC for approach (a). > > > > > > The idea is > > > 1) Before launching the worker divide

Re: [HACKERS] Block level parallel vacuum

2019-10-24 Thread Amit Kapila
On Thu, Oct 24, 2019 at 8:12 PM Masahiko Sawada wrote: > > On Thu, Oct 24, 2019 at 3:21 PM Dilip Kumar wrote: > > > > I have come up with the POC for approach (a). > > > > The idea is > > 1) Before launching the worker divide the current VacuumCostBalance > > among workers so that workers start

Re: Duplicate Workers entries in some EXPLAIN plans

2019-10-24 Thread Andres Freund
Hi, On 2019-10-22 11:58:35 -0700, Maciek Sakrejda wrote: > I originally reported this in pgsql-bugs [0], but there wasn't much > feedback there, so moving the discussion here. When using JSON, YAML, or > XML-format EXPLAIN on a plan that uses a parallelized sort, the Sort nodes > list two

Re: v12.0: reindex CONCURRENTLY: lock ShareUpdateExclusiveLock on object 14185/39327/0 is already held

2019-10-24 Thread Michael Paquier
On Wed, Oct 23, 2019 at 10:08:21PM -0500, Justin Pryzby wrote: > On Thu, Oct 24, 2019 at 11:42:04AM +0900, Michael Paquier wrote: >> Please see the attached. Justin, does it fix your problems regarding >> the locks? > > Confirmed. Okay, committed and back-patched. I have checked manually all

Re: Creating foreign key on partitioned table is too slow

2019-10-24 Thread Andres Freund
Hi, On 2019-10-23 05:59:01 +, kato-...@fujitsu.com wrote: > To benchmark with tpcb model, I tried to create a foreign key in the > partitioned history table, but backend process killed by OOM. > the number of partitions is 8192. I tried in master(commit: ad4b7aeb84). Obviously this should

Re: Creating foreign key on partitioned table is too slow

2019-10-24 Thread Tomas Vondra
On Fri, Oct 25, 2019 at 12:17:58AM +0200, Tomas Vondra wrote: ... FWIW, even with this fix it still takes an awful lot to create the foreign key, because the CPU is stuck doing this 60.78%60.78% postgres postgres[.] bms_equal 32.58%32.58% postgres postgres

Re: WIP: expression evaluation improvements

2019-10-24 Thread Andres Freund
Hi, On 2019-10-25 00:43:37 +0200, Andreas Karlsson wrote: > On 10/23/19 6:38 PM, Andres Freund wrote: > > In the attached *prototype* patch series there's a lot of incremental > > improvements (and some cleanups) (in time, not importance order): > > You may already know this but your patch set

Re: WIP: expression evaluation improvements

2019-10-24 Thread Andreas Karlsson
On 10/23/19 6:38 PM, Andres Freund wrote: In the attached *prototype* patch series there's a lot of incremental improvements (and some cleanups) (in time, not importance order): You may already know this but your patch set seems to require clang 9. I get the below compilation error which is

Re: WIP: expression evaluation improvements

2019-10-24 Thread Andres Freund
Hi, On 2019-10-24 14:59:21 -0700, Soumyadeep Chakraborty wrote: > After looking at > v2-0006-jit-Reference-functions-by-name-in-IOCOERCE-steps.patch, I was > wondering > about other places in the code where we have const pointers to functions > outside > LLVM's purview: specially EEOP_FUNCEXPR*

Re: Creating foreign key on partitioned table is too slow

2019-10-24 Thread Tomas Vondra
On Thu, Oct 24, 2019 at 03:48:57PM -0300, Alvaro Herrera wrote: On 2019-Oct-23, kato-...@fujitsu.com wrote: Hello To benchmark with tpcb model, I tried to create a foreign key in the partitioned history table, but backend process killed by OOM. the number of partitions is 8192. I tried in

Re: WIP: expression evaluation improvements

2019-10-24 Thread Soumyadeep Chakraborty
Hey Andres, After looking at v2-0006-jit-Reference-functions-by-name-in-IOCOERCE-steps.patch, I was wondering about other places in the code where we have const pointers to functions outside LLVM's purview: specially EEOP_FUNCEXPR* for any function call expressions, EEOP_DISTINCT and EEOP_NULLIF

Re: psql tab-complete

2019-10-24 Thread Victor Spirin
Yes, I found, that VA_ARGS_NARGS(__ VA_ARGS__) macros always return 1 on Windows. Victor Spirin Postgres Professional:http://www.postgrespro.com The Russian Postgres Company 25.10.2019 0:48, Tom Lane пишет: Victor Spirin writes: I found some problem with tab-complete in the 12 version.  I

Re: psql tab-complete

2019-10-24 Thread Tom Lane
Victor Spirin writes: > I found some problem with tab-complete in the 12 version.  I checked > this in the Windows. This change seems to break the case intended by the comment, ie given the context SELECT * FROM tablename WHERE we want to offer the columns of "tablename" as

Re: psql tab-complete

2019-10-24 Thread Victor Spirin
Sorry for wrong place and contents of my message. It seems that the VA_ARGS_NARGS (__ VA_ARGS__) macros always return 1 on Windows. Victor Spirin Postgres Professional:http://www.postgrespro.com The Russian Postgres Company 24.10.2019 19:11, Victor Spirin пишет: I found some problem with

Re: tuplesort test coverage

2019-10-24 Thread Andres Freund
Hi, On 2019-10-24 11:10:34 -0700, Andres Freund wrote: > On 2019-10-15 13:05:32 +0100, Peter Geoghegan wrote: > > > - aborted abbreviated keys (lines 1522, 1608, 1774, 3620, 3739, 3867, > > > 4266) > > > > Also hard to test -- there was a bug here when abbreviated keys first > > went in -- that

TOAST corruption in standby database

2019-10-24 Thread Alex Adriaanse
We have primary and hot standby databases running Postgres 11.3 inside Docker, with their data directories bind-mounted to a reflink-enabled XFS filesystem. The VM is running Debian's 4.19.16-1~bpo9+1 kernel inside an AWS EC2 instance. I've seen TOAST corruption in one of the standby databases

Re: Creating foreign key on partitioned table is too slow

2019-10-24 Thread Alvaro Herrera
On 2019-Oct-23, kato-...@fujitsu.com wrote: > Hello > > To benchmark with tpcb model, I tried to create a foreign key in the > partitioned history table, but backend process killed by OOM. > the number of partitions is 8192. I tried in master(commit: ad4b7aeb84). > > I did the same thing in

Re: Consider low startup cost in add_partial_path

2019-10-24 Thread James Coleman
On Fri, Oct 4, 2019 at 8:36 AM Robert Haas wrote: > > On Wed, Oct 2, 2019 at 10:22 AM James Coleman wrote: > > In all cases I've been starting with: > > > > set enable_hashjoin = off; > > set enable_nestloop = off; > > set max_parallel_workers_per_gather = 4; > > set min_parallel_index_scan_size

Re: tuplesort test coverage

2019-10-24 Thread Andres Freund
Hi, On 2019-10-15 13:05:32 +0100, Peter Geoghegan wrote: > > - aborted abbreviated keys (lines 1522, 1608, 1774, 3620, 3739, 3867, 4266) > > Also hard to test -- there was a bug here when abbreviated keys first > went in -- that was detected by amcheck. > > All of the places where we abort are

Re: Add json_object(text[], json[])?

2019-10-24 Thread Tom Lane
Paul A Jungwirth writes: > On Thu, Oct 24, 2019 at 8:52 AM Tom Lane wrote: >> Also, as the prototype implementation shows, it's not like you >> can't get this functionality today ... you just need to cast >> jsonb to text. Admittedly that's annoying and wasteful. > I don't think that gives the

Re: v12.0: ERROR: could not find pathkey item to sort

2019-10-24 Thread Tom Lane
Amit Langote writes: > On Mon, Oct 14, 2019 at 11:54 PM Tom Lane wrote: >> In view of the proposed patches being dependent on some other >> 13-only changes, I wonder if we should fix v12 by reverting >> d25ea0127. The potential planner performance loss for large >> partition sets could be

Re: Add json_object(text[], json[])?

2019-10-24 Thread Paul A Jungwirth
On Thu, Oct 24, 2019 at 8:45 AM Nikita Glukhov wrote: > You can simply use jsonb_object_agg() to build a jsonb object from a sequence > of transformed key-value pairs: I've even used that function before. :-) I tried finding it on the JSON functions page but couldn't, so I thought maybe I was

Re: Add json_object(text[], json[])?

2019-10-24 Thread Paul A Jungwirth
On Thu, Oct 24, 2019 at 8:52 AM Tom Lane wrote: > I think a potential problem is creation of ambiguity where there was > none before. I agree that's not nice, and it seems like a new name might be better. > Also, as the prototype implementation shows, it's not like you > can't get this

psql tab-complete

2019-10-24 Thread Victor Spirin
I found some problem with tab-complete in the 12 version.  I checked this in the Windows. Victor Spirin Postgres Professional:http://www.postgrespro.com The Russian Postgres Company diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c index e00dbab5aa..5d7f24e57a 100644 ---

Re: Add json_object(text[], json[])?

2019-10-24 Thread Tom Lane
Paul Jungwirth writes: > I noticed that our existing 2-param json{,b}_object functions take > text[] for both keys and values, so they are only able to build > one-layer-deep JSON objects. I'm interested in adding json{,b}_object > functions that take text[] for the keys and json{,b}[] for the

Re: WIP: System Versioned Temporal Table

2019-10-24 Thread Vik Fearing
On 24/10/2019 16:54, Surafel Temesgen wrote: > > hi Vik, > On Wed, Oct 23, 2019 at 9:02 PM Vik Fearing > mailto:vik.fear...@2ndquadrant.com>> wrote: >   > > > If we're going to be implicitly adding stuff to the PK, we also > need to > add that stuff to the other unique constraints, no? 

Re: Add json_object(text[], json[])?

2019-10-24 Thread Nikita Glukhov
On 24.10.2019 18:17, Paul Jungwirth wrote: Hello, I noticed that our existing 2-param json{,b}_object functions take text[] for both keys and values, so they are only able to build one-layer-deep JSON objects. I'm interested in adding json{,b}_object functions that take text[] for the keys

Add json_object(text[], json[])?

2019-10-24 Thread Paul Jungwirth
Hello, I noticed that our existing 2-param json{,b}_object functions take text[] for both keys and values, so they are only able to build one-layer-deep JSON objects. I'm interested in adding json{,b}_object functions that take text[] for the keys and json{,b}[] for the values. It would

Re: pgbench - extend initialization phase control

2019-10-24 Thread Fabien COELHO
Hello, Yep. Or anything else, including without (), to allow checking the performance impact or non impact of transactions on the initialization phase. Is there actually such performance impact? AFAIR most time-consuming part in initialization phase is the generation of pgbench_accounts

Re: WIP: System Versioned Temporal Table

2019-10-24 Thread Surafel Temesgen
hi Vik, On Wed, Oct 23, 2019 at 9:02 PM Vik Fearing wrote: > > If we're going to be implicitly adding stuff to the PK, we also need to > add that stuff to the other unique constraints, no? And I think it > would be better to add both the start and the end column to these keys. > Most of the

Re: dropdb --force

2019-10-24 Thread Pavel Stehule
čt 24. 10. 2019 v 11:10 odesílatel Amit Kapila napsal: > On Wed, Oct 23, 2019 at 12:59 PM Amit Kapila > wrote: > > > > On Tue, Oct 22, 2019 at 4:51 PM Pavel Stehule > wrote: > > > > > > út 22. 10. 2019 v 5:09 odesílatel Amit Kapila > napsal: > > >> > > >> > > >> CountOtherDBBackends is called

Re: [HACKERS] Block level parallel vacuum

2019-10-24 Thread Masahiko Sawada
On Thu, Oct 24, 2019 at 3:21 PM Dilip Kumar wrote: > > On Fri, Oct 18, 2019 at 12:18 PM Dilip Kumar wrote: > > > > On Fri, Oct 18, 2019 at 11:25 AM Amit Kapila > > wrote: > > > > > > On Fri, Oct 18, 2019 at 8:45 AM Dilip Kumar wrote: > > > > > > > > On Thu, Oct 17, 2019 at 4:00 PM Amit Kapila

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2019-10-24 Thread Amit Kapila
On Tue, Oct 22, 2019 at 10:30 AM Dilip Kumar wrote: > > I have merged bugs_and_review_comments_fix.patch changes to 0001 and 0002. > I was wondering whether we have checked the code coverage after this patch? Previously, the existing tests seem to be covering most parts of the function

Re: pgbench - extend initialization phase control

2019-10-24 Thread Fujii Masao
On Thu, Oct 24, 2019 at 9:16 PM Fabien COELHO wrote: > > > Hello Masao-san, > > >> The benefit of controlling where begin/end actually occur is that it may > >> have an impact on performance, and it allows to check that. > > > > I still fail to understand the benefit of addition of () settings. >

Re: pgbench - extend initialization phase control

2019-10-24 Thread Fabien COELHO
Hello Masao-san, The benefit of controlling where begin/end actually occur is that it may have an impact on performance, and it allows to check that. I still fail to understand the benefit of addition of () settings. Could you clarify what case () settings are useful for? You are thinking to

Re: Fix of fake unlogged LSN initialization

2019-10-24 Thread Michael Paquier
On Thu, Oct 24, 2019 at 11:57:33AM +0100, Simon Riggs wrote: > I wonder why is that value 1000, rather than an aligned value or a whole > WAL page? Good question. Heikki, why this choice? -- Michael signature.asc Description: PGP signature

Re: WIP/PoC for parallel backup

2019-10-24 Thread Asif Rehman
On Thu, Oct 24, 2019 at 3:21 PM Ibrar Ahmed wrote: > > > On Fri, Oct 18, 2019 at 4:12 PM Jeevan Chalke < > jeevan.cha...@enterprisedb.com> wrote: > >> >> >> On Thu, Oct 17, 2019 at 10:51 AM Asif Rehman >> wrote: >> >>> >>> Attached are the updated patches. >>> >> >> I had a quick look over

Re: [HACKERS] Block level parallel vacuum

2019-10-24 Thread Dilip Kumar
On Thu, Oct 24, 2019 at 4:21 PM Amit Kapila wrote: > > On Thu, Oct 24, 2019 at 11:51 AM Dilip Kumar wrote: > > > > On Fri, Oct 18, 2019 at 12:18 PM Dilip Kumar wrote: > > > > > > On Fri, Oct 18, 2019 at 11:25 AM Amit Kapila > > > wrote: > > > > > > > > I am thinking if we can write the patch

Re: Fix of fake unlogged LSN initialization

2019-10-24 Thread Simon Riggs
On Mon, 21 Oct 2019 at 06:03, Michael Paquier wrote: > On Sat, Oct 19, 2019 at 05:03:00AM +, tsunakawa.ta...@fujitsu.com > wrote: > > The attached trivial patch fixes the initialization of the fake > > unlogged LSN. Currently, BootstrapXLOG() in initdb sets the initial > > fake unlogged LSN

Re: [HACKERS] Block level parallel vacuum

2019-10-24 Thread Amit Kapila
On Thu, Oct 24, 2019 at 11:51 AM Dilip Kumar wrote: > > On Fri, Oct 18, 2019 at 12:18 PM Dilip Kumar wrote: > > > > On Fri, Oct 18, 2019 at 11:25 AM Amit Kapila > > wrote: > > > > > > I am thinking if we can write the patch for both the approaches (a. > > > compute shared costs and try to

Re: WIP/PoC for parallel backup

2019-10-24 Thread Ibrar Ahmed
On Fri, Oct 18, 2019 at 4:12 PM Jeevan Chalke < jeevan.cha...@enterprisedb.com> wrote: > > > On Thu, Oct 17, 2019 at 10:51 AM Asif Rehman > wrote: > >> >> Attached are the updated patches. >> > > I had a quick look over these changes and they look good overall. > However, here are my few review

Re: v12.0: ERROR: could not find pathkey item to sort

2019-10-24 Thread Amit Langote
Sorry about the late reply. On Mon, Oct 14, 2019 at 11:54 PM Tom Lane wrote: > Justin Pryzby writes: > > On Sun, Oct 13, 2019 at 01:30:29PM -0500, Justin Pryzby wrote: > >> BTW it probably should've been documented as an "Open Item" for v12. > > > https://commitfest.postgresql.org/25/2278/ > >

Re:Re: [BUG] standby node can not provide service even it replays all log files

2019-10-24 Thread Thunder
Thanks for replay.I feel confused about snapshot. At 2019-10-23 11:51:19, "Kyotaro Horiguchi" wrote: >Hello. > >At Tue, 22 Oct 2019 20:42:21 +0800 (CST), Thunder wrote in >> Update the patch. >> >> 1. The STANDBY_SNAPSHOT_PENDING state is set when we replay the first >> XLOG_RUNNING_XACTS

Re: Zedstore - compressed in-core columnar storage

2019-10-24 Thread Ashutosh Sharma
On Thu, Oct 17, 2019 at 2:11 PM Heikki Linnakangas wrote: > > On 15/10/2019 13:49, Ashutosh Sharma wrote: > > Hi, > > > > I got chance to spend some time looking into the recent changes done > > in the zedstore code, basically the functions for packing datums into > > the attribute streams and

Re: Ordering of header file inclusion

2019-10-24 Thread Amit Kapila
On Wed, Oct 23, 2019 at 10:23 AM Amit Kapila wrote: > > Attached are patches for (a) and (b) after another round of review and > fixes by Vignesh. I am planning to commit the first one (a) tomorrow > morning and then if everything is fine on buildfarm, I will commit the > second one (b) and

Re: dropdb --force

2019-10-24 Thread Amit Kapila
eliminate this race condition and whether it is a good idea to accept such a race condition even though it exists in other parts of code. What do you think? BTW, I have added/revised some comments in the code and done few other cosmetic changes, the result of which is attached. With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com drop-database-force-20191024.amit.patch Description: Binary data

Re: Fix of fake unlogged LSN initialization

2019-10-24 Thread Dilip Kumar
On Sat, Oct 19, 2019 at 3:18 PM tsunakawa.ta...@fujitsu.com wrote: > > Hello, > > > The attached trivial patch fixes the initialization of the fake unlogged LSN. > Currently, BootstrapXLOG() in initdb sets the initial fake unlogged LSN to > FirstNormalUnloggedLSN (=1000), but the recovery and

Re: pgbench - refactor init functions with buffers

2019-10-24 Thread Fabien COELHO
Hello Jeevan, +static void +executeStatementExpect(PGconn *con, const char *sql, const ExecStatusType expected, bool errorOK) +{ I think some instances like this need 80 column alignment? Yep. Applying the pgindent is kind-of a pain, so I tend to do a reasonable job by hand and rely on the

Re: [HACKERS] Block level parallel vacuum

2019-10-24 Thread Dilip Kumar
On Fri, Oct 18, 2019 at 12:18 PM Dilip Kumar wrote: > > On Fri, Oct 18, 2019 at 11:25 AM Amit Kapila wrote: > > > > On Fri, Oct 18, 2019 at 8:45 AM Dilip Kumar wrote: > > > > > > On Thu, Oct 17, 2019 at 4:00 PM Amit Kapila > > > wrote: > > > > > > > > On Thu, Oct 17, 2019 at 3:25 PM Dilip