Re: CVE-2017-7484-induced bugs, or, btree cmp functions are not leakproof?

2019-09-05 Thread Dilip Kumar
; to hit the no-mapping case. The original proposal was to fall back > > to current behavior (test the child-table permissions) if we couldn't > > match the var to the top parent, and I think that that is still a > > sane proposal. > > OK, I've removed the Assert. For child Vars that can't be translated > to root parent's, permissions are checked with the child relation, > like before. Instead of falling back to the child, isn't it make more sense to check the permissions on the parent upto which we could translate (it may not be the root parent)? -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: CVE-2017-7484-induced bugs, or, btree cmp functions are not leakproof?

2019-09-05 Thread Dilip Kumar
On Thu, Sep 5, 2019 at 2:48 PM Dilip Kumar wrote: > > On Thu, Sep 5, 2019 at 2:12 PM Amit Langote wrote: > > Thanks for the patch, I was almost about to press the send button with > my patch. But, this looks similar to my version. > > > > On Wed, Sep 4, 2019

Re: CVE-2017-7484-induced bugs, or, btree cmp functions are not leakproof?

2019-09-05 Thread Dilip Kumar
On Thu, Sep 5, 2019 at 3:26 PM Amit Langote wrote: > > On Thu, Sep 5, 2019 at 6:33 PM Dilip Kumar wrote: > > /* > > + * For inheritance child relations, we also need to remember > > + * the root parent. > > + */ > > + if (parent->rtekind == RTE_RELAT

Re: pgbench - allow to create partitioned tables

2019-09-12 Thread Dilip Kumar
good idea to expand it to support multi-level partitioning? -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: pgbench - allow to create partitioned tables

2019-09-13 Thread Dilip Kumar
ORMAT, (p-1) * part_size + 1); then also it will give 1 as minvalue for the first partition and that will be the right thing to do. Am I missing something here? -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: pgbench - allow to create partitioned tables

2019-09-13 Thread Dilip Kumar
lways within the defined range minvalue=1 and maxvalue=scale*10, right? > > It also probably reduces the cost of checking whether a value belongs to > the first partition because one test is removed, so there is a small > additional performance benefit beyond principle and cove

Re: POC: Cleaning up orphaned files using undo logs

2019-07-30 Thread Dilip Kumar
On Tue, Jul 30, 2019 at 12:21 PM Thomas Munro wrote: > > On Tue, Jul 30, 2019 at 5:03 PM Dilip Kumar wrote: > > > > I think this idea is good for the DO time but during REDO time it will > > not work as we will not have the transaction state. Having said that > &g

Re: POC: Cleaning up orphaned files using undo logs

2019-08-07 Thread Dilip Kumar
h a bunch of branches, is > even more expensive, especially when it's shared memory, especially when > all that shared meory is locked at once. My idea is, indeed of keeping all these fields duplicated in the context, just allocate a single memory segment equal to the expected record size (maybe the payload data can keep separate). Now, based on uur_info pack all the field of UnpackedUndoRecord in that memory segment. After that In InsertUndoData, we just need one call to InsertUndoBytes for copying complete header in one shot and another call for copying payload data. Does this sound reasonable to you? -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: POC: Cleaning up orphaned files using undo logs

2019-08-07 Thread Dilip Kumar
atch set we always need to fetch one record so instead of repeating this pattern everywhere we can write one function and move this sequence of calls in that function. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: POC: Cleaning up orphaned files using undo logs

2019-08-07 Thread Dilip Kumar
On Tue, Jul 30, 2019 at 1:54 PM Dilip Kumar wrote: > > On Tue, Jul 30, 2019 at 12:21 PM Thomas Munro wrote: > > > > One data structure that could perhaps hold this would be > > UndoLogTableEntry (the per-backend cache, indexed by undo log number, > > with pretty

Re: CVE-2017-7484-induced bugs, or, btree cmp functions are not leakproof?

2019-07-29 Thread Dilip Kumar
On Wed, Jul 17, 2019 at 2:39 PM Amit Langote wrote: > > On Wed, Jul 10, 2019 at 2:43 PM Dilip Kumar wrote: > > On Wed, Jul 10, 2019 at 10:15 AM Amit Langote > > wrote: > > > Thanks for checking. There has been a lot of churn in the inheritance > > &g

Re: POC: Cleaning up orphaned files using undo logs

2019-07-19 Thread Dilip Kumar
> We don't write CID in the WAL. Because In hot-standby or after recovery we don't need actual CID for the visibility. So during REDO while generating the undo record we set CID as 'FirstCommandId' which is different from the DO time. That's the reason we mask it. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: POC: Cleaning up orphaned files using undo logs

2019-07-19 Thread Dilip Kumar
sh table for any concurrent backend to insert the request and this seems too restrictive. Having said that I think we must ensure MaxBackends*2 empty slots in the hash table as each backend can enter 2 requests in the hash table. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: POC: Cleaning up orphaned files using undo logs

2019-07-20 Thread Dilip Kumar
ached to undo-log at any point in time, then we can have a > more precise check. Maybe we can make ProcGlobal->xactsHavingPendingUndo an atomic variable. We can increment its value atomically whenever a) A transaction writes the first undo record for each persistence level. b) For each abor

Re: POC: Cleaning up orphaned files using undo logs

2019-07-23 Thread Dilip Kumar
On Mon, Jul 22, 2019 at 3:51 PM Dilip Kumar wrote: > Please find my review comments for 0013-Allow-foreground-transactions-to-perform-undo-action + /* initialize undo record locations for the transaction */ + for (i = 0; i < UndoLogCategories; i++) + { + s->start_urec_ptr[i] = InvalidU

Re: POC: Cleaning up orphaned files using undo logs

2019-07-24 Thread Dilip Kumar
get > * initialized. After these many attempts it will return error and the user > * can retry the operation. > */ > #define ROLLBACK_HT_INIT_WAIT_TRY 60 > > %s/error/an error > This macro also added under 0014. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: POC: Cleaning up orphaned files using undo logs

2019-07-16 Thread Dilip Kumar
On Wed, Jul 17, 2019 at 9:27 AM Thomas Munro wrote: > > On Wed, Jul 17, 2019 at 3:44 PM Dilip Kumar wrote: > > Right, actually I got that point. But, I was thinking that we are > > wasting one logno from undo log addressing space no?. Instead, if we > > can keep

Re: POC: Cleaning up orphaned files using undo logs

2019-07-16 Thread Dilip Kumar
On Wed, Jul 17, 2019 at 3:48 AM Thomas Munro wrote: > > On Tue, Jul 16, 2019 at 11:33 PM Dilip Kumar wrote: > > On Mon, Jul 1, 2019 at 1:24 PM Thomas Munro wrote: > > /* If we discarded everything, the slot can be given up. */ > > + if (entirely_discarded) >

Re: POC: Cleaning up orphaned files using undo logs

2019-07-24 Thread Dilip Kumar
size) <= slot->meta.end) + { + *need_xact_header = false; + return try_offset; + } Here also you are returning directly try_offset instead of UndoRecPtr -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: POC: Cleaning up orphaned files using undo logs

2019-07-25 Thread Dilip Kumar
On Thu, Jul 25, 2019 at 11:25 AM Dilip Kumar wrote: > > Hi Thomas, > > I have started reviewing 0003-Add-undo-log-manager, I haven't yet > reviewed but some places I noticed that instead of UndoRecPtr you are > directly > using UndoLogOffset. Which seems like bugs to me

Re: Warning messages appearing twice

2019-07-25 Thread Dilip Kumar
as you have not enabled the logging collector the WARNING is printed on your console. > > I'm not sure if this is already known. > I'm not sure if this is widely used scenario or not. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: POC: Cleaning up orphaned files using undo logs

2019-07-22 Thread Dilip Kumar
oFetchRecord should directly rely on the field because ideally all the fields in UnpackUndoRecord must be set and uur_txt or uur_logswitch will be allocated when those headers present. I think this needs to be improved in undo interface patch as well (in UndoBulkFetchRecord). -- Regard

Re: pgbench - allow to create partitioned tables

2019-09-19 Thread Dilip Kumar
following this thread, would > like to comment? It is important to know others opinion on this > because this will change how pgbench behaves with prior versions. IMHO, we don't need to invent the error handling at the pgbench instead we can rely on the server's error. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: A problem about partitionwise join

2019-09-21 Thread Dilip Kumar
On Fri, Sep 20, 2019 at 2:33 PM Richard Guo wrote: > > Hi Dilip, > > Thank you for reviewing this patch. > > On Fri, Sep 20, 2019 at 12:48 PM Dilip Kumar wrote: >> >> On Thu, Aug 29, 2019 at 3:15 PM Richard Guo wrote: >> > >> > >> >

Re: A problem about partitionwise join

2019-09-19 Thread Dilip Kumar
; + I am wondering isn't it possible to just process the missing join clause? I mean 'generate_join_implied_equalities' has only skipped the ECs which has const so can't we create join clause only for those ECs and append it the "Restrictlist" we already have? I might be missing

Re: dropdb --force

2019-09-19 Thread Dilip Kumar
better to keep it how it was earlier. -extern bool CountOtherDBBackends(Oid databaseId, - int *nbackends, int *nprepared); +extern bool CountOtherDBBackends(Oid databaseId, int *nbackends, int *nprepared); Unrelated change -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2019-09-28 Thread Dilip Kumar
an infrastructure such that we can check the status of the transaction for which we are decoding the change. Also, I think we need to handle the ERRCODE_TRANSACTION_ROLLBACK and ignore it. I have attached a small patch to handle this which can be applied on top of your patch set. -- Regards, Dilip Kumar

Re: abort-time portal cleanup

2019-09-24 Thread Dilip Kumar
e set portal->createSubid to InvalidSubTransactionId. So it seems to me that the second condition will never reach. Am I missing something? -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: POC: Cleaning up orphaned files using undo logs

2019-07-06 Thread Dilip Kumar
bug fixes in undo log. - Fixup for Orphan file cleanup to pass dbid in PrepareUndoInsert -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com undo_20190706.tar.gz Description: GNU Zip compressed data

Re: CVE-2017-7484-induced bugs, or, btree cmp functions are not leakproof?

2019-07-09 Thread Dilip Kumar
On Wed, Jul 10, 2019 at 9:44 AM Dilip Kumar wrote: > > On Fri, Nov 2, 2018 at 1:34 PM Amit Langote > wrote: > > > > On 2018/11/01 20:34, Dilip Kumar wrote: > > > On Mon, Oct 29, 2018 at 2:53 PM Amit Langote wrote: > > >> Anyway, why don't we just

Re: CVE-2017-7484-induced bugs, or, btree cmp functions are not leakproof?

2019-07-09 Thread Dilip Kumar
On Fri, Nov 2, 2018 at 1:34 PM Amit Langote wrote: > > On 2018/11/01 20:34, Dilip Kumar wrote: > > On Mon, Oct 29, 2018 at 2:53 PM Amit Langote wrote: > >> Anyway, why don't we just use the child table's AppendRelInfo to get the > >> parent's version of varattno inst

Re: CVE-2017-7484-induced bugs, or, btree cmp functions are not leakproof?

2019-07-09 Thread Dilip Kumar
On Wed, Jul 10, 2019 at 10:15 AM Amit Langote wrote: > > Hi Dilip, > > On Wed, Jul 10, 2019 at 1:29 PM Dilip Kumar wrote: > > On Wed, Jul 10, 2019 at 9:44 AM Dilip Kumar wrote: > > > On Fri, Nov 2, 2018 at 1:34 PM Amit Langote wrote: > > > > Okay, here are

Re: [HACKERS] Block level parallel vacuum

2019-10-30 Thread Dilip Kumar
On Tue, Oct 29, 2019 at 3:11 PM Dilip Kumar wrote: > > On Tue, Oct 29, 2019 at 1:59 PM Masahiko Sawada wrote: > > > > On Tue, Oct 29, 2019 at 4:06 PM Masahiko Sawada > > wrote: > > > > > > On Mon, Oct 28, 2019 at 2:13 PM Dilip Kumar wrote: > > &

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2019-10-30 Thread Dilip Kumar
ling if the subform->subworkmem is NULL. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2019-11-04 Thread Dilip Kumar
r by hacking the code such that it serializes after every change and > then if you abort after one change, it should hit this problem. > I think you might need to kill the server after all changes are serialized otherwise normal abort will hit the ReorderBufferAbort and that will remove your ReorderBufferTXN entry and you will never hit this case. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: [HACKERS] Block level parallel vacuum

2019-11-02 Thread Dilip Kumar
On Fri, Nov 1, 2019 at 2:21 PM Masahiko Sawada wrote: > > On Thu, Oct 31, 2019 at 3:45 PM Dilip Kumar wrote: > > > > On Thu, Oct 31, 2019 at 11:33 AM Dilip Kumar wrote: > > > > > > On Tue, Oct 29, 2019 at 1:59 PM Masahiko Sawada > > > wrote: >

Re: cost based vacuum (parallel)

2019-11-10 Thread Dilip Kumar
On Fri, Nov 8, 2019 at 11:49 AM Amit Kapila wrote: > > On Fri, Nov 8, 2019 at 9:39 AM Dilip Kumar wrote: > > > > I have done some experiments on this line. I have first produced a > > case where we can show the problem with the existing shared costing > > patch (

Re: [HACKERS] Block level parallel vacuum

2019-11-10 Thread Dilip Kumar
of whether bulkdelete is called or not e.g. gin? If so, along with an amcanparallelcleanup flag, we need to consider vacrelstats->num_index_scans right? So if vacrelstats->num_index_scans == 0 then we need to launch parallel worker for all the indexes who support amcanparallelvacuum and if vacrelstats->num_index_scans > 0 then only for those who has amcanparallelcleanup as true. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: [HACKERS] Block level parallel vacuum

2019-11-11 Thread Dilip Kumar
nce_work_mem; Am I missing something? -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: cost based vacuum (parallel)

2019-11-11 Thread Dilip Kumar
On Mon, Nov 11, 2019 at 4:23 PM Amit Kapila wrote: > > On Mon, Nov 11, 2019 at 12:59 PM Dilip Kumar wrote: > > > > On Mon, Nov 11, 2019 at 9:43 AM Dilip Kumar wrote: > > > > > > On Fri, Nov 8, 2019 at 11:49 AM Amit Kapila > > > wrote: > >

Re: [HACKERS] Block level parallel vacuum

2019-11-10 Thread Dilip Kumar
On Mon, Nov 11, 2019 at 12:26 PM Masahiko Sawada wrote: > > On Mon, 11 Nov 2019 at 15:06, Dilip Kumar wrote: > > > > On Mon, Nov 11, 2019 at 9:57 AM Masahiko Sawada > > wrote: > > > > > > On Fri, 8 Nov 2019 at 18:48, Amit Kapila wrote: > > >

Re: cost based vacuum (parallel)

2019-11-10 Thread Dilip Kumar
On Mon, Nov 11, 2019 at 9:43 AM Dilip Kumar wrote: > > On Fri, Nov 8, 2019 at 11:49 AM Amit Kapila wrote: > > > > On Fri, Nov 8, 2019 at 9:39 AM Dilip Kumar wrote: > > > > > > I have done some experiments on this line. I have first produced a >

Re: [HACKERS] Block level parallel vacuum

2019-11-06 Thread Dilip Kumar
On Wed, Nov 6, 2019 at 3:50 PM Masahiko Sawada wrote: > > On Wed, 6 Nov 2019 at 18:42, Dilip Kumar wrote: > > > > On Wed, Nov 6, 2019 at 2:01 PM Mahendra Singh wrote: > > > > > > Hi > > > I took all attached patches(v32-01 to v32-4) and one Dili

Re: Reorderbuffer crash during recovery

2019-11-06 Thread Dilip Kumar
gt; 2) Have many open transactions with subtransactions open. > 3) Attach one of the transaction from gdb and call abort(). Do you need subtransactions for the issue1? It appears that after the restart if the changes list is empty it will hit the assert. Am I missing something? > > I'm not sure of the fix for this. If I get time I will try to spend > more time to find out the fix. > Thoughts? > -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: cost based vacuum (parallel)

2019-11-06 Thread Dilip Kumar
limit. So If we get such a mechanism in the future then it seems that it will be easily expandable to the parallel vacuum, isn't it? Because across workers also we can track tablespace wise shared balance (if we go with the shared costing approach for example). -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: [HACKERS] Block level parallel vacuum

2019-11-06 Thread Dilip Kumar
parallel plan then the user can force and check. But, vacuum parallelism is itself forced by the user so there is no point in doing it with force_parallel_mode=on. However, force_parallel_mode=regress is useful for testing the vacuum with an existing test suit. > > Please let me know your thoughts for this patch. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2019-11-07 Thread Dilip Kumar
illTxns, rb->spillCount, rb->spillBytes); > > Changed the above elog to DEBUG1 as otherwise it was getting printed > very frequently. I think we can make it DEBUG2 if we want. Yeah, it should not be WARNING. > > 6. There was an extra space in rules.out due to which test was > failing. I have fixed it. My Bad. I have induced while separating out the changes for the spilling. > What do you think? I have reviewed your changes and looks fine to me. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: cost based vacuum (parallel)

2019-11-07 Thread Dilip Kumar
Here I have changed the patch2 as per (b) If local balance reaches to 50% of the local limit and shared balance hit the vacuum cost limit then go for the delay. patch 3: Shared costing patch: (delay condition -> VacuumSharedCostBalance > VacuumCostLimit && VacuumLocalBalance >

Re: [HACKERS] Block level parallel vacuum

2019-10-31 Thread Dilip Kumar
7665, total 104149 [1] https://www.postgresql.org/message-id/CAD21AoAqT17QwKJ_sWOqRxNvg66wMw1oZZzf9Rt-E-zD%2BXOh_Q%40mail.gmail.com -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: [HACKERS] Block level parallel vacuum

2019-10-31 Thread Dilip Kumar
On Thu, Oct 31, 2019 at 11:33 AM Dilip Kumar wrote: > > On Tue, Oct 29, 2019 at 1:59 PM Masahiko Sawada wrote: > > Actually after increased shared_buffer I got expected results: > > > > * Test1 (after increased shared_buffers) > > normal : 2807 ms (hit 56295

Re: [HACKERS] Block level parallel vacuum

2019-11-13 Thread Dilip Kumar
information via two variables but the > above seems like a better idea to all the people involved. > > Any suggestions? Anyone thinks this is not the right way to expose > this information or there is no need to expose this information or > they have a better idea for this? > > Sawada-San, Dilip, feel free to correct me. Looks fine to me. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: [HACKERS] Block level parallel vacuum

2019-11-13 Thread Dilip Kumar
On Wed, Nov 13, 2019 at 11:39 AM Masahiko Sawada wrote: > > On Wed, 13 Nov 2019 at 12:43, Dilip Kumar wrote: > > > > On Tue, Nov 12, 2019 at 5:31 PM Masahiko Sawada > > wrote: > > > > > > On Tue, 12 Nov 2019 at 20:29, Dilip Kumar wrote: > > >

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2019-11-13 Thread Dilip Kumar
On Wed, Nov 13, 2019 at 5:55 PM Amit Kapila wrote: > > On Thu, Oct 3, 2019 at 1:18 PM Dilip Kumar wrote: > > > > As mentioned by me a few days back that the first patch in this series > is ready to go [1] (I am hoping Tomas will pick it up), so I have > started the

Re: cost based vacuum (parallel)

2019-11-15 Thread Dilip Kumar
On Thu, Nov 14, 2019 at 5:02 PM Mahendra Singh wrote: > > On Mon, 11 Nov 2019 at 17:56, Amit Kapila wrote: > > > > On Mon, Nov 11, 2019 at 5:14 PM Dilip Kumar wrote: > > > > > > On Mon, Nov 11, 2019 at 4:23 PM Amit Kapila > > > wrote: > &

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2019-11-18 Thread Dilip Kumar
On Fri, Nov 15, 2019 at 4:19 PM Amit Kapila wrote: > > On Fri, Nov 15, 2019 at 4:01 PM Dilip Kumar wrote: > > > > On Fri, Nov 15, 2019 at 3:50 PM Amit Kapila wrote: > > > > > > > > > Few other comments on this patch: > > &

Re: [HACKERS] Block level parallel vacuum

2019-11-12 Thread Dilip Kumar
vacuumcleanup can be done in > parallel if bulkdelete is not performed (Indexes nbtree, brin, hash, > gin, gist, spgist, bloom will set this flag) > VACUUM_OPTION_PARALLEL_CLEANUP 4 # vacuumcleanup can be done in > parallel even if bulkdelete is already performed (Indexes gin, brin, > and bloom will set this flag) > > Does something like this make sense? Yeah, something like that seems better to me. > If we all agree on this, then I > think we can summarize the part of the discussion related to this API > and get feedback from a broader audience. Make sense. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: [HACKERS] Block level parallel vacuum

2019-11-12 Thread Dilip Kumar
On Tue, Nov 12, 2019 at 4:04 PM Masahiko Sawada wrote: > > On Mon, 11 Nov 2019 at 17:57, Dilip Kumar wrote: > > > > On Tue, Oct 29, 2019 at 12:37 PM Masahiko Sawada > > wrote: > > > I realized that v31-0006 patch doesn't work fine so I've attached the >

Re: cost based vacuum (parallel)

2019-11-12 Thread Dilip Kumar
On Tue, Nov 12, 2019 at 10:47 AM Dilip Kumar wrote: > > On Mon, Nov 11, 2019 at 4:23 PM Amit Kapila wrote: > > > > On Mon, Nov 11, 2019 at 12:59 PM Dilip Kumar wrote: > > > > > > On Mon, Nov 11, 2019 at 9:43 AM Dilip Kumar wrote: > > > > >

Re: [HACKERS] Block level parallel vacuum

2019-11-12 Thread Dilip Kumar
On Tue, Nov 12, 2019 at 5:31 PM Masahiko Sawada wrote: > > On Tue, 12 Nov 2019 at 20:29, Dilip Kumar wrote: > > > > On Tue, Nov 12, 2019 at 4:04 PM Masahiko Sawada > > wrote: > > > > > > On Mon, 11 Nov 2019 at 17:57, Dilip Kumar wrote: > > >

Re: [HACKERS] Block level parallel vacuum

2019-11-12 Thread Dilip Kumar
On Wed, Nov 13, 2019 at 9:12 AM Dilip Kumar wrote: > > On Tue, Nov 12, 2019 at 5:31 PM Masahiko Sawada > wrote: > > > > On Tue, 12 Nov 2019 at 20:29, Dilip Kumar wrote: > > > > > > On Tue, Nov 12, 2019 at 4:04 PM Masahiko Sawada > > > wrote: >

Re: [HACKERS] Block level parallel vacuum

2019-11-12 Thread Dilip Kumar
cleanup. > > > > > > > > VACUUM_OPTION_PARALLEL_NO_BULKDEL 1 << 0 > > > > VACUUM_OPTION_PARALLEL_BULKDEL 1 << 1 > > > > VACUUM_OPTION_PARALLEL_NO_CLEANUP 1 << 2 > > > > VACUUM_OPTION_PARALLEL_COND_CLEANUP 1 << 3 > > > > VACUUM_OPTION_PARALLEL_CLEANUP 1 << 4 > > > > > > > > > > This also looks reasonable, but if there is an index that doesn't want > > > to support a parallel vacuum, it needs to set multiple flags. > > > > Right. It would be better to use uint16 as two uint8. I mean that if > > first 8 bits are 0 it means VACUUM_OPTION_PARALLEL_NO_BULKDEL and if > > next 8 bits are 0 means VACUUM_OPTION_PARALLEL_NO_CLEANUP. Other flags > > could be followings: > > > > VACUUM_OPTION_PARALLEL_BULKDEL 0x0001 > > VACUUM_OPTION_PARALLEL_COND_CLEANUP 0x0100 > > VACUUM_OPTION_PARALLEL_CLEANUP 0x0200 > > > > Hmm, I think we should define these flags in the most simple way. > Your previous proposal sounds okay to me. > > -- > With Regards, > Amit Kapila. > EnterpriseDB: http://www.enterprisedb.com -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2019-11-15 Thread Dilip Kumar
On Fri, Nov 15, 2019 at 3:50 PM Amit Kapila wrote: > > On Thu, Nov 14, 2019 at 3:40 PM Dilip Kumar wrote: > > > > > > Apart from this, I have another question in > > 0003-Issue-individual-invalidations-with-wal_level-logical.patch > > > > @@ -543,6 +

Re: Reorderbuffer crash during recovery

2019-11-06 Thread Dilip Kumar
On Thu, Nov 7, 2019 at 9:55 AM vignesh C wrote: > > On Wed, Nov 6, 2019 at 5:41 PM Dilip Kumar wrote: > > > > On Wed, Nov 6, 2019 at 5:20 PM vignesh C wrote: > > > > > > Hi, > > > > > > I found couple of crashes in reorderbuffer whil

Re: [HACKERS] Block level parallel vacuum

2019-11-06 Thread Dilip Kumar
On Wed, 6 Nov 2019, 20:07 Masahiko Sawada, wrote: > On Wed, 6 Nov 2019 at 20:29, Dilip Kumar wrote: > > > > On Wed, Nov 6, 2019 at 3:50 PM Masahiko Sawada > > wrote: > > > > > > On Wed, 6 Nov 2019 at 18:42, Dilip Kumar > wrote: > > > >

Re: [HACKERS] Block level parallel vacuum

2019-11-03 Thread Dilip Kumar
On Mon, Nov 4, 2019 at 10:45 AM Amit Kapila wrote: > > On Sun, Nov 3, 2019 at 9:49 AM Dilip Kumar wrote: > > > > On Fri, Nov 1, 2019 at 2:21 PM Masahiko Sawada > > wrote: > > > > > > > > > I think that two approaches make parallel vac

Re: [HACKERS] Block level parallel vacuum

2019-11-03 Thread Dilip Kumar
t limit evenly. > > > > I am also not completely sure which approach is better but I slightly > lean towards approach (b). I think we need input from some other > people as well. I will start a separate thread to discuss this and > see if that helps to get the input from others. +1 -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: cost based vacuum (parallel)

2019-11-05 Thread Dilip Kumar
allel vacuum with different ring buffer size) 8 index ring buffer size 246kb-> non-parallel: 7.6 seconds parallel (2 worker): 3.9 seconds ring buffer size 256mb-> non-parallel: 6.1 seconds parallel (2 worker): 3.2 seconds 4 index ring buffer size 246kb -> non-parallel: 4.8 seconds parallel (2 worker): 3.2 seconds ring buffer size 256mb -> non-parallel: 3.8 seconds parallel (2 worker): 2.6 seconds -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: cost based vacuum (parallel)

2019-11-05 Thread Dilip Kumar
On Tue, Nov 5, 2019 at 8:49 PM Andres Freund wrote: > > Hi, > > On November 5, 2019 7:16:41 AM PST, Dilip Kumar wrote: > >On Tue, Nov 5, 2019 at 2:40 PM Amit Kapila > >wrote: > >> > >> On Mon, Nov 4, 2019 at 11:58 PM Andres Freund > >wrote: >

Re: [HACKERS] Block level parallel vacuum

2019-11-21 Thread Dilip Kumar
On Thu, 21 Nov 2019, 13:52 Masahiko Sawada, wrote: > On Thu, 21 Nov 2019 at 14:16, Dilip Kumar wrote: > > > > On Wed, Nov 20, 2019 at 11:01 AM Masahiko Sawada > > wrote: > > > > > > On Mon, 18 Nov 2019 at 15:38, Masahiko Sawada > > > wrote: &g

Re: [HACKERS] Block level parallel vacuum

2019-11-21 Thread Dilip Kumar
On Thu, 21 Nov 2019, 14:15 Masahiko Sawada, wrote: > On Thu, 21 Nov 2019 at 14:32, Dilip Kumar wrote: > > > > On Thu, Nov 21, 2019 at 10:46 AM Dilip Kumar > wrote: > > > > > > On Wed, Nov 20, 2019 at 11:01 AM Masahiko Sawada > > > wrote: > >

Re: [HACKERS] Block level parallel vacuum

2019-11-20 Thread Dilip Kumar
mSharedCostBalance, +_balance, +new_balance)) + { + /* Updated successfully, break */ + break; + } While looking at the shared costing delay part, I have noticed that while checking the delay condition, we are considering local_balance which is VacuumCostBalanceLocal + VacuumCostBalance, but while computing the new balance we only reduce shared balance by VacuumCostBalanceLocal, I think it should be reduced with local_balance? I see that later we are adding VacuumCostBalance to the VacuumCostBalanceLocal so we are not loosing accounting for this balance. But, I feel it is not right that we compare based on one value and operate based on other. I think we can immediately set VacuumCostBalanceLocal += VacuumCostBalance before checking the condition. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: [HACKERS] Block level parallel vacuum

2019-11-20 Thread Dilip Kumar
On Thu, Nov 21, 2019 at 10:46 AM Dilip Kumar wrote: > > On Wed, Nov 20, 2019 at 11:01 AM Masahiko Sawada > wrote: > > > > On Mon, 18 Nov 2019 at 15:38, Masahiko Sawada > > wrote: > > > > > > On Mon, 18 Nov 2019 at 15:34, Amit Kapila wrote: >

Re: [HACKERS] Block level parallel vacuum

2019-12-04 Thread Dilip Kumar
index statistics after exited from the > parallel > > The first of these sentences ("Note that all...") is not very clear to > me, and seems like it may amount to a statement that the leader > doesn't try to destroy the parallel context too early, but since I > don't understand it, maybe that's not what it is saying. The second > sentence needs exited -> exiting, and maybe some more work on the > grammar, too. > -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: [HACKERS] Block level parallel vacuum

2019-12-04 Thread Dilip Kumar
On Thu, Dec 5, 2019 at 12:21 AM Masahiko Sawada wrote: > > On Wed, 4 Dec 2019 at 04:57, Dilip Kumar wrote: > > > > On Wed, Dec 4, 2019 at 9:12 AM Amit Kapila wrote: > > > > > > On Wed, Dec 4, 2019 at 1:58 AM Masahiko Sawada > > > wrote: > > &

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2019-12-08 Thread Dilip Kumar
On Mon, Dec 2, 2019 at 2:01 PM Dilip Kumar wrote: > > On Sun, Dec 1, 2019 at 7:58 AM Michael Paquier wrote: > > > > On Fri, Nov 22, 2019 at 01:18:11PM +0530, Dilip Kumar wrote: > > > I have rebased the patch on the latest head and also fix the issue of >

Re: Fastpath while arranging the changes in LSN order in logical decoding

2019-12-16 Thread Dilip Kumar
On Mon, Nov 25, 2019 at 9:22 AM Dilip Kumar wrote: > > In logical decoding, while sending the changes to the output plugin we > need to arrange them in the LSN order. But, if there is only one > transaction which is a very common case then we can avoid building the > binary heap.

Re: [HACKERS] Block level parallel vacuum

2019-12-05 Thread Dilip Kumar
On Fri, Dec 6, 2019 at 12:55 AM Mahendra Singh wrote: > > On Thu, 5 Dec 2019 at 19:54, Robert Haas wrote: > > > > [ Please trim excess quoted material from your replies. ] > > > > On Thu, Dec 5, 2019 at 12:27 AM Dilip Kumar wrote: > > > I agree that t

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2019-12-11 Thread Dilip Kumar
On Wed, Dec 11, 2019 at 5:22 PM Amit Kapila wrote: > > On Mon, Dec 9, 2019 at 1:27 PM Dilip Kumar wrote: > > > > I have review the patch set and here are few comments/questions > > > > 1. > > +static void > > +pg_decode_stream_change(LogicalDecodin

Re: Questions/Observations related to Gist vacuum

2019-10-18 Thread Dilip Kumar
On Fri, Oct 18, 2019 at 10:55 AM Amit Kapila wrote: > > On Fri, Oct 18, 2019 at 9:41 AM Dilip Kumar wrote: > > > > On Wed, Oct 16, 2019 at 7:22 PM Heikki Linnakangas wrote: > > > > > > On 16 October 2019 12:57:03 CEST, Amit Kapila > > > wrote:

Re: [HACKERS] Block level parallel vacuum

2019-10-28 Thread Dilip Kumar
On Mon, Oct 28, 2019 at 12:20 PM Amit Kapila wrote: > > On Sun, Oct 27, 2019 at 12:52 PM Dilip Kumar wrote: > > > > On Fri, Oct 25, 2019 at 9:19 PM Masahiko Sawada > > wrote: > > > > > > > > I haven't yet read the new set of the patch. But, I

Re: [HACKERS] Block level parallel vacuum

2019-10-27 Thread Dilip Kumar
On Thu, Oct 24, 2019 at 4:33 PM Dilip Kumar wrote: > > On Thu, Oct 24, 2019 at 4:21 PM Amit Kapila wrote: > > > > On Thu, Oct 24, 2019 at 11:51 AM Dilip Kumar wrote: > > > > > > On Fri, Oct 18, 2019 at 12:18 PM Dilip Kumar > > > wrote: > >

Re: [HACKERS] Block level parallel vacuum

2019-10-28 Thread Dilip Kumar
On Fri, Oct 25, 2019 at 9:19 PM Masahiko Sawada wrote: > > On Fri, Oct 25, 2019 at 2:06 PM Dilip Kumar wrote: > > > > On Fri, Oct 25, 2019 at 10:22 AM Masahiko Sawada > > wrote: > > > > > > For more detail of my idea it is that the first worker who en

Re: Questions/Observations related to Gist vacuum

2019-10-17 Thread Dilip Kumar
On Thu, Oct 17, 2019 at 6:32 PM Dilip Kumar wrote: > > On Thu, 17 Oct 2019, 14:59 Amit Kapila, wrote: >> >> On Thu, Oct 17, 2019 at 1:47 PM Dilip Kumar wrote: >> > >> > On Thu, Oct 17, 2019 at 12:27 PM Heikki Linnakangas >> > wrote: >>

Re: [HACKERS] Block level parallel vacuum

2019-10-17 Thread Dilip Kumar
On Thu, Oct 17, 2019 at 4:00 PM Amit Kapila wrote: > > On Thu, Oct 17, 2019 at 3:25 PM Dilip Kumar wrote: > > > > On Thu, Oct 17, 2019 at 2:12 PM Masahiko Sawada > > wrote: > > > > > > On Thu, Oct 17, 2019 at 5:30 PM Amit Kapila > > >

Re: Questions/Observations related to Gist vacuum

2019-10-17 Thread Dilip Kumar
is called from parallel vacuum workers or in general? -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: [HACKERS] Block level parallel vacuum

2019-10-18 Thread Dilip Kumar
On Fri, Oct 18, 2019 at 11:25 AM Amit Kapila wrote: > > On Fri, Oct 18, 2019 at 8:45 AM Dilip Kumar wrote: > > > > On Thu, Oct 17, 2019 at 4:00 PM Amit Kapila wrote: > > > > > > On Thu, Oct 17, 2019 at 3:25 PM Dilip Kumar wrote: > > > > > &g

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2019-10-20 Thread Dilip Kumar
true. From next time onwards if we try to serialize the same transaction we will not increment the rb->spillTxns so that we count each transaction only once. > > Also, isn't spillTxns count bit confusing, because in some cases it > will include subtransactions and other cases (where the largest picked > transaction is a subtransaction) it won't include it? I did not understand your comment completely. Basically, every transaction which we are serializing we will increase the count first time right? whether it is the main transaction or the sub-transaction. Am I missing something? -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: Questions/Observations related to Gist vacuum

2019-10-21 Thread Dilip Kumar
er patch, there is no leak. I will commit the patch early next > > week unless Heikki or someone wants some more tests. > > > > Pushed. Thanks. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2019-10-21 Thread Dilip Kumar
On Mon, Oct 21, 2019 at 2:50 PM Amit Kapila wrote: > > On Mon, Oct 21, 2019 at 10:48 AM Dilip Kumar wrote: > > > > On Fri, Oct 18, 2019 at 5:32 PM Amit Kapila wrote: > > > > > 3. > > > @@ -2479,7 +2480,7 @@ ReorderBufferSerializeTXN(Reor

Re: Questions/Observations related to Gist vacuum

2019-10-21 Thread Dilip Kumar
On Mon, Oct 21, 2019 at 2:58 PM Andrey Borodin wrote: > > > > > 21 окт. 2019 г., в 11:12, Dilip Kumar написал(а): > > > > On Mon, Oct 21, 2019 at 2:30 PM Andrey Borodin wrote: > >> > >> I've took a look into the patch, and cannot understan

Re: Questions/Observations related to Gist vacuum

2019-10-21 Thread Dilip Kumar
On Mon, Oct 21, 2019 at 2:30 PM Andrey Borodin wrote: > > Hi! > > > 18 окт. 2019 г., в 13:21, Dilip Kumar написал(а): > > > > On Fri, Oct 18, 2019 at 10:55 AM Amit Kapila > > wrote: > >> > >> > >> I think we can do it in general as ad

Re: Questions/Observations related to Gist vacuum

2019-10-15 Thread Dilip Kumar
; vacuum, in a multi-pass vacuum. Here is a patch to fix this issue. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com user_correct_memorycontext_gist_stat.patch Description: Binary data

Re: [HACKERS] Block level parallel vacuum

2019-10-15 Thread Dilip Kumar
t I suggest "amestimatestat" or "amstatsize" and for copy stat data we can add "amcopystat"? It would be helpful to extend the parallel vacuum for the indexes which has extended stats. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2019-10-21 Thread Dilip Kumar
On Tue, Oct 22, 2019 at 10:46 AM Amit Kapila wrote: > > On Thu, Oct 3, 2019 at 1:18 PM Dilip Kumar wrote: > > > > I have attempted to test the performance of (Stream + Spill) vs > > (Stream + BGW pool) and I can see the similar gain what Alexey had > > shown[1].

Re: pgbench - refactor init functions with buffers

2019-10-22 Thread Dilip Kumar
convention. Is there any specific reason to do this? -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: pgbench - refactor init functions with buffers

2019-10-22 Thread Dilip Kumar
oreover here > I'm actually doing a refactoring patch, so it seems ok to include that. > I see. I was under impression that we don't use this style in PG. But, since we are already using this style other places so no objection from my side for this particular point. Sorry for the noise. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: Questions/Observations related to Gist vacuum

2019-10-22 Thread Dilip Kumar
On Tue, Oct 22, 2019 at 10:53 AM Amit Kapila wrote: > > On Tue, Oct 22, 2019 at 10:50 AM Dilip Kumar wrote: > > > > On Tue, Oct 22, 2019 at 9:10 AM Amit Kapila wrote: > > > > > > On Fri, Oct 18, 2019 at 4:51 PM Dilip Kumar wrote: > > > > >

Re: Questions/Observations related to Gist vacuum

2019-10-21 Thread Dilip Kumar
On Tue, Oct 22, 2019 at 9:10 AM Amit Kapila wrote: > > On Fri, Oct 18, 2019 at 4:51 PM Dilip Kumar wrote: > > > > I have prepared a first version of the patch. Currently, I am > > performing an empty page deletion for all the cases. > > > > Few co

Re: [HACKERS] Block level parallel vacuum

2019-10-24 Thread Dilip Kumar
On Fri, Oct 18, 2019 at 12:18 PM Dilip Kumar wrote: > > On Fri, Oct 18, 2019 at 11:25 AM Amit Kapila wrote: > > > > On Fri, Oct 18, 2019 at 8:45 AM Dilip Kumar wrote: > > > > > > On Thu, Oct 17, 2019 at 4:00 PM Amit Kapila > > > wrote: > >

Re: Fix of fake unlogged LSN initialization

2019-10-24 Thread Dilip Kumar
NED) XLogCtl->unloggedLSN = ControlFile->unloggedLSN; else XLogCtl->unloggedLSN = 1; -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

<    1   2   3   4   5   6   7   8   9   10   >