Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

2017-03-22 Thread Amit Kapila
ted the table, we have exactly typo. /inserted the table/inserted in the table 14. + lp [1] [2] + [, ]->[111, ] Here, after the update, the first column should be . -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Supporting huge pages on Windows

2017-03-22 Thread Amit Kapila
xed. As Ashutosh referred, I added a very >> simple suggestion to use Windows Group Policy tool. > > > Amit, Magnus, you are signed up as reviewers for this patch. Do you know > when you'll have a chance to take a look? > I have provided my feedback and I could not test it on

Re: [HACKERS] pageinspect and hash indexes

2017-03-22 Thread Amit Kapila
, + errmsg("index table contains empty page"))); Do we want to give a separate message for EMPTY and NEW pages? Isn't it better that the same error message can be given for both of them as from user perspective there is not much difference between both the messages? -- With

Re: [HACKERS] segfault in hot standby for hash indexes

2017-03-22 Thread Amit Kapila
On Wed, Mar 22, 2017 at 3:39 PM, Ashutosh Sharma wrote: > Hi, > > On Wed, Mar 22, 2017 at 8:41 AM, Amit Kapila wrote: > > To fix this, I think we should pass 'REGBUF_KEEP_DATA' while > registering the buffer. Something like this, > > -

Re: [HACKERS] segfault in hot standby for hash indexes

2017-03-22 Thread Amit Kapila
On Thu, Mar 23, 2017 at 8:43 AM, Amit Kapila wrote: > On Wed, Mar 22, 2017 at 3:39 PM, Ashutosh Sharma > wrote: >> Hi, >> >> On Wed, Mar 22, 2017 at 8:41 AM, Amit Kapila wrote: >> >> To fix this, I think we should pass 'REGBUF_KEEP_DATA' while

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

2017-03-22 Thread Amit Kapila
On Wed, Mar 22, 2017 at 4:06 PM, Amit Kapila wrote: > On Tue, Mar 21, 2017 at 6:47 PM, Pavan Deolasee > wrote: >> >>> >> >> Please find attached rebased patches. >> > > Few comments on 0005_warm_updates_v18.patch: > Few more comments on 0

Re: [HACKERS] pageinspect and hash indexes

2017-03-22 Thread Amit Kapila
n be given for both of them as >> from user perspective there is not much difference between both the >> messages? > > I think we should show separate message because they are two different > type of pages. In the sense like, one is initialised whereas other is > complete

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

2017-03-23 Thread Amit Kapila
() check in master code as well and then you will most probably again see the same regression. Also, I think if we test at fillfactor 80 or 75 (which is not unrealistic considering an update-intensive workload), then we might again see regression. -- With Regards, Amit Kapila. EnterpriseDB: ht

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

2017-03-23 Thread Amit Kapila
On Thu, Mar 23, 2017 at 3:44 PM, Pavan Deolasee wrote: > On Wed, Mar 22, 2017 at 4:06 PM, Amit Kapila > wrote: >> > >> 3. >> + /* >> + * HASH indexes compute a hash value of the key and store that in the >> + * index. So >> we must first obtain the ha

Re: [HACKERS] pageinspect and hash indexes

2017-03-23 Thread Amit Kapila
HASHO_PAGE_ID) ereport(ERROR, spurious white space. > > Also, I have attached > '0001-Mark-freed-overflow-page-as-UNUSED-pagetype-v2.patch' that > handles your comment mentioned in [1]. > In general, we have to initialize prevblock with max_bucket, but here i

Re: [HACKERS] segfault in hot standby for hash indexes

2017-03-23 Thread Amit Kapila
On Thu, Mar 23, 2017 at 4:26 PM, Ashutosh Sharma wrote: > On Thu, Mar 23, 2017 at 9:17 AM, Amit Kapila wrote: >> >> On Thu, Mar 23, 2017 at 8:43 AM, Amit Kapila wrote: >> > >> > I think this will work, but not sure if there is a merit to deviate >> >

Re: [HACKERS] segfault in hot standby for hash indexes

2017-03-24 Thread Amit Kapila
ks, this version looks good to me. -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] pageinspect and hash indexes

2017-03-24 Thread Amit Kapila
ched wrong patch. Here are the correct > set of patches. > The latest version of patches looks fine to me. -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

2017-03-24 Thread Amit Kapila
On Fri, Mar 24, 2017 at 12:25 AM, Pavan Deolasee wrote: > > > On Thu, Mar 23, 2017 at 7:53 PM, Amit Kapila > wrote: >> >> > >> >> I am not sure on what basis user can set such parameters, it will be >> quite difficult to tune those parameters. I thi

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

2017-03-24 Thread Amit Kapila
On Thu, Mar 23, 2017 at 3:54 PM, Pavan Deolasee wrote: > Thanks Amit. v19 addresses some of the comments below. > > On Thu, Mar 23, 2017 at 10:28 AM, Amit Kapila > wrote: >> >> On Wed, Mar 22, 2017 at 4:06 PM, Amit Kapila >> wrote: >> > On Tue,

Re: [HACKERS] crashes due to setting max_parallel_workers=0

2017-03-24 Thread Amit Kapila
tion. I have added this to open items list. -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Problem in Parallel Bitmap Heap Scan?

2017-03-24 Thread Amit Kapila
with force_parallel_mode=regress and all the test are > passing. > Thomas, did you get chance to verify Dilip's latest patch? I have added this issue in PostgreSQL 10 Open Items list so that we don't loose track of this issue. -- With Regards, Amit Kapila. EnterpriseDB: http:

Re: [HACKERS] Add pgstathashindex() to get hash index table statistics.

2017-03-24 Thread Amit Kapila
On Thu, Mar 23, 2017 at 11:24 PM, Ashutosh Sharma wrote: > Hi, > > On Tue, Feb 7, 2017 at 9:23 AM, Robert Haas wrote: >> On Mon, Feb 6, 2017 at 10:40 PM, Amit Kapila wrote: >>>> Maybe we should call them "unused pages". >>> >>> +1. If we

Re: [HACKERS] comments in hash_alloc_buckets

2017-03-24 Thread Amit Kapila
ndex. I think saying ".. last page in hash index" sounds slightly awkward as this is the last page for current split point, how about just "Initialize the page. ..." -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-hacker

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

2017-03-25 Thread Amit Kapila
On Fri, Mar 24, 2017 at 11:49 PM, Pavan Deolasee wrote: > > > On Fri, Mar 24, 2017 at 6:46 PM, Amit Kapila > wrote: >> >> >> >> I was worried for the case if the index is created non-default >> collation, will the datumIsEqual() suffice the need. Now a

Re: [HACKERS] Add pgstathashindex() to get hash index table statistics.

2017-03-25 Thread Amit Kapila
On Sat, Mar 25, 2017 at 12:33 PM, Ashutosh Sharma wrote: > On Sat, Mar 25, 2017 at 11:02 AM, Amit Kapila wrote: >> On Thu, Mar 23, 2017 at 11:24 PM, Ashutosh Sharma >> wrote: >>> Hi, >>> >>> On Tue, Feb 7, 2017 at 9:23 AM, Robert Haas wrote: >>

Re: [HACKERS] crashes due to setting max_parallel_workers=0

2017-03-25 Thread Amit Kapila
t; My vote would be to leave the GUC behaviour as is and add some tests > to select_parallel.sql which exploit setting max_parallel_workers to 0 > for running some tests. > I think force_parallel_mode=regress should test the same thing. -- With Regards, Amit Kapila. EnterpriseDB: http://www.ent

Re: [HACKERS] [POC] A better way to expand hash indexes.

2017-03-25 Thread Amit Kapila
On Sat, Mar 25, 2017 at 10:13 AM, Mithun Cy wrote: > On Tue, Mar 21, 2017 at 8:16 AM, Amit Kapila wrote: >> Sure, I was telling you based on that. If you are implicitly treating >> it as 2-dimensional array, it might be easier to compute the array >>offsets. > >

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

2017-03-25 Thread Amit Kapila
On Sat, Mar 25, 2017 at 11:24 PM, Pavan Deolasee wrote: > > On Sat, 25 Mar 2017 at 11:03 PM, Peter Geoghegan wrote: >> >> On Sat, Mar 25, 2017 at 12:54 AM, Amit Kapila >> wrote: >> > I am not sure how do you want to binary compare two datums, if you are >>

Re: [HACKERS] [POC] A better way to expand hash indexes.

2017-03-26 Thread Amit Kapila
On Sun, Mar 26, 2017 at 11:26 AM, Mithun Cy wrote: > Thanks, Amit for the review. > On Sat, Mar 25, 2017 at 7:03 PM, Amit Kapila wrote: >> >> I think one-dimensional patch has fewer places to touch, so that looks >> better to me. However, I think there is still hard codi

Re: [HACKERS] [POC] A better way to expand hash indexes.

2017-03-27 Thread Amit Kapila
On Mon, Mar 27, 2017 at 11:21 AM, Amit Kapila wrote: > On Sun, Mar 26, 2017 at 11:26 AM, Mithun Cy > wrote: >> Thanks, Amit for the review. >> On Sat, Mar 25, 2017 at 7:03 PM, Amit Kapila wrote: >>> >>> I think one-dimensional patch has fewer places to t

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

2017-03-27 Thread Amit Kapila
On Sat, Mar 25, 2017 at 1:24 PM, Amit Kapila wrote: > On Fri, Mar 24, 2017 at 11:49 PM, Pavan Deolasee > wrote: >> >> On Fri, Mar 24, 2017 at 6:46 PM, Amit Kapila >> wrote: >>> > >> While looking at this problem, it occurred to me that the assumptio

Re: [HACKERS] parallelize queries containing initplans

2017-03-27 Thread Amit Kapila
On Thu, Mar 16, 2017 at 2:34 AM, Kuntal Ghosh wrote: > On Tue, Mar 14, 2017 at 3:20 PM, Amit Kapila wrote: >> Based on that idea, I have modified the patch such that it will >> compute the set of initplans Params that are required below gather >> node and store them as bitm

Re: [HACKERS] [POC] A better way to expand hash indexes.

2017-03-27 Thread Amit Kapila
On Tue, Mar 28, 2017 at 10:43 AM, Mithun Cy wrote: > On Mon, Mar 27, 2017 at 11:21 AM, Amit Kapila wrote: > > > As you have said we can solve it if we allocate buckets always in > power-of-2 when we do hash index meta page init. But on other > occasions, when we try to d

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

2017-03-28 Thread Amit Kapila
On Mon, Mar 27, 2017 at 2:19 PM, Pavan Deolasee wrote: > On Fri, Mar 24, 2017 at 11:49 PM, Pavan Deolasee > wrote: >> >> >> >> On Fri, Mar 24, 2017 at 6:46 PM, Amit Kapila >> wrote: >>> >>> >>> >>> I was worried

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

2017-03-28 Thread Amit Kapila
On Mon, Mar 27, 2017 at 2:19 PM, Pavan Deolasee wrote: > > > On Fri, Mar 24, 2017 at 11:49 PM, Pavan Deolasee > wrote: >> >> >> >> On Fri, Mar 24, 2017 at 6:46 PM, Amit Kapila >> wrote: >>> >>> >>> >>> I was wo

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

2017-03-28 Thread Amit Kapila
On Tue, Mar 28, 2017 at 4:05 PM, Amit Kapila wrote: > On Mon, Mar 27, 2017 at 2:19 PM, Pavan Deolasee > wrote: >> On Fri, Mar 24, 2017 at 11:49 PM, Pavan Deolasee >> wrote: >>> >>> >>> >>> On Fri, Mar 24, 2017 at 6:46 PM, Amit Kapila >&g

Re: [HACKERS] [POC] A better way to expand hash indexes.

2017-03-28 Thread Amit Kapila
LY_ONE_PHASE)) & \ + SPLITPOINT_PHASE_MASK) This won't work if we change SPLITPOINT_GROUPS_WITH_ONLY_ONE_PHASE to number other than 3. I think you should change it so that it can work with any value of SPLITPOINT_GROUPS_WITH_ONLY_ONE_PHASE. I think you should name this define as SP

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

2017-03-28 Thread Amit Kapila
On Tue, Mar 28, 2017 at 10:35 PM, Pavan Deolasee wrote: > > On Tue, Mar 28, 2017 at 7:04 PM, Amit Kapila > wrote: >> >> >> >> For such an heap insert, we will pass >> the actual value of column to index_form_tuple during index insert. >> However

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

2017-03-28 Thread Amit Kapila
On Wed, Mar 29, 2017 at 11:52 AM, Amit Kapila wrote: > On Tue, Mar 28, 2017 at 10:35 PM, Pavan Deolasee > wrote: >> >> On Tue, Mar 28, 2017 at 7:04 PM, Amit Kapila >> wrote: >>> >>> For such an heap insert, we will pass >>> the actual valu

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

2017-03-28 Thread Amit Kapila
On Tue, Mar 28, 2017 at 10:31 PM, Pavan Deolasee wrote: > > On Tue, Mar 28, 2017 at 4:05 PM, Amit Kapila > wrote: >> >> As asked previously, can you explain me on what basis are you >> considering it robust? The comments on top of datumIsEqual() clearly >> ind

Re: [HACKERS] [POC] A better way to expand hash indexes.

2017-03-29 Thread Amit Kapila
On Wed, Mar 29, 2017 at 12:51 PM, Mithun Cy wrote: > On Wed, Mar 29, 2017 at 10:12 AM, Amit Kapila wrote: >> Few other comments: >> +/* >> + * This is just a trick to save a division operation. If you look into the >> + * bitmap of 0-based bucket_num 2nd and 3r

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

2017-03-29 Thread Amit Kapila
On Wed, Mar 29, 2017 at 1:10 PM, Pavan Deolasee wrote: > > On Wed, Mar 29, 2017 at 12:02 PM, Amit Kapila > wrote: >> >> On Wed, Mar 29, 2017 at 11:52 AM, Amit Kapila >> wrote: > > Then during recheck, we pass already compressed values to > index_form_tuple().

Re: [HACKERS] inconsistent page found on STANDBY server

2017-03-30 Thread Amit Kapila
g registered and > therefore, data (xl_hash_update_meta_page) is not completely recorded > into the wal record. > > Fix: > === > Attached patch fixes this issue. > The fix looks good to me. I have scanned the hash index code to see if there is any other similar problem, but

Re: [HACKERS] Supporting huge pages on Windows

2017-03-30 Thread Amit Kapila
similar to what we display in sysv_shmem.c as below: elog(DEBUG1, "mmap(%zu) with MAP_HUGETLB failed, huge pages disabled: %m", allocsize); -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

2017-03-30 Thread Amit Kapila
On Thu, Mar 30, 2017 at 4:07 PM, Pavan Deolasee wrote: > > On Wed, Mar 29, 2017 at 4:42 PM, Amit Kapila > wrote: >> >> On Wed, Mar 29, 2017 at 1:10 PM, Pavan Deolasee >> wrote: >> > >> > On Wed, Mar 29, 2017 at 12:02 PM, Amit Kapila >> &g

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

2017-03-30 Thread Amit Kapila
On Thu, Mar 30, 2017 at 5:55 PM, Pavan Deolasee wrote: > > On Thu, Mar 30, 2017 at 5:27 PM, Amit Kapila > wrote: >> >> >> >> How have you verified that? Have you checked that in >> heap_prepare_insert it has called toast_insert_or_update() and then >

Re: [HACKERS] Supporting huge pages on Windows

2017-03-31 Thread Amit Kapila
On Fri, Mar 31, 2017 at 8:05 AM, Tsunakawa, Takayuki wrote: > From: Amit Kapila [mailto:amit.kapil...@gmail.com] >> The latest patch looks good to me apart from one Debug message, so I have >> marked it as Ready For Committer. > > Thank you so much! > > >&g

Re: [HACKERS] TPC-H Q20 from 1 hour to 19 hours!

2017-03-31 Thread Amit Kapila
x27;make > check'. > Your patch looks good to me. I have verified some join cases as well where the behaviour is sane after patch. I have also done testing with force_parallel_mode=regress (ran make check-world) and everything seems good. -- With Regards, Amit Kapila. EnterpriseDB:

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

2017-03-31 Thread Amit Kapila
Do you think it make sense to have a separate thread to discuss and get feedback on same as I am not seeing much input on the knobs you are proposing to handle second pass over index? -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Page Scan Mode in Hash Index

2017-04-01 Thread Amit Kapila
.. + next_buf = _hash_getbuf_with_strategy(rel, blkno, HASH_WRITE, + LH_OVERFLOW_PAGE, + bstrategy); + After this change, you need to modify comments on top of function hashbucketcleanup() and _hash_squeezebucket(). -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com -

Re: [HACKERS] [sqlsmith] Failed assertion in _hash_kill_items/MarkBufferDirtyHint

2017-04-01 Thread Amit Kapila
that patch, then it is good to proceed with this fix. [1] - https://commitfest.postgresql.org/13/999/ -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Page Scan Mode in Hash Index

2017-04-04 Thread Amit Kapila
so->hashso_split_bucket_buf) + { + so->currPos.prevPage = InvalidBlockNumber; + LockBuffer(so->currPos.buf, BUFFER_LOCK_UNLOCK); + } + else + { + so->currPos.prevPage = (opaque)->hasho_prevblkno; + _hash_relbuf(rel, so->currPos.buf); + } + + so->currPos.nextPage = (opaque)->ha

Re: [HACKERS] Page Scan Mode in Hash Index

2017-04-04 Thread Amit Kapila
f. There won't be any case where we will be having pin > on overflow buf at the end of scan. > What if mark the buffer as invalid after releasing the pin? We are already doing it that way in btree code, refer _bt_drop_lock_and_maybe_pin(). I think if we do that way, then we can do what Ro

Re: [HACKERS] increasing the default WAL segment size

2017-04-04 Thread Amit Kapila
t part to allow internal representation change (only). > > No commitment yet to increasing wal-segsize in the way this patch has it. > What part of patch you don't like and do you have any suggestions to improve the same? -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterpr

Re: [HACKERS] strange parallel query behavior after OOM crashes

2017-04-05 Thread Amit Kapila
gt; some reason? > AFAICU, during crash recovery, we wait for all non-syslogger children > to exit, then reset shmem(call BackgroundWorkerShmemInit) and perform > StartupDataBase. While starting the startup process we check if any > bgworker is scheduled for a restart. > In general, your theory appears right, but can you check how it behaves in standby server because there is a difference in how the startup process behaves during master and standby startup? In master, it stops after recovery whereas in standby it will keep on running to receive WAL. -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] [sqlsmith] ERROR: badly formatted node string "RESTRICTINFO...

2017-04-11 Thread Amit Kapila
rker will never execute such a plan as we don't generate a plan where unsafe sublan/initplan is referenced in the node passed to the worker. If we want to avoid passing parallel-unsafe subplans to workers, then I think we can maintain a list of parallel safe subplans along with subplans in P

Re: [HACKERS] [sqlsmith] ERROR: badly formatted node string "RESTRICTINFO...

2017-04-11 Thread Amit Kapila
On Wed, Apr 12, 2017 at 12:40 AM, Tom Lane wrote: > Amit Kapila writes: > >> I think there is a possibility of doing ExecInitNode in a parallel >> worker for a parallel-unsafe subplan, because we pass a list of all >> the sublans stored in planned statement. > > It&

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

2017-04-11 Thread Amit Kapila
Amit brought up > where only one of the indexes receive WARM inserts. > 4. Added code to kill wrong index pointers to do online cleanup. > 5. Added code to set a CLEAR pointer to a WARM pointer when we know that the > entire chain is WARM. This should address the workload Dilip ran and

Re: [HACKERS] [sqlsmith] ERROR: badly formatted node string "RESTRICTINFO...

2017-04-12 Thread Amit Kapila
On Wed, Apr 12, 2017 at 12:40 AM, Tom Lane wrote: > Amit Kapila writes: > >> However, the worker will >> never execute such a plan as we don't generate a plan where unsafe >> sublan/initplan is referenced in the node passed to the worker. If we >> want to avo

Re: [HACKERS] TAP tests take a long time

2017-04-12 Thread Amit Kapila
eed it for the sake of testing that operation. Mithun, as you are the original author of these tests, can you please try some of the above optimizations and any others you can think of and see if we can reduce the time for hash index tests? -- With Regards, Amit Kapila. EnterpriseDB: http://www.

Re: [HACKERS] Inadequate parallel-safety check for SubPlans

2017-04-12 Thread Amit Kapila
I'm knocking off for the day a bit early today, but I'll have > a look at it tomorrow, unless anyone in the Far East beats me to it. > Thanks for looking into it. -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] [sqlsmith] ERROR: badly formatted node string "RESTRICTINFO...

2017-04-12 Thread Amit Kapila
On Wed, Apr 12, 2017 at 10:30 PM, Tom Lane wrote: > Amit Kapila writes: >> On Wed, Apr 12, 2017 at 12:40 AM, Tom Lane wrote: >>> Anyone want to draft a patch for this? > >> Please find patch attached based on above discussion. > > This patch seems fairly incomp

[HACKERS] Typo in htup_details.h

2017-04-15 Thread Amit Kapila
Attached patch to fix $SUBJECT. -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com typo_htup_details.patch Description: Binary data -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org

Re: [HACKERS] Inadequate parallel-safety check for SubPlans

2017-04-16 Thread Amit Kapila
whitelist of params for parallel safety, but if you think that is the better way to go, I can give it a try. -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Inadequate parallel-safety check for SubPlans

2017-04-18 Thread Amit Kapila
On Mon, Apr 17, 2017 at 10:54 AM, Tom Lane wrote: > Amit Kapila writes: >> On Sun, Apr 16, 2017 at 9:30 PM, Tom Lane wrote: >>> FYI, I have this on my to-look-at list, and expect to fix it before Robert >>> returns from vacation. > >> Let me know if an

Re: [HACKERS] Inadequate parallel-safety check for SubPlans

2017-04-18 Thread Amit Kapila
On Wed, Apr 19, 2017 at 1:27 AM, Tom Lane wrote: > Amit Kapila writes: >> I have ended up doing something along the lines suggested by you (or >> at least what I have understood from your e-mail). Basically, pass >> the safe-param-ids list to parallel safety function and de

Re: [HACKERS] OK, so culicidae is *still* broken

2017-04-18 Thread Amit Kapila
ct we'd need some retry > logic anyway for that case. > Yeah, that kind of thing can work assuming we don't get conflicts too often, but it could be possible that conflicts are not reported from ASLR enabled environments because of commit 7f3e17b4. [1] - https://blogs.msdn.

Re: [HACKERS] dtrace probes

2017-04-20 Thread Amit Kapila
n PG-10 which has caused this behavior? If not, then I don't think it should be added to open items of PG-10. -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Getting server crash on Windows when using ICU collation

2017-06-16 Thread Amit Kapila
ucol_strcollUTF8 to compare the same, however, with patch, it will always ucol_strcoll as we never define HAVE_UCOL_STRCOLLUTF8 flag on Windows. We have some multi-byte tests in src/test/mb directory, see if we can use those to verify these changes. I admit that I have not tried to execute those on

Re: [HACKERS] Broken hint bits (freeze)

2017-06-16 Thread Amit Kapila
ither it is mentioned in comments why we have done it that way. -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Broken hint bits (freeze)

2017-06-16 Thread Amit Kapila
On Fri, Jun 16, 2017 at 11:03 PM, Sergey Burladyan wrote: > Bruce Momjian writes: > >> On Fri, Jun 16, 2017 at 08:10:13PM +0530, Amit Kapila wrote: >> > On Fri, Jun 16, 2017 at 7:03 AM, Sergey Burladyan >> > wrote: >> > > Bruce Momjian writes: >>

Re: [HACKERS] Getting server crash on Windows when using ICU collation

2017-06-16 Thread Amit Kapila
On Fri, Jun 16, 2017 at 7:42 PM, Peter Eisentraut wrote: > On 6/16/17 06:30, Amit Kapila wrote: >> How will this compare UTF-8 strings in UTF-8 encoding? It seems to me >> that ideally, it should use ucol_strcollUTF8 to compare the same, >> however, with patch, it will alwa

Re: [HACKERS] Getting server crash on Windows when using ICU collation

2017-06-16 Thread Amit Kapila
On Sat, Jun 17, 2017 at 6:57 AM, Peter Eisentraut wrote: > On 6/16/17 10:12, Peter Eisentraut wrote: >> On 6/16/17 06:30, Amit Kapila wrote: >>> How will this compare UTF-8 strings in UTF-8 encoding? It seems to me >>> that ideally, it should use ucol_strcollUTF8 to co

Re: [HACKERS] Getting server crash on Windows when using ICU collation

2017-06-18 Thread Amit Kapila
On Sat, Jun 17, 2017 at 8:27 PM, Ashutosh Sharma wrote: > Hi, > > On Sat, Jun 17, 2017 at 6:38 PM, Peter Eisentraut > wrote: >> On 6/16/17 23:46, Amit Kapila wrote: >>> I have just posted one way >>> to determine if icu library has support for ucol_strcollUTF

Re: [HACKERS] BUG: pg_dump generates corrupted gzip file in Windows

2017-06-18 Thread Amit Kapila
you intended to attach the patch to this e-mail or are you referring to Kuntal's patch up thread? If later, then it is better to mention the link of mail which has a patch that you have verified. -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Setting pd_lower in GIN metapage

2017-06-19 Thread Amit Kapila
n't hold in the case of GIN metapage, so any GIN indexes contain > corrupted metapage after recovery (metadata overwritten with zeros). > Why can't you do a special check for metapage identification? It should be easy to add such a check. I have seen one of such tools (pg_filedump) ha

Re: [HACKERS] Broken hint bits (freeze)

2017-06-20 Thread Amit Kapila
file >> > from the new master to the standby, right, and I think pg_controldata >> > too, so it should be fine. Have you tested to see if it fails? > > It need old WAL files from old version for correct restore heap > files. New WAL files from new version does not have this information. > So in such a case can we run rsync once before pg_upgrade? -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Setting pd_lower in GIN metapage

2017-06-20 Thread Amit Kapila
On Tue, Jun 20, 2017 at 1:50 PM, Amit Langote wrote: > On 2017/06/19 23:31, Tom Lane wrote: >> Amit Kapila writes: >>> On Mon, Jun 19, 2017 at 11:37 AM, Amit Langote >>> wrote: >>>> What are some arguments against setting pd_lower in the GIN metapage as &g

Re: [HACKERS] Getting server crash on Windows when using ICU collation

2017-06-20 Thread Amit Kapila
where we may have > multiple ICU versions installed on our system and there might be a > possibility that the uconv command may get executed from the ICU > version that we are not trying to link with postgres. > To avoid that why can't we use the same icu path for executing uconv

Re: [HACKERS] Broken hint bits (freeze)

2017-06-20 Thread Amit Kapila
On Tue, Jun 20, 2017 at 7:05 PM, Sergey Burladyan wrote: > Amit Kapila writes: > >> On Tue, Jun 20, 2017 at 3:40 PM, Sergey Burladyan >> wrote: > I use pg 9.2 and "skipping restartpoint, already performed at" is from > src/backend/access/transam/xlog.c:864

Re: [HACKERS] Getting server crash on Windows when using ICU collation

2017-06-20 Thread Amit Kapila
On Wed, Jun 21, 2017 at 12:15 AM, Peter Eisentraut wrote: > On 6/20/17 09:23, Amit Kapila wrote: >> To avoid that why can't we use the same icu path for executing uconv >> as we are using for linking? > > Yeah, you'd need to use prefix $self->{options}->{ic

Re: [HACKERS] Broken hint bits (freeze)

2017-06-21 Thread Amit Kapila
On Tue, Jun 20, 2017 at 7:24 PM, Amit Kapila wrote: > On Tue, Jun 20, 2017 at 7:05 PM, Sergey Burladyan wrote: >> Amit Kapila writes: >> >>> On Tue, Jun 20, 2017 at 3:40 PM, Sergey Burladyan >>> wrote: >> I use pg 9.2 and "skipping restartpoint,

Re: [HACKERS] Broken hint bits (freeze)

2017-06-22 Thread Amit Kapila
On Wed, Jun 21, 2017 at 10:03 PM, Bruce Momjian wrote: > On Wed, Jun 21, 2017 at 07:49:21PM +0530, Amit Kapila wrote: >> On Tue, Jun 20, 2017 at 7:24 PM, Amit Kapila wrote: >> > Hmm. I think we need something that works with lesser effort because >> > not all users

Re: [HACKERS] intermittent failures in Cygwin from select_parallel tests

2017-06-22 Thread Amit Kapila
aybe we can try to print dsm_segment handle value both before launching the worker and in parallel worker (ParallelWorkerMain). That might help us in identifying the reason of error "could not map dynamic shared memory segment". -- With Regards, Amit Kapila. EnterpriseDB: http://www.ent

Re: [HACKERS] An attempt to reduce WALWriteLock contention

2017-06-23 Thread Amit Kapila
calls out of walwritelock. Can you share patch or some details about how you have done it and how have you measured the contention you are seeing? -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) T

Re: [HACKERS] Broken hint bits (freeze)

2017-06-23 Thread Amit Kapila
On Fri, Jun 23, 2017 at 8:47 PM, Sergey Burladyan wrote: > Bruce Momjian writes: > >> On Wed, Jun 21, 2017 at 07:49:21PM +0530, Amit Kapila wrote: >> > On Tue, Jun 20, 2017 at 7:24 PM, Amit Kapila >> > wrote: >> > > Hmm. I think we need somet

Re: [HACKERS] Broken hint bits (freeze)

2017-06-23 Thread Amit Kapila
On Fri, Jun 23, 2017 at 8:18 PM, Bruce Momjian wrote: > On Fri, Jun 23, 2017 at 08:10:17AM +0530, Amit Kapila wrote: >> On Wed, Jun 21, 2017 at 10:03 PM, Bruce Momjian wrote: >> > On Wed, Jun 21, 2017 at 07:49:21PM +0530, Amit Kapila wrote: >> >> On Tue, Jun 20

Re: [HACKERS] intermittent failures in Cygwin from select_parallel tests

2017-06-26 Thread Amit Kapila
he log contents or is there something else? The attached log looks strange to me in the sense that the first worker that gets invoked is Parallel worker number 2 and it finds that somebody else has already set the sender handle. -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.

Re: [HACKERS] intermittent failures in Cygwin from select_parallel tests

2017-06-26 Thread Amit Kapila
On Mon, Jun 26, 2017 at 8:09 PM, Andrew Dunstan wrote: > > > On 06/26/2017 10:36 AM, Amit Kapila wrote: >> On Fri, Jun 23, 2017 at 9:12 AM, Andrew Dunstan >> wrote: >>> >>> On 06/22/2017 10:24 AM, Tom Lane wrote: >>>> Andrew Dunstan writes: &g

Re: [HACKERS] Pluggable storage

2017-06-27 Thread Amit Kapila
then you do need to worry about places wherever we are using HeapTuple like TupleTableSlots, Visibility routines, etc. (just think if somebody changes tuple header, then all such places are susceptible to change). Similarly, if the page format is changed you need to consider all page scan API's

Re: [HACKERS] Pluggable storage

2017-06-27 Thread Amit Kapila
t vacuum with its parameters can be used so easily. One thing that might need some thoughts is that is it sufficient to say that keep autovacuum as off and call some different API for places where the vacuum can be invoked manually like Vacuum command to the developer implementing some different strategy for vacuum or we need something more as well. -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Pluggable storage

2017-06-28 Thread Amit Kapila
On Tue, Jun 27, 2017 at 7:23 PM, Alexander Korotkov wrote: > On Tue, Jun 27, 2017 at 4:08 PM, Amit Kapila > wrote: >> >> Similarly, >> if the page format is changed you need to consider all page scan API's >> like heapgettup_pagemode/heapgetpage. &g

Re: [HACKERS] Pluggable storage

2017-06-28 Thread Amit Kapila
On Wed, Jun 28, 2017 at 7:43 AM, Haribabu Kommi wrote: > > On Wed, Jun 28, 2017 at 12:00 AM, Alexander Korotkov > wrote: >> >> On Tue, Jun 27, 2017 at 4:19 PM, Amit Kapila >> wrote: >>> >>> On Thu, Jun 22, 2017 at 5:46 PM, Alexander Korotkov >&

Re: [HACKERS] Request more documentation for incompatibility of parallelism and plpgsql exec_run_select

2017-06-29 Thread Amit Kapila
n be easier for users to understand. [1] - https://www.postgresql.org/docs/devel/static/when-can-parallel-query-be-used.html -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Broken hint bits (freeze)

2017-06-30 Thread Amit Kapila
s doesn't fail on PG 10 since we WAL-log hash indexes. > > I think we have two questions: > > 1. do we fix this in the server If we want to fix this in the server then we need to log (write WAL) the init fork for hash indexes. > 2. if not, do we fix pg_upgrade > I think

[HACKERS] hash index on unlogged tables doesn't behave as expected

2017-07-02 Thread Amit Kapila
bitmappage) which make it difficult to follow that approach. I think this should be considered a PostgreSQL 10 open item. [1] - https://www.postgresql.org/message-id/20170630005634.GA4448%40momjian.us -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com fix_unlogged_hash_index_i

Re: [HACKERS] Broken hint bits (freeze)

2017-07-02 Thread Amit Kapila
On Thu, Jun 29, 2017 at 6:57 AM, Bruce Momjian wrote: > On Sat, Jun 24, 2017 at 09:24:21AM +0530, Amit Kapila wrote: >> > I was not clear. I was not saying there can be only one extra WAL file. >> > I am saying the "Latest checkpoint location" should be one WAL

Re: [HACKERS] Proposal : For Auto-Prewarm.

2017-07-02 Thread Amit Kapila
e only a server log for the same. > I think that > is okay as the objective was to get one worker up and running. > You are right that the objective will be met, but still, I feel the behavior of this API will be unpredictable which in my opinion should be fixed. If it is really not pos

Re: [HACKERS] Proposal : For Auto-Prewarm.

2017-07-03 Thread Amit Kapila
rrno; + + FreeFile(file); I think you don't need to close the file in case of error, it will be automatically taken care in case of error (via transaction abort path). 9. + /* Register autoprewarm load. */ + setup_autoprewarm(&autoprewarm_worker, "autoprewarm", "autoprewarm_main", +

Re: [HACKERS] Proposal : For Auto-Prewarm.

2017-07-03 Thread Amit Kapila
ocateFile(). If the number of blocks to be dumped is large, then the file can remain open for the significant period of time. -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] hash index on unlogged tables doesn't behave as expected

2017-07-03 Thread Amit Kapila
On Mon, Jul 3, 2017 at 3:24 PM, Ashutosh Sharma wrote: > On Mon, Jul 3, 2017 at 9:12 AM, Amit Kapila wrote: >> The basic issue was that the WAL logging for Create Index operation >> was oblivion of the fact that for unlogged tables only INIT forks need >> to be logged. A

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

2017-07-03 Thread Amit Kapila
k waits in this patch as is done by Robert in commit d4116a771925379c33cf4c6634ca620ed08b551d for ProcArrayGroupUpdate. -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com group_update_clog_v12.patch Description: Binary data -- Sent via pgsql-hackers mailing list (pgsql-hack

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

2017-07-03 Thread Amit Kapila
On Mon, Jul 3, 2017 at 6:15 PM, Amit Kapila wrote: > On Thu, Mar 23, 2017 at 1:18 PM, Ashutosh Sharma > wrote: >> >> Conclusion: >> As seen from the test results mentioned above, there is some performance >> improvement with 3 SP(s), with 5 SP(s) the results with pa

Re: [HACKERS] Request more documentation for incompatibility of parallelism and plpgsql exec_run_select

2017-07-03 Thread Amit Kapila
On Mon, Jul 3, 2017 at 8:57 PM, Simon Riggs wrote: > On 30 June 2017 at 05:14, Amit Kapila wrote: > >> This is explained in section 15.2 [1], refer below para: >> "The query might be suspended during execution. In any situation in >> which the system thinks that par

<    3   4   5   6   7   8   9   10   11   12   >