Re: [HACKERS] VACUUM and ANALYZE disagreeing on what reltuples means

Haribabu Kommi writes: > The changes are fine and now it reports proper live tuples in both > pg_class and stats. The other issue of continuous vacuum operation > leading to decrease of number of live tuples is not related to this > patch and that can be handled

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

On Sun, Oct 1, 2017 at 8:36 PM, Daniel Gustafsson wrote: >> On 18 Aug 2017, at 13:39, Claudio Freire wrote: >> >> On Fri, Apr 7, 2017 at 10:51 PM, Claudio Freire >> wrote: >>> Indeed they do, and that's what motivated this patch.

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

> On 18 Aug 2017, at 13:39, Claudio Freire wrote: > > On Fri, Apr 7, 2017 at 10:51 PM, Claudio Freire > wrote: >> Indeed they do, and that's what motivated this patch. But I'd need >> TB-sized tables to set up something like that. I don't have

Re: [HACKERS] VACUUM and ANALYZE disagreeing on what reltuples means

On Mon, Sep 25, 2017 at 4:39 AM, Tomas Vondra wrote: > > > On 09/06/2017 09:45 AM, Haribabu Kommi wrote: > > > > > > On Tue, Jul 25, 2017 at 9:33 PM, Tomas Vondra > > > > wrote: > > > > On

Re: [HACKERS] VACUUM and ANALYZE disagreeing on what reltuples means

Hi, Apologies, I forgot to respond to the second part of your message. On 09/06/2017 09:45 AM, Haribabu Kommi wrote: > > While testing this patch, I found another problem that is not related to > this patch. When the vacuum command is executed mutiple times on > a table with no dead rows, the

Re: [HACKERS] VACUUM and ANALYZE disagreeing on what reltuples means

On 09/06/2017 09:45 AM, Haribabu Kommi wrote: > > > On Tue, Jul 25, 2017 at 9:33 PM, Tomas Vondra > > wrote: > > On 7/25/17 12:55 AM, Tom Lane wrote: > > Tomas Vondra

Re: [HACKERS] VACUUM and ANALYZE disagreeing on what reltuples means

> On 06 Sep 2017, at 09:45, Haribabu Kommi wrote: > > On Tue, Jul 25, 2017 at 9:33 PM, Tomas Vondra > wrote: > On 7/25/17 12:55 AM, Tom Lane wrote: > Tomas Vondra

Re: [HACKERS] VACUUM and ANALYZE disagreeing on what reltuples means

On Tue, Jul 25, 2017 at 9:33 PM, Tomas Vondra wrote: > On 7/25/17 12:55 AM, Tom Lane wrote: > >> Tomas Vondra writes: >> >>> It seems to me that VACUUM and ANALYZE somewhat disagree on what >>> exactly reltuples means. VACUUM seems to

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

On Fri, Apr 7, 2017 at 10:51 PM, Claudio Freire wrote: > Indeed they do, and that's what motivated this patch. But I'd need > TB-sized tables to set up something like that. I don't have the > hardware or time available to do that (vacuum on bloated TB-sized > tables can

Re: [HACKERS] VACUUM and ANALYZE disagreeing on what reltuples means

On Tue, Jul 25, 2017 at 07:02:28PM +0200, Tomas Vondra wrote: > On 7/25/17 5:04 PM, Tom Lane wrote: > >Tomas Vondra writes: > >>Attached is a patch that (I think) does just that. The disagreement > >>was caused by VACUUM treating recently dead tuples as live, while >

Re: [HACKERS] VACUUM and ANALYZE disagreeing on what reltuples means

On 7/25/17 5:04 PM, Tom Lane wrote: Tomas Vondra writes: On 7/25/17 12:55 AM, Tom Lane wrote: I think the planner basically assumes that reltuples is the live tuple count, so maybe we'd better change VACUUM to get in step. Attached is a patch that (I think)

Re: [HACKERS] VACUUM and ANALYZE disagreeing on what reltuples means

Tomas Vondra writes: > On 7/25/17 12:55 AM, Tom Lane wrote: >> I think the planner basically assumes that reltuples is the live >> tuple count, so maybe we'd better change VACUUM to get in step. > Attached is a patch that (I think) does just that. The disagreement

Re: [HACKERS] VACUUM and ANALYZE disagreeing on what reltuples means

On 7/25/17 12:55 AM, Tom Lane wrote: Tomas Vondra writes: It seems to me that VACUUM and ANALYZE somewhat disagree on what exactly reltuples means. VACUUM seems to be thinking that reltuples = live + dead while ANALYZE apparently believes that reltuples = live

Re: [HACKERS] VACUUM and ANALYZE disagreeing on what reltuples means

Tomas Vondra writes: > It seems to me that VACUUM and ANALYZE somewhat disagree on what exactly > reltuples means. VACUUM seems to be thinking that > reltuples = live + dead > while ANALYZE apparently believes that > reltuples = live > The question is -

[HACKERS] VACUUM and ANALYZE disagreeing on what reltuples means

Hi, It seems to me that VACUUM and ANALYZE somewhat disagree on what exactly reltuples means. VACUUM seems to be thinking that reltuples = live + dead while ANALYZE apparently believes that reltuples = live This causes somewhat bizarre changes in the value, depending on which of

Re: Fwd: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

On Wed, Jul 12, 2017 at 1:29 PM, Claudio Freire wrote: > On Wed, Jul 12, 2017 at 1:08 PM, Claudio Freire > wrote: >> On Wed, Jul 12, 2017 at 11:48 AM, Alexey Chernyshov >> wrote: >>> Thank you for the patch and

Re: Fwd: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

On Wed, Jul 12, 2017 at 1:08 PM, Claudio Freire wrote: > On Wed, Jul 12, 2017 at 11:48 AM, Alexey Chernyshov > wrote: >> Thank you for the patch and benchmark results, I have a couple remarks. >> Firstly, padding in DeadTuplesSegment >> >>

Re: Fwd: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

On Wed, Jul 12, 2017 at 11:48 AM, Alexey Chernyshov wrote: > Thank you for the patch and benchmark results, I have a couple remarks. > Firstly, padding in DeadTuplesSegment > > typedef struct DeadTuplesSegment > > { > > ItemPointerData last_dead_tuple;/* Copy

Re: Fwd: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

Thank you for the patch and benchmark results, I have a couple remarks. Firstly, padding in DeadTuplesSegment typedef struct DeadTuplesSegment { ItemPointerData last_dead_tuple;/* Copy of the last dead tuple (unset * until the segment is

Fwd: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

Resending without the .tar.bz2 that get blocked Sorry for the delay, I had extended vacations that kept me away from my test rigs, and afterward testing too, liteally, a few weeks. I built a more thoroguh test script that produced some interesting results. Will attach the results. For now, to

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

On Fri, Apr 21, 2017 at 6:24 AM, Claudio Freire wrote: > On Wed, Apr 12, 2017 at 4:35 PM, Robert Haas wrote: >> On Tue, Apr 11, 2017 at 4:38 PM, Claudio Freire >> wrote: >>> In essence, the patch as it is proposed, doesn't

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

On Mon, Apr 24, 2017 at 3:57 PM, Claudio Freire wrote: > I wouldn't fret over the slight slowdown vs the old patch, it could be > noise (the script only completed a single run at scale 400). Yeah, seems fine. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

On Sun, Apr 23, 2017 at 12:41 PM, Robert Haas wrote: >> That's after inlining the compare on both the linear and sequential >> code, and it seems it lets the compiler optimize the binary search to >> the point where it outperforms the sequential search. >> >> That's not the

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

On Thu, Apr 20, 2017 at 5:24 PM, Claudio Freire wrote: >> What's not clear to me is how sensitive the performance of vacuum is >> to the number of cycles used here. For a large index, the number of >> searches will presumably be quite large, so it does seem worth >>

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

On Wed, Apr 12, 2017 at 4:35 PM, Robert Haas wrote: > On Tue, Apr 11, 2017 at 4:38 PM, Claudio Freire > wrote: >> In essence, the patch as it is proposed, doesn't *need* a binary >> search, because the segment list can only grow up to 15 segments

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

On Tue, Apr 11, 2017 at 4:38 PM, Claudio Freire wrote: > In essence, the patch as it is proposed, doesn't *need* a binary > search, because the segment list can only grow up to 15 segments at > its biggest, and that's a size small enough that linear search will >

Re: [HACKERS] Vacuum full stats reporting

On Mon, Apr 10, 2017 at 4:36 AM, Magnus Hagander wrote: > Right now, VACUUM FULL are not reported in pgstat. That seems bad:ish. I can > see two reasonable ways to proceed: > > 1. Start reporting VACUUM FULL as regular vacuums, so they count up > vacuum_count and last_vacuum

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

On Tue, Apr 11, 2017 at 4:17 PM, Robert Haas wrote: > On Tue, Apr 11, 2017 at 2:59 PM, Claudio Freire > wrote: >> On Tue, Apr 11, 2017 at 3:53 PM, Robert Haas wrote: >>> 1TB / 8kB per page * 60 tuples/page * 20% * 6

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

On Tue, Apr 11, 2017 at 2:59 PM, Claudio Freire wrote: > On Tue, Apr 11, 2017 at 3:53 PM, Robert Haas wrote: >> 1TB / 8kB per page * 60 tuples/page * 20% * 6 bytes/tuple = 9216MB of >> maintenance_work_mem >> >> So we'll allocate

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

On Tue, Apr 11, 2017 at 3:59 PM, Claudio Freire wrote: > On Tue, Apr 11, 2017 at 3:53 PM, Robert Haas wrote: >> 1TB / 8kB per page * 60 tuples/page * 20% * 6 bytes/tuple = 9216MB of >> maintenance_work_mem >> >> So we'll allocate

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

On Tue, Apr 11, 2017 at 3:53 PM, Robert Haas wrote: > 1TB / 8kB per page * 60 tuples/page * 20% * 6 bytes/tuple = 9216MB of > maintenance_work_mem > > So we'll allocate 128MB+256MB+512MB+1GB+2GB+4GB which won't be quite > enough so we'll allocate another 8GB, for a total of

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

On Fri, Apr 7, 2017 at 9:12 PM, Andres Freund wrote: >> Why do you say exponential growth fragments memory? AFAIK, all those >> allocations are well beyond the point where malloc starts mmaping >> memory, so each of those segments should be a mmap segment, >> independently

[HACKERS] Vacuum full stats reporting

Right now, VACUUM FULL are not reported in pgstat. That seems bad:ish. I can see two reasonable ways to proceed: 1. Start reporting VACUUM FULL as regular vacuums, so they count up vacuum_count and last_vacuum in pg_stat_*_tables. 2. Create a new set of counters for CLUSTER and VACUUM FULL

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

On 4/7/17 10:19 PM, Claudio Freire wrote: > > I rebased the early free patch (patch 3) to apply on top of the v9 > patch 2 (it needed some changes). I recognize the early free patch > didn't get nearly as much scrutiny, so I'm fine with commiting only 2 > if that one's ready to go but 3 isn't. >

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

On Fri, Apr 7, 2017 at 10:06 PM, Claudio Freire wrote: >>> >> + if (seg->num_dead_tuples >= seg->max_dead_tuples) >>> >> + { >>> >> + /* >>> >> + * The segment is overflowing, so we must allocate >>> >> a

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

On Fri, Apr 7, 2017 at 10:12 PM, Andres Freund wrote: > On 2017-04-07 22:06:13 -0300, Claudio Freire wrote: >> On Fri, Apr 7, 2017 at 9:56 PM, Andres Freund wrote: >> > Hi, >> > >> > >> > On 2017-04-07 19:43:39 -0300, Claudio Freire wrote: >> >> On Fri,

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

On 2017-04-07 22:06:13 -0300, Claudio Freire wrote: > On Fri, Apr 7, 2017 at 9:56 PM, Andres Freund wrote: > > Hi, > > > > > > On 2017-04-07 19:43:39 -0300, Claudio Freire wrote: > >> On Fri, Apr 7, 2017 at 5:05 PM, Andres Freund wrote: > >> > Hi, > >> > >

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

On Fri, Apr 7, 2017 at 9:56 PM, Andres Freund wrote: > Hi, > > > On 2017-04-07 19:43:39 -0300, Claudio Freire wrote: >> On Fri, Apr 7, 2017 at 5:05 PM, Andres Freund wrote: >> > Hi, >> > >> > I've *not* read the history of this thread. So I really might

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

Hi, On 2017-04-07 19:43:39 -0300, Claudio Freire wrote: > On Fri, Apr 7, 2017 at 5:05 PM, Andres Freund wrote: > > Hi, > > > > I've *not* read the history of this thread. So I really might be > > missing some context. > > > > > >> From

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

On Fri, Apr 7, 2017 at 5:05 PM, Andres Freund wrote: > Hi, > > I've *not* read the history of this thread. So I really might be > missing some context. > > >> From e37d29c26210a0f23cd2e9fe18a264312fecd383 Mon Sep 17 00:00:00 2001 >> From: Claudio Freire

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

On Fri, Apr 7, 2017 at 7:43 PM, Claudio Freire wrote: >>> + * Lookup in that structure proceeds sequentially in the list of segments, >>> + * and with a binary search within each segment. Since segment's size grows >>> + * exponentially, this retains O(N log N) lookup

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

Hi, I've *not* read the history of this thread. So I really might be missing some context. > From e37d29c26210a0f23cd2e9fe18a264312fecd383 Mon Sep 17 00:00:00 2001 > From: Claudio Freire > Date: Mon, 12 Sep 2016 23:36:42 -0300 > Subject: [PATCH] Vacuum: allow using

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

On Wed, Feb 1, 2017 at 7:55 PM, Claudio Freire wrote: > On Wed, Feb 1, 2017 at 6:13 PM, Masahiko Sawada wrote: >> On Wed, Feb 1, 2017 at 10:02 PM, Claudio Freire >> wrote: >>> On Wed, Feb 1, 2017 at 5:47 PM, Masahiko Sawada

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

On Wed, Feb 1, 2017 at 11:55 PM, Claudio Freire wrote: > On Wed, Feb 1, 2017 at 6:13 PM, Masahiko Sawada wrote: >> On Wed, Feb 1, 2017 at 10:02 PM, Claudio Freire >> wrote: >>> On Wed, Feb 1, 2017 at 5:47 PM, Masahiko

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

On Wed, Feb 1, 2017 at 6:13 PM, Masahiko Sawada wrote: > On Wed, Feb 1, 2017 at 10:02 PM, Claudio Freire > wrote: >> On Wed, Feb 1, 2017 at 5:47 PM, Masahiko Sawada >> wrote: >>> Thank you for updating the patch. >>> >>>

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

On Wed, Feb 1, 2017 at 10:02 PM, Claudio Freire wrote: > On Wed, Feb 1, 2017 at 5:47 PM, Masahiko Sawada wrote: >> Thank you for updating the patch. >> >> Whole patch looks good to me except for the following one comment. >> This is the final

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

On Wed, Feb 1, 2017 at 5:47 PM, Masahiko Sawada wrote: > Thank you for updating the patch. > > Whole patch looks good to me except for the following one comment. > This is the final comment from me. > > /* > * lazy_tid_reaped() -- is a particular tid deletable? > * > *

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

On Tue, Jan 31, 2017 at 3:05 AM, Claudio Freire wrote: > On Mon, Jan 30, 2017 at 5:51 AM, Masahiko Sawada > wrote: >> >> * We are willing to use at most maintenance_work_mem (or perhaps >> * autovacuum_work_mem) memory space to keep track of

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

On Tue, Jan 31, 2017 at 11:05 AM, Claudio Freire wrote: > Updated and rebased v7 attached. Moved to CF 2017-03. -- Michael -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription:

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

On Mon, Jan 30, 2017 at 5:51 AM, Masahiko Sawada wrote: > > * We are willing to use at most maintenance_work_mem (or perhaps > * autovacuum_work_mem) memory space to keep track of dead tuples. We > * initially allocate an array of TIDs of that size, with an upper

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

On Thu, Jan 26, 2017 at 5:11 AM, Claudio Freire wrote: > On Wed, Jan 25, 2017 at 1:54 PM, Masahiko Sawada > wrote: >> Thank you for updating the patch! >> >> + /* >> +* Quickly rule out by lower bound (should happen a lot) Upper bound

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

On Wed, Jan 25, 2017 at 1:54 PM, Masahiko Sawada wrote: > Thank you for updating the patch! > > + /* > +* Quickly rule out by lower bound (should happen a lot) Upper bound > was > +* already checked by segment search > +*/ > + if

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

On Tue, Jan 24, 2017 at 1:49 AM, Claudio Freire wrote: > On Fri, Jan 20, 2017 at 6:24 AM, Masahiko Sawada > wrote: >> On Thu, Jan 19, 2017 at 8:31 PM, Claudio Freire >> wrote: >>> On Thu, Jan 19, 2017 at 6:33 AM, Anastasia

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

On Fri, Jan 20, 2017 at 6:24 AM, Masahiko Sawada wrote: > On Thu, Jan 19, 2017 at 8:31 PM, Claudio Freire > wrote: >> On Thu, Jan 19, 2017 at 6:33 AM, Anastasia Lubennikova >> wrote: >>> 28.12.2016 23:43, Claudio

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

I think this patch no longer applies because of conflicts with the one I just pushed. Please rebase. Thanks -- Álvaro Herrerahttps://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services -- Sent via pgsql-hackers mailing list

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

I pushed this patch after rewriting it rather completely. I added tracing notices to inspect the blocks it was prefetching and observed that the original coding was failing to prefetch the final streak of blocks in the table, which is an important oversight considering that it may very well be

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

Alvaro Herrera wrote: > There was no discussion whatsoever of the "prefetch" patch in this > thread; and as far as I can see, nobody even mentioned such an idea in > the thread. This prefetch patch appeared out of the blue and there was > no discussion about it that I can see. Now I was about

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

You posted two patches with this preamble: Claudio Freire wrote: > Attached is the raw output of the test, the script used to create it, > and just in case the patch set used. I believe it's the same as the > last one I posted, just rebased. There was no discussion whatsoever of the "prefetch"

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

On Thu, Jan 19, 2017 at 8:31 PM, Claudio Freire wrote: > On Thu, Jan 19, 2017 at 6:33 AM, Anastasia Lubennikova > wrote: >> 28.12.2016 23:43, Claudio Freire: >> >> Attached v4 patches with the requested fixes. >> >> >> Sorry for being late,

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

On Thu, Jan 19, 2017 at 6:33 AM, Anastasia Lubennikova wrote: > 28.12.2016 23:43, Claudio Freire: > > Attached v4 patches with the requested fixes. > > > Sorry for being late, but the tests took a lot of time. I know. Takes me several days to run my test scripts

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

28.12.2016 23:43, Claudio Freire: Attached v4 patches with the requested fixes. Sorry for being late, but the tests took a lot of time. create table t1 as select i, md5(random()::text) from generate_series(0,4) as i; create index md5_idx ON t1(md5); update t1 set md5 = md5((random()

Re: [HACKERS] VACUUM FULL Error

Hi there Can this be forwarded to someone who can assist with this query? Thank you, Patrick Hayes From: Hayes, Patrick Sent: 29 December 2016 12:26 To: 'pgsql-hackers@postgresql.org' Subject: FW: VACUUM FULL Error Hi there Any suggestion how to get around this

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

On Wed, Dec 28, 2016 at 3:41 PM, Claudio Freire wrote: >> Anyway, I found the problem that had caused segfault. >> >> for (segindex = 0; segindex <= vacrelstats->dead_tuples.last_seg; tupindex = >> 0, segindex++) >> { >> DeadTuplesSegment *seg = >>

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

On Wed, Dec 28, 2016 at 10:26 AM, Anastasia Lubennikova wrote: > 27.12.2016 20:14, Claudio Freire: > > On Tue, Dec 27, 2016 at 10:41 AM, Anastasia Lubennikova > wrote: > > Program terminated with signal SIGSEGV, Segmentation fault. > #0

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

27.12.2016 20:14, Claudio Freire: On Tue, Dec 27, 2016 at 10:41 AM, Anastasia Lubennikova wrote: Program terminated with signal SIGSEGV, Segmentation fault. #0 0x006941e7 in lazy_vacuum_heap (onerel=0x1ec2360, vacrelstats=0x1ef6e00) at vacuumlazy.c:1417

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

27.12.2016 16:54, Alvaro Herrera: Anastasia Lubennikova wrote: I ran configure using following set of flags: ./configure --enable-tap-tests --enable-cassert --enable-debug --enable-depend CFLAGS="-O0 -g3 -fno-omit-frame-pointer" And then ran make check. Here is the stacktrace: Program

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

On Tue, Dec 27, 2016 at 10:41 AM, Anastasia Lubennikova wrote: > Program terminated with signal SIGSEGV, Segmentation fault. > #0 0x006941e7 in lazy_vacuum_heap (onerel=0x1ec2360, > vacrelstats=0x1ef6e00) at vacuumlazy.c:1417 > 1417tblk = >

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

On Tue, Dec 27, 2016 at 10:54 AM, Alvaro Herrera wrote: > Anastasia Lubennikova wrote: > >> I ran configure using following set of flags: >> ./configure --enable-tap-tests --enable-cassert --enable-debug >> --enable-depend CFLAGS="-O0 -g3 -fno-omit-frame-pointer" >> And

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

Anastasia Lubennikova wrote: > I ran configure using following set of flags: > ./configure --enable-tap-tests --enable-cassert --enable-debug > --enable-depend CFLAGS="-O0 -g3 -fno-omit-frame-pointer" > And then ran make check. Here is the stacktrace: > > Program terminated with signal SIGSEGV,

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

23.12.2016 22:54, Claudio Freire: On Fri, Dec 23, 2016 at 1:39 PM, Anastasia Lubennikova wrote: I found the reason. I configure postgres with CFLAGS="-O0" and it causes Segfault on initdb. It works fine and passes tests with default configure flags, but I'm

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

On Fri, Dec 23, 2016 at 1:39 PM, Anastasia Lubennikova wrote: >> On Thu, Dec 22, 2016 at 12:22 PM, Claudio Freire >> wrote: >>> >>> On Thu, Dec 22, 2016 at 12:15 PM, Anastasia Lubennikova >>> wrote: The

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

22.12.2016 21:18, Claudio Freire: On Thu, Dec 22, 2016 at 12:22 PM, Claudio Freire wrote: On Thu, Dec 22, 2016 at 12:15 PM, Anastasia Lubennikova wrote: The following review has been posted through the commitfest application: make

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

On Thu, Dec 22, 2016 at 12:22 PM, Claudio Freire wrote: > On Thu, Dec 22, 2016 at 12:15 PM, Anastasia Lubennikova > wrote: >> The following review has been posted through the commitfest application: >> make installcheck-world: tested, failed >>

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

On Thu, Dec 22, 2016 at 12:15 PM, Anastasia Lubennikova wrote: > The following review has been posted through the commitfest application: > make installcheck-world: tested, failed > Implements feature: not tested > Spec compliant: not tested >

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

The following review has been posted through the commitfest application: make installcheck-world: tested, failed Implements feature: not tested Spec compliant: not tested Documentation:not tested Hi, I haven't read through the thread yet, just tried to apply the patch

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

On Tue, Nov 22, 2016 at 4:53 AM, Claudio Freire wrote: > On Mon, Nov 21, 2016 at 2:15 PM, Masahiko Sawada > wrote: > > On Fri, Nov 18, 2016 at 6:54 AM, Claudio Freire > wrote: > >> On Thu, Nov 17, 2016 at 6:34 PM, Robert

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

On Mon, Nov 21, 2016 at 2:15 PM, Masahiko Sawada wrote: > On Fri, Nov 18, 2016 at 6:54 AM, Claudio Freire > wrote: >> On Thu, Nov 17, 2016 at 6:34 PM, Robert Haas wrote: >>> On Thu, Nov 17, 2016 at 1:42 PM, Claudio Freire

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

On Fri, Nov 18, 2016 at 6:54 AM, Claudio Freire wrote: > On Thu, Nov 17, 2016 at 6:34 PM, Robert Haas wrote: >> On Thu, Nov 17, 2016 at 1:42 PM, Claudio Freire >> wrote: >>> Attached is patch 0002 with pgindent applied over

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

On Thu, Nov 17, 2016 at 6:34 PM, Robert Haas wrote: > On Thu, Nov 17, 2016 at 1:42 PM, Claudio Freire > wrote: >> Attached is patch 0002 with pgindent applied over it >> >> I don't think there's any other formatting issue, but feel free to >> point

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

On Thu, Nov 17, 2016 at 1:42 PM, Claudio Freire wrote: > Attached is patch 0002 with pgindent applied over it > > I don't think there's any other formatting issue, but feel free to > point a finger to it if I missed any Hmm, I had imagined making all of the segments the

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

On Thu, Nov 17, 2016 at 1:42 PM, Claudio Freire wrote: > Attached is patch 0002 with pgindent applied over it > > I don't think there's any other formatting issue, but feel free to > point a finger to it if I missed any Hmm, I had imagined making all of the segments the

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

On Thu, Nov 17, 2016 at 2:51 PM, Claudio Freire wrote: > On Thu, Nov 17, 2016 at 2:34 PM, Masahiko Sawada > wrote: >> I glanced at the patches but the both patches don't obey the coding >> style of PostgreSQL. >> Please refer to [1]. >> >> [1] >>

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

On Thu, Nov 17, 2016 at 2:34 PM, Masahiko Sawada wrote: > I glanced at the patches but the both patches don't obey the coding > style of PostgreSQL. > Please refer to [1]. > > [1] >

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

On Thu, Oct 27, 2016 at 5:25 AM, Claudio Freire wrote: > On Thu, Sep 15, 2016 at 1:16 PM, Claudio Freire > wrote: >> On Wed, Sep 14, 2016 at 12:24 PM, Claudio Freire >> wrote: >>> On Wed, Sep 14, 2016 at 12:17 PM, Robert

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

On Fri, Sep 16, 2016 at 9:47 AM, Pavan Deolasee wrote: > On Fri, Sep 16, 2016 at 7:03 PM, Robert Haas wrote: >> On Thu, Sep 15, 2016 at 11:39 PM, Pavan Deolasee >> wrote: >> > But I actually wonder if we are over

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

On Fri, Sep 16, 2016 at 7:03 PM, Robert Haas wrote: > On Thu, Sep 15, 2016 at 11:39 PM, Pavan Deolasee > wrote: > > But I actually wonder if we are over engineering things and > overestimating > > cost of memmove etc. How about this simpler

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

On Thu, Sep 15, 2016 at 11:39 PM, Pavan Deolasee wrote: > But I actually wonder if we are over engineering things and overestimating > cost of memmove etc. How about this simpler approach: Don't forget that you need to handle the case where maintenance_work_mem is quite

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

On Fri, Sep 16, 2016 at 9:09 AM, Pavan Deolasee wrote: > > I also realised that we can compact the TID array in step (b) above > because we only need to store 17 bits for block numbers (we already know > which 1GB segment they belong to). Given that usable offsets are

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

On Fri, Sep 16, 2016 at 12:24 AM, Claudio Freire wrote: > On Thu, Sep 15, 2016 at 3:48 PM, Tomas Vondra > wrote: > > For example, we always allocate the TID array as large as we can fit into > > m_w_m, but maybe we don't need to wait with

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

On Thu, Sep 15, 2016 at 3:48 PM, Tomas Vondra wrote: > For example, we always allocate the TID array as large as we can fit into > m_w_m, but maybe we don't need to wait with switching to the bitmap until > filling the whole array - we could wait as long as the

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

On 09/15/2016 06:40 PM, Robert Haas wrote: On Thu, Sep 15, 2016 at 12:22 PM, Tom Lane wrote: Tomas Vondra writes: On 09/14/2016 07:57 PM, Tom Lane wrote: People who are vacuuming because they are out of disk space will be very very unhappy

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

On Thu, Sep 15, 2016 at 12:22 PM, Tom Lane wrote: > Tomas Vondra writes: >> On 09/14/2016 07:57 PM, Tom Lane wrote: >>> People who are vacuuming because they are out of disk space will be very >>> very unhappy with that solution. > >> The people

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

Tomas Vondra writes: > On 09/14/2016 07:57 PM, Tom Lane wrote: >> People who are vacuuming because they are out of disk space will be very >> very unhappy with that solution. > The people are usually running out of space for data, while these files > would be

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

On Thu, Sep 15, 2016 at 12:50 PM, Tomas Vondra wrote: > On 09/14/2016 07:57 PM, Tom Lane wrote: >> >> Pavan Deolasee writes: >>> >>> On Wed, Sep 14, 2016 at 10:53 PM, Alvaro Herrera >>> >>> wrote: One

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

On 09/14/2016 05:17 PM, Robert Haas wrote: I am kind of doubtful about this whole line of investigation because we're basically trying pretty hard to fix something that I'm not sure is broken.I do agree that, all other things being equal, the TID lookups will probably be faster with a

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

On 09/14/2016 07:57 PM, Tom Lane wrote: Pavan Deolasee writes: On Wed, Sep 14, 2016 at 10:53 PM, Alvaro Herrera wrote: One thing not quite clear to me is how do we create the bitmap representation starting from the array representation in

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

On Thu, Sep 15, 2016 at 2:40 AM, Simon Riggs wrote: > On 14 September 2016 at 11:19, Pavan Deolasee > wrote: > >>> In >>> theory we could even start with the list of TIDs and switch to the >>> bitmap if the TID list becomes larger than the

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

On Wed, Sep 14, 2016 at 1:23 PM, Alvaro Herrera wrote: > Robert Haas wrote: >> Actually, I think that probably *is* worthwhile, specifically because >> it might let us avoid multiple index scans in cases where we currently >> require them. Right now, our default

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

Pavan Deolasee writes: > On Wed, Sep 14, 2016 at 10:53 PM, Alvaro Herrera > wrote: >> One thing not quite clear to me is how do we create the bitmap >> representation starting from the array representation in midflight >> without using twice as

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

On Wed, Sep 14, 2016 at 10:53 PM, Alvaro Herrera wrote: > > > One thing not quite clear to me is how do we create the bitmap > representation starting from the array representation in midflight > without using twice as much memory transiently. Are we going to write >

  1   2   3   4   5   6   7   8   9   >