Re: [HACKERS] EXPLAIN BUFFERS

2009-11-23 Thread Jeff Janes
On Thu, Oct 15, 2009 at 3:29 AM, Itagaki Takahiro itagaki.takah...@oss.ntt.co.jp wrote: Robert Haas robertmh...@gmail.com wrote: In this case, I think that the auto_explain changes out to be part of the same patch as the core EXPLAIN changes Here is a rewritten patch to add EXPLAIN

[HACKERS] TODO: fix priority of ordering of read and write light-weight locks

2009-08-11 Thread Jeff Janes
The wiki todo has the topic Fix priority ordering of read and write light-weight locks and references http://archives.postgresql.org/pgsql-hackers/2004-11/msg00893.php (lwlocks and starvation) Having read the referenced discussion, I'm trying to figure out what remains to be done. Tom proposed a

[HACKERS] Wisconsin benchmark

2009-08-13 Thread Jeff Janes
The Wisconsin Benchmark in src/test/bench is broken, probably since 8.2. Attached is a tested patch that fixes it. However, it might be better to just remove src/test/bench. The benchmark is quite useless, because it is single user test that runs in the stand alone mode, and because it is

Fwd: [HACKERS] TODO: fix priority of ordering of read and write light-weight locks

2009-08-14 Thread Jeff Janes
Sorry, I screwed up. The below was supposed to go to the list, not Tom personally. -- Forwarded message -- From: Jeff Janes jeff.ja...@gmail.com Date: Thu, Aug 13, 2009 at 9:32 PM Subject: Re: [HACKERS] TODO: fix priority of ordering of read and write light-weight locks To: Tom

Re: [HACKERS] REGRESS_OPTS versus MSVC build scripts

2009-08-19 Thread Jeff Janes
-- Forwarded message -- From: David Fetter da...@fetter.org To: Andrew Dunstan and...@dunslane.net Date: Tue, 18 Aug 2009 11:31:41 -0700 Subject: Re: REGRESS_OPTS versus MSVC build scripts On Tue, Aug 18, 2009 at 02:15:48PM -0400, Andrew Dunstan wrote: Andrew Dunstan

[HACKERS] XLogInsert

2009-08-19 Thread Jeff Janes
In XLogInsert (src/backend/access/transam/xlog.c), the part that adds back-up blocks into the rdata chain is described: /* * Make additional rdata chain entries for the backup blocks, so that we * don't need to special-case them in the write loop. Note that we have

[HACKERS] XLogFlush

2009-08-21 Thread Jeff Janes
Maybe this is one of those things that is obvious when someone points it out to you, but right now I am not seeing it. If you look at the last eight lines of this snippet from XLogFlush, you see that if we obtain WriteRqstPtr under the WALInsertLock, then we both write and flush up to the highest

[HACKERS] Bug in visibility hint bit

2009-08-24 Thread Jeff Janes
There seems to be a bug in the visibility map in 8.4.0, introduced to cvs on 2008-12-03. It results in tuples being called visible that shouldn't be. In heap_update function from heapam.c: /* * Note: we mustn't clear PD_ALL_VISIBLE flags before writing the WAL *

Re: [HACKERS] Bug in visibility hint bit

2009-08-24 Thread Jeff Janes
On Mon, Aug 24, 2009 at 6:23 PM, Jeff Janesjeff.ja...@gmail.com wrote: There seems to be a bug in the visibility map in 8.4.0, introduced to cvs on 2008-12-03. It results in tuples being called visible that shouldn't be. Well, never mind. It took me a few days to track down the bug and in

Re: [HACKERS] [pgsql-hackers] Daily digest v1.9418 (15 messages)

2009-08-27 Thread Jeff Janes
-- Forwarded message -- From: Kevin Grittner kevin.gritt...@wicourts.gov To: Robert Haas robertmh...@gmail.com, Bruce Momjian br...@momjian.us Date: Thu, 27 Aug 2009 09:07:05 -0500 Subject: Re: 8.5 release timetable, again Robert Haas robertmh...@gmail.com wrote: Maybe

Re: [HACKERS] [pgsql-hackers] Daily digest v1.9418 (15 messages)

2009-08-27 Thread Jeff Janes
-- Forwarded message -- From: Tom Lane t...@sss.pgh.pa.us To: Robert Haas robertmh...@gmail.com Date: Thu, 27 Aug 2009 10:11:24 -0400 Subject: Re: 8.5 release timetable, again What I'd like to see is some sort of test mechanism for WAL recovery. What I've done sometimes in

[HACKERS] LWLock Queue Jumping

2009-08-28 Thread Jeff Janes
-- Forwarded message -- From: Simon Riggs si...@2ndquadrant.com To: pgsql-hackers pgsql-hackers@postgresql.org Date: Fri, 28 Aug 2009 20:07:32 +0100 Subject: LWLock Queue Jumping WALInsertLock is heavily contended and likely always will be even if we apply some of the

Re: [HACKERS] LWLock Queue Jumping

2009-08-29 Thread Jeff Janes
On Sat, Aug 29, 2009 at 4:02 AM, Simon Riggs si...@2ndquadrant.com wrote: On Fri, 2009-08-28 at 14:44 -0700, Jeff Janes wrote: I'd previously implemented this just by copying and pasting and making some changes, perhaps not the most desirable way but I thought adding another parameter

Re: [HACKERS] [pgsql-hackers] Daily digest v1.9430 (16 messages)

2009-08-30 Thread Jeff Janes
-- Forwarded message -- From: Greg Stark gsst...@mit.edu To: Simon Riggs si...@2ndquadrant.com Date: Sun, 30 Aug 2009 00:28:14 +0100 Subject: Re: LWLock Queue Jumping On Fri, Aug 28, 2009 at 8:07 PM, Simon Riggssi...@2ndquadrant.com wrote: WALInsertLock is heavily

Re: [HACKERS] LWLock Queue Jumping

2009-08-30 Thread Jeff Janes
-- Forwarded message -- From: Stefan Kaltenbrunner ste...@kaltenbrunner.cc To: Heikki Linnakangas heikki.linnakan...@enterprisedb.com Date: Sun, 30 Aug 2009 11:48:47 +0200 Subject: Re: LWLock Queue Jumping Heikki Linnakangas wrote: Greg Stark wrote: On Fri, Aug 28, 2009

Re: [HACKERS] LWLock Queue Jumping

2009-08-30 Thread Jeff Janes
On Sun, Aug 30, 2009 at 11:01 AM, Stefan Kaltenbrunner ste...@kaltenbrunner.cc wrote: Jeff Janes wrote: -- Forwarded message -- From: Stefan Kaltenbrunner ste...@kaltenbrunner.cc To: Heikki Linnakangas heikki.linnakan...@enterprisedb.com mailto:heikki.linnakan

Re: [HACKERS] XLogFlush

2009-08-31 Thread Jeff Janes
On Fri, Aug 21, 2009 at 1:18 AM, Jeff Janes jeff.ja...@gmail.com wrote: Maybe this is one of those things that is obvious when someone points it out to you, but right now I am not seeing it. If you look at the last eight lines of this snippet from XLogFlush, you see that if we obtain

[HACKERS] pgbench hard coded constants

2009-08-31 Thread Jeff Janes
pgbench has #defines for number of branches, tellers, and accounts. There are used to populate the tables with -i, but when running actual benchmark it has values separately hard-coded in the query metacommands. This patch makes the metacommands obtain their values from the relevant #defines.

[HACKERS] test_fsync file overrun

2009-09-13 Thread Jeff Janes
test_fsync in tools/fsync pre-creates a 16MB file. If it is given a number of iterations greater than 1024 (like one might use if trying to see what happens when NVRAM gets filled, or on a journaling file system), than one of the writes being timed will have to extend the size of the pre-created

Re: [HACKERS] XLogInsert

2009-09-13 Thread Jeff Janes
On Wed, Aug 19, 2009 at 9:49 AM, Tom Lane t...@sss.pgh.pa.us wrote: Jeff Janes jeff.ja...@gmail.com writes: If I read the code correctly, the only thing that is irrevocable is that it writes into rdt-next, and if it saved an old copy of rdt first, then it could revoke the changes just

Re: [HACKERS] Patch LWlocks instrumentation

2009-09-14 Thread Jeff Janes
2009/9/14 Pierre Frédéric Caillaud li...@peufeu.com A little bit of a reply to Jeff's email about WALInsertLock. This patch instruments LWLocks, it is controlled with the following #define's in lwlock.c : LWLOCK_STATS LWLOCK_TIMING_STATS It is an upgrade of current lwlocks stats. Hi

Re: [HACKERS] Bulk Inserts

2009-09-14 Thread Jeff Janes
2009/9/14 Pierre Frédéric Caillaud li...@peufeu.com I've done a little experiment with bulk inserts. = heap_bulk_insert() Behaves like heap_insert except it takes an array of tuples (HeapTuple *tups, int ntups). - Grabs a page (same as heap_insert) - While holding exclusive lock,

Re: [HACKERS] Bulk Inserts

2009-09-14 Thread Jeff Janes
2009/9/14 Pierre Frédéric Caillaud li...@peufeu.com Replying to myself... Jeff suggested to build pages in local memory and insert them later in the table. This is what's used in CLUSTER for instance, I believe. It has some drawbacks though : - To insert the tuples in indexes, the tuples

Re: [HACKERS] Bulk Inserts

2009-09-15 Thread Jeff Janes
2009/9/15 Pierre Frédéric Caillaud li...@peufeu.com Does that heuristic change the timings much? If not, it seems like it would better to keep it simple and always do the same thing, like log the tuples (if it is done under one WALInsertLock, which I am assuming it is..) It is the logging

Re: [HACKERS] opportunistic tuple freezing

2009-09-15 Thread Jeff Janes
to test for and also be of limited real-world applicability. Cheers, Jeff (Janes)

Re: [HACKERS] opportunistic tuple freezing

2009-09-17 Thread Jeff Janes
evaluate this patch and because it seems like something that should be reported. Cheers, Jeff Janes

Re: [HACKERS] Bulk Inserts

2009-09-17 Thread Jeff Janes
2009/9/16 Pierre Frédéric Caillaud li...@peufeu.com OK, that makes sense. I thought you had hacked either XLogInsert or the heap WAL replay code so that you could just accumulate tuples in the rdata chain and then submit them all under the cover of a single WALInsertLock. Ah, no, I did

Re: [HACKERS] Hot Standby 0.2.1

2009-09-18 Thread Jeff Janes
On Tue, Sep 15, 2009 at 2:41 PM, Simon Riggs si...@2ndquadrant.com wrote: OK, here is the latest version of the Hot Standby patchset. This is about version 30+ by now, but we should regard this as 0.2.1 Patch against CVS HEAD (now): clean apply, compile, no known bugs. Hi Simon, Is there

Re: [HACKERS] SELECT ... FOR UPDATE [WAIT integer | NOWAIT] for 8.5

2009-09-18 Thread Jeff Janes
On Thu, Sep 3, 2009 at 6:47 AM, Boszormenyi Zoltan z...@cybertec.at wrote: Boszormenyi Zoltan írta: Okay, we implemented only the lock_timeout GUC. Patch attached, hopefully in an acceptable form. Documentation included in the patch, lock_timeout works the same way as

Re: [HACKERS] SELECT ... FOR UPDATE [WAIT integer | NOWAIT] for 8.5

2009-09-19 Thread Jeff Janes
On Thu, Sep 3, 2009 at 6:47 AM, Boszormenyi Zoltan z...@cybertec.at wrote: Boszormenyi Zoltan írta: Alvaro Herrera írta: Boszormenyi Zoltan wrote: The vague consensus for syntax options was that the GUC 'lock_timeout' and WAIT [N] extension (wherever NOWAIT is allowed) both

Re: [HACKERS] Hot Standby 0.2.1

2009-09-21 Thread Jeff Janes
On Tue, Sep 15, 2009 at 2:41 PM, Simon Riggs si...@2ndquadrant.com wrote: OK, here is the latest version of the Hot Standby patchset. This is about version 30+ by now, but we should regard this as 0.2.1 Patch against CVS HEAD (now): clean apply, compile, no known bugs. OVERVIEW You can

Re: [HACKERS] Hot Standby 0.2.1

2009-09-23 Thread Jeff Janes
On Tue, Sep 22, 2009 at 10:02 AM, Alvaro Herrera alvhe...@commandprompt.com wrote: Heikki Linnakangas escribió: Simon Riggs wrote: On Mon, 2009-09-21 at 19:42 -0700, Jeff Janes wrote: jjanes=# begin; BEGIN jjanes=# lock table pgbench_history in access exclusive mode; LOCK TABLE

Re: [HACKERS] SELECT ... FOR UPDATE [WAIT integer | NOWAIT] for 8.5

2009-09-23 Thread Jeff Janes
On Mon, Sep 21, 2009 at 3:07 AM, Boszormenyi Zoltan z...@cybertec.at wrote: Jeff Janes írta: Maybe I am biased in this because I am primarily thinking about how I would use such a feature, rather than how Hans-Juergen intends to use it, and maybe those uses differ. Hans-Juergen, could you

Re: [HACKERS] Lock Wait Statistics (next commitfest)

2009-09-29 Thread Jeff Janes
On Mon, Sep 28, 2009 at 12:14 AM, Jaime Casanova jcasa...@systemguards.com.ec wrote: On Sat, Aug 8, 2009 at 7:47 PM, Mark Kirkwood mar...@paradise.net.nz wrote: Patch with max(wait time). Still TODO - amalgamate individual transaction lock waits - redo (rather ugly) temporary

Re: [HACKERS] Lock Wait Statistics (next commitfest)

2009-10-04 Thread Jeff Janes
On Mon, Sep 28, 2009 at 12:14 AM, Jaime Casanova jcasa...@systemguards.com.ec wrote: On Sat, Aug 8, 2009 at 7:47 PM, Mark Kirkwood mar...@paradise.net.nz wrote: Patch with max(wait time). Still TODO - amalgamate individual transaction lock waits - redo (rather ugly) temporary

[HACKERS] Making hash indexes worthwhile

2009-10-04 Thread Jeff Janes
I see that the docs were recently changed from discouraging hash indexes both because they were no known uses in which they were meaningfully better than btree, and because they aren't recoverable; to now just discouraging them because they are not recoverable. Does that mean there are now uses

Re: [HACKERS] Making hash indexes worthwhile

2009-10-05 Thread Jeff Janes
On Sun, Oct 4, 2009 at 7:13 PM, Tom Lane t...@sss.pgh.pa.us wrote: Jeff Janes jeff.ja...@gmail.com writes: I've played around a bit with hash indexes, and it seems to me that making them generally worthwhile will take (at least) reducing or entirely doing away with the heavy-wait locks

Re: [HACKERS] Making hash indexes worthwhile

2009-10-06 Thread Jeff Janes
On Mon, Oct 5, 2009 at 6:49 AM, Tom Lane t...@sss.pgh.pa.us wrote: Jeff Janes jeff.ja...@gmail.com writes: Do you know why that should be?  I've done some work with gprof, and the results are pretty suspect, because the total gprof time adds up to only about 1/3 of the total time the backend

Re: [HACKERS] Concurrency testing

2009-10-07 Thread Jeff Janes
On Wed, Oct 7, 2009 at 10:38 AM, David Fetter da...@fetter.org wrote: On Wed, Oct 07, 2009 at 10:28:08AM -0700, David Wheeler wrote: On Oct 7, 2009, at 9:53 AM, David Fetter wrote: At the moment, we have no way to test this, although with certain Perl modules, it would be pretty trivial. No

Re: [HACKERS] Concurrency testing

2009-10-07 Thread Jeff Janes
On Wed, Oct 7, 2009 at 6:01 PM, Robert Haas robertmh...@gmail.com wrote: On Wed, Oct 7, 2009 at 8:33 PM, David E. Wheeler da...@kineticode.com wrote: Do we need to restrict ourselves to core?  Developers already need flex and bison, which aren't needed when installing from the tarball.

Re: [HACKERS] Concurrency testing

2009-10-07 Thread Jeff Janes
On Wed, Oct 7, 2009 at 5:33 PM, David E. Wheeler da...@kineticode.com wrote: On Oct 7, 2009, at 5:18 PM, Jeff Janes wrote: I'd much rather live without Test::More and use DBD::Pg, then have Test::More but need to open pipes to psql to talk to the database, rather than using DBI to do

Re: [HACKERS] per table random-page-cost?

2009-10-19 Thread Jeff Janes
On Mon, Oct 19, 2009 at 4:29 PM, Greg Stark gsst...@mit.edu wrote: On Mon, Oct 19, 2009 at 4:21 PM, Tom Lane t...@sss.pgh.pa.us wrote: marcin mank marcin.m...@gmail.com writes: This proposal is just hints by the back door, ISTM. As Tom says, there is a justification for having it on

[HACKERS] misleading comments in pgbench

2009-10-23 Thread Jeff Janes
From contrib/pgbench/pgbench.c starting in revision 1.77 * Note: TPC-B requires at least 100 bytes per row, and the filler * fields in these table declarations were intended to comply with that. * But because they default to NULLs, they don't actually take any * space. We

Re: [HACKERS] Making hash indexes worthwhile

2009-10-30 Thread Jeff Janes
On Sun, Oct 4, 2009 at 7:13 PM, Tom Lane t...@sss.pgh.pa.us wrote: Jeff Janes jeff.ja...@gmail.com writes: I see that the docs were recently changed from discouraging hash indexes both because they were no known uses in which they were meaningfully better than btree, and because they aren't

Re: [HACKERS] Page Checksums + Double Writes

2011-12-23 Thread Jeff Janes
On 12/23/11, Robert Haas robertmh...@gmail.com wrote: On Fri, Dec 23, 2011 at 11:14 AM, Kevin Grittner kevin.gritt...@wicourts.gov wrote: Thoughts? Those are good thoughts. Here's another random idea, which might be completely nuts. Maybe we could consider some kind of summarization of

Re: [HACKERS] 16-bit page checksums for 9.2

2011-12-30 Thread Jeff Janes
On 12/29/11, Ants Aasma ants.aa...@eesti.ee wrote: Unless I'm missing something, double-writes are needed for all writes, not only the first page after a checkpoint. Consider this sequence of events: 1. Checkpoint 2. Double-write of page A (DW buffer write, sync, heap write) 3. Sync of

Re: [HACKERS] LWLOCK_STATS

2012-01-06 Thread Jeff Janes
On Fri, Jan 6, 2012 at 2:24 PM, Robert Haas robertmh...@gmail.com wrote: It's been a while since I did any testing with LWLOCK_STATS defined, so I thought it might be about time to do that again and see how things look.  Here's how they looked back in July:

[HACKERS] bgwriter holds onto file handles of deleted files

2012-01-07 Thread Jeff Janes
On an overnight test run, I ran into an unexpected out of space condition. The writer process was holding on to dozens of file descriptors for long-ago deleted files from dropped tables and indexes, preventing their disk space from being freed for reuse. I think the checkpoint writer process

Re: [HACKERS] bgwriter holds onto file handles of deleted files

2012-01-08 Thread Jeff Janes
On 1/8/12, Alex Shulgin a...@commandprompt.com wrote: Jeff Janes jeff.ja...@gmail.com writes: It looks like it hold the file handles until either is it is killed and restarted, or until they get flushed out of vfd cache (which holds ~1000 files on my machine, so that can be a long time

Re: [HACKERS] checkpoint writeback via sync_file_range

2012-01-13 Thread Jeff Janes
On Thu, Jan 12, 2012 at 7:26 PM, Greg Smith g...@2ndquadrant.com wrote: On 1/11/12 9:25 AM, Andres Freund wrote: The heavy pressure putting it directly in the writeback queue leads to less efficient io because quite often it won't reorder sensibly with other io anymore and thelike. At least

Re: [HACKERS] Standalone synchronous master

2012-01-13 Thread Jeff Janes
On Fri, Jan 13, 2012 at 2:30 AM, Alexander Björnhagen alex.bjornha...@gmail.com wrote: At this point I feel that this new functionality might be a bit overkill for postgres, maybe it's better to stay lean and mean rather than add a controversial feature like this. I don't understand why this

Re: [HACKERS] read transaction and sync rep

2012-01-13 Thread Jeff Janes
On Fri, Jan 13, 2012 at 3:44 AM, Simon Riggs si...@2ndquadrant.com wrote: On Fri, Jan 13, 2012 at 11:30 AM, Fujii Masao masao.fu...@gmail.com wrote: I found that even read transaction waits for sync rep when it generates WAL records even if XID is not assigned. For example, imagine the case

Re: [HACKERS] Standalone synchronous master

2012-01-15 Thread Jeff Janes
On Fri, Jan 13, 2012 at 9:50 AM, Tom Lane t...@sss.pgh.pa.us wrote: Jeff Janes jeff.ja...@gmail.com writes: I don't understand why this is controversial.  In the current code, if you have a master and a single sync standby, and the master disappears and you promote the standby, now the new

Re: [HACKERS] Standalone synchronous master

2012-01-15 Thread Jeff Janes
On Fri, Jan 13, 2012 at 10:12 AM, Kevin Grittner kevin.gritt...@wicourts.gov wrote: Jeff Janes jeff.ja...@gmail.com wrote:\ I don't understand why this is controversial. I'm having a hard time seeing why this is considered a feature.  It seems to me what is being proposed is a mode

[HACKERS] Memory usage during sorting

2012-01-15 Thread Jeff Janes
In tuplesort.c, it initially reads tuples into memory until availMem is exhausted. It then switches to the tape sort algorithm, and allocates buffer space for each tape it will use. This substantially over-runs the allowedMem, and drives availMem negative. It works off this deficit by writing

Re: [HACKERS] automating CF submissions (was xlog location arithmetic)

2012-01-16 Thread Jeff Janes
On Mon, Jan 16, 2012 at 1:10 PM, Alvaro Herrera alvhe...@commandprompt.com wrote: Excerpts from Josh Berkus's message of lun ene 16 17:48:41 -0300 2012: Putting submitters aside, I have to say based on teaching people how to use the CF stuff on Thursday night that the process of submitting a

Re: [HACKERS] Publish checkpoint timing and sync files summary data to pg_stat_bgwriter

2012-01-21 Thread Jeff Janes
On Sun, Jan 15, 2012 at 10:28 PM, Greg Smith g...@2ndquadrant.com wrote: I would like to be able to tell people they need never turn on log_checkpoints if they monitor pg_stat_bgwriter instead, because that sets a good precedent for what direction the database is going.  It would be nice for

Re: [HACKERS] show Heap Fetches in EXPLAIN for index-only scans

2012-01-21 Thread Jeff Janes
On Fri, Jan 13, 2012 at 7:21 AM, Robert Haas robertmh...@gmail.com wrote: I think that people who are using index-only scans are going to want some way to find out the degree to which the scans are in fact index-only. So here's a 5-line patch that adds the number of heap fetches to the

Re: [HACKERS] Review of: explain / allow collecting row counts without timing info

2012-01-21 Thread Jeff Janes
On Fri, Jan 13, 2012 at 3:07 PM, Tomas Vondra t...@fuzzy.cz wrote: Fixed. The default value of TIMING option did not work as intended, it was set to true even for plain EXPLAIN (without ANALYZE). In that case the EXPLAIN failed. I've applied this over the show Heap Fetches in EXPLAIN ANALYZE

Re: [HACKERS] Avoid FK validations for no-rewrite ALTER TABLE ALTER TYPE

2012-01-21 Thread Jeff Janes
On Wed, Jan 4, 2012 at 12:35 PM, Noah Misch n...@leadboat.com wrote: I neglected to commit after revising the text of a few comments; use this version instead.  Thanks. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription:

Re: [HACKERS] JSON for PG 9.2

2012-01-21 Thread Jeff Janes
On Sun, Jan 15, 2012 at 8:08 AM, Andrew Dunstan and...@dunslane.net wrote: On 01/14/2012 03:06 PM, Andrew Dunstan wrote: OK, here's a patch that does both query_to_json and array_to_json, along with docs and regression tests. It include Robert's original patch, although I can produce a

Re: [HACKERS] Avoid FK validations for no-rewrite ALTER TABLE ALTER TYPE

2012-01-21 Thread Jeff Janes
On Sat, Jan 21, 2012 at 9:05 PM, Noah Misch n...@leadboat.com wrote: On Sat, Jan 21, 2012 at 08:04:20PM -0800, Jeff Janes wrote: This is failing make check for me. *** /tmp/foo/src/test/regress/expected/alter_table.out  Sat Jan 21 19:51:46 2012 --- /tmp/foo/src/test/regress/results

Re: [HACKERS] [v9.2] Fix Leaky View Problem

2012-01-22 Thread Jeff Janes
On Tue, Jan 17, 2012 at 7:08 PM, Robert Haas robertmh...@gmail.com wrote: On Sun, Jan 8, 2012 at 10:32 AM, Kohei KaiGai kai...@kaigai.gr.jp wrote: I guess you concerned about that expected/select_views_1.out is patched, not expected/select_views.out. I'm not sure the reason why regression test

Re: [HACKERS] CLOG contention, part 2

2012-01-22 Thread Jeff Janes
On Fri, Jan 20, 2012 at 6:44 AM, Simon Riggs si...@2ndquadrant.com wrote: OT: It would save lots of time if we had 2 things for the CF app: .. 2. Something that automatically tests patches. If you submit a patch we run up a blank VM and run patch applies on all patches. As soon as we get a

Re: Removing freelist (was Re: [HACKERS] Should I implement DROP INDEX CONCURRENTLY?)

2012-01-24 Thread Jeff Janes
On Sat, Jan 21, 2012 at 2:29 PM, Jim Nasby j...@nasby.net wrote: On Jan 20, 2012, at 11:54 AM, Heikki Linnakangas wrote: So, you're proposing that we remove freelist altogether? Sounds reasonable, but that needs to be performance tested somehow. I'm not sure what exactly the test should

Re: Removing freelist (was Re: [HACKERS] Should I implement DROP INDEX CONCURRENTLY?)

2012-01-24 Thread Jeff Janes
On Mon, Jan 23, 2012 at 5:06 AM, Robert Haas robertmh...@gmail.com wrote: On Mon, Jan 23, 2012 at 12:12 AM, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: On Sat, Jan 21, 2012 at 5:29 PM, Jim Nasby j...@nasby.net wrote: We should also look at having the freelist

Re: [HACKERS] some longer, larger pgbench tests with various performance-related patches

2012-01-25 Thread Jeff Janes
On Tue, Jan 24, 2012 at 12:53 PM, Robert Haas robertmh...@gmail.com wrote: Early yesterday morning, I was able to use Nate Boley's test machine do a single 30-minute pgbench run at scale factor 300 using a variety of trees built with various patches, and with the -l option added to track

Re: [HACKERS] some longer, larger pgbench tests with various performance-related patches

2012-01-25 Thread Jeff Janes
On Wed, Jan 25, 2012 at 9:09 AM, Robert Haas robertmh...@gmail.com wrote: On Wed, Jan 25, 2012 at 12:00 PM, Jeff Janes jeff.ja...@gmail.com wrote: On Tue, Jan 24, 2012 at 12:53 PM, Robert Haas robertmh...@gmail.com wrote: Early yesterday morning, I was able to use Nate Boley's test machine do

Re: [HACKERS] Simulating Clog Contention

2012-01-27 Thread Jeff Janes
On Thu, Jan 12, 2012 at 4:31 AM, Simon Riggs si...@2ndquadrant.com wrote: The following patch adds a pgbench option -I to load data using INSERTs, so that we can begin benchmark testing with rows that have large numbers of distinct un-hinted transaction ids. With a database pre-created using

Re: [HACKERS] CLOG contention, part 2

2012-01-27 Thread Jeff Janes
On Sat, Jan 21, 2012 at 7:31 AM, Simon Riggs si...@2ndquadrant.com wrote: Yes, it was. Sorry about that. New version attached, retesting while you read this. In my hands I could never get this patch to do anything. The new cache was never used. I think that that was because RecentXminPageno

Re: [HACKERS] CLOG contention, part 2

2012-01-27 Thread Jeff Janes
On Fri, Jan 27, 2012 at 3:16 PM, Merlin Moncure mmonc...@gmail.com wrote: On Fri, Jan 27, 2012 at 4:05 PM, Jeff Janes jeff.ja...@gmail.com wrote: Also, I think the general approach is wrong.  The only reason to have these pages in shared memory is that we can control access to them to prevent

Re: [HACKERS] initdb and fsync

2012-01-28 Thread Jeff Janes
On Sat, Jan 28, 2012 at 7:31 AM, Andrew Dunstan and...@dunslane.net wrote: On 01/27/2012 11:52 PM, Noah Misch wrote: Is a platform-independent fsync be available at initdb time? Not sure. It's a macro on Windows that calls _commit(fd), so it should be portable enough. I'm curious what

Re: [HACKERS] initdb and fsync

2012-01-28 Thread Jeff Janes
On Sat, Jan 28, 2012 at 10:18 AM, Tom Lane t...@sss.pgh.pa.us wrote: Andrew Dunstan and...@dunslane.net writes: I'm curious what problem we're actually solving here, though. I've run the buildfarm countless thousands of times on different VMs, and five of my seven current animals run in VMs,

Re: [HACKERS] Simulating Clog Contention

2012-01-28 Thread Jeff Janes
On Fri, Jan 27, 2012 at 1:45 PM, Jeff Janes jeff.ja...@gmail.com wrote: On Thu, Jan 12, 2012 at 4:31 AM, Simon Riggs si...@2ndquadrant.com wrote: The following patch adds a pgbench option -I to load data using INSERTs, so that we can begin benchmark testing with rows that have large numbers

Re: [HACKERS] CLOG contention

2012-01-28 Thread Jeff Janes
On Thu, Jan 12, 2012 at 4:49 AM, Simon Riggs si...@2ndquadrant.com wrote: On Thu, Jan 5, 2012 at 6:26 PM, Simon Riggs si...@2ndquadrant.com wrote: Patch to remove clog contention caused by dirty clog LRU. v2, minor changes, updated for recent commits This no longer applies to file

Re: [HACKERS] Group commit, revised

2012-01-28 Thread Jeff Janes
On Fri, Jan 27, 2012 at 5:35 AM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: On 26.01.2012 04:10, Robert Haas wrote: I think you should break this off into a new function, LWLockWaitUntilFree(), rather than treating it as a new LWLockMode. Also, instead of adding

Re: [HACKERS] CLOG contention, part 2

2012-01-29 Thread Jeff Janes
On Sun, Jan 29, 2012 at 12:18 PM, Simon Riggs si...@2ndquadrant.com wrote: On Fri, Jan 27, 2012 at 10:05 PM, Jeff Janes jeff.ja...@gmail.com wrote: On Sat, Jan 21, 2012 at 7:31 AM, Simon Riggs si...@2ndquadrant.com wrote: Yes, it was. Sorry about that. New version attached, retesting while

Re: [HACKERS] CLOG contention, part 2

2012-01-29 Thread Jeff Janes
On Sun, Jan 29, 2012 at 1:41 PM, Jeff Janes jeff.ja...@gmail.com wrote: On Sun, Jan 29, 2012 at 12:18 PM, Simon Riggs si...@2ndquadrant.com wrote: On Fri, Jan 27, 2012 at 10:05 PM, Jeff Janes jeff.ja...@gmail.com wrote: On Sat, Jan 21, 2012 at 7:31 AM, Simon Riggs si...@2ndquadrant.com wrote

Re: [HACKERS] Vacuum rate limit in KBps

2012-01-29 Thread Jeff Janes
On Sun, Jan 15, 2012 at 12:24 AM, Greg Smith g...@2ndquadrant.com wrote: If you then turn that equation around, making the maximum write rate the input, for any given cost delay and dirty page cost you can solve for the cost limit--the parameter in fictitious units everyone hates.  It works

Re: [HACKERS] Simulating Clog Contention

2012-01-30 Thread Jeff Janes
On Mon, Jan 30, 2012 at 7:24 AM, Robert Haas robertmh...@gmail.com wrote: On Sat, Jan 28, 2012 at 3:32 PM, Jeff Janes jeff.ja...@gmail.com wrote: I think that even in normal (non-initialization) usage, this message should be suppressed when the provided scale factor is equal

Re: [HACKERS] Simulating Clog Contention

2012-01-30 Thread Jeff Janes
On Thu, Jan 26, 2012 at 6:18 AM, Abhijit Menon-Sen a...@toroid.org wrote: At 2012-01-12 12:31:20 +, si...@2ndquadrant.com wrote: The following patch adds a pgbench option -I to load data using INSERTs This is just to confirm that the patch applies and builds and works fine (though of

Re: [HACKERS] Group commit, revised

2012-01-30 Thread Jeff Janes
On Sun, Jan 29, 2012 at 1:20 PM, Greg Smith g...@2ndquadrant.com wrote: On 01/28/2012 07:48 PM, Jeff Janes wrote: I haven't inspected that deep fall off at 30 clients for the patch. By way of reference, if I turn off synchronous commit, I get tps=1245.8 which is 100% CPU limited.  This sets

Re: [HACKERS] CLOG contention, part 2

2012-01-31 Thread Jeff Janes
On Mon, Jan 30, 2012 at 12:24 PM, Robert Haas robertmh...@gmail.com wrote: On Fri, Jan 27, 2012 at 8:21 PM, Jeff Janes jeff.ja...@gmail.com wrote: On Fri, Jan 27, 2012 at 3:16 PM, Merlin Moncure mmonc...@gmail.com wrote: On Fri, Jan 27, 2012 at 4:05 PM, Jeff Janes jeff.ja...@gmail.com wrote

Re: [HACKERS] Vacuum rate limit in KBps

2012-02-02 Thread Jeff Janes
On Sun, Jan 15, 2012 at 12:24 AM, Greg Smith g...@2ndquadrant.com wrote: So far the reaction I've gotten from my recent submission to make autovacuum log its read/write in MB/s has been rather positive.  I've been surprised at the unprecedented (to me at least) amount of backporting onto big

Re: [HACKERS] Review of: explain / allow collecting row counts without timing info

2012-02-03 Thread Jeff Janes
On Fri, Feb 3, 2012 at 8:12 AM, Robert Haas robertmh...@gmail.com wrote: On Sat, Jan 21, 2012 at 10:32 PM, Jeff Janes jeff.ja...@gmail.com wrote: On Fri, Jan 13, 2012 at 3:07 PM, Tomas Vondra t...@fuzzy.cz wrote: Fixed. The default value of TIMING option did not work as intended, it was set

Re: [HACKERS] Checkpoint sync pause

2012-02-03 Thread Jeff Janes
On Mon, Jan 16, 2012 at 5:59 PM, Greg Smith g...@2ndquadrant.com wrote: On 01/16/2012 11:00 AM, Robert Haas wrote: Also, I am still struggling with what the right benchmarking methodology even is to judge whether any patch in this area works.  Can you provide more details about your test

Re: [HACKERS] LWLockWaitUntilFree (was: Group commit, revised)

2012-02-03 Thread Jeff Janes
On Fri, Feb 3, 2012 at 12:57 PM, Robert Haas robertmh...@gmail.com wrote: I think I recommended a bad name for this function.  It's really LWLockAcquireOrWaitUntilFree.  Can we rename that? We might also want to consider what all the behaviors are that we might want here.  It strikes me that

Re: [HACKERS] Group commit, revised

2012-02-03 Thread Jeff Janes
On Mon, Jan 30, 2012 at 6:57 AM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: On 27.01.2012 15:38, Robert Haas wrote: On Fri, Jan 27, 2012 at 8:35 AM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com  wrote: Yeah, we have to be careful with any overhead in there, it

Re: [HACKERS] Memory usage during sorting

2012-02-04 Thread Jeff Janes
On Sat, Jan 21, 2012 at 4:51 PM, Peter Geoghegan pe...@2ndquadrant.com wrote: On 16 January 2012 00:59, Jeff Janes jeff.ja...@gmail.com wrote: I think it would be better to pre-deduct the tape overhead amount we will need if we decide to switch to tape sort from the availMem before we even

Re: [HACKERS] Memory usage during sorting

2012-02-04 Thread Jeff Janes
On Sat, Feb 4, 2012 at 10:06 AM, Jeff Janes jeff.ja...@gmail.com wrote: Attached is a completely uncommitable proof of concept/work in progress patch to get around the limitation.  It shows a 2 fold improvement when indexing an integer column on a 50,000,000 row randomly ordered table. Oops

Re: [HACKERS] some longer, larger pgbench tests with various performance-related patches

2012-02-04 Thread Jeff Janes
On Wed, Feb 1, 2012 at 9:47 AM, Robert Haas robertmh...@gmail.com wrote: On Wed, Jan 25, 2012 at 8:49 AM, Robert Haas robertmh...@gmail.com wrote: On Tue, Jan 24, 2012 at 4:28 PM, Simon Riggs si...@2ndquadrant.com wrote: I think we should be working to commit XLogInsert and then Group Commit,

Re: Scaling XLog insertion (was Re: [HACKERS] Moving more work outside WALInsertLock)

2012-02-08 Thread Jeff Janes
On Wed, Feb 1, 2012 at 11:46 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: On 31.01.2012 17:35, Fujii Masao wrote: On Fri, Jan 20, 2012 at 11:11 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com  wrote: On 20.01.2012 15:32, Robert Haas wrote: On Sat, Jan 14,

Re: [HACKERS] Memory usage during sorting

2012-02-11 Thread Jeff Janes
On Wed, Feb 8, 2012 at 1:01 AM, Hitoshi Harada umi.tan...@gmail.com wrote: On Sun, Jan 15, 2012 at 4:59 PM, Jeff Janes jeff.ja...@gmail.com wrote: The attached patch allows it to reuse that memory.  On my meager system it reduced the building of an index on an integer column in a skinny 200

Re: [HACKERS] random_page_cost vs seq_page_cost

2012-02-11 Thread Jeff Janes
On Tue, Feb 7, 2012 at 2:06 PM, Greg Smith g...@2ndquadrant.com wrote: On 02/07/2012 03:23 PM, Bruce Momjian wrote: Where did you see that there will be an improvement in the 9.2 documentation?  I don't see an improvement. I commented that I'm hoping for an improvement in the documentation

Re: [HACKERS] random_page_cost vs seq_page_cost

2012-02-11 Thread Jeff Janes
On Tue, Feb 7, 2012 at 4:58 PM, Bruce Momjian br...@momjian.us wrote: I was initially concerned that tuning advice in this part of the docs would look out of place, but now see the 25% shared_buffers recommentation, and it looks fine, so we are OK.  (Should we caution against more than 8GB of

Re: [HACKERS] some longer, larger pgbench tests with various performance-related patches

2012-02-11 Thread Jeff Janes
On Mon, Feb 6, 2012 at 6:38 AM, Robert Haas robertmh...@gmail.com wrote: On Sat, Feb 4, 2012 at 2:13 PM, Jeff Janes jeff.ja...@gmail.com wrote: We really need to nail that down.  Could you post the scripts (on the wiki) you use for running the benchmark and making the graph?  I'd like to see

Re: [HACKERS] Checkpoint sync pause

2012-02-12 Thread Jeff Janes
On Tue, Feb 7, 2012 at 1:22 PM, Greg Smith gsm...@gregsmith.com wrote: On 02/03/2012 11:41 PM, Jeff Janes wrote: -The steady stream of backend writes that happen between checkpoints have filled up most of the OS write cache.  A look at /proc/meminfo shows around 2.5GB Dirty: backend writes

Re: [HACKERS] Should I implement DROP INDEX CONCURRENTLY?

2012-02-12 Thread Jeff Janes
On Sun, Jan 8, 2012 at 8:19 AM, Simon Riggs si...@2ndquadrant.com wrote: On Wed, Jan 4, 2012 at 11:14 AM, Simon Riggs si...@2ndquadrant.com wrote: Not having a freelist at all is probably a simpler way of avoiding the lock contention, so I'll happily back that suggestion instead. Patch

Re: Scaling XLog insertion (was Re: [HACKERS] Moving more work outside WALInsertLock)

2012-02-12 Thread Jeff Janes
On Thu, Feb 9, 2012 at 3:02 AM, Fujii Masao masao.fu...@gmail.com wrote: On Thu, Feb 9, 2012 at 7:25 PM, Fujii Masao masao.fu...@gmail.com wrote: On Thu, Feb 9, 2012 at 3:32 AM, Jeff Janes jeff.ja...@gmail.com wrote: After applying this patch and then forcing crashes, upon recovery

Re: Scaling XLog insertion (was Re: [HACKERS] Moving more work outside WALInsertLock)

2012-02-18 Thread Jeff Janes
On Fri, Feb 17, 2012 at 7:36 AM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: On 17.02.2012 07:27, Fujii Masao wrote: Got another problem: when I ran pg_stop_backup to take an online backup, it got stuck until I had generated new WAL record. This happens because, in the

  1   2   3   4   5   6   7   8   9   10   >