Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-05-16 Thread Greg Smith
before its worth adding one for it. -- Greg Smith 2ndQuadrant USg...@2ndquadrant.com Baltimore, MD PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http

Re: [HACKERS] Move unused buffers to freelist

2013-05-14 Thread Greg Smith
of your patch. -- Greg Smith 2ndQuadrant USg...@2ndquadrant.com Baltimore, MD PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org

Re: [HACKERS] corrupt pages detected by enabling checksums

2013-05-10 Thread Greg Smith
order until that point. All you can rely on is throwing up a stop sign that says tell me when all of them are done. In between those, you have no idea of the ordering. -- Greg Smith 2ndQuadrant USg...@2ndquadrant.com Baltimore, MD PostgreSQL Training, Services, and 24x7 Support www

Re: [HACKERS] [PATCH] add --throttle to pgbench (submission 3)

2013-05-01 Thread Greg Smith
interesting, but it's not necessarily relevant in many cases. -- Greg Smith 2ndQuadrant USg...@2ndquadrant.com Baltimore, MD PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your

Re: [HACKERS] Substituting Checksum Algorithm (was: Enabling Checksums)

2013-04-30 Thread Greg Smith
. -- Greg Smith 2ndQuadrant USg...@2ndquadrant.com Baltimore, MD PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.com ChecksumMethods.xls Description: MS-Excel spreadsheet -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription

Re: [HACKERS] Substituting Checksum Algorithm (was: Enabling Checksums)

2013-04-30 Thread Greg Smith
without expecting any block differences, that will make both routine quality auditing and forensics of broken clusters so much easier. -- Greg Smith 2ndQuadrant USg...@2ndquadrant.com Baltimore, MD PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.com -- Sent via pgsql

Re: [HACKERS] Substituting Checksum Algorithm (was: Enabling Checksums)

2013-04-26 Thread Greg Smith
considering regardless. The optimizations seem to have a very significant impact on the checksum feature, but I'd like to quantify how they change the code a little bit before even getting into that. -- Greg Smith 2ndQuadrant USg...@2ndquadrant.com Baltimore, MD PostgreSQL Training

Re: [HACKERS] Enabling Checksums

2013-04-17 Thread Greg Smith
have now dug into how exactly that and specific CPU optimization concerns impact the best approach for 8K database pages. This is very clearly a 9.4 project that is just getting started. -- Greg Smith 2ndQuadrant USg...@2ndquadrant.com Baltimore, MD PostgreSQL Training, Services, and 24x7

Re: [HACKERS] Enabling Checksums

2013-04-17 Thread Greg Smith
(1), but I wouldn't give it 100% odds of happening either. The user demand that's motivated me to work on this will be happy with any of (1) through (3), and in two of them optimizing the 16 bit checksums now turns out to be premature. -- Greg Smith 2ndQuadrant USg...@2ndquadrant.com

Re: [HACKERS] PROPOSAL: tracking aggregated numbers from pg_stat_database

2013-04-16 Thread Greg Smith
. It's not a big difference, but thinking in that direction doesn't help your suggestion. -- Greg Smith 2ndQuadrant USg...@2ndquadrant.com Baltimore, MD PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org

Re: [HACKERS] Enabling Checksums

2013-04-05 Thread Greg Smith
, along with one of Seagate's new drives with a built-in BBWC. (Their latest SSHD flash hybrid model caches writes with a capacitor for clean shutdown on power loss) -- Greg Smith 2ndQuadrant USg...@2ndquadrant.com Baltimore, MD PostgreSQL Training, Services, and 24x7 Support www

Re: [HACKERS] Page replacement algorithm in buffer cache

2013-04-03 Thread Greg Smith
of these problems available all the time. What I did this week on that front was just go buy a 24 core server with 64GB of RAM that lives in my house. I just need to keep it two floors away if I want to sleep at night. -- Greg Smith 2ndQuadrant USg...@2ndquadrant.com Baltimore, MD PostgreSQL

Re: [HACKERS] Patch for checking file parameters to psql before password prompt

2013-04-03 Thread Greg Smith
again later. I'll do the kicking myself now. -- Greg Smith 2ndQuadrant USg...@2ndquadrant.com Baltimore, MD PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http

Re: [HACKERS] Proposal for Allow postgresql.conf values to be changed via SQL [review]

2013-04-02 Thread Greg Smith
to being ready, and I hope you won't get discouraged just because it's probably going to slip to 9.4. -- Greg Smith 2ndQuadrant USg...@2ndquadrant.com Baltimore, MD PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers

Re: [HACKERS] Proposal for Allow postgresql.conf values to be changed via SQL [review]

2013-04-01 Thread Greg Smith
for people who don't use the feature. It's not that much code, but it is going to take a good bit of committer level review to accept due to its risk. -- Greg Smith 2ndQuadrant USg...@2ndquadrant.com Baltimore, MD PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.com -- Sent via

Re: [HACKERS] Page replacement algorithm in buffer cache

2013-03-24 Thread Greg Smith
that any tuning you do for one type of workload might accidentally slow another. Starting with a lot of baseline workloads is the only way to move usefully forward when facing that problem. -- Greg Smith 2ndQuadrant USg...@2ndquadrant.com Baltimore, MD PostgreSQL Training, Services

Re: [HACKERS] Let's invent a function to report lock-wait-blocking PIDs

2013-03-24 Thread Greg Smith
they grabbed in?, that would solve the hardest of the questions I see in the field. -- Greg Smith 2ndQuadrant USg...@2ndquadrant.com Baltimore, MD PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make

Re: [HACKERS] Limiting setting of hint bits by read-only queries; vacuum_delay

2013-03-24 Thread Greg Smith
tongue about the design on all these, and return to working on one of the patches already in the CF queue instead. -- Greg Smith 2ndQuadrant USg...@2ndquadrant.com Baltimore, MD PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list

Re: [HACKERS] Proposal for Allow postgresql.conf values to be changed via SQL [review]

2013-03-21 Thread Greg Smith
nothing changed. It should be as obvious as we can make it to someone that the explicit reload is necessary. -- Greg Smith 2ndQuadrant USg...@2ndquadrant.com Baltimore, MD PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql

Re: [HACKERS] Proposal for Allow postgresql.conf values to be changed via SQL [review]

2013-03-21 Thread Greg Smith
the change. Maybe that's a NOTICE plus a HINT. -- Greg Smith 2ndQuadrant USg...@2ndquadrant.com Baltimore, MD PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription

Re: [HACKERS] Enabling Checksums

2013-03-19 Thread Greg Smith
. -- Greg Smith 2ndQuadrant USg...@2ndquadrant.com Baltimore, MD PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Enabling Checksums

2013-03-19 Thread Greg Smith
to be a multiple of 1K is ever going to be an unreasonable limitation, if that's what it takes to get useful hardware acceleration. -- Greg Smith 2ndQuadrant USg...@2ndquadrant.com Baltimore, MD PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list

Re: [HACKERS] Enabling Checksums

2013-03-19 Thread Greg Smith
this efficiently toward the zlib developers, let them drop into assembly to get the best possible Intel acceleration etc. one day. -- Greg Smith 2ndQuadrant USg...@2ndquadrant.com Baltimore, MD PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list

Re: [HACKERS] Enabling Checksums

2013-03-19 Thread Greg Smith
CRC32, the zlib CRC32, and the Intel-derived versions Ants hacked on to compare. -- Greg Smith 2ndQuadrant USg...@2ndquadrant.com Baltimore, MD PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org

Re: [HACKERS] Enabling Checksums

2013-03-19 Thread Greg Smith
this committed with a safe option and then y'all can discuss the fine merits of each algorithm at leisure. Yes, that's what we're already doing--it just looks like work :) -- Greg Smith 2ndQuadrant USg...@2ndquadrant.com Baltimore, MD PostgreSQL Training, Services, and 24x7 Support www

Re: [HACKERS] Enabling Checksums

2013-03-19 Thread Greg Smith
itching to assemble a 24 core AMD box at home anyway, this gives me an excuse to pull the trigger on that. Thanks for the summary of how you view the zlib/libpng project state. I saw 4 releases from zlib in 2012, so it seemed possible development might still move forward there. -- Greg Smith

Re: [HACKERS] Enabling Checksums

2013-03-18 Thread Greg Smith
and manually fixed them, basically go on a hunt for torn pages and the last known good copy via full-page write. Without checksums, there really was nowhere to go with them except dump/reload. -- Greg Smith 2ndQuadrant USg...@2ndquadrant.com Baltimore, MD PostgreSQL Training, Services, and 24x7

Re: [HACKERS] Enabling Checksums

2013-03-18 Thread Greg Smith
summarizing how I use them for that tomorrow, just out of juice for that tonight. -- Greg Smith 2ndQuadrant USg...@2ndquadrant.com Baltimore, MD PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org

Re: [HACKERS] Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

2013-03-17 Thread Greg Smith
management. Just because that change introduces an entire directory being scanned, I don't see that as an excuse to clutter it with a long list of files too. -- Greg Smith 2ndQuadrant USg...@2ndquadrant.com Baltimore, MD PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.com

Re: [HACKERS] Enabling Checksums

2013-03-17 Thread Greg Smith
like to do that with a CRC16 implementation or two. I'm not sure if it's possible to get a quicker implementation because the target is a CRC16, or whether it's useful to consider truncating a CRC32 into a CRC16. -- Greg Smith 2ndQuadrant USg...@2ndquadrant.com Baltimore, MD PostgreSQL

Re: [HACKERS] Enabling Checksums

2013-03-17 Thread Greg Smith
generate during (and beyond) beta. I have my own stress tests I'll keep running too. If the bug rate from the beta adopters is bad and doesn't improve, there's is always the uncomfortable possibility of reverting it before the first RC. -- Greg Smith 2ndQuadrant USg...@2ndquadrant.com

[HACKERS] Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

2013-03-12 Thread Greg Smith
the perspective of what a useful minimal commit looks like, I would think 90% of the use cases here expect immediate reload of just the postgresql.conf--but probably not pg_hba.conf--and any other refinement could wait until a next release. -- Greg Smith 2ndQuadrant USg...@2ndquadrant.com

Re: [HACKERS] Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

2013-03-11 Thread Greg Smith
to be the slowest way, too, that is acceptable. -- Greg Smith 2ndQuadrant USg...@2ndquadrant.com Baltimore, MD PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http

Re: [HACKERS] Btrfs clone WIP patch

2013-03-11 Thread Greg Smith
likely to be accepted if they're demonstrated to work on two OSes. That's much better evidence that the API for the OS specific work is a good one. -- Greg Smith 2ndQuadrant USg...@2ndquadrant.com Baltimore, MD PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.com

Re: [HACKERS] Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

2013-03-10 Thread Greg Smith
*also* happening at server start. But I could understand that other people might not like that. And having this pop up on every reload, appearing to a client next to another statement altogether, that isn't acceptable though. -- Greg Smith 2ndQuadrant USg...@2ndquadrant.com Baltimore

Re: [HACKERS] Btrfs clone WIP patch

2013-03-09 Thread Greg Smith
Regardless, you need to keep most of the structure to copydir anyway. Error handling, handling cancellation, and fsync calls are all vital things. You probably have to make the forked command copy a single file at a time to get the same interrupt handling behavior. -- Greg Smith 2ndQuadrant US

Re: [HACKERS] Enabling Checksums

2013-03-08 Thread Greg Smith
in the database will be valuable. -- Greg Smith 2ndQuadrant USg...@2ndquadrant.com Baltimore, MD PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http

Re: [HACKERS] Request for vote to move forward with recovery.conf overhaul

2013-03-08 Thread Greg Smith
back to facing bit rot and remembering what was going on again, which is what killed the momentum toward committing this the last time. -- Greg Smith 2ndQuadrant USg...@2ndquadrant.com Baltimore, MD PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.com -- Sent via pgsql

Re: [HACKERS] Enabling Checksums

2013-03-06 Thread Greg Smith
Linux with filesystem choices skewed toward conservative. Forget about the leading edge--I'd be happy if I could get one large customer to migrate off of ext3... -- Greg Smith 2ndQuadrant USg...@2ndquadrant.com Baltimore, MD PostgreSQL Training, Services, and 24x7 Support www

Re: [HACKERS] Enabling Checksums

2013-03-06 Thread Greg Smith
, a moderately large lookup table. I don't see that there's any advantage to having all that baggage around if you're just going to throw away half of the result anyway. More on CRC32Cs in my next message. -- Greg Smith 2ndQuadrant USg...@2ndquadrant.com Baltimore, MD PostgreSQL

Re: [HACKERS] Enabling Checksums

2013-03-06 Thread Greg Smith
as strongly though, as the timetime before we can expect a hardware accelerated version to be available is much further off than a Linux kernel developer's future. -- Greg Smith 2ndQuadrant USg...@2ndquadrant.com Baltimore, MD PostgreSQL Training, Services, and 24x7 Support www

Re: [HACKERS] Enabling Checksums

2013-03-06 Thread Greg Smith
were encountered here though. -- Greg Smith 2ndQuadrant USg...@2ndquadrant.com Baltimore, MD PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.com attachment: clients-sets.pngattachment: scaling-sets.png Checksum-pgbench.xls Description: MS-Excel spreadsheet -- Sent via

Re: [HACKERS] Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

2013-03-06 Thread Greg Smith
beats me to it. I would find it very useful to me personally if this feature were committed, and we know there's plenty of user demand for it too. -- Greg Smith 2ndQuadrant USg...@2ndquadrant.com Baltimore, MD PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.com -- Sent

Re: [HACKERS] Enabling Checksums

2013-03-06 Thread Greg Smith
to be a real average slowdown of 6%. I think way more deployments are going to be like that case, which matches most of my pgbench runs, than the worse case workloads. -- Greg Smith 2ndQuadrant USg...@2ndquadrant.com Baltimore, MD PostgreSQL Training, Services, and 24x7 Support www

Re: [HACKERS] Enabling Checksums

2013-03-04 Thread Greg Smith
on database filesystem mechanics perfectly. -- Greg Smith 2ndQuadrant USg...@2ndquadrant.com Baltimore, MD PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription

Re: [HACKERS] Enabling Checksums

2013-03-04 Thread Greg Smith
almost invariably do. If I didn't see being able to make noise about the improved reliability of PostgreSQL as valuable enough to consider it anyway, I wouldn't even be working on this thing. -- Greg Smith 2ndQuadrant USg...@2ndquadrant.com Baltimore, MD PostgreSQL Training, Services

Re: [HACKERS] Enabling Checksums

2013-03-03 Thread Greg Smith
And here's an updated version of the checksum corruption testing wrapper script already. This includes an additional safety check that you've set PGDATA to a location that can be erased. Presumably no one else would like to accidentally do this: rm -rf /* Like I just did. -- Greg Smith

Re: [HACKERS] Enabling Checksums

2013-03-03 Thread Greg Smith
hitting one of them. -- Greg Smith 2ndQuadrant USg...@2ndquadrant.com Baltimore, MD PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.com ChecksumPerfResults.xls Description: MS-Excel spreadsheet # Configuration for testing checksum performance. # # Primary goals

Re: [HACKERS] buffer assertion tripping under repeat pgbench load

2013-03-03 Thread Greg Smith
message. To keep the output from a test like that from being completely overwhelming, I also set: shared_buffers = 16 Which is its minimum value. -- Greg Smith 2ndQuadrant USg...@2ndquadrant.com Baltimore, MD PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.com diff --git

Re: [HACKERS] Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

2013-03-03 Thread Greg Smith
pg_reload_conf(); + select pg_sleep(1); Why the pause? There are some regression tests with sleeps in them, such as the stats collector and the timezone tests. I would expect pg_reload_conf would not return until its configuration is stable, so this is a bit of a surprise. -- Greg Smith

Re: [HACKERS] Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

2013-03-03 Thread Greg Smith
the file more likely to be processed before other things in that directory. I could live with that order too. I think it is important to name the file such that it suggests being connected to the SET PERSISTENT feature though. -- Greg Smith 2ndQuadrant USg...@2ndquadrant.com Baltimore, MD

Re: [HACKERS] Suggested new CF status: Pending Discussion

2013-03-03 Thread Greg Smith
point. This should probably move to Waiting on Author when it happens, presuming that the person who wrote something is motivated to see the change committed. (If they weren't, why did they write it?) -- Greg Smith 2ndQuadrant USg...@2ndquadrant.com Baltimore, MD PostgreSQL Training

Re: [HACKERS] Request for vote to move forward with recovery.conf overhaul

2013-03-03 Thread Greg Smith
recovery_end_command available for that. Let's say you wanted to force the old name and did include_if_exists config/recovery.conf. Now you could do: recovery_end_command = 'rm -f /tmp/pgsql.trigger.5432 mv conf.d/recovery.conf conf.d/recovery.done' -- Greg Smith 2ndQuadrant USg...@2ndquadrant.com

Re: [HACKERS] Enabling Checksums

2013-03-03 Thread Greg Smith
that though? Perhaps unvalidated only be allowed in a read-only transaction? -- Greg Smith 2ndQuadrant USg...@2ndquadrant.com Baltimore, MD PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make

Re: [HACKERS] Enabling Checksums

2013-03-03 Thread Greg Smith
(but feasible) situation you're describing. It would need to be something stronger than a GUC. Once you start writing out pages without checksums, you're back into the fuzzy state where some pages have them, others don't, and there's no good way to deal with that yet. -- Greg Smith 2ndQuadrant

[HACKERS] initdb ignoring options?

2013-02-26 Thread Greg Smith
was the code refactoring of http://www.postgresql.org/message-id/e1teyox-0005w4...@gemulon.postgresql.org (there were a few additional initdb commits around then too) -- Greg Smith 2ndQuadrant USg...@2ndquadrant.com Baltimore, MD PostgreSQL Training, Services, and 24x7 Support www

Re: [HACKERS] initdb ignoring options?

2013-02-26 Thread Greg Smith
On 2/26/13 5:51 PM, Mark Kirkwood wrote: So looks like something odd you are doing - are you using any unusual build options? The unusual part looks to be the build environment or libraries of this Mac I'm trying to use. The build options are the normal boring set: CONFIGURE =

Re: [HACKERS] initdb ignoring options?

2013-02-26 Thread Greg Smith
/gsmith/pgwork/inst/latest/bin/initdb $ initdb --version Database cluster already exists in /Users/gsmith/pgwork/data/latest -- Greg Smith 2ndQuadrant USg...@2ndquadrant.com Baltimore, MD PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.com -- Sent via pgsql-hackers mailing

Re: [HACKERS] initdb ignoring options due to bash environment on OS X

2013-02-26 Thread Greg Smith
.dylib dyld: loaded: /usr/lib/libSystem.B.dylib dyld: loaded: /usr/lib/system/libcache.dylib ... Making me think about the /bin/sh environment. Looking into the diff from the working/not working output of set env.txt figured out what was going on. -- Greg Smith 2ndQuadrant USg

Re: [HACKERS] initdb ignoring options due to bash environment on OS X

2013-02-26 Thread Greg Smith
On 2/26/13 9:18 PM, Greg Smith wrote: $ otool -L `which initdb` /Users/gsmith/pgwork/inst/latest/bin/initdb: /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 159.1.0) Two last bits of trivia and I'll stop talking to myself. You can get the full list of things

Re: [HACKERS] Enabling Checksums

2013-01-15 Thread Greg Smith
Smith 2ndQuadrant USg...@2ndquadrant.com Baltimore, MD PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.com #!/usr/bin/env python # # pg_corrupt # # Read in a byte of a PostgreSQL relation (a table or index) and allow writing # it back with an altered value. # # Greg Smith g

Re: [HACKERS] Timing events WIP v1

2013-01-14 Thread Greg Smith
on similarity to the generic logging hook that went into 9.2. This new implementation idea reminds me more of the query length decoration needed for normalized pg_stat_statements though--something that wasn't easy to just extract out from the consumer at all. -- Greg Smith 2ndQuadrant USg

Re: [HACKERS] buffer assertion tripping under repeat pgbench load

2013-01-12 Thread Greg Smith
=302295, flags=0x106, refcount=0 1073741824) WARNING: buffers with non-zero refcount is 1 TRAP: FailedAssertion(!(RefCountErrors == 0), File: bufmgr.c, Line: 1712) -- Greg Smith 2ndQuadrant USg...@2ndquadrant.com Baltimore, MD PostgreSQL Training, Services, and 24x7 Support www

Re: [HACKERS] Enabling Checksums

2013-01-12 Thread Greg Smith
it a bit; computing wal bytes / commit seems to work. pgbench-tools also launches vmstat and isstat in a way that it's possible to graph the values later. The interesting results I'm seeing are when the disk is about 80% busy and when it's 100% busy. -- Greg Smith 2ndQuadrant USg

Re: [HACKERS] buffer assertion tripping under repeat pgbench load

2013-01-09 Thread Greg Smith
to make the i variable in AtEOXact_Buffers be a Buffer number. Then you access PrivateRefCount[buffer - 1] in that code, making it look like more of the surrounding references. And the value goes directly into PrintBufferLeakWarning without a problem. -- Greg Smith 2ndQuadrant USg

Re: [HACKERS] buffer assertion tripping under repeat pgbench load

2012-12-29 Thread Greg Smith
On 12/27/12 7:43 AM, Greg Stark wrote: If it's always the first buffer then it could conceivably still be some other heap allocated object that always lands before LocalRefCount. It does seem a bit weird to be storing 130 though -- there are no 130 constants that we might be storing for example.

Re: [HACKERS] buffer assertion tripping under repeat pgbench load

2012-12-26 Thread Greg Smith
after that. Does this need an update or is there just a devel version I should grab? -- Greg Smith 2ndQuadrant USg...@2ndquadrant.com Baltimore, MD PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org

Re: [HACKERS] buffer assertion tripping under repeat pgbench load

2012-12-26 Thread Greg Smith
if there was at least one refcounted buffer? I've been doing that for a while, the only thing that's new is getting the block number. It's only ever been one buffer involved. -- Greg Smith 2ndQuadrant USg...@2ndquadrant.com Baltimore, MD PostgreSQL Training, Services, and 24x7 Support

Re: [HACKERS] buffer assertion tripping under repeat pgbench load

2012-12-26 Thread Greg Smith
system here. If I can reproduce this on a second system, I'll push all the details out to the list, hopeful that other people might see it too. -- Greg Smith 2ndQuadrant USg...@2ndquadrant.com Baltimore, MD PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.com -- Sent via

Re: [HACKERS] buffer assertion tripping under repeat pgbench load

2012-12-26 Thread Greg Smith
this is an obscure bug rather than something with my hardware or install. The CPU is an Intel i7 870 using HyperThreading to simulate 8 cores, 16GB of RAM. The database is on a single regular SATA drive. I simplified the storage setup before I started public reports. -- Greg Smith 2ndQuadrant US

Re: [HACKERS] buffer assertion tripping under repeat pgbench load

2012-12-24 Thread Greg Smith
this be an off by one error in one of those, where the wrong form was used? -- Greg Smith 2ndQuadrant USg...@2ndquadrant.com Baltimore, MD PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes

Re: [HACKERS] buffer assertion tripping under repeat pgbench load

2012-12-24 Thread Greg Smith
by tomorrow at the latest. The range of runtime before crash is 3 to 14 hours so far. -- Greg Smith 2ndQuadrant USg...@2ndquadrant.com Baltimore, MD PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org

Re: [HACKERS] buffer assertion tripping under repeat pgbench load

2012-12-23 Thread Greg Smith
On 12/23/12 1:10 PM, Tom Lane wrote: It might also be interesting to know if there is more than one still-pinned buffer --- that is, if you're going to hack the code, fix it to elog(LOG) each pinned buffer and then panic after completing the loop. Easy enough; I kept it so the actual source

Re: [HACKERS] buffer assertion tripping under repeat pgbench load

2012-12-23 Thread Greg Smith
intermittent that I'm trying not to change anything until I get more productive output out of one. The last run I kicked off has been chugging for 14 hours without a hiccup yet. -- Greg Smith 2ndQuadrant USg...@2ndquadrant.com Baltimore, MD PostgreSQL Training, Services, and 24x7 Support www

[HACKERS] buffer assertion tripping under repeat pgbench load

2012-12-22 Thread Greg Smith
goes away. Other ideas are welcome. -- Greg Smith 2ndQuadrant USg...@2ndquadrant.com Baltimore, MD PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http

Re: [HACKERS] Enabling Checksums

2012-12-18 Thread Greg Smith
just that--the first one, to be joined by others soon afterward. My experience at how drives fail says the second error is a lot more likely after you've seen one. -- Greg Smith 2ndQuadrant USg...@2ndquadrant.com Baltimore, MD PostgreSQL Training, Services, and 24x7 Support www

Re: [HACKERS] Enabling Checksums

2012-12-14 Thread Greg Smith
nailed down yet. Building a corruption testing harness is going to involve a lot of creating new clusters and test data to torture. It's a different style of problem than injecting faults in the first place. -- Greg Smith 2ndQuadrant USg...@2ndquadrant.com Baltimore, MD PostgreSQL Training

Re: [HACKERS] Enabling Checksums

2012-12-12 Thread Greg Smith
. The randomness seed should be printed as part of the output, so that it's possible re-create the damage exactly later. If the server doesn't handle it correctly, we'll want to be able to replicate the condition it choked on exactly later, just based on the tool's log output. Any other requests? -- Greg

Re: [HACKERS] [WIP PATCH] for Performance Improvement in Buffer Management

2012-12-11 Thread Greg Smith
running an init/test pair will often create. You don't need pg_prewarm to do it. If you initialize the database, then restart the server to clear shared_buffers, the result will be similar to what you're doing. -- Greg Smith 2ndQuadrant USg...@2ndquadrant.com Baltimore, MD PostgreSQL

Re: [HACKERS] Timing events WIP v1

2012-12-11 Thread Greg Smith
once it's identified. I can't claim any good expertise on client encoding/transfer issues though; that's not a strong area for me. -- Greg Smith 2ndQuadrant USg...@2ndquadrant.com Baltimore, MD PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.com -- Sent via pgsql-hackers

Re: [HACKERS] WIP patch for hint bit i/o mitigation

2012-11-21 Thread Greg Smith
with this one quite a bit. I'd suggest reviewers here focus on code quality, correctness, and targeted optimization testing. I'm working heavily on a larger benchmarking framework that both this and checksums will fit into right now. -- Greg Smith 2ndQuadrant USg...@2ndquadrant.com

Re: [HACKERS] auto_explain WAS: RFC: Timing Events

2012-11-21 Thread Greg Smith
going and accept the loss. -- Greg Smith 2ndQuadrant USg...@2ndquadrant.com Baltimore, MD PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org

Re: [HACKERS] Timing events WIP v1

2012-11-20 Thread Greg Smith
. -- Greg Smith 2ndQuadrant USg...@2ndquadrant.com Baltimore, MD PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql

Re: [HACKERS] Doc patch making firm recommendation for setting the value of commit_delay

2012-11-15 Thread Greg Smith
. -- Greg Smith 2ndQuadrant USg...@2ndquadrant.com Baltimore, MD PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

[HACKERS] Timing events WIP v1

2012-11-15 Thread Greg Smith
a moment and lookup the serial number of the last record I should display. Then I could drop to a share lock looping over the elements, stopping if I find one with a serial number over that. -- Greg Smith 2ndQuadrant USg...@2ndquadrant.com Baltimore, MD PostgreSQL Training, Services

Re: [HACKERS] Materialized views WIP patch

2012-11-15 Thread Greg Smith
isn't implemented yet and the author never tested that. Having some known good samples you have tested, even if they're not proper regression tests, would be helpful for establishing the code baseline works. -- Greg Smith 2ndQuadrant USg...@2ndquadrant.com Baltimore, MD PostgreSQL

Re: [HACKERS] Timing events WIP v1

2012-11-15 Thread Greg Smith
mean it needs to pull into core. If there's a better approach for adding a hook that avoids that I'd be happy to adopt it, I just haven't thought of one. -- Greg Smith 2ndQuadrant USg...@2ndquadrant.com Baltimore, MD PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.com

Re: [HACKERS] Enabling Checksums

2012-11-12 Thread Greg Smith
the potentially suspicious write to a new place. -- Greg Smith 2ndQuadrant USg...@2ndquadrant.com Baltimore, MD PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http

Re: [HACKERS] Enabling Checksums

2012-11-12 Thread Greg Smith
for the checksum conversion, that would be very helpful to larger sites. It doesn't matter if adding checksums to the old data takes a week if you throttle the load down, so long as you're not forced to hold an open snapshot the whole time. -- Greg Smith 2ndQuadrant USg...@2ndquadrant.com

Re: [HACKERS] Enabling Checksums

2012-11-11 Thread Greg Smith
see. As you say, you don't ever gain the ability to tell pages that were checksummed but have since been corrupted from ones that were corrupt all along in that path. -- Greg Smith 2ndQuadrant USg...@2ndquadrant.com Baltimore, MD PostgreSQL Training, Services, and 24x7 Support www

Re: [HACKERS] WIP checksums patch

2012-11-11 Thread Greg Smith
if they're available on a given database/table or not. The sort of text I'm thinking of for the manual and/or warning is: You can't use pg_upgrade to migrate from a cluster where checksums are not available to one where they are. This limitation may be removed by a future version. -- Greg

Re: [HACKERS] Enabling Checksums

2012-11-11 Thread Greg Smith
that, and allow coordinating the work with the challenges of vacuuming larger systems too. -- Greg Smith 2ndQuadrant USg...@2ndquadrant.com Baltimore, MD PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers

Re: [HACKERS] Proposal for Allow postgresql.conf values to be changed via SQL

2012-11-11 Thread Greg Smith
file locking comes into play. I wouldn't even bother aiming for that part in a first patch. It's obviously a solvable problem in a number of ways. You need a rock solid way to figure out what to write there before that solution is useful though. -- Greg Smith 2ndQuadrant USg

Re: [HACKERS] Proposal for Allow postgresql.conf values to be changed via SQL

2012-11-07 Thread Greg Smith
-xr-x postgres postgres /etc/postgresql/8.4/main/ $ ls -ld /etc/postgresql/9.1/main/postgresql.conf -rw-r--r-- postgres postgres /etc/postgresql/9.1/main/postgresql.conf $ ls -ld /etc/postgresql/8.4/main/postgresql.conf -rw-r--r-- postgres postgres /etc/postgresql/8.4/main/postgresql.conf -- Greg

[HACKERS] RFC: Timing Events

2012-11-01 Thread Greg Smith
that, you can certainly afford to write a copy to memory instead, and probably in addition too. -- Greg Smith 2ndQuadrant USg...@2ndquadrant.com Baltimore, MD PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers

Re: [HACKERS] RFC: Timing Events

2012-11-01 Thread Greg Smith
on. -- Greg Smith 2ndQuadrant USg...@2ndquadrant.com Baltimore, MD PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Proposal for Allow postgresql.conf values to be changed via SQL

2012-11-01 Thread Greg Smith
to revive usefully now. And as much as I hate to expand scope by saying it, both changes should probably be tackled at once. It's important to make sure they're both solved well by whatever is adopted, they are going to co-exist as committed one day. -- Greg Smith 2ndQuadrant USg

Re: [HACKERS] [PATCH] Prefetch index pages for B-Tree index scans

2012-11-01 Thread Greg Smith
on the io and can't issue a read as long as the first process has the buffer locked and io in progress. I think aio makes a lot more sense if you're using threads so you can start a thread to wait for the io to complete. -- Greg Smith 2ndQuadrant USg...@2ndquadrant.com Baltimore, MD

Re: [HACKERS] Re: [COMMITTERS] pgsql: Revert commit_delay change; just add comment that we don't hav

2012-09-05 Thread Greg Smith
get even more useful than before. -- Greg Smith 2ndQuadrant USg...@2ndquadrant.com Baltimore, MD PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http

Re: [HACKERS] Timing overhead and Linux clock sources

2012-08-27 Thread Greg Smith
, and that's not where people should focus first. -- Greg Smith 2ndQuadrant USg...@2ndquadrant.com Baltimore, MD PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http

<    1   2   3   4   5   6   7   8   9   10   >