as well, at least enough to
cover this particular issue.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
--
Sent via pgsql-patches mailing list (pgsql-patches@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-patches
then,
then runs VACCUM ANALYZE. There is no notion of what script you'll end up
executing yet. If you have a truly custom script that works against other
data instead of the pgbench tables, you won't even be executing this
initialization bit.
--
* Greg Smith [EMAIL PROTECTED] http
a round of portability concerns,
too.
Where did Marc's patch come from? I'd like to be able to separate out
that change from the rest if necessary.
Also, if you have any specific test cases you ran that I could start by
trying to replicate a speedup on, those would be handy as well.
--
* Greg
to me how to test the existing bitmap scan
patch, yours seems a more straightforward patch to use as a testing ground
for fadvise effectiveness.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
--
Sent via pgsql-patches mailing list (pgsql-patches@postgresql.org)
To make
On Wed, 7 May 2008, Bruce Momjian wrote:
Tom Lane wrote:
Greg Smith [EMAIL PROTECTED] writes:
The way the option parsing code is done would make complaining in the case
where your parameter is ignored a bit of a contortion.
Yeah. But couldn't we have that part issue a warning if -s had
now that
I look at this again.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
--
Sent via pgsql-patches mailing list (pgsql-patches@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-patches
the unprocessed part of the queue always
shrinking, and as long as people know that they can get it reconsidered by
submitting new results it's not unfair to them.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
--
Sent via pgsql-patches mailing list (pgsql-patches@postgresql.org
stuck sorting out a number of OS level issue right now
before my testing system is online again). Was planning to take a longer
look at Greg Stark's prefetching work at that point as well.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
--
Sent via pgsql-patches mailing
be for you to bounce this into the
rejected pile as Returned for testing immediately, to clearly remove it
from the main queue. A reasonable expectation there is that you might
consider it again during May if someone gets back with said testing
results before the 'fest ends.
--
* Greg Smith
as
much of a penalty for writing that way because lining up with the spinning
disk isn't important. Someone who put one of DB/WAL on SSD and the other
on traditional disk might end up with very different DB/WAL block sizes to
match.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com
and packed into an integer
I personally don't feel it's worth making the code any more complicated
than it needs to be just to save a fraction of a percent of the total
memory used by the buffer pool.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
--
Sent via pgsql
structure (5 ints). Let's call it 40
bytes; even that's only a 0.5% overhead relative to the shared buffer
allocation. If we can speed checkpoints significantly with that much
overhead it sounds like a good tradeoff to me.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
, and that wasn't being used. Given
some of the other corruption found later and the bad memory issues
discovered, a bit flipping in the do I need to checkpoint now? code or
data seems just as likely as any other explanation.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
. That's the theory at
least.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
--
Sent via pgsql-patches mailing list (pgsql-patches@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-patches
be good enough to trigger an alert
rather than needing a real monitoring probe.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
--
Sent via pgsql-patches mailing list (pgsql-patches@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref
the checkpoints are doing there. I
don't see why that sort of stuff should go into core when it's now easy to
do outside of it. I have a whole stack of scripts in that area I plan to
release over the summer.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
--
Sent via pgsql-patches
they do still vary even in 8.3. I'm just not sure if
the current methods available for that really aren't good enough, or if
it's just the case that not everyone is aware of all of them.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
--
Sent via pgsql-patches mailing list
places people would like to look at.
I will apologize now for suggesting this, followed by not having enough
time to code it in the near future.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
--
Sent via pgsql-patches mailing list (pgsql-patches@postgresql.org)
To make
that shows up with 8 escapes per line.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
--
Sent via pgsql-patches mailing list (pgsql-patches@postgresql.org)
To make changes to your subscription:
http://mail.postgresql.org/mj/mj_wwwusr?domain=postgresql.orgextra=pgsql
of a SATA drive that had its write cache disabled by
default. They're always on unless you force them off, and even then they
can turn themselves back on again if there's a device reset and you didn't
change the drive's default using the manufacturer's adjustment utility.
--
* Greg Smith
delete if I'm understanding this correctly.
I left behind the link I was just playing with and I'll see if I can get
tmpwatch to eat it tomorrow, that seems like the most appropriate test
here.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
want to use -S (sort memory) or -f (forbid
scan/join types). If I can get someone to clarify what is supported there
I can put together a pgbench doc patch that addresses this topic.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
---(end
documentation is less clear
than it could be on the subject of what you can usefully put into
PGOPTIONS. That's two small documentation patches I should be able to
handle.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
---(end of broadcast
for bgwriter_delay feasible. None of the tunables
*should* have to be adjusted if you need to run the bgwriter much more
often to support some new HOT-related activity in there as well; this is
actually my next area I wanted to do a bit more testing on myself.
--
* Greg Smith [EMAIL PROTECTED
.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
---(end of broadcast)---
TIP 4: Have you searched our list archives?
http://archives.postgresql.org
)
...
executed INSERT in 12.212027 sec (2nd)
results-patch:
...
executed DELETE in 18.062664 sec
executed VACUUM in 28.487570 sec
executed INSERT in 25.638022 sec (1st)
...
executed INSERT in 40.759404 sec (2nd)
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
how to count
some of what it does; I marked the section I'm concerned about with an XXX
comment.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
buf-alloc-stats-2.patch
Description: Binary data
---(end of broadcast
was done against 8.2 and then ported
forward). If that already shows clearly the start and end of each
autovacuum section, ignore that I even brought this up.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
---(end of broadcast
appreciate them as well.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
---(end of broadcast)---
TIP 5: don't forget to increase your free space map settings
checkpoint which doesn't seem to recycle as
many segments usefully
Any change that would be more proactive about creating segments in these
situations than the current code would be a benefit, even though these are
not common paths people encounter.
--
* Greg Smith [EMAIL PROTECTED] http
along. I'll add this idea
to my list of things that would be nice to have as part of a larger
rewriter, I think it's more trouble than it's worth to chase right now.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
---(end of broadcast
cases floating around this area is what makes me feel that removing it
altogether is still a bit premature.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
---(end of broadcast)---
TIP 6: explain analyze is your friend
this way right now though, and
the fact that LDC gives you a parameter to aim this particular foot-gun
more precisely isn't a big deal IMHO as long as the documentation is
clear.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
---(end
like it to be the case that one day the database's I/O
scheduling would eventually get to those, in order to optimize performance
in the kind of bursty scenarios I've been mentioning lately.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
---(end
like to see this get committed relatively soon
because there's two interleaved merges stuck behind this one (the more
verbose logging patch and the LRU modifications).
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
---(end of broadcast
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
---(end of broadcast)---
TIP 3: Have you checked our extensive FAQ?
http://www.postgresql.org/docs/faq
to get expensive CPU-wise.
In addition to being a CPU pig, that still won't necessarily work because
the way the LRU writer ignores things with a non-zero usage_count. If
it's dirty, it's probably been used recently as well.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore
to the other types of transactions; having more
of those seemed to aggrevate my LDC-related issues because they leave a
different pattern of dirty+used buffers around than other operations.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
---(end
is extrapolation.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
---(end of broadcast)---
TIP 4: Have you searched our list archives?
http://archives.postgresql.org
to disillusion you :-(
It's having been on the painful receiving end of that fact that makes me
so paranoid now :)
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
---(end of broadcast)---
TIP 7: You can help support
it up, decrease
checkpoint_segments, and get used to having more checkpoints if you have
to keep the same recovery time.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
---(end of broadcast)---
TIP 6: explain analyze is your
sync writes get mixed with buffered WAL
writes seems to cause some weird I/O scheduling issues in Linux that can
make worst-case latency degrade. But since I can't prove that, I guess I
might as well not even mention that either.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com
path instead.
The checkpoint logging patch I submitted logs when this happens
specifically because that particular issue messed with some operations and
I found it important to be aware of.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
the LDC can
be spread out a bit but not across the entire interval, that makes it
easier to control how much expansion there is relative to the current
behavior.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
---(end of broadcast
this week on the topic
Controlling Load Distributed Checkpoints, and check out the Automatic
adjustment of bgwriter_lru_maxpages patch whose latest version is at
http://archives.postgresql.org/pgsql-patches/2007-05/msg00142.php
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
On Fri, 1 Jun 2007, Andrew Dunstan wrote:
Greg Smith wrote:
Since the rotation size feature causes other issues anyway
that make importing more complicated, documenting the issue seemed
sufficient.
What are the other issues? I'm not happy about producing files with split
lines.
Just
of an unreasonable situation to go out of your way to accommodate.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
---(end of broadcast)---
TIP 2: Don't 'kill -9' the postmaster
to see that truncated too much.
Anyway, I have a bunch of data on this subject being collected at this
moment, and I'll rescale the results based on what I see after analyzing
that this week.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
applications will have to work with this data, so you can
see that my contrary preference isn't completely random here.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
---(end of broadcast)---
TIP 3: Have you checked our
it's worthless precision or worth capturing.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
---(end of broadcast)---
TIP 9: In versions below 8.0, the planner will ignore your desire to
choose an index scan if your
, so
I'll add another column for the line number to the output. Thanks for
pointing that out, I can finish cleaning up all the functional
implementation work on this patch myself now.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
---(end
of
obvious bugs to re-submit ready by next weekend.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
---(end of broadcast)---
TIP 1: if posting/reading through Usenet, please send an appropriate
subscribe-nomail command
easier to write that if you can say You can safely set
bgwriter_lru_maxpages high because it only writes what it needs to based
on your usage.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
---(end of broadcast)---
TIP 9
/PGBuildFarm-HOWTO.txt
and those aren't quite focused right if the goal is to work on new patches
while keeping in sync with the repository.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
---(end of broadcast)---
TIP 9
that
during a period, they should shrink the delay instead. The kinds of
systems where 1000 isn't high enough for bgwriter_lru_maxpages are going
to be compelled to adjust these parameters anyway for good performance.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
sent
in what I thought was ready to go because I didn't want to hold up
reviewing the bulk of the code over some of these fine details.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
---(end of broadcast)---
TIP 1
/unlocked all in one section of code right now, and I didn't see how
to move any parts of that to the callers without disrupting that clean
interface.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
---(end of broadcast)---
TIP
with no expected functional impact that alternate
approaches might be built on.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MDIndex: src/backend/storage/buffer/bufmgr.c
===
RCS file: /d3/pgsql/cvs/pgsql-local/src
the
other way around is easier to work with. Piecing together log entries is
a pain, splitting them is easy.
If I had to only keep one line out of this, it would be the one with the
summary. It would be nice to have it logged at INFO.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com
information
than all that and you really have very little basis on which to guess how
long it's going to take.
Other operating systems are going to give completely different behavior
here, which of course makes the problem even worse.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com
--it gets real
messy. Good luck drumming up support for all that when the initial
benchmarks suggest it's going to be a big step back.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
---(end of broadcast)---
TIP 5: don't
on the
patches list about a feature useful *right now* in this area.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
---(end of broadcast)---
TIP 6: explain analyze is your friend
of the checkpoint, to
tell when it was bogged down with slow writes versus being held up in sync
for various (possibly fixed with your CheckpointStartLock) issues.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
---(end of broadcast
suggested here) will be sufficient to fix
all of them. Was already on my to-do list to investigate that further.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
---(end of broadcast)---
TIP 7: You can help support
here.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
---(end of broadcast)---
TIP 5: don't forget to increase your free space map settings
to here. Ideally, it would be nice to call something
like WillLog(DEBUG2) and get a boolean back saying whether something at
that log level will be output anywhere; I don't know enough about the
logging code to add such a thing to it myself.
--
* Greg Smith [EMAIL PROTECTED] http
On Thu, 29 Mar 2007, Magnus Hagander wrote:
I've included a couple of more counters per ideas from Greg Smith in his
logging patch.
I just submitted a patch that logs the remaining things of value from my
original that couldn't be derived from the information you're collecting.
Between them
haven't had enough time track all the code paths involved to
prove otherwise. Glad to hear it was never an issue. Doesn't change what
I submitted though.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
---(end of broadcast
in TPS (checkpoints are particularly visible
in the graphs), after further testing I concluded running a VACUUM VERBOSE
and CHECKPOINT in a script afterwards and analyzing the results was more
useful than integrating something into pgbench itself.
--
* Greg Smith [EMAIL PROTECTED] http
be
passing along all that code once I get it ready for other people to use.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MDIndex: README.pg_buffercache
===
RCS file: /projects/cvsroot/pgsql/contrib/pg_buffercache
they could
get evicted, which would take quite a while. My bet is that bufferid gets
expanded from 32 bits before that happens, which would break
pg_buffercache similarly and isn't being checked for either.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
issues even worse on my Linux system. I'm trying to evaluate this fairly.
-This code operates on the assumption you have a good value for the
checkpoint timeout. Have you tested its behavior when checkpoints are
being triggered by checkpoint_segments being reached instead?
--
* Greg Smith
the production system I
mentioned above, they're from my server at home, which is similar to the
system your results came from).
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
---(end of broadcast)---
TIP 1: if posting
logs are be based on my
requirements, which is to include close enough to everything that it might
as well be the whole set, in case I forgot something I find I need later.
The SQL logs are *completely* different from the syslog setup.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com
an installation that can support the
overhead.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
---(end of broadcast)---
TIP 4: Have you searched our list archives?
http://archives.postgresql.org
to log
importing that I'd be happy with any workable improvement.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
---(end of broadcast)---
TIP 6: explain analyze is your friend
just the fields you want into another table.
I would guess this simplifies the patch as well.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
---(end of broadcast)---
TIP 2: Don't 'kill -9' the postmaster
--incorporating all
the feedback I've gotten here as either code changes or additional
documentation--and resubmit in another week or so.
Thanks for the feedback.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
---(end of broadcast
of additional scripts
I'll be releasing shortly that do interesting analysis of this extended
latency data from pgbench (graphs of TPS and latency, that sort of thing),
and I'd hate for that to only be available on 8.3.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MDIndex
in
results.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
---(end of broadcast)---
TIP 6: explain analyze is your friend
80 matches
Mail list logo