Windows as well, at least enough to
cover this particular issue.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
--
Sent via pgsql-patches mailing list (pgsql-patches@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-patches
results were reported on, but so far I don't see
any significant difference after applying the sorted writes patch.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
--
Sent via pgsql-patches mailing list (pgsql-patches@postgresql.org)
To make changes to your subs
then,
then runs VACCUM ANALYZE. There is no notion of what script you'll end up
executing yet. If you have a truly custom script that works against other
data instead of the pgbench tables, you won't even be executing this
initialization bit.
--
* Greg Smith [EMAIL
a less popular API I expect a round of portability concerns,
too.
Where did Marc's patch come from? I'd like to be able to separate out
that change from the rest if necessary.
Also, if you have any specific test cases you ran that I could start by
trying to replicate a speedup on, t
o see how well
it works. It wasn't as clear to me how to test the existing bitmap scan
patch, yours seems a more straightforward patch to use as a testing ground
for fadvise effectiveness.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
--
Sent via pgsql-patches m
On Wed, 7 May 2008, Bruce Momjian wrote:
Tom Lane wrote:
Greg Smith <[EMAIL PROTECTED]> writes:
The way the option parsing code is done would make complaining in the case
where your parameter is ignored a bit of a contortion.
Yeah. But couldn't we have that part issue a warnin
re it should go now that
I look at this again.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
--
Sent via pgsql-patches mailing list (pgsql-patches@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-patches
iginal source to re-open with more
information. That keeps the unprocessed part of the queue always
shrinking, and as long as people know that they can get it reconsidered by
submitting new results it's not unfair to them.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
for this situation might be for you to bounce this into the
rejected pile as "Returned for testing" immediately, to clearly remove it
from the main queue. A reasonable expectation there is that you might
consider it again during May if someone gets back with said testing
results befor
then (I'm stuck sorting out a number of OS level issue right now
before my testing system is online again). Was planning to take a longer
look at Greg Stark's prefetching work at that point as well.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
--
Sent via p
ocks to match the underlying flash better, and you don't pay as
much of a penalty for writing that way because lining up with the spinning
disk isn't important. Someone who put one of DB/WAL on SSD and the other
on traditional disk might end up with very different DB/WAL block sizes t
emory for it, RelFileNode in BufferTag could
be hashed and packed into an integer
I personally don't feel it's worth making the code any more complicated
than it needs to be just to save a fraction of a percent of the total
memory used by the buffer pool.
--
* Greg Smith [EMAIL PRO
BufAndTag is a relatively small structure (5 ints). Let's call it 40
bytes; even that's only a 0.5% overhead relative to the shared buffer
allocation. If we can speed checkpoints significantly with that much
overhead it sounds like a good tradeoff to me.
--
* Greg Smith [EMAIL PROTEC
collected some actual useful information here. That's the theory at
least.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
--
Sent via pgsql-patches mailing list (pgsql-patches@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-patches
27;t working anymore, and that wasn't being used. Given
some of the other corruption found later and the bad memory issues
discovered, a bit flipping in the "do I need to checkpoint now?" code or
data seems just as likely as any other explanation.
--
* Greg Smith [EMAIL PROTECTED
then start sprinkling exports of those probe points in some popular
places people would like to look at.
I will apologize now for suggesting this, followed by not having enough
time to code it in the near future.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
--
S
cause they do still vary even in 8.3. I'm just not sure if
the current methods available for that really aren't good enough, or if
it's just the case that not everyone is aware of all of them.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
--
Sent via p
table so
you can compute statistics about how the checkpoints are doing there. I
don't see why that sort of stuff should go into core when it's now easy to
do outside of it. I have a whole stack of scripts in that area I plan to
release over the summer.
--
* Greg Smith [EMAIL PROTECTED] http:
t easier to answer with a simple command which has some
value; a little SQL in a cron job would be good enough to trigger an alert
rather than needing a real monitoring probe.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
--
Sent via pgsql-patches mailing list (pgs
gression
that shows up with >8 escapes per line.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
--
Sent via pgsql-patches mailing list (pgsql-patches@postgresql.org)
To make changes to your subscription:
http://mail.postgresql.org/mj/mj_wwwusr?domain=postgresql.org&
disk cache.
I've never heard of a SATA drive that had its write cache disabled by
default. They're always on unless you force them off, and even then they
can turn themselves back on again if there's a device reset and you didn't
change the drive's default using t
d ever find a short symlink
it can delete if I'm understanding this correctly.
I left behind the link I was just playing with and I'll see if I can get
tmpwatch to eat it tomorrow, that seems like the most appropriate test
here.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmit
nch will use
PGOPTIONS. In addition to that, the current documentation is less clear
than it could be on the subject of what you can usefully put into
PGOPTIONS. That's two small documentation patches I should be able to
handle.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.
n imagine
pgbench users might also want to use -S (sort memory) or -f (forbid
scan/join types). If I can get someone to clarify what is supported there
I can put together a pgbench doc patch that addresses this topic.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
for bgwriter_delay feasible. None of the tunables
*should* have to be adjusted if you need to run the bgwriter much more
often to support some new HOT-related activity in there as well; this is
actually my next area I wanted to do a bit more testing on myself.
--
* Greg Smith [EMAIL PROTECTED
tinct walwriter.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
---(end of broadcast)---
TIP 4: Have you searched our list archives?
http://archives.postgresql.org
ssume this is a typo on Benford's Law:
http://en.wikipedia.org/wiki/Benford's_law which notes there are far more
ones in real-world data sets.
If there were a Benfold's Law, it would surely involve the number 5
instead.
--
* Greg Smith [EMAIL PROTECTED] http://www.gr
ually trying to draw
conclusions from was probably real enough.
Agreed, just figuring out the test ramp-up situation, and your explanation
for that quirk sounds reasonable.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
---(end of broadcast)-
2.449248 sec (1st)
...
executed INSERT in 12.212027 sec (2nd)
results-patch:
...
executed DELETE in 18.062664 sec
executed VACUUM in 28.487570 sec
executed INSERT in 25.638022 sec (1st)
...
executed INSERT in 40.759404 sec (2nd)
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmi
me.
3) The recent strategy changes in freelist.c left me unsure how to count
some of what it does; I marked the section I'm concerned about with an XXX
comment.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
buf-alloc-stats-2.patch
(most of this patch was done against 8.2 and then ported
forward). If that already shows clearly the start and end of each
autovacuum section, ignore that I even brought this up.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
(int) pid,
port->sock)));
Little stuff, but all things I've found valuable on several occasions,
which suggests eventually someone else may appreciate them as well.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
---(end of broadcast)---
TIP 5: don't forget to increase your free space map settings
ts are built back up again
3) You ran an early manual checkpoint which doesn't seem to recycle as
many segments usefully
Any change that would be more proactive about creating segments in these
situations than the current code would be a benefit, even though these are
not common path
they're
illuminated. I'd actually like to see this get committed relatively soon
because there's two interleaved merges stuck behind this one (the more
verbose logging patch and the LRU modifications).
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore
nt process. That's OK for
now, but I'd like it to be the case that one day the database's I/O
scheduling would eventually get to those, in order to optimize performance
in the kind of bursty scenarios I've been mentioning lately.
--
* Greg Smith [EMAIL PROTECTED] http://www.gre
ill yourself exactly this way right now though, and
the fact that LDC gives you a parameter to aim this particular foot-gun
more precisely isn't a big deal IMHO as long as the documentation is
clear.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
-
hat there are corner
cases floating around this area is what makes me feel that removing it
altogether is still a bit premature.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
---(end of broadcast)---
TIP 6: explain analyze is your friend
will have moved along. I'll add this idea
to my list of things that would be nice to have as part of a larger
rewriter, I think it's more trouble than it's worth to chase right now.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
--
ore UPDATEs relative to the other types of transactions; having more
of those seemed to aggrevate my LDC-related issues because they leave a
different pattern of dirty+used buffers around than other operations.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
---
ts to get expensive CPU-wise.
In addition to being a CPU pig, that still won't necessarily work because
the way the LRU writer ignores things with a non-zero usage_count. If
it's dirty, it's probably been used recently as well.
--
* Greg Smith [EMAIL PROTECTED] http://www.gre
my favorite web
sites: http://www.channel101.com/shows/show.php?show_id=152
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
---(end of broadcast)---
TIP 3: Have you checked our extensive FAQ?
http://www.postgresql.org/docs/faq
you think that the current code had enormous amounts of testing
before it went in, I've got to disillusion you :-(
It's having been on the painful receiving end of that fact that makes me
so paranoid now :)
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
t the old behavior back. Suck it up, decrease
checkpoint_segments, and get used to having more checkpoints if you have
to keep the same recovery time".
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
---(end of broadcast)--
presented so far is extrapolation.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
---(end of broadcast)---
TIP 4: Have you searched our list archives?
http://archives.postgresql.org
are without cost is
completely unrealistic IMHO. In the situation I just described above, I
also noticed the way O_DIRECT sync writes get mixed with buffered WAL
writes seems to cause some weird I/O scheduling issues in Linux that can
make worst-case latency degrade. But since I can'
vates the issue. Given a knob where the LDC can
be spread out a bit but not across the entire interval, that makes it
easier to control how much expansion there is relative to the current
behavior.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
e the rest of the xlog
writes do? That write goes through the regular buffered path instead.
The checkpoint logging patch I submitted logs when this happens
specifically because that particular issue messed with some operations and
I found it important to be awar
chives for the discussion this week on the topic
"Controlling Load Distributed Checkpoints", and check out the "Automatic
adjustment of bgwriter_lru_maxpages" patch whose latest version is at
http://archives.postgresql.org/pgsql-patches/2007-05/msg00142.php
--
* Greg Smith [E
On Fri, 1 Jun 2007, Andrew Dunstan wrote:
Greg Smith wrote:
Since the rotation size feature causes other issues anyway
that make importing more complicated, documenting the issue seemed
sufficient.
What are the other issues? I'm not happy about producing files with split
lines.
Just
range,
and I wouldn't want to see that truncated too much.
Anyway, I have a bunch of data on this subject being collected at this
moment, and I'll rescale the results based on what I see after analyzing
that this week.
--
* Greg Smith [EMAIL PROTECTED
kind of an unreasonable situation to go out of your way to accommodate.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
---(end of broadcast)---
TIP 2: Don't 'kill -9' the postmaster
t server this week to confirm
whether's it's worthless precision or worth capturing.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
---(end of broadcast)---
TIP 9: In versions below 8.0, the planner will ignore
the usage_count regardless.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
---(end of broadcast)---
TIP 6: explain analyze is your friend
cts the dynamics of
how downstream applications will have to work with this data, so you can
see that my contrary preference isn't completely random here.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
---(end of broadcast)---
, so
I'll add another column for the line number to the output. Thanks for
pointing that out, I can finish cleaning up all the functional
implementation work on this patch myself now.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
---(e
fix that too when I'm revising. I plan to have a version free of
obvious bugs to re-submit ready by next weekend.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
---(end of broadcast)---
TIP 1: if posting/reading throug
to what's already been done here, I sent
in what I thought was ready to go because I didn't want to hold up
reviewing the bulk of the code over some of these fine details.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
---(end of br
y're allocating more than that
during a period, they should shrink the delay instead. The kinds of
systems where 1000 isn't high enough for bgwriter_lru_maxpages are going
to be compelled to adjust these parameters anyway for good performance.
--
* Greg Smith [EMAIL PROTECTED] htt
w.php/140/4/PGBuildFarm-HOWTO.txt
and those aren't quite focused right if the goal is to work on new patches
while keeping in sync with the repository.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
---(end of broadcast)-
easier to write that if you can say "You can safely set
bgwriter_lru_maxpages high because it only writes what it needs to based
on your usage".
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
---(end of broadcast)---
f back into
the callers would end up being a cut and paste job that would duplicate
many lines. That's on top of the fact that the buffer is cleanly
locked/unlocked all in one section of code right now, and I didn't see how
to move any parts of that to the c
the LRU
limiting behavior of the second patch, at which point it's strictly
internals code with no expected functional impact that alternate
approaches might be built on.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MDIndex: src/backend/storage/buffer/bufmgr.c
==
the shoes of a log file parser the
other way around is easier to work with. Piecing together log entries is
a pain, splitting them is easy.
If I had to only keep one line out of this, it would be the one with the
summary. It would be nice to have it logged at INFO.
--
* Greg Smith [EMAIL
ible to make a reasonable estimate. Any less information
than all that and you really have very little basis on which to guess how
long it's going to take.
Other operating systems are going to give completely different behavior
here, which of course makes the problem ev
a discussion on the
patches list about a feature useful *right now* in this area.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
---(end of broadcast)---
TIP 6: explain analyze is your friend
ould come from things like AIO--it gets real
messy. Good luck drumming up support for all that when the initial
benchmarks suggest it's going to be a big step back.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
---(end of broadcast)-
using a moving-average approach (as I did in my auto-tuning
writer prototype and as Tom just suggested here) will be sufficient to fix
all of them. Was already on my to-do list to investigate that further.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
---
h a redesign there, maybe you can get away with it at this
late moment. But I ask that you please don't remove anything from the
current design until you have significant test results to back up that
change.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
--
ious portions of the checkpoint, to
tell when it was bogged down with slow writes versus being held up in sync
for various (possibly fixed with your CheckpointStartLock) issues.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
ting I consider a bonus rather than the main focus here.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
---(end of broadcast)---
TIP 5: don't forget to increase your free space map settings
a partial result
because I haven't had enough time track all the code paths involved to
prove otherwise. Glad to hear it was never an issue. Doesn't change what
I submitted though.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
-
On Thu, 29 Mar 2007, Magnus Hagander wrote:
I've included a couple of more counters per ideas from Greg Smith in his
logging patch.
I just submitted a patch that logs the remaining things of value from my
original that couldn't be derived from the information you're collecting
g a dependency on its
current structure to here. Ideally, it would be nice to call something
like WillLog(DEBUG2) and get a boolean back saying whether something at
that log level will be output anywhere; I don't know enough about the
logging code to add such a thing to it mys
uffer cache before they could
get evicted, which would take quite a while. My bet is that bufferid gets
expanded from 32 bits before that happens, which would break
pg_buffercache similarly and isn't being checked for either.
--
* Greg Smith [EMAIL PROTECTED] http://www
massive cost in overhead, of course. I'll be
passing along all that code once I get it ready for other people to use.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MDIndex: README.pg_buffercache
===
RCS f
un variation in TPS (checkpoints are particularly visible
in the graphs), after further testing I concluded running a VACUUM VERBOSE
and CHECKPOINT in a script afterwards and analyzing the results was more
useful than integrating something into pgbench itself.
--
* Greg Smith [EMAIL
the example graphs there aren't from the production system I
mentioned above, they're from my server at home, which is similar to the
system your results came from).
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
---(end of broad
slow checkpoint
issues even worse on my Linux system. I'm trying to evaluate this fairly.
-This code operates on the assumption you have a good value for the
checkpoint timeout. Have you tested its behavior when checkpoints are
being triggered by checkpoint_segments being reached instea
ley related mechanics. But the SQL logs are be based on my
requirements, which is to include close enough to everything that it might
as well be the whole set, in case I forgot something I find I need later.
The SQL logs are *completely* different from the syslog setup.
--
* Greg Smith [EMAIL
ir logs probably has an installation that can support the
overhead.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
---(end of broadcast)---
TIP 4: Have you searched our list archives?
http://archives.postgresql.org
wrong with the approach you suggest for a first version of
this feature. I know I'm so desparate for a better solution to log
importing that I'd be happy with any workable improvement.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
---(end of broadcast)---
TIP 6: explain analyze is your friend
se entries instead of having so many ad-hoc approaches. You want a
subset, use a view or copy just the fields you want into another table.
I would guess this simplifies the patch as well.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
---(end of broadcast)---
TIP 2: Don't 'kill -9' the postmaster
or pgbench.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
---(end of broadcast)---
TIP 7: You can help support the PostgreSQL project by donating at
http://www.postgresql.org/about/donate
I
will prepare a slightly different patch aimed at 8.3--incorporating all
the feedback I've gotten here as either code changes or additional
documentation--and resubmit in another week or so.
Thanks for the feedback.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com
I suggest trying it out. You may be as
surprised as I was at exactly how much vacuuming work is leftover after a
long pgbench run, and how dramatically it lowers TPS results if you
consider that cleanup essential to the test.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com
sure haven't been able to measure any change in
results.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
---(end of broadcast)---
TIP 6: explain analyze is your friend
series of additional scripts
I'll be releasing shortly that do interesting analysis of this extended
latency data from pgbench (graphs of TPS and latency, that sort of thing),
and I'd hate for that to only be available on 8.3.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.co
87 matches
Mail list logo