> We may have a good idea of how to define a custom language, still we
> are going to need to design a clean interface at catalog level more or
> less close to what is written here. If we can get a clean interface,
> the custom language implemented, and TAP tests that take advantage of
> this
All,
So the proximate cause of late releases are the following:
1. patches are getting kicked down the road from one CF to another, creating a
big pileup in the final CF. This is exactly like the huge pile of unreviewed
patches we used to have before the CF system.
2. Once the last CF is
- Original Message -
* Josh Berkus (j...@agliodbs.com) wrote:
Is there anyone else on the committer list with similar circumstances?
I'll just flip it around and offer to be publically flogged whenever I'm
not helping out with a commitfest. :) Perhaps this should be more
opt-in
Robert,
I'm sure it's possible; I don't *think* it's terribly easy. The
usual
algorithm for cycle detection is to have each node send to the next
node the path that the data has taken. But, there's no unique
identifier for each slave that I know of - you could use IP address,
but that's
As ever, we spent much energy on debating backwards compatibility
rather than just solving the problem it posed, which is fairly easy
to
solve.
Well, IIRC, the debate was primarily of *your* making. Almost everyone else on
the thread was fine with the original patch, and it was nearly done
I just committed a patch that should make the requested WAL segment
00020003 has already been removed errors go away.
The
trick was for walsenders to not switch to the new timeline until at
least one record has been replayed on it. That closes the window
where
the
Robert,
What would such a test look like? It's not obvious to me that
there's any rapid way for a user to detect this situation, without
checking each server individually.
Change something on the master and observe that none of the supposed
standbys notice?
That doesn't sound like an
Andreas,
Do you want the node one step up or the top-level in the chain?
Because
I don't think we can do the latter without complicating the
replication
protocol noticably.
Well, clearly a whole chain would be nice for the user. But even just one step
up would be very useful.
--Josh
Heikki,
The problem goes away after some time, after the 1st standby has
streamed the contents of 00020003 and written it to
disk, and the cascaded standby reconnects. But it would be nice to
avoid
that situation. I'm not sure how to do that yet, we might need to
track
the
Heikki,
The next time I get the issue, and I'm not paying for 5 cloud servers by the
hour, I'll give you a login.
--Josh
- Original Message -
On 19.12.2012 17:27, Heikki Linnakangas wrote:
On 19.12.2012 15:55, Heikki Linnakangas wrote:
On 19.12.2012 04:57, Josh Berkus wrote:
Simon,
My logic is that if you make a 1 minute test you will notice your
mistake, which is glaringly obvious. That is sufficient to prevent
that mistake, IMHO.
What would such a test look like? It's not obvious to me that there's any
rapid way for a user to detect this situation, without
It stalled because the patch author decided not to implement the
request to detect recovery.conf in data directory, which allows
backwards compatibility.
Well, I don't think we had agreement on how important backwards compatibility
for recovery.conf was, particularly not on the whole
This sounds like my previous suggestion of returning the primary
conninfo value, but with just ip. That one came with a pretty bad
patch, and was later postponed until we folded recovery.conf into
the main configuration file parsing. I'm not really sure what
happened to that project? (the
For my part, while that's certainly an interesting idea, it's far
more
complicated than even providing GUCs and the idea is to make PG just
do
it right, not to offer the user more ways to get it wrong...
Yes, please let's not replace the existing too-simplistic knobs with giant
complicated
Simon,
I think its sad we can't even attempt a technical conversation
without
you making snide ad hominem attacks that aren't even close to being
true on a personal level, nor accurate in a technical sense.
I would prefer it if you actually addressed my substantive arguments, which, so
far,
Ah. Okay, maybe we can agree that that wasn't a good idea.
Oh, I'd say there's no question it was a mistake. We just didn't have the data
at the time to realize it.
I don't really see that we need to bend over backwards to exactly
match
some data points that you made up out of thin air.
Robert,
Hmmm. I was assuming September, given how late the beta came out, and that
nobody has previously talked seriously about a June release. I'll also point
out that while there's a beta2 tarball, there was no announcement and no
packages for it.
If we decide to do June, then PR will be
So currently we have a major limitation in binary replication, where it is not
possible to remaster your system (that is, designate the most caught-up
standby as the new master) based on streaming replication only. This is a
major limitation because the requirement to copy physical logs over
It might be easy to detect the situation where the standby has
connected to itself,
e.g., by assigning ID for each instance and checking whether IDs of
two servers
are the same. But it seems not easy to detect the
circularly-connected
two or more
standbys.
Well, I think it would be fine
Fujii,
You mean that remaster is, after promoting one of standby servers,
to make
remaining standby servers reconnect to new master and resolve the
timeline
gap without the shared archive? Yep, that's one of my TODO items, but
I'm not
sure if I have enough time to implement that for
17, 2012 at 6:08 AM, Joshua Berkus j...@agliodbs.com
wrote:
As you can see, the indexonlyscan version of the query spends 5% as
much time reading the data as the seq scan version, and doesn't
have to read the heap at all. Yet it spends 20 seconds doing ...
what, exactly?
BTW, kudos
Erik,
Are you taking the counts *while* the table is loading? In sync replication,
it's possible for the counts to differ for a short time due to one of three
things:
* transaction has been saved to the replica and confirm message hasn't reached
the master yet
* replica has synched the
Jim, Fujii,
Even more fun:
1) Set up a server as a cascading replica (e.g. max_wal_senders = 3,
standby_mode = on )
2) Connect the server to *itself* as a replica.
3) This will work and report success, up until you do your first write.
4) Then ... segfault!
- Original Message -
lookups which happen once per leaf
node instead of in bulk.Certainly the performance I'm seeing would be
consistent with that idea.
I'll try some multi-column covering indexes next to see how it looks.
- Original Message -
On Thu, May 17, 2012 at 5:22 AM, Joshua Berkus j
a...@cybertec.at
wrote:
On Thu, May 17, 2012 at 3:42 PM, Joshua Berkus j...@agliodbs.com
wrote:
Even more fun:
1) Set up a server as a cascading replica (e.g. max_wal_senders =
3, standby_mode = on )
2) Connect the server to *itself* as a replica.
3) This will work and report success, up
And: if we still have to ship logs, what's the point in even having
cascading replication?
At least cascading replication (1) allows you to adopt more flexible
configuration of servers,
I'm just pretty shocked. The last time we talked about this, at the end of the
9.1 development
So, I set up a test which should have been ideal setup for index-only scan.
The index was 1/10 the size of the table, and fit in RAM (1G) which the table
does not:
bench2=# select pg_size_pretty(pg_relation_size('pgbench_accounts_pkey'));
pg_size_pretty
428 MB
(1 row)
Jim,
I didn't get as far as running any tests, actually. All I did was try to set
up 3 servers in cascading replication. Then I tried shutting down
master-master and promoting master-replica. That's it.
- Original Message -
On May 13, 2012, at 3:08 PM, Josh Berkus wrote:
More
Fujii,
Wait, are you telling me that we *still* can't remaster from streaming
replication? Why wasn't that fixed in 9.2?
And: if we still have to ship logs, what's the point in even having cascading
replication?
- Original Message -
On Wed, May 16, 2012 at 1:36 AM, Thom Brown
Before restarting it, you need to do pg_basebackup and make a base
backup
onto the standby again. Since you started the standby without
recovery.conf,
a series of WAL in the standby has gotten inconsistent with that in
the master.
So you need a fresh backup to restart the standby.
You're
If we were actually using git branches for it, the CF app could
automatically close entries when they were committed. But that
requires them to be committed *unmodified*, and I'm not sure that's
reasonable. I also think requiring a git branch for the *simple*
changes is adding more tooling
I think the big take-away, education-wise, is that for our project,
committer == grunt work. Remember, I used to be the big committer of
non-committer patches --- need I say more. ;-) LOL
Well, promoting several people to committer specifically and publically because
of their review work
All,
From my observation, the CF process ... in fact, all development processes
we've had in Postgres ... have suffered from only one problem: lack of
consensus on how the process should work. For example, we've *never* had
consensus around the criteria for kicking a patch out of a
Ultimately, we're herding cats here. I don't think you're going to
get
the community to suddenly be willing to march in lockstep instead.
If you, Peter, Simon, Robert, Heikki, Magnus, Peter G., Greg, Bruce and Andrew
agreed on a calendar-driven, mostly unambiguous process and adhered to
Qi,
Yeah, I can see that. That's a sign that you had a good idea for a project,
actually: your idea is interesting enough that people want to debate it. Make
a proposal on Monday and our potential mentors will help you refine the idea.
- Original Message -
Date: Thu, 22 Mar
Billy,
I've done a brief search of the postgresql mail archives, and I've
noticed a few projects for adding query caches to postgresql, (for
example, Masanori Yamazaki's query cache proposal for GSOC 2011),
... which was completed, btw. Take a look at the current release of pgPool.
Are you
Hackers,
NTT Open Source has requested that I convene the 3rd Cluster Hackers summit at
pgCon this year. As last year, it will be held on Tuesday (May 15th) during
tutorials (and not conflicting with the Developer Summit).
If you are a contributor to any of PostgreSQL's various replication,
Compiling on Ubuntu 10.04 LTS AMD64 on a GoGrid virtual machine from 2012-01-12
git checkout.
Patch applied fine.
Docs are present, build, look good and are clear.
Changes to gram.y required Bison 2.5 to compile. Are we requiring Bison 2.5
now? There's no configure check for it, so it took
Bruce,
I don't see any of this reaching the level that it needs to be
backpatched, so I think we have to accept that this will be 9.2-only
change.
Agreed. If users encounter issues with the prefix in the field, it will be
easy enough for them to back-patch. But we don't want to be
Greg,
I'm not attached to the name, which I just pulled out of the air for
the
documentation. Could just as easily call them built-in modules or
extensions. If the objection is that extensions isn't technically
correct for auto-explain, you might call them core add-ons instead.
My
Peter,
I consider contrib/isn to be quite broken. It hard codes ISBN
prefixes
for the purposes of sanitising ISBNs, even though their assignment is
actually controlled by a decentralised body of regional authorities.
I'd vote for kicking it out of contrib.
Submit a patch to fix it then.
All,
I agree. The argument that this code is useful as example code has
been offered before, but the justification is pretty thin when the
example code is an example of a horrible design that no one should
ever copy.
People are already using ISN (or at least ISBN) in production. It's been
Here is a patch for that for pg_dump. The sections provided for are
pre-data, data and post-data, as discussed elsewhere. I still feel that
anything finer grained should be handled via pg_restore's --use-list
functionality. I'll provide a patch to do the same switch for pg_restore
Robert,
In most cases we either break backwards compatibility or require some
type of switch to turn on backwards compatibility for those who want
it. While the above plan tries to do one better, it leaves me feeling
that the thing I don't like about this is that it sounds like you are
Folks,
What happens currently if we have an \include in postgresql.conf for a file
which doesn't exist? Is it ignored, or do we error out?
If it could just be ignored, maybe with a note in the logs, then we could be a
lot more flexible.
--Josh Berkus
--
Sent via pgsql-hackers mailing list
There might be a use case for a separate directive include_if_exists,
or some such name. But I think the user should have to tell us very
clearly that it's okay for the file to not be found.
Better to go back to include_directory, then.
--Josh Berkus
--
Sent via pgsql-hackers mailing
Reminder: BETWEEEN supports the SYMMETRIC keyword, so there is
a precedent for this.
And I don't see it as valuable enough to justify changing the
grammar.
I agree that we should leave symmetry until 9.3.
--Josh Berkus
--
Sent via pgsql-hackers mailing list
All,
I'd love to see someone evaluate the impact of Marti's patch on JDBC
applications which use named prepared statements. Anyone have a benchmark
handy?
--Josh
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
I rather like Tom's suggestion of include_if_exists.
include_if_exists certainly solves the recovery.conf/recovery.done problem. We
can even phase it out, like this:
9.2: include_if_exists = 'recovery.conf' in the default postgresql.conf file.
9.3: include_if_exists = 'recovery.conf'
Since we haven't yet come up with a reasonable way of machine-editing
postgresql.conf, this seems like a fairly serious objection to
getting
rid of recovery.conf. I wonder if there's a way we can work around
that...
Well, we *did* actually come up with a reasonable way, but it died under
All,
We might want to have a system where an extension can declare that
it
provides capabilites, and then have another extension require
those
capabilities. That would be a neater solution to the case that
there are
multiple extensions that all provide the same capability.
+1
As
I'm in favor of defining a separate, content-free trigger file to
enable
archive recovery. Not sure about the name recovery.ready, though
---
that makes it look like one of the WAL archive transfer trigger
files,
which does not seem like a great analogy. The pg_standby
documentation
That's not my recollection. Obviously, it's hard to measure this one
way or the other, but I don't recall there being a lot of test
reports
from people who are not already contributors and could have used some
other way to get the code.
Do we have download stats for the alphas? Dave?
Download numbers for the installers were bordering on noise compared
to the GA builds last time I looked, double figures iirc. I don't
know about the tarballs offhand and can't check ATM.
Can you check when you get a chance? I know that the DL numbers for the first
alphas were very low,
All,
Where are we on RC1 or Beta4 for PostgreSQL 9.1?
While I know we're doing going to do a final release in August due to the
europeans, it would be nice to move things along before then. There don't seem
to be any blockers open.
--
Josh Berkus
PostgreSQL Experts Inc.
Robert,
We need to start marking the patches that are Waiting on Author as
Returned with Feedback, ideally after checking that the status in
the CF application is in fact up to date. With a week left in the
CommitFest at this point, anything that has been reviewed and still
has issues is
Simon,
The point I have made is that I disagree with a feature freeze date
fixed ahead of time without regard to the content of the forthcoming
release. I've not said I disagree with feature freezes altogether,
which would be utterly ridiculous. Fixed dates are IMHO much less
important than
iew. The
reason we usually skip the summer isn't actually a wholesale lack of
people - it's because it's not so good from a publicity perspective,
and it's hard to get all the packagers around at the same time.
Actually, the summer is *excellent* from a publicity perspective ... at least,
Robert,
Oh, I get that. I'm just dismayed that we can't have a discussion
about the patch without getting sidetracked into a conversation about
whether we should throw feature freeze out the window.
That's not something you can change. Whatever the patch is, even if it's a
psql
MauMau,
Could you give me your frank opinions about which of 8.4 or 9.0 you
recommend to ISVs who embed PostgreSQL?
So, first of all, you posted this question to the wrong list. pgsql-general or
pgsql-admin would have been more appropriate for this question.
That being said, I find your
Robert,
Another point is that parsing overhead is quite obviously not the
reason for the massive performance gap between one core running simple
selects on PostgreSQL and one core running simple selects on MySQL.
Even if I had (further) eviscerated the parser to cover only the
syntax those
All,
I agree that we should not reduce the support window. The fact that we
can do in place upgrades of the data only addresses one pain point in
upgrading. Large legacy apps require large retesting efforts when
upgrading, often followed by lots more work renovating the code for
backwards
Chris,
Not totally Idle thought: it would be nice if the holding
corporation doesn't need a bank account, as they impose burdens of
fees (not huge, but not providing us notable value), and more
importantly, impose administrative burdens. Our banks like to impose
holds on accounts any time
I think the main thing we have to think about before choosing is
whether
we believe that we can shorten the CFs at all. Josh's proposal had
3-week CFs after the first one, which makes it a lot easier to have a
fest in November or December, but only if you really can end it on
time.
I
Pavel,
Actually we had to solve a issue with slow SELECT. The problem was in
low value of JOIN_COLLAPSE_LIMITS. Can we increase a default of this
value. I checked some complex query, and planner needed about 200ms
for JOIN_COLLAPSE_LIMIT = 16. So some around 12 can be well.
I'm not
Dimitri,
I'll bet someone a fancy drink at a conference that this thread goes
to at least 100 posts.
Of course, if we all are to argue about this bet… :)
Darn! You've uncovered by sinister plan. Foiled again!
--
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com
San Francisco
Robert,
Tom and I were talking about starting maybe June 1, rather than July
1. You seem opposed but I'm not sure why.
Because I think -- strictly based on history and the complexity of the new
features -- we'll still be fixing major issues with the beta in June, which was
what Tom said as
I'll bet someone a fancy drink at a conference that this thread goes to at
least 100 posts.
Let the bikeshedding begin!
--
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com
San Francisco
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your
If CF1 is June1, though, when will CF4 be? Having a CF start Dec. 1
is probably a bad idea.
Well, I made a suggestion on this topic in my previous email on the
subject...
I just searched backwards on this thread and I can't find it. There's been a
lot of posts.
--
Josh Berkus
All,
+1 from me for keeping it as-is as well.
So it sounds like most committers want to keep the CFs on their existing
schedule for another year. Also that we don't want to branch until RC1. While
it would be nice to get some feedback from those who had bad experiences with
the CF cycle,
Every time I've gotten pulled into discussions of setting parameters
based on live monitoring, it's turned into a giant black hole--absorbs a
lot of energy, nothing useful escapes from it. I credit completely
ignoring that idea altogether, and using the simplest possible static
settings
Tom,
I'd like a pony, too. Let's be perfectly clear about this: there is
no
part of plpgsql that can run outside a transaction today, and
probably
no part of the other PLs either, and changing that without major
changes is wishful thinking of the first order.
I always thought that it
Robert, Tom,
Hm ... there are people out there who think *I* get high off rejecting
patches. I have a t-shirt to prove it. But I seem to be pretty
ineffective at it too, judging from these numbers.
It's a question of how we reject patches, especially first-time patches. We
can reject them
Robert,
Actually, I'd walk through fire for a 10% performance improvement if
it meant only a *risk* to stability.
Depends on the degree of risk. MMAP has the potential to introduce instability
into areas of the code which have been completely reliable for years. Adding
20 new coredump
All,
Never, and that's not true. Heikki was being nice; I wouldn't have
even
slogged through it long enough to ask the questions he did before
kicking it back as unusable. A badly formatted patch makes it
impossible to evaluate whether the changes from a submission are
reasonable or not
Radoslaw,
I think 10% is quite good, as my stand-alone test of mmap vs. read
shown that
speed up of copying 100MB data to mem may be from ~20ms to ~100ms
(depends on
destination address). Of course deeper, system test simulating real
usage will
say more. In any case after good deals with
Radoslaw,
10% improvement isn't very impressive from a switch to mmap. What workload did
you test with? What I'd really like to see is testing with databases which are
50%, 90% and 200% the size of RAM ... that's where I'd expect the greatest gain
from limiting copying.
Netbeans is
All,
While it would be nice to improve our performance on this workload, let me
point out that it's not a very important workload from the point of view of
real performance challenges. Yes, there are folks out there with 100MB
databases who only run one-liner select queries. But frankly,
Magnus,
That could certainly be useful, yes. But I have a feeling whomever
tries to get that into 9.1 will be killed - but it's certainly good to
put ont he list of things for 9.2.
Oh, no question. At some point in 9.2 we should also discuss how basebackup
considers emtpy directories.
All,
We left this out of 9.0; let's not leave it out of 9.1. We need an example
replication line in pg_hba.conf, commented out. e.g.
# host replication all samenet md5
Also, what happened to having a replication user defined by default? We
talked this to death last year, I
All,
If I have the following line in pg_hba.conf:
hostreplication replication all md5
pg_basebackup -x -v -P -h master1 -U replication -D $PGDATA
pg_basebackup: could not connect to server: FATAL: no pg_hba.conf entry for
replication connection from
Magnus, all:
It seems a bit annoying to have to do an rm -rf * $PGDATA/ before resynching a
standby using pg_basebackup. This means that I still need to wrap basebackup
in a shell script, instead of having it do everything for me ... especially if
I have multiple tablespaces.
Couldn't we
I would think it would be purely syntatic sugar really, which does
incorporate a familiar interface for those who are working in
different
worlds (.Net/Drupal/JAVA) etc...
I wouldn't mind having something more standard supported; I'm always looking up
the conninfo for the options I don't
The fifth alpha release for PostgreSQL version 9.1, 9.1alpha5, is now
available. There are no new major features in this alpha release as compared
to 9.1alpha4, but there are many minor bug fixes and improvements to features
added in 9.1alpha4 and earlier alpha releases. It is expected that
Making DDL serializable is *not* simple, and half-baked hacks won't
make that situation better ...
That seemed unnecessary. Whether or not you approve of Stephen's solution, he
is dealing with a real issue.
--
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com
San Francisco
--
Sent
Tomas,
I spoke to a teacher from a local university last week, mainly as we
were looking for a place where a local PUG could meet regularly. I
realized this could be a good opportunity to head-hunt some students
to
participate in this GSoC. Are we still interested in new students?
Yes,
Tom,
Personally I'd vote for *not* having any such dangerous semantics as
that. We should have learned better by now from plpgsql experience.
I think the best idea is to throw error for ambiguous references,
period.
As a likely heavy user of this feature, I agree with Tom here. I really
87 matches
Mail list logo
