> We may have a good idea of how to define a custom language, still we
> are going to need to design a clean interface at catalog level more or
> less close to what is written here. If we can get a clean interface,
> the custom language implemented, and TAP tests that take advantage of
> this user
All,
So the proximate cause of late releases are the following:
1. patches are getting kicked down the road from one CF to another, creating a
big pileup in the final CF. This is exactly like the huge pile of unreviewed
patches we used to have before the CF system.
2. Once the last CF is clos
- Original Message -
> * Josh Berkus (j...@agliodbs.com) wrote:
> > Is there anyone else on the committer list with similar circumstances?
>
> I'll just flip it around and offer to be publically flogged whenever I'm
> not helping out with a commitfest. :) Perhaps this should be more
> "
Robert,
> I'm sure it's possible; I don't *think* it's terribly easy. The
> usual
> algorithm for cycle detection is to have each node send to the next
> node the path that the data has taken. But, there's no unique
> identifier for each slave that I know of - you could use IP address,
> but tha
Andreas,
> Do you want the node one step up or the top-level in the chain?
> Because
> I don't think we can do the latter without complicating the
> replication
> protocol noticably.
Well, clearly a whole chain would be nice for the user. But even just one step
up would be very useful.
--Josh
Robert,
> > What would such a test look like? It's not obvious to me that
> > there's any rapid way for a user to detect this situation, without
> > checking each server individually.
>
> Change something on the master and observe that none of the supposed
> standbys notice?
That doesn't sound
> I just committed a patch that should make the "requested WAL segment
> 00020003 has already been removed" errors go away.
> The
> trick was for walsenders to not switch to the new timeline until at
> least one record has been replayed on it. That closes the window
> where
> the
> As ever, we spent much energy on debating backwards compatibility
> rather than just solving the problem it posed, which is fairly easy
> to
> solve.
Well, IIRC, the debate was primarily of *your* making. Almost everyone else on
the thread was fine with the original patch, and it was nearly d
> This sounds like my previous suggestion of returning the primary
> conninfo value, but with just ip. That one came with a pretty bad
> patch, and was later postponed until we folded recovery.conf into
> the main configuration file parsing. I'm not really sure what
> happened to that project? (th
> It stalled because the patch author decided not to implement the
> request to detect recovery.conf in data directory, which allows
> backwards compatibility.
Well, I don't think we had agreement on how important backwards compatibility
for recovery.conf was, particularly not on the whole
reco
Simon,
> My logic is that if you make a 1 minute test you will notice your
> mistake, which is glaringly obvious. That is sufficient to prevent
> that mistake, IMHO.
What would such a test look like? It's not obvious to me that there's any
rapid way for a user to detect this situation, without
Heikki,
The next time I get the issue, and I'm not paying for 5 cloud servers by the
hour, I'll give you a login.
--Josh
- Original Message -
> On 19.12.2012 17:27, Heikki Linnakangas wrote:
> > On 19.12.2012 15:55, Heikki Linnakangas wrote:
> >> On 19.12.2012 04:57, Josh Berkus wrote:
Heikki,
> The problem goes away after some time, after the 1st standby has
> streamed the contents of 00020003 and written it to
> disk, and the cascaded standby reconnects. But it would be nice to
> avoid
> that situation. I'm not sure how to do that yet, we might need to
> track
> Ah. Okay, maybe we can agree that that wasn't a good idea.
Oh, I'd say there's no question it was a mistake. We just didn't have the data
at the time to realize it.
> I don't really see that we need to bend over backwards to exactly
> match
> some data points that you made up out of thin ai
Simon,
> I think its sad we can't even attempt a technical conversation
> without
> you making snide ad hominem attacks that aren't even close to being
> true on a personal level, nor accurate in a technical sense.
I would prefer it if you actually addressed my substantive arguments, which, so
f
> For my part, while that's certainly an interesting idea, it's far
> more
> complicated than even providing GUCs and the idea is to make PG just
> "do
> it right", not to offer the user more ways to get it wrong...
Yes, please let's not replace the existing too-simplistic knobs with giant
compl
Robert,
Hmmm. I was assuming September, given how late the beta came out, and that
nobody has previously talked seriously about a June release. I'll also point
out that while there's a beta2 tarball, there was no announcement and no
packages for it.
If we decide to do June, then PR will be m
So currently we have a major limitation in binary replication, where it is not
possible to "remaster" your system (that is, designate the most caught-up
standby as the new master) based on streaming replication only. This is a
major limitation because the requirement to copy physical logs over
Fujii,
>
> You mean that "remaster" is, after promoting one of standby servers,
> to make
> remaining standby servers reconnect to new master and resolve the
> timeline
> gap without the shared archive? Yep, that's one of my TODO items, but
> I'm not
> sure if I have enough time to implement that
> It might be easy to detect the situation where the standby has
> connected to itself,
> e.g., by assigning ID for each instance and checking whether IDs of
> two servers
> are the same. But it seems not easy to detect the
> circularly-connected
> two or more
> standbys.
Well, I think it would b
PM, Ants Aasma
> wrote:
> > On Thu, May 17, 2012 at 3:42 PM, Joshua Berkus
> > wrote:
> >> Even more fun:
> >>
> >> 1) Set up a server as a cascading replica (e.g. max_wal_senders =
> >> 3, standby_mode = on )
> >>
> >> 2) Con
especially lookups which happen once per leaf
node instead of in bulk.Certainly the performance I'm seeing would be
consistent with that idea.
I'll try some multi-column covering indexes next to see how it looks.
- Original Message -
> On Thu, May 17, 2012 at 5:22 AM,
-
> On 5/16/12 10:53 AM, Fujii Masao wrote:
> > On Wed, May 16, 2012 at 3:43 AM, Joshua Berkus
> > wrote:
> >>
> >>> Before restarting it, you need to do pg_basebackup and make a
> >>> base
> >>> backup
> >>> onto the standby a
Erik,
Are you taking the counts *while* the table is loading? In sync replication,
it's possible for the counts to differ for a short time due to one of three
things:
* transaction has been saved to the replica and confirm message hasn't reached
the master yet
* replica has synched the transa
; On Thu, May 17, 2012 at 6:08 AM, Joshua Berkus
> wrote:
> > As you can see, the indexonlyscan version of the query spends 5% as
> > much time reading the data as the seq scan version, and doesn't
> > have to read the heap at all. Yet it spends 20 seconds doing ...
>
So, I set up a test which should have been ideal setup for index-only scan.
The index was 1/10 the size of the table, and fit in RAM (1G) which the table
does not:
bench2=# select pg_size_pretty(pg_relation_size('pgbench_accounts_pkey'));
pg_size_pretty
428 MB
(1 row)
be
> > And: if we still have to ship logs, what's the point in even having
> > cascading replication?
>
> At least cascading replication (1) allows you to adopt more flexible
> configuration of servers,
I'm just pretty shocked. The last time we talked about this, at the end of the
9.1 development
> Before restarting it, you need to do pg_basebackup and make a base
> backup
> onto the standby again. Since you started the standby without
> recovery.conf,
> a series of WAL in the standby has gotten inconsistent with that in
> the master.
> So you need a fresh backup to restart the standby.
Y
Fujii,
Wait, are you telling me that we *still* can't remaster from streaming
replication? Why wasn't that fixed in 9.2?
And: if we still have to ship logs, what's the point in even having cascading
replication?
- Original Message -
> On Wed, May 16, 2012 at 1:36 AM, Thom Brown wrote
Jim,
I didn't get as far as running any tests, actually. All I did was try to set
up 3 servers in cascading replication. Then I tried shutting down
master-master and promoting master-replica. That's it.
- Original Message -
> On May 13, 2012, at 3:08 PM, Josh Berkus wrote:
> > More i
> I think the big take-away, education-wise, is that for our project,
> committer == grunt work. Remember, I used to be the big committer of
> non-committer patches --- need I say more. ;-) LOL
Well, promoting several people to committer specifically and publically because
of their review wor
> If we were actually using git branches for it, the CF app could
> automatically close entries when they were committed. But that
> requires them to be committed *unmodified*, and I'm not sure that's
> reasonable. I also think requiring a git branch for the *simple*
> changes is adding more tool
> Ultimately, we're herding cats here. I don't think you're going to
> get
> the community to suddenly be willing to march in lockstep instead.
If you, Peter, Simon, Robert, Heikki, Magnus, Peter G., Greg, Bruce and Andrew
agreed on a calendar-driven, mostly unambiguous process and adhered to t
All,
>From my observation, the CF process ... in fact, all development processes
>we've had in Postgres ... have suffered from only one problem: lack of
>consensus on how the process should work. For example, we've *never* had
>consensus around the criteria for kicking a patch out of a commitf
Billy,
> I've done a brief search of the postgresql mail archives, and I've
> noticed a few projects for adding query caches to postgresql, (for
> example, Masanori Yamazaki's query cache proposal for GSOC 2011),
... which was completed, btw. Take a look at the current release of pgPool.
Are yo
Qi,
Yeah, I can see that. That's a sign that you had a good idea for a project,
actually: your idea is interesting enough that people want to debate it. Make
a proposal on Monday and our potential mentors will help you refine the idea.
- Original Message -
>
>
>
>
> > Date: Thu, 2
Hackers,
NTT Open Source has requested that I convene the 3rd Cluster Hackers summit at
pgCon this year. As last year, it will be held on Tuesday (May 15th) during
tutorials (and not conflicting with the Developer Summit).
If you are a contributor to any of PostgreSQL's various replication,
Compiling on Ubuntu 10.04 LTS AMD64 on a GoGrid virtual machine from 2012-01-12
git checkout.
Patch applied fine.
Docs are present, build, look good and are clear.
Changes to gram.y required Bison 2.5 to compile. Are we requiring Bison 2.5
now? There's no configure check for it, so it took
Bruce,
> I don't see any of this reaching the level that it needs to be
> backpatched, so I think we have to accept that this will be 9.2-only
> change.
Agreed. If users encounter issues with the prefix in the field, it will be
easy enough for them to back-patch. But we don't want to be respo
> > Here is a patch for that for pg_dump. The sections provided for are
> > pre-data, data and post-data, as discussed elsewhere. I still feel that
> > anything finer grained should be handled via pg_restore's --use-list
> > functionality. I'll provide a patch to do the same switch for pg_restore
All,
> I agree. The argument that this code is useful as example code has
> been offered before, but the justification is pretty thin when the
> example code is an example of a horrible design that no one should
> ever copy.
People are already using ISN (or at least ISBN) in production. It's be
Peter,
> I consider contrib/isn to be quite broken. It hard codes ISBN
> prefixes
> for the purposes of sanitising ISBNs, even though their assignment is
> actually controlled by a decentralised body of regional authorities.
> I'd vote for kicking it out of contrib.
Submit a patch to fix it then.
Greg,
> I'm not attached to the name, which I just pulled out of the air for
> the
> documentation. Could just as easily call them built-in modules or
> extensions. If the objection is that "extensions" isn't technically
> correct for auto-explain, you might call them core add-ons instead.
> My
Robert,
> In most cases we either break backwards compatibility or require some
> type of switch to turn on backwards compatibility for those who want
> it. While the above plan tries to do one better, it leaves me feeling
> that the thing I don't like about this is that it sounds like you are
> f
> I rather like Tom's suggestion of include_if_exists.
include_if_exists certainly solves the recovery.conf/recovery.done problem. We
can even phase it out, like this:
9.2: include_if_exists = 'recovery.conf' in the default postgresql.conf file.
9.3: include_if_exists = 'recovery.conf' commen
All,
I'd love to see someone evaluate the impact of Marti's patch on JDBC
applications which use named prepared statements. Anyone have a benchmark
handy?
--Josh
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql
> > Reminder: BETWEEEN supports the SYMMETRIC keyword, so there is
> > a precedent for this.
>
> And I don't see it as valuable enough to justify changing the
> grammar.
I agree that we should leave symmetry until 9.3.
--Josh Berkus
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postg
> There might be a use case for a separate directive include_if_exists,
> or some such name. But I think the user should have to tell us very
> clearly that it's okay for the file to not be found.
Better to go back to include_directory, then.
--Josh Berkus
--
Sent via pgsql-hackers mailing l
Folks,
What happens currently if we have an \include in postgresql.conf for a file
which doesn't exist? Is it ignored, or do we error out?
If it could just be ignored, maybe with a note in the logs, then we could be a
lot more flexible.
--Josh Berkus
--
Sent via pgsql-hackers mailing list (
All,
> >> We might want to have a system where an extension can declare that
> >> it
> >> "provides" capabilites, and then have another extension "require"
> >> those
> >> capabilities. That would be a neater solution to the case that
> >> there are
> >> multiple extensions that all provide the sa
> Since we haven't yet come up with a reasonable way of machine-editing
> postgresql.conf, this seems like a fairly serious objection to
> getting
> rid of recovery.conf. I wonder if there's a way we can work around
> that...
Well, we *did* actually come up with a reasonable way, but it died und
> I'm in favor of defining a separate, content-free trigger file to
> enable
> archive recovery. Not sure about the name "recovery.ready", though
> ---
> that makes it look like one of the WAL archive transfer trigger
> files,
> which does not seem like a great analogy. The pg_standby
> document
> Download numbers for the installers were bordering on noise compared
> to the GA builds last time I looked, double figures iirc. I don't
> know about the tarballs offhand and can't check ATM.
Can you check when you get a chance? I know that the DL numbers for the first
alphas were very low,
> That's not my recollection. Obviously, it's hard to measure this one
> way or the other, but I don't recall there being a lot of test
> reports
> from people who are not already contributors and could have used some
> other way to get the code.
Do we have download stats for the alphas? Dave?
All,
Where are we on RC1 or Beta4 for PostgreSQL 9.1?
While I know we're doing going to do a final release in August due to the
europeans, it would be nice to move things along before then. There don't seem
to be any blockers open.
--
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.co
Robert,
> We need to start marking the patches that are "Waiting on Author" as
> "Returned with Feedback", ideally after checking that the status in
> the CF application is in fact up to date. With a week left in the
> CommitFest at this point, anything that has been reviewed and still
> has is
Simon,
> The point I have made is that I disagree with a feature freeze date
> fixed ahead of time without regard to the content of the forthcoming
> release. I've not said I disagree with feature freezes altogether,
> which would be utterly ridiculous. Fixed dates are IMHO much less
> important t
Robert,
> Oh, I get that. I'm just dismayed that we can't have a discussion
> about the patch without getting sidetracked into a conversation about
> whether we should throw feature freeze out the window.
That's not something you can change. Whatever the patch is, even if it's a
psql improveme
> iew. The
> reason we usually skip the summer isn't actually a wholesale lack of
> people - it's because it's not so good from a publicity perspective,
> and it's hard to get all the packagers around at the same time.
Actually, the summer is *excellent* from a publicity perspective ... at least,
Robert,
> Another point is that parsing overhead is quite obviously not the
> reason for the massive performance gap between one core running simple
> selects on PostgreSQL and one core running simple selects on MySQL.
> Even if I had (further) eviscerated the parser to cover only the
> syntax tho
MauMau,
> Could you give me your frank opinions about which of 8.4 or 9.0 you
> recommend to ISVs who embed PostgreSQL?
So, first of all, you posted this question to the wrong list. pgsql-general or
pgsql-admin would have been more appropriate for this question.
That being said, I find your st
All,
> I agree that we should not reduce the support window. The fact that we
> can do in place upgrades of the data only addresses one pain point in
> upgrading. Large legacy apps require large retesting efforts when
> upgrading, often followed by lots more work renovating the code for
> backward
Chris,
> Not totally Idle thought: it would be nice if the "holding
> corporation" doesn't need a bank account, as they impose burdens of
> fees (not huge, but not providing us notable value), and more
> importantly, impose administrative burdens. Our banks like to impose
> holds on accounts any t
Dimitri,
> > I'll bet someone a fancy drink at a conference that this thread goes
> > to at least 100 posts.
>
> Of course, if we all are to argue about this bet… :)
Darn! You've uncovered by sinister plan. Foiled again!
--
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com
San Francis
Pavel,
> Actually we had to solve a issue with slow SELECT. The problem was in
> low value of JOIN_COLLAPSE_LIMITS. Can we increase a default of this
> value. I checked some complex query, and planner needed about 200ms
> for JOIN_COLLAPSE_LIMIT = 16. So some around 12 can be well.
I'm not comfo
> I think the main thing we have to think about before choosing is
> whether
> we believe that we can shorten the CFs at all. Josh's proposal had
> 3-week CFs after the first one, which makes it a lot easier to have a
> fest in November or December, but only if you really can end it on
> time.
I
> > If CF1 is June1, though, when will CF4 be? Having a CF start Dec. 1
> > is probably a bad idea.
>
> Well, I made a suggestion on this topic in my previous email on the
> subject...
I just searched backwards on this thread and I can't find it. There's been a
lot of posts.
--
Josh Berkus
P
I'll bet someone a fancy drink at a conference that this thread goes to at
least 100 posts.
Let the bikeshedding begin!
--
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com
San Francisco
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subsc
Robert,
> Tom and I were talking about starting maybe June 1, rather than July
> 1. You seem opposed but I'm not sure why.
Because I think -- strictly based on history and the complexity of the new
features -- we'll still be fixing major issues with the beta in June, which was
what Tom said as
All,
> +1 from me for keeping it as-is as well.
So it sounds like most committers want to keep the CFs on their existing
schedule for another year. Also that we don't want to branch until RC1. While
it would be nice to get some feedback from those who had bad experiences with
the CF cycle,
> Every time I've gotten pulled into discussions of setting parameters
> based on live monitoring, it's turned into a giant black hole--absorbs a
> lot of energy, nothing useful escapes from it. I credit completely
> ignoring that idea altogether, and using the simplest possible static
> sett
Tom,
> >> I'd like a pony, too. Let's be perfectly clear about this: there is
> >> no
> >> part of plpgsql that can run outside a transaction today, and
> >> probably
> >> no part of the other PLs either, and changing that "without major
> >> changes" is wishful thinking of the first order.
I alw
Robert, Tom,
> Hm ... there are people out there who think *I* get high off rejecting
> patches. I have a t-shirt to prove it. But I seem to be pretty
> ineffective at it too, judging from these numbers.
It's a question of how we reject patches, especially first-time patches. We
can reject the
Robert,
> Actually, I'd walk through fire for a 10% performance improvement if
> it meant only a *risk* to stability.
Depends on the degree of risk. MMAP has the potential to introduce instability
into areas of the code which have been completely reliable for years. Adding
20 new coredump cas
Radoslaw,
> I think 10% is quite good, as my stand-alone test of mmap vs. read
> shown that
> speed up of copying 100MB data to mem may be from ~20ms to ~100ms
> (depends on
> destination address). Of course deeper, system test simulating real
> usage will
> say more. In any case after good deals
All,
> Never, and that's not true. Heikki was being nice; I wouldn't have
> even
> slogged through it long enough to ask the questions he did before
> kicking it back as unusable. A badly formatted patch makes it
> impossible to evaluate whether the changes from a submission are
> reasonable or no
All,
While it would be nice to improve our performance on this workload, let me
point out that it's not a very important workload from the point of view of
real performance challenges. Yes, there are folks out there with 100MB
databases who only run one-liner select queries. But frankly,
Val
Radoslaw,
10% improvement isn't very impressive from a switch to mmap. What workload did
you test with? What I'd really like to see is testing with databases which are
50%, 90% and 200% the size of RAM ... that's where I'd expect the greatest gain
from limiting copying.
> Netbeans is possib
Magnus,
> That could certainly be useful, yes. But I have a feeling whomever
> tries to get that into 9.1 will be killed - but it's certainly good to
> put ont he list of things for 9.2.
Oh, no question. At some point in 9.2 we should also discuss how basebackup
considers "emtpy" directories.
Magnus, all:
It seems a bit annoying to have to do an rm -rf * $PGDATA/ before resynching a
standby using pg_basebackup. This means that I still need to wrap basebackup
in a shell script, instead of having it do everything for me ... especially if
I have multiple tablespaces.
Couldn't we have
All,
If I have the following line in pg_hba.conf:
hostreplication replication all md5
pg_basebackup -x -v -P -h master1 -U replication -D $PGDATA
pg_basebackup: could not connect to server: FATAL: no pg_hba.conf entry for
replication connection from host
All,
We left this out of 9.0; let's not leave it out of 9.1. We need an example
"replication" line in pg_hba.conf, commented out. e.g.
# host replication all samenet md5
Also, what happened to having a "replication" user defined by default? We
talked this to death last year, I tho
> I would think it would be purely syntatic sugar really, which does
> incorporate a familiar interface for those who are working in
> different
> worlds (.Net/Drupal/JAVA) etc...
I wouldn't mind having something more standard supported; I'm always looking up
the conninfo for the options I don't
The fifth alpha release for PostgreSQL version 9.1, 9.1alpha5, is now
available. There are no new major features in this alpha release as compared
to 9.1alpha4, but there are many minor bug fixes and improvements to features
added in 9.1alpha4 and earlier alpha releases. It is expected that no
Tom,
> Personally I'd vote for *not* having any such dangerous semantics as
> that. We should have learned better by now from plpgsql experience.
> I think the best idea is to throw error for ambiguous references,
> period.
As a likely heavy user of this feature, I agree with Tom here. I really
Tomas,
> I spoke to a teacher from a local university last week, mainly as we
> were looking for a place where a local PUG could meet regularly. I
> realized this could be a good opportunity to head-hunt some students
> to
> participate in this GSoC. Are we still interested in new students?
Yes,
> Making DDL serializable is *not* simple, and half-baked hacks won't
> make that situation better ...
That seemed unnecessary. Whether or not you approve of Stephen's solution, he
is dealing with a real issue.
--
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com
San Francisco
--
Sen
87 matches
Mail list logo
