[HACKERS] Re: Hot Standby query cancellation and Streaming Replication integration

2010-03-01 Thread Greg Stark
josh, nobody is talking about it because it doesn't make sense. you could only retry if it was the first query in the transaction and only if you could prove there were no side-effects outside the database and then you would have no reason to think the retry would be any more likely to work. greg

Re: [HACKERS] Hung postmaster (8.3.9)

2010-03-02 Thread Greg Stark
We should probably also check and prohibit including directories as files. On Tuesday, March 2, 2010, Tom Lane wrote: > In the meantime, it seems like we ought to take two defensive steps: -- greg -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your

Re: [HACKERS] pg_stop_backup does not complete

2010-03-02 Thread Greg Stark
On Tue, Mar 2, 2010 at 9:48 AM, Simon Riggs wrote: >> > Setting archive_mode to a command that does nothing but return true, e.g. >> > /bin/true, >> >> "return true" seems ambiguous for me. How about writing clearly >> "return a zero exit status" instead? > > Docs are already quite clear on that

Re: [HACKERS] Anyone know if Alvaro is OK?

2010-03-02 Thread Greg Stark
On Mon, Mar 1, 2010 at 11:11 PM, Chris Browne wrote: > Nobody really notices the carnage on the highways, because, > stochastically, there are such a large number of events, both positive > and negative (e.g. - millions of people making it home safely, and a > tiny number that don't) that it's dif

Re: [HACKERS] HS/SR and smart shutdown

2010-03-04 Thread Greg Stark
On Thu, Mar 4, 2010 at 12:11 PM, Fujii Masao wrote: > There is no post about this for over a month. Can I remove this > from TODO item of SR for 9.0? Thought? Objection? > Does smart shutdown still fail to shut down a slave? -- greg -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgr

Re: [HACKERS] HS/SR and smart shutdown

2010-03-04 Thread Greg Stark
On Thu, Mar 4, 2010 at 3:56 PM, Robert Haas wrote: > On Thu, Mar 4, 2010 at 10:17 AM, Fujii Masao wrote: >> >> Yes. More precisely, smart shutdown during recovery does not complete >> until recovery ends. > > Well, I don't think we should let smart shutdown just never terminate > when standby_mod

Re: [HACKERS] Re: Hot Standby query cancellation and Streaming Replication integration

2010-03-10 Thread Greg Stark
On Wed, Mar 10, 2010 at 6:29 AM, Josh Berkus wrote: > Then I increased vacuum_defer_cleanup_age to 10, which represents > about 5 minutes of transactions on the test system.  This eliminated all > query cancels for the reporting query, which takes an average of 10s. > > Next is a database bloa

Re: [HACKERS] [patch] build issues on Win32

2010-03-11 Thread Greg Stark
2010/3/10 David Fetter : >> > --disable-shared, as previously mentioned. >> >> Oh.  Well, we don't really support that, and there is a proposal on >> the table to remove it altogether from the configure script.  I >> don't think we're going to contort our source code in order to make >> a marginal

Re: [HACKERS] operator exclusion constraints

2010-03-11 Thread Greg Stark
On Thu, Mar 11, 2010 at 5:29 AM, Tom Lane wrote: >> Indexes: >>     "foo_pkey" PRIMARY KEY, btree (f1), tablespace "ts1" >>     "foo_f2_exclusion" btree (f2), tablespace "ts1" >>     "foo_f3_exclusion" btree (f3) DEFERRABLE INITIALLY DEFERRED >> Exclusion constraints: >>     "foo_f2_exclusion" EXC

Re: [HACKERS] gothic_moth, codlin_moth failures on REL8_2_STABLE

2010-03-11 Thread Greg Stark
On Wed, Mar 10, 2010 at 11:37 PM, Tom Lane wrote: > My conclusion is that this is probably a compiler bug.  Both buildfarm > animals appear to be using Sun Studio, although on different > architectures which weakens the compiler-bug theory a bit.  Even though > we are not seeing the failure in lat

Re: [HACKERS] gothic_moth, codlin_moth failures on REL8_2_STABLE

2010-03-11 Thread Greg Stark
Incidentally Zdenek came to the same conclusion that it was a compiler bug in <4aa775a9.80...@sun.com> -- greg -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] [patch] build issues on Win32

2010-03-11 Thread Greg Stark
2010/3/11 Tom Lane : > Now libpq doesn't often have critical security bugs filed against it, > but it certainly has bugs.  Do you really want to have to remember to > rebuild every piece of dependent software when you update it? I absolutely agree that linking statically is a terrible idea for dis

[HACKERS] An idle thought

2010-03-16 Thread Greg Stark
A few days ago there was a thread on one of our lists where someone was suprised it took as much i/o to delete data as it took to insert it in the first place. At first this does seem surprising but the fact that Postgres stores its transaction information inline with the data and does all i/o in b

Re: [HACKERS] Command to prune archive at restartpoints

2010-03-17 Thread Greg Stark
On Wed, Mar 17, 2010 at 9:37 AM, Heikki Linnakangas wrote: > One awkward omission in the new built-in standby mode, mainly used for > streaming replication, is that there is no easy way to delete old > archived files like you do with the %r parameter to restore_command. I'm still finding this kin

Re: [HACKERS] An idle thought

2010-03-18 Thread Greg Stark
On Thu, Mar 18, 2010 at 9:07 PM, Jeff Davis wrote: > The VM cause wrong results if a bit is set that's not supposed to be -- > right? Am I missing something? How does a seq scan skip visibility > checks and still produce right results, if it doesn't rely on the bit? > There's also a PD_ALL_VISIBL

Re: [HACKERS] Command to prune archive at restartpoints

2010-03-22 Thread Greg Stark
On Thu, Mar 18, 2010 at 9:43 AM, Simon Riggs wrote: > On Wed, 2010-03-17 at 11:37 +0200, Heikki Linnakangas wrote: > >> One awkward omission in the new built-in standby mode, mainly used for >> streaming replication, is that there is no easy way to delete old >> archived files like you do with the

Re: [HACKERS] Comments on Exclusion Constraints and related datatypes

2010-03-22 Thread Greg Stark
On Mon, Mar 22, 2010 at 1:15 PM, Simon Riggs wrote: > > * Circles, Boxes and other geometric datatypes defined "overlaps" to > include touching shapes. So > > * inet datatypes don't have a commutative operator on which a unique > index can be built. There is no "overlaps" equivalent, which again i

Re: [HACKERS] [postgis-users] ERROR: array size exceeds themaximumallowed(134217727)

2010-03-22 Thread Greg Stark
On Mon, Mar 22, 2010 at 4:45 PM, Paul Ramsey wrote: > Did you already try replacing your postgis functions with array_agg > calls to see if we can push the problem back over the fence to pgsql > land? On Sat, Mar 20, 2010 at 9:17 AM, Mike Leahy wrote: > Running this query on various data will pr

[HACKERS] More idle thoughts

2010-03-26 Thread Greg Stark
The Linux kernel had a big push to reduce latency, and one of the tricks they did was they replaced the usual interrupt points with a call which noted how long it had been since the last interrupt point. It occurs to me we could do the same for CHECK_FOR_INTERRUPTS() by conditionally having it call

[HACKERS] Re: [COMMITTERS] pgsql: Augment WAL records for btree delete with GetOldestXmin() to

2010-03-27 Thread Greg Stark
On Sat, Mar 27, 2010 at 10:10 AM, Simon Riggs wrote: > It appears that in practice many of the index items point to heap items > that are LP_DEAD. So for the purposes of accessing a heap tuple's xmin, > then we're both right. To the current purpose the tuple has been > removed, though you are also

[HACKERS] Re: [COMMITTERS] pgsql: Augment WAL records for btree delete with GetOldestXmin() to

2010-03-27 Thread Greg Stark
On Sat, Mar 27, 2010 at 7:36 PM, Simon Riggs wrote: > On Sat, 2010-03-27 at 19:15 +, Greg Stark wrote: > > If we're pruning an index entry to a heap tuple that has been HOT > > pruned wouldn't the HOT pruning record have already conflicted with > > any querie

[HACKERS] A maze of twisty mailing lists all the same

2010-04-07 Thread Greg Stark
I've often said in the past that we have too many mailing lists with overlapping and vague charters. I submit the following thread as evidence that this causes real problems. http://archives.postgresql.org/message-id/g2o4b46b5f01004010610ib8625426uae6ee90ac1435...@mail.gmail.com Because the poste

Re: [HACKERS] A maze of twisty mailing lists all the same

2010-04-08 Thread Greg Stark
On Thu, Apr 8, 2010 at 5:09 PM, Kevin Grittner wrote: > My set is different, but the principle is the same -- I can't find > the time to read all messages to all lists (really, I've tried), so > I limit by list to try to target the issues of most interest to me. But all it means is you get a rand

Re: [HACKERS] A maze of twisty mailing lists all the same

2010-04-08 Thread Greg Stark
On Thu, Apr 8, 2010 at 7:58 PM, Ned Lilly wrote: > +1 for the idea, and +1 for the Zork reference.  Hello sailor. fwiw it's older than Zork. It comes from Adventure (http://en.wikipedia.org/wiki/Colossal_Cave_Adventure) -- greg -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.

Re: [HACKERS] a faster compression algorithm for pg_dump

2010-04-08 Thread Greg Stark
On Fri, Apr 9, 2010 at 12:17 AM, Joachim Wieland wrote: > One question that I do not yet see answered is, do we risk violating a > patent even if we just link against a compression library, for example > liblzf, without shipping the actual code? > Generally patents are infringed on when the proce

Re: [HACKERS] Differential backup

2010-04-28 Thread Greg Stark
On Tue, Apr 27, 2010 at 8:15 PM, Craig Ringer wrote: > > Another avenue possibly worth investigating may be using the in-heap > mvcc information to do SQL-level differential backups of individual > tables or of the whole database. think: > You can't use the mvcc information to do incremental back

Re: [HACKERS] max_standby_delay considered harmful

2010-05-04 Thread Greg Stark
On Mon, May 3, 2010 at 4:37 PM, Tom Lane wrote: > 1. The timestamps we are reading from the log might be historical, > 2. There could be clock skew between the master and slave servers. > 3. There could be significant propagation delay from master to slave, So it sounds like what you're expecti

Re: [HACKERS] max_standby_delay considered harmful

2010-05-04 Thread Greg Stark
On Wed, May 5, 2010 at 5:01 AM, Andrew Dunstan wrote: > Our process is shot to pieces. But then, we knew that, didn't we ;-) > Franky I think these kinds of usability questions are things that we'll never have great success getting feedback on without users banging on the system. There are soluti

Re: [HACKERS] max_standby_delay considered harmful

2010-05-05 Thread Greg Stark
On Thu, May 6, 2010 at 2:36 AM, Tom Lane wrote: > One reason I believe this isn't so critical as all that is that it only > matters for cases where the operation on the master took an exclusive > lock. Uhm, or a vacuum ran. Or a HOT page cleanup occurred, or a btree page split deleted old tuples.

Re: [HACKERS] LD_LIBRARY_PATH versus rpath

2010-05-06 Thread Greg Stark
On Thu, May 6, 2010 at 12:20 AM, Tom Lane wrote: > Now, pg_regress tries to ensure that the temporary installation > will work as desired by setting LD_LIBRARY_PATH to point at the > temp installation's lib/ directory.  However, the psql executable > will by default get built with a DT_RPATH entry

Re: [HACKERS] max_standby_delay considered harmful

2010-05-09 Thread Greg Stark
On Sun, May 9, 2010 at 4:00 AM, Greg Smith wrote: >  The use cases are covered as best they can be without better support from > expected future SR features like heartbeats and XID loopback. For what it's worth I think deferring these extra complications is a very useful exercise. I would like to

Re: [HACKERS] max_standby_delay considered harmful

2010-05-10 Thread Greg Stark
On Mon, May 10, 2010 at 5:20 PM, Bruce Momjian wrote: > You are right that we are much more flexible about changing > administrative configuration parameters (like this one) than SQL. In the > past, we even renamed logging parameters to be more consistent, and I > think that proves the bar is quit

Re: [HACKERS] no universally correct setting for fsync

2010-05-10 Thread Greg Stark
On Mon, May 10, 2010 at 4:55 PM, Kevin Grittner wrote: > Robert Haas wrote: > >> "It might be safe" is a bit of a waffle.  It would be nice if we >> could provide some more clear guidance as to whether it is or is >> not, or how someone could go about testing their hardware to find >> out. > > I

Re: [HACKERS] max_standby_delay considered harmful

2010-05-12 Thread Greg Stark
On Wed, May 12, 2010 at 12:26 PM, Simon Riggs wrote: > On Wed, 2010-05-12 at 07:10 -0400, Robert Haas wrote: > >> I'm not sure what to make of this.  Sometimes not shutting down >> doesn't sound like a feature to me. > > It acts exactly the same in recovery as in normal running. It is not a > spec

Re: [HACKERS] max_standby_delay considered harmful

2010-05-12 Thread Greg Stark
On Wed, May 12, 2010 at 5:49 PM, Simon Riggs wrote: > On Wed, 2010-05-12 at 12:04 -0400, Robert Haas wrote: > >> Huh?? The evidence that this bug is linked with HS is that it occurs >> on a server running in HS mode, and not otherwise.  As for whether the >> bug is code I committed, that's certain

Re: [HACKERS] List traffic

2010-05-13 Thread Greg Stark
On Wed, May 12, 2010 at 5:24 PM, Robert Haas wrote: > The difference between discussing a patch and discussing an idea that > might lead to a patch is fairly fine. And importantly -- who would be able to subscribe to one and not the other? If you have to subscribe to both to get make any sense of

Re: [HACKERS] Row-level Locks & SERIALIZABLE transactions, postgres vs. Oracle

2010-05-13 Thread Greg Stark
On Thu, May 13, 2010 at 10:25 PM, Florian Pflug wrote: > C1: BEGIN > C1: SELECT * FROM t WHERE id = 1 FOR UPDATE > C2: BEGIN > C2: SET TRANSACTION ISOLATION LEVEL SERIALIZABLE > C2: SELECT * FROM t -- Take snapshot before C1 commits > C1: COMMIT > C2: DELETE FROM t WHERE id = 1 > C2: COMMIT > Can

Re: [HACKERS] quoting and recovery.conf

2010-05-13 Thread Greg Stark
On Thu, May 13, 2010 at 11:33 PM, Robert Haas wrote: > > I think we should add a TODO to parse recovery.conf with the same code > we use to parse postgresql.conf, or possibly merge the two files. > This issue was previously alluded to here: > > http://archives.postgresql.org/pgsql-hackers/2010-04/

Re: [HACKERS] List traffic

2010-05-14 Thread Greg Stark
On Fri, May 14, 2010 at 4:39 AM, Greg Smith wrote: > Is it > helpful to novices that they can subscribe to a list when they won't be > overwhelmed by traffic, and can ask questions without being too concerned > about being harassed for being newbies?  Probably. Only if they aren't hoping to get a

Re: [HACKERS] including PID or backend ID in relpath of temp rels

2010-05-17 Thread Greg Stark
On Tue, May 4, 2010 at 5:17 PM, Tom Lane wrote: > FWIW, that's not the case, anymore than it is for blocks in shared > buffer cache for regular rels.  smgrextend() results in an observable > extension of the file EOF immediately, whether or not you can see > up-to-date data for those pages. > > No

Re: [HACKERS] Partitioning/inherited tables vs FKs

2010-05-18 Thread Greg Stark
On Thu, May 6, 2010 at 10:38 AM, Tom Lane wrote: > * the index grows as the size of the total data set, it's not limited > by partition size > > * can't cheaply drop one partition any more, you have to vacuum the > (big) index first So I wholeheartedly agree with the general sentiment that if you

Re: [HACKERS] Clarifications of licences on pgfoundry

2010-05-18 Thread Greg Stark
On Tue, May 18, 2010 at 4:06 AM, Simon Riggs wrote: > If, as you say, the licence is unclear then whether-or-not it is an open > source licence must also be unclear. I would suggest you, or anyone else who notices, open bugs on any packages you want to use for which you find no LICENSE file match

Re: [HACKERS] Memory usage during sorting

2012-04-13 Thread Greg Stark
On Mon, Mar 19, 2012 at 7:23 PM, Robert Haas wrote: > On Sun, Mar 18, 2012 at 11:25 AM, Tom Lane wrote: >> So has somebody found a hole in the n log n lower bound on the cost of >> comparison-based sorting?  I thought that had been proven pretty >> rigorously. > > There's not much danger of anyon

Re: [HACKERS] Patch: add timing of buffer I/O requests

2012-04-13 Thread Greg Stark
On Fri, Apr 13, 2012 at 8:15 PM, Robert Haas wrote: > That's probably true, but I'm not sure it's worth worrying about - > one-in-four-billion is a pretty small probability. Is this not subject to the birthday paradox? If you have a given hash you're worried about a collision with then you have a

Re: [HACKERS] Memory usage during sorting

2012-04-14 Thread Greg Stark
On Fri, Apr 13, 2012 at 7:01 PM, Peter Geoghegan wrote: > Well, timsort is specifically designed to take advantage of pre-sorted > data. It does appear to have a lot of traction, as wikipedia points > out: I hadn't heard of it. But reading up on it it does seem like a good fit for us. It trades s

Re: [HACKERS] Memory usage during sorting

2012-04-17 Thread Greg Stark
On Mon, Apr 16, 2012 at 10:42 PM, Peter Geoghegan wrote: > All but 4 regression tests pass, but they don't really count > as failures, since they're down to an assumption in the tests that the > order certain tuples appear should be the same as our current > quicksort implementation returns them,

Re: [HACKERS] 9.3 Pre-proposal: Range Merge Join

2012-04-17 Thread Greg Stark
On Mon, Apr 16, 2012 at 10:20 PM, Simon Riggs wrote: > The thing I like most about temp indexes is that they needn't be temporary. > > I'd like to see something along the lines of demand-created optional > indexes, that we reclaim space/maintenance overhead on according to > some cache management

Re: [HACKERS] Gsoc2012 idea, tablesample

2012-04-17 Thread Greg Stark
On Tue, Apr 17, 2012 at 2:49 PM, Stephen Frost wrote: > * Heikki Linnakangas (heikki.linnakan...@enterprisedb.com) wrote: >> 1. We probably don't want the SQL syntax to be added to the grammar. >> This should be written as an extension, using custom functions as >> the API, instead of extra SQL sy

Re: [HACKERS] Gsoc2012 idea, tablesample

2012-04-17 Thread Greg Stark
On Tue, Apr 17, 2012 at 5:33 PM, Christopher Browne wrote: > I get the feeling that this is a somewhat-magical feature (in that > users haven't much hope of understanding in what ways the results are > deterministic) that is sufficiently "magical" that anyone serious > about their result sets is l

[HACKERS] Re: patch submission: truncate trailing nulls from heap rows to reduce the size of the null bitmap

2012-04-17 Thread Greg Stark
On Tue, Apr 17, 2012 at 5:38 PM, Tom Lane wrote: > This has been discussed before, but it always seemed that the > cost-benefit ratio was exceedingly questionable.  You don't get any > savings whatsoever unless you reduce the size of the null bitmap across > a MAXALIGN boundary, which more and mor

Re: [HACKERS] Parameterized-path cost comparisons need some work

2012-04-18 Thread Greg Stark
On Tue, Apr 17, 2012 at 5:14 PM, Tom Lane wrote: > I've been hacking away on a patch to do this, and attached is something > that I think is pretty close to committable. I have nothing to add but I just wanted to say thank you for taking the time to write up this explanation. Even when some of us

Re: [HACKERS] 9.3: summary of corruption detection / checksums / CRCs discussion

2012-04-21 Thread Greg Stark
On Sat, Apr 21, 2012 at 10:40 PM, Jeff Davis wrote: > * In addition to detecting random garbage, we also need to be able to > detect zeroing of pages. Right now, a zero page is not considered > corrupt, so that's a problem. We'll need to WAL table extension > operations, and we'll need to mitigate

Re: [HACKERS] Timsort performance, quicksort (was: Re: Memory usage during sorting)

2012-04-24 Thread Greg Stark
On Tue, Apr 24, 2012 at 4:17 PM, Robert Haas wrote: > Based on that, I'm inclined to propose rejiggering things so that the > presorted-input check runs only at the top level, and not during any > recursive steps. Just a thought. What about running only every nth step. Maybe something like every

Re: [HACKERS] 9.3: summary of corruption detection / checksums / CRCs discussion

2012-04-24 Thread Greg Stark
On Tue, Apr 24, 2012 at 9:40 PM, Robert Haas wrote: >  For three things, index pages > have hint-type changes that are not single-bit changes. ? Just how big are these? Part of the reason hint bit updates are safe is because one bit definitely absolutely has to be entirely in one page. You can't

Re: [HACKERS] remove dead ports?

2012-04-24 Thread Greg Stark
On Tue, Apr 24, 2012 at 9:49 PM, Robert Haas wrote: >  I'm > suspicious of s_lock.h's support for National Semiconductor 32K, > Renesas' M32R, Renesas' SuperH, UNIVEL, SINIX / Reliant UNIX, > Nextstep, and Sun3 Were there ever multiprocessor Nextstep or Sun3 machines anyways? Wouldn't someone on

Re: [HACKERS] Patch: add timing of buffer I/O requests

2012-04-25 Thread Greg Stark
On Wed, Apr 25, 2012 at 5:47 PM, Tom Lane wrote: > Also, as was pointed out upthread, the underlying data in shared memory > is almost certainly never going to be infinite-precision; so using > numeric in the API seems to me to be more likely to convey a false > impression of exactness than to do

[HACKERS] Re: patch submission: truncate trailing nulls from heap rows to reduce the size of the null bitmap

2012-04-26 Thread Greg Stark
On Thu, Apr 26, 2012 at 8:27 AM, Simon Riggs wrote: >> The source table had 5 integer columns, and was populated with 10 million >> rows. ... >>   2) target has all nullable columns, only the first column is set: the >> patch was slightly faster ... >> By slightly faster I'm talking on order of 10

Re: [HACKERS] Temporary tables under hot standby

2012-04-26 Thread Greg Stark
On Thu, Apr 26, 2012 at 9:18 AM, Simon Riggs wrote: > Implementing a feature that *requires* those things is madness and > obscuring those crucial points is not balanced or fair. I think this whole discussion started the wrong way around. If the goal of implementing GTTs is to solve a need with r

[HACKERS] Re: patch submission: truncate trailing nulls from heap rows to reduce the size of the null bitmap

2012-04-28 Thread Greg Stark
On Fri, Apr 27, 2012 at 1:51 AM, Josh Berkus wrote: > 1. Out of 700 columns, columns 301+ are all Null, so we map them away. > 2. User updates column 688 to non-null > 3. Suddenly we have a MUCH larger row which will no longer fit on the page. Note that this is only actually 48 bytes more in the

Re: [HACKERS] smart shutdown at end of transaction (was: Default mode for shutdown)

2012-04-30 Thread Greg Stark
On Mon, Apr 30, 2012 at 9:55 AM, Wolfgang Wilhelm wrote: > Just for the ones interested in a view on another turf: > > In Oracle "shutdown immediate" is the fastest _clean_ shutdown and "shutdown > abort" is equal to "shutdown immediate" in PG. > The other modes are called "shutdown normal" and "s

Re: [HACKERS] Patch: add timing of buffer I/O requests

2012-04-30 Thread Greg Stark
On Sun, Apr 29, 2012 at 12:26 AM, Robert Haas wrote: > As for track_iotiming -> track_io_timing, I'm fine with that as well. I'm still grumpy about the idea of a GUC changing the explain analyze output. How would people feel about adding an explain option that explicitly requests io timing for th

Re: [HACKERS] Gsoc2012 idea, tablesample

2012-05-11 Thread Greg Stark
On Fri, May 11, 2012 at 6:16 PM, Kevin Grittner wrote: >> MaxHeapTuplesPerPage? > > What about dead line pointers without corresponding tuples? Actually we don't allow there to be more than MaxHeapTuplesPerPage line pointers even if some of them are dead line pointers. I think the argument then

Re: [HACKERS] Why is indexonlyscan so darned slow?

2012-05-23 Thread Greg Stark
On Mon, May 21, 2012 at 9:37 PM, Simon Riggs wrote: >>> This is exactly what we do for VACUUM and it works faster there. >> >> The reason that's okay for vacuum is that vacuum doesn't care if it >> visits the same index tuple multiple times.  It will not work for real >> queries, unless you would

Re: [HACKERS] 9.3: load path to mitigate load penalty for checksums

2012-06-12 Thread Greg Stark
On Tue, Jun 12, 2012 at 5:41 PM, Merlin Moncure wrote: > Also it's not safe to assume that insertion heavy clients > can be migrated to COPY.  For example, JDBC bulk loading AFAIK does > not use COPY and even if it did wouldn't be able to decorate the > command for a long time for most production

[HACKERS] Re: [COMMITTERS] pgsql: Send new protocol keepalive messages to standby servers.

2012-06-13 Thread Greg Stark
On Mon, Jun 4, 2012 at 8:49 PM, Christopher Browne wrote: >> What if the two servers are in different time zones? > > NTP shouldn't have any problem; it uses UTC underneath.  As does > PostgreSQL, underneath. As an aside, this is not strictly speaking true. NTP doesn't "use UTC" -- afaik it doesn

Re: [HACKERS] sortsupport for text

2012-06-20 Thread Greg Stark
On Sun, Jun 17, 2012 at 9:26 PM, Tom Lane wrote: > The trick for hashing such datatypes is to be able to guarantee that > "equal" values hash to the same hash code, which is typically possible > as long as you know the equality rules well enough.  We could possibly > do that for text with pure-str

Re: [HACKERS] sortsupport for text

2012-06-20 Thread Greg Stark
On Wed, Jun 20, 2012 at 3:19 PM, Peter Geoghegan wrote: >> It occurs to me that strxfrm would answer this question. If we made >> the hash function hash the result of strxfrm then we could make >> equality use strcoll and not fall back to strcmp. > > What about per-column collations? Well collati

Re: [HACKERS] WAL format changes

2012-07-06 Thread Greg Stark
On Thu, Jun 14, 2012 at 10:01 PM, Heikki Linnakangas wrote: > This has the advantage that you can calculate the CRC for all the other > fields before acquiring WALInsertLock. For xl_prev, you need to know where > exactly the record is inserted, so it's handy that it's the last field > before CRC.

[HACKERS] Re: Allow replacement of bloated primary key indexes without foreign key rebuilds

2012-07-10 Thread Greg Stark
On Sat, Jul 7, 2012 at 4:53 AM, Gurjeet Singh wrote: > All we need to do is allow swapping of pg_class.relfilenode of two indexes. > This will let the dependency entries stand as they are and allow us to drop > the bloated primary key index structure without having to rebuild the > foreign key con

Re: [HACKERS] Re: Allow replacement of bloated primary key indexes without foreign key rebuilds

2012-07-10 Thread Greg Stark
On Tue, Jul 10, 2012 at 3:44 PM, Tom Lane wrote: >> The problem you describe is one of constraints and dependencies and >> not one of indexes. It seems what you really want is a way to alter >> foreign key dependencies to depend on a new index. Either an explicit >> command that lets you set the n

[HACKERS] Re: Checkpointer split has broken things dramatically (was Re: DELETE vs TRUNCATE explanation)

2012-07-24 Thread Greg Stark
On Wed, Jul 18, 2012 at 1:13 AM, Craig Ringer wrote: > That makes me wonder if on top of the buildfarm, extending some buildfarm > machines into a "crashfarm" is needed: > > - Keep kvm instances with copy-on-write snapshot disks and the build env > on them > - Fire up the VM, do a build, and star

Re: [HACKERS] Path question

2010-09-01 Thread Greg Stark
2010/9/1 Boszormenyi Zoltan : > we are experimenting with modifying table partitioning > so the ORDER BY clause can be pushed down to > child nodes on the grounds that: > 1. smaller datasets are faster to sort, e.g. two datasets that almost >    spill out to disk are faster to sort in memory and la

Re: [HACKERS] Streaming a base backup from master

2010-09-03 Thread Greg Stark
On Fri, Sep 3, 2010 at 12:19 PM, Heikki Linnakangas wrote: > * We need a smarter way to do pg_start/stop_backup() with this. At the > moment, you can only have one backup running at a time, but we shouldn't > have that limitation with this built-in mechanism. Well there's no particular reason we

Re: [HACKERS] Streaming a base backup from master

2010-09-04 Thread Greg Stark
On Fri, Sep 3, 2010 at 8:30 PM, Martijn van Oosterhout wrote: > > rsync is not rocket science. All you need is for the receiving end to > send a checksum for each block it has. The server side does the same > checksum and for each block sends back "same" or "new data". Well rsync is closer to roc

[HACKERS] Re: Interruptible sleeps (was Re: CommitFest 2009-07: Yay, Kevin! Thanks, reviewers!)

2010-09-05 Thread Greg Stark
On Fri, Sep 3, 2010 at 9:20 PM, Heikki Linnakangas wrote: > So we now have the same process nested twice inside a semop() call. Looking > at the Linux signal (7) man page, it is undefined what happens if semop() is > re-entered in a signal handler while another semop() call is happening in > main

Re: [HACKERS] Streaming a base backup from master

2010-09-06 Thread Greg Stark
On Sun, Sep 5, 2010 at 4:51 PM, Martijn van Oosterhout wrote: > If you're working from a known good version of the database at some > point, yes you are right you have more interesting options. If you > don't you want something that will fix it. Sure, in that case you want to restore from backup

Re: [HACKERS] Synchronization levels in SR

2010-09-06 Thread Greg Stark
On Mon, Sep 6, 2010 at 10:02 PM, Simon Riggs wrote: > Then I respectfully suggest that you're releasing locks too early. > > Your proposal would allow a 2nd user to see the results of the 1st > user's transaction before the 1st user knew about whether it had > committed or not. > > I know why you

Re: [HACKERS] What happened to the is_ family of functions proposal?

2010-09-21 Thread Greg Stark
On Tue, Sep 21, 2010 at 6:02 PM, Alvaro Herrera wrote: > So we could refactor the input functions so that there's an internal > function that returns the accepted datum in the OK case and an ErrorData > for the failure case.  The regular input function would just throw the > error data in the latt

[HACKERS] Magnus? Is that you?

2010-09-24 Thread Greg Stark
Some voter in Sweden has an interesting sense of humour http://alicebobandmallory.com/articles/2010/09/23/did-little-bobby-tables-migrate-to-sweden -- greg -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/

Re: [HACKERS] Serializable Snapshot Isolation

2010-09-25 Thread Greg Stark
On Thu, Sep 23, 2010 at 4:08 PM, Kevin Grittner wrote: > One place I'm particularly interested in using such a feature is in > pg_dump. Without it we have the choice of using a SERIALIZABLE > transaction, which might fail or cause failures (which doesn't seem > good for a backup program) or using

Re: [HACKERS] Serializable Snapshot Isolation

2010-09-25 Thread Greg Stark
On Sat, Sep 25, 2010 at 4:24 PM, Kevin Grittner wrote: > OK, to get back to the question -- pg_dump's transaction (T0) could > see an inconsistent version of the database if one transaction (TN) > writes to a table, another transaction (T1) overlaps TN and can't > read something written by TN beca

Re: [HACKERS] bg worker: general purpose requirements

2010-09-25 Thread Greg Stark
On Sat, Sep 18, 2010 at 4:21 AM, Robert Haas wrote: >> (It's exactly what apache pre-fork does, no? Is anybody concerned about the >> idle processes there? Or do they consume much less resources?) > > I don't know whether an idle Apache worker consumes more or less > memory than an idle Postg

Re: [HACKERS] Serializable Snapshot Isolation

2010-09-25 Thread Greg Stark
Just to be clear I wasn't saying it was or wasn't a problem, I was just trying to see if I understand the problem and if I do maybe help bring others up to speed. On 25 Sep 2010 23:28, "Kevin Grittner" wrote: > Greg Stark wrote: > >> So T1 must have happened befo

Re: [HACKERS] Per-column collation, work in progress

2010-09-26 Thread Greg Stark
On Sun, Sep 26, 2010 at 1:15 PM, Pavel Stehule wrote: > Is there any reason why you prohibit a different encodings per one > database? Actually people expect from collate per column a possibility > to store a two or more different encodings per one database. These are two completely separate prob

Re: [HACKERS] Path question

2010-09-29 Thread Greg Stark
2010/9/23 Robert Haas : > All of this leaves me wondering why Greg ended up ifdefing out this > code in the first place.  There's probably something I'm missing > here...  but for now I can't think of a better idea than just removing > the #ifdefs and hoping that whatever problem they were causing

Re: [HACKERS] O_DSYNC broken on MacOS X?

2010-09-30 Thread Greg Stark
On Thu, Sep 30, 2010 at 2:22 PM, A.M. wrote: > That is not correct. fsync and friends on Darwin synchronizes I/O and flushes > dirty kernel caches to the disk which meets the specification and is > distinctly different from doing nothing. How exactly is it different from doing nothing? That is,

[HACKERS] Adding getrusage profiling data to EXPLAIN output

2010-09-30 Thread Greg Stark
Attached is a patch to display getrusage output to EXPLAIN output. This is the patch I mentioned previously in http://archives.postgresql.org/pgsql-hackers/2010-02/msg00684.php and it raises the same issues we were talking about then. Should the resource usage stats displayed be per-iteration total

Re: [HACKERS] Adding getrusage profiling data to EXPLAIN output

2010-10-01 Thread Greg Stark
On Thu, Sep 30, 2010 at 6:51 PM, Itagaki Takahiro wrote: > How much overhead do you have with "resource" option? > getrusage() calls for each tuple might have considerable overheads. > How much difference between (analyze) and (analyze, resource) options? Here's strace -c for a select count(*) fr

Re: [HACKERS] git diff --patience

2010-10-01 Thread Greg Stark
On Fri, Oct 1, 2010 at 7:15 AM, Kevin Grittner wrote: > An interesting exercise it so think about what > real-life lines you could have which would have multiple occurrences > in this pattern, and think about whether you would then prefer the > --patience output, especially if this were part of a

Re: [HACKERS] wip: functions median and percentile

2010-10-03 Thread Greg Stark
On Sun, Oct 3, 2010 at 7:06 AM, Hitoshi Harada wrote: > And I'm now thinking about how to make median happen in window > aggregate. If you were to do this by extending tuplesort what extra features would tuplesort need? Would tuplesort need the ability to insert additional records into an alread

Re: [HACKERS] wip: functions median and percentile

2010-10-04 Thread Greg Stark
On Sun, Oct 3, 2010 at 11:58 PM, Dean Rasheed wrote: > The problem is that performance really sucks, > because it is an O(n^2 log(n)) algorithm. I don't see an easy way > around that without significant new infrastructure, as Greg describes, > or a completely different algorithm. Resorting for ea

Re: [HACKERS] Adding getrusage profiling data to EXPLAIN output

2010-10-04 Thread Greg Stark
On Mon, Oct 4, 2010 at 3:29 AM, Itagaki Takahiro wrote: >  * There are some overlaps between the feature and DTrace hooks. >   If we need such extension even though we have DTrace hooks, >   it might mean DTrace hooks are hard to use for average users >   and maybe also for postgres' hackers... I

Re: [HACKERS] MIT benchmarks pgsql multicore (up to 48)performance

2010-10-04 Thread Greg Stark
Here's a video on lock-free hashing for example: http://video.google.com/videoplay?docid=2139967204534450862# I guess by "lock-free in the uncontended case" they mean the buffer cache manager is lock-free unless you're actually contending on the same buffer? -- Sent via pgsql-hackers mailing li

Re: [HACKERS] leaky views, yet again

2010-10-05 Thread Greg Stark
On Tue, Oct 5, 2010 at 11:01 AM, Robert Haas wrote: > Well, the only thing I've ever wanted to do this for was to allow > sales reps to see their own customers but not the customers of other > sales reps (because if they could pull our complete customer list, > then once they left and went to work

Re: [HACKERS] host name support in pg_hba.conf

2010-10-06 Thread Greg Stark
On Wed, Oct 6, 2010 at 6:21 AM, Magnus Hagander wrote: > It's not common, but i've certainly come across a number of virtual > machines where localhost resolves (through /etc/hosts) to the machines > "real" IP rather than 127.0.01, because 127.0.0.1 simply doesn't > exist. It's perfectly fine for

Re: [HACKERS] On Scalability

2010-10-07 Thread Greg Stark
Firstly I want to say I think this discussion is over-looking some benefits of the current system in other use cases. I don't think we should get rid of the current system even once we have "proper" partitioning. It solves use cases such as data warehouse queries that need to do a full table scan o

Re: [HACKERS] standby registration (was: is sync rep stalled?)

2010-10-07 Thread Greg Stark
On Thu, Oct 7, 2010 at 10:27 AM, Heikki Linnakangas wrote: > The standby name is a GUC in the standby's configuration file: > > standby_name='bostonserver' > Fwiw I was hoping it would be possible to set every machine up with an identical postgresql.conf file. That doesn't preclude this idea sinc

Re: [HACKERS] On Scalability

2010-10-08 Thread Greg Stark
On Fri, Oct 8, 2010 at 3:20 AM, Vincenzo Romano wrote: > Do the same conclusions apply to partial indexes? > I mean, if I have a large number (n>=100 or n>=1000) of partial indexes > on a single very large table (m>=10**12), how good is the planner to choose > the > right indexes to plan a query?

Re: [HACKERS] wip: functions median and percentile

2010-10-11 Thread Greg Stark
On Sun, Oct 10, 2010 at 2:16 PM, Tom Lane wrote: > It was pointed out upthread that while median isn't presently > in the standard, Oracle defines it in terms of percentile_cont(0.5) > which *is* in the standard. Uhmm, then why don't we implement that? We could provide median() as a short-cut but

Re: [HACKERS] Issues with two-server Synch Rep

2010-10-11 Thread Greg Stark
On Mon, Oct 11, 2010 at 5:41 PM, Robert Haas wrote: > That's probably not going to happen until we have a way to update > postgresql.conf via SQL.  Which, I maintain, as I have maintained > before, is not going to happen until we get rid of the comments, > because otherwise absolutely any implemen

<    3   4   5   6   7   8   9   10   11   12   >