Re: [HACKERS] Errors on missing pg_subtrans/ files with 9.3

2013-12-10 Thread Andres Freund
On 2013-12-10 11:12:03 -0800, Josh Berkus wrote: > On 12/10/2013 10:48 AM, Andres Freund wrote: > > On 2013-12-10 10:44:30 -0800, Josh Berkus wrote: > >> On 12/10/2013 10:39 AM, Andres Freund wrote: > >>> Hi, > >>> > >>> On 2013-12-10 10:38:32 -0800, Josh Berkus wrote: > We've just run across

Re: [HACKERS] Errors on missing pg_subtrans/ files with 9.3

2013-12-10 Thread Josh Berkus
On 12/10/2013 10:48 AM, Andres Freund wrote: > On 2013-12-10 10:44:30 -0800, Josh Berkus wrote: >> On 12/10/2013 10:39 AM, Andres Freund wrote: >>> Hi, >>> >>> On 2013-12-10 10:38:32 -0800, Josh Berkus wrote: We've just run across a case of this exact issue on 9.2.4. I thought it was sup

Re: [HACKERS] Errors on missing pg_subtrans/ files with 9.3

2013-12-10 Thread Andres Freund
On 2013-12-10 10:44:30 -0800, Josh Berkus wrote: > On 12/10/2013 10:39 AM, Andres Freund wrote: > > Hi, > > > > On 2013-12-10 10:38:32 -0800, Josh Berkus wrote: > >> We've just run across a case of this exact issue on 9.2.4. I thought it > >> was supposed to be 9.3-only? > > > > Could you please

Re: [HACKERS] Errors on missing pg_subtrans/ files with 9.3

2013-12-10 Thread Josh Berkus
On 12/10/2013 10:39 AM, Andres Freund wrote: > Hi, > > On 2013-12-10 10:38:32 -0800, Josh Berkus wrote: >> We've just run across a case of this exact issue on 9.2.4. I thought it >> was supposed to be 9.3-only? > > Could you please describe "this exact issue"? Fatal errors due to missing pg_sub

Re: [HACKERS] Errors on missing pg_subtrans/ files with 9.3

2013-12-10 Thread Andres Freund
Hi, On 2013-12-10 10:38:32 -0800, Josh Berkus wrote: > We've just run across a case of this exact issue on 9.2.4. I thought it > was supposed to be 9.3-only? Could you please describe "this exact issue"? Greetings, Andres Freund -- Andres Freund http://www.2ndQuadrant.co

Re: [HACKERS] Errors on missing pg_subtrans/ files with 9.3

2013-12-10 Thread Josh Berkus
Andres, all: We've just run across a case of this exact issue on 9.2.4. I thought it was supposed to be 9.3-only? -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www

Re: [HACKERS] Errors on missing pg_subtrans/ files with 9.3

2013-12-05 Thread J Smith
On Thu, Nov 28, 2013 at 5:15 AM, Andres Freund wrote: > Hi, > > Do you still have the core file around? If so could you 'p > *ShmemVariableCache' and 'p *ControlFile'? > So sorry, I didn't see this message until just today. Seems it was accidentally archived before hitting my eyeballs. I see tha

Re: [HACKERS] Errors on missing pg_subtrans/ files with 9.3

2013-11-28 Thread Andres Freund
Hi, On 2013-11-24 16:56:26 -0500, J Smith wrote: > coredumper worked like a charm. Useful tool, that is... although as a > bit of advice, I'd try not to run it on Postgres if your various > memory settings are tweaked towards production use -- the core dump > that was captured on my server weighed

Re: [HACKERS] Errors on missing pg_subtrans/ files with 9.3

2013-11-27 Thread Andres Freund
On 2013-11-27 13:57:52 -0300, Alvaro Herrera wrote: > Per bug report by J Smith in > cadfupgc5bmtv-yg9znxv-vcfkb+jprqs7m2oesqxam_4z1j...@mail.gmail.com > diagnosed by Andres Freund. Alvaro, do you see a way this could actually have caused J.'s problems? I thought about a few, but each

Re: [HACKERS] Errors on missing pg_subtrans/ files with 9.3

2013-11-27 Thread Alvaro Herrera
Alvaro Herrera escribió: > Andres Freund escribió: > This seems simple to handle by adding the check you propose to the loop. > Basically if the xmax doesn't match the xmin, we reached the end, > there's nothing more to lock and we can return success without any > further work: As mentioned in th

Re: [HACKERS] Errors on missing pg_subtrans/ files with 9.3

2013-11-25 Thread Andres Freund
On 2013-11-25 18:06:30 -0300, Alvaro Herrera wrote: > > I mean that in the !KEYS_UPDATED case we don't need to abort if we're > > only acquiring a key share... > > Hm, I think that's correct -- we don't need to abort. But we still need > to wait until the updater completes. So this proposed patc

Re: [HACKERS] Errors on missing pg_subtrans/ files with 9.3

2013-11-25 Thread Alvaro Herrera
Andres Freund escribió: > On 2013-11-25 17:10:39 -0300, Alvaro Herrera wrote: > > Let me point out that this is exactly the same code that would be > > affected by my proposed fix for #8434, which would have this check the > > updateXid in all cases, not only in KEYS_UPDATED as currently. > > Hm.

Re: [HACKERS] Errors on missing pg_subtrans/ files with 9.3

2013-11-25 Thread Andres Freund
On 2013-11-25 17:10:39 -0300, Alvaro Herrera wrote: > > I am not sure whether that's the origin of the problem but at the very > > least it seems to me that heap_lock_updated_tuple_rec() is missing > > several important pieces: > > a) do the priorXmax==xmin dance to check we're still following the

Re: [HACKERS] Errors on missing pg_subtrans/ files with 9.3

2013-11-25 Thread Alvaro Herrera
Andres Freund escribió: > Ok, this is helpful. Do you rather longrunning transactions? The > transaction that does foreign key checks has an xid of 10260613, while > the row that's getting checked has 13514992. Thanks for the analysis. > #4 0x00635dc7 in XactLockTableWait (xid=13514992)

Re: [HACKERS] Errors on missing pg_subtrans/ files with 9.3

2013-11-25 Thread J Smith
On Mon, Nov 25, 2013 at 11:46 AM, Alvaro Herrera wrote: > J Smith escribió: > >> We did have some long-running transactions, yes. We refactored a bit >> and removed them and the problem ceased on our end. We ended up >> reverting our changes for the sake of running this experiment over the >> week

Re: [HACKERS] Errors on missing pg_subtrans/ files with 9.3

2013-11-25 Thread Alvaro Herrera
J Smith escribió: > We did have some long-running transactions, yes. We refactored a bit > and removed them and the problem ceased on our end. We ended up > reverting our changes for the sake of running this experiment over the > weekend and the errors returned. We've since restored our fix and >

Re: [HACKERS] Errors on missing pg_subtrans/ files with 9.3

2013-11-25 Thread J Smith
On Mon, Nov 25, 2013 at 6:47 AM, Andres Freund wrote: > Hi, > > On 2013-11-24 16:56:26 -0500, J Smith wrote: > >> Nov 23 14:38:32 dev postgres[23810]: [4-1] user=dev,db=dev ERROR: could not >> access status of transaction 13514992 >> Nov 23 14:38:32 dev postgres[23810]: [4-2] user=dev,db=dev DET

Re: [HACKERS] Errors on missing pg_subtrans/ files with 9.3

2013-11-25 Thread Andres Freund
Hi, On 2013-11-24 16:56:26 -0500, J Smith wrote: > coredumper worked like a charm. Useful tool, that is... although as a > bit of advice, I'd try not to run it on Postgres if your various > memory settings are tweaked towards production use -- the core dump > that was captured on my server weighed

Re: [HACKERS] Errors on missing pg_subtrans/ files with 9.3

2013-11-24 Thread J Smith
coredumper worked like a charm. Useful tool, that is... although as a bit of advice, I'd try not to run it on Postgres if your various memory settings are tweaked towards production use -- the core dump that was captured on my server weighed in at 16 GB. Anyways, I've attached both the log entries

Re: [HACKERS] Errors on missing pg_subtrans/ files with 9.3

2013-11-23 Thread J Smith
On Tue, Nov 19, 2013 at 10:16 AM, J Smith wrote: > Alright, we'll look into doing that heading into the weekend. > Interestingly, we haven't experienced the issue since our main Java > developer made some modifications to our backend system. I'm not > entirely sure what the changes entail except t

Re: [HACKERS] Errors on missing pg_subtrans/ files with 9.3

2013-11-19 Thread J Smith
Alright, we'll look into doing that heading into the weekend. Interestingly, we haven't experienced the issue since our main Java developer made some modifications to our backend system. I'm not entirely sure what the changes entail except that it's a one-liner that involves re-SELECTing a table du

Re: [HACKERS] Errors on missing pg_subtrans/ files with 9.3

2013-11-19 Thread Robert Haas
On Fri, Nov 15, 2013 at 4:01 PM, J Smith wrote: > On Fri, Nov 15, 2013 at 3:21 PM, Robert Haas wrote: >> I think what would help the most is if you could arrange to obtain a >> stack backtrace at the point when the error is thrown. Maybe put a >> long sleep call in just before the error happens,

Re: [HACKERS] Errors on missing pg_subtrans/ files with 9.3

2013-11-15 Thread J Smith
On Fri, Nov 15, 2013 at 3:21 PM, Robert Haas wrote: > > I think what would help the most is if you could arrange to obtain a > stack backtrace at the point when the error is thrown. Maybe put a > long sleep call in just before the error happens, and when it gets > stuck there, attach gdb and run

Re: [HACKERS] Errors on missing pg_subtrans/ files with 9.3

2013-11-15 Thread Robert Haas
On Wed, Nov 13, 2013 at 12:29 PM, J Smith wrote: > Looks like we got another set of errors overnight. Here's the log file > from the errors. (Log file scrubbed slightly to remove private data, > but still representative of the problem I believe.) > > Nov 13 05:34:34 dev postgres[6084]: [4-1] user=

Re: [HACKERS] Errors on missing pg_subtrans/ files with 9.3

2013-11-13 Thread J Smith
Looks like we got another set of errors overnight. Here's the log file from the errors. (Log file scrubbed slightly to remove private data, but still representative of the problem I believe.) Nov 13 05:34:34 dev postgres[6084]: [4-1] user=dev,db=dev ERROR: could not access status of transaction 63

Re: [HACKERS] Errors on missing pg_subtrans/ files with 9.3

2013-11-12 Thread J Smith
On Tue, Nov 12, 2013 at 11:55 AM, Andres Freund wrote: > Hi, > > On 2013-11-12 11:46:19 -0500, J Smith wrote: >> > * Does SELECT count(*) FROM pg_prepared_xacts; return 0? >> >> Yes it does. > > Could you show the output? Do you actually use prepared xacts actively? jay:dev@jagger=# select * from

Re: [HACKERS] Errors on missing pg_subtrans/ files with 9.3

2013-11-12 Thread J Smith
On Tue, Nov 12, 2013 at 11:54 AM, Stephen Frost wrote: > > Did you also upgrade to PostGIS 2.x as part of this..? Seems like it'd > be unrelated, but one never knows. Any chance you could distill this > down into a small test case which exhibits the problem? I'm guessing > 'no', but figured I'd

Re: [HACKERS] Errors on missing pg_subtrans/ files with 9.3

2013-11-12 Thread Andres Freund
Hi, On 2013-11-12 11:46:19 -0500, J Smith wrote: > > * Does SELECT count(*) FROM pg_prepared_xacts; return 0? > > Yes it does. Could you show the output? Do you actually use prepared xacts actively? Do you actively use row level locking? Is there high concurrency in that environment? In short,

Re: [HACKERS] Errors on missing pg_subtrans/ files with 9.3

2013-11-12 Thread Stephen Frost
* J Smith (dark.panda+li...@gmail.com) wrote: > We haven't been able to use pg_upgrade as we rely heavily on PostGIS > and do hard upgrades via pg_dump and postgis_restore.pl when we > upgrade. Did you also upgrade to PostGIS 2.x as part of this..? Seems like it'd be unrelated, but one never know

Re: [HACKERS] Errors on missing pg_subtrans/ files with 9.3

2013-11-12 Thread Stephen Frost
* Andres Freund (and...@2ndquadrant.com) wrote: > He referred to using pg_dumpall/pg_dump. But that bug was erroring out > on pg_clog, not pg_subtrans, right? Yeah, that was pg_clog. Obviously responded before really looking at it. :) > My gut feeling is thats it's related to foreign key locks d

Re: [HACKERS] Errors on missing pg_subtrans/ files with 9.3

2013-11-12 Thread J Smith
G'day Andres. On Tue, Nov 12, 2013 at 11:13 AM, Andres Freund wrote: > Hi, > > On 2013-11-12 10:56:55 -0500, J Smith wrote: >> G'day list. Didn't get any interest in pgsql-general, thought I'd try >> my luck here, which perhaps would be more fitting in case I've >> stumbled upon an edge case issu

Re: [HACKERS] Errors on missing pg_subtrans/ files with 9.3

2013-11-12 Thread J Smith
On Tue, Nov 12, 2013 at 11:25 AM, Stephen Frost wrote: > > How was this upgrade done? If you used pg_upgrade, what version of the > pg_upgrade code did you use? As I recall, there was a bug in older > versions which could exhibit in this way.. > > http://wiki.postgresql.org/wiki/20110408pg_upgra

Re: [HACKERS] Errors on missing pg_subtrans/ files with 9.3

2013-11-12 Thread Andres Freund
On 2013-11-12 11:25:03 -0500, Stephen Frost wrote: > * J Smith (dark.panda+li...@gmail.com) wrote: > > I've recently upgraded a number of servers from PostgreSQL 9.2.5 to > > 9.3.1 and have started getting the following errors every couple of > > hours along with some failed transactions. > > How

Re: [HACKERS] Errors on missing pg_subtrans/ files with 9.3

2013-11-12 Thread Stephen Frost
* J Smith (dark.panda+li...@gmail.com) wrote: > I've recently upgraded a number of servers from PostgreSQL 9.2.5 to > 9.3.1 and have started getting the following errors every couple of > hours along with some failed transactions. How was this upgrade done? If you used pg_upgrade, what version of

Re: [HACKERS] Errors on missing pg_subtrans/ files with 9.3

2013-11-12 Thread Andres Freund
Hi, On 2013-11-12 10:56:55 -0500, J Smith wrote: > G'day list. Didn't get any interest in pgsql-general, thought I'd try > my luck here, which perhaps would be more fitting in case I've > stumbled upon an edge case issue or something... Normally the bug report for/the -bugs mailing list is the ri