Re: [HACKERS] logging in high performance systems.

2012-03-15 Thread Robert Haas
On Fri, Mar 9, 2012 at 9:53 AM, Robert Haas robertmh...@gmail.com wrote: Tom recently committed something by another author that is along similar lines to what you have here (I think).  Can you comment on whether you think more is still needed and what the differences are between that approach

Re: [HACKERS] logging in high performance systems.

2012-03-09 Thread Robert Haas
On Wed, Nov 23, 2011 at 9:28 PM, Theo Schlossnagle je...@omniti.com wrote: We have a need for logging in systems where it isn't feasible to log to disk as it negatively impacts performance. I'd like to be able to creatively solve this problem without modifying the core, but today I cannot.

Re: [HACKERS] logging in high performance systems.

2012-03-06 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: I would dismissed this out of hand at this if you said it a year ago, but I'm older and wiser now. At some point this cycle, I did some benchmarking of the subtransaction abort path, since the slowness of things like EXCEPTION blocks in PL/pgsql is a

Re: [HACKERS] logging in high performance systems.

2012-03-06 Thread Robert Haas
On Tue, Mar 6, 2012 at 2:11 PM, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: I would dismissed this out of hand at this if you said it a year ago, but I'm older and wiser now.  At some point this cycle, I did some benchmarking of the subtransaction abort path,

Re: [HACKERS] logging in high performance systems.

2012-01-19 Thread Robert Haas
On Mon, Jan 16, 2012 at 3:51 AM, Greg Smith g...@2ndquadrant.com wrote: There is an important distinction to make here.  Statement logging is one of the largest producers of logging data you want to worry about optimizing for on a high performance system.  The decision about whether to log or

Re: [HACKERS] logging in high performance systems.

2012-01-17 Thread Marti Raudsepp
On Thu, Nov 24, 2011 at 04:28, Theo Schlossnagle je...@omniti.com wrote: So... here's my first whack at solving this with some flexibility. The first thing I did was add hook points where immediate statement logging happens pre_exec and those that present duration post_exec.  These should,

Re: [HACKERS] logging in high performance systems.

2012-01-17 Thread Alvaro Herrera
Excerpts from Marti Raudsepp's message of mar ene 17 12:12:50 -0300 2012: On Thu, Nov 24, 2011 at 04:28, Theo Schlossnagle je...@omniti.com wrote: So... here's my first whack at solving this with some flexibility. The first thing I did was add hook points where immediate statement

Re: [HACKERS] logging in high performance systems.

2012-01-16 Thread Greg Smith
On 11/23/2011 09:28 PM, Theo Schlossnagle wrote: The first thing I did was add hook points where immediate statement logging happens pre_exec and those that present duration post_exec. These should, with optimization turned on, have only a few instructions of impact when no hooks are registered

Re: [HACKERS] logging in high performance systems.

2011-12-13 Thread Greg Smith
On 11/24/2011 11:33 AM, Theo Schlossnagle wrote: I see the next steps being: 1) agreeing that a problem exists (I know one does, but I suppose consensus is req'd) 2) agreeing that hooks are the right approach, if not propose a different approach. (fwiw, it's incredible common) 3) reworking

Re: [HACKERS] logging in high performance systems.

2011-12-13 Thread Michael Glaesemann
On Dec 13, 2011, at 13:57, Greg Smith wrote: With this idea still being pretty new, and several of the people popping out opinions in this thread being local--Theo, Stephen, myself--we've decided to make our local Baltimore/Washington PUG meeting this month be an excuse to hash some of

Re: [HACKERS] logging in high performance systems.

2011-12-04 Thread Martin Pihlak
On 11/24/2011 06:33 PM, Theo Schlossnagle wrote: I see the next steps being: 1) agreeing that a problem exists (I know one does, but I suppose consensus is req'd) +1, high volume logging is also a huge problem here at Skype. Some of the issues that immediately come to mind: - extracting

Re: [HACKERS] logging in high performance systems.

2011-11-27 Thread Stephen Frost
* Theo Schlossnagle (je...@omniti.com) wrote: I'd like to be able to creatively solve this problem without modifying the core, but today I cannot. I thought a hook already existed for this (there's already a module which uses a hook to log commands done as a superuser, for example).. Perhaps it

Re: [HACKERS] logging in high performance systems.

2011-11-24 Thread Theo Schlossnagle
On Wed, Nov 23, 2011 at 11:45 PM, Greg Smith g...@2ndquadrant.com wrote: My assumption has been that eventually a lossy logger was going to be necessary for busier sites, I just haven't been suffering from one enough to hack on it yet.  If it's possible to work this out in enough detail to

Re: [HACKERS] logging in high performance systems.

2011-11-24 Thread Robert Haas
On Wed, Nov 23, 2011 at 9:28 PM, Theo Schlossnagle je...@omniti.com wrote: Thoughts? Feedback? Can you add it to the next CommitFest? https://commitfest.postgresql.org/action/commitfest_view/open -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company --

Re: [HACKERS] logging in high performance systems.

2011-11-23 Thread Greg Smith
On 11/23/2011 09:28 PM, Theo Schlossnagle wrote: The second thing I did was write a sample use of those hooks to implement a completely non-blocking fifo logger. (if it would block, it drops the log line). The concept is that we could run this without risk of negative performance impact due to