Re: [PERFORM] performance for high-volume log insertion

2009-04-20 Thread Stephen Frost
David, * da...@lang.hm (da...@lang.hm) wrote: > is this as simple as creating a database and doing an explain on each of > these? or do I need to actually measure the time (at which point the > specific hardware and tuning settings become an issue again) No, you need to measure the time. An

Re: [PERFORM] performance for high-volume log insertion

2009-04-20 Thread Stephen Frost
* da...@lang.hm (da...@lang.hm) wrote: > while I fully understand the 'benchmark your situation' need, this isn't > that simple. It really is. You know your application, you know it's primary use cases, and probably have some data to play with. You're certainly in a much better situation to at

Re: [PERFORM] performance for high-volume log insertion

2009-04-20 Thread Stephen Frost
David, * da...@lang.hm (da...@lang.hm) wrote: > I thought that part of the 'efficiancy' and 'performance' to be gained > from binary modes were avoiding the need to parse commands, if it's only > the savings in converting column contents from text to specific types, > it's much less importan

Re: [PERFORM] performance for high-volume log insertion

2009-04-20 Thread david
On Mon, 20 Apr 2009, Stephen Frost wrote: Greg, * Greg Smith (gsm...@gregsmith.com) wrote: The win from switching from INSERT to COPY can be pretty big, further optimizing to BINARY you'd really need to profile to justify. Have you done any testing to compare COPY vs. INSERT using prepared s

Re: [PERFORM] performance for high-volume log insertion

2009-04-20 Thread david
On Mon, 20 Apr 2009, Greg Smith wrote: On Mon, 20 Apr 2009, da...@lang.hm wrote: any idea what sort of difference binary mode would result in? The win from switching from INSERT to COPY can be pretty big, further optimizing to BINARY you'd really need to profile to justify. I haven't foun

Re: [PERFORM] performance for high-volume log insertion

2009-04-20 Thread david
On Tue, 21 Apr 2009, Stephen Frost wrote: David, * da...@lang.hm (da...@lang.hm) wrote: the database structure is not being defined by (or specificly for) rsyslog. so at compile time we have _no_ idea how many variables of what type there are going to be. my example of ($timestamp,$msg) was in

Re: [PERFORM] performance for high-volume log insertion

2009-04-20 Thread Stephen Frost
David, * da...@lang.hm (da...@lang.hm) wrote: > any idea what sort of difference binary mode would result in? It depends a great deal on your application.. > currently rsyslog makes use of it's extensive formatting capabilities to > format a string along the lines of > $DBformat="insert into t

Re: [PERFORM] performance for high-volume log insertion

2009-04-20 Thread Stephen Frost
David, * da...@lang.hm (da...@lang.hm) wrote: > the database structure is not being defined by (or specificly for) > rsyslog. so at compile time we have _no_ idea how many variables of what > type there are going to be. my example of ($timestamp,$msg) was intended > to just be a sample (avoi

Re: [PERFORM] performance for high-volume log insertion

2009-04-20 Thread david
m would let you do DBinit="PREPARE rsyslog_insert(date, text) AS\nINSERT INTO foo VALUES(\$1, \$2);" DBstart = "begini;B\n" DBjoin = "" DBend = "end;" DBitem = "EXECUTE rsyslog_insert('$timestamp','$msg');\n" which would become

Re: [PERFORM] performance for high-volume log insertion

2009-04-20 Thread Stephen Frost
Greg, * Greg Smith (gsm...@gregsmith.com) wrote: > The win from switching from INSERT to COPY can be pretty big, further > optimizing to BINARY you'd really need to profile to justify. Have you done any testing to compare COPY vs. INSERT using prepared statements? I'd be curious to know how

Re: [PERFORM] performance for high-volume log insertion

2009-04-20 Thread Greg Smith
On Mon, 20 Apr 2009, da...@lang.hm wrote: any idea what sort of difference binary mode would result in? The win from switching from INSERT to COPY can be pretty big, further optimizing to BINARY you'd really need to profile to justify. I haven't found any significant difference in binary mo

Re: [PERFORM] performance for high-volume log insertion

2009-04-20 Thread david
On Mon, 20 Apr 2009, Stephen Frost wrote: David, * da...@lang.hm (da...@lang.hm) wrote: I am working with the rsyslog developers to improve it's performance in inserting log messages to databases. Great! currently they have a postgres interface that works like all the other ones, where rsy

Re: [PERFORM] performance for high-volume log insertion

2009-04-20 Thread Stephen Frost
David, * da...@lang.hm (da...@lang.hm) wrote: > I am working with the rsyslog developers to improve it's performance in > inserting log messages to databases. Great! > currently they have a postgres interface that works like all the other > ones, where rsyslog formats an insert statement, pa

[PERFORM] performance for high-volume log insertion

2009-04-20 Thread david
I am working with the rsyslog developers to improve it's performance in inserting log messages to databases. currently they have a postgres interface that works like all the other ones, where rsyslog formats an insert statement, passes that the the interface module, that sends it to postgres (

Re: [PERFORM] SQL With Dates

2009-04-20 Thread Mark Lewis
It sounds like what you're doing is comparing the planner's cost estimate from running EXPLAIN on a few different queries. The planner's cost estimate was never intended to do what you're trying to do; it's not an absolute scale of cost, it's just a tool that the planner uses to get relative compa

Re: [PERFORM] SQL With Dates

2009-04-20 Thread Scott Marlowe
On Mon, Apr 20, 2009 at 7:55 AM, Rafael Domiciano wrote: > Hello People, > > I have initiated a work to review the sqls of our internal software. > Lot of them he problem are about sql logic, or join with table unecessary, > and so on. > But software has lot of sql with date, doing thinks like: >

Re: [PERFORM] SQL With Dates

2009-04-20 Thread Rafael Domiciano
Hello Grzegorz, Thnks for response, but lot of the selects is using BETWEEN and the cost is the same. 2009/4/20 Grzegorz Jaśkiewicz > BETWEEN X AND Y > > On Mon, Apr 20, 2009 at 2:55 PM, Rafael Domiciano > wrote: > > Hello People, > > > > I have initiated a work to review the sqls of our inter

[PERFORM] SQL With Dates

2009-04-20 Thread Rafael Domiciano
Hello People, I have initiated a work to review the sqls of our internal software. Lot of them he problem are about sql logic, or join with table unecessary, and so on. But software has lot of sql with date, doing thinks like: [..] date >= '2009-04-01' AND date <= '2009-04-15' [..] Redoing th

Re: [PERFORM] GiST index performance

2009-04-20 Thread Tom Lane
Matthew Wakeling writes: > I have found a bug in the contrib package seg, which has been copied into > the bioseg data type as well. It causes the index to be created with > horribly bad unselective trees, so that when a search is performed many of > the branches of the tree need to be followed

Re: [PERFORM] SQL With Dates

2009-04-20 Thread Grzegorz Jaśkiewicz
BETWEEN X AND Y On Mon, Apr 20, 2009 at 2:55 PM, Rafael Domiciano wrote: > Hello People, > > I have initiated a work to review the sqls of our internal software. > Lot of them he problem are about sql logic, or join with table unecessary, > and so on. > But software has lot of sql with date, doin

Re: [PERFORM] GiST index performance

2009-04-20 Thread Matthew Wakeling
On Fri, 17 Apr 2009, Matthew Wakeling wrote: I have done a bit of investigation, and I think I might have found the smoking gun I was looking for. I have found a bug in the contrib package seg, which has been copied into the bioseg data type as well. It causes the index to be created with hor