On 14 August 2009 at 03:18 Jeff Janes jeff.ja...@gmail.com wrote:
This my rough understanding. Oracle never
takes a snapshot, it computes one the fly, if and when it is needed. It
maintains a
structure of recently committed transactions, with the XID for when they
committed. If a
On Fri, Aug 14, 2009 at 4:21 PM, Tom Lanet...@sss.pgh.pa.us wrote:
Jeff Janes jeff.ja...@gmail.com writes:
I apologize if it is bad form to respond to a message that is two
months old, but I did not see this question answered elsewhere and
thought it would be helpful to have it answered. This
On Thu, 4 Jun 2009 06:57:57 -0400, Robert Haas robertmh...@gmail.com wrote
in http://archives.postgresql.org/pgsql-performance/2009-06/msg00065.php :
I think I see the distinction you're drawing here. IIUC, you're
arguing that other database products use connection pooling to handle
rapid
Jeff Janes jeff.ja...@gmail.com writes:
I apologize if it is bad form to respond to a message that is two
months old, but I did not see this question answered elsewhere and
thought it would be helpful to have it answered. This my rough
understanding. Oracle never takes a snapshot, it
Mark Mielke m...@mark.mielke.cc wrote:
Kevin Grittner wrote:
James Mansion ja...@mansionfamily.plus.com wrote:
Kevin Grittner wrote:
Sure, but the architecture of those products is based around all
the work being done by engines which try to establish affinity
to different CPUs, and loop
Scott Carey sc...@richrelevance.com wrote:
If you wake up 10,000 threads, and they all can get significant work
done before yielding no matter what order they run, the system will
scale extremely well.
But with roughly twice the average response time you would get
throttling active requests
On Fri, Jun 5, 2009 at 12:33 AM, da...@lang.hm wrote:
On Fri, 5 Jun 2009, Greg Smith wrote:
On Thu, 4 Jun 2009, Robert Haas wrote:
That's because this thread has altogether too much theory and
altogether too little gprof.
But running benchmarks and profiling is actual work; that's so much
Greg Smith wrote:
No amount of theoretical discussion advances that any until
you're at least staring at a very specific locking problem you've
already characterized extensively via profiling. And even then,
profiling trumps theory every time.
In theory, there is no difference between
On Fri, 5 Jun 2009, Mark Mielke wrote:
I disagree that profiling trumps theory every time.
That's an interesting theory. Unfortunately, profiling shows it doesn't
work that way.
Let's see if I can summarize the state of things a bit better here:
1) PostgreSQL stops working as efficiently
On Thu, 4 Jun 2009, Mark Mielke wrote:
At it's very simplest, this is the difference between wake one thread
(which is then responsible for waking the next thread) vs wake all
threadsAny system which actually wakes all threads will probably
exhibit scaling limitations.
The prototype
On Fri, Jun 5, 2009 at 1:02 PM, Greg Smithgsm...@gregsmith.com wrote:
On Fri, 5 Jun 2009, Mark Mielke wrote:
I disagree that profiling trumps theory every time.
That's an interesting theory. Unfortunately, profiling shows it doesn't
work that way.
I had a laugh when I read this, but I can
It's not that trivial with Oracle either. I guess you had to use shared
servers to get to that amount of sessions. They're most of the time not
activated by default (dispatchers is at 0).
Granted, they are part of the 'main' product, so you just have to set up
dispatchers, shared servers,
On Wed, Jun 3, 2009 at 5:09 PM, Scott Carey sc...@richrelevance.com wrote:
On 6/3/09 11:39 AM, Robert Haas robertmh...@gmail.com wrote:
On Wed, Jun 3, 2009 at 2:12 PM, Scott Carey sc...@richrelevance.com wrote:
Postgres could fix its connection scalability issues -- that is entirely
On 6/4/09 3:57 AM, Robert Haas robertmh...@gmail.com wrote:
On Wed, Jun 3, 2009 at 5:09 PM, Scott Carey sc...@richrelevance.com wrote:
On 6/3/09 11:39 AM, Robert Haas robertmh...@gmail.com wrote:
On Wed, Jun 3, 2009 at 2:12 PM, Scott Carey sc...@richrelevance.com wrote:
Postgres could fix
On Thu, Jun 4, 2009 at 2:04 PM, Scott Carey sc...@richrelevance.com wrote:
To clarify if needed:
I'm not saying the two issues are unrelated. I'm saying that the
relationship between connection pooling and a database is multi-dimensional,
and the scalability improvement does not have a hard
Kevin Grittner wrote:
Sure, but the architecture of those products is based around all the
work being done by engines which try to establish affinity to
different CPUs, and loop through the various tasks to be done. You
don't get a context switch storm because you normally have the number
of
James Mansion ja...@mansionfamily.plus.com wrote:
Kevin Grittner wrote:
Sure, but the architecture of those products is based around all
the work being done by engines which try to establish affinity to
different CPUs, and loop through the various tasks to be done. You
don't get a context
James Mansion ja...@mansionfamily.plus.com wrote:
they spend a lot of time spinning around queue access to see if
anything has become available to do -- which causes them not to
play nice with other processes on the same box.
UNIX systems have routinely managed large numbers of runnable
Kevin Grittner wrote:
James Mansion ja...@mansionfamily.plus.com wrote:
Kevin Grittner wrote:
Sure, but the architecture of those products is based around all
the work being done by engines which try to establish affinity to
different CPUs, and loop through the various tasks to be
On Thu, 4 Jun 2009, Robert Haas wrote:
On Wed, Jun 3, 2009 at 5:09 PM, Scott Carey sc...@richrelevance.com wrote:
On 6/3/09 11:39 AM, Robert Haas robertmh...@gmail.com wrote:
On Wed, Jun 3, 2009 at 2:12 PM, Scott Carey sc...@richrelevance.com wrote:
Postgres could fix its connection
On 6/4/09 3:08 PM, Kevin Grittner kevin.gritt...@wicourts.gov wrote:
James Mansion ja...@mansionfamily.plus.com wrote:
I'm sorry, but (in particular) UNIX systems have routinely
managed large numbers of runnable processes where the run queue
lengths are long without such an issue.
Well,
On Thu, 4 Jun 2009, Mark Mielke wrote:
Kevin Grittner wrote:
James Mansion ja...@mansionfamily.plus.com wrote:
I know that if you do use a large number of threads, you have to be
pretty adaptive. In our Java app that pulls data from 72 sources and
replicates it to eight, plus feeding it to
On Thu, Jun 4, 2009 at 8:51 PM, da...@lang.hm wrote:
if this is the case, how hard would it be to have threads add and remove
themselves from some list as they get busy/become idle?
I've been puzzled as I've been watching this conversation on what internal
locking/lookup is happening that is
da...@lang.hm wrote:
On Thu, 4 Jun 2009, Mark Mielke wrote:
You should really only have as 1X or 2X many threads as there are
CPUs waiting on one monitor. Beyond that is waste. The idle threads
can be pooled away, and only activated (with individual monitors
which can be far more easily and
On Thu, 4 Jun 2009, Robert Haas wrote:
That's because this thread has altogether too much theory and
altogether too little gprof.
But running benchmarks and profiling is actual work; that's so much less
fun than just speculating about what's going on!
This thread reminds me of Jignesh's
On Thu, 4 Jun 2009, Mark Mielke wrote:
da...@lang.hm wrote:
On Thu, 4 Jun 2009, Mark Mielke wrote:
An alternative approach might be: 1) Idle processes not currently running
a transaction do not need to be consulted for their snapshot (and other
related expenses) - if they are idle for a
On Fri, 5 Jun 2009, Greg Smith wrote:
On Thu, 4 Jun 2009, Robert Haas wrote:
That's because this thread has altogether too much theory and
altogether too little gprof.
But running benchmarks and profiling is actual work; that's so much less fun
than just speculating about what's going on!
Greg Smith wrote:
This thread reminds me of Jignesh's Proposal of tunable fix for
scalability of 8.4 thread from March, except with only a fraction of
the real-world detail. There are multiple high-profile locks causing
scalability concerns at quadruple digit high user counts in the
Greg Smith wrote:
3500 active connections across them. That doesn't work, and what
happens
is exactly the sort of context switch storm you're showing data for.
Think about it for a minute: how many of those can really be doing
work at any time? 32, that's how many. Now, you need some
James Mansion ja...@mansionfamily.plus.com wrote:
I'm sure most of us evaluating Postgres from a background in Sybase
or SQLServer would regard 5000 connections as no big deal.
Sure, but the architecture of those products is based around all the
work being done by engines which try to
Few weeks ago tested a customer application on 16 cores with Oracle:
- 20,000 sessions in total
- 70,000 queries/sec
without any problem on a mid-range Sun box + Solaris 10..
Rgds,
-Dimitri
On 6/3/09, Kevin Grittner kevin.gritt...@wicourts.gov wrote:
James Mansion
Dimitri dimitrik...@gmail.com wrote:
Few weeks ago tested a customer application on 16 cores with Oracle:
- 20,000 sessions in total
- 70,000 queries/sec
without any problem on a mid-range Sun box + Solaris 10..
I'm not sure what point you are trying to make. Could you elaborate?
On 6/3/09 10:45 AM, Kevin Grittner kevin.gritt...@wicourts.gov wrote:
Dimitri dimitrik...@gmail.com wrote:
Few weeks ago tested a customer application on 16 cores with Oracle:
- 20,000 sessions in total
- 70,000 queries/sec
without any problem on a mid-range Sun box + Solaris 10..
Just to say you don't need a mega server to keep thousands connections
with Oracle, it's just trivial, nor CPU affinity and other stuff you
may or may not need with Sybase :-)
Regarding PostgreSQL, I think it'll only benefit to have an integrated
connection pooler as it'll make happy all
On Wed, Jun 3, 2009 at 2:12 PM, Scott Carey sc...@richrelevance.com wrote:
Postgres could fix its connection scalability issues -- that is entirely
independent of connection pooling.
Really? I'm surprised. I thought the two were very closely related.
Could you expand on your thinking here?
On 6/3/09 11:39 AM, Robert Haas robertmh...@gmail.com wrote:
On Wed, Jun 3, 2009 at 2:12 PM, Scott Carey sc...@richrelevance.com wrote:
Postgres could fix its connection scalability issues -- that is entirely
independent of connection pooling.
Really? I'm surprised. I thought the two
On 5/31/09 9:37 AM, Fabrix fabrix...@gmail.com wrote:
2009/5/29 Scott Carey sc...@richrelevance.com
On 5/28/09 6:54 PM, Greg Smith gsm...@gregsmith.com wrote:
2) You have very new hardware and a very old kernel. Once you've done the
above, if you're still not happy with performance,
Grzegorz Jaśkiewicz wrote:
I thought that's where the difference is between postgresql and oracle
mostly, ability to handle more transactions and better scalability .
Which were you suggesting had this better scalability?
I recall someone summarizing to a CFO where I used to work:
Oracle
On Sat, 30 May 2009, Scott Marlowe wrote:
8.04 was a frakking train wreck in many ways. It wasn't until 8.04.2
came out that it was even close to useable as a server OS, and even
then, not for databases yet. It's still got broken bits and pieces
marked fixed in 8.10... Uh, hello, it's your
On Sat, May 30, 2009 at 9:41 PM, Greg Smith gsm...@gregsmith.com wrote:
On Fri, 29 May 2009, Scott Carey wrote:
There are operations/IT people won't touch Ubuntu etc with a ten foot pole
yet for production.
The only thing I was suggesting is that because 2.6.28 is the latest Ubuntu
kernel,
2009/5/29 Scott Carey sc...@richrelevance.com
On 5/28/09 6:54 PM, Greg Smith gsm...@gregsmith.com wrote:
2) You have very new hardware and a very old kernel. Once you've done
the
above, if you're still not happy with performance, at that point you
should consider using a newer one.
On Fri, 29 May 2009, Scott Carey wrote:
There are operations/IT people won't touch Ubuntu etc with a ten foot pole
yet for production.
The only thing I was suggesting is that because 2.6.28 is the latest
Ubuntu kernel, that means it's gotten a lot more exposure and testing
than, say, other
On Fri, May 29, 2009 at 2:54 AM, Greg Smith gsm...@gregsmith.com wrote:
The PostgreSQL connection handler is known to be bad at handling high
connection loads compared to the popular pooling projects, so you really
shouldn't throw this problem at it. While kernel problems stack on top of
2009/5/29 Grzegorz Jaśkiewicz gryz...@gmail.com:
On Fri, May 29, 2009 at 2:54 AM, Greg Smith gsm...@gregsmith.com wrote:
The PostgreSQL connection handler is known to be bad at handling high
connection loads compared to the popular pooling projects, so you really
shouldn't throw this problem
2009/5/29 Grzegorz Jaśkiewicz gryz...@gmail.com:
On Fri, May 29, 2009 at 2:54 AM, Greg Smith gsm...@gregsmith.com wrote:
The PostgreSQL connection handler is known to be bad at handling high
connection loads compared to the popular pooling projects, so you really
shouldn't throw this problem
2009/5/29 Scott Marlowe scott.marl...@gmail.com:
if it is implemented somewhere else better, shouldn't that make it
obvious that postgresql should solve it internally ? It is really
annoying to hear all the time that you should add additional path of
execution to already complex stack, and
2009/5/29 Grzegorz Jaśkiewicz gryz...@gmail.com:
2009/5/29 Scott Marlowe scott.marl...@gmail.com:
if it is implemented somewhere else better, shouldn't that make it
obvious that postgresql should solve it internally ? It is really
annoying to hear all the time that you should add additional
2009/5/29 Scott Marlowe scott.marl...@gmail.com:
Both Oracle and PostgreSQL have fairly heavy backend processes, and
running hundreds of them on either database is a mistake. Sure,
Oracle can handle more transactions and scales a bit better, but no
one wants to have to buy a 128 way E15K
2009/5/29 Grzegorz Jaśkiewicz gryz...@gmail.com:
2009/5/29 Scott Marlowe scott.marl...@gmail.com:
Both Oracle and PostgreSQL have fairly heavy backend processes, and
running hundreds of them on either database is a mistake. Sure,
Oracle can handle more transactions and scales a bit
damn I agree with you Scott. I wish I had enough cash here to employ
Tom and other pg magicians to improve performance for all of us ;)
Thing is tho, postgresql is mostly used by companies, that either
don't have that sort of cash, but still like to get the performance,
or companies that have
2009/5/29 Grzegorz Jaśkiewicz gryz...@gmail.com:
damn I agree with you Scott. I wish I had enough cash here to employ
Tom and other pg magicians to improve performance for all of us ;)
Thing is tho, postgresql is mostly used by companies, that either
don't have that sort of cash, but still
On Fri, 29 May 2009, Grzegorz Ja?kiewicz wrote:
if it is implemented somewhere else better, shouldn't that make it
obvious that postgresql should solve it internally ?
Opening a database connection has some overhead to it that can't go away
without losing *something* in the process that you
2009/5/28 Greg Smith gsm...@gregsmith.com
On Thu, 28 May 2009, Flavio Henrique Araque Gurgel wrote:
It is 2.6.24 We had to apply the kswapd patch also. It's important
specially if you see your system % going as high as 99% in top and loosing
the machine's control. I have read something
On Fri, 29 May 2009, Fabrix wrote:
In this application is not closing the connection, the development team
is makeing the change for close the connection after getting the job
done. So most connections are in idle state. How much would this help?
Does this could be the real problem?
Ah,
2009/5/29 Greg Smith gsm...@gregsmith.com
On Fri, 29 May 2009, Grzegorz Ja?kiewicz wrote:
if it is implemented somewhere else better, shouldn't that make it
obvious that postgresql should solve it internally ?
Opening a database connection has some overhead to it that can't go away
On Fri, May 29, 2009 at 12:20 PM, Scott Mead
scott.li...@enterprisedb.com wrote:
This sounds like a dirty plug (sorry sorry sorry, it's for informative
purposes only)...
(Commercial applications mentioned deleted for brevity.)
Just sounded like useful information to me. I'm not anti
2009/5/29 Scott Mead scott.li...@enterprisedb.com
2009/5/29 Greg Smith gsm...@gregsmith.com
On Fri, 29 May 2009, Grzegorz Ja?kiewicz wrote:
if it is implemented somewhere else better, shouldn't that make it
obvious that postgresql should solve it internally ?
Opening a database
On 5/28/09 6:54 PM, Greg Smith gsm...@gregsmith.com wrote:
2) You have very new hardware and a very old kernel. Once you've done the
above, if you're still not happy with performance, at that point you
should consider using a newer one. It's fairly simple to build a Linux
kernel using the
HI.
Someone had some experience of bad performance with postgres in some server
with many processors?
I have a server with 4 CPUS dual core and gives me a very good performance
but I have experienced problems with another server that has 8 CPUS quad
core (32 cores). The second one only gives me
On Thu, May 28, 2009 at 11:50 AM, Fabrix fabrix...@gmail.com wrote:
Monitoring (nmon, htop, vmstat) see that everything is fine (memory, HD,
eth, etc) except that processors regularly climb to 100%.
What kind of load are you putting the server under when this happens?
I can see that the
On Thu, May 28, 2009 at 12:50 PM, Fabrix fabrix...@gmail.com wrote:
HI.
Someone had some experience of bad performance with postgres in some server
with many processors?
Seems to depend on the processors and chipset a fair bit.
I have a server with 4 CPUS dual core and gives me a very
Thanks David...
2009/5/28 David Rees dree...@gmail.com
On Thu, May 28, 2009 at 11:50 AM, Fabrix fabrix...@gmail.com wrote:
Monitoring (nmon, htop, vmstat) see that everything is fine (memory, HD,
eth, etc) except that processors regularly climb to 100%.
What kind of load are you putting
On Thu, May 28, 2009 at 4:53 PM, Fabrix fabrix...@gmail.com wrote:
Wow, that's some serious context-switching right there - 300k context
switches a second mean that the processors are spending a lot of their
time fighting for CPU time instead of doing any real work.
There is a bug in
On Thu, May 28, 2009 at 2:53 PM, Fabrix fabrix...@gmail.com wrote:
yes, i have max_connections = 5000
can lower, but at least i need 3500 connections
Whoa, that's a lot. Can you look into connection pooling of some sort?
--
Sent via pgsql-performance mailing list
Thanks Scott
2009/5/28 Scott Marlowe scott.marl...@gmail.com
On Thu, May 28, 2009 at 12:50 PM, Fabrix fabrix...@gmail.com wrote:
HI.
Someone had some experience of bad performance with postgres in some
server
with many processors?
Seems to depend on the processors and chipset a
2009/5/28 Scott Mead scott.li...@enterprisedb.com
On Thu, May 28, 2009 at 4:53 PM, Fabrix fabrix...@gmail.com wrote:
Wow, that's some serious context-switching right there - 300k context
switches a second mean that the processors are spending a lot of their
time fighting for CPU time
- Scott Marlowe scott.marl...@gmail.com escreveu:
On Thu, May 28, 2009 at 12:50 PM, Fabrix fabrix...@gmail.com wrote:
HI.
Someone had some experience of bad performance with postgres in some server
with many processors?
I had.
but I have experienced problems with another
2009/5/28 Flavio Henrique Araque Gurgel fla...@4linux.com.br
- Scott Marlowe scott.marl...@gmail.com escreveu:
On Thu, May 28, 2009 at 12:50 PM, Fabrix fabrix...@gmail.com wrote:
HI.
Someone had some experience of bad performance with postgres in some
server
with many
I would ask for your kernel version. uname -a please?
sure, and thanks for you answer Flavio...
uname -a
Linux SERVIDOR-A 2.6.18-92.el5 #1 SMP Tue Apr 29 13:16:15 EDT 2008 x86_64
x86_64 x86_64 GNU/Linux
cat /etc/redhat-release
Red Hat Enterprise Linux Server release 5.2
On Thu, May 28, 2009 at 7:04 PM, Fabrix fabrix...@gmail.com wrote:
I would ask for your kernel version. uname -a please?
sure, and thanks for you answer Flavio...
uname -a
Linux SERVIDOR-A 2.6.18-92.el5 #1 SMP Tue Apr 29 13:16:15 EDT 2008 x86_64
x86_64 x86_64 GNU/Linux
cat
70 matches
Mail list logo
