Re: [PATCHES] A couple of patches for PostgreSQL 64bit support

2005-08-20 Thread Tom Lane
Koichi Suzuki [EMAIL PROTECTED] writes:
 Here're a couple of patches for PostgreSQL 64bit support.  There're just
 two extension to 64bit, size of shared memory and transaction ID.

I've applied the part of this that seemed reasonably noncontroversial,
namely the fixes to do shared memory size calculation in size_t
arithmetic instead of int arithmetic.  (While at it, I extended the Size
convention to all the shared memory sizing routines, not just buffers,
and added code to detect overflows in the calculations.  That way we
don't need a 64 bit configure switch.)  While I still remain
unconvinced that there's any real-world need for more than 2Gb of
shared_buffers, this change certainly makes the code more robust against
configuration errors, and it has essentially zero cost (except maybe a
few more milliseconds during postmaster start).

On the other hand, I think the 64-bit XID idea needs considerably more
work before being proposed again.  That would incur serious costs due
to the expansion of tuple headers, and there's no evidence that the
distributed cost would be bought back by avoiding one vacuum pass every
billion transactions.  (Your description of the patch claimed that
vacuuming requires shutting down the database, which is simply wrong.)
Also, as previously noted, you can't just whack the size of XID around
without considering side-effects on CID, OID, Datum, etc.

regards, tom lane

---(end of broadcast)---
TIP 6: explain analyze is your friend


Re: [PATCHES] A couple of patches for PostgreSQL 64bit support

2005-07-19 Thread Koichi Suzuki

Mark,

I've not seen CVS in detail.   I begain this work against 8.0.1 and 
continued thru 8.0.2 to 8.0.3.  It was not a great work.   The patch is 
rather straightforward and I appreciate if you try to port against CVS.


Mark Wong wrote:

Hi,

I grabbed the patches to try, but I was wondering if it would be more
interesting to try them against CVS rather than 8.0.3 (and if it would
be easy to port :)?

Mark




--
---
Koichi Suzuki
Open Source Engineeering Departmeent,
NTT DATA Intellilink Corporation
Phone: +81-3-5566-9628  WWW: http://www.intellilink.co.jp
--

---(end of broadcast)---
TIP 5: don't forget to increase your free space map settings


Re: [PATCHES] A couple of patches for PostgreSQL 64bit support

2005-07-18 Thread Mark Wong
Hi,

I grabbed the patches to try, but I was wondering if it would be more
interesting to try them against CVS rather than 8.0.3 (and if it would
be easy to port :)?

Mark

---(end of broadcast)---
TIP 4: Have you searched our list archives?

   http://archives.postgresql.org


Re: [PATCHES] A couple of patches for PostgreSQL 64bit support

2005-07-14 Thread Koichi Suzuki

 I asked originally for some experimental evidence showing any value
 in having more than 2Gb of shared buffers.  In the absence of any
 convincing demonstration, I'm not very inclined to worry about whether
 we can handle wider-than-int shared memory size.

Hi,

Attached is a result of pgbench with 64bit patch PostgreSQL (base is
8.0.1).  Benchmark machine is dual opteron (1.4GHz, 1MB cache each) with
8GB of memory and 120GB of IDE hard disk.

Koichi Suzuki wrote:

 I have some experimeltal data about this extension.   I will gather it
 and post hopefully this weekend.

---
Koichi Suzuki
Open Source Engineeering Departmeent,
NTT DATA Intellilink Corporation
Phone: +81-3-5566-9628  WWW: http://www.intellilink.co.jp
--


64-bit-pgbench20050712.pdf
Description: Adobe PDF document

---(end of broadcast)---
TIP 3: Have you checked our extensive FAQ?

   http://www.postgresql.org/docs/faq


Re: [PATCHES] A couple of patches for PostgreSQL 64bit support

2005-07-12 Thread Koichi Suzuki
Hi,

Attached is a result of pgbench with 64bit patch PostgreSQL (base is
8.0.1).  Benchmark machine is dual opteron (1.4GHz, 1MB cache each) with
8GB of memory and 120GB of IDE hard disk.

Koichi Suzuki wrote:
 I have some experimeltal data about this extension.   I will gather it
 and post hopefully this weekend.


-- 
---
Koichi Suzuki
Open Source Engineeering Departmeent,
NTT DATA Intellilink Corporation
Phone: +81-3-5566-9628  WWW: http://www.intellilink.co.jp
--


64-bit-pgbench20050712.pdf
Description: Adobe PDF document

---(end of broadcast)---
TIP 1: if posting/reading through Usenet, please send an appropriate
   subscribe-nomail command to [EMAIL PROTECTED] so that your
   message can get through to the mailing list cleanly


Re: [PATCHES] A couple of patches for PostgreSQL 64bit support

2005-07-07 Thread Tom Lane
Koichi Suzuki [EMAIL PROTECTED] writes:
 Here're a couple of patches for PostgreSQL 64bit support.  There're just
 two extension to 64bit, size of shared memory and transaction ID.

I asked originally for some experimental evidence showing any value
in having more than 2Gb of shared buffers.  In the absence of any
convincing demonstration, I'm not very inclined to worry about whether
we can handle wider-than-int shared memory size.

As for the XID change, I don't think this patch accurately reflects the
size of the impact.  There are a lot of things that in practice need to
be the same size as XID (CID, most obviously, but I suspect also OID).
And again, some demonstration of the performance impact would be
appropriate.  Here, not only do you have to prove that widening XID
isn't a big performance hit in itself, but you also have to convince
us that it's a win compared to the existing approach of vacuuming at
least every billion transactions.

regards, tom lane

---(end of broadcast)---
TIP 6: Have you searched our list archives?

   http://archives.postgresql.org


Re: [PATCHES] A couple of patches for PostgreSQL 64bit support

2005-07-07 Thread Hans-Jürgen Schönig

Tom Lane wrote:

Koichi Suzuki [EMAIL PROTECTED] writes:


Here're a couple of patches for PostgreSQL 64bit support.  There're just
two extension to 64bit, size of shared memory and transaction ID.



I asked originally for some experimental evidence showing any value
in having more than 2Gb of shared buffers.  In the absence of any
convincing demonstration, I'm not very inclined to worry about whether
we can handle wider-than-int shared memory size. 


There is some practical evidence. Recently the number of large boxes in 
the field is almost growing exponentially. Today I have heard somebody 
say this box has 'just 4 gig of ram' .
On large installations we have already seen problems with too small 
caches (= 2gb).
Surprisingly this has turned out to be a quite important issue in the 
field. Tests have shown that the cache provided by the OS is a lot worse 
for the database.


64-bit XIDs seem to be an overkill - the only practical impact I can see 
is an even larger tuple header (this can be an issue on large boxes too 
- at least compared to Oracle).


Best regards,

Hans

---(end of broadcast)---
TIP 3: if posting/reading through Usenet, please send an appropriate
  subscribe-nomail command to [EMAIL PROTECTED] so that your
  message can get through to the mailing list cleanly


Re: [PATCHES] A couple of patches for PostgreSQL 64bit support

2005-07-07 Thread Tom Lane
=?ISO-8859-1?Q?Hans-J=FCrgen_Sch=F6nig?= [EMAIL PROTECTED] writes:
 There is some practical evidence. Recently the number of large boxes in 
 the field is almost growing exponentially. Today I have heard somebody 
 say this box has 'just 4 gig of ram' .
 On large installations we have already seen problems with too small 
 caches (= 2gb).
 Surprisingly this has turned out to be a quite important issue in the 
 field. Tests have shown that the cache provided by the OS is a lot worse 
 for the database.

*What* tests?  This is all handwaving :-(

What I would find credible is a set of, say, OSDL test runs, showing a
continuing increase of performance with shared_buffers right up to the
2Gb limit.  Everything done to date says that you hit the point of
diminishing returns well before that.

regards, tom lane

---(end of broadcast)---
TIP 7: don't forget to increase your free space map settings


Re: [PATCHES] A couple of patches for PostgreSQL 64bit support

2005-07-07 Thread Hans-Jürgen Schönig

Tom Lane wrote:

=?ISO-8859-1?Q?Hans-J=FCrgen_Sch=F6nig?= [EMAIL PROTECTED] writes:

There is some practical evidence. Recently the number of large boxes in 
the field is almost growing exponentially. Today I have heard somebody 
say this box has 'just 4 gig of ram' .
On large installations we have already seen problems with too small 
caches (= 2gb).
Surprisingly this has turned out to be a quite important issue in the 
field. Tests have shown that the cache provided by the OS is a lot worse 
for the database.



*What* tests?  This is all handwaving :-(

What I would find credible is a set of, say, OSDL test runs, showing a
continuing increase of performance with shared_buffers right up to the
2Gb limit.  Everything done to date says that you hit the point of
diminishing returns well before that.

regards, tom lane



well, you can easily try it on a big machine with gigs of ram and 
nothing but the database running. using a very low number of shared 
buffers will lead to worse performance than many shared buffers - even 
if the operating system caches some disk i/O (which is done by linux if 
nobody else want to have some ram). i have no public hard figures i 
could post here but customers have told me that 2Q cache vs. kernel 
cache is around 5-10 times faster (it varies of course).


the 2gb thing is especially important for data crunchers - not 
necessarily for 'normal' OLTP databases. just assume somebody getting 5 
gig of data and doing some repeated computation with this data. you 
definitely don't want to go to disk in this case. people will assume 
that postgresql can work with large caches (it is a good database - why 
do i get errors on startup - this is the usual story). people rather 
tend to rely on PostgreSQL than on some operating system thing ;).


i might have some time to provide some real hard facts to prove this but 
i am a bit busy at the moment.


best regards,

hans



---(end of broadcast)---
TIP 3: if posting/reading through Usenet, please send an appropriate
  subscribe-nomail command to [EMAIL PROTECTED] so that your
  message can get through to the mailing list cleanly


Re: [PATCHES] A couple of patches for PostgreSQL 64bit support

2005-07-07 Thread Koichi Suzuki
I have some experimeltal data about this extension.   I will gather it
and post hopefully this weekend.

Tom Lane wrote:
 Koichi Suzuki [EMAIL PROTECTED] writes:
 
Here're a couple of patches for PostgreSQL 64bit support.  There're just
two extension to 64bit, size of shared memory and transaction ID.
 
 
 I asked originally for some experimental evidence showing any value
 in having more than 2Gb of shared buffers.  In the absence of any
 convincing demonstration, I'm not very inclined to worry about whether
 we can handle wider-than-int shared memory size.
 
 As for the XID change, I don't think this patch accurately reflects the
 size of the impact.  There are a lot of things that in practice need to
 be the same size as XID (CID, most obviously, but I suspect also OID).
 And again, some demonstration of the performance impact would be
 appropriate.  Here, not only do you have to prove that widening XID
 isn't a big performance hit in itself, but you also have to convince
 us that it's a win compared to the existing approach of vacuuming at
 least every billion transactions.
 
   regards, tom lane
 


-- 
---
Koichi Suzuki
Open Source Engineeering Departmeent,
NTT DATA Intellilink Corporation
Phone: +81-3-5566-9628  WWW: http://www.intellilink.co.jp
--

---(end of broadcast)---
TIP 5: Have you checked our extensive FAQ?

   http://www.postgresql.org/docs/faq


Re: [PATCHES] A couple of patches for PostgreSQL 64bit support

2005-07-07 Thread ITAGAKI Takahiro
Hans-J|rgen Schvnig [EMAIL PROTECTED] wrote:

 64-bit XIDs seem to be an overkill - the only practical impact I can see 
 is an even larger tuple header (this can be an issue on large boxes too 
 - at least compared to Oracle).

I agreed, too. The changes of XIDs cannot be ignored because the overhead
will be 32bytes per tuple.

Avoiding overheads, can XIDs/CIDs be different bit length? For example,
can XIDs/CIDs be changed to 48/16-bit or 40/24-bit?

---
ITAGAKI Takahiro
NTT Cyber Space Laboratories


---(end of broadcast)---
TIP 8: explain analyze is your friend