Re: [PERFORM] Looking for ideas on how to speed up warehouse loading

2004-04-22 Thread Joe Conway
Sean Shanny wrote: explain analyze SELECT t1.id, t2.url FROM referral_temp t2 LEFT OUTER JOIN d_referral t1 ON t2.url = t1.referral_raw_url ORDER BY t1.id; What I would like to know is if there are better ways to do the join? I need to get all the rows back from the referral_temp table as they

Re: [PERFORM] Wierd context-switching issue on Xeon

2004-04-22 Thread Anjan Dave
Tested the sql on Quad 2.0GHz XEON/8GB RAM: During the first run, the CS shooted up more than 100k, and was randomly high/low Second process made it consistently high 100k+ Third brought it down to anaverage 80-90k Fourth brought it down to an average of 50-60k/s By cancelling the queries one-b

Re: [PERFORM] Looking for ideas on how to speed up warehouse loading

2004-04-22 Thread Tom Lane
Sean Shanny <[EMAIL PROTECTED]> writes: > explain analyze SELECT t1.id, t2.url FROM referral_temp t2 LEFT OUTER > JOIN d_referral t1 ON t2.url = t1.referral_raw_url ORDER BY t1.id; > QUERY PLAN > ---

Re: [PERFORM] Looking for ideas on how to speed up warehouse loading

2004-04-22 Thread Nicholas Shanny
One other thing: we are running with a block size of 32K. Nick Shanny (Brother of above person) On Apr 22, 2004, at 7:30 PM, Sean Shanny wrote: I should have included this as well: show all; name |setting + add_missing

Re: [PERFORM] Looking for ideas on how to speed up warehouse loading

2004-04-22 Thread Sean Shanny
I should have included this as well: show all; name |setting + add_missing_from | on australian_timezones | off authentication_timeout | 60 check_function_bodies | on checkpoint_s

Re: [PERFORM] Wierd context-switching issue on Xeon patch for 7.4.1

2004-04-22 Thread Rod Taylor
On Thu, 2004-04-22 at 13:55, Tom Lane wrote: > Josh Berkus <[EMAIL PROTECTED]> writes: > > This may be a moot point, since you've stated that changing the loop timing > > won't solve the problem, but what about making the test part of make? I > > don't think too many systems are going to change

Re: [PERFORM] Setting Shared Buffers , Effective Cache, Sort Mem

2004-04-22 Thread scott.marlowe
On Thu, 22 Apr 2004, Pallav Kalva wrote: > Hi > > We are in the process of building a new machine for our production > database. Below you will see some of the harware specs for the machine. > I need some help with setting these parameters (shared buffers, > effective cache, sort mem) in t

Re: [PERFORM] Wierd context-switching issue on Xeon patch for 7.4.1

2004-04-22 Thread Magnus Hagander
>> Having to recompile to run on single- vs dual-processor >machines doesn't >> seem like it would fly. > >Oh, I don't know. Many applications require compiling for a target >architecture; SQL Server, for example, won't use a 2nd >processor without >re-installation. I'm not sure about Oracle

Re: [PERFORM] Setting Shared Buffers , Effective Cache, Sort Mem Parameters

2004-04-22 Thread Manfred Koizar
On Thu, 22 Apr 2004 13:51:42 -0400, Pallav Kalva <[EMAIL PROTECTED]> wrote: >I need some help with setting these parameters (shared buffers, >effective cache, sort mem) in the pg_conf file. It really depends on the kind of queries you intend to run, the number of concurrent active connections, th

[PERFORM] Looking for ideas on how to speed up warehouse loading

2004-04-22 Thread Sean Shanny
To all, Essentials: Running 7.4.1 on OSX on a loaded G5 with dual procs, 8GB memory, direct attached via fibre channel to a fully optioned 3.5TB XRaid (14 spindles, 2 sets of 7 in RAID 5) box running RAID 50. Background: We are loading what are essentially xml based access logs from about 20

Re: [PERFORM] Wierd context-switching issue on Xeon patch for 7.4.1

2004-04-22 Thread Josh Berkus
Tom, > The tricky > part is that a slow adaptation rate means we can't have every backend > figuring this out for itself --- the right value would have to be > maintained globally, and I'm not sure how to do that without adding a > lot of overhead. This may be a moot point, since you've stated th

Re: [pgsql-advocacy] [PERFORM] MySQL vs PG TPC-H benchmarks

2004-04-22 Thread Eduardo Almeida
Folks, I forgot to mention that I used Shell scripts to load the data and use Java just to run the refresh functions. Talking about sort_mem config, I used 65000 but in the TPCH specification they said that you are not able to change the configs when you start the benchmark, is that a big problem

Re: [PERFORM] Wierd context-switching issue on Xeon patch for 7.4.1

2004-04-22 Thread Tom Lane
Josh Berkus <[EMAIL PROTECTED]> writes: > This may be a moot point, since you've stated that changing the loop timing > won't solve the problem, but what about making the test part of make? I > don't think too many systems are going to change processor architectures once > in production, and th

Re: [PERFORM] Wierd context-switching issue on Xeon patch for 7.4.1

2004-04-22 Thread Josh Berkus
Tom, > Having to recompile to run on single- vs dual-processor machines doesn't > seem like it would fly. Oh, I don't know. Many applications require compiling for a target architecture; SQL Server, for example, won't use a 2nd processor without re-installation. I'm not sure about Oracle. I

Re: [pgsql-advocacy] [PERFORM] MySQL vs PG TPC-H benchmarks

2004-04-22 Thread Markus Bertheau
Ð ÐÑÐ, 22.04.2004, Ð 17:54, Tom Lane ÐÐÑÐÑ: > Eduardo Almeida <[EMAIL PROTECTED]> writes: > > About 7hs:30min to load the data and 16:09:25 to > > create the indexes > > You could probably improve the index-create time by temporarily > increasing sort_mem. It wouldn't be unreasonable to give CREA

Re: [PERFORM] Wierd context-switching issue on Xeon patch for 7.4.1

2004-04-22 Thread Bruce Momjian
Josh Berkus wrote: > Tom, > > > Having to recompile to run on single- vs dual-processor machines doesn't > > seem like it would fly. > > Oh, I don't know. Many applications require compiling for a target > architecture; SQL Server, for example, won't use a 2nd processor without > re-installati

Re: [PERFORM] 225 times slower

2004-04-22 Thread Pailloncy Jean-Gérard
The planner is guessing that scanning in rec_id order will produce a matching row fairly quickly (sooner than selecting all the matching rows and sorting them would do). It's wrong in this case, but I'm not sure it could do better without very detailed cross-column statistics. Am I right to gues

Re: [pgsql-advocacy] [PERFORM] MySQL vs PG TPC-H benchmarks

2004-04-22 Thread Tom Lane
Markus Bertheau <[EMAIL PROTECTED]> writes: >> You could probably improve the index-create time by temporarily >> increasing sort_mem. It wouldn't be unreasonable to give CREATE INDEX >> several hundred meg to work in. (You don't want sort_mem that big >> normally, because there may be many sorts

[PERFORM] Setting Shared Buffers , Effective Cache, Sort Mem Parameters

2004-04-22 Thread Pallav Kalva
Hi We are in the process of building a new machine for our production database. Below you will see some of the harware specs for the machine. I need some help with setting these parameters (shared buffers, effective cache, sort mem) in the pg_conf file. Also can anyone explain the differen

Re: [pgsql-advocacy] [PERFORM] MySQL vs PG TPC-H benchmarks

2004-04-22 Thread Tom Lane
Eduardo Almeida <[EMAIL PROTECTED]> writes: > About 7hs:30min to load the data and 16:09:25 to > create the indexes You could probably improve the index-create time by temporarily increasing sort_mem. It wouldn't be unreasonable to give CREATE INDEX several hundred meg to work in. (You don't wan

Re: [pgsql-advocacy] [PERFORM] MySQL vs PG TPC-H benchmarks

2004-04-22 Thread Grega Bremec
...and on Thu, Apr 22, 2004 at 05:53:18AM -0700, Eduardo Almeida used the keyboard: > > - The configuration of the machine is: > Dual opteron 64 bits model 240 > 4GB RAM > 960 GB on RAID 0 > Mandrake Linux 64 with Kernel 2.6.5 (I compiled a > kernel for this test) > Java SDK java version "1.4.2_

Re: [pgsql-advocacy] [PERFORM] MySQL vs PG TPC-H benchmarks

2004-04-22 Thread Jan Wieck
Eduardo Almeida wrote: Folks, I’m doing the 100GB TPC-H and I’ll show the previous results to our community (Postgres) in 3 weeks before finishing the study. My intention is to carry through a test with a VLDB in a low cost platform (PostgreSQL, Linux and cheap HW) and not to compare with another

Re: [pgsql-advocacy] [PERFORM] MySQL vs PG TPC-H benchmarks

2004-04-22 Thread Eduardo Almeida
Grega, That´s why I used java 32bits and needed to compile the kernel 2.6.5 with the 32bits modules. To reference, Sun has java 64bits just to IA64 and Solaris Sparc 64 not to Opteron. regards, Eduardo --- Grega Bremec <[EMAIL PROTECTED]> wrote: > ...and on Thu, Apr 22, 2004 at 05:53:18AM -0700,

Re: [pgsql-advocacy] [PERFORM] MySQL vs PG TPC-H benchmarks

2004-04-22 Thread Eduardo Almeida
Folks, I’m doing the 100GB TPC-H and I’ll show the previous results to our community (Postgres) in 3 weeks before finishing the study. My intention is to carry through a test with a VLDB in a low cost platform (PostgreSQL, Linux and cheap HW) and not to compare with another DBMS. So far I can te

Re: [PERFORM] Wierd context-switching issue on Xeon patch for 7.4.1

2004-04-22 Thread Tom Lane
Dave Cramer <[EMAIL PROTECTED]> writes: > My hypothesis is that if you spin approximately the same or more time > than the average time it takes to get finished with the shared resource > then this should reduce cs. The only thing we use spinlocks for nowadays is to protect LWLocks, so the "averag

Re: [PERFORM] Wierd context-switching issue on Xeon

2004-04-22 Thread Tom Lane
Paul Tuckfield <[EMAIL PROTECTED]> writes: >> I used the taskset command: >> taskset 01 -p >> taskset 01 -p >> >> I guess that 0 and 1 are the two cores (pipelines? hyper-threads?) on >> the first Xeon processor in the box. AFAICT, what you've actually done here is to bind both backends to the

Re: [PERFORM] Wierd context-switching issue on Xeon patch for 7.4.1

2004-04-22 Thread Dave Cramer
More data On a dual xeon with HTT enabled: I tried increasing the NUM_SPINS to 1000 and it works better. NUM_SPINLOCKS CS ID pgbench 100 250K59% 230 TPS 1000125K55% 228 TPS This is certainly heading in the right direction ? Although it lo