Potluri Srikanth wrote:
> Hi all,
>
> I need to do a bulk data loading around 704GB (log file size) at
> present in 8 hrs (1 am - 9am). The data file size may increase 3 to
> 5 times in future.
>
> Using COPY it takes 96 hrs to finish the task.
> What is the best way to do it ?
>
> HARDWARE: S
On Mon, Apr 07, 2008 at 11:01:18PM -0400, Christian Bourque wrote:
> I have a performance problem with a script that does massive bulk
> insert in 6 tables. When the script starts the performance is really
> good but will degrade minute after minute and take almost a day to
> finish!
how do you do
Hell, Robert wrote:
> > I tried different other tools for random IO (including a self
written one which does random lseek and read).
> >
> > This tool, started during one of our tests, achieves 2 iops (8k each).
> > Started alone I get something about 1,500 iops with an avg latency
of 100 ms.
1500
On Tue, 8 Apr 2008, Ow Mun Heng wrote:
I moved from multicolumn indexes to individual indexes because the
queries does not always utilise the same few indexes, some users would
use
eg: index F, A, B or D,A,E or any other combination.
Yes, that does make it more tricky, but it still may be best
Worst latency was at ~600 ms.
In this test case we have only selects - so it's read only.
I tried 8.3 - it's better there (10-15 %) - but IO rates stay the same.
I use 18 GB shared memory on a machine with 32 GB during the test, I think this
shouldn't be a problem.
I did some further testing an
Oleg Bartunov wrote:
Mark,
do you know about our sky segmentation code Q3C,
see details http://www.sai.msu.su/~megera/wiki/SkyPixelization
We use it for billions objects in database and quite happy.
Oleg,
Thanks for the response. That sounds interesting, but it's not clear to
me how I would
Christian Bourque wrote:
Any idea? Is there any other improvements I could do?
Are you using the "COPY" syntax in the import script or individual
insert statements? Using COPY will always be *much* faster.
I believe COPY always appends to tables rather than replacing the
contents, you can
On Tue, 8 Apr 2008, Mark Stosberg wrote:
Any idea? Is there any other improvements I could do?
Are you using the "COPY" syntax in the import script or individual insert
statements? Using COPY will always be *much* faster.
PostgreSQL (latest versions at least) has an optimisation if you creat
samantha mahindrakar wrote:
Iam using a temporary table where in i put data from other partitoined
table. I then query this table to get the desired data.But the thing
is this temporary table has to be craeted for every record that i need
to correct and there are thousands of such records that ne
samantha mahindrakar escribió:
> So the program necessarily creates a temporary table evrytime it has
> to correct a record. However this table is dropeed after each record
> is corrected.
Perhaps it would be better to truncate the temp table instead.
> Iam not sure if i can use a cursor to repl
On Tue, 8 Apr 2008, Mark Stosberg wrote:
Oleg Bartunov wrote:
Mark,
do you know about our sky segmentation code Q3C,
see details http://www.sai.msu.su/~megera/wiki/SkyPixelization
We use it for billions objects in database and quite happy.
Oleg,
Thanks for the response. That sounds interest
Well instead of creating a temp table everytime i just created a
permanant table and insert the data into it everytime and truncate it.
I created indexes on this permanent table too. This did improve the
performance to some extent.
Does using permanant tables also bloat the catalog or hinder the p
samantha mahindrakar escribió:
> Well instead of creating a temp table everytime i just created a
> permanant table and insert the data into it everytime and truncate it.
> I created indexes on this permanent table too. This did improve the
> performance to some extent.
>
> Does using permanant ta
When traffic to our PostgreSQL-backed website spikes, the first resource
we see being exhausted is the DB slots on the master server (currently
set to about 400).
I expect that as new Apache/mod_perl children are being put to us, they
are creating new database connections.
I'm interested in re
I have a performance problem with a script that does massive bulk
insert in 6 tables. When the script starts the performance is really
good but will degrade minute after minute and take almost a day to
finish!
Looks like foreign key checks slow you down.
- Batch INSERTS in trans
15 matches
Mail list logo
