I am using npgsql in my c# program,
I have a table in which we need to dump data from a flat file approx. 5.5
million rows once in every 24 hours.
Soon it will grow to 10 million rows.
We are using npgsql binary import and all this is done (5.5 million)
inserts in < 3 minutes.
Then we create two in
On Wed, Feb 5, 2020 at 12:25 PM Arya F wrote:
> If I run the database on a server that has enough ram to load all the
> indexes and tables into ram. And then it would update the index on the HDD
> every x seconds. Would that work to increase performance dramatically?
>
Perhaps. Probably not dra
On Wed, Feb 5, 2020 at 9:12 AM Laurenz Albe
wrote:
> One idea I can come up with is a table that is partitioned by a column
> that appears
> in a selective search condition, but have no indexes on the table, so that
> you always get
> away with a sequential scan of a single partition.
>
>
This is
Arya,
We ran into the issue of decreasing insert performance for tables of
hundreds of millions of rows and they are indeed due to index updates.
We tested TimescaleDB (a pgsql plugin) with success in all use cases that
we have. It does a "behind the scenes" single-level partitioning that is
indeed
If I run the database on a server that has enough ram to load all the
indexes and tables into ram. And then it would update the index on the HDD
every x seconds. Would that work to increase performance dramatically?
On Wed, Feb 5, 2020, 12:15 PM Justin Pryzby wrote:
> On Wed, Feb 05, 2020 at 12:
On Wed, Feb 05, 2020 at 12:03:52PM -0500, Arya F wrote:
> I'm looking to write about 1100 rows per second to tables up to 100 million
> rows. I'm trying to come up with a design that I can do all the writes to a
> database with no indexes. When having indexes the write performance slows
> down dram
On Wed, 2020-02-05 at 12:03 -0500, Arya F wrote:
> I'm looking to write about 1100 rows per second to tables up to 100 million
> rows. I'm trying to
> come up with a design that I can do all the writes to a database with no
> indexes. When having
> indexes the write performance slows down dramati
I'm looking to write about 1100 rows per second to tables up to 100 million
rows. I'm trying to come up with a design that I can do all the writes to a
database with no indexes. When having indexes the write performance slows
down dramatically after the table gets bigger than 30 million rows.
I w
