Re: Writing 1100 rows per second

2020-02-09 Thread Jeff Janes
On Wed, Feb 5, 2020 at 12:25 PM Arya F wrote: > If I run the database on a server that has enough ram to load all the > indexes and tables into ram. And then it would update the index on the HDD > every x seconds. Would that work to increase performance dramatically? > Perhaps. Probably not

Re: Writing 1100 rows per second

2020-02-06 Thread Ogden Brash
On Wed, Feb 5, 2020 at 9:12 AM Laurenz Albe wrote: > One idea I can come up with is a table that is partitioned by a column > that appears > in a selective search condition, but have no indexes on the table, so that > you always get > away with a sequential scan of a single partition. > > This

Re: Writing 1100 rows per second

2020-02-05 Thread Haroldo Kerry
Arya, We ran into the issue of decreasing insert performance for tables of hundreds of millions of rows and they are indeed due to index updates. We tested TimescaleDB (a pgsql plugin) with success in all use cases that we have. It does a "behind the scenes" single-level partitioning that is

Re: Writing 1100 rows per second

2020-02-05 Thread Arya F
If I run the database on a server that has enough ram to load all the indexes and tables into ram. And then it would update the index on the HDD every x seconds. Would that work to increase performance dramatically? On Wed, Feb 5, 2020, 12:15 PM Justin Pryzby wrote: > On Wed, Feb 05, 2020 at

Re: Writing 1100 rows per second

2020-02-05 Thread Justin Pryzby
On Wed, Feb 05, 2020 at 12:03:52PM -0500, Arya F wrote: > I'm looking to write about 1100 rows per second to tables up to 100 million > rows. I'm trying to come up with a design that I can do all the writes to a > database with no indexes. When having indexes the write performance slows > down

Re: Writing 1100 rows per second

2020-02-05 Thread Laurenz Albe
On Wed, 2020-02-05 at 12:03 -0500, Arya F wrote: > I'm looking to write about 1100 rows per second to tables up to 100 million > rows. I'm trying to > come up with a design that I can do all the writes to a database with no > indexes. When having > indexes the write performance slows down

Writing 1100 rows per second

2020-02-05 Thread Arya F
I'm looking to write about 1100 rows per second to tables up to 100 million rows. I'm trying to come up with a design that I can do all the writes to a database with no indexes. When having indexes the write performance slows down dramatically after the table gets bigger than 30 million rows. I