Re: [PERFORM] Millions of tables

2016-09-25 Thread julyanto SUTANDANG
-sorry for my last email, which also not bottom posting- Hi Greg, On Mon, Sep 26, 2016 at 11:19 AM, Greg Spiegelberg wrote: > I did look at PostgresXL and CitusDB. Both are admirable however neither > could support the need to read a random record consistently under 30ms. > It's a similar probl

Re: [PERFORM] Millions of tables

2016-09-25 Thread Jeff Janes
On Sun, Sep 25, 2016 at 7:50 PM, Greg Spiegelberg wrote: > Hey all, > > Obviously everyone who's been in PostgreSQL or almost any RDBMS for a time > has said not to have millions of tables. I too have long believed it until > recently. > > AWS d2.8xlarge instance with 9.5 is my test rig using XF

Re: [PERFORM] Millions of tables

2016-09-25 Thread Gavin Flower
Hi Greg, Please follow the conventions of this mailing list, to avoid confusion - see bottom of this posting for further comments On 26/09/16 17:05, Greg Spiegelberg wrote: Precisely why I shared with the group. I must understand the risks involved. I need to explore if it can be stable a

Re: [PERFORM] Millions of tables

2016-09-25 Thread Greg Spiegelberg
I did look at PostgresXL and CitusDB. Both are admirable however neither could support the need to read a random record consistently under 30ms. It's a similar problem Cassandra and others have: network latency. At this scale, to provide the ability to access any given record amongst trillions it

Re: [PERFORM] Millions of tables

2016-09-25 Thread Greg Spiegelberg
Precisely why I shared with the group. I must understand the risks involved. I need to explore if it can be stable at this size when does it become unstable? Aside from locking down user access to superuser, is there a way to prohibit database-wide VACUUM & ANALYZE? Certainly putting my trust i

Re: [PERFORM] Millions of tables

2016-09-25 Thread Mike Sofen
From: Greg Spiegelberg Sent: Sunday, September 25, 2016 7:50 PM … Over the weekend, I created 8M tables with 16M indexes on those tables. … A system or database crash could take potentially hours to days to recover. There are likely other issues ahead. You may wonder, "why is Greg attempti

Re: [PERFORM] Millions of tables

2016-09-25 Thread julyanto SUTANDANG
Dear Greg, Have you checked PostgresXL ? with millions of table, how the apps choose which table is approriate? in my opinion, with that scale it should go with parallel query with data sharing like what PostgresXL is done. Thanks, Julyanto SUTANDANG Equnix Business Solutions, PT (An Open Sour

[PERFORM] Millions of tables

2016-09-25 Thread Greg Spiegelberg
Hey all, Obviously everyone who's been in PostgreSQL or almost any RDBMS for a time has said not to have millions of tables. I too have long believed it until recently. AWS d2.8xlarge instance with 9.5 is my test rig using XFS on EBS (io1) for PGDATA. Over the weekend, I created 8M tables with

Re: [PERFORM] Storing large documents - one table or partition by doc?

2016-09-25 Thread Jeff Janes
On Fri, Sep 23, 2016 at 3:12 AM, Dev Nop wrote: > I’m storing thousands of independent documents each containing around 20k > rows. The larger the document, the more likely it is to be active with > inserts and updates (1000s/day). The most common read query is to get all > the rows for a single