Re: [PERFORM] Millions of tables

2016-09-25 Thread Gavin Flower
Hi Greg, Please follow the conventions of this mailing list, to avoid confusion - see bottom of this posting for further comments On 26/09/16 17:05, Greg Spiegelberg wrote: Precisely why I shared with the group. I must understand the risks involved. I need to explore if it can be stable

Re: [PERFORM] Millions of tables

2016-09-25 Thread Greg Spiegelberg
I did look at PostgresXL and CitusDB. Both are admirable however neither could support the need to read a random record consistently under 30ms. It's a similar problem Cassandra and others have: network latency. At this scale, to provide the ability to access any given record amongst trillions

Re: [PERFORM] Millions of tables

2016-09-25 Thread Greg Spiegelberg
Precisely why I shared with the group. I must understand the risks involved. I need to explore if it can be stable at this size when does it become unstable? Aside from locking down user access to superuser, is there a way to prohibit database-wide VACUUM & ANALYZE? Certainly putting my trust

Re: [PERFORM] Millions of tables

2016-09-25 Thread Mike Sofen
From: Greg Spiegelberg Sent: Sunday, September 25, 2016 7:50 PM … Over the weekend, I created 8M tables with 16M indexes on those tables. … A system or database crash could take potentially hours to days to recover. There are likely other issues ahead. You may wonder, "why is Greg

Re: [PERFORM] Millions of tables

2016-09-25 Thread julyanto SUTANDANG
Dear Greg, Have you checked PostgresXL ? with millions of table, how the apps choose which table is approriate? in my opinion, with that scale it should go with parallel query with data sharing like what PostgresXL is done. Thanks, Julyanto SUTANDANG Equnix Business Solutions, PT (An Open

[PERFORM] Millions of tables

2016-09-25 Thread Greg Spiegelberg
Hey all, Obviously everyone who's been in PostgreSQL or almost any RDBMS for a time has said not to have millions of tables. I too have long believed it until recently. AWS d2.8xlarge instance with 9.5 is my test rig using XFS on EBS (io1) for PGDATA. Over the weekend, I created 8M tables with