Hi Yong, Thanks for writing this up, and for connecting it with Anand's load-test write-up. The connection math is a good place to start. With JDBC persistence, the Agroal pool is the limit on each pod (quarkus.datasource.jdbc.max-size, commonly 50), and max_connections on the primary is the limit on the fleet. That is a real scaling axis, and using read replicas is worth discussing.
I would be careful about treating ~5,333 rps as a planning target :) Anand's run was ~400 rps sustained and ~800 rps peak on 15 pods, with low CPU and well under one in-flight request per pod on average. The 50-connection pool was not what was biting, most of those connections were idle. Scaling from there is likely to hit database CPU, I/O, WAL, or lock contention before a clean 5K-connection wall. There are also cheaper levers in front of replicas.. right-sizing pool and pod count, and multiplexing with something like PgBouncer or RDS Proxy so pod count and database sessions do not grow 1:1. On Polaris-Readonly, I am a bit hesitant to make this a client-side routing decision. main concern is consistency. Iceberg clients can expect a read after a write in the same session to see that write, for example commitTable followed by loadTable. RDS/Aurora replicas are asynchronous, so sending those reads to a replica is a change in consistency model. That should be an explicit contract, not something controlled by a header. There is also a practical issue with standard Iceberg clients. Spark, Flink, Trino, and PyIceberg do not know about Polaris-Readonly. A static header applies to the whole catalog, so a stray mutation, or something like an Iceberg metrics report, could land on the replica too. And some operations that look like reads can still write in Polaris (persisted events, idempotency bookkeeping). The header does not turn those side effects off. If a mutation does hit the replica, I agree we should not silently fall back to the primary, but Polaris should fail that itself with a clear 4xx, not with a Postgres read-only error. I think a better path is to keep today's behavior as the default and make replica use opt-in on the server: - Add an optional second Quarkus datasource pointing at the replica. - Keep mutations, auth, and anything that must be linearizable with a prior write on the primary; pure metadata reads may use the replica. - Do not silently fall back to the primary if a replica call fails. - Document that opted-in reads can see replica lag. That also fits the two-workload setup you described. The ingestion fleet can use the writer endpoint and the serving fleet the reader endpoint. The mixed case (token issue on RW, catalog reads on RO) can then be handled inside Polaris, rather than asking Iceberg clients to understand a new header. One other point from Anand's article, around 60% of the 30M requests were listNamespaces against a catalog that barely changes. Replicas would absorb that, but so would connector-side caching or a longer refresh interval. I think this server-side dual-datasource approach is the right way to move forward. WDYT? Regards, Prithvi S On Sun, Sep 13, 2026 at 9:45 AM Yong Zheng <[email protected]> wrote: > Hello, > > When using JDBC as the backend metastore for Polaris, the JDBC connection > capacity of the active primary can become a limiting factor for the > throughput of a given Polaris deployment. Taking RDS with PostgreSQL as an > example, if we are using an AWS db.m9g.4xlarge (16 cores and 64 GB) as the > backend store, we can get up to 5K connections by default. AWS uses > "LEAST({DBInstanceClassMemory/9531392}, 5000)" for the default PostgreSQL > "max_connections", and this can be increased manually. > > Now, assuming we put 50 connections per pod, we can have up to 100 pods > max (in reality, it will be less as a couple of connections are reserved > for the superuser, but let's stick with 100 pods to make the math simpler). > > With 50 connections per pod, this matches what Anand reported in > https://medium.com/@obelix74/a-30-million-request-load-test-for-apache-polaris-fb5690040154. > If we assume linear scaling from the benchmark (which may not hold due to > database CPU, I/O, locking, and other bottlenecks), the theoretical > throughput would be ~5,333 requests per second. This is great, but if we > ever want to handle more throughput, we would have no other option other > than scaling up the RDS instance and overriding the default max allowed > connections on the DB server when using the native AWS solution. There are > other solutions out there, such as using a multi-master deployment for the > backend DB or switching to a NoSQL backend, which we can scale up a lot > easier (e.g. MongoDB). > > While there are solutions to scale up the infrastructure to support more > connections, one particular thing that caught my eye is that we are not > using read-only replicas at all. > > Assuming we have two primary workloads: > > 1. Ingestion layer: this has both read and write. > 2. Query/Serving layer: this is read-only queries to power various > dashboards. > > We could also have two DB connection endpoints: > > 1. RW connection endpoint (active primary) > 2. RO connection endpoint (read replicas) > > By default, everything should go to the RW endpoint. This matches the > current workflow we support. However, if we know a query/serving layer is > read-only, we should route those requests to the RO connection endpoint; > the auth token request/renewal would still be fulfilled through the RW > connection endpoint for the query/serving layer. > > Now, to decide where a client should be sending requests to RW/RO, we can > check the following: > > 1. Is this an auth request? If yes, always use the RW endpoint. > 2. Does this request have a special header (e.g. "Polaris-Readonly", which > defaults to false or unset)? If yes, offload the request to the RO endpoint. > > In this case, if a non-read-only request is ever sent by the client by > accident, it will be failed by the backend DB because writes are not > supported on RO replicas, and this is expected behavior. We should not > silently fall back to the RW endpoint in this case. > > With this approach, we can squeeze higher requests per second out of a > given setup by offloading the read-heavy query/serving workload from the > primary. If the load from the query/serving layer is significant, this > could give us another dimension to scale the infrastructure without having > to keep scaling up the primary DB. > > Thanks, > Yong Zheng >
