Hi I think Prithvi is right about server-side/opt-in.
DatasourceOperations (in persistence/relational-jdbc) already separates executeSelect* (no explicit transaction) from executeUpdate/runWithinTransaction (explicit commit/rollback). A second read-replica DataSource could plausibly be swapped in right: this is "natural". Routing on a client supplied header is risky for a spec-compliant Iceberg REST server like Polaris: Spark, Flink and other query engines have no idea Polaris-Readonly exists, so the header can only ever be set by custom callers. In a way, it breaks the Polaris interoperability value. Also, routing on HTTP verb alone (server-side, blind GET vs non-GET) is also unfafe imho: JdbcBasePersistenceImpl has read-like calls with side effects (idempotency-key bookkeeping, persisted events). Verb-based routing at the HTTP filter layer (there's a precedent for that kind of filter with RealmContextFilter) doesn't know which "reads" quietly write. So, I think the routing decision needs to happen at the persistence-method level (lookupEntity vs writeEntity, inside JdbcBasePersistenceImpl/DatasourceOperations), where operation semantics are actually clearly known (not at the HTTP edge based on verb or a header). Today there's no multi-datasource support: multi-tenancy is pure row-level realmId filtering on one shared pool. A read-replica split is a new persistence-layer pattern, not a config tweak imho. As it touches retry semantics too (I have to admit that the executeSelect() return fix is already a bit fragile), it probably deserves some design discussion, rather than converging via header-name bikeshedding. Regards JB On Mon, Sep 14, 2026 at 5:32 AM Yong Zheng <[email protected]> wrote: > > Hello, > > Thanks for the quick review Prithvi. Yes around the rps as those are only > provided as a quick math to show the scaling issue that people can face (or > may already faced) when having a large lakehouse in an enterprise environment. > > I do agreed that we will make the RO DB endpoint optional (so existed > deployment will continue to behavior the same way) and can be add when there > is a needed for scaling by splitting readonly requests into this optional RO > DB endpoint. However, I don't think this is a good idea to make the server > blindly route non-GET requests to default DB endpoint and GET requests to > optional RO DB endpoint. As called out in your response, replication can have > latency and it can have big impacts when those happened and caused > un-intensional retry from custom Iceberg client (e.g. if people are doing a > write then checking if write completed, then retry if write was not > committed...in this case, latency can cause un-necessary retry as well as > potentially dup ingestion for custom Iceberg writer). > > As we do use custom header to route requests to different REALM in Polaris, I > think it is safer for client to decide if they should route requests to RO > endpoint via another customer header as oppose as let server handles this > blindly as described above. The workload used by Anand's is not very likely a > real world workload or may only covered part of the use cases on how people > are using Iceberg. I don' think we should use that as a way to say 60% of the > workload is listNamespaces which doesn't change. For deployment where there > are high number of tables (e.g. tenant level tables under different > namespaces) and custom data-ops, checking if table maintenance among all > tables is critical and those do does very frequent. Thoughts? > > Thanks, > Yong Zheng > > > > On 2026/09/13 11:44:27 Prithvi S wrote: > > Hi Yong, > > > > Thanks for writing this up, and for connecting it with Anand's load-test > > write-up. The connection math is a good place to start. With JDBC > > persistence, the Agroal pool is the limit on each pod > > (quarkus.datasource.jdbc.max-size, commonly 50), and max_connections on the > > primary is the limit on the fleet. That is a real scaling axis, and using > > read replicas is worth discussing. > > > > I would be careful about treating ~5,333 rps as a planning target :) > > Anand's run was ~400 rps sustained and ~800 rps peak on 15 pods, with low > > CPU and well under one in-flight request per pod on average. The > > 50-connection pool was not what was biting, most of those connections were > > idle. Scaling from there is likely to hit database CPU, I/O, WAL, or lock > > contention before a clean 5K-connection wall. There are also cheaper levers > > in front of replicas.. right-sizing pool and pod count, and multiplexing > > with something like PgBouncer or RDS Proxy so pod count and database > > sessions do not grow 1:1. > > > > On Polaris-Readonly, I am a bit hesitant to make this a client-side routing > > decision. main concern is consistency. Iceberg clients can expect a read > > after a write in the same session to see that write, for example > > commitTable followed by loadTable. RDS/Aurora replicas are asynchronous, so > > sending those reads to a replica is a change in consistency model. That > > should be an explicit contract, not something controlled by a header. > > > > There is also a practical issue with standard Iceberg clients. Spark, > > Flink, Trino, and PyIceberg do not know about Polaris-Readonly. A static > > header applies to the whole catalog, so a stray mutation, or something like > > an Iceberg metrics report, could land on the replica too. And some > > operations that look like reads can still write in Polaris (persisted > > events, idempotency bookkeeping). The header does not turn those side > > effects off. If a mutation does hit the replica, I agree we should not > > silently fall back to the primary, but Polaris should fail that itself with > > a clear 4xx, not with a Postgres read-only error. > > > > I think a better path is to keep today's behavior as the default and make > > replica use opt-in on the server: > > > > - Add an optional second Quarkus datasource pointing at the replica. > > - Keep mutations, auth, and anything that must be linearizable with a > > prior write on the primary; pure metadata reads may use the replica. > > - Do not silently fall back to the primary if a replica call fails. > > - Document that opted-in reads can see replica lag. > > > > That also fits the two-workload setup you described. The ingestion fleet > > can use the writer endpoint and the serving fleet the reader endpoint. The > > mixed case (token issue on RW, catalog reads on RO) can then be handled > > inside Polaris, rather than asking Iceberg clients to understand a new > > header. One other point from Anand's article, around 60% of the 30M > > requests were listNamespaces against a catalog that barely changes. > > Replicas would absorb that, but so would connector-side caching or a longer > > refresh interval. > > > > I think this server-side dual-datasource approach is the right way to move > > forward. WDYT? > > > > Regards, > > Prithvi S > > > > On Sun, Sep 13, 2026 at 9:45 AM Yong Zheng <[email protected]> wrote: > > > > > Hello, > > > > > > When using JDBC as the backend metastore for Polaris, the JDBC connection > > > capacity of the active primary can become a limiting factor for the > > > throughput of a given Polaris deployment. Taking RDS with PostgreSQL as an > > > example, if we are using an AWS db.m9g.4xlarge (16 cores and 64 GB) as the > > > backend store, we can get up to 5K connections by default. AWS uses > > > "LEAST({DBInstanceClassMemory/9531392}, 5000)" for the default PostgreSQL > > > "max_connections", and this can be increased manually. > > > > > > Now, assuming we put 50 connections per pod, we can have up to 100 pods > > > max (in reality, it will be less as a couple of connections are reserved > > > for the superuser, but let's stick with 100 pods to make the math > > > simpler). > > > > > > With 50 connections per pod, this matches what Anand reported in > > > https://medium.com/@obelix74/a-30-million-request-load-test-for-apache-polaris-fb5690040154. > > > If we assume linear scaling from the benchmark (which may not hold due to > > > database CPU, I/O, locking, and other bottlenecks), the theoretical > > > throughput would be ~5,333 requests per second. This is great, but if we > > > ever want to handle more throughput, we would have no other option other > > > than scaling up the RDS instance and overriding the default max allowed > > > connections on the DB server when using the native AWS solution. There are > > > other solutions out there, such as using a multi-master deployment for the > > > backend DB or switching to a NoSQL backend, which we can scale up a lot > > > easier (e.g. MongoDB). > > > > > > While there are solutions to scale up the infrastructure to support more > > > connections, one particular thing that caught my eye is that we are not > > > using read-only replicas at all. > > > > > > Assuming we have two primary workloads: > > > > > > 1. Ingestion layer: this has both read and write. > > > 2. Query/Serving layer: this is read-only queries to power various > > > dashboards. > > > > > > We could also have two DB connection endpoints: > > > > > > 1. RW connection endpoint (active primary) > > > 2. RO connection endpoint (read replicas) > > > > > > By default, everything should go to the RW endpoint. This matches the > > > current workflow we support. However, if we know a query/serving layer is > > > read-only, we should route those requests to the RO connection endpoint; > > > the auth token request/renewal would still be fulfilled through the RW > > > connection endpoint for the query/serving layer. > > > > > > Now, to decide where a client should be sending requests to RW/RO, we can > > > check the following: > > > > > > 1. Is this an auth request? If yes, always use the RW endpoint. > > > 2. Does this request have a special header (e.g. "Polaris-Readonly", which > > > defaults to false or unset)? If yes, offload the request to the RO > > > endpoint. > > > > > > In this case, if a non-read-only request is ever sent by the client by > > > accident, it will be failed by the backend DB because writes are not > > > supported on RO replicas, and this is expected behavior. We should not > > > silently fall back to the RW endpoint in this case. > > > > > > With this approach, we can squeeze higher requests per second out of a > > > given setup by offloading the read-heavy query/serving workload from the > > > primary. If the load from the query/serving layer is significant, this > > > could give us another dimension to scale the infrastructure without having > > > to keep scaling up the primary DB. > > > > > > Thanks, > > > Yong Zheng > > > > >
