Hi Yong, Prithvi, I think Yong's header idea is reasonable for some deployments. I agree that in general routing R/O requests to a replica can violate clients' consistency expectations. However, if in a particular situation the client (admin user) is aware of the expectations, using this as an optimization switch can be acceptable.
Also, all GET requests are read-only by definition, but not all GETs may be able to tolerate reading stale data. Perhaps the matter can be clarified by using a different header name to highlight the implications. Something like "Polaris-Out-Of-Band: true". In this form the header is applicable to all requests, but will alter execution only for GETs. WDYT? Implementation-wise, if we have two different connection pools, the InMemoryEntityCache may become more of an obstacle than an aid since replication delays can cause odd effects in the cache (if it is shared between the two connection pools). For the sake of sanity, it may be necessary to use different entity caches for each connection pool (or not use the cache at all [1]). [1] https://lists.apache.org/thread/tl6z48gdblsko3x0b9nt2917wb2fgtor Cheers, Dmitri. On Sun, Sep 13, 2026 at 11:32 PM Yong Zheng <[email protected]> wrote: > Hello, > > Thanks for the quick review Prithvi. Yes around the rps as those are only > provided as a quick math to show the scaling issue that people can face (or > may already faced) when having a large lakehouse in an enterprise > environment. > > I do agreed that we will make the RO DB endpoint optional (so existed > deployment will continue to behavior the same way) and can be add when > there is a needed for scaling by splitting readonly requests into this > optional RO DB endpoint. However, I don't think this is a good idea to make > the server blindly route non-GET requests to default DB endpoint and GET > requests to optional RO DB endpoint. As called out in your response, > replication can have latency and it can have big impacts when those > happened and caused un-intensional retry from custom Iceberg client (e.g. > if people are doing a write then checking if write completed, then retry if > write was not committed...in this case, latency can cause un-necessary > retry as well as potentially dup ingestion for custom Iceberg writer). > > As we do use custom header to route requests to different REALM in > Polaris, I think it is safer for client to decide if they should route > requests to RO endpoint via another customer header as oppose as let server > handles this blindly as described above. The workload used by Anand's is > not very likely a real world workload or may only covered part of the use > cases on how people are using Iceberg. I don' think we should use that as a > way to say 60% of the workload is listNamespaces which doesn't change. For > deployment where there are high number of tables (e.g. tenant level tables > under different namespaces) and custom data-ops, checking if table > maintenance among all tables is critical and those do does very frequent. > Thoughts? > > Thanks, > Yong Zheng > > > > On 2026/09/13 11:44:27 Prithvi S wrote: > > Hi Yong, > > > > Thanks for writing this up, and for connecting it with Anand's load-test > > write-up. The connection math is a good place to start. With JDBC > > persistence, the Agroal pool is the limit on each pod > > (quarkus.datasource.jdbc.max-size, commonly 50), and max_connections on > the > > primary is the limit on the fleet. That is a real scaling axis, and using > > read replicas is worth discussing. > > > > I would be careful about treating ~5,333 rps as a planning target :) > > Anand's run was ~400 rps sustained and ~800 rps peak on 15 pods, with low > > CPU and well under one in-flight request per pod on average. The > > 50-connection pool was not what was biting, most of those connections > were > > idle. Scaling from there is likely to hit database CPU, I/O, WAL, or lock > > contention before a clean 5K-connection wall. There are also cheaper > levers > > in front of replicas.. right-sizing pool and pod count, and multiplexing > > with something like PgBouncer or RDS Proxy so pod count and database > > sessions do not grow 1:1. > > > > On Polaris-Readonly, I am a bit hesitant to make this a client-side > routing > > decision. main concern is consistency. Iceberg clients can expect a read > > after a write in the same session to see that write, for example > > commitTable followed by loadTable. RDS/Aurora replicas are asynchronous, > so > > sending those reads to a replica is a change in consistency model. That > > should be an explicit contract, not something controlled by a header. > > > > There is also a practical issue with standard Iceberg clients. Spark, > > Flink, Trino, and PyIceberg do not know about Polaris-Readonly. A static > > header applies to the whole catalog, so a stray mutation, or something > like > > an Iceberg metrics report, could land on the replica too. And some > > operations that look like reads can still write in Polaris (persisted > > events, idempotency bookkeeping). The header does not turn those side > > effects off. If a mutation does hit the replica, I agree we should not > > silently fall back to the primary, but Polaris should fail that itself > with > > a clear 4xx, not with a Postgres read-only error. > > > > I think a better path is to keep today's behavior as the default and make > > replica use opt-in on the server: > > > > - Add an optional second Quarkus datasource pointing at the replica. > > - Keep mutations, auth, and anything that must be linearizable with a > > prior write on the primary; pure metadata reads may use the replica. > > - Do not silently fall back to the primary if a replica call fails. > > - Document that opted-in reads can see replica lag. > > > > That also fits the two-workload setup you described. The ingestion fleet > > can use the writer endpoint and the serving fleet the reader endpoint. > The > > mixed case (token issue on RW, catalog reads on RO) can then be handled > > inside Polaris, rather than asking Iceberg clients to understand a new > > header. One other point from Anand's article, around 60% of the 30M > > requests were listNamespaces against a catalog that barely changes. > > Replicas would absorb that, but so would connector-side caching or a > longer > > refresh interval. > > > > I think this server-side dual-datasource approach is the right way to > move > > forward. WDYT? > > > > Regards, > > Prithvi S > > > > On Sun, Sep 13, 2026 at 9:45 AM Yong Zheng <[email protected]> wrote: > > > > > Hello, > > > > > > When using JDBC as the backend metastore for Polaris, the JDBC > connection > > > capacity of the active primary can become a limiting factor for the > > > throughput of a given Polaris deployment. Taking RDS with PostgreSQL > as an > > > example, if we are using an AWS db.m9g.4xlarge (16 cores and 64 GB) as > the > > > backend store, we can get up to 5K connections by default. AWS uses > > > "LEAST({DBInstanceClassMemory/9531392}, 5000)" for the default > PostgreSQL > > > "max_connections", and this can be increased manually. > > > > > > Now, assuming we put 50 connections per pod, we can have up to 100 pods > > > max (in reality, it will be less as a couple of connections are > reserved > > > for the superuser, but let's stick with 100 pods to make the math > simpler). > > > > > > With 50 connections per pod, this matches what Anand reported in > > > > https://medium.com/@obelix74/a-30-million-request-load-test-for-apache-polaris-fb5690040154 > . > > > If we assume linear scaling from the benchmark (which may not hold due > to > > > database CPU, I/O, locking, and other bottlenecks), the theoretical > > > throughput would be ~5,333 requests per second. This is great, but if > we > > > ever want to handle more throughput, we would have no other option > other > > > than scaling up the RDS instance and overriding the default max allowed > > > connections on the DB server when using the native AWS solution. There > are > > > other solutions out there, such as using a multi-master deployment for > the > > > backend DB or switching to a NoSQL backend, which we can scale up a lot > > > easier (e.g. MongoDB). > > > > > > While there are solutions to scale up the infrastructure to support > more > > > connections, one particular thing that caught my eye is that we are not > > > using read-only replicas at all. > > > > > > Assuming we have two primary workloads: > > > > > > 1. Ingestion layer: this has both read and write. > > > 2. Query/Serving layer: this is read-only queries to power various > > > dashboards. > > > > > > We could also have two DB connection endpoints: > > > > > > 1. RW connection endpoint (active primary) > > > 2. RO connection endpoint (read replicas) > > > > > > By default, everything should go to the RW endpoint. This matches the > > > current workflow we support. However, if we know a query/serving layer > is > > > read-only, we should route those requests to the RO connection > endpoint; > > > the auth token request/renewal would still be fulfilled through the RW > > > connection endpoint for the query/serving layer. > > > > > > Now, to decide where a client should be sending requests to RW/RO, we > can > > > check the following: > > > > > > 1. Is this an auth request? If yes, always use the RW endpoint. > > > 2. Does this request have a special header (e.g. "Polaris-Readonly", > which > > > defaults to false or unset)? If yes, offload the request to the RO > endpoint. > > > > > > In this case, if a non-read-only request is ever sent by the client by > > > accident, it will be failed by the backend DB because writes are not > > > supported on RO replicas, and this is expected behavior. We should not > > > silently fall back to the RW endpoint in this case. > > > > > > With this approach, we can squeeze higher requests per second out of a > > > given setup by offloading the read-heavy query/serving workload from > the > > > primary. If the load from the query/serving layer is significant, this > > > could give us another dimension to scale the infrastructure without > having > > > to keep scaling up the primary DB. > > > > > > Thanks, > > > Yong Zheng > > > > > >
