I think consistency needs to come first. Clients should be able to read their changes after a successful write, regardless of how we route requests.
Have we actually seen database connection limits become a bottleneck for reads? The connection-count calculation suggests a possible limit, but it would help to check what slows us down first. For example, loadTable also reads metadata files, and that I/O could take much longer than getting a connection from the pool. We might hit that bottleneck before running out of connections, in which case read replicas may not help much. Could we test this with a realistic workload and look at how many connections are in use, how long requests wait for a connection, and how much time goes into database queries versus metadata-file I/O? That would give us a clearer idea of whether read replicas would help and if the added complexity is justified. Thanks, Yufei On Mon, Sep 14, 2026 at 7:59 AM Dmitri Bourlatchkov <[email protected]> wrote: > Hi Yong, Prithvi, > > I think Yong's header idea is reasonable for some deployments. I agree that > in general routing R/O requests to a replica can > violate clients' consistency expectations. However, if in a particular > situation the client (admin user) is aware of the expectations, using this > as an optimization switch can be acceptable. > > Also, all GET requests are read-only by definition, but not all GETs may be > able to tolerate reading stale data. > > Perhaps the matter can be clarified by using a different header name to > highlight the implications. Something like "Polaris-Out-Of-Band: true". In > this form the header is applicable to all requests, but will alter > execution only for GETs. > > WDYT? > > Implementation-wise, if we have two different connection pools, > the InMemoryEntityCache may become more of an obstacle than an aid since > replication delays can cause odd effects in the cache (if it is shared > between the two connection pools). For the sake of sanity, it may be > necessary to use different entity caches for each connection pool (or not > use the cache at all [1]). > > [1] https://lists.apache.org/thread/tl6z48gdblsko3x0b9nt2917wb2fgtor > > Cheers, > Dmitri. > > On Sun, Sep 13, 2026 at 11:32 PM Yong Zheng <[email protected]> wrote: > > > Hello, > > > > Thanks for the quick review Prithvi. Yes around the rps as those are only > > provided as a quick math to show the scaling issue that people can face > (or > > may already faced) when having a large lakehouse in an enterprise > > environment. > > > > I do agreed that we will make the RO DB endpoint optional (so existed > > deployment will continue to behavior the same way) and can be add when > > there is a needed for scaling by splitting readonly requests into this > > optional RO DB endpoint. However, I don't think this is a good idea to > make > > the server blindly route non-GET requests to default DB endpoint and GET > > requests to optional RO DB endpoint. As called out in your response, > > replication can have latency and it can have big impacts when those > > happened and caused un-intensional retry from custom Iceberg client (e.g. > > if people are doing a write then checking if write completed, then retry > if > > write was not committed...in this case, latency can cause un-necessary > > retry as well as potentially dup ingestion for custom Iceberg writer). > > > > As we do use custom header to route requests to different REALM in > > Polaris, I think it is safer for client to decide if they should route > > requests to RO endpoint via another customer header as oppose as let > server > > handles this blindly as described above. The workload used by Anand's is > > not very likely a real world workload or may only covered part of the use > > cases on how people are using Iceberg. I don' think we should use that > as a > > way to say 60% of the workload is listNamespaces which doesn't change. > For > > deployment where there are high number of tables (e.g. tenant level > tables > > under different namespaces) and custom data-ops, checking if table > > maintenance among all tables is critical and those do does very frequent. > > Thoughts? > > > > Thanks, > > Yong Zheng > > > > > > > > On 2026/09/13 11:44:27 Prithvi S wrote: > > > Hi Yong, > > > > > > Thanks for writing this up, and for connecting it with Anand's > load-test > > > write-up. The connection math is a good place to start. With JDBC > > > persistence, the Agroal pool is the limit on each pod > > > (quarkus.datasource.jdbc.max-size, commonly 50), and max_connections on > > the > > > primary is the limit on the fleet. That is a real scaling axis, and > using > > > read replicas is worth discussing. > > > > > > I would be careful about treating ~5,333 rps as a planning target :) > > > Anand's run was ~400 rps sustained and ~800 rps peak on 15 pods, with > low > > > CPU and well under one in-flight request per pod on average. The > > > 50-connection pool was not what was biting, most of those connections > > were > > > idle. Scaling from there is likely to hit database CPU, I/O, WAL, or > lock > > > contention before a clean 5K-connection wall. There are also cheaper > > levers > > > in front of replicas.. right-sizing pool and pod count, and > multiplexing > > > with something like PgBouncer or RDS Proxy so pod count and database > > > sessions do not grow 1:1. > > > > > > On Polaris-Readonly, I am a bit hesitant to make this a client-side > > routing > > > decision. main concern is consistency. Iceberg clients can expect a > read > > > after a write in the same session to see that write, for example > > > commitTable followed by loadTable. RDS/Aurora replicas are > asynchronous, > > so > > > sending those reads to a replica is a change in consistency model. That > > > should be an explicit contract, not something controlled by a header. > > > > > > There is also a practical issue with standard Iceberg clients. Spark, > > > Flink, Trino, and PyIceberg do not know about Polaris-Readonly. A > static > > > header applies to the whole catalog, so a stray mutation, or something > > like > > > an Iceberg metrics report, could land on the replica too. And some > > > operations that look like reads can still write in Polaris (persisted > > > events, idempotency bookkeeping). The header does not turn those side > > > effects off. If a mutation does hit the replica, I agree we should not > > > silently fall back to the primary, but Polaris should fail that itself > > with > > > a clear 4xx, not with a Postgres read-only error. > > > > > > I think a better path is to keep today's behavior as the default and > make > > > replica use opt-in on the server: > > > > > > - Add an optional second Quarkus datasource pointing at the replica. > > > - Keep mutations, auth, and anything that must be linearizable with a > > > prior write on the primary; pure metadata reads may use the replica. > > > - Do not silently fall back to the primary if a replica call fails. > > > - Document that opted-in reads can see replica lag. > > > > > > That also fits the two-workload setup you described. The ingestion > fleet > > > can use the writer endpoint and the serving fleet the reader endpoint. > > The > > > mixed case (token issue on RW, catalog reads on RO) can then be handled > > > inside Polaris, rather than asking Iceberg clients to understand a new > > > header. One other point from Anand's article, around 60% of the 30M > > > requests were listNamespaces against a catalog that barely changes. > > > Replicas would absorb that, but so would connector-side caching or a > > longer > > > refresh interval. > > > > > > I think this server-side dual-datasource approach is the right way to > > move > > > forward. WDYT? > > > > > > Regards, > > > Prithvi S > > > > > > On Sun, Sep 13, 2026 at 9:45 AM Yong Zheng <[email protected]> wrote: > > > > > > > Hello, > > > > > > > > When using JDBC as the backend metastore for Polaris, the JDBC > > connection > > > > capacity of the active primary can become a limiting factor for the > > > > throughput of a given Polaris deployment. Taking RDS with PostgreSQL > > as an > > > > example, if we are using an AWS db.m9g.4xlarge (16 cores and 64 GB) > as > > the > > > > backend store, we can get up to 5K connections by default. AWS uses > > > > "LEAST({DBInstanceClassMemory/9531392}, 5000)" for the default > > PostgreSQL > > > > "max_connections", and this can be increased manually. > > > > > > > > Now, assuming we put 50 connections per pod, we can have up to 100 > pods > > > > max (in reality, it will be less as a couple of connections are > > reserved > > > > for the superuser, but let's stick with 100 pods to make the math > > simpler). > > > > > > > > With 50 connections per pod, this matches what Anand reported in > > > > > > > https://medium.com/@obelix74/a-30-million-request-load-test-for-apache-polaris-fb5690040154 > > . > > > > If we assume linear scaling from the benchmark (which may not hold > due > > to > > > > database CPU, I/O, locking, and other bottlenecks), the theoretical > > > > throughput would be ~5,333 requests per second. This is great, but if > > we > > > > ever want to handle more throughput, we would have no other option > > other > > > > than scaling up the RDS instance and overriding the default max > allowed > > > > connections on the DB server when using the native AWS solution. > There > > are > > > > other solutions out there, such as using a multi-master deployment > for > > the > > > > backend DB or switching to a NoSQL backend, which we can scale up a > lot > > > > easier (e.g. MongoDB). > > > > > > > > While there are solutions to scale up the infrastructure to support > > more > > > > connections, one particular thing that caught my eye is that we are > not > > > > using read-only replicas at all. > > > > > > > > Assuming we have two primary workloads: > > > > > > > > 1. Ingestion layer: this has both read and write. > > > > 2. Query/Serving layer: this is read-only queries to power various > > > > dashboards. > > > > > > > > We could also have two DB connection endpoints: > > > > > > > > 1. RW connection endpoint (active primary) > > > > 2. RO connection endpoint (read replicas) > > > > > > > > By default, everything should go to the RW endpoint. This matches the > > > > current workflow we support. However, if we know a query/serving > layer > > is > > > > read-only, we should route those requests to the RO connection > > endpoint; > > > > the auth token request/renewal would still be fulfilled through the > RW > > > > connection endpoint for the query/serving layer. > > > > > > > > Now, to decide where a client should be sending requests to RW/RO, we > > can > > > > check the following: > > > > > > > > 1. Is this an auth request? If yes, always use the RW endpoint. > > > > 2. Does this request have a special header (e.g. "Polaris-Readonly", > > which > > > > defaults to false or unset)? If yes, offload the request to the RO > > endpoint. > > > > > > > > In this case, if a non-read-only request is ever sent by the client > by > > > > accident, it will be failed by the backend DB because writes are not > > > > supported on RO replicas, and this is expected behavior. We should > not > > > > silently fall back to the RW endpoint in this case. > > > > > > > > With this approach, we can squeeze higher requests per second out of > a > > > > given setup by offloading the read-heavy query/serving workload from > > the > > > > primary. If the load from the query/serving layer is significant, > this > > > > could give us another dimension to scale the infrastructure without > > having > > > > to keep scaling up the primary DB. > > > > > > > > Thanks, > > > > Yong Zheng > > > > > > > > > >
