Hello,
When using JDBC as the backend metastore for Polaris, the JDBC connection
capacity of the active primary can become a limiting factor for the throughput
of a given Polaris deployment. Taking RDS with PostgreSQL as an example, if we
are using an AWS db.m9g.4xlarge (16 cores and 64 GB) as the backend store, we
can get up to 5K connections by default. AWS uses
"LEAST({DBInstanceClassMemory/9531392}, 5000)" for the default PostgreSQL
"max_connections", and this can be increased manually.
Now, assuming we put 50 connections per pod, we can have up to 100 pods max (in
reality, it will be less as a couple of connections are reserved for the
superuser, but let's stick with 100 pods to make the math simpler).
With 50 connections per pod, this matches what Anand reported in
https://medium.com/@obelix74/a-30-million-request-load-test-for-apache-polaris-fb5690040154.
If we assume linear scaling from the benchmark (which may not hold due to
database CPU, I/O, locking, and other bottlenecks), the theoretical throughput
would be ~5,333 requests per second. This is great, but if we ever want to
handle more throughput, we would have no other option other than scaling up the
RDS instance and overriding the default max allowed connections on the DB
server when using the native AWS solution. There are other solutions out there,
such as using a multi-master deployment for the backend DB or switching to a
NoSQL backend, which we can scale up a lot easier (e.g. MongoDB).
While there are solutions to scale up the infrastructure to support more
connections, one particular thing that caught my eye is that we are not using
read-only replicas at all.
Assuming we have two primary workloads:
1. Ingestion layer: this has both read and write.
2. Query/Serving layer: this is read-only queries to power various dashboards.
We could also have two DB connection endpoints:
1. RW connection endpoint (active primary)
2. RO connection endpoint (read replicas)
By default, everything should go to the RW endpoint. This matches the current
workflow we support. However, if we know a query/serving layer is read-only, we
should route those requests to the RO connection endpoint; the auth token
request/renewal would still be fulfilled through the RW connection endpoint for
the query/serving layer.
Now, to decide where a client should be sending requests to RW/RO, we can check
the following:
1. Is this an auth request? If yes, always use the RW endpoint.
2. Does this request have a special header (e.g. "Polaris-Readonly", which
defaults to false or unset)? If yes, offload the request to the RO endpoint.
In this case, if a non-read-only request is ever sent by the client by
accident, it will be failed by the backend DB because writes are not supported
on RO replicas, and this is expected behavior. We should not silently fall back
to the RW endpoint in this case.
With this approach, we can squeeze higher requests per second out of a given
setup by offloading the read-heavy query/serving workload from the primary. If
the load from the query/serving layer is significant, this could give us
another dimension to scale the infrastructure without having to keep scaling up
the primary DB.
Thanks,
Yong Zheng
