aamir306 opened a new pull request, #16332:
URL: https://github.com/apache/iceberg/pull/16332
## Summary
Replaces the unbounded single-thread `ASYNC_PLANNING_POOL` in
`CatalogHandlers` with a bounded, configurable thread pool. This prevents
thread exhaustion under high concurrency and allows operators to tune the pool
size for their workloads.
## Motivation
The current `ASYNC_PLANNING_POOL` is a single-threaded, unbounded-queue
executor:
```java
private static final ExecutorService ASYNC_PLANNING_POOL =
ThreadPools.newExitingWorkerPool("iceberg-async-planning", 1);
```
**Problems:**
1. **Head-of-line blocking** – All async planning requests serialize through
a single thread, creating a bottleneck for concurrent REST catalog clients.
2. **Unbounded queue growth** – Under load, the unbounded queue can grow
without limit, consuming memory and increasing latency for queued requests.
3. **No tuning knob** – Operators have no way to adjust the pool size for
their deployment's characteristics.
## Changes
- Added `ICEBERG_REST_ASYNC_PLANNING_POOL_SIZE` system config (default: 4,
minimum: 1)
- Created bounded thread pool with `ArrayBlockingQueue(256)` to prevent
runaway queue growth
- Uses `CallerRunsPolicy` for graceful degradation when the queue is full
- Preserved thread naming for observability
## Configuration
```bash
# Set via system property
-Diceberg.rest.async-planning.pool-size=8
# Or environment variable
ICEBERG_REST_ASYNC_PLANNING_POOL_SIZE=8
```
## Test plan
- Added `TestAsyncPlanningPoolConfig` with tests for:
- Default value (4)
- Custom configuration
- Minimum bound enforcement
- Verified existing REST catalog tests pass
## Compatibility
- **Backward compatible** – Default of 4 threads is a strict improvement
over the previous single thread
- **No API changes** – Only internal implementation detail
Made with [Cursor](https://cursor.com)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
