Hi all, I have implemented a dynamic batching design for AINode inference locally and would like to discuss the direction before opening a PR.
Background AINode inference today uses FIFO scheduling plus exact-shape batching: only requests with identical target_count, input_length, and output_length can be merged. Under concurrency, similar requests often cannot be batched together, so GPU utilization stays low. There is also no waiting-window or deadline mechanism, making it hard to trade off throughput vs. latency by scenario. Architecture comparison Before (Basic): Requests enter a waiting queue and are dequeued FIFO; only requests with identical shapes can be merged into a batch before model inference. After (Dynamic): Requests still flow through waiting queue → running queue, but scheduling adds similar-length grouping plus waiting-window/deadline control; batches are formed with padding alignment. Inference also emits QPS, latency, and batch-size stats. CALL INFERENCE and FORECAST semantics stay the same; only the internal scheduling and batching path inside the inference pool changes. Core mechanisms: Similar-length grouping — Round input/output lengths up to buckets so nearby shapes can batch together. Waiting window + deadline — Avoid dispatching tiny batches too early and starving requests; dispatch when the batch is full. Dynamic padding batching — Pad to the max length in a group, run one inference, split results back per request. Inference stats — Each pool periodically logs QPS, latency percentiles, and average batch size. Operations configuration 1. What operators need to care about Defaults are fine — No inference settings required; use balanced plus hardware auto-tuning. If tuning is needed, only 3 keys in iotdb-ainode.properties: Config keyMeaning ain_inference_memory_usage_ratioShare of memory for inference (auto ~55% GPU, ~25% CPU) ain_inference_scheduling_profilelatency (low latency) / balanced / throughput (high throughput) ain_inference_max_batch_sizeMax requests per batch (optional; auto-estimated from VRAM) Typical values: 16 GB GPU → 0.55 + balanced; 24 GB GPU, heavy concurrency → 0.65 + throughput; CPU-only → 0.25 + latency. 2. How it works under the hood Previously only memory_usage_ratio and extra_memory_ratio were configurable; how long to wait and how to form batches was hard-coded. Dynamic batching needs many internal parameters — exposing all of them to operators is impractical. Approach: detect device and memory at startup → apply the three common settings (or defaults) → auto-derive the rest. scheduling_profile maps internally to waiting window, deadline, etc., so operators do not set millisecond values by hand. Startup logs one summary line (total memory, inference budget, suggested pool count, batch cap) for verification. Legacy keys remain supported for advanced overrides. Questions for discussion Scheduling — Are latency / balanced / throughput enough for main use cases? Do we need finer-grained control? Operations — Is “3 common settings + auto-derivation” sufficient? Are the default ratios reasonable? Observability — Is pool-level logging enough, or should we expose a metrics API? I have a working branch and local tests; feedback welcome. I will open a PR once we align on direction. Thanks! Hongzhi Gao Hongzhi Gao [email protected]
