Hi Hongzhi, Thanks for writing this up — +1 on the direction. Exact-shape merging leaving GPU utilization on the table matches what I'd expect, and I like that the operations surface stays at three keys with profiles doing the heavy lifting. Some thoughts on your three discussion questions, plus a couple of things I'm curious about:
On observability: I lean toward wiring these stats into the existing metrics framework (the Prometheus-facing service) rather than pool-level logs — mainly because tuning a profile usually means watching how queue depth and batch shape respond over time, which is awkward to do from log lines. If that sounds reasonable, two stats I'd find particularly useful alongside QPS/latency/batch-size are padding waste (padded vs. useful tokens) and a reject or deadline-miss count. Curious whether you already track something like padding waste internally. One thing I didn't see in the write-up and would love to hear your plan on: what happens under sustained overload? Is the waiting queue bounded, and if it fills up, does the caller block or get rejected? I recently worked on a batch-write path elsewhere in the ecosystem (bounded queue + size/linger flush) and we went back and forth on this — we ended up preferring a bounded queue with fast rejection over blocking, since an unbounded queue tends to turn overload into unbounded tail latency that's hard to see. Not sure how well that maps to inference workloads though, where a dropped request may be more costly than a late one. What's your current behavior there? On padding, two questions. Are the length buckets fixed or do they adapt to the observed distribution? I'm wondering about the case where input-length variance is high — would the throughput profile keep merging even when most of the batch is padding, or is there some cutoff? And on correctness: do you plan to include a test comparing padded-batch output against single-request output on the same inputs? For some models padding + masking subtleties can shift results numerically, so it would be reassuring to see "semantics stay the same" pinned by a test. On the profiles question: I think latency / balanced / throughput covers the main cases and I wouldn't add finer knobs now — keeping legacy keys as the escape hatch seems right. It would help though to have the profile → derived-parameter mapping (window, deadline, bucket granularity) written down in the user docs, so when someone does need to debug the auto-derivation they can see what it decided. Overall this looks like a solid improvement — looking forward to the PR. Best, Xuan Wang Hongzhi Gao <[email protected]> 于2026年7月6日周一 23:43写道： > Hi all, > > I have implemented a dynamic batching design for AINode inference > locally and would like to discuss the direction before opening a PR. > > Background > > AINode inference today uses FIFO scheduling plus exact-shape batching: > only requests with identical target_count, input_length, and > output_length can be merged. Under concurrency, similar requests often > cannot be batched together, so GPU utilization stays low. There is also no > waiting-window or deadline mechanism, making it hard to trade off > throughput vs. latency by scenario. > > Architecture comparison > > Before (Basic): Requests enter a waiting queue and are dequeued FIFO; > only requests with identical shapes can be merged into a batch before model > inference. > > After (Dynamic): Requests still flow through waiting queue → running > queue, but scheduling adds similar-length grouping plus > waiting-window/deadline control; batches are formed with padding alignment. > Inference also emits QPS, latency, and batch-size stats. > > CALL INFERENCE and FORECAST semantics stay the same; only the > internal scheduling and batching path inside the inference pool changes. > > Core mechanisms: > > Similar-length grouping — Round input/output lengths up to buckets so > nearby shapes can batch together. > > Waiting window + deadline — Avoid dispatching tiny batches too early > and starving requests; dispatch when the batch is full. > > Dynamic padding batching — Pad to the max length in a group, run one > inference, split results back per request. > > Inference stats — Each pool periodically logs QPS, latency > percentiles, and average batch size. > > Operations configuration > > 1. What operators need to care about > > Defaults are fine — No inference settings required; use > balanced plus hardware auto-tuning. > > If tuning is needed, only 3 keys in iotdb-ainode.properties: > Config keyMeaning > ain_inference_memory_usage_ratioShare of memory for inference (auto ~55% > GPU, ~25% CPU) > ain_inference_scheduling_profilelatency (low latency) / > balanced / throughput (high throughput) > ain_inference_max_batch_sizeMax requests per batch (optional; > auto-estimated from VRAM) > > Typical values: 16 GB GPU → 0.55 + balanced; 24 GB GPU, heavy > concurrency → 0.65 + throughput; CPU-only → 0.25 + latency. > > 2. How it works under the hood > > Previously only memory_usage_ratio and extra_memory_ratio were > configurable; how long to wait and how to form batches was hard-coded. > Dynamic batching needs many internal parameters — exposing all of them to > operators is impractical. > > Approach: detect device and memory at startup → apply the three common > settings (or defaults) → auto-derive the rest. scheduling_profile maps > internally to waiting window, deadline, etc., so operators do not set > millisecond values by hand. Startup logs one summary line (total memory, > inference budget, suggested pool count, batch cap) for verification. Legacy > keys remain supported for advanced overrides. > > Questions for discussion > > Scheduling — Are latency / balanced / > throughput enough for main use cases? Do we need finer-grained control? > > Operations — Is “3 common settings + auto-derivation” sufficient? Are > the default ratios reasonable? > > Observability — Is pool-level logging enough, or should we expose a > metrics API? > > I have a working branch and local tests; feedback welcome. I will open a > PR once we align on direction. > > Thanks! > > Hongzhi Gao > > > > > > Hongzhi Gao > [email protected]
