Hi Hongzhi, Thanks for the thorough reply — really appreciate you going through each point and laying out the planned scope clearly.
No objection to merging without metrics wiring. The bounded-queue + fast-failure plan and the torch.allclose correctness tests both sound right to me. Best, Xuan Wang Hongzhi Gao <[email protected]> 于2026年7月7日周二 23:03写道： > Hi all, > > Thanks to Xuan for the detailed feedback. Below is a point-by-point > response, plus some notes on design direction. > > (Note: the design has been validated on a local branch; the PR is not open > yet. “Planned for this PR” below is the intended scope — details will > follow the PR description.) > > > 1. Observability: logs vs. metrics > > Xuan suggested wiring stats into the existing Prometheus metrics framework > rather than relying only on pool-level logs, with particular interest in > padding waste and reject / deadline-miss style counters. I agree > with that direction. > > Current design (planned): each inference pool periodically logs QPS, > latency percentiles, and average batch size. > > Not covered yet; planned for this PR: > > padding waste (useful tokens vs. padding) > > queue depth > > admission reject count (queue full, fast failure) > > deadline trigger count (meaning “forced dispatch”; see §2) > > My inclination: ship enhanced log stats first to validate batching > behavior; Prometheus integration as a near-term follow-up (AINode has > no metrics plumbing today — doing it separately keeps the dynamic-batching > PR focused and easier to review). > > Question for the list: is enhanced log stats alone acceptable for the > first PR, or should metrics be a merge prerequisite? > > > 2. Sustained overload: bounded queue? block or reject? > > This was missing from my original mail — thanks Xuan for raising it. > > Before (current mainline): per-pool waiting queue is unbounded; > callers block until results return; deadline only forces > dispatch (no rejection). Under sustained overload, queues grow and > tail latency increases. > > Planned for this PR (vLLM-style): vLLM caps the waiting queue with > max_queue_size and fails fast when full (e.g. HTTP 503) instead > of letting requests pile up in an unbounded queue. AINode uses Thrift RPC; > we’ll follow the same idea — when the queue is full, return a clear > overload error to the caller so upstream can retry or degrade: > > Bounded admission queue — cap waiting depth; avoid unbounded tail > latency under overload > > Fast failure when full — no indefinite blocking > > deadline retained — for admitted requests, force dispatch after > max wait to avoid starvation > > > 3. Padding: bucket strategy and high length variance > > Current design (planned): > > Round input/output lengths up to fixed buckets > > Default buckets: input=32, output=16 (overridable via legacy keys) > > No distribution-adaptive buckets; no padding-ratio cutoff > > throughput profile mainly widens waiting window / deadline to improve > batching > > Why these defaults: > > 32 / 16 match common time-series length granularity (many > patch/stride steps are multiples of 16) — a trade-off between batching > efficiency and padding waste > > Buckets too small → fragmented groups, similar lengths don’t batch, GPU > utilization stays low > > Buckets too large → more padding when variance is high, less effective > compute > > If defaults aren’t right, how to adjust: > > Ops: override ain_inference_input_length_bucket_size / > ain_inference_output_length_bucket_size > > Product: use padding-waste stats from this PR; if waste is too high, > consider scheduler cutoffs or per-model/scenario defaults later > > With very high length variance, padding-heavy batches are a known > trade-off — quantify with stats first, then decide on cutoffs. > > > 4. Correctness: padded batch vs. single-request output > > I agree we need to pin semantics, especially since padding/masking can > differ across models. > > Planned for this PR: > > Python unit tests: same inputs, single-request path vs. dynamic > padded batch, torch.allclose > > Integration tests: cover forecast-style built-in models that use the > inference pool, ensuring padded-batch output matches single-request output > > > 5. Profiles and documentation > > I agree three profiles are enough for now; no finer operator knobs; legacy > keys remain for advanced overrides. > > User docs will document profile → derived parameters. Planned mapping (all > three profiles share default buckets input=32 / output=16): > ProfilePoll intervalWaiting windowDeadline > latency5ms5ms200ms > balanced15ms15ms500ms > throughput30ms40ms1500ms > > Parameter meanings: > > Poll interval (batch_interval_ms): how often the scheduler polls > waiting / running queues to check whether batches can be dispatched > > Waiting window (batch_waiting_window_ms): minimum wait before > dispatching a partial batch — hold briefly so more similar requests can > arrive and improve batching > > Deadline (batch_deadline_ms): max time a request may wait in queue; > after that, force dispatch (even as a small batch) to avoid starvation > > > 6. Scope for this PR (draft) > > Planned for this PR: > > Dynamic batching (similar-length grouping + window/deadline + padding) > > Three operator knobs + hardware auto-tuning + profile mapping > > Bounded admission queue + fast failure when full (vLLM-style > max_queue_size) > > Enhanced pool log stats (padding waste, queue depth, admission reject, > deadline triggers) > > Correctness unit tests + IT for forecast-style built-in models > > Profile documentation > > Planned follow-up (later PRs): > > Prometheus / metrics integration > > Padding cutoff / adaptive buckets (based on stats) > > > Feedback welcome — especially on whether metrics should block merging the > first PR. I’ll open the PR once we’re aligned on direction. > > Thanks! > > Hongzhi Gao > > > > 原始邮件 > > 发件人：王旋 [email protected] > 发件时间：2026年7月7日 11:23 > 收件人：dev [email protected] > 主题：Re: [Design Discussion] AINode Inference Engine: Dynamic Batching & > Simplified Operations Config > Hi Hongzhi, > > Thanks for writing this up — +1 on the direction. Exact-shape merging > leaving GPU utilization on the table matches what I’d expect, and I like > that the operations surface stays at three keys with profiles doing the > heavy lifting. Some thoughts on your three discussion questions, plus a > couple of things I’m curious about: > > On observability: I lean toward wiring these stats into the existing > metrics framework (the Prometheus-facing service) rather than pool-level > logs — mainly because tuning a profile usually means watching how queue > depth and batch shape respond over time, which is awkward to do from log > lines. If that sounds reasonable, two stats I’d find particularly useful > alongside QPS/latency/batch-size are padding waste (padded vs. useful > tokens) and a reject or deadline-miss count. Curious whether you already > track something like padding waste internally. > > One thing I didn’t see in the write-up and would love to hear your plan on: > what happens under sustained overload? Is the waiting queue bounded, and if > it fills up, does the caller block or get rejected? I recently worked on a > batch-write path elsewhere in the ecosystem (bounded queue + size/linger > flush) and we went back and forth on this — we ended up preferring a > bounded queue with fast rejection over blocking, since an unbounded queue > tends to turn overload into unbounded tail latency that’s hard to see. Not > sure how well that maps to inference workloads though, where a dropped > request may be more costly than a late one. What’s your current behavior > there? > > On padding, two questions. Are the length buckets fixed or do they adapt to > the observed distribution? I’m wondering about the case where input-length > variance is high — would the throughput profile keep merging even when most > of the batch is padding, or is there some cutoff? And on correctness: do > you plan to include a test comparing padded-batch output against > single-request output on the same inputs? For some models padding + masking > subtleties can shift results numerically, so it would be reassuring to see > “semantics stay the same” pinned by a test. > > On the profiles question: I think latency / balanced / throughput covers > the main cases and I wouldn’t add finer knobs now — keeping legacy keys as > the escape hatch seems right. It would help though to have the profile → > derived-parameter mapping (window, deadline, bucket granularity) written > down in the user docs, so when someone does need to debug the > auto-derivation they can see what it decided. > > Overall this looks like a solid improvement — looking forward to the PR. > > Best, > Xuan Wang > > Hongzhi Gao [email protected] 于2026年7月6日周一 23:43写道： > > Hi all, > > I have implemented a dynamic batching design for AINode inference > locally and would like to discuss the direction before opening a PR. > > Background > > AINode inference today uses FIFO scheduling plus exact-shape batching: > only requests with identical target_count, input_length, and > output_length can be merged. Under concurrency, similar requests often > cannot be batched together, so GPU utilization stays low. There is also no > waiting-window or deadline mechanism, making it hard to trade off > throughput vs. latency by scenario. > > Architecture comparison > > Before (Basic): Requests enter a waiting queue and are dequeued FIFO; > only requests with identical shapes can be merged into a batch before model > inference. > > After (Dynamic): Requests still flow through waiting queue → running > queue, but scheduling adds similar-length grouping plus > waiting-window/deadline control; batches are formed with padding alignment. > Inference also emits QPS, latency, and batch-size stats. > > CALL INFERENCE and FORECAST semantics stay the same; only the > internal scheduling and batching path inside the inference pool changes. > > Core mechanisms: > > Similar-length grouping — Round input/output lengths up to buckets so > nearby shapes can batch together. > > Waiting window + deadline — Avoid dispatching tiny batches too early > and starving requests; dispatch when the batch is full. > > Dynamic padding batching — Pad to the max length in a group, run one > inference, split results back per request. > > Inference stats — Each pool periodically logs QPS, latency > percentiles, and average batch size. > > Operations configuration > > What operators need to care about > > Defaults are fine — No inference settings required; use > balanced plus hardware auto-tuning. > > If tuning is needed, only 3 keys in iotdb-ainode.properties: > Config keyMeaning > ain_inference_memory_usage_ratioShare of memory for inference (auto ~55% > GPU, ~25% CPU) > ain_inference_scheduling_profilelatency (low latency) / > balanced / throughput (high throughput) > ain_inference_max_batch_sizeMax requests per batch (optional; > auto-estimated from VRAM) > > Typical values: 16 GB GPU → 0.55 + balanced; 24 GB GPU, heavy > concurrency → 0.65 + throughput; CPU-only → 0.25 + latency. > > How it works under the hood > > Previously only memory_usage_ratio and extra_memory_ratio were > configurable; how long to wait and how to form batches was hard-coded. > Dynamic batching needs many internal parameters — exposing all of them to > operators is impractical. > > Approach: detect device and memory at startup → apply the three common > settings (or defaults) → auto-derive the rest. scheduling_profile maps > internally to waiting window, deadline, etc., so operators do not set > millisecond values by hand. Startup logs one summary line (total memory, > inference budget, suggested pool count, batch cap) for verification. Legacy > keys remain supported for advanced overrides. > > Questions for discussion > > Scheduling — Are latency / balanced / > throughput enough for main use cases? Do we need finer-grained > control? > > Operations — Is “3 common settings + auto-derivation” sufficient? Are > the default ratios reasonable? > > Observability — Is pool-level logging enough, or should we expose a > metrics API? > > I have a working branch and local tests; feedback welcome. I will open a > PR once we align on direction. > > Thanks! > > Hongzhi Gao > > > Hongzhi Gao > [email protected] >
