Hi all,
Thanks to Xuan for the detailed feedback. Below is a point-by-point response,
plus some notes on design direction.
(Note: the design has been validated on a local branch; the PR is not open yet.
“Planned for this PR” below is the intended scope — details will follow the PR
description.)
1. Observability: logs vs. metrics
Xuan suggested wiring stats into the existing Prometheus metrics framework
rather than relying only on pool-level logs, with particular interest in
padding waste and reject / deadline-miss style counters. I agree with
that direction.
Current design (planned): each inference pool periodically logs QPS,
latency percentiles, and average batch size.
Not covered yet; planned for this PR:
padding waste (useful tokens vs. padding)
queue depth
admission reject count (queue full, fast failure)
deadline trigger count (meaning “forced dispatch”; see §2)
My inclination: ship enhanced log stats first to validate batching
behavior; Prometheus integration as a near-term follow-up (AINode has no
metrics plumbing today — doing it separately keeps the dynamic-batching PR
focused and easier to review).
Question for the list: is enhanced log stats alone acceptable for the
first PR, or should metrics be a merge prerequisite?
2. Sustained overload: bounded queue? block or reject?
This was missing from my original mail — thanks Xuan for raising it.
Before (current mainline): per-pool waiting queue is unbounded; callers
block until results return; deadline only forces dispatch (no
rejection). Under sustained overload, queues grow and tail latency increases.
Planned for this PR (vLLM-style): vLLM caps the waiting queue with
max_queue_size and fails fast when full (e.g. HTTP 503) instead of
letting requests pile up in an unbounded queue. AINode uses Thrift RPC; we’ll
follow the same idea — when the queue is full, return a clear overload
error to the caller so upstream can retry or degrade:
Bounded admission queue — cap waiting depth; avoid unbounded tail latency
under overload
Fast failure when full — no indefinite blocking
deadline retained — for admitted requests, force dispatch after max
wait to avoid starvation
3. Padding: bucket strategy and high length variance
Current design (planned):
Round input/output lengths up to fixed buckets
Default buckets: input=32, output=16 (overridable via legacy keys)
No distribution-adaptive buckets; no padding-ratio cutoff
throughput profile mainly widens waiting window / deadline to improve
batching
Why these defaults:
32 / 16 match common time-series length granularity (many patch/stride
steps are multiples of 16) — a trade-off between batching efficiency and
padding waste
Buckets too small → fragmented groups, similar lengths don’t batch, GPU
utilization stays low
Buckets too large → more padding when variance is high, less effective compute
If defaults aren’t right, how to adjust:
Ops: override ain_inference_input_length_bucket_size /
ain_inference_output_length_bucket_size
Product: use padding-waste stats from this PR; if waste is too high,
consider scheduler cutoffs or per-model/scenario defaults later
With very high length variance, padding-heavy batches are a known trade-off —
quantify with stats first, then decide on cutoffs.
4. Correctness: padded batch vs. single-request output
I agree we need to pin semantics, especially since padding/masking can differ
across models.
Planned for this PR:
Python unit tests: same inputs, single-request path vs. dynamic padded
batch, torch.allclose
Integration tests: cover forecast-style built-in models that use the
inference pool, ensuring padded-batch output matches single-request output
5. Profiles and documentation
I agree three profiles are enough for now; no finer operator knobs; legacy keys
remain for advanced overrides.
User docs will document profile → derived parameters. Planned mapping (all
three profiles share default buckets input=32 / output=16):
ProfilePoll intervalWaiting windowDeadline
latency5ms5ms200ms
balanced15ms15ms500ms
throughput30ms40ms1500ms
Parameter meanings:
Poll interval (batch_interval_ms): how often the scheduler polls waiting /
running queues to check whether batches can be dispatched
Waiting window (batch_waiting_window_ms): minimum wait before dispatching
a partial batch — hold briefly so more similar requests can arrive and improve
batching
Deadline (batch_deadline_ms): max time a request may wait in queue; after
that, force dispatch (even as a small batch) to avoid starvation
6. Scope for this PR (draft)
Planned for this PR:
Dynamic batching (similar-length grouping + window/deadline + padding)
Three operator knobs + hardware auto-tuning + profile mapping
Bounded admission queue + fast failure when full (vLLM-style max_queue_size)
Enhanced pool log stats (padding waste, queue depth, admission reject, deadline
triggers)
Correctness unit tests + IT for forecast-style built-in models
Profile documentation
Planned follow-up (later PRs):
Prometheus / metrics integration
Padding cutoff / adaptive buckets (based on stats)
Feedback welcome — especially on whether metrics should block merging the first
PR. I’ll open the PR once we’re aligned on direction.
Thanks!
Hongzhi Gao
原始邮件
发件人：王旋 [email protected]
发件时间：2026年7月7日 11:23
收件人：dev [email protected]
主题：Re: [Design Discussion] AINode Inference Engine: Dynamic Batching &
Simplified Operations Config
Hi Hongzhi,
Thanks for writing this up — +1 on the direction. Exact-shape merging
leaving GPU utilization on the table matches what I’d expect, and I like
that the operations surface stays at three keys with profiles doing the
heavy lifting. Some thoughts on your three discussion questions, plus a
couple of things I’m curious about:
On observability: I lean toward wiring these stats into the existing
metrics framework (the Prometheus-facing service) rather than pool-level
logs — mainly because tuning a profile usually means watching how queue
depth and batch shape respond over time, which is awkward to do from log
lines. If that sounds reasonable, two stats I’d find particularly useful
alongside QPS/latency/batch-size are padding waste (padded vs. useful
tokens) and a reject or deadline-miss count. Curious whether you already
track something like padding waste internally.
One thing I didn’t see in the write-up and would love to hear your plan on:
what happens under sustained overload? Is the waiting queue bounded, and if
it fills up, does the caller block or get rejected? I recently worked on a
batch-write path elsewhere in the ecosystem (bounded queue + size/linger
flush) and we went back and forth on this — we ended up preferring a
bounded queue with fast rejection over blocking, since an unbounded queue
tends to turn overload into unbounded tail latency that’s hard to see. Not
sure how well that maps to inference workloads though, where a dropped
request may be more costly than a late one. What’s your current behavior
there?
On padding, two questions. Are the length buckets fixed or do they adapt to
the observed distribution? I’m wondering about the case where input-length
variance is high — would the throughput profile keep merging even when most
of the batch is padding, or is there some cutoff? And on correctness: do
you plan to include a test comparing padded-batch output against
single-request output on the same inputs? For some models padding + masking
subtleties can shift results numerically, so it would be reassuring to see
“semantics stay the same” pinned by a test.
On the profiles question: I think latency / balanced / throughput covers
the main cases and I wouldn’t add finer knobs now — keeping legacy keys as
the escape hatch seems right. It would help though to have the profile →
derived-parameter mapping (window, deadline, bucket granularity) written
down in the user docs, so when someone does need to debug the
auto-derivation they can see what it decided.
Overall this looks like a solid improvement — looking forward to the PR.
Best,
Xuan Wang
Hongzhi Gao [email protected] 于2026年7月6日周一 23:43写道：
Hi all,
I have implemented a dynamic batching design for AINode inference
locally and would like to discuss the direction before opening a PR.
Background
AINode inference today uses FIFO scheduling plus exact-shape batching:
only requests with identical target_count, input_length, and
output_length can be merged. Under concurrency, similar requests often
cannot be batched together, so GPU utilization stays low. There is also no
waiting-window or deadline mechanism, making it hard to trade off
throughput vs. latency by scenario.
Architecture comparison
Before (Basic): Requests enter a waiting queue and are dequeued FIFO;
only requests with identical shapes can be merged into a batch before model
inference.
After (Dynamic): Requests still flow through waiting queue → running
queue, but scheduling adds similar-length grouping plus
waiting-window/deadline control; batches are formed with padding alignment.
Inference also emits QPS, latency, and batch-size stats.
CALL INFERENCE and FORECAST semantics stay the same; only the
internal scheduling and batching path inside the inference pool changes.
Core mechanisms:
Similar-length grouping — Round input/output lengths up to buckets so
nearby shapes can batch together.
Waiting window + deadline — Avoid dispatching tiny batches too early
and starving requests; dispatch when the batch is full.
Dynamic padding batching — Pad to the max length in a group, run one
inference, split results back per request.
Inference stats — Each pool periodically logs QPS, latency
percentiles, and average batch size.
Operations configuration
What operators need to care about
Defaults are fine — No inference settings required; use
balanced plus hardware auto-tuning.
If tuning is needed, only 3 keys in iotdb-ainode.properties:
Config keyMeaning
ain_inference_memory_usage_ratioShare of memory for inference (auto ~55%
GPU, ~25% CPU)
ain_inference_scheduling_profilelatency (low latency) /
balanced / throughput (high throughput)
ain_inference_max_batch_sizeMax requests per batch (optional;
auto-estimated from VRAM)
Typical values: 16 GB GPU → 0.55 + balanced; 24 GB GPU, heavy
concurrency → 0.65 + throughput; CPU-only → 0.25 + latency.
How it works under the hood
Previously only memory_usage_ratio and extra_memory_ratio were
configurable; how long to wait and how to form batches was hard-coded.
Dynamic batching needs many internal parameters — exposing all of them to
operators is impractical.
Approach: detect device and memory at startup → apply the three common
settings (or defaults) → auto-derive the rest. scheduling_profile maps
internally to waiting window, deadline, etc., so operators do not set
millisecond values by hand. Startup logs one summary line (total memory,
inference budget, suggested pool count, batch cap) for verification. Legacy
keys remain supported for advanced overrides.
Questions for discussion
Scheduling — Are latency / balanced /
throughput enough for main use cases? Do we need finer-grained control?
Operations — Is “3 common settings + auto-derivation” sufficient? Are
the default ratios reasonable?
Observability — Is pool-level logging enough, or should we expose a
metrics API?
I have a working branch and local tests; feedback welcome. I will open a
PR once we align on direction.
Thanks!
Hongzhi Gao
Hongzhi Gao
[email protected]
