This is an automated email from the ASF dual-hosted git repository. shuwenwei pushed a commit to branch supports_subscription in repository https://gitbox.apache.org/repos/asf/iotdb-client-go.git
commit fbadffdb33122245ac7f0ad5f6a2f2af0a123244 Author: shuwenwei <[email protected]> AuthorDate: Fri Nov 13 12:09:22 2026 +0800 Add built-in subscription processors --- subscription/README.md | 43 ++++- subscription/column_align_processor.go | 262 +++++++++++++++++++++++++++++ subscription/doc.go | 5 +- subscription/watermark_processor.go | 297 +++++++++++++++++++++++++++++++++ 4 files changed, 603 insertions(+), 4 deletions(-) diff --git a/subscription/README.md b/subscription/README.md index aa49988..7cebe42 100644 --- a/subscription/README.md +++ b/subscription/README.md @@ -19,7 +19,7 @@ under the License. # Go Subscription API -The module implements `SubscriptionSession` management, Pull consumption with record polling, commits, progress, seek, processor pipelines, and background connection coordination, and Push consumption with sequential callbacks and acknowledgment strategies. Poll response decoding, record decoding, and message release are also connected. Built-in processors remain for a subsequent step. These implementations have not been compiled or exercised at runtime; this document describes the code [...] +The module implements `SubscriptionSession` management, Pull consumption with record polling, commits, progress, seek, processor pipelines, and background connection coordination, and Push consumption with sequential callbacks and acknowledgment strategies. Poll response decoding, record decoding, and message release are also connected. Built-in Watermark and ColumnAlign processors are available for the Pull pipeline. These implementations have not been compiled or exercised at runtime; [...] The module path is `github.com/apache/iotdb-client-go/subscription`. It uses Go 1.25 and maintains dependencies and versions separately from the existing `github.com/apache/iotdb-client-go/v2` Session module. Its direct dependencies are `github.com/apache/iotdb-client-go/v2` at `v2.0.0-20260914074454-a25f68aa9681`, which provides existing types such as `TSDataType` and `ColumnCategory`, and `github.com/apache/thrift` at `v0.24.0`, which provides the RPC transport and protocol. It adds no [...] @@ -259,9 +259,48 @@ The `SubscriptionPushConsumer.Errors()` channel buffers one error. A TsFile deco | `BufferingProcessor.Buffered()`, `ProcessorBuffer` | `getBufferedCount/getBufferedCommitContexts` | An optional buffering interface reports both the count and all renewable contexts | | `TopicResetter.ResetTopic(topic)` | `supportsTopicScopedReset/reset(topic)` | Interface implementation replaces Java's capability flag | | `WatermarkReporter.Watermark()` | `WatermarkProcessor.getWatermark()` | Optionally exposes a processor's ordering watermark, distinct from the Consumer's latest server watermark | +| `NewWatermarkProcessor(config)`, `WatermarkProcessorConfig` | Both `WatermarkProcessor` constructors | One configuration structure; event-time distance retains server precision, while timeout fields use `time.Duration` | +| `WatermarkProcessor.Process/Flush/Buffered/Reset/ResetTopic/Watermark` | `process/flush/getBufferedCount/getBufferedCommitContexts/reset/getWatermark` | Buffers whole messages by maximum timestamp, renews every retained commit context, and resets by topic | +| `NewColumnAlignProcessor(dialect)` | Both `ColumnAlignProcessor` constructors and its nested `Dialect` | Reuses the module's `Dialect`; empty selects tree and table requires `TableDialect` | +| `ColumnAlignProcessor.Process/Flush/Reset/ResetTopic` | Methods with the same roles | Preserves delivery identity and returns independently owned record buffers with NULLs forward-filled | A `SubscriptionMessageProcessor` that retains messages must implement `BufferingProcessor` so heartbeats can renew undelivered records. Seek clears only its target topic. With multiple subscribed topics, a processor without `TopicResetter` makes seek fail before server positions change. A single-topic consumer can fall back to `Reset()`. Messages returned by `Flush` pass through subsequent stages; the processor must not retain or modify messages after transferring them. +## Built-in Pull Processors + +Create processors and install them with `AddProcessor` before the first Poll. A processor instance belongs to one consumer. Installation order determines pipeline order; putting watermark ordering before column alignment lets aligned records benefit from message reordering. + +```go +// This example assumes the server uses millisecond timestamps. +watermark, err := subscription.NewWatermarkProcessor(subscription.WatermarkProcessorConfig{ + MaxOutOfOrderness: 5000, + Timeout: 30 * time.Second, +}) +if err != nil { + return err +} +alignment, err := subscription.NewColumnAlignProcessor(subscription.TableDialect) +if err != nil { + return err +} +if err := consumer.AddProcessor(watermark); err != nil { + return err +} +if err := consumer.AddProcessor(alignment); err != nil { + return err +} +``` + +`WatermarkProcessor` tracks sources by topic, DataNode ID, and region ID, matching Java's region identity. Its watermark is the minimum latest timestamp among active sources minus `MaxOutOfOrderness`. Repeated or older timestamps do not refresh source activity. Zero `StaleRegionTimeout` selects 30 seconds, a negative value disables stale-source exclusion, and zero `MaxBufferBytes` selects 64 MiB. Negative event-time distance, timeout, or byte limit is rejected. `Timeout == 0` immediately [...] + +Messages are ordered by their maximum raw timestamp and emitted whole when that maximum is at most the watermark. This does not globally sort rows across messages or batches. Equal message maxima have no guaranteed order. Capacity overflow force-flushes all remaining messages after normal emission. A timeout force-flushes when no message was emitted and the interval since the previous emission has elapsed. When every source is stale, the previous watermark remains in effect. New sources [...] + +`Buffered()` snapshots the message count and every committable context for heartbeat renewal. `Flush()` transfers buffered records in maximum-timestamp order without clearing source progress. `Reset()` releases all retained records and clears ordering state; `ResetTopic(topic)` releases only that topic and recomputes ordering from the remaining sources. Reset discards unconfirmed records without acknowledging them. `Watermark()` returns `ok == false` before progress is known or when rese [...] + +`ColumnAlignProcessor` preserves raw row order, as Java does. It fills a NULL only from a cached non-null value with a strictly earlier timestamp; a non-null observation updates the cache at equal or greater timestamps. Columns are identified by name and data type, so reordered columns remain independent and changed types cannot reuse incompatible values. Tree state is isolated by topic and device path. Table state is isolated by topic, database, table name, and ordered TAG values per ro [...] + +Alignment reads inputs through the existing public RecordBatch API, copies record buffers, and returns replacement messages with the same commit contexts and projection. It releases consumed originals only after all input access succeeds. Input errors and dialect mismatches leave inputs and cache unchanged. Cache byte values and filled byte values are independently copied so earlier Release calls cannot invalidate later records. Watermarks pass through unchanged. It buffers no messages, [...] + ## Data Ownership and Context Contracts - Session query collections, Consumer identity, commit contexts, and progress are caller-owned snapshots; progress maps are deep snapshots. Seek copies its input rather than retaining the caller's maps. @@ -285,4 +324,4 @@ A `SubscriptionMessageProcessor` that retains messages must implement `Buffering Operation errors can wrap these sentinels; callers identify them with `errors.Is`. Unsupported TsFile errors must not trigger connection retries or file-payload acknowledgments. Pull returns an error and no messages for that round, including records obtained by parallel branches. That round must not advance positions or enter processor or automatic-commit buffers. Previously delivered records and confirmed progress are not rolled back. Callers can unsubscribe the offending topic and poll again. -The remaining implementation step is the built-in Watermark/ColumnAlign processors; further consumption examples and behavior tests are also pending. Management queries reuse the root module's decoder; this step does not change its handling of malformed data. Verification has been static only: no compilation, tests, or formatting tools have run, and no runtime validation result is claimed. +The built-in Watermark/ColumnAlign processors are implemented; further consumption examples, behavior tests, and runtime verification are pending. Management queries reuse the root module's decoder; this step does not change its handling of malformed data. Verification has been static only: no compilation, tests, or formatting tools have run, and no runtime validation result is claimed. diff --git a/subscription/column_align_processor.go b/subscription/column_align_processor.go new file mode 100644 index 0000000..4dd5630 --- /dev/null +++ b/subscription/column_align_processor.go @@ -0,0 +1,262 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package subscription + +import ( + "errors" + "fmt" + "strconv" + "strings" + "sync" + + "github.com/apache/iotdb-client-go/v2/client" +) + +// ColumnAlignProcessor forward-fills NULL columns with the latest known non-null +// value for the same topic, device, and column. It does not retain messages. +// A cached value fills a NULL only when its timestamp is strictly earlier; +// an observed non-null value updates the cache at equal or greater timestamps. +// Processing follows raw Tablet row order, matching Java, rather than Rows order. +// Tree devices use the device path; table devices use database, table, and TAG +// values for each row, including tables with no TAG columns. Use one instance +// per consumer. Cache size grows with observed devices and columns until reset. +type ColumnAlignProcessor struct { + mu sync.Mutex + dialect Dialect + lastValues map[string]map[columnAlignDevice]map[columnAlignColumn]columnAlignValue +} + +type columnAlignDevice struct { + database string + target string + tags string +} + +type columnAlignColumn struct { + name string + dataType client.TSDataType +} + +type columnAlignValue struct { + timestamp int64 + value any +} + +type columnAlignMessage struct { + message SubscriptionMessage + batches []columnAlignBatch +} + +type columnAlignBatch struct { + database string + timeSelected bool + tablet decodedSubscriptionTablet +} + +// NewColumnAlignProcessor creates a processor for one model. An empty dialect +// selects TreeDialect, matching Java's no-argument constructor. Table consumers +// must select TableDialect. Different databases never share cached table values. +func NewColumnAlignProcessor(dialect Dialect) (*ColumnAlignProcessor, error) { + if dialect == "" { + dialect = TreeDialect + } + if dialect != TreeDialect && dialect != TableDialect { + return nil, fmt.Errorf("subscription: invalid column alignment dialect %q", dialect) + } + return &ColumnAlignProcessor{ + dialect: dialect, + lastValues: make(map[string]map[columnAlignDevice]map[columnAlignColumn]columnAlignValue), + }, nil +} + +// Process returns record messages with aligned, independently owned buffers. +// It consumes and releases the original record messages only after every input +// has been read successfully. Kind, commit identity, schema, raw row order, and +// time projection are preserved. Watermarks pass through unchanged. Cached byte +// values are copied so releasing an earlier message cannot invalidate the cache. +// Access failures or model mismatches leave cache and input ownership unchanged. +func (p *ColumnAlignProcessor) Process(messages []SubscriptionMessage) ([]SubscriptionMessage, error) { + p.mu.Lock() + defer p.mu.Unlock() + incoming := make([]columnAlignMessage, 0, len(messages)) + for _, message := range messages { + if message == nil { + return nil, errors.New("subscription: column alignment received a nil message") + } + entry := columnAlignMessage{message: message} + if message.Kind() == WatermarkMessage { + incoming = append(incoming, entry) + continue + } + batches, err := message.Records() + if err != nil { + return nil, err + } + for _, batch := range batches { + if batch == nil { + return nil, errors.New("subscription: column alignment received a nil batch") + } + if batch.Dialect() != p.dialect { + return nil, fmt.Errorf("subscription: column alignment dialect %q does not match batch dialect %q", p.dialect, batch.Dialect()) + } + tablet := decodedSubscriptionTablet{ + Target: batch.Target(), + Columns: append([]Column(nil), batch.Columns()...), + RowCount: batch.Len(), + } + tablet.Timestamps = make([]int64, tablet.RowCount) + tablet.Values = make([][]any, len(tablet.Columns)) + for column := range tablet.Values { + tablet.Values[column] = make([]any, tablet.RowCount) + } + for row := 0; row < tablet.RowCount; row++ { + tablet.Timestamps[row], err = batch.Timestamp(row) + if err != nil { + return nil, err + } + for column := range tablet.Columns { + value, err := batch.Value(row, column) + if err != nil { + return nil, err + } + tablet.Values[column][row] = cloneColumnAlignValue(value) + } + } + entry.batches = append(entry.batches, columnAlignBatch{ + database: batch.Database(), + timeSelected: batch.TimeSelected(), + tablet: tablet, + }) + } + incoming = append(incoming, entry) + } + result := make([]SubscriptionMessage, 0, len(incoming)) + for _, entry := range incoming { + if entry.message.Kind() == WatermarkMessage { + result = append(result, entry.message) + continue + } + context := entry.message.CommitContext() + var batches []RecordBatch + for _, batch := range entry.batches { + p.fillTablet(context.Topic, batch.database, &batch.tablet) + batches = append(batches, newSubscriptionRecordBatch(p.dialect, batch.database, batch.tablet, batch.timeSelected)) + } + result = append(result, newSubscriptionRecordMessage(context, batches)) + } + for _, entry := range incoming { + if entry.message.Kind() != WatermarkMessage { + entry.message.Release() + } + } + return result, nil +} + +func (p *ColumnAlignProcessor) fillTablet(topic, database string, tablet *decodedSubscriptionTablet) { + topicCache := p.lastValues[topic] + if topicCache == nil { + topicCache = make(map[columnAlignDevice]map[columnAlignColumn]columnAlignValue) + p.lastValues[topic] = topicCache + } + for row := 0; row < tablet.RowCount; row++ { + timestamp := tablet.Timestamps[row] + device := columnAlignDevice{target: tablet.Target} + if p.dialect == TableDialect { + device.database = database + device.tags = columnAlignTags(tablet, row) + } + cache := topicCache[device] + if cache == nil { + cache = make(map[columnAlignColumn]columnAlignValue) + topicCache[device] = cache + } + for column, schema := range tablet.Columns { + // A schema type change cannot reuse an incompatible cached value. + key := columnAlignColumn{name: schema.Name, dataType: schema.DataType} + cached, ok := cache[key] + value := tablet.Values[column][row] + if value == nil { + if ok && cached.timestamp < timestamp { + tablet.Values[column][row] = cloneColumnAlignValue(cached.value) + } + } else if !ok || timestamp >= cached.timestamp { + cache[key] = columnAlignValue{timestamp: timestamp, value: cloneColumnAlignValue(value)} + } + } + } +} + +// columnAlignTags encodes ordered TAG segments without delimiter collisions. +// Trailing NULLs are ignored, matching TsFile's table device identity; a NULL +// segment remains distinct from an empty string and from embedded delimiters. +func columnAlignTags(tablet *decodedSubscriptionTablet, row int) string { + var tags []any + lastNonNull := 0 + for column, schema := range tablet.Columns { + if schema.Category == client.TAG { + value := tablet.Values[column][row] + tags = append(tags, value) + if value != nil { + lastNonNull = len(tags) + } + } + } + var key strings.Builder + for _, value := range tags[:lastNonNull] { + if value == nil { + key.WriteString("-1:") + continue + } + segment := fmt.Sprint(value) + if binary, ok := value.([]byte); ok { + segment = string(binary) + } + key.WriteString(strconv.Itoa(len(segment))) + key.WriteByte(':') + key.WriteString(segment) + } + return key.String() +} + +func cloneColumnAlignValue(value any) any { + if binary, ok := value.([]byte); ok { + return append([]byte(nil), binary...) + } + return value +} + +// Flush returns no messages because alignment does not buffer deliveries. +func (p *ColumnAlignProcessor) Flush() ([]SubscriptionMessage, error) { + return nil, nil +} + +// Reset clears cached values for every topic. +func (p *ColumnAlignProcessor) Reset() { + p.mu.Lock() + defer p.mu.Unlock() + p.lastValues = make(map[string]map[columnAlignDevice]map[columnAlignColumn]columnAlignValue) +} + +// ResetTopic clears only the target topic's cached alignment values. +func (p *ColumnAlignProcessor) ResetTopic(topic string) { + p.mu.Lock() + defer p.mu.Unlock() + delete(p.lastValues, topic) +} diff --git a/subscription/doc.go b/subscription/doc.go index 9cb40cd..45e1458 100644 --- a/subscription/doc.go +++ b/subscription/doc.go @@ -27,8 +27,9 @@ // NewSubscriptionPullConsumer provides pull consumption, record continuation, // commit and seek coordination, background heartbeats, and provider recovery. // NewSubscriptionPushConsumer adds sequential callback delivery with before- or -// after-consumption acknowledgment and a terminal-error channel. Built-in -// processors remain a subsequent implementation step. +// after-consumption acknowledgment and a terminal-error channel. +// NewWatermarkProcessor and NewColumnAlignProcessor provide optional Pull +// pipeline stages for message reordering and per-device NULL forward filling. // // Record messages are the supported payload format. Topic management and // subscription do not reject TsFile topics; decoding a TsFile payload must return diff --git a/subscription/watermark_processor.go b/subscription/watermark_processor.go new file mode 100644 index 0000000..e528995 --- /dev/null +++ b/subscription/watermark_processor.go @@ -0,0 +1,297 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package subscription + +import ( + "container/heap" + "errors" + "math" + "sync" + "time" +) + +// WatermarkProcessorConfig controls client-side message reordering. Timestamp +// distances use server precision; wall-clock waits use time.Duration. +type WatermarkProcessorConfig struct { + // MaxOutOfOrderness is subtracted from the minimum active region timestamp. + // It must be nonnegative and uses the same unit as RecordBatch.Timestamp. + MaxOutOfOrderness int64 + // Timeout force-flushes buffered records when this long has passed without + // output. Zero flushes on every Process call that produces no ordered output. + Timeout time.Duration + // StaleRegionTimeout excludes a region whose timestamp has not increased + // for this long. Zero selects 30 seconds; a negative value disables exclusion. + StaleRegionTimeout time.Duration + // MaxBufferBytes limits estimated retained message bytes. Zero selects + // 64 MiB. Exceeding the limit after ordered emission flushes the remainder. + MaxBufferBytes int64 +} + +// WatermarkProcessor buffers whole record messages in maximum-timestamp order. +// Its watermark is the minimum latest timestamp across active topic/node/region +// sources, minus MaxOutOfOrderness. Only increasing timestamps refresh source +// activity; repeated server watermarks cannot keep a stalled source active. +// All-stale sources leave the watermark unchanged. Timeout and capacity flushing +// prevent indefinite buffering, but can emit records beyond the watermark. +// This is a reference processor; large out-of-order streams may require a +// downstream stream processing framework. Use one instance per consumer. +type WatermarkProcessor struct { + mu sync.Mutex + config WatermarkProcessorConfig + buffer watermarkMessageHeap + regions map[watermarkRegion]watermarkRegionProgress + bufferedBytes int64 + lastEmit time.Time + watermark int64 + hasWatermark bool +} + +type watermarkRegion struct { + topic string + node int32 + region string +} + +type watermarkRegionProgress struct { + timestamp int64 + advanced time.Time +} + +type watermarkBufferedMessage struct { + message SubscriptionMessage + maximum int64 + size int64 +} + +type watermarkMessageHeap []watermarkBufferedMessage + +func (h watermarkMessageHeap) Len() int { return len(h) } +func (h watermarkMessageHeap) Less(i, j int) bool { return h[i].maximum < h[j].maximum } +func (h watermarkMessageHeap) Swap(i, j int) { h[i], h[j] = h[j], h[i] } +func (h *watermarkMessageHeap) Push(value any) { + *h = append(*h, value.(watermarkBufferedMessage)) +} +func (h *watermarkMessageHeap) Pop() any { + last := len(*h) - 1 + value := (*h)[last] + (*h)[last] = watermarkBufferedMessage{} + *h = (*h)[:last] + return value +} + +// NewWatermarkProcessor validates configuration without taking message ownership. +// Timeout must be nonnegative. StaleRegionTimeout and MaxBufferBytes select their +// documented defaults independently. Emission is message-level, not a global +// row sort; Rows still sorts each batch independently. +func NewWatermarkProcessor(config WatermarkProcessorConfig) (*WatermarkProcessor, error) { + if config.MaxOutOfOrderness < 0 || config.Timeout < 0 || config.MaxBufferBytes < 0 { + return nil, errors.New("subscription: watermark distance, timeout, and buffer limit must be nonnegative") + } + if config.StaleRegionTimeout == 0 { + config.StaleRegionTimeout = 30 * time.Second + } + if config.MaxBufferBytes == 0 { + config.MaxBufferBytes = 64 * 1024 * 1024 + } + return &WatermarkProcessor{ + config: config, + regions: make(map[watermarkRegion]watermarkRegionProgress), + lastEmit: time.Now(), + }, nil +} + +// Process retains records until their maximum timestamp is at most the +// watermark, or a timeout/capacity flush occurs. Watermark events update source +// progress and are consumed. Empty input still evaluates staleness and timeout. +// An input-access error leaves processor state and input ownership unchanged. +func (p *WatermarkProcessor) Process(messages []SubscriptionMessage) ([]SubscriptionMessage, error) { + p.mu.Lock() + defer p.mu.Unlock() + now := time.Now() + incoming := make([]watermarkBufferedMessage, 0, len(messages)) + for _, message := range messages { + if message == nil { + return nil, errors.New("subscription: watermark processor received a nil message") + } + if timestamp, ok := message.Watermark(); ok { + incoming = append(incoming, watermarkBufferedMessage{message: message, maximum: timestamp}) + continue + } + batches, err := message.Records() + if err != nil { + return nil, err + } + maximum, found := int64(math.MinInt64), false + for _, batch := range batches { + if batch == nil { + return nil, errors.New("subscription: watermark processor received a nil batch") + } + for row := 0; row < batch.Len(); row++ { + timestamp, err := batch.Timestamp(row) + if err != nil { + return nil, err + } + if !found || timestamp > maximum { + maximum = timestamp + } + found = true + } + } + // As in Java, an empty message falls back to the wall clock in ms. + if !found { + maximum = now.UnixMilli() + } + size := message.EstimatedSize() + if size < 0 { + size = 0 + } + incoming = append(incoming, watermarkBufferedMessage{message: message, maximum: maximum, size: size}) + } + for _, entry := range incoming { + context := entry.message.CommitContext() + key := watermarkRegion{topic: context.Topic, node: context.DataNodeID, region: context.RegionID} + previous, ok := p.regions[key] + if !ok || entry.maximum > previous.timestamp { + p.regions[key] = watermarkRegionProgress{timestamp: entry.maximum, advanced: now} + } + if entry.message.Kind() == WatermarkMessage { + entry.message.Release() + continue + } + heap.Push(&p.buffer, entry) + p.bufferedBytes += entry.size + } + p.updateWatermark(now) + var emitted []SubscriptionMessage + for len(p.buffer) > 0 && p.hasWatermark && p.buffer[0].maximum <= p.watermark { + entry := heap.Pop(&p.buffer).(watermarkBufferedMessage) + p.bufferedBytes -= entry.size + emitted = append(emitted, entry.message) + } + if p.bufferedBytes > p.config.MaxBufferBytes || + (len(emitted) == 0 && len(p.buffer) > 0 && now.Sub(p.lastEmit) >= p.config.Timeout) { + emitted = append(emitted, p.flushLocked(now)...) + } + if len(emitted) > 0 { + p.lastEmit = now + } + return emitted, nil +} + +// Flush transfers every buffered record in maximum-timestamp order without +// resetting region progress or the current watermark. +func (p *WatermarkProcessor) Flush() ([]SubscriptionMessage, error) { + p.mu.Lock() + defer p.mu.Unlock() + return p.flushLocked(time.Now()), nil +} + +func (p *WatermarkProcessor) flushLocked(now time.Time) []SubscriptionMessage { + messages := make([]SubscriptionMessage, 0, len(p.buffer)) + for len(p.buffer) > 0 { + messages = append(messages, heap.Pop(&p.buffer).(watermarkBufferedMessage).message) + } + p.bufferedBytes = 0 + p.lastEmit = now + return messages +} + +// Buffered atomically snapshots the retained count and every renewable context. +func (p *WatermarkProcessor) Buffered() ProcessorBuffer { + p.mu.Lock() + defer p.mu.Unlock() + result := ProcessorBuffer{Count: len(p.buffer)} + for _, entry := range p.buffer { + context := entry.message.CommitContext() + if flowCommitContextCommittable(context) { + result.Contexts = append(result.Contexts, context) + } + } + return result +} + +// Watermark reports the current ordering watermark and whether one is known. +// It can move backward when new regions appear or after a topic reset, as in Java. +func (p *WatermarkProcessor) Watermark() (int64, bool) { + p.mu.Lock() + defer p.mu.Unlock() + return p.watermark, p.hasWatermark +} + +// Reset releases buffered records and clears all ordering progress. +func (p *WatermarkProcessor) Reset() { + p.mu.Lock() + defer p.mu.Unlock() + for _, entry := range p.buffer { + entry.message.Release() + } + p.buffer = nil + p.regions = make(map[watermarkRegion]watermarkRegionProgress) + p.bufferedBytes = 0 + p.lastEmit = time.Now() + p.watermark, p.hasWatermark = 0, false +} + +// ResetTopic releases only this topic's records and recomputes the watermark +// from the remaining active sources. Other topics keep their buffered records. +func (p *WatermarkProcessor) ResetTopic(topic string) { + p.mu.Lock() + defer p.mu.Unlock() + retained := p.buffer[:0] + for _, entry := range p.buffer { + if entry.message.CommitContext().Topic == topic { + p.bufferedBytes -= entry.size + entry.message.Release() + } else { + retained = append(retained, entry) + } + } + clear(p.buffer[len(retained):]) + p.buffer = retained + heap.Init(&p.buffer) + for key := range p.regions { + if key.topic == topic { + delete(p.regions, key) + } + } + p.watermark, p.hasWatermark = 0, false + p.updateWatermark(time.Now()) +} + +func (p *WatermarkProcessor) updateWatermark(now time.Time) { + var minimum int64 + found := false + for _, progress := range p.regions { + if p.config.StaleRegionTimeout >= 0 && now.Sub(progress.advanced) > p.config.StaleRegionTimeout { + continue + } + if !found || progress.timestamp < minimum { + minimum, found = progress.timestamp, true + } + } + if found { + // Saturate subtraction to preserve ordering at the timestamp boundary. + p.watermark = math.MinInt64 + if minimum >= math.MinInt64 + p.config.MaxOutOfOrderness { + p.watermark = minimum - p.config.MaxOutOfOrderness + } + p.hasWatermark = true + } +}
