This is an automated email from the ASF dual-hosted git repository.

shuwenwei pushed a commit to branch supports_subscription
in repository https://gitbox.apache.org/repos/asf/iotdb-client-go.git

commit fbadffdb33122245ac7f0ad5f6a2f2af0a123244
Author: shuwenwei <[email protected]>
AuthorDate: Fri Nov 13 12:09:22 2026 +0800

    Add built-in subscription processors
---
 subscription/README.md                 |  43 ++++-
 subscription/column_align_processor.go | 262 +++++++++++++++++++++++++++++
 subscription/doc.go                    |   5 +-
 subscription/watermark_processor.go    | 297 +++++++++++++++++++++++++++++++++
 4 files changed, 603 insertions(+), 4 deletions(-)

diff --git a/subscription/README.md b/subscription/README.md
index aa49988..7cebe42 100644
--- a/subscription/README.md
+++ b/subscription/README.md
@@ -19,7 +19,7 @@ under the License.
 
 # Go Subscription API
 
-The module implements `SubscriptionSession` management, Pull consumption with 
record polling, commits, progress, seek, processor pipelines, and background 
connection coordination, and Push consumption with sequential callbacks and 
acknowledgment strategies. Poll response decoding, record decoding, and message 
release are also connected. Built-in processors remain for a subsequent step. 
These implementations have not been compiled or exercised at runtime; this 
document describes the code  [...]
+The module implements `SubscriptionSession` management, Pull consumption with 
record polling, commits, progress, seek, processor pipelines, and background 
connection coordination, and Push consumption with sequential callbacks and 
acknowledgment strategies. Poll response decoding, record decoding, and message 
release are also connected. Built-in Watermark and ColumnAlign processors are 
available for the Pull pipeline. These implementations have not been compiled 
or exercised at runtime;  [...]
 
 The module path is `github.com/apache/iotdb-client-go/subscription`. It uses 
Go 1.25 and maintains dependencies and versions separately from the existing 
`github.com/apache/iotdb-client-go/v2` Session module. Its direct dependencies 
are `github.com/apache/iotdb-client-go/v2` at 
`v2.0.0-20260914074454-a25f68aa9681`, which provides existing types such as 
`TSDataType` and `ColumnCategory`, and `github.com/apache/thrift` at `v0.24.0`, 
which provides the RPC transport and protocol. It adds no [...]
 
@@ -259,9 +259,48 @@ The `SubscriptionPushConsumer.Errors()` channel buffers 
one error. A TsFile deco
 | `BufferingProcessor.Buffered()`, `ProcessorBuffer` | 
`getBufferedCount/getBufferedCommitContexts` | An optional buffering interface 
reports both the count and all renewable contexts |
 | `TopicResetter.ResetTopic(topic)` | `supportsTopicScopedReset/reset(topic)` 
| Interface implementation replaces Java's capability flag |
 | `WatermarkReporter.Watermark()` | `WatermarkProcessor.getWatermark()` | 
Optionally exposes a processor's ordering watermark, distinct from the 
Consumer's latest server watermark |
+| `NewWatermarkProcessor(config)`, `WatermarkProcessorConfig` | Both 
`WatermarkProcessor` constructors | One configuration structure; event-time 
distance retains server precision, while timeout fields use `time.Duration` |
+| `WatermarkProcessor.Process/Flush/Buffered/Reset/ResetTopic/Watermark` | 
`process/flush/getBufferedCount/getBufferedCommitContexts/reset/getWatermark` | 
Buffers whole messages by maximum timestamp, renews every retained commit 
context, and resets by topic |
+| `NewColumnAlignProcessor(dialect)` | Both `ColumnAlignProcessor` 
constructors and its nested `Dialect` | Reuses the module's `Dialect`; empty 
selects tree and table requires `TableDialect` |
+| `ColumnAlignProcessor.Process/Flush/Reset/ResetTopic` | Methods with the 
same roles | Preserves delivery identity and returns independently owned record 
buffers with NULLs forward-filled |
 
 A `SubscriptionMessageProcessor` that retains messages must implement 
`BufferingProcessor` so heartbeats can renew undelivered records. Seek clears 
only its target topic. With multiple subscribed topics, a processor without 
`TopicResetter` makes seek fail before server positions change. A single-topic 
consumer can fall back to `Reset()`. Messages returned by `Flush` pass through 
subsequent stages; the processor must not retain or modify messages after 
transferring them.
 
+## Built-in Pull Processors
+
+Create processors and install them with `AddProcessor` before the first Poll. 
A processor instance belongs to one consumer. Installation order determines 
pipeline order; putting watermark ordering before column alignment lets aligned 
records benefit from message reordering.
+
+```go
+// This example assumes the server uses millisecond timestamps.
+watermark, err := 
subscription.NewWatermarkProcessor(subscription.WatermarkProcessorConfig{
+    MaxOutOfOrderness: 5000,
+    Timeout: 30 * time.Second,
+})
+if err != nil {
+    return err
+}
+alignment, err := 
subscription.NewColumnAlignProcessor(subscription.TableDialect)
+if err != nil {
+    return err
+}
+if err := consumer.AddProcessor(watermark); err != nil {
+    return err
+}
+if err := consumer.AddProcessor(alignment); err != nil {
+    return err
+}
+```
+
+`WatermarkProcessor` tracks sources by topic, DataNode ID, and region ID, 
matching Java's region identity. Its watermark is the minimum latest timestamp 
among active sources minus `MaxOutOfOrderness`. Repeated or older timestamps do 
not refresh source activity. Zero `StaleRegionTimeout` selects 30 seconds, a 
negative value disables stale-source exclusion, and zero `MaxBufferBytes` 
selects 64 MiB. Negative event-time distance, timeout, or byte limit is 
rejected. `Timeout == 0` immediately [...]
+
+Messages are ordered by their maximum raw timestamp and emitted whole when 
that maximum is at most the watermark. This does not globally sort rows across 
messages or batches. Equal message maxima have no guaranteed order. Capacity 
overflow force-flushes all remaining messages after normal emission. A timeout 
force-flushes when no message was emitted and the interval since the previous 
emission has elapsed. When every source is stale, the previous watermark 
remains in effect. New sources  [...]
+
+`Buffered()` snapshots the message count and every committable context for 
heartbeat renewal. `Flush()` transfers buffered records in maximum-timestamp 
order without clearing source progress. `Reset()` releases all retained records 
and clears ordering state; `ResetTopic(topic)` releases only that topic and 
recomputes ordering from the remaining sources. Reset discards unconfirmed 
records without acknowledging them. `Watermark()` returns `ok == false` before 
progress is known or when rese [...]
+
+`ColumnAlignProcessor` preserves raw row order, as Java does. It fills a NULL 
only from a cached non-null value with a strictly earlier timestamp; a non-null 
observation updates the cache at equal or greater timestamps. Columns are 
identified by name and data type, so reordered columns remain independent and 
changed types cannot reuse incompatible values. Tree state is isolated by topic 
and device path. Table state is isolated by topic, database, table name, and 
ordered TAG values per ro [...]
+
+Alignment reads inputs through the existing public RecordBatch API, copies 
record buffers, and returns replacement messages with the same commit contexts 
and projection. It releases consumed originals only after all input access 
succeeds. Input errors and dialect mismatches leave inputs and cache unchanged. 
Cache byte values and filled byte values are independently copied so earlier 
Release calls cannot invalidate later records. Watermarks pass through 
unchanged. It buffers no messages,  [...]
+
 ## Data Ownership and Context Contracts
 
 - Session query collections, Consumer identity, commit contexts, and progress 
are caller-owned snapshots; progress maps are deep snapshots. Seek copies its 
input rather than retaining the caller's maps.
@@ -285,4 +324,4 @@ A `SubscriptionMessageProcessor` that retains messages must 
implement `Buffering
 
 Operation errors can wrap these sentinels; callers identify them with 
`errors.Is`. Unsupported TsFile errors must not trigger connection retries or 
file-payload acknowledgments. Pull returns an error and no messages for that 
round, including records obtained by parallel branches. That round must not 
advance positions or enter processor or automatic-commit buffers. Previously 
delivered records and confirmed progress are not rolled back. Callers can 
unsubscribe the offending topic and poll again.
 
-The remaining implementation step is the built-in Watermark/ColumnAlign 
processors; further consumption examples and behavior tests are also pending. 
Management queries reuse the root module's decoder; this step does not change 
its handling of malformed data. Verification has been static only: no 
compilation, tests, or formatting tools have run, and no runtime validation 
result is claimed.
+The built-in Watermark/ColumnAlign processors are implemented; further 
consumption examples, behavior tests, and runtime verification are pending. 
Management queries reuse the root module's decoder; this step does not change 
its handling of malformed data. Verification has been static only: no 
compilation, tests, or formatting tools have run, and no runtime validation 
result is claimed.
diff --git a/subscription/column_align_processor.go 
b/subscription/column_align_processor.go
new file mode 100644
index 0000000..4dd5630
--- /dev/null
+++ b/subscription/column_align_processor.go
@@ -0,0 +1,262 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package subscription
+
+import (
+       "errors"
+       "fmt"
+       "strconv"
+       "strings"
+       "sync"
+
+       "github.com/apache/iotdb-client-go/v2/client"
+)
+
+// ColumnAlignProcessor forward-fills NULL columns with the latest known 
non-null
+// value for the same topic, device, and column. It does not retain messages.
+// A cached value fills a NULL only when its timestamp is strictly earlier;
+// an observed non-null value updates the cache at equal or greater timestamps.
+// Processing follows raw Tablet row order, matching Java, rather than Rows 
order.
+// Tree devices use the device path; table devices use database, table, and TAG
+// values for each row, including tables with no TAG columns. Use one instance
+// per consumer. Cache size grows with observed devices and columns until 
reset.
+type ColumnAlignProcessor struct {
+       mu sync.Mutex
+       dialect Dialect
+       lastValues 
map[string]map[columnAlignDevice]map[columnAlignColumn]columnAlignValue
+}
+
+type columnAlignDevice struct {
+       database string
+       target string
+       tags string
+}
+
+type columnAlignColumn struct {
+       name string
+       dataType client.TSDataType
+}
+
+type columnAlignValue struct {
+       timestamp int64
+       value any
+}
+
+type columnAlignMessage struct {
+       message SubscriptionMessage
+       batches []columnAlignBatch
+}
+
+type columnAlignBatch struct {
+       database string
+       timeSelected bool
+       tablet decodedSubscriptionTablet
+}
+
+// NewColumnAlignProcessor creates a processor for one model. An empty dialect
+// selects TreeDialect, matching Java's no-argument constructor. Table 
consumers
+// must select TableDialect. Different databases never share cached table 
values.
+func NewColumnAlignProcessor(dialect Dialect) (*ColumnAlignProcessor, error) {
+       if dialect == "" {
+               dialect = TreeDialect
+       }
+       if dialect != TreeDialect && dialect != TableDialect {
+               return nil, fmt.Errorf("subscription: invalid column alignment 
dialect %q", dialect)
+       }
+       return &ColumnAlignProcessor{
+               dialect: dialect,
+               lastValues: 
make(map[string]map[columnAlignDevice]map[columnAlignColumn]columnAlignValue),
+       }, nil
+}
+
+// Process returns record messages with aligned, independently owned buffers.
+// It consumes and releases the original record messages only after every input
+// has been read successfully. Kind, commit identity, schema, raw row order, 
and
+// time projection are preserved. Watermarks pass through unchanged. Cached 
byte
+// values are copied so releasing an earlier message cannot invalidate the 
cache.
+// Access failures or model mismatches leave cache and input ownership 
unchanged.
+func (p *ColumnAlignProcessor) Process(messages []SubscriptionMessage) 
([]SubscriptionMessage, error) {
+       p.mu.Lock()
+       defer p.mu.Unlock()
+       incoming := make([]columnAlignMessage, 0, len(messages))
+       for _, message := range messages {
+               if message == nil {
+                       return nil, errors.New("subscription: column alignment 
received a nil message")
+               }
+               entry := columnAlignMessage{message: message}
+               if message.Kind() == WatermarkMessage {
+                       incoming = append(incoming, entry)
+                       continue
+               }
+               batches, err := message.Records()
+               if err != nil {
+                       return nil, err
+               }
+               for _, batch := range batches {
+                       if batch == nil {
+                               return nil, errors.New("subscription: column 
alignment received a nil batch")
+                       }
+                       if batch.Dialect() != p.dialect {
+                               return nil, fmt.Errorf("subscription: column 
alignment dialect %q does not match batch dialect %q", p.dialect, 
batch.Dialect())
+                       }
+                       tablet := decodedSubscriptionTablet{
+                               Target: batch.Target(),
+                               Columns: append([]Column(nil), 
batch.Columns()...),
+                               RowCount: batch.Len(),
+                       }
+                       tablet.Timestamps = make([]int64, tablet.RowCount)
+                       tablet.Values = make([][]any, len(tablet.Columns))
+                       for column := range tablet.Values {
+                               tablet.Values[column] = make([]any, 
tablet.RowCount)
+                       }
+                       for row := 0; row < tablet.RowCount; row++ {
+                               tablet.Timestamps[row], err = 
batch.Timestamp(row)
+                               if err != nil {
+                                       return nil, err
+                               }
+                               for column := range tablet.Columns {
+                                       value, err := batch.Value(row, column)
+                                       if err != nil {
+                                               return nil, err
+                                       }
+                                       tablet.Values[column][row] = 
cloneColumnAlignValue(value)
+                               }
+                       }
+                       entry.batches = append(entry.batches, columnAlignBatch{
+                               database: batch.Database(),
+                               timeSelected: batch.TimeSelected(),
+                               tablet: tablet,
+                       })
+               }
+               incoming = append(incoming, entry)
+       }
+       result := make([]SubscriptionMessage, 0, len(incoming))
+       for _, entry := range incoming {
+               if entry.message.Kind() == WatermarkMessage {
+                       result = append(result, entry.message)
+                       continue
+               }
+               context := entry.message.CommitContext()
+               var batches []RecordBatch
+               for _, batch := range entry.batches {
+                       p.fillTablet(context.Topic, batch.database, 
&batch.tablet)
+                       batches = append(batches, 
newSubscriptionRecordBatch(p.dialect, batch.database, batch.tablet, 
batch.timeSelected))
+               }
+               result = append(result, newSubscriptionRecordMessage(context, 
batches))
+       }
+       for _, entry := range incoming {
+               if entry.message.Kind() != WatermarkMessage {
+                       entry.message.Release()
+               }
+       }
+       return result, nil
+}
+
+func (p *ColumnAlignProcessor) fillTablet(topic, database string, tablet 
*decodedSubscriptionTablet) {
+       topicCache := p.lastValues[topic]
+       if topicCache == nil {
+               topicCache = 
make(map[columnAlignDevice]map[columnAlignColumn]columnAlignValue)
+               p.lastValues[topic] = topicCache
+       }
+       for row := 0; row < tablet.RowCount; row++ {
+               timestamp := tablet.Timestamps[row]
+               device := columnAlignDevice{target: tablet.Target}
+               if p.dialect == TableDialect {
+                       device.database = database
+                       device.tags = columnAlignTags(tablet, row)
+               }
+               cache := topicCache[device]
+               if cache == nil {
+                       cache = make(map[columnAlignColumn]columnAlignValue)
+                       topicCache[device] = cache
+               }
+               for column, schema := range tablet.Columns {
+                       // A schema type change cannot reuse an incompatible 
cached value.
+                       key := columnAlignColumn{name: schema.Name, dataType: 
schema.DataType}
+                       cached, ok := cache[key]
+                       value := tablet.Values[column][row]
+                       if value == nil {
+                               if ok && cached.timestamp < timestamp {
+                                       tablet.Values[column][row] = 
cloneColumnAlignValue(cached.value)
+                               }
+                       } else if !ok || timestamp >= cached.timestamp {
+                               cache[key] = columnAlignValue{timestamp: 
timestamp, value: cloneColumnAlignValue(value)}
+                       }
+               }
+       }
+}
+
+// columnAlignTags encodes ordered TAG segments without delimiter collisions.
+// Trailing NULLs are ignored, matching TsFile's table device identity; a NULL
+// segment remains distinct from an empty string and from embedded delimiters.
+func columnAlignTags(tablet *decodedSubscriptionTablet, row int) string {
+       var tags []any
+       lastNonNull := 0
+       for column, schema := range tablet.Columns {
+               if schema.Category == client.TAG {
+                       value := tablet.Values[column][row]
+                       tags = append(tags, value)
+                       if value != nil {
+                               lastNonNull = len(tags)
+                       }
+               }
+       }
+       var key strings.Builder
+       for _, value := range tags[:lastNonNull] {
+               if value == nil {
+                       key.WriteString("-1:")
+                       continue
+               }
+               segment := fmt.Sprint(value)
+               if binary, ok := value.([]byte); ok {
+                       segment = string(binary)
+               }
+               key.WriteString(strconv.Itoa(len(segment)))
+               key.WriteByte(':')
+               key.WriteString(segment)
+       }
+       return key.String()
+}
+
+func cloneColumnAlignValue(value any) any {
+       if binary, ok := value.([]byte); ok {
+               return append([]byte(nil), binary...)
+       }
+       return value
+}
+
+// Flush returns no messages because alignment does not buffer deliveries.
+func (p *ColumnAlignProcessor) Flush() ([]SubscriptionMessage, error) {
+       return nil, nil
+}
+
+// Reset clears cached values for every topic.
+func (p *ColumnAlignProcessor) Reset() {
+       p.mu.Lock()
+       defer p.mu.Unlock()
+       p.lastValues = 
make(map[string]map[columnAlignDevice]map[columnAlignColumn]columnAlignValue)
+}
+
+// ResetTopic clears only the target topic's cached alignment values.
+func (p *ColumnAlignProcessor) ResetTopic(topic string) {
+       p.mu.Lock()
+       defer p.mu.Unlock()
+       delete(p.lastValues, topic)
+}
diff --git a/subscription/doc.go b/subscription/doc.go
index 9cb40cd..45e1458 100644
--- a/subscription/doc.go
+++ b/subscription/doc.go
@@ -27,8 +27,9 @@
 // NewSubscriptionPullConsumer provides pull consumption, record continuation,
 // commit and seek coordination, background heartbeats, and provider recovery.
 // NewSubscriptionPushConsumer adds sequential callback delivery with before- 
or
-// after-consumption acknowledgment and a terminal-error channel. Built-in
-// processors remain a subsequent implementation step.
+// after-consumption acknowledgment and a terminal-error channel.
+// NewWatermarkProcessor and NewColumnAlignProcessor provide optional Pull
+// pipeline stages for message reordering and per-device NULL forward filling.
 //
 // Record messages are the supported payload format. Topic management and
 // subscription do not reject TsFile topics; decoding a TsFile payload must 
return
diff --git a/subscription/watermark_processor.go 
b/subscription/watermark_processor.go
new file mode 100644
index 0000000..e528995
--- /dev/null
+++ b/subscription/watermark_processor.go
@@ -0,0 +1,297 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package subscription
+
+import (
+       "container/heap"
+       "errors"
+       "math"
+       "sync"
+       "time"
+)
+
+// WatermarkProcessorConfig controls client-side message reordering. Timestamp
+// distances use server precision; wall-clock waits use time.Duration.
+type WatermarkProcessorConfig struct {
+       // MaxOutOfOrderness is subtracted from the minimum active region 
timestamp.
+       // It must be nonnegative and uses the same unit as 
RecordBatch.Timestamp.
+       MaxOutOfOrderness int64
+       // Timeout force-flushes buffered records when this long has passed 
without
+       // output. Zero flushes on every Process call that produces no ordered 
output.
+       Timeout time.Duration
+       // StaleRegionTimeout excludes a region whose timestamp has not 
increased
+       // for this long. Zero selects 30 seconds; a negative value disables 
exclusion.
+       StaleRegionTimeout time.Duration
+       // MaxBufferBytes limits estimated retained message bytes. Zero selects
+       // 64 MiB. Exceeding the limit after ordered emission flushes the 
remainder.
+       MaxBufferBytes int64
+}
+
+// WatermarkProcessor buffers whole record messages in maximum-timestamp order.
+// Its watermark is the minimum latest timestamp across active 
topic/node/region
+// sources, minus MaxOutOfOrderness. Only increasing timestamps refresh source
+// activity; repeated server watermarks cannot keep a stalled source active.
+// All-stale sources leave the watermark unchanged. Timeout and capacity 
flushing
+// prevent indefinite buffering, but can emit records beyond the watermark.
+// This is a reference processor; large out-of-order streams may require a
+// downstream stream processing framework. Use one instance per consumer.
+type WatermarkProcessor struct {
+       mu sync.Mutex
+       config WatermarkProcessorConfig
+       buffer watermarkMessageHeap
+       regions map[watermarkRegion]watermarkRegionProgress
+       bufferedBytes int64
+       lastEmit time.Time
+       watermark int64
+       hasWatermark bool
+}
+
+type watermarkRegion struct {
+       topic string
+       node int32
+       region string
+}
+
+type watermarkRegionProgress struct {
+       timestamp int64
+       advanced time.Time
+}
+
+type watermarkBufferedMessage struct {
+       message SubscriptionMessage
+       maximum int64
+       size int64
+}
+
+type watermarkMessageHeap []watermarkBufferedMessage
+
+func (h watermarkMessageHeap) Len() int { return len(h) }
+func (h watermarkMessageHeap) Less(i, j int) bool { return h[i].maximum < 
h[j].maximum }
+func (h watermarkMessageHeap) Swap(i, j int) { h[i], h[j] = h[j], h[i] }
+func (h *watermarkMessageHeap) Push(value any) {
+       *h = append(*h, value.(watermarkBufferedMessage))
+}
+func (h *watermarkMessageHeap) Pop() any {
+       last := len(*h) - 1
+       value := (*h)[last]
+       (*h)[last] = watermarkBufferedMessage{}
+       *h = (*h)[:last]
+       return value
+}
+
+// NewWatermarkProcessor validates configuration without taking message 
ownership.
+// Timeout must be nonnegative. StaleRegionTimeout and MaxBufferBytes select 
their
+// documented defaults independently. Emission is message-level, not a global
+// row sort; Rows still sorts each batch independently.
+func NewWatermarkProcessor(config WatermarkProcessorConfig) 
(*WatermarkProcessor, error) {
+       if config.MaxOutOfOrderness < 0 || config.Timeout < 0 || 
config.MaxBufferBytes < 0 {
+               return nil, errors.New("subscription: watermark distance, 
timeout, and buffer limit must be nonnegative")
+       }
+       if config.StaleRegionTimeout == 0 {
+               config.StaleRegionTimeout = 30 * time.Second
+       }
+       if config.MaxBufferBytes == 0 {
+               config.MaxBufferBytes = 64 * 1024 * 1024
+       }
+       return &WatermarkProcessor{
+               config: config,
+               regions: make(map[watermarkRegion]watermarkRegionProgress),
+               lastEmit: time.Now(),
+       }, nil
+}
+
+// Process retains records until their maximum timestamp is at most the
+// watermark, or a timeout/capacity flush occurs. Watermark events update 
source
+// progress and are consumed. Empty input still evaluates staleness and 
timeout.
+// An input-access error leaves processor state and input ownership unchanged.
+func (p *WatermarkProcessor) Process(messages []SubscriptionMessage) 
([]SubscriptionMessage, error) {
+       p.mu.Lock()
+       defer p.mu.Unlock()
+       now := time.Now()
+       incoming := make([]watermarkBufferedMessage, 0, len(messages))
+       for _, message := range messages {
+               if message == nil {
+                       return nil, errors.New("subscription: watermark 
processor received a nil message")
+               }
+               if timestamp, ok := message.Watermark(); ok {
+                       incoming = append(incoming, 
watermarkBufferedMessage{message: message, maximum: timestamp})
+                       continue
+               }
+               batches, err := message.Records()
+               if err != nil {
+                       return nil, err
+               }
+               maximum, found := int64(math.MinInt64), false
+               for _, batch := range batches {
+                       if batch == nil {
+                               return nil, errors.New("subscription: watermark 
processor received a nil batch")
+                       }
+                       for row := 0; row < batch.Len(); row++ {
+                               timestamp, err := batch.Timestamp(row)
+                               if err != nil {
+                                       return nil, err
+                               }
+                               if !found || timestamp > maximum {
+                                       maximum = timestamp
+                               }
+                               found = true
+                       }
+               }
+               // As in Java, an empty message falls back to the wall clock in 
ms.
+               if !found {
+                       maximum = now.UnixMilli()
+               }
+               size := message.EstimatedSize()
+               if size < 0 {
+                       size = 0
+               }
+               incoming = append(incoming, watermarkBufferedMessage{message: 
message, maximum: maximum, size: size})
+       }
+       for _, entry := range incoming {
+               context := entry.message.CommitContext()
+               key := watermarkRegion{topic: context.Topic, node: 
context.DataNodeID, region: context.RegionID}
+               previous, ok := p.regions[key]
+               if !ok || entry.maximum > previous.timestamp {
+                       p.regions[key] = watermarkRegionProgress{timestamp: 
entry.maximum, advanced: now}
+               }
+               if entry.message.Kind() == WatermarkMessage {
+                       entry.message.Release()
+                       continue
+               }
+               heap.Push(&p.buffer, entry)
+               p.bufferedBytes += entry.size
+       }
+       p.updateWatermark(now)
+       var emitted []SubscriptionMessage
+       for len(p.buffer) > 0 && p.hasWatermark && p.buffer[0].maximum <= 
p.watermark {
+               entry := heap.Pop(&p.buffer).(watermarkBufferedMessage)
+               p.bufferedBytes -= entry.size
+               emitted = append(emitted, entry.message)
+       }
+       if p.bufferedBytes > p.config.MaxBufferBytes ||
+               (len(emitted) == 0 && len(p.buffer) > 0 && now.Sub(p.lastEmit) 
>= p.config.Timeout) {
+               emitted = append(emitted, p.flushLocked(now)...)
+       }
+       if len(emitted) > 0 {
+               p.lastEmit = now
+       }
+       return emitted, nil
+}
+
+// Flush transfers every buffered record in maximum-timestamp order without
+// resetting region progress or the current watermark.
+func (p *WatermarkProcessor) Flush() ([]SubscriptionMessage, error) {
+       p.mu.Lock()
+       defer p.mu.Unlock()
+       return p.flushLocked(time.Now()), nil
+}
+
+func (p *WatermarkProcessor) flushLocked(now time.Time) []SubscriptionMessage {
+       messages := make([]SubscriptionMessage, 0, len(p.buffer))
+       for len(p.buffer) > 0 {
+               messages = append(messages, 
heap.Pop(&p.buffer).(watermarkBufferedMessage).message)
+       }
+       p.bufferedBytes = 0
+       p.lastEmit = now
+       return messages
+}
+
+// Buffered atomically snapshots the retained count and every renewable 
context.
+func (p *WatermarkProcessor) Buffered() ProcessorBuffer {
+       p.mu.Lock()
+       defer p.mu.Unlock()
+       result := ProcessorBuffer{Count: len(p.buffer)}
+       for _, entry := range p.buffer {
+               context := entry.message.CommitContext()
+               if flowCommitContextCommittable(context) {
+                       result.Contexts = append(result.Contexts, context)
+               }
+       }
+       return result
+}
+
+// Watermark reports the current ordering watermark and whether one is known.
+// It can move backward when new regions appear or after a topic reset, as in 
Java.
+func (p *WatermarkProcessor) Watermark() (int64, bool) {
+       p.mu.Lock()
+       defer p.mu.Unlock()
+       return p.watermark, p.hasWatermark
+}
+
+// Reset releases buffered records and clears all ordering progress.
+func (p *WatermarkProcessor) Reset() {
+       p.mu.Lock()
+       defer p.mu.Unlock()
+       for _, entry := range p.buffer {
+               entry.message.Release()
+       }
+       p.buffer = nil
+       p.regions = make(map[watermarkRegion]watermarkRegionProgress)
+       p.bufferedBytes = 0
+       p.lastEmit = time.Now()
+       p.watermark, p.hasWatermark = 0, false
+}
+
+// ResetTopic releases only this topic's records and recomputes the watermark
+// from the remaining active sources. Other topics keep their buffered records.
+func (p *WatermarkProcessor) ResetTopic(topic string) {
+       p.mu.Lock()
+       defer p.mu.Unlock()
+       retained := p.buffer[:0]
+       for _, entry := range p.buffer {
+               if entry.message.CommitContext().Topic == topic {
+                       p.bufferedBytes -= entry.size
+                       entry.message.Release()
+               } else {
+                       retained = append(retained, entry)
+               }
+       }
+       clear(p.buffer[len(retained):])
+       p.buffer = retained
+       heap.Init(&p.buffer)
+       for key := range p.regions {
+               if key.topic == topic {
+                       delete(p.regions, key)
+               }
+       }
+       p.watermark, p.hasWatermark = 0, false
+       p.updateWatermark(time.Now())
+}
+
+func (p *WatermarkProcessor) updateWatermark(now time.Time) {
+       var minimum int64
+       found := false
+       for _, progress := range p.regions {
+               if p.config.StaleRegionTimeout >= 0 && 
now.Sub(progress.advanced) > p.config.StaleRegionTimeout {
+                       continue
+               }
+               if !found || progress.timestamp < minimum {
+                       minimum, found = progress.timestamp, true
+               }
+       }
+       if found {
+               // Saturate subtraction to preserve ordering at the timestamp 
boundary.
+               p.watermark = math.MinInt64
+               if minimum >= math.MinInt64 + p.config.MaxOutOfOrderness {
+                       p.watermark = minimum - p.config.MaxOutOfOrderness
+               }
+               p.hasWatermark = true
+       }
+}

Reply via email to