This is an automated email from the ASF dual-hosted git repository.

shuwenwei pushed a commit to branch supports_subscription
in repository https://gitbox.apache.org/repos/asf/iotdb-client-go.git

commit 37ef7f2dcf30ef8a04e3d907a5d04e8fe80ffaa1
Author: shuwenwei <[email protected]>
AuthorDate: Fri Nov 13 10:21:55 2026 +0800

    Decode subscription record messages and poll responses
---
 subscription/README.md        |  18 ++-
 subscription/doc.go           |   6 +-
 subscription/message_impl.go  | 332 ++++++++++++++++++++++++++++++++++++++++++
 subscription/provider.go      |   8 +-
 subscription/tablet_decode.go | 226 ++++++++++++++++++++++++++++
 subscription/wire_poll.go     | 273 ++++++++++++++++++++++++++++++++++
 6 files changed, 854 insertions(+), 9 deletions(-)

diff --git a/subscription/README.md b/subscription/README.md
index 8a78bd0..af8a373 100644
--- a/subscription/README.md
+++ b/subscription/README.md
@@ -19,7 +19,7 @@ under the License.
 
 # Go subscription 公共接口
 
-当前已实现 `SubscriptionSession` 的默认配置、构造、普通 Thrift 连接与认证,以及 Topic/Subscription 
管理;Consumer 默认配置、内部控制协议和单节点 Provider 也已补齐。公共 
Consumer、记录解析和处理器仍没有可运行的消费实现。下文对应表同时列出已实现能力和后续消费实现需要遵守的契约。
+当前已实现 `SubscriptionSession` 的默认配置、构造、普通 Thrift 连接与认证,以及 Topic/Subscription 
管理;Consumer 默认配置、内部控制协议和单节点 Provider、Poll 响应与记录解析、消息及记录访问实现也已补齐。公共 Consumer 
和处理器仍没有可运行的消费实现。下文对应表同时列出已实现能力和后续消费实现需要遵守的契约。
 
 模块路径为 `github.com/apache/iotdb-client-go/subscription`,使用 Go 1.25,与现有 
`github.com/apache/iotdb-client-go/v2` Session 模块分别维护依赖和版本。当前仅依赖现有模块,以复用 
`TSDataType` 和 `ColumnCategory`;没有 TsFile、cgo 或 TLS 依赖,也没有改动现有 Session 接口。
 
@@ -73,7 +73,19 @@ topics, err := s.Topics(ctx)
 - 帧限制取本地与服务端较小值并预留协议空间;旧服务端没有属性字段或不支持属性探测 RPC 时回退本地限制。可分片请求的所有片段保持同一连接、顺序和批次身份。
 - 心跳按报文大小分批发送缓存上下文;提交预留响应空间分批,后续批次失败时仍返回先前已接受的上下文和进度。没有响应 body 
时保留空结果,不推断整批提交成功。
 - 状态码保留普通错误、owner fencing、consumer fencing、连接替换和致命错误的区别。Poll 返回消费组 Broker 
丢失诊断时使旧连接失效,供后续 Consumer 重建;失效连接不会被旧响应重新启用。
-- 本轮只编码记录拉取请求,不解析其响应,也不实现文件拉取。TsFile 在解析时报错的契约保留,实际分派与记录解码在后续阶段接入。
+- Provider 现在解析全部非空 Poll 响应 body,保留记录、错误、终止和水位线事件。文件响应在类型分派处返回 
`ErrUnsupportedTsFile`;混合响应不会返回已经解析的部分记录,不实现文件拉取。
+
+## 记录消息与解析
+
+记录数据采用 Java 当前依赖的 `2.5.1-261023-SNAPSHOT` Tablet 序列化格式,与写入 RPC 和查询 TsBlock 
格式分开处理,不依赖 TsFile Go 或原生库。
+
+- 支持 BOOLEAN、INT32、INT64、FLOAT、DOUBLE、TEXT、STRING、TIMESTAMP、DATE、BLOB 和 
OBJECT,保留 TAG/FIELD/ATTRIBUTE 类别。TEXT/STRING 暴露为 string,BLOB/OBJECT 为 
`[]byte`,DATE 为 UTC 零点的 `time.Time`。
+- NULL 由 Bitmap 和缺失值列决定。Binary 的缺省标记先恢复为空内容,再应用 Bitmap;不把空字符串或空 BLOB 自动变成 
NULL。日期按 Java 的日期整数规则读取。
+- 原始行号用于 `Timestamp/Value`;`Rows()` 
通过独立的稳定排序视图按时间递增遍历,相同时间戳保持原始顺序。行值切片不在迭代间复用,调用用户的迭代函数时不持内部锁。
+- 树模型始终显示时间戳;表模型优先采用按数据库、表指定的时间投影,未指定时回退消息级标志。旧响应缺少投影尾部时保持 Java 的默认选择时间行为。
+- 消息释放幂等并清理记录缓存;提交上下文及其可选 Writer 元数据仍以独立快照返回,水位线和消息类型继续可访问。已借用的批次在释放后数据访问报错。
+- 记录续取组装校验完整提交上下文,逐页合并数据和时间投影;最终数量按 Tablet 个数核对,不按行数核对。该组装能力尚未接入 Consumer 
的实际续取循环。
+- 空记录交付和终止事件在解析层保留上下文,不自动 ACK 或退订;服务端 ERROR 保留严重级别,水位线可转为处理器消息。消费循环中的处理仍待下一步实现。
 
 ## 管理 Session
 
@@ -167,4 +179,4 @@ SubscriptionMessageProcessor 保留消息时必须实现 `BufferingProcessor`,
 
 这些哨兵可以被后续操作错误包装,调用方使用 `errors.Is` 识别。TsFile 不支持错误不得触发连接重试或文件载荷确认。Pull 
本轮返回错误而不交付任何消息,包括并行分支已经取得的记录;本轮记录不得推进位置、进入处理器或自动提交缓存,此前成功交付及提交的状态不回滚。调用方可退订相关 
Topic 后继续拉取。
 
-后续步骤尚需实现记录及 Poll 响应解析、公共 Consumer 和构造、多节点协调、提交进度与恢复、后台任务、内置 
Watermark/ColumnAlign Processor,以及消费示例和行为测试。内部 Provider 已有提交 RPC,但尚未实现公共 
Consumer 
的交付和确认链。管理查询复用根模块解码器,本轮不修改其对畸形数据的处理逻辑。当前仅完成静态核对,没有编译、运行测试或执行格式工具,不提供运行验证结论。
+后续步骤尚需实现公共 Consumer 和构造、实际记录续取循环、多节点协调、提交进度与恢复、后台任务、内置 Watermark/ColumnAlign 
Processor,以及消费示例和行为测试。内部 Provider 已有提交 RPC,但尚未实现公共 Consumer 
的交付和确认链。管理查询复用根模块解码器,本轮不修改其对畸形数据的处理逻辑。当前仅完成静态核对,没有编译、运行测试或执行格式工具,不提供运行验证结论。
diff --git a/subscription/doc.go b/subscription/doc.go
index 8f1dece..1864210 100644
--- a/subscription/doc.go
+++ b/subscription/doc.go
@@ -22,8 +22,10 @@
 //
 // SubscriptionSession provides topic and subscription management through
 // NewSubscriptionSession. Consumer configuration defaults and internal control
-// protocol/provider support are available. Public consumer implementations,
-// record decoding, and built-in processors remain subsequent implementation 
steps.
+// protocol/provider support are available. Poll response and record decoding,
+// record views, message release, and watermark messages are implemented 
internally.
+// Public consumers, background coordination, and built-in processors remain
+// subsequent implementation steps.
 //
 // Record messages are the supported payload format. Topic management and
 // subscription do not reject TsFile topics; decoding a TsFile payload must 
return
diff --git a/subscription/message_impl.go b/subscription/message_impl.go
new file mode 100644
index 0000000..5c957c5
--- /dev/null
+++ b/subscription/message_impl.go
@@ -0,0 +1,332 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package subscription
+
+import (
+       "fmt"
+       "iter"
+       "sort"
+       "sync"
+       "time"
+)
+
+// subscriptionMessage keeps immutable delivery metadata separate from 
releasable
+// record buffers. Its factories take ownership of decoded batches.
+type subscriptionMessage struct {
+       mu sync.RWMutex
+       kind SubscriptionMessageKind
+       context CommitContext
+       watermark int64
+       batches []RecordBatch
+       released bool
+}
+
+func newSubscriptionRecordMessage(commitContext CommitContext, batches 
[]RecordBatch) SubscriptionMessage {
+       return &subscriptionMessage{
+               kind: RecordMessage,
+               context: cloneMessageCommitContext(commitContext),
+               batches: append([]RecordBatch(nil), batches...),
+       }
+}
+
+func newSubscriptionWatermarkMessage(commitContext CommitContext, timestamp 
int64) SubscriptionMessage {
+       return &subscriptionMessage{
+               kind: WatermarkMessage,
+               context: cloneMessageCommitContext(commitContext),
+               watermark: timestamp,
+       }
+}
+
+func cloneMessageCommitContext(commitContext CommitContext) CommitContext {
+       if commitContext.Writer != nil {
+               writer := *commitContext.Writer
+               commitContext.Writer = &writer
+       }
+       return commitContext
+}
+
+func (m *subscriptionMessage) Kind() SubscriptionMessageKind {
+       return m.kind
+}
+
+func (m *subscriptionMessage) CommitContext() CommitContext {
+       return cloneMessageCommitContext(m.context)
+}
+
+func (m *subscriptionMessage) Records() ([]RecordBatch, error) {
+       if m.kind != RecordMessage {
+               return nil, ErrIncompatibleMessage
+       }
+       m.mu.RLock()
+       defer m.mu.RUnlock()
+       if m.released {
+               return nil, ErrDataReleased
+       }
+       return append([]RecordBatch(nil), m.batches...), nil
+}
+
+func (m *subscriptionMessage) Watermark() (int64, bool) {
+       return m.watermark, m.kind == WatermarkMessage
+}
+
+func (m *subscriptionMessage) EstimatedSize() int64 {
+       m.mu.RLock()
+       defer m.mu.RUnlock()
+       // Include a rough object/metadata allowance and retained variable data.
+       // This estimate is intended for buffering limits, not heap accounting.
+       size := int64(128) + int64(len(m.kind)) + int64(len(m.context.Topic)) +
+               int64(len(m.context.GroupID)) + int64(len(m.context.RegionID))
+       if m.context.Writer != nil {
+               size += 32 + int64(len(m.context.Writer.ID.RegionID))
+       }
+       size += int64(len(m.batches)) * 16
+       for _, batch := range m.batches {
+               if records, ok := batch.(*subscriptionRecordBatch); ok {
+                       size += records.estimatedSize()
+               }
+       }
+       return size
+}
+
+func (m *subscriptionMessage) Release() {
+       m.mu.Lock()
+       defer m.mu.Unlock()
+       if m.released {
+               return
+       }
+       m.released = true
+       for _, batch := range m.batches {
+               if records, ok := batch.(*subscriptionRecordBatch); ok {
+                       records.release()
+               }
+       }
+       m.batches = nil
+}
+
+// subscriptionRecordBatch preserves the decoded Tablet's indexed order and 
uses
+// a separate stable permutation for user-visible iteration. Immutable target
+// metadata remains available after release; Columns and Len return empty 
values.
+type subscriptionRecordBatch struct {
+       mu sync.RWMutex
+       dialect Dialect
+       database string
+       target string
+       timeSelected bool
+       rowCount int
+       columns []Column
+       timestamps []int64
+       values [][]any
+       rowOrder []int
+       released bool
+}
+
+func newSubscriptionRecordBatch(dialect Dialect, database string, tablet 
decodedSubscriptionTablet, timeSelected bool) RecordBatch {
+       if dialect == TreeDialect {
+               database = ""
+               timeSelected = true
+       }
+       batch := &subscriptionRecordBatch{
+               dialect: dialect,
+               database: database,
+               target: tablet.Target,
+               timeSelected: timeSelected,
+               rowCount: tablet.RowCount,
+               columns: tablet.Columns,
+               timestamps: tablet.Timestamps,
+               values: tablet.Values,
+       }
+       // A missing timestamp array represents zero for every row. Its 
ordering is
+       // already stable, so keep that representation sparse and iterate 
lazily.
+       if tablet.Timestamps != nil {
+               batch.rowOrder = make([]int, tablet.RowCount)
+               for index := range batch.rowOrder {
+                       batch.rowOrder[index] = index
+               }
+               sort.SliceStable(batch.rowOrder, func(left, right int) bool {
+                       return batch.timestamps[batch.rowOrder[left]] < 
batch.timestamps[batch.rowOrder[right]]
+               })
+       }
+       return batch
+}
+
+func (b *subscriptionRecordBatch) Dialect() Dialect {
+       return b.dialect
+}
+
+func (b *subscriptionRecordBatch) Database() string {
+       return b.database
+}
+
+func (b *subscriptionRecordBatch) Target() string {
+       return b.target
+}
+
+func (b *subscriptionRecordBatch) Columns() []Column {
+       b.mu.RLock()
+       defer b.mu.RUnlock()
+       return b.columns
+}
+
+func (b *subscriptionRecordBatch) TimeSelected() bool {
+       return b.timeSelected
+}
+
+func (b *subscriptionRecordBatch) Len() int {
+       b.mu.RLock()
+       defer b.mu.RUnlock()
+       return b.rowCount
+}
+
+func (b *subscriptionRecordBatch) Timestamp(row int) (int64, error) {
+       b.mu.RLock()
+       defer b.mu.RUnlock()
+       if b.released {
+               return 0, ErrDataReleased
+       }
+       if row < 0 || row >= b.rowCount {
+               return 0, fmt.Errorf("subscription: row index %d out of range", 
row)
+       }
+       if b.timestamps == nil {
+               return 0, nil
+       }
+       return b.timestamps[row], nil
+}
+
+func (b *subscriptionRecordBatch) Value(row, column int) (any, error) {
+       b.mu.RLock()
+       defer b.mu.RUnlock()
+       if b.released {
+               return nil, ErrDataReleased
+       }
+       if row < 0 || row >= b.rowCount {
+               return nil, fmt.Errorf("subscription: row index %d out of 
range", row)
+       }
+       if column < 0 || column >= len(b.columns) {
+               return nil, fmt.Errorf("subscription: column index %d out of 
range", column)
+       }
+       return b.valueLocked(row, column)
+}
+
+func (b *subscriptionRecordBatch) valueLocked(row, column int) (any, error) {
+       if b.values == nil {
+               return nil, nil
+       }
+       if column >= len(b.values) {
+               return nil, fmt.Errorf("subscription: record column %d has no 
value storage", column)
+       }
+       if b.values[column] == nil {
+               return nil, nil
+       }
+       if row >= len(b.values[column]) {
+               return nil, fmt.Errorf("subscription: record column %d has no 
value for row %d", column, row)
+       }
+       return b.values[column][row], nil
+}
+
+func (b *subscriptionRecordBatch) Rows() iter.Seq2[Row, error] {
+       return func(yield func(Row, error) bool) {
+               for position := 0; ; position++ {
+                       b.mu.RLock()
+                       if b.released {
+                               b.mu.RUnlock()
+                               yield(Row{}, ErrDataReleased)
+                               return
+                       }
+                       if position >= b.rowCount {
+                               b.mu.RUnlock()
+                               return
+                       }
+                       index := position
+                       if b.rowOrder != nil {
+                               index = b.rowOrder[position]
+                       }
+                       row := Row{Values: make([]any, len(b.columns))}
+                       if b.timeSelected {
+                               var timestamp int64
+                               if b.timestamps != nil {
+                                       timestamp = b.timestamps[index]
+                               }
+                               row.Timestamp = &timestamp
+                       }
+                       var err error
+                       for column := range row.Values {
+                               row.Values[column], err = b.valueLocked(index, 
column)
+                               if err != nil {
+                                       break
+                               }
+                       }
+                       b.mu.RUnlock()
+                       // The caller may release the message or stop iteration 
from yield.
+                       // Never hold a batch lock while invoking 
caller-controlled code.
+                       if err != nil {
+                               yield(Row{}, err)
+                               return
+                       }
+                       if !yield(row, nil) {
+                               return
+                       }
+               }
+       }
+}
+
+func (b *subscriptionRecordBatch) release() {
+       b.mu.Lock()
+       defer b.mu.Unlock()
+       if b.released {
+               return
+       }
+       b.released = true
+       b.rowCount = 0
+       b.columns = nil
+       b.timestamps = nil
+       b.values = nil
+       b.rowOrder = nil
+}
+
+func (b *subscriptionRecordBatch) estimatedSize() int64 {
+       b.mu.RLock()
+       defer b.mu.RUnlock()
+       size := int64(128) + int64(len(b.dialect)) + int64(len(b.database)) +
+               int64(len(b.target)) + int64(len(b.timestamps))*8 + 
int64(len(b.rowOrder))*8
+       size += int64(len(b.columns))*32 + int64(len(b.values))*24
+       for _, column := range b.columns {
+               size += int64(len(column.Name))
+       }
+       for _, values := range b.values {
+               size += int64(len(values))*16
+               for _, value := range values {
+                       switch value := value.(type) {
+                       case bool:
+                               size++
+                       case int32, float32:
+                               size += 4
+                       case int64, float64:
+                               size += 8
+                       case string:
+                               size += 16 + int64(len(value))
+                       case []byte:
+                               size += 24 + int64(len(value))
+                       case time.Time:
+                               size += 24
+                       }
+               }
+       }
+       return size
+}
diff --git a/subscription/provider.go b/subscription/provider.go
index 472652a..238002d 100644
--- a/subscription/provider.go
+++ b/subscription/provider.go
@@ -337,7 +337,7 @@ func (p *subscriptionProvider) seek(ctx context.Context, 
topic string, kind int1
        return err
 }
 
-func (p *subscriptionProvider) poll(ctx context.Context, topics []string, 
timeoutMs, maxBytes int64, progress map[string]TopicProgress) ([][]byte, error) 
{
+func (p *subscriptionProvider) poll(ctx context.Context, topics []string, 
timeoutMs, maxBytes int64, progress map[string]TopicProgress) 
([]subscriptionPollResponse, error) {
        request, err := encodeSubscriptionPoll(topics, timeoutMs, maxBytes, 
progress)
        if err != nil {
                return nil, err
@@ -345,7 +345,7 @@ func (p *subscriptionProvider) poll(ctx context.Context, 
topics []string, timeou
        return p.pollPayloads(ctx, request)
 }
 
-func (p *subscriptionProvider) pollTablets(ctx context.Context, commitContext 
CommitContext, offset int32, timeoutMs, maxBytes int64) ([][]byte, error) {
+func (p *subscriptionProvider) pollTablets(ctx context.Context, commitContext 
CommitContext, offset int32, timeoutMs, maxBytes int64) 
([]subscriptionPollResponse, error) {
        request, err := encodeSubscriptionPollTablets(commitContext, offset, 
timeoutMs, maxBytes)
        if err != nil {
                return nil, err
@@ -353,7 +353,7 @@ func (p *subscriptionProvider) pollTablets(ctx 
context.Context, commitContext Co
        return p.pollPayloads(ctx, request)
 }
 
-func (p *subscriptionProvider) pollPayloads(ctx context.Context, request 
*rpc.TPipeSubscribeReq) ([][]byte, error) {
+func (p *subscriptionProvider) pollPayloads(ctx context.Context, request 
*rpc.TPipeSubscribeReq) ([]subscriptionPollResponse, error) {
        response, err := p.call(ctx, request)
        if err != nil {
                var statusErr *subscriptionStatusError
@@ -369,7 +369,7 @@ func (p *subscriptionProvider) pollPayloads(ctx 
context.Context, request *rpc.TP
                }
                return nil, err
        }
-       return response.Body, nil
+       return decodeSubscriptionPollResponses(response.Body)
 }
 
 // close preserves subscriptions during fenced/recovery teardown. A regular 
close
diff --git a/subscription/tablet_decode.go b/subscription/tablet_decode.go
new file mode 100644
index 0000000..4988b06
--- /dev/null
+++ b/subscription/tablet_decode.go
@@ -0,0 +1,226 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package subscription
+
+import (
+       "fmt"
+       "io"
+       "math"
+       "time"
+
+       "github.com/apache/iotdb-client-go/v2/client"
+)
+
+// decodedSubscriptionTablet follows TsFile 2.5.1-261023-SNAPSHOT Tablet's 
stream
+// format, independently of the Session insert protocol and TsBlock query 
format.
+// Nil Timestamps represents RowCount zero timestamps. Each nil Values column
+// represents RowCount null values without allocating storage for absent data.
+type decodedSubscriptionTablet struct {
+       Target string
+       Columns []Column
+       RowCount int
+       Timestamps []int64
+       Values [][]any
+}
+
+func decodeSubscriptionTablet(reader *subscriptionWireReader) 
(decodedSubscriptionTablet, error) {
+       tablet := decodedSubscriptionTablet{Target: reader.readString()}
+       rowCount := reader.readInt32()
+       if reader.err != nil {
+               return decodedSubscriptionTablet{}, reader.err
+       }
+       if rowCount < 0 {
+               return decodedSubscriptionTablet{}, fmt.Errorf("subscription: 
invalid Tablet row count %d", rowCount)
+       }
+       tablet.RowCount = int(rowCount)
+       if reader.readBool() {
+               columnCount := reader.readCount(1, false)
+               for index := int32(0); index < columnCount && reader.err == 
nil; index++ {
+                       hasSchema := reader.readBool()
+                       if reader.err != nil {
+                               return decodedSubscriptionTablet{}, reader.err
+                       }
+                       if !hasSchema {
+                               return decodedSubscriptionTablet{}, 
fmt.Errorf("subscription: null Tablet schema at column %d is not supported by 
the record handler", index)
+                       }
+                       column := Column{Name: reader.readString(), DataType: 
client.TSDataType(reader.readByte())}
+                       // Encoding and compression are metadata in this 
uncompressed array
+                       // representation; they do not select a value decoder 
here.
+                       reader.readByte()
+                       reader.readByte()
+                       properties := reader.readCount(8, false)
+                       for property := int32(0); property < properties && 
reader.err == nil; property++ {
+                               reader.readString()
+                               reader.readString()
+                       }
+                       category := reader.readByte()
+                       if reader.err != nil {
+                               break
+                       }
+                       switch category {
+                       case 0:
+                               column.Category = client.TAG
+                       case 1:
+                               column.Category = client.FIELD
+                       case 2:
+                               column.Category = client.ATTRIBUTE
+                       default:
+                               return decodedSubscriptionTablet{}, 
fmt.Errorf("subscription: unsupported Tablet column category %d", category)
+                       }
+                       if _, err := 
subscriptionTabletValueSize(column.DataType); err != nil {
+                               return decodedSubscriptionTablet{}, err
+                       }
+                       tablet.Columns = append(tablet.Columns, column)
+               }
+       }
+       if reader.err != nil {
+               return decodedSubscriptionTablet{}, reader.err
+       }
+       if reader.readBool() {
+               if err := requireSubscriptionTabletValues(reader, 
tablet.RowCount, 8); err != nil {
+                       return decodedSubscriptionTablet{}, err
+               }
+               tablet.Timestamps = make([]int64, tablet.RowCount)
+               for row := range tablet.Timestamps {
+                       tablet.Timestamps[row] = reader.readInt64()
+               }
+       }
+       bitmaps := make([][]byte, len(tablet.Columns))
+       if reader.readBool() {
+               for column := range tablet.Columns {
+                       if !reader.readBool() {
+                               continue
+                       }
+                       size := reader.readInt32()
+                       length := reader.readCount(1, false)
+                       if reader.err != nil {
+                               break
+                       }
+                       if size < rowCount || int64(length) < 
(int64(rowCount)+7)/8 {
+                               return decodedSubscriptionTablet{}, 
fmt.Errorf("subscription: Tablet bitmap cannot represent %d rows", rowCount)
+                       }
+                       bitmaps[column] = reader.take(int(length))
+               }
+       }
+       if reader.err != nil {
+               return decodedSubscriptionTablet{}, reader.err
+       }
+       tablet.Values = make([][]any, len(tablet.Columns))
+       if reader.readBool() {
+               for column, schema := range tablet.Columns {
+                       if !reader.readBool() {
+                               continue
+                       }
+                       values, err := decodeSubscriptionTabletColumn(reader, 
schema.DataType, tablet.RowCount)
+                       if err != nil {
+                               return decodedSubscriptionTablet{}, err
+                       }
+                       for row := range values {
+                               if bitmap := bitmaps[column]; bitmap != nil && 
bitmap[row/8]&(1<<uint(row%8)) != 0 {
+                                       values[row] = nil
+                               }
+                       }
+                       tablet.Values[column] = values
+               }
+       }
+       if reader.err != nil {
+               return decodedSubscriptionTablet{}, reader.err
+       }
+       return tablet, nil
+}
+
+// subscriptionTabletValueSize returns the minimum wire bytes per array entry.
+// Binary-backed types use a presence byte before their optional length and 
data.
+func subscriptionTabletValueSize(dataType client.TSDataType) (int, error) {
+       switch dataType {
+       case client.BOOLEAN, client.TEXT, client.STRING, client.BLOB, 
client.OBJECT:
+               return 1, nil
+       case client.INT32, client.FLOAT, client.DATE:
+               return 4, nil
+       case client.INT64, client.DOUBLE, client.TIMESTAMP:
+               return 8, nil
+       default:
+               return 0, fmt.Errorf("subscription: unsupported Tablet data 
type %d", dataType)
+       }
+}
+
+func requireSubscriptionTabletValues(reader *subscriptionWireReader, count, 
size int) error {
+       if reader.err != nil {
+               return reader.err
+       }
+       if count > reader.remaining()/size {
+               reader.err = fmt.Errorf("subscription: Tablet array of %d 
entries exceeds remaining payload: %w", count, io.ErrUnexpectedEOF)
+               return reader.err
+       }
+       return nil
+}
+
+func decodeSubscriptionTabletColumn(reader *subscriptionWireReader, dataType 
client.TSDataType, rows int) ([]any, error) {
+       size, err := subscriptionTabletValueSize(dataType)
+       if err != nil {
+               return nil, err
+       }
+       if err := requireSubscriptionTabletValues(reader, rows, size); err != 
nil {
+               return nil, err
+       }
+       values := make([]any, rows)
+       for row := range values {
+               switch dataType {
+               case client.BOOLEAN:
+                       values[row] = reader.readBool()
+               case client.INT32:
+                       values[row] = reader.readInt32()
+               case client.INT64, client.TIMESTAMP:
+                       values[row] = reader.readInt64()
+               case client.FLOAT:
+                       values[row] = 
math.Float32frombits(uint32(reader.readInt32()))
+               case client.DOUBLE:
+                       values[row] = 
math.Float64frombits(uint64(reader.readInt64()))
+               case client.DATE:
+                       date := reader.readInt32()
+                       if reader.err != nil {
+                               return nil, reader.err
+                       }
+                       year, month, day := int(date/10000), 
time.Month((date/100)%100), int(date%100)
+                       value := time.Date(year, month, day, 0, 0, 0, 0, 
time.UTC)
+                       if value.Year() != year || value.Month() != month || 
value.Day() != day {
+                               return nil, fmt.Errorf("subscription: invalid 
Tablet DATE value %d", date)
+                       }
+                       values[row] = value
+               case client.TEXT, client.STRING, client.BLOB, client.OBJECT:
+                       var binary []byte
+                       if reader.readBool() {
+                               length := reader.readCount(1, false)
+                               binary = reader.take(int(length))
+                       }
+                       if dataType == client.TEXT || dataType == client.STRING 
{
+                               values[row] = string(binary)
+                       } else {
+                               // Type.deserializeArray uses 
Binary.EMPTY_VALUE for a missing
+                               // Binary. SQL NULL is supplied separately by 
the Tablet bitmap.
+                               values[row] = append([]byte{}, binary...)
+                       }
+               }
+               if reader.err != nil {
+                       return nil, reader.err
+               }
+       }
+       return values, nil
+}
diff --git a/subscription/wire_poll.go b/subscription/wire_poll.go
new file mode 100644
index 0000000..4dbbc88
--- /dev/null
+++ b/subscription/wire_poll.go
@@ -0,0 +1,273 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package subscription
+
+import (
+       "encoding/binary"
+       "fmt"
+       "strings"
+)
+
+const (
+       subscriptionResponseError int16 = 0
+       subscriptionResponseTablets int16 = 1
+       subscriptionResponseFileInit int16 = 2
+       subscriptionResponseFilePiece int16 = 3
+       subscriptionResponseFileSeal int16 = 4
+       subscriptionResponseTermination int16 = 5
+       subscriptionResponseWatermark int16 = 7
+)
+
+type subscriptionPollBatch struct {
+       Database string
+       Tablet decodedSubscriptionTablet
+}
+
+// subscriptionPollResponse retains transport events separately from 
deliverable
+// messages. In particular, termination and empty record deliveries still need
+// consumer-level unsubscribe or confirmation; decoding does neither.
+type subscriptionPollResponse struct {
+       Type int16
+       CommitContext CommitContext
+       Records []subscriptionPollBatch
+       NextOffset int32
+       TimeSelected bool
+       TimeSelectedByTable map[string]map[string]bool
+       Watermark int64
+       DataNodeID int32
+       ErrorMessage string
+       Critical bool
+}
+
+// decodeSubscriptionPollResponses parses every nonempty body entry. No 
partially
+// decoded response is returned if any entry fails, including an unsupported 
file
+// entry that follows otherwise valid records.
+func decodeSubscriptionPollResponses(bodies [][]byte) 
([]subscriptionPollResponse, error) {
+       // Reject files before decoding records in sibling entries. This 
examines the
+       // response tag only; it does not parse file payloads or attempt file 
I/O.
+       for _, body := range bodies {
+               if len(body) < 2 {
+                       continue
+               }
+               kind := int16(binary.BigEndian.Uint16(body[:2]))
+               if isSubscriptionFileResponse(kind) {
+                       return nil, fmt.Errorf("subscription: decode poll 
response type %d: %w", kind, ErrUnsupportedTsFile)
+               }
+       }
+       var responses []subscriptionPollResponse
+       for index, body := range bodies {
+               if len(body) == 0 {
+                       continue
+               }
+               response, err := decodeSubscriptionPollResponse(body)
+               if err != nil {
+                       return nil, fmt.Errorf("subscription: decode poll body 
%d: %w", index, err)
+               }
+               responses = append(responses, response)
+       }
+       return responses, nil
+}
+
+func decodeSubscriptionPollResponse(body []byte) (subscriptionPollResponse, 
error) {
+       reader := newSubscriptionWireReader(body)
+       response := subscriptionPollResponse{Type: reader.readInt16(), 
TimeSelected: true}
+       if reader.err != nil {
+               return subscriptionPollResponse{}, reader.err
+       }
+       switch response.Type {
+       case subscriptionResponseTablets:
+               databaseCount := reader.readCount(8, false)
+               for databaseIndex := int32(0); databaseIndex < databaseCount && 
reader.err == nil; databaseIndex++ {
+                       database := reader.readString()
+                       tabletCount := reader.readCount(12, false)
+                       for tabletIndex := int32(0); tabletIndex < tabletCount 
&& reader.err == nil; tabletIndex++ {
+                               tablet, err := decodeSubscriptionTablet(reader)
+                               if err != nil {
+                                       return subscriptionPollResponse{}, err
+                               }
+                               response.Records = append(response.Records, 
subscriptionPollBatch{Database: database, Tablet: tablet})
+                       }
+               }
+               response.NextOffset = reader.readInt32()
+       case subscriptionResponseError:
+               response.ErrorMessage = reader.readString()
+               response.Critical = reader.readBool()
+       case subscriptionResponseTermination:
+               // The termination payload is empty; its topic comes from the 
context.
+       case subscriptionResponseWatermark:
+               response.Watermark = reader.readInt64()
+               response.DataNodeID = reader.readInt32()
+       case subscriptionResponseFileInit, subscriptionResponseFilePiece, 
subscriptionResponseFileSeal:
+               return subscriptionPollResponse{}, fmt.Errorf("subscription: 
decode poll response type %d: %w", response.Type, ErrUnsupportedTsFile)
+       default:
+               return subscriptionPollResponse{}, fmt.Errorf("subscription: 
unsupported poll response type %d", response.Type)
+       }
+       if reader.err != nil {
+               return subscriptionPollResponse{}, reader.err
+       }
+       response.CommitContext = readSubscriptionCommitContext(reader)
+       // Older responses end after the context or after the global projection 
flag.
+       if reader.remaining() > 0 {
+               response.TimeSelected = reader.readBool()
+       }
+       if reader.remaining() > 0 {
+               response.TimeSelectedByTable = 
readSubscriptionTimeSelection(reader)
+       }
+       if reader.err != nil {
+               return subscriptionPollResponse{}, reader.err
+       }
+       if reader.remaining() != 0 {
+               return subscriptionPollResponse{}, fmt.Errorf("subscription: 
poll response has %d trailing bytes", reader.remaining())
+       }
+       return response, nil
+}
+
+func isSubscriptionFileResponse(kind int16) bool {
+       return kind == subscriptionResponseFileInit || kind == 
subscriptionResponseFilePiece || kind == subscriptionResponseFileSeal
+}
+
+func readSubscriptionTimeSelection(reader *subscriptionWireReader) 
map[string]map[string]bool {
+       count := reader.readCount(8, false)
+       selection := make(map[string]map[string]bool)
+       for databaseIndex := int32(0); databaseIndex < count && reader.err == 
nil; databaseIndex++ {
+               database, hasDatabase := readSubscriptionProjectionKey(reader)
+               tableCount := reader.readCount(5, false)
+               tables := make(map[string]bool)
+               for tableIndex := int32(0); tableIndex < tableCount && 
reader.err == nil; tableIndex++ {
+                       table, hasTable := readSubscriptionProjectionKey(reader)
+                       selected := reader.readBool()
+                       if hasTable {
+                               
tables[strings.ToLower(strings.TrimSpace(table))] = selected
+                       }
+               }
+               if hasDatabase && len(tables) != 0 {
+                       selection[strings.ToLower(strings.TrimSpace(database))] 
= tables
+               }
+       }
+       return selection
+}
+
+// Projection maps discard null keys but preserve empty keys, unlike ordinary
+// Go string protocol fields where null and empty both map to the zero value.
+func readSubscriptionProjectionKey(reader *subscriptionWireReader) (string, 
bool) {
+       length := reader.readInt32()
+       if reader.err != nil || length == -1 {
+               return "", false
+       }
+       if length < -1 {
+               reader.err = fmt.Errorf("subscription: invalid projection key 
length %d", length)
+               return "", false
+       }
+       key := reader.take(int(length))
+       return string(key), reader.err == nil
+}
+
+// appendTablets merges a continuation page only after validating its delivery
+// identity. Global time projection is ANDed; table overrides use the latest 
page.
+func (response *subscriptionPollResponse) appendTablets(next 
subscriptionPollResponse) error {
+       if response.Type != subscriptionResponseTablets || response.NextOffset 
<= 0 {
+               return fmt.Errorf("subscription: record response does not need 
a continuation")
+       }
+       if next.Type != subscriptionResponseTablets {
+               return fmt.Errorf("subscription: expected tablet continuation, 
received type %d", next.Type)
+       }
+       if !sameSubscriptionCommitContext(response.CommitContext, 
next.CommitContext) {
+               return fmt.Errorf("subscription: tablet continuation has a 
different commit context")
+       }
+       response.Records = append(response.Records, next.Records...)
+       response.NextOffset = next.NextOffset
+       response.TimeSelected = response.TimeSelected && next.TimeSelected
+       if response.TimeSelectedByTable == nil {
+               response.TimeSelectedByTable = make(map[string]map[string]bool)
+       }
+       for database, tables := range next.TimeSelectedByTable {
+               if response.TimeSelectedByTable[database] == nil {
+                       response.TimeSelectedByTable[database] = 
make(map[string]bool)
+               }
+               for table, selected := range tables {
+                       response.TimeSelectedByTable[database][table] = selected
+               }
+       }
+       return nil
+}
+
+// message creates a complete record view or a processor watermark. A complete
+// zero-tablet delivery returns nil; its context remains available for the 
future
+// consumer's empty-delivery ACK. No protocol state changes occur here.
+func (response *subscriptionPollResponse) message(dialect Dialect) 
(SubscriptionMessage, error) {
+       if dialect != TreeDialect && dialect != TableDialect {
+               return nil, fmt.Errorf("subscription: unsupported dialect %q", 
dialect)
+       }
+       switch response.Type {
+       case subscriptionResponseTablets:
+               if response.NextOffset > 0 {
+                       return nil, fmt.Errorf("subscription: record message 
requires tablet continuation at offset %d", response.NextOffset)
+               }
+               if int64(len(response.Records)) != -int64(response.NextOffset) {
+                       return nil, fmt.Errorf("subscription: tablet count %d 
does not match final offset %d", len(response.Records), response.NextOffset)
+               }
+               if len(response.Records) == 0 {
+                       return nil, nil
+               }
+               batches := make([]RecordBatch, 0, len(response.Records))
+               for _, record := range response.Records {
+                       selected := response.TimeSelected
+                       if tables := 
response.TimeSelectedByTable[strings.ToLower(strings.TrimSpace(record.Database))];
 tables != nil {
+                               if value, ok := 
tables[strings.ToLower(strings.TrimSpace(record.Tablet.Target))]; ok {
+                                       selected = value
+                               }
+                       }
+                       batches = append(batches, 
newSubscriptionRecordBatch(dialect, record.Database, record.Tablet, selected))
+               }
+               return newSubscriptionRecordMessage(response.CommitContext, 
batches), nil
+       case subscriptionResponseWatermark:
+               return newSubscriptionWatermarkMessage(response.CommitContext, 
response.Watermark), nil
+       case subscriptionResponseError:
+               return nil, &subscriptionPollError{Message: 
response.ErrorMessage, Critical: response.Critical}
+       case subscriptionResponseTermination:
+               return nil, nil
+       default:
+               return nil, fmt.Errorf("subscription: poll type %d cannot 
produce a record message", response.Type)
+       }
+}
+
+func sameSubscriptionCommitContext(left, right CommitContext) bool {
+       leftWriter, rightWriter := left.Writer, right.Writer
+       left.Writer, right.Writer = nil, nil
+       if left != right {
+               return false
+       }
+       if leftWriter == nil || rightWriter == nil {
+               return leftWriter == nil && rightWriter == nil
+       }
+       return *leftWriter == *rightWriter
+}
+
+// subscriptionPollError preserves the server payload's critical flag 
separately
+// from transport status errors so the consumer can apply the right retry 
policy.
+type subscriptionPollError struct {
+       Message string
+       Critical bool
+}
+
+func (failure *subscriptionPollError) Error() string {
+       return "subscription: poll payload: " + failure.Message
+}
+

Reply via email to