This is an automated email from the ASF dual-hosted git repository. shuwenwei pushed a commit to branch supports_subscription in repository https://gitbox.apache.org/repos/asf/iotdb-client-go.git
commit 37ef7f2dcf30ef8a04e3d907a5d04e8fe80ffaa1 Author: shuwenwei <[email protected]> AuthorDate: Fri Nov 13 10:21:55 2026 +0800 Decode subscription record messages and poll responses --- subscription/README.md | 18 ++- subscription/doc.go | 6 +- subscription/message_impl.go | 332 ++++++++++++++++++++++++++++++++++++++++++ subscription/provider.go | 8 +- subscription/tablet_decode.go | 226 ++++++++++++++++++++++++++++ subscription/wire_poll.go | 273 ++++++++++++++++++++++++++++++++++ 6 files changed, 854 insertions(+), 9 deletions(-) diff --git a/subscription/README.md b/subscription/README.md index 8a78bd0..af8a373 100644 --- a/subscription/README.md +++ b/subscription/README.md @@ -19,7 +19,7 @@ under the License. # Go subscription 公共接口 -当前已实现 `SubscriptionSession` 的默认配置、构造、普通 Thrift 连接与认证,以及 Topic/Subscription 管理;Consumer 默认配置、内部控制协议和单节点 Provider 也已补齐。公共 Consumer、记录解析和处理器仍没有可运行的消费实现。下文对应表同时列出已实现能力和后续消费实现需要遵守的契约。 +当前已实现 `SubscriptionSession` 的默认配置、构造、普通 Thrift 连接与认证,以及 Topic/Subscription 管理;Consumer 默认配置、内部控制协议和单节点 Provider、Poll 响应与记录解析、消息及记录访问实现也已补齐。公共 Consumer 和处理器仍没有可运行的消费实现。下文对应表同时列出已实现能力和后续消费实现需要遵守的契约。 模块路径为 `github.com/apache/iotdb-client-go/subscription`,使用 Go 1.25,与现有 `github.com/apache/iotdb-client-go/v2` Session 模块分别维护依赖和版本。当前仅依赖现有模块,以复用 `TSDataType` 和 `ColumnCategory`;没有 TsFile、cgo 或 TLS 依赖,也没有改动现有 Session 接口。 @@ -73,7 +73,19 @@ topics, err := s.Topics(ctx) - 帧限制取本地与服务端较小值并预留协议空间;旧服务端没有属性字段或不支持属性探测 RPC 时回退本地限制。可分片请求的所有片段保持同一连接、顺序和批次身份。 - 心跳按报文大小分批发送缓存上下文;提交预留响应空间分批,后续批次失败时仍返回先前已接受的上下文和进度。没有响应 body 时保留空结果,不推断整批提交成功。 - 状态码保留普通错误、owner fencing、consumer fencing、连接替换和致命错误的区别。Poll 返回消费组 Broker 丢失诊断时使旧连接失效,供后续 Consumer 重建;失效连接不会被旧响应重新启用。 -- 本轮只编码记录拉取请求,不解析其响应,也不实现文件拉取。TsFile 在解析时报错的契约保留,实际分派与记录解码在后续阶段接入。 +- Provider 现在解析全部非空 Poll 响应 body,保留记录、错误、终止和水位线事件。文件响应在类型分派处返回 `ErrUnsupportedTsFile`;混合响应不会返回已经解析的部分记录,不实现文件拉取。 + +## 记录消息与解析 + +记录数据采用 Java 当前依赖的 `2.5.1-261023-SNAPSHOT` Tablet 序列化格式,与写入 RPC 和查询 TsBlock 格式分开处理,不依赖 TsFile Go 或原生库。 + +- 支持 BOOLEAN、INT32、INT64、FLOAT、DOUBLE、TEXT、STRING、TIMESTAMP、DATE、BLOB 和 OBJECT,保留 TAG/FIELD/ATTRIBUTE 类别。TEXT/STRING 暴露为 string,BLOB/OBJECT 为 `[]byte`,DATE 为 UTC 零点的 `time.Time`。 +- NULL 由 Bitmap 和缺失值列决定。Binary 的缺省标记先恢复为空内容,再应用 Bitmap;不把空字符串或空 BLOB 自动变成 NULL。日期按 Java 的日期整数规则读取。 +- 原始行号用于 `Timestamp/Value`;`Rows()` 通过独立的稳定排序视图按时间递增遍历,相同时间戳保持原始顺序。行值切片不在迭代间复用,调用用户的迭代函数时不持内部锁。 +- 树模型始终显示时间戳;表模型优先采用按数据库、表指定的时间投影,未指定时回退消息级标志。旧响应缺少投影尾部时保持 Java 的默认选择时间行为。 +- 消息释放幂等并清理记录缓存;提交上下文及其可选 Writer 元数据仍以独立快照返回,水位线和消息类型继续可访问。已借用的批次在释放后数据访问报错。 +- 记录续取组装校验完整提交上下文,逐页合并数据和时间投影;最终数量按 Tablet 个数核对,不按行数核对。该组装能力尚未接入 Consumer 的实际续取循环。 +- 空记录交付和终止事件在解析层保留上下文,不自动 ACK 或退订;服务端 ERROR 保留严重级别,水位线可转为处理器消息。消费循环中的处理仍待下一步实现。 ## 管理 Session @@ -167,4 +179,4 @@ SubscriptionMessageProcessor 保留消息时必须实现 `BufferingProcessor`, 这些哨兵可以被后续操作错误包装,调用方使用 `errors.Is` 识别。TsFile 不支持错误不得触发连接重试或文件载荷确认。Pull 本轮返回错误而不交付任何消息,包括并行分支已经取得的记录;本轮记录不得推进位置、进入处理器或自动提交缓存,此前成功交付及提交的状态不回滚。调用方可退订相关 Topic 后继续拉取。 -后续步骤尚需实现记录及 Poll 响应解析、公共 Consumer 和构造、多节点协调、提交进度与恢复、后台任务、内置 Watermark/ColumnAlign Processor,以及消费示例和行为测试。内部 Provider 已有提交 RPC,但尚未实现公共 Consumer 的交付和确认链。管理查询复用根模块解码器,本轮不修改其对畸形数据的处理逻辑。当前仅完成静态核对,没有编译、运行测试或执行格式工具,不提供运行验证结论。 +后续步骤尚需实现公共 Consumer 和构造、实际记录续取循环、多节点协调、提交进度与恢复、后台任务、内置 Watermark/ColumnAlign Processor,以及消费示例和行为测试。内部 Provider 已有提交 RPC,但尚未实现公共 Consumer 的交付和确认链。管理查询复用根模块解码器,本轮不修改其对畸形数据的处理逻辑。当前仅完成静态核对,没有编译、运行测试或执行格式工具,不提供运行验证结论。 diff --git a/subscription/doc.go b/subscription/doc.go index 8f1dece..1864210 100644 --- a/subscription/doc.go +++ b/subscription/doc.go @@ -22,8 +22,10 @@ // // SubscriptionSession provides topic and subscription management through // NewSubscriptionSession. Consumer configuration defaults and internal control -// protocol/provider support are available. Public consumer implementations, -// record decoding, and built-in processors remain subsequent implementation steps. +// protocol/provider support are available. Poll response and record decoding, +// record views, message release, and watermark messages are implemented internally. +// Public consumers, background coordination, and built-in processors remain +// subsequent implementation steps. // // Record messages are the supported payload format. Topic management and // subscription do not reject TsFile topics; decoding a TsFile payload must return diff --git a/subscription/message_impl.go b/subscription/message_impl.go new file mode 100644 index 0000000..5c957c5 --- /dev/null +++ b/subscription/message_impl.go @@ -0,0 +1,332 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package subscription + +import ( + "fmt" + "iter" + "sort" + "sync" + "time" +) + +// subscriptionMessage keeps immutable delivery metadata separate from releasable +// record buffers. Its factories take ownership of decoded batches. +type subscriptionMessage struct { + mu sync.RWMutex + kind SubscriptionMessageKind + context CommitContext + watermark int64 + batches []RecordBatch + released bool +} + +func newSubscriptionRecordMessage(commitContext CommitContext, batches []RecordBatch) SubscriptionMessage { + return &subscriptionMessage{ + kind: RecordMessage, + context: cloneMessageCommitContext(commitContext), + batches: append([]RecordBatch(nil), batches...), + } +} + +func newSubscriptionWatermarkMessage(commitContext CommitContext, timestamp int64) SubscriptionMessage { + return &subscriptionMessage{ + kind: WatermarkMessage, + context: cloneMessageCommitContext(commitContext), + watermark: timestamp, + } +} + +func cloneMessageCommitContext(commitContext CommitContext) CommitContext { + if commitContext.Writer != nil { + writer := *commitContext.Writer + commitContext.Writer = &writer + } + return commitContext +} + +func (m *subscriptionMessage) Kind() SubscriptionMessageKind { + return m.kind +} + +func (m *subscriptionMessage) CommitContext() CommitContext { + return cloneMessageCommitContext(m.context) +} + +func (m *subscriptionMessage) Records() ([]RecordBatch, error) { + if m.kind != RecordMessage { + return nil, ErrIncompatibleMessage + } + m.mu.RLock() + defer m.mu.RUnlock() + if m.released { + return nil, ErrDataReleased + } + return append([]RecordBatch(nil), m.batches...), nil +} + +func (m *subscriptionMessage) Watermark() (int64, bool) { + return m.watermark, m.kind == WatermarkMessage +} + +func (m *subscriptionMessage) EstimatedSize() int64 { + m.mu.RLock() + defer m.mu.RUnlock() + // Include a rough object/metadata allowance and retained variable data. + // This estimate is intended for buffering limits, not heap accounting. + size := int64(128) + int64(len(m.kind)) + int64(len(m.context.Topic)) + + int64(len(m.context.GroupID)) + int64(len(m.context.RegionID)) + if m.context.Writer != nil { + size += 32 + int64(len(m.context.Writer.ID.RegionID)) + } + size += int64(len(m.batches)) * 16 + for _, batch := range m.batches { + if records, ok := batch.(*subscriptionRecordBatch); ok { + size += records.estimatedSize() + } + } + return size +} + +func (m *subscriptionMessage) Release() { + m.mu.Lock() + defer m.mu.Unlock() + if m.released { + return + } + m.released = true + for _, batch := range m.batches { + if records, ok := batch.(*subscriptionRecordBatch); ok { + records.release() + } + } + m.batches = nil +} + +// subscriptionRecordBatch preserves the decoded Tablet's indexed order and uses +// a separate stable permutation for user-visible iteration. Immutable target +// metadata remains available after release; Columns and Len return empty values. +type subscriptionRecordBatch struct { + mu sync.RWMutex + dialect Dialect + database string + target string + timeSelected bool + rowCount int + columns []Column + timestamps []int64 + values [][]any + rowOrder []int + released bool +} + +func newSubscriptionRecordBatch(dialect Dialect, database string, tablet decodedSubscriptionTablet, timeSelected bool) RecordBatch { + if dialect == TreeDialect { + database = "" + timeSelected = true + } + batch := &subscriptionRecordBatch{ + dialect: dialect, + database: database, + target: tablet.Target, + timeSelected: timeSelected, + rowCount: tablet.RowCount, + columns: tablet.Columns, + timestamps: tablet.Timestamps, + values: tablet.Values, + } + // A missing timestamp array represents zero for every row. Its ordering is + // already stable, so keep that representation sparse and iterate lazily. + if tablet.Timestamps != nil { + batch.rowOrder = make([]int, tablet.RowCount) + for index := range batch.rowOrder { + batch.rowOrder[index] = index + } + sort.SliceStable(batch.rowOrder, func(left, right int) bool { + return batch.timestamps[batch.rowOrder[left]] < batch.timestamps[batch.rowOrder[right]] + }) + } + return batch +} + +func (b *subscriptionRecordBatch) Dialect() Dialect { + return b.dialect +} + +func (b *subscriptionRecordBatch) Database() string { + return b.database +} + +func (b *subscriptionRecordBatch) Target() string { + return b.target +} + +func (b *subscriptionRecordBatch) Columns() []Column { + b.mu.RLock() + defer b.mu.RUnlock() + return b.columns +} + +func (b *subscriptionRecordBatch) TimeSelected() bool { + return b.timeSelected +} + +func (b *subscriptionRecordBatch) Len() int { + b.mu.RLock() + defer b.mu.RUnlock() + return b.rowCount +} + +func (b *subscriptionRecordBatch) Timestamp(row int) (int64, error) { + b.mu.RLock() + defer b.mu.RUnlock() + if b.released { + return 0, ErrDataReleased + } + if row < 0 || row >= b.rowCount { + return 0, fmt.Errorf("subscription: row index %d out of range", row) + } + if b.timestamps == nil { + return 0, nil + } + return b.timestamps[row], nil +} + +func (b *subscriptionRecordBatch) Value(row, column int) (any, error) { + b.mu.RLock() + defer b.mu.RUnlock() + if b.released { + return nil, ErrDataReleased + } + if row < 0 || row >= b.rowCount { + return nil, fmt.Errorf("subscription: row index %d out of range", row) + } + if column < 0 || column >= len(b.columns) { + return nil, fmt.Errorf("subscription: column index %d out of range", column) + } + return b.valueLocked(row, column) +} + +func (b *subscriptionRecordBatch) valueLocked(row, column int) (any, error) { + if b.values == nil { + return nil, nil + } + if column >= len(b.values) { + return nil, fmt.Errorf("subscription: record column %d has no value storage", column) + } + if b.values[column] == nil { + return nil, nil + } + if row >= len(b.values[column]) { + return nil, fmt.Errorf("subscription: record column %d has no value for row %d", column, row) + } + return b.values[column][row], nil +} + +func (b *subscriptionRecordBatch) Rows() iter.Seq2[Row, error] { + return func(yield func(Row, error) bool) { + for position := 0; ; position++ { + b.mu.RLock() + if b.released { + b.mu.RUnlock() + yield(Row{}, ErrDataReleased) + return + } + if position >= b.rowCount { + b.mu.RUnlock() + return + } + index := position + if b.rowOrder != nil { + index = b.rowOrder[position] + } + row := Row{Values: make([]any, len(b.columns))} + if b.timeSelected { + var timestamp int64 + if b.timestamps != nil { + timestamp = b.timestamps[index] + } + row.Timestamp = ×tamp + } + var err error + for column := range row.Values { + row.Values[column], err = b.valueLocked(index, column) + if err != nil { + break + } + } + b.mu.RUnlock() + // The caller may release the message or stop iteration from yield. + // Never hold a batch lock while invoking caller-controlled code. + if err != nil { + yield(Row{}, err) + return + } + if !yield(row, nil) { + return + } + } + } +} + +func (b *subscriptionRecordBatch) release() { + b.mu.Lock() + defer b.mu.Unlock() + if b.released { + return + } + b.released = true + b.rowCount = 0 + b.columns = nil + b.timestamps = nil + b.values = nil + b.rowOrder = nil +} + +func (b *subscriptionRecordBatch) estimatedSize() int64 { + b.mu.RLock() + defer b.mu.RUnlock() + size := int64(128) + int64(len(b.dialect)) + int64(len(b.database)) + + int64(len(b.target)) + int64(len(b.timestamps))*8 + int64(len(b.rowOrder))*8 + size += int64(len(b.columns))*32 + int64(len(b.values))*24 + for _, column := range b.columns { + size += int64(len(column.Name)) + } + for _, values := range b.values { + size += int64(len(values))*16 + for _, value := range values { + switch value := value.(type) { + case bool: + size++ + case int32, float32: + size += 4 + case int64, float64: + size += 8 + case string: + size += 16 + int64(len(value)) + case []byte: + size += 24 + int64(len(value)) + case time.Time: + size += 24 + } + } + } + return size +} diff --git a/subscription/provider.go b/subscription/provider.go index 472652a..238002d 100644 --- a/subscription/provider.go +++ b/subscription/provider.go @@ -337,7 +337,7 @@ func (p *subscriptionProvider) seek(ctx context.Context, topic string, kind int1 return err } -func (p *subscriptionProvider) poll(ctx context.Context, topics []string, timeoutMs, maxBytes int64, progress map[string]TopicProgress) ([][]byte, error) { +func (p *subscriptionProvider) poll(ctx context.Context, topics []string, timeoutMs, maxBytes int64, progress map[string]TopicProgress) ([]subscriptionPollResponse, error) { request, err := encodeSubscriptionPoll(topics, timeoutMs, maxBytes, progress) if err != nil { return nil, err @@ -345,7 +345,7 @@ func (p *subscriptionProvider) poll(ctx context.Context, topics []string, timeou return p.pollPayloads(ctx, request) } -func (p *subscriptionProvider) pollTablets(ctx context.Context, commitContext CommitContext, offset int32, timeoutMs, maxBytes int64) ([][]byte, error) { +func (p *subscriptionProvider) pollTablets(ctx context.Context, commitContext CommitContext, offset int32, timeoutMs, maxBytes int64) ([]subscriptionPollResponse, error) { request, err := encodeSubscriptionPollTablets(commitContext, offset, timeoutMs, maxBytes) if err != nil { return nil, err @@ -353,7 +353,7 @@ func (p *subscriptionProvider) pollTablets(ctx context.Context, commitContext Co return p.pollPayloads(ctx, request) } -func (p *subscriptionProvider) pollPayloads(ctx context.Context, request *rpc.TPipeSubscribeReq) ([][]byte, error) { +func (p *subscriptionProvider) pollPayloads(ctx context.Context, request *rpc.TPipeSubscribeReq) ([]subscriptionPollResponse, error) { response, err := p.call(ctx, request) if err != nil { var statusErr *subscriptionStatusError @@ -369,7 +369,7 @@ func (p *subscriptionProvider) pollPayloads(ctx context.Context, request *rpc.TP } return nil, err } - return response.Body, nil + return decodeSubscriptionPollResponses(response.Body) } // close preserves subscriptions during fenced/recovery teardown. A regular close diff --git a/subscription/tablet_decode.go b/subscription/tablet_decode.go new file mode 100644 index 0000000..4988b06 --- /dev/null +++ b/subscription/tablet_decode.go @@ -0,0 +1,226 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package subscription + +import ( + "fmt" + "io" + "math" + "time" + + "github.com/apache/iotdb-client-go/v2/client" +) + +// decodedSubscriptionTablet follows TsFile 2.5.1-261023-SNAPSHOT Tablet's stream +// format, independently of the Session insert protocol and TsBlock query format. +// Nil Timestamps represents RowCount zero timestamps. Each nil Values column +// represents RowCount null values without allocating storage for absent data. +type decodedSubscriptionTablet struct { + Target string + Columns []Column + RowCount int + Timestamps []int64 + Values [][]any +} + +func decodeSubscriptionTablet(reader *subscriptionWireReader) (decodedSubscriptionTablet, error) { + tablet := decodedSubscriptionTablet{Target: reader.readString()} + rowCount := reader.readInt32() + if reader.err != nil { + return decodedSubscriptionTablet{}, reader.err + } + if rowCount < 0 { + return decodedSubscriptionTablet{}, fmt.Errorf("subscription: invalid Tablet row count %d", rowCount) + } + tablet.RowCount = int(rowCount) + if reader.readBool() { + columnCount := reader.readCount(1, false) + for index := int32(0); index < columnCount && reader.err == nil; index++ { + hasSchema := reader.readBool() + if reader.err != nil { + return decodedSubscriptionTablet{}, reader.err + } + if !hasSchema { + return decodedSubscriptionTablet{}, fmt.Errorf("subscription: null Tablet schema at column %d is not supported by the record handler", index) + } + column := Column{Name: reader.readString(), DataType: client.TSDataType(reader.readByte())} + // Encoding and compression are metadata in this uncompressed array + // representation; they do not select a value decoder here. + reader.readByte() + reader.readByte() + properties := reader.readCount(8, false) + for property := int32(0); property < properties && reader.err == nil; property++ { + reader.readString() + reader.readString() + } + category := reader.readByte() + if reader.err != nil { + break + } + switch category { + case 0: + column.Category = client.TAG + case 1: + column.Category = client.FIELD + case 2: + column.Category = client.ATTRIBUTE + default: + return decodedSubscriptionTablet{}, fmt.Errorf("subscription: unsupported Tablet column category %d", category) + } + if _, err := subscriptionTabletValueSize(column.DataType); err != nil { + return decodedSubscriptionTablet{}, err + } + tablet.Columns = append(tablet.Columns, column) + } + } + if reader.err != nil { + return decodedSubscriptionTablet{}, reader.err + } + if reader.readBool() { + if err := requireSubscriptionTabletValues(reader, tablet.RowCount, 8); err != nil { + return decodedSubscriptionTablet{}, err + } + tablet.Timestamps = make([]int64, tablet.RowCount) + for row := range tablet.Timestamps { + tablet.Timestamps[row] = reader.readInt64() + } + } + bitmaps := make([][]byte, len(tablet.Columns)) + if reader.readBool() { + for column := range tablet.Columns { + if !reader.readBool() { + continue + } + size := reader.readInt32() + length := reader.readCount(1, false) + if reader.err != nil { + break + } + if size < rowCount || int64(length) < (int64(rowCount)+7)/8 { + return decodedSubscriptionTablet{}, fmt.Errorf("subscription: Tablet bitmap cannot represent %d rows", rowCount) + } + bitmaps[column] = reader.take(int(length)) + } + } + if reader.err != nil { + return decodedSubscriptionTablet{}, reader.err + } + tablet.Values = make([][]any, len(tablet.Columns)) + if reader.readBool() { + for column, schema := range tablet.Columns { + if !reader.readBool() { + continue + } + values, err := decodeSubscriptionTabletColumn(reader, schema.DataType, tablet.RowCount) + if err != nil { + return decodedSubscriptionTablet{}, err + } + for row := range values { + if bitmap := bitmaps[column]; bitmap != nil && bitmap[row/8]&(1<<uint(row%8)) != 0 { + values[row] = nil + } + } + tablet.Values[column] = values + } + } + if reader.err != nil { + return decodedSubscriptionTablet{}, reader.err + } + return tablet, nil +} + +// subscriptionTabletValueSize returns the minimum wire bytes per array entry. +// Binary-backed types use a presence byte before their optional length and data. +func subscriptionTabletValueSize(dataType client.TSDataType) (int, error) { + switch dataType { + case client.BOOLEAN, client.TEXT, client.STRING, client.BLOB, client.OBJECT: + return 1, nil + case client.INT32, client.FLOAT, client.DATE: + return 4, nil + case client.INT64, client.DOUBLE, client.TIMESTAMP: + return 8, nil + default: + return 0, fmt.Errorf("subscription: unsupported Tablet data type %d", dataType) + } +} + +func requireSubscriptionTabletValues(reader *subscriptionWireReader, count, size int) error { + if reader.err != nil { + return reader.err + } + if count > reader.remaining()/size { + reader.err = fmt.Errorf("subscription: Tablet array of %d entries exceeds remaining payload: %w", count, io.ErrUnexpectedEOF) + return reader.err + } + return nil +} + +func decodeSubscriptionTabletColumn(reader *subscriptionWireReader, dataType client.TSDataType, rows int) ([]any, error) { + size, err := subscriptionTabletValueSize(dataType) + if err != nil { + return nil, err + } + if err := requireSubscriptionTabletValues(reader, rows, size); err != nil { + return nil, err + } + values := make([]any, rows) + for row := range values { + switch dataType { + case client.BOOLEAN: + values[row] = reader.readBool() + case client.INT32: + values[row] = reader.readInt32() + case client.INT64, client.TIMESTAMP: + values[row] = reader.readInt64() + case client.FLOAT: + values[row] = math.Float32frombits(uint32(reader.readInt32())) + case client.DOUBLE: + values[row] = math.Float64frombits(uint64(reader.readInt64())) + case client.DATE: + date := reader.readInt32() + if reader.err != nil { + return nil, reader.err + } + year, month, day := int(date/10000), time.Month((date/100)%100), int(date%100) + value := time.Date(year, month, day, 0, 0, 0, 0, time.UTC) + if value.Year() != year || value.Month() != month || value.Day() != day { + return nil, fmt.Errorf("subscription: invalid Tablet DATE value %d", date) + } + values[row] = value + case client.TEXT, client.STRING, client.BLOB, client.OBJECT: + var binary []byte + if reader.readBool() { + length := reader.readCount(1, false) + binary = reader.take(int(length)) + } + if dataType == client.TEXT || dataType == client.STRING { + values[row] = string(binary) + } else { + // Type.deserializeArray uses Binary.EMPTY_VALUE for a missing + // Binary. SQL NULL is supplied separately by the Tablet bitmap. + values[row] = append([]byte{}, binary...) + } + } + if reader.err != nil { + return nil, reader.err + } + } + return values, nil +} diff --git a/subscription/wire_poll.go b/subscription/wire_poll.go new file mode 100644 index 0000000..4dbbc88 --- /dev/null +++ b/subscription/wire_poll.go @@ -0,0 +1,273 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package subscription + +import ( + "encoding/binary" + "fmt" + "strings" +) + +const ( + subscriptionResponseError int16 = 0 + subscriptionResponseTablets int16 = 1 + subscriptionResponseFileInit int16 = 2 + subscriptionResponseFilePiece int16 = 3 + subscriptionResponseFileSeal int16 = 4 + subscriptionResponseTermination int16 = 5 + subscriptionResponseWatermark int16 = 7 +) + +type subscriptionPollBatch struct { + Database string + Tablet decodedSubscriptionTablet +} + +// subscriptionPollResponse retains transport events separately from deliverable +// messages. In particular, termination and empty record deliveries still need +// consumer-level unsubscribe or confirmation; decoding does neither. +type subscriptionPollResponse struct { + Type int16 + CommitContext CommitContext + Records []subscriptionPollBatch + NextOffset int32 + TimeSelected bool + TimeSelectedByTable map[string]map[string]bool + Watermark int64 + DataNodeID int32 + ErrorMessage string + Critical bool +} + +// decodeSubscriptionPollResponses parses every nonempty body entry. No partially +// decoded response is returned if any entry fails, including an unsupported file +// entry that follows otherwise valid records. +func decodeSubscriptionPollResponses(bodies [][]byte) ([]subscriptionPollResponse, error) { + // Reject files before decoding records in sibling entries. This examines the + // response tag only; it does not parse file payloads or attempt file I/O. + for _, body := range bodies { + if len(body) < 2 { + continue + } + kind := int16(binary.BigEndian.Uint16(body[:2])) + if isSubscriptionFileResponse(kind) { + return nil, fmt.Errorf("subscription: decode poll response type %d: %w", kind, ErrUnsupportedTsFile) + } + } + var responses []subscriptionPollResponse + for index, body := range bodies { + if len(body) == 0 { + continue + } + response, err := decodeSubscriptionPollResponse(body) + if err != nil { + return nil, fmt.Errorf("subscription: decode poll body %d: %w", index, err) + } + responses = append(responses, response) + } + return responses, nil +} + +func decodeSubscriptionPollResponse(body []byte) (subscriptionPollResponse, error) { + reader := newSubscriptionWireReader(body) + response := subscriptionPollResponse{Type: reader.readInt16(), TimeSelected: true} + if reader.err != nil { + return subscriptionPollResponse{}, reader.err + } + switch response.Type { + case subscriptionResponseTablets: + databaseCount := reader.readCount(8, false) + for databaseIndex := int32(0); databaseIndex < databaseCount && reader.err == nil; databaseIndex++ { + database := reader.readString() + tabletCount := reader.readCount(12, false) + for tabletIndex := int32(0); tabletIndex < tabletCount && reader.err == nil; tabletIndex++ { + tablet, err := decodeSubscriptionTablet(reader) + if err != nil { + return subscriptionPollResponse{}, err + } + response.Records = append(response.Records, subscriptionPollBatch{Database: database, Tablet: tablet}) + } + } + response.NextOffset = reader.readInt32() + case subscriptionResponseError: + response.ErrorMessage = reader.readString() + response.Critical = reader.readBool() + case subscriptionResponseTermination: + // The termination payload is empty; its topic comes from the context. + case subscriptionResponseWatermark: + response.Watermark = reader.readInt64() + response.DataNodeID = reader.readInt32() + case subscriptionResponseFileInit, subscriptionResponseFilePiece, subscriptionResponseFileSeal: + return subscriptionPollResponse{}, fmt.Errorf("subscription: decode poll response type %d: %w", response.Type, ErrUnsupportedTsFile) + default: + return subscriptionPollResponse{}, fmt.Errorf("subscription: unsupported poll response type %d", response.Type) + } + if reader.err != nil { + return subscriptionPollResponse{}, reader.err + } + response.CommitContext = readSubscriptionCommitContext(reader) + // Older responses end after the context or after the global projection flag. + if reader.remaining() > 0 { + response.TimeSelected = reader.readBool() + } + if reader.remaining() > 0 { + response.TimeSelectedByTable = readSubscriptionTimeSelection(reader) + } + if reader.err != nil { + return subscriptionPollResponse{}, reader.err + } + if reader.remaining() != 0 { + return subscriptionPollResponse{}, fmt.Errorf("subscription: poll response has %d trailing bytes", reader.remaining()) + } + return response, nil +} + +func isSubscriptionFileResponse(kind int16) bool { + return kind == subscriptionResponseFileInit || kind == subscriptionResponseFilePiece || kind == subscriptionResponseFileSeal +} + +func readSubscriptionTimeSelection(reader *subscriptionWireReader) map[string]map[string]bool { + count := reader.readCount(8, false) + selection := make(map[string]map[string]bool) + for databaseIndex := int32(0); databaseIndex < count && reader.err == nil; databaseIndex++ { + database, hasDatabase := readSubscriptionProjectionKey(reader) + tableCount := reader.readCount(5, false) + tables := make(map[string]bool) + for tableIndex := int32(0); tableIndex < tableCount && reader.err == nil; tableIndex++ { + table, hasTable := readSubscriptionProjectionKey(reader) + selected := reader.readBool() + if hasTable { + tables[strings.ToLower(strings.TrimSpace(table))] = selected + } + } + if hasDatabase && len(tables) != 0 { + selection[strings.ToLower(strings.TrimSpace(database))] = tables + } + } + return selection +} + +// Projection maps discard null keys but preserve empty keys, unlike ordinary +// Go string protocol fields where null and empty both map to the zero value. +func readSubscriptionProjectionKey(reader *subscriptionWireReader) (string, bool) { + length := reader.readInt32() + if reader.err != nil || length == -1 { + return "", false + } + if length < -1 { + reader.err = fmt.Errorf("subscription: invalid projection key length %d", length) + return "", false + } + key := reader.take(int(length)) + return string(key), reader.err == nil +} + +// appendTablets merges a continuation page only after validating its delivery +// identity. Global time projection is ANDed; table overrides use the latest page. +func (response *subscriptionPollResponse) appendTablets(next subscriptionPollResponse) error { + if response.Type != subscriptionResponseTablets || response.NextOffset <= 0 { + return fmt.Errorf("subscription: record response does not need a continuation") + } + if next.Type != subscriptionResponseTablets { + return fmt.Errorf("subscription: expected tablet continuation, received type %d", next.Type) + } + if !sameSubscriptionCommitContext(response.CommitContext, next.CommitContext) { + return fmt.Errorf("subscription: tablet continuation has a different commit context") + } + response.Records = append(response.Records, next.Records...) + response.NextOffset = next.NextOffset + response.TimeSelected = response.TimeSelected && next.TimeSelected + if response.TimeSelectedByTable == nil { + response.TimeSelectedByTable = make(map[string]map[string]bool) + } + for database, tables := range next.TimeSelectedByTable { + if response.TimeSelectedByTable[database] == nil { + response.TimeSelectedByTable[database] = make(map[string]bool) + } + for table, selected := range tables { + response.TimeSelectedByTable[database][table] = selected + } + } + return nil +} + +// message creates a complete record view or a processor watermark. A complete +// zero-tablet delivery returns nil; its context remains available for the future +// consumer's empty-delivery ACK. No protocol state changes occur here. +func (response *subscriptionPollResponse) message(dialect Dialect) (SubscriptionMessage, error) { + if dialect != TreeDialect && dialect != TableDialect { + return nil, fmt.Errorf("subscription: unsupported dialect %q", dialect) + } + switch response.Type { + case subscriptionResponseTablets: + if response.NextOffset > 0 { + return nil, fmt.Errorf("subscription: record message requires tablet continuation at offset %d", response.NextOffset) + } + if int64(len(response.Records)) != -int64(response.NextOffset) { + return nil, fmt.Errorf("subscription: tablet count %d does not match final offset %d", len(response.Records), response.NextOffset) + } + if len(response.Records) == 0 { + return nil, nil + } + batches := make([]RecordBatch, 0, len(response.Records)) + for _, record := range response.Records { + selected := response.TimeSelected + if tables := response.TimeSelectedByTable[strings.ToLower(strings.TrimSpace(record.Database))]; tables != nil { + if value, ok := tables[strings.ToLower(strings.TrimSpace(record.Tablet.Target))]; ok { + selected = value + } + } + batches = append(batches, newSubscriptionRecordBatch(dialect, record.Database, record.Tablet, selected)) + } + return newSubscriptionRecordMessage(response.CommitContext, batches), nil + case subscriptionResponseWatermark: + return newSubscriptionWatermarkMessage(response.CommitContext, response.Watermark), nil + case subscriptionResponseError: + return nil, &subscriptionPollError{Message: response.ErrorMessage, Critical: response.Critical} + case subscriptionResponseTermination: + return nil, nil + default: + return nil, fmt.Errorf("subscription: poll type %d cannot produce a record message", response.Type) + } +} + +func sameSubscriptionCommitContext(left, right CommitContext) bool { + leftWriter, rightWriter := left.Writer, right.Writer + left.Writer, right.Writer = nil, nil + if left != right { + return false + } + if leftWriter == nil || rightWriter == nil { + return leftWriter == nil && rightWriter == nil + } + return *leftWriter == *rightWriter +} + +// subscriptionPollError preserves the server payload's critical flag separately +// from transport status errors so the consumer can apply the right retry policy. +type subscriptionPollError struct { + Message string + Critical bool +} + +func (failure *subscriptionPollError) Error() string { + return "subscription: poll payload: " + failure.Message +} +
