This is an automated email from the ASF dual-hosted git repository. shuwenwei pushed a commit to branch supports_subscription in repository https://gitbox.apache.org/repos/asf/iotdb-client-go.git
commit aea1a00f63d69c47283aee842de8bb48f03424da Author: shuwenwei <[email protected]> AuthorDate: Thu Nov 12 18:11:40 2026 +0800 Add standalone subscription module API contracts --- subscription/LICENSE | 207 ++++++++++++++++++++++++++++++++++++++++++++++ subscription/NOTICE | 19 +++++ subscription/README.md | 140 +++++++++++++++++++++++++++++++ subscription/config.go | 137 ++++++++++++++++++++++++++++++ subscription/consumer.go | 153 ++++++++++++++++++++++++++++++++++ subscription/doc.go | 35 ++++++++ subscription/errors.go | 51 ++++++++++++ subscription/go.mod | 7 ++ subscription/go.sum | 4 + subscription/message.go | 137 ++++++++++++++++++++++++++++++ subscription/processor.go | 88 ++++++++++++++++++++ subscription/progress.go | 114 +++++++++++++++++++++++++ subscription/session.go | 130 +++++++++++++++++++++++++++++ 13 files changed, 1222 insertions(+) diff --git a/subscription/LICENSE b/subscription/LICENSE new file mode 100644 index 0000000..ef4019b --- /dev/null +++ b/subscription/LICENSE @@ -0,0 +1,207 @@ + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + +-------------------------------------------------------------------------------- + APACHE IOTDB SUBCOMPONENTS +-------------------------------------------------------------------------------- + + diff --git a/subscription/NOTICE b/subscription/NOTICE new file mode 100644 index 0000000..b4df141 --- /dev/null +++ b/subscription/NOTICE @@ -0,0 +1,19 @@ +Apache IoTDB +Copyright 2018-2020 The Apache Software Foundation. + +This product includes software developed at +The Apache Software Foundation (http://www.apache.org/). + +============================================================================ + +IoTDB project uses 4 Chinese Patents: +* 201711384490X +* 201810111712.9 +* 201711322631.5 +* 201711319331.1 + +According to the Apache 2.0 License. The owner of the patents, Tsinghua University, +grant the users the right to the use of patent under the requirement of Apache 2.0 License. + +============================================================================ + diff --git a/subscription/README.md b/subscription/README.md new file mode 100644 index 0000000..86c392d --- /dev/null +++ b/subscription/README.md @@ -0,0 +1,140 @@ +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +# Go subscription 公共接口 + +当前模块只定义公共接口、配置和数据类型,以及可用 `errors.Is` 判断的错误哨兵。尚无构造函数、连接、消费者或消息解析实现,不能据此创建可运行的订阅客户端。下面描述的是后续实现必须遵守的接口契约,不代表这些行为已经实现。 + +模块路径为 `github.com/apache/iotdb-client-go/subscription`,使用 Go 1.25,与现有 `github.com/apache/iotdb-client-go/v2` Session 模块分别维护依赖和版本。当前仅依赖现有模块,以复用 `TSDataType` 和 `ColumnCategory`;没有 TsFile、cgo 或 TLS 依赖,也没有改动现有 Session 接口。 + +Java 对照基线为 IoTDB `d9122d837610` 的 `iotdb-client/subscription`。本次支持范围是记录消息:不定义 TsFile handler、下载、文件操作或 Reader API。Topic 管理和 `Subscribe` 不预先拒绝 TsFile Topic;后续解析分派到文件载荷时必须返回 `ErrUnsupportedTsFile`,不能交付或确认该载荷。 + +## 配置与模型选择 + +Go 使用 `Dialect` 区分树、表模型,管理和消费接口各自共用一份契约,避免复制 Java 的树、表类型及 Builder 层次。 + +Java 的 `ISubscriptionTreeSession/ISubscriptionTableSession`、树/表 Pull 接口、树/表 Push 接口各自具有相同的方法签名。因此 Go 分别用 `SubscriptionSession`、`SubscriptionPullConsumer`、`SubscriptionPushConsumer` 表达这些共同操作,由配置中的 `Dialect` 选择模型;每个实例对应一个模型。 + +这里共用的是对外方法契约。后续连接、握手和管理 SQL 必须传递对应模型,记录解析仍需分别处理树模型的设备路径与表模型的数据库、表和 TAG 身份。相同接口不要求树、表使用同一套内部实现。当前仅有接口声明,尚未实现上述模型选择逻辑。 + +| Go 公共类型 | Java 对应 | Go 中的表达方式 | +| --- | --- | --- | +| `Dialect`、`TreeDialect`、`TableDialect` | 树、表 Session/Consumer 类型及 SQL dialect | 在 `SubscriptionSessionConfig` 或 `SubscriptionConsumerConfig` 中选择模型 | +| `ConnectionConfig` | Session/Consumer Builder 的节点、认证、连接超时、最大帧配置 | `NodeURLs` 同时表示单节点和集群种子地址;时间使用 `time.Duration`;明文与加密密码互斥 | +| `SubscriptionSessionConfig` | `SubscriptionTreeSessionBuilder`、`SubscriptionTableSessionBuilder` | 组合连接配置和 `Dialect`,本步不提供 Builder 或构造函数 | +| `SubscriptionConsumerConfig` | `AbstractSubscriptionConsumerBuilder` | 公共消费配置,包含消费身份、owner、心跳、端点同步及并行度 | +| `SubscriptionPullConsumerConfig` | `AbstractSubscriptionPullConsumerBuilder` | `AutoCommit == nil` 表示沿用 Java 默认值 `true`,指向 `false` 表示显式手动提交 | +| `SubscriptionPushConsumerConfig` | `AbstractSubscriptionPushConsumerBuilder` | 组合 Handler、ACK 策略及拉取间隔、超时 | +| `AckStrategy`、`AckAfterConsume`、`AckBeforeConsume` | `AckStrategy.AFTER_CONSUME`、`BEFORE_CONSUME` | 零值为消费后确认;消费前确认下,回调失败不导致重新投递 | + +`SubscriptionConsumerConfig.OwnerEpoch == nil` 表示未提供 epoch,指向零表示显式 epoch 0。配置的运行时默认值和校验会在构造函数实现阶段补齐,本步不执行配置校验。 + +## 管理 Session + +`SubscriptionSession` 对应 Java 的 `ISubscriptionTreeSession` 和 `ISubscriptionTableSession`。方法接收显式 `context.Context`,存在性条件通过参数结构表达。 + +| Go 方法或类型 | Java 对应 | 合并方式 | +| --- | --- | --- | +| `Open(ctx)`、`Close()` | `open()`、`AutoCloseable.close()` | 打开操作可取消;关闭统一负责释放资源 | +| `CreateTopic(ctx, name, options)`、`CreateTopicOptions` | 四个 `createTopic/createTopicIfNotExists` 重载 | `Attributes` 表示可选属性,`IfNotExists` 表示条件创建 | +| `AlterTopic(ctx, name, attributes)` | `alterTopic(name, properties)` | 属性用 `map[string]string` | +| `AlterTopicOwner(ctx, name, owner)`、`TopicOwner` | 两个 `alterTopicOwner` 重载 | `LeaseDuration` 指针表达可选租约时长 | +| `DropTopic(ctx, name, options)`、`DropOptions` | `dropTopic/dropTopicIfExists` | `IfExists` 表示条件删除 | +| `Topics(ctx)`、`Topic(ctx, name)` | `getTopics()`、`getTopic(name)` | 集合用 slice;单 Topic 不存在时返回 `found == false` | +| `Subscriptions(ctx, topic)` | `getSubscriptions()`、`getSubscriptions(topic)` | `topic == ""` 表示查询全部 | +| `DropSubscription(ctx, id, options)` | `dropSubscription/dropSubscriptionIfExists` | 复用 `DropOptions` | +| `Topic` | `model.Topic` | 保留服务端返回的属性原始文本,不假定其可直接解析为 map | +| `Subscription` | `model.Subscription` | 保留服务端的 `ConsumerIDs` 原始文本,不假定其是 CSV | + +## Consumer、提交与进度 + +`SubscriptionConsumer` 是 Pull 和 Push 的共同接口。它包含 `Positioner`,因此两类消费者共用进度与 seek 契约;这也覆盖 Java 具体 Consumer 从基类继承的公开能力。 + +| Go 方法或类型 | Java 对应 | 合并方式 | +| --- | --- | --- | +| `SubscriptionConsumer.Open(ctx)`、`Close()` | 各 Consumer 的 `open/close` | 共用生命周期接口 | +| `Subscribe(ctx, topics...)`、`Unsubscribe(ctx, topics...)` | 单个、数组、集合重载 | 统一为 variadic Topic 名称 | +| `Info()`、`SubscriptionConsumerInfo` | `getConsumerId/getConsumerGroupId/getOwnerId/getOwnerEpoch` | 一次返回不含凭据的身份快照,包含服务端分配的 ID | +| `LatestWatermark()` | `getLatestWatermarkTimestamp()` | 额外返回 `ok`,区分尚无水位线与有效时间值 | +| `AllConsumed()` | `allTopicMessagesHaveBeenConsumed()` | 无订阅时为 true;活跃的无界 Topic 阻止完成 | +| `SubscriptionPullConsumer.Poll(ctx, timeout, topics...)` | 四个 `poll` 重载 | `time.Duration` 替代毫秒和 Java Duration 重载;不传 Topic 表示当前全部订阅 | +| `PollWithInfo(ctx, timeout, topics...)`、`PollResult` | 具体 Pull 类的 `pollWithInfo`、`payload.PollResult` | 一并返回消息、缓存条数及处理器水位线 | +| `DrainBufferedMessages(ctx)` | `drainBufferedMessages()` | 显式排空处理器或待交付缓存 | +| `Commit(ctx, messages...)` | 单条/批量 `commitSync` | 同步提交统一为 variadic 消息 | +| `CommitAsync(ctx, messages...)` | `CompletableFuture` 及 `AsyncCommitCallback` 重载 | 返回 `<-chan error`;调用方可直接等待,或用 goroutine 等待后执行回调 | +| `AddProcessor(processor)` | 具体 Pull 类的 `addProcessor` | 按调用顺序添加处理阶段,返回错误而非链式 Builder | +| `ConsumeFunc`、`SubscriptionPushConsumerConfig.Handler` | `ConsumeListener.onReceive`、`ConsumeResult` | `func(ctx, message) error`,nil 表示成功,非 nil 表示消费失败 | +| `SubscriptionPushConsumer.Errors()` | Java 自动拉取异常路径的 Go 显式错误出口 | 首次终止性后台错误通知一次;普通 Handler 失败不视为终止错误 | +| `Positioner.Positions(topic)` | `positions(topic)` | 返回当前已拉取位置的深快照 | +| `CommittedPositions(topic)` | `committedPositions(topic)` | 单独保留已确认、可恢复的位置 | +| `SeekToBeginning(ctx, topic)`、`SeekToEnd(ctx, topic)` | 同名 Java 方法 | 保留两种定位行为 | +| `Seek(ctx, topic, progress)`、`SeekAfter(ctx, topic, progress)` | `seek/seekAfter` | 分别从指定进度重放和从其后继续 | +| `TopicProgress`、`RegionProgress` | 同名 Java 类 | Topic → Region → Writer 的 map,不合并成单 offset | +| `WriterID`、`WriterProgress` | `WriterId`、`WriterProgress` | 可比较 Writer ID 作为 map key;进度保留物理时间和本地序号 | +| `WriterPosition`、`CommitContext`、`InvalidCommitID` | `SubscriptionCommitContext` 的可选 writer 进度及旧式 commit 元数据 | writer 身份与进度成对保留,同时保留路由、重启和 seek-generation 信息 | + +`CommitAsync` 的返回 channel 容量为 1,只发送一次结果(成功时为 nil)后关闭。调用方不读取结果不能阻塞清理。取消 context 不撤回服务端已接受的提交;同步或异步提交报错也不代表整批都失败。 + +`SubscriptionPushConsumer.Errors()` 的 channel 至少缓冲一个错误。TsFile 解析错误必须先停止自动拉取再报告,失败轮次不能进入 Handler 或 ACK;channel 关闭由统一生命周期管理,避免与后台发送竞争。 + +## 消息、记录与 Processor + +| Go 方法或类型 | Java 对应 | Go 中的表达方式 | +| --- | --- | --- | +| `SubscriptionMessage`、`SubscriptionMessageKind`、`RecordMessage`、`WatermarkMessage` | `SubscriptionMessage`、记录和水位线消息类型 | 公共消息种类与协议数字解耦;Pull 和 Push 只交付记录,Processor 可接收水位线 | +| `SubscriptionMessage.Kind()`、`CommitContext()` | `getMessageType/getCommitContext` | 方法返回公开种类和提交上下文快照 | +| `SubscriptionMessage.Records()` | `getResultSets/getRecordTabletIterator`、`SubscriptionRecordHandler` | 返回多数据库、多 Tablet 的 `[]RecordBatch`,不复制 Java ResultSet 继承层次 | +| `SubscriptionMessage.Watermark()` | `getWatermarkTimestamp()` | 用 `ok` 标识是否为水位线消息 | +| `EstimatedSize()`、`Release()` | `estimateSize/removeUserData` | 释放记录缓存后保留消息类型与提交身份,仍可提交 | +| `RecordBatch` | `SubscriptionRecordHandler.SubscriptionResultSet` 和底层 Tablet 视图 | 提供模型、数据库、目标、列信息、时间投影、行数和按索引访问 | +| `Column` | measurement schema、列类型与类别 | 复用现有 `client.TSDataType/ColumnCategory`;时间列单独表示 | +| `RecordBatch.Rows()`、`Row` | `ResultSet`、`RowRecord`、`TSRecord` 的遍历视图 | 使用 `iter.Seq2[Row, error]`;索引从 0 开始 | +| `RecordBatch.TimeSelected()`、`Row.Timestamp` | 消息和表级时间列选择信息 | 每个批次独立保留投影;未选择时间列时 `Row.Timestamp == nil` | +| `RecordBatch.Timestamp(row)` | 底层 Tablet 的时间戳访问 | 即使不投影时间列,处理器仍可读取原始时间戳 | +| `SubscriptionMessageProcessor.Process/Flush/Reset` | `SubscriptionMessageProcessor` 的同名能力 | 处理和排空显式返回 error;consumer 串行调用每个 Processor | +| `BufferingProcessor.Buffered()`、`ProcessorBuffer` | `getBufferedCount/getBufferedCommitContexts` | 缓存能力用可选接口表达,同时报告条数和全部可续租上下文 | +| `TopicResetter.ResetTopic(topic)` | `supportsTopicScopedReset/reset(topic)` | 是否实现接口代替 Java 的能力布尔值 | +| `WatermarkReporter.Watermark()` | `WatermarkProcessor.getWatermark()` | 可选暴露处理器计算的排序水位线;不等同于 Consumer 最新服务端水位线 | + +SubscriptionMessageProcessor 保留消息时必须实现 `BufferingProcessor`,让后续心跳实现能够续租未交付记录。seek 只清理目标 Topic;同时订阅多个 Topic 时,未实现 `TopicResetter` 的 Processor 会使 seek 在修改服务端位置前失败;单 Topic 场景允许回退到 `Reset()`。`Flush` 返回的消息经后续处理阶段继续处理,Processor 不得再保留或修改已移交的消息。 + +## 数据所有权与 context 契约 + +- Session 查询集合、Consumer 身份、提交上下文和进度返回调用方拥有的快照;进度 map 为深快照。seek 复制输入,不长期借用调用方的 map。 +- `SubscriptionMessage.Records()` 的批次、列和行数据是借用的只读视图,有效期到所属消息 `Release()`。需要跨越该有效期保留数据时,调用方负责复制值和字节数组。 +- `Release()` 幂等。释放后不再使用已借用的批次、列和行数据;`Records()` 返回 `ErrDataReleased`,消息类型、提交上下文和水位线仍可访问。 +- `Row.Values` 的 nil 表示 SQL NULL;DATE 使用 `time.Time`;TIMESTAMP 和记录时间戳使用服务端精度的 `int64`;BLOB/OBJECT 使用 `[]byte`。 +- Push Handler 不得释放收到的消息;回调返回后若仍需记录数据,必须先复制。回调 context 随消费交付停止而取消。 +- `Open(ctx)` 的 context 只限制打开操作,不持有后台任务生命周期;后台任务随 `Close()` 停止。其他操作 context 仅限制对应操作。 +- Pull 超时受 `timeout` 和 context deadline 中较早者约束。操作超时使用 `time.Duration`,不改变消息中的时间戳单位。 +- 已打开时重复 `Open`、已关闭时重复 `Close` 无副作用。成功关闭后不能重新打开;要求先排空缓存的关闭失败仍保留排空和提交的恢复路径。 + +## 错误与当前未实现内容 + +| Go 错误哨兵 | Java 对应或行为 | +| --- | --- | +| `ErrUnsupportedTsFile` | 本轮明确排除的 TsFile handler;仅解析文件载荷时返回,管理及订阅不提前拒绝 | +| `ErrClosed`、`ErrNotOpen` | Consumer 生命周期检查 | +| `ErrDataReleased` | 记录数据已释放后的访问 | +| `ErrIncompatibleMessage` | `SubscriptionIncompatibleHandlerException` 所表达的消息访问不匹配 | +| `ErrConsumerFenced`、`ErrOwnerFenced` | `SubscriptionConsumerFencedException`、`SubscriptionOwnerFencedException` | + +这些哨兵可以被后续操作错误包装,调用方使用 `errors.Is` 识别。TsFile 不支持错误不得触发连接重试或文件载荷确认。Pull 本轮返回错误而不交付任何消息,包括并行分支已经取得的记录;本轮记录不得推进位置、进入处理器或自动提交缓存,此前成功交付及提交的状态不回滚。调用方可退订相关 Topic 后继续拉取。 + +后续步骤尚需实现构造函数与默认配置、配置校验、连接和协议编解码、管理 SQL、记录解析、Consumer、提交和恢复、后台任务、内置 Watermark/ColumnAlign Processor,以及示例和行为测试。本步没有编译、运行测试或执行格式工具,不提供运行验证结论。 diff --git a/subscription/config.go b/subscription/config.go new file mode 100644 index 0000000..f74cbea --- /dev/null +++ b/subscription/config.go @@ -0,0 +1,137 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package subscription + +import "time" + +// Dialect selects one data model for an instance. A shared subscription interface +// does not imply that tree and table connections or record decoding are identical. +type Dialect string + +const ( + // TreeDialect selects tree-model connections and record interpretation. + TreeDialect Dialect = "tree" + // TableDialect selects table-model connections and record interpretation. + TableDialect Dialect = "table" +) + +// ConnectionConfig describes the RPC connections owned by a subscription client. +// Password and EncryptedPassword are mutually exclusive. NodeURLs contains +// host:port addresses and supports a single node as well as a cluster seed list. +type ConnectionConfig struct { + // NodeURLs lists host:port endpoints to use when establishing connections. + // A single entry selects a single seed; multiple entries provide alternatives. + NodeURLs []string + // Username is the IoTDB user used for session and subscription authentication. + Username string + // Password supplies the plaintext password and must not be combined with + // EncryptedPassword. + Password string + // EncryptedPassword supplies an already encrypted password and must not be + // combined with Password. + EncryptedPassword string + // ConnectTimeout limits connection establishment. The Java consumer default + // is 10 seconds; default resolution belongs to the future constructors. + ConnectTimeout time.Duration + // MaxFrameSize is the maximum accepted Thrift frame size in bytes. + MaxFrameSize int32 +} + +// SubscriptionSessionConfig configures a topic and subscription management session. +type SubscriptionSessionConfig struct { + // Connection supplies the endpoint, authentication, and transport settings. + Connection ConnectionConfig + // Dialect selects the model used by the management session's connection and SQL. + Dialect Dialect +} + +// SubscriptionConsumerConfig holds settings shared by pull and push consumers. Empty consumer +// and group IDs request server-assigned identities. A nil OwnerEpoch means that +// no owner epoch was supplied; a pointer to zero is an explicit epoch. +type SubscriptionConsumerConfig struct { + // Connection supplies the seed endpoints, authentication, and transport settings. + Connection ConnectionConfig + // Dialect selects the model used by the consumer's connections and records. + Dialect Dialect + // ConsumerID identifies this consumer. Empty requests an assigned identity. + ConsumerID string + // GroupID identifies the consumer group. Empty requests an assigned group ID. + GroupID string + // OwnerID supplies the identity used when a topic requires ownership fencing. + OwnerID string + // OwnerEpoch supplies the owner's epoch. Nil omits it; a pointer to zero + // explicitly supplies epoch zero. + OwnerEpoch *int64 + // HeartbeatInterval controls provider heartbeat frequency. Java consumers + // default to 30 seconds and enforce a minimum of one second. + HeartbeatInterval time.Duration + // EndpointsSyncInterval controls how often the consumer refreshes cluster + // endpoints. Java consumers default to two minutes and enforce a five-second + // minimum. + EndpointsSyncInterval time.Duration + // MaxPollParallelism limits parallel polling work. The Java default and + // minimum are both one, allowing polling to use a separate worker when + // resources are available so its timeout can be enforced. + MaxPollParallelism int +} + +// SubscriptionPullConsumerConfig adds automatic commit settings. A nil AutoCommit selects the Java +// default, true; an explicit false selects manual commit. Timing defaults will +// follow the Java consumer defaults when constructors are implemented. +type SubscriptionPullConsumerConfig struct { + // Consumer contains the shared connection, identity, and polling settings. + Consumer SubscriptionConsumerConfig + // AutoCommit selects periodic acknowledgment of delivered messages. Nil + // selects true; an explicit false leaves acknowledgment to the caller. + AutoCommit *bool + // AutoCommitInterval controls periodic acknowledgment when AutoCommit is + // enabled. The Java default is five seconds, with a 500-millisecond minimum. + AutoCommitInterval time.Duration +} + +// AckStrategy controls when push messages are acknowledged. +type AckStrategy uint8 + +const ( + // AckAfterConsume acknowledges successful callbacks and nacks failed ones. + AckAfterConsume AckStrategy = iota + // AckBeforeConsume acknowledges before invoking callbacks; callback failure + // does not cause redelivery under this strategy. + AckBeforeConsume +) + +// SubscriptionPushConsumerConfig configures background polling and delivery. Handler is required. +// The zero AckStrategy is AckAfterConsume. +type SubscriptionPushConsumerConfig struct { + // Consumer contains the shared connection, identity, and polling settings. + Consumer SubscriptionConsumerConfig + // AckStrategy selects acknowledgment before delivery or after successful + // processing. Its zero value is AckAfterConsume. + AckStrategy AckStrategy + // Handler receives record messages. It is required; nil returned by a call + // means successful processing, and a non-nil error means processing failed. + Handler ConsumeFunc + // PollInterval controls the interval between automatic polling work. Java + // defaults to 100 milliseconds and requires an interval greater than zero. + PollInterval time.Duration + // PollTimeout limits one automatic poll. Java defaults to ten seconds and + // enforces a one-second minimum. Message timestamps retain server precision. + PollTimeout time.Duration +} diff --git a/subscription/consumer.go b/subscription/consumer.go new file mode 100644 index 0000000..1473026 --- /dev/null +++ b/subscription/consumer.go @@ -0,0 +1,153 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package subscription + +import ( + "context" + "time" +) + +// SubscriptionConsumer is the shared lifecycle, subscription, and positioning contract. +// Repeated Open while open and repeated Close are harmless. A successfully +// closed consumer cannot be reopened. A failed Close that requires buffered +// messages to be drained leaves that recovery path available. +type SubscriptionConsumer interface { + Positioner + + // Open establishes the consumer's connections and starts its background + // work. It returns an error if opening fails. The context limits this + // operation only; canceling it after Open returns does not stop consumption. + Open(ctx context.Context) error + // Close stops background work and releases connections. In manual-commit + // mode, buffered processor output must first be drained, processed, and + // committed before retrying a close that reports undelivered messages. + // Closing does not guarantee that every outstanding commit succeeded. + Close() error + // Subscribe establishes subscriptions to one or more named topics. It + // returns an error if the subscriptions cannot be established. TsFile topic + // configuration is accepted; unsupported payloads fail during decoding. + Subscribe(ctx context.Context, topics ...string) error + // Unsubscribe removes subscriptions to one or more named topics and returns + // an error if removal fails. It does not mean "all topics" when none are + // supplied; the all-topics shorthand belongs to polling only. + Unsubscribe(ctx context.Context, topics ...string) error + // Info returns an independent identity and ownership snapshot. Consumer and + // group IDs omitted from configuration are assigned when Open succeeds. + Info() SubscriptionConsumerInfo + // LatestWatermark reports the latest received server watermark. The boolean + // is false until a watermark is observed. This is not a processor watermark. + LatestWatermark() (timestamp int64, ok bool) + // AllConsumed reports whether all subscribed topics have completed and been + // automatically unsubscribed. An active unbounded topic prevents completion; + // no subscriptions means true. This does not assert that delivered messages + // were committed or that client-side processor buffers have been drained. + AllConsumed() bool +} + +// SubscriptionConsumerInfo contains identity and ownership, without exposing credentials. +type SubscriptionConsumerInfo struct { + // ConsumerID identifies this consumer and is assigned on Open if omitted. + ConsumerID string + // GroupID identifies the consumer group and is assigned on Open if omitted. + GroupID string + // OwnerID is the configured topic owner identity, if any. + OwnerID string + // OwnerEpoch is nil when no epoch was supplied; a pointer to zero is an + // explicitly supplied epoch. The pointer is part of the returned snapshot. + OwnerEpoch *int64 +} + +// SubscriptionPullConsumer fetches record messages explicitly. Poll and PollWithInfo use +// server timestamp units for message data and time.Duration for timeouts. The +// shorter of the timeout and the context deadline bounds a polling operation. +// No topics means all subscribed topics. +// +// A TsFile decoding error returns no messages for that poll round, including +// records from parallel requests. That round is not acknowledged, added to the +// processor or auto-commit buffers, or used to advance positions. Previously +// delivered records and confirmed progress are unaffected. The caller can +// unsubscribe the offending topic and poll again; no automatic retry is made. +type SubscriptionPullConsumer interface { + SubscriptionConsumer + + // Poll retrieves messages from the named topics, waiting up to timeout. + // Omitting topics selects all current subscriptions. An empty result with a + // nil error is normal when no records arrive within the polling period. + // It returns an error when polling, context cancellation, or decoding fails. + Poll(ctx context.Context, timeout time.Duration, topics ...string) ([]SubscriptionMessage, error) + // PollWithInfo polls with the same topic, timeout, and error semantics as + // Poll and also reports processor buffering and ordering-watermark state. + PollWithInfo(ctx context.Context, timeout time.Duration, topics ...string) (PollResult, error) + // DrainBufferedMessages retrieves user-visible records held by client-side + // processors without starting a new server poll. The result can be empty. + // Manual-commit callers must process and explicitly commit these records + // before closing when processors still have buffered output. + DrainBufferedMessages(ctx context.Context) ([]SubscriptionMessage, error) + + // Commit synchronously confirms that one or more previously delivered + // messages were successfully processed. A returned error does not imply + // that every message failed; accepted progress is not rolled back. + Commit(ctx context.Context, messages ...SubscriptionMessage) error + // CommitAsync confirms the same processing outcome as Commit without + // waiting for completion. Its result channel has capacity one, receives + // exactly one error or nil on success, and is then closed. An unread result + // never blocks cleanup. Context cancellation does not undo accepted commits. + CommitAsync(ctx context.Context, messages ...SubscriptionMessage) <-chan error + + // AddProcessor appends a processor to the pipeline. Each poll passes messages + // through processors in their addition order before making records visible. + AddProcessor(processor SubscriptionMessageProcessor) error +} + +// ConsumeFunc handles a delivered record message. A nil error means success; +// a non-nil error means failure. Acknowledgment and redelivery depend on the +// configured AckStrategy rather than on the callback alone. +// The callback context is canceled when the consumer stops delivery. The message +// remains valid for the callback; the callback must not release it or retain +// borrowed record data after returning without copying that data. +type ConsumeFunc func(ctx context.Context, message SubscriptionMessage) error + +// SubscriptionPushConsumer polls in the background and delivers records to SubscriptionPushConsumerConfig.Handler. +// A TsFile decoding error stops automatic polling before it is reported; no +// record callback or acknowledgment is performed for that failed poll round. +type SubscriptionPushConsumer interface { + SubscriptionConsumer + + // Errors reports the first terminal background error, then delivery stops. + // The channel has capacity at least one; the error is sent once even if the + // caller is not receiving. The lifecycle owner closes it after all senders + // have stopped. An ordinary callback failure is not a terminal error. + Errors() <-chan error +} + +// PollResult adds processing metadata to a successful poll. Watermark belongs to +// the last WatermarkReporter in the pipeline, not SubscriptionConsumer.LatestWatermark. +type PollResult struct { + // Messages contains the user-visible records emitted by the processing chain. + Messages []SubscriptionMessage + // BufferedCount is the sum of messages retained by all processors. + BufferedCount int + // Watermark is the last reporter's ordering watermark, valid only when + // HasWatermark is true. It is not the latest received server watermark. + Watermark int64 + // HasWatermark distinguishes an available watermark from an uninitialized + // value; zero can itself be a valid watermark. + HasWatermark bool +} diff --git a/subscription/doc.go b/subscription/doc.go new file mode 100644 index 0000000..4232d56 --- /dev/null +++ b/subscription/doc.go @@ -0,0 +1,35 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +// Package subscription defines the public contracts for IoTDB tree-model and +// table-model subscriptions. It is a separate Go module from the Session client. +// +// This initial module contains interfaces and data types only. Connections, +// consumers, constructors, protocol decoding, and built-in processors will be +// implemented in subsequent steps. +// +// Record messages are the supported payload format. Topic management and +// subscription do not reject TsFile topics; decoding a TsFile payload must return +// ErrUnsupportedTsFile before delivering or acknowledging it. This API exposes +// no TsFile handlers or TLS configuration and adds no native library dependency. +// +// Contexts passed to Open limit the opening operation. Background work belongs +// to the consumer and stops on Close, rather than when that context is canceled. +// Other operation contexts bound the corresponding operation only. +package subscription diff --git a/subscription/errors.go b/subscription/errors.go new file mode 100644 index 0000000..45e9caf --- /dev/null +++ b/subscription/errors.go @@ -0,0 +1,51 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package subscription + +import "errors" + +// Sentinel errors can be matched through wrapped operation errors with errors.Is. +var ( + // ErrUnsupportedTsFile is raised only when decoding a TsFile payload. It + // must not cause automatic connection retries or acknowledgment of that + // payload. Management and Subscribe do not preemptively return this error. + ErrUnsupportedTsFile = errors.New("subscription: TsFile payloads are not supported") + // ErrClosed indicates that an operation cannot proceed after the client's + // final close. A successfully closed consumer cannot reclaim its identity + // by reopening the same instance. + ErrClosed = errors.New("subscription: client is closed") + // ErrNotOpen indicates that an operation requiring an open session or + // consumer was called before its connection was established. + ErrNotOpen = errors.New("subscription: client is not open") + // ErrDataReleased indicates access to record buffers after message Release. + // Releasing user data does not invalidate the message's commit identity. + ErrDataReleased = errors.New("subscription: message data has been released") + // ErrIncompatibleMessage indicates an accessor used with the wrong payload + // kind, such as requesting records from a watermark message. + ErrIncompatibleMessage = errors.New("subscription: incompatible message kind") + // ErrConsumerFenced indicates that another connection has taken over the + // same consumer identity. The fenced instance must stop issuing requests + // and must not automatically reclaim ownership. + ErrConsumerFenced = errors.New("subscription: consumer is fenced") + // ErrOwnerFenced indicates that the server rejected topic ownership, for + // example because the owner, epoch, or lease is no longer valid. It is + // distinct from replacement of a consumer instance. + ErrOwnerFenced = errors.New("subscription: owner is fenced") +) diff --git a/subscription/go.mod b/subscription/go.mod new file mode 100644 index 0000000..8647ca7 --- /dev/null +++ b/subscription/go.mod @@ -0,0 +1,7 @@ +module github.com/apache/iotdb-client-go/subscription + +go 1.25 + +require github.com/apache/iotdb-client-go/v2 v2.0.0-20260914074454-a25f68aa9681 + +require github.com/apache/thrift v0.24.0 // indirect diff --git a/subscription/go.sum b/subscription/go.sum new file mode 100644 index 0000000..f62225e --- /dev/null +++ b/subscription/go.sum @@ -0,0 +1,4 @@ +github.com/apache/iotdb-client-go/v2 v2.0.0-20260914074454-a25f68aa9681 h1:UW2FAA1Y+wEsHnPnCFgu0UprlmMhwZeyDeRbut+a4BQ= +github.com/apache/iotdb-client-go/v2 v2.0.0-20260914074454-a25f68aa9681/go.mod h1:oflS5qOZUFOxOQbpQvkDkjTxmsDoN6qB1y6/1pk3lus= +github.com/apache/thrift v0.24.0 h1:zy31L1a49QTNB2bG1BBfMXol3yJrTH975G3pPubQVLQ= +github.com/apache/thrift v0.24.0/go.mod h1:zPt6WxgvTOM6hF92y8C+MkEM5LMxZuk4JcQOiU4Esvs= diff --git a/subscription/message.go b/subscription/message.go new file mode 100644 index 0000000..8957db7 --- /dev/null +++ b/subscription/message.go @@ -0,0 +1,137 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package subscription + +import ( + "iter" + + "github.com/apache/iotdb-client-go/v2/client" +) + +// SubscriptionMessageKind identifies a record or watermark message independently +// of the protocol's numeric response types. +type SubscriptionMessageKind string + +const ( + // RecordMessage carries decoded records for application consumption. + RecordMessage SubscriptionMessageKind = "records" + // WatermarkMessage carries a server-side timestamp-progress signal for + // client-side processing rather than application record delivery. + WatermarkMessage SubscriptionMessageKind = "watermark" +) + +// SubscriptionMessage exposes records or an internal watermark signal. Poll results and +// push callbacks contain records only; processors can also receive watermarks. +// TsFile messages are never constructed: decoding them returns +// ErrUnsupportedTsFile instead. +type SubscriptionMessage interface { + // Kind returns the message kind used to select the applicable payload view. + Kind() SubscriptionMessageKind + // CommitContext returns a caller-owned snapshot of the routing and progress + // information needed to acknowledge this message. It remains available + // after Release so record buffers can be freed before committing. + CommitContext() CommitContext + // Records returns the record-handler views of the message's decoded + // Tablets. A message can contain batches from multiple databases and tables; + // each batch preserves its own target, schema, and time-column projection. + // The returned data is borrowed and read-only until Release. + // + // Records returns ErrIncompatibleMessage for a watermark and + // ErrDataReleased after the record buffers have been released. + Records() ([]RecordBatch, error) + // Watermark returns the timestamp carried by a watermark message. The + // boolean is false for other kinds, replacing Java's exception when a + // watermark timestamp is requested from a record message. + Watermark() (timestamp int64, ok bool) + // EstimatedSize estimates this message's heap memory usage in bytes, + // including its decoded record buffers. The estimate is useful for + // processor buffering limits and is not a serialized payload size. + EstimatedSize() int64 + // Release drops record buffers and is idempotent. Kind, CommitContext, and + // Watermark remain usable, so a released message can still be committed. + Release() +} + +// Column describes a data column. Time is represented separately by Row.Timestamp +// and is not included in this schema. Existing Session data types and column +// categories are reused without adding a second enum mapping. +type Column struct { + // Name is the data column name exposed by the record handler. + Name string + // DataType determines the Go representation of a non-null column value. + DataType client.TSDataType + // Category identifies a table column as TAG, FIELD, or ATTRIBUTE. + Category client.ColumnCategory +} + +// RecordBatch is the record-handler view of one decoded Tablet. Returned data is +// borrowed, read-only, and valid until its message is released. Copy row values +// and byte slices before retaining them beyond that lifetime. All indexes are +// zero-based. Rows uses the record handler's user-visible ordering; raw indexed +// access addresses the decoded Tablet's row order. +type RecordBatch interface { + // Dialect identifies whether this batch contains tree-model or table-model + // data. Sharing this interface does not merge the models' target semantics. + Dialect() Dialect + // Database returns the database associated with table-model data. It is + // empty for tree-model data, which is addressed by its device path. + Database() string + // Target returns the tree-model device path or table-model table name + // associated with this decoded Tablet. + Target() string + // Columns returns the ordered data-column metadata used by Value and + // Row.Values. The time column is represented separately, even when selected. + Columns() []Column + // TimeSelected reports whether timestamps are exposed in the batch's + // user-visible rows. This is resolved per batch, including any table-specific + // projection, rather than assumed to be uniform across the whole message. + TimeSelected() bool + // Len returns the number of rows in the decoded Tablet, not its backing + // buffer capacity. + Len() int + // Timestamp exposes the raw timestamp for processing even when the time + // column is excluded from user-visible rows. Its unit is server precision. + // The row index refers to the original Tablet order; an out-of-range index + // returns an error. + Timestamp(row int) (int64, error) + // Value returns the value at the zero-based raw row and data-column indexes. + // A nil value denotes SQL NULL; non-null representations are described by + // Row. An out-of-range row or column returns an error. + Value(row, column int) (any, error) + // Rows iterates the user-visible records in ascending timestamp order, + // retaining original row order for equal timestamps. It does not reorder + // the underlying Tablet used by Timestamp and Value. Iteration failures are + // reported through the second yielded value. + Rows() iter.Seq2[Row, error] +} + +// Row contains data columns in RecordBatch.Columns order. A nil Timestamp means +// that the time column was not selected; a nil value is SQL NULL. Values use +// bool, int32, int64, float32, float64, string (TEXT/STRING), time.Time (DATE), or +// []byte (BLOB/OBJECT). TIMESTAMP values remain int64 in server precision. +// Rows are borrowed views with the same lifetime as their message. +type Row struct { + // Timestamp is the selected timestamp in server precision, or nil when the + // time column was not selected. It is not an entry in Values. + Timestamp *int64 + // Values follows RecordBatch.Columns and preserves null entries as nil. + // The slice and any byte values are borrowed from the owning message. + Values []any +} diff --git a/subscription/processor.go b/subscription/processor.go new file mode 100644 index 0000000..b69e4f6 --- /dev/null +++ b/subscription/processor.go @@ -0,0 +1,88 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package subscription + +// SubscriptionMessageProcessor transforms, filters, or enriches messages in the +// pull consumer pipeline. Processors run in their configured order on each poll, +// and the consumer invokes each processor serially. A processor can retain +// messages and return them during a later Process call or through Flush. +// +// A processor that retains messages must also implement BufferingProcessor so +// heartbeat renewal can protect undelivered records. A processor must not release +// messages it returns or retains. +type SubscriptionMessageProcessor interface { + // Process receives messages from the previous pipeline stage, or the raw + // poll when it is the first stage. It returns messages for the next stage + // or application and may return fewer, more, or different messages than + // it received. A non-nil error reports a processing failure. + Process(messages []SubscriptionMessage) ([]SubscriptionMessage, error) + // Flush returns all remaining buffered messages when the consumer drains + // or closes. An empty result means no buffered output remains. A non-nil + // error reports a failure to flush. + // + // Returned messages transfer to the consumer and continue through later + // pipeline stages. The processor must not retain or subsequently modify + // the returned messages. + Flush() ([]SubscriptionMessage, error) + // Reset clears all state that must not survive a seek, including buffered + // records, ordering progress, and cached alignment values. It applies to + // all topics; TopicResetter provides a separate per-topic capability. + Reset() +} + +// BufferingProcessor is the optional capability of a processor that retains +// messages between calls. It reports count and renewable commit contexts together +// so retained, uncommitted records can be included in heartbeat renewal. +type BufferingProcessor interface { + SubscriptionMessageProcessor + // Buffered returns a snapshot of the messages currently held by this + // processor. The snapshot must describe the same buffered state for both + // its count and commit contexts. + Buffered() ProcessorBuffer +} + +// ProcessorBuffer is a snapshot of records retained by one processor. Contexts +// contains every committable buffered message's context, not just one per topic. +type ProcessorBuffer struct { + // Count is the total number of messages currently retained by the processor. + Count int + // Contexts contains the commit context of every committable buffered + // message so the consumer can renew those records before delivery. + Contexts []CommitContext +} + +// TopicResetter is an optional processor capability. ResetTopic clears only the +// named topic, preserving every other topic. Its presence replaces Java's +// supportsTopicScopedReset flag. +type TopicResetter interface { + // ResetTopic clears buffered records and processing state belonging to + // topic after a topic-scoped seek, without touching any other topic. + ResetTopic(topic string) +} + +// WatermarkReporter is an optional capability for processors that calculate an +// ordering watermark. The boolean is false before a watermark is available. +type WatermarkReporter interface { + // Watermark returns the processor's current ordering watermark. It is a + // client-side processing result, distinct from the latest server watermark + // observed by SubscriptionConsumer.LatestWatermark. The boolean is false + // until this processor has a watermark to report. + Watermark() (timestamp int64, ok bool) +} diff --git a/subscription/progress.go b/subscription/progress.go new file mode 100644 index 0000000..00aca94 --- /dev/null +++ b/subscription/progress.go @@ -0,0 +1,114 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package subscription + +import "context" + +// Positioner separates fetched positions from confirmed, recoverable positions. +// Progress results are deep snapshots owned by the caller. Seek operations copy +// their inputs and reset state for the named topic only, including pending +// automatic commits and processor state. With multiple subscribed topics, every +// processor must implement TopicResetter; otherwise seek fails before changing +// server positions. With a single topic, SubscriptionMessageProcessor.Reset is a safe fallback. +type Positioner interface { + // Positions returns the latest observed fetch positions for topic. They are + // hints for subsequent polls, not durable checkpoints. No observed position + // returns empty progress; a consumer that is not open or is fenced returns + // an error. The returned map and its nested maps are independent snapshots. + Positions(topic string) (TopicProgress, error) + // CommittedPositions returns the latest confirmed positions for topic. + // Callers can persist this snapshot as a recovery checkpoint. Modifying it + // does not modify the consumer's progress. An unopened or fenced consumer + // returns an error. + CommittedPositions(topic string) (TopicProgress, error) + // SeekToBeginning seeks to the earliest WAL position still retained for + // topic. Reclaimed WAL segments cannot be recovered by this operation. + SeekToBeginning(ctx context.Context, topic string) error + // SeekToEnd seeks to the current WAL tail so subsequent consumption starts + // with newly written data for topic. + SeekToEnd(ctx context.Context, topic string) error + // Seek replays topic from the supplied writer progress. It copies progress + // and clears obsolete local acknowledgment and processor state for topic. + // A failure across multiple providers does not imply atomic rollback. + Seek(ctx context.Context, topic string, progress TopicProgress) error + // SeekAfter resumes topic after the supplied writer progress, with the same + // snapshot, reset, and multi-provider failure semantics as Seek. + SeekAfter(ctx context.Context, topic string, progress TopicProgress) error +} + +// TopicProgress maps a region ID to that region's writer positions. +type TopicProgress map[string]RegionProgress + +// RegionProgress preserves independent progress for each writer. +type RegionProgress map[WriterID]WriterProgress + +// WriterID is comparable and can be used as a map key. +type WriterID struct { + // RegionID identifies the region in which this writer produces records. + RegionID string + // NodeID identifies the writer node within RegionID. Both fields form the + // identity; NodeID alone is not sufficient. + NodeID int32 +} + +// WriterProgress contains protocol progress, not an application event timestamp. +type WriterProgress struct { + // PhysicalTime is the physical-time component of protocol progress. It is + // not the record's event timestamp and has no client-imposed time unit. + PhysicalTime int64 + // LocalSeq orders writes for the writer; it is not a global stream offset. + LocalSeq int64 +} + +// WriterPosition keeps optional writer identity and progress together. +type WriterPosition struct { + // ID is the writer whose position is described. + ID WriterID + // Progress is the position associated with ID. + Progress WriterProgress +} + +// InvalidCommitID marks a context without a legacy commit ID. Writer progress, +// when present, can still identify a committable message. +const InvalidCommitID int64 = -1 + +// CommitContext preserves routing, restart, seek-generation, and writer metadata. +// A non-nil Writer selects writer progress; nil preserves legacy commit metadata. +// Values returned by SubscriptionMessage.CommitContext are caller-owned snapshots. +type CommitContext struct { + // DataNodeID selects the node to which confirmation is routed. + DataNodeID int32 + // RebootTimes distinguishes different restart instances of that node. + RebootTimes int32 + // Topic identifies the topic that produced the message. + Topic string + // GroupID identifies the consumer group owning the delivery. + GroupID string + // CommitID is the legacy delivery identifier, or InvalidCommitID when absent. + CommitID int64 + // SeekGeneration distinguishes deliveries from different seek generations. + SeekGeneration int64 + // RegionID preserves the region component of legacy progress metadata. + RegionID string + // PhysicalTime preserves the physical-time component of legacy progress. + PhysicalTime int64 + // Writer carries paired writer identity and progress when present. + Writer *WriterPosition +} diff --git a/subscription/session.go b/subscription/session.go new file mode 100644 index 0000000..12d8bc2 --- /dev/null +++ b/subscription/session.go @@ -0,0 +1,130 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package subscription + +import ( + "context" + "time" +) + +// SubscriptionSession manages topics and subscriptions for one configured +// dialect. Tree and table sessions share this method contract, while their +// connection and SQL dialect remain model-specific. TsFile topic attributes pass +// through to the server; support is checked only when a consumer decodes a +// payload. Management operations require an open session and report connection, +// statement-execution, or context errors. Returned collections are caller-owned +// snapshots. +type SubscriptionSession interface { + // Open establishes the connection to IoTDB using the configured dialect and + // credentials. The context limits this opening operation; it does not own + // the lifetime of a successfully opened session. + Open(ctx context.Context) error + // Close closes the session and releases its connection resources, returning + // any error encountered while closing them. + Close() error + + // CreateTopic creates name with the supplied attributes and existence + // condition. A name containing single quotes must be enclosed in backticks. + // Attribute keys and values are quoted and escaped for SQL automatically; + // a nil or empty Attributes map creates the topic without a WITH clause. + // Connection failures, invalid identifiers, and rejected statements are + // returned as errors. + CreateTopic(ctx context.Context, name string, options CreateTopicOptions) error + // AlterTopic changes the supplied properties of name. Attributes must not + // be empty; ownership properties must be changed with AlterTopicOwner. + // A name containing single quotes must be enclosed in backticks. Invalid + // input, connection failures, and rejected statements are returned as errors. + AlterTopic(ctx context.Context, name string, attributes map[string]string) error + // AlterTopicOwner transfers name to the supplied owner ID and monotonically + // increasing epoch, optionally setting its lease duration. The owner ID must + // not be empty. A name containing single quotes must be enclosed in backticks. + // Ownership validation and statement failures are returned as errors. + AlterTopicOwner(ctx context.Context, name string, owner TopicOwner) error + // DropTopic removes name, using options.IfExists to allow a missing topic. + // A name containing single quotes must be enclosed in backticks. Connection + // failures and rejected statements are returned as errors. + DropTopic(ctx context.Context, name string, options DropOptions) error + // Topics retrieves all existing topics. A failed query or connection is + // reported as an error rather than an empty successful result. + Topics(ctx context.Context) ([]Topic, error) + // Topic retrieves name and returns found=false when it does not exist. + // A name containing single quotes must be enclosed in backticks. Query and + // connection failures are reported through err. + Topic(ctx context.Context, name string) (topic Topic, found bool, err error) + + // Subscriptions retrieves subscriptions belonging to topic, or every + // subscription when topic is empty. A nonempty topic containing single + // quotes must be enclosed in backticks. Query and connection failures are + // returned as errors. + Subscriptions(ctx context.Context, topic string) ([]Subscription, error) + // DropSubscription removes the subscription identified by id. When + // options.IfExists is true, a missing subscription is not an error. Invalid + // identifiers, connection failures, and rejected statements return errors. + DropSubscription(ctx context.Context, id string, options DropOptions) error +} + +// CreateTopicOptions combines topic attributes and conditional creation. +type CreateTopicOptions struct { + // Attributes contains the topic properties to include in the CREATE TOPIC + // statement. Nil or empty uses the server's topic defaults. + Attributes map[string]string + // IfNotExists adds IF NOT EXISTS, allowing an existing topic without error. + IfNotExists bool +} + +// DropOptions controls conditional deletion of a topic or subscription. +type DropOptions struct { + // IfExists adds IF EXISTS, allowing the named object to be absent. + IfExists bool +} + +// TopicOwner sets the owner and its epoch. A nil LeaseDuration preserves the +// server's default lease behavior; a non-nil duration is an explicit setting. +type TopicOwner struct { + // ID is the nonempty owner identity to assign to the topic. + ID string + // Epoch is the new monotonically increasing owner epoch. + Epoch int64 + // LeaseDuration optionally sets the owner's lease. Unlike the Java API's + // millisecond argument, this field is expressed as a time.Duration. + LeaseDuration *time.Duration +} + +// Topic contains the name and the server's textual topic attributes. The raw +// representation is preserved rather than assuming it is a client-side map. +type Topic struct { + // Name is the topic name returned by the server. + Name string + // Attributes is the topic's property description as returned by the server. + Attributes string +} + +// Subscription is one server-side subscription description. ConsumerIDs retains +// the server's textual representation, which is not assumed to be a CSV list. +type Subscription struct { + // ID uniquely identifies the server-side subscription. + ID string + // Topic is the topic to which this subscription belongs. + Topic string + // GroupID identifies the consumer group sharing the subscription. + GroupID string + // ConsumerIDs is the server's description of the group's consumer IDs. + ConsumerIDs string +}
