This is an automated email from the ASF dual-hosted git repository. shuwenwei pushed a commit to branch supports_subscription in repository https://gitbox.apache.org/repos/asf/iotdb-client-go.git
commit a219300a987f153629fb6006ae6a8b10eb773067 Author: shuwenwei <[email protected]> AuthorDate: Fri Nov 13 12:23:12 2026 +0800 Add tree and table subscription examples and clarify session filenames --- subscription/README.md | 4 +- subscription/example/README.md | 84 ++++++ subscription/example/table/incremental/main.go | 313 ++++++++++++++++++++ subscription/example/table/main.go | 316 +++++++++++++++++++++ subscription/example/tree/incremental/main.go | 315 ++++++++++++++++++++ subscription/example/tree/main.go | 280 ++++++++++++++++++ .../{session.go => subscription_session.go} | 0 ...on_config.go => subscription_session_config.go} | 0 ...ction.go => subscription_session_connection.go} | 0 ...ession_impl.go => subscription_session_impl.go} | 0 ...ession_rows.go => subscription_session_rows.go} | 0 ...{session_sql.go => subscription_session_sql.go} | 0 12 files changed, 1311 insertions(+), 1 deletion(-) diff --git a/subscription/README.md b/subscription/README.md index 7cebe42..db9c106 100644 --- a/subscription/README.md +++ b/subscription/README.md @@ -25,6 +25,8 @@ The module path is `github.com/apache/iotdb-client-go/subscription`. It uses Go The Java reference is `iotdb-client/subscription` at IoTDB commit `d9122d837610`. The supported payload format is records: this module defines no TsFile handler, download, file-management, or Reader API. Topic management and `Subscribe` do not reject TsFile topics in advance. Dispatching a file payload during decoding returns `ErrUnsupportedTsFile`; that payload must not be delivered or acknowledged. +Standalone [tree and table examples](example/README.md) cover ordinary record Pull/Push consumption and incremental subscriptions with progress, checkpoint resume, and optional processors. They are part of this module and use the existing Session client only to prepare sample data. + ## Configuration and Data Model Selection Go uses `Dialect` to distinguish the tree and table models. Management and consumption each share a contract rather than duplicating Java's tree/table types and Builder hierarchy. @@ -324,4 +326,4 @@ Alignment reads inputs through the existing public RecordBatch API, copies recor Operation errors can wrap these sentinels; callers identify them with `errors.Is`. Unsupported TsFile errors must not trigger connection retries or file-payload acknowledgments. Pull returns an error and no messages for that round, including records obtained by parallel branches. That round must not advance positions or enter processor or automatic-commit buffers. Previously delivered records and confirmed progress are not rolled back. Callers can unsubscribe the offending topic and poll again. -The built-in Watermark/ColumnAlign processors are implemented; further consumption examples, behavior tests, and runtime verification are pending. Management queries reuse the root module's decoder; this step does not change its handling of malformed data. Verification has been static only: no compilation, tests, or formatting tools have run, and no runtime validation result is claimed. +The built-in Watermark/ColumnAlign processors and standalone tree/table examples are implemented; behavior tests and runtime verification are pending. Management queries reuse the root module's decoder; this step does not change its handling of malformed data. No compilation, tests, formatting tools, or compiler/syntax checks have run as part of the example work, and no runtime validation result is claimed. diff --git a/subscription/example/README.md b/subscription/example/README.md new file mode 100644 index 0000000..947e3f2 --- /dev/null +++ b/subscription/example/README.md @@ -0,0 +1,84 @@ +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +# Subscription Examples + +These programs belong to the standalone `github.com/apache/iotdb-client-go/subscription` module. Run commands from `subscription/` with Go 1.25 and a server that supports record subscriptions. The incremental programs additionally require server support for incremental subscriptions and per-writer ordering. + +| Program | Scenario | Java reference | +| --- | --- | --- | +| [tree](tree/main.go) | Tree path and time filtering, topic management, automatic-commit Pull or record Push | `SubscriptionSessionExample.dataSubscription1` | +| [table](table/main.go) | Database/table and time filtering, table schema metadata, automatic-commit Pull or record Push | `TableModelSubscriptionSessionExample.dataSubscription` | +| [tree/incremental](tree/incremental/main.go) | Tree incremental subscription, manual commit, progress, checkpoint resume, optional processors | `ConsensusSubscriptionSessionExample` | +| [table/incremental](table/incremental/main.go) | Table incremental subscription, manual commit, progress, checkpoint resume, optional processors | `ConsensusTableModelSubscriptionSessionExample` | + +The Java references are under `example/subscription/src/main/java/org/apache/iotdb` in the IoTDB checkout. The Go programs use the current topic mode name, `incremental`, for the scenarios whose Java class names contain `Consensus`. Every program explicitly requests `format=SubscriptionRecordHandler`. Java's TsFile examples are excluded because this module returns `ErrUnsupportedTsFile` when decoding file payloads. + +Record Push, progress/seek, and processor usage extend the Java record examples. They demonstrate the Go APIs implemented by this module; the Java example directory does not contain equivalent record-only examples for those combinations. + +## Ordinary Record Subscriptions + +The ordinary programs prepare historical sample data, create a topic with a path or database/table filter and a time range, show topic and subscription metadata, and consume records. Pull is the default and uses automatic commits of delivered messages. Use the incremental programs' manual-commit pattern when processing success must control acknowledgment. Each table batch also prints its database, table, and column metadata. + +```sh +cd subscription +go run ./example/tree +go run ./example/table +``` + +Select Push to deliver records through a sequential callback. `-ack=after` confirms successful callbacks; `-ack=before` confirms the batch before callbacks. The Push examples observe `Errors()` and stop after `-duration` or a termination signal. + +```sh +go run ./example/tree -consumer=push -ack=after -duration=10s +go run ./example/table -consumer=push -ack=before -duration=10s +``` + +The Push callback reads borrowed records without releasing the message. Pull releases records after processing; the consumer retains their commit identities for acknowledgment. + +## Incremental Subscriptions + +The incremental programs prepare a bootstrap record and flush it to establish the source region. They then create a topic with `mode=incremental` and `order-mode=per-writer`, subscribe, and write five new records. This order follows the Java incremental examples: the records of interest are written after the subscription exists. + +```sh +go run ./example/tree/incremental +go run ./example/table/incremental +``` + +Each program disables automatic commits, calls `PollWithInfo`, processes records, and commits only successfully processed messages. It prints fetched and committed positions, takes an in-memory checkpoint of confirmed progress, and demonstrates `SeekAfter` once. The checkpoint is scoped to that run and is not saved to disk. + +Enable the optional Pull processor chain with `-processors`: + +```sh +go run ./example/tree/incremental -processors +go run ./example/table/incremental -processors +``` + +The chain installs `WatermarkProcessor` before the model-specific `ColumnAlignProcessor`. Its out-of-order distance is one raw server timestamp unit, and its wall-clock timeout is two seconds. Message ordering is based on each message's maximum timestamp; it does not globally sort rows across messages. The examples drain retained records and commit successfully processed output before closing a manual-commit consumer. + +## Connection Settings and Resources + +All programs accept `-host`, `-port`, and `-username`, defaulting to `127.0.0.1`, `6667`, and `root`. `IOTDB_PASSWORD` supplies the password; when it is unset, the default is `root`. An explicitly empty environment value remains an empty password. + +```sh +IOTDB_PASSWORD='your-password' go run ./example/table -host=127.0.0.1 -port=6667 -username=root +``` + +Each run uses its own database, topic, consumer, and group. Database and topic names are generated locally; consumer identities are either supplied by the program or assigned by the server. The programs print resource and subscription information, create sample data through the existing Session client, and remove only the topic they created after closing the consumer. The sample database and records remain available for inspection. Topic cleanup and manual-commit shutdown use bounded con [...] + +These commands are provided for manual use. The programs have not been compiled or executed, and no test or runtime validation result is claimed. diff --git a/subscription/example/table/incremental/main.go b/subscription/example/table/incremental/main.go new file mode 100644 index 0000000..85173fa --- /dev/null +++ b/subscription/example/table/incremental/main.go @@ -0,0 +1,313 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +// This example follows Java ConsensusTableModelSubscriptionSessionExample's +// incremental record flow, adding progress, SeekAfter, and optional processors. +package main + +import ( + "context" + "errors" + "flag" + "fmt" + "log" + "net" + "os" + "os/signal" + "strconv" + "syscall" + "time" + + "github.com/apache/iotdb-client-go/subscription" + "github.com/apache/iotdb-client-go/v2/client" +) + +func main() { + if err := run(); err != nil { + log.Fatal(err) + } +} + +func run() (err error) { + host := flag.String("host", "127.0.0.1", "IoTDB host") + port := flag.Int("port", 6667, "IoTDB RPC port") + username := flag.String("username", "root", "IoTDB username") + processors := flag.Bool("processors", false, "Enable Watermark followed by table ColumnAlign processing") + flag.Parse() + if *port < 1 || *port > 65535 { + return errors.New("port must be between 1 and 65535") + } + password, supplied := os.LookupEnv("IOTDB_PASSWORD") + if !supplied { + password = "root" + } + address := net.JoinHostPort(*host, strconv.Itoa(*port)) + runID := strconv.FormatInt(time.Now().UnixNano(), 10) + database := "go_subscription_table_incremental_" + runID + table := "events" + topic := "go_subscription_table_incremental_topic_" + runID + ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM) + defer stop() + + dataSession, err := client.NewTableSession(&client.Config{ + Host: *host, + Port: strconv.Itoa(*port), + UserName: *username, + Password: password, + }, false, 10000) + if err != nil { + return err + } + defer func() { err = errors.Join(err, dataSession.Close()) }() + if err := prepareBootstrapData(dataSession, database, table); err != nil { + return err + } + if _, err := fmt.Printf("Database %s and table %s are left in place for inspection.\n", database, table); err != nil { + return err + } + + managementConfig := subscription.DefaultSubscriptionSessionConfig() + managementConfig.Dialect = subscription.TableDialect + managementConfig.Connection.NodeURLs = []string{address} + managementConfig.Connection.Username = *username + managementConfig.Connection.Password = password + management, err := subscription.NewSubscriptionSession(managementConfig) + if err != nil { + return err + } + if err := management.Open(ctx); err != nil { + return err + } + defer func() { err = errors.Join(err, management.Close()) }() + if err := management.CreateTopic(ctx, topic, subscription.CreateTopicOptions{ + Attributes: map[string]string{ + "mode": "incremental", + "format": "SubscriptionRecordHandler", + "database": database, + "table": table, + "order-mode": "per-writer", + }, + }); err != nil { + return err + } + // This cleanup owns only the topic successfully created above, not user data. + defer func() { + cleanupCtx, cancel := context.WithTimeout(context.Background(), 10*time.Second) + defer cancel() + err = errors.Join(err, management.DropTopic(cleanupCtx, topic, subscription.DropOptions{IfExists: true})) + }() + + config := subscription.DefaultSubscriptionPullConsumerConfig() + config.Consumer.Dialect = subscription.TableDialect + config.Consumer.Connection = managementConfig.Connection + config.Consumer.ConsumerID = "table_incremental_consumer_" + runID + config.Consumer.GroupID = "table_incremental_group_" + runID + autoCommit := false + config.AutoCommit = &autoCommit + consumer, err := subscription.NewSubscriptionPullConsumer(config) + if err != nil { + return err + } + var watermark *subscription.WatermarkProcessor + var alignment *subscription.ColumnAlignProcessor + if *processors { + watermark, err = subscription.NewWatermarkProcessor(subscription.WatermarkProcessorConfig{ + MaxOutOfOrderness: 1, + Timeout: 2 * time.Second, + }) + if err != nil { + return err + } + alignment, err = subscription.NewColumnAlignProcessor(subscription.TableDialect) + if err != nil { + return err + } + if err := consumer.AddProcessor(watermark); err != nil { + return err + } + if err := consumer.AddProcessor(alignment); err != nil { + return err + } + } + if err := consumer.Open(ctx); err != nil { + return err + } + defer func() { + if err != nil { + // An error is not permission to acknowledge unprocessed data. Discard + // processor state without ACK so Close can release the connections. + if watermark != nil { + watermark.Reset() + } + if alignment != nil { + alignment.Reset() + } + cleanupCtx, cancel := context.WithTimeout(context.Background(), 10*time.Second) + defer cancel() + discarded, drainErr := consumer.DrainBufferedMessages(cleanupCtx) + for _, message := range discarded { + message.Release() + } + err = errors.Join(err, drainErr) + } + err = errors.Join(err, consumer.Close()) + }() + if err := consumer.Subscribe(ctx, topic); err != nil { + return err + } + defer func() { + cleanupCtx, cancel := context.WithTimeout(context.Background(), 10*time.Second) + defer cancel() + err = errors.Join(err, consumer.Unsubscribe(cleanupCtx, topic)) + }() + // The bootstrap only creates the region. Incremental consumption starts + // after Subscribe, so write the five demonstrated records afterward. + if err := writeRealtimeData(dataSession, table); err != nil { + return err + } + + rows := 0 + for round := 1; round <= 20 && rows < 5; round++ { + result, err := consumer.PollWithInfo(ctx, time.Second) + if err != nil { + return err + } + count, err := processAndCommit(ctx, consumer, result.Messages) + if err != nil { + return err + } + rows += count + if _, err := fmt.Printf("poll=%d rows=%d buffered=%d watermark=%d hasWatermark=%t\n", round, rows, result.BufferedCount, result.Watermark, result.HasWatermark); err != nil { + return err + } + if err := printProgress(consumer, topic); err != nil { + return err + } + } + // Flush processor output before checkpointing or seeking. Only successfully + // printed records are committed, using a live operation context. + drained, err := consumer.DrainBufferedMessages(ctx) + if err != nil { + return err + } + count, err := processAndCommit(ctx, consumer, drained) + if err != nil { + return err + } + rows += count + if err := printProgress(consumer, topic); err != nil { + return err + } + if rows < 5 { + return fmt.Errorf("received %d of at least 5 realtime rows; check incremental subscription support on the server", rows) + } + checkpoint, err := consumer.CommittedPositions(topic) + if err != nil { + return err + } + if len(checkpoint) == 0 { + return errors.New("no committed writer progress is available for SeekAfter") + } + // This checkpoint is an in-memory snapshot for this run, not durable storage. + if err := consumer.SeekAfter(ctx, topic, checkpoint); err != nil { + return err + } + _, err = fmt.Printf("SeekAfter completed with the in-memory checkpoint: %+v\n", checkpoint) + return err +} + +func prepareBootstrapData(session client.ITableSession, database, table string) error { + for _, statement := range []string{ + "CREATE DATABASE " + database, + "USE " + database, + "CREATE TABLE " + table + "(tag1 STRING TAG, s1 INT64 FIELD)", + "INSERT INTO " + table + "(time, tag1, s1) VALUES (0, 'bootstrap', 0)", + "FLUSH", + } { + if err := session.ExecuteNonQueryStatement(statement); err != nil { + return err + } + } + return nil +} + +func writeRealtimeData(session client.ITableSession, table string) error { + for index := 1; index <= 5; index++ { + statement := fmt.Sprintf("INSERT INTO %s(time, tag1, s1) VALUES (%d, 'device_%d', %d)", table, index, index, index*10) + if err := session.ExecuteNonQueryStatement(statement); err != nil { + return err + } + } + return session.ExecuteNonQueryStatement("FLUSH") +} + +func processAndCommit(ctx context.Context, consumer subscription.SubscriptionPullConsumer, messages []subscription.SubscriptionMessage) (int, error) { + defer func() { + for _, message := range messages { + message.Release() + } + }() + rows := 0 + for _, message := range messages { + batches, err := message.Records() + if err != nil { + return rows, err + } + for _, batch := range batches { + if _, err := fmt.Printf("database=%s table=%s timeSelected=%t columns=%+v\n", batch.Database(), batch.Target(), batch.TimeSelected(), batch.Columns()); err != nil { + return rows, err + } + for row, err := range batch.Rows() { + if err != nil { + return rows, err + } + if err := ctx.Err(); err != nil { + return rows, err + } + var timestamp any + if row.Timestamp != nil { + timestamp = *row.Timestamp + } + if _, err := fmt.Printf("time=%v values=%v\n", timestamp, row.Values); err != nil { + return rows, err + } + rows++ + } + } + } + if len(messages) != 0 { + if err := consumer.Commit(ctx, messages...); err != nil { + return rows, err + } + } + return rows, nil +} + +func printProgress(consumer subscription.SubscriptionPullConsumer, topic string) error { + positions, err := consumer.Positions(topic) + if err != nil { + return err + } + committed, err := consumer.CommittedPositions(topic) + if err != nil { + return err + } + _, err = fmt.Printf("positions=%+v committed=%+v\n", positions, committed) + return err +} diff --git a/subscription/example/table/main.go b/subscription/example/table/main.go new file mode 100644 index 0000000..6eb284b --- /dev/null +++ b/subscription/example/table/main.go @@ -0,0 +1,316 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +// This example adapts Java TableModelSubscriptionSessionExample's record topic +// and metadata traversal. Record-format Push delivery is an additional Go path. +package main + +import ( + "context" + "errors" + "flag" + "fmt" + "log" + "net" + "os" + "os/signal" + "strconv" + "syscall" + "time" + + "github.com/apache/iotdb-client-go/subscription" + "github.com/apache/iotdb-client-go/v2/client" +) + +func main() { + if err := run(); err != nil { + log.Fatal(err) + } +} + +func run() (err error) { + host := flag.String("host", "127.0.0.1", "IoTDB host") + port := flag.Int("port", 6667, "IoTDB RPC port") + username := flag.String("username", "root", "IoTDB username") + mode := flag.String("consumer", "pull", "Record consumer: pull or push") + ack := flag.String("ack", "after", "Push acknowledgment strategy: after or before") + duration := flag.Duration("duration", 10*time.Second, "How long to receive Push records") + flag.Parse() + if *port < 1 || *port > 65535 { + return errors.New("port must be between 1 and 65535") + } + if *mode != "pull" && *mode != "push" { + return errors.New("consumer must be pull or push") + } + if *ack != "after" && *ack != "before" { + return errors.New("ack must be after or before") + } + if *mode == "push" && *duration <= 0 { + return errors.New("duration must be positive for Push consumption") + } + password, supplied := os.LookupEnv("IOTDB_PASSWORD") + if !supplied { + password = "root" + } + address := net.JoinHostPort(*host, strconv.Itoa(*port)) + runID := strconv.FormatInt(time.Now().UnixNano(), 10) + database := "go_subscription_table_" + runID + table := "events" + topic := "go_subscription_table_topic_" + runID + ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM) + defer stop() + + dataSession, err := client.NewTableSession(&client.Config{ + Host: *host, + Port: strconv.Itoa(*port), + UserName: *username, + Password: password, + }, false, 10000) + if err != nil { + return err + } + defer func() { err = errors.Join(err, dataSession.Close()) }() + if err := prepareData(dataSession, database, table); err != nil { + return err + } + if _, err := fmt.Printf("Database %s and table %s are left in place for inspection.\n", database, table); err != nil { + return err + } + + managementConfig := subscription.DefaultSubscriptionSessionConfig() + managementConfig.Dialect = subscription.TableDialect + managementConfig.Connection.NodeURLs = []string{address} + managementConfig.Connection.Username = *username + managementConfig.Connection.Password = password + management, err := subscription.NewSubscriptionSession(managementConfig) + if err != nil { + return err + } + if err := management.Open(ctx); err != nil { + return err + } + defer func() { err = errors.Join(err, management.Close()) }() + if err := management.CreateTopic(ctx, topic, subscription.CreateTopicOptions{ + Attributes: map[string]string{ + "mode": "initial", + "format": "SubscriptionRecordHandler", + "database": database, + "table": table, + "start-time": "25", + "end-time": "75", + "strict": "true", + }, + }); err != nil { + return err + } + // Drop only the unique topic whose creation succeeded in this run. + defer func() { + cleanupCtx, cancel := context.WithTimeout(context.Background(), 10*time.Second) + defer cancel() + err = errors.Join(err, management.DropTopic(cleanupCtx, topic, subscription.DropOptions{IfExists: true})) + }() + if topics, err := management.Topics(ctx); err != nil { + return err + } else { + if _, err := fmt.Printf("Topics: %+v\n", topics); err != nil { + return err + } + } + + consumerConfig := subscription.DefaultSubscriptionConsumerConfig() + consumerConfig.Dialect = subscription.TableDialect + consumerConfig.Connection = managementConfig.Connection + consumerConfig.ConsumerID = "table_consumer_" + runID + consumerConfig.GroupID = "table_group_" + runID + if *mode == "push" { + config := subscription.DefaultSubscriptionPushConsumerConfig() + config.Consumer = consumerConfig + if *ack == "before" { + config.AckStrategy = subscription.AckBeforeConsume + } + config.Handler = func(callbackCtx context.Context, message subscription.SubscriptionMessage) error { + _, err := printMessage(callbackCtx, message) + // Push owns the message; the Handler must not call Release. + return err + } + consumer, err := subscription.NewSubscriptionPushConsumer(config) + if err != nil { + return err + } + if err := consumer.Open(ctx); err != nil { + return err + } + defer func() { + cleanupCtx, cancel := context.WithTimeout(context.Background(), 10*time.Second) + defer cancel() + err = errors.Join(err, consumer.Close()) + // Close cancels delivery; a callback may still be returning. Wait for + // the loop with a fresh budget, even if the run context was canceled. + for { + select { + case terminal, ok := <-consumer.Errors(): + if !ok { + return + } + err = errors.Join(err, terminal) + case <-cleanupCtx.Done(): + err = errors.Join(err, fmt.Errorf("waiting for Push shutdown: %w", cleanupCtx.Err())) + return + } + } + }() + if err := consumer.Subscribe(ctx, topic); err != nil { + return err + } + if err := showSubscriptions(ctx, management, topic); err != nil { + return err + } + timer := time.NewTimer(*duration) + defer timer.Stop() + select { + case terminal, ok := <-consumer.Errors(): + if ok { + return terminal + } + return nil + case <-timer.C: + return nil + case <-ctx.Done(): + return ctx.Err() + } + } + + config := subscription.DefaultSubscriptionPullConsumerConfig() + config.Consumer = consumerConfig + // Keep the default automatic commit behavior of the ordinary Java example. + consumer, err := subscription.NewSubscriptionPullConsumer(config) + if err != nil { + return err + } + if err := consumer.Open(ctx); err != nil { + return err + } + // Close commits pending automatic acknowledgments before unregistering the + // consumer. Unsubscribing first could invalidate those outstanding contexts. + defer func() { err = errors.Join(err, consumer.Close()) }() + if err := consumer.Subscribe(ctx, topic); err != nil { + return err + } + if err := showSubscriptions(ctx, management, topic); err != nil { + return err + } + rows, emptyRounds := 0, 0 + for round := 0; round < 20; round++ { + messages, err := consumer.Poll(ctx, time.Second) + if err != nil { + return err + } + if len(messages) == 0 { + emptyRounds++ + if rows >= 5 && emptyRounds >= 3 { + break + } + continue + } + emptyRounds = 0 + count, err := printPullMessages(ctx, messages) + if err != nil { + return err + } + rows += count + } + if rows < 5 { + return fmt.Errorf("received %d of at least 5 expected rows; check subscription support on the server", rows) + } + return nil +} + +func prepareData(session client.ITableSession, database, table string) error { + for _, statement := range []string{ + "CREATE DATABASE " + database, + "USE " + database, + "CREATE TABLE " + table + "(tag1 STRING TAG, s1 INT64 FIELD)", + } { + if err := session.ExecuteNonQueryStatement(statement); err != nil { + return err + } + } + for index := 0; index < 5; index++ { + statement := fmt.Sprintf("INSERT INTO %s(time, tag1, s1) VALUES (%d, 'device_%d', %d)", table, 25+index, index, index*10) + if err := session.ExecuteNonQueryStatement(statement); err != nil { + return err + } + } + return session.ExecuteNonQueryStatement("FLUSH") +} + +func showSubscriptions(ctx context.Context, management subscription.SubscriptionSession, topic string) error { + subscriptions, err := management.Subscriptions(ctx, topic) + if err == nil { + _, err = fmt.Printf("Subscriptions: %+v\n", subscriptions) + } + return err +} + +func printPullMessages(ctx context.Context, messages []subscription.SubscriptionMessage) (int, error) { + defer func() { + for _, message := range messages { + message.Release() + } + }() + rows := 0 + for _, message := range messages { + count, err := printMessage(ctx, message) + if err != nil { + return rows, err + } + rows += count + } + return rows, nil +} + +func printMessage(ctx context.Context, message subscription.SubscriptionMessage) (int, error) { + batches, err := message.Records() + if err != nil { + return 0, err + } + rows := 0 + for _, batch := range batches { + if _, err := fmt.Printf("database=%s table=%s timeSelected=%t columns=%+v\n", batch.Database(), batch.Target(), batch.TimeSelected(), batch.Columns()); err != nil { + return rows, err + } + for row, err := range batch.Rows() { + if err != nil { + return rows, err + } + if err := ctx.Err(); err != nil { + return rows, err + } + var timestamp any + if row.Timestamp != nil { + timestamp = *row.Timestamp + } + if _, err := fmt.Printf("time=%v values=%v\n", timestamp, row.Values); err != nil { + return rows, err + } + rows++ + } + } + return rows, nil +} diff --git a/subscription/example/tree/incremental/main.go b/subscription/example/tree/incremental/main.go new file mode 100644 index 0000000..c4a8e7f --- /dev/null +++ b/subscription/example/tree/incremental/main.go @@ -0,0 +1,315 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +// This example mirrors ConsensusSubscriptionSessionExample using incremental +// record topics. The generated database is retained for inspection. +package main + +import ( + "context" + "errors" + "flag" + "fmt" + "log" + "net" + "os" + "os/signal" + "strconv" + "syscall" + "time" + + "github.com/apache/iotdb-client-go/v2/client" + "github.com/apache/iotdb-client-go/subscription" +) + +func main() { + host := flag.String("host", "127.0.0.1", "IoTDB host") + port := flag.Int("port", 6667, "IoTDB RPC port") + username := flag.String("username", "root", "IoTDB username") + processors := flag.Bool("processors", false, "Order messages by watermark, then forward-fill NULL columns") + flag.Parse() + if *port < 1 || *port > 65535 { + log.Fatal("Port must be between 1 and 65535") + } + password, supplied := os.LookupEnv("IOTDB_PASSWORD") + if !supplied { + password = "root" + } + ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM) + defer stop() + if err := run(ctx, *host, *port, *username, password, *processors); err != nil { + log.Print(err) + os.Exit(1) + } +} + +func run(ctx context.Context, host string, port int, username, password string, processors bool) (result error) { + runID := strconv.FormatInt(time.Now().UnixNano(), 10) + database := "root.subscription_tree_incremental_" + runID + device := database + ".d0" + topic := "subscription_tree_incremental_" + runID + if _, err := fmt.Printf("Database retained for inspection: %s; device=%s; topic=%s\n", database, device, topic); err != nil { + return err + } + writer := client.NewSession(&client.Config{Host: host, Port: strconv.Itoa(port), UserName: username, Password: password}) + if err := writer.Open(false, 10000); err != nil { + return err + } + defer func() { result = errors.Join(result, writer.Close()) }() + // Bootstrap and flush before topic creation so the server has a data region. + for _, sql := range []string{ + "CREATE DATABASE " + database, + "CREATE TIMESERIES " + device + ".s1 WITH DATATYPE=INT64, ENCODING=RLE", + "INSERT INTO " + device + "(time,s1) VALUES (0,0)", + "FLUSH", + } { + if err := ctx.Err(); err != nil { + return err + } + if err := writer.ExecuteNonQueryStatement(sql); err != nil { + return err + } + } + cfg := subscription.DefaultSubscriptionSessionConfig() + cfg.Connection.NodeURLs = []string{net.JoinHostPort(host, strconv.Itoa(port))} + cfg.Connection.Username, cfg.Connection.Password = username, password + manager, err := subscription.NewSubscriptionSession(cfg) + if err != nil { + return err + } + created := false + defer func() { + cleanupCtx, cancel := context.WithTimeout(context.Background(), 10*time.Second) + defer cancel() + if created { + result = errors.Join(result, manager.DropTopic(cleanupCtx, topic, subscription.DropOptions{IfExists: true})) + } + result = errors.Join(result, manager.Close()) + }() + if err := manager.Open(ctx); err != nil { + return err + } + if err := manager.CreateTopic(ctx, topic, subscription.CreateTopicOptions{Attributes: map[string]string{ + "mode": "incremental", + "format": "SubscriptionRecordHandler", + "path": database + ".**", + "order-mode": "per-writer", + }}); err != nil { + return err + } + created = true + return consume(ctx, cfg.Connection, &writer, device, topic, processors) +} + +func consume(ctx context.Context, connection subscription.ConnectionConfig, writer *client.Session, device, topic string, processors bool) (result error) { + cfg := subscription.DefaultSubscriptionPullConsumerConfig() + cfg.Consumer.Connection = connection + autoCommit := false + cfg.AutoCommit = &autoCommit + consumer, err := subscription.NewSubscriptionPullConsumer(cfg) + if err != nil { + return err + } + opened, subscribed, drained := false, false, false + defer func() { + // Cleanup uses its own live context even when a signal canceled ctx. + cleanupCtx, cancel := context.WithTimeout(context.Background(), 10*time.Second) + defer cancel() + if opened && !drained { + _, drainErr := drainAndCommit(cleanupCtx, consumer) + result = errors.Join(result, drainErr) + } + if subscribed { + result = errors.Join(result, consumer.Unsubscribe(cleanupCtx, topic)) + } + result = errors.Join(result, consumer.Close()) + }() + if processors { + watermark, err := subscription.NewWatermarkProcessor(subscription.WatermarkProcessorConfig{ + MaxOutOfOrderness: 1, + Timeout: 2 * time.Second, + }) + if err != nil { + return err + } + alignment, err := subscription.NewColumnAlignProcessor(subscription.TreeDialect) + if err != nil { + return err + } + if err := consumer.AddProcessor(watermark); err != nil { + return err + } + if err := consumer.AddProcessor(alignment); err != nil { + return err + } + } + if err := consumer.Open(ctx); err != nil { + return err + } + opened = true + if err := consumer.Subscribe(ctx, topic); err != nil { + return err + } + subscribed = true + fmt.Printf("Consumer: %+v\n", consumer.Info()) + // These five writes occur after Subscribe; the bootstrap row is historical. + for timestamp := int64(1); timestamp <= 5; timestamp++ { + if err := ctx.Err(); err != nil { + return err + } + if err := writer.InsertRecord(device, []string{"s1"}, []client.TSDataType{client.INT64}, []interface{}{timestamp * 10}, timestamp); err != nil { + return err + } + } + if err := writer.ExecuteNonQueryStatement("FLUSH"); err != nil { + return err + } + totalRows, emptyPolls := 0, 0 + for round := 1; round <= 20; round++ { + poll, err := consumer.PollWithInfo(ctx, time.Second) + if err != nil { + return err + } + fmt.Printf("Poll round=%d buffered=%d watermark=%d hasWatermark=%t\n", round, poll.BufferedCount, poll.Watermark, poll.HasWatermark) + rows, err := processAndCommit(ctx, consumer, poll.Messages) + if err != nil { + return err + } + totalRows += rows + if err := printPositions(consumer, topic); err != nil { + return err + } + if len(poll.Messages) == 0 { + emptyPolls++ + } else { + emptyPolls = 0 + } + if totalRows >= 5 && emptyPolls >= 3 { + break + } + } + // Drain before checkpoint/seek so seek cannot discard unprocessed buffers. + drainCtx, cancelDrain := context.WithTimeout(context.Background(), 10*time.Second) + rows, err := drainAndCommit(drainCtx, consumer) + cancelDrain() + if err != nil { + return err + } + drained = true + totalRows += rows + if totalRows != 5 { + return fmt.Errorf("expected 5 realtime rows, consumed %d; check server incremental subscription support", totalRows) + } + if err := printPositions(consumer, topic); err != nil { + return err + } + // This checkpoint is an owned in-memory snapshot, not persistent storage. + checkpoint, err := consumer.CommittedPositions(topic) + if err != nil { + return err + } + fmt.Printf("In-memory checkpoint: %+v\n", checkpoint) + if len(checkpoint) == 0 { + return errors.New("server returned no writer checkpoint for incremental consumption") + } + if err := consumer.SeekAfter(ctx, topic, checkpoint); err != nil { + return err + } + fmt.Println("SeekAfter resumed once from the confirmed in-memory checkpoint") + return printPositions(consumer, topic) +} + +func drainAndCommit(ctx context.Context, consumer subscription.SubscriptionPullConsumer) (int, error) { + messages, err := consumer.DrainBufferedMessages(ctx) + if err != nil { + return 0, err + } + return processAndCommit(ctx, consumer, messages) +} + +func processAndCommit(ctx context.Context, consumer subscription.SubscriptionPullConsumer, messages []subscription.SubscriptionMessage) (int, error) { + defer func() { + for _, message := range messages { + message.Release() + } + }() + total := 0 + for _, message := range messages { + rows, err := printMessage(ctx, message) + if err != nil { + return total, err + } + // Never confirm a message whose processing or context check failed. + if err := ctx.Err(); err != nil { + return total, err + } + if err := consumer.Commit(ctx, message); err != nil { + return total, err + } + total += rows + } + return total, nil +} + +func printPositions(consumer subscription.SubscriptionPullConsumer, topic string) error { + positions, err := consumer.Positions(topic) + if err != nil { + return err + } + committed, err := consumer.CommittedPositions(topic) + if err != nil { + return err + } + _, err = fmt.Printf("Positions: %+v\nCommitted positions: %+v\n", positions, committed) + return err +} + +func printMessage(ctx context.Context, message subscription.SubscriptionMessage) (int, error) { + batches, err := message.Records() + if err != nil { + return 0, err + } + if _, err := fmt.Printf("Commit context: %+v\n", message.CommitContext()); err != nil { + return 0, err + } + total := 0 + for _, batch := range batches { + if _, err := fmt.Printf("dialect=%s database=%s target=%s timeSelected=%t columns=%+v\n", batch.Dialect(), batch.Database(), batch.Target(), batch.TimeSelected(), batch.Columns()); err != nil { + return total, err + } + for row, err := range batch.Rows() { + if err != nil { + return total, err + } + if err := ctx.Err(); err != nil { + return total, err + } + if row.Timestamp == nil { + _, err = fmt.Printf("values=%v\n", row.Values) + } else { + _, err = fmt.Printf("time=%d values=%v\n", *row.Timestamp, row.Values) + } + if err != nil { + return total, err + } + total++ + } + } + return total, nil +} diff --git a/subscription/example/tree/main.go b/subscription/example/tree/main.go new file mode 100644 index 0000000..c9033f7 --- /dev/null +++ b/subscription/example/tree/main.go @@ -0,0 +1,280 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +// This example mirrors SubscriptionSessionExample.dataSubscription1 and adds +// record-based push delivery. The generated database is retained for inspection. +package main + +import ( + "context" + "errors" + "flag" + "fmt" + "log" + "net" + "os" + "os/signal" + "strconv" + "syscall" + "time" + + "github.com/apache/iotdb-client-go/v2/client" + "github.com/apache/iotdb-client-go/subscription" +) + +func main() { + host := flag.String("host", "127.0.0.1", "IoTDB host") + port := flag.Int("port", 6667, "IoTDB RPC port") + username := flag.String("username", "root", "IoTDB username") + consumer := flag.String("consumer", "pull", "Consumer type: pull or push") + ack := flag.String("ack", "after", "Push acknowledgment: after or before") + duration := flag.Duration("duration", 10*time.Second, "Push consumption duration; ignored by pull") + flag.Parse() + if *port < 1 || *port > 65535 || (*consumer != "pull" && *consumer != "push") || + (*ack != "after" && *ack != "before") || (*consumer == "push" && *duration <= 0) { + log.Fatal("Invalid port, consumer, acknowledgment strategy, or push duration") + } + password, supplied := os.LookupEnv("IOTDB_PASSWORD") + if !supplied { + password = "root" + } + ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM) + defer stop() + if err := run(ctx, *host, *port, *username, password, *consumer, *ack, *duration); err != nil { + log.Print(err) + os.Exit(1) + } +} + +func run(ctx context.Context, host string, port int, username, password, kind, ack string, duration time.Duration) (result error) { + runID := strconv.FormatInt(time.Now().UnixNano(), 10) + database := "root.subscription_tree_" + runID + device := database + ".d1" + topic := "subscription_tree_" + runID + if _, err := fmt.Printf("Database retained for inspection: %s; device=%s; topic=%s\n", database, device, topic); err != nil { + return err + } + if err := prepareData(ctx, host, port, username, password, database, device); err != nil { + return err + } + cfg := subscription.DefaultSubscriptionSessionConfig() + cfg.Connection.NodeURLs = []string{net.JoinHostPort(host, strconv.Itoa(port))} + cfg.Connection.Username, cfg.Connection.Password = username, password + manager, err := subscription.NewSubscriptionSession(cfg) + if err != nil { + return err + } + created := false + defer func() { + cleanupCtx, cancel := context.WithTimeout(context.Background(), 10*time.Second) + defer cancel() + if created { + result = errors.Join(result, manager.DropTopic(cleanupCtx, topic, subscription.DropOptions{IfExists: true})) + } + result = errors.Join(result, manager.Close()) + }() + if err := manager.Open(ctx); err != nil { + return err + } + if err := manager.CreateTopic(ctx, topic, subscription.CreateTopicOptions{Attributes: map[string]string{ + "mode": "initial", + "format": "SubscriptionRecordHandler", + "path": device + ".s1", + "start-time": "25", + "end-time": "75", + }}); err != nil { + return err + } + created = true + if kind == "push" { + return consumePush(ctx, cfg.Connection, manager, topic, ack, duration) + } + return consumePull(ctx, cfg.Connection, manager, topic) +} + +func prepareData(ctx context.Context, host string, port int, username, password, database, device string) (result error) { + session := client.NewSession(&client.Config{Host: host, Port: strconv.Itoa(port), UserName: username, Password: password}) + if err := session.Open(false, 10000); err != nil { + return err + } + defer func() { result = errors.Join(result, session.Close()) }() + for _, sql := range []string{ + "CREATE DATABASE " + database, + "CREATE TIMESERIES " + device + ".s1 WITH DATATYPE=INT64, ENCODING=RLE", + "CREATE TIMESERIES " + device + ".s2 WITH DATATYPE=INT64, ENCODING=RLE", + } { + if err := ctx.Err(); err != nil { + return err + } + if err := session.ExecuteNonQueryStatement(sql); err != nil { + return err + } + } + for timestamp := int64(0); timestamp < 100; timestamp++ { + if err := ctx.Err(); err != nil { + return err + } + if err := session.InsertRecord(device, []string{"s1", "s2"}, []client.TSDataType{client.INT64, client.INT64}, []interface{}{int64(1), int64(2)}, timestamp); err != nil { + return err + } + } + return session.ExecuteNonQueryStatement("FLUSH") +} + +func consumePull(ctx context.Context, connection subscription.ConnectionConfig, manager subscription.SubscriptionSession, topic string) (result error) { + cfg := subscription.DefaultSubscriptionPullConsumerConfig() + cfg.Consumer.Connection = connection + consumer, err := subscription.NewSubscriptionPullConsumer(cfg) + if err != nil { + return err + } + // Close flushes pending automatic commits while the subscription still exists. + defer func() { result = errors.Join(result, consumer.Close()) }() + if err := consumer.Open(ctx); err != nil { + return err + } + if err := consumer.Subscribe(ctx, topic); err != nil { + return err + } + if err := printManagement(ctx, manager, topic); err != nil { + return err + } + emptyPolls := 0 + for emptyPolls < 3 && !consumer.AllConsumed() { + messages, err := consumer.Poll(ctx, 10*time.Second) + if err != nil { + return err + } + if len(messages) == 0 { + emptyPolls++ + } else { + emptyPolls = 0 + } + for index, message := range messages { + err := printMessage(ctx, message) + message.Release() + if err != nil { + for _, remaining := range messages[index+1:] { + remaining.Release() + } + return err + } + } + } + return nil +} + +func consumePush(ctx context.Context, connection subscription.ConnectionConfig, manager subscription.SubscriptionSession, topic, ack string, duration time.Duration) (result error) { + cfg := subscription.DefaultSubscriptionPushConsumerConfig() + cfg.Consumer.Connection = connection + if ack == "before" { + cfg.AckStrategy = subscription.AckBeforeConsume + } + // The delivery loop owns messages; a push Handler must never call Release. + cfg.Handler = printMessage + consumer, err := subscription.NewSubscriptionPushConsumer(cfg) + if err != nil { + return err + } + defer func() { + result = errors.Join(result, consumer.Close()) + // Close cancels callbacks but may return before the Handler finishes. + waitCtx, cancel := context.WithTimeout(context.Background(), 10*time.Second) + defer cancel() + for { + select { + case failure, ok := <-consumer.Errors(): + if !ok { + return + } + result = errors.Join(result, failure) + case <-waitCtx.Done(): + result = errors.Join(result, waitCtx.Err()) + return + } + } + }() + if err := consumer.Open(ctx); err != nil { + return err + } + if err := consumer.Subscribe(ctx, topic); err != nil { + return err + } + if err := printManagement(ctx, manager, topic); err != nil { + return err + } + timer := time.NewTimer(duration) + defer timer.Stop() + select { + case err, ok := <-consumer.Errors(): + if ok { + return err + } + return nil + case <-ctx.Done(): + return ctx.Err() + case <-timer.C: + return nil + } +} + +func printManagement(ctx context.Context, manager subscription.SubscriptionSession, topic string) error { + topics, err := manager.Topics(ctx) + if err != nil { + return err + } + subscriptions, err := manager.Subscriptions(ctx, topic) + if err != nil { + return err + } + _, err = fmt.Printf("Topics: %+v\nSubscriptions for %s: %+v\n", topics, topic, subscriptions) + return err +} + +func printMessage(ctx context.Context, message subscription.SubscriptionMessage) error { + batches, err := message.Records() + if err != nil { + return err + } + if _, err := fmt.Printf("Commit context: %+v\n", message.CommitContext()); err != nil { + return err + } + for _, batch := range batches { + if _, err := fmt.Printf("dialect=%s database=%s target=%s timeSelected=%t columns=%+v\n", batch.Dialect(), batch.Database(), batch.Target(), batch.TimeSelected(), batch.Columns()); err != nil { + return err + } + for row, err := range batch.Rows() { + if err != nil { + return err + } + if err := ctx.Err(); err != nil { + return err + } + if row.Timestamp == nil { + _, err = fmt.Printf("values=%v\n", row.Values) + } else { + _, err = fmt.Printf("time=%d values=%v\n", *row.Timestamp, row.Values) + } + if err != nil { + return err + } + } + } + return nil +} diff --git a/subscription/session.go b/subscription/subscription_session.go similarity index 100% rename from subscription/session.go rename to subscription/subscription_session.go diff --git a/subscription/session_config.go b/subscription/subscription_session_config.go similarity index 100% rename from subscription/session_config.go rename to subscription/subscription_session_config.go diff --git a/subscription/session_connection.go b/subscription/subscription_session_connection.go similarity index 100% rename from subscription/session_connection.go rename to subscription/subscription_session_connection.go diff --git a/subscription/session_impl.go b/subscription/subscription_session_impl.go similarity index 100% rename from subscription/session_impl.go rename to subscription/subscription_session_impl.go diff --git a/subscription/session_rows.go b/subscription/subscription_session_rows.go similarity index 100% rename from subscription/session_rows.go rename to subscription/subscription_session_rows.go diff --git a/subscription/session_sql.go b/subscription/subscription_session_sql.go similarity index 100% rename from subscription/session_sql.go rename to subscription/subscription_session_sql.go
