This is an automated email from the ASF dual-hosted git repository. shuwenwei pushed a commit to branch supports_subscription in repository https://gitbox.apache.org/repos/asf/iotdb-client-go.git
commit 16bf05838385de1f8a6689f7876a451a1edd6104 Author: shuwenwei <[email protected]> AuthorDate: Fri Nov 13 11:25:46 2026 +0800 Implement subscription push consumer --- subscription/README.md | 71 ++++++- subscription/config.go | 19 +- subscription/consumer.go | 34 +++- subscription/doc.go | 7 +- subscription/pull_consumer.go | 10 + subscription/push_consumer.go | 449 ++++++++++++++++++++++++++++++++++++++++++ 6 files changed, 568 insertions(+), 22 deletions(-) diff --git a/subscription/README.md b/subscription/README.md index f470148..aa49988 100644 --- a/subscription/README.md +++ b/subscription/README.md @@ -19,7 +19,7 @@ under the License. # Go Subscription API -The module implements `SubscriptionSession` management and the configuration, construction, subscription management, record polling, commits, progress, seek, processor pipeline, and background connection coordination of `SubscriptionPullConsumer`. Poll response decoding, record decoding, and message release are also connected. Push consumers and built-in processors remain for subsequent steps. These implementations have not been compiled or exercised at runtime; this document describes t [...] +The module implements `SubscriptionSession` management, Pull consumption with record polling, commits, progress, seek, processor pipelines, and background connection coordination, and Push consumption with sequential callbacks and acknowledgment strategies. Poll response decoding, record decoding, and message release are also connected. Built-in processors remain for a subsequent step. These implementations have not been compiled or exercised at runtime; this document describes the code [...] The module path is `github.com/apache/iotdb-client-go/subscription`. It uses Go 1.25 and maintains dependencies and versions separately from the existing `github.com/apache/iotdb-client-go/v2` Session module. Its direct dependencies are `github.com/apache/iotdb-client-go/v2` at `v2.0.0-20260914074454-a25f68aa9681`, which provides existing types such as `TSDataType` and `ColumnCategory`, and `github.com/apache/thrift` at `v0.24.0`, which provides the RPC transport and protocol. It adds no [...] @@ -43,7 +43,7 @@ Only the public method contract is shared. Management sessions pass the model wh | `SubscriptionPushConsumerConfig` | `AbstractSubscriptionPushConsumerBuilder` | Combines a Handler, ACK strategy, and polling interval and timeout | | `AckStrategy`, `AckAfterConsume`, `AckBeforeConsume` | `AckStrategy.AFTER_CONSUME`, `BEFORE_CONSUME` | The zero value acknowledges after consumption; callback failure does not cause redelivery when acknowledgment precedes consumption | -`SubscriptionConsumerConfig.OwnerEpoch == nil` means that no epoch was supplied; a pointer to zero explicitly supplies epoch 0. `DefaultSubscriptionConsumerConfig`, `DefaultSubscriptionPullConsumerConfig`, and `DefaultSubscriptionPushConsumerConfig` are available. The default Push Handler remains nil, so callers must supply a real handler. Internal normalization copies endpoint slices and optional pointers. Zero durations select defaults, negative durations return errors, and positive va [...] +`SubscriptionConsumerConfig.OwnerEpoch == nil` means that no epoch was supplied; a pointer to zero explicitly supplies epoch 0. `DefaultSubscriptionConsumerConfig`, `DefaultSubscriptionPullConsumerConfig`, and `DefaultSubscriptionPushConsumerConfig` are available. The default Push Handler remains nil, so callers must supply a real handler. Internal normalization copies endpoint slices and optional pointers. Zero durations select defaults, negative durations return errors, and positive va [...] The management session defaults are `localhost:6667`, the tree model, root/root authentication, a 64 MiB frame limit, and a ten-second connection and authentication timeout. Java management sessions do not limit connection time by default; the Go entry point uses a bounded timeout here. The constructor fills empty endpoints, username, and dialect, as well as zero timeout and frame size, validates addresses and mutually exclusive authentication, and copies the endpoint slice. Passwords ar [...] @@ -65,7 +65,7 @@ Management connections own query pagination and closing while reusing the existi ## Internal Subscription Protocol and Providers -The internal implementation covers handshakes, heartbeats, subscribe/unsubscribe, close, commits, seek, Poll and Tablet-continuation requests, and oversized-request slicing. These are connected to the public Pull consumer. +The internal implementation covers handshakes, heartbeats, subscribe/unsubscribe, close, commits, seek, Poll and Tablet-continuation requests, and oversized-request slicing. Pull and Push share these internal capabilities. - Primitive encoding uses Java's big-endian layout. Commit contexts use version 2 and retain legacy commit fields plus optional Writer identity and progress. Topic/Region/Writer progress follows the original protocol structure. - Topic sets in subscribe and Poll requests use Java ObjectSet string tags, separately from the ordinary StringList used by heartbeat unsubscribe lists. Set encoding is sorted for stability. Null strings map to empty strings in existing Go string fields. @@ -136,6 +136,61 @@ return consumer.Commit(ctx, messages...) - Heartbeats are scheduled independently per Provider. Endpoint synchronization and connection failure trigger recovery. Background recovery attempts are at least five seconds apart. Late old-connection responses cannot reactivate an old Provider or incorrectly fence the replacement connection's Consumer. - `Close` interrupts blocked polling and stops background work. In manual mode, buffered processor messages make closing return an error while preserving the drain path. Call `DrainBufferedMessages`, process and commit its records, then close again. Automatic mode makes a best-effort commit of delivered and drained records during closing. Commit failures return errors; a successful close does not imply that all historical messages were processed. +## Push Consumption + +`NewSubscriptionPushConsumer` normalizes configuration once, requires a non-nil Handler, and does not connect. It reuses the manual Pull core for providers, progress, and polling, without installing processors or enabling automatic commits. `Open(ctx)` starts one fixed-delay polling loop; its background lifetime is independent of the opening context. Handlers run sequentially, and the next round starts only after the current round and `PollInterval` delay finish. + +```go +cfg := subscription.DefaultSubscriptionPushConsumerConfig() +cfg.Consumer.Dialect = subscription.TableDialect +cfg.AckStrategy = subscription.AckAfterConsume +cfg.Handler = func(callbackCtx context.Context, message subscription.SubscriptionMessage) error { + batches, err := message.Records() + if err != nil { + return err + } + for _, batch := range batches { + for row, err := range batch.Rows() { + if err != nil { + return err + } + if err := callbackCtx.Err(); err != nil { + return err + } + fmt.Println(row.Timestamp, row.Values) + } + } + return nil +} +consumer, err := subscription.NewSubscriptionPushConsumer(cfg) +if err != nil { + return err +} +if err := consumer.Open(ctx); err != nil { + return err +} +defer consumer.Close() +if err := consumer.Subscribe(ctx, "measurements"); err != nil { + return err +} +select { +case err, ok := <-consumer.Errors(): + if !ok { + return nil + } + return err +case <-ctx.Done(): + return ctx.Err() +} +``` + +- `AckBeforeConsume` acknowledges the whole batch before invoking any Handler. A partial or failed ACK prevents callbacks for that batch. A later Handler error or panic does not trigger NACK under this strategy. +- `AckAfterConsume` runs all callbacks in order, then acknowledges the successful group and nacks the failed group. An ACK error skips the subsequent NACK request. A nil Handler result means success; an error or panic means consumption failed, not that the background loop has terminated. +- Ordinary network failures are retried in a later polling round. TsFile decoding, fencing, and critical background errors stop delivery. `Errors()` buffers one result, sends the first terminal error once, and is closed by the loop when it exits. Normal closing does not invent a terminal error. A TsFile failure never enters the Handler or acknowledgment path. +- Handlers run without an internal consumer operation gate and may call Subscribe, Seek, or Close. Push seek invalidates pending delivery for the target topic before sending the request. That topic's active callback context is canceled after the seek request finishes, whether it succeeds or fails, so a Handler can use its own context to call Seek without canceling that request at entry. An already-started callback can run until it returns; later callbacks and ACKs for the old delivery ge [...] +- `Close` cancels the active callback context and synchronously closes core connection resources, but does not wait for user Handler code to return. This permits calling Close from inside a Handler without waiting on itself. The active message is released only after its callback returns, so borrowed data remains valid during that callback. Handlers should return promptly after cancellation. If a callback has not returned, the loop and `Errors()` channel may finish after Close returns. +- Push messages are owned by the delivery loop. Do not call `Release` in a Handler; copy any record data that must outlive the callback. Watermarks update consumer state but do not reach record Handlers. Push does not expose the Pull processor pipeline. + ## Management Sessions `SubscriptionSession` corresponds to Java's `ISubscriptionTreeSession` and `ISubscriptionTableSession`. Methods take an explicit `context.Context`, while parameter structures express existence conditions. @@ -159,6 +214,8 @@ return consumer.Commit(ctx, messages...) | Go method or type | Java counterpart | Consolidation | | --- | --- | --- | +| `NewSubscriptionPullConsumer(config)` | `SubscriptionTreePullConsumerBuilder.build`, `SubscriptionTablePullConsumerBuilder.build` | One constructor with `Dialect`; validates configuration without connecting | +| `NewSubscriptionPushConsumer(config)` | `SubscriptionTreePushConsumerBuilder.build`, `SubscriptionTablePushConsumerBuilder.build` | One constructor with `Dialect` and a required Go Handler; validates configuration without connecting | | `SubscriptionConsumer.Open(ctx)`, `Close()` | Each Consumer's `open/close` | Shared lifecycle interface | | `Subscribe(ctx, topics...)`, `Unsubscribe(ctx, topics...)` | Single-topic, array, and collection overloads | Variadic topic names | | `Info()`, `SubscriptionConsumerInfo` | `getConsumerId/getConsumerGroupId/getOwnerId/getOwnerEpoch` | Returns a credential-free identity snapshot, including server-assigned IDs | @@ -182,7 +239,7 @@ return consumer.Commit(ctx, messages...) `CommitAsync` returns a channel with capacity 1. It sends one result, nil on success, then closes. An unread result cannot block cleanup. Canceling the context does not undo commits accepted by the server, and a synchronous or asynchronous commit error does not mean that the whole batch failed. -The `SubscriptionPushConsumer.Errors()` channel buffers at least one error. A TsFile decoding error must stop automatic polling before notification; the failed round cannot enter the Handler or ACK path. The lifecycle owner closes the channel after coordinating background senders. +The `SubscriptionPushConsumer.Errors()` channel buffers one error. A TsFile decoding error stops automatic polling before notification; the failed round cannot enter the Handler or ACK path. The polling loop owns channel closure. If Close cancels a still-running Handler, channel closure can occur after Close returns, when that Handler returns and the loop finishes. ## Messages, Records, and Processors @@ -211,8 +268,8 @@ A `SubscriptionMessageProcessor` that retains messages must implement `Buffering - Batches, columns, and row data from `SubscriptionMessage.Records()` are borrowed, read-only views valid until the message's `Release()`. Callers must copy values and byte slices that need to outlive that point. - `Release()` is idempotent. Do not reuse borrowed batch, column, or row data after release. `Records()` returns `ErrDataReleased`, while kind, commit context, and watermark remain available. - Nil in `Row.Values` means SQL NULL. DATE uses `time.Time`; TIMESTAMP and record timestamps use `int64` in server precision; BLOB/OBJECT use `[]byte`. -- A Push Handler must not release its message. Copy record data before returning if it is needed afterward. The callback context is canceled when delivery stops. -- The `Open(ctx)` context bounds opening only, not the background lifetime. Background work stops with `Close()`. Other operation contexts bound their corresponding operations only. +- A Push Handler must not release its message. Copy record data before returning if it is needed afterward. The callback context is canceled when delivery stops or after a seek request for its topic finishes, regardless of that request's outcome. Close does not wait for the Handler; its message remains valid until the Handler returns. +- The `Open(ctx)` context bounds opening only, not the background lifetime. Close stops new work and cancels active work. Other operation contexts bound their corresponding operations only; Push user code can finish after Close returns. - A positive Pull timeout bounds foreground queue waiting and network polling, subject to an earlier context deadline. Zero has the single-round behavior described above; processor callbacks must return promptly. Operation timeouts use `time.Duration` without changing message timestamp units. - Repeated Consumer `Open` while open and repeated `Close` after closing are harmless. A successfully closed Consumer cannot reopen. A failed close requiring buffered data to be drained preserves the drain-and-commit recovery path. @@ -228,4 +285,4 @@ A `SubscriptionMessageProcessor` that retains messages must implement `Buffering Operation errors can wrap these sentinels; callers identify them with `errors.Is`. Unsupported TsFile errors must not trigger connection retries or file-payload acknowledgments. Pull returns an error and no messages for that round, including records obtained by parallel branches. That round must not advance positions or enter processor or automatic-commit buffers. Previously delivered records and confirmed progress are not rolled back. Callers can unsubscribe the offending topic and poll again. -Remaining work includes Push consumers, built-in Watermark/ColumnAlign processors, further consumption examples, and behavior tests. Management queries reuse the root module's decoder; this step does not change its handling of malformed data. Verification has been static only: no compilation, tests, or formatting tools have run, and no runtime validation result is claimed. +The remaining implementation step is the built-in Watermark/ColumnAlign processors; further consumption examples and behavior tests are also pending. Management queries reuse the root module's decoder; this step does not change its handling of malformed data. Verification has been static only: no compilation, tests, or formatting tools have run, and no runtime validation result is claimed. diff --git a/subscription/config.go b/subscription/config.go index 9039c4c..d1fe2fa 100644 --- a/subscription/config.go +++ b/subscription/config.go @@ -116,10 +116,12 @@ type SubscriptionPullConsumerConfig struct { type AckStrategy uint8 const ( - // AckAfterConsume acknowledges successful callbacks and nacks failed ones. + // AckAfterConsume runs callbacks sequentially, acknowledges the successful + // group, then nacks the failed group. An ACK error skips the subsequent NACK. AckAfterConsume AckStrategy = iota - // AckBeforeConsume acknowledges before invoking callbacks; callback failure - // does not cause redelivery under this strategy. + // AckBeforeConsume acknowledges the entire batch before invoking callbacks. + // A partial or failed ACK prevents callbacks for that batch; callback failure + // does not cause a NACK or redelivery under this strategy. AckBeforeConsume ) @@ -131,11 +133,14 @@ type SubscriptionPushConsumerConfig struct { // AckStrategy selects acknowledgment before delivery or after successful // processing. Its zero value is AckAfterConsume. AckStrategy AckStrategy - // Handler receives record messages. It is required; nil returned by a call - // means successful processing, and a non-nil error means processing failed. + // Handler receives record messages sequentially. It is required; nil returned + // by a call means success, while an error or panic means consumption failed. + // It must not release the borrowed message and should return promptly when + // its context is canceled. Close does not wait for Handler to return. Handler ConsumeFunc - // PollInterval controls the interval between automatic polling work. Java - // defaults to 100 milliseconds and requires an interval greater than zero. + // PollInterval is the fixed delay after one polling and delivery round + // finishes before another starts. Java defaults to 100 milliseconds and + // requires an interval greater than zero. PollInterval time.Duration // PollTimeout limits one automatic poll. Java defaults to ten seconds and // enforces a one-second minimum. Message timestamps retain server precision. diff --git a/subscription/consumer.go b/subscription/consumer.go index 23b5b3e..8097f6a 100644 --- a/subscription/consumer.go +++ b/subscription/consumer.go @@ -35,7 +35,9 @@ type SubscriptionConsumer interface { // work. It returns an error if opening fails. The context limits this // operation only; canceling it after Open returns does not stop consumption. Open(ctx context.Context) error - // Close stops background work and releases connections. In manual-commit + // Close stops background work and releases connections. Push cancels the + // current callback context but does not wait for the callback to return; + // that callback's message remains valid until it returns. In manual-commit // mode, buffered processor output must first be drained, processed, and // committed before retrying a close that reports undelivered messages. // Closing does not guarantee that every outstanding commit succeeded. @@ -125,21 +127,39 @@ type SubscriptionPullConsumer interface { // ConsumeFunc handles a delivered record message. A nil error means success; // a non-nil error means failure. Acknowledgment and redelivery depend on the // configured AckStrategy rather than on the callback alone. -// The callback context is canceled when the consumer stops delivery. The message -// remains valid for the callback; the callback must not release it or retain +// Callbacks run sequentially and without holding an internal consumer operation +// gate. They may call Subscribe, Seek, or Close. A panic is treated as a failed +// consumption attempt, not as a terminal background error. +// The callback context is canceled when delivery stops or after a seek request +// for its topic finishes, whether that request succeeds or fails. Close does +// not wait for the callback, so it should return promptly when +// its context is canceled. The message remains valid for the callback; the +// callback must not release it or retain // borrowed record data after returning without copying that data. type ConsumeFunc func(ctx context.Context, message SubscriptionMessage) error -// SubscriptionPushConsumer polls in the background and delivers records to SubscriptionPushConsumerConfig.Handler. +// SubscriptionPushConsumer polls in the background and delivers records to +// SubscriptionPushConsumerConfig.Handler. One fixed-delay loop invokes handlers +// sequentially and does not install record processors. Before-consumption ACK +// confirms the whole batch before any callback; a failed ACK prevents delivery. +// After-consumption ACK confirms successful callbacks before nacking failed ones. +// A failed ACK skips that batch's subsequent NACK request. // A TsFile decoding error stops automatic polling before it is reported; no // record callback or acknowledgment is performed for that failed poll round. +// Seeking invalidates pending delivery for the target topic before its request +// starts, then cancels that topic's active callback context after the request +// finishes. No new callbacks or ACKs are started for invalidated records, +// even if seek fails, because server positions might have changed partially. +// Already accepted acknowledgments are not undone. type SubscriptionPushConsumer interface { SubscriptionConsumer - // Errors reports the first terminal background error, then delivery stops. + // Errors reports the first terminal background error after new delivery stops. // The channel has capacity at least one; the error is sent once even if the - // caller is not receiving. The lifecycle owner closes it after all senders - // have stopped. An ordinary callback failure is not a terminal error. + // caller is not receiving. The polling loop closes it when that loop exits. + // A Close during an active callback may return before this + // channel closes. An ordinary callback error or panic is not terminal; + // TsFile decoding, fencing, and critical background errors stop delivery. Errors() <-chan error } diff --git a/subscription/doc.go b/subscription/doc.go index 6f68007..9cb40cd 100644 --- a/subscription/doc.go +++ b/subscription/doc.go @@ -26,7 +26,9 @@ // record views, message release, and watermark messages are implemented internally. // NewSubscriptionPullConsumer provides pull consumption, record continuation, // commit and seek coordination, background heartbeats, and provider recovery. -// Push consumers and built-in processors remain subsequent implementation steps. +// NewSubscriptionPushConsumer adds sequential callback delivery with before- or +// after-consumption acknowledgment and a terminal-error channel. Built-in +// processors remain a subsequent implementation step. // // Record messages are the supported payload format. Topic management and // subscription do not reject TsFile topics; decoding a TsFile payload must return @@ -36,4 +38,7 @@ // Contexts passed to Open limit the opening operation. Background work belongs // to the consumer and stops on Close, rather than when that context is canceled. // Other operation contexts bound the corresponding operation only. +// Push Close cancels the active callback context and closes connection resources +// without waiting for user code to return. Its borrowed message is released only +// after that callback returns; the error channel can therefore close later. package subscription diff --git a/subscription/pull_consumer.go b/subscription/pull_consumer.go index b8ecbbc..78d081b 100644 --- a/subscription/pull_consumer.go +++ b/subscription/pull_consumer.go @@ -42,6 +42,16 @@ func NewSubscriptionPullConsumer(config SubscriptionPullConsumerConfig) (Subscri if err != nil { return nil, err } + consumer, err := newSubscriptionPullConsumer(resolved) + if err != nil { + return nil, err + } + return consumer, nil +} + +// newSubscriptionPullConsumer accepts normalized configuration so push consumers +// can reuse the connection and progress engine without parsing identities twice. +func newSubscriptionPullConsumer(resolved SubscriptionPullConsumerConfig) (*subscriptionPullConsumer, error) { var suffix [8]byte if _, err := rand.Read(suffix[:]); err != nil { return nil, fmt.Errorf("subscription: generate consumer instance identity: %w", err) diff --git a/subscription/push_consumer.go b/subscription/push_consumer.go new file mode 100644 index 0000000..89f40e6 --- /dev/null +++ b/subscription/push_consumer.go @@ -0,0 +1,449 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package subscription + +import ( + "context" + "errors" + "fmt" + "sync" + "time" +) + +// NewSubscriptionPushConsumer copies and validates config without connecting. +// Handler is required. Open starts sequential background delivery with the +// configured acknowledgment strategy; its context limits opening only. +// Close cancels delivery and closes connections without waiting for an active +// Handler. Its message remains valid until that Handler returns. +func NewSubscriptionPushConsumer(config SubscriptionPushConsumerConfig) (SubscriptionPushConsumer, error) { + resolved, err := normalizeSubscriptionPushConsumerConfig(config) + if err != nil { + return nil, err + } + autoCommit := false + consumer, err := newSubscriptionPullConsumer(SubscriptionPullConsumerConfig{ + Consumer: resolved.Consumer, + AutoCommit: &autoCommit, + AutoCommitInterval: DefaultSubscriptionPullConsumerConfig().AutoCommitInterval, + }) + if err != nil { + return nil, err + } + return &subscriptionPushConsumer{ + config: resolved, + consumer: consumer, + operations: make(chan struct{}, 1), + generations: make(map[string]uint64), + errors: make(chan error, 1), + }, nil +} + +type subscriptionPushConsumer struct { + config SubscriptionPushConsumerConfig + consumer *subscriptionPullConsumer + operations chan struct{} + mu sync.Mutex + generations map[string]uint64 + started bool + closed bool + terminal error + deliveryCancel context.CancelFunc + callbackTopic string + callbackCancel context.CancelFunc + errors chan error + errorsClosed sync.Once +} + +type subscriptionPushDelivery struct { + message SubscriptionMessage + generation uint64 +} + +func (c *subscriptionPushConsumer) Open(ctx context.Context) error { + if ctx == nil { + return errors.New("subscription: context must not be nil") + } + if err := ctx.Err(); err != nil { + return err + } + c.mu.Lock() + err := c.deliveryStateLocked() + c.mu.Unlock() + if err != nil { + return err + } + if err := c.consumer.Open(ctx); err != nil { + return err + } + c.mu.Lock() + if err := c.deliveryStateLocked(); err != nil { + c.mu.Unlock() + return err + } + if c.started { + c.mu.Unlock() + return nil + } + c.consumer.mu.RLock() + lifeCtx := c.consumer.lifeCtx + c.consumer.mu.RUnlock() + deliveryCtx, cancel := context.WithCancel(lifeCtx) + c.deliveryCancel = cancel + c.started = true + c.mu.Unlock() + go c.runDelivery(deliveryCtx) + return nil +} + +func (c *subscriptionPushConsumer) Close() error { + c.mu.Lock() + if c.closed { + c.mu.Unlock() + return nil + } + c.closed = true + if c.deliveryCancel != nil { + c.deliveryCancel() + } + started := c.started + c.mu.Unlock() + // The loop is not a core worker: a Handler may close its own consumer. + // It retains ownership of its messages until the Handler has returned. + err := c.consumer.Close() + if !started { + c.closeErrors() + } + return err +} + +func (c *subscriptionPushConsumer) Errors() <-chan error { + return c.errors +} + +func (c *subscriptionPushConsumer) Info() SubscriptionConsumerInfo { + return c.consumer.Info() +} + +func (c *subscriptionPushConsumer) LatestWatermark() (int64, bool) { + return c.consumer.LatestWatermark() +} + +func (c *subscriptionPushConsumer) AllConsumed() bool { + return c.consumer.AllConsumed() +} + +func (c *subscriptionPushConsumer) Subscribe(ctx context.Context, topics ...string) error { + return c.consumer.Subscribe(ctx, topics...) +} + +func (c *subscriptionPushConsumer) Unsubscribe(ctx context.Context, topics ...string) error { + return c.consumer.Unsubscribe(ctx, topics...) +} + +func (c *subscriptionPushConsumer) Positions(topic string) (TopicProgress, error) { + return c.consumer.Positions(topic) +} + +func (c *subscriptionPushConsumer) CommittedPositions(topic string) (TopicProgress, error) { + return c.consumer.CommittedPositions(topic) +} + +func (c *subscriptionPushConsumer) SeekToBeginning(ctx context.Context, topic string) error { + return c.seek(ctx, topic, subscriptionSeekBeginning, nil) +} + +func (c *subscriptionPushConsumer) SeekToEnd(ctx context.Context, topic string) error { + return c.seek(ctx, topic, subscriptionSeekEnd, nil) +} + +func (c *subscriptionPushConsumer) Seek(ctx context.Context, topic string, progress TopicProgress) error { + return c.seek(ctx, topic, subscriptionSeekProgress, progress) +} + +func (c *subscriptionPushConsumer) SeekAfter(ctx context.Context, topic string, progress TopicProgress) error { + return c.seek(ctx, topic, subscriptionSeekAfterProgress, progress) +} + +func (c *subscriptionPushConsumer) seek(ctx context.Context, topic string, kind int16, progress TopicProgress) error { + name, err := parseSubscriptionIdentifier(topic) + if err != nil { + return err + } + if err := c.acquireOperations(ctx); err != nil { + return err + } + defer c.releaseOperations() + if err := c.consumer.checkOpen(); err != nil { + return err + } + if err := ctx.Err(); err != nil { + return err + } + c.mu.Lock() + if c.closed { + c.mu.Unlock() + return ErrClosed + } + // Even a failed multi-provider seek can change server state. Never confirm + // or start another callback for an older delivery of the affected topic. + c.generations[name]++ + var cancelCallback context.CancelFunc + if c.callbackTopic == name { + cancelCallback = c.callbackCancel + } + c.mu.Unlock() + // Keep a callback's context usable for a seek invoked from that callback. + // Cancel the obsolete delivery after the request completes instead. + if cancelCallback != nil { + defer cancelCallback() + } + return c.consumer.seekProgress(ctx, topic, kind, progress) +} + +func (c *subscriptionPushConsumer) acquireOperations(ctx context.Context) error { + if ctx == nil { + return errors.New("subscription: context must not be nil") + } + if err := ctx.Err(); err != nil { + return err + } + select { + case c.operations <- struct{}{}: + return nil + case <-ctx.Done(): + return ctx.Err() + } +} + +func (c *subscriptionPushConsumer) releaseOperations() { + <-c.operations +} + +func (c *subscriptionPushConsumer) deliveryStateLocked() error { + if c.closed { + return ErrClosed + } + return c.terminal +} + +func (c *subscriptionPushConsumer) pollDeliveries(ctx context.Context) ([]subscriptionPushDelivery, error) { + if err := c.acquireOperations(ctx); err != nil { + return nil, err + } + defer c.releaseOperations() + c.mu.Lock() + err := c.deliveryStateLocked() + c.mu.Unlock() + if err != nil { + return nil, err + } + messages, err := c.consumer.Poll(ctx, c.config.PollTimeout) + if err != nil { + return nil, err + } + c.mu.Lock() + defer c.mu.Unlock() + if err := c.deliveryStateLocked(); err != nil { + for _, message := range messages { + message.Release() + } + return nil, err + } + deliveries := make([]subscriptionPushDelivery, 0, len(messages)) + for _, message := range messages { + deliveries = append(deliveries, subscriptionPushDelivery{ + message: message, + generation: c.generations[message.CommitContext().Topic], + }) + } + return deliveries, nil +} + +func (c *subscriptionPushConsumer) runDelivery(ctx context.Context) { + var terminal error + defer func() { c.finishDelivery(terminal) }() + for { + if err := ctx.Err(); err != nil { + terminal = c.terminalDeliveryError(err) + return + } + deliveries, err := c.pollDeliveries(ctx) + if err == nil { + err = c.deliver(ctx, deliveries) + } + if err != nil { + terminal = c.terminalDeliveryError(err) + if terminal != nil || ctx.Err() != nil { + return + } + } + // Wait after the entire round, including callbacks and acknowledgment. + // A slow round therefore never overlaps the next round. + timer := time.NewTimer(c.config.PollInterval) + select { + case <-ctx.Done(): + timer.Stop() + terminal = c.terminalDeliveryError(ctx.Err()) + return + case <-timer.C: + } + } +} + +func (c *subscriptionPushConsumer) deliver(ctx context.Context, deliveries []subscriptionPushDelivery) error { + defer func() { + for _, delivery := range deliveries { + delivery.message.Release() + } + }() + if c.config.AckStrategy == AckBeforeConsume { + if err := c.commitDeliveries(ctx, deliveries, false); err != nil { + return err + } + } + var succeeded, failed []subscriptionPushDelivery + for _, delivery := range deliveries { + if err := ctx.Err(); err != nil { + return err + } + callbackCtx, finish, current := c.beginCallback(ctx, delivery) + if !current { + continue + } + err := c.invokeHandler(callbackCtx, delivery.message) + finish() + delivery.message.Release() + if err == nil { + succeeded = append(succeeded, delivery) + } else { + failed = append(failed, delivery) + } + } + if c.config.AckStrategy == AckBeforeConsume { + return nil + } + // Java confirms successful callbacks before nacking failures. An ACK error + // leaves the failed group unconfirmed for server-side recovery. + if err := c.commitDeliveries(ctx, succeeded, false); err != nil { + return err + } + return c.commitDeliveries(ctx, failed, true) +} + +func (c *subscriptionPushConsumer) beginCallback(ctx context.Context, delivery subscriptionPushDelivery) (context.Context, func(), bool) { + topic := delivery.message.CommitContext().Topic + c.mu.Lock() + defer c.mu.Unlock() + if c.deliveryStateLocked() != nil || ctx.Err() != nil || c.generations[topic] != delivery.generation { + return nil, nil, false + } + callbackCtx, cancel := context.WithCancel(ctx) + c.callbackTopic = topic + c.callbackCancel = cancel + return callbackCtx, func() { + cancel() + c.mu.Lock() + c.callbackTopic = "" + c.callbackCancel = nil + c.mu.Unlock() + }, true +} + +func (c *subscriptionPushConsumer) invokeHandler(ctx context.Context, message SubscriptionMessage) (err error) { + defer func() { + if failure := recover(); failure != nil { + err = fmt.Errorf("subscription: push handler panicked: %v", failure) + } + }() + return c.config.Handler(ctx, message) +} + +func (c *subscriptionPushConsumer) commitDeliveries(ctx context.Context, deliveries []subscriptionPushDelivery, nack bool) error { + if len(deliveries) == 0 { + return nil + } + if err := c.acquireOperations(ctx); err != nil { + return err + } + defer c.releaseOperations() + c.mu.Lock() + if err := c.deliveryStateLocked(); err != nil { + c.mu.Unlock() + return err + } + messages := make([]SubscriptionMessage, 0, len(deliveries)) + for _, delivery := range deliveries { + if c.generations[delivery.message.CommitContext().Topic] == delivery.generation { + messages = append(messages, delivery.message) + } + } + c.mu.Unlock() + if len(messages) == 0 { + return nil + } + commitCtx, cancel := context.WithTimeout(ctx, c.config.Consumer.Connection.ConnectTimeout) + defer cancel() + operationCtx, finish, err := c.consumer.linkContext(commitCtx) + if err != nil { + return err + } + defer finish() + contexts, err := c.consumer.messageCommitContexts(messages) + if err != nil { + return err + } + return c.consumer.commitContexts(operationCtx, contexts, nack) +} + +func (c *subscriptionPushConsumer) terminalDeliveryError(failure error) error { + c.mu.Lock() + closed := c.closed + c.mu.Unlock() + if closed { + return nil + } + if err := c.consumer.checkOpen(); err != nil { + return err + } + if flowFatalPollError(failure) || flowCriticalPollError(failure) { + return failure + } + return nil +} + +func (c *subscriptionPushConsumer) finishDelivery(failure error) { + c.mu.Lock() + notify := failure != nil && !c.closed + if notify { + c.terminal = failure + } + if c.deliveryCancel != nil { + c.deliveryCancel() + } + c.mu.Unlock() + if notify { + c.errors <- failure + } + c.closeErrors() +} + +func (c *subscriptionPushConsumer) closeErrors() { + c.errorsClosed.Do(func() { close(c.errors) }) +}
