This is an automated email from the ASF dual-hosted git repository.
urfree pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/pulsar-site.git
The following commit(s) were added to refs/heads/main by this push:
new 7215eb052b2 Docs sync done from apache/pulsar(#f453e0a)
7215eb052b2 is described below
commit 7215eb052b25a513df278e6726d55aa8b7bdb480
Author: Pulsar Site Updater <[email protected]>
AuthorDate: Wed Sep 7 06:23:44 2022 +0000
Docs sync done from apache/pulsar(#f453e0a)
---
site2/website-next/docs/client-libraries-go.md | 661 +++++++++------------
site2/website-next/docs/client-libraries-java.md | 2 +-
site2/website-next/docs/cookbooks-compaction.md | 28 +-
site2/website-next/docs/functions-overview.md | 2 +-
site2/website-next/docs/getting-started-docker.md | 6 +-
site2/website-next/docs/getting-started-helm.md | 14 +-
site2/website-next/docs/getting-started-home.md | 8 +-
.../docs/getting-started-standalone.md | 271 ++-------
site2/website-next/docs/reference-metrics.md | 4 +-
.../docs/security-tls-authentication.md | 2 +
site2/website-next/sidebars.json | 7 +-
.../version-2.10.x/client-libraries-java.md | 6 +-
.../version-2.10.x/cookbooks-compaction.md | 32 +-
.../version-2.10.x/io-debezium-source.md | 4 +-
.../version-2.10.x/reference-configuration.md | 2 +-
.../security-policy-and-supported-versions.md | 10 +-
.../version-2.10.x/tiered-storage-filesystem.md | 8 +-
.../version-2.8.x/client-libraries-java.md | 2 +-
.../version-2.8.x/cookbooks-compaction.md | 32 +-
.../version-2.8.x/developing-binary-protocol.md | 3 +-
.../version-2.8.x/reference-configuration.md | 2 +-
.../version-2.8.x/tiered-storage-filesystem.md | 8 +-
.../version-2.9.x/cookbooks-compaction.md | 32 +-
.../version-2.9.x/io-debezium-source.md | 4 +-
.../version-2.9.x/reference-configuration.md | 2 +-
.../version-2.9.x/tiered-storage-filesystem.md | 6 +-
26 files changed, 484 insertions(+), 674 deletions(-)
diff --git a/site2/website-next/docs/client-libraries-go.md
b/site2/website-next/docs/client-libraries-go.md
index a7a72cdf2f2..7ab2b416aab 100644
--- a/site2/website-next/docs/client-libraries-go.md
+++ b/site2/website-next/docs/client-libraries-go.md
@@ -6,7 +6,7 @@ sidebar_label: "Go"
You can use Pulsar [Go client](https://github.com/apache/pulsar-client-go) to
create Pulsar [producers](#producers), [consumers](#consumers), and
[readers](#readers) in Golang.
-API docs are available on the
[Godoc](https://godoc.org/github.com/apache/pulsar-client-go/pulsar) page
+API docs are available on the
[Godoc](https://pkg.go.dev/github.com/apache/pulsar-client-go/pulsar) page
## Installation
@@ -65,27 +65,27 @@ pulsar+ssl://pulsar.us-west.example.com:6651
## Create a client
-To interact with Pulsar, you need a `Client` object first. You can create a
client object using the `NewClient` function, passing in a `ClientOptions`
object (more on configuration [below](#client-configuration)). Here's an
example:
+To interact with Pulsar, you need a
[`Client`](https://pkg.go.dev/github.com/apache/pulsar-client-go/pulsar#Client)
object first. You can create a client object using the
[`NewClient`](https://pkg.go.dev/github.com/apache/pulsar-client-go/pulsar#NewClient)
function, passing in a
[`ClientOptions`](https://pkg.go.dev/github.com/apache/pulsar-client-go/pulsar#ClientOptions)
object (more on configuration [below](#client-configuration)). Here's an
example:
```go
import (
- "log"
- "time"
+ "log"
+ "time"
- "github.com/apache/pulsar-client-go/pulsar"
+ "github.com/apache/pulsar-client-go/pulsar"
)
func main() {
- client, err := pulsar.NewClient(pulsar.ClientOptions{
- URL: "pulsar://localhost:6650",
- OperationTimeout: 30 * time.Second,
- ConnectionTimeout: 30 * time.Second,
- })
- if err != nil {
- log.Fatalf("Could not instantiate Pulsar client: %v", err)
- }
+ client, err := pulsar.NewClient(pulsar.ClientOptions{
+ URL: "pulsar://localhost:6650",
+ OperationTimeout: 30 * time.Second,
+ ConnectionTimeout: 30 * time.Second,
+ })
+ if err != nil {
+ log.Fatalf("Could not instantiate Pulsar client: %v", err)
+ }
- defer client.Close()
+ defer client.Close()
}
```
@@ -112,21 +112,7 @@ func main() {
}
```
-The following configurable parameters are available for Pulsar clients:
-
-| Name | Description | Default|
-| :-------- | :---------- |:---------- |
-| URL | Configure the service URL for the Pulsar service.<br /><br />If you
have multiple brokers, you can set multiple Pulsar cluster addresses for a
client. <br /><br />This parameter is **required**. |None |
-| ConnectionTimeout | Timeout for the establishment of a TCP connection | 30s |
-| OperationTimeout| Set the operation timeout. Producer-create, subscribe and
unsubscribe operations will be retried until this interval, after which the
operation will be marked as failed| 30s|
-| Authentication | Configure the authentication provider. Example:
`Authentication: NewAuthenticationTLS("my-cert.pem", "my-key.pem")` | no
authentication |
-| TLSTrustCertsFilePath | Set the path to the trusted TLS certificate file | |
-| TLSAllowInsecureConnection | Configure whether the Pulsar client accept
untrusted TLS certificate from broker | false |
-| TLSValidateHostname | Configure whether the Pulsar client verify the
validity of the host name from broker | false |
-| ListenerName | Configure the net model for VPC users to connect to the
Pulsar broker | |
-| MaxConnectionsPerBroker | Max number of connections to a single broker that
is kept in the pool | 1 |
-| CustomMetricsLabels | Add custom labels to all the metrics reported by this
client instance | |
-| Logger | Configure the logger used by the client | logrus.StandardLogger |
+All configurable parameters for `ClientOptions` are
[here](https://pkg.go.dev/github.com/apache/pulsar-client-go/pulsar#ClientOptions).
## Producers
@@ -134,121 +120,131 @@ Pulsar producers publish messages to Pulsar topics. You
can [configure](#produce
```go
producer, err := client.CreateProducer(pulsar.ProducerOptions{
- Topic: "my-topic",
+ Topic: "my-topic",
})
if err != nil {
- log.Fatal(err)
+ log.Fatal(err)
}
_, err = producer.Send(context.Background(), &pulsar.ProducerMessage{
- Payload: []byte("hello"),
+ Payload: []byte("hello"),
})
defer producer.Close()
if err != nil {
- fmt.Println("Failed to publish message", err)
+ fmt.Println("Failed to publish message", err)
}
fmt.Println("Published message")
```
### Producer operations
-Pulsar Go producers have the following methods available:
-
-Method | Description | Return type
-:------|:------------|:-----------
-`Topic()` | Fetches the producer's [topic](reference-terminology.md#topic)|
`string`
-`Name()` | Fetches the producer's name | `string`
-`Send(context.Context, *ProducerMessage)` | Publishes a [message](#messages)
to the producer's topic. This call will block until the message is successfully
acknowledged by the Pulsar broker, or an error will be thrown if the timeout
set using the `SendTimeout` in the producer's
[configuration](#producer-configuration) is exceeded. | (MessageID, error)
-`SendAsync(context.Context, *ProducerMessage, func(MessageID,
*ProducerMessage, error))`| Send a message, this call will be blocking until is
successfully acknowledged by the Pulsar broker. |
-`LastSequenceID()` | Get the last sequence id that was published by this
producer. his represent either the automatically assigned or custom sequence id
(set on the ProducerMessage) that was published and acknowledged by the broker.
| int64
-`Flush()`| Flush all the messages buffered in the client and wait until all
messages have been successfully persisted. | error
-`Close()` | Closes the producer and releases all resources allocated to it. If
`Close()` is called then no more messages will be accepted from the publisher.
This method will block until all pending publish requests have been persisted
by Pulsar. If an error is thrown, no pending writes will be retried. |
+All available methods of `Producer` interface are
[here](https://pkg.go.dev/github.com/apache/pulsar-client-go/pulsar#Producer).
### Producer Example
#### How to use message router in producer
```go
-client, err := NewClient(pulsar.ClientOptions{
- URL: serviceURL,
+client, err := pulsar.NewClient(pulsar.ClientOptions{
+ URL: "pulsar://localhost:6650",
})
if err != nil {
- log.Fatal(err)
+ log.Fatal(err)
}
defer client.Close()
-// Only subscribe on the specific partition
-consumer, err := client.Subscribe(pulsar.ConsumerOptions{
- Topic: "my-partitioned-topic-partition-2",
- SubscriptionName: "my-sub",
+producer, err := client.CreateProducer(pulsar.ProducerOptions{
+ Topic: "my-partitioned-topic",
+ MessageRouter: func(msg *pulsar.ProducerMessage, tm pulsar.TopicMetadata)
int {
+ fmt.Println("Topic has", tm.NumPartitions(), "partitions. Routing
message ", msg, " to partition 2.")
+ // always push msg to partition 2
+ return 2
+ },
})
if err != nil {
- log.Fatal(err)
+ log.Fatal(err)
}
-defer consumer.Close()
+defer producer.Close()
-producer, err := client.CreateProducer(pulsar.ProducerOptions{
- Topic: "my-partitioned-topic",
- MessageRouter: func(msg *ProducerMessage, tm TopicMetadata) int {
- fmt.Println("Routing message ", msg, " -- Partitions: ",
tm.NumPartitions())
- return 2
- },
-})
+for i := 0; i < 10; i++ {
+ if msgId, err := producer.Send(context.Background(),
&pulsar.ProducerMessage{
+ Payload: []byte(fmt.Sprintf("message-%d", i)),
+ }); err != nil {
+ log.Fatal(err)
+ } else {
+ log.Println("Published message: ", msgId)
+ }
+}
+// subscribe a specific partition of a topic
+// for demos only, not recommend to subscribe a specific partition
+consumer, err := client.Subscribe(pulsar.ConsumerOptions{
+ // pulsar partition is a special topic has the suffix '-partition-xx'
+ Topic: "my-partitioned-topic-partition-2",
+ SubscriptionName: "my-sub",
+ Type: pulsar.Shared,
+})
if err != nil {
- log.Fatal(err)
+ log.Fatal(err)
+}
+defer consumer.Close()
+
+for i := 0; i < 10; i++ {
+ msg, err := consumer.Receive(context.Background())
+ if err != nil {
+ log.Fatal(err)
+ }
+ fmt.Printf("Received message msgId: %#v -- content: '%s'\n", msg.ID(),
string(msg.Payload()))
+ consumer.Ack(msg)
}
-defer producer.Close()
```
#### How to use schema interface in producer
```go
type testJSON struct {
- ID int `json:"id"`
- Name string `json:"name"`
+ ID int `json:"id"`
+ Name string `json:"name"`
}
-```
-
-```go
var (
- exampleSchemaDef =
"{\"type\":\"record\",\"name\":\"Example\",\"namespace\":\"test\"," +
-
"\"fields\":[{\"name\":\"ID\",\"type\":\"int\"},{\"name\":\"Name\",\"type\":\"string\"}]}"
+ exampleSchemaDef =
"{\"type\":\"record\",\"name\":\"Example\",\"namespace\":\"test\"," +
+
"\"fields\":[{\"name\":\"ID\",\"type\":\"int\"},{\"name\":\"Name\",\"type\":\"string\"}]}"
)
-```
-```go
-client, err := NewClient(pulsar.ClientOptions{
- URL: "pulsar://localhost:6650",
+client, err := pulsar.NewClient(pulsar.ClientOptions{
+ URL: "pulsar://localhost:6650",
})
if err != nil {
- log.Fatal(err)
+ log.Fatal(err)
}
defer client.Close()
properties := make(map[string]string)
properties["pulsar"] = "hello"
-jsonSchemaWithProperties := NewJSONSchema(exampleSchemaDef, properties)
-producer, err := client.CreateProducer(ProducerOptions{
- Topic: "jsonTopic",
- Schema: jsonSchemaWithProperties,
+jsonSchemaWithProperties := pulsar.NewJSONSchema(exampleSchemaDef, properties)
+producer, err := client.CreateProducer(pulsar.ProducerOptions{
+ Topic: "jsonTopic",
+ Schema: jsonSchemaWithProperties,
})
-assert.Nil(t, err)
-_, err = producer.Send(context.Background(), &ProducerMessage{
- Value: &testJSON{
- ID: 100,
- Name: "pulsar",
- },
+if err != nil {
+ log.Fatal(err)
+}
+
+_, err = producer.Send(context.Background(), &pulsar.ProducerMessage{
+ Value: &testJSON{
+ ID: 100,
+ Name: "pulsar",
+ },
})
if err != nil {
- log.Fatal(err)
+ log.Fatal(err)
}
producer.Close()
```
@@ -256,58 +252,58 @@ producer.Close()
#### How to use delay relative in producer
```go
-client, err := NewClient(pulsar.ClientOptions{
- URL: "pulsar://localhost:6650",
+client, err := pulsar.NewClient(pulsar.ClientOptions{
+ URL: "pulsar://localhost:6650",
})
if err != nil {
- log.Fatal(err)
+ log.Fatal(err)
}
defer client.Close()
-topicName := newTopicName()
+topicName := "topic-1"
producer, err := client.CreateProducer(pulsar.ProducerOptions{
Topic: topicName,
DisableBatching: true,
})
if err != nil {
- log.Fatal(err)
+ log.Fatal(err)
}
defer producer.Close()
consumer, err := client.Subscribe(pulsar.ConsumerOptions{
- Topic: topicName,
- SubscriptionName: "subName",
- Type: Shared,
+ Topic: topicName,
+ SubscriptionName: "subName",
+ Type: pulsar.Shared,
})
if err != nil {
- log.Fatal(err)
+ log.Fatal(err)
}
defer consumer.Close()
ID, err := producer.Send(context.Background(), &pulsar.ProducerMessage{
- Payload: []byte(fmt.Sprintf("test")),
- DeliverAfter: 3 * time.Second,
+ Payload: []byte(fmt.Sprintf("test")),
+ DeliverAfter: 3 * time.Second,
})
if err != nil {
- log.Fatal(err)
+ log.Fatal(err)
}
fmt.Println(ID)
-ctx, canc := context.WithTimeout(context.Background(), 1*time.Second)
+ctx, cancel := context.WithTimeout(context.Background(), 1*time.Second)
msg, err := consumer.Receive(ctx)
if err != nil {
- log.Fatal(err)
+ log.Fatal(err)
}
fmt.Println(msg.Payload())
-canc()
+cancel()
-ctx, canc = context.WithTimeout(context.Background(), 5*time.Second)
+ctx, cancel = context.WithTimeout(context.Background(), 5*time.Second)
msg, err = consumer.Receive(ctx)
if err != nil {
- log.Fatal(err)
+ log.Fatal(err)
}
fmt.Println(msg.Payload())
-canc()
+cancel()
```
#### How to use Prometheus metrics in producer
@@ -319,7 +315,7 @@ Pulsar Go client registers client metrics using Prometheus.
This section demonst
```go
// Create a Pulsar client
client, err := pulsar.NewClient(pulsar.ClientOptions{
- URL: "pulsar://localhost:6650",
+ URL: "pulsar://localhost:6650",
})
if err != nil {
log.Fatal(err)
@@ -387,26 +383,7 @@ Now you can query Pulsar client metrics on Prometheus.
### Producer configuration
- Name | Description | Default
-| :-------- | :---------- |:---------- |
-| Topic | Topic specify the topic this consumer will subscribe to. This
argument is required when constructing the reader. | |
-| Name | Name specify a name for the producer. If not assigned, the system
will generate a globally unique name which can be access with
Producer.ProducerName(). | |
-| Properties | Properties attach a set of application defined properties to
the producer This properties will be visible in the topic stats | |
-| SendTimeout | SendTimeout set the timeout for a message that is not
acknowledged by the server | 30s |
-| DisableBlockIfQueueFull | DisableBlockIfQueueFull control whether Send and
SendAsync block if producer's message queue is full | false |
-| MaxPendingMessages| MaxPendingMessages set the max size of the queue holding
the messages pending to receive an acknowledgment from the broker. | |
-| HashingScheme | HashingScheme change the `HashingScheme` used to chose the
partition on where to publish a particular message. | JavaStringHash |
-| CompressionType | CompressionType set the compression type for the producer.
| not compressed |
-| CompressionLevel | Define the desired compression level. Options: Default,
Faster and Better | Default |
-| MessageRouter | MessageRouter set a custom message routing policy by passing
an implementation of MessageRouter | |
-| DisableBatching | DisableBatching control whether automatic batching of
messages is enabled for the producer. | false |
-| BatchingMaxPublishDelay | BatchingMaxPublishDelay set the time period within
which the messages sent will be batched | 1ms |
-| BatchingMaxMessages | BatchingMaxMessages set the maximum number of messages
permitted in a batch. | 1000 |
-| BatchingMaxSize | BatchingMaxSize sets the maximum number of bytes permitted
in a batch. | 128KB |
-| Schema | Schema set a custom schema type by passing an implementation of
`Schema` | bytes[] |
-| Interceptors | A chain of interceptors. These interceptors are called at
some points defined in the `ProducerInterceptor` interface. | None |
-| MaxReconnectToBroker | MaxReconnectToBroker set the maximum retry number of
reconnectToBroker | ultimate |
-| BatcherBuilderType | BatcherBuilderType sets the batch builder type. This is
used to create a batch container when batching is enabled. Options:
DefaultBatchBuilder and KeyBasedBatchBuilder | DefaultBatchBuilder |
+All available options of `ProducerOptions` are
[here](https://pkg.go.dev/github.com/apache/pulsar-client-go/pulsar#ProducerOptions).
## Consumers
@@ -414,168 +391,161 @@ Pulsar consumers subscribe to one or more Pulsar topics
and listen for incoming
```go
consumer, err := client.Subscribe(pulsar.ConsumerOptions{
- Topic: "topic-1",
- SubscriptionName: "my-sub",
- Type: pulsar.Shared,
+ Topic: "topic-1",
+ SubscriptionName: "my-sub",
+ Type: pulsar.Shared,
})
if err != nil {
- log.Fatal(err)
+ log.Fatal(err)
}
defer consumer.Close()
for i := 0; i < 10; i++ {
- msg, err := consumer.Receive(context.Background())
- if err != nil {
- log.Fatal(err)
- }
+ // may block here
+ msg, err := consumer.Receive(context.Background())
+ if err != nil {
+ log.Fatal(err)
+ }
- fmt.Printf("Received message msgId: %#v -- content: '%s'\n",
- msg.ID(), string(msg.Payload()))
+ fmt.Printf("Received message msgId: %#v -- content: '%s'\n",
+ msg.ID(), string(msg.Payload()))
- consumer.Ack(msg)
+ consumer.Ack(msg)
}
if err := consumer.Unsubscribe(); err != nil {
- log.Fatal(err)
+ log.Fatal(err)
}
```
### Consumer operations
-Pulsar Go consumers have the following methods available:
-
-Method | Description | Return type
-:------|:------------|:-----------
-`Subscription()` | Returns the consumer's subscription name | `string`
-`Unsubcribe()` | Unsubscribes the consumer from the assigned topic. Throws an
error if the unsubscribe operation is somehow unsuccessful. | `error`
-`Receive(context.Context)` | Receives a single message from the topic. This
method blocks until a message is available. | `(Message, error)`
-`Chan()` | Chan returns a channel from which to consume messages. | `<-chan
ConsumerMessage`
-`Ack(Message)` | [Acknowledges](reference-terminology.md#acknowledgment-ack) a
message to the Pulsar [broker](reference-terminology.md#broker) |
-`AckID(MessageID)` |
[Acknowledges](reference-terminology.md#acknowledgment-ack) a message to the
Pulsar [broker](reference-terminology.md#broker) by message ID |
-`ReconsumeLater(msg Message, delay time.Duration)` | ReconsumeLater mark a
message for redelivery after custom delay |
-`Nack(Message)` | Acknowledge the failure to process a single message. |
-`NackID(MessageID)` | Acknowledge the failure to process a single message. |
-`Seek(msgID MessageID)` | Reset the subscription associated with this consumer
to a specific message id. The message id can either be a specific message or
represent the first or last messages in the topic. | `error`
-`SeekByTime(time time.Time)` | Reset the subscription associated with this
consumer to a specific message publish time. | `error`
-`Close()` | Closes the consumer, disabling its ability to receive messages
from the broker |
-`Name()` | Name returns the name of consumer | `string`
+All available methods of `Consumer` interface are
[here](https://pkg.go.dev/github.com/apache/pulsar-client-go/pulsar#Consumer).
-### Receive example
-
-#### How to use regex consumer
+#### Create single-topic consumer
```go
-client, err := pulsar.NewClient(pulsar.ClientOptions{
- URL: "pulsar://localhost:6650",
-})
+client, err := pulsar.NewClient(pulsar.ClientOptions{URL:
"pulsar://localhost:6650"})
+if err != nil {
+ log.Fatal(err)
+}
defer client.Close()
-p, err := client.CreateProducer(pulsar.ProducerOptions{
- Topic: topicInRegex,
- DisableBatching: true,
+consumer, err := client.Subscribe(pulsar.ConsumerOptions{
+ // fill `Topic` field will create a single-topic consumer
+ Topic: "topic-1",
+ SubscriptionName: "my-sub",
+ Type: pulsar.Shared,
})
if err != nil {
- log.Fatal(err)
+ log.Fatal(err)
}
-defer p.Close()
+defer consumer.Close()
+```
-topicsPattern := fmt.Sprintf("persistent://%s/foo.*", namespace)
+#### Create regex-topic consumer
+
+```go
+client, err := pulsar.NewClient(pulsar.ClientOptions{
+ URL: "pulsar://localhost:6650",
+})
+defer client.Close()
+
+topicsPattern := "persistent://public/default/topic.*"
opts := pulsar.ConsumerOptions{
- TopicsPattern: topicsPattern,
- SubscriptionName: "regex-sub",
+ // fill `TopicsPattern` field will create a regex consumer
+ TopicsPattern: topicsPattern,
+ SubscriptionName: "regex-sub",
}
+
consumer, err := client.Subscribe(opts)
if err != nil {
- log.Fatal(err)
+ log.Fatal(err)
}
defer consumer.Close()
```
-#### How to use multi topics Consumer
+#### Create multi-topic consumer
```go
-func newTopicName() string {
- return fmt.Sprintf("my-topic-%v", time.Now().Nanosecond())
-}
-
-
-topic1 := "topic-1"
-topic2 := "topic-2"
-
-client, err := NewClient(pulsar.ClientOptions{
- URL: "pulsar://localhost:6650",
+client, err := pulsar.NewClient(pulsar.ClientOptions{
+ URL: "pulsar://localhost:6650",
})
if err != nil {
- log.Fatal(err)
+ log.Fatal(err)
}
-topics := []string{topic1, topic2}
+
+topics := []string{"topic-1", "topic-2"}
consumer, err := client.Subscribe(pulsar.ConsumerOptions{
- Topics: topics,
- SubscriptionName: "multi-topic-sub",
+ // fill `Topics` field will create a multi-topic consumer
+ Topics: topics,
+ SubscriptionName: "multi-topic-sub",
})
if err != nil {
- log.Fatal(err)
+ log.Fatal(err)
}
defer consumer.Close()
```
-#### How to use consumer listener
+#### Create consumer listener
```go
import (
- "fmt"
- "log"
+ "fmt"
+ "log"
- "github.com/apache/pulsar-client-go/pulsar"
+ "github.com/apache/pulsar-client-go/pulsar"
)
func main() {
- client, err := pulsar.NewClient(pulsar.ClientOptions{URL:
"pulsar://localhost:6650"})
- if err != nil {
- log.Fatal(err)
- }
-
- defer client.Close()
+ client, err := pulsar.NewClient(pulsar.ClientOptions{URL:
"pulsar://localhost:6650"})
+ if err != nil {
+ log.Fatal(err)
+ }
- channel := make(chan pulsar.ConsumerMessage, 100)
+ defer client.Close()
- options := pulsar.ConsumerOptions{
- Topic: "topic-1",
- SubscriptionName: "my-subscription",
- Type: pulsar.Shared,
- }
+ // we can listen this channel
+ channel := make(chan pulsar.ConsumerMessage, 100)
- options.MessageChannel = channel
+ options := pulsar.ConsumerOptions{
+ Topic: "topic-1",
+ SubscriptionName: "my-subscription",
+ Type: pulsar.Shared,
+ // fill `MessageChannel` field will create a listener
+ MessageChannel: channel,
+ }
- consumer, err := client.Subscribe(options)
- if err != nil {
- log.Fatal(err)
- }
+ consumer, err := client.Subscribe(options)
+ if err != nil {
+ log.Fatal(err)
+ }
- defer consumer.Close()
+ defer consumer.Close()
- // Receive messages from channel. The channel returns a struct which
contains message and the consumer from where
- // the message was received. It's not necessary here since we have 1
single consumer, but the channel could be
- // shared across multiple consumers as well
- for cm := range channel {
- msg := cm.Message
- fmt.Printf("Received message msgId: %v -- content: '%s'\n",
- msg.ID(), string(msg.Payload()))
+ // Receive messages from channel. The channel returns a struct
`ConsumerMessage` which contains message and the consumer from where
+ // the message was received. It's not necessary here since we have 1
single consumer, but the channel could be
+ // shared across multiple consumers as well
+ for cm := range channel {
+ consumer := cm.Consumer
+ msg := cm.Message
+ fmt.Printf("Consumer %s received a message, msgId: %v, content:
'%s'\n",
+ consumer.Name(), msg.ID(), string(msg.Payload()))
- consumer.Ack(msg)
- }
+ consumer.Ack(msg)
+ }
}
```
-#### How to use consumer receive timeout
+#### Receive message with timeout
```go
-client, err := NewClient(pulsar.ClientOptions{
- URL: "pulsar://localhost:6650",
+client, err := pulsar.NewClient(pulsar.ClientOptions{
+ URL: "pulsar://localhost:6650",
})
if err != nil {
- log.Fatal(err)
+ log.Fatal(err)
}
defer client.Close()
@@ -585,64 +555,66 @@ defer cancel()
// create consumer
consumer, err := client.Subscribe(pulsar.ConsumerOptions{
- Topic: topic,
- SubscriptionName: "my-sub1",
- Type: Shared,
+ Topic: topic,
+ SubscriptionName: "my-sub1",
+ Type: pulsar.Shared,
})
if err != nil {
- log.Fatal(err)
+ log.Fatal(err)
}
defer consumer.Close()
+// receive message with a timeout
msg, err := consumer.Receive(ctx)
-fmt.Println(msg.Payload())
if err != nil {
- log.Fatal(err)
+ log.Fatal(err)
}
+fmt.Println(msg.Payload())
```
-#### How to use schema in consumer
+#### Use schema in consumer
```go
type testJSON struct {
- ID int `json:"id"`
- Name string `json:"name"`
+ ID int `json:"id"`
+ Name string `json:"name"`
}
-```
-```go
var (
- exampleSchemaDef =
"{\"type\":\"record\",\"name\":\"Example\",\"namespace\":\"test\"," +
-
"\"fields\":[{\"name\":\"ID\",\"type\":\"int\"},{\"name\":\"Name\",\"type\":\"string\"}]}"
+ exampleSchemaDef =
"{\"type\":\"record\",\"name\":\"Example\",\"namespace\":\"test\"," +
+
"\"fields\":[{\"name\":\"ID\",\"type\":\"int\"},{\"name\":\"Name\",\"type\":\"string\"}]}"
)
-```
-```go
-client, err := NewClient(pulsar.ClientOptions{
- URL: "pulsar://localhost:6650",
+client, err := pulsar.NewClient(pulsar.ClientOptions{
+ URL: "pulsar://localhost:6650",
})
if err != nil {
- log.Fatal(err)
+ log.Fatal(err)
}
defer client.Close()
var s testJSON
-consumerJS := NewJSONSchema(exampleSchemaDef, nil)
-consumer, err := client.Subscribe(ConsumerOptions{
- Topic: "jsonTopic",
- SubscriptionName: "sub-1",
- Schema: consumerJS,
- SubscriptionInitialPosition: SubscriptionPositionEarliest,
+consumerJS := pulsar.NewJSONSchema(exampleSchemaDef, nil)
+consumer, err := client.Subscribe(pulsar.ConsumerOptions{
+ Topic: "jsonTopic",
+ SubscriptionName: "sub-1",
+ Schema: consumerJS,
+ SubscriptionInitialPosition: pulsar.SubscriptionPositionEarliest,
})
-assert.Nil(t, err)
+if err != nil {
+ log.Fatal(err)
+}
+
msg, err := consumer.Receive(context.Background())
-assert.Nil(t, err)
-err = msg.GetSchemaValue(&s)
if err != nil {
- log.Fatal(err)
+ log.Fatal(err)
}
+err = msg.GetSchemaValue(&s)
+if err != nil {
+ log.Fatal(err)
+}
defer consumer.Close()
```
@@ -716,35 +688,14 @@ scrape_configs:
scrape_interval: 10s
static_configs:
- targets:
- - localhost:2112
+ - localhost: 2112
```
Now you can query Pulsar client metrics on Prometheus.
### Consumer configuration
- Name | Description | Default
-| :-------- | :---------- |:---------- |
-| Topic | Topic specify the topic this consumer will subscribe to. This
argument is required when constructing the reader. | |
-| Topics | Specify a list of topics this consumer will subscribe on. Either a
topic, a list of topics or a topics pattern are required when subscribing| |
-| TopicsPattern | Specify a regular expression to subscribe to multiple topics
under the same namespace. Either a topic, a list of topics or a topics pattern
are required when subscribing | |
-| AutoDiscoveryPeriod | Specify the interval in which to poll for new
partitions or new topics if using a TopicsPattern. | |
-| SubscriptionName | Specify the subscription name for this consumer. This
argument is required when subscribing | |
-| Name | Set the consumer name | |
-| Properties | Properties attach a set of application defined properties to
the producer This properties will be visible in the topic stats | |
-| Type | Select the subscription type to be used when subscribing to the
topic. | Exclusive |
-| SubscriptionInitialPosition | InitialPosition at which the cursor will be
set when subscribe | Latest |
-| DLQ | Configuration for Dead Letter Queue consumer policy. | no DLQ |
-| MessageChannel | Sets a `MessageChannel` for the consumer. When a message is
received, it will be pushed to the channel for consumption | |
-| ReceiverQueueSize | Sets the size of the consumer receive queue. | 1000|
-| NackRedeliveryDelay | The delay after which to redeliver the messages that
failed to be processed | 1min |
-| ReadCompacted | If enabled, the consumer will read messages from the
compacted topic rather than reading the full message backlog of the topic |
false |
-| ReplicateSubscriptionState | Mark the subscription as replicated to keep it
in sync across clusters | false |
-| KeySharedPolicy | Configuration for Key Shared consumer policy. | |
-| RetryEnable | Auto retry send messages to default filled DLQPolicy topics |
false |
-| Interceptors | A chain of interceptors. These interceptors are called at
some points defined in the `ConsumerInterceptor` interface. | |
-| MaxReconnectToBroker | MaxReconnectToBroker set the maximum retry number of
reconnectToBroker. | ultimate |
-| Schema | Schema set a custom schema type by passing an implementation of
`Schema` | bytes[] |
+All available options of `ConsumerOptions` are
[here](https://pkg.go.dev/github.com/apache/pulsar-client-go/pulsar#ConsumerOptions).
## Readers
@@ -752,27 +703,18 @@ Pulsar readers process messages from Pulsar topics.
Readers are different from c
```go
reader, err := client.CreateReader(pulsar.ReaderOptions{
- Topic: "topic-1",
- StartMessageID: pulsar.EarliestMessageID(),
+ Topic: "topic-1",
+ StartMessageID: pulsar.EarliestMessageID(),
})
if err != nil {
- log.Fatal(err)
+ log.Fatal(err)
}
defer reader.Close()
```
### Reader operations
-Pulsar Go readers have the following methods available:
-
-Method | Description | Return type
-:------|:------------|:-----------
-`Topic()` | Returns the reader's [topic](reference-terminology.md#topic) |
`string`
-`Next(context.Context)` | Receives the next message on the topic (analogous to
the `Receive` method for [consumers](#consumer-operations)). This method blocks
until a message is available. | `(Message, error)`
-`HasNext()` | Check if there is any message available to read from the current
position| (bool, error)
-`Close()` | Closes the reader, disabling its ability to receive messages from
the broker | `error`
-`Seek(MessageID)` | Reset the subscription associated with this reader to a
specific message ID | `error`
-`SeekByTime(time time.Time)` | Reset the subscription associated with this
reader to a specific message publish time | `error`
+All available methods of the `Reader` interface are
[here](https://pkg.go.dev/github.com/apache/pulsar-client-go/pulsar#Reader).
### Reader example
@@ -782,39 +724,39 @@ Here's an example usage of a Go reader that uses the
`Next()` method to process
```go
import (
- "context"
- "fmt"
- "log"
+ "context"
+ "fmt"
+ "log"
- "github.com/apache/pulsar-client-go/pulsar"
+ "github.com/apache/pulsar-client-go/pulsar"
)
func main() {
- client, err := pulsar.NewClient(pulsar.ClientOptions{URL:
"pulsar://localhost:6650"})
- if err != nil {
- log.Fatal(err)
- }
-
- defer client.Close()
-
- reader, err := client.CreateReader(pulsar.ReaderOptions{
- Topic: "topic-1",
- StartMessageID: pulsar.EarliestMessageID(),
- })
- if err != nil {
- log.Fatal(err)
- }
- defer reader.Close()
-
- for reader.HasNext() {
- msg, err := reader.Next(context.Background())
- if err != nil {
- log.Fatal(err)
- }
-
- fmt.Printf("Received message msgId: %#v -- content: '%s'\n",
- msg.ID(), string(msg.Payload()))
- }
+ client, err := pulsar.NewClient(pulsar.ClientOptions{URL:
"pulsar://localhost:6650"})
+ if err != nil {
+ log.Fatal(err)
+ }
+
+ defer client.Close()
+
+ reader, err := client.CreateReader(pulsar.ReaderOptions{
+ Topic: "topic-1",
+ StartMessageID: pulsar.EarliestMessageID(),
+ })
+ if err != nil {
+ log.Fatal(err)
+ }
+ defer reader.Close()
+
+ for reader.HasNext() {
+ msg, err := reader.Next(context.Background())
+ if err != nil {
+ log.Fatal(err)
+ }
+
+ fmt.Printf("Received message msgId: %#v -- content: '%s'\n",
+ msg.ID(), string(msg.Payload()))
+ }
}
```
@@ -829,15 +771,15 @@ reader, err := client.CreateReader(pulsar.ReaderOptions{
})
```
-#### How to use reader to read specific message
+#### Use reader to read specific message
```go
-client, err := NewClient(pulsar.ClientOptions{
- URL: lookupURL,
+client, err := pulsar.NewClient(pulsar.ClientOptions{
+ URL: "pulsar://localhost:6650",
})
if err != nil {
- log.Fatal(err)
+ log.Fatal(err)
}
defer client.Close()
@@ -846,69 +788,59 @@ ctx := context.Background()
// create producer
producer, err := client.CreateProducer(pulsar.ProducerOptions{
- Topic: topic,
- DisableBatching: true,
+ Topic: topic,
+ DisableBatching: true,
})
if err != nil {
- log.Fatal(err)
+ log.Fatal(err)
}
defer producer.Close()
// send 10 messages
-msgIDs := [10]MessageID{}
+msgIDs := [10]pulsar.MessageID{}
for i := 0; i < 10; i++ {
- msgID, err := producer.Send(ctx, &pulsar.ProducerMessage{
- Payload: []byte(fmt.Sprintf("hello-%d", i)),
- })
- assert.NoError(t, err)
- assert.NotNil(t, msgID)
- msgIDs[i] = msgID
+ msgID, _ := producer.Send(ctx, &pulsar.ProducerMessage{
+ Payload: []byte(fmt.Sprintf("hello-%d", i)),
+ })
+ msgIDs[i] = msgID
}
// create reader on 5th message (not included)
reader, err := client.CreateReader(pulsar.ReaderOptions{
- Topic: topic,
- StartMessageID: msgIDs[4],
+ Topic: topic,
+ StartMessageID: msgIDs[4],
+ StartMessageIDInclusive: false,
})
if err != nil {
- log.Fatal(err)
+ log.Fatal(err)
}
defer reader.Close()
// receive the remaining 5 messages
for i := 5; i < 10; i++ {
- msg, err := reader.Next(context.Background())
- if err != nil {
- log.Fatal(err)
+ msg, err := reader.Next(context.Background())
+ if err != nil {
+ log.Fatal(err)
+ }
+ fmt.Printf("Read %d-th msg: %s\n", i, string(msg.Payload()))
}
-
// create reader on 5th message (included)
readerInclusive, err := client.CreateReader(pulsar.ReaderOptions{
- Topic: topic,
- StartMessageID: msgIDs[4],
- StartMessageIDInclusive: true,
+ Topic: topic,
+ StartMessageID: msgIDs[4],
+ StartMessageIDInclusive: true,
})
if err != nil {
- log.Fatal(err)
+ log.Fatal(err)
}
defer readerInclusive.Close()
```
### Reader configuration
- Name | Description | Default
-| :-------- | :---------- |:---------- |
-| Topic | Topic specify the topic this consumer will subscribe to. This
argument is required when constructing the reader. | |
-| Name | Name set the reader name. | |
-| Properties | Attach a set of application defined properties to the reader.
This properties will be visible in the topic stats | |
-| StartMessageID | StartMessageID initial reader positioning is done by
specifying a message id. | |
-| StartMessageIDInclusive | If true, the reader will start at the
`StartMessageID`, included. Default is `false` and the reader will start from
the "next" message | false |
-| MessageChannel | MessageChannel sets a `MessageChannel` for the consumer
When a message is received, it will be pushed to the channel for consumption| |
-| ReceiverQueueSize | ReceiverQueueSize sets the size of the consumer receive
queue. | 1000 |
-| SubscriptionRolePrefix| SubscriptionRolePrefix set the subscription role
prefix. | "reader" |
-| ReadCompacted | If enabled, the reader will read messages from the compacted
topic rather than reading the full message backlog of the topic. ReadCompacted
can only be enabled when reading from a persistent topic. | false|
+All available options of `ReaderOptions` are
[here](https://pkg.go.dev/github.com/apache/pulsar-client-go/pulsar#ReaderOptions).
## Messages
@@ -930,20 +862,7 @@ if _, err := producer.send(msg); err != nil {
}
```
-The following methods parameters are available for `ProducerMessage` objects:
-
-Parameter | Description
-:---------|:-----------
-`Payload` | The actual data payload of the message
-`Value` | Value and payload is mutually exclusive, `Value interface{}` for
schema message.
-`Key` | The optional key associated with the message (particularly useful for
things like topic compaction)
-`OrderingKey` | OrderingKey sets the ordering key of the message.
-`Properties` | A key-value map (both keys and values must be strings) for any
application-specific metadata attached to the message
-`EventTime` | The timestamp associated with the message
-`ReplicationClusters` | The clusters to which this message will be replicated.
Pulsar brokers handle message replication automatically; you should only change
this setting if you want to override the broker default.
-`SequenceID` | Set the sequence id to assign to the current message
-`DeliverAfter` | Request to deliver the message only after the specified
relative delay
-`DeliverAt` | Deliver the message only at or after the specified absolute
timestamp
+All methods of `ProducerMessage` object are
[here](https://pkg.go.dev/github.com/apache/pulsar-client-go/pulsar#ProducerMessage).
## TLS encryption and authentication
@@ -959,7 +878,7 @@ Here's an example:
opts := pulsar.ClientOptions{
URL: "pulsar+ssl://my-cluster.com:6651",
TLSTrustCertsFilePath: "/path/to/certs/my-cert.csr",
- Authentication: NewAuthenticationTLS("my-cert.pem", "my-key.pem"),
+ Authentication: pulsar.NewAuthenticationTLS("my-cert.pem", "my-key.pem"),
}
```
@@ -971,15 +890,15 @@ This example shows how to configure OAuth2 authentication.
```go
oauth := pulsar.NewAuthenticationOAuth2(map[string]string{
- "type": "client_credentials",
- "issuerUrl": "https://dev-kt-aa9ne.us.auth0.com";,
- "audience": "https://dev-kt-aa9ne.us.auth0.com/api/v2/";,
- "privateKey": "/path/to/privateKey",
- "clientId": "0Xx...Yyxeny",
- })
+ "type": "client_credentials",
+ "issuerUrl": "https://dev-kt-aa9ne.us.auth0.com";,
+ "audience": "https://dev-kt-aa9ne.us.auth0.com/api/v2/";,
+ "privateKey": "/path/to/privateKey",
+ "clientId": "0Xx...Yyxeny",
+})
client, err := pulsar.NewClient(pulsar.ClientOptions{
- URL: "pulsar://my-cluster:6650",
- Authentication: oauth,
+ URL: "pulsar://my-cluster:6650",
+ Authentication: oauth,
})
```
diff --git a/site2/website-next/docs/client-libraries-java.md
b/site2/website-next/docs/client-libraries-java.md
index 30f88956079..82568086b8d 100644
--- a/site2/website-next/docs/client-libraries-java.md
+++ b/site2/website-next/docs/client-libraries-java.md
@@ -1431,7 +1431,7 @@ With TableView, Pulsar clients can fetch all the message
updates from a topic an
:::note
-Each TableView uses one Reader instance per partition, and reads the topic
starting from the compacted view by default. It is highly recommended to enable
automatic compaction by [configuring the topic compaction
policies](cookbooks-compaction.md#configuring-compaction-to-run-automatically)
for the given topic or namespace. More frequent compaction results in shorter
startup times because less data is replayed to reconstruct the TableView of the
topic.
+Each TableView uses one Reader instance per partition, and reads the topic
starting from the compacted view by default. It is highly recommended to enable
automatic compaction by [configuring the topic compaction
policies](cookbooks-compaction.md#configure-compaction-to-run-automatically)
for the given topic or namespace. More frequent compaction results in shorter
startup times because less data is replayed to reconstruct the TableView of the
topic.
:::
diff --git a/site2/website-next/docs/cookbooks-compaction.md
b/site2/website-next/docs/cookbooks-compaction.md
index a20ebd188ad..95403af461d 100644
--- a/site2/website-next/docs/cookbooks-compaction.md
+++ b/site2/website-next/docs/cookbooks-compaction.md
@@ -9,8 +9,8 @@ Pulsar's [topic
compaction](concepts-topic-compaction.md#compaction) feature ena
To use compaction:
* You need to give messages keys, as topic compaction in Pulsar takes place on
a *per-key basis* (i.e. messages are compacted based on their key). For a stock
ticker use case, the stock symbol---e.g. `AAPL` or `GOOG`---could serve as the
key (more on this [below](#when-should-i-use-compacted-topics)). Messages
without keys will be left alone by the compaction process.
-* Compaction can be configured to run
[automatically](#configuring-compaction-to-run-automatically), or you can
manually [trigger](#triggering-compaction-manually) compaction using the Pulsar
administrative API.
-* Your consumers must be [configured](#consumer-configuration) to read from
compacted topics ([Java consumers](#java), for example, have a `readCompacted`
setting that must be set to `true`). If this configuration is not set,
consumers will still be able to read from the non-compacted topic.
+* Compaction can be configured to run
[automatically](#configure-compaction-to-run-automatically), or you can
manually [trigger](#trigger-compaction-manually) compaction using the Pulsar
administrative API.
+* Your consumers must be [configured](#configure-consumers) to read from
compacted topics (Java consumers, for example, have a `readCompacted` setting
that must be set to `true`). If this configuration is not set, consumers will
still be able to read from the non-compacted topic.
> Compaction only works on messages that have keys (as in the stock ticker
> example the stock symbol serves as the key for each message). Keys can thus
> be thought of as the axis along which compaction is applied. Messages that
> don't have keys are simply ignored by compaction.
@@ -22,25 +22,25 @@ The classic example of a topic that could benefit from
compaction would be a sto
* They can read from the "original," non-compacted topic in case they need
access to "historical" values, i.e. the entirety of the topic's messages.
* They can read from the compacted topic if they only want to see the most
up-to-date messages.
-Thus, if you're using a Pulsar topic called `stock-values`, some consumers
could have access to all messages in the topic (perhaps because they're
performing some kind of number crunching of all values in the last hour) while
the consumers used to power the real-time stock ticker only see the compacted
topic (and thus aren't forced to process outdated messages). Which variant of
the topic any given consumer pulls messages from is determined by the
consumer's [configuration](#consumer-con [...]
+Thus, if you're using a Pulsar topic called `stock-values`, some consumers
could have access to all messages in the topic (perhaps because they're
performing some kind of number crunching of all values in the last hour) while
the consumers used to power the real-time stock ticker only see the compacted
topic (and thus aren't forced to process outdated messages). Which variant of
the topic any given consumer pulls messages from is determined by the
consumer's [configuration](#configure-co [...]
> One of the benefits of compaction in Pulsar is that you aren't forced to
> choose between compacted and non-compacted topics, as the compaction process
> leaves the original topic as-is and essentially adds an alternate topic. In
> other words, you can run compaction on a topic and consumers that need
> access to the non-compacted version of the topic will not be adversely
> affected.
-## Configuring compaction to run automatically
+## Configure compaction to run automatically
-Tenant administrators can configure a policy for compaction at the namespace
level. The policy specifies how large the topic backlog can grow before
compaction is triggered.
+Compaction policy specifies how large the topic backlog can grow before
compaction is triggered.
-For example, to trigger compaction when the backlog reaches 100MB:
+Tenant administrators can configure a compaction policy at namespace or topic
levels. Configuring the compaction policy at the namespace level applies to all
topics within that namespace.
+
+For example, to trigger compaction in a namespace when the backlog reaches
100MB:
```bash
bin/pulsar-admin namespaces set-compaction-threshold \
--threshold 100M my-tenant/my-namespace
```
-Configuring the compaction threshold on a namespace will apply to all topics
within that namespace.
-
-## Triggering compaction manually
+## Trigger compaction manually
To run compaction on a topic, you need to use the [`topics
compact`](/tools/pulsar-admin/) command for the
[`pulsar-admin`](/tools/pulsar-admin/) CLI tool. Here's an example:
@@ -70,15 +70,15 @@ bin/pulsar compact-topic \
--topic persistent://my-tenant/my-namespace/my-topic
```
-#### When should I trigger compaction?
+:::tip
-How often you [trigger compaction](#triggering-compaction-manually) will vary
widely based on the use case. If you want a compacted topic to be extremely
speedy on read, then you should run compaction fairly frequently.
+The frequency to trigger topic compaction varies widely based on use cases. If
you want a compacted topic to be extremely speedy on read, then you need to run
compaction fairly frequently.
-## Consumer configuration
+:::
-Pulsar consumers and readers need to be configured to read from compacted
topics. The sections below show you how to enable compacted topic reads for
Pulsar's language clients.
+## Configure consumers
-### Java
+Pulsar consumers and readers need to be configured to read from compacted
topics. The section below introduces how to enable compacted topic reads for
Java clients.
To read from a compacted topic using a Java consumer, the `readCompacted`
parameter must be set to `true`. Here's an example consumer for a compacted
topic:
diff --git a/site2/website-next/docs/functions-overview.md
b/site2/website-next/docs/functions-overview.md
index cfc39081fb3..e017695ddb9 100644
--- a/site2/website-next/docs/functions-overview.md
+++ b/site2/website-next/docs/functions-overview.md
@@ -8,7 +8,7 @@ This section introduces the following content:
* [What are Pulsar Functions](#what-are-pulsar-functions)
* [Why use Pulsar Functions](#why-use-pulsar-functions)
* [Use cases](#use-cases)
-* [User flow](#user-flow)
+* [What's next?](#whats-next)
## What are Pulsar Functions
diff --git a/site2/website-next/docs/getting-started-docker.md
b/site2/website-next/docs/getting-started-docker.md
index 8dd7779ca55..1c4f01600a0 100644
--- a/site2/website-next/docs/getting-started-docker.md
+++ b/site2/website-next/docs/getting-started-docker.md
@@ -1,10 +1,10 @@
---
id: getting-started-docker
-title: Set up a standalone Pulsar in Docker
+title: Run a standalone Pulsar cluster in Docker
sidebar_label: "Run Pulsar in Docker"
---
-For local development and testing, you can run Pulsar in standalone mode on
your own machine within a Docker container.
+For local development and testing, you can run Pulsar in standalone mode on
your own machine within a Docker container.
If you have not installed Docker, download the [Community
edition](https://www.docker.com/community-edition) and follow the instructions
for your OS.
@@ -48,7 +48,7 @@ After starting Pulsar successfully, you can see `INFO`-level
log messages like t
## Use Pulsar in Docker
-Pulsar offers a variety of [client libraries](client-libraries.md), such as
[Java](client-libraries-java.md), [Go](client-libraries-go.md),
[Python](client-libraries-python.md), [C++](client-libraries-cpp.md).
+Pulsar offers a variety of [client libraries](client-libraries.md), such as
[Java](client-libraries-java.md), [Go](client-libraries-go.md),
[Python](client-libraries-python.md), [C++](client-libraries-cpp.md).
If you're running a local standalone cluster, you can use one of these root
URLs to interact with your cluster:
* `pulsar://localhost:6650`
diff --git a/site2/website-next/docs/getting-started-helm.md
b/site2/website-next/docs/getting-started-helm.md
index 95e3c8079cb..1a373d1893e 100644
--- a/site2/website-next/docs/getting-started-helm.md
+++ b/site2/website-next/docs/getting-started-helm.md
@@ -1,6 +1,6 @@
---
id: getting-started-helm
-title: Get started in Kubernetes
+title: Run a standalone Pulsar cluster in Kubernetes
sidebar_label: "Run Pulsar in Kubernetes"
---
@@ -52,7 +52,7 @@ We use [Minikube](https://minikube.sigs.k8s.io/docs/start/)
in this quick start
minikube dashboard
```
- The command automatically triggers opening a webpage in your browser.
+ The command automatically triggers opening a webpage in your browser.
## Step 1: Install Pulsar Helm chart
@@ -88,7 +88,7 @@ We use [Minikube](https://minikube.sigs.k8s.io/docs/start/)
in this quick start
-c
```
-4. Use the Pulsar Helm chart to install a Pulsar cluster to Kubernetes.
+4. Use the Pulsar Helm chart to install a Pulsar cluster to Kubernetes.
```bash
helm install \
@@ -169,7 +169,7 @@ We use [Minikube](https://minikube.sigs.k8s.io/docs/start/)
in this quick start
bin/pulsar-admin tenants list
```
- You should see a similar output as below. The tenant `apache` has been
successfully created.
+ You should see a similar output as below. The tenant `apache` has been
successfully created.
```bash
"apache"
@@ -189,7 +189,7 @@ We use [Minikube](https://minikube.sigs.k8s.io/docs/start/)
in this quick start
bin/pulsar-admin namespaces list apache
```
- You should see a similar output as below. The namespace `apache/pulsar` has
been successfully created.
+ You should see a similar output as below. The namespace `apache/pulsar` has
been successfully created.
```bash
"apache/pulsar"
@@ -303,7 +303,7 @@ Then you can proceed with the following steps:
- From the producer side
**Output**
-
+
The messages have been produced successfully.
```bash
@@ -351,7 +351,7 @@ Then you can proceed with the following steps:
2. The Pulsar Manager UI will be open in your browser. You can use the
username `pulsar` and password `pulsar` to log into Pulsar Manager.
-3. In Pulsar Manager UI, you can create an environment.
+3. In Pulsar Manager UI, you can create an environment.
- Click **New Environment** in the upper-left corner.
- Type `pulsar-mini` for the field `Environment Name` in the pop-up window.
diff --git a/site2/website-next/docs/getting-started-home.md
b/site2/website-next/docs/getting-started-home.md
index 64ea38a8120..fd6f0d329e5 100644
--- a/site2/website-next/docs/getting-started-home.md
+++ b/site2/website-next/docs/getting-started-home.md
@@ -4,9 +4,9 @@ title: Get started
sidebar_label: "Get Started"
---
-Getting up and running with Pulsar is simple. Download it, install it, and try
it out.
+Getting up and running with Pulsar is simple. Download it, install it, and try
it out.
You have three options. Click any of these links to begin your Pulsar journey!
-* [Run a standalone Pulsar locally](getting-started-standalone.md) - Run a
single instance of Pulsar in standalone mode on a single machine.
-* [Run a standalone Pulsar in Docker](getting-started-docker.md) - Run one or
more instances of Pulsar in a Docker container.
-* [Run a standalone Pulsar in Kubernetes](getting-started-helm.md) - Run one
or more instances of Pulsar in Kubernetes using a Helm chart.
\ No newline at end of file
+* [Run a standalone Pulsar cluster locally](getting-started-standalone.md) -
Run a single instance of Pulsar in standalone mode on a single machine.
+* [Run a standalone Pulsar cluster in Docker](getting-started-docker.md) - Run
one or more instances of Pulsar in a Docker container.
+* [Run a standalone Pulsar cluster in Kubernetes](getting-started-helm.md) -
Run one or more instances of Pulsar in Kubernetes using a Helm chart.
diff --git a/site2/website-next/docs/getting-started-standalone.md
b/site2/website-next/docs/getting-started-standalone.md
index f89bad1da59..5cba7f981b9 100644
--- a/site2/website-next/docs/getting-started-standalone.md
+++ b/site2/website-next/docs/getting-started-standalone.md
@@ -1,278 +1,137 @@
---
id: getting-started-standalone
-title: Set up a standalone Pulsar locally
+title: Run a standalone Pulsar cluster locally
sidebar_label: "Run Pulsar locally"
---
-For local development and testing, you can run Pulsar in standalone mode on
your machine. The standalone mode includes a Pulsar broker, the necessary
[RocksDB](http://rocksdb.org/) and BookKeeper components running inside of a
single Java Virtual Machine (JVM) process.
-
-> **Pulsar in production?**
-> If you're looking to run a full production Pulsar installation, see the
[Deploying a Pulsar instance](deploy-bare-metal.md) guide.
-
-## Install Pulsar standalone
-
-This tutorial guides you through every step of installing Pulsar locally.
-
-### System requirements
-
-Currently, Pulsar is available for 64-bit **macOS**, **Linux**, and
**Windows**. To use Pulsar, you need to install 64-bit JRE/JDK.
-For the runtime Java version, see [Pulsar Runtime Java Version
Recommendation](https://github.com/apache/pulsar/blob/master/README.md#pulsar-runtime-java-version-recommendation)
according to your target Pulsar version.
+For local development and testing, you can run Pulsar in standalone mode on
your machine. The standalone mode runs all components inside a single Java
Virtual Machine (JVM) process.
:::tip
-By default, Pulsar allocates 2G JVM heap memory to start. It can be changed in
`conf/pulsar_env.sh` file under `PULSAR_MEM`. This is an extra option passed
into JVM.
+If you're looking to run a full production Pulsar installation, see the
[Deploying a Pulsar instance](deploy-bare-metal.md) guide.
:::
-:::note
+## Prerequisites
-Broker is only supported on 64-bit JVM.
+- JRE (64-bit). Different Pulsar versions rely on different JRE versions. For
how to choose the JRE version, see [Pulsar Runtime Java Version
Recommendation](https://github.com/apache/pulsar/blob/master/README.md#pulsar-runtime-java-version-recommendation).
-:::
-
-#### Install JDK on M1
-In the current version, Pulsar uses a BookKeeper version which in turn uses
RocksDB. RocksDB is compiled to work on x86 architecture and not ARM.
Therefore, Pulsar can only work with x86 JDK. This is planned to be fixed in
future versions of Pulsar.
-
-One of the ways to easily install an x86 JDK is to use
[SDKMan](http://sdkman.io). Follow instructions on the SDKMan website.
-
-2. Turn on Rosetta2 compatibility for SDKMan by editing `~/.sdkman/etc/config`
and changing the following property from `false` to `true`.
-
-```properties
-sdkman_rosetta2_compatible=true
-```
+## Step 1. Download Pulsar distribution
-3. Close the current shell / terminal window and open a new one.
-4. Make sure you don't have any previously installed JVM of the same version
by listing existing installed versions.
+Download the official Apache Pulsar distribution:
-```shell
-sdk list java|grep installed
-```
-
-Example output:
-
-```text
- | >>> | 17.0.3.6.1 | amzn | installed | 17.0.3.6.1-amzn
-```
-
-If you have any Java 17 version installed, uninstall it.
-
-```shell
-sdk uinstall java 17.0.3.6.1
+```bash
+wget
https://archive.apache.org/dist/pulsar/pulsar-@pulsar:version@/apache-pulsar-@pulsar:[email protected]
```
-5. Install any Java versions greater than Java 8.
+Once downloaded, unpack the tar file:
-```shell
- sdk install java 17.0.3.6.1-amzn
+```bash
+tar xvfz apache-pulsar-@pulsar:[email protected]
```
-### Install Pulsar using binary release
-
-To get started with Pulsar, download a binary tarball release in one of the
following ways:
-
-* download from the Apache mirror (<a href="pulsar:binary_release_url"
download>Pulsar @pulsar:version@ binary release</a>)
-
-* download from the Pulsar [downloads page](pulsar:download_page_url)
-
-* download from the Pulsar [releases
page](https://github.com/apache/pulsar/releases/latest)
-
-* use [wget](https://www.gnu.org/software/wget):
-
- ```shell
- wget pulsar:binary_release_url
- ```
-
-After you download the tarball, untar it and use the `cd` command to navigate
to the resulting directory:
+For the rest of this quickstart all commands are run from the root of the
distribution folder, so switch to it:
```bash
-tar xvfz apache-pulsar-@pulsar:[email protected]
cd apache-pulsar-@pulsar:version@
```
-#### What your package contains
-
-The Pulsar binary package initially contains the following directories:
-
-Directory | Contains
-:---------|:--------
-`bin` | Pulsar's command-line tools, such as
[`pulsar`](reference-cli-tools.md#pulsar) and
[`pulsar-admin`](/tools/pulsar-admin/).
-`conf` | Configuration files for Pulsar, including [broker
configuration](reference-configuration.md#broker) and more.<br />**Note:**
Pulsar standalone uses RocksDB as the local metadata store and its
configuration file path [`metadataStoreConfigPath`](reference-configuration.md)
is configurable in the `standalone.conf` file. For more information about the
configurations of RocksDB, see
[here](https://github.com/facebook/rocksdb/blob/main/examples/rocksdb_option_file_example.ini)
and rel [...]
-`examples` | A Java JAR file containing [Pulsar
Functions](functions-overview.md) example.
-`instances` | Artifacts created for [Pulsar Functions](functions-overview.md).
-`lib` | The [JAR](https://en.wikipedia.org/wiki/JAR_(file_format)) files used
by Pulsar.
-`licenses` | License files, in the`.txt` form, for various components of the
Pulsar [codebase](https://github.com/apache/pulsar).
-
-These directories are created once you begin running Pulsar.
-
-Directory | Contains
-:---------|:--------
-`data` | The data storage directory used by RocksDB and BookKeeper.
-`logs` | Logs created by the installation.
+List the contents by executing:
-:::tip
-
-If you want to use built-in connectors and tiered storage offloaders, you can
install them according to the following instructions：
-* [Install built-in connectors
(optional)](#install-built-in-connectors-optional)
-* [Install tiered storage offloaders
(optional)](#install-tiered-storage-offloaders-optional)
-Otherwise, skip this step and perform the next step [Start Pulsar
standalone](#start-pulsar-standalone). Pulsar can be successfully installed
without installing built-in connectors and tiered storage offloaders.
-
-:::
-
-### Install built-in connectors (optional)
-
-Since `2.1.0-incubating` release, Pulsar releases a separate binary
distribution, containing all the `built-in` connectors.
-To enable those `built-in` connectors, you can download the connectors tarball
release in one of the following ways:
-
-* download from the Apache mirror <a href="pulsar:connector_release_url"
download>Pulsar IO Connectors @pulsar:version@ release</a>
-
-* download from the Pulsar [downloads page](pulsar:download_page_url)
+```bash
+ls -1F
+```
-* download from the Pulsar [releases
page](https://github.com/apache/pulsar/releases/latest)
+You may want to note that:
-* use [wget](https://www.gnu.org/software/wget):
+| Directory | Description
|
+| ------------- |
---------------------------------------------------------------------------------------------------
|
+| **bin** | The [`pulsar`](reference-cli-tools.md#pulsar) entry point
script, and many other command-line tools |
+| **conf** | Configuration files, including `broker.conf`
|
+| **lib** | JARs used by Pulsar
|
+| **examples** | [Pulsar Functions](functions-overview.md) examples
|
+| **instances** | Artifacts for [Pulsar Functions](functions-overview.md)
|
- ```shell
- wget pulsar:connector_release_url/{connector}-@pulsar:[email protected]
- ```
+## Step 2. Start a Pulsar standalone cluster
-After you download the NAR file, copy the file to the `connectors` directory
in the pulsar directory.
-For example, if you download the `pulsar-io-aerospike-@pulsar:[email protected]`
connector file, enter the following commands:
+Run this command to start a standalone Pulsar cluster:
```bash
-mkdir connectors
-mv pulsar-io-aerospike-@pulsar:[email protected] connectors
-
-ls connectors
-pulsar-io-aerospike-@pulsar:[email protected]
-...
+bin/pulsar standalone
```
-:::note
-
-* If you are running Pulsar in a bare metal cluster, make sure `connectors`
tarball is unzipped in every pulsar directory of the broker (or in every pulsar
directory of function-worker if you are running a separate worker cluster for
Pulsar Functions).
-* If you are [running Pulsar in Docker](getting-started-docker.md) or
deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes.md) or
[DC/OS](https://dcos.io/), you can use the `apachepulsar/pulsar-all` image
instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has
already bundled [all built-in
connectors](io-overview.md#working-with-connectors).
+These directories are created once you started the Pulsar cluster:
-:::
-
-### Install tiered storage offloaders (optional)
+| Directory | Description |
+| --------- | ------------------------------------------ |
+| **data** | All data created by BookKeeper and RocksDB |
+| **logs** | All server-side logs |
:::tip
-- Since `2.2.0` release, Pulsar releases a separate binary distribution,
containing the tiered storage offloaders.
-- To enable the tiered storage feature, follow the instructions below;
otherwise skip this section.
+* To run the service as a background process, you can use the
`bin/pulsar-daemon start standalone` command. For more information, see
[pulsar-daemon](reference-cli-tools.md#pulsar-daemon).
+* The `public/default` namespace is created when you start a Pulsar cluster.
This namespace is for development purposes. All Pulsar topics are managed
within namespaces. For more information, see
[Namespaces](concepts-messaging.md#namespaces) and
[Topics](concepts-messaging.md#topics).
:::
-To get started with [tiered storage offloaders](concepts-tiered-storage.md),
you need to download the offloaders tarball release on every broker node in one
of the following ways:
-
-* download from the Apache mirror <a href="pulsar:offloader_release_url"
download>Pulsar Tiered Storage Offloaders @pulsar:version@ release</a>
+## Step 3. Create a topic
-* download from the Pulsar [downloads page](pulsar:download_page_url)
+Pulsar stores messages in topics. It's a good practice to explicitly create
topics before using them, even if Pulsar can automatically create topics when
they are referenced.
-* download from the Pulsar [releases
page](https://github.com/apache/pulsar/releases/latest)
-
-* use [wget](https://www.gnu.org/software/wget):
-
- ```shell
- wget pulsar:offloader_release_url
- ```
-
-After you download the tarball, untar the offloaders package and copy the
offloaders as `offloaders`
-in the pulsar directory:
+To create a new topic, run this command:
```bash
-tar xvfz apache-pulsar-offloaders-@pulsar:[email protected]
-
-// you will find a directory named `apache-pulsar-offloaders-@pulsar:version@`
in the pulsar directory
-// then copy the offloaders
-
-mv apache-pulsar-offloaders-@pulsar:version@/offloaders offloaders
-
-ls offloaders
-tiered-storage-jcloud-@pulsar:[email protected]
+bin/pulsar-admin topics create persistent://public/default/my-topic
```
-For more information on how to configure tiered storage, see [Tiered storage
cookbook](cookbooks-tiered-storage.md).
-
-:::note
-
-* If you are running Pulsar in a bare metal cluster, make sure that
`offloaders` tarball is unzipped in every broker's pulsar directory.
-* If you are [running Pulsar in Docker](getting-started-docker.md) or
deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes.md) or
DC/OS), you can use the `apachepulsar/pulsar-all` image instead of the
`apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already
bundled tiered storage offloaders.
-
-:::
+## Step 4. Write messages to the topic
-## Start Pulsar standalone
+You can use the `pulsar` command line tool to write messages to a topic. This
is useful for experimentation, but in practice you'll use the Producer API in
your application code, or Pulsar IO connectors for pulling data in from other
systems to Pulsar.
-Once you have an up-to-date local copy of the release, you can start a local
cluster using the [`pulsar`](reference-cli-tools.md#pulsar) command, which is
stored in the `bin` directory, and specifying that you want to start Pulsar in
standalone mode.
+Run this command to produce a message:
```bash
-bin/pulsar standalone
+bin/pulsar-client produce my-topic --messages 'Hello Pulsar!'
```
-If you have started Pulsar successfully, you will see `INFO`-level log
messages like this:
+## Step 5. Read messages from the topic
-```bash
-21:59:29.327 [DLM-/stream/storage-OrderedScheduler-3-0] INFO
org.apache.bookkeeper.stream.storage.impl.sc.StorageContainerImpl -
Successfully started storage container (0).
-21:59:34.576 [main] INFO
org.apache.pulsar.broker.authentication.AuthenticationService - Authentication
is disabled
-21:59:34.576 [main] INFO org.apache.pulsar.websocket.WebSocketService -
Pulsar WebSocket Service started
-```
-
-:::tip
-
-* The service is running on your terminal, which is under your direct control.
If you need to run other commands, open a new terminal window.
-* To run the service as a background process, you can use the
`bin/pulsar-daemon start standalone` command. For more information, see
[pulsar-daemon](/docs/en/reference-cli-tools/#pulsar-daemon).
-* To perform a health check, you can use the `bin/pulsar-admin brokers
healthcheck` command. For more information, see [Pulsar-admin
docs](/tools/pulsar-admin/).
-* When you start a local standalone cluster, a `public/default`
[namespace](concepts-messaging.md#namespaces) is created automatically. The
namespace is used for development purposes. All Pulsar topics are managed
within namespaces. For more information, see
[Topics](concepts-messaging.md#topics).
-* By default, there is no encryption, authentication, or authorization
configured. Apache Pulsar can be accessed from a remote server without any
authorization. See [Security Overview](security-overview.md) for how to secure
your deployment.
-
-:::
-
-## Use Pulsar standalone
-
-Pulsar provides a CLI tool called
[`pulsar-client`](reference-cli-tools.md#pulsar-client). The pulsar-client tool
enables you to consume and produce messages to a Pulsar topic in a running
cluster.
-
-### Consume a message
-
-The following command consumes a message with the subscription name
`first-subscription` to the `my-topic` topic:
+Now that some messages have been written to the topic, run this command to
launch the consumer and read those messages back:
```bash
-bin/pulsar-client consume my-topic -s "first-subscription"
+bin/pulsar-client consume my-topic -s 'my-subscription' -p Earliest -n 0
```
-If the message has been successfully consumed, you will see a confirmation
like the following in the `pulsar-client` logs:
+Earliest means consuming from the earliest **unconsumed** message. `-n`
configures the number of messages to consume, 0 means to consume forever.
-```
-22:17:16.781 [main] INFO org.apache.pulsar.client.cli.PulsarClientTool - 1
messages successfully consumed
-```
+As before, this is useful for trialling things on the command line, but in
practice you'll use the Consumer API in your application code, or Pulsar IO
connectors for reading data from Pulsar to push to other systems.
-:::tip
+You'll see the messages that you produce in the previous step:
-As you have noticed that we do not explicitly create the `my-topic` topic,
from which we consume the message. When you consume a message from a topic that
does not yet exist, Pulsar creates that topic for you automatically. Producing
a message to a topic that does not exist will automatically create that topic
for you as well.
+```text
+----- got message -----
+key:[null], properties:[], content:Hello Pulsar!
+```
-:::
+## Step 6. Write some more messages
-### Produce a message
+Leave the consume command from the previous step running. If you've already
closed it, just re-run it.
-The following command produces a message saying `hello-pulsar` to the
`my-topic` topic:
+Now open a new terminal window and produce more messages, the default message
separator is `,`:
```bash
-bin/pulsar-client produce my-topic --messages "hello-pulsar"
-```
-
-If the message has been successfully published to the topic, you will see a
confirmation like the following in the `pulsar-client` logs:
-
-```
-22:21:08.693 [main] INFO org.apache.pulsar.client.cli.PulsarClientTool - 1
messages successfully produced
+bin/pulsar-client produce my-topic --messages "$(seq -s, -f 'Message NO.%g' -t
'\n' 1 10)"
```
-## Stop Pulsar standalone
+Note how they are displayed almost instantaneously in the consumer terminal.
-Press `Ctrl+C` to stop a local standalone Pulsar.
+## Step 7. Stop the Pulsar cluster
-:::tip
+Once you've finished you can shut down the Pulsar cluster. Press **Ctrl-C** in
the terminal window in which you started the cluster.
-If the service runs as a background process using the `bin/pulsar-daemon start
standalone` command, then use the `bin/pulsar-daemon stop standalone` command
to stop the service.
-For more information, see
[pulsar-daemon](reference-cli-tools.md#pulsar-daemon).
-
-:::
+## Further readings
+* Read [Pulsar Concepts and Architecture](concepts-architecture-overview.md)
to learn more about Pulsar fundamentals.
+* Read [Pulsar Client Libraries](client-libraries.md) to connect Pulsar with
your application.
+* Read [Pulsar Connectors](io-overview.md) to connect Pulsar with your
existing data pipelines.
+* Read [Pulsar Functions](functions-overview.md) to run serverless
computations against Pulsar.
diff --git a/site2/website-next/docs/reference-metrics.md
b/site2/website-next/docs/reference-metrics.md
index c70b444cb44..c4c7f5c771c 100644
--- a/site2/website-next/docs/reference-metrics.md
+++ b/site2/website-next/docs/reference-metrics.md
@@ -1,7 +1,7 @@
---
id: reference-metrics
-title: Pulsar Metrics
-sidebar_label: "Pulsar Metrics"
+title: Pulsar metrics
+sidebar_label: "Pulsar metrics"
---
diff --git a/site2/website-next/docs/security-tls-authentication.md
b/site2/website-next/docs/security-tls-authentication.md
index 761203f34c4..dfa6b7cdb37 100644
--- a/site2/website-next/docs/security-tls-authentication.md
+++ b/site2/website-next/docs/security-tls-authentication.md
@@ -71,6 +71,7 @@ To configure brokers to authenticate clients, add the
following parameters to `b
# Configuration to enable authentication
authenticationEnabled=true
authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderTls
+tlsRequireTrustedClientCertOnConnect=true
brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls
brokerClientAuthenticationParameters={"tlsCertFile":"/path/my-ca/admin.cert.pem","tlsKeyFile":"/path/my-ca/admin.key-pk8.pem"}
@@ -87,6 +88,7 @@ The proxy should have its own client key pair for connecting
to brokers. You nee
# For clients connecting to the proxy
authenticationEnabled=true
authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderTls
+tlsRequireTrustedClientCertOnConnect=true
# For the proxy to connect to brokers
brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls
diff --git a/site2/website-next/sidebars.json b/site2/website-next/sidebars.json
index 7c51ae83678..5879f3b36b7 100644
--- a/site2/website-next/sidebars.json
+++ b/site2/website-next/sidebars.json
@@ -343,7 +343,12 @@
"reference-cli-tools",
"reference-configuration",
"reference-metrics",
- "reference-rest-api-overview"
+ "reference-rest-api-overview",
+ {
+ "type": "link",
+ "href": "/release-notes/",
+ "label": "Release notes"
+ }
]
}
]
diff --git
a/site2/website-next/versioned_docs/version-2.10.x/client-libraries-java.md
b/site2/website-next/versioned_docs/version-2.10.x/client-libraries-java.md
index 5dacf669e76..38d26f70ee8 100644
--- a/site2/website-next/versioned_docs/version-2.10.x/client-libraries-java.md
+++ b/site2/website-next/versioned_docs/version-2.10.x/client-libraries-java.md
@@ -1390,7 +1390,11 @@ The TableView interface serves an encapsulated access
pattern, providing a conti
With TableView, Pulsar clients can fetch all the message updates from a topic
and construct a map with the latest values of each key. These values can then
be used to build a local cache of data. In addition, you can register consumers
with the TableView by specifying a listener to perform a scan of the map and
then receive notifications when new messages are received. Consequently, event
handling can be triggered to serve use cases, such as event-driven applications
and message monitoring.
-> **Note:** Each TableView uses one Reader instance per partition, and reads
the topic starting from the compacted view by default. It is highly recommended
to enable automatic compaction by [configuring the topic compaction
policies](cookbooks-compaction.md#configuring-compaction-to-run-automatically)
for the given topic or namespace. More frequent compaction results in shorter
startup times because less data is replayed to reconstruct the TableView of the
topic.
+:::note
+
+Each TableView uses one Reader instance per partition, and reads the topic
starting from the compacted view by default. It is highly recommended to enable
automatic compaction by [configuring the topic compaction
policies](cookbooks-compaction.md#configure-compaction-to-run-automatically)
for the given topic or namespace. More frequent compaction results in shorter
startup times because less data is replayed to reconstruct the TableView of the
topic.
+
+:::
The following figure illustrates the dynamic construction of a TableView
updated with newer values of each key.
diff --git
a/site2/website-next/versioned_docs/version-2.10.x/cookbooks-compaction.md
b/site2/website-next/versioned_docs/version-2.10.x/cookbooks-compaction.md
index dfa31472724..8e4ed064438 100644
--- a/site2/website-next/versioned_docs/version-2.10.x/cookbooks-compaction.md
+++ b/site2/website-next/versioned_docs/version-2.10.x/cookbooks-compaction.md
@@ -10,8 +10,8 @@ Pulsar's [topic
compaction](concepts-topic-compaction.md#compaction) feature ena
To use compaction:
* You need to give messages keys, as topic compaction in Pulsar takes place on
a *per-key basis* (i.e. messages are compacted based on their key). For a stock
ticker use case, the stock symbol---e.g. `AAPL` or `GOOG`---could serve as the
key (more on this [below](#when-should-i-use-compacted-topics)). Messages
without keys will be left alone by the compaction process.
-* Compaction can be configured to run
[automatically](#configuring-compaction-to-run-automatically), or you can
manually [trigger](#triggering-compaction-manually) compaction using the Pulsar
administrative API.
-* Your consumers must be [configured](#consumer-configuration) to read from
compacted topics ([Java consumers](#java), for example, have a `readCompacted`
setting that must be set to `true`). If this configuration is not set,
consumers will still be able to read from the non-compacted topic.
+* Compaction can be configured to run
[automatically](#configure-compaction-to-run-automatically), or you can
manually [trigger](#trigger-compaction-manually) compaction using the Pulsar
administrative API.
+* Your consumers must be [configured](#configure-consumers) to read from
compacted topics (Java consumers, for example, have a `readCompacted` setting
that must be set to `true`). If this configuration is not set, consumers will
still be able to read from the non-compacted topic.
> Compaction only works on messages that have keys (as in the stock ticker
> example the stock symbol serves as the key for each message). Keys can thus
> be thought of as the axis along which compaction is applied. Messages that
> don't have keys are simply ignored by compaction.
@@ -23,16 +23,18 @@ The classic example of a topic that could benefit from
compaction would be a sto
* They can read from the "original," non-compacted topic in case they need
access to "historical" values, i.e. the entirety of the topic's messages.
* They can read from the compacted topic if they only want to see the most
up-to-date messages.
-Thus, if you're using a Pulsar topic called `stock-values`, some consumers
could have access to all messages in the topic (perhaps because they're
performing some kind of number crunching of all values in the last hour) while
the consumers used to power the real-time stock ticker only see the compacted
topic (and thus aren't forced to process outdated messages). Which variant of
the topic any given consumer pulls messages from is determined by the
consumer's [configuration](#consumer-con [...]
+Thus, if you're using a Pulsar topic called `stock-values`, some consumers
could have access to all messages in the topic (perhaps because they're
performing some kind of number crunching of all values in the last hour) while
the consumers used to power the real-time stock ticker only see the compacted
topic (and thus aren't forced to process outdated messages). Which variant of
the topic any given consumer pulls messages from is determined by the
consumer's [configuration](#configure-co [...]
> One of the benefits of compaction in Pulsar is that you aren't forced to
> choose between compacted and non-compacted topics, as the compaction process
> leaves the original topic as-is and essentially adds an alternate topic. In
> other words, you can run compaction on a topic and consumers that need
> access to the non-compacted version of the topic will not be adversely
> affected.
-## Configuring compaction to run automatically
+## Configure compaction to run automatically
-Tenant administrators can configure a policy for compaction at the namespace
level. The policy specifies how large the topic backlog can grow before
compaction is triggered.
+Compaction policy specifies how large the topic backlog can grow before
compaction is triggered.
-For example, to trigger compaction when the backlog reaches 100MB:
+Tenant administrators can configure a compaction policy at namespace or topic
levels. Configuring the compaction policy at the namespace level applies to all
topics within that namespace.
+
+For example, to trigger compaction in a namespace when the backlog reaches
100MB:
```bash
@@ -41,9 +43,13 @@ $ bin/pulsar-admin namespaces set-compaction-threshold \
```
-Configuring the compaction threshold on a namespace will apply to all topics
within that namespace.
+:::note
+
+To configure the compaction policy at the topic level, you need to enable
[topic-level
policy](concepts-multi-tenancy.md#namespace-change-events-and-topic-level-policies)
first.
+
+:::
-## Triggering compaction manually
+## Trigger compaction manually
In order to run compaction on a topic, you need to use the [`topics
compact`](reference-pulsar-admin.md#topics-compact) command for the
[`pulsar-admin`](reference-pulsar-admin.md) CLI tool. Here's an example:
@@ -79,15 +85,15 @@ $ bin/pulsar compact-topic \
```
-#### When should I trigger compaction?
+:::tip
-How often you [trigger compaction](#triggering-compaction-manually) will vary
widely based on the use case. If you want a compacted topic to be extremely
speedy on read, then you should run compaction fairly frequently.
+The frequency to trigger topic compaction varies widely based on use cases. If
you want a compacted topic to be extremely speedy on read, then you need to run
compaction fairly frequently.
-## Consumer configuration
+:::
-Pulsar consumers and readers need to be configured to read from compacted
topics. The sections below show you how to enable compacted topic reads for
Pulsar's language clients.
+## Configure consumers
-### Java
+Pulsar consumers and readers need to be configured to read from compacted
topics. The section below introduces how to enable compacted topic reads for
Java clients.
In order to read from a compacted topic using a Java consumer, the
`readCompacted` parameter must be set to `true`. Here's an example consumer for
a compacted topic:
diff --git
a/site2/website-next/versioned_docs/version-2.10.x/io-debezium-source.md
b/site2/website-next/versioned_docs/version-2.10.x/io-debezium-source.md
index aedbd18dce4..b487ab6477e 100644
--- a/site2/website-next/versioned_docs/version-2.10.x/io-debezium-source.md
+++ b/site2/website-next/versioned_docs/version-2.10.x/io-debezium-source.md
@@ -25,10 +25,10 @@ The configuration of Debezium source connector has the
following properties.
| `key.converter` | true | null | The converter provided by Kafka Connect to
convert record key. |
| `value.converter` | true | null | The converter provided by Kafka Connect to
convert record value. |
| `database.history` | true | null | The name of the database history class. |
-| `database.history.pulsar.topic` | true | null | The name of the database
history topic where the connector writes and recovers DDL statements. <br /><br
/>**Note: this topic is for internal use only and should not be used by
consumers.** |
+| `database.history.pulsar.topic` | true | null | The name of the database
history topic where the connector writes and recovers DDL statements. <br /><br
/>**Note: This topic is for internal use only and should not be used by
consumers.** |
| `database.history.pulsar.service.url` | true | null | Pulsar cluster service
URL for history topic. |
| `offset.storage.topic` | true | null | Record the last committed offsets
that the connector successfully completes. |
-| `json-with-envelope` | false | false | Present the message only consist of
payload.
+| `json-with-envelope` | false | false | Present the message that only
consists of payload.|
| `database.history.pulsar.reader.config` | false | null | The configs of the
reader for the database schema history topic, in the form of a JSON string with
key-value pairs. <br />**Note:** This property is only available in 2.10.2 and
later versions. |
| `offset.storage.reader.config` | false | null | The configs of the reader
for the kafka connector offsets topic, in the form of a JSON string with
key-value pairs. <br />**Note:** This property is only available in 2.10.2 and
later versions.|
diff --git
a/site2/website-next/versioned_docs/version-2.10.x/reference-configuration.md
b/site2/website-next/versioned_docs/version-2.10.x/reference-configuration.md
index cfd6a794957..d21b890b99d 100644
---
a/site2/website-next/versioned_docs/version-2.10.x/reference-configuration.md
+++
b/site2/website-next/versioned_docs/version-2.10.x/reference-configuration.md
@@ -40,7 +40,7 @@ BookKeeper is a replicated log storage system that Pulsar
uses for durable stora
|zkLedgersRootPath|The root ZooKeeper path used to store ledger metadata. This
parameter is used by the ZooKeeper-based ledger manager as a root znode to
store all ledgers.|/ledgers|
|ledgerStorageClass|Ledger storage implementation
class|org.apache.bookkeeper.bookie.storage.ldb.DbLedgerStorage|
|entryLogFilePreallocationEnabled|Enable or disable entry logger
preallocation|true|
-|logSizeLimit|Max file size of the entry logger, in bytes. A new entry log
file will be created when the old one reaches the file size
limitation.|2147483648|
+|logSizeLimit|Max file size of the entry logger, in bytes. A new entry log
file will be created when the old one reaches the file size
limitation.|1073741824|
|minorCompactionThreshold|Threshold of minor compaction. Entry log files whose
remaining size percentage reaches below this threshold will be compacted in a
minor compaction. If set to less than zero, the minor compaction is
disabled.|0.2|
|minorCompactionInterval|Time interval to run minor compaction, in seconds. If
set to less than zero, the minor compaction is disabled. Note: should be
greater than gcWaitTime. |3600|
|majorCompactionThreshold|The threshold of major compaction. Entry log files
whose remaining size percentage reaches below this threshold will be compacted
in a major compaction. Those entry log files whose remaining size percentage is
still higher than the threshold will never be compacted. If set to less than
zero, the minor compaction is disabled.|0.5|
diff --git
a/site2/website-next/versioned_docs/version-2.10.x/security-policy-and-supported-versions.md
b/site2/website-next/versioned_docs/version-2.10.x/security-policy-and-supported-versions.md
index b7fe6f2e156..637147a5dc2 100644
---
a/site2/website-next/versioned_docs/version-2.10.x/security-policy-and-supported-versions.md
+++
b/site2/website-next/versioned_docs/version-2.10.x/security-policy-and-supported-versions.md
@@ -2,7 +2,6 @@
id: security-policy-and-supported-versions
title: Security Policy and Supported Versions
sidebar_label: "Security Policy and Supported Versions"
-original_id: security-policy-and-supported-versions
---
## Using Pulsar's Security Features
@@ -10,10 +9,13 @@ original_id: security-policy-and-supported-versions
You can find documentation on Pulsar's available security features and how to
use them here:
https://pulsar.apache.org/docs/en/security-overview/.
-## Security Vulnerability Announcements
+## Security Vulnerability Process
-The Pulsar community will announce security vulnerabilities and how to
mitigate them on the [[email protected]](mailto:[email protected]).
-For instructions on how to subscribe, please see
https://pulsar.apache.org/contact/.
+The Pulsar community follows the ASF [security vulnerability handling
process](https://apache.org/security/#vulnerability-handling).
+
+To report a new vulnerability you have discovered, please follow the [ASF
security vulnerability reporting
process](https://apache.org/security/#reporting-a-vulnerability). To report a
vulnerability for Pulsar, contact the [Apache Security
Team](https://www.apache.org/security/). When reporting a vulnerability to
[[email protected]](mailto:[email protected]), you can copy your email to
[[email protected]](mailto:[email protected]) to send your
report to the Apache Pul [...]
+
+It is the responsibility of the security vulnerability handling project team
(Apache Pulsar PMC in most cases) to make public security vulnerability
announcements. You can follow announcements on the
[[email protected]](mailto:[email protected]) mailing list. For
instructions on how to subscribe, please see https://pulsar.apache.org/contact/.
## Versioning Policy
diff --git
a/site2/website-next/versioned_docs/version-2.10.x/tiered-storage-filesystem.md
b/site2/website-next/versioned_docs/version-2.10.x/tiered-storage-filesystem.md
index fb39290ef8f..bb399b500cb 100644
---
a/site2/website-next/versioned_docs/version-2.10.x/tiered-storage-filesystem.md
+++
b/site2/website-next/versioned_docs/version-2.10.x/tiered-storage-filesystem.md
@@ -109,7 +109,7 @@ You can configure the filesystem offloader driver in the
`broker.conf` or `stand
`managedLedgerOffloadDriver` | Offloader driver name, which is
case-insensitive. | filesystem
`fileSystemURI` | Connection address, which is the URI to access the default
Hadoop distributed file system. | hdfs://127.0.0.1:9000
`offloadersDirectory` | Offloader directory | offloaders
- `fileSystemProfilePath` | Hadoop profile path. The configuration file is
stored in the Hadoop profile path. It contains various settings for Hadoop
performance tuning. | ../conf/filesystem_offload_core_site.xml
+ `fileSystemProfilePath` | Hadoop profile path. The configuration file is
stored in the Hadoop profile path. It contains various settings for Hadoop
performance tuning. | conf/filesystem_offload_core_site.xml
- **Optional** configurations are as below.
@@ -128,7 +128,7 @@ You can configure the filesystem offloader driver in the
`broker.conf` or `stand
|---|---|---
`managedLedgerOffloadDriver` | Offloader driver name, which is
case-insensitive. | filesystem
`offloadersDirectory` | Offloader directory | offloaders
- `fileSystemProfilePath` | NFS profile path. The configuration file is stored
in the NFS profile path. It contains various settings for performance tuning. |
../conf/filesystem_offload_core_site.xml
+ `fileSystemProfilePath` | NFS profile path. The configuration file is stored
in the NFS profile path. It contains various settings for performance tuning. |
conf/filesystem_offload_core_site.xml
- **Optional** configurations are as below.
@@ -370,7 +370,7 @@ Set the following configurations in the
`conf/standalone.conf` file.
managedLedgerOffloadDriver=filesystem
fileSystemURI=hdfs://127.0.0.1:9000
-fileSystemProfilePath=../conf/filesystem_offload_core_site.xml
+fileSystemProfilePath=conf/filesystem_offload_core_site.xml
```
@@ -421,7 +421,7 @@ As indicated in the [configuration](#configuration)
section, you need to configu
```conf
managedLedgerOffloadDriver=filesystem
- fileSystemProfilePath=../conf/filesystem_offload_core_site.xml
+ fileSystemProfilePath=conf/filesystem_offload_core_site.xml
```
diff --git
a/site2/website-next/versioned_docs/version-2.8.x/client-libraries-java.md
b/site2/website-next/versioned_docs/version-2.8.x/client-libraries-java.md
index 8c149ec7b71..d430e7ca2c0 100644
--- a/site2/website-next/versioned_docs/version-2.8.x/client-libraries-java.md
+++ b/site2/website-next/versioned_docs/version-2.8.x/client-libraries-java.md
@@ -128,7 +128,7 @@ long|`statsIntervalSeconds`|Interval between each stats
info<br /><br />Stats is
int|`numIoThreads`| The number of threads used for handling connections to
brokers | 1
int|`numListenerThreads`|The number of threads used for handling message
listeners. The listener thread pool is shared across all the consumers and
readers using the "listener" model to get messages. For a given consumer, the
listener is always invoked from the same thread to ensure ordering. If you want
multiple threads to process a single topic, you need to create a
[`shared`](https://pulsar.apache.org/docs/en/next/concepts-messaging/#shared)
subscription and multiple consumers for thi [...]
boolean|`useTcpNoDelay`|Whether to use TCP no-delay flag on the connection to
disable Nagle algorithm |true
-boolean |`useTls` |Whether to use TLS encryption on the connection| false
+boolean |`enableTls` |Whether to use TLS encryption on the connection. Note
that this parameter is **deprecated**. If you want to enable TLS, use
`pulsar+ssl://` in `serviceUrl` instead.| false
string | `tlsTrustCertsFilePath` |Path to the trusted TLS certificate file|None
boolean|`tlsAllowInsecureConnection`|Whether the Pulsar client accepts
untrusted TLS certificate from broker | false
boolean | `tlsHostnameVerificationEnable` | Whether to enable TLS hostname
verification|false
diff --git
a/site2/website-next/versioned_docs/version-2.8.x/cookbooks-compaction.md
b/site2/website-next/versioned_docs/version-2.8.x/cookbooks-compaction.md
index dfa31472724..8e4ed064438 100644
--- a/site2/website-next/versioned_docs/version-2.8.x/cookbooks-compaction.md
+++ b/site2/website-next/versioned_docs/version-2.8.x/cookbooks-compaction.md
@@ -10,8 +10,8 @@ Pulsar's [topic
compaction](concepts-topic-compaction.md#compaction) feature ena
To use compaction:
* You need to give messages keys, as topic compaction in Pulsar takes place on
a *per-key basis* (i.e. messages are compacted based on their key). For a stock
ticker use case, the stock symbol---e.g. `AAPL` or `GOOG`---could serve as the
key (more on this [below](#when-should-i-use-compacted-topics)). Messages
without keys will be left alone by the compaction process.
-* Compaction can be configured to run
[automatically](#configuring-compaction-to-run-automatically), or you can
manually [trigger](#triggering-compaction-manually) compaction using the Pulsar
administrative API.
-* Your consumers must be [configured](#consumer-configuration) to read from
compacted topics ([Java consumers](#java), for example, have a `readCompacted`
setting that must be set to `true`). If this configuration is not set,
consumers will still be able to read from the non-compacted topic.
+* Compaction can be configured to run
[automatically](#configure-compaction-to-run-automatically), or you can
manually [trigger](#trigger-compaction-manually) compaction using the Pulsar
administrative API.
+* Your consumers must be [configured](#configure-consumers) to read from
compacted topics (Java consumers, for example, have a `readCompacted` setting
that must be set to `true`). If this configuration is not set, consumers will
still be able to read from the non-compacted topic.
> Compaction only works on messages that have keys (as in the stock ticker
> example the stock symbol serves as the key for each message). Keys can thus
> be thought of as the axis along which compaction is applied. Messages that
> don't have keys are simply ignored by compaction.
@@ -23,16 +23,18 @@ The classic example of a topic that could benefit from
compaction would be a sto
* They can read from the "original," non-compacted topic in case they need
access to "historical" values, i.e. the entirety of the topic's messages.
* They can read from the compacted topic if they only want to see the most
up-to-date messages.
-Thus, if you're using a Pulsar topic called `stock-values`, some consumers
could have access to all messages in the topic (perhaps because they're
performing some kind of number crunching of all values in the last hour) while
the consumers used to power the real-time stock ticker only see the compacted
topic (and thus aren't forced to process outdated messages). Which variant of
the topic any given consumer pulls messages from is determined by the
consumer's [configuration](#consumer-con [...]
+Thus, if you're using a Pulsar topic called `stock-values`, some consumers
could have access to all messages in the topic (perhaps because they're
performing some kind of number crunching of all values in the last hour) while
the consumers used to power the real-time stock ticker only see the compacted
topic (and thus aren't forced to process outdated messages). Which variant of
the topic any given consumer pulls messages from is determined by the
consumer's [configuration](#configure-co [...]
> One of the benefits of compaction in Pulsar is that you aren't forced to
> choose between compacted and non-compacted topics, as the compaction process
> leaves the original topic as-is and essentially adds an alternate topic. In
> other words, you can run compaction on a topic and consumers that need
> access to the non-compacted version of the topic will not be adversely
> affected.
-## Configuring compaction to run automatically
+## Configure compaction to run automatically
-Tenant administrators can configure a policy for compaction at the namespace
level. The policy specifies how large the topic backlog can grow before
compaction is triggered.
+Compaction policy specifies how large the topic backlog can grow before
compaction is triggered.
-For example, to trigger compaction when the backlog reaches 100MB:
+Tenant administrators can configure a compaction policy at namespace or topic
levels. Configuring the compaction policy at the namespace level applies to all
topics within that namespace.
+
+For example, to trigger compaction in a namespace when the backlog reaches
100MB:
```bash
@@ -41,9 +43,13 @@ $ bin/pulsar-admin namespaces set-compaction-threshold \
```
-Configuring the compaction threshold on a namespace will apply to all topics
within that namespace.
+:::note
+
+To configure the compaction policy at the topic level, you need to enable
[topic-level
policy](concepts-multi-tenancy.md#namespace-change-events-and-topic-level-policies)
first.
+
+:::
-## Triggering compaction manually
+## Trigger compaction manually
In order to run compaction on a topic, you need to use the [`topics
compact`](reference-pulsar-admin.md#topics-compact) command for the
[`pulsar-admin`](reference-pulsar-admin.md) CLI tool. Here's an example:
@@ -79,15 +85,15 @@ $ bin/pulsar compact-topic \
```
-#### When should I trigger compaction?
+:::tip
-How often you [trigger compaction](#triggering-compaction-manually) will vary
widely based on the use case. If you want a compacted topic to be extremely
speedy on read, then you should run compaction fairly frequently.
+The frequency to trigger topic compaction varies widely based on use cases. If
you want a compacted topic to be extremely speedy on read, then you need to run
compaction fairly frequently.
-## Consumer configuration
+:::
-Pulsar consumers and readers need to be configured to read from compacted
topics. The sections below show you how to enable compacted topic reads for
Pulsar's language clients.
+## Configure consumers
-### Java
+Pulsar consumers and readers need to be configured to read from compacted
topics. The section below introduces how to enable compacted topic reads for
Java clients.
In order to read from a compacted topic using a Java consumer, the
`readCompacted` parameter must be set to `true`. Here's an example consumer for
a compacted topic:
diff --git
a/site2/website-next/versioned_docs/version-2.8.x/developing-binary-protocol.md
b/site2/website-next/versioned_docs/version-2.8.x/developing-binary-protocol.md
index 7a2a034b044..1f225c52df1 100644
---
a/site2/website-next/versioned_docs/version-2.8.x/developing-binary-protocol.md
+++
b/site2/website-next/versioned_docs/version-2.8.x/developing-binary-protocol.md
@@ -302,7 +302,8 @@ subscription is not already there, a new one will be
created.
:::note
-In 2.8.4 and later versions, if the client does not receive a response
indicating the success or failure of consumer creation, it first sends a
command to close the original consumer before sending a command to re-attempt
consumer creation.
+* Before creating or connecting a consumer, you need to perform [topic
lookup](#topic-lookup) first.
+* In 2.8.4 and later versions, if the client does not receive a response
indicating the success or failure of consumer creation, it first sends a
command to close the original consumer before sending a command to re-attempt
consumer creation.
:::
diff --git
a/site2/website-next/versioned_docs/version-2.8.x/reference-configuration.md
b/site2/website-next/versioned_docs/version-2.8.x/reference-configuration.md
index b58d3f23166..6ef75bac740 100644
--- a/site2/website-next/versioned_docs/version-2.8.x/reference-configuration.md
+++ b/site2/website-next/versioned_docs/version-2.8.x/reference-configuration.md
@@ -41,7 +41,7 @@ BookKeeper is a replicated log storage system that Pulsar
uses for durable stora
|zkLedgersRootPath|The root ZooKeeper path used to store ledger metadata. This
parameter is used by the ZooKeeper-based ledger manager as a root znode to
store all ledgers.|/ledgers|
|ledgerStorageClass|Ledger storage implementation
class|org.apache.bookkeeper.bookie.storage.ldb.DbLedgerStorage|
|entryLogFilePreallocationEnabled|Enable or disable entry logger
preallocation|true|
-|logSizeLimit|Max file size of the entry logger, in bytes. A new entry log
file will be created when the old one reaches the file size
limitation.|2147483648|
+|logSizeLimit|Max file size of the entry logger, in bytes. A new entry log
file will be created when the old one reaches the file size
limitation.|1073741824|
|minorCompactionThreshold|Threshold of minor compaction. Entry log files whose
remaining size percentage reaches below this threshold will be compacted in a
minor compaction. If set to less than zero, the minor compaction is
disabled.|0.2|
|minorCompactionInterval|Time interval to run minor compaction, in seconds. If
set to less than zero, the minor compaction is disabled. Note: should be
greater than gcWaitTime. |3600|
|majorCompactionThreshold|The threshold of major compaction. Entry log files
whose remaining size percentage reaches below this threshold will be compacted
in a major compaction. Those entry log files whose remaining size percentage is
still higher than the threshold will never be compacted. If set to less than
zero, the minor compaction is disabled.|0.5|
diff --git
a/site2/website-next/versioned_docs/version-2.8.x/tiered-storage-filesystem.md
b/site2/website-next/versioned_docs/version-2.8.x/tiered-storage-filesystem.md
index 4456b615afa..85a1644120f 100644
---
a/site2/website-next/versioned_docs/version-2.8.x/tiered-storage-filesystem.md
+++
b/site2/website-next/versioned_docs/version-2.8.x/tiered-storage-filesystem.md
@@ -109,7 +109,7 @@ You can configure the filesystem offloader driver in the
`broker.conf` or `stand
`managedLedgerOffloadDriver` | Offloader driver name, which is
case-insensitive. | filesystem
`fileSystemURI` | Connection address, which is the URI to access the default
Hadoop distributed file system. | hdfs://127.0.0.1:9000
`offloadersDirectory` | Offloader directory | offloaders
- `fileSystemProfilePath` | Hadoop profile path. The configuration file is
stored in the Hadoop profile path. It contains various settings for Hadoop
performance tuning. | ../conf/filesystem_offload_core_site.xml
+ `fileSystemProfilePath` | Hadoop profile path. The configuration file is
stored in the Hadoop profile path. It contains various settings for Hadoop
performance tuning. | conf/filesystem_offload_core_site.xml
- **Optional** configurations are as below.
@@ -127,7 +127,7 @@ You can configure the filesystem offloader driver in the
`broker.conf` or `stand
|---|---|---
`managedLedgerOffloadDriver` | Offloader driver name, which is
case-insensitive. | filesystem
`offloadersDirectory` | Offloader directory | offloaders
- `fileSystemProfilePath` | NFS profile path. The configuration file is stored
in the NFS profile path. It contains various settings for performance tuning. |
../conf/filesystem_offload_core_site.xml
+ `fileSystemProfilePath` | NFS profile path. The configuration file is stored
in the NFS profile path. It contains various settings for performance tuning. |
conf/filesystem_offload_core_site.xml
- **Optional** configurations are as below.
@@ -369,7 +369,7 @@ Set the following configurations in the
`conf/standalone.conf` file.
managedLedgerOffloadDriver=filesystem
fileSystemURI=hdfs://127.0.0.1:9000
-fileSystemProfilePath=../conf/filesystem_offload_core_site.xml
+fileSystemProfilePath=conf/filesystem_offload_core_site.xml
```
@@ -420,7 +420,7 @@ As indicated in the [configuration](#configuration)
section, you need to configu
```conf
managedLedgerOffloadDriver=filesystem
- fileSystemProfilePath=../conf/filesystem_offload_core_site.xml
+ fileSystemProfilePath=conf/filesystem_offload_core_site.xml
```
diff --git
a/site2/website-next/versioned_docs/version-2.9.x/cookbooks-compaction.md
b/site2/website-next/versioned_docs/version-2.9.x/cookbooks-compaction.md
index dfa31472724..8e4ed064438 100644
--- a/site2/website-next/versioned_docs/version-2.9.x/cookbooks-compaction.md
+++ b/site2/website-next/versioned_docs/version-2.9.x/cookbooks-compaction.md
@@ -10,8 +10,8 @@ Pulsar's [topic
compaction](concepts-topic-compaction.md#compaction) feature ena
To use compaction:
* You need to give messages keys, as topic compaction in Pulsar takes place on
a *per-key basis* (i.e. messages are compacted based on their key). For a stock
ticker use case, the stock symbol---e.g. `AAPL` or `GOOG`---could serve as the
key (more on this [below](#when-should-i-use-compacted-topics)). Messages
without keys will be left alone by the compaction process.
-* Compaction can be configured to run
[automatically](#configuring-compaction-to-run-automatically), or you can
manually [trigger](#triggering-compaction-manually) compaction using the Pulsar
administrative API.
-* Your consumers must be [configured](#consumer-configuration) to read from
compacted topics ([Java consumers](#java), for example, have a `readCompacted`
setting that must be set to `true`). If this configuration is not set,
consumers will still be able to read from the non-compacted topic.
+* Compaction can be configured to run
[automatically](#configure-compaction-to-run-automatically), or you can
manually [trigger](#trigger-compaction-manually) compaction using the Pulsar
administrative API.
+* Your consumers must be [configured](#configure-consumers) to read from
compacted topics (Java consumers, for example, have a `readCompacted` setting
that must be set to `true`). If this configuration is not set, consumers will
still be able to read from the non-compacted topic.
> Compaction only works on messages that have keys (as in the stock ticker
> example the stock symbol serves as the key for each message). Keys can thus
> be thought of as the axis along which compaction is applied. Messages that
> don't have keys are simply ignored by compaction.
@@ -23,16 +23,18 @@ The classic example of a topic that could benefit from
compaction would be a sto
* They can read from the "original," non-compacted topic in case they need
access to "historical" values, i.e. the entirety of the topic's messages.
* They can read from the compacted topic if they only want to see the most
up-to-date messages.
-Thus, if you're using a Pulsar topic called `stock-values`, some consumers
could have access to all messages in the topic (perhaps because they're
performing some kind of number crunching of all values in the last hour) while
the consumers used to power the real-time stock ticker only see the compacted
topic (and thus aren't forced to process outdated messages). Which variant of
the topic any given consumer pulls messages from is determined by the
consumer's [configuration](#consumer-con [...]
+Thus, if you're using a Pulsar topic called `stock-values`, some consumers
could have access to all messages in the topic (perhaps because they're
performing some kind of number crunching of all values in the last hour) while
the consumers used to power the real-time stock ticker only see the compacted
topic (and thus aren't forced to process outdated messages). Which variant of
the topic any given consumer pulls messages from is determined by the
consumer's [configuration](#configure-co [...]
> One of the benefits of compaction in Pulsar is that you aren't forced to
> choose between compacted and non-compacted topics, as the compaction process
> leaves the original topic as-is and essentially adds an alternate topic. In
> other words, you can run compaction on a topic and consumers that need
> access to the non-compacted version of the topic will not be adversely
> affected.
-## Configuring compaction to run automatically
+## Configure compaction to run automatically
-Tenant administrators can configure a policy for compaction at the namespace
level. The policy specifies how large the topic backlog can grow before
compaction is triggered.
+Compaction policy specifies how large the topic backlog can grow before
compaction is triggered.
-For example, to trigger compaction when the backlog reaches 100MB:
+Tenant administrators can configure a compaction policy at namespace or topic
levels. Configuring the compaction policy at the namespace level applies to all
topics within that namespace.
+
+For example, to trigger compaction in a namespace when the backlog reaches
100MB:
```bash
@@ -41,9 +43,13 @@ $ bin/pulsar-admin namespaces set-compaction-threshold \
```
-Configuring the compaction threshold on a namespace will apply to all topics
within that namespace.
+:::note
+
+To configure the compaction policy at the topic level, you need to enable
[topic-level
policy](concepts-multi-tenancy.md#namespace-change-events-and-topic-level-policies)
first.
+
+:::
-## Triggering compaction manually
+## Trigger compaction manually
In order to run compaction on a topic, you need to use the [`topics
compact`](reference-pulsar-admin.md#topics-compact) command for the
[`pulsar-admin`](reference-pulsar-admin.md) CLI tool. Here's an example:
@@ -79,15 +85,15 @@ $ bin/pulsar compact-topic \
```
-#### When should I trigger compaction?
+:::tip
-How often you [trigger compaction](#triggering-compaction-manually) will vary
widely based on the use case. If you want a compacted topic to be extremely
speedy on read, then you should run compaction fairly frequently.
+The frequency to trigger topic compaction varies widely based on use cases. If
you want a compacted topic to be extremely speedy on read, then you need to run
compaction fairly frequently.
-## Consumer configuration
+:::
-Pulsar consumers and readers need to be configured to read from compacted
topics. The sections below show you how to enable compacted topic reads for
Pulsar's language clients.
+## Configure consumers
-### Java
+Pulsar consumers and readers need to be configured to read from compacted
topics. The section below introduces how to enable compacted topic reads for
Java clients.
In order to read from a compacted topic using a Java consumer, the
`readCompacted` parameter must be set to `true`. Here's an example consumer for
a compacted topic:
diff --git
a/site2/website-next/versioned_docs/version-2.9.x/io-debezium-source.md
b/site2/website-next/versioned_docs/version-2.9.x/io-debezium-source.md
index f739a4cdc49..4ed7f4a8d26 100644
--- a/site2/website-next/versioned_docs/version-2.9.x/io-debezium-source.md
+++ b/site2/website-next/versioned_docs/version-2.9.x/io-debezium-source.md
@@ -25,10 +25,10 @@ The configuration of Debezium source connector has the
following properties.
| `key.converter` | true | null | The converter provided by Kafka Connect to
convert record key. |
| `value.converter` | true | null | The converter provided by Kafka Connect to
convert record value. |
| `database.history` | true | null | The name of the database history class. |
-| `database.history.pulsar.topic` | true | null | The name of the database
history topic where the connector writes and recovers DDL statements. <br /><br
/>**Note: this topic is for internal use only and should not be used by
consumers.** |
+| `database.history.pulsar.topic` | true | null | The name of the database
history topic where the connector writes and recovers DDL statements. <br /><br
/>**Note: This topic is for internal use only and should not be used by
consumers.** |
| `database.history.pulsar.service.url` | true | null | Pulsar cluster service
URL for history topic. |
| `offset.storage.topic` | true | null | Record the last committed offsets
that the connector successfully completes. |
-| `json-with-envelope` | false | false | Present the message only consist of
payload.
+| `json-with-envelope` | false | false | Present the message that only
consists of payload.|
| `database.history.pulsar.reader.config` | false | null | The configs of the
reader for the database schema history topic, in the form of a JSON string with
key-value pairs. <br />**Note:** This property is only available in 2.9.4 and
later versions. |
| `offset.storage.reader.config` | false | null | The configs of the reader
for the kafka connector offsets topic, in the form of a JSON string with
key-value pairs. <br />**Note:** This property is only available in 2.9.4 and
later versions.|
diff --git
a/site2/website-next/versioned_docs/version-2.9.x/reference-configuration.md
b/site2/website-next/versioned_docs/version-2.9.x/reference-configuration.md
index 11b77269900..e90124fbe1d 100644
--- a/site2/website-next/versioned_docs/version-2.9.x/reference-configuration.md
+++ b/site2/website-next/versioned_docs/version-2.9.x/reference-configuration.md
@@ -40,7 +40,7 @@ BookKeeper is a replicated log storage system that Pulsar
uses for durable stora
|zkLedgersRootPath|The root ZooKeeper path used to store ledger metadata. This
parameter is used by the ZooKeeper-based ledger manager as a root znode to
store all ledgers.|/ledgers|
|ledgerStorageClass|Ledger storage implementation
class|org.apache.bookkeeper.bookie.storage.ldb.DbLedgerStorage|
|entryLogFilePreallocationEnabled|Enable or disable entry logger
preallocation|true|
-|logSizeLimit|Max file size of the entry logger, in bytes. A new entry log
file will be created when the old one reaches the file size
limitation.|2147483648|
+|logSizeLimit|Max file size of the entry logger, in bytes. A new entry log
file will be created when the old one reaches the file size
limitation.|1073741824|
|minorCompactionThreshold|Threshold of minor compaction. Entry log files whose
remaining size percentage reaches below this threshold will be compacted in a
minor compaction. If set to less than zero, the minor compaction is
disabled.|0.2|
|minorCompactionInterval|Time interval to run minor compaction, in seconds. If
set to less than zero, the minor compaction is disabled. Note: should be
greater than gcWaitTime. |3600|
|majorCompactionThreshold|The threshold of major compaction. Entry log files
whose remaining size percentage reaches below this threshold will be compacted
in a major compaction. Those entry log files whose remaining size percentage is
still higher than the threshold will never be compacted. If set to less than
zero, the minor compaction is disabled.|0.5|
diff --git
a/site2/website-next/versioned_docs/version-2.9.x/tiered-storage-filesystem.md
b/site2/website-next/versioned_docs/version-2.9.x/tiered-storage-filesystem.md
index a5844d22fb5..8164e68208b 100644
---
a/site2/website-next/versioned_docs/version-2.9.x/tiered-storage-filesystem.md
+++
b/site2/website-next/versioned_docs/version-2.9.x/tiered-storage-filesystem.md
@@ -98,7 +98,7 @@ You can configure filesystem offloader driver in the
configuration file `broker.
|---|---|---
`managedLedgerOffloadDriver` | Offloader driver name, which is
case-insensitive. | filesystem
`fileSystemURI` | Connection address | hdfs://127.0.0.1:9000
- `fileSystemProfilePath` | Hadoop profile path |
../conf/filesystem_offload_core_site.xml
+ `fileSystemProfilePath` | Hadoop profile path |
conf/filesystem_offload_core_site.xml
- **Optional** configurations are as below.
@@ -139,11 +139,11 @@ The configuration file is stored in the Hadoop profile
path. It contains various
##### Example
-This example sets the Hadoop profile path as
_../conf/filesystem_offload_core_site.xml_.
+This example sets the Hadoop profile path as
_conf/filesystem_offload_core_site.xml_.
```conf
-fileSystemProfilePath=../conf/filesystem_offload_core_site.xml
+fileSystemProfilePath=conf/filesystem_offload_core_site.xml
```
