merlimat closed pull request #2237: [website] Fix security page links and
includes
URL: https://github.com/apache/incubator-pulsar/pull/2237
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/site2/docs/adaptors-kafka.md b/site2/docs/adaptors-kafka.md
index a3f309db53..4c945909f3 100644
--- a/site2/docs/adaptors-kafka.md
+++ b/site2/docs/adaptors-kafka.md
@@ -25,7 +25,7 @@ Then include this dependency for the Pulsar Kafka wrapper:
<dependency>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar-client-kafka</artifactId>
- <version>pulsar:version</version>
+ <version>{{pulsar:version}}</version>
</dependency>
```
@@ -44,7 +44,7 @@ unshaded pulsar kafka client wrapper.
<dependency>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar-client-kafka-original</artifactId>
- <version>pulsar:version</version>
+ <version>{{pulsar:version}}</version>
</dependency>
```
diff --git a/site2/docs/adaptors-spark.md b/site2/docs/adaptors-spark.md
index 1cecaa9dfe..bb92db121f 100644
--- a/site2/docs/adaptors-spark.md
+++ b/site2/docs/adaptors-spark.md
@@ -18,7 +18,7 @@ If you're using Maven, add this to your `pom.xml`:
```xml
<!-- in your <properties> block -->
-<pulsar.version>pulsar:version</pulsar.version>
+<pulsar.version>{{pulsar:version}}</pulsar.version>
<!-- in your <dependencies> block -->
<dependency>
@@ -33,7 +33,7 @@ If you're using Maven, add this to your `pom.xml`:
If you're using Gradle, add this to your `build.gradle` file:
```groovy
-def pulsarVersion = "pulsar:version"
+def pulsarVersion = "{{pulsar:version}}"
dependencies {
compile group: 'org.apache.pulsar', name: 'pulsar-spark', version:
pulsarVersion
diff --git a/site2/docs/admin-api-brokers.md b/site2/docs/admin-api-brokers.md
index 62199bfdcb..5ee9295645 100644
--- a/site2/docs/admin-api-brokers.md
+++ b/site2/docs/admin-api-brokers.md
@@ -17,9 +17,7 @@ Pulsar brokers consist of two components:
In addition to being configurable when you start them up, brokers can also be
[dynamically configured](#dynamic-broker-configuration).
-{% include admonition.html type="info" content="
-See the [Configuration](reference-configuration.md#broker) page for a full
listing of broker-specific configuration parameters.
-" %}
+> See the [Configuration](reference-configuration.md#broker) page for a full
listing of broker-specific configuration parameters.
## Brokers resources
diff --git a/site2/docs/admin-api-clusters.md b/site2/docs/admin-api-clusters.md
index d808b4de9d..bbaf337571 100644
--- a/site2/docs/admin-api-clusters.md
+++ b/site2/docs/admin-api-clusters.md
@@ -20,8 +20,6 @@ New clusters can be provisioned using the admin interface.
> Please note that this operation requires superuser privileges.
-{% include message.html id="superuser" %}
-
#### pulsar-admin
You can provision a new cluster using the
[`create`](reference-pulsar-admin.md#clusters-create) subcommand. Here's an
example:
@@ -62,9 +60,12 @@ When provision a new cluster, you need to initialize that
cluster's [metadata](g
You must initialize cluster metadata *before* starting up any
[brokers](admin-api-brokers.md) that will belong to the cluster.
-{% include admonition.html type="warning" title="No cluster metadata
initialization through the REST API or the Java admin API" content='
-Unlike most other admin functions in Pulsar, cluster metadata initialization
cannot be performed via the admin REST API or the admin Java client, as
metadata initialization involves communicating with ZooKeeper directly.
Instead, you can use the [`pulsar`](reference-cli-tools.md#pulsar) CLI tool, in
particular the
[`initialize-cluster-metadata`](reference-cli-tools.md#pulsar-initialize-cluster-metadata)
command.
-' %}
+> #### No cluster metadata initialization through the REST API or the Java
admin API
+>
+> Unlike most other admin functions in Pulsar, cluster metadata initialization
cannot be performed via the admin REST API
+> or the admin Java client, as metadata initialization involves communicating
with ZooKeeper directly.
+> Instead, you can use the [`pulsar`](reference-cli-tools.md#pulsar) CLI tool,
in particular
+> the
[`initialize-cluster-metadata`](reference-cli-tools.md#pulsar-initialize-cluster-metadata)
command.
Here's an example cluster metadata initialization command:
@@ -79,7 +80,7 @@ bin/pulsar initialize-cluster-metadata \
--broker-service-url-tls pulsar+ssl://pulsar.us-west.example.com:6651/
```
-You'll need to use `--*-tls` flags only if you're using [TLS
authentication](administration-auth.md#tls-client-auth) in your instance.
+You'll need to use `--*-tls` flags only if you're using [TLS
authentication](security-tls.md) in your instance.
### Get configuration
diff --git a/site2/docs/admin-api-non-persistent-topics.md
b/site2/docs/admin-api-non-persistent-topics.md
index eef44079a3..65b0ae769e 100644
--- a/site2/docs/admin-api-non-persistent-topics.md
+++ b/site2/docs/admin-api-non-persistent-topics.md
@@ -10,7 +10,9 @@ persisting messages.
In all of the instructions and commands below, the topic name structure is:
-{% include topic.html ten="tenant" n="namespace" t="topic" %}
+```shell
+persistent://tenant/namespace/topic
+```
## Non-persistent topics resources
diff --git a/site2/docs/admin-api-overview.md b/site2/docs/admin-api-overview.md
index a730a7d130..f85752df8e 100644
--- a/site2/docs/admin-api-overview.md
+++ b/site2/docs/admin-api-overview.md
@@ -24,21 +24,25 @@ Full documentation for this tool can be found in the
[Pulsar command-line tools]
> #### The REST API is the admin interface
> Under the hood, both the `pulsar-admin` CLI tool and the Java client both
> use the REST API. If you’d like to implement your own admin interface
> client, you should use the REST API as well. Full documentation can be found
> here.
-{% include message.html id="admin_rest_api" %}
-
In this document, examples from each of the three available interfaces will be
shown.
## Admin setup
-{% include explanations/admin-setup.md %}
-
-Each of Pulsar's three admin interfaces---the
[`pulsar-admin`](reference-pulsar-admin.md) CLI tool, the [Java admin
API](/api/admin), and the [REST API](reference-rest-api.md)---requires some
special setup if you have
[authentication](administration-auth.md#authentication-providers) enabled in
your Pulsar {% popover instance %}.
+Each of Pulsar's three admin interfaces---the
[`pulsar-admin`](reference-pulsar-admin.md) CLI tool, the [Java admin
API](/api/admin), and the [REST API](reference-rest-api.md)---requires some
special setup if you have
[authentication](security-overview.md#authentication-providers) enabled in your
Pulsar {% popover instance %}.
### pulsar-admin
-If you have [authentication](administration-auth.md#authentication-providers)
enabled, you will need to provide an auth configuration to use the
[`pulsar-admin`](reference-pulsar-admin.md) tool. By default, the configuration
for the `pulsar-admin` tool is found in the
[`conf/client.conf`](reference-configuration.md#client) file. Here are the
available parameters:
-
-{% include config.html id="client" %}
+If you have [authentication](security-overview.md#authentication-providers)
enabled, you will need to provide an auth configuration to use the
[`pulsar-admin`](reference-pulsar-admin.md) tool. By default, the configuration
for the `pulsar-admin` tool is found in the
[`conf/client.conf`](reference-configuration.md#client) file. Here are the
available parameters:
+
+|Name|Description|Default|
+|----|-----------|-------|
+|webServiceUrl|The web URL for the cluster.|http://localhost:8080/|
+|brokerServiceUrl|The Pulsar protocol URL for the
cluster.|pulsar://localhost:6650/|
+|authPlugin|The authentication plugin.| |
+|authParams|The authentication parameters for the cluster, as a
comma-separated string.| |
+|useTls|Whether or not TLS authentication will be enforced in the
cluster.|false|
+|tlsAllowInsecureConnection|Accept untrusted TLS certificate from
client.|false|
+|tlsTrustCertsFilePath|Path for the trusted TLS certificate file.| |
### REST API
diff --git a/site2/docs/admin-api-partitioned-topics.md
b/site2/docs/admin-api-partitioned-topics.md
index db497186d0..f9b94f9401 100644
--- a/site2/docs/admin-api-partitioned-topics.md
+++ b/site2/docs/admin-api-partitioned-topics.md
@@ -9,7 +9,9 @@ You can use Pulsar's [admin API](admin-api-overview.md) to
create and manage par
In all of the instructions and commands below, the topic name structure is:
-{% include topic.html ten="tenant" n="namespace" t="topic" %}
+```shell
+persistent://tenant/namespace/topic
+```
## Partitioned topics resources
diff --git a/site2/docs/admin-api-persistent-topics.md
b/site2/docs/admin-api-persistent-topics.md
index 6a7cb2e1da..7138e3b0a4 100644
--- a/site2/docs/admin-api-persistent-topics.md
+++ b/site2/docs/admin-api-persistent-topics.md
@@ -8,7 +8,10 @@ Persistent helps to access topic which is a logical endpoint
for publishing and
In all of the instructions and commands below, the topic name structure is:
-{% include topic.html ten="tenant" n="namespace" t="topic" %}
+
+```shell
+persistent://tenant/namespace/topic
+```
## Persistent topics resources
diff --git a/site2/docs/administration-auth.md
b/site2/docs/administration-auth.md
deleted file mode 100644
index 392ce48060..0000000000
--- a/site2/docs/administration-auth.md
+++ /dev/null
@@ -1,434 +0,0 @@
----
-id: administration-auth
-title: Authentication and authorization in Pulsar
-sidebar_label: Authentication and authorization
----
-
-Pulsar supports a pluggable authentication mechanism that Pulsar clients can
use to authenticate with {% popover brokers %}. Pulsar can also be configured
to support multiple authentication sources.
-
-## Role tokens
-
-In Pulsar, a *role* is a string, like `admin` or `app1`, that can represent a
single client or multiple clients. Roles are used to control permission for
clients to produce or consume from certain topics, administer the configuration
for properties, and more.
-
-The purpose of the [authentication provider](#authentication-providers) is to
establish the identity of the client and then assign that client a *role
token*. This role token is then used to determine what the client is authorized
to do.
-
-## Authentication providers
-
-Out of the box, Pulsar supports two authentication providers:
-
-* [TLS client auth](#tls-client-auth)
-* [Athenz](#athenz)
-
-### TLS client auth
-
-In addition to providing connection encryption between Pulsar clients and {%
popover brokers %}, [Transport Layer
Security](https://en.wikipedia.org/wiki/Transport_Layer_Security) (TLS) can be
used to identify clients through a certificate signed by a trusted certificate
authority.
-
-#### Creating certificates
-
-Creating TLS certificates for Pulsar involves creating a [certificate
authority](#certificate-authority) (CA), [broker
certificate](#broker-certificate), and [client
certificate](#client-certificate).
-
-##### Certificate authority
-
-The first step is to create the certificate for the CA. The CA will be used to
sign both the broker and client certificates, in order to ensure that each
party will trust the others.
-
-###### Linux
-
-```bash
-$ CA.pl -newca
-```
-
-###### macOS
-
-```bash
-$ /System/Library/OpenSSL/misc/CA.pl -newca
-```
-
-After answering the question prompts, this will store CA-related files in the
`./demoCA` directory. Within that directory:
-
-* `demoCA/cacert.pem` is the public certificate. It is meant to be distributed
to all parties involved.
-* `demoCA/private/cakey.pem` is the private key. This is only needed when
signing a new certificate for either broker or clients and it must be safely
guarded.
-
-##### Broker certificate
-
-Once a CA certificate has been created, you can create certificate requests
and sign them with the CA.
-
-The following commands will ask you a few questions and then create the
certificates. When asked for the common name, you need to match the hostname of
the broker. You could also use a wildcard to match a group of broker hostnames,
for example `*.broker.usw.example.com`. This ensures that the same certificate
can be reused on multiple machines.
-
-```shell
-$ openssl req \
- -newkey rsa:2048 \
- -sha256 \
- -nodes \
- -out broker-cert.csr \
- -outform PEM
-```
-
-Convert the key to [PKCS 8](https://en.wikipedia.org/wiki/PKCS_8) format:
-
-```shell
-$ openssl pkcs8 \
- -topk8 \
- -inform PEM \
- -outform PEM \
- -in privkey.pem \
- -out broker-key.pem \
- -nocrypt
-```
-
-This will create two broker certificate files named `broker-cert.csr` and
`broker-key.pem`. Now you can create the signed certificate:
-
-```shell
-$ openssl ca \
- -out broker-cert.pem \
- -infiles broker-cert.csr
-```
-
-At this point, you should have a `broker-cert.pem` and `broker-key.pem` file.
These will be needed for the broker.
-
-##### Client certificate
-
-To create a client certificate, repeat the steps in the previous section, but
did create `client-cert.pem` and `client-key.pem` files instead.
-
-For the client common name, you need to use a string that you intend to use as
the [role token](#role-tokens) for this client, though it doesn't need to match
the client hostname.
-
-#### Configure the broker for TLS
-
-To configure a Pulsar broker to use TLS authentication, you'll need to make
some changes to the `broker.conf` configuration file, which is located in the
`conf` directory of your [Pulsar installation](getting-started-standalone.md).
-
-Add these values to the configuration file (substituting the appropriate
certificate paths where necessary):
-
-```properties
-# Enable TLS and point the broker to the right certs
-tlsEnabled=true
-tlsCertificateFilePath=/path/to/broker-cert.pem
-tlsKeyFilePath=/path/to/broker-key.pem
-tlsTrustCertsFilePath=/path/to/cacert.pem
-
-# Enable the TLS auth provider
-authenticationEnabled=true
-authorizationEnabled=true
-authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderTls
-```
-
-> A full listing of parameters available in the `conf/broker.conf` file, as
well as the default values for those parameters, can be found in Broker
Configuration.
-
-
-#### Configure the discovery service
-
-The discovery service used by Pulsar brokers needs to redirect all HTTPS
requests, which means that it needs to be trusted by the client as well. Add
this configuration in `conf/discovery.conf` in your Pulsar installation:
-
-```properties
-tlsEnabled=true
-tlsCertificateFilePath=/path/to/broker-cert.pem
-tlsKeyFilePath=/path/to/broker-key.pem
-```
-
-#### Configure clients
-
-For more information on Pulsar client authentication using TLS, see the
following language-specific docs:
-
-* [Java client](client-libraries-java.md)
-* [C++ client](client-libraries-cpp.md)
-
-#### Configure CLI tools
-
-[Command-line tools](reference-cli-tools.md) like
[`pulsar-admin`](reference-pulsar-admin.md),
[`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and
[`pulsar-client`](reference-cli-tools.md#pulsar-client) use the
`conf/client.conf` config file in a Pulsar installation.
-
-You'll need to add the following authentication parameters to that file to use
TLS with Pulsar's CLI tools:
-
-```properties
-serviceUrl=https://broker.example.com:8443/
-authPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls
-authParams=tlsCertFile:/path/to/client-cert.pem,tlsKeyFile:/path/to/client-key.pem
-useTls=true
-tlsAllowInsecureConnection=false
-tlsTrustCertsFilePath=/path/to/cacert.pem
-```
-
-### Athenz
-
-[Athenz](https://github.com/yahoo/athenz) is a role-based
authentication/authorization system. In Pulsar, Athenz [role
tokens](#role-tokens) (aka *z-tokens*) can be used to establish the identify of
the client.
-
-#### Athenz authentication settings
-
-In a [decentralized Athenz
system](https://github.com/yahoo/athenz/blob/master/docs/dev_decentralized_access.md)
there is both an [authori**Z**ation **M**anagement
**S**ystem](https://github.com/yahoo/athenz/blob/master/docs/setup_zms.md)
(ZMS) server and an [authori**Z**ation **T**oken
**S**ystem](https://github.com/yahoo/athenz/blob/master/docs/setup_zts.md)
(ZTS) server.
-
-To begin, you need to set up Athenz service access control. You should create
domains for the *provider* (which provides some resources to other services
with some authentication/authorization policies) and the *tenant* (which is
provisioned to access some resources in a provider). In this case, the provider
corresponds to the Pulsar service itself and the tenant corresponds to each
application using Pulsar (typically, a property in Pulsar).
-
-##### Create the tenant domain and service
-
-On the tenant side, you need to:
-
-1. Create a domain, such as `shopping`
-2. Generate a private/public key pair
-3. Create a service, such as `some_app`, on the domain with the public key
-
-Note that the private key generated in step 2 needs to be specified when the
Pulsar client connects to the broker (see client configuration examples for
[Java](client-libraries-java.md#tls-authentication) and
[C++](client-libraries-cpp.md#tls-authentication)).
-
-For more specific steps involving the Athenz UI, please refer to [this
doc](https://github.com/yahoo/athenz/blob/master/docs/example_service_athenz_setup.md#client-tenant-domain).
-
-##### Create the provider domain and add the tenant service to some role
members
-
-On the provider side, you need to:
-
-1. Create a domain, such as `pulsar`
-2. Create a role
-3. Add the tenant service to members of the role
-
-Note that in step 2 any action and resource can be specified since they are
not used on Pulsar. In other words, Pulsar uses the Athenz role token only for
authentication, *not* for authorization.
-
-For more specific steps involving UI, please refer to [this
doc](https://github.com/yahoo/athenz/blob/master/docs/example_service_athenz_setup.md#server-provider-domain).
-
-#### Configure the broker for Athenz
-
-{% include message.html id="tls_role_tokens" %}
-
-> #### TLS encryption strongly recommended
-> Please note that using TLS encryption is strongly recommended when using
Athenz as an authentication provider, as it can protect role tokens from being
intercepted and reused (see also [this doc]()).
-
-
-In the `conf/broker.conf` configuration file in your Pulsar installation, you
need to provide the class name of the Athenz authentication provider as well as
a comma-separated list of provider domain names.
-
-```properties
-# Add the Athenz auth provider
-authenticationEnabled=true
-authorizationEnabled=true
-authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderAthenz
-athenzDomainNames=pulsar
-
-# Enable TLS
-tlsEnabled=true
-tlsCertificateFilePath=/path/to/broker-cert.pem
-tlsKeyFilePath=/path/to/broker-key.pem
-```
-
-> A full listing of parameters available in the conf/broker.conf file, as well
as the default values for those parameters, can be found in [Broker
Configuration]().
-
-#### Configure clients for Athenz
-
-For more information on Pulsar client authentication using Athenz, see the
following language-specific docs:
-
-* [Java client](client-libraries-java.md#athenz)
-
-#### Configure CLI tools for Athenz
-
-[Command-line tools](reference-cli-tools.md) like
[`pulsar-admin`](reference-pulsar-admin.md),
[`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and
[`pulsar-client`](reference-cli-tools.md#pulsar-client) use the
`conf/client.conf` config file in a Pulsar installation.
-
-You’ll need to add the following authentication parameters to that file to use
Athenz with Pulsar’s CLI tools:
-
-```properties
-# URL for the broker
-serviceUrl=https://broker.example.com:8443/
-
-# Set Athenz auth plugin and its parameters
-authPlugin=org.apache.pulsar.client.impl.auth.AuthenticationAthenz
-authParams={"tenantDomain":"shopping","tenantService":"some_app","providerDomain":"pulsar","privateKey":"file:///path/to/private.pem","keyId":"v1"}
-
-# Enable TLS
-useTls=true
-tlsAllowInsecureConnection=false
-tlsTrustCertsFilePath=/path/to/cacert.pem
-```
-
-## Authorization
-
-In Pulsar, the [authentication provider](#authentication-providers) is charged
with properly identifying clients and associating them with [role
tokens](#role-tokens). *Authorization* is the process that determines *what*
clients are able to do.
-
-Authorization in Pulsar is managed at the tenant level, which means that you
can have multiple authorization schemes active in a single Pulsar instance. You
could, for example, create a `shopping` tenant that has one set of
[roles](#role-tokens) and applies to a shopping application used by your
company, while an `inventory` tenant would be used only by an inventory
application.
-
-> When working with properties, you can specify which of your Pulsar clusters
your tenant is allowed to use. This enables you to also have cluster-level
authorization schemes.
-
-## Creating a new tenant
-
-A Pulsar tenant is typically provisioned by Pulsar {% popover instance %}
administrators or by some kind of self-service portal.
-
-Properties are managed using the [`pulsar-admin`](reference-pulsar-admin.md)
tool. Here's an example property creation command:
-
-```shell
-$ bin/pulsar-admin tenants create my-tenant \
- --admin-roles my-admin-role \
- --allowed-clusters us-west,us-east
-```
-
-This command will create a new property `my-tenant` that will be allowed to
use the clusters `us-west` and `us-east`.
-
-A client that successfully identified itself as having the role
`my-admin-role` would then be allowed to perform all administrative tasks on
this tenant.
-
-The structure of topic names in Pulsar reflects the hierarchy between tenants,
clusters, and [namespaces](#managing-namespaces):
-
-```http
-persistent://tenant/namespace/topic
-```
-
-## Managing permissions
-
-Permissions in Pulsar are managed at the namespace level (that is, within
tenants and clusters).
-
-### Grant permissions
-
-You can grant permissions to specific roles for lists of operations such as
`produce` and `consume`.
-
-#### pulsar-admin
-
-Use the
[`grant-permission`](reference-pulsar-admin.md#namespaces-grant-permission)
subcommand and specify a namespace, actions using the `--actions` flag, and a
role using the `--role` flag:
-
-```shell
-$ pulsar-admin namespaces grant-permission test-tenant/ns1 \
- --actions produce,consume \
- --role admin10
-```
-
-Wildcard authorization can be performed when
`authorizationAllowWildcardsMatching` is set to `true` in `broker.conf`.
-
-e.g.
-```shell
-$ pulsar-admin namespaces grant-permission test-tenant/ns1 \
- --actions produce,consume \
- --role 'my.role.*'
-```
-
-Then, roles `my.role.1`, `my.role.2`, `my.role.foo`, `my.role.bar`, etc. can
produce and consume.
-
-```shell
-$ pulsar-admin namespaces grant-permission test-tenant/ns1 \
- --actions produce,consume \
- --role '*.role.my'
-```
-
-Then, roles `1.role.my`, `2.role.my`, `foo.role.my`, `bar.role.my`, etc. can
produce and consume.
-
-**Note**: A wildcard matching works at **the beginning or end of the role name
only**.
-
-e.g.
-```shell
-$ pulsar-admin namespaces grant-permission test-tenant/ns1 \
- --actions produce,consume \
- --role 'my.*.role'
-```
-
-In this case, only the role `my.*.role` has permissions.
-Roles `my.1.role`, `my.2.role`, `my.foo.role`, `my.bar.role`, etc. **cannot**
produce and consume.
-
-#### REST API
-
-```http
-POST /admin/v2/namespaces/:tenant/:namespace/permissions/:role
-```
-[More
info](reference-rest-api.md#/admin/namespaces/:property/:cluster/:namespace/permissions/:role)
-
-#### Java
-
-```java
-admin.namespaces().grantPermissionOnNamespace(namespace, role,
getAuthActions(actions));
-```
-
-### Get permission
-
-You can see which permissions have been granted to which roles in a namespace.
-
-#### pulsar-admin
-
-Use the [`permissions`](reference-pulsar-admin.md#namespaces-permissions)
subcommand and specify a namespace:
-
-```shell
-$ pulsar-admin namespaces permissions test-tenant/ns1
-{
- "admin10": [
- "produce",
- "consume"
- ]
-}
-```
-
-#### REST API
-
-```http
-GET /admin/v2/namespaces/:tenant/:namespace/permissions
-```
-
-[More
info](reference-rest-api.md#/admin/namespaces/:property/:cluster/:namespace/permissions)
-
-#### Java
-
-```java
-admin.namespaces().getPermissions(namespace);
-```
-
-### Revoke permissions
-
-You can revoke permissions from specific roles, which means that those roles
will no longer have access to the specified namespace.
-
-#### pulsar-admin
-
-Use the [`revoke-permission`](reference-pulsar-admin.md#revoke-permission)
subcommand and specify a namespace and a role using the `--role` flag:
-
-```shell
-$ pulsar-admin namespaces revoke-permission test-tenant/ns1 \
- --role admin10
-```
-
-#### REST API
-
-```http
-DELETE /admin/v2/namespaces/:tenant/:namespace/permissions/:role
-```
-
-[More
info](reference-rest-api.md#/admin/namespaces/:property/:cluster/:namespace/permissions/:role)
-
-#### Java
-
-```java
-admin.namespaces().revokePermissionsOnNamespace(namespace, role);
-```
-
-
-## Superusers
-
-In Pulsar you can assign certain roles to be *superusers* of the system. A
superuser is allowed to perform all administrative tasks on all properties and
namespaces, as well as to publish and subscribe to all topics.
-
-Superusers are configured in the broker configuration file in
[`conf/broker.conf`](reference-configuration.md#broker) configuration file,
using the [`superUserRoles`](reference-configuration.md#broker-superUserRoles)
parameter:
-
-```properties
-superUserRoles=my-super-user-1,my-super-user-2
-```
-
-> A full listing of parameters available in the `conf/broker.conf` file, as
well as the default values for those parameters, can be found in [Broker
Configuration]().
-
-Typically, superuser roles are used for administrators and clients but also
for broker-to-broker authorization. When using
[geo-replication](administration-geo.md), every broker
-needs to be able to publish to other clusters' topics.
-
-## Pulsar admin authentication
-
-```java
-String authPluginClassName = "com.org.MyAuthPluginClass";
-String authParams = "param1:value1";
-boolean useTls = false;
-boolean tlsAllowInsecureConnection = false;
-String tlsTrustCertsFilePath = null;
-
-ClientConfiguration config = new ClientConfiguration();
-config.setAuthentication(authPluginClassName, authParams);
-config.setUseTls(useTls);
-config.setTlsAllowInsecureConnection(tlsAllowInsecureConnection);
-config.setTlsTrustCertsFilePath(tlsTrustCertsFilePath);
-
-PulsarAdmin admin = new PulsarAdmin(url, config);
-```
-
-To use TLS:
-
-```java
-String authPluginClassName = "com.org.MyAuthPluginClass";
-String authParams = "param1:value1";
-boolean useTls = false;
-boolean tlsAllowInsecureConnection = false;
-String tlsTrustCertsFilePath = null;
-
-ClientConfiguration config = new ClientConfiguration();
-config.setAuthentication(authPluginClassName, authParams);
-config.setUseTls(useTls);
-config.setTlsAllowInsecureConnection(tlsAllowInsecureConnection);
-config.setTlsTrustCertsFilePath(tlsTrustCertsFilePath);
-
-PulsarAdmin admin = new PulsarAdmin(url, config);
-```
diff --git a/site2/docs/administration-dashboard.md
b/site2/docs/administration-dashboard.md
index e4e3625f0d..b129091197 100644
--- a/site2/docs/administration-dashboard.md
+++ b/site2/docs/administration-dashboard.md
@@ -35,4 +35,4 @@ Once the Docker container is running, the web dashboard will
be accessible via `
### Known issues
-Pulsar [authentication](administration-auth.md#authentication-providers) is
not supported at this point. The dashboard's data collector does not pass any
authentication-related data and will be denied access if the Pulsar broker
requires authentication.
+Pulsar [authentication](security-overview.md#authentication-providers) is not
supported at this point. The dashboard's data collector does not pass any
authentication-related data and will be denied access if the Pulsar broker
requires authentication.
diff --git a/site2/docs/administration-zk-bk.md
b/site2/docs/administration-zk-bk.md
index 79ee037a0f..54d00eaeee 100644
--- a/site2/docs/administration-zk-bk.md
+++ b/site2/docs/administration-zk-bk.md
@@ -146,9 +146,6 @@ In Pulsar, ZooKeeper configuration is handled by two
separate configuration file
Configuration for local ZooKeeper is handled by the
[`conf/zookeeper.conf`](reference-configuration.md#zookeeper) file. The table
below shows the available parameters:
-{% include config.html id="zookeeper" %}
-
-
|Name|Description|Default|
|---|---|---|
|tickTime| The tick is the basic unit of time in ZooKeeper, measured in
milliseconds and used to regulate things like heartbeats and timeouts. tickTime
is the length of a single tick. |2000|
@@ -174,8 +171,6 @@ BookKeeper is responsible for all durable message storage
in Pulsar. BookKeeper
### Deploying BookKeeper
-{% include explanations/deploying-bk.md %}
-
BookKeeper provides [persistent message
storage](getting-started-concepts-and-architecture.md#persistent-storage) for
Pulsar.
Each Pulsar broker needs to have its own cluster of bookies. The BookKeeper
cluster shares a local ZooKeeper quorum with the Pulsar cluster.
diff --git a/site2/docs/client-libraries-go.md
b/site2/docs/client-libraries-go.md
index 0f5283b896..92a88df336 100644
--- a/site2/docs/client-libraries-go.md
+++ b/site2/docs/client-libraries-go.md
@@ -48,7 +48,7 @@ A URL for a production Pulsar cluster may look something like
this:
pulsar://pulsar.us-west.example.com:6650
```
-If you're using [TLS](administration-auth.md#tls-client-auth) authentication,
the URL will look like something like this:
+If you're using [TLS](security-tls.md) authentication, the URL will look like
something like this:
```http
pulsar+ssl://pulsar.us-west.example.com:6651
@@ -435,7 +435,7 @@ Parameter | Description
## TLS encryption and authentication
-In order to use [TLS encryption](administration-auth.md#), you'll need to
configure your client to do so:
+In order to use [TLS encryption](security-tls.md), you'll need to configure
your client to do so:
* Use `pulsar+ssl` URL type
* Set `TLSTrustCertsFilePath` to the path to the TLS certs used by your
client and the Pulsar broker
diff --git a/site2/docs/client-libraries-java.md
b/site2/docs/client-libraries-java.md
index 9fe37849fb..97a1ef1b76 100644
--- a/site2/docs/client-libraries-java.md
+++ b/site2/docs/client-libraries-java.md
@@ -4,20 +4,20 @@ title: The Pulsar Java client
sidebar_label: Java
---
-The Pulsar Java client can be used both to create Java producers, consumers,
and [readers](#readers) of messages and to perform [administrative
tasks](admin-api-overview.md). The current version of the Java client is
**pulsar:version**.
+The Pulsar Java client can be used both to create Java producers, consumers,
and [readers](#readers) of messages and to perform [administrative
tasks](admin-api-overview.md). The current version of the Java client is
**{{pulsar:version}}**.
Javadoc for the Pulsar client is divided up into two domains, by package:
Package | Description | Maven Artifact
:-------|:------------|:--------------
-[`org.apache.pulsar.client.api`](/api/client) | The producer and consumer API
|
[org.apache.pulsar:pulsar-client:pulsar:version](http://search.maven.org/#artifactdetails%7Corg.apache.pulsar%7Cpulsar-client%7Cpulsar:version%7Cjar)
-[`org.apache.pulsar.client.admin`](/api/admin) | The Java [admin
API](admin-api-overview.md) |
[org.apache.pulsar:pulsar-client-admin:pulsar:version](http://search.maven.org/#artifactdetails%7Corg.apache.pulsar%7Cpulsar-client-admin%7Cpulsar:version%7Cjar)
+[`org.apache.pulsar.client.api`](/api/client) | The producer and consumer API
|
[org.apache.pulsar:pulsar-client:{{pulsar:version}}](http://search.maven.org/#artifactdetails%7Corg.apache.pulsar%7Cpulsar-client%7C{{pulsar:version}}%7Cjar)
+[`org.apache.pulsar.client.admin`](/api/admin) | The Java [admin
API](admin-api-overview.md) |
[org.apache.pulsar:pulsar-client-admin:{{pulsar:version}}](http://search.maven.org/#artifactdetails%7Corg.apache.pulsar%7Cpulsar-client-admin%7C{{pulsar:version}}%7Cjar)
This document will focus only on the client API for producing and consuming
messages on Pulsar topics. For a guide to using the Java admin client, see [The
Pulsar admin interface](admin-api-overview.md).
## Installation
-The latest version of the Pulsar Java client library is available via [Maven
Central](http://search.maven.org/#artifactdetails%7Corg.apache.pulsar%7Cpulsar-client%7Cpulsar:version%7Cjar).
To use the latest version, add the `pulsar-client` library to your build
configuration.
+The latest version of the Pulsar Java client library is available via [Maven
Central](http://search.maven.org/#artifactdetails%7Corg.apache.pulsar%7Cpulsar-client%7C{{pulsar:version}}%7Cjar).
To use the latest version, add the `pulsar-client` library to your build
configuration.
### Maven
@@ -25,7 +25,7 @@ If you're using Maven, add this to your `pom.xml`:
```xml
<!-- in your <properties> block -->
-<pulsar.version>pulsar:version</pulsar.version>
+<pulsar.version>{{pulsar:version}}</pulsar.version>
<!-- in your <dependencies> block -->
<dependency>
@@ -40,7 +40,7 @@ If you're using Maven, add this to your `pom.xml`:
If you're using Gradle, add this to your `build.gradle` file:
```groovy
-def pulsarVersion = 'pulsar:version'
+def pulsarVersion = '{{pulsar:version}}'
dependencies {
compile group: 'org.apache.pulsar', name: 'pulsar-client', version:
pulsarVersion
@@ -63,7 +63,7 @@ A URL for a production Pulsar cluster may look something like
this:
pulsar://pulsar.us-west.example.com:6650
```
-If you're using [TLS](administration-auth.md#tls-client-auth) authentication,
the URL will look like something like this:
+If you're using [TLS](security-tls.md) authentication, the URL will look like
something like this:
```http
pulsar+ssl://pulsar.us-west.example.com:6651
@@ -407,11 +407,11 @@ The following schema formats are currently available for
Java:
## Authentication
-Pulsar currently supports two authentication schemes:
[TLS](administration-auth.md#tls-client-auth) and
[Athenz](administration-auth.md#athenz). The Pulsar Java client can be used
with both.
+Pulsar currently supports two authentication schemes: [TLS](security-tls.md)
and [Athenz](security-athenz.md). The Pulsar Java client can be used with both.
### TLS Authentication
-To use [TLS](administration-auth.md#tls-client-auth), you need to set TLS to
`true` using the `setUseTls` method, point your Pulsar client to a TLS cert
path, and provide paths to cert and key files.
+To use [TLS](security-tls.md), you need to set TLS to `true` using the
`setUseTls` method, point your Pulsar client to a TLS cert path, and provide
paths to cert and key files.
Here's an example configuration:
@@ -433,7 +433,7 @@ PulsarClient client = PulsarClient.builder()
### Athenz
-To use [Athenz](administration-auth.md#athenz) as an authentication provider,
you need to [use TLS](#tls-authentication) and provide values for four
parameters in a hash:
+To use [Athenz](security-athenz.md) as an authentication provider, you need to
[use TLS](#tls-authentication) and provide values for four parameters in a hash:
* `tenantDomain`
* `tenantService`
diff --git a/site2/docs/client-libraries-websocket.md
b/site2/docs/client-libraries-websocket.md
index c71dfca56e..64884c735b 100644
--- a/site2/docs/client-libraries-websocket.md
+++ b/site2/docs/client-libraries-websocket.md
@@ -250,7 +250,7 @@ Error Code | Error Message
7 | Invalid payload encoding
8 | Unknown error
-{% include admonition.html type='warning' content='The application is
responsible for re-establishing a new WebSocket session after a backoff
period.' %}
+> The application is responsible for re-establishing a new WebSocket session
after a backoff period.
## Client examples
diff --git a/site2/docs/cookbooks-partitioned.md
b/site2/docs/cookbooks-partitioned.md
index 34ecb3e103..61db75f412 100644
--- a/site2/docs/cookbooks-partitioned.md
+++ b/site2/docs/cookbooks-partitioned.md
@@ -4,7 +4,7 @@ title: Non-persistent messaging
sidebar_label: Partitioned Topics
---
-By default, Pulsar topics are served by a single broker. Using only a single
broker, however, limits a topic's maximum throughput. *Partitioned topics* are
a special type of topic that can span multiple brokers and thus allow for much
higher throughput. For an explanation of how partitioned topics work, see the
[Concepts](#concepts) section below.
+By default, Pulsar topics are served by a single broker. Using only a single
broker, however, limits a topic's maximum throughput. *Partitioned topics* are
a special type of topic that can span multiple brokers and thus allow for much
higher throughput. For an explanation of how partitioned topics work, see the
[Partitioned
Topics](getting-started-concepts-and-architecture.md#partitioned-topics)
concepts.
You can [publish](#publishing-to-partitioned-topics) to partitioned topics
using Pulsar's client libraries and you can [create and
manage](#managing-partitioned-topics) partitioned topics using Pulsar's [admin
API](admin-api-overview.md).
@@ -70,16 +70,6 @@ Producer producer = client.createProducer(topic, config);
producer.send("Partitioned topic message".getBytes());
```
-
-## Pulsar admin setup
-
-{% include explanations/admin-setup.md %}
-
## Managing partitioned topics
-{% include explanations/partitioned-topic-admin.md %}
-
-## Concepts
-
-{% include explanations/partitioned-topics.md %}
-
+You can use Pulsar's [admin API](admin-api-overview.md) to create and manage
[partitioned topics](admin-api-partitioned-topics.md).
diff --git a/site2/docs/cookbooks-tiered-storage.md
b/site2/docs/cookbooks-tiered-storage.md
index 003cd14a7e..74a17ce33a 100644
--- a/site2/docs/cookbooks-tiered-storage.md
+++ b/site2/docs/cookbooks-tiered-storage.md
@@ -16,8 +16,6 @@ A topic in Pulsar is backed by a log, known as a managed
ledger. This log is com
-{% include figure.html src="/img/pulsar-tiered-storage.png" alt="Tiered
Storage" width="80" %}
-
The Tiered Storage offloading mechanism takes advantage of this segment
oriented architecture. When offloading is requested, the segments of the log
are copied, one-by-one, to tiered storage. All segments of the log, apart from
the segment currently being written to can be offloaded.
## Amazon S3
diff --git a/site2/docs/deploy-bare-metal-multi-cluster.md
b/site2/docs/deploy-bare-metal-multi-cluster.md
index 177740ccb6..d20047efc7 100644
--- a/site2/docs/deploy-bare-metal-multi-cluster.md
+++ b/site2/docs/deploy-bare-metal-multi-cluster.md
@@ -27,21 +27,21 @@ To get started running Pulsar, download a binary tarball
release in one of the f
* by clicking the link below and downloading the release from an Apache mirror:
- * <a href="pulsar:binary_release_url" download>Pulsar pulsar:version binary
release</a>
+ * <a href="pulsar:binary_release_url" download>Pulsar {{pulsar:version}}
binary release</a>
* from the Pulsar [downloads page](pulsar:download_page_url)
* from the Pulsar [releases
page](https://github.com/apache/incubator-pulsar/releases/latest)
* using [wget](https://www.gnu.org/software/wget):
```shell
- $ wget
'https://www.apache.org/dyn/mirrors/mirrors.cgi?action=download&filename=incubator/pulsar/pulsar-pulsar:version/apache-pulsar-pulsar:version-bin.tar.gz'
-O apache-pulsar-pulsar:version-bin.tar.gz
+ $ wget
'https://www.apache.org/dyn/mirrors/mirrors.cgi?action=download&filename=incubator/pulsar/pulsar-{{pulsar:version}}/apache-pulsar-{{pulsar:version}}-bin.tar.gz'
-O apache-pulsar-{{pulsar:version}}-bin.tar.gz
```
Once the tarball is downloaded, untar it and `cd` into the resulting directory:
```bash
-$ tar xvfz apache-pulsar-pulsar:version-bin.tar.gz
-$ cd apache-pulsar-pulsar:version
+$ tar xvfz apache-pulsar-{{pulsar:version}}-bin.tar.gz
+$ cd apache-pulsar-{{pulsar:version}}
```
## What your package contains
@@ -211,7 +211,7 @@ As you can see from the example above, the following needs
to be specified:
* The web service URL for the cluster
* A broker service URL enabling interaction with the {% popover brokers %} in
the cluster
-If you're using [TLS](administration-auth.md#tls-client-auth), you'll also
need to specify a TLS web service URL for the cluster as well as a TLS broker
service URL for the brokers in the cluster.
+If you're using [TLS](security-tls.md), you'll also need to specify a TLS web
service URL for the cluster as well as a TLS broker service URL for the brokers
in the cluster.
Make sure to run `initialize-cluster-metadata` for each cluster in your
instance.
diff --git a/site2/docs/deploy-bare-metal.md b/site2/docs/deploy-bare-metal.md
index 8b3497fc7b..792aa39dcc 100644
--- a/site2/docs/deploy-bare-metal.md
+++ b/site2/docs/deploy-bare-metal.md
@@ -130,9 +130,9 @@ Flag | Description
`--zookeeper` | A "local" ZooKeeper connection string for the cluster. This
connection string only needs to include *one* machine in the ZooKeeper cluster.
`--configuration-store` | The configuration store connection string for the
entire instance. As with the `--zookeeper` flag, this connection string only
needs to include *one* machine in the ZooKeeper cluster.
`--web-service-url` | The web service URL for the cluster, plus a port. This
URL should be a standard DNS name. The default port is 8080 (we don't recommend
using a different port).
-`--web-service-url-tls` | If you're using
[TLS](administration-auth.md#tls-client-auth), you'll also need to specify a
TLS web service URL for the cluster. The default port is 8443 (we don't
recommend using a different port).
+`--web-service-url-tls` | If you're using [TLS](security-tls.md), you'll also
need to specify a TLS web service URL for the cluster. The default port is 8443
(we don't recommend using a different port).
`--broker-service-url` | A broker service URL enabling interaction with the
brokers in the cluster. This URL should use the same DNS name as the web
service URL but should use the `pulsar` scheme instead. The default port is
6650 (we don't recommend using a different port).
-`--broker-service-url-tls` | If you're using
[TLS](administration-auth.md#tls-client-auth), you'll also need to specify a
TLS web service URL for the cluster as well as a TLS broker service URL for the
brokers in the cluster. The default port is 6651 (we don't recommend using a
different port).
+`--broker-service-url-tls` | If you're using [TLS](security-tls.md), you'll
also need to specify a TLS web service URL for the cluster as well as a TLS
broker service URL for the brokers in the cluster. The default port is 6651 (we
don't recommend using a different port).
## Deploying a BookKeeper cluster
diff --git a/site2/docs/deploy-kubernetes.md b/site2/docs/deploy-kubernetes.md
index 75f2d920c9..1bdd1b4162 100644
--- a/site2/docs/deploy-kubernetes.md
+++ b/site2/docs/deploy-kubernetes.md
@@ -199,7 +199,7 @@ $ kubectl get pods -w -l app=pulsar
Once all of the components are up and running, you'll need to create at least
one Pulsar tenant and at least one namespace.
-> This step is not strictly required if Pulsar [authentication and
authorization](administration-auth.md) is turned on, though it allows you to
change [policies](admin-api-namespaces.md) for each of the namespaces later.
+> This step is not strictly required if Pulsar [authentication and
authorization](security-overview.md) is turned on, though it allows you to
change [policies](admin-api-namespaces.md) for each of the namespaces later.
You can create properties and namespaces (and perform any other administrative
tasks) using the `pulsar-admin` pod that is already configured to act as an
admin client for your newly created Pulsar cluster. One easy way to perform
administrative tasks is to create an alias for the
[`pulsar-admin`](reference-pulsar-admin.md) tool installed on the admin pod.
diff --git a/site2/docs/getting-started-clients.md
b/site2/docs/getting-started-clients.md
index efb67c588a..8a924096c0 100644
--- a/site2/docs/getting-started-clients.md
+++ b/site2/docs/getting-started-clients.md
@@ -4,7 +4,7 @@ title: Pulsar client libraries
sidebar_label: Client libraries
---
-Pulsar currently has client libraries available for three languages:
+Pulsar currently has client libraries available for following languages:
* [Java](#java-client)
* [Go](#go-client)
diff --git a/site2/docs/getting-started-concepts-and-architecture.md
b/site2/docs/getting-started-concepts-and-architecture.md
index e290753038..14eb69f16d 100644
--- a/site2/docs/getting-started-concepts-and-architecture.md
+++ b/site2/docs/getting-started-concepts-and-architecture.md
@@ -22,7 +22,7 @@ Messages are the basic "unit" of Pulsar. They're what
producers publish to topic
Component | Purpose
:---------|:-------
Value / data payload | The data carried by the message. All Pulsar messages
carry raw bytes, although message data can also conform to data
[schemas](#schema-registry)
-Key | Messages can optionally be tagged with keys, which can be useful for
things like [topic compaction](#compaction)
+Key | Messages can optionally be tagged with keys, which can be useful for
things like [topic compaction](#topic-compaction)
Properties | An optional key/value map of user-defined properties
Producer name | The name of the producer that produced the message (producers
are automatically given default names, but you can apply your own explicitly as
well)
Sequence ID | Each Pulsar message belongs to an ordered sequence on its topic.
A message's sequence ID is its ordering in that sequence.
@@ -100,12 +100,12 @@ Topic name component | Description
:--------------------|:-----------
`persistent` / `non-persistent` | This identifies the type of topic. Pulsar
supports two kind of topics: [persistent](#persistent-storage) and
[non-persistent](#non-persistent-topics) (persistent is the default, so if you
don't specify a type the topic will be persistent). With persistent topics, all
messages are durably [persisted](#persistent-storage) on disk (that means on
multiple disks unless the broker is standalone), whereas data for
[non-persistent](#non-persistent-topics) topics isn't persisted to storage
disks.
`tenant` | The topic's tenant within the instance. Tenants are
essential to multi-tenancy in Pulsar and can be spread across clusters.
-`namespace` | The administrative unit of the topic, which acts as a
grouping mechanism for related topics. Most topic configuration is performed at
the [namespace](#namespace) level. Each tenant can have multiple namespaces.
+`namespace` | The administrative unit of the topic, which acts as a
grouping mechanism for related topics. Most topic configuration is performed at
the [namespace](#namespaces) level. Each tenant can have multiple namespaces.
`topic` | The final part of the name. Topic names are freeform
and have no special meaning in a Pulsar instance.
> #### No need to explicitly create new topics
-> You don't need to explicitly create topics in Pulsar. If a client attempts
to write or receive messages to/from a topic that does not yet exist, Pulsar
will automatically create that topic under the [namespace](#namespace) provided
in the [topic name](#topics).
+> You don't need to explicitly create topics in Pulsar. If a client attempts
to write or receive messages to/from a topic that does not yet exist, Pulsar
will automatically create that topic under the [namespace](#namespaces)
provided in the [topic name](#topics).
### Namespaces
@@ -243,7 +243,7 @@ In non-persistent topics, brokers immediately deliver
messages to all connected
> With non-persistent topics, message data lives only in memory. If a message
> broker fails or message data can otherwise not be retrieved from memory,
> your message data may be lost. Use non-persistent topics only if you're
> *certain* that your use case requires it and can sustain it.
-By default, non-persistent topics are enabled on Pulsar brokers. You can
disable them in the broker's
[configuration](reference-configuration.md#broker-enableNonPersistentTopics).
You can manage non-persistent topics using the [`pulsar-admin
non-persistent`](reference-cli-tools.md#pulsar-admin-non-persistent) interface.
+By default, non-persistent topics are enabled on Pulsar brokers. You can
disable them in the broker's
[configuration](reference-configuration.md#broker-enableNonPersistentTopics).
You can manage non-persistent topics using the [`pulsar-admin
topics`](referencereference--pulsar-admin/#topics-1) interface.
#### Performance
@@ -253,7 +253,7 @@ Non-persistent messaging is usually faster than persistent
messaging because bro
Producers and consumers can connect to non-persistent topics in the same way
as persistent topics, with the crucial difference that the topic name must
start with `non-persistent`. All three subscription
modes---[exclusive](#exclusive), [shared](#shared), and
[failover](#failover)---are supported for non-persistent topics.
-Here's an example [Java consumer](client-libraries-java.md#consumer) for a
non-persistent topic:
+Here's an example [Java consumer](client-libraries-java.md#consumers) for a
non-persistent topic:
```java
PulsarClient client = PulsarClient.create("pulsar://localhost:6650");
@@ -271,7 +271,7 @@ Producer producer = client.createProducer(npTopic);
## Architecture overview
-At the highest level, a Pulsar instance is composed of one or more Pulsar
clusters. Clusters within an instance can [replicate](#replicate) data amongst
themselves.
+At the highest level, a Pulsar instance is composed of one or more Pulsar
clusters. Clusters within an instance can [replicate](#replication) data
amongst themselves.
In a Pulsar cluster:
@@ -285,45 +285,45 @@ The diagram below provides an illustration of a Pulsar
cluster:
At the broader instance level, an instance-wide ZooKeeper cluster called the
configuration store handles coordination tasks involving multiple clusters, for
example [geo-replication](#replication).
-## Brokers
+### Brokers
The Pulsar message broker is a stateless component that's primarily
responsible for running two other components:
* An HTTP server that exposes a REST API for both [administrative
tasks](reference-rest-api.md) and [topic lookup](#client-setup-phase) for
producers and consumers
* A dispatcher, which is an asynchronous TCP server over a custom [binary
protocol](developing-binary-protocol.md) used for all data transfers
-Messages are typically dispatched out of a [managed ledger](#managed-ledger)
cache for the sake of performance, *unless* the backlog exceeds the cache size.
If the backlog grows too large for the cache, the broker will start reading
entries from BookKeeper.
+Messages are typically dispatched out of a [managed ledger](#managed-ledgers)
cache for the sake of performance, *unless* the backlog exceeds the cache size.
If the backlog grows too large for the cache, the broker will start reading
entries from BookKeeper.
Finally, to support geo-replication on global topics, the broker manages
replicators that tail the entries published in the local region and republish
them to the remote region using the Pulsar [Java client
library](client-libraries-java.md).
> For a guide to managing Pulsar brokers, see the
> [brokers](admin-api-brokers.md) guide.
-## Clusters
+### Clusters
A Pulsar instance consists of one or more Pulsar *clusters*. Clusters, in
turn, consist of:
-* One or more Pulsar [brokers](#broker)
+* One or more Pulsar [brokers](#brokers)
* A ZooKeeper quorum used for cluster-level configuration and coordination
* An ensemble of bookies used for [persistent storage](#persistent-storage) of
messages
-Clusters can replicate amongst themselves using
[geo-replication](#geo-replication).
+Clusters can replicate amongst themselves using
[geo-replication](#replication).
-> For a guide to managing Pulsar clusters, see the [Clusters and
brokers](admin-api-brokers#managing-clusters) guide.
+> For a guide to managing Pulsar clusters, see the
[clusters](admin-api-clusters.md) guide.
-## Metadata store
+### Metadata store
Pulsar uses [Apache Zookeeper](https://zookeeper.apache.org/) for metadata
storage, cluster configuration, and coordination. In a Pulsar instance:
* A configuration store quorum stores configuration for tenants, namespaces,
and other entities that need to be globally consistent.
* Each cluster has its own local ZooKeeper ensemble that stores
cluster-specific configuration and coordination such as ownership metadata,
broker load reports, BookKeeper ledger metadata, and more.
-## Persistent storage
+### Persistent storage
Pulsar provides guaranteed message delivery for applications. If a message
successfully reaches a Pulsar broker, it will be delivered to its intended
target.
This guarantee requires that non-acknowledged messages are stored in a durable
manner until they can be delivered to and acknowledged by consumers. This mode
of messaging is commonly called *persistent messaging*. In Pulsar, N copies of
all messages are stored and synced on disk, for example 4 copies across two
servers with mirrored [RAID](https://en.wikipedia.org/wiki/RAID) volumes on
each server.
-### Apache BookKeeper
+#### Apache BookKeeper
Pulsar uses a system called [Apache BookKeeper](http://bookkeeper.apache.org/)
for persistent message storage. BookKeeper is a distributed [write-ahead
log](https://en.wikipedia.org/wiki/Write-ahead_logging) (WAL) system that
provides a number of crucial advantages for Pulsar:
@@ -350,7 +350,7 @@ You can see an illustration of how brokers and bookies
interact in the diagram b
-### Ledgers
+#### Ledgers
A ledger is an append-only data structure with a single writer that is
assigned to multiple BookKeeper storage nodes, or bookies. Ledger entries are
replicated to multiple bookies. Ledgers themselves have very simple semantics:
@@ -358,11 +358,11 @@ A ledger is an append-only data structure with a single
writer that is assigned
* After the ledger has been closed---either explicitly or because the writer
process crashed---it can then be opened only in read-only mode.
* Finally, when entries in the ledger are no longer needed, the whole ledger
can be deleted from the system (across all bookies).
-#### Ledger read consistency
+##### Ledger read consistency
The main strength of Bookkeeper is that it guarantees read consistency in
ledgers in the presence of failures. Since the ledger can only be written to by
a single process, that process is free to append entries very efficiently,
without need to obtain consensus. After a failure, the ledger will go through a
recovery process that will finalize the state of the ledger and establish which
entry was last committed to the log. After that point, all readers of the
ledger are guaranteed to see the exact same content.
-#### Managed ledgers
+##### Managed ledgers
Given that Bookkeeper ledgers provide a single log abstraction, a library was
developed on top of the ledger called the *managed ledger* that represents the
storage layer for a single topic. A managed ledger represents the abstraction
of a stream of messages with a single writer that keeps appending at the end of
the stream and multiple cursors that are consuming the stream, each with its
own associated position.
@@ -371,14 +371,10 @@ Internally, a single managed ledger uses multiple
BookKeeper ledgers to store th
1. After a failure, a ledger is no longer writable and a new one needs to be
created.
2. A ledger can be deleted when all cursors have consumed the messages it
contains. This allows for periodic rollover of ledgers.
-### Journal storage
+#### Journal storage
In BookKeeper, *journal* files contain BookKeeper transaction logs. Before
making an update to a [ledger](#ledgers), a bookie needs to ensure that a
transaction describing the update is written to persistent (non-volatile)
storage. A new journal file is created once the bookie starts or the older
journal file reaches the journal file size threshold (configured using the
[`journalMaxSizeMB`](reference-configuration.md#bookkeeper-journalMaxSizeMB)
parameter).
-### Non-persistent storage
-
-A future version of BookKeeper will support *non-persistent messaging* and
thus multiple durability modes at the topic level. This will enable you to set
the durability mode at the topic level, replacing the `persistent` in topic
names with a `non-persistent` indicator.
-
## Message retention and expiry
By default, Pulsar message brokers:
@@ -401,10 +397,6 @@ With message retention, shown at the top, a <span
style="color: #89b557;">retent
With message expiry, shown at the bottom, some messages are <span
style="color: #bb3b3e;">deleted</span>, even though they <span style="color:
#337db6;">haven't been acknowledged</span>, because they've expired according
to the <span style="color: #e39441;">TTL applied to the namespace</span> (for
example because a TTL of 5 minutes has been applied and the messages haven't
been acknowledged but are 10 minutes old).
-## Pulsar Functions
-
-For an in-depth look at Pulsar Functions, see the [Pulsar Functions
overview](functions-overview.md).
-
## Replication
Pulsar enables messages to be produced and consumed in different
geo-locations. For instance, your application may be publishing data in one
region or market and you would like to process it for consumption in other
regions or markets. [Geo-replication](administration-geo.md) in Pulsar enables
you to do that.
@@ -438,7 +430,7 @@ Message deduplication makes Pulsar an ideal messaging
system to be used in conju
## Multi-tenancy
-Pulsar was created from the ground up as a multi-tenant system. To support
multi-tenancy, Pulsar has a concept of tenants. Tenants can be spread across
clusters and can each have their own [authentication and
authorization](administration-auth.md) scheme applied to them. They are also
the administrative unit at which storage quotas, [message
TTL](cookbooks-retention-expiry.md#time-to-live-ttl), and isolation policies
can be managed.
+Pulsar was created from the ground up as a multi-tenant system. To support
multi-tenancy, Pulsar has a concept of tenants. Tenants can be spread across
clusters and can each have their own [authentication and
authorization](security-overview.md) scheme applied to them. They are also the
administrative unit at which storage quotas, [message
TTL](cookbooks-retention-expiry.md#time-to-live-ttl), and isolation policies
can be managed.
The multi-tenant nature of Pulsar is reflected mostly visibly in topic URLs,
which have this structure:
@@ -448,17 +440,11 @@ persistent://tenant/namespace/topic
As you can see, the tenant is the most basic unit of categorization for topics
(more fundamental than the namespace and topic name).
-### Tenants and namespaces
-
-{% include explanations/tenants-namespaces.md %}
-
-Pulsar was designed from the ground up to be a multi-tenant system. In Pulsar,
tenants are the highest administrative unit within a Pulsar instance.
-
### Tenants
-To each property in a Pulsar instance you can assign:
+To each tenant in a Pulsar instance you can assign:
-* An [authorization](administration-auth.md#authorization) scheme
+* An [authorization](security-authorization.md) scheme
* The set of {% popover clusters %} to which the tenant's configuration applies
### Namespaces
@@ -481,7 +467,7 @@ persistent://tenant/app1/topic-3
## Authentication and Authorization
-Pulsar supports a pluggable [authentication](administration-auth.md) mechanism
which can be configured at broker and it also supports authorization to
identify client and its access rights on topics and tenants.
+Pulsar supports a pluggable [authentication](security-overview.md) mechanism
which can be configured at broker and it also supports authorization to
identify client and its access rights on topics and tenants.
## Client interface
@@ -525,7 +511,7 @@ $ bin/pulsar proxy \
Some important things to know about the Pulsar proxy:
* Connecting clients don't need to provide *any* specific configuration to use
the Pulsar proxy. You won't need to update the client configuration for
existing applications beyond updating the IP used for the service URL (for
example if you're running a load balancer over the Pulsar proxy).
-* [TLS encryption and authentication](administration-auth.md#tls-client-auth)
is supported by the Pulsar proxy
+* [TLS encryption and authentication](security-tls.md) is supported by the
Pulsar proxy
## Service discovery
@@ -647,43 +633,6 @@ Pulsar currently supports S3 as a long term store.
Offloading to S3 triggered vi
> For a guide for setting up tiered storage, see the [Tiered storage
> cookbook](cookbooks-tiered-storage.md).
-
-## Pulsar IO
-
-Messaging systems are most powerful when you can easily use them in
conjunction with external systems like databases and other messaging systems.
**Pulsar IO** is a feature of Pulsar that enables you to easily create, deploy,
and manage Pulsar **connectors** that interact with external systems, such as
[Apache Cassandra](https://cassandra.apache.org),
[Aerospike](https://www.aerospike.com), and many others.
-
-> #### Pulsar IO and Pulsar Functions
-> Under the hood, Pulsar IO connectors are specialized [Pulsar
Functions](#pulsar-functions) purpose-built to interface with external systems.
The [administrative interface](io-overview.md) for Pulsar IO is, in fact, quite
similar to that of Pulsar Functions.
-
-### Sources and sinks
-
-Pulsar IO connectors come in two types:
-
-* **Sources** feed data *into* Pulsar from other systems. Common sources
include other messaging systems and "firehose"-style data pipeline APIs.
-* **Sinks** are fed data *from* Pulsar. Common sinks include other messaging
systems and SQL and NoSQL databases.
-
-This diagram illustrates the relationship between sources, sinks, and Pulsar:
-
-
-
-### Working with connectors
-
-Pulsar IO connectors can be managed via the
[`pulsar-admin`](reference-pulsar-admin.md) CLI tool, in particular the
[`source`](reference-pulsar-admin.md#source) and
[`sink`](reference-pulsar-admin.md#cooks) commands.
-
-> For a guide to managing connectors in your Pulsar installation, see the
[Pulsar IO cookbook](io-overview.md#managing-connectors).
-
-The following connectors are currently available for Pulsar:
-
-|Name|Java Class|
-|---|---|
-|[Aerospike
sink](https://www.aerospike.com/)|[`org.apache.pulsar.io.aerospike.AerospikeSink`](https://github.com/apache/incubator-pulsar/blob/master/pulsar-io/aerospike/src/main/java/org/apache/pulsar/io/aerospike/AerospikeAbstractSink.java)|
-|[Cassandra
sink](https://cassandra.apache.org)|[`org.apache.pulsar.io.cassandra.CassandraSink`](https://github.com/apache/incubator-pulsar/blob/master/pulsar-io/cassandra/src/main/java/org/apache/pulsar/io/cassandra/CassandraAbstractSink.java)|
-|[Kafka
source](https://kafka.apache.org)|[`org.apache.pulsar.io.kafka.KafkaSource`](https://github.com/apache/incubator-pulsar/blob/master/pulsar-io/kafka/src/main/java/org/apache/pulsar/io/kafka/KafkaAbstractSource.java)|
-|[Kafka
sink](https://kafka.apache.org)|[`org.apache.pulsar.io.kafka.KafkaSink`](https://github.com/apache/incubator-pulsar/blob/master/pulsar-io/kafka/src/main/java/org/apache/pulsar/io/kafka/KafkaAbstractSink.java)|
-|[RabbitMQ
source](https://www.rabbitmq.com)|[`org.apache.pulsar.io.rabbitmq.RabbitMQSource`](https://github.com/apache/incubator-pulsar/blob/master/pulsar-io/rabbitmq/src/main/java/org/apache/pulsar/io/rabbitmq/RabbitMQSource.java)|
-|[Twitter Firehose
source](https://developer.twitter.com/en/docs)|[org.apache.pulsar.io.twitter.TwitterFireHose](https://github.com/apache/incubator-pulsar/blob/master/pulsar-io/twitter/src/main/java/org/apache/pulsar/io/twitter/TwitterFireHose.java)|
-
-
## Schema registry
Type safety is extremely important in any application built around a message
bus like Pulsar. Producers and consumers need some kind of mechanism for
coordinating types at the topic level lest a wide variety of potential problems
arise (for example serialization and deserialization issues). Applications
typically adopt one of two basic approaches to type safety in messaging:
diff --git a/site2/docs/getting-started-docker.md
b/site2/docs/getting-started-docker.md
index e53b9095ed..f1e7483986 100644
--- a/site2/docs/getting-started-docker.md
+++ b/site2/docs/getting-started-docker.md
@@ -17,7 +17,7 @@ $ docker run -it \
-p 6650:6650 \
-p 8080:8080 \
-v $PWD/data:/pulsar/data \
- apachepulsar/pulsar:{{site.current_version}} \
+ apachepulsar/pulsar:{{pulsar:version}} \
bin/pulsar standalone
```
@@ -38,7 +38,7 @@ If Pulsar has been successfully started, you should see
`INFO`-level log message
> #### Automatically created namespace
> When you start a local standalone cluster, Pulsar will automatically create
> a `public/default`
namespace that you can use for development purposes. All Pulsar topics are
managed within namespaces.
-For more info, see
[Topics](getting-started-concepts-and-architecture.md#Topics).
+For more info, see
[Topics](getting-started-concepts-and-architecture.md#topics).
## Start publishing and consuming messages
diff --git a/site2/docs/getting-started-standalone.md
b/site2/docs/getting-started-standalone.md
index 523ae18fbc..99bd923a16 100644
--- a/site2/docs/getting-started-standalone.md
+++ b/site2/docs/getting-started-standalone.md
@@ -77,7 +77,7 @@ If Pulsar has been successfully started, you should see
`INFO`-level log message
```
> #### Automatically created namespace
-> When you start a local standalone cluster, Pulsar will automatically create
a `public/default`
[namespace](getting-started-concepts-and-architecture.md#namespace) that you
can use for development purposes. All Pulsar topics are managed within
namespaces. For more info, see
[Topics](getting-started-concepts-and-architecture.md#topics).
+> When you start a local standalone cluster, Pulsar will automatically create
a `public/default`
[namespace](getting-started-concepts-and-architecture.md#namespaces) that you
can use for development purposes. All Pulsar topics are managed within
namespaces. For more info, see
[Topics](getting-started-concepts-and-architecture.md#topics).
## Testing your cluster setup
diff --git a/site2/docs/reference-auth.md b/site2/docs/reference-auth.md
index 08364924af..cc7e8fc42c 100644
--- a/site2/docs/reference-auth.md
+++ b/site2/docs/reference-auth.md
@@ -9,7 +9,7 @@ Pulsar provides a way to use custom authentication and
authorization mechanisms
## Authentication
Pulsar supports mutual TLS and Athenz authentication plugins, and these can be
used as described
-[here](administration-auth.md).
+[here](security-overview.md).
It is possible to use a custom authentication mechanism by providing the
implementation in the
form of two plugins one for the Client library and the other for the Pulsar
Broker to validate
diff --git a/site2/docs/security-athenz.md b/site2/docs/security-athenz.md
index c349cae1bb..27b84a31f7 100644
--- a/site2/docs/security-athenz.md
+++ b/site2/docs/security-athenz.md
@@ -38,7 +38,10 @@ For more specific steps involving UI, please refer to [this
doc](https://github.
## Configure the broker for Athenz
-{% include message.html id="tls_role_tokens" %}
+> ### TLS encryption strongly recommended
+>
+> Please note that using TLS encryption is strongly recommended when using
Athenz as an authentication provider,
+> as it can protect role tokens from being intercepted and reused (see also
[this doc](https://github.com/yahoo/athenz/blob/master/docs/data_model.md)).
In the `conf/broker.conf` configuration file in your Pulsar installation, you
need to provide the class name of the Athenz authentication provider as well as
a comma-separated list of provider domain names.
@@ -55,7 +58,8 @@ tlsCertificateFilePath=/path/to/broker-cert.pem
tlsKeyFilePath=/path/to/broker-key.pem
```
-{% include message.html id="broker_conf_doc" %}
+> A full listing of parameters available in the `conf/broker.conf` file, as
well as the default
+> values for those parameters, can be found in [Broker
Configuration](reference-configuration.md#broker).
## Configure clients for Athenz
diff --git a/site2/docs/security-authorization.md
b/site2/docs/security-authorization.md
index 4b879a08d0..8fa50a912f 100644
--- a/site2/docs/security-authorization.md
+++ b/site2/docs/security-authorization.md
@@ -4,14 +4,15 @@ title: Authentication and authorization in Pulsar
sidebar_label: Authorization and ACLs
---
-n Pulsar, the [authentication
provider](security-overview.md#authentication-providers) is charged with
properly identifying clients and
+In Pulsar, the [authentication
provider](security-overview.md#authentication-providers) is charged with
properly identifying clients and
associating them with [role tokens](security-overview.md#role-tokens).
*Authorization* is the process that determines *what* clients are able to do.
Authorization in Pulsar is managed at the {% popover tenant %} level, which
means that you can have multiple authorization schemes active
in a single Pulsar instance. You could, for example, create a `shopping`
tenant that has one set of [roles](security-overview.md#role-tokens)
and applies to a shopping application used by your company, while an
`inventory` tenant would be used only by an inventory application.
-{% include message.html id="properties_multiple_clusters" %}
+> When working with properties, you can specify which of your Pulsar clusters
your property is allowed to use.
+> This enables you to also have cluster-level authorization schemes.
## Creating a new tenant
@@ -31,7 +32,9 @@ A client that successfully identified itself as having the
role `my-admin-role`
The structure of topic names in Pulsar reflects the hierarchy between tenants,
clusters, and namespaces:
-{% include topic.html ten="tenant" n="namespace" t="topic" %}
+```shell
+persistent://tenant/namespace/topic
+```
## Managing permissions
@@ -47,7 +50,8 @@ Superusers are configured in the broker configuration file in
[`conf/broker.conf
superUserRoles=my-super-user-1,my-super-user-2
```
-{% include message.html id="broker_conf_doc" %}
+> A full listing of parameters available in the `conf/broker.conf` file, as
well as the default
+> values for those parameters, can be found in [Broker
Configuration](reference-configuration.md#broker).
Typically, superuser roles are used for administrators and clients but also
for broker-to-broker authorization. When using
[geo-replication](administration-geo.md), every broker
needs to be able to publish to other clusters' topics.
diff --git a/site2/docs/security-tls.md b/site2/docs/security-tls.md
index ed27c69e70..fb197f63ec 100644
--- a/site2/docs/security-tls.md
+++ b/site2/docs/security-tls.md
@@ -105,7 +105,8 @@ authorizationEnabled=true
authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderTls
```
-{% include message.html id="broker_conf_doc" %}
+> A full listing of parameters available in the `conf/broker.conf` file, as
well as the default
+> values for those parameters, can be found in [Broker
Configuration](reference-configuration.md#broker).
## Configure the discovery service
diff --git a/site2/website/scripts/replace.js b/site2/website/scripts/replace.js
index 0ec9264e3d..b6bcaf99c9 100644
--- a/site2/website/scripts/replace.js
+++ b/site2/website/scripts/replace.js
@@ -49,7 +49,7 @@ const versions = getVersions();
const latestVersion = versions[0];
const from = [
- /pulsar:version/g,
+ /{{pulsar:version}}/g,
/pulsar:binary_release_url/g,
/pulsar:download_page_url/g,
/pulsar:repo_url/g
@@ -65,10 +65,10 @@ const options = {
from: from,
to: [
`${latestVersion}-incubating`,
- binaryReleaseUrl(`${latestVersion}-incubating`),
- downloadPageUrl(),
- pulsarRepoUrl()
- ],
+ binaryReleaseUrl(`${latestVersion}-incubating`),
+ downloadPageUrl(),
+ pulsarRepoUrl()
+ ],
dry: false
};
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
