This is an automated email from the ASF dual-hosted git repository.
hjf pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/master by this push:
new 1e106a5 [Issue 2404][doc]Add proxy auth data forwarding and refine
pulsar proxy content (#7297)
1e106a5 is described below
commit 1e106a5a98a5b7675f3afa24e18f8de35f461712
Author: Jennifer Huang <[email protected]>
AuthorDate: Mon Jun 22 11:58:13 2020 +0800
[Issue 2404][doc]Add proxy auth data forwarding and refine pulsar proxy
content (#7297)
* add proxy auth data forwarding and refine pulsar proxy content
* fix PR- 5845
* add a note
* Update site2/docs/administration-proxy.md
Co-authored-by: Yu Liu <[email protected]>
* Update site2/docs/administration-proxy.md
Co-authored-by: HuanliMeng <[email protected]>
* Update site2/docs/administration-proxy.md
Co-authored-by: HuanliMeng <[email protected]>
Co-authored-by: Yu Liu <[email protected]>
Co-authored-by: HuanliMeng <[email protected]>
---
site2/docs/administration-proxy.md | 68 ++++++++++-------------------------
site2/docs/reference-configuration.md | 7 +++-
2 files changed, 25 insertions(+), 50 deletions(-)
diff --git a/site2/docs/administration-proxy.md
b/site2/docs/administration-proxy.md
index f4497d5..07d1aab 100644
--- a/site2/docs/administration-proxy.md
+++ b/site2/docs/administration-proxy.md
@@ -1,16 +1,16 @@
---
id: administration-proxy
-title: The Pulsar proxy
+title: Pulsar proxy
sidebar_label: Pulsar proxy
---
-The [Pulsar proxy](concepts-architecture-overview.md#pulsar-proxy) is an
optional gateway that you can run in front of the brokers in a Pulsar cluster.
You can run a Pulsar proxy in cases when direction connections between clients
and Pulsar brokers are either infeasible, undesirable, or both, for example
when you run Pulsar in a cloud environment or on
[Kubernetes](https://kubernetes.io) or an analogous platform.
+Pulsar proxy is an optional gateway. Pulsar proxy is used when direction
connections between clients and Pulsar brokers are either infeasible or
undesirable. For example, when you run Pulsar in a cloud environment or on
[Kubernetes](https://kubernetes.io) or an analogous platform, you can run
Pulsar proxy.
## Configure the proxy
-The proxy must have some way to find the addresses of the brokers of the
cluster. You can do this by either configuring the proxy to connect directly to
service discovery or by specifying a broker URL in the configuration.
+Before using the proxy, you need to configure it with the brokers addresses in
the cluster. You can configure the proxy to connect directly to service
discovery, or specify a broker URL in the configuration.
-### Option 1: Use service discovery
+### Use service discovery
Pulsar uses [ZooKeeper](https://zookeeper.apache.org) for service discovery.
To connect the proxy to ZooKeeper, specify the following in `conf/proxy.conf`.
```properties
@@ -18,13 +18,15 @@ zookeeperServers=zk-0,zk-1,zk-2
configurationStoreServers=zk-0:2184,zk-remote:2184
```
-> If you use service discovery, the network ACL must allow the proxy to talk
to the ZooKeeper nodes on the zookeeper client port, which is usually 2181, and
on the configuration store client port, which is 2184 by default. Opening the
network ACLs means that if someone compromises a proxy, they have full access
to ZooKeeper. For this reason, using broker URLs to configure the proxy is more
secure.
+> To use service discovery, you need to open the network ACLs, so the proxy
can connects to the ZooKeeper nodes through the ZooKeeper client port (port
`2181`) and the configuration store client port (port `2184`).
-### Option 2: Use broker URLs
+> However, it is not secure to use service discovery. Because if the network
ACL is open, when someone compromises a proxy, they have full access to
ZooKeeper.
-The more secure method of configuring the proxy is to specify a URL to connect
to the brokers.
+### Use broker URLs
->
[Authorization](security-authorization#enable-authorization-and-assign-superusers)
at the proxy requires access to ZooKeeper, so if you use these broker URLs to
connect to the brokers, you should disable the Proxy level authorization.
Brokers still authorize requests after the proxy forwards them.
+It is more secure to specify a URL to connect to the brokers.
+
+Proxy authorization requires access to ZooKeeper, so if you use these broker
URLs to connect to the brokers, you need to disable authorization at the Proxy
level. Brokers still authorize requests after the proxy forwards them.
You can configure the broker URLs in `conf/proxy.conf` as follows.
@@ -34,18 +36,18 @@ brokerWebServiceURL=http://brokers.example.com:8080
functionWorkerWebServiceURL=http://function-workers.example.com:8080
```
-Or if you use TLS:
+If you use TLS, configure the broker URLs in the following way:
```properties
brokerServiceURLTLS=pulsar+ssl://brokers.example.com:6651
brokerWebServiceURLTLS=https://brokers.example.com:8443
functionWorkerWebServiceURL=https://function-workers.example.com:8443
```
-The hostname in the URLs provided should be a DNS entry which points to
multiple brokers or a Virtual IP which is backed by multiple broker IP
addresses so that the proxy does not lose connectivity to the pulsar cluster if
a single broker becomes unavailable.
+The hostname in the URLs provided should be a DNS entry which points to
multiple brokers or a virtual IP address, which is backed by multiple broker IP
addresses, so that the proxy does not lose connectivity to Pulsar cluster if a
single broker becomes unavailable.
The ports to connect to the brokers (6650 and 8080, or in the case of TLS,
6651 and 8443) should be open in the network ACLs.
-Note that if you do not use functions, then you do not need to configure
`functionWorkerWebServiceURL`.
+Note that if you do not use functions, you do not need to configure
`functionWorkerWebServiceURL`.
## Start the proxy
@@ -56,50 +58,18 @@ $ cd /path/to/pulsar/directory
$ bin/pulsar proxy
```
-> You can run as many instances of the Pulsar proxy in a cluster as you want.
-
+> You can run multiple instances of the Pulsar proxy in a cluster.
## Stop the proxy
-The Pulsar proxy runs by default in the foreground. To stop the proxy, simply
stop the process in which the proxy is running.
+Pulsar proxy runs in the foreground by default. To stop the proxy, simply stop
the process in which the proxy is running.
## Proxy frontends
-You can run the Pulsar proxy behind some kind of load-distributing frontend,
such as an
[HAProxy](https://www.digitalocean.com/community/tutorials/an-introduction-to-haproxy-and-load-balancing-concepts)
load balancer.
+You can run Pulsar proxy behind some kind of load-distributing frontend, such
as an
[HAProxy](https://www.digitalocean.com/community/tutorials/an-introduction-to-haproxy-and-load-balancing-concepts)
load balancer.
## Use Pulsar clients with the proxy
-Once your Pulsar proxy is up and running, preferably behind a
load-distributing [frontend](#proxy-frontends), clients can connect to the
proxy via whichever address that the frontend uses. If the address is the DNS
address `pulsar.cluster.default`, for example, then the connection URL for
clients is `pulsar://pulsar.cluster.default:6650`.
-
-## Proxy configuration
-
-You can configure the Pulsar proxy using the
[`proxy.conf`](reference-configuration.md#proxy) configuration file. The
following parameters are available in that file:
-
-|Name|Description|Default|
-|---|---|---|
-|zookeeperServers| The ZooKeeper quorum connection string (as a
comma-separated list) ||
-|configurationStoreServers| Configuration store connection string (as a
comma-separated list) ||
-|zookeeperSessionTimeoutMs| ZooKeeper session timeout (in milliseconds) |30000|
-|servicePort| The port to use for server binary Protobuf requests |6650|
-|servicePortTls| The port to use to server binary Protobuf TLS requests
|6651|
-|statusFilePath | Path for the file used to determine the rotation status for
the proxy instance when responding to service discovery health checks ||
-|advertisedAddress|Hostname or IP address the service advertises to the
outside world.|`InetAddress.getLocalHost().getHostname()`|
-|authenticationEnabled| Whether authentication is enabled for the Pulsar proxy
|false|
-|authenticateMetricsEndpoint| Whether the '/metrics' endpoint requires
authentication. Defaults to true. 'authenticationEnabled' must also be set for
this to take effect. |true|
-|authenticationProviders| Authentication provider name list (a comma-separated
list of class names) ||
-|authorizationEnabled| Whether authorization is enforced by the Pulsar proxy
|false|
-|authorizationProvider| Authorization provider as a fully qualified class name
|org.apache.pulsar.broker.authorization.PulsarAuthorizationProvider|
-|brokerClientAuthenticationPlugin| The authentication plugin used by the
Pulsar proxy to authenticate with Pulsar brokers ||
-|brokerClientAuthenticationParameters| The authentication parameters used by
the Pulsar proxy to authenticate with Pulsar brokers ||
-|brokerClientTrustCertsFilePath| The path to trusted certificates used by the
Pulsar proxy to authenticate with Pulsar brokers ||
-|superUserRoles| Role names that are treated as “super-users,” meaning that
they are able to perform all admin ||
-|forwardAuthorizationCredentials| Whether client authorization credentials are
forwared to the broker for re-authorization. Authentication must be enabled via
authenticationEnabled=true for this to take effect. |false|
-|maxConcurrentInboundConnections| Max concurrent inbound connections. The
proxy rejects requests beyond that. |10000|
-|maxConcurrentLookupRequests| Max concurrent outbound connections. The proxy
errors out requests beyond that. |50000|
-|tlsEnabledInProxy| Whether TLS is enabled for the proxy |false|
-|tlsEnabledWithBroker| Whether TLS is enabled when communicating with Pulsar
brokers |false|
-|tlsCertificateFilePath| Path for the TLS certificate file ||
-|tlsKeyFilePath| Path for the TLS private key file ||
-|tlsTrustCertsFilePath| Path for the trusted TLS certificate pem file ||
-|tlsHostnameVerificationEnabled| Whether the hostname is validated when the
proxy creates a TLS connection with brokers |false|
-|tlsRequireTrustedClientCertOnConnect| Whether client certificates are
required for TLS. Connections are rejected if the client certificate is not
trusted. |false|
+Once your Pulsar proxy is up and running, preferably behind a
load-distributing [frontend](#proxy-frontends), clients can connect to the
proxy via whichever address that the frontend uses. If the address is the DNS
address `pulsar.cluster.default`, for example, the connection URL for clients
is `pulsar://pulsar.cluster.default:6650`.
+
+For more information on Proxy configuration, refer to [Pulsar
proxy](reference-configuration.md#pulsar-proxy).
diff --git a/site2/docs/reference-configuration.md
b/site2/docs/reference-configuration.md
index 900e4df..3b0ef40 100644
--- a/site2/docs/reference-configuration.md
+++ b/site2/docs/reference-configuration.md
@@ -107,6 +107,7 @@ Pulsar brokers are responsible for handling incoming
messages from producers, di
|---|---|---|
|advertisedListeners|Specify multiple advertised listeners for the
broker.<br><br>The format is
`<listener_name>:pulsar://<host>:<port>`.<br><br>If there are multiple
listeners, separate them with commas.<br><br>**Note**: do not use this
configuration with `advertisedAddress` and `brokerServicePort`. If the value of
this configuration is empty, the broker uses `advertisedAddress` and
`brokerServicePort`|/|
internalListenerName|Specify the internal listener name for the
broker.<br><br>**Note**: the listener name must be contained in
`advertisedListeners`.<br><br> If the value of this configuration is empty, the
broker uses the first listener as the internal listener.|/|
+|authenticateOriginalAuthData| If this flag is set to `true`, the broker
authenticates the original Auth data; else it just accepts the
originalPrincipal and authorizes it (if required). |false|
|enablePersistentTopics| Whether persistent topics are enabled on the broker
|true|
|enableNonPersistentTopics| Whether non-persistent topics are enabled on the
broker |true|
|functionsWorkerEnabled| Whether the Pulsar Functions worker service is
enabled in the broker |false|
@@ -313,7 +314,6 @@ The [`pulsar-client`](reference-cli-tools.md#pulsar-client)
CLI tool can be used
## Log4j
-
|Name|Default|
|---|---|
|pulsar.root.logger| WARN,CONSOLE|
@@ -335,6 +335,9 @@ The [`pulsar-client`](reference-cli-tools.md#pulsar-client)
CLI tool can be used
|log4j.appender.TRACEFILE.layout| org.apache.log4j.PatternLayout|
|log4j.appender.TRACEFILE.layout.ConversionPattern| %d{ISO8601} - %-5p
[%t:%C{1}@%L][%x] - %m%n|
+> Note: 'topic' in log4j2.appender is configurable.
+> - If you want to append all logs to a single topic, set the same topic name.
+> - If you want to append logs to different topics, you can set different
topic names.
## Log4j shell
@@ -355,6 +358,7 @@ The [`pulsar-client`](reference-cli-tools.md#pulsar-client)
CLI tool can be used
|Name|Description|Default|
|---|---|---|
+|authenticateOriginalAuthData| If this flag is set to `true`, the broker
authenticates the original Auth data; else it just accepts the
originalPrincipal and authorizes it (if required). |false|
|zookeeperServers| The quorum connection string for local ZooKeeper ||
|zooKeeperCacheExpirySeconds|ZooKeeper cache expiry time in seconds|300
|configurationStoreServers| Configuration store connection string (as a
comma-separated list) ||
@@ -484,6 +488,7 @@ The [Pulsar
proxy](concepts-architecture-overview.md#pulsar-proxy) can be config
|Name|Description|Default|
|---|---|---|
+|forwardAuthorizationCredentials| Forward client authorization credentials to
Broker for re-authorization, and make sure authentication is enabled for this
to take effect. |false|
|zookeeperServers| The ZooKeeper quorum connection string (as a
comma-separated list) ||
|configurationStoreServers| Configuration store connection string (as a
comma-separated list) ||
|zookeeperSessionTimeoutMs| ZooKeeper session timeout (in milliseconds) |30000|
