This is an automated email from the ASF dual-hosted git repository.
spacewander pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/apisix.git
The following commit(s) were added to refs/heads/master by this push:
new 7945e6d98 docs: set up multiple CA certificates (#7540)
7945e6d98 is described below
commit 7945e6d98b4f03576af92c2255b52e367952d5f7
Author: tzssangglass <[email protected]>
AuthorDate: Fri Jul 29 17:15:38 2022 +0800
docs: set up multiple CA certificates (#7540)
---
docs/en/latest/certificate.md | 143 ++++++++++++++++++++++++++++++++++++++++++
docs/zh/latest/certificate.md | 143 ++++++++++++++++++++++++++++++++++++++++++
2 files changed, 286 insertions(+)
diff --git a/docs/en/latest/certificate.md b/docs/en/latest/certificate.md
index 5507e5ee3..38140f36c 100644
--- a/docs/en/latest/certificate.md
+++ b/docs/en/latest/certificate.md
@@ -171,3 +171,146 @@ private keys by `certs` and `keys`.
`APISIX` will pair certificate and private key with the same indice as a SSL
key
pair. So the length of `certs` and `keys` must be same.
+
+### set up multiple CA certificates
+
+APISIX currently uses CA certificates in several places, such as [Protect
Admin API](./mtls.md#protect-admin-api), [etcd with
mTLS](./mtls.md#etcd-with-mtls), and [Deployment
Modes](./architecture-design/deployment-role.md).
+
+In these places, `ssl_trusted_certificate` or `trusted_ca_cert` will be used
to set up the CA certificate, but these configurations will eventually be
translated into
[lua_ssl_trusted_certificate](https://github.com/openresty/lua-nginx-module#lua_ssl_trusted_certificate)
directive in OpenResty.
+
+If you need to set up different CA certificates in different places, then you
can package these CA certificates into a CA bundle file and point to this file
when you need to set up CAs. This will avoid the problem that the generated
`lua_ssl_trusted_certificate` has multiple locations and overwrites each other.
+
+The following is a complete example to show how to set up multiple CA
certificates in APISIX.
+
+Suppose we let client and APISIX Admin API, APISIX and ETCD communicate with
each other using mTLS protocol, and currently there are two CA certificates,
`foo_ca.crt` and `bar_ca.crt`, and use each of these two CA certificates to
issue client and server certificate pairs, `foo_ca.crt` and its issued
certificate pair are used to protect Admin API, and `bar_ca.crt` and its issued
certificate pair are used to protect ETCD.
+
+The following table details the configurations involved in this example and
what they do:
+
+| Configuration | Type | Description
|
+| ------------- | ------- |
-------------------------------------------------------------------------------------------------------------------------------------------------------------
|
+| foo_ca.crt | CA cert | Issues the secondary certificate required for
the client to communicate with the APISIX Admin API over mTLS.
|
+| foo_client.crt | cert | A certificate issued by `foo_ca.crt` and used
by the client to prove its identity when accessing the APISIX Admin API.
|
+| foo_client.key | key | Issued by `foo_ca.crt`, used by the client,
the key file required to access the APISIX Admin API.
|
+| foo_server.crt | cert | Issued by `foo_ca.crt`, used by APISIX,
corresponding to the `apisix.admin_api_mtls.admin_ssl_cert` configuration
entry. |
+| foo_server.key | key | Issued by `foo_ca.crt`, used by APISIX,
corresponding to the `apisix.admin_api_mtls.admin_ssl_cert_key` configuration
entry. |
+| admin.apisix.dev | doname | Common Name used in issuing `foo_server.crt`
certificate, through which the client accesses APISIX Admin API
|
+| bar_ca.crt | CA cert | Issues the secondary certificate required for
APISIX to communicate with ETCD over mTLS.
|
+| bar_etcd.crt | cert | Issued by `bar_ca.crt` and used by ETCD,
corresponding to the `-cert-file` option in the ETCD startup command.
|
+| bar_etcd.key | key | Issued by `bar_ca.crt` and used by ETCD,
corresponding to the `--key-file` option in the ETCD startup command.
|
+| bar_apisix.crt | cert | Issued by `bar_ca.crt`, used by APISIX,
corresponding to the `etcd.tls.cert` configuration entry.
|
+| bar_apisix.key | key | Issued by `bar_ca.crt`, used by APISIX,
corresponding to the `etcd.tls.key` configuration entry.
|
+| etcd.cluster.dev | key | Common Name used in issuing `bar_etcd.crt`
certificate, which is used as SNI when APISIX communicates with ETCD over mTLS.
corresponds to `etcd.tls.sni` configuration item. |
+| apisix.ca-bundle | CA bundle | Merged from `foo_ca.crt` and `bar_ca.crt`,
replacing `foo_ca.crt` and `bar_ca.crt`.
|
+
+1. Create CA bundle files
+
+```
+cat /path/to/foo_ca.crt /path/to/bar_ca.crt > apisix.ca-bundle
+```
+
+2. Start the ETCD cluster and enable client authentication
+
+Start by writing a `goreman` configuration named
`Procfile-single-enable-mtls`, the content as:
+
+```text
+# Use goreman to run `go get github.com/mattn/goreman`
+etcd1: etcd --name infra1 --listen-client-urls https://127.0.0.1:12379
--advertise-client-urls https://127.0.0.1:12379 --listen-peer-urls
http://127.0.0.1:12380 --initial-advertise-peer-urls http://127.0.0.1:12380
--initial-cluster-token etcd-cluster-1 --initial-cluster
'infra1=http://127.0.0.1:12380,infra2=http://127.0.0.1:22380,infra3=http://127.0.0.1:32380'
--initial-cluster-state new --cert-file /path/to/bar_etcd.crt --key-file
/path/to/bar_etcd.key --client-cert-auth --trusted-ca-fi [...]
+etcd2: etcd --name infra2 --listen-client-urls https://127.0.0.1:22379
--advertise-client-urls https://127.0.0.1:22379 --listen-peer-urls
http://127.0.0.1:22380 --initial-advertise-peer-urls http://127.0.0.1:22380
--initial-cluster-token etcd-cluster-1 --initial-cluster
'infra1=http://127.0.0.1:12380,infra2=http://127.0.0.1:22380,infra3=http://127.0.0.1:32380'
--initial-cluster-state new --cert-file /path/to/bar_etcd.crt --key-file
/path/to/bar_etcd.key --client-cert-auth --trusted-ca-fi [...]
+etcd3: etcd --name infra3 --listen-client-urls https://127.0.0.1:32379
--advertise-client-urls https://127.0.0.1:32379 --listen-peer-urls
http://127.0.0.1:32380 --initial-advertise-peer-urls http://127.0.0.1:32380
--initial-cluster-token etcd-cluster-1 --initial-cluster
'infra1=http://127.0.0.1:12380,infra2=http://127.0.0.1:22380,infra3=http://127.0.0.1:32380'
--initial-cluster-state new --cert-file /path/to/bar_etcd.crt --key-file
/path/to/bar_etcd.key --client-cert-auth --trusted-ca-fi [...]
+```
+
+Use `goreman` to start the ETCD cluster:
+
+```shell
+goreman -f Procfile-single-enable-mtls start > goreman.log 2>&1 &
+```
+
+3. Update `config.yaml`
+
+```yaml
+apisix:
+ admin_key:
+ - name: admin
+ key: edd1c9f034335f136f87ad84b625c8f1
+ role: admin
+ port_admin: 9180
+ https_admin: true
+
+ admin_api_mtls:
+ admin_ssl_ca_cert: /path/to/apisix.ca-bundle
+ admin_ssl_cert: /path/to/foo_server.crt
+ admin_ssl_cert_key: /path/to/foo_server.key
+
+ ssl:
+ ssl_trusted_certificate: /path/to/apisix.ca-bundle
+
+etcd:
+ host:
+ - "https://127.0.0.1:12379";
+ - "https://127.0.0.1:22379";
+ - "https://127.0.0.1:32379";
+ tls:
+ cert: /path/to/bar_apisix.crt
+ key: /path/to/bar_apisix.key
+ sni: etcd.cluster.dev
+```
+
+4. Test APISIX Admin API
+
+Start APISIX, if APISIX starts successfully and there is no abnormal output in
`logs/error.log`, it means that mTLS communication between APISIX and ETCD is
normal.
+
+Use curl to simulate a client, communicate with APISIX Admin API with mTLS,
and create a route:
+
+```shell
+curl -vvv \
+ --resolve 'admin.apisix.dev:9180:127.0.0.1'
https://admin.apisix.dev:9180/apisix/admin/routes/1 \
+ --cert /path/to/foo_client.crt \
+ --key /path/to/foo_client.key \
+ --cacert /path/to/apisix.ca-bundle \
+ -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -i -d '
+{
+ "uri": "/get",
+ "upstream": {
+ "type": "roundrobin",
+ "nodes": {
+ "httpbin.org:80": 1
+ }
+ }
+}'
+```
+
+A successful mTLS communication between curl and the APISIX Admin API is
indicated if the following SSL handshake process is output:
+
+```shell
+* TLSv1.3 (OUT), TLS handshake, Client hello (1):
+* TLSv1.3 (IN), TLS handshake, Server hello (2):
+* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
+* TLSv1.3 (IN), TLS handshake, Request CERT (13):
+* TLSv1.3 (IN), TLS handshake, Certificate (11):
+* TLSv1.3 (IN), TLS handshake, CERT verify (15):
+* TLSv1.3 (IN), TLS handshake, Finished (20):
+* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
+* TLSv1.3 (OUT), TLS handshake, Certificate (11):
+* TLSv1.3 (OUT), TLS handshake, CERT verify (15):
+* TLSv1.3 (OUT), TLS handshake, Finished (20):
+* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
+```
+
+5. Verify APISIX proxy
+
+```shell
+curl http://127.0.0.1:9080/get -i
+
+HTTP/1.1 200 OK
+Content-Type: application/json
+Content-Length: 298
+Connection: keep-alive
+Date: Tue, 26 Jul 2022 16:31:00 GMT
+Access-Control-Allow-Origin: *
+Access-Control-Allow-Credentials: true
+Server: APISIX/2.14.1
+
+...
+```
+
+APISIX proxied the request to the `/get` path of the upstream `httpbin.org`
and returned `HTTP/1.1 200 OK`. The whole process is working fine using CA
bundle instead of CA certificate.
diff --git a/docs/zh/latest/certificate.md b/docs/zh/latest/certificate.md
index e6c433e2f..8b89d6f6c 100644
--- a/docs/zh/latest/certificate.md
+++ b/docs/zh/latest/certificate.md
@@ -168,3 +168,146 @@ curl --resolve 'www.test.com:9443:127.0.0.1'
https://www.test.com:9443/hello -v
* `keys`：PEM 格式的 SSL 证书私钥列表
`APISIX` 会将相同下标的证书和私钥配对使用，因此 `certs` 和 `keys` 列表的长度必须一致。
+
+### 设置多个 CA 证书
+
+APISIX 目前支持在多处设置 CA 证书，比如 [保护 Admin API](./mtls.md#保护-admin-api)，[保护
ETCD](./mtls.md#保护-etcd)，以及
[部署模式](../../en/latest/architecture-design/deployment-role.md) 等。
+
+在这些地方，使用 `ssl_trusted_certificate` 或 `trusted_ca_cert` 来配置 CA 证书，但是这些配置最终将转化为
OpenResty 的
[lua_ssl_trusted_certificate](https://github.com/openresty/lua-nginx-module#lua_ssl_trusted_certificate)
指令。
+
+如果你需要在不同的地方指定不同的 CA 证书，你可以将这些 CA 证书制作成一个 CA bundle 文件，在需要用到 CA
证书的地方将配置指向这个文件。这样可以避免生成的 `lua_ssl_trusted_certificate` 存在多处并且互相覆盖的问题。
+
+下面用一个完整的例子来展示如何在 APISIX 设置多个 CA 证书。
+
+假设让 client 与 APISIX Admin API，APISIX 与 ETCD 之间都使用 mTLS 协议进行通信，目前有两张 CA 证书，分别是
`foo_ca.crt` 和 `bar_ca.crt`，用这两张 CA 证书各自签发 client 与 server 证书对，`foo_ca.crt`
及其签发的证书对用于保护 Admin API，`bar_ca.crt` 及其签发的证书对用于保护 ETCD。
+
+下表详细列出这个示例所涉及到的配置及其作用：
+
+| 配置 | 类型 | 用途
|
+| ------------- | ------- |
-----------------------------------------------------------------------------------------------------------
|
+| foo_ca.crt | CA 证书 | 签发客户端与 APISIX Admin API 进行 mTLS 通信所需的次级证书。
|
+| foo_client.crt | 证书 | 由 `foo_ca.crt` 签发，客户端使用，访问 APISIX Admin API
时证明自身身份的证书。 |
+| foo_client.key | 密钥文件 | 由 `foo_ca.crt` 签发，客户端使用，访问 APISIX Admin API
所需的密钥文件。 |
+| foo_server.crt | 证书 | 由 `foo_ca.crt` 签发，APISIX 使用，对应
`apisix.admin_api_mtls.admin_ssl_cert` 配置项。 |
+| foo_server.key | 密钥文件 | 由 `foo_ca.crt` 签发，APISIX 使用，对应
`apisix.admin_api_mtls.admin_ssl_cert_key` 配置项。 |
+| admin.apisix.dev | 域名 | 签发 `foo_server.crt` 证书时使用的 Common
Name，客户端通过该域名访问 APISIX Admin API |
+| bar_ca.crt | CA 证书 | 签发 APISIX 与 ETCD 进行 mTLS 通信所需的次级证书。
|
+| bar_etcd.crt | 证书 | 由 `bar_ca.crt` 签发，ETCD 使用，对应 ETCD 启动命令中的
`--cert-file` 选项。 |
+| bar_etcd.key | 密钥文件 | 由 `bar_ca.crt` 签发，ETCD 使用，对应 ETCD 启动命令中的
`--key-file` 选项。 |
+| bar_apisix.crt | 证书 | 由 `bar_ca.crt` 签发，APISIX 使用，对应 `etcd.tls.cert`
配置项。 |
+| bar_apisix.key | 密钥文件 | 由 `bar_ca.crt` 签发，APISIX 使用，对应 `etcd.tls.key`
配置项。 |
+| etcd.cluster.dev | 域名 | 签发 `bar_etcd.crt` 证书时使用的 Common Name，APISIX 与
ETCD 进行 mTLS 通信时，使用该域名作为 SNI。对应 `etcd.tls.sni` 配置项。|
+| apisix.ca-bundle | CA bundle | 由 `foo_ca.crt` 与 `bar_ca.crt` 合并而成，替代
`foo_ca.crt` 与 `bar_ca.crt`。 |
+
+1. 制作 CA bundle 文件
+
+```
+cat /path/to/foo_ca.crt /path/to/bar_ca.crt > apisix.ca-bundle
+```
+
+2. 启动 ETCD 集群，并开启客户端验证
+
+先编写 `goreman` 配置，命名为 `Procfile-single-enable-mtls`，内容如下：
+
+```text
+# 运行 `go get github.com/mattn/goreman` 安装 goreman，用 goreman 执行以下命令：
+etcd1: etcd --name infra1 --listen-client-urls https://127.0.0.1:12379
--advertise-client-urls https://127.0.0.1:12379 --listen-peer-urls
http://127.0.0.1:12380 --initial-advertise-peer-urls http://127.0.0.1:12380
--initial-cluster-token etcd-cluster-1 --initial-cluster
'infra1=http://127.0.0.1:12380,infra2=http://127.0.0.1:22380,infra3=http://127.0.0.1:32380'
--initial-cluster-state new --cert-file /path/to/bar_etcd.crt --key-file
/path/to/bar_etcd.key --client-cert-auth --trusted-ca-fi [...]
+etcd2: etcd --name infra2 --listen-client-urls https://127.0.0.1:22379
--advertise-client-urls https://127.0.0.1:22379 --listen-peer-urls
http://127.0.0.1:22380 --initial-advertise-peer-urls http://127.0.0.1:22380
--initial-cluster-token etcd-cluster-1 --initial-cluster
'infra1=http://127.0.0.1:12380,infra2=http://127.0.0.1:22380,infra3=http://127.0.0.1:32380'
--initial-cluster-state new --cert-file /path/to/bar_etcd.crt --key-file
/path/to/bar_etcd.key --client-cert-auth --trusted-ca-fi [...]
+etcd3: etcd --name infra3 --listen-client-urls https://127.0.0.1:32379
--advertise-client-urls https://127.0.0.1:32379 --listen-peer-urls
http://127.0.0.1:32380 --initial-advertise-peer-urls http://127.0.0.1:32380
--initial-cluster-token etcd-cluster-1 --initial-cluster
'infra1=http://127.0.0.1:12380,infra2=http://127.0.0.1:22380,infra3=http://127.0.0.1:32380'
--initial-cluster-state new --cert-file /path/to/bar_etcd.crt --key-file
/path/to/bar_etcd.key --client-cert-auth --trusted-ca-fi [...]
+```
+
+使用 `goreman` 来启动 ETCD 集群：
+
+```shell
+goreman -f Procfile-single-enable-mtls start > goreman.log 2>&1 &
+```
+
+3. 更新 `config.yaml`
+
+```yaml
+apisix:
+ admin_key:
+ - name: admin
+ key: edd1c9f034335f136f87ad84b625c8f1
+ role: admin
+ port_admin: 9180
+ https_admin: true
+
+ admin_api_mtls:
+ admin_ssl_ca_cert: /path/to/apisix.ca-bundle
+ admin_ssl_cert: /path/to/foo_server.crt
+ admin_ssl_cert_key: /path/to/foo_server.key
+
+ ssl:
+ ssl_trusted_certificate: /path/to/apisix.ca-bundle
+
+etcd:
+ host:
+ - "https://127.0.0.1:12379";
+ - "https://127.0.0.1:22379";
+ - "https://127.0.0.1:32379";
+ tls:
+ cert: /path/to/bar_apisix.crt
+ key: /path/to/bar_apisix.key
+ sni: etcd.cluster.dev
+```
+
+4. 测试 Admin API
+
+启动 APISIX，如果 APISIX 启动成功，`logs/error.log` 中没有异常输出，表示 APISIX 与 ETCD 之间进行 mTLS
通信正常。
+
+用 curl 模拟客户端，与 APISIX Admin API 进行 mTLS 通信，并创建一条路由：
+
+```shell
+curl -vvv \
+ --resolve 'admin.apisix.dev:9180:127.0.0.1'
https://admin.apisix.dev:9180/apisix/admin/routes/1 \
+ --cert /path/to/foo_client.crt \
+ --key /path/to/foo_client.key \
+ --cacert /path/to/apisix.ca-bundle \
+ -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -i -d '
+{
+ "uri": "/get",
+ "upstream": {
+ "type": "roundrobin",
+ "nodes": {
+ "httpbin.org:80": 1
+ }
+ }
+}'
+```
+
+如果输出以下 SSL 握手过程，表示 curl 与 APISIX Admin API 之间 mTLS 通信成功：
+
+```shell
+* TLSv1.3 (OUT), TLS handshake, Client hello (1):
+* TLSv1.3 (IN), TLS handshake, Server hello (2):
+* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
+* TLSv1.3 (IN), TLS handshake, Request CERT (13):
+* TLSv1.3 (IN), TLS handshake, Certificate (11):
+* TLSv1.3 (IN), TLS handshake, CERT verify (15):
+* TLSv1.3 (IN), TLS handshake, Finished (20):
+* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
+* TLSv1.3 (OUT), TLS handshake, Certificate (11):
+* TLSv1.3 (OUT), TLS handshake, CERT verify (15):
+* TLSv1.3 (OUT), TLS handshake, Finished (20):
+* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
+```
+
+5. 验证 APISIX 代理
+
+```shell
+curl http://127.0.0.1:9080/get -i
+
+HTTP/1.1 200 OK
+Content-Type: application/json
+Content-Length: 298
+Connection: keep-alive
+Date: Tue, 26 Jul 2022 16:31:00 GMT
+Access-Control-Allow-Origin: *
+Access-Control-Allow-Credentials: true
+Server: APISIX/2.14.1
+
+……
+```
+
+APISIX 将请求代理到了上游 `httpbin.org` 的 `/get` 路径，并返回了 `HTTP/1.1 200 OK`。整个过程使用 CA
bundle 替代 CA 证书是正常可用的。
