spacewander commented on code in PR #7540: URL: https://github.com/apache/apisix/pull/7540#discussion_r930641519
########## docs/en/latest/certificate.md: ########## @@ -171,3 +171,134 @@ private keys by `certs` and `keys`. `APISIX` will pair certificate and private key with the same indice as a SSL key pair. So the length of `certs` and `keys` must be same. + +### set up multiple CA certificates + +APISIX currently uses CA certificates in several places, such as [Protect Admin API](./mtls.md#protect-admin-api), [etcd with mTLS](./mtls.md#etcd-with-mtls), and [Deployment Modes](./architecture-design/deployment-role.md). + +In these places, `ssl_trusted_certificate` or `trusted_ca_cert` will be used to set up the CA certificate, but these configurations will eventually be translated into [lua_ssl_trusted_certificate](https://github.com/openresty/lua-nginx-module#lua_ssl_trusted_certificate) directive in OpenResty. + +If you need to set up different CA certificates in different places, then you can package these CA certificates into a CA bundle file and point to this file when you need to set up CAs. This will avoid the problem that the generated `lua_ssl_trusted_certificate` has multiple locations and overwrites each other. + +The following is a complete example to show how to set up multiple CA certificates in APISXI. + +Suppose we let Client and APISIX Admin API, APISXI and ETCD communicate with each other using mTLS protocol, and currently there are two CA certificates, `foo_ca.crt` and `bar_ca.crt`, and use each of these two CA certificates to issue client and server certificate pairs, `foo_ca. crt` and its issued certificate pair are used to protect Admin API, and `bar_ca.crt` and its issued certificate pair are used to protect ETCD. + +The following table details the configurations involved in this example and what they do: + +| 配置 | 类型 | 用途 | Review Comment: Please use English ########## docs/en/latest/certificate.md: ########## @@ -171,3 +171,134 @@ private keys by `certs` and `keys`. `APISIX` will pair certificate and private key with the same indice as a SSL key pair. So the length of `certs` and `keys` must be same. + +### set up multiple CA certificates + +APISIX currently uses CA certificates in several places, such as [Protect Admin API](./mtls.md#protect-admin-api), [etcd with mTLS](./mtls.md#etcd-with-mtls), and [Deployment Modes](./architecture-design/deployment-role.md). + +In these places, `ssl_trusted_certificate` or `trusted_ca_cert` will be used to set up the CA certificate, but these configurations will eventually be translated into [lua_ssl_trusted_certificate](https://github.com/openresty/lua-nginx-module#lua_ssl_trusted_certificate) directive in OpenResty. + +If you need to set up different CA certificates in different places, then you can package these CA certificates into a CA bundle file and point to this file when you need to set up CAs. This will avoid the problem that the generated `lua_ssl_trusted_certificate` has multiple locations and overwrites each other. + +The following is a complete example to show how to set up multiple CA certificates in APISXI. + +Suppose we let Client and APISIX Admin API, APISXI and ETCD communicate with each other using mTLS protocol, and currently there are two CA certificates, `foo_ca.crt` and `bar_ca.crt`, and use each of these two CA certificates to issue client and server certificate pairs, `foo_ca. crt` and its issued certificate pair are used to protect Admin API, and `bar_ca.crt` and its issued certificate pair are used to protect ETCD. + +The following table details the configurations involved in this example and what they do: + +| 配置 | 类型 | 用途 | +| ------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| foo_ca.crt | CA cert | Issues the secondary certificate required for the client to communicate with the APISIX Admin API over mTLS. | +| foo_client.crt | cert | A certificate issued by `foo_ca.crt` and used by the client to prove its identity when accessing the APISIX Admin API. | +| foo_client.key | key | Issued by `foo_ca.crt`, used by the client, the key file required to access the APISIX Admin API. | +| foo_server.crt | cert | Issued by `foo_ca.crt`, used by APISIX, corresponding to the `apisix.admin_api_mtls.admin_ssl_cert` configuration entry. | +| foo_server.key | key | Issued by `foo_ca.crt`, used by APISIX, corresponding to the `apisix.admin_api_mtls.admin_ssl_cert_key` configuration entry. | +| admin.apisix.dev | doname | Common Name used in issuing `foo_server.crt` certificate, through which the client accesses APISIX Admin API | +| bar_ca.crt | CA cert | Issues the secondary certificate required for APISIX to communicate with ETCD over mTLS. | +| bar_etcd.crt | cert | Issued by `bar_ca.crt` and used by ETCD, corresponding to the `-cert-file` option in the ETCD startup command. | +| bar_etcd.key | key | Issued by `bar_ca.crt` and used by ETCD, corresponding to the `--key-file` option in the ETCD startup command. | +| bar_apisix.crt | cert | Issued by `bar_ca.crt`, used by APISIX, corresponding to the `etcd.tls.cert` configuration entry. | +| bar_apisix.key | key | Issued by `bar_ca.crt`, used by APISIX, corresponding to the `etcd.tls.key` configuration entry. | +| etcd.cluster.dev | key | Common Name used in issuing `bar_etcd.crt` certificate, which is used as SNI when APISIX communicates with ETCD over mTLS. corresponds to `etcd.tls.sni` configuration item. | +| apisix.ca-bundle | CA bundle | Merged from `foo_ca.crt` and `bar_ca.crt`, replacing `foo_ca.crt` and `bar_ca.crt`. | + +1. Create CA bundle files + +``` +cat /path/to/foo_ca.crt /path/to/bar_ca.crt > apisix.ca-bundle +``` + +2. Start the ETCD cluster and enable client authentication + +```shell +# Use goreman to run `go get github.com/mattn/goreman` +etcd1: etcd --name infra1 --listen-client-urls https://127.0.0.1:12379 --advertise-client-urls https://127.0.0.1:12379 --listen-peer-urls http://127.0.0.1:12380 --initial-advertise-peer-urls http://127.0.0.1:12380 --initial-cluster-token etcd-cluster-1 --initial-cluster 'infra1=http://127.0.0.1:12380,infra2=http://127.0.0.1:22380,infra3=http://127.0.0.1:32380' --initial-cluster-state new --cert-file /path/to/bar_etcd.crt --key-file /path/to/bar_etcd.key --client-cert-auth --trusted-ca-file /path/to/apisix.ca-bundle Review Comment: Look like this is some configuration instead of shell command? ########## docs/en/latest/certificate.md: ########## @@ -171,3 +171,134 @@ private keys by `certs` and `keys`. `APISIX` will pair certificate and private key with the same indice as a SSL key pair. So the length of `certs` and `keys` must be same. + +### set up multiple CA certificates + +APISIX currently uses CA certificates in several places, such as [Protect Admin API](./mtls.md#protect-admin-api), [etcd with mTLS](./mtls.md#etcd-with-mtls), and [Deployment Modes](./architecture-design/deployment-role.md). + +In these places, `ssl_trusted_certificate` or `trusted_ca_cert` will be used to set up the CA certificate, but these configurations will eventually be translated into [lua_ssl_trusted_certificate](https://github.com/openresty/lua-nginx-module#lua_ssl_trusted_certificate) directive in OpenResty. + +If you need to set up different CA certificates in different places, then you can package these CA certificates into a CA bundle file and point to this file when you need to set up CAs. This will avoid the problem that the generated `lua_ssl_trusted_certificate` has multiple locations and overwrites each other. + +The following is a complete example to show how to set up multiple CA certificates in APISXI. + +Suppose we let Client and APISIX Admin API, APISXI and ETCD communicate with each other using mTLS protocol, and currently there are two CA certificates, `foo_ca.crt` and `bar_ca.crt`, and use each of these two CA certificates to issue client and server certificate pairs, `foo_ca. crt` and its issued certificate pair are used to protect Admin API, and `bar_ca.crt` and its issued certificate pair are used to protect ETCD. + +The following table details the configurations involved in this example and what they do: + +| 配置 | 类型 | 用途 | +| ------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| foo_ca.crt | CA cert | Issues the secondary certificate required for the client to communicate with the APISIX Admin API over mTLS. | +| foo_client.crt | cert | A certificate issued by `foo_ca.crt` and used by the client to prove its identity when accessing the APISIX Admin API. | +| foo_client.key | key | Issued by `foo_ca.crt`, used by the client, the key file required to access the APISIX Admin API. | +| foo_server.crt | cert | Issued by `foo_ca.crt`, used by APISIX, corresponding to the `apisix.admin_api_mtls.admin_ssl_cert` configuration entry. | +| foo_server.key | key | Issued by `foo_ca.crt`, used by APISIX, corresponding to the `apisix.admin_api_mtls.admin_ssl_cert_key` configuration entry. | +| admin.apisix.dev | doname | Common Name used in issuing `foo_server.crt` certificate, through which the client accesses APISIX Admin API | +| bar_ca.crt | CA cert | Issues the secondary certificate required for APISIX to communicate with ETCD over mTLS. | +| bar_etcd.crt | cert | Issued by `bar_ca.crt` and used by ETCD, corresponding to the `-cert-file` option in the ETCD startup command. | +| bar_etcd.key | key | Issued by `bar_ca.crt` and used by ETCD, corresponding to the `--key-file` option in the ETCD startup command. | +| bar_apisix.crt | cert | Issued by `bar_ca.crt`, used by APISIX, corresponding to the `etcd.tls.cert` configuration entry. | +| bar_apisix.key | key | Issued by `bar_ca.crt`, used by APISIX, corresponding to the `etcd.tls.key` configuration entry. | +| etcd.cluster.dev | key | Common Name used in issuing `bar_etcd.crt` certificate, which is used as SNI when APISIX communicates with ETCD over mTLS. corresponds to `etcd.tls.sni` configuration item. | +| apisix.ca-bundle | CA bundle | Merged from `foo_ca.crt` and `bar_ca.crt`, replacing `foo_ca.crt` and `bar_ca.crt`. | + +1. Create CA bundle files + +``` +cat /path/to/foo_ca.crt /path/to/bar_ca.crt > apisix.ca-bundle +``` + +2. Start the ETCD cluster and enable client authentication + +```shell +# Use goreman to run `go get github.com/mattn/goreman` +etcd1: etcd --name infra1 --listen-client-urls https://127.0.0.1:12379 --advertise-client-urls https://127.0.0.1:12379 --listen-peer-urls http://127.0.0.1:12380 --initial-advertise-peer-urls http://127.0.0.1:12380 --initial-cluster-token etcd-cluster-1 --initial-cluster 'infra1=http://127.0.0.1:12380,infra2=http://127.0.0.1:22380,infra3=http://127.0.0.1:32380' --initial-cluster-state new --cert-file /path/to/bar_etcd.crt --key-file /path/to/bar_etcd.key --client-cert-auth --trusted-ca-file /path/to/apisix.ca-bundle +etcd2: etcd --name infra2 --listen-client-urls https://127.0.0.1:22379 --advertise-client-urls https://127.0.0.1:22379 --listen-peer-urls http://127.0.0.1:22380 --initial-advertise-peer-urls http://127.0.0.1:22380 --initial-cluster-token etcd-cluster-1 --initial-cluster 'infra1=http://127.0.0.1:12380,infra2=http://127.0.0.1:22380,infra3=http://127.0.0.1:32380' --initial-cluster-state new --cert-file /path/to/bar_etcd.crt --key-file /path/to/bar_etcd.key --client-cert-auth --trusted-ca-file /path/to/apisix.ca-bundle +etcd3: etcd --name infra3 --listen-client-urls https://127.0.0.1:32379 --advertise-client-urls https://127.0.0.1:32379 --listen-peer-urls http://127.0.0.1:32380 --initial-advertise-peer-urls http://127.0.0.1:32380 --initial-cluster-token etcd-cluster-1 --initial-cluster 'infra1=http://127.0.0.1:12380,infra2=http://127.0.0.1:22380,infra3=http://127.0.0.1:32380' --initial-cluster-state new --cert-file /path/to/bar_etcd.crt --key-file /path/to/bar_etcd.key --client-cert-auth --trusted-ca-file /path/to/apisix.ca-bundle +``` + +3. Update `config.yaml` + +```yaml +apisix: + admin_key: + - name: admin + key: edd1c9f034335f136f87ad84b625c8f1 + role: admin + port_admin: 9180 + https_admin: true + + admin_api_mtls: + admin_ssl_ca_cert: /path/to/apisix.ca-bundle + admin_ssl_cert: /path/to/foo_server.crt + admin_ssl_cert_key: /path/to/foo_server.key + + ssl: + ssl_trusted_certificate: /path/to/apisix.ca-bundle + +etcd: + host: + - "https://127.0.0.1:12379"; + - "https://127.0.0.1:22379"; + - "https://127.0.0.1:32379"; + tls: + cert: /path/to/bar_apisix.crt + key: /path/to/bar_apisix.key + sni: etcd.cluster.dev +``` + +4. Test APISIX Admin API + +Start APISIX, if APISIX starts successfully and there is no abnormal output in `logs/error.log`, it means that mTLS communication between APISIX and ETCD is normal. + +Use curl to simulate a client, communicate with APISIX Admin API with mTLS, and create a route: + +```shell +curl -vvv --resolve 'admin.apisix.dev:9180:127.0.0.1' https://admin.apisix.dev:9180/apisix/admin/routes/1 --cert /path/to/foo_client.crt --key /path/to/foo_client.key --cacert/path/to/apisix.ca-bundle -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -i -d ' +{ + "uri": "/get", + "upstream": { + "type": "roundrobin", + "nodes": { + "httpbin.org:80": 1 + } + } +}' +``` + +A successful mTLS communication between curl and the APISIX Admin API is indicated if the following SSL handshake process is output: + +``` +shell Review Comment: Bad syntax ########## docs/en/latest/certificate.md: ########## @@ -171,3 +171,134 @@ private keys by `certs` and `keys`. `APISIX` will pair certificate and private key with the same indice as a SSL key pair. So the length of `certs` and `keys` must be same. + +### set up multiple CA certificates + +APISIX currently uses CA certificates in several places, such as [Protect Admin API](./mtls.md#protect-admin-api), [etcd with mTLS](./mtls.md#etcd-with-mtls), and [Deployment Modes](./architecture-design/deployment-role.md). + +In these places, `ssl_trusted_certificate` or `trusted_ca_cert` will be used to set up the CA certificate, but these configurations will eventually be translated into [lua_ssl_trusted_certificate](https://github.com/openresty/lua-nginx-module#lua_ssl_trusted_certificate) directive in OpenResty. + +If you need to set up different CA certificates in different places, then you can package these CA certificates into a CA bundle file and point to this file when you need to set up CAs. This will avoid the problem that the generated `lua_ssl_trusted_certificate` has multiple locations and overwrites each other. + +The following is a complete example to show how to set up multiple CA certificates in APISXI. + +Suppose we let Client and APISIX Admin API, APISXI and ETCD communicate with each other using mTLS protocol, and currently there are two CA certificates, `foo_ca.crt` and `bar_ca.crt`, and use each of these two CA certificates to issue client and server certificate pairs, `foo_ca. crt` and its issued certificate pair are used to protect Admin API, and `bar_ca.crt` and its issued certificate pair are used to protect ETCD. + +The following table details the configurations involved in this example and what they do: + +| 配置 | 类型 | 用途 | +| ------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| foo_ca.crt | CA cert | Issues the secondary certificate required for the client to communicate with the APISIX Admin API over mTLS. | +| foo_client.crt | cert | A certificate issued by `foo_ca.crt` and used by the client to prove its identity when accessing the APISIX Admin API. | +| foo_client.key | key | Issued by `foo_ca.crt`, used by the client, the key file required to access the APISIX Admin API. | +| foo_server.crt | cert | Issued by `foo_ca.crt`, used by APISIX, corresponding to the `apisix.admin_api_mtls.admin_ssl_cert` configuration entry. | +| foo_server.key | key | Issued by `foo_ca.crt`, used by APISIX, corresponding to the `apisix.admin_api_mtls.admin_ssl_cert_key` configuration entry. | +| admin.apisix.dev | doname | Common Name used in issuing `foo_server.crt` certificate, through which the client accesses APISIX Admin API | +| bar_ca.crt | CA cert | Issues the secondary certificate required for APISIX to communicate with ETCD over mTLS. | +| bar_etcd.crt | cert | Issued by `bar_ca.crt` and used by ETCD, corresponding to the `-cert-file` option in the ETCD startup command. | +| bar_etcd.key | key | Issued by `bar_ca.crt` and used by ETCD, corresponding to the `--key-file` option in the ETCD startup command. | +| bar_apisix.crt | cert | Issued by `bar_ca.crt`, used by APISIX, corresponding to the `etcd.tls.cert` configuration entry. | +| bar_apisix.key | key | Issued by `bar_ca.crt`, used by APISIX, corresponding to the `etcd.tls.key` configuration entry. | +| etcd.cluster.dev | key | Common Name used in issuing `bar_etcd.crt` certificate, which is used as SNI when APISIX communicates with ETCD over mTLS. corresponds to `etcd.tls.sni` configuration item. | +| apisix.ca-bundle | CA bundle | Merged from `foo_ca.crt` and `bar_ca.crt`, replacing `foo_ca.crt` and `bar_ca.crt`. | + +1. Create CA bundle files + +``` +cat /path/to/foo_ca.crt /path/to/bar_ca.crt > apisix.ca-bundle +``` + +2. Start the ETCD cluster and enable client authentication + +```shell +# Use goreman to run `go get github.com/mattn/goreman` +etcd1: etcd --name infra1 --listen-client-urls https://127.0.0.1:12379 --advertise-client-urls https://127.0.0.1:12379 --listen-peer-urls http://127.0.0.1:12380 --initial-advertise-peer-urls http://127.0.0.1:12380 --initial-cluster-token etcd-cluster-1 --initial-cluster 'infra1=http://127.0.0.1:12380,infra2=http://127.0.0.1:22380,infra3=http://127.0.0.1:32380' --initial-cluster-state new --cert-file /path/to/bar_etcd.crt --key-file /path/to/bar_etcd.key --client-cert-auth --trusted-ca-file /path/to/apisix.ca-bundle +etcd2: etcd --name infra2 --listen-client-urls https://127.0.0.1:22379 --advertise-client-urls https://127.0.0.1:22379 --listen-peer-urls http://127.0.0.1:22380 --initial-advertise-peer-urls http://127.0.0.1:22380 --initial-cluster-token etcd-cluster-1 --initial-cluster 'infra1=http://127.0.0.1:12380,infra2=http://127.0.0.1:22380,infra3=http://127.0.0.1:32380' --initial-cluster-state new --cert-file /path/to/bar_etcd.crt --key-file /path/to/bar_etcd.key --client-cert-auth --trusted-ca-file /path/to/apisix.ca-bundle +etcd3: etcd --name infra3 --listen-client-urls https://127.0.0.1:32379 --advertise-client-urls https://127.0.0.1:32379 --listen-peer-urls http://127.0.0.1:32380 --initial-advertise-peer-urls http://127.0.0.1:32380 --initial-cluster-token etcd-cluster-1 --initial-cluster 'infra1=http://127.0.0.1:12380,infra2=http://127.0.0.1:22380,infra3=http://127.0.0.1:32380' --initial-cluster-state new --cert-file /path/to/bar_etcd.crt --key-file /path/to/bar_etcd.key --client-cert-auth --trusted-ca-file /path/to/apisix.ca-bundle +``` + +3. Update `config.yaml` + +```yaml +apisix: + admin_key: + - name: admin + key: edd1c9f034335f136f87ad84b625c8f1 + role: admin + port_admin: 9180 + https_admin: true + + admin_api_mtls: + admin_ssl_ca_cert: /path/to/apisix.ca-bundle + admin_ssl_cert: /path/to/foo_server.crt + admin_ssl_cert_key: /path/to/foo_server.key + + ssl: + ssl_trusted_certificate: /path/to/apisix.ca-bundle + +etcd: + host: + - "https://127.0.0.1:12379"; + - "https://127.0.0.1:22379"; + - "https://127.0.0.1:32379"; + tls: + cert: /path/to/bar_apisix.crt + key: /path/to/bar_apisix.key + sni: etcd.cluster.dev +``` + +4. Test APISIX Admin API + +Start APISIX, if APISIX starts successfully and there is no abnormal output in `logs/error.log`, it means that mTLS communication between APISIX and ETCD is normal. + +Use curl to simulate a client, communicate with APISIX Admin API with mTLS, and create a route: + +```shell +curl -vvv --resolve 'admin.apisix.dev:9180:127.0.0.1' https://admin.apisix.dev:9180/apisix/admin/routes/1 --cert /path/to/foo_client.crt --key /path/to/foo_client.key --cacert/path/to/apisix.ca-bundle -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -i -d ' +{ + "uri": "/get", + "upstream": { + "type": "roundrobin", + "nodes": { + "httpbin.org:80": 1 + } + } +}' +``` + +A successful mTLS communication between curl and the APISIX Admin API is indicated if the following SSL handshake process is output: + +``` +shell +* TLSv1.3 (OUT), TLS handshake, Client hello (1): +* TLSv1.3 (IN), TLS handshake, Server hello (2): +* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8): +* TLSv1.3 (IN), TLS handshake, Request CERT (13): +* TLSv1.3 (IN), TLS handshake, Certificate (11): +* TLSv1.3 (IN), TLS handshake, CERT verify (15): +* TLSv1.3 (IN), TLS handshake, Finished (20): +* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1): +* TLSv1.3 (OUT), TLS handshake, Certificate (11): +* TLSv1.3 (OUT), TLS handshake, CERT verify (15): +* TLSv1.3 (OUT), TLS handshake, Finished (20): +* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 +``` + +5. Verify APISIX proxy + +```shell +curl http://127.0.0.1:9080/get -i + +HTTP/1.1 200 OK +Content-Type: application/json +Content-Length: 298 +Connection: keep-alive +Date: Tue, 26 Jul 2022 16:31:00 GMT +Access-Control-Allow-Origin: * +Access-Control-Allow-Credentials: true +Server: APISIX/2.14.1 + +…… Review Comment: Chinese punctuation in English doc? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
