hansedong commented on issue #9336:
URL: https://github.com/apache/apisix/issues/9336#issuecomment-1619896950
I would like to add some information. I compiled RPM based on version 3.4.0
and reinstalled APISIX.
Here is my configuration:
```
deployment:
role: traditional
role_traditional:
config_provider: etcd
admin:
# Default token when use API to call for Admin API.
# *NOTE*: Highly recommended to modify this value to protect APISIX's
Admin API.
# Disabling this configuration item means that the Admin API does not
# require any authentication.
admin_key:
-
name: admin
key: xxx
role: admin # admin: manage all configuration data
# viewer: only can view configuration
data
-
name: viewer
key: xxx
role: viewer
enable_admin_cors: true # Admin API support CORS response
headers.
allow_admin: #
http://nginx.org/en/docs/http/ngx_http_access_module.html#allow
- 0.0.0.0/0 # If we don't set any IP list, then any IP
access is allowed by default.
#- "::/64"
admin_listen: # use a separate port
ip: 0.0.0.0 # Specific IP, if not set, the default
value is `0.0.0.0`.
port: 9180 # Specific port, which must be different
from node_listen's port.
#https_admin: true # enable HTTPS when use a separate port
for Admin API.
# Admin API will use
conf/apisix_admin_api.crt and conf/apisix_admin_api.key as certificate.
admin_api_mtls: # Depends on `admin_listen` and
`https_admin`.
admin_ssl_cert: "" # Path of your self-signed server side
cert.
admin_ssl_cert_key: "" # Path of your self-signed server side key.
admin_ssl_ca_cert: "" # Path of your self-signed ca cert.The CA
is used to sign all admin api callers' certificates.
admin_api_version: v3 # The version of admin api, latest version
is v3.
etcd:
host: # it's possible to define multiple etcd
hosts addresses of the same etcd cluster.
- "https://10.72.8.66:2479"; # multiple etcd address, if your etcd
cluster enables TLS, please use https scheme,
- "https://10.72.8.67:2479"; # e.g. https://127.0.0.1:2379.
- "https://10.72.8.68:2479";
```
Here are some questions that I want to clarify:
1. When I use gRPC to connect to Etcd (configuration: `use_grpc: true`), I
expect this method to reduce the number of connections between APISIX and Etcd.
However, in reality, I found that for a single Nginx instance, there are still
many gRPC connections between it and Etcd
```
[root@knode10-72-8-63 logs]# ps -ef |grep nginx |grep worker
nobody 74123 74122 0 17:19 ? 00:00:01 nginx: worker process
nobody 74124 74122 0 17:19 ? 00:00:01 nginx: worker process
nobody 74125 74122 0 17:19 ? 00:00:01 nginx: worker process
nobody 74126 74122 0 17:19 ? 00:00:01 nginx: worker process
```
```
[root@knode10-72-8-63 logs]# netstat -apn |grep ESTABLISHED |grep nginx|grep
-v unix|grep 2479
tcp 0 0 10.72.8.63:52689 10.72.8.67:2479
ESTABLISHED 74124/nginx: worker
tcp 0 0 10.72.8.63:52695 10.72.8.67:2479
ESTABLISHED 74126/nginx: worker
tcp 0 0 10.72.8.63:64456 10.72.8.68:2479
ESTABLISHED 74124/nginx: worker
tcp 0 0 10.72.8.63:53235 10.72.8.67:2479
ESTABLISHED 74124/nginx: worker
tcp 0 0 10.72.8.63:40532 10.72.8.66:2479
ESTABLISHED 74124/nginx: worker
tcp 0 0 10.72.8.63:52685 10.72.8.67:2479
ESTABLISHED 74124/nginx: worker
tcp 0 0 10.72.8.63:52681 10.72.8.67:2479
ESTABLISHED 74126/nginx: worker
tcp 0 0 10.72.8.63:52751 10.72.8.67:2479
ESTABLISHED 74124/nginx: worker
tcp 0 0 10.72.8.63:52679 10.72.8.67:2479
ESTABLISHED 74126/nginx: worker
tcp 0 0 10.72.8.63:51521 10.72.8.67:2479
ESTABLISHED 74124/nginx: worker
tcp 0 0 10.72.8.63:40604 10.72.8.66:2479
ESTABLISHED 74123/nginx: worker
tcp 0 0 10.72.8.63:40588 10.72.8.66:2479
ESTABLISHED 74123/nginx: worker
tcp 0 0 10.72.8.63:40544 10.72.8.66:2479
ESTABLISHED 74126/nginx: worker
tcp 0 0 10.72.8.63:64418 10.72.8.68:2479
ESTABLISHED 74124/nginx: worker
tcp 0 0 10.72.8.63:40526 10.72.8.66:2479
ESTABLISHED 74124/nginx: worker
tcp 0 0 10.72.8.63:64430 10.72.8.68:2479
ESTABLISHED 74126/nginx: worker
tcp 0 0 10.72.8.63:40522 10.72.8.66:2479
ESTABLISHED 74124/nginx: worker
tcp 0 0 10.72.8.63:41104 10.72.8.66:2479
ESTABLISHED 74124/nginx: worker
tcp 0 0 10.72.8.63:52713 10.72.8.67:2479
ESTABLISHED 74123/nginx: worker
tcp 0 0 10.72.8.63:52723 10.72.8.67:2479
ESTABLISHED 74123/nginx: worker
tcp 0 0 10.72.8.63:40524 10.72.8.66:2479
ESTABLISHED 74124/nginx: worker
tcp 0 0 10.72.8.63:52741 10.72.8.67:2479
ESTABLISHED 74124/nginx: worker
tcp 0 0 10.72.8.63:64496 10.72.8.68:2479
ESTABLISHED 74124/nginx: worker
tcp 0 0 10.72.8.63:64498 10.72.8.68:2479
ESTABLISHED 74124/nginx: worker
tcp 0 0 10.72.8.63:40626 10.72.8.66:2479
ESTABLISHED 74124/nginx: worker
tcp 0 0 10.72.8.63:40610 10.72.8.66:2479
ESTABLISHED 74124/nginx: worker
tcp 0 0 10.72.8.63:64440 10.72.8.68:2479
ESTABLISHED 74126/nginx: worker
tcp 0 0 10.72.8.63:40618 10.72.8.66:2479
ESTABLISHED 74124/nginx: worker
tcp 0 0 10.72.8.63:52671 10.72.8.67:2479
ESTABLISHED 74126/nginx: worker
tcp 0 0 10.72.8.63:40554 10.72.8.66:2479
ESTABLISHED 74126/nginx: worker
tcp 0 0 10.72.8.63:40596 10.72.8.66:2479
ESTABLISHED 74123/nginx: worker
tcp 0 0 10.72.8.63:52711 10.72.8.67:2479
ESTABLISHED 74124/nginx: worker
tcp 0 0 10.72.8.63:64458 10.72.8.68:2479
ESTABLISHED 74124/nginx: worker
tcp 0 0 10.72.8.63:64422 10.72.8.68:2479
ESTABLISHED 74126/nginx: worker
tcp 0 0 10.72.8.63:39390 10.72.8.66:2479
ESTABLISHED 74124/nginx: worker
tcp 0 0 10.72.8.63:64410 10.72.8.68:2479
ESTABLISHED 74124/nginx: worker
tcp 0 0 10.72.8.63:64412 10.72.8.68:2479
ESTABLISHED 74124/nginx: worker
tcp 0 0 10.72.8.63:40608 10.72.8.66:2479
ESTABLISHED 74124/nginx: worker
tcp 0 0 10.72.8.63:52667 10.72.8.67:2479
ESTABLISHED 74124/nginx: worker
tcp 0 0 10.72.8.63:40546 10.72.8.66:2479
ESTABLISHED 74126/nginx: worker
tcp 0 0 10.72.8.63:64488 10.72.8.68:2479
ESTABLISHED 74124/nginx: worker
tcp 0 0 10.72.8.63:64474 10.72.8.68:2479
ESTABLISHED 74123/nginx: worker
tcp 0 0 10.72.8.63:52701 10.72.8.67:2479
ESTABLISHED 74124/nginx: worker
tcp 0 0 10.72.8.63:64472 10.72.8.68:2479
ESTABLISHED 74123/nginx: worker
tcp 0 0 10.72.8.63:64506 10.72.8.68:2479
ESTABLISHED 74124/nginx: worker
tcp 0 0 10.72.8.63:64444 10.72.8.68:2479
ESTABLISHED 74124/nginx: worker
tcp 0 0 10.72.8.63:64484 10.72.8.68:2479
ESTABLISHED 74123/nginx: worker
tcp 0 0 10.72.8.63:64450 10.72.8.68:2479
ESTABLISHED 74124/nginx: worker
tcp 0 0 10.72.8.63:52749 10.72.8.67:2479
ESTABLISHED 74124/nginx: worker
tcp 0 0 10.72.8.63:52709 10.72.8.67:2479
ESTABLISHED 74124/nginx: worker
tcp 0 0 10.72.8.63:52715 10.72.8.67:2479
ESTABLISHED 74123/nginx: worker
tcp 0 0 10.72.8.63:40534 10.72.8.66:2479
ESTABLISHED 74124/nginx: worker
tcp 0 0 10.72.8.63:40578 10.72.8.66:2479
ESTABLISHED 74124/nginx: worker
tcp 0 0 10.72.8.63:40598 10.72.8.66:2479
ESTABLISHED 74123/nginx: worker
tcp 0 0 10.72.8.63:64482 10.72.8.68:2479
ESTABLISHED 74123/nginx: worker
tcp 0 0 10.72.8.63:52699 10.72.8.67:2479
ESTABLISHED 74124/nginx: worker
tcp 0 0 10.72.8.63:64446 10.72.8.68:2479
ESTABLISHED 74126/nginx: worker
```
It is evident that there are only 4 Nginx processes in total, but each
process has a relatively high number of connections with Etcd. The crucial
point is that the process 74124 has 35 connections with Etcd, while the
processes 74123 and 74126 have only 11 connections each. There is a significant
difference between them. I would like to ask if this situation is expected or
if there might be an issue in my environment?
2. In APISIX 3.4.0 version, I noticed that it is possible to use an HTTP
long connection to connect Etcd (https://github.com/apache/apisix/pull/9456).
Indeed, compared to using gRPC, the number of connections has been greatly
reduced.
```
[root@knode10-72-8-63 operation]# netstat -apn |grep nginx|grep -v unix|grep
2479
tcp 0 0 10.72.8.63:18922 10.72.8.66:2479
ESTABLISHED 69001/nginx: worker
tcp 0 0 10.72.8.63:31047 10.72.8.67:2479
ESTABLISHED 69003/nginx: worker
tcp 0 0 10.72.8.63:18926 10.72.8.66:2479
ESTABLISHED 69004/nginx: worker
tcp 0 0 10.72.8.63:43182 10.72.8.68:2479
ESTABLISHED 69004/nginx: worker
tcp 0 0 10.72.8.63:42862 10.72.8.68:2479
ESTABLISHED 69002/nginx: worker
tcp 0 0 10.72.8.63:31427 10.72.8.67:2479
ESTABLISHED 69004/nginx: worker
tcp 0 0 10.72.8.63:29825 10.72.8.67:2479
ESTABLISHED 69002/nginx: worker
```
From the above, there are a total of 4 Nginx processes. However, each
process has a different number of connections with Etcd.
For example, processes 69001 and 69003 only have 1 connection, while
processes 69002 and 69004 have 2 connections. In reality, these connections are
dynamically changing. Sometimes, an Nginx process may not have any connection
with Etcd or even have 3 connections. I would like to ask what is going on here?
@kingluo Can you help me clarify the points of confusion mentioned above?
Thank you very much.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
