[GitHub] [apisix] tzssangglass commented on issue #6694: bug: using DNS discovery SRV and CHash algorithm can't work like node upstream type
tzssangglass commented on issue #6694: URL: https://github.com/apache/apisix/issues/6694#issuecomment-1175156189 > As the [service-lookups](https://www.consul.io/docs/discovery/dns#service-lookups) documentation said, `When DNS SRV responses are sent, the order is randomized`. Please see my latest verification process, I got the result in random order, but could not reproduce your problem. Now there are two ways. 1. you give the minimum reproducible use case based on my verification approach. 2. give the minimum reproducible exertion in your own way, but it needs to be clearer like the steps I described so that I can reproduce it step by step. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: notifications-unsubscr...@apisix.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [apisix] tzssangglass commented on issue #6694: bug: using DNS discovery SRV and CHash algorithm can't work like node upstream type
tzssangglass commented on issue #6694: URL: https://github.com/apache/apisix/issues/6694#issuecomment-1173319136 BTW, I used the master branch to test. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: notifications-unsubscr...@apisix.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [apisix] tzssangglass commented on issue #6694: bug: using DNS discovery SRV and CHash algorithm can't work like node upstream type
tzssangglass commented on issue #6694:
URL: https://github.com/apache/apisix/issues/6694#issuecomment-1173316698
> Which version of consul did you use in this case?
consul version
Consul v1.12.2
Revision 19041f20
> And when you dug the DNS search repeated, did consul return random `;;
ANSWER SECTION:` result?
no, maybe because I didn't add the `check` configuration.
> In addition, I recommend that you may try the test case again after the
discovery refreshes the DNS search result.
OK, I change step 2 preparing the consul DNS configuration, the consul DNS
configs like
```
$ cat consul.d/mock-2980.json
{
"service": {
"id": "mock-2980",
"name": "openresty",
"address": "127.0.0.1",
"port": 2980,
"check": {
"http": "http://127.0.0.1:2980/healthy;,
"interval": "3s",
"timeout": "1s"
}
}
}
$ cat consul.d/mock-2981.json
{
"service": {
"id": "mock-2981",
"name": "openresty",
"address": "127.0.0.1",
"port": 2981,
"check": {
"http": "http://127.0.0.1:2981/healthy;,
"interval": "3s",
"timeout": "1s"
}
}
}
$ cat consul.d/mock-2982.json
{
"service": {
"id": "mock-2982",
"name": "openresty",
"address": "127.0.0.1",
"port": 2982,
"check": {
"http": "http://127.0.0.1:2982/healthy;,
"interval": "3s",
"timeout": "1s"
}
}
}
```
and dig return random `ANSWER SECTION` result(order change), like:
```
$ dig @127.0.0.1 -p 8600 openresty.service.consul -t srv
; <<>> DiG 9.11.4-P2-RedHat-9.11.4-26.P2.el7_9.8 <<>> @127.0.0.1 -p 8600
openresty.service.consul -t srv
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 7328
;; flags: qr aa rd; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 7
;; WARNING: recursion requested but not available
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;openresty.service.consul. IN SRV
;; ANSWER SECTION:
openresty.service.consul. 0 IN SRV 1 1 2982
7f01.addr.dc1.consul.
openresty.service.consul. 0 IN SRV 1 1 2980
7f01.addr.dc1.consul.
openresty.service.consul. 0 IN SRV 1 1 2981
7f01.addr.dc1.consul.
;; ADDITIONAL SECTION:
7f01.addr.dc1.consul. 0 IN A 127.0.0.1
machine.node.dc1.consul. 0 IN TXT "consul-network-segment="
7f01.addr.dc1.consul. 0 IN A 127.0.0.1
machine.node.dc1.consul. 0 IN TXT "consul-network-segment="
7f01.addr.dc1.consul. 0 IN A 127.0.0.1
machine.node.dc1.consul. 0 IN TXT "consul-network-segment="
;; Query time: 0 msec
;; SERVER: 127.0.0.1#8600(127.0.0.1)
;; WHEN: Mon Jul 04 12:02:18 CST 2022
;; MSG SIZE rcvd: 354
$ dig @127.0.0.1 -p 8600 openresty.service.consul -t srv
; <<>> DiG 9.11.4-P2-RedHat-9.11.4-26.P2.el7_9.8 <<>> @127.0.0.1 -p 8600
openresty.service.consul -t srv
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 38061
;; flags: qr aa rd; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 7
;; WARNING: recursion requested but not available
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;openresty.service.consul. IN SRV
;; ANSWER SECTION:
openresty.service.consul. 0 IN SRV 1 1 2981
7f01.addr.dc1.consul.
openresty.service.consul. 0 IN SRV 1 1 2980
7f01.addr.dc1.consul.
openresty.service.consul. 0 IN SRV 1 1 2982
7f01.addr.dc1.consul.
;; ADDITIONAL SECTION:
7f01.addr.dc1.consul. 0 IN A 127.0.0.1
machine.node.dc1.consul. 0 IN TXT "consul-network-segment="
7f01.addr.dc1.consul. 0 IN A 127.0.0.1
machine.node.dc1.consul. 0 IN TXT "consul-network-segment="
7f01.addr.dc1.consul. 0 IN A 127.0.0.1
machine.node.dc1.consul. 0 IN TXT "consul-network-segment="
;; Query time: 0 msec
;; SERVER: 127.0.0.1#8600(127.0.0.1)
;; WHEN: Mon Jul 04 12:02:20 CST 2022
;; MSG SIZE rcvd: 354
```
then I repeated step 4 at different times, like:
```
$ curl http://127.0.0.1:9080/echo -H "Host: fatpa.me"
2982
$ curl http://127.0.0.1:9080/echo -H "Host: fatpa.me"
2982
$ curl http://127.0.0.1:9080/echo -H "Host: fatpa.me"
2982
```
still no recurrence.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail:
[GitHub] [apisix] tzssangglass commented on issue #6694: bug: using DNS discovery SRV and CHash algorithm can't work like node upstream type
tzssangglass commented on issue #6694:
URL: https://github.com/apache/apisix/issues/6694#issuecomment-1173017986
OK, I retried and now I can construct upstream and DNS query results similar
to your description.
1. prepare upstream
I use openresty to simulate upstream, and nginx.conf is configured as
follows:
```
master_process on;
worker_processes 2;
error_log logs/error.log warn;
pid logs/nginx.pid;
worker_rlimit_nofile 20480;
events {
accept_mutex off;
worker_connections 10620;
}
worker_shutdown_timeout 3;
http {
server {
listen 2980;
listen 2981;
listen 2982;
access_log off;
location / {
content_by_lua_block {
local port = ngx.var.server_port
ngx.say(port)
}
}
}
}
```
access any of the ports it listens to and it will return that port, e.g.:
```
$ curl http://127.0.0.1:2980/test
2980
$ curl http://127.0.0.1:2981/test
2981
$ curl http://127.0.0.1:2982/test
2982
```
2. preparing the consul DNS configuration
```
$ ls ./consul.d
mock-2980.json mock-2981.json mock-2982.json
$ cat consul.d/mock-2980.json
{
"service": {
"id": "mock-2980",
"name": "openresty",
"address": "127.0.0.1",
"port": 2980
}
}
$ cat consul.d/mock-2981.json
{
"service": {
"id": "mock-2981",
"name": "openresty",
"address": "127.0.0.1",
"port": 2981
}
}
$ cat consul.d/mock-2982.json
{
"service": {
"id": "mock-2982",
"name": "openresty",
"address": "127.0.0.1",
"port": 2982
}
}
```
and use
```
consul agent -dev -config-dir=./consul.d -node=machine
```
to start consul DNS. service, and test dns search, like:
```
dig @127.0.0.1 -p 8600 openresty.service.consul -t srv
; <<>> DiG 9.11.4-P2-RedHat-9.11.4-26.P2.el7_9.8 <<>> @127.0.0.1 -p 8600
openresty.service.consul -t srv
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 42772
;; flags: qr aa rd; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 7
;; WARNING: recursion requested but not available
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;openresty.service.consul. IN SRV
;; ANSWER SECTION:
openresty.service.consul. 0 IN SRV 1 1 2981
7f01.addr.dc1.consul.
openresty.service.consul. 0 IN SRV 1 1 2980
7f01.addr.dc1.consul.
openresty.service.consul. 0 IN SRV 1 1 2982
7f01.addr.dc1.consul.
;; ADDITIONAL SECTION:
7f01.addr.dc1.consul. 0 IN A 127.0.0.1
machine.node.dc1.consul. 0 IN TXT "consul-network-segment="
7f01.addr.dc1.consul. 0 IN A 127.0.0.1
machine.node.dc1.consul. 0 IN TXT "consul-network-segment="
7f01.addr.dc1.consul. 0 IN A 127.0.0.1
machine.node.dc1.consul. 0 IN TXT "consul-network-segment="
;; Query time: 0 msec
;; SERVER: 127.0.0.1#8600(127.0.0.1)
;; WHEN: Sun Jul 03 13:48:54 CST 2022
;; MSG SIZE rcvd: 354
```
3. start APISIX, the config.yaml like
```
apisix:
admin_key:
- name: admin
key: edd1c9f034335f136f87ad84b625c8f1 # using fixed API token has
security risk, please update it when you deploy to production environment
role: admin
discovery:
dns:
servers:
- "127.0.0.1:8600"
```
4. set route and upstream
upstream
```
curl http://127.0.0.1:9080/apisix/admin/upstreams/1 -H 'X-API-KEY:
edd1c9f034335f136f87ad84b625c8f1' -X PUT -d '
{
"timeout": {
"connect": 6,
"send": 6,
"read": 6
},
"type": "chash",
"hash_on": "vars",
"key": "request_uri",
"scheme": "http",
"discovery_type": "dns",
"pass_host": "pass",
"name": "dns_tester",
"service_name": "openresty.service.consul",
"keepalive_pool": {
"idle_timeout": 60,
"requests": 1000,
"size": 320
}
}'
```
route
```
curl http://127.0.0.1:9080/apisix/admin/routes/1 -H 'X-API-KEY:
edd1c9f034335f136f87ad84b625c8f1' -X PUT -d '
{
"uri": "/*",
"name": "Dns_Tester",
"methods": [
"GET",
"POST",
"PUT",
"DELETE",
"PATCH",
"HEAD",
"OPTIONS",
"CONNECT",
"TRACE"
],
"host": "fatpa.me",
"upstream_id": "1",
"status": 1
}'
```
5. test
```
curl http://127.0.0.1:9080/echo -H "Host: fatpa.me"
2982
$ curl http://127.0.0.1:9080/echo -H "Host:
[GitHub] [apisix] tzssangglass commented on issue #6694: bug: using DNS discovery SRV and CHash algorithm can't work like node upstream type
tzssangglass commented on issue #6694: URL: https://github.com/apache/apisix/issues/6694#issuecomment-1172936488 Hi @Fatpa, I would like to register some service, like the scenario you describe. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: notifications-unsubscr...@apisix.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [apisix] tzssangglass commented on issue #6694: bug: using DNS discovery SRV and CHash algorithm can't work like node upstream type
tzssangglass commented on issue #6694:
URL: https://github.com/apache/apisix/issues/6694#issuecomment-1172936393
I tried to install consul, here are some steps:
1. install
```
sudo yum install -y yum-utils
sudo yum-config-manager --add-repo
https://rpm.releases.hashicorp.com/RHEL/hashicorp.repo
sudo yum -y install consul
```
2. start consul
```
consul agent -dev
```
3. register service
```
# register
curl http://127.0.0.1:8500/v1/agent/service/register -X PUT -d '
{
"id":"pushgateway-9091",
"name":"pushgateway",
"address":"127.0.0.1",
"port":9091,
"check":{
"http":"http://1.0.0.1:9091/-/healthy;,
"interval":"10s",
"timeout":"3s"
}
}'
# check
curl http://127.0.0.1:8500/v1/agent/service/pushgateway-9091
{
"ID": "pushgateway-9091",
"Service": "pushgateway",
"Tags": [],
"Meta": {},
"Port": 9091,
"Address": "127.0.0.1",
"TaggedAddresses": {
"lan_ipv4": {
"Address": "127.0.0.1",
"Port": 9091
},
"wan_ipv4": {
"Address": "127.0.0.1",
"Port": 9091
}
},
"Weights": {
"Passing": 1,
"Warning": 1
},
"EnableTagOverride": false,
"ContentHash": "2f65415818755fba",
"Datacenter": "dc1"
}
```
4. fetch service by dig
```
dig @127.0.0.1 -p 8600 pushgateway.service.dc1.consul -t srv
; <<>> DiG 9.11.4-P2-RedHat-9.11.4-26.P2.el7_9.8 <<>> @127.0.0.1 -p 8600
pushgateway.service.dc1.consul -t srv
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 58617
;; flags: qr aa rd; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1
;; WARNING: recursion requested but not available
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;pushgateway.service.dc1.consul. IN SRV
;; AUTHORITY SECTION:
consul. 0 IN SOA ns.consul.
hostmaster.consul. 1656784437 3600 600 86400 0
;; Query time: 0 msec
;; SERVER: 127.0.0.1#8600(127.0.0.1)
;; WHEN: Sun Jul 03 01:53:57 CST 2022
;; MSG SIZE rcvd: 109
```
Here's how I should get to the service I just registered via dig, I tried a
few ways by got no answer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscr...@apisix.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [apisix] tzssangglass commented on issue #6694: bug: using DNS discovery SRV and CHash algorithm can't work like node upstream type
tzssangglass commented on issue #6694: URL: https://github.com/apache/apisix/issues/6694#issuecomment-1148128749 And I would like to know how to build the Consul DNS Service you mentioned to reproduce it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: notifications-unsubscr...@apisix.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [apisix] tzssangglass commented on issue #6694: bug: using DNS discovery SRV and CHash algorithm can't work like node upstream type
tzssangglass commented on issue #6694: URL: https://github.com/apache/apisix/issues/6694#issuecomment-1148128514 > If all the points can visit, this test case can't reproduce my problem. > My problem is the same request_uri var should visit the same endpoint. I know your problem. You misunderstood, the key in my test case is `device_id`, not `request_uri`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: notifications-unsubscr...@apisix.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [apisix] tzssangglass commented on issue #6694: bug: using DNS discovery SRV and CHash algorithm can't work like node upstream type
tzssangglass commented on issue #6694: URL: https://github.com/apache/apisix/issues/6694#issuecomment-1147426733 > And I think maybe you should reproduce it with Consul DNS Service instead of CoreDNS. Have you tried this: https://github.com/apache/apisix/blob/master/docs/en/latest/discovery/consul_kv.md -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: notifications-unsubscr...@apisix.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [apisix] tzssangglass commented on issue #6694: bug: using DNS discovery SRV and CHash algorithm can't work like node upstream type
tzssangglass commented on issue #6694: URL: https://github.com/apache/apisix/issues/6694#issuecomment-1147424137 > According to your test cases, `127.0.0.2` can't visit, right? No, in my test case, `127.0.0.2` can be visited. `127.0.0.1` ~ `127.0.0.8` all point to the local loopback. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: notifications-unsubscr...@apisix.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [apisix] tzssangglass commented on issue #6694: bug: using DNS discovery SRV and CHash algorithm can't work like node upstream type
tzssangglass commented on issue #6694:
URL: https://github.com/apache/apisix/issues/6694#issuecomment-1078623920
Here are my reproduction steps:
1. start DNS server
execute this script
https://github.com/apache/apisix/blob/master/utils/set-dns.sh
2. verify SRV records
```shell
$ dig port.srv.test.local @127.0.0.1 -p 1053 -t srv
; <<>> DiG 9.10.6 <<>> port.srv.test.local @127.0.0.1 -p 1053 -t srv
;; global options: +cmd
;; Got answer:
;; WARNING: .local is reserved for Multicast DNS
;; You are currently testing what happens when an mDNS query is leaked to DNS
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 9039
;; flags: qr aa rd; QUERY: 1, ANSWER: 2, AUTHORITY: 2, ADDITIONAL: 3
;; WARNING: recursion requested but not available
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;port.srv.test.local. IN SRV
;; ANSWER SECTION:
port.srv.test.local.86400 IN SRV 10 60 1980 a.test.local.
port.srv.test.local.86400 IN SRV 10 20 1981 b.test.local.
;; AUTHORITY SECTION:
test.local. 3600IN NS a.iana-servers.net.
test.local. 3600IN NS b.iana-servers.net.
;; ADDITIONAL SECTION:
a.test.local. 1 IN A 127.0.0.1
b.test.local. 1 IN A 127.0.0.2
;; Query time: 0 msec
;; SERVER: 127.0.0.1#1053(127.0.0.1)
;; WHEN: Fri Mar 25 11:38:48 CST 2022
;; MSG SIZE rcvd: 290
```
3. test cases
```
use t::APISIX 'no_plan';
repeat_each(1);
worker_connections(1024);
no_root_location();
no_shuffle();
add_block_preprocessor(sub {
my ($block) = @_;
if (!$block->request) {
$block->set_value("request", "GET /t");
}
if ((!defined $block->error_log) && (!defined $block->no_error_log)) {
$block->set_value("no_error_log", "[error]");
}
});
run_tests();
__DATA__
=== TEST 1: add route
--- config
location /t {
content_by_lua_block {
local t = require("lib.test_admin").test
local code, body = t('/apisix/admin/routes/1',
ngx.HTTP_PUT,
[[{
"upstream": {
"service_name": "port.srv.test.local",
"discovery_type": "dns",
"type": "chash",
"key": "arg_device_id"
},
"uri": "/echo"
}]]
)
if code >= 300 then
ngx.status = code
end
ngx.say(body)
}
}
--- response_body
passed
=== TEST 2: proxy request to upstream(127.0.0.1:1980) by chash
key(device_id=1)
--- yaml_config
discovery:
dns:
servers:
- "127.0.0.1:1053"
--- config
location /t {
content_by_lua_block {
local http = require "resty.http"
local t = {}
local ids = {}
for i = 1, 180 do
local th = assert(ngx.thread.spawn(function()
local httpc = http.new()
local uri = "http://127.0.0.1:; .. ngx.var.server_port
.. "/echo?device_id=1"
local res, err = httpc:request_uri(uri,
{
method = "GET",
headers = {
["Content-Type"] = "application/json",
}
}
)
if not res then
ngx.log(ngx.ERR, err)
return
end
end, i))
table.insert(t, th)
end
for i, th in ipairs(t) do
ngx.thread.wait(th)
end
ngx.say("true")
}
}
--- wait: 5
--- response_body
true
--- no_error_log
proxy request to 127.0.0.2:1981
=== TEST 3: proxy request to upstream(127.0.0.2:1981) by chash
key(device_id=gagalgeobeotbhrtnfrnsrt)
--- yaml_config
discovery:
dns:
servers:
- "127.0.0.1:1053"
--- config
location /t {
content_by_lua_block {
local http = require "resty.http"
local t = {}
local ids = {}
for i = 1, 180 do
local th = assert(ngx.thread.spawn(function()
local httpc = http.new()
local uri = "http://127.0.0.1:; .. ngx.var.server_port
.. "/echo?device_id=gagalgeobeotbhrtnfrnsrt"
[GitHub] [apisix] tzssangglass commented on issue #6694: bug: using DNS discovery SRV and CHash algorithm can't work like node upstream type
tzssangglass commented on issue #6694: URL: https://github.com/apache/apisix/issues/6694#issuecomment-1077028329 hi @Fatpa pls describe this step in detail and paste the configuration of route and upstream. I am unable to reproduce. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: notifications-unsubscr...@apisix.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
