Re: [SR-Users] DMQ problems
Hello Henning,
Thanks for precious indications!
I've tried to disable dns caching but nothing changed
Also captured everything on the network interface, but nothing discovered!
I've changed also dns server (from kube DNs to core dns) without success.
I'll try to reproduce in vm environment with power dns and capturing also
on the dns side
Il Gio 2 Ago 2018, 20:40 Henning Westerholt ha scritto:
> Am Donnerstag, 2. August 2018, 20:33:28 CEST schrieb Paolo Visintin -
> evosip.cloud:
> > I am able to see
> > 0(710) DEBUG: dmq [notification_peer.c:240]: get_dmq_host_list(): adding
> > DMQ node A host dmq-router-service=sip:42.100.109.113:5062
> >
> > but in tcpdump no DNS query
> >
> > Also doing a loop of "kamcmd dns.lookup A dmq-router-service.paolo.svc.c
> > luster.local" and starting kamailio at the same time the result is always
> > the correct ip, no 42.100.109.113 coming up so this is very strange!
>
> Hello,
>
> without looking too much into that, I have two remarks which may worth
> investigating:
>
> - for DNS names in the configuration file Kamailio does a lookup during
> startup and "compiles" this to structure which is evaluated during run time
>
> - there is also a DNS caching functionality, which could cause something
> like
> you observed. You can deactivate this with a core variable, I think.
>
> Best regards,
>
> Henning
>
>
> > 2018-08-02 15:29 GMT+02:00 Charles Chance >:
> > > Hi Paolo,
> > >
> > > kamailio "resolves" A host but does not query the DNS
> > >
> > >
> > > What do you mean by this?
> > >
> > > I am not overly familiar with Kamailio's DNS internals, but what do you
> > > see if you run the following commands?
> > >
> > > kamcmd dns.lookup SRV
> _sip._udp.dmq-router-service.paolo.svc.cluster.local
> > >
> > > kamcmd dns.lookup A dmq-router-service.paolo.svc.cluster.local
> > >
> > > Cheers,
> > >
> > > Charles
> > >
> > >
> > > On 2 August 2018 at 13:27, Paolo Visintin - evosip.cloud <
> > >
> > > paolo.visintin@evosip.cloud> wrote:
> > >> Hello again!
> > >> Another interesting thing found running in debug=3
> > >>
> > >> 0(710) DEBUG: dmq [notification_peer.c:240]: get_dmq_host_list():
> adding
> > >> DMQ node A host dmq-router-service=sip:42.100.109.113:5062
> > >>
> > >> so , kamailio "resolves" A host but does not query the DNS
> > >> this is dump of dns request inside kamailio container:
> > >>
> > >> 147 0.597474 10.42.1.251 → 10.96.0.10 DNS 102 Standard query
> 0xf31d
> > >> SRV dmq-router-service.paolo.svc.cluster.local
> > >> 151 0.598306 10.96.0.10 → 10.42.1.251 DNS 155 Standard query
> > >> response 0xf31d SRV dmq-router-service.paolo.svc.cluster.local SRV 10
> > >> 100 0 3934376365383035.dmq-router-service.paolo.svc.cluster.local A
> > >> 172.28.0.240
> > >> 156 0.598538 10.42.1.251 → 10.96.0.10 DNS 102 Standard query
> 0x1904
> > >> A dmq-router-service.paolo.svc.cluster.local
> > >> 163 0.599275 10.96.0.10 → 10.42.1.251 DNS 118 Standard query
> > >> response 0x1904 A dmq-router-service.paolo.svc.cluster.local A
> > >> 172.28.0.240
> > >>
> > >>
> > >>
> > >> *Paolo Visintin*
> > >> *CTO*
> > >> evosip.cloud
> > >> [image: Risultati immagini per evosip]
> > >>
> > >> 2018-08-02 0:24 GMT+02:00 Paolo Visintin - evosip.cloud <
> > >>
> > >> paolo.visintin@evosip.cloud>:
> > >>> Hello Charles,
> > >>> sorry for late reply, we've had some issues rebuilding docker images
> > >>> with kamailo nightly build!
> > >>>
> > >>> We are running on our local, private on premise kubernetes cluster so
> > >>> connections from outside is simply impossible (in dev we are using a
> > >>> "closed" network system)
> > >>>
> > >>> I've just tried to :
> > >>> - open a tcpdump trace in the instance tracing only the DMQ port
> defined
> > >>> [5062] - (tcpdump -i eth0 port 5062)
> > >>> - packet coming from nowhere
> > >>> - started kamailio ($kamailio -f $PATH_KAMAILIO_CFG -m 512 -DD -E -e)
> > >>> - seeing this in tcpdump :
> > >>>
> > >>> root@router-0:/# tcpdump -i eth0 port 5062
> > >>> tcpdump: verbose output suppressed, use -v or -vv for full protocol
> > >>> decode
> > >>> listening on eth0, link-type EN10MB (Ethernet), capture size 262144
> > >>> bytes
> > >>> 22:06:44.966991 IP router-0.router.paolo.svc.cluster.local.5062 >
> > >>> 42.100.109.113.5062: UDP, length 497
> > >>> 22:06:45.572084 IP router-0.router.paolo.svc.cluster.local.5062 >
> > >>> 42.100.109.113.5062: UDP, length 497
> > >>> 22:06:46.571875 IP router-0.router.paolo.svc.cluster.local.5062 >
> > >>> 42.100.109.113.5062: UDP, length 497
> > >>>
> > >>> so seems that kamailio is trying to contact the foreign ip
> > >>>
> > >>> if i check inside the configuration:
> > >>> modparam("dmq", "server_address", DMQ_SERVER_ADDRESS)
> > >>> modparam("dmq", "notification_address", DMQ_NOTIFICATION_ADDRESS)
> > >>> modparam("dmq", "multi_notify", 1)
> > >>> modparam("dmq", "num_workers", 4)
> > >>> modparam("dmq", "ping_interval", 15)
> > >>>
> > >>>
> > >>> ...
> > >>>
> > >>> if(is_method("KDMQ")
Re: [SR-Users] DMQ problems
Am Donnerstag, 2. August 2018, 20:33:28 CEST schrieb Paolo Visintin -
evosip.cloud:
> I am able to see
> 0(710) DEBUG: dmq [notification_peer.c:240]: get_dmq_host_list(): adding
> DMQ node A host dmq-router-service=sip:42.100.109.113:5062
>
> but in tcpdump no DNS query
>
> Also doing a loop of "kamcmd dns.lookup A dmq-router-service.paolo.svc.c
> luster.local" and starting kamailio at the same time the result is always
> the correct ip, no 42.100.109.113 coming up so this is very strange!
Hello,
without looking too much into that, I have two remarks which may worth
investigating:
- for DNS names in the configuration file Kamailio does a lookup during
startup and "compiles" this to structure which is evaluated during run time
- there is also a DNS caching functionality, which could cause something like
you observed. You can deactivate this with a core variable, I think.
Best regards,
Henning
> 2018-08-02 15:29 GMT+02:00 Charles Chance :
> > Hi Paolo,
> >
> > kamailio "resolves" A host but does not query the DNS
> >
> >
> > What do you mean by this?
> >
> > I am not overly familiar with Kamailio's DNS internals, but what do you
> > see if you run the following commands?
> >
> > kamcmd dns.lookup SRV _sip._udp.dmq-router-service.paolo.svc.cluster.local
> >
> > kamcmd dns.lookup A dmq-router-service.paolo.svc.cluster.local
> >
> > Cheers,
> >
> > Charles
> >
> >
> > On 2 August 2018 at 13:27, Paolo Visintin - evosip.cloud <
> >
> > paolo.visintin@evosip.cloud> wrote:
> >> Hello again!
> >> Another interesting thing found running in debug=3
> >>
> >> 0(710) DEBUG: dmq [notification_peer.c:240]: get_dmq_host_list(): adding
> >> DMQ node A host dmq-router-service=sip:42.100.109.113:5062
> >>
> >> so , kamailio "resolves" A host but does not query the DNS
> >> this is dump of dns request inside kamailio container:
> >>
> >> 147 0.597474 10.42.1.251 → 10.96.0.10 DNS 102 Standard query 0xf31d
> >> SRV dmq-router-service.paolo.svc.cluster.local
> >> 151 0.598306 10.96.0.10 → 10.42.1.251 DNS 155 Standard query
> >> response 0xf31d SRV dmq-router-service.paolo.svc.cluster.local SRV 10
> >> 100 0 3934376365383035.dmq-router-service.paolo.svc.cluster.local A
> >> 172.28.0.240
> >> 156 0.598538 10.42.1.251 → 10.96.0.10 DNS 102 Standard query 0x1904
> >> A dmq-router-service.paolo.svc.cluster.local
> >> 163 0.599275 10.96.0.10 → 10.42.1.251 DNS 118 Standard query
> >> response 0x1904 A dmq-router-service.paolo.svc.cluster.local A
> >> 172.28.0.240
> >>
> >>
> >>
> >> *Paolo Visintin*
> >> *CTO*
> >> evosip.cloud
> >> [image: Risultati immagini per evosip]
> >>
> >> 2018-08-02 0:24 GMT+02:00 Paolo Visintin - evosip.cloud <
> >>
> >> paolo.visintin@evosip.cloud>:
> >>> Hello Charles,
> >>> sorry for late reply, we've had some issues rebuilding docker images
> >>> with kamailo nightly build!
> >>>
> >>> We are running on our local, private on premise kubernetes cluster so
> >>> connections from outside is simply impossible (in dev we are using a
> >>> "closed" network system)
> >>>
> >>> I've just tried to :
> >>> - open a tcpdump trace in the instance tracing only the DMQ port defined
> >>> [5062] - (tcpdump -i eth0 port 5062)
> >>> - packet coming from nowhere
> >>> - started kamailio ($kamailio -f $PATH_KAMAILIO_CFG -m 512 -DD -E -e)
> >>> - seeing this in tcpdump :
> >>>
> >>> root@router-0:/# tcpdump -i eth0 port 5062
> >>> tcpdump: verbose output suppressed, use -v or -vv for full protocol
> >>> decode
> >>> listening on eth0, link-type EN10MB (Ethernet), capture size 262144
> >>> bytes
> >>> 22:06:44.966991 IP router-0.router.paolo.svc.cluster.local.5062 >
> >>> 42.100.109.113.5062: UDP, length 497
> >>> 22:06:45.572084 IP router-0.router.paolo.svc.cluster.local.5062 >
> >>> 42.100.109.113.5062: UDP, length 497
> >>> 22:06:46.571875 IP router-0.router.paolo.svc.cluster.local.5062 >
> >>> 42.100.109.113.5062: UDP, length 497
> >>>
> >>> so seems that kamailio is trying to contact the foreign ip
> >>>
> >>> if i check inside the configuration:
> >>> modparam("dmq", "server_address", DMQ_SERVER_ADDRESS)
> >>> modparam("dmq", "notification_address", DMQ_NOTIFICATION_ADDRESS)
> >>> modparam("dmq", "multi_notify", 1)
> >>> modparam("dmq", "num_workers", 4)
> >>> modparam("dmq", "ping_interval", 15)
> >>>
> >>>
> >>> ...
> >>>
> >>> if(is_method("KDMQ") && dst_port == DMQ_PORT){
> >>>
> >>> dmq_handle_message();
> >>> exit;
> >>>
> >>> }
> >>>
> >>> and in DMQ_NOTIFICATION_ADDRESS I have :
> >>> #!define DMQ_NOTIFICATION_ADDRESS "sip:dmq-router-service:5062"
> >>>
> >>> dmq-router-service DNS resolves:
> >>> > dmq-router-service.paolo.svc.cluster.local
> >>>
> >>> Server: 10.96.0.10
> >>> Address: 10.96.0.10#53
> >>>
> >>> Non-authoritative answer:
> >>> Name: dmq-router-service.paolo.svc.cluster.local
> >>> Address: 172.28.0.240
> >>>
> >>>
> >>> so -nowhere- i am able to find 42.100.109.113
> >>>
> >>>
> >>> and in kamailio log:
> >>>
Re: [SR-Users] DMQ problems
Hi Charles,
when I say kamailio "resolves" but does not query DNS I mean that in DEBUG
I am able to see
0(710) DEBUG: dmq [notification_peer.c:240]: get_dmq_host_list(): adding
DMQ node A host dmq-router-service=sip:42.100.109.113:5062
but in tcpdump no DNS query
Also doing a loop of "kamcmd dns.lookup A dmq-router-service.paolo.svc.c
luster.local" and starting kamailio at the same time the result is always
the correct ip, no 42.100.109.113 coming up so this is very strange!
Cheers
*Paolo Visintin*
*CTO*
evosip.cloud
[image: Risultati immagini per evosip]
2018-08-02 15:29 GMT+02:00 Charles Chance :
> Hi Paolo,
>
> kamailio "resolves" A host but does not query the DNS
>
>
> What do you mean by this?
>
> I am not overly familiar with Kamailio's DNS internals, but what do you
> see if you run the following commands?
>
> kamcmd dns.lookup SRV _sip._udp.dmq-router-service.paolo.svc.cluster.local
>
> kamcmd dns.lookup A dmq-router-service.paolo.svc.cluster.local
>
> Cheers,
>
> Charles
>
>
> On 2 August 2018 at 13:27, Paolo Visintin - evosip.cloud <
> paolo.visintin@evosip.cloud> wrote:
>
>> Hello again!
>> Another interesting thing found running in debug=3
>>
>> 0(710) DEBUG: dmq [notification_peer.c:240]: get_dmq_host_list(): adding
>> DMQ node A host dmq-router-service=sip:42.100.109.113:5062
>>
>> so , kamailio "resolves" A host but does not query the DNS
>> this is dump of dns request inside kamailio container:
>>
>> 147 0.597474 10.42.1.251 → 10.96.0.10 DNS 102 Standard query 0xf31d
>> SRV dmq-router-service.paolo.svc.cluster.local
>> 151 0.598306 10.96.0.10 → 10.42.1.251 DNS 155 Standard query
>> response 0xf31d SRV dmq-router-service.paolo.svc.cluster.local SRV 10
>> 100 0 3934376365383035.dmq-router-service.paolo.svc.cluster.local A
>> 172.28.0.240
>> 156 0.598538 10.42.1.251 → 10.96.0.10 DNS 102 Standard query 0x1904
>> A dmq-router-service.paolo.svc.cluster.local
>> 163 0.599275 10.96.0.10 → 10.42.1.251 DNS 118 Standard query
>> response 0x1904 A dmq-router-service.paolo.svc.cluster.local A
>> 172.28.0.240
>>
>>
>>
>> *Paolo Visintin*
>> *CTO*
>> evosip.cloud
>> [image: Risultati immagini per evosip]
>>
>> 2018-08-02 0:24 GMT+02:00 Paolo Visintin - evosip.cloud <
>> paolo.visintin@evosip.cloud>:
>>
>>>
>>> Hello Charles,
>>> sorry for late reply, we've had some issues rebuilding docker images
>>> with kamailo nightly build!
>>>
>>> We are running on our local, private on premise kubernetes cluster so
>>> connections from outside is simply impossible (in dev we are using a
>>> "closed" network system)
>>>
>>> I've just tried to :
>>> - open a tcpdump trace in the instance tracing only the DMQ port defined
>>> [5062] - (tcpdump -i eth0 port 5062)
>>> - packet coming from nowhere
>>> - started kamailio ($kamailio -f $PATH_KAMAILIO_CFG -m 512 -DD -E -e)
>>> - seeing this in tcpdump :
>>>
>>> root@router-0:/# tcpdump -i eth0 port 5062
>>> tcpdump: verbose output suppressed, use -v or -vv for full protocol
>>> decode
>>> listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
>>> 22:06:44.966991 IP router-0.router.paolo.svc.cluster.local.5062 >
>>> 42.100.109.113.5062: UDP, length 497
>>> 22:06:45.572084 IP router-0.router.paolo.svc.cluster.local.5062 >
>>> 42.100.109.113.5062: UDP, length 497
>>> 22:06:46.571875 IP router-0.router.paolo.svc.cluster.local.5062 >
>>> 42.100.109.113.5062: UDP, length 497
>>>
>>> so seems that kamailio is trying to contact the foreign ip
>>>
>>> if i check inside the configuration:
>>> modparam("dmq", "server_address", DMQ_SERVER_ADDRESS)
>>> modparam("dmq", "notification_address", DMQ_NOTIFICATION_ADDRESS)
>>> modparam("dmq", "multi_notify", 1)
>>> modparam("dmq", "num_workers", 4)
>>> modparam("dmq", "ping_interval", 15)
>>>
>>>
>>> ...
>>>
>>> if(is_method("KDMQ") && dst_port == DMQ_PORT){
>>> dmq_handle_message();
>>> exit;
>>> }
>>>
>>>
>>> and in DMQ_NOTIFICATION_ADDRESS I have :
>>> #!define DMQ_NOTIFICATION_ADDRESS "sip:dmq-router-service:5062"
>>>
>>> dmq-router-service DNS resolves:
>>> > dmq-router-service.paolo.svc.cluster.local
>>> Server: 10.96.0.10
>>> Address: 10.96.0.10#53
>>>
>>> Non-authoritative answer:
>>> Name: dmq-router-service.paolo.svc.cluster.local
>>> Address: 172.28.0.240
>>>
>>>
>>> so -nowhere- i am able to find 42.100.109.113
>>>
>>>
>>> and in kamailio log:
>>> 13(821) ERROR: dmq [notification_peer.c:599]:
>>> notification_resp_callback_f(): deleting server sip:42.100.109.113:5062
>>> because of failed request
>>>
>>>
>>>
>>> Cheers,
>>>
>>> Paolo
>>>
>>> 2018-07-26 18:14 GMT+02:00 Charles Chance >> >:
>>>
Hi Aleksandar,
The initial depopulation of the nodes (following a period of 'pending'
state) is due to no response being received from them. Are you able to
trace the messages to/from one of them to confirm what is happening there?
As for the unrecognised IP, I'm afraid I can't answer that one. It must
be coming from somewhere - it's just a case
Re: [SR-Users] DMQ problems
Hi Paolo,
kamailio "resolves" A host but does not query the DNS
What do you mean by this?
I am not overly familiar with Kamailio's DNS internals, but what do you see
if you run the following commands?
kamcmd dns.lookup SRV _sip._udp.dmq-router-service.paolo.svc.cluster.local
kamcmd dns.lookup A dmq-router-service.paolo.svc.cluster.local
Cheers,
Charles
On 2 August 2018 at 13:27, Paolo Visintin - evosip.cloud <
paolo.visintin@evosip.cloud> wrote:
> Hello again!
> Another interesting thing found running in debug=3
>
> 0(710) DEBUG: dmq [notification_peer.c:240]: get_dmq_host_list(): adding
> DMQ node A host dmq-router-service=sip:42.100.109.113:5062
>
> so , kamailio "resolves" A host but does not query the DNS
> this is dump of dns request inside kamailio container:
>
> 147 0.597474 10.42.1.251 → 10.96.0.10 DNS 102 Standard query 0xf31d
> SRV dmq-router-service.paolo.svc.cluster.local
> 151 0.598306 10.96.0.10 → 10.42.1.251 DNS 155 Standard query response
> 0xf31d SRV dmq-router-service.paolo.svc.cluster.local SRV 10 100 0
> 3934376365383035.dmq-router-service.paolo.svc.cluster.local A 172.28.0.240
> 156 0.598538 10.42.1.251 → 10.96.0.10 DNS 102 Standard query 0x1904 A
> dmq-router-service.paolo.svc.cluster.local
> 163 0.599275 10.96.0.10 → 10.42.1.251 DNS 118 Standard query response
> 0x1904 A dmq-router-service.paolo.svc.cluster.local A 172.28.0.240
>
>
>
> *Paolo Visintin*
> *CTO*
> evosip.cloud
> [image: Risultati immagini per evosip]
>
> 2018-08-02 0:24 GMT+02:00 Paolo Visintin - evosip.cloud <
> paolo.visintin@evosip.cloud>:
>
>>
>> Hello Charles,
>> sorry for late reply, we've had some issues rebuilding docker images with
>> kamailo nightly build!
>>
>> We are running on our local, private on premise kubernetes cluster so
>> connections from outside is simply impossible (in dev we are using a
>> "closed" network system)
>>
>> I've just tried to :
>> - open a tcpdump trace in the instance tracing only the DMQ port defined
>> [5062] - (tcpdump -i eth0 port 5062)
>> - packet coming from nowhere
>> - started kamailio ($kamailio -f $PATH_KAMAILIO_CFG -m 512 -DD -E -e)
>> - seeing this in tcpdump :
>>
>> root@router-0:/# tcpdump -i eth0 port 5062
>> tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
>> listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
>> 22:06:44.966991 IP router-0.router.paolo.svc.cluster.local.5062 >
>> 42.100.109.113.5062: UDP, length 497
>> 22:06:45.572084 IP router-0.router.paolo.svc.cluster.local.5062 >
>> 42.100.109.113.5062: UDP, length 497
>> 22:06:46.571875 IP router-0.router.paolo.svc.cluster.local.5062 >
>> 42.100.109.113.5062: UDP, length 497
>>
>> so seems that kamailio is trying to contact the foreign ip
>>
>> if i check inside the configuration:
>> modparam("dmq", "server_address", DMQ_SERVER_ADDRESS)
>> modparam("dmq", "notification_address", DMQ_NOTIFICATION_ADDRESS)
>> modparam("dmq", "multi_notify", 1)
>> modparam("dmq", "num_workers", 4)
>> modparam("dmq", "ping_interval", 15)
>>
>>
>> ...
>>
>> if(is_method("KDMQ") && dst_port == DMQ_PORT){
>> dmq_handle_message();
>> exit;
>> }
>>
>>
>> and in DMQ_NOTIFICATION_ADDRESS I have :
>> #!define DMQ_NOTIFICATION_ADDRESS "sip:dmq-router-service:5062"
>>
>> dmq-router-service DNS resolves:
>> > dmq-router-service.paolo.svc.cluster.local
>> Server: 10.96.0.10
>> Address: 10.96.0.10#53
>>
>> Non-authoritative answer:
>> Name: dmq-router-service.paolo.svc.cluster.local
>> Address: 172.28.0.240
>>
>>
>> so -nowhere- i am able to find 42.100.109.113
>>
>>
>> and in kamailio log:
>> 13(821) ERROR: dmq [notification_peer.c:599]:
>> notification_resp_callback_f(): deleting server sip:42.100.109.113:5062
>> because of failed request
>>
>>
>>
>> Cheers,
>>
>> Paolo
>>
>> 2018-07-26 18:14 GMT+02:00 Charles Chance
>> :
>>
>>> Hi Aleksandar,
>>>
>>> The initial depopulation of the nodes (following a period of 'pending'
>>> state) is due to no response being received from them. Are you able to
>>> trace the messages to/from one of them to confirm what is happening there?
>>>
>>> As for the unrecognised IP, I'm afraid I can't answer that one. It must
>>> be coming from somewhere - it's just a case of finding out where. Following
>>> the initial population/depopulation of the node list, DMQ simply repeats
>>> the process over and over until it eventually connects to one or more other
>>> nodes. Are you running on a public cloud somewhere or is it your own,
>>> locally running Kubernetes cluster?
>>>
>>> Cheers,
>>>
>>> Charles
>>>
>>>
>>> On 26 July 2018 at 15:30, Aleksandar Sosic
>>> wrote:
>>>
Hi Charles,
We're so glad about the improvements you just committed! Thanks!
Now I'm using the latest nightly: 5.2.0~dev6+0~20180726010431.11
65+xenial
Kamailio starts even if the DNS record does not exist at first, that's
great. I'm having this nodes up and running:
```
proxy-66f79498cc-8ws6d
Re: [SR-Users] DMQ problems
Hello again!
Another interesting thing found running in debug=3
0(710) DEBUG: dmq [notification_peer.c:240]: get_dmq_host_list(): adding
DMQ node A host dmq-router-service=sip:42.100.109.113:5062
so , kamailio "resolves" A host but does not query the DNS
this is dump of dns request inside kamailio container:
147 0.597474 10.42.1.251 → 10.96.0.10 DNS 102 Standard query 0xf31d
SRV dmq-router-service.paolo.svc.cluster.local
151 0.598306 10.96.0.10 → 10.42.1.251 DNS 155 Standard query response
0xf31d SRV dmq-router-service.paolo.svc.cluster.local SRV 10 100 0
3934376365383035.dmq-router-service.paolo.svc.cluster.local A 172.28.0.240
156 0.598538 10.42.1.251 → 10.96.0.10 DNS 102 Standard query 0x1904 A
dmq-router-service.paolo.svc.cluster.local
163 0.599275 10.96.0.10 → 10.42.1.251 DNS 118 Standard query response
0x1904 A dmq-router-service.paolo.svc.cluster.local A 172.28.0.240
*Paolo Visintin*
*CTO*
evosip.cloud
[image: Risultati immagini per evosip]
2018-08-02 0:24 GMT+02:00 Paolo Visintin - evosip.cloud <
paolo.visintin@evosip.cloud>:
>
> Hello Charles,
> sorry for late reply, we've had some issues rebuilding docker images with
> kamailo nightly build!
>
> We are running on our local, private on premise kubernetes cluster so
> connections from outside is simply impossible (in dev we are using a
> "closed" network system)
>
> I've just tried to :
> - open a tcpdump trace in the instance tracing only the DMQ port defined
> [5062] - (tcpdump -i eth0 port 5062)
> - packet coming from nowhere
> - started kamailio ($kamailio -f $PATH_KAMAILIO_CFG -m 512 -DD -E -e)
> - seeing this in tcpdump :
>
> root@router-0:/# tcpdump -i eth0 port 5062
> tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
> listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
> 22:06:44.966991 IP router-0.router.paolo.svc.cluster.local.5062 >
> 42.100.109.113.5062: UDP, length 497
> 22:06:45.572084 IP router-0.router.paolo.svc.cluster.local.5062 >
> 42.100.109.113.5062: UDP, length 497
> 22:06:46.571875 IP router-0.router.paolo.svc.cluster.local.5062 >
> 42.100.109.113.5062: UDP, length 497
>
> so seems that kamailio is trying to contact the foreign ip
>
> if i check inside the configuration:
> modparam("dmq", "server_address", DMQ_SERVER_ADDRESS)
> modparam("dmq", "notification_address", DMQ_NOTIFICATION_ADDRESS)
> modparam("dmq", "multi_notify", 1)
> modparam("dmq", "num_workers", 4)
> modparam("dmq", "ping_interval", 15)
>
>
> ...
>
> if(is_method("KDMQ") && dst_port == DMQ_PORT){
> dmq_handle_message();
> exit;
> }
>
>
> and in DMQ_NOTIFICATION_ADDRESS I have :
> #!define DMQ_NOTIFICATION_ADDRESS "sip:dmq-router-service:5062"
>
> dmq-router-service DNS resolves:
> > dmq-router-service.paolo.svc.cluster.local
> Server: 10.96.0.10
> Address: 10.96.0.10#53
>
> Non-authoritative answer:
> Name: dmq-router-service.paolo.svc.cluster.local
> Address: 172.28.0.240
>
>
> so -nowhere- i am able to find 42.100.109.113
>
>
> and in kamailio log:
> 13(821) ERROR: dmq [notification_peer.c:599]:
> notification_resp_callback_f(): deleting server sip:42.100.109.113:5062
> because of failed request
>
>
>
> Cheers,
>
> Paolo
>
> 2018-07-26 18:14 GMT+02:00 Charles Chance :
>
>> Hi Aleksandar,
>>
>> The initial depopulation of the nodes (following a period of 'pending'
>> state) is due to no response being received from them. Are you able to
>> trace the messages to/from one of them to confirm what is happening there?
>>
>> As for the unrecognised IP, I'm afraid I can't answer that one. It must
>> be coming from somewhere - it's just a case of finding out where. Following
>> the initial population/depopulation of the node list, DMQ simply repeats
>> the process over and over until it eventually connects to one or more other
>> nodes. Are you running on a public cloud somewhere or is it your own,
>> locally running Kubernetes cluster?
>>
>> Cheers,
>>
>> Charles
>>
>>
>> On 26 July 2018 at 15:30, Aleksandar Sosic
>> wrote:
>>
>>> Hi Charles,
>>>
>>> We're so glad about the improvements you just committed! Thanks!
>>>
>>> Now I'm using the latest nightly: 5.2.0~dev6+0~20180726010431.11
>>> 65+xenial
>>> Kamailio starts even if the DNS record does not exist at first, that's
>>> great. I'm having this nodes up and running:
>>> ```
>>> proxy-66f79498cc-8ws6d3/3 Running 0
>>> 4m172.28.1.4
>>> proxy-66f79498cc-b68dd3/3 Running 0
>>> 4m172.28.1.5
>>> proxy-66f79498cc-lhbxg3/3 Running 0
>>> 4m172.28.1.6
>>> proxy-66f79498cc-png6p3/3 Running 0
>>> 4m172.28.1.7
>>> ```
>>>
>>> In those 4-5 minutes I've experienced a population and depopulation of
>>> the dmq nodes.
>>>
>>> For example:
>>> ```
>>> kamcmd> dmq.list_nodes
>>> {
>>> host: 172.28.1.4
>>> port: 5061
>>> resolved_ip: 172.28.1.4
>>> status: pending
>>> last_notification: 0
>>> local: 0
>>> }
>>> {
>>>
Re: [SR-Users] DMQ problems
Hello Charles,
sorry for late reply, we've had some issues rebuilding docker images with
kamailo nightly build!
We are running on our local, private on premise kubernetes cluster so
connections from outside is simply impossible (in dev we are using a
"closed" network system)
I've just tried to :
- open a tcpdump trace in the instance tracing only the DMQ port defined
[5062] - (tcpdump -i eth0 port 5062)
- packet coming from nowhere
- started kamailio ($kamailio -f $PATH_KAMAILIO_CFG -m 512 -DD -E -e)
- seeing this in tcpdump :
root@router-0:/# tcpdump -i eth0 port 5062
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
22:06:44.966991 IP router-0.router.paolo.svc.cluster.local.5062 >
42.100.109.113.5062: UDP, length 497
22:06:45.572084 IP router-0.router.paolo.svc.cluster.local.5062 >
42.100.109.113.5062: UDP, length 497
22:06:46.571875 IP router-0.router.paolo.svc.cluster.local.5062 >
42.100.109.113.5062: UDP, length 497
so seems that kamailio is trying to contact the foreign ip
if i check inside the configuration:
modparam("dmq", "server_address", DMQ_SERVER_ADDRESS)
modparam("dmq", "notification_address", DMQ_NOTIFICATION_ADDRESS)
modparam("dmq", "multi_notify", 1)
modparam("dmq", "num_workers", 4)
modparam("dmq", "ping_interval", 15)
...
if(is_method("KDMQ") && dst_port == DMQ_PORT){
dmq_handle_message();
exit;
}
and in DMQ_NOTIFICATION_ADDRESS I have :
#!define DMQ_NOTIFICATION_ADDRESS "sip:dmq-router-service:5062"
dmq-router-service DNS resolves:
> dmq-router-service.paolo.svc.cluster.local
Server: 10.96.0.10
Address: 10.96.0.10#53
Non-authoritative answer:
Name: dmq-router-service.paolo.svc.cluster.local
Address: 172.28.0.240
so -nowhere- i am able to find 42.100.109.113
and in kamailio log:
13(821) ERROR: dmq [notification_peer.c:599]:
notification_resp_callback_f(): deleting server sip:42.100.109.113:5062
because of failed request
Cheers,
Paolo
2018-07-26 18:14 GMT+02:00 Charles Chance :
> Hi Aleksandar,
>
> The initial depopulation of the nodes (following a period of 'pending'
> state) is due to no response being received from them. Are you able to
> trace the messages to/from one of them to confirm what is happening there?
>
> As for the unrecognised IP, I'm afraid I can't answer that one. It must be
> coming from somewhere - it's just a case of finding out where. Following
> the initial population/depopulation of the node list, DMQ simply repeats
> the process over and over until it eventually connects to one or more other
> nodes. Are you running on a public cloud somewhere or is it your own,
> locally running Kubernetes cluster?
>
> Cheers,
>
> Charles
>
>
> On 26 July 2018 at 15:30, Aleksandar Sosic
> wrote:
>
>> Hi Charles,
>>
>> We're so glad about the improvements you just committed! Thanks!
>>
>> Now I'm using the latest nightly: 5.2.0~dev6+0~20180726010431.1165+xenial
>> Kamailio starts even if the DNS record does not exist at first, that's
>> great. I'm having this nodes up and running:
>> ```
>> proxy-66f79498cc-8ws6d3/3 Running 0
>> 4m172.28.1.4
>> proxy-66f79498cc-b68dd3/3 Running 0
>> 4m172.28.1.5
>> proxy-66f79498cc-lhbxg3/3 Running 0
>> 4m172.28.1.6
>> proxy-66f79498cc-png6p3/3 Running 0
>> 4m172.28.1.7
>> ```
>>
>> In those 4-5 minutes I've experienced a population and depopulation of
>> the dmq nodes.
>>
>> For example:
>> ```
>> kamcmd> dmq.list_nodes
>> {
>> host: 172.28.1.4
>> port: 5061
>> resolved_ip: 172.28.1.4
>> status: pending
>> last_notification: 0
>> local: 0
>> }
>> {
>> host: 172.28.1.6
>> port: 5061
>> resolved_ip: 172.28.1.6
>> status: pending
>> last_notification: 0
>> local: 0
>> }
>> {
>> host: 40.100.109.113
>> port: 5061
>> resolved_ip: 40.100.109.113
>> status: pending
>> last_notification: 0
>> local: 0
>> }
>> {
>> host: 172.28.1.5
>> port: 5061
>> resolved_ip: 172.28.1.5
>> status: active
>> last_notification: 0
>> local: 1
>> }
>> ```
>>
>> And after a few seconds only the loca node itself:
>> ```
>> kamcmd> dmq.list_nodes
>> {
>> host: 172.28.1.5
>> port: 5061
>> resolved_ip: 172.28.1.5
>> status: active
>> last_notification: 0
>> local: 1
>> }
>> ```
>> And then again the 3 nodes and the 40.100.109.113 which is a Microsoft
>> IP and I don't know what's doing that IP into my dmq server list.
>>
>> The DNS record is this:
>> ```
>> root@proxy-66f79498cc-b68dd:/# nslookup dmq-proxy-service
>> Server: 10.96.0.10
>> Address: 10.96.0.10#53
>>
>> Non-authoritative answer:
>> Name: dmq-proxy-service.alex.svc.cluster.local
>> Address: 172.28.1.7
>> Name: dmq-proxy-service.alex.svc.cluster.local
>> Address: 172.28.1.6
>> Name: dmq-proxy-service.alex.svc.cluster.local
>> Address: 172.28.1.5
>> Name: dmq-proxy-service.alex.svc.cluster.local
>> Address: 172.28.1.4
>> ```
>> Any clues why that ip is in the
Re: [SR-Users] DMQ problems
Hi Aleksandar,
The initial depopulation of the nodes (following a period of 'pending'
state) is due to no response being received from them. Are you able to
trace the messages to/from one of them to confirm what is happening there?
As for the unrecognised IP, I'm afraid I can't answer that one. It must be
coming from somewhere - it's just a case of finding out where. Following
the initial population/depopulation of the node list, DMQ simply repeats
the process over and over until it eventually connects to one or more other
nodes. Are you running on a public cloud somewhere or is it your own,
locally running Kubernetes cluster?
Cheers,
Charles
On 26 July 2018 at 15:30, Aleksandar Sosic wrote:
> Hi Charles,
>
> We're so glad about the improvements you just committed! Thanks!
>
> Now I'm using the latest nightly: 5.2.0~dev6+0~20180726010431.1165+xenial
> Kamailio starts even if the DNS record does not exist at first, that's
> great. I'm having this nodes up and running:
> ```
> proxy-66f79498cc-8ws6d3/3 Running 0
> 4m172.28.1.4
> proxy-66f79498cc-b68dd3/3 Running 0
> 4m172.28.1.5
> proxy-66f79498cc-lhbxg3/3 Running 0
> 4m172.28.1.6
> proxy-66f79498cc-png6p3/3 Running 0
> 4m172.28.1.7
> ```
>
> In those 4-5 minutes I've experienced a population and depopulation of
> the dmq nodes.
>
> For example:
> ```
> kamcmd> dmq.list_nodes
> {
> host: 172.28.1.4
> port: 5061
> resolved_ip: 172.28.1.4
> status: pending
> last_notification: 0
> local: 0
> }
> {
> host: 172.28.1.6
> port: 5061
> resolved_ip: 172.28.1.6
> status: pending
> last_notification: 0
> local: 0
> }
> {
> host: 40.100.109.113
> port: 5061
> resolved_ip: 40.100.109.113
> status: pending
> last_notification: 0
> local: 0
> }
> {
> host: 172.28.1.5
> port: 5061
> resolved_ip: 172.28.1.5
> status: active
> last_notification: 0
> local: 1
> }
> ```
>
> And after a few seconds only the loca node itself:
> ```
> kamcmd> dmq.list_nodes
> {
> host: 172.28.1.5
> port: 5061
> resolved_ip: 172.28.1.5
> status: active
> last_notification: 0
> local: 1
> }
> ```
> And then again the 3 nodes and the 40.100.109.113 which is a Microsoft
> IP and I don't know what's doing that IP into my dmq server list.
>
> The DNS record is this:
> ```
> root@proxy-66f79498cc-b68dd:/# nslookup dmq-proxy-service
> Server: 10.96.0.10
> Address: 10.96.0.10#53
>
> Non-authoritative answer:
> Name: dmq-proxy-service.alex.svc.cluster.local
> Address: 172.28.1.7
> Name: dmq-proxy-service.alex.svc.cluster.local
> Address: 172.28.1.6
> Name: dmq-proxy-service.alex.svc.cluster.local
> Address: 172.28.1.5
> Name: dmq-proxy-service.alex.svc.cluster.local
> Address: 172.28.1.4
> ```
> Any clues why that ip is in the dmq nodes list? And why .7 is missing?
> Kind regards,
> --
> Aleksandar Sosic
> mail: alex.sosic@evosip.cloud
>
> On Wed, Jul 25, 2018 at 3:06 PM Charles Chance
> wrote:
> >
> > Hi,
> >
> > I have just pushed some changes to master - one of these allows startup
> to continue even if initial node resolution fails.
> >
> > There are some other improvements, too, which I have been planning to
> push for some time and which should also help in your situation.
> >
> > Can you try again with these changes applied and let me know the outcome?
> >
> > Cheers,
> >
> > Charles
>
> ___
> Kamailio (SER) - Users Mailing List
> sr-users@lists.kamailio.org
> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
>
--
*Charles Chance*
Managing Director
t. 0330 120 1200m. 07932 063 891
--
Sipcentric Ltd.
Company registered in England & Wales no.
7365592. Registered
office: Faraday Wharf, Innovation
Birmingham Campus, Holt Street, Birmingham Science Park, Birmingham B7 4BB.
___
Kamailio (SER) - Users Mailing List
sr-users@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Re: [SR-Users] DMQ problems
Hi Charles,
We're so glad about the improvements you just committed! Thanks!
Now I'm using the latest nightly: 5.2.0~dev6+0~20180726010431.1165+xenial
Kamailio starts even if the DNS record does not exist at first, that's
great. I'm having this nodes up and running:
```
proxy-66f79498cc-8ws6d3/3 Running 0
4m172.28.1.4
proxy-66f79498cc-b68dd3/3 Running 0
4m172.28.1.5
proxy-66f79498cc-lhbxg3/3 Running 0
4m172.28.1.6
proxy-66f79498cc-png6p3/3 Running 0
4m172.28.1.7
```
In those 4-5 minutes I've experienced a population and depopulation of
the dmq nodes.
For example:
```
kamcmd> dmq.list_nodes
{
host: 172.28.1.4
port: 5061
resolved_ip: 172.28.1.4
status: pending
last_notification: 0
local: 0
}
{
host: 172.28.1.6
port: 5061
resolved_ip: 172.28.1.6
status: pending
last_notification: 0
local: 0
}
{
host: 40.100.109.113
port: 5061
resolved_ip: 40.100.109.113
status: pending
last_notification: 0
local: 0
}
{
host: 172.28.1.5
port: 5061
resolved_ip: 172.28.1.5
status: active
last_notification: 0
local: 1
}
```
And after a few seconds only the loca node itself:
```
kamcmd> dmq.list_nodes
{
host: 172.28.1.5
port: 5061
resolved_ip: 172.28.1.5
status: active
last_notification: 0
local: 1
}
```
And then again the 3 nodes and the 40.100.109.113 which is a Microsoft
IP and I don't know what's doing that IP into my dmq server list.
The DNS record is this:
```
root@proxy-66f79498cc-b68dd:/# nslookup dmq-proxy-service
Server: 10.96.0.10
Address: 10.96.0.10#53
Non-authoritative answer:
Name: dmq-proxy-service.alex.svc.cluster.local
Address: 172.28.1.7
Name: dmq-proxy-service.alex.svc.cluster.local
Address: 172.28.1.6
Name: dmq-proxy-service.alex.svc.cluster.local
Address: 172.28.1.5
Name: dmq-proxy-service.alex.svc.cluster.local
Address: 172.28.1.4
```
Any clues why that ip is in the dmq nodes list? And why .7 is missing?
Kind regards,
--
Aleksandar Sosic
mail: alex.sosic@evosip.cloud
On Wed, Jul 25, 2018 at 3:06 PM Charles Chance
wrote:
>
> Hi,
>
> I have just pushed some changes to master - one of these allows startup to
> continue even if initial node resolution fails.
>
> There are some other improvements, too, which I have been planning to push
> for some time and which should also help in your situation.
>
> Can you try again with these changes applied and let me know the outcome?
>
> Cheers,
>
> Charles
___
Kamailio (SER) - Users Mailing List
sr-users@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Re: [SR-Users] DMQ problems
Hi,
I have just pushed some changes to master - one of these allows startup to
continue even if initial node resolution fails.
There are some other improvements, too, which I have been planning to push
for some time and which should also help in your situation.
Can you try again with these changes applied and let me know the outcome?
Cheers,
Charles
On 6 July 2018 at 11:35, Aleksandar Sosic wrote:
> Similar thing with a different type of nodes:
>
> ```
> proxy-94b6ccf46-6n49v 3/3 Running 0
> 1m172.22.5.99 master.alex.cloud.evox.it
> proxy-94b6ccf46-7jrgj 3/3 Running 0
> 1m172.22.5.98 master.alex.cloud.evox.it
> proxy-94b6ccf46-rbskb 3/3 Running 0
> 1m172.22.5.95 master.alex.cloud.evox.it
> proxy-94b6ccf46-rtzh9 3/3 Running 0
> 1m172.22.5.96 master.alex.cloud.evox.it`
> ```
>
> ```
> root@proxy-94b6ccf46-6n49v:/# kamcmd
> kamcmd 1.5
> Copyright 2006 iptelorg GmbH
> This is free software with ABSOLUTELY NO WARRANTY.
> For details type `warranty'.
> kamcmd> dmq.list_nodes
> {
> host: 172.22.5.99
> port: 5061
> resolved_ip: 172.22.5.99
> status: active
> last_notification: 0
> local: 1
> }
> ```
>
> ```
> root@proxy-94b6ccf46-7jrgj:/# kamcmd
> kamcmd 1.5
> Copyright 2006 iptelorg GmbH
> This is free software with ABSOLUTELY NO WARRANTY.
> For details type `warranty'.
> kamcmd> dmq.list_nodes
> {
> host: 172.22.5.95
> port: 5061
> resolved_ip: 172.22.5.95
> status: active
> last_notification: 0
> local: 0
> }
> {
> host: 172.22.5.98
> port: 5061
> resolved_ip: 172.22.5.98
> status: active
> last_notification: 0
> local: 1
> }
> ```
>
> ```
> root@proxy-94b6ccf46-rbskb:/# kamcmd
> kamcmd 1.5
> Copyright 2006 iptelorg GmbH
> This is free software with ABSOLUTELY NO WARRANTY.
> For details type `warranty'.
> kamcmd> dmq.list_nodes
> {
> host: 172.22.5.95
> port: 5061
> resolved_ip: 172.22.5.95
> status: active
> last_notification: 0
> local: 1
> }
> ```
>
> ```
> root@proxy-94b6ccf46-rtzh9:/# kamcmd
> kamcmd 1.5
> Copyright 2006 iptelorg GmbH
> This is free software with ABSOLUTELY NO WARRANTY.
> For details type `warranty'.
> kamcmd> dmq.list_nodes
> {
> host: 172.22.5.96
> port: 5061
> resolved_ip: 172.22.5.96
> status: active
> last_notification: 0
> local: 1
> }
> ```
>
> ```
> nslookup dmq-proxy-service
> Server:10.96.0.10
> Address:10.96.0.10#53
>
> Non-authoritative answer:
> Name:dmq-proxy-service.default.svc.cluster.local
> Address: 172.22.5.99
> Name:dmq-proxy-service.default.svc.cluster.local
> Address: 172.22.5.98
> Name:dmq-proxy-service.default.svc.cluster.local
> Address: 172.22.5.96
> Name:dmq-proxy-service.default.svc.cluster.local
> Address: 172.22.5.95
> Name:dmq-proxy-service.default.svc.cluster.local
> Address: 10.32.68.218
> ```
>
> What am I missing out?
> --
> Aleksandar Sosic
> mail: alex.sosic@evosip.cloud
> On Fri, Jul 6, 2018 at 12:26 PM Aleksandar Sosic
> wrote:
> >
> > Here's another example:
> >
> > ```
> > router-0 3/3 Running 0
> > 13m 172.22.5.94 master.alex.cloud.evox.it
> > router-1 3/3 Running 0
> > 13m 172.22.5.3 master.alex.cloud.evox.it
> > router-2 3/3 Running 0
> > 12m 172.22.5.4 master.alex.cloud.evox.it
> > router-3 3/3 Running 0
> > 12m 172.22.5.5 master.alex.cloud.evox.it
> > ```
> >
> > ```
> > root@router-0:/# kamcmd
> > kamcmd 1.5
> > Copyright 2006 iptelorg GmbH
> > This is free software with ABSOLUTELY NO WARRANTY.
> > For details type `warranty'.
> > kamcmd> dmq.list_nodes
> > {
> > host: 172.22.5.3
> > port: 5062
> > resolved_ip: 172.22.5.3
> > status: active
> > last_notification: 0
> > local: 0
> > }
> > {
> > host: 172.22.5.4
> > port: 5062
> > resolved_ip: 172.22.5.4
> > status: active
> > last_notification: 0
> > local: 0
> > }
> > {
> > host: 172.22.5.94
> > port: 5062
> > resolved_ip: 172.22.5.94
> > status: active
> > last_notification: 0
> > local: 1
> > }
> > ```
> >
> > ```
> > root@router-1:/# kamcmd
> > kamcmd 1.5
> > Copyright 2006 iptelorg GmbH
> > This is free software with ABSOLUTELY NO WARRANTY.
> > For details type `warranty'.
> > kamcmd> dmq.list_nodes
> > {
> > host: 172.22.5.4
> > port: 5062
> > resolved_ip: 172.22.5.4
> > status: active
> > last_notification: 0
> > local: 0
> > }
> > {
> > host: 172.22.5.94
> > port: 5062
> > resolved_ip: 172.22.5.94
> > status: active
> > last_notification: 0
> > local: 0
> > }
> > {
> > host: 172.22.5.3
> > port: 5062
> > resolved_ip: 172.22.5.3
> > status: active
> >
Re: [SR-Users] DMQ problems
Similar thing with a different type of nodes:
```
proxy-94b6ccf46-6n49v 3/3 Running 0
1m172.22.5.99 master.alex.cloud.evox.it
proxy-94b6ccf46-7jrgj 3/3 Running 0
1m172.22.5.98 master.alex.cloud.evox.it
proxy-94b6ccf46-rbskb 3/3 Running 0
1m172.22.5.95 master.alex.cloud.evox.it
proxy-94b6ccf46-rtzh9 3/3 Running 0
1m172.22.5.96 master.alex.cloud.evox.it`
```
```
root@proxy-94b6ccf46-6n49v:/# kamcmd
kamcmd 1.5
Copyright 2006 iptelorg GmbH
This is free software with ABSOLUTELY NO WARRANTY.
For details type `warranty'.
kamcmd> dmq.list_nodes
{
host: 172.22.5.99
port: 5061
resolved_ip: 172.22.5.99
status: active
last_notification: 0
local: 1
}
```
```
root@proxy-94b6ccf46-7jrgj:/# kamcmd
kamcmd 1.5
Copyright 2006 iptelorg GmbH
This is free software with ABSOLUTELY NO WARRANTY.
For details type `warranty'.
kamcmd> dmq.list_nodes
{
host: 172.22.5.95
port: 5061
resolved_ip: 172.22.5.95
status: active
last_notification: 0
local: 0
}
{
host: 172.22.5.98
port: 5061
resolved_ip: 172.22.5.98
status: active
last_notification: 0
local: 1
}
```
```
root@proxy-94b6ccf46-rbskb:/# kamcmd
kamcmd 1.5
Copyright 2006 iptelorg GmbH
This is free software with ABSOLUTELY NO WARRANTY.
For details type `warranty'.
kamcmd> dmq.list_nodes
{
host: 172.22.5.95
port: 5061
resolved_ip: 172.22.5.95
status: active
last_notification: 0
local: 1
}
```
```
root@proxy-94b6ccf46-rtzh9:/# kamcmd
kamcmd 1.5
Copyright 2006 iptelorg GmbH
This is free software with ABSOLUTELY NO WARRANTY.
For details type `warranty'.
kamcmd> dmq.list_nodes
{
host: 172.22.5.96
port: 5061
resolved_ip: 172.22.5.96
status: active
last_notification: 0
local: 1
}
```
```
nslookup dmq-proxy-service
Server:10.96.0.10
Address:10.96.0.10#53
Non-authoritative answer:
Name:dmq-proxy-service.default.svc.cluster.local
Address: 172.22.5.99
Name:dmq-proxy-service.default.svc.cluster.local
Address: 172.22.5.98
Name:dmq-proxy-service.default.svc.cluster.local
Address: 172.22.5.96
Name:dmq-proxy-service.default.svc.cluster.local
Address: 172.22.5.95
Name:dmq-proxy-service.default.svc.cluster.local
Address: 10.32.68.218
```
What am I missing out?
--
Aleksandar Sosic
mail: alex.sosic@evosip.cloud
On Fri, Jul 6, 2018 at 12:26 PM Aleksandar Sosic
wrote:
>
> Here's another example:
>
> ```
> router-0 3/3 Running 0
> 13m 172.22.5.94 master.alex.cloud.evox.it
> router-1 3/3 Running 0
> 13m 172.22.5.3 master.alex.cloud.evox.it
> router-2 3/3 Running 0
> 12m 172.22.5.4 master.alex.cloud.evox.it
> router-3 3/3 Running 0
> 12m 172.22.5.5 master.alex.cloud.evox.it
> ```
>
> ```
> root@router-0:/# kamcmd
> kamcmd 1.5
> Copyright 2006 iptelorg GmbH
> This is free software with ABSOLUTELY NO WARRANTY.
> For details type `warranty'.
> kamcmd> dmq.list_nodes
> {
> host: 172.22.5.3
> port: 5062
> resolved_ip: 172.22.5.3
> status: active
> last_notification: 0
> local: 0
> }
> {
> host: 172.22.5.4
> port: 5062
> resolved_ip: 172.22.5.4
> status: active
> last_notification: 0
> local: 0
> }
> {
> host: 172.22.5.94
> port: 5062
> resolved_ip: 172.22.5.94
> status: active
> last_notification: 0
> local: 1
> }
> ```
>
> ```
> root@router-1:/# kamcmd
> kamcmd 1.5
> Copyright 2006 iptelorg GmbH
> This is free software with ABSOLUTELY NO WARRANTY.
> For details type `warranty'.
> kamcmd> dmq.list_nodes
> {
> host: 172.22.5.4
> port: 5062
> resolved_ip: 172.22.5.4
> status: active
> last_notification: 0
> local: 0
> }
> {
> host: 172.22.5.94
> port: 5062
> resolved_ip: 172.22.5.94
> status: active
> last_notification: 0
> local: 0
> }
> {
> host: 172.22.5.3
> port: 5062
> resolved_ip: 172.22.5.3
> status: active
> last_notification: 0
> local: 1
> }
> ```
>
> ```
> root@router-2:/# kamcmd
> kamcmd 1.5
> Copyright 2006 iptelorg GmbH
> This is free software with ABSOLUTELY NO WARRANTY.
> For details type `warranty'.
> kamcmd> dmq.list_nodes
> {
> host: 172.22.5.94
> port: 5062
> resolved_ip: 172.22.5.94
> status: active
> last_notification: 0
> local: 0
> }
> {
> host: 172.22.5.3
> port: 5062
> resolved_ip: 172.22.5.3
> status: active
> last_notification: 0
> local: 0
> }
> {
> host: 172.22.5.4
> port: 5062
> resolved_ip: 172.22.5.4
> status: active
> last_notification: 0
> local: 1
> }
> ```
>
> ```
> root@router-3:/# kamcmd
> kamcmd 1.5
> Copyright 2006 iptelorg GmbH
> This is free software with ABSOLUTELY NO
Re: [SR-Users] DMQ problems
Here's another example:
```
router-0 3/3 Running 0
13m 172.22.5.94 master.alex.cloud.evox.it
router-1 3/3 Running 0
13m 172.22.5.3 master.alex.cloud.evox.it
router-2 3/3 Running 0
12m 172.22.5.4 master.alex.cloud.evox.it
router-3 3/3 Running 0
12m 172.22.5.5 master.alex.cloud.evox.it
```
```
root@router-0:/# kamcmd
kamcmd 1.5
Copyright 2006 iptelorg GmbH
This is free software with ABSOLUTELY NO WARRANTY.
For details type `warranty'.
kamcmd> dmq.list_nodes
{
host: 172.22.5.3
port: 5062
resolved_ip: 172.22.5.3
status: active
last_notification: 0
local: 0
}
{
host: 172.22.5.4
port: 5062
resolved_ip: 172.22.5.4
status: active
last_notification: 0
local: 0
}
{
host: 172.22.5.94
port: 5062
resolved_ip: 172.22.5.94
status: active
last_notification: 0
local: 1
}
```
```
root@router-1:/# kamcmd
kamcmd 1.5
Copyright 2006 iptelorg GmbH
This is free software with ABSOLUTELY NO WARRANTY.
For details type `warranty'.
kamcmd> dmq.list_nodes
{
host: 172.22.5.4
port: 5062
resolved_ip: 172.22.5.4
status: active
last_notification: 0
local: 0
}
{
host: 172.22.5.94
port: 5062
resolved_ip: 172.22.5.94
status: active
last_notification: 0
local: 0
}
{
host: 172.22.5.3
port: 5062
resolved_ip: 172.22.5.3
status: active
last_notification: 0
local: 1
}
```
```
root@router-2:/# kamcmd
kamcmd 1.5
Copyright 2006 iptelorg GmbH
This is free software with ABSOLUTELY NO WARRANTY.
For details type `warranty'.
kamcmd> dmq.list_nodes
{
host: 172.22.5.94
port: 5062
resolved_ip: 172.22.5.94
status: active
last_notification: 0
local: 0
}
{
host: 172.22.5.3
port: 5062
resolved_ip: 172.22.5.3
status: active
last_notification: 0
local: 0
}
{
host: 172.22.5.4
port: 5062
resolved_ip: 172.22.5.4
status: active
last_notification: 0
local: 1
}
```
```
root@router-3:/# kamcmd
kamcmd 1.5
Copyright 2006 iptelorg GmbH
This is free software with ABSOLUTELY NO WARRANTY.
For details type `warranty'.
kamcmd> dmq.list_nodes
{
host: 172.22.5.5
port: 5062
resolved_ip: 172.22.5.5
status: active
last_notification: 0
local: 1
}
```
```
nslookup dmq-router-service
Server: 10.96.0.10
Address: 10.96.0.10#53
Non-authoritative answer:
Name: dmq-router-service.default.svc.cluster.local
Address: 172.22.5.5
Name: dmq-router-service.default.svc.cluster.local
Address: 172.22.5.4
Name: dmq-router-service.default.svc.cluster.local
Address: 172.22.5.3
Name: dmq-router-service.default.svc.cluster.local
Address: 10.32.68.217
Name: dmq-router-service.default.svc.cluster.local
Address: 172.22.5.94
```
There is no difference in configuration between the router nodes.
--
Aleksandar Sosic
mail: alex.sosic@evosip.cloud
On Fri, Jul 6, 2018 at 11:42 AM Aleksandar Sosic
wrote:
>
> Hi,
>
> I'm now creating a dns record inside kubernetes with a headless service.
> Unfortunately I must use a busybox that will start before the kamailio
> nodes so the dns record will be created before kamailio starts because
> otherwise it will crash as I told you before. IMHO it will be useful
> to have a flag that allows me to choose if kamailio should start even
> if the notification_address of the DMQ module does not resolve. That
> would allow me to start nodes, populate the dns and then slowly add
> nodes to the cluster without having a busybox to do that.
>
> As you can see from the logs kamailio notices the busybox IP
> (10.32.68.202) which obviously not being a kamailio node gives me this
> errors:
> ```
> 17(109) ERROR: dispatcher [dispatch.c:1910]: ds_select_dst_limit(): no
> destination sets
> 33(125) ERROR: dmq [notification_peer.c:596]:
> notification_resp_callback_f(): deleting server sip:10.32.68.202:5061
> because of failed request
> 33(125) ERROR: dmq [notification_peer.c:596]:
> notification_resp_callback_f(): deleting server
> sip:43.100.109.113:5061 because of failed request
> ```
> Being a workaround for creating the DNS record before starting
> kamailio nodes that's ok.
> But notice the IP 43.100.109.113 which is not an IP that the DNS
> record for the notification_address resolves:
>
> ```
> $ nslookup dmq-proxy-service
> Server: 10.96.0.10
> Address: 10.96.0.10#53
>
> Name: dmq-proxy-service.default.svc.cluster.local
> Address: 172.22.5.83
> Name: dmq-proxy-service.default.svc.cluster.local
> Address: 172.22.5.84
> Name: dmq-proxy-service.default.svc.cluster.local
> Address: 10.32.68.202
> Name: dmq-proxy-service.default.svc.cluster.local
> Address: 172.22.5.81
> Name: dmq-proxy-service.default.svc.cluster.local
> Address: 172.22.5.82
> ```
>
> So why is that?
>
> Then I noticed that on one node I do have all the other IPs:
> ```kamcmd> dmq.list_nodes
> {
Re: [SR-Users] DMQ problems
Hi,
I'm now creating a dns record inside kubernetes with a headless service.
Unfortunately I must use a busybox that will start before the kamailio
nodes so the dns record will be created before kamailio starts because
otherwise it will crash as I told you before. IMHO it will be useful
to have a flag that allows me to choose if kamailio should start even
if the notification_address of the DMQ module does not resolve. That
would allow me to start nodes, populate the dns and then slowly add
nodes to the cluster without having a busybox to do that.
As you can see from the logs kamailio notices the busybox IP
(10.32.68.202) which obviously not being a kamailio node gives me this
errors:
```
17(109) ERROR: dispatcher [dispatch.c:1910]: ds_select_dst_limit(): no
destination sets
33(125) ERROR: dmq [notification_peer.c:596]:
notification_resp_callback_f(): deleting server sip:10.32.68.202:5061
because of failed request
33(125) ERROR: dmq [notification_peer.c:596]:
notification_resp_callback_f(): deleting server
sip:43.100.109.113:5061 because of failed request
```
Being a workaround for creating the DNS record before starting
kamailio nodes that's ok.
But notice the IP 43.100.109.113 which is not an IP that the DNS
record for the notification_address resolves:
```
$ nslookup dmq-proxy-service
Server: 10.96.0.10
Address: 10.96.0.10#53
Name: dmq-proxy-service.default.svc.cluster.local
Address: 172.22.5.83
Name: dmq-proxy-service.default.svc.cluster.local
Address: 172.22.5.84
Name: dmq-proxy-service.default.svc.cluster.local
Address: 10.32.68.202
Name: dmq-proxy-service.default.svc.cluster.local
Address: 172.22.5.81
Name: dmq-proxy-service.default.svc.cluster.local
Address: 172.22.5.82
```
So why is that?
Then I noticed that on one node I do have all the other IPs:
```kamcmd> dmq.list_nodes
{
host: 172.22.5.81
port: 5061
resolved_ip: 172.22.5.81
status: active
last_notification: 0
local: 0
}
{
host: 172.22.5.82
port: 5061
resolved_ip: 172.22.5.82
status: active
last_notification: 0
local: 0
}
{
host: 172.22.5.83
port: 5061
resolved_ip: 172.22.5.83
status: active
last_notification: 0
local: 0
}
{
host: 172.22.5.84
port: 5061
resolved_ip: 172.22.5.84
status: active
last_notification: 0
local: 1
}
```
But on another one:
```
kamcmd> dmq.list_nodes
{
host: 172.22.5.83
port: 5061
resolved_ip: 172.22.5.83
status: active
last_notification: 0
local: 0
}
{
host: 172.22.5.81
port: 5061
resolved_ip: 172.22.5.81
status: active
last_notification: 0
local: 0
}
{
host: 172.22.5.82
port: 5061
resolved_ip: 172.22.5.82
status: active
last_notification: 0
local: 1
}
```
I miss a node? And all the kamailio instances in this range are the
same with the same config. Also the missing node is not the ip of the
node itself but another one.
--
Aleksandar Sosic
mail: alex.sosic@evosip.cloudOn Fri, Jul 6, 2018 at 9:21 AM Aleksandar
Sosic wrote:
>
> On Thu, Jul 5, 2018 at 12:35 PM Charles Chance
> wrote:
> > I'll take a look - which version are you using?
>
> 5.2.0~dev6+0~20180616010152.1138+xenial
>
> Thank you!
> --
> Aleksandar Sosic
> mail: alex.sosic@evosip.cloud
___
Kamailio (SER) - Users Mailing List
sr-users@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Re: [SR-Users] DMQ problems
On Thu, Jul 5, 2018 at 12:35 PM Charles Chance wrote: > I'll take a look - which version are you using? 5.2.0~dev6+0~20180616010152.1138+xenial Thank you! -- Aleksandar Sosic mail: alex.sosic@evosip.cloud ___ Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Re: [SR-Users] DMQ problems
I'll take a look - which version are you using? Cheers, Charles On 5 July 2018 at 11:17, Aleksandar Sosic wrote: > On Mon, Jul 2, 2018 at 2:55 PM Charles Chance > wrote: > > Hi Aleksandar, > > [...] > > You do not, in fact, need to maintain a dedicated node who's only role > is "DMQ server" as you have described it. I would recommend removing it > completely and allowing your other nodes to discover/communicate between > themselves. > > Ok Thanks for your inputs, seems a good idea and I think we're on a > good path to solve our issues. > > > Omitting the notification address should not cause an issue - what error > do you see in your log when it fails to start? If you get rid of the "DMQ > server' node as suggested above, though, this should be a none-issue. > > It does cause an issue: > ``` > 0(51) DEBUG: [core/parser/parse_uri.c:1254]: parse_uri(): uri > too short: <> (0) > 0(51) ERROR: dmq [dmq.c:218]: mod_init(): notification address invalid > 0(51) ERROR: [core/sr_module.c:990]: init_mod(): Error while > initializing module dmq > (/usr/lib/x86_64-linux-gnu/kamailio/modules/dmq.so) > ERROR: error while initializing modules > ``` > > Maybe a feature request could be to just start kamailio even if it > does not resolve a dns record for the dmq notification_address? This > would solve us some problems in a highly mutable infrastructure like > ours: > > ``` 0(88) ERROR: dmq [notification_peer.c:336]: > add_server_and_notify(): error adding notification node > 0(88) ERROR: dmq [dmq.c:304]: child_init(): cannot retrieve initial > nodelist from sip:proxy-service:5061 > 0(88) ERROR: [core/sr_module.c:944]: init_mod_child(): error > while initializing module dmq > (/usr/lib/x86_64-linux-gnu/kamailio/modules/dmq.so) (idx: 0 rank: 0 > desc: [main]) > 0(88) ERROR: [main.c:1711]: main_loop(): error in init_child``` > > So just to recap: > - Not specifying a notification_address DOES give issues > - Having a DNS record for the notification_address that does not > resolve makes kamailio to crash > > Any possibility to have this solved/enhanced? > > Thanks, > -- > Aleksandar Sosic > mail: alex.sosic@evosip.cloud > > ___ > Kamailio (SER) - Users Mailing List > sr-users@lists.kamailio.org > https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users > -- Sipcentric Ltd. Company registered in England & Wales no. 7365592. Registered office: Faraday Wharf, Innovation Birmingham Campus, Holt Street, Birmingham Science Park, Birmingham B7 4BB. ___ Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Re: [SR-Users] DMQ problems
On Mon, Jul 2, 2018 at 2:55 PM Charles Chance wrote: > Hi Aleksandar, > [...] > You do not, in fact, need to maintain a dedicated node who's only role is > "DMQ server" as you have described it. I would recommend removing it > completely and allowing your other nodes to discover/communicate between > themselves. Ok Thanks for your inputs, seems a good idea and I think we're on a good path to solve our issues. > Omitting the notification address should not cause an issue - what error do > you see in your log when it fails to start? If you get rid of the "DMQ > server' node as suggested above, though, this should be a none-issue. It does cause an issue: ``` 0(51) DEBUG: [core/parser/parse_uri.c:1254]: parse_uri(): uri too short: <> (0) 0(51) ERROR: dmq [dmq.c:218]: mod_init(): notification address invalid 0(51) ERROR: [core/sr_module.c:990]: init_mod(): Error while initializing module dmq (/usr/lib/x86_64-linux-gnu/kamailio/modules/dmq.so) ERROR: error while initializing modules ``` Maybe a feature request could be to just start kamailio even if it does not resolve a dns record for the dmq notification_address? This would solve us some problems in a highly mutable infrastructure like ours: ``` 0(88) ERROR: dmq [notification_peer.c:336]: add_server_and_notify(): error adding notification node 0(88) ERROR: dmq [dmq.c:304]: child_init(): cannot retrieve initial nodelist from sip:proxy-service:5061 0(88) ERROR: [core/sr_module.c:944]: init_mod_child(): error while initializing module dmq (/usr/lib/x86_64-linux-gnu/kamailio/modules/dmq.so) (idx: 0 rank: 0 desc: [main]) 0(88) ERROR: [main.c:1711]: main_loop(): error in init_child``` So just to recap: - Not specifying a notification_address DOES give issues - Having a DNS record for the notification_address that does not resolve makes kamailio to crash Any possibility to have this solved/enhanced? Thanks, -- Aleksandar Sosic mail: alex.sosic@evosip.cloud ___ Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Re: [SR-Users] DMQ problems
Hi Aleksandar,
On 1 July 2018 at 07:08, Aleksandar Sosic wrote:
> Hi everyone,
>
> so adding some xlogs We've managed to find out what's happening.
> The request route was using:
> ```
> if(is_method("KDMQ")){
> ```
> but this if was never triggered(!?)
> We changed that to
> ```
> if($rm == "KDMQ"){
> ```
> And now it's working.
>
> So why is that? In the documentation there's an example with is_method:
> https://kamailio.org/docs/modules/5.2.x/modules/dmq.
> html#dmq.f.dmq_handle_message
> and I can recall it working a month ago.
>
>
Either way should work - there should be no difference between them. We are
using is_method everywhere and it works just fine.
Everything is working except for this message:
> ```25(42) ERROR: dmq [notification_peer.c:596]:
> notification_resp_callback_f(): deleting server sip:127.0.0.1:5060
> because of failed request```
> And that's because of the notification address on the kamailio node we
> use as DMQ server
>
>
I suspect setting localhost as the notification address is causing you some
problems - it's certainly not supported and although I've not tested it, it
wouldn't surprise me if this has something to do with your earlier issue.
You do not, in fact, need to maintain a dedicated node who's only role is
"DMQ server" as you have described it. I would recommend removing it
completely and allowing your other nodes to discover/communicate between
themselves.
> This portion of DMQ documentation says that `Default value is "NULL".`
> for the notification_address:
> https://kamailio.org/docs/modules/5.2.x/modules/dmq.
> html#dmq.p.notification_address
> But with NULL or NOT specifying a value kamailio does not start.
> Am I misunderstanding the documentation? Could this be a feature
> request or it's a bug?
>
Omitting the notification address should not cause an issue - what error do
you see in your log when it fails to start? If you get rid of the "DMQ
server' node as suggested above, though, this should be a none-issue.
> In a situation like our where kamailio nodes start and stop on demand
> without any prior defined topology or IP addresses defined it's hard
> to use DMQ module like that. That's why we introduced a DMQ server
> with a fixed IP address that it's handling only those packets (but
> it's a workaround).
>
> So my next question is:
> Do you guys have an idea on how could we handle DMQ on a very
> mutational infrastructure?
> Example: I have 2 kamailio nodes in the begining (no fixed IPs - using
> DHCP), then when there's demand the infrastructure starts another 2
> nodes, now I have 4 of them, then 3, one of the nodes randomly is
> stopped.
> And I want to use DMQ on these node to propagate information without
> using a DB. How could we do that?
>
Use multi_notify -
https://www.kamailio.org/docs/modules/stable/modules/dmq.html#dmq.p.multi_notify
If nodes are being added/removed dynamically then you'll probably also want
to employ some kind of service discovery mechanism, e.g. consul.
> Also another thing that we noticed is that the last node to be added
> is the one with no prior information regarding usrloc for example, it
> has only the changes happening from when it starts on. Could there be
> a method or modparam flag to force dmq sync of all the information
> upon the startup of another node (another possible feature request)?
>
>
See sync parameter -
https://www.kamailio.org/docs/modules/stable/modules/dmq_usrloc.html#usrloc_dmq.p.sync
Cheers,
Charles
--
Sipcentric Ltd.
Company registered in England & Wales no.
7365592. Registered
office: Faraday Wharf, Innovation
Birmingham Campus, Holt Street, Birmingham Science Park, Birmingham B7 4BB.
___
Kamailio (SER) - Users Mailing List
sr-users@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Re: [SR-Users] DMQ problems
Hi everyone,
so adding some xlogs We've managed to find out what's happening.
The request route was using:
```
if(is_method("KDMQ")){
```
but this if was never triggered(!?)
We changed that to
```
if($rm == "KDMQ"){
```
And now it's working.
So why is that? In the documentation there's an example with is_method:
https://kamailio.org/docs/modules/5.2.x/modules/dmq.html#dmq.f.dmq_handle_message
and I can recall it working a month ago.
Everything is working except for this message:
```25(42) ERROR: dmq [notification_peer.c:596]:
notification_resp_callback_f(): deleting server sip:127.0.0.1:5060
because of failed request```
And that's because of the notification address on the kamailio node we
use as DMQ server
This portion of DMQ documentation says that `Default value is "NULL".`
for the notification_address:
https://kamailio.org/docs/modules/5.2.x/modules/dmq.html#dmq.p.notification_address
But with NULL or NOT specifying a value kamailio does not start.
Am I misunderstanding the documentation? Could this be a feature
request or it's a bug?
In a situation like our where kamailio nodes start and stop on demand
without any prior defined topology or IP addresses defined it's hard
to use DMQ module like that. That's why we introduced a DMQ server
with a fixed IP address that it's handling only those packets (but
it's a workaround).
So my next question is:
Do you guys have an idea on how could we handle DMQ on a very
mutational infrastructure?
Example: I have 2 kamailio nodes in the begining (no fixed IPs - using
DHCP), then when there's demand the infrastructure starts another 2
nodes, now I have 4 of them, then 3, one of the nodes randomly is
stopped.
And I want to use DMQ on these node to propagate information without
using a DB. How could we do that?
Also another thing that we noticed is that the last node to be added
is the one with no prior information regarding usrloc for example, it
has only the changes happening from when it starts on. Could there be
a method or modparam flag to force dmq sync of all the information
upon the startup of another node (another possible feature request)?
Kind regards,
--
Aleksandar Sosic
mail: alex.sosic@evosip.cloud
On Sat, Jun 30, 2018 at 11:12 AM Aleksandar Sosic
wrote:
> With a more verbose kamailio (not sure if it helps):
___
Kamailio (SER) - Users Mailing List
sr-users@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Re: [SR-Users] DMQ problems
With a more verbose kamailio (not sure if it helps): ```3(32) DEBUG: [core/udp_server.c:491]: udp_rcv_loop(): received on udp socket: (106/100/520) [[KDMQ sip:notification_peer@10.0.0.102:5060 SIP/2.0 0D 0A Via: SIP/2.0/UDP 10.0.0.49;branch=z9hG4bK361]] 3(32) DEBUG: [core/parser/msg_parser.c:604]: parse_msg(): SIP Request: 3(32) DEBUG: [core/parser/msg_parser.c:606]: parse_msg(): method: 3(32) DEBUG: [core/parser/msg_parser.c:608]: parse_msg(): uri: 3(32) DEBUG: [core/parser/msg_parser.c:610]: parse_msg(): version: 3(32) DEBUG: [core/parser/parse_via.c:1303]: parse_via_param(): Found param type 232, = ; state=16 3(32) DEBUG: [core/parser/parse_via.c:2639]: parse_via(): end of header reached, state=5 3(32) DEBUG: [core/parser/msg_parser.c:492]: parse_headers(): Via found, flags=2 3(32) DEBUG: [core/parser/msg_parser.c:494]: parse_headers(): this is the first via 3(32) DEBUG: [core/parser/parse_addr_spec.c:864]: parse_addr_spec(): end of header reached, state=10 3(32) DEBUG: [core/parser/msg_parser.c:171]: get_hdr_field(): [43]; uri=[sip:notification_peer@10.0.0.102:5060] 3(32) DEBUG: [core/parser/msg_parser.c:172]: get_hdr_field(): to body [ ], to tag [] 3(32) DEBUG: [core/parser/msg_parser.c:152]: get_hdr_field(): cseq : <10> 3(32) DEBUG: [core/receive.c:232]: receive_msg(): --- received sip message - request - call-id: [3d4783ff3ccce81f-64@10.0.0.49] - cseq: [10 KDMQ] 3(32) DEBUG: [core/receive.c:287]: receive_msg(): preparing to run routing scripts... 3(32) DEBUG: [core/parser/msg_parser.c:183]: get_hdr_field(): content_length=73 3(32) DEBUG: [core/parser/msg_parser.c:89]: get_hdr_field(): found end of header 3(32) DEBUG: [core/receive.c:344]: receive_msg(): request-route executed in: 203 usec 3(32) DEBUG: [core/usr_avp.c:636]: destroy_avp_list(): destroying list (nil) 3(32) DEBUG: [core/usr_avp.c:636]: destroy_avp_list(): destroying list (nil) 3(32) DEBUG: [core/usr_avp.c:636]: destroy_avp_list(): destroying list (nil) 3(32) DEBUG: [core/usr_avp.c:636]: destroy_avp_list(): destroying list (nil) 3(32) DEBUG: [core/usr_avp.c:636]: destroy_avp_list(): destroying list (nil) 3(32) DEBUG: [core/usr_avp.c:636]: destroy_avp_list(): destroying list (nil) 3(32) DEBUG: [core/xavp.c:495]: xavp_destroy_list(): destroying xavp list (nil) 3(32) DEBUG: [core/receive.c:447]: receive_msg(): cleaning up ``` ... ``` 9(38) DEBUG: [core/receive.c:232]: receive_msg(): --- received sip message - request - call-id: [0eb7f0c66155bdad-46@10.0.0.102] - cseq: [10 KDMQ] 9(38) DEBUG: [core/receive.c:287]: receive_msg(): preparing to run routing scripts... 9(38) DEBUG: [core/parser/msg_parser.c:183]: get_hdr_field(): content_length=71 9(38) DEBUG: [core/parser/msg_parser.c:89]: get_hdr_field(): found end of header 9(38) DEBUG: [core/receive.c:344]: receive_msg(): request-route executed in: 102 usec 9(38) DEBUG: [core/usr_avp.c:636]: destroy_avp_list(): destroying list (nil) 9(38) DEBUG: [core/usr_avp.c:636]: destroy_avp_list(): destroying list (nil) 9(38) DEBUG: [core/usr_avp.c:636]: destroy_avp_list(): destroying list (nil) 9(38) DEBUG: [core/usr_avp.c:636]: destroy_avp_list(): destroying list (nil) 9(38) DEBUG: [core/usr_avp.c:636]: destroy_avp_list(): destroying list (nil) 9(38) DEBUG: [core/usr_avp.c:636]: destroy_avp_list(): destroying list (nil) 9(38) DEBUG: [core/xavp.c:495]: xavp_destroy_list(): destroying xavp list (nil) 9(38) DEBUG: [core/receive.c:447]: receive_msg(): cleaning up 17(46) DEBUG: tm [t_reply.c:1262]: t_should_relay_response(): -> T_code=0, new_code=408 17(46) DEBUG: tm [t_reply.c:2092]: local_reply(): branch=0, save=0, winner=0 17(46) DEBUG: tm [t_reply.c:2131]: local_reply(): local transaction completed 408/0 (totag retr: 0/1024) 17(46) DEBUG: tm [t_hooks.c:258]: run_trans_callbacks_internal(): DBG: trans=0x7f9093424300, callback type 1024, id 0 entered 17(46) DEBUG: dmq [dmq_funcs.c:61]: dmq_tm_callback(): dmq_tm_callback start 17(46) DEBUG: dmq [notification_peer.c:586]: notification_resp_callback_f(): notification_callback_f triggered [0x 408 (nil)] 17(46) ERROR: dmq [notification_peer.c:596]: notification_resp_callback_f(): deleting server sip:127.0.0.1:5060 because of failed request 17(46) ERROR: dmq [notification_peer.c:599]: notification_resp_callback_f(): not deleting notification_peer 17(46) DEBUG: dmq [dmq_funcs.c:69]: dmq_tm_callback(): dmq_tm_callback done ``` -- Aleksandar Sosic mail: alex.sosic@evosip.cloud On Sat, Jun 30, 2018 at 8:32 AM Aleksandar Sosic wrote: > > Hi Charles, > > The notification address is set to localhost only for the server node > because I have a mutual architecture and don't know which nodes are up > and with which IPs. There could be a possibility that there are no > other kamailio nodes beside the dmq-server. > I'm pretty sure this configuration was working some time ago, We've > since then changed some configuration but no dmq configuration was > changed
Re: [SR-Users] DMQ problems
Hi Charles, The notification address is set to localhost only for the server node because I have a mutual architecture and don't know which nodes are up and with which IPs. There could be a possibility that there are no other kamailio nodes beside the dmq-server. I'm pretty sure this configuration was working some time ago, We've since then changed some configuration but no dmq configuration was changed as I'm aware of. The other thing that changes is the kamailio version. I will try to use a stable version and retry. I've tried a capture with tshark and on the local interface I've got: ```3 0.116690338 10.0.0.101 ? 127.0.0.1SIP 557 Unknown request: KDMQ sip:notification_peer@127.0.0.1:5060 | (text/plain) 4 0.116771925 10.0.0.101 ? 127.0.0.1SIP 557 Unknown request: KDMQ sip:notification_peer@127.0.0.1:5060 | (text/plain) 5 1.116940867 10.0.0.101 ? 127.0.0.1SIP 557 Unknown request: KDMQ sip:notification_peer@127.0.0.1:5060 | (text/plain) 6 3.116729798 10.0.0.101 ? 127.0.0.1SIP 557 Unknown request: KDMQ sip:notification_peer@127.0.0.1:5060 | (text/plain) 7 4.116611072 10.0.0.101 ? 127.0.0.1SIP 557 Unknown request: KDMQ sip:notification_peer@127.0.0.1:5060 | (text/plain) 8 7.116677754 10.0.0.101 ? 127.0.0.1SIP 557 Unknown request: KDMQ sip:notification_peer@127.0.0.1:5060 | (text/plain) 9 8.116711422 10.0.0.101 ? 127.0.0.1SIP 557 Unknown request: KDMQ sip:notification_peer@127.0.0.1:5060 | (text/plain) ``` On the public interface there are packets arriving from the other two kamailio nodes: ``` 221 218.212276413 10.0.0.50 ? 172.22.5.102 SIP 561 Unknown request: KDMQ sip:notification_peer@10.0.0.102:5060 | (text/plain) 222 219.212053642 10.0.0.50 ? 172.22.5.102 SIP 561 Unknown request: KDMQ sip:notification_peer@10.0.0.102:5060 | (text/plain) 223 219.955864183 10.0.0.49 ? 172.22.5.102 SIP 562 Unknown request: KDMQ sip:notification_peer@10.0.0.102:5060 | (text/plain) 224 220.955964403 10.0.0.49 ? 172.22.5.102 SIP 562 Unknown request: KDMQ sip:notification_peer@10.0.0.102:5060 | (text/plain) ``` But in the logs of all three kamailio nodes I've got: ``` 9(64) ERROR: dmq [notification_peer.c:596]: notification_resp_callback_f(): deleting server sip:10.0.0.102:5060 because of failed request 9(64) ERROR: dmq [notification_peer.c:599]: notification_resp_callback_f(): not deleting notification_peer ``` Also no luck with the dmq.list_nodes. Kind regards and thank you for your time, I really appreciate it! -- Aleksandar Sosic mail: alex.sosic@evosip.cloudOn Sat, Jun 23, 2018 at 6:27 PM Charles Chance wrote: > > Hello, > > Is there anything preventing the messages from reaching Kamailio? If you have > a pcap from one of the servers we may be able to see what’s happening. > > Also, you have the notification address set to localhost - this should > instead point to one of the other nodes. > > Cheers, > > Charles ___ Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Re: [SR-Users] DMQ problems
Hello,
Is there anything preventing the messages from reaching Kamailio? If you
have a pcap from one of the servers we may be able to see what’s happening.
Also, you have the notification address set to localhost - this should
instead point to one of the other nodes.
Cheers,
Charles
On Fri, 22 Jun 2018 at 21:35, Aleksandar Sosic
wrote:
> Hi,
>
> I've set up a dmq server for other kamailio nodes with this parameters:
> ```
> modparam("dmq", "server_address","sip:10.0.0.101:5060")
> modparam("dmq", "notification_address", "sip:127.0.0.1:5060")
> modparam("dmq", "multi_notify", 1)
> modparam("dmq", "num_workers", 4)
> modparam("dmq", "ping_interval", 15)
> ```
>
Why are you setting the notification address as localhost?
> On the kamailio nodes the config is the following:
> ```
> modparam("dmq", "server_address", "sip:10.0.0.x:5060")
> modparam("dmq", "notification_address", "sip:10.0.0.101:5060")
> modparam("dmq", "multi_notify", 1)
> modparam("dmq", "num_workers", 4)
> modparam("dmq", "ping_interval", 15)
> ```
>
> On the kamailio dmq server I've got this in the logs:
> ```
> 25(41) ERROR: dmq [notification_peer.c:596]:
> notification_resp_callback_f(): deleting server sip:127.0.0.1:5060
> because of failed request
> 25(41) ERROR: dmq [notification_peer.c:599]:
> notification_resp_callback_f(): not deleting notification_peer
> ```
>
> On the kamailio nodes the same thing with the ip of the dmq server:
> ```
> 25(112) ERROR: dmq [notification_peer.c:596]:
> notification_resp_callback_f(): deleting server sip:10.0.0.101:5060
> because of failed request
> 25(112) ERROR: dmq [notification_peer.c:599]:
> notification_resp_callback_f(): not deleting notification_peer
> ```
>
> The request route on the dmq server is simple as this:
> ```
> request_route {
> if(is_method("KDMQ")){
> dmq_handle_message();
> }
> }#end request_route
> ```
>
> I've tried with some xlog-s inside the if(is_method("KDMQ")) but they
> don't show up so the dmq_handle_message() is not even executed.
>
> I've also tried to bind the dmq server to a different port but with no
> success on making it work.
>
> The dmq.list_nodes shows only the dmq server IP and the current node
> IP, the IPs of other nodes are not shown:
> ```
> root@proxy-94b6ccf46-t8fx2:/# kamcmd dmq.list_nodes
> {
> host: 10.0.0.101
> port: 5060
> resolved_ip: 10.0.0.101
> status: active
> last_notification: 0
> local: 0
> }
> {
> host: 10.0.0.45
> port: 5060
> resolved_ip: 10.0.0.45
> status: active
> last_notification: 0
> local: 1
> }
> ```
> And it's the same on all other kamailio nodes: dmq server ip and local ip.
>
> The kamailio version on all the nodes is:
> 5.2.0~dev6+0~20180616010152.1138+xenial
>
> Thanks,
> --
> Aleksandar Sosic
> mail: alex.sosic@evosip.cloud
>
> ___
> Kamailio (SER) - Users Mailing List
> sr-users@lists.kamailio.org
> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
>
--
*Charles Chance*
Managing Director
t. 0330 120 1200m. 07932 063 891
--
Sipcentric Ltd.
Company registered in England & Wales no.
7365592. Registered
office: Faraday Wharf, Innovation
Birmingham Campus, Holt Street, Birmingham Science Park, Birmingham B7 4BB.
___
Kamailio (SER) - Users Mailing List
sr-users@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
