Re: [SR-Users] Kamctl Stats Update

[Daniel Greenwald]

Point was that
kamctl rpc stats.fetch usrloc
or
kamctl rpc stats.fetch shmem

both return

{
 "jsonrpc":  "2.0",
 "error":  {
   "code": 500,
   "message":  "Method Not Found"
 },
 "id": 29575
}

shmem was the specific example given by Daniel Constantin Mierla so I'm not
sure why its not working. I ended up using "kamcmd stats.get_statistics
all" and parsing the raw text returned there.


On Mon, Aug 6, 2018 at 2:53 PM, Henning Westerholt  wrote:

> Am Montag, 6. August 2018, 19:23:33 CEST schrieb Daniel Greenwald:
> > Looking to get usrloc stats in pure json format.
> >
> > When trying to run from the example above:
> >
> >  kamctl rpc stats.fetch shmem:
> >
> > I get a message of Method Not Found
> >
> >
> > Was this command not released, or is there another way to do this?
> > [..]
>
> Hello,
>
> probably a stupid question, but do you tried to execute the command with
> ":"
> at the end? Do you get a result with kamctl rpc stats.fetch shmem?
>
> And why are you looking into the shmem stats for usrloc statistics?
>
> Best regards,
>
> Henning
>
> --
> Henning Westerholt
> https://skalatan.de/blog/
>
___
Kamailio (SER) - Users Mailing List
sr-users@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

Re: [SR-Users] RTPEngine UDP receive queue issue

[Alex Balashov]

From the 5.1 TB transferred on the 'lo' interface (RTPEngine running on
same host as Kamailio):

lo: flags=73  mtu 65536
inet 127.0.0.1  netmask 255.0.0.0
inet6 ::1  prefixlen 128  scopeid 0x10
loop  txqueuelen 1  (Local Loopback)
RX packets 27694476978  bytes 5678003789723 (5.1 TiB)
RX errors 0  dropped 0  overruns 0  frame 0
TX packets 27694476978  bytes 5678003789723 (5.1 TiB)
TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

... I would surmise that this is some sort of loop / infinite loop
condition, but I can't quite put my finger on what it is.

From an RTPEngine logging standpoint, the scenario begins looking rather
normally:

Aug  6 18:11:04 gw1 rtpengine[25476]: INFO: [1074537188_44545952@20.20.20.20]: 
Received command 'offer' from 127.0.0.1:48525
Aug  6 18:11:04 gw1 rtpengine[25476]: NOTICE: 
[1074537188_44545952@20.20.20.20]: Creating new call
Aug  6 18:11:04 gw1 rtpengine[25476]: INFO: [1074537188_44545952@20.20.20.20]: 
offer time = 0.000865 sec
Aug  6 18:11:04 gw1 rtpengine[25476]: INFO: [1074537188_44545952@20.20.20.20]: 
Replying to 'offer' from 127.0.0.1:48525
Aug  6 18:11:14 gw1 rtpengine[25476]: INFO: [1074537188_44545952@20.20.20.20]: 
Received command 'answer' from 127.0.0.1:58668
Aug  6 18:11:14 gw1 rtpengine[25476]: INFO: [1074537188_44545952@20.20.20.20]: 
answer time = 0.000123 sec
Aug  6 18:11:14 gw1 rtpengine[25476]: INFO: [1074537188_44545952@20.20.20.20]: 
Replying to 'answer' from 127.0.0.1:58668
Aug  6 18:11:18 gw1 rtpengine[25476]: INFO: [1074537188_44545952@20.20.20.20 
port 61666]: Confirmed peer address as 2.2.2.2:38236
Aug  6 18:11:23 gw1 rtpengine[25476]: INFO: [1074537188_44545952@20.20.20.20]: 
Received command 'offer' from 127.0.0.1:50995
Aug  6 18:11:23 gw1 rtpengine[25476]: INFO: [1074537188_44545952@20.20.20.20]: 
offer time = 0.82 sec
Aug  6 18:11:23 gw1 rtpengine[25476]: INFO: [1074537188_44545952@20.20.20.20]: 
Replying to 'offer' from 127.0.0.1:50995
Aug  6 18:11:23 gw1 rtpengine[25476]: INFO: [1074537188_44545952@20.20.20.20]: 
Received command 'offer' from 127.0.0.1:47321
Aug  6 18:11:23 gw1 rtpengine[25476]: INFO: [1074537188_44545952@20.20.20.20]: 
offer time = 0.000151 sec
Aug  6 18:11:23 gw1 rtpengine[25476]: INFO: [1074537188_44545952@20.20.20.20]: 
Replying to 'offer' from 127.0.0.1:47321
Aug  6 18:11:23 gw1 rtpengine[25476]: INFO: [1074537188_44545952@20.20.20.20 
port 61666]: Switching local interface to 1.1.1.1:61666
Aug  6 18:11:23 gw1 rtpengine[25476]: ERR: [1074537188_44545952@20.20.20.20 
port 61666]: Too many packets in UDP receive queue (more than 50), aborting 
loop. Dropped packets possible
[... ad nauseum... ]

The only thing unusual about it from a SIP perspective is that it
involves an SRTP endpoint and several reinvites which don't seem to be
handled as one would expect:

Aug  6 18:11:04 gw1 /usr/local/sbin/kamailio[22070]: INFO: 
[R-INBOUND-ROUTE-BRANCH:1074537188_44545952@2.2.2.2] Relaying media through 
RTPEngine
Aug  6 18:11:04 gw1 /usr/local/sbin/kamailio[22070]: INFO: 
[R-INBOUND-ROUTE-BRANCH:1074537188_44545952@2.2.2.2] -> Forcing Secure RTP 
(SRTP) profile in offer
Aug  6 18:11:04 gw1 /usr/local/sbin/kamailio[22233]: INFO: 
[R-MAIN-REPLY:1074537188_44545952@2.2.2.2] Reply '100 Trying' from 3.3.3.3:54246
Aug  6 18:11:04 gw1 /usr/local/sbin/kamailio[22233]: INFO: 
[R-MAIN-REPLY:1074537188_44545952@2.2.2.2] Reply '180 Ringing' from 
3.3.3.3:54246
Aug  6 18:11:14 gw1 /usr/local/sbin/kamailio[22258]: INFO: 
[R-MAIN-REPLY:1074537188_44545952@2.2.2.2] Reply '200 OK' from 3.3.3.3:54246
Aug  6 18:11:14 gw1 /usr/local/sbin/kamailio[22258]: INFO: 
[R-MAIN-REPLY:1074537188_44545952@2.2.2.2] -> Reply contains SDP;  diverting 
through RTPEngine
Aug  6 18:11:14 gw1 /usr/local/sbin/kamailio[22112]: INFO: 
[R-MAIN:1074537188_44545952@2.2.2.2] -> Other sequential ACK received from 
2.2.2.2:5060
Aug  6 18:11:23 gw1 /usr/local/sbin/kamailio[22263]: INFO: 
[R-MAIN:1074537188_44545952@2.2.2.2] -> Re-invite received on rtpengine'd call 
-- updating RTPEngine
Aug  6 18:11:23 gw1 /usr/local/sbin/kamailio[22068]: INFO: 
[R-MAIN:1074537188_44545952@2.2.2.2] -> Re-invite received on rtpengine'd call 
-- updating RTPEngine
Aug  6 18:11:23 gw1 /usr/local/sbin/kamailio[22074]: INFO: 
[R-MAIN-REPLY:1074537188_44545952@2.2.2.2] Reply '100 Trying' from 1.1.1.1:5060
Aug  6 18:11:23 gw1 /usr/local/sbin/kamailio[22057]: INFO: 
[R-MAIN-REPLY:1074537188_44545952@2.2.2.2] Reply '100 Trying' from 2.2.2.2:5060
Aug  6 18:11:23 gw1 /usr/local/sbin/kamailio[22070]: INFO: 
[R-MAIN-REPLY:1074537188_44545952@2.2.2.2] Reply '200 OK' from 2.2.2.2:5060
Aug  6 18:11:23 gw1 /usr/local/sbin/kamailio[22070]: INFO: 
[R-MAIN-REPLY:1074537188_44545952@2.2.2.2] -> Reply contains SDP;  diverting 
through RTPEngine
Aug  6 18:11:27 gw1 /usr/local/sbin/kamailio[22065]: INFO: 
[R-MAIN-REPLY:1074537188_44545952@2.2.2.2] Reply '200 OK' from 2.2.2.2:5060
Aug  6 18:11:27 gw1 /usr/local/sbin/kamailio[22065]: INFO: 

[SR-Users] RTPEngine UDP receive queue issue

[Alex Balashov]

Hi, 

We have an RTPEngine that occasionally finds itself in this state of CPU
usage:

  PID USER  PR  NIVIRTRESSHR S  %CPU %MEM TIME+ COMMAND
25476 root  20   0 2929600  80820   1900 S 156.2  0.1 259:05.54 rtpengine

The journal is a dizzying stream of:

Aug 06 20:58:47 gw1 rtpengine[25476]: ERR: [1074537188_44545952 port 61666]: 
Too many packets in UDP receive queue (more than 50), aborting loop. Dropped 
packets possible

It eventually leads to increasing CPU usage and a degeneration of audio
quality until RTPEngine is restarted.

It would be appreciated if someone could shed some light on the nature
of this queue and what we can do about this problem. Is this referring
to a netfilter packet forwarding queue, or is it a receive queue of the
RTPengine user space control process? Can its size be increased or
tweaked with kernel parameters?

Thanks,

-- Alex

-- 
Alex Balashov | Principal | Evariste Systems LLC

Tel: +1-706-510-6800 / +1-800-250-5920 (toll-free) 
Web: http://www.evaristesys.com/, http://www.csrpswitch.com/

___
Kamailio (SER) - Users Mailing List
sr-users@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

Re: [SR-Users] IRC Channel

[Daniel-Constantin Mierla]

Hello,

I tried to do it via web chat earlier today, being on a trip with limited
internet access, I am not sure if succeeded. Can someone connected to
#kamailio channel on irc freenode.net confirm that spam messages stopped?

Cheers,
Daniel

On Mon, Aug 6, 2018 at 2:13 PM, Enrico Bandiera <
enrico.bandi...@cloud.timenet.it> wrote:

> Hi, currenty (the last few days) there has been a spam attack on freenode,
> #Kamailio is currently full of spam messages, I was wondering if who has
> operator rights could set the channel to +r (only registered users can send
> messages) as suggested by freeenode admins
>
> The channel usually has only few messages and with all this spam it
> becomes impossible at all to use it.
>
> Greetings,
> Enrico.
>
> ___
> Kamailio (SER) - Users Mailing List
> sr-users@lists.kamailio.org
> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
>
>


-- 
Daniel-Constantin Mierla - http://www.asipto.com
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
___
Kamailio (SER) - Users Mailing List
sr-users@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

Re: [SR-Users] DMQ mem leak issues

[Rogelio Perez]

Charles, Julien, Daniel,

The results are pretty much the same, the mem leak is still there and we
need to restart Kamailio when it reaches certain threshold.
https://www.dropbox.com/s/enxx6b7t0c8vl49/Selection_539.png?dl=0

Is there anything else we can try?
Will a core dump file tell us what's causing it?

Thanks,
Rogelio

On Thu, Aug 2, 2018 at 2:57 PM Rogelio Perez  wrote:

> Thanks Charles, it's working now.
> I'm deploying to production and confirming results soon.
>
> Rogelio
>


-- 

Rogelio Perez | engineering | telnyx 
chicago: +1 312 270 8119 | dublin: +353 1 912 6119
___
Kamailio (SER) - Users Mailing List
sr-users@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

Re: [SR-Users] Kamctl Stats Update

[Henning Westerholt]

Am Montag, 6. August 2018, 19:23:33 CEST schrieb Daniel Greenwald:
> Looking to get usrloc stats in pure json format.
> 
> When trying to run from the example above:
> 
>  kamctl rpc stats.fetch shmem:
> 
> I get a message of Method Not Found
> 
> 
> Was this command not released, or is there another way to do this?
> [..]

Hello,

probably a stupid question, but do you tried to execute the command with ":" 
at the end? Do you get a result with kamctl rpc stats.fetch shmem?

And why are you looking into the shmem stats for usrloc statistics?

Best regards,

Henning

-- 
Henning Westerholt
https://skalatan.de/blog/

___
Kamailio (SER) - Users Mailing List
sr-users@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

Re: [SR-Users] Kamctl Stats Update

[Daniel Greenwald]

Looking to get usrloc stats in pure json format.

When trying to run from the example above:

 kamctl rpc stats.fetch shmem:

I get a message of Method Not Found


Was this command not released, or is there another way to do this?

Thanks


On Thu, Sep 21, 2017 at 10:53 AM, Daniel-Constantin Mierla <
mico...@gmail.com> wrote:

> Hello,
>
> jsonrcps is the module required by kamctl starting with version 5.0
> (being the module renamed from jsonrpc-s).
>
> Then, back to the main topic here -- the output for 'kamctl stats' was
> inherited from old times when MI (with no standard format in its output)
> was used to interact with kamailio and it was like a printed string in
> the form of "group.name = value" for each of the available statistics.
>
> As I also wanted for quite long time to get a more json friendly output
> for stats, this discussion brought it back in my attention and I just
> added the rpc command stats.fetch. This one returns a json structure
> like in next example for getting shared memory stats:
>
> # kamctl rpc stats.fetch shmem:
> {
>   "jsonrpc":  "2.0",
>   "result": {
> "shmem.fragments":  "1",
> "shmem.free_size":  "64288976",
> "shmem.max_used_size":  "2819888",
> "shmem.real_used_size": "2819888",
> "shmem.total_size": "67108864",
> "shmem.used_size":  "2578288"
>   },
>   "id": 44590
> }
>
> I left the value as string in order to accommodate large numbers (as the
> rpc interface works usually with integers), but if people finds it
> inconvenient, I can look at seeing if large numbers are actually needed
> here.
>
> Cheers,
> Daniel
>
>
> On 20.09.17 19:34, Noah Mehl wrote:
> > Alex,
> >
> > We are using this for time series monitoring (e.g. Zabbix).  It doesn’t
> make sense, at least to me, to implement jsonrpc-s just to get the kamctl
> stats output.  I mean, currently I’m just chaining the output with cut and
> tr, and that’s fine.  I just suggest utilizing JSON a bit better here.
> >
> > Thanks!
> >
> > ~Noah
> >
> >> On Sep 20, 2017, at 1:17 PM, Alex Balashov 
> wrote:
> >>
> >> You may want to consider an alternate and more streamlined method of
> pulling these.
> >>
> >> On September 20, 2017 1:16:49 PM EDT, Noah Mehl 
> wrote:
> >>> Alex,
> >>>
> >>> This is how that output was generated:
> >>>
> >>> # kamctl stats shmem | jq .
> >>>
> >>> Thanks!
> >>>
> >>> ~Noah
> >>>
>  On Sep 20, 2017, at 1:14 PM, Alex Balashov
> >>>  wrote:
>  Hello,
> 
>  The jsonrpc-s module has a pretty_print option. Or is that not where
> >>> you're dispatching this JSON output from?
> 
>  -- Alex
> 
>  --
>  Sent via mobile, please forgive typos and brevity.
> 
>  ___
>  Kamailio (SER) - Users Mailing List
>  sr-users@lists.kamailio.org
>  https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
> >>>
> >>> ___
> >>> Kamailio (SER) - Users Mailing List
> >>> sr-users@lists.kamailio.org
> >>> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
> >>
> >> -- Alex
> >>
> >> --
> >> Sent via mobile, please forgive typos and brevity.
> >>
> >> ___
> >> Kamailio (SER) - Users Mailing List
> >> sr-users@lists.kamailio.org
> >> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
> >
> > ___
> > Kamailio (SER) - Users Mailing List
> > sr-users@lists.kamailio.org
> > https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
>
> --
> Daniel-Constantin Mierla
> www.twitter.com/miconda -- www.linkedin.com/in/miconda
> Kamailio Advanced Training - www.asipto.com
> Kamailio World Conference - www.kamailioworld.com
>
>
> ___
> Kamailio (SER) - Users Mailing List
> sr-users@lists.kamailio.org
> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
>
___
Kamailio (SER) - Users Mailing List
sr-users@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

Re: [SR-Users] UDP workers block when one or more rtpengine instances go offline

[Daniel Tryba]

On Mon, Aug 06, 2018 at 02:37:15PM +0200, Sebastian Damm wrote:
> Of course, that's probably the better way. But as far as I know, that
> command wasn't available before 5.0. So I guess our blocking of the
> control port was the way we did it before that.
> 
> But anyhow, if an rtpengine crashes, Kamailio shouldn't block.

BTW forgot to ask: are you REJECTing or DROPing packets?
A reject should trigger a failover in the rtpengine_* calls immediately.
A drop will result in a timeout mechanism triggering, which according to
your description blocks the thread.


signature.asc
Description: PGP signature
___
Kamailio (SER) - Users Mailing List
sr-users@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

Re: [SR-Users] UDP workers block when one or more rtpengine instances go offline

[Daniel Tryba]

On Mon, Aug 06, 2018 at 02:37:15PM +0200, Sebastian Damm wrote:
> > kamcmd rtpengine.enable udp:x.y.z.a:port 0/1
> > on the kamailio machines?
> 
> Of course, that's probably the better way. But as far as I know, that
> command wasn't available before 5.0. So I guess our blocking of the
> control port was the way we did it before that.

In the past (4.x) the command was:
kamctl fifo nh_enable_rtpp udp:x.y.z.a:port 0/1




signature.asc
Description: PGP signature
___
Kamailio (SER) - Users Mailing List
sr-users@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

Re: [SR-Users] UDP workers block when one or more rtpengine instances go offline

[Sebastian Damm]

Hi Daniel,

On Mon, Aug 6, 2018 at 2:17 PM Daniel Tryba  wrote:
> No answer to you question (which sounds like a legit problem), but why
> not do it with:
> kamcmd rtpengine.enable udp:x.y.z.a:port 0/1
> on the kamailio machines?

Of course, that's probably the better way. But as far as I know, that
command wasn't available before 5.0. So I guess our blocking of the
control port was the way we did it before that.

But anyhow, if an rtpengine crashes, Kamailio shouldn't block.

Sebastian

___
Kamailio (SER) - Users Mailing List
sr-users@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

Re: [SR-Users] UDP workers block when one or more rtpengine instances go offline

[Daniel Tryba]

On Mon, Aug 06, 2018 at 12:58:00PM +0200, Sebastian Damm wrote:
> we run multiple rtpengine servers to share the load. Whenever we need
> to take an rtpengine server offline, we used to just block the control
> port via iptables, then no new calls ended up on this instance of
> rtpengine. This worked pretty well in Kamailio 4.4.5.

No answer to you question (which sounds like a legit problem), but why
not do it with:
kamcmd rtpengine.enable udp:x.y.z.a:port 0/1
on the kamailio machines?
 
By simply blocking you might interrupt updates in SDP for running calls
and prevent hangups from being sent.



signature.asc
Description: PGP signature
___
Kamailio (SER) - Users Mailing List
sr-users@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

[SR-Users] IRC Channel

[Enrico Bandiera]

Hi, currenty (the last few days) there has been a spam attack on freenode,
#Kamailio is currently full of spam messages, I was wondering if who has
operator rights could set the channel to +r (only registered users can send
messages) as suggested by freeenode admins

The channel usually has only few messages and with all this spam it becomes
impossible at all to use it.

Greetings,
Enrico.
___
Kamailio (SER) - Users Mailing List
sr-users@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

[SR-Users] UDP workers block when one or more rtpengine instances go offline

[Sebastian Damm]

Hi,

we run multiple rtpengine servers to share the load. Whenever we need
to take an rtpengine server offline, we used to just block the control
port via iptables, then no new calls ended up on this instance of
rtpengine. This worked pretty well in Kamailio 4.4.5.

However, since Kamailio 5.0, and the problem persists with 5.1.4,
Kamailio hangs almost immediately after we block the control port
traffic. In the log file there are almost no packets processed except
every few seconds, which looks like a timeout thing.

Did we configure anything wrong there? Or is the "dead rtpengine
detection" just broken?

Our configuration:

loadmodule "rtpengine.so"
modparam("rtpengine", "rtpengine_disable_tout", 120)
modparam("rtpengine", "setid_avp", "$avp(rtpsetid)")
modparam("rtpengine", "rtpengine_sock", "0 == udp:1.2.3.4:9001=2
udp:1.2.3.5:9001=2 udp:1.2.3.6:9001=2")
modparam("rtpengine", "rtpengine_sock", "1 == udp:2.3.4.5:9001=2")

Any help appreciated.

Regards,
Sebastian

___
Kamailio (SER) - Users Mailing List
sr-users@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users