Re: [sr-dev] [kamailio/kamailio] t_replicate transaction failures keeps transaction in memory forever (#1220)

[Olle E. Johansson]

I mentioned that, but did not change the topic of the issue... Latest 5.0 from 
git being used here, not 5.1. Wanted to find this issue before upgrading.

-- 
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1220#issuecomment-357650887___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

Re: [sr-dev] [kamailio/kamailio] t_replicate transaction failures keeps transaction in memory forever (#1220)

[Daniel-Constantin Mierla]

What version are you running now there?

If the issue is reproducible with a lot of calls via sipp, then it sounds like 
not being related to t_replicate() of registrations...

-- 
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1220#issuecomment-357607076___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

Re: [sr-dev] [kamailio/kamailio] t_replicate transaction failures keeps transaction in memory forever (#1220)

[Olle E. Johansson]

The server was running over christmas with no issues (but no calls either). 
Registrations doesn't seem to be the major problem (and thus not replication). 
I can force a memory issue by running a lot of calls with SIPP.

Another observation is that the timer process runs up to 99% of the CPU it's 
running on (observed with top). I have moved dialog, usrloc, rtimer and 
everything else I can move out of the generic timer process, so this points to 
a problem with TM and timers. I will review our settings there and keep testing.

-- 
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1220#issuecomment-357606036___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

Re: [sr-dev] [kamailio/kamailio] t_replicate transaction failures keeps transaction in memory forever (#1220)

[Olle E. Johansson]

Finally have a day where it's possible to run som tests. Will try to fire up 
GDB :-)
Building latest 5.0 from git.

-- 
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1220#issuecomment-345624236___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

Re: [sr-dev] [kamailio/kamailio] t_replicate transaction failures keeps transaction in memory forever (#1220)

[Olle E. Johansson]

This is still a problem, even though we managed to push it back a little. After 
a few weeks of service, kamailio stopped working properly because it was out of 
memory. Had to restart quickly because of other dependencies (important tests 
of devices) but will try to isolate the problem with small configuration files 
this week.


-- 
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1220#issuecomment-340797794___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

Re: [sr-dev] [kamailio/kamailio] t_replicate transaction failures keeps transaction in memory forever (#1220)

[Olle E. Johansson]

Sorry, can't disturb the system until middle of week. Will test then.

-- 
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1220#issuecomment-330149090___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

Re: [sr-dev] [kamailio/kamailio] t_replicate transaction failures keeps transaction in memory forever (#1220)

[Daniel-Constantin Mierla]

Any chance that you can test with latest master branch? I pushed some new 
features to list transactions via rpc as well as trigger a clean via rpc. 
Alternative would be to backport last patches from tm module in your clone.

I will also add an option to do hard clean on time, but it would be good to 
know if it works when triggered via rpc. Also, the list of transactions via rpc 
will be useful to do when the issue is exposed, as alternative to gdb script.

-- 
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1220#issuecomment-329402631___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

Re: [sr-dev] [kamailio/kamailio] t_replicate transaction failures keeps transaction in memory forever (#1220)

[Daniel-Constantin Mierla]

I couldn't find a reason after looking around, likely something is triggered at 
some point, in certain conditions.

As a workaround, I am considering to add a hard cleanup on timer to the 
transactions that leave too long after lifetime, with a log message to gather 
some useful information for troubleshooting.

If you can still get the output for gdb script on such transactions is still 
useful.

-- 
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1220#issuecomment-328803937___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

Re: [sr-dev] [kamailio/kamailio] t_replicate transaction failures keeps transaction in memory forever (#1220)

[Daniel-Constantin Mierla]

Did you get the chance to run the gdb script from my previous comment?

Is the shm memory usage still growing without t_replicate?

I master I added a rpc command that prints some details of the transactions in 
memory, that should make it easier in the future to see what's there.

-- 
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1220#issuecomment-328452832___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

Re: [sr-dev] [kamailio/kamailio] t_replicate transaction failures keeps transaction in memory forever (#1220)

[Olle E. Johansson]

There is something fishy going on here. I removed t_replicate and the server 
has been running for six days without getting into trouble. 
However when I run kamcmd tm.stats there are 767 waiting transactions, not 
freed. It's a very basic configuration using the dialog module, the dialog 
event routes. A few calls, but not much over the weekend. I will try to debug a 
big harder and find out where these come from.  T_replicate really forced the 
issue and the server bogged down quickly, but it seems like ordinary calls also 
get transactions that hang around. That's not good.
```

{
current: 780
waiting: 767
total: 792
total_local: 0
rpl_received: 1389
rpl_generated: 339
rpl_sent: 1514
6xx: 4
5xx: 285
4xx: 172
3xx: 0
2xx: 519
created: 792
freed: 11
delayed_free: 0
}

```

-- 
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1220#issuecomment-328437649___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

Re: [sr-dev] [kamailio/kamailio] t_replicate transaction failures keeps transaction in memory forever (#1220)

[Daniel-Constantin Mierla]

Here is a gdb script snippet to print transaction structures from memory -- 
hopefully works ok, not tested here yet:

```
set $i=0
while($ientries[$i]
set $tcell = _tm_table->entries[$i].next_c

while($ehead!=(void*)$tcell)
p $tcell
p *$tcell
set $tcell = $tcell->next_c;
end

set $i = $i + 1
end
```

Try to attach to last udp worker, so traffic routing goes ok. You may need to 
replace `TABLE_ENTRIES` with `2^16` if gdb complains that it cannot find its 
value.

-- 
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1220#issuecomment-327177060___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

Re: [sr-dev] [kamailio/kamailio] t_replicate transaction failures keeps transaction in memory forever (#1220)

[Olle E. Johansson]

replication stops. Can't find anything unusual in the syslogs. Can attach GDB, 
but have no clue on how to look at the transaction structures. Need some help 
:-)

-- 
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1220#issuecomment-326979140___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

Re: [sr-dev] [kamailio/kamailio] t_replicate transaction failures keeps transaction in memory forever (#1220)

[Daniel-Constantin Mierla]

Quite strange ... any relevant log messages in syslog around the time when it 
goes astray? Are any new registrations still replicated (sent out) and replied?

Would you be able to attach with gdb to one of the kamailio processes and look 
at the transaction structures?

-- 
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1220#issuecomment-326971517___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

Re: [sr-dev] [kamailio/kamailio] t_replicate transaction failures keeps transaction in memory forever (#1220)

[Olle E. Johansson]

Started test 12:42, all seems quiet and current goes up and down around 20. At 
13:14 it begins to climb again, so around 30 minutes quiet. We have two-minute 
registration timeouts, so at the time it starts running astray we have 
processed 6762 transactions. All over UDP.

-- 
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1220#issuecomment-326965067___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

Re: [sr-dev] [kamailio/kamailio] t_replicate transaction failures keeps transaction in memory forever (#1220)

[Olle E. Johansson]

After 30 minutes, everything looks fine... So it takes a while before the TM 
queue starts to build


-- 
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1220#issuecomment-326936424___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

Re: [sr-dev] [kamailio/kamailio] t_replicate transaction failures keeps transaction in memory forever (#1220)

[Olle E. Johansson]

After three hours I have 14726 transactions waiting using around 300 accounts 
for REGISTER.
I will restart now and see how fast I notice a new queue.

-- 
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1220#issuecomment-326930453___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

Re: [sr-dev] [kamailio/kamailio] t_replicate transaction failures keeps transaction in memory forever (#1220)

[Daniel-Constantin Mierla]

Could you spot if it appears from first replication or starts after a while? 
Are you having any test scenario that could be reused to reproduce it? As wrote 
previously, I tested with default config + t_replicate_tcp() for each register 
and worked.

-- 
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1220#issuecomment-326892948___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

Re: [sr-dev] [kamailio/kamailio] t_replicate transaction failures keeps transaction in memory forever (#1220)

[Olle E. Johansson]

After a bit mor testing of t_replicate_udp I am sorry to say that we still have 
a problem with that. Firewall is open. We see the same locked transactions and 
replication has stopped, the other side has an empty usrloc. Kamailio from the 
5.0 branch of git before the 5.0.3 release.
Version:
```
# kamailio -v
version: kamailio 5.0.2 (x86_64/linux) 
```

```
# kamcmd tm.stats
{
current: 19974
waiting: 19974
total: 534594
total_local: 0
rpl_received: 544239
rpl_generated: 577
rpl_sent: 534606
6xx: 0
5xx: 4533
4xx: 584
3xx: 0
2xx: 529484
created: 534594
freed: 514619
delayed_free: 0
}

```

One server replicating with udp, the other with tcp. I see TCP keepalives 
flowing, so no blocking any more.

-- 
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1220#issuecomment-326885271___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

Re: [sr-dev] [kamailio/kamailio] t_replicate transaction failures keeps transaction in memory forever (#1220)

[Daniel-Constantin Mierla]

Latest release as snapshot/tarball 5.0.2, or latest branch 5.0 at some point in 
time?

-- 
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1220#issuecomment-326302491___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

Re: [sr-dev] [kamailio/kamailio] t_replicate transaction failures keeps transaction in memory forever (#1220)

[Olle E. Johansson]

Latest release. Can't access parameters until monday.

-- 
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1220#issuecomment-326301700___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

Re: [sr-dev] [kamailio/kamailio] t_replicate transaction failures keeps transaction in memory forever (#1220)

[Daniel-Constantin Mierla]

I tried to reproduce using master, but all ok. Based on default kamailio.cfg, I 
have added t_replicate_tcp:

```
# Handle SIP registrations
route[REGISTRAR] {
if (!is_method("REGISTER")) return;

if(isflagset(FLT_NATS)) {
setbflag(FLB_NATB);
#!ifdef WITH_NATSIPPING
# do SIP NAT pinging
setbflag(FLB_NATSIPPING);
#!endif
}
if (!save("location")) {
sl_reply_error();
}
t_replicate_tcp("127.0.0.1", 9);
exit;
}
```

In logs I get:

```
39(81014) ERROR:  [core/tcp_main.c:4135]: handle_tcpconn_ev(): connect 
127.0.0.1:9 failed
```

and after a while (delete wait time) transaction is cleaned:

```
25(81000) DEBUG: tm [t_reply.c:1263]: t_should_relay_response(): -> 
T_code=0, new_code=408
25(81000) DEBUG: tm [t_reply.c:2072]: local_reply(): branch=0, save=0, winner=0
25(81000) DEBUG: tm [t_reply.c:2111]: local_reply(): local transaction 
completed 408/0 (totag retr: 0/0)
26(81001) DEBUG: tm [h_table.c:129]: free_cell_helper(): freeing transaction 
0x105712170 from timer.c:651
```

`tm.stats` showing 0 current transactions.

What version are you using? What are the parameters you set for tm module?

-- 
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1220#issuecomment-326297453___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

Re: [sr-dev] [kamailio/kamailio] t_replicate transaction failures keeps transaction in memory forever (#1220)

[Daniel-Constantin Mierla]

So connections are no ok, but the transactions still live forever, right?

Do you know if the replicated requests were replied? Or the replicated 
registrations were handled in the location table of the pair server?

-- 
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1220#issuecomment-326246424___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

Re: [sr-dev] [kamailio/kamailio] t_replicate transaction failures keeps transaction in memory forever (#1220)

[Daniel-Constantin Mierla]

@oej - were those transactions counted in `current` field of the stats?

-- 
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1220#issuecomment-325574130___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

Re: [sr-dev] [kamailio/kamailio] t_replicate transaction failures keeps transaction in memory forever (#1220)

[Olle E. Johansson]

Note for the future: Maybe an event_route for failed replications would be a 
good thing.

-- 
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1220#issuecomment-325332824___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

Re: [sr-dev] [kamailio/kamailio] t_replicate transaction failures keeps transaction in memory forever (#1220)

[Olle E. Johansson]

Not iptables

-- 
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1220#issuecomment-325331558___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

Re: [sr-dev] [kamailio/kamailio] t_replicate transaction failures keeps transaction in memory forever (#1220)

[Alekzander Spiridonov]

If FW was iptables - it doesn't send neither FIN nor RST to existing 
connections.

-- 
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1220#issuecomment-325330627___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

Re: [sr-dev] [kamailio/kamailio] t_replicate transaction failures keeps transaction in memory forever (#1220)

[Olle E. Johansson]

What happened was that the firewall closed the TCP connection after a while, 
Kamailio continued sending but got dropped. 

-- 
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1220#issuecomment-325330425___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

Re: [sr-dev] [kamailio/kamailio] t_replicate transaction failures keeps transaction in memory forever (#1220)

[Alekzander Spiridonov]

I bet the problem was with established connections been dropped with a 
firewall. TCP takes ages to timeout on Linux.

-- 
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1220#issuecomment-325330403___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

Re: [sr-dev] [kamailio/kamailio] t_replicate transaction failures keeps transaction in memory forever (#1220)

[Alekzander Spiridonov]

If transaction relies on tcp stack than it could depend on whether firewall
dropped or rejected connections in your initial case. As for "UDP with a
random port" it was most probably icmp rejected.

2017-08-28 14:30 GMT+03:00 Olle E. Johansson :

> Changed to t_replicate_udp with a random port and it seems to not behave
> in the same way. I see that transactions are freed after a timeout.
>
> # kamcmd tm.stats
> {
>   current: 119
>   waiting: 17
>   total: 462
>   total_local: 0
>   rpl_received: 0
>   rpl_generated: 360
>   rpl_sent: 360
>   6xx: 0
>   5xx: 0
>   4xx: 360
>   3xx: 0
>   2xx: 0
>   created: 462
>   freed: 343
>   delayed_free: 0
> }
>
>
>
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> ,
> or mute the thread
> 
> .
>
> ___
> Kamailio (SER) - Development Mailing List
> sr-dev@lists.kamailio.org
> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev
>
>


-- 
Best regards,
Alekzander Spiridonov
___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

Re: [sr-dev] [kamailio/kamailio] t_replicate transaction failures keeps transaction in memory forever (#1220)

[Olle E. Johansson]

Changed to t_replicate_udp with a random port and it seems to not behave in the 
same way. I see that transactions are freed after a timeout.

```
# kamcmd tm.stats
{
current: 119
waiting: 17
total: 462
total_local: 0
rpl_received: 0
rpl_generated: 360
rpl_sent: 360
6xx: 0
5xx: 0
4xx: 360
3xx: 0
2xx: 0
created: 462
freed: 343
delayed_free: 0
}


```

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1220#issuecomment-325328754___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

Re: [sr-dev] [kamailio/kamailio] t_replicate transaction failures keeps transaction in memory forever (#1220)

[Olle E. Johansson]

I can check with UDP next week. 

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1220#issuecomment-325139566___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev