Re: [sr-dev] [kamailio/kamailio] Segfaults in 5.1.4 (maybe after one rtpengine disappeared) (#1613)

[Daniel-Constantin Mierla]

Closed #1613.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1613#event-2193112660___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

Re: [sr-dev] [kamailio/kamailio] Segfaults in 5.1.4 (maybe after one rtpengine disappeared) (#1613)

[Daniel-Constantin Mierla]

This should be fixed in master and 5.2 branches (commit 
814d5cc1f4f5b1e4b95737108dffc1e7d7bd566f).

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1613#issuecomment-471442072___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

Re: [sr-dev] [kamailio/kamailio] Segfaults in 5.1.4 (maybe after one rtpengine disappeared) (#1613)

[marcinkowalczyk]

Hi will try to test it tomorrow and check how it goes

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1613#issuecomment-444208955___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

Re: [sr-dev] [kamailio/kamailio] Segfaults in 5.1.4 (maybe after one rtpengine disappeared) (#1613)

[Daniel-Constantin Mierla]

Anyone able to test with the commit referenced above (I just pushed the patch 
to master)? Might have been an issue caused by a delayed processing (eg., 
waiting for rtp engine command execution timeout) that also induced some 
response retransmissions. I want to be sure it doesn't have side effects in 
some way, so be careful, watching closely if you push it to production.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1613#issuecomment-443242943___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

Re: [sr-dev] [kamailio/kamailio] Segfaults in 5.1.4 (maybe after one rtpengine disappeared) (#1613)

[marcinkowalczyk]

Hi,

Seems that 5.1.6 is afected too, but I don't have coredump from 5.1.6 (yet)

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1613#issuecomment-442837050___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

Re: [sr-dev] [kamailio/kamailio] Segfaults in 5.1.4 (maybe after one rtpengine disappeared) (#1613)

[marcinkowalczyk]

Hi,

>From kam logs looks crash was related to ACK for CANCEL message when proxy 
>offered in INVITE did not reply back

```
Nov 28 10:07:51 ip-10-23-247-205 /usr/sbin/kamailio[14702]: [INVITE 
domain.example.com_87665 push_server:3010@udp]--> INFO: 

Re: [sr-dev] [kamailio/kamailio] Segfaults in 5.1.4 (maybe after one rtpengine disappeared) (#1613)

[Daniel-Constantin Mierla]

@marcinkowalczyk it sounds like it can be reproduced, which will make 
troubleshooting easier.

But to understand properly, what do you mean by "RTPEngine disappears between 
offer in invite and ack". Is the SIP ACK, which is after 200ok response? Can 
you narrow a bit more when it happens or provide the steps to reproduce?

The backtrace you posted points to sending out the INVITE.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1613#issuecomment-442404429___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

Re: [sr-dev] [kamailio/kamailio] Segfaults in 5.1.4 (maybe after one rtpengine disappeared) (#1613)

[marcinkowalczyk]

This seems to be more tm issue rather then rtpengengine module. I think it may 
happen if for any reason RTPEngine dissapears between offer in invite and ack

```
Nov 28 10:08:11 ip-10-23-247-205 /usr/sbin/kamailio[14702]: [INVITE 
eldnybgg6e_87665 src_ip:3010@udp]--> ERROR: rtpengine [rtpengine.c:2538]: 
send_rtpp_command(): timeout waiting reply from RTP proxy 

Nov 28 10:08:11 ip-10-23-247-205 /usr/sbin/kamailio[14702]: [INVITE 
eldnybgg6e_87665 src_ip:3010@udp]--> ERROR: rtpengine [rtpengine.c:2412]: 
rtpp_test(): proxy did not respond to ping
Nov 28 10:08:11 ip-10-23-247-205 /usr/sbin/kamailio[14702]: [INVITE 
eldnybgg6e_87665 src_ip:3010@udp]--> ERROR: rtpengine [rtpengine.c:2749]: 
select_rtpp_node(): rtpengine failed to select new for calllen=30 
callid=eldnybgg6e_87665
Nov 28 10:08:11 ip-10-23-247-205 /usr/sbin/kamailio[14702]: [INVITE 
eldnybgg6e_87665 src_ip:3010@udp]--> ERROR: rtpengine [rtpengine.c:2239]: 
rtpp_function_call(): no available proxies
Nov 28 10:08:11 ip-10-23-247-205 kernel: [586868.258562] show_signal_msg: 30 
callbacks suppressed
Nov 28 10:08:11 ip-10-23-247-205 kernel: [586868.258566] kamailio[14702]: 
segfault at 24 ip 7efca782242e sp 7fffbef66410 error 4 in 
tm.so[7efca775c000+138000]

```

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1613#issuecomment-442397711___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

Re: [sr-dev] [kamailio/kamailio] Segfaults in 5.1.4 (maybe after one rtpengine disappeared) (#1613)

[Surendra Tiwari]

Hi, 

i also can see the same issue on 5.1.4.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1613#issuecomment-440156131___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

Re: [sr-dev] [kamailio/kamailio] Segfaults in 5.1.4 (maybe after one rtpengine disappeared) (#1613)

[marcinkowalczyk]

Hi,

 I can see same issue on 5.1.4 too

```
version: kamailio 5.1.4 (x86_64/linux)
flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, DISABLE_NAGLE, 
USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, 
TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, 
USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, 
MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB
poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
id: unknown
compiled with gcc 5.3.1

```

but without crash. It  just stops processing sip messages (happened to me 
couple of times after **kamcmd rtpengine.reload** )
As in my case it's running on top of docker I've implemented simple sipp 
heathcheck that restarts container. 

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1613#issuecomment-424676515___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

Re: [sr-dev] [kamailio/kamailio] Segfaults in 5.1.4 (maybe after one rtpengine disappeared) (#1613)

[Victor Seva]

I had a coredump with version 5.1.4+0~20180725075519.79+trusty ( 
bf6468f7a4a54600e04dc6fd8aab537bde5ee464 ) maybe it's related:

```
(gdb) bt
#0  0x7f1d7c438814 in rtpengine_manage (msg=0x7f1d87bcc858, 
flags=0x7f1d87bfc9b0 "direction=public direction=public replace-origin 
replace-session-connection trust-address") at rtpengine.c:3247
#1  0x7f1d7c4389a8 in rtpengine_manage_wrap (msg=0x7f1d87bcc858, 
d=0x7f1d87bfc9b0, more=0) at rtpengine.c:3270
#2  0x7f1d7c436ab4 in rtpengine_rtpp_set_wrap (msg=0x7f1d87bcc858, 
func=0x7f1d7c438982 , data=0x7f1d87bfc9b0, direction=1) 
at rtpengine.c:3074
#3  0x7f1d7c438cfe in rtpengine_manage1_f (msg=0x7f1d87bcc858, 
str1=0x7f1d87fa7fd0 "\260f\372\207\035\177", str2=0x0) at rtpengine.c:3286
#4  0x00434807 in do_action (h=0x7ffc50479920, a=0x7f1d87fa7668, 
msg=0x7f1d87bcc858) at core/action.c:1073
#5  0x00441630 in run_actions (h=0x7ffc50479920, a=0x7f1d87fa7668, 
msg=0x7f1d87bcc858) at core/action.c:1565
#6  0x004346ee in do_action (h=0x7ffc50479920, a=0x7f1d87fa8b60, 
msg=0x7f1d87bcc858) at core/action.c:1058
#7  0x00441630 in run_actions (h=0x7ffc50479920, a=0x7f1d87fa4920, 
msg=0x7f1d87bcc858) at core/action.c:1565
#8  0x00430f48 in do_action (h=0x7ffc50479920, a=0x7f1d87f83ad8, 
msg=0x7f1d87bcc858) at core/action.c:691
#9  0x00441630 in run_actions (h=0x7ffc50479920, a=0x7f1d87f837b8, 
msg=0x7f1d87bcc858) at core/action.c:1565
#10 0x00441df9 in run_top_route (a=0x7f1d87f837b8, msg=0x7f1d87bcc858, 
c=0x7ffc50479920) at core/action.c:1654
#11 0x7f1d814b8b0e in prepare_new_uac (t=0x7f1d734543f8, 
i_req=0x7f1d87bcc858, branch=0, uri=0x7ffc504798c0, path=0x7ffc504798e0, 
next_hop=0x7ffc504798d0, fsocket=0x0, snd_flags=..., fproto=0, flags=0, 
instance=0x7ffc504798f0, ruid=0x7ffc50479900, location_ua=0x7ffc50479910)
at t_fwd.c:356
#12 0x7f1d814bd457 in add_uac (t=0x7f1d734543f8, request=0x7f1d87bcc858, 
uri=0x7f1d87bccad0, next_hop=0x7f1d87bccae0, path=0x7f1d87bcce90, proxy=0x0, 
fsocket=0x0, snd_flags=..., proto=0, flags=0, instance=0x7f1d87bccea0, 
ruid=0x7f1d87bcceb8, location_ua=0x7f1d87bccec8)
at t_fwd.c:800
#13 0x7f1d814c5758 in t_forward_nonack (t=0x7f1d734543f8, 
p_msg=0x7f1d87bcc858, proxy=0x0, proto=0) at t_fwd.c:1680
#14 0x7f1d814af0ac in t_relay_to (p_msg=0x7f1d87bcc858, proxy=0x0, proto=0, 
replicate=0) at t_funcs.c:331
#15 0x7f1d8153c3e8 in _w_t_relay_to (p_msg=0x7f1d87bcc858, proxy=0x0, 
force_proto=0) at tm.c:1555
#16 0x7f1d8153d66d in w_t_relay (p_msg=0x7f1d87bcc858, _foo=0x0, _bar=0x0) 
at tm.c:1755
#17 0x0043477f in do_action (h=0x7ffc5047a700, a=0x7f1d87f9b358, 
msg=0x7f1d87bcc858) at core/action.c:1067
#18 0x00441630 in run_actions (h=0x7ffc5047a700, a=0x7f1d87f9b358, 
msg=0x7f1d87bcc858) at core/action.c:1565
#19 0x00441d10 in run_actions_safe (h=0x7ffc5047d670, a=0x7f1d87f9b358, 
msg=0x7f1d87bcc858) at core/action.c:1633
#20 0x0057c66b in rval_get_int (h=0x7ffc5047d670, msg=0x7f1d87bcc858, 
i=0x7ffc5047aaf4, rv=0x7f1d87f9b618, cache=0x0) at core/rvalue.c:912
#21 0x00580cf6 in rval_expr_eval_int (h=0x7ffc5047d670, 
msg=0x7f1d87bcc858, res=0x7ffc5047aaf4, rve=0x7f1d87f9b610) at 
core/rvalue.c:1910
#22 0x00581158 in rval_expr_eval_int (h=0x7ffc5047d670, 
msg=0x7f1d87bcc858, res=0x7ffc5047ac64, rve=0x7f1d87f9bd40) at 
core/rvalue.c:1918
#23 0x0043422a in do_action (h=0x7ffc5047d670, a=0x7f1d87f9e8c0, 
msg=0x7f1d87bcc858) at core/action.c:1043
#24 0x00441630 in run_actions (h=0x7ffc5047d670, a=0x7f1d87f983f8, 
msg=0x7f1d87bcc858) at core/action.c:1565
#25 0x00430f48 in do_action (h=0x7ffc5047d670, a=0x7f1d87d5bfd0, 
msg=0x7f1d87bcc858) at core/action.c:691
#26 0x00441630 in run_actions (h=0x7ffc5047d670, a=0x7f1d87d5bb60, 
msg=0x7f1d87bcc858) at core/action.c:1565
#27 0x004346ee in do_action (h=0x7ffc5047d670, a=0x7f1d87d5eb00, 
msg=0x7f1d87bcc858) at core/action.c:1058
#28 0x00441630 in run_actions (h=0x7ffc5047d670, a=0x7f1d87d49d78, 
msg=0x7f1d87bcc858) at core/action.c:1565
#29 0x00430f48 in do_action (h=0x7ffc5047d670, a=0x7f1d87e07e80, 
msg=0x7f1d87bcc858) at core/action.c:691
#30 0x00441630 in run_actions (h=0x7ffc5047d670, a=0x7f1d87e07e80, 
msg=0x7f1d87bcc858) at core/action.c:1565
#31 0x0043e47a in do_action (h=0x7ffc5047d670, a=0x7f1d87e08748, 
msg=0x7f1d87bcc858) at core/action.c:1314
#32 0x00441630 in run_actions (h=0x7ffc5047d670, a=0x7f1d87e062e0, 
msg=0x7f1d87bcc858) at core/action.c:1565
#33 0x004346ee in do_action (h=0x7ffc5047d670, a=0x7f1d87e08898, 
msg=0x7f1d87bcc858) at core/action.c:1058
#34 0x00441630 in run_actions (h=0x7ffc5047d670, a=0x7f1d87dff690, 
msg=0x7f1d87bcc858) at core/action.c:1565
#35 0x00441df9 in run_top_route (a=0x7f1d87dff690, msg=0x7f1d87bcc858, 
c=0x0) at core/action.c:1654
#36 0x0053e22f in receive_msg (buf=0xb010c0  "INVITE 

Re: [sr-dev] [kamailio/kamailio] Segfaults in 5.1.4 (maybe after one rtpengine disappeared) (#1613)

[Sebastian Damm]

No, we don't have an tm event routes at all. The only event routes are:

```
event_route[xhttp:request] {
event_route[dispatcher:dst-down] {
event_route[dispatcher:dst-up] {
```


-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1613#issuecomment-414230848___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

Re: [sr-dev] [kamailio/kamailio] Segfaults in 5.1.4 (maybe after one rtpengine disappeared) (#1613)

[Daniel-Constantin Mierla]

For *The other core* is looks a bit strange. The first backtrace shows and 
INVITE, but in the last comment is an OPTIONS.

It looks like the OPTIONS gets to a t_relay() (or something similar), although 
the request is generated locally, so this should not happen. Do you have any tm 
event route that does t_relay()?

I pushed a commit with a safety check, but this situation was not supposed to 
happen and may still have side effects even after this safety check.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1613#issuecomment-413157322___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

Re: [sr-dev] [kamailio/kamailio] Segfaults in 5.1.4 (maybe after one rtpengine disappeared) (#1613)

[Sebastian Damm]

First core file:

```
GNU gdb (GDB) 7.4.1-debian
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
For bug reporting instructions, please see:
...
Reading symbols from /usr/sbin/kamailio...Reading symbols from 
/usr/lib/debug/.build-id/bf/0f481eacd5661e7adec0c358237730d825ea2a.debug...done.
done.
[New LWP 31359]

warning: Can't read pathname for load map: Input/output error.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/sbin/kamailio -f 
/etc/kamailio/kamailio_sip_proxy.cfg -P /var/run/kamailio'.
Program terminated with signal 11, Segmentation fault.
#0  0x00663091 in destroy_avp_list_unsafe (list=0x7efe49050bd0) at 
core/usr_avp.c:625
625 core/usr_avp.c: No such file or directory.
(gdb) frame 1
#1  0x7efe6b4daadc in free_cell_helper (dead_cell=0x7efe49050a48, silent=0, 
fname=0x7efe6b5db100 "timer.c", fline=654) at h_table.c:242
242 h_table.c: No such file or directory.
(gdb) p *dead_cell 
$1 = {next_c = 0x0, prev_c = 0x0, hash_index = 30681, label = 705812179, flags 
= 32, nr_of_outgoings = 1, ref_count = {val = 0}, from = {
s = 0x7efe48f3796d "From: 
;tag=7fcea877-80e3-4069-b\r\nCall-ID: 
DF246541ADA4EDA2@84.149.15.231\r\nEvent: message-summary\r\nSubscription-State: 
Active\r\nRoute: \r\nConte"..., len = 57}, 
callid = {
s = 0x7efe48f379a6 "Call-ID: DF246541ADA4EDA2@84.149.15.231\r\nEvent: 
message-summary\r\nSubscription-State: Active\r\nRoute: 
\r\nContent-Type: 
application/simple-message-summary\r\nVia: SIP/2.0"..., len = 41}, cseq_n = {
s = 0x7efe48f3792d "CSeq: 7950 NOTIFY\r\nTo: 
;tag=3951995175\r\nFrom: 
;tag=7fcea877-80e3-4069-b\r\nCall-ID: 
DF246541ADA4EDA2@84.149.15.231\r\nEvent: message-summary\r\nSubscription-S"..., 
len = 10}, to = {
s = 0x7efe48f37940 "To: ;tag=3951995175\r\nFrom: 
;tag=7fcea877-80e3-4069-b\r\nCall-ID: 
DF246541ADA4EDA2@84.149.15.231\r\nEvent: message-summary\r\nSubscription-State: 
Active\r\nRoute"..., len = 45}, method = {
s = 0x7efe48f378e0 "NOTIFY 
sip:1442863@84.149.15.231;uniq=29A7ECC65F8884BD635517C89AFD8 SIP/2.0\r\nCSeq: 
7950 NOTIFY\r\nTo: ;tag=3951995175\r\nFrom: 
;tag=7fcea877-80e3-4069-b\r\nCa"..., len = 6}, tmcb_hl 
= {first = 0x0, reg_types = 0}, wait_timer = {next = 0x0, prev = 0x0, expire = 
1289862248, initial_timeout = 320, data = 0x7efe49050a48, 
f = 0x7efe6b5875fa , flags = 513, slow_idx = 0}, uas = 
{request = 0x7efe48f371c8, end_request = 0x7efe48f38070 "\210", response = 
{activ_type = 481, flags = 128, t_active = 0 '\000', 
  branch = 0, buffer_len = 323, 
  buffer = 0x7efe49077660 "SIP/2.0 481 Call Leg/Transaction Does Not 
Exist\r\nVia: SIP/2.0/UDP 217.10.76.144:5060\r\nFrom: 
;tag=7fcea877-80e3-4069-b\r\nTo: 
;tag=3951995175\r\nCall-ID: DF2"..., my_T = 
0x7efe49050a48, timer = {next = 0x0, prev = 0x0, expire = 0, initial_timeout = 
0, data = 0x0, f = 0x7efe6b5870b2 , flags = 0, slow_idx = 0}, 
dst = {
send_sock = 0x7efe6dff77e0, to = {s = {sa_family = 2, sa_data = 
"\023\304\331\nL\220\000\000\000\000\000\000\000"}, sin = {sin_family = 2, 
sin_port = 50195, sin_addr = {s_addr = 2420902617}, 
sin_zero = "\000\000\000\000\000\000\000"}, sin6 = {sin6_family = 
2, sin6_port = 50195, sin6_flowinfo = 2420902617, sin6_addr = {__in6_u = 
{__u6_addr8 = '\000' , __u6_addr16 = {0, 
  0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, 
sin6_scope_id = 0}}, id = 0, proto = 1 '\001', send_flags = {f = 0, blst_imask 
= 0}}, retr_expire = 0, fr_expire = 0}, local_totag = {
  s = 0x0, len = 0}, cancel_reas = 0x0, status = 481}, uac = 
0x7efe49050c50, async_backup = {backup_route = 0, backup_branch = 0, blind_uac 
= 0, ruri_new = 0}, fwded_totags = 0x0, 
  uri_avps_from = 0x7efe49030900, uri_avps_to = 0x0, user_avps_from = 0x0, 
user_avps_to = 0x0, domain_avps_from = 0x0, domain_avps_to = 0x0, xavps_list = 
0x0, reply_mutex = {val = 0}, reply_locker_pid = {
val = 0}, reply_rec_lock_level = 0, fr_timeout = 320, fr_inv_timeout = 
2000, rt_t1_timeout_ms = 500, rt_t2_timeout_ms = 4000, end_of_life = 
1289862439, relayed_reply_branch = 0, on_failure = 0, 
  on_branch_failure = 0, on_reply = 0, on_branch = 0, on_branch_delayed = 0, 
md5 = 0x7efe49050c30 "f17e9566ecae744152b7aed3c5ba0273"}
```

The other core:
```
GNU gdb (GDB) 7.4.1-debian
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.