I managed to get a core dump. Here is the backtrace:
#0 0xffffe410 in __kernel_vsyscall ()
#1 0xb7e5dfc1 in raise () from /lib/libpthread.so.0
#2 0x0805ace4 in pss_amf_csi_set_all (invocation=4148166670,
compName=0xb53059d0, haState=SA_AMF_HA_ACTIVE, csiDescriptor=
{csiFlags = 4, csiName = {length = 0, value = '\0' <repeats 255 times>}, csiStateDescriptor =
{activeDescriptor = {transitionDescriptor = SA_AMF_CSI_NOT_QUIESCED, activeCompName = {length = 52, value =
"safComp=CompT_PSS,safSu=SuT_NCS_CNTLR,safNode=SC_2_1", '\0' <repeats 203 times>}}, standbyDescriptor
= {activeCompName = {length = 3, value =
"\000\0004\000safComp=CompT_PSS,safSu=SuT_NCS_CNTLR,safNode=SC_2_1", '\0' <repeats 199 times>},
standbyRank = 0}}, csiAttr = {attr = 0x0, number = 0}}) at ./psr_amf.c:1081
#3 0x0805a36e in pss_amf_csi_set (invocation=4148166670, compName=0xb53059d0,
haState=SA_AMF_HA_ACTIVE, csiDescriptor=
{csiFlags = 4, csiName = {length = 0, value = '\0' <repeats 255 times>}, csiStateDescriptor =
{activeDescriptor = {transitionDescriptor = SA_AMF_CSI_NOT_QUIESCED, activeCompName = {length = 52, value =
"safComp=CompT_PSS,safSu=SuT_NCS_CNTLR,safNode=SC_2_1", '\0' <repeats 203 times>}}, standbyDescriptor
= {activeCompName = {length = 3, value =
"\000\0004\000safComp=CompT_PSS,safSu=SuT_NCS_CNTLR,safNode=SC_2_1", '\0' <repeats 199 times>},
standbyRank = 0}}, csiAttr = {attr = 0x0, number = 0}}) at ./psr_amf.c:757
#4 0xb7e73609 in ava_hdl_cbk_rec_prc (info=0xb53059bc, reg_cbk=0xb4afe194) at
./ava_hdl.c:744
#5 0xb7e72ab1 in ava_hdl_cbk_dispatch_one (cb=0xb4afe228, hdl_rec=0xb4afe224)
at ./ava_hdl.c:430
#6 0xb7e7282e in ava_hdl_cbk_dispatch (cb=0xb4afe228, hdl_rec=0xb4afe224,
flags=SA_DISPATCH_ONE) at ./ava_hdl.c:366
#7 0xb7e6e1b4 in saAmfDispatch (hdl=4269801474, flags=SA_DISPATCH_ONE) at
./ava_api.c:301
#8 0x0805922e in pss_mbx_amf_process (pss_mbx=0x8094d28) at ./psr_amf.c:281
#9 0xb7e562ab in start_thread () from /lib/libpthread.so.0
#10 0xb7deda4e in clone () from /lib/libc.so.6
What do you want more? I can easily reproduce the problem. I just reboot the
active controller and this crash happens on the standby.
But there must an error in the pss_reformat function. The loops are not
terminated correctly or something. Here is my PSSV content:
SC_2_2# find /var/opt/opensaf/pssv_store/2/current/1/
/var/opt/opensaf/pssv_store/2/current/1/
/var/opt/opensaf/pssv_store/2/current/1/AVD
/var/opt/opensaf/pssv_store/2/current/1/AVD/13
/var/opt/opensaf/pssv_store/2/current/1/AVD/15
/var/opt/opensaf/pssv_store/2/current/1/AVD/17
/var/opt/opensaf/pssv_store/2/current/1/AVD/24
/var/opt/opensaf/pssv_store/2/current/1/AVD/26
/var/opt/opensaf/pssv_store/2/current/1/AVD/25
/var/opt/opensaf/pssv_store/2/current/1/AVD/12
/var/opt/opensaf/pssv_store/2/current/1/AVD/11
/var/opt/opensaf/pssv_store/2/current/1/AVD/10
/var/opt/opensaf/pssv_store/2/current/1/AVD/22
/var/opt/opensaf/pssv_store/2/current/1/AVD/23
/var/opt/opensaf/pssv_store/2/current/1/AVD/28
/var/opt/opensaf/pssv_store/2/current/1/AVD/8
/var/opt/opensaf/pssv_store/2/current/1/AVD/6
/var/opt/opensaf/pssv_store/2/current/1/AVD/18
/var/opt/opensaf/pssv_store/2/current/1/AVM
/var/opt/opensaf/pssv_store/2/current/1/AVM/39
Why does it do a scandir on '/var/opt/opensaf/pssv_store/2/current/1/1'?
Attached is the PSSV console log.
Thanks,
Hans
Mathivanan Np-G19859 wrote:
The standby PSS reports an error to AMF when it is about the
become active during fail-over. I tracked it down to a failed
scandir in psr_rfmt.c#1347. The path is non-existent in the
file system.
PSR reporting an error just seems to be fine since the data
"/var/opt/opensaf/pssv_store/2/current/1/1" is anyway non-existent.
But the real problem would be that, in the normal flow, while reading
from the persistent store, this value
===>"/var/opt/opensaf/pssv_store/2/current/1/1"
itself is unexpected/incorrect and should not occur.
This could happen if either the persistent-store was tampered
with/corrupt(?) OR
If a decode failed!
We can try generating a core of the 'ncs_psr' process the next time we
hit this issue.
That would help closing-in on to the problem.
- Mathi.
Should I open a ticket on this?
Any ideas?
Thanks,
Hans
syslog:
Feb 6 15:39:15 SC_2_1 ncs_scap: NCS_AvSv: Card going for reboot
-safComp=CompT_PSS,safSu=SuT_NCS_CNTLR,safNode=SC_2_1
faulted due to 1
-rcvr=9
DTS log:
NOTICE : 0x0002010f 2625322986 49 1 PSS 06Feb2008_15.39.15.107
HEADLINE: PSS PWE found: data: 1 NOTICE : 0x0002010f
2625322986 49 1
PSS 06Feb2008_15.39.15.107 HEADLINE: PSS PCN found: String: AVM
NOTICE : 0x0002010f 2625322986 49 1 PSS 06Feb2008_15.39.15.121
HEADLINE: PSS PCN found: String: AVD NOTICE : 0x0002010f
2625322986
49 1 PSS 06Feb2008_15.39.15.133 HEADLINE: PSS PWE found: data: 1
NOTICE : 0x0002010f 2625322986 49 1 PSS 06Feb2008_15.39.15.133
HEADLINE: PSS PCN found: String: 1 ERROR : 0x0002010f
2625322986 49
1 PSS 06Feb2008_15.39.15.133 HEADLINE : Scanning the directory for
Table IDs failed
SC_2_1# cat NID_PSSV.log
<snip>
pwe_id: 1
pcn_cnt: 1
str_len: 2
PCN_PATH: /var/opt/opensaf/pssv_store/2/current/1/1
table_cnt: -1
SC_2_1# find /var/opt/opensaf/pssv_store/ /var/opt/opensaf/pssv_store/
/var/opt/opensaf/pssv_store/2
/var/opt/opensaf/pssv_store/2/current
/var/opt/opensaf/pssv_store/2/current/1
/var/opt/opensaf/pssv_store/2/current/1/AVM
/var/opt/opensaf/pssv_store/2/current/1/AVM/39
/var/opt/opensaf/pssv_store/2/current/1/AVD
/var/opt/opensaf/pssv_store/2/current/1/AVD/13
/var/opt/opensaf/pssv_store/2/current/1/AVD/15
/var/opt/opensaf/pssv_store/2/current/1/AVD/17
/var/opt/opensaf/pssv_store/2/current/1/AVD/24
/var/opt/opensaf/pssv_store/2/current/1/AVD/26
/var/opt/opensaf/pssv_store/2/current/1/AVD/25
/var/opt/opensaf/pssv_store/2/current/1/AVD/12
/var/opt/opensaf/pssv_store/2/current/1/AVD/11
/var/opt/opensaf/pssv_store/2/current/1/AVD/10
/var/opt/opensaf/pssv_store/2/current/1/AVD/22
/var/opt/opensaf/pssv_store/2/current/1/AVD/23
/var/opt/opensaf/pssv_store/2/current/1/AVD/28
/var/opt/opensaf/pssv_store/2/current/1/AVD/8
/var/opt/opensaf/pssv_store/2/current/1/AVD/6
/var/opt/opensaf/pssv_store/2/current/1/AVD/18
/var/opt/opensaf/pssv_store/2/current/_ISU
/var/opt/opensaf/pssv_store/2/current/_ISU/1
/var/opt/opensaf/pssv_store/2/current/_ISU/1/AVD
/var/opt/opensaf/pssv_store/2/current/_ISU/1/AVD/13
/var/opt/opensaf/pssv_store/2/current/_ISU/1/AVD/15
/var/opt/opensaf/pssv_store/2/current/_ISU/1/AVD/17
/var/opt/opensaf/pssv_store/2/current/_ISU/1/AVD/24
/var/opt/opensaf/pssv_store/2/current/_ISU/1/AVD/26
/var/opt/opensaf/pssv_store/2/current/_ISU/1/AVD/25
/var/opt/opensaf/pssv_store/2/current/_ISU/1/AVM
/var/opt/opensaf/pssv_store/2/current/_ISU/1/AVM/39
_______________________________________________
Users mailing list
Users@list.opensaf.org
http://list.opensaf.org/maillist/listinfo/users
______________________________________________________________________
This email has been scanned by the MessageLabs Email Security System.
For more information please visit http://www.messagelabs.com/email
______________________________________________________________________
Entered pss_reformat, reformat_type: 2
cur_ps_format: 2
Profile found: current
buff_len: 26
pwe_cnt: 2
pwe_id: 1
pcn_cnt: 2
str_len: 4
PCN_PATH: /var/opt/opensaf/pssv_store/2/current/1/AVD
table_cnt: 1718
18
Entered case PSS_REFORMAT_TYPE_MIB_IS_READY
dest_file_path: /var/opt/opensaf/pssv_store/2/current/1/AVD/18_tbl_details
Read Header details:
header_len: 12
PS_FORMAT: 2
TableVersion: 1
max_row_length: 523
max_key_length: 261
bitmap_length: 1
file_size : 3150, hdr.max_row_length: 523
No change in the table_version and the table details are consistent: 18
Done with table 18
6
6
Entered case PSS_REFORMAT_TYPE_MIB_IS_READY
dest_file_path: /var/opt/opensaf/pssv_store/2/current/1/AVD/6_tbl_details
Read Header details:
header_len: 12
PS_FORMAT: 2
TableVersion: 1
max_row_length: 25
max_key_length: 0
bitmap_length: 1
file_size : 37, hdr.max_row_length: 25
No change in the table_version and the table details are consistent: 6
Done with table 6
8
8
Entered case PSS_REFORMAT_TYPE_MIB_IS_READY
dest_file_path: /var/opt/opensaf/pssv_store/2/current/1/AVD/8_tbl_details
Read Header details:
header_len: 12
PS_FORMAT: 2
TableVersion: 1
max_row_length: 535
max_key_length: 0
bitmap_length: 1
file_size : 547, hdr.max_row_length: 535
No change in the table_version and the table details are consistent: 8
Done with table 8
28
28
Entered case PSS_REFORMAT_TYPE_MIB_IS_READY
dest_file_path: /var/opt/opensaf/pssv_store/2/current/1/AVD/28_tbl_details
Read Header details:
header_len: 12
PS_FORMAT: 2
TableVersion: 1
max_row_length: 312
max_key_length: 291
bitmap_length: 1
file_size : 40572, hdr.max_row_length: 312
No change in the table_version and the table details are consistent: 28
Done with table 28
23
23
Entered case PSS_REFORMAT_TYPE_MIB_IS_READY
dest_file_path: /var/opt/opensaf/pssv_store/2/current/1/AVD/23_tbl_details
Read Header details:
header_len: 12
PS_FORMAT: 2
TableVersion: 1
max_row_length: 519
max_key_length: 514
bitmap_length: 1
file_size : 76824, hdr.max_row_length: 519
No change in the table_version and the table details are consistent: 23
Done with table 23
22
22
Entered case PSS_REFORMAT_TYPE_MIB_IS_READY
dest_file_path: /var/opt/opensaf/pssv_store/2/current/1/AVD/22_tbl_details
Read Header details:
header_len: 12
PS_FORMAT: 2
TableVersion: 1
max_row_length: 1980
max_key_length: 257
bitmap_length: 5
file_size : 293052, hdr.max_row_length: 1980
No change in the table_version and the table details are consistent: 22
Done with table 22
10
10
Entered case PSS_REFORMAT_TYPE_MIB_IS_READY
dest_file_path: /var/opt/opensaf/pssv_store/2/current/1/AVD/10_tbl_details
Read Header details:
header_len: 12
PS_FORMAT: 2
TableVersion: 1
max_row_length: 267
max_key_length: 258
bitmap_length: 1
file_size : 3750, hdr.max_row_length: 267
No change in the table_version and the table details are consistent: 10
Done with table 10
11
11
Entered case PSS_REFORMAT_TYPE_MIB_IS_READY
dest_file_path: /var/opt/opensaf/pssv_store/2/current/1/AVD/11_tbl_details
Read Header details:
header_len: 12
PS_FORMAT: 2
TableVersion: 1
max_row_length: 278
max_key_length: 257
bitmap_length: 1
file_size : 3904, hdr.max_row_length: 278
No change in the table_version and the table details are consistent: 11
Done with table 11
12
12
Entered case PSS_REFORMAT_TYPE_MIB_IS_READY
dest_file_path: /var/opt/opensaf/pssv_store/2/current/1/AVD/12_tbl_details
Read Header details:
header_len: 12
PS_FORMAT: 2
TableVersion: 1
max_row_length: 267
max_key_length: 258
bitmap_length: 1
file_size : 1881, hdr.max_row_length: 267
No change in the table_version and the table details are consistent: 12
Done with table 12
25
25
Entered case PSS_REFORMAT_TYPE_MIB_IS_READY
dest_file_path: /var/opt/opensaf/pssv_store/2/current/1/AVD/25_tbl_details
Read Header details:
header_len: 12
PS_FORMAT: 2
TableVersion: 1
max_row_length: 776
max_key_length: 514
bitmap_length: 1
file_size : 27172, hdr.max_row_length: 776
No change in the table_version and the table details are consistent: 25
Done with table 25
26
26
Entered case PSS_REFORMAT_TYPE_MIB_IS_READY
dest_file_path: /var/opt/opensaf/pssv_store/2/current/1/AVD/26_tbl_details
Read Header details:
header_len: 12
PS_FORMAT: 2
TableVersion: 1
max_row_length: 519
max_key_length: 514
bitmap_length: 1
file_size : 4683, hdr.max_row_length: 519
No change in the table_version and the table details are consistent: 26
Done with table 26
24
24
Entered case PSS_REFORMAT_TYPE_MIB_IS_READY
dest_file_path: /var/opt/opensaf/pssv_store/2/current/1/AVD/24_tbl_details
Read Header details:
header_len: 12
PS_FORMAT: 2
TableVersion: 1
max_row_length: 523
max_key_length: 257
bitmap_length: 1
file_size : 68525, hdr.max_row_length: 523
No change in the table_version and the table details are consistent: 24
Done with table 24
17
17
Entered case PSS_REFORMAT_TYPE_MIB_IS_READY
dest_file_path: /var/opt/opensaf/pssv_store/2/current/1/AVD/17_tbl_details
Read Header details:
header_len: 12
PS_FORMAT: 2
TableVersion: 1
max_row_length: 548
max_key_length: 257
bitmap_length: 2
file_size : 35632, hdr.max_row_length: 548
No change in the table_version and the table details are consistent: 17
Done with table 17
15
15
Entered case PSS_REFORMAT_TYPE_MIB_IS_READY
dest_file_path: /var/opt/opensaf/pssv_store/2/current/1/AVD/15_tbl_details
Read Header details:
header_len: 12
PS_FORMAT: 2
TableVersion: 1
max_row_length: 564
max_key_length: 257
bitmap_length: 2
file_size : 41748, hdr.max_row_length: 564
No change in the table_version and the table details are consistent: 15
Done with table 15
13
13
Entered case PSS_REFORMAT_TYPE_MIB_IS_READY
dest_file_path: /var/opt/opensaf/pssv_store/2/current/1/AVD/13_tbl_details
Read Header details:
header_len: 12
PS_FORMAT: 2
TableVersion: 1
max_row_length: 336
max_key_length: 257
bitmap_length: 3
file_size : 4716, hdr.max_row_length: 336
No change in the table_version and the table details are consistent: 13
Done with table 13
..
.
str_len: 4
PCN_PATH: /var/opt/opensaf/pssv_store/2/current/1/AVM
table_cnt: 339
39
Entered case PSS_REFORMAT_TYPE_MIB_IS_READY
dest_file_path: /var/opt/opensaf/pssv_store/2/current/1/AVM/39_tbl_details
Read Header details:
header_len: 12
PS_FORMAT: 2
TableVersion: 1
max_row_length: 3683
max_key_length: 261
bitmap_length: 4
file_size : 55257, hdr.max_row_length: 3683
No change in the table_version and the table details are consistent: 39
Done with table 39
..
.
pwe_id: 1
pcn_cnt: 1
str_len: 2
PCN_PATH: /var/opt/opensaf/pssv_store/2/current/1/1
_______________________________________________
Users mailing list
Users@list.opensaf.org
http://list.opensaf.org/maillist/listinfo/users
