Re: [devel] [PATCH 1/1] amf: fix Comp stuck in RESTARTING presence state [#3011]

2019-02-28 Thread Minh Hon Chau 

Hi Thuan,

ack (review + test).

Thanks

Minh

:

During SU (many COMPs) restart recovery, if any COMP finish instantiated
then crash while other COMPs are still instantiating, AMF recovery it by
restarting but AMF only cleanup without re-instantiation because AMF see
the COMP is not eligible for instantiation. The error COMP is stuck in
RESTARTING without further action from AMF.

AMF should allow COMP re-instantiation if SU state is INSTANTIATING and
error COMP state is RESTARTING.
---
  src/amf/amfnd/clc.cc | 3 ++-
  1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/src/amf/amfnd/clc.cc b/src/amf/amfnd/clc.cc
index 463c5de..7a62a56 100644
--- a/src/amf/amfnd/clc.cc
+++ b/src/amf/amfnd/clc.cc
@@ -1805,7 +1805,8 @@ static bool 
is_failed_comp_eligible_for_instantiation(AVND_COMP *comp) {
/*During surestart recovery, after cleanup of all components, amfnd 
starts
  instantiation of components. A component may fault at this stage. 
Such a
  component is eligible for instantiation.*/
-  if ((comp->pres == SA_AMF_PRESENCE_INSTANTIATING) &&
+  if (((comp->pres == SA_AMF_PRESENCE_RESTARTING) ||
+   (comp->pres == SA_AMF_PRESENCE_INSTANTIATING)) &&
(comp->su->pres == SA_AMF_PRESENCE_INSTANTIATING))
  return true;
  
--

2.7.4


___
Opensaf-devel mailing list
Opensaf-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/opensaf-devel

Re: [devel] [PATCH 1/1] nid: fix opensafd crashed during start-up [#3013]

2019-02-28 Thread Hans Nordebäck 
Hi Vu,
fine,  perhaps also changing the  static bool svc_monitor_thread_running = 
false  to std::atomic?/BR Hans

From: Vu Minh Nguyen 
Sent: den 28 februari 2019 09:30
To: Hans Nordebäck ; Gary Lee 

Cc: opensaf-devel@lists.sourceforge.net
Subject: RE: [PATCH 1/1] nid: fix opensafd crashed during start-up [#3013]

Thanks Hans. I will send the V2 for these updates.

Regards, Vu

From: Hans Nordebäck 
mailto:hans.nordeb...@ericsson.com>>
Sent: Thursday, February 28, 2019 2:16 PM
To: Vu Minh Nguyen 
mailto:vu.m.ngu...@dektech.com.au>>; Gary Lee 
mailto:gary@dektech.com.au>>
Cc: 
opensaf-devel@lists.sourceforge.net
Subject: Re: [PATCH 1/1] nid: fix opensafd crashed during start-up [#3013]


Hi Vu,

you can keep your patch for the ready state, but also change SOCK_STREAM to 
SOCK_DGRAM and change

the read(svc_mon_thr_fd, nid_name, NID_MAXSNAME) in svc_monitor_thread to

recv(svc_mon_thr_fd, nid_name, NID_MAXSNAME, MSG_DONTWAIT) and also handle 
EAGAIN and

EWOULDBLOCK. Then only one nid_name per read/recv will be given instead of 
several nid_names

as in the SOCK_STREAM case.

/BR Hans


On 2/28/19 05:30, Vu Minh Nguyen wrote:

Hi Hans,



Thanks for your comment.



But I has a concern that the service-monitoring function may not fully work

if a service is crashed before the svc_monitor_thread goes to ready state?



Is it mandatory for monitoring thread to enter ready state before spawning

SAF services?



Regards, Vu



-Original Message-

From: Hans Nordebäck 


Sent: Wednesday, February 27, 2019 8:23 PM

To: Vu Minh Nguyen 
; Gary Lee



Cc: 
opensaf-devel@lists.sourceforge.net;
 Vu Minh Nguyen



Subject: RE: [PATCH 1/1] nid: fix opensafd crashed during start-up [#3013]



Hi Vu,

I discussed a bit with Anders, likely it should work if the socketpair is

changed

to socketpair(AF_UNIX, SOCK_DGRAM .. from SOCK_STREAM. /BR Hans



-Original Message-

From: Hans Nordebäck

Sent: den 27 februari 2019 11:55

To: 'Vu Minh Nguyen' 
; Gary Lee



Cc: 
opensaf-devel@lists.sourceforge.net;
 Vu Minh Nguyen



Subject: RE: [PATCH 1/1] nid: fix opensafd crashed during start-up [#3013]



Hi Vu,

ack, code review only/Thanks HansN



-Original Message-

From: Vu Minh Nguyen 


Sent: den 27 februari 2019 11:48

To: Hans Nordebäck 
; Gary Lee



Cc: 
opensaf-devel@lists.sourceforge.net;
 Vu Minh Nguyen



Subject: [PATCH 1/1] nid: fix opensafd crashed during start-up [#3013]



There is a dependency b/w svc_monitor_thread and spawn_services.

The coredump happens when spawn_services is executed while the thread

has not yet started. In this case, data is sent to the pipe but no one

consumed

it. Later on, reading data from the pipe, will get unexpected data and

crash

the program.



This patch ensures the order: svc_monitor_thread must be in ready state

before spawn_services() is executed.

---

 src/nid/nodeinit.cc | 11 +++

 1 file changed, 11 insertions(+)



diff --git a/src/nid/nodeinit.cc b/src/nid/nodeinit.cc index

5f15916b4..b4945b05c 100644

--- a/src/nid/nodeinit.cc

+++ b/src/nid/nodeinit.cc

@@ -134,6 +134,7 @@ static int start_monitor_svc(const char *svc);

 /* Data declarations for service monitoring */  static int svc_mon_fd =

-1;

static int next_svc_fds_slot = 0;

+static bool svc_monitor_thread_running = false;



 struct SAFServices {

   const std::string fifo_dir = PKGLOCALSTATEDIR; @@ -1517,6 +1518,7 @@

void *svc_monitor_thread(void *fd) {

   next_svc_fds_slot++;



   while (true) {

+svc_monitor_thread_running = true;

 unsigned rc = osaf_poll(fds, next_svc_fds_slot, -1);

 if (rc > 0) {

   // check if any monitored service has exit @@ -1655,6 +1657,15 @@

int

main(int argc, char *argv[]) {

 exit(EXIT_FAILURE);

   }



+  // Waiting until svc_monitor_thread is up and in ready state.

+  // If spawn_services runs before the thread is in ready state,  //

+ receive side of the pipe s_pair will get unexpected data and  // may

+ crash the process.

+  while (svc_monitor_thread_running == false) {

+usleep(100);

+  }

+

+  LOG_NO("svc_monitor_thread is up and in ready state");

   if (parse_nodeinit_conf(sbuf) != NCSCC_RC_SUCCESS) {

 LOG_ER("Failed to parse file %s. Exiting", sbuf);

 exit(EXIT_FAILURE);

--

2.19.2





___
Opensaf-devel mailing list
Opensaf-devel@lists.sourceforge.net

Re: [devel] [PATCH 1/1] nid: fix opensafd crashed during start-up [#3013]

2019-02-28 Thread Vu Minh Nguyen 
Thanks Hans. I will send the V2 for these updates.

 

Regards, Vu

 

From: Hans Nordebäck  
Sent: Thursday, February 28, 2019 2:16 PM
To: Vu Minh Nguyen ; Gary Lee 

Cc: opensaf-devel@lists.sourceforge.net
Subject: Re: [PATCH 1/1] nid: fix opensafd crashed during start-up [#3013]

 

Hi Vu,

you can keep your patch for the ready state, but also change SOCK_STREAM to 
SOCK_DGRAM and change 

the read(svc_mon_thr_fd, nid_name, NID_MAXSNAME) in svc_monitor_thread to

recv(svc_mon_thr_fd, nid_name, NID_MAXSNAME, MSG_DONTWAIT) and also handle 
EAGAIN and

EWOULDBLOCK. Then only one nid_name per read/recv will be given instead of 
several nid_names

as in the SOCK_STREAM case.

/BR Hans

 

On 2/28/19 05:30, Vu Minh Nguyen wrote:

Hi Hans,
 
Thanks for your comment. 
 
But I has a concern that the service-monitoring function may not fully work 
if a service is crashed before the svc_monitor_thread goes to ready state?
 
Is it mandatory for monitoring thread to enter ready state before spawning
SAF services?
 
Regards, Vu
 

-Original Message-
From: Hans Nordebäck   

Sent: Wednesday, February 27, 2019 8:23 PM
To: Vu Minh Nguyen   
; Gary Lee
  
Cc: opensaf-devel@lists.sourceforge.net 
 ; Vu Minh Nguyen
  
Subject: RE: [PATCH 1/1] nid: fix opensafd crashed during start-up [#3013]
 
Hi Vu,
I discussed a bit with Anders, likely it should work if the socketpair is

changed

to socketpair(AF_UNIX, SOCK_DGRAM .. from SOCK_STREAM. /BR Hans
 
-Original Message-
From: Hans Nordebäck
Sent: den 27 februari 2019 11:55
To: 'Vu Minh Nguyen'   
; Gary Lee
  
Cc: opensaf-devel@lists.sourceforge.net 
 ; Vu Minh Nguyen
  
Subject: RE: [PATCH 1/1] nid: fix opensafd crashed during start-up [#3013]
 
Hi Vu,
ack, code review only/Thanks HansN
 
-Original Message-
From: Vu Minh Nguyen   

Sent: den 27 februari 2019 11:48
To: Hans Nordebäck   
; Gary Lee
  
Cc: opensaf-devel@lists.sourceforge.net 
 ; Vu Minh Nguyen
  
Subject: [PATCH 1/1] nid: fix opensafd crashed during start-up [#3013]
 
There is a dependency b/w svc_monitor_thread and spawn_services.
The coredump happens when spawn_services is executed while the thread
has not yet started. In this case, data is sent to the pipe but no one

consumed

it. Later on, reading data from the pipe, will get unexpected data and

crash

the program.
 
This patch ensures the order: svc_monitor_thread must be in ready state
before spawn_services() is executed.
---
 src/nid/nodeinit.cc | 11 +++
 1 file changed, 11 insertions(+)
 
diff --git a/src/nid/nodeinit.cc b/src/nid/nodeinit.cc index
5f15916b4..b4945b05c 100644
--- a/src/nid/nodeinit.cc
+++ b/src/nid/nodeinit.cc
@@ -134,6 +134,7 @@ static int start_monitor_svc(const char *svc);
 /* Data declarations for service monitoring */  static int svc_mon_fd =

-1;

static int next_svc_fds_slot = 0;
+static bool svc_monitor_thread_running = false;
 
 struct SAFServices {
   const std::string fifo_dir = PKGLOCALSTATEDIR; @@ -1517,6 +1518,7 @@
void *svc_monitor_thread(void *fd) {
   next_svc_fds_slot++;
 
   while (true) {
+svc_monitor_thread_running = true;
 unsigned rc = osaf_poll(fds, next_svc_fds_slot, -1);
 if (rc > 0) {
   // check if any monitored service has exit @@ -1655,6 +1657,15 @@

int

main(int argc, char *argv[]) {
 exit(EXIT_FAILURE);
   }
 
+  // Waiting until svc_monitor_thread is up and in ready state.
+  // If spawn_services runs before the thread is in ready state,  //
+ receive side of the pipe s_pair will get unexpected data and  // may
+ crash the process.
+  while (svc_monitor_thread_running == false) {
+usleep(100);
+  }
+
+  LOG_NO("svc_monitor_thread is up and in ready state");
   if (parse_nodeinit_conf(sbuf) != NCSCC_RC_SUCCESS) {
 LOG_ER("Failed to parse file %s. Exiting", sbuf);
 exit(EXIT_FAILURE);
--
2.19.2

 
 


___
Opensaf-devel mailing list
Opensaf-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/opensaf-devel