Re: [vpp-dev] api msg deadlock

[NUAA无痕]

Hi ,florin

if i add many message once in a while, svm_queue_add will deadlock due to
q->cursize == q->maxsize?
Maybe this reason cause deadlock

int
svm_queue_add (svm_queue_t * q, u8 * elem, int nowait)
{
  i8 *tailp;
  int need_broadcast = 0;

  if (nowait)
{
  /* zero on success */
  if (svm_queue_trylock (q))
{
 return (-1);
}
}
  else
svm_queue_lock (q);

  if (PREDICT_FALSE (q->cursize == q->maxsize))
{
  if (nowait)
{
 svm_queue_unlock (q);
 return (-2);
}
  while (q->cursize == q->maxsize)
svm_queue_wait_inline (q);
}

  tailp = (i8 *) (>data[0] + q->elsize * q->tail);
  clib_memcpy_fast (tailp, elem, q->elsize);

  q->tail++;
  q->cursize++;

  need_broadcast = (q->cursize == 1);

  if (q->tail == q->maxsize)
q->tail = 0;

  if (need_broadcast)
svm_queue_send_signal_inline (q, 1);

  svm_queue_unlock (q);

  return 0;
}

NUAA无痕 via lists.fd.io  于2022年6月6日周一
14:52写道：

> Hi, florin
>
> i have study jvpp code and found that maybe jvpp is ok, error occurs in
> vpp svm_queue
>
> jvpp connect vpp by vl_client_connect_to_vlib function
> (vpp/src/vlibmemory/memory_client.c)
>
> int
> vl_client_connect_to_vlib (const char *svm_name,
>   const char *client_name, int rx_queue_size)
> {
>   return connect_to_vlib_internal (svm_name, client_name, rx_queue_size,
>   rx_thread_fn, 0 /* thread fn arg */ ,
>   1 /* do map */ );
> }
>
> jvpp example is
>  jvpp-registry/jvpp_registry.c:if
> (vl_client_connect_to_vlib(shm_prefix, name, 32) < 0)
>
> i change 32 to 4096, now vpp has run three days and not deadlock
>
> so i think that whether rx_queue_size parameter is too small, resulting in
> function error handling
>
> I also compare vpp-2101 with vpp-2206 and found that binary api code(
> svm_queue in queue.c ) is no change
>
> If i send a large message that size over rx_queue_size will cause problem
> ?
> Looking forward to your opinion
>
> Best regards
> wanghe
>
> Florin Coras  于2022年6月3日周五 23:28写道：
>
>> Hi Wanghe,
>>
>> The only api bindings supported today are c, python and golang. Maybe
>> somebody familiar with the jvpp code can help you out but otherwise I’d
>> recommend to switch if possible.
>>
>> Regards,
>> Florin
>>
>> > On Jun 3, 2022, at 7:55 AM, NUAA无痕  wrote:
>> >
>> > Hi, florin
>> >
>> > About this question, i compare c++ code with jvpp code， then i found
>> that jvpp maybe have a bug and even if update vpp also cannot resolve it
>> >
>> > jvpp code according to vpp version 1901, that has jvpp example
>> >
>> vpp-1901/extras/japi/java/jvpp-core/io/fd/vpp/jvpp/core/examples/CreateSubInterfaceExample.java
>> has jvpp example and our code according to it
>> >
>> > now vpp version is 2101
>> > then when java code connected to vpp and then use "close“ function it
>> will hint "peer unresponsive, give up"
>> > this error from src/vlibmemory/memory_client.c vl_client_disconnect
>> function
>> >
>> > why this error is that svm_queue_sub always return -2 until timeout
>> >
>> > this is code , the reason is that "vl_input_queue->cursize == 0 " and
>> vl_input_queue->head == vl_input_queue->tail
>> >
>> > int
>> > vl_client_disconnect (void)
>> > {
>> >   vl_api_memclnt_delete_reply_t *rp;
>> >   svm_queue_t *vl_input_queue;
>> >   api_main_t *am = vlibapi_get_main ();
>> >   time_t begin;
>> >
>> >   vl_input_queue = am->vl_input_queue;
>> >   vl_client_send_disconnect (0 /* wait for reply */ );
>> >
>> >   /*
>> >* Have to be careful here, in case the client is disconnecting
>> >* because e.g. the vlib process died, or is unresponsive.
>> >*/
>> >   begin = time (0);
>> >   while (1)
>> > {
>> >   time_t now;
>> >
>> >   now = time (0);
>> >
>> >   if (now >= (begin + 2))
>> > {
>> >  clib_warning ("peer unresponsive, give up");
>> >  am->my_client_index = ~0;
>> >  am->my_registration = 0;
>> >  am->shmem_hdr = 0;
>> >  return -1;
>> > }
>> >
>> > /* this error because vl_input_queue->cursize == 0  */
>> >   if (svm_queue_sub (vl_input_queue, (u8 *) & rp, SVM_Q_NOWAIT, 0)
>> < 0)
>> > continue;
>> >
>> >   VL_MSG_API_UNPOISON (rp);
>> >
>> >   /* drain the queue */
>> >   if (ntohs (rp->_vl_msg_id) != VL_API_MEMCLNT_DELETE_REPLY)
>> > {
>> >  clib_warning ("queue drain: %d", ntohs (rp->_vl_msg_id));
>> >  vl_msg_api_handler ((void *) rp);
>> >  continue;
>> > }
>> >   vl_msg_api_handler ((void *) rp);
>> >   break;
>> > }
>> >
>> >   vl_api_name_and_crc_free ();
>> >   return 0;
>> > }
>> >
>> > when i use c++ for vpp binary api,  vl_input_queue->cursize == 1 and
>> vl_input_queue->head != vl_input_queue->tail
>> >
>> > so c++ use binary api is correct that about svm_queue_* series functions
>> >
>> > Although JVpp is no longer supported， but this is important for me!
>> >
>> > Can you give a patch for jvpp? Thanks
>> >
>> > Best regards
>> >
>> > Wanghe
>> >
>> >
>>
>>
> 
>
>

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#21508): 

Re: [vpp-dev] api msg deadlock

[NUAA无痕]

Hi, florin

i have study jvpp code and found that maybe jvpp is ok, error occurs in vpp
svm_queue

jvpp connect vpp by vl_client_connect_to_vlib function
(vpp/src/vlibmemory/memory_client.c)

int
vl_client_connect_to_vlib (const char *svm_name,
  const char *client_name, int rx_queue_size)
{
  return connect_to_vlib_internal (svm_name, client_name, rx_queue_size,
  rx_thread_fn, 0 /* thread fn arg */ ,
  1 /* do map */ );
}

jvpp example is
 jvpp-registry/jvpp_registry.c:if
(vl_client_connect_to_vlib(shm_prefix, name, 32) < 0)

i change 32 to 4096, now vpp has run three days and not deadlock

so i think that whether rx_queue_size parameter is too small, resulting in
function error handling

I also compare vpp-2101 with vpp-2206 and found that binary api code(
svm_queue in queue.c ) is no change

If i send a large message that size over rx_queue_size will cause problem ?
Looking forward to your opinion

Best regards
wanghe

Florin Coras  于2022年6月3日周五 23:28写道：

> Hi Wanghe,
>
> The only api bindings supported today are c, python and golang. Maybe
> somebody familiar with the jvpp code can help you out but otherwise I’d
> recommend to switch if possible.
>
> Regards,
> Florin
>
> > On Jun 3, 2022, at 7:55 AM, NUAA无痕  wrote:
> >
> > Hi, florin
> >
> > About this question, i compare c++ code with jvpp code， then i found
> that jvpp maybe have a bug and even if update vpp also cannot resolve it
> >
> > jvpp code according to vpp version 1901, that has jvpp example
> >
> vpp-1901/extras/japi/java/jvpp-core/io/fd/vpp/jvpp/core/examples/CreateSubInterfaceExample.java
> has jvpp example and our code according to it
> >
> > now vpp version is 2101
> > then when java code connected to vpp and then use "close“ function it
> will hint "peer unresponsive, give up"
> > this error from src/vlibmemory/memory_client.c vl_client_disconnect
> function
> >
> > why this error is that svm_queue_sub always return -2 until timeout
> >
> > this is code , the reason is that "vl_input_queue->cursize == 0 " and
> vl_input_queue->head == vl_input_queue->tail
> >
> > int
> > vl_client_disconnect (void)
> > {
> >   vl_api_memclnt_delete_reply_t *rp;
> >   svm_queue_t *vl_input_queue;
> >   api_main_t *am = vlibapi_get_main ();
> >   time_t begin;
> >
> >   vl_input_queue = am->vl_input_queue;
> >   vl_client_send_disconnect (0 /* wait for reply */ );
> >
> >   /*
> >* Have to be careful here, in case the client is disconnecting
> >* because e.g. the vlib process died, or is unresponsive.
> >*/
> >   begin = time (0);
> >   while (1)
> > {
> >   time_t now;
> >
> >   now = time (0);
> >
> >   if (now >= (begin + 2))
> > {
> >  clib_warning ("peer unresponsive, give up");
> >  am->my_client_index = ~0;
> >  am->my_registration = 0;
> >  am->shmem_hdr = 0;
> >  return -1;
> > }
> >
> > /* this error because vl_input_queue->cursize == 0  */
> >   if (svm_queue_sub (vl_input_queue, (u8 *) & rp, SVM_Q_NOWAIT, 0) <
> 0)
> > continue;
> >
> >   VL_MSG_API_UNPOISON (rp);
> >
> >   /* drain the queue */
> >   if (ntohs (rp->_vl_msg_id) != VL_API_MEMCLNT_DELETE_REPLY)
> > {
> >  clib_warning ("queue drain: %d", ntohs (rp->_vl_msg_id));
> >  vl_msg_api_handler ((void *) rp);
> >  continue;
> > }
> >   vl_msg_api_handler ((void *) rp);
> >   break;
> > }
> >
> >   vl_api_name_and_crc_free ();
> >   return 0;
> > }
> >
> > when i use c++ for vpp binary api,  vl_input_queue->cursize == 1 and
> vl_input_queue->head != vl_input_queue->tail
> >
> > so c++ use binary api is correct that about svm_queue_* series functions
> >
> > Although JVpp is no longer supported， but this is important for me!
> >
> > Can you give a patch for jvpp? Thanks
> >
> > Best regards
> >
> > Wanghe
> >
> >
>
>

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#21507): https://lists.fd.io/g/vpp-dev/message/21507
Mute This Topic: https://lists.fd.io/mt/91372330/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/leave/1480452/21656/631435203/xyzzy 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] api msg deadlock

[Florin Coras]

Hi Wanghe, 

The only api bindings supported today are c, python and golang. Maybe somebody 
familiar with the jvpp code can help you out but otherwise I’d recommend to 
switch if possible. 

Regards,
Florin

> On Jun 3, 2022, at 7:55 AM, NUAA无痕  wrote:
> 
> Hi, florin
> 
> About this question, i compare c++ code with jvpp code， then i found that 
> jvpp maybe have a bug and even if update vpp also cannot resolve it
> 
> jvpp code according to vpp version 1901, that has jvpp example
> vpp-1901/extras/japi/java/jvpp-core/io/fd/vpp/jvpp/core/examples/CreateSubInterfaceExample.java
>   has jvpp example and our code according to it
> 
> now vpp version is 2101
> then when java code connected to vpp and then use "close“ function it will 
> hint "peer unresponsive, give up" 
> this error from src/vlibmemory/memory_client.c vl_client_disconnect function
> 
> why this error is that svm_queue_sub always return -2 until timeout
> 
> this is code , the reason is that "vl_input_queue->cursize == 0 " and 
> vl_input_queue->head == vl_input_queue->tail
> 
> int
> vl_client_disconnect (void)
> {
>   vl_api_memclnt_delete_reply_t *rp;
>   svm_queue_t *vl_input_queue;
>   api_main_t *am = vlibapi_get_main ();
>   time_t begin;
> 
>   vl_input_queue = am->vl_input_queue;
>   vl_client_send_disconnect (0 /* wait for reply */ );
> 
>   /*
>* Have to be careful here, in case the client is disconnecting
>* because e.g. the vlib process died, or is unresponsive.
>*/
>   begin = time (0);
>   while (1)
> {
>   time_t now;
> 
>   now = time (0);
> 
>   if (now >= (begin + 2))
> {
>  clib_warning ("peer unresponsive, give up");
>  am->my_client_index = ~0;
>  am->my_registration = 0;
>  am->shmem_hdr = 0;
>  return -1;
> }
> 
> /* this error because vl_input_queue->cursize == 0  */
>   if (svm_queue_sub (vl_input_queue, (u8 *) & rp, SVM_Q_NOWAIT, 0) < 0)
> continue;
> 
>   VL_MSG_API_UNPOISON (rp);
> 
>   /* drain the queue */
>   if (ntohs (rp->_vl_msg_id) != VL_API_MEMCLNT_DELETE_REPLY)
> {
>  clib_warning ("queue drain: %d", ntohs (rp->_vl_msg_id));
>  vl_msg_api_handler ((void *) rp);
>  continue;
> }
>   vl_msg_api_handler ((void *) rp);
>   break;
> }
> 
>   vl_api_name_and_crc_free ();
>   return 0;
> }
> 
> when i use c++ for vpp binary api,  vl_input_queue->cursize == 1 and 
> vl_input_queue->head != vl_input_queue->tail
> 
> so c++ use binary api is correct that about svm_queue_* series functions
> 
> Although JVpp is no longer supported， but this is important for me!
> 
> Can you give a patch for jvpp? Thanks
> 
> Best regards
> 
> Wanghe
> 
> 


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#21501): https://lists.fd.io/g/vpp-dev/message/21501
Mute This Topic: https://lists.fd.io/mt/91372330/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] api msg deadlock

[NUAA无痕]

Hi, florin

About this question, i compare c++ code with jvpp code， then i found that
jvpp maybe have a bug and even if update vpp also cannot resolve it

jvpp code according to vpp version 1901, that has jvpp example
vpp-1901/extras/japi/java/jvpp-core/io/fd/vpp/jvpp/core/examples/CreateSubInterfaceExample.java
has jvpp example and our code according to it

now vpp version is 2101
then when java code connected to vpp and then use "close“ function it will
hint "peer unresponsive, give up"
this error from src/vlibmemory/memory_client.c vl_client_disconnect function

why this error is that svm_queue_sub always return -2 until timeout

this is code , the reason is that "vl_input_queue->cursize == 0 " and
vl_input_queue->head == vl_input_queue->tail

int
vl_client_disconnect (void)
{
  vl_api_memclnt_delete_reply_t *rp;
  svm_queue_t *vl_input_queue;
  api_main_t *am = vlibapi_get_main ();
  time_t begin;

  vl_input_queue = am->vl_input_queue;
  vl_client_send_disconnect (0 /* wait for reply */ );

  /*
   * Have to be careful here, in case the client is disconnecting
   * because e.g. the vlib process died, or is unresponsive.
   */
  begin = time (0);
  while (1)
{
  time_t now;

  now = time (0);

  if (now >= (begin + 2))
{
 clib_warning ("peer unresponsive, give up");
 am->my_client_index = ~0;
 am->my_registration = 0;
 am->shmem_hdr = 0;
 return -1;
}

/* this error because vl_input_queue->cursize == 0  */
  if (svm_queue_sub (vl_input_queue, (u8 *) & rp, SVM_Q_NOWAIT, 0) < 0)
continue;

  VL_MSG_API_UNPOISON (rp);

  /* drain the queue */
  if (ntohs (rp->_vl_msg_id) != VL_API_MEMCLNT_DELETE_REPLY)
{
 clib_warning ("queue drain: %d", ntohs (rp->_vl_msg_id));
 vl_msg_api_handler ((void *) rp);
 continue;
}
  vl_msg_api_handler ((void *) rp);
  break;
}

  vl_api_name_and_crc_free ();
  return 0;
}

when i use c++ for vpp binary api,  vl_input_queue->cursize == 1 and
vl_input_queue->head != vl_input_queue->tail

so c++ use binary api is correct that about svm_queue_* series functions

Although JVpp is no longer supported， but this is important for me!

Can you give a patch for jvpp? Thanks

Best regards

Wanghe

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#21500): https://lists.fd.io/g/vpp-dev/message/21500
Mute This Topic: https://lists.fd.io/mt/91372330/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] api msg deadlock

[NUAA无痕]

Hi florin,
Thanks for your suggestion, i will update vpp to new version for this
question

In addition that is vpp has example for c++/java binary api use， so i can
compare it with our project code
our c++ code refer to https://pantheon.tech/vpp-101-plugins-binary-api/

Thank u very much

Best regards
wanghe

Florin Coras  于2022年5月29日周日 03:37写道：

> Hi wanghe,
>
> Neither vpp 21.01 nor jvpp are supported, so your only options are to try
> back porting fixes from newer versions, if any exist, or to debug the
> problem.
>
> As previously mentioned, the deadlock seem to be in a reply message, so
> the issue is probably in the java/c++ implementation of the binary api
> client or the way the api is used by the client. Either the api client is
> not dequeueing messages, e.g., maybe it’s stuck waiting on vpp, or, if did
> dequeue, it did not broadcast on the condvar or the broadcast was missed by
> vpp.
>
> Try to check what your api client is doing. That might shed some light on
> the issue.
>
> Hope this helps.
>
> Regards,
> Florin
>
> On May 27, 2022, at 7:57 PM, NUAA无痕  wrote:
>
> Hi，florin
>
> I would appreciate it if you can resolve it
>
> My project use java web to control vpp, many binary api are used, such as
> interfaces, ip, our customize api etc.
>
> half a year ago， i use one c++ restful framework ( google‘s pistache) to
> use vpp binary api, there also has deadlock problem , then i don't know to
> send mail for help (dont know must subscribe mail  can send mail to vpp-dev
> haha)
>
> i think maybe i m not proficient with c++ multithreading that cause
> deadlock, that time i fount if use one thread also api communication will
> deadlock
>
> so i think many times use api will cause vpp deadlock
>
> For resolve this problem, we decide to change it to jvpp, maybe jvpp
> resolve this deadlock （now this problem also exist）
>
> i found git log for vpp new version that vpp resolve a deadlock about svm
> （i dont know if it can resolve this）, but now we update vpp need at least
> one month （maybe too long）
>
> So florin expert , can you analyze this problem?  thank you very much!
>
> Best regards
> wanghe
>
>
>
>
> Florin Coras  于2022年5月28日周六 01:02写道：
>
>> Hi wanghe,
>>
>> Unfortunately, jvpp is no longer supported so probably there’s no recent
>> fix for the issue you’re hitting. By the looks of it, an api msg handler is
>> trying to enqueue something (probably a reply towards the client) and ends
>> up stuck because the svm queue is full and a condvar broadcast never comes.
>>
>> If you really need to fix this, I’d check jvpp code to see if condvar
>> broadcasts on dequeue are done properly.
>>
>> Regards,
>> Florin
>>
>> > On May 27, 2022, at 12:53 AM, NUAA无痕  wrote:
>> >
>> > Hi, vpp experts
>> >
>> > im use vpp 2101 version
>> > my project use jvpp communicate with vpp by binary api, but now when
>> long time run(about 14h) it will deadlock, this is info
>> >
>> > 0x7f2687783a35 in pthread_cond_wait@@GLIBC_2.3.2 () from
>> /usr/lib64/libpthread.so.0
>> > (gdb) bt
>> > #0  0x7f2687783a35 in pthread_cond_wait@@GLIBC_2.3.2 () from
>> /usr/lib64/libpthread.so.0
>> > #1  0x7f2688f198e6 in svm_queue_add () from
>> /usr/local/zfp/lib/libsvm.so.21.01.1
>> > #2  0x55cdda6856f3 in ?? ()
>> > #3  0x7f268904c909 in vl_msg_api_handler_with_vm_node () from
>> /usr/local/zfp/lib/libvlibmemory.so.21.01.1
>> > #4  0x7f2689033521 in vl_mem_api_handle_msg_main () from
>> /usr/local/zfp/lib/libvlibmemory.so.21.01.1
>> > #5  0x7f2689043fce in ?? () from
>> /usr/local/zfp/lib/libvlibmemory.so.21.01.1
>> > #6  0x7f2688fba5a7 in ?? () from
>> /usr/local/zfp/lib/libvlib.so.21.01.1
>> > #7  0x7f2688ef8de0 in clib_calljmp () from
>> /usr/local/zfp/lib/libvppinfra.so.21.01.1
>> > #8  0x7f263d673dd0 in ?? ()
>> > #9  0x7f2688fbdf67 in ?? () from
>> /usr/local/zfp/lib/libvlib.so.21.01.1
>> > #10 0x in ?? ()
>> >
>> > because i use release version so some info is not show, i found that
>> vpp new version has change a lot about svm.
>> >
>> > for some reason，i need some time to update vpp and now must resolve
>> this problem
>> >
>> > so experts can you give patch for this bug for vpp 2101 version
>> >
>> > Best regards
>> > wanghe
>> >
>> > 
>> >
>>
>>
>

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#21481): https://lists.fd.io/g/vpp-dev/message/21481
Mute This Topic: https://lists.fd.io/mt/91372330/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] api msg deadlock

[Florin Coras]

Hi wanghe, 

Neither vpp 21.01 nor jvpp are supported, so your only options are to try back 
porting fixes from newer versions, if any exist, or to debug the problem. 

As previously mentioned, the deadlock seem to be in a reply message, so the 
issue is probably in the java/c++ implementation of the binary api client or 
the way the api is used by the client. Either the api client is not dequeueing 
messages, e.g., maybe it’s stuck waiting on vpp, or, if did dequeue, it did not 
broadcast on the condvar or the broadcast was missed by vpp. 

Try to check what your api client is doing. That might shed some light on the 
issue.

Hope this helps. 

Regards,
Florin

> On May 27, 2022, at 7:57 PM, NUAA无痕  wrote:
> 
> Hi，florin
> 
> I would appreciate it if you can resolve it
> 
> My project use java web to control vpp, many binary api are used, such as 
> interfaces, ip, our customize api etc.
> 
> half a year ago， i use one c++ restful framework ( google‘s pistache) to use 
> vpp binary api, there also has deadlock problem , then i don't know to send 
> mail for help (dont know must subscribe mail  can send mail to vpp-dev haha)
> 
> i think maybe i m not proficient with c++ multithreading that cause deadlock, 
> that time i fount if use one thread also api communication will deadlock
> 
> so i think many times use api will cause vpp deadlock
> 
> For resolve this problem, we decide to change it to jvpp, maybe jvpp resolve 
> this deadlock （now this problem also exist）
> 
> i found git log for vpp new version that vpp resolve a deadlock about svm （i 
> dont know if it can resolve this）, but now we update vpp need at least one 
> month （maybe too long）
> 
> So florin expert , can you analyze this problem?  thank you very much!
> 
> Best regards
> wanghe
> 
> 
> 
> 
> Florin Coras mailto:fcoras.li...@gmail.com>> 
> 于2022年5月28日周六 01:02写道：
> Hi wanghe, 
> 
> Unfortunately, jvpp is no longer supported so probably there’s no recent fix 
> for the issue you’re hitting. By the looks of it, an api msg handler is 
> trying to enqueue something (probably a reply towards the client) and ends up 
> stuck because the svm queue is full and a condvar broadcast never comes. 
> 
> If you really need to fix this, I’d check jvpp code to see if condvar 
> broadcasts on dequeue are done properly. 
> 
> Regards,
> Florin
> 
> > On May 27, 2022, at 12:53 AM, NUAA无痕  > > wrote:
> > 
> > Hi, vpp experts
> > 
> > im use vpp 2101 version
> > my project use jvpp communicate with vpp by binary api, but now when long 
> > time run(about 14h) it will deadlock, this is info
> > 
> > 0x7f2687783a35 in pthread_cond_wait@@GLIBC_2.3.2 () from 
> > /usr/lib64/libpthread.so.0
> > (gdb) bt
> > #0  0x7f2687783a35 in pthread_cond_wait@@GLIBC_2.3.2 () from 
> > /usr/lib64/libpthread.so.0
> > #1  0x7f2688f198e6 in svm_queue_add () from 
> > /usr/local/zfp/lib/libsvm.so.21.01.1
> > #2  0x55cdda6856f3 in ?? ()
> > #3  0x7f268904c909 in vl_msg_api_handler_with_vm_node () from 
> > /usr/local/zfp/lib/libvlibmemory.so.21.01.1
> > #4  0x7f2689033521 in vl_mem_api_handle_msg_main () from 
> > /usr/local/zfp/lib/libvlibmemory.so.21.01.1
> > #5  0x7f2689043fce in ?? () from 
> > /usr/local/zfp/lib/libvlibmemory.so.21.01.1
> > #6  0x7f2688fba5a7 in ?? () from /usr/local/zfp/lib/libvlib.so.21.01.1
> > #7  0x7f2688ef8de0 in clib_calljmp () from 
> > /usr/local/zfp/lib/libvppinfra.so.21.01.1
> > #8  0x7f263d673dd0 in ?? ()
> > #9  0x7f2688fbdf67 in ?? () from /usr/local/zfp/lib/libvlib.so.21.01.1
> > #10 0x in ?? ()
> > 
> > because i use release version so some info is not show, i found that vpp 
> > new version has change a lot about svm.
> > 
> > for some reason，i need some time to update vpp and now must resolve this 
> > problem
> > 
> > so experts can you give patch for this bug for vpp 2101 version
> > 
> > Best regards
> > wanghe
> > 
> > 
> > 
> 


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#21480): https://lists.fd.io/g/vpp-dev/message/21480
Mute This Topic: https://lists.fd.io/mt/91372330/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] api msg deadlock

[NUAA无痕]

Hi，florin

I would appreciate it if you can resolve it

My project use java web to control vpp, many binary api are used, such as
interfaces, ip, our customize api etc.

half a year ago， i use one c++ restful framework ( google‘s pistache) to
use vpp binary api, there also has deadlock problem , then i don't know to
send mail for help (dont know must subscribe mail  can send mail to vpp-dev
haha)

i think maybe i m not proficient with c++ multithreading that cause
deadlock, that time i fount if use one thread also api communication will
deadlock

so i think many times use api will cause vpp deadlock

For resolve this problem, we decide to change it to jvpp, maybe jvpp
resolve this deadlock （now this problem also exist）

i found git log for vpp new version that vpp resolve a deadlock about svm
（i dont know if it can resolve this）, but now we update vpp need at least
one month （maybe too long）

So florin expert , can you analyze this problem?  thank you very much!

Best regards
wanghe




Florin Coras  于2022年5月28日周六 01:02写道：

> Hi wanghe,
>
> Unfortunately, jvpp is no longer supported so probably there’s no recent
> fix for the issue you’re hitting. By the looks of it, an api msg handler is
> trying to enqueue something (probably a reply towards the client) and ends
> up stuck because the svm queue is full and a condvar broadcast never comes.
>
> If you really need to fix this, I’d check jvpp code to see if condvar
> broadcasts on dequeue are done properly.
>
> Regards,
> Florin
>
> > On May 27, 2022, at 12:53 AM, NUAA无痕  wrote:
> >
> > Hi, vpp experts
> >
> > im use vpp 2101 version
> > my project use jvpp communicate with vpp by binary api, but now when
> long time run(about 14h) it will deadlock, this is info
> >
> > 0x7f2687783a35 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /usr/lib64/libpthread.so.0
> > (gdb) bt
> > #0  0x7f2687783a35 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /usr/lib64/libpthread.so.0
> > #1  0x7f2688f198e6 in svm_queue_add () from
> /usr/local/zfp/lib/libsvm.so.21.01.1
> > #2  0x55cdda6856f3 in ?? ()
> > #3  0x7f268904c909 in vl_msg_api_handler_with_vm_node () from
> /usr/local/zfp/lib/libvlibmemory.so.21.01.1
> > #4  0x7f2689033521 in vl_mem_api_handle_msg_main () from
> /usr/local/zfp/lib/libvlibmemory.so.21.01.1
> > #5  0x7f2689043fce in ?? () from
> /usr/local/zfp/lib/libvlibmemory.so.21.01.1
> > #6  0x7f2688fba5a7 in ?? () from
> /usr/local/zfp/lib/libvlib.so.21.01.1
> > #7  0x7f2688ef8de0 in clib_calljmp () from
> /usr/local/zfp/lib/libvppinfra.so.21.01.1
> > #8  0x7f263d673dd0 in ?? ()
> > #9  0x7f2688fbdf67 in ?? () from
> /usr/local/zfp/lib/libvlib.so.21.01.1
> > #10 0x in ?? ()
> >
> > because i use release version so some info is not show, i found that vpp
> new version has change a lot about svm.
> >
> > for some reason，i need some time to update vpp and now must resolve this
> problem
> >
> > so experts can you give patch for this bug for vpp 2101 version
> >
> > Best regards
> > wanghe
> >
> > 
> >
>
>

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#21479): https://lists.fd.io/g/vpp-dev/message/21479
Mute This Topic: https://lists.fd.io/mt/91372330/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] api msg deadlock

[Florin Coras]

Hi wanghe, 

Unfortunately, jvpp is no longer supported so probably there’s no recent fix 
for the issue you’re hitting. By the looks of it, an api msg handler is trying 
to enqueue something (probably a reply towards the client) and ends up stuck 
because the svm queue is full and a condvar broadcast never comes. 

If you really need to fix this, I’d check jvpp code to see if condvar 
broadcasts on dequeue are done properly. 

Regards,
Florin

> On May 27, 2022, at 12:53 AM, NUAA无痕  wrote:
> 
> Hi, vpp experts
> 
> im use vpp 2101 version
> my project use jvpp communicate with vpp by binary api, but now when long 
> time run(about 14h) it will deadlock, this is info
> 
> 0x7f2687783a35 in pthread_cond_wait@@GLIBC_2.3.2 () from 
> /usr/lib64/libpthread.so.0
> (gdb) bt
> #0  0x7f2687783a35 in pthread_cond_wait@@GLIBC_2.3.2 () from 
> /usr/lib64/libpthread.so.0
> #1  0x7f2688f198e6 in svm_queue_add () from 
> /usr/local/zfp/lib/libsvm.so.21.01.1
> #2  0x55cdda6856f3 in ?? ()
> #3  0x7f268904c909 in vl_msg_api_handler_with_vm_node () from 
> /usr/local/zfp/lib/libvlibmemory.so.21.01.1
> #4  0x7f2689033521 in vl_mem_api_handle_msg_main () from 
> /usr/local/zfp/lib/libvlibmemory.so.21.01.1
> #5  0x7f2689043fce in ?? () from 
> /usr/local/zfp/lib/libvlibmemory.so.21.01.1
> #6  0x7f2688fba5a7 in ?? () from /usr/local/zfp/lib/libvlib.so.21.01.1
> #7  0x7f2688ef8de0 in clib_calljmp () from 
> /usr/local/zfp/lib/libvppinfra.so.21.01.1
> #8  0x7f263d673dd0 in ?? ()
> #9  0x7f2688fbdf67 in ?? () from /usr/local/zfp/lib/libvlib.so.21.01.1
> #10 0x in ?? ()
> 
> because i use release version so some info is not show, i found that vpp new 
> version has change a lot about svm.
> 
> for some reason，i need some time to update vpp and now must resolve this 
> problem
> 
> so experts can you give patch for this bug for vpp 2101 version
> 
> Best regards
> wanghe
> 
> 
> 


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#21478): https://lists.fd.io/g/vpp-dev/message/21478
Mute This Topic: https://lists.fd.io/mt/91372330/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-