Dear list, when I was debugging message loss in a core test case last year, I observed that the sending of messages got stuck in the following part of core:
In the beginning of `try_transmission()` the queue size of the core
session gets checked for its size. If it is full, the function issues a
debug message and returns. To my current understanding this raises some
questions and halts the communication between client and service.
See `src/service/core/gnunet-service-core_sessions.c:475` and following.
1. The queue size is defined with `MAX_ENCRYPTED_MESSAGE_QUEUE_SIZE` as
`4`. My intuition tells me that this might be quite short, but my
intuition is not based on many data points.
2. The dropping of the message to be transmitted is quite silent. Just
issuing a debug is too little. I would expect a warning and a
counter in statistics.
3. The comment talks about 'excess bandwidth'. Either the comment is
misplaced and comments the block after the check or it is correctly
placed and the check for excess bandwidth is redundant with the
next block.
4. The biggest issue is - again, to my current understanding - that
once this condition is hit a single time, the communication between
client and service halts in an unintended/defunct state: All other
code branches that leave the function `try_transmission()` call
`solicit_messages()`, which in turn sends a message of type
`GNUNET_MESSAGE_TYPE_CORE_SEND_READY` to the client. Without
receiving this message, the client never calls
`GNUNET_MQ_impl_send_continue()` and thus can not send any other
message to the service. I may have missed something, but that is
what it looks like to me.
Severity/Age:
I am not so sure about the severity of it - `git blame` tells me that
this code has been around since 2015/2019. I assume it hasn't been hit
that often. But would explain some hanging of core.
Potential fixes:
0. Disable the check. (I am actually not sure whether we need this
limitation of maximum queue size. There is a lot of checks in the
function in question that might already cover resources exhaustion
and might do so in a much cleaner way.)
1. To smoothen out the problem, the maximum queue size could be
increased.
2. Feedback to the client: Send a message to the client that signals
that the current sending queue is full and let the client deal with
re-issuing the message. At least send something back to the client
so the client can get unstuck.
3. Implement a timer in the client that assumes message loss and gets
the client unstuck.
Please have a look yourself. Check whether I wrote applies. I very well
might have missed something. Maybe there are other code paths or
mechanisms I'm not aware of. The code across different processes with
all the callbacks and its size makes it difficult for me to understand
well.
Moving forward:
I remember disabeling the size check last year. I think it improved the
results of my test, but it was still loosing messages in other places. I
will try to find these next. Before I go to more debugging, I will try
to get my testing-based testcase in a good shape, so I have a good basis
to work against.
Best,
ch3
bin675UwYMvly.bin
Description: PGP Key 0x494E1084E37E08C8F539C0C1FA15A4D6BF109DF5.
signature.asc
Description: PGP signature
