So, we've been discussing IRL about the osmo-hnbgw implementation of
interacting with nftables from a separate thread.
The idea now is that a separate thread gets the results from nftables, caches
them and enqueues them, so that the main thread can update the hnbgw counters
in-sync. A second separate thread does nft add/del rule maintenance.
There is another complication, and I want to make sure we agree on the solution:
Problem: to delete a rule, I need to retrieve its 'handle' first.
With the amount of threading we want, this becomes a bit complex.
So. We 'add rule' and 'delete rule', as hNodeB attach and detach.
But, to be able to 'delete rule', we need to retrieve an id that nftables
assigned to this rule when it was added, the 'handle':
add rule inet osmo-hnbgw gtpu-ul ip saddr 1.2.3.4 counter comment
"ul:1-2-3-4";
...
list table inet osmo-hnbgw
-->
table inet osmo-hnbgw {
chain gtpu-ul {
ip saddr 1.2.3.4 counter packets 5 bytes 100 comment "ul:1-2-3-4"
# handle 23;
}
^^^^^^^^^
}
...
delete rule inet osmo-hnbgw gtpu-ul handle 23;
^^^^^^^^^
AFAICT I cannot set a handle right from the start. I can also not delete a rule
by just stating it again as it is, nor by using the 'comment' as a handle. nft
wants a 'handle', period. Does anyone know better?
Every time we read counters, osmo-hnbgw also retrieves the handles for each
rule, and correlates via the 'comment'. Similarly to counters, we can only get
all handles at once, via a complete table dump (possibly inefficient).
It works out fine in most practical cases. But the complex corner case is when
an hNodeB detaches quickly, before the counters had a chance to run and update
the hNodeB's state, including the handle.
In the current agreement on our implementation with two threads for nft, with
queues back to the main thread, it would go something like this, to avoid
races:
main() nft_maintenance_thread() nft_read_counters_thread()
| | |
|---add hNodeB-------->| |
|<--ok-----------------| |
X | |
|---del hNodeB-------->| |
| [rule handle unknown!] |
..|<--EAGAIN-------------| |
. | | |
. |---do counters--------------------------------->|
. | | [work]
. | | [work]
. | | [work]
. |<--update-counters-and-handles------------------|
. | | |
..|---del hNodeB-------->| |
| [delete rule] |
|<--done---------------| |
Simplifying a bit...
main() nft_maintenance_thread() nft_read_counters_thread()
| | |
|---add hNodeB-------->| |
|<--ok-----------------| |
X | |
[want: del hNodeB] | |
[rule handle unknown!] | |
[set some "del" flag] | |
. | | |
. |---do counters--------------------------------->|
. | | [work]
. | | [work]
. | | [work]
. |<--update-counters-and-handles------------------|
. | | |
[del flag!] | |
|---del hNodeB-------->| |
| [delete rule] |
|<--done---------------| |
I'm not liking very much the amount of complexity spawning here, but
this last approach is ok I guess, do you agree?
i.e. when the handle is not set yet, set a flag on the hNodeB,
and trigger the 'delete' the next time the counter thread reports back a handle
for the hNodeB.
~N
--
- Neels Hofmeyr <nhofm...@sysmocom.de> https://www.sysmocom.de/
=======================================================================
* sysmocom - systems for mobile communications GmbH
* Siemensstr. 26a
* 10551 Berlin, Germany
* Sitz / Registered office: Berlin, HRB 134158 B
* Geschaeftsfuehrer / Managing Director: Harald Welte
