Re: [linux-yocto] [PATCH 1/2] netatop: Initial integration

[Bruce Ashfield]

On 2018-04-14 11:27 AM, He Zhe wrote:



On 2018年04月10日 03:19, Bruce Ashfield wrote:

On 2018-04-05 3:08 PM, Bruce Ashfield wrote:

On 2018-03-28 11:25 PM, He Zhe wrote:

This is a net statistics collector for atop from:
http://atoptool.nl/netatop.php
https://atoptool.nl/download/netatop-2.0.tar.gz

Two changes:
Adjust pathes for netatop.h and netatopversion.h
Add module_init and module_exit


Which kernel versions have you tested this on ? Is there a way we could get
a README that describes how to build and test it ?


Also, is there a recipe that builds this that I can see/test ?


Sorry for slow reply, I'm on vacation and will handle this asap.
Is this expected to be built as a module via a recipe? Or as a patch/feature in 
yocto-kernel-cache?



I'd expect something like the perf recipe to build this directly
from the source tree.

That recipe can live wherever, i.e. a github based meta-layer, but
it should be available so that others can see how to build the code.

Bruce


Zhe



Bruce



Bruce



Signed-off-by: He Zhe 
---
   drivers/staging/netatop/netatop.c    | 1843 
++
   drivers/staging/netatop/netatop.h    |   47 +
   drivers/staging/netatop/netatopversion.h |    2 +
   3 files changed, 1892 insertions(+)
   create mode 100644 drivers/staging/netatop/netatop.c
   create mode 100644 drivers/staging/netatop/netatop.h
   create mode 100644 drivers/staging/netatop/netatopversion.h

diff --git a/drivers/staging/netatop/netatop.c 
b/drivers/staging/netatop/netatop.c
new file mode 100644
index ..3baa6520416a
--- /dev/null
+++ b/drivers/staging/netatop/netatop.c
@@ -0,0 +1,1843 @@
+/*
+** This module uses the netfilter interface to maintain statistics
+** about the network traffic per task, on level of thread group
+** and individual thread.
+**
+** General setup
+** -
+** Once the module is active, it is called for every packet that is
+** transmitted by a local process and every packet that is received
+** from an interface. Not only the packets that contain the user data
+** are passed but also the TCP related protocol packets (SYN, ACK, ...).
+**
+** When the module discovers a packet for a connection (TCP) or local
+** port (UDP) that is new, it creates a sockinfo structure. As soon as
+** possible the sockinfo struct will be connected to a taskinfo struct
+** that represents the proces or thread that is related to the socket.
+** However, the task can only be determined when a packet is transmitted,
+** i.e. the module is called during system call handling in the context
+** of the transmitting process. At that moment the tgid (process) and
+** pid (thread) can be obtained from the process administration to
+** be stored in the module's own taskinfo structs (one for the process,
+** one for the thread).
+** For the time that the sockinfo struct can not be related to a taskinfo
+** struct (e.g. when only packets are received), counters are maintained
+** temporarily in the sockinfo struct. As soon as a related taskinfo struct
+** is discovered when the task transmits, counters will be maintained in
+** the taskinfo struct itself.
+** When packets are only received for a socket (e.g. another machine is
+** sending UDP packets to the local machine) while the local task
+** never responds, no match to a process can be made and the packets
+** remain unidentified by the netatop module. At least one packet should
+** have been sent by a local process to be able to match packets for such
+** socket.
+** In the file /proc/netatop counters can be found that show the total
+** number of packets sent/received and how many of these packets were
+** unidentified (i.e. not accounted to a process/thread).
+**
+** Garbage collection
+** --
+** The module uses a garbage collector to cleanup the unused sockinfo
+** structs if connections do not exist any more (TCP) or have not been
+** used for some time (TCP/UDP).
+** Furthermore, the garbage collector checks if the taskinfo structs
+** still represent existing processes or threads. If not, the taskinfo struct
+** is destroyed (in case of a thread) or it is moved to a separate list of
+** finished processes (in case of a process). Analysis programs can read
+** the taskinfo of such finished process. When the taskinfo of a finished
+** process is not read within 15 seconds, the taskinfo will be destroyed.
+**
+** A garbage collector cycle can be triggered by issueing a getsockopt
+** call from an analysis program (e.g. atop). Apart from that, a time-based
+** garbage collector cycle is issued anyhow every 15 seconds by the
+** knetatop kernel thread.
+**
+** Interface with user mode
+** 
+** Programs can open an IP socket and use the getsockopt() system call
+** to issue commands to this module. With the command ATOP_GETCNT_TGID
+** the current counters can be obtained on process level (thread group)
+** and with the command ATOP_GETCNT_PID 

Re: [linux-yocto] [PATCH 1/2] netatop: Initial integration

[Bruce Ashfield]

On 2018-04-05 3:08 PM, Bruce Ashfield wrote:

On 2018-03-28 11:25 PM, He Zhe wrote:

This is a net statistics collector for atop from:
http://atoptool.nl/netatop.php
https://atoptool.nl/download/netatop-2.0.tar.gz

Two changes:
Adjust pathes for netatop.h and netatopversion.h
Add module_init and module_exit


Which kernel versions have you tested this on ? Is there a way we could get
a README that describes how to build and test it ?


Also, is there a recipe that builds this that I can see/test ?

Bruce



Bruce



Signed-off-by: He Zhe 
---
  drivers/staging/netatop/netatop.c    | 1843 
++

  drivers/staging/netatop/netatop.h    |   47 +
  drivers/staging/netatop/netatopversion.h |    2 +
  3 files changed, 1892 insertions(+)
  create mode 100644 drivers/staging/netatop/netatop.c
  create mode 100644 drivers/staging/netatop/netatop.h
  create mode 100644 drivers/staging/netatop/netatopversion.h

diff --git a/drivers/staging/netatop/netatop.c 
b/drivers/staging/netatop/netatop.c

new file mode 100644
index ..3baa6520416a
--- /dev/null
+++ b/drivers/staging/netatop/netatop.c
@@ -0,0 +1,1843 @@
+/*
+** This module uses the netfilter interface to maintain statistics
+** about the network traffic per task, on level of thread group
+** and individual thread.
+**
+** General setup
+** -
+** Once the module is active, it is called for every packet that is
+** transmitted by a local process and every packet that is received
+** from an interface. Not only the packets that contain the user data
+** are passed but also the TCP related protocol packets (SYN, ACK, ...).
+**
+** When the module discovers a packet for a connection (TCP) or local
+** port (UDP) that is new, it creates a sockinfo structure. As soon as
+** possible the sockinfo struct will be connected to a taskinfo struct
+** that represents the proces or thread that is related to the socket.
+** However, the task can only be determined when a packet is 
transmitted,

+** i.e. the module is called during system call handling in the context
+** of the transmitting process. At that moment the tgid (process) and
+** pid (thread) can be obtained from the process administration to
+** be stored in the module's own taskinfo structs (one for the process,
+** one for the thread).
+** For the time that the sockinfo struct can not be related to a 
taskinfo

+** struct (e.g. when only packets are received), counters are maintained
+** temporarily in the sockinfo struct. As soon as a related taskinfo 
struct

+** is discovered when the task transmits, counters will be maintained in
+** the taskinfo struct itself.
+** When packets are only received for a socket (e.g. another machine is
+** sending UDP packets to the local machine) while the local task
+** never responds, no match to a process can be made and the packets
+** remain unidentified by the netatop module. At least one packet should
+** have been sent by a local process to be able to match packets for 
such

+** socket.
+** In the file /proc/netatop counters can be found that show the total
+** number of packets sent/received and how many of these packets were
+** unidentified (i.e. not accounted to a process/thread).
+**
+** Garbage collection
+** --
+** The module uses a garbage collector to cleanup the unused sockinfo
+** structs if connections do not exist any more (TCP) or have not been
+** used for some time (TCP/UDP).
+** Furthermore, the garbage collector checks if the taskinfo structs
+** still represent existing processes or threads. If not, the 
taskinfo struct
+** is destroyed (in case of a thread) or it is moved to a separate 
list of

+** finished processes (in case of a process). Analysis programs can read
+** the taskinfo of such finished process. When the taskinfo of a 
finished
+** process is not read within 15 seconds, the taskinfo will be 
destroyed.

+**
+** A garbage collector cycle can be triggered by issueing a getsockopt
+** call from an analysis program (e.g. atop). Apart from that, a 
time-based

+** garbage collector cycle is issued anyhow every 15 seconds by the
+** knetatop kernel thread.
+**
+** Interface with user mode
+** 
+** Programs can open an IP socket and use the getsockopt() system call
+** to issue commands to this module. With the command ATOP_GETCNT_TGID
+** the current counters can be obtained on process level (thread group)
+** and with the command ATOP_GETCNT_PID the counters on thread level.
+** For both commands, the tgid/pid has to be passed of the required 
thread
+** (group). When the required thread (group) does not exist, an errno 
ESRCH

+** is given.
+**
+** The command ATOP_GETCNT_EXIT can be issued to obtain the counters of
+** an exited process. As stated above, such command has to be issued
+** within 15 seconds after a process has been declared 'finished' by
+** the garbage collector. Whenever this command is issued and no