Re: [linux-yocto] [PATCH 1/2] netatop: Initial integration
On 2018-04-14 11:27 AM, He Zhe wrote: On 2018年04月10日 03:19, Bruce Ashfield wrote: On 2018-04-05 3:08 PM, Bruce Ashfield wrote: On 2018-03-28 11:25 PM, He Zhe wrote: This is a net statistics collector for atop from: http://atoptool.nl/netatop.php https://atoptool.nl/download/netatop-2.0.tar.gz Two changes: Adjust pathes for netatop.h and netatopversion.h Add module_init and module_exit Which kernel versions have you tested this on ? Is there a way we could get a README that describes how to build and test it ? Also, is there a recipe that builds this that I can see/test ? Sorry for slow reply, I'm on vacation and will handle this asap. Is this expected to be built as a module via a recipe? Or as a patch/feature in yocto-kernel-cache? I'd expect something like the perf recipe to build this directly from the source tree. That recipe can live wherever, i.e. a github based meta-layer, but it should be available so that others can see how to build the code. Bruce Zhe Bruce Bruce Signed-off-by: He Zhe --- drivers/staging/netatop/netatop.c | 1843 ++ drivers/staging/netatop/netatop.h | 47 + drivers/staging/netatop/netatopversion.h | 2 + 3 files changed, 1892 insertions(+) create mode 100644 drivers/staging/netatop/netatop.c create mode 100644 drivers/staging/netatop/netatop.h create mode 100644 drivers/staging/netatop/netatopversion.h diff --git a/drivers/staging/netatop/netatop.c b/drivers/staging/netatop/netatop.c new file mode 100644 index ..3baa6520416a --- /dev/null +++ b/drivers/staging/netatop/netatop.c @@ -0,0 +1,1843 @@ +/* +** This module uses the netfilter interface to maintain statistics +** about the network traffic per task, on level of thread group +** and individual thread. +** +** General setup +** - +** Once the module is active, it is called for every packet that is +** transmitted by a local process and every packet that is received +** from an interface. Not only the packets that contain the user data +** are passed but also the TCP related protocol packets (SYN, ACK, ...). +** +** When the module discovers a packet for a connection (TCP) or local +** port (UDP) that is new, it creates a sockinfo structure. As soon as +** possible the sockinfo struct will be connected to a taskinfo struct +** that represents the proces or thread that is related to the socket. +** However, the task can only be determined when a packet is transmitted, +** i.e. the module is called during system call handling in the context +** of the transmitting process. At that moment the tgid (process) and +** pid (thread) can be obtained from the process administration to +** be stored in the module's own taskinfo structs (one for the process, +** one for the thread). +** For the time that the sockinfo struct can not be related to a taskinfo +** struct (e.g. when only packets are received), counters are maintained +** temporarily in the sockinfo struct. As soon as a related taskinfo struct +** is discovered when the task transmits, counters will be maintained in +** the taskinfo struct itself. +** When packets are only received for a socket (e.g. another machine is +** sending UDP packets to the local machine) while the local task +** never responds, no match to a process can be made and the packets +** remain unidentified by the netatop module. At least one packet should +** have been sent by a local process to be able to match packets for such +** socket. +** In the file /proc/netatop counters can be found that show the total +** number of packets sent/received and how many of these packets were +** unidentified (i.e. not accounted to a process/thread). +** +** Garbage collection +** -- +** The module uses a garbage collector to cleanup the unused sockinfo +** structs if connections do not exist any more (TCP) or have not been +** used for some time (TCP/UDP). +** Furthermore, the garbage collector checks if the taskinfo structs +** still represent existing processes or threads. If not, the taskinfo struct +** is destroyed (in case of a thread) or it is moved to a separate list of +** finished processes (in case of a process). Analysis programs can read +** the taskinfo of such finished process. When the taskinfo of a finished +** process is not read within 15 seconds, the taskinfo will be destroyed. +** +** A garbage collector cycle can be triggered by issueing a getsockopt +** call from an analysis program (e.g. atop). Apart from that, a time-based +** garbage collector cycle is issued anyhow every 15 seconds by the +** knetatop kernel thread. +** +** Interface with user mode +** +** Programs can open an IP socket and use the getsockopt() system call +** to issue commands to this module. With the command ATOP_GETCNT_TGID +** the current counters can be obtained on process level (thread group) +** and with the command ATOP_GETCNT_PID the counters on thread l
Re: [linux-yocto] [PATCH 1/2] netatop: Initial integration
On 2018年04月10日 03:19, Bruce Ashfield wrote: > On 2018-04-05 3:08 PM, Bruce Ashfield wrote: >> On 2018-03-28 11:25 PM, He Zhe wrote: >>> This is a net statistics collector for atop from: >>> http://atoptool.nl/netatop.php >>> https://atoptool.nl/download/netatop-2.0.tar.gz >>> >>> Two changes: >>> Adjust pathes for netatop.h and netatopversion.h >>> Add module_init and module_exit >> >> Which kernel versions have you tested this on ? Is there a way we could get >> a README that describes how to build and test it ? > > Also, is there a recipe that builds this that I can see/test ? Sorry for slow reply, I'm on vacation and will handle this asap. Is this expected to be built as a module via a recipe? Or as a patch/feature in yocto-kernel-cache? Zhe > > Bruce > >> >> Bruce >> >>> >>> Signed-off-by: He Zhe >>> --- >>> drivers/staging/netatop/netatop.c | 1843 >>> ++ >>> drivers/staging/netatop/netatop.h | 47 + >>> drivers/staging/netatop/netatopversion.h | 2 + >>> 3 files changed, 1892 insertions(+) >>> create mode 100644 drivers/staging/netatop/netatop.c >>> create mode 100644 drivers/staging/netatop/netatop.h >>> create mode 100644 drivers/staging/netatop/netatopversion.h >>> >>> diff --git a/drivers/staging/netatop/netatop.c >>> b/drivers/staging/netatop/netatop.c >>> new file mode 100644 >>> index ..3baa6520416a >>> --- /dev/null >>> +++ b/drivers/staging/netatop/netatop.c >>> @@ -0,0 +1,1843 @@ >>> +/* >>> +** This module uses the netfilter interface to maintain statistics >>> +** about the network traffic per task, on level of thread group >>> +** and individual thread. >>> +** >>> +** General setup >>> +** - >>> +** Once the module is active, it is called for every packet that is >>> +** transmitted by a local process and every packet that is received >>> +** from an interface. Not only the packets that contain the user data >>> +** are passed but also the TCP related protocol packets (SYN, ACK, ...). >>> +** >>> +** When the module discovers a packet for a connection (TCP) or local >>> +** port (UDP) that is new, it creates a sockinfo structure. As soon as >>> +** possible the sockinfo struct will be connected to a taskinfo struct >>> +** that represents the proces or thread that is related to the socket. >>> +** However, the task can only be determined when a packet is transmitted, >>> +** i.e. the module is called during system call handling in the context >>> +** of the transmitting process. At that moment the tgid (process) and >>> +** pid (thread) can be obtained from the process administration to >>> +** be stored in the module's own taskinfo structs (one for the process, >>> +** one for the thread). >>> +** For the time that the sockinfo struct can not be related to a taskinfo >>> +** struct (e.g. when only packets are received), counters are maintained >>> +** temporarily in the sockinfo struct. As soon as a related taskinfo struct >>> +** is discovered when the task transmits, counters will be maintained in >>> +** the taskinfo struct itself. >>> +** When packets are only received for a socket (e.g. another machine is >>> +** sending UDP packets to the local machine) while the local task >>> +** never responds, no match to a process can be made and the packets >>> +** remain unidentified by the netatop module. At least one packet should >>> +** have been sent by a local process to be able to match packets for such >>> +** socket. >>> +** In the file /proc/netatop counters can be found that show the total >>> +** number of packets sent/received and how many of these packets were >>> +** unidentified (i.e. not accounted to a process/thread). >>> +** >>> +** Garbage collection >>> +** -- >>> +** The module uses a garbage collector to cleanup the unused sockinfo >>> +** structs if connections do not exist any more (TCP) or have not been >>> +** used for some time (TCP/UDP). >>> +** Furthermore, the garbage collector checks if the taskinfo structs >>> +** still represent existing processes or threads. If not, the taskinfo >>> struct >>> +** is destroyed (in case of a thread) or it is moved to a separate list of >>> +** finished processes (in case of a process). Analysis programs can read >>> +** the taskinfo of such finished process. When the taskinfo of a finished >>> +** process is not read within 15 seconds, the taskinfo will be destroyed. >>> +** >>> +** A garbage collector cycle can be triggered by issueing a getsockopt >>> +** call from an analysis program (e.g. atop). Apart from that, a time-based >>> +** garbage collector cycle is issued anyhow every 15 seconds by the >>> +** knetatop kernel thread. >>> +** >>> +** Interface with user mode >>> +** >>> +** Programs can open an IP socket and use the getsockopt() system call >>> +** to issue commands to this module. With the command ATOP_GETCNT_TGID >>> +** the current counters can be obtained on process level (t
Re: [linux-yocto] [PATCH 1/2] netatop: Initial integration
On 2018-04-05 3:08 PM, Bruce Ashfield wrote: On 2018-03-28 11:25 PM, He Zhe wrote: This is a net statistics collector for atop from: http://atoptool.nl/netatop.php https://atoptool.nl/download/netatop-2.0.tar.gz Two changes: Adjust pathes for netatop.h and netatopversion.h Add module_init and module_exit Which kernel versions have you tested this on ? Is there a way we could get a README that describes how to build and test it ? Also, is there a recipe that builds this that I can see/test ? Bruce Bruce Signed-off-by: He Zhe --- drivers/staging/netatop/netatop.c | 1843 ++ drivers/staging/netatop/netatop.h | 47 + drivers/staging/netatop/netatopversion.h | 2 + 3 files changed, 1892 insertions(+) create mode 100644 drivers/staging/netatop/netatop.c create mode 100644 drivers/staging/netatop/netatop.h create mode 100644 drivers/staging/netatop/netatopversion.h diff --git a/drivers/staging/netatop/netatop.c b/drivers/staging/netatop/netatop.c new file mode 100644 index ..3baa6520416a --- /dev/null +++ b/drivers/staging/netatop/netatop.c @@ -0,0 +1,1843 @@ +/* +** This module uses the netfilter interface to maintain statistics +** about the network traffic per task, on level of thread group +** and individual thread. +** +** General setup +** - +** Once the module is active, it is called for every packet that is +** transmitted by a local process and every packet that is received +** from an interface. Not only the packets that contain the user data +** are passed but also the TCP related protocol packets (SYN, ACK, ...). +** +** When the module discovers a packet for a connection (TCP) or local +** port (UDP) that is new, it creates a sockinfo structure. As soon as +** possible the sockinfo struct will be connected to a taskinfo struct +** that represents the proces or thread that is related to the socket. +** However, the task can only be determined when a packet is transmitted, +** i.e. the module is called during system call handling in the context +** of the transmitting process. At that moment the tgid (process) and +** pid (thread) can be obtained from the process administration to +** be stored in the module's own taskinfo structs (one for the process, +** one for the thread). +** For the time that the sockinfo struct can not be related to a taskinfo +** struct (e.g. when only packets are received), counters are maintained +** temporarily in the sockinfo struct. As soon as a related taskinfo struct +** is discovered when the task transmits, counters will be maintained in +** the taskinfo struct itself. +** When packets are only received for a socket (e.g. another machine is +** sending UDP packets to the local machine) while the local task +** never responds, no match to a process can be made and the packets +** remain unidentified by the netatop module. At least one packet should +** have been sent by a local process to be able to match packets for such +** socket. +** In the file /proc/netatop counters can be found that show the total +** number of packets sent/received and how many of these packets were +** unidentified (i.e. not accounted to a process/thread). +** +** Garbage collection +** -- +** The module uses a garbage collector to cleanup the unused sockinfo +** structs if connections do not exist any more (TCP) or have not been +** used for some time (TCP/UDP). +** Furthermore, the garbage collector checks if the taskinfo structs +** still represent existing processes or threads. If not, the taskinfo struct +** is destroyed (in case of a thread) or it is moved to a separate list of +** finished processes (in case of a process). Analysis programs can read +** the taskinfo of such finished process. When the taskinfo of a finished +** process is not read within 15 seconds, the taskinfo will be destroyed. +** +** A garbage collector cycle can be triggered by issueing a getsockopt +** call from an analysis program (e.g. atop). Apart from that, a time-based +** garbage collector cycle is issued anyhow every 15 seconds by the +** knetatop kernel thread. +** +** Interface with user mode +** +** Programs can open an IP socket and use the getsockopt() system call +** to issue commands to this module. With the command ATOP_GETCNT_TGID +** the current counters can be obtained on process level (thread group) +** and with the command ATOP_GETCNT_PID the counters on thread level. +** For both commands, the tgid/pid has to be passed of the required thread +** (group). When the required thread (group) does not exist, an errno ESRCH +** is given. +** +** The command ATOP_GETCNT_EXIT can be issued to obtain the counters of +** an exited process. As stated above, such command has to be issued +** within 15 seconds after a process has been declared 'finished' by +** the garbage collector. Whenever this command is issued and no exited +** process is in t
Re: [linux-yocto] [PATCH 1/2] netatop: Initial integration
On 2018-03-28 11:25 PM, He Zhe wrote: This is a net statistics collector for atop from: http://atoptool.nl/netatop.php https://atoptool.nl/download/netatop-2.0.tar.gz Two changes: Adjust pathes for netatop.h and netatopversion.h Add module_init and module_exit Which kernel versions have you tested this on ? Is there a way we could get a README that describes how to build and test it ? Bruce Signed-off-by: He Zhe --- drivers/staging/netatop/netatop.c| 1843 ++ drivers/staging/netatop/netatop.h| 47 + drivers/staging/netatop/netatopversion.h |2 + 3 files changed, 1892 insertions(+) create mode 100644 drivers/staging/netatop/netatop.c create mode 100644 drivers/staging/netatop/netatop.h create mode 100644 drivers/staging/netatop/netatopversion.h diff --git a/drivers/staging/netatop/netatop.c b/drivers/staging/netatop/netatop.c new file mode 100644 index ..3baa6520416a --- /dev/null +++ b/drivers/staging/netatop/netatop.c @@ -0,0 +1,1843 @@ +/* +** This module uses the netfilter interface to maintain statistics +** about the network traffic per task, on level of thread group +** and individual thread. +** +** General setup +** - +** Once the module is active, it is called for every packet that is +** transmitted by a local process and every packet that is received +** from an interface. Not only the packets that contain the user data +** are passed but also the TCP related protocol packets (SYN, ACK, ...). +** +** When the module discovers a packet for a connection (TCP) or local +** port (UDP) that is new, it creates a sockinfo structure. As soon as +** possible the sockinfo struct will be connected to a taskinfo struct +** that represents the proces or thread that is related to the socket. +** However, the task can only be determined when a packet is transmitted, +** i.e. the module is called during system call handling in the context +** of the transmitting process. At that moment the tgid (process) and +** pid (thread) can be obtained from the process administration to +** be stored in the module's own taskinfo structs (one for the process, +** one for the thread). +** For the time that the sockinfo struct can not be related to a taskinfo +** struct (e.g. when only packets are received), counters are maintained +** temporarily in the sockinfo struct. As soon as a related taskinfo struct +** is discovered when the task transmits, counters will be maintained in +** the taskinfo struct itself. +** When packets are only received for a socket (e.g. another machine is +** sending UDP packets to the local machine) while the local task +** never responds, no match to a process can be made and the packets +** remain unidentified by the netatop module. At least one packet should +** have been sent by a local process to be able to match packets for such +** socket. +** In the file /proc/netatop counters can be found that show the total +** number of packets sent/received and how many of these packets were +** unidentified (i.e. not accounted to a process/thread). +** +** Garbage collection +** -- +** The module uses a garbage collector to cleanup the unused sockinfo +** structs if connections do not exist any more (TCP) or have not been +** used for some time (TCP/UDP). +** Furthermore, the garbage collector checks if the taskinfo structs +** still represent existing processes or threads. If not, the taskinfo struct +** is destroyed (in case of a thread) or it is moved to a separate list of +** finished processes (in case of a process). Analysis programs can read +** the taskinfo of such finished process. When the taskinfo of a finished +** process is not read within 15 seconds, the taskinfo will be destroyed. +** +** A garbage collector cycle can be triggered by issueing a getsockopt +** call from an analysis program (e.g. atop). Apart from that, a time-based +** garbage collector cycle is issued anyhow every 15 seconds by the +** knetatop kernel thread. +** +** Interface with user mode +** +** Programs can open an IP socket and use the getsockopt() system call +** to issue commands to this module. With the command ATOP_GETCNT_TGID +** the current counters can be obtained on process level (thread group) +** and with the command ATOP_GETCNT_PID the counters on thread level. +** For both commands, the tgid/pid has to be passed of the required thread +** (group). When the required thread (group) does not exist, an errno ESRCH +** is given. +** +** The command ATOP_GETCNT_EXIT can be issued to obtain the counters of +** an exited process. As stated above, such command has to be issued +** within 15 seconds after a process has been declared 'finished' by +** the garbage collector. Whenever this command is issued and no exited +** process is in the exitlist, the requesting process is blocked until +** an exited process is available. +** +** The command NETATOP_FORCE_GC activates the ga