Hi David,
I will try to comment as well as I can below...
David Goulet wrote:
Hi everyone,
Introduce a complete new proposal for LTTng and UST tracing
infrastructure and control.
This is the fourth revision (v0.4) of that proposal. This version also
introduce the concept of a global and per-user registry. See the
changelog at the beginning of the file.
Hope I did not forget anything and sorry if I did... the last replies
for the v0.3 were quite enormous.
Comments from Mathieu Desnoyers and Michel Dagenais were added. Nothing
final, we are still in the development process.
Again, questions and comments are more than welcome!
Cheers!
David
RFC - New processes model for UST and LTTng
Author: David Goulet <[email protected]>
Contributors:
* Mathieu Desnoyers <[email protected]>
* Yannick Brosseau <[email protected]>
* Nils Carlson <[email protected]>
* Michel Dagenais <[email protected]>
* Stefan Hajnoczi <[email protected]>
Version:
- v0.1: 17/01/2011
* Initial proposal
- v0.2: 19/01/2011
After multiple reply from all the contributors above, here is
the list
of what has changed:
* Change/Add Terminology elements from the initial model
* New figures for four new scenarios
* Add inprocess library section
* LTTng kernel tracer support proposition
* More details for the Model and Components
* Improve the basic model. Quite different from the last one
- v0.3: 28/01/2011
In response from Michel Dagenais and Nils Carlson comments:
* Add scaling to reasons of this re-engineering
* Purpose of the session ID
* Explain why ltt-sessiond creates the tracing buffers
* ust-consumerd interaction schema
* Clarify inprocess library behavior
- v0.4: 01/02/2011
After Mathieu Desnoyers and Michel Dagenais comments:
* Add section Introduction
* Define the global and per-user ltt-sessiond
* Add details for ltt-sessiond in the inprocess lib section
* Session ID are now UUID
* Add buffer snapshot schema for ust-consumerd
* ltt-sessiond validate inprocess lib version
* ltt-sessiond socket validation by the inprocess lib.
* Add drtrace definition
* Add consumer to the Model section
Terminology
-----------------
ltt-sessiond - Main daemon for trace session registry for UST and LTTng
ust-consumerd - Daemon that consume UST buffers for a specific
application
ltt-consumerd - Daemon that consume LTTng buffers
tracing session - A trace linked to a set of specific tracepoints and
to a set
of tracing buffers
tracing buffers - Buffers containing tracing data
tracing data - Data created by tracing an application
inprocess library - UST library linked with the application
shared memory - system V shared memory
application common named pipe - Global named pipe that triggers
application
registration, on pipe event, to
ltt-sessiond
drtrace - New command line tool for LTTng and UST tracing control
Introduction
-----------------
This RFC propose a brand new UST and LTTng daemon model. This
re-engineering
was mostly driven by the need of:
* Better security in terms of access rights on tracing data
* Manage tracing session
* Scaling in terms of thread/processes needed to perform tracing
* LTTng and UST integration in terms of merging traces and session
control
* Networking such as streaming and remote control over different
traces
The new model follows the basic principles of having a session registry
(ltt-sessiond) and consumers for each tracing session (ust-consumerd and
ltt-consumerd).
With this proposal, LTTng and UST will share the same tracing session, be
managed by the same tool and bring a complete integration between
these two
powerful tools.
NOTE: This proposal does NOT makes UST dependent on LTTng and vice versa.
Model
-----------------
A global and/or per-user registry keeps track of all tracing sessions.
Any user
that wants to manage either a kernel trace using LTTng or an
application trace
with UST must interact with that registry for any possible actions.
We can start here... :-)
What do we need the session daemon for in the case where a single user
has downloaded
and compiled UST in his home directory? What is wrong with the socket
per app
and direct connection approach. Please clarify WHAT exactly we need a
session daemon for.
The model address multiple tracing use cases based on the fact that we
introduce a tracing Unix group (tracing group). Only users in that
group or
root can use the global registry. Other users will create a local
registry
(per-user registry) that will be completely independent from the
global one.
Two cases:
1) Users in the tracing group, it's tracing session can consume
all tracing
buffers from all applications and the kernel.
2) Users NOT in the tracing group, it's tracing session can only
consume
data from its own applications' buffers hence tracing his
applications.
A session stored by the registry consist of:
* Session name (given by the user or automatically assigned)
* List of traces (LTTng or UST)
* Tracepoints/markers associated to a trace of that session
* UUID
* Associated user (UID)
Then, consumers are used to extract data from tracing buffers. These
consumers
are daemon consuming either UST or/and LTTng buffers. For a single
session,
only one UST consumer and one LTTng consumer is necessary. The daemon CAN
handle multiple tracing buffers for network streaming by example or
for quick
snapshot. These consumers are told by the inprocess library or the
kernel to
start getting out data on disk or network.
For the next subsections, every components of this new proposal is
explained
from the global and per-user registry perspective.
LTT-SESSIOND:
The ltt-sessiond daemon acts as a session registry i.e. by keeping
reference to
all active session and, by active, it means a session in any state
other than
destroyed. Each entity we are keeping track of, here session, will have a
universal unique identifier (UUID) assigned to it. The purpose of this
UUID is
to track a session in order to apply any kind of actions (Ex: Attach,
Destroy).
A human readable version SHOULD be consider in order to facilitate the
session
identification when listed by drtrace.
The daemon creates two local Unix sockets (AF_UNIX). The first one is
for what
we call client communication i.e. interaction with drtrace (or any other
compatible tools). That socket is set with the ltt-sessiond
credentials with
read-write mode for both user and group. The second one is a global
socket for
application registration for the UST case (see inprocess lib
subsection below).
This daemon is also responsible for tracing buffers creation. Two main
reasons
motivate this design:
* The ltt-sessiond needs to keep track of all the shared memory
segments in
order to be able to give reference to any other possible consumer.
* For the case of sharing tracing buffers between all userspace
applications, having the registry allocating them will allow that
but, if
the inprocess library was allocating them, we will need to
redesign the
whole model.
Ok, here I once again need a motivation in the text.
App is user A
Session daemon user root with gid trace
consumer has gid trace.
Here is another approach that I'd like you to consider.
app creates buffer with mode 0700.
the session daemon is connected so the app sends a reference to the
session daemon containing the shmid.
The trace daemon changes the mode to 0750 and the gid to the tracing group
and then discards the reference. Hej presto, we don't need to keep a
reference
to the shared memory and the consumers can happily consume.
For all tracing actions either to interact with a session or a
specific trace,
the drtrace client MUST go through ltt-sessiond. The daemon will take
care of
routing the command to the write inprocess library or the kernel.
Global registry:
A global registry SHOULD be started, idealy at boot, with credentials
UID root
and GID of the tracing group. Only user within the tracing group will
be able
to interact with that registry. All applications will try to register
to that
registry using the global socket (second one discuss above).
Per-user registry:
This type of registry address two use cases. The first one is when a
session
creation is requested from drtrace but no global ltt-sessiond exist.
So, a
ltt-sessiond will be spawned in order to manage the tracing of that
user. The
second use case is when a user is not in the tracing group thus he cannot
communication with the global registry.
Ok, these are use-cases that assume a session daemon is a requirement
for tracing
at all, but not really a use-case for a session daemon.
However, care MUST be put in order to manage the socket's daemon. They
are not
global anymore so they should be created in the home directory of the
user
requesting tracing.
In both cases, for global and per-user registry, all applications MUST
try to
register to both ltt-sessiond. (see inprocess library subsection for
details)
The trace roles of ltt-sessiond:
Trace interaction - Create, Destroy, Pause, Stop, Start, Set options
Registry - keep track of trace's information:
* shared memory location (only the keyid)
* application PID (UST)
* type (kernel or UST)
* session name
* UID
Can a non-priviled user trace the kernel?
Also, how aware are multiple session daemons of each other?
Buffers creation - creates shared memory for the tracing buffers.
I really think buffer creation here is not a good idea.
I think you will end up putting state in way too many places and have to
replicate
information hither and thither in a way that will be very painful for you.
UST-CONSUMERD:
Ok, we can start with a little note. ust-consumerd could just as well
have been
called ust-disk-consumer. It instantiates libustconsumer which does the
work.
I would want you to re-write this chapter taking into account multiple
instances
of libustconsumer.
The purpose of this daemon is to consume the UST trace buffers for only a
specific session. The session MAY have several traces for example two
different
applications. The client tool, drtrace has to create the ust-consumerd
if NONE
is available for that session. It is very important to understand that
for a
tracing session, there is only one ust-consumerd for all the traced
applications.
So we are back at one libustconsumer instance per session? Why do we
want so many
processes hanging about?
This daemon basically empty the tracing buffers when asked for and
writes that
data to disk for future analysis using LTTv or/and TMF (Tracing
Monitoring
Frameworks). The inprocess library is the one that tells the
ust-consumerd
daemon that the buffers are ready for consumption.
Here is a flow of action to illustrate the ust-consumerd life span:
1)
+-----------+ ops +--------------+
| drtrace A |<---------->| ltt-sessiond |
+-----------+ +--------------+
drtrace ask for tracing an application using the PID and the session
UUID. The
shared memory reference is given to drtrace and the ust-consumerd
communication
socket if ust-consumerd already exist.
2a) If ust-consumerd EXIST
+-----------+
| drtrace A |
+-----------+
| mem ref.
| +---------------+ read +------------+
+-->| ust-consumerd |--------->| shared mem |
+---------------+ +------------+
In that case, drtrace only ask ust-consumerd to consume the buffers using
the reference it previously got from ltt-sessiond.
2b) If ust-consumerd DOES NOT EXIST
+-----------+ +--------------+
| drtrace A | +---->| ltt-sessiond |
+-----------+ | +--------------+
| ID |
| mem ref. | register
| +---------------+
+-->| ust-consumerd |
+---------------+
drtrace spawns the ust-consumerd for the session using the session
UUID in
order for the daemon to register as a consumer to ltt-sessiond for that
session.
Quick buffer snapshot:
1) Here, drtrace will request a buffer snapshot for an already running
session.
+-----------+ +--------------+
| drtrace A |-------- ops ------->| ltt-sessiond |
+-----------+ +--------------+
| | command
| +-----------------+ +-------+<--+
| | ust-consumerd 1 |<----| app_1 |-+
| +-----------------+ +-------+ | write
| 1 | v
| | +-------------+
| +--- read ----->| shared mem. |
| +-------------+
| ^
| +-----------------+ |
+->| ust-consumerd 2 |----------+
+-----------------+ snapshot
| write
|
+---> disk/network
Hm, here you have the session daemon acting as a command router. I
really don't
recommend that.
I think I have to end here... But there is an old adage from somewhere
that goes
"show us the code". This form of programming on paper and design by
committee
is very painful, takes a lot of time and creates a lot of strife.
I recommend you put down the pen and start coding away instead. Put some
patches
up and we can review them and try them out. I think that you will
realize that
you want to minimize the amount of state that you replicate as well as
the total
amount of information you keep track of. Once you've done that we can start
discussing models again.
Even if we were to rubber stamp a final Daemon model proposal you can
very well
run into design issues along the way that force you to completely change
your
approach.
/Nils
The first ust-consumerd (1) was already consuming buffers for the current
session. So, drtrace ask for a live snapshot. A new ust-consumerd (2) is
spawned, snapshot the buffers using the shared memory reference from
ltt-sessiond, writes date to disk and die after all.
On the security side, the ust-consumerd gets UID/GID from the drtrace
credentials since it was spawned by drtrace and so the files
containing the
tracing data will also be set to UID/GID of the drtrace client. No
setuid or
setgid is used, we only use the credentials of the user.
The roles of ust-consumerd:
Register to ltt-sessiond - Using a session UUID and credentials
(UID/GID)
Consume buffers - Write data to a file descriptor (on disk,
network, ...)
Buffer consumption is triggered by the inprocess library which tells
ust-consumerd when to consume.
LTT-CONSUMERD:
The purpose of this daemon is to consume the LTTng trace buffers for
only a
specific session.
For that kernel consumer, ltt-sessiond will pass different anonymous file
descriptors to the ltt-consumerd using a Unix socket. From these file
desriptors, it will be able to get the data from a special function
export by
the LTTng kernel.
ltt-consumerd will be managed by the exact same way as ust-consumerd.
However,
in order to trace the kernel, you are either root (UID=0) or in the
tracing
group.
The roles of ltt-consumerd:
Register to ltt-sessiond - Using a session UUID and credentials
(UID/GID)
Consume buffers - Write data to a file descriptor (on disk,
network, ...)
Kernel triggers ltt-consumerd for buffer consumption.
UST INPROCESS LIBRARY:
When the application starts, this library will check for the global
named pipe
of ltt-sessiond. If present, it MUST validate that root is the owner.
This
check is very important to prevent ltt-sessiond spoofing. If the pipe
is root,
we are certain that it's the privileged user that operates tracing.
Then, using
it's UID, the application will try to register to the per-user
ltt-sessiond
again verifying before the owner ship of the named pipe that should
match the
UID.
Before registration, the inprocess library MUST validate with the
ltt-sessiond
the library version for compatibility reason. This is mechanism is
useful for
library compatibility but also to see if ltt-sessiond socket is valid
(means
that an actual ltt-sessiond is listening on the other side). Having no
response
for over 10 seconds, the application will cut communication on that
socket and
fallback to the application common named pipe (explain below).
If the socket is valid, it will register as a traceable application
using the
apps credentials and will open a local Unix socket, passed to
ltt-sessiond, in
order to receive an eventual shared memory reference. It will then
wait on it
if any other command are given by the drtrace client. This socket
becomes the
only channel of communication between the registry and the application.
If no ltt-sessiond is present at registration, the application tries
to open
the application common named pipe or create it if it does not exist
and wait on
it (using poll or epoll Linux API). Having any type of event on that
pipe, the
inprocess library will then try to register to the global and per-user
ltt-sessiond. If it fails again, it goes back again to wait on that pipe.
SHARED MEMORY
For UST, this is the memory area where the tracing buffers will be
held and
given access in read-write mode for the inprocess library of the
application.
On the LTTng side (for ltt-consumerd), these buffers are in the kernel
space
and given access by opening a file in the debugfs file system. With an
anonymous file desriptor, this consumer will be able to extract the data.
This memory is ONLY used for the tracing data. No communication between
components is done using that memory.
A shared memory segment for tracing MUST be set with the tracing group
GID for
the UST buffers. This is the job of ltt-sessiond.
PREREQUISITES:
The global ltt-sessiond daemon MUST always be running as "root" or an
equivalent user having the same privilege as root (UID = 0).
The ltt-sessiond daemon SHOULD be up and running at all time in order
to trace
a tracable application.
The new drtrace library API MUST be used to interact with the
ltt-sessiond registry daemon for every trace action needed by the user.
A tracing group MUST be created. Whoever is in that group is able to
access the
tracing data of any buffers and is able to trace any application or
the kernel.
WARNING: The tracing group name COULD interfere with other already
existing
groups. Care should be put at install time for that (from source and
packages)
The next section illustrates different use cases using that new model.
Use Cases
-----------------
Each case considers these :
* user A - UID: A; GID: A, tracing
* user B - UID: B; GID: B, tracing
Scenario 1 - Single user tracing app_1
------
This first scenario shows how user A will start a trace for
application app_1
that is not running.
1) drtrace ask ltt-sessiond for a new session through a Unix socket. If
allowed, ltt-sessiond returns a session UUID to the client.
(Ex: ops --> new session)
+-----------+ ops +--------------+
| drtrace A |<---------->| ltt-sessiond |
+-----------+ +--------------+
2) The app_1 is spawned by drtrace having the user A credentials.
Then, app_1
automatically register to ltt-sessiond has a "tracable apps" through
the global
named pipe of ltt-sessiond using the UID/GID and session UUID.
The shared memory is created with the app_1 UID (rw-) and tracing
group GID
(r--) and a reference is given back to app_1
+-----------+ +--------------+
| drtrace A | | ltt-sessiond |
+-----------+ +--------------+
| ^ |
| +-------+ | | +-------------+
+-->| app_1 |<--------+ +-->| shared mem. |
+-------+ +-------------+
3) app_1 connect to the shared memory and ust-consumerd is spawned
with the
session UUID and drtrace credentials (user A). It then register to
ltt-sessiond
for a valid session to consume using the previous session UUID and
credentials.
+-----------+ +--------------+
| drtrace A | +-->| ltt-sessiond |----------+
+-----------+ | +--------------+ |
| | |
| +---------------+ read | commands
+-->| ust-consumerd |---------+ | and
+---------------+ v | options
^ | +-------------+ |
| v +------>| shared mem. | |
+-------+ | +-------------+ |
| app_1 |-------- |
+-------+ write |
^ |
+---------------------------------------
Scenario 2 - Single user tracing already running app_1
------
1) drtrace ask ltt-sessiond for a new session through a Unix socket.
If allowed
(able to write on socket), ltt-sessiond returns a session UUID to the
client.
+-----------+ ops +--------------+
| drtrace A |<---------->| ltt-sessiond |
+-----------+ +--------------+
^
+-------+ read |
| app_1 |----------+
+-------+
NOTE: At this stage, since app_1 is already running, the registration
of app_1
to ltt-sessiond has already been done. However, the shared memory
segment is
not allocated yet until a trace session is initiated. Having no shared
memory,
the inprocess library of app_1 will wait on the local Unix socket
connected to
ltt-sessiond for the reference.
+-----------+ +--------------+
| drtrace A | | ltt-sessiond |
+-----------+ +--------------+
^ |
+-------+ | | +-------------+
| app_1 |<--------+ +-->| shared mem. |
+-------+ commands +-------------+
| ^
+---------- write ----------+
2) drtrace spawns a ust-consumerd for the session. We get the same
figure as
step 3 in the first scenario.
There is a small difference though. The application MAY NOT be using
the same
credentials as user A (drtrace). However, the shared memory is always
GID of
the tracing group. So, in order for user A to trace app_1, is MUST be
in the
tracing group otherwise, if the application is not set with the user
credentials, user A will not be able to trace app_1
Scenario 3 - Multiple users tracing the same running application
------
1) Session are created for the two users. Using the same exact
mechanism as
before, the shared memory and consumers are created. Two users, two
sessions,
two consumers and two shared memories for the same application.
+-----------+ +--------------+
| drtrace A |-------- ops ------->| ltt-sessiond |
+-----------+ ^ +--------------+
| ^ commands
+-----------+ | +-------+<--+
| drtrace B |------+ +--->| app_1 |------- write -----+
+-----------+ | +-------+ |
| |
+-----------------+ | +-------------+ |
| ust-consumerd A |--O--- read ----->| shared mem. |<-+
+-----------------+ | +-------------+ |
| |
+-----------------+ v +-------------+ |
| ust-consumerd B |--+--- read ----->| shared mem. |<-+
+-----------------+ +-------------+
ust-consumerd A - UID: user A (rw-), GID: tracing (r--)
ust-consumerd B - UID: user B (rw-), GID: tracing (r--)
Scenario 4 - User not in the tracing group
------
For this particular case, it's all goes back to the first scenario.
The user
MUST start the application using his credentials. The session will be
created
by the per-user ltt-sessiond but he will not be able to trace anything
that the
user does not owned.
_______________________________________________
ltt-dev mailing list
[email protected]
http://lists.casi.polymtl.ca/cgi-bin/mailman/listinfo/ltt-dev
