My comments as well. :-)
David Goulet wrote:
Hi everyone,
I'm submitting a RFC about a "new daemon model" for UST.
This is the *first* draft... so a lot of feedback will be appreciated
for improvements and ideas!
English might be bad sometimes, please feel free to submit correction
about that!
Thanks to all
RFC - New processes model for UST
Author: David Goulet <[email protected]>
Acknowledgement for helping creating that model:
- Mathieu Desnoyers
- Yannick Brosseau
Version:
- v0.1: 17/01/2011
Initial proposal
Terminology
-----------------
ustd - Main daemon that act as a trace session registry
I would name this ust-sessiond, mostly because ustd has already been used
and it might be good to not reuse the name.
ust-consumerd - Daemon that reads UST buffers for a speficic application
tracing session - A trace linked to set of specific tracepoints
tracing buffers - Buffers containing tracing data
tracing data - Data created by tracing an application
inprocess library - UST library linked with the application
Model
-----------------
This RFC propose a complete new UST daemon model. This re-engineering was
mostly driven by the needs of better security in terms of access rights,
tracing session and networking such as streaming and remote control over
different traces.
The new model follows the basic principles of having a trace registry
(ustd),
consumers for each tracing buffers (ust-consumerd) and having these
buffers
shared between the application and one consumer (shared memory).
From now on, each component mention above is presented with their
associated
roles and existence meaning.
USTD:
The ustd daemon act as a trace registry i.e by keeping reference to
all active
session and, by active, it means a session in any state other then
destroyed.
Each entity we are keeping track of, here traces, will have a unique
identifier
(ID) assign to it.
An ID SHOULD be a unique hash of the session name, trace path name,
date/time, PID and/or UID
What is the ID used for?
The trace roles of ustd:
Trace interaction - Create, Destroy, Pause, Stop, Start, Set options
Registry - keep track of all tracing session information (basics):
* shared memory location
* UID and GID
* application PID
* trace ID (unique identifier)
* session name
This is a bit of a tricky one... I've been thinking a bit about this too.
If we look at a traceable application we could say that we have a hierachy
like so:
+--------+
| app_1 |
+---+----+
|
+-------------+-------------+
+---+------+ +---+------+
| trace_1 | | trace_2 |
+---+------+ +---+------+
| |
+-------------+ +....
| | |
+---+------+ +---+------+ +---+------+
| buffer_1 | | buffer_2 | | buffer_2 |
+---+------+ +---+------+ +---+------+....
(Hope this picture is legible...)
Now, the the consumer obviosuly has to be aware of the lowest levels of
the hierarchy
but does the session daemon need to be aware? I'm not sure...
Basic access control - allow or deny trace interaction based on
the user
UID/GID.
Consumer creation - spawns ust-consumerd.
Don't agree with this... At boot ust-consumerd should start.
Buffers creation - creates shared memory for the tracing buffers.
In order for ustd to be stateless, it should save all the registry
information
to disk. If ustd is killed or crashes, it will be able to restart and
get the
tracing state back without loosing track of all sessions.
UST-CONSUMERD:
This daemon basically consume the tracing buffers and write that data
to disk
for future analysis using LTTv or/and TMF (Tracing Monitoring
Frameworks). Upon
creation, that daemon UID is set to the application user.
hmhm.... I will start bottom up now as I think people will be reading
linearly.
As I seet it we will always start two daemons at boot, ust-sessiond and
ust-consumerd
ust-consumerd is the default disk consumer daemon, ust-sessiond
coordinates traces.
Sequence of events
Figure 1 ust-sessiond started:
+--------------+
| ust-sessiond |
+---+----------+
Figure 2 ust-consumerd started, ust-consumerd connect to ust-sessiond
and registers
itself as a consumer of type "disk":
+--------------+
| ust-sessiond |
+---+----------+
^
|
register consumer
|
+---+-----------+
| ust-consumerd |
+---+-----------+
Figure 3 application starts, connects to ust-sessiond and registers itself.
+--------------+ +--------+
| ust-sessiond |<--register--| app_1 |
+---+----------+ +---+----+
|
|
|
|
+---+-----------+
| ust-consumerd |
+---+-----------+
Figure 4 ustctl starts a trace, telling the sessiond what
the name of the trace should be, which application pid and what consumer it
wants.
+--------+
| ustctl |
+---+----+
|
Start trace
|
v
+--------------+ +--------+
| ust-sessiond |-------------| app_1 |
+---+----------+ +---+----+
|
|
|
|
+---+-----------+
| ust-consumerd |
+---+-----------+
Figure 5 trace still starting, ust-sessiond creates two socketpairs and
sends one fd
to the consumer and one to the app, connecting them bidirectionally (the
consumer can
send commands to the app across one fd and receive commands on the other
fd.)
+--------+
| ustctl |
+---+----+
|
Start trace
|
v
+--------------+ +--------+
| ust-sessiond |--socket fd->| app_1 |
+---+----------+ +---+----+
|
socket fd
|
v
+---+-----------+
| ust-consumerd |
+---+-----------+
Figure 6 trace still starting, app and consumerd are now bidirectionally
connected.
+--------+
| ustctl |
+---+----+
|
Start trace
|
v
+--------------+ +--------+
| ust-sessiond |-------------| app_1 |
+---+----------+ +---+----+
| ^
| |
| |
| |
+---+-----------+ |
| ust-consumerd +<---------------+
+---+-----------+
Figure 7 trace still starting, app allocates buffers in shared memory
and passes
each to the consumerd. The consumerd now maps each of them in turn and
checks which
cpu the belong to and passes them to the corresponding cpu thread where
they are added
to the epoll set.
+---------------+ +--------+
| ust-consumerd |<-map buffers-| app_1 |
+---------------+ +---+----+
I would draw internal diagrams for ust-consumerd but I'm getting tired
of drawing. :-/
What is different is the following:
1. We have only one instance of a libustconsumer normally. This instance
can handle multiple
traces. We thus scale well.
c = number of cpus
n = number of libustconsumer instances
total thread count = n * c + 1
For a typical 12 cpu system tracing 20 applications to disk we then have
c = 12
n = 1
total thread count = 13
In the current model we have a scalability of
c = number of cpus
ch = number of channels
i = number of libustconsumer instances (one per trace)
total thread count = c * ch * i + i
c = 12
ch = 2
i = 20 (one trace per app assumed)
total thread count = 500
2. There is no need for the sessiond to keep track of indivdual buffers.
3. If we want to implement access control we can do it by passing
credentials across the socket (see man unix). These can then be used
by both the consumerd to open files in the right places with the right
credentials. DBUS does this. We don't want to re-implement DBUS though.
4. ust-sessiond doesn't have to work as a router... Routing is painful
and I don't want to have to implement it. Connecting the consumerd with
the app is much simpler. I thought about this, hard. I even had a prototype
routing messages, and I promise, you don't want to do this.
This is the general idea, a lot of stuff is still loitering in my head.
SHARED MEMORY:
This is the memory area where the tracing buffers of the application
will be
held and given access in write mode for the inprocess library of the
application and in readonly mode for the ust-consumerd.
This memory is ONLY used for the tracing buffers. No communication
between
components is done by that memory.
PREREQUISITES:
The ustd daemon MUST always be running as "root" or an equivalent user
having
the same privilege as root.
This daemon MUST be up and running at all time in order to trace a
tracable
application. (Future works will propose multiple registry coexistence).
A "tracing" group SHOULD be created. Whoever is in that group is ABLE
to access
the tracing data of ANY buffers and is ABLE to operate a tracing
session for
ANY application.
WARNING: This group name might interfer with other linux apps using
the same
group name. Carefull care should be put at install time for that.
The new "lttngtrace" command line tool MUST be use to interact with
the ustd
registry daemon for every trace action needed by the user.
We currently have a library (libustcmd) for this? ustctl wraps
libustcmd. I have
a small plan of renaming the library libustctl because it sounds better.
The library has to stay as it will in not too distant a future be used
to implement
RPC calls to ust using tcf or some such.
Have to get to work now... If someone feels like reading up on
credential passing that
would be great.
/Nils
The next section illustrates different use cases using that new model.
Use Cases
-----------------
Every case considers these :
* user A - UID: A; GID: A, tracing
* user B - UID: B; GID: B, tracing
* user C - UID: C; GID: C
This first scenario shows how a certain user A will start a trace for the
specific already running application app_1:
Single user:
+--------+ +------+ +-------+
| user A |---- new ---->| ustd |<------>| app_1 |
+--------+ +------+ +-------+
| | | write
fork create v
+-----------------+ | | +--------+
| ust-consumerd A |<----+ +--------->| shared |
+-----------------+ | memory |
^ +--------+
| |
+-------------- read -------------+
The user A ask for a tracing session (new) to ustd using the PID of
app_1. At
that point, the buffers (shared memory) that will contains the traced
data is
created by the ustd registry daemon and gives to app_1 the location of
that
memory with write access (write). Important to understand here that
this memory
segment will be set with UID of the application (app_1) and with the
tracing
group GID (tracing).
Then, the ust-consumerd is spawned (fork) having these security steps
taken:
* setuid(<user A>) and seteuid()
* setgid(<tracing>) and setegid()
* attach to shared memory in readonly (shmat)
ust-consumerd A - UID: user A (rw-), GID: tracing (r--)
Note here that the GID is set to "tracing" because user A is in that
group. So,
for example, the user B, who is also in the group tracing, will be
ABLE to read
the tracing data of that application since ust-consumerd has the read
access.
Two users tracing the same application:
+--------+ +------+ +-------+
| user A |-------- new ------->| ustd |<------------>| app_1 |--+
+--------+ ^ +------+ +-------+ |
| | | |
+--------+ | | | |
| user B |------+ fork create | write
+--------+ | | |
| | |
+-----------------+ | | +-------------+ |
| ust-consumerd A |<---+ +-------->| shared mem. |<-+
+-----------------+ | | +-------------+ |
^ | | | |
+-------------O----O---- read ------+ |
| | |
+-----------------+ | | +-------------+ |
| ust-consumerd B |<---+ +-------->| shared mem. |<-+
+-----------------+ +-------------+
^ |
+------------- read ----------------+
This is essentially the same scheme as before but the previous figure
shows the
memory and daemon separation for two different users tracing the same
application (app_1). This applies for two or more users. Again, as
mention
earlier, access rights for ust-consumerd:
ust-consumerd A - UID: user A (rw-), GID: tracing (r--)
ust-consumerd B - UID: user B (rw-), GID: tracing (r--)
The next case shows a single user NOT in the tracing group and shows the
process scheme associate to that particular case which is different then
before since ustd will NOT register that session.
Single user NOT in tracing group:
+--------+ +------+
| user C |---- new ---->| ustd |
+--------+ +------+
| +-------+
spawn +------------------->| app_1 |
| v +-------+
| +-----------------+ |
+-->| ust-consumerd C |-create+ | write
+-----------------+ | |
^ v v
| +--------+
| | shared |
+----- read -----| memory |
+--------+
Access rights will look like this:
ust-consumerd C - UID: user C (rw-), GID: C (r--)
For someone NOT in the tracing group, he CAN NOT use the global
registry for
creating a new session in order to prevent any user of being able to
trace any
application.
So ustd will simply denied lttngtrace the ability to create a session
and thus
lttngtrace will have to spawn and inform ust-consumerd to create the
shared
memory and give it to the application.
_______________________________________________
ltt-dev mailing list
[email protected]
http://lists.casi.polymtl.ca/cgi-bin/mailman/listinfo/ltt-dev
