Re: [systemd-devel] Can apps ship their own copy of libudev?
On sexta-feira, 25 de outubro de 2013 10:20:46, Greg KH wrote: On Fri, Oct 25, 2013 at 08:39:49AM +, Koehne Kai wrote: Finally, just searching for 'libudev.so.0' on the web shows that we're far from the only ones who ran into problems because of this ... Learning that the differences between libudev.so.0 and libudev.so.1 are very minor somewhat helps (we can e.g. dlopen it), Even if it isn't that minor of an issue, the correct thing to do here is to dlopen it, as that removes this type of issue completely. It's what a number of other programs have done for years to deal with core system libraries and is a valid solution for Linux. That's what we'll do. It's just a sub-optimal solution, compared to direct normal linking -- which is something we need to enable for distros to do. The drawbacks of this solution are: - no link-time dependency for packaging tools to detect and create the package dependency without intervention - sub-optimal resolution and calling (hand-written code worse than linker- generated PLT) - two separate codepaths for us to maintain - need to scan for both libudev.so.0 and libudev.so.1 I understand the long-term need to break binary compatibility. It's just annoying when it happens... -- Thiago Macieira - thiago.macieira (AT) intel.com Software Architect - Intel Open Source Technology Center signature.asc Description: This is a digitally signed message part. ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Can apps ship their own copy of libudev?
On sexta-feira, 25 de outubro de 2013 10:36:12, Greg KH wrote: Then, as others have stated, be like a distro and use their tools to build your packages. OBS can build stuff for every major distro except Gentoo, so you should have no problem at all to use that. And that's the real solution you should be using here, as that will resolve issues that you will have if you don't that are bigger than just the minor udev library issue. We'll do that and packagers do help us with final versions. Kai already explained where we have problems and where distro packages aren't enough. -- Thiago Macieira - thiago.macieira (AT) intel.com Software Architect - Intel Open Source Technology Center signature.asc Description: This is a digitally signed message part. ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Can apps ship their own copy of libudev?
On sexta-feira, 25 de outubro de 2013 11:57:25, Jan Engelhardt wrote: The drawbacks of this solution are: - no link-time dependency for packaging tools to detect and create the package dependency without intervention Given most .run-type software bypasses package tools already, there is not much more to lose. Good point. I was thinking of the case of distributions that potentially forgot to enable direct linking and instead fell back to the dynamic loading code. We probably have to make our code smart enough so that this won't happen. - sub-optimal resolution and calling (hand-written code worse than linker- generated PLT) How worse is it, actually? 5 void *h = dlopen(libc.so.6, RTLD_LAZY); (gdb) n 6 int (*scmp)(const char *, const char *) = dlsym(h, strcmp); (gdb) 7 return scmp(argv[0], x.c) == 0; (gdb) p scmp $1 = (int (*)(const char *, const char *)) 0x77953c40 __strcmp_ssse3 So even though I used the, as you call it, suboptimal dlopen mechanism, I do get the SSSE3-optimized variant. Can't be _that_ bad. That's not what I meant. Yes, both variants will call the same function. I meant that the code you wrote above (dlopen-dlsym-call, plus handling dlopen failing) is worse than the equivalent solution generated by the linker and dynamic linker for the case when we are allowed to link directly. -- Thiago Macieira - thiago.macieira (AT) intel.com Software Architect - Intel Open Source Technology Center signature.asc Description: This is a digitally signed message part. ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] [HEADSUP] libsystemd-bus + kdbus plans
On segunda-feira, 25 de março de 2013 13.06.57, Simon McVittie wrote: Please talk to the D-Bus maintainers about any reimplementations or replacements for D-Bus; we are on d...@lists.freedesktop.org. (Cross-posted.) Thanks for the cross-posting, Simon. I have now found the thread on systemd- devel showing the plans for kdbus. I'm wondering only why the plans for kdbus were never posted to the dbus@fd.o mailing list... I'll just attribute that to a minor oops. Anyway, here's the original for those in the D-Bus list and not in the systemd list (trimmed to the relevant parts about D-Bus): http://lists.freedesktop.org/archives/systemd-devel/2013-March/009797.html ==kdbus is a new kernel implementation of D-Bus that Kay and Greg have been working on. The userspace component of it will live in systemd, and libsystemd-bus is the first step towards it. This library will speak both kdbus and traditional dbus (we need this, in order to allow tools like systemctl -H to work, across hosts via a stream connection). Currently, the code I just pushed only does the latter. Moving things onto kdbus/libsystemd-bus is going to be a step-by-step process, happening over the coming year or so. We will try hard to make this transition smooth for distributions. Our rough plan is: a) stabilize what we have now b) port over systemctl and the other client side tools to libsystemd-bus, to see if it all works, and give the API a bit of a real-life check-up. When finished we will drop the dependency on classic libdbus.so, but will still use dbus-daemon. c) Finish up kdbus and port libsystemd-bus over to it. Get kdbus merged. d) Port gdbus + classic libdbus.so to become clients for kdbus, too. e) Flag day, switch over the system bus to kdbus, leave session bus as dbus-daemon. At the same time we will no longer support dbus-daemon for the system. This will add a hard dependency of systemd on a very new kernel version. However, to make this palatable we will try hard to keep kdbus.ko compilable out-of-tree and easily backportable. f) Flag day, switch over the session/user bus to kdbus too. dbus-daemon is not used anymore now. All three client libraries will be able to talk to either the new or the old server, so that apps don't really have to care. At least in the ideal case. We try hard to keep breakage minimal, but we might not be able to avoid it entirely. The userspace part for setting up the busses will live in systemd. kdbus is not going to be available for non-systemd systems unless somebody sits down and writes an entirely independent bus userspace. kdbus is entirely different from AF_BUS. == -- Thiago Macieira - thiago (AT) macieira.info - thiago (AT) kde.org Software Architect - Intel Open Source Technology Center PGP/GPG: 0x6EF45358; fingerprint: E067 918B B660 DBD1 105C 966C 33F5 F005 6EF4 5358 signature.asc Description: This is a digitally signed message part. ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Compatibility between D-Bus and kdbus
to be fixed.
=== KDBUS_CMD_MSG_CANCEL ===
Clarification: the docs say that this call cancels a blocking wait. Does it
mean that this command is to be used from a different thread to cause the
blocking call to return? If it has other purposes, this needs documenting.
=== KDBUS_CMD_NAME_ACQUIRE and RELEASE ===
These calls could allow multiple names to be requested or released in one go.
Low-priority future improvement, I guess.
I also don't see anything specifying how a connection is notified that it got a
name it had queued for. Should it use KDBUS_ITEM_NAME_{ADD,CHANGE} and look
for its own connection ID?
If one requests a name and allows queueing, is there any way to tell from the
return value whether the acquisition happened immediately? Not that important,
I guess, since queueing implies async anyway.
=== kdbus_cmd_match user data ===
One of the problems we have in userspace bindings is to figure out what to do
with a message that was received, especially broadcasts. It would really,
really help us if we could specify some user data in the match rule and have
that be provided by the kernel when the message is received.
A single u64 per match rule would be enough, since we can store a pointer
there (i.e., the cookie that is already there). The received message should
contain a list of user data (cookies) that it matched.
This could be a future improvement, since right now we make do without it.
=== kdbus_policy_access ===
After reading through the description, you can be sure that there will be a
request to add the ability to match a given SMACK label, not just UID or GID.
Hopefully a patch will be sent alongside it, but please make it possible to
pass labels not just IDs.
=== Wildcards ===
Are you sure that * not matching a dot is a good idea? What is the rationale
behind it?
Can I suggest using IMAP wildcard matching instead? * matches anything
including dots, % matches everything except dots.
=== KDBUS_ATTACH_NAMES ===
Documentation for metadata says that userspace must cope with some metadata
not being delivered. Can we at least require that KDBUS_ATTACH_NAMES be
delivered if requested? If the cookie in the match rule isn't provided in the
message reception, having the source's names would help solve the problem of
the signal delivery.
The timestamp should also be mandatory.
=== KDBUS_PAYLOAD_DBUS ===
The docs say it reads DBusDBus, but on little-endian systems it actually
reads suBDsuBD :-)
Not relevant, though.
--
Thiago Macieira - thiago (AT) macieira.info - thiago (AT) kde.org
Software Architect - Intel Open Source Technology Center
PGP/GPG: 0x6EF45358; fingerprint:
E067 918B B660 DBD1 105C 966C 33F5 F005 6EF4 5358
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Compatibility between D-Bus and kdbus
* the interruption if any; with the timestamp, there's always one before the syscall. If this isn't to be changed, please change it at least to be a struct timespec, so it's easier to calculate it from the output of clock_gettime(). Conversion is trivial actually... timespec→nanosecs is easy, granted (just multiplication and addition). Doing the reverse would require divisions and modulus, but I can't think of where it would be necessary. PS: the documentation says that it's on CLOCK_MONOTONIC, but glibc does not define _POSIX_MONOTONIC_CLOCK to be larger than zero. That implies that there are Linux systems where no monotonic clock is present. Either kdbus or glibc needs to be fixed. No, the monotonic clock is *not* optional on Linux. Then glibc should be fixed to have _POSIX_MONOTONIC_CLOCK set to 200809L. That saves us a sysconf() call to verify whether it's present or not. http://osxr.org/glibc/source/sysdeps/unix/sysv/linux/bits/posix_opt.h#0093 http://osxr.org/glibc/source/nptl/sysdeps/unix/sysv/linux/bits/posix_opt.h#0161 if you know someone influential there to make it happen, it would be most welcome. Well, this has been requested before, but this has problems. For starters this would mean that each reciever would have to recieve a different message, which we currently try to avoid, everybdoy gets the same. Also, it means that the kernel would always have to iterate through all rules that are installed, instead of being able to return quickly if the first rule that matches is found (why? because there might be two rules that match the same message). Also not that bloom filters are probabilistic anyway, hence you have to match in userspace anyway, in order not to get false positives. But if you do that you can just build the matching data structure so that you also use it to find the appropriate binding for each message. sd-bus does that actuallly pretty neatly by building a decision tree that solves both problems with the minimal number of checks traversing a decision tree. QtDBus has a hashing table based on the most common rules for dbus1 and then does more precise matching. For kdbus support, I suppose we'll use the hashing table on the definite information and then build the decision tree on the rest of the data. The user cookie would allow us to bypass one or both steps. I understand it would require parsing all rules, but is that so bad? As for having the same message, I was thinking that the cookies should be present for all destinations along with the IDs of those destinations. Except that that might balloon the size of the message (100 services listening for a signal, each matching 4 rules, with 16 bytes per match rule = 6400 bytes of overhead). See below on the names. === Wildcards === Are you sure that * not matching a dot is a good idea? What is the rationale behind it? Hmm, what precisely is this about? wildcards about? Just wondering why the * does not match the dot. I'd assume the more common case is to match a full prefix and that includes match dots. === KDBUS_ATTACH_NAMES === Documentation for metadata says that userspace must cope with some metadata not being delivered. Can we at least require that KDBUS_ATTACH_NAMES be delivered if requested? If the cookie in the match rule isn't provided in the message reception, having the source's names would help solve the problem of the signal delivery. The timestamp should also be mandatory. Yes, they are mandatory. process credentials might be suppressed hwover, for example if they cannot be translated due to namespaces. Thanks. Could you clarify in the docs? -- Thiago Macieira - thiago (AT) macieira.info - thiago (AT) kde.org Software Architect - Intel Open Source Technology Center PGP/GPG: 0x6EF45358; fingerprint: E067 918B B660 DBD1 105C 966C 33F5 F005 6EF4 5358 ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Compatibility between D-Bus and kdbus
. If that's not the case, then the only reason to do this would be to simplify implementations. Also, is there any way to ask an endpoint to stop accepting new connections without tearing down the existing ones? You could just take away the access bits. Thanks. Because I thought that the activator may be one process for all possible services. I'm guessing this is not the way you'd envisioned it. Otherwise, if you have 200 activatable services, there are 200 connections by one or more process. There's no bus daemon to run out of fd's here, but they would count towards the user's system-wide file descriptor limit. Yes, systemd maintains one fd per bus-activatable name, that is correct. And it bumps the NOFILES limits to make sure that works. Correct me if I'm wrong, but doesn't the kernel impose a per-UID limit on the number of FDs open? If so, wouldn't a user with tons of activatable services cause a considerable consumption of this limited resource because systemd-user opened a lot of FDs? The docs say that it only succeeds if there are no more messages, at which point no further messages will be accepted. There doesn't seem to be a way of doing a shutdown()-equivalent: stop reception of new messages but still process the queued ones. What's the precise usecase for this? I've been requested to exit, so I am going to exit now This tells the kernel to stop sending me messages, so I am able to exit. If there are more after this, they'll be queued for the activator again, if there's one, rejected otherwise. Well, but you could just process what you want, and not read from the fd anymore. Then you exit, leaving the messages in the fd unread. The kernel will then activate the process again and pass the new messages to it. I am not really sure what the usecase is for telling the kernel explicitly that you don't want more messages... Right that works for activatable services. For non-activatable services, any unhandled calls will get KDBUS_ITEM_REPLY_DEAD sent back to the sender. The only difference from what I was asking is timing: with a KDBUS_CMD_SHUTDOWN, the error condition would come as soon as the message was sent, as opposed to when the receiver closed the fd. I don't think that is a problem. Then glibc should be fixed to have _POSIX_MONOTONIC_CLOCK set to 200809L. That saves us a sysconf() call to verify whether it's present or not. http://osxr.org/glibc/source/sysdeps/unix/sysv/linux/bits/posix_opt.h#0093 http://osxr.org/glibc/source/nptl/sysdeps/unix/sysv/linux/bits/posix_opt.h #0161 if you know someone influential there to make it happen, it would be most welcome. File a bug to glibc. Right. I was just wondering if you knew someone influential so we can get this simple change in quickly. === Wildcards === Are you sure that * not matching a dot is a good idea? What is the rationale behind it? Hmm, what precisely is this about? wildcards about? Just wondering why the * does not match the dot. I'd assume the more common case is to match a full prefix and that includes match dots. Hmm? * in what precisely? missing the context here... 11.2 Wildcard names Policy holder connections may upload names that contain the wildcard suffix (.*). That way, a policy can be uploaded that is effective for every well-kwown name that extends the provided name by exactly one more level. As I said, I kind of expect that the normal case is that the policy applies to the entire tree, not just one more level. -- Thiago Macieira - thiago (AT) macieira.info - thiago (AT) kde.org Software Architect - Intel Open Source Technology Center PGP/GPG: 0x6EF45358; fingerprint: E067 918B B660 DBD1 105C 966C 33F5 F005 6EF4 5358 ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Compatibility between D-Bus and kdbus
as one full second (less than 5% error). -- Thiago Macieira - thiago (AT) macieira.info - thiago (AT) kde.org Software Architect - Intel Open Source Technology Center PGP/GPG: 0x6EF45358; fingerprint: E067 918B B660 DBD1 105C 966C 33F5 F005 6EF4 5358 ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Compatibility between D-Bus and kdbus
you can fake p2p connections by allocating a bus and only connecting two peers to it (busses are relatively cheap now), but I am not sure why. Because kdbus has a few extra advantages that sockets don't, like the zero- or single-copy from process to process. And because we can: if the necessary delta to support P2P is minimal, just a convention, then it's more practical to keep everything centralised than to have an AF_UNIX socket handler and a kdbus fd parser. Well, but in those cases you can just use the user bus or system bus, there's no need to create a new bus just for the purpose of P2P... That implies changes to the user application. I want to provide a replacement API for DBusServer. Well, I doubt the usecase for direct links. I mean, the reasons for peer-to-peer links I am aware of are: a) performance b) network transparency c) IPC before dbus-daemon is around a) and c) don't apply on kdbus anymore. And kdbus is inherently not a network transport, hence you have to use AF_INET there anyway. Plus d) connection between processes of different privileges (e.g., different UIDs) Why not use the regular system bus for that? Because if both processes are unprivileged, neither can own a name in the system bus nor send messages to each other. Correct me if I'm wrong, but doesn't the kernel impose a per-UID limit on the number of FDs open? To my knowledge only one per-process. There's a system-wide limit, but it's much higher. Ok, so there doesn't seem to be a problem here. For non-activatable services, any unhandled calls will get KDBUS_ITEM_REPLY_DEAD sent back to the sender. The only difference from what I was asking is timing: with a KDBUS_CMD_SHUTDOWN, the error condition would come as soon as the message was sent, as opposed to when the receiver closed the fd. I don't think that is a problem. But does that timing fix anything? I don't see any race that was fixed by such differen behaviour. I don't think that the timing change makes a difference to anyone. 11.2 Wildcard names Policy holder connections may upload names that contain the wildcard suffix (.*). That way, a policy can be uploaded that is effective for every well-kwown name that extends the provided name by exactly one more level. As I said, I kind of expect that the normal case is that the policy applies to the entire tree, not just one more level. Ah, OK, no I get the context! So, dbus1 policy had this concept (see own_prefix in the man page) and we deemed it to actually be a useful one, given that this opens up using names as the synchronization primitive they are. I think the reason to limit this to one label only is to keep a clear distinction between what is namespace and what is instance. And ensuring the instance is just the last label (for whatever you encode into it) and the other labels are namespace... In general it is always easy to be strict initially and then open things up further when there's a strong usecase for it, rather than the other way round. Fair enough. I guess that, if needed, you'll make the full hierarchy wildcard a double asterisk, e.g. org.example.**. -- Thiago Macieira - thiago (AT) macieira.info - thiago (AT) kde.org Software Architect - Intel Open Source Technology Center PGP/GPG: 0x6EF45358; fingerprint: E067 918B B660 DBD1 105C 966C 33F5 F005 6EF4 5358 ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Compatibility between D-Bus and kdbus
On Wednesday 26 November 2014 19:30:16 Lennart Poettering wrote: I must be misunderstanding something. The kernel enforces that each bus name is prefixed with $UID-. This is why the system bus is /sys/fs/kdbus/0-system rather than just /sys/fs/kdbus/system. This makes sure that users cannot play games with other users. Thanks for the clarification. That tidbit is nowhere in the kdbus.txt docs. It should probably be mentioned in the section about creating buses. Also, shouldn't CAP_IPC_OWNER-privileged processes be able to create a bus with an arbitrary name? My argument was for a more generic name, such as something in the org.freedesktop.DBus hierarchy. That way, if someone else wants to provide kdbus-powered buses, they'd implement that interface instead of trying to emulate systemd and playing catch-up and possibly bug compatibility. I am really not keen in defining a new abstract bus interface for two functions of which at least one is very questionnable anyway, and where it's not clear that there will ever be a second implementation of... I understand that reluctance. I mean, the two operations are not even really abstractable since both the update-env stuff and the reload stuff have different semantics on systemd: they are much more comprehensive and apply to all services at once instead of just the bus services. I mean, it sounds OK to me to pass this to systemd for compat reasons in a quirky way. But it if we define an abstract, well-defined, standardized interface for this with inherently unclean semantics this is really nothing I want to do. Well, whatever it is, we need a sane API. I'm willing to standardise on the systemd behaviour if the current one is quirky anyway. If we do that, we'll need a new interface name anyway. Further, if it does acquire the org.freedesktop.DBus name itself, libraries won't have to emulate most of the compat behaviour by intercepting outgoing messages. Intercepting means EVERY message needs to be inspected at one point or another, instead of letting it fallback when the call actually happens. In other words, providing a fallback service optimises for the right use-case. This kind of half/half solution sounds messy to me. Either emulate it or don't. But having some of the calls emulated on the client-side and some elsewhere sounds messy. I don't want to emulate any in the library. Any calls that are made are sent to the service. AddMatch and RemoveMatch would reply with error, but most of the others should just work. But given the other email, I guess I am only looking at my own problem. I didn't know that other applications, using other bindings, had legitimate uses for calling AddMatch directly. Thanks for the clarification. I recommend adding a section on what steps to do to create a bus (including setting up policies, activators), one on creating an endpoint, and then one on how a controlling process should operate to manage the bus/endpoint properly. To create a bus you open the control file, issue an ioctl (MAKE_BUS) on the fd, and the fd is then *converted* into a reference to the bus. You can use it then only for updating policy and for eventually closing it to make the bus go away. To create a bus connection you open the bus file, issue an ioctl (HELLO) on the fd, and the fd is then *converted* into a connection fd. You can then use that for send/recv messages, or acquiring/releasing names. To create a bus EP you open the bus file, issue an ioctl (MAKE_EP) on the fd, and the fd is then *converted* into a reference to the EP. You can use it then only to update the EP's policy, and for eventually closing it to make the EP go away. The systemd PID 1 instance has only two fds open hence: one bus reference, and one bus connection. A sandboxing app executor has one, maybe two fds open: the EP reference, and maybe a bus connection in case it wants to offer some services... Understood. My recommendation for clarification in kdbus.txt remains. Well, but in those cases you can just use the user bus or system bus, there's no need to create a new bus just for the purpose of P2P... That implies changes to the user application. I want to provide a replacement API for DBusServer. So, if this is about compat with old apps then continue to use AF_UNIX for that... No need to push them to something new... Well, one thing I've just realised is that the other app needs to know about kdbus too. So we have a double compat issue. Still, I think that a binding *can* provide a P2P-style bus by abusing endpoints with just a bit of convention rules. -- Thiago Macieira - thiago (AT) macieira.info - thiago (AT) kde.org Software Architect - Intel Open Source Technology Center PGP/GPG: 0x6EF45358; fingerprint: E067 918B B660 DBD1 105C 966C 33F5 F005 6EF4 5358 ___ systemd-devel mailing list systemd
Re: [systemd-devel] Compatibility between D-Bus and kdbus
On Wednesday 26 November 2014 19:44:46 Lennart Poettering wrote: Now, that's the reason why acquirename/releasename need to be implemented client side. With that knowledge we can punch holes in the other calls too. For example, the GetConnectionUnixProcessID() call you brought up: let's now issue a GetConnectionUnixProcessID() async first, immediately followed by an AcquireName() for the same name. On dbus this will tell you the old owner, and then acquire the name for you. On kdbus with an external driver this would be different though. While the GetConnecitonUnixProcessID() is queued and waiting to be processed the AcquireName() might already have been executed (because it needs to be an ioctl as we have shown above), hence you would get your own PID returned, not the old one! hence, this calls really should be sync local ioctl()s, not half-half... Ok, thanks. I understand the issue now. Given that and the fact that there are legitimate uses for AddMatch, I am in agreement that bindings need to provide the interception themselves. -- Thiago Macieira - thiago (AT) macieira.info - thiago (AT) kde.org Software Architect - Intel Open Source Technology Center PGP/GPG: 0x6EF45358; fingerprint: E067 918B B660 DBD1 105C 966C 33F5 F005 6EF4 5358 ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
