Re: Clarification on upstart-0.5 and dbus usage
On Thu, Jun 19, 2008 at 8:42 AM, Marcel Holtmann [EMAIL PROTECTED] wrote: Hi Garrett, Marcel, seems to think that just by allowing the implementation of alternatives to D-Bus in Upstart would cause bloating in Upstart. I find that argument without merit. There is a bunch of functions in nih/dbus.c which essentially define the D-Bus communication into Upstart. The rest of upstart don't seem to have much that is really D-Bus specific. So as long we as we continue to keep the D-Bus specific stuff to the nih/dbus.c files i.e. outside of the main upstart code, people could implement alternatives to nih/dbus.c and use the Makefile to decide which one gets compiled in. Why would this bloat Upstart? That won't work. Looks like a good idea, but in the end doesn't work. I have been there and tried it. This is nearly impossible to maintain and create a complex abstraction. It will bloat your code. If you don't believe me, that is perfectly fine. I am not going to stop you from doing it, but I did warn you :) Agreed. Maintaining a second IPC layer would be unwise for continued work. D-Bus isn't going away anytime soon from what I can see and specs can change between versions if we pose a good enough case to the D-Bus folks; we might be able to 'contract out' D-Bus folk's help too if we have a clear set of requirements, pose a good case, and are willing to help contribute (provide funds, manpower, other resources, etc to the project) to motivate our combined goals. One solution that was brought up a while ago though is we could essentially write an dbusd 'replacement' with a limited subset of D-Bus protocol instructions, such that we could maintain Cisco I.P. absolutely while being flexible to include D-Bus in the base system in the future. Maintaining it of course would require more responsibility from our end and would in the end prove to be more involved than just contributing essential components back to the community and working with the community to positively evolve said source code. I think that is a really good idea. Start looking at the D-Bus specification and look at the protocol. If you write your own daemon for handling the D-Bus system bus, you can do it really small in the case you have constraint requirements. So for example: Target a specific OS, Linux in your case, and don't bother with compat or old stuff. Use what the latest Linux kernels allows you to do. If you have a constraint set of users in your system. Maybe root and another one. Then you don't need the complex security handling that the current implementation provides. You can simplify this a lot. If everything runs as root anyway and you don't use SELinux, then you can actually throw it away completely. The bus daemon is a message dispatcher and thus you can do a lot of optimization for the message parsing. For example don't bother with the body, simply hand it over to its destination etc. Only provide a system bus if you don't need the session bus. So that extra code for session handling is not unneeded. Also the service activation part can go away since you have Upstart. And there is more that can be done. My estimate is that in the end you end up with a simple and small dbusd that will follow the full D-Bus specification, but it nicely suited for an embedded Linux system and you will love it. It will talk the same protocol and so Upstart and all the users don't really care. Since I have been through most of these steps already, in some cases you will ask yourself what the f*ck is this. Simple go with it. The D-Bus specification and protocol are stable and the D-Bus community tries to be fully backward compatible. The first issue you will run into is the authentication step between the client and the bus. It is braindead, but simply see it as part of the protocol. Thanks Marcel for the input. I'll sync up with Sarvi and the rest of the team to see if there's any way we can take a subset of the protocol and run with it. We'll be in touch. Cheers, -Garrett -- upstart-devel mailing list upstart-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/upstart-devel
Re: Clarification on upstart-0.5 and dbus usage
Garrett Cooper wrote: Hi again, There have been a flurry of emails passed internal dev lists, and I believe (given past experience with dbus) there is some confusion as to what the role is of dbus in the upstart project. So, I was wondering if someone could briefly discuss from a high-level what dbus is going to do within upstart, or point me to [a series of] link[s] with more information about the architectural interfacing between upstart, dbus, and monitored processes. Thanks, -Garrett DBus is used to give instructions to upstart (start this, stop that) and get information from it (this is now running. This has stopped). The DBus interface can be accessed in one of 2 ways. One is actually running the dbus daemon, and accessing the com.ubuntu.Upstart object on the system bus. The other is by connecting directly to upstart through the /com/ubuntu/Upstart unix-domain socket and communicating in DBus protocol. --CJD -- upstart-devel mailing list upstart-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/upstart-devel
Re: Clarification on upstart-0.5 and dbus usage
On Wed, Jun 18, 2008 at 1:11 PM, Casey Dahlin [EMAIL PROTECTED] wrote: Garrett Cooper wrote: Hi again, There have been a flurry of emails passed internal dev lists, and I believe (given past experience with dbus) there is some confusion as to what the role is of dbus in the upstart project. So, I was wondering if someone could briefly discuss from a high-level what dbus is going to do within upstart, or point me to [a series of] link[s] with more information about the architectural interfacing between upstart, dbus, and monitored processes. Thanks, -Garrett DBus is used to give instructions to upstart (start this, stop that) and get information from it (this is now running. This has stopped). The DBus interface can be accessed in one of 2 ways. One is actually running the dbus daemon, and accessing the com.ubuntu.Upstart object on the system bus. The other is by connecting directly to upstart through the /com/ubuntu/Upstart unix-domain socket and communicating in DBus protocol. --CJD Ok, so there isn't a means to control upstart from a terminal screen, exec, system, or something simple without knowing the API's or talking dbus if need be? I assume that's due (in part) to the chicken-and-egg issue provided by init spawning a terminal. -Garrett -- upstart-devel mailing list upstart-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/upstart-devel
Re: Clarification on upstart-0.5 and dbus usage
Garrett Cooper wrote: On Wed, Jun 18, 2008 at 1:11 PM, Casey Dahlin [EMAIL PROTECTED] wrote: Garrett Cooper wrote: Hi again, There have been a flurry of emails passed internal dev lists, and I believe (given past experience with dbus) there is some confusion as to what the role is of dbus in the upstart project. So, I was wondering if someone could briefly discuss from a high-level what dbus is going to do within upstart, or point me to [a series of] link[s] with more information about the architectural interfacing between upstart, dbus, and monitored processes. Thanks, -Garrett DBus is used to give instructions to upstart (start this, stop that) and get information from it (this is now running. This has stopped). The DBus interface can be accessed in one of 2 ways. One is actually running the dbus daemon, and accessing the com.ubuntu.Upstart object on the system bus. The other is by connecting directly to upstart through the /com/ubuntu/Upstart unix-domain socket and communicating in DBus protocol. --CJD Ok, so there isn't a means to control upstart from a terminal screen, exec, system, or something simple without knowing the API's or talking dbus if need be? I assume that's due (in part) to the chicken-and-egg issue provided by init spawning a terminal. -Garrett There are command line tools that will be provided, but they will be communicating to Upstart via the DBus protocol. You don't need to know the DBus API to use them, by any means, but your system does need to provide the library. --CJD -- upstart-devel mailing list upstart-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/upstart-devel
Re: Clarification on upstart-0.5 and dbus usage
Scott James Remnant wrote: On Wed, 2008-06-18 at 16:11 -0400, Casey Dahlin wrote: The DBus interface can be accessed in one of 2 ways. One is actually running the dbus daemon, and accessing the com.ubuntu.Upstart object on the system bus. Actually, this is the only supported way. While there is another, secret, way -- it's not intended for general use and may change or be taken away without notice. Scott I'm assuming by this you mean over the actual bus? --CJD -- upstart-devel mailing list upstart-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/upstart-devel
Re: Clarification on upstart-0.5 and dbus usage
Hi,
Scott James Remnant:
Actually, this is the only supported way.
In most circumstances (i.e. on anything but limited embedded systems)
this makes sense.
While there is another, secret, way
Well, at minimum you'd need _some_ method to tell Upstart that the dbus
daemon has started up and may now be registered with. :-P
--
Matthias Urlichs | {M:U} IT Design @ m-u-it.de | [EMAIL PROTECTED]
Disclaimer: The quote was selected randomly. Really. | http://smurf.noris.de
- -
Blinding speed can compensate for a lot of deficiencies.
-- David Nichols
--
upstart-devel mailing list
upstart-devel@lists.ubuntu.com
Modify settings or unsubscribe at:
https://lists.ubuntu.com/mailman/listinfo/upstart-devel
Re: Clarification on upstart-0.5 and dbus usage
On Wed, Jun 18, 2008 at 2:29 PM, Scott James Remnant [EMAIL PROTECTED] wrote: On Wed, 2008-06-18 at 22:36 +0200, Matthias Urlichs wrote: While there is another, secret, way Well, at minimum you'd need _some_ method to tell Upstart that the dbus daemon has started up and may now be registered with. :-P Upstart starts the D-Bus daemon, it knows ;) Scott Ok, so the takeaway is: D-Bus provides status info and is method of communication between a given process and upstart. Maybe where my wonderment comes from is will all apps that previously were just able to communicate with sysv now have to explicitly implement callbacks with upstart via C-api's (or D-bus compatible methods) or can other, simpler methods like the sysv / BSD /etc/[init,rc].d/some_script status method be supported too? Or perhaps this point of view is different from what my group is looking for (if a process tanks, restart it), which isn't supported out of the box in the original sysv / BSD style of obtaining status. -Garrett -- just plain confused; needs specs =(... -- upstart-devel mailing list upstart-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/upstart-devel
Re: Clarification on upstart-0.5 and dbus usage
On Wed, 2008-06-18 at 15:06 -0700, Garrett Cooper wrote: Maybe where my wonderment comes from is will all apps that previously were just able to communicate with sysv Which applications were those? ;-) sysvinit never had a documented IPC protocol. Scott -- Have you ever, ever felt like this? Had strange things happen? Are you going round the twist? signature.asc Description: This is a digitally signed message part -- upstart-devel mailing list upstart-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/upstart-devel
Re: Clarification on upstart-0.5 and dbus usage
2008/6/18 Scott James Remnant [EMAIL PROTECTED]: On Wed, 2008-06-18 at 16:11 -0400, Casey Dahlin wrote: The DBus interface can be accessed in one of 2 ways. One is actually running the dbus daemon, and accessing the com.ubuntu.Upstart object on the system bus. Actually, this is the only supported way. While there is another, secret, way -- it's not intended for general use and may change or be taken away without notice. Hm, I'd actually prefer somehow, if core tools, like initctl/runlevel/telinit etc would talk to upstart directly without the need of a running dbus system bus. For 3rd party tools, the obvious way to access upstart should be the system bus. Just my 2ยข, mivhsrl -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? -- upstart-devel mailing list upstart-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/upstart-devel
Re: Clarification on upstart-0.5 and dbus usage
On Wed, Jun 18, 2008 at 3:12 PM, Scott James Remnant [EMAIL PROTECTED] wrote: On Wed, 2008-06-18 at 15:06 -0700, Garrett Cooper wrote: Maybe where my wonderment comes from is will all apps that previously were just able to communicate with sysv Which applications were those? ;-) sysvinit never had a documented IPC protocol. Scott I think I'm just confusing init with the scripts that typically come with it. Ack... -Garrett -- upstart-devel mailing list upstart-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/upstart-devel
Re: Clarification on upstart-0.5 and dbus usage
2008/6/19 Scott James Remnant [EMAIL PROTECTED]: On Thu, 2008-06-19 at 00:10 +0200, Michael Biebl wrote: 2008/6/18 Scott James Remnant [EMAIL PROTECTED]: On Wed, 2008-06-18 at 16:11 -0400, Casey Dahlin wrote: The DBus interface can be accessed in one of 2 ways. One is actually running the dbus daemon, and accessing the com.ubuntu.Upstart object on the system bus. Actually, this is the only supported way. While there is another, secret, way -- it's not intended for general use and may change or be taken away without notice. Hm, I'd actually prefer somehow, if core tools, like initctl/runlevel/telinit etc would talk to upstart directly without the need of a running dbus system bus. Any particular reason? - Someone deletes his dbus job file. - dbus-daemon fails to start (misconfiguration, whatever) - upstart would be usable without the complete dbus package (it would only have to depend on libdbus) It's more of a gut feeling, that relying on the system bus for these core tools, makes upstart more fragile and error prone. Cheers, Michael -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? -- upstart-devel mailing list upstart-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/upstart-devel
Re: Clarification on upstart-0.5 and dbus usage
On Wed, Jun 18, 2008 at 3:19 PM, Michael Biebl [EMAIL PROTECTED] wrote: 2008/6/19 Scott James Remnant [EMAIL PROTECTED]: On Thu, 2008-06-19 at 00:10 +0200, Michael Biebl wrote: 2008/6/18 Scott James Remnant [EMAIL PROTECTED]: On Wed, 2008-06-18 at 16:11 -0400, Casey Dahlin wrote: The DBus interface can be accessed in one of 2 ways. One is actually running the dbus daemon, and accessing the com.ubuntu.Upstart object on the system bus. Actually, this is the only supported way. While there is another, secret, way -- it's not intended for general use and may change or be taken away without notice. Hm, I'd actually prefer somehow, if core tools, like initctl/runlevel/telinit etc would talk to upstart directly without the need of a running dbus system bus. Any particular reason? - Someone deletes his dbus job file. - dbus-daemon fails to start (misconfiguration, whatever) - upstart would be usable without the complete dbus package (it would only have to depend on libdbus) It's more of a gut feeling, that relying on the system bus for these core tools, makes upstart more fragile and error prone. Cheers, Michael I have to agree with Michael. More possible points of human input for upstart during critical stages in system startup just make it more brittle when dealing with confused users or misconfigured systems, input from rogue scripts, etc, esp when dbus is a shared 'resource' amongst different system applications. -Garrett -- upstart-devel mailing list upstart-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/upstart-devel
Re: Clarification on upstart-0.5 and dbus usage
Hi Garrett, Hm, I'd actually prefer somehow, if core tools, like initctl/runlevel/telinit etc would talk to upstart directly without the need of a running dbus system bus. Any particular reason? - Someone deletes his dbus job file. - dbus-daemon fails to start (misconfiguration, whatever) - upstart would be usable without the complete dbus package (it would only have to depend on libdbus) It's more of a gut feeling, that relying on the system bus for these core tools, makes upstart more fragile and error prone. I have to agree with Michael. More possible points of human input for upstart during critical stages in system startup just make it more brittle when dealing with confused users or misconfigured systems, input from rogue scripts, etc, esp when dbus is a shared 'resource' amongst different system applications. believe me that when I mentioned that we are going to fix this, I meant it. I am not going into details here, but expect something soon :) Scott knows about it and so simply stop this discussion and wait for me to actually finish it. Regards Marcel -- upstart-devel mailing list upstart-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/upstart-devel
Re: Clarification on upstart-0.5 and dbus usage
2008/6/19 Marcel Holtmann [EMAIL PROTECTED]: Hi Garrett, Hm, I'd actually prefer somehow, if core tools, like initctl/runlevel/telinit etc would talk to upstart directly without the need of a running dbus system bus. Any particular reason? - Someone deletes his dbus job file. - dbus-daemon fails to start (misconfiguration, whatever) - upstart would be usable without the complete dbus package (it would only have to depend on libdbus) It's more of a gut feeling, that relying on the system bus for these core tools, makes upstart more fragile and error prone. I have to agree with Michael. More possible points of human input for upstart during critical stages in system startup just make it more brittle when dealing with confused users or misconfigured systems, input from rogue scripts, etc, esp when dbus is a shared 'resource' amongst different system applications. believe me that when I mentioned that we are going to fix this, I meant it. I am not going into details here, but expect something soon :) Scott knows about it and so simply stop this discussion and wait for me to actually finish it. Ok then. I was just commenting on the status quo. Will shut up and eagerly await what's coming... Cheers, Michael -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? -- upstart-devel mailing list upstart-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/upstart-devel
Re: Clarification on upstart-0.5 and dbus usage
On Wed, 2008-06-18 at 15:21 -0700, Garrett Cooper wrote: Would the definition of simple case be a straight up SIGCHLD? If the process dies, Upstart will receive SIGCHLD, and if marked respawn it will be restarted. Detecting things like a stoned process is more interesting. You'd want to do something like this on a regular interval: [via D-Bus] obtain job object from Upstart controller (if none, assume job has gone away and exit) [via D-Bus] obtain running instance object (if none, assume job is disabled and exit) [via D-Bus] obtain process id of main job (if you need it for your poking) (if none, assume Upstart is respawning job and sleep) do whatever you need to do to the process to determine whether it is alive or stoned if stoned, you can simply KILL the process knowing that Upstart will respawn it, or [via D-Bus] execute the Restart() method on the instance This is actually where D-Bus really benefits. Upstart's model is much more complex and flexible than sysvinit's, so you need to be able to be quite expressive across IPC. There's already native D-Bus bindings in just about every language you can care to think of, so that's a huge amount of coding you don't need to do. For example, that might look in Python: import dbus bus = dbus.SystemBus() upstart = bus.get_object(com.ubuntu.Upstart, /com/ubuntu/Upstart) try: job = upstart.GetJobByName(my_job) instance = upstart.GetInstanceByName() except: # No such job, or not running return # Note this bit may change try: pid = instance.Get(pid)[main] except KeyError: # Being started, stopped or restarted return # Muck around with pid as you see fit if os.path.isfile(/tmp/%d % pid): return # Oh dear, it's broken instance.Restart() Scott -- Have you ever, ever felt like this? Had strange things happen? Are you going round the twist? signature.asc Description: This is a digitally signed message part -- upstart-devel mailing list upstart-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/upstart-devel
Re: Clarification on upstart-0.5 and dbus usage
Hi,
Scott James Remnant:
pid = instance.Get(pid)[main]
# Muck around with pid as you see fit
if os.path.isfile(/tmp/%d % pid):
return
# Oh dear, it's broken
instance.Restart()
Nice race condition there. :-P
This example just shows that starting jobs cleanly is not quite as
simple as you'd think (well ... isn't that a surprise ;-) which
demonstrates that an expressive IPC protocol is a Good Thing,
ultimately.
--
Matthias Urlichs | {M:U} IT Design @ m-u-it.de | [EMAIL PROTECTED]
Disclaimer: The quote was selected randomly. Really. | http://smurf.noris.de
- -
:locked up: adj. Syn. for {hung}, {wedged}.
--
upstart-devel mailing list
upstart-devel@lists.ubuntu.com
Modify settings or unsubscribe at:
https://lists.ubuntu.com/mailman/listinfo/upstart-devel
RE: Clarification on upstart-0.5 and dbus usage
Hi Sarvi, Its high time we stepped up a layer from plain Unix domain sockets and talked a higher level API. And I agree D-Bus is very likely the equivalent 'Unix Domain Sockets' for this purpose. But unfortunatley its not there yet. All this means is that D-Bus needs some more time to mature as a spec and as an implementation to reach that status. Until then people will want other ways to talk to Upstart, than D-Bus, for various reasons. Code Size, integratiion with existing communication infrastructure, etc. If my team were to comeup with a modular compile-time plugable interface between Upstart and D-Bus, would something like Glib Object system be a good way to do that. If I understand right, D-Bus works Glib Objects and Glib Object system can be made to work with other messaging mechanisms as well. D-Bus is not depending on GLib or GObject. The D-Bus low-level C library has no extra dependencies. The D-Bus daemon also needs an XML library, but that is only for parsing the security files and that could be made optional. So D-Bus as itself doesn't have any dependencies and is already pretty small. The bloats that people associate with D-Bus are mainly rumors :) Regards Marcel -- upstart-devel mailing list upstart-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/upstart-devel
RE: Clarification on upstart-0.5 and dbus usage
Hi Sarvi, But that said, D-Bus is a fine choice for now. I hope though, the Upstart community is open to code contributions from us that allow for modular alternatives to D-Bus. Ofcourse without compromising on performance or clean code. I think that Scott and I explained that this would only increase complexity inside Upstart and that this makes basically no sense. We don't need support for two IPC mechanism. We use D-Bus. And if D-Bus looks like such a problem to you, then even Upstart might not be the right solution for you. Sarvi Are you implying that if my team was willing to put the effort into making Upstart more suitable for Embedded systems as well, that it would be contradictory to the direction/goals of Upstart. Is the intention of Upstart only big heavy Workstations. Note that LaunchD the Apple equivalent of Upstart runs on the iPhone as well, and I don't see why Upstart needs to aim any lower. To tie the destiny of Upstart to D-Bus does not seem to be in the best interest of Upstart. it actually is and I have made that point already. So has Scott. If you wanna put effort into adding an extra IPC layer and thus also an IPC abstraction into Upstart, you gonna bloat Upstart and make it only more complex. This is not helping. It is really not. I had all these discussion before and have heard it all, but start looking into D-Bus and start using it. Don't waste time and resources into adding something to Upstart. Use the manpower to make D-Bus better and more suitable for embedded if that is needed. Personally, I don't see that since systems like Maemo, OpenMoko, Android, ALP and many more are using D-Bus successfully in its current form. For some reason everybody thinks that D-Bus is bloated, but it is not. The low-level part is a well defined communication protocol and a message bus. It is not CORBA or some crazy stuff like DCOP or something. It is small and well suited for embedded. Nobody is stopping you from trying to add another IPC to Upstart, but the success of this idea is really questionable and once you use D-Bus for any second component in your system, this effort becomes useless and it bloats your system. So trust me when we all say that you should accept D-Bus as way of talking to Upstart and roll with it. You will see the benefits really quickly. Regards Marcel -- upstart-devel mailing list upstart-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/upstart-devel
