Re: [systemd-devel] [RFC 0/6] A network proxy management daemon, systemd-proxy-discoveryd
Hi Lucas, Maybe, maybe not. It was not said publicly in that thread, but some systemd guys are nacking this due to the JS engine it would provide. Tom contacted me in private to tell me this is put on hold (though himself, he really want that service to get in, as Lennart and David). I guess the time people get convinced somehow. I'll probably come back to that in the end of September, I am fully busy with other things right now. If there is really nothing to be done on systemd side, then there loss. We will improve PACrunner and that's it. Br, Tomasz Hi Tomasz, On Fri, Apr 10, 2015 at 9:17 AM, Tomasz Bursztyka tomasz.burszt...@linux.intel.com wrote: Hi, As it has been discussed in the systemd hackfest during the Linux Conference Europe, one daemon could centralize the management of all network proxy configurations. Idea is something user can do per-application (like in firefox for instance) or broader (per-DM like in Gnome), user could do it once and for all through such daemon and applications would then request it to know whether or not a proxy has to be used and which one. As a notice, this is nothing new. Such standalone daemon has been already done by the past, pacrunner. systemd-proxy-discoveryd will more or less implement the same ideas with improvements. It will get rid of big JS engines like spidermonkey or v8 which are overkill for the tiny PAC files to be executed on, for instance. From pacrunner experience, APIs will be also improved. This one is using - at least in this RFC - the duktape JS engine to run the PAC files. Note it is not provided in this patchset. Latest version 1.2.x was used. Next features to come are a bit detailed in the TODO (last patch). Tomasz Bursztyka (6): proxy-discoveryd: Basic core added proxy-discoveryd: Add the basics for parsing the default configuration proxy-discoveryd: Add PAC support through duktape js engine proxy-discoveryd: Execute the PAC based proxy in a thread proxy-discoveryd: Add the basic parts for handling DBus methods update TODO What happened to this patch set? Are you going to send a new version? Lucas De Marchi ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] [RFC 0/6] A network proxy management daemon, systemd-proxy-discoveryd
Too used to reply to all. well... Hi Lucas, Maybe, maybe not. It was not said publicly in that thread, but some systemd guys are nacking this due to the JS engine it would provide. Tom contacted me in private to tell me this is put on hold (though himself, he really want that service to get in, as Lennart and David). I guess the time people get convinced somehow. I'll probably come back to that in the end of September, I am fully busy with other things right now. If there is really nothing to be done on systemd side, then there loss. We will improve PACrunner and that's it. Br, Tomasz Hi Tomasz, On Fri, Apr 10, 2015 at 9:17 AM, Tomasz Bursztyka tomasz.burszt...@linux.intel.com wrote: Hi, As it has been discussed in the systemd hackfest during the Linux Conference Europe, one daemon could centralize the management of all network proxy configurations. Idea is something user can do per-application (like in firefox for instance) or broader (per-DM like in Gnome), user could do it once and for all through such daemon and applications would then request it to know whether or not a proxy has to be used and which one. As a notice, this is nothing new. Such standalone daemon has been already done by the past, pacrunner. systemd-proxy-discoveryd will more or less implement the same ideas with improvements. It will get rid of big JS engines like spidermonkey or v8 which are overkill for the tiny PAC files to be executed on, for instance. From pacrunner experience, APIs will be also improved. This one is using - at least in this RFC - the duktape JS engine to run the PAC files. Note it is not provided in this patchset. Latest version 1.2.x was used. Next features to come are a bit detailed in the TODO (last patch). Tomasz Bursztyka (6): proxy-discoveryd: Basic core added proxy-discoveryd: Add the basics for parsing the default configuration proxy-discoveryd: Add PAC support through duktape js engine proxy-discoveryd: Execute the PAC based proxy in a thread proxy-discoveryd: Add the basic parts for handling DBus methods update TODO What happened to this patch set? Are you going to send a new version? Lucas De Marchi ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] [RFC 0/6] A network proxy management daemon, systemd-proxy-discoveryd
Hi Tomasz, On Fri, Apr 10, 2015 at 9:17 AM, Tomasz Bursztyka tomasz.burszt...@linux.intel.com wrote: Hi, As it has been discussed in the systemd hackfest during the Linux Conference Europe, one daemon could centralize the management of all network proxy configurations. Idea is something user can do per-application (like in firefox for instance) or broader (per-DM like in Gnome), user could do it once and for all through such daemon and applications would then request it to know whether or not a proxy has to be used and which one. As a notice, this is nothing new. Such standalone daemon has been already done by the past, pacrunner. systemd-proxy-discoveryd will more or less implement the same ideas with improvements. It will get rid of big JS engines like spidermonkey or v8 which are overkill for the tiny PAC files to be executed on, for instance. From pacrunner experience, APIs will be also improved. This one is using - at least in this RFC - the duktape JS engine to run the PAC files. Note it is not provided in this patchset. Latest version 1.2.x was used. Next features to come are a bit detailed in the TODO (last patch). Tomasz Bursztyka (6): proxy-discoveryd: Basic core added proxy-discoveryd: Add the basics for parsing the default configuration proxy-discoveryd: Add PAC support through duktape js engine proxy-discoveryd: Execute the PAC based proxy in a thread proxy-discoveryd: Add the basic parts for handling DBus methods update TODO What happened to this patch set? Are you going to send a new version? Lucas De Marchi ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] [RFC 0/6] A network proxy management daemon, systemd-proxy-discoveryd
On Fri Apr 10 05:17:37 PDT 2015, Tomasz Bursztyka wrote: As it has been discussed in the systemd hackfest during the Linux Conference Europe, one daemon could centralize the management of all network proxy configurations. Idea is something user can do per-application (like in firefox for instance) or broader (per-DM like in Gnome), user could do it once and for all through such daemon and applications would then request it to know whether or not a proxy has to be used and which one. What overriding reason is there for doing this instead of just using PacRunner? If it's just the JS engine, that's *already* a plugin for PacRunner and it's easy enough to add new options. Hell *I* managed to add v8 support at some point in the dim and distant past, and I don't even admit to knowing any C++. Or JavaScript. You seem to be reimplementing the part of the solution that was already basically *working*, while there are other parts which desperately need to be completed. PacRunner works. There is a libproxy plugin which uses it¹ — and alternatively, PacRunner even provides its own drop-in replacement for libproxy, implementing the nice simple API without the complete horror the libproxy turned in to. So we have the dæmon working, and we have a simple way for client libraries and applications to use it. The *only* thing that has so far held me back from proposing a packaging guideline in my Linux distribution of choice which says applications SHALL use libproxy or query PacRunner by default has been the fact that NetworkManager does not pass on the proxy information to PacRunner, so applications which query it won't yet get the *right* results². If you're going to look at this stuff, I wish you'd fix *that* (for both NetworkManager and systemd-networkd) instead of polishing something that already existed. Then again, as long as it continues to work, I don't really care too much. In some way, having it be part of systemd would at least give more credence to the idea that applications SHOULD be using it by default. -- David WoodhouseOpen Source Technology Centre david.woodho...@intel.com Intel Corporation ¹ https://code.google.com/p/libproxy/issues/detail?id=152² https://bugzilla.gnome.org/show_bug.cgi?id=701824 and https://mail.gnome.org/archives/networkmanager-list/2013-June/msg00093.html smime.p7s Description: S/MIME cryptographic signature ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] [RFC 0/6] A network proxy management daemon, systemd-proxy-discoveryd
Hi David, On Fri Apr 10 05:17:37 PDT 2015, Tomasz Bursztyka wrote: As it has been discussed in the systemd hackfest during the Linux Conference Europe, one daemon could centralize the management of all network proxy configurations. Idea is something user can do per-application (like in firefox for instance) or broader (per-DM like in Gnome), user could do it once and for all through such daemon and applications would then request it to know whether or not a proxy has to be used and which one. What overriding reason is there for doing this instead of just using PacRunner? If it's just the JS engine, that's *already* a plugin for PacRunner and it's easy enough to add new options. Hell *I* managed to add v8 support at some point in the dim and distant past, and I don't even admit to knowing any C++. Or JavaScript. You seem to be reimplementing the part of the solution that was already basically *working*, while there are other parts which desperately need to be completed. PacRunner works. There is a libproxy plugin which uses it¹ — and alternatively, PacRunner even provides its own drop-in replacement for libproxy, implementing the nice simple API without the complete horror the libproxy turned in to. So we have the dæmon working, and we have a simple way for client libraries and applications to use it. The *only* thing that has so far held me back from proposing a packaging guideline in my Linux distribution of choice which says applications SHALL use libproxy or query PacRunner by default has been the fact that NetworkManager does not pass on the proxy information to PacRunner, so applications which query it won't yet get the *right* results². If you're going to look at this stuff, I wish you'd fix *that* (for both NetworkManager and systemd-networkd) instead of polishing something that already existed. Then again, as long as it continues to work, I don't really care too much. In some way, having it be part of systemd would at least give more credence to the idea that applications SHOULD be using it by default. Ok maybe that's was unclear in my mail, but systemd guys want such feature in systemd. I know the work you did to integrate PACrunner properly. And Marcel - who created PACrunner - was attending this hackfest as well. So all parties involved know what are the reasons for that proposal. Tomasz ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] [RFC 0/6] A network proxy management daemon, systemd-proxy-discoveryd
On 11 April 2015 at 13:41, Zbigniew Jędrzejewski-Szmek zbys...@in.waw.pl wrote: On Fri, Apr 10, 2015 at 03:17:37PM +0300, Tomasz Bursztyka wrote: Hi, As it has been discussed in the systemd hackfest during the Linux Conference Europe, one daemon could centralize the management of all network proxy configurations. Idea is something user can do per-application (like in firefox for instance) or broader (per-DM like in Gnome), user could do it once and for all through such daemon and applications would then request it to know whether or not a proxy has to be used and which one. As a notice, this is nothing new. Such standalone daemon has been already done by the past, pacrunner. systemd-proxy-discoveryd will more or less implement the same ideas with improvements. It will get rid of big JS engines like spidermonkey or v8 which are overkill for the tiny PAC files to be executed on, for instance. From pacrunner experience, APIs will be also improved. Hi, the idea of having centralized proxy config is certainly nice. But the PAC files make me shiver. So the first question: is it really necessary to support PAC files? Are they widely used in corporate setting or something? Is there no saner standard? Yes. Yes. No. I only wish for system-wide WPAD support and PAC auto-parsing. The standard started by netscape or some such, hence interpreted javascript, and nobody came up with something more declarative / deterministic that catched on. If the PAC files must be interpreted, I think this is the hardest part. FindProxyForURL is started for every request, potentially hundreds of times per second and more. This means that starting a process per invocation is out of the question, and even starting a thread per invocation seems to be too much. But each call fall into an infinite loop and hang. So the run time of FindProxyForURL should be bounded. FindProxyForURL can also resolve names over the network, which would best be done asynchronously. Well pac file is generally cached, and is static it its contents (possibly many complex clauses if this / if that) but it's ideal to keep it around for subsequent queries. Things in systemd are usually implemented through poll loops, which makes it easy to support thousands of concurrent jobs. I'd think that this would be the best option here too, with a number of workers executing FindProxyForURL()s and stopping when name resolution is requested and continuing when the name is resolved. Zbyszek ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel -- Regards, Dimitri. https://clearlinux.org Open Source Technology Center Intel Corporation (UK) Ltd. - Co. Reg. #1134945 - Pipers Way, Swindon SN3 1RJ. ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] [RFC 0/6] A network proxy management daemon, systemd-proxy-discoveryd
Hi, Have you looked into MuJS instead of duktape? http://mujs.com/ It has a C api similar to Lua, with all state encapsulated in an opaque structure, that you interface with via a virtual stack. It could be easily tested. I did so the PAC related code is contained in a specific place, sharing nothing specific but a somehow generic internal api. I have personally no bold opinion at this time about which JS engine to use. Tomasz ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] [RFC 0/6] A network proxy management daemon, systemd-proxy-discoveryd
Hi Tomasz, Have you looked into MuJS instead of duktape? http://mujs.com/ It has a C api similar to Lua, with all state encapsulated in an opaque structure, that you interface with via a virtual stack. It could be easily tested. I did so the PAC related code is contained in a specific place, sharing nothing specific but a somehow generic internal api. I have personally no bold opinion at this time about which JS engine to use. interesting to test, but it clearly states that its license is AGPL which is clearly more restrictive than the permissive license of duktape. Regards Marcel ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] [RFC 0/6] A network proxy management daemon, systemd-proxy-discoveryd
On Sun, 2015-04-12 at 20:31 +0200, Lennart Poettering wrote: On Fri, 10.04.15 14:05, Dan Williams (d...@redhat.com) wrote: So idea would basically be that we provide in all three daemons calls like: SetAdditionalNTP(ias) SetAdditionalDNS(ia(uay)) I would strongly suggest using strings in the API for IP addresses, not byte arrays. Why so? It's much easier to use for languages other than C, like Python or JS or Ruby or dbus-send or even C in many cases. We originally used binary addresses in the D-Bus interface for NetworkManager, and eventually found that was the wrong choice for these reasons. It's also easier to pull in and out of D-Bus, with dbus-glib or GIO. I'm not sure about sd_bus though. I am very much convinced that passing normalized data through dbus is the right thing. And that implies byte arrays for binary data such as IP addresses... We have structured data in dbus, that's pretty much the biggest benefit of dbus over raw sockets. We should use it and not chicken out, because our bindings suck and encode everything in formatted strings after all... If it's not easy to decode structured data with your bus binding, then the answer is to fix the bus binding, not to just revert to unstructured data. Also, remember that we want to push domains too, for split DNS in the VPN or other cases. So it really should be something like: SetAdditionalDNS(ia(sas)) (index, array[domain, array[server]]) what were the (uay) arguments you proposed? resolved supprots split DNS, but it will not allow multiple DNS servers with different domains on the same interface. Instead, you bind a set of DNS servers and a set of domains to an interface, and that's it, there's no further connection between servers and domains. If I understand correctly, this means you cannot direct *.foobar.com to one set of DNS servers,and *.baz.com to another set, assuming those are bound to the same interface? Why not allow that? Well, resolved is not supposed to support every crazy set up people can think of. I mean, if they want that they can certainly run their own local DNS server. resolved is about figuring out the common usecases and covering them nicely and automatically, and little else. I fail to see the strong usecase for allowing per-server domains. I mean in the VPn case you have one explicit interface for the VPN connection, so this should be covered with the current design. IPSec doesn't give you an kernel netdev at all, it's just routing on an existing interface. So for that interface, you'd have both the VPN servers and upstream servers, all bound to the same interface, and no ability to direct VPN domains to the VPN servers since they are all on the same interface? IPSec used in that mode does not convey DNS server info, does it? In that case it's supposed to be transparent, but with such domain server config it wouldn't be transparent after all... It certainly can convey DNS information. Try the Red Hat VPN servers with libreswan for an example. Your eth0 will now have the original, DHCP/static/etc provided LAN DNS servers, and secondary VPN-provided DNS servers that can be tied to one or more domains that the VPN also provided. Dan ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] [RFC 0/6] A network proxy management daemon, systemd-proxy-discoveryd
On Fri, Apr 10, 2015 at 5:17 AM, Tomasz Bursztyka tomasz.burszt...@linux.intel.com wrote: Hi, [snip] As a notice, this is nothing new. Such standalone daemon has been already done by the past, pacrunner. systemd-proxy-discoveryd will more or less implement the same ideas with improvements. It will get rid of big JS engines like spidermonkey or v8 which are overkill for the tiny PAC files to be executed on, for instance. From pacrunner experience, APIs will be also improved. This one is using - at least in this RFC - the duktape JS engine to run the PAC files. Note it is not provided in this patchset. Latest version 1.2.x was used. It seems that duktape is really not in a suitable shape to be packaged in distributions (https://github.com/svaarala/duktape/issues/94). Do you have any plans to get it into shape? Also, and I am just curious, what is the specific reasoning duktape is preferred? Smaller memory footprint? Thanks, -- Cameron Norman ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] [RFC 0/6] A network proxy management daemon, systemd-proxy-discoveryd
On 13 April 2015 at 07:12, Cameron Norman camerontnor...@gmail.com wrote: On Fri, Apr 10, 2015 at 5:17 AM, Tomasz Bursztyka tomasz.burszt...@linux.intel.com wrote: Hi, [snip] As a notice, this is nothing new. Such standalone daemon has been already done by the past, pacrunner. systemd-proxy-discoveryd will more or less implement the same ideas with improvements. It will get rid of big JS engines like spidermonkey or v8 which are overkill for the tiny PAC files to be executed on, for instance. From pacrunner experience, APIs will be also improved. This one is using - at least in this RFC - the duktape JS engine to run the PAC files. Note it is not provided in this patchset. Latest version 1.2.x was used. It seems that duktape is really not in a suitable shape to be packaged in distributions (https://github.com/svaarala/duktape/issues/94). Do you have any plans to get it into shape? Also, and I am just curious, what is the specific reasoning duktape is preferred? Smaller memory footprint? Thanks, -- Cameron Norman Have you looked into MuJS instead of duktape? http://mujs.com/ It has a C api similar to Lua, with all state encapsulated in an opaque structure, that you interface with via a virtual stack. ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] [RFC 0/6] A network proxy management daemon, systemd-proxy-discoveryd
В Sat, 11 Apr 2015 19:41:15 + Zbigniew Jędrzejewski-Szmek zbys...@in.waw.pl пишет: On Fri, Apr 10, 2015 at 03:17:37PM +0300, Tomasz Bursztyka wrote: Hi, As it has been discussed in the systemd hackfest during the Linux Conference Europe, one daemon could centralize the management of all network proxy configurations. Idea is something user can do per-application (like in firefox for instance) or broader (per-DM like in Gnome), user could do it once and for all through such daemon and applications would then request it to know whether or not a proxy has to be used and which one. As a notice, this is nothing new. Such standalone daemon has been already done by the past, pacrunner. systemd-proxy-discoveryd will more or less implement the same ideas with improvements. It will get rid of big JS engines like spidermonkey or v8 which are overkill for the tiny PAC files to be executed on, for instance. From pacrunner experience, APIs will be also improved. Hi, the idea of having centralized proxy config is certainly nice. But the PAC files make me shiver. So the first question: is it really necessary to support PAC files? Are they widely used in corporate setting or something? I am aware of several large companies that use it, yes. Is there no saner standard? I'm not aware of it. Alternative would be central service that answers the same question. If the PAC files must be interpreted, I think this is the hardest part. FindProxyForURL is started for every request, potentially hundreds of times per second and more. This means that starting a process per invocation is out of the question, and even starting a thread per invocation seems to be too much. But each call fall into an infinite loop and hang. So the run time of FindProxyForURL should be bounded. FindProxyForURL can also resolve names over the network, which would best be done asynchronously. Those PAC files I have seen used relatively dumb URL string comparison, at most doing wildcard domain match. I.e. they do not even attempt to resolve anything but just take literal URL as entered by user. Things in systemd are usually implemented through poll loops, which makes it easy to support thousands of concurrent jobs. I'd think that this would be the best option here too, with a number of workers executing FindProxyForURL()s and stopping when name resolution is requested and continuing when the name is resolved. Zbyszek ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] [RFC 0/6] A network proxy management daemon, systemd-proxy-discoveryd
On Sun, 12.04.15 12:51, Marcel Holtmann (mar...@holtmann.org) wrote: Hi Lennart, As it has been discussed in the systemd hackfest during the Linux Conference Europe, one daemon could centralize the management of all network proxy configurations. Idea is something user can do per-application (like in firefox for instance) or broader (per-DM like in Gnome), user could do it once and for all through such daemon and applications would then request it to know whether or not a proxy has to be used and which one. As a notice, this is nothing new. Such standalone daemon has been already done by the past, pacrunner. systemd-proxy-discoveryd will more or less implement the same ideas with improvements. It will get rid of big JS engines like spidermonkey or v8 which are overkill for the tiny PAC files to be executed on, for instance. From pacrunner experience, APIs will be also improved. Hi, the idea of having centralized proxy config is certainly nice. But the PAC files make me shiver. So the first question: is it really necessary to support PAC files? Yes, it's kinda necessary. PAC is pretty widely used in corporate setting. Windows does the WPAD stuff (the protocol to discover PAD) in all its versions since a long time. In fact, it immediately issues the wpad requests as first thing when you plug in an ethernet cable, in addition to DHCP. Are they widely used in corporate setting or something? Is there no saner standard? Nope, not really, I fear. If the PAC files must be interpreted, I think this is the hardest part. FindProxyForURL is started for every request, potentially hundreds of times per second and more. This means that starting a process per invocation is out of the question, and even starting a thread per invocation seems to be too much. But each call fall into an infinite loop and hang. So the run time of FindProxyForURL should be bounded. FindProxyForURL can also resolve names over the network, which would best be done asynchronously. Things in systemd are usually implemented through poll loops, which makes it easy to support thousands of concurrent jobs. I'd think that this would be the best option here too, with a number of workers executing FindProxyForURL()s and stopping when name resolution is requested and continuing when the name is resolved. Well, it's Java script code. People can just add code like for (;;); , and we must be able to cancel the lookup then safely. I doubt that's cleanly doable with either thread-based or event loop based solutions. I am pretty sure a worker process logic is the way to go. The worker process should get the PAC data when it is forked off, and then read FindProxyForURL data from a pipe and output the result on another, or something similar, and easily sandboxable... are you sure that you are not overthinking this? I think that duktape actually allows just canceling the JS engine execution, no matter what operation it runs. So you could just cancel the JS context. How is this implemented in duktype? I mean, cancelling threads is fricking awful. POSIX thread cancellation is awful, and I am pretty sure NSS calls are incompatible with it anway. Which means you have to check flags after each javascript instruction -- which however doesn't really work too well either, since NSS calls will block as long as they want, hence you couldn't use this to cancel those... I really would prefer doing this out-of-process. Then we can kill the stuff, regardless what it is doing, without interfering with anything else. Heck, we can even let the kernel help us with the timeout thing with RLIMIT_CPU... Lennart -- Lennart Poettering, Red Hat ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] [RFC 0/6] A network proxy management daemon, systemd-proxy-discoveryd
On Sat, 11.04.15 19:41, Zbigniew Jędrzejewski-Szmek (zbys...@in.waw.pl) wrote: On Fri, Apr 10, 2015 at 03:17:37PM +0300, Tomasz Bursztyka wrote: Hi, As it has been discussed in the systemd hackfest during the Linux Conference Europe, one daemon could centralize the management of all network proxy configurations. Idea is something user can do per-application (like in firefox for instance) or broader (per-DM like in Gnome), user could do it once and for all through such daemon and applications would then request it to know whether or not a proxy has to be used and which one. As a notice, this is nothing new. Such standalone daemon has been already done by the past, pacrunner. systemd-proxy-discoveryd will more or less implement the same ideas with improvements. It will get rid of big JS engines like spidermonkey or v8 which are overkill for the tiny PAC files to be executed on, for instance. From pacrunner experience, APIs will be also improved. Hi, the idea of having centralized proxy config is certainly nice. But the PAC files make me shiver. So the first question: is it really necessary to support PAC files? Yes, it's kinda necessary. PAC is pretty widely used in corporate setting. Windows does the WPAD stuff (the protocol to discover PAD) in all its versions since a long time. In fact, it immediately issues the wpad requests as first thing when you plug in an ethernet cable, in addition to DHCP. Are they widely used in corporate setting or something? Is there no saner standard? Nope, not really, I fear. If the PAC files must be interpreted, I think this is the hardest part. FindProxyForURL is started for every request, potentially hundreds of times per second and more. This means that starting a process per invocation is out of the question, and even starting a thread per invocation seems to be too much. But each call fall into an infinite loop and hang. So the run time of FindProxyForURL should be bounded. FindProxyForURL can also resolve names over the network, which would best be done asynchronously. Things in systemd are usually implemented through poll loops, which makes it easy to support thousands of concurrent jobs. I'd think that this would be the best option here too, with a number of workers executing FindProxyForURL()s and stopping when name resolution is requested and continuing when the name is resolved. Well, it's Java script code. People can just add code like for (;;); , and we must be able to cancel the lookup then safely. I doubt that's cleanly doable with either thread-based or event loop based solutions. I am pretty sure a worker process logic is the way to go. The worker process should get the PAC data when it is forked off, and then read FindProxyForURL data from a pipe and output the result on another, or something similar, and easily sandboxable... Lennart -- Lennart Poettering, Red Hat ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] [RFC 0/6] A network proxy management daemon, systemd-proxy-discoveryd
Hi Zbyszek, As it has been discussed in the systemd hackfest during the Linux Conference Europe, one daemon could centralize the management of all network proxy configurations. Idea is something user can do per-application (like in firefox for instance) or broader (per-DM like in Gnome), user could do it once and for all through such daemon and applications would then request it to know whether or not a proxy has to be used and which one. As a notice, this is nothing new. Such standalone daemon has been already done by the past, pacrunner. systemd-proxy-discoveryd will more or less implement the same ideas with improvements. It will get rid of big JS engines like spidermonkey or v8 which are overkill for the tiny PAC files to be executed on, for instance. From pacrunner experience, APIs will be also improved. Hi, the idea of having centralized proxy config is certainly nice. But the PAC files make me shiver. So the first question: is it really necessary to support PAC files? Are they widely used in corporate setting or something? Is there no saner standard? If the PAC files must be interpreted, I think this is the hardest part. FindProxyForURL is started for every request, potentially hundreds of times per second and more. This means that starting a process per invocation is out of the question, and even starting a thread per invocation seems to be too much. But each call fall into an infinite loop and hang. So the run time of FindProxyForURL should be bounded. FindProxyForURL can also resolve names over the network, which would best be done asynchronously. Things in systemd are usually implemented through poll loops, which makes it easy to support thousands of concurrent jobs. I'd think that this would be the best option here too, with a number of workers executing FindProxyForURL()s and stopping when name resolution is requested and continuing when the name is resolved. PACrunner is an existing implementation of this concept. It uses threads and seems to work just fine. We bridged libproxy API compatible library that talks to the PACrunner over D-Bus. I am confused why everybody worries about DNS here. Just use C library name resolving calls. Let it resolve it and be done with it. You are synchronous anyway since the name resolving happens as a Javascript function call. It just happens that this is mapping to actually system code internally. Regards Marcel ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] [RFC 0/6] A network proxy management daemon, systemd-proxy-discoveryd
Hi Lennart, As it has been discussed in the systemd hackfest during the Linux Conference Europe, one daemon could centralize the management of all network proxy configurations. Idea is something user can do per-application (like in firefox for instance) or broader (per-DM like in Gnome), user could do it once and for all through such daemon and applications would then request it to know whether or not a proxy has to be used and which one. As a notice, this is nothing new. Such standalone daemon has been already done by the past, pacrunner. systemd-proxy-discoveryd will more or less implement the same ideas with improvements. It will get rid of big JS engines like spidermonkey or v8 which are overkill for the tiny PAC files to be executed on, for instance. From pacrunner experience, APIs will be also improved. Hi, the idea of having centralized proxy config is certainly nice. But the PAC files make me shiver. So the first question: is it really necessary to support PAC files? Yes, it's kinda necessary. PAC is pretty widely used in corporate setting. Windows does the WPAD stuff (the protocol to discover PAD) in all its versions since a long time. In fact, it immediately issues the wpad requests as first thing when you plug in an ethernet cable, in addition to DHCP. Are they widely used in corporate setting or something? Is there no saner standard? Nope, not really, I fear. If the PAC files must be interpreted, I think this is the hardest part. FindProxyForURL is started for every request, potentially hundreds of times per second and more. This means that starting a process per invocation is out of the question, and even starting a thread per invocation seems to be too much. But each call fall into an infinite loop and hang. So the run time of FindProxyForURL should be bounded. FindProxyForURL can also resolve names over the network, which would best be done asynchronously. Things in systemd are usually implemented through poll loops, which makes it easy to support thousands of concurrent jobs. I'd think that this would be the best option here too, with a number of workers executing FindProxyForURL()s and stopping when name resolution is requested and continuing when the name is resolved. Well, it's Java script code. People can just add code like for (;;); , and we must be able to cancel the lookup then safely. I doubt that's cleanly doable with either thread-based or event loop based solutions. I am pretty sure a worker process logic is the way to go. The worker process should get the PAC data when it is forked off, and then read FindProxyForURL data from a pipe and output the result on another, or something similar, and easily sandboxable... are you sure that you are not overthinking this? I think that duktape actually allows just canceling the JS engine execution, no matter what operation it runs. So you could just cancel the JS context. How is this implemented in duktype? I mean, cancelling threads is fricking awful. POSIX thread cancellation is awful, and I am pretty sure NSS calls are incompatible with it anway. Which means you have to check flags after each javascript instruction -- which however doesn't really work too well either, since NSS calls will block as long as they want, hence you couldn't use this to cancel those... I am not saying that we cancel the thread. I know that this is painful. I am saying that we just cancel the duktape context and its execution, then the thread would just yield all by itself. So I think the question is if a master thread could just tell the duktape context to quit. That is something we might want to figure out. I really would prefer doing this out-of-process. Then we can kill the stuff, regardless what it is doing, without interfering with anything else. Heck, we can even let the kernel help us with the timeout thing with RLIMIT_CPU... We would need to define some IPC and managing the pool of these processes. Of course this is possible since almost any browser does it this way. It however sounds like a lot of effort and complexity if we can do the same with threads and just cancel the duktape execution instead. Regards Marcel ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] [RFC 0/6] A network proxy management daemon, systemd-proxy-discoveryd
On Fri, 10.04.15 14:05, Dan Williams (d...@redhat.com) wrote: So idea would basically be that we provide in all three daemons calls like: SetAdditionalNTP(ias) SetAdditionalDNS(ia(uay)) I would strongly suggest using strings in the API for IP addresses, not byte arrays. Why so? It's much easier to use for languages other than C, like Python or JS or Ruby or dbus-send or even C in many cases. We originally used binary addresses in the D-Bus interface for NetworkManager, and eventually found that was the wrong choice for these reasons. It's also easier to pull in and out of D-Bus, with dbus-glib or GIO. I'm not sure about sd_bus though. I am very much convinced that passing normalized data through dbus is the right thing. And that implies byte arrays for binary data such as IP addresses... We have structured data in dbus, that's pretty much the biggest benefit of dbus over raw sockets. We should use it and not chicken out, because our bindings suck and encode everything in formatted strings after all... If it's not easy to decode structured data with your bus binding, then the answer is to fix the bus binding, not to just revert to unstructured data. Also, remember that we want to push domains too, for split DNS in the VPN or other cases. So it really should be something like: SetAdditionalDNS(ia(sas)) (index, array[domain, array[server]]) what were the (uay) arguments you proposed? resolved supprots split DNS, but it will not allow multiple DNS servers with different domains on the same interface. Instead, you bind a set of DNS servers and a set of domains to an interface, and that's it, there's no further connection between servers and domains. If I understand correctly, this means you cannot direct *.foobar.com to one set of DNS servers,and *.baz.com to another set, assuming those are bound to the same interface? Why not allow that? Well, resolved is not supposed to support every crazy set up people can think of. I mean, if they want that they can certainly run their own local DNS server. resolved is about figuring out the common usecases and covering them nicely and automatically, and little else. I fail to see the strong usecase for allowing per-server domains. I mean in the VPn case you have one explicit interface for the VPN connection, so this should be covered with the current design. IPSec doesn't give you an kernel netdev at all, it's just routing on an existing interface. So for that interface, you'd have both the VPN servers and upstream servers, all bound to the same interface, and no ability to direct VPN domains to the VPN servers since they are all on the same interface? IPSec used in that mode does not convey DNS server info, does it? In that case it's supposed to be transparent, but with such domain server config it wouldn't be transparent after all... Lennart -- Lennart Poettering, Red Hat ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] [RFC 0/6] A network proxy management daemon, systemd-proxy-discoveryd
Hi Lennart, As it has been discussed in the systemd hackfest during the Linux Conference Europe, one daemon could centralize the management of all network proxy configurations. Idea is something user can do per-application (like in firefox for instance) or broader (per-DM like in Gnome), user could do it once and for all through such daemon and applications would then request it to know whether or not a proxy has to be used and which one. As a notice, this is nothing new. Such standalone daemon has been already done by the past, pacrunner. systemd-proxy-discoveryd will more or less implement the same ideas with improvements. It will get rid of big JS engines like spidermonkey or v8 which are overkill for the tiny PAC files to be executed on, for instance. From pacrunner experience, APIs will be also improved. Hi, the idea of having centralized proxy config is certainly nice. But the PAC files make me shiver. So the first question: is it really necessary to support PAC files? Yes, it's kinda necessary. PAC is pretty widely used in corporate setting. Windows does the WPAD stuff (the protocol to discover PAD) in all its versions since a long time. In fact, it immediately issues the wpad requests as first thing when you plug in an ethernet cable, in addition to DHCP. Are they widely used in corporate setting or something? Is there no saner standard? Nope, not really, I fear. If the PAC files must be interpreted, I think this is the hardest part. FindProxyForURL is started for every request, potentially hundreds of times per second and more. This means that starting a process per invocation is out of the question, and even starting a thread per invocation seems to be too much. But each call fall into an infinite loop and hang. So the run time of FindProxyForURL should be bounded. FindProxyForURL can also resolve names over the network, which would best be done asynchronously. Things in systemd are usually implemented through poll loops, which makes it easy to support thousands of concurrent jobs. I'd think that this would be the best option here too, with a number of workers executing FindProxyForURL()s and stopping when name resolution is requested and continuing when the name is resolved. Well, it's Java script code. People can just add code like for (;;); , and we must be able to cancel the lookup then safely. I doubt that's cleanly doable with either thread-based or event loop based solutions. I am pretty sure a worker process logic is the way to go. The worker process should get the PAC data when it is forked off, and then read FindProxyForURL data from a pipe and output the result on another, or something similar, and easily sandboxable... are you sure that you are not overthinking this? I think that duktape actually allows just canceling the JS engine execution, no matter what operation it runs. So you could just cancel the JS context. Regards Marcel ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] [RFC 0/6] A network proxy management daemon, systemd-proxy-discoveryd
On Sun, 12.04.15 12:49, Marcel Holtmann (mar...@holtmann.org) wrote: PACrunner is an existing implementation of this concept. It uses threads and seems to work just fine. We bridged libproxy API compatible library that talks to the PACrunner over D-Bus. How does the abort-after-max-runtime logic work in PACrunner? How do you abort the thread? Do you use POSIX thread cancellation (yuck!), or do you check a cancelation flag after each javascript instruction? or what do you do? Lennart -- Lennart Poettering, Red Hat ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] [RFC 0/6] A network proxy management daemon, systemd-proxy-discoveryd
On Fri, Apr 10, 2015 at 03:17:37PM +0300, Tomasz Bursztyka wrote: Hi, As it has been discussed in the systemd hackfest during the Linux Conference Europe, one daemon could centralize the management of all network proxy configurations. Idea is something user can do per-application (like in firefox for instance) or broader (per-DM like in Gnome), user could do it once and for all through such daemon and applications would then request it to know whether or not a proxy has to be used and which one. As a notice, this is nothing new. Such standalone daemon has been already done by the past, pacrunner. systemd-proxy-discoveryd will more or less implement the same ideas with improvements. It will get rid of big JS engines like spidermonkey or v8 which are overkill for the tiny PAC files to be executed on, for instance. From pacrunner experience, APIs will be also improved. Hi, the idea of having centralized proxy config is certainly nice. But the PAC files make me shiver. So the first question: is it really necessary to support PAC files? Are they widely used in corporate setting or something? Is there no saner standard? If the PAC files must be interpreted, I think this is the hardest part. FindProxyForURL is started for every request, potentially hundreds of times per second and more. This means that starting a process per invocation is out of the question, and even starting a thread per invocation seems to be too much. But each call fall into an infinite loop and hang. So the run time of FindProxyForURL should be bounded. FindProxyForURL can also resolve names over the network, which would best be done asynchronously. Things in systemd are usually implemented through poll loops, which makes it easy to support thousands of concurrent jobs. I'd think that this would be the best option here too, with a number of workers executing FindProxyForURL()s and stopping when name resolution is requested and continuing when the name is resolved. Zbyszek ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
