On Tue, Oct 25, 2016 at 5:17 PM, Carsten Haitzler <ras...@rasterman.com> wrote: > On Tue, 25 Oct 2016 14:32:29 -0700 Cedric BAIL <cedric.b...@free.fr> said: >> On Mon, Oct 24, 2016 at 10:31 PM, Carsten Haitzler <ras...@rasterman.com> >> wrote: >> > On Fri, 21 Oct 2016 15:14:20 -0700 Cedric BAIL <cedric.b...@free.fr> said: >> > i've been mulling this. i followed the feedback so far. basically 2 main >> > things: >> > >> > 1. this is intended to improve performance. especially startup times. >> > 2. this is highly problematic in terms of portability and is going to >> > create >> > lots of code and build paths. >> >> It won't create lots of code and build paths. In fact it doesn't even >> need two build paths. It just needs to uses all .la files for building >> quicklaunch and can be rolled in, in parallel to current system. Just >> the link of the quicklaunch binary would be different. Next step would >> be to make a build option that setup symlink to that binary. Finally >> that option would be the default, likely only afffecting the install >> process. I don't think that create a lots of code (There should be no >> code change) and build paths change at all. > > why are you doing a symlink if its JUST a quicklaunch binary change (that > statically compiles all of efl into it)?
As said it is a multiple stage update path. First just a change to quicklaunch, which should work by itself. Then we can switch from real libevas.so.1 to a symlink to that binary. This allow us a progressive try without breaking everyone right away and making it possible for people/packager on other OS to try it first. > you are trying to replace all of libevas.so.1, libeina.so.1, etc. with > symlinks > TO this quicklaunch binary ... to retain binary compatibility. you cant have > the .so's AND this ql binary both have the same set of symbols. asking for > trouble there. It shouldn't be a problem thanks to DSO on most system. System without DSO may be an issue. > but to USe this binaries HAVE to be PIe and this makes for a non-portable pain > in the arse of creating apps. which is why i did .so's for quicklaunch > before.. > because it also works with dll's on windows. PIE does not. not that i could > find. Most system have some form of PIE due to the need to implement ALSR. For windows, let follow up the discussion with Vincent. >> > 3. this makes debugging and other things a nightmare (ps names are all the >> > same .. well depending on tool etc.) >> >> This is exagerated. Again just need to pass the right command line to >> ps/top and problem is solved. Can easily be documented and we can also >> provide a script that does it until we provide some nicer application >> there. > > i mean simple user debugging. someone runs ps or top and all they see is 20 > processes called "quicklaunch". because depending on the ps tool it may use > one > proc field or another. so a user goes "quicklaunch process is using N% cpu or > N > % ram". what real process is that? they don't know. because procps tools will > not do what you want and thus the average person will become confused and find > anything using efl like this to be horribly hard to work with. > > if top, htop, ps, killall, etc. don't work out of the box, it isby everyone > (except yours) definition... a nightmare. i've messed with this before and it > does not work across all the tools because of where they source command (and > that can change based on config and cmdline options too). As said, this is just a matter of alias to be documented. I don't think it is so much of a big deal. >> > 4. generally the issue of complexity in our build requires for portability >> > etc. >> > >> > #4 isn't something we can solve right now, but has a better solution if we >> > #go >> > for a greater internal redesign. but not today. some other day. we have to >> > live with what we have right now. >> >> Hum ? How ? I don't see how we can have good portability without a >> complex build system. Would be interested to know more by what you >> mean here ! > > we can isolate the portability layers in a much smaller area and always > provide > a portable interface wrapped on top of the real os one. currently our > portability code is scattered all through efl. it's not in one place. by > merging it into one location we'll be able to simplify it too so in the end we > just have some "efl port sys" lib that provides an "always works" set of > library calls and everything above that is 100% portable and never changes > platform to platform. We kind of have that and kind of not. The issue is that depending on how far from an Unix system an OS is the deeper the change are. Windows require a base layer like Evil to handle things, most RTOS would also require something like that, while Mac OS X doesn't but will require alternate code for display, input, c&p and drag&drop. Moving everything down to a very low level means that even for Unix we need to abstract all this things into a lower layer. I am not to sure of the benefit and that it would work. Can we make xlib call looks like windows call for c&p ? Basically I think we will just end up moving the code we have into another library without much benefit, but I may missunderstand what you have in mind. >> > i am always for improvements. but this really makes this method a very very >> > very hard ask to do this. i've thought about it and i think this is far TOO >> > radical a change as it means different execution methods, library >> > production, linking and so much more os by os. it's kind of insane though >> > theoretically doable on *nix. i've checked and PIE seems to be urtterly >> > unportable (windows can't do it based on my reading). the proposal would be >> > workable ONLY if we could do it unversally so it wouldn't bitrot and it'd >> > work. this is actually why quicklaunch did .so's instead of PIE because >> > that WORKS portably.... >> >> Windows is anyway another platform with a lot of specificity. >> Quicklaunch will never work there as there is no equivalent to fork >> anyway. Emulating what PIE allow is also mostly impossible as > > actually it does work. fork works. it's emulated and expensive, bit it WORKS. > it means you have an already "partially launched" app around. if we had a fork > POOL sitting around "ready to go" and they get ipc'd the command to run... > it'd > work the same on windows and on *nix. it'd be faster on both because when an > app runs, the init/liking phase is already done by a process that is sitting > as > a hot spare ready to go. Hum, so instead of doing the fork on the spot, we would have to make ql have a pool of waiting instance. Sounds quite doable everywhere. > so ql dlopen()'s the apps' DLL (as now it has to be a dll and not an exe) and > the app is basically a loadable module. and loadable module dlls WORK on > windows. > > the problem is symlinks cannot be done in windows (technically ntfs does > support them i believe but we cant assume ntfs etc.), so we have to have a > different build path that creates empty .so's that simply have dependency > (linking) info so we retain binary compatibility. on *nix we can use symlinks > to do that. Yes. That is still an open problem. And I think it requires two differents solution. <snip> >> As for the change, they will be the same for all Unix. It is a > > not quite. osx doesnt even use the same names. its dylib, not .so. and i think > its libxxx.1.2.3.dylib ... from memory. so no it's not all the same. I am not sure of your point. This is mostly a toolchain and libtool issue that is already dealt with by the build system. It is not something that has to be handle specifically. >> compilation option on just the binary plus a static linking of all efl >> to it (basically an ifdef with a lot a +=). So I disagree here with >> your assesment of the complexity introduced in our build system. The >> main change is going to be on install which will be made by doing >> symlink to that executable. > > its the linking of efl libs into a PIE executable which doesn't work on > windows. you have to magically figure out a portable way to do that with > libtool (good luck) across *nix's. you need to deal with windows too. i don't > think having a completely different spread of libs in windows is good at all > vs > *nix and things will bit-rot. they already do. it's already bad enough. i > don't > think this should be done unless its done universally everywhere the same > exact > way. in fact technically ql does not need to be PIE at all. it'd be more > efficient to NOT be PIE. normal executable with exported symbols. just how > enlightenment works with modules. you now cant link against it though. you > need > dummy libs with symbols to link against but at runtime don't load them in. > this > actually would make the _thread vars in efl api calls far more efficient (i've > measured it and it basically makes _thread a nop If its in a fixed location > executable which is the standard, as opposed to PIC or PIE, but of course we > just broke ASLR for all of efl if we do that... :)) PIE is mandatory on Unix to handle ALSR which is where every one of them is going for security reason. I think that latest Ubuntu force all binary to be compiled as PIE. So Unix support for PIE is quite well there as far as I know since a few year now (At least that's the case for Linux, Mac OS X, FreeBSD, OpenBSD and NetBSD). Basically, PIE is there and it will be used more and more for security reason. We can leverage that without problem I think. Expecting non-PIE support is going to be the exception more than the rules. Windows does also have ALSR and provide the same ability, we need to clear out the initialization of that binary that may be a problem. >> > so here is what i think is sane: >> > >> > 1. merge libraries aggressively. this reduces linking time BETWEEN efl libs >> > and thus an efl lib .so when loaded has the libs that were merged already >> > linked together. i propose we try and get down to maybe 3 or 4 .so's this >> > will close the gap. >> > >> > how do we make this portable? well on *nix we can install symlinks from our >> > old .so major versions TO the new merged ones. but what on windows? the >> > best i can think of for windows is produce "empty" dll's that ONLY contain >> > like a dummy function and the correct linking to other efl libs they depend >> > on eg if we merged eina, eo, ecore, efl into "efl" then eina depends on >> > efl, eo, depend on efl and ecore depends on efl. thus link any one of these >> > and you got your symbols from before plus more. this solution ALSO works on >> > *nix ... so we could use it as a start with symlinks being an optimization >> > of it. yes it means 2 install paths unfortunately. we COULD do it as a >> > post-install that "rm's" the .so's we installed and replaces them by >> > symlinks which makes this kind of sane to keep around. >> >> Wouldn't that not work due to DSO ? No idea if that would work with > > with symlinks - no, because symlinks resolve to the same file and i think DSO > checks look for linking to the FILE and yet not using any symbols out of it at > all (which is actually not strictly true and we could try turn off DSO checks > as it is legal to link to a lib then use dlsym(NULL, ...) to hunt for symbols > in it at runtime - DSO checks are just trying to add sanity checks that are > generally true but not always). with .so's that have dependencies and link > back > to one big merged .so - DSO will not like that, but this would be needed for > WINDOWS because windows does not have symlinks. (replace .so with .dll on > windows). Yes, I think that with symlink, we do not have any DSO issue, but without we have. Which is my point here. DSO is usually enforced by the packager and we have no control over that. We can't rely on bypassing it in my opinion, which is why I have dismissed the empty libraries so far. This does apply to both proposal of merging some efl libraries together or merging all of them in ql. > regardless if its a PIE quicklaunch executable OR its just merging eina, > ecore, > eo, etc. - its the same problem. just merging the fewer libs is doing it on a > smaller scale so problems to fix are on a smaller scale too. Well, the scale of the problem doesn't matter. If we have no solution on Unix due to DSO and on Windows due to the absence of symlink, we have to have two differents solution for both system. This is something that indeed apply on both proposal. > to DO this merge on WINDOWS we HAVE to produce "empty DLL's" that just have > linking info back to a single big DLL (or a few bigg-ish DLL's). otherwise > symbols will be missing at runtime. > > for *nix symlinks will do. but to avoid noty testing the build and install of > the empty dll's (or .so's) we should do so on all builds and symlink > delete/replace the .so file installed with a synlink on *nix in an install > hook > after installation. we chmod to add +s in e for binary tools for example post > install. this hook/feature does exist. So you do propose that we have empty library on Windows and symlink on Unix, right ? > but let's not launch into a single PIE which is we cannot make work across all > platforms. we CAN merge libs into fewer real .so's or DLL's and we CAN have > this be universal on all platforms. as long as we don't start messing with > PIE. Why ? We already have two differents solutions with two differents code path which are unrelated to using PIE/ALSR or not ! PIE/ALSR are completely orthogonal to the above problem of symlink vs empty dll. > if then quicklaunch links in 2 or 3 efl libs rather than 20, it will start > much > faster, AND regular executables will also start faster as linking between efl > libs is reduced AND quicklaunch will be able to be portable as it will > run/load > a.os/dll not a exeutable binary assuming PIE. > >> Windows. Also, I am not to sure it would improve things that much as >> you still need to load a library, agreed an empty one, but still need >> to load one. It will also be a lot more work than my proposed change >> as it require to shuffle all the code around and create a set of new >> empty libraries. > > tyhe empty bin wont have symbols to resulv between eachother. eg eo does not > have to resole eina symbols. evas does not need to resolve eo and eina > symbols, > edje does not need to resolve evas, eo and eina symbols as they are already > resolved internally. IO on Windows are pretty bad as far as I know, so just dlopen a file and parsing it is going to be costly, not much we can do there I think. Which is why I said it wouldn't change much in my opinion. <snip> >> Also what do you mean by further down in the stack ? I like it in >> elementary as it enable us to later optimize loading of profile/theme >> which wouldn't be possible in the lower level. > > because this is a low level feature. i'd hook higher level INTO the lower > level > api (extend it by "call "preload func x" for me in quicklaunch) so higher > level > code just extends it. the whole infra for quicklaunch is actualyl the same as > "single process, multiple instances" where you ipc yourself new cmdline args > and new env vars, cwd etc. and the single ap opens a new window (and switches > cwd etc.)... - its similar. it's the same blob of context data and likely > should use the same core. Still, I don't see the point of moving it down the stack. All application/platform that are going to use it will require the initialisation of up to elementary, ideally including preloading configuration/theme and even pre-connecting to the display manager should be doable, so what is the benefit of moving it down the stack adding infrastructure for elementary to always init itself ? >> > 3. use quicklaunch mode for e. since efl will be there then this will work. >> > this does not MEAN we have to use it for all apps nor does it mean we MUST >> > use it for e... but this would help >> >> That is what I want to avoid. Having a random path where we may have >> it for some application for some user on some platform. I really would >> like this to either be used by a large portion of the user base or not >> bother. In my opinion 3. doesn't change at all from current situation >> as in dead code lying around. > > and yet you dismiss this as being impossible on windows, so you want to have a > completely different setup on windows vs *nix and that's ok. i'm saying we > should do this universally in such a way that it works on ALL platforms and > ALL > cases. we HAVE to keep supporting normal binaries. it HAS to be done. this > isn't a choice. e itself HAS to change to work with quicklaunch because of the > way it is built anyway. Enlightenment is not working on Windows. Why would you want to complexify the code of Enlightenment to allow people to turn on/off quicklaunch while you argue so much that for the rest of EFL it is a problem to have different code path. I am very confused here. It is possible that for all the platform where we use Enlightenment to have only one code path, why would you introduce complexity there when there is no reason for ? EFL has to deal with OS that are tricky, like Windows or Mac OS X, but Enlightenment hasn't. I don't see a point of adding option there while arguing there shouldn't be in EFL. > we CAN do this on windows AND on *nix IF all apps are dll's or .so's. we still > need our normal .so libraries there that work for those that do not build > their > apps as a dll or .so. to encourage this to happen we need to make it EASY and > portable. this also needs to be maintainable over the years across platforms > and not bitrot because one platform doesn't do it, or bitrot because most > things don't use quicklaunch and thus it bitrots. we can't have it in a corner > and ignored. but we can't fragment things. right now quicklaunch is bitrotting > in a corner being ignored... so... right now we use a traditional launch setup > with executables actually running as they always have linking in libs. moving > from this to quicklaunch everywhere AND being portable AND maintanable is VERY > painful. > > the easiest path is to start work on merging libs and work this out on a small > scale. you'd need the same solutions for a quicklaunch PIE anyway (and i'm > against doing this). As said before, the issue of portability is not PIE, it is symlink/DSO. Merging library doesn't solve that bit. PIE enable us to work around the issue, as we can even continue to ship the normal library for some time and introduce a ql.pc or efl-2.pc that will link to the quicklaunch binary on all platform avoinding the problem with symlink/PSO all completely if we can not figure it out. As said before PIE allow us a gradual roll out of the feature. <snip> >> enlightenment always rely on quicklaunch or it never rely on it, >> because the alternative of it relying maybe on it, is a receipe for >> duplicated code that bit rot in my opinion. Enabling quicklaunch in >> enlightenment means moving process monitoring to quicklaunch and >> adding the ability to start a binary in case of a crash. Btw that >> means that we can provide the same backtrace feature we do provide >> with enlightenment for all efl application started by quicklaunch. >> This would improve debugging users issues quite a lot, something we >> can't do currently. > > this now adds one of the hug pains/badnesses of quicklaunch. if you want to > run > something it now is not a child process. look at rage with rage_thumb for > example. you cant use stdin/out to talk to your child anymore. look at evas > generic loaders... etc. No. rage_thumb is started directly by rage. If rage doesn't enforce the call to quicklaunch and I don't see why it would, it will not start rage_thumb as a quicklaunch binary. The application binary is unchanged. It is a PIE binary that start normally without ql if you execute it directly. This is not going to affect rage_thumb, evas generic loaders or anything else that is just started with system or ecore_exe. It has to effectively ask ql to launch that program. That's why I said that we can easily have an optout in Enlightenment. <snip> >> > these above should get almost all the benefit of the PIE thing with the >> > least of the downsides. we still have ps name issues though. :( we need to >> > dig more to try our best to make this "ok" but this means we still work >> > normally. as long as the "quicklaunch" binary uses very few efl symbols, >> > loading efl libs and linking will be a much lower cost as we do less >> > intra-lib linking and minimal app <-> lib linking. in fact we could make a >> > single libefl.so as a build option if symlinks and dependency .so's as >> > above work. this should be almost the same cost as the pie binary proposal >> > but is portable. >> >> I have absolutely no idea what you mean here. The point of quicklaunch >> is to load efl and force a resolution of all symbol once (this reduce >> future memory usage and runtime cost), you would even start it by >> enforcing the linker to resolve all symbol. I am confused ! > > my point is do this without quicklaunch first. merge libs. see what we gain. > the work to merge libs would HAVE to be done for a PIE or whatever solution > anyway. it's the same. you just dont add in the complexity of a forking daemon > ipc flow control, ps being useless etc. change. you get some of the benefit > with some of the work that is a step to a final destination anyway. what if > this buys 80% of the gain without creating all the quicklaunch complexity > issues for everyone (because we need this for everyone if it is to not > bitrot). This doesn't match what you were talking about just before. You were saying just above that ql should use very few efl symbol and I don't see why, quite the opposite. The point of ql is also to do all the symbol resolution once. This means that the dirty page related to relocation is shared among all executable started by ql. This save times and memory and one of the reason you want to use "all" efl symbol. I just wanted clarification on the above point. -- Cedric BAIL ------------------------------------------------------------------------------ The Command Line: Reinvented for Modern Developers Did the resurgence of CLI tooling catch you by surprise? Reconnect with the command line and become more productive. Learn the new .NET and ASP.NET CLI. Get your free copy! http://sdm.link/telerik _______________________________________________ enlightenment-devel mailing list enlightenment-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/enlightenment-devel
