On Tue, 25 Oct 2016 14:32:29 -0700 Cedric BAIL <cedric.b...@free.fr> said:
> On Mon, Oct 24, 2016 at 10:31 PM, Carsten Haitzler <ras...@rasterman.com> > wrote: > > On Fri, 21 Oct 2016 15:14:20 -0700 Cedric BAIL <cedric.b...@free.fr> said: > > > > i've been mulling this. i followed the feedback so far. basically 2 main > > things: > > > > 1. this is intended to improve performance. especially startup times. > > 2. this is highly problematic in terms of portability and is going to create > > lots of code and build paths. > > It won't create lots of code and build paths. In fact it doesn't even > need two build paths. It just needs to uses all .la files for building > quicklaunch and can be rolled in, in parallel to current system. Just > the link of the quicklaunch binary would be different. Next step would > be to make a build option that setup symlink to that binary. Finally > that option would be the default, likely only afffecting the install > process. I don't think that create a lots of code (There should be no > code change) and build paths change at all. why are you doing a symlink if its JUST a quicklaunch binary change (that statically compiles all of efl into it)? you are trying to replace all of libevas.so.1, libeina.so.1, etc. with symlinks TO this quicklaunch binary ... to retain binary compatibility. you cant have the .so's AND this ql binary both have the same set of symbols. asking for trouble there. but to USe this binaries HAVE to be PIe and this makes for a non-portable pain in the arse of creating apps. which is why i did .so's for quicklaunch before.. because it also works with dll's on windows. PIE does not. not that i could find. > > 3. this makes debugging and other things a nightmare (ps names are all the > > same .. well depending on tool etc.) > > This is exagerated. Again just need to pass the right command line to > ps/top and problem is solved. Can easily be documented and we can also > provide a script that does it until we provide some nicer application > there. i mean simple user debugging. someone runs ps or top and all they see is 20 processes called "quicklaunch". because depending on the ps tool it may use one proc field or another. so a user goes "quicklaunch process is using N% cpu or N % ram". what real process is that? they don't know. because procps tools will not do what you want and thus the average person will become confused and find anything using efl like this to be horribly hard to work with. if top, htop, ps, killall, etc. don't work out of the box, it isby everyone (except yours) definition... a nightmare. i've messed with this before and it does not work across all the tools because of where they source command (and that can change based on config and cmdline options too). > > 4. generally the issue of complexity in our build requires for portability > > etc. > > > > #4 isn't something we can solve right now, but has a better solution if we > > #go > > for a greater internal redesign. but not today. some other day. we have to > > live with what we have right now. > > Hum ? How ? I don't see how we can have good portability without a > complex build system. Would be interested to know more by what you > mean here ! we can isolate the portability layers in a much smaller area and always provide a portable interface wrapped on top of the real os one. currently our portability code is scattered all through efl. it's not in one place. by merging it into one location we'll be able to simplify it too so in the end we just have some "efl port sys" lib that provides an "always works" set of library calls and everything above that is 100% portable and never changes platform to platform. > > i am always for improvements. but this really makes this method a very very > > very hard ask to do this. i've thought about it and i think this is far TOO > > radical a change as it means different execution methods, library > > production, linking and so much more os by os. it's kind of insane though > > theoretically doable on *nix. i've checked and PIE seems to be urtterly > > unportable (windows can't do it based on my reading). the proposal would be > > workable ONLY if we could do it unversally so it wouldn't bitrot and it'd > > work. this is actually why quicklaunch did .so's instead of PIE because > > that WORKS portably.... > > Windows is anyway another platform with a lot of specificity. > Quicklaunch will never work there as there is no equivalent to fork > anyway. Emulating what PIE allow is also mostly impossible as actually it does work. fork works. it's emulated and expensive, bit it WORKS. it means you have an already "partially launched" app around. if we had a fork POOL sitting around "ready to go" and they get ipc'd the command to run... it'd work the same on windows and on *nix. it'd be faster on both because when an app runs, the init/liking phase is already done by a process that is sitting as a hot spare ready to go. so ql dlopen()'s the apps' DLL (as now it has to be a dll and not an exe) and the app is basically a loadable module. and loadable module dlls WORK on windows. the problem is symlinks cannot be done in windows (technically ntfs does support them i believe but we cant assume ntfs etc.), so we have to have a different build path that creates empty .so's that simply have dependency (linking) info so we retain binary compatibility. on *nix we can use symlinks to do that. > initialization of a library is different as what a binary require > (Even with ALSR). Anyway we already have different code path/build > differrence for Windows and Linux and this one is not a new one. As > in, it won't increase the difference much. not different link and install paths though. build paths are simple build a.c or b.c depending on platform and link with x.c, y.c and z.c that dont care. this is a bigger difference and it affects every single lib we install. > As for the change, they will be the same for all Unix. It is a not quite. osx doesnt even use the same names. its dylib, not .so. and i think its libxxx.1.2.3.dylib ... from memory. so no it's not all the same. > compilation option on just the binary plus a static linking of all efl > to it (basically an ifdef with a lot a +=). So I disagree here with > your assesment of the complexity introduced in our build system. The > main change is going to be on install which will be made by doing > symlink to that executable. its the linking of efl libs into a PIE executable which doesn't work on windows. you have to magically figure out a portable way to do that with libtool (good luck) across *nix's. you need to deal with windows too. i don't think having a completely different spread of libs in windows is good at all vs *nix and things will bit-rot. they already do. it's already bad enough. i don't think this should be done unless its done universally everywhere the same exact way. in fact technically ql does not need to be PIE at all. it'd be more efficient to NOT be PIE. normal executable with exported symbols. just how enlightenment works with modules. you now cant link against it though. you need dummy libs with symbols to link against but at runtime don't load them in. this actually would make the _thread vars in efl api calls far more efficient (i've measured it and it basically makes _thread a nop If its in a fixed location executable which is the standard, as opposed to PIC or PIE, but of course we just broke ASLR for all of efl if we do that... :)) > > so here is what i think is sane: > > > > 1. merge libraries aggressively. this reduces linking time BETWEEN efl libs > > and thus an efl lib .so when loaded has the libs that were merged already > > linked together. i propose we try and get down to maybe 3 or 4 .so's this > > will close the gap. > > > > how do we make this portable? well on *nix we can install symlinks from our > > old .so major versions TO the new merged ones. but what on windows? the > > best i can think of for windows is produce "empty" dll's that ONLY contain > > like a dummy function and the correct linking to other efl libs they depend > > on eg if we merged eina, eo, ecore, efl into "efl" then eina depends on > > efl, eo, depend on efl and ecore depends on efl. thus link any one of these > > and you got your symbols from before plus more. this solution ALSO works on > > *nix ... so we could use it as a start with symlinks being an optimization > > of it. yes it means 2 install paths unfortunately. we COULD do it as a > > post-install that "rm's" the .so's we installed and replaces them by > > symlinks which makes this kind of sane to keep around. > > Wouldn't that not work due to DSO ? No idea if that would work with with symlinks - no, because symlinks resolve to the same file and i think DSO checks look for linking to the FILE and yet not using any symbols out of it at all (which is actually not strictly true and we could try turn off DSO checks as it is legal to link to a lib then use dlsym(NULL, ...) to hunt for symbols in it at runtime - DSO checks are just trying to add sanity checks that are generally true but not always). with .so's that have dependencies and link back to one big merged .so - DSO will not like that, but this would be needed for WINDOWS because windows does not have symlinks. (replace .so with .dll on windows). regardless if its a PIE quicklaunch executable OR its just merging eina, ecore, eo, etc. - its the same problem. just merging the fewer libs is doing it on a smaller scale so problems to fix are on a smaller scale too. to DO this merge on WINDOWS we HAVE to produce "empty DLL's" that just have linking info back to a single big DLL (or a few bigg-ish DLL's). otherwise symbols will be missing at runtime. for *nix symlinks will do. but to avoid noty testing the build and install of the empty dll's (or .so's) we should do so on all builds and symlink delete/replace the .so file installed with a synlink on *nix in an install hook after installation. we chmod to add +s in e for binary tools for example post install. this hook/feature does exist. but let's not launch into a single PIE which is we cannot make work across all platforms. we CAN merge libs into fewer real .so's or DLL's and we CAN have this be universal on all platforms. as long as we don't start messing with PIE. if then quicklaunch links in 2 or 3 efl libs rather than 20, it will start much faster, AND regular executables will also start faster as linking between efl libs is reduced AND quicklaunch will be able to be portable as it will run/load a.os/dll not a exeutable binary assuming PIE. > Windows. Also, I am not to sure it would improve things that much as > you still need to load a library, agreed an empty one, but still need > to load one. It will also be a lot more work than my proposed change > as it require to shuffle all the code around and create a set of new > empty libraries. tyhe empty bin wont have symbols to resulv between eachother. eg eo does not have to resole eina symbols. evas does not need to resolve eo and eina symbols, edje does not need to resolve evas, eo and eina symbols as they are already resolved internally. > > 2. move quicklaunch further down to core (efl core) so its not only up in > > elm land. this would make it a core capability. we STILL should > > load .so's/dylibs or dll's if there, and try PIE binaries... but we should > > work here on windows too. make it a core feature low down the stack > > We are already doing PIE binary. As stated above Windows can never > have quicklaunch in any form (No fork, and library vs binary can't be > initialized in the same way). windows has an emulated fork. so it can. see above. as long as it loads dll's AS executables. and thats how QL was built from the beginning to work. > Also what do you mean by further down in the stack ? I like it in > elementary as it enable us to later optimize loading of profile/theme > which wouldn't be possible in the lower level. because this is a low level feature. i'd hook higher level INTO the lower level api (extend it by "call "preload func x" for me in quicklaunch) so higher level code just extends it. the whole infra for quicklaunch is actualyl the same as "single process, multiple instances" where you ipc yourself new cmdline args and new env vars, cwd etc. and the single ap opens a new window (and switches cwd etc.)... - its similar. it's the same blob of context data and likely should use the same core. > > 3. use quicklaunch mode for e. since efl will be there then this will work. > > this does not MEAN we have to use it for all apps nor does it mean we MUST > > use it for e... but this would help > > That is what I want to avoid. Having a random path where we may have > it for some application for some user on some platform. I really would > like this to either be used by a large portion of the user base or not > bother. In my opinion 3. doesn't change at all from current situation > as in dead code lying around. and yet you dismiss this as being impossible on windows, so you want to have a completely different setup on windows vs *nix and that's ok. i'm saying we should do this universally in such a way that it works on ALL platforms and ALL cases. we HAVE to keep supporting normal binaries. it HAS to be done. this isn't a choice. e itself HAS to change to work with quicklaunch because of the way it is built anyway. we CAN do this on windows AND on *nix IF all apps are dll's or .so's. we still need our normal .so libraries there that work for those that do not build their apps as a dll or .so. to encourage this to happen we need to make it EASY and portable. this also needs to be maintainable over the years across platforms and not bitrot because one platform doesn't do it, or bitrot because most things don't use quicklaunch and thus it bitrots. we can't have it in a corner and ignored. but we can't fragment things. right now quicklaunch is bitrotting in a corner being ignored... so... right now we use a traditional launch setup with executables actually running as they always have linking in libs. moving from this to quicklaunch everywhere AND being portable AND maintanable is VERY painful. the easiest path is to start work on merging libs and work this out on a small scale. you'd need the same solutions for a quicklaunch PIE anyway (and i'm against doing this). > Also if we start to enable quicklaunch in enlightenment, due to the > way enlightenment_start monitor enlightenment, this means we have to > seriously change the way enlightenment_start work. So either yes. correct. > enlightenment always rely on quicklaunch or it never rely on it, > because the alternative of it relying maybe on it, is a receipe for > duplicated code that bit rot in my opinion. Enabling quicklaunch in > enlightenment means moving process monitoring to quicklaunch and > adding the ability to start a binary in case of a crash. Btw that > means that we can provide the same backtrace feature we do provide > with enlightenment for all efl application started by quicklaunch. > This would improve debugging users issues quite a lot, something we > can't do currently. this now adds one of the hug pains/badnesses of quicklaunch. if you want to run something it now is not a child process. look at rage with rage_thumb for example. you cant use stdin/out to talk to your child anymore. look at evas generic loaders... etc. this is a fundamental architectural change of process hierarchy and control data flow etc. - do not just jump into this. not without a LOT of details design thoughts on how to solve these things too (fd passing - but then you cant do fd passing on windows. hooray! so proxying?). let's start with things that already have enough work on portability etc. merge efl libs from 20 to about 4 and this should already have an impact. measure the impact? does this solve 70% of the problems? 50%? 20%? until we do it we don't know. (by problems i mean time spend linking and launch time). this we can manage portably without redesigning the parent/child process design that has been standard and assumed for 40 years. and surely dont redesign this overnight. > > these above should get almost all the benefit of the PIE thing with the > > least of the downsides. we still have ps name issues though. :( we need to > > dig more to try our best to make this "ok" but this means we still work > > normally. as long as the "quicklaunch" binary uses very few efl symbols, > > loading efl libs and linking will be a much lower cost as we do less > > intra-lib linking and minimal app <-> lib linking. in fact we could make a > > single libefl.so as a build option if symlinks and dependency .so's as > > above work. this should be almost the same cost as the pie binary proposal > > but is portable. > > I have absolutely no idea what you mean here. The point of quicklaunch > is to load efl and force a resolution of all symbol once (this reduce > future memory usage and runtime cost), you would even start it by > enforcing the linker to resolve all symbol. I am confused ! my point is do this without quicklaunch first. merge libs. see what we gain. the work to merge libs would HAVE to be done for a PIE or whatever solution anyway. it's the same. you just dont add in the complexity of a forking daemon ipc flow control, ps being useless etc. change. you get some of the benefit with some of the work that is a step to a final destination anyway. what if this buys 80% of the gain without creating all the quicklaunch complexity issues for everyone (because we need this for everyone if it is to not bitrot). > -- > Cedric BAIL > > ------------------------------------------------------------------------------ > The Command Line: Reinvented for Modern Developers > Did the resurgence of CLI tooling catch you by surprise? > Reconnect with the command line and become more productive. > Learn the new .NET and ASP.NET CLI. Get your free copy! > http://sdm.link/telerik > _______________________________________________ > enlightenment-devel mailing list > enlightenment-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/enlightenment-devel > -- ------------- Codito, ergo sum - "I code, therefore I am" -------------- The Rasterman (Carsten Haitzler) ras...@rasterman.com ------------------------------------------------------------------------------ The Command Line: Reinvented for Modern Developers Did the resurgence of CLI tooling catch you by surprise? Reconnect with the command line and become more productive. Learn the new .NET and ASP.NET CLI. Get your free copy! http://sdm.link/telerik _______________________________________________ enlightenment-devel mailing list enlightenment-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/enlightenment-devel
