On Wed, 26 Oct 2016 11:28:30 -0700 Cedric BAIL <cedric.b...@free.fr> said:
> On Tue, Oct 25, 2016 at 5:17 PM, Carsten Haitzler <ras...@rasterman.com> > wrote: > > > > why are you doing a symlink if its JUST a quicklaunch binary change (that > > statically compiles all of efl into it)? > > As said it is a multiple stage update path. First just a change to > quicklaunch, which should work by itself. Then we can switch from real > libevas.so.1 to a symlink to that binary. This allow us a progressive > try without breaking everyone right away and making it possible for > people/packager on other OS to try it first. so let's not do this quicklaunch executable thing. as per exchange with vincent. windows cant loadlibrary an executable and have it WORK. it can find symbols, but not run (unless we do some linker level work it seems like resolve some libc symbols etc.). let's just merge .so's to fewer .so's (or fewer dll's) and get compatibility working without any quicklaunch craziness. let us see how much better things get then. we haven't measured it so we don't know. this is portable with symlinks on *nix OR "empty DLL's" on windows. my suggestion is we ALWAYS build the empty dll's and have a post install hook that deletes the files and replaces them with symnlinks on *nix only. this way we exercise the windows compatibility path ALWAYS on build+install and then add a post install step to optimize for *nix. we could even make the post install step explicit like make install make install-symlink > > you are trying to replace all of libevas.so.1, libeina.so.1, etc. with > > symlinks TO this quicklaunch binary ... to retain binary compatibility. you > > cant have the .so's AND this ql binary both have the same set of symbols. > > asking for trouble there. > > It shouldn't be a problem thanks to DSO on most system. System without > DSO may be an issue. my point is we can't maintain .so's AND a quicklaunch binary with libs statically linked in. its one or the other. you build an app normally even as a PIE exec, it will INSIST on linking to libevas.so and when it links libevas.so it must find symbols evas_* etc. from there. same for every library. if we have these .so's AND we have a quicklaunch lib with things compiled/linked inside we have multiple evas libs and evas symbols.. one from libevas.so and one from the ql lib. the only way to do this is replace ALL .so's with symlinks to the quicklaunch binary. so its this OR a set of .so's. you can't have both. let's not jump all that way (which cant work on windows as already mentioned. it can be a quicklaunch dll or a quicklaunch exe we can find symbols from but cant run anything from). let's just work on merging our libs and make it portable. let's just get this done. this is more than enough work for now and likely will buy us a lot of improvements on its own. let's measure them and see. > > but to USe this binaries HAVE to be PIe and this makes for a non-portable > > pain in the arse of creating apps. which is why i did .so's for quicklaunch > > before.. because it also works with dll's on windows. PIE does not. not > > that i could find. > > Most system have some form of PIE due to the need to implement ALSR. > For windows, let follow up the discussion with Vincent. yes. but we cant have a combined exe AND dll. you can find symbols and data but not run code according to vincent (unless we ALSO do some custom linker work ourselves which i think is going way way way too far). > > if top, htop, ps, killall, etc. don't work out of the box, it isby everyone > > (except yours) definition... a nightmare. i've messed with this before and > > it does not work across all the tools because of where they source command > > (and that can change based on config and cmdline options too). > > As said, this is just a matter of alias to be documented. I don't > think it is so much of a big deal. see simotek's response. it's not. it's a dealbreaker and will be for the vast vast vast majority of distributors and users. it means a quicklaunch/zygote kind of setup is going to be special and custom always for SOME systems where people are willing to pay this price. well until all ps/top like tools use the fields that allow process renaming to work. if we have this tool MY suggestion is to not even do PIE. do one big NON-RELOCATABLE executable (i know it kills ASLR for efl and anything linked into it, but you could force it to be PIE if you want this). this executable inits up to some specific point then CAN fork. maybe it forks 1 or 2 "spares" that sit and wait for commands. before they can accept commands though they do do some "app local" init like connect to x or wayland servers etc. and THEN sit and wait. when tols to run they dlopen the target and find the efl_main() func and launch from there by calling it. THIS works on windows. it works on *nix. BUT the app HAS to be a dll or .so to work. yes it needs special changes in building of the app, BUT you HAVE to force PIE (change app builds) to have this work ANYWAY. but let's worry about this later when we have also solved how to provide application CONTROL like stdin/out and being able to listen for when the app exits. enlightenment will need this itself anyway. enlightenment shoudl NOT be this process. it should be also a quicklaunched app and symly speak the quicklaunch protocol via api's provided by efl for quicklaunching apps and getting stdin/out, being able to hear when the app exits and get exit codes and so on. let's build these controls and infra first before launching into a PIE executable that is not portable. people have to do special stuff to support such a zygote system anyway. their builds HAVE to alter to do it. if you just build a .so (or .dll) for your app and DO NOT LINK it to anything in efl at all, then this will work just fine. you can link to other things. this ALSO will make the _thread vars in eo for resolve cache much faster because _thread in a non-PIC/PIE binary are totally free. if you are relocatable they turn into full tls (pthread_getspecific) lookups. you want massive raw speed. do this. we can even make quickluanch possibly statically link in everything (libjpeg, libpng, zlib, libc etc.) and get it to have ZERO linking at startup. for full on raw speed this should be the best that can be done, but we can't realistically make this the default for every app and everyone. we could do it for e. we should support it at least and test it and make it work. but we can't sensibly do it for everyone. > > merging it into one location we'll be able to simplify it too so in the end > > we just have some "efl port sys" lib that provides an "always works" set of > > library calls and everything above that is 100% portable and never changes > > platform to platform. > > We kind of have that and kind of not. The issue is that depending on > how far from an Unix system an OS is the deeper the change are. > Windows require a base layer like Evil to handle things, most RTOS > would also require something like that, while Mac OS X doesn't but > will require alternate code for display, input, c&p and drag&drop. > Moving everything down to a very low level means that even for Unix we > need to abstract all this things into a lower layer. I am not to sure > of the benefit and that it would work. Can we make xlib call looks > like windows call for c&p ? Basically I think we will just end up > moving the code we have into another library without much benefit, but > I may missunderstand what you have in mind. we just have a single "efl-system-abstract.so/dll" and this is where that all goes. all complexity in build then goes into this for system abstractions. we have lots of complexity because of OPTIONS like gst vs vlc vs vlc as module vs xin, lots of img loaders optional, etc. etc. - so many things that are not system abstractions but are "but i dont want to install the dependency, or its too now a dep" or whatever to make people happy. hell xlib vs xcb alone adds a lot of code. x vs wl adds a lot etc. either way this is a side topic and not relevant here in this thread. > >> > i am always for improvements. but this really makes this method a very > >> > very very hard ask to do this. i've thought about it and i think this is > >> > far TOO radical a change as it means different execution methods, library > >> > production, linking and so much more os by os. it's kind of insane though > >> > theoretically doable on *nix. i've checked and PIE seems to be urtterly > >> > unportable (windows can't do it based on my reading). the proposal would > >> > be workable ONLY if we could do it unversally so it wouldn't bitrot and > >> > it'd work. this is actually why quicklaunch did .so's instead of PIE > >> > because that WORKS portably.... > >> > >> Windows is anyway another platform with a lot of specificity. > >> Quicklaunch will never work there as there is no equivalent to fork > >> anyway. Emulating what PIE allow is also mostly impossible as > > > > actually it does work. fork works. it's emulated and expensive, bit it > > WORKS. it means you have an already "partially launched" app around. if we > > had a fork POOL sitting around "ready to go" and they get ipc'd the command > > to run... it'd work the same on windows and on *nix. it'd be faster on both > > because when an app runs, the init/liking phase is already done by a > > process that is sitting as a hot spare ready to go. > > Hum, so instead of doing the fork on the spot, we would have to make > ql have a pool of waiting instance. Sounds quite doable everywhere. yes. correct. this is why i say "dont think of this as a *nix only feature". the more we do it across all platforms the less likely it is to bitrot/break. but reality is it cant really be a DEFAULT path for many reasons and we can support this but it's going to have to be special and a PIe "ql binary" is not the best way imho. > > so ql dlopen()'s the apps' DLL (as now it has to be a dll and not an exe) > > and the app is basically a loadable module. and loadable module dlls WORK on > > windows. > > > > the problem is symlinks cannot be done in windows (technically ntfs does > > support them i believe but we cant assume ntfs etc.), so we have to have a > > different build path that creates empty .so's that simply have dependency > > (linking) info so we retain binary compatibility. on *nix we can use > > symlinks to do that. > > Yes. That is still an open problem. And I think it requires two > differents solution. yes. this i know. lets tackle this first just on its own. its enough to tackle. > >> As for the change, they will be the same for all Unix. It is a > > > > not quite. osx doesnt even use the same names. its dylib, not .so. and i > > think its libxxx.1.2.3.dylib ... from memory. so no it's not all the same. > > I am not sure of your point. This is mostly a toolchain and libtool > issue that is already dealt with by the build system. It is not > something that has to be handle specifically. libtool abstracts what the name of the file is. making something to install symlinks means knowing WHAT the real file installed was and its name. we kind of don't know. we need to now make assumptions os by os. well osx vs other unixes for sure. > > its the linking of efl libs into a PIE executable which doesn't work on > > windows. you have to magically figure out a portable way to do that with > > libtool (good luck) across *nix's. you need to deal with windows too. i > > don't think having a completely different spread of libs in windows is good > > at all vs *nix and things will bit-rot. they already do. it's already bad > > enough. i don't think this should be done unless its done universally > > everywhere the same exact way. in fact technically ql does not need to be > > PIE at all. it'd be more efficient to NOT be PIE. normal executable with > > exported symbols. just how enlightenment works with modules. you now cant > > link against it though. you need dummy libs with symbols to link against > > but at runtime don't load them in. this actually would make the _thread > > vars in efl api calls far more efficient (i've measured it and it basically > > makes _thread a nop If its in a fixed location executable which is the > > standard, as opposed to PIC or PIE, but of course we just broke ASLR for > > all of efl if we do that... :)) > > PIE is mandatory on Unix to handle ALSR which is where every one of but binaries by DEFAULT are not PIE. you have to patch autofoo tools or force it in CFLAGS etc. > them is going for security reason. I think that latest Ubuntu force sure but the LIBS (libc, etc.) are ASLR so this does still help for shared libs. btw with quicklaunch and forkign we kill ASLR at least in that every binary we fork has the same memory mapping then... it may be unique to that system but its the same on all binaries "quicklaunched" > all binary to be compiled as PIE. So Unix support for PIE is quite > well there as far as I know since a few year now (At least that's the > case for Linux, Mac OS X, FreeBSD, OpenBSD and NetBSD). Basically, PIE > is there and it will be used more and more for security reason. We can > leverage that without problem I think. Expecting non-PIE support is > going to be the exception more than the rules. Windows does also have > ALSR and provide the same ability, we need to clear out the > initialization of that binary that may be a problem. but doesnt let us do a binary .exe which is ALSO a .so - as per vincent. > > with symlinks - no, because symlinks resolve to the same file and i think > > DSO checks look for linking to the FILE and yet not using any symbols out > > of it at all (which is actually not strictly true and we could try turn off > > DSO checks as it is legal to link to a lib then use dlsym(NULL, ...) to > > hunt for symbols in it at runtime - DSO checks are just trying to add > > sanity checks that are generally true but not always). with .so's that have > > dependencies and link back to one big merged .so - DSO will not like that, > > but this would be needed for WINDOWS because windows does not have > > symlinks. (replace .so with .dll on windows). > > Yes, I think that with symlink, we do not have any DSO issue, but > without we have. Which is my point here. DSO is usually enforced by > the packager and we have no control over that. We can't rely on > bypassing it in my opinion, which is why I have dismissed the empty > libraries so far. This does apply to both proposal of merging some efl > libraries together or merging all of them in ql. i'm just saying we should MAKE empty .so's because this is what we need for empty .dll's for windows anyway. do it and then have a post-install rm and fixup with symlinks. > > regardless if its a PIE quicklaunch executable OR its just merging eina, > > ecore, eo, etc. - its the same problem. just merging the fewer libs is > > doing it on a smaller scale so problems to fix are on a smaller scale too. > > Well, the scale of the problem doesn't matter. If we have no solution > on Unix due to DSO and on Windows due to the absence of symlink, we > have to have two differents solution for both system. This is > something that indeed apply on both proposal. let's just do .so's and dll's for now and then we do things the same EXCEPT empty dll's on win, symlinks on *nix. let's just do this for now and see where we get in terms of improvements. > > for *nix symlinks will do. but to avoid noty testing the build and install > > of the empty dll's (or .so's) we should do so on all builds and symlink > > delete/replace the .so file installed with a synlink on *nix in an install > > hook after installation. we chmod to add +s in e for binary tools for > > example post install. this hook/feature does exist. > > So you do propose that we have empty library on Windows and symlink on > Unix, right ? yes. it's the only way. > > but let's not launch into a single PIE which is we cannot make work across > > all platforms. we CAN merge libs into fewer real .so's or DLL's and we CAN > > have this be universal on all platforms. as long as we don't start messing > > with PIE. > > Why ? We already have two differents solutions with two differents > code path which are unrelated to using PIE/ALSR or not ! PIE/ALSR are > completely orthogonal to the above problem of symlink vs empty dll. but just rm'ing empty .so's and replacing with symlinks on *nix is EASY having a whole PIE executable on *nix cannot be mirrored on windows. it can only be a DLL. > > tyhe empty bin wont have symbols to resulv between eachother. eg eo does not > > have to resole eina symbols. evas does not need to resolve eo and eina > > symbols, edje does not need to resolve evas, eo and eina symbols as they > > are already resolved internally. > > IO on Windows are pretty bad as far as I know, so just dlopen a file > and parsing it is going to be costly, not much we can do there I > think. Which is why I said it wouldn't change much in my opinion. merging our code into fewer dll's means less cross-library symbol resolution but still loads/opens the same number of dll's. it means saving memory with merging writable pages etc. it is a win anyway. > > because this is a low level feature. i'd hook higher level INTO the lower > > level api (extend it by "call "preload func x" for me in quicklaunch) so > > higher level code just extends it. the whole infra for quicklaunch is > > actualyl the same as "single process, multiple instances" where you ipc > > yourself new cmdline args and new env vars, cwd etc. and the single ap > > opens a new window (and switches cwd etc.)... - its similar. it's the same > > blob of context data and likely should use the same core. > > Still, I don't see the point of moving it down the stack. All > application/platform that are going to use it will require the > initialisation of up to elementary, ideally including preloading > configuration/theme and even pre-connecting to the display manager > should be doable, so what is the benefit of moving it down the stack > adding infrastructure for elementary to always init itself ? because the args events and so on are down the stack. > >> > 3. use quicklaunch mode for e. since efl will be there then this will > >> > work. this does not MEAN we have to use it for all apps nor does it mean > >> > we MUST use it for e... but this would help > >> > >> That is what I want to avoid. Having a random path where we may have > >> it for some application for some user on some platform. I really would > >> like this to either be used by a large portion of the user base or not > >> bother. In my opinion 3. doesn't change at all from current situation > >> as in dead code lying around. > > > > and yet you dismiss this as being impossible on windows, so you want to > > have a completely different setup on windows vs *nix and that's ok. i'm > > saying we should do this universally in such a way that it works on ALL > > platforms and ALL cases. we HAVE to keep supporting normal binaries. it HAS > > to be done. this isn't a choice. e itself HAS to change to work with > > quicklaunch because of the way it is built anyway. > > Enlightenment is not working on Windows. Why would you want to > complexify the code of Enlightenment to allow people to turn on/off > quicklaunch while you argue so much that for the rest of EFL it is a > problem to have different code path. I am very confused here. It is > possible that for all the platform where we use Enlightenment to have > only one code path, why would you introduce complexity there when > there is no reason for ? EFL has to deal with OS that are tricky, like > Windows or Mac OS X, but Enlightenment hasn't. I don't see a point of > adding option there while arguing there shouldn't be in EFL. without e's code to be modified to support ql you can NEVER do what you want anyway. it has to be. otherwise then you wont get a faster boot time as e will start as a full binary on its own anyway and link in efl libs. > > we CAN do this on windows AND on *nix IF all apps are dll's or .so's. we > > still need our normal .so libraries there that work for those that do not > > build their apps as a dll or .so. to encourage this to happen we need to > > make it EASY and portable. this also needs to be maintainable over the > > years across platforms and not bitrot because one platform doesn't do it, > > or bitrot because most things don't use quicklaunch and thus it bitrots. we > > can't have it in a corner and ignored. but we can't fragment things. right > > now quicklaunch is bitrotting in a corner being ignored... so... right now > > we use a traditional launch setup with executables actually running as they > > always have linking in libs. moving from this to quicklaunch everywhere AND > > being portable AND maintanable is VERY painful. > > > > the easiest path is to start work on merging libs and work this out on a > > small scale. you'd need the same solutions for a quicklaunch PIE anyway > > (and i'm against doing this). > > As said before, the issue of portability is not PIE, it is > symlink/DSO. Merging library doesn't solve that bit. PIE enable us to no it's PIE too. as per vincent. the symlink issue is *nix only and solves things best. the empty dll's is windows. we can do empty dll's on *nix and i suggest doing ot to keep exercising that code path used for windows anyway. dso isn't an issue on windows. it's an issue in some distros on *nix. symlinks make that all work. or should in theory. lets tackle this first and not try and do everything at once especially when the final goal is NOT portable on windows at all and you are stick to a .dll anyway. > work around the issue, as we can even continue to ship the normal > library for some time and introduce a ql.pc or efl-2.pc that will link > to the quicklaunch binary on all platform avoinding the problem with > symlink/PSO all completely if we can not figure it out. As said before > PIE allow us a gradual roll out of the feature. > > <snip> > > >> enlightenment always rely on quicklaunch or it never rely on it, > >> because the alternative of it relying maybe on it, is a receipe for > >> duplicated code that bit rot in my opinion. Enabling quicklaunch in > >> enlightenment means moving process monitoring to quicklaunch and > >> adding the ability to start a binary in case of a crash. Btw that > >> means that we can provide the same backtrace feature we do provide > >> with enlightenment for all efl application started by quicklaunch. > >> This would improve debugging users issues quite a lot, something we > >> can't do currently. > > > > this now adds one of the hug pains/badnesses of quicklaunch. if you want to > > run something it now is not a child process. look at rage with rage_thumb > > for example. you cant use stdin/out to talk to your child anymore. look at > > evas generic loaders... etc. > > No. rage_thumb is started directly by rage. If rage doesn't enforce > the call to quicklaunch and I don't see why it would, it will not > start rage_thumb as a quicklaunch binary. The application binary is but this SHOULD be possible. E will need it for launching apps.it needs to be able to get stdin/out AND has to know when it exeited even when launched by some 3rd party launcher. without that we CANT use ql to replace launching of anything. > unchanged. It is a PIE binary that start normally without ql if you > execute it directly. This is not going to affect rage_thumb, evas > generic loaders or anything else that is just started with system or > ecore_exe. It has to effectively ask ql to launch that program. That's > why I said that we can easily have an optout in Enlightenment. > > <snip> > > >> > these above should get almost all the benefit of the PIE thing with the > >> > least of the downsides. we still have ps name issues though. :( we need > >> > to dig more to try our best to make this "ok" but this means we still > >> > work normally. as long as the "quicklaunch" binary uses very few efl > >> > symbols, loading efl libs and linking will be a much lower cost as we do > >> > less intra-lib linking and minimal app <-> lib linking. in fact we could > >> > make a single libefl.so as a build option if symlinks and > >> > dependency .so's as above work. this should be almost the same cost as > >> > the pie binary proposal but is portable. > >> > >> I have absolutely no idea what you mean here. The point of quicklaunch > >> is to load efl and force a resolution of all symbol once (this reduce > >> future memory usage and runtime cost), you would even start it by > >> enforcing the linker to resolve all symbol. I am confused ! > > > > my point is do this without quicklaunch first. merge libs. see what we gain. > > the work to merge libs would HAVE to be done for a PIE or whatever solution > > anyway. it's the same. you just dont add in the complexity of a forking > > daemon ipc flow control, ps being useless etc. change. you get some of the > > benefit with some of the work that is a step to a final destination anyway. > > what if this buys 80% of the gain without creating all the quicklaunch > > complexity issues for everyone (because we need this for everyone if it is > > to not bitrot). > > This doesn't match what you were talking about just before. You were > saying just above that ql should use very few efl symbol and I don't > see why, quite the opposite. The point of ql is also to do all the > symbol resolution once. This means that the dirty page related to > relocation is shared among all executable started by ql. This save > times and memory and one of the reason you want to use "all" efl > symbol. I just wanted clarification on the above point. a non-pie/pic binary doesn't need to relocate symbols. they are fixed. so you dont have to dirty pages for them at all. well you do in the apps you load. not in a NON-PIE ql binary. -- ------------- Codito, ergo sum - "I code, therefore I am" -------------- The Rasterman (Carsten Haitzler) ras...@rasterman.com ------------------------------------------------------------------------------ The Command Line: Reinvented for Modern Developers Did the resurgence of CLI tooling catch you by surprise? Reconnect with the command line and become more productive. Learn the new .NET and ASP.NET CLI. Get your free copy! http://sdm.link/telerik _______________________________________________ enlightenment-devel mailing list enlightenment-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/enlightenment-devel
