Re: FreeBSD MachO File format, your comments on it.
Hi again. I've implemented FatElf support for Elf image activator too[1] :) Any comments/reviews on this ? I also pinged Ryan Gordon about this. Also im curious to know if any comments/reviews are done on MachO implementation[2] Best wishes and thousands of regards, Mokhi. == [1] https://github.com/m0khi/FreeBSD_FatElf [2] https://github.com/m0khi/FreeBSD_MachO ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: FreeBSD MachO File format, your comments on it.
errr, typo ... s/an FatELF both/and FatELF both/g :) On 3/25/16, mokhi wrote: > Adrian, thanks for your +1 :P > > So, what about EULA related things that 'David' pointed to? > If this isn't really a big problem, I have no problem to continue on > working on it, an FatELF both ;) > ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: FreeBSD MachO File format, your comments on it.
Adrian, thanks for your +1 :P So, what about EULA related things that 'David' pointed to? If this isn't really a big problem, I have no problem to continue on working on it, an FatELF both ;) ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: FreeBSD MachO File format, your comments on it.
+1, get mach-o up, see if we can twist other people to work on the other missing bits to run apple stuff on freebsd. :P -a On 24 March 2016 at 07:26, mokhi wrote: > So, I'll try to port FatELF as well as MachO. > Choosing the better one is up to you ;) > > All opinions/idea are welcome and I appreciate. > > Best wishes, Mokhi. > ___ > freebsd-current@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-current > To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org" ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: FreeBSD MachO File format, your comments on it.
So, I'll try to port FatELF as well as MachO. Choosing the better one is up to you ;) All opinions/idea are welcome and I appreciate. Best wishes, Mokhi. ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: FreeBSD MachO File format, your comments on it.
On 24 Mar 2016, at 13:42, Damjan Jovanovic wrote: > > ELF itself is a disaster. Symbol lookup in ELF is process scoped, not > library scoped like Windows's PE and Mac's Mach-O, so same named > symbols from different libraries in the same process (loaded through > any number of levels of indirection) can and do clash, resulting in > memory corruption. This is why hacks like symbol versioning, > RTLD_DEEPBIND on GNU's libc and -Bdirect on Solaris were invented. This problem is addressed by some of the work that Sony has done recently that they are about to upstream to Clang/LLVM. > We suffer from this problem badly on FreeBSD, as Clang's C++ standard > library and GCC's standard library don't have fully compatible ABIs, > so when both are loaded into the same process and the incompatible C++ > features are used -> memory corruption -> crash. Eg. compile Apache > OpenOffice with GCC on a system built with Clang, and you'll see even > the unit tests crash. That shouldn’t happen, as libstd++ and libc++ have different symbols (libc++ puts its symbols in the __v1 namespace). The problem can come from mixing libsupc++ and libcxxrt, but that’s only an issue if you have not built libstdc++ against libcxxrt. David ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: FreeBSD MachO File format, your comments on it.
ELF itself is a disaster. Symbol lookup in ELF is process scoped, not library scoped like Windows's PE and Mac's Mach-O, so same named symbols from different libraries in the same process (loaded through any number of levels of indirection) can and do clash, resulting in memory corruption. This is why hacks like symbol versioning, RTLD_DEEPBIND on GNU's libc and -Bdirect on Solaris were invented. We suffer from this problem badly on FreeBSD, as Clang's C++ standard library and GCC's standard library don't have fully compatible ABIs, so when both are loaded into the same process and the incompatible C++ features are used -> memory corruption -> crash. Eg. compile Apache OpenOffice with GCC on a system built with Clang, and you'll see even the unit tests crash. This is why I am personally interested in alternatives like Mach-O. Damjan On Thu, Mar 24, 2016 at 12:51 PM, David Chisnall wrote: > Hi, > > I’d slightly question the assertion that Mach-O is a well-designed format. > For example, it has a hard limit of 16 section types, doesn’t support COMDATs > and so on. OS X uses a load of magic section names to work around these > limitations. > > Note that a Mach-O image activator is relatively easy, but a Mach-O rtld is > far more complex. It might be possible to port dyld from OS X, but as I > recall it depends quite heavily on the Mach kernel interfaces. > > On fat binaries, note that the support in the file format is pretty trivial. > Far more support is needed in the image activator and rtld to determine the > correct parts and map only them. If you’re interested in doing this work, > then I’d recommend looking at this proposed specification for fat ELF > binaries: > > https://icculus.org/fatelf/ > > That said, I’m not totally convinced that fat binaries are actually a good > solution (unless you’re willing to go a step further than Apple did and merge > data sections) - NeXT managed very well shipping fat bundles without using > fat binaries and even had a special mode in ditto to strip out the foreign > architectures when copying a bundle from a network share to a local > filesystem. > > Persuading clang to emit FreeBSD Mach-O binaries is probably harder than you > think. It’s quite easy to persuade it that Mach is a valid file format for > FreeBSD, but there are a *lot* of places where people conflate ‘is Mach’ with > ‘is Darwin’ in the Clang and LLVM sources. Finding all of these and making > sure that they’re really checking the correct one is difficult. > > Emulating OS X binaries may be interesting. NetBSD had a Mach / XNU compat > layer for a while. The problem here is that the graphics stack interfaces on > OS X are completely different from any other *NIX system (as are the kernel > interfaces for sound), so the most that they could do was run command-line > and X11 Mac apps - not especially useful. Actually emulating OS X apps will > need far more than that - OS X ships with about 500MB of frameworks, many of > which are used by most applications. The GNUstep project is undermanned and > hasn’t been able to keep up with the changes to the core Foundation and > AppKit frameworks, let alone the rest. > > David > >> On 24 Mar 2016, at 09:13, mokhi wrote: >> >> Hi guys. >> I'm Mahdi Mokhtari (aka Mokhi between FreeBSD friends). >> >> I am working on adding Mach-O binary format to supported formats for FreeBSD. >> Not for emulations on first step, but as a native supported format >> just like a.out [or Elf] >> (though it can go in both ways too). >> >> There are good reasons to have Mach-O format support IMO. >> It's well/clear designed file format. >> Can supports multiple Arch by default (It's Fat Format). >> Because of its Fat Format support, it can even help porting/packaging easier >> for >> projects such as Freebsd-arm or others IMO :D. >> At end (even not among its interesting parts, maybe :D) point, it >> leads and helps to have >> OSX emulation support on FreeBSD. >> >> BTW, i've coded[1] Mach-O support for FreeBSD with helps of >> FreeBSD-ppl on IRC about various aspects of this works (from >> fundamental points of VM-MAP, to SysEntVec for Mach-O format) and >> with help of Elf and a.out format codes and mach-o references. >> It's in Alpha state (or before it) IMO, as I'm not sure about some of >> its parts, but I've tested a mach-o formatted binary with it and it at >> least loads and maps it segments correctly :D. (it was actually a >> simple "return 0" C Code, compiled in a OSX, if you know how can I >> force my FreeBSD clang to produce mach-o files instead of ELF I'd be >> happy to know it, and I appreciate :D) >> >> I'd like to have your helps and comments on it, in hope to make it better >> and make it ready for review. >> >> Thanks and thousands of regards, Mokhi. >> >> == >> [1] https://github.com/m0khi/FreeBSD_MachO >> ___ >> freebsd-current@freebsd.org mailing list >>
Re: FreeBSD MachO File format, your comments on it.
On 3/24/16, mokhi wrote: > Then you think the better idea is porting FatELF to FreeBSD, rather > than working on MachO? > If yes, I am ready to put dedicated time on it :) [as I did for MachO] But before that, you think, is there any changes we can/should make on it? * I read something about FatELF when I was doing research for macho :) ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: FreeBSD MachO File format, your comments on it.
Then you think the better idea is porting FatELF to FreeBSD, rather than working on MachO? ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: FreeBSD MachO File format, your comments on it.
On 3/24/16, Achim Patzner wrote: > Would that project end in having an intelligent loader that will only map > the relevant architecture to memory (i. e. I’ll have extremely fat > executables supporting any known architecture in the universe on /usr/local > or even for all files that can be shared between machines) and keep the load > on memory and network as low as possible? It'll be one of its results, (at least) I expect :D though I'm not sure how it will have effect on network. > automatic cross-compilation to building I feel like Christmas came early. Then happy vacation ;) Thanks and thousands of regards, Mokhi. ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: FreeBSD MachO File format, your comments on it.
On 24 Mar 2016, at 12:05, mokhi wrote: > > Hi. > > I'm agreed with point you told about improvements we can do for fat > format (or more). > And I'm ready to do them (with your helps, sure :D). > > But we need short steps and more of them (a local proverb :D) IMO. > If we completely do this image activator, then we can have 2 sub plans > for OSX emulation and/or fat data segment redesign. FatELF binaries do not depend on this work. Fat Mach-O binaries do, but the pain of working with Mach-O is not worth it (talk to some of the Apple toolchain team some time about how much effort Mach-O is - I’m glad it’s their problem and not mine). I don’t believe that the work to support FatELF would be particularly large. The format is pretty simple (basically a small header that tells you where within the binaries to find the real ELF for your architecture). Teaching all of the associated bits of the toolchain (especially debuggers) about it is a lot of tedious work, but not particularly hard if someone is motivated to do it. Teaching clang and lld to produce fat binaries as part of normal ELF compilation would be a bit more work. > I saw netbsd's way of mach-kernel/darwin emulation. > They have been stopped in porting/simulating quartz (the reason > described lack of developers' interest IIRC), and that relates to OSX > emulating. That wasn’t the only reason. The XNU kernel interfaces for graphics and sound are large and mostly undocumented (at least, publicly) and change between OS X revisions. Even if you implement *all* of this, then you’d still need most of an OS X userland to be able to run OS X applications. This would involve violating the OS X EULA unless you ran it on a Mac and the only thing that you’d then be able to do that you couldn’t with OS X is run FreeBSD binaries in the background or in XQuartz (which you can already do pretty well with xhyve on OS X). If you are willing to violate the OS X EULA then you should probably just run OS X in a VM. > If we wanna complete/continue that way, first we need this image > activator, what's your opinion about it? > > BTW, in brief I believe we can have strategies to do (sub plans) and > it worth (at least for me, because I'll learn good things). What's > your opinion? As a learning exercise, I definitely encourage you to continue. Writing a new image activator will teach you a lot. If you want to do some of the rtld work to make a partial Mach-O rtld then you’ll learn even more. I just don’t think that the end result will be something that’s particularly useful to anyone. David ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: FreeBSD MachO File format, your comments on it.
Hi. I'm agreed with point you told about improvements we can do for fat format (or more). And I'm ready to do them (with your helps, sure :D). But we need short steps and more of them (a local proverb :D) IMO. If we completely do this image activator, then we can have 2 sub plans for OSX emulation and/or fat data segment redesign. I saw netbsd's way of mach-kernel/darwin emulation. They have been stopped in porting/simulating quartz (the reason described lack of developers' interest IIRC), and that relates to OSX emulating. If we wanna complete/continue that way, first we need this image activator, what's your opinion about it? BTW, in brief I believe we can have strategies to do (sub plans) and it worth (at least for me, because I'll learn good things). What's your opinion? Best wishes, Mokhi. ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: FreeBSD MachO File format, your comments on it.
Hi, I’d slightly question the assertion that Mach-O is a well-designed format. For example, it has a hard limit of 16 section types, doesn’t support COMDATs and so on. OS X uses a load of magic section names to work around these limitations. Note that a Mach-O image activator is relatively easy, but a Mach-O rtld is far more complex. It might be possible to port dyld from OS X, but as I recall it depends quite heavily on the Mach kernel interfaces. On fat binaries, note that the support in the file format is pretty trivial. Far more support is needed in the image activator and rtld to determine the correct parts and map only them. If you’re interested in doing this work, then I’d recommend looking at this proposed specification for fat ELF binaries: https://icculus.org/fatelf/ That said, I’m not totally convinced that fat binaries are actually a good solution (unless you’re willing to go a step further than Apple did and merge data sections) - NeXT managed very well shipping fat bundles without using fat binaries and even had a special mode in ditto to strip out the foreign architectures when copying a bundle from a network share to a local filesystem. Persuading clang to emit FreeBSD Mach-O binaries is probably harder than you think. It’s quite easy to persuade it that Mach is a valid file format for FreeBSD, but there are a *lot* of places where people conflate ‘is Mach’ with ‘is Darwin’ in the Clang and LLVM sources. Finding all of these and making sure that they’re really checking the correct one is difficult. Emulating OS X binaries may be interesting. NetBSD had a Mach / XNU compat layer for a while. The problem here is that the graphics stack interfaces on OS X are completely different from any other *NIX system (as are the kernel interfaces for sound), so the most that they could do was run command-line and X11 Mac apps - not especially useful. Actually emulating OS X apps will need far more than that - OS X ships with about 500MB of frameworks, many of which are used by most applications. The GNUstep project is undermanned and hasn’t been able to keep up with the changes to the core Foundation and AppKit frameworks, let alone the rest. David > On 24 Mar 2016, at 09:13, mokhi wrote: > > Hi guys. > I'm Mahdi Mokhtari (aka Mokhi between FreeBSD friends). > > I am working on adding Mach-O binary format to supported formats for FreeBSD. > Not for emulations on first step, but as a native supported format > just like a.out [or Elf] > (though it can go in both ways too). > > There are good reasons to have Mach-O format support IMO. > It's well/clear designed file format. > Can supports multiple Arch by default (It's Fat Format). > Because of its Fat Format support, it can even help porting/packaging easier > for > projects such as Freebsd-arm or others IMO :D. > At end (even not among its interesting parts, maybe :D) point, it > leads and helps to have > OSX emulation support on FreeBSD. > > BTW, i've coded[1] Mach-O support for FreeBSD with helps of > FreeBSD-ppl on IRC about various aspects of this works (from > fundamental points of VM-MAP, to SysEntVec for Mach-O format) and > with help of Elf and a.out format codes and mach-o references. > It's in Alpha state (or before it) IMO, as I'm not sure about some of > its parts, but I've tested a mach-o formatted binary with it and it at > least loads and maps it segments correctly :D. (it was actually a > simple "return 0" C Code, compiled in a OSX, if you know how can I > force my FreeBSD clang to produce mach-o files instead of ELF I'd be > happy to know it, and I appreciate :D) > > I'd like to have your helps and comments on it, in hope to make it better > and make it ready for review. > > Thanks and thousands of regards, Mokhi. > > == > [1] https://github.com/m0khi/FreeBSD_MachO > ___ > freebsd-current@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-current > To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org" ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: FreeBSD MachO File format, your comments on it.
> Am 24.03.2016 um 10:13 schrieb mokhi : > > Hi guys. > I'm Mahdi Mokhtari (aka Mokhi between FreeBSD friends). > > I am working on adding Mach-O binary format to supported formats for FreeBSD. Would that project end in having an intelligent loader that will only map the relevant architecture to memory (i. e. I’ll have extremely fat executables supporting any known architecture in the universe on /usr/local or even for all files that can be shared between machines) and keep the load on memory and network as low as possible? Cool. I’ll buy immediately. Now if someone would add completely hassle-free automatic cross-compilation to building I feel like Christmas came early. (Just imagine an “one stick fits them all”-installer…) Achim smime.p7s Description: S/MIME cryptographic signature
FreeBSD MachO File format, your comments on it.
Hi guys. I'm Mahdi Mokhtari (aka Mokhi between FreeBSD friends). I am working on adding Mach-O binary format to supported formats for FreeBSD. Not for emulations on first step, but as a native supported format just like a.out [or Elf] (though it can go in both ways too). There are good reasons to have Mach-O format support IMO. It's well/clear designed file format. Can supports multiple Arch by default (It's Fat Format). Because of its Fat Format support, it can even help porting/packaging easier for projects such as Freebsd-arm or others IMO :D. At end (even not among its interesting parts, maybe :D) point, it leads and helps to have OSX emulation support on FreeBSD. BTW, i've coded[1] Mach-O support for FreeBSD with helps of FreeBSD-ppl on IRC about various aspects of this works (from fundamental points of VM-MAP, to SysEntVec for Mach-O format) and with help of Elf and a.out format codes and mach-o references. It's in Alpha state (or before it) IMO, as I'm not sure about some of its parts, but I've tested a mach-o formatted binary with it and it at least loads and maps it segments correctly :D. (it was actually a simple "return 0" C Code, compiled in a OSX, if you know how can I force my FreeBSD clang to produce mach-o files instead of ELF I'd be happy to know it, and I appreciate :D) I'd like to have your helps and comments on it, in hope to make it better and make it ready for review. Thanks and thousands of regards, Mokhi. == [1] https://github.com/m0khi/FreeBSD_MachO ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"