Re: FreeBSD MachO File format, your comments on it.

2016-03-27 Thread mokhi
Hi again.

I've implemented FatElf support for Elf image activator too[1] :)
Any comments/reviews on this ?
I also pinged Ryan Gordon about this.

Also im curious to know if any comments/reviews are done on MachO
implementation[2]

Best wishes and thousands of regards, Mokhi.

==
[1] https://github.com/m0khi/FreeBSD_FatElf
[2] https://github.com/m0khi/FreeBSD_MachO
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


Re: FreeBSD MachO File format, your comments on it.

2016-03-25 Thread mokhi
errr, typo ...

s/an FatELF both/and FatELF both/g

:)

On 3/25/16, mokhi  wrote:
> Adrian, thanks for your +1 :P
>
> So, what about EULA related things that 'David' pointed to?
> If this isn't really a big problem, I have no problem to continue on
> working on it, an FatELF both ;)
>
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


Re: FreeBSD MachO File format, your comments on it.

2016-03-25 Thread mokhi
Adrian, thanks for your +1 :P

So, what about EULA related things that 'David' pointed to?
If this isn't really a big problem, I have no problem to continue on
working on it, an FatELF both ;)
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


Re: FreeBSD MachO File format, your comments on it.

2016-03-24 Thread Adrian Chadd
+1, get mach-o up, see if we can twist other people to work on the
other missing bits to run apple stuff on freebsd. :P


-a


On 24 March 2016 at 07:26, mokhi  wrote:
> So, I'll try to port FatELF as well as MachO.
> Choosing the better one is up to you ;)
>
> All opinions/idea are welcome and I appreciate.
>
> Best wishes, Mokhi.
> ___
> freebsd-current@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-current
> To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


Re: FreeBSD MachO File format, your comments on it.

2016-03-24 Thread mokhi
So, I'll try to port FatELF as well as MachO.
Choosing the better one is up to you ;)

All opinions/idea are welcome and I appreciate.

Best wishes, Mokhi.
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


Re: FreeBSD MachO File format, your comments on it.

2016-03-24 Thread David Chisnall
On 24 Mar 2016, at 13:42, Damjan Jovanovic  wrote:
> 
> ELF itself is a disaster. Symbol lookup in ELF is process scoped, not
> library scoped like Windows's PE and Mac's Mach-O, so same named
> symbols from different libraries in the same process (loaded through
> any number of levels of indirection) can and do clash, resulting in
> memory corruption. This is why hacks like symbol versioning,
> RTLD_DEEPBIND on GNU's libc and -Bdirect on Solaris were invented.

This problem is addressed by some of the work that Sony has done recently that 
they are about to upstream to Clang/LLVM.

> We suffer from this problem badly on FreeBSD, as Clang's C++ standard
> library and GCC's standard library don't have fully compatible ABIs,
> so when both are loaded into the same process and the incompatible C++
> features are used -> memory corruption -> crash. Eg. compile Apache
> OpenOffice with GCC on a system built with Clang, and you'll see even
> the unit tests crash.

That shouldn’t happen, as libstd++ and libc++ have different symbols (libc++ 
puts its symbols in the __v1 namespace).  The problem can come from mixing 
libsupc++ and libcxxrt, but that’s only an issue if you have not built 
libstdc++ against libcxxrt.

David

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: FreeBSD MachO File format, your comments on it.

2016-03-24 Thread Damjan Jovanovic
ELF itself is a disaster. Symbol lookup in ELF is process scoped, not
library scoped like Windows's PE and Mac's Mach-O, so same named
symbols from different libraries in the same process (loaded through
any number of levels of indirection) can and do clash, resulting in
memory corruption. This is why hacks like symbol versioning,
RTLD_DEEPBIND on GNU's libc and -Bdirect on Solaris were invented.

We suffer from this problem badly on FreeBSD, as Clang's C++ standard
library and GCC's standard library don't have fully compatible ABIs,
so when both are loaded into the same process and the incompatible C++
features are used -> memory corruption -> crash. Eg. compile Apache
OpenOffice with GCC on a system built with Clang, and you'll see even
the unit tests crash.

This is why I am personally interested in alternatives like Mach-O.

Damjan

On Thu, Mar 24, 2016 at 12:51 PM, David Chisnall  wrote:
> Hi,
>
> I’d slightly question the assertion that Mach-O is a well-designed format.  
> For example, it has a hard limit of 16 section types, doesn’t support COMDATs 
> and so on.  OS X uses a load of magic section names to work around these 
> limitations.
>
> Note that a Mach-O image activator is relatively easy, but a Mach-O rtld is 
> far more complex.  It might be possible to port dyld from OS X, but as I 
> recall it depends quite heavily on the Mach kernel interfaces.
>
> On fat binaries, note that the support in the file format is pretty trivial.  
> Far more support is needed in the image activator and rtld to determine the 
> correct parts and map only them.  If you’re interested in doing this work, 
> then I’d recommend looking at this proposed specification for fat ELF 
> binaries:
>
> https://icculus.org/fatelf/
>
> That said, I’m not totally convinced that fat binaries are actually a good 
> solution (unless you’re willing to go a step further than Apple did and merge 
> data sections) - NeXT managed very well shipping fat bundles without using 
> fat binaries and even had a special mode in ditto to strip out the foreign 
> architectures when copying a bundle from a network share to a local 
> filesystem.
>
> Persuading clang to emit FreeBSD Mach-O binaries is probably harder than you 
> think.  It’s quite easy to persuade it that Mach is a valid file format for 
> FreeBSD, but there are a *lot* of places where people conflate ‘is Mach’ with 
> ‘is Darwin’ in the Clang and LLVM sources.  Finding all of these and making 
> sure that they’re really checking the correct one is difficult.
>
> Emulating OS X binaries may be interesting.  NetBSD had a Mach / XNU compat 
> layer for a while.  The problem here is that the graphics stack interfaces on 
> OS X are completely different from any other *NIX system (as are the kernel 
> interfaces for sound), so the most that they could do was run command-line 
> and X11 Mac apps - not especially useful.  Actually emulating OS X apps will 
> need far more than that - OS X ships with about 500MB of frameworks, many of 
> which are used by most applications.  The GNUstep project is undermanned and 
> hasn’t been able to keep up with the changes to the core Foundation and 
> AppKit frameworks, let alone the rest.
>
> David
>
>> On 24 Mar 2016, at 09:13, mokhi  wrote:
>>
>> Hi guys.
>> I'm Mahdi Mokhtari (aka Mokhi between FreeBSD friends).
>>
>> I am working on adding Mach-O binary format to supported formats for FreeBSD.
>> Not for emulations on first step, but as a native supported format
>> just like a.out [or Elf]
>> (though it can go in both ways too).
>>
>> There are good reasons to have Mach-O format support IMO.
>> It's well/clear designed file format.
>> Can supports multiple Arch by default (It's Fat Format).
>> Because of its Fat Format support, it can even help porting/packaging easier 
>> for
>> projects such as Freebsd-arm or others IMO :D.
>> At end (even not among its interesting parts, maybe :D) point, it
>> leads and helps to have
>> OSX emulation support on FreeBSD.
>>
>> BTW, i've coded[1] Mach-O support for FreeBSD with helps of
>> FreeBSD-ppl on IRC about various aspects of this works (from
>> fundamental points of VM-MAP, to SysEntVec for Mach-O format) and
>> with help of Elf and a.out format codes and mach-o references.
>> It's in Alpha state (or before it) IMO, as I'm not sure about some of
>> its parts, but I've tested a mach-o formatted binary with it and it at
>> least loads and maps it segments correctly :D. (it was actually a
>> simple "return 0" C Code, compiled in a OSX, if you know how can I
>> force my FreeBSD clang to produce mach-o files instead of ELF I'd be
>> happy to know it, and I appreciate :D)
>>
>> I'd like to have your helps and comments on it, in hope to make it better
>> and make it ready for review.
>>
>> Thanks and thousands of regards, Mokhi.
>>
>> ==
>> [1] https://github.com/m0khi/FreeBSD_MachO
>> ___
>> 

Re: FreeBSD MachO File format, your comments on it.

2016-03-24 Thread mokhi
On 3/24/16, mokhi  wrote:
> Then you think the better idea is porting FatELF to FreeBSD, rather
> than working on MachO?
>


If yes,
I am ready to put dedicated time on it :) [as I did for MachO]

But before that, you think, is there any changes we can/should make on it?

* I read something about FatELF when I was doing research for macho :)
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


Re: FreeBSD MachO File format, your comments on it.

2016-03-24 Thread mokhi
Then you think the better idea is porting FatELF to FreeBSD, rather
than working on MachO?
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


Re: FreeBSD MachO File format, your comments on it.

2016-03-24 Thread mokhi
On 3/24/16, Achim Patzner  wrote:
> Would that project end in having an intelligent loader that will only map
> the relevant architecture to memory (i. e. I’ll have extremely fat
> executables supporting any known architecture in the universe on /usr/local
> or even for all files that can be shared between machines) and keep the load
> on memory and network as low as possible?
It'll be one of its results, (at least) I expect :D though I'm not
sure how it will have effect on network.

> automatic cross-compilation to building I feel like Christmas came early.
Then happy vacation  ;)

Thanks and thousands of regards, Mokhi.
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: FreeBSD MachO File format, your comments on it.

2016-03-24 Thread David Chisnall
On 24 Mar 2016, at 12:05, mokhi  wrote:
> 
> Hi.
> 
> I'm agreed with point you told about improvements we can do for fat
> format (or more).
> And I'm ready to do them (with your helps, sure :D).
> 
> But we need short steps and more of them (a local proverb :D) IMO.
> If we completely do this image activator, then we can have 2 sub plans
> for OSX emulation and/or fat data segment redesign.

FatELF binaries do not depend on this work.  Fat Mach-O binaries do, but the 
pain of working with Mach-O is not worth it (talk to some of the Apple 
toolchain team some time about how much effort Mach-O is - I’m glad it’s their 
problem and not mine).

I don’t believe that the work to support FatELF would be particularly large.  
The format is pretty simple (basically a small header that tells you where 
within the binaries to find the real ELF for your architecture).  Teaching all 
of the associated bits of the toolchain (especially debuggers) about it is a 
lot of tedious work, but not particularly hard if someone is motivated to do 
it.  Teaching clang and lld to produce fat binaries as part of normal ELF 
compilation would be a bit more work.

> I saw netbsd's way of mach-kernel/darwin emulation.
> They have been stopped in porting/simulating quartz (the reason
> described lack of developers' interest IIRC), and that relates to OSX
> emulating.

That wasn’t the only reason.  The XNU kernel interfaces for graphics and sound 
are large and mostly undocumented (at least, publicly) and change between OS X 
revisions.  Even if you implement *all* of this, then you’d still need most of 
an OS X userland to be able to run OS X applications.  This would involve 
violating the OS X EULA unless you ran it on a Mac and the only thing that 
you’d then be able to do that you couldn’t with OS X is run FreeBSD binaries in 
the background or in XQuartz (which you can already do pretty well with xhyve 
on OS X).  If you are willing to violate the OS X EULA then you should probably 
just run OS X in a VM.

> If we wanna complete/continue that way, first we need this image
> activator, what's your opinion about it?
> 
> BTW, in brief I believe we can have strategies to do (sub plans) and
> it worth (at least for me, because I'll learn good things). What's
> your opinion?

As a learning exercise, I definitely encourage you to continue.  Writing a new 
image activator will teach you a lot.  If you want to do some of the rtld work 
to make a partial Mach-O rtld then you’ll learn even more.  I just don’t think 
that the end result will be something that’s particularly useful to anyone.

David

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: FreeBSD MachO File format, your comments on it.

2016-03-24 Thread mokhi
Hi.

I'm agreed with point you told about improvements we can do for fat
format (or more).
And I'm ready to do them (with your helps, sure :D).

But we need short steps and more of them (a local proverb :D) IMO.
If we completely do this image activator, then we can have 2 sub plans
for OSX emulation and/or fat data segment redesign.

I saw netbsd's way of mach-kernel/darwin emulation.
They have been stopped in porting/simulating quartz (the reason
described lack of developers' interest IIRC), and that relates to OSX
emulating.
If we wanna complete/continue that way, first we need this image
activator, what's your opinion about it?

BTW, in brief I believe we can have strategies to do (sub plans) and
it worth (at least for me, because I'll learn good things). What's
your opinion?


Best wishes, Mokhi.
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


Re: FreeBSD MachO File format, your comments on it.

2016-03-24 Thread David Chisnall
Hi,

I’d slightly question the assertion that Mach-O is a well-designed format.  For 
example, it has a hard limit of 16 section types, doesn’t support COMDATs and 
so on.  OS X uses a load of magic section names to work around these 
limitations.

Note that a Mach-O image activator is relatively easy, but a Mach-O rtld is far 
more complex.  It might be possible to port dyld from OS X, but as I recall it 
depends quite heavily on the Mach kernel interfaces.

On fat binaries, note that the support in the file format is pretty trivial.  
Far more support is needed in the image activator and rtld to determine the 
correct parts and map only them.  If you’re interested in doing this work, then 
I’d recommend looking at this proposed specification for fat ELF binaries:

https://icculus.org/fatelf/

That said, I’m not totally convinced that fat binaries are actually a good 
solution (unless you’re willing to go a step further than Apple did and merge 
data sections) - NeXT managed very well shipping fat bundles without using fat 
binaries and even had a special mode in ditto to strip out the foreign 
architectures when copying a bundle from a network share to a local filesystem.

Persuading clang to emit FreeBSD Mach-O binaries is probably harder than you 
think.  It’s quite easy to persuade it that Mach is a valid file format for 
FreeBSD, but there are a *lot* of places where people conflate ‘is Mach’ with 
‘is Darwin’ in the Clang and LLVM sources.  Finding all of these and making 
sure that they’re really checking the correct one is difficult.

Emulating OS X binaries may be interesting.  NetBSD had a Mach / XNU compat 
layer for a while.  The problem here is that the graphics stack interfaces on 
OS X are completely different from any other *NIX system (as are the kernel 
interfaces for sound), so the most that they could do was run command-line and 
X11 Mac apps - not especially useful.  Actually emulating OS X apps will need 
far more than that - OS X ships with about 500MB of frameworks, many of which 
are used by most applications.  The GNUstep project is undermanned and hasn’t 
been able to keep up with the changes to the core Foundation and AppKit 
frameworks, let alone the rest.

David

> On 24 Mar 2016, at 09:13, mokhi  wrote:
> 
> Hi guys.
> I'm Mahdi Mokhtari (aka Mokhi between FreeBSD friends).
> 
> I am working on adding Mach-O binary format to supported formats for FreeBSD.
> Not for emulations on first step, but as a native supported format
> just like a.out [or Elf]
> (though it can go in both ways too).
> 
> There are good reasons to have Mach-O format support IMO.
> It's well/clear designed file format.
> Can supports multiple Arch by default (It's Fat Format).
> Because of its Fat Format support, it can even help porting/packaging easier 
> for
> projects such as Freebsd-arm or others IMO :D.
> At end (even not among its interesting parts, maybe :D) point, it
> leads and helps to have
> OSX emulation support on FreeBSD.
> 
> BTW, i've coded[1] Mach-O support for FreeBSD with helps of
> FreeBSD-ppl on IRC about various aspects of this works (from
> fundamental points of VM-MAP, to SysEntVec for Mach-O format) and
> with help of Elf and a.out format codes and mach-o references.
> It's in Alpha state (or before it) IMO, as I'm not sure about some of
> its parts, but I've tested a mach-o formatted binary with it and it at
> least loads and maps it segments correctly :D. (it was actually a
> simple "return 0" C Code, compiled in a OSX, if you know how can I
> force my FreeBSD clang to produce mach-o files instead of ELF I'd be
> happy to know it, and I appreciate :D)
> 
> I'd like to have your helps and comments on it, in hope to make it better
> and make it ready for review.
> 
> Thanks and thousands of regards, Mokhi.
> 
> ==
> [1] https://github.com/m0khi/FreeBSD_MachO
> ___
> freebsd-current@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-current
> To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: FreeBSD MachO File format, your comments on it.

2016-03-24 Thread Achim Patzner

> Am 24.03.2016 um 10:13 schrieb mokhi :
> 
> Hi guys.
> I'm Mahdi Mokhtari (aka Mokhi between FreeBSD friends).
> 
> I am working on adding Mach-O binary format to supported formats for FreeBSD.

Would that project end in having an intelligent loader that will only map the 
relevant architecture to memory (i. e. I’ll have extremely fat executables 
supporting any known architecture in the universe on /usr/local or even for all 
files that can be shared between machines) and keep the load on memory and 
network as low as possible?

Cool. I’ll buy immediately. Now if someone would add completely hassle-free 
automatic cross-compilation to building I feel like Christmas came early.

(Just imagine an “one stick fits them all”-installer…)


Achim



smime.p7s
Description: S/MIME cryptographic signature