Re: Making (useful subsets of) bytecode portable between targets

[MarLinn via ghc-devs]

On 2016-11-25 12:11, Simon Marlow wrote:
We basically have two worlds: first, the compile-time world. In this 
world, we need all the packages and modules of the current package 
built for the host platform. Secondly, we need the runtime world, with 
all the packages and modules of the current package cross-compiled for 
the target platform.


Maybe this separation and the preceding discussion of the two possible 
solutions suggests a usable approach to the architecture of this future 
system?


First, let me reframe the "runner" idea. In a real-world environment, 
this seems like a viable solution either with two separate machines or 
with a VM nested in the main build machine. In both cases, we would need 
two parts of the compiler, communicating over customized channels.
The cross-compiler approach is more or less just a variation on this 
with far less overhead.

So why not build an architecture that supports both solutions?

I practice, this would mean we need a tightly defined, but flexible API 
between at least two "architecture plugins" and one controller that 
could run on either side. To me, this sounds more like a build system 
than a mere compiler. And I'm okay with that, but I don't think 
GHC+Cabal alone can and should shoulder the complexity. There are nice, 
working build-systems out there that could take over the role of the 
controller, so all GHC and Cabal would have to offer are parsing, 
modularized steps, and nice hooks. In other words, /a //kind of 
meta-language to describe compiler deployments/ – and Haskell is great 
for describing languages.


Here's yet another idea I'd like to add, although it is rather silly. 
The idea of a meta-language that describes a conversion structure seems 
very close to what Pandoc is doing for documents. And while Pandoc's 
architecture and history make it a bit static, GHC can still learn from 
it. Maybe, someday, there could even be a bigger, even more over-arching 
build language that describes the program, the documentation, and the 
deployment processes of the whole system?


Cheers,
MarLinn
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: Making (useful subsets of) bytecode portable between targets

[Edward Z. Yang]

Excerpts from Shea Levy's message of 2016-11-25 10:30:25 -0500:
> > The right thing is to have a clean separation between runtime
> > imports and compile-time imports.  Perhaps we just annotate some imports to
> > say they aren't needed at compile-time for running the TH code.  but then
> > we also need compile-time vs. runtime build-depends in our .cabal files,
> > and so on.
> 
> Yes, I was just looking into this yesterday. We already have something
> similar for plugins, though of course the TH story is much more
> involved (in part because GHC has to compile haskell code, not just load
> and run a pre-existing object file).

Actually, we don't really have this for plugins either. It is true that
you can load a plugin independently of the imports of the module it is
compiling, and that you can setup an independent package database for
plugins, but the interfaces of the plugin are still all dumped in
the same interface database that GHC uses for compile-time interfaces
as well.  It's delicate and has caused bugs before 
(https://ghc.haskell.org/trac/ghc/ticket/10420)

Edward
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: Making (useful subsets of) bytecode portable between targets

[Moritz Angermann]

There is https://github.com/ghc-proposals/ghc-proposals

Sent from my iPhone

> On 25 Nov 2016, at 11:30 PM, Shea Levy  wrote:
> 
> Simon Marlow  writes:
> 
>> The right thing is to have a clean separation between runtime
>> imports and compile-time imports.  Perhaps we just annotate some imports to
>> say they aren't needed at compile-time for running the TH code.  but then
>> we also need compile-time vs. runtime build-depends in our .cabal files,
>> and so on.
> 
> Yes, I was just looking into this yesterday. We already have something
> similar for plugins, though of course the TH story is much more
> involved (in part because GHC has to compile haskell code, not just load
> and run a pre-existing object file).
> 
>> Its a big project, but ultimately we do have to tackle it, because it's the
>> right thing to do.  Anyone interested in working on this?  Maybe start a
>> new proposal to flesh out the details.
> 
> Yeah, I'm going to at least try. Don't know where formal proposals go,
> but what I've got so far (definitely subject to change!):
> 
> 1. Add a separate package database like we have for plugins for TH
>   imports, defaulting to the normal package database when empty
> 2. Use the TH package db for code to be *run*, but resolve all
>   reifications etc. against the normal target scope.
> 3. Add {-# TH #-}-annotated imports to specify modules to import
>   compile-time code from (if a module *also* has definitions to be
>   reified, it needs to be imported twice)
>   3a. If a module has any {-# TH #-} imports, enforce that *all*
>   compile-time executed code is pulled in via {-# TH #-} imports.
>   3b. Optionally add a warning for TH-using code that doesn't use
>   {-# TH #-}, as the first start of a migration path to enforcing
>   compiletime/runtime separation across the ecosystem.
>   3c. Not sure what the specific difficulties are that require the
>   staging restrictions, but possibly annotating same-module
>   definitions with {-# TH -} might make this easier?
> 4. Teach cabal how to add packages to the TH database
>   4a. Not sure how all this works, but probably this is the place to
>   e.g. request the non-profiled version of a package if ghc is
>   non-profiled and the target code is profiled
> 5. Modify the build process of the cross-compiler to build a stage-2
>   compiler that builds code for the target *and* host (at least, enough
>   of the host to build BCOs), and runs that host-targeted code in the
>   interpreter.
>   5a. This will require some way to distinguish
>   stage-2-as-target-native-compiler VS stage-2-as-hybrid-compiler.
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: Making (useful subsets of) bytecode portable between targets

[Shea Levy]

Simon Marlow  writes:

> The right thing is to have a clean separation between runtime
> imports and compile-time imports.  Perhaps we just annotate some imports to
> say they aren't needed at compile-time for running the TH code.  but then
> we also need compile-time vs. runtime build-depends in our .cabal files,
> and so on.

Yes, I was just looking into this yesterday. We already have something
similar for plugins, though of course the TH story is much more
involved (in part because GHC has to compile haskell code, not just load
and run a pre-existing object file).

> Its a big project, but ultimately we do have to tackle it, because it's the
> right thing to do.  Anyone interested in working on this?  Maybe start a
> new proposal to flesh out the details.

Yeah, I'm going to at least try. Don't know where formal proposals go,
but what I've got so far (definitely subject to change!):

1. Add a separate package database like we have for plugins for TH
   imports, defaulting to the normal package database when empty
2. Use the TH package db for code to be *run*, but resolve all
   reifications etc. against the normal target scope.
3. Add {-# TH #-}-annotated imports to specify modules to import
   compile-time code from (if a module *also* has definitions to be
   reified, it needs to be imported twice)
   3a. If a module has any {-# TH #-} imports, enforce that *all*
   compile-time executed code is pulled in via {-# TH #-} imports.
   3b. Optionally add a warning for TH-using code that doesn't use
   {-# TH #-}, as the first start of a migration path to enforcing
   compiletime/runtime separation across the ecosystem.
   3c. Not sure what the specific difficulties are that require the
   staging restrictions, but possibly annotating same-module
   definitions with {-# TH -} might make this easier?
4. Teach cabal how to add packages to the TH database
   4a. Not sure how all this works, but probably this is the place to
   e.g. request the non-profiled version of a package if ghc is
   non-profiled and the target code is profiled
5. Modify the build process of the cross-compiler to build a stage-2
   compiler that builds code for the target *and* host (at least, enough
   of the host to build BCOs), and runs that host-targeted code in the
   interpreter.
   5a. This will require some way to distinguish
   stage-2-as-target-native-compiler VS stage-2-as-hybrid-compiler.


signature.asc
Description: PGP signature
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: Making (useful subsets of) bytecode portable between targets

[Simon Marlow]

On 25 November 2016 at 07:23, Moritz Angermann 
wrote:

[snip]


> To get this back on topic, if we have a architecture independent
> interpretable bytecode,
> for ghc, could we sidestep the runner solution altogether and have TH for
> the target
> be evaluated on the host?  Is this what Shea initially wanted to go after?
>

Yes, but architecture-independent bytecode is the least of the problems.
Doing this properly is a really big change.

We basically have two worlds: first, the compile-time world. In this world,
we need all the packages and modules of the current package built for the
host platform. Secondly, we need the runtime world, with all the packages
and modules of the current package cross-compiled for the target platform.

When compiling a module that uses TH, we need to
- compile it as if we were compiling for the host platform, reading .hi
files from the host world
- run the TH code in the host world
- restart the compilation, using the .hi files from the target world, and
the results of running the splices

But even this isn't going to be enough.  What if your code imports some
modules that are only available on the runtime platform? iOS APIs, for
example.  The right thing is to have a clean separation between runtime
imports and compile-time imports.  Perhaps we just annotate some imports to
say they aren't needed at compile-time for running the TH code.  but then
we also need compile-time vs. runtime build-depends in our .cabal files,
and so on.

This is just off the top of my head, I'm sure there are more complexities I
haven't thought about.

Its a big project, but ultimately we do have to tackle it, because it's the
right thing to do.  Anyone interested in working on this?  Maybe start a
new proposal to flesh out the details.

Cheers
Simon



> cheers,
>  moritz
>
> > On Nov 25, 2016, at 2:38 PM, Manuel M T Chakravarty <
> c...@justtesting.org> wrote:
> >
> > Ok, I am not saying that it is technical impossible. I am saying that it
> is *impractical*.
> >
> > Imagine Travis CI needing to run stuff on my phone that is attached to
> my Mac (if we are lucky), which is behind NAT somewhere in Australia.
> >
> > Running stuff in the simulator during a build would be pretty awkward,
> but running it on the device is not practical.
> >
> > Manuel
> >
> > PS: BTW, shipping binary code to the device means it has to be code
> signed using a distribution profile of a registered developer. That is one
> thing if Xcode does all the magic behind the scenes, but do you really want
> to make that part of the GHC build process?
> >
> >> Edward Z. Yang :
> >>
> >> At least for Travis, you can generate a private key that only Travis
> >> has access to, and use this to authenticate access to the runner.
> >> See https://docs.travis-ci.com/user/encryption-keys/
> >>
> >> Edward
> >>
> >> Excerpts from Manuel M T Chakravarty's message of 2016-11-24 16:38:34
> +1100:
> >>> If you use Travis CI or such, do you really want to have a runner
> accessible from an arbitrary host on the Internet?
> >>>
>  Moritz Angermann :
> 
>  It's certainly far from ideal, but for CI, what obstacles are there
> besides needing a runner accessible from cross compiling machine?
> 
>  E.g. Start the runner app on an iPhone plugged in into a USB power
> source and leave it there?
> 
>  Sent from my iPhone
> 
> > On 24 Nov 2016, at 12:42 PM, Manuel M T Chakravarty <
> c...@justtesting.org> wrote:
> >
> > Sorry, but I don’t think running on the device is practical. How do
> you want to do CI, for example?
> >
> > Manuel
> >
> >> Moritz Angermann :
> >>
> >>
> >>> On Nov 23, 2016, at 7:50 PM, Simon Marlow 
> wrote:
> >>>
> >>> […]
> >>>
> >>> My question would be: are you *sure* you can't run target code at
> compile time?  Not even with an iphone simulator?
> >>
> >> This should be possible. However for proper development one would
> need to run on the
> >> device (iPhone, iPad, …) for armv7 or arm64, as the Simulator is
> i386 or x86_64.
> >>
> >> There is a bit of additional engineering required here to get the
> shipping of
> >> code from ghc to the runner on the target required (e.g. via
> network).  As executing
> >> and controlling applications on the actual hardware is limited, I
> guess a custom
> >> ghc-runner application would have to be manually started on the
> device, which could
> >> trivially be discovered using bonjour/zeroconf (or just giving ghc
> the host:port information).
> >>
> >> In general though, the runner does not have to obey all the
> restrictions apple puts
> >> onto app-store distributed apps, as I expect that everyone could
> build and install
> >> the runner themselves when intending to do iOS development with ghc.
> >>
> >> cheers,
> >> moritz
> 

Re: Making (useful subsets of) bytecode portable between targets

[Manuel M T Chakravarty]

I totally agree that GHC with TH is crippled and that this is a major 
restriction off the cross-compilation story. All I am saying is that I see a 
device runner (and to a degree a simulator runner) not as a solution to this 
*important* problem. Architecture independent interpretable byte code seems a 
much more attractive avenue for me. (Sorry if I didn’t make this clear.)

Manuel

> Moritz Angermann :
> 
> Aren’t we taking this a bit too far off topic? I feared as much when I wrote 
> my
> initial reply. Please excuse.
> 
> I agree that ghc + runner is not an optimal, and maybe even for some tasks 
> (iOS)
> a pretty convoluted solution.
> 
> This is only if we follow the proven solution to TH that luite in ghcjs 
> pioneered,
> and which later found it’s way into ghc through iserv.  If someone proposes a
> solution to TH that does not require a runner and allows the TH to be fully
> evaluated on the host with no need to evaluate on the target for cross 
> compilation,
> that would be great!
> 
> If the runner would just require the same architecture, maybe qemu would be a 
> solution
> that would not require a device running?  Then again I’m not sure how that 
> would
> work with TH that directly or indirectly accesses libraries only available on 
> iOS for
> example.
> 
> Please don’t get me wrong. IMO ghc without TH is quite crippled, and 
> therefore so is a 
> cross compiling ghc. From the solutions I saw to this problem (zeroth, 
> evil-splicer, and
> the ghcjs runner approach), the ghcjs runner approach, seems to me at least 
> as the most
> promising, that would work for the largest subset of TH.
> 
> To get this back on topic, if we have a architecture independent 
> interpretable bytecode,
> for ghc, could we sidestep the runner solution altogether and have TH for the 
> target
> be evaluated on the host?  Is this what Shea initially wanted to go after?
> 
> cheers,
> moritz
> 
>> On Nov 25, 2016, at 2:38 PM, Manuel M T Chakravarty  
>> wrote:
>> 
>> Ok, I am not saying that it is technical impossible. I am saying that it is 
>> *impractical*. 
>> 
>> Imagine Travis CI needing to run stuff on my phone that is attached to my 
>> Mac (if we are lucky), which is behind NAT somewhere in Australia. 
>> 
>> Running stuff in the simulator during a build would be pretty awkward, but 
>> running it on the device is not practical.
>> 
>> Manuel
>> 
>> PS: BTW, shipping binary code to the device means it has to be code signed 
>> using a distribution profile of a registered developer. That is one thing if 
>> Xcode does all the magic behind the scenes, but do you really want to make 
>> that part of the GHC build process?
>> 
>>> Edward Z. Yang :
>>> 
>>> At least for Travis, you can generate a private key that only Travis
>>> has access to, and use this to authenticate access to the runner.
>>> See https://docs.travis-ci.com/user/encryption-keys/
>>> 
>>> Edward
>>> 
>>> Excerpts from Manuel M T Chakravarty's message of 2016-11-24 16:38:34 +1100:
 If you use Travis CI or such, do you really want to have a runner 
 accessible from an arbitrary host on the Internet?
 
> Moritz Angermann :
> 
> It's certainly far from ideal, but for CI, what obstacles are there 
> besides needing a runner accessible from cross compiling machine?
> 
> E.g. Start the runner app on an iPhone plugged in into a USB power source 
> and leave it there?
> 
> Sent from my iPhone
> 
>> On 24 Nov 2016, at 12:42 PM, Manuel M T Chakravarty 
>>  wrote:
>> 
>> Sorry, but I don’t think running on the device is practical. How do you 
>> want to do CI, for example?
>> 
>> Manuel
>> 
>>> Moritz Angermann :
>>> 
>>> 
 On Nov 23, 2016, at 7:50 PM, Simon Marlow  wrote:
 
 […]
 
 My question would be: are you *sure* you can't run target code at 
 compile time?  Not even with an iphone simulator?
>>> 
>>> This should be possible. However for proper development one would need 
>>> to run on the
>>> device (iPhone, iPad, …) for armv7 or arm64, as the Simulator is i386 
>>> or x86_64.
>>> 
>>> There is a bit of additional engineering required here to get the 
>>> shipping of
>>> code from ghc to the runner on the target required (e.g. via network).  
>>> As executing
>>> and controlling applications on the actual hardware is limited, I guess 
>>> a custom
>>> ghc-runner application would have to be manually started on the device, 
>>> which could
>>> trivially be discovered using bonjour/zeroconf (or just giving ghc the 
>>> host:port information).
>>> 
>>> In general though, the runner does not have to obey all the 
>>> restrictions apple puts
>>> onto app-store distributed apps, 

Re: Making (useful subsets of) bytecode portable between targets

[Moritz Angermann]

Aren’t we taking this a bit too far off topic? I feared as much when I wrote my
initial reply. Please excuse.

I agree that ghc + runner is not an optimal, and maybe even for some tasks (iOS)
a pretty convoluted solution.

This is only if we follow the proven solution to TH that luite in ghcjs 
pioneered,
and which later found it’s way into ghc through iserv.  If someone proposes a
solution to TH that does not require a runner and allows the TH to be fully
evaluated on the host with no need to evaluate on the target for cross 
compilation,
that would be great!

If the runner would just require the same architecture, maybe qemu would be a 
solution
that would not require a device running?  Then again I’m not sure how that would
work with TH that directly or indirectly accesses libraries only available on 
iOS for
example.

Please don’t get me wrong. IMO ghc without TH is quite crippled, and therefore 
so is a 
cross compiling ghc. From the solutions I saw to this problem (zeroth, 
evil-splicer, and
the ghcjs runner approach), the ghcjs runner approach, seems to me at least as 
the most
promising, that would work for the largest subset of TH.

To get this back on topic, if we have a architecture independent interpretable 
bytecode,
for ghc, could we sidestep the runner solution altogether and have TH for the 
target
be evaluated on the host?  Is this what Shea initially wanted to go after?

cheers,
 moritz

> On Nov 25, 2016, at 2:38 PM, Manuel M T Chakravarty  
> wrote:
> 
> Ok, I am not saying that it is technical impossible. I am saying that it is 
> *impractical*. 
> 
> Imagine Travis CI needing to run stuff on my phone that is attached to my Mac 
> (if we are lucky), which is behind NAT somewhere in Australia. 
> 
> Running stuff in the simulator during a build would be pretty awkward, but 
> running it on the device is not practical.
> 
> Manuel
> 
> PS: BTW, shipping binary code to the device means it has to be code signed 
> using a distribution profile of a registered developer. That is one thing if 
> Xcode does all the magic behind the scenes, but do you really want to make 
> that part of the GHC build process?
> 
>> Edward Z. Yang :
>> 
>> At least for Travis, you can generate a private key that only Travis
>> has access to, and use this to authenticate access to the runner.
>> See https://docs.travis-ci.com/user/encryption-keys/
>> 
>> Edward
>> 
>> Excerpts from Manuel M T Chakravarty's message of 2016-11-24 16:38:34 +1100:
>>> If you use Travis CI or such, do you really want to have a runner 
>>> accessible from an arbitrary host on the Internet?
>>> 
 Moritz Angermann :
 
 It's certainly far from ideal, but for CI, what obstacles are there 
 besides needing a runner accessible from cross compiling machine?
 
 E.g. Start the runner app on an iPhone plugged in into a USB power source 
 and leave it there?
 
 Sent from my iPhone
 
> On 24 Nov 2016, at 12:42 PM, Manuel M T Chakravarty 
>  wrote:
> 
> Sorry, but I don’t think running on the device is practical. How do you 
> want to do CI, for example?
> 
> Manuel
> 
>> Moritz Angermann :
>> 
>> 
>>> On Nov 23, 2016, at 7:50 PM, Simon Marlow  wrote:
>>> 
>>> […]
>>> 
>>> My question would be: are you *sure* you can't run target code at 
>>> compile time?  Not even with an iphone simulator?
>> 
>> This should be possible. However for proper development one would need 
>> to run on the
>> device (iPhone, iPad, …) for armv7 or arm64, as the Simulator is i386 or 
>> x86_64.
>> 
>> There is a bit of additional engineering required here to get the 
>> shipping of
>> code from ghc to the runner on the target required (e.g. via network).  
>> As executing
>> and controlling applications on the actual hardware is limited, I guess 
>> a custom
>> ghc-runner application would have to be manually started on the device, 
>> which could
>> trivially be discovered using bonjour/zeroconf (or just giving ghc the 
>> host:port information).
>> 
>> In general though, the runner does not have to obey all the restrictions 
>> apple puts
>> onto app-store distributed apps, as I expect that everyone could build 
>> and install
>> the runner themselves when intending to do iOS development with ghc.
>> 
>> cheers,
>> moritz
>> ___
>> ghc-devs mailing list
>> ghc-devs@haskell.org
>> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
> 
 
>>> 
>> ___
>> ghc-devs mailing list
>> ghc-devs@haskell.org
>> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
> 
> ___
> ghc-devs 

Re: Making (useful subsets of) bytecode portable between targets

[Manuel M T Chakravarty]

Ok, I am not saying that it is technical impossible. I am saying that it is 
*impractical*. 

Imagine Travis CI needing to run stuff on my phone that is attached to my Mac 
(if we are lucky), which is behind NAT somewhere in Australia. 

Running stuff in the simulator during a build would be pretty awkward, but 
running it on the device is not practical.

Manuel

PS: BTW, shipping binary code to the device means it has to be code signed 
using a distribution profile of a registered developer. That is one thing if 
Xcode does all the magic behind the scenes, but do you really want to make that 
part of the GHC build process?

> Edward Z. Yang :
> 
> At least for Travis, you can generate a private key that only Travis
> has access to, and use this to authenticate access to the runner.
> See https://docs.travis-ci.com/user/encryption-keys/
> 
> Edward
> 
> Excerpts from Manuel M T Chakravarty's message of 2016-11-24 16:38:34 +1100:
>> If you use Travis CI or such, do you really want to have a runner accessible 
>> from an arbitrary host on the Internet?
>> 
>>> Moritz Angermann :
>>> 
>>> It's certainly far from ideal, but for CI, what obstacles are there besides 
>>> needing a runner accessible from cross compiling machine?
>>> 
>>> E.g. Start the runner app on an iPhone plugged in into a USB power source 
>>> and leave it there?
>>> 
>>> Sent from my iPhone
>>> 
 On 24 Nov 2016, at 12:42 PM, Manuel M T Chakravarty  
 wrote:
 
 Sorry, but I don’t think running on the device is practical. How do you 
 want to do CI, for example?
 
 Manuel
 
> Moritz Angermann :
> 
> 
>> On Nov 23, 2016, at 7:50 PM, Simon Marlow  wrote:
>> 
>> […]
>> 
>> My question would be: are you *sure* you can't run target code at 
>> compile time?  Not even with an iphone simulator?
> 
> This should be possible. However for proper development one would need to 
> run on the
> device (iPhone, iPad, …) for armv7 or arm64, as the Simulator is i386 or 
> x86_64.
> 
> There is a bit of additional engineering required here to get the 
> shipping of
> code from ghc to the runner on the target required (e.g. via network).  
> As executing
> and controlling applications on the actual hardware is limited, I guess a 
> custom
> ghc-runner application would have to be manually started on the device, 
> which could
> trivially be discovered using bonjour/zeroconf (or just giving ghc the 
> host:port information).
> 
> In general though, the runner does not have to obey all the restrictions 
> apple puts
> onto app-store distributed apps, as I expect that everyone could build 
> and install
> the runner themselves when intending to do iOS development with ghc.
> 
> cheers,
> moritz
> ___
> ghc-devs mailing list
> ghc-devs@haskell.org
> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
 
>>> 
>> 
> ___
> ghc-devs mailing list
> ghc-devs@haskell.org
> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: Making (useful subsets of) bytecode portable between targets

[Edward Z. Yang]

Excerpts from Ben Gamari's message of 2016-11-21 16:06:14 -0500:
> > 3. Teach cabal to compile dependencies and modules for the build and
> >target when cross-compiling and TH is used
> 
> You'll need to speak with the Cabal folks about this. However, it's
> probably safe to ignore this for now; there's a lot of work to done
> before you'll be in a position to start thinking about this.

Some related work that has been done for just compiling Setup scripts
with the host system: https://github.com/haskell/cabal/issues/1493
and https://github.com/haskell/cabal/issues/4047 is where we have
been going with it.

John Ericson is in charge of the effort, I've CC'ed him to this list.

Edward
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: Making (useful subsets of) bytecode portable between targets

[Manuel M T Chakravarty]

If you use Travis CI or such, do you really want to have a runner accessible 
from an arbitrary host on the Internet?

> Moritz Angermann :
> 
> It's certainly far from ideal, but for CI, what obstacles are there besides 
> needing a runner accessible from cross compiling machine?
> 
> E.g. Start the runner app on an iPhone plugged in into a USB power source and 
> leave it there?
> 
> Sent from my iPhone
> 
>> On 24 Nov 2016, at 12:42 PM, Manuel M T Chakravarty  
>> wrote:
>> 
>> Sorry, but I don’t think running on the device is practical. How do you want 
>> to do CI, for example?
>> 
>> Manuel
>> 
>>> Moritz Angermann :
>>> 
>>> 
 On Nov 23, 2016, at 7:50 PM, Simon Marlow  wrote:
 
 […]
 
 My question would be: are you *sure* you can't run target code at compile 
 time?  Not even with an iphone simulator?
>>> 
>>> This should be possible. However for proper development one would need to 
>>> run on the
>>> device (iPhone, iPad, …) for armv7 or arm64, as the Simulator is i386 or 
>>> x86_64.
>>> 
>>> There is a bit of additional engineering required here to get the shipping 
>>> of
>>> code from ghc to the runner on the target required (e.g. via network).  As 
>>> executing
>>> and controlling applications on the actual hardware is limited, I guess a 
>>> custom
>>> ghc-runner application would have to be manually started on the device, 
>>> which could
>>> trivially be discovered using bonjour/zeroconf (or just giving ghc the 
>>> host:port information).
>>> 
>>> In general though, the runner does not have to obey all the restrictions 
>>> apple puts
>>> onto app-store distributed apps, as I expect that everyone could build and 
>>> install
>>> the runner themselves when intending to do iOS development with ghc.
>>> 
>>> cheers,
>>> moritz
>>> ___
>>> ghc-devs mailing list
>>> ghc-devs@haskell.org
>>> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
>> 
> 

___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: Making (useful subsets of) bytecode portable between targets

[Moritz Angermann]

Hi,

just to also note: that this completely ignores any host / target IO actions 
that
TH might want to run.

cheers,
 moritz

> On Nov 24, 2016, at 12:51 PM, Moritz Angermann  wrote:
> 
> It's certainly far from ideal, but for CI, what obstacles are there besides 
> needing a runner accessible from cross compiling machine?
> 
> E.g. Start the runner app on an iPhone plugged in into a USB power source and 
> leave it there?
> 
> Sent from my iPhone
> 
>> On 24 Nov 2016, at 12:42 PM, Manuel M T Chakravarty  
>> wrote:
>> 
>> Sorry, but I don’t think running on the device is practical. How do you want 
>> to do CI, for example?
>> 
>> Manuel
>> 
>>> Moritz Angermann :
>>> 
>>> 
 On Nov 23, 2016, at 7:50 PM, Simon Marlow  wrote:
 
 […]
 
 My question would be: are you *sure* you can't run target code at compile 
 time?  Not even with an iphone simulator?
>>> 
>>> This should be possible. However for proper development one would need to 
>>> run on the
>>> device (iPhone, iPad, …) for armv7 or arm64, as the Simulator is i386 or 
>>> x86_64.
>>> 
>>> There is a bit of additional engineering required here to get the shipping 
>>> of
>>> code from ghc to the runner on the target required (e.g. via network).  As 
>>> executing
>>> and controlling applications on the actual hardware is limited, I guess a 
>>> custom
>>> ghc-runner application would have to be manually started on the device, 
>>> which could
>>> trivially be discovered using bonjour/zeroconf (or just giving ghc the 
>>> host:port information).
>>> 
>>> In general though, the runner does not have to obey all the restrictions 
>>> apple puts
>>> onto app-store distributed apps, as I expect that everyone could build and 
>>> install
>>> the runner themselves when intending to do iOS development with ghc.
>>> 
>>> cheers,
>>> moritz
>>> ___
>>> ghc-devs mailing list
>>> ghc-devs@haskell.org
>>> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
>> 
> 
> ___
> ghc-devs mailing list
> ghc-devs@haskell.org
> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: Making (useful subsets of) bytecode portable between targets

[Moritz Angermann]

It's certainly far from ideal, but for CI, what obstacles are there besides 
needing a runner accessible from cross compiling machine?

E.g. Start the runner app on an iPhone plugged in into a USB power source and 
leave it there?

Sent from my iPhone

> On 24 Nov 2016, at 12:42 PM, Manuel M T Chakravarty  
> wrote:
> 
> Sorry, but I don’t think running on the device is practical. How do you want 
> to do CI, for example?
> 
> Manuel
> 
>> Moritz Angermann :
>> 
>> 
>>> On Nov 23, 2016, at 7:50 PM, Simon Marlow  wrote:
>>> 
>>> […]
>>> 
>>> My question would be: are you *sure* you can't run target code at compile 
>>> time?  Not even with an iphone simulator?
>> 
>> This should be possible. However for proper development one would need to 
>> run on the
>> device (iPhone, iPad, …) for armv7 or arm64, as the Simulator is i386 or 
>> x86_64.
>> 
>> There is a bit of additional engineering required here to get the shipping of
>> code from ghc to the runner on the target required (e.g. via network).  As 
>> executing
>> and controlling applications on the actual hardware is limited, I guess a 
>> custom
>> ghc-runner application would have to be manually started on the device, 
>> which could
>> trivially be discovered using bonjour/zeroconf (or just giving ghc the 
>> host:port information).
>> 
>> In general though, the runner does not have to obey all the restrictions 
>> apple puts
>> onto app-store distributed apps, as I expect that everyone could build and 
>> install
>> the runner themselves when intending to do iOS development with ghc.
>> 
>> cheers,
>> moritz
>> ___
>> ghc-devs mailing list
>> ghc-devs@haskell.org
>> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
> 

___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: Making (useful subsets of) bytecode portable between targets

[Manuel M T Chakravarty]

Sorry, but I don’t think running on the device is practical. How do you want to 
do CI, for example?

Manuel

> Moritz Angermann :
> 
> 
>> On Nov 23, 2016, at 7:50 PM, Simon Marlow  wrote:
>> 
>> […]
>> 
>> My question would be: are you *sure* you can't run target code at compile 
>> time?  Not even with an iphone simulator?
> 
> This should be possible. However for proper development one would need to run 
> on the
> device (iPhone, iPad, …) for armv7 or arm64, as the Simulator is i386 or 
> x86_64.
> 
> There is a bit of additional engineering required here to get the shipping of
> code from ghc to the runner on the target required (e.g. via network).  As 
> executing
> and controlling applications on the actual hardware is limited, I guess a 
> custom
> ghc-runner application would have to be manually started on the device, which 
> could
> trivially be discovered using bonjour/zeroconf (or just giving ghc the 
> host:port information).
> 
> In general though, the runner does not have to obey all the restrictions 
> apple puts
> onto app-store distributed apps, as I expect that everyone could build and 
> install
> the runner themselves when intending to do iOS development with ghc.
> 
> cheers,
> moritz
> ___
> ghc-devs mailing list
> ghc-devs@haskell.org
> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: Making (useful subsets of) bytecode portable between targets

[Ben Gamari]

CCing Luite who has done this sort of thing in the context of GHCJS.


Shea Levy  writes:

> Hi all,
>
> I'm interested in implementing a general solution for TH during
> cross-compilation, which if my naive lack-of-understanding is correct
> will broadly involve the following three tasks:
>
> 1. Make the generation of byte code, or at least the subset needed for
>useful TH use, target-independent

The bytecode machine itself (defined by ByteCodeInstr) is reasonably
target-independent. However, I suspect primops will be
problematic since they can appear in BCOs and are quite target dependent
(namely due to word size). I honestly don't know how much of a problem
this will be in practice; I tried loading a few example programs into
ghci -ddump-bcos and didn't see any PUSH_PRIMOP instructions, so perhaps
this won't be as problematic as I expect; it is worrisome though.

In general it's hard to identify "the subset needed for useful TH use"
but the fact that I can load the gitrev package (a fairly non-trivial
use of TH) into GHCi with no apparently platform dependent BCOs being
produced is promising.

> 2. Teach the external-interpreter to launch a build-native iserv when
>cross-compiling

For this you'll want to have a careful look through the Binary instances
for the things that are sent over the wire to iserv. You'll find these
occur in libraries/ghci/*.hs.

The first thing to look for is places where the system word-size may
leak into the serialized representation (e.g. instances where Ints and
Words are being used). Many of these can probably just be narrowed into
32-bit representations (e.g. in ResolvedBCO, where I think it's fair to
assume that four-billion bytecode instructions is an unreasonable BCO
size).

Don't forget to consider the platform dependence of the "primitive"
encodings like the Binary instance for ByteString [1]; you'll need to
take care to replace these with platform-independent encodings.

[1] 
https://hackage.haskell.org/package/binary-0.8.4.1/docs/src/Data.Binary.Class.html#line-611

> 3. Teach cabal to compile dependencies and modules for the build and
>target when cross-compiling and TH is used

You'll need to speak with the Cabal folks about this. However, it's
probably safe to ignore this for now; there's a lot of work to done
before you'll be in a position to start thinking about this.


> Of course, due to the generality of TH there will be code you can write
> that would be different in this approach from what you would get with a
> fully native compilation (e.g. due to GHC conditional compilation in the
> TH functions or FFI that does runtime host introspection), but since
> requiring a target device at build time is in many cases impractical (do
> you want to hook up an iPhone to your build farm?) and in some cases
> probably impossible (for targets without the resources to run a full GHC
> even if they can run GHC-compiled code) I think this is a reasonable
> restriction to require.
>
> My questions to the list are:
>
> * Is 1 above a pipe dream, keeping in mind that we are assuming the
>   availability of build-native versions of all dependencies and
>   already-compiled modules?

This sounds quite feasible to me.

> * Any pointers for getting started with 1?

See above. The first thing to do is to rid the Binary instances of
platform dependent encodings. This will be a bit of a game of
whack-a-mole, but it shouldn't be that hard.

Don't hesitate to ask if you get stuck!

Cheers,

- Ben



signature.asc
Description: PGP signature
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Making (useful subsets of) bytecode portable between targets

[Shea Levy]

Hi all,

I'm interested in implementing a general solution for TH during
cross-compilation, which if my naive lack-of-understanding is correct
will broadly involve the following three tasks:

1. Make the generation of byte code, or at least the subset needed for
   useful TH use, target-independent
2. Teach the external-interpreter to launch a build-native iserv when
   cross-compiling
3. Teach cabal to compile dependencies and modules for the build and
   target when cross-compiling and TH is used

Of course, due to the generality of TH there will be code you can write
that would be different in this approach from what you would get with a
fully native compilation (e.g. due to GHC conditional compilation in the
TH functions or FFI that does runtime host introspection), but since
requiring a target device at build time is in many cases impractical (do
you want to hook up an iPhone to your build farm?) and in some cases
probably impossible (for targets without the resources to run a full GHC
even if they can run GHC-compiled code) I think this is a reasonable
restriction to require.

My questions to the list are:

* Is 1 above a pipe dream, keeping in mind that we are assuming the
  availability of build-native versions of all dependencies and
  already-compiled modules?
* Any pointers for getting started with 1?
* Anything I'm missing?

Thanks,
Shea


signature.asc
Description: PGP signature
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs