Re: Emscripten SDK and WASI filesystem access

[scot...@gmail.com]

>From their zulipchat:

> 

Dan Gohman: The original reason is that cloudlibc did it that way, and the 
code comes from there.

Dan Gohman: From a libc maintainer perspective, these values are 
implementation details. musl only has a single value because musl only 
supports Linux, where O_RDONLY always has the same value. But across other 
platforms, it can have different values.

Dan Gohman: wasi-libc is not ABI-compatible with Emscripten in numerous 
ways, and no one has ever volunteered to do the work to make it be so.

Dan Gohman: That said, from a practical perspective, some C / POSIX 
constants are sufficiently "well known" that we do make efforts to align 
their values with the well-known values. This is somewhat subjective, so 
whether O_RDONLY should qualify as having a well-known value would come 
down to how it's used in practice, how you're using it.
 In summary, while it would be easier from my perspective if the libcs 
where ABI compatible, its perhaps not a reasonable expectation, and the 
"right" thing would not to use libc for things that WASI supplies, but go 
straight to the WASI calls.
On Tuesday, March 8, 2022 at 4:54:29 PM UTC-5 scot...@gmail.com wrote:

> I did a little more digging and emscripten is the same as musl, and 
> wasi-libc is different.  I will ask  wasi-libc why they didn't use the 
> original constants as there is nothing in their git history about it - it's 
> always been different
>
> On Tuesday, March 1, 2022 at 11:32:09 AM UTC-5 scot...@gmail.com wrote:
>
>> For clarity, here is the linke to `O_RDONLY` for wasi:
>>
>> https://github.com/WebAssembly/wasi-libc/blob/ad5133410f66b93a2381db5b542aad5e0964db96/libc-bottom-half/headers/public/__header_fcntl.h#L20
>>
>> On Saturday, February 26, 2022 at 5:21:11 PM UTC-5 scot...@gmail.com 
>> wrote:
>>
>>> Its the constants in fcntl.h / in wasi: __header_fcntl.h and api.h :
>>>
>>> ```
>>>
>>> #define O_WASI_RDONLY (0x0400) 
>>> #define O_WASI_WRONLY (0x1000) 
>>> #define O_WASI_RDWR (O_WASI_RDONLY | O_WASI_WRONLY) 
>>>
>>>
>>> typedef uint16_t __wasi_oflags_t; 
>>> typedef uint16_t __wasi_fdflags_t; 
>>>
>>>
>>> #define WASI_OFLAGS_CREAT ((__wasi_oflags_t)(1 << 0)) 
>>> #define WASI_OFLAGS_EXCL ((__wasi_oflags_t)(1 << 2)) 
>>> #define WASI_OFLAGS_TRUNC ((__wasi_oflags_t)(1 << 3)) 
>>> #define WASI_FDFLAGS_SYNC ((__wasi_fdflags_t)(1 << 4)) 
>>>
>>>
>>> #define O_WASI_CREAT (WASI_OFLAGS_CREAT << 12) 
>>> #define O_WASI_EXCL (WASI_OFLAGS_EXCL << 12) 
>>> #define O_WASI_TRUNC (WASI_OFLAGS_TRUNC << 12) 
>>> #define O_WASI_SYNC WASI_FDFLAGS_SYNC #define O_WASI_CLOEXEC 0 
>>> ```
>>> Remove the WASI_ from these and you see that in emscripten they are 
>>> different.  E.g. for O_RDONLY in fcntl.h under system/lib/libc/musl/include 
>>> it is:
>>> ```
>>> #define O_RDONLY  00
>>> ```
>>> But For WASI I "overwrite" it with 0x0400.  It's a hack (
>>> https://github.com/dotnet/runtimelab/pull/1850/files#diff-5d86cce7b76e1bece0aec26ee2bc4830503046428aecbbdefbf845832573020f),
>>>  
>>> and I do that switch with a "special" options flag.   
>>>
>>> Interesting that these constants are different between emscripten and 
>>> WASI as both are musl.  There must be some history there I suppose.
>>> On Thursday, February 10, 2022 at 12:34:43 PM UTC-5 alon...@gmail.com 
>>> wrote:
>>>
 I see... yeah, mixing and matching object files from different 
 toolchains is not guaranteed to work. They need C ABI compatibility, I'm 
 afraid.

 It is a little sad to need separate runtimes, though, so I hope we can 
 find a way to avoid that for you. What is the specific code that would be 
 different between the two runtimes?

 On Thu, Feb 10, 2022 at 5:23 AM yowl yowlxx  wrote:

> " (Of course, header constants *can* be an issue if you link wasm 
> object files together that were built using different SDKs or even 
> different versions of the same SDK.)"
>
> Yes, that is what I naively tried to do.  The CoreCLR runtime is 
> compiled using emscripten's SDK, but I wanted to link it with the WASI 
> SDK 
> libc++ to get file open support.  What I'll probably have to do in the 
> long 
> run is have two CoreCLR runtimes, one for Emscripten and one for WASI SDK.
>
> On Mon, 7 Feb 2022 at 14:39, Alon Zakai  wrote:
>
>> Header constants like O_CREAT should not be a problem between wasm 
>> and the host. WASI defines a very clear API between wasm and the 
>> runtime, 
>> and those constants are not part of it. That is, "wasi-sdk host" is a 
>> confusing way to put it: there are VMs with WASI support, but they are 
>> not 
>> tied to the wasi-sdk toolchain, which is just one toolchain that emits 
>> WASI.
>>
>> (Of course, header constants *can* be an issue if you link wasm 
>> object files together that were built using different SDKs or even 
>> different versions of the same SDK.)
>>
>> Emscripten could 

Re: Emscripten SDK and WASI filesystem access

[scot...@gmail.com]

I did a little more digging and emscripten is the same as musl, and 
wasi-libc is different.  I will ask  wasi-libc why they didn't use the 
original constants as there is nothing in their git history about it - it's 
always been different

On Tuesday, March 1, 2022 at 11:32:09 AM UTC-5 scot...@gmail.com wrote:

> For clarity, here is the linke to `O_RDONLY` for wasi:
>
> https://github.com/WebAssembly/wasi-libc/blob/ad5133410f66b93a2381db5b542aad5e0964db96/libc-bottom-half/headers/public/__header_fcntl.h#L20
>
> On Saturday, February 26, 2022 at 5:21:11 PM UTC-5 scot...@gmail.com 
> wrote:
>
>> Its the constants in fcntl.h / in wasi: __header_fcntl.h and api.h :
>>
>> ```
>>
>> #define O_WASI_RDONLY (0x0400) 
>> #define O_WASI_WRONLY (0x1000) 
>> #define O_WASI_RDWR (O_WASI_RDONLY | O_WASI_WRONLY) 
>>
>>
>> typedef uint16_t __wasi_oflags_t; 
>> typedef uint16_t __wasi_fdflags_t; 
>>
>>
>> #define WASI_OFLAGS_CREAT ((__wasi_oflags_t)(1 << 0)) 
>> #define WASI_OFLAGS_EXCL ((__wasi_oflags_t)(1 << 2)) 
>> #define WASI_OFLAGS_TRUNC ((__wasi_oflags_t)(1 << 3)) 
>> #define WASI_FDFLAGS_SYNC ((__wasi_fdflags_t)(1 << 4)) 
>>
>>
>> #define O_WASI_CREAT (WASI_OFLAGS_CREAT << 12) 
>> #define O_WASI_EXCL (WASI_OFLAGS_EXCL << 12) 
>> #define O_WASI_TRUNC (WASI_OFLAGS_TRUNC << 12) 
>> #define O_WASI_SYNC WASI_FDFLAGS_SYNC #define O_WASI_CLOEXEC 0 
>> ```
>> Remove the WASI_ from these and you see that in emscripten they are 
>> different.  E.g. for O_RDONLY in fcntl.h under system/lib/libc/musl/include 
>> it is:
>> ```
>> #define O_RDONLY  00
>> ```
>> But For WASI I "overwrite" it with 0x0400.  It's a hack (
>> https://github.com/dotnet/runtimelab/pull/1850/files#diff-5d86cce7b76e1bece0aec26ee2bc4830503046428aecbbdefbf845832573020f),
>>  
>> and I do that switch with a "special" options flag.   
>>
>> Interesting that these constants are different between emscripten and 
>> WASI as both are musl.  There must be some history there I suppose.
>> On Thursday, February 10, 2022 at 12:34:43 PM UTC-5 alon...@gmail.com 
>> wrote:
>>
>>> I see... yeah, mixing and matching object files from different 
>>> toolchains is not guaranteed to work. They need C ABI compatibility, I'm 
>>> afraid.
>>>
>>> It is a little sad to need separate runtimes, though, so I hope we can 
>>> find a way to avoid that for you. What is the specific code that would be 
>>> different between the two runtimes?
>>>
>>> On Thu, Feb 10, 2022 at 5:23 AM yowl yowlxx  wrote:
>>>
 " (Of course, header constants *can* be an issue if you link wasm 
 object files together that were built using different SDKs or even 
 different versions of the same SDK.)"

 Yes, that is what I naively tried to do.  The CoreCLR runtime is 
 compiled using emscripten's SDK, but I wanted to link it with the WASI SDK 
 libc++ to get file open support.  What I'll probably have to do in the 
 long 
 run is have two CoreCLR runtimes, one for Emscripten and one for WASI SDK.

 On Mon, 7 Feb 2022 at 14:39, Alon Zakai  wrote:

> Header constants like O_CREAT should not be a problem between wasm and 
> the host. WASI defines a very clear API between wasm and the runtime, and 
> those constants are not part of it. That is, "wasi-sdk host" is a 
> confusing 
> way to put it: there are VMs with WASI support, but they are not tied to 
> the wasi-sdk toolchain, which is just one toolchain that emits WASI.
>
> (Of course, header constants *can* be an issue if you link wasm object 
> files together that were built using different SDKs or even different 
> versions of the same SDK.)
>
> Emscripten could add more complete WASI support, if that were useful - 
> it's just a matter of implementing the APIs. PRs would be welcome!
>
>
>
> On Sat, Feb 5, 2022 at 9:59 AM scot...@gmail.com  
> wrote:
>
>> My own observations mixing Emscripten and WASI-SDK are that some 
>> constants, e.g. `O_CREAT` are different in Emscripten (0x64) to Wasi-SDK 
>> (0x1000).  Allowing emscripten to build something that would run 
>> under 
>> a wasi-sdk host, like wasmtime, is tricky - I think.
>>
>> On Friday, December 17, 2021 at 8:01:14 AM UTC-5 Floh wrote:
>>
>>> Ah alright, that toolchain file looks a lot simpler (it's under 
>>> "share/cmake" in the SDK download). I'll use that as base for my own 
>>> cmake 
>>> toolchain file instead. Thanks!
>>>
>>> On Thursday, 16 December 2021 at 21:18:46 UTC+1 s...@google.com 
>>> wrote:
>>>
 On Thu, Dec 16, 2021 at 5:53 AM Floh  wrote:

> > Unless you plan to run those tools on the web then I think 
> wasi-sdk is most likely the way to go.   How do you plan to run the 
> binaries BTW? 
>
> Currently I'm using wasmtime.
>
> I was actually successful to build my shader compiler through 
> wasi-sdk, 

Re: Emscripten SDK and WASI filesystem access

[scot...@gmail.com]

For clarity, here is the linke to `O_RDONLY` for wasi:
https://github.com/WebAssembly/wasi-libc/blob/ad5133410f66b93a2381db5b542aad5e0964db96/libc-bottom-half/headers/public/__header_fcntl.h#L20

On Saturday, February 26, 2022 at 5:21:11 PM UTC-5 scot...@gmail.com wrote:

> Its the constants in fcntl.h / in wasi: __header_fcntl.h and api.h :
>
> ```
>
> #define O_WASI_RDONLY (0x0400) 
> #define O_WASI_WRONLY (0x1000) 
> #define O_WASI_RDWR (O_WASI_RDONLY | O_WASI_WRONLY) 
>
>
> typedef uint16_t __wasi_oflags_t; 
> typedef uint16_t __wasi_fdflags_t; 
>
>
> #define WASI_OFLAGS_CREAT ((__wasi_oflags_t)(1 << 0)) 
> #define WASI_OFLAGS_EXCL ((__wasi_oflags_t)(1 << 2)) 
> #define WASI_OFLAGS_TRUNC ((__wasi_oflags_t)(1 << 3)) 
> #define WASI_FDFLAGS_SYNC ((__wasi_fdflags_t)(1 << 4)) 
>
>
> #define O_WASI_CREAT (WASI_OFLAGS_CREAT << 12) 
> #define O_WASI_EXCL (WASI_OFLAGS_EXCL << 12) 
> #define O_WASI_TRUNC (WASI_OFLAGS_TRUNC << 12) 
> #define O_WASI_SYNC WASI_FDFLAGS_SYNC #define O_WASI_CLOEXEC 0 
> ```
> Remove the WASI_ from these and you see that in emscripten they are 
> different.  E.g. for O_RDONLY in fcntl.h under system/lib/libc/musl/include 
> it is:
> ```
> #define O_RDONLY  00
> ```
> But For WASI I "overwrite" it with 0x0400.  It's a hack (
> https://github.com/dotnet/runtimelab/pull/1850/files#diff-5d86cce7b76e1bece0aec26ee2bc4830503046428aecbbdefbf845832573020f),
>  
> and I do that switch with a "special" options flag.   
>
> Interesting that these constants are different between emscripten and WASI 
> as both are musl.  There must be some history there I suppose.
> On Thursday, February 10, 2022 at 12:34:43 PM UTC-5 alon...@gmail.com 
> wrote:
>
>> I see... yeah, mixing and matching object files from different toolchains 
>> is not guaranteed to work. They need C ABI compatibility, I'm afraid.
>>
>> It is a little sad to need separate runtimes, though, so I hope we can 
>> find a way to avoid that for you. What is the specific code that would be 
>> different between the two runtimes?
>>
>> On Thu, Feb 10, 2022 at 5:23 AM yowl yowlxx  wrote:
>>
>>> " (Of course, header constants *can* be an issue if you link wasm object 
>>> files together that were built using different SDKs or even different 
>>> versions of the same SDK.)"
>>>
>>> Yes, that is what I naively tried to do.  The CoreCLR runtime is 
>>> compiled using emscripten's SDK, but I wanted to link it with the WASI SDK 
>>> libc++ to get file open support.  What I'll probably have to do in the long 
>>> run is have two CoreCLR runtimes, one for Emscripten and one for WASI SDK.
>>>
>>> On Mon, 7 Feb 2022 at 14:39, Alon Zakai  wrote:
>>>
 Header constants like O_CREAT should not be a problem between wasm and 
 the host. WASI defines a very clear API between wasm and the runtime, and 
 those constants are not part of it. That is, "wasi-sdk host" is a 
 confusing 
 way to put it: there are VMs with WASI support, but they are not tied to 
 the wasi-sdk toolchain, which is just one toolchain that emits WASI.

 (Of course, header constants *can* be an issue if you link wasm object 
 files together that were built using different SDKs or even different 
 versions of the same SDK.)

 Emscripten could add more complete WASI support, if that were useful - 
 it's just a matter of implementing the APIs. PRs would be welcome!



 On Sat, Feb 5, 2022 at 9:59 AM scot...@gmail.com  
 wrote:

> My own observations mixing Emscripten and WASI-SDK are that some 
> constants, e.g. `O_CREAT` are different in Emscripten (0x64) to Wasi-SDK 
> (0x1000).  Allowing emscripten to build something that would run 
> under 
> a wasi-sdk host, like wasmtime, is tricky - I think.
>
> On Friday, December 17, 2021 at 8:01:14 AM UTC-5 Floh wrote:
>
>> Ah alright, that toolchain file looks a lot simpler (it's under 
>> "share/cmake" in the SDK download). I'll use that as base for my own 
>> cmake 
>> toolchain file instead. Thanks!
>>
>> On Thursday, 16 December 2021 at 21:18:46 UTC+1 s...@google.com 
>> wrote:
>>
>>> On Thu, Dec 16, 2021 at 5:53 AM Floh  wrote:
>>>
 > Unless you plan to run those tools on the web then I think 
 wasi-sdk is most likely the way to go.   How do you plan to run the 
 binaries BTW? 

 Currently I'm using wasmtime.

 I was actually successful to build my shader compiler through 
 wasi-sdk, and run it through wasmtime, the file system access worked 
 as 
 expected.

 Installing the wasi-sdk is just downloading and unpacking somewhere 
 (e.g. no env-variables or search path updating needed), and then I 
 created 
 a cmake toolchain file which looks a lot like my Emscripten toolchain 
 file, 
 but with all the Emscripten-specific parts removed:

Re: Emscripten SDK and WASI filesystem access

[scot...@gmail.com]

Its the constants in fcntl.h / in wasi: __header_fcntl.h and api.h :

```

#define O_WASI_RDONLY (0x0400) 
#define O_WASI_WRONLY (0x1000) 
#define O_WASI_RDWR (O_WASI_RDONLY | O_WASI_WRONLY) 


typedef uint16_t __wasi_oflags_t; 
typedef uint16_t __wasi_fdflags_t; 


#define WASI_OFLAGS_CREAT ((__wasi_oflags_t)(1 << 0)) 
#define WASI_OFLAGS_EXCL ((__wasi_oflags_t)(1 << 2)) 
#define WASI_OFLAGS_TRUNC ((__wasi_oflags_t)(1 << 3)) 
#define WASI_FDFLAGS_SYNC ((__wasi_fdflags_t)(1 << 4)) 


#define O_WASI_CREAT (WASI_OFLAGS_CREAT << 12) 
#define O_WASI_EXCL (WASI_OFLAGS_EXCL << 12) 
#define O_WASI_TRUNC (WASI_OFLAGS_TRUNC << 12) 
#define O_WASI_SYNC WASI_FDFLAGS_SYNC #define O_WASI_CLOEXEC 0 
```
Remove the WASI_ from these and you see that in emscripten they are 
different.  E.g. for O_RDONLY in fcntl.h under system/lib/libc/musl/include 
it is:
```
#define O_RDONLY  00
```
But For WASI I "overwrite" it with 0x0400.  It's a hack 
(https://github.com/dotnet/runtimelab/pull/1850/files#diff-5d86cce7b76e1bece0aec26ee2bc4830503046428aecbbdefbf845832573020f),
 
and I do that switch with a "special" options flag.   

Interesting that these constants are different between emscripten and WASI 
as both are musl.  There must be some history there I suppose.
On Thursday, February 10, 2022 at 12:34:43 PM UTC-5 alon...@gmail.com wrote:

> I see... yeah, mixing and matching object files from different toolchains 
> is not guaranteed to work. They need C ABI compatibility, I'm afraid.
>
> It is a little sad to need separate runtimes, though, so I hope we can 
> find a way to avoid that for you. What is the specific code that would be 
> different between the two runtimes?
>
> On Thu, Feb 10, 2022 at 5:23 AM yowl yowlxx  wrote:
>
>> " (Of course, header constants *can* be an issue if you link wasm object 
>> files together that were built using different SDKs or even different 
>> versions of the same SDK.)"
>>
>> Yes, that is what I naively tried to do.  The CoreCLR runtime is compiled 
>> using emscripten's SDK, but I wanted to link it with the WASI SDK libc++ to 
>> get file open support.  What I'll probably have to do in the long run is 
>> have two CoreCLR runtimes, one for Emscripten and one for WASI SDK.
>>
>> On Mon, 7 Feb 2022 at 14:39, Alon Zakai  wrote:
>>
>>> Header constants like O_CREAT should not be a problem between wasm and 
>>> the host. WASI defines a very clear API between wasm and the runtime, and 
>>> those constants are not part of it. That is, "wasi-sdk host" is a confusing 
>>> way to put it: there are VMs with WASI support, but they are not tied to 
>>> the wasi-sdk toolchain, which is just one toolchain that emits WASI.
>>>
>>> (Of course, header constants *can* be an issue if you link wasm object 
>>> files together that were built using different SDKs or even different 
>>> versions of the same SDK.)
>>>
>>> Emscripten could add more complete WASI support, if that were useful - 
>>> it's just a matter of implementing the APIs. PRs would be welcome!
>>>
>>>
>>>
>>> On Sat, Feb 5, 2022 at 9:59 AM scot...@gmail.com  
>>> wrote:
>>>
 My own observations mixing Emscripten and WASI-SDK are that some 
 constants, e.g. `O_CREAT` are different in Emscripten (0x64) to Wasi-SDK 
 (0x1000).  Allowing emscripten to build something that would run under 
 a wasi-sdk host, like wasmtime, is tricky - I think.

 On Friday, December 17, 2021 at 8:01:14 AM UTC-5 Floh wrote:

> Ah alright, that toolchain file looks a lot simpler (it's under 
> "share/cmake" in the SDK download). I'll use that as base for my own 
> cmake 
> toolchain file instead. Thanks!
>
> On Thursday, 16 December 2021 at 21:18:46 UTC+1 s...@google.com wrote:
>
>> On Thu, Dec 16, 2021 at 5:53 AM Floh  wrote:
>>
>>> > Unless you plan to run those tools on the web then I think 
>>> wasi-sdk is most likely the way to go.   How do you plan to run the 
>>> binaries BTW? 
>>>
>>> Currently I'm using wasmtime.
>>>
>>> I was actually successful to build my shader compiler through 
>>> wasi-sdk, and run it through wasmtime, the file system access worked as 
>>> expected.
>>>
>>> Installing the wasi-sdk is just downloading and unpacking somewhere 
>>> (e.g. no env-variables or search path updating needed), and then I 
>>> created 
>>> a cmake toolchain file which looks a lot like my Emscripten toolchain 
>>> file, 
>>> but with all the Emscripten-specific parts removed:
>>>
>>>
>>> https://github.com/floooh/fips/blob/master/cmake-toolchains/wasisdk.toolchain.cmake
>>>
>>
>> FYI wasi-sdk includes a cmake toolchain fail upstream:  
>> https://github.com/WebAssembly/wasi-sdk/blob/main/wasi-sdk.cmake
>>
>> Maybe we forgot to include it in the download?   But it should be 
>> usable I think.
>>
>>
>>> The only problem (vs the Emscripten SDK) was that the wasi-sdk 
>>> 

Re: Emscripten SDK and WASI filesystem access

[Alon Zakai]

I see... yeah, mixing and matching object files from different toolchains
is not guaranteed to work. They need C ABI compatibility, I'm afraid.

It is a little sad to need separate runtimes, though, so I hope we can find
a way to avoid that for you. What is the specific code that would be
different between the two runtimes?

On Thu, Feb 10, 2022 at 5:23 AM yowl yowlxx  wrote:

> " (Of course, header constants *can* be an issue if you link wasm object
> files together that were built using different SDKs or even different
> versions of the same SDK.)"
>
> Yes, that is what I naively tried to do.  The CoreCLR runtime is compiled
> using emscripten's SDK, but I wanted to link it with the WASI SDK libc++ to
> get file open support.  What I'll probably have to do in the long run is
> have two CoreCLR runtimes, one for Emscripten and one for WASI SDK.
>
> On Mon, 7 Feb 2022 at 14:39, Alon Zakai  wrote:
>
>> Header constants like O_CREAT should not be a problem between wasm and
>> the host. WASI defines a very clear API between wasm and the runtime, and
>> those constants are not part of it. That is, "wasi-sdk host" is a confusing
>> way to put it: there are VMs with WASI support, but they are not tied to
>> the wasi-sdk toolchain, which is just one toolchain that emits WASI.
>>
>> (Of course, header constants *can* be an issue if you link wasm object
>> files together that were built using different SDKs or even different
>> versions of the same SDK.)
>>
>> Emscripten could add more complete WASI support, if that were useful -
>> it's just a matter of implementing the APIs. PRs would be welcome!
>>
>>
>>
>> On Sat, Feb 5, 2022 at 9:59 AM scot...@gmail.com 
>> wrote:
>>
>>> My own observations mixing Emscripten and WASI-SDK are that some
>>> constants, e.g. `O_CREAT` are different in Emscripten (0x64) to Wasi-SDK
>>> (0x1000).  Allowing emscripten to build something that would run under
>>> a wasi-sdk host, like wasmtime, is tricky - I think.
>>>
>>> On Friday, December 17, 2021 at 8:01:14 AM UTC-5 Floh wrote:
>>>
 Ah alright, that toolchain file looks a lot simpler (it's under
 "share/cmake" in the SDK download). I'll use that as base for my own cmake
 toolchain file instead. Thanks!

 On Thursday, 16 December 2021 at 21:18:46 UTC+1 s...@google.com wrote:

> On Thu, Dec 16, 2021 at 5:53 AM Floh  wrote:
>
>> > Unless you plan to run those tools on the web then I think wasi-sdk
>> is most likely the way to go.   How do you plan to run the binaries BTW?
>>
>> Currently I'm using wasmtime.
>>
>> I was actually successful to build my shader compiler through
>> wasi-sdk, and run it through wasmtime, the file system access worked as
>> expected.
>>
>> Installing the wasi-sdk is just downloading and unpacking somewhere
>> (e.g. no env-variables or search path updating needed), and then I 
>> created
>> a cmake toolchain file which looks a lot like my Emscripten toolchain 
>> file,
>> but with all the Emscripten-specific parts removed:
>>
>>
>> https://github.com/floooh/fips/blob/master/cmake-toolchains/wasisdk.toolchain.cmake
>>
>
> FYI wasi-sdk includes a cmake toolchain fail upstream:
> https://github.com/WebAssembly/wasi-sdk/blob/main/wasi-sdk.cmake
>
> Maybe we forgot to include it in the download?   But it should be
> usable I think.
>
>
>> The only problem (vs the Emscripten SDK) was that the wasi-sdk
>> doesn't come with pthread headers/stubs, so I had to tinker a bit with 
>> the
>> glslangValidator source code (which for some reason needs thread-local 
>> data
>> and mutexes, but then doesn't actually spawn any threads).
>>
>> The popen() calls (to run native Metal or D3D shader compiler
>> executables) simply seem to fail at runtime (which I'm already handling 
>> in
>> my code).
>>
>> All in all pretty smooth sailing, at the moment it's just an
>> experiment though, but a promising one :)
>>
>>
>> On Tuesday, 14 December 2021 at 23:41:10 UTC+1 s...@google.com wrote:
>>
>>> On Tue, Dec 14, 2021 at 9:28 AM Floh  wrote:
>>>
 Oki doki, thanks for the clarification. I just wanted to check if
 I'm missing something, using the wasi-sdk makes sense in that case.

 Maybe my use case helps a bit to prioritize "proper" WASI support a
 bit :)

 I basically want to replace native command line tools (in my case:
 a shader compiler built out of the the Khronos GLSL compiler, 
 SPIRVTools
 and SPIRVCross) with a WASI version, because right now I need to build 
 this
 tool in 4 variants (Windows x86_64, Linux x86_64, macOS x86_64 and 
 macOS
 arm64) and then "distribute" the binaries through a git repository. My 
 plan
 is to replace this with a single WASI binary (building on 

Re: Emscripten SDK and WASI filesystem access

[yowl yowlxx]

" (Of course, header constants *can* be an issue if you link wasm object
files together that were built using different SDKs or even different
versions of the same SDK.)"

Yes, that is what I naively tried to do.  The CoreCLR runtime is compiled
using emscripten's SDK, but I wanted to link it with the WASI SDK libc++ to
get file open support.  What I'll probably have to do in the long run is
have two CoreCLR runtimes, one for Emscripten and one for WASI SDK.

On Mon, 7 Feb 2022 at 14:39, Alon Zakai  wrote:

> Header constants like O_CREAT should not be a problem between wasm and the
> host. WASI defines a very clear API between wasm and the runtime, and those
> constants are not part of it. That is, "wasi-sdk host" is a confusing way
> to put it: there are VMs with WASI support, but they are not tied to the
> wasi-sdk toolchain, which is just one toolchain that emits WASI.
>
> (Of course, header constants *can* be an issue if you link wasm object
> files together that were built using different SDKs or even different
> versions of the same SDK.)
>
> Emscripten could add more complete WASI support, if that were useful -
> it's just a matter of implementing the APIs. PRs would be welcome!
>
>
>
> On Sat, Feb 5, 2022 at 9:59 AM scot...@gmail.com 
> wrote:
>
>> My own observations mixing Emscripten and WASI-SDK are that some
>> constants, e.g. `O_CREAT` are different in Emscripten (0x64) to Wasi-SDK
>> (0x1000).  Allowing emscripten to build something that would run under
>> a wasi-sdk host, like wasmtime, is tricky - I think.
>>
>> On Friday, December 17, 2021 at 8:01:14 AM UTC-5 Floh wrote:
>>
>>> Ah alright, that toolchain file looks a lot simpler (it's under
>>> "share/cmake" in the SDK download). I'll use that as base for my own cmake
>>> toolchain file instead. Thanks!
>>>
>>> On Thursday, 16 December 2021 at 21:18:46 UTC+1 s...@google.com wrote:
>>>
 On Thu, Dec 16, 2021 at 5:53 AM Floh  wrote:

> > Unless you plan to run those tools on the web then I think wasi-sdk
> is most likely the way to go.   How do you plan to run the binaries BTW?
>
> Currently I'm using wasmtime.
>
> I was actually successful to build my shader compiler through
> wasi-sdk, and run it through wasmtime, the file system access worked as
> expected.
>
> Installing the wasi-sdk is just downloading and unpacking somewhere
> (e.g. no env-variables or search path updating needed), and then I created
> a cmake toolchain file which looks a lot like my Emscripten toolchain 
> file,
> but with all the Emscripten-specific parts removed:
>
>
> https://github.com/floooh/fips/blob/master/cmake-toolchains/wasisdk.toolchain.cmake
>

 FYI wasi-sdk includes a cmake toolchain fail upstream:
 https://github.com/WebAssembly/wasi-sdk/blob/main/wasi-sdk.cmake

 Maybe we forgot to include it in the download?   But it should be
 usable I think.


> The only problem (vs the Emscripten SDK) was that the wasi-sdk doesn't
> come with pthread headers/stubs, so I had to tinker a bit with the
> glslangValidator source code (which for some reason needs thread-local 
> data
> and mutexes, but then doesn't actually spawn any threads).
>
> The popen() calls (to run native Metal or D3D shader compiler
> executables) simply seem to fail at runtime (which I'm already handling in
> my code).
>
> All in all pretty smooth sailing, at the moment it's just an
> experiment though, but a promising one :)
>
>
> On Tuesday, 14 December 2021 at 23:41:10 UTC+1 s...@google.com wrote:
>
>> On Tue, Dec 14, 2021 at 9:28 AM Floh  wrote:
>>
>>> Oki doki, thanks for the clarification. I just wanted to check if
>>> I'm missing something, using the wasi-sdk makes sense in that case.
>>>
>>> Maybe my use case helps a bit to prioritize "proper" WASI support a
>>> bit :)
>>>
>>> I basically want to replace native command line tools (in my case: a
>>> shader compiler built out of the the Khronos GLSL compiler, SPIRVTools 
>>> and
>>> SPIRVCross) with a WASI version, because right now I need to build this
>>> tool in 4 variants (Windows x86_64, Linux x86_64, macOS x86_64 and macOS
>>> arm64) and then "distribute" the binaries through a git repository. My 
>>> plan
>>> is to replace this with a single WASI binary (building on the target
>>> machine is also not an option because these are complex C++ dependencies
>>> which can take up to 15 minutes to build).
>>>
>>
>> Unless you plan to run those tools on the web then I think wasi-sdk
>> is most likely the way to go.   How do you plan to run the binaries BTW?
>>
>>
>>>
>>> One missing piece in the WASI API is popen() support though. The
>>> shader compiler optionally needs to run the proprietaty D3D and Metal
>>> shader compilers to generate shader binary 

Re: Emscripten SDK and WASI filesystem access

[Alon Zakai]

Header constants like O_CREAT should not be a problem between wasm and the
host. WASI defines a very clear API between wasm and the runtime, and those
constants are not part of it. That is, "wasi-sdk host" is a confusing way
to put it: there are VMs with WASI support, but they are not tied to the
wasi-sdk toolchain, which is just one toolchain that emits WASI.

(Of course, header constants *can* be an issue if you link wasm object
files together that were built using different SDKs or even different
versions of the same SDK.)

Emscripten could add more complete WASI support, if that were useful - it's
just a matter of implementing the APIs. PRs would be welcome!



On Sat, Feb 5, 2022 at 9:59 AM scot...@gmail.com 
wrote:

> My own observations mixing Emscripten and WASI-SDK are that some
> constants, e.g. `O_CREAT` are different in Emscripten (0x64) to Wasi-SDK
> (0x1000).  Allowing emscripten to build something that would run under
> a wasi-sdk host, like wasmtime, is tricky - I think.
>
> On Friday, December 17, 2021 at 8:01:14 AM UTC-5 Floh wrote:
>
>> Ah alright, that toolchain file looks a lot simpler (it's under
>> "share/cmake" in the SDK download). I'll use that as base for my own cmake
>> toolchain file instead. Thanks!
>>
>> On Thursday, 16 December 2021 at 21:18:46 UTC+1 s...@google.com wrote:
>>
>>> On Thu, Dec 16, 2021 at 5:53 AM Floh  wrote:
>>>
 > Unless you plan to run those tools on the web then I think wasi-sdk
 is most likely the way to go.   How do you plan to run the binaries BTW?

 Currently I'm using wasmtime.

 I was actually successful to build my shader compiler through wasi-sdk,
 and run it through wasmtime, the file system access worked as expected.

 Installing the wasi-sdk is just downloading and unpacking somewhere
 (e.g. no env-variables or search path updating needed), and then I created
 a cmake toolchain file which looks a lot like my Emscripten toolchain file,
 but with all the Emscripten-specific parts removed:


 https://github.com/floooh/fips/blob/master/cmake-toolchains/wasisdk.toolchain.cmake

>>>
>>> FYI wasi-sdk includes a cmake toolchain fail upstream:
>>> https://github.com/WebAssembly/wasi-sdk/blob/main/wasi-sdk.cmake
>>>
>>> Maybe we forgot to include it in the download?   But it should be usable
>>> I think.
>>>
>>>
 The only problem (vs the Emscripten SDK) was that the wasi-sdk doesn't
 come with pthread headers/stubs, so I had to tinker a bit with the
 glslangValidator source code (which for some reason needs thread-local data
 and mutexes, but then doesn't actually spawn any threads).

 The popen() calls (to run native Metal or D3D shader compiler
 executables) simply seem to fail at runtime (which I'm already handling in
 my code).

 All in all pretty smooth sailing, at the moment it's just an experiment
 though, but a promising one :)


 On Tuesday, 14 December 2021 at 23:41:10 UTC+1 s...@google.com wrote:

> On Tue, Dec 14, 2021 at 9:28 AM Floh  wrote:
>
>> Oki doki, thanks for the clarification. I just wanted to check if I'm
>> missing something, using the wasi-sdk makes sense in that case.
>>
>> Maybe my use case helps a bit to prioritize "proper" WASI support a
>> bit :)
>>
>> I basically want to replace native command line tools (in my case: a
>> shader compiler built out of the the Khronos GLSL compiler, SPIRVTools 
>> and
>> SPIRVCross) with a WASI version, because right now I need to build this
>> tool in 4 variants (Windows x86_64, Linux x86_64, macOS x86_64 and macOS
>> arm64) and then "distribute" the binaries through a git repository. My 
>> plan
>> is to replace this with a single WASI binary (building on the target
>> machine is also not an option because these are complex C++ dependencies
>> which can take up to 15 minutes to build).
>>
>
> Unless you plan to run those tools on the web then I think wasi-sdk is
> most likely the way to go.   How do you plan to run the binaries BTW?
>
>
>>
>> One missing piece in the WASI API is popen() support though. The
>> shader compiler optionally needs to run the proprietaty D3D and Metal
>> shader compilers to generate shader binary blobs. Not sure yet how I'll
>> tackle that eventually, but a WASI executable which just generates shader
>> source code (not binary blobs) would be a good start nonetheless.
>>
>> Cheers!
>> -Floh.
>>
>> On Tuesday, 14 December 2021 at 17:43:10 UTC+1 s...@google.com wrote:
>>
>>> The standalone/wasi support in emscripten is very basic and doesn't
>>> have full fileystem support yet.   I would certainly recommend using
>>> wasi-sdk if you want to run something on wasmtime.
>>>
>>> If I ever get around to landing this PR then a lot more of the FS
>>> stuff might start working:

Re: Emscripten SDK and WASI filesystem access

[scot...@gmail.com]

My own observations mixing Emscripten and WASI-SDK are that some constants, 
e.g. `O_CREAT` are different in Emscripten (0x64) to Wasi-SDK 
(0x1000).  Allowing emscripten to build something that would run under 
a wasi-sdk host, like wasmtime, is tricky - I think.

On Friday, December 17, 2021 at 8:01:14 AM UTC-5 Floh wrote:

> Ah alright, that toolchain file looks a lot simpler (it's under 
> "share/cmake" in the SDK download). I'll use that as base for my own cmake 
> toolchain file instead. Thanks!
>
> On Thursday, 16 December 2021 at 21:18:46 UTC+1 s...@google.com wrote:
>
>> On Thu, Dec 16, 2021 at 5:53 AM Floh  wrote:
>>
>>> > Unless you plan to run those tools on the web then I think wasi-sdk is 
>>> most likely the way to go.   How do you plan to run the binaries BTW? 
>>>
>>> Currently I'm using wasmtime.
>>>
>>> I was actually successful to build my shader compiler through wasi-sdk, 
>>> and run it through wasmtime, the file system access worked as expected.
>>>
>>> Installing the wasi-sdk is just downloading and unpacking somewhere 
>>> (e.g. no env-variables or search path updating needed), and then I created 
>>> a cmake toolchain file which looks a lot like my Emscripten toolchain file, 
>>> but with all the Emscripten-specific parts removed:
>>>
>>>
>>> https://github.com/floooh/fips/blob/master/cmake-toolchains/wasisdk.toolchain.cmake
>>>
>>
>> FYI wasi-sdk includes a cmake toolchain fail upstream:  
>> https://github.com/WebAssembly/wasi-sdk/blob/main/wasi-sdk.cmake
>>
>> Maybe we forgot to include it in the download?   But it should be usable 
>> I think.
>>
>>
>>> The only problem (vs the Emscripten SDK) was that the wasi-sdk doesn't 
>>> come with pthread headers/stubs, so I had to tinker a bit with the 
>>> glslangValidator source code (which for some reason needs thread-local data 
>>> and mutexes, but then doesn't actually spawn any threads).
>>>
>>> The popen() calls (to run native Metal or D3D shader compiler 
>>> executables) simply seem to fail at runtime (which I'm already handling in 
>>> my code).
>>>
>>> All in all pretty smooth sailing, at the moment it's just an experiment 
>>> though, but a promising one :)
>>>
>>>
>>> On Tuesday, 14 December 2021 at 23:41:10 UTC+1 s...@google.com wrote:
>>>
 On Tue, Dec 14, 2021 at 9:28 AM Floh  wrote:

> Oki doki, thanks for the clarification. I just wanted to check if I'm 
> missing something, using the wasi-sdk makes sense in that case.
>
> Maybe my use case helps a bit to prioritize "proper" WASI support a 
> bit :)
>
> I basically want to replace native command line tools (in my case: a 
> shader compiler built out of the the Khronos GLSL compiler, SPIRVTools 
> and 
> SPIRVCross) with a WASI version, because right now I need to build this 
> tool in 4 variants (Windows x86_64, Linux x86_64, macOS x86_64 and macOS 
> arm64) and then "distribute" the binaries through a git repository. My 
> plan 
> is to replace this with a single WASI binary (building on the target 
> machine is also not an option because these are complex C++ dependencies 
> which can take up to 15 minutes to build).
>

 Unless you plan to run those tools on the web then I think wasi-sdk is 
 most likely the way to go.   How do you plan to run the binaries BTW? 
  

>
> One missing piece in the WASI API is popen() support though. The 
> shader compiler optionally needs to run the proprietaty D3D and Metal 
> shader compilers to generate shader binary blobs. Not sure yet how I'll 
> tackle that eventually, but a WASI executable which just generates shader 
> source code (not binary blobs) would be a good start nonetheless.
>
> Cheers!
> -Floh.
>
> On Tuesday, 14 December 2021 at 17:43:10 UTC+1 s...@google.com wrote:
>
>> The standalone/wasi support in emscripten is very basic and doesn't 
>> have full fileystem support yet.   I would certainly recommend using 
>> wasi-sdk if you want to run something on wasmtime.
>>
>> If I ever get around to landing this PR then a lot more of the FS 
>> stuff might start working: 
>> https://github.com/emscripten-core/emscripten/pull/12704.   But this 
>> has not been a priority recently.   The interesting part for me would be 
>> that it might allow existing WASI applications to be run in the JS glue 
>> code.  i.e. take a pre-built wasi module and run `emcc --post-link` to 
>> run 
>> on the web. 
>>
>> On Tue, Dec 14, 2021 at 6:45 AM Floh  wrote:
>>
>>> I'm currently tinkering with Emscripten's WASI output and can't get 
>>> filesystem access to work. In short, everything compiles, but then when 
>>> running via:
>>>
>>> wasmtime --dir . bla.wasm
>>>
>>> ...all filesystem operations fail.
>>>
>>> When compiling with the clang included in the wasi-sdk it works as 
>>> expected. Is this 

Re: Emscripten SDK and WASI filesystem access

[Floh]

Ah alright, that toolchain file looks a lot simpler (it's under 
"share/cmake" in the SDK download). I'll use that as base for my own cmake 
toolchain file instead. Thanks!

On Thursday, 16 December 2021 at 21:18:46 UTC+1 s...@google.com wrote:

> On Thu, Dec 16, 2021 at 5:53 AM Floh  wrote:
>
>> > Unless you plan to run those tools on the web then I think wasi-sdk is 
>> most likely the way to go.   How do you plan to run the binaries BTW? 
>>
>> Currently I'm using wasmtime.
>>
>> I was actually successful to build my shader compiler through wasi-sdk, 
>> and run it through wasmtime, the file system access worked as expected.
>>
>> Installing the wasi-sdk is just downloading and unpacking somewhere (e.g. 
>> no env-variables or search path updating needed), and then I created a 
>> cmake toolchain file which looks a lot like my Emscripten toolchain file, 
>> but with all the Emscripten-specific parts removed:
>>
>>
>> https://github.com/floooh/fips/blob/master/cmake-toolchains/wasisdk.toolchain.cmake
>>
>
> FYI wasi-sdk includes a cmake toolchain fail upstream:  
> https://github.com/WebAssembly/wasi-sdk/blob/main/wasi-sdk.cmake
>
> Maybe we forgot to include it in the download?   But it should be usable I 
> think.
>
>
>> The only problem (vs the Emscripten SDK) was that the wasi-sdk doesn't 
>> come with pthread headers/stubs, so I had to tinker a bit with the 
>> glslangValidator source code (which for some reason needs thread-local data 
>> and mutexes, but then doesn't actually spawn any threads).
>>
>> The popen() calls (to run native Metal or D3D shader compiler 
>> executables) simply seem to fail at runtime (which I'm already handling in 
>> my code).
>>
>> All in all pretty smooth sailing, at the moment it's just an experiment 
>> though, but a promising one :)
>>
>>
>> On Tuesday, 14 December 2021 at 23:41:10 UTC+1 s...@google.com wrote:
>>
>>> On Tue, Dec 14, 2021 at 9:28 AM Floh  wrote:
>>>
 Oki doki, thanks for the clarification. I just wanted to check if I'm 
 missing something, using the wasi-sdk makes sense in that case.

 Maybe my use case helps a bit to prioritize "proper" WASI support a bit 
 :)

 I basically want to replace native command line tools (in my case: a 
 shader compiler built out of the the Khronos GLSL compiler, SPIRVTools and 
 SPIRVCross) with a WASI version, because right now I need to build this 
 tool in 4 variants (Windows x86_64, Linux x86_64, macOS x86_64 and macOS 
 arm64) and then "distribute" the binaries through a git repository. My 
 plan 
 is to replace this with a single WASI binary (building on the target 
 machine is also not an option because these are complex C++ dependencies 
 which can take up to 15 minutes to build).

>>>
>>> Unless you plan to run those tools on the web then I think wasi-sdk is 
>>> most likely the way to go.   How do you plan to run the binaries BTW? 
>>>  
>>>

 One missing piece in the WASI API is popen() support though. The shader 
 compiler optionally needs to run the proprietaty D3D and Metal shader 
 compilers to generate shader binary blobs. Not sure yet how I'll tackle 
 that eventually, but a WASI executable which just generates shader source 
 code (not binary blobs) would be a good start nonetheless.

 Cheers!
 -Floh.

 On Tuesday, 14 December 2021 at 17:43:10 UTC+1 s...@google.com wrote:

> The standalone/wasi support in emscripten is very basic and doesn't 
> have full fileystem support yet.   I would certainly recommend using 
> wasi-sdk if you want to run something on wasmtime.
>
> If I ever get around to landing this PR then a lot more of the FS 
> stuff might start working: 
> https://github.com/emscripten-core/emscripten/pull/12704.   But this 
> has not been a priority recently.   The interesting part for me would be 
> that it might allow existing WASI applications to be run in the JS glue 
> code.  i.e. take a pre-built wasi module and run `emcc --post-link` to 
> run 
> on the web. 
>
> On Tue, Dec 14, 2021 at 6:45 AM Floh  wrote:
>
>> I'm currently tinkering with Emscripten's WASI output and can't get 
>> filesystem access to work. In short, everything compiles, but then when 
>> running via:
>>
>> wasmtime --dir . bla.wasm
>>
>> ...all filesystem operations fail.
>>
>> When compiling with the clang included in the wasi-sdk it works as 
>> expected. Is this something that can be easily fixed or worked around on 
>> my 
>> side, or should I switch to the wasi-sdk instead?
>>
>> Thanks!
>>
>>
>> -- 
>> You received this message because you are subscribed to the Google 
>> Groups "emscripten-discuss" group.
>> To unsubscribe from this group and stop receiving emails from it, 
>> send an email to emscripten-disc...@googlegroups.com.
>> To view this 

Re: Emscripten SDK and WASI filesystem access

['Sam Clegg' via emscripten-discuss]

On Thu, Dec 16, 2021 at 5:53 AM Floh  wrote:

> > Unless you plan to run those tools on the web then I think wasi-sdk is
> most likely the way to go.   How do you plan to run the binaries BTW?
>
> Currently I'm using wasmtime.
>
> I was actually successful to build my shader compiler through wasi-sdk,
> and run it through wasmtime, the file system access worked as expected.
>
> Installing the wasi-sdk is just downloading and unpacking somewhere (e.g.
> no env-variables or search path updating needed), and then I created a
> cmake toolchain file which looks a lot like my Emscripten toolchain file,
> but with all the Emscripten-specific parts removed:
>
>
> https://github.com/floooh/fips/blob/master/cmake-toolchains/wasisdk.toolchain.cmake
>

FYI wasi-sdk includes a cmake toolchain fail upstream:
https://github.com/WebAssembly/wasi-sdk/blob/main/wasi-sdk.cmake

Maybe we forgot to include it in the download?   But it should be usable I
think.


> The only problem (vs the Emscripten SDK) was that the wasi-sdk doesn't
> come with pthread headers/stubs, so I had to tinker a bit with the
> glslangValidator source code (which for some reason needs thread-local data
> and mutexes, but then doesn't actually spawn any threads).
>
> The popen() calls (to run native Metal or D3D shader compiler executables)
> simply seem to fail at runtime (which I'm already handling in my code).
>
> All in all pretty smooth sailing, at the moment it's just an experiment
> though, but a promising one :)
>
>
> On Tuesday, 14 December 2021 at 23:41:10 UTC+1 s...@google.com wrote:
>
>> On Tue, Dec 14, 2021 at 9:28 AM Floh  wrote:
>>
>>> Oki doki, thanks for the clarification. I just wanted to check if I'm
>>> missing something, using the wasi-sdk makes sense in that case.
>>>
>>> Maybe my use case helps a bit to prioritize "proper" WASI support a bit
>>> :)
>>>
>>> I basically want to replace native command line tools (in my case: a
>>> shader compiler built out of the the Khronos GLSL compiler, SPIRVTools and
>>> SPIRVCross) with a WASI version, because right now I need to build this
>>> tool in 4 variants (Windows x86_64, Linux x86_64, macOS x86_64 and macOS
>>> arm64) and then "distribute" the binaries through a git repository. My plan
>>> is to replace this with a single WASI binary (building on the target
>>> machine is also not an option because these are complex C++ dependencies
>>> which can take up to 15 minutes to build).
>>>
>>
>> Unless you plan to run those tools on the web then I think wasi-sdk is
>> most likely the way to go.   How do you plan to run the binaries BTW?
>>
>>
>>>
>>> One missing piece in the WASI API is popen() support though. The shader
>>> compiler optionally needs to run the proprietaty D3D and Metal shader
>>> compilers to generate shader binary blobs. Not sure yet how I'll tackle
>>> that eventually, but a WASI executable which just generates shader source
>>> code (not binary blobs) would be a good start nonetheless.
>>>
>>> Cheers!
>>> -Floh.
>>>
>>> On Tuesday, 14 December 2021 at 17:43:10 UTC+1 s...@google.com wrote:
>>>
 The standalone/wasi support in emscripten is very basic and doesn't
 have full fileystem support yet.   I would certainly recommend using
 wasi-sdk if you want to run something on wasmtime.

 If I ever get around to landing this PR then a lot more of the FS stuff
 might start working:
 https://github.com/emscripten-core/emscripten/pull/12704.   But this
 has not been a priority recently.   The interesting part for me would be
 that it might allow existing WASI applications to be run in the JS glue
 code.  i.e. take a pre-built wasi module and run `emcc --post-link` to run
 on the web.

 On Tue, Dec 14, 2021 at 6:45 AM Floh  wrote:

> I'm currently tinkering with Emscripten's WASI output and can't get
> filesystem access to work. In short, everything compiles, but then when
> running via:
>
> wasmtime --dir . bla.wasm
>
> ...all filesystem operations fail.
>
> When compiling with the clang included in the wasi-sdk it works as
> expected. Is this something that can be easily fixed or worked around on 
> my
> side, or should I switch to the wasi-sdk instead?
>
> Thanks!
>
>
> --
> You received this message because you are subscribed to the Google
> Groups "emscripten-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to emscripten-disc...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/emscripten-discuss/527adec2-ee4d-44f3-a783-ce901632c30cn%40googlegroups.com
> 
> .
>
 --
>>> You received this message because you are subscribed to the Google
>>> Groups "emscripten-discuss" group.
>>> To 

Re: Emscripten SDK and WASI filesystem access

[Floh]

> Unless you plan to run those tools on the web then I think wasi-sdk is 
most likely the way to go.   How do you plan to run the binaries BTW? 

Currently I'm using wasmtime.

I was actually successful to build my shader compiler through wasi-sdk, and 
run it through wasmtime, the file system access worked as expected.

Installing the wasi-sdk is just downloading and unpacking somewhere (e.g. 
no env-variables or search path updating needed), and then I created a 
cmake toolchain file which looks a lot like my Emscripten toolchain file, 
but with all the Emscripten-specific parts removed:

https://github.com/floooh/fips/blob/master/cmake-toolchains/wasisdk.toolchain.cmake

The only problem (vs the Emscripten SDK) was that the wasi-sdk doesn't come 
with pthread headers/stubs, so I had to tinker a bit with the 
glslangValidator source code (which for some reason needs thread-local data 
and mutexes, but then doesn't actually spawn any threads).

The popen() calls (to run native Metal or D3D shader compiler executables) 
simply seem to fail at runtime (which I'm already handling in my code).

All in all pretty smooth sailing, at the moment it's just an experiment 
though, but a promising one :)


On Tuesday, 14 December 2021 at 23:41:10 UTC+1 s...@google.com wrote:

> On Tue, Dec 14, 2021 at 9:28 AM Floh  wrote:
>
>> Oki doki, thanks for the clarification. I just wanted to check if I'm 
>> missing something, using the wasi-sdk makes sense in that case.
>>
>> Maybe my use case helps a bit to prioritize "proper" WASI support a bit :)
>>
>> I basically want to replace native command line tools (in my case: a 
>> shader compiler built out of the the Khronos GLSL compiler, SPIRVTools and 
>> SPIRVCross) with a WASI version, because right now I need to build this 
>> tool in 4 variants (Windows x86_64, Linux x86_64, macOS x86_64 and macOS 
>> arm64) and then "distribute" the binaries through a git repository. My plan 
>> is to replace this with a single WASI binary (building on the target 
>> machine is also not an option because these are complex C++ dependencies 
>> which can take up to 15 minutes to build).
>>
>
> Unless you plan to run those tools on the web then I think wasi-sdk is 
> most likely the way to go.   How do you plan to run the binaries BTW? 
>  
>
>>
>> One missing piece in the WASI API is popen() support though. The shader 
>> compiler optionally needs to run the proprietaty D3D and Metal shader 
>> compilers to generate shader binary blobs. Not sure yet how I'll tackle 
>> that eventually, but a WASI executable which just generates shader source 
>> code (not binary blobs) would be a good start nonetheless.
>>
>> Cheers!
>> -Floh.
>>
>> On Tuesday, 14 December 2021 at 17:43:10 UTC+1 s...@google.com wrote:
>>
>>> The standalone/wasi support in emscripten is very basic and doesn't have 
>>> full fileystem support yet.   I would certainly recommend using wasi-sdk if 
>>> you want to run something on wasmtime.
>>>
>>> If I ever get around to landing this PR then a lot more of the FS stuff 
>>> might start working: 
>>> https://github.com/emscripten-core/emscripten/pull/12704.   But this 
>>> has not been a priority recently.   The interesting part for me would be 
>>> that it might allow existing WASI applications to be run in the JS glue 
>>> code.  i.e. take a pre-built wasi module and run `emcc --post-link` to run 
>>> on the web. 
>>>
>>> On Tue, Dec 14, 2021 at 6:45 AM Floh  wrote:
>>>
 I'm currently tinkering with Emscripten's WASI output and can't get 
 filesystem access to work. In short, everything compiles, but then when 
 running via:

 wasmtime --dir . bla.wasm

 ...all filesystem operations fail.

 When compiling with the clang included in the wasi-sdk it works as 
 expected. Is this something that can be easily fixed or worked around on 
 my 
 side, or should I switch to the wasi-sdk instead?

 Thanks!


 -- 
 You received this message because you are subscribed to the Google 
 Groups "emscripten-discuss" group.
 To unsubscribe from this group and stop receiving emails from it, send 
 an email to emscripten-disc...@googlegroups.com.
 To view this discussion on the web visit 
 https://groups.google.com/d/msgid/emscripten-discuss/527adec2-ee4d-44f3-a783-ce901632c30cn%40googlegroups.com
  
 
 .

>>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "emscripten-discuss" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to emscripten-disc...@googlegroups.com.
>>
> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/emscripten-discuss/17b87056-ad33-404a-850b-141028b20f43n%40googlegroups.com
>>  
>> 

Re: Emscripten SDK and WASI filesystem access

['Sam Clegg' via emscripten-discuss]

On Tue, Dec 14, 2021 at 9:28 AM Floh  wrote:

> Oki doki, thanks for the clarification. I just wanted to check if I'm
> missing something, using the wasi-sdk makes sense in that case.
>
> Maybe my use case helps a bit to prioritize "proper" WASI support a bit :)
>
> I basically want to replace native command line tools (in my case: a
> shader compiler built out of the the Khronos GLSL compiler, SPIRVTools and
> SPIRVCross) with a WASI version, because right now I need to build this
> tool in 4 variants (Windows x86_64, Linux x86_64, macOS x86_64 and macOS
> arm64) and then "distribute" the binaries through a git repository. My plan
> is to replace this with a single WASI binary (building on the target
> machine is also not an option because these are complex C++ dependencies
> which can take up to 15 minutes to build).
>

Unless you plan to run those tools on the web then I think wasi-sdk is most
likely the way to go.   How do you plan to run the binaries BTW?


>
> One missing piece in the WASI API is popen() support though. The shader
> compiler optionally needs to run the proprietaty D3D and Metal shader
> compilers to generate shader binary blobs. Not sure yet how I'll tackle
> that eventually, but a WASI executable which just generates shader source
> code (not binary blobs) would be a good start nonetheless.
>
> Cheers!
> -Floh.
>
> On Tuesday, 14 December 2021 at 17:43:10 UTC+1 s...@google.com wrote:
>
>> The standalone/wasi support in emscripten is very basic and doesn't have
>> full fileystem support yet.   I would certainly recommend using wasi-sdk if
>> you want to run something on wasmtime.
>>
>> If I ever get around to landing this PR then a lot more of the FS stuff
>> might start working:
>> https://github.com/emscripten-core/emscripten/pull/12704.   But this has
>> not been a priority recently.   The interesting part for me would be that
>> it might allow existing WASI applications to be run in the JS glue code.
>> i.e. take a pre-built wasi module and run `emcc --post-link` to run on the
>> web.
>>
>> On Tue, Dec 14, 2021 at 6:45 AM Floh  wrote:
>>
>>> I'm currently tinkering with Emscripten's WASI output and can't get
>>> filesystem access to work. In short, everything compiles, but then when
>>> running via:
>>>
>>> wasmtime --dir . bla.wasm
>>>
>>> ...all filesystem operations fail.
>>>
>>> When compiling with the clang included in the wasi-sdk it works as
>>> expected. Is this something that can be easily fixed or worked around on my
>>> side, or should I switch to the wasi-sdk instead?
>>>
>>> Thanks!
>>>
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "emscripten-discuss" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to emscripten-disc...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/emscripten-discuss/527adec2-ee4d-44f3-a783-ce901632c30cn%40googlegroups.com
>>> 
>>> .
>>>
>> --
> You received this message because you are subscribed to the Google Groups
> "emscripten-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to emscripten-discuss+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/emscripten-discuss/17b87056-ad33-404a-850b-141028b20f43n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"emscripten-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to emscripten-discuss+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/emscripten-discuss/CAL_va2_0%2BNab2kLK6DcfVrtHDcv9v-KrqqKRZZOneHZSH77B2Q%40mail.gmail.com.

Re: Emscripten SDK and WASI filesystem access

[Floh]

Oki doki, thanks for the clarification. I just wanted to check if I'm 
missing something, using the wasi-sdk makes sense in that case.

Maybe my use case helps a bit to prioritize "proper" WASI support a bit :)

I basically want to replace native command line tools (in my case: a shader 
compiler built out of the the Khronos GLSL compiler, SPIRVTools and 
SPIRVCross) with a WASI version, because right now I need to build this 
tool in 4 variants (Windows x86_64, Linux x86_64, macOS x86_64 and macOS 
arm64) and then "distribute" the binaries through a git repository. My plan 
is to replace this with a single WASI binary (building on the target 
machine is also not an option because these are complex C++ dependencies 
which can take up to 15 minutes to build).

One missing piece in the WASI API is popen() support though. The shader 
compiler optionally needs to run the proprietaty D3D and Metal shader 
compilers to generate shader binary blobs. Not sure yet how I'll tackle 
that eventually, but a WASI executable which just generates shader source 
code (not binary blobs) would be a good start nonetheless.

Cheers!
-Floh.

On Tuesday, 14 December 2021 at 17:43:10 UTC+1 s...@google.com wrote:

> The standalone/wasi support in emscripten is very basic and doesn't have 
> full fileystem support yet.   I would certainly recommend using wasi-sdk if 
> you want to run something on wasmtime.
>
> If I ever get around to landing this PR then a lot more of the FS stuff 
> might start working: 
> https://github.com/emscripten-core/emscripten/pull/12704.   But this has 
> not been a priority recently.   The interesting part for me would be that 
> it might allow existing WASI applications to be run in the JS glue code.  
> i.e. take a pre-built wasi module and run `emcc --post-link` to run on the 
> web. 
>
> On Tue, Dec 14, 2021 at 6:45 AM Floh  wrote:
>
>> I'm currently tinkering with Emscripten's WASI output and can't get 
>> filesystem access to work. In short, everything compiles, but then when 
>> running via:
>>
>> wasmtime --dir . bla.wasm
>>
>> ...all filesystem operations fail.
>>
>> When compiling with the clang included in the wasi-sdk it works as 
>> expected. Is this something that can be easily fixed or worked around on my 
>> side, or should I switch to the wasi-sdk instead?
>>
>> Thanks!
>>
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "emscripten-discuss" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to emscripten-disc...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/emscripten-discuss/527adec2-ee4d-44f3-a783-ce901632c30cn%40googlegroups.com
>>  
>> 
>> .
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"emscripten-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to emscripten-discuss+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/emscripten-discuss/17b87056-ad33-404a-850b-141028b20f43n%40googlegroups.com.

Re: Emscripten SDK and WASI filesystem access

['Sam Clegg' via emscripten-discuss]

The standalone/wasi support in emscripten is very basic and doesn't have
full fileystem support yet.   I would certainly recommend using wasi-sdk if
you want to run something on wasmtime.

If I ever get around to landing this PR then a lot more of the FS stuff
might start working:
https://github.com/emscripten-core/emscripten/pull/12704.   But this has
not been a priority recently.   The interesting part for me would be that
it might allow existing WASI applications to be run in the JS glue code.
i.e. take a pre-built wasi module and run `emcc --post-link` to run on the
web.

On Tue, Dec 14, 2021 at 6:45 AM Floh  wrote:

> I'm currently tinkering with Emscripten's WASI output and can't get
> filesystem access to work. In short, everything compiles, but then when
> running via:
>
> wasmtime --dir . bla.wasm
>
> ...all filesystem operations fail.
>
> When compiling with the clang included in the wasi-sdk it works as
> expected. Is this something that can be easily fixed or worked around on my
> side, or should I switch to the wasi-sdk instead?
>
> Thanks!
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "emscripten-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to emscripten-discuss+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/emscripten-discuss/527adec2-ee4d-44f3-a783-ce901632c30cn%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"emscripten-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to emscripten-discuss+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/emscripten-discuss/CAL_va2_7oCfMzAdrv_QFA4TYE%2BUdSULXGntHZ6gAHsZVfXKZ3g%40mail.gmail.com.

Emscripten SDK and WASI filesystem access

[Floh]

I'm currently tinkering with Emscripten's WASI output and can't get 
filesystem access to work. In short, everything compiles, but then when 
running via:

wasmtime --dir . bla.wasm

...all filesystem operations fail.

When compiling with the clang included in the wasi-sdk it works as 
expected. Is this something that can be easily fixed or worked around on my 
side, or should I switch to the wasi-sdk instead?

Thanks!


-- 
You received this message because you are subscribed to the Google Groups 
"emscripten-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to emscripten-discuss+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/emscripten-discuss/527adec2-ee4d-44f3-a783-ce901632c30cn%40googlegroups.com.