Re: Moving large data between JS & wasm with IDL binder

[キャロウ マーク]

> On Jul 14, 2020, at 12:00, Beuc  wrote:
> 
>> 
>> XMLHttpRequest most definitely supports receiving responses into 
>> ArrayBuffers. You set responseType=‘arrayBuffer’. However now I look more 
>> closely at the API there doesn’t seem to be any way to provide it with the 
>> destination ArrayBuffer. It looks like XMLHttpRequest always creates it 
>> itself which makes my question moot.
> One optimization I did for RenPyWeb was importing XMLHttpRequest's buffer 
> into the filesystem (FS):
> https://github.com/renpy/renpy/blob/8b0d3dd4986a5a8656b3a5ad634c62bd52b1afbd/renpy/webloader.py#L64
>  
> 
> FS.writeFile(path, new Uint8Array(xhr.response), {canOwn:true});
> I'm not sure what's your use case is, it might help.
> 
> 

My use case is an inflater and transcoder for universal supercompressed 
textures. the incoming texture file, received by XMLHttpRequest into am 
ArrayBuffer, is parsed in JS from which a typed array with the data for a mip 
level is created. That has, maybe, it’s file dependent, to be passed to a Zstd 
decoder compiled with Emscripten. Back in JS-land, JS identifies the position 
and length of each image  and then must pass the image data to the transcoder, 
also compiled with Emscripten.

My goal is to reduce data copies to the minimum so I don’t think saving in the 
filesystem will help. Thanks for the suggestion though.

I have some further questions to help guide me on the right path w.r.t WebIDL 
binder vs Embind and moving the data.

1. Is it possible for 2 separately compiled .wasm packages to use the same 
HEAP8, possibly by not modularizing them? My initial plan was to make a 
separate Zstd module but a different HEAP8 from that used by the transcoder 
means a data copy.

2. If I use WebIDL binder then to pass data into, e.g. the Zstd decoder, I will 
have to do in Javascript:

var levelData = new Uint8Array( this.arrayBuffer, this.levels[ level 
].byteOffs, this.levels[ level ].byteLength);
var deflatedData = this.zstdModule._malloc(this.levels[ level 
].byteLength);
var inflatedData = this.zstdModule._malloc(this.levels[ level 
].uncompressedByteLength);
this.zstdModule.HEAPU8.set(levelData, deflatedData);
var byteCount = zstdDecoder.decompress(inflatedData,
  this.levels[ level 
].uncompressedByteLength,
  deflatedData,
  this.levels[ level ].byteLength);


I have a c++ wrapper around the Zstd decoder, which is c code. This class is 
called ZstdDecoder.  Is it possible to add JS to this class so instead of users 
of the decoder doing the above they could just call something more in the 
spirit of JS that hides the mallocs and takes a typed array in?


var byteCount = zstdDecoder.decompress(i
  this.levels[ level 
].uncompressedByteLength,
  levelData,
  );

var inflatedData = zstdDecoder.getInflatedData();


Regards

-Mark

-- 
You received this message because you are subscribed to the Google Groups 
"emscripten-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to emscripten-discuss+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/emscripten-discuss/8128FE9C-6D80-4292-B240-0D74B364AA8D%40callow.im.


signature.asc
Description: Message signed with OpenPGP

Re: Moving large data between JS & wasm with IDL binder

[Beuc]

Hi,

On 14/07/2020 18:49, キャロウ マーク wrote:
>
>
>> On Jul 14, 2020, at 8:37, 'Sam Clegg' via emscripten-discuss
>> > > wrote:
>>
>>
>>
>> On Mon, Jul 13, 2020 at 6:43 PM キャロウ　マーク > > wrote:
>>
>> Is is possible to use a typed array created on HEAP8 with
>> XMLHttpRequest as the destination for the incoming data?
>>
>>
>> I don't think so no.   IIRC  XMLHttpRequest API pre-dates the typed
>> array APIs.
>
> XMLHttpRequest most definitely supports receiving responses into
> ArrayBuffers. You set responseType=‘arrayBuffer’. However now I look
> more closely at the API there doesn’t seem to be any way to provide it
> with the destination ArrayBuffer. It looks like XMLHttpRequest always
> creates it itself which makes my question moot.
One optimization I did for RenPyWeb was importing XMLHttpRequest's
buffer into the filesystem (FS):
https://github.com/renpy/renpy/blob/8b0d3dd4986a5a8656b3a5ad634c62bd52b1afbd/renpy/webloader.py#L64
FS.writeFile(path, new Uint8Array(xhr.response), {canOwn:true});

I'm not sure what's your use case is, it might help.

Cheers!
Beuc

-- 
You received this message because you are subscribed to the Google Groups 
"emscripten-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to emscripten-discuss+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/emscripten-discuss/cb9bca95-877c-3a6d-921a-e9b093f57b33%40beuc.net.

Re: Moving large data between JS & wasm with IDL binder

[キャロウ マーク]

> On Jul 14, 2020, at 8:37, 'Sam Clegg' via emscripten-discuss 
>  wrote:
> 
> 
> 
> On Mon, Jul 13, 2020 at 6:43 PM キャロウ　マーク  > wrote:
> Is is possible to use a typed array created on HEAP8 with XMLHttpRequest as 
> the destination for the incoming data?
> 
> 
> I don't think so no.   IIRC  XMLHttpRequest API pre-dates the typed array 
> APIs.

XMLHttpRequest most definitely supports receiving responses into ArrayBuffers. 
You set responseType=‘arrayBuffer’. However now I look more closely at the API 
there doesn’t seem to be any way to provide it with the destination 
ArrayBuffer. It looks like XMLHttpRequest always creates it itself which makes 
my question moot.


Regards

-Mark

-- 
You received this message because you are subscribed to the Google Groups 
"emscripten-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to emscripten-discuss+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/emscripten-discuss/467D97EB-1B79-4C4C-A6AF-43108D8E2F27%40callow.im.


signature.asc
Description: Message signed with OpenPGP

Re: Moving large data between JS & wasm with IDL binder

['Sam Clegg' via emscripten-discuss]

On Mon, Jul 13, 2020 at 6:43 PM キャロウ マーク  wrote:

> Is is possible to use a typed array created on HEAP8 with XMLHttpRequest
> as the destination for the incoming data?
>
>
I don't think so no.   IIRC  XMLHttpRequest API pre-dates the typed array
APIs.

Regards
>
> -Mark
>
> On Jul 9, 2020, at 9:44, Alon Zakai  wrote:
>
> HEAP8 is one of the global variables the emscripten JS always has. It's
> always present for code in a JS library, --pre-js, or other code that is
> part of the main JS output (that is all optimized together).
>
> On Wed, Jul 8, 2020 at 6:39 PM キャロウ マーク  wrote:
>
>> Thanks Alon.
>>
>> > On Jul 8, 2020, at 15:59, Alon Zakai  wrote:
>> >
>> > Embind and the WebIDL binder are separate tools. They don't have any
>> special support for mixing between them.
>>
>> That’s what I had concluded. Thanks for confirming. My confusion arose
>> because it is not clear (to me at least) that emscripten::val is strictly
>> part of embind.
>>
>> >
>> > The WebIDL binder doesn't have direct support for copying data from JS
>> into wasm. You can do it manually though, without the WebIDL binder, by
>> malloc()ing some room, copying the data (all in JS, just a copy from the JS
>> data into HEAP8 for example), then you can pass the malloc()ed pointer as a
>> parameter to the compiled code (with the WebIDL binder, passing it as a
>> void* would work, or an int).
>>
>> So in JS do I create a typed array view on the right place in HEAP8 and
>> then copy in and out of that? How does the JS find HEAP8? I need to create
>> a wrapper for the zstd decoder so I need to be as efficient as possible in
>> moving the data.
>>
>>
> --
> You received this message because you are subscribed to the Google Groups
> "emscripten-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to emscripten-discuss+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/emscripten-discuss/66D3C2D4-C9C8-4665-85F2-0B8EAE8882C6%40callow.im
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"emscripten-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to emscripten-discuss+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/emscripten-discuss/CAL_va29UtEf3npHqHziiVqsYmvxND_Q_TFgQzqu2NYs%3DFvWwyw%40mail.gmail.com.

Re: Moving large data between JS & wasm with IDL binder

[キャロウ マーク]

Is is possible to use a typed array created on HEAP8 with XMLHttpRequest as the 
destination for the incoming data?

Regards

-Mark

> On Jul 9, 2020, at 9:44, Alon Zakai  wrote:
> 
> HEAP8 is one of the global variables the emscripten JS always has. It's 
> always present for code in a JS library, --pre-js, or other code that is part 
> of the main JS output (that is all optimized together).
> 
> On Wed, Jul 8, 2020 at 6:39 PM キャロウ　マーク  > wrote:
> Thanks Alon.
> 
> > On Jul 8, 2020, at 15:59, Alon Zakai  > > wrote:
> >
> > Embind and the WebIDL binder are separate tools. They don't have any 
> > special support for mixing between them.
> 
> That’s what I had concluded. Thanks for confirming. My confusion arose 
> because it is not clear (to me at least) that emscripten::val is strictly 
> part of embind.
> 
> >
> > The WebIDL binder doesn't have direct support for copying data from JS into 
> > wasm. You can do it manually though, without the WebIDL binder, by 
> > malloc()ing some room, copying the data (all in JS, just a copy from the JS 
> > data into HEAP8 for example), then you can pass the malloc()ed pointer as a 
> > parameter to the compiled code (with the WebIDL binder, passing it as a 
> > void* would work, or an int).
> 
> So in JS do I create a typed array view on the right place in HEAP8 and then 
> copy in and out of that? How does the JS find HEAP8? I need to create a 
> wrapper for the zstd decoder so I need to be as efficient as possible in 
> moving the data.
> 

-- 
You received this message because you are subscribed to the Google Groups 
"emscripten-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to emscripten-discuss+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/emscripten-discuss/66D3C2D4-C9C8-4665-85F2-0B8EAE8882C6%40callow.im.


signature.asc
Description: Message signed with OpenPGP

Re: Moving large data between JS & wasm with IDL binder

[Alon Zakai]

HEAP8 is one of the global variables the emscripten JS always has. It's
always present for code in a JS library, --pre-js, or other code that is
part of the main JS output (that is all optimized together).

On Wed, Jul 8, 2020 at 6:39 PM キャロウ マーク  wrote:

> Thanks Alon.
>
> > On Jul 8, 2020, at 15:59, Alon Zakai  wrote:
> >
> > Embind and the WebIDL binder are separate tools. They don't have any
> special support for mixing between them.
>
> That’s what I had concluded. Thanks for confirming. My confusion arose
> because it is not clear (to me at least) that emscripten::val is strictly
> part of embind.
>
> >
> > The WebIDL binder doesn't have direct support for copying data from JS
> into wasm. You can do it manually though, without the WebIDL binder, by
> malloc()ing some room, copying the data (all in JS, just a copy from the JS
> data into HEAP8 for example), then you can pass the malloc()ed pointer as a
> parameter to the compiled code (with the WebIDL binder, passing it as a
> void* would work, or an int).
>
> So in JS do I create a typed array view on the right place in HEAP8 and
> then copy in and out of that? How does the JS find HEAP8? I need to create
> a wrapper for the zstd decoder so I need to be as efficient as possible in
> moving the data.
>
> Regards
>
> -Mark
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "emscripten-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to emscripten-discuss+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/emscripten-discuss/1ECB6FF7-0049-4F3B-93AB-0CDD0879A908%40callow.im
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"emscripten-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to emscripten-discuss+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/emscripten-discuss/CAEX4NpQW%3DuafqKph1dJ7S%3DiNAp3rCDY%2B_9T8oeDh0kRtk6r%3Djg%40mail.gmail.com.

Re: Moving large data between JS & wasm with IDL binder

[キャロウ マーク]

Thanks Alon.

> On Jul 8, 2020, at 15:59, Alon Zakai  wrote:
> 
> Embind and the WebIDL binder are separate tools. They don't have any special 
> support for mixing between them.

That’s what I had concluded. Thanks for confirming. My confusion arose because 
it is not clear (to me at least) that emscripten::val is strictly part of 
embind.

> 
> The WebIDL binder doesn't have direct support for copying data from JS into 
> wasm. You can do it manually though, without the WebIDL binder, by 
> malloc()ing some room, copying the data (all in JS, just a copy from the JS 
> data into HEAP8 for example), then you can pass the malloc()ed pointer as a 
> parameter to the compiled code (with the WebIDL binder, passing it as a void* 
> would work, or an int).

So in JS do I create a typed array view on the right place in HEAP8 and then 
copy in and out of that? How does the JS find HEAP8? I need to create a wrapper 
for the zstd decoder so I need to be as efficient as possible in moving the 
data.

Regards

-Mark


-- 
You received this message because you are subscribed to the Google Groups 
"emscripten-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to emscripten-discuss+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/emscripten-discuss/1ECB6FF7-0049-4F3B-93AB-0CDD0879A908%40callow.im.


signature.asc
Description: Message signed with OpenPGP

Re: Moving large data between JS & wasm with IDL binder

[Alon Zakai]

Embind and the WebIDL binder are separate tools. They don't have any
special support for mixing between them.

The WebIDL binder doesn't have direct support for copying data from JS into
wasm. You can do it manually though, without the WebIDL binder, by
malloc()ing some room, copying the data (all in JS, just a copy from the JS
data into HEAP8 for example), then you can pass the malloc()ed pointer as a
parameter to the compiled code (with the WebIDL binder, passing it as a
void* would work, or an int).

On Wed, Jul 8, 2020 at 12:34 PM キャロウ マーク  wrote:

> Can anyone here explain, or point me at an example, of how to move large
> data, e.g an image, between JS memory and wasm memory using a function
> defined with WebIDL binder?
>
> In embind for moving data from JS to wasm, I use an emscripten::val
> parameter to pass in the JS handle then emscripten::val::module_property to
> get a handle to the wasm memory on which I make a memory view to which the
> data is copied.
>
> To move data from wasm to JS I use emscripten::val::get_typed_memory_view
> to create a view onto the item in wasm memory and return the view via a
> emscripten::val.
>
> I tried to use emscripten::val with WebIDL binder but I could not get it
> to compile. There is a short section in the WebIDL docs "Pointers,
> References, Value types (Ref and Value)”, which is not very illuminating,
> but does suggest that emscripten::val is not the right way to move data
> with WebIDL binder.
>
> I’d prefer to use WebIDL binder because in order to document my previous
> embind wrappers I ended up writing what looked very much like web IDL
> because it seems the best way to tie all the pieces together.
>
> Regards
>
> -Mark
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "emscripten-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to emscripten-discuss+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/emscripten-discuss/005C24B2-1D79-47CC-A291-D7A6B749582F%40callow.im
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"emscripten-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to emscripten-discuss+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/emscripten-discuss/CAEX4NpTgyrmUFDs_1B0CtJyvcEAZUuk9v9Oed5H6hAeTOQ1DEQ%40mail.gmail.com.

Moving large data between JS & wasm with IDL binder

[キャロウ マーク]

Can anyone here explain, or point me at an example, of how to move large data, 
e.g an image, between JS memory and wasm memory using a function defined with 
WebIDL binder?

In embind for moving data from JS to wasm, I use an emscripten::val parameter 
to pass in the JS handle then emscripten::val::module_property to get a handle 
to the wasm memory on which I make a memory view to which the data is copied.

To move data from wasm to JS I use emscripten::val::get_typed_memory_view to 
create a view onto the item in wasm memory and return the view via a 
emscripten::val.

I tried to use emscripten::val with WebIDL binder but I could not get it to 
compile. There is a short section in the WebIDL docs "Pointers, References, 
Value types (Ref and Value)”, which is not very illuminating, but does suggest 
that emscripten::val is not the right way to move data with WebIDL binder.

I’d prefer to use WebIDL binder because in order to document my previous embind 
wrappers I ended up writing what looked very much like web IDL because it seems 
the best way to tie all the pieces together.

Regards

-Mark


-- 
You received this message because you are subscribed to the Google Groups 
"emscripten-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to emscripten-discuss+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/emscripten-discuss/005C24B2-1D79-47CC-A291-D7A6B749582F%40callow.im.


signature.asc
Description: Message signed with OpenPGP