If performance is not a concern, then you could call into JS to reach each
byte, e.g. using EM_ASM. Or, if speed matters, you can read in chunks of 1K
or 1MB, writing them into a temporary spot in emscripten memory. This
avoids an extra copy of the whole 100MB file. e.g.
void* temp = malloc(1024*1024);
for (int i = 0; i < numChunks; i++) {
EM_ASM_({
HEAPU8.set(theBigBuffer.subarray($1 * 1024 * 1024, ($1 + 1) * 1024 *
1024), $0); // copy 1MB from the middle of theBigBuffer into temp
}, temp, i);
// process that 1MB that was copied to temp
}
(A more radical approach could use the new experimental split-memory mode,
https://github.com/kripken/emscripten/wiki/Split-Memory which actually
allows avoiding even those copies. However, it causes a very large slowdown
as a downside, so almost certainly not worth it.)
On Tue, Oct 13, 2015 at 9:52 AM, Richard Bateman <[email protected]>
wrote:
> Is it possible from emscripten code to read from an ArrayBuffer which is
> not part of the heap? I have a very large file which I read from disk using
> FileReader and would like to stream it as though from disk; I run out of
> memory very quickly when copying it around before using it.
>
> If I could write a function which would be able to stream from the
> ArrayBuffer then I could read a bit at a time rather than needing enough
> memory to store an extra copy of my 100MB+ file.
>
> Thoughts? Suggestions?
>
> Thanks!
>
> Richard Bateman
> GradeCam
>
> --
> You received this message because you are subscribed to the Google Groups
> "emscripten-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [email protected].
> For more options, visit https://groups.google.com/d/optout.
>
--
You received this message because you are subscribed to the Google Groups
"emscripten-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/d/optout.
