I have a comparable setup in the ogv.js media player: On the main thread I have a JS front-end and an emscripten C module for the demuxer, which extracts packets of compressed data to be sent to Workers with additional emscripten C modules which decode the data and send back uncompressed video or audio to be handled by JS in the main thread (WebGL and Web Audio used directly).
I think this maps to scenario 3 in your mail. It should be fairly easy to send a Float32Array and an Int32Array through worker messages; since they're backed by a buffer, it's mostly the cost of copying that backing buffer. (You can also send an ArrayBuffer directly and wrap it into a typed array on the other end. Shouldn't be much difference in performance.) Main thing to watch out for: make sure you are not accidentally sending the entire emscripten heap buffer! If you extracted live heap views, then the copy may be very slow as it tries to copy the entire 16M or larger backing buffer. If you have extracted fresh buffers containing only the bytes needed, then they should copy cleanly. If you have live buffers from the interface you're using, you can copy them using the copy constructor of the appropriate typed array: var newArr = new Float32Array(extractedArr); and then send that copy instead of the original that referenced all of the heap. You can also optimize the postMessage data transfer by using the 'transferList' parameter, putting the buffer property of each typed array in the list. This will avoid an extra copy of the smaller backing buffer, as long as you no longer need the array in the worker. https://developer.mozilla.org/en-US/docs/Web/API/Worker/postMessage -- brion On Jun 17, 2016 3:26 AM, "Sohail Siadat" <[email protected]> wrote: > > Efficient transfer of large arrays with an Emscripten C++ Web Worker: which JavaScript design is better? > > I have trouble deciding between the following three designs. My code is already working well on web in HTML JavaScript using a core algorithm implemented in C++, but I want to turn it into a web-worker, because it can be a time consuming process, so I don't want to block the 3D Designer's function and UI. > > > I have an Emscripten C++ algorithm. Which design is more efficient to transfer large data to a JavaScript program? Since a web worker does clone() and serialise, to transfer through the web worker message system, there is some overhead here. Also some code is needed to translate the resulting data on the C++ side, from HEAP32 into JavaScript arrays ( C -> JS ). > > By efficient, I mean which design is faster, i.e. which design leads to triggering less new and gc()(constructing and destructing JS objects). My Web Worker uses a core function written in C++which returns large arrays (two arrays of float[V][3] and int[N][3] with N=V=10000. It will be used to update a ThreeJS Geometry, and will be called tens of thousands of times over a long period on a web page. Apart being slow, this also may cause the browser to slow down, freeze or crash. > > Solution 1: > To write a Web Worker using JS which imports a JS code compiles using Emscripten. Cons: This option seems not possible, as the web-worker side needs to import the compiles JS file. Data exchange: C++ -> JS -> message(serialise) -> JS. Design: (C++)JS <-WW-> JS. Files: core_mc_algorithm.cpp, worker.js, main.js . > > Solution 2: > Use a C++ Web Worker compiled using -s BUILD_AS_WORKER=1, write some other C++ code on the main side that received the data, and convert the results from HEAP to JS on the main side: (WebWorker data traser handled by Emscripten): Pros: efficient transfer, but required two conversions. Risk: on C++ side, it requires multiple copying from vector to array, etc. Data exchange: C++ -> message(serialise) -> C++ -> JS, Design: (C++) <-WW-> C++(JS) . Files: worker.cpp, main.cpp, main.js . > Solution 3: > Again a C++ Web Worker, but the web worker function are contacted directly by the main program which is in JavaScript. Pros: The conversions/exchanges are not done twice. There is no separate exchange between C++ and JS, this conversion is done at the same time with WW serialisation. Risks: The decoding may be difficult and messy (the protocol will need to be reimplemented, which itself requires multiple conversions, which may not be very efficient). Also, the exchange may be not actually efficient and may not actually improve performance. Data exchange: C++ -> message(serialise) -> JS, Design: (C++) <-WW-> JS. Files: worker.cpp, main.js . > > I have a function like this in C++, I want to run it as a Web Worker (this is not the exact prototype, just as an example.): > > void produce_object ( REAL* verts_output, int number_of_vertices, int* faces_output, int number_of_triangles ) { // Run Marching cubes, which produces a vector<int> and a vector<float>. // fills in the arrays verts_output[] with coordinates (size: 3*number_of_vertices), // fill in faces_output[] with triangle vertex indices (size: 3*number_of_triangles ), using some numerical code which includes the Marching Cubes algorithm. } > > I need the following JavaScript callback function to get called with the right parameters. It is defined in an HTML file: > > function update_mesh_geometry_callback (verts, faces) { /* verts and faces are of type Float32Array and Int32Array of size (3*N) and (3*V). In this function they are used to create the following object, which is added to the scene.*/ var geo = new THREE.Geometry(verts, faces); // a subclass scene.add(new THREE.Mesh(gro, mat, etc)); } > > Typical size at least: number_of_vertices == 90000 = N, number_of_triangles == 8000 = V. > > > Thanks, > Sohail > > Also cross posted here but received no answer: > http://stackoverflow.com/questions/37867511/efficient-transfer-of-large-arrays-with-an-emscripten-c-web-worker-which-java?noredirect=1#comment63192712_37867511 > > > -- > You received this message because you are subscribed to the Google Groups "emscripten-discuss" group. > To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. > For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "emscripten-discuss" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
