On Wednesday, February 12, 2014 5:15:50 PM UTC-5, Dave Nicponski wrote:
>
>
>
> On Wednesday, February 12, 2014 5:13:32 PM UTC-5, Dave Nicponski wrote:
>>
>> Hello, and thanks for the reply Alon.
>>
>> In the process of making a simple repo case, i better understand the
>> problem(s) involved here.
>> Executive summary: When a worker is run under nodejs with typed arrays
>> option set for the worker, an emscripten assertion is triggered:
>>
>> "Assertion failed: Cannot fallback to non-typed array case: Code is too
>> specialized"
>>
>> However, compiling the worker with USE_TYPED_ARRAYS=0 causes "no memory
>> initializer" to be inserted into the generated code, instead of the
>> traditional memory initializer block.
>> This causes the emlink.py script to fail when it cannot find the memory
>> initializer section.
>>
>> Attached find a pretty minimal repro. Note that where nodejs triggers
>> the above listed emscripten assertion failure, the same code appears to
>> work fine in a browser (at least under latest firefox). It's not entirely
>> clear how i should continue development. Using the browser is incredibly
>> inefficient for development for me compared to nodejs. I suspect a 'quick
>> fix' here would be to fix the emlink.py script to handle this case
>> correctly. Meanwhile, i'll try giving fastcomp a try if i can get it to
>> work for me.
>>
>> Thanks,
>> -dave-
>>
>> -------8<-------8<-------8<-------8<-------8<-------8<-------
>> Building sequence follows
>> -------8<-------8<-------8<-------8<-------8<-------8<-------
>> # This shows the simple clang setup I use. It mostly just sets up the
>> various
>> # include paths and forces C++ mode and c++11
>> $ echo $(./clang_compile_flags.py)
>> -Qunused-arguments -Wall -Wextra -Werror -Wno-unused-parameter
>> -fexceptions -std=c++11 -x c++ -iquote ./src -iquote
>> /home/daven/src/emscripen/emscripten/system/include/ -I . -isystem
>> ./include/protorpc/ -isystem ./include/openssl/ -isystem
>> ./include/tinyxml/tinyxml/
>>
>> # Build the MAIN (shared, static) library for main()
>> $ emcc $(./clang_compile_flags.py) -O1 -s MAIN_MODULE=1 -o
>> src/trash/poc/split_worker_with_common_lib/build/lib.js
>> src/trash/poc/split_worker_with_common_lib/lib.cc
>>
>> # Build the MAIN (shared, static) library for worker
>> $ emcc $(./clang_compile_flags.py) -O1 -s MAIN_MODULE=1 -s
>> BUILD_AS_WORKER=1 -o
>> src/trash/poc/split_worker_with_common_lib/build/lib.js
>> src/trash/poc/split_worker_with_common_lib/lib.cc
>>
>>
> Oops, this should obviously have read:
> $ emcc $(./clang_compile_flags.py) -O1 -s MAIN_MODULE=1 -s
> BUILD_AS_WORKER=1 -o
> src/trash/poc/split_worker_with_common_lib/build/lib_worker.js
> src/trash/poc/split_worker_with_common_lib/lib.cc
>
Ahh, remembered one more important detail.
I slightly modified emscripten/src/shell.js to enable webworker emulated
support in nodejs.
shell.js modifications:
src/shell.js-67- };
src/shell.js-68-
src/shell.js-69- Module['arguments'] = process['argv'].slice(2);
src/shell.js-70-+
src/shell.js-71-+ // XXX(daven): Hack in webworker emulation.
src/shell.js:72:+ Module['Worker'] = Worker =
require('webworker-threads').Worker;
src/shell.js-73-+ // XXX
src/shell.js-74-
src/shell.js-75- module['exports'] = Module;
src/shell.js-76-}
src/shell.js-77-else if (ENVIRONMENT_IS_SHELL) {
This uses the nodejs webworker-threads module (npm install
webworker-threads), which works fine in other tests i've done.
Cheers,
-dave-
>
>
>> # Build the SIDE (main module) library
>> $ emcc $(./clang_compile_flags.py) -O1 -s SIDE_MODULE=1 -o
>> src/trash/poc/split_worker_with_common_lib/build/main.js
>> src/trash/poc/split_worker_with_common_lib/main.cc
>>
>> # Link the main() module (SUCCESS)
>> $ emlink.py
>> src/trash/poc/split_worker_with_common_lib/build/{lib,main,main_module}.js
>> #Main module: src/trash/poc/split_worker_with_common_lib/build/lib.js
>> #Side module: src/trash/poc/split_worker_with_common_lib/build/main.js
>> #Output: src/trash/poc/split_worker_with_common_lib/build/main_module.js
>>
>> # Build the SIDE (worker module) library
>> $ emcc $(./clang_compile_flags.py) -O1 -s BUILD_AS_WORKER=1 -s
>> SIDE_MODULE=1 -s EXPORTED_FUNCTIONS="['_do_work']" -o
>> src/trash/poc/split_worker_with_common_lib/build/worker.js
>> src/trash/poc/split_worker_with_common_lib/worker.cc
>>
>> # Link the worker do_work() module
>> $ emlink.py
>> src/trash/poc/split_worker_with_common_lib/build/{lib_worker,worker,worker_module}.js
>> #Main module: src/trash/poc/split_worker_with_common_lib/build/lib.js
>> #Side module: src/trash/poc/split_worker_with_common_lib/build/worker.js
>> #Output: src/trash/poc/split_worker_with_common_lib/build/worker_module.js
>>
>> # Try running this in nodejs. It fails in emscripten worker setup code,
>> # apparently. It works fine from a browser (firefox, at least) though.
>> $ nodejs main_module.js
>> Lib function returns This is some string
>> Assertion failed: Cannot fallback to non-typed array case: Code is too
>> specialized
>>
>> # Try again building SIDE (worker module) library without typed arrays, so
>> # this can be run in nodejs
>> $ emcc $(./clang_compile_flags.py) -O1 -s USE_TYPED_ARRAYS=0 -s
>> BUILD_AS_WORKER=1 -s SIDE_MODULE=1 -s EXPORTED_FUNCTIONS="['_do_work']" -o
>> src/trash/poc/split_worker_with_common_lib/build/worker.js
>> src/trash/poc/split_worker_with_common_lib/worker.cc
>>
>> # Try again relinking the worker do_work module
>> $ emlink.py
>> src/trash/poc/split_worker_with_common_lib/build/{lib_worker,worker,worker_module}.js
>> #Main module: src/trash/poc/split_worker_with_common_lib/build/lib.js
>> #Side module: src/trash/poc/split_worker_with_common_lib/build/worker.js
>> #Output: src/trash/poc/split_worker_with_common_lib/build/worker_module.js
>> #Traceback (most recent call last):
>> # File "/home/daven/symlinks/override/emlink.py", line 33, in <module>
>> # run()
>> # File "/home/daven/symlinks/override/emlink.py", line 27, in run
>> # side = AsmModule(side)
>> # File "/home/daven/src/emscripen/emscripten/tools/asm_module.py", line
>> 24, in __init__
>> # self.mem_init_js = re.search(shared.JS.memory_initializer_pattern,
>> self.pre_js).group(0)
>> #AttributeError: 'NoneType' object has no attribute 'group'
>>
>> # Sure enough... The same is true with USE_TYPED_ARRAYS=1 as well.
>> $ grep 'no memory initializer'
>> src/trash/poc/split_worker_with_common_lib/build/worker.js
>> #/* no memory initializer */
>>
>>
>>
>>
>> On Tuesday, February 11, 2014 1:35:43 PM UTC-5, Alon Zakai wrote:
>>>
>>> This is an intended use case, that sounds like a bug. Any chance you can
>>> make a testcase showing it?
>>>
>>> Fastcomp is likely a better way to improve your iteration time though,
>>> as it is much more heavily tested than the static linking feature (at least
>>> currently). But please make a testcase if you can, that's how we can make
>>> static linking more stable.
>>>
>>> - Alon
>>>
>>>
>>>
>>> On Mon, Feb 10, 2014 at 11:30 PM, Dave Nicponski
>>> <[email protected]>wrote:
>>>
>>>> Hello all,
>>>>
>>>> Sorry for the terse email, but it's 2:30am here and i'm about to give
>>>> up for the night.
>>>> I have a project roughly as follows:
>>>>
>>>> The project contains currently a main method and a set of worker code,
>>>> intended to be run in web workers in browsers, or via the
>>>> webworker-threads
>>>> module in nodejs.
>>>>
>>>> Each of these requires a rather large support library (statically
>>>> compiled, rarely changing) to work. It contains things like the protocol
>>>> buffer libraries, for instance.
>>>>
>>>> The support libraries have been precompiled into llvm bitcode. Most of
>>>> the development iteration is done on the main and worker modules. As
>>>> such,
>>>> compilation time after modifying either of those is very important to me.
>>>>
>>>> To minimize this, here's what I had in mind.
>>>> 1) compile the static library bitcode into a javascript MAIN library.
>>>>
>>>> iterate one of:
>>>>
>>>> 2a) Modify and compile main() module into a SIDE library
>>>> 2b) run "emlink.py static.js, main.js main_out.js"
>>>>
>>>> or
>>>>
>>>> 3a) Modify and compile worker module into a SIDE library with
>>>> BUILD_AS_WORKER=1 option
>>>> 3b) run "emlink.py static.js worker.js worker_out.js"
>>>>
>>>> Sadly, step 3a reliably fails since worker.js doesn't have a memory
>>>> initialization section! This causes tools/asm_module.py to fail trying to
>>>> spot memory_initializer_patter, when in fact only
>>>> no_memory_initilizer_pattern is present in that file (worker.js).
>>>>
>>>> Is this know / meant to be an unsupported use case? Is there an easy
>>>> workaround?
>>>> Tomorrow i'll try using fastcomp (which i suspect would eventually
>>>> work), but for tonight i don't have the energy.
>>>>
>>>> I'll supply additional info tomorrow if requested.
>>>>
>>>> Thanks,
>>>> -dave-
>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google
>>>> Groups "emscripten-discuss" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>> an email to [email protected].
>>>> For more options, visit https://groups.google.com/groups/opt_out.
>>>>
>>>
>>>
--
You received this message because you are subscribed to the Google Groups
"emscripten-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/groups/opt_out.
