Thanks Alon - that makes sense. The part that got me thinking along this
path was - is shutil.copy_file atomic? Consider the following code from
cache.py:
# Request a cached file. If it isn't in the cache, it will be created with
# the given creator function
def get(self, shortname, creator, extension='.bc', what=None):
if not shortname.endswith(extension): shortname += extension
cachename = os.path.join(self.dirname, shortname)
if os.path.exists(cachename):
return cachename
if what is None:
if shortname.endswith(('.bc', '.so', '.a')): what = 'system library'
else: what = 'system asset'
message = 'generating ' + what + ': ' + shortname + '...'
logging.warn(message)
self.ensure()
temp = creator()
if temp != cachename:
shutil.copyfile(temp, cachename)
logging.warn(' '*len(message) + 'ok')
return cachename
If both target A and B were to cross the first part of the function and
target A then copied the file to the cache and proceeded to use it, could
there be a case where target B invoked shutil.copyfile while A was using
libcxx.a?
On Wednesday, January 27, 2016 at 10:17:00 AM UTC-8, Alon Zakai wrote:
>
> That is a possible issue in the sense that we don't have a global system
> lock for creating system libs (because it's hard to do cross-platform - if
> someone knows a good way, let me know!). If you start multiple emccs, they
> will all build libc++. A solution to that is to use embuilder.py, and to
> build libc++ and other things you need before starting all the threads.
>
> However, this still shouldn't be able to cause the problem you see -
> building libc++ should be deterministic, so all the threads building it in
> parallel should end up with the same thing. They will overwrite each other,
> but with the exact same data. I believe on all modern filesystems this is
> atomic, so all those threads should see the same full libc++. Or, they
> should see no libc++ if the OS can't do an atomic replace (i.e. if it
> deletes first then writes), but that should result in a link error, not in
> missing symbols.
>
>
> On Wed, Jan 27, 2016 at 9:54 AM, arnab choudhury <achou...@gmail.com
> <javascript:>> wrote:
>
>> I think I have identified what may be the problem, although since I can't
>> repro on demand, its only a hunch. The C++ codebase that I am working on
>> has multiple Emscripten CMake targets. The symbols below all come from
>> libcxx.a which is generated if it is not present in the Emscripten cache
>> (set via EM_CACHE). The CMake generator generates a ninja file which will
>> then schedule all the targets. Since my build machine always starts from a
>> clean state, all targets will simultaneously attempt to populate the
>> Emscripten cache with libcxx.a as they will all initially detect that it is
>> not present. If target A were writing libcxx.a to the cache when target B
>> was trying to read from it, we could see issues like this.
>>
>> So, I think the underlying problem is that building multiple CMake
>> targets simultaneously with Emscripten may be a problem if there is a
>> common cache - is this a known issue? Note that since I haven't root
>> caused, this is only a hunch - but from looking at cache.py, in theory at
>> least, it feels like there could be a problem.
>>
>> On Tuesday, January 26, 2016 at 6:15:53 PM UTC-8, Alon Zakai wrote:
>>>
>>> That's very strange, all of that should be fully deterministic.
>>>
>>> Perhaps build with EMCC_DEBUG=1 in the env, then if you see the error,
>>> that log could help.
>>>
>>> On Tue, Jan 26, 2016 at 6:02 PM, arnab choudhury <achou...@gmail.com>
>>> wrote:
>>>
>>>> Unfortunately I don't have a repro :-/. About one in 150 builds of a
>>>> large C++ codebase produces this error. Digging in more, it looks like on
>>>> the builds that fail, we are missing libcxx symbols. These seem to be
>>>> libcxx symbols which I think get included by default while building C++
>>>> projects with Emscripten.
>>>>
>>>> warning: unresolved symbol:
>>>> _ZNSt3__112basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEEC1ERKS5_jjRKS4_
>>>> warning: unresolved symbol: _ZNSt3__119__shared_weak_count10__add_weakEv
>>>> warning: unresolved symbol:
>>>> _ZNSt3__112basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEEaSERKS5_
>>>> warning: unresolved symbol:
>>>> _ZNSt3__113basic_ostreamIcNS_11char_traitsIcEEE5flushEv
>>>> warning: unresolved symbol:
>>>> _ZNSt3__112basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEE9push_backEc
>>>> warning: unresolved symbol:
>>>> _ZNSt3__112basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEE6resizeEjc
>>>> warning: unresolved symbol:
>>>> _ZNSt3__112basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEE6appendEPKc
>>>> warning: unresolved symbol: _ZNSt3__119__shared_weak_countD2Ev
>>>> warning: unresolved symbol:
>>>> _ZNSt3__115basic_streambufIcNS_11char_traitsIcEEE4syncEv
>>>> warning: unresolved symbol:
>>>> _ZNSt3__113basic_ostreamIcNS_11char_traitsIcEEE3putEc
>>>> warning: unresolved symbol:
>>>> _ZNSt3__115basic_streambufIcNS_11char_traitsIcEEE6xsgetnEPci
>>>> warning: unresolved symbol: _ZNSt3__18ios_base5clearEj
>>>> warning: unresolved symbol: _ZNSt3__119__shared_weak_count4lockEv
>>>> warning: unresolved symbol:
>>>> _ZTv0_n12_NSt3__113basic_ostreamIcNS_11char_traitsIcEEED1Ev
>>>> warning: unresolved symbol:
>>>> _ZNKSt3__120__codecvt_utf8_utf16IDsE10do_unshiftER11__mbstate_tPcS4_RS4_
>>>> warning: unresolved symbol:
>>>> _ZNSt3__115basic_streambufIcNS_11char_traitsIcEEE6setbufEPci
>>>> warning: unresolved symbol:
>>>> _ZNKSt3__112basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEE7compareEPKc
>>>> warning: unresolved symbol:
>>>> _ZNKSt3__120__codecvt_utf8_utf16IDsE13do_max_lengthEv
>>>> warning: unresolved symbol:
>>>> _ZNKSt3__120__vector_base_commonILb1EE20__throw_out_of_rangeEv
>>>> warning: unresolved symbol:
>>>> _ZNSt3__113basic_ostreamIcNS_11char_traitsIcEEElsEs
>>>> warning: unresolved symbol:
>>>> _ZNSt3__113basic_ostreamIcNS_11char_traitsIcEEElsEt
>>>> warning: unresolved symbol:
>>>> _ZNSt3__113basic_ostreamIcNS_11char_traitsIcEEED0Ev
>>>> warning: unresolved symbol:
>>>> _ZNSt3__113basic_ostreamIcNS_11char_traitsIcEEElsEx
>>>> warning: unresolved symbol:
>>>> _ZNSt3__113basic_ostreamIcNS_11char_traitsIcEEE6sentryC1ERS3_
>>>> warning: unresolved symbol:
>>>> _ZNSt3__113basic_ostreamIcNS_11char_traitsIcEEElsEb
>>>> warning: unresolved symbol:
>>>> _ZNSt3__113basic_ostreamIcNS_11char_traitsIcEEElsEf
>>>> warning: unresolved symbol:
>>>> _ZNSt3__119__shared_weak_count16__release_sharedEv
>>>> warning: unresolved symbol:
>>>> _ZNSt3__113basic_ostreamIcNS_11char_traitsIcEEElsEj
>>>> warning: unresolved symbol:
>>>> _ZNSt3__113basic_ostreamIcNS_11char_traitsIcEEElsEi
>>>> warning: unresolved symbol:
>>>> _ZNSt3__113basic_ostreamIcNS_11char_traitsIcEEElsEl
>>>> warning: unresolved symbol:
>>>> _ZTv0_n12_NSt3__113basic_istreamIcNS_11char_traitsIcEEED0Ev
>>>> warning: unresolved symbol: _ZNSt11logic_errorC2EPKc
>>>> warning: unresolved symbol:
>>>> _ZNSt3__115basic_streambufIcNS_11char_traitsIcEEEC2Ev
>>>> warning: unresolved symbol:
>>>> _ZNSt3__112basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEE7reserveEj
>>>> warning: unresolved symbol: _ZNSt3__16locale5facet16__on_zero_sharedEv
>>>> warning: unresolved symbol: _ZNSt3__112__next_primeEj
>>>> warning: unresolved symbol:
>>>> _ZNSt3__112basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEE6assignEPKc
>>>> warning: unresolved symbol:
>>>> _ZNSt3__19basic_iosIcNS_11char_traitsIcEEED2Ev
>>>> warning: unresolved symbol:
>>>> _ZNSt3__113basic_ostreamIcNS_11char_traitsIcEEElsEPNS_15basic_streambufIcS2_EE
>>>> warning: unresolved symbol:
>>>> _ZNKSt3__112basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEE4findEcj
>>>> warning: unresolved symbol: _ZNSt3__17codecvtIDsc11__mbstate_tED2Ev
>>>> warning: unresolved symbol:
>>>> _ZNSt3__113basic_ostreamIcNS_11char_traitsIcEEE6sentryD1Ev
>>>> warning: unresolved symbol:
>>>> _ZNSt3__113basic_istreamIcNS_11char_traitsIcEEE3getEv
>>>> warning: unresolved symbol:
>>>> _ZNSt3__115basic_streambufIcNS_11char_traitsIcEEE5uflowEv
>>>> warning: unresolved symbol:
>>>> _ZNSt3__113basic_istreamIcNS_11char_traitsIcEEED1Ev
>>>> warning: unresolved symbol:
>>>> _ZNKSt3__120__codecvt_utf8_utf16IDsE9do_lengthER11__mbstate_tPKcS5_j
>>>> warning: unresolved symbol:
>>>> _ZNSt3__112basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEE6insertEjPKc
>>>> warning: unresolved symbol:
>>>> _ZNSt3__112basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEE9__grow_byEjjjjjj
>>>> warning: unresolved symbol: _ZNSt13runtime_errorC1EPKc
>>>> warning: unresolved symbol:
>>>> _ZTv0_n12_NSt3__113basic_istreamIcNS_11char_traitsIcEEED1Ev
>>>> warning: unresolved symbol:
>>>> _ZNSt3__115basic_streambufIcNS_11char_traitsIcEEE5imbueERKNS_6localeE
>>>> warning: unresolved symbol:
>>>> _ZNKSt3__120__codecvt_utf8_utf16IDsE11do_encodingEv
>>>> warning: unresolved symbol:
>>>> _ZNSt3__112basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEEC1ERKS5_
>>>> warning: unresolved symbol:
>>>> _ZNSt3__113basic_ostreamIcNS_11char_traitsIcEEElsEy
>>>> warning: unresolved symbol: _ZNSt3__19to_stringEi
>>>> warning: unresolved symbol:
>>>> _ZNSt3__112basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEE6__initEPKcjj
>>>> warning: unresolved symbol:
>>>> _ZNSt3__113basic_ostreamIcNS_11char_traitsIcEEElsEPKv
>>>> warning: unresolved symbol:
>>>> _ZNKSt3__120__codecvt_utf8_utf16IDsE16do_always_noconvEv
>>>> warning: unresolved symbol:
>>>> _ZNSt3__112basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEE6__initEjc
>>>> warning: unresolved symbol:
>>>> _ZNSt3__112basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEE6__initEPKcj
>>>> warning: unresolved symbol: _ZNSt13runtime_errorC2EPKc
>>>> warning: unresolved symbol:
>>>> _ZNSt3__113basic_ostreamIcNS_11char_traitsIcEEED1Ev
>>>> warning: unresolved symbol:
>>>> _ZNSt3__113basic_istreamIcNS_11char_traitsIcEEED2Ev
>>>> warning: unresolved symbol:
>>>> _ZNSt3__113basic_ostreamIcNS_11char_traitsIcEEED2Ev
>>>> warning: unresolved symbol:
>>>> _ZNSt11logic_errorC2ERKNSt3__112basic_stringIcNS0_11char_traitsIcEENS0_9allocatorIcEEEE
>>>> warning: unresolved symbol:
>>>> _ZNSt3__119__shared_weak_count14__release_weakEv
>>>> warning: unresolved symbol:
>>>> _ZNKSt3__120__codecvt_utf8_utf16IDsE5do_inER11__mbstate_tPKcS5_RS5_PDsS7_RS7_
>>>> warning: unresolved symbol: _ZNKSt3__18ios_base6getlocEv
>>>> warning: unresolved symbol: _ZNSt3__112bad_weak_ptrD1Ev
>>>> warning: unresolved symbol:
>>>> _ZNSt3__115basic_streambufIcNS_11char_traitsIcEEE6xsputnEPKci
>>>> warning: unresolved symbol: _ZNKSt3__16locale9use_facetERNS0_2idE
>>>> warning: unresolved symbol:
>>>> _ZNSt3__112basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEED1Ev
>>>> warning: unresolved symbol: _ZNSt3__18ios_base4initEPv
>>>> warning: unresolved symbol:
>>>> _ZNSt3__113basic_istreamIcNS_11char_traitsIcEEED0Ev
>>>> warning: unresolved symbol:
>>>> _ZNSt3__112basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEE6appendEPKcj
>>>> warning: unresolved symbol: _ZNSt3__16localeD1Ev
>>>> warning: unresolved symbol:
>>>> _ZNSt3__18ios_base33__set_badbit_and_consider_rethrowEv
>>>> warning: unresolved symbol:
>>>> _ZNSt3__115basic_streambufIcNS_11char_traitsIcEEED2Ev
>>>> warning: unresolved symbol:
>>>> _ZTv0_n12_NSt3__113basic_ostreamIcNS_11char_traitsIcEEED0Ev
>>>> warning: unresolved symbol:
>>>> _ZNSt3__119__shared_weak_count12__add_sharedEv
>>>> warning: unresolved symbol:
>>>> _ZNKSt3__119__shared_weak_count13__get_deleterERKSt9type_info
>>>> warning: unresolved symbol:
>>>> _ZNSt3__115basic_streambufIcNS_11char_traitsIcEEE9showmanycEv
>>>> warning: unresolved symbol: _ZNSt3__13cinE
>>>> warning: unresolved symbol: _ZNSt3__14cerrE
>>>> warning: unresolved symbol: _ZNSt3__14coutE
>>>> warning: unresolved symbol: _ZNSt3__15ctypeIcE2idE
>>>> warning: unresolved symbol: _ZTINSt3__112bad_weak_ptrE
>>>> warning: unresolved symbol:
>>>> _ZTINSt3__113basic_istreamIcNS_11char_traitsIcEEEE
>>>> warning: unresolved symbol:
>>>> _ZTINSt3__113basic_ostreamIcNS_11char_traitsIcEEEE
>>>> warning: unresolved symbol:
>>>> _ZTINSt3__115basic_streambufIcNS_11char_traitsIcEEEE
>>>> warning: unresolved symbol: _ZTINSt3__119__shared_weak_countE
>>>> warning: unresolved symbol: _ZTINSt3__120__codecvt_utf8_utf16IDsEE
>>>> warning: unresolved symbol: _ZTVNSt3__112bad_weak_ptrE
>>>>
>>>> On Thursday, January 21, 2016 at 2:22:20 PM UTC-8, Alon Zakai wrote:
>>>>>
>>>>> That's very strange, and not a known issue. Do you have a testcase
>>>>> showing the problem?
>>>>>
>>>>> On Thu, Jan 21, 2016 at 12:17 PM, arnab choudhury <achou...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Another thing to note is that only optimized builds are showing this
>>>>>> behavior - we are not seeing this on debug (unoptimized) builds.
>>>>>>
>>>>>>
>>>>>> On Thursday, January 21, 2016 at 12:04:56 PM UTC-8, arnab choudhury
>>>>>> wrote:
>>>>>>>
>>>>>>> Hey guys
>>>>>>>
>>>>>>> After updating my local fork of Emscripten to 1.35.17, I am seeing
>>>>>>> sporadic runtime failures where Emscripten identifies certain STL
>>>>>>> functions
>>>>>>> as missing:
>>>>>>>
>>>>>>> Some examples of functions:
>>>>>>> _ZNSt3__112basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEE6__initEPKcj
>>>>>>>
>>>>>>> (Demangled: std::__1::basic_string<char, std::__1::char_traits<char>,
>>>>>>> std::__1::allocator<char> >::__init(char const*, unsigned int)
>>>>>>> _ZNSt3__115basic_streambufIcNS_11char_traitsIcEEEC2Ev (Demangled:
>>>>>>> std::__1::basic_streambuf<char, std::__1::char_traits<char>
>>>>>>> >::basic_streambuf())
>>>>>>>
>>>>>>> Is this a known issue with 1.35.17? I wouldn't expect these
>>>>>>> functions to be optimized away. Digging in a bit more, it looks like
>>>>>>> jsifier will identify these functions as missing when adding library
>>>>>>> functions to the ASM JS preamble code.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Arnab
>>>>>>>
>>>>>> --
>>>>>> You received this message because you are subscribed to the Google
>>>>>> Groups "emscripten-discuss" group.
>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>> send an email to emscripten-discuss+unsubscr...@googlegroups.com.
>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>>
>>>>>
>>>>> --
>>>> You received this message because you are subscribed to the Google
>>>> Groups "emscripten-discuss" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>> an email to emscripten-discuss+unsubscr...@googlegroups.com.
>>>> For more options, visit https://groups.google.com/d/optout.
>>>>
>>>
>>> --
>> You received this message because you are subscribed to the Google Groups
>> "emscripten-discuss" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to emscripten-discuss+unsubscr...@googlegroups.com <javascript:>.
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
--
You received this message because you are subscribed to the Google Groups
"emscripten-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to emscripten-discuss+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
