This stackoverflow thread seems to indicate that it isn't - I'll see if I can set something up where I have one python process continuously invoking copy_file on libcxx.a while another process tries to build an emscripten target.
http://stackoverflow.com/questions/20873723/is-pythons-shutil-copyfile-atomic On Wednesday, January 27, 2016 at 10:26:51 AM UTC-8, arnab choudhury wrote: > > Thanks Alon - that makes sense. The part that got me thinking along this > path was - is shutil.copy_file atomic? Consider the following code from > cache.py: > > # Request a cached file. If it isn't in the cache, it will be created > with > # the given creator function > def get(self, shortname, creator, extension='.bc', what=None): > if not shortname.endswith(extension): shortname += extension > cachename = os.path.join(self.dirname, shortname) > if os.path.exists(cachename): > return cachename > if what is None: > if shortname.endswith(('.bc', '.so', '.a')): what = 'system library' > else: what = 'system asset' > message = 'generating ' + what + ': ' + shortname + '...' > logging.warn(message) > self.ensure() > temp = creator() > if temp != cachename: > shutil.copyfile(temp, cachename) > logging.warn(' '*len(message) + 'ok') > return cachename > > If both target A and B were to cross the first part of the function and > target A then copied the file to the cache and proceeded to use it, could > there be a case where target B invoked shutil.copyfile while A was using > libcxx.a? > > On Wednesday, January 27, 2016 at 10:17:00 AM UTC-8, Alon Zakai wrote: >> >> That is a possible issue in the sense that we don't have a global system >> lock for creating system libs (because it's hard to do cross-platform - if >> someone knows a good way, let me know!). If you start multiple emccs, they >> will all build libc++. A solution to that is to use embuilder.py, and to >> build libc++ and other things you need before starting all the threads. >> >> However, this still shouldn't be able to cause the problem you see - >> building libc++ should be deterministic, so all the threads building it in >> parallel should end up with the same thing. They will overwrite each other, >> but with the exact same data. I believe on all modern filesystems this is >> atomic, so all those threads should see the same full libc++. Or, they >> should see no libc++ if the OS can't do an atomic replace (i.e. if it >> deletes first then writes), but that should result in a link error, not in >> missing symbols. >> >> >> On Wed, Jan 27, 2016 at 9:54 AM, arnab choudhury <achou...@gmail.com> >> wrote: >> >>> I think I have identified what may be the problem, although since I >>> can't repro on demand, its only a hunch. The C++ codebase that I am working >>> on has multiple Emscripten CMake targets. The symbols below all come from >>> libcxx.a which is generated if it is not present in the Emscripten cache >>> (set via EM_CACHE). The CMake generator generates a ninja file which will >>> then schedule all the targets. Since my build machine always starts from a >>> clean state, all targets will simultaneously attempt to populate the >>> Emscripten cache with libcxx.a as they will all initially detect that it is >>> not present. If target A were writing libcxx.a to the cache when target B >>> was trying to read from it, we could see issues like this. >>> >>> So, I think the underlying problem is that building multiple CMake >>> targets simultaneously with Emscripten may be a problem if there is a >>> common cache - is this a known issue? Note that since I haven't root >>> caused, this is only a hunch - but from looking at cache.py, in theory at >>> least, it feels like there could be a problem. >>> >>> On Tuesday, January 26, 2016 at 6:15:53 PM UTC-8, Alon Zakai wrote: >>>> >>>> That's very strange, all of that should be fully deterministic. >>>> >>>> Perhaps build with EMCC_DEBUG=1 in the env, then if you see the error, >>>> that log could help. >>>> >>>> On Tue, Jan 26, 2016 at 6:02 PM, arnab choudhury <achou...@gmail.com> >>>> wrote: >>>> >>>>> Unfortunately I don't have a repro :-/. About one in 150 builds of a >>>>> large C++ codebase produces this error. Digging in more, it looks like on >>>>> the builds that fail, we are missing libcxx symbols. These seem to be >>>>> libcxx symbols which I think get included by default while building C++ >>>>> projects with Emscripten. >>>>> >>>>> warning: unresolved symbol: >>>>> _ZNSt3__112basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEEC1ERKS5_jjRKS4_ >>>>> warning: unresolved symbol: >>>>> _ZNSt3__119__shared_weak_count10__add_weakEv >>>>> warning: unresolved symbol: >>>>> _ZNSt3__112basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEEaSERKS5_ >>>>> warning: unresolved symbol: >>>>> _ZNSt3__113basic_ostreamIcNS_11char_traitsIcEEE5flushEv >>>>> warning: unresolved symbol: >>>>> _ZNSt3__112basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEE9push_backEc >>>>> warning: unresolved symbol: >>>>> _ZNSt3__112basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEE6resizeEjc >>>>> warning: unresolved symbol: >>>>> _ZNSt3__112basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEE6appendEPKc >>>>> warning: unresolved symbol: _ZNSt3__119__shared_weak_countD2Ev >>>>> warning: unresolved symbol: >>>>> _ZNSt3__115basic_streambufIcNS_11char_traitsIcEEE4syncEv >>>>> warning: unresolved symbol: >>>>> _ZNSt3__113basic_ostreamIcNS_11char_traitsIcEEE3putEc >>>>> warning: unresolved symbol: >>>>> _ZNSt3__115basic_streambufIcNS_11char_traitsIcEEE6xsgetnEPci >>>>> warning: unresolved symbol: _ZNSt3__18ios_base5clearEj >>>>> warning: unresolved symbol: _ZNSt3__119__shared_weak_count4lockEv >>>>> warning: unresolved symbol: >>>>> _ZTv0_n12_NSt3__113basic_ostreamIcNS_11char_traitsIcEEED1Ev >>>>> warning: unresolved symbol: >>>>> _ZNKSt3__120__codecvt_utf8_utf16IDsE10do_unshiftER11__mbstate_tPcS4_RS4_ >>>>> warning: unresolved symbol: >>>>> _ZNSt3__115basic_streambufIcNS_11char_traitsIcEEE6setbufEPci >>>>> warning: unresolved symbol: >>>>> _ZNKSt3__112basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEE7compareEPKc >>>>> warning: unresolved symbol: >>>>> _ZNKSt3__120__codecvt_utf8_utf16IDsE13do_max_lengthEv >>>>> warning: unresolved symbol: >>>>> _ZNKSt3__120__vector_base_commonILb1EE20__throw_out_of_rangeEv >>>>> warning: unresolved symbol: >>>>> _ZNSt3__113basic_ostreamIcNS_11char_traitsIcEEElsEs >>>>> warning: unresolved symbol: >>>>> _ZNSt3__113basic_ostreamIcNS_11char_traitsIcEEElsEt >>>>> warning: unresolved symbol: >>>>> _ZNSt3__113basic_ostreamIcNS_11char_traitsIcEEED0Ev >>>>> warning: unresolved symbol: >>>>> _ZNSt3__113basic_ostreamIcNS_11char_traitsIcEEElsEx >>>>> warning: unresolved symbol: >>>>> _ZNSt3__113basic_ostreamIcNS_11char_traitsIcEEE6sentryC1ERS3_ >>>>> warning: unresolved symbol: >>>>> _ZNSt3__113basic_ostreamIcNS_11char_traitsIcEEElsEb >>>>> warning: unresolved symbol: >>>>> _ZNSt3__113basic_ostreamIcNS_11char_traitsIcEEElsEf >>>>> warning: unresolved symbol: >>>>> _ZNSt3__119__shared_weak_count16__release_sharedEv >>>>> warning: unresolved symbol: >>>>> _ZNSt3__113basic_ostreamIcNS_11char_traitsIcEEElsEj >>>>> warning: unresolved symbol: >>>>> _ZNSt3__113basic_ostreamIcNS_11char_traitsIcEEElsEi >>>>> warning: unresolved symbol: >>>>> _ZNSt3__113basic_ostreamIcNS_11char_traitsIcEEElsEl >>>>> warning: unresolved symbol: >>>>> _ZTv0_n12_NSt3__113basic_istreamIcNS_11char_traitsIcEEED0Ev >>>>> warning: unresolved symbol: _ZNSt11logic_errorC2EPKc >>>>> warning: unresolved symbol: >>>>> _ZNSt3__115basic_streambufIcNS_11char_traitsIcEEEC2Ev >>>>> warning: unresolved symbol: >>>>> _ZNSt3__112basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEE7reserveEj >>>>> warning: unresolved symbol: _ZNSt3__16locale5facet16__on_zero_sharedEv >>>>> warning: unresolved symbol: _ZNSt3__112__next_primeEj >>>>> warning: unresolved symbol: >>>>> _ZNSt3__112basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEE6assignEPKc >>>>> warning: unresolved symbol: >>>>> _ZNSt3__19basic_iosIcNS_11char_traitsIcEEED2Ev >>>>> warning: unresolved symbol: >>>>> _ZNSt3__113basic_ostreamIcNS_11char_traitsIcEEElsEPNS_15basic_streambufIcS2_EE >>>>> warning: unresolved symbol: >>>>> _ZNKSt3__112basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEE4findEcj >>>>> warning: unresolved symbol: _ZNSt3__17codecvtIDsc11__mbstate_tED2Ev >>>>> warning: unresolved symbol: >>>>> _ZNSt3__113basic_ostreamIcNS_11char_traitsIcEEE6sentryD1Ev >>>>> warning: unresolved symbol: >>>>> _ZNSt3__113basic_istreamIcNS_11char_traitsIcEEE3getEv >>>>> warning: unresolved symbol: >>>>> _ZNSt3__115basic_streambufIcNS_11char_traitsIcEEE5uflowEv >>>>> warning: unresolved symbol: >>>>> _ZNSt3__113basic_istreamIcNS_11char_traitsIcEEED1Ev >>>>> warning: unresolved symbol: >>>>> _ZNKSt3__120__codecvt_utf8_utf16IDsE9do_lengthER11__mbstate_tPKcS5_j >>>>> warning: unresolved symbol: >>>>> _ZNSt3__112basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEE6insertEjPKc >>>>> warning: unresolved symbol: >>>>> _ZNSt3__112basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEE9__grow_byEjjjjjj >>>>> warning: unresolved symbol: _ZNSt13runtime_errorC1EPKc >>>>> warning: unresolved symbol: >>>>> _ZTv0_n12_NSt3__113basic_istreamIcNS_11char_traitsIcEEED1Ev >>>>> warning: unresolved symbol: >>>>> _ZNSt3__115basic_streambufIcNS_11char_traitsIcEEE5imbueERKNS_6localeE >>>>> warning: unresolved symbol: >>>>> _ZNKSt3__120__codecvt_utf8_utf16IDsE11do_encodingEv >>>>> warning: unresolved symbol: >>>>> _ZNSt3__112basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEEC1ERKS5_ >>>>> warning: unresolved symbol: >>>>> _ZNSt3__113basic_ostreamIcNS_11char_traitsIcEEElsEy >>>>> warning: unresolved symbol: _ZNSt3__19to_stringEi >>>>> warning: unresolved symbol: >>>>> _ZNSt3__112basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEE6__initEPKcjj >>>>> warning: unresolved symbol: >>>>> _ZNSt3__113basic_ostreamIcNS_11char_traitsIcEEElsEPKv >>>>> warning: unresolved symbol: >>>>> _ZNKSt3__120__codecvt_utf8_utf16IDsE16do_always_noconvEv >>>>> warning: unresolved symbol: >>>>> _ZNSt3__112basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEE6__initEjc >>>>> warning: unresolved symbol: >>>>> _ZNSt3__112basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEE6__initEPKcj >>>>> warning: unresolved symbol: _ZNSt13runtime_errorC2EPKc >>>>> warning: unresolved symbol: >>>>> _ZNSt3__113basic_ostreamIcNS_11char_traitsIcEEED1Ev >>>>> warning: unresolved symbol: >>>>> _ZNSt3__113basic_istreamIcNS_11char_traitsIcEEED2Ev >>>>> warning: unresolved symbol: >>>>> _ZNSt3__113basic_ostreamIcNS_11char_traitsIcEEED2Ev >>>>> warning: unresolved symbol: >>>>> _ZNSt11logic_errorC2ERKNSt3__112basic_stringIcNS0_11char_traitsIcEENS0_9allocatorIcEEEE >>>>> warning: unresolved symbol: >>>>> _ZNSt3__119__shared_weak_count14__release_weakEv >>>>> warning: unresolved symbol: >>>>> _ZNKSt3__120__codecvt_utf8_utf16IDsE5do_inER11__mbstate_tPKcS5_RS5_PDsS7_RS7_ >>>>> warning: unresolved symbol: _ZNKSt3__18ios_base6getlocEv >>>>> warning: unresolved symbol: _ZNSt3__112bad_weak_ptrD1Ev >>>>> warning: unresolved symbol: >>>>> _ZNSt3__115basic_streambufIcNS_11char_traitsIcEEE6xsputnEPKci >>>>> warning: unresolved symbol: _ZNKSt3__16locale9use_facetERNS0_2idE >>>>> warning: unresolved symbol: >>>>> _ZNSt3__112basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEED1Ev >>>>> warning: unresolved symbol: _ZNSt3__18ios_base4initEPv >>>>> warning: unresolved symbol: >>>>> _ZNSt3__113basic_istreamIcNS_11char_traitsIcEEED0Ev >>>>> warning: unresolved symbol: >>>>> _ZNSt3__112basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEE6appendEPKcj >>>>> warning: unresolved symbol: _ZNSt3__16localeD1Ev >>>>> warning: unresolved symbol: >>>>> _ZNSt3__18ios_base33__set_badbit_and_consider_rethrowEv >>>>> warning: unresolved symbol: >>>>> _ZNSt3__115basic_streambufIcNS_11char_traitsIcEEED2Ev >>>>> warning: unresolved symbol: >>>>> _ZTv0_n12_NSt3__113basic_ostreamIcNS_11char_traitsIcEEED0Ev >>>>> warning: unresolved symbol: >>>>> _ZNSt3__119__shared_weak_count12__add_sharedEv >>>>> warning: unresolved symbol: >>>>> _ZNKSt3__119__shared_weak_count13__get_deleterERKSt9type_info >>>>> warning: unresolved symbol: >>>>> _ZNSt3__115basic_streambufIcNS_11char_traitsIcEEE9showmanycEv >>>>> warning: unresolved symbol: _ZNSt3__13cinE >>>>> warning: unresolved symbol: _ZNSt3__14cerrE >>>>> warning: unresolved symbol: _ZNSt3__14coutE >>>>> warning: unresolved symbol: _ZNSt3__15ctypeIcE2idE >>>>> warning: unresolved symbol: _ZTINSt3__112bad_weak_ptrE >>>>> warning: unresolved symbol: >>>>> _ZTINSt3__113basic_istreamIcNS_11char_traitsIcEEEE >>>>> warning: unresolved symbol: >>>>> _ZTINSt3__113basic_ostreamIcNS_11char_traitsIcEEEE >>>>> warning: unresolved symbol: >>>>> _ZTINSt3__115basic_streambufIcNS_11char_traitsIcEEEE >>>>> warning: unresolved symbol: _ZTINSt3__119__shared_weak_countE >>>>> warning: unresolved symbol: _ZTINSt3__120__codecvt_utf8_utf16IDsEE >>>>> warning: unresolved symbol: _ZTVNSt3__112bad_weak_ptrE >>>>> >>>>> On Thursday, January 21, 2016 at 2:22:20 PM UTC-8, Alon Zakai wrote: >>>>>> >>>>>> That's very strange, and not a known issue. Do you have a testcase >>>>>> showing the problem? >>>>>> >>>>>> On Thu, Jan 21, 2016 at 12:17 PM, arnab choudhury <achou...@gmail.com >>>>>> > wrote: >>>>>> >>>>>>> Another thing to note is that only optimized builds are showing this >>>>>>> behavior - we are not seeing this on debug (unoptimized) builds. >>>>>>> >>>>>>> >>>>>>> On Thursday, January 21, 2016 at 12:04:56 PM UTC-8, arnab choudhury >>>>>>> wrote: >>>>>>>> >>>>>>>> Hey guys >>>>>>>> >>>>>>>> After updating my local fork of Emscripten to 1.35.17, I am seeing >>>>>>>> sporadic runtime failures where Emscripten identifies certain STL >>>>>>>> functions >>>>>>>> as missing: >>>>>>>> >>>>>>>> Some examples of functions: >>>>>>>> _ZNSt3__112basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEE6__initEPKcj >>>>>>>> >>>>>>>> (Demangled: std::__1::basic_string<char, std::__1::char_traits<char>, >>>>>>>> std::__1::allocator<char> >::__init(char const*, unsigned int) >>>>>>>> _ZNSt3__115basic_streambufIcNS_11char_traitsIcEEEC2Ev (Demangled: >>>>>>>> std::__1::basic_streambuf<char, std::__1::char_traits<char> >>>>>>>> >::basic_streambuf()) >>>>>>>> >>>>>>>> Is this a known issue with 1.35.17? I wouldn't expect these >>>>>>>> functions to be optimized away. Digging in a bit more, it looks like >>>>>>>> jsifier will identify these functions as missing when adding library >>>>>>>> functions to the ASM JS preamble code. >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Arnab >>>>>>>> >>>>>>> -- >>>>>>> You received this message because you are subscribed to the Google >>>>>>> Groups "emscripten-discuss" group. >>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>> send an email to emscripten-discuss+unsubscr...@googlegroups.com. >>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>> >>>>>> >>>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "emscripten-discuss" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to emscripten-discuss+unsubscr...@googlegroups.com. >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>> >>>> -- >>> You received this message because you are subscribed to the Google >>> Groups "emscripten-discuss" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to emscripten-discuss+unsubscr...@googlegroups.com. >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> -- You received this message because you are subscribed to the Google Groups "emscripten-discuss" group. To unsubscribe from this group and stop receiving emails from it, send an email to emscripten-discuss+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
