Welp, spoke too soon. I was able to make it run by recompiling sklearn without openmp using the SKLEARN_NO_OPENMP flag. ( https://scikit-learn.org/dev/developers/advanced_installation.html) If I knew c++ better I'd implement a real sem_open, although I'm almost sure stubbing it is enough for sklearn.
On Saturday, November 2, 2019 at 2:49:18 AM UTC-5, Henrique Fingler wrote: > > Hi Waldek, > > Any update on this issue? I implemented a extremely horrible sem_open and > stubbed close and unlink, but I'm getting a mmap fault somewhere in > LogisticRegression > fit ( > https://github.com/scikit-learn/scikit-learn/blob/1495f6924/sklearn/linear_model/logistic.py#L1466 > ). > > 0x000000004046537b <osv::generate_signal(siginfo&, exception_frame*)+59> > 0x00000000404653ea <osv::handle_mmap_fault(unsigned long, int, > exception_frame*)+26> > 0x000000004032f509 <mmu::vm_fault(unsigned long, exception_frame*)+185> > 0x0000000040393a36 <page_fault+166> > 0x0000000040392876 <???+1077487734> > 0x0000100000c11e5f <???+12656223> > > > On Wednesday, August 28, 2019 at 10:24:42 AM UTC-5, Waldek Kozaczuk wrote: >> >> Hi, >> >> Given you are the second person asking about it, I started implementing >> it last night given it is not that difficult. But it would still take a bit >> of time to make it correct and covered with unit tests. >> >> However, I peeked more at the details of >> https://github.com/cloudius-systems/osv/issues/1025 and realized >> scikit-learn uses the multiprocessing module ( >> https://docs.python.org/3.4/library/multiprocessing.html?highlight=process) >> which allows multiprocessing based on forking new processes vs >> multithreading. OSv supports pthreads very well even on multiple cores >> (SMP) but being unikernel it *does NOT and never intended to support >> fork()/spawn()*. That being said I suspect that even if implemented name >> semaphores you will hit the fork() obstacle down the road. >> >> Based on this doc - >> https://scipy.github.io/old-wiki/pages/ParallelProgramming, and >> https://scipy.github.io/old-wiki/pages/Cookbook/Multithreading.html you >> can use SciPy with multithreading which should work on OSv. >> >> There is also GraalVM Python which is supposed to run python much faster >> and is focused on SciPy but it is still experimental - >> https://www.graalvm.org/docs/reference-manual/languages/python/. I have >> had a lot of luck running native images (JVM code AOT-compiled to machine >> code) on OSv. >> >> Regards, >> Waldek >> >> On Tuesday, August 27, 2019 at 10:21:22 PM UTC-4, Michael De Lucia wrote: >>> >>> Waldek, >>> >>> >>> >>> I finally got sklearn installed and got to the same issue you pointed >>> out described here - https://github.com/cloudius-systems/osv/issues/1025 >>> >>> >>> >>> Has this been implemented yet? >>> >>> >>> >>> Thanks, >>> >>> >>> >>> Mike >>> >>> >>> >>> *From:* [email protected] <[email protected]> *On Behalf Of >>> *Waldek Kozaczuk >>> *Sent:* Tuesday, August 27, 2019 11:40 AM >>> *To:* OSv Development <[email protected]> >>> *Subject:* Re: [osv-dev] Re: Python 3 Scikitlearn >>> >>> >>> >>> It looks like you have one of the libraries, >>> libopenblasp-r0-2ecf47d5.3.7.dev.so >>> <http://www.google.com/url?q=http%3A%2F%2Flibopenblasp-r0-2ecf47d5.3.7.dev.so&sa=D&sntz=1&usg=AFQjCNEzhQZhSHkxBkhVADvoaOnGZdTRaQ> >>> depends >>> on, missing. >>> >>> >>> >>> You can use: >>> >>> >>> >>> ./scripts/manifest_from_host.sh -l /lib/python3.7/numpy/.libs/ >>> libopenblasp-r0-2ecf47d5.3.7.dev.so >>> <http://www.google.com/url?q=http%3A%2F%2Flibopenblasp-r0-2ecf47d5.3.7.dev.so&sa=D&sntz=1&usg=AFQjCNEzhQZhSHkxBkhVADvoaOnGZdTRaQ> >>> >>> >>> >>> to see which libraries have to be added. >>> >>> >>> On Tuesday, August 27, 2019 at 11:15:51 AM UTC-4, Michael De Lucia wrote: >>> >>> Hi Waldek, >>> >>> >>> >>> No that does not seem to be the problem. I started with just trying to >>> see with numpy first, since that is a dependency. Here is what I did. I >>> used your docker image build on Fedora29, which does a git and pulls down >>> the OSv repo. Then I built the python3x osv-app in osv-apps/python3x/ on >>> the github repo. >>> >>> >>> >>> I have python 3.7 installed on the fedora29 image and did pip3 install >>> numpy to get that installed in my python 3.7 environment. I modified the >>> GET file under the python3x directory to include the numpy module. >>> >>> >>> >>> Next I ran ./scripts/build image=python3x and the build completed >>> successfully. >>> >>> >>> >>> Then I ran ./scripts/run.py –api -e “/python3 -c \”import numpy; >>> print(numpy.__version__)\”” >>> >>> >>> >>> That is when it failed in an unresolved symbol for blas_memory_alloc >>> >>> >>> >>> [root@8ec8090052ab osv]# ./scripts/run.py --api -e "/python3 -c \"import >>> numpy; print(numpy.__version__)\"" >>> >>> OSv v0.53.0-74-gef56fde7 >>> >>> eth0: 192.168.122.15 >>> >>> Booted up in 611.56 ms >>> >>> /lib/python3.7/numpy/.libs/libopenblasp-r0-2ecf47d5.3.7.dev.so >>> <http://www.google.com/url?q=http%3A%2F%2Flibopenblasp-r0-2ecf47d5.3.7.dev.so&sa=D&sntz=1&usg=AFQjCNEzhQZhSHkxBkhVADvoaOnGZdTRaQ>: >>> >>> failed looking up symbol blas_memory_alloc >>> >>> >>> >>> [backtrace] >>> >>> 0x0000000040356209 <elf::object::symbol(unsigned int, bool)+969> >>> >>> 0x00000000403562cf <elf::object::resolve_pltgot(unsigned int)+127> >>> >>> 0x0000000040356494 <elf_resolve_pltgot+52> >>> >>> 0x000000004039b82f <???+1077524527> >>> >>> 0x000020000077ff6f <???+7864175> >>> >>> 0x0000000000000003 <???+3> >>> >>> >>> >>> Any ideas on how to resolve this? >>> >>> >>> >>> Thanks, >>> >>> >>> >>> Mike >>> >>> >>> >>> *From:* [email protected] <[email protected]> *On Behalf Of >>> *Waldek Kozaczuk >>> *Sent:* Monday, August 26, 2019 6:03 PM >>> *To:* OSv Development <[email protected]> >>> *Subject:* Re: [osv-dev] Re: Python 3 Scikitlearn >>> >>> >>> >>> Is it possibly the same issue as described here - >>> https://github.com/cloudius-systems/osv/issues/1025? >>> >>> On Wednesday, August 21, 2019 at 10:08:15 PM UTC-4, Michael De Lucia >>> wrote: >>> >>> I Waldek, >>> >>> >>> >>> I will get the exact command and stack trace to you on Friday. >>> >>> >>> >>> Thanks, >>> >>> >>> >>> Mike >>> >>> Sent from my iPhone >>> >>> >>> On Aug 21, 2019, at 9:39 PM, Waldek Kozaczuk <[email protected]> wrote: >>> >>> Hi, >>> >>> >>> >>> Can you provide exact build command you used to build the image? Also, >>> can you provide the exact stack trace so we know which symbol is was >>> unresolved? >>> >>> >>> >>> Thanks, >>> >>> Waldek >>> >>> On Wednesday, August 21, 2019 at 9:19:36 PM UTC-4, [email protected] >>> wrote: >>> >>> Hi, >>> >>> >>> >>> I tried to use the OSv app for python 3 support. I built the image with >>> the system installed python 3 version. I also installed scikit learn and >>> added that to the modules to keep from the site packages. Scikit also >>> relies on numpy and scipy. However, I got it to import and build an image. >>> But when I tried to run it, failed with an unresolved symbol. I have a >>> centos system and was building within the docker container fedora 29 >>> template. Any suggestions or thoughts on why this would fail? >>> >>> >>> >>> Thanks, >>> >>> >>> >>> Mike >>> >>> -- >>> You received this message because you are subscribed to a topic in the >>> Google Groups "OSv Development" group. >>> To unsubscribe from this topic, visit >>> https://groups.google.com/d/topic/osv-dev/V9fch2WFX7o/unsubscribe. >>> To unsubscribe from this group and all its topics, send an email to >>> [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/osv-dev/61cb76c8-63f4-490c-b040-5634148c3930%40googlegroups.com >>> >>> <https://groups.google.com/d/msgid/osv-dev/61cb76c8-63f4-490c-b040-5634148c3930%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >>> -- >>> You received this message because you are subscribed to a topic in the >>> Google Groups "OSv Development" group. >>> To unsubscribe from this topic, visit >>> https://groups.google.com/d/topic/osv-dev/V9fch2WFX7o/unsubscribe. >>> To unsubscribe from this group and all its topics, send an email to >>> [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/osv-dev/dba8eead-7d65-4f86-bef0-98eb80925d6d%40googlegroups.com >>> >>> <https://groups.google.com/d/msgid/osv-dev/dba8eead-7d65-4f86-bef0-98eb80925d6d%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >>> -- >>> You received this message because you are subscribed to a topic in the >>> Google Groups "OSv Development" group. >>> To unsubscribe from this topic, visit >>> https://groups.google.com/d/topic/osv-dev/V9fch2WFX7o/unsubscribe. >>> To unsubscribe from this group and all its topics, send an email to >>> [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/osv-dev/e1bdaf15-e259-4bad-8beb-15f81c4506f8%40googlegroups.com >>> >>> <https://groups.google.com/d/msgid/osv-dev/e1bdaf15-e259-4bad-8beb-15f81c4506f8%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >> -- You received this message because you are subscribed to the Google Groups "OSv Development" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/osv-dev/e8ecc4f5-ccac-4f5d-84a1-6f50e4586f10%40googlegroups.com.
