Dear Henrique/Friends, Maybe we could "reshape" freebsd[1]/musl libc implementation [2] Of course, it might take a while but it's doable
[1] https://github.com/freebsd/freebsd/search?q=sem_open&unscoped_q=sem_open [2] https://git.musl-libc.org/cgit/musl/tree/src/thread Keep Rocking, Geraldo Netto Sapere Aude => Non dvcor, dvco site: http://exdev.sf.net/ github: https://github.com/geraldo-netto linkedin: https://www.linkedin.com/in/geraldonetto/ facebook: https://web.facebook.com/geraldo.netto.161 On Sat, 2 Nov 2019 at 10:13, Henrique Fingler <[email protected]> wrote: > > Welp, spoke too soon. I was able to make it run by recompiling sklearn > without openmp using the SKLEARN_NO_OPENMP flag. > (https://scikit-learn.org/dev/developers/advanced_installation.html) > If I knew c++ better I'd implement a real sem_open, although I'm almost sure > stubbing it is enough for sklearn. > > On Saturday, November 2, 2019 at 2:49:18 AM UTC-5, Henrique Fingler wrote: >> >> Hi Waldek, >> >> Any update on this issue? I implemented a extremely horrible sem_open and >> stubbed close and unlink, but I'm getting a mmap fault somewhere in >> LogisticRegression fit >> (https://github.com/scikit-learn/scikit-learn/blob/1495f6924/sklearn/linear_model/logistic.py#L1466). >> >> 0x000000004046537b <osv::generate_signal(siginfo&, exception_frame*)+59> >> 0x00000000404653ea <osv::handle_mmap_fault(unsigned long, int, >> exception_frame*)+26> >> 0x000000004032f509 <mmu::vm_fault(unsigned long, exception_frame*)+185> >> 0x0000000040393a36 <page_fault+166> >> 0x0000000040392876 <???+1077487734> >> 0x0000100000c11e5f <???+12656223> >> >> >> On Wednesday, August 28, 2019 at 10:24:42 AM UTC-5, Waldek Kozaczuk wrote: >>> >>> Hi, >>> >>> Given you are the second person asking about it, I started implementing it >>> last night given it is not that difficult. But it would still take a bit of >>> time to make it correct and covered with unit tests. >>> >>> However, I peeked more at the details of >>> https://github.com/cloudius-systems/osv/issues/1025 and realized >>> scikit-learn uses the multiprocessing module >>> (https://docs.python.org/3.4/library/multiprocessing.html?highlight=process) >>> which allows multiprocessing based on forking new processes vs >>> multithreading. OSv supports pthreads very well even on multiple cores >>> (SMP) but being unikernel it does NOT and never intended to support >>> fork()/spawn(). That being said I suspect that even if implemented name >>> semaphores you will hit the fork() obstacle down the road. >>> >>> Based on this doc - >>> https://scipy.github.io/old-wiki/pages/ParallelProgramming, and >>> https://scipy.github.io/old-wiki/pages/Cookbook/Multithreading.html you can >>> use SciPy with multithreading which should work on OSv. >>> >>> There is also GraalVM Python which is supposed to run python much faster >>> and is focused on SciPy but it is still experimental - >>> https://www.graalvm.org/docs/reference-manual/languages/python/. I have had >>> a lot of luck running native images (JVM code AOT-compiled to machine code) >>> on OSv. >>> >>> Regards, >>> Waldek >>> >>> On Tuesday, August 27, 2019 at 10:21:22 PM UTC-4, Michael De Lucia wrote: >>>> >>>> Waldek, >>>> >>>> >>>> >>>> I finally got sklearn installed and got to the same issue you pointed out >>>> described here - https://github.com/cloudius-systems/osv/issues/1025 >>>> >>>> >>>> >>>> Has this been implemented yet? >>>> >>>> >>>> >>>> Thanks, >>>> >>>> >>>> >>>> Mike >>>> >>>> >>>> >>>> From: [email protected] <[email protected]> On Behalf Of >>>> Waldek Kozaczuk >>>> Sent: Tuesday, August 27, 2019 11:40 AM >>>> To: OSv Development <[email protected]> >>>> Subject: Re: [osv-dev] Re: Python 3 Scikitlearn >>>> >>>> >>>> >>>> It looks like you have one of the libraries, >>>> libopenblasp-r0-2ecf47d5.3.7.dev.so depends on, missing. >>>> >>>> >>>> >>>> You can use: >>>> >>>> >>>> >>>> ./scripts/manifest_from_host.sh -l >>>> /lib/python3.7/numpy/.libs/libopenblasp-r0-2ecf47d5.3.7.dev.so >>>> >>>> >>>> >>>> to see which libraries have to be added. >>>> >>>> >>>> On Tuesday, August 27, 2019 at 11:15:51 AM UTC-4, Michael De Lucia wrote: >>>> >>>> Hi Waldek, >>>> >>>> >>>> >>>> No that does not seem to be the problem. I started with just trying to see >>>> with numpy first, since that is a dependency. Here is what I did. I used >>>> your docker image build on Fedora29, which does a git and pulls down the >>>> OSv repo. Then I built the python3x osv-app in osv-apps/python3x/ on the >>>> github repo. >>>> >>>> >>>> >>>> I have python 3.7 installed on the fedora29 image and did pip3 install >>>> numpy to get that installed in my python 3.7 environment. I modified the >>>> GET file under the python3x directory to include the numpy module. >>>> >>>> >>>> >>>> Next I ran ./scripts/build image=python3x and the build completed >>>> successfully. >>>> >>>> >>>> >>>> Then I ran ./scripts/run.py –api -e “/python3 -c \”import numpy; >>>> print(numpy.__version__)\”” >>>> >>>> >>>> >>>> That is when it failed in an unresolved symbol for blas_memory_alloc >>>> >>>> >>>> >>>> [root@8ec8090052ab osv]# ./scripts/run.py --api -e "/python3 -c \"import >>>> numpy; print(numpy.__version__)\"" >>>> >>>> OSv v0.53.0-74-gef56fde7 >>>> >>>> eth0: 192.168.122.15 >>>> >>>> Booted up in 611.56 ms >>>> >>>> /lib/python3.7/numpy/.libs/libopenblasp-r0-2ecf47d5.3.7.dev.so: failed >>>> looking up symbol blas_memory_alloc >>>> >>>> >>>> >>>> [backtrace] >>>> >>>> 0x0000000040356209 <elf::object::symbol(unsigned int, bool)+969> >>>> >>>> 0x00000000403562cf <elf::object::resolve_pltgot(unsigned int)+127> >>>> >>>> 0x0000000040356494 <elf_resolve_pltgot+52> >>>> >>>> 0x000000004039b82f <???+1077524527> >>>> >>>> 0x000020000077ff6f <???+7864175> >>>> >>>> 0x0000000000000003 <???+3> >>>> >>>> >>>> >>>> Any ideas on how to resolve this? >>>> >>>> >>>> >>>> Thanks, >>>> >>>> >>>> >>>> Mike >>>> >>>> >>>> >>>> From: [email protected] <[email protected]> On Behalf Of >>>> Waldek Kozaczuk >>>> Sent: Monday, August 26, 2019 6:03 PM >>>> To: OSv Development <[email protected]> >>>> Subject: Re: [osv-dev] Re: Python 3 Scikitlearn >>>> >>>> >>>> >>>> Is it possibly the same issue as described here - >>>> https://github.com/cloudius-systems/osv/issues/1025? >>>> >>>> On Wednesday, August 21, 2019 at 10:08:15 PM UTC-4, Michael De Lucia wrote: >>>> >>>> I Waldek, >>>> >>>> >>>> >>>> I will get the exact command and stack trace to you on Friday. >>>> >>>> >>>> >>>> Thanks, >>>> >>>> >>>> >>>> Mike >>>> >>>> Sent from my iPhone >>>> >>>> >>>> On Aug 21, 2019, at 9:39 PM, Waldek Kozaczuk <[email protected]> wrote: >>>> >>>> Hi, >>>> >>>> >>>> >>>> Can you provide exact build command you used to build the image? Also, can >>>> you provide the exact stack trace so we know which symbol is was >>>> unresolved? >>>> >>>> >>>> >>>> Thanks, >>>> >>>> Waldek >>>> >>>> On Wednesday, August 21, 2019 at 9:19:36 PM UTC-4, [email protected] wrote: >>>> >>>> Hi, >>>> >>>> >>>> >>>> I tried to use the OSv app for python 3 support. I built the image with >>>> the system installed python 3 version. I also installed scikit learn and >>>> added that to the modules to keep from the site packages. Scikit also >>>> relies on numpy and scipy. However, I got it to import and build an image. >>>> But when I tried to run it, failed with an unresolved symbol. I have a >>>> centos system and was building within the docker container fedora 29 >>>> template. Any suggestions or thoughts on why this would fail? >>>> >>>> >>>> >>>> Thanks, >>>> >>>> >>>> >>>> Mike >>>> >>>> -- >>>> You received this message because you are subscribed to a topic in the >>>> Google Groups "OSv Development" group. >>>> To unsubscribe from this topic, visit >>>> https://groups.google.com/d/topic/osv-dev/V9fch2WFX7o/unsubscribe. >>>> To unsubscribe from this group and all its topics, send an email to >>>> [email protected]. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/osv-dev/61cb76c8-63f4-490c-b040-5634148c3930%40googlegroups.com. >>>> >>>> -- >>>> You received this message because you are subscribed to a topic in the >>>> Google Groups "OSv Development" group. >>>> To unsubscribe from this topic, visit >>>> https://groups.google.com/d/topic/osv-dev/V9fch2WFX7o/unsubscribe. >>>> To unsubscribe from this group and all its topics, send an email to >>>> [email protected]. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/osv-dev/dba8eead-7d65-4f86-bef0-98eb80925d6d%40googlegroups.com. >>>> >>>> -- >>>> You received this message because you are subscribed to a topic in the >>>> Google Groups "OSv Development" group. >>>> To unsubscribe from this topic, visit >>>> https://groups.google.com/d/topic/osv-dev/V9fch2WFX7o/unsubscribe. >>>> To unsubscribe from this group and all its topics, send an email to >>>> [email protected]. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/osv-dev/e1bdaf15-e259-4bad-8beb-15f81c4506f8%40googlegroups.com. > > -- > You received this message because you are subscribed to the Google Groups > "OSv Development" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/osv-dev/e8ecc4f5-ccac-4f5d-84a1-6f50e4586f10%40googlegroups.com. -- You received this message because you are subscribed to the Google Groups "OSv Development" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/osv-dev/CACepeQ9fJ72SnzfUC6N6HNqf%2BL3gWqhyKsvtT0Za%3DD2tq2W9xg%40mail.gmail.com.
