I am hoping we can have beta support from the 0.5 release onwards for sequential julia. We were able to do this work due to some small support from IBM. Hopefully, as more users use Julia on Power and HPC systems, there may be a way to sustain this through the right form of funding. For now, I just keep pushing julia on other architectures as a hobby. :-)
-viral > On Jul 22, 2016, at 3:54 PM, Viral Shah <vi...@fourthlion.in> wrote: > > I am hoping we can have beta support from the 0.5 release onwards for > sequential julia. We were able to do this work due to some small support from > IBM. Hopefully, as more users use Julia on Power and HPC systems, there may > be a way to sustain this through the right form of funding. For now, I just > keep building julia on newer architectures as a hobby. :-) > > -viral > > > >> On Jul 22, 2016, at 3:28 PM, Paulito Palmes <ppal...@gmail.com> wrote: >> >> Hi Viral, >> >> Thanks. I have already informed the sysad to check the time sync of the >> servers to avoid future issues. >> >> I'll try to compile in local disks and get back to you. >> >> Yes, I understand it may take some work to create the bin image but highly >> appreciate the fact that Julia will run now in power architecture. I think >> it was several months ago when a fellow IBMer from US tried to port it in >> power machine and had discussion here also but had some issues. It will be >> cool if Julia will support power architecture in time for 1.0 release... >> >> On Fri, Jul 22, 2016 at 7:51 PM, Viral Shah <vi...@mayin.org> wrote: >> It appears to me that you are building on some kind of a network filesystem >> and timestamps are getting messed up. Can you build in /tmp or somewhere, >> which may be guaranteed to have local storage and hence avoid these clock >> issues? >> >> If still doesn’t work, I will create some binaries and upload and put up all >> the instructions. Since openblas still has some issues to be fixed on power, >> I am using an ATLAS build (of which we are using the latest RC), and >> packaging it all up is going to be some work. >> >> >> -viral >> >> >> >>> On Jul 22, 2016, at 1:50 PM, Paulito Palmes <ppal...@gmail.com> wrote: >>> >>> This problem might be specific to Redhat? Unfortunately, I don't have >>> debian-based Power system running to test. >>> >>> On Fri, Jul 22, 2016 at 6:40 PM, Paulito Palmes <ppal...@gmail.com> wrote: >>> Here's the error using Power8: same error which happens during linking. >>> >>> Can you send me the tarballs of successfully compiled image in Power8? >>> >>> ----------------------------------------- >>> >>> LINK usr/lib/libjulia.so.0.5.0 >>> make[1]: warning: Clock skew detected. Your build may be incomplete. >>> make[1]: Warning: File `../src/julia_version.h' has modification time 228 s >>> in the future >>> CC ui/repl.o >>> LINK usr/bin/julia >>> make[1]: warning: Clock skew detected. Your build may be incomplete. >>> PERL base/pcre_h.jl >>> PERL base/errno_h.jl >>> PERL base/build_h.jl.phony >>> PERL base/fenv_constants.jl >>> PERL base/file_constants.jl >>> PERL base/uv_constants.jl >>> PERL base/version_git.jl.phony >>> JULIA usr/lib/julia/inference.ji >>> LLVM ERROR: unable to evaluate offset to undefined symbol '' >>> make[1]: *** [/home/paulpalm/power8/julia/usr/lib/julia/inference.ji] Error >>> 1 >>> make: *** [julia-inference] Error 2 >>> >>> >>> On Fri, Jul 22, 2016 at 5:01 PM, Paulito Palmes <ppal...@gmail.com> wrote: >>> It caused a core-dump during linking ;( in Power7 >>> >>> ... >>> CC src/cgmemmgr.o >>> LINK usr/lib/libjulia.so.0.5.0 >>> CC ui/repl.o >>> LINK usr/bin/julia >>> PERL base/pcre_h.jl >>> PERL base/errno_h.jl >>> PERL base/build_h.jl.phony >>> PERL base/fenv_constants.jl >>> PERL base/file_constants.jl >>> PERL base/uv_constants.jl >>> PERL base/version_git.jl.phony >>> JULIA usr/lib/julia/inference.ji >>> /bin/sh: line 1: 28699 Segmentation fault (core dumped) >>> /home/paulpalm/julia/usr/bin/julia -C native --output-ji >>> /home/paulpalm/julia/usr/lib/julia/inference.ji --startup-file=no coreimg.jl >>> make[1]: *** [/home/paulpalm/julia/usr/lib/julia/inference.ji] Error 139 >>> make: *** [julia-inference] Error 2 >>> >>> >>> On Fri, Jul 22, 2016 at 4:48 PM, Paulito Palmes <ppal...@gmail.com> wrote: >>> I'm compiling in both Power7 and Power8 and not yet finished. I started >>> with make -j 20 but after some time, it had errors. Then I switch to make >>> -j 1 but takes time to finish. I hope it will compile in both power7 and >>> power8. >>> >>> I'll wait and try to check if I can fix some obvious errors. Otherwise, >>> I'll ask the tarball so that I can check the difference from what I have >>> specially which objects were not compiled. >>> >>> thanks. >>> >>> On Fri, Jul 22, 2016 at 4:40 PM, Viral Shah <vi...@mayin.org> wrote: >>> Did it go through? I usually build with make -j 20 on the machines that >>> have enough cores. Let me know if you still want me to create the tarball. >>> Using ATLAS makes the whole thing a bit nonstandard for creating >>> distributions, but I’ll see what I can do if needed. >>> >>> -viral >>> >>>> On Jul 22, 2016, at 10:50 AM, Paulito Palmes <ppal...@gmail.com> wrote: >>>> >>>> Ah, forgot to update Make.user.... >>>> >>>> >>>> On Fri, Jul 22, 2016 at 3:50 PM, Paulito Palmes <ppal...@gmail.com> wrote: >>>> Hi, >>>> >>>> Got this error using Power7: Linux abc.com 3.10.0-327.4.5.el7.ppc64 #1 SMP >>>> Thu Jan 21 04:12:40 EST 2016 ppc64 ppc64 ppc64 GNU/Linux >>>> >>>> Red Hat Enterprise Linux Server release 7.2 (Maipo) >>>> -------------- >>>> make -C build/openblas-12ab1804b6ebcd38b26960d65d254314d8bc33d6/ CC=gcc >>>> FC=gfortran RANLIB=ranlib FFLAGS= -O2 -fPIC TARGET= BINARY= USE_THREAD=1 >>>> GEMM_MULTITHREADING_THRESHOLD=50 NUM_THREADS=8 NO_AFFINITY=1 >>>> DYNAMIC_ARCH=1 MAKE_NB_JOBS=0 >>>> dynamic.c: In function ‘support_avx’: >>>> dynamic.c:107:3: warning: implicit declaration of function ‘cpuid’ >>>> [-Wimplicit-function-declaration] >>>> cpuid(1, &eax, &ebx, &ecx, &edx); >>>> ^ >>>> dynamic.c:97:3: error: impossible register constraint in ‘asm’ >>>> __asm__ __volatile__ >>>> ^ >>>> make[3]: *** [dynamic.o] Error 1 >>>> make[2]: *** [libs] Error 1 >>>> \033[33;1m*** Clean the OpenBLAS build with 'make -C deps clean-openblas'. >>>> Rebuild with 'make OPENBLAS_USE_THREAD=0' if OpenBLAS had trouble linking >>>> libpthread.so, and with 'make OPENBLAS_TARGET_ARCH=NEHALEM' if there were >>>> errors building SandyBridge support. Both these options can also be used >>>> simultaneously. ***\033[0m >>>> make[1]: *** >>>> [build/openblas-12ab1804b6ebcd38b26960d65d254314d8bc33d6/libopenblas.so] >>>> Error 1 >>>> make: *** [julia-deps] Error 2 >>>> ---------------------- >>>> >>>> On Fri, Jul 22, 2016 at 3:10 PM, Paulito Palmes <ppal...@gmail.com> wrote: >>>> Hi Viral, >>>> >>>> uname -a is: >>>>> Linux abc.com 3.10.0-327.18.2.el7.ppc64le #1 SMP Fri Apr 8 05:10:45 EDT >>>>> 2016 ppc64le ppc64le ppc64le GNU/Linux >>>> >>>> and using Power8: Red Hat Enterprise Linux Server release 7.2 (Maipo) >>>> >>>> >>>> Can you send me the tarball image of Julia compiled in Power8? I would >>>> like to test it if the problem is only during compilation. Since I have no >>>> git access on this machine, I used make -C deps getall to pull the deps >>>> before copying the source files to Power8. It may be possible the deps >>>> were not properly resolved? >>>> >>>> On Fri, Jul 22, 2016 at 1:58 PM, Viral Shah <vi...@mayin.org> wrote: >>>> Not sure what to do here. Is the filesystem NFS or something? Perhaps just >>>> try building again with make -j 1. >>>> >>>> What is uname -a? >>>> >>>> -viral >>>> >>>> >>>> On Jul 22, 2016 8:52 AM, "Paulito Palmes" <ppal...@gmail.com> wrote: >>>> Hi, >>>> >>>> I'm compiling using Red Hat Enterprise Linux Server release 7.2 in Power8 >>>> and I got this error: >>>> >>>> ./julia/usr/lib64/libunwind*.a: No such file or directory >>>> >>>> >>>> the make process didn't create usr/lib64 and the libunwind*.a are in >>>> usr/lib >>>> >>>> I created the links of libunwind*.a in usr/lib64 >>>> >>>> but got these errors during compile: >>>> >>>> make[2]: Warning: File >>>> `/home/paulpalm/julia/deps/srccache/patchelf-0.9/aclocal.m4' has >>>> modification time 293 s in the future >>>> >>>> cd /home/paulpalm/julia/deps/srccache/patchelf-0.9 && /bin/sh >>>> /home/paulpalm/julia/deps/srccache/patchelf-0.9/build-aux/missing >>>> automake-1.15 --foreign >>>> >>>> /home/paulpalm/julia/deps/srccache/patchelf-0.9/build-aux/missing: line >>>> 81: automake-1.15: command not found >>>> >>>> WARNING: 'automake-1.15' is missing on your system. >>>> >>>> You should only need it if you modified 'Makefile.am' or >>>> >>>> 'configure.ac' or m4 files included by 'configure.ac'. >>>> >>>> The 'automake' program is part of the GNU Automake package: >>>> >>>> <http://www.gnu.org/software/automake> >>>> >>>> It also requires GNU Autoconf, GNU m4 and Perl in order to run: >>>> >>>> <http://www.gnu.org/software/autoconf> >>>> >>>> <http://www.gnu.org/software/m4/> >>>> >>>> <http://www.perl.org/> >>>> >>>> make[2]: *** [julia/deps/srccache/patchelf-0.9/Makefile.in] Error 1 >>>> >>>> make[1]: *** [build/patchelf-0.9/src/patchelf] Error 2 >>>> >>>> make: *** [julia-deps] Error 2 >>>> >>>> >>>> I have automake-1.13 while it looks for automake-1.15. >>>> >>>> On Wednesday, 20 July 2016 20:57:36 UTC+1, Viral Shah wrote: >>>> Just a heads up, there is a bug in atlas that causes the linalg/lq test to >>>> fail (thanks Andreas for finding!). The bugfix is going to be in the next >>>> atlas release - 3.10.3 >>>> >>>> https://sourceforge.net/p/math-atlas/bugs/254/#64f2 >>>> >>>> Turns out at this moment, both atlas and openblas are not passing julia’s >>>> tests on power. >>>> >>>> -viral >>>> >>>> >>>> >>>>> On Jul 20, 2016, at 3:11 PM, James Fairbanks <jpfai...@gmail.com> wrote: >>>>> >>>>> Updating to master and using that Make.user worked for me. >>>>> >>>>> Success! >>>>> [jpf@power8 julia-dev]$ ./julia >>>>> _ >>>>> _ _ _(_)_ | A fresh approach to technical computing >>>>> (_) | (_) (_) | Documentation: http://docs.julialang.org >>>>> _ _ _| |_ __ _ | Type "?help" for help. >>>>> | | | | | | |/ _` | | >>>>> | | |_| | | | (_| | | Version 0.5.0-dev+5549 (2016-07-20 15:47 UTC) >>>>> _/ |\__'_|_|_|\__'_| | Commit 80fcb57 (0 days old master) >>>>> |__/ | ppc64le-redhat-linux >>>>> >>>>> Thanks for going through this with me. Hopefully it will be easier for >>>>> the next person. >>>>> >>>>> On Wednesday, July 20, 2016 at 10:27:27 AM UTC-4, Viral Shah wrote: >>>>> >>>>> Julia now builds on master. No longer need to disable threading. >>>>> Make.user for me is now: >>>>> >>>>> override USE_SYSTEM_BLAS = 1 >>>>> override LIBBLAS = -L/usr/lib64/atlas -lsatlas >>>>> override LIBBLASNAME = libsatlas >>>>> override USE_BLAS64 = 0 >>>>> >>>>> -viral >>>>> >>>>> >>>>>> On Jul 19, 2016, at 7:00 PM, Viral Shah <vi...@fourthlion.in> wrote: >>>>>> >>>>>> I am testing on CentOS Linux release 7.2.1511 (AltArch) >>>>>> >>>>>> This is the line I am using: >>>>>> export LD_LIBRARY_PATH=/usr/lib64/atlas:$LD_LIBRARY_PATH >>>>>> >>>>>> Our README says LAPACK >= 3.5, so I suspect the system lapack won’t work >>>>>> out. Assuming that symbol is present in the libsatlas file, perhaps try >>>>>> also copying it in julia’s usr/lib. >>>>>> >>>>>> The other thing is to run make again to make sure that all the rpath and >>>>>> linking stuff has gone through fine. Or perhaps make cleanall and make. >>>>>> >>>>>> -viral >>>>>> >>>>>> >>>>>> >>>>>>> On Jul 19, 2016, at 6:55 PM, James Fairbanks <jpfai...@gmail.com> wrote: >>>>>>> >>>>>>> For completeness I should say that the julia executable appears not to >>>>>>> have the libjulia.so correctly linked. >>>>>>> >>>>>>> $ ./julia >>>>>>> ./julia: symbol lookup error: >>>>>>> /home/jpf/julia/usr/bin/../lib/liblapack.so: undefined symbol: lsame_ >>>>>>> $ ldd ./julia >>>>>>> linux-vdso64.so.1 => (0x00003fff9d140000) >>>>>>> libjulia.so.0.5 => not found >>>>>>> libdl.so.2 => /lib64/libdl.so.2 (0x00003fff9d100000) >>>>>>> librt.so.1 => /lib64/power8/librt.so.1 (0x00003fff9d0d0000) >>>>>>> libpthread.so.0 => /lib64/power8/libpthread.so.0 (0x00003fff9d090000) >>>>>>> libc.so.6 => /lib64/power8/libc.so.6 (0x00003fff9ceb0000) >>>>>>> /lib64/ld64.so.2 (0x00000000251e0000) >>>>>>> >>>>>>> $ ldd ./usr/lib/libjulia.so >>>>>>> linux-vdso64.so.1 => (0x00003fff99680000) >>>>>>> libLLVM-3.8.so => /home/jpf/julia/./usr/lib/libLLVM-3.8.so >>>>>>> (0x00003fff97d80000) >>>>>>> libdl.so.2 => /lib64/libdl.so.2 (0x00003fff97d40000) >>>>>>> librt.so.1 => /lib64/power8/librt.so.1 (0x00003fff97d10000) >>>>>>> libpthread.so.0 => /lib64/power8/libpthread.so.0 (0x00003fff97cd0000) >>>>>>> libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00003fff97b50000) >>>>>>> libm.so.6 => /lib64/power8/libm.so.6 (0x00003fff97a60000) >>>>>>> libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00003fff97a20000) >>>>>>> libc.so.6 => /lib64/power8/libc.so.6 (0x00003fff97840000) >>>>>>> libz.so.1 => /lib64/libz.so.1 (0x00003fff97800000) >>>>>>> /lib64/ld64.so.2 (0x0000000035a00000) >>>>>>> >>>>>>> On Tuesday, July 19, 2016 at 6:30:34 PM UTC-4, James Fairbanks wrote: >>>>>>> Adding /usr/lib64/atlas was insufficient so I created a symlink in ln >>>>>>> -s /usr/lib/atlas/libsatlas.so.3 /usr/lib/atlas/libsatlas.so >>>>>>> This allowed the build process to complete but leads to an error when >>>>>>> running the julia executable. >>>>>>> >>>>>>> The error is: >>>>>>> /home/jpf/julia/usr/bin/julia: symbol lookup error: >>>>>>> /home/jpf/julia/usr/bin/../lib/liblapack.so: undefined symbol: lsame_ >>>>>>> >>>>>>> Is it a good idea to go with the RHEL ppc64le system lapack instead of >>>>>>> having julia build lapack against system blas? >>>>>>> I have this version of lapack available from the system. >>>>>>> >>>>>>> Available Packages >>>>>>> Name : lapack >>>>>>> Arch : ppc64le >>>>>>> Version : 3.4.2 >>>>>>> Release : 5.el7 >>>>>>> Size : 4.9 M >>>>>>> Repo : rhel-7-for-power-le-rpms/7Server/ppc64le >>>>>>> >>>>>>> >>>>>>> Thanks, >>>>>>> James >>>>>>> On Tuesday, July 19, 2016 at 4:46:22 PM UTC-4, Viral Shah wrote: >>>>>>> >>>>>>> Yes - I forgot to mention that. On the machine I am using, I had to add >>>>>>> /usr/lib64/atlas or something like that to LD_LIBRARY_PATH. I can’t >>>>>>> login at the moment for some reason, or else I could have retrieved the >>>>>>> exact line. >>>>>>> >>>>>>> >>>>>>> -viral >>>>>>> >>>>>>> >>>>>>> >>>>>>>> On Jul 19, 2016, at 4:12 PM, James Fairbanks <jpfai...@gmail.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>> I set it up like you said and got the following error >>>>>>>> >>>>>>>> $ make >>>>>>>> ... a bunch of stuff successfully building ... >>>>>>>> /usr/bin/ld: cannot find -lsatlas >>>>>>>> collect2: error: ld returned 1 exit status >>>>>>>> make[1]: *** [build/lapack-3.5.0/liblapack.so] Error 1 >>>>>>>> make: *** [julia-deps] Error 2 >>>>>>>> >>>>>>>> Do I need to add something to LD_LIBRARY_PATH in the makefile or bash >>>>>>>> environment? >>>>>>>> >>>>>>>> On Tuesday, July 19, 2016 at 3:59:55 PM UTC-4, Viral Shah wrote: >>>>>>>> There is some old ATLAS stuff in there to build atlas that hasn’t been >>>>>>>> used for a very long time. We are going to delete it. If we find this >>>>>>>> atlas stuff to be generally useful, we can bring it into the Makefile >>>>>>>> - but I definitely don’t want to support a source build. >>>>>>>> >>>>>>>> override USE_SYSTEM_BLAS = 1 >>>>>>>> override LIBBLAS = -L/usr/lib64/atlas -lsatlas >>>>>>>> override LIBBLASNAME = libsatlas >>>>>>>> override USE_BLAS64 = 0 >>>>>>>> override JULIA_THREADS := 0 >>>>>>>> >>>>>>>> Use the vs/ccall-ppc branch that I just pushed, with the above >>>>>>>> Make.user. Master still doesn’t work. >>>>>>>> >>>>>>>> -viral >>>>>>>> >>>>>>>> >>>>>>>>> On Jul 19, 2016, at 3:49 PM, James Fairbanks <jpfai...@gmail.com> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>> You are overriding the SYSTEM_BLAS with ATLAS. There is a USE_ATLAS >>>>>>>>> option. Sould that work? >>>>>>>>> >>>>>>>>> the relevant part from Make.inc is >>>>>>>>> >>>>>>>>> USE_ATLAS := 0 >>>>>>>>> ATLAS_LIBDIR := $(build_libdir) >>>>>>>>> # or ATLAS_LIBDIR := /path/to/atlas >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Saturday, July 9, 2016 at 2:07:38 AM UTC-4, Viral Shah wrote: >>>>>>>>> The current master now seems to be in good shape for Power, for those >>>>>>>>> interested in trying it out. OpenBLAS is still working out a few >>>>>>>>> bugs, but in the meanwhile, I was able to successfully link against >>>>>>>>> Atlas using the following Make.user: >>>>>>>>> >>>>>>>>> override USE_SYSTEM_BLAS = 1 >>>>>>>>> override LIBBLAS = -L/usr/lib64/atlas -lsatlas >>>>>>>>> override LIBBLASNAME = libsatlas >>>>>>>>> override USE_BLAS64 = 0 >>>>>>>>> >>>>>>>>> Apart from multi-threading, all the other tests passed. >>>>>>>>> >>>>>>>>> -viral >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>>> >>>> >>>> >>> >>> >>> >>> >>> >> >> >
