Can you try running python from gdb? This may give some information. On Sat, Oct 15, 2016, Michael Harradon wrote: > I'm hitting this problem myself, as well. > > Ubuntu 14.04 > gcc 4.8.4 > amdlibm-3.1 > libopenblas-dev 0.2.8 > > Would appreciate any suggestions - running with amdlibm off for the moment. > > Best, > Michael > > On Friday, March 11, 2016 at 12:49:04 PM UTC-5, Pascal Lamblin wrote: > > > > On Fri, Mar 11, 2016, Juan Camilo Gamboa Higuera wrote: > > > I tried changing the architecture, but this still produces the > > segmentation > > > fault. How can I see all the flags that are passed to the gcc command > > from > > > theano? > > > > You can change the log level of the following call [1], or just add > > "print(cmd)" there. > > > > [1] > > https://github.com/Theano/Theano/blob/master/theano/gof/cmodule.py#L2165 > > > > > > > > -- Juan > > > > > > On Thursday, March 10, 2016 at 1:42:59 PM UTC-5, Pascal Lamblin wrote: > > > > > > > > Can you try passing explicitly a target architecture to g++ in theano, > > > > for instance THEANO_FLAGS=gcc.cxxflags='-march=core2'? > > > > > > > > By default, Theano tries to emulate '-march=native', which might cause > > > > trouble. > > > > > > > > On Thu, Mar 10, 2016, Juan Camilo Gamboa Higuera wrote: > > > > > I tried another machine with he following software: > > > > > Ubuntu 14.04.4, 4.2 kernel > > > > > gcc 4.8.4 > > > > > libopenblas-dev 0.2.8 > > > > > amd libm 3.1 > > > > > latest theano version from the git repository > > > > > > > > > > and I got again a segmentation fault. The main difference between > > the > > > > > machines is the processor (The first a 6th gen i7, the second a > > Xeon, > > > > the > > > > > third a 5th gen i7). > > > > > Any other ideas of what might be causing the segmentation fault? > > > > > > > > > > -- Juan Camilo > > > > > > > > > > On Thursday, March 10, 2016 at 9:36:38 AM UTC-5, Juan Camilo Gamboa > > > > Higuera > > > > > wrote: > > > > > > > > > > > > Hi Pascal, > > > > > > > > > > > > Thanks for your time! > > > > > > > > > > > > I'm using the latest version of Theano from the git repository, > > and > > > > AMD > > > > > > libm version 3.1. The md5 sums are: > > > > > > > > > > > > fd9f86e040a4d5f26013fc5f60182680 libamdlibm.so > > > > > > 701474c84f1e3dff6bbb1a11d07eb215 libamdlibm.a > > > > > > > > > > > > I tried disabling blas (by setting > > > > > > *THEANO_FLAGS="lib.amdlibm=True,blas.ldflags=''"*) but I still get > > the > > > > > > segmentation fault. > > > > > > > > > > > > I've also tried it on another machine, and I get no segmentation > > > > fault. > > > > > > > > > > > > The machine where the segfault happens: > > > > > > Ubuntu 15.10, 4.2 kernel > > > > > > gcc 4.9.3 > > > > > > libopenblas-dev 0.2.14 > > > > > > > > > > > > The machine where it works > > > > > > Ubuntu 14.04.4, 4.2 kernel > > > > > > gcc 4.8.4 > > > > > > libopenblas-dev 0.2.8 > > > > > > > > > > > > I will try installing the older versions of openblas and gcc on my > > > > machine > > > > > > and see if it makes a difference. > > > > > > > > > > > > Do you have any other suggestions? > > > > > > > > > > > > -- Juan Camilo > > > > > > > > > > > > On Wednesday, March 9, 2016 at 7:52:39 PM UTC-5, Pascal Lamblin > > wrote: > > > > > >> > > > > > >> So I just tried your case, on Linux, with a version of amdlibm > > that > > > > is > > > > > >> possibly quite old but that I'm unable to identify (md5sum > > below). > > > > > >> > > > > > >> I was unfortunately unable to reproduce the segfault. > > > > > >> > > > > > >> It may be possible that for some reason, the output of the dot > > > > product > > > > > >> is not aligned, and that amdlibm does not support that. In that > > case, > > > > > >> maybe disabling BLAS could help. > > > > > >> > > > > > >> > > > > > >> 9211766f5cef4ce1a35dc44701dcac6a libamdlibm.a > > > > > >> 3ce0e1c4c7afbfe514639fd8482ed220 libamdlibm.so > > > > > >> > > > > > >> > > > > > >> On Wed, Mar 09, 2016, Juan Camilo Gamboa Higuera wrote: > > > > > >> > Another development! > > > > > >> > > > > > > >> > If i disable the inplace optimization ( > > > > optimizer_excluding=inplace_opt > > > > > >> ) > > > > > >> > the segmentation fault disappears. Which narrows down the > > problem > > > > to > > > > > >> > situations where > > > > > >> > > > > > > >> > * you take the sine or cosine of a dot product between two > > matrices > > > > and > > > > > >> > inplace optimization for elementwise ops is enabled.*Cheers! > > > > > >> > > > > > > >> > -- Juan Camilo > > > > > >> > > > > > > >> > On Wednesday, March 9, 2016 at 10:23:22 AM UTC-5, Juan Camilo > > > > Gamboa > > > > > >> > Higuera wrote: > > > > > >> > > > > > > > >> > > Hi all, > > > > > >> > > > > > > > >> > > *Here is a simpler version of the code that causes the > > > > segmentation > > > > > >> fault > > > > > >> > > when using amdlibm. It looks like it happens when you take > > the > > > > sine > > > > > >> or > > > > > >> > > cosine of a dot product:* > > > > > >> > > > > > > > >> > > import theano > > > > > >> > > import theano.tensor as T > > > > > >> > > import numpy as np > > > > > >> > > > > > > > >> > > np.set_printoptions(linewidth=200) > > > > > >> > > n_samples = 500 > > > > > >> > > n_basis = 100 > > > > > >> > > idims = 4 > > > > > >> > > > > > > > >> > > # generate some test data > > > > > >> > > X_ = > > > > np.random.randn(n_samples,idims).astype(theano.config.floatX) > > > > > >> > > w_ = > > np.random.randn(n_basis,idims).astype(theano.config.floatX) > > > > > >> > > > > > > > >> > > X = T.matrix('X') > > > > > >> > > w = T.matrix('w') > > > > > >> > > > > > > > >> > > f1 = theano.function( [X,w], T.sin(w.dot(X.T)) ) > > > > > >> > > print f1(X_,w_) > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> > > *This is the output from gdb:* > > > > > >> > > Program received signal SIGSEGV, Segmentation fault. > > > > > >> > > (anonymous > > > > > >> > > > > > > > >> > > > > namespace)::__struct_compiled_op_d97cada2f52aa7922278b46580b84db2::run > > > > > >> > > (this=0x2308200) at > > > > > >> > > > > > > > >> > > > > > > /home/juancamilog/.theano/compiledir_Linux-4.2--generic-x86_64-with-Ubuntu-15.10-wily-x86_64-2.7.10-64/tmpZ_b9g2/mod.cpp:356 > > > > > > > > > > > > > >> > > > > > >> > > 356 PyObject* old = PyList_GET_ITEM(storage_V1, 0); > > > > > >> > > (gdb) bt > > > > > >> > > #0 (anonymous > > > > > >> > > > > > > > >> > > > > namespace)::__struct_compiled_op_d97cada2f52aa7922278b46580b84db2::run > > > > > >> > > (this=0x2308200) at > > > > > >> > > > > > > > >> > > > > > > /home/juancamilog/.theano/compiledir_Linux-4.2--generic-x86_64-with-Ubuntu-15.10-wily-x86_64-2.7.10-64/tmpZ_b9g2/mod.cpp:356 > > > > > > > > > > > > > >> > > > > > >> > > #1 > > > > __struct_compiled_op_d97cada2f52aa7922278b46580b84db2_executor > > > > > >> > > (self=0x2308200) at > > > > > >> > > > > > > > >> > > > > > > /home/juancamilog/.theano/compiledir_Linux-4.2--generic-x86_64-with-Ubuntu-15.10-wily-x86_64-2.7.10-64/tmpZ_b9g2/mod.cpp:402 > > > > > > > > > > > > > >> > > > > > >> > > #2 0x00007ffff30f875e in c_call (self=0x7fffd3900b48, > > > > node_idx=2, > > > > > >> > > verbose=0) at > > > > > >> > > > > > > > >> > > > > > > /home/juancamilog/.theano/compiledir_Linux-4.2--generic-x86_64-with-Ubuntu-15.10-wily-x86_64-2.7.10-64/lazylinker_ext/mod.cpp:533 > > > > > > > > > > > > > >> > > > > > >> > > #3 0x00007ffff30f919b in lazy_rec_eval (self=0x7fffd3900b48, > > > > > >> var_idx=4, > > > > > >> > > one=0x981158, zero=0x981170) at > > > > > >> > > > > > > > >> > > > > > > /home/juancamilog/.theano/compiledir_Linux-4.2--generic-x86_64-with-Ubuntu-15.10-wily-x86_64-2.7.10-64/lazylinker_ext/mod.cpp:699 > > > > > > > > > > > > > >> > > > > > >> > > #4 0x00007ffff30f9842 in CLazyLinker_call > > (_self=0x7fffd3900b48, > > > > > >> > > args=0x7ffff7f7f050, kwds=0x0) at > > > > > >> > > > > > > > >> > > > > > > /home/juancamilog/.theano/compiledir_Linux-4.2--generic-x86_64-with-Ubuntu-15.10-wily-x86_64-2.7.10-64/lazylinker_ext/mod.cpp:838 > > > > > > > > > > > > > >> > > > > > >> > > #5 0x00000000004bf3cc in PyEval_EvalFrameEx () > > > > > >> > > #6 0x00000000004b7986 in PyEval_EvalCodeEx () > > > > > >> > > #7 0x00000000004d3bb9 in ?? () > > > > > >> > > #8 0x00000000004ece7e in ?? () > > > > > >> > > #9 0x00000000004a45de in PyObject_Call () > > > > > >> > > #10 0x0000000000546a93 in ?? () > > > > > >> > > #11 0x00000000004bf3cc in PyEval_EvalFrameEx () > > > > > >> > > #12 0x00000000004b7986 in PyEval_EvalCodeEx () > > > > > >> > > #13 0x00000000004e8f3f in ?? () > > > > > >> > > #14 0x00000000004e3b02 in PyRun_FileExFlags () > > > > > >> > > #15 0x00000000004e22e6 in PyRun_SimpleFileExFlags () > > > > > >> > > #16 0x0000000000490fe1 in Py_Main () > > > > > >> > > #17 0x00007ffff7811a40 in __libc_start_main (main=0x4909f0 > > > > <main>, > > > > > >> argc=2, > > > > > >> > > argv=0x7fffffffd608, init=<optimized out>, fini=<optimized > > out>, > > > > > >> > > rtld_fini=<optimized out>, stack_end=0x7fffffffd5f8) at > > > > > >> libc-start.c:289 > > > > > >> > > #18 0x0000000000490919 in _start () > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> > > *I've attached the generated mod.cpp file to this messageIt > > looks > > > > > >> like a > > > > > >> > > bug in theano. Can anyone confirm this?* > > > > > >> > > > > > > > >> > > Thanks! > > > > > >> > > > > > > > >> > > -- Juan Camilo > > > > > >> > > > > > > > >> > > > > > > > >> > > On Monday, March 7, 2016 at 3:57:53 PM UTC-5, Juan Camilo > > Gamboa > > > > > >> Higuera > > > > > >> > > wrote: > > > > > >> > >> > > > > > >> > >> Hi, > > > > > >> > >> > > > > > >> > >> I have a piece of code that produces a segmentation fault > > when > > > > > >> running > > > > > >> > >> with the lib.amdlibm flag to True. > > > > > >> > >> > > > > > >> > >> The code is available here: > > > > > >> > >> > > > > > >> > >> https://gist.github.com/juancamilog/eb1443f0f2fa45ffa6f3 > > > > > >> > >> > > > > > >> > >> After some experimentation, I've been able to determine that > > the > > > > > >> > >> segmentation fault appears when I'm using trigonometric > > > > operations > > > > > >> in my > > > > > >> > >> function, i.e. I replace the sin and cos operations by > > something > > > > > >> else, and > > > > > >> > >> it works. > > > > > >> > >> > > > > > >> > >> This is the backtrace when debugging the code with gdb: > > > > > >> > >> > > > > > >> > >> (anonymous > > > > > >> > >> > > > > > >> > > > > namespace)::__struct_compiled_op_bdfa1d4ff1cc646e65d68ff3a2c807db::run > > > > > >> > >> (this=0x27d3830) at > > > > > >> > >> > > > > > >> > > > > > > /home/juancamilog/.theano/compiledir_Linux-4.2--generic-x86_64-with-Ubuntu-15.10-wily-x86_64-2.7.10-64/tmpomVkWr/mod.cpp:517 > > > > > > > > > > > > > >> > > > > > >> > >> 517 PyObject* old = PyList_GET_ITEM(storage_V1, 0); > > > > > >> > >> (gdb) > > > > > >> > >> (gdb) > > > > > >> > >> (gdb) bt > > > > > >> > >> #0 (anonymous > > > > > >> > >> > > > > > >> > > > > namespace)::__struct_compiled_op_bdfa1d4ff1cc646e65d68ff3a2c807db::run > > > > > >> > >> (this=0x27d3830) at > > > > > >> > >> > > > > > >> > > > > > > /home/juancamilog/.theano/compiledir_Linux-4.2--generic-x86_64-with-Ubuntu-15.10-wily-x86_64-2.7.10-64/tmpomVkWr/mod.cpp:517 > > > > > > > > > > > > > >> > > > > > >> > >> #1 > > > > __struct_compiled_op_bdfa1d4ff1cc646e65d68ff3a2c807db_executor > > > > > >> > >> (self=0x27d3830) at > > > > > >> > >> > > > > > >> > > > > > > /home/juancamilog/.theano/compiledir_Linux-4.2--generic-x86_64-with-Ubuntu-15.10-wily-x86_64-2.7.10-64/tmpomVkWr/mod.cpp:563 > > > > > > > > > > > > > >> > > > > > >> > >> #2 0x00007ffff10ea75e in c_call (self=0x7fffd6dff7a0, > > > > node_idx=26, > > > > > >> > >> verbose=0) at > > > > > >> > >> > > > > > >> > > > > > > /home/juancamilog/.theano/compiledir_Linux-4.2--generic-x86_64-with-Ubuntu-15.10-wily-x86_64-2.7.10-64/lazylinker_ext/mod.cpp:533 > > > > > > > > > > > > > >> > > > > > >> > >> #3 0x00007ffff10eb19b in lazy_rec_eval > > (self=0x7fffd6dff7a0, > > > > > >> var_idx=39, > > > > > >> > >> one=0x981158, zero=0x981170) at > > > > > >> > >> > > > > > >> > > > > > > /home/juancamilog/.theano/compiledir_Linux-4.2--generic-x86_64-with-Ubuntu-15.10-wily-x86_64-2.7.10-64/lazylinker_ext/mod.cpp:699 > > > > > > > > > > > > > >> > > > > > >> > >> #4 0x00007ffff10eb0d7 in lazy_rec_eval > > (self=0x7fffd6dff7a0, > > > > > >> var_idx=43, > > > > > >> > >> one=0x981158, zero=0x981170) at > > > > > >> > >> > > > > > >> > > > > > > /home/juancamilog/.theano/compiledir_Linux-4.2--generic-x86_64-with-Ubuntu-15.10-wily-x86_64-2.7.10-64/lazylinker_ext/mod.cpp:690 > > > > > > > > > > > > > >> > > > > > >> > >> #5 0x00007ffff10eb0d7 in lazy_rec_eval > > (self=0x7fffd6dff7a0, > > > > > >> var_idx=49, > > > > > >> > >> one=0x981158, zero=0x981170) at > > > > > >> > >> > > > > > >> > > > > > > /home/juancamilog/.theano/compiledir_Linux-4.2--generic-x86_64-with-Ubuntu-15.10-wily-x86_64-2.7.10-64/lazylinker_ext/mod.cpp:690 > > > > > > > > > > > > > >> > > > > > >> > >> #6 0x00007ffff10eb0d7 in lazy_rec_eval > > (self=0x7fffd6dff7a0, > > > > > >> var_idx=66, > > > > > >> > >> one=0x981158, zero=0x981170) at > > > > > >> > >> > > > > > >> > > > > > > /home/juancamilog/.theano/compiledir_Linux-4.2--generic-x86_64-with-Ubuntu-15.10-wily-x86_64-2.7.10-64/lazylinker_ext/mod.cpp:690 > > > > > > > > > > > > > >> > > > > > >> > >> #7 0x00007ffff10eb0d7 in lazy_rec_eval > > (self=0x7fffd6dff7a0, > > > > > >> var_idx=67, > > > > > >> > >> one=0x981158, zero=0x981170) at > > > > > >> > >> > > > > > >> > > > > > > /home/juancamilog/.theano/compiledir_Linux-4.2--generic-x86_64-with-Ubuntu-15.10-wily-x86_64-2.7.10-64/lazylinker_ext/mod.cpp:690 > > > > > > > > > > > > > >> > > > > > >> > >> #8 0x00007ffff10eb0d7 in lazy_rec_eval > > (self=0x7fffd6dff7a0, > > > > > >> var_idx=69, > > > > > >> > >> one=0x981158, zero=0x981170) at > > > > > >> > >> > > > > > >> > > > > > > /home/juancamilog/.theano/compiledir_Linux-4.2--generic-x86_64-with-Ubuntu-15.10-wily-x86_64-2.7.10-64/lazylinker_ext/mod.cpp:690 > > > > > > > > > > > > > >> > > > > > >> > >> #9 0x00007ffff10eb0d7 in lazy_rec_eval > > (self=0x7fffd6dff7a0, > > > > > >> var_idx=70, > > > > > >> > >> one=0x981158, zero=0x981170) at > > > > > >> > >> > > > > > >> > > > > > > /home/juancamilog/.theano/compiledir_Linux-4.2--generic-x86_64-with-Ubuntu-15.10-wily-x86_64-2.7.10-64/lazylinker_ext/mod.cpp:690 > > > > > > > > > > > > > >> > > > > > >> > >> #10 0x00007ffff10eb0d7 in lazy_rec_eval > > (self=0x7fffd6dff7a0, > > > > > >> var_idx=71, > > > > > >> > >> one=0x981158, zero=0x981170) at > > > > > >> > >> > > > > > >> > > > > > > /home/juancamilog/.theano/compiledir_Linux-4.2--generic-x86_64-with-Ubuntu-15.10-wily-x86_64-2.7.10-64/lazylinker_ext/mod.cpp:690 > > > > > > > > > > > > > >> > > > > > >> > >> #11 0x00007ffff10eb0d7 in lazy_rec_eval > > (self=0x7fffd6dff7a0, > > > > > >> var_idx=72, > > > > > >> > >> one=0x981158, zero=0x981170) at > > > > > >> > >> > > > > > >> > > > > > > /home/juancamilog/.theano/compiledir_Linux-4.2--generic-x86_64-with-Ubuntu-15.10-wily-x86_64-2.7.10-64/lazylinker_ext/mod.cpp:690 > > > > > > > > > > > > > >> > > > > > >> > >> #12 0x00007ffff10eb0d7 in lazy_rec_eval > > (self=0x7fffd6dff7a0, > > > > > >> var_idx=73, > > > > > >> > >> one=0x981158, zero=0x981170) at > > > > > >> > >> > > > > > >> > > > > > > /home/juancamilog/.theano/compiledir_Linux-4.2--generic-x86_64-with-Ubuntu-15.10-wily-x86_64-2.7.10-64/lazylinker_ext/mod.cpp:690 > > > > > > > > > > > > > >> > > > > > >> > >> #13 0x00007ffff10eb842 in CLazyLinker_call > > > > (_self=0x7fffd6dff7a0, > > > > > >> > >> args=0x7ffff7f7f050, kwds=0x0) at > > > > > >> > >> > > > > > >> > > > > > > /home/juancamilog/.theano/compiledir_Linux-4.2--generic-x86_64-with-Ubuntu-15.10-wily-x86_64-2.7.10-64/lazylinker_ext/mod.cpp:838 > > > > > > > > > > > > > >> > > > > > >> > >> #14 0x00000000004bf3cc in PyEval_EvalFrameEx () > > > > > >> > >> #15 0x00000000004b7986 in PyEval_EvalCodeEx () > > > > > >> > >> #16 0x00000000004d3bb9 in ?? () > > > > > >> > >> #17 0x00000000004ece7e in ?? () > > > > > >> > >> #18 0x00000000004a45de in PyObject_Call () > > > > > >> > >> #19 0x0000000000546a93 in ?? () > > > > > >> > >> #20 0x00000000004bf3cc in PyEval_EvalFrameEx () > > > > > >> > >> #21 0x00000000004b7986 in PyEval_EvalCodeEx () > > > > > >> > >> #22 0x00000000004e8f3f in ?? () > > > > > >> > >> #23 0x00000000004e3b02 in PyRun_FileExFlags () > > > > > >> > >> #24 0x00000000004e22e6 in PyRun_SimpleFileExFlags () > > > > > >> > >> #25 0x0000000000490fe1 in Py_Main () > > > > > >> > >> #26 0x00007ffff7811a40 in __libc_start_main (main=0x4909f0 > > > > <main>, > > > > > >> > >> argc=2, argv=0x7fffffffd5d8, init=<optimized out>, > > > > fini=<optimized > > > > > >> out>, > > > > > >> > >> rtld_fini=<optimized out>, stack_end=0x7fffffffd5c8) at > > > > > >> libc-start.c:289 > > > > > >> > >> #27 0x0000000000490919 in _start () > > > > > >> > >> > > > > > >> > >> I've attached the mod.cpp file generated by theano. > > > > > >> > >> > > > > > >> > >> First question, any idea what is going on here? Second, how > > does > > > > one > > > > > >> > >> debug this? > > > > > >> > >> > > > > > >> > >> Thanks! > > > > > >> > >> > > > > > >> > >> -- Juan Camilo > > > > > >> > >> > > > > > >> > >> > > > > > >> > > > > > > >> > -- > > > > > >> > > > > > > >> > --- > > > > > >> > You received this message because you are subscribed to the > > Google > > > > > >> Groups "theano-users" group. > > > > > >> > To unsubscribe from this group and stop receiving emails from > > it, > > > > send > > > > > >> an email to [email protected]. > > > > > >> > For more options, visit https://groups.google.com/d/optout. > > > > > >> > > > > > >> > > > > > >> -- > > > > > >> Pascal > > > > > >> > > > > > > > > > > > > > > > > -- > > > > > > > > > > --- > > > > > You received this message because you are subscribed to the Google > > > > Groups "theano-users" group. > > > > > To unsubscribe from this group and stop receiving emails from it, > > send > > > > an email to [email protected] <javascript:>. > > > > > For more options, visit https://groups.google.com/d/optout. > > > > > > > > > > > > -- > > > > Pascal > > > > > > > > > > -- > > > > > > --- > > > You received this message because you are subscribed to the Google > > Groups "theano-users" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > an email to [email protected] <javascript:>. > > > For more options, visit https://groups.google.com/d/optout. > > > > > > -- > > Pascal > > > > -- > > --- > You received this message because you are subscribed to the Google Groups > "theano-users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/d/optout.
-- Pascal -- --- You received this message because you are subscribed to the Google Groups "theano-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
