Re: segmentation fault in scipy?
If I run it from the shell (unix) I get: Segmentation fault and see a core dump in my processes. If I run it in the python shell I get as above: File D:\Python24\Lib\site-packages\numpy\core\defmatrix.py, line 149, in That's a Window's path... Does Windows even make full use of 4GB? Im developing on a unix machine. However, you do have a sharp eye... So you win the blue ribbon and a free roundtable pizza! I'm actually running experiments on an HP workstation while I work (emailed myself the error), that's one of the beauties of python. Yes, Dennis there is a max of 4gb of addressable mem on that machine and 2 cpus. This is what a unix path looks like. Can you tell me what OS and shell Im running (both are obvious from the prompt) for the bonus prize? [gremlins:~/desktop/CR_2] conorrob% python run.py AL fold: 0 Done Loading Input Ready Network Training Initial SSE: Segmentation fault [gremlins:~/desktop/CR_2] conorrob% -- http://mail.python.org/mailman/listinfo/python-list
Re: segmentation fault in scipy?
I'm afraid you're using terminology (and abbreviations!) that I can't follow. Let me try to restate what's going on and you can correct me as I screw up. You have a neural net that has 80 output units. You have 25000 observations that you are using to train the neural net. Each observation vector (and consequently, each error vector) has 80 elements. First, sorry for not quoting and using abbreviations. Second, your obsevation is correct, except that I have *one* output unit, not 80. I have ~80 input units + bias (for each of the 25000 observations), x + bias number of hidden units and one output unit leaving me with an output array/matrix of shape =(25000,), as well as my desired output having the same shape. RPROP = Resilliant BackPropogation, uses chages in the error gradiant ignores the magnitude of the gradiant, which can be harmful. See A Direct Adaptive Method for Faster Backpropogation Learning: The RPROP Algorithm -Martin Riedmiller and Heinrich Braun, just google it and read Section D, it's well written. tanh = hyperbolic tangent function [-1, 1] y vals, often times better for its steeper derivative and wider range. sigmoid function = popular for its [0,1] range and can calculate posterior probabilities when using a the cross entropy error function which I have commented out since it takes more time to process and I'm not using my error function in this specific case at this point in time, thus SSE is not really needed, however I'd like use it as a check. Also, popular for its easily calculatable derivative. The answer I gave also used the wrong name for the result. It seems that you want the sum of the squared errors across all of the observations. In this case, you can use axis=None to specify that every element should be summed: SSE = sum(multiply(error, error), axis=None) trace(dot(error, transpose(error))) wastes a *huge* amount of time and memory since you are calculating (if your machine was capable of it) a gigantic matrix, then throwing away all of the off-diagonal elements. The method I give above wastes a little memory; there is one temporary matrix the size of error. This is great advise and much appreciated. It was the answer to my problem, thank you. However, isn't this faster... scipy.sum(scipy.array(scipy.matrix(error)*scipy.matrix(error)), axis = None) as we you explained in my other posting? Thanks again. -- http://mail.python.org/mailman/listinfo/python-list
Re: Nested loops confusion
I'm still not sure what was stopping the inner loop from working earlier - but removing the redundancy in j=0 and so on seems to have solved it. Call me crazy, but be careful when programming python in different text editors and in general, ie cutting and pasting, tabing and spacing. Loops can look fine and not work (try moving around test print statements for iterators), in this case try re-tabing your indents so that everything is uniform in the editor you are using. -- http://mail.python.org/mailman/listinfo/python-list
segmentation fault in scipy?
I'm running operations large arrays of floats, approx 25,000 x 80. Python (scipy) does not seem to come close to using 4GB of wired mem, but segments at around a gig. Everything works fine on smaller batches of data around 10,000 x 80 and uses a max of ~600mb of mem. Any Ideas? Is this just too much data for scipy? Thanks Conor Traceback (most recent call last): File C:\Temp\CR_2\run.py, line 68, in ? net.rProp(1.2, .5, .01, 50.0, input, output, 1) File /Users/conorrob/Desktop/CR_2/Network.py, line 230, in rProp print scipy.trace(error*scipy.transpose(error)) File D:\Python24\Lib\site-packages\numpy\core\defmatrix.py, line 149, in __mul__ return N.dot(self, other) MemoryError -- http://mail.python.org/mailman/listinfo/python-list
Re: segmentation fault in scipy?
If I run it from the shell (unix) I get: Segmentation fault and see a core dump in my processes. If I run it in the python shell I get as above: File D:\Python24\Lib\site-packages\numpy\core\defmatrix.py, line 149, in __mul__ return N.dot(self, other) MemoryError I your experience as one of the dev of scipy, is this too much data? thank you -- http://mail.python.org/mailman/listinfo/python-list
Re: segmentation fault in scipy?
Good point. Finding the SSE using an absolute error matrix of (25000 x 1) is insane. I pulled out the error function (for now) and I'm back in business. Thanks for all the great advise. -- http://mail.python.org/mailman/listinfo/python-list
Best form when using matrices and arrays in scipy...
Using large arrays of data I found it is MUCH faster to cast arrays to matricies and then multiply the two matricies togther (scipy.matrix(ARRAY1)*scipy.matrix(ARRAY2)) in order to do a matrix multipy of two arrays vs. scipy.matrixmultipy(ARRAY1, ARRAY2). Are there any logical/efficiency errors with this train of thought and is there a valid reason for the speed increase? Thanks, Conor -- http://mail.python.org/mailman/listinfo/python-list
Re: segmentation fault in scipy?
Im using rprop (not dependent on error function in this case ie. standard rprop vs. irprop or arprop) for an MLP tanh, sigmod nnet as part of a hybrid model. I guess I was using a little Matlab thought when I wrote the SSE funtion. My batches are about 25,000 x 80 so my absolute error (diff between net outputs and desired outputs) when using *one* output unit is shape(~25000,), am I wrong to assume trace(error*transpose(error)) is the sum of the squared errors which should be an shape(1,)? I'm just now starting to dig a little deeper into scipy, and I need to get the full doc. Thanks for all your input. -- http://mail.python.org/mailman/listinfo/python-list
Re: very strange problem in 2.4
Ok, so I found out that even though mylist[] and all objects in it were fine ie id(mylist[i]) != id(mylist[all others]) what was happening is that during a reproduction function a shallow copies were being made making all offspring (genetic algorithm) have different id(mylist[0..n]), however the actual attributes were referenced for offspring on the same parent(s). Fixed it with a deepcopy and used properties insead of java like get sets. thanks for all the help, Conor -- http://mail.python.org/mailman/listinfo/python-list
Re: very strange problem in 2.4
Ok, so I found out that even though mylist[] and all objects in it were fine ie id(mylist[i]) != id(mylist[all others]) what was happening is that during a reproduction function a shallow copies were being made making all offspring (genetic algorithm) have different id(mylist[0..n]), however the actual attributes were referenced for offspring on the same parent(s). Fixed it with a deepcopy and used properties insead of java like get sets. thanks for all the help, Conor -- http://mail.python.org/mailman/listinfo/python-list
Re: very strange problem in 2.4
Ok, so I found out that even though mylist[] and all objects in it were fine ie id(mylist[i]) != id(mylist[all others]) what was happening is that during a reproduction function a shallow copies were being made making all offspring (genetic algorithm) have different id(mylist[0..n]), however the actual attributes were referenced for offspring on the same parent(s). Fixed it with a deepcopy and used properties insead of java like get sets. thanks for all the help, Conor -- http://mail.python.org/mailman/listinfo/python-list
Re: very strange problem in 2.4
Ok, so I found out that even though mylist[] and all objects in it were fine ie id(mylist[i]) != id(mylist[all others]) what was happening is that during a reproduction function a shallow copies were being made making all offspring (genetic algorithm) have different id(mylist[0..n]), however the actual attributes were referenced for offspring on the same parent(s). Fixed it with a deepcopy and used properties insead of java like get sets. thanks for all the help, Conor -- http://mail.python.org/mailman/listinfo/python-list
Re: very strange problem in 2.4
Ok, so I found out that even though mylist[] and all objects in it were fine ie id(mylist[i]) != id(mylist[all others]) what was happening is that during a reproduction function a shallow copies were being made making all offspring (genetic algorithm) have different id(mylist[0..n]), however the actual attributes were referenced for offspring on the same parent(s). Fixed it with a deepcopy and used properties insead of java like get sets. thanks for all the help, Conor -- http://mail.python.org/mailman/listinfo/python-list
very strange problem in 2.4
The Problem (very basic, but strange): I have a list holding a population of objects, each object has 5 vars and appropriate funtions to get or modify the vars. When objects in the list have identical vars (like all = 5 for var a and all = 10 for var b across all vars and objects) and i change self.mylist[i].change_var_a(5) to a new value, in this case var a in object i to 5, now all vars of type a in all objects in my list are changed to 5 instead of just var a in object mylist[i], which is my goal. if i print self.mylist[i].return_var_a() right after I change var a in object i, I get the correct change, however when i print out the whole list, the last change to var a in the last object modified takes over for all objects in the list. note: all the vars not being modified must be the same across all objects in the list, the var being modified need not be the same as the one before it in the list (but will be once just one of the identical object are changed). The value changed in the last object var modified takes over for all object vars making them exactly identical. If, for example, half the list has objects with random vars init. and the other half is identical, as above, and I perform the same operation, as above, to one of the identical var objects self.mylist[i].change_var_a(5) (to an object that has identicals in the list) all the identicals are changed in the same way as above, however the objects that have different var values are unchanged. What is python doing? Am I missing something? Any ideas at all would be wonderful? -- http://mail.python.org/mailman/listinfo/python-list
