Re: [Numpy-discussion] RAM problem during code execution - Numpya arrays
Hi José, The code is somewhat longish for a pure visual inspection, but my advice is that you install memory profiler ( https://pypi.python.org/pypi/memory_profiler). This will help you determine which line or lines are hugging the memory the most. Saludos, Francesc On Fri, Aug 23, 2013 at 3:58 PM, Josè Luis Mietta joseluismie...@yahoo.com.ar wrote: Hi ecperts. I need your help with a RAM porblem during execution of my script. I wrote the next code. I use SAGE. In 1-2 hours of execution time the RAM of my laptop (8gb) is filled and the sistem crash: from scipy.stats import uniformimport numpy as np cant_de_cadenas =[700,800,900] cantidad_de_cadenas=np.array([]) for k in cant_de_cadenas: cantidad_de_cadenas=np.append(cantidad_de_cadenas,k) cantidad_de_cadenas=np.transpose(cantidad_de_cadenas) b=10 h=bLongitud=1 numero_experimentos=150 densidad_de_cadenas =cantidad_de_cadenas/(b**2) prob_perc=np.array([]) tiempos=np.array([]) S_int=np.array([]) S_medio=np.array([]) desviacion_standard=np.array([]) desviacion_standard_nuevo=np.array([]) anisotropia_macroscopica_porcentual=np.array([]) componente_y=np.array([]) componente_x=np.array([]) import time for N in cant_de_cadenas: empieza=time.clock() PERCOLACION=np.array([]) size_medio_intuitivo = np.array([]) size_medio_nuevo = np.array([]) std_dev_size_medio_intuitivo = np.array([]) std_dev_size_medio_nuevo = np.array([]) comp_y = np.array([]) comp_x = np.array([]) for u in xrange(numero_experimentos): perco = False array_x1=uniform.rvs(loc=-b/2, scale=b, size=N) array_y1=uniform.rvs(loc=-h/2, scale=h, size=N) array_angle=uniform.rvs(loc=-0.5*(np.pi), scale=np.pi, size=N) array_pendiente_x=1./np.tan(array_angle) random=uniform.rvs(loc=-1, scale=2, size=N) lambda_sign=np.zeros([N]) for t in xrange(N): if random[t]0: lambda_sign[t]=-1 else: lambda_sign[t]=1 array_lambdas=(lambda_sign*Longitud)/np.sqrt(1+array_pendiente_x**2) array_x2= array_x1 + array_lambdas*array_pendiente_x array_y2= array_y1 + array_lambdas*1 array_x1 = np.append(array_x1, [-b/2, b/2, -b/2, -b/2]) array_y1 = np.append(array_y1, [-h/2, -h/2, -h/2, h/2]) array_x2 = np.append(array_x2, [-b/2, b/2, b/2, b/2]) array_y2 = np.append(array_y2, [h/2, h/2, -h/2, h/2]) M = np.zeros([N+4,N+4]) for j in xrange(N+4): if j0: x_A1B1 = array_x2[j]-array_x1[j] y_A1B1 = array_y2[j]-array_y1[j] x_A1A2 = array_x1[0:j]-array_x1[j] y_A1A2 = array_y1[0:j]-array_y1[j] x_A2A1 = -1*x_A1A2 y_A2A1 = -1*y_A1A2 x_A2B2 = array_x2[0:j]-array_x1[0:j] y_A2B2 = array_y2[0:j]-array_y1[0:j] x_A1B2 = array_x2[0:j]-array_x1[j] y_A1B2 = array_y2[0:j]-array_y1[j] x_A2B1 = array_x2[j]-array_x1[0:j] y_A2B1 = array_y2[j]-array_y1[0:j] p1 = x_A1B1*y_A1A2 - y_A1B1*x_A1A2 p2 = x_A1B1*y_A1B2 - y_A1B1*x_A1B2 p3 = x_A2B2*y_A2B1 - y_A2B2*x_A2B1 p4 = x_A2B2*y_A2A1 - y_A2B2*x_A2A1 condicion_1=p1*p2 condicion_2=p3*p4 for k in xrange (j): if condicion_1[k]=0 and condicion_2[k]=0: M[j,k]=1 del condicion_1 del condicion_2 if j+1N+4: x_A1B1 = array_x2[j]-array_x1[j] y_A1B1 = array_y2[j]-array_y1[j] x_A1A2 = array_x1[j+1:]-array_x1[j] y_A1A2 = array_y1[j+1:]-array_y1[j] x_A2A1 = -1*x_A1A2 y_A2A1 = -1*y_A1A2 x_A2B2 = array_x2[j+1:]-array_x1[j+1:] y_A2B2 = array_y2[j+1:]-array_y1[j+1:] x_A1B2 = array_x2[j+1:]-array_x1[j] y_A1B2 = array_y2[j+1:]-array_y1[j] x_A2B1 = array_x2[j]-array_x1[j+1:] y_A2B1 = array_y2[j]-array_y1[j+1:] p1 = x_A1B1*y_A1A2 - y_A1B1*x_A1A2 p2 = x_A1B1*y_A1B2 - y_A1B1*x_A1B2 p3 = x_A2B2*y_A2B1 - y_A2B2*x_A2B1 p4 = x_A2B2*y_A2A1 - y_A2B2*x_A2A1 condicion_1=p1*p2 condicion_2=p3*p4 for k in xrange ((N+4)-j-1): if condicion_1[k]=0 and condicion_2[k]=0: M[j,k+j+1]=1 del condicion_1 del condicion_2 M[N,N+2]=0 M[N,N+3]=0 M[N+1,N+2]=0 M[N+1,N+3]=0 M[N+2,N]=0 M[N+2,N+1]=0 M[N+3,N]=0 M[N+3,N+1]=0
Re: [Numpy-discussion] RAM problem during code execution - Numpya arrays
On Fri, Aug 23, 2013 at 10:34 AM, Francesc Alted franc...@continuum.iowrote: Hi José, The code is somewhat longish for a pure visual inspection, but my advice is that you install memory profiler ( https://pypi.python.org/pypi/memory_profiler). This will help you determine which line or lines are hugging the memory the most. Saludos, Francesc On Fri, Aug 23, 2013 at 3:58 PM, Josè Luis Mietta joseluismie...@yahoo.com.ar wrote: Hi ecperts. I need your help with a RAM porblem during execution of my script. I wrote the next code. I use SAGE. In 1-2 hours of execution time the RAM of my laptop (8gb) is filled and the sistem crash: from scipy.stats import uniformimport numpy as np cant_de_cadenas =[700,800,900] cantidad_de_cadenas=np.array([]) for k in cant_de_cadenas: cantidad_de_cadenas=np.append(cantidad_de_cadenas,k) cantidad_de_cadenas=np.transpose(cantidad_de_cadenas) b=10 h=bLongitud=1 numero_experimentos=150 densidad_de_cadenas =cantidad_de_cadenas/(b**2) prob_perc=np.array([]) tiempos=np.array([]) S_int=np.array([]) S_medio=np.array([]) desviacion_standard=np.array([]) desviacion_standard_nuevo=np.array([]) anisotropia_macroscopica_porcentual=np.array([]) componente_y=np.array([]) componente_x=np.array([]) import time for N in cant_de_cadenas: empieza=time.clock() PERCOLACION=np.array([]) size_medio_intuitivo = np.array([]) size_medio_nuevo = np.array([]) std_dev_size_medio_intuitivo = np.array([]) std_dev_size_medio_nuevo = np.array([]) comp_y = np.array([]) comp_x = np.array([]) for u in xrange(numero_experimentos): perco = False array_x1=uniform.rvs(loc=-b/2, scale=b, size=N) array_y1=uniform.rvs(loc=-h/2, scale=h, size=N) array_angle=uniform.rvs(loc=-0.5*(np.pi), scale=np.pi, size=N) array_pendiente_x=1./np.tan(array_angle) random=uniform.rvs(loc=-1, scale=2, size=N) lambda_sign=np.zeros([N]) for t in xrange(N): if random[t]0: lambda_sign[t]=-1 else: lambda_sign[t]=1 array_lambdas=(lambda_sign*Longitud)/np.sqrt(1+array_pendiente_x**2) array_x2= array_x1 + array_lambdas*array_pendiente_x array_y2= array_y1 + array_lambdas*1 array_x1 = np.append(array_x1, [-b/2, b/2, -b/2, -b/2]) array_y1 = np.append(array_y1, [-h/2, -h/2, -h/2, h/2]) array_x2 = np.append(array_x2, [-b/2, b/2, b/2, b/2]) array_y2 = np.append(array_y2, [h/2, h/2, -h/2, h/2]) M = np.zeros([N+4,N+4]) for j in xrange(N+4): if j0: x_A1B1 = array_x2[j]-array_x1[j] y_A1B1 = array_y2[j]-array_y1[j] x_A1A2 = array_x1[0:j]-array_x1[j] y_A1A2 = array_y1[0:j]-array_y1[j] x_A2A1 = -1*x_A1A2 y_A2A1 = -1*y_A1A2 x_A2B2 = array_x2[0:j]-array_x1[0:j] y_A2B2 = array_y2[0:j]-array_y1[0:j] x_A1B2 = array_x2[0:j]-array_x1[j] y_A1B2 = array_y2[0:j]-array_y1[j] x_A2B1 = array_x2[j]-array_x1[0:j] y_A2B1 = array_y2[j]-array_y1[0:j] p1 = x_A1B1*y_A1A2 - y_A1B1*x_A1A2 p2 = x_A1B1*y_A1B2 - y_A1B1*x_A1B2 p3 = x_A2B2*y_A2B1 - y_A2B2*x_A2B1 p4 = x_A2B2*y_A2A1 - y_A2B2*x_A2A1 condicion_1=p1*p2 condicion_2=p3*p4 for k in xrange (j): if condicion_1[k]=0 and condicion_2[k]=0: M[j,k]=1 del condicion_1 del condicion_2 if j+1N+4: x_A1B1 = array_x2[j]-array_x1[j] y_A1B1 = array_y2[j]-array_y1[j] x_A1A2 = array_x1[j+1:]-array_x1[j] y_A1A2 = array_y1[j+1:]-array_y1[j] x_A2A1 = -1*x_A1A2 y_A2A1 = -1*y_A1A2 x_A2B2 = array_x2[j+1:]-array_x1[j+1:] y_A2B2 = array_y2[j+1:]-array_y1[j+1:] x_A1B2 = array_x2[j+1:]-array_x1[j] y_A1B2 = array_y2[j+1:]-array_y1[j] x_A2B1 = array_x2[j]-array_x1[j+1:] y_A2B1 = array_y2[j]-array_y1[j+1:] p1 = x_A1B1*y_A1A2 - y_A1B1*x_A1A2 p2 = x_A1B1*y_A1B2 - y_A1B1*x_A1B2 p3 = x_A2B2*y_A2B1 - y_A2B2*x_A2B1 p4 = x_A2B2*y_A2A1 - y_A2B2*x_A2A1 condicion_1=p1*p2 condicion_2=p3*p4 for k in xrange ((N+4)-j-1): if condicion_1[k]=0 and condicion_2[k]=0: M[j,k+j+1]=1 del condicion_1 del condicion_2 M[N,N+2]=0 M[N,N+3]=0 M[N+1,N+2]=0 M[N+1,N+3]=0 M[N+2,N]=0
Re: [Numpy-discussion] RAM problem during code execution - Numpya arrays
On 23 August 2013 16:59, Benjamin Root ben.r...@ou.edu wrote: A lot of the code you have here can be greatly simplified. I would start with just trying to get rid of appends as much as possible and use preallocated arrays with np.empty() or np.ones() or the likes. Also, if you don't know beforehand the final size of the array (I find difficult to follow the flow of the program, it is quite lengthy), you can use lists as a temporary thing: results = [] while SomeCondition: do_something() results.append(res) results = np.array(res) Also, it may also help to trace things down encapsulate pieces of code inside functions. There are two reasons for this: it will make the code more readable, easier to test, and you could run independently pieces of code to find where is your memory growing. I hope it is of any help. David. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
