Good afternoon, I am new to web and python development, but I have managed until now to get pretty far along into a statistical analysis application without any real problems. Web2py is great. The simplified concept of my app is that Users connect to the local database and pull in data, the app stores the data as a numpy array and allows the User to do analysis, then the app writes the results of the analysis back to a local database table. I am running web2py 2.4.7 (I know, I should upgrade soon) and python 2.7.4.
I need the numpy array, which the User has loaded, to be available throughout the User's time in the app. So, I thought storing it in session was the obvious place. When the User loads in an array around 1 MB and it is stored in session, there are no problems. The app runs well. When the User loads arrays that are much larger, around 20 MB for example, however, the application slows down considerably and basically stops loading pages. In this case, the session file is around 65 MB. The average amount of system memory used is about 100 MB with over 1 GB on standby and free memory. I had a different problem previously because I think my array was too large. I decreased the amount of memory used considerably by cutting down the size of the numpy array. I was able to decrease the size from 190 MB to the 20 MB it is now by cutting out some long text fields that were not necessary for the analysis. Before that, I was getting python memory errors even though I had about 1 GB of standby and free memory available in Windows 7 and the application was only using about 600 MB of memory on average. After I decreased the size of the array to 20 MB, the python memory errors have gone away, but I still have the massive slowdowns that I detailed in the previous paragraph. My current guess is that these 20 MB numpy arrays are too big to be stored in session and that my bottleneck is accessing the large session file (in this case, around 65 MB). I am new, so I have no idea whether I am on the right track here. Does anyone have any general advice for me? What do you all do with large numpy arrays that need to be accessed across the app in different functions by just the current User? How big of a session file is too big? Josh -- Resources: - http://web2py.com - http://web2py.com/book (Documentation) - http://github.com/web2py/web2py (Source code) - https://code.google.com/p/web2py/issues/list (Report Issues) --- You received this message because you are subscribed to the Google Groups "web2py-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/groups/opt_out.
