Hi
I have loaded around 2200 table total of 1.1GB (I have a lot more to
load - csv size is 12GB). The load crash with a memory error:
Script run-time error
---------------------
Traceback (most recent call last):
File "c:\QuantView\App\a2\ubs-packages\qa\scripting.py", line 48, in
executeScript
result = eval(bytecodes, globalNamespace, localNamespace)
# allow eval errors to be propagated up the stack
File "<string>", line 1, in <module>
File "c:\QuantView\App\a2\ubs-packages\qa\timeseries_cache.py", line
412, in openDB
ts._firstPersistentTS = table.read(start=0, field="Date")[0]
File "c:\python26qv\lib\site-packages\tables\table.py", line 1565, in
read
arr = self._read(start, stop, step, field)
File "c:\python26qv\lib\site-packages\tables\table.py", line 1527, in
_read
self.row._fillCol(result, start, stop, step, field)
File "c:\python26qv\lib\site-packages\tables\utils.py", line 228, in
newfget
mydict[name] = value = fget(self)
File "c:\python26qv\lib\site-packages\tables\table.py", line 299, in
row
return tableExtension.Row(self)
File "tableExtension.pyx", line 751, in
tables.tableExtension.Row.__cinit__ (tables\tableExtension.c:6736)
File "tableExtension.pyx", line 786, in
tables.tableExtension.Row._newBuffer (tables\tableExtension.c:7192)
File "c:\python26qv\lib\site-packages\tables\table.py", line 687, in
_get_container
return numpy.empty(shape=shape, dtype=self._v_dtype)
MemoryError
When I open the db I wish to cycle through each table and store the
attributes in memory, the first row of first column, the last row of the
last column but no other table data.
If I open the db with the NODE_CACHE_SLOTS=-10000 python uses about 1GB
of ram then stops with a memory error, same for NODE_CACHE_SLOTS=-2000.
If I run it with NODE_CACHE_SLOTS=2000 then I still get a memory error
followed by a performance warning thus:
c:\python26qv\lib\site-packages\tables\table.py:2518:
PerformanceWarning: table ``/data/_2279`` is being preempted from alive
nodes without
its buffers being flushed or with some index being dirty. This may lead
to very ineficient use of resources and even to fatal errors in cer
tain situations. Please do a call to the .flush() or .reIndexDirty()
methods on this table before start using other nodes.
PerformanceWarning)
Here's the code that causes the memory error:
def openDB(self, DBLocator, DBType):
# basic checks
if self._DBByDBLocator.has_key(DBLocator): raise TSCError("DB
already open")
if DBType.upper() <> "PYTABLESPRO": raise TSCError("DBType can
only be PyTablesPro at the moment")
if not os.path.exists(DBLocator): raise TSCError("DB doesn't
exist")
# open it and add it to _DBTypeByDBLocator
ptFile = pytables.openFile(DBLocator, mode="a",
NODE_CACHE_SLOTS=2000)
self._DBByDBLocator[DBLocator] = ptFile
# for each table in the DB create a Timeseries keyed by the
table's attributes
tables = pt_utils.getTables(ptFile, where="/data")
newTSByTSID = {}
for table in tables:
attributes = {}
for name in table._v_attrs._f_list():
attributes[name] = table._v_attrs[name] # I'm not sure
if the _v_attrs can be added in one go
if self.hasTS(attributes):
# if any tables are already open then fail the whole
lot!!
ptFile.close()
raise TSCError("Error in openDB - TS with attributes
(%s) already exists" % attributesAsString(attributes))
self._TSIDSeed += 1
ts = Timeseries(attributes, table.dtype, self._TSIDSeed,
PyTablesPersistentTSAccessor(self, DBLocator, table._v_pathname))
ts._persistentSize = table.nrows
if ts._persistentSize > 0:
ts._firstPersistentTS = table.read(start=0,
field="Date")[0]
ts._lastPersistentTS = table.read(start=table.nrows-1,
field="Date")[0]
newTSByTSID[self._TSIDSeed] = ts
self._timeseriesByTSID.update(newTSByTSID)
Nothing there that's obviously holding on memory.
What information do I need to provide to figure out what is going on
here?
Thanks
David
Visit our website at http://www.ubs.com
This message contains confidential information and is intended only
for the individual named. If you are not the named addressee you
should not disseminate, distribute or copy this e-mail. Please
notify the sender immediately by e-mail if you have received this
e-mail by mistake and delete this e-mail from your system.
E-mails are not encrypted and cannot be guaranteed to be secure or
error-free as information could be intercepted, corrupted, lost,
destroyed, arrive late or incomplete, or contain viruses. The sender
therefore does not accept liability for any errors or omissions in the
contents of this message which arise as a result of e-mail transmission.
If verification is required please request a hard-copy version. This
message is provided for informational purposes and should not be
construed as a solicitation or offer to buy or sell any securities
or related financial instruments.
UBS Limited is a company limited by shares incorporated in the United
Kingdom registered in England and Wales with number 2035362.
Registered office: 1 Finsbury Avenue, London EC2M 2PP. UBS Limited
is authorised and regulated by the Financial Services Authority.
UBS AG is a public company incorporated with limited liability in
Switzerland domiciled in the Canton of Basel-City and the Canton of
Zurich respectively registered at the Commercial Registry offices in
those Cantons with Identification No: CH-270.3.004.646-4 and having
respective head offices at Aeschenvorstadt 1, 4051 Basel and
Bahnhofstrasse 45, 8001 Zurich, Switzerland. Registered in the
United Kingdom as a foreign company with No: FC021146 and having a
UK Establishment registered at Companies House, Cardiff, with No:
BR 004507. The principal office of UK Establishment: 1 Finsbury Avenue,
London EC2M 2PP. In the United Kingdom, UBS AG is authorised and
regulated by the Financial Services Authority.
UBS reserves the right to retain all messages. Messages are protected
and accessed only in legally justified cases.
------------------------------------------------------------------------------
What happens now with your Lotus Notes apps - do you make another costly
upgrade, or settle for being marooned without product support? Time to move
off Lotus Notes and onto the cloud with Force.com, apps are easier to build,
use, and manage than apps on traditional platforms. Sign up for the Lotus
Notes Migration Kit to learn more. http://p.sf.net/sfu/salesforce-d2d
_______________________________________________
Pytables-users mailing list
Pytables-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/pytables-users
