Feng wrote:
> Hi all, when I query into a big table, it lead to memory error.
MySQL-python fully buffers a result set into memory before providing it to
the client via fetchone(). Its a behavior of the DBAPI.
>
> Here is the small script which lead to the memory error:
> ---------------------------------------------------------
> engine = sqlalchemy.create_engine("mysql://user:[email protected]/
> homo_sapiens_variation_56_37a")
> meta = sqlalchemy.MetaData()
> meta.bind = engine
> Session = sessionmaker(bind = engine)
> table_variation = sqlalchemy.Table('variation', meta, autoload=True)
> class Variation(object): pass
> mapper(Variation, table_variation)
> session = Session()
> al_records = session.query(Variation)
> for record in al_records: #there are 18 million records in the table
> variation
> pass
>
> ---------------------------------------------------------
> And here are the error information. I am using the SQLAlchemy of 0.56,
> python 2.5, and OS X. Is it a bug of SQLAlchemy? Could anyone please
> help me out? Thanks.
>
> Python(43867) malloc: *** mmap(size=262144) failed (error code=12)
> *** error: can't allocate region
> *** set a breakpoint in malloc_error_break to debug
> ...
> Python(43867) malloc: *** mmap(size=262144) failed (error code=12)
> *** error: can't allocate region
> *** set a breakpoint in malloc_error_break to debug
> Python(43867) malloc: *** mmap(size=2097152) failed (error code=12)
> *** error: can't allocate region
> *** set a breakpoint in malloc_error_break to debug
> Traceback (most recent call last):
> File "test_memory_leak.py", line 28, in <module>
> for record in al_records: # load all 18 million records into the
> memory
> File "/Library/Python/2.5/site-packages/SQLAlchemy-0.5.6-py2.5.egg/
> sqlalchemy/orm/query.py", line 1361, in __iter__
> return self._execute_and_instances(context)
> File "/Library/Python/2.5/site-packages/SQLAlchemy-0.5.6-py2.5.egg/
> sqlalchemy/orm/query.py", line 1364, in _execute_and_instances
> result = self.session.execute(querycontext.statement,
> params=self._params, mapper=self._mapper_zero_or_none())
> File "/Library/Python/2.5/site-packages/SQLAlchemy-0.5.6-py2.5.egg/
> sqlalchemy/orm/session.py", line 755, in execute
> clause, params or {})
> File "/Library/Python/2.5/site-packages/SQLAlchemy-0.5.6-py2.5.egg/
> sqlalchemy/engine/base.py", line 824, in execute
> return Connection.executors[c](self, object, multiparams, params)
> File "/Library/Python/2.5/site-packages/SQLAlchemy-0.5.6-py2.5.egg/
> sqlalchemy/engine/base.py", line 874, in _execute_clauseelement
> return self.__execute_context(context)
> File "/Library/Python/2.5/site-packages/SQLAlchemy-0.5.6-py2.5.egg/
> sqlalchemy/engine/base.py", line 896, in __execute_context
> self._cursor_execute(context.cursor, context.statement,
> context.parameters[0], context=context)
> File "/Library/Python/2.5/site-packages/SQLAlchemy-0.5.6-py2.5.egg/
> sqlalchemy/engine/base.py", line 948, in _cursor_execute
> self.dialect.do_execute(cursor, statement, parameters,
> context=context)
> File "/Library/Python/2.5/site-packages/SQLAlchemy-0.5.6-py2.5.egg/
> sqlalchemy/engine/default.py", line 123, in do_execute
> cursor.execute(statement, parameters)
> File "build/bdist.macosx-10.5-i386/egg/MySQLdb/cursors.py", line
> 173, in execute
> File "build/bdist.macosx-10.5-i386/egg/MySQLdb/connections.py", line
> 36, in defaulterrorhandler
> MemoryError
>
> --
>
> You received this message because you are subscribed to the Google Groups
> "sqlalchemy" group.
> To post to this group, send email to [email protected].
> To unsubscribe from this group, send email to
> [email protected].
> For more options, visit this group at
> http://groups.google.com/group/sqlalchemy?hl=en.
>
>
>
--
You received this message because you are subscribed to the Google Groups
"sqlalchemy" group.
To post to this group, send email to [email protected].
To unsubscribe from this group, send email to
[email protected].
For more options, visit this group at
http://groups.google.com/group/sqlalchemy?hl=en.
