Re: [SQLObject] Vacation
Have a nice time off Oleg:) Juan Manuel From: Oleg Broytman To: SQLObject discussion Date: Wednesday 26 May 2010 > Hello. I'm leaving the Net and diving into the Real Life. Will be back at > 14th of June. > > Oleg. -- ___ sqlobject-discuss mailing list sqlobject-discuss@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/sqlobject-discuss
Re: [SQLObject] Reply to list (was: Search for Full Name)
From: Bill Campbell To: sqlobject-discuss@lists.sourceforge.net Date: Wednesday 14 April 2010 > On Wed, Apr 14, 2010, j...@qlf.com wrote: > >FWIW, the other 3 Python-related lists I'm on all default to reply to the > >list. I use a mixture of Outlook and Thunderbird. Thunderbird has the > >'Reply to List' option, but sadly, Outlook doesn't offer that. > > I'm in the reply munging harmful camp, and use mutt which makes > it very simple with it's ``L'' to respond to the list. > > I have seen too many cases were somebody thought they were making > a private reply to the sender that went to a list causing much > embarassment. > > IMHO, it should require a tiny bit of effort before sending a > message to hundreds of recipients. > > Bill I have to admit that I was a bit annoyed that my zombie "reply to" on this mailing list always ended up in the last poster's mailbox. But after reading the article posted here about munging considered harmful, I'm definitely on that side now. It can be seen more clearly when you think about it in these terms: which one is worse? To send a private message to an entire list, or to send a list message to a private mailbox? The second can be amended (the recipient was going to receive the message anyway). The first one can't. Just my 0.02 Cheers Juan Manuel -- Download Intel® Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev ___ sqlobject-discuss mailing list sqlobject-discuss@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/sqlobject-discuss
Re: [SQLObject] More connections for more tables
From: Visgean Skeloru To: sqlobject-discuss@lists.sourceforge.net Date: Tuesday 13 April 2010 > Hello is it possible to set a specific connections database for each class? > > because i have 2 class and one of them is temporary so i would like to use > sqlite in memory. Yes, it's possible. One way is to pass a connection parameter at class creation (that's at least the way I do it). This may look like: from sqlobject import SQLObject, connectionForURI class AnObject(SQLObject): one_attr = StringCol() con = connectionForURI("sqlite:/:memory:") obj = AnObject(one_attr="one attr", connection=con) The other way is defining the _connection attribute on the class definition. I have never tried it but from reading the doc I think I remember it quite like this: con = connectionForURI("sqlite:/:memory:") class AnObject(SQLObject) one_attr = StringCol() _connection = con With the first one you have to keep in mind to pass the connection parameters in later operations, such as select (again, that's at least the way I do it). I have never fiddled with the second one, so I can't provide some experience on the matter. Please do post your results :) Cheers Juan Manuel Santos -- Download Intel® Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev ___ sqlobject-discuss mailing list sqlobject-discuss@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/sqlobject-discuss
Re: [SQLObject] rdbhost
On Fri, Apr 9, 2010 at 6:05 PM, Oleg Broytman wrote: > On Fri, Apr 09, 2010 at 05:51:46PM -0300, Juan Manuel Santos wrote: > > > > http://svn.colorstudy.com/SQLObject/trunk/sqlobject/ > > > > > >Aha, I see - I have forgotten to add it to setup.py. Will be fixed. > > > > No probs, just post when it's ready and I'll test > >Must be ready now. > > Oleg. > -- > Oleg Broytmanhttp://phd.pp.ru/p...@phd.pp.ru > Programmers don't die, they just GOSUB without RETURN. > Works fine Oleg, thanks :) Juan Manuel -- Download Intel® Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev___ sqlobject-discuss mailing list sqlobject-discuss@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/sqlobject-discuss
Re: [SQLObject] rdbhost
On Fri, Apr 9, 2010 at 5:34 PM, Oleg Broytman wrote: > On Sat, Apr 10, 2010 at 12:32:02AM +0400, Oleg Broytman wrote: > > > import rdbhost > > > ImportError: No module named rdbhost > > > >That's strange. I see the directory in SVN: > > http://svn.colorstudy.com/SQLObject/trunk/sqlobject/ > >and I can import it. > >Aha, I see - I have forgotten to add it to setup.py. Will be fixed. > > Oleg. > -- > Oleg Broytmanhttp://phd.pp.ru/p...@phd.pp.ru > Programmers don't die, they just GOSUB without RETURN. > > > -- > Download Intel® Parallel Studio Eval > Try the new software tools for yourself. Speed compiling, find bugs > proactively, and fine-tune applications for parallel performance. > See why Intel Parallel Studio got high marks during beta. > http://p.sf.net/sfu/intel-sw-dev > ___ > sqlobject-discuss mailing list > sqlobject-discuss@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/sqlobject-discuss > No probs, just post when it's ready and I'll test Cheers Juan Manuel -- Download Intel® Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev___ sqlobject-discuss mailing list sqlobject-discuss@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/sqlobject-discuss
Re: [SQLObject] AttributeError: No connection has been defined for this thread or process
On Fri, Apr 9, 2010 at 4:31 PM, Oleg Broytman wrote: > On Fri, Apr 09, 2010 at 03:56:56PM -0300, Juan Manuel Santos wrote: > > Thanks! Now it's working well with latest trunk. > >Thank you for reporting. > > > Only thing is I had to > > comment the rdbhost import since it can't find the module > >Oops, what's up? What did you comment out? What was the error? > > > Thanks Oleg, you're a lifesaver :) > >You are welcome! > > Oleg. > -- > Oleg Broytmanhttp://phd.pp.ru/p...@phd.pp.ru > Programmers don't die, they just GOSUB without RETURN. > > > -- > Download Intel® Parallel Studio Eval > Try the new software tools for yourself. Speed compiling, find bugs > proactively, and fine-tune applications for parallel performance. > See why Intel Parallel Studio got high marks during beta. > http://p.sf.net/sfu/intel-sw-dev > ___ > sqlobject-discuss mailing list > sqlobject-discuss@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/sqlobject-discuss > There's a "import rdbhost" statement in dbconnection.py: godl...@snorlax:~/indexor/indexor$ ./runner.py Traceback (most recent call last): File "./runner.py", line 21, in from controller.mainhandler import MainHandler File "/home/godlike/indexor/indexor/controller/mainhandler.py", line 27, in from fs.entities import File, Directory, Video, Audio, Photo File "/home/godlike/indexor/indexor/fs/entities.py", line 18, in from sqlobject import SQLObject, StringCol, FloatCol, BoolCol, IntCol, \ File "/usr/local/lib/python2.6/dist-packages/SQLObject-0.13dev_r4161-py2.6.egg/sqlobject/__init__.py", line 8, in from main import * File "/usr/local/lib/python2.6/dist-packages/SQLObject-0.13dev_r4161-py2.6.egg/sqlobject/main.py", line 32, in import dbconnection File "/usr/local/lib/python2.6/dist-packages/SQLObject-0.13dev_r4161-py2.6.egg/sqlobject/dbconnection.py", line 969, in import rdbhost ImportError: No module named rdbhost I just commented that one import line out, since it apparently didn't had anything related. I have no use/intention for it in this test environment :P. -- Download Intel® Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev___ sqlobject-discuss mailing list sqlobject-discuss@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/sqlobject-discuss
Re: [SQLObject] AttributeError: No connection has been defined for this thread or process
On Fri, Apr 9, 2010 at 3:16 PM, Oleg Broytman wrote: > On Thu, Apr 08, 2010 at 07:50:59PM -0300, Juan Manuel Santos wrote: > > if self._connection is not connection: > [skip] > > "No connection has been defined for this thread " > > AttributeError: No connection has been defined for this thread or process > >I think I fixed it. Please try the code from SVN - any branch will do, > 0.11, 0.12 or the trunk. > > Oleg. > -- > Oleg Broytmanhttp://phd.pp.ru/p...@phd.pp.ru > Programmers don't die, they just GOSUB without RETURN. > > > -- > Download Intel® Parallel Studio Eval > Try the new software tools for yourself. Speed compiling, find bugs > proactively, and fine-tune applications for parallel performance. > See why Intel Parallel Studio got high marks during beta. > http://p.sf.net/sfu/intel-sw-dev > ___ > sqlobject-discuss mailing list > sqlobject-discuss@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/sqlobject-discuss > Thanks! Now it's working well with latest trunk. Only thing is I had to comment the rdbhost import since it can't find the module and I don't know where is such module/package for Jaunty. But everything else seems to work fine. Thanks Oleg, you're a lifesaver :) Cheers Juan Manuel Santos -- Download Intel® Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev___ sqlobject-discuss mailing list sqlobject-discuss@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/sqlobject-discuss
Re: [SQLObject] AttributeError: No connection has been defined for this thread or process
On Fri, Apr 9, 2010 at 1:10 PM, Oleg Broytman wrote: > On Fri, Apr 09, 2010 at 12:34:43PM -0300, Juan Manuel Santos wrote: > > I've been doing some testing. On a Jaunty installation, with most recent > > SQLObject (SVN), it doesn't make a difference if I set fromDatabase as a > > parameter when creating a class or doing a select. If I specify it inside > > the SQLObject class definitions (inside class sqlmeta), I get the same > "No > > connection has been defined" but at starting time, when the first import > of > > the class is being done. > >No, you get a different error. Really, the error message is the same > 'cause it came from the same __get__, but the code that triggered call to > that __get__ is different. I think I understand what is going on. > > > With the SQLObject version that comes with Jaunty (0.10.2-3), > fromDatabase > > is not recognized as a parameter to be passed at object creation (and I > > would assume for select isn't either, though I haven't tested). > >fromDatabase can only be used as a variable (actually, constant) of the > inner sqlmeta because it only affects class creation. > > Oleg. > -- > Oleg Broytmanhttp://phd.pp.ru/p...@phd.pp.ru > Programmers don't die, they just GOSUB without RETURN. > > > -- > Download Intel® Parallel Studio Eval > Try the new software tools for yourself. Speed compiling, find bugs > proactively, and fine-tune applications for parallel performance. > See why Intel Parallel Studio got high marks during beta. > http://p.sf.net/sfu/intel-sw-dev > ___ > sqlobject-discuss mailing list > sqlobject-discuss@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/sqlobject-discuss > Alright! Let me know if you need any additional info/tracebacks. Cheers Juan Manuel -- Download Intel® Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev___ sqlobject-discuss mailing list sqlobject-discuss@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/sqlobject-discuss
Re: [SQLObject] AttributeError: No connection has been defined for this thread or process
On Fri, Apr 9, 2010 at 10:49 AM, Oleg Broytman wrote: > On Fri, Apr 09, 2010 at 10:37:06AM -0300, Juan Manuel Santos wrote: > > This error was raised on the first object creation the program does. > However, > > If I make the code in main.py workaround this, it fails on the first > select > > the program does. > > And no, I don't use fromDatabase = True, but would that help? > >Yes, it helps to understand what is going on. The test suite is used a > global connection so it cannot catch the bug in the code if there was no > connection. > I will look into it and fix the bug. > > Oleg. > -- > Oleg Broytmanhttp://phd.pp.ru/p...@phd.pp.ru > Programmers don't die, they just GOSUB without RETURN. > > > -- > Download Intel® Parallel Studio Eval > Try the new software tools for yourself. Speed compiling, find bugs > proactively, and fine-tune applications for parallel performance. > See why Intel Parallel Studio got high marks during beta. > http://p.sf.net/sfu/intel-sw-dev > ___ > sqlobject-discuss mailing list > sqlobject-discuss@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/sqlobject-discuss > I've been doing some testing. On a Jaunty installation, with most recent SQLObject (SVN), it doesn't make a difference if I set fromDatabase as a parameter when creating a class or doing a select. If I specify it inside the SQLObject class definitions (inside class sqlmeta), I get the same "No connection has been defined" but at starting time, when the first import of the class is being done. With the SQLObject version that comes with Jaunty (0.10.2-3), fromDatabase is not recognized as a parameter to be passed at object creation (and I would assume for select isn't either, though I haven't tested). However, if I run the same test with this older version, it doesn't break at class creation but on the first select, and this is the traceback: Traceback (most recent call last): File "/home/godlike/indexor/indexor/controller/mainhandler.py", line 514, in check_if_indexing_finished self._tvhandler.print_output() File "/home/godlike/indexor/indexor/controller/tvhandler.py", line 153, in print_output root = rootselect[0] File "/usr/lib/python2.6/dist-packages/sqlobject/sresults.py", line 173, in __getitem__ return list(self.clone(start=start, end=start+1))[0] File "/usr/lib/python2.6/dist-packages/sqlobject/sresults.py", line 179, in __iter__ return iter(list(self.lazyIter())) File "/usr/lib/python2.6/dist-packages/sqlobject/inheritance/iteration.py", line 26, in next if not self.lazyColumns: self.fetchChildren() File "/usr/lib/python2.6/dist-packages/sqlobject/inheritance/iteration.py", line 72, in fetchChildren childUpdate=True) File "/usr/lib/python2.6/dist-packages/sqlobject/inheritance/__init__.py", line 447, in select clause, *args, **kwargs) File "/usr/lib/python2.6/dist-packages/sqlobject/main.py", line 1345, in select join=join, forUpdate=forUpdate) File "/usr/lib/python2.6/dist-packages/sqlobject/inheritance/__init__.py", line 34, in __init__ dbName = (ops.get('connection',None) or sourceClass._connection).dbName File "/usr/lib/python2.6/dist-packages/sqlobject/dbconnection.py", line 819, in __get__ return self.getConnection() File "/usr/lib/python2.6/dist-packages/sqlobject/dbconnection.py", line 832, in getConnection "No connection has been defined for this thread " AttributeError: No connection has been defined for this thread or process Hope it helps. Please do tell me if you need additional testing or info Juan Manuel Santos -- Download Intel® Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev___ sqlobject-discuss mailing list sqlobject-discuss@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/sqlobject-discuss
Re: [SQLObject] AttributeError: No connection has been defined for this thread or process
From: Oleg Broytman To: sqlobject-discuss@lists.sourceforge.net Date: Friday 09 April 2010 > On Thu, Apr 08, 2010 at 07:50:59PM -0300, Juan Manuel Santos wrote: > > File "/usr/local/lib/python2.6/dist-packages/sqlobject/main.py", line > > > > 1211, in __init__ > > > > if self._connection is not connection: > [skip] > > > "No connection has been defined for this thread " > > > > AttributeError: No connection has been defined for this thread or process > >This is a very fresh code, I added it in the last releases. Try a > release before the very last - 0.11.3 or 0.12.1. > > Oleg. Actually I have 0.11.2 on my Gentoo box (if that suffices). I had a friend test the code on another Gentoo box and it worked alright. This problem is raised both on Ubuntu (Jaunty) and Debian (a mix between stable, testing and unstable), which IIRC have older versions (I checked the number yesterday but forgot it ;p but it was 0.10 or older). The strange thing is that a similar error (the one I posted) happened with the svn version. From: Oleg Broytman To: sqlobject-discuss@lists.sourceforge.net Date: Friday 09 April 2010 > On Thu, Apr 08, 2010 at 06:47:00PM -0300, Juan Manuel Santos wrote: > > I don't > > set any connection globally (since my plans are for the application to be > > able to manage several connections to different databases at the same > > time). Instead, I pass a connection parameter to every select, class > > creation and any SQLObject method which may require it. > >At waht phase the error is raised? Do you use fromDatabase=True? > > Oleg. This error was raised on the first object creation the program does. However, If I make the code in main.py workaround this, it fails on the first select the program does. And no, I don't use fromDatabase = True, but would that help? I'd gladly try it out if it does. Thanks, Juan Manuel Santos -- Download Intel® Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev ___ sqlobject-discuss mailing list sqlobject-discuss@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/sqlobject-discuss
Re: [SQLObject] AttributeError: No connection has been defined for this thread or process
Sorry, forgot to add this. A relevant traceback: Exception in thread Thread-2: Traceback (most recent call last): File "/usr/lib/python2.6/threading.py", line 525, in __bootstrap_inner self.run() File "/usr/lib/python2.6/threading.py", line 477, in run self.__target(*self.__args, **self.__kwargs) File "/home/godlike/indexor/indexor/logic/input/indexer.py", line 285, in do_index_process self.parse_size(size), absname) File "/home/godlike/indexor/indexor/logic/input/factory.py", line 73, in new_dir connection = self._conn) File "/usr/local/lib/python2.6/dist-packages/sqlobject/main.py", line 1211, in __init__ if self._connection is not connection: File "/usr/local/lib/python2.6/dist-packages/sqlobject/dbconnection.py", line 837, in __get__ return self.getConnection() File "/usr/local/lib/python2.6/dist-packages/sqlobject/dbconnection.py", line 850, in getConnection "No connection has been defined for this thread " AttributeError: No connection has been defined for this thread or process On Thu, Apr 8, 2010 at 6:47 PM, Juan Manuel Santos wrote: > Hello again list, > > I've been testing a program I'm coding on gentoo (SQLObject 0.11) on > Ubuntu/Debian. I tried both with older SQLObject (0.10 IIRC) and also > current SVN. But I'm getting the same error from the subject line. > > I currently can't paste a code example, but it is worth noting that I don't > set any connection globally (since my plans are for the application to be > able to manage several connections to different databases at the same time). > Instead, I pass a connection parameter to every select, class creation and > any SQLObject method which may require it. > > On Gentoo this behaviour does not get triggered at all. Any ideas as to why > this could be happening? > > Thanks > Juan Manuel Santos > -- Download Intel® Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev___ sqlobject-discuss mailing list sqlobject-discuss@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/sqlobject-discuss
[SQLObject] AttributeError: No connection has been defined for this thread or process
Hello again list, I've been testing a program I'm coding on gentoo (SQLObject 0.11) on Ubuntu/Debian. I tried both with older SQLObject (0.10 IIRC) and also current SVN. But I'm getting the same error from the subject line. I currently can't paste a code example, but it is worth noting that I don't set any connection globally (since my plans are for the application to be able to manage several connections to different databases at the same time). Instead, I pass a connection parameter to every select, class creation and any SQLObject method which may require it. On Gentoo this behaviour does not get triggered at all. Any ideas as to why this could be happening? Thanks Juan Manuel Santos -- Download Intel® Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev___ sqlobject-discuss mailing list sqlobject-discuss@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/sqlobject-discuss
Re: [SQLObject] Speed comparison
(grr it happened again >.<, my apologies to the guy who received this mail, it was intended for the list) From: Petr Jakeš To: vicariou...@gmail.com Date: Thursday 01 April 2010 > Some examples of the code, so we all can learn from your experiences, will > be nice. > > Regards > > Petr > Well, for this particular case, I was using a factory class that was in charge of creating the several objects that were required. It worked more or less like so: there is an indexer class, which takes care of crawling the filesystem. For every file or directory it encounters, it passes a series of parameters to the factory class (or object ;)), which the factory uses to instantiate the object that represents said file/dir. Yes, you may say that has too much coupling, but given the handling that one must do regarding connections and transactions (and taking care not to use them in a different thread than the one they were created on), it seemed the easiest and more or less "right" way to do it. So, let's say that there's one method in this Factory class/object to instantiate each of the objects that can be scanned from the filesystem (regular file, directory, image/video/audio file). Instantiation in itself isn't too expensive (or at least there isn't much that you can do about it). But when it comes to extracting the metadata for some particular files (namely, audio, video and image files) I found that I was setting fields on the object one at a time. This caused several UPDATE statements, which slowed everything down. So, in order to alleviate this, I did the following: def new_photo(self, parent, name, relpath, mimetype, atime, mtime, root, size, strsize, strabs): trans = self._conn.transaction() atime, mtime = self._check_stat(atime, mtime) mimetype = mimetype[0] photo = Photo(parent = parent, name = name, relpath = relpath, mimetype = mimetype, atime = atime, mtime = mtime, size = size, strsize = strsize, root = root, isdir = False, strabs = strabs, connection = trans) self._get_photo_metadata(photo) trans.commit(close = True) return photo The photo (I didn't use "Image" as a name so I wouldn't overwrite some other class already defined within Python) was the heaviest class in this regard, since it extracted two scaled images from the original (that's just something the pogram does, you can try it out if you want to). This extraction is done in the _get_photo_metadata method. So I got a transaction from the regular SQLObject connection, and used it as a parameter at object creation time, instead of the connection. After that I commit it with the close parameter on True, since then the method ends and I don't want that transaction lying around so that after working I still have a journal file for the SQLite db :P. Hope it is useful for somebody in the future. Don't forget that the transaction object you get from the connection.transaction() method, is already begun. You just have to work directly with it and DON'T FORGET to close it :) (via commit or rollback). Thanks for the help people. Cheers Juan Manuel Santos -- Download Intel® Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev ___ sqlobject-discuss mailing list sqlobject-discuss@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/sqlobject-discuss
Re: [SQLObject] Speed comparison
I believe I fixed the speed issue. I wrapped the most expensive functions/calls to SQLObject (expensive in terms of I/O) in a transaction. I didn't realize that that's the way SQLAlchemy works, and maybe that's why it was being faster (I had some functions which modified several attributes one at a time, which resulted in several UPDATE statements). Anyway in the end, and for the record, when wrapping the most expensive calls in a transaction (so they get executed all at once), there is little to no speed difference between SQLO and SQLA (and even SQLO turns out to be the faster when there's a difference). Nice tip to keep in mind :) Thanks everybody for their help, it was truly priceless! Cheers Juan Manuel Santos -- Download Intel® Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev ___ sqlobject-discuss mailing list sqlobject-discuss@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/sqlobject-discuss
Re: [SQLObject] SQLObject mass insertion
Got it, thanks One last thing and maybe a bit offtopic: how do I set "PRAGMA asyncrhonous = OFF" for the SQLite connection? I'm trying to see if the speed difference is due to this (as I suspect that SQLA uses PRAGMA), but this: self._conn.queryAll("PRAGMA synchronous=OFF;") or this: self._conn.query("PRAGMA synchronous=OFF;") Doesn't change anything. Any ideas? Thanks again, Juan Manuel From: Oleg Broytman To: sqlobject-discuss@lists.sourceforge.net Date: Thursday 01 April 2010 > On Thu, Apr 01, 2010 at 06:18:35AM -0300, Juan Manuel Santos wrote: > > 1/Query : CREATE TABLE meta_dir ( > > [...] > > 1/QueryR : CREATE TABLE meta_dir ( > > [...] > > > > Any idea why debug gets printed twice, but the second time with a capital > > 'r' after "Query"? :P > > > > Two debugging output are from > > > > > different methods. > > $ grep -F printDebug sqlobject/dbconnection.py > > def _executeRetry(self, conn, cursor, query): > if self.debug: > self.printDebug(conn, query, 'QueryR') > return cursor.execute(query) > > def _query(self, conn, s): > if self.debug: > self.printDebug(conn, s, 'Query') > self._executeRetry(conn, conn.cursor(), s) > > def _queryAll(self, conn, s): > if self.debug: > self.printDebug(conn, s, 'QueryAll') > [skip] > if self.debugOutput: > self.printDebug(conn, value, 'QueryAll', 'result') > >._query(), ._executeRetry(), etc can be called from different methods in > a different order so these extra debugging allows to trace the call stack. > > Oleg. -- Download Intel® Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev ___ sqlobject-discuss mailing list sqlobject-discuss@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/sqlobject-discuss
Re: [SQLObject] SQLObject mass insertion
Yes, you're right and that's a bit odd. I've set the debug variable in the usual way: self._conn = connectionForURI(con_str) self._conn.debug = True And I noticed this also: 1/Query : CREATE TABLE meta_dir ( [...] 1/QueryR : CREATE TABLE meta_dir ( [...] Any idea why debug gets printed twice, but the second time with a capital 'r' after "Query"? :P From: Oleg Broytman To: sqlobject-discuss@lists.sourceforge.net Date: Thursday 01 April 2010 > On Thu, Apr 01, 2010 at 05:47:06AM -0300, Juan Manuel Santos wrote: > > http://pastebin.com/raw.php?i=Xp1TjmZ0 > > > > Basically, if you look in the previous link for "Setting icon" or > > "Setting thumb" you will see that both the icon and the thumb are being > > set TWICE! with the same data. > >There are two debugging entries for every query (see CREATE TABLE for > examples) but every query runs only once. Two debugging output are from > different methods. > > Oleg. -- Download Intel® Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev ___ sqlobject-discuss mailing list sqlobject-discuss@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/sqlobject-discuss
Re: [SQLObject] SQLObject mass insertion (was: Speed comparison)
(Sorry, little mistake on the To: field and this didn't go to the list. My apologies for the one who received this :P) Hi everybody again, sorry for the delay in answering. I've been doing some more testing, with both ORMs (SQLO and SQLA) in debug mode, and seeing the SQL that gets sent to the db. In the meantime I also added somewhat of a counter for my app, which counts in 10ms multiples, the time that it takes to scan the directory. Pretty rough but it serves its purpose. So I started comparing both ORMs on different directories. An since SQLA almost halved the time it took SQLO to scan a directory full of big (1920x1200) wallpapers, I decided to debug it by scanning a directory with only one image. First of all, the structure of the class that maps the filesystem images onto my app, looks something like this: class Photo(File): hasthumb = BoolCol(default = None) author = StringCol(default = None) res = StringCol(default = None) date_taken = StringCol(default = None) soft = StringCol(default = None) _thumb = PickleCol(default = None) _icon = PickleCol(default = None) def get_thumb(self): return gtk.gdk.pixbuf_new_from_array(self._thumb, gtk.gdk.COLORSPACE_RGB, 8) def set_thumb(self, value): if isinstance(value, gtk.gdk.Pixbuf): self._thumb = value.get_pixels_array() def get_icon(self): return gtk.gdk.pixbuf_new_from_array(self._icon, gtk.gdk.COLORSPACE_RGB, 8) def set_icon(self, value): if isinstance(value, gtk.gdk.Pixbuf): self._icon = value.get_pixels_array() thumb = property(get_thumb, set_thumb) icon = property(get_icon, set_icon) Those properties are there because in order to pickle an image and be able to successfully unpickle it, you have to convert it to its pixels' array. Otherwise, you can't unpickle it. Here is the code that loads both thumb and icon into the object: photo.icon = \ gtk.gdk.pixbuf_new_from_file_at_size(photo.strabs, SETTINGS.thumblistsize, SETTINGS.thumblistsize) photo.thumb = \ gtk.gdk.pixbuf_new_from_file_at_size(photo.strabs, SETTINGS.thumbpanesize, SETTINGS.thumbpanesize) photo.hasthumb = True So, in my app, the thumb and icon for the Photo class are only loaded once, on the original scan. Then, by looking at SQLO's output I found out this: http://pastebin.com/raw.php?i=Xp1TjmZ0 Basically, if you look in the previous link for "Setting icon" or "Setting thumb" you will see that both the icon and the thumb are being set TWICE! with the same data. Now I honestly don't know where this comes from. SQLA apparently doesn't do this (hence, its better scanning time), and indeed it is an odd thing to do. I debugged a little more and found that indeed the properties for both icon and thumb are being called twice, though I don't know why. Is there maybe something in the code of the PickleCol that triggers this behaviour? Thanks for your help and time :) Cheers Juan Manuel Santos From: Oleg Broytman To: sqlobject-discuss@lists.sourceforge.net Date: Tuesday 23 March 2010 > On Tue, Mar 23, 2010 at 12:57:19AM -0300, Juan Manuel Santos wrote: > > Are there any ideas on why would SQLO > > be a bit slower > >When you declare a table > > class MyTable(SQLObject): >...columns... > > and do an INSERT by calling MyTable(columns) SQLObject immediately does a > SELECT to get back autogenerated fields (timestamps and the like) This is > slow. It's ok to create rows like this occasionally but it is certainly bad > for mass-insertion. >For mass-insertion use SQLBuilder. Alas, it's underdocumented. Go by > example: > > record = {'column1': value1, 'column2': value2, ...} > > connection.query(connection.sqlrepr( >sqlbuilder.Insert(MyTable.sqlmeta.table, values=record))) > >These are simple straightforward INSERTs without any additional > high-level burden - no SELECT, no caching, nothing. Fire and forget. >It is not as high-level as calling MyTable() but it is still high > enough - sqlrepr() does value conversion and quoting, e.g. > > Oleg. -- Download Intel® Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev ___ sqlobject-discuss mailing list sqlobject-discuss@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/sqlobject-discuss
Re: [SQLObject] Speed comparison
Sorry, missed that detail. In all cases I used SQLite, with pysqlite at version 2.5.6. Python is version 2.6.4 Cheers Juan Manuel From: Simon Cross To: sqlobject-discuss@lists.sourceforge.net Date: Tuesday 23 March 2010 > On Tue, Mar 23, 2010 at 5:57 AM, Juan Manuel Santos > > wrote: > > SQLObject: > > Text files: > > real0m25.704s > > user0m3.240s > > sys 0m1.424s > > > > SQLAlchemy/Elixir: > > Text files: > > real0m13.754s > > user0m6.696s > > sys 0m0.504s > > Which database backend are you using? > > Schiavo > Simon > > --- > --- Download Intel® Parallel Studio Eval > Try the new software tools for yourself. Speed compiling, find bugs > proactively, and fine-tune applications for parallel performance. > See why Intel Parallel Studio got high marks during beta. > http://p.sf.net/sfu/intel-sw-dev > ___ > sqlobject-discuss mailing list > sqlobject-discuss@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/sqlobject-discuss -- Download Intel® Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev ___ sqlobject-discuss mailing list sqlobject-discuss@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/sqlobject-discuss
Re: [SQLObject] Speed comparison
Sorry, missed that detail. In all cases I used SQLite, with pysqlite at version 2.5.6. Python is version 2.6.4 Cheers Juan Manuel From: Simon Cross To: sqlobject-discuss@lists.sourceforge.net Date: Tuesday 23 March 2010 > On Tue, Mar 23, 2010 at 5:57 AM, Juan Manuel Santos > > wrote: > > SQLObject: > > Text files: > > real0m25.704s > > user0m3.240s > > sys 0m1.424s > > > > SQLAlchemy/Elixir: > > Text files: > > real0m13.754s > > user0m6.696s > > sys 0m0.504s > > Which database backend are you using? > > Schiavo > Simon > > --- > --- Download Intel® Parallel Studio Eval > Try the new software tools for yourself. Speed compiling, find bugs > proactively, and fine-tune applications for parallel performance. > See why Intel Parallel Studio got high marks during beta. > http://p.sf.net/sfu/intel-sw-dev > ___ > sqlobject-discuss mailing list > sqlobject-discuss@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/sqlobject-discuss -- Download Intel® Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev ___ sqlobject-discuss mailing list sqlobject-discuss@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/sqlobject-discuss
[SQLObject] Speed comparison
Hi everybody, First of all, before anybody attacks me, I do not intend to start a flamewar. I have simply been testing out SQLObject and SQLAlchemy/Elixir for a program I'm writing (I previously posted on some issues I had). If I am to be honest, I like SQLObject a lot more, because its syntax and way of doing things is, IMHO, simpler and more Python-like. And I wouldn't have tried out Alchemy in the first place if it wasn't for Elixir. After battling between tutorials and documentation, I finally managed to create another development branch, based on my SQLObject one, fully working with Alchemy/Elixir. I did this (keeping two copies of the program in the same repo, with different backends) so as to do a speed comparison between the two. Again, not that I don't like SQLO, but I just wanted to see which one was faster, since it benefits my application, and WHY it was faster. The app I am talking about is a filesystem indexer, like Gnomecatalog or Basenji. Basically it scans the filesystem recursively and for every entry (file or directory), it extracts some metadata about it and stores it in a DB, so it can be accessed later (perfect for those storage HDs you have full of stuff you don't want to index by hand :)). Originally I just kept the whole scan content in memory and pickled it when asked, but with huge directories it turned into a problem, so I started looking for an alternative in the DB field. I haven't thoroughly scanned it, neither is my scanning method the most scientific one, but the results I got so far seem to point out that Alchemy/Elixir is faster than SQLO. I did two scans, one with some small text files in just one directory, and another one with plenty (1GB+) of wallpapers split into two directories. Here are the results: SQLObject: Text files: real0m25.704s user0m3.240s sys 0m1.424s Wallpapers: real5m25.644s user3m6.760s sys 0m16.881s SQLAlchemy/Elixir: Text files: real0m13.754s user0m6.696s sys 0m0.504s Wallpapers: real3m58.949s user3m4.392s sys 0m7.244s Of course, numbers may not be that big, but there is a difference which I figure would increase on bigger scans. Are there any ideas on why would SQLO be a bit slower than Alchemy? Is there something that can be done to amend this? If anybody wants to check out the code, its at git://vicarious.com.ar/indexor.git, or at github on http://github.com/godlike64/indexor. The branches are dev/godlike/elixir for the Elixir one, and dev/godlike/bleeding for the SQLObject one. Any questions feel free to ask me :) Cheers Juan Manuel Santos PS: the program is of course in development, so some things may not quite work. Actually, I timed it by opening, ordering it to scan a directory, and closing the window. After the scan finishes (in complete background), the program exits and you get the time prompt). -- Download Intel® Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev ___ sqlobject-discuss mailing list sqlobject-discuss@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/sqlobject-discuss
Re: [SQLObject] Manage objects in different databases under the same program instance
On Saturday 06 February 2010 06:54:31 Petr Jakeš wrote: > > In order to create the DB objects inside the approppriate DB, AFAIK I can > > pass a connection parameter in the class creation, and all will be fine. > > But how do I go about doing selects and obtaining results from a specific > > DB? I may have two DBs open, and each with its file & dir structure > > inside. If I want to do, say, File.select(), how can I specify that I > > want results from only one of the DBs? > > What about: > > from sqlobject import * > > connection01 = connectionForURI('sqlite:/:memory:') > connection02 = connectionForURI('sqlite:/:memory:') > > class File01(SQLObject): > _connection = connection01 > > class File02(SQLObject): > _connection = connection02 > I don't think that serves the purpose. The program is supposed to let the user scan whatever structure he/she wants, and this way i'm limiting that, and it is also not scalable. Thanks anyway -- The Planet: dedicated and managed hosting, cloud storage, colocation Stay online with enterprise data centers and the best network in the business Choose flexible plans and management services without long-term contracts Personal 24x7 support from experience hosting pros just a phone call away. http://p.sf.net/sfu/theplanet-com ___ sqlobject-discuss mailing list sqlobject-discuss@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/sqlobject-discuss
[SQLObject] Manage objects in different databases under the same program instance
Hello again everybody, As I mentioned in a post some days ago, I'm writing a program that scans a file & directory structure as part of an indexing process. My next development target is to make it able to manage several of these structures at the same time (for example, opening each one in a different tab). Maybe even being able to scan different structures at the same time (through threads, of course :)) Since I don't like huge DBs, each structure is saved to its own sqlite db upon scanning. Until now I've been using sqlhub.processConnection in order to create the classes into their appropriate DBs, but I believe that sqlhub.processConnection forces the assigned connection to be used througout the entire program instance, so if in another class (after assigning the connection) I create an instance of a SQLObject, it goes to that DB, no matter if the connection assigment was done elsewhere. In order to create the DB objects inside the approppriate DB, AFAIK I can pass a connection parameter in the class creation, and all will be fine. But how do I go about doing selects and obtaining results from a specific DB? I may have two DBs open, and each with its file & dir structure inside. If I want to do, say, File.select(), how can I specify that I want results from only one of the DBs? Thanks Juan Manuel -- The Planet: dedicated and managed hosting, cloud storage, colocation Stay online with enterprise data centers and the best network in the business Choose flexible plans and management services without long-term contracts Personal 24x7 support from experience hosting pros just a phone call away. http://p.sf.net/sfu/theplanet-com ___ sqlobject-discuss mailing list sqlobject-discuss@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/sqlobject-discuss
Re: [SQLObject] Problem with InheritableSQLObject and ForeignKeys/MultipleJoin
On Friday 29 January 2010 08:53:26 Oleg Broytman wrote: > On Thu, Jan 28, 2010 at 09:35:50PM -0300, Juan Manuel Santos wrote: > > class File(InheritableSQLObject): > > name = StringCol() > > root = ForeignKey("Directory", default=None) > > > > class Directory(File): > > dirs = MultipleJoin("Directory", joinColumn="root_id") > > files = MultipleJoin("File", joinColumn="root_id") > > > > sqlobject.dberrors.OperationalError: no such column: root_id > > > > 1/QueryR : SELECT id FROM directory WHERE root_id = (1) > >Table 'directory' doesn't have root_id, only 'file' as it. > >The problem is that InheritableSQLObject is rather limited thing. These > limitations are documented: > http://sqlobject.org/Inheritance.html#limitations-and-notes >The problem you are stumbled upon is "Inheritance works in two stages - > first it draws the IDs from the parent table and then it draws the rows > from the children tables. The first stage could fail if you try to do > complex things. For example, Children.select(orderBy=Children.q.column, > distinct=True) could fail because at the first stage inheritance generates > a SELECT query for the parent table with ORDER BY the column from the > children table." >Your code choked on the first stage while trying to SELECT IDs from the > child table using a column that only exists in the parent. > >You can try simple Python inheritance instead: > > class File(SQLObject): > name = StringCol() > root = ForeignKey("Directory", default=None) > > class Directory(File): > dirs = MultipleJoin("Directory", joinColumn="root_id") > files = MultipleJoin("File", joinColumn="root_id") > >This works by copying all parent columns to the child table. > > Oleg. > Yes, that's what I ended up doing, after I thought it might be a limitation. But well, it works fine for the rest of the classes. Thanks :) Juan Manuel -- The Planet: dedicated and managed hosting, cloud storage, colocation Stay online with enterprise data centers and the best network in the business Choose flexible plans and management services without long-term contracts Personal 24x7 support from experience hosting pros just a phone call away. http://p.sf.net/sfu/theplanet-com ___ sqlobject-discuss mailing list sqlobject-discuss@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/sqlobject-discuss
[SQLObject] Problem with InheritableSQLObject and ForeignKeys/MultipleJoin
Hi everybody. I've recently started using SQLObject for a Python program I'm writing to replace the previous object structure (was using Pickle but it got too big :P). Anyway, the relevant part is this: I have a File object (that I defined) and a Directory object, like so: class File(InheritableSQLObject): name = StringCol() root = ForeignKey("Directory", default=None) class Directory(File): dirs = MultipleJoin("Directory", joinColumn="root_id") files = MultipleJoin("File", joinColumn="root_id") File has more attributes, but I only put the relevant ones. As can be seen, each file has a root FK which points to the parent directory. Since directories are files, they inherit from File and the only extra attributes they provide are two lists, one for the files it contains and one for the subdirectories it contains. While I could do without this last part, it is there for compatibility's sake (since the old structure worked more or less like this). Changing this would require another further rewrite, which I'm trying to avoid. So, the problem: wether I retrieve a Directory through Directory.select or Directory.selectBy, I can access directory.files fine, but whenever I try to access directory.dirs, I get this: .select: File "/home/godlike/workspace/python/indexor/controller/tvhandler.py", line 133, in append_directories print _dir.dirs File "", line 1, in File "/usr/lib64/python2.6/site-packages/sqlobject/joins.py", line 144, in performJoin inst.id) File "/usr/lib64/python2.6/site-packages/sqlobject/dbconnection.py", line 549, in _SO_selectJoin self.sqlrepr(value))) File "/usr/lib64/python2.6/site-packages/sqlobject/dbconnection.py", line 362, in queryAll return self._runWithConnection(self._queryAll, s) File "/usr/lib64/python2.6/site-packages/sqlobject/dbconnection.py", line 262, in _runWithConnection val = meth(conn, *args) File "/usr/lib64/python2.6/site-packages/sqlobject/dbconnection.py", line 355, in _queryAll self._executeRetry(conn, c, s) File "/usr/lib64/python2.6/site- packages/sqlobject/sqlite/sqliteconnection.py", line 183, in _executeRetry raise OperationalError(ErrorMessage(e)) sqlobject.dberrors.OperationalError: no such column: root_id Select performed: 1/Select : SELECT file.id, file.parent, file.name, file.relpath, file.mimetype, file.atime, file.mtime, file.size, file.strsize, file.isdir, file.root_id, file.child_name FROM file WHERE (((file.name) = ('Desktop')) AND ((file.child_name) = ('Directory'))) LIMIT 1 1/QueryR : SELECT file.id, file.parent, file.name, file.relpath, file.mimetype, file.atime, file.mtime, file.size, file.strsize, file.isdir, file.root_id, file.child_name FROM file WHERE (((file.name) = ('Desktop')) AND ((file.child_name) = ('Directory'))) LIMIT 1 1/Select children of the class Directory: SELECT directory.id, directory.child_name FROM directory WHERE ((directory.id) = (1)) 1/QueryR : SELECT directory.id, directory.child_name FROM directory WHERE ((directory.id) = (1)) 1/COMMIT : auto 1/QueryAll: SELECT id FROM directory WHERE root_id = (1) 1/QueryR : SELECT id FROM directory WHERE root_id = (1) 1/COMMIT : auto .selectBy: File "/home/godlike/workspace/python/indexor/controller/tvhandler.py", line 133, in append_directories print _dir.dirs File "", line 1, in File "/usr/lib64/python2.6/site-packages/sqlobject/joins.py", line 144, in performJoin inst.id) File "/usr/lib64/python2.6/site-packages/sqlobject/dbconnection.py", line 549, in _SO_selectJoin self.sqlrepr(value))) File "/usr/lib64/python2.6/site-packages/sqlobject/dbconnection.py", line 362, in queryAll return self._runWithConnection(self._queryAll, s) File "/usr/lib64/python2.6/site-packages/sqlobject/dbconnection.py", line 262, in _runWithConnection val = meth(conn, *args) File "/usr/lib64/python2.6/site-packages/sqlobject/dbconnection.py", line 355, in _queryAll self._executeRetry(conn, c, s) File "/usr/lib64/python2.6/site- packages/sqlobject/sqlite/sqliteconnection.py", line 183, in _executeRetry raise OperationalError(ErrorMessage(e)) sqlobject.dberrors.OperationalError: no such column: root_id Select performed: 1/Select : SELECT directory.id, directory.child_name FROM directory, file WHERE file.name) = ('Desktop')) AND ((directory.id) = (file.id))) AND ((directory.id) = (file.id))) LIMIT 1 1/QueryR : SELECT directory.id, directory.child_name FROM directory, file WHERE file.name) = ('Desktop')) AND ((directory.id) = (file.id))) AND ((directory.id) = (file.id))) LIMIT 1 1/COMMIT : auto 1/QueryAll: SELECT id FROM directory WHERE root_id = (1) 1/QueryR : SELECT id FROM directory WHERE root_id = (1) 1/COMMIT : auto Any ideas? Thanks Juan Manuel -- The Planet: dedicated and managed hosting, cloud storage, colocation