Re: [Vala] sqlheavy?
Hi everyone, I need to use sqlite in my app. Does anyone have any experience using the sqlheavy wrapper? Does it make working with sqlite in Vala more friendly in your experience? Is it kept up to date? thanks Did you read this document[1]? Seemed pretty user-friendly to me, although I haven't personally used sqlheavy yet. Considering that Geary (from the Shotwell devs) is using it, I assume it's pretty up to date. [1] http://code.google.com/p/sqlheavy/wiki/UserGuide ___ vala-list mailing list vala-list@gnome.org https://mail.gnome.org/mailman/listinfo/vala-list
Re: [Vala] sqlheavy?
On Thu, Jun 21, 2012 at 9:44 AM, Jonas Kulla nyocu...@googlemail.comwrote: I assume it's pretty up to date. Sadly that's not the case, which is why we're in the process of removing SQLHeavy from Geary. - Eric ___ vala-list mailing list vala-list@gnome.org https://mail.gnome.org/mailman/listinfo/vala-list
Re: [Vala] sqlheavy?
On Mon, Jul 2, 2012 at 2:22 PM, Eric Gregory e...@yorba.org wrote: On Thu, Jun 21, 2012 at 9:44 AM, Jonas Kulla nyocu...@googlemail.com wrote: I assume it's pretty up to date. Sadly that's not the case, which is why we're in the process of removing SQLHeavy from Geary. - Eric Would you mind revealing what you'll be replacing it with? Something in house? ___ vala-list mailing list vala-list@gnome.org https://mail.gnome.org/mailman/listinfo/vala-list
Re: [Vala] sqlheavy?
On Mon, 2012-07-02 at 11:22 -0700, Eric Gregory wrote: On Thu, Jun 21, 2012 at 9:44 AM, Jonas Kulla nyocu...@googlemail.comwrote: I assume it's pretty up to date. Sadly that's not the case, which is why we're in the process of removing SQLHeavy from Geary. I don't really understand this. First, based on http://redmine.yorba.org/issues/5034 it sounds like that's not the primary reason. If it were the issue, it would be trivial to resolve. There isn't really anything that needs to be done to keep SQLHeavy up to date, since SQLite isn't exactly exploding with new features. If you just want new tarballs released more regularly that's really not a problem--I released one last time you guys requested it and I can do it again. The other option is that I can just add someone from Yorba to the list of maintainers and you're welcome to make releases whenever you want. Removing the sqlheavy-gen-orm utility would make this even more simple (and likely unnecessary), since it would remove the dependency on libvala. The text of that bug leads me to believe that the primary reason is that ... it's not clear that it will ever support concurrent database access from multiple threads That's actually not completely true--you can still create multiple Database objects pointed to the same file on disk (just like in SQLite). What is missing is a layer to have SQLHeavy.Datbase automatically create multiple connections to a single database in order to support concurrent queries transparently. That's not a matter of keeping things up to date, that's a *major* new feature which has basically necessitated the creation of a whole new library (http://code.google.com/p/bump) as well as a rewrite of the SQLHeavy internals. I would be disappointed to see Geary move away from SQLHeavy, but if you have an alternative which works I certainly can't fault you for migrating to it. I think I need about a week of full-time work in order to finish the database/connection split, and since I'm very busy at work these days (plus other open source stuff) I don't know when I'll have that kind of time. Until I do, SQLHeavy is in basically the same situation as SQLite, which isn't a problem for the majority of users (including the software I created SQLHeavy for in the first place). -Evan ___ vala-list mailing list vala-list@gnome.org https://mail.gnome.org/mailman/listinfo/vala-list
Re: [Vala] sqlheavy?
Let me explain the situation with Geary and SQLHeavy. The ticket Eric referred to earlier does not fully describe our issues. (Warning: this is *long*.) First, I have to correct what Eric said earlier. It's not that SQLHeavy isn't up-to-date, rather, the features Geary most heavily relies upon are not as well- or fully-implemented as we need. Specifically, the Geary engine is a fully asynchronous library designed to handle multiple requests at once. Virtually all our interaction with SQLHeavy is through its asynchronous interfaces. That was the strongest motivation toward using it rather than coding directly to SQLite. However, during development we discovered serious flaws in SQLHeavy's async implementation. But Evan's right, as far as using it as a direct replacement for SQLite, it seems as full-featured as one would want it to be. Perhaps the ORM stuff isn't up to snuff, as Evan said, so people wanting to adopt SQLHeavy should consider that. I would also examine the bug database (found at http://code.google.com/p/sqlheavy/issues/list) before jumping in with both feet. What's motivating our decision to move away from SQLHeavy are the following: * Each asynchronous operation creates and destroys a thread: http://code.google.com/p/sqlheavy/issues/detail?id=11 When I say operation, I don't mean transaction, I mean each *operation*. Thus, if your QueryResult has 50 items in its set, 50 threads are created and destroyed while iterating the result. 1000 items, 1000 threads. This is a major performance hit, especially at startup, as Geary does a lot of database work while connecting to the server. It's painful to run Geary under gdb because of this problem. Every time I go into Geary to tune or optimize, this problem is in the mix affecting performance. I reported this bug a year ago, almost to the day. * Async transactions aren't asynchronous: http://code.google.com/p/sqlheavy/issues/detail?id=12 This was almost a showstopper for us. Without diving in too deep to the mechanics of this one, Geary uses transactions throughout its database layer to guarantee atomicity. SQLHeavy's async transactions can't make that guarantee. I didn't discover this until gobs of database code had already been written. The only short-term solution I could find meant, in essence, there's a master lock on Geary's database that guarantees only one block of async code may perform operations at a time. In other words, we're serializing database access. It's an ugly hack, but I coded it thinking the problem would be solved shortly, at which time I could rip it out. This bug is also almost a year old. * Cancelling an async operation doesn't cancel the scheduled thread: http://code.google.com/p/sqlheavy/issues/detail?id=17 * Use IOError.CANCELLED: http://code.google.com/p/sqlheavy/issues/detail?id=18 * QueryResult.finished doesn't respect Cancellable: http://code.google.com/p/sqlheavy/issues/detail?id=19 * next_async() can segfault if cancelled: http://code.google.com/p/sqlheavy/issues/detail?id=20 These are related in an oblique way. The first one hurts performance because there might be a lot of I/O against the database when the user switches Folders, causing all of it to be cancelled at once. Because the scheduled threads are not cancelled, the fresh, relevant I/O the user has requested is queued up waiting for all those threads ahead of it to die. The second bug is not as serious, but it means that callers to the Geary engine had to do special-case checking for two different types of exceptions rather than one (since CANCELLED is ok if the caller did, indeed, cancel the operation). I had to work around this to convert SQLHeavy's INTERRUPTED error to CANCELLED. This check has to happen throughout Geary's database code. The third and fourth required checking Cancellable.is_cancelled() everywhere in our database code. Again, a utility method that must be called from a lot of places. These three problems required me to go into our database layer and add a lot of special checking to correct and work around them. These were reported 6 to 8 months ago. * VersionedDatabase upgrade scripts don't run: http://code.google.com/p/sqlheavy/issues/detail?id=21 This is one of those features I was really excited about, so I was pretty disappointed when it didn't work. In Shotwell, we do all our database upgrades through manual code. I thought it was a great idea to handle it all through SQL scripts. But it looks like an internal SQLHeavy lock prevents the upgrade scripts from running, causing the app to hang at startup. We worked around this by implementing our own upgrade path. I want to make it clear: Evan is fulfilling a major need in the GNOME community with SQLHeavy. The alternative (GNOME-DB) is unpalatable for an app that needs an in-proc database and knows it's going to use SQLite. He's done a ton of work and I think it's a solid foundation for the future. I also don't think Evan's responsible to
Re: [Vala] sqlheavy?
Jim, Thanks, this is good information. I'm going to reply in-line people who are interested, but here is the TL;DR version (of this whole thread, really): SQLHeavy currently has issues with parallelism. If you need parallel execution of queries, SQLHeavy is currently as big of a pain as SQLite. If you just want an API for SQLite which easier to use, or if you just want to avoid blocking the UI (the original use case for the async stuff), SQLHeavy may be a good fit. To anyone who is reading this at some point in the future: if there is an SQLHeavy = 0.2 release, the issues with parallelism have been resolved. On Mon, 2012-07-02 at 20:07 -0700, Jim Nelson wrote: Let me explain the situation with Geary and SQLHeavy. The ticket Eric referred to earlier does not fully describe our issues. (Warning: this is *long*.) First, I have to correct what Eric said earlier. It's not that SQLHeavy isn't up-to-date, rather, the features Geary most heavily relies upon are not as well- or fully-implemented as we need. Specifically, the Geary engine is a fully asynchronous library designed to handle multiple requests at once. Virtually all our interaction with SQLHeavy is through its asynchronous interfaces. That was the strongest motivation toward using it rather than coding directly to SQLite. However, during development we discovered serious flaws in SQLHeavy's async implementation. You're absolutely right about this, though I'd like to emphasize the distinction between async and parallel. Geary's problems with SQLHeavy come mostly from trying to use the async interfaces to achieve parallelism, as opposed to simply not blocking the UI thread (which was the original purpose of the async API). Most of the issues below are actually related pretty closely under the hood and the proper fix is really to separate the connection from the database so you can transparently execute multiple queries, which completely replaces the existing async support (as well as the synchronous support). Some of these issues are probably resolvable within the existing code base, but it would take a lot of time and only mostly work. I would rather use that time to get the proper fix in place. For anything which is missing a response, just imagine I wrote something like The right solution for this issue relies on the parallel execution stuff. I think that actually covers most of this message. But Evan's right, as far as using it as a direct replacement for SQLite, it seems as full-featured as one would want it to be. Perhaps the ORM stuff isn't up to snuff, as Evan said, so people wanting to adopt SQLHeavy should consider that. I would also examine the bug database (found at http://code.google.com/p/sqlheavy/issues/list) before jumping in with both feet. The ORM stuff is, AFAIK, fully functional. It is a bit limited since I don't actually have a lot of experience with ORM (and no experience that I enjoyed)... I had hoped people who actually liked ORM would have some ideas about how to make it a bit more full-featured, but nobody has come up with anything so far (and now even if they did I don't really have time to implement them anymore). What *is* a problem is the sqlheavy-gen-orm tool (which generates Vala source code which uses the ORM stuff in the library). Though it will live on in the 0.1 branch for all eternity, I'm removing it in master (which will become 0.2 when I'm done with the parallel execution stuff). What's motivating our decision to move away from SQLHeavy are the following: * Each asynchronous operation creates and destroys a thread: http://code.google.com/p/sqlheavy/issues/detail?id=11 When I say operation, I don't mean transaction, I mean each *operation*. Thus, if your QueryResult has 50 items in its set, 50 threads are created and destroyed while iterating the result. 1000 items, 1000 threads. This is a major performance hit, especially at startup, as Geary does a lot of database work while connecting to the server. It's painful to run Geary under gdb because of this problem. Every time I go into Geary to tune or optimize, this problem is in the mix affecting performance. I reported this bug a year ago, almost to the day. * Async transactions aren't asynchronous: http://code.google.com/p/sqlheavy/issues/detail?id=12 This was almost a showstopper for us. Without diving in too deep to the mechanics of this one, Geary uses transactions throughout its database layer to guarantee atomicity. SQLHeavy's async transactions can't make that guarantee. I didn't discover this until gobs of database code had already been written. The only short-term solution I could find meant, in essence, there's a master lock on Geary's database that guarantees only one block of async code may perform operations at a time. In other words, we're serializing database access. It's an ugly hack, but I coded it thinking the problem would be solved shortly, at which time
Re: [Vala] SQLHeavy memory leak?
2012/3/22 ant blowb...@gmail.com It works very well, but I have a catastrophic memory leak. Hunting the problem down, it seems to be SQLHeavy.Query that is the culprit. In an attempt to distill the problem to its essentials, I crufted up the attached program; it's nonsense in that all it does is create lots of Queries and throw them away, but if I run it as 'dbtest 0' my memory usage is 1.3MB, and with 'dbtest 10' it's 171MB. I think you're right that this is a memory leak in SqlHeavy. The problem is that statements are compiled by the Query object, and finalized by the QueryResult object. So if a query is not executed, its statement is never finalized. This is a branch that fixes the problem [1]. Can you confirm that it works? I haven't tested it in any other scenario so I hope it doesn't introduce problems of its own. I've forwarded it to the issue tracker [2]. [1] https://gitorious.org/~tkluck/sqlheavy/tklucks-sqlheavy/commit/c91da3ff5fe5a819d0349bdab155b1d7a1f1cf54 [2] https://code.google.com/p/sqlheavy/issues/detail?id=24 ___ vala-list mailing list vala-list@gnome.org https://mail.gnome.org/mailman/listinfo/vala-list
Re: [Vala] SQLHeavy memory leak?
On 24 June 2012 12:35, Timo Kluck tkl...@infty.nl wrote: I think you're right that this is a memory leak in SqlHeavy. The problem is that statements are compiled by the Query object, and finalized by the QueryResult object. So if a query is not executed, its statement is never finalized. Hi, My original use case did use the QueryResult object; and leaked memory. The example given was the shortest possible I could come up with that illustrated the problem. This is a branch that fixes the problem [1]. Can you confirm that it works? I'm afraid I don't use SqlHeavy in this project any longer. cheers ant ___ vala-list mailing list vala-list@gnome.org https://mail.gnome.org/mailman/listinfo/vala-list
[Vala] sqlheavy?
Hi everyone, I need to use sqlite in my app. Does anyone have any experience using the sqlheavy wrapper? Does it make working with sqlite in Vala more friendly in your experience? Is it kept up to date? thanks Brian -- Duff ___ vala-list mailing list vala-list@gnome.org https://mail.gnome.org/mailman/listinfo/vala-list
Re: [Vala] sqlheavy?
I know they Yorba is using it for their new Geary client. Check here: http://git.yorba.org/cgit.cgi/geary/ Steven N. Oliver On Thu, Jun 21, 2012 at 10:00 AM, Brian Duffy brdu...@gmail.com wrote: Hi everyone, I need to use sqlite in my app. Does anyone have any experience using the sqlheavy wrapper? Does it make working with sqlite in Vala more friendly in your experience? Is it kept up to date? thanks Brian -- Duff ___ vala-list mailing list vala-list@gnome.org https://mail.gnome.org/mailman/listinfo/vala-list ___ vala-list mailing list vala-list@gnome.org https://mail.gnome.org/mailman/listinfo/vala-list
Re: [Vala] sqlheavy?
On Thu, 2012-06-21 at 10:00 -0400, Brian Duffy wrote: Hi everyone, I need to use sqlite in my app. Does anyone have any experience using the sqlheavy wrapper? Does it make working with sqlite in Vala more friendly in your experience? Is it kept up to date? Discalaimer: I wrote SQLHeavy so I can hardly be considered unbiased, but I'll try to be. I've used SQLHeavy quite a bit in some proprietary software (which is why I wrote it), and in my experience it really does make SQLite a lot easier to use. As for keeping it up to date, it's not like SQLite is changing heavily with each release... it's really quite stable, and SQLHeavy doesn't actually need to change. There are a few problems: * Asynchronous queries can be problematic, especially when not serialized. I'm working on rewriting some of the internals to use Bump [1] to fix that by transparently creating multiple connections to the database, but my spare time is a bit limited these days so it is going slowly. * The ORM generator tool (sqlheavy-gen-orm) is basically crap. I'm planning on removing it. I do plan to keep the ORM code in the library(SQLHeavy.Table, Row, etc.), just don't rely on the sqlheavy-gen-orm command-line tool. * libsqlheavy-gtk isn't ready for production use yet. [1] http://code.google.com/p/bump ___ vala-list mailing list vala-list@gnome.org https://mail.gnome.org/mailman/listinfo/vala-list
Re: [Vala] sqlheavy?
On Thu, 2012-06-21 at 19:56 +0100, ant wrote: SQLHeavy is an awesome interface..unfortunately I had to abandon it in my project because there is (or was) a memory leak in SQLHeavy.Query. If you create one, and let it go out of scope, not all the memory is recovered. Search the archive for an example. I don't remember seeing that, and I don't see anything in the archives... If you can find a link (or supply a test case) I'd really like to take a look. Having said that, I was creating tens of thousands of the buggers, which rapidly became an issue on a 4GB VM. If your requirements are less taxing, SQLHeavy is well worth the price of a few K leakage. I ended up swiping all the ideas and writing my own wrapper that did just enough for what I neededsorry Evan, I'm too thick to fix it properly ;) If I needed lots of raw performance I'd probably just use sqlite directly, too. Actually, I'd probably use something else (leveldb, tokyo cabinet, etc.) if I could. cheers ant On 21 June 2012 19:24, Evan Nemerson e...@coeus-group.com wrote: On Thu, 2012-06-21 at 10:00 -0400, Brian Duffy wrote: Hi everyone, I need to use sqlite in my app. Does anyone have any experience using the sqlheavy wrapper? Does it make working with sqlite in Vala more friendly in your experience? Is it kept up to date? Discalaimer: I wrote SQLHeavy so I can hardly be considered unbiased, but I'll try to be. I've used SQLHeavy quite a bit in some proprietary software (which is why I wrote it), and in my experience it really does make SQLite a lot easier to use. As for keeping it up to date, it's not like SQLite is changing heavily with each release... it's really quite stable, and SQLHeavy doesn't actually need to change. There are a few problems: * Asynchronous queries can be problematic, especially when not serialized. I'm working on rewriting some of the internals to use Bump [1] to fix that by transparently creating multiple connections to the database, but my spare time is a bit limited these days so it is going slowly. * The ORM generator tool (sqlheavy-gen-orm) is basically crap. I'm planning on removing it. I do plan to keep the ORM code in the library(SQLHeavy.Table, Row, etc.), just don't rely on the sqlheavy-gen-orm command-line tool. * libsqlheavy-gtk isn't ready for production use yet. [1] http://code.google.com/p/bump ___ vala-list mailing list vala-list@gnome.org https://mail.gnome.org/mailman/listinfo/vala-list ___ vala-list mailing list vala-list@gnome.org https://mail.gnome.org/mailman/listinfo/vala-list ___ vala-list mailing list vala-list@gnome.org https://mail.gnome.org/mailman/listinfo/vala-list
[Vala] SQLHeavy memory leak?
Hi all,
I have the classic Gtk enormous TreeView backed by SQL database
problem, which I've attempted to solve with the entirely excellent
SQLHeavy and the accompanying TreeModel it provides.
It works very well, but I have a catastrophic memory leak. Hunting the
problem down, it seems to be SQLHeavy.Query that is the culprit.
In an attempt to distill the problem to its essentials, I crufted up
the attached program; it's nonsense in that all it does is create lots
of Queries and throw them away, but if I run it as 'dbtest 0' my
memory usage is 1.3MB, and with 'dbtest 10' it's 171MB.
I've tried fiddling with the various db options, they don't seem to be
especially relevant.
Am I doing something fundamentally stupid, or is this a leak?
cheers
ant
using Gtk;
class Main : GLib.Object {
public int queries = 0;
public SQLHeavy.Database db;
public SQLHeavy.Transaction trans;
public SQLHeavy.Query ins_query;
public Main(int n) {
queries = n;
}
public void run() {
try {
db = new SQLHeavy.Database(test.sqlite);
db.sql_executed.connect((sql) = { debug(::%s, sql); });
/* make it as speedy as possible */
db.synchronous = SQLHeavy.SynchronousMode.OFF; // don't
wait for disk writes
db.count_changes = false; // don't return number of rows
changed for INSERT, UPDATE, DELETE
db.temp_store = SQLHeavy.TempStoreMode.MEMORY; // store
temp tables and indices in RAM
db.journal_mode = SQLHeavy.JournalMode.OFF; // disable journalling
db.cache_size = 8000; // allow 8000 1KB pages for cache
db.enable_profiling = false;
/*db.auto_vacuum = SQLHeavy.AutoVacuum.FULL;*/ // makes no
difference
db.execute(DROP TABLE IF EXISTS `log`);
db.execute(CREATE TABLE `log` ( `secs` INTEGER, `msecs`
INTEGER ));
trans = db.begin_transaction();
ins_query = trans.prepare(INSERT INTO log VALUES(?,?););
/* create some records */
for(var i = 0; i 10; i++) {
ins_query.bind_int(0, 42);
ins_query.bind_int(1, 42);
ins_query.execute();
}
trans.commit();
/* create some queries */
for(var i = 0; i queries; i++) {
SQLHeavy.Query q = db.prepare(SELECT `ROWID` FROM log);
/*q.clear();*/ // makes no difference
}
stderr.printf(done, %d queries\n, queries);
} catch(SQLHeavy.Error err) {
error(err.message);
}
}
public static int main(string[] args) {
Gtk.init(ref args);
Main app = new Main(args[1].to_int());
app.run();
Gtk.main();
return 0;
}
}
___
vala-list mailing list
vala-list@gnome.org
http://mail.gnome.org/mailman/listinfo/vala-list
