Re: [sqlite] SQLite server using execnet ?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 02/21/2011 12:37 PM, Jay A. Kreibich wrote: > Yes, but in something like memcached, the database is not aware of > that structure, and can't take advantage of it. Memcached does understand some types and has atomic increment/decrement etc but does not understand structure. > When storing serialized objects, it is all too common to see code that > fetches an object, un-marshals it, alters some simple value, re-marshals > the whole object, and then write the whole thing back. This all circles back to what you are doing and in particular what the OP was doing. If you need to do queries, modifications and durability then what you really need falls under the category of 'database'. On the other hand if there is no need for querying or changes then something like memcached is a great way for a bunch of machines/processes to get the data. > Redis Redis is a database :-) > Yes and no. Redis, like memcached, is essentially an always in-memory > key/value store. *All* databases are in-memory for practical purposes. Their working set will need to be in memory either explicitly due to their implementation, or implicitly via the operating system or through administration (eg indices). If accesses to the working set of data require disk accesses then the performance will be dismal. (Some exceptions for data only accessed sequentially.) > Its main selling point is memcached-like speed, Incidentally MongoDB claims the same thing :-) The places I use memcached are where I do not want disk touched. >... but it is a good fit ... It looks like we are seeing what happened with the first generation of DVCS. Relational representation is being changed to be less constrained/structured. There are numerous "databases" with varying and overlapping sweet spots in terms of querying, persistence, performance, distribution etc. I expect we'll see similar shakeouts and end up with a small number of strong products., > Like SQLite itself, I tend do all my virtual table modules in > extremely vanilla C. Brave :-) My personal preference is to do the initial development in Python and then reimplement in C if needed for portability/performance reasons. The Python development is a lot quicker and then acts as a test suite for the C implementation. I wonder how many of the other bindings for SQLite have bothered to implement virtual tables as that probably holds back usage of virtual tables a lot. (A 'hello world' virtual table in Python/APSW is about half a screenful of code. An example one I have that represents information about files on disk is just under a screenful.) > I happen to think virtual tables are one of the more powerful features > of SQLite, but also one of the most under-utilized features. Agreed. Unfortunately it does require that the underlying data be representable in a relational manner which is also very constraining. > Since a big part of writing these is to get them out for other people > to use them, Where do you publish them? It is probably also worth trying to encourage a 'contrib' location for SQLite that is more active and in wider use than http://www.sqlite.org/contrib > Working in C avoids adding complexity, like someone working > in Java wanting to use your MongoDB module. I suppose it could be > done, but I wouldn't want to be the one trying to make it all work. MongoDB is client server so this issue does not arise. (Nor do they have virtual tables.) In order to perform "programming" on the server side such as for map/reduce you have to use Javascript which is slowly becoming the most popular language for that kind of thing including on the desktop. (Eg see node.js and Seed.) Roger -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk1i/zkACgkQmOOfHg372QT9cgCgyV7NaECzQUrrrDZr9zYri0tq RkkAoKSuRlclVshN/oIxSXOy0dtXZcot =xEyA -END PGP SIGNATURE- ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] SQLite server using execnet ?
On Feb 21, 2011, at 9:37 PM, Jay A. Kreibich wrote: > I was once forced to look at SOAP over SMTP Ah, yes... double S! The S stands for Simple http://wanderingbarque.com/nonintersecting/2006/11/15/the-s-stands-for-simple/ ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] SQLite server using execnet ?
On Sun, Feb 20, 2011 at 08:37:04PM -0800, Roger Binns scratched on the wall: > On 02/20/2011 06:23 PM, Jay A. Kreibich wrote: > > On Sun, Feb 20, 2011 at 05:23:09PM -0800, Roger Binns scratched on the wall: > >> If you want to use SQL then use Postfix. > > > > I might suggest PostgreSQL instead. > > (Sorry, Roger, I couldn't resist.) > > Yeah, long night :-) However, technically SQL over SMTP is possible and > would actually work. And if anyone is insane enough to try that then using > Postfix and Postgres are a good combination. I was once forced to look at SOAP over SMTP. Thankfully we were able to talk them out of it. As if SOAP wasn't bad enough, I sure didn't want to mix it with SMTP. "Any networking protocol with 'Simple' in the name isn't." > >> If you need lots of processes on the network to access data quickly then > >> consider memcached. > > > > More seriously, in this category you might also consider Redis. > > Redis allows your data to have some structure, > > The Python binding pylibmc does structure the data for you automagically. Yes, but in something like memcached, the database is not aware of that structure, and can't take advantage of it. When storing serialized objects, it is all too common to see code that fetches an object, un-marshals it, alters some simple value, re-marshals the whole object, and then write the whole thing back. While this is fine for complex or less frequently accessed objects, where the ease of use outweighs the overhead, it seems like a lot of work for simpler items, such as lists and basic dictionaries or hashes. Of course, these types of objects are also the most frequently modified data-types in many applications. The best way to do key lists in memcached is practically a religious topic. Having the database aware of a few very simple and primitive structures can provide significant improvements for these basic, most common operations. For example, Redis can append an element to a list with a single O(1) database operation that is independent of list size, and the only payload data that crosses the network is the list key and the value to insert. It has a few other tricks, like atomically incrementing counter values. > > plus it has the ability to persist the data to disk. > > The moment you talk about persistence you then have significant overlap with > databases. My personal favourite is MongoDB but there are loads of others > such as Cassandra, HBase, Tokyo Cabinet etc. Yes and no. Redis, like memcached, is essentially an always in-memory key/value store. Functionally, it is much more like memcached than most of these other examples. Its main selling point is memcached-like speed, without the "cache" aspects of memcached-- you're data stays there until you get rid of it, even across restarts. In many cases, that also allows you to get rid of your backing database. Redis also provides just enough internal structure to be useful, without really getting in the way, in much the same way that a very basic container library provides building-block tools without defining a whole class tree. Those features definitely bring it closer into the domain of NoSQL databases, but I'd argue that's only because memcached is so far removed from the rest of these. Thanks to its caching nature, it can't really be considered a "database" at all (useful tool, yes; database, not really). Each of these products has its place. I don't mean to sound like such a Redis fan-boy, but I've been messing around with it a lot lately, and found it to be both extremely simple to setup and configure (something most of these other products cannot claim), while also extremely useful. You're not going to use Redis to replace something like Cassandra or HBase, but it is a good fit for situations where memcached is a good fit, but you want more structure and a known life-cycle to your data. > What programming language are you using to implement the virtual tables? Like SQLite itself, I tend do all my virtual table modules in extremely vanilla C. I happen to think virtual tables are one of the more powerful features of SQLite, but also one of the most under-utilized features. As a way to relax and explore, I sometimes write virtual table modules to bolt together odd data stores or storage formats. Partly this is just for fun, but the process also helps develop a deep understanding of the native data model used by these different products, and that knowledge is useful for other work I do. If it opens some eyes and helps promote virtual tables, and SQLite in general, that's also a great bonus. As such, much of the virtual table development work I do is somewhat isolated, outside of any environment or problem context. My approach tends to be extremely general-purpose, since I'm often approaching the problem from a very
Re: [sqlite] SQLite server using execnet ?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 02/20/2011 06:23 PM, Jay A. Kreibich wrote: > On Sun, Feb 20, 2011 at 05:23:09PM -0800, Roger Binns scratched on the wall: >> If you want to use SQL then use Postfix. > > I might suggest PostgreSQL instead. > (Sorry, Roger, I couldn't resist.) Yeah, long night :-) However, technically SQL over SMTP is possible and would actually work. And if anyone is insane enough to try that then using Postfix and Postgres are a good combination. >> If you need lots of processes on the network to access data quickly then >> consider memcached. > > More seriously, in this category you might also consider Redis. > Redis allows your data to have some structure, The Python binding pylibmc does structure the data for you automagically. > plus it has the > ability to persist the data to disk. The moment you talk about persistence you then have significant overlap with databases. My personal favourite is MongoDB but there are loads of others such as Cassandra, HBase, Tokyo Cabinet etc. I like the ones that don't have a schema the best. > I've been playing around with connecting SQLite virtual tables to > a Redis server, and it is producing some interesting results. A while back I implemented a virtual table that talks to CouchDB. I suspect you'll have similar issues. You may find the documentation of interest as you'll likely encounter similar issues. http://apidoc.apsw.googlecode.com/hg/couchdb.html What programming language are you using to implement the virtual tables? Roger -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk1h6+wACgkQmOOfHg372QT/DACfQFOCo/Ku/kHXZGQ0eiXqWDcJ XQwAnRaBR8/uNgSKNBKXKiG5i/y7G1wm =1v5a -END PGP SIGNATURE- ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] SQLite server using execnet ?
On Sun, Feb 20, 2011 at 05:23:09PM -0800, Roger Binns scratched on the wall: > If you want a close match to the Python object model then use MongoDB. > > If you want to use SQL then use Postfix. I might suggest PostgreSQL instead. Postgres is likely to give you much better results than the Postfix SMTP email server. (Sorry, Roger, I couldn't resist.) > If you need lots of processes on the network to access data quickly then > consider memcached. More seriously, in this category you might also consider Redis. Redis allows your data to have some structure, plus it has the ability to persist the data to disk. I've been playing around with connecting SQLite virtual tables to a Redis server, and it is producing some interesting results. -j -- Jay A. Kreibich < J A Y @ K R E I B I.C H > "Intelligence is like underwear: it is important that you have it, but showing it to the wrong people has the tendency to make them feel uncomfortable." -- Angela Johnson ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] SQLite server using execnet ?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 02/20/2011 04:55 PM, Stef Mientki wrote: > still want to see if a simple client/server setup would solve my current > problems There is no such thing as "simple" client/server. You have to worry about issues like authentication and authorization. You have to deal with naming. You have to think about APIs being synchronous or asynchronous. You have to worry about state and if state is maintained across connections or dropped. You have to worry about new error codes that couldn't occur before. You have to deal with race conditions and latency. Sweeping all this under the rug will appear simple until you do real deployments and start painfully encountering and addressing the issues. That is why networked databases are not simple. If the data is not valuable then all that doesn't matter. > (and I realize that I'm a great optimist) Indeed :-) You should look closely at what it is you actually need. If you want a close match to the Python object model then use MongoDB. If you want to use SQL then use Postfix. If you want a stronger binding to SQLite and the ability to operate with and without a network then consider using SQLite virtual tables with the backend talking over the network or locally as needed. If you need lots of processes on the network to access data quickly then consider memcached. If you eventually intend to go for Amazon or Google cloud deployments then look at what they provide to run locally. If transactions and ACID matter then carefully research what meets your needs and deploying using as much redundancy and backup as appropriate. If you can't make your mind up, write a server that provides your data REST style and make the clients use HTTP. Roger -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk1hvnkACgkQmOOfHg372QSFZACfbWZXHwD3+q9xfmfIVAZr9ITO yHAAn1s3y6w6FV0pW0VPAL1cTfoscB96 =Id/K -END PGP SIGNATURE- ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
[sqlite] SQLite server using execnet ?
hello, knowing that SQllite is not a client/server database, still want to see if a simple client/server setup would solve my current problems for the moment (because I love the simplicity of SQLlite, and planned to go to a client / server database in the future) Now I wonder if anyone has considered to use Python execnet-module to realize a simple SLQlite client / server application. If I look at the documentation of execnet, (and I realize that I'm a great optimist) it would take between 20 and 50 lines of Python code. thanks very much for your opinions. cheers, Stef Mientki ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
