RE: [sqlite] Encryption?
On Wed, 2007-12-19 at 16:10 -0800, James Dennett wrote: > > -Original Message- > > From: Jason Tudor [mailto:[EMAIL PROTECTED] > > Sent: Wednesday, December 19, 2007 4:06 PM > > To: sqlite-users@sqlite.org > > Subject: [sqlite] Encryption? > > > > Is there any encryption functionality built into SQLite? > > Not in the public domain version. >From the ministry of silly thoughts, maybe dm_crypt for Linux only ;-) http://www.saout.de/misc/dm-crypt/ Can be used to encrypt/decrypt whole file systems on the fly! JS. - To unsubscribe, send email to [EMAIL PROTECTED] -
Re: [sqlite] Improving performance of SQLite. Anyone heard ofDevice SQL?
On Mon, 2007-12-17 at 19:22 -0500, John Elrick wrote: > John Elrick wrote: > > SNIP > > When you can hire a forklift operator to program (well, that is), > > > To avoid a misunderstanding...I mean - right off the forklift. I'm sure > anyone with the proper motivation can learn to program, but it took me > 25 years to realize how little I really knew. I didn't write the original about forklift operators. It was Fred Williams, AFAIK. I am in agreement with you, and I have nothing against forklift operators either. IMHO, this has gone completely off topic, and I shall hence forth cease to contribute to this, and related threads. Regards, James. - To unsubscribe, send email to [EMAIL PROTECTED] -
RE: [sqlite] Improving performance of SQLite. Anyone heard of Devic eSQL?
On Mon, 2007-12-17 at 15:30 -0600, Fred Williams wrote: > A hundred or so Visual Basic programmers are cheaper to replace and > "maintain" than one good Delphi/C++ programmer. ;-) That is the reason > management likes "Visual ." Been there, learned that. Hire the > staff from the largest pool, not the most effective. Besides it's damn > hard to be a prima donna, when your replacement is ready to jump off > that forklift and learn a cushy job. Ouch. Lucky Visual is not a cross platform language. Show the Visual programmers a bit of Tcl/Tk and watch them wilt! It looks like Xilinx wrote their entire ISE GUI in Tcl/Tk, and the backend apps are cross compiled. I can run it on Linux as well as NoDose, and the GUI is identical. Try that in Visual poop. You might have cheap programmers today, but tomorrow they will be less useful. Look out for Tux! At every planning meeting I push open source, and cross platform solutions, because I know today the majority is still under Bill's spell, but the magic in Vista is fading... Ciao. James. - To unsubscribe, send email to [EMAIL PROTECTED] -
Re: [sqlite] DeviceSQL
steveweick wrote: Do you need to read the code to verify reliability as your next few sentences seems to imply? For that to be true, the reader would have to be able to spot bugs through inspection. While that is certainly one way to spot bugs, I seriously doubt that any shop would rely on code inspection, when millions of dollars of potential recall costs are on the line. I think many would agree that code inspections do find (serious) bugs, that may not show up from testing. I'm sure your company conducts code inspection meetings as a part of all code development. We (the company I work for) certainly do. I know I've seen change logs that read something like "Fixed possible buffer overflow in foo..." for open source projects here and there as well. In fact the SQLite marketing does not rely on code inspection as its argument for why the code is reliable. Check it out. That would be bad if they did, I agree. But all the testing in the world won't uncover all the bugs either, in a complex piece of code. See http://www.eecs.berkeley.edu/Pubs/TechRpts/2006/EECS-2006-1.pdf . "The Ptolemy II system itself began to be widely used, and every use of the system exercised this code. No problems were observed until the code deadlocked on April 26, 2004, four years later." And that was after code inspections, regression tests and belt and braces programming techniques! All of that said, I do admire the elegance of the SQLite code. It makes entertaining reading. Unfortunately elegance does not translate into performance or reliability. Not necessarily, but it often does, and can make for better maintainability too. I've not trawled through to SQLite code myself, so couldn't comment. But it does have quite a few big name users, and an active and helpful user forum, which gives me good vibes at least. Cheerio, James. - To unsubscribe, send email to [EMAIL PROTECTED] -
Re: [sqlite] DeviceSQL
steveweick wrote: Richard has it right this time. Today DeviceSQL uses no SQLite code. One of the things we might consider is bolting the SQLite parser/front end to our table engine, in theory to get the both worlds. Just an idea at the moment. Such an interesting discussion to be following. I must say though, it seems DeviceSQL has opened the door to speculation due to unsubstantiated claims in advertising, as far as I see it. IMHO, so long as there is no independent, unbiased, side by side test results presented somewhere by some reliable source, there will always be some room for "ifs" and "buts" by both sides. Maybe DeviceSQL should go open source, so the public can judge for them selves the qualities of the two products. There would still be money to be made from paid support. Who knows, both parties could benefit, and customers too. At least there'd be a clearer view of the pros and cons. There is something to be said for a product being open source, that is the code is scrutinized by the world. Closed shop code can possibly still be very good, but without seeing it, how would we know? Reminds me of a story about a cat: dead or alive, we won't know until we open the box it's in, and prior to that, is it only half dead? One only has to look at the MSDN code examples to see the ugliness of closed source code development...(sorry Bill) JS. - To unsubscribe, send email to [EMAIL PROTECTED] -
Re: [sqlite] Seg fault after calling sqlite3_exec while running under cygwin
On Mon, 2007-12-10 at 20:37 -0700, John Williams wrote: > Since I'm not really sure where the problem is I've attached a zip of my c > files. AptAssist.c is my main file and contains the full program. > Temp.cis a simple pull out of the problem function. I didn't get an attachment. Did anyone? JS. - To unsubscribe, send email to [EMAIL PROTECTED] -
Re: [sqlite] Performance tuning, and other (silly?) SQLitequestions.
> Ken <[EMAIL PROTECTED]> wrote: > > Can you do a test load to a memory database. What is the performance > (row/sec) of your data using an in memory database? I tried a RAMDisk on Windows. I haven't tried at home on Linux. I'm not sure if I tried the C API to RAMDisk database combination, only the Tcl API to RAMDisk database combination. Maybe Tcl was the bottleneck here. > Is that a burst rate or a sustained data rate? You need to achieve > about 70k rows per second to be able to maintain 66k rps. Well, at the moment it is mostly only bursts of data, but I have no control over the possibility that it will become even faster, or continuous, so I want need to push the envelope. > Maximum TPS is 60 on a 7200 rpm drive. A 15000 RPM drive should get > about 120, which is double the TPS. So maybe a faster drive would be > helpful. Quite possible. At the moment I'm not sure what HD is installed on the vehicle where the data is produced, but it can always be improved upon by throwing $$ at it ;-) Cheers, James. - To unsubscribe, send email to [EMAIL PROTECTED] -
Re: [sqlite] Re: Performance tuning, and other (silly?) SQLitequestions.
> James Steward <[EMAIL PROTECTED]> wrote: > > Dennis Cote <[EMAIL PROTECTED]> wrote: > > > You have said you tried both the TCL and C APIs, but you didn't say if > > you were using prepared insert statements in the C API. If not, that > > will save the overhead of parsing and code generation for each insert > > statement. The prepare, bind step, reset mechanism will give better > > perfomance. > > Is there an example you know of somewhere? I've been reading... http://www.sqlite.org/capi3ref.html#sqlite3_prepare_v2 Seems fairly straight forward. I'll try tomorrow (perhaps). Regards, James. - To unsubscribe, send email to [EMAIL PROTECTED] -
Re: [sqlite] Re: Performance tuning, and other (silly?) SQLitequestions.
Dennis Cote <[EMAIL PROTECTED]> wrote: Hi Dennis, > You haven't said what your system is, but to record your data in real > time you will need to insert about 70K records per second. That is high, > > but not impossible for SQLite, so I wouldn't give up yet. I have had > SQLite doing 60K inserts per second on a standard 7200 RPM hard drive. That sounds good. > Do you actually need to insert records at this rate continuously, or > just for a short 30 second burst? If it is bursty, how much time do you > have between bursts? Usually there are bursts of data, and we can buffer it for a while, but at times, and to allow for system growth, I was hoping to match the system maximum continuous data rate. > You have said you tried both the TCL and C APIs, but you didn't say if > you were using prepared insert statements in the C API. If not, that > will save the overhead of parsing and code generation for each insert > statement. The prepare, bind step, reset mechanism will give better > perfomance. Is there an example you know of somewhere? > If you can do post processing on the data, then you could look at > storing the data into separate databases on separate high speed (i.e.15K > RPM) hard drives. This should give you the raw I/O speed you need to get > all the info to disk. The you can run a second program that merges the > separate databases into a single one. I will need to ponder this one. Thanks for the idea. > Do you need the ACID properties of SQLite, or can you simple repeat the > collection process if you have an OS crash or power fail while > collecting data? If not, then you can turn off the synchronous writing > with Pragma Synchronous=Off which should increase your write rate again. This is also a possibility. I shall investigate. Cheers, James. - To unsubscribe, send email to [EMAIL PROTECTED] -
Re: [sqlite] Re: Performance tuning, and other (silly?) SQLitequestions.
> Michael Ruck <[EMAIL PROTECTED]> wrote: > > I know that a natural join exists, but it is not automatic as > it seems to be in MySQL. Thanks , and thanks to all who replied to my questions. I've been testing SQLite's speed, for inserting the type of data I gather from the field. I've tried encapsulating multiple inserts between begin and commit statements, fiddling the pragmas and with both Tcl and C interfaces, even using a RAM disk to store the database file. I can receive up to 2,000,000 records in about 30 seconds from the field, and I can't seem to jam them into an SQLite database any faster than about 100 seconds at best, on my system. So unless anyone can suggest some magic I have not thought of, I will have to abandon the SQL database dream, and write a less functional, data and application specific, database of my own. Cheers, James. - To unsubscribe, send email to [EMAIL PROTECTED] -
Re: [sqlite] Where is sqlite3.h?
On Tue, 2007-11-20 at 14:23 +1100, James Steward wrote: > Any hints? I answer my own question. I found it in the preprocessed source zip file. I hope this will do... Cheerio. JS. - To unsubscribe, send email to [EMAIL PROTECTED] -
[sqlite] Where is sqlite3.h?
Hi, I wanted to build a C app, copied the example C code from the Documentation on the web site, and downloaded the prebuilt binaries. There's no header in with the dll and def (for crummy windows). I tried building the source with MSYS/MinGW, but it didn't work out of the box. Any hints? Regards, James. - To unsubscribe, send email to [EMAIL PROTECTED] -
Re: [sqlite] Re: Performance tuning, and other (silly?) SQLite questions.
On Mon, 2007-11-19 at 18:14 -0500, Igor Tandetnik wrote:
> James Steward
> wrote:
> > db eval {CREATE TABLE position(\
> > position_id INTEGER PRIMARY_KEY, \
> > odo INTEGER, \
> > time CURRENT_TIMESTAMP);}
>
> You probably meant
>
> -- note no underscore between PRIMARY and KEY
> position_id INTEGER PRIMARY KEY
Oops! Fixed now. Thanks.
> -- note DEFAULT keyword
> time DEFAULT CURRENT_TIMESTAMP
I discovered that just before I saw your reply ;-)
> > #A more complicated query...runs quite slowly. How can this be sped
> > up?
> > db eval {SELECT position.odo, data.x, data.y from position, data WHERE
> > position.odo BETWEEN 1 AND 10020;}
>
> First, you want an index on position.odo. Second, you don't specify any
> relation between position and data tables, so you generate a full
> cross-product. You want
>
> SELECT position.odo, data.x, data.y
> FROM position JOIN data ON (position.position_id = data.position_id)
> WHERE position.odo BETWEEN 1 AND 10020;
Ah. I have been reading a PHP/MySQL book, that I thought said a MySQL
server would see the common column names and automagically join the 2.
Either I misremember what the book said (it's not with me here), or this
is a feature of MySQL, not present in SQLite. Anyway, what you suggest
works just fine.
> And for that to work efficiently, you want another index on
> data.position_id
I'm guessing that is with;
CREATE INDEX odo_index ON data (position_id ASC);
If the data is streaming in, and insertions are being made on the fly,
will an index need to be regenerated periodically, i.e. REINDEX?
Cheers,
James.
-
To unsubscribe, send email to [EMAIL PROTECTED]
-
