Re: [sqlite] How people test db failures
On 3 May 2013, at 4:03am, Igor Korotwrote: > How do people prove to their customers that no matter what As others have commented, this is too vague. Many disasters will destroy the hardware the data is stored on. They need to integrate what the software does for recovery with their own recovery precautions like offsite backup, live duplication, etc.. > the software > will either finish gracefully or will not break, crash or anything to that > matter? This was years ago, and not working with SQL but with a far earlier and more primitive DBMS. But I once did a demo for a customer where I had the database on an external 40Meg hard disk, and I let the manager in the meeting pull the external drive cable out while the program was updating a record, and later turn off the power to the computer while doing an end-of-month update. Both times, when restarted the software detected the problem, recovered the last usable dataaset, and continued. The customer was suitably impressed and continued with purchase negotiations. Technically, the demo did not demonstrate what the customer thought it did. There was no way to do that that wouldn't have taken up a 10 hour meeting and involved lots of technical discussion, which was not appropriate with this customer. Our software did handle most of these situations correctly, but there was no way to walk a non-technical customer negotiator through how it did it. They would have felt embarrassed because they didn't understand the explanation and that's not a good way to run a meeting. Simon. ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] How people test db failures
> > > My question was a little different. > > > I am developing desktop application which utilizes SQLite. At some > > > point in time my customer will want to know what happen when the DB > > > operation breaks. > > This depends on your definition of "breaks". > Let me give you an example. > I start the transaction, do couple of inserts and then I get an I/O error. > In this case I need to rollback tell the user about failure and go back to > the screen I started the transaction from. > Or the simplest one - the database is moved and inaccessible. If it > happens on the start, just tell the user and exit the program. If it happens > during the program execution, again tell the user that current operation > failed > and go back to the screen that originated the operation. Both of these should be fatal. As should things like running out of memory. You cannot "recover" from these sorts of errors and when you come upon them your application should detect that there is an error condition that cannot be recovered and stop. There is no way that you application can deal gracefully with these errors. If the application flow is well designed, then your data will be preserved (well, maybe not in the case of an I/O error, depending on what the underlying cause is). However, it is unsafe for the application to ignore the error and continue. ACID should prevent your data from being corrupted or inconsistent, if that is at all possible (depending on the error). However, there is nothing that *your application* can do when faced with such an error except to crash. Maybe the error is transient and will not appear when you restart the application. Maybe a sector went bad in the middle of file and you can do nothing except replace the disk and restore the database from backup. An ABEND is a perfectly valid response to an unexpected error condition and prevents things from being made *worse* after an error occurs. --- () ascii ribbon campaign against html e-mail /\ www.asciiribbon.org ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] How people test db failures
At 07:31 03/05/2013, you wrote: Problem is, how do I prove to the customer/user that it is working as expected no matter what? There are two important points in your question. The second (!) is "prove". The first is "as expected". Let me expand on why. For a system (irrespective of its nature or implementation technology) to be "working as expected", you first have to define exactly how it is expected to behave in every situation it may encounter. That's called functional specifications. Specs need to be complete and deterministic down to the lowest levels (of spec, not implementation!) to be useful later. If you want to "prove" something based on these specifications, you need to express them as a formal model by using one of the formal methods available. This model needs to be proven complete (no unspecified state) and non-contradictory under the formal methodology used. Once you have formal specifications defined as a formal model (we call this a "machine" in B) you can enter the process of refinement. To refine a formal model is to gradually introduce every aspect of implementation while using adequate proofing tools to demonstrate (and here I mean "mathematically") at every step that the refined specs are mathematically consistent with the initial formal specs. Actual large systems require many refinement steps to achieve a final implementable model (B0 in the B method). Once here and if the method chosen has the capability, you can actually produce executable code (typically secure ADA or C) for a given hardware target, along with a formal proof that the executable will behave "as expected" in all aspects. B is such a method. The excellent news is that you don't need ANY test step: just deliver the production to the client, re-play the proof before him, get payed and fly to some sunny place with white sand beaches and no phone or Internet for a well deserved rest. There exists a valid mathematical proof that the production will always work "correctly", that is conforms to the initial formal model, in every aspect. The less pleasant point is that a complete coverage by a top-notch formal method is really hard: you need experts in this method and ample time. The "simple" initial formal model is the first hard work because the human language --even used for specifications-- is incredibly ambiguous in practice and more than often uses implicit references to untold implementation details left vague (e.g. implicit "knowledge" about the behavior of the underlying OS for a software product). What you do when you use a formal method is that you merge the two branches of the V development process. In fact, you intermix tests (the right branch of the V) which translate into proof of correctness, with the classical development right branch. Many real-world systems requiring high dependability for mission-critical applications have shown that the time invested in refinements (where most of the time and efforts are used) grossly balance the time and efforts normally spent on incomplete or unconclusive tests and, after release, in fixing issues left over and discovered in the wild. Of course with a formal development you benefit of a program proven correct by construction. SQLite would be a very good candidate for a formal development if its design was not somehow a moving target. I'm not saying DRH and his team are amateur developpers, all I mean is that when significant new features are introduced in the user specifications, the whole formal process has to be restarted. Many of the changes introduced in SQLite over the last few years have deep implications on its behavior. Depending on how the changes introduced fit the formal model or can't cope with its structure, a new version may require much work, up to complete rework. Formal methods are best suited for fixed specifications applications. In short, "yes you can" reach part of your goal, but if you take your own words "prove" and "working as expected no matter what" at face value, then you definitely heavyweighted formal methods to prove your application, the OS and the hardware all this runs on, possibly using redundant (majority) architecture or a secure coded processor. Needless to say, this is overly expensive for most applications. JcD ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] How people test db failures
You'll have to explain to the customer that "no matter what" carries too much weight. Are you going to protect against hardware failures? Fire? Floods? Under what level would an acceptable failure be, or an unacceptable failure? What specifically is the user looking to break? If the user gets smart enough and starts modifying your schemas, all bets are off, so, does that mean you have to start encrypting the database to keep their fingers out of the cookie jar? If files are being moved around and the database is pushed off somewhere else, do you keep a secret backup of the database somewhere else on the computer? Upload it to the cloud behind the users back? Do you somehow write the database directly onto the EXE and just make the user guess where the file is? You can protect your software as best as you can acknowledging that users are inventive and will break your software, so the best you can do is defend against at least data entry issues, and perhaps, if you want to go the extra mile, DO save out to the cloud somewhere else for backup purposes. But there is a limit to what you can do without building a virtual Fort Knox around a simple phone book database application, and even then you can't save the data in case of a meteor strike. So sanitize your data on any database transaction, ensure that anything being put in and pulled out matches that of the constraint that the application expects (IE: User enters a letter instead of a number. Since SQLite will accept that, when reading the data out, your app may freak out). It is absolutely impossible to protect against EVERY contingency (Which is what "No Matter What" means), so definitions are going to have to be laid out about what the expectations are to be for failure defenses. ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] How people test db failures
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 02/05/13 22:31, Igor Korot wrote: > Or the simplest one - the database is moved ... That is sufficient to break the database. A SQLite database is not just the database file, it is also the journal and WAL. If you don't move all as a single unit then you break the database. Often you'll get lucky ... The only safe way to move a database is to use the backup api to make a new copy with the new name. If you do it outside of SQLite then great care has to be taken, and errors won't necessarily be immediately apparent or even show up under pragma check. Roger -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.12 (GNU/Linux) iEYEARECAAYFAlGDT0QACgkQmOOfHg372QSkKwCgysgQWSsCWgSN1zMS0p9uOOWb TL8An1MuVaYjysden5anmCnKk2G3+b5w =8HDa -END PGP SIGNATURE- ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] How people test db failures
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 02/05/13 20:03, Igor Korot wrote: > How do people prove to their customers that no matter what the > software will either finish gracefully or will not break, crash or > anything to that matter? That is easy - use SQLite. Put your work inside transactions. You can use savepoints to nest transactions. A transaction either fully completes, or has no effect. Even if your app is forcefully killed this will still hold. (The database will be cleaned up the next time it is opened removing information about the transaction that was under way.) On the other hand your code interacting with SQLite could have bugs. At a simple level you could ignore the error code returned by functions and assume they succeeded. The only way to test that is to add in hooks to cause various pieces to fail as needed, and use coverage analysis to help ensure you have tested a reasonable amount of code. An example of how thorough you can be is the SQLite testing: http://www.sqlite.org/testing.html All that aside, there is still no protection from a hostile environment. An overclocked/overheated cpu can make arithmetic errors. Cosmic rays can change bits in memory. Filesystems without checksums can have bit flips on the storage or in transmission. Roger -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.12 (GNU/Linux) iEYEARECAAYFAlGDTf0ACgkQmOOfHg372QSuhgCfZaiQPow1ioBbZnez569/oVPt 3s0AoLdwcTPuEB8apk5Dv1VNpLS2bMXC =VYvk -END PGP SIGNATURE- ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] How people test db failures
Keith, On Thu, May 2, 2013 at 9:44 PM, Keith Medcalfwrote: > > > My question was a little different. > > > I am developing desktop application which utilizes SQLite. At some point > > in time my customer will want to know what happen when the DB operation > > breaks. > > This depends on your definition of "breaks". > Let me give you an example. I start the transaction, do couple of inserts and then I get an I/O error. In this case I need to rollback tell the user about failure and go back to the screen I started the transaction from. Or the simplest one - the database is moved and inaccessible. If it happens on the start, just tell the user and exit the program. If it happens during the program execution, again tell the user that current operation failed and go back to the screen that originated the operation. > > > How do people prove to their customers that no matter what the software > > will either finish gracefully or will not break, crash or anything to > that > > matter? > > Oftentimes, it is preferable to "break" or "crash" rather than do > something in error -- or worse yet fail to detect the error and proceed > merrily but incorrectly. > > > So far the guy tested the program and he made sure that working with DB > > was OK if everything succeeds. But now I want to let him know that he can > > setup the environment and try check what happen if something fails. > > The answer to this depends on what failed, and if you can anticipate such > failure being a valid outcome of what you are doing. > > > How do you achieve something like this? > > If everything succeeds, you are fine. If it fails in an anticipated way, > then you deal with it in accordance with your design. If the failure is > unanticipated, you vomit noisily presenting sufficient information to > diagnose and repair the problem. If appropriate, you add specific handling > for the error that you didn't anticipate happening once you know the > circumstance under which it occurs and can validate that the specific > circumstance has occurred so that your fault handler is dealing with the > fault correctly. > > Of course, if the "error condition" is anticipated, then it is not an > error, is it? > > Having something which guarantees consistency in the face of unexpected > failure, in other words ACID, makes the issue not really much of an issue > at all since it should be impossible for sudden "de-rugging" (ie, crash or > killing the process) of the application at any time to cause corruption and > inconsistency (if it can, then you have designed your application badly). > > You can handle it in the same way that Apple does: crash with a little > comic bomb on the screen, then display a picture of a little application > with a thermometer in its mouth, then after a time simple restart yourself > over cleanly. > Well I'm handling it. Problem is, how do I prove to the customer/user that it is working as expected no matter what? Thank you. > > --- > () ascii ribbon campaign against html e-mail > /\ www.asciiribbon.org > > > > > ___ > sqlite-users mailing list > sqlite-users@sqlite.org > http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users > ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] How people test db failures
> My question was a little different. > I am developing desktop application which utilizes SQLite. At some point > in time my customer will want to know what happen when the DB operation > breaks. This depends on your definition of "breaks". > How do people prove to their customers that no matter what the software > will either finish gracefully or will not break, crash or anything to that > matter? Oftentimes, it is preferable to "break" or "crash" rather than do something in error -- or worse yet fail to detect the error and proceed merrily but incorrectly. > So far the guy tested the program and he made sure that working with DB > was OK if everything succeeds. But now I want to let him know that he can > setup the environment and try check what happen if something fails. The answer to this depends on what failed, and if you can anticipate such failure being a valid outcome of what you are doing. > How do you achieve something like this? If everything succeeds, you are fine. If it fails in an anticipated way, then you deal with it in accordance with your design. If the failure is unanticipated, you vomit noisily presenting sufficient information to diagnose and repair the problem. If appropriate, you add specific handling for the error that you didn't anticipate happening once you know the circumstance under which it occurs and can validate that the specific circumstance has occurred so that your fault handler is dealing with the fault correctly. Of course, if the "error condition" is anticipated, then it is not an error, is it? Having something which guarantees consistency in the face of unexpected failure, in other words ACID, makes the issue not really much of an issue at all since it should be impossible for sudden "de-rugging" (ie, crash or killing the process) of the application at any time to cause corruption and inconsistency (if it can, then you have designed your application badly). You can handle it in the same way that Apple does: crash with a little comic bomb on the screen, then display a picture of a little application with a thermometer in its mouth, then after a time simple restart yourself over cleanly. --- () ascii ribbon campaign against html e-mail /\ www.asciiribbon.org ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] How people test db failures
Hi, guys, On Thu, May 2, 2013 at 6:52 PM, Keith Medcalfwrote: > > >> My pet peeve is the lousy error message like Microsoft used to produce > > >> "dll not found". What DLL? What error? E.g. File not found or > > permission > > >> problem? > > > > > > My favorite is from the service manager: > > > > > > "The Service could not be started because the file could not be found". > > > > > > Wouldn't it be nice to tell me (a) what service or (b) what file and > (c) > > where you were looking. > > > > In a previous version of Mac OS X sometimes deleting a file worked > > properly but triggered an error message anyway. The error message said > > something like > > > > The file "" could not be deleted because the file no longer exists. > > > > . The filename inside the quotes was always blank because it couldn't > > find out the name because the file no longer existed. It's really hard > > explaining to users why it won't tell them what the problem is or what > > file caused it. > > Of course the file name is known -- how else would it know that it no > longer exists? > Failing to use the correct variable in the error message is a defect in > the processing logic. > > BTW, the bug you point out is quite common and exists in many operating > systems (and applications). > This is all good, but... My question was a little different. I am developing desktop application which utilizes SQLite. At some point in time my customer will want to know what happen when the DB operation breaks. How do people prove to their customers that no matter what the software will either finish gracefully or will not break, crash or anything to that matter? So far the guy tested the program and he made sure that working with DB was OK if everything succeeds. But now I want to let him know that he can setup the environment and try check what happen if something fails. How do you achieve something like this? Thank you. > --- > () ascii ribbon campaign against html e-mail > /\ www.asciiribbon.org > > > > > > ___ > sqlite-users mailing list > sqlite-users@sqlite.org > http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users > ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] How people test db failures
> >> My pet peeve is the lousy error message like Microsoft used to produce > >> "dll not found". What DLL? What error? E.g. File not found or > permission > >> problem? > > > > My favorite is from the service manager: > > > > "The Service could not be started because the file could not be found". > > > > Wouldn't it be nice to tell me (a) what service or (b) what file and (c) > where you were looking. > > In a previous version of Mac OS X sometimes deleting a file worked > properly but triggered an error message anyway. The error message said > something like > > The file "" could not be deleted because the file no longer exists. > > . The filename inside the quotes was always blank because it couldn't > find out the name because the file no longer existed. It's really hard > explaining to users why it won't tell them what the problem is or what > file caused it. Of course the file name is known -- how else would it know that it no longer exists? Failing to use the correct variable in the error message is a defect in the processing logic. BTW, the bug you point out is quite common and exists in many operating systems (and applications). --- () ascii ribbon campaign against html e-mail /\ www.asciiribbon.org ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] How people test db failures
On 3 May 2013, at 1:49am, Keith Medcalfwrote: >> My pet peeve is the lousy error message like Microsoft used to produce >> "dll not found". What DLL? What error? E.g. File not found or permission >> problem? > > My favorite is from the service manager: > > "The Service could not be started because the file could not be found". > > Wouldn't it be nice to tell me (a) what service or (b) what file and (c) > where you were looking. In a previous version of Mac OS X sometimes deleting a file worked properly but triggered an error message anyway. The error message said something like The file "" could not be deleted because the file no longer exists. . The filename inside the quotes was always blank because it couldn't find out the name because the file no longer existed. It's really hard explaining to users why it won't tell them what the problem is or what file caused it. Simon. ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] How people test db failures
> My pet peeve is the lousy error message like Microsoft used to produce > "dll not found". What DLL? What error? E.g. File not found or permission > problem? My favorite is from the service manager: "The Service could not be started because the file could not be found". Wouldn't it be nice to tell me (a) what service or (b) what file and (c) where you were looking. All three would be best. And this isn't a "used too". This is the current state-of-the-art in Microsoft error messages. Completely and utterly useless. ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] How people test db failures
There are several ways people handle errors. #1 Path of least resistanceyou assume all is well and ignore thembad idea but too common #2 Catch as catch can...you put in error handling as you experience themalso a bad idea as it's too similar to #1 #3 Catch with careful thought about what could happencan be difficult but this is what experienced people do...can still miss things. #4 Catch all...check all return codes and print out somethingthen do #3 for known error and graceful/proper handling...this is how you learn. #5 Force all possible errors to occur...there's a lot. I don't know anybody that does that. A combo of #3 and #4 is best IMHO. If you want to be really conservative you could write a wrapper around every sqlite function to randomly return one of its possible error codes. That's would take a lot of time though and I've never heard of anybody doing that. My pet peeve is the lousy error message like Microsoft used to produce "dll not found". What DLL? What error? E.g. File not found or permission problem? Or the very common "cannot open file". What file, why not? Error messages need to be explicit. SQlite is pretty good about it. Developers not so much. -Original Message- From: sqlite-users-boun...@sqlite.org [mailto:sqlite-users-boun...@sqlite.org] On Behalf Of Igor Korot Sent: Wednesday, May 01, 2013 2:00 PM To: General Discussion of SQLite Database Subject: [sqlite] How people test db failures Hi, ALL, I wrote a code that executes fine. There is no memory leaks and no issues. However, trying to check whether my error handling is written correctly, I set breakpoint before accessing db and when I hit it I manually changes the value returned. This is not how it should be done and I feel that the person I'm working with will need to check it at some point. So, how people check whether db failure result in graceful program termination/proper flow? I'm working on the desktop application under Windows and am giving the release version of the program compiled under MSVC 2010. Thank you. ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] How people test db failures
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 01/05/13 12:00, Igor Korot wrote: > So, how people check whether db failure result in graceful program > termination/proper flow? I use a macro that takes this form: #define TESTPOINT(name, normal, failure) I use it like this: TESTPOINT(StepNoMem, rc=sqlite3_step(stmt), rc=SQLITE_NOMEM); The macro expands to something like this with debug builds: do { if(_point_should_fail(#name) { failure; } else { normal; } } while(0) The _point_should_fail function returns true once and then false from then on. My test suite would then look like: def test2c(): set_point_should_fail("StepNoMem") ... code that should handle the failure correctly ... Of course reality is a little messier than this. Here is some code that sets a failure point: https://code.google.com/p/apsw/source/browse/src/apsw.c#217 Here is the corresponding test: https://code.google.com/p/apsw/source/browse/tests.py#7562 (I use 1/0 to ensure that there will always be an exception, and if it turns out to be ZeroDivision then I know the previous line didn't cause one and should have.) Roger -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.12 (GNU/Linux) iEYEARECAAYFAlGBp+YACgkQmOOfHg372QQsrACgiDu/z2nn+NZdf/Q/Ep4JBrFu U24An3J26qO54dT89vSVKo/Js60O7pwc =mc1D -END PGP SIGNATURE- ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
[sqlite] How people test db failures
Hi, ALL, I wrote a code that executes fine. There is no memory leaks and no issues. However, trying to check whether my error handling is written correctly, I set breakpoint before accessing db and when I hit it I manually changes the value returned. This is not how it should be done and I feel that the person I'm working with will need to check it at some point. So, how people check whether db failure result in graceful program termination/proper flow? I'm working on the desktop application under Windows and am giving the release version of the program compiled under MSVC 2010. Thank you. ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users