Re: [sqlite] Propose minor incompatible API change

[Andy Gibbs]

On Tuesday, January 11, 2011 1:54 AM, Richard Hipp wrote:

> So the question to you, gentle reader, is should we make this change, and
> break backwards compatibility, albeit in a very obscure way, or should we 
> be
> hard-nosed and force hundreds or perhaps thousands of smartphone 
> application
> developers fix their programs?

I see that this issue is now resolved 
(http://www.sqlite.org/src/info/053ce76deb).  Thank you - I think the way 
you have done it is probably the best in the circumstances and fits the 
needs of most people.

Regards
Andy


___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Propose minor incompatible API change

[Philip Graham Willoughby]

On 12 Jan 2011, at 12:49, Andy Gibbs wrote:

> On Wednesday, January 12, 2011 10:08 AM, Philip Graham Willoughby wrote:
> 
>> unfortunately 3.7.2 shipped in Ubuntu Maverick and
>> 3.6.23.1 shipped in a maintenance update for Fedora
>> Core 14. So lots of people already have both behaviours
>> in the wild.
> 
> Actually, the first alteration happened *after* 3.6.23.1 which released on 
> 30th March; the alteration happened 17th April --  
> http://www.sqlite.org/src/info/3e646e3f4c -- so maybe that limits a little 
> the number of people affected?

Well, it means Fedora doesn't have the change, and anyone with 3.7.0 or later 
does.

The list I gave was based on which operating systems were easy for me to find 
out about rather than any attempt to be exhaustive.

Incidentally sqlite3_libversion() on iOS 4.2.1 (as used on iPhone/iPad) reports 
3.6.23.2, which is not listed on http://www.sqlite.org/changes.html but was 
presumably cut at some point after 3.6.23.1: your guess is as good as mine as 
to whether it contains the April 17 change or not.

The docs say versions 'after version 3.6.23.1' have the relevant behavioural 
change, so perhaps that means a lot of smartphones already have it.

Best Regards,

Phil Willoughby
-- 
Managing Director, StrawberryCat Limited

StrawberryCat Limited is registered in England and Wales with Company No. 
7234809.

The registered office address of StrawberryCat Limited is:

107 Morgan Le Fay Drive
Eastleigh
SO53 4JH

___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Propose minor incompatible API change

[Andy Gibbs]

On Wednesday, January 12, 2011 10:08 AM, Philip Graham Willoughby wrote:

> unfortunately 3.7.2 shipped in Ubuntu Maverick and
> 3.6.23.1 shipped in a maintenance update for Fedora
> Core 14. So lots of people already have both behaviours
> in the wild.

Actually, the first alteration happened *after* 3.6.23.1 which released on 
30th March; the alteration happened 17th April --  
http://www.sqlite.org/src/info/3e646e3f4c -- so maybe that limits a little 
the number of people affected?

Andy

___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Propose minor incompatible API change

[Philip Graham Willoughby]

Please don't top-post and include everything that went before.

On 11 Jan 2011, at 17:10, Scott A Mintz wrote:

> There's the issue of "this is what I meant" vs. "this is what I did." When 
> you have a couple hundred customer's, changing the code is painful but 
> doable.  When you have a couple million customer's, then what is "out 
> there" is the "true" API and must be kept around.

If that were generally true no-one would ever change (or even add) anything in 
any software library product.

The problem here is that there are two mutually incompatible versions out there 
already, pre-3.6.23.1 which generates SQLITE_MISUSE in many cases and 
post-3.6.23.1 which generates SQLITE_MISUSE practically never. Introducing a 
third incompatible version (as proposed to start this thread) which generates 
SQLITE_MISUSE with a frequency between the two is unlikely to make things 
better; if anything it will make things worse.

If we knew that no-one had shipped dynamic libraries for a version >=3.6.23.1 
and http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Propose minor incompatible API change

[Scott A Mintz]

There's the issue of "this is what I meant" vs. "this is what I did." When 
you have a couple hundred customer's, changing the code is painful but 
doable.  When you have a couple million customer's, then what is "out 
there" is the "true" API and must be kept around.  Microsoft has bent over 
backwards to keep old bugs around because of major applications taking 
advantage of undocumented behavior. 

When MS made the decision to break with backwards compatibility in favor 
of security and forward progress they introduced Vista and the world hated 
it.

-Scott

sqlite-users-boun...@sqlite.org wrote on 01/11/2011 07:57:13 AM:

> 
> On 11 Jan 2011, at 12:15, Richard Hipp wrote:
> 
> > That new OS release includes the latest shared library for
> > SQLite.
> 
> You didn't put it there, and the consequences of putting it there 
> are not your responsibility. Nor are the consequences of someone 
> else's app breaking because they didn't read the docs and coded to 
> observed behaviour instead of specified behaviour.
> 
> Why does the vendor of this OS release not ship multiple versions of
> SQLite so that apps coded to bind to 3.7.4 get 3.7.4 and apps bound 
> to 3.6.23.1 get 3.6.23.1 and so forth for all previous versions they
> have shipped? This is pretty basic dynamic linking functionality 
> that exists on all modern platforms.
> 
> > Sure, the problem really is that the apps were incorrectly
> > coded.  But does your mom really care about that?  They worked before.
> 
> This situation, or analogous ones, crop up any time you're supplying
> a product with a documented API and design when you have shipped a 
> version which did not do exactly what that design specifies and 
> later wish to correct it.
> 
> The two extremes of responses are:
> 1) We noticed a bug in that we weren't enforcing the documented API.
> We have now fixed it. If your app was relying on the previous faulty
> behaviour you will need to rectify it. Remember that the API as 
> documented is what you should code to and not the current behaviour 
> which may (due to human error) deviate from the spec at any given 
> point in time and therefore may be subject to revision to correct 
> it. If you observe such a deviation please report it to us so that 
> it can be fixed.
> 2) We will never change the external behaviour of our product.
> 
> Path (2) realistically means you never change anything, never add a 
> new feature, never fix any bug, in case someone is reliant on it.
> 
> Path (1) will annoy people from time-to-time but it leaves people 
> knowing where they stand.
> 
> As a developer-customer I prefer (1). I don't want to have to test 
> that the code does what it is documented to do all the time, I want 
> to read the docs and code to that, and to know that if I find a 
> discrepancy between the documented and observed behaviour that I can
> report a bug and you will (probably, if it will affecting more than 
> just my code or I pay you) fix it, or accept a patch to fix it.
> 
> Best Regards,
> 
> Phil Willoughby
> -- 
> Managing Director, StrawberryCat Limited
> 
> StrawberryCat Limited is registered in England and Wales with 
> Company No. 7234809.
> 
> The registered office address of StrawberryCat Limited is:
> 
> 107 Morgan Le Fay Drive
> Eastleigh
> SO53 4JH
> 
> ___
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Propose minor incompatible API change

[Jean-Christophe Deschamps]

>An end user (think: your mom) wants to upgrade her smartphone to the 
>latest
>OS release.  That new OS release includes the latest shared library for
>SQLite.  But in so doing, some percentage of the apps she has downloaded
>cease to work.  Sure, the problem really is that the apps were incorrectly
>coded.  But does your mom really care about that?  They worked before.  Do
>we really want thousand, perhaps millions, of moms screaming that SQLite
>broke their phone when they upgraded?

Yes if the choice is the only one left, then I for one want zillions of 
users complaining worldwide!  Software liability has to find its way 
someday.  It has become too easy for major software vendors to release 
crap without care and consider that the next release will fix the most 
crying bugs/misfeatures.

Now if it has been that common and easy to download and install a new 
whole OS, then how difficult is it to scan the smartphone at the same 
time for broken applications versions and push in fixed ones?  Again, 
all those companies have plenty of resource for planing such a move and 
the incentive for the phone maker is to maintain a good reputation vs 
competition, for the OS maker to show that his OS can be kept up to 
date and still working fine and for the app supplier that he is careful 
and responsive to his customers.

Taking up the responsability for this bug and changing SQLite for 
coping with incompetence will only open the avenue that every time a 
significant widespread application will prove buggy, then the author 
will be tempted to ask you to put in a specific workaround.

I remind a post by Roger Binns (I believe) regarding the history of 
SMB/Samba and the awfull number of such "workarounds" that have cropped 
into it (to overcome bugs in protocols, in implementations and possibly 
in common big name applications) to such an extent that the whole stuff 
has become unmanageable and unreliable, which seems to be one of the 
main reasons why for instance SQLite can't be used over a network reliably.

Again, and since installing a new OS version into smartphones has been 
that easy, why don't OS makers force download of a revised OS version 
including a specific SQLite version suited to the park?

Even if I agree and understand that the proposed change has minor or 
zero implications to most non-smartphone users, I still advocate 
against making it part of the main core, where it doesn't belong, just 
for the reasons invoked.

I sincerely doubt even Cisco would succeed in forcing a change in the 
IPv6 protocols to work around a bug in some of their routers.

Please read me well: I really don't care which corner case behavior is 
the best  in the future.  I trust SQlite dev team to make the right 
choice(s) just like they have done already so many times.  I also don't 
care as the library I'm maintaining (AutoIt UDF) works fine with both 
old and new versions and so do my own applications.  What makes me 
answer to your poll is that the principle of changing a core component 
due to applicative bugs opens the route to endless negociations about 
which applicative bug should be worked around at SQLite level, which is 
completely backwards.

___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Propose minor incompatible API change

[Adam DeVita]

I think that a bunch of good points have been made, especially as to why you
should hold your ground.

(I don't have sympathy for poor code that doesn't follow the documentation,
especially when there is a large, competent, and helpful community group
that usually comes to people's aid in less than 1 day.  Since when is my
app's bug your core problem?)

If the smartphone providers are willing to pay "enough" to fund this project
to ensure it keeps going, then I guess the idea of  "He who pays the piper
calls the tune."  applies.   Having lots of apps that work well are a way
these companies compete with each other.  I don't like the idea of a
technical decision being determined in the spirit of a bribe, but this
doesn't affect well written code. so I don't much care which side this
falls on.

The idea of a examples library for the docs is good, probably should have
its own thread to discuss how to implement.

best wishes,
Adam

On Tue, Jan 11, 2011 at 9:09 AM, Philip Graham Willoughby <
phil.willoug...@strawberrycat.com> wrote:

> On 11 Jan 2011, at 13:36, Andy Gibbs wrote:
>
> > On Tuesday, January 11, 2011 1:35 PM, Jean-Denis Muys wrote:
> >
> >> Don't encumber SQLite with workarounds and special cases
> >> to cater to bugs in client software.
> >
> > Isn't an accurate synopsis of the problem this: that Sqlite has *already*
> > implemented a workaround in 3.7.0, and that this workaround has actually
> > caused a bigger problem, albeit only for "incorrectly" written code.
>
> It has (also) caused problems for code which was correct (if not pretty)
> given the API as documented in the last release before 3.6.23.1.
>
> > Therefore, shouldn't this original workaround be fixed, in the way
> > prescribed (since for all intents and purposes the new fix is better than
> > the old fix)?
>
> Arguable - either 'fix' is undesirable if you have pre-3.6.23.1 code which
> is expecting to see SQLITE_MISUSE when it used to see it. It also makes a
> certain class of bugs more likely - if you get SQLITE_BUSY within an
> explicit transaction you should roll-back that transaction and begin it
> again; IMO you are more likely to notice and obey that requirement if you
> cannot just immediately call sqlite3_step again.
>
> That said, I like the current behaviour best of the three options, as it's
> less code to write in applications and it's consistent with itself. The
> <3.6.23.1 behaviour is also consistent, and there is a case for going back
> to that if the current behaviour is (with hindsight) a more-incompatible
> change than should have been introduced mid-release.
>
> I like the proposed new fix least, as it still requires sqlite3_reset on
> the normal path and creates an inconsistency between that and the abnormal
> path.
>
> > The issue of whether or not sqlite should provide workarounds (in future)
> to
> > cater for bugs in client software is another question, isn't it?
>
> Yes; I would expect future workaround-requesters to appeal to the precedent
> set this time.
>
> The precedent that SQLite can be improved at any time, and that's what
> happened in 3.6.23.1 so it won't be reverted is one option. This is probably
> what developers expect in the open-source world.
>
> The precedent that flow-affecting changes will not be put in mid-release
> and this 3.6.23.1 change was therefore an error that will be reverted is
> another. This is probably what developers expect in the commercial world.
>
> The third is the precedent that developers don't need to worry about
> reading the documentation and handling errors correctly as SQLite will
> usually be changed in a future release to make their code work. And if this
> change breaks someone else then SQLite will be changed again.
>
> Best Regards,
>
> Phil Willoughby
> --
> Managing Director, StrawberryCat Limited
>
> StrawberryCat Limited is registered in England and Wales with Company No.
> 7234809.
>
> The registered office address of StrawberryCat Limited is:
>
> 107 Morgan Le Fay Drive
> Eastleigh
> SO53 4JH
>
> ___
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>



-- 
VerifEye Technologies Inc.
905-948-0015x245
151 Whitehall Dr, Unit 2
Markham ON, L3R 9T1
Canada
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Propose minor incompatible API change

[Philip Graham Willoughby]

On 11 Jan 2011, at 13:36, Andy Gibbs wrote:

> On Tuesday, January 11, 2011 1:35 PM, Jean-Denis Muys wrote:
> 
>> Don't encumber SQLite with workarounds and special cases
>> to cater to bugs in client software.
> 
> Isn't an accurate synopsis of the problem this: that Sqlite has *already* 
> implemented a workaround in 3.7.0, and that this workaround has actually 
> caused a bigger problem, albeit only for "incorrectly" written code.

It has (also) caused problems for code which was correct (if not pretty) given 
the API as documented in the last release before 3.6.23.1.

> Therefore, shouldn't this original workaround be fixed, in the way 
> prescribed (since for all intents and purposes the new fix is better than 
> the old fix)?

Arguable - either 'fix' is undesirable if you have pre-3.6.23.1 code which is 
expecting to see SQLITE_MISUSE when it used to see it. It also makes a certain 
class of bugs more likely - if you get SQLITE_BUSY within an explicit 
transaction you should roll-back that transaction and begin it again; IMO you 
are more likely to notice and obey that requirement if you cannot just 
immediately call sqlite3_step again.

That said, I like the current behaviour best of the three options, as it's less 
code to write in applications and it's consistent with itself. The <3.6.23.1 
behaviour is also consistent, and there is a case for going back to that if the 
current behaviour is (with hindsight) a more-incompatible change than should 
have been introduced mid-release.

I like the proposed new fix least, as it still requires sqlite3_reset on the 
normal path and creates an inconsistency between that and the abnormal path.

> The issue of whether or not sqlite should provide workarounds (in future) to 
> cater for bugs in client software is another question, isn't it?

Yes; I would expect future workaround-requesters to appeal to the precedent set 
this time.

The precedent that SQLite can be improved at any time, and that's what happened 
in 3.6.23.1 so it won't be reverted is one option. This is probably what 
developers expect in the open-source world.

The precedent that flow-affecting changes will not be put in mid-release and 
this 3.6.23.1 change was therefore an error that will be reverted is another. 
This is probably what developers expect in the commercial world.

The third is the precedent that developers don't need to worry about reading 
the documentation and handling errors correctly as SQLite will usually be 
changed in a future release to make their code work. And if this change breaks 
someone else then SQLite will be changed again.

Best Regards,

Phil Willoughby
-- 
Managing Director, StrawberryCat Limited

StrawberryCat Limited is registered in England and Wales with Company No. 
7234809.

The registered office address of StrawberryCat Limited is:

107 Morgan Le Fay Drive
Eastleigh
SO53 4JH

___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Propose minor incompatible API change

[Andy Gibbs]

On Tuesday, January 11, 2011 1:35 PM, Jean-Denis Muys wrote:

> Don't encumber SQLite with workarounds and special cases
> to cater to bugs in client software.

Isn't an accurate synopsis of the problem this: that Sqlite has *already* 
implemented a workaround in 3.7.0, and that this workaround has actually 
caused a bigger problem, albeit only for "incorrectly" written code.

Therefore, shouldn't this original workaround be fixed, in the way 
prescribed (since for all intents and purposes the new fix is better than 
the old fix)?

The issue of whether or not sqlite should provide workarounds (in future) to 
cater for bugs in client software is another question, isn't it?


And on Tuesday, January 11, 2011 1:56 PM, Michael Black wrote:

> The only thing I would ask is a clear example of what "correctly"
> written isdoes that exist?  I can't find it in the documentataion.

Sorry, not a serious answer to what is a good question, but it might raise a 
smile: http://www.xkcd.com/844/

Andy

___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Propose minor incompatible API change

[Philip Graham Willoughby]

On 11 Jan 2011, at 12:15, Richard Hipp wrote:

> That new OS release includes the latest shared library for
> SQLite.

You didn't put it there, and the consequences of putting it there are not your 
responsibility. Nor are the consequences of someone else's app breaking because 
they didn't read the docs and coded to observed behaviour instead of specified 
behaviour.

Why does the vendor of this OS release not ship multiple versions of SQLite so 
that apps coded to bind to 3.7.4 get 3.7.4 and apps bound to 3.6.23.1 get 
3.6.23.1 and so forth for all previous versions they have shipped? This is 
pretty basic dynamic linking functionality that exists on all modern platforms.

> Sure, the problem really is that the apps were incorrectly
> coded.  But does your mom really care about that?  They worked before.

This situation, or analogous ones, crop up any time you're supplying a product 
with a documented API and design when you have shipped a version which did not 
do exactly what that design specifies and later wish to correct it.

The two extremes of responses are:
1) We noticed a bug in that we weren't enforcing the documented API. We have 
now fixed it. If your app was relying on the previous faulty behaviour you will 
need to rectify it. Remember that the API as documented is what you should code 
to and not the current behaviour which may (due to human error) deviate from 
the spec at any given point in time and therefore may be subject to revision to 
correct it. If you observe such a deviation please report it to us so that it 
can be fixed.
2) We will never change the external behaviour of our product.

Path (2) realistically means you never change anything, never add a new 
feature, never fix any bug, in case someone is reliant on it.

Path (1) will annoy people from time-to-time but it leaves people knowing where 
they stand.

As a developer-customer I prefer (1). I don't want to have to test that the 
code does what it is documented to do all the time, I want to read the docs and 
code to that, and to know that if I find a discrepancy between the documented 
and observed behaviour that I can report a bug and you will (probably, if it 
will affecting more than just my code or I pay you) fix it, or accept a patch 
to fix it.

Best Regards,

Phil Willoughby
-- 
Managing Director, StrawberryCat Limited

StrawberryCat Limited is registered in England and Wales with Company No. 
7234809.

The registered office address of StrawberryCat Limited is:

107 Morgan Le Fay Drive
Eastleigh
SO53 4JH

___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Propose minor incompatible API change

[Black, Michael (IS)]

If you can change the API so "correctly" written apps don't break and "poorly" 
written ones work too...yeah rah...
 
The only thing I would ask is a clear example of what "correctly" written 
isdoes that exist?  I can't find it in the documentataion.  Plus, a clear 
example of "poorly" written...but stil works due to 3.7.5 changesso that 
smart app developers can see what they probably did and must do in the 
future...and how to maintain compatibility across versions.
 
The "Simple C program" is wholly inadequate to show anybody how to do anything 
correctly.  Perhaps a more complex example that shows complete usage is what is 
needed for developers that don't read every line of the documentaion and 
instead rely on other poorly written examples to learn...
 
An example of confusing documentation...

SQLITE_DONE 
<https://legacy2.ngc.com/Exchange/Michael.Black2/Drafts/c3ref/c_abort.html>  
means that the statement has finished executing successfully. sqlite3_step() 
should not be called again on this virtual machine without first calling 
sqlite3_reset() 
<https://legacy2.ngc.com/Exchange/Michael.Black2/Drafts/c3ref/reset.html>  to 
reset the virtual machine back to its initial state.

I think what's meant here is that you can't call the SAME stmt again without 
sqlite3_reset() -- but it doesn't say that...no doubt obvioius to the 
author...makes it sound like if I create a new stmt I have to call it too...

A more complex example would clarify...people take the least painful way to do 
things...and poorly written examples are the bane of good programming...long 
live bubble sort!!

 

 
 
 
Michael D. Black
Senior Scientist
Advanced Analytics Directorate
Northrop Grumman Information Systems
 



From: sqlite-users-boun...@sqlite.org on behalf of Richard Hipp
Sent: Tue 1/11/2011 6:15 AM
To: General Discussion of SQLite Database
Subject: EXTERNAL:Re: [sqlite] Propose minor incompatible API change




I don't think I explained the problem clearly.

The proposed change is for the benefit of the applications customers, not
the application developers.

An end user (think: your mom) wants to upgrade her smartphone to the latest
OS release.  That new OS release includes the latest shared library for
SQLite.  But in so doing, some percentage of the apps she has downloaded
cease to work.  Sure, the problem really is that the apps were incorrectly
coded.  But does your mom really care about that?  They worked before.  Do
we really want thousand, perhaps millions, of moms screaming that SQLite
broke their phone when they upgraded?

--
D. Richard Hipp
d...@sqlite.org
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users


___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Propose minor incompatible API change

[Jean-Denis Muys]

On 11 janv. 2011, at 13:15, Richard Hipp wrote:

> On Tue, Jan 11, 2011 at 6:59 AM, Jean-Christophe Deschamps 
> wrote:
> 
>> 
>> My rationale is that either it's technically or economically feasible
>> for the offending applications' developpers to change their code to use
>> the library correctly (and that doesn't seem to be the case), OR very
>> simply avoid upgrading to the new SQLite versions.  I don't get what
>> good reason they had to switch to new SQLite versions.
>> 
> 
> 
> I don't think I explained the problem clearly.
> 
> The proposed change is for the benefit of the applications customers, not
> the application developers.
> 
> An end user (think: your mom) wants to upgrade her smartphone to the latest
> OS release.  That new OS release includes the latest shared library for
> SQLite.  But in so doing, some percentage of the apps she has downloaded
> cease to work.  Sure, the problem really is that the apps were incorrectly
> coded.  But does your mom really care about that?  They worked before.  Do
> we really want thousand, perhaps millions, of moms screaming that SQLite
> broke their phone when they upgraded?

moms won't know that SQLite is involved. They can blame either the crashing app 
or the OS. And they will probably blame the app. An OS upgrade almost always 
lead to incompatibilities for some app, for a number of reasons. This is the 
least of the reasons for app developers to update their app when an OS is 
upgraded. There is absolutely no reason to think that the app cannot be revised 
(fixed) at this point in time (or any other point in time). Unless the software 
is abandonware. Then, a bug in its SQLite code is the least of mom's problem.

Don't encumber SQLite with workarounds and special cases to cater to bugs in 
client software. If so many moms are using it, the software developers will 
have enough incentives to fix their problems on their side.

Making the API more complex to use or document increases the cognitive load of 
*all* users of SQLite, for the sake of a few programmers who don't bother. 
Don't do it.

If anything, make sure to crash early and often when the API is misused. In 
that way, the client software can't avoid noticing the bug and fixing it.

Jean-Denis

___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Propose minor incompatible API change

[Richard Hipp]

On Tue, Jan 11, 2011 at 6:59 AM, Jean-Christophe Deschamps 
wrote:

>
> My rationale is that either it's technically or economically feasible
> for the offending applications' developpers to change their code to use
> the library correctly (and that doesn't seem to be the case), OR very
> simply avoid upgrading to the new SQLite versions.  I don't get what
> good reason they had to switch to new SQLite versions.
>


I don't think I explained the problem clearly.

The proposed change is for the benefit of the applications customers, not
the application developers.

An end user (think: your mom) wants to upgrade her smartphone to the latest
OS release.  That new OS release includes the latest shared library for
SQLite.  But in so doing, some percentage of the apps she has downloaded
cease to work.  Sure, the problem really is that the apps were incorrectly
coded.  But does your mom really care about that?  They worked before.  Do
we really want thousand, perhaps millions, of moms screaming that SQLite
broke their phone when they upgraded?

-- 
D. Richard Hipp
d...@sqlite.org
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Propose minor incompatible API change

[Jean-Christophe Deschamps]

>This is, technically, a compatibility break.  On the other hand, there
>appear to be vast numbers of smartphone applications that currently depend
>on undefined behavior and will suddenly stop working if we don't make this
>change.

I understand the proposed change will have no incidence for correctly 
written application but as a principle, I really can't agree with the 
idea of changing an engine core for the benefit of poorly written apps 
disseminated in the wild.

My rationale is that either it's technically or economically feasible 
for the offending applications' developpers to change their code to use 
the library correctly (and that doesn't seem to be the case), OR very 
simply avoid upgrading to the new SQLite versions.  I don't get what 
good reason they had to switch to new SQLite versions.  Any basic app 
testing would have revealed incompatibility before they upgrade to 
newer SQLite, so they knew about the issue.

It makes full sense that if a third-party library evolves, which 
changes break your own bad code, then you keep on using the old version 
without asking the library author to take extra steps backward to cope 
with your own incompetence.  That's specially true when the said 
library is free beer.

Since that market share (smartphones) managed to upgrade the 
smartphones in use with the new SQLite version, how can we understand 
that they can't upgrade their apps as well using a conforming API use?

Now if for some weird reason they can change SQLite lib but can't 
upgrade the offending apps in the field, they can still either 
downgrade to the latest SQLite version which works for them or (and 
they all have enough resource for doing so) fork their own private 
SQLite source from the current version and apply any change they 
want/need, without bragging over the rest of the community.

All this doesn't make any sense to me. 

___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Propose minor incompatible API change

[Sylvain Pointeau]

I agree with this change because it doesn't change all well written
programs.
It just tries to correct all incorrectly written ones, which is a good thing
for the end-users.
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Propose minor incompatible API change

[Andy Gibbs]

On Tuesday, January 11, 2011 1:54 AM, Richard Hipp wrote:

> This is, technically, a compatibility break.

But wasn't the original change also a compatibility break?  But this time it 
may make it into the short-form software history at 
http://www.sqlite.org/changes.html?

Having studied the changes made, first to 3.7.0 and then for 3.7.5, my 
opinion is that this could be a change for the better: whereas the original 
change to 3.7.0 may now be considered half-baked, the change proposed here 
actually makes quite a nice feature in some ways.


And on Tuesday, January 11, 2011 3:11 AM, Tod Wulff wrote:

> However for new dev efforts, does it not make sense to consider deploying
> sqlite3_step_v2() which functions as the devs intend - i.e. leave
> sqlite3_step() in place for the legacy code and made sqlite3_step_v2() the
> new defacto standard for current/future dev efforts.?.


I think this can be a good idea, but of course, people may just end up 
misusing it in the same way...

Andy

___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Propose minor incompatible API change

[Jean-Denis Muys]

On 11 janv. 2011, at 01:54, Richard Hipp wrote:

> 
> So the question to you, gentle reader, is should we make this change, and
> break backwards compatibility, albeit in a very obscure way, or should we be
> hard-nosed and force hundreds or perhaps thousands of smartphone application
> developers fix their programs?
> 
> -- 

I have very little sympathy for poorly written apps, and for their developers. 
Accommodating them by increasing the complexity of SQLite may seem the right 
investment in the short term, but is almost certainly a losing proposition in 
the long term (as your question here shows already).

I vote for keeping the API clean. The lazy developers will have to fix their 
code or be driven out of the market (for the benefit of everybody).

Perhaps make the topic more visible in the documentation. Nothing more.

Don't be accommodationist. You stand to lose more than to gain.

Jean-Denis

___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Propose minor incompatible API change

[Philip Graham Willoughby]

On 11 Jan 2011, at 08:20, Max Vlasov wrote:

> On Tue, Jan 11, 2011 at 3:54 AM, Richard Hipp  wrote:
> 
>> 
>> This is, technically, a compatibility break.  On the other hand, there
>> appear to be vast numbers of smartphone applications that currently depend
>> on undefined behavior and will suddenly stop working if we don't make this
>> change.
>> 
>> So the question to you, gentle reader, is should we make this change, and
>> break backwards compatibility, albeit in a very obscure way, or should we
>> be
>> hard-nosed and force hundreds or perhaps thousands of smartphone
>> application
>> developers fix their programs?
>> 
>> 
> A thought came to me. Are we talking about recompiling thousands of
> application or just about applications that use sqlite with some other way
> (dynamic linking or managed layer) ? If the latter (and this is possible the
> case for example for managed-coded platforms such as Android), isn't it
> better to introduce a define for the new functionality that doesn't exists
> in the source, but the developers of the platform should add it to makefile
> in order to this new feature to take effect at once for all these thousand
> applications? I don't think the change itself is bad,  the problem is that
> "minor change syndrome" will never die and life teaches me this again and
> again :)

I assume we are talking about dynamic linking: otherwise thousands of 
programmers have got a new amalgamation, rebuilt their apps and shipped them 
untested. It's not just Android: Apple ship, as part of the iPhone/iPad OS, 
SQLite in a dynamic library which any application can use.

The problem here is twofold:
1 A variety of badly-written applications do not work with the pre-3.6.23.1 API 
specification when a relatively rare condition happens.
2 There is more than one possible response to this condition, and any of the 
available responses will be considered 'best' by someone for their application.

Obviously (1) is a problem, but it isn't immediately obvious to me that it's a 
problem which can or should be fixed by a code change to SQLite rather than by 
telling the producers of badly-written applications to (a) read the docs for 
the APIs they use and (b) test their applications before shipping them.

>From (2) it is obviously not possible to automatically handle the condition in 
>such a way that everyone will be happy; and any automatic handling will be an 
>unwelcome change in behaviour to someone. Worse, that someone may have read 
>the docs and written their application to the specification therein in the 
>first place. Incidentally the second group of apps which were broken by 
>3.6.23.1 may well be in this category - it is harsh to say they are 
>incorrectly coded, as the pre-3.6.23.1 docs and behaviour guaranteed that 
>SQLITE_MISUSE would eventually be returned.

I think the behaviour it has now (implicit reset when that or finalize is all 
you can do) is a very sensible way for it to work. However, we know that breaks 
people who had read the pre-3.6.23.1 docs and observed that they could use 
SQLITE_MISUSE as a simple way to detect they were done with a statement. The 
proposed reset-on-error-only feels the wrong way round to me 
(reset-on-success-only would make more sense), and it will almost certainly 
still break some people.

The problem I see with Max's suggested route is that you potentially end up 
with different semantics from SQLite on all the platforms which provide it as 
part of the OS, which in the long run makes it much harder for application 
developers.

If this were a democracy I'd vote for the pre-3.6.23.1 semantics, as that's the 
documented API for sqlite 3. If the weight of history wasn't there, I'd 
probably vote for the 3.6.23.1 semantics.

Orthogonally: providing some help with application testing would be useful if 
it's possible - for instance a #define or pragma to make all these rare 
conditions more likely so that they can be tested with less effort. I don't 
know how feasible this is.

Best Regards,

Phil Willoughby
-- 
Managing Director, StrawberryCat Limited

StrawberryCat Limited is registered in England and Wales with Company No. 
7234809.

The registered office address of StrawberryCat Limited is:

107 Morgan Le Fay Drive
Eastleigh
SO53 4JH

___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Propose minor incompatible API change

[Max Vlasov]

On Tue, Jan 11, 2011 at 3:54 AM, Richard Hipp  wrote:

>
> This is, technically, a compatibility break.  On the other hand, there
> appear to be vast numbers of smartphone applications that currently depend
> on undefined behavior and will suddenly stop working if we don't make this
> change.
>
> So the question to you, gentle reader, is should we make this change, and
> break backwards compatibility, albeit in a very obscure way, or should we
> be
> hard-nosed and force hundreds or perhaps thousands of smartphone
> application
> developers fix their programs?
>
>
A thought came to me. Are we talking about recompiling thousands of
application or just about applications that use sqlite with some other way
(dynamic linking or managed layer) ? If the latter (and this is possible the
case for example for managed-coded platforms such as Android), isn't it
better to introduce a define for the new functionality that doesn't exists
in the source, but the developers of the platform should add it to makefile
in order to this new feature to take effect at once for all these thousand
applications? I don't think the change itself is bad,  the problem is that
"minor change syndrome" will never die and life teaches me this again and
again :)

Max Vlasov
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Propose minor incompatible API change

[Tod Wulff]

On Mon, Jan 10, 2011 at 9:00 PM, Igor Tandetnik  wrote:
...  They are stuck calling sqlite3_step - incorrectly - so the only
remaining move is to modify the behavior of sqlite3_step itself, to allow
them to muddle through.  ...

Understood and concurred.  It makes sense to capitulate and have the current
implementation of sqlite3_step() support how the market has decided to use
it.  *Right, wrong, or otherwise, it is what it is.*

However for new dev efforts, does it not make sense to consider deploying
sqlite3_step_v2() which functions as the devs intend - i.e. leave
sqlite3_step() in place for the legacy code and made sqlite3_step_v2() the
new defacto standard for current/future dev efforts.?.

-t

On Mon, Jan 10, 2011 at 9:00 PM, Igor Tandetnik  wrote:

> On 1/10/2011 8:52 PM, Chris Peachment wrote:
> > On Mon, 2011-01-10 at 19:54 -0500, Richard Hipp wrote:
> >
> > 
> >
> >> This is, technically, a compatibility break.  On the other hand, there
> >> appear to be vast numbers of smartphone applications that currently
> depend
> >> on undefined behavior and will suddenly stop working if we don't make
> this
> >> change.
> >>
> >
> > What's wrong with using a new function: sqlite3_step_v2()
>
> Presumably, it would be just as difficult to get those broken apps to
> switch to sqlite3_step_v2 as it would be to get them fixed in the first
> place. They are stuck calling sqlite3_step - incorrectly - so the only
> remaining move is to modify the behavior of sqlite3_step itself, to
> allow them to muddle through.
> --
> Igor Tandetnik
>
> ___
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Propose minor incompatible API change

[Igor Tandetnik]

On 1/10/2011 8:52 PM, Chris Peachment wrote:
> On Mon, 2011-01-10 at 19:54 -0500, Richard Hipp wrote:
>
> 
>
>> This is, technically, a compatibility break.  On the other hand, there
>> appear to be vast numbers of smartphone applications that currently depend
>> on undefined behavior and will suddenly stop working if we don't make this
>> change.
>>
>
> What's wrong with using a new function: sqlite3_step_v2()

Presumably, it would be just as difficult to get those broken apps to 
switch to sqlite3_step_v2 as it would be to get them fixed in the first 
place. They are stuck calling sqlite3_step - incorrectly - so the only 
remaining move is to modify the behavior of sqlite3_step itself, to 
allow them to muddle through.
-- 
Igor Tandetnik

___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Propose minor incompatible API change

[Richard Hipp]

On Mon, Jan 10, 2011 at 8:52 PM, Chris Peachment  wrote:

> On Mon, 2011-01-10 at 19:54 -0500, Richard Hipp wrote:
>
> 
>
> > This is, technically, a compatibility break.  On the other hand, there
> > appear to be vast numbers of smartphone applications that currently
> depend
> > on undefined behavior and will suddenly stop working if we don't make
> this
> > change.
> >
>
> What's wrong with using a new function: sqlite3_step_v2()
>

The legacy applications we are trying to fix all link against
sqlite3_step().  If we were to make them changes to sqlite3_step_v2(), we
might as well just ask them to fix their code.  But, as noted in the
original post, that is impractical due to the large number of broken legacy
applications.


>
>
> ___
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>



-- 
D. Richard Hipp
d...@sqlite.org
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Propose minor incompatible API change

[Chris Peachment]

On Mon, 2011-01-10 at 19:54 -0500, Richard Hipp wrote:



> This is, technically, a compatibility break.  On the other hand, there
> appear to be vast numbers of smartphone applications that currently depend
> on undefined behavior and will suddenly stop working if we don't make this
> change.
> 

What's wrong with using a new function: sqlite3_step_v2()


___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

[sqlite] Propose minor incompatible API change

[Richard Hipp]

As you probably know, the sqlite3_step() interface is used to step through
the results of an SQLite query.  The sqlite3_step() function returns
SQLITE_ROW for every row that is returned, then SQLITE_DONE when there are
no more result rows.  After SQLITE_DONE (or an error) is returned, one can
use sqlite3_reset() to rewind the statement back to the beginning then start
calling sqlite3_step() again.

Prior to 3.6.23.1, if you called sqlite3_step() after it returned
SQLITE_DONE and without first calling sqlite3_reset(), you would get an
SQLITE_MISUSE error.  But, we were told, this was causing problems for some
poorly coded applications that were getting back SQLITE_BUSY or
SQLITE_LOCKED and were immediately retrying sqlite3_step() without the
intervening sqlite3_reset().  Rather than force a huge cloud of (mostly
smartphone) apps to be fixed, we decided to change the way sqlite3_step()
worked so that if you called sqlite3_step() again after an error or an
SQLITE_DONE it would reset automatically instead of returning SQLITE_MISUSE.

But now it seems, that this 3.6.23.1 change is causing problems with a
different set of poorly (read: incorrectly) coded (smartphone)
applications.  We are told that there are many apps that terminate their
loop upon receiving SQLITE_MISUSE rather than SQLITE_DONE.  When those
applications are linked against newer versions of SQLite, their loops spin
forever since the sqlite3_step() now resets automatically and never returns
SQLITE_MISUSE.

The proposed solution is to modify the behavior of sqlite3_step() yet again
so that it only automatically resets after returning either the SQLITE_BUSY
or SQLITE_LOCKED error.  Any call to sqlite3_step() after returning
SQLITE_DONE or some error other than BUSY or LOCKED will revert generating
an SQLITE_MISUSE, just as it did for all versions of SQLite prior to
3.6.23.1.

This is, technically, a compatibility break.  On the other hand, there
appear to be vast numbers of smartphone applications that currently depend
on undefined behavior and will suddenly stop working if we don't make this
change.

So the question to you, gentle reader, is should we make this change, and
break backwards compatibility, albeit in a very obscure way, or should we be
hard-nosed and force hundreds or perhaps thousands of smartphone application
developers fix their programs?

-- 
D. Richard Hipp
d...@sqlite.org
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users