Re: [sqlite] New draft document on the new pointer-passing interfaces
If you’re in a position to perform an attack that address space randomization can defend against, then you have already successfully performed a code execution attack. And so far as I can tell... *any* constant strings in the executable, including things like elements of the SQL language itself, provide a mechanism for locating code. Meanwhile, what this design does is remove an attack surface for promoting an SQL injection attack to a code execution attack. The comments about randomizing strings in the previous threads were about possible alternate ways of mitigating that attack, and have nothing to do with ASLR. ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] New draft document on the new pointer-passing interfaces
>...the attacker is already able to read the process’s address space. The rest of us here are saying that, in that case, as far as we’re concerned the attacker has already won. I understand that. What I'm saying is your standard is not nuanced. Applying a security standard that amounts to best practice web server administration is insufficient for all environments. Extra care is warranted where compromised apps can be installed alongside yours. Maybe there is no exploit to take over the app or host but there could be very good ones for your app to assist in eavesdropping and boosting social engineering attacks. The goal should be "secure in the wild" rather than merely secure on the server. A bit of reading reveals how the most spectacular SQLite exploits so far occurred in desktop and portable environments marketed as a platforms for users to commingle apps of their choice and quality. Therefore, a more realistic security threat standard comes from imagining an environment where unknown threats and partially effective countermeasures are presently in a precarious stalemate equilibrium. Introducing an app with secure in isolation but unorthodox implementation may not have the secure in the wild guarantee that is expected. At this point I've got no dog in this fight. The new pointer API is now fairly complete and can be mechanically patched to use a more sensible key type. Serious implementors anticipating widespread desktop or mobile deployment can do this easily for themselves. ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] New draft document on the new pointer-passing interfaces
> On Jul 28, 2017, at 2:46 PM, peternwrote: > > The attack vector is described well enough. A penetration search harness > would work directly from predictable accesses to the published key > constants as mapped into process address space. [Full ASLR decoding, see > the paper, isn't needed for this.] These well advertised accesses "light > up" otherwise randomized program and stack layouts very well since there is > no access for any other purpose. CPU security might prevent directly > forging pseudo-null pointers but they are as observable as subtype pointers > and leak more address space layout information. You’re getting _closer_ to something a non-expert could understand, but you’re not there yet. Keep trying, please. It would help if you described what the attack requires, and what it would allow the attacker to do. Reading between the lines, I think you’re assuming the attacker is already able to read the process’s address space. The rest of us here are saying that, in that case, as far as we’re concerned the attacker has already won. And I don’t understand how this pointer feature would help such an attacker get any further. —Jens ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] New draft document on the new pointer-passing interfaces
Here I'm replying to Richard and subsequent comments on Richard's question. The attack vector is described well enough. A penetration search harness would work directly from predictable accesses to the published key constants as mapped into process address space. [Full ASLR decoding, see the paper, isn't needed for this.] These well advertised accesses "light up" otherwise randomized program and stack layouts very well since there is no access for any other purpose. CPU security might prevent directly forging pseudo-null pointers but they are as observable as subtype pointers and leak more address space layout information. I suppose what you're really after is to see a demonstrable exploit to know whether or not is a SQLite component or side channel. For that, you'll have to wait. I don't have one at my fingertips but can imagine possibilities and incentives. About that 2017 paper. It details a practical universal decoder against the randomized memory space countermeasures on every major CPU architecture. It's not really that technical. In fact, pointer leaks to randomized memory were cited in Richard's essay where, in a jarring leap, he went from the goal of preserving randomization to a new API with new countermeasures. This reader was left wondering how he got there rather than to a plan that finished up the deliberately incomplete but proven subtype pointer API. By introducing the paper, I was hoping for an appreciation of irony. For it is ironic how the new pseudo-null API leans even more heavily on a waning shield of address space randomization than the subtype API did. On Thu, Jul 27, 2017 at 11:41 AM, Jens Alfkewrote: > > > On Jul 27, 2017, at 10:02 AM, petern > wrote: > > > > Are you able to put two facts together? > > > > What prevents stack busting or other code injection attacks on an > otherwise > > valid pseudo-null pointer by simply decoding the address space and > > observing where strcmp() loads a register to one of the pointer "keys" > > you've insisted be conveniently published for hackers in the data > segment? > > Peter, you’re > (a) assuming a lot of specialized domain knowledge, by using jargon and > referring to highly technical papers; > (b) referring vaguely to threats instead of describing a clear problem or > attack vector; > (c) speaking very condescendingly to the people you’re trying to convince. > > None of these help your case at all. I’m interested in the security > implications of this API, and I’ve got some security knowledge, but not > apparently enough to follow along. Richard is right: please describe > clearly a situation where this API results in an attack vector. > > —Jens > ___ > sqlite-users mailing list > sqlite-users@mailinglists.sqlite.org > http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users > ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] New draft document on the new pointer-passing interfaces
On Thursday, 27 July, 2017 11:03, peternwrote: >Richard. Were you aware of this paper? >http://www.cs.vu.nl/~herbertb/download/papers/anc_ndss17.pdf >Are you able to put two facts together? >What prevents stack busting or other code injection attacks on an >otherwise valid pseudo-null pointer by simply decoding the address >space and observing where strcmp() loads a register to one of the >pointer "keys" you've insisted be conveniently published for >hackers in the data segment? Absolutely nothing prevents it. However, this would be an application problem and not an SQLite problem. The purpose of using the pointer interface is to disable the ability of "unsanitized user created SQL" (aka Johhny Tables) from being able to pass pointers into and out of the SQLite3 library via crafted SQL. The application developer must specifically code the ability to send and receive pointers. This can no longer be done and will no longer return meaningful results. It does nothing to secure the application code from exploitation by "other means". In other words, if a "badly written application" were subject to any sort of machine code injection attack, there is nothing to whatsoever to prevent the injected code from accessing the SQLite engine functions (or the OS kernel or anything else), binding pointers using the new interface, and extracting pointers. Nothing except that if the application already has this vulnerability (code injection via whatever means) there are much easier (not to mention smaller and simpler) methods to retrieve the same data which does not depend on using the database engine. ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] New draft document on the new pointer-passing interfaces
> On Jul 27, 2017, at 10:02 AM, peternwrote: > > Are you able to put two facts together? > > What prevents stack busting or other code injection attacks on an otherwise > valid pseudo-null pointer by simply decoding the address space and > observing where strcmp() loads a register to one of the pointer "keys" > you've insisted be conveniently published for hackers in the data segment? Peter, you’re (a) assuming a lot of specialized domain knowledge, by using jargon and referring to highly technical papers; (b) referring vaguely to threats instead of describing a clear problem or attack vector; (c) speaking very condescendingly to the people you’re trying to convince. None of these help your case at all. I’m interested in the security implications of this API, and I’ve got some security knowledge, but not apparently enough to follow along. Richard is right: please describe clearly a situation where this API results in an attack vector. —Jens ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] New draft document on the new pointer-passing interfaces
On 7/27/17, peternwrote: > > What prevents stack busting or other code injection attacks on an otherwise > valid pseudo-null pointer by simply decoding the address space and > observing where strcmp() loads a register to one of the pointer "keys" > you've insisted be conveniently published for hackers in the data segment? > I do not understand what this sentence means. Can you explain it again in simpler terms? Refresh my memory please: What exactly (and succinctly) is your complain with the current sqlite3_bind_pointer(), sqlite3_result_pointer(), and sqlite3_value_pointer() design? Are there multiple complains? Can you enumerate them? Please be as specific as possible. -- D. Richard Hipp d...@sqlite.org ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] New draft document on the new pointer-passing interfaces
Richard. Were you aware of this paper? http://www.cs.vu.nl/~herbertb/download/papers/anc_ndss17.pdf Are you able to put two facts together? What prevents stack busting or other code injection attacks on an otherwise valid pseudo-null pointer by simply decoding the address space and observing where strcmp() loads a register to one of the pointer "keys" you've insisted be conveniently published for hackers in the data segment? On Tue, Jul 25, 2017 at 10:43 AM, Richard Hippwrote: > On 7/24/17, petern wrote: > > Great. But, if this is an ultimate replacement for BLOB'ed pointers, > these > > new pseudo-null pointers must support SQLITE_STATIC and destructor > function > > pointer lifetime disposition for those migrating their code. > > Nobody is forcing you to migrate your legacy code to the new API. > Anything that worked for you in 3.19.3 (or earlier) will continue to > work just as well in 3.20.0. If what you are doing now works for you, > then you are welcomed to keep doing exactly the same in the future. > > > > > Why can't the producer destructor disposition be preserved within a chain > > of application functions by subsequent consumers passing SQLITE_STATIC > > disposition as they do now? > > I cannot say with precision because your proposal is vague. But what > you want to do will very likely use extra memory and extra CPU cycles > for the overwhelmingly common case where pointers are not being > passed. We do not want to burden the common case for the convenience > of an outlier. > > -- > D. Richard Hipp > d...@sqlite.org > ___ > sqlite-users mailing list > sqlite-users@mailinglists.sqlite.org > http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users > ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] New draft document on the new pointer-passing interfaces
On 7/24/17, peternwrote: > Great. But, if this is an ultimate replacement for BLOB'ed pointers, these > new pseudo-null pointers must support SQLITE_STATIC and destructor function > pointer lifetime disposition for those migrating their code. Nobody is forcing you to migrate your legacy code to the new API. Anything that worked for you in 3.19.3 (or earlier) will continue to work just as well in 3.20.0. If what you are doing now works for you, then you are welcomed to keep doing exactly the same in the future. > > Why can't the producer destructor disposition be preserved within a chain > of application functions by subsequent consumers passing SQLITE_STATIC > disposition as they do now? I cannot say with precision because your proposal is vague. But what you want to do will very likely use extra memory and extra CPU cycles for the overwhelmingly common case where pointers are not being passed. We do not want to burden the common case for the convenience of an outlier. -- D. Richard Hipp d...@sqlite.org ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] New draft document on the new pointer-passing interfaces
My tone isn't about the technical development discussion. Its about my subscribing to this forum and seeing my 11 year olds mentality shine through with his "I'm not getting the attention I want, so I'm going to yell and scream and pout until I get what I want". Perhaps it is a language barrier, or a cultural difference I don't know about, but, in the last SEVERAL notes (Pretty much the post right after Dr Hipps notice of the draft in this particular thread), this hasn't been levied to the technical side of things, but an individual who is screaming and making demands. Making demands isn't a discussion. No one is perfect, I get that. And please, do discuss the possible or impossibility of the security faults of the library. I'm interested in that. I'm not interested in seeing the demands. On Tue, Jul 25, 2017 at 1:00 PM, Jens Alfkewrote: > > > On Jul 25, 2017, at 9:39 AM, Stephen Chrzanowski > wrote: > > > > Your attitude towards a public forum and bully attempts isn't required > > here. I'd ask YOU to leave based on the fact that your behavior is > > anything but professional, as I'm not interested in your self proclaimed > > expertise. > > Stephen, I don’t think this is called for. Peter is being somewhat > vehement (as are you) but I don’t think anything he’s said counts as > bullying or is otherwise out of bounds in a technical discussion. > > > My expertise extends to the knowledge that what Dr Hipp has written is of > > the best interest of all the BILLIONS of devices that are out in the > field > > Argument by appeal to authority isn’t a good idea here. I’m sure Dr Hipp > doesn’t consider himself infallible (even if he is a doctor), and part of > the reason for open development is to catch the mistakes and incorrect > assumptions that all of us are prone to. > > I have some modest knowledge of computer security, and I know it’s > vulnerable to very, very subtle design errors that can leave openings for > attack. (Security bugs are unique in that they are not triggered randomly, > but are actively hunted down and exploited by human-level intelligences.) > It’s in everyone’s best interest that any security related issues in the > design be thoroughly debated before this is released. > > > If there is an aspect of this software you do not like, fork it, and > write > > your own code to remove what Dr Hipp and his team has put into the code. > > Again, this is unhelpful. We all understand the basics of open source > development. Forking isn’t a solution, it’s an admission of failure. > > —Jens > ___ > sqlite-users mailing list > sqlite-users@mailinglists.sqlite.org > http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users > ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] New draft document on the new pointer-passing interfaces
> On Jul 25, 2017, at 9:39 AM, Stephen Chrzanowskiwrote: > > Your attitude towards a public forum and bully attempts isn't required > here. I'd ask YOU to leave based on the fact that your behavior is > anything but professional, as I'm not interested in your self proclaimed > expertise. Stephen, I don’t think this is called for. Peter is being somewhat vehement (as are you) but I don’t think anything he’s said counts as bullying or is otherwise out of bounds in a technical discussion. > My expertise extends to the knowledge that what Dr Hipp has written is of > the best interest of all the BILLIONS of devices that are out in the field Argument by appeal to authority isn’t a good idea here. I’m sure Dr Hipp doesn’t consider himself infallible (even if he is a doctor), and part of the reason for open development is to catch the mistakes and incorrect assumptions that all of us are prone to. I have some modest knowledge of computer security, and I know it’s vulnerable to very, very subtle design errors that can leave openings for attack. (Security bugs are unique in that they are not triggered randomly, but are actively hunted down and exploited by human-level intelligences.) It’s in everyone’s best interest that any security related issues in the design be thoroughly debated before this is released. > If there is an aspect of this software you do not like, fork it, and write > your own code to remove what Dr Hipp and his team has put into the code. Again, this is unhelpful. We all understand the basics of open source development. Forking isn’t a solution, it’s an admission of failure. —Jens ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] New draft document on the new pointer-passing interfaces
On Tue, Jul 25, 2017 at 12:25 PM, peternwrote: > You're trying to change the topic to the security model. This thread is > supposed to be about a lengthy beyond the pale proposal that named all > manner of hypothetical boogie men before concluding the only way is a > "nuclear solution" as in: "Let's just nuke it, that's only way to be > safe". I'll ask again. Did you read the proposal itself? If not, why are > you responding on this thread? What is your expertise exactly? > > My expertise extends to the knowledge that what Dr Hipp has written is of the best interest of all the BILLIONS of devices that are out in the field, actively used knowingly, or unknowingly by the users who use the library. Your attitude towards a public forum and bully attempts isn't required here. I'd ask YOU to leave based on the fact that your behavior is anything but professional, as I'm not interested in your self proclaimed expertise. Perhaps in the software development area, you are an expert in your own products you deploy, but absolutely NOT in the area of what this public domain library is being used for. If there is an aspect of this software you do not like, fork it, and write your own code to remove what Dr Hipp and his team has put into the code. ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] New draft document on the new pointer-passing interfaces
On 7/25/17, 11:25 AM, "sqlite-users on behalf of petern"wrote: > You're trying to change the topic to the security model. All I was doing was pointing out that hiding the type information from attackers is not a requirement, so the fact that it’s visible if you examine the binary or source is not relevant. Now you’ve acknowledged that you’re not pointing out any actual security flaw, we’re done, right? ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] New draft document on the new pointer-passing interfaces
Richard, Your description of the rational for the new pointer values is sound and quite reasonable. I believe that it meets all the requirements you have set out to prevent SQL from being used to either retrieve or set arbitrary internal use (that is internal to both the application and SQLite3) pointers that was not directly intended (and pre-coded) by the application developer. How much of a security risk being able to do so would present is not particularly relevant to the implementation and is difficult to estimate because it certainly varies considerably by host OS. Simply eliminating the possibility of vulnerability via this method is adequate and commendable. It should be noted however that this does not in any way prevent or mitigate that a "malicious application" will in fact be able to access or set pointer values, however, this is not within the scope of the change. The scope of the change is to protect existing and future applications from being used to perform side channel pointer attacks via crafted SQL injections that were not intended by the application author. At this it will succeed brilliantly. --- The fact that there's a Highway to Hell but only a Stairway to Heaven says a lot about anticipated traffic volume. >-Original Message- >From: sqlite-users [mailto:sqlite-users- >boun...@mailinglists.sqlite.org] On Behalf Of Richard Hipp >Sent: Monday, 24 July, 2017 05:54 >To: General Discussion of SQLite Database >Subject: [sqlite] New draft document on the new pointer-passing >interfaces > >https://www.sqlite.org/draft/bindptr.html > >-- >D. Richard Hipp >d...@sqlite.org >___ >sqlite-users mailing list >sqlite-users@mailinglists.sqlite.org >http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] New draft document on the new pointer-passing interfaces
You're trying to change the topic to the security model. This thread is supposed to be about a lengthy beyond the pale proposal that named all manner of hypothetical boogie men before concluding the only way is a "nuclear solution" as in: "Let's just nuke it, that's only way to be safe". I'll ask again. Did you read the proposal itself? If not, why are you responding on this thread? What is your expertise exactly? Also, where is your critique of my suggestion to simply and logically expand the well proven and secure subtype mechanism? Why do you presume that Richard's idea to suddenly create a parallel API is automatically the best plan? What is your expertise on the main issue of how to pass around runtime object pointers? If you reply to this thread again, please demonstrate some expertise on the issue at hand. Now I'll address the question of remote attack path in case somebody mistakes a superficial question for a valid point. I don't know any exact attack path, local or remote, and neither does Richard. That's the point. However, all unknowns being equal, if there is a local way to inject pointers then there could be a remote way. Whatever the method, the attacker can then use the supposedly secure constant space names everywhere the application is deployed without further opposition. If you recall, I pointed out that the programmer is free to rotate or randomize integer subtypes of the current API at runtime if it's keeping them from sleep. So in this vein of these hypothetical threats, constant space names are no magic bullet. They have drawbacks too. That being said, let me be very clear once again for short attention spans. There is no attack path in the current API. It works fine and that was proven in the field! So please, BEFORE RESPONDING TO THIS POST, DO READ THE PROPOSAL WHERE THIS IS CLEARLY STATED! On Tue, Jul 25, 2017 at 6:11 AM, Peter Da Silva < peter.dasi...@flightaware.com> wrote: > On 7/24/17, 7:20 PM, "sqlite-users on behalf of petern" < > sqlite-users-boun...@mailinglists.sqlite.org on behalf of > peter.nichvolo...@gmail.com> wrote: > > Justifications presented in the proposal claim hardwired constants for > mandatory lock and key style pointer value receiving are a great idea > because SQL can't generate constant space strings. > > This has nothing to do with the secrecy or otherwise of these strings, but > to prevent developers from _even in principle_ implementing a mechanism for > passing these strings in from SQL. If it is not possible to inject the > string, then it doesn’t matter if they’re secret or not. Think of it as > preventing the creation of a “cross domain” attack from an insecure module. > > You might as well complain that the names of internal functions in SQLite > are known. There’s no way from SQL to call them, so it’s not an attack > surface. > > > On the local device the hacker attack space would be immediately > narrowed to constants listed in the executable which, I might add, are > guaranteed to work on remote copy of the same application! > > How do you figure? What is the attack path you’re postulating? > > > ___ > sqlite-users mailing list > sqlite-users@mailinglists.sqlite.org > http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users > ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] New draft document on the new pointer-passing interfaces
Justifications presented in the proposal claim hardwired constants for mandatory lock and key style pointer value receiving are a great idea because SQL can't generate constant space strings. And, this is true provided the executable is secret and remote. I know there are a lot of web server jockeys on this forum that are quick with the PHP answers so, perhaps, this assertion makes sense to many. What I am pointing out is how those same mandatory hardwired secret constants work against security in the domain of local DB on a portable device. On the local device the hacker attack space would be immediately narrowed to constants listed in the executable which, I might add, are guaranteed to work on remote copy of the same application! As well, this particular justification apparently is the reason to make something completely new and utterly parallel with the existing subtype solution which works fine and could be extended to do the job. Did you read the part of the proposal essay where the existing API is mentioned? Now, what about the first part of my reply? No comment on that? You accept what I said there? I'm glad to learn people are coming to their senses. :-) On Mon, Jul 24, 2017 at 1:52 PM, Peter Da Silva < peter.dasi...@flightaware.com> wrote: > On 7/24/17, 3:50 PM, "sqlite-users on behalf of petern" < > sqlite-users-boun...@mailinglists.sqlite.org on behalf of > peter.nichvolo...@gmail.com> wrote: > > BTW, if the hypothetical attacker has a copy of the application, aren't > the constant space pointer access keys' string addresses all there in clear > text > > But that’s not part of the security model, so what’s the problem? > > > ___ > sqlite-users mailing list > sqlite-users@mailinglists.sqlite.org > http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users > ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] New draft document on the new pointer-passing interfaces
On 7/24/17, 3:50 PM, "sqlite-users on behalf of petern"wrote: > BTW, if the hypothetical attacker has a copy of the application, aren't the > constant space pointer access keys' string addresses all there in clear text But that’s not part of the security model, so what’s the problem? ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] New draft document on the new pointer-passing interfaces
Your proposal does not walk through the alternative of sticking with subtypes to add non-persistent sqlite3_bind_subtype() and corresponding sqlite3_column_subtype() methods. With a few extra lines and some imagination can't this more straightforward alternative be combined with the existing BLOB pointer interface to reach the desired outcome in FTS and carray? BTW, if the hypothetical attacker has a copy of the application, aren't the constant space pointer access keys' string addresses all there in clear text. The castle walls will be no higher than those of a discretionary pointer access protocol with subtypes. In fact, subtypes could afford greater security at runtime if the programmer rotates or otherwise randomizes the type id's. On Mon, Jul 24, 2017 at 10:05 AM, Richard Hippwrote: > On 7/24/17, petern wrote: > > > > Are sqlite3_result_subtype() and sqlite3_value_subtype() being deprecated > > in light of the duplicate functionality? > > > > No. The subtype() interfaces were originally created for completely > unrelated purposes (specifically to identify validated JSON text in > the JSON1 extension) and will continue to live on to serve those > unrelated purposes. > > -- > D. Richard Hipp > d...@sqlite.org > ___ > sqlite-users mailing list > sqlite-users@mailinglists.sqlite.org > http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users > ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] New draft document on the new pointer-passing interfaces
Gwendal. Yes. You've missed something. My application is working code not a hypothetical feature request. BLOB application object pointer lifetime presently works precisely as I've described and without memory leak. My point (and Dominique's point) was that this proposal as it stands isn't a drop in replacement for that BLOB'ed application object pointer protocol. Your reply also suggests superficial understanding. For example, regarding the SQLITE_TRANSIENT BLOB disposition you mentioned. That one is generally not useful here unless the particular application object can be completely materialized elsewhere in memory from shallow copy without loss of fidelity or introduction of memory leak. I'm not talking about copying dumb strings. These are highly stateful and possibly third party objects containing still more pointers and reference counters. They cannot be memcopied around willy-nilly. Take a look at my earlier email that really applied the BLOB object technique to functions if you have genuine interest. This isn't some half baked idea that came about in a flight of fancy. There is a thread of emails leading up to this point. On Mon, Jul 24, 2017 at 11:12 AM, Gwendal Rouéwrote: > > > Le 24 juil. 2017 à 19:02, petern a écrit : > > > > Great. But, if this is an ultimate replacement for BLOB'ed pointers, > these > > new pseudo-null pointers must support SQLITE_STATIC and destructor > function > > pointer lifetime disposition for those migrating their code. > > You're right that the new APIs make pointer values unsuitable for > *intermediate* values. > > For example, `SELECT free(use(initialize(alloc(123`, while possible, > is out of scope. One can define such functions, but it's very risky to > misuse them, leak memory, or double-free. > > The new pointer values indeed look *designed* so that the lifetime of > pointed values is managed *outside* of any statement execution. > > I fail to see how blob'ed pointers did not have the exact same issue. > SQLITE_STATIC and SQLITE_TRANSIENT manage the lifetime of the blob content, > not of the content pointed by a blob'ed pointer. Or did I miss something? > > Gwendal Roué > > ___ > sqlite-users mailing list > sqlite-users@mailinglists.sqlite.org > http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users > ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] New draft document on the new pointer-passing interfaces
> Le 24 juil. 2017 à 19:02, peterna écrit : > > Great. But, if this is an ultimate replacement for BLOB'ed pointers, these > new pseudo-null pointers must support SQLITE_STATIC and destructor function > pointer lifetime disposition for those migrating their code. You're right that the new APIs make pointer values unsuitable for *intermediate* values. For example, `SELECT free(use(initialize(alloc(123`, while possible, is out of scope. One can define such functions, but it's very risky to misuse them, leak memory, or double-free. The new pointer values indeed look *designed* so that the lifetime of pointed values is managed *outside* of any statement execution. I fail to see how blob'ed pointers did not have the exact same issue. SQLITE_STATIC and SQLITE_TRANSIENT manage the lifetime of the blob content, not of the content pointed by a blob'ed pointer. Or did I miss something? Gwendal Roué ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] New draft document on the new pointer-passing interfaces
> Le 24 juil. 2017 à 19:02, peterna écrit : > > To those posting low information congratulatory notes on this thread, you'd > better hold off on popping those champagne corks. The current API already > contains irreversible additions to solve this problem that fell short. Congrats can also go to clear documentation, scope, and rationale, even if the feature can be discussed :-) Gwendal ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] New draft document on the new pointer-passing interfaces
On 7/24/17, peternwrote: > > Are sqlite3_result_subtype() and sqlite3_value_subtype() being deprecated > in light of the duplicate functionality? > No. The subtype() interfaces were originally created for completely unrelated purposes (specifically to identify validated JSON text in the JSON1 extension) and will continue to live on to serve those unrelated purposes. -- D. Richard Hipp d...@sqlite.org ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] New draft document on the new pointer-passing interfaces
Great. But, if this is an ultimate replacement for BLOB'ed pointers, these new pseudo-null pointers must support SQLITE_STATIC and destructor function pointer lifetime disposition for those migrating their code. Why can't the producer destructor disposition be preserved within a chain of application functions by subsequent consumers passing SQLITE_STATIC disposition as they do now? Isn't this feature just an accident of statement scope controlled destruction that will continue to work with tracked lifetime pseudo-null pointers? BTW, let's call them what they are. These are explicit pseudo-nulls for the purpose of keeping pointer bits out of band from hacker SQL. Also. What is to stop black budget funded developers from creating popular applications in the wild which preserve penetration channels of BLOB pointers the original way? Total security improvement justifications for the pseudo-null pointer API are specious if the API is merely another alternative. Are sqlite3_result_subtype() and sqlite3_value_subtype() being deprecated in light of the duplicate functionality? Supplementing/deprecating the already secure sqlite3_X_subtype() API with a more complete and pointer leak opaque replacement sqlite3_X_pointer() API seems a worthy goal. But, if that's the plan, where is the rest to the proposal? Honestly, it appears all you've proposed so far is yet another way to pass pointers more aligned with the whims of your present tastes for FTS3 MATCH, FTS5 extensions, and one code sample. To those posting low information congratulatory notes on this thread, you'd better hold off on popping those champagne corks. The current API already contains irreversible additions to solve this problem that fell short. On Mon, Jul 24, 2017 at 4:54 AM, Richard Hippwrote: > https://www.sqlite.org/draft/bindptr.html > > -- > D. Richard Hipp > d...@sqlite.org > ___ > sqlite-users mailing list > sqlite-users@mailinglists.sqlite.org > http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users > ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] New draft document on the new pointer-passing interfaces
Minor typo's and grammo's: 2.1) suppose -> supposed For example, the first argument to the snippet() function is suppose to be a special column space instead of the fts3cursor object it was suppose to be operating on 2.2) forge pointer -> forge a pointer forge pointer without the help of SQLite. 3) forge a pointers -> forge pointers Hence, it is not possible for SQL to forge a pointers. 3.1) and -> an been passed and invalid pointer. 4) operating -> operation insert a sort operating prior 4) insert -> inserted pointer value insert into a query --- The fact that there's a Highway to Hell but only a Stairway to Heaven says a lot about anticipated traffic volume. >-Original Message- >From: sqlite-users [mailto:sqlite-users- >boun...@mailinglists.sqlite.org] On Behalf Of Richard Hipp >Sent: Monday, 24 July, 2017 05:54 >To: General Discussion of SQLite Database >Subject: [sqlite] New draft document on the new pointer-passing >interfaces > >https://www.sqlite.org/draft/bindptr.html > >-- >D. Richard Hipp >d...@sqlite.org >___ >sqlite-users mailing list >sqlite-users@mailinglists.sqlite.org >http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] New draft document on the new pointer-passing interfaces
On Mon, Jul 24, 2017 at 1:54 PM, Richard Hippwrote: > https://www.sqlite.org/draft/bindptr.html Thanks. Very helpful. Still unsure whether not having a destructor D for pointer P is a good thing though. The text explicitly says the pointer is "destroyed" when not flowing directly from producer to consumer, which could mean calling the D(P) in that case. The current semantic seem to imply the pointer producer is longer lived than the pointer consumer, and thus no lifetime management is necessary, but will that always be the case? Also, the new pointer APIs are said to be an enhanced version of *_subtype(), so there's some overlap, but I guess one can use both at the same time, no? Or is _subtype() deprecated? JSON1 uses _subtype I believe, so _subtype() has more uses than the one _pointer() APIs replace it for? Maybe an extra paragraph regarding _pointer vs _subtype() would be helpful? At least I'd like your perspective on it. Thanks, --DD Minor fixes: s/Upping The Thread Level/Upping The Threat Level/ s/Consideration was giving to using integer values as the pointer type/Consideration was giveb to using integer values as the pointer type/ s/if a pointer value insert into a query/if a pointer value inserted into a query/ ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] New draft document on the new pointer-passing interfaces
If you’re concerned about people faking pointers, you could always go with something like “org.sqlite.fts3.01Yzg5x.snippet”. But given this design, you need to have already injected a C extension to even access the pointers, so I’m not sure what the issue is. On 7/24/17, 9:19 AM, "sqlite-users on behalf of Hick Gunter" <sqlite-users-boun...@mailinglists.sqlite.org on behalf of h...@scigames.at> wrote: Naming conventions (if strictly adhered to) are moderately good at avoiding conflicts, but take the guesswork out of "faking a pointer". It also assumes that pointers for "whatever" are interchangeable between different queries in a process, i.e. passing a "whatever" pointer from statement A to a function in statement B does not pose a risk. -Ursprüngliche Nachricht- Von: sqlite-users [mailto:sqlite-users-boun...@mailinglists.sqlite.org] Im Auftrag von Peter Da Silva Gesendet: Montag, 24. Juli 2017 15:37 An: SQLite mailing list <sqlite-users@mailinglists.sqlite.org> Betreff: Re: [sqlite] New draft document on the new pointer-passing interfaces What about imposing some structure on the pointer type strings that uses a guaranteed unique substring, for example “org.sqlite.fts3.snippet”, to ensure there wouldn’t be accidental conflicts? On 7/24/17, 6:54 AM, "sqlite-users on behalf of Richard Hipp" <sqlite-users-boun...@mailinglists.sqlite.org on behalf of d...@sqlite.org> wrote: https://www.sqlite.org/draft/bindptr.html -- D. Richard Hipp d...@sqlite.org ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users ___ Gunter Hick Software Engineer Scientific Games International GmbH FN 157284 a, HG Wien Klitschgasse 2-4, A-1130 Vienna, Austria Tel: +43 1 80100 0 E-Mail: h...@scigames.at This communication (including any attachments) is intended for the use of the intended recipient(s) only and may contain information that is confidential, privileged or legally protected. Any unauthorized use or dissemination of this communication is strictly prohibited. If you have received this communication in error, please immediately notify the sender by return e-mail message and delete all copies of the original communication. Thank you for your cooperation. ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] New draft document on the new pointer-passing interfaces
Naming conventions (if strictly adhered to) are moderately good at avoiding conflicts, but take the guesswork out of "faking a pointer". It also assumes that pointers for "whatever" are interchangeable between different queries in a process, i.e. passing a "whatever" pointer from statement A to a function in statement B does not pose a risk. -Ursprüngliche Nachricht- Von: sqlite-users [mailto:sqlite-users-boun...@mailinglists.sqlite.org] Im Auftrag von Peter Da Silva Gesendet: Montag, 24. Juli 2017 15:37 An: SQLite mailing list <sqlite-users@mailinglists.sqlite.org> Betreff: Re: [sqlite] New draft document on the new pointer-passing interfaces What about imposing some structure on the pointer type strings that uses a guaranteed unique substring, for example “org.sqlite.fts3.snippet”, to ensure there wouldn’t be accidental conflicts? On 7/24/17, 6:54 AM, "sqlite-users on behalf of Richard Hipp" <sqlite-users-boun...@mailinglists.sqlite.org on behalf of d...@sqlite.org> wrote: https://www.sqlite.org/draft/bindptr.html -- D. Richard Hipp d...@sqlite.org ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users ___ Gunter Hick Software Engineer Scientific Games International GmbH FN 157284 a, HG Wien Klitschgasse 2-4, A-1130 Vienna, Austria Tel: +43 1 80100 0 E-Mail: h...@scigames.at This communication (including any attachments) is intended for the use of the intended recipient(s) only and may contain information that is confidential, privileged or legally protected. Any unauthorized use or dissemination of this communication is strictly prohibited. If you have received this communication in error, please immediately notify the sender by return e-mail message and delete all copies of the original communication. Thank you for your cooperation. ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] New draft document on the new pointer-passing interfaces
What about imposing some structure on the pointer type strings that uses a guaranteed unique substring, for example “org.sqlite.fts3.snippet”, to ensure there wouldn’t be accidental conflicts? On 7/24/17, 6:54 AM, "sqlite-users on behalf of Richard Hipp"wrote: https://www.sqlite.org/draft/bindptr.html -- D. Richard Hipp d...@sqlite.org ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] New draft document on the new pointer-passing interfaces
>https://www.sqlite.org/draft/bindptr.html Thank you very much for this, detailed, informative write-up, Dr Hipp. It's very helpful to see the sensible rationale behind the new interfaces. Thanks for continuing to enhance the API! ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] New draft document on the new pointer-passing interfaces
> Le 24 juil. 2017 à 13:54, Richard Hippa écrit : > > https://www.sqlite.org/draft/bindptr.html Thank you very much for this detailed rationale! Gwendal Roué ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
[sqlite] New draft document on the new pointer-passing interfaces
https://www.sqlite.org/draft/bindptr.html -- D. Richard Hipp d...@sqlite.org ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users