Re: [sqlite] SQL_STATIC unterminated strings, and sqlite3_result_text

[David de Regt]

Sorry to threadjack here, but this made me think of something...

Does this mean that sqlite3_column_text always makes a copy of the string to 
put a null terminator on the end?  My ORM uses std::strings in UTF8 everywhere, 
so does that mean it would be quite a bit faster to pull strings out using 
sqlite3_column_bytes?  When I'm inserting, I'm always using bound columns with 
sqlite3_bind_text, explicitly stating the number of characters (which doesn't 
include a trailing null).  So, I can easily reconstruct std::strings from a 
void * and a number of bytes, without sqlite having to make a copy of it and 
null terminate it for me, if sqlite is doing extra work in my circumstance.

Thanks!
-David

-Original Message-
From: sqlite-users-boun...@sqlite.org [mailto:sqlite-users-boun...@sqlite.org] 
On Behalf Of Richard Hipp
Sent: Sunday, December 15, 2013 5:10 PM
To: General Discussion of SQLite Database
Subject: Re: [sqlite] SQL_STATIC unterminated strings, and sqlite3_result_text

On Sun, Dec 15, 2013 at 6:04 PM, James K. Lowden
wrote:

> http://www.sqlite.org/c3ref/result_blob.html
>
> I found a documentation typo and have a question about SQLITE_STATIC.
>
> The documentation for sqlite3_result_text says,
>
> "If the 3rd parameter is non-negative, then it must be the 
> byte offset into the string where the NUL terminator would appear if 
> the string where NUL terminated."
>
> I believe the intent is subjunctive,
>
> "if the string were NUL terminated".
>
> meaning no NUL is required.  It continues:
>
> "If the 4th parameter to the sqlite3_result_text* interfaces 
> or to sqlite3_result_blob is the special constant SQLITE_STATIC, then 
> SQLite assumes that the text or BLOB result is in constant space and 
> does not copy the content"
>

That statement would be more precise if it read:  "... does not copy the
content RIGHT AWAY..."   If you are inserting into the database, obviously
SQLite needs to copy the content in order to put it on disk.  If you later 
query for the content, then it will copy off of disk again.

If your statements is:

SELECT ?1;

And you bind a string that is not zero-terminated then request the result using 
sqlite3_column_text(), then SQLite will make a copy of the string in order to 
add the zero-terminator.  But, if you request the string using
sqlite3_column_blob() it will not make a copy.  In other words, it delays 
copying the string until it really must, and avoids making a copy if possible.

The application has no idea if and when SQLite might make a copy of the 
SQLITE_STATIC-bound string, so the application must guarantee that the string 
does not change until the statement is finalized or until the same parameter is 
rebound to a different value.



>
> My data are static (a read-only mmap'ed file), and the columns are not 
> null-terminated.
>
> The documentation for the usual column-reading function
> sqlite3_column_text() says it always returns a null-terminated string:
>
> "Strings returned by sqlite3_column_text() and 
> sqlite3_column_text16(), even empty strings, are always 
> zero-terminated."
>
> According to the above, the user gets a null-terminated string from 
> static data in a virtual table that is not copied and need not contain 
> a NUL terminator. That seems unlikely.  I would think either a copy is 
> made or the supplied static data must end in NUL.
>
> If sqlite3_result_text() is provided data with a nonzero length marked 
> as SQLITE_STATIC, where does the NUL come from that is returned to the 
> application by sqlite3_column_text()?
>
> --jkl
> ___
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>



--
D. Richard Hipp
d...@sqlite.org
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] SQL_STATIC unterminated strings, and sqlite3_result_text

[Richard Hipp]

On Sun, Dec 15, 2013 at 6:04 PM, James K. Lowden
wrote:

> http://www.sqlite.org/c3ref/result_blob.html
>
> I found a documentation typo and have a question about SQLITE_STATIC.
>
> The documentation for sqlite3_result_text says,
>
> "If the 3rd parameter is non-negative, then it must be the byte
> offset into the string where the NUL terminator would appear if the
> string where NUL terminated."
>
> I believe the intent is subjunctive,
>
> "if the string were NUL terminated".
>
> meaning no NUL is required.  It continues:
>
> "If the 4th parameter to the sqlite3_result_text* interfaces or
> to sqlite3_result_blob is the special constant SQLITE_STATIC, then
> SQLite assumes that the text or BLOB result is in constant space and
> does not copy the content"
>

That statement would be more precise if it read:  "... does not copy the
content RIGHT AWAY..."   If you are inserting into the database, obviously
SQLite needs to copy the content in order to put it on disk.  If you later
query for the content, then it will copy off of disk again.

If your statements is:

SELECT ?1;

And you bind a string that is not zero-terminated then request the result
using sqlite3_column_text(), then SQLite will make a copy of the string in
order to add the zero-terminator.  But, if you request the string using
sqlite3_column_blob() it will not make a copy.  In other words, it delays
copying the string until it really must, and avoids making a copy if
possible.

The application has no idea if and when SQLite might make a copy of the
SQLITE_STATIC-bound string, so the application must guarantee that the
string does not change until the statement is finalized or until the same
parameter is rebound to a different value.



>
> My data are static (a read-only mmap'ed file), and the columns are not
> null-terminated.
>
> The documentation for the usual column-reading function
> sqlite3_column_text() says it always returns a null-terminated string:
>
> "Strings returned by sqlite3_column_text() and
> sqlite3_column_text16(), even empty strings, are always
> zero-terminated."
>
> According to the above, the user gets a null-terminated string from
> static data in a virtual table that is not copied and need not contain a
> NUL terminator. That seems unlikely.  I would think either a copy is
> made or the supplied static data must end in NUL.
>
> If sqlite3_result_text() is provided data with a nonzero length
> marked as SQLITE_STATIC, where does the NUL come from that is returned
> to the application by sqlite3_column_text()?
>
> --jkl
> ___
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>



-- 
D. Richard Hipp
d...@sqlite.org
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

[sqlite] SQL_STATIC unterminated strings, and sqlite3_result_text

[James K. Lowden]

http://www.sqlite.org/c3ref/result_blob.html

I found a documentation typo and have a question about SQLITE_STATIC. 

The documentation for sqlite3_result_text says, 

"If the 3rd parameter is non-negative, then it must be the byte
offset into the string where the NUL terminator would appear if the
string where NUL terminated." 

I believe the intent is subjunctive, 

"if the string were NUL terminated".  

meaning no NUL is required.  It continues: 

"If the 4th parameter to the sqlite3_result_text* interfaces or
to sqlite3_result_blob is the special constant SQLITE_STATIC, then
SQLite assumes that the text or BLOB result is in constant space and
does not copy the content"

My data are static (a read-only mmap'ed file), and the columns are not
null-terminated.  

The documentation for the usual column-reading function
sqlite3_column_text() says it always returns a null-terminated string: 

"Strings returned by sqlite3_column_text() and
sqlite3_column_text16(), even empty strings, are always
zero-terminated."

According to the above, the user gets a null-terminated string from
static data in a virtual table that is not copied and need not contain a
NUL terminator. That seems unlikely.  I would think either a copy is
made or the supplied static data must end in NUL. 

If sqlite3_result_text() is provided data with a nonzero length
marked as SQLITE_STATIC, where does the NUL come from that is returned
to the application by sqlite3_column_text()? 

--jkl
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] An "unable to open database file" error that has nothing to do with opening database file

[Kees Nuyt]

On Sun, 15 Dec 2013 07:28:52 -0800 (PST), margave  wrote:

>Addendum: I ran "file" on sqlite3.exe ...
>
>On computer B (where sqlite3 works): PE32 executable (console) Intel 80386,
>for MS Windows
>On computer A (where it does not):   PE32+ executable (console) x86-64, for
>MS Windows

Well, something is not completely the same.

I'd start with examining :

* Exact windows version (start/computer/system properties)
* Exact cygwin version
* Package choices in cygwin
* sqlite3 --version
* Security properties of the database file
(roperties / security / advanced / etc.)

Remark: Do you know that concurrent access to sqlite files 
on network shared filesystems is not safe?

-- 
Groet, Cordialement, Pozdrawiam, Regards,

Kees Nuyt

___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] An "unable to open database file" error that has nothing to do with opening database file

[Jan Nijtmans]

2013/12/15 margave :
> I noticed that the sqlite3.exe file on computer A has a different cksum than
> the one on computer B. (Seems odd! The Cygwin installer shows the same
> version on both!)
Which version? SQLite 3.8.2-2?
Apparently, you are running Cygwin on A and
Cygwin64 on B. SQLite is compiled with
exactly the same source-code on Cygwin
and Cygwin64, but with a different compiler.
So this suggests it isn't a problem in SQLite
but either in the compiler (unlikely), ore in
cygwin1.dll. Hard to tell without some more
information.

You could try "sqlite3  -vfs win32-longpath"
which uses the win32 API in stead of the cygwin
API to open the file. Does this make a difference?

Regards,
Jan Nijtmans
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] An "unable to open database file" error that has nothing to do with opening database file

[margave]

Addendum: I ran "file" on sqlite3.exe ...

On computer B (where sqlite3 works): PE32 executable (console) Intel 80386,
for MS Windows
On computer A (where it does not):   PE32+ executable (console) x86-64, for
MS Windows



--
View this message in context: 
http://sqlite.1065341.n5.nabble.com/An-unable-to-open-database-file-error-that-has-nothing-to-do-with-opening-database-file-tp69199p72907.html
Sent from the SQLite mailing list archive at Nabble.com.
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] An "unable to open database file" error that has nothing to do with opening database file

[margave]

I have the same problem. I have .sqlite db files, and I wish to read them
with sqlite3.exe (under Cygwin x64 on Win 7 x64).  I want this to work on
both computer A and computer B. (Both have the same Win 7 and Cygwin
installed).

But it only works on computer B, not on computer A.

I have the SAME .sqlite db file on the desktop on both computers. I run
sqlite3 filename from a Cygwin bash shell (under mintty). As soon as I enter
a sql command (select * from xxx;) I get the "unable to open database file
error".

The two computers have fs mounts on one-another. So I can run sqlite3 on
computer B, referencing the db file on computer A ... and it works, just
like when I reference the local file.

But none of this works when running on computer A. (And ... I tried it from
a CMD prompt, as suggested above. No dice.)

I noticed that the sqlite3.exe file on computer A has a different cksum than
the one on computer B. (Seems odd! The Cygwin installer shows the same
version on both!) Anyway, I copied the working exe from B over to A. And it
doesn't run at all on A.

So I uninstalled sqlite on A and reinstalled. Still no go.

I'm confuzzled.



--
View this message in context: 
http://sqlite.1065341.n5.nabble.com/An-unable-to-open-database-file-error-that-has-nothing-to-do-with-opening-database-file-tp69199p72906.html
Sent from the SQLite mailing list archive at Nabble.com.
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users