Re: [sqlite] how to know which database is corrupted

[J Decker]

On Tue, Jul 24, 2018 at 5:11 PM Simon Slavin  wrote:

> On 24 Jul 2018, at 11:34pm, J Decker  wrote:
>
> > If the system rebooted; did a screen size change, and terminated the
> > program, it's possible it coild cause corruption.
>
> Step 1: use the command-line tool to fix your existing corruption.
>
I don't have the vfs driver as a sqlite3 loadable module; if I did, I don't
know the keys I could know but that's a lot of work.

It could have hapepened because the program was at a breakpoint when the
system ended.

>
> Step 2: prevent more corruption.
>
ya; turns out it'll be a bunch of code that never runs...

Quick question though - do I hae to attempt to keep afinity for values in
an insert?

Insert into * values ( '1234', 'asdf', "0.3422" );
/* where * is a list built from querying select * fields */
Other than haveing to escape anything with ' in it?  (or \0 )



>
> Ignoring the possibbility of bad programming (overwriting SQLite's memory
> or files) my guess is that your corruption is caused by not closing the
> database properly before the computer reboots.
>
> Could you close the database if no changes are made to it for one minute ?
> Reopen it when it's needed again.  Seems like a lot of window position
> changes would be done for a short time, then no more for a long time.
>
> Simon.
> ___
> sqlite-users mailing list
> sqlite-users@mailinglists.sqlite.org
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
>
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

[sqlite] Out-of-bounds read in FTS5 on 3.24.0 and 201807110327 snapshot

[David Yip]

Hi all,

On x86-64 Linux with SQLite 3.24.0 and the 201807110327 SQLite snapshot,
the
following program causes FTS5 to do an out-of-bounds access:
https://gitlab.peach-bun.com/snippets/157

Sample ASan and Valgrind outputs are here:
https://gitlab.peach-bun.com/snippets/158

It looks like if you feed in the byte sequence E3 81 BE E3 82 8A E3 82 84
(the
UTF-8 encoding of まりや), then the loop

while( (p[n] & 0xc0)==0x80 ) n++;

in sqlite3Fts5IndexCharlenToBytelen will attempt to read past the end of
the
string when building 3-character prefixes.

I don't know what (if any) security/stability implications this
out-of-bounds read has, but it is
inconvenient to hit it when statically linking SQLite into an application
has
has ASan enabled (because it'll cause a program abort).

Please let me know if I can provide any additional information that would
help
with a fix.

Thanks,

- David
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] how to know which database is corrupted

[Simon Slavin]

On 24 Jul 2018, at 11:34pm, J Decker  wrote:

> If the system rebooted; did a screen size change, and terminated the
> program, it's possible it coild cause corruption.

Step 1: use the command-line tool to fix your existing corruption.

Step 2: prevent more corruption.

Ignoring the possibbility of bad programming (overwriting SQLite's memory or 
files) my guess is that your corruption is caused by not closing the database 
properly before the computer reboots.

Could you close the database if no changes are made to it for one minute ? 
Reopen it when it's needed again.  Seems like a lot of window position changes 
would be done for a short time, then no more for a long time.

Simon.
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] how to know which database is corrupted

[J Decker]

On Tue, Jul 24, 2018 at 1:47 PM Simon Slavin  wrote:

> On 24 Jul 2018, at 8:43pm, J Decker  wrote:
>
> > I have a database that got corrupted; was working on implementing
> automatic
> > recovery
>
> It would be a million times better to figure out how the corruption occurs
> and prevent it.
>

Well...I've been considering that.   that database is written to when the
window moves (so I restore the position); and during startup.  It's just a
database for program options, which are loaded at the start, created if
they don't exist, given defaults, and can be updated during startup with a
script that gets loaded.

If the system rebooted; did a screen size change, and terminated the
program, it's possible it coild cause corruption.

On tree page 41 cell 37: Rowid 45783 out of order
... 37 to 0
On tree page 41 cell 0: Rowid 45787 out of order
On tree page 6 cell 166: Rowid 45787 out of order
On tree page 42 cell 40: Rowid 45787 out of order
... 40-0
On tree page 42 cell 0: Rowid 45787 out of order
On tree page 6 cell 165: Rowid 45787 out





>
> Simon.
> ___
> sqlite-users mailing list
> sqlite-users@mailinglists.sqlite.org
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
>
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] how to know which database is corrupted

[Simon Slavin]

On 24 Jul 2018, at 8:43pm, J Decker  wrote:

> I have a database that got corrupted; was working on implementing automatic
> recovery

It would be a million times better to figure out how the corruption occurs and 
prevent it.

Simon.
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] how to know which database is corrupted

[Richard Hipp]

On 7/24/18, J Decker  wrote:
>
> I have a callback configured on SQLITE_CONFIG_LOG
> sqlite3_config( SQLITE_CONFIG_LOG, errorLogCallback, 0);
>
> Which is global, and does not identify the instance.  I figured, I could
> just easily register the same callback on the db connection object
>
> sqlite3_db_config( odbc->db, SQLITE_CONFIG_LOG, dbErrorLogCallback, odbc );
>
> But that's not supported per connection.
> So how do I know which connection is executing a statement that failed?

There is no way to do that.  The reason is that these errors are often
dispatched from way down inside the I/O routines which have no
knowledge of which database connection they are servicing.

-- 
D. Richard Hipp
d...@sqlite.org
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

[sqlite] how to know which database is corrupted

[J Decker]

I have a database that got corrupted; was working on implementing automatic
recovery

I have a callback configured on SQLITE_CONFIG_LOG
sqlite3_config( SQLITE_CONFIG_LOG, errorLogCallback, 0);

Which is global, and does not identify the instance.  I figured, I could
just easily register the same callback on the db connection object

sqlite3_db_config( odbc->db, SQLITE_CONFIG_LOG, dbErrorLogCallback, odbc );

But that's not supported per connection.
So how do I know which connection is executing a statement that failed?  (I
could potentially have more than one query active at the same time in
different databases)

Also; should I post that error as an async error?  I mean... in the
callback for the log, a query is outstanding on a statement in the db; so I
suppose I should let that unwind before doing further operations
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

[sqlite] Web page issue

[R Smith]

Problem:

I've come across a small niggle in the sqlite web pages' html tree, 
which may or may not be important, it certainly isn't urgent, but not my 
place to judge what is important and what not, so posting it anyway.
(Details pasted at the bottom of this mail if it merits further looking 
into.)



Description:

There is a innocuous but persistent occurrence in the website source 
HTML pages where it emits an unopened extra closing tag ( to be 
precise). It's near everywhere (everywhere I've checked, but my checking 
was not exhaustive) and in consistent relative placement, which leads me 
to believe the problem to be programmatic or due to a template.


I found this making a parser for web documents (so it can be reformatted 
to a more compact view) to use in an SQLite manager utility.  Now both 
my parser and any other modern browser will swallow the dangling tag 
without complaint, which is why it is innocuous.  I guess the worst 
problem is the slight tickle of an OCD nerve. :)


That said, perhaps it is of interest.


Detail:

This example for the ...sqlite.org/lang_analyze.html (chosen for having 
relatively little html code, but any page could have been chosen).
Problem occurs on line 93 in this file, always after the closing tag of 
the script section (See highlight):


Alternatively, open in modern browser, then open the "View page source" 
- it should be nicely highlighted (Firefox, for instance, marks it red).


"http://www.w3.org/TR/html4/strict.dtd;>name="viewport" content="width=device-width, initial-scale=1.0">http-equiv="content-type" content="text/html; charset=UTF-8">href="sqlite.css" rel="stylesheet">SQLite Query Language: 
ANALYZEhref="index.html">alt="SQLite" border="0">Small. Fast. 
Reliable.Choose any three. Homehref="javascript:void(0)" onclick='toggle_div("submenu")'>Menuclass='wideonly'>Aboutclass='desktoponly'>Documentationclass='desktoponly'>Downloadclass='wideonly'>Licenseclass='desktoponly'>Supportclass='desktoponly'>Purchaseclass='search' id='search_menubutton'>onclick='toggle_search()'>Searchid="submenu">Abouthref='docs.html'>Documentationhref='download.html'>Downloadhref='support.html'>Supporthref='prosupport.html'>Purchaseid="searchmenu">id="searchtype">Search Documentationvalue="c">Search Changelogid="searchbox" value="">value="Go">function toggle_div(nm) { var w = 
document.getElementById(nm); if( w.style.display=="block" ){ 
w.style.display = "none"; }else{ w.style.display = "block"; } } function 
toggle_search() { var w = document.getElementById("searchmenu"); if( 
w.style.display=="block" ){ w.style.display = "none"; } else { 
w.style.display = "block"; setTimeout(function(){ 
document.getElementById("searchbox").focus() }, 30); } } function 
div_off(nm){document.getElementById(nm).style.display="none";} 
window.onbeforeunload = function(e){div_off("submenu");} /* Disable the 
Search feature if we are not operating from CGI, since */ /* Search is 
accomplished using CGI and will not work without it. */ if( 
!location.origin.match || !location.origin.match(/http/) ){ 
document.getElementById("search_menubutton").style.display = "none"; } 
/* Used by the Hide/Show button beside syntax diagrams, to toggle the */ 
function hideorshow(btn,obj){ var x = document.getElementById(obj); var 
b = document.getElementById(btn); if( x.style.display!='none' ){ 
x.style.display = 'none'; b.innerHTML='show'; }else{ x.style.display = 
''; b.innerHTML='hide'; } return false; } *# This 
is the dangling tag -->* align="center">SQL As Understood By SQLitehref="lang.html">[Top]ANALYZEhref="syntax/analyze-stmt.html">analyze-stmt:onclick='hideorshow("x829","x830")'>hideclass='imgcontainer'>src="images/syntax/analyze-stmt.gif" />The ANALYZE command 
gathers statistics about tables and indices and stores the collected 
information in internal 
tablesof the database where the query optimizer can access the 
information and use it to help make better query planning choices. If no 
arguments are given, all attached databases are analyzed. If a schema 
name is given as the argument, then all tables and indices in that one 
database are analyzed. If the argument is a table name, then only that 
table and the indices associated with that table are analyzed. If the 
argument is an index name, then only that one index is 
analyzed.The default implementation stores all statistics in a 
single table named "href="fileformat2.html#stat1tab">sqlite_stat1". If SQLite is 
compiled with the href="compile.html#enable_stat3">SQLITE_ENABLE_STAT3option and 
without the href="compile.html#enable_stat4">SQLITE_ENABLE_STAT4option, then 
additional histogram data is collected and stored in href="fileformat2.html#stat3tab">sqlite_stat3. If SQLite is compiled 
with the href="compile.html#enable_stat4">SQLITE_ENABLE_STAT4option, then 
additional