Re: [sqlite] [EXTERNAL] Error in recover sqlite3 database

2019-07-31 Thread Hick Gunter
What is this tbllog table? -Ursprüngliche Nachricht- Von: sqlite-users [mailto:sqlite-users-boun...@mailinglists.sqlite.org] Im Auftrag von bhandari_nikhil Gesendet: Donnerstag, 01. August 2019 07:39 An: sqlite-users@mailinglists.sqlite.org Betreff: [EXTERNAL] [sqlite] Error in recover

[sqlite] Error in recover sqlite3 database

2019-07-31 Thread bhandari_nikhil
My sqlite3 database keeps on increasing in memory (although it is limited by memory). If I try to rebuild the database, it is giving the following error: sqlite> INSERT INTO tbllog(tbllog) VALUES('rebuild'); Error: database or disk is full Is the

Re: [sqlite] Floating point literals

2019-07-31 Thread Simon Slavin
On 1 Aug 2019, at 12:55am, Keith Medcalf wrote: > Columns declared with no affinity behave as if they had been declared with > BLOB infinity and v/v. Okay, so leaving out the affinity just lets it default to BLOB. That clarifies and simplifies things.

Re: [sqlite] Floating point literals

2019-07-31 Thread Keith Medcalf
On Wednesday, 31 July, 2019 17:29, Simon Slavin wrote: >On 31 Jul 2019, at 11:58pm, Keith Medcalf wrote: >> it depends on the application of affinity. If you are storing the >floating point value in a column that does not have an affinity (ie, >no conversions are performed), then it is

Re: [sqlite] Explicit "read transaction" with journal_mode=WAL.

2019-07-31 Thread Jean-Christophe Deschamps
Simon, Consider a bank which takes an audit every day at close-of-business. This might be declared to be 5pm. However, accounts are continued to be debited and credited all night, due to interest being added, ATM transactions, etc.. Nevertheless, the audit needs to see a snapshot as of

Re: [sqlite] Floating point literals

2019-07-31 Thread Simon Slavin
On 31 Jul 2019, at 11:58pm, Keith Medcalf wrote: > it depends on the application of affinity. If you are storing the floating > point value in a column that does not have an affinity (ie, no conversions > are performed), then it is stored exactly (except for NaN). Application of > affinity

Re: [sqlite] Floating point literals

2019-07-31 Thread Donald Shepherd
Thanks, that's an interesting wrinkle that I don't remember being raised in previous discussions but if known it should be mentioned up front as many/most use affinities. Regards, Donald Shepherd. On Thu, 1 Aug 2019 at 08:58, Keith Medcalf wrote: > False, as it depends on the application of

Re: [sqlite] Floating point literals

2019-07-31 Thread Keith Medcalf
False, as it depends on the application of affinity. If you are storing the floating point value in a column that does not have an affinity (ie, no conversions are performed), then it is stored exactly (except for NaN). Application of affinity (ie, real) will cause the -0.0 to be stored as

Re: [sqlite] Floating point literals

2019-07-31 Thread Donald Shepherd
That's not correct, verified several times by my own testing and re-verified on the recent discussion about -0.0 on this mailing list. If you store -0.0 as a double, it will be stored as an integer as a space-saving mechanism. That integer is 0. When you retrieve the value as a double it will

Re: [sqlite] Floating point literals

2019-07-31 Thread Keith Medcalf
The -0.0 is only for conversion to text. Otherwise -0.0 is preserved both on input and output (including input text conversions). It is only the conversion of -0.0 TO text that drops the sign. NaN becomes a NULL (ie, a double is not stored, a NULL value is stored). Everything else is

Re: [sqlite] Floating point literals

2019-07-31 Thread Donald Shepherd
Plus (as Igor noted) -0.0 returns as 0.0. Regards, Donald Shepherd. On Thu, 1 Aug 2019 at 08:41, Keith Medcalf wrote: > > sqlite3_bind_double and sqlite3_column_double will round trip IEEE > floating point values EXCEPT for NaN. NaN will be stored as a NULL. > > -- > The fact that there's a

Re: [sqlite] Floating point literals

2019-07-31 Thread Keith Medcalf
sqlite3_bind_double and sqlite3_column_double will round trip IEEE floating point values EXCEPT for NaN. NaN will be stored as a NULL. -- The fact that there's a Highway to Hell but only a Stairway to Heaven says a lot about anticipated traffic volume. >-Original Message- >From:

Re: [sqlite] Quick way to determine optimal page size?

2019-07-31 Thread Tony Papadimitriou
Tens of databases (from a few MBs to almost GB), so it's good to keep them at their minimum size (for disk and backup savings). I often save several megabytes by going to the 'right' size, eg., just today I went from ~110MB down to ~80MB in one of them ('vacuum'ed before and after so it's

Re: [sqlite] Explicit "read transaction" with journal_mode=WAL.

2019-07-31 Thread test user
Quote: Connection 2 just happened to write lots of data and commit before connection 1 obtained a read transaction Quote: if SELECT on Connection 1 just happens to beat the write on Connection 2 - The order in the example is exact, not a guess what might happen. - Each step runs in a single

Re: [sqlite] Floating point literals

2019-07-31 Thread Igor Tandetnik
On 7/31/2019 5:15 PM, Eric Reischer wrote: I understand you can *retrieve* a non-quantized value using sqlite3_column_double(), but I don't see a way to set one without having to printf() the floating point value. sqlite3_bind_double Can this be done using sqlite3_bind_* interfaces, or do

Re: [sqlite] Floating point literals

2019-07-31 Thread Simon Slavin
On 31 Jul 2019, at 10:15pm, Eric Reischer wrote: > Is there a way to pass binary representations of floating point numbers to a > SQL query? If sqlite's internal representation of floating point numbers is > 8-byte IEEE doubles, it would be convenient to be able to pass the literal > value

Re: [sqlite] [SPAM?] Re: Explicit "read transaction" with journal_mode=WAL.

2019-07-31 Thread Richard Damon
If, and it is possible for there to be sequencing to enforce it, You KNOW that step 1, get a read transaction (which the OP presumes implies includes getting the lock) occurs before step 2, and thus step 4 seeing changes from step 3 says something is wrong. Yes, if you only can use the

[sqlite] Floating point literals

2019-07-31 Thread Eric Reischer
Is there a way to pass binary representations of floating point numbers to a SQL query? If sqlite's internal representation of floating point numbers is 8-byte IEEE doubles, it would be convenient to be able to pass the literal value of a float or double to the underlying SQL parser without

Re: [sqlite] Explicit "read transaction" with journal_mode=WAL.

2019-07-31 Thread Igor Tandetnik
On 7/31/2019 12:32 PM, test user wrote: In some runtimes, the scheduling of functions is unpredictable, so although you will not have a `sleep 5` in the code, the runtime can produce this effect on loaded systems or with programs with long running sync functions. An example of how you might

Re: [sqlite] Quick way to determine optimal page size?

2019-07-31 Thread Simon Slavin
On 31 Jul 2019, at 8:28pm, Tony Papadimitriou wrote: > Instead of brute force “pragma page_size=xxx; vacuum;” for each page size and > each database to determine which one produces the smallest file, is there > some quicker way? It might be faster to make a new file, set page size, ATTACH

Re: [sqlite] Quick way to determine optimal page size?

2019-07-31 Thread David Raymond
Not that I'm aware of no. How much of a difference are you seeing for your database size depending on the page size you try? -Original Message- From: sqlite-users On Behalf Of Tony Papadimitriou Sent: Wednesday, July 31, 2019 3:29 PM To: General Discussion of SQLite Database Subject:

[sqlite] Quick way to determine optimal page size?

2019-07-31 Thread Tony Papadimitriou
Instead of brute force “pragma page_size=xxx; vacuum;” for each page size and each database to determine which one produces the smallest file, is there some quicker way? Thanks. ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org

[sqlite] Double CTRL-C in shell get you completely out

2019-07-31 Thread Tony Papadimitriou
Recently CTRL-C was improved to abort the query and stay in the CLI. This is very good. However, if (accidentally, e.g., key bounce) a second CTRL-C is entered, it will escape back to the console. Could it be changed so that either: 1. Only CTRL-D (Linux) or CTRL-Z (Windows) is used to exit

Re: [sqlite] Explicit "read transaction" with journal_mode=WAL.

2019-07-31 Thread Olivier Mascia
> Le 31 juil. 2019 à 18:53, Keith Medcalf a écrit : > > I believe the idea would be to permit syntax something like: > > BEGIN IMMEDIATE [SHARED|[UPDATE]] [TRANSACTION] Keith, I mostly share your view and I like the fact that the proposal uses SHARED and not READ or anything like that because

Re: [sqlite] Explicit "read transaction" with journal_mode=WAL.

2019-07-31 Thread Richard Hipp
On 7/31/19, Thomas Kurz wrote: > Would it be possible for you to give some feedback (just an estimation) > whether or not a suggestion might be considered? Low probability at this time. The suggestion does not provide any new capability, but it is something that we would need to test and

Re: [sqlite] Explicit "read transaction" with journal_mode=WAL.

2019-07-31 Thread Keith Medcalf
On Wednesday, 31 July, 2019 10:21, Simon Slavin wrote: >But I think the BEGIN command can be retrofitted without breaking >backward compatibility. The words WRITE and IMMEDIATE should be seen >as options. WRITE means that you want a write lock as well as a read >lock. IMMEDIATE means that you

Re: [sqlite] Explicit "read transaction" with journal_mode=WAL.

2019-07-31 Thread test user
Quote: Why would that distinction matter to it in the first place? So its clear in the API what lock you have at what time. This would make predicting what happens in concurrent scenarios much easier. An explicit "read transaction" is a single line of a program. With an implicit "read

Re: [sqlite] [SPAM?] Re: Explicit "read transaction" with journal_mode=WAL.

2019-07-31 Thread Thomas Kurz
Would it be possible for you to give some feedback (just an estimation) whether or not a suggestion might be considered? - Original Message - From: Richard Hipp To: SQLite mailing list Sent: Wednesday, July 31, 2019, 16:10:13 Subject: [sqlite] [SPAM?] Re: Explicit "read transaction"

Re: [sqlite] Explicit "read transaction" with journal_mode=WAL.

2019-07-31 Thread Simon Slavin
On 31 Jul 2019, at 5:04pm, Larry Brasfield wrote: > I do not personally see the benefit of moving the repeatable read guarantee > to the BEGIN point rather than the first database read after the BEGIN > because only fully committed transactions will be visible anyway -- … I can imagine two

Re: [sqlite] Explicit "read transaction" with journal_mode=WAL.

2019-07-31 Thread Larry Brasfield
Keith writes: I do not personally see the benefit of moving the repeatable read guarantee to the BEGIN point rather than the first database read after the BEGIN because only fully committed transactions will be visible anyway -- … Improved modularity would be facilitated by making it possible

Re: [sqlite] [EXTERNAL] sqlite3_exec without ubiqitous text conversions

2019-07-31 Thread James K. Lowden
On Wed, 31 Jul 2019 12:05:05 +0200 Olivier Mascia wrote: > Nothing stops any piece of your own programming or anyone using the > CLI to do: > > INSERT INTO "VALUES"(value_int) VALUES('something'); Not nothing, just nothing automatic. value_int INTEGER not NULL

Re: [sqlite] Explicit "read transaction" with journal_mode=WAL.

2019-07-31 Thread Igor Tandetnik
On 7/31/2019 6:36 AM, test user wrote: As an example, the client of this library could: - A. Obtain a "read transaction", *without running any SELECTs*. - B. Complete 20 write transactions in another process. - C. Begin reading from the read transaction (A) at the point before the transactions

Re: [sqlite] [SPAM?] Re: Explicit "read transaction" with journal_mode=WAL.

2019-07-31 Thread Richard Hipp
On 7/31/19, Simon Slavin wrote: > On 31 Jul 2019, at 12:57pm, test user wrote: > >> Is there a standard place where people can request features to be added to >> SQLite? > > Here. You've already done it. The developers of SQLite read this list and > will consider the things you wanted to do

Re: [sqlite] [SPAM?] Re: Explicit "read transaction" with journal_mode=WAL.

2019-07-31 Thread Simon Slavin
On 31 Jul 2019, at 12:57pm, test user wrote: > Is there a standard place where people can request features to be added to > SQLite? Here. You've already done it. The developers of SQLite read this list and will consider the things you wanted to do and whether it's worth providing a better

Re: [sqlite] [EXTERNAL] storing blobs in a separate table

2019-07-31 Thread Hick Gunter
SQLite stores rows in a compressed format that requires decoding. To access the nth field, all the fields that come before it need to be decoded. If there is a large blob stoed in a blob field, any field after that will suffer a performance penalty (unless, of course, both fields are required).

Re: [sqlite] [SPAM?] storing blobs in a separate table

2019-07-31 Thread Richard Damon
On 7/31/19 7:49 AM, Rael Bauer wrote: > Hi, > > I am aware that in sqlite for a given "entity" one should generally > store blobs in a separate table to the other standard fields (such as > text/number etc..) > > So for maximum efficiency, in such a blob table, is there a problem > storing

Re: [sqlite] [SPAM?] Re: Explicit "read transaction" with journal_mode=WAL.

2019-07-31 Thread test user
Yeh I think this is the way to go currently. I just wanted to be sure I understood it correctly and that it doesn’t exist already. Is there a standard place where people can request features to be added to SQLite? The Fossil repo perhaps? On Wed, 31 Jul 2019 at 12:53, Richard Damon wrote: >

Re: [sqlite] [SPAM?] Re: Explicit "read transaction" with journal_mode=WAL.

2019-07-31 Thread Richard Damon
As has been pointed out, your function that is called can do the BEGIN and then a SELECT that hits the database to force the obtaining of the read lock. The fact that the BEGIN didn't get the lock is then not seen by the users of your API. IF at some point a new BEGIN SHARED IMMEDIATE operation

[sqlite] storing blobs in a separate table

2019-07-31 Thread Rael Bauer
Hi, I am aware that in sqlite for a given "entity" one should generally store blobs in a separate table to the other standard fields (such as text/number etc..) So for maximum efficiency, in such a blob table, is there a problem storing multiple blob fields? E.g.  data1, data2, picture,

Re: [sqlite] Explicit "read transaction" with journal_mode=WAL.

2019-07-31 Thread Simon Slavin
On 31 Jul 2019, at 11:36am, test user wrote: > BEGIN IMMEDIATE forces a "write transaction", but there is no counterpart for > a "read transaction". The difference is that the connection can write as well as read. So it is a lock for reading. ___

Re: [sqlite] Explicit "read transaction" with journal_mode=WAL.

2019-07-31 Thread test user
Quote: What importance does it have for you that it already holds an "end-mark? Quote: Why would it matter that a writer did write and commit between the "reader" BEGIN and its first read? Im writing a library and would like to have an API where the "read transaction" has a clear beginning in

Re: [sqlite] [EXTERNAL] sqlite3_exec without ubiqitous text conversions

2019-07-31 Thread Barone Ashura
Thanks! this helped me a lot in understanding the red flags. I think I have read the 'type affinity' documentation pages quite a few times (including this morning). Il giorno mer 31 lug 2019 alle ore 12:05 Olivier Mascia ha scritto: > > Le 31 juil. 2019 à 10:03, Barone Ashura a > écrit : > >

Re: [sqlite] [EXTERNAL] sqlite3_exec without ubiqitous text conversions

2019-07-31 Thread Olivier Mascia
> Le 31 juil. 2019 à 10:03, Barone Ashura a écrit : > > CREATE TABLE "VALUES" ( > id INTEGER PRIMARY KEY ASC, > value_int INTEGER, > value_double REAL, > value_text TEXT > ); > > and that I want to execute the following simple query: > > SELECT * FROM "VALUES"; > > For this very specific

Re: [sqlite] sqlite3_exec without ubiqitous text conversions

2019-07-31 Thread Barone Ashura
Suggestion Acknowledged :) Il giorno mar 30 lug 2019 alle ore 18:56 ha scritto: > Hi! > > I use a simple trick: > > A clone of the sqlite3_exec that passes the sqlite3_stmt as an argument > to the callback function. > > So no conversion is made and we can use the sqlite3_column... functions >

Re: [sqlite] [EXTERNAL] sqlite3_exec without ubiqitous text conversions

2019-07-31 Thread Barone Ashura
> Why do it this way? Why not write your own custom_sqlite3_exec(...) that uses the standard, > stable, documented interfaces? Because sometimes I take wrong turns despite my best intentions :) But I usually smell it, when I am taking wrong turns, and that is why I can up here asking for

Re: [sqlite] [EXTERNAL] sqlite3_exec without ubiqitous text conversions

2019-07-31 Thread Hick Gunter
Sorry if I came across impolite or destructive. The way you are attempting to do what you feel you need to raises a whole bunch of red flags, which I have been trying to explain, based on 40+ years of coding experience from assembler upwards (including COBOL, FORTRAN, BASIC, FORTH, PASCAL, C,