Re: [sqlite] new Error database disk image is malformed

[Ron Barnes]

I'm coding that now.  I never thought about it.  Thank you!

-Original Message-
From: sqlite-users [mailto:sqlite-users-boun...@mailinglists.sqlite.org] On 
Behalf Of Jens Alfke
Sent: Friday, January 5, 2018 1:34 PM
To: SQLite mailing list <sqlite-users@mailinglists.sqlite.org>
Subject: Re: [sqlite] new Error database disk image is malformed



> On Jan 4, 2018, at 6:29 PM, Peter Da Silva <peter.dasi...@flightaware.com> 
> wrote:
> 
> Since you're I/O bound on socket connections, and not CPU or database bound, 
> you might want to just have one database thread that communicates using 
> native inter-thread messaging to pass out work and accept responses from the 
> worker threads.

+1. Also, this will make it easy to batch multiple updates into a single 
transaction, which greatly improves write performance.

—Jens

___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] new Error database disk image is malformed

[Jens Alfke]

> On Jan 4, 2018, at 6:29 PM, Peter Da Silva  
> wrote:
> 
> Since you're I/O bound on socket connections, and not CPU or database bound, 
> you might want to just have one database thread that communicates using 
> native inter-thread messaging to pass out work and accept responses from the 
> worker threads.

+1. Also, this will make it easy to batch multiple updates into a single 
transaction, which greatly improves write performance.

—Jens

___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] new Error database disk image is malformed

[Peter Da Silva]

> Ok. I've read the Document and I think I may be having a rouge thread issue.  
> I hope not since I use synclock in my code when ever a thread is attempting a 
> write to the database.  That seems like the only issue from that page that I 
> may be doing.  I could have up to 30 or more threads reading from the DB but 
> only one to three active threads writing.  The way my code is set up, though 
> is that each thread would have to wait for the previous thread to finish 
> writing before its turn to write. 

Since you're I/O bound on socket connections, and not CPU or database bound, 
you might want to just have one database thread that communicates using native 
inter-thread messaging to pass out work and accept responses from the worker 
threads.
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] new Error database disk image is malformed

[Richard Damon]

On 1/4/18 8:02 PM, Jens Alfke wrote:



On Jan 4, 2018, at 4:51 PM, Ron Barnes  wrote:

I hope not since I use synclock in my code when ever a thread is attempting a 
write to the database.  That seems like the only issue from that page that I 
may be doing.  I could have up to 30 or more threads reading from the DB but 
only one to three active threads writing.

It depends on the value of SQLITE_THREADSAFE that SQLite was compiled with. 
(Check the docs for details.) If it’s set to 1, you can do what you’re doing. 
Otherwise, you cannot use a single SQLite connection on multiple threads 
without using your own mutex (if it’s 2), or at all (if it’s 0).

30 threads sounds like overkill, BTW. Generally the appropriate number is equal 
to the number of CPU cores. And if the task is I/O-bound there’s not much 
benefit to having more than one since only one of them can be inside SQLite at 
a time.

—Jens
Limiting to number of CPU cores make sense for homogeneous threads 
(multiple threads doing the same sort of work, so make it go in 
parallel). One case where more can make sense is if they are 
heterogeneous, each thread doing some unique task to keep the worker 
threads simple.


--
Richard Damon

___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] new Error database disk image is malformed

[Ron Barnes]

Hi Jens,

Thank you for the response.  
I left it at default so it should be a 1.  The threads are not I/O.  They each 
execute only a few short lines of code to gather network information and then 
hit the DB for a read using the information they just gathered to populate a 
key.  After the read they add the formatted data to a table that is eventually 
displayed to the user.

A quick overview of the code.
I check 64K ports for a response to determine if they are open or not (Nothing 
nefarious).  I use 30+ threads because it could take up to 10 seconds for a 
port to respond while other ports respond immediately.

If you are curious to see the app in action and get a better picture of what 
I'm doing.  It's on the Microsoft store for free.

Do a search of the MS Store for an App named Lo and Behold.

Or click this link..

https://www.microsoft.com/en-us/store/p/lo-and-behold/9nblggh533kc

-Ron

-Original Message-
From: sqlite-users [mailto:sqlite-users-boun...@mailinglists.sqlite.org] On 
Behalf Of Jens Alfke
Sent: Thursday, January 4, 2018 8:03 PM
To: SQLite mailing list <sqlite-users@mailinglists.sqlite.org>
Subject: Re: [sqlite] new Error database disk image is malformed



> On Jan 4, 2018, at 4:51 PM, Ron Barnes <rbar...@njdevils.net> wrote:
> 
> I hope not since I use synclock in my code when ever a thread is attempting a 
> write to the database.  That seems like the only issue from that page that I 
> may be doing.  I could have up to 30 or more threads reading from the DB but 
> only one to three active threads writing.

It depends on the value of SQLITE_THREADSAFE that SQLite was compiled with. 
(Check the docs for details.) If it’s set to 1, you can do what you’re doing. 
Otherwise, you cannot use a single SQLite connection on multiple threads 
without using your own mutex (if it’s 2), or at all (if it’s 0).

30 threads sounds like overkill, BTW. Generally the appropriate number is equal 
to the number of CPU cores. And if the task is I/O-bound there’s not much 
benefit to having more than one since only one of them can be inside SQLite at 
a time.

—Jens
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] new Error database disk image is malformed

[Jens Alfke]

> On Jan 4, 2018, at 4:51 PM, Ron Barnes  wrote:
> 
> I hope not since I use synclock in my code when ever a thread is attempting a 
> write to the database.  That seems like the only issue from that page that I 
> may be doing.  I could have up to 30 or more threads reading from the DB but 
> only one to three active threads writing.

It depends on the value of SQLITE_THREADSAFE that SQLite was compiled with. 
(Check the docs for details.) If it’s set to 1, you can do what you’re doing. 
Otherwise, you cannot use a single SQLite connection on multiple threads 
without using your own mutex (if it’s 2), or at all (if it’s 0).

30 threads sounds like overkill, BTW. Generally the appropriate number is equal 
to the number of CPU cores. And if the task is I/O-bound there’s not much 
benefit to having more than one since only one of them can be inside SQLite at 
a time.

—Jens
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] new Error database disk image is malformed

[Ron Barnes]

Hi,

Ok. I've read the Document and I think I may be having a rouge thread issue.  I 
hope not since I use synclock in my code when ever a thread is attempting a 
write to the database.  That seems like the only issue from that page that I 
may be doing.  I could have up to 30 or more threads reading from the DB but 
only one to three active threads writing.  The way my code is set up, though is 
that each thread would have to wait for the previous thread to finish writing 
before its turn to write.  

Each process works as follows...

Gather information.
Format the information

Enter the write routine using synclock (Visual Basic)

Write the record
Exit the write routine.

I'm using Visual Studio 2015
Visual Basic
Samsung 2TB SSD
64GB RAM
8 Core AMD Processor

Could any of the 30 or so concurrent reads be messing up my writes?

Here is my Create string...

Dim CreateNewDBConString As String = "Data Source=" + Trim(MyPath) + 
"\LoAndBehold.DB3;Version=3;New=True;Max Page 
Count=10485760;Compress=True;journal_mode=WAL;"

Here is my read/write string...

Dim Myconstring As String = "Data Source=" + Trim(MyPath) + 
"\LoAndBehold.DB3;Version=3;New=False;Max Page 
Count=10485760;Compress=True;journal_mode=WAL;"


Regards,

-Ron

-Original Message-
From: sqlite-users [mailto:sqlite-users-boun...@mailinglists.sqlite.org] On 
Behalf Of R Smith
Sent: Thursday, January 4, 2018 3:14 PM
To: sqlite-users@mailinglists.sqlite.org
Subject: Re: [sqlite] new Error database disk image is malformed


On 2018/01/04 9:49 PM, Ron Barnes wrote:
> Hi All,
>
> I keep generating this error and I can't figure out why.  I have deleted and 
> re-created the database but it keeps popping up.
>
> Error
>
> database disk image is malformed
>
>
> Any ideas why?

This is a wild guess, but I'm thinking it's because the disk image is 
malformed...? :)

More seriously, what you probably meant to ask is: "Any idea why my file gets 
corrupt in this way?"

To which the best answer is: No idea, but here are some things that are typical 
culprits:
https://sqlite.org/howtocorrupt.html

Once you've checked all those, and you still are not sure what could cause the 
corruption, please write again but then include all information, OS, storage 
media, file system, use case, etc. and likely someone here would have had 
similar experience or run a similar system (or perhaps see some other obvious 
problem).

Good luck!
Ryan


___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] new Error database disk image is malformed

[Ron Barnes]

Thank you - I will and report back!

-Original Message-
From: sqlite-users [mailto:sqlite-users-boun...@mailinglists.sqlite.org] On 
Behalf Of R Smith
Sent: Thursday, January 4, 2018 3:14 PM
To: sqlite-users@mailinglists.sqlite.org
Subject: Re: [sqlite] new Error database disk image is malformed


On 2018/01/04 9:49 PM, Ron Barnes wrote:
> Hi All,
>
> I keep generating this error and I can't figure out why.  I have deleted and 
> re-created the database but it keeps popping up.
>
> Error
>
> database disk image is malformed
>
>
> Any ideas why?

This is a wild guess, but I'm thinking it's because the disk image is 
malformed...? :)

More seriously, what you probably meant to ask is: "Any idea why my file gets 
corrupt in this way?"

To which the best answer is: No idea, but here are some things that are typical 
culprits:
https://sqlite.org/howtocorrupt.html

Once you've checked all those, and you still are not sure what could cause the 
corruption, please write again but then include all information, OS, storage 
media, file system, use case, etc. and likely someone here would have had 
similar experience or run a similar system (or perhaps see some other obvious 
problem).

Good luck!
Ryan


___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] new Error database disk image is malformed

[Jens Alfke]

(1) Try opening & querying the database file using the `sqlite3` tool, to 
verify whether it’s actually corrupted.
(2) If it is, check the documentation page “How To Corrupt A SQLite Database” 
and see if you’re making any of those mistakes.

—Jens
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] new Error database disk image is malformed

[R Smith]

On 2018/01/04 9:49 PM, Ron Barnes wrote:

Hi All,

I keep generating this error and I can't figure out why.  I have deleted and 
re-created the database but it keeps popping up.

Error

database disk image is malformed


Any ideas why?


This is a wild guess, but I'm thinking it's because the disk image is 
malformed...? :)


More seriously, what you probably meant to ask is: "Any idea why my file 
gets corrupt in this way?"


To which the best answer is: No idea, but here are some things that are 
typical culprits:

https://sqlite.org/howtocorrupt.html

Once you've checked all those, and you still are not sure what could 
cause the corruption, please write again but then include all 
information, OS, storage media, file system, use case, etc. and likely 
someone here would have had similar experience or run a similar system 
(or perhaps see some other obvious problem).


Good luck!
Ryan


___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

[sqlite] new Error database disk image is malformed

[Ron Barnes]

Hi All,

I keep generating this error and I can't figure out why.  I have deleted and 
re-created the database but it keeps popping up.

Error

database disk image is malformed


Any ideas why?

-Ron


___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users