Re: [sqlite] SQLITE Header access

[Dan Kennedy]

On 05/04/2012 12:52 AM, Igor Tandetnik wrote:

On 5/3/2012 1:42 PM, msantha wrote:

I am using sqlite in my application only for read access.


But someone else might open and modify the same database. You may know
this doesn't happen, but SQLite doesn't.


The DB gets hit
often by my application and I could see that the header(100 bytes) of the
database is read every time when i access the database.


Most likely, checking the schema cookie, to confirm that the database
schema remains unchanged and prepared statements are still valid.


It's checking if the "change-counter", a field updated each time the
database is written, has changed. If the change-counter has changed
since the last time the db was written, all cached pages are discarded
from the cache and SQLite moves on to checking the schema cookie.


can we make it read it only once?


Try starting a transaction at the beginning, and keeping it open the
whole time the application runs.


You could also use "PRAGMA locking_mode=EXCLUSIVE". That way, SQLite
could be as confident as you are that no other process is modifying
the db. And it won't feel the need to inspect the db header each
time you read.

___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] is SQLite the right tool to analyze a 44GB file

[Oliver Peters]

Am 03.05.2012 19:59, schrieb peter korinis:

I have R but really haven't used it much. I know it's a great stats package
and great for data reduction ... but I want to perform queries against my
44GB of data, filtering records by a variety of attributes, comparing those
subsets in a variety of ad hoc ways, perhaps summing/counting other fields,
etc.


I prefer

http://www.yunqa.de/delphi/doku.php/products/sqlitespy/index

for creating the database (tables, views) and doing the queries cause 
it's fast and reliable but I prefer to write(!) SQL code and not to 
create it through a generator (as it is done in Access).


sqlitespy can't do the import job; I always do this with the CLI by 
creating INSERT statements with my scripting language in a separate file


since sqlite 3.7.11 you don't need a statement like

INSERT INTO table(col01,col02,col03) VALUES(1,2,3);
INSERT INTO table(col01,col02,col03) VALUES(4,5,6);

you can make it shorter:

INSERT INTO table(col01,col02,col03) VALUES(1,2,3),(4,5,6);

this is a great advantage if you need to do many INSERTs cause your file 
won't become so large




This is the kind of job excel is good at ... but the data is too bit!
Seems like a database plus a good query GUI or some BI app would work. is R
a good query tool?


afaik there is no other way than to write (!) SQL Code - depending on 
the problem this can be done in an R script or directly in the database 
(i.e. as a VIEW) or as a combination


[...]

Oliver
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] SQlite exclusive table lock

[Pavel Ivanov]

> So currently from .net code I just say BeginTransaction(ReadCommitted) and do 
> a write operation this locks the complete database file and the write 
> operations are really long. So currently I want to solve the
> locking issue by bringing them down to table locks.

To solve locking issue in SQLite you have to use WAL journal mode.

If you think you absolutely need to use table locks (e.g. to allow
several parallel writers to different tables), you have to use other
DBMS.

But even in other DBMS you will have hard time trying to force readers
to wait while writer transaction is in progress. For that sort of
behavior you need to use your programming language.


Pavel


On Thu, May 3, 2012 at 3:29 PM, Harnek Manj  wrote:
> Hi Simon,
>
> Yes I have multiple Threads which are accessing the database. Currently if I 
> am doing a write operation the whole database file is locked, I want the 
> locking applied only to the table in operation.
>
> So currently from .net code I just say BeginTransaction(ReadCommitted) and do 
> a write operation this locks the complete database file and the write 
> operations are really long. So currently I want to solve the
> locking issue by bringing them down to table locks.
>
> You mentioned "BEGIN EXCLUSIVE TRANSACTION", but I don't see where I can set, 
> that I want to start a transaction with exclusive lock, when I call 
> BeginTransaction on the connection.
>
> Thanks
> Harnek
>
> -Original Message-
> From: sqlite-users-boun...@sqlite.org 
> [mailto:sqlite-users-boun...@sqlite.org] On Behalf Of Simon Slavin
> Sent: May-02-12 5:28 PM
> To: General Discussion of SQLite Database
> Subject: Re: [sqlite] SQlite exclusive table lock
>
>
> On 3 May 2012, at 1:03am, Harnek Manj  wrote:
>
>> So does it mean that in SQlite there is no way to stop the read operation 
>> while a write operation is running, like table level exclusive lock.
>
> You seem to have jumped straight over the basic features of SQL and looked at 
> some of the most advanced and complicated features.  When you want to block 
> everything else use
>
> BEGIN EXCLUSIVE TRANSACTION
>
> See
>
> 
>
> However, I don't understand why you would want to block a read operation.  
> The difference can matter only when you have multiple threads or processes 
> running at once.  And if you're doing that then there's no harm in getting 
> the data as it was before the write happened.  If the first operation had to 
> finish before the second started, who wouldn't you be doing both operations 
> in the same thread of the same process ?
>
> Simon.
> ___
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>
> ___
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] SQlite exclusive table lock

[Simon Davies]

On 3 May 2012 20:29, Harnek Manj  wrote:
> Hi Simon,
>
> Yes I have multiple Threads which are accessing the database. Currently if I 
> am doing a write operation the whole database file is locked, I want the 
> locking applied only to the table in operation.
>
> So currently from .net code I just say BeginTransaction(ReadCommitted) and do 
> a write operation this locks the complete database file and the write 
> operations are really long. So currently I want to solve the
> locking issue by bringing them down to table locks.

http://www.sqlite.org/whentouse.html
look at last paragraph - SQLite locks the whole database file.

>
> You mentioned "BEGIN EXCLUSIVE TRANSACTION", but I don't see where I can set, 
> that I want to start a transaction with exclusive lock, when I call 
> BeginTransaction on the connection.
>
> Thanks
> Harnek
>

Regards,
Simon
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Serialized access faster than multithreaded?

[Simon Slavin]

On 3 May 2012, at 10:30pm, Igor Tandetnik  wrote:

> On 5/3/2012 5:04 PM, Marco Era wrote:
>> Now that I have some time, I'm stress testing it to see how much I can get 
>> from it; what I want to check is its performance in a multithreading 
>> environment.
>> 
>> To my surprise, it seems that serialized access to the database (which is 
>> the default in the source code for Windows) is ~20% faster than 
>> multithreading (no matter how the shared cache option is set).
> 
> Multithreading helps CPU-bound applications improve performance by taking 
> advantage of multiple cores. But SQLite is I/O-bound most of the time. Your 
> hard drive only has one set of heads: trying to pull them in four directions 
> at once just leads to excessive seek activity, hurting performance.

If you have one available, you might try the same test with a computer which 
has a solid state drive (SSD) instead of a rotating drive.  Your bottleneck is 
still the drive itself, but you no longer spend most of your time waiting for 
the disk to rotate to the right place to read/write the sector you want.  It's 
this 'rotational latency' which kills conventional hard disk speed.  I'd be 
interested in seeing results from anyone who could run such a comparison.

Simon.
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Serialized access faster than multithreaded?

[Pavel Ivanov]

> Serialized test (SQLITE_THREADSAFE=1):
>
> The main thread opens the database, sets the cache and busy timeout to huge 
> values and passes the db descriptor to the threads.
>
> Multithreaded test (SQLITE_THREADSAFE=2):
> The main thread creates the other threads and wait; each thread opens its own 
> database connection and sets the cache and timeout (using a shared cache 
> required some changes to handle the LOCKED return codes, but didn't give 
> better results).
>
> Is this the expected behavior or am I missing something?

Yes, you made completely different tests. In first case everybody used
the same connection, thus all worked in the same transaction space.
I.e. transaction was committed only when all threads finalized or
reset their statements. In a stress test environment I bet that was a
very rare occasion, most of the time some thread still had its
statement unfinished. So in the serialized test SQLite had less
locking/unlocking operations to do and thus it worked faster.


Pavel


On Thu, May 3, 2012 at 5:04 PM, Marco Era  wrote:
> Hi there,
>
> I've been using sqlite for a few years now and it worked fine in my 
> experience.
>
> Now that I have some time, I'm stress testing it to see how much I can get 
> from it; what I want to check is its performance in a multithreading 
> environment.
>
> To my surprise, it seems that serialized access to the database (which is the 
> default in the source code for Windows) is ~20% faster than multithreading 
> (no matter how the shared cache option is set).
>
>
>
> My test program creates 4 threads that do some work on a single table of a 
> ~200MB WAL-based database.
>
> First two threads do only SELECTs; third thread does UPDATEs; fourth thread 
> does INSERTs; the SQL commands are not wrapped into explicit transactions.
>
>
> Serialized test (SQLITE_THREADSAFE=1):
>
> The main thread opens the database, sets the cache and busy timeout to huge 
> values and passes the db descriptor to the threads.
>
>
> Multithreaded test (SQLITE_THREADSAFE=2):
> The main thread creates the other threads and wait; each thread opens its own 
> database connection and sets the cache and timeout (using a shared cache 
> required some changes to handle the LOCKED return codes, but didn't give 
> better results).
>
>
>
> Is this the expected behavior or am I missing something?
> Thanks in advance.
>
> Marco
> ___
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Serialized access faster than multithreaded?

[Igor Tandetnik]

On 5/3/2012 5:04 PM, Marco Era wrote:

Now that I have some time, I'm stress testing it to see how much I can get from 
it; what I want to check is its performance in a multithreading environment.

To my surprise, it seems that serialized access to the database (which is the 
default in the source code for Windows) is ~20% faster than multithreading (no 
matter how the shared cache option is set).


Multithreading helps CPU-bound applications improve performance by 
taking advantage of multiple cores. But SQLite is I/O-bound most of the 
time. Your hard drive only has one set of heads: trying to pull them in 
four directions at once just leads to excessive seek activity, hurting 
performance.

--
Igor Tandetnik

___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

[sqlite] Serialized access faster than multithreaded?

[Marco Era]

Hi there,

I've been using sqlite for a few years now and it worked fine in my experience.

Now that I have some time, I'm stress testing it to see how much I can get from 
it; what I want to check is its performance in a multithreading environment.

To my surprise, it seems that serialized access to the database (which is the 
default in the source code for Windows) is ~20% faster than multithreading (no 
matter how the shared cache option is set). 



My test program creates 4 threads that do some work on a single table of a 
~200MB WAL-based database. 

First two threads do only SELECTs; third thread does UPDATEs; fourth thread 
does INSERTs; the SQL commands are not wrapped into explicit transactions.


Serialized test (SQLITE_THREADSAFE=1):

The main thread opens the database, sets the cache and busy timeout to huge 
values and passes the db descriptor to the threads.


Multithreaded test (SQLITE_THREADSAFE=2):
The main thread creates the other threads and wait; each thread opens its own 
database connection and sets the cache and timeout (using a shared cache 
required some changes to handle the LOCKED return codes, but didn't give better 
results).



Is this the expected behavior or am I missing something?
Thanks in advance.

Marco
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] is SQLite the right tool to analyze a 44GB file

[Warren Young]

On 5/3/2012 11:59 AM, peter korinis wrote:

is R a good query tool?


It's a programming language.  It can do anything within your power to 
persuade the interpreter.


One of the fundamental data types in R is the data frame, which is 
roughly equivalent to a SQLite table.


This is an R equivalent to "SELECT * FROM MYTABLE WHERE V2 < 9":

results <- subset(my.table, V2 < 9)

But, this is not the place for an R tutorial.  Take the question up on 
an R mailing list if you want to learn more of its capabilities.

___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] is SQLite the right tool to analyze a 44GB file

[Freddy López]

As said Oliver, I don't think the real issue is have to choose one
software: or SQLite or R.

I've used SQLite with data around 10GB under Windows 7 with 4GB of RAM and
it worked perfectly. Yes, it size is less than yours but I learned that the
use of every GUI was a problem.

Since that, I always work simply with sqlite3.exe and other tools such as
SCITE to write queries.

Cheers.


On Thu, May 3, 2012 at 1:29 PM, peter korinis  wrote:

> I have R but really haven't used it much. I know it's a great stats package
> and great for data reduction ... but I want to perform queries against my
> 44GB of data, filtering records by a variety of attributes, comparing those
> subsets in a variety of ad hoc ways, perhaps summing/counting other fields,
> etc.
> This is the kind of job excel is good at ... but the data is too bit!
> Seems like a database plus a good query GUI or some BI app would work. is R
> a good query tool?
>
> Thanks,
> peter
>
>
> -Original Message-
> From: sqlite-users-boun...@sqlite.org
> [mailto:sqlite-users-boun...@sqlite.org] On Behalf Of Warren Young
> Sent: Thursday, May 03, 2012 9:36 AM
> To: General Discussion of SQLite Database
>  Subject: Re: [sqlite] is SQLite the right tool to analyze a 44GB file
>
>  On 5/1/2012 2:06 PM, peter korinis wrote:
> > Is SQLite the wrong tool for this project?
>
> Probably.
>
> SQLite is a data storage tool.  With enough SQL cleverness, you can turn it
> into a data *reduction* tool.  But a data analysis tool?  No, not without
> marrying it to a real programming language.
>
> Granted, that's what almost everyone does do with SQLite, but if you're
> going to learn a programming language, I'd recommend you learn R, a
> language
> and environment made for the sort of problem you find yourself stuck with.
> http://r-project.org/
>
> There are several R GUIs out there.  I like R Studio best:
> http://www.rstudio.org/
>
> You'll still find R Studio a sharp shock compared to Excel.  And yes, it
> will require some programming, and yes, I know you said you aren't a
> programmer.  But in the rest of the thread, it looks like people have
> convinced you to use SQLite from the command line, typing in raw SQL
> commands; guess what, that's programming.  Not on the level of R code, but
> R
> isn't far down the same slippery slope.
>
> It may help you to know that R is most popular in the statistics community,
> which of course is populated by statisticians, not programmers.
>
> R isn't the easiest programming language to pick up, but it's far from the
> hardest.  It's very similar to JavaScript, though a bit tougher to learn,
> mostly due to having accumulated some strange syntax over its 36 years.
> (That's counting R's predecessor, S.)
> ___
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>
> ___
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>



-- 
«But Gwindor answered: 'The doom lies in yourself, not in your name.'»

JRR Tolkien
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] SQlite exclusive table lock

[Harnek Manj]

Hi Simon,

Yes I have multiple Threads which are accessing the database. Currently if I am 
doing a write operation the whole database file is locked, I want the locking 
applied only to the table in operation.

So currently from .net code I just say BeginTransaction(ReadCommitted) and do a 
write operation this locks the complete database file and the write operations 
are really long. So currently I want to solve the
locking issue by bringing them down to table locks.

You mentioned "BEGIN EXCLUSIVE TRANSACTION", but I don't see where I can set, 
that I want to start a transaction with exclusive lock, when I call 
BeginTransaction on the connection.

Thanks
Harnek

-Original Message-
From: sqlite-users-boun...@sqlite.org [mailto:sqlite-users-boun...@sqlite.org] 
On Behalf Of Simon Slavin
Sent: May-02-12 5:28 PM
To: General Discussion of SQLite Database
Subject: Re: [sqlite] SQlite exclusive table lock


On 3 May 2012, at 1:03am, Harnek Manj  wrote:

> So does it mean that in SQlite there is no way to stop the read operation 
> while a write operation is running, like table level exclusive lock.

You seem to have jumped straight over the basic features of SQL and looked at 
some of the most advanced and complicated features.  When you want to block 
everything else use

BEGIN EXCLUSIVE TRANSACTION

See



However, I don't understand why you would want to block a read operation.  The 
difference can matter only when you have multiple threads or processes running 
at once.  And if you're doing that then there's no harm in getting the data as 
it was before the write happened.  If the first operation had to finish before 
the second started, who wouldn't you be doing both operations in the same 
thread of the same process ?

Simon.
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] SQLITE Header access

[Kees Nuyt]

On Thu, 3 May 2012 10:42:38 -0700 (PDT), msantha 
wrote:

>
> I am using sqlite in my application only for read access. The DB gets hit
> often by my application and I could see that the header(100 bytes) of the
> database is read every time when i access the database.
> Precisely speaking, 16 bytes from the 24th byte of the header is read
> everytime. My question is , if the database is used only for read purpose,
> why the header is read everytime as the database connection is not
> closed?..

Your process may promise that it will only read the database, but there
might be some other process writing to it.
Not being a server, sqlite has no other way to find that out than by
reading the header over and over again. It has to check whether the
schema was changed, or whatever other info is in those bytes.

> can we make it read it only once?

I don't think so, unless you patch the source code.
I wouldn't worry about it too much, the operating system will notice
that particular page is read often and keep it in it's file system
cache, so access is fast.

> Thanks!!

Hope this helps.

-- 
Regards,

Kees Nuyt

___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Data Import Techniques

[peter korinis]

Forgive me ... but what is SQLiteman?
Will "import table data" help me load a csv file into a SQLite table?

Thanks,
peter

-Original Message-
From: sqlite-users-boun...@sqlite.org
[mailto:sqlite-users-boun...@sqlite.org] On Behalf Of Simon
Sent: Wednesday, May 02, 2012 3:03 PM
To: General Discussion of SQLite Database
Subject: Re: [sqlite] Data Import Techniques

> Using the "Import Table Data" function in SQLiteman, the data loads very
quickly. However in my application, using either an SQL "insert" command or
a resultset, the import is very much slower. Is there another technique I
can use to speed things up?

This FAQ entry might interest you...

(19) INSERT is really slow - I can only do few dozen INSERTs per second
  ->  http://sqlite.org/faq.html#q19
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] is SQLite the right tool to analyze a 44GB file

[peter korinis]

I have R but really haven't used it much. I know it's a great stats package
and great for data reduction ... but I want to perform queries against my
44GB of data, filtering records by a variety of attributes, comparing those
subsets in a variety of ad hoc ways, perhaps summing/counting other fields,
etc. 
This is the kind of job excel is good at ... but the data is too bit!
Seems like a database plus a good query GUI or some BI app would work. is R
a good query tool?

Thanks,
peter


-Original Message-
From: sqlite-users-boun...@sqlite.org
[mailto:sqlite-users-boun...@sqlite.org] On Behalf Of Warren Young
Sent: Thursday, May 03, 2012 9:36 AM
To: General Discussion of SQLite Database
Subject: Re: [sqlite] is SQLite the right tool to analyze a 44GB file

On 5/1/2012 2:06 PM, peter korinis wrote:
> Is SQLite the wrong tool for this project?

Probably.

SQLite is a data storage tool.  With enough SQL cleverness, you can turn it
into a data *reduction* tool.  But a data analysis tool?  No, not without
marrying it to a real programming language.

Granted, that's what almost everyone does do with SQLite, but if you're
going to learn a programming language, I'd recommend you learn R, a language
and environment made for the sort of problem you find yourself stuck with.
http://r-project.org/

There are several R GUIs out there.  I like R Studio best: 
http://www.rstudio.org/

You'll still find R Studio a sharp shock compared to Excel.  And yes, it
will require some programming, and yes, I know you said you aren't a
programmer.  But in the rest of the thread, it looks like people have
convinced you to use SQLite from the command line, typing in raw SQL
commands; guess what, that's programming.  Not on the level of R code, but R
isn't far down the same slippery slope.

It may help you to know that R is most popular in the statistics community,
which of course is populated by statisticians, not programmers.

R isn't the easiest programming language to pick up, but it's far from the
hardest.  It's very similar to JavaScript, though a bit tougher to learn,
mostly due to having accumulated some strange syntax over its 36 years.
(That's counting R's predecessor, S.)
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] SQLITE Header access

[Pavel Ivanov]

> My question is , if the database is used only for read purpose,
> why the header is read everytime as the database connection is not
> closed?..can we make it read it only once?

Even though your connection is used only for reading, some other
connection (maybe in a different process) can be used for writing.
Thus SQLite have to check before each transaction if anything was
changed in the database.


Pavel


On Thu, May 3, 2012 at 1:42 PM, msantha  wrote:
>
> I am using sqlite in my application only for read access. The DB gets hit
> often by my application and I could see that the header(100 bytes) of the
> database is read every time when i access the database.
> Precisely speaking, 16 bytes from the 24th byte of the header is read
> everytime. My question is , if the database is used only for read purpose,
> why the header is read everytime as the database connection is not
> closed?..can we make it read it only once?
>
> Thanks!!
>
> --
> View this message in context: 
> http://old.nabble.com/SQLITE-Header-access-tp33763427p33763427.html
> Sent from the SQLite mailing list archive at Nabble.com.
>
> ___
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] SQLITE Header access

[Igor Tandetnik]

On 5/3/2012 1:42 PM, msantha wrote:

I am using sqlite in my application only for read access.


But someone else might open and modify the same database. You may know 
this doesn't happen, but SQLite doesn't.



The DB gets hit
often by my application and I could see that the header(100 bytes) of the
database is read every time when i access the database.


Most likely, checking the schema cookie, to confirm that the database 
schema remains unchanged and prepared statements are still valid.



can we make it read it only once?


Try starting a transaction at the beginning, and keeping it open the 
whole time the application runs.

--
Igor Tandetnik

___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

[sqlite] SQLITE Header access

[msantha]

I am using sqlite in my application only for read access. The DB gets hit
often by my application and I could see that the header(100 bytes) of the
database is read every time when i access the database.
Precisely speaking, 16 bytes from the 24th byte of the header is read
everytime. My question is , if the database is used only for read purpose,
why the header is read everytime as the database connection is not
closed?..can we make it read it only once?

Thanks!!

-- 
View this message in context: 
http://old.nabble.com/SQLITE-Header-access-tp33763427p33763427.html
Sent from the SQLite mailing list archive at Nabble.com.

___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] is SQLite the right tool to analyze a 44GB file

[Oliver Peters]

Am Do 03 Mai 2012 15:35:46 CEST schrieb Warren Young:

On 5/1/2012 2:06 PM, peter korinis wrote:

Is SQLite the wrong tool for this project?


Probably.

SQLite is a data storage tool. With enough SQL cleverness, you can
turn it into a data *reduction* tool. But a data analysis tool? No,
not without marrying it to a real programming language.

Granted, that's what almost everyone does do with SQLite, but if
you're going to learn a programming language, I'd recommend you learn
R, a language and environment made for the sort of problem you find
yourself stuck with. http://r-project.org/

There are several R GUIs out there. I like R Studio best:
http://www.rstudio.org/

You'll still find R Studio a sharp shock compared to Excel. And yes,
it will require some programming, and yes, I know you said you aren't
a programmer. But in the rest of the thread, it looks like people have
convinced you to use SQLite from the command line, typing in raw SQL
commands; guess what, that's programming. Not on the level of R code,
but R isn't far down the same slippery slope.

It may help you to know that R is most popular in the statistics
community, which of course is populated by statisticians, not
programmers.

R isn't the easiest programming language to pick up, but it's far from
the hardest. It's very similar to JavaScript, though a bit tougher to
learn, mostly due to having accumulated some strange syntax over its
36 years. (That's counting R's predecessor, S.)
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users



I don't think that the question is "R or sqlite?". As you already wrote 
sqlite is only a container for more or less structured data (I prefer 
more structure) and that is the job it really does well. If he only 
needs to aggregate data the functionality sqlite offers is enough. If 
more complex analysis is needed (graphical or not) the combination of R 
with sqlite (i.e. via RODBC) would do a superb job.


___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] is SQLite the right tool to analyze a 44GB file

[Warren Young]

On 5/1/2012 2:06 PM, peter korinis wrote:

Is SQLite the wrong tool for this project?


Probably.

SQLite is a data storage tool.  With enough SQL cleverness, you can turn 
it into a data *reduction* tool.  But a data analysis tool?  No, not 
without marrying it to a real programming language.


Granted, that's what almost everyone does do with SQLite, but if you're 
going to learn a programming language, I'd recommend you learn R, a 
language and environment made for the sort of problem you find yourself 
stuck with.  http://r-project.org/


There are several R GUIs out there.  I like R Studio best: 
http://www.rstudio.org/


You'll still find R Studio a sharp shock compared to Excel.  And yes, it 
will require some programming, and yes, I know you said you aren't a 
programmer.  But in the rest of the thread, it looks like people have 
convinced you to use SQLite from the command line, typing in raw SQL 
commands; guess what, that's programming.  Not on the level of R code, 
but R isn't far down the same slippery slope.


It may help you to know that R is most popular in the statistics 
community, which of course is populated by statisticians, not programmers.


R isn't the easiest programming language to pick up, but it's far from 
the hardest.  It's very similar to JavaScript, though a bit tougher to 
learn, mostly due to having accumulated some strange syntax over its 36 
years.  (That's counting R's predecessor, S.)

___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users