Re: [sqlite] is SQLite the right tool to analyze a 44GB file

On 5/1/2012 2:06 PM, peter korinis wrote: Is SQLite the wrong tool for this project? Probably. SQLite is a data storage tool. With enough SQL cleverness, you can turn it into a data *reduction* tool. But a data analysis tool? No, not without marrying it to a real programming language.

Re: [sqlite] is SQLite the right tool to analyze a 44GB file

Am Do 03 Mai 2012 15:35:46 CEST schrieb Warren Young: On 5/1/2012 2:06 PM, peter korinis wrote: Is SQLite the wrong tool for this project? Probably. SQLite is a data storage tool. With enough SQL cleverness, you can turn it into a data *reduction* tool. But a data analysis tool? No, not

[sqlite] SQLITE Header access

I am using sqlite in my application only for read access. The DB gets hit often by my application and I could see that the header(100 bytes) of the database is read every time when i access the database. Precisely speaking, 16 bytes from the 24th byte of the header is read everytime. My question

Re: [sqlite] SQLITE Header access

On 5/3/2012 1:42 PM, msantha wrote: I am using sqlite in my application only for read access. But someone else might open and modify the same database. You may know this doesn't happen, but SQLite doesn't. The DB gets hit often by my application and I could see that the header(100 bytes)

Re: [sqlite] SQLITE Header access

> My question is , if the database is used only for read purpose, > why the header is read everytime as the database connection is not > closed?..can we make it read it only once? Even though your connection is used only for reading, some other connection (maybe in a different process) can be

Re: [sqlite] is SQLite the right tool to analyze a 44GB file

I have R but really haven't used it much. I know it's a great stats package and great for data reduction ... but I want to perform queries against my 44GB of data, filtering records by a variety of attributes, comparing those subsets in a variety of ad hoc ways, perhaps summing/counting other

Re: [sqlite] Data Import Techniques

Forgive me ... but what is SQLiteman? Will "import table data" help me load a csv file into a SQLite table? Thanks, peter -Original Message- From: sqlite-users-boun...@sqlite.org [mailto:sqlite-users-boun...@sqlite.org] On Behalf Of Simon Sent: Wednesday, May 02, 2012 3:03 PM To: General

Re: [sqlite] SQLITE Header access

On Thu, 3 May 2012 10:42:38 -0700 (PDT), msantha wrote: > > I am using sqlite in my application only for read access. The DB gets hit > often by my application and I could see that the header(100 bytes) of the > database is read every time when i access the database. >

Re: [sqlite] SQlite exclusive table lock

Hi Simon, Yes I have multiple Threads which are accessing the database. Currently if I am doing a write operation the whole database file is locked, I want the locking applied only to the table in operation. So currently from .net code I just say BeginTransaction(ReadCommitted) and do a write

Re: [sqlite] is SQLite the right tool to analyze a 44GB file

As said Oliver, I don't think the real issue is have to choose one software: or SQLite or R. I've used SQLite with data around 10GB under Windows 7 with 4GB of RAM and it worked perfectly. Yes, it size is less than yours but I learned that the use of every GUI was a problem. Since that, I always

Re: [sqlite] is SQLite the right tool to analyze a 44GB file

On 5/3/2012 11:59 AM, peter korinis wrote: is R a good query tool? It's a programming language. It can do anything within your power to persuade the interpreter. One of the fundamental data types in R is the data frame, which is roughly equivalent to a SQLite table. This is an R

[sqlite] Serialized access faster than multithreaded?

Hi there, I've been using sqlite for a few years now and it worked fine in my experience. Now that I have some time, I'm stress testing it to see how much I can get from it; what I want to check is its performance in a multithreading environment. To my surprise, it seems that serialized access

Re: [sqlite] Serialized access faster than multithreaded?

On 5/3/2012 5:04 PM, Marco Era wrote: Now that I have some time, I'm stress testing it to see how much I can get from it; what I want to check is its performance in a multithreading environment. To my surprise, it seems that serialized access to the database (which is the default in the

Re: [sqlite] Serialized access faster than multithreaded?

> Serialized test (SQLITE_THREADSAFE=1): > > The main thread opens the database, sets the cache and busy timeout to huge > values and passes the db descriptor to the threads. > > Multithreaded test (SQLITE_THREADSAFE=2): > The main thread creates the other threads and wait; each thread opens its

Re: [sqlite] Serialized access faster than multithreaded?

On 3 May 2012, at 10:30pm, Igor Tandetnik wrote: > On 5/3/2012 5:04 PM, Marco Era wrote: >> Now that I have some time, I'm stress testing it to see how much I can get >> from it; what I want to check is its performance in a multithreading >> environment. >> >> To my

Re: [sqlite] SQlite exclusive table lock

On 3 May 2012 20:29, Harnek Manj wrote: > Hi Simon, > > Yes I have multiple Threads which are accessing the database. Currently if I > am doing a write operation the whole database file is locked, I want the > locking applied only to the table in operation. > > So

Re: [sqlite] SQlite exclusive table lock

> So currently from .net code I just say BeginTransaction(ReadCommitted) and do > a write operation this locks the complete database file and the write > operations are really long. So currently I want to solve the > locking issue by bringing them down to table locks. To solve locking issue in

Re: [sqlite] is SQLite the right tool to analyze a 44GB file

Am 03.05.2012 19:59, schrieb peter korinis: I have R but really haven't used it much. I know it's a great stats package and great for data reduction ... but I want to perform queries against my 44GB of data, filtering records by a variety of attributes, comparing those subsets in a variety of ad

Re: [sqlite] SQLITE Header access

On 05/04/2012 12:52 AM, Igor Tandetnik wrote: On 5/3/2012 1:42 PM, msantha wrote: I am using sqlite in my application only for read access. But someone else might open and modify the same database. You may know this doesn't happen, but SQLite doesn't. The DB gets hit often by my application