[sqlite] excessive malloc() calls

[Matt_Arrington]

I switched my application over to SQLite3 and did some performance
profiling and found that the majority of the processing time spent is
making calls to malloc().

sqlite3_step() is the function that is making all the excessive calls, one
call per row fetched.

The program is a stock scanning / data mining program. It keeps about 6-7
years worth of daily stock price data for a company in a table, and I have
about 3000 of these tables. One complete scan of all 3000 companies will
make roughly  5.5 million malloc() calls!

I create each table using SQL command:
CREATE TABLE 'SPY' (key INTEGER PRIMARYKEY UNIQUE, open FLOAT, low FLOAT,
high FLOAT, close FLOAT, volume FLOAT)

I need each row sorted by date, which I setup as my primary key. Ideally
each table would be stored sorted on disk since I do a very small amount of
updating of only one record per day per table. I could not figure out a way
to make  do that.

When I read a table I use the following SQL select statement.
SELECT * FROM 'SPY' ORDER BY 1 ASC

This all seems like it should be pretty basic stuff, that I'd expect an SQL
engine to easily handle. I'm a novice when it comes to SQL, so maybe I'm
doing something dumb?

I debugged / stepped in to the step() code and noticed that it is the op
code COLUMN making the memory allocation. I think it may be my ORDER BY
clause that is causing it to take the path to the malloc() call.

The actual code making the call is:

/* Read and parse the table header.  Store the results of the parse
  ** into the record header cache fields of the cursor.
  */
  if( pC && pC->cacheValid ){
    aType = pC->aType;
    aOffset = pC->aOffset;
  }else{
    int avail;    /* Number of bytes of available data */
    if( pC && pC->aType ){
      aType = pC->aType;
    }else{
      aType = sqliteMallocRaw( 2*nField*sizeof(aType) );
<<<<<<<<<<<<<<<<<<<<<<< HERE
    }
    aOffset = &aType[nField];
    if( aType==0 ){
      goto no_mem;
    }


Thanks,

Matt Arrington