page size is max (32768). These sqlt files are around 100Gig with 200 million rows. I have query_cache set to 0.
I am running single queries against these attach tables and then putting the data in memcached. The script runs thousands of queries before it runs out of memory. If I use the same width table ( a product of all 3 attached tables, flattened into 1 table) and run it with the same script I have no memory issues. I am hoping there is something I am missing with caching and attach. Maybe a pragma or something? 2009/9/25 P Kishor <punk.k...@gmail.com> > On Thu, Sep 24, 2009 at 3:30 PM, John Lynch <johntly...@gmail.com> wrote: > > Sqlite Users, > > > > I am using sqliute user for a very large customer database. I have 3 > > seperate DB files with a table in each file. The files are cust,acct, > and > > web. > > > > Each DB table has ~ 200 Million Rows. When I attach web and acct to > cust > > with perl DBI/DBD::Sqlite the process runs out of memory on windows (2G > > 32bit process limit). The memory usage grows out of control while > querying > > all table joined together via the attach. > > > > When I used a flattened table of the same data (ie create table as select > * > > from ...) via same attach to create the flattened table. I have no > > problems. > > > > Are you running out of memory when you ATTACH, or when you actually > perform a specific query? If the only difference between your two > instances is "ATTACH," and you are doing everything exactly the same > and running out of memory with ATTACH and not without ATTACH, then > there is something going on at the point of ATTACH. More info from you > will shed light on this. > > Besides providing more info on this here, you might also want to ask > this question at perlmonks, in case this is a Perl-specific issue. > > > > > > > > Any ideas on how to reduce the memory consumption when using attach? > > > > Thank You > > > > -- > > -- > > John Lynch > > (217) 953-0557 > > johntly...@gmail.com > > _______________________________________________ > > sqlite-users mailing list > > sqlite-users@sqlite.org > > http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users > > > > > > -- > Puneet Kishor http://www.punkish.org > Carbon Model http://carbonmodel.org > Charter Member, Open Source Geospatial Foundation http://www.osgeo.org > Science Commons Fellow, http://sciencecommons.org/about/whoweare/kishor > Nelson Institute, UW-Madison http://www.nelson.wisc.edu > ----------------------------------------------------------------------- > Assertions are politics; backing up assertions with evidence is science > ======================================================================= > Sent from Madison, WI, United States > _______________________________________________ > sqlite-users mailing list > sqlite-users@sqlite.org > http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users > -- -- John Lynch (217) 953-0557 johntly...@gmail.com _______________________________________________ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
