It will take a few minutes per Gigabyte to load the data from database into your system. So expecting real-time (few seconds) processing, the data need to be pre-loaded. While at it, it might as well make sense to cache the data in local files that then can be mapped.
The system be best used if you are not thinking in terms of wholly contained in-memory database, but as local mapped storage. Then you should break the processing (eg one report or analysis) into small simple requests, which will map only necessary columns into memory, run query, unmap columns. Mapping/unmapping works very fast, and the OS will optimize more frequently mapped data for you. BTW, how many tables and columns are there? The tables can also be partitioned horizontally. With such design the capacity of your local cache is limited not by memory, but by disk space. Only atomic requests are limited by (virtual) memory, which can be tuned and optimized. With the domain-specific user operations you are thinking about, the mapping/unmapping can be done seamlessly behind the scenes. Further notes below... --- On Mon, 4/21/08, amit bolakani <[EMAIL PROTECTED]> wrote: > Some of the questions that I have following up to your > answers are: > 1. Is there an equivalent of MAXWS in Dyalog APL in J? AFAIU, MAXWS is a rudimentary artifact of legacy APL memory management. Dyalog also requires contiguous memory http://www.dyalog.dk/faq.htm#Config_2 J should just be able to allocate available memory and delegate the maintenance to OS, while optimizing only small chunks of memory. So you should not worry much about J ability to utilize the available system memory. > 2. Are there any examples of C# interaction with J? There are a number of examples under .NET in http://www.jsoftware.com/jwiki/Interfaces > 3. Can one produce word documents from J. From what I see > in Publish, it > seems mainly for PDF output reports. I guess one could > easily generate HTML > output and rely on the MS WORD HTML processor to get the > equivalent Word > files. Correct? HTML is one possibility, another would be concatenating RTF of producing Word 2003 XML. > 4. Is the NYCJUG forum open to rank newbies like me :) > Thanks, > -Amit > > On Mon, Apr 21, 2008 at 11:18 PM, Sherlock, Ric > <[EMAIL PROTECTED]> > wrote: > > > Thanks for the info Amit, > > I'm sure other are better qualified to give you > guidance here, (and so I'm > > happy to be corrected by others if I'm wrong!) > nevertheless my thoughts are > > as follows: > > > > I think you will be able to create an effective > interface using a J GUI > > fine, but if you want a pretty interface then the C# > road will probably be > > the way to go. > > > > If your data is essentially one big database table, > then I think that > > using one or more J arrays to store and manipulate > your data will be much > > simpler, faster and more flexible than storing the > data in a database. > > > > I don't think there are any requirements specified > that would be a problem > > for J. > > > > Some resources you might find useful are: > > * The grid demo: Studio|Demos...|grid > > * http://www.jsoftware.com/jwiki/Grid > > * Chris pointed you to > http://www.jsoftware.com/jwiki/DB which I would > > second. > > - The links under Data Models may be useful for > deciding how to manage > > your data within J. > > - Some of the other links may be useful for > interfacing with databases > > for importing data > > * The controls demo (Studio|Demos...|controls) will > give you an idea of > > what controls are available in a J GUI. > > * The Publish Addon may be useful for creating > reports: > > http://www.jsoftware.com/jwiki/Publish > > > > I'm sure (never having been myself ;-)) that you > would be welcome at the > > NYCJUG meetings. http://www.jsoftware.com/jwiki/NYCJUG > > > > > > ---amit bolakani wrote: > > > Hi guys, > > > Thanks for all the responses. I really appreciate > all you > > > guys trying to > > > help me out with my problem. And I think I owe > you guys a > > > better explanation > > > of the problem. This is an interesting set of > requirements > > > and I don't know > > > if something like this has been done with J > before. Here is > > > what I am trying > > > to achieve: > > > I want to build a tool for Windows which will > have a GUI > > > based interface and > > > will be able to interface with: > > > 1. Databases (SQL, Oracle, Mysql etc) > > > 2. Excel and CSV files > > > 3. Text files > > > These interfaces will be used to pull data into > the tool. > > > While inputting > > > the data, I want the user to be able to specify > some data > > > transformations on > > > the incoming data and also be able to add new > user defined > > > columns in the > > > dataset which would be derived from already > existing columns. > > > Once I have > > > imported this data I would want to have the > ability to store > > > it in memory so > > > that I can easily manipulate it. The tool should > be > > > restricted only by the > > > amount of RAM on the PC (ideally something like > MAXWS that > > > Dyalog APL has). > > > Ideally I would want to build an in memory > database so that I > > > have easy > > > access and manipulation capabilities. I would > then like to > > > run things like > > > frequency reports, averages, means, variances of > the various > > > columns and the > > > like on the data and show reports and produce > output > > > excel/csv/word files. > > > I think J could be ideal for such a project > though I am a newbie to J > > > programming. > > > There are 2 approaches that I have been > considering before I dive in: > > > 1. Use a C# front end for the GUI and > Excel/CSV/DB interfaces > > > and read the > > > data into a SQLite database while operating it in > an > > > in-memory mode. Though > > > I am not sure how I would easily be able to apply > column > > > transformations and > > > add new columns (would the transformations be > done in J and > > > the new columns > > > be added using SQLite) and moreover what the > performance of > > > this would be > > > for large files (like say a million records -- > basically > > > limited by the PC > > > RAM size). With this approach I would use J for > processing > > > the operations. > > > Note that I don't need to persist the data -- > so I would effectively > > > construct an temporary in memory db session with > SQLite (and > > > I understand > > > that SQLite already has an addon for J). > > > 2. Use J for GUI and use an in memory database (I > don't think > > > anything is > > > happening with jdatabase project for extending J > with database > > > capabilities?) like kdb (I believe there is an > addon for J with kdb as > > > well). I would also in this approach use J for > the processing the > > > operations. > > > Lastly, I would want to give the users the > capability to > > > apply operations on > > > the data and the set of operations should only be > restricted > > > by the rich > > > functionality that is offered by J. I would want > to basically > > > map the user > > > defined operations easily onto J operations, > almost as if the > > > user defined > > > operations are disguised J operations. > > > I am still not sure if J can handle all the above > > > requirements and all your > > > inputs would be extremely valuable to help me > make my decision. > > > All in all, I want to build an analytical > database with rich > > > user defined > > > functionality with is only restricted by the > features > > > offered by J and > > > want to have optimum performance (in seconds) > without the > > > need to persist > > > the data and do it all on Windows with a GUI. > > > Thanks to everyone in advance. > > > -Amit > > > NOTE: If there are any J experts in NYC, I would > like to sit > > > down with them > > > (if they are kind enough and interested enough to > spend time > > > on this) and > > > work out more details ____________________________________________________________________________________ Be a better friend, newshound, and know-it-all with Yahoo! Mobile. Try it now. http://mobile.yahoo.com/;_ylt=Ahu06i62sR8HDtDypao8Wcj9tAcJ ---------------------------------------------------------------------- For information about J forums see http://www.jsoftware.com/forums.htm
