--- Mattia Landoni <[EMAIL PROTECTED]> wrote: > Hi all, > > I am an economist and I discovered J a few days ago. I haven't been so
> Data Management > - I import data from several sources. Not always are they in straightforward > formats. Are there libraries or built-in function to import text (e.g. .csv, > .tab, fixed format) and non-text (e.g. Excel, 1-2-3) data? In addition to existing libraries for CSV etc., it is easyto construct text parsers with J operations and powerful PCRE regex. See http://www.jsoftware.com/jwiki/Addons/xml/loose > - (*) I often merge datasets (sort of SQL join). The other day I saw that it > is possible to embed a database (SQLite) through a library. Are there > interfaces to other databases? I usually use MySQL (last time I checked > SQLite did not implement enough SQL for my purposes - that was probably 2 > years ago). Are there in-built functions to perform similar operations? > (although I'd be very happy to do all the merging in SQL). Besides SQLite, there is generic ODBC interface supported, which can be hooked up in Unix too. Inverted tables and functional queries are easy to construct and work with interactively. See Flwor under DB in Wiki. > Performance > - (***) How does J deal with very large datasets? currently I am dealing > with a 65-Gb dataset. So far only software I can use is SAS. Performing an > SQL query [SELECT, GROUP BY] in SAS on a dedicated server takes me six > hours, of which a large part of the time is network I/O (I guess SAS's > computing time would be an hour, perhaps two). The data is divided in 7 > chunks of 7 to 13 Gb each. Having the same amount of data on a good > computer, would I be able to perform the same operations with J? Assume > plentiful RAM and speedy processor: what's the order of magnitude of the > time it would take? > - I read something about memory mapping in past posts and I intuitively > understand what it means but I never did it. What are the limits of memory > mapping? In general, what are the techniques to deal with large datasets? Combined with Flwor, mapped inverted columns should scale to >2Gb on 64-bit architecture. > Computation > - Is there a numerical optimizer/solver? (e.g., given a certain function, > find local maxima and minima; given an equation, find the zeros). I could > program this one, but is there one already? > - Is there a sufficiently painless interface to Maxima (symbolic calculus > toolbox)? If not already, J has simple yet powerful portable DLL interface. So connecting external libraries is easy. > Learning > - What's the fastest way to learn the basics for a greedy person who learns > the average C-like programming language in a week? Normally what I do is to > learn "what can be done" and then start programming right away with a > reference at hand. Here it does not seem so simple... right? Cover everything in Introduction and Sample Topics with J sessions open http://www.jsoftware.com/help/dictionary/contents.htm and then run the labs from J sessions. It's not a C-like language, so allocate some quality time to it. It will be rewarding. ____________________________________________________________________________________ Like movies? Here's a limited-time offer: Blockbuster Total Access for one month at no cost. http://tc.deals.yahoo.com/tc/blockbuster/text4.com ---------------------------------------------------------------------- For information about J forums see http://www.jsoftware.com/forums.htm
