Hi all, I am an economist and I discovered J a few days ago. I haven't been so excited since when I was 13 and Santa brought me an 8-bit Nintendo Entertainment System. Yet before taking a week off from work to study J (just kidding) I would like to be sure it does everything I need. Here are questions in four main topics: data management, performance, actual computation, and learning. Every answer to any question is very welcome. Answers to questions marked with a (*) are particularly important to me. Thank you in advance!
Data Management - I import data from several sources. Not always are they in straightforward formats. Are there libraries or built-in function to import text (e.g. .csv, .tab, fixed format) and non-text (e.g. Excel, 1-2-3) data? - (*) I often merge datasets (sort of SQL join). The other day I saw that it is possible to embed a database (SQLite) through a library. Are there interfaces to other databases? I usually use MySQL (last time I checked SQLite did not implement enough SQL for my purposes - that was probably 2 years ago). Are there in-built functions to perform similar operations? (although I'd be very happy to do all the merging in SQL). Performance - (***) How does J deal with very large datasets? currently I am dealing with a 65-Gb dataset. So far only software I can use is SAS. Performing an SQL query [SELECT, GROUP BY] in SAS on a dedicated server takes me six hours, of which a large part of the time is network I/O (I guess SAS's computing time would be an hour, perhaps two). The data is divided in 7 chunks of 7 to 13 Gb each. Having the same amount of data on a good computer, would I be able to perform the same operations with J? Assume plentiful RAM and speedy processor: what's the order of magnitude of the time it would take? - I read something about memory mapping in past posts and I intuitively understand what it means but I never did it. What are the limits of memory mapping? In general, what are the techniques to deal with large datasets? Computation - Is there a numerical optimizer/solver? (e.g., given a certain function, find local maxima and minima; given an equation, find the zeros). I could program this one, but is there one already? - Is there a sufficiently painless interface to Maxima (symbolic calculus toolbox)? Learning - What's the fastest way to learn the basics for a greedy person who learns the average C-like programming language in a week? Normally what I do is to learn "what can be done" and then start programming right away with a reference at hand. Here it does not seem so simple... right? Thank you again and, again, any answer to even just one question is welcome. Mattia -- Mattia Landoni 1201 S Eads St Apt 417 Arlington, VA 22202-2837 USA Greenwich -5 hours Office: +1 202 62 35922 Cell: +1 202 492 3404 Home: +1 360 968 1684 Govern a great country as you would fry a small fish: do not poke at it too much. -- Lao Tzu ---------------------------------------------------------------------- For information about J forums see http://www.jsoftware.com/forums.htm
