Alex - Thanks for your detailed reply. This is giving me some good insights into where J actually should be used (or not used) in real-world applications.
> > From my development environment, the normal way was to develop first a > whole module in C# or any .Net language then optimize them afterwards with > J. There are of course cases where J was integrated from the first step but > this are special cases. I normally get called in a development meeting when > the team already has a performance problem. > ** Run-time Performance - there is no doubt that tweaking a module with J > always-always have an increase in performance. Unfortunately saying a > "percentage" is relative to what was being accomplished. > Ok, it's nice to see that J is useful in situations where performance is important. This is as I suspected: with its array-processing capabilities, J ought to be able to shine in this area. > ** Customizability - since most of the implementation of J in my > development environment is very specific ... customizability is also > relative. Take for example an EDI parser. I have built a bunch of EDI > parsers for A&F, Polo Raph Lauren, adidas, SPRIT, etc. For each of this > companies, I actually have a base code which I customize for each of our > clients. I initially hard-coded everything and later-on created J Nouns to > handle specific data structures per client and the base code customization > became minimal. > It's also encouraging to hear that you're able to use J for customizability. I'm down in Brazil where the tax authorities are moving from an older EDI format called CNAB to XML, and they're starting to require near-real-time on-line electronic submission of digitally signed tax documents for all sales. This is actually what got me back to looking at J again this week: this past Sunday I again surveyed what's out there for open-source ERP (enterprise resource planning) systems which have been customized to Brazil's language (Portuguese) and its new EDI formats for taxation, and I decided to re-evaluate an open-source ERP system implemented in Java+PosgreSQL+JBoss. After spending four days trying to install the thing (since this app's community of open-source developers apparently take the whole cathedral-and-bazaar thing a bit too literally, eschewing modern lifecycle management tools such as Maven which would have avoided a lot of headaches involving the need to deploy various bits of the system against various JRE versions), I was confronted with a monstrous application involving a slow, bloated JBoss application server mediating between a slow, bloated Java front-end client and a PostgreSQL back-end database server... which basically just presented the user with a giant CRUD system with a bit of workflow thrown in. And I thought: - This thing is slow, it's bloated, it's just a glorified CRUD with some workflow, and it's going to need lots of customization both initially and probably over time. - There's no fancy GUI (no ability to freeze left-hand columns in a datagrid, no custom in-cell editors in the data grids aside from a drop-down combo-box here and there, no visual query builder). So that's what initially motivated me to think about J and write into the forum this week. I thought maybe it would be worth it for me to try porting a mid-size open-source ERP database application like this from Java+SQL+JBoss to J. I figured the number-crunching efficiency in J would be as good or better than in Java+SQL, and in this case a simple interface in J could be just as good or better than the current one in Java - and maybe by eliminating some of the layers or tiers (getting rid of JBoss), and by writing everything in a single language rather than Java + SQL (plus the procedural language used in the database - currently a weird thing called PL/Java where PostgreSQL makes calls to Java JARs, and in later releases they've moved to using PosgreSQL's standard built-in procedural language called PL/PgSQL), the whole thing might be simpler and more efficient in the long run - and more enjoyable for the programmer to program and the end-user to use. > ** Programmer Efficiency - the problem with this is that I technically am > the only main J resource in our company. I have trained a lot of in-house > developers over the years but most of them use J to just support the > existing J modules that they have. What they do use J for is problem solving > prior to coding the solution to the preferred .NET language. > I'm not sure, but it sounds like maybe there is a lack of understanding among many programmers who may have been trained in more "classical" programming languages such as Java or C#, how to really be productive with a language like J. Sounds like you're in a nice position being one of the few at your company who really knows how to be productive with J! :-) > So you asked: > > Would it make sense, from a practical standpoint, to port a typical, > mid-size Java+SQL+JBoss application to J (involving roughly 500 tables, with > roughly a million records and 100 columns in the biggest table)? > > Short answer: NO > > Long answer: Re-engineer your system using the tools that you or your > resources are most familiar and identify where J can best help. > > Some suggestion: > 1. Process where computation takes more than 2 minutes. > 2. Process/Computations that uses more than 3 tables. > 3. Process/Computations that uses data/information coming from different > sources like > a. data coming from different databases or database servers > b. consolidation of data coming from different source like: EDI, CSV, > output from Mainfraimes and MINIs, data coming from OEM/hardware (like Time > Tracking devices). > c. Conversion of data from one format to another (EDI->XML, XML->CSV, > CSV to FIXED WIDTH files, etc) > 4. Really complicated data displays (i.e. reports that requires cubes, > pivots, etc) > 5. Processing data for complicated reports > OK, so here it sounds like you're saying that J is a good calculation engine for big batches of data. Which again is as I suspected - due to its array-processing capabilities, it should do well in this area. > > Some weakness of current version of J6 (J7 is still in beta but looking > good) > 1. Complicated GUI (for those WD masters ... don't hurt me) > 2. Data Entry modules > 3. Operations that require row by row processing (never use J with this) > This is the stuff I was worried about: the stuff J might not be so good at. Regarding a complicated GUI: I have seen some very nice GUI add-ons for environments such as .NET (for example, the Xceed data grid looks very nice, with all kinds of different in-cell editors, a single master-detail data grid, etc.) Seeing the part of J's Form Builder which apparently allows dropping in any installed OCX control gave me some encouragement about being able to make a complicated GUI using J. I thought that in cases where a complicated GUI would be needed, maybe I could just hunt around for (or create?) a decent OCX control that did that. But I don't know the details about how such custom OCX controls might "hook into" the J data being processed in the back-end. Some concerns I have: - Would there be some loss of efficiency due to piping data back and forth? - Would data be fully updatable in both directions? - Would it be easily formattable and validatable? (I remember the attributes ..f and ..g in the language K which did this quite well using K's older Windows native UI - I'm not sure how doable this kind of validation and formatting might be using J plus an OCX control.) - Would there be a loss in runtime customizability due to J's interpreter having to interact with the pre-compiled libraries which implement an OCX control? I also have become very attached to the visual query builder in good ole MS-Access - and I've seen similar query builders in the various database administration tools put out by EMS (for PostgreSQL, Firebird, MySQL, etc.) I don't know if such a visual query builder is currently available as a control that can just be dropped onto a form. A visual query builder is quite useful in any environment (both for programmers and end users), and I've been curious about whether people think a decent one could be built to work with J using some more basic GUI components which are already out there. You say that J should not be used for data-entry modules. This seems to be the big stumbling block and I'm curious as to why this might be so. - Does J's column-oriented approach make it prohibitively inefficient to have say several users simultaneously trying to add or edit rows in the same table from separate workstations? - Would J be simply inefficient in such a case - or could it also lead to actual record contention errors - say problems generating unique ids for each rows, or problems trying to simultaneously write multiple records to the same table? I have no idea if J is able to handle concurrency the way some of the popular RDMBSs do. Mainly I'm talking about "record contention" - two people trying to edit the same table (different records) - or even the same record - at the same time. I understand that Firebird uses something called MVCC (multi-version concurrency control) to handle this. I also understand that most databases have some sort of built-in mechanism for performing a two-phase commit on a transaction which might involve multiple writes across multiple tables. (For example, the classic situation where a user wants to withdraw $10 from a savings account and transfer it to a checking account. Either both transactions should succeed and commit, or both should fail and rollback - because obviously, if only one succeeds and the other one fails, then you've either created or destroyed $10, and the accounts won't balance any more). However, even if J doesn't currently have built-in logic for handling this kind of situation, that doesn't mean that it never could have. I remember reading in the literature that many years ago, even MS-SQL lacked a built-in two-phase commit protocol, so database developers were "rolling their own" until the RDMBS vendor got around to adding it as a built-in feature. If such a feature is currently lacking in J, maybe it's something that I could work on adding. There are apparently several open-source implementations of this sort of thing out there which I might be able to use as guidance - even more exotic, high-end multi-phase multi-processor commit protocols such as the Paxos algorithm which I believe was developed by Google and which is now open-source. To summarize the above: I guess you are confirming my concerns about where J might be a bit weak: - fancy GUIs - transaction- or row-oriented processing I guess what I'm wondering is: - Is J weak in these areas because of any inherent unsurmountable aspects of the way it's designed? - Or if I did a bit of scrounging around for decent compatible GUI controls which could be dropped into the interface to a J system, and for decent multi-user row-oriented transaction-processing algorithms that could be implemented on top of J, would J then be able to have reasonable performance and usability in these "weaker" areas as well? I understand that you are recommending not creating a pure J system, but rather a hybrid. Maybe I'm being off-the-wall here, but I see J as being strong not only because of its efficient array-processing, but also specifically because it is *interpreted*. I'm absolutely blown away by how easy it is to use J to create a window, or a control on a form, at *run-time*, simply by making an appropriate call via 'wd' (which I only started scratching the surface of yesterday in some of the J Labs) to the appropriate Windows API. I think many programmers have been grown accustomed to thinking of all on-screen widgets as somehow being dependent on code which needs to go through the entire compile, build, link cycle - simply because of their habit of programming such widgets using statically typed languages. So programmers may tend to they can only create a widget statically. It was really an epiphany when I saw how J is able to create the same widgets *dynamically*. So while I like J for it's array-processing efficiency, I also like J for its interpreted nature, its ability to make fairly major updates to a GUI at runtime, creating and destroying arbitrary on-screen widgets dynamically, instead of having to go through a whole coding-compiling-building-linking cycle just in order to create or modify a on-screen widget. This is why, although J may not be optimized for row-based or transaction-oriented processing, I'd like to investigate further whether it could be made competitive in the areas of user interface and transaction processing - perhaps by finding or building some additional useful OCX controls for it to work with, and implementing some additional algorithms on top of J to support concurrency. > > Another note, as much as possible, minimize data marshaling from one > language to J (i.e. J -> C#, C# -> J). A few years back, we built a > production planning system in our company using pure VB.NET. I got called > in because re-scheduling a production run technically freezes the user's PC. > The original system uses GDI+ to draw the GANTT chart and an MS-SQL server > at the backend with VB.NET as the main language. This was a pretty big > system because the each Production Plant has its own database and each > country may have more than 1 Production Plant ... then we allow a > user/manager to play around with putting his/her order on any production > plants in the World (well, at least at countries where we have a factory). > When I was pulled-in to re-engineer the project, the first thing I did was > gutted the middle-tier of the system and replacing it with J. The database > is still MS-SQL server and the front-end is GDI+ using VB.NET but this > module concern is now only to draw the GUI and acc > ept inputs. J now handles all data processing which meant I actually > implemented a local database cache in J because data is consolidated from > different sources. Of course, this is an oversimplification of what we did > (there were a lot of other changes to make sure that high level managers get > the data from off-site factories). > This is a very interesting real-world industrial application of how J is being used. I should clarify my own background and say that I haven't done any "real-world" programming in any commercial language - not C/C++ or Java or C# or Scala - although I've studied them quite a bit over the years. The only languages that I've ever put to any serious use come from the realm of theoretical computer science: mainly the algebraic term-rewriting language Maude (which almost nobody has heard of), and I've studied other languages which also tend to have a theoretical bent such as ML, Scheme, Haskell, Lisp, Clean, BETA, and I've looked at the related array-processing language K. I'm interested in J because it has some of the nice features of these less-well-known languages (functional or function-level programming, array processing, interpreted)... plus with its ability to use OCX controls it looks like it could have a much nicer Windows GUI than, say, the old one which K eventually abandoned. So this is why I've been curious about how J can be used or is being used in real-world applications. Due to the complexity of languages like C/C++, Java or even C# and Scala, I don't know if I could ever do any real work using these languages: I just get overwhelmed whenever I look at the source of any project built using them, I get mentally exhausted from distractions after a few hours of trying to tinker with it, and I also lose my patience waiting for the resulting massive bloated mess to compile, load and run. I do have some hope, however, that I could become a decent J programmer. I've occasionally spent a few weeks or months writing stuff for myself using Maude or using K - and I found that it was never mentally exhausting, it always left me feeling clear-headed, and I didn't get bogged down in a morass of distractions. So I'm always interested in finding out more about practical applications of J, as this seems to be the only language out there which is enjoyable for me to work with and which also can be applied to real-world programming tasks - perhaps in a hybrid situation as you recommended, only using J in areas where it's strongest. > > So to summarize (I am rambling again), I would not create a pure-J system > but instead a hybrid where I would use J where it would be at its best. :) > r/Alex > Thanks for all your detailed comments - you've helped clear up a lot of stuff, and it's very interesting learning how you're actually using J in real-world commercial applications. I understand what you're recommending about doing hybrid systems leveraging the areas where J is strongest, rather than "pure J" systems. I'm also still curious whether anyone has any further opinions about whether there's any possibility that J could also become a contender in areas where it's not traditionally regarded as being strong: transaction-oriented / record-based processing (perhaps by implementing additional needed concurrency mechanisms on top of J), and complicated GUIs (by using J602's ability to incorporate OCX controls). - Scott ---------------------------------------------------------------------- For information about J forums see http://www.jsoftware.com/forums.htm
