I was re-reading this and I saw this part: > What I did was abandoned the server side concept entirely and moved > the processing on the client side. It is now running in this > configuration, where the client system downloads all the data that it > needs and computes it and J creates SQL statements that would insert, > delete and/or update the database records.
My statement implies that J only creates the SQL statement. This is incorrect. Specifically, the C# application retrieves data from the SQL server over the internet and submit all data into the current J session. J now computes everything and just returns a string containing 3 type of SQL-DDL statements which are DELETE, INSERT and UPDATE statements. The C# calling application now has the responsibility of executing/submitting this SQL statements to the MS-SQL Server. Hahaha, I actually saw and instance where ADO.NET choked on a 25,000 SQL-DDL statement generated by J in less than a minute. -----Original Message----- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Alex Rufon Sent: Thursday, August 21, 2008 10:30 PM To: General forum Subject: RE: [Jgeneral] Successfully Migrated to J602 from J601 Yes your right. The system was designed to be synchronous. When I was reading your suggestion, I actually thought about just letting the client insert a record to a database table which is read by a server side application that is invoked by windows scheduler. :) It would be so simple that J would just update the record periodically and the client application will also just poll that same database table for its specific request. Using the database would satisfy all the requirements that you specified. Well, I was thinking about this while listening to Bob Dylan's "All Along The Watchtower" then a though occurred to me. I may be overlooking a crucial step. You see, the RMR calculation is actually composed of computation by categories. Categories are by Color, by Size, by Destination, by Color and Size, by Size and Destination and a lot more combination. This implies that materials to be computed by color are expanded to the number of colors. Like there are 2 fabrics that varies by 4 colors. I actually end up with 8 records. So materials that vary by Size and Destination (2 fabrics, 3 sizes - s/m/l, 2 destination - US/Japan) would have a 12 (2*3*2) number of records. There variations are simple and would only take J less than 3 minutes to compute. The RMR calculation that was taking more than an hour was the order which require the all three calculation: 1. Color by Size by Sub-Style 2. Color by Size by Purchase Order 3. Color by Size by Sub-Style by Purchase Order Looking at my J scripts, I fear that my code is not using J in its best form. My plan now is to still implement the progress dialog initially and spend the sometime on "optimizing" the code again. If this still fails (still un-acceptable performance) ... thats the time I'll consider running the script on the server side again. :) -----Original Message----- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Oleg Kobchenko Sent: Thursday, August 21, 2008 4:22 PM To: General forum Subject: Re: [Jgeneral] Successfully Migrated to J602 from J601 > From: Alex Rufon <[EMAIL PROTECTED]> > > **J Usage > For this project, we used J to primarily calculate the following: > 1. Raw Material Consumption – computes how many items of each material > is needed to make one garment of a certain style, size and color. > 2. Raw Material Requirement – based on the raw material consumption, J > computes for the actual number of number of materials that are needed > to be ordered from the supplier based on the number of actual garments > ordered by the customer. > J was a life saver because of its inherent capability to do matrix > calculations. > Original computations made in AS400 and VB was taking some time to > complete because each material needed has to be looped through one by > one. The number of materials not only includes the fabric but > everything from thread, buttons, tags to adhesive tapes, plastic bags > and packing cartons. The challenge here is how to make the computation > run faster because of the iterative process of coming up with the raw > material consumption which leads to changes in the raw material > requirement. The operator would run the calculation a number of times > a day because of availability materials or changes in the consumption > and allowances or replacement of the materials itself. With the > competitiveness in the world market, a factory has to send out their > orders to suppliers within 24 hours from receiving the confirmed > orders from the customers. Missing the 24 hours limit normally means that the > material is already out stock from suppliers and delays ensues. > > ** Some Design Constraints > The initial design of this system had J running on the server as a > Windows Service. This lead to J using up 100% of only 1 CPU of the 4 > CPU application server. With concurrent users increasing, the Windows > Service started queuing up requests that the web server started > timing-out the requests … even though they were still running and > would complete successfully. Since the web server already gave-up, > the client was notified that an error occurred and the users reaction was to > submit the same operation again and further slowing things down. > It was a vicious circle. > > What I did was abandoned the server side concept entirely and moved > the processing on the client side. It is now running in this > configuration, where the client system downloads all the data that it > needs and computes it and J creates SQL statements that would insert, > delete and/or update the database records. > > When I first decided to change my execution design strategy, I had an > idea that they may be some performance problems but I was thinking > that I’ll just burn the bridge when I get there. So after more than > half a year after implementation I encountered my first performance > problem. You see, client machines on the factory were not really that > powerful. A typical setup would be with the following specs: > Windows XP Professional > 1.0GHz CPU > 20GB HD > 384 MB RAM > > Normal J usage peaks at 70MB RAM but last month, there were some RMR > calculation that the user says doesn’t finish. So I investigated and > found out that because of the number of materials involved in the > order, J usage was peaking at 130MB RAM. I then timed it and it took > more than an hour to finish with J using up the CPU. So now, some > users would kill the J process from the Task Manager and would > complain that the system is “Hanged”. :D > > For now, I’m trying to put in a progress dialog box into the mix > (Oleg’s Progress Dialog to be precise) as an immediate solution but I > am looking into an “easy” solution of running the code on the server. > What I’m looking for are J libraries that would make it relatively > easy to implement execution of J scripts on the server side (web server or > application server behind the web server). > Right now I’m looking at JHP but I’m still doing the leg work since I > really need to complete the QA documentation requirements. As I can see the focus was on synchronous processing either at the server or at the client. However, what we see here is an example of a long-running processes-- anything that exceeds finite amount of user attention for an atomic operation. Here's a design pattern for asynchronous scalable distributed process management. Asynchronous means that process is broken into life-cycle operations steps and corresponding operations: initiate, suspend, resume, abort, get status etc; statuses: waiting, running, suspended, success, error. The GUI needs to be decoupled from process execution and only interact indirectly with quick atomic status query and change operations. User session (client application) needs to be decoupled from server, which would allow to login/logout any time, continue to observe server operations running independently. Server process management is done with a queue that maintains these statuses and mediates communication between the client request handler, worker processes manager, and worker processes. Queue item also contains parameters. Scalability is made easy with such queue model as the managers and handlers can be deployed across many commodity machines while processing the same queue. Scheduling allows to uses this process management framework for off-line batch execution. Configurable load-balancing and throttling ensures even spread of tasks over available hardware. To implement (parts of) this, an existing framework could be used, such as WWF in .NET, BPEL in Java, to some extent Quatz scheduler in Java. But you want to make sure all aspects above are satisfied. So it may be better and simpler to implement your own solution using a database and combination of standard framework components: ASP.NET, command-line executables, COM+ or J2EE apps etc. ---------------------------------------------------------------------- For information about J forums see http://www.jsoftware.com/forums.htm
---------------------------------------------------------------------- For information about J forums see http://www.jsoftware.com/forums.htm
