Yes your right. The system was designed to be synchronous. When I was reading your suggestion, I actually thought about just letting the client insert a record to a database table which is read by a server side application that is invoked by windows scheduler. :) It would be so simple that J would just update the record periodically and the client application will also just poll that same database table for its specific request. Using the database would satisfy all the requirements that you specified.
Well, I was thinking about this while listening to Bob Dylan's "All Along The Watchtower" then a though occurred to me. I may be overlooking a crucial step. You see, the RMR calculation is actually composed of computation by categories. Categories are by Color, by Size, by Destination, by Color and Size, by Size and Destination and a lot more combination. This implies that materials to be computed by color are expanded to the number of colors. Like there are 2 fabrics that varies by 4 colors. I actually end up with 8 records. So materials that vary by Size and Destination (2 fabrics, 3 sizes - s/m/l, 2 destination - US/Japan) would have a 12 (2*3*2) number of records. There variations are simple and would only take J less than 3 minutes to compute. The RMR calculation that was taking more than an hour was the order which require the all three calculation: 1. Color by Size by Sub-Style 2. Color by Size by Purchase Order 3. Color by Size by Sub-Style by Purchase Order Looking at my J scripts, I fear that my code is not using J in its best form. My plan now is to still implement the progress dialog initially and spend the sometime on "optimizing" the code again. If this still fails (still un-acceptable performance) ... thats the time I'll consider running the script on the server side again. :) -----Original Message----- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Oleg Kobchenko Sent: Thursday, August 21, 2008 4:22 PM To: General forum Subject: Re: [Jgeneral] Successfully Migrated to J602 from J601 > From: Alex Rufon <[EMAIL PROTECTED]> > > **J Usage > For this project, we used J to primarily calculate the following: > 1. Raw Material Consumption – computes how many items of each material is > needed > to make one garment of a certain style, size and color. > 2. Raw Material Requirement – based on the raw material consumption, J > computes > for the actual number of number of materials that are needed to be ordered > from > the supplier based on the number of actual garments ordered by the customer. > J was a life saver because of its inherent capability to do matrix > calculations. > Original computations made in AS400 and VB was taking some time to complete > because each material needed has to be looped through one by one. The number > of > materials not only includes the fabric but everything from thread, buttons, > tags > to adhesive tapes, plastic bags and packing cartons. The challenge here is > how > to make the computation run faster because of the iterative process of coming > up > with the raw material consumption which leads to changes in the raw material > requirement. The operator would run the calculation a number of times a day > because of availability materials or changes in the consumption and > allowances > or replacement of the materials itself. With the competitiveness in the > world > market, a factory has to send out their orders to suppliers within 24 hours > from > receiving the confirmed orders from the customers. Missing the 24 hours limit > normally means that the material is already out stock from suppliers and > delays > ensues. > > ** Some Design Constraints > The initial design of this system had J running on the server as a Windows > Service. This lead to J using up 100% of only 1 CPU of the 4 CPU application > server. With concurrent users increasing, the Windows Service started queuing > up > requests that the web server started timing-out the requests … even though > they > were still running and would complete successfully. Since the web server > already gave-up, the client was notified that an error occurred and the users > reaction was to submit the same operation again and further slowing things > down. > It was a vicious circle. > > What I did was abandoned the server side concept entirely and moved the > processing on the client side. It is now running in this configuration, where > the client system downloads all the data that it needs and computes it and J > creates SQL statements that would insert, delete and/or update the database > records. > > When I first decided to change my execution design strategy, I had an idea > that > they may be some performance problems but I was thinking that I’ll just burn > the > bridge when I get there. So after more than half a year after implementation > I > encountered my first performance problem. You see, client machines on the > factory were not really that powerful. A typical setup would be with the > following specs: > Windows XP Professional > 1.0GHz CPU > 20GB HD > 384 MB RAM > > Normal J usage peaks at 70MB RAM but last month, there were some RMR > calculation > that the user says doesn’t finish. So I investigated and found out that > because > of the number of materials involved in the order, J usage was peaking at > 130MB > RAM. I then timed it and it took more than an hour to finish with J using up > the > CPU. So now, some users would kill the J process from the Task Manager and > would > complain that the system is “Hanged”. :D > > For now, I’m trying to put in a progress dialog box into the mix (Oleg’s > Progress Dialog to be precise) as an immediate solution but I am looking into > an > “easy” solution of running the code on the server. What I’m looking for are J > libraries that would make it relatively easy to implement execution of J > scripts > on the server side (web server or application server behind the web server). > Right now I’m looking at JHP but I’m still doing the leg work since I really > need to complete the QA documentation requirements. As I can see the focus was on synchronous processing either at the server or at the client. However, what we see here is an example of a long-running processes-- anything that exceeds finite amount of user attention for an atomic operation. Here's a design pattern for asynchronous scalable distributed process management. Asynchronous means that process is broken into life-cycle operations steps and corresponding operations: initiate, suspend, resume, abort, get status etc; statuses: waiting, running, suspended, success, error. The GUI needs to be decoupled from process execution and only interact indirectly with quick atomic status query and change operations. User session (client application) needs to be decoupled from server, which would allow to login/logout any time, continue to observe server operations running independently. Server process management is done with a queue that maintains these statuses and mediates communication between the client request handler, worker processes manager, and worker processes. Queue item also contains parameters. Scalability is made easy with such queue model as the managers and handlers can be deployed across many commodity machines while processing the same queue. Scheduling allows to uses this process management framework for off-line batch execution. Configurable load-balancing and throttling ensures even spread of tasks over available hardware. To implement (parts of) this, an existing framework could be used, such as WWF in .NET, BPEL in Java, to some extent Quatz scheduler in Java. But you want to make sure all aspects above are satisfied. So it may be better and simpler to implement your own solution using a database and combination of standard framework components: ASP.NET, command-line executables, COM+ or J2EE apps etc. ---------------------------------------------------------------------- For information about J forums see http://www.jsoftware.com/forums.htm
---------------------------------------------------------------------- For information about J forums see http://www.jsoftware.com/forums.htm
