> instance where ADO.NET choked on a 25,000 You cleverly used active verb aspect here. Though, I am sure there's literature on how to make efficient bulk operations in ADO.NET.
Regarding client-side J processing, although progress bar is a nice indicator, users may want to have more control over execution. Although J does not support multi-threadedness, so you could run a child J process, controlled with a parent process with GUI. They can communicate using mapped memory. You don't wait until the process finishes, but query the status and use control that the child process polls periodically. (Basically, that's how Break is done in J.) So the parent process is free to update the GUI and process user events. The status updates can be polled by timer event. This would allow you to practice process management without the the database queue. If successful, you can then manage more than one child process. > From: Alex Rufon <[EMAIL PROTECTED]> > > I was re-reading this and I saw this part: > > What I did was abandoned the server side concept entirely and moved > > the processing on the client side. It is now running in this > > configuration, where the client system downloads all the data that it > > needs and computes it and J creates SQL statements that would insert, > > delete and/or update the database records. > > My statement implies that J only creates the SQL statement. This is > incorrect. > Specifically, the C# application retrieves data from the SQL server over the > internet and submit all data into the current J session. J now computes > everything and just returns a string containing 3 type of SQL-DDL statements > which are DELETE, INSERT and UPDATE statements. The C# calling application > now > has the responsibility of executing/submitting this SQL statements to the > MS-SQL > Server. Hahaha, I actually saw and instance where ADO.NET choked on a 25,000 > SQL-DDL statement generated by J in less than a minute. > > -----Original Message----- > From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On > Behalf Of Alex Rufon > Sent: Thursday, August 21, 2008 10:30 PM > To: General forum > Subject: RE: [Jgeneral] Successfully Migrated to J602 from J601 > > Yes your right. The system was designed to be synchronous. > > When I was reading your suggestion, I actually thought about just letting the > client insert a record to a database table which is read by a server side > application that is invoked by windows scheduler. :) It would be so simple > that > J would just update the record periodically and the client application will > also > just poll that same database table for its specific request. Using the > database > would satisfy all the requirements that you specified. > > Well, I was thinking about this while listening to Bob Dylan's "All Along The > Watchtower" then a though occurred to me. I may be overlooking a crucial > step. > You see, the RMR calculation is actually composed of computation by > categories. > Categories are by Color, by Size, by Destination, by Color and Size, by Size > and > Destination and a lot more combination. This implies that materials to be > computed by color are expanded to the number of colors. Like there are 2 > fabrics > that varies by 4 colors. I actually end up with 8 records. So materials that > vary by Size and Destination (2 fabrics, 3 sizes - s/m/l, 2 destination - > US/Japan) would have a 12 (2*3*2) number of records. There variations are > simple > and would only take J less than 3 minutes to compute. The RMR calculation > that > was taking more than an hour was the order which require the all three > calculation: > 1. Color by Size by Sub-Style > 2. Color by Size by Purchase Order > 3. Color by Size by Sub-Style by Purchase Order > > Looking at my J scripts, I fear that my code is not using J in its best form. > My > plan now is to still implement the progress dialog initially and spend the > sometime on "optimizing" the code again. If this still fails (still > un-acceptable performance) ... thats the time I'll consider running the > script > on the server side again. :) > > -----Original Message----- > From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On > Behalf Of Oleg Kobchenko > Sent: Thursday, August 21, 2008 4:22 PM > To: General forum > Subject: Re: [Jgeneral] Successfully Migrated to J602 from J601 > > > From: Alex Rufon > > > > > **J Usage > > For this project, we used J to primarily calculate the following: > > 1. Raw Material Consumption – computes how many items of each material > > is needed to make one garment of a certain style, size and color. > > 2. Raw Material Requirement – based on the raw material consumption, J > > computes for the actual number of number of materials that are needed > > to be ordered from the supplier based on the number of actual garments > > ordered > by the customer. > > J was a life saver because of its inherent capability to do matrix > calculations. > > Original computations made in AS400 and VB was taking some time to > > complete because each material needed has to be looped through one by > > one. The number of materials not only includes the fabric but > > everything from thread, buttons, tags to adhesive tapes, plastic bags > > and packing cartons. The challenge here is how to make the computation > > run faster because of the iterative process of coming up with the raw > > material consumption which leads to changes in the raw material > > requirement. The operator would run the calculation a number of times > > a day because of availability materials or changes in the consumption > > and allowances or replacement of the materials itself. With the > > competitiveness in the world market, a factory has to send out their > > orders to suppliers within 24 hours from receiving the confirmed > > orders from the customers. Missing the 24 hours limit normally means that > > the > material is already out stock from suppliers and delays ensues. > > > > ** Some Design Constraints > > The initial design of this system had J running on the server as a > > Windows Service. This lead to J using up 100% of only 1 CPU of the 4 > > CPU application server. With concurrent users increasing, the Windows > > Service started queuing up requests that the web server started > > timing-out the requests … even though they were still running and > > would complete successfully. Since the web server already gave-up, > > the client was notified that an error occurred and the users reaction was > > to > submit the same operation again and further slowing things down. > > It was a vicious circle. > > > > What I did was abandoned the server side concept entirely and moved > > the processing on the client side. It is now running in this > > configuration, where the client system downloads all the data that it > > needs and computes it and J creates SQL statements that would insert, > > delete and/or update the database records. > > > > When I first decided to change my execution design strategy, I had an > > idea that they may be some performance problems but I was thinking > > that I’ll just burn the bridge when I get there. So after more than > > half a year after implementation I encountered my first performance > > problem. You see, client machines on the factory were not really that > > powerful. A typical setup would be with the following specs: > > Windows XP Professional > > 1.0GHz CPU > > 20GB HD > > 384 MB RAM > > > > Normal J usage peaks at 70MB RAM but last month, there were some RMR > > calculation that the user says doesn’t finish. So I investigated and > > found out that because of the number of materials involved in the > > order, J usage was peaking at 130MB RAM. I then timed it and it took > > more than an hour to finish with J using up the CPU. So now, some > > users would kill the J process from the Task Manager and would > > complain that the system is “Hanged”. :D > > > > For now, I’m trying to put in a progress dialog box into the mix > > (Oleg’s Progress Dialog to be precise) as an immediate solution but I > > am looking into an “easy” solution of running the code on the server. > > What I’m looking for are J libraries that would make it relatively > > easy to implement execution of J scripts on the server side (web server or > application server behind the web server). > > Right now I’m looking at JHP but I’m still doing the leg work since I > > really need to complete the QA documentation requirements. > > As I can see the focus was on synchronous processing either at the server or > at > the client. However, what we see here is an example of a long-running > processes-- anything that exceeds finite amount of user attention for an > atomic > operation. > Here's a design pattern for asynchronous scalable distributed process > management. > > Asynchronous means that process is broken into life-cycle operations steps > and > corresponding operations: initiate, suspend, resume, abort, get status etc; > statuses: waiting, running, suspended, success, error. The GUI needs to be > decoupled from process execution and only interact indirectly with quick > atomic > status query and change operations. > > User session (client application) needs to be decoupled from server, which > would > allow to login/logout any time, continue to observe server operations running > independently. > > Server process management is done with a queue that maintains these statuses > and > mediates communication between the client request handler, worker processes > manager, and worker processes. Queue item also contains parameters. > > Scalability is made easy with such queue model as the managers and handlers > can > be deployed across many commodity machines while processing the same queue. > > Scheduling allows to uses this process management framework for off-line > batch > execution. Configurable load-balancing and throttling ensures even spread of > tasks over available hardware. > > To implement (parts of) this, an existing framework could be used, such as > WWF > in .NET, BPEL in Java, to some extent Quatz scheduler in Java. > But you want to make sure all aspects above are satisfied. > So it may be better and simpler to implement your own solution using a > database > and combination of standard framework components: ASP.NET, command-line > executables, COM+ or J2EE apps etc. > > > > ---------------------------------------------------------------------- > For information about J forums see http://www.jsoftware.com/forums.htm ---------------------------------------------------------------------- For information about J forums see http://www.jsoftware.com/forums.htm
