Hi guys, Just adding my thoughts...
Tibor Biro wrote: >> If we built a decent programming interface, I think we could leave >> this as a configurable option on the Manager or Executors. Maybe >> something that we could even have auto-select based upon the current >> grid configuration and Executor preferences. (Or even allow for a >> hybrid solution, with some Executors mirroring, and others just >> accessing the mirrored copies.) >> >> Because which implementation used really does depend on the >> application and administrative preferences. A machine used for >> development or CAD visualization or as a server should minimize the >> Executor's memory usage, but for a machine used by "management" (web >> browsing, email, wordprocessing and spreadsheet), memory usage by the >> Executor process is less critical. >> > > [Tibor Biro] Yes, this is what I had in mind too. In some applications it > makes sense to mirror while in others it makes sense to spread. Ideally the > Grid would have to be able to support both. > [Krishna] : Yes, I agree that this would be a flexible option, and the grid does need to support both. > >> But, I think we're agreed on the fact that shared memory, via some >> implementation, is a real feature that's desired. >> > > [Tibor Biro] Agreed. So I'll start by adding a small feature request to > sourceforge. I can take a stab at adding a Manager-centric implementation to > see how things go. Then we can build on that. > > Nice to see some life on the list :). > > Tibor [Krishna]: Just to let you know, our group has started research on two issues mentioned below, which are similar to what is being discussed here. In addition there is a .Net based distributed file system, which provides various useful features, (dotDFS), which appears in the Grid2006 conference. The author of this work has been in touch with us and is very interested to see the integration of dotDFS and Alchemi. However, we still haven't done any real implementation. Back to the work we are doing: 1. a .Net based Dataflow engine for designing and running inter-linked tasks on Alchemi. 2. P2P-based Alchemi Grid federation (using the Pastry protocol) to connect multiple Managers. (The Managers themselves will still manage 'n' Executors...). You guys may want to get in touch with the guys in our group who are doing the work mentioned, so efforts can be co-ordinated ,and there is least duplication of the software produced. Another thing I wanted to mention, was the architecture of Alchemi and possible modifications to it. Currently the entire communication infrastructure is kind of "hard-coded" to use tcp-binary remoting, and a "grid-thread" programming model. This limits distributed applications running on Alchemi to independent tasks which are threads. It also limits Alchemi to the Microsoft platform (since we use remoting using the binaryformatter, which is not compatible with Mono, for example). It might be worthwhile, to think of aligning all of our thoughts, and efforts, to modify / abstract out some parts of the design, so that components in Alchemi are clearly demarcated into things like: - a communication layer - a management layer (for managing nodes, administration) - a security layer - a data sharing layer (and a possibly seperate memory-sharing layer) - an application + scheduling layer - fault tolerance (via persistence at both Manager, Executor) - ...etc... When I say "layer"... each of these could just be one component. But the aim is to have flexibility and extensibility at various levels. For example : a recently concluded work by some students in our group, involved building a BSP (bulk-synchronous-processing) programing model on top of Alchemi. The implementation of this had to deal with some issues where Alchemi's design was unsuitable for BSP (and MPI-like) programming models. The Dataflow engine I mentioned above, will also benefit hugely from a clearer seperation of these components, and we can imagine other programming models being build on top of the Alchemi platform. In other words, specifically, the primitives / APIs available on the Manager and Executor currently are too oriented towards just grid-threads, and this limits extensibility to an extent. We should look to reduce the number of primitives on the Manager / Executor, and make the grid-thread APIs a seperate module, and define a clean way to plugin other such modules. In doing this, we can re-use a lot of existing work, to build the layered architecture for Alchemi. Things such as GridFTP.Net, WSRF.Net (http://www.cs.virginia.edu/~gsw2c/wsrf.net.html) , dotDFS ... and so on. Anyway, I will put in some more thoughts after discussing with the guys in our group... Cheers Krishna. Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 _______________________________________________ Alchemi-developers mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/alchemi-developers
