Hi Jonathan, The simple implementation would be to store the shared application data in the Manager which would in turn store it in the storage (memory, SQL or whatever is implemented). It could look more or less like a hash table with key/value pairs being written or set by each Executor. This can be done today pretty easily and it would be fit small amounts of data. If anybody needs this let's add it to the Feature Requests on sourceforge.
Looking at a distributed hashtable fit for caching the data from massive calculations... Clearly not feasible with a single Manager setup... This would require the Executors to be working in a P2P-like setup where they can more-or-less discover each other's presence. It would make sense to have something like this only on a fast LAN so the ability to work in both P2P and Manager-centric modes is required. I think it is doable though with some effort. A rough path would look like this: 1. Add multiple Manager support on Executors. This would allow an Executor to negotiate work with a number of Managers. 2. Add executor and manager advertising and discovery services. This would allow Executors and Managers to find each other in a LAN. 3. At this point we have a P2P capable grid, nodes come up and go as they please. One problem will be redundancy on the Manager side as an application owner could talk to multiple managers and it should somehow be able to find its application even if it moves through the grid. 4. Add cooperative Manager capabilities. This should allow Managers to share their data and their results. If a Manager goes down then it can try to transfer its knowledge to another Manager. One special example would be Managers that share the same storage (like a SQL server cluster). 5. Add a shared memory layer to the Alchemi framework. This would be based on an algorithm that allows us to "map" memory pieces to existing computers. Now if you think back to the Managers having a primitive way of storing data in their storage (memory or database) this would be a way of putting a tag on each Manager's storage. Since the Managers could cooperate at this point the data could also get some redundancy. In a setup as above you can have various combinations that would fit a lot of the possible applications. Some quick examples: - Distributed hashtable based cache. Configure the nodes in P2P mode. Setup the Managers on each node to use InMemory storage. Since the data is only cached for speed purposes a node going down only means that some calculations would have to be performed again on the fly. The Managers might cooperate to share some data to prevent some recalculations but the exact mix depends on the application and how expensive it was to get to the data in the first place. - The application needs the grid for CPU power and to store the results in the shared storage. You can either use P2P mode or have a handful of Managers. Setup the Managers with a shared SQL Cluster. For ultimate scalability have several SQL clusters. Regards, Tibor > -----Original Message----- > From: Jonathan Mitchem [mailto:[EMAIL PROTECTED] > Sent: Tuesday, July 04, 2006 5:27 PM > To: Tibor Biro > Cc: Atali DAOUD; [email protected] > Subject: Re: [Alchemi-developers] data in App.config > > This question brings up a problem I've seen repeatedly, both in my own > work and in others, so I've been brainstorming a bit. > > Perhaps this is out of the scope of Alchemi, and I really don't have a > good way of doing it yet... but would it make sense to provide some > sort of GApplication wide shared memory, so that we could share common > data that each of the GThreads may read and access? > > Basically, the functionality of storing a database (i.e., concurrency > handling without explicitly handling mutexes and semaphores) but > without the overhead of a database, and with the ease of use of > accessing a .NET object within code. (I'm personally not much of a fan > of mixing C#/VB.NET and SQL.) > > If we could provide some application wide [virtual] object pool, and > add objects to this pool that could be accessed by each Executor, I > think that would be very convenient. (And very tricky to implement > and ensure that every executor has the same data...) > > A database has the advantage of higher-level built-in concurrency > handling; a common file has the advantage of lower overhead and OS > level concurrency handling. However, both solutions have the > potential to lead to bandwidth and processor bottlenecks on the > machine hosting the data. Distributing the data among the nodes using > some sort of shared object would be nice, but synchronization issues > could be a nightmare. However, there are some solutions such as > broadcasting DataChanged messages with the specific data that changed, > including timestamps, and then using something of a Compare and Swap > operation to ensure we've got the most recent data whenever we access > the object locally. > > Thoughts? > > Jonathan > > On 7/4/06, Tibor Biro <[EMAIL PROTECTED]> wrote: > > Hi, > > > > You can use several scenarios: > > 1. Put the data in the app.config of the owner application. When the > GThread > > is created read in the data and set it as data in the GThread. > > 2. Put the data in a database (or file) that is accessible from all > > executors. > > > > Let me know if you need more details. > > > > Regards, > > Tibor > > > > > > > -----Original Message----- > > > From: [EMAIL PROTECTED] > [mailto:alchemi- > > > [EMAIL PROTECTED] On Behalf Of Atali DAOUD > > > Sent: Tuesday, July 04, 2006 2:49 PM > > > To: [email protected] > > > Subject: [Alchemi-developers] data in App.config > > > > > > Hi everybody, > > > > > > I have an application with a config file. I would like to transform it > > > for using Alchemi.Net. > > > > > > I will convert App.exe to App.dll and send it to the executor. > > > > > > My question is where I have to store my datas from App.config in the > > > executor? in Alchemi.ExecutorExec.exe.config ? > > > > > > Can someone help me . > > > > > > > > > > > > Using Tomcat but need to do more? Need to support web services, > security? > > > Get stuff done quickly with pre-integrated technology to make your job > > > easier > > > Download IBM WebSphere Application Server v.1.0.1 based on Apache > Geronimo > > > http://sel.as- > us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > > > _______________________________________________ > > > Alchemi-developers mailing list > > > [email protected] > > > https://lists.sourceforge.net/lists/listinfo/alchemi-developers > > > > > > Using Tomcat but need to do more? Need to support web services, > security? > > Get stuff done quickly with pre-integrated technology to make your job > easier > > Download IBM WebSphere Application Server v.1.0.1 based on Apache > Geronimo > > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > > _______________________________________________ > > Alchemi-developers mailing list > > [email protected] > > https://lists.sourceforge.net/lists/listinfo/alchemi-developers > > Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 _______________________________________________ Alchemi-developers mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/alchemi-developers
