Hi All, I have finished the initial version of the ZK integration. Now we can start multiple thrift gfac services (still the communication between orchestrator and gfac is RPC) and orchestrator submit jobs to multiple gfac nodes.
I can kill a gfac node and orchestrator will make sure jobs are not lost, it simply take those jobs and re-submit to gfac. Since GFac is a generic framework and we have multiple plugins developed for that framework checkpointing the plugin is up to the plugin developers but gfac checkpoints whether those plugins invoked or not. I have introduced a new interface for plugin development called Recoverable (RecoverableHandlers and RecoverableProvider). So state-full plugins has to implement their recover method and gfac framework will make sure it will be invoked during a re-run scenario. If a plugin is not recoverable and already ran(can be found using framework checkpointing) during the re-run that plugin will not be invoked. For now I just implemented recoverability to few plugins and I have tested submitting a job to trestles and let it submit and come to monitoring state and kill that gfac instance. Now Orchestrator pick that execution and re-submit to another gfac node and that gfac node does not re-run that job to the computing resource, but simply start monitoring once the job is done outputs are downloaded from the original output location. When a particular experiment is finished all the ZK data is removed. At this point following things needs to be done, 1. Figure out all the state-full handlers/providers and implement recoverability, Ex: Input handler is transfering 1000 files and with 500 files gfac instance crashed, during the re-run it should be able to tranfer from 501 file. Same logic can be applied to a single huge file. Those things are completely up to the plugin developer. 2. Then we have to do remove the RPC invocation and make gfac nodes as worker nodes. Regards Lahiru On Wed, Jun 18, 2014 at 12:11 PM, Lahiru Gunathilake <[email protected]> wrote: > Hi Eran, > > > On Tue, Jun 17, 2014 at 4:06 PM, Eran Chinthaka Withana < > [email protected]> wrote: > >> Storm has a Kafka spout which manages the cursor location (pointer to the >> head of the queue representing the next message to be processed) inside >> ZK. >> Each storm spout instance uses this information to get the next item to >> process. Storm kafka spout won't advance to the next message until it gets >> an ack from the storm topology. >> > If we have 10 jobs in the queue and 5 GFAC instances picked 1 at a time > and successfully submitted and have to start taking rest of the jobs. But > all 5 GFAC instances are responsible for initially picked 5 jobs because > they are still running and gfac instances are monitoring them until its > done but at the same time we have to move the cursor to pick other jobs > too. > > If we Ack and moved the cursor just after submission without waiting until > the job is actually finished how are we going to know which gfac is > monitoring which set of jobs ? > > I am not getting how achieve above requirement with this suggestion. May > be I am missing something here. > > Regards > Lahiru > >> >> So, if there is an exception in the topology and ack is sent only by the >> last bolt, then storm bolt make sure all messages are processed since >> exceptions won't generate acks. >> >> Thanks, >> Eran Chinthaka Withana >> >> >> On Tue, Jun 17, 2014 at 12:30 PM, Lahiru Gunathilake <[email protected]> >> wrote: >> >> > Hi Eran, >> > >> > I think I should take back my last email. When I carefully look at >> storm I >> > have following question. >> > >> > How are we going to store the Job statuses and relaunch the jobs which >> was >> > running in failure nodes ? Its true that storm is starting new workers >> but >> > there should be a way to find missing jobs by someone in the system. >> Since >> > we are not having a data stream there is no use to start new workers >> unless >> > we handler the missing jobs. I think we need to have a better control of >> > our component and persist the states of jobs each GFAC node is handling. >> > Directly using zookeeper will let us to do a proper fault tolerance >> > implementation. >> > >> > Regards >> > Lahiru >> > >> > >> > >> > On Tue, Jun 17, 2014 at 3:14 PM, Lahiru Gunathilake <[email protected]> >> > wrote: >> > >> > > Hi Supun, >> > > >> > > I think in this usecase we only use storm topology to do the >> > communication >> > > among workers and we are completely ignoring the stream processing >> part. >> > > Orchestrator will talk to Nimbus and GFAC nodes will be Worker nodes >> in >> > the >> > > storm topology. But I think we can achieve extremely fault tolerance >> > system >> > > by directly using storm based on following statement in storm site >> with >> > > minimum changes in airavata. >> > > >> > > Additionally, the Nimbus daemon and Supervisor daemons are fail-fast >> and >> > > stateless; all state is kept in Zookeeper or on local disk. This means >> > you >> > > can kill -9 Nimbus or the Supervisors and they’ll start back up like >> > > nothing happened. This design leads to Storm clusters being incredibly >> > > stable. >> > > >> > > >> > > >> > > >> > > On Tue, Jun 17, 2014 at 3:02 PM, Supun Kamburugamuva < >> [email protected]> >> > > wrote: >> > > >> > >> Hi Eran, >> > >> >> > >> I'm using Storm every day and this is one of the strangest things >> I've >> > >> heard about using Storm. My be there are more use cases for Storm >> other >> > >> than Distributed Stream processing. AFAIK the Bolts, spouts are >> built to >> > >> handle a stream of events that doesn't take much time to process. In >> > >> Airavata we don't process the messages. Instead we run experiments >> based >> > >> on >> > >> the commands given. >> > >> >> > >> If you want process isolation, distributed execution, cluster >> resource >> > >> management Yarn would be a better thing to explore. >> > >> >> > >> Thanks, >> > >> Supun.. >> > >> >> > >> >> > >> On Tue, Jun 17, 2014 at 2:27 PM, Eran Chinthaka Withana < >> > >> [email protected]> wrote: >> > >> >> > >> > Hi Lahiru, >> > >> > >> > >> > good summarization. Thanks Lahiru. >> > >> > >> > >> > I think you are trying to stick to a model where Orchestrator >> > >> distributing >> > >> > to work for GFac worker and trying to do the impedance mismatch >> > through >> > >> a >> > >> > messaging solution. If you step back and think, we don't even want >> the >> > >> > orchestrator to handle everything. From its point of view, it >> should >> > >> submit >> > >> > jobs to the framework, and will wait or get notified once the job >> is >> > >> done. >> > >> > >> > >> > There are multiple ways of doing this. And here is one method. >> > >> > >> > >> > Orchestrator submits all its jobs to Job queue (implemented using >> any >> > MQ >> > >> > impl like Rabbit or Kafka). A storm topology is implemented to >> dequeue >> > >> > messages, process them (i.e. submit those jobs and get those >> executed) >> > >> and >> > >> > notify the Orchestrator with the status (either through another >> > >> > JobCompletionQueue or direct invocation). >> > >> > >> > >> > With this approach, the MQ provider will help to match impedance >> > between >> > >> > job submission and consumption. Storm helps with worker >> coordination, >> > >> load >> > >> > balancing, throttling on your job execution framework, worker pool >> > >> > management and fault tolerance. >> > >> > >> > >> > Of course, you can implement this based only on ZK and handle >> > everything >> > >> > else on your own but storm had done exactly that with the use of ZK >> > >> > underneath. >> > >> > >> > >> > Finally, if you go for a model like this, then even beyond job >> > >> submission, >> > >> > you can use the same model to do anything within the framework for >> > >> internal >> > >> > communication. For example, the workflow engine will submit its >> jobs >> > to >> > >> > queues based on what it has to do. Storm topologies exists for each >> > >> queues >> > >> > to dequeue messages and carry out the work in a reliable manner. >> > >> Consider >> > >> > this as mini-workflows within a larger workflow framework. >> > >> > >> > >> > We can have a voice chat if its more convenient. But not at 7am >> PST :) >> > >> > >> > >> > >> > >> > Thanks, >> > >> > Eran Chinthaka Withana >> > >> > >> > >> > >> > >> > On Tue, Jun 17, 2014 at 10:12 AM, Lahiru Gunathilake < >> > [email protected] >> > >> > >> > >> > wrote: >> > >> > >> > >> > > Hi All, >> > >> > > >> > >> > > Ignoring the tool that we are going to use to implement fault >> > >> tolerance I >> > >> > > have summarized the model we have decided so far. I will use the >> > tool >> > >> > name >> > >> > > as X, we can use Zookeeper or some other implementation. >> Following >> > >> design >> > >> > > assume tool X and Registry have high availability. >> > >> > > >> > >> > > 1. Orchestrator and GFAC worker node communication is going to be >> > >> queue >> > >> > > based and tool X is going to be used for this communication. (We >> > have >> > >> to >> > >> > > implement this with considering race condition between different >> > gfac >> > >> > > workers). >> > >> > > 2. We are having multiple instances of GFAC which are identical >> (In >> > >> > future >> > >> > > we can group gfac workers). Existence of each worker node is >> > >> identified >> > >> > > using X. If node goes down orchestrator will be notified by X. >> > >> > > 3. When a particular request comes and accepted by one gfac >> worker >> > >> that >> > >> > > information will be replicated in tool X and a place where this >> > >> > information >> > >> > > is persisted even the worker failed. >> > >> > > 4. When a job comes to a final state like failed or cancelled or >> > >> > completed >> > >> > > above information will be removed. So at a given time >> orchestrator >> > can >> > >> > poll >> > >> > > active jobs in each worker by giving a worker ID. >> > >> > > 5. Tool X will make sure that when a worker goes down it will >> notify >> > >> > > orchestrator. During a worker failure, based on step 3 and 4 >> > >> orchestrator >> > >> > > can poll all the active jobs of that worker and do the same thing >> > >> like in >> > >> > > step 1 (store the experiment ID to the queue) and gfac worker >> will >> > >> pick >> > >> > the >> > >> > > jobs. >> > >> > > >> > >> > > 6. When GFAC receive a job like in step 5 it have to carefully >> > >> evaluate >> > >> > the >> > >> > > state from registry and decide what to be done (If the job is >> > pending >> > >> > then >> > >> > > gfac just have to monitor, if job state is like input transferred >> > not >> > >> > even >> > >> > > submitted gfac has to execute rest of the chain and submit the >> job >> > to >> > >> the >> > >> > > resource and start monitoring). >> > >> > > >> > >> > > If we can find a tool X which supports all these features and >> tool >> > >> itself >> > >> > > is fault tolerance and support atomicity, high availability and >> > simply >> > >> > API >> > >> > > to implement we can use that tool. >> > >> > > >> > >> > > WDYT ? >> > >> > > >> > >> > > Lahiru >> > >> > > >> > >> > > >> > >> > > On Mon, Jun 16, 2014 at 2:38 PM, Supun Kamburugamuva < >> > >> [email protected]> >> > >> > > wrote: >> > >> > > >> > >> > > > Hi Lahiru, >> > >> > > > >> > >> > > > Before moving with an implementation it may be worth to >> consider >> > >> some >> > >> > of >> > >> > > > the following aspects as well. >> > >> > > > >> > >> > > > 1. How to report the progress of an experiment as state in >> > >> ZooKeeper? >> > >> > > What >> > >> > > > happens if a GFac instance crashes while executing an >> experiment? >> > >> Are >> > >> > > there >> > >> > > > check-points we can save so that another GFac instance can take >> > >> over? >> > >> > > > 2. What is the threading model of GFac instances? (I consider >> this >> > >> as a >> > >> > > > very important aspect) >> > >> > > > 3. What are the information needed to be stored in the >> ZooKeeper? >> > >> You >> > >> > may >> > >> > > > need to store other information about an experiment apart from >> its >> > >> > > > experiment ID. >> > >> > > > 4. How to report errors? >> > >> > > > 5. For GFac weather you need a threading model or worker >> process >> > >> model? >> > >> > > > >> > >> > > > Thanks, >> > >> > > > Supun.. >> > >> > > > >> > >> > > > >> > >> > > > >> > >> > > > >> > >> > > > >> > >> > > > On Mon, Jun 16, 2014 at 2:22 PM, Lahiru Gunathilake < >> > >> [email protected] >> > >> > > >> > >> > > > wrote: >> > >> > > > >> > >> > > > > Hi All, >> > >> > > > > >> > >> > > > > I think the conclusion is like this, >> > >> > > > > >> > >> > > > > 1, We make the gfac as a worker not a thrift service and we >> can >> > >> start >> > >> > > > > multiple workers either with bunch of providers and handlers >> > >> > configured >> > >> > > > in >> > >> > > > > each worker or provider specific workers to handle the class >> > path >> > >> > > issues >> > >> > > > > (not the common scenario). >> > >> > > > > >> > >> > > > > 2. Gfac workers can be configured to watch for a given path >> in >> > >> > > zookeeper, >> > >> > > > > and multiple workers can listen to the same path. Default >> path >> > >> can be >> > >> > > > > /airavata/gfac or can configure paths like >> /airavata/gfac/gsissh >> > >> > > > > /airavata/gfac/bes. >> > >> > > > > >> > >> > > > > 3. Orchestrator can configure with a logic to store >> experiment >> > >> IDs in >> > >> > > > > zookeeper with a path, and orchestrator can be configured to >> > >> provider >> > >> > > > > specific path logic too. So when a new request come >> orchestrator >> > >> > store >> > >> > > > the >> > >> > > > > experimentID and these experiments IDs are stored in Zk as a >> > >> queue. >> > >> > > > > >> > >> > > > > 4. Since gfac workers are watching they will be notified and >> as >> > >> supun >> > >> > > > > suggested can use a leader selection algorithm[1] and one >> gfac >> > >> worker >> > >> > > > will >> > >> > > > > take the leadership for each experiment. If there are gfac >> > >> instances >> > >> > > for >> > >> > > > > each provider same logic will apply among those nodes with >> same >> > >> > > provider >> > >> > > > > type. >> > >> > > > > >> > >> > > > > [1] >> > http://curator.apache.org/curator-recipes/leader-election.html >> > >> > > > > >> > >> > > > > I would like to implement this if there are no objections. >> > >> > > > > >> > >> > > > > Lahiru >> > >> > > > > >> > >> > > > > >> > >> > > > > On Mon, Jun 16, 2014 at 11:51 AM, Supun Kamburugamuva < >> > >> > > [email protected] >> > >> > > > > >> > >> > > > > wrote: >> > >> > > > > >> > >> > > > > > Hi Marlon, >> > >> > > > > > >> > >> > > > > > I think you are exactly correct. >> > >> > > > > > >> > >> > > > > > Supun.. >> > >> > > > > > >> > >> > > > > > >> > >> > > > > > On Mon, Jun 16, 2014 at 11:48 AM, Marlon Pierce < >> > >> [email protected]> >> > >> > > > wrote: >> > >> > > > > > >> > >> > > > > > > Let me restate this, and please tell me if I'm wrong. >> > >> > > > > > > >> > >> > > > > > > Orchestrator decides (somehow) that a particular job >> > requires >> > >> > > > JSDL/BES, >> > >> > > > > > so >> > >> > > > > > > it places the Experiment ID in Zookeeper's >> > >> > /airavata/gfac/jsdl-bes >> > >> > > > > node. >> > >> > > > > > > GFAC servers associated with this instance notice the >> > update. >> > >> > The >> > >> > > > > first >> > >> > > > > > > GFAC to claim the job gets it, uses the Experiment ID to >> get >> > >> the >> > >> > > > > detailed >> > >> > > > > > > information it needs from the Registry. ZooKeeper >> handles >> > the >> > >> > > > locking, >> > >> > > > > > etc >> > >> > > > > > > to make sure that only one GFAC at a time is trying to >> > handle >> > >> an >> > >> > > > > > experiment. >> > >> > > > > > > >> > >> > > > > > > Marlon >> > >> > > > > > > >> > >> > > > > > > >> > >> > > > > > > On 6/16/14, 11:42 AM, Lahiru Gunathilake wrote: >> > >> > > > > > > >> > >> > > > > > >> Hi Supun, >> > >> > > > > > >> >> > >> > > > > > >> Thanks for the clarification. >> > >> > > > > > >> >> > >> > > > > > >> Regards >> > >> > > > > > >> Lahiru >> > >> > > > > > >> >> > >> > > > > > >> >> > >> > > > > > >> On Mon, Jun 16, 2014 at 11:38 AM, Supun Kamburugamuva < >> > >> > > > > > [email protected]> >> > >> > > > > > >> wrote: >> > >> > > > > > >> >> > >> > > > > > >> Hi Lahiru, >> > >> > > > > > >>> >> > >> > > > > > >>> My suggestion is that may be you don't need a Thrift >> > service >> > >> > > > between >> > >> > > > > > >>> Orchestrator and the component executing the >> experiment. >> > >> When a >> > >> > > new >> > >> > > > > > >>> experiment is submitted, orchestrator decides who can >> > >> execute >> > >> > > this >> > >> > > > > job. >> > >> > > > > > >>> Then it put the information about this experiment >> > execution >> > >> in >> > >> > > > > > ZooKeeper. >> > >> > > > > > >>> The component which wants to executes the experiment is >> > >> > listening >> > >> > > > to >> > >> > > > > > this >> > >> > > > > > >>> ZooKeeper path and when it sees the experiment it will >> > >> execute >> > >> > > it. >> > >> > > > So >> > >> > > > > > >>> that >> > >> > > > > > >>> the communication happens through an state change in >> > >> ZooKeeper. >> > >> > > > This >> > >> > > > > > can >> > >> > > > > > >>> potentially simply your architecture. >> > >> > > > > > >>> >> > >> > > > > > >>> Thanks, >> > >> > > > > > >>> Supun. >> > >> > > > > > >>> >> > >> > > > > > >>> >> > >> > > > > > >>> On Mon, Jun 16, 2014 at 11:14 AM, Lahiru Gunathilake < >> > >> > > > > > [email protected]> >> > >> > > > > > >>> wrote: >> > >> > > > > > >>> >> > >> > > > > > >>> Hi Supun, >> > >> > > > > > >>>> >> > >> > > > > > >>>> So your suggestion is to create a znode for each >> thrift >> > >> > service >> > >> > > we >> > >> > > > > > have >> > >> > > > > > >>>> and >> > >> > > > > > >>>> when the request comes that node gets modified with >> input >> > >> data >> > >> > > for >> > >> > > > > > that >> > >> > > > > > >>>> request and thrift service is having a watch for that >> > node >> > >> and >> > >> > > it >> > >> > > > > will >> > >> > > > > > >>>> be >> > >> > > > > > >>>> notified because of the watch and it can read the >> input >> > >> from >> > >> > > > > zookeeper >> > >> > > > > > >>>> and >> > >> > > > > > >>>> invoke the operation? >> > >> > > > > > >>>> >> > >> > > > > > >>>> Lahiru >> > >> > > > > > >>>> >> > >> > > > > > >>>> >> > >> > > > > > >>>> On Thu, Jun 12, 2014 at 11:50 PM, Supun Kamburugamuva >> < >> > >> > > > > > >>>> [email protected]> >> > >> > > > > > >>>> wrote: >> > >> > > > > > >>>> >> > >> > > > > > >>>> Hi all, >> > >> > > > > > >>>>> >> > >> > > > > > >>>>> Here is what I think about Airavata and ZooKeeper. In >> > >> > Airavata >> > >> > > > > there >> > >> > > > > > >>>>> are >> > >> > > > > > >>>>> many components and these components must be >> stateless >> > to >> > >> > > achieve >> > >> > > > > > >>>>> scalability and reliability.Also there must be a >> > >> mechanism to >> > >> > > > > > >>>>> >> > >> > > > > > >>>> communicate >> > >> > > > > > >>>> >> > >> > > > > > >>>>> between the components. At the moment Airavata uses >> RPC >> > >> calls >> > >> > > > based >> > >> > > > > > on >> > >> > > > > > >>>>> Thrift for the communication. >> > >> > > > > > >>>>> >> > >> > > > > > >>>>> ZooKeeper can be used both as a place to hold state >> and >> > >> as a >> > >> > > > > > >>>>> >> > >> > > > > > >>>> communication >> > >> > > > > > >>>> >> > >> > > > > > >>>>> layer between the components. I'm involved with a >> > project >> > >> > that >> > >> > > > has >> > >> > > > > > many >> > >> > > > > > >>>>> distributed components like AIravata. Right now we >> use >> > >> Thrift >> > >> > > > > > services >> > >> > > > > > >>>>> >> > >> > > > > > >>>> to >> > >> > > > > > >>>> >> > >> > > > > > >>>>> communicate among the components. But we find it >> > >> difficult to >> > >> > > use >> > >> > > > > RPC >> > >> > > > > > >>>>> >> > >> > > > > > >>>> calls >> > >> > > > > > >>>> >> > >> > > > > > >>>>> and achieve stateless behaviour and thinking of >> > replacing >> > >> > > Thrift >> > >> > > > > > >>>>> >> > >> > > > > > >>>> services >> > >> > > > > > >>>> >> > >> > > > > > >>>>> with ZooKeeper based communication layer. So I think >> it >> > is >> > >> > > better >> > >> > > > > to >> > >> > > > > > >>>>> explore the possibility of removing the Thrift >> services >> > >> > between >> > >> > > > the >> > >> > > > > > >>>>> components and use ZooKeeper as a communication >> > mechanism >> > >> > > between >> > >> > > > > the >> > >> > > > > > >>>>> services. If you do this you will have to move the >> state >> > >> to >> > >> > > > > ZooKeeper >> > >> > > > > > >>>>> >> > >> > > > > > >>>> and >> > >> > > > > > >>>> >> > >> > > > > > >>>>> will automatically achieve the stateless behaviour in >> > the >> > >> > > > > components. >> > >> > > > > > >>>>> >> > >> > > > > > >>>>> Also I think trying to make ZooKeeper optional is a >> bad >> > >> idea. >> > >> > > If >> > >> > > > we >> > >> > > > > > are >> > >> > > > > > >>>>> trying to integrate something fundamentally >> important to >> > >> > > > > architecture >> > >> > > > > > >>>>> as >> > >> > > > > > >>>>> how to store state, we shouldn't make it optional. >> > >> > > > > > >>>>> >> > >> > > > > > >>>>> Thanks, >> > >> > > > > > >>>>> Supun.. >> > >> > > > > > >>>>> >> > >> > > > > > >>>>> >> > >> > > > > > >>>>> On Thu, Jun 12, 2014 at 10:57 PM, Shameera >> Rathnayaka < >> > >> > > > > > >>>>> [email protected]> wrote: >> > >> > > > > > >>>>> >> > >> > > > > > >>>>> Hi Lahiru, >> > >> > > > > > >>>>>> >> > >> > > > > > >>>>>> As i understood, not only reliability , you are >> trying >> > >> to >> > >> > > > achieve >> > >> > > > > > >>>>>> some >> > >> > > > > > >>>>>> other requirement by introducing zookeeper, like >> health >> > >> > > > monitoring >> > >> > > > > > of >> > >> > > > > > >>>>>> >> > >> > > > > > >>>>> the >> > >> > > > > > >>>> >> > >> > > > > > >>>>> services, categorization with service implementation >> etc >> > >> ... >> > >> > . >> > >> > > In >> > >> > > > > > that >> > >> > > > > > >>>>>> case, i think we can get use of zookeeper's features >> > but >> > >> if >> > >> > we >> > >> > > > > only >> > >> > > > > > >>>>>> >> > >> > > > > > >>>>> focus >> > >> > > > > > >>>> >> > >> > > > > > >>>>> on reliability, i have little bit of concern, why >> can't >> > we >> > >> > use >> > >> > > > > > >>>>>> >> > >> > > > > > >>>>> clustering + >> > >> > > > > > >>>> >> > >> > > > > > >>>>> LB ? >> > >> > > > > > >>>>>> >> > >> > > > > > >>>>>> Yes it is better we add Zookeeper as a prerequisite >> if >> > >> user >> > >> > > need >> > >> > > > > to >> > >> > > > > > >>>>>> use >> > >> > > > > > >>>>>> it. >> > >> > > > > > >>>>>> >> > >> > > > > > >>>>>> Thanks, >> > >> > > > > > >>>>>> Shameera. >> > >> > > > > > >>>>>> >> > >> > > > > > >>>>>> >> > >> > > > > > >>>>>> On Thu, Jun 12, 2014 at 5:19 AM, Lahiru Gunathilake >> < >> > >> > > > > > >>>>>> [email protected] >> > >> > > > > > >>>>>> wrote: >> > >> > > > > > >>>>>> >> > >> > > > > > >>>>>> Hi Gagan, >> > >> > > > > > >>>>>>> >> > >> > > > > > >>>>>>> I need to start another discussion about it, but I >> had >> > >> an >> > >> > > > offline >> > >> > > > > > >>>>>>> discussion with Suresh about auto-scaling. I will >> > start >> > >> > > another >> > >> > > > > > >>>>>>> thread >> > >> > > > > > >>>>>>> about this topic too. >> > >> > > > > > >>>>>>> >> > >> > > > > > >>>>>>> Regards >> > >> > > > > > >>>>>>> Lahiru >> > >> > > > > > >>>>>>> >> > >> > > > > > >>>>>>> >> > >> > > > > > >>>>>>> On Wed, Jun 11, 2014 at 4:10 PM, Gagan Juneja < >> > >> > > > > > >>>>>>> >> > >> > > > > > >>>>>> [email protected] >> > >> > > > > > >>>> >> > >> > > > > > >>>>> wrote: >> > >> > > > > > >>>>>>> >> > >> > > > > > >>>>>>> Thanks Lahiru for pointing to nice library, added >> to >> > my >> > >> > > > > dictionary >> > >> > > > > > >>>>>>>> >> > >> > > > > > >>>>>>> :). >> > >> > > > > > >>>> >> > >> > > > > > >>>>> I would like to know how are we planning to start >> > >> multiple >> > >> > > > > servers. >> > >> > > > > > >>>>>>>> 1. Spawning new servers based on load? Some times >> we >> > >> call >> > >> > it >> > >> > > > as >> > >> > > > > > auto >> > >> > > > > > >>>>>>>> scalable. >> > >> > > > > > >>>>>>>> 2. To make some specific number of nodes available >> > >> such as >> > >> > > we >> > >> > > > > > want 2 >> > >> > > > > > >>>>>>>> servers to be available at any time so if one goes >> > down >> > >> > > then I >> > >> > > > > > need >> > >> > > > > > >>>>>>>> >> > >> > > > > > >>>>>>> to >> > >> > > > > > >>>> >> > >> > > > > > >>>>> spawn one new to make available servers count 2. >> > >> > > > > > >>>>>>>> 3. Initially start all the servers. >> > >> > > > > > >>>>>>>> >> > >> > > > > > >>>>>>>> In scenario 1 and 2 zookeeper does make sense but >> I >> > >> don't >> > >> > > > > believe >> > >> > > > > > >>>>>>>> >> > >> > > > > > >>>>>>> existing >> > >> > > > > > >>>>>>> >> > >> > > > > > >>>>>>>> architecture support this? >> > >> > > > > > >>>>>>>> >> > >> > > > > > >>>>>>>> Regards, >> > >> > > > > > >>>>>>>> Gagan >> > >> > > > > > >>>>>>>> On 12-Jun-2014 1:19 am, "Lahiru Gunathilake" < >> > >> > > > [email protected] >> > >> > > > > > >> > >> > > > > > >>>>>>>> >> > >> > > > > > >>>>>>> wrote: >> > >> > > > > > >>>>>>> >> > >> > > > > > >>>>>>>> Hi Gagan, >> > >> > > > > > >>>>>>>>> >> > >> > > > > > >>>>>>>>> Thanks for your response. Please see my inline >> > >> comments. >> > >> > > > > > >>>>>>>>> >> > >> > > > > > >>>>>>>>> >> > >> > > > > > >>>>>>>>> On Wed, Jun 11, 2014 at 3:37 PM, Gagan Juneja < >> > >> > > > > > >>>>>>>>> >> > >> > > > > > >>>>>>>> [email protected]> >> > >> > > > > > >>>>>>> >> > >> > > > > > >>>>>>>> wrote: >> > >> > > > > > >>>>>>>>> >> > >> > > > > > >>>>>>>>> Hi Lahiru, >> > >> > > > > > >>>>>>>>>> Just my 2 cents. >> > >> > > > > > >>>>>>>>>> >> > >> > > > > > >>>>>>>>>> I am big fan of zookeeper but also against >> adding >> > >> > multiple >> > >> > > > > hops >> > >> > > > > > in >> > >> > > > > > >>>>>>>>>> >> > >> > > > > > >>>>>>>>> the >> > >> > > > > > >>>>>>> >> > >> > > > > > >>>>>>>> system which can add unnecessary complexity. Here >> I >> > am >> > >> not >> > >> > > > able >> > >> > > > > to >> > >> > > > > > >>>>>>>>>> understand the requirement of zookeeper may be >> I am >> > >> > wrong >> > >> > > > > > because >> > >> > > > > > >>>>>>>>>> >> > >> > > > > > >>>>>>>>> of >> > >> > > > > > >>>> >> > >> > > > > > >>>>> less >> > >> > > > > > >>>>>>> >> > >> > > > > > >>>>>>>> knowledge of the airavata system in whole. So I >> would >> > >> like >> > >> > > to >> > >> > > > > > >>>>>>>>>> >> > >> > > > > > >>>>>>>>> discuss >> > >> > > > > > >>>> >> > >> > > > > > >>>>> following point. >> > >> > > > > > >>>>>>>>>> >> > >> > > > > > >>>>>>>>>> 1. How it will help us in making system more >> > >> reliable. >> > >> > > > > Zookeeper >> > >> > > > > > >>>>>>>>>> >> > >> > > > > > >>>>>>>>> is >> > >> > > > > > >>>> >> > >> > > > > > >>>>> not >> > >> > > > > > >>>>>>> >> > >> > > > > > >>>>>>>> able to restart services. At max it can tell >> whether >> > >> > service >> > >> > > > is >> > >> > > > > up >> > >> > > > > > >>>>>>>>>> >> > >> > > > > > >>>>>>>>> or not >> > >> > > > > > >>>>>>> >> > >> > > > > > >>>>>>>> which could only be the case if airavata service >> goes >> > >> down >> > >> > > > > > >>>>>>>>>> >> > >> > > > > > >>>>>>>>> gracefully and >> > >> > > > > > >>>>>>> >> > >> > > > > > >>>>>>>> we have any automated way to restart it. If this >> is >> > >> just >> > >> > > > matter >> > >> > > > > of >> > >> > > > > > >>>>>>>>>> >> > >> > > > > > >>>>>>>>> routing >> > >> > > > > > >>>>>>> >> > >> > > > > > >>>>>>>> client requests to the available thrift servers >> then >> > >> this >> > >> > > can >> > >> > > > be >> > >> > > > > > >>>>>>>>>> >> > >> > > > > > >>>>>>>>> achieved >> > >> > > > > > >>>>>>> >> > >> > > > > > >>>>>>>> with the help of load balancer which I guess is >> > already >> > >> > > there >> > >> > > > in >> > >> > > > > > >>>>>>>>>> >> > >> > > > > > >>>>>>>>> thrift >> > >> > > > > > >>>>>>> >> > >> > > > > > >>>>>>>> wish list. >> > >> > > > > > >>>>>>>>>> >> > >> > > > > > >>>>>>>>>> We have multiple thrift services and currently >> we >> > >> start >> > >> > > > only >> > >> > > > > > one >> > >> > > > > > >>>>>>>>> >> > >> > > > > > >>>>>>>> instance >> > >> > > > > > >>>>>>> >> > >> > > > > > >>>>>>>> of them and each thrift service is a stateless >> > >> service. To >> > >> > > > keep >> > >> > > > > > the >> > >> > > > > > >>>>>>>>> >> > >> > > > > > >>>>>>>> high >> > >> > > > > > >>>>>>> >> > >> > > > > > >>>>>>>> availability we have to start multiple instances >> of >> > >> them >> > >> > in >> > >> > > > > > >>>>>>>>> >> > >> > > > > > >>>>>>>> production >> > >> > > > > > >>>> >> > >> > > > > > >>>>> scenario. So for clients to get an available thrift >> > >> service >> > >> > we >> > >> > > > can >> > >> > > > > > >>>>>>>>> >> > >> > > > > > >>>>>>>> use >> > >> > > > > > >>>> >> > >> > > > > > >>>>> zookeeper znodes to represent each available >> service. >> > >> There >> > >> > > are >> > >> > > > > > >>>>>>>>> >> > >> > > > > > >>>>>>>> some >> > >> > > > > > >>>> >> > >> > > > > > >>>>> libraries which is doing similar[1] and I think we >> can >> > >> use >> > >> > > them >> > >> > > > > > >>>>>>>>> >> > >> > > > > > >>>>>>>> directly. >> > >> > > > > > >>>>>>> >> > >> > > > > > >>>>>>>> 2. As far as registering of different providers is >> > >> > concerned >> > >> > > > do >> > >> > > > > > >>>>>>>>>> >> > >> > > > > > >>>>>>>>> you >> > >> > > > > > >>>> >> > >> > > > > > >>>>> think for that we really need external store. >> > >> > > > > > >>>>>>>>>> >> > >> > > > > > >>>>>>>>>> Yes I think so, because its light weight and >> > >> reliable >> > >> > and >> > >> > > > we >> > >> > > > > > have >> > >> > > > > > >>>>>>>>> >> > >> > > > > > >>>>>>>> to >> > >> > > > > > >>>> >> > >> > > > > > >>>>> do >> > >> > > > > > >>>>>>> >> > >> > > > > > >>>>>>>> very minimal amount of work to achieve all these >> > >> features >> > >> > to >> > >> > > > > > >>>>>>>>> >> > >> > > > > > >>>>>>>> Airavata >> > >> > > > > > >>>> >> > >> > > > > > >>>>> because zookeeper handle all the complexity. >> > >> > > > > > >>>>>>>>> >> > >> > > > > > >>>>>>>>> I have seen people using zookeeper more for >> state >> > >> > > management >> > >> > > > > in >> > >> > > > > > >>>>>>>>>> distributed environments. >> > >> > > > > > >>>>>>>>>> >> > >> > > > > > >>>>>>>>>> +1, we might not be the most effective users of >> > >> > zookeeper >> > >> > > > > > because >> > >> > > > > > >>>>>>>>> >> > >> > > > > > >>>>>>>> all >> > >> > > > > > >>>> >> > >> > > > > > >>>>> of >> > >> > > > > > >>>>>>> >> > >> > > > > > >>>>>>>> our services are stateless services, but my point >> is >> > to >> > >> > > > achieve >> > >> > > > > > >>>>>>>>> fault-tolerance we can use zookeeper and with >> > minimal >> > >> > work. >> > >> > > > > > >>>>>>>>> >> > >> > > > > > >>>>>>>>> I would like to understand more how can we >> > leverage >> > >> > > > > zookeeper >> > >> > > > > > in >> > >> > > > > > >>>>>>>>>> airavata to make system reliable. >> > >> > > > > > >>>>>>>>>> >> > >> > > > > > >>>>>>>>>> >> > >> > > > > > >>>>>>>>>> [1] >> https://github.com/eirslett/thrift-zookeeper >> > >> > > > > > >>>>>>>>> >> > >> > > > > > >>>>>>>>> >> > >> > > > > > >>>>>>>>> >> > >> > > > > > >>>>>>>>> Regards, >> > >> > > > > > >>>>>>>>>> Gagan >> > >> > > > > > >>>>>>>>>> On 12-Jun-2014 12:33 am, "Marlon Pierce" < >> > >> > [email protected] >> > >> > > > >> > >> > > > > > wrote: >> > >> > > > > > >>>>>>>>>> >> > >> > > > > > >>>>>>>>>> Thanks for the summary, Lahiru. I'm cc'ing the >> > >> > > Architecture >> > >> > > > > > list >> > >> > > > > > >>>>>>>>>>> >> > >> > > > > > >>>>>>>>>> for >> > >> > > > > > >>>> >> > >> > > > > > >>>>> additional comments. >> > >> > > > > > >>>>>>>>>>> >> > >> > > > > > >>>>>>>>>>> Marlon >> > >> > > > > > >>>>>>>>>>> >> > >> > > > > > >>>>>>>>>>> On 6/11/14 2:27 PM, Lahiru Gunathilake wrote: >> > >> > > > > > >>>>>>>>>>> >> > >> > > > > > >>>>>>>>>>>> Hi All, >> > >> > > > > > >>>>>>>>>>>> >> > >> > > > > > >>>>>>>>>>>> I did little research about Apache >> Zookeeper[1] >> > and >> > >> > how >> > >> > > to >> > >> > > > > use >> > >> > > > > > >>>>>>>>>>>> >> > >> > > > > > >>>>>>>>>>> it >> > >> > > > > > >>>> >> > >> > > > > > >>>>> in >> > >> > > > > > >>>>>>> >> > >> > > > > > >>>>>>>> airavata. Its really a nice way to achieve fault >> > >> > tolerance >> > >> > > > and >> > >> > > > > > >>>>>>>>>>>> >> > >> > > > > > >>>>>>>>>>> reliable >> > >> > > > > > >>>>>>>>>>> >> > >> > > > > > >>>>>>>>>>>> communication between our thrift services and >> > >> clients. >> > >> > > > > > >>>>>>>>>>>> >> > >> > > > > > >>>>>>>>>>> Zookeeper >> > >> > > > > > >>>> >> > >> > > > > > >>>>> is a >> > >> > > > > > >>>>>>> >> > >> > > > > > >>>>>>>> distributed, fault tolerant system to do a >> reliable >> > >> > > > > > >>>>>>>>>>>> >> > >> > > > > > >>>>>>>>>>> communication >> > >> > > > > > >>>> >> > >> > > > > > >>>>> between >> > >> > > > > > >>>>>>>>>>> >> > >> > > > > > >>>>>>>>>>>> distributed applications. This is like an >> > in-memory >> > >> > file >> > >> > > > > > system >> > >> > > > > > >>>>>>>>>>>> >> > >> > > > > > >>>>>>>>>>> which >> > >> > > > > > >>>>>>> >> > >> > > > > > >>>>>>>> has >> > >> > > > > > >>>>>>>>>>> >> > >> > > > > > >>>>>>>>>>>> nodes in a tree structure and each node can >> have >> > >> small >> > >> > > > > amount >> > >> > > > > > >>>>>>>>>>>> >> > >> > > > > > >>>>>>>>>>> of >> > >> > > > > > >>>> >> > >> > > > > > >>>>> data >> > >> > > > > > >>>>>>> >> > >> > > > > > >>>>>>>> associated with it and these nodes are called >> > znodes. >> > >> > > Clients >> > >> > > > > > >>>>>>>>>>>> >> > >> > > > > > >>>>>>>>>>> can >> > >> > > > > > >>>> >> > >> > > > > > >>>>> connect >> > >> > > > > > >>>>>>>>>>> >> > >> > > > > > >>>>>>>>>>>> to a zookeeper server and add/delete and >> update >> > >> these >> > >> > > > > znodes. >> > >> > > > > > >>>>>>>>>>>> >> > >> > > > > > >>>>>>>>>>>> In Apache Airavata we start multiple thrift >> > >> > services >> > >> > > > and >> > >> > > > > > >>>>>>>>>>>> >> > >> > > > > > >>>>>>>>>>> these >> > >> > > > > > >>>> >> > >> > > > > > >>>>> can >> > >> > > > > > >>>>>>> >> > >> > > > > > >>>>>>>> go >> > >> > > > > > >>>>>>>>>>> >> > >> > > > > > >>>>>>>>>>>> down for maintenance or these can crash, if we >> > use >> > >> > > > zookeeper >> > >> > > > > > to >> > >> > > > > > >>>>>>>>>>>> >> > >> > > > > > >>>>>>>>>>> store >> > >> > > > > > >>>>>>> >> > >> > > > > > >>>>>>>> these >> > >> > > > > > >>>>>>>>>>> >> > >> > > > > > >>>>>>>>>>>> configuration(thrift service configurations) >> we >> > can >> > >> > > > achieve >> > >> > > > > a >> > >> > > > > > >>>>>>>>>>>> >> > >> > > > > > >>>>>>>>>>> very >> > >> > > > > > >>>> >> > >> > > > > > >>>>> reliable >> > >> > > > > > >>>>>>>>>>> >> > >> > > > > > >>>>>>>>>>>> system. Basically thrift clients can >> dynamically >> > >> > > discover >> > >> > > > > > >>>>>>>>>>>> >> > >> > > > > > >>>>>>>>>>> available >> > >> > > > > > >>>>>>> >> > >> > > > > > >>>>>>>> service >> > >> > > > > > >>>>>>>>>>> >> > >> > > > > > >>>>>>>>>>>> by using ephemeral znodes(Here we do not have >> to >> > >> > change >> > >> > > > the >> > >> > > > > > >>>>>>>>>>>> >> > >> > > > > > >>>>>>>>>>> generated >> > >> > > > > > >>>>>>> >> > >> > > > > > >>>>>>>> thrift client code but we have to change the >> > >> locations we >> > >> > > are >> > >> > > > > > >>>>>>>>>>>> >> > >> > > > > > >>>>>>>>>>> invoking >> > >> > > > > > >>>>>>> >> > >> > > > > > >>>>>>>> them). ephemeral znodes will be removed when the >> > >> thrift >> > >> > > > service >> > >> > > > > > >>>>>>>>>>>> >> > >> > > > > > >>>>>>>>>>> goes >> > >> > > > > > >>>>>>> >> > >> > > > > > >>>>>>>> down >> > >> > > > > > >>>>>>>>>>> >> > >> > > > > > >>>>>>>>>>>> and zookeeper guarantee the atomicity between >> > these >> > >> > > > > > operations. >> > >> > > > > > >>>>>>>>>>>> >> > >> > > > > > >>>>>>>>>>> With >> > >> > > > > > >>>>>>> >> > >> > > > > > >>>>>>>> this >> > >> > > > > > >>>>>>>>>>> >> > >> > > > > > >>>>>>>>>>>> approach we can have a node hierarchy for >> > multiple >> > >> of >> > >> > > > > > airavata, >> > >> > > > > > >>>>>>>>>>>> orchestrator,appcatalog and gfac thrift >> services. >> > >> > > > > > >>>>>>>>>>>> >> > >> > > > > > >>>>>>>>>>>> For specifically for gfac we can have >> different >> > >> types >> > >> > of >> > >> > > > > > >>>>>>>>>>>> >> > >> > > > > > >>>>>>>>>>> services >> > >> > > > > > >>>> >> > >> > > > > > >>>>> for >> > >> > > > > > >>>>>>> >> > >> > > > > > >>>>>>>> each >> > >> > > > > > >>>>>>>>>>> >> > >> > > > > > >>>>>>>>>>>> provider implementation. This can be achieved >> by >> > >> using >> > >> > > the >> > >> > > > > > >>>>>>>>>>>> >> > >> > > > > > >>>>>>>>>>> hierarchical >> > >> > > > > > >>>>>>>>>>> >> > >> > > > > > >>>>>>>>>>>> support in zookeeper and providing some logic >> in >> > >> > > > gfac-thrift >> > >> > > > > > >>>>>>>>>>>> >> > >> > > > > > >>>>>>>>>>> service >> > >> > > > > > >>>>>>> >> > >> > > > > > >>>>>>>> to >> > >> > > > > > >>>>>>>>>>> >> > >> > > > > > >>>>>>>>>>>> register it to a defined path. Using the same >> > logic >> > >> > > > > > >>>>>>>>>>>> >> > >> > > > > > >>>>>>>>>>> orchestrator >> > >> > > > > > >>>> >> > >> > > > > > >>>>> can >> > >> > > > > > >>>>>>> >> > >> > > > > > >>>>>>>> discover the provider specific gfac thrift >> service >> > and >> > >> > > route >> > >> > > > > > >>>>>>>>>>>> >> > >> > > > > > >>>>>>>>>>> the >> > >> > > > > > >>>> >> > >> > > > > > >>>>> message to >> > >> > > > > > >>>>>>>>>>> >> > >> > > > > > >>>>>>>>>>>> the correct thrift service. >> > >> > > > > > >>>>>>>>>>>> >> > >> > > > > > >>>>>>>>>>>> With this approach I think we simply have >> write >> > >> some >> > >> > > > client >> > >> > > > > > >>>>>>>>>>>> >> > >> > > > > > >>>>>>>>>>> code >> > >> > > > > > >>>> >> > >> > > > > > >>>>> in >> > >> > > > > > >>>>>>> >> > >> > > > > > >>>>>>>> thrift >> > >> > > > > > >>>>>>>>>>> >> > >> > > > > > >>>>>>>>>>>> services and clients and zookeeper server >> > >> installation >> > >> > > can >> > >> > > > > be >> > >> > > > > > >>>>>>>>>>>> >> > >> > > > > > >>>>>>>>>>> done as >> > >> > > > > > >>>>>>> >> > >> > > > > > >>>>>>>> a >> > >> > > > > > >>>>>>>>>>> >> > >> > > > > > >>>>>>>>>>>> separate process and it will be easier to keep >> > the >> > >> > > > Zookeeper >> > >> > > > > > >>>>>>>>>>>> >> > >> > > > > > >>>>>>>>>>> server >> > >> > > > > > >>>>>>> >> > >> > > > > > >>>>>>>> separate from Airavata because installation of >> > >> Zookeeper >> > >> > > > server >> > >> > > > > > >>>>>>>>>>>> >> > >> > > > > > >>>>>>>>>>> little >> > >> > > > > > >>>>>>> >> > >> > > > > > >>>>>>>> complex in production scenario. I think we have >> to >> > >> make >> > >> > > sure >> > >> > > > > > >>>>>>>>>>>> >> > >> > > > > > >>>>>>>>>>> everything >> > >> > > > > > >>>>>>>>>>> >> > >> > > > > > >>>>>>>>>>>> works fine when there is no Zookeeper running, >> > ex: >> > >> > > > > > >>>>>>>>>>>> >> > >> > > > > > >>>>>>>>>>> enable.zookeeper=false >> > >> > > > > > >>>>>>>>>>> >> > >> > > > > > >>>>>>>>>>>> should works fine and users doesn't have to >> > >> download >> > >> > and >> > >> > > > > start >> > >> > > > > > >>>>>>>>>>>> >> > >> > > > > > >>>>>>>>>>> zookeeper. >> > >> > > > > > >>>>>>>>>>> >> > >> > > > > > >>>>>>>>>>>> >> > >> > > > > > >>>>>>>>>>>> >> > >> > > > > > >>>>>>>>>>>> [1]http://zookeeper.apache.org/ >> > >> > > > > > >>>>>>>>>>>> >> > >> > > > > > >>>>>>>>>>>> Thanks >> > >> > > > > > >>>>>>>>>>>> Lahiru >> > >> > > > > > >>>>>>>>>>>> >> > >> > > > > > >>>>>>>>>>> >> > >> > > > > > >>>>>>>>>>> >> > >> > > > > > >>>>>>>>> -- >> > >> > > > > > >>>>>>>>> System Analyst Programmer >> > >> > > > > > >>>>>>>>> PTI Lab >> > >> > > > > > >>>>>>>>> Indiana University >> > >> > > > > > >>>>>>>>> >> > >> > > > > > >>>>>>>>> >> > >> > > > > > >>>>>>> -- >> > >> > > > > > >>>>>>> System Analyst Programmer >> > >> > > > > > >>>>>>> PTI Lab >> > >> > > > > > >>>>>>> Indiana University >> > >> > > > > > >>>>>>> >> > >> > > > > > >>>>>>> >> > >> > > > > > >>>>>> >> > >> > > > > > >>>>>> -- >> > >> > > > > > >>>>>> Best Regards, >> > >> > > > > > >>>>>> Shameera Rathnayaka. >> > >> > > > > > >>>>>> >> > >> > > > > > >>>>>> email: shameera AT apache.org , shameerainfo AT >> > >> gmail.com >> > >> > > > > > >>>>>> Blog : http://shameerarathnayaka.blogspot.com/ >> > >> > > > > > >>>>>> >> > >> > > > > > >>>>>> >> > >> > > > > > >>>>> >> > >> > > > > > >>>>> -- >> > >> > > > > > >>>>> Supun Kamburugamuva >> > >> > > > > > >>>>> Member, Apache Software Foundation; >> > http://www.apache.org >> > >> > > > > > >>>>> E-mail: [email protected]; Mobile: +1 812 369 6762 >> > >> > > > > > >>>>> Blog: http://supunk.blogspot.com >> > >> > > > > > >>>>> >> > >> > > > > > >>>>> >> > >> > > > > > >>>>> >> > >> > > > > > >>>> -- >> > >> > > > > > >>>> System Analyst Programmer >> > >> > > > > > >>>> PTI Lab >> > >> > > > > > >>>> Indiana University >> > >> > > > > > >>>> >> > >> > > > > > >>>> >> > >> > > > > > >>> >> > >> > > > > > >>> -- >> > >> > > > > > >>> Supun Kamburugamuva >> > >> > > > > > >>> Member, Apache Software Foundation; >> http://www.apache.org >> > >> > > > > > >>> E-mail: [email protected]; Mobile: +1 812 369 6762 >> > >> > > > > > >>> Blog: http://supunk.blogspot.com >> > >> > > > > > >>> >> > >> > > > > > >>> >> > >> > > > > > >>> >> > >> > > > > > >> >> > >> > > > > > > >> > >> > > > > > >> > >> > > > > > >> > >> > > > > > -- >> > >> > > > > > Supun Kamburugamuva >> > >> > > > > > Member, Apache Software Foundation; http://www.apache.org >> > >> > > > > > E-mail: [email protected]; Mobile: +1 812 369 6762 >> > >> > > > > > Blog: http://supunk.blogspot.com >> > >> > > > > > >> > >> > > > > >> > >> > > > > >> > >> > > > > >> > >> > > > > -- >> > >> > > > > System Analyst Programmer >> > >> > > > > PTI Lab >> > >> > > > > Indiana University >> > >> > > > > >> > >> > > > >> > >> > > > >> > >> > > > >> > >> > > > -- >> > >> > > > Supun Kamburugamuva >> > >> > > > Member, Apache Software Foundation; http://www.apache.org >> > >> > > > E-mail: [email protected]; Mobile: +1 812 369 6762 >> > >> > > > Blog: http://supunk.blogspot.com >> > >> > > > >> > >> > > >> > >> > > >> > >> > > >> > >> > > -- >> > >> > > System Analyst Programmer >> > >> > > PTI Lab >> > >> > > Indiana University >> > >> > > >> > >> > >> > >> >> > >> >> > >> >> > >> -- >> > >> Supun Kamburugamuva >> > >> Member, Apache Software Foundation; http://www.apache.org >> > >> E-mail: [email protected]; Mobile: +1 812 369 6762 >> > >> Blog: http://supunk.blogspot.com >> > >> >> > > >> > > >> > > >> > > -- >> > > System Analyst Programmer >> > > PTI Lab >> > > Indiana University >> > > >> > >> > >> > >> > -- >> > System Analyst Programmer >> > PTI Lab >> > Indiana University >> > >> > > > > -- > System Analyst Programmer > PTI Lab > Indiana University > -- System Analyst Programmer PTI Lab Indiana University
