Hi Tom, I have written a draft proposal and shared it with the Apache Software Foundation through the GSoC dashboard. If you get a chance, can you please review it and let me know what are the improvements required to be made.
Thanks in advance! Kind Regards, *Imesha Sudasingha* Undergraduate of Department of Computer Science and Engineering, University of Moratuwa, Sri Lanka. <https://lk.linkedin.com/in/imeshasudasingha> <https://github.com/IMS94> <http://stackoverflow.com/users/4012073/imesha-sudasingha> <https://twitter.com/Imesha94> On 22 March 2017 at 14:36, Imesha Sudasingha <[email protected]> wrote: > Hi Tom, > > I think I had missed your point previously. I was referring to just > storing the configuration of different components in different ZNodes and > querying them later. Then I understood that we also need to have > inheritance at zookeeper level in order to make it easy to configure OODT > clusters as you have mentioned in the issue [1]. To configure clusters of > OODT components with minimum configuration, I will have to review my design > again and make a lot of improvements. For example, I have to think about a > way to store configuration in zookeeper with minimum duplications and give > components enough flexibility to initialize with no or minimum manual > configuration. > > Since I have to submit the GSoC proposal before 3rd of April, I think that > I won't have enough time to review the design and make the necessary > improvements before that. Will that negatively affect the possibility of me > being able to work on this project for GSoC 2017? Please let me know you > opinion. > > Thank you! > > [1] https://issues.apache.org/jira/browse/OODT-945 > > Kind Regards, > *Imesha Sudasingha* > Undergraduate of Department of Computer Science and Engineering, > University of Moratuwa, > Sri Lanka. > > <https://lk.linkedin.com/in/imeshasudasingha> <https://github.com/IMS94> > <http://stackoverflow.com/users/4012073/imesha-sudasingha> > <https://twitter.com/Imesha94> > > On 21 March 2017 at 10:52, Imesha Sudasingha <[email protected]> > wrote: > >> Hi Tom, >> >> Sorry for the late reply. Zookeeper is just providing an ZNode structure, >> which can be considered as a tree where a node is accessed as directories. >> For example, in [1] the file manager properties of the instance with name " >> *node1*" is stored at the ZPath (Path to that ZNode) " >> */oodt/node1/file-mgr/filemgr.properties*". >> >> Therefore, my idea is that we can make such directories (ZPaths) per each >> instance in our cluster (ex: */oodt/node1 for instance "node1"*, */oodt/node2 >> for instance "node2"* and so on). Then inside each such path (assigned >> per node), we will have separate ZNodes for each component. For example, in >> */oodt/node1* we will have an ZNode /oodt/node1/file-mgr which stores >> the configurations of file manager of node1. Similarly, an ZNode >> /oodt/node1/res-mgr will store the configuration of the resource manager of >> node1. >> >> Hope you got the idea. Since zookeeper is not some kind of a graph >> database, I couldn't think of a method to keep inheritance among >> configuration at Zookeeper's side. However, the design I suggest will store >> the configuration independently and will allow the users/developers to >> query those configuration. Therefore in my opinion, we should let the >> developer to compare the configuration and make decisions. This way, we can >> allow this distributed configuration manager to manage configuration of any >> current component or any upcoming component making it more extend-able. >> >> Can you let me know what do you think of it? >> >> Thank you! >> >> [1] https://drive.google.com/open?id=0B415Bnipaf1lNThKYXZMN3pFemc >> >> Kind Regards, >> *Imesha Sudasingha* >> Undergraduate of Department of Computer Science and Engineering, >> University of Moratuwa, >> Sri Lanka. >> >> <https://lk.linkedin.com/in/imeshasudasingha> <https://github.com/IMS94> >> <http://stackoverflow.com/users/4012073/imesha-sudasingha> >> <https://twitter.com/Imesha94> >> >> On 20 March 2017 at 05:02, Tom Barber <[email protected]> wrote: >> >>> Hi Imesha >>> >>> Sorry for the delay, I've been travelling and only just got myself >>> through >>> my backlog. >>> >>> The overall design looks good. I was also thinking it would make sense if >>> possible to have some form of inheritance in the config. So, for example, >>> if you have 5 file managers, and they all share the same config except a >>> base url or whatever, it would make sense to have a single set of configs >>> and then individual overrides for configs that change. (My knowledge of >>> Zookeeper isn't great, I'm assuming something along these lines is >>> possible). >>> >>> It would also make sense, I think to register this stuff with the >>> resource >>> manager, but I'm not sure how that integration would work quite yet, so >>> we'll need to figure that one out. >>> >>> Let me know if you have any questions about the above. >>> >>> Tom >>> >>> >>> On Sat, Mar 18, 2017 at 12:39 PM, Imesha Sudasingha < >>> [email protected] >>> > wrote: >>> >>> > Hi Tom, >>> > >>> > What do you think of the design? Do you agree on implementing this >>> > configuration management section in a separate module? >>> > >>> > I was thinking on starting on the implementation as a proof of >>> concept. Can >>> > you give me your kind feedback before I start on that? >>> > >>> > Thank you! >>> > >>> > Kind Regards, >>> > *Imesha Sudasingha* >>> > Undergraduate of Department of Computer Science and Engineering, >>> > University of Moratuwa, >>> > Sri Lanka. >>> > >>> > <https://lk.linkedin.com/in/imeshasudasingha> < >>> https://github.com/IMS94> >>> > <http://stackoverflow.com/users/4012073/imesha-sudasingha> >>> > <https://twitter.com/Imesha94> >>> > >>> > On 16 March 2017 at 17:51, Imesha Sudasingha <[email protected]> >>> > wrote: >>> > >>> > > Hi Tom, >>> > > >>> > > Thanks for the quick reply. I came up with a small design [1]. >>> > > >>> > > What I suggest is a separate module for configuration management, >>> which >>> > > can be extended in future and used by many components as well. >>> > > >>> > > As shown in the design [1], File Manager (or any component that is >>> > willing >>> > > to have this extended functionality of distributed configuration >>> > > management) will have an instance of *ConfigurationManager* which is >>> an >>> > > interface defined to load configuration (specifically, .properties >>> files) >>> > > and get configurations of each running instance of the same >>> component (If >>> > > distributed configuration management is enabled, this will return the >>> > > configuration of all the other instances). >>> > > >>> > > There will be two implementing classes of that interface, >>> > > *StandaloneConfigurationManager* and *DistributedConfigurationManag >>> er.* >>> > As >>> > > the name implies, standalone configuration manager is to be used if >>> the >>> > > users do not intend to used distributed configuration management. >>> Also >>> > this >>> > > will be same as the current mechanism of configuration management. >>> Then >>> > the >>> > > distributed configuration manager will be the implementation using >>> > > Zookeeper. >>> > > >>> > > At zookeeper level, we will need to have a proper naming mechanism >>> and a >>> > > structure to store configuration. For example,* >>> > > /${nodeName}/${componentName}/{$configFileName}* will be the place >>> where >>> > > the configuration file contents will be stored. *getFile(String >>> fileName) >>> > > and getFiles(String directiryName)* of *NodeConfiguration* class will >>> > > return the content of the configuration files and configuration >>> > directories >>> > > respectively. I have demonstrated the ZNode structre I'm expecting to >>> > > design in [2] >>> > > >>> > > Please note that this is just a very high level overview of my >>> design and >>> > > the real difficulty of implementation comes when testing this and >>> > > addressing the edge cases of zookeeper. Also, I have only added few >>> > methods >>> > > that came to my mind to the *ConfigurationManager* interface. If you >>> find >>> > > any fault in this design or have a better design in mind please let >>> me >>> > know >>> > > so that I can optimize this design and do my best to come up with the >>> > best >>> > > possible solution. >>> > > >>> > > Thank you! >>> > > >>> > > [1] https://drive.google.com/open?id=0B415Bnipaf1ldnV1dWt4VVVLam8 >>> > > [2] https://drive.google.com/open?id=0B415Bnipaf1lNThKYXZMN3pFemc >>> > > >>> > > Kind Regards, >>> > > *Imesha Sudasingha* >>> > > Undergraduate of Department of Computer Science and Engineering, >>> > > University of Moratuwa, >>> > > Sri Lanka. >>> > > >>> > > <https://lk.linkedin.com/in/imeshasudasingha> < >>> https://github.com/IMS94 >>> > > >>> > > <http://stackoverflow.com/users/4012073/imesha-sudasingha> >>> > > <https://twitter.com/Imesha94> >>> > > >>> > > On 16 March 2017 at 16:55, Tom Barber <[email protected]> >>> wrote: >>> > > >>> > >> Hi Imesha, >>> > >> >>> > >> Basically, just allowing the dynamic registration of file managers >>> so >>> > that >>> > >> new ones can come and go. >>> > >> >>> > >> Tom >>> > >> >>> > >> On Thu, Mar 16, 2017 at 10:00 AM, Imesha Sudasingha < >>> > >> [email protected] >>> > >> > wrote: >>> > >> >>> > >> > Hi Tom, >>> > >> > >>> > >> > Need a small clarification. In the last reply, you mentioned that >>> file >>> > >> > managers may be designed to be *transient. *What do you mean by >>> that? >>> > A >>> > >> > quick response would really help me as I'm trying to come up with >>> a >>> > >> design. >>> > >> > >>> > >> > Thank you! >>> > >> > >>> > >> > *Imesha Sudasingha* >>> > >> > Undergraduate of Department of Computer Science and Engineering, >>> > >> > University of Moratuwa, >>> > >> > Sri Lanka. >>> > >> > >>> > >> > <https://lk.linkedin.com/in/imeshasudasingha> < >>> > >> https://github.com/IMS94> >>> > >> > <http://stackoverflow.com/users/4012073/imesha-sudasingha> >>> > >> > <https://twitter.com/Imesha94> >>> > >> > >>> > >> > On 13 March 2017 at 22:03, Imesha Sudasingha < >>> [email protected] >>> > > >>> > >> > wrote: >>> > >> > >>> > >> > > Hi Tom, >>> > >> > > >>> > >> > > Thanks for the clarification. It really helped a lot to fill >>> most of >>> > >> the >>> > >> > > gaps I had unanswered. I have a draft design in my mind and will >>> > post >>> > >> for >>> > >> > > your review with details soon. >>> > >> > > >>> > >> > > As for my current understanding, my task is to implement a >>> mechanism >>> > >> to >>> > >> > > switch/chose configuration mechanism (between local files and >>> > zookeer >>> > >> for >>> > >> > > the moment) and then allow APIs to query those mechanisms (in >>> the >>> > >> > zookeeper >>> > >> > > scenario) for future use (For File Managers to know about other >>> > >> instances >>> > >> > > and make decisions accordingly). That include synchronization, >>> > >> fetching >>> > >> > > configuration from zookeeper if an instance go down and come >>> again >>> > >> and so >>> > >> > > on. Am I correct? >>> > >> > > >>> > >> > > I will try to come up with the design I mentioned, so that both >>> of >>> > us >>> > >> can >>> > >> > > discuss more about the improvements and requirements soon. >>> > >> > > >>> > >> > > Thank you for the support! >>> > >> > > >>> > >> > > Kind Regards, >>> > >> > > *Imesha Sudasingha* >>> > >> > > Undergraduate of Department of Computer Science and >>> Engineering, >>> > >> > > University of Moratuwa, >>> > >> > > Sri Lanka. >>> > >> > > >>> > >> > > <https://lk.linkedin.com/in/imeshasudasingha> < >>> > >> https://github.com/IMS94 >>> > >> > > >>> > >> > > <http://stackoverflow.com/users/4012073/imesha-sudasingha> >>> > >> > > <https://twitter.com/Imesha94> >>> > >> > > >>> > >> > > On 13 March 2017 at 20:19, Tom Barber <[email protected]> >>> > >> wrote: >>> > >> > > >>> > >> > >> Hi Imesha >>> > >> > >> >>> > >> > >> The best way to demonstrate this is to show you a reference >>> build >>> > for >>> > >> > >> OODT.. >>> > >> > >> >>> > >> > >> https://dl.dropboxusercontent.com/u/8503756/oodt-example.tgz >>> > >> > >> >>> > >> > >> If you can grab that archive and extract it, you'll see that >>> is how >>> > >> > >> currently OODT is deployed. >>> > >> > >> >>> > >> > >> If you take the file manager as the reference module inside >>> > >> oodt/filemgr >>> > >> > >> you'll see >>> > >> > >> >>> > >> > >> etc/filemgr.properties >>> > >> > >> >>> > >> > >> and >>> > >> > >> >>> > >> > >> policy/oodt/* >>> > >> > >> >>> > >> > >> These are the configuration locations of a single file manager. >>> > >> > >> >>> > >> > >> The problem I want to solve is this: >>> > >> > >> >>> > >> > >> a) If I run > 1 file manager they are completely separate and >>> they >>> > >> don't >>> > >> > >> know of each other. So using Zookeeper will allow multiple file >>> > >> managers >>> > >> > >> to >>> > >> > >> register against ZK and learn of each other. >>> > >> > >> >>> > >> > >> b) Backing them onto ZK will allow file managers to come and >>> go on >>> > >> the >>> > >> > >> fly, >>> > >> > >> providing scaling capabilities or maybe fm's that are designed >>> to >>> > be >>> > >> > >> transient. >>> > >> > >> >>> > >> > >> Just for clarity, the ZK configuration will be an optional >>> > >> requirement, >>> > >> > so >>> > >> > >> we need to extend the existing configuration mechanism, to >>> support >>> > >> both >>> > >> > >> the >>> > >> > >> current config and the ZK config instead of ZK being >>> mandatory. >>> > >> > >> >>> > >> > >> Tom >>> > >> > >> >>> > >> > >> On Mon, Mar 13, 2017 at 3:20 AM, Imesha Sudasingha < >>> > >> > >> [email protected]> >>> > >> > >> wrote: >>> > >> > >> >>> > >> > >> > Hi Tom, >>> > >> > >> > >>> > >> > >> > Related to the conversation on JIRA [1] ; >>> > >> > >> > >>> > >> > >> > I went through the CAS-File manager module and I now >>> understand >>> > >> how to >>> > >> > >> > address ".properties" files. Since you mentioned in JIRA [2] >>> that >>> > >> > there >>> > >> > >> are >>> > >> > >> > occasions where we use *spring configuration files *for >>> > >> configuration >>> > >> > >> > purposes, I want to have a look at that type of configuration >>> > too. >>> > >> Can >>> > >> > >> you >>> > >> > >> > give me an example module where spring configuration files >>> are >>> > >> used to >>> > >> > >> load >>> > >> > >> > configuration? >>> > >> > >> > >>> > >> > >> > Also, will the content of a configuration file exceed 1MB >>> > (Maximum >>> > >> > data >>> > >> > >> > size allowed in an ZNode) in any case? >>> > >> > >> > >>> > >> > >> > Thank you! >>> > >> > >> > >>> > >> > >> > [1] >>> > >> > >> > https://issues.apache.org/jira/browse/OODT-945?focusedCommen >>> > >> > >> tId=15906204& >>> > >> > >> > page=com.atlassian.jira.plugin.system.issuetabpanels: >>> > >> > >> > comment-tabpanel#comment-15906204 >>> > >> > >> > >>> > >> > >> > [2] >>> > >> > >> > https://issues.apache.org/jira/browse/OODT-945?focusedCommen >>> > >> > >> tId=15906524& >>> > >> > >> > page=com.atlassian.jira.plugin.system.issuetabpanels: >>> > >> > >> > comment-tabpanel#comment-15906524 >>> > >> > >> > >>> > >> > >> > Kind Regards, >>> > >> > >> > *Imesha Sudasingha* >>> > >> > >> > Undergraduate of Department of Computer Science and >>> Engineering, >>> > >> > >> > University of Moratuwa, >>> > >> > >> > Sri Lanka. >>> > >> > >> > >>> > >> > >> > <https://lk.linkedin.com/in/imeshasudasingha> < >>> > >> > >> https://github.com/IMS94> >>> > >> > >> > <http://stackoverflow.com/users/4012073/imesha-sudasingha> >>> > >> > >> > <https://twitter.com/Imesha94> >>> > >> > >> > >>> > >> > >> >>> > >> > > >>> > >> > > >>> > >> > >>> > >> >>> > > >>> > > >>> > >>> >> >> >
