Hi Tom, What do you think of the design? Do you agree on implementing this configuration management section in a separate module?
I was thinking on starting on the implementation as a proof of concept. Can you give me your kind feedback before I start on that? Thank you! Kind Regards, *Imesha Sudasingha* Undergraduate of Department of Computer Science and Engineering, University of Moratuwa, Sri Lanka. <https://lk.linkedin.com/in/imeshasudasingha> <https://github.com/IMS94> <http://stackoverflow.com/users/4012073/imesha-sudasingha> <https://twitter.com/Imesha94> On 16 March 2017 at 17:51, Imesha Sudasingha <[email protected]> wrote: > Hi Tom, > > Thanks for the quick reply. I came up with a small design [1]. > > What I suggest is a separate module for configuration management, which > can be extended in future and used by many components as well. > > As shown in the design [1], File Manager (or any component that is willing > to have this extended functionality of distributed configuration > management) will have an instance of *ConfigurationManager* which is an > interface defined to load configuration (specifically, .properties files) > and get configurations of each running instance of the same component (If > distributed configuration management is enabled, this will return the > configuration of all the other instances). > > There will be two implementing classes of that interface, > *StandaloneConfigurationManager* and *DistributedConfigurationManager.* As > the name implies, standalone configuration manager is to be used if the > users do not intend to used distributed configuration management. Also this > will be same as the current mechanism of configuration management. Then the > distributed configuration manager will be the implementation using > Zookeeper. > > At zookeeper level, we will need to have a proper naming mechanism and a > structure to store configuration. For example,* > /${nodeName}/${componentName}/{$configFileName}* will be the place where > the configuration file contents will be stored. *getFile(String fileName) > and getFiles(String directiryName)* of *NodeConfiguration* class will > return the content of the configuration files and configuration directories > respectively. I have demonstrated the ZNode structre I'm expecting to > design in [2] > > Please note that this is just a very high level overview of my design and > the real difficulty of implementation comes when testing this and > addressing the edge cases of zookeeper. Also, I have only added few methods > that came to my mind to the *ConfigurationManager* interface. If you find > any fault in this design or have a better design in mind please let me know > so that I can optimize this design and do my best to come up with the best > possible solution. > > Thank you! > > [1] https://drive.google.com/open?id=0B415Bnipaf1ldnV1dWt4VVVLam8 > [2] https://drive.google.com/open?id=0B415Bnipaf1lNThKYXZMN3pFemc > > Kind Regards, > *Imesha Sudasingha* > Undergraduate of Department of Computer Science and Engineering, > University of Moratuwa, > Sri Lanka. > > <https://lk.linkedin.com/in/imeshasudasingha> <https://github.com/IMS94> > <http://stackoverflow.com/users/4012073/imesha-sudasingha> > <https://twitter.com/Imesha94> > > On 16 March 2017 at 16:55, Tom Barber <[email protected]> wrote: > >> Hi Imesha, >> >> Basically, just allowing the dynamic registration of file managers so that >> new ones can come and go. >> >> Tom >> >> On Thu, Mar 16, 2017 at 10:00 AM, Imesha Sudasingha < >> [email protected] >> > wrote: >> >> > Hi Tom, >> > >> > Need a small clarification. In the last reply, you mentioned that file >> > managers may be designed to be *transient. *What do you mean by that? A >> > quick response would really help me as I'm trying to come up with a >> design. >> > >> > Thank you! >> > >> > *Imesha Sudasingha* >> > Undergraduate of Department of Computer Science and Engineering, >> > University of Moratuwa, >> > Sri Lanka. >> > >> > <https://lk.linkedin.com/in/imeshasudasingha> < >> https://github.com/IMS94> >> > <http://stackoverflow.com/users/4012073/imesha-sudasingha> >> > <https://twitter.com/Imesha94> >> > >> > On 13 March 2017 at 22:03, Imesha Sudasingha <[email protected]> >> > wrote: >> > >> > > Hi Tom, >> > > >> > > Thanks for the clarification. It really helped a lot to fill most of >> the >> > > gaps I had unanswered. I have a draft design in my mind and will post >> for >> > > your review with details soon. >> > > >> > > As for my current understanding, my task is to implement a mechanism >> to >> > > switch/chose configuration mechanism (between local files and zookeer >> for >> > > the moment) and then allow APIs to query those mechanisms (in the >> > zookeeper >> > > scenario) for future use (For File Managers to know about other >> instances >> > > and make decisions accordingly). That include synchronization, >> fetching >> > > configuration from zookeeper if an instance go down and come again >> and so >> > > on. Am I correct? >> > > >> > > I will try to come up with the design I mentioned, so that both of us >> can >> > > discuss more about the improvements and requirements soon. >> > > >> > > Thank you for the support! >> > > >> > > Kind Regards, >> > > *Imesha Sudasingha* >> > > Undergraduate of Department of Computer Science and Engineering, >> > > University of Moratuwa, >> > > Sri Lanka. >> > > >> > > <https://lk.linkedin.com/in/imeshasudasingha> < >> https://github.com/IMS94 >> > > >> > > <http://stackoverflow.com/users/4012073/imesha-sudasingha> >> > > <https://twitter.com/Imesha94> >> > > >> > > On 13 March 2017 at 20:19, Tom Barber <[email protected]> >> wrote: >> > > >> > >> Hi Imesha >> > >> >> > >> The best way to demonstrate this is to show you a reference build for >> > >> OODT.. >> > >> >> > >> https://dl.dropboxusercontent.com/u/8503756/oodt-example.tgz >> > >> >> > >> If you can grab that archive and extract it, you'll see that is how >> > >> currently OODT is deployed. >> > >> >> > >> If you take the file manager as the reference module inside >> oodt/filemgr >> > >> you'll see >> > >> >> > >> etc/filemgr.properties >> > >> >> > >> and >> > >> >> > >> policy/oodt/* >> > >> >> > >> These are the configuration locations of a single file manager. >> > >> >> > >> The problem I want to solve is this: >> > >> >> > >> a) If I run > 1 file manager they are completely separate and they >> don't >> > >> know of each other. So using Zookeeper will allow multiple file >> managers >> > >> to >> > >> register against ZK and learn of each other. >> > >> >> > >> b) Backing them onto ZK will allow file managers to come and go on >> the >> > >> fly, >> > >> providing scaling capabilities or maybe fm's that are designed to be >> > >> transient. >> > >> >> > >> Just for clarity, the ZK configuration will be an optional >> requirement, >> > so >> > >> we need to extend the existing configuration mechanism, to support >> both >> > >> the >> > >> current config and the ZK config instead of ZK being mandatory. >> > >> >> > >> Tom >> > >> >> > >> On Mon, Mar 13, 2017 at 3:20 AM, Imesha Sudasingha < >> > >> [email protected]> >> > >> wrote: >> > >> >> > >> > Hi Tom, >> > >> > >> > >> > Related to the conversation on JIRA [1] ; >> > >> > >> > >> > I went through the CAS-File manager module and I now understand >> how to >> > >> > address ".properties" files. Since you mentioned in JIRA [2] that >> > there >> > >> are >> > >> > occasions where we use *spring configuration files *for >> configuration >> > >> > purposes, I want to have a look at that type of configuration too. >> Can >> > >> you >> > >> > give me an example module where spring configuration files are >> used to >> > >> load >> > >> > configuration? >> > >> > >> > >> > Also, will the content of a configuration file exceed 1MB (Maximum >> > data >> > >> > size allowed in an ZNode) in any case? >> > >> > >> > >> > Thank you! >> > >> > >> > >> > [1] >> > >> > https://issues.apache.org/jira/browse/OODT-945?focusedCommen >> > >> tId=15906204& >> > >> > page=com.atlassian.jira.plugin.system.issuetabpanels: >> > >> > comment-tabpanel#comment-15906204 >> > >> > >> > >> > [2] >> > >> > https://issues.apache.org/jira/browse/OODT-945?focusedCommen >> > >> tId=15906524& >> > >> > page=com.atlassian.jira.plugin.system.issuetabpanels: >> > >> > comment-tabpanel#comment-15906524 >> > >> > >> > >> > Kind Regards, >> > >> > *Imesha Sudasingha* >> > >> > Undergraduate of Department of Computer Science and Engineering, >> > >> > University of Moratuwa, >> > >> > Sri Lanka. >> > >> > >> > >> > <https://lk.linkedin.com/in/imeshasudasingha> < >> > >> https://github.com/IMS94> >> > >> > <http://stackoverflow.com/users/4012073/imesha-sudasingha> >> > >> > <https://twitter.com/Imesha94> >> > >> > >> > >> >> > > >> > > >> > >> > >
