Hi Imesha Sorry for the delay, I've been travelling and only just got myself through my backlog.
The overall design looks good. I was also thinking it would make sense if possible to have some form of inheritance in the config. So, for example, if you have 5 file managers, and they all share the same config except a base url or whatever, it would make sense to have a single set of configs and then individual overrides for configs that change. (My knowledge of Zookeeper isn't great, I'm assuming something along these lines is possible). It would also make sense, I think to register this stuff with the resource manager, but I'm not sure how that integration would work quite yet, so we'll need to figure that one out. Let me know if you have any questions about the above. Tom On Sat, Mar 18, 2017 at 12:39 PM, Imesha Sudasingha <[email protected] > wrote: > Hi Tom, > > What do you think of the design? Do you agree on implementing this > configuration management section in a separate module? > > I was thinking on starting on the implementation as a proof of concept. Can > you give me your kind feedback before I start on that? > > Thank you! > > Kind Regards, > *Imesha Sudasingha* > Undergraduate of Department of Computer Science and Engineering, > University of Moratuwa, > Sri Lanka. > > <https://lk.linkedin.com/in/imeshasudasingha> <https://github.com/IMS94> > <http://stackoverflow.com/users/4012073/imesha-sudasingha> > <https://twitter.com/Imesha94> > > On 16 March 2017 at 17:51, Imesha Sudasingha <[email protected]> > wrote: > > > Hi Tom, > > > > Thanks for the quick reply. I came up with a small design [1]. > > > > What I suggest is a separate module for configuration management, which > > can be extended in future and used by many components as well. > > > > As shown in the design [1], File Manager (or any component that is > willing > > to have this extended functionality of distributed configuration > > management) will have an instance of *ConfigurationManager* which is an > > interface defined to load configuration (specifically, .properties files) > > and get configurations of each running instance of the same component (If > > distributed configuration management is enabled, this will return the > > configuration of all the other instances). > > > > There will be two implementing classes of that interface, > > *StandaloneConfigurationManager* and *DistributedConfigurationManager.* > As > > the name implies, standalone configuration manager is to be used if the > > users do not intend to used distributed configuration management. Also > this > > will be same as the current mechanism of configuration management. Then > the > > distributed configuration manager will be the implementation using > > Zookeeper. > > > > At zookeeper level, we will need to have a proper naming mechanism and a > > structure to store configuration. For example,* > > /${nodeName}/${componentName}/{$configFileName}* will be the place where > > the configuration file contents will be stored. *getFile(String fileName) > > and getFiles(String directiryName)* of *NodeConfiguration* class will > > return the content of the configuration files and configuration > directories > > respectively. I have demonstrated the ZNode structre I'm expecting to > > design in [2] > > > > Please note that this is just a very high level overview of my design and > > the real difficulty of implementation comes when testing this and > > addressing the edge cases of zookeeper. Also, I have only added few > methods > > that came to my mind to the *ConfigurationManager* interface. If you find > > any fault in this design or have a better design in mind please let me > know > > so that I can optimize this design and do my best to come up with the > best > > possible solution. > > > > Thank you! > > > > [1] https://drive.google.com/open?id=0B415Bnipaf1ldnV1dWt4VVVLam8 > > [2] https://drive.google.com/open?id=0B415Bnipaf1lNThKYXZMN3pFemc > > > > Kind Regards, > > *Imesha Sudasingha* > > Undergraduate of Department of Computer Science and Engineering, > > University of Moratuwa, > > Sri Lanka. > > > > <https://lk.linkedin.com/in/imeshasudasingha> <https://github.com/IMS94 > > > > <http://stackoverflow.com/users/4012073/imesha-sudasingha> > > <https://twitter.com/Imesha94> > > > > On 16 March 2017 at 16:55, Tom Barber <[email protected]> wrote: > > > >> Hi Imesha, > >> > >> Basically, just allowing the dynamic registration of file managers so > that > >> new ones can come and go. > >> > >> Tom > >> > >> On Thu, Mar 16, 2017 at 10:00 AM, Imesha Sudasingha < > >> [email protected] > >> > wrote: > >> > >> > Hi Tom, > >> > > >> > Need a small clarification. In the last reply, you mentioned that file > >> > managers may be designed to be *transient. *What do you mean by that? > A > >> > quick response would really help me as I'm trying to come up with a > >> design. > >> > > >> > Thank you! > >> > > >> > *Imesha Sudasingha* > >> > Undergraduate of Department of Computer Science and Engineering, > >> > University of Moratuwa, > >> > Sri Lanka. > >> > > >> > <https://lk.linkedin.com/in/imeshasudasingha> < > >> https://github.com/IMS94> > >> > <http://stackoverflow.com/users/4012073/imesha-sudasingha> > >> > <https://twitter.com/Imesha94> > >> > > >> > On 13 March 2017 at 22:03, Imesha Sudasingha <[email protected] > > > >> > wrote: > >> > > >> > > Hi Tom, > >> > > > >> > > Thanks for the clarification. It really helped a lot to fill most of > >> the > >> > > gaps I had unanswered. I have a draft design in my mind and will > post > >> for > >> > > your review with details soon. > >> > > > >> > > As for my current understanding, my task is to implement a mechanism > >> to > >> > > switch/chose configuration mechanism (between local files and > zookeer > >> for > >> > > the moment) and then allow APIs to query those mechanisms (in the > >> > zookeeper > >> > > scenario) for future use (For File Managers to know about other > >> instances > >> > > and make decisions accordingly). That include synchronization, > >> fetching > >> > > configuration from zookeeper if an instance go down and come again > >> and so > >> > > on. Am I correct? > >> > > > >> > > I will try to come up with the design I mentioned, so that both of > us > >> can > >> > > discuss more about the improvements and requirements soon. > >> > > > >> > > Thank you for the support! > >> > > > >> > > Kind Regards, > >> > > *Imesha Sudasingha* > >> > > Undergraduate of Department of Computer Science and Engineering, > >> > > University of Moratuwa, > >> > > Sri Lanka. > >> > > > >> > > <https://lk.linkedin.com/in/imeshasudasingha> < > >> https://github.com/IMS94 > >> > > > >> > > <http://stackoverflow.com/users/4012073/imesha-sudasingha> > >> > > <https://twitter.com/Imesha94> > >> > > > >> > > On 13 March 2017 at 20:19, Tom Barber <[email protected]> > >> wrote: > >> > > > >> > >> Hi Imesha > >> > >> > >> > >> The best way to demonstrate this is to show you a reference build > for > >> > >> OODT.. > >> > >> > >> > >> https://dl.dropboxusercontent.com/u/8503756/oodt-example.tgz > >> > >> > >> > >> If you can grab that archive and extract it, you'll see that is how > >> > >> currently OODT is deployed. > >> > >> > >> > >> If you take the file manager as the reference module inside > >> oodt/filemgr > >> > >> you'll see > >> > >> > >> > >> etc/filemgr.properties > >> > >> > >> > >> and > >> > >> > >> > >> policy/oodt/* > >> > >> > >> > >> These are the configuration locations of a single file manager. > >> > >> > >> > >> The problem I want to solve is this: > >> > >> > >> > >> a) If I run > 1 file manager they are completely separate and they > >> don't > >> > >> know of each other. So using Zookeeper will allow multiple file > >> managers > >> > >> to > >> > >> register against ZK and learn of each other. > >> > >> > >> > >> b) Backing them onto ZK will allow file managers to come and go on > >> the > >> > >> fly, > >> > >> providing scaling capabilities or maybe fm's that are designed to > be > >> > >> transient. > >> > >> > >> > >> Just for clarity, the ZK configuration will be an optional > >> requirement, > >> > so > >> > >> we need to extend the existing configuration mechanism, to support > >> both > >> > >> the > >> > >> current config and the ZK config instead of ZK being mandatory. > >> > >> > >> > >> Tom > >> > >> > >> > >> On Mon, Mar 13, 2017 at 3:20 AM, Imesha Sudasingha < > >> > >> [email protected]> > >> > >> wrote: > >> > >> > >> > >> > Hi Tom, > >> > >> > > >> > >> > Related to the conversation on JIRA [1] ; > >> > >> > > >> > >> > I went through the CAS-File manager module and I now understand > >> how to > >> > >> > address ".properties" files. Since you mentioned in JIRA [2] that > >> > there > >> > >> are > >> > >> > occasions where we use *spring configuration files *for > >> configuration > >> > >> > purposes, I want to have a look at that type of configuration > too. > >> Can > >> > >> you > >> > >> > give me an example module where spring configuration files are > >> used to > >> > >> load > >> > >> > configuration? > >> > >> > > >> > >> > Also, will the content of a configuration file exceed 1MB > (Maximum > >> > data > >> > >> > size allowed in an ZNode) in any case? > >> > >> > > >> > >> > Thank you! > >> > >> > > >> > >> > [1] > >> > >> > https://issues.apache.org/jira/browse/OODT-945?focusedCommen > >> > >> tId=15906204& > >> > >> > page=com.atlassian.jira.plugin.system.issuetabpanels: > >> > >> > comment-tabpanel#comment-15906204 > >> > >> > > >> > >> > [2] > >> > >> > https://issues.apache.org/jira/browse/OODT-945?focusedCommen > >> > >> tId=15906524& > >> > >> > page=com.atlassian.jira.plugin.system.issuetabpanels: > >> > >> > comment-tabpanel#comment-15906524 > >> > >> > > >> > >> > Kind Regards, > >> > >> > *Imesha Sudasingha* > >> > >> > Undergraduate of Department of Computer Science and Engineering, > >> > >> > University of Moratuwa, > >> > >> > Sri Lanka. > >> > >> > > >> > >> > <https://lk.linkedin.com/in/imeshasudasingha> < > >> > >> https://github.com/IMS94> > >> > >> > <http://stackoverflow.com/users/4012073/imesha-sudasingha> > >> > >> > <https://twitter.com/Imesha94> > >> > >> > > >> > >> > >> > > > >> > > > >> > > >> > > > > >
