Thanks Imesha I shall check it tomorrow and get back to you ASAP.
Tom On Tue, Mar 28, 2017 at 2:33 AM, Imesha Sudasingha <[email protected]> wrote: > Hi Tom, > > As you suggested, I have created a copy of my proposal at [1] and shared > with comment permissions. It will be a great help if you can review it > sooner and provide a kind feedback. > > Thank you! > > [1] > https://docs.google.com/document/d/1bnFEVViCwMACHJRB1OiYZ- > GAywCNJRxrUJPpHttpgpE/edit?usp=sharing > > Kind Regards, > *Imesha Sudasingha* > Undergraduate of Department of Computer Science and Engineering, > University of Moratuwa, > Sri Lanka. > > <https://lk.linkedin.com/in/imeshasudasingha> <https://github.com/IMS94> > <http://stackoverflow.com/users/4012073/imesha-sudasingha> > <https://twitter.com/Imesha94> > > On 28 March 2017 at 01:13, Tom Barber <[email protected]> wrote: > > > Hi Imesha > > > > I'm trying to figure out if/how I can get access to it on the dashboard. > In > > the mean time if you want to pastebin it, chuck it in a word doc, or > > something completely different I'd be happy to take a look. > > > > Tom > > > > On Sun, Mar 26, 2017 at 3:09 PM, Imesha Sudasingha < > > [email protected]> > > wrote: > > > > > Hi Tom, > > > > > > Did you get a chance to look at my proposal? > > > > > > Kind Regards, > > > *Imesha Sudasingha* > > > Undergraduate of Department of Computer Science and Engineering, > > > University of Moratuwa, > > > Sri Lanka. > > > > > > <https://lk.linkedin.com/in/imeshasudasingha> < > https://github.com/IMS94 > > > > > > <http://stackoverflow.com/users/4012073/imesha-sudasingha> > > > <https://twitter.com/Imesha94> > > > > > > On 23 March 2017 at 10:40, Imesha Sudasingha <[email protected]> > > > wrote: > > > > > > > Hi Tom, > > > > > > > > I have written a draft proposal and shared it with the Apache > Software > > > > Foundation through the GSoC dashboard. If you get a chance, can you > > > please > > > > review it and let me know what are the improvements required to be > > made. > > > > > > > > Thanks in advance! > > > > > > > > Kind Regards, > > > > *Imesha Sudasingha* > > > > Undergraduate of Department of Computer Science and Engineering, > > > > University of Moratuwa, > > > > Sri Lanka. > > > > > > > > <https://lk.linkedin.com/in/imeshasudasingha> < > > https://github.com/IMS94 > > > > > > > > <http://stackoverflow.com/users/4012073/imesha-sudasingha> > > > > <https://twitter.com/Imesha94> > > > > > > > > On 22 March 2017 at 14:36, Imesha Sudasingha < > [email protected]> > > > > wrote: > > > > > > > >> Hi Tom, > > > >> > > > >> I think I had missed your point previously. I was referring to just > > > >> storing the configuration of different components in different > ZNodes > > > and > > > >> querying them later. Then I understood that we also need to have > > > >> inheritance at zookeeper level in order to make it easy to configure > > > OODT > > > >> clusters as you have mentioned in the issue [1]. To configure > clusters > > > of > > > >> OODT components with minimum configuration, I will have to review my > > > design > > > >> again and make a lot of improvements. For example, I have to think > > > about a > > > >> way to store configuration in zookeeper with minimum duplications > and > > > give > > > >> components enough flexibility to initialize with no or minimum > manual > > > >> configuration. > > > >> > > > >> Since I have to submit the GSoC proposal before 3rd of April, I > think > > > >> that I won't have enough time to review the design and make the > > > necessary > > > >> improvements before that. Will that negatively affect the > possibility > > > of me > > > >> being able to work on this project for GSoC 2017? Please let me know > > you > > > >> opinion. > > > >> > > > >> Thank you! > > > >> > > > >> [1] https://issues.apache.org/jira/browse/OODT-945 > > > >> > > > >> Kind Regards, > > > >> *Imesha Sudasingha* > > > >> Undergraduate of Department of Computer Science and Engineering, > > > >> University of Moratuwa, > > > >> Sri Lanka. > > > >> > > > >> <https://lk.linkedin.com/in/imeshasudasingha> < > > > https://github.com/IMS94> > > > >> <http://stackoverflow.com/users/4012073/imesha-sudasingha> > > > >> <https://twitter.com/Imesha94> > > > >> > > > >> On 21 March 2017 at 10:52, Imesha Sudasingha < > [email protected] > > > > > > >> wrote: > > > >> > > > >>> Hi Tom, > > > >>> > > > >>> Sorry for the late reply. Zookeeper is just providing an ZNode > > > >>> structure, which can be considered as a tree where a node is > accessed > > > as > > > >>> directories. For example, in [1] the file manager properties of the > > > >>> instance with name "*node1*" is stored at the ZPath (Path to that > > > >>> ZNode) "*/oodt/node1/file-mgr/filemgr.properties*". > > > >>> > > > >>> Therefore, my idea is that we can make such directories (ZPaths) > per > > > >>> each instance in our cluster (ex: */oodt/node1 for instance > "node1"*, > > > */oodt/node2 > > > >>> for instance "node2"* and so on). Then inside each such path > > (assigned > > > >>> per node), we will have separate ZNodes for each component. For > > > example, in > > > >>> */oodt/node1* we will have an ZNode /oodt/node1/file-mgr which > stores > > > >>> the configurations of file manager of node1. Similarly, an ZNode > > > >>> /oodt/node1/res-mgr will store the configuration of the resource > > > manager of > > > >>> node1. > > > >>> > > > >>> Hope you got the idea. Since zookeeper is not some kind of a graph > > > >>> database, I couldn't think of a method to keep inheritance among > > > >>> configuration at Zookeeper's side. However, the design I suggest > will > > > store > > > >>> the configuration independently and will allow the users/developers > > to > > > >>> query those configuration. Therefore in my opinion, we should let > the > > > >>> developer to compare the configuration and make decisions. This > way, > > > we can > > > >>> allow this distributed configuration manager to manage > configuration > > > of any > > > >>> current component or any upcoming component making it more > > extend-able. > > > >>> > > > >>> Can you let me know what do you think of it? > > > >>> > > > >>> Thank you! > > > >>> > > > >>> [1] https://drive.google.com/open?id=0B415Bnipaf1lNThKYXZMN3pFemc > > > >>> > > > >>> Kind Regards, > > > >>> *Imesha Sudasingha* > > > >>> Undergraduate of Department of Computer Science and Engineering, > > > >>> University of Moratuwa, > > > >>> Sri Lanka. > > > >>> > > > >>> <https://lk.linkedin.com/in/imeshasudasingha> > > > >>> <https://github.com/IMS94> > > > >>> <http://stackoverflow.com/users/4012073/imesha-sudasingha> > > > >>> <https://twitter.com/Imesha94> > > > >>> > > > >>> On 20 March 2017 at 05:02, Tom Barber <[email protected]> > > wrote: > > > >>> > > > >>>> Hi Imesha > > > >>>> > > > >>>> Sorry for the delay, I've been travelling and only just got myself > > > >>>> through > > > >>>> my backlog. > > > >>>> > > > >>>> The overall design looks good. I was also thinking it would make > > sense > > > >>>> if > > > >>>> possible to have some form of inheritance in the config. So, for > > > >>>> example, > > > >>>> if you have 5 file managers, and they all share the same config > > > except a > > > >>>> base url or whatever, it would make sense to have a single set of > > > >>>> configs > > > >>>> and then individual overrides for configs that change. (My > knowledge > > > of > > > >>>> Zookeeper isn't great, I'm assuming something along these lines is > > > >>>> possible). > > > >>>> > > > >>>> It would also make sense, I think to register this stuff with the > > > >>>> resource > > > >>>> manager, but I'm not sure how that integration would work quite > yet, > > > so > > > >>>> we'll need to figure that one out. > > > >>>> > > > >>>> Let me know if you have any questions about the above. > > > >>>> > > > >>>> Tom > > > >>>> > > > >>>> > > > >>>> On Sat, Mar 18, 2017 at 12:39 PM, Imesha Sudasingha < > > > >>>> [email protected] > > > >>>> > wrote: > > > >>>> > > > >>>> > Hi Tom, > > > >>>> > > > > >>>> > What do you think of the design? Do you agree on implementing > this > > > >>>> > configuration management section in a separate module? > > > >>>> > > > > >>>> > I was thinking on starting on the implementation as a proof of > > > >>>> concept. Can > > > >>>> > you give me your kind feedback before I start on that? > > > >>>> > > > > >>>> > Thank you! > > > >>>> > > > > >>>> > Kind Regards, > > > >>>> > *Imesha Sudasingha* > > > >>>> > Undergraduate of Department of Computer Science and > Engineering, > > > >>>> > University of Moratuwa, > > > >>>> > Sri Lanka. > > > >>>> > > > > >>>> > <https://lk.linkedin.com/in/imeshasudasingha> < > > > >>>> https://github.com/IMS94> > > > >>>> > <http://stackoverflow.com/users/4012073/imesha-sudasingha> > > > >>>> > <https://twitter.com/Imesha94> > > > >>>> > > > > >>>> > On 16 March 2017 at 17:51, Imesha Sudasingha < > > > [email protected] > > > >>>> > > > > >>>> > wrote: > > > >>>> > > > > >>>> > > Hi Tom, > > > >>>> > > > > > >>>> > > Thanks for the quick reply. I came up with a small design [1]. > > > >>>> > > > > > >>>> > > What I suggest is a separate module for configuration > > management, > > > >>>> which > > > >>>> > > can be extended in future and used by many components as well. > > > >>>> > > > > > >>>> > > As shown in the design [1], File Manager (or any component > that > > is > > > >>>> > willing > > > >>>> > > to have this extended functionality of distributed > configuration > > > >>>> > > management) will have an instance of *ConfigurationManager* > > which > > > >>>> is an > > > >>>> > > interface defined to load configuration (specifically, > > .properties > > > >>>> files) > > > >>>> > > and get configurations of each running instance of the same > > > >>>> component (If > > > >>>> > > distributed configuration management is enabled, this will > > return > > > >>>> the > > > >>>> > > configuration of all the other instances). > > > >>>> > > > > > >>>> > > There will be two implementing classes of that interface, > > > >>>> > > *StandaloneConfigurationManager* and > > > *DistributedConfigurationManag > > > >>>> er.* > > > >>>> > As > > > >>>> > > the name implies, standalone configuration manager is to be > used > > > if > > > >>>> the > > > >>>> > > users do not intend to used distributed configuration > > management. > > > >>>> Also > > > >>>> > this > > > >>>> > > will be same as the current mechanism of configuration > > management. > > > >>>> Then > > > >>>> > the > > > >>>> > > distributed configuration manager will be the implementation > > using > > > >>>> > > Zookeeper. > > > >>>> > > > > > >>>> > > At zookeeper level, we will need to have a proper naming > > mechanism > > > >>>> and a > > > >>>> > > structure to store configuration. For example,* > > > >>>> > > /${nodeName}/${componentName}/{$configFileName}* will be the > > > place > > > >>>> where > > > >>>> > > the configuration file contents will be stored. > *getFile(String > > > >>>> fileName) > > > >>>> > > and getFiles(String directiryName)* of *NodeConfiguration* > class > > > >>>> will > > > >>>> > > return the content of the configuration files and > configuration > > > >>>> > directories > > > >>>> > > respectively. I have demonstrated the ZNode structre I'm > > expecting > > > >>>> to > > > >>>> > > design in [2] > > > >>>> > > > > > >>>> > > Please note that this is just a very high level overview of my > > > >>>> design and > > > >>>> > > the real difficulty of implementation comes when testing this > > and > > > >>>> > > addressing the edge cases of zookeeper. Also, I have only > added > > > few > > > >>>> > methods > > > >>>> > > that came to my mind to the *ConfigurationManager* interface. > If > > > >>>> you find > > > >>>> > > any fault in this design or have a better design in mind > please > > > let > > > >>>> me > > > >>>> > know > > > >>>> > > so that I can optimize this design and do my best to come up > > with > > > >>>> the > > > >>>> > best > > > >>>> > > possible solution. > > > >>>> > > > > > >>>> > > Thank you! > > > >>>> > > > > > >>>> > > [1] https://drive.google.com/open?id= > > 0B415Bnipaf1ldnV1dWt4VVVLam8 > > > >>>> > > [2] https://drive.google.com/open?id= > > 0B415Bnipaf1lNThKYXZMN3pFemc > > > >>>> > > > > > >>>> > > Kind Regards, > > > >>>> > > *Imesha Sudasingha* > > > >>>> > > Undergraduate of Department of Computer Science and > > Engineering, > > > >>>> > > University of Moratuwa, > > > >>>> > > Sri Lanka. > > > >>>> > > > > > >>>> > > <https://lk.linkedin.com/in/imeshasudasingha> < > > > >>>> https://github.com/IMS94 > > > >>>> > > > > > >>>> > > <http://stackoverflow.com/users/4012073/imesha-sudasingha> > > > >>>> > > <https://twitter.com/Imesha94> > > > >>>> > > > > > >>>> > > On 16 March 2017 at 16:55, Tom Barber < > [email protected]> > > > >>>> wrote: > > > >>>> > > > > > >>>> > >> Hi Imesha, > > > >>>> > >> > > > >>>> > >> Basically, just allowing the dynamic registration of file > > > managers > > > >>>> so > > > >>>> > that > > > >>>> > >> new ones can come and go. > > > >>>> > >> > > > >>>> > >> Tom > > > >>>> > >> > > > >>>> > >> On Thu, Mar 16, 2017 at 10:00 AM, Imesha Sudasingha < > > > >>>> > >> [email protected] > > > >>>> > >> > wrote: > > > >>>> > >> > > > >>>> > >> > Hi Tom, > > > >>>> > >> > > > > >>>> > >> > Need a small clarification. In the last reply, you > mentioned > > > >>>> that file > > > >>>> > >> > managers may be designed to be *transient. *What do you > mean > > by > > > >>>> that? > > > >>>> > A > > > >>>> > >> > quick response would really help me as I'm trying to come > up > > > >>>> with a > > > >>>> > >> design. > > > >>>> > >> > > > > >>>> > >> > Thank you! > > > >>>> > >> > > > > >>>> > >> > *Imesha Sudasingha* > > > >>>> > >> > Undergraduate of Department of Computer Science and > > > Engineering, > > > >>>> > >> > University of Moratuwa, > > > >>>> > >> > Sri Lanka. > > > >>>> > >> > > > > >>>> > >> > <https://lk.linkedin.com/in/imeshasudasingha> < > > > >>>> > >> https://github.com/IMS94> > > > >>>> > >> > <http://stackoverflow.com/users/4012073/imesha-sudasingha> > > > >>>> > >> > <https://twitter.com/Imesha94> > > > >>>> > >> > > > > >>>> > >> > On 13 March 2017 at 22:03, Imesha Sudasingha < > > > >>>> [email protected] > > > >>>> > > > > > >>>> > >> > wrote: > > > >>>> > >> > > > > >>>> > >> > > Hi Tom, > > > >>>> > >> > > > > > >>>> > >> > > Thanks for the clarification. It really helped a lot to > > fill > > > >>>> most of > > > >>>> > >> the > > > >>>> > >> > > gaps I had unanswered. I have a draft design in my mind > and > > > >>>> will > > > >>>> > post > > > >>>> > >> for > > > >>>> > >> > > your review with details soon. > > > >>>> > >> > > > > > >>>> > >> > > As for my current understanding, my task is to implement > a > > > >>>> mechanism > > > >>>> > >> to > > > >>>> > >> > > switch/chose configuration mechanism (between local files > > and > > > >>>> > zookeer > > > >>>> > >> for > > > >>>> > >> > > the moment) and then allow APIs to query those mechanisms > > (in > > > >>>> the > > > >>>> > >> > zookeeper > > > >>>> > >> > > scenario) for future use (For File Managers to know about > > > other > > > >>>> > >> instances > > > >>>> > >> > > and make decisions accordingly). That include > > > synchronization, > > > >>>> > >> fetching > > > >>>> > >> > > configuration from zookeeper if an instance go down and > > come > > > >>>> again > > > >>>> > >> and so > > > >>>> > >> > > on. Am I correct? > > > >>>> > >> > > > > > >>>> > >> > > I will try to come up with the design I mentioned, so > that > > > >>>> both of > > > >>>> > us > > > >>>> > >> can > > > >>>> > >> > > discuss more about the improvements and requirements > soon. > > > >>>> > >> > > > > > >>>> > >> > > Thank you for the support! > > > >>>> > >> > > > > > >>>> > >> > > Kind Regards, > > > >>>> > >> > > *Imesha Sudasingha* > > > >>>> > >> > > Undergraduate of Department of Computer Science and > > > >>>> Engineering, > > > >>>> > >> > > University of Moratuwa, > > > >>>> > >> > > Sri Lanka. > > > >>>> > >> > > > > > >>>> > >> > > <https://lk.linkedin.com/in/imeshasudasingha> < > > > >>>> > >> https://github.com/IMS94 > > > >>>> > >> > > > > > >>>> > >> > > <http://stackoverflow.com/users/4012073/imesha- > sudasingha> > > > >>>> > >> > > <https://twitter.com/Imesha94> > > > >>>> > >> > > > > > >>>> > >> > > On 13 March 2017 at 20:19, Tom Barber < > > > [email protected] > > > >>>> > > > > >>>> > >> wrote: > > > >>>> > >> > > > > > >>>> > >> > >> Hi Imesha > > > >>>> > >> > >> > > > >>>> > >> > >> The best way to demonstrate this is to show you a > > reference > > > >>>> build > > > >>>> > for > > > >>>> > >> > >> OODT.. > > > >>>> > >> > >> > > > >>>> > >> > >> https://dl.dropboxusercontent. > > > com/u/8503756/oodt-example.tgz > > > >>>> > >> > >> > > > >>>> > >> > >> If you can grab that archive and extract it, you'll see > > that > > > >>>> is how > > > >>>> > >> > >> currently OODT is deployed. > > > >>>> > >> > >> > > > >>>> > >> > >> If you take the file manager as the reference module > > inside > > > >>>> > >> oodt/filemgr > > > >>>> > >> > >> you'll see > > > >>>> > >> > >> > > > >>>> > >> > >> etc/filemgr.properties > > > >>>> > >> > >> > > > >>>> > >> > >> and > > > >>>> > >> > >> > > > >>>> > >> > >> policy/oodt/* > > > >>>> > >> > >> > > > >>>> > >> > >> These are the configuration locations of a single file > > > >>>> manager. > > > >>>> > >> > >> > > > >>>> > >> > >> The problem I want to solve is this: > > > >>>> > >> > >> > > > >>>> > >> > >> a) If I run > 1 file manager they are completely > separate > > > and > > > >>>> they > > > >>>> > >> don't > > > >>>> > >> > >> know of each other. So using Zookeeper will allow > multiple > > > >>>> file > > > >>>> > >> managers > > > >>>> > >> > >> to > > > >>>> > >> > >> register against ZK and learn of each other. > > > >>>> > >> > >> > > > >>>> > >> > >> b) Backing them onto ZK will allow file managers to come > > and > > > >>>> go on > > > >>>> > >> the > > > >>>> > >> > >> fly, > > > >>>> > >> > >> providing scaling capabilities or maybe fm's that are > > > >>>> designed to > > > >>>> > be > > > >>>> > >> > >> transient. > > > >>>> > >> > >> > > > >>>> > >> > >> Just for clarity, the ZK configuration will be an > optional > > > >>>> > >> requirement, > > > >>>> > >> > so > > > >>>> > >> > >> we need to extend the existing configuration mechanism, > to > > > >>>> support > > > >>>> > >> both > > > >>>> > >> > >> the > > > >>>> > >> > >> current config and the ZK config instead of ZK being > > > >>>> mandatory. > > > >>>> > >> > >> > > > >>>> > >> > >> Tom > > > >>>> > >> > >> > > > >>>> > >> > >> On Mon, Mar 13, 2017 at 3:20 AM, Imesha Sudasingha < > > > >>>> > >> > >> [email protected]> > > > >>>> > >> > >> wrote: > > > >>>> > >> > >> > > > >>>> > >> > >> > Hi Tom, > > > >>>> > >> > >> > > > > >>>> > >> > >> > Related to the conversation on JIRA [1] ; > > > >>>> > >> > >> > > > > >>>> > >> > >> > I went through the CAS-File manager module and I now > > > >>>> understand > > > >>>> > >> how to > > > >>>> > >> > >> > address ".properties" files. Since you mentioned in > JIRA > > > >>>> [2] that > > > >>>> > >> > there > > > >>>> > >> > >> are > > > >>>> > >> > >> > occasions where we use *spring configuration files > *for > > > >>>> > >> configuration > > > >>>> > >> > >> > purposes, I want to have a look at that type of > > > >>>> configuration > > > >>>> > too. > > > >>>> > >> Can > > > >>>> > >> > >> you > > > >>>> > >> > >> > give me an example module where spring configuration > > files > > > >>>> are > > > >>>> > >> used to > > > >>>> > >> > >> load > > > >>>> > >> > >> > configuration? > > > >>>> > >> > >> > > > > >>>> > >> > >> > Also, will the content of a configuration file exceed > > 1MB > > > >>>> > (Maximum > > > >>>> > >> > data > > > >>>> > >> > >> > size allowed in an ZNode) in any case? > > > >>>> > >> > >> > > > > >>>> > >> > >> > Thank you! > > > >>>> > >> > >> > > > > >>>> > >> > >> > [1] > > > >>>> > >> > >> > https://issues.apache.org/jira > > > >>>> /browse/OODT-945?focusedCommen > > > >>>> > >> > >> tId=15906204& > > > >>>> > >> > >> > page=com.atlassian.jira.plugin.system.issuetabpanels: > > > >>>> > >> > >> > comment-tabpanel#comment-15906204 > > > >>>> > >> > >> > > > > >>>> > >> > >> > [2] > > > >>>> > >> > >> > https://issues.apache.org/jira > > > >>>> /browse/OODT-945?focusedCommen > > > >>>> > >> > >> tId=15906524& > > > >>>> > >> > >> > page=com.atlassian.jira.plugin.system.issuetabpanels: > > > >>>> > >> > >> > comment-tabpanel#comment-15906524 > > > >>>> > >> > >> > > > > >>>> > >> > >> > Kind Regards, > > > >>>> > >> > >> > *Imesha Sudasingha* > > > >>>> > >> > >> > Undergraduate of Department of Computer Science and > > > >>>> Engineering, > > > >>>> > >> > >> > University of Moratuwa, > > > >>>> > >> > >> > Sri Lanka. > > > >>>> > >> > >> > > > > >>>> > >> > >> > <https://lk.linkedin.com/in/imeshasudasingha> < > > > >>>> > >> > >> https://github.com/IMS94> > > > >>>> > >> > >> > <http://stackoverflow.com/users/4012073/imesha- > > sudasingha > > > > > > > >>>> > >> > >> > <https://twitter.com/Imesha94> > > > >>>> > >> > >> > > > > >>>> > >> > >> > > > >>>> > >> > > > > > >>>> > >> > > > > > >>>> > >> > > > > >>>> > >> > > > >>>> > > > > > >>>> > > > > > >>>> > > > > >>>> > > > >>> > > > >>> > > > >> > > > > > > > > > >
