[jira] [Commented] (OODT-945) Rework OODT configuration to make use of Zookeeper for distributed configuration management

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/OODT-945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16200234#comment-16200234
 ] 

ASF GitHub Bot commented on OODT-945:
-

Github user IMS94 closed the pull request at:

https://github.com/apache/oodt/pull/51


> Rework OODT configuration to make use of  Zookeeper for distributed 
> configuration management
> 
>
> Key: OODT-945
> URL: https://issues.apache.org/jira/browse/OODT-945
> Project: OODT
>  Issue Type: Bug
>Reporter: Tom Barber
>Assignee: Imesha Sudasingha
>  Labels: gsoc2017, mentor
> Fix For: 1.3
>
>
> OODT is made up of a number of different components, each of which have their 
> own set of configuration files and locations. This is complex to manage and 
> causes additional problems when the platform requires distribution across 
> servers or geographically.
> To combat this we are proposing that we migrate the OODT configuration to an 
> opional Zookeeper module that would allow users to commission a zookeeper 
> cluster and register the various OODT components against it, which will allow 
> the zookeeper cluster to maintain the state of the various OODT components 
> regardless of scale.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OODT-945) Rework OODT configuration to make use of Zookeeper for distributed configuration management

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/OODT-945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16200233#comment-16200233
 ] 

ASF GitHub Bot commented on OODT-945:
-

Github user IMS94 commented on the issue:

https://github.com/apache/oodt/pull/51
  
Included in #52 


> Rework OODT configuration to make use of  Zookeeper for distributed 
> configuration management
> 
>
> Key: OODT-945
> URL: https://issues.apache.org/jira/browse/OODT-945
> Project: OODT
>  Issue Type: Bug
>Reporter: Tom Barber
>Assignee: Imesha Sudasingha
>  Labels: gsoc2017, mentor
> Fix For: 1.3
>
>
> OODT is made up of a number of different components, each of which have their 
> own set of configuration files and locations. This is complex to manage and 
> causes additional problems when the platform requires distribution across 
> servers or geographically.
> To combat this we are proposing that we migrate the OODT configuration to an 
> opional Zookeeper module that would allow users to commission a zookeeper 
> cluster and register the various OODT components against it, which will allow 
> the zookeeper cluster to maintain the state of the various OODT components 
> regardless of scale.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OODT-945) Rework OODT configuration to make use of Zookeeper for distributed configuration management

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/OODT-945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16200231#comment-16200231
 ] 

ASF GitHub Bot commented on OODT-945:
-

GitHub user IMS94 opened a pull request:

https://github.com/apache/oodt/pull/52

 Fixed surefire test failures in workflow manager due to 

Workflow manager tests failed in when running `mvn clean install`. The 
problem was with the sure fire plugin in workflow manager. `` was set 
to `never` earlier. This has caused some environment variables I was setting 
through surefire to affect other tests as well. Changed it to `pertest`. Tests 
are passing now, except 
`org.apache.oodt.cas.workflow.system.TestAvroRpcWorkflowManager` due to a NPE 
which requires to be addressed separately.

This PR also include some missing commits which adds the feature to watch 
for configuration changes and act accordingly for **[OODT-945]**

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/IMS94/oodt development

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/oodt/pull/52.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #52


commit 711b445cf3d5964ebff8fd72e06fbfc451a19ff0
Author: Imesha Sudasingha 
Date:   2017-08-19T10:51:10Z

Added configuration change listening feature

commit bd86cd657516eb221759920e941b84322a8a52e6
Author: Imesha Sudasingha 
Date:   2017-08-23T14:02:48Z

Improved java docs

commit fb6c2fad13c6ff7f107b6f6147e661996ed74a27
Author: Imesha Sudasingha 
Date:   2017-10-11T10:00:34Z

Merge branch 'feature/zookeeper-config' of https://github.com/IMS94/oodt 
into development

commit f8a9b284a7fed1b5e1b120cfdb2a52fedb539c26
Author: Imesha Sudasingha 
Date:   2017-10-11T12:49:28Z

Fixed surefire test failures in workflow manager due to 




> Rework OODT configuration to make use of  Zookeeper for distributed 
> configuration management
> 
>
> Key: OODT-945
> URL: https://issues.apache.org/jira/browse/OODT-945
> Project: OODT
>  Issue Type: Bug
>Reporter: Tom Barber
>Assignee: Imesha Sudasingha
>  Labels: gsoc2017, mentor
> Fix For: 1.3
>
>
> OODT is made up of a number of different components, each of which have their 
> own set of configuration files and locations. This is complex to manage and 
> causes additional problems when the platform requires distribution across 
> servers or geographically.
> To combat this we are proposing that we migrate the OODT configuration to an 
> opional Zookeeper module that would allow users to commission a zookeeper 
> cluster and register the various OODT components against it, which will allow 
> the zookeeper cluster to maintain the state of the various OODT components 
> regardless of scale.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OODT-945) Rework OODT configuration to make use of Zookeeper for distributed configuration management

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/OODT-945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16130697#comment-16130697
 ] 

ASF GitHub Bot commented on OODT-945:
-

Github user chrismattmann commented on the issue:

https://github.com/apache/oodt/pull/50
  
reviewed, and tested! LGTM and committed!


> Rework OODT configuration to make use of  Zookeeper for distributed 
> configuration management
> 
>
> Key: OODT-945
> URL: https://issues.apache.org/jira/browse/OODT-945
> Project: OODT
>  Issue Type: Bug
>Reporter: Tom Barber
>Assignee: Tom Barber
>  Labels: gsoc2017, mentor
> Fix For: 1.3
>
>
> OODT is made up of a number of different components, each of which have their 
> own set of configuration files and locations. This is complex to manage and 
> causes additional problems when the platform requires distribution across 
> servers or geographically.
> To combat this we are proposing that we migrate the OODT configuration to an 
> opional Zookeeper module that would allow users to commission a zookeeper 
> cluster and register the various OODT components against it, which will allow 
> the zookeeper cluster to maintain the state of the various OODT components 
> regardless of scale.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OODT-945) Rework OODT configuration to make use of Zookeeper for distributed configuration management

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/OODT-945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16129981#comment-16129981
 ] 

ASF GitHub Bot commented on OODT-945:
-

GitHub user IMS94 opened a pull request:

https://github.com/apache/oodt/pull/50

[OODT-945] Introduced distributed configuration management for workflow and 
resource managers

Up to now, distributed configuration management was tested and implemented 
for file manager only (#44 and #48). This PR is introducing the same for both 
workflow manager and resource manager. 

This also include corresponding unit tests to ensure functionality of those 
components with distributed configuration management enabled.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/IMS94/oodt feature/zookeeper-config

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/oodt/pull/50.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #50


commit 57a9235e71a9d869291e80ba9bd16eeddd4a8e10
Author: Imesha Sudasingha 
Date:   2017-08-06T09:34:36Z

Merge branch 'master' of https://github.com/apache/oodt into 
feature/zookeeper-config

commit 9ee8af48ba45904a44d4e32b3030e8ad7d0bee21
Author: Imesha Sudasingha 
Date:   2017-08-10T13:02:27Z

Improvements to filemgr and config-publisher bash scripts

commit c437106fb9292547f8d87ada4601aab6b9d97ee9
Author: Imesha Sudasingha 
Date:   2017-08-10T13:03:18Z

Merge branch 'feature/zookeeper-config' of https://github.com/apache/oodt 
into feature/zookeeper-config

commit 57e544658f0716395c78b3c0cdaa3875b1c2cdff
Author: Imesha Sudasingha 
Date:   2017-08-12T06:51:29Z

Added cleaning up directories created when testing file manager

commit ef0aa5a75ff5fad964b7a0c5dfb7f6d5ea500484
Author: Tom Barber 
Date:   2017-08-14T22:07:38Z

update changes for 1.2

commit 7443a7c37a993afffa66270282a6a6c5aa8074ba
Author: Tom Barber 
Date:   2017-08-14T22:31:51Z

update version

commit 3df0e43de5f8c78348cff5da766e75f62343a380
Author: Imesha Sudasingha 
Date:   2017-08-15T17:01:30Z

Merge branch 'master' of https://github.com/apache/oodt into 
feature/zookeeper-config

commit fc6311db2ed4e5ccc6129cb88c71bb5428037b98
Author: Imesha Sudasingha 
Date:   2017-08-15T19:47:42Z

Introduced distributed configuration management to workflow and resource 
managers + tests




> Rework OODT configuration to make use of  Zookeeper for distributed 
> configuration management
> 
>
> Key: OODT-945
> URL: https://issues.apache.org/jira/browse/OODT-945
> Project: OODT
>  Issue Type: Bug
>Reporter: Tom Barber
>Assignee: Tom Barber
>  Labels: gsoc2017, mentor
> Fix For: 1.3
>
>
> OODT is made up of a number of different components, each of which have their 
> own set of configuration files and locations. This is complex to manage and 
> causes additional problems when the platform requires distribution across 
> servers or geographically.
> To combat this we are proposing that we migrate the OODT configuration to an 
> opional Zookeeper module that would allow users to commission a zookeeper 
> cluster and register the various OODT components against it, which will allow 
> the zookeeper cluster to maintain the state of the various OODT components 
> regardless of scale.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OODT-945) Rework OODT configuration to make use of Zookeeper for distributed configuration management

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/OODT-945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16120267#comment-16120267
 ] 

ASF GitHub Bot commented on OODT-945:
-

Github user chrismattmann commented on the issue:

https://github.com/apache/oodt/pull/48
  
build passes, all tests pass, and code looks  

committing!

```
[INFO] >>> maven-javadoc-plugin:2.10.3:javadoc (attach-javadocs) > 
generate-sources @ oodt >>>
[INFO] 
[INFO] <<< maven-javadoc-plugin:2.10.3:javadoc (attach-javadocs) < 
generate-sources @ oodt <<<
[INFO] 
[INFO] --- maven-javadoc-plugin:2.10.3:javadoc (attach-javadocs) @ oodt ---
[INFO] 
[INFO] --- maven-install-plugin:2.5.2:install (default-install) @ oodt ---
[INFO] Installing /Users/mattmann/git/oodt/pom.xml to 
/Users/mattmann/.m2/repository/org/apache/oodt/oodt/1.2-SNAPSHOT/oodt-1.2-SNAPSHOT.pom
[INFO] 

[INFO] Reactor Summary:
[INFO] 
[INFO] OODT Core .. SUCCESS [  
1.465 s]
[INFO] Common Utilities ... SUCCESS [ 
17.148 s]
[INFO] CAS Command Line Interface . SUCCESS [  
4.770 s]
[INFO] OODT - Configuration Management  SUCCESS [  
5.153 s]
[INFO] Process Control System Input Data Package .. SUCCESS [  
1.305 s]
[INFO] Catalog and Archive Service Generic Multi-valued Metadata Container 
SUCCESS [  2.218 s]
[INFO] Catalog and Archive File Management Component .. SUCCESS [02:20 
min]
[INFO] Catalog and Archive Resource Management Component .. SUCCESS [ 
15.761 s]
[INFO] Catalog and Archive Workflow Management Component .. SUCCESS [ 
25.660 s]
[INFO] Catalog and Archive Crawling Framework . SUCCESS [ 
14.219 s]
[INFO] OODT CAS Curator Single Sign On Security Package ... SUCCESS [  
1.307 s]
[INFO] CAS Curation Web Services .. SUCCESS [  
6.256 s]
[INFO] Process Control System Core Package  SUCCESS [ 
11.753 s]
[INFO] OODT Wicket Web Components . SUCCESS [  
3.793 s]
[INFO] CAS Curation Interface . SUCCESS [  
7.610 s]
[INFO] CAS PGE Adaptor Framework .. SUCCESS [ 
13.339 s]
[INFO] CAS Installer Maven Mojo ... SUCCESS [  
2.329 s]
[INFO] OODT :: Archetypes :: OpsUI  SUCCESS [  
0.821 s]
[INFO] OODT :: Archetypes :: RADiX  SUCCESS [  
0.978 s]
[INFO] OODT :: Archetypes . SUCCESS [  
0.024 s]
[INFO] CAS File Manager Browser Web App ... SUCCESS [  
7.333 s]
[INFO] CAS Workflow Manager Monitor Web App ... SUCCESS [  
3.690 s]
[INFO] CAS Product Server Web Application . SUCCESS [  
9.914 s]
[INFO] CAS Workflow REST Services . SUCCESS [  
2.872 s]
[INFO] Process Control System Operator Interface Webapp ... SUCCESS [ 
11.993 s]
[INFO] OODT Process Control System JAX-RS service layer ... SUCCESS [  
4.930 s]
[INFO] Apache OODT  SUCCESS [  
0.019 s]
[INFO] 

[INFO] BUILD SUCCESS
[INFO] 

[INFO] Total time: 05:17 min
[INFO] Finished at: 2017-08-09T10:07:27-07:00
[INFO] Final Memory: 107M/1465M
[INFO] 

LMC-053601:oodt mattmann$ 
```
thanks @IMS94 


> Rework OODT configuration to make use of  Zookeeper for distributed 
> configuration management
> 
>
> Key: OODT-945
> URL: https://issues.apache.org/jira/browse/OODT-945
> Project: OODT
>  Issue Type: Bug
>Reporter: Tom Barber
>Assignee: Tom Barber
>  Labels: gsoc2017, mentor
>
> OODT is made up of a number of different components, each of which have their 
> own set of configuration files and locations. This is complex to manage and 
> causes additional problems when the platform requires distribution across 
> servers or geographically.
> To combat this we are proposing that we migrate the OODT configuration to an 
> opional Zookeeper module that would allow users to commission a zookeeper 
> cluster and register the various OODT components against it, which will allow 
> the zookeeper cluster to maintain the state of the various OODT components 
> regardless of scale.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OODT-945) Rework OODT configuration to make use of Zookeeper for distributed configuration management

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/OODT-945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16120269#comment-16120269
 ] 

ASF GitHub Bot commented on OODT-945:
-

Github user asfgit closed the pull request at:

https://github.com/apache/oodt/pull/48


> Rework OODT configuration to make use of  Zookeeper for distributed 
> configuration management
> 
>
> Key: OODT-945
> URL: https://issues.apache.org/jira/browse/OODT-945
> Project: OODT
>  Issue Type: Bug
>Reporter: Tom Barber
>Assignee: Tom Barber
>  Labels: gsoc2017, mentor
>
> OODT is made up of a number of different components, each of which have their 
> own set of configuration files and locations. This is complex to manage and 
> causes additional problems when the platform requires distribution across 
> servers or geographically.
> To combat this we are proposing that we migrate the OODT configuration to an 
> opional Zookeeper module that would allow users to commission a zookeeper 
> cluster and register the various OODT components against it, which will allow 
> the zookeeper cluster to maintain the state of the various OODT components 
> regardless of scale.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OODT-945) Rework OODT configuration to make use of Zookeeper for distributed configuration management

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/OODT-945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16104466#comment-16104466
 ] 

ASF GitHub Bot commented on OODT-945:
-

Github user chrismattmann commented on the issue:

https://github.com/apache/oodt/pull/48
  
will review tomorrow


> Rework OODT configuration to make use of  Zookeeper for distributed 
> configuration management
> 
>
> Key: OODT-945
> URL: https://issues.apache.org/jira/browse/OODT-945
> Project: OODT
>  Issue Type: Bug
>Reporter: Tom Barber
>Assignee: Tom Barber
>  Labels: gsoc2017, mentor
>
> OODT is made up of a number of different components, each of which have their 
> own set of configuration files and locations. This is complex to manage and 
> causes additional problems when the platform requires distribution across 
> servers or geographically.
> To combat this we are proposing that we migrate the OODT configuration to an 
> opional Zookeeper module that would allow users to commission a zookeeper 
> cluster and register the various OODT components against it, which will allow 
> the zookeeper cluster to maintain the state of the various OODT components 
> regardless of scale.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OODT-945) Rework OODT configuration to make use of Zookeeper for distributed configuration management

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/OODT-945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16103592#comment-16103592
 ] 

ASF GitHub Bot commented on OODT-945:
-

Github user IMS94 commented on the issue:

https://github.com/apache/oodt/pull/48
  
hey @chrismattmann, Should be ok now?


> Rework OODT configuration to make use of  Zookeeper for distributed 
> configuration management
> 
>
> Key: OODT-945
> URL: https://issues.apache.org/jira/browse/OODT-945
> Project: OODT
>  Issue Type: Bug
>Reporter: Tom Barber
>Assignee: Tom Barber
>  Labels: gsoc2017, mentor
>
> OODT is made up of a number of different components, each of which have their 
> own set of configuration files and locations. This is complex to manage and 
> causes additional problems when the platform requires distribution across 
> servers or geographically.
> To combat this we are proposing that we migrate the OODT configuration to an 
> opional Zookeeper module that would allow users to commission a zookeeper 
> cluster and register the various OODT components against it, which will allow 
> the zookeeper cluster to maintain the state of the various OODT components 
> regardless of scale.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OODT-945) Rework OODT configuration to make use of Zookeeper for distributed configuration management

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/OODT-945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16103494#comment-16103494
 ] 

ASF GitHub Bot commented on OODT-945:
-

Github user chrismattmann commented on the issue:

https://github.com/apache/oodt/pull/48
  
hey @IMS94 can you squash/rebase your local and then resubmit your PR? I'm 
getting a bunch of changes that are already there in the remote branch. Try git 
pull before you rebase.


> Rework OODT configuration to make use of  Zookeeper for distributed 
> configuration management
> 
>
> Key: OODT-945
> URL: https://issues.apache.org/jira/browse/OODT-945
> Project: OODT
>  Issue Type: Bug
>Reporter: Tom Barber
>Assignee: Tom Barber
>  Labels: gsoc2017, mentor
>
> OODT is made up of a number of different components, each of which have their 
> own set of configuration files and locations. This is complex to manage and 
> causes additional problems when the platform requires distribution across 
> servers or geographically.
> To combat this we are proposing that we migrate the OODT configuration to an 
> opional Zookeeper module that would allow users to commission a zookeeper 
> cluster and register the various OODT components against it, which will allow 
> the zookeeper cluster to maintain the state of the various OODT components 
> regardless of scale.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OODT-945) Rework OODT configuration to make use of Zookeeper for distributed configuration management

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/OODT-945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16088419#comment-16088419
 ] 

ASF GitHub Bot commented on OODT-945:
-

Github user IMS94 commented on the issue:

https://github.com/apache/oodt/pull/44
  
@chrismattmann Thanks for your support too. Did the tests pass for you 
without a problem?


> Rework OODT configuration to make use of  Zookeeper for distributed 
> configuration management
> 
>
> Key: OODT-945
> URL: https://issues.apache.org/jira/browse/OODT-945
> Project: OODT
>  Issue Type: Bug
>Reporter: Tom Barber
>Assignee: Tom Barber
>  Labels: gsoc2017, mentor
>
> OODT is made up of a number of different components, each of which have their 
> own set of configuration files and locations. This is complex to manage and 
> causes additional problems when the platform requires distribution across 
> servers or geographically.
> To combat this we are proposing that we migrate the OODT configuration to an 
> opional Zookeeper module that would allow users to commission a zookeeper 
> cluster and register the various OODT components against it, which will allow 
> the zookeeper cluster to maintain the state of the various OODT components 
> regardless of scale.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OODT-945) Rework OODT configuration to make use of Zookeeper for distributed configuration management

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/OODT-945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16088408#comment-16088408
 ] 

ASF GitHub Bot commented on OODT-945:
-

Github user chrismattmann commented on the issue:

https://github.com/apache/oodt/pull/44
  
merged this thanks @IMS94 

```
LMC-053601:oodt mattmann$ git push -u origin feature/zookeeper-config
Counting objects: 475, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (203/203), done.
Writing objects: 100% (475/475), 81.85 KiB | 0 bytes/s, done.
Total 475 (delta 155), reused 355 (delta 113)
remote: oodt git commit: Update log4j config.
remote: oodt git commit: Merge branch 'feature/zookeeper-config' of 
https://github.com/IMS94/oodt into feature/zookeeper-config
remote: oodt git commit: assert not null.
remote: oodt git commit: Fix for empty component home variable
remote: oodt git commit: Merge and apply robust changes for stack trace 
print in tests.
remote: oodt git commit: Merge branch 'feature/zookeeper-config' of 
https://github.com/IMS94/oodt into feature/zookeeper-config
remote: oodt git commit: fix for presence of component home environment 
variables in tests
remote: oodt git commit: Merge branch 'feature/zookeeper-config' of 
https://github.com/IMS94/oodt into feature/zookeeper-config
remote: oodt git commit: fix pom.xml
remote: oodt git commit: Fixed parent artifactId in config/pom.xml
remote: oodt git commit: Removed * imports. Improved code formatting and 
refactored code.
remote: oodt git commit: Merge branch 'master' of 
https://github.com/apache/oodt into feature/zookeeper-config
remote: oodt git commit: Update README.md
remote: oodt git commit: Merge branch 'OODT-847' of 
http://github.com/adhulipa/oodt
remote: oodt git commit: Record change for OODT-847.
remote: oodt git commit: Merge branch 'master' into feature/zookeeper-config
remote: oodt git commit: Improvements to file downloading from zookeeper 
and tests
remote: oodt git commit: Minor improvements for configuration manager tests
remote: oodt git commit: Created test cases for distributed configuration 
manager and publisher
remote: oodt git commit: Added zip packaging for distributed configuration 
publisher
remote: oodt git commit: Implemented configuration fetching for 
DistributedConfigurationManager
remote: oodt git commit: Altered configuration publishing mechanism to use 
spring configuration
remote: oodt git commit: Implemented global configuration publishing 
mechanism.
remote: oodt git commit: Defined initial API for ConfigurationManager
remote: oodt git commit: Fix for OODT-847 contributed by adhulipa & rverma
To https://git-wip-us.apache.org/repos/asf/oodt.git
   ec5b201..3fb7660  feature/zookeeper-config -> feature/zookeeper-config
Branch feature/zookeeper-config set up to track remote branch 
feature/zookeeper-config from origin.
LMC-053601:oodt mattmann$ 
```


> Rework OODT configuration to make use of  Zookeeper for distributed 
> configuration management
> 
>
> Key: OODT-945
> URL: https://issues.apache.org/jira/browse/OODT-945
> Project: OODT
>  Issue Type: Bug
>Reporter: Tom Barber
>Assignee: Tom Barber
>  Labels: gsoc2017, mentor
>
> OODT is made up of a number of different components, each of which have their 
> own set of configuration files and locations. This is complex to manage and 
> causes additional problems when the platform requires distribution across 
> servers or geographically.
> To combat this we are proposing that we migrate the OODT configuration to an 
> opional Zookeeper module that would allow users to commission a zookeeper 
> cluster and register the various OODT components against it, which will allow 
> the zookeeper cluster to maintain the state of the various OODT components 
> regardless of scale.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OODT-945) Rework OODT configuration to make use of Zookeeper for distributed configuration management

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/OODT-945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16087529#comment-16087529
 ] 

ASF GitHub Bot commented on OODT-945:
-

Github user IMS94 commented on the issue:

https://github.com/apache/oodt/pull/44
  
Hi @chrismattmann, Thanks for the invitation. 

I have pushed an update. There's still a chance for the tests to fail 
because I store configuration files in `${COMPONENT_HOME}` directory if 
`${COMPONENT_HOME}` is present, else in the working directory. If it fails 
again, I will find a more convenient method for my implementation.


> Rework OODT configuration to make use of  Zookeeper for distributed 
> configuration management
> 
>
> Key: OODT-945
> URL: https://issues.apache.org/jira/browse/OODT-945
> Project: OODT
>  Issue Type: Bug
>Reporter: Tom Barber
>Assignee: Tom Barber
>  Labels: gsoc2017, mentor
>
> OODT is made up of a number of different components, each of which have their 
> own set of configuration files and locations. This is complex to manage and 
> causes additional problems when the platform requires distribution across 
> servers or geographically.
> To combat this we are proposing that we migrate the OODT configuration to an 
> opional Zookeeper module that would allow users to commission a zookeeper 
> cluster and register the various OODT components against it, which will allow 
> the zookeeper cluster to maintain the state of the various OODT components 
> regardless of scale.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OODT-945) Rework OODT configuration to make use of Zookeeper for distributed configuration management

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/OODT-945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16087514#comment-16087514
 ] 

ASF GitHub Bot commented on OODT-945:
-

Github user chrismattmann commented on the issue:

https://github.com/apache/oodt/pull/44
  
OK @IMS94 I have invited you to the USC Data Science slack 


> Rework OODT configuration to make use of  Zookeeper for distributed 
> configuration management
> 
>
> Key: OODT-945
> URL: https://issues.apache.org/jira/browse/OODT-945
> Project: OODT
>  Issue Type: Bug
>Reporter: Tom Barber
>Assignee: Tom Barber
>  Labels: gsoc2017, mentor
>
> OODT is made up of a number of different components, each of which have their 
> own set of configuration files and locations. This is complex to manage and 
> causes additional problems when the platform requires distribution across 
> servers or geographically.
> To combat this we are proposing that we migrate the OODT configuration to an 
> opional Zookeeper module that would allow users to commission a zookeeper 
> cluster and register the various OODT components against it, which will allow 
> the zookeeper cluster to maintain the state of the various OODT components 
> regardless of scale.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OODT-945) Rework OODT configuration to make use of Zookeeper for distributed configuration management

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/OODT-945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16087511#comment-16087511
 ] 

ASF GitHub Bot commented on OODT-945:
-

Github user chrismattmann commented on the issue:

https://github.com/apache/oodt/pull/44
  
@IMS94 feel free to request an invite to uscdatascience.slack.com I'll make 
a channel for slack chat if you want real time comm on this


> Rework OODT configuration to make use of  Zookeeper for distributed 
> configuration management
> 
>
> Key: OODT-945
> URL: https://issues.apache.org/jira/browse/OODT-945
> Project: OODT
>  Issue Type: Bug
>Reporter: Tom Barber
>Assignee: Tom Barber
>  Labels: gsoc2017, mentor
>
> OODT is made up of a number of different components, each of which have their 
> own set of configuration files and locations. This is complex to manage and 
> causes additional problems when the platform requires distribution across 
> servers or geographically.
> To combat this we are proposing that we migrate the OODT configuration to an 
> opional Zookeeper module that would allow users to commission a zookeeper 
> cluster and register the various OODT components against it, which will allow 
> the zookeeper cluster to maintain the state of the various OODT components 
> regardless of scale.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OODT-945) Rework OODT configuration to make use of Zookeeper for distributed configuration management

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/OODT-945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16087499#comment-16087499
 ] 

ASF GitHub Bot commented on OODT-945:
-

Github user IMS94 commented on the issue:

https://github.com/apache/oodt/pull/44
  
Hi @chrismattmann, Thanks for the update. That is the reason for the test 
failure. Actually this is a case I should have addressed within my code. I have 
wrote the fix. Will push it now. 


> Rework OODT configuration to make use of  Zookeeper for distributed 
> configuration management
> 
>
> Key: OODT-945
> URL: https://issues.apache.org/jira/browse/OODT-945
> Project: OODT
>  Issue Type: Bug
>Reporter: Tom Barber
>Assignee: Tom Barber
>  Labels: gsoc2017, mentor
>
> OODT is made up of a number of different components, each of which have their 
> own set of configuration files and locations. This is complex to manage and 
> causes additional problems when the platform requires distribution across 
> servers or geographically.
> To combat this we are proposing that we migrate the OODT configuration to an 
> opional Zookeeper module that would allow users to commission a zookeeper 
> cluster and register the various OODT components against it, which will allow 
> the zookeeper cluster to maintain the state of the various OODT components 
> regardless of scale.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OODT-945) Rework OODT configuration to make use of Zookeeper for distributed configuration management

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/OODT-945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16087494#comment-16087494
 ] 

ASF GitHub Bot commented on OODT-945:
-

Github user IMS94 commented on the issue:

https://github.com/apache/oodt/pull/44
  
Hi @chrismattmann, No. Tests do not require any environment variable to be 
set. But if any of the above mentioned env variables are set, that could be a 
possible reason for your test failure (because I didn't encounter test 
failures). I'm using environment variables to determine each OODT component's 
HOME when downloading and storing configuration files from zookeeper.

Therefore, what I wanted know was whether you have any of those env vars 
set.


> Rework OODT configuration to make use of  Zookeeper for distributed 
> configuration management
> 
>
> Key: OODT-945
> URL: https://issues.apache.org/jira/browse/OODT-945
> Project: OODT
>  Issue Type: Bug
>Reporter: Tom Barber
>Assignee: Tom Barber
>  Labels: gsoc2017, mentor
>
> OODT is made up of a number of different components, each of which have their 
> own set of configuration files and locations. This is complex to manage and 
> causes additional problems when the platform requires distribution across 
> servers or geographically.
> To combat this we are proposing that we migrate the OODT configuration to an 
> opional Zookeeper module that would allow users to commission a zookeeper 
> cluster and register the various OODT components against it, which will allow 
> the zookeeper cluster to maintain the state of the various OODT components 
> regardless of scale.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OODT-945) Rework OODT configuration to make use of Zookeeper for distributed configuration management

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/OODT-945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16087492#comment-16087492
 ] 

ASF GitHub Bot commented on OODT-945:
-

Github user chrismattmann commented on the issue:

https://github.com/apache/oodt/pull/44
  
So I checked, and I do have those environment variables set, except for 
```RESMGR_HOME``` e.g., 

```
LMC-053601:oodt mattmann$ echo $FILEMGR_HOME
/usr/local/bigtranslate/filemgr
LMC-053601:oodt mattmann$ echo $FILEMGR_URL
http://localhost:9000
LMC-053601:oodt mattmann$ 
LMC-053601:oodt mattmann$ echo $RESMGR_URL
http://localhost:9002
LMC-053601:oodt mattmann$ 
```

I was thinking about your code - is it that if an environment variable from 
local config isn't that set - you are publishing a null value to ZK or there is 
a getPropertyOrNull call somewhere that is causing this? If so, we should 
probably have a sensibleDefaults() configuration which are all the sensible 
defaults in case the environment is Null. Even better would be to use PathUtils 
in o.a.cas.metadata.util to do EnvVarOrMetadata replacement on all paths you 
read from config.




> Rework OODT configuration to make use of  Zookeeper for distributed 
> configuration management
> 
>
> Key: OODT-945
> URL: https://issues.apache.org/jira/browse/OODT-945
> Project: OODT
>  Issue Type: Bug
>Reporter: Tom Barber
>Assignee: Tom Barber
>  Labels: gsoc2017, mentor
>
> OODT is made up of a number of different components, each of which have their 
> own set of configuration files and locations. This is complex to manage and 
> causes additional problems when the platform requires distribution across 
> servers or geographically.
> To combat this we are proposing that we migrate the OODT configuration to an 
> opional Zookeeper module that would allow users to commission a zookeeper 
> cluster and register the various OODT components against it, which will allow 
> the zookeeper cluster to maintain the state of the various OODT components 
> regardless of scale.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OODT-945) Rework OODT configuration to make use of Zookeeper for distributed configuration management

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/OODT-945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16087383#comment-16087383
 ] 

ASF GitHub Bot commented on OODT-945:
-

Github user chrismattmann commented on the issue:

https://github.com/apache/oodt/pull/44
  
hi @IMS94 do I need to set those env vars? This will screw up tests since 
they should not have environment variable requirements, or if so, should be 
mock'ed in the tests themselves.


> Rework OODT configuration to make use of  Zookeeper for distributed 
> configuration management
> 
>
> Key: OODT-945
> URL: https://issues.apache.org/jira/browse/OODT-945
> Project: OODT
>  Issue Type: Bug
>Reporter: Tom Barber
>Assignee: Tom Barber
>  Labels: gsoc2017, mentor
>
> OODT is made up of a number of different components, each of which have their 
> own set of configuration files and locations. This is complex to manage and 
> causes additional problems when the platform requires distribution across 
> servers or geographically.
> To combat this we are proposing that we migrate the OODT configuration to an 
> opional Zookeeper module that would allow users to commission a zookeeper 
> cluster and register the various OODT components against it, which will allow 
> the zookeeper cluster to maintain the state of the various OODT components 
> regardless of scale.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OODT-945) Rework OODT configuration to make use of Zookeeper for distributed configuration management

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/OODT-945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16086892#comment-16086892
 ] 

ASF GitHub Bot commented on OODT-945:
-

Github user IMS94 commented on the issue:

https://github.com/apache/oodt/pull/44
  
Hi @chrismattmann, Can let me know do you have any of previously set up 
`FILEMGR_HOME` or `RESMGR_HOME` environment variables available in your system? 


> Rework OODT configuration to make use of  Zookeeper for distributed 
> configuration management
> 
>
> Key: OODT-945
> URL: https://issues.apache.org/jira/browse/OODT-945
> Project: OODT
>  Issue Type: Bug
>Reporter: Tom Barber
>Assignee: Tom Barber
>  Labels: gsoc2017, mentor
>
> OODT is made up of a number of different components, each of which have their 
> own set of configuration files and locations. This is complex to manage and 
> causes additional problems when the platform requires distribution across 
> servers or geographically.
> To combat this we are proposing that we migrate the OODT configuration to an 
> opional Zookeeper module that would allow users to commission a zookeeper 
> cluster and register the various OODT components against it, which will allow 
> the zookeeper cluster to maintain the state of the various OODT components 
> regardless of scale.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OODT-945) Rework OODT configuration to make use of Zookeeper for distributed configuration management

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/OODT-945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16086866#comment-16086866
 ] 

ASF GitHub Bot commented on OODT-945:
-

Github user chrismattmann commented on the issue:

https://github.com/apache/oodt/pull/44
  
hi @IMS94 got further, but failing on unit tests, thoughts?

```
---
 T E S T S
---
Running 
org.apache.oodt.config.distributed.DistributedConfigurationManagerTest
log4j:WARN No appenders could be found for logger 
(org.apache.zookeeper.server.quorum.QuorumPeerConfig).
log4j:WARN Please initialize the log4j system properly.
Starting configuration publishing

Processing commands for component : FILE_MANAGER
Publishing configuration for : FILE_MANAGER
Published configuration for : FILE_MANAGER


Processing commands for component : RESOURCE_MANAGER
Publishing configuration for : RESOURCE_MANAGER
Published configuration for : RESOURCE_MANAGER

Starting configuration publishing

Processing commands for component : FILE_MANAGER
Clearing configuration for : FILE_MANAGER
Cleared configuration for : FILE_MANAGER


Processing commands for component : RESOURCE_MANAGER
Clearing configuration for : RESOURCE_MANAGER
Cleared configuration for : RESOURCE_MANAGER

Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 1.209 sec 
<<< FAILURE! - in 
org.apache.oodt.config.distributed.DistributedConfigurationManagerTest

loadConfigurationTest(org.apache.oodt.config.distributed.DistributedConfigurationManagerTest)
  Time elapsed: 0.049 sec  <<< FAILURE!
java.lang.AssertionError: null
at org.junit.Assert.fail(Assert.java:86)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertTrue(Assert.java:52)
at 
org.apache.oodt.config.distributed.DistributedConfigurationManagerTest.loadConfigurationTest(DistributedConfigurationManagerTest.java:91)

Running 
org.apache.oodt.config.distributed.DistributedConfigurationPublisherTest
Starting configuration publishing

Processing commands for component : FILE_MANAGER
Publishing configuration for : FILE_MANAGER
Published configuration for : FILE_MANAGER

Verifying configuration for : FILE_MANAGER
OK... Configuration verified
Verified configuration for : FILE_MANAGER


Processing commands for component : RESOURCE_MANAGER
Publishing configuration for : RESOURCE_MANAGER
Published configuration for : RESOURCE_MANAGER

Verifying configuration for : RESOURCE_MANAGER
OK... Configuration verified
Verified configuration for : RESOURCE_MANAGER

Starting configuration publishing

Processing commands for component : FILE_MANAGER
Clearing configuration for : FILE_MANAGER
Cleared configuration for : FILE_MANAGER


Processing commands for component : RESOURCE_MANAGER
Clearing configuration for : RESOURCE_MANAGER
Cleared configuration for : RESOURCE_MANAGER

Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.512 sec - 
in org.apache.oodt.config.distributed.DistributedConfigurationPublisherTest

Results :

Failed tests: 
  DistributedConfigurationManagerTest.loadConfigurationTest:91 null

Tests run: 2, Failures: 1, Errors: 0, Skipped: 0

[INFO] 

[INFO] Reactor Summary:
[INFO] 
[INFO] OODT Core .. SUCCESS [  
1.820 s]
[INFO] Common Utilities ... SUCCESS [ 
17.797 s]
[INFO] CAS Command Line Interface . SUCCESS [  
6.046 s]
[INFO] OODT - Configuration Management  FAILURE [  
5.426 s]
[INFO] Process Control System Input Data Package .. SKIPPED
[INFO] Catalog and Archive Service Generic Multi-valued Metadata Container 
SKIPPED
[INFO] CAS Protocol ... SKIPPED
[INFO] CAS Protocol FTP Implementation  SKIPPED
[INFO] CAS Protocol HTTP Implementation ... SKIPPED
[INFO] CAS Protocol IMAPS Implementation .. SKIPPED
[INFO] CAS Protocol SFTP Implementation ... SKIPPED
[INFO] Query Expression ... SKIPPED
[INFO] OODT Single Sign On Security Package ... SKIPPED
[INFO] Catalog and Archive File Management Component .. SKIPPED
[INFO] OODT CAS Virtual Catalog and Integration Service. .. SKIPPED
[INFO] Catalog and Archive Resource Management Component .. SKIPPED
[INFO] Catalog and Archive Workflow Management Component 

[jira] [Commented] (OODT-945) Rework OODT configuration to make use of Zookeeper for distributed configuration management

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/OODT-945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16086761#comment-16086761
 ] 

ASF GitHub Bot commented on OODT-945:
-

Github user IMS94 commented on the issue:

https://github.com/apache/oodt/pull/44
  
Hi @chrismattmann, can you check now? I had put wrong parent artifactId 
name in `config/pom.xml` earlier. It should build now.


> Rework OODT configuration to make use of  Zookeeper for distributed 
> configuration management
> 
>
> Key: OODT-945
> URL: https://issues.apache.org/jira/browse/OODT-945
> Project: OODT
>  Issue Type: Bug
>Reporter: Tom Barber
>Assignee: Tom Barber
>  Labels: gsoc2017, mentor
>
> OODT is made up of a number of different components, each of which have their 
> own set of configuration files and locations. This is complex to manage and 
> causes additional problems when the platform requires distribution across 
> servers or geographically.
> To combat this we are proposing that we migrate the OODT configuration to an 
> opional Zookeeper module that would allow users to commission a zookeeper 
> cluster and register the various OODT components against it, which will allow 
> the zookeeper cluster to maintain the state of the various OODT components 
> regardless of scale.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OODT-945) Rework OODT configuration to make use of Zookeeper for distributed configuration management

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/OODT-945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16086055#comment-16086055
 ] 

ASF GitHub Bot commented on OODT-945:
-

Github user chrismattmann commented on the issue:

https://github.com/apache/oodt/pull/44
  
hi @IMS94 I have errors building this locally can you check:

```
LMC-053601:oodt mattmann$ mvn clean install
[INFO] Scanning for projects...
[ERROR] [ERROR] Some problems were encountered while processing the POMs:
[WARNING] 'parent.relativePath' of POM 
org.apache.oodt:oodt-conf:[unknown-version] 
(/Users/mattmann/git/oodt/config/pom.xml) points at org.apache.oodt:oodt-core 
instead of org.apache.oodt:oodt, please verify your project structure @ 
org.apache.oodt:oodt-conf:[unknown-version], 
/Users/mattmann/git/oodt/config/pom.xml, line 5, column 13
[ERROR] 'dependencies.dependency.version' for 
org.apache.curator:curator-framework:jar is missing. @ 
org.apache.oodt:oodt-conf:[unknown-version], 
/Users/mattmann/git/oodt/config/pom.xml, line 19, column 21
[ERROR] 'dependencies.dependency.version' for args4j:args4j:jar is missing. 
@ org.apache.oodt:oodt-conf:[unknown-version], 
/Users/mattmann/git/oodt/config/pom.xml, line 39, column 21
[ERROR] 'dependencies.dependency.version' for 
org.apache.curator:curator-test:jar is missing. @ 
org.apache.oodt:oodt-conf:[unknown-version], 
/Users/mattmann/git/oodt/config/pom.xml, line 55, column 21
 @ 
[ERROR] The build could not read 1 project -> [Help 1]
[ERROR]   
[ERROR]   The project org.apache.oodt:oodt-conf:1.1-SNAPSHOT 
(/Users/mattmann/git/oodt/config/pom.xml) has 3 errors
[ERROR] 'dependencies.dependency.version' for 
org.apache.curator:curator-framework:jar is missing. @ 
org.apache.oodt:oodt-conf:[unknown-version], 
/Users/mattmann/git/oodt/config/pom.xml, line 19, column 21
[ERROR] 'dependencies.dependency.version' for args4j:args4j:jar is 
missing. @ org.apache.oodt:oodt-conf:[unknown-version], 
/Users/mattmann/git/oodt/config/pom.xml, line 39, column 21
[ERROR] 'dependencies.dependency.version' for 
org.apache.curator:curator-test:jar is missing. @ 
org.apache.oodt:oodt-conf:[unknown-version], 
/Users/mattmann/git/oodt/config/pom.xml, line 55, column 21
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e 
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, 
please read the following articles:
[ERROR] [Help 1] 
http://cwiki.apache.org/confluence/display/MAVEN/ProjectBuildingException
```



> Rework OODT configuration to make use of  Zookeeper for distributed 
> configuration management
> 
>
> Key: OODT-945
> URL: https://issues.apache.org/jira/browse/OODT-945
> Project: OODT
>  Issue Type: Bug
>Reporter: Tom Barber
>Assignee: Tom Barber
>  Labels: gsoc2017, mentor
>
> OODT is made up of a number of different components, each of which have their 
> own set of configuration files and locations. This is complex to manage and 
> causes additional problems when the platform requires distribution across 
> servers or geographically.
> To combat this we are proposing that we migrate the OODT configuration to an 
> opional Zookeeper module that would allow users to commission a zookeeper 
> cluster and register the various OODT components against it, which will allow 
> the zookeeper cluster to maintain the state of the various OODT components 
> regardless of scale.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OODT-945) Rework OODT configuration to make use of Zookeeper for distributed configuration management

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/OODT-945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16082018#comment-16082018
 ] 

ASF GitHub Bot commented on OODT-945:
-

Github user chrismattmann commented on a diff in the pull request:

https://github.com/apache/oodt/pull/44#discussion_r126652721
  
--- Diff: README.md ---
@@ -53,8 +50,7 @@ To contribute a patch, follow these instructions (note 
that installing
 ```
 
 
-License (see also LICENSE.txt)
-==
+## License (see also LICENSE.txt)
--- End diff --

Hi @IMS94 I haven't applied *all of them*, just the *first one*, so can you 
please open a new PR with the other changes you did and remove/rollback this 
one.


> Rework OODT configuration to make use of  Zookeeper for distributed 
> configuration management
> 
>
> Key: OODT-945
> URL: https://issues.apache.org/jira/browse/OODT-945
> Project: OODT
>  Issue Type: Bug
>Reporter: Tom Barber
>Assignee: Tom Barber
>  Labels: gsoc2017, mentor
>
> OODT is made up of a number of different components, each of which have their 
> own set of configuration files and locations. This is complex to manage and 
> causes additional problems when the platform requires distribution across 
> servers or geographically.
> To combat this we are proposing that we migrate the OODT configuration to an 
> opional Zookeeper module that would allow users to commission a zookeeper 
> cluster and register the various OODT components against it, which will allow 
> the zookeeper cluster to maintain the state of the various OODT components 
> regardless of scale.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OODT-945) Rework OODT configuration to make use of Zookeeper for distributed configuration management

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/OODT-945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16081683#comment-16081683
 ] 

ASF GitHub Bot commented on OODT-945:
-

Github user chrismattmann commented on a diff in the pull request:

https://github.com/apache/oodt/pull/44#discussion_r126597857
  
--- Diff: config/src/main/resources/examples/filemgr/cmd-line-actions.xml 
---
@@ -0,0 +1,123 @@
+
--- End diff --

thanks check out Maven Antrun plugin


> Rework OODT configuration to make use of  Zookeeper for distributed 
> configuration management
> 
>
> Key: OODT-945
> URL: https://issues.apache.org/jira/browse/OODT-945
> Project: OODT
>  Issue Type: Bug
>Reporter: Tom Barber
>Assignee: Tom Barber
>  Labels: gsoc2017, mentor
>
> OODT is made up of a number of different components, each of which have their 
> own set of configuration files and locations. This is complex to manage and 
> causes additional problems when the platform requires distribution across 
> servers or geographically.
> To combat this we are proposing that we migrate the OODT configuration to an 
> opional Zookeeper module that would allow users to commission a zookeeper 
> cluster and register the various OODT components against it, which will allow 
> the zookeeper cluster to maintain the state of the various OODT components 
> regardless of scale.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OODT-945) Rework OODT configuration to make use of Zookeeper for distributed configuration management

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/OODT-945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16081489#comment-16081489
 ] 

ASF GitHub Bot commented on OODT-945:
-

Github user IMS94 commented on a diff in the pull request:

https://github.com/apache/oodt/pull/44#discussion_r126577868
  
--- Diff: 
filemgr/src/main/java/org/apache/oodt/cas/filemgr/system/XmlRpcFileManager.java 
---
@@ -52,29 +41,20 @@
 import org.apache.oodt.cas.metadata.Metadata;
 import org.apache.oodt.cas.metadata.exceptions.MetExtractionException;
 import org.apache.oodt.commons.date.DateUtils;
+import org.apache.oodt.config.ConfigurationManager;
+import org.apache.oodt.config.ConfigurationManagerFactory;
+import org.apache.oodt.config.Constants;
 import org.apache.xmlrpc.WebServer;
 
-import com.google.common.collect.Lists;
-
-import java.io.File;
-import java.io.FileInputStream;
-import java.io.FileNotFoundException;
-import java.io.FileOutputStream;
-import java.io.IOException;
+import java.io.*;
 import java.net.URI;
 import java.net.URISyntaxException;
 import java.net.URL;
-import java.util.Arrays;
-import java.util.Hashtable;
-import java.util.LinkedList;
-import java.util.List;
-import java.util.Map;
-import java.util.Vector;
+import java.util.*;
--- End diff --

@chrismattmann Sorry, it was my fault. I was using IntelliJ IDEA's optimize 
imports command. I will fix this.


> Rework OODT configuration to make use of  Zookeeper for distributed 
> configuration management
> 
>
> Key: OODT-945
> URL: https://issues.apache.org/jira/browse/OODT-945
> Project: OODT
>  Issue Type: Bug
>Reporter: Tom Barber
>Assignee: Tom Barber
>  Labels: gsoc2017, mentor
>
> OODT is made up of a number of different components, each of which have their 
> own set of configuration files and locations. This is complex to manage and 
> causes additional problems when the platform requires distribution across 
> servers or geographically.
> To combat this we are proposing that we migrate the OODT configuration to an 
> opional Zookeeper module that would allow users to commission a zookeeper 
> cluster and register the various OODT components against it, which will allow 
> the zookeeper cluster to maintain the state of the various OODT components 
> regardless of scale.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OODT-945) Rework OODT configuration to make use of Zookeeper for distributed configuration management

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/OODT-945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16081491#comment-16081491
 ] 

ASF GitHub Bot commented on OODT-945:
-

Github user IMS94 commented on a diff in the pull request:

https://github.com/apache/oodt/pull/44#discussion_r126577969
  
--- Diff: config/src/main/resources/examples/filemgr/cmd-line-actions.xml 
---
@@ -0,0 +1,123 @@
+
--- End diff --

@chrismattmann I will look into that. Thanks for pointing that out. Didn't 
know we can do that with a plugin before.


> Rework OODT configuration to make use of  Zookeeper for distributed 
> configuration management
> 
>
> Key: OODT-945
> URL: https://issues.apache.org/jira/browse/OODT-945
> Project: OODT
>  Issue Type: Bug
>Reporter: Tom Barber
>Assignee: Tom Barber
>  Labels: gsoc2017, mentor
>
> OODT is made up of a number of different components, each of which have their 
> own set of configuration files and locations. This is complex to manage and 
> causes additional problems when the platform requires distribution across 
> servers or geographically.
> To combat this we are proposing that we migrate the OODT configuration to an 
> opional Zookeeper module that would allow users to commission a zookeeper 
> cluster and register the various OODT components against it, which will allow 
> the zookeeper cluster to maintain the state of the various OODT components 
> regardless of scale.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OODT-945) Rework OODT configuration to make use of Zookeeper for distributed configuration management

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/OODT-945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16081487#comment-16081487
 ] 

ASF GitHub Bot commented on OODT-945:
-

Github user IMS94 commented on a diff in the pull request:

https://github.com/apache/oodt/pull/44#discussion_r126577697
  
--- Diff: README.md ---
@@ -53,8 +50,7 @@ To contribute a patch, follow these instructions (note 
that installing
 ```
 
 
-License (see also LICENSE.txt)
-==
+## License (see also LICENSE.txt)
--- End diff --

@chrismattmann Sure. But I think you have applied these changes to the 
Readme.md with the commit [update 
README.md](https://github.com/apache/oodt/commit/1b42cacacfc96e96b69f0c28d7eefcb89ae85856).
 Therefore, I will rollback this change


> Rework OODT configuration to make use of  Zookeeper for distributed 
> configuration management
> 
>
> Key: OODT-945
> URL: https://issues.apache.org/jira/browse/OODT-945
> Project: OODT
>  Issue Type: Bug
>Reporter: Tom Barber
>Assignee: Tom Barber
>  Labels: gsoc2017, mentor
>
> OODT is made up of a number of different components, each of which have their 
> own set of configuration files and locations. This is complex to manage and 
> causes additional problems when the platform requires distribution across 
> servers or geographically.
> To combat this we are proposing that we migrate the OODT configuration to an 
> opional Zookeeper module that would allow users to commission a zookeeper 
> cluster and register the various OODT components against it, which will allow 
> the zookeeper cluster to maintain the state of the various OODT components 
> regardless of scale.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OODT-945) Rework OODT configuration to make use of Zookeeper for distributed configuration management

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/OODT-945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16080897#comment-16080897
 ] 

ASF GitHub Bot commented on OODT-945:
-

Github user chrismattmann commented on a diff in the pull request:

https://github.com/apache/oodt/pull/44#discussion_r126509803
  
--- Diff: config/src/main/resources/examples/filemgr/filemgr.properties ---
@@ -0,0 +1,114 @@
+#
--- End diff --

same why copy these when we can use some maven copy task


> Rework OODT configuration to make use of  Zookeeper for distributed 
> configuration management
> 
>
> Key: OODT-945
> URL: https://issues.apache.org/jira/browse/OODT-945
> Project: OODT
>  Issue Type: Bug
>Reporter: Tom Barber
>Assignee: Tom Barber
>  Labels: gsoc2017, mentor
>
> OODT is made up of a number of different components, each of which have their 
> own set of configuration files and locations. This is complex to manage and 
> causes additional problems when the platform requires distribution across 
> servers or geographically.
> To combat this we are proposing that we migrate the OODT configuration to an 
> opional Zookeeper module that would allow users to commission a zookeeper 
> cluster and register the various OODT components against it, which will allow 
> the zookeeper cluster to maintain the state of the various OODT components 
> regardless of scale.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OODT-945) Rework OODT configuration to make use of Zookeeper for distributed configuration management

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/OODT-945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16080900#comment-16080900
 ] 

ASF GitHub Bot commented on OODT-945:
-

Github user chrismattmann commented on the issue:

https://github.com/apache/oodt/pull/44
  
please see my review comments above - great start!


> Rework OODT configuration to make use of  Zookeeper for distributed 
> configuration management
> 
>
> Key: OODT-945
> URL: https://issues.apache.org/jira/browse/OODT-945
> Project: OODT
>  Issue Type: Bug
>Reporter: Tom Barber
>Assignee: Tom Barber
>  Labels: gsoc2017, mentor
>
> OODT is made up of a number of different components, each of which have their 
> own set of configuration files and locations. This is complex to manage and 
> causes additional problems when the platform requires distribution across 
> servers or geographically.
> To combat this we are proposing that we migrate the OODT configuration to an 
> opional Zookeeper module that would allow users to commission a zookeeper 
> cluster and register the various OODT components against it, which will allow 
> the zookeeper cluster to maintain the state of the various OODT components 
> regardless of scale.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OODT-945) Rework OODT configuration to make use of Zookeeper for distributed configuration management

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/OODT-945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16080901#comment-16080901
 ] 

ASF GitHub Bot commented on OODT-945:
-

Github user chrismattmann commented on the issue:

https://github.com/apache/oodt/pull/44
  
@IMS94 


> Rework OODT configuration to make use of  Zookeeper for distributed 
> configuration management
> 
>
> Key: OODT-945
> URL: https://issues.apache.org/jira/browse/OODT-945
> Project: OODT
>  Issue Type: Bug
>Reporter: Tom Barber
>Assignee: Tom Barber
>  Labels: gsoc2017, mentor
>
> OODT is made up of a number of different components, each of which have their 
> own set of configuration files and locations. This is complex to manage and 
> causes additional problems when the platform requires distribution across 
> servers or geographically.
> To combat this we are proposing that we migrate the OODT configuration to an 
> opional Zookeeper module that would allow users to commission a zookeeper 
> cluster and register the various OODT components against it, which will allow 
> the zookeeper cluster to maintain the state of the various OODT components 
> regardless of scale.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OODT-945) Rework OODT configuration to make use of Zookeeper for distributed configuration management

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/OODT-945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16080899#comment-16080899
 ] 

ASF GitHub Bot commented on OODT-945:
-

Github user chrismattmann commented on a diff in the pull request:

https://github.com/apache/oodt/pull/44#discussion_r126509940
  
--- Diff: 
filemgr/src/main/java/org/apache/oodt/cas/filemgr/system/XmlRpcFileManager.java 
---
@@ -52,29 +41,20 @@
 import org.apache.oodt.cas.metadata.Metadata;
 import org.apache.oodt.cas.metadata.exceptions.MetExtractionException;
 import org.apache.oodt.commons.date.DateUtils;
+import org.apache.oodt.config.ConfigurationManager;
+import org.apache.oodt.config.ConfigurationManagerFactory;
+import org.apache.oodt.config.Constants;
 import org.apache.xmlrpc.WebServer;
 
-import com.google.common.collect.Lists;
-
-import java.io.File;
-import java.io.FileInputStream;
-import java.io.FileNotFoundException;
-import java.io.FileOutputStream;
-import java.io.IOException;
+import java.io.*;
 import java.net.URI;
 import java.net.URISyntaxException;
 import java.net.URL;
-import java.util.Arrays;
-import java.util.Hashtable;
-import java.util.LinkedList;
-import java.util.List;
-import java.util.Map;
-import java.util.Vector;
+import java.util.*;
--- End diff --

we don't use star imports and these changes are orthogonal


> Rework OODT configuration to make use of  Zookeeper for distributed 
> configuration management
> 
>
> Key: OODT-945
> URL: https://issues.apache.org/jira/browse/OODT-945
> Project: OODT
>  Issue Type: Bug
>Reporter: Tom Barber
>Assignee: Tom Barber
>  Labels: gsoc2017, mentor
>
> OODT is made up of a number of different components, each of which have their 
> own set of configuration files and locations. This is complex to manage and 
> causes additional problems when the platform requires distribution across 
> servers or geographically.
> To combat this we are proposing that we migrate the OODT configuration to an 
> opional Zookeeper module that would allow users to commission a zookeeper 
> cluster and register the various OODT components against it, which will allow 
> the zookeeper cluster to maintain the state of the various OODT components 
> regardless of scale.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OODT-945) Rework OODT configuration to make use of Zookeeper for distributed configuration management

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/OODT-945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16080894#comment-16080894
 ] 

ASF GitHub Bot commented on OODT-945:
-

Github user chrismattmann commented on a diff in the pull request:

https://github.com/apache/oodt/pull/44#discussion_r126509639
  
--- Diff: config/src/main/resources/examples/filemgr/cmd-line-actions.xml 
---
@@ -0,0 +1,123 @@
+
--- End diff --

maybe we should use maven-ant-tasks to copy these in the prepare-resources 
phase or in the compile phase (whatever you need them for) instead of 
physically copying the files?


> Rework OODT configuration to make use of  Zookeeper for distributed 
> configuration management
> 
>
> Key: OODT-945
> URL: https://issues.apache.org/jira/browse/OODT-945
> Project: OODT
>  Issue Type: Bug
>Reporter: Tom Barber
>Assignee: Tom Barber
>  Labels: gsoc2017, mentor
>
> OODT is made up of a number of different components, each of which have their 
> own set of configuration files and locations. This is complex to manage and 
> causes additional problems when the platform requires distribution across 
> servers or geographically.
> To combat this we are proposing that we migrate the OODT configuration to an 
> opional Zookeeper module that would allow users to commission a zookeeper 
> cluster and register the various OODT components against it, which will allow 
> the zookeeper cluster to maintain the state of the various OODT components 
> regardless of scale.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OODT-945) Rework OODT configuration to make use of Zookeeper for distributed configuration management

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/OODT-945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16080818#comment-16080818
 ] 

ASF GitHub Bot commented on OODT-945:
-

Github user chrismattmann commented on a diff in the pull request:

https://github.com/apache/oodt/pull/44#discussion_r126501802
  
--- Diff: README.md ---
@@ -53,8 +50,7 @@ To contribute a patch, follow these instructions (note 
that installing
 ```
 
 
-License (see also LICENSE.txt)
-==
+## License (see also LICENSE.txt)
--- End diff --

great changes, these are orthogonal though - can you separate them into a 
new PR apart from this?


> Rework OODT configuration to make use of  Zookeeper for distributed 
> configuration management
> 
>
> Key: OODT-945
> URL: https://issues.apache.org/jira/browse/OODT-945
> Project: OODT
>  Issue Type: Bug
>Reporter: Tom Barber
>Assignee: Tom Barber
>  Labels: gsoc2017, mentor
>
> OODT is made up of a number of different components, each of which have their 
> own set of configuration files and locations. This is complex to manage and 
> causes additional problems when the platform requires distribution across 
> servers or geographically.
> To combat this we are proposing that we migrate the OODT configuration to an 
> opional Zookeeper module that would allow users to commission a zookeeper 
> cluster and register the various OODT components against it, which will allow 
> the zookeeper cluster to maintain the state of the various OODT components 
> regardless of scale.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OODT-945) Rework OODT configuration to make use of Zookeeper for distributed configuration management

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/OODT-945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16080405#comment-16080405
 ] 

ASF GitHub Bot commented on OODT-945:
-

GitHub user IMS94 opened a pull request:

https://github.com/apache/oodt/pull/44

OODT-945 Rework OODT configuration to make use of Zookeeper for distributed 
configuration management (GSoC 2017)

This PR includes the Distributed Configuration Management implementation 
for OODT [OODT-945] along with the distributed configuration publishing CLI 
[OODT-952].

Explanation on the design of this feature is available at [Rework OODT 
configuration to make use of Zookeeper for distributed configuration 
management](https://cwiki.apache.org/confluence/display/OODT/Rework+OODT+configuration+to+make+use+of+Zookeeper+for+distributed+configuration+management).

User documentation on how to use this distributed configuration management 
feature is available at [OODT Distributed Configuration 
Management](https://cwiki.apache.org/confluence/display/OODT/OODT+Distributed+Configuration+Management)

Corresponding GSoC project proposal is available at 
[https://cwiki.apache.org/confluence/display/OODT/Proposal](https://cwiki.apache.org/confluence/display/OODT/Proposal)

**Since this is the first glance of the distributed configuration 
management feature, there can be bugs and improvements required to be 
addressed. Please review this PR and provide me with your kind feedback so that 
I can improve on that.**

Thank you!

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/IMS94/oodt feature/zookeeper-config

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/oodt/pull/44.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #44


commit ac7170bdf5c973374f141c6f3e031fa748e0af13
Author: Imesha Sudasingha 
Date:   2017-04-01T07:00:55Z

Defined initial API for ConfigurationManager

commit 4b88f71497263675bb313462075fd552696ba717
Author: Imesha Sudasingha 
Date:   2017-06-16T10:23:43Z

Implemented global configuration publishing mechanism.

commit fadf149cb77dd943af9e81c0f127ce74f203c478
Author: Imesha Sudasingha 
Date:   2017-06-17T15:39:58Z

Altered configuration publishing mechanism to use spring configuration

commit 5fcf2220125b64afba080c6c9c6c5592daae2dea
Author: Imesha Sudasingha 
Date:   2017-06-17T17:51:12Z

Implemented configuration fetching for DistributedConfigurationManager

commit 27a28daa3d3c448b462bd9a33eb6ac4fd859b75a
Author: Imesha Sudasingha 
Date:   2017-07-02T15:23:49Z

Added zip packaging for distributed configuration publisher

commit a973c531fee5bf37ebf4448334470c75943af1ec
Author: Imesha Sudasingha 
Date:   2017-07-03T10:25:22Z

Created test cases for distributed configuration manager and publisher

commit e697405223fe17c96739ae8f156639a0f5ef321c
Author: Imesha Sudasingha 
Date:   2017-07-09T16:48:40Z

Minor improvements for configuration manager tests

commit 99b086b9887cafc8b89de3ff65fa61d3f80cc56c
Author: Imesha Sudasingha 
Date:   2017-07-10T11:51:51Z

Improvements to file downloading from zookeeper and tests

commit a8092af5484d066a4dda623395bf21612d696d62
Author: Imesha Sudasingha 
Date:   2017-07-10T14:24:28Z

Merge branch 'master' into feature/zookeeper-config




> Rework OODT configuration to make use of  Zookeeper for distributed 
> configuration management
> 
>
> Key: OODT-945
> URL: https://issues.apache.org/jira/browse/OODT-945
> Project: OODT
>  Issue Type: Bug
>Reporter: Tom Barber
>Assignee: Tom Barber
>  Labels: gsoc2017, mentor
>
> OODT is made up of a number of different components, each of which have their 
> own set of configuration files and locations. This is complex to manage and 
> causes additional problems when the platform requires distribution across 
> servers or geographically.
> To combat this we are proposing that we migrate the OODT configuration to an 
> opional Zookeeper module that would allow users to commission a zookeeper 
> cluster and register the various OODT components against it, which will allow 
> the zookeeper cluster to maintain the state of the various OODT components 
> regardless of scale.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OODT-945) Rework OODT configuration to make use of Zookeeper for distributed configuration management

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/OODT-945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16080378#comment-16080378
 ] 

ASF GitHub Bot commented on OODT-945:
-

Github user IMS94 commented on the issue:

https://github.com/apache/oodt/pull/43
  
Will open this again on a feature branch.


> Rework OODT configuration to make use of  Zookeeper for distributed 
> configuration management
> 
>
> Key: OODT-945
> URL: https://issues.apache.org/jira/browse/OODT-945
> Project: OODT
>  Issue Type: Bug
>Reporter: Tom Barber
>Assignee: Tom Barber
>  Labels: gsoc2017, mentor
>
> OODT is made up of a number of different components, each of which have their 
> own set of configuration files and locations. This is complex to manage and 
> causes additional problems when the platform requires distribution across 
> servers or geographically.
> To combat this we are proposing that we migrate the OODT configuration to an 
> opional Zookeeper module that would allow users to commission a zookeeper 
> cluster and register the various OODT components against it, which will allow 
> the zookeeper cluster to maintain the state of the various OODT components 
> regardless of scale.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OODT-945) Rework OODT configuration to make use of Zookeeper for distributed configuration management

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/OODT-945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16080377#comment-16080377
 ] 

ASF GitHub Bot commented on OODT-945:
-

GitHub user IMS94 reopened a pull request:

https://github.com/apache/oodt/pull/43

OODT-945 Rework OODT configuration to make use of Zookeeper for distributed 
configuration management (GSoC 2017)

This PR includes the Distributed Configuration Management implementation 
for OODT [OODT-945] along with the distributed configuration publishing CLI 
[OODT-952].

Explanation on the design of this feature is available at [Rework OODT 
configuration to make use of Zookeeper for distributed configuration 
management](https://cwiki.apache.org/confluence/display/OODT/Rework+OODT+configuration+to+make+use+of+Zookeeper+for+distributed+configuration+management).

User documentation on how to use this distributed configuration management 
feature is available at [OODT Distributed Configuration 
Management](https://cwiki.apache.org/confluence/display/OODT/OODT+Distributed+Configuration+Management)

Corresponding GSoC project proposal is available at 
[https://cwiki.apache.org/confluence/display/OODT/Proposal](https://cwiki.apache.org/confluence/display/OODT/Proposal)

**Since this is the first glance of the distributed configuration 
management feature, there can be bugs and improvements required to be 
addressed. Please review this PR and provide me with your kind feedback so that 
I can improve on that**

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/IMS94/oodt master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/oodt/pull/43.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #43






> Rework OODT configuration to make use of  Zookeeper for distributed 
> configuration management
> 
>
> Key: OODT-945
> URL: https://issues.apache.org/jira/browse/OODT-945
> Project: OODT
>  Issue Type: Bug
>Reporter: Tom Barber
>Assignee: Tom Barber
>  Labels: gsoc2017, mentor
>
> OODT is made up of a number of different components, each of which have their 
> own set of configuration files and locations. This is complex to manage and 
> causes additional problems when the platform requires distribution across 
> servers or geographically.
> To combat this we are proposing that we migrate the OODT configuration to an 
> opional Zookeeper module that would allow users to commission a zookeeper 
> cluster and register the various OODT components against it, which will allow 
> the zookeeper cluster to maintain the state of the various OODT components 
> regardless of scale.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OODT-945) Rework OODT configuration to make use of Zookeeper for distributed configuration management

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/OODT-945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16080376#comment-16080376
 ] 

ASF GitHub Bot commented on OODT-945:
-

Github user IMS94 closed the pull request at:

https://github.com/apache/oodt/pull/43


> Rework OODT configuration to make use of  Zookeeper for distributed 
> configuration management
> 
>
> Key: OODT-945
> URL: https://issues.apache.org/jira/browse/OODT-945
> Project: OODT
>  Issue Type: Bug
>Reporter: Tom Barber
>Assignee: Tom Barber
>  Labels: gsoc2017, mentor
>
> OODT is made up of a number of different components, each of which have their 
> own set of configuration files and locations. This is complex to manage and 
> causes additional problems when the platform requires distribution across 
> servers or geographically.
> To combat this we are proposing that we migrate the OODT configuration to an 
> opional Zookeeper module that would allow users to commission a zookeeper 
> cluster and register the various OODT components against it, which will allow 
> the zookeeper cluster to maintain the state of the various OODT components 
> regardless of scale.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OODT-945) Rework OODT configuration to make use of Zookeeper for distributed configuration management

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/OODT-945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16080343#comment-16080343
 ] 

ASF GitHub Bot commented on OODT-945:
-

GitHub user IMS94 opened a pull request:

https://github.com/apache/oodt/pull/43

OODT-945 Rework OODT configuration to make use of Zookeeper for distributed 
configuration management (GSoC 2017)

This PR includes the Distributed Configuration Management implementation 
for OODT [OODT-945] along with the distributed configuration publishing CLI 
[OODT-952].

Explanation on the design of this feature is available at [Rework OODT 
configuration to make use of Zookeeper for distributed configuration 
management](https://cwiki.apache.org/confluence/display/OODT/Rework+OODT+configuration+to+make+use+of+Zookeeper+for+distributed+configuration+management).

User documentation on how to use this distributed configuration management 
feature is available at [OODT Distributed Configuration 
Management](https://cwiki.apache.org/confluence/display/OODT/OODT+Distributed+Configuration+Management)

Corresponding GSoC project proposal is available at 
[https://cwiki.apache.org/confluence/display/OODT/Proposal](https://cwiki.apache.org/confluence/display/OODT/Proposal)

**Since this is the first glance of the distributed configuration 
management feature, there can be bugs and improvements required to be 
addressed. Please review this PR and provide me with your kind feedback so that 
I can improve on that**

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/IMS94/oodt master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/oodt/pull/43.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #43


commit ac7170bdf5c973374f141c6f3e031fa748e0af13
Author: Imesha Sudasingha 
Date:   2017-04-01T07:00:55Z

Defined initial API for ConfigurationManager

commit 4b88f71497263675bb313462075fd552696ba717
Author: Imesha Sudasingha 
Date:   2017-06-16T10:23:43Z

Implemented global configuration publishing mechanism.

commit fadf149cb77dd943af9e81c0f127ce74f203c478
Author: Imesha Sudasingha 
Date:   2017-06-17T15:39:58Z

Altered configuration publishing mechanism to use spring configuration

commit 5fcf2220125b64afba080c6c9c6c5592daae2dea
Author: Imesha Sudasingha 
Date:   2017-06-17T17:51:12Z

Implemented configuration fetching for DistributedConfigurationManager

commit 27a28daa3d3c448b462bd9a33eb6ac4fd859b75a
Author: Imesha Sudasingha 
Date:   2017-07-02T15:23:49Z

Added zip packaging for distributed configuration publisher

commit a973c531fee5bf37ebf4448334470c75943af1ec
Author: Imesha Sudasingha 
Date:   2017-07-03T10:25:22Z

Created test cases for distributed configuration manager and publisher

commit e697405223fe17c96739ae8f156639a0f5ef321c
Author: Imesha Sudasingha 
Date:   2017-07-09T16:48:40Z

Minor improvements for configuration manager tests

commit 99b086b9887cafc8b89de3ff65fa61d3f80cc56c
Author: Imesha Sudasingha 
Date:   2017-07-10T11:51:51Z

Improvements to file downloading from zookeeper and tests




> Rework OODT configuration to make use of  Zookeeper for distributed 
> configuration management
> 
>
> Key: OODT-945
> URL: https://issues.apache.org/jira/browse/OODT-945
> Project: OODT
>  Issue Type: Bug
>Reporter: Tom Barber
>Assignee: Tom Barber
>  Labels: gsoc2017, mentor
>
> OODT is made up of a number of different components, each of which have their 
> own set of configuration files and locations. This is complex to manage and 
> causes additional problems when the platform requires distribution across 
> servers or geographically.
> To combat this we are proposing that we migrate the OODT configuration to an 
> opional Zookeeper module that would allow users to commission a zookeeper 
> cluster and register the various OODT components against it, which will allow 
> the zookeeper cluster to maintain the state of the various OODT components 
> regardless of scale.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OODT-945) Rework OODT configuration to make use of Zookeeper for distributed configuration management

[Imesha Sudasingha (JIRA)]

[ 
https://issues.apache.org/jira/browse/OODT-945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15906547#comment-15906547
 ] 

Imesha Sudasingha commented on OODT-945:


Hi [~bugg_tb],

Thanks for the quick response. Since I have some experience in zookeeper and 
curator, I will first go through the OODT project and let you know if I need 
anymore clarifications.

Imesha

> Rework OODT configuration to make use of  Zookeeper for distributed 
> configuration management
> 
>
> Key: OODT-945
> URL: https://issues.apache.org/jira/browse/OODT-945
> Project: OODT
>  Issue Type: Bug
>Reporter: Tom Barber
>Assignee: Tom Barber
>  Labels: gsoc2017, mentor
>
> OODT is made up of a number of different components, each of which have their 
> own set of configuration files and locations. This is complex to manage and 
> causes additional problems when the platform requires distribution across 
> servers or geographically.
> To combat this we are proposing that we migrate the OODT configuration to an 
> opional Zookeeper module that would allow users to commission a zookeeper 
> cluster and register the various OODT components against it, which will allow 
> the zookeeper cluster to maintain the state of the various OODT components 
> regardless of scale.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (OODT-945) Rework OODT configuration to make use of Zookeeper for distributed configuration management

[Tom Barber (JIRA)]

[ 
https://issues.apache.org/jira/browse/OODT-945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15906524#comment-15906524
 ] 

Tom Barber commented on OODT-945:
-

Hi [~imesha]

Thanks for asking!

Yeah one of the core plans for OODT 2.0 is to add an option for developers to 
deploy OODT with ZK as the backing config store. This will allow us to scale 
the platform whilst keeping the config manageable and dynamic as components 
come and go in deployments.

In order of the questions asked:

Yes, pretty much, there are lots of OODT components so storing in 
namspace/componentName makes sense
Again, yes, currently the configs for most OODT components are based on Spring 
configs and property files, so it would be a question of reworking these. We'd 
likely start with the File Manager as the first component as its the most 
critical for scale and a good reference implementation
And finally, yes Apache Curator would be fine, simplifying the complexity is 
always good :)

Tom

> Rework OODT configuration to make use of  Zookeeper for distributed 
> configuration management
> 
>
> Key: OODT-945
> URL: https://issues.apache.org/jira/browse/OODT-945
> Project: OODT
>  Issue Type: Bug
>Reporter: Tom Barber
>Assignee: Tom Barber
>  Labels: gsoc2017, mentor
>
> OODT is made up of a number of different components, each of which have their 
> own set of configuration files and locations. This is complex to manage and 
> causes additional problems when the platform requires distribution across 
> servers or geographically.
> To combat this we are proposing that we migrate the OODT configuration to an 
> opional Zookeeper module that would allow users to commission a zookeeper 
> cluster and register the various OODT components against it, which will allow 
> the zookeeper cluster to maintain the state of the various OODT components 
> regardless of scale.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (OODT-945) Rework OODT configuration to make use of Zookeeper for distributed configuration management

[Imesha Sudasingha (JIRA)]

[ 
https://issues.apache.org/jira/browse/OODT-945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15906204#comment-15906204
 ] 

Imesha Sudasingha commented on OODT-945:


I am a final year student of University of Moratuwa, Sri Lanka. I want to know 
few things about this project.

* The task is to store configuration of each component under a Zookeeper 
namespace in a unique ZNode (say {{/{$namespace}/{$componentName}}})?
* A suitable architecture for this configuration management has to be 
introduced?
* Can we use Apache Curator as the client?

Thanks in advance!

> Rework OODT configuration to make use of  Zookeeper for distributed 
> configuration management
> 
>
> Key: OODT-945
> URL: https://issues.apache.org/jira/browse/OODT-945
> Project: OODT
>  Issue Type: Bug
>Reporter: Tom Barber
>Assignee: Tom Barber
>  Labels: gsoc2017, mentor
>
> OODT is made up of a number of different components, each of which have their 
> own set of configuration files and locations. This is complex to manage and 
> causes additional problems when the platform requires distribution across 
> servers or geographically.
> To combat this we are proposing that we migrate the OODT configuration to an 
> opional Zookeeper module that would allow users to commission a zookeeper 
> cluster and register the various OODT components against it, which will allow 
> the zookeeper cluster to maintain the state of the various OODT components 
> regardless of scale.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)