Re: Review Request 64751: Fix Broken Symlinks on Stack Distribution

2017-12-26 Thread Dmitro Lisnichenko 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64751/
---

(Updated Dec. 26, 2017, 7:10 p.m.)


Review request for Ambari, Jonathan Hurley and Nate Cole.


Changes
---

added Livy2


Bugs: AMBARI-22678
https://issues.apache.org/jira/browse/AMBARI-22678


Repository: ambari


Description
---

There are two scenarios to cover here:

# Ambari never conf-select'd a component (maybe because of a bug or because the 
component didn't support it)
# The conf pointers of a component are broken

In either event, when distributing a new stack, the code detects this problem 
(as it would on a first-time install) and tries to fix it:
{code}
/etc/component/conf (directory)
/usr/hdp/current/component -> /usr/hdp/v1/component
/usr/hdp/v1/component -> /etc/component/conf
{code}

The stack distribution thinks this is a first-time installed and tries to fix 
the symlinks. We end up with:
{code}
/etc/component/conf -> /usr/hdp/current/component
/usr/hdp/current/component -> /usr/hdp/v1/component
/usr/hdp/v1/component -> /etc/component/conf
/usr/hdp/v2/component -> /etc/component/v2/0
{code}

Because we're only conf-selecting v2, v1 never gets corrected since it's 
already installed. Thus, we have a circular symlink.

Most likely the proper fix will be:
- Iterate over the entire known conf-select structure
- Check to see the state /etc/component/conf - if it's bad, fix it to defaults

Chances are we can do this directly in 
{{conf_select.convert_conf_directories_to_symlinks}}:
{code}
stack_name = Script.get_stack_name()
for directory_struct in dirs:
if not os.path.exists(directory_struct['conf_dir']):
Logger.info("Skipping the conf-select tool on {0} since {1} does not 
exist.".format(
package, directory_struct['conf_dir']))

return
{code}


Diffs (updated)
-

  
ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/livy_service.py
 add5a7789e 
  
ambari-server/src/main/resources/common-services/SPARK2/2.0.0/package/scripts/livy2_service.py
 0180a315b6 
  ambari-server/src/main/resources/custom_actions/scripts/install_packages.py 
5cff342076 
  
ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_packages.json
 dc71b4da4f 


Diff: https://reviews.apache.org/r/64751/diff/3/

Changes: https://reviews.apache.org/r/64751/diff/2-3/


Testing
---

mvn clean test

Live cluster check


Thanks,

Dmitro Lisnichenko

Re: Review Request 64751: Fix Broken Symlinks on Stack Distribution

2017-12-21 Thread Nate Cole 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64751/#review194353
---


Ship it!




Ship It!

- Nate Cole


On Dec. 21, 2017, 8:25 a.m., Dmitro Lisnichenko wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64751/
> ---
> 
> (Updated Dec. 21, 2017, 8:25 a.m.)
> 
> 
> Review request for Ambari, Jonathan Hurley and Nate Cole.
> 
> 
> Bugs: AMBARI-22678
> https://issues.apache.org/jira/browse/AMBARI-22678
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> There are two scenarios to cover here:
> 
> # Ambari never conf-select'd a component (maybe because of a bug or because 
> the component didn't support it)
> # The conf pointers of a component are broken
> 
> In either event, when distributing a new stack, the code detects this problem 
> (as it would on a first-time install) and tries to fix it:
> {code}
> /etc/component/conf (directory)
> /usr/hdp/current/component -> /usr/hdp/v1/component
> /usr/hdp/v1/component -> /etc/component/conf
> {code}
> 
> The stack distribution thinks this is a first-time installed and tries to fix 
> the symlinks. We end up with:
> {code}
> /etc/component/conf -> /usr/hdp/current/component
> /usr/hdp/current/component -> /usr/hdp/v1/component
> /usr/hdp/v1/component -> /etc/component/conf
> /usr/hdp/v2/component -> /etc/component/v2/0
> {code}
> 
> Because we're only conf-selecting v2, v1 never gets corrected since it's 
> already installed. Thus, we have a circular symlink.
> 
> Most likely the proper fix will be:
> - Iterate over the entire known conf-select structure
> - Check to see the state /etc/component/conf - if it's bad, fix it to defaults
> 
> Chances are we can do this directly in 
> {{conf_select.convert_conf_directories_to_symlinks}}:
> {code}
> stack_name = Script.get_stack_name()
> for directory_struct in dirs:
> if not os.path.exists(directory_struct['conf_dir']):
> Logger.info("Skipping the conf-select tool on {0} since {1} does not 
> exist.".format(
> package, directory_struct['conf_dir']))
> 
> return
> {code}
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/livy_service.py
>  cce2148ab6 
>   
> ambari-server/src/main/resources/common-services/SPARK2/2.0.0/package/scripts/livy2_service.py
>  dfadd84613 
>   ambari-server/src/main/resources/custom_actions/scripts/install_packages.py 
> c8497cd2cb 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_packages.json
>  62a46b91bd 
> 
> 
> Diff: https://reviews.apache.org/r/64751/diff/2/
> 
> 
> Testing
> ---
> 
> mvn clean test
> 
> Live cluster check
> 
> 
> Thanks,
> 
> Dmitro Lisnichenko
> 
>

Re: Review Request 64751: Fix Broken Symlinks on Stack Distribution

2017-12-21 Thread Jonathan Hurley 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64751/#review194333
---


Ship it!




Ship It!

- Jonathan Hurley


On Dec. 21, 2017, 8:25 a.m., Dmitro Lisnichenko wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64751/
> ---
> 
> (Updated Dec. 21, 2017, 8:25 a.m.)
> 
> 
> Review request for Ambari, Jonathan Hurley and Nate Cole.
> 
> 
> Bugs: AMBARI-22678
> https://issues.apache.org/jira/browse/AMBARI-22678
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> There are two scenarios to cover here:
> 
> # Ambari never conf-select'd a component (maybe because of a bug or because 
> the component didn't support it)
> # The conf pointers of a component are broken
> 
> In either event, when distributing a new stack, the code detects this problem 
> (as it would on a first-time install) and tries to fix it:
> {code}
> /etc/component/conf (directory)
> /usr/hdp/current/component -> /usr/hdp/v1/component
> /usr/hdp/v1/component -> /etc/component/conf
> {code}
> 
> The stack distribution thinks this is a first-time installed and tries to fix 
> the symlinks. We end up with:
> {code}
> /etc/component/conf -> /usr/hdp/current/component
> /usr/hdp/current/component -> /usr/hdp/v1/component
> /usr/hdp/v1/component -> /etc/component/conf
> /usr/hdp/v2/component -> /etc/component/v2/0
> {code}
> 
> Because we're only conf-selecting v2, v1 never gets corrected since it's 
> already installed. Thus, we have a circular symlink.
> 
> Most likely the proper fix will be:
> - Iterate over the entire known conf-select structure
> - Check to see the state /etc/component/conf - if it's bad, fix it to defaults
> 
> Chances are we can do this directly in 
> {{conf_select.convert_conf_directories_to_symlinks}}:
> {code}
> stack_name = Script.get_stack_name()
> for directory_struct in dirs:
> if not os.path.exists(directory_struct['conf_dir']):
> Logger.info("Skipping the conf-select tool on {0} since {1} does not 
> exist.".format(
> package, directory_struct['conf_dir']))
> 
> return
> {code}
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/livy_service.py
>  cce2148ab6 
>   
> ambari-server/src/main/resources/common-services/SPARK2/2.0.0/package/scripts/livy2_service.py
>  dfadd84613 
>   ambari-server/src/main/resources/custom_actions/scripts/install_packages.py 
> c8497cd2cb 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_packages.json
>  62a46b91bd 
> 
> 
> Diff: https://reviews.apache.org/r/64751/diff/2/
> 
> 
> Testing
> ---
> 
> mvn clean test
> 
> Live cluster check
> 
> 
> Thanks,
> 
> Dmitro Lisnichenko
> 
>

Re: Review Request 64751: Fix Broken Symlinks on Stack Distribution

2017-12-21 Thread Dmitro Lisnichenko 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64751/
---

(Updated Dec. 21, 2017, 3:25 p.m.)


Review request for Ambari, Jonathan Hurley and Nate Cole.


Changes
---

Addressed comments


Bugs: AMBARI-22678
https://issues.apache.org/jira/browse/AMBARI-22678


Repository: ambari


Description
---

There are two scenarios to cover here:

# Ambari never conf-select'd a component (maybe because of a bug or because the 
component didn't support it)
# The conf pointers of a component are broken

In either event, when distributing a new stack, the code detects this problem 
(as it would on a first-time install) and tries to fix it:
{code}
/etc/component/conf (directory)
/usr/hdp/current/component -> /usr/hdp/v1/component
/usr/hdp/v1/component -> /etc/component/conf
{code}

The stack distribution thinks this is a first-time installed and tries to fix 
the symlinks. We end up with:
{code}
/etc/component/conf -> /usr/hdp/current/component
/usr/hdp/current/component -> /usr/hdp/v1/component
/usr/hdp/v1/component -> /etc/component/conf
/usr/hdp/v2/component -> /etc/component/v2/0
{code}

Because we're only conf-selecting v2, v1 never gets corrected since it's 
already installed. Thus, we have a circular symlink.

Most likely the proper fix will be:
- Iterate over the entire known conf-select structure
- Check to see the state /etc/component/conf - if it's bad, fix it to defaults

Chances are we can do this directly in 
{{conf_select.convert_conf_directories_to_symlinks}}:
{code}
stack_name = Script.get_stack_name()
for directory_struct in dirs:
if not os.path.exists(directory_struct['conf_dir']):
Logger.info("Skipping the conf-select tool on {0} since {1} does not 
exist.".format(
package, directory_struct['conf_dir']))

return
{code}


Diffs (updated)
-

  
ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/livy_service.py
 cce2148ab6 
  
ambari-server/src/main/resources/common-services/SPARK2/2.0.0/package/scripts/livy2_service.py
 dfadd84613 
  ambari-server/src/main/resources/custom_actions/scripts/install_packages.py 
c8497cd2cb 
  
ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_packages.json
 62a46b91bd 


Diff: https://reviews.apache.org/r/64751/diff/2/

Changes: https://reviews.apache.org/r/64751/diff/1-2/


Testing
---

mvn clean test

Live cluster check


Thanks,

Dmitro Lisnichenko

Re: Review Request 64751: Fix Broken Symlinks on Stack Distribution

2017-12-21 Thread Dmitro Lisnichenko 


> On Dec. 20, 2017, 5:03 p.m., Jonathan Hurley wrote:
> > ambari-server/src/main/resources/custom_actions/scripts/install_packages.py
> > Lines 199-201 (patched)
> > 
> >
> > I'd say just return earlier like this:
> > 
> > if not stack_version:
> >   Logger.warn("Unable to fix {0} since there is no known installed 
> > version for this component".format(package_name))

Stack is being checked for every package in a loop, so return here would break 
the execution flow


- Dmitro


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64751/#review194249
---


On Dec. 20, 2017, 4:33 p.m., Dmitro Lisnichenko wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64751/
> ---
> 
> (Updated Dec. 20, 2017, 4:33 p.m.)
> 
> 
> Review request for Ambari, Jonathan Hurley and Nate Cole.
> 
> 
> Bugs: AMBARI-22678
> https://issues.apache.org/jira/browse/AMBARI-22678
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> There are two scenarios to cover here:
> 
> # Ambari never conf-select'd a component (maybe because of a bug or because 
> the component didn't support it)
> # The conf pointers of a component are broken
> 
> In either event, when distributing a new stack, the code detects this problem 
> (as it would on a first-time install) and tries to fix it:
> {code}
> /etc/component/conf (directory)
> /usr/hdp/current/component -> /usr/hdp/v1/component
> /usr/hdp/v1/component -> /etc/component/conf
> {code}
> 
> The stack distribution thinks this is a first-time installed and tries to fix 
> the symlinks. We end up with:
> {code}
> /etc/component/conf -> /usr/hdp/current/component
> /usr/hdp/current/component -> /usr/hdp/v1/component
> /usr/hdp/v1/component -> /etc/component/conf
> /usr/hdp/v2/component -> /etc/component/v2/0
> {code}
> 
> Because we're only conf-selecting v2, v1 never gets corrected since it's 
> already installed. Thus, we have a circular symlink.
> 
> Most likely the proper fix will be:
> - Iterate over the entire known conf-select structure
> - Check to see the state /etc/component/conf - if it's bad, fix it to defaults
> 
> Chances are we can do this directly in 
> {{conf_select.convert_conf_directories_to_symlinks}}:
> {code}
> stack_name = Script.get_stack_name()
> for directory_struct in dirs:
> if not os.path.exists(directory_struct['conf_dir']):
> Logger.info("Skipping the conf-select tool on {0} since {1} does not 
> exist.".format(
> package, directory_struct['conf_dir']))
> 
> return
> {code}
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/livy_service.py
>  cce2148 
>   
> ambari-server/src/main/resources/common-services/SPARK2/2.0.0/package/scripts/livy2_service.py
>  dfadd84 
>   ambari-server/src/main/resources/custom_actions/scripts/install_packages.py 
> c8497cd 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_packages.json
>  62a46b9 
> 
> 
> Diff: https://reviews.apache.org/r/64751/diff/1/
> 
> 
> Testing
> ---
> 
> mvn clean test
> 
> Live cluster check
> 
> 
> Thanks,
> 
> Dmitro Lisnichenko
> 
>

Re: Review Request 64751: Fix Broken Symlinks on Stack Distribution

2017-12-20 Thread Jonathan Hurley 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64751/#review194249
---


Fix it, then Ship it!





ambari-server/src/main/resources/custom_actions/scripts/install_packages.py
Lines 179 (patched)


nit: Attempting to fix any configuration symlinks which are not in the 
correct state



ambari-server/src/main/resources/custom_actions/scripts/install_packages.py
Lines 199-201 (patched)


I'd say just return earlier like this:

if not stack_version:
  Logger.warn("Unable to fix {0} since there is no known installed version 
for this component".format(package_name))



ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_packages.json
Lines 1075-1080 (original), 1090-1096 (patched)


nifi is supported by hdp-select?


- Jonathan Hurley


On Dec. 20, 2017, 9:33 a.m., Dmitro Lisnichenko wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64751/
> ---
> 
> (Updated Dec. 20, 2017, 9:33 a.m.)
> 
> 
> Review request for Ambari, Jonathan Hurley and Nate Cole.
> 
> 
> Bugs: AMBARI-22678
> https://issues.apache.org/jira/browse/AMBARI-22678
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> There are two scenarios to cover here:
> 
> # Ambari never conf-select'd a component (maybe because of a bug or because 
> the component didn't support it)
> # The conf pointers of a component are broken
> 
> In either event, when distributing a new stack, the code detects this problem 
> (as it would on a first-time install) and tries to fix it:
> {code}
> /etc/component/conf (directory)
> /usr/hdp/current/component -> /usr/hdp/v1/component
> /usr/hdp/v1/component -> /etc/component/conf
> {code}
> 
> The stack distribution thinks this is a first-time installed and tries to fix 
> the symlinks. We end up with:
> {code}
> /etc/component/conf -> /usr/hdp/current/component
> /usr/hdp/current/component -> /usr/hdp/v1/component
> /usr/hdp/v1/component -> /etc/component/conf
> /usr/hdp/v2/component -> /etc/component/v2/0
> {code}
> 
> Because we're only conf-selecting v2, v1 never gets corrected since it's 
> already installed. Thus, we have a circular symlink.
> 
> Most likely the proper fix will be:
> - Iterate over the entire known conf-select structure
> - Check to see the state /etc/component/conf - if it's bad, fix it to defaults
> 
> Chances are we can do this directly in 
> {{conf_select.convert_conf_directories_to_symlinks}}:
> {code}
> stack_name = Script.get_stack_name()
> for directory_struct in dirs:
> if not os.path.exists(directory_struct['conf_dir']):
> Logger.info("Skipping the conf-select tool on {0} since {1} does not 
> exist.".format(
> package, directory_struct['conf_dir']))
> 
> return
> {code}
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/livy_service.py
>  cce2148 
>   
> ambari-server/src/main/resources/common-services/SPARK2/2.0.0/package/scripts/livy2_service.py
>  dfadd84 
>   ambari-server/src/main/resources/custom_actions/scripts/install_packages.py 
> c8497cd 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_packages.json
>  62a46b9 
> 
> 
> Diff: https://reviews.apache.org/r/64751/diff/1/
> 
> 
> Testing
> ---
> 
> mvn clean test
> 
> Live cluster check
> 
> 
> Thanks,
> 
> Dmitro Lisnichenko
> 
>

Review Request 64751: Fix Broken Symlinks on Stack Distribution

2017-12-20 Thread Dmitro Lisnichenko 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64751/
---

Review request for Ambari, Jonathan Hurley and Nate Cole.


Bugs: AMBARI-22678
https://issues.apache.org/jira/browse/AMBARI-22678


Repository: ambari


Description
---

There are two scenarios to cover here:

# Ambari never conf-select'd a component (maybe because of a bug or because the 
component didn't support it)
# The conf pointers of a component are broken

In either event, when distributing a new stack, the code detects this problem 
(as it would on a first-time install) and tries to fix it:
{code}
/etc/component/conf (directory)
/usr/hdp/current/component -> /usr/hdp/v1/component
/usr/hdp/v1/component -> /etc/component/conf
{code}

The stack distribution thinks this is a first-time installed and tries to fix 
the symlinks. We end up with:
{code}
/etc/component/conf -> /usr/hdp/current/component
/usr/hdp/current/component -> /usr/hdp/v1/component
/usr/hdp/v1/component -> /etc/component/conf
/usr/hdp/v2/component -> /etc/component/v2/0
{code}

Because we're only conf-selecting v2, v1 never gets corrected since it's 
already installed. Thus, we have a circular symlink.

Most likely the proper fix will be:
- Iterate over the entire known conf-select structure
- Check to see the state /etc/component/conf - if it's bad, fix it to defaults

Chances are we can do this directly in 
{{conf_select.convert_conf_directories_to_symlinks}}:
{code}
stack_name = Script.get_stack_name()
for directory_struct in dirs:
if not os.path.exists(directory_struct['conf_dir']):
Logger.info("Skipping the conf-select tool on {0} since {1} does not 
exist.".format(
package, directory_struct['conf_dir']))

return
{code}


Diffs
-

  
ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/livy_service.py
 cce2148 
  
ambari-server/src/main/resources/common-services/SPARK2/2.0.0/package/scripts/livy2_service.py
 dfadd84 
  ambari-server/src/main/resources/custom_actions/scripts/install_packages.py 
c8497cd 
  
ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_packages.json
 62a46b9 


Diff: https://reviews.apache.org/r/64751/diff/1/


Testing
---

mvn clean test

Live cluster check


Thanks,

Dmitro Lisnichenko