subject:"Review Request 55016\: AMBARI\-19293. Component install and after\-install should not run in parallel"

Re: Review Request 55016: AMBARI-19293. Component install and after-install should not run in parallel

2017-01-12 Thread Alejandro Fernandez


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/55016/#review161441
---


Ship it!




Ship It!

- Alejandro Fernandez


On Jan. 12, 2017, 9:14 p.m., Attila Doroszlai wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/55016/
> ---
> 
> (Updated Jan. 12, 2017, 9:14 p.m.)
> 
> 
> Review request for Ambari, Andrew Onischuk, Jonathan Hurley, Laszlo Puskas, 
> Sandor Magyari, and Sebastian Toader.
> 
> 
> Bugs: AMBARI-19293
> https://issues.apache.org/jira/browse/AMBARI-19293
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> * Perform `configure` called from `install` with shared lock
> * Use exclusive lock in after-install hook
> 
> 
> Diffs
> -
> 
>   
> ambari-common/src/main/python/resource_management/libraries/script/script.py 
> ccb09c75a2a6e15c9f639c5eea4a9a60e8706942 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/after-INSTALL/scripts/params.py
>  ed34217757540278a8730d58797cc47536e58874 
>   
> ambari-server/src/main/resources/stacks/HDP/3.0/hooks/after-INSTALL/scripts/params.py
>  566f5b3f2608ef0a3393ed163a216c1b3696d380 
> 
> Diff: https://reviews.apache.org/r/55016/diff/
> 
> 
> Testing
> ---
> 
> Manual test:
> 
> * install cluster via blueprint with parallel execution disabled (this is the 
> default)
> * install cluster via blueprint with parallel execution enabled (needs 
> packages pre-installed and users pre-created)
> 
> Unit tests:
> 
> ```
> $ mvn -am -pl ambari-agent,ambari-server -DskipSurefireTests clean test
> ...
> [INFO] --- exec-maven-plugin:1.2.1:exec (python-test) @ ambari-server ---
> ...
> Total run:1158
> Total errors:0
> Total failures:0
> OK
> ...
> [INFO] --- exec-maven-plugin:1.2.1:exec (python-test) @ ambari-agent ---
> ...
> Ran 452 tests in 22.538s
> 
> OK
> ...
> [INFO] BUILD SUCCESS
> ```
> 
> (note: ambari-server test-compile may run into error, see AMBARI-19458)
> 
> 
> Thanks,
> 
> Attila Doroszlai
> 
>

Re: Review Request 55016: AMBARI-19293. Component install and after-install should not run in parallel

2017-01-12 Thread Attila Doroszlai


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/55016/
---

(Updated Jan. 12, 2017, 10:14 p.m.)


Review request for Ambari, Andrew Onischuk, Jonathan Hurley, Laszlo Puskas, 
Sandor Magyari, and Sebastian Toader.


Changes
---

Include HDP 3.0 hook


Bugs: AMBARI-19293
https://issues.apache.org/jira/browse/AMBARI-19293


Repository: ambari


Description
---

* Perform `configure` called from `install` with shared lock
* Use exclusive lock in after-install hook


Diffs (updated)
-

  ambari-common/src/main/python/resource_management/libraries/script/script.py 
ccb09c75a2a6e15c9f639c5eea4a9a60e8706942 
  
ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/after-INSTALL/scripts/params.py
 ed34217757540278a8730d58797cc47536e58874 
  
ambari-server/src/main/resources/stacks/HDP/3.0/hooks/after-INSTALL/scripts/params.py
 566f5b3f2608ef0a3393ed163a216c1b3696d380 

Diff: https://reviews.apache.org/r/55016/diff/


Testing
---

Manual test:

* install cluster via blueprint with parallel execution disabled (this is the 
default)
* install cluster via blueprint with parallel execution enabled (needs packages 
pre-installed and users pre-created)

Unit tests:

```
$ mvn -am -pl ambari-agent,ambari-server -DskipSurefireTests clean test
...
[INFO] --- exec-maven-plugin:1.2.1:exec (python-test) @ ambari-server ---
...
Total run:1158
Total errors:0
Total failures:0
OK
...
[INFO] --- exec-maven-plugin:1.2.1:exec (python-test) @ ambari-agent ---
...
Ran 452 tests in 22.538s

OK
...
[INFO] BUILD SUCCESS
```

(note: ambari-server test-compile may run into error, see AMBARI-19458)


Thanks,

Attila Doroszlai

Re: Review Request 55016: AMBARI-19293. Component install and after-install should not run in parallel

2017-01-12 Thread Attila Doroszlai



> On Jan. 12, 2017, 6:52 p.m., Jonathan Hurley wrote:
> > `configure` intercepting looks fine ...
> > 
> > One question is that you removed the conf-select lock. Are you saying we 
> > won't have a problem with conf-select anymore b/c of the `configure` lock? 
> > I'm not fully convinced of that. If there are two components running 
> > concurrently, couldn't they both be doing the symlink restructuring on the 
> > same directories? For example, NameNode and DataNode both mess with 
> > /etc/hadoop/conf ...

I don't think I removed the conf-select lock.  Relative to the current code, I 
just made sure it uses the same file as `configure` (when called from .  
Relative to the first patch, I moved the definition of the lock filename from 
`conf_select.py` to `script.py`.

This reminds me: trunk needs a new patch including the HDP 3.0 hook (since it 
doesn't inherit from 2.0.6).


- Attila


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/55016/#review161396
---


On Jan. 11, 2017, 11:03 p.m., Attila Doroszlai wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/55016/
> ---
> 
> (Updated Jan. 11, 2017, 11:03 p.m.)
> 
> 
> Review request for Ambari, Andrew Onischuk, Jonathan Hurley, Laszlo Puskas, 
> Sandor Magyari, and Sebastian Toader.
> 
> 
> Bugs: AMBARI-19293
> https://issues.apache.org/jira/browse/AMBARI-19293
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> * Perform `configure` called from `install` with shared lock
> * Use exclusive lock in after-install hook
> 
> 
> Diffs
> -
> 
>   
> ambari-common/src/main/python/resource_management/libraries/script/script.py 
> ccb09c75a2a6e15c9f639c5eea4a9a60e8706942 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/after-INSTALL/scripts/params.py
>  ed34217757540278a8730d58797cc47536e58874 
> 
> Diff: https://reviews.apache.org/r/55016/diff/
> 
> 
> Testing
> ---
> 
> Manual test:
> 
> * install cluster via blueprint with parallel execution disabled (this is the 
> default)
> * install cluster via blueprint with parallel execution enabled (needs 
> packages pre-installed and users pre-created)
> 
> Unit tests:
> 
> ```
> $ mvn -am -pl ambari-agent,ambari-server -DskipSurefireTests clean test
> ...
> [INFO] --- exec-maven-plugin:1.2.1:exec (python-test) @ ambari-server ---
> ...
> Total run:1158
> Total errors:0
> Total failures:0
> OK
> ...
> [INFO] --- exec-maven-plugin:1.2.1:exec (python-test) @ ambari-agent ---
> ...
> Ran 452 tests in 22.538s
> 
> OK
> ...
> [INFO] BUILD SUCCESS
> ```
> 
> (note: ambari-server test-compile may run into error, see AMBARI-19458)
> 
> 
> Thanks,
> 
> Attila Doroszlai
> 
>

Re: Review Request 55016: AMBARI-19293. Component install and after-install should not run in parallel

2017-01-12 Thread Jonathan Hurley


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/55016/#review161396
---



`configure` intercepting looks fine ...

One question is that you removed the conf-select lock. Are you saying we won't 
have a problem with conf-select anymore b/c of the `configure` lock? I'm not 
fully convinced of that. If there are two components running concurrently, 
couldn't they both be doing the symlink restructuring on the same directories? 
For example, NameNode and DataNode both mess with /etc/hadoop/conf ...

- Jonathan Hurley


On Jan. 11, 2017, 5:03 p.m., Attila Doroszlai wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/55016/
> ---
> 
> (Updated Jan. 11, 2017, 5:03 p.m.)
> 
> 
> Review request for Ambari, Andrew Onischuk, Jonathan Hurley, Laszlo Puskas, 
> Sandor Magyari, and Sebastian Toader.
> 
> 
> Bugs: AMBARI-19293
> https://issues.apache.org/jira/browse/AMBARI-19293
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> * Perform `configure` called from `install` with shared lock
> * Use exclusive lock in after-install hook
> 
> 
> Diffs
> -
> 
>   
> ambari-common/src/main/python/resource_management/libraries/script/script.py 
> ccb09c75a2a6e15c9f639c5eea4a9a60e8706942 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/after-INSTALL/scripts/params.py
>  ed34217757540278a8730d58797cc47536e58874 
> 
> Diff: https://reviews.apache.org/r/55016/diff/
> 
> 
> Testing
> ---
> 
> Manual test:
> 
> * install cluster via blueprint with parallel execution disabled (this is the 
> default)
> * install cluster via blueprint with parallel execution enabled (needs 
> packages pre-installed and users pre-created)
> 
> Unit tests:
> 
> ```
> $ mvn -am -pl ambari-agent,ambari-server -DskipSurefireTests clean test
> ...
> [INFO] --- exec-maven-plugin:1.2.1:exec (python-test) @ ambari-server ---
> ...
> Total run:1158
> Total errors:0
> Total failures:0
> OK
> ...
> [INFO] --- exec-maven-plugin:1.2.1:exec (python-test) @ ambari-agent ---
> ...
> Ran 452 tests in 22.538s
> 
> OK
> ...
> [INFO] BUILD SUCCESS
> ```
> 
> (note: ambari-server test-compile may run into error, see AMBARI-19458)
> 
> 
> Thanks,
> 
> Attila Doroszlai
> 
>

Re: Review Request 55016: AMBARI-19293. Component install and after-install should not run in parallel

2017-01-12 Thread Jonathan Hurley


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/55016/#review161395
---


Ship it!




Ship It!

- Jonathan Hurley


On Jan. 11, 2017, 5:03 p.m., Attila Doroszlai wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/55016/
> ---
> 
> (Updated Jan. 11, 2017, 5:03 p.m.)
> 
> 
> Review request for Ambari, Andrew Onischuk, Jonathan Hurley, Laszlo Puskas, 
> Sandor Magyari, and Sebastian Toader.
> 
> 
> Bugs: AMBARI-19293
> https://issues.apache.org/jira/browse/AMBARI-19293
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> * Perform `configure` called from `install` with shared lock
> * Use exclusive lock in after-install hook
> 
> 
> Diffs
> -
> 
>   
> ambari-common/src/main/python/resource_management/libraries/script/script.py 
> ccb09c75a2a6e15c9f639c5eea4a9a60e8706942 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/after-INSTALL/scripts/params.py
>  ed34217757540278a8730d58797cc47536e58874 
> 
> Diff: https://reviews.apache.org/r/55016/diff/
> 
> 
> Testing
> ---
> 
> Manual test:
> 
> * install cluster via blueprint with parallel execution disabled (this is the 
> default)
> * install cluster via blueprint with parallel execution enabled (needs 
> packages pre-installed and users pre-created)
> 
> Unit tests:
> 
> ```
> $ mvn -am -pl ambari-agent,ambari-server -DskipSurefireTests clean test
> ...
> [INFO] --- exec-maven-plugin:1.2.1:exec (python-test) @ ambari-server ---
> ...
> Total run:1158
> Total errors:0
> Total failures:0
> OK
> ...
> [INFO] --- exec-maven-plugin:1.2.1:exec (python-test) @ ambari-agent ---
> ...
> Ran 452 tests in 22.538s
> 
> OK
> ...
> [INFO] BUILD SUCCESS
> ```
> 
> (note: ambari-server test-compile may run into error, see AMBARI-19458)
> 
> 
> Thanks,
> 
> Attila Doroszlai
> 
>

Re: Review Request 55016: AMBARI-19293. Component install and after-install should not run in parallel

2017-01-11 Thread Sebastian Toader


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/55016/#review161356
---


Ship it!




Ship It!

- Sebastian Toader


On Jan. 11, 2017, 11:03 p.m., Attila Doroszlai wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/55016/
> ---
> 
> (Updated Jan. 11, 2017, 11:03 p.m.)
> 
> 
> Review request for Ambari, Andrew Onischuk, Jonathan Hurley, Laszlo Puskas, 
> Sandor Magyari, and Sebastian Toader.
> 
> 
> Bugs: AMBARI-19293
> https://issues.apache.org/jira/browse/AMBARI-19293
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> * Perform `configure` called from `install` with shared lock
> * Use exclusive lock in after-install hook
> 
> 
> Diffs
> -
> 
>   
> ambari-common/src/main/python/resource_management/libraries/script/script.py 
> ccb09c75a2a6e15c9f639c5eea4a9a60e8706942 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/after-INSTALL/scripts/params.py
>  ed34217757540278a8730d58797cc47536e58874 
> 
> Diff: https://reviews.apache.org/r/55016/diff/
> 
> 
> Testing
> ---
> 
> Manual test:
> 
> * install cluster via blueprint with parallel execution disabled (this is the 
> default)
> * install cluster via blueprint with parallel execution enabled (needs 
> packages pre-installed and users pre-created)
> 
> Unit tests:
> 
> ```
> $ mvn -am -pl ambari-agent,ambari-server -DskipSurefireTests clean test
> ...
> [INFO] --- exec-maven-plugin:1.2.1:exec (python-test) @ ambari-server ---
> ...
> Total run:1158
> Total errors:0
> Total failures:0
> OK
> ...
> [INFO] --- exec-maven-plugin:1.2.1:exec (python-test) @ ambari-agent ---
> ...
> Ran 452 tests in 22.538s
> 
> OK
> ...
> [INFO] BUILD SUCCESS
> ```
> 
> (note: ambari-server test-compile may run into error, see AMBARI-19458)
> 
> 
> Thanks,
> 
> Attila Doroszlai
> 
>

Re: Review Request 55016: AMBARI-19293. Component install and after-install should not run in parallel

2017-01-11 Thread Alejandro Fernandez


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/55016/#review161333
---


Ship it!




Ship It!

- Alejandro Fernandez


On Jan. 11, 2017, 10:03 p.m., Attila Doroszlai wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/55016/
> ---
> 
> (Updated Jan. 11, 2017, 10:03 p.m.)
> 
> 
> Review request for Ambari, Andrew Onischuk, Jonathan Hurley, Laszlo Puskas, 
> Sandor Magyari, and Sebastian Toader.
> 
> 
> Bugs: AMBARI-19293
> https://issues.apache.org/jira/browse/AMBARI-19293
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> * Perform `configure` called from `install` with shared lock
> * Use exclusive lock in after-install hook
> 
> 
> Diffs
> -
> 
>   
> ambari-common/src/main/python/resource_management/libraries/script/script.py 
> ccb09c75a2a6e15c9f639c5eea4a9a60e8706942 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/after-INSTALL/scripts/params.py
>  ed34217757540278a8730d58797cc47536e58874 
> 
> Diff: https://reviews.apache.org/r/55016/diff/
> 
> 
> Testing
> ---
> 
> Manual test:
> 
> * install cluster via blueprint with parallel execution disabled (this is the 
> default)
> * install cluster via blueprint with parallel execution enabled (needs 
> packages pre-installed and users pre-created)
> 
> Unit tests:
> 
> ```
> $ mvn -am -pl ambari-agent,ambari-server -DskipSurefireTests clean test
> ...
> [INFO] --- exec-maven-plugin:1.2.1:exec (python-test) @ ambari-server ---
> ...
> Total run:1158
> Total errors:0
> Total failures:0
> OK
> ...
> [INFO] --- exec-maven-plugin:1.2.1:exec (python-test) @ ambari-agent ---
> ...
> Ran 452 tests in 22.538s
> 
> OK
> ...
> [INFO] BUILD SUCCESS
> ```
> 
> (note: ambari-server test-compile may run into error, see AMBARI-19458)
> 
> 
> Thanks,
> 
> Attila Doroszlai
> 
>

Re: Review Request 55016: AMBARI-19293. Component install and after-install should not run in parallel

2017-01-11 Thread Attila Doroszlai


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/55016/
---

(Updated Jan. 11, 2017, 11:03 p.m.)


Review request for Ambari, Andrew Onischuk, Jonathan Hurley, Laszlo Puskas, 
Sandor Magyari, and Sebastian Toader.


Changes
---

added comment on local import


Bugs: AMBARI-19293
https://issues.apache.org/jira/browse/AMBARI-19293


Repository: ambari


Description
---

* Perform `configure` called from `install` with shared lock
* Use exclusive lock in after-install hook


Diffs (updated)
-

  ambari-common/src/main/python/resource_management/libraries/script/script.py 
ccb09c75a2a6e15c9f639c5eea4a9a60e8706942 
  
ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/after-INSTALL/scripts/params.py
 ed34217757540278a8730d58797cc47536e58874 

Diff: https://reviews.apache.org/r/55016/diff/


Testing
---

Manual test:

* install cluster via blueprint with parallel execution disabled (this is the 
default)
* install cluster via blueprint with parallel execution enabled (needs packages 
pre-installed and users pre-created)

Unit tests:

```
$ mvn -am -pl ambari-agent,ambari-server -DskipSurefireTests clean test
...
[INFO] --- exec-maven-plugin:1.2.1:exec (python-test) @ ambari-server ---
...
Total run:1158
Total errors:0
Total failures:0
OK
...
[INFO] --- exec-maven-plugin:1.2.1:exec (python-test) @ ambari-agent ---
...
Ran 452 tests in 22.538s

OK
...
[INFO] BUILD SUCCESS
```

(note: ambari-server test-compile may run into error, see AMBARI-19458)


Thanks,

Attila Doroszlai

Re: Review Request 55016: AMBARI-19293. Component install and after-install should not run in parallel

2017-01-11 Thread Alejandro Fernandez


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/55016/#review161301
---


Fix it, then Ship it!





ambari-common/src/main/python/resource_management/libraries/script/script.py 
(line 112)


Can you add a quick comment for that?

Things like that are not obvious, especially if someone else has to look at 
the code or refactor.


- Alejandro Fernandez


On Jan. 11, 2017, 10:40 a.m., Attila Doroszlai wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/55016/
> ---
> 
> (Updated Jan. 11, 2017, 10:40 a.m.)
> 
> 
> Review request for Ambari, Andrew Onischuk, Jonathan Hurley, Laszlo Puskas, 
> Sandor Magyari, and Sebastian Toader.
> 
> 
> Bugs: AMBARI-19293
> https://issues.apache.org/jira/browse/AMBARI-19293
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> * Perform `configure` called from `install` with shared lock
> * Use exclusive lock in after-install hook
> 
> 
> Diffs
> -
> 
>   
> ambari-common/src/main/python/resource_management/libraries/script/script.py 
> ccb09c75a2a6e15c9f639c5eea4a9a60e8706942 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/after-INSTALL/scripts/params.py
>  ed34217757540278a8730d58797cc47536e58874 
> 
> Diff: https://reviews.apache.org/r/55016/diff/
> 
> 
> Testing
> ---
> 
> Manual test:
> 
> * install cluster via blueprint with parallel execution disabled (this is the 
> default)
> * install cluster via blueprint with parallel execution enabled (needs 
> packages pre-installed and users pre-created)
> 
> Unit tests:
> 
> ```
> $ mvn -am -pl ambari-agent,ambari-server -DskipSurefireTests clean test
> ...
> [INFO] --- exec-maven-plugin:1.2.1:exec (python-test) @ ambari-server ---
> ...
> Total run:1158
> Total errors:0
> Total failures:0
> OK
> ...
> [INFO] --- exec-maven-plugin:1.2.1:exec (python-test) @ ambari-agent ---
> ...
> Ran 452 tests in 22.538s
> 
> OK
> ...
> [INFO] BUILD SUCCESS
> ```
> 
> (note: ambari-server test-compile may run into error, see AMBARI-19458)
> 
> 
> Thanks,
> 
> Attila Doroszlai
> 
>

Re: Review Request 55016: AMBARI-19293. Component install and after-install should not run in parallel

2017-01-11 Thread Attila Doroszlai



> On Jan. 11, 2017, 10:32 p.m., Alejandro Fernandez wrote:
> > ambari-common/src/main/python/resource_management/libraries/script/script.py,
> >  line 112
> > 
> >
> > I prefer to move imports like this to the top of the file

I tried that first, but it doesn't work, I'd guess due to `default.py` also 
importing `Script`.


- Attila


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/55016/#review161299
---


On Jan. 11, 2017, 11:40 a.m., Attila Doroszlai wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/55016/
> ---
> 
> (Updated Jan. 11, 2017, 11:40 a.m.)
> 
> 
> Review request for Ambari, Andrew Onischuk, Jonathan Hurley, Laszlo Puskas, 
> Sandor Magyari, and Sebastian Toader.
> 
> 
> Bugs: AMBARI-19293
> https://issues.apache.org/jira/browse/AMBARI-19293
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> * Perform `configure` called from `install` with shared lock
> * Use exclusive lock in after-install hook
> 
> 
> Diffs
> -
> 
>   
> ambari-common/src/main/python/resource_management/libraries/script/script.py 
> ccb09c75a2a6e15c9f639c5eea4a9a60e8706942 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/after-INSTALL/scripts/params.py
>  ed34217757540278a8730d58797cc47536e58874 
> 
> Diff: https://reviews.apache.org/r/55016/diff/
> 
> 
> Testing
> ---
> 
> Manual test:
> 
> * install cluster via blueprint with parallel execution disabled (this is the 
> default)
> * install cluster via blueprint with parallel execution enabled (needs 
> packages pre-installed and users pre-created)
> 
> Unit tests:
> 
> ```
> $ mvn -am -pl ambari-agent,ambari-server -DskipSurefireTests clean test
> ...
> [INFO] --- exec-maven-plugin:1.2.1:exec (python-test) @ ambari-server ---
> ...
> Total run:1158
> Total errors:0
> Total failures:0
> OK
> ...
> [INFO] --- exec-maven-plugin:1.2.1:exec (python-test) @ ambari-agent ---
> ...
> Ran 452 tests in 22.538s
> 
> OK
> ...
> [INFO] BUILD SUCCESS
> ```
> 
> (note: ambari-server test-compile may run into error, see AMBARI-19458)
> 
> 
> Thanks,
> 
> Attila Doroszlai
> 
>

Re: Review Request 55016: AMBARI-19293. Component install and after-install should not run in parallel

2017-01-11 Thread Alejandro Fernandez


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/55016/#review161299
---


Fix it, then Ship it!





ambari-common/src/main/python/resource_management/libraries/script/script.py 
(line 112)


I prefer to move imports like this to the top of the file


- Alejandro Fernandez


On Jan. 11, 2017, 10:40 a.m., Attila Doroszlai wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/55016/
> ---
> 
> (Updated Jan. 11, 2017, 10:40 a.m.)
> 
> 
> Review request for Ambari, Andrew Onischuk, Jonathan Hurley, Laszlo Puskas, 
> Sandor Magyari, and Sebastian Toader.
> 
> 
> Bugs: AMBARI-19293
> https://issues.apache.org/jira/browse/AMBARI-19293
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> * Perform `configure` called from `install` with shared lock
> * Use exclusive lock in after-install hook
> 
> 
> Diffs
> -
> 
>   
> ambari-common/src/main/python/resource_management/libraries/script/script.py 
> ccb09c75a2a6e15c9f639c5eea4a9a60e8706942 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/after-INSTALL/scripts/params.py
>  ed34217757540278a8730d58797cc47536e58874 
> 
> Diff: https://reviews.apache.org/r/55016/diff/
> 
> 
> Testing
> ---
> 
> Manual test:
> 
> * install cluster via blueprint with parallel execution disabled (this is the 
> default)
> * install cluster via blueprint with parallel execution enabled (needs 
> packages pre-installed and users pre-created)
> 
> Unit tests:
> 
> ```
> $ mvn -am -pl ambari-agent,ambari-server -DskipSurefireTests clean test
> ...
> [INFO] --- exec-maven-plugin:1.2.1:exec (python-test) @ ambari-server ---
> ...
> Total run:1158
> Total errors:0
> Total failures:0
> OK
> ...
> [INFO] --- exec-maven-plugin:1.2.1:exec (python-test) @ ambari-agent ---
> ...
> Ran 452 tests in 22.538s
> 
> OK
> ...
> [INFO] BUILD SUCCESS
> ```
> 
> (note: ambari-server test-compile may run into error, see AMBARI-19458)
> 
> 
> Thanks,
> 
> Attila Doroszlai
> 
>

Re: Review Request 55016: AMBARI-19293. Component install and after-install should not run in parallel

2017-01-11 Thread Attila Doroszlai


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/55016/
---

(Updated Jan. 11, 2017, 11:40 a.m.)


Review request for Ambari, Andrew Onischuk, Jonathan Hurley, Laszlo Puskas, 
Sandor Magyari, and Sebastian Toader.


Changes
---

* Use exclusive lock for `configure`, too, to avoid concurrent config file 
creation
* Implement with metaclass instead of having to add to all subclasses


Bugs: AMBARI-19293
https://issues.apache.org/jira/browse/AMBARI-19293


Repository: ambari


Description
---

* Perform `configure` called from `install` with shared lock
* Use exclusive lock in after-install hook


Diffs (updated)
-

  ambari-common/src/main/python/resource_management/libraries/script/script.py 
ccb09c75a2a6e15c9f639c5eea4a9a60e8706942 
  
ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/after-INSTALL/scripts/params.py
 ed34217757540278a8730d58797cc47536e58874 

Diff: https://reviews.apache.org/r/55016/diff/


Testing (updated)
---

Manual test:

* install cluster via blueprint with parallel execution disabled (this is the 
default)
* install cluster via blueprint with parallel execution enabled (needs packages 
pre-installed and users pre-created)

Unit tests:

```
$ mvn -am -pl ambari-agent,ambari-server -DskipSurefireTests clean test
...
[INFO] --- exec-maven-plugin:1.2.1:exec (python-test) @ ambari-server ---
...
Total run:1158
Total errors:0
Total failures:0
OK
...
[INFO] --- exec-maven-plugin:1.2.1:exec (python-test) @ ambari-agent ---
...
Ran 452 tests in 22.538s

OK
...
[INFO] BUILD SUCCESS
```

(note: ambari-server test-compile may run into error, see AMBARI-19458)


Thanks,

Attila Doroszlai

Re: Review Request 55016: AMBARI-19293. Component install and after-install should not run in parallel

2017-01-09 Thread Attila Doroszlai



> On Jan. 9, 2017, 6:42 p.m., Jonathan Hurley wrote:
> > ambari-server/src/main/resources/common-services/ACCUMULO/1.6.1.2.2.0/package/scripts/accumulo_client.py,
> >  lines 38-39
> > 
> >
> > This pattern doesn't seem very maintainable. What defines whether or 
> > not a component needs to get a shared lock before calling configure? Should 
> > this instead be moved to the super's `configure` ?

There are two problems with moving it to `super.configure`:

1. `super.configure` (in `script.py`) is currently this:

```
787   def configure(self, env, upgrade_type=None, config_dir=None):
788 """
789 To be overridden by subclasses
790 :param upgrade_type: only valid during RU/EU, otherwise will be None
791 :param config_dir: for some clients during RU, the location to save 
configs to, otherwise None
792 """
793 self.fail_with_error('configure method isn\'t implemented')
```

So not only are scripts not required to call `super.configure`, they are not 
allowed to do so.  If we put the lock in `script`, we'd need to change even 
more subclasses (224 per `grep`) to add the missing call.

2. We only want to lock `configure` called from `install`, not from `start` or 
anywhere else, since that's what `after-install` is related to.


- Attila


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/55016/#review160909
---


On Jan. 9, 2017, 11:51 a.m., Attila Doroszlai wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/55016/
> ---
> 
> (Updated Jan. 9, 2017, 11:51 a.m.)
> 
> 
> Review request for Ambari, Andrew Onischuk, Jonathan Hurley, Laszlo Puskas, 
> Sandor Magyari, and Sebastian Toader.
> 
> 
> Bugs: AMBARI-19293
> https://issues.apache.org/jira/browse/AMBARI-19293
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> * Perform `configure` called from `install` with shared lock
> * Use exclusive lock in after-install hook
> 
> 
> Diffs
> -
> 
>   
> ambari-agent/src/test/python/resource_management/TestFcntlBasedProcessLock.py 
> be6ff80 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/conf_select.py
>  ce00f0c 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/fcntl_based_process_lock.py
>  67fadac 
>   
> ambari-server/src/main/resources/common-services/ACCUMULO/1.6.1.2.2.0/package/scripts/accumulo_client.py
>  67ca525 
>   
> ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/atlas_client.py
>  26742ae 
>   
> ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/package/scripts/falcon_client.py
>  6d1dbc5 
>   
> ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/hbase_client.py
>  c8128ab 
>   
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs_client.py
>  4dabdbc 
>   
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/namenode.py
>  bd05a95 
>   
> ambari-server/src/main/resources/common-services/HDFS/3.0.0.3.0/package/scripts/hdfs_client.py
>  4dabdbc 
>   
> ambari-server/src/main/resources/common-services/HDFS/3.0.0.3.0/package/scripts/namenode.py
>  08578bc 
>   
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_client.py
>  83d82df 
>   
> ambari-server/src/main/resources/common-services/MAHOUT/1.0.0.2.3/package/scripts/mahout_client.py
>  3486add 
>   
> ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie_client.py
>  f98ecfd 
>   
> ambari-server/src/main/resources/common-services/PIG/0.12.0.2.0/package/scripts/pig_client.py
>  6fdcef8 
>   
> ambari-server/src/main/resources/common-services/RANGER/0.4.0/package/scripts/ranger_admin.py
>  b849d58 
>   
> ambari-server/src/main/resources/common-services/RANGER/0.4.0/package/scripts/ranger_tagsync.py
>  008fb99 
>   
> ambari-server/src/main/resources/common-services/SLIDER/0.60.0.2.2/package/scripts/slider_client.py
>  cf9f830 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/spark_client.py
>  31bf4c6 
>   
> ambari-server/src/main/resources/common-services/SPARK2/2.0.0/package/scripts/spark_client.py
>  2c19b88 
>   
> ambari-server/src/main/resources/common-services/SQOOP/1.4.4.2.0/package/scripts/sqoop_client.py
>  d420fab 
>   
> ambari-server/src/main/resources/common-services/STORM/0.9.1/package/scripts/drpc_server.py
>  178c043 
>   
> ambari-server/src/main/resources/common-services/STORM/0.9.1/package/scripts/nimbus.py
>  a974103 
>   
>

Re: Review Request 55016: AMBARI-19293. Component install and after-install should not run in parallel

2017-01-09 Thread Nate Cole


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/55016/#review160911
---



-1 on this patch.  If you are protecting against directory creation as part of 
conf-select invocation, then that is where the locking should occur.  
Definitely NOT per-component.  We'll never be able to enforce developers to 
make this a part of normal lifecycle for a component.

- Nate Cole


On Jan. 9, 2017, 5:51 a.m., Attila Doroszlai wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/55016/
> ---
> 
> (Updated Jan. 9, 2017, 5:51 a.m.)
> 
> 
> Review request for Ambari, Andrew Onischuk, Jonathan Hurley, Laszlo Puskas, 
> Sandor Magyari, and Sebastian Toader.
> 
> 
> Bugs: AMBARI-19293
> https://issues.apache.org/jira/browse/AMBARI-19293
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> * Perform `configure` called from `install` with shared lock
> * Use exclusive lock in after-install hook
> 
> 
> Diffs
> -
> 
>   
> ambari-agent/src/test/python/resource_management/TestFcntlBasedProcessLock.py 
> be6ff80 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/conf_select.py
>  ce00f0c 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/fcntl_based_process_lock.py
>  67fadac 
>   
> ambari-server/src/main/resources/common-services/ACCUMULO/1.6.1.2.2.0/package/scripts/accumulo_client.py
>  67ca525 
>   
> ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/atlas_client.py
>  26742ae 
>   
> ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/package/scripts/falcon_client.py
>  6d1dbc5 
>   
> ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/hbase_client.py
>  c8128ab 
>   
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs_client.py
>  4dabdbc 
>   
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/namenode.py
>  bd05a95 
>   
> ambari-server/src/main/resources/common-services/HDFS/3.0.0.3.0/package/scripts/hdfs_client.py
>  4dabdbc 
>   
> ambari-server/src/main/resources/common-services/HDFS/3.0.0.3.0/package/scripts/namenode.py
>  08578bc 
>   
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_client.py
>  83d82df 
>   
> ambari-server/src/main/resources/common-services/MAHOUT/1.0.0.2.3/package/scripts/mahout_client.py
>  3486add 
>   
> ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie_client.py
>  f98ecfd 
>   
> ambari-server/src/main/resources/common-services/PIG/0.12.0.2.0/package/scripts/pig_client.py
>  6fdcef8 
>   
> ambari-server/src/main/resources/common-services/RANGER/0.4.0/package/scripts/ranger_admin.py
>  b849d58 
>   
> ambari-server/src/main/resources/common-services/RANGER/0.4.0/package/scripts/ranger_tagsync.py
>  008fb99 
>   
> ambari-server/src/main/resources/common-services/SLIDER/0.60.0.2.2/package/scripts/slider_client.py
>  cf9f830 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/spark_client.py
>  31bf4c6 
>   
> ambari-server/src/main/resources/common-services/SPARK2/2.0.0/package/scripts/spark_client.py
>  2c19b88 
>   
> ambari-server/src/main/resources/common-services/SQOOP/1.4.4.2.0/package/scripts/sqoop_client.py
>  d420fab 
>   
> ambari-server/src/main/resources/common-services/STORM/0.9.1/package/scripts/drpc_server.py
>  178c043 
>   
> ambari-server/src/main/resources/common-services/STORM/0.9.1/package/scripts/nimbus.py
>  a974103 
>   
> ambari-server/src/main/resources/common-services/STORM/0.9.1/package/scripts/nimbus_prod.py
>  39bda4d 
>   
> ambari-server/src/main/resources/common-services/STORM/0.9.1/package/scripts/pacemaker.py
>  a56c0cd 
>   
> ambari-server/src/main/resources/common-services/STORM/0.9.1/package/scripts/rest_api.py
>  f9b3b80 
>   
> ambari-server/src/main/resources/common-services/STORM/0.9.1/package/scripts/supervisor.py
>  ec3f533 
>   
> ambari-server/src/main/resources/common-services/STORM/0.9.1/package/scripts/supervisor_prod.py
>  d6c3545 
>   
> ambari-server/src/main/resources/common-services/STORM/0.9.1/package/scripts/ui_server.py
>  63acecf 
>   
> ambari-server/src/main/resources/common-services/TEZ/0.4.0.2.1/package/scripts/tez_client.py
>  8018f0f 
>   
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/mapreduce2_client.py
>  424157b 
>   
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/yarn_client.py
>  beea8b9 
>   
> ambari-server/src/main/resources/common-services/YARN/3.0.0.3.0/package/scripts/mapreduce2_client.py
>  424157b 
>   
>

Re: Review Request 55016: AMBARI-19293. Component install and after-install should not run in parallel

2017-01-09 Thread Jonathan Hurley


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/55016/#review160909
---


Fix it, then Ship it!




Changing every packages `install` function to get a lock around `configure` 
doesn't seem very maintainable as new services and components are added. Can 
you just place this at a higher level?


ambari-server/src/main/resources/common-services/ACCUMULO/1.6.1.2.2.0/package/scripts/accumulo_client.py
 (lines 38 - 39)


This pattern doesn't seem very maintainable. What defines whether or not a 
component needs to get a shared lock before calling configure? Should this 
instead be moved to the super's `configure` ?


- Jonathan Hurley


On Jan. 9, 2017, 5:51 a.m., Attila Doroszlai wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/55016/
> ---
> 
> (Updated Jan. 9, 2017, 5:51 a.m.)
> 
> 
> Review request for Ambari, Andrew Onischuk, Jonathan Hurley, Laszlo Puskas, 
> Sandor Magyari, and Sebastian Toader.
> 
> 
> Bugs: AMBARI-19293
> https://issues.apache.org/jira/browse/AMBARI-19293
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> * Perform `configure` called from `install` with shared lock
> * Use exclusive lock in after-install hook
> 
> 
> Diffs
> -
> 
>   
> ambari-agent/src/test/python/resource_management/TestFcntlBasedProcessLock.py 
> be6ff80 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/conf_select.py
>  ce00f0c 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/fcntl_based_process_lock.py
>  67fadac 
>   
> ambari-server/src/main/resources/common-services/ACCUMULO/1.6.1.2.2.0/package/scripts/accumulo_client.py
>  67ca525 
>   
> ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/atlas_client.py
>  26742ae 
>   
> ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/package/scripts/falcon_client.py
>  6d1dbc5 
>   
> ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/hbase_client.py
>  c8128ab 
>   
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs_client.py
>  4dabdbc 
>   
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/namenode.py
>  bd05a95 
>   
> ambari-server/src/main/resources/common-services/HDFS/3.0.0.3.0/package/scripts/hdfs_client.py
>  4dabdbc 
>   
> ambari-server/src/main/resources/common-services/HDFS/3.0.0.3.0/package/scripts/namenode.py
>  08578bc 
>   
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_client.py
>  83d82df 
>   
> ambari-server/src/main/resources/common-services/MAHOUT/1.0.0.2.3/package/scripts/mahout_client.py
>  3486add 
>   
> ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie_client.py
>  f98ecfd 
>   
> ambari-server/src/main/resources/common-services/PIG/0.12.0.2.0/package/scripts/pig_client.py
>  6fdcef8 
>   
> ambari-server/src/main/resources/common-services/RANGER/0.4.0/package/scripts/ranger_admin.py
>  b849d58 
>   
> ambari-server/src/main/resources/common-services/RANGER/0.4.0/package/scripts/ranger_tagsync.py
>  008fb99 
>   
> ambari-server/src/main/resources/common-services/SLIDER/0.60.0.2.2/package/scripts/slider_client.py
>  cf9f830 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/spark_client.py
>  31bf4c6 
>   
> ambari-server/src/main/resources/common-services/SPARK2/2.0.0/package/scripts/spark_client.py
>  2c19b88 
>   
> ambari-server/src/main/resources/common-services/SQOOP/1.4.4.2.0/package/scripts/sqoop_client.py
>  d420fab 
>   
> ambari-server/src/main/resources/common-services/STORM/0.9.1/package/scripts/drpc_server.py
>  178c043 
>   
> ambari-server/src/main/resources/common-services/STORM/0.9.1/package/scripts/nimbus.py
>  a974103 
>   
> ambari-server/src/main/resources/common-services/STORM/0.9.1/package/scripts/nimbus_prod.py
>  39bda4d 
>   
> ambari-server/src/main/resources/common-services/STORM/0.9.1/package/scripts/pacemaker.py
>  a56c0cd 
>   
> ambari-server/src/main/resources/common-services/STORM/0.9.1/package/scripts/rest_api.py
>  f9b3b80 
>   
> ambari-server/src/main/resources/common-services/STORM/0.9.1/package/scripts/supervisor.py
>  ec3f533 
>   
> ambari-server/src/main/resources/common-services/STORM/0.9.1/package/scripts/supervisor_prod.py
>  d6c3545 
>   
> ambari-server/src/main/resources/common-services/STORM/0.9.1/package/scripts/ui_server.py
>  63acecf 
>   
> ambari-server/src/main/resources/common-services/TEZ/0.4.0.2.1/package/scripts/tez_client.py
>  8018f0f 
>   
>

Re: Review Request 55016: AMBARI-19293. Component install and after-install should not run in parallel

2017-01-09 Thread Sebastian Toader


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/55016/#review160887
---


Ship it!




Ship It!

- Sebastian Toader


On Jan. 9, 2017, 11:51 a.m., Attila Doroszlai wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/55016/
> ---
> 
> (Updated Jan. 9, 2017, 11:51 a.m.)
> 
> 
> Review request for Ambari, Andrew Onischuk, Jonathan Hurley, Laszlo Puskas, 
> Sandor Magyari, and Sebastian Toader.
> 
> 
> Bugs: AMBARI-19293
> https://issues.apache.org/jira/browse/AMBARI-19293
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> * Perform `configure` called from `install` with shared lock
> * Use exclusive lock in after-install hook
> 
> 
> Diffs
> -
> 
>   
> ambari-agent/src/test/python/resource_management/TestFcntlBasedProcessLock.py 
> be6ff80 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/conf_select.py
>  ce00f0c 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/fcntl_based_process_lock.py
>  67fadac 
>   
> ambari-server/src/main/resources/common-services/ACCUMULO/1.6.1.2.2.0/package/scripts/accumulo_client.py
>  67ca525 
>   
> ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/atlas_client.py
>  26742ae 
>   
> ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/package/scripts/falcon_client.py
>  6d1dbc5 
>   
> ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/hbase_client.py
>  c8128ab 
>   
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs_client.py
>  4dabdbc 
>   
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/namenode.py
>  bd05a95 
>   
> ambari-server/src/main/resources/common-services/HDFS/3.0.0.3.0/package/scripts/hdfs_client.py
>  4dabdbc 
>   
> ambari-server/src/main/resources/common-services/HDFS/3.0.0.3.0/package/scripts/namenode.py
>  08578bc 
>   
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_client.py
>  83d82df 
>   
> ambari-server/src/main/resources/common-services/MAHOUT/1.0.0.2.3/package/scripts/mahout_client.py
>  3486add 
>   
> ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie_client.py
>  f98ecfd 
>   
> ambari-server/src/main/resources/common-services/PIG/0.12.0.2.0/package/scripts/pig_client.py
>  6fdcef8 
>   
> ambari-server/src/main/resources/common-services/RANGER/0.4.0/package/scripts/ranger_admin.py
>  b849d58 
>   
> ambari-server/src/main/resources/common-services/RANGER/0.4.0/package/scripts/ranger_tagsync.py
>  008fb99 
>   
> ambari-server/src/main/resources/common-services/SLIDER/0.60.0.2.2/package/scripts/slider_client.py
>  cf9f830 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/spark_client.py
>  31bf4c6 
>   
> ambari-server/src/main/resources/common-services/SPARK2/2.0.0/package/scripts/spark_client.py
>  2c19b88 
>   
> ambari-server/src/main/resources/common-services/SQOOP/1.4.4.2.0/package/scripts/sqoop_client.py
>  d420fab 
>   
> ambari-server/src/main/resources/common-services/STORM/0.9.1/package/scripts/drpc_server.py
>  178c043 
>   
> ambari-server/src/main/resources/common-services/STORM/0.9.1/package/scripts/nimbus.py
>  a974103 
>   
> ambari-server/src/main/resources/common-services/STORM/0.9.1/package/scripts/nimbus_prod.py
>  39bda4d 
>   
> ambari-server/src/main/resources/common-services/STORM/0.9.1/package/scripts/pacemaker.py
>  a56c0cd 
>   
> ambari-server/src/main/resources/common-services/STORM/0.9.1/package/scripts/rest_api.py
>  f9b3b80 
>   
> ambari-server/src/main/resources/common-services/STORM/0.9.1/package/scripts/supervisor.py
>  ec3f533 
>   
> ambari-server/src/main/resources/common-services/STORM/0.9.1/package/scripts/supervisor_prod.py
>  d6c3545 
>   
> ambari-server/src/main/resources/common-services/STORM/0.9.1/package/scripts/ui_server.py
>  63acecf 
>   
> ambari-server/src/main/resources/common-services/TEZ/0.4.0.2.1/package/scripts/tez_client.py
>  8018f0f 
>   
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/mapreduce2_client.py
>  424157b 
>   
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/yarn_client.py
>  beea8b9 
>   
> ambari-server/src/main/resources/common-services/YARN/3.0.0.3.0/package/scripts/mapreduce2_client.py
>  424157b 
>   
> ambari-server/src/main/resources/common-services/YARN/3.0.0.3.0/package/scripts/yarn_client.py
>  beea8b9 
>   
> ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5/package/scripts/zookeeper_client.py
>  3137f1a 
>   
>

Re: Review Request 55016: AMBARI-19293. Component install and after-install should not run in parallel

2017-01-08 Thread Attila Doroszlai


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/55016/
---

(Updated Jan. 8, 2017, 5:59 p.m.)


Review request for Ambari, Andrew Onischuk, Balázs Bence Sári, Jonathan Hurley, 
Laszlo Puskas, and Sebastian Toader.


Changes
---

Dropped package-specific locks, now using read-write lock


Bugs: AMBARI-19293
https://issues.apache.org/jira/browse/AMBARI-19293


Repository: ambari


Description (updated)
---

* Perform `configure` called from `install` with shared lock
* Use exclusive lock in after-install hook


Diffs (updated)
-

  ambari-agent/src/test/python/resource_management/TestFcntlBasedProcessLock.py 
be6ff80 
  
ambari-common/src/main/python/resource_management/libraries/functions/conf_select.py
 ce00f0c 
  
ambari-common/src/main/python/resource_management/libraries/functions/fcntl_based_process_lock.py
 67fadac 
  
ambari-server/src/main/resources/common-services/ACCUMULO/1.6.1.2.2.0/package/scripts/accumulo_client.py
 67ca525 
  
ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/atlas_client.py
 26742ae 
  
ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/package/scripts/falcon_client.py
 6d1dbc5 
  
ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/hbase_client.py
 c8128ab 
  
ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs_client.py
 4dabdbc 
  
ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/namenode.py
 bd05a95 
  
ambari-server/src/main/resources/common-services/HDFS/3.0.0.3.0/package/scripts/hdfs_client.py
 4dabdbc 
  
ambari-server/src/main/resources/common-services/HDFS/3.0.0.3.0/package/scripts/namenode.py
 08578bc 
  
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_client.py
 83d82df 
  
ambari-server/src/main/resources/common-services/MAHOUT/1.0.0.2.3/package/scripts/mahout_client.py
 3486add 
  
ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie_client.py
 f98ecfd 
  
ambari-server/src/main/resources/common-services/PIG/0.12.0.2.0/package/scripts/pig_client.py
 6fdcef8 
  
ambari-server/src/main/resources/common-services/RANGER/0.4.0/package/scripts/ranger_admin.py
 b849d58 
  
ambari-server/src/main/resources/common-services/RANGER/0.4.0/package/scripts/ranger_tagsync.py
 008fb99 
  
ambari-server/src/main/resources/common-services/SLIDER/0.60.0.2.2/package/scripts/slider_client.py
 cf9f830 
  
ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/spark_client.py
 31bf4c6 
  
ambari-server/src/main/resources/common-services/SPARK2/2.0.0/package/scripts/spark_client.py
 2c19b88 
  
ambari-server/src/main/resources/common-services/SQOOP/1.4.4.2.0/package/scripts/sqoop_client.py
 d420fab 
  
ambari-server/src/main/resources/common-services/STORM/0.9.1/package/scripts/drpc_server.py
 178c043 
  
ambari-server/src/main/resources/common-services/STORM/0.9.1/package/scripts/nimbus.py
 a974103 
  
ambari-server/src/main/resources/common-services/STORM/0.9.1/package/scripts/nimbus_prod.py
 39bda4d 
  
ambari-server/src/main/resources/common-services/STORM/0.9.1/package/scripts/pacemaker.py
 a56c0cd 
  
ambari-server/src/main/resources/common-services/STORM/0.9.1/package/scripts/rest_api.py
 f9b3b80 
  
ambari-server/src/main/resources/common-services/STORM/0.9.1/package/scripts/supervisor.py
 ec3f533 
  
ambari-server/src/main/resources/common-services/STORM/0.9.1/package/scripts/supervisor_prod.py
 d6c3545 
  
ambari-server/src/main/resources/common-services/STORM/0.9.1/package/scripts/ui_server.py
 63acecf 
  
ambari-server/src/main/resources/common-services/TEZ/0.4.0.2.1/package/scripts/tez_client.py
 8018f0f 
  
ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/mapreduce2_client.py
 424157b 
  
ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/yarn_client.py
 beea8b9 
  
ambari-server/src/main/resources/common-services/YARN/3.0.0.3.0/package/scripts/mapreduce2_client.py
 424157b 
  
ambari-server/src/main/resources/common-services/YARN/3.0.0.3.0/package/scripts/yarn_client.py
 beea8b9 
  
ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5/package/scripts/zookeeper_client.py
 3137f1a 
  
ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5/package/scripts/zookeeper_server.py
 3f2502a 
  
ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/after-INSTALL/scripts/params.py
 ed34217 
  
ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/after-INSTALL/scripts/shared_initialization.py
 e9f2283 
  ambari-server/src/test/python/stacks/2.2/common/test_conf_select.py d445d74 

Diff: https://reviews.apache.org/r/55016/diff/


Testing (updated)
---

Manual test:

* install cluster via

Re: Review Request 55016: AMBARI-19293. Component install and after-install should not run in parallel

2017-01-03 Thread Jonathan Hurley



> On Dec. 28, 2016, 9:36 a.m., Jonathan Hurley wrote:
> > Although I do not see anything wrong with this implementation, I'm curious 
> > why we simply can't use a single lock for this. Why do we need specific 
> > locks per component? 
> > 
> > Shouldn't it be enough to have a single lock which is used during the 
> > installation phase as well as the conf-select symlinking phase? The goal 
> > here is to just ensure they don't run in parallel.
> 
> Sebastian Toader wrote:
> In order to improve cluster creation time we need to run in parallel as 
> much as possible and synchronize only those bits we know can't run in 
> parallel. During installation of two components we we should run in parallel 
> the bits that are independent.
> 
> Jonathan Hurley wrote:
> I think that's my point - none of the install commands should ever run at 
> the same time as the conf-select work. conf-select should take less than a 
> second to run and re-link. I don't think it's going to break the bank here. I 
> think it's much safer to ensure that they are 100% mutually exclusive.
> 
> Attila Doroszlai wrote:
> This change introduces lock usage for configure() only for TEZ.  With 
> this setup, there is no noticeable difference between single lock and 
> multiple locks.  If we were to lock more/all configure() calls, the 
> difference would be more significant.
> 
> Jonathan Hurley wrote:
> Would it be that much more significant? You're locking on running yum 
> installs in parallel with conf-select post-install work, right? The 
> conf-select work is extremely fast, especially compared to the time it takes 
> to actually do an install. Therefore, doesn't it make sense to use a less 
> complicated implementation with a single lock?
> 
> Attila Doroszlai wrote:
> No, yum installs are already done by that time.  Only configure() (which 
> also copies and links configs) and conf-select run with mutex.

Hmmm, looking at this patch again, you're only using `get_config_lock` in two 
places:
- After Tez's package install before calling `configure`
- On AFTER-INSTALL while iterating every package when linking configs

This still poses a problem since you could have a parallel install of another 
component while AFTER-INSTALL iterates packages and tries to symlink that 
component. That's why I'm saying that the fine grain locks here aren't really 
solving the problem.


- Jonathan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/55016/#review160228
---


On Jan. 2, 2017, 10:47 a.m., Attila Doroszlai wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/55016/
> ---
> 
> (Updated Jan. 2, 2017, 10:47 a.m.)
> 
> 
> Review request for Ambari, Andrew Onischuk, Balázs Bence Sári, Jonathan 
> Hurley, Laszlo Puskas, and Sebastian Toader.
> 
> 
> Bugs: AMBARI-19293
> https://issues.apache.org/jira/browse/AMBARI-19293
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> * Perform TEZ `configure` step with lock (will add other services later).
> * Use service-specific locks in after-install hook to decrease contention
> 
> 
> Diffs
> -
> 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/conf_select.py
>  ce00f0c6f69f255690f1ab9528bea6cee184a73e 
>   
> ambari-server/src/main/resources/common-services/TEZ/0.4.0.2.1/package/scripts/tez_client.py
>  8018f0f960322aac196caf3915a0a944e7765d8e 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/after-INSTALL/scripts/params.py
>  ed34217757540278a8730d58797cc47536e58874 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/after-INSTALL/scripts/shared_initialization.py
>  e9f22834b5479330a6f00ca69991cc690780b3c3 
>   ambari-server/src/test/python/stacks/2.2/common/test_conf_select.py 
> d445d741ba4225a33e2a763eed6dfcf8394dceb6 
> 
> Diff: https://reviews.apache.org/r/55016/diff/
> 
> 
> Testing
> ---
> 
> Manual test:
> 
> * install cluster via blueprint with parallel execution disabled (this is the 
> default)
> * install cluster via blueprint with parallel execution enabled (needs 
> packages pre-installed and users pre-created)
> 
> Unit tests:
> 
> ```
> $ mvn -am -pl ambari-agent,ambari-server -DskipSurefireTests clean test
> ...
> [INFO] --- exec-maven-plugin:1.2.1:exec (python-test) @ ambari-server ---
> ...
> Total run:1160
> Total errors:0
> Total failures:0
> OK
> ...
> [INFO] --- exec-maven-plugin:1.2.1:exec (python-test) @ ambari-agent ---
> ...
> Ran 451 tests in 15.344s
> 
> OK
> ...
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Ambari Main ... SUCCESS [6.650s]
> [INFO] Apache Ambari Project POM . SUCCESS [0.027s]
> [INFO]

Re: Review Request 55016: AMBARI-19293. Component install and after-install should not run in parallel

2017-01-03 Thread Attila Doroszlai



> On Dec. 28, 2016, 3:36 p.m., Jonathan Hurley wrote:
> > Although I do not see anything wrong with this implementation, I'm curious 
> > why we simply can't use a single lock for this. Why do we need specific 
> > locks per component? 
> > 
> > Shouldn't it be enough to have a single lock which is used during the 
> > installation phase as well as the conf-select symlinking phase? The goal 
> > here is to just ensure they don't run in parallel.
> 
> Sebastian Toader wrote:
> In order to improve cluster creation time we need to run in parallel as 
> much as possible and synchronize only those bits we know can't run in 
> parallel. During installation of two components we we should run in parallel 
> the bits that are independent.
> 
> Jonathan Hurley wrote:
> I think that's my point - none of the install commands should ever run at 
> the same time as the conf-select work. conf-select should take less than a 
> second to run and re-link. I don't think it's going to break the bank here. I 
> think it's much safer to ensure that they are 100% mutually exclusive.
> 
> Attila Doroszlai wrote:
> This change introduces lock usage for configure() only for TEZ.  With 
> this setup, there is no noticeable difference between single lock and 
> multiple locks.  If we were to lock more/all configure() calls, the 
> difference would be more significant.
> 
> Jonathan Hurley wrote:
> Would it be that much more significant? You're locking on running yum 
> installs in parallel with conf-select post-install work, right? The 
> conf-select work is extremely fast, especially compared to the time it takes 
> to actually do an install. Therefore, doesn't it make sense to use a less 
> complicated implementation with a single lock?

No, yum installs are already done by that time.  Only configure() (which also 
copies and links configs) and conf-select run with mutex.


- Attila


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/55016/#review160228
---


On Jan. 2, 2017, 4:47 p.m., Attila Doroszlai wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/55016/
> ---
> 
> (Updated Jan. 2, 2017, 4:47 p.m.)
> 
> 
> Review request for Ambari, Andrew Onischuk, Balázs Bence Sári, Jonathan 
> Hurley, Laszlo Puskas, and Sebastian Toader.
> 
> 
> Bugs: AMBARI-19293
> https://issues.apache.org/jira/browse/AMBARI-19293
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> * Perform TEZ `configure` step with lock (will add other services later).
> * Use service-specific locks in after-install hook to decrease contention
> 
> 
> Diffs
> -
> 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/conf_select.py
>  ce00f0c6f69f255690f1ab9528bea6cee184a73e 
>   
> ambari-server/src/main/resources/common-services/TEZ/0.4.0.2.1/package/scripts/tez_client.py
>  8018f0f960322aac196caf3915a0a944e7765d8e 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/after-INSTALL/scripts/params.py
>  ed34217757540278a8730d58797cc47536e58874 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/after-INSTALL/scripts/shared_initialization.py
>  e9f22834b5479330a6f00ca69991cc690780b3c3 
>   ambari-server/src/test/python/stacks/2.2/common/test_conf_select.py 
> d445d741ba4225a33e2a763eed6dfcf8394dceb6 
> 
> Diff: https://reviews.apache.org/r/55016/diff/
> 
> 
> Testing
> ---
> 
> Manual test:
> 
> * install cluster via blueprint with parallel execution disabled (this is the 
> default)
> * install cluster via blueprint with parallel execution enabled (needs 
> packages pre-installed and users pre-created)
> 
> Unit tests:
> 
> ```
> $ mvn -am -pl ambari-agent,ambari-server -DskipSurefireTests clean test
> ...
> [INFO] --- exec-maven-plugin:1.2.1:exec (python-test) @ ambari-server ---
> ...
> Total run:1160
> Total errors:0
> Total failures:0
> OK
> ...
> [INFO] --- exec-maven-plugin:1.2.1:exec (python-test) @ ambari-agent ---
> ...
> Ran 451 tests in 15.344s
> 
> OK
> ...
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Ambari Main ... SUCCESS [6.650s]
> [INFO] Apache Ambari Project POM . SUCCESS [0.027s]
> [INFO] Ambari Views .. SUCCESS [2.355s]
> [INFO] utility ... SUCCESS [1.182s]
> [INFO] ambari-metrics  SUCCESS [0.315s]
> [INFO] Ambari Metrics Common . SUCCESS [0.730s]
> [INFO] Ambari Server . SUCCESS [1:38.241s]
> [INFO] Ambari Agent .. SUCCESS [17.247s]
> [INFO] 
>

Re: Review Request 55016: AMBARI-19293. Component install and after-install should not run in parallel

2017-01-03 Thread Jonathan Hurley



> On Dec. 28, 2016, 9:36 a.m., Jonathan Hurley wrote:
> > Although I do not see anything wrong with this implementation, I'm curious 
> > why we simply can't use a single lock for this. Why do we need specific 
> > locks per component? 
> > 
> > Shouldn't it be enough to have a single lock which is used during the 
> > installation phase as well as the conf-select symlinking phase? The goal 
> > here is to just ensure they don't run in parallel.
> 
> Sebastian Toader wrote:
> In order to improve cluster creation time we need to run in parallel as 
> much as possible and synchronize only those bits we know can't run in 
> parallel. During installation of two components we we should run in parallel 
> the bits that are independent.
> 
> Jonathan Hurley wrote:
> I think that's my point - none of the install commands should ever run at 
> the same time as the conf-select work. conf-select should take less than a 
> second to run and re-link. I don't think it's going to break the bank here. I 
> think it's much safer to ensure that they are 100% mutually exclusive.
> 
> Attila Doroszlai wrote:
> This change introduces lock usage for configure() only for TEZ.  With 
> this setup, there is no noticeable difference between single lock and 
> multiple locks.  If we were to lock more/all configure() calls, the 
> difference would be more significant.

Would it be that much more significant? You're locking on running yum installs 
in parallel with conf-select post-install work, right? The conf-select work is 
extremely fast, especially compared to the time it takes to actually do an 
install. Therefore, doesn't it make sense to use a less complicated 
implementation with a single lock?


- Jonathan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/55016/#review160228
---


On Jan. 2, 2017, 10:47 a.m., Attila Doroszlai wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/55016/
> ---
> 
> (Updated Jan. 2, 2017, 10:47 a.m.)
> 
> 
> Review request for Ambari, Andrew Onischuk, Balázs Bence Sári, Jonathan 
> Hurley, Laszlo Puskas, and Sebastian Toader.
> 
> 
> Bugs: AMBARI-19293
> https://issues.apache.org/jira/browse/AMBARI-19293
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> * Perform TEZ `configure` step with lock (will add other services later).
> * Use service-specific locks in after-install hook to decrease contention
> 
> 
> Diffs
> -
> 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/conf_select.py
>  ce00f0c6f69f255690f1ab9528bea6cee184a73e 
>   
> ambari-server/src/main/resources/common-services/TEZ/0.4.0.2.1/package/scripts/tez_client.py
>  8018f0f960322aac196caf3915a0a944e7765d8e 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/after-INSTALL/scripts/params.py
>  ed34217757540278a8730d58797cc47536e58874 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/after-INSTALL/scripts/shared_initialization.py
>  e9f22834b5479330a6f00ca69991cc690780b3c3 
>   ambari-server/src/test/python/stacks/2.2/common/test_conf_select.py 
> d445d741ba4225a33e2a763eed6dfcf8394dceb6 
> 
> Diff: https://reviews.apache.org/r/55016/diff/
> 
> 
> Testing
> ---
> 
> Manual test:
> 
> * install cluster via blueprint with parallel execution disabled (this is the 
> default)
> * install cluster via blueprint with parallel execution enabled (needs 
> packages pre-installed and users pre-created)
> 
> Unit tests:
> 
> ```
> $ mvn -am -pl ambari-agent,ambari-server -DskipSurefireTests clean test
> ...
> [INFO] --- exec-maven-plugin:1.2.1:exec (python-test) @ ambari-server ---
> ...
> Total run:1160
> Total errors:0
> Total failures:0
> OK
> ...
> [INFO] --- exec-maven-plugin:1.2.1:exec (python-test) @ ambari-agent ---
> ...
> Ran 451 tests in 15.344s
> 
> OK
> ...
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Ambari Main ... SUCCESS [6.650s]
> [INFO] Apache Ambari Project POM . SUCCESS [0.027s]
> [INFO] Ambari Views .. SUCCESS [2.355s]
> [INFO] utility ... SUCCESS [1.182s]
> [INFO] ambari-metrics  SUCCESS [0.315s]
> [INFO] Ambari Metrics Common . SUCCESS [0.730s]
> [INFO] Ambari Server . SUCCESS [1:38.241s]
> [INFO] Ambari Agent .. SUCCESS [17.247s]
> [INFO] 
> 
> [INFO] BUILD SUCCESS
> ```
> 
> 
> Thanks,
> 
> Attila Doroszlai
> 
>

Re: Review Request 55016: AMBARI-19293. Component install and after-install should not run in parallel

2017-01-03 Thread Attila Doroszlai



> On Dec. 28, 2016, 3:36 p.m., Jonathan Hurley wrote:
> > Although I do not see anything wrong with this implementation, I'm curious 
> > why we simply can't use a single lock for this. Why do we need specific 
> > locks per component? 
> > 
> > Shouldn't it be enough to have a single lock which is used during the 
> > installation phase as well as the conf-select symlinking phase? The goal 
> > here is to just ensure they don't run in parallel.
> 
> Sebastian Toader wrote:
> In order to improve cluster creation time we need to run in parallel as 
> much as possible and synchronize only those bits we know can't run in 
> parallel. During installation of two components we we should run in parallel 
> the bits that are independent.
> 
> Jonathan Hurley wrote:
> I think that's my point - none of the install commands should ever run at 
> the same time as the conf-select work. conf-select should take less than a 
> second to run and re-link. I don't think it's going to break the bank here. I 
> think it's much safer to ensure that they are 100% mutually exclusive.

This change introduces lock usage for configure() only for TEZ.  With this 
setup, there is no noticeable difference between single lock and multiple 
locks.  If we were to lock more/all configure() calls, the difference would be 
more significant.


- Attila


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/55016/#review160228
---


On Jan. 2, 2017, 4:47 p.m., Attila Doroszlai wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/55016/
> ---
> 
> (Updated Jan. 2, 2017, 4:47 p.m.)
> 
> 
> Review request for Ambari, Andrew Onischuk, Balázs Bence Sári, Jonathan 
> Hurley, Laszlo Puskas, and Sebastian Toader.
> 
> 
> Bugs: AMBARI-19293
> https://issues.apache.org/jira/browse/AMBARI-19293
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> * Perform TEZ `configure` step with lock (will add other services later).
> * Use service-specific locks in after-install hook to decrease contention
> 
> 
> Diffs
> -
> 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/conf_select.py
>  ce00f0c6f69f255690f1ab9528bea6cee184a73e 
>   
> ambari-server/src/main/resources/common-services/TEZ/0.4.0.2.1/package/scripts/tez_client.py
>  8018f0f960322aac196caf3915a0a944e7765d8e 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/after-INSTALL/scripts/params.py
>  ed34217757540278a8730d58797cc47536e58874 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/after-INSTALL/scripts/shared_initialization.py
>  e9f22834b5479330a6f00ca69991cc690780b3c3 
>   ambari-server/src/test/python/stacks/2.2/common/test_conf_select.py 
> d445d741ba4225a33e2a763eed6dfcf8394dceb6 
> 
> Diff: https://reviews.apache.org/r/55016/diff/
> 
> 
> Testing
> ---
> 
> Manual test:
> 
> * install cluster via blueprint with parallel execution disabled (this is the 
> default)
> * install cluster via blueprint with parallel execution enabled (needs 
> packages pre-installed and users pre-created)
> 
> Unit tests:
> 
> ```
> $ mvn -am -pl ambari-agent,ambari-server -DskipSurefireTests clean test
> ...
> [INFO] --- exec-maven-plugin:1.2.1:exec (python-test) @ ambari-server ---
> ...
> Total run:1160
> Total errors:0
> Total failures:0
> OK
> ...
> [INFO] --- exec-maven-plugin:1.2.1:exec (python-test) @ ambari-agent ---
> ...
> Ran 451 tests in 15.344s
> 
> OK
> ...
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Ambari Main ... SUCCESS [6.650s]
> [INFO] Apache Ambari Project POM . SUCCESS [0.027s]
> [INFO] Ambari Views .. SUCCESS [2.355s]
> [INFO] utility ... SUCCESS [1.182s]
> [INFO] ambari-metrics  SUCCESS [0.315s]
> [INFO] Ambari Metrics Common . SUCCESS [0.730s]
> [INFO] Ambari Server . SUCCESS [1:38.241s]
> [INFO] Ambari Agent .. SUCCESS [17.247s]
> [INFO] 
> 
> [INFO] BUILD SUCCESS
> ```
> 
> 
> Thanks,
> 
> Attila Doroszlai
> 
>

Re: Review Request 55016: AMBARI-19293. Component install and after-install should not run in parallel

2017-01-02 Thread Attila Doroszlai


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/55016/
---

(Updated Jan. 2, 2017, 4:47 p.m.)


Review request for Ambari, Andrew Onischuk, Balázs Bence Sári, Jonathan Hurley, 
Laszlo Puskas, and Sebastian Toader.


Bugs: AMBARI-19293
https://issues.apache.org/jira/browse/AMBARI-19293


Repository: ambari


Description
---

* Perform TEZ `configure` step with lock (will add other services later).
* Use service-specific locks in after-install hook to decrease contention


Diffs
-

  
ambari-common/src/main/python/resource_management/libraries/functions/conf_select.py
 ce00f0c6f69f255690f1ab9528bea6cee184a73e 
  
ambari-server/src/main/resources/common-services/TEZ/0.4.0.2.1/package/scripts/tez_client.py
 8018f0f960322aac196caf3915a0a944e7765d8e 
  
ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/after-INSTALL/scripts/params.py
 ed34217757540278a8730d58797cc47536e58874 
  
ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/after-INSTALL/scripts/shared_initialization.py
 e9f22834b5479330a6f00ca69991cc690780b3c3 
  ambari-server/src/test/python/stacks/2.2/common/test_conf_select.py 
d445d741ba4225a33e2a763eed6dfcf8394dceb6 

Diff: https://reviews.apache.org/r/55016/diff/


Testing (updated)
---

Manual test:

* install cluster via blueprint with parallel execution disabled (this is the 
default)
* install cluster via blueprint with parallel execution enabled (needs packages 
pre-installed and users pre-created)

Unit tests:

```
$ mvn -am -pl ambari-agent,ambari-server -DskipSurefireTests clean test
...
[INFO] --- exec-maven-plugin:1.2.1:exec (python-test) @ ambari-server ---
...
Total run:1160
Total errors:0
Total failures:0
OK
...
[INFO] --- exec-maven-plugin:1.2.1:exec (python-test) @ ambari-agent ---
...
Ran 451 tests in 15.344s

OK
...
[INFO] Reactor Summary:
[INFO]
[INFO] Ambari Main ... SUCCESS [6.650s]
[INFO] Apache Ambari Project POM . SUCCESS [0.027s]
[INFO] Ambari Views .. SUCCESS [2.355s]
[INFO] utility ... SUCCESS [1.182s]
[INFO] ambari-metrics  SUCCESS [0.315s]
[INFO] Ambari Metrics Common . SUCCESS [0.730s]
[INFO] Ambari Server . SUCCESS [1:38.241s]
[INFO] Ambari Agent .. SUCCESS [17.247s]
[INFO] 
[INFO] BUILD SUCCESS
```


Thanks,

Attila Doroszlai

Re: Review Request 55016: AMBARI-19293. Component install and after-install should not run in parallel

2016-12-28 Thread Jonathan Hurley



> On Dec. 28, 2016, 9:36 a.m., Jonathan Hurley wrote:
> > Although I do not see anything wrong with this implementation, I'm curious 
> > why we simply can't use a single lock for this. Why do we need specific 
> > locks per component? 
> > 
> > Shouldn't it be enough to have a single lock which is used during the 
> > installation phase as well as the conf-select symlinking phase? The goal 
> > here is to just ensure they don't run in parallel.
> 
> Sebastian Toader wrote:
> In order to improve cluster creation time we need to run in parallel as 
> much as possible and synchronize only those bits we know can't run in 
> parallel. During installation of two components we we should run in parallel 
> the bits that are independent.

I think that's my point - none of the install commands should ever run at the 
same time as the conf-select work. conf-select should take less than a second 
to run and re-link. I don't think it's going to break the bank here. I think 
it's much safer to ensure that they are 100% mutually exclusive.


- Jonathan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/55016/#review160228
---


On Dec. 23, 2016, 1:05 p.m., Attila Doroszlai wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/55016/
> ---
> 
> (Updated Dec. 23, 2016, 1:05 p.m.)
> 
> 
> Review request for Ambari, Andrew Onischuk, Balázs Bence Sári, Jonathan 
> Hurley, Laszlo Puskas, and Sebastian Toader.
> 
> 
> Bugs: AMBARI-19293
> https://issues.apache.org/jira/browse/AMBARI-19293
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> * Perform TEZ `configure` step with lock (will add other services later).
> * Use service-specific locks in after-install hook to decrease contention
> 
> 
> Diffs
> -
> 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/conf_select.py
>  ce00f0c6f69f255690f1ab9528bea6cee184a73e 
>   
> ambari-server/src/main/resources/common-services/TEZ/0.4.0.2.1/package/scripts/tez_client.py
>  8018f0f960322aac196caf3915a0a944e7765d8e 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/after-INSTALL/scripts/params.py
>  ed34217757540278a8730d58797cc47536e58874 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/after-INSTALL/scripts/shared_initialization.py
>  e9f22834b5479330a6f00ca69991cc690780b3c3 
>   ambari-server/src/test/python/stacks/2.2/common/test_conf_select.py 
> d445d741ba4225a33e2a763eed6dfcf8394dceb6 
> 
> Diff: https://reviews.apache.org/r/55016/diff/
> 
> 
> Testing
> ---
> 
> Manual test (still in progress):
> 
> * install cluster via blueprint with parallel execution disabled (this is the 
> default)
> * install cluster via blueprint with parallel execution enabled (needs 
> packages pre-installed and users pre-created)
> 
> Unit tests:
> 
> ```
> $ mvn -am -pl ambari-agent,ambari-server -DskipSurefireTests clean test
> ...
> [INFO] --- exec-maven-plugin:1.2.1:exec (python-test) @ ambari-server ---
> ...
> Total run:1160
> Total errors:0
> Total failures:0
> OK
> ...
> [INFO] --- exec-maven-plugin:1.2.1:exec (python-test) @ ambari-agent ---
> ...
> Ran 451 tests in 15.344s
> 
> OK
> ...
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Ambari Main ... SUCCESS [6.650s]
> [INFO] Apache Ambari Project POM . SUCCESS [0.027s]
> [INFO] Ambari Views .. SUCCESS [2.355s]
> [INFO] utility ... SUCCESS [1.182s]
> [INFO] ambari-metrics  SUCCESS [0.315s]
> [INFO] Ambari Metrics Common . SUCCESS [0.730s]
> [INFO] Ambari Server . SUCCESS [1:38.241s]
> [INFO] Ambari Agent .. SUCCESS [17.247s]
> [INFO] 
> 
> [INFO] BUILD SUCCESS
> ```
> 
> 
> Thanks,
> 
> Attila Doroszlai
> 
>

Re: Review Request 55016: AMBARI-19293. Component install and after-install should not run in parallel

2016-12-28 Thread Sebastian Toader



> On Dec. 28, 2016, 3:36 p.m., Jonathan Hurley wrote:
> > Although I do not see anything wrong with this implementation, I'm curious 
> > why we simply can't use a single lock for this. Why do we need specific 
> > locks per component? 
> > 
> > Shouldn't it be enough to have a single lock which is used during the 
> > installation phase as well as the conf-select symlinking phase? The goal 
> > here is to just ensure they don't run in parallel.

In order to improve cluster creation time we need to run in parallel as much as 
possible and synchronize only those bits we know can't run in parallel. During 
installation of two components we we should run in parallel the bits that are 
independent.


- Sebastian


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/55016/#review160228
---


On Dec. 23, 2016, 7:05 p.m., Attila Doroszlai wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/55016/
> ---
> 
> (Updated Dec. 23, 2016, 7:05 p.m.)
> 
> 
> Review request for Ambari, Andrew Onischuk, Balázs Bence Sári, Jonathan 
> Hurley, Laszlo Puskas, and Sebastian Toader.
> 
> 
> Bugs: AMBARI-19293
> https://issues.apache.org/jira/browse/AMBARI-19293
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> * Perform TEZ `configure` step with lock (will add other services later).
> * Use service-specific locks in after-install hook to decrease contention
> 
> 
> Diffs
> -
> 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/conf_select.py
>  ce00f0c6f69f255690f1ab9528bea6cee184a73e 
>   
> ambari-server/src/main/resources/common-services/TEZ/0.4.0.2.1/package/scripts/tez_client.py
>  8018f0f960322aac196caf3915a0a944e7765d8e 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/after-INSTALL/scripts/params.py
>  ed34217757540278a8730d58797cc47536e58874 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/after-INSTALL/scripts/shared_initialization.py
>  e9f22834b5479330a6f00ca69991cc690780b3c3 
>   ambari-server/src/test/python/stacks/2.2/common/test_conf_select.py 
> d445d741ba4225a33e2a763eed6dfcf8394dceb6 
> 
> Diff: https://reviews.apache.org/r/55016/diff/
> 
> 
> Testing
> ---
> 
> Manual test (still in progress):
> 
> * install cluster via blueprint with parallel execution disabled (this is the 
> default)
> * install cluster via blueprint with parallel execution enabled (needs 
> packages pre-installed and users pre-created)
> 
> Unit tests:
> 
> ```
> $ mvn -am -pl ambari-agent,ambari-server -DskipSurefireTests clean test
> ...
> [INFO] --- exec-maven-plugin:1.2.1:exec (python-test) @ ambari-server ---
> ...
> Total run:1160
> Total errors:0
> Total failures:0
> OK
> ...
> [INFO] --- exec-maven-plugin:1.2.1:exec (python-test) @ ambari-agent ---
> ...
> Ran 451 tests in 15.344s
> 
> OK
> ...
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Ambari Main ... SUCCESS [6.650s]
> [INFO] Apache Ambari Project POM . SUCCESS [0.027s]
> [INFO] Ambari Views .. SUCCESS [2.355s]
> [INFO] utility ... SUCCESS [1.182s]
> [INFO] ambari-metrics  SUCCESS [0.315s]
> [INFO] Ambari Metrics Common . SUCCESS [0.730s]
> [INFO] Ambari Server . SUCCESS [1:38.241s]
> [INFO] Ambari Agent .. SUCCESS [17.247s]
> [INFO] 
> 
> [INFO] BUILD SUCCESS
> ```
> 
> 
> Thanks,
> 
> Attila Doroszlai
> 
>

Re: Review Request 55016: AMBARI-19293. Component install and after-install should not run in parallel

2016-12-28 Thread Jonathan Hurley


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/55016/#review160228
---



Although I do not see anything wrong with this implementation, I'm curious why 
we simply can't use a single lock for this. Why do we need specific locks per 
component? 

Shouldn't it be enough to have a single lock which is used during the 
installation phase as well as the conf-select symlinking phase? The goal here 
is to just ensure they don't run in parallel.

- Jonathan Hurley


On Dec. 23, 2016, 1:05 p.m., Attila Doroszlai wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/55016/
> ---
> 
> (Updated Dec. 23, 2016, 1:05 p.m.)
> 
> 
> Review request for Ambari, Andrew Onischuk, Balázs Bence Sári, Jonathan 
> Hurley, Laszlo Puskas, and Sebastian Toader.
> 
> 
> Bugs: AMBARI-19293
> https://issues.apache.org/jira/browse/AMBARI-19293
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> * Perform TEZ `configure` step with lock (will add other services later).
> * Use service-specific locks in after-install hook to decrease contention
> 
> 
> Diffs
> -
> 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/conf_select.py
>  ce00f0c6f69f255690f1ab9528bea6cee184a73e 
>   
> ambari-server/src/main/resources/common-services/TEZ/0.4.0.2.1/package/scripts/tez_client.py
>  8018f0f960322aac196caf3915a0a944e7765d8e 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/after-INSTALL/scripts/params.py
>  ed34217757540278a8730d58797cc47536e58874 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/after-INSTALL/scripts/shared_initialization.py
>  e9f22834b5479330a6f00ca69991cc690780b3c3 
>   ambari-server/src/test/python/stacks/2.2/common/test_conf_select.py 
> d445d741ba4225a33e2a763eed6dfcf8394dceb6 
> 
> Diff: https://reviews.apache.org/r/55016/diff/
> 
> 
> Testing
> ---
> 
> Manual test (still in progress):
> 
> * install cluster via blueprint with parallel execution disabled (this is the 
> default)
> * install cluster via blueprint with parallel execution enabled (needs 
> packages pre-installed and users pre-created)
> 
> Unit tests:
> 
> ```
> $ mvn -am -pl ambari-agent,ambari-server -DskipSurefireTests clean test
> ...
> [INFO] --- exec-maven-plugin:1.2.1:exec (python-test) @ ambari-server ---
> ...
> Total run:1160
> Total errors:0
> Total failures:0
> OK
> ...
> [INFO] --- exec-maven-plugin:1.2.1:exec (python-test) @ ambari-agent ---
> ...
> Ran 451 tests in 15.344s
> 
> OK
> ...
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Ambari Main ... SUCCESS [6.650s]
> [INFO] Apache Ambari Project POM . SUCCESS [0.027s]
> [INFO] Ambari Views .. SUCCESS [2.355s]
> [INFO] utility ... SUCCESS [1.182s]
> [INFO] ambari-metrics  SUCCESS [0.315s]
> [INFO] Ambari Metrics Common . SUCCESS [0.730s]
> [INFO] Ambari Server . SUCCESS [1:38.241s]
> [INFO] Ambari Agent .. SUCCESS [17.247s]
> [INFO] 
> 
> [INFO] BUILD SUCCESS
> ```
> 
> 
> Thanks,
> 
> Attila Doroszlai
> 
>

Review Request 55016: AMBARI-19293. Component install and after-install should not run in parallel

2016-12-23 Thread Attila Doroszlai


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/55016/
---

Review request for Ambari, Balázs Bence Sári, Laszlo Puskas, and Sebastian 
Toader.


Bugs: AMBARI-19293
https://issues.apache.org/jira/browse/AMBARI-19293


Repository: ambari


Description
---

* Perform TEZ `configure` step with lock (will add other services later).
* Use service-specific locks in after-install hook to decrease contention


Diffs
-

  
ambari-common/src/main/python/resource_management/libraries/functions/conf_select.py
 ce00f0c6f69f255690f1ab9528bea6cee184a73e 
  
ambari-server/src/main/resources/common-services/TEZ/0.4.0.2.1/package/scripts/tez_client.py
 8018f0f960322aac196caf3915a0a944e7765d8e 
  
ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/after-INSTALL/scripts/params.py
 ed34217757540278a8730d58797cc47536e58874 
  
ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/after-INSTALL/scripts/shared_initialization.py
 e9f22834b5479330a6f00ca69991cc690780b3c3 
  ambari-server/src/test/python/stacks/2.2/common/test_conf_select.py 
d445d741ba4225a33e2a763eed6dfcf8394dceb6 

Diff: https://reviews.apache.org/r/55016/diff/


Testing
---

Manual test (still in progress):

* install cluster via blueprint with parallel execution disabled (this is the 
default)
* install cluster via blueprint with parallel execution enabled (needs packages 
pre-installed and users pre-created)

Unit tests:

```
$ mvn -am -pl ambari-agent,ambari-server -DskipSurefireTests clean test
...
[INFO] --- exec-maven-plugin:1.2.1:exec (python-test) @ ambari-server ---
...
Total run:1160
Total errors:0
Total failures:0
OK
...
[INFO] --- exec-maven-plugin:1.2.1:exec (python-test) @ ambari-agent ---
...
Ran 451 tests in 15.344s

OK
...
[INFO] Reactor Summary:
[INFO]
[INFO] Ambari Main ... SUCCESS [6.650s]
[INFO] Apache Ambari Project POM . SUCCESS [0.027s]
[INFO] Ambari Views .. SUCCESS [2.355s]
[INFO] utility ... SUCCESS [1.182s]
[INFO] ambari-metrics  SUCCESS [0.315s]
[INFO] Ambari Metrics Common . SUCCESS [0.730s]
[INFO] Ambari Server . SUCCESS [1:38.241s]
[INFO] Ambari Agent .. SUCCESS [17.247s]
[INFO] 
[INFO] BUILD SUCCESS
```


Thanks,

Attila Doroszlai

Re: Review Request 55016: AMBARI-19293. Component install and after-install should not run in parallel

Re: Review Request 55016: AMBARI-19293. Component install and after-install should not run in parallel

Re: Review Request 55016: AMBARI-19293. Component install and after-install should not run in parallel

Re: Review Request 55016: AMBARI-19293. Component install and after-install should not run in parallel

Re: Review Request 55016: AMBARI-19293. Component install and after-install should not run in parallel

Re: Review Request 55016: AMBARI-19293. Component install and after-install should not run in parallel

Re: Review Request 55016: AMBARI-19293. Component install and after-install should not run in parallel

Re: Review Request 55016: AMBARI-19293. Component install and after-install should not run in parallel

Re: Review Request 55016: AMBARI-19293. Component install and after-install should not run in parallel

Re: Review Request 55016: AMBARI-19293. Component install and after-install should not run in parallel

Re: Review Request 55016: AMBARI-19293. Component install and after-install should not run in parallel

Re: Review Request 55016: AMBARI-19293. Component install and after-install should not run in parallel

Re: Review Request 55016: AMBARI-19293. Component install and after-install should not run in parallel

Re: Review Request 55016: AMBARI-19293. Component install and after-install should not run in parallel

Re: Review Request 55016: AMBARI-19293. Component install and after-install should not run in parallel

Re: Review Request 55016: AMBARI-19293. Component install and after-install should not run in parallel

Re: Review Request 55016: AMBARI-19293. Component install and after-install should not run in parallel

Re: Review Request 55016: AMBARI-19293. Component install and after-install should not run in parallel

Re: Review Request 55016: AMBARI-19293. Component install and after-install should not run in parallel

Re: Review Request 55016: AMBARI-19293. Component install and after-install should not run in parallel

Re: Review Request 55016: AMBARI-19293. Component install and after-install should not run in parallel

Re: Review Request 55016: AMBARI-19293. Component install and after-install should not run in parallel

Re: Review Request 55016: AMBARI-19293. Component install and after-install should not run in parallel

Re: Review Request 55016: AMBARI-19293. Component install and after-install should not run in parallel

Re: Review Request 55016: AMBARI-19293. Component install and after-install should not run in parallel

Review Request 55016: AMBARI-19293. Component install and after-install should not run in parallel

26 matches

Site Navigation

Mail list logo

Footer information