Re: Review Request 57792: Fixes for Express Upgrade on large-scale clusters: batch execute-stage for hdp-select set all, alert for hdp-select versions, EU from HDP 2.2 to 2.5

2017-03-20 Thread Alejandro Fernandez 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57792/
---

(Updated March 21, 2017, 12:49 a.m.)


Review request for Ambari, Jonathan Hurley, Nate Cole, Sumit Mohanty, and Sid 
Wagle.


Changes
---

Fixed unit tests


Bugs: AMBARI-20490
https://issues.apache.org/jira/browse/AMBARI-20490


Repository: ambari


Description
---

On large-scale clusters over 1000 nodes, performance during Express Upgrade can 
be very slow and it takes more time to fix potential errors.
We can remedy these with the following fixes:

* "hdp-select set all" is currently ran in the ClusterGroup with a single stage 
instead of batching. This can cause Ambari Server to be the bottleneck when 
processing 1000+ requests. Allow the execute-stage inside a ClusterGroup to use 
batching.

* If some hosts have symlinks or dirs that don't belong in /usr/hdp/ then it 
will prevent the hosts from running "hdp-select versions". We can find these 
problematic hosts with an alert.

* There is currently no upgrade path from HDP 2.2 directly to 2.5. Doing a 
2-step upgrade is very time consuming (only for branch 2.4 since HDP 2.2 no 
longer supported in Ambari 2.5).


Diffs (updated)
-

  
ambari-common/src/main/python/resource_management/libraries/functions/stack_select.py
 20b4cbd 
  
ambari-server/src/main/java/org/apache/ambari/server/checks/AtlasPresenceCheck.java
 8556436 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/UpgradeResourceProvider.java
 ca9ce07 
  ambari-server/src/main/java/org/apache/ambari/server/state/UpgradeHelper.java 
ee5451f 
  
ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/ClusterGrouping.java
 c45ec24 
  
ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/Grouping.java
 8003223 
  
ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/StageWrapper.java
 669d50f 
  
ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/StageWrapperBuilder.java
 3cf3f8b 
  ambari-server/src/main/resources/alerts.json 2559b3a 
  ambari-server/src/main/resources/host_scripts/alert_version_select.py 
PRE-CREATION 
  
ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.3.xml
 28ca939 
  
ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.4.xml
 9aff1a9 
  
ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.5.xml
 adccac0 
  
ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/nonrolling-upgrade-2.4.xml
 3c0104b 
  
ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/nonrolling-upgrade-2.5.xml
 cd63651 
  
ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/nonrolling-upgrade-2.5.xml
 39cb0dc 
  
ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/nonrolling-upgrade-2.6.xml
 57a5203 
  
ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/nonrolling-upgrade-2.6.xml
 035cb59 
  ambari-server/src/main/resources/upgrade-pack.xsd 1f11aa1 
  
ambari-server/src/test/java/org/apache/ambari/server/api/services/AmbariMetaInfoTest.java
 f1af66f 
  
ambari-server/src/test/java/org/apache/ambari/server/controller/AmbariManagementControllerTest.java
 bf6c41a 
  
ambari-server/src/test/java/org/apache/ambari/server/metadata/AgentAlertDefinitionsTest.java
 cbc5e69 
  
ambari-server/src/test/java/org/apache/ambari/server/state/UpgradeHelperTest.java
 8e5ad0a 
  
ambari-server/src/test/resources/stacks/HDP/2.1.1/upgrades/upgrade_test_batch.xml
 PRE-CREATION 
  
ambari-server/src/test/resources/stacks/HDP/2.1.1/upgrades/upgrade_to_new_stack.xml
 fe387f3 


Diff: https://reviews.apache.org/r/57792/diff/2/

Changes: https://reviews.apache.org/r/57792/diff/1-2/


Testing
---

mvn clean test


Thanks,

Alejandro Fernandez

Re: Review Request 57795: While generating hash for ambari-server to fill agent cache, all the symlinks should be traversed too.

2017-03-20 Thread Madhuvanthi Radhakrishnan 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57795/
---

(Updated March 21, 2017, 12:27 a.m.)


Review request for Ambari, Alejandro Fernandez, Jayush Luniya, and Sumit 
Mohanty.


Bugs: AMBARI-20512
https://issues.apache.org/jira/browse/AMBARI-20512


Repository: ambari


Description
---

the archive.zip and .hash is not getting generated on the server side when the 
package folder is under
stacks///services//package
instead of common-services///package
On closer look, the issue was that when the directories are being traversed to 
look for hooks and package folders that need to be hashed/archived for having 
it in the agent cache, those directories that were symlinks were being skipped.
Fix is to include them.


Diffs (updated)
-

  ambari-server/src/main/python/ambari_server/resourceFilesKeeper.py baf8179 


Diff: https://reviews.apache.org/r/57795/diff/2/

Changes: https://reviews.apache.org/r/57795/diff/1-2/


Testing
---

Tested using a management pack


Thanks,

Madhuvanthi Radhakrishnan

Re: Review Request 57687: AMBARI-20443 No need to show `hive_ambari_database`

2017-03-20 Thread Masahiro Tanaka 


> On 3月 17, 2017, 5:21 a.m., Alejandro Fernandez wrote:
> > ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/configuration/hive-env.xml
> > Line 97 (original), 97 (patched)
> > 
> >
> > Does this config need to be deleted from existing clusters? Which stack 
> > versions does this affect?
> > 
> > If so, you can mark the property as , which will also cause 
> > Ambari to delete it during EU/RU

Sorry for late reply, and thank you for your reviewing.
I'll check if existing clusters do not need this property.


- Masahiro


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57687/#review169205
---


On 3月 16, 2017, 9:55 p.m., Masahiro Tanaka wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/57687/
> ---
> 
> (Updated 3月 16, 2017, 9:55 p.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Jayush Luniya, and Yusaku 
> Sako.
> 
> 
> Bugs: AMBARI-20443
> https://issues.apache.org/jira/browse/AMBARI-20443
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> In Advanced hive-env, there is a parameter named hive_ambari_database. Though 
> I change the parameter, it looks like nothing has changed. I greped the 
> entire ambari source code, the parameter is not referenced.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/configuration/hive-env.xml
>  be79331 
> 
> 
> Diff: https://reviews.apache.org/r/57687/diff/1/
> 
> 
> Testing
> ---
> 
> -1 overall. Here are the results of testing the latest attachment 
> http://issues.apache.org/jira/secure/attachment/12858664/AMBARI-20443.0.patch
> against trunk revision .
> +1 @author. The patch does not contain any @author tags.
> -1 tests included. The patch doesn't appear to include any new or modified 
> tests.
> Please justify why no new tests are needed for this patch.
> Also please list what manual steps were performed to verify this patch.
> +1 javac. The applied patch does not increase the total number of javac 
> compiler warnings.
> +1 release audit. The applied patch does not increase the total number of 
> release audit warnings.
> +1 core tests. The patch passed unit tests in ambari-server.
> Test results: 
> https://builds.apache.org/job/Ambari-trunk-test-patch/11023//testReport/
> Console output: 
> https://builds.apache.org/job/Ambari-trunk-test-patch/11023//console
> This message is automatically generated.
> 
> 
> Thanks,
> 
> Masahiro Tanaka
> 
>

Re: Review Request 57795: While generating hash for ambari-server to fill agent cache, all the symlinks should be traversed too.

2017-03-20 Thread Alejandro Fernandez 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57795/#review169517
---


Fix it, then Ship it!





ambari-server/src/main/python/ambari_server/resourceFilesKeeper.py
Line 79 (original), 79 (patched)


Small nitpick, put a space after the comma


- Alejandro Fernandez


On March 20, 2017, 10:02 p.m., Madhuvanthi Radhakrishnan wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/57795/
> ---
> 
> (Updated March 20, 2017, 10:02 p.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Jayush Luniya, and Sumit 
> Mohanty.
> 
> 
> Bugs: AMBARI-20512
> https://issues.apache.org/jira/browse/AMBARI-20512
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> the archive.zip and .hash is not getting generated on the server side when 
> the package folder is under
> stacks///services//package
> instead of common-services///package
> On closer look, the issue was that when the directories are being traversed 
> to look for hooks and package folders that need to be hashed/archived for 
> having it in the agent cache, those directories that were symlinks were being 
> skipped.
> Fix is to include them.
> 
> 
> Diffs
> -
> 
>   ambari-server/src/main/python/ambari_server/resourceFilesKeeper.py baf8179 
> 
> 
> Diff: https://reviews.apache.org/r/57795/diff/1/
> 
> 
> Testing
> ---
> 
> Tested using a management pack
> 
> 
> Thanks,
> 
> Madhuvanthi Radhakrishnan
> 
>

Re: Review Request 57793: YARN Service Check failing due to incorrect jar name, Slider failing due to storm plugin conf not present at the moment

2017-03-20 Thread Sid Wagle 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57793/#review169512
---


Ship it!




Ship It!

- Sid Wagle


On March 20, 2017, 9:47 p.m., Alejandro Fernandez wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/57793/
> ---
> 
> (Updated March 20, 2017, 9:47 p.m.)
> 
> 
> Review request for Ambari, Dmytro Sen, Sumit Mohanty, Swapan Shridhar, Sid 
> Wagle, and Vitalyi Brodetskyi.
> 
> 
> Bugs: AMBARI-20511
> https://issues.apache.org/jira/browse/AMBARI-20511
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> YARN service check is failing on HDP 3.0 since Ambari is still looking for a 
> 2.* jar instead of 3.*
> 
> ```
> resource_management.core.exceptions.ExecutionFailed: Execution of 'hadoop 
> --config /usr/hdp/current/hadoop-client/conf jar 
> /usr/hdp/current/hadoop-mapreduce-client/hadoop-mapreduce-examples-2.*.jar 
> wordcount /user/ambari-qa/mapredsmokeinput /user/ambari-qa/mapredsmokeoutput' 
> returned 255. JAR does not exist or is not a normal file: 
> /usr/hdp/3.0.0.0-154/hadoop-mapreduce/hadoop-mapreduce-examples-2.*.jar
> ```
> 
> Slider is failing because it currently does not have the Storm plugin, which 
> is expected this early in the release, so comment it out for now.
> ```
> raise Fail("Cannot create directory '{0}' as '{1}' is a broken 
> symlink".format(path, dirname))
> resource_management.core.exceptions.Fail: Cannot create directory 
> '/usr/hdp/current/storm-slider-client/conf' as 
> '/usr/hdp/current/storm-slider-client' is a broken symlink
> ```
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/SLIDER/0.91.0.3.0/package/scripts/slider.py
>  f7d7fbf 
>   
> ambari-server/src/main/resources/common-services/YARN/3.0.0.3.0/package/scripts/params_linux.py
>  d88f518 
> 
> 
> Diff: https://reviews.apache.org/r/57793/diff/1/
> 
> 
> Testing
> ---
> 
> Python unit tests passed.
> --
> Total run:1192
> Total errors:0
> Total failures:0
> OK
> 
> Verified on live cluster.
> 
> 
> Thanks,
> 
> Alejandro Fernandez
> 
>

Review Request 57793: YARN Service Check failing due to incorrect jar name, Slider failing due to storm plugin conf not present at the moment

2017-03-20 Thread Alejandro Fernandez 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57793/
---

Review request for Ambari, Dmytro Sen, Sumit Mohanty, Swapan Shridhar, Sid 
Wagle, and Vitalyi Brodetskyi.


Bugs: AMBARI-20511
https://issues.apache.org/jira/browse/AMBARI-20511


Repository: ambari


Description
---

YARN service check is failing on HDP 3.0 since Ambari is still looking for a 
2.* jar instead of 3.*

```
resource_management.core.exceptions.ExecutionFailed: Execution of 'hadoop 
--config /usr/hdp/current/hadoop-client/conf jar 
/usr/hdp/current/hadoop-mapreduce-client/hadoop-mapreduce-examples-2.*.jar 
wordcount /user/ambari-qa/mapredsmokeinput /user/ambari-qa/mapredsmokeoutput' 
returned 255. JAR does not exist or is not a normal file: 
/usr/hdp/3.0.0.0-154/hadoop-mapreduce/hadoop-mapreduce-examples-2.*.jar
```

Slider is failing because it currently does not have the Storm plugin, which is 
expected this early in the release, so comment it out for now.
```
raise Fail("Cannot create directory '{0}' as '{1}' is a broken 
symlink".format(path, dirname))
resource_management.core.exceptions.Fail: Cannot create directory 
'/usr/hdp/current/storm-slider-client/conf' as 
'/usr/hdp/current/storm-slider-client' is a broken symlink
```


Diffs
-

  
ambari-server/src/main/resources/common-services/SLIDER/0.91.0.3.0/package/scripts/slider.py
 f7d7fbf 
  
ambari-server/src/main/resources/common-services/YARN/3.0.0.3.0/package/scripts/params_linux.py
 d88f518 


Diff: https://reviews.apache.org/r/57793/diff/1/


Testing
---

Python unit tests passed.
--
Total run:1192
Total errors:0
Total failures:0
OK

Verified on live cluster.


Thanks,

Alejandro Fernandez

Re: Review Request 57792: Fixes for Express Upgrade on large-scale clusters: batch execute-stage for hdp-select set all, alert for hdp-select versions, EU from HDP 2.2 to 2.5

2017-03-20 Thread Alejandro Fernandez 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57792/#review169503
---




ambari-common/src/main/python/resource_management/libraries/functions/stack_select.py
Lines 293 (patched)


This is needed so that the alert can find if "hdp-select versions" is 
broken.



ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/Grouping.java
Line 222 (original)


This variable was unused.



ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/StageWrapper.java
Lines 139 (patched)


Used so I can preserve the original text of the stage in order to create 
the name of the commands.



ambari-server/src/main/resources/alerts.json
Lines 184 (patched)


New alert definition.



ambari-server/src/main/resources/host_scripts/alert_version_select.py
Lines 89 (patched)


New alert to detect if "hdp-select versions" won't work because of 
incorrect symlinks/dirs in /usr/hdp.


- Alejandro Fernandez


On March 20, 2017, 8:55 p.m., Alejandro Fernandez wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/57792/
> ---
> 
> (Updated March 20, 2017, 8:55 p.m.)
> 
> 
> Review request for Ambari, Jonathan Hurley, Nate Cole, Sumit Mohanty, and Sid 
> Wagle.
> 
> 
> Bugs: AMBARI-20490
> https://issues.apache.org/jira/browse/AMBARI-20490
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> On large-scale clusters over 1000 nodes, performance during Express Upgrade 
> can be very slow and it takes more time to fix potential errors.
> We can remedy these with the following fixes:
> 
> * "hdp-select set all" is currently ran in the ClusterGroup with a single 
> stage instead of batching. This can cause Ambari Server to be the bottleneck 
> when processing 1000+ requests. Allow the execute-stage inside a ClusterGroup 
> to use batching.
> 
> * If some hosts have symlinks or dirs that don't belong in /usr/hdp/ then it 
> will prevent the hosts from running "hdp-select versions". We can find these 
> problematic hosts with an alert.
> 
> * There is currently no upgrade path from HDP 2.2 directly to 2.5. Doing a 
> 2-step upgrade is very time consuming (only for branch 2.4 since HDP 2.2 no 
> longer supported in Ambari 2.5).
> 
> 
> Diffs
> -
> 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/stack_select.py
>  20b4cbd 
>   
> ambari-server/src/main/java/org/apache/ambari/server/checks/AtlasPresenceCheck.java
>  8556436 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/UpgradeResourceProvider.java
>  ca9ce07 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/UpgradeHelper.java 
> ee5451f 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/ClusterGrouping.java
>  c45ec24 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/Grouping.java
>  8003223 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/StageWrapper.java
>  669d50f 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/StageWrapperBuilder.java
>  3cf3f8b 
>   ambari-server/src/main/resources/alerts.json 2559b3a 
>   ambari-server/src/main/resources/host_scripts/alert_version_select.py 
> PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.3.xml
>  28ca939 
>   
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.4.xml
>  9aff1a9 
>   
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.5.xml
>  adccac0 
>   
> ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/nonrolling-upgrade-2.4.xml
>  3c0104b 
>   
> ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/nonrolling-upgrade-2.5.xml
>  cd63651 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/nonrolling-upgrade-2.5.xml
>  39cb0dc 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/nonrolling-upgrade-2.6.xml
>  57a5203 
>   
> ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/nonrolling-upgrade-2.6.xml
>  035cb59 
>   ambari-server/src/main/resources/upgrade-pack.xsd 1f11aa1 
>   
> ambari-server/src/test/java/org/apache/ambari/server/api/services/AmbariMetaInfoTest.java
>  f1af66f 
>   
> ambari-server/src/test/java/org/apache/ambari/server/controller/AmbariManagementControllerTest.java
>  bf6c41a 
>   
>

Review Request 57792: Fixes for Express Upgrade on large-scale clusters: batch execute-stage for hdp-select set all, alert for hdp-select versions, EU from HDP 2.2 to 2.5

2017-03-20 Thread Alejandro Fernandez 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57792/
---

Review request for Ambari, Jonathan Hurley, Nate Cole, Sumit Mohanty, and Sid 
Wagle.


Bugs: AMBARI-20490
https://issues.apache.org/jira/browse/AMBARI-20490


Repository: ambari


Description
---

On large-scale clusters over 1000 nodes, performance during Express Upgrade can 
be very slow and it takes more time to fix potential errors.
We can remedy these with the following fixes:

* "hdp-select set all" is currently ran in the ClusterGroup with a single stage 
instead of batching. This can cause Ambari Server to be the bottleneck when 
processing 1000+ requests. Allow the execute-stage inside a ClusterGroup to use 
batching.

* If some hosts have symlinks or dirs that don't belong in /usr/hdp/ then it 
will prevent the hosts from running "hdp-select versions". We can find these 
problematic hosts with an alert.

* There is currently no upgrade path from HDP 2.2 directly to 2.5. Doing a 
2-step upgrade is very time consuming (only for branch 2.4 since HDP 2.2 no 
longer supported in Ambari 2.5).


Diffs
-

  
ambari-common/src/main/python/resource_management/libraries/functions/stack_select.py
 20b4cbd 
  
ambari-server/src/main/java/org/apache/ambari/server/checks/AtlasPresenceCheck.java
 8556436 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/UpgradeResourceProvider.java
 ca9ce07 
  ambari-server/src/main/java/org/apache/ambari/server/state/UpgradeHelper.java 
ee5451f 
  
ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/ClusterGrouping.java
 c45ec24 
  
ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/Grouping.java
 8003223 
  
ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/StageWrapper.java
 669d50f 
  
ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/StageWrapperBuilder.java
 3cf3f8b 
  ambari-server/src/main/resources/alerts.json 2559b3a 
  ambari-server/src/main/resources/host_scripts/alert_version_select.py 
PRE-CREATION 
  
ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.3.xml
 28ca939 
  
ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.4.xml
 9aff1a9 
  
ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.5.xml
 adccac0 
  
ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/nonrolling-upgrade-2.4.xml
 3c0104b 
  
ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/nonrolling-upgrade-2.5.xml
 cd63651 
  
ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/nonrolling-upgrade-2.5.xml
 39cb0dc 
  
ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/nonrolling-upgrade-2.6.xml
 57a5203 
  
ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/nonrolling-upgrade-2.6.xml
 035cb59 
  ambari-server/src/main/resources/upgrade-pack.xsd 1f11aa1 
  
ambari-server/src/test/java/org/apache/ambari/server/api/services/AmbariMetaInfoTest.java
 f1af66f 
  
ambari-server/src/test/java/org/apache/ambari/server/controller/AmbariManagementControllerTest.java
 bf6c41a 
  
ambari-server/src/test/java/org/apache/ambari/server/metadata/AgentAlertDefinitionsTest.java
 cbc5e69 
  
ambari-server/src/test/java/org/apache/ambari/server/state/UpgradeHelperTest.java
 8e5ad0a 
  
ambari-server/src/test/resources/stacks/HDP/2.1.1/upgrades/upgrade_test_batch.xml
 PRE-CREATION 
  
ambari-server/src/test/resources/stacks/HDP/2.1.1/upgrades/upgrade_to_new_stack.xml
 fe387f3 


Diff: https://reviews.apache.org/r/57792/diff/1/


Testing
---

mvn clean test


Thanks,

Alejandro Fernandez

Re: Review Request 57781: Configuration Markdown Not Properly Parsed By Some Editors

2017-03-20 Thread Jonathan Hurley 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57781/
---

(Updated March 20, 2017, 2:52 p.m.)


Review request for Ambari, Nate Cole and Robert Levas.


Summary (updated)
-

Configuration Markdown Not Properly Parsed By Some Editors


Bugs: AMBARI-20502
https://issues.apache.org/jira/browse/AMBARI-20502


Repository: ambari


Description
---

Some markdown displays, like GitHub, don't properly display the configuration 
markdown. Sections appear as:

{code}
Alerts & Notifications | Property Name | 10 Hosts | ~50 Hosts | ~100 Hosts 
| 500+ Hosts | | --- | --- | --- | --- | --- | | 
alerts.execution.scheduler.threadpool.size.core | 2 | 2 | 4 | 4 | | 
alerts.execution.scheduler.threadpool.size.max | 2 | 2 | 8 | 8 | | 
alerts.execution.scheduler.threadpool.worker.size | 400 | 2000 | 4000 | 2 | 
| alerts.cache.enabled | false | false | false | true | | 
alerts.cache.flush.interval | 10 | 10 | 10 | 10 | | alerts.cache.size | 5 | 
5 | 10 | 10 | Jetty API & Agent Thread Pools | Property Name | 
10 Hosts | ~50 Hosts | ~100 Hosts | 500+ Hosts | | --- | --- | --- | --- | --- 
| | client.threadpool.size.max | 25 | 35 | 50 | 65 | | 
agent.threadpool.size.max | 25 | 35 | 75 | 100 |
{code}


Diffs
-

  ambari-server/docs/configuration/index.md 7a7f3b9 
  
ambari-server/src/main/java/org/apache/ambari/server/configuration/Configuration.java
 20431f6 


Diff: https://reviews.apache.org/r/57781/diff/1/


Testing
---

Created a feature branch on Github to test this change. Verified that the 
Markdown has been corrected.


Thanks,

Jonathan Hurley

Re: Review Request 57582: Create a prototype of ambari-server swagger integration

2017-03-20 Thread Oliver Szabo 


> On March 15, 2017, 8:31 p.m., Oliver Szabo wrote:
> > ambari-web/api-docs/index.html
> > Lines 1 (patched)
> > 
> >
> > Instead of storing all of html + css files from swagger in the source 
> > code, I think its better idea to use swagger ui webjar: 
> > 
> > https://mvnrepository.com/artifact/org.webjars/swagger-ui (i think the 
> > exact swagger-ui version is not there that you used, but newer one can work)
> > 
> > I used that for Log Search. (you can unpack that during the build or 
> > use resource files from the jar somehow else)
> > 
> > If you will need any custom files you can override those files during 
> > the build. In Log Search i needed a specific index.html which supports 
> > basic authentication instead of api key, so i changed the index.html 
> > (https://github.com/apache/ambari/blob/trunk/ambari-logsearch/ambari-logsearch-portal/src/main/resources/swagger/swagger.html),
> >  but for every other file is used from the webjar, so i did not need to 
> > store swagger html+css in the source code.
> 
> Oliver Szabo wrote:
> additon: you can drop my issue, if for using the modified swagger-ui, its 
> not enough just include the some new files (extra css files etc.) from 
> https://github.com/jensoleg/swagger-ui
> 
> Jaimin Jetly wrote:
> Since ambari-web using custom look on top of swagger-ui, ambari-web will 
> need to maintain it's override of html+css file. 
> But swagger-ui JS code is something we want to use from 
> https://www.npmjs.com/package/swagger-ui. There is an existing task 
> (https://issues.apache.org/jira/browse/AMBARI-20508) to do analysis on how 
> much of the code we can use from the vendor library instead of maintaining 
> all in ambari-web source code.

thanks. then im dropping the issue.


- Oliver


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57582/#review169051
---


On March 20, 2017, 6:20 p.m., Jaimin Jetly wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/57582/
> ---
> 
> (Updated March 20, 2017, 6:20 p.m.)
> 
> 
> Review request for Ambari, Jonathan Hurley, Jayush Luniya, Nate Cole, Oliver 
> Szabo, Robert Nettleton, Sumit Mohanty, Sid Wagle, and Yusaku Sako.
> 
> 
> Bugs: AMBARI-20436
> https://issues.apache.org/jira/browse/AMBARI-20436
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> A separate branch is created for this work: ambari-rest-api-explorer
> 
> ## As part of this task, following changes are done:
> 1. Users, Groups and Views API are integrated with swagger and exposed from 
> ambari rest api explorer ui (swagger ui) on a deployed cluster at path: 
> http://c6404.ambari.apache.org:8080/api-docs
> 2. swagger-maven-plugin is used to generate swagger.json file on compile 
> time. This file is published in web resources directory. Note that this file 
> is generated build time and will be available on deployed ambari-server host 
> at web resources location but it is not yet decided to be committed and 
> maintained in Ambari source code
> 3. swagger2markup-maven-plugin is used to generate asciidoc from swagger.json 
> file (that can be shown as markdown in github). More information about this 
> format an be found at http://asciidoc.org/ and http://asciidoctor.org/. This 
> generates files in docs/api/asciidoc/** location at build time. This 
> directory is currently intended to be committed and maintained in ambari 
> source code
> 4. swagger-ui (version: v2.1.1-M2) compiled code with the different css skin 
> (adopted from [link|https://github.com/jensoleg/swagger-ui]) is committed to 
> ambari-web/api-docs directory with certain modification to make it work with 
> ambari api. Further ui polishing will be done in subsequent tasks. Also there 
> is a strong possibility to maintain the fork code of swagger-ui and compile 
> (minify and concanate) it during ambari compile time rather than directly 
> using swagger-ui dist files. Doing so will help developers when customization 
> done over swagger-ui will increase
> 5. swagger-annotation expects application to define schema of request body 
> and response for each endpoint to be encapsulated in a class. While Ambari 
> follows this pattern for some of the endpoint, there are many others which 
> does not do so. For The ones which do not does so, new request and response 
> classes were defined. Going forward at the completion of this epic, either 
> each resource type or each resource provider should be coupled with a 
> resource response class and a resource request class. As part of this patch, 
> each resourceprovider worked upon introduces

Re: Review Request 57582: Create a prototype of ambari-server swagger integration

2017-03-20 Thread Jaimin Jetly 


> On March 16, 2017, 3:16 p.m., Nate Cole wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/api/services/groupApi/GroupPrivilegeService.java
> > Lines 19-26 (patched)
> > 
> >
> > Why do these classes need to be in a different package?
> > 
> > This is an odd package name, should just be 
> > "org.apache.ambari.server.api.services.group"
> > 
> > I don't think I've ever seen a camel-case package name.

The package name issue has been addressed in the last revision of the patch


- Jaimin


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57582/#review169161
---


On March 20, 2017, 6:20 p.m., Jaimin Jetly wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/57582/
> ---
> 
> (Updated March 20, 2017, 6:20 p.m.)
> 
> 
> Review request for Ambari, Jonathan Hurley, Jayush Luniya, Nate Cole, Oliver 
> Szabo, Robert Nettleton, Sumit Mohanty, Sid Wagle, and Yusaku Sako.
> 
> 
> Bugs: AMBARI-20436
> https://issues.apache.org/jira/browse/AMBARI-20436
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> A separate branch is created for this work: ambari-rest-api-explorer
> 
> ## As part of this task, following changes are done:
> 1. Users, Groups and Views API are integrated with swagger and exposed from 
> ambari rest api explorer ui (swagger ui) on a deployed cluster at path: 
> http://c6404.ambari.apache.org:8080/api-docs
> 2. swagger-maven-plugin is used to generate swagger.json file on compile 
> time. This file is published in web resources directory. Note that this file 
> is generated build time and will be available on deployed ambari-server host 
> at web resources location but it is not yet decided to be committed and 
> maintained in Ambari source code
> 3. swagger2markup-maven-plugin is used to generate asciidoc from swagger.json 
> file (that can be shown as markdown in github). More information about this 
> format an be found at http://asciidoc.org/ and http://asciidoctor.org/. This 
> generates files in docs/api/asciidoc/** location at build time. This 
> directory is currently intended to be committed and maintained in ambari 
> source code
> 4. swagger-ui (version: v2.1.1-M2) compiled code with the different css skin 
> (adopted from [link|https://github.com/jensoleg/swagger-ui]) is committed to 
> ambari-web/api-docs directory with certain modification to make it work with 
> ambari api. Further ui polishing will be done in subsequent tasks. Also there 
> is a strong possibility to maintain the fork code of swagger-ui and compile 
> (minify and concanate) it during ambari compile time rather than directly 
> using swagger-ui dist files. Doing so will help developers when customization 
> done over swagger-ui will increase
> 5. swagger-annotation expects application to define schema of request body 
> and response for each endpoint to be encapsulated in a class. While Ambari 
> follows this pattern for some of the endpoint, there are many others which 
> does not do so. For The ones which do not does so, new request and response 
> classes were defined. Going forward at the completion of this epic, either 
> each resource type or each resource provider should be coupled with a 
> resource response class and a resource request class. As part of this patch, 
> each resourceprovider worked upon introduces a new method "getResponse". At 
> completion of this epic ResourceProvider interface should also declare 
> methods like "getResponse" and "getRequest" that returns response schema and 
> request schema instances for the resource API endpoints
> 6. Currently it seems that swagger has a limitation in supporting  
> "subresource locator methods". This issue is been reported to swagger 
> community and is being tracked at 
> [link|https://github.com/swagger-api/swagger-core/issues/2136]. As a result 
> of which currently as a temporary workaround, all subresources are converted 
> to root resources. Also all root resources on similar path are moved under 
> same subpackages (like userApi, groupApi and viewApi).
> 
> 
> Diffs
> -
> 
>   LICENSE.txt f05016f 
>   ambari-server/docs/api/asciidoc/definitions.adoc PRE-CREATION 
>   ambari-server/docs/api/asciidoc/overview.adoc PRE-CREATION 
>   ambari-server/docs/api/asciidoc/paths.adoc PRE-CREATION 
>   ambari-server/pom.xml f0c73e4 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/ActiveWidgetLayoutService.java
>  40cd6e0 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/GroupPrivilegeService.java
>  9c1f1a3 
>   
>

Re: Review Request 57582: Create a prototype of ambari-server swagger integration

2017-03-20 Thread Jaimin Jetly 


> On March 15, 2017, 8:31 p.m., Oliver Szabo wrote:
> > ambari-web/api-docs/index.html
> > Lines 1 (patched)
> > 
> >
> > Instead of storing all of html + css files from swagger in the source 
> > code, I think its better idea to use swagger ui webjar: 
> > 
> > https://mvnrepository.com/artifact/org.webjars/swagger-ui (i think the 
> > exact swagger-ui version is not there that you used, but newer one can work)
> > 
> > I used that for Log Search. (you can unpack that during the build or 
> > use resource files from the jar somehow else)
> > 
> > If you will need any custom files you can override those files during 
> > the build. In Log Search i needed a specific index.html which supports 
> > basic authentication instead of api key, so i changed the index.html 
> > (https://github.com/apache/ambari/blob/trunk/ambari-logsearch/ambari-logsearch-portal/src/main/resources/swagger/swagger.html),
> >  but for every other file is used from the webjar, so i did not need to 
> > store swagger html+css in the source code.
> 
> Oliver Szabo wrote:
> additon: you can drop my issue, if for using the modified swagger-ui, its 
> not enough just include the some new files (extra css files etc.) from 
> https://github.com/jensoleg/swagger-ui

Since ambari-web using custom look on top of swagger-ui, ambari-web will need 
to maintain it's override of html+css file. 
But swagger-ui JS code is something we want to use from 
https://www.npmjs.com/package/swagger-ui. There is an existing task 
(https://issues.apache.org/jira/browse/AMBARI-20508) to do analysis on how much 
of the code we can use from the vendor library instead of maintaining all in 
ambari-web source code.


- Jaimin


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57582/#review169051
---


On March 20, 2017, 6:20 p.m., Jaimin Jetly wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/57582/
> ---
> 
> (Updated March 20, 2017, 6:20 p.m.)
> 
> 
> Review request for Ambari, Jonathan Hurley, Jayush Luniya, Nate Cole, Oliver 
> Szabo, Robert Nettleton, Sumit Mohanty, Sid Wagle, and Yusaku Sako.
> 
> 
> Bugs: AMBARI-20436
> https://issues.apache.org/jira/browse/AMBARI-20436
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> A separate branch is created for this work: ambari-rest-api-explorer
> 
> ## As part of this task, following changes are done:
> 1. Users, Groups and Views API are integrated with swagger and exposed from 
> ambari rest api explorer ui (swagger ui) on a deployed cluster at path: 
> http://c6404.ambari.apache.org:8080/api-docs
> 2. swagger-maven-plugin is used to generate swagger.json file on compile 
> time. This file is published in web resources directory. Note that this file 
> is generated build time and will be available on deployed ambari-server host 
> at web resources location but it is not yet decided to be committed and 
> maintained in Ambari source code
> 3. swagger2markup-maven-plugin is used to generate asciidoc from swagger.json 
> file (that can be shown as markdown in github). More information about this 
> format an be found at http://asciidoc.org/ and http://asciidoctor.org/. This 
> generates files in docs/api/asciidoc/** location at build time. This 
> directory is currently intended to be committed and maintained in ambari 
> source code
> 4. swagger-ui (version: v2.1.1-M2) compiled code with the different css skin 
> (adopted from [link|https://github.com/jensoleg/swagger-ui]) is committed to 
> ambari-web/api-docs directory with certain modification to make it work with 
> ambari api. Further ui polishing will be done in subsequent tasks. Also there 
> is a strong possibility to maintain the fork code of swagger-ui and compile 
> (minify and concanate) it during ambari compile time rather than directly 
> using swagger-ui dist files. Doing so will help developers when customization 
> done over swagger-ui will increase
> 5. swagger-annotation expects application to define schema of request body 
> and response for each endpoint to be encapsulated in a class. While Ambari 
> follows this pattern for some of the endpoint, there are many others which 
> does not do so. For The ones which do not does so, new request and response 
> classes were defined. Going forward at the completion of this epic, either 
> each resource type or each resource provider should be coupled with a 
> resource response class and a resource request class. As part of this patch, 
> each resourceprovider worked upon introduces a new method "getResponse". At 
> completion of this epic ResourceProvider

Re: Review Request 57582: Create a prototype of ambari-server swagger integration

2017-03-20 Thread Jaimin Jetly 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57582/
---

(Updated March 20, 2017, 6:20 p.m.)


Review request for Ambari, Jonathan Hurley, Jayush Luniya, Nate Cole, Oliver 
Szabo, Robert Nettleton, Sumit Mohanty, Sid Wagle, and Yusaku Sako.


Changes
---

Addresses the concerns raised in review comments


Bugs: AMBARI-20436
https://issues.apache.org/jira/browse/AMBARI-20436


Repository: ambari


Description
---

A separate branch is created for this work: ambari-rest-api-explorer

## As part of this task, following changes are done:
1. Users, Groups and Views API are integrated with swagger and exposed from 
ambari rest api explorer ui (swagger ui) on a deployed cluster at path: 
http://c6404.ambari.apache.org:8080/api-docs
2. swagger-maven-plugin is used to generate swagger.json file on compile time. 
This file is published in web resources directory. Note that this file is 
generated build time and will be available on deployed ambari-server host at 
web resources location but it is not yet decided to be committed and maintained 
in Ambari source code
3. swagger2markup-maven-plugin is used to generate asciidoc from swagger.json 
file (that can be shown as markdown in github). More information about this 
format an be found at http://asciidoc.org/ and http://asciidoctor.org/. This 
generates files in docs/api/asciidoc/** location at build time. This directory 
is currently intended to be committed and maintained in ambari source code
4. swagger-ui (version: v2.1.1-M2) compiled code with the different css skin 
(adopted from [link|https://github.com/jensoleg/swagger-ui]) is committed to 
ambari-web/api-docs directory with certain modification to make it work with 
ambari api. Further ui polishing will be done in subsequent tasks. Also there 
is a strong possibility to maintain the fork code of swagger-ui and compile 
(minify and concanate) it during ambari compile time rather than directly using 
swagger-ui dist files. Doing so will help developers when customization done 
over swagger-ui will increase
5. swagger-annotation expects application to define schema of request body and 
response for each endpoint to be encapsulated in a class. While Ambari follows 
this pattern for some of the endpoint, there are many others which does not do 
so. For The ones which do not does so, new request and response classes were 
defined. Going forward at the completion of this epic, either each resource 
type or each resource provider should be coupled with a resource response class 
and a resource request class. As part of this patch, each resourceprovider 
worked upon introduces a new method "getResponse". At completion of this epic 
ResourceProvider interface should also declare methods like "getResponse" and 
"getRequest" that returns response schema and request schema instances for the 
resource API endpoints
6. Currently it seems that swagger has a limitation in supporting  "subresource 
locator methods". This issue is been reported to swagger community and is being 
tracked at [link|https://github.com/swagger-api/swagger-core/issues/2136]. As a 
result of which currently as a temporary workaround, all subresources are 
converted to root resources. Also all root resources on similar path are moved 
under same subpackages (like userApi, groupApi and viewApi).


Diffs (updated)
-

  LICENSE.txt f05016f 
  ambari-server/docs/api/asciidoc/definitions.adoc PRE-CREATION 
  ambari-server/docs/api/asciidoc/overview.adoc PRE-CREATION 
  ambari-server/docs/api/asciidoc/paths.adoc PRE-CREATION 
  ambari-server/pom.xml f0c73e4 
  
ambari-server/src/main/java/org/apache/ambari/server/api/services/ActiveWidgetLayoutService.java
 40cd6e0 
  
ambari-server/src/main/java/org/apache/ambari/server/api/services/GroupPrivilegeService.java
 9c1f1a3 
  
ambari-server/src/main/java/org/apache/ambari/server/api/services/GroupService.java
 8ec0097 
  
ambari-server/src/main/java/org/apache/ambari/server/api/services/MemberService.java
 28e53e6 
  
ambari-server/src/main/java/org/apache/ambari/server/api/services/UserAuthorizationService.java
 26a7107 
  
ambari-server/src/main/java/org/apache/ambari/server/api/services/UserPrivilegeService.java
 f9c95e7 
  
ambari-server/src/main/java/org/apache/ambari/server/api/services/UserService.java
 31f3a8c 
  
ambari-server/src/main/java/org/apache/ambari/server/api/services/ViewDataMigrationService.java
 2a9aa64 
  
ambari-server/src/main/java/org/apache/ambari/server/api/services/ViewExternalSubResourceService.java
 66ccae7 
  
ambari-server/src/main/java/org/apache/ambari/server/api/services/ViewInstanceService.java
 53d4918 
  
ambari-server/src/main/java/org/apache/ambari/server/api/services/ViewPermissionService.java
 8f7f4ef

Re: Review Request 57781: Configuration Markdown Not Property Parsed By Some Editors

2017-03-20 Thread Robert Levas 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57781/#review169475
---


Ship it!




Ship It!

- Robert Levas


On March 20, 2017, 12:13 p.m., Jonathan Hurley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/57781/
> ---
> 
> (Updated March 20, 2017, 12:13 p.m.)
> 
> 
> Review request for Ambari, Nate Cole and Robert Levas.
> 
> 
> Bugs: AMBARI-20502
> https://issues.apache.org/jira/browse/AMBARI-20502
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Some markdown displays, like GitHub, don't properly display the configuration 
> markdown. Sections appear as:
> 
> {code}
> Alerts & Notifications | Property Name | 10 Hosts | ~50 Hosts | ~100 
> Hosts | 500+ Hosts | | --- | --- | --- | --- | --- | | 
> alerts.execution.scheduler.threadpool.size.core | 2 | 2 | 4 | 4 | | 
> alerts.execution.scheduler.threadpool.size.max | 2 | 2 | 8 | 8 | | 
> alerts.execution.scheduler.threadpool.worker.size | 400 | 2000 | 4000 | 2 
> | | alerts.cache.enabled | false | false | false | true | | 
> alerts.cache.flush.interval | 10 | 10 | 10 | 10 | | alerts.cache.size | 5 
> | 5 | 10 | 10 | Jetty API & Agent Thread Pools | Property 
> Name | 10 Hosts | ~50 Hosts | ~100 Hosts | 500+ Hosts | | --- | --- | --- | 
> --- | --- | | client.threadpool.size.max | 25 | 35 | 50 | 65 | | 
> agent.threadpool.size.max | 25 | 35 | 75 | 100 |
> {code}
> 
> 
> Diffs
> -
> 
>   ambari-server/docs/configuration/index.md 7a7f3b9 
>   
> ambari-server/src/main/java/org/apache/ambari/server/configuration/Configuration.java
>  20431f6 
> 
> 
> Diff: https://reviews.apache.org/r/57781/diff/1/
> 
> 
> Testing
> ---
> 
> Created a feature branch on Github to test this change. Verified that the 
> Markdown has been corrected.
> 
> 
> Thanks,
> 
> Jonathan Hurley
> 
>

Re: Review Request 57494: Description incorrect for the method ExtensionModeule

2017-03-20 Thread Alejandro Fernandez 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57494/#review169474
---



Please close this review.

- Alejandro Fernandez


On March 12, 2017, 2:21 a.m., zhangxiaolu zhangxiaolu wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/57494/
> ---
> 
> (Updated March 12, 2017, 2:21 a.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Andrew Onischuk, Dmytro Sen, 
> Jaimin Jetly, Srimanth Gunturi, Sid Wagle, and Vitalyi Brodetskyi.
> 
> 
> Bugs: AMBARI-18259
> https://issues.apache.org/jira/browse/AMBARI-18259
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> ExtensionModeule's description is not corrent
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/ExtensionModule.java
>  e3f8d0d 
> 
> 
> Diff: https://reviews.apache.org/r/57494/diff/1/
> 
> 
> Testing
> ---
> 
> done it
> 
> 
> Thanks,
> 
> zhangxiaolu zhangxiaolu
> 
>

Re: Review Request 57785: Ambari-server clean up

2017-03-20 Thread Sumit Mohanty 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57785/#review169472
---


Ship it!




Ship It!

- Sumit Mohanty


On March 20, 2017, 5:35 p.m., Vitalyi Brodetskyi wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/57785/
> ---
> 
> (Updated March 20, 2017, 5:35 p.m.)
> 
> 
> Review request for Ambari, Dmytro Sen and Sumit Mohanty.
> 
> 
> Bugs: AMBARI-20506
> https://issues.apache.org/jira/browse/AMBARI-20506
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> clean up
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/security/CertificateManager.java
>  c9e7c6e 
>   
> ambari-server/src/test/java/org/apache/ambari/server/security/CertGenerationTest.java
>  4e0549d 
> 
> 
> Diff: https://reviews.apache.org/r/57785/diff/1/
> 
> 
> Testing
> ---
> 
> mvn clean test
> 
> 
> Thanks,
> 
> Vitalyi Brodetskyi
> 
>

Review Request 57785: Ambari-server clean up

2017-03-20 Thread Vitalyi Brodetskyi 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57785/
---

Review request for Ambari, Dmytro Sen and Sumit Mohanty.


Bugs: AMBARI-20506
https://issues.apache.org/jira/browse/AMBARI-20506


Repository: ambari


Description
---

clean up


Diffs
-

  
ambari-server/src/main/java/org/apache/ambari/server/security/CertificateManager.java
 c9e7c6e 
  
ambari-server/src/test/java/org/apache/ambari/server/security/CertGenerationTest.java
 4e0549d 


Diff: https://reviews.apache.org/r/57785/diff/1/


Testing
---

mvn clean test


Thanks,

Vitalyi Brodetskyi

Re: Review Request 57451: The Installation of hbase-client failed

2017-03-20 Thread Alejandro Fernandez 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57451/#review169470
---




ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/hbase.py
Lines 59 (patched)


Python unit tests are failing.
Please see comment in Jira


- Alejandro Fernandez


On March 16, 2017, 12:51 a.m., zhangxiaolu zhangxiaolu wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/57451/
> ---
> 
> (Updated March 16, 2017, 12:51 a.m.)
> 
> 
> Review request for Ambari, Dmytro Sen, Jaimin Jetly, Srimanth Gunturi, Sid 
> Wagle, and Yusaku Sako.
> 
> 
> Bugs: AMBARI-20368
> https://issues.apache.org/jira/browse/AMBARI-20368
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> The problem is caused by the wrong installation order, which means 
> hbase-client will be installed before the installation of hdfs-client. Under 
> such circumstance, installation of hdfs-client will fail because the 
> hadoop-conf-dir does not exist. 
> Two solutions can be provided to solve this problem:
> 1. Creating the hadoop-conf-dir in hbase.py;
> 2. Setting correct installation order in role_command_order.json;
> Considering that hdfs-client may not beinstalled on the node especially when 
> adding new host, therefore, in my opinion, the second choice would be more 
> appropriate.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/hbase.py
>  20894ec 
> 
> 
> Diff: https://reviews.apache.org/r/57451/diff/1/
> 
> 
> Testing
> ---
> 
> done it
> 
> 
> Thanks,
> 
> zhangxiaolu zhangxiaolu
> 
>

Re: Review Request 57782: Ambari-agent log rotation is broken

2017-03-20 Thread Alejandro Fernandez 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57782/#review169469
---


Ship it!




Ship It!

- Alejandro Fernandez


On March 20, 2017, 4:33 p.m., Eugene Chekanskiy wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/57782/
> ---
> 
> (Updated March 20, 2017, 4:33 p.m.)
> 
> 
> Review request for Ambari, Andrew Onischuk, Dmitro Lisnichenko, and Vitalyi 
> Brodetskyi.
> 
> 
> Bugs: AMBARI-20504
> https://issues.apache.org/jira/browse/AMBARI-20504
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Ambari-agent log rotation broken due to multiple RotatingFileHandler on same 
> file.
> 
> 
> Diffs
> -
> 
>   ambari-agent/src/main/python/ambari_agent/main.py 604b234 
> 
> 
> Diff: https://reviews.apache.org/r/57782/diff/1/
> 
> 
> Testing
> ---
> 
> mvn clean test
> 
> 
> Thanks,
> 
> Eugene Chekanskiy
> 
>

Re: Review Request 57781: Configuration Markdown Not Property Parsed By Some Editors

2017-03-20 Thread Alejandro Fernandez 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57781/#review169467
---


Ship it!




Typo in review title, "Properly"

- Alejandro Fernandez


On March 20, 2017, 4:13 p.m., Jonathan Hurley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/57781/
> ---
> 
> (Updated March 20, 2017, 4:13 p.m.)
> 
> 
> Review request for Ambari, Nate Cole and Robert Levas.
> 
> 
> Bugs: AMBARI-20502
> https://issues.apache.org/jira/browse/AMBARI-20502
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Some markdown displays, like GitHub, don't properly display the configuration 
> markdown. Sections appear as:
> 
> {code}
> Alerts & Notifications | Property Name | 10 Hosts | ~50 Hosts | ~100 
> Hosts | 500+ Hosts | | --- | --- | --- | --- | --- | | 
> alerts.execution.scheduler.threadpool.size.core | 2 | 2 | 4 | 4 | | 
> alerts.execution.scheduler.threadpool.size.max | 2 | 2 | 8 | 8 | | 
> alerts.execution.scheduler.threadpool.worker.size | 400 | 2000 | 4000 | 2 
> | | alerts.cache.enabled | false | false | false | true | | 
> alerts.cache.flush.interval | 10 | 10 | 10 | 10 | | alerts.cache.size | 5 
> | 5 | 10 | 10 | Jetty API & Agent Thread Pools | Property 
> Name | 10 Hosts | ~50 Hosts | ~100 Hosts | 500+ Hosts | | --- | --- | --- | 
> --- | --- | | client.threadpool.size.max | 25 | 35 | 50 | 65 | | 
> agent.threadpool.size.max | 25 | 35 | 75 | 100 |
> {code}
> 
> 
> Diffs
> -
> 
>   ambari-server/docs/configuration/index.md 7a7f3b9 
>   
> ambari-server/src/main/java/org/apache/ambari/server/configuration/Configuration.java
>  20431f6 
> 
> 
> Diff: https://reviews.apache.org/r/57781/diff/1/
> 
> 
> Testing
> ---
> 
> Created a feature branch on Github to test this change. Verified that the 
> Markdown has been corrected.
> 
> 
> Thanks,
> 
> Jonathan Hurley
> 
>

Re: Review Request 57782: Ambari-agent log rotation is broken

2017-03-20 Thread Dmytro Sen 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57782/#review169458
---


Ship it!




Ship It!

- Dmytro Sen


On Март 20, 2017, 4:33 п.п., Eugene Chekanskiy wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/57782/
> ---
> 
> (Updated Март 20, 2017, 4:33 п.п.)
> 
> 
> Review request for Ambari, Andrew Onischuk, Dmitro Lisnichenko, and Vitalyi 
> Brodetskyi.
> 
> 
> Bugs: AMBARI-20504
> https://issues.apache.org/jira/browse/AMBARI-20504
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Ambari-agent log rotation broken due to multiple RotatingFileHandler on same 
> file.
> 
> 
> Diffs
> -
> 
>   ambari-agent/src/main/python/ambari_agent/main.py 604b234 
> 
> 
> Diff: https://reviews.apache.org/r/57782/diff/1/
> 
> 
> Testing
> ---
> 
> mvn clean test
> 
> 
> Thanks,
> 
> Eugene Chekanskiy
> 
>

Re: Review Request 57771: Unlimited Key JCE policy is enabled or not on each host

2017-03-20 Thread Robert Levas 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57771/#review169456
---


Ship it!




Thanks for making those changes.

- Robert Levas


On March 20, 2017, 12:10 p.m., Attila Magyar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/57771/
> ---
> 
> (Updated March 20, 2017, 12:10 p.m.)
> 
> 
> Review request for Ambari, Attila Doroszlai, Jaimin Jetly, Robert Levas, and 
> Sebastian Toader.
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> In many cases support engineers want to look at if Unlimited Strength Java 
> Cryptography Extension is installed on all hosts quickly and easily.
> 
> This information is now visible on the host summary page, and can be accessed 
> via api call: /api/v1/hosts?fields=Hosts/last_agent_env.
> 
> 
> Diffs
> -
> 
>   ambari-agent/src/main/python/ambari_agent/HostInfo.py 7f041eb 
>   ambari-agent/src/test/python/ambari_agent/TestHostInfo.py 67294e0 
>   
> ambari-common/src/main/python/resource_management/core/resources/jcepolicyinfo.py
>  a4e4f6e 
>   ambari-server/src/main/java/org/apache/ambari/server/agent/AgentEnv.java 
> 64d0865 
>   
> ambari-server/src/main/java/org/apache/ambari/server/configuration/Configuration.java
>  20431f6 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-START/scripts/shared_initialization.py
>  3c36962 
>   
> ambari-server/src/main/resources/stacks/HDP/3.0/hooks/before-START/scripts/shared_initialization.py
>  71161e0 
>   
> ambari-server/src/test/java/org/apache/ambari/server/agent/AgentResourceTest.java
>  3f62366 
>   ambari-web/app/controllers/global/update_controller.js 59eaf3e 
>   ambari-web/app/mappers/hosts_mapper.js e4dd08f 
>   ambari-web/app/messages.js c9926e6 
>   ambari-web/app/models/host.js 1b7b326 
>   ambari-web/app/templates/main/host/summary.hbs 5d0b09c 
> 
> 
> Diff: https://reviews.apache.org/r/57771/diff/2/
> 
> 
> Testing
> ---
> 
> added new unittests, tested the UI manually
> 
> existing tests ran successfully with one irrelevant exception:
> 
> ==
> FAIL: test_get_params_from_filesystem 
> (TestSecurityCommons.TestSecurityCommons)
> --
> Traceback (most recent call last):
>   File 
> "/Users/amagyar/development/ambari/ambari-common/src/test/python/mock/mock.py",
>  line 1199, in patched
> return func(*args, **keywargs)
>   File 
> "/Users/amagyar/development/ambari/ambari-agent/src/test/python/resource_management/TestSecurityCommons.py",
>  line 203, in test_get_params_from_filesystem
> et_parser_mock.assert_called_with(conf_dir + os.sep + "config.xml")
>   File 
> "/Users/amagyar/development/ambari/ambari-common/src/test/python/mock/mock.py",
>  line 829, in assert_called_with
> raise AssertionError('Expected call: %s\nNot called' % (expected,))
> AssertionError: Expected call: 
> parse('/var/folders/hp/5rnr2yx14q7csg8xbtp_mkr8gn/T/config.xml')
> Not called
> 
> --
> Ran 465 tests in 16.659s
> 
> 
> Thanks,
> 
> Attila Magyar
> 
>

Review Request 57782: Ambari-agent log rotation is broken

2017-03-20 Thread Eugene Chekanskiy 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57782/
---

Review request for Ambari, Andrew Onischuk, Dmitro Lisnichenko, and Vitalyi 
Brodetskyi.


Bugs: AMBARI-20504
https://issues.apache.org/jira/browse/AMBARI-20504


Repository: ambari


Description
---

Ambari-agent log rotation broken due to multiple RotatingFileHandler on same 
file.


Diffs
-

  ambari-agent/src/main/python/ambari_agent/main.py 604b234 


Diff: https://reviews.apache.org/r/57782/diff/1/


Testing
---

mvn clean test


Thanks,

Eugene Chekanskiy

Re: Review Request 57776: Hive View2.0: Upload table fails, if one column is deleted

2017-03-20 Thread Gaurav Nagar 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57776/#review169455
---


Ship it!




Ship It!

- Gaurav Nagar


On March 20, 2017, 2:12 p.m., Pallav Kulshreshtha wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/57776/
> ---
> 
> (Updated March 20, 2017, 2:12 p.m.)
> 
> 
> Review request for Ambari, Abhishek Kumar, DIPAYAN BHOWMICK, Gaurav Nagar, 
> Nitiraj Rathore, Rohit Choudhary, and venkat sairam.
> 
> 
> Bugs: AMBARI-20497
> https://issues.apache.org/jira/browse/AMBARI-20497
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Added a var 'fromUploadTable' as true when it comes from upload table 
> otherwise false.
> 
> 
> Diffs
> -
> 
>   
> contrib/views/hive20/src/main/resources/ui/app/routes/databases/database/tables/upload-table.js
>  e6b71fe 
>   
> contrib/views/hive20/src/main/resources/ui/app/templates/components/column-item.hbs
>  762c77c 
>   
> contrib/views/hive20/src/main/resources/ui/app/templates/components/create-table.hbs
>  f369783 
>   
> contrib/views/hive20/src/main/resources/ui/app/templates/components/table-columns.hbs
>  b942136 
>   
> contrib/views/hive20/src/main/resources/ui/app/templates/components/upload-table.hbs
>  0ee6b81 
>   
> contrib/views/hive20/src/main/resources/ui/app/templates/databases/database/tables/upload-table.hbs
>  0091ede 
> 
> 
> Diff: https://reviews.apache.org/r/57776/diff/2/
> 
> 
> Testing
> ---
> 
> manually test.
> 
> 
> Thanks,
> 
> Pallav Kulshreshtha
> 
>

Review Request 57781: Configuration Markdown Not Property Parsed By Some Editors

2017-03-20 Thread Jonathan Hurley 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57781/
---

Review request for Ambari, Nate Cole and Robert Levas.


Bugs: AMBARI-20502
https://issues.apache.org/jira/browse/AMBARI-20502


Repository: ambari


Description
---

Some markdown displays, like GitHub, don't properly display the configuration 
markdown. Sections appear as:

{code}
Alerts & Notifications | Property Name | 10 Hosts | ~50 Hosts | ~100 Hosts 
| 500+ Hosts | | --- | --- | --- | --- | --- | | 
alerts.execution.scheduler.threadpool.size.core | 2 | 2 | 4 | 4 | | 
alerts.execution.scheduler.threadpool.size.max | 2 | 2 | 8 | 8 | | 
alerts.execution.scheduler.threadpool.worker.size | 400 | 2000 | 4000 | 2 | 
| alerts.cache.enabled | false | false | false | true | | 
alerts.cache.flush.interval | 10 | 10 | 10 | 10 | | alerts.cache.size | 5 | 
5 | 10 | 10 | Jetty API & Agent Thread Pools | Property Name | 
10 Hosts | ~50 Hosts | ~100 Hosts | 500+ Hosts | | --- | --- | --- | --- | --- 
| | client.threadpool.size.max | 25 | 35 | 50 | 65 | | 
agent.threadpool.size.max | 25 | 35 | 75 | 100 |
{code}


Diffs
-

  ambari-server/docs/configuration/index.md 7a7f3b9 
  
ambari-server/src/main/java/org/apache/ambari/server/configuration/Configuration.java
 20431f6 


Diff: https://reviews.apache.org/r/57781/diff/1/


Testing
---

Created a feature branch on Github to test this change. Verified that the 
Markdown has been corrected.


Thanks,

Jonathan Hurley

Re: Review Request 57777: Python UT fail on trunk

2017-03-20 Thread Vitalyi Brodetskyi 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5/#review169447
---


Ship it!




Ship It!

- Vitalyi Brodetskyi


On Березень 20, 2017, 3:09 після полудня, Andrew Onischuk wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/5/
> ---
> 
> (Updated Березень 20, 2017, 3:09 після полудня)
> 
> 
> Review request for Ambari and Eugene Chekanskiy.
> 
> 
> Bugs: AMBARI-20500
> https://issues.apache.org/jira/browse/AMBARI-20500
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> ==
> FAIL: test_get_params_from_filesystem 
> (TestSecurityCommons.TestSecurityCommons)
> --
> Traceback (most recent call last):
>   File "/home/user/ambari/ambari-common/src/test/python/mock/mock.py", 
> line 1199, in patched
> return func(*args, **keywargs)
>   File 
> "/home/user/ambari/ambari-agent/src/test/python/resource_management/TestSecurityCommons.py",
>  line 203, in test_get_params_from_filesystem
> et_parser_mock.assert_called_with(conf_dir + os.sep + "config.xml")
>   File "/home/user/ambari/ambari-common/src/test/python/mock/mock.py", 
> line 829, in assert_called_with
> raise AssertionError('Expected call: %s\nNot called' % (expected,))
> AssertionError: Expected call: parse('/tmp/config.xml')
> Not called
> 
> 
> Diffs
> -
> 
>   ambari-agent/src/test/python/resource_management/TestSecurityCommons.py 
> b44b454 
> 
> 
> Diff: https://reviews.apache.org/r/5/diff/1/
> 
> 
> Testing
> ---
> 
> mvn clean test
> 
> 
> Thanks,
> 
> Andrew Onischuk
> 
>

Review Request 57777: Python UT fail on trunk

2017-03-20 Thread Andrew Onischuk 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5/
---

Review request for Ambari and Eugene Chekanskiy.


Bugs: AMBARI-20500
https://issues.apache.org/jira/browse/AMBARI-20500


Repository: ambari


Description
---

==
FAIL: test_get_params_from_filesystem 
(TestSecurityCommons.TestSecurityCommons)
--
Traceback (most recent call last):
  File "/home/user/ambari/ambari-common/src/test/python/mock/mock.py", line 
1199, in patched
return func(*args, **keywargs)
  File 
"/home/user/ambari/ambari-agent/src/test/python/resource_management/TestSecurityCommons.py",
 line 203, in test_get_params_from_filesystem
et_parser_mock.assert_called_with(conf_dir + os.sep + "config.xml")
  File "/home/user/ambari/ambari-common/src/test/python/mock/mock.py", line 
829, in assert_called_with
raise AssertionError('Expected call: %s\nNot called' % (expected,))
AssertionError: Expected call: parse('/tmp/config.xml')
Not called


Diffs
-

  ambari-agent/src/test/python/resource_management/TestSecurityCommons.py 
b44b454 


Diff: https://reviews.apache.org/r/5/diff/1/


Testing
---

mvn clean test


Thanks,

Andrew Onischuk

Re: Review Request 57686: AMBARI-20472 MapRed History server start does not create directories in HDFS when WEBHDFS is disabled.

2017-03-20 Thread Di Li 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57686/
---

(Updated March 20, 2017, 3:05 p.m.)


Review request for Ambari, Alejandro Fernandez and Tim Thorpe.


Bugs: AMBARI-20472
https://issues.apache.org/jira/browse/AMBARI-20472


Repository: ambari


Description
---

MapRed HistoryServer start logic (implemented in yarn.py) creates various dirs 
in HDFS. The dir creation only runs thru when WEBHDFS is enabled. When WEBHDFS 
is disabled, HdfsResource calls bailed with NPE as the following one.
Creating: Resource [source=null, target=/app-logs, type=directory, 
action=create, owner=yarn, group=hadoop, mode=777, recursiveChown=false, 
recursiveChmod=true, changePermissionforParents=false, manageIfExists=false]
Exception occurred, Reason: null
java.lang.NullPointerException
at org.apache.hadoop.fs.FileSystem.fixRelativePart(FileSystem.java:2211)
at 
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1300)
at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1426)
at org.apache.ambari.fast_hdfs_resource.Runner.main(Runner.java:80)


Diffs
-

  
contrib/fast-hdfs-resource/src/main/java/org/apache/ambari/fast_hdfs_resource/Runner.java
 e210876 


Diff: https://reviews.apache.org/r/57686/diff/2/


Testing
---

build fast-hdfs-resource jar, patch a trunk cluster with the new jar, verify 
the dirs , such as /mapred, are created when WEBHDFS is disabled.


File Attachments (updated)


fast-hdfs-resource-trunk.jar
  
https://reviews.apache.org/media/uploaded/files/2017/03/20/1ad0fe56-6efa-4d46-aa74-edf738c41a3f__fast-hdfs-resource-trunk.jar


Thanks,

Di Li

Re: Review Request 57771: Unlimited Key JCE policy is enabled or not on each host

2017-03-20 Thread Robert Levas 


> On March 20, 2017, 10:18 a.m., Robert Levas wrote:
> > ambari-agent/src/main/python/ambari_agent/HostInfo.py
> > Lines 293-298 (patched)
> > 
> >
> > Would it be possible to use 
> > `resource_management.core.resources.jcepolicyinfo.JcePolicyInfo#is_unlimited_key_jce_policy`?
> >   
> > 
> > If we keep the implementation for determining if the unlimited key JCE 
> > policy contained to one location, maintenance will be easier.
> 
> Attila Magyar wrote:
> This one uses Execute instead of shell.call and fails with this error:
> 
> Traceback (most recent call last):
>   File "/usr/lib/python2.6/site-packages/ambari_agent/HostInfo.py", line 
> 296, in checkUnlimitedJce
> logoutput=True
>   File 
> "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 
> 118, in __new__
> env = env or Environment.get_instance()
>   File 
> "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", 
> line 182, in get_instance
> raise Exception("No Environment present for retrieving for thread %s" 
> % threading.current_thread())
> 
> 
> Any insight why?
> 
> Attila Magyar wrote:
> The environment instance is stored in a thread local and this is not set 
> in the heartbeat related code.

Bummer.  Could you move your implementation into 
`resource_management.core.resources.jcepolicyinfo.JcePolicyInfo#is_unlimited_key_jce_policy`?
  Maybe other invocations of 
`resource_management.core.resources.jcepolicyinfo.JcePolicyInfo#is_unlimited_key_jce_policy`
 will suffer the same issue, and you implementation appears to avoid it.


- Robert


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57771/#review169436
---


On March 20, 2017, 8:49 a.m., Attila Magyar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/57771/
> ---
> 
> (Updated March 20, 2017, 8:49 a.m.)
> 
> 
> Review request for Ambari, Attila Doroszlai, Jaimin Jetly, Robert Levas, and 
> Sebastian Toader.
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> In many cases support engineers want to look at if Unlimited Strength Java 
> Cryptography Extension is installed on all hosts quickly and easily.
> 
> This information is now visible on the host summary page, and can be accessed 
> via api call: /api/v1/hosts?fields=Hosts/last_agent_env.
> 
> 
> Diffs
> -
> 
>   ambari-agent/src/main/python/ambari_agent/HostInfo.py 7f041eb 
>   ambari-agent/src/test/python/ambari_agent/TestHostInfo.py 67294e0 
>   ambari-server/src/main/java/org/apache/ambari/server/agent/AgentEnv.java 
> 64d0865 
>   
> ambari-server/src/main/java/org/apache/ambari/server/configuration/Configuration.java
>  20431f6 
>   
> ambari-server/src/test/java/org/apache/ambari/server/agent/AgentResourceTest.java
>  3f62366 
>   ambari-web/app/controllers/global/update_controller.js 59eaf3e 
>   ambari-web/app/mappers/hosts_mapper.js e4dd08f 
>   ambari-web/app/messages.js c9926e6 
>   ambari-web/app/models/host.js 1b7b326 
>   ambari-web/app/templates/main/host/summary.hbs 5d0b09c 
> 
> 
> Diff: https://reviews.apache.org/r/57771/diff/1/
> 
> 
> Testing
> ---
> 
> added new unittests, tested the UI manually
> 
> existing tests ran successfully with one irrelevant exception:
> 
> ==
> FAIL: test_get_params_from_filesystem 
> (TestSecurityCommons.TestSecurityCommons)
> --
> Traceback (most recent call last):
>   File 
> "/Users/amagyar/development/ambari/ambari-common/src/test/python/mock/mock.py",
>  line 1199, in patched
> return func(*args, **keywargs)
>   File 
> "/Users/amagyar/development/ambari/ambari-agent/src/test/python/resource_management/TestSecurityCommons.py",
>  line 203, in test_get_params_from_filesystem
> et_parser_mock.assert_called_with(conf_dir + os.sep + "config.xml")
>   File 
> "/Users/amagyar/development/ambari/ambari-common/src/test/python/mock/mock.py",
>  line 829, in assert_called_with
> raise AssertionError('Expected call: %s\nNot called' % (expected,))
> AssertionError: Expected call: 
> parse('/var/folders/hp/5rnr2yx14q7csg8xbtp_mkr8gn/T/config.xml')
> Not called
> 
> --
> Ran 465 tests in 16.659s
> 
> 
> Thanks,
> 
> Attila Magyar
> 
>

Re: Review Request 57771: Unlimited Key JCE policy is enabled or not on each host

2017-03-20 Thread Attila Magyar 


> On March 20, 2017, 2:18 p.m., Robert Levas wrote:
> > ambari-agent/src/main/python/ambari_agent/HostInfo.py
> > Lines 293-298 (patched)
> > 
> >
> > Would it be possible to use 
> > `resource_management.core.resources.jcepolicyinfo.JcePolicyInfo#is_unlimited_key_jce_policy`?
> >   
> > 
> > If we keep the implementation for determining if the unlimited key JCE 
> > policy contained to one location, maintenance will be easier.
> 
> Attila Magyar wrote:
> This one uses Execute instead of shell.call and fails with this error:
> 
> Traceback (most recent call last):
>   File "/usr/lib/python2.6/site-packages/ambari_agent/HostInfo.py", line 
> 296, in checkUnlimitedJce
> logoutput=True
>   File 
> "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 
> 118, in __new__
> env = env or Environment.get_instance()
>   File 
> "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", 
> line 182, in get_instance
> raise Exception("No Environment present for retrieving for thread %s" 
> % threading.current_thread())
> 
> 
> Any insight why?

The environment instance is stored in a thread local and this is not set in the 
heartbeat related code.


- Attila


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57771/#review169436
---


On March 20, 2017, 12:49 p.m., Attila Magyar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/57771/
> ---
> 
> (Updated March 20, 2017, 12:49 p.m.)
> 
> 
> Review request for Ambari, Attila Doroszlai, Jaimin Jetly, Robert Levas, and 
> Sebastian Toader.
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> In many cases support engineers want to look at if Unlimited Strength Java 
> Cryptography Extension is installed on all hosts quickly and easily.
> 
> This information is now visible on the host summary page, and can be accessed 
> via api call: /api/v1/hosts?fields=Hosts/last_agent_env.
> 
> 
> Diffs
> -
> 
>   ambari-agent/src/main/python/ambari_agent/HostInfo.py 7f041eb 
>   ambari-agent/src/test/python/ambari_agent/TestHostInfo.py 67294e0 
>   ambari-server/src/main/java/org/apache/ambari/server/agent/AgentEnv.java 
> 64d0865 
>   
> ambari-server/src/main/java/org/apache/ambari/server/configuration/Configuration.java
>  20431f6 
>   
> ambari-server/src/test/java/org/apache/ambari/server/agent/AgentResourceTest.java
>  3f62366 
>   ambari-web/app/controllers/global/update_controller.js 59eaf3e 
>   ambari-web/app/mappers/hosts_mapper.js e4dd08f 
>   ambari-web/app/messages.js c9926e6 
>   ambari-web/app/models/host.js 1b7b326 
>   ambari-web/app/templates/main/host/summary.hbs 5d0b09c 
> 
> 
> Diff: https://reviews.apache.org/r/57771/diff/1/
> 
> 
> Testing
> ---
> 
> added new unittests, tested the UI manually
> 
> existing tests ran successfully with one irrelevant exception:
> 
> ==
> FAIL: test_get_params_from_filesystem 
> (TestSecurityCommons.TestSecurityCommons)
> --
> Traceback (most recent call last):
>   File 
> "/Users/amagyar/development/ambari/ambari-common/src/test/python/mock/mock.py",
>  line 1199, in patched
> return func(*args, **keywargs)
>   File 
> "/Users/amagyar/development/ambari/ambari-agent/src/test/python/resource_management/TestSecurityCommons.py",
>  line 203, in test_get_params_from_filesystem
> et_parser_mock.assert_called_with(conf_dir + os.sep + "config.xml")
>   File 
> "/Users/amagyar/development/ambari/ambari-common/src/test/python/mock/mock.py",
>  line 829, in assert_called_with
> raise AssertionError('Expected call: %s\nNot called' % (expected,))
> AssertionError: Expected call: 
> parse('/var/folders/hp/5rnr2yx14q7csg8xbtp_mkr8gn/T/config.xml')
> Not called
> 
> --
> Ran 465 tests in 16.659s
> 
> 
> Thanks,
> 
> Attila Magyar
> 
>

Re: Review Request 57771: Unlimited Key JCE policy is enabled or not on each host

2017-03-20 Thread Attila Magyar 


> On March 20, 2017, 2:18 p.m., Robert Levas wrote:
> > ambari-agent/src/main/python/ambari_agent/HostInfo.py
> > Lines 293-298 (patched)
> > 
> >
> > Would it be possible to use 
> > `resource_management.core.resources.jcepolicyinfo.JcePolicyInfo#is_unlimited_key_jce_policy`?
> >   
> > 
> > If we keep the implementation for determining if the unlimited key JCE 
> > policy contained to one location, maintenance will be easier.

This one uses Execute instead of shell.call and fails with this error:

Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/ambari_agent/HostInfo.py", line 296, 
in checkUnlimitedJce
logoutput=True
  File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", 
line 118, in __new__
env = env or Environment.get_instance()
  File 
"/usr/lib/python2.6/site-packages/resource_management/core/environment.py", 
line 182, in get_instance
raise Exception("No Environment present for retrieving for thread %s" % 
threading.current_thread())


Any insight why?


- Attila


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57771/#review169436
---


On March 20, 2017, 12:49 p.m., Attila Magyar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/57771/
> ---
> 
> (Updated March 20, 2017, 12:49 p.m.)
> 
> 
> Review request for Ambari, Attila Doroszlai, Jaimin Jetly, Robert Levas, and 
> Sebastian Toader.
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> In many cases support engineers want to look at if Unlimited Strength Java 
> Cryptography Extension is installed on all hosts quickly and easily.
> 
> This information is now visible on the host summary page, and can be accessed 
> via api call: /api/v1/hosts?fields=Hosts/last_agent_env.
> 
> 
> Diffs
> -
> 
>   ambari-agent/src/main/python/ambari_agent/HostInfo.py 7f041eb 
>   ambari-agent/src/test/python/ambari_agent/TestHostInfo.py 67294e0 
>   ambari-server/src/main/java/org/apache/ambari/server/agent/AgentEnv.java 
> 64d0865 
>   
> ambari-server/src/main/java/org/apache/ambari/server/configuration/Configuration.java
>  20431f6 
>   
> ambari-server/src/test/java/org/apache/ambari/server/agent/AgentResourceTest.java
>  3f62366 
>   ambari-web/app/controllers/global/update_controller.js 59eaf3e 
>   ambari-web/app/mappers/hosts_mapper.js e4dd08f 
>   ambari-web/app/messages.js c9926e6 
>   ambari-web/app/models/host.js 1b7b326 
>   ambari-web/app/templates/main/host/summary.hbs 5d0b09c 
> 
> 
> Diff: https://reviews.apache.org/r/57771/diff/1/
> 
> 
> Testing
> ---
> 
> added new unittests, tested the UI manually
> 
> existing tests ran successfully with one irrelevant exception:
> 
> ==
> FAIL: test_get_params_from_filesystem 
> (TestSecurityCommons.TestSecurityCommons)
> --
> Traceback (most recent call last):
>   File 
> "/Users/amagyar/development/ambari/ambari-common/src/test/python/mock/mock.py",
>  line 1199, in patched
> return func(*args, **keywargs)
>   File 
> "/Users/amagyar/development/ambari/ambari-agent/src/test/python/resource_management/TestSecurityCommons.py",
>  line 203, in test_get_params_from_filesystem
> et_parser_mock.assert_called_with(conf_dir + os.sep + "config.xml")
>   File 
> "/Users/amagyar/development/ambari/ambari-common/src/test/python/mock/mock.py",
>  line 829, in assert_called_with
> raise AssertionError('Expected call: %s\nNot called' % (expected,))
> AssertionError: Expected call: 
> parse('/var/folders/hp/5rnr2yx14q7csg8xbtp_mkr8gn/T/config.xml')
> Not called
> 
> --
> Ran 465 tests in 16.659s
> 
> 
> Thanks,
> 
> Attila Magyar
> 
>

Re: Review Request 53686: Stage and Request status should be persisted in the database

2017-03-20 Thread Nate Cole 


> On March 15, 2017, 11:20 a.m., Nate Cole wrote:
> > It looks like this was pushed to trunk - can it be closed?

Ping, can this be closed?


- Nate


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53686/#review169012
---


On Feb. 6, 2017, 6:09 p.m., Jaimin Jetly wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/53686/
> ---
> 
> (Updated Feb. 6, 2017, 6:09 p.m.)
> 
> 
> Review request for Ambari, Jonathan Hurley, Nate Cole, Sumit Mohanty, and Sid 
> Wagle.
> 
> 
> Bugs: AMBARI-18868
> https://issues.apache.org/jira/browse/AMBARI-18868
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Stage and Request status should be persisted in the database.
> 
> upgrading to ambari-3.0.0 should add status for all present stages and 
> request for the cluster.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/actionmanager/ActionDBAccessorImpl.java
>  7837a7b 
>   
> ambari-server/src/main/java/org/apache/ambari/server/actionmanager/ActionScheduler.java
>  dabcb98 
>   
> ambari-server/src/main/java/org/apache/ambari/server/actionmanager/Request.java
>  31e11c1 
>   
> ambari-server/src/main/java/org/apache/ambari/server/actionmanager/Stage.java 
> 4a05b32 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/CalculatedStatus.java
>  3c415df 
>   
> ambari-server/src/main/java/org/apache/ambari/server/events/TaskCreateEvent.java
>  PRE-CREATION 
>   ambari-server/src/main/java/org/apache/ambari/server/events/TaskEvent.java 
> PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/events/TaskUpdateEvent.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/events/listeners/tasks/TaskStatusListener.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/events/publishers/TaskEventPublisher.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/orm/dao/HostRoleCommandDAO.java
>  02c4091 
>   
> ambari-server/src/main/java/org/apache/ambari/server/orm/dao/RequestDAO.java 
> 1c4d0a3 
>   ambari-server/src/main/java/org/apache/ambari/server/orm/dao/StageDAO.java 
> d2f899f 
>   
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/HostRoleCommandEntity.java
>  74271b9 
>   
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/RequestEntity.java
>  7944d21 
>   
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/StageEntity.java
>  f9c8810 
>   
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/StageEntityPK.java
>  9ca0470 
>   
> ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog300.java
>  4f90ef3 
>   ambari-server/src/main/resources/Ambari-DDL-Derby-CREATE.sql b79c945 
>   ambari-server/src/main/resources/Ambari-DDL-MySQL-CREATE.sql 1c502bc 
>   ambari-server/src/main/resources/Ambari-DDL-Oracle-CREATE.sql c6d4ad0 
>   ambari-server/src/main/resources/Ambari-DDL-Postgres-CREATE.sql 1be87bb 
>   ambari-server/src/main/resources/Ambari-DDL-SQLAnywhere-CREATE.sql abe48e8 
>   ambari-server/src/main/resources/Ambari-DDL-SQLServer-CREATE.sql 169a464 
>   
> ambari-server/src/test/java/org/apache/ambari/server/actionmanager/TestActionDBAccessorImpl.java
>  177ac70 
>   
> ambari-server/src/test/java/org/apache/ambari/server/actionmanager/TestActionScheduler.java
>  ade625a 
>   
> ambari-server/src/test/java/org/apache/ambari/server/alerts/AmbariPerformanceRunnableTest.java
>  7b1a5a2 
>   
> ambari-server/src/test/java/org/apache/ambari/server/controller/internal/UpgradeResourceProviderTest.java
>  a0701b6 
>   
> ambari-server/src/test/java/org/apache/ambari/server/controller/internal/UpgradeSummaryResourceProviderTest.java
>  619e367 
>   
> ambari-server/src/test/java/org/apache/ambari/server/events/listeners/tasks/TaskStatusListenerTest.java
>  PRE-CREATION 
>   
> ambari-server/src/test/java/org/apache/ambari/server/state/ConfigHelperTest.java
>  b1c10f5 
>   
> ambari-server/src/test/java/org/apache/ambari/server/state/cluster/ClusterEffectiveVersionTest.java
>  9d339e2 
>   
> ambari-server/src/test/java/org/apache/ambari/server/state/services/RetryUpgradeActionServiceTest.java
>  ed95b0b 
>   
> ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog300Test.java
>  d7979e8 
> 
> 
> Diff: https://reviews.apache.org/r/53686/diff/8/
> 
> 
> Testing
> ---
> 
> Verified manually on a cluster by making api requests and upgrading ambari.
> Add unit tests.
> Verified that the patch does not break any existing unit tests on dev box. 
> Jenkins job overall unit test result pending..
> Verified on 1000 node cluster

Re: Review Request 57771: Unlimited Key JCE policy is enabled or not on each host

2017-03-20 Thread Robert Levas 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57771/#review169436
---


Fix it, then Ship it!





ambari-agent/src/main/python/ambari_agent/HostInfo.py
Lines 293-298 (patched)


Would it be possible to use 
`resource_management.core.resources.jcepolicyinfo.JcePolicyInfo#is_unlimited_key_jce_policy`?
  

If we keep the implementation for determining if the unlimited key JCE 
policy contained to one location, maintenance will be easier.


- Robert Levas


On March 20, 2017, 8:49 a.m., Attila Magyar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/57771/
> ---
> 
> (Updated March 20, 2017, 8:49 a.m.)
> 
> 
> Review request for Ambari, Attila Doroszlai, Jaimin Jetly, Robert Levas, and 
> Sebastian Toader.
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> In many cases support engineers want to look at if Unlimited Strength Java 
> Cryptography Extension is installed on all hosts quickly and easily.
> 
> This information is now visible on the host summary page, and can be accessed 
> via api call: /api/v1/hosts?fields=Hosts/last_agent_env.
> 
> 
> Diffs
> -
> 
>   ambari-agent/src/main/python/ambari_agent/HostInfo.py 7f041eb 
>   ambari-agent/src/test/python/ambari_agent/TestHostInfo.py 67294e0 
>   ambari-server/src/main/java/org/apache/ambari/server/agent/AgentEnv.java 
> 64d0865 
>   
> ambari-server/src/main/java/org/apache/ambari/server/configuration/Configuration.java
>  20431f6 
>   
> ambari-server/src/test/java/org/apache/ambari/server/agent/AgentResourceTest.java
>  3f62366 
>   ambari-web/app/controllers/global/update_controller.js 59eaf3e 
>   ambari-web/app/mappers/hosts_mapper.js e4dd08f 
>   ambari-web/app/messages.js c9926e6 
>   ambari-web/app/models/host.js 1b7b326 
>   ambari-web/app/templates/main/host/summary.hbs 5d0b09c 
> 
> 
> Diff: https://reviews.apache.org/r/57771/diff/1/
> 
> 
> Testing
> ---
> 
> added new unittests, tested the UI manually
> 
> existing tests ran successfully with one irrelevant exception:
> 
> ==
> FAIL: test_get_params_from_filesystem 
> (TestSecurityCommons.TestSecurityCommons)
> --
> Traceback (most recent call last):
>   File 
> "/Users/amagyar/development/ambari/ambari-common/src/test/python/mock/mock.py",
>  line 1199, in patched
> return func(*args, **keywargs)
>   File 
> "/Users/amagyar/development/ambari/ambari-agent/src/test/python/resource_management/TestSecurityCommons.py",
>  line 203, in test_get_params_from_filesystem
> et_parser_mock.assert_called_with(conf_dir + os.sep + "config.xml")
>   File 
> "/Users/amagyar/development/ambari/ambari-common/src/test/python/mock/mock.py",
>  line 829, in assert_called_with
> raise AssertionError('Expected call: %s\nNot called' % (expected,))
> AssertionError: Expected call: 
> parse('/var/folders/hp/5rnr2yx14q7csg8xbtp_mkr8gn/T/config.xml')
> Not called
> 
> --
> Ran 465 tests in 16.659s
> 
> 
> Thanks,
> 
> Attila Magyar
> 
>

Re: Review Request 57776: Hive View2.0: Upload table fails, if one column is deleted

2017-03-20 Thread Pallav Kulshreshtha 


> On March 20, 2017, 1:11 p.m., Nitiraj Rathore wrote:
> > contrib/views/hive20/src/main/resources/ui/app/components/column-item.js
> > Line 27 (original), 27 (patched)
> > 
> >
> > Make this options object instead.

Niti, uploaded the code with suggested changes.Pls have a look.


- Pallav


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57776/#review169432
---


On March 20, 2017, 2:12 p.m., Pallav Kulshreshtha wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/57776/
> ---
> 
> (Updated March 20, 2017, 2:12 p.m.)
> 
> 
> Review request for Ambari, Abhishek Kumar, DIPAYAN BHOWMICK, Gaurav Nagar, 
> Nitiraj Rathore, Rohit Choudhary, and venkat sairam.
> 
> 
> Bugs: AMBARI-20497
> https://issues.apache.org/jira/browse/AMBARI-20497
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Added a var 'fromUploadTable' as true when it comes from upload table 
> otherwise false.
> 
> 
> Diffs
> -
> 
>   
> contrib/views/hive20/src/main/resources/ui/app/routes/databases/database/tables/upload-table.js
>  e6b71fe 
>   
> contrib/views/hive20/src/main/resources/ui/app/templates/components/column-item.hbs
>  762c77c 
>   
> contrib/views/hive20/src/main/resources/ui/app/templates/components/create-table.hbs
>  f369783 
>   
> contrib/views/hive20/src/main/resources/ui/app/templates/components/table-columns.hbs
>  b942136 
>   
> contrib/views/hive20/src/main/resources/ui/app/templates/components/upload-table.hbs
>  0ee6b81 
>   
> contrib/views/hive20/src/main/resources/ui/app/templates/databases/database/tables/upload-table.hbs
>  0091ede 
> 
> 
> Diff: https://reviews.apache.org/r/57776/diff/2/
> 
> 
> Testing
> ---
> 
> manually test.
> 
> 
> Thanks,
> 
> Pallav Kulshreshtha
> 
>

Re: Review Request 57770: Disabling security fails with AttributeError

2017-03-20 Thread Robert Levas 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57770/
---

(Updated March 20, 2017, 10:05 a.m.)


Review request for Ambari, Attila Magyar, Balázs Bence Sári, Eugene Chekanskiy, 
Laszlo Puskas, and Sebastian Toader.


Bugs: AMBARI-20494
https://issues.apache.org/jira/browse/AMBARI-20494


Repository: ambari


Description
---

Disabling security is failing with :
```
Stack Advisor reported an error: AttributeError: 'NoneType' object has no 
attribute 'replace'
StdOut file: /var/run/ambari-server/stack-recommendations/26/stackadvisor.out

StdErr file: /var/run/ambari-server/stack-recommendations/26/stackadvisor.err
```

Error file shows :
```
Traceback (most recent call last):
  File "/var/lib/ambari-server/resources/scripts/stack_advisor.py", line 166, 
in 
main(sys.argv)
  File "/var/lib/ambari-server/resources/scripts/stack_advisor.py", line 116, 
in main
result = stackAdvisor.recommendConfigurations(services, hosts)
  File "/var/lib/ambari-server/resources/scripts/../stacks/stack_advisor.py", 
line 775, in recommendConfigurations
calculation(configurations, clusterSummary, services, hosts)
  File 
"/var/lib/ambari-server/resources/scripts/./../stacks/HDP/2.5/services/stack_advisor.py",
 line 480, in recommendStormConfigurations
storm_nimbus_impersonation_acl.replace('{{storm_bare_jaas_principal}}', 
storm_bare_jaas_principal)
AttributeError: 'NoneType' object has no attribute 'replace'
```


Diffs
-

  ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py 
13151eb 
  ambari-server/src/test/python/stacks/2.5/common/test_stack_advisor.py 693b067 


Diff: https://reviews.apache.org/r/57770/diff/1/


Testing (updated)
---

Manually tested

# Local test results: 
```
[INFO] 
[INFO] BUILD SUCCESS
[INFO] 
[INFO] Total time: 1:15.026s
[INFO] Finished at: Sat Mar 18 21:16:02 EDT 2017
[INFO] Final Memory: 71M/1372M
[INFO] 
```

# Jenkins test results: 
```
{color:green}+1 overall{color}.  Here are the results of testing the latest 
attachment
  
http://issues.apache.org/jira/secure/attachment/12859515/AMBARI-20494_trunk_01.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 1 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
ambari-server.

Test results: 
https://builds.apache.org/job/Ambari-trunk-test-patch/11080//testReport/
Console output: 
https://builds.apache.org/job/Ambari-trunk-test-patch/11080//console
```


Thanks,

Robert Levas

Re: Review Request 57776: Hive View2.0: Upload table fails, if one column is deleted

2017-03-20 Thread Pallav Kulshreshtha 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57776/
---

(Updated March 20, 2017, 2:12 p.m.)


Review request for Ambari, Abhishek Kumar, DIPAYAN BHOWMICK, Gaurav Nagar, 
Nitiraj Rathore, Rohit Choudhary, and venkat sairam.


Bugs: AMBARI-20497
https://issues.apache.org/jira/browse/AMBARI-20497


Repository: ambari


Description
---

Added a var 'fromUploadTable' as true when it comes from upload table otherwise 
false.


Diffs (updated)
-

  
contrib/views/hive20/src/main/resources/ui/app/routes/databases/database/tables/upload-table.js
 e6b71fe 
  
contrib/views/hive20/src/main/resources/ui/app/templates/components/column-item.hbs
 762c77c 
  
contrib/views/hive20/src/main/resources/ui/app/templates/components/create-table.hbs
 f369783 
  
contrib/views/hive20/src/main/resources/ui/app/templates/components/table-columns.hbs
 b942136 
  
contrib/views/hive20/src/main/resources/ui/app/templates/components/upload-table.hbs
 0ee6b81 
  
contrib/views/hive20/src/main/resources/ui/app/templates/databases/database/tables/upload-table.hbs
 0091ede 


Diff: https://reviews.apache.org/r/57776/diff/2/

Changes: https://reviews.apache.org/r/57776/diff/1-2/


Testing
---

manually test.


Thanks,

Pallav Kulshreshtha

Re: Review Request 57733: Configuration types are validated before the cluster resources are persisted.

2017-03-20 Thread Robert Levas 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57733/#review169434
---


Ship it!




Ship It!

- Robert Levas


On March 20, 2017, 7:48 a.m., Laszlo Puskas wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/57733/
> ---
> 
> (Updated March 20, 2017, 7:48 a.m.)
> 
> 
> Review request for Ambari, Robert Levas, Sandor Magyari, and Sebastian Toader.
> 
> 
> Bugs: AMBARI-20488
> https://issues.apache.org/jira/browse/AMBARI-20488
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> When a cluster is created using blueprints it is possible to pass 
> configuration in the cluster creation template (that is posted after the 
> blueprint isent in)
> On cluster provisioning configurations are validated and if errors are found 
> the cluster provisioning is interrupted. As the configuration validation is 
> done after the cluster resources are persisted it's not possible to resend 
> the (possibly corrected) cluster template and restart the cluster 
> provisioning.
> Solution:
> Perform the configuration type validation before the cluster resources are 
> persisted.
> 
> Note:
> Created new package for topology validators and moved existing validators 
> into it.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/ProvisionClusterRequest.java
>  2c8d09a 
>   
> ambari-server/src/main/java/org/apache/ambari/server/topology/RequiredPasswordValidator.java
>  41812c3 
>   
> ambari-server/src/main/java/org/apache/ambari/server/topology/validators/ClusterConfigTypeValidator.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/topology/validators/RequiredPasswordValidator.java
>  PRE-CREATION 
>   
> ambari-server/src/test/java/org/apache/ambari/server/controller/internal/ProvisionClusterRequestTest.java
>  c64da62 
>   
> ambari-server/src/test/java/org/apache/ambari/server/topology/RequiredPasswordValidatorTest.java
>  8ab3a43 
>   
> ambari-server/src/test/java/org/apache/ambari/server/topology/validators/ClusterConfigTypeValidatorTest.java
>  PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/57733/diff/6/
> 
> 
> Testing
> ---
> 
> Manually tested on local environment.
> Unit tests succeeded.
> 
> 
> Thanks,
> 
> Laszlo Puskas
> 
>

Re: Review Request 57776: Hive View2.0: Upload table fails, if one column is deleted

2017-03-20 Thread Nitiraj Rathore 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57776/#review169432
---




contrib/views/hive20/src/main/resources/ui/app/components/column-item.js
Line 27 (original), 27 (patched)


Make this options object instead.


- Nitiraj Rathore


On March 20, 2017, 12:59 p.m., Pallav Kulshreshtha wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/57776/
> ---
> 
> (Updated March 20, 2017, 12:59 p.m.)
> 
> 
> Review request for Ambari, Abhishek Kumar, DIPAYAN BHOWMICK, Gaurav Nagar, 
> Nitiraj Rathore, Rohit Choudhary, and venkat sairam.
> 
> 
> Bugs: AMBARI-20497
> https://issues.apache.org/jira/browse/AMBARI-20497
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Added a var 'fromUploadTable' as true when it comes from upload table 
> otherwise false.
> 
> 
> Diffs
> -
> 
>   contrib/views/hive20/src/main/resources/ui/app/components/column-item.js 
> f2e45bd 
>   contrib/views/hive20/src/main/resources/ui/app/components/create-table.js 
> 322aab5 
>   contrib/views/hive20/src/main/resources/ui/app/components/table-columns.js 
> 7d83353 
>   
> contrib/views/hive20/src/main/resources/ui/app/routes/databases/database/tables/upload-table.js
>  e6b71fe 
>   
> contrib/views/hive20/src/main/resources/ui/app/templates/components/column-item.hbs
>  762c77c 
>   
> contrib/views/hive20/src/main/resources/ui/app/templates/components/create-table.hbs
>  f369783 
>   
> contrib/views/hive20/src/main/resources/ui/app/templates/components/table-columns.hbs
>  b942136 
>   
> contrib/views/hive20/src/main/resources/ui/app/templates/components/upload-table.hbs
>  0ee6b81 
>   
> contrib/views/hive20/src/main/resources/ui/app/templates/databases/database/tables/upload-table.hbs
>  0091ede 
> 
> 
> Diff: https://reviews.apache.org/r/57776/diff/1/
> 
> 
> Testing
> ---
> 
> manually test.
> 
> 
> Thanks,
> 
> Pallav Kulshreshtha
> 
>

Review Request 57776: Hive View2.0: Upload table fails, if one column is deleted

2017-03-20 Thread Pallav Kulshreshtha 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57776/
---

Review request for Ambari, Abhishek Kumar, DIPAYAN BHOWMICK, Gaurav Nagar, 
Nitiraj Rathore, Rohit Choudhary, and venkat sairam.


Bugs: AMBARI-20497
https://issues.apache.org/jira/browse/AMBARI-20497


Repository: ambari


Description
---

Added a var 'fromUploadTable' as true when it comes from upload table otherwise 
false.


Diffs
-

  contrib/views/hive20/src/main/resources/ui/app/components/column-item.js 
f2e45bd 
  contrib/views/hive20/src/main/resources/ui/app/components/create-table.js 
322aab5 
  contrib/views/hive20/src/main/resources/ui/app/components/table-columns.js 
7d83353 
  
contrib/views/hive20/src/main/resources/ui/app/routes/databases/database/tables/upload-table.js
 e6b71fe 
  
contrib/views/hive20/src/main/resources/ui/app/templates/components/column-item.hbs
 762c77c 
  
contrib/views/hive20/src/main/resources/ui/app/templates/components/create-table.hbs
 f369783 
  
contrib/views/hive20/src/main/resources/ui/app/templates/components/table-columns.hbs
 b942136 
  
contrib/views/hive20/src/main/resources/ui/app/templates/components/upload-table.hbs
 0ee6b81 
  
contrib/views/hive20/src/main/resources/ui/app/templates/databases/database/tables/upload-table.hbs
 0091ede 


Diff: https://reviews.apache.org/r/57776/diff/1/


Testing
---

manually test.


Thanks,

Pallav Kulshreshtha

Re: Review Request 57771: Unlimited Key JCE policy is enabled or not on each host

2017-03-20 Thread Attila Magyar 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57771/
---

(Updated March 20, 2017, 12:49 p.m.)


Review request for Ambari, Attila Doroszlai, Jaimin Jetly, Robert Levas, and 
Sebastian Toader.


Changes
---

unittest results


Repository: ambari


Description
---

In many cases support engineers want to look at if Unlimited Strength Java 
Cryptography Extension is installed on all hosts quickly and easily.

This information is now visible on the host summary page, and can be accessed 
via api call: /api/v1/hosts?fields=Hosts/last_agent_env.


Diffs
-

  ambari-agent/src/main/python/ambari_agent/HostInfo.py 7f041eb 
  ambari-agent/src/test/python/ambari_agent/TestHostInfo.py 67294e0 
  ambari-server/src/main/java/org/apache/ambari/server/agent/AgentEnv.java 
64d0865 
  
ambari-server/src/main/java/org/apache/ambari/server/configuration/Configuration.java
 20431f6 
  
ambari-server/src/test/java/org/apache/ambari/server/agent/AgentResourceTest.java
 3f62366 
  ambari-web/app/controllers/global/update_controller.js 59eaf3e 
  ambari-web/app/mappers/hosts_mapper.js e4dd08f 
  ambari-web/app/messages.js c9926e6 
  ambari-web/app/models/host.js 1b7b326 
  ambari-web/app/templates/main/host/summary.hbs 5d0b09c 


Diff: https://reviews.apache.org/r/57771/diff/1/


Testing (updated)
---

added new unittests, tested the UI manually

existing tests ran successfully with one irrelevant exception:

==
FAIL: test_get_params_from_filesystem (TestSecurityCommons.TestSecurityCommons)
--
Traceback (most recent call last):
  File 
"/Users/amagyar/development/ambari/ambari-common/src/test/python/mock/mock.py", 
line 1199, in patched
return func(*args, **keywargs)
  File 
"/Users/amagyar/development/ambari/ambari-agent/src/test/python/resource_management/TestSecurityCommons.py",
 line 203, in test_get_params_from_filesystem
et_parser_mock.assert_called_with(conf_dir + os.sep + "config.xml")
  File 
"/Users/amagyar/development/ambari/ambari-common/src/test/python/mock/mock.py", 
line 829, in assert_called_with
raise AssertionError('Expected call: %s\nNot called' % (expected,))
AssertionError: Expected call: 
parse('/var/folders/hp/5rnr2yx14q7csg8xbtp_mkr8gn/T/config.xml')
Not called

--
Ran 465 tests in 16.659s


Thanks,

Attila Magyar

Re: Review Request 57771: Unlimited Key JCE policy is enabled or not on each host

2017-03-20 Thread Sebastian Toader 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57771/#review169427
---


Ship it!




Ship It!

- Sebastian Toader


On March 20, 2017, 12:50 p.m., Attila Magyar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/57771/
> ---
> 
> (Updated March 20, 2017, 12:50 p.m.)
> 
> 
> Review request for Ambari, Jaimin Jetly, Robert Levas, and Sebastian Toader.
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> In many cases support engineers want to look at if Unlimited Strength Java 
> Cryptography Extension is installed on all hosts quickly and easily.
> 
> This information is now visible on the host summary page, and can be accessed 
> via api call: /api/v1/hosts?fields=Hosts/last_agent_env.
> 
> 
> Diffs
> -
> 
>   ambari-agent/src/main/python/ambari_agent/HostInfo.py 7f041eb 
>   ambari-agent/src/test/python/ambari_agent/TestHostInfo.py 67294e0 
>   ambari-server/src/main/java/org/apache/ambari/server/agent/AgentEnv.java 
> 64d0865 
>   
> ambari-server/src/main/java/org/apache/ambari/server/configuration/Configuration.java
>  20431f6 
>   
> ambari-server/src/test/java/org/apache/ambari/server/agent/AgentResourceTest.java
>  3f62366 
>   ambari-web/app/controllers/global/update_controller.js 59eaf3e 
>   ambari-web/app/mappers/hosts_mapper.js e4dd08f 
>   ambari-web/app/messages.js c9926e6 
>   ambari-web/app/models/host.js 1b7b326 
>   ambari-web/app/templates/main/host/summary.hbs 5d0b09c 
> 
> 
> Diff: https://reviews.apache.org/r/57771/diff/1/
> 
> 
> Testing
> ---
> 
> added new unittests, tested the UI manually
> 
> existing tests: PENDING..
> 
> 
> Thanks,
> 
> Attila Magyar
> 
>

Review Request 57771: Unlimited Key JCE policy is enabled or not on each host

2017-03-20 Thread Attila Magyar 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57771/
---

Review request for Ambari, Jaimin Jetly, Robert Levas, and Sebastian Toader.


Repository: ambari


Description
---

In many cases support engineers want to look at if Unlimited Strength Java 
Cryptography Extension is installed on all hosts quickly and easily.

This information is now visible on the host summary page, and can be accessed 
via api call: /api/v1/hosts?fields=Hosts/last_agent_env.


Diffs
-

  ambari-agent/src/main/python/ambari_agent/HostInfo.py 7f041eb 
  ambari-agent/src/test/python/ambari_agent/TestHostInfo.py 67294e0 
  ambari-server/src/main/java/org/apache/ambari/server/agent/AgentEnv.java 
64d0865 
  
ambari-server/src/main/java/org/apache/ambari/server/configuration/Configuration.java
 20431f6 
  
ambari-server/src/test/java/org/apache/ambari/server/agent/AgentResourceTest.java
 3f62366 
  ambari-web/app/controllers/global/update_controller.js 59eaf3e 
  ambari-web/app/mappers/hosts_mapper.js e4dd08f 
  ambari-web/app/messages.js c9926e6 
  ambari-web/app/models/host.js 1b7b326 
  ambari-web/app/templates/main/host/summary.hbs 5d0b09c 


Diff: https://reviews.apache.org/r/57771/diff/1/


Testing
---

added new unittests, tested the UI manually

existing tests: PENDING..


Thanks,

Attila Magyar

Re: Review Request 57733: Configuration types are validated before the cluster resources are persisted.

2017-03-20 Thread Laszlo Puskas 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57733/
---

(Updated March 20, 2017, 11:48 a.m.)


Review request for Ambari, Robert Levas, Sandor Magyari, and Sebastian Toader.


Changes
---

Fixed import order.


Bugs: AMBARI-20488
https://issues.apache.org/jira/browse/AMBARI-20488


Repository: ambari


Description
---

When a cluster is created using blueprints it is possible to pass configuration 
in the cluster creation template (that is posted after the blueprint isent in)
On cluster provisioning configurations are validated and if errors are found 
the cluster provisioning is interrupted. As the configuration validation is 
done after the cluster resources are persisted it's not possible to resend the 
(possibly corrected) cluster template and restart the cluster provisioning.
Solution:
Perform the configuration type validation before the cluster resources are 
persisted.

Note:
Created new package for topology validators and moved existing validators into 
it.


Diffs (updated)
-

  
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/ProvisionClusterRequest.java
 2c8d09a 
  
ambari-server/src/main/java/org/apache/ambari/server/topology/RequiredPasswordValidator.java
 41812c3 
  
ambari-server/src/main/java/org/apache/ambari/server/topology/validators/ClusterConfigTypeValidator.java
 PRE-CREATION 
  
ambari-server/src/main/java/org/apache/ambari/server/topology/validators/RequiredPasswordValidator.java
 PRE-CREATION 
  
ambari-server/src/test/java/org/apache/ambari/server/controller/internal/ProvisionClusterRequestTest.java
 c64da62 
  
ambari-server/src/test/java/org/apache/ambari/server/topology/RequiredPasswordValidatorTest.java
 8ab3a43 
  
ambari-server/src/test/java/org/apache/ambari/server/topology/validators/ClusterConfigTypeValidatorTest.java
 PRE-CREATION 


Diff: https://reviews.apache.org/r/57733/diff/6/

Changes: https://reviews.apache.org/r/57733/diff/5-6/


Testing
---

Manually tested on local environment.
Unit tests succeeded.


Thanks,

Laszlo Puskas

Re: Review Request 57733: Configuration types are validated before the cluster resources are persisted.

2017-03-20 Thread Sandor Magyari 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57733/#review169416
---


Ship it!




Ship It!

- Sandor Magyari


On March 20, 2017, 10:38 a.m., Laszlo Puskas wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/57733/
> ---
> 
> (Updated March 20, 2017, 10:38 a.m.)
> 
> 
> Review request for Ambari, Robert Levas, Sandor Magyari, and Sebastian Toader.
> 
> 
> Bugs: AMBARI-20488
> https://issues.apache.org/jira/browse/AMBARI-20488
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> When a cluster is created using blueprints it is possible to pass 
> configuration in the cluster creation template (that is posted after the 
> blueprint isent in)
> On cluster provisioning configurations are validated and if errors are found 
> the cluster provisioning is interrupted. As the configuration validation is 
> done after the cluster resources are persisted it's not possible to resend 
> the (possibly corrected) cluster template and restart the cluster 
> provisioning.
> Solution:
> Perform the configuration type validation before the cluster resources are 
> persisted.
> 
> Note:
> Created new package for topology validators and moved existing validators 
> into it.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/ProvisionClusterRequest.java
>  2c8d09a 
>   
> ambari-server/src/main/java/org/apache/ambari/server/topology/RequiredPasswordValidator.java
>  41812c3 
>   
> ambari-server/src/main/java/org/apache/ambari/server/topology/validators/ClusterConfigTypeValidator.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/topology/validators/RequiredPasswordValidator.java
>  PRE-CREATION 
>   
> ambari-server/src/test/java/org/apache/ambari/server/controller/internal/ProvisionClusterRequestTest.java
>  c64da62 
>   
> ambari-server/src/test/java/org/apache/ambari/server/topology/RequiredPasswordValidatorTest.java
>  8ab3a43 
>   
> ambari-server/src/test/java/org/apache/ambari/server/topology/validators/ClusterConfigTypeValidatorTest.java
>  PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/57733/diff/5/
> 
> 
> Testing
> ---
> 
> Manually tested on local environment.
> Unit tests succeeded.
> 
> 
> Thanks,
> 
> Laszlo Puskas
> 
>

Re: Review Request 57770: Disabling security fails with AttributeError

2017-03-20 Thread Laszlo Puskas 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57770/#review169414
---


Ship it!




Ship It!

- Laszlo Puskas


On March 20, 2017, 9:07 a.m., Robert Levas wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/57770/
> ---
> 
> (Updated March 20, 2017, 9:07 a.m.)
> 
> 
> Review request for Ambari, Attila Magyar, Balázs Bence Sári, Eugene 
> Chekanskiy, Laszlo Puskas, and Sebastian Toader.
> 
> 
> Bugs: AMBARI-20494
> https://issues.apache.org/jira/browse/AMBARI-20494
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Disabling security is failing with :
> ```
> Stack Advisor reported an error: AttributeError: 'NoneType' object has no 
> attribute 'replace'
> StdOut file: /var/run/ambari-server/stack-recommendations/26/stackadvisor.out
> 
> StdErr file: /var/run/ambari-server/stack-recommendations/26/stackadvisor.err
> ```
> 
> Error file shows :
> ```
> Traceback (most recent call last):
>   File "/var/lib/ambari-server/resources/scripts/stack_advisor.py", line 166, 
> in 
> main(sys.argv)
>   File "/var/lib/ambari-server/resources/scripts/stack_advisor.py", line 116, 
> in main
> result = stackAdvisor.recommendConfigurations(services, hosts)
>   File "/var/lib/ambari-server/resources/scripts/../stacks/stack_advisor.py", 
> line 775, in recommendConfigurations
> calculation(configurations, clusterSummary, services, hosts)
>   File 
> "/var/lib/ambari-server/resources/scripts/./../stacks/HDP/2.5/services/stack_advisor.py",
>  line 480, in recommendStormConfigurations
> storm_nimbus_impersonation_acl.replace('{{storm_bare_jaas_principal}}', 
> storm_bare_jaas_principal)
> AttributeError: 'NoneType' object has no attribute 'replace'
> ```
> 
> 
> Diffs
> -
> 
>   ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py 
> 13151eb 
>   ambari-server/src/test/python/stacks/2.5/common/test_stack_advisor.py 
> 693b067 
> 
> 
> Diff: https://reviews.apache.org/r/57770/diff/1/
> 
> 
> Testing
> ---
> 
> Manually tested
> 
> # Local test results: 
> ```
> [INFO] 
> 
> [INFO] BUILD SUCCESS
> [INFO] 
> 
> [INFO] Total time: 1:15.026s
> [INFO] Finished at: Sat Mar 18 21:16:02 EDT 2017
> [INFO] Final Memory: 71M/1372M
> [INFO] 
> 
> ```
> 
> # Jenkins test results: PENDING
> 
> 
> Thanks,
> 
> Robert Levas
> 
>

Re: Review Request 57733: Configuration types are validated before the cluster resources are persisted.

2017-03-20 Thread Laszlo Puskas 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57733/
---

(Updated March 20, 2017, 10:38 a.m.)


Review request for Ambari, Robert Levas, Sandor Magyari, and Sebastian Toader.


Changes
---

- Added null checks
- added test for equals
- validator list extracted into a member


Bugs: AMBARI-20488
https://issues.apache.org/jira/browse/AMBARI-20488


Repository: ambari


Description
---

When a cluster is created using blueprints it is possible to pass configuration 
in the cluster creation template (that is posted after the blueprint isent in)
On cluster provisioning configurations are validated and if errors are found 
the cluster provisioning is interrupted. As the configuration validation is 
done after the cluster resources are persisted it's not possible to resend the 
(possibly corrected) cluster template and restart the cluster provisioning.
Solution:
Perform the configuration type validation before the cluster resources are 
persisted.

Note:
Created new package for topology validators and moved existing validators into 
it.


Diffs (updated)
-

  
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/ProvisionClusterRequest.java
 2c8d09a 
  
ambari-server/src/main/java/org/apache/ambari/server/topology/RequiredPasswordValidator.java
 41812c3 
  
ambari-server/src/main/java/org/apache/ambari/server/topology/validators/ClusterConfigTypeValidator.java
 PRE-CREATION 
  
ambari-server/src/main/java/org/apache/ambari/server/topology/validators/RequiredPasswordValidator.java
 PRE-CREATION 
  
ambari-server/src/test/java/org/apache/ambari/server/controller/internal/ProvisionClusterRequestTest.java
 c64da62 
  
ambari-server/src/test/java/org/apache/ambari/server/topology/RequiredPasswordValidatorTest.java
 8ab3a43 
  
ambari-server/src/test/java/org/apache/ambari/server/topology/validators/ClusterConfigTypeValidatorTest.java
 PRE-CREATION 


Diff: https://reviews.apache.org/r/57733/diff/5/

Changes: https://reviews.apache.org/r/57733/diff/4-5/


Testing
---

Manually tested on local environment.
Unit tests succeeded.


Thanks,

Laszlo Puskas

Re: Review Request 57733: Configuration types are validated before the cluster resources are persisted.

2017-03-20 Thread Laszlo Puskas 


> On March 20, 2017, 8:54 a.m., Sebastian Toader wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/topology/validators/ClusterConfigTypeValidator.java
> > Lines 30 (patched)
> > 
> >
> > Implement equals and hashCode such as instances of this validator can 
> > be stored in hashmaps, sets etc.

As the class doesn't have any members the default (super) implementation is 
enough here.


> On March 20, 2017, 8:54 a.m., Sebastian Toader wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/topology/validators/ClusterConfigTypeValidator.java
> > Lines 37 (patched)
> > 
> >
> > Could ```topology.getConfiguration()``` ever be null?

This is always set to a value, i can add a null check however.


> On March 20, 2017, 8:54 a.m., Sebastian Toader wrote:
> > ambari-server/src/test/java/org/apache/ambari/server/topology/validators/ClusterConfigTypeValidatorTest.java
> > Lines 37 (patched)
> > 
> >
> > You could use ```EqualsVerifier``` to ensure that the ```equals``` and 
> > ```hashCode``` methods are properly implemented and are not broken 
> > inadvertently in the future.

Added a simple check.


- Laszlo


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57733/#review169394
---


On March 18, 2017, 9:59 a.m., Laszlo Puskas wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/57733/
> ---
> 
> (Updated March 18, 2017, 9:59 a.m.)
> 
> 
> Review request for Ambari, Robert Levas, Sandor Magyari, and Sebastian Toader.
> 
> 
> Bugs: AMBARI-20488
> https://issues.apache.org/jira/browse/AMBARI-20488
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> When a cluster is created using blueprints it is possible to pass 
> configuration in the cluster creation template (that is posted after the 
> blueprint isent in)
> On cluster provisioning configurations are validated and if errors are found 
> the cluster provisioning is interrupted. As the configuration validation is 
> done after the cluster resources are persisted it's not possible to resend 
> the (possibly corrected) cluster template and restart the cluster 
> provisioning.
> Solution:
> Perform the configuration type validation before the cluster resources are 
> persisted.
> 
> Note:
> Created new package for topology validators and moved existing validators 
> into it.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/ProvisionClusterRequest.java
>  2c8d09a 
>   
> ambari-server/src/main/java/org/apache/ambari/server/topology/RequiredPasswordValidator.java
>  41812c3 
>   
> ambari-server/src/main/java/org/apache/ambari/server/topology/validators/ClusterConfigTypeValidator.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/topology/validators/RequiredPasswordValidator.java
>  PRE-CREATION 
>   
> ambari-server/src/test/java/org/apache/ambari/server/controller/internal/ProvisionClusterRequestTest.java
>  c64da62 
>   
> ambari-server/src/test/java/org/apache/ambari/server/topology/RequiredPasswordValidatorTest.java
>  8ab3a43 
>   
> ambari-server/src/test/java/org/apache/ambari/server/topology/validators/ClusterConfigTypeValidatorTest.java
>  PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/57733/diff/4/
> 
> 
> Testing
> ---
> 
> Manually tested on local environment.
> Unit tests succeeded.
> 
> 
> Thanks,
> 
> Laszlo Puskas
> 
>

Re: Review Request 57536: Restart of Spark2-client fails at setup_spark()

2017-03-20 Thread Andrew Onischuk 


> On March 11, 2017, 3:27 p.m., Sumit Mohanty wrote:
> > Making this change will allow Ambari to produce the properties file - would 
> > it cause trouble when service read it - e.g. when Spark is starting? IOW, 
> > do they need to set something in the env file to be able to read UTF-8.

I don't think it would require any change. We have all xml files written in 
utf-8 for a long and it never was a problem.


- Andrew


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57536/#review168710
---


On March 11, 2017, 11:43 a.m., Andrew Onischuk wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/57536/
> ---
> 
> (Updated March 11, 2017, 11:43 a.m.)
> 
> 
> Review request for Ambari and Sumit Mohanty.
> 
> 
> Bugs: AMBARI-20404
> https://issues.apache.org/jira/browse/AMBARI-20404
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Scenario:
> 
>   * Added below configs to spark2 default 
> 
> 
> spark.io.encryption.enabled = true
> spark.io.encryption.keySizeBits = 128
> spark.io.encryption.keygen.algorithm = HmacSHA1? 
> 
>   * Restart affected services.  
> Spark2-client is failing to restart with below error.
> 
> **stderr**
> 
> 
> Traceback (most recent call last):
>   File 
> "/var/lib/ambari-agent/cache/common-services/SPARK2/2.0.0/package/scripts/spark_client.py",
>  line 60, in 
> SparkClient().execute()
>   File 
> "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py",
>  line 313, in execute
> method(env)
>   File 
> "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py",
>  line 721, in restart
> self.install(env)
>   File 
> "/var/lib/ambari-agent/cache/common-services/SPARK2/2.0.0/package/scripts/spark_client.py",
>  line 36, in install
> self.configure(env)
>   File 
> "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py",
>  line 116, in locking_configure
> original_configure(obj, *args, **kw)
>   File 
> "/var/lib/ambari-agent/cache/common-services/SPARK2/2.0.0/package/scripts/spark_client.py",
>  line 42, in configure
> setup_spark(env, 'client', upgrade_type=upgrade_type, action = 
> 'config')
>   File 
> "/var/lib/ambari-agent/cache/common-services/SPARK2/2.0.0/package/scripts/setup_spark.py",
>  line 56, in setup_spark
> mode=0644
>   File 
> "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 
> 155, in __init__
> self.env.run()
>   File 
> "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", 
> line 160, in run
> self.run_action(resource, action)
>   File 
> "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", 
> line 124, in run_action
> provider_action()
>   File 
> "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/properties_file.py",
>  line 54, in action_create
> mode = self.resource.mode
>   File 
> "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 
> 155, in __init__
> self.env.run()
>   File 
> "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", 
> line 160, in run
> self.run_action(resource, action)
>   File 
> "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", 
> line 124, in run_action
> provider_action()
>   File 
> "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py",
>  line 123, in action_create
> content = self._get_content()
>   File 
> "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py",
>  line 160, in _get_content
> return content()
>   File 
> "/usr/lib/python2.6/site-packages/resource_management/core/source.py", line 
> 52, in __call__
> return self.get_content()
>   File 
> "/usr/lib/python2.6/site-packages/resource_management/core/source.py", line 
> 143, in get_content
> rendered = self.template.render(self.context)
>   File "/usr/lib/python2.6/site-packages/ambari_jinja2/environment.py", 
> line 891, in render
> return self.environment.handle_exception(exc_info, True)
>   File "", line 3, in top-level template code
> UnicodeEncodeError: 'ascii' codec can't encode character u'\u200b' in 
> position 8: ordinal not in range(128)
> 
> 
> ambari-hash : d54803dd86c35f92637a978c58cf06818f8ca0d5
> 
> 
> Diffs
> -
> 
>   
> ambari-agent/src/test/python/resource_management/TestPropertiesFileResource.py
>  3dd60e1 
>

Re: Review Request 57770: Disabling security fails with AttributeError

2017-03-20 Thread Sebastian Toader 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57770/#review169403
---


Ship it!




Ship It!

- Sebastian Toader


On March 20, 2017, 10:07 a.m., Robert Levas wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/57770/
> ---
> 
> (Updated March 20, 2017, 10:07 a.m.)
> 
> 
> Review request for Ambari, Attila Magyar, Balázs Bence Sári, Eugene 
> Chekanskiy, Laszlo Puskas, and Sebastian Toader.
> 
> 
> Bugs: AMBARI-20494
> https://issues.apache.org/jira/browse/AMBARI-20494
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Disabling security is failing with :
> ```
> Stack Advisor reported an error: AttributeError: 'NoneType' object has no 
> attribute 'replace'
> StdOut file: /var/run/ambari-server/stack-recommendations/26/stackadvisor.out
> 
> StdErr file: /var/run/ambari-server/stack-recommendations/26/stackadvisor.err
> ```
> 
> Error file shows :
> ```
> Traceback (most recent call last):
>   File "/var/lib/ambari-server/resources/scripts/stack_advisor.py", line 166, 
> in 
> main(sys.argv)
>   File "/var/lib/ambari-server/resources/scripts/stack_advisor.py", line 116, 
> in main
> result = stackAdvisor.recommendConfigurations(services, hosts)
>   File "/var/lib/ambari-server/resources/scripts/../stacks/stack_advisor.py", 
> line 775, in recommendConfigurations
> calculation(configurations, clusterSummary, services, hosts)
>   File 
> "/var/lib/ambari-server/resources/scripts/./../stacks/HDP/2.5/services/stack_advisor.py",
>  line 480, in recommendStormConfigurations
> storm_nimbus_impersonation_acl.replace('{{storm_bare_jaas_principal}}', 
> storm_bare_jaas_principal)
> AttributeError: 'NoneType' object has no attribute 'replace'
> ```
> 
> 
> Diffs
> -
> 
>   ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py 
> 13151eb 
>   ambari-server/src/test/python/stacks/2.5/common/test_stack_advisor.py 
> 693b067 
> 
> 
> Diff: https://reviews.apache.org/r/57770/diff/1/
> 
> 
> Testing
> ---
> 
> Manually tested
> 
> # Local test results: 
> ```
> [INFO] 
> 
> [INFO] BUILD SUCCESS
> [INFO] 
> 
> [INFO] Total time: 1:15.026s
> [INFO] Finished at: Sat Mar 18 21:16:02 EDT 2017
> [INFO] Final Memory: 71M/1372M
> [INFO] 
> 
> ```
> 
> # Jenkins test results: PENDING
> 
> 
> Thanks,
> 
> Robert Levas
> 
>

Re: Review Request 57756: Make set-hdp-repo-url.sh version agnostic for HDP stacks

2017-03-20 Thread Vitalyi Brodetskyi 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57756/#review169401
---


Ship it!




Ship It!

- Vitalyi Brodetskyi


On Март 19, 2017, 11:25 д.п., Andrew Onischuk wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/57756/
> ---
> 
> (Updated Март 19, 2017, 11:25 д.п.)
> 
> 
> Review request for Ambari and Sumit Mohanty.
> 
> 
> Bugs: AMBARI-20493
> https://issues.apache.org/jira/browse/AMBARI-20493
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> As `ambari-server/set-hdp-repo-url.sh` has a hard-coded list of HDP versions,
> it usually ends up as a late discovery of the problem on the day of the
> release. E.g. for HDP-2.6 release, RE provided the modified value for the hdp
> urlinfo json file but it did not take effect.
> 
> 
> Diffs
> -
> 
>   ambari-server/pom.xml 66213e9 
>   ambari-server/set-hdp-repo-url.sh 1835705 
> 
> 
> Diff: https://reviews.apache.org/r/57756/diff/1/
> 
> 
> Testing
> ---
> 
> mvn clean test
> 
> 
> Thanks,
> 
> Andrew Onischuk
> 
>

Review Request 57770: Disabling security fails with AttributeError

2017-03-20 Thread Robert Levas 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57770/
---

Review request for Ambari, Attila Magyar, Balázs Bence Sári, Eugene Chekanskiy, 
Laszlo Puskas, and Sebastian Toader.


Bugs: AMBARI-20494
https://issues.apache.org/jira/browse/AMBARI-20494


Repository: ambari


Description
---

Disabling security is failing with :
```
Stack Advisor reported an error: AttributeError: 'NoneType' object has no 
attribute 'replace'
StdOut file: /var/run/ambari-server/stack-recommendations/26/stackadvisor.out

StdErr file: /var/run/ambari-server/stack-recommendations/26/stackadvisor.err
```

Error file shows :
```
Traceback (most recent call last):
  File "/var/lib/ambari-server/resources/scripts/stack_advisor.py", line 166, 
in 
main(sys.argv)
  File "/var/lib/ambari-server/resources/scripts/stack_advisor.py", line 116, 
in main
result = stackAdvisor.recommendConfigurations(services, hosts)
  File "/var/lib/ambari-server/resources/scripts/../stacks/stack_advisor.py", 
line 775, in recommendConfigurations
calculation(configurations, clusterSummary, services, hosts)
  File 
"/var/lib/ambari-server/resources/scripts/./../stacks/HDP/2.5/services/stack_advisor.py",
 line 480, in recommendStormConfigurations
storm_nimbus_impersonation_acl.replace('{{storm_bare_jaas_principal}}', 
storm_bare_jaas_principal)
AttributeError: 'NoneType' object has no attribute 'replace'
```


Diffs
-

  ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py 
13151eb 
  ambari-server/src/test/python/stacks/2.5/common/test_stack_advisor.py 693b067 


Diff: https://reviews.apache.org/r/57770/diff/1/


Testing
---

Manually tested

# Local test results: 
```
[INFO] 
[INFO] BUILD SUCCESS
[INFO] 
[INFO] Total time: 1:15.026s
[INFO] Finished at: Sat Mar 18 21:16:02 EDT 2017
[INFO] Final Memory: 71M/1372M
[INFO] 
```

# Jenkins test results: PENDING


Thanks,

Robert Levas

Re: Review Request 57733: Configuration types are validated before the cluster resources are persisted.

2017-03-20 Thread Sebastian Toader 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57733/#review169394
---


Fix it, then Ship it!





ambari-server/src/main/java/org/apache/ambari/server/controller/internal/ProvisionClusterRequest.java
Line 272 (original), 271-274 (patched)


Is the collection of topology validators being changed in other methods? If 
not than make this a singleton readonly field. (e.g. you could use guava's 
```ImmutableSet```)



ambari-server/src/main/java/org/apache/ambari/server/topology/validators/ClusterConfigTypeValidator.java
Lines 30 (patched)


Implement equals and hashCode such as instances of this validator can be 
stored in hashmaps, sets etc.



ambari-server/src/main/java/org/apache/ambari/server/topology/validators/ClusterConfigTypeValidator.java
Lines 37 (patched)


Could ```topology.getConfiguration()``` ever be null?



ambari-server/src/test/java/org/apache/ambari/server/topology/validators/ClusterConfigTypeValidatorTest.java
Lines 37 (patched)


You could use ```EqualsVerifier``` to ensure that the ```equals``` and 
```hashCode``` methods are properly implemented and are not broken 
inadvertently in the future.


- Sebastian Toader


On March 18, 2017, 10:59 a.m., Laszlo Puskas wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/57733/
> ---
> 
> (Updated March 18, 2017, 10:59 a.m.)
> 
> 
> Review request for Ambari, Robert Levas, Sandor Magyari, and Sebastian Toader.
> 
> 
> Bugs: AMBARI-20488
> https://issues.apache.org/jira/browse/AMBARI-20488
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> When a cluster is created using blueprints it is possible to pass 
> configuration in the cluster creation template (that is posted after the 
> blueprint isent in)
> On cluster provisioning configurations are validated and if errors are found 
> the cluster provisioning is interrupted. As the configuration validation is 
> done after the cluster resources are persisted it's not possible to resend 
> the (possibly corrected) cluster template and restart the cluster 
> provisioning.
> Solution:
> Perform the configuration type validation before the cluster resources are 
> persisted.
> 
> Note:
> Created new package for topology validators and moved existing validators 
> into it.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/ProvisionClusterRequest.java
>  2c8d09a 
>   
> ambari-server/src/main/java/org/apache/ambari/server/topology/RequiredPasswordValidator.java
>  41812c3 
>   
> ambari-server/src/main/java/org/apache/ambari/server/topology/validators/ClusterConfigTypeValidator.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/topology/validators/RequiredPasswordValidator.java
>  PRE-CREATION 
>   
> ambari-server/src/test/java/org/apache/ambari/server/controller/internal/ProvisionClusterRequestTest.java
>  c64da62 
>   
> ambari-server/src/test/java/org/apache/ambari/server/topology/RequiredPasswordValidatorTest.java
>  8ab3a43 
>   
> ambari-server/src/test/java/org/apache/ambari/server/topology/validators/ClusterConfigTypeValidatorTest.java
>  PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/57733/diff/4/
> 
> 
> Testing
> ---
> 
> Manually tested on local environment.
> Unit tests succeeded.
> 
> 
> Thanks,
> 
> Laszlo Puskas
> 
>