date:20171211

Re: Review Request 64478: Blueprint V2 API: object model, resource provider, cluster creation

2017-12-11 Thread Attila Doroszlai



> On Dec. 11, 2017, 11:23 p.m., Madhuvanthi Radhakrishnan wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementControllerImpl.java
> > Line 1029 (original), 1023 (patched)
> > 
> >
> > Hi Attila,
> > Since we havent completely gotten rid of /cluster/configuration yet, 
> > the serviceId in the request can be null. The change you have made is what 
> > we will do eventually but right now this might break the existing feature 
> > branch deployments.

Both code versions have the same result:

 * if `request.getServiceId()` is not `null`, then its value is used
 * if `request.getServiceId()` is `null`, then `null` is used, which is exactly 
`request.getServiceId()`'s value in this case

The `?:` is unnecessary.


> On Dec. 11, 2017, 11:23 p.m., Madhuvanthi Radhakrishnan wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/controller/internal/BaseClusterRequest.java
> > Line 137 (original), 110 (patched)
> > 
> >
> > Missing throws clause

Throws `IllegalArgumentException` instead.  Used in lambdas, hence cannot throw 
checked exception.  Does not make much difference, because 
`InvalidTopologyTemplateException` is translated to `IllegalArgumentException` 
in `ClusterResourceProvider`, `BlueprintV2ResourceProvider` etc. anyway.


> On Dec. 11, 2017, 11:23 p.m., Madhuvanthi Radhakrishnan wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/state/ConfigMergeHelper.java
> > Line 66 (original), 66 (patched)
> > 
> >
> > If each service instance can have its own config properties then 
> > shouldnt it be serviceName (unique)?
> > Correct me if i am wrong.

It should be `serviceType`, because `AmbariMetaInfo.getServiceProperties` 
expects `stackServiceName` (which is the same).

This line of change is not directly related to the blueprints implementation, 
it is an addendum for 
[AMBARI-22244](https://issues.apache.org/jira/browse/AMBARI-22244).


> On Dec. 11, 2017, 11:23 p.m., Madhuvanthi Radhakrishnan wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/topology/HostGroupInfo.java
> > Line 194 (original), 200 (patched)
> > 
> >
> > Missing throws clause

Same as above.


- Attila


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64478/#review193429
---


On Dec. 9, 2017, 1:19 p.m., Attila Doroszlai wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64478/
> ---
> 
> (Updated Dec. 9, 2017, 1:19 p.m.)
> 
> 
> Review request for Ambari, Balázs Bence Sári, Jayush Luniya, Madhuvanthi 
> Radhakrishnan, Robert Nettleton, and Swapan Shridhar.
> 
> 
> Bugs: AMBARI-22253, AMBARI-22297 and AMBARI-22325
> https://issues.apache.org/jira/browse/AMBARI-22253
> https://issues.apache.org/jira/browse/AMBARI-22297
> https://issues.apache.org/jira/browse/AMBARI-22325
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Implementation of Blueprint V2 API POST that allows cluster creation.  
> Co-authored by Balázs Bence Sári and Attila Doroszlai.
> 
> Includes minor fixes for:
> 
>  * AMBARI-22196 -- yum repo file contains parsing errors due to initial 
> whitespace
>  * AMBARI-22249 -- AmbariContext.serviceGroupDependencyResourceProvider 
> initialization
>  * AMBARI-22164 -- wrong assignment due to leftover parameter name
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/ObjectNotFoundException.java
>  75c9f3b5d50102f3beeeaa8041178cc08e22e241 
>   
> ambari-server/src/main/java/org/apache/ambari/server/ParentObjectNotFoundException.java
>  15bd7cb6bbe5026bf0a5ae9d3fc0bec575137643 
>   
> ambari-server/src/main/java/org/apache/ambari/server/StackAccessException.java
>  b8bfff3991a4e49514801dbb480958941cff01ac 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java
>  fd43edfa88339cd94686ddc13822b36ab7c74d6f 
>   
> ambari-server/src/main/java/org/apache/ambari/server/checks/ServiceCheckValidityCheck.java
>  2582c71eb95255f0f8081e90473183243f12cdd0 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementControllerImpl.java
>  59e6622f708e7d8d27c9e28ce2ce58f014a4a5dc 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariServer.java
>  c88b23c8af374dd9bd695b70560079871f3366fd 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/ConfigurationRequest.java
>

Re: Review Request 64528: Zeppelin Interpreter settings are getting updated after zeppelin restart

2017-12-11 Thread venkat sairam


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64528/#review193507
---


Ship it!




Ship It!

- venkat sairam


On Dec. 12, 2017, 7:39 a.m., Prabhjyot Singh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64528/
> ---
> 
> (Updated Dec. 12, 2017, 7:39 a.m.)
> 
> 
> Review request for Ambari, DIPAYAN BHOWMICK, Jaimin Jetly, Jayush Luniya, 
> Rohit Choudhary, Sumit Mohanty, and venkat sairam.
> 
> 
> Bugs: AMBARI-22626
> https://issues.apache.org/jira/browse/AMBARI-22626
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Zeppelin Interpreter settings are getting updated after zeppelin restart.
> 
> Steps to reproduce :
> 1) Update the zeppelin.pyspark.python to /base/tools/python-2.7.14/bin/python 
> for spark2 interpreter
> 2) Restart zeppelin
> 3) After restart zeppelin.pyspark.python is getting overridden with value 
> 'python'.
> Same is observed with livy2 interpreter.
> 
> But same steps for spark interpreter is working fine even after zeppelin 
> restart.
> 
> So looks like issue is happening only for spark2 and livy2 interpreter.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/ZEPPELIN/0.7.0/package/scripts/interpreter_json_template.py
>  713db232b1 
>   
> ambari-server/src/main/resources/common-services/ZEPPELIN/0.7.0/package/scripts/livy2_config_template.py
>  c9030bd33e 
>   
> ambari-server/src/main/resources/common-services/ZEPPELIN/0.7.0/package/scripts/master.py
>  bb4c5b227b 
>   
> ambari-server/src/main/resources/common-services/ZEPPELIN/0.7.0/package/scripts/spark2_config_template.py
>  28a63c6f1b 
>   
> ambari-server/src/test/python/stacks/2.6/ZEPPELIN/interpreter_json_generated.py
>  4b4dc1fe48 
> 
> 
> Diff: https://reviews.apache.org/r/64528/diff/1/
> 
> 
> Testing
> ---
> 
> Manually
> 
> 
> Thanks,
> 
> Prabhjyot Singh
> 
>

Re: Review Request 64528: Zeppelin Interpreter settings are getting updated after zeppelin restart

2017-12-11 Thread Prabhjyot Singh


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64528/
---

(Updated Dec. 12, 2017, 1:09 p.m.)


Review request for Ambari, DIPAYAN BHOWMICK, Jaimin Jetly, Jayush Luniya, Rohit 
Choudhary, Sumit Mohanty, and venkat sairam.


Bugs: AMBARI-22626
https://issues.apache.org/jira/browse/AMBARI-22626


Repository: ambari


Description (updated)
---

Zeppelin Interpreter settings are getting updated after zeppelin restart.

Steps to reproduce :
1) Update the zeppelin.pyspark.python to /base/tools/python-2.7.14/bin/python 
for spark2 interpreter
2) Restart zeppelin
3) After restart zeppelin.pyspark.python is getting overridden with value 
'python'.
Same is observed with livy2 interpreter.

But same steps for spark interpreter is working fine even after zeppelin 
restart.

So looks like issue is happening only for spark2 and livy2 interpreter.


Diffs
-

  
ambari-server/src/main/resources/common-services/ZEPPELIN/0.7.0/package/scripts/interpreter_json_template.py
 713db232b1 
  
ambari-server/src/main/resources/common-services/ZEPPELIN/0.7.0/package/scripts/livy2_config_template.py
 c9030bd33e 
  
ambari-server/src/main/resources/common-services/ZEPPELIN/0.7.0/package/scripts/master.py
 bb4c5b227b 
  
ambari-server/src/main/resources/common-services/ZEPPELIN/0.7.0/package/scripts/spark2_config_template.py
 28a63c6f1b 
  
ambari-server/src/test/python/stacks/2.6/ZEPPELIN/interpreter_json_generated.py 
4b4dc1fe48 


Diff: https://reviews.apache.org/r/64528/diff/1/


Testing
---

Manually


Thanks,

Prabhjyot Singh

Review Request 64528: Zeppelin Interpreter settings are getting updated after zeppelin restart

2017-12-11 Thread Prabhjyot Singh


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64528/
---

Review request for Ambari, DIPAYAN BHOWMICK, Jaimin Jetly, Jayush Luniya, Rohit 
Choudhary, Sumit Mohanty, and venkat sairam.


Bugs: AMBARI-22626
https://issues.apache.org/jira/browse/AMBARI-22626


Repository: ambari


Description
---

Zeppelin Interpreter settings are getting updated after zeppelin restart.
Live cluster : 
http://ctr-e135-1512069032975-20895-02-09.hwx.site:9995/#/interpreter
Steps to reproduce :
1) Update the zeppelin.pyspark.python to /base/tools/python-2.7.14/bin/python 
for spark2 interpreter
2) Restart zeppelin
3) After restart zeppelin.pyspark.python is getting overridden with value 
'python'.
Same is observed with livy2 interpreter.

But same steps for spark interpreter is working fine even after zeppelin 
restart.

So looks like issue is happening only for spark2 and livy2 interpreter.


Diffs
-

  
ambari-server/src/main/resources/common-services/ZEPPELIN/0.7.0/package/scripts/interpreter_json_template.py
 713db232b1 
  
ambari-server/src/main/resources/common-services/ZEPPELIN/0.7.0/package/scripts/livy2_config_template.py
 c9030bd33e 
  
ambari-server/src/main/resources/common-services/ZEPPELIN/0.7.0/package/scripts/master.py
 bb4c5b227b 
  
ambari-server/src/main/resources/common-services/ZEPPELIN/0.7.0/package/scripts/spark2_config_template.py
 28a63c6f1b 
  
ambari-server/src/test/python/stacks/2.6/ZEPPELIN/interpreter_json_generated.py 
4b4dc1fe48 


Diff: https://reviews.apache.org/r/64528/diff/1/


Testing
---

Manually


Thanks,

Prabhjyot Singh

Re: Review Request 64513: Update the default storm log search configuration (use wildcards)

2017-12-11 Thread Krisztian Kasa


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64513/#review193502
---


Ship it!




Ship It!

- Krisztian Kasa


On Dec. 11, 2017, 11:47 p.m., Oliver Szabo wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64513/
> ---
> 
> (Updated Dec. 11, 2017, 11:47 p.m.)
> 
> 
> Review request for Ambari, Krisztian Kasa, Miklos Gergely, and Sid Wagle.
> 
> 
> Bugs: AMBARI-22630
> https://issues.apache.org/jira/browse/AMBARI-22630
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> set the storm config defaults for logseach as other teams needs it.
> 
> this patch is for branch-2.6, but if cannot be delivered in 2.6.1, that can 
> be used in a hotfix
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/STORM/0.9.1/configuration/storm-logsearch-conf.xml
>  d485017 
> 
> 
> Diff: https://reviews.apache.org/r/64513/diff/1/
> 
> 
> Testing
> ---
> 
> not yet.
> 
> 
> Thanks,
> 
> Oliver Szabo
> 
>

Re: Review Request 64397: AMBARI-22602. Add 'clusterSettings' and 'stackSettings' parameters in Execution Command.

2017-12-11 Thread Swapan Shridhar


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64397/
---

(Updated Dec. 12, 2017, 4:07 a.m.)


Review request for Ambari, Attila Doroszlai, Jayush Luniya, and Madhuvanthi 
Radhakrishnan.


Changes
---

Code updated as per review comments.


Bugs: AMBARI-22602
https://issues.apache.org/jira/browse/AMBARI-22602


Repository: ambari


Description
---

AMBARI-22198 added "stack settings", and AMBARI-22196 introduced "cluster 
settings" in Ambari.

This review adds 2 new parameters to Execution Command : **(1).** 
clusterSettings and **(2).** stackSettings, and adds their respective set of 
key-values pairs.

- This enables these parameters to be passed in to the ambari-agent's 
**command*.json**.


Test cases are tracked in : AMBARI-22603


Diffs (updated)
-

  
ambari-server/src/main/java/org/apache/ambari/server/agent/ExecutionCommand.java
 5ee4bf6 
  
ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java
 fd43edf 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariCustomCommandExecutionHelper.java
 e7dea06 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementControllerImpl.java
 59e6622 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/ClientConfigResourceProvider.java
 9043297 
  
ambari-server/src/main/java/org/apache/ambari/server/resources/RootLevelSettingsManager.java
 8bd24f2 
  ambari-server/src/main/java/org/apache/ambari/server/state/Cluster.java 
c8768dd 
  
ambari-server/src/main/java/org/apache/ambari/server/state/cluster/ClusterImpl.java
 0b38b36 


Diff: https://reviews.apache.org/r/64397/diff/2/

Changes: https://reviews.apache.org/r/64397/diff/1-2/


Testing
---

Tested on Live cluster doing following : 
(1). Start Service 
(2). Restart Service 
(3). Stop Service 
(4). Refreshing client configs
(5). Adding Client.


**Snippet from command*.json as part of testing:**
{
...
...
"clusterSettings": {
"security_enabled": "false",
"hide_yarn_memory_widget": "false",
"enable_external_ranger": "false",
"override_uid": "true",
"kerberos_domain": "EXAMPLE.COM",
"one_dir_per_partition": "false",
"repo_ubuntu_template": "{{package_type}} {{base_url}} {{components}}",
"ignore_groupsusers_create": "false",
"alerts_repeat_tolerance": "1",
"namenode_rolling_restart_timeout": "4200",
"fetch_nonlocal_groups": "true",
"manage_dirs_on_root": "true",
"recovery_lifetime_max_count": "1024",
"agent_mounts_ignore_list": "",
"ignore_bad_mounts": "false",
"recovery_window_in_minutes": "60",
"sysprep_skip_copy_tarballs_hdfs": "false",
"recovery_type": "AUTO_START",
"user_group": "hadoop",
"namenode_rolling_restart_safemode_exit_timeout": "3600",
"recovery_retry_interval": "5",
"sysprep_skip_copy_oozie_share_lib_to_hdfs": "false",
"sysprep_skip_setup_jce": "false",
"manage_hive_fsroot": "true",
"service_check_type": "full",
"recovery_enabled": "true",
"recovery_max_count": "6",
"sysprep_skip_create_users_and_groups": "false",
"smokeuser_keytab": "/etc/security/keytabs/smokeuser.headless.keytab",
"managed_hdfs_resource_property_names": "false",
"smokeuser": "ambari-qa",
"sysprep_skip_copy_fast_jar_hdfs": "false"
},
...
...
...
...
...
...
...
"stackSettings": {
"stack_features": "{\n  \"HDP\": {\n\"stack_features\": [\n  
{\n\"name\": \"snappy\",\n\"description\": \"Snappy 
compressor/decompressor support\",\n\"min_version\": \"2.0.0.0\",\n 
   \"max_version\": \"2.2.0.0\"\n  },\n  {\n\"name\": 
\"lzo\",\n\"description\": \"LZO libraries support\",\n
\"min_version\": \"2.2.1.0\"\n  },\n  {\n\"name\": 
\"express_upgrade\",\n\"description\": \"Express upgrade support\",\n   
 \"min_version\": \"2.1.0.0\"\n  },\n  {\n\"name\": 
\"rolling_upgrade\",\n\"description\": \"Rolling upgrade support\",\n   
 \"min_version\": \"2.2.0.0\"\n  },\n  {\n\"name\": 
\"kafka_acl_migration_support\",\n\"description\": \"ACL migration 
support\",\n\"min_version\": \"2.3.4.0\"\n  },\n  {\n
\"name\": \"secure_zookeeper\",\n\"description\": \"Protect ZNodes with 
SASL acl in secure clusters\"
 ,\n\"min_version\": \"2.6.0.0\"\n  },\n  {\n\"name\": 
\"config_versioning\",\n\"description\": \"Configurable versions 
support\",\n\"min_version\": \"2.3.0.0\"\n  },\n  {\n
\"name\": \"datanode_non_root\",\n\"description\":

Re: Review Request 64397: AMBARI-22602. Add 'clusterSettings' and 'stackSettings' parameters in Execution Command.

2017-12-11 Thread Swapan Shridhar



> On Dec. 11, 2017, 5:44 p.m., Jayush Luniya wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/agent/ExecutionCommand.java
> > Lines 291 (patched)
> > 
> >
> > settings or clusterSettings instead of params?

Fixed.


> On Dec. 11, 2017, 5:44 p.m., Jayush Luniya wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/agent/ExecutionCommand.java
> > Lines 300 (patched)
> > 
> >
> > settings or stackSettings instead of params?

Fixed.


- Swapan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64397/#review193406
---


On Dec. 7, 2017, 8:11 a.m., Swapan Shridhar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64397/
> ---
> 
> (Updated Dec. 7, 2017, 8:11 a.m.)
> 
> 
> Review request for Ambari, Jayush Luniya and Madhuvanthi Radhakrishnan.
> 
> 
> Bugs: AMBARI-22602
> https://issues.apache.org/jira/browse/AMBARI-22602
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> AMBARI-22198 added "stack settings", and AMBARI-22196 introduced "cluster 
> settings" in Ambari.
> 
> This review adds 2 new parameters to Execution Command : **(1).** 
> clusterSettings and **(2).** stackSettings, and adds their respective set of 
> key-values pairs.
> 
> - This enables these parameters to be passed in to the ambari-agent's 
> **command*.json**.
> 
> 
> Test cases are tracked in : AMBARI-22603
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/agent/ExecutionCommand.java
>  5ee4bf6 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariCustomCommandExecutionHelper.java
>  e7dea06 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementControllerImpl.java
>  59e6622 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/ClientConfigResourceProvider.java
>  9043297 
>   
> ambari-server/src/main/java/org/apache/ambari/server/resources/RootLevelSettingsManager.java
>  8bd24f2 
> 
> 
> Diff: https://reviews.apache.org/r/64397/diff/1/
> 
> 
> Testing
> ---
> 
> Tested on Live cluster doing following : 
> (1). Start Service 
> (2). Restart Service 
> (3). Stop Service 
> (4). Refreshing client configs
> (5). Adding Client.
> 
> 
> **Snippet from command*.json as part of testing:**
> {
> ...
> ...
> "clusterSettings": {
> "security_enabled": "false",
> "hide_yarn_memory_widget": "false",
> "enable_external_ranger": "false",
> "override_uid": "true",
> "kerberos_domain": "EXAMPLE.COM",
> "one_dir_per_partition": "false",
> "repo_ubuntu_template": "{{package_type}} {{base_url}} 
> {{components}}",
> "ignore_groupsusers_create": "false",
> "alerts_repeat_tolerance": "1",
> "namenode_rolling_restart_timeout": "4200",
> "fetch_nonlocal_groups": "true",
> "manage_dirs_on_root": "true",
> "recovery_lifetime_max_count": "1024",
> "agent_mounts_ignore_list": "",
> "ignore_bad_mounts": "false",
> "recovery_window_in_minutes": "60",
> "sysprep_skip_copy_tarballs_hdfs": "false",
> "recovery_type": "AUTO_START",
> "user_group": "hadoop",
> "namenode_rolling_restart_safemode_exit_timeout": "3600",
> "recovery_retry_interval": "5",
> "sysprep_skip_copy_oozie_share_lib_to_hdfs": "false",
> "sysprep_skip_setup_jce": "false",
> "manage_hive_fsroot": "true",
> "service_check_type": "full",
> "recovery_enabled": "true",
> "recovery_max_count": "6",
> "sysprep_skip_create_users_and_groups": "false",
> "smokeuser_keytab": "/etc/security/keytabs/smokeuser.headless.keytab",
> "managed_hdfs_resource_property_names": "false",
> "smokeuser": "ambari-qa",
> "sysprep_skip_copy_fast_jar_hdfs": "false"
> },
> ...
> ...
> ...
> ...
> ...
> ...
> ...
> "stackSettings": {
> "stack_features": "{\n  \"HDP\": {\n\"stack_features\": [\n  
> {\n\"name\": \"snappy\",\n\"description\": \"Snappy 
> compressor/decompressor support\",\n\"min_version\": \"2.0.0.0\",\n   
>  \"max_version\": \"2.2.0.0\"\n  },\n  {\n\"name\": 
> \"lzo\",\n\"description\": \"LZO libraries support\",\n
> \"min_version\": \"2.2.1.0\"\n  },\n  {\n\"name\": 
> \"express_upgrade\",\n\"description\": \"Express upgrade support\",\n 
>\"min_version\": \"2.1.0.0\"\n  },\n

Re: Review Request 64397: AMBARI-22602. Add 'clusterSettings' and 'stackSettings' parameters in Execution Command.

2017-12-11 Thread Swapan Shridhar



> On Dec. 8, 2017, 8:01 p.m., Attila Doroszlai wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariCustomCommandExecutionHelper.java
> > Lines 436 (patched)
> > 
> >
> > * no need for explicit type arg in Maps.newHashMap()
> > * empty map should be created lazily in `else` below

Fixed.


> On Dec. 8, 2017, 8:01 p.m., Attila Doroszlai wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariCustomCommandExecutionHelper.java
> > Lines 438 (patched)
> > 
> >
> > According to IDEA there is no need to include the 
> > `clusterSettings.values() != null` check, as it is always true if 
> > `clusterSettings != null` is true.

Removed.


> On Dec. 8, 2017, 8:01 p.m., Attila Doroszlai wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariCustomCommandExecutionHelper.java
> > Lines 449-451 (patched)
> > 
> >
> > Please use `collect(toMap())` (as above) instead of `forEach` and `put`

Fixed.


> On Dec. 8, 2017, 8:01 p.m., Attila Doroszlai wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementControllerImpl.java
> > Lines 2676-2695 (patched)
> > 
> >
> > Can you please extract the logic to 2 utility methods to avoid code 
> > duplication among `AmbariCustomCommandExecutionHelper`, 
> > `AmbariManagementControllerImpl` and `ClientConfigResourceProvider`?

Refactored.


- Swapan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64397/#review193274
---


On Dec. 7, 2017, 8:11 a.m., Swapan Shridhar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64397/
> ---
> 
> (Updated Dec. 7, 2017, 8:11 a.m.)
> 
> 
> Review request for Ambari, Jayush Luniya and Madhuvanthi Radhakrishnan.
> 
> 
> Bugs: AMBARI-22602
> https://issues.apache.org/jira/browse/AMBARI-22602
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> AMBARI-22198 added "stack settings", and AMBARI-22196 introduced "cluster 
> settings" in Ambari.
> 
> This review adds 2 new parameters to Execution Command : **(1).** 
> clusterSettings and **(2).** stackSettings, and adds their respective set of 
> key-values pairs.
> 
> - This enables these parameters to be passed in to the ambari-agent's 
> **command*.json**.
> 
> 
> Test cases are tracked in : AMBARI-22603
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/agent/ExecutionCommand.java
>  5ee4bf6 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariCustomCommandExecutionHelper.java
>  e7dea06 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementControllerImpl.java
>  59e6622 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/ClientConfigResourceProvider.java
>  9043297 
>   
> ambari-server/src/main/java/org/apache/ambari/server/resources/RootLevelSettingsManager.java
>  8bd24f2 
> 
> 
> Diff: https://reviews.apache.org/r/64397/diff/1/
> 
> 
> Testing
> ---
> 
> Tested on Live cluster doing following : 
> (1). Start Service 
> (2). Restart Service 
> (3). Stop Service 
> (4). Refreshing client configs
> (5). Adding Client.
> 
> 
> **Snippet from command*.json as part of testing:**
> {
> ...
> ...
> "clusterSettings": {
> "security_enabled": "false",
> "hide_yarn_memory_widget": "false",
> "enable_external_ranger": "false",
> "override_uid": "true",
> "kerberos_domain": "EXAMPLE.COM",
> "one_dir_per_partition": "false",
> "repo_ubuntu_template": "{{package_type}} {{base_url}} 
> {{components}}",
> "ignore_groupsusers_create": "false",
> "alerts_repeat_tolerance": "1",
> "namenode_rolling_restart_timeout": "4200",
> "fetch_nonlocal_groups": "true",
> "manage_dirs_on_root": "true",
> "recovery_lifetime_max_count": "1024",
> "agent_mounts_ignore_list": "",
> "ignore_bad_mounts": "false",
> "recovery_window_in_minutes": "60",
> "sysprep_skip_copy_tarballs_hdfs": "false",
> "recovery_type": "AUTO_START",
> "user_group": "hadoop",
> "namenode_rolling_restart_safemode_exit_timeout": "3600",
> "recovery_retry_interval": "5",
> "sysprep_skip_copy_oozie_share_lib_to_hdfs": "false",
> "sysprep_skip_setup_jce": "false",
>

Re: Review Request 64397: AMBARI-22602. Add 'clusterSettings' and 'stackSettings' parameters in Execution Command.

2017-12-11 Thread Swapan Shridhar



> On Dec. 8, 2017, 11:41 p.m., Madhuvanthi Radhakrishnan wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/resources/RootLevelSettingsManager.java
> > Line 57 (original)
> > 
> >
> > Are we using this map anywhere?

Nope. Its part of the cleanup.


- Swapan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64397/#review193284
---


On Dec. 7, 2017, 8:11 a.m., Swapan Shridhar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64397/
> ---
> 
> (Updated Dec. 7, 2017, 8:11 a.m.)
> 
> 
> Review request for Ambari, Jayush Luniya and Madhuvanthi Radhakrishnan.
> 
> 
> Bugs: AMBARI-22602
> https://issues.apache.org/jira/browse/AMBARI-22602
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> AMBARI-22198 added "stack settings", and AMBARI-22196 introduced "cluster 
> settings" in Ambari.
> 
> This review adds 2 new parameters to Execution Command : **(1).** 
> clusterSettings and **(2).** stackSettings, and adds their respective set of 
> key-values pairs.
> 
> - This enables these parameters to be passed in to the ambari-agent's 
> **command*.json**.
> 
> 
> Test cases are tracked in : AMBARI-22603
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/agent/ExecutionCommand.java
>  5ee4bf6 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariCustomCommandExecutionHelper.java
>  e7dea06 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementControllerImpl.java
>  59e6622 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/ClientConfigResourceProvider.java
>  9043297 
>   
> ambari-server/src/main/java/org/apache/ambari/server/resources/RootLevelSettingsManager.java
>  8bd24f2 
> 
> 
> Diff: https://reviews.apache.org/r/64397/diff/1/
> 
> 
> Testing
> ---
> 
> Tested on Live cluster doing following : 
> (1). Start Service 
> (2). Restart Service 
> (3). Stop Service 
> (4). Refreshing client configs
> (5). Adding Client.
> 
> 
> **Snippet from command*.json as part of testing:**
> {
> ...
> ...
> "clusterSettings": {
> "security_enabled": "false",
> "hide_yarn_memory_widget": "false",
> "enable_external_ranger": "false",
> "override_uid": "true",
> "kerberos_domain": "EXAMPLE.COM",
> "one_dir_per_partition": "false",
> "repo_ubuntu_template": "{{package_type}} {{base_url}} 
> {{components}}",
> "ignore_groupsusers_create": "false",
> "alerts_repeat_tolerance": "1",
> "namenode_rolling_restart_timeout": "4200",
> "fetch_nonlocal_groups": "true",
> "manage_dirs_on_root": "true",
> "recovery_lifetime_max_count": "1024",
> "agent_mounts_ignore_list": "",
> "ignore_bad_mounts": "false",
> "recovery_window_in_minutes": "60",
> "sysprep_skip_copy_tarballs_hdfs": "false",
> "recovery_type": "AUTO_START",
> "user_group": "hadoop",
> "namenode_rolling_restart_safemode_exit_timeout": "3600",
> "recovery_retry_interval": "5",
> "sysprep_skip_copy_oozie_share_lib_to_hdfs": "false",
> "sysprep_skip_setup_jce": "false",
> "manage_hive_fsroot": "true",
> "service_check_type": "full",
> "recovery_enabled": "true",
> "recovery_max_count": "6",
> "sysprep_skip_create_users_and_groups": "false",
> "smokeuser_keytab": "/etc/security/keytabs/smokeuser.headless.keytab",
> "managed_hdfs_resource_property_names": "false",
> "smokeuser": "ambari-qa",
> "sysprep_skip_copy_fast_jar_hdfs": "false"
> },
> ...
> ...
> ...
> ...
> ...
> ...
> ...
> "stackSettings": {
> "stack_features": "{\n  \"HDP\": {\n\"stack_features\": [\n  
> {\n\"name\": \"snappy\",\n\"description\": \"Snappy 
> compressor/decompressor support\",\n\"min_version\": \"2.0.0.0\",\n   
>  \"max_version\": \"2.2.0.0\"\n  },\n  {\n\"name\": 
> \"lzo\",\n\"description\": \"LZO libraries support\",\n
> \"min_version\": \"2.2.1.0\"\n  },\n  {\n\"name\": 
> \"express_upgrade\",\n\"description\": \"Express upgrade support\",\n 
>\"min_version\": \"2.1.0.0\"\n  },\n  {\n\"name\": 
> \"rolling_upgrade\",\n\"description\": \"Rolling upgrade support\",\n 
>\"min_version\": \"2.2.0.0\"\n  },\n  {\n\"name\": 
> \"kafka_acl_migration_support\",\n\"description\": \"ACL migration 
> support\",\n

Re: Review Request 64242: AMBARI-22476: Modify bootstrap API to pass Ambari repo URL from UI and retrieve Host OS types

2017-12-11 Thread Nate Cole



> On Dec. 4, 2017, 11:20 a.m., Nate Cole wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/bootstrap/SshHostInfo.java
> > Lines 130-132 (patched)
> > 
> >
> > How many URLs are we talking about here.  If more than one, should this 
> > bet a Set?
> 
> Sonia Garudi wrote:
> ambariRepoUrls is a json string having ambari repo URLs for each os type 
> which is sent from the UI.
> 
> Nate Cole wrote:
> Can I see an example?
> 
> Sonia Garudi wrote:
> Example : 
> 
> "[{"os_type":"os1","ambari_repo":"http://ambari-repo1","hosts":["host1","host2"],"ambariRepoUIError":"","hasError":false},{"os_type":"os2","ambari_repo":"http://ambari-repo2","hosts":["host3"],"ambariRepoUIError":"","hasError":false}];
> The above json string consists of list of 2 objects; each containing the 
> os_type, ambari repository URL and the list of host names for the 
> corresponding os_type. The 2 other fields(i.e. ambariRepoUIError and 
> hasError) are used for UI validation.
> 
> Nate Cole wrote:
> This seems strange in that we end up with two different ways to "get" the 
> URL.  One is that when the UI does not recognize an OS, we ask for it and 
> (presumably) save it in ambari.properties.  Seems as though when we should 
> always get the value from ambari.properties and fail if it's not there.  
> Also, why are we setting all this information  If we have a command for 
> host1, host2, why would we need to know that os-type-2 goes on host3?
> 
> Sonia Garudi wrote:
> The bootstrap.py file first checks if the os type of a host is not same 
> as server os type. If its different it checks for the ambari repo property 
> for that os type in the ambari.properties file. 
> case 1 : If the property is not found, the bootstrap for that host fails 
> with code 44. The UI prompts for the ambari repo URL for os_type of hosts 
> which fail with code 44. These hosts are bootstrapped with the ambariRepoUrls 
> json string which contains the ambari repo urls. These URLs are then saved in 
> the ambari.properties file for the respective os type.
> case 2 : If the property is found in the properties file, the 
> installation will continue without showing a prompt for that os_type.
> 
> The hosts field in each object is used for removal of prompt when a host 
> is removed.
> Implementation of storing of ambari repo url in properties file in done 
> in upcoming patches.
> 
> Yussuf Shaikh wrote:
> Hi Nate,
> 
> Sonia just dumped the data structure used on step3 controller to the 
> bootstrap API, hence it contains the hostnames, but at server side only 
> os_type and corresponding repo URL is used.
> 
> We can just do away with hosts information in the json data which would 
> look like below:
> 
> [{"os_type":"os1","ambari_repo":"http://ambari-repo1"},{"os_type":"os2","ambari_repo":"http://ambari-repo2"}];
> 
> Let us know if you really need to change this in this patch or we could 
> change it in next patch (AMBARI-22394).

I was just looking for clarification of what's in this string.  If it's just a 
singular URL for the repo of that host, or what purpose the structure serves.

Think of it this way.  With no UI interaction at all, what should the 
bootstrapping do?  Would we make people create that odd json structure and pass 
it as a string?  I'm ok with this patch as-is, but we should be careful what is 
really being asked of the API here in the next patch.


- Nate


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64242/#review192705
---


On Dec. 1, 2017, 2:47 a.m., Sonia Garudi wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64242/
> ---
> 
> (Updated Dec. 1, 2017, 2:47 a.m.)
> 
> 
> Review request for Ambari and Nate Cole.
> 
> 
> Bugs: AMBARI-22476
> https://issues.apache.org/jira/browse/AMBARI-22476
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> The hosts failing with error code 44 are bootstrapped along with the ambari 
> repository URLs. The required java beans are changed to include the ambari 
> repository URLs to be passed to bootstrap.py.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/bootstrap/BSRunner.java 
> c7976ee5bb48209478258625532e3420973b4b8c 
>   
> ambari-server/src/main/java/org/apache/ambari/server/bootstrap/SshHostInfo.java
>  86888f499f15aa23807067ee1af745525b9d4619 
>   ambari-server/src/main/python/bootstrap.py 
> f1c53ce45316843fd38624536eed656b029a230a 
>   
> ambari-server/src/test/java/org/apache/ambari/server/bootstrap/BootStrapTest.java
>

Re: Review Request 64513: Update the default storm log search configuration (use wildcards)

2017-12-11 Thread Jungtaek Lim


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64513/#review193486
---


Ship it!




Is it safe to put same configuration value from Storm logfeeder config? Then 
LGTM.

- Jungtaek Lim


On 12월 11, 2017, 11:47 오후, Oliver Szabo wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64513/
> ---
> 
> (Updated 12월 11, 2017, 11:47 오후)
> 
> 
> Review request for Ambari, Krisztian Kasa, Miklos Gergely, and Sid Wagle.
> 
> 
> Bugs: AMBARI-22630
> https://issues.apache.org/jira/browse/AMBARI-22630
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> set the storm config defaults for logseach as other teams needs it.
> 
> this patch is for branch-2.6, but if cannot be delivered in 2.6.1, that can 
> be used in a hotfix
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/STORM/0.9.1/configuration/storm-logsearch-conf.xml
>  d485017 
> 
> 
> Diff: https://reviews.apache.org/r/64513/diff/1/
> 
> 
> Testing
> ---
> 
> not yet.
> 
> 
> Thanks,
> 
> Oliver Szabo
> 
>

Review Request 64513: Update the default storm log search configuration (use wildcards)

2017-12-11 Thread Oliver Szabo


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64513/
---

Review request for Ambari, Krisztian Kasa, Miklos Gergely, and Sid Wagle.


Bugs: AMBARI-22630
https://issues.apache.org/jira/browse/AMBARI-22630


Repository: ambari


Description
---

set the storm config defaults for logseach as other teams needs it.

this patch is for branch-2.6, but if cannot be delivered in 2.6.1, that can be 
used in a hotfix


Diffs
-

  
ambari-server/src/main/resources/common-services/STORM/0.9.1/configuration/storm-logsearch-conf.xml
 d485017 


Diff: https://reviews.apache.org/r/64513/diff/1/


Testing
---

not yet.


Thanks,

Oliver Szabo

Re: Review Request 64478: Blueprint V2 API: object model, resource provider, cluster creation

2017-12-11 Thread Madhuvanthi Radhakrishnan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64478/#review193429
---




ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementControllerImpl.java
Line 1029 (original), 1023 (patched)


Hi Attila,
Since we havent completely gotten rid of /cluster/configuration yet, the 
serviceId in the request can be null. The change you have made is what we will 
do eventually but right now this might break the existing feature branch 
deployments.



ambari-server/src/main/java/org/apache/ambari/server/controller/internal/BaseClusterRequest.java
Line 137 (original), 110 (patched)


Missing throws clause



ambari-server/src/main/java/org/apache/ambari/server/state/ConfigMergeHelper.java
Line 66 (original), 66 (patched)


If each service instance can have its own config properties then shouldnt 
it be serviceName (unique)?
Correct me if i am wrong.



ambari-server/src/main/java/org/apache/ambari/server/topology/HostGroupInfo.java
Line 194 (original), 200 (patched)


Missing throws clause



ambari-server/src/main/resources/Ambari-DDL-Derby-CREATE.sql
Line 42 (original)


Where are we setting this value? I saw we removed it from the StackManager 
code. So, in UI deployment it might fail.


- Madhuvanthi Radhakrishnan


On Dec. 9, 2017, 12:19 p.m., Attila Doroszlai wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64478/
> ---
> 
> (Updated Dec. 9, 2017, 12:19 p.m.)
> 
> 
> Review request for Ambari, Balázs Bence Sári, Jayush Luniya, Madhuvanthi 
> Radhakrishnan, Robert Nettleton, and Swapan Shridhar.
> 
> 
> Bugs: AMBARI-22253, AMBARI-22297 and AMBARI-22325
> https://issues.apache.org/jira/browse/AMBARI-22253
> https://issues.apache.org/jira/browse/AMBARI-22297
> https://issues.apache.org/jira/browse/AMBARI-22325
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Implementation of Blueprint V2 API POST that allows cluster creation.  
> Co-authored by Balázs Bence Sári and Attila Doroszlai.
> 
> Includes minor fixes for:
> 
>  * AMBARI-22196 -- yum repo file contains parsing errors due to initial 
> whitespace
>  * AMBARI-22249 -- AmbariContext.serviceGroupDependencyResourceProvider 
> initialization
>  * AMBARI-22164 -- wrong assignment due to leftover parameter name
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/ObjectNotFoundException.java
>  75c9f3b5d50102f3beeeaa8041178cc08e22e241 
>   
> ambari-server/src/main/java/org/apache/ambari/server/ParentObjectNotFoundException.java
>  15bd7cb6bbe5026bf0a5ae9d3fc0bec575137643 
>   
> ambari-server/src/main/java/org/apache/ambari/server/StackAccessException.java
>  b8bfff3991a4e49514801dbb480958941cff01ac 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java
>  fd43edfa88339cd94686ddc13822b36ab7c74d6f 
>   
> ambari-server/src/main/java/org/apache/ambari/server/checks/ServiceCheckValidityCheck.java
>  2582c71eb95255f0f8081e90473183243f12cdd0 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementControllerImpl.java
>  59e6622f708e7d8d27c9e28ce2ce58f014a4a5dc 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariServer.java
>  c88b23c8af374dd9bd695b70560079871f3366fd 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/ConfigurationRequest.java
>  d26cf16b59897573173ea258133a1c1b4e6f31f1 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/ControllerModule.java
>  f2ff181da0327a82adb67e1405ab9ae238b5c4d4 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/DeleteIdentityHandler.java
>  cc84ce24b5e769257ac0e8744a93e3e236bb04e3 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/RepositoryResponse.java
>  6d0f3fbd4288d31c3e077445e20c3b2bac696930 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/ServiceComponentHostRequest.java
>  ea082c66e128a30368d6be195f3ee270a08e8518 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/StackV2.java 
> 0b0329c31bb5800eb32ba4b437bd57a41a618dbe 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/StackV2Factory.java
>  784b3683ba9d3909bb9e6d900e4175b6a00aec9a 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/AbstractControllerResourceProvider.java
>  5104354fce7701fe023723c19d56d25e7adf5219 
>

Re: Review Request 64502: YARN Shuffle Service Can't Be Found On Client-Only Nodes After New Cluster Install

2017-12-11 Thread Nate Cole


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64502/#review193416
---


Ship it!




Ship It!

- Nate Cole


On Dec. 11, 2017, 12:04 p.m., Jonathan Hurley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64502/
> ---
> 
> (Updated Dec. 11, 2017, 12:04 p.m.)
> 
> 
> Review request for Ambari, Dmitro Lisnichenko and Nate Cole.
> 
> 
> Bugs: AMBARI-22628
> https://issues.apache.org/jira/browse/AMBARI-22628
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Installing a new cluster can create values in yarn-site.xml which have 
> {{None}} specified in the classpath for Spark
> 
> ```
> 
>   yarn.nodemanager.aux-services.spark2_shuffle.classpath
>   /usr/hdp/None/spark2/aux/*
> 
> 
>  
>   yarn.nodemanager.aux-services.spark_shuffle.classpath
>   /usr/hdp/None/spark/aux/*
> 
> 
> 
>   
> yarn.timeline-service.entity-group-fs-store.group-id-plugin-classpath
>   /usr/hdp/None/spark/hdpLib/*
> 
> ```
> 
> The cause for this is that YARN Clients on hosts without daemons never get a 
> restart command after the initial {{yarn-site.xml}}, and can never fill in 
> the correct values. This causes problems when jobs are run on these nodes:
> 
> ```
> 2017-12-04 10:16:41,789 INFO  service.AbstractService 
> (AbstractService.java:noteFailure(272)) - Service 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices failed 
> in state INITED; cause: java.lang.ClassNotFoundException: 
> org.apache.spark.network.yarn.YarnShuffleService
> java.lang.ClassNotFoundException: 
> org.apache.spark.network.yarn.YarnShuffleService
> ```
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/agent/ExecutionCommand.java
>  9d5e29ee8a 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/ClientConfigResourceProvider.java
>  a7c712bd1a 
>   ambari-server/src/main/java/org/apache/ambari/server/state/Cluster.java 
> 15efcd2173 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/cluster/ClusterImpl.java
>  ce328f91ff 
>   
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/params_linux.py
>  98141456c7 
> 
> 
> Diff: https://reviews.apache.org/r/64502/diff/3/
> 
> 
> Testing
> ---
> 
> Manual install via UI and Blueprint
> 
> 
> Thanks,
> 
> Jonathan Hurley
> 
>

Re: Review Request 64502: YARN Shuffle Service Can't Be Found On Client-Only Nodes After New Cluster Install

2017-12-11 Thread Dmitro Lisnichenko


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64502/#review193410
---


Ship it!




Ship It!

- Dmitro Lisnichenko


On Dec. 11, 2017, 7:04 p.m., Jonathan Hurley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64502/
> ---
> 
> (Updated Dec. 11, 2017, 7:04 p.m.)
> 
> 
> Review request for Ambari, Dmitro Lisnichenko and Nate Cole.
> 
> 
> Bugs: AMBARI-22628
> https://issues.apache.org/jira/browse/AMBARI-22628
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Installing a new cluster can create values in yarn-site.xml which have 
> {{None}} specified in the classpath for Spark
> 
> ```
> 
>   yarn.nodemanager.aux-services.spark2_shuffle.classpath
>   /usr/hdp/None/spark2/aux/*
> 
> 
>  
>   yarn.nodemanager.aux-services.spark_shuffle.classpath
>   /usr/hdp/None/spark/aux/*
> 
> 
> 
>   
> yarn.timeline-service.entity-group-fs-store.group-id-plugin-classpath
>   /usr/hdp/None/spark/hdpLib/*
> 
> ```
> 
> The cause for this is that YARN Clients on hosts without daemons never get a 
> restart command after the initial {{yarn-site.xml}}, and can never fill in 
> the correct values. This causes problems when jobs are run on these nodes:
> 
> ```
> 2017-12-04 10:16:41,789 INFO  service.AbstractService 
> (AbstractService.java:noteFailure(272)) - Service 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices failed 
> in state INITED; cause: java.lang.ClassNotFoundException: 
> org.apache.spark.network.yarn.YarnShuffleService
> java.lang.ClassNotFoundException: 
> org.apache.spark.network.yarn.YarnShuffleService
> ```
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/agent/ExecutionCommand.java
>  9d5e29ee8a 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/ClientConfigResourceProvider.java
>  a7c712bd1a 
>   ambari-server/src/main/java/org/apache/ambari/server/state/Cluster.java 
> 15efcd2173 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/cluster/ClusterImpl.java
>  ce328f91ff 
>   
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/params_linux.py
>  98141456c7 
> 
> 
> Diff: https://reviews.apache.org/r/64502/diff/3/
> 
> 
> Testing
> ---
> 
> Manual install via UI and Blueprint
> 
> 
> Thanks,
> 
> Jonathan Hurley
> 
>

Re: Review Request 64397: AMBARI-22602. Add 'clusterSettings' and 'stackSettings' parameters in Execution Command.

2017-12-11 Thread Jayush Luniya


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64397/#review193406
---




ambari-server/src/main/java/org/apache/ambari/server/agent/ExecutionCommand.java
Lines 291 (patched)


settings or clusterSettings instead of params?



ambari-server/src/main/java/org/apache/ambari/server/agent/ExecutionCommand.java
Lines 300 (patched)


settings or stackSettings instead of params?


- Jayush Luniya


On Dec. 7, 2017, 8:11 a.m., Swapan Shridhar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64397/
> ---
> 
> (Updated Dec. 7, 2017, 8:11 a.m.)
> 
> 
> Review request for Ambari, Jayush Luniya and Madhuvanthi Radhakrishnan.
> 
> 
> Bugs: AMBARI-22602
> https://issues.apache.org/jira/browse/AMBARI-22602
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> AMBARI-22198 added "stack settings", and AMBARI-22196 introduced "cluster 
> settings" in Ambari.
> 
> This review adds 2 new parameters to Execution Command : **(1).** 
> clusterSettings and **(2).** stackSettings, and adds their respective set of 
> key-values pairs.
> 
> - This enables these parameters to be passed in to the ambari-agent's 
> **command*.json**.
> 
> 
> Test cases are tracked in : AMBARI-22603
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/agent/ExecutionCommand.java
>  5ee4bf6 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariCustomCommandExecutionHelper.java
>  e7dea06 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementControllerImpl.java
>  59e6622 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/ClientConfigResourceProvider.java
>  9043297 
>   
> ambari-server/src/main/java/org/apache/ambari/server/resources/RootLevelSettingsManager.java
>  8bd24f2 
> 
> 
> Diff: https://reviews.apache.org/r/64397/diff/1/
> 
> 
> Testing
> ---
> 
> Tested on Live cluster doing following : 
> (1). Start Service 
> (2). Restart Service 
> (3). Stop Service 
> (4). Refreshing client configs
> (5). Adding Client.
> 
> 
> **Snippet from command*.json as part of testing:**
> {
> ...
> ...
> "clusterSettings": {
> "security_enabled": "false",
> "hide_yarn_memory_widget": "false",
> "enable_external_ranger": "false",
> "override_uid": "true",
> "kerberos_domain": "EXAMPLE.COM",
> "one_dir_per_partition": "false",
> "repo_ubuntu_template": "{{package_type}} {{base_url}} 
> {{components}}",
> "ignore_groupsusers_create": "false",
> "alerts_repeat_tolerance": "1",
> "namenode_rolling_restart_timeout": "4200",
> "fetch_nonlocal_groups": "true",
> "manage_dirs_on_root": "true",
> "recovery_lifetime_max_count": "1024",
> "agent_mounts_ignore_list": "",
> "ignore_bad_mounts": "false",
> "recovery_window_in_minutes": "60",
> "sysprep_skip_copy_tarballs_hdfs": "false",
> "recovery_type": "AUTO_START",
> "user_group": "hadoop",
> "namenode_rolling_restart_safemode_exit_timeout": "3600",
> "recovery_retry_interval": "5",
> "sysprep_skip_copy_oozie_share_lib_to_hdfs": "false",
> "sysprep_skip_setup_jce": "false",
> "manage_hive_fsroot": "true",
> "service_check_type": "full",
> "recovery_enabled": "true",
> "recovery_max_count": "6",
> "sysprep_skip_create_users_and_groups": "false",
> "smokeuser_keytab": "/etc/security/keytabs/smokeuser.headless.keytab",
> "managed_hdfs_resource_property_names": "false",
> "smokeuser": "ambari-qa",
> "sysprep_skip_copy_fast_jar_hdfs": "false"
> },
> ...
> ...
> ...
> ...
> ...
> ...
> ...
> "stackSettings": {
> "stack_features": "{\n  \"HDP\": {\n\"stack_features\": [\n  
> {\n\"name\": \"snappy\",\n\"description\": \"Snappy 
> compressor/decompressor support\",\n\"min_version\": \"2.0.0.0\",\n   
>  \"max_version\": \"2.2.0.0\"\n  },\n  {\n\"name\": 
> \"lzo\",\n\"description\": \"LZO libraries support\",\n
> \"min_version\": \"2.2.1.0\"\n  },\n  {\n\"name\": 
> \"express_upgrade\",\n\"description\": \"Express upgrade support\",\n 
>\"min_version\": \"2.1.0.0\"\n  },\n  {\n\"name\": 
> \"rolling_upgrade\",\n\"description\": \"Rolling upgrade support\",\n 
>\"min_version\": \"2.2.0.0\"\n  },\n  {\n\"name\": 
>

Re: Review Request 64502: YARN Shuffle Service Can't Be Found On Client-Only Nodes After New Cluster Install

2017-12-11 Thread Jonathan Hurley


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64502/
---

(Updated Dec. 11, 2017, 12:04 p.m.)


Review request for Ambari, Dmitro Lisnichenko and Nate Cole.


Changes
---

I realized that downloading configurations is also problematic since it's done 
on the Ambari server and not on the real cluster. As such, we should also pass 
down the component version structure in client config download commands.


Bugs: AMBARI-22628
https://issues.apache.org/jira/browse/AMBARI-22628


Repository: ambari


Description
---

Installing a new cluster can create values in yarn-site.xml which have {{None}} 
specified in the classpath for Spark

```

  yarn.nodemanager.aux-services.spark2_shuffle.classpath
  /usr/hdp/None/spark2/aux/*


 
  yarn.nodemanager.aux-services.spark_shuffle.classpath
  /usr/hdp/None/spark/aux/*



  
yarn.timeline-service.entity-group-fs-store.group-id-plugin-classpath
  /usr/hdp/None/spark/hdpLib/*

```

The cause for this is that YARN Clients on hosts without daemons never get a 
restart command after the initial {{yarn-site.xml}}, and can never fill in the 
correct values. This causes problems when jobs are run on these nodes:

```
2017-12-04 10:16:41,789 INFO  service.AbstractService 
(AbstractService.java:noteFailure(272)) - Service 
org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices failed 
in state INITED; cause: java.lang.ClassNotFoundException: 
org.apache.spark.network.yarn.YarnShuffleService
java.lang.ClassNotFoundException: 
org.apache.spark.network.yarn.YarnShuffleService
```


Diffs (updated)
-

  
ambari-server/src/main/java/org/apache/ambari/server/agent/ExecutionCommand.java
 9d5e29ee8a 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/ClientConfigResourceProvider.java
 a7c712bd1a 
  ambari-server/src/main/java/org/apache/ambari/server/state/Cluster.java 
15efcd2173 
  
ambari-server/src/main/java/org/apache/ambari/server/state/cluster/ClusterImpl.java
 ce328f91ff 
  
ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/params_linux.py
 98141456c7 


Diff: https://reviews.apache.org/r/64502/diff/3/

Changes: https://reviews.apache.org/r/64502/diff/2-3/


Testing
---

Manual install via UI and Blueprint


Thanks,

Jonathan Hurley

Re: Review Request 64502: YARN Shuffle Service Can't Be Found On Client-Only Nodes After New Cluster Install

2017-12-11 Thread Jonathan Hurley


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64502/#review193403
---




ambari-server/src/main/java/org/apache/ambari/server/state/cluster/ClusterImpl.java
Lines 2812-2841 (patched)


There is change to the logic here. We're only going to send down a 
component version if it advertises a version AND its been resolved.

There's no point in sending down a version if it's not trusted, as is the 
case with "latest" installations.


- Jonathan Hurley


On Dec. 11, 2017, 12:04 p.m., Jonathan Hurley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64502/
> ---
> 
> (Updated Dec. 11, 2017, 12:04 p.m.)
> 
> 
> Review request for Ambari, Dmitro Lisnichenko and Nate Cole.
> 
> 
> Bugs: AMBARI-22628
> https://issues.apache.org/jira/browse/AMBARI-22628
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Installing a new cluster can create values in yarn-site.xml which have 
> {{None}} specified in the classpath for Spark
> 
> ```
> 
>   yarn.nodemanager.aux-services.spark2_shuffle.classpath
>   /usr/hdp/None/spark2/aux/*
> 
> 
>  
>   yarn.nodemanager.aux-services.spark_shuffle.classpath
>   /usr/hdp/None/spark/aux/*
> 
> 
> 
>   
> yarn.timeline-service.entity-group-fs-store.group-id-plugin-classpath
>   /usr/hdp/None/spark/hdpLib/*
> 
> ```
> 
> The cause for this is that YARN Clients on hosts without daemons never get a 
> restart command after the initial {{yarn-site.xml}}, and can never fill in 
> the correct values. This causes problems when jobs are run on these nodes:
> 
> ```
> 2017-12-04 10:16:41,789 INFO  service.AbstractService 
> (AbstractService.java:noteFailure(272)) - Service 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices failed 
> in state INITED; cause: java.lang.ClassNotFoundException: 
> org.apache.spark.network.yarn.YarnShuffleService
> java.lang.ClassNotFoundException: 
> org.apache.spark.network.yarn.YarnShuffleService
> ```
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/agent/ExecutionCommand.java
>  9d5e29ee8a 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/ClientConfigResourceProvider.java
>  a7c712bd1a 
>   ambari-server/src/main/java/org/apache/ambari/server/state/Cluster.java 
> 15efcd2173 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/cluster/ClusterImpl.java
>  ce328f91ff 
>   
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/params_linux.py
>  98141456c7 
> 
> 
> Diff: https://reviews.apache.org/r/64502/diff/3/
> 
> 
> Testing
> ---
> 
> Manual install via UI and Blueprint
> 
> 
> Thanks,
> 
> Jonathan Hurley
> 
>

Re: Review Request 64242: AMBARI-22476: Modify bootstrap API to pass Ambari repo URL from UI and retrieve Host OS types

2017-12-11 Thread Yussuf Shaikh



> On Dec. 4, 2017, 4:20 p.m., Nate Cole wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/bootstrap/SshHostInfo.java
> > Lines 130-132 (patched)
> > 
> >
> > How many URLs are we talking about here.  If more than one, should this 
> > bet a Set?
> 
> Sonia Garudi wrote:
> ambariRepoUrls is a json string having ambari repo URLs for each os type 
> which is sent from the UI.
> 
> Nate Cole wrote:
> Can I see an example?
> 
> Sonia Garudi wrote:
> Example : 
> 
> "[{"os_type":"os1","ambari_repo":"http://ambari-repo1","hosts":["host1","host2"],"ambariRepoUIError":"","hasError":false},{"os_type":"os2","ambari_repo":"http://ambari-repo2","hosts":["host3"],"ambariRepoUIError":"","hasError":false}];
> The above json string consists of list of 2 objects; each containing the 
> os_type, ambari repository URL and the list of host names for the 
> corresponding os_type. The 2 other fields(i.e. ambariRepoUIError and 
> hasError) are used for UI validation.
> 
> Nate Cole wrote:
> This seems strange in that we end up with two different ways to "get" the 
> URL.  One is that when the UI does not recognize an OS, we ask for it and 
> (presumably) save it in ambari.properties.  Seems as though when we should 
> always get the value from ambari.properties and fail if it's not there.  
> Also, why are we setting all this information  If we have a command for 
> host1, host2, why would we need to know that os-type-2 goes on host3?
> 
> Sonia Garudi wrote:
> The bootstrap.py file first checks if the os type of a host is not same 
> as server os type. If its different it checks for the ambari repo property 
> for that os type in the ambari.properties file. 
> case 1 : If the property is not found, the bootstrap for that host fails 
> with code 44. The UI prompts for the ambari repo URL for os_type of hosts 
> which fail with code 44. These hosts are bootstrapped with the ambariRepoUrls 
> json string which contains the ambari repo urls. These URLs are then saved in 
> the ambari.properties file for the respective os type.
> case 2 : If the property is found in the properties file, the 
> installation will continue without showing a prompt for that os_type.
> 
> The hosts field in each object is used for removal of prompt when a host 
> is removed.
> Implementation of storing of ambari repo url in properties file in done 
> in upcoming patches.

Hi Nate,

Sonia just dumped the data structure used on step3 controller to the bootstrap 
API, hence it contains the hostnames, but at server side only os_type and 
corresponding repo URL is used.

We can just do away with hosts information in the json data which would look 
like below:
[{"os_type":"os1","ambari_repo":"http://ambari-repo1"},{"os_type":"os2","ambari_repo":"http://ambari-repo2"}];

Let us know if you really need to change this in this patch or we could change 
it in next patch (AMBARI-22394).


- Yussuf


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64242/#review192705
---


On Dec. 1, 2017, 7:47 a.m., Sonia Garudi wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64242/
> ---
> 
> (Updated Dec. 1, 2017, 7:47 a.m.)
> 
> 
> Review request for Ambari and Nate Cole.
> 
> 
> Bugs: AMBARI-22476
> https://issues.apache.org/jira/browse/AMBARI-22476
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> The hosts failing with error code 44 are bootstrapped along with the ambari 
> repository URLs. The required java beans are changed to include the ambari 
> repository URLs to be passed to bootstrap.py.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/bootstrap/BSRunner.java 
> c7976ee5bb48209478258625532e3420973b4b8c 
>   
> ambari-server/src/main/java/org/apache/ambari/server/bootstrap/SshHostInfo.java
>  86888f499f15aa23807067ee1af745525b9d4619 
>   ambari-server/src/main/python/bootstrap.py 
> f1c53ce45316843fd38624536eed656b029a230a 
>   
> ambari-server/src/test/java/org/apache/ambari/server/bootstrap/BootStrapTest.java
>  b9da013524fee556c6faa2f0236d2c0c4b032750 
>   ambari-server/src/test/python/TestBootstrap.py 
> bea47f498cbb523fbe24622c63b48918d433c717 
>   ambari-web/app/controllers/wizard/step3_controller.js 
> 5ff2cdc1b4bb9da20565fe387765b4fc6728e096 
>   ambari-web/test/controllers/wizard/step3_test.js 
> be1affe28fdcdb3404ebd3fb3fc6e61315483ed2 
> 
> 
> Diff: https://reviews.apache.org/r/64242/diff/1/
> 
> 
> Testing
> ---
> 
> Tests were successful in ambari-web and ambari-server.
> 
> 
> Thanks,
> 
> Sonia Garudi
> 
>

Re: Review Request 64501: Fix Stacks UT based on secured.json/default.json

2017-12-11 Thread Eugene Chekanskiy


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64501/#review193392
---


Ship it!




Ship It!

- Eugene Chekanskiy


On Гру. 11, 2017, 1:59 після полудня, Andrew Onischuk wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64501/
> ---
> 
> (Updated Гру. 11, 2017, 1:59 після полудня)
> 
> 
> Review request for Ambari, Sid Wagle and Vitalyi Brodetskyi.
> 
> 
> Bugs: AMBARI-22627
> https://issues.apache.org/jira/browse/AMBARI-22627
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> 
> Diffs
> -
> 
>   ambari-agent/src/main/python/ambari_agent/HostStatusReporter.py c27ee552f6 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/setup_ranger_plugin_xml.py
>  485c1a6743 
>   
> ambari-common/src/main/python/resource_management/libraries/script/script.py 
> 2c45b16f6c 
>   
> ambari-funtest/src/test/resources/stacks/HDP/2.0.7/services/HIVE/package/scripts/hive.py
>  f81375597c 
>   
> ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/package/scripts/ams.py
>  c0ad75f9b3 
>   
> ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/package/scripts/params_linux.py
>  b6030b9b0c 
>   
> ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/metadata.py
>  240ecae28b 
>   
> ambari-server/src/main/resources/common-services/ATLAS/0.7.0.3.0/package/scripts/metadata.py
>  36c4598a21 
>   
> ambari-server/src/main/resources/common-services/ATLAS/0.7.0.3.0/package/scripts/params.py
>  234da4c918 
>   
> ambari-server/src/main/resources/common-services/ATLAS/0.7.0.3.0/package/scripts/setup_ranger_atlas.py
>  c47c75c906 
>   
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive.py
>  645447d917 
>   
> ambari-server/src/main/resources/common-services/KAFKA/0.10.0.3.0/package/scripts/params.py
>  32ca71cabb 
>   
> ambari-server/src/main/resources/common-services/KNOX/0.5.0.3.0/package/scripts/knox.py
>  34b5643b8b 
>   
> ambari-server/src/main/resources/common-services/KNOX/0.5.0.3.0/package/scripts/params_linux.py
>  c4da932445 
>   
> ambari-server/src/main/resources/common-services/KNOX/0.5.0.3.0/package/scripts/setup_ranger_knox.py
>  c486ef7c8c 
>   
> ambari-server/src/main/resources/common-services/KNOX/0.5.0.3.0/package/scripts/status_params.py
>  8b441d5317 
>   
> ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie_client.py
>  164cc9130c 
>   
> ambari-server/src/main/resources/common-services/OOZIE/4.2.0.3.0/package/scripts/oozie.py
>  96f1f1afa9 
>   
> ambari-server/src/main/resources/common-services/OOZIE/4.2.0.3.0/package/scripts/oozie_client.py
>  6fbd66a95f 
>   
> ambari-server/src/main/resources/common-services/OOZIE/4.2.0.3.0/package/scripts/status_params.py
>  c559d1041f 
>   
> ambari-server/src/main/resources/common-services/RANGER/1.0.0.3.0/package/scripts/params.py
>  912329d277 
>   
> ambari-server/src/main/resources/common-services/RANGER/1.0.0.3.0/package/scripts/setup_ranger_xml.py
>  c84ae96d9a 
>   
> ambari-server/src/main/resources/common-services/RANGER/1.0.0.3.0/package/scripts/status_params.py
>  b71f8bd75d 
>   
> ambari-server/src/main/resources/common-services/RANGER_KMS/1.0.0.3.0/package/scripts/kms.py
>  6e4a1714f9 
>   
> ambari-server/src/main/resources/common-services/RANGER_KMS/1.0.0.3.0/package/scripts/params.py
>  9ef127bc89 
>   
> ambari-server/src/main/resources/common-services/RANGER_KMS/1.0.0.3.0/package/scripts/status_params.py
>  f9e903dc44 
>   
> ambari-server/src/main/resources/common-services/SPARK/2.2.0/package/scripts/params.py
>  a7df33a269 
>   
> ambari-server/src/main/resources/common-services/SQOOP/1.4.4.3.0/package/scripts/params_linux.py
>  b29e8ef86b 
>   
> ambari-server/src/main/resources/common-services/SQOOP/1.4.4.3.0/package/scripts/sqoop.py
>  436402cf4e 
>   
> ambari-server/src/main/resources/common-services/STORM/1.0.1.3.0/package/scripts/setup_ranger_storm.py
>  c04496eb99 
>   
> ambari-server/src/main/resources/common-services/STORM/1.0.1.3.0/package/scripts/status_params.py
>  d08880b137 
>   
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/hooks/after-INSTALL/scripts/params.py
>  dd66d58f24 
>   
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/hooks/after-INSTALL/scripts/shared_initialization.py
>  80941290d0 
>   
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/hooks/before-ANY/scripts/params.py
>  97e560a3f0 
>   
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/hooks/before-START/scripts/params.py
>  3dca7f01e4 
>   
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/HBASE/package/scripts/hbase.py
>  9a48a43138

Re: Review Request 64499: Reporting host status was broken by merge

2017-12-11 Thread Eugene Chekanskiy


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64499/#review193391
---


Ship it!




Ship It!

- Eugene Chekanskiy


On Гру. 11, 2017, 10:57 до полудня, Andrew Onischuk wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64499/
> ---
> 
> (Updated Гру. 11, 2017, 10:57 до полудня)
> 
> 
> Review request for Ambari, Sid Wagle and Vitalyi Brodetskyi.
> 
> 
> Bugs: AMBARI-22623
> https://issues.apache.org/jira/browse/AMBARI-22623
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> 
> Diffs
> -
> 
>   ambari-agent/src/main/python/ambari_agent/HostStatusReporter.py c27ee552f6 
> 
> 
> Diff: https://reviews.apache.org/r/64499/diff/1/
> 
> 
> Testing
> ---
> 
> mvn clean test
> 
> 
> Thanks,
> 
> Andrew Onischuk
> 
>

Re: Review Request 64498: NFSGateway start failing with error : "ERROR: You must be a privileged user in order to run a secure service."

2017-12-11 Thread Eugene Chekanskiy


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64498/#review193390
---


Ship it!




Ship It!

- Eugene Chekanskiy


On Гру. 11, 2017, 10:39 до полудня, Andrew Onischuk wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64498/
> ---
> 
> (Updated Гру. 11, 2017, 10:39 до полудня)
> 
> 
> Review request for Ambari, Sid Wagle and Vitalyi Brodetskyi.
> 
> 
> Bugs: AMBARI-22622
> https://issues.apache.org/jira/browse/AMBARI-22622
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Traceback (most recent call last):
>   File 
> "/var/lib/ambari-agent/cache/stack-hooks/before-START/scripts/hook.py", line 
> 43, in 
> BeforeStartHook().execute()
>   File 
> "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py",
>  line 368, in execute
> method(env)
>   File 
> "/var/lib/ambari-agent/cache/stack-hooks/before-START/scripts/hook.py", line 
> 34, in hook
> setup_hadoop()
>   File 
> "/var/lib/ambari-agent/cache/stack-hooks/before-START/scripts/shared_initialization.py",
>  line 45, in setup_hadoop
> cd_access='a',
>   File 
> "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 
> 166, in __init__
> self.env.run()
>   File 
> "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", 
> line 160, in run
> self.run_action(resource, action)
>   File 
> "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", 
> line 124, in run_action
> provider_action()
>   File 
> "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py",
>  line 185, in action_create
> sudo.makedirs(path, self.resource.mode or 0755)
>   File 
> "/usr/lib/python2.6/site-packages/resource_management/core/sudo.py", line 
> 102, in makedirs
> os.makedirs(path, mode)
>   File "/usr/lib64/python2.7/os.py", line 157, in makedirs
> mkdir(name, mode)
> OSError: [Errno 17] File exists: '/grid/0/log/hdfs'
> Traceback (most recent call last):
>   File 
> "/var/lib/ambari-agent/cache/common-services/HDFS/3.0.0.3.0/package/scripts/nfsgateway.py",
>  line 88, in 
> NFSGateway().execute()
>   File 
> "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py",
>  line 368, in execute
> method(env)
>   File 
> "/var/lib/ambari-agent/cache/common-services/HDFS/3.0.0.3.0/package/scripts/nfsgateway.py",
>  line 53, in start
> nfsgateway(action="start")
>   File 
> "/var/lib/ambari-agent/cache/common-services/HDFS/3.0.0.3.0/package/scripts/hdfs_nfsgateway.py",
>  line 74, in nfsgateway
> create_log_dir=True
>   File 
> "/var/lib/ambari-agent/cache/common-services/HDFS/3.0.0.3.0/package/scripts/utils.py",
>  line 273, in service
> Execute(daemon_cmd, not_if=process_id_exists_command, 
> environment=hadoop_env_exports)
>   File 
> "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 
> 166, in __init__
> self.env.run()
>   File 
> "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", 
> line 160, in run
> self.run_action(resource, action)
>   File 
> "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", 
> line 124, in run_action
> provider_action()
>   File 
> "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py",
>  line 262, in action_run
> tries=self.resource.tries, try_sleep=self.resource.try_sleep)
>   File 
> "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 
> 72, in inner
> result = function(command, **kwargs)
>   File 
> "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 
> 102, in checked_call
> tries=tries, try_sleep=try_sleep, 
> timeout_kill_strategy=timeout_kill_strategy)
>   File 
> "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 
> 150, in _call_wrapper
> result = _call(command, **kwargs_copy)
>   File 
> "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 
> 303, in _call
> raise ExecutionFailed(err_msg, code, out, err)
> resource_management.core.exceptions.ExecutionFailed: Execution of 
> 'ambari-sudo.sh su hdfs -l -s /bin/bash -c 'ulimit -c unlimited ;  
> /usr/hdp/3.0.0.0-555/hadoop/bin/hdfs --config 
> /usr/hdp/3.0.0.0-555/hadoop/conf --daemon start nfs3'' returned 1. WARNING: 
> HADOOP_PRIVILEGED_NFS_USER has been replaced by HDFS_NFS3_SECURE_USER. Using 
> value of HADOOP_PRIVILEGED_NFS_USER.
> WARNING:

Review Request 64502: Downloaded client configs have invalid values for spark properties in yarn-site.xml

2017-12-11 Thread Jonathan Hurley


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64502/
---

Review request for Ambari, Dmitro Lisnichenko and Nate Cole.


Bugs: AMBARI-22628
https://issues.apache.org/jira/browse/AMBARI-22628


Repository: ambari


Description
---

Downloaded client configs have invalid values for spark properties in 
yarn-site.xml.

Issue: spark_version variable is replaced by 'None' in the spark related config 
properties in yarn-site in the client configs downloaded.

Attaching downloaded yarn-site.xml
 [^yarn-site.xml] 

Properties with issue:

{code:java}

  yarn.nodemanager.aux-services.spark2_shuffle.classpath
  /usr/hdp/None/spark2/aux/*


 
  yarn.nodemanager.aux-services.spark_shuffle.classpath
  /usr/hdp/None/spark/aux/*



  
yarn.timeline-service.entity-group-fs-store.group-id-plugin-classpath
  /usr/hdp/None/spark/hdpLib/*

{code}

The cause for this is that YARN Clients on hosts without daemons never get a 
restart command after the initial {{yarn-site.xml}}, and can never fill in the 
correct values.


Diffs
-

  
ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/params_linux.py
 98141456c7 


Diff: https://reviews.apache.org/r/64502/diff/1/


Testing
---

Manual install via UI and Blueprint


Thanks,

Jonathan Hurley

Review Request 64501: Fix Stacks UT based on secured.json/default.json

2017-12-11 Thread Andrew Onischuk


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64501/
---

Review request for Ambari, Sid Wagle and Vitalyi Brodetskyi.


Bugs: AMBARI-22627
https://issues.apache.org/jira/browse/AMBARI-22627


Repository: ambari


Description
---


Diffs
-

  ambari-agent/src/main/python/ambari_agent/HostStatusReporter.py c27ee552f6 
  
ambari-common/src/main/python/resource_management/libraries/functions/setup_ranger_plugin_xml.py
 485c1a6743 
  ambari-common/src/main/python/resource_management/libraries/script/script.py 
2c45b16f6c 
  
ambari-funtest/src/test/resources/stacks/HDP/2.0.7/services/HIVE/package/scripts/hive.py
 f81375597c 
  
ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/package/scripts/ams.py
 c0ad75f9b3 
  
ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/package/scripts/params_linux.py
 b6030b9b0c 
  
ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/metadata.py
 240ecae28b 
  
ambari-server/src/main/resources/common-services/ATLAS/0.7.0.3.0/package/scripts/metadata.py
 36c4598a21 
  
ambari-server/src/main/resources/common-services/ATLAS/0.7.0.3.0/package/scripts/params.py
 234da4c918 
  
ambari-server/src/main/resources/common-services/ATLAS/0.7.0.3.0/package/scripts/setup_ranger_atlas.py
 c47c75c906 
  
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive.py
 645447d917 
  
ambari-server/src/main/resources/common-services/KAFKA/0.10.0.3.0/package/scripts/params.py
 32ca71cabb 
  
ambari-server/src/main/resources/common-services/KNOX/0.5.0.3.0/package/scripts/knox.py
 34b5643b8b 
  
ambari-server/src/main/resources/common-services/KNOX/0.5.0.3.0/package/scripts/params_linux.py
 c4da932445 
  
ambari-server/src/main/resources/common-services/KNOX/0.5.0.3.0/package/scripts/setup_ranger_knox.py
 c486ef7c8c 
  
ambari-server/src/main/resources/common-services/KNOX/0.5.0.3.0/package/scripts/status_params.py
 8b441d5317 
  
ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie_client.py
 164cc9130c 
  
ambari-server/src/main/resources/common-services/OOZIE/4.2.0.3.0/package/scripts/oozie.py
 96f1f1afa9 
  
ambari-server/src/main/resources/common-services/OOZIE/4.2.0.3.0/package/scripts/oozie_client.py
 6fbd66a95f 
  
ambari-server/src/main/resources/common-services/OOZIE/4.2.0.3.0/package/scripts/status_params.py
 c559d1041f 
  
ambari-server/src/main/resources/common-services/RANGER/1.0.0.3.0/package/scripts/params.py
 912329d277 
  
ambari-server/src/main/resources/common-services/RANGER/1.0.0.3.0/package/scripts/setup_ranger_xml.py
 c84ae96d9a 
  
ambari-server/src/main/resources/common-services/RANGER/1.0.0.3.0/package/scripts/status_params.py
 b71f8bd75d 
  
ambari-server/src/main/resources/common-services/RANGER_KMS/1.0.0.3.0/package/scripts/kms.py
 6e4a1714f9 
  
ambari-server/src/main/resources/common-services/RANGER_KMS/1.0.0.3.0/package/scripts/params.py
 9ef127bc89 
  
ambari-server/src/main/resources/common-services/RANGER_KMS/1.0.0.3.0/package/scripts/status_params.py
 f9e903dc44 
  
ambari-server/src/main/resources/common-services/SPARK/2.2.0/package/scripts/params.py
 a7df33a269 
  
ambari-server/src/main/resources/common-services/SQOOP/1.4.4.3.0/package/scripts/params_linux.py
 b29e8ef86b 
  
ambari-server/src/main/resources/common-services/SQOOP/1.4.4.3.0/package/scripts/sqoop.py
 436402cf4e 
  
ambari-server/src/main/resources/common-services/STORM/1.0.1.3.0/package/scripts/setup_ranger_storm.py
 c04496eb99 
  
ambari-server/src/main/resources/common-services/STORM/1.0.1.3.0/package/scripts/status_params.py
 d08880b137 
  
ambari-server/src/main/resources/stacks/BIGTOP/0.8/hooks/after-INSTALL/scripts/params.py
 dd66d58f24 
  
ambari-server/src/main/resources/stacks/BIGTOP/0.8/hooks/after-INSTALL/scripts/shared_initialization.py
 80941290d0 
  
ambari-server/src/main/resources/stacks/BIGTOP/0.8/hooks/before-ANY/scripts/params.py
 97e560a3f0 
  
ambari-server/src/main/resources/stacks/BIGTOP/0.8/hooks/before-START/scripts/params.py
 3dca7f01e4 
  
ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/HBASE/package/scripts/hbase.py
 9a48a43138 
  
ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/HDFS/package/scripts/hdfs.py
 33543c42ed 
  
ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/HDFS/package/scripts/params.py
 3a711c721a 
  
ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/HIVE/package/scripts/hcat.py
 af75d3691c 
  
ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/HIVE/package/scripts/hive.py
 2e382ff962 
  
ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/HIVE/package/scripts/params.py
 115949145f 
  
ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/HIVE/package/scripts/webhcat.py
 9a7fc0c44c

Review Request 64499: Reporting host status was broken by merge

2017-12-11 Thread Andrew Onischuk


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64499/
---

Review request for Ambari, Sid Wagle and Vitalyi Brodetskyi.


Bugs: AMBARI-22623
https://issues.apache.org/jira/browse/AMBARI-22623


Repository: ambari


Description
---


Diffs
-

  ambari-agent/src/main/python/ambari_agent/HostStatusReporter.py c27ee552f6 


Diff: https://reviews.apache.org/r/64499/diff/1/


Testing
---

mvn clean test


Thanks,

Andrew Onischuk

Review Request 64498: NFSGateway start failing with error : "ERROR: You must be a privileged user in order to run a secure service."

2017-12-11 Thread Andrew Onischuk


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64498/
---

Review request for Ambari, Sid Wagle and Vitalyi Brodetskyi.


Bugs: AMBARI-22622
https://issues.apache.org/jira/browse/AMBARI-22622


Repository: ambari


Description
---

Traceback (most recent call last):
  File 
"/var/lib/ambari-agent/cache/stack-hooks/before-START/scripts/hook.py", line 
43, in 
BeforeStartHook().execute()
  File 
"/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py",
 line 368, in execute
method(env)
  File 
"/var/lib/ambari-agent/cache/stack-hooks/before-START/scripts/hook.py", line 
34, in hook
setup_hadoop()
  File 
"/var/lib/ambari-agent/cache/stack-hooks/before-START/scripts/shared_initialization.py",
 line 45, in setup_hadoop
cd_access='a',
  File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", 
line 166, in __init__
self.env.run()
  File 
"/usr/lib/python2.6/site-packages/resource_management/core/environment.py", 
line 160, in run
self.run_action(resource, action)
  File 
"/usr/lib/python2.6/site-packages/resource_management/core/environment.py", 
line 124, in run_action
provider_action()
  File 
"/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py",
 line 185, in action_create
sudo.makedirs(path, self.resource.mode or 0755)
  File "/usr/lib/python2.6/site-packages/resource_management/core/sudo.py", 
line 102, in makedirs
os.makedirs(path, mode)
  File "/usr/lib64/python2.7/os.py", line 157, in makedirs
mkdir(name, mode)
OSError: [Errno 17] File exists: '/grid/0/log/hdfs'
Traceback (most recent call last):
  File 
"/var/lib/ambari-agent/cache/common-services/HDFS/3.0.0.3.0/package/scripts/nfsgateway.py",
 line 88, in 
NFSGateway().execute()
  File 
"/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py",
 line 368, in execute
method(env)
  File 
"/var/lib/ambari-agent/cache/common-services/HDFS/3.0.0.3.0/package/scripts/nfsgateway.py",
 line 53, in start
nfsgateway(action="start")
  File 
"/var/lib/ambari-agent/cache/common-services/HDFS/3.0.0.3.0/package/scripts/hdfs_nfsgateway.py",
 line 74, in nfsgateway
create_log_dir=True
  File 
"/var/lib/ambari-agent/cache/common-services/HDFS/3.0.0.3.0/package/scripts/utils.py",
 line 273, in service
Execute(daemon_cmd, not_if=process_id_exists_command, 
environment=hadoop_env_exports)
  File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", 
line 166, in __init__
self.env.run()
  File 
"/usr/lib/python2.6/site-packages/resource_management/core/environment.py", 
line 160, in run
self.run_action(resource, action)
  File 
"/usr/lib/python2.6/site-packages/resource_management/core/environment.py", 
line 124, in run_action
provider_action()
  File 
"/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py",
 line 262, in action_run
tries=self.resource.tries, try_sleep=self.resource.try_sleep)
  File 
"/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 72, 
in inner
result = function(command, **kwargs)
  File 
"/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 102, 
in checked_call
tries=tries, try_sleep=try_sleep, 
timeout_kill_strategy=timeout_kill_strategy)
  File 
"/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 150, 
in _call_wrapper
result = _call(command, **kwargs_copy)
  File 
"/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 303, 
in _call
raise ExecutionFailed(err_msg, code, out, err)
resource_management.core.exceptions.ExecutionFailed: Execution of 
'ambari-sudo.sh su hdfs -l -s /bin/bash -c 'ulimit -c unlimited ;  
/usr/hdp/3.0.0.0-555/hadoop/bin/hdfs --config /usr/hdp/3.0.0.0-555/hadoop/conf 
--daemon start nfs3'' returned 1. WARNING: HADOOP_PRIVILEGED_NFS_USER has been 
replaced by HDFS_NFS3_SECURE_USER. Using value of HADOOP_PRIVILEGED_NFS_USER.
WARNING: HADOOP_NFS3_OPTS has been replaced by HDFS_NFS3_OPTS. Using value 
of HADOOP_NFS3_OPTS.
ERROR: You must be a privileged user in order to run a secure service.
Traceback (most recent call last):
  File 
"/var/lib/ambari-agent/cache/common-services/HDFS/3.0.0.3.0/package/scripts/nfsgateway.py",
 line 88, in 
NFSGateway().execute()
  File 
"/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py",
 line 368, in execute
method(env)
  File 
"/var/lib/ambari-agent/cache/common-services/HDFS/3.0.0.3.0/package/scripts/nfsgateway.py",
 line 53, in start

Re: Review Request 64478: Blueprint V2 API: object model, resource provider, cluster creation

Re: Review Request 64528: Zeppelin Interpreter settings are getting updated after zeppelin restart

Re: Review Request 64528: Zeppelin Interpreter settings are getting updated after zeppelin restart

Review Request 64528: Zeppelin Interpreter settings are getting updated after zeppelin restart

Re: Review Request 64513: Update the default storm log search configuration (use wildcards)

Re: Review Request 64397: AMBARI-22602. Add 'clusterSettings' and 'stackSettings' parameters in Execution Command.

Re: Review Request 64397: AMBARI-22602. Add 'clusterSettings' and 'stackSettings' parameters in Execution Command.

Re: Review Request 64397: AMBARI-22602. Add 'clusterSettings' and 'stackSettings' parameters in Execution Command.

Re: Review Request 64397: AMBARI-22602. Add 'clusterSettings' and 'stackSettings' parameters in Execution Command.

Re: Review Request 64242: AMBARI-22476: Modify bootstrap API to pass Ambari repo URL from UI and retrieve Host OS types

Re: Review Request 64513: Update the default storm log search configuration (use wildcards)

Review Request 64513: Update the default storm log search configuration (use wildcards)

Re: Review Request 64478: Blueprint V2 API: object model, resource provider, cluster creation

Re: Review Request 64502: YARN Shuffle Service Can't Be Found On Client-Only Nodes After New Cluster Install

Re: Review Request 64502: YARN Shuffle Service Can't Be Found On Client-Only Nodes After New Cluster Install

Re: Review Request 64397: AMBARI-22602. Add 'clusterSettings' and 'stackSettings' parameters in Execution Command.

Re: Review Request 64502: YARN Shuffle Service Can't Be Found On Client-Only Nodes After New Cluster Install

Re: Review Request 64502: YARN Shuffle Service Can't Be Found On Client-Only Nodes After New Cluster Install

Re: Review Request 64242: AMBARI-22476: Modify bootstrap API to pass Ambari repo URL from UI and retrieve Host OS types

Re: Review Request 64501: Fix Stacks UT based on secured.json/default.json

Re: Review Request 64499: Reporting host status was broken by merge

Re: Review Request 64498: NFSGateway start failing with error : "ERROR: You must be a privileged user in order to run a secure service."

Review Request 64502: Downloaded client configs have invalid values for spark properties in yarn-site.xml

Review Request 64501: Fix Stacks UT based on secured.json/default.json

Review Request 64499: Reporting host status was broken by merge

Review Request 64498: NFSGateway start failing with error : "ERROR: You must be a privileged user in order to run a secure service."

26 matches

Site Navigation

Mail list logo

Footer information