date:20160321

Re: Review Request 45150: AMBARI-15476 : [AMS / Grafana] Rate calculation with sum() is wrong

2016-03-21 Thread Aravindan Vijayan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45150/
---

(Updated March 22, 2016, 5:08 a.m.)


Review request for Ambari, Dmytro Sen, Sumit Mohanty, and Sid Wagle.


Changes
---

Stack Advisor test changes.


Bugs: AMBARI-15476
https://issues.apache.org/jira/browse/AMBARI-15476


Repository: ambari


Description
---

Fix TimelineMetricAggregatorSecond such that time slices with no datapoints for 
a series are given interpolated values based on nearby values.

Change METRIC_RECORD TTL to 2 days in distributed mode.

Add stack advisor recommendation for AMS HBase log directory based on AMS log 
directory.


Diffs (updated)
-

  
ambari-metrics/ambari-metrics-common/src/main/java/org/apache/hadoop/metrics2/sink/timeline/PostProcessingUtil.java
 PRE-CREATION 
  
ambari-metrics/ambari-metrics-common/src/test/java/org/apache/hadoop/metrics2/sink/timeline/cache/PostProcessingUtilTest.java
 PRE-CREATION 
  
ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/TimelineMetricClusterAggregatorSecond.java
 e8e16a7 
  
ambari-metrics/ambari-metrics-timelineservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/TimelineMetricClusterAggregatorSecondTest.java
 PRE-CREATION 
  
ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog222.java
 6c8d153 
  
ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/configuration/ams-hbase-env.xml
 b99d1f9 
  
ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/configuration/ams-site.xml
 7a8984a 
  ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py 
d33e081 
  
ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog222Test.java
 599ac3e 
  ambari-server/src/test/python/stacks/2.2/common/test_stack_advisor.py 41da868 

Diff: https://reviews.apache.org/r/45150/diff/


Testing
---

Manual testing done.

Added unit tests.


Thanks,

Aravindan Vijayan

Re: Review Request 45150: AMBARI-15476 : [AMS / Grafana] Rate calculation with sum() is wrong

2016-03-21 Thread Aravindan Vijayan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45150/
---

(Updated March 22, 2016, 4:37 a.m.)


Review request for Ambari, Dmytro Sen, Sumit Mohanty, and Sid Wagle.


Changes
---

Rest of the files.


Bugs: AMBARI-15476
https://issues.apache.org/jira/browse/AMBARI-15476


Repository: ambari


Description
---

Fix TimelineMetricAggregatorSecond such that time slices with no datapoints for 
a series are given interpolated values based on nearby values.

Change METRIC_RECORD TTL to 2 days in distributed mode.

Add stack advisor recommendation for AMS HBase log directory based on AMS log 
directory.


Diffs (updated)
-

  
ambari-metrics/ambari-metrics-common/src/main/java/org/apache/hadoop/metrics2/sink/timeline/PostProcessingUtil.java
 PRE-CREATION 
  
ambari-metrics/ambari-metrics-common/src/test/java/org/apache/hadoop/metrics2/sink/timeline/cache/PostProcessingUtilTest.java
 PRE-CREATION 
  
ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/TimelineMetricClusterAggregatorSecond.java
 e8e16a7 
  
ambari-metrics/ambari-metrics-timelineservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/TimelineMetricClusterAggregatorSecondTest.java
 PRE-CREATION 
  
ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog222.java
 6c8d153 
  
ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/configuration/ams-hbase-env.xml
 b99d1f9 
  
ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/configuration/ams-site.xml
 7a8984a 
  ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py 
d33e081 
  
ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog222Test.java
 599ac3e 

Diff: https://reviews.apache.org/r/45150/diff/


Testing
---

Manual testing done.

Added unit tests.


Thanks,

Aravindan Vijayan

Re: Review Request 45141: Apply the stack featurization prototype detailed on AMBARI-13364 to ZOOKEEPER service.

2016-03-21 Thread Jayush Luniya


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45141/#review124702
---




ambari-common/src/main/python/resource_management/libraries/functions/stack_features.py
 (line 32)


Use default function instead? 

  from resource_management.libraries.functions.default import default
  stack_features_config = 
default("/configurations/cluster-env/stack_tools", None)
  data = _DEFAULT_STACK_FEATURES
  if stack_features_config:
data = json.loads(stack_features_config)



ambari-common/src/main/python/resource_management/libraries/functions/stack_features.py
 (line 33)


To avoid issues with ambari-server upgrade, lets set a 
DEFAULT_STACK_FEATURES same as what we have for HDP until we fix the 
ambari-server upgrade scenario (i.e. on ambari-server upgrade we will not have 
stack_features config in the cluster). (See stack_tools.py, copy_tarball.py as 
reference).



ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/params_linux.py
 (line 35)


Add:-

stack_root = Script.get_stack_root()



ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/params_linux.py
 (line 55)


Replace "/usr/hdp" with "{stack_root}"



ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/params_linux.py
 (line 58)


Leave config_dir logic as is as noted in my comment on status_params.py



ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/status_params.py
 


We need to define config_dir in status_params as it will cause issue during 
zookeeper_server::security_status checks as only status_params is imported in 
security_status()

  def security_status(self, env):
import status_params
env.set_params(status_params)
if status_params.security_enabled:   
...
  security_params = get_params_from_filesystem(status_params.config_dir,
   {'zookeeper_jaas.conf': 
FILE_TYPE_JAAS_CONF})



ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/zookeeper_server.py
 (line 132)


We are not importing params, only status_params (See line#115). Lets keep 
this as is.



ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_features.json
 (line 3)


I believe max_version for snappy should be 2.2.0.0?


- Jayush Luniya


On March 22, 2016, 1:10 a.m., Juanjo  Marron wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/45141/
> ---
> 
> (Updated March 22, 2016, 1:10 a.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez and Jayush Luniya.
> 
> 
> Bugs: AMBARI-14435
> https://issues.apache.org/jira/browse/AMBARI-14435
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Apply the  stack featurization prototype detailed on AMBARI-13364 to 
> ZOOKEEPER service plus the comments and reviews received on:
> https://reviews.apache.org/r/44933/
> 
> This patch removes the hardcoded stack versions in ZK common-services code 
> (centralized version and parameterized stack configurations will be faced in 
> other JIRAS)
> 
> stack_features is now a property in .json format in 
> HDP/2.0.6/properties/stack_features.json
> With the structure:
> {
> "stack_features": [
> { "name": "feature1", "description" : "Feature1 support", "min_version" : 
> "Y.Y.Y.Y" , "max_version" : "X.X.X.X"}
> , 
> ...
> ]
> }
> where min_version/max_version are optional constraints.
> 
> The upgrade constants, such has ROLLING_UPGRADE = "rolling_upgrade" has been 
> added to in a new StackFeature in 
> resource_management/libraries/functions/constants.py
> 
> It is added as part of the HDP stack configurations on 
> /HDP/2.0.6/configuration/cluster-env.xml, introducing a new stack_features 
> property
> 
> New resource_management/libraries/functions/stack_fetaures.py has been 
> introduced to parse the json file and called from service code to check if 
> the stack supports the required feature.
> 
> The stack version used in params_linux.py comparison is  
> stack_version_formatted = hostLevelParams\stack_version
> 
> 
> Diffs
> -
> 
>   
>

Re: Review Request 44933: Apply the stack featurization prototype detailed on AMBARI-13364 to ZOOKEEPER service.

2016-03-21 Thread Juanjo Marron



> On March 21, 2016, 8:01 p.m., Jayush Luniya wrote:
> > ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/zookeeper.py,
> >  line 40
> > 
> >
> > Add constants to 
> > ambari/ambari-common/src/main/python/resource_management/libraries/functions/constants.py
> > 
> > class StackFeatures:
> >   ROLLING_UPGRADE = "rolling_upgrade"

Added new review request with all the new design changes: 
https://reviews.apache.org/r/45141/

Im not sure how is the right procedure in this case, but it was hard to me to 
keep track of the diff changes after the new prototype design


- Juanjo


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44933/#review124636
---


On March 22, 2016, 1:03 a.m., Juanjo  Marron wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44933/
> ---
> 
> (Updated March 22, 2016, 1:03 a.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez and Jayush Luniya.
> 
> 
> Bugs: AMBARI-14435
> https://issues.apache.org/jira/browse/AMBARI-14435
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Apply the  stack featurization prototype  detailed on AMBARI-13364 to 
> ZOOKEEPER service.
> This patch removes the hardcoded stack versions in ZK common-services code 
> (centralized version and parameterized stack configurations will be faced in 
> other JIRAS)
> stack_features is now a property in .json format in 
> configuration/cluster-env.xml for each stack version.
> With the structure:
> {
> "stack_features": [
> { "name": "feature1", "description" : "Feature1 support", "min_version" : 
> "Y.Y.Y.Y" , "max_version" : "X.X.X.X"}
> , 
> ...
> ]
> }
> where min_version/max_version are optional constraints.
> New resource_management/libraries/functions/stack_fetaures.py has been 
> introduced to parse the json file and called from service code to check if 
> the stack supports the required feature.
> 
> 
> Diffs
> -
> 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/stack_fetaures.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/params_linux.py
>  9ff9125 
>   
> ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/status_params.py
>  d18e4d7 
>   
> ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/zookeeper.py
>  5712ce4 
>   
> ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/zookeeper_client.py
>  25ace24 
>   
> ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/zookeeper_server.py
>  d5b6898 
>   
> ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/zookeeper_service.py
>  6fe0772 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/configuration/cluster-env.xml
>  3fb82e9 
>   
> ambari-server/src/main/resources/stacks/HDP/2.2/configuration/cluster-env.xml 
> f1fa4de 
>   
> ambari-server/src/main/resources/stacks/HDP/2.3/configuration/cluster-env.xml 
> PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/44933/diff/
> 
> 
> Testing
> ---
> 
> Zookeeper fresh installation has been tested using HDP stack 2.3 and 
> AMBARI-13364 branch code.
> 
> 
> Thanks,
> 
> Juanjo  Marron
> 
>

Re: Review Request 45140: Hive interactive component should be created/deleted when interactive query is enabled/disabled

2016-03-21 Thread Jaimin Jetly


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45140/
---

(Updated March 22, 2016, 1:30 a.m.)


Review request for Ambari, Srimanth Gunturi, Swapan Shridhar, and Yusaku Sako.


Changes
---

small fix related to deletion of hive interactive component on disbaling 
interactive query.


Bugs: AMBARI-15503
https://issues.apache.org/jira/browse/AMBARI-15503


Repository: ambari


Description
---

1. Show dialog to user for choosing HiveServer2 interactive host component when 
Interactive query is turned ON
2. If Manual kerberos is enabled on the cluster then the dialog should ask user 
to add appropriate keytabs for newly installed host components
3. HiveServer2 interactive host-component and its dependencies should be 
created and started when "Enable Interactive" flag is turned ON.
4. HiveServer2 interactive host-component should be stopped and deleted when 
"Enable Interactive" flag is turned OFF.
5. UI should ask user to install slider service if not present in the cluster
6. "Enable Interactive" flag should be disabled when installing another service.


Diffs (updated)
-

  ambari-server/src/main/resources/stacks/HDP/2.6/role_command_order.json 
PRE-CREATION 
  
ambari-server/src/main/resources/stacks/HDP/2.6/services/HIVE/configuration/hive-interactive-env.xml
 PRE-CREATION 
  
ambari-server/src/main/resources/stacks/HDP/2.6/services/HIVE/configuration/hive-interactive-site.xml
 fb55d2d 
  
ambari-server/src/main/resources/stacks/HDP/2.6/services/HIVE/configuration/llap-env.xml
 24a95cf 
  ambari-server/src/main/resources/stacks/HDP/2.6/services/HIVE/metainfo.xml 
ff0f318 
  
ambari-server/src/main/resources/stacks/HDP/2.6/services/HIVE/themes/theme.json 
26b9532 
  
ambari-server/src/main/resources/stacks/HDP/2.6/services/HIVE/themes/theme_version_2.json
 PRE-CREATION 
  ambari-web/app/controllers/main/service/info/configs.js 93775fb 
  ambari-web/app/controllers/wizard/step7/assign_master_controller.js 496968b 
  ambari-web/app/controllers/wizard/step7_controller.js 06488b1 
  ambari-web/app/mappers/configs/themes_mapper.js 3e68e67 
  ambari-web/app/messages.js 421143a 
  ambari-web/app/mixins.js 0a686b3 
  ambari-web/app/mixins/common/configs/configs_saver.js 82741fa 
  ambari-web/app/mixins/main/service/configs/component_actions_by_configs.js 
PRE-CREATION 
  ambari-web/app/mixins/wizard/assign_master_components.js 78af279 
  ambari-web/app/models/configs/theme/config_action.js 288c674 
  ambari-web/app/templates/common/assign_master_components.hbs ea77c73 
  ambari-web/app/templates/common/configs/widgets/label_config_widget.hbs 
PRE-CREATION 
  ambari-web/app/utils/ajax/ajax.js 4e7b045 
  ambari-web/app/utils/configs/theme/theme.js f0cf966 
  ambari-web/app/views.js 8e8154c 
  ambari-web/app/views/common/assign_master_components_view.js da45ee9 
  ambari-web/app/views/common/configs/service_config_layout_tab_view.js def75d1 
  ambari-web/app/views/common/configs/widgets/config_widget_view.js 3ad9e3b 
  ambari-web/app/views/common/configs/widgets/label_config_widget_view.js 
PRE-CREATION 
  ambari-web/app/views/common/configs/widgets/time_interval_spinner_view.js 
8663ac3 
  ambari-web/app/views/wizard/step7/assign_master_view.js 2bc2ce3 
  ambari-web/test/controllers/wizard/step5_test.js 4140210 
  ambari-web/test/mixins/wizard/assign_master_components_test.js ecdb456 

Diff: https://reviews.apache.org/r/45140/diff/


Testing
---

Manually tested the patch on Installer, Add Service wizard and service->config 
page
Verified that all ambari-web unit tests passes:

  24639 tests complete (34 seconds)
  145 tests pending


Thanks,

Jaimin Jetly

Re: Review Request 44986: AMBARI-15474: Listen for changes to auto-start configuration and send them to the agent during heartbeats

2016-03-21 Thread Nahappan Somasundaram



> On March 21, 2016, 5:38 p.m., Alejandro Fernandez wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/agent/HeartBeatHandler.java,
> >  line 286
> > 
> >
> > It's possible for this to be called if clusterName is still null.
> 
> Nahappan Somasundaram wrote:
> Yes, there is a possibility. Before creating a cluster, if we register 
> the agent using the API and start it, it will attempt to register with the 
> server. At this time the cluster name will be null. And subsequent in 
> subsequent heartbeats, isConfigStale() is called. Though, hostname will not 
> be null.

Thinking about this, we should not get the recovery configuration if the 
cluster name is not available. I'll go ahead and make cluster name and host 
name mandatory for the recovery configuration calls. Thanks.


- Nahappan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44986/#review124692
---


On March 21, 2016, 12:24 p.m., Nahappan Somasundaram wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44986/
> ---
> 
> (Updated March 21, 2016, 12:24 p.m.)
> 
> 
> Review request for Ambari, Jonathan Hurley, Nate Cole, Sumit Mohanty, and Sid 
> Wagle.
> 
> 
> Bugs: AMBARI-15474
> https://issues.apache.org/jira/browse/AMBARI-15474
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> AMBARI-15474: Listen for changes to auto-start configuration and send them to 
> the agent during heartbeats.
> 
> In HeartbeatHandler::handleHeartBeat(), send the recovery configuration to 
> the agent. Instead of pulling the recovery configuration from the application 
> cache, implement change events and get notified whenever there are changes to 
> the recovery configuration.
> 
> **Changes**:
> 1. Instead of sending the recovery configuration to the agent during every 
> heartbeat (10 seconds interval), send the configuration only when changes are 
> detected.
> 2. Used AmbariEventPublisher to publish the changes. RecoveryConfigHelper is 
> the subscriber which listens to changes to the configuration.
> 3. RecoveryConfigHelper maintains a map of Cluster-->Host-->Timestamp which 
> is the last updated timestamp of the recovery configuration. This timestamp 
> is used as a token and is sent to the agent with the configuration. The agent 
> provides this timestamp during each heartbeat. If the timestamp from the 
> heartbeat is the same as the one on the server side, no configuration is sent 
> to the agent.
> 4. Whenever changes are detected, the timestamp is invalidated. During the 
> next heartbeat, the recovery configuration is read from the DB and sent to 
> the agent along with the latest timestamp.
> 5. Added new unit tests to test each configuration change event.
> 
> 
> Diffs
> -
> 
>   ambari-agent/src/main/python/ambari_agent/Controller.py 
> c1c16ac97cca0d2677da2e26eee7b4949a4bf15c 
>   ambari-agent/src/main/python/ambari_agent/Heartbeat.py 
> b28644c834a2387d2fb0ad17d104224e830a0245 
>   ambari-agent/src/main/python/ambari_agent/RecoveryManager.py 
> ed537ca6928ce376c5f3e906f7a47c3f1919e309 
>   ambari-server/src/main/java/org/apache/ambari/server/agent/HeartBeat.java 
> 1430aa20cf68db071be85c4ad2ebb9acdaccecc0 
>   
> ambari-server/src/main/java/org/apache/ambari/server/agent/HeartBeatHandler.java
>  3a80803724bcb6ca9e4ec875aa17c9ac1c66fbe8 
>   
> ambari-server/src/main/java/org/apache/ambari/server/agent/HeartBeatResponse.java
>  617b04b5b8768ba28a3c8ecb6e5e00601f153396 
>   
> ambari-server/src/main/java/org/apache/ambari/server/agent/RecoveryConfig.java
>  c2c18462c586292caaf40bdb6a25a8fe9a39d76c 
>   
> ambari-server/src/main/java/org/apache/ambari/server/agent/RecoveryConfigHelper.java
>  92a622117c71ddfc3bd2562c04ee525491bd6ffd 
>   
> ambari-server/src/main/java/org/apache/ambari/server/events/AmbariEvent.java 
> 78731854fe80c3d6087a73ebf00b0ffe9e287354 
>   
> ambari-server/src/main/java/org/apache/ambari/server/events/ClusterConfigChangedEvent.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/events/ServiceComponentInstalledEvent.java
>  2ce01236b3bb1ab4934fe456e5669fa474987426 
>   
> ambari-server/src/main/java/org/apache/ambari/server/events/ServiceComponentRecoveryChangedEvent.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/events/ServiceComponentUninstalledEvent.java
>  19712cdb18592816f82898cd7aa08a3600c7e1f4 
>   ambari-server/src/main/java/org/apache/ambari/server/state/ConfigImpl.java 
> 2cc3d0013140c9c7d3453d7c5e05c3aa02bc5794 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/ServiceComponentImpl.java
>

Review Request 45141: Apply the stack featurization prototype detailed on AMBARI-13364 to ZOOKEEPER service.

2016-03-21 Thread Juanjo Marron


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45141/
---

Review request for Ambari, Alejandro Fernandez and Jayush Luniya.


Bugs: AMBARI-14435
https://issues.apache.org/jira/browse/AMBARI-14435


Repository: ambari


Description
---

Apply the  stack featurization prototype detailed on AMBARI-13364 to ZOOKEEPER 
service plus the comments and reviews received on:
https://reviews.apache.org/r/44933/

This patch removes the hardcoded stack versions in ZK common-services code 
(centralized version and parameterized stack configurations will be faced in 
other JIRAS)

stack_features is now a property in .json format in 
HDP/2.0.6/properties/stack_features.json
With the structure:
{
"stack_features": [
{ "name": "feature1", "description" : "Feature1 support", "min_version" : 
"Y.Y.Y.Y" , "max_version" : "X.X.X.X"}
, 
...
]
}
where min_version/max_version are optional constraints.

The upgrade constants, such has ROLLING_UPGRADE = "rolling_upgrade" has been 
added to in a new StackFeature in 
resource_management/libraries/functions/constants.py

It is added as part of the HDP stack configurations on 
/HDP/2.0.6/configuration/cluster-env.xml, introducing a new stack_features 
property

New resource_management/libraries/functions/stack_fetaures.py has been 
introduced to parse the json file and called from service code to check if the 
stack supports the required feature.

The stack version used in params_linux.py comparison is  
stack_version_formatted = hostLevelParams\stack_version


Diffs
-

  
ambari-common/src/main/python/resource_management/libraries/functions/constants.py
 0adaef9 
  
ambari-common/src/main/python/resource_management/libraries/functions/stack_features.py
 PRE-CREATION 
  
ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/params_linux.py
 9ff9125 
  
ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/status_params.py
 9ae8440 
  
ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/zookeeper.py
 5712ce4 
  
ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/zookeeper_client.py
 25ace24 
  
ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/zookeeper_server.py
 d5b6898 
  
ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/zookeeper_service.py
 6fe0772 
  
ambari-server/src/main/resources/stacks/HDP/2.0.6/configuration/cluster-env.xml 
5f77f48 
  
ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_features.json
 PRE-CREATION 

Diff: https://reviews.apache.org/r/45141/diff/


Testing
---

Zookeeper fresh installation on top of latest trunk


Thanks,

Juanjo  Marron

Re: Review Request 44933: Apply the stack featurization prototype detailed on AMBARI-13364 to ZOOKEEPER service.

2016-03-21 Thread Juanjo Marron


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44933/
---

(Updated March 22, 2016, 1:03 a.m.)


Review request for Ambari, Alejandro Fernandez and Jayush Luniya.


Bugs: AMBARI-14435
https://issues.apache.org/jira/browse/AMBARI-14435


Repository: ambari


Description
---

Apply the  stack featurization prototype  detailed on AMBARI-13364 to ZOOKEEPER 
service.
This patch removes the hardcoded stack versions in ZK common-services code 
(centralized version and parameterized stack configurations will be faced in 
other JIRAS)
stack_features is now a property in .json format in 
configuration/cluster-env.xml for each stack version.
With the structure:
{
"stack_features": [
{ "name": "feature1", "description" : "Feature1 support", "min_version" : 
"Y.Y.Y.Y" , "max_version" : "X.X.X.X"}
, 
...
]
}
where min_version/max_version are optional constraints.
New resource_management/libraries/functions/stack_fetaures.py has been 
introduced to parse the json file and called from service code to check if the 
stack supports the required feature.


Diffs
-

  
ambari-common/src/main/python/resource_management/libraries/functions/stack_fetaures.py
 PRE-CREATION 
  
ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/params_linux.py
 9ff9125 
  
ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/status_params.py
 d18e4d7 
  
ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/zookeeper.py
 5712ce4 
  
ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/zookeeper_client.py
 25ace24 
  
ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/zookeeper_server.py
 d5b6898 
  
ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/zookeeper_service.py
 6fe0772 
  
ambari-server/src/main/resources/stacks/HDP/2.0.6/configuration/cluster-env.xml 
3fb82e9 
  ambari-server/src/main/resources/stacks/HDP/2.2/configuration/cluster-env.xml 
f1fa4de 
  ambari-server/src/main/resources/stacks/HDP/2.3/configuration/cluster-env.xml 
PRE-CREATION 

Diff: https://reviews.apache.org/r/44933/diff/


Testing
---

Zookeeper fresh installation has been tested using HDP stack 2.3 and 
AMBARI-13364 branch code.


Thanks,

Juanjo  Marron

Review Request 45140: Hive interactive component should be created/deleted when interactive query is enabled/disabled

2016-03-21 Thread Jaimin Jetly


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45140/
---

Review request for Ambari, Srimanth Gunturi, Swapan Shridhar, and Yusaku Sako.


Bugs: AMBARI-15503
https://issues.apache.org/jira/browse/AMBARI-15503


Repository: ambari


Description
---

1. Show dialog to user for choosing HiveServer2 interactive host component when 
Interactive query is turned ON
2. If Manual kerberos is enabled on the cluster then the dialog should ask user 
to add appropriate keytabs for newly installed host components
3. HiveServer2 interactive host-component and its dependencies should be 
created and started when "Enable Interactive" flag is turned ON.
4. HiveServer2 interactive host-component should be stopped and deleted when 
"Enable Interactive" flag is turned OFF.
5. UI should ask user to install slider service if not present in the cluster
6. "Enable Interactive" flag should be disabled when installing another service.


Diffs
-

  ambari-server/src/main/resources/stacks/HDP/2.6/role_command_order.json 
PRE-CREATION 
  
ambari-server/src/main/resources/stacks/HDP/2.6/services/HIVE/configuration/hive-interactive-env.xml
 PRE-CREATION 
  
ambari-server/src/main/resources/stacks/HDP/2.6/services/HIVE/configuration/hive-interactive-site.xml
 fb55d2d 
  
ambari-server/src/main/resources/stacks/HDP/2.6/services/HIVE/configuration/llap-env.xml
 24a95cf 
  ambari-server/src/main/resources/stacks/HDP/2.6/services/HIVE/metainfo.xml 
ff0f318 
  
ambari-server/src/main/resources/stacks/HDP/2.6/services/HIVE/themes/theme.json 
26b9532 
  
ambari-server/src/main/resources/stacks/HDP/2.6/services/HIVE/themes/theme_version_2.json
 PRE-CREATION 
  ambari-web/app/controllers/main/service/info/configs.js 93775fb 
  ambari-web/app/controllers/wizard/step7/assign_master_controller.js 496968b 
  ambari-web/app/controllers/wizard/step7_controller.js 06488b1 
  ambari-web/app/mappers/configs/themes_mapper.js 3e68e67 
  ambari-web/app/messages.js 421143a 
  ambari-web/app/mixins.js 0a686b3 
  ambari-web/app/mixins/common/configs/configs_saver.js 82741fa 
  ambari-web/app/mixins/main/service/configs/component_actions_by_configs.js 
PRE-CREATION 
  ambari-web/app/mixins/wizard/assign_master_components.js 78af279 
  ambari-web/app/models/configs/theme/config_action.js 288c674 
  ambari-web/app/templates/common/assign_master_components.hbs ea77c73 
  ambari-web/app/templates/common/configs/widgets/label_config_widget.hbs 
PRE-CREATION 
  ambari-web/app/utils/ajax/ajax.js 4e7b045 
  ambari-web/app/utils/configs/theme/theme.js f0cf966 
  ambari-web/app/views.js 8e8154c 
  ambari-web/app/views/common/assign_master_components_view.js da45ee9 
  ambari-web/app/views/common/configs/service_config_layout_tab_view.js def75d1 
  ambari-web/app/views/common/configs/widgets/config_widget_view.js 3ad9e3b 
  ambari-web/app/views/common/configs/widgets/label_config_widget_view.js 
PRE-CREATION 
  ambari-web/app/views/common/configs/widgets/time_interval_spinner_view.js 
8663ac3 
  ambari-web/app/views/wizard/step7/assign_master_view.js 2bc2ce3 
  ambari-web/test/controllers/wizard/step5_test.js 4140210 
  ambari-web/test/mixins/wizard/assign_master_components_test.js ecdb456 

Diff: https://reviews.apache.org/r/45140/diff/


Testing
---

Manually tested the patch on Installer, Add Service wizard and service->config 
page
Verified that all ambari-web unit tests passes:

  24639 tests complete (34 seconds)
  145 tests pending


Thanks,

Jaimin Jetly

Re: Review Request 44986: AMBARI-15474: Listen for changes to auto-start configuration and send them to the agent during heartbeats

2016-03-21 Thread Alejandro Fernandez


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44986/#review124692
---


Fix it, then Ship it!




Fix It and Ship It


ambari-agent/src/main/python/ambari_agent/Heartbeat.py (line 81)


What does repr of these 2 objects look like?



ambari-agent/src/main/python/ambari_agent/RecoveryManager.py (line 595)


Perhaps add a message that -1 is an acceptable value.



ambari-server/src/main/java/org/apache/ambari/server/agent/HeartBeatHandler.java
 (line 277)


I wouldn't worry too much about this case.



ambari-server/src/main/java/org/apache/ambari/server/agent/HeartBeatHandler.java
 (line 286)


It's possible for this to be called if clusterName is still null.



ambari-server/src/main/java/org/apache/ambari/server/agent/RecoveryConfigHelper.java
 (line 88)


Does this really have to be a concurrent hashmap?



ambari-server/src/main/java/org/apache/ambari/server/agent/RecoveryConfigHelper.java
 (line 131)


Why should empty cluster or host return true instead of raising an 
exception.



ambari-server/src/main/java/org/apache/ambari/server/events/ServiceComponentRecoveryChangedEvent.java
 (line 23)


Please add a description of what "Recovery Enabled" means


- Alejandro Fernandez


On March 21, 2016, 7:24 p.m., Nahappan Somasundaram wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44986/
> ---
> 
> (Updated March 21, 2016, 7:24 p.m.)
> 
> 
> Review request for Ambari, Jonathan Hurley, Nate Cole, Sumit Mohanty, and Sid 
> Wagle.
> 
> 
> Bugs: AMBARI-15474
> https://issues.apache.org/jira/browse/AMBARI-15474
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> AMBARI-15474: Listen for changes to auto-start configuration and send them to 
> the agent during heartbeats.
> 
> In HeartbeatHandler::handleHeartBeat(), send the recovery configuration to 
> the agent. Instead of pulling the recovery configuration from the application 
> cache, implement change events and get notified whenever there are changes to 
> the recovery configuration.
> 
> **Changes**:
> 1. Instead of sending the recovery configuration to the agent during every 
> heartbeat (10 seconds interval), send the configuration only when changes are 
> detected.
> 2. Used AmbariEventPublisher to publish the changes. RecoveryConfigHelper is 
> the subscriber which listens to changes to the configuration.
> 3. RecoveryConfigHelper maintains a map of Cluster-->Host-->Timestamp which 
> is the last updated timestamp of the recovery configuration. This timestamp 
> is used as a token and is sent to the agent with the configuration. The agent 
> provides this timestamp during each heartbeat. If the timestamp from the 
> heartbeat is the same as the one on the server side, no configuration is sent 
> to the agent.
> 4. Whenever changes are detected, the timestamp is invalidated. During the 
> next heartbeat, the recovery configuration is read from the DB and sent to 
> the agent along with the latest timestamp.
> 5. Added new unit tests to test each configuration change event.
> 
> 
> Diffs
> -
> 
>   ambari-agent/src/main/python/ambari_agent/Controller.py 
> c1c16ac97cca0d2677da2e26eee7b4949a4bf15c 
>   ambari-agent/src/main/python/ambari_agent/Heartbeat.py 
> b28644c834a2387d2fb0ad17d104224e830a0245 
>   ambari-agent/src/main/python/ambari_agent/RecoveryManager.py 
> ed537ca6928ce376c5f3e906f7a47c3f1919e309 
>   ambari-server/src/main/java/org/apache/ambari/server/agent/HeartBeat.java 
> 1430aa20cf68db071be85c4ad2ebb9acdaccecc0 
>   
> ambari-server/src/main/java/org/apache/ambari/server/agent/HeartBeatHandler.java
>  3a80803724bcb6ca9e4ec875aa17c9ac1c66fbe8 
>   
> ambari-server/src/main/java/org/apache/ambari/server/agent/HeartBeatResponse.java
>  617b04b5b8768ba28a3c8ecb6e5e00601f153396 
>   
> ambari-server/src/main/java/org/apache/ambari/server/agent/RecoveryConfig.java
>  c2c18462c586292caaf40bdb6a25a8fe9a39d76c 
>   
> ambari-server/src/main/java/org/apache/ambari/server/agent/RecoveryConfigHelper.java
>  92a622117c71ddfc3bd2562c04ee525491bd6ffd 
>   
> ambari-server/src/main/java/org/apache/ambari/server/events/AmbariEvent.java 
> 78731854fe80c3d6087a73ebf00b0ffe9e287354 
>   
> ambari-server/src/main/java/org/apache/ambari/server/events/ClusterConfigChangedEvent.java
>  PRE-CREATION 
>   
>

Re: Review Request 45133: AMBARI-15501. HiveServerInteractive. Change the 'role command order' so that 'Hive Metastore' start comes before 'Hive Interactive Server'.

2016-03-21 Thread Alejandro Fernandez


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45133/#review124690
---


Ship it!




Ship It!

- Alejandro Fernandez


On March 22, 2016, 12:22 a.m., Swapan Shridhar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/45133/
> ---
> 
> (Updated March 22, 2016, 12:22 a.m.)
> 
> 
> Review request for Ambari, Jaimin Jetly and Sumit Mohanty.
> 
> 
> Bugs: AMBARI-15501
> https://issues.apache.org/jira/browse/AMBARI-15501
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Change the 'role command order' so that 'Hive Metastore' start comes before 
> 'Hive Interactive Server'.
> 
> 
> Diffs
> -
> 
>   ambari-server/src/main/resources/stacks/HDP/2.6/role_command_order.json 
> PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/45133/diff/
> 
> 
> Testing
> ---
> 
> - HDP 2.6 - Tested the behavior of Hive Server Interactive -> Start came 
> after Hive Metastore.
> - HDP 2.6 - Tested the behavior w/o Hive Server Interactive (Isolation) -> 
> Hive Service came up fine.
> 
> 
> - Ran Service Checks for both the Scenarios mentioned above -> Works fine.
> - 
> - As file is created in 2.6 stack, shoudln't affect before stacks.
> 
> 
> Thanks,
> 
> Swapan Shridhar
> 
>

Re: Review Request 45125: 'ambari-server check-database' has errors after upgrade to ambari 2.2.2.0 [grafana errors]

2016-03-21 Thread Myroslav Papirkovskyy


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45125/#review124642
---


Ship it!




Ship It!

- Myroslav Papirkovskyy


On Березень 21, 2016, 10:12 після полудня, Vitalyi Brodetskyi wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/45125/
> ---
> 
> (Updated Березень 21, 2016, 10:12 після полудня)
> 
> 
> Review request for Ambari, Sumit Mohanty and Sid Wagle.
> 
> 
> Bugs: AMBARI-15495
> https://issues.apache.org/jira/browse/AMBARI-15495
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> STR: 
> 1)Deploy old version 
> 2)Make Ambari only Upgrade 
> 4)'ambari-server check-database'
> Actual result:
> 'ambari-server check-database' has errors after upgrade to ambari 2.2.2.0
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/checks/CheckDatabaseHelper.java
>  652a49f 
>   
> ambari-server/src/test/java/org/apache/ambari/server/checks/CheckDatabaseHelperTest.java
>  66c6667 
> 
> Diff: https://reviews.apache.org/r/45125/diff/
> 
> 
> Testing
> ---
> 
> mvn clean test
> 
> 
> Thanks,
> 
> Vitalyi Brodetskyi
> 
>

Re: Review Request 45110: 'ambari-server check-database' reports error for spark-thrift-fairscheduler config post EU

2016-03-21 Thread Alejandro Fernandez


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45110/#review124585
---


Ship it!




Ship It!

- Alejandro Fernandez


On March 21, 2016, 4:38 p.m., Vitalyi Brodetskyi wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/45110/
> ---
> 
> (Updated March 21, 2016, 4:38 p.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez and Myroslav Papirkovskyy.
> 
> 
> Bugs: AMBARI-15493
> https://issues.apache.org/jira/browse/AMBARI-15493
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> *Steps*
> 1. Run an EU from 2.3.4.0 to 2.4.2.0 (with Ambari 2.2.2.0)
> 2. Run ambari-server check-database
> 
> *Result:*
> {code}
> Using python  /usr/bin/python
> Checking database
> Some error(s) or/and warning(s) was(were) found. Please check 
> ambari-server-check-database.log for problem(s).
> Ambari Server 'check-database' completed successfully.
> {code}
> 
> Log file shows below:
> {code}
> cat ambari-server-check-database.log
> 2016-03-21 09:58:06,995  INFO - *** Check 
> database started ***
> 2016-03-21 09:58:21,551 ERROR - Required config(s): 
> spark-thrift-fairscheduler is(are) not available for service SPARK with 
> service config version 1 in cluster cl1
> 2016-03-21 09:58:21,554 ERROR - Required config(s): 
> spark-thrift-fairscheduler is(are) not available for service SPARK with 
> service config version 2 in cluster cl1
> 2016-03-21 09:58:21,647  INFO - *** Check 
> database completed ***
> {code}
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/checks/CheckDatabaseHelper.java
>  766e2c0 
>   
> ambari-server/src/test/java/org/apache/ambari/server/checks/CheckDatabaseHelperTest.java
>  51ed42b 
> 
> Diff: https://reviews.apache.org/r/45110/diff/
> 
> 
> Testing
> ---
> 
> mvn clean test
> 
> 
> Thanks,
> 
> Vitalyi Brodetskyi
> 
>

Re: Review Request 44966: In IE, logging out of Ambari Admin View doesn't log user out of web console.

2016-03-21 Thread Alexandr Antonenko


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44966/#review124579
---


Ship it!




Ship It!

- Alexandr Antonenko


On March 18, 2016, 4:52 p.m., Sangeeta Ravindran wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44966/
> ---
> 
> (Updated March 18, 2016, 4:52 p.m.)
> 
> 
> Review request for Ambari, Alexandr Antonenko and Denys Buzhor.
> 
> 
> Bugs: AMBARI-15319
> https://issues.apache.org/jira/browse/AMBARI-15319
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Install latest Ambari trunk build.
> Open Ambari web using IE. In the Admin view, click on Sign out.
> The login page is displayed briefly and then the page returns to the Admin 
> view.
> This works fine on FF and Chrome.
> 
> This seems to be happening because of the additional parameters added to the 
> url in signout() method in 
> ambari-admin/src/main/resources/ui/admin-web/app/scripts/services/Auth.js
> 
> // Workaround for sign off within Basic Authorization
>   var origin = $window.location.protocol + '//' + Date.now() + ':' + 
> Date.now() + '@' +
> $window.location.hostname + ($window.location.port ? ':' + 
> $window.location.port : '');
>   return $http({
> method: 'GET',
> url: origin + Settings.baseUrl + '/logout'
>   });
> 
> This causes an exception in IE (A security problem occurred.) when executing 
> urlParsingNode.setAttribute("href", href); 
> 
> As a result, the server side performLogout method is not invoked and after 
> temporarily displaying the login page, the user is brought back in to the 
> Dashboard.
> 
> To fix this, we will instead pass an invalid username and password in the 
> request header.
> 
> //use an invalid username and password in the request header
>   $http.defaults.headers.common['Authorization'] = 'Basic ' + 
> btoa('invalid_username:password');
>   return $http({
> method: 'GET',
> url: Settings.baseUrl + '/logout'
>});
> 
> 
> Diffs
> -
> 
>   ambari-admin/src/main/resources/ui/admin-web/app/scripts/services/Auth.js 
> 14c04c1 
> 
> Diff: https://reviews.apache.org/r/44966/diff/
> 
> 
> Testing
> ---
> 
> Manually tested fix on IE, Chrome and FF.
> 
> 
> Thanks,
> 
> Sangeeta Ravindran
> 
>

Re: Review Request 45101: Configuration with tag 'TOPOLOGY_RESOLVED' exists for 'cluster-env' error when creating Kerberized cluster with Blueprints

2016-03-21 Thread Robert Levas


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45101/#review124555
---


Fix it, then Ship it!





ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementControllerImpl.java
 (lines 1437 - 1438)


I would have thought that the fix would have been somewhere within the 
blueprint processing code rather than here since it attempts to set the 
`cluster-env` config multiple times with the same tag.

If you skip the setting of this, wont you loose data?


- Robert Levas


On March 21, 2016, 10:23 a.m., Sebastian Toader wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/45101/
> ---
> 
> (Updated March 21, 2016, 10:23 a.m.)
> 
> 
> Review request for Ambari, Myroslav Papirkovskyy, Robert Levas, and Sandor 
> Magyari.
> 
> 
> Bugs: AMBARI-15489
> https://issues.apache.org/jira/browse/AMBARI-15489
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> The logic that determines if a new version has to be created for a 
> configuration should skip cluster wide config types when verifying current 
> desired config against service configs as these are not service configs.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementControllerImpl.java
>  ac2fb22 
> 
> Diff: https://reviews.apache.org/r/45101/diff/
> 
> 
> Testing
> ---
> 
> Manual testing done.
> 
> Unit tests results:
> Results :
> 
> Tests run: 3534, Failures: 0, Errors: 0, Skipped: 36
> 
> 
> Thanks,
> 
> Sebastian Toader
> 
>

Re: Review Request 45035: Restarting HDFS Before Upgrade Finalizing Does Not Supply the rollingUpgrade Flag

2016-03-21 Thread Jonathan Hurley


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45035/
---

(Updated March 21, 2016, 10:48 a.m.)


Review request for Ambari, Alejandro Fernandez, Nate Cole, and Robert Levas.


Bugs: AMBARI-15482
https://issues.apache.org/jira/browse/AMBARI-15482


Repository: ambari


Description
---

During an upgrade, the HDFS NameNode(s) are restarted with the 
{{-rollingUpgrade}} flag. However, it's possible to get to the end of an 
upgrade and decide to "Finalize Later".

This allows the cluster to run in the upgraded state before committing to the 
upgrade. Full cluster control is returned via the Ambari web client.

Administrators can then decide to restart a NameNode. Upon restarting the 
NameNode, it will produce an error that it was not started with the 
{{rollingUpgrade}} flag. 

It seems that as long as an upgrade has not been finalized, the NameNode(s) 
must be started with the {{rollingUpgrade}} to allow them to function properly. 

STR:
- Perform a rolling upgrade from HDP 2.2 to 2.3 (or 2.3 to 2.4); as long as 
there is a major version change.
- Before finalization, say "Finalize Later". All services should be up and 
green.
- Now restart a NameNode


Diffs
-

  
ambari-server/src/main/java/org/apache/ambari/server/agent/ExecutionCommand.java
 9ea541e 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariActionExecutionHelper.java
 4ef215c 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariCustomCommandExecutionHelper.java
 303f3a4 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/UpgradeResourceProvider.java
 07061e1 
  
ambari-server/src/main/java/org/apache/ambari/server/orm/entities/UpgradeEntity.java
 3f1a52b 
  ambari-server/src/main/java/org/apache/ambari/server/state/Cluster.java 
ed3c772 
  
ambari-server/src/main/java/org/apache/ambari/server/state/cluster/ClusterImpl.java
 1c7ff61 
  
ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog222.java
 1e69db7 
  ambari-server/src/main/resources/Ambari-DDL-Derby-CREATE.sql e01e693 
  ambari-server/src/main/resources/Ambari-DDL-MySQL-CREATE.sql a07c6fc 
  ambari-server/src/main/resources/Ambari-DDL-Oracle-CREATE.sql b2b450a 
  ambari-server/src/main/resources/Ambari-DDL-Postgres-CREATE.sql cec122e 
  ambari-server/src/main/resources/Ambari-DDL-Postgres-EMBEDDED-CREATE.sql 
96fc720 
  ambari-server/src/main/resources/Ambari-DDL-SQLAnywhere-CREATE.sql c425d6f 
  ambari-server/src/main/resources/Ambari-DDL-SQLServer-CREATE.sql 2a89e26 
  
ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs_namenode.py
 e4c8c9c 
  
ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/namenode.py
 02905ec 
  
ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/namenode_upgrade.py
 a154b73 
  
ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/params_linux.py
 905802f 
  
ambari-server/src/test/java/org/apache/ambari/server/controller/internal/UpgradeResourceProviderTest.java
 17c52d2 
  ambari-web/app/controllers/main/admin/stack_and_upgrade_controller.js 2dceccc 
  ambari-web/app/utils/ajax/ajax.js 29d0715 

Diff: https://reviews.apache.org/r/45035/diff/


Testing (updated)
---

Tests run: 3968, Failures: 0, Errors: 0, Skipped: 33

[INFO] 
[INFO] BUILD SUCCESS
[INFO] 
[INFO] Total time: 32:16 min
[INFO] Finished at: 2016-03-18T13:41:19-04:00
[INFO] Final Memory: 37M/701M
[INFO] 


Thanks,

Jonathan Hurley

Re: Review Request 45035: Restarting HDFS Before Upgrade Finalizing Does Not Supply the rollingUpgrade Flag

2016-03-21 Thread Jonathan Hurley


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45035/
---

(Updated March 21, 2016, 10:48 a.m.)


Review request for Ambari, Alejandro Fernandez, Nate Cole, and Robert Levas.


Changes
---

Updating after merge conflicts resolved.


Bugs: AMBARI-15482
https://issues.apache.org/jira/browse/AMBARI-15482


Repository: ambari


Description
---

During an upgrade, the HDFS NameNode(s) are restarted with the 
{{-rollingUpgrade}} flag. However, it's possible to get to the end of an 
upgrade and decide to "Finalize Later".

This allows the cluster to run in the upgraded state before committing to the 
upgrade. Full cluster control is returned via the Ambari web client.

Administrators can then decide to restart a NameNode. Upon restarting the 
NameNode, it will produce an error that it was not started with the 
{{rollingUpgrade}} flag. 

It seems that as long as an upgrade has not been finalized, the NameNode(s) 
must be started with the {{rollingUpgrade}} to allow them to function properly. 

STR:
- Perform a rolling upgrade from HDP 2.2 to 2.3 (or 2.3 to 2.4); as long as 
there is a major version change.
- Before finalization, say "Finalize Later". All services should be up and 
green.
- Now restart a NameNode


Diffs (updated)
-

  
ambari-server/src/main/java/org/apache/ambari/server/agent/ExecutionCommand.java
 9ea541e 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariActionExecutionHelper.java
 4ef215c 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariCustomCommandExecutionHelper.java
 303f3a4 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/UpgradeResourceProvider.java
 07061e1 
  
ambari-server/src/main/java/org/apache/ambari/server/orm/entities/UpgradeEntity.java
 3f1a52b 
  ambari-server/src/main/java/org/apache/ambari/server/state/Cluster.java 
ed3c772 
  
ambari-server/src/main/java/org/apache/ambari/server/state/cluster/ClusterImpl.java
 1c7ff61 
  
ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog222.java
 1e69db7 
  ambari-server/src/main/resources/Ambari-DDL-Derby-CREATE.sql e01e693 
  ambari-server/src/main/resources/Ambari-DDL-MySQL-CREATE.sql a07c6fc 
  ambari-server/src/main/resources/Ambari-DDL-Oracle-CREATE.sql b2b450a 
  ambari-server/src/main/resources/Ambari-DDL-Postgres-CREATE.sql cec122e 
  ambari-server/src/main/resources/Ambari-DDL-Postgres-EMBEDDED-CREATE.sql 
96fc720 
  ambari-server/src/main/resources/Ambari-DDL-SQLAnywhere-CREATE.sql c425d6f 
  ambari-server/src/main/resources/Ambari-DDL-SQLServer-CREATE.sql 2a89e26 
  
ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs_namenode.py
 e4c8c9c 
  
ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/namenode.py
 02905ec 
  
ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/namenode_upgrade.py
 a154b73 
  
ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/params_linux.py
 905802f 
  
ambari-server/src/test/java/org/apache/ambari/server/controller/internal/UpgradeResourceProviderTest.java
 17c52d2 
  ambari-web/app/controllers/main/admin/stack_and_upgrade_controller.js 2dceccc 
  ambari-web/app/utils/ajax/ajax.js 29d0715 

Diff: https://reviews.apache.org/r/45035/diff/


Testing
---

Pending...


Thanks,

Jonathan Hurley

Re: Review Request 45101: Configuration with tag 'TOPOLOGY_RESOLVED' exists for 'cluster-env' error when creating Kerberized cluster with Blueprints

2016-03-21 Thread Myroslav Papirkovskyy


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45101/#review124552
---




ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementControllerImpl.java
 (lines 1437 - 1438)


We chould not hardcode config types here.
Simple way to make this dynamic is to use following method:
org.apache.ambari.server.state.cluster.ClusterImpl#getServiceByConfigType

So after adding this method to Cluster interface check will look like 
following:

if ( cluster.getServiceByCOnfigType != null && 
clusterConfig.getServiceConfigVersions().isEmpty() )


- Myroslav Papirkovskyy


On Березень 21, 2016, 4:23 після полудня, Sebastian Toader wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/45101/
> ---
> 
> (Updated Березень 21, 2016, 4:23 після полудня)
> 
> 
> Review request for Ambari, Myroslav Papirkovskyy, Robert Levas, and Sandor 
> Magyari.
> 
> 
> Bugs: AMBARI-15489
> https://issues.apache.org/jira/browse/AMBARI-15489
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> The logic that determines if a new version has to be created for a 
> configuration should skip cluster wide config types when verifying current 
> desired config against service configs as these are not service configs.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementControllerImpl.java
>  ac2fb22 
> 
> Diff: https://reviews.apache.org/r/45101/diff/
> 
> 
> Testing
> ---
> 
> Manual testing done.
> 
> Unit tests results:
> Results :
> 
> Tests run: 3534, Failures: 0, Errors: 0, Skipped: 36
> 
> 
> Thanks,
> 
> Sebastian Toader
> 
>

Re: Review Request 45101: Configuration with tag 'TOPOLOGY_RESOLVED' exists for 'cluster-env' error when creating Kerberized cluster with Blueprints

2016-03-21 Thread Sebastian Toader


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45101/
---

(Updated March 21, 2016, 3:23 p.m.)


Review request for Ambari, Myroslav Papirkovskyy, Robert Levas, and Sandor 
Magyari.


Changes
---

Added unit test results.


Bugs: AMBARI-15489
https://issues.apache.org/jira/browse/AMBARI-15489


Repository: ambari


Description
---

The logic that determines if a new version has to be created for a 
configuration should skip cluster wide config types when verifying current 
desired config against service configs as these are not service configs.


Diffs
-

  
ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementControllerImpl.java
 ac2fb22 

Diff: https://reviews.apache.org/r/45101/diff/


Testing (updated)
---

Manual testing done.

Unit tests results:
Results :

Tests run: 3534, Failures: 0, Errors: 0, Skipped: 36


Thanks,

Sebastian Toader

Re: Review Request 45101: Configuration with tag 'TOPOLOGY_RESOLVED' exists for 'cluster-env' error when creating Kerberized cluster with Blueprints

2016-03-21 Thread Sebastian Toader


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45101/
---

(Updated March 21, 2016, 3:15 p.m.)


Review request for Ambari, Myroslav Papirkovskyy, Robert Levas, and Sandor 
Magyari.


Changes
---

Linked apache jira to the review


Bugs: AMBARI-15489
https://issues.apache.org/jira/browse/AMBARI-15489


Repository: ambari


Description
---

The logic that determines if a new version has to be created for a 
configuration should skip cluster wide config types when verifying current 
desired config against service configs as these are not service configs.


Diffs
-

  
ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementControllerImpl.java
 ac2fb22 

Diff: https://reviews.apache.org/r/45101/diff/


Testing
---

Manual testing done.

Unit tests are running.


Thanks,

Sebastian Toader

Re: Review Request 45106: Non NameNode-HA properties still in hdfs-site.xml after enabling HA (scripts refer to non-existent property values)

2016-03-21 Thread Dmitro Lisnichenko


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45106/#review124547
---


Ship it!




Ship It!

- Dmitro Lisnichenko


On March 21, 2016, 3:33 p.m., Andrew Onischuk wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/45106/
> ---
> 
> (Updated March 21, 2016, 3:33 p.m.)
> 
> 
> Review request for Ambari and Dmitro Lisnichenko.
> 
> 
> Bugs: AMBARI-15491
> https://issues.apache.org/jira/browse/AMBARI-15491
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> This is a copy from AMBARI-13946, but I believe it has enough information
> 
> After enabling NameNode-HA, hdfs-site.xml does still contain non-HA
> properties, including  
> dfs.namenode.rpc-address  
> dfs.namenode.http-address  
> dfs.namenode.https-address  
> This cause the balancer to fail with the following symptoms in Balancer:  
> ...
> 
> 
> 
> 
> 15/11/18 15:48:30 INFO balancer.Balancer: namenodes  = [hdfs://daplab2, 
> hdfs://daplab-rt-11.fri.lan:8020]
> ...
> 
> java.io.IOException: Another Balancer is running.. Exiting ...  
> And ATS:
> 
> 
> 
> 
> _assert_valid
> self.target_status = self._get_file_status(target)
>   File 
> "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py",
>  line 292, in _get_file_status
> list_status = self.util.run_command(target, 'GETFILESTATUS', 
> method='GET', ignore_status_codes=['404'], assertable_result=False)
>   File 
> "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py",
>  line 210, in run_command
> raise Fail(err_msg)
> resource_management.core.exceptions.Fail: Execution of 'curl -sS -L -w 
> '%{http_code}' -X GET 
> 'http://pvvsccmn1-brn1:50070/webhdfs/v1/ats/done?op=GETFILESTATUS=hdfs''
>  returned status_code=403. 
> {
>   "RemoteException": {
> "exception": "StandbyException", 
> "javaClassName": "org.apache.hadoop.ipc.StandbyException", 
> "message": "Operation category READ is not supported in state standby"
>   }
> }
> 
> 
> 
> These should be removed from the config.
> 
> Steps to reproduce:Enable namenode HA on Ambari 2.2
> 
> Expected behavior: after turning on NameNode HA, grep grep dfs.namenode.http-
> address /etc/hadoop/conf/hdfs-site.xml shouldn't return anything
> 
> 
> Diffs
> -
> 
>   ambari-server/src/main/resources/stacks/HDP/2.2/upgrades/config-upgrade.xml 
> 440bd50 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/config-upgrade.xml 
> d826446 
> 
> Diff: https://reviews.apache.org/r/45106/diff/
> 
> 
> Testing
> ---
> 
> mvn clean test
> 
> 
> Thanks,
> 
> Andrew Onischuk
> 
>

Re: Review Request 45105: DB connection check installs JDK with incorrect permissions. umask 027, non root

2016-03-21 Thread Dmitro Lisnichenko


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45105/#review124546
---


Ship it!




Ship It!

- Dmitro Lisnichenko


On March 21, 2016, 3:32 p.m., Andrew Onischuk wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/45105/
> ---
> 
> (Updated March 21, 2016, 3:32 p.m.)
> 
> 
> Review request for Ambari and Dmitro Lisnichenko.
> 
> 
> Bugs: AMBARI-15490
> https://issues.apache.org/jira/browse/AMBARI-15490
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> **STR**  
> 1) Setup umask 027, non root agent  
> 2) During the wizard set Oozie server and Ambari-server on different hosts  
> 3) On service configuration step set Oozie db as "Existing MySQL Database" and
> hit "Test Connection" button  
> 4) Deploy the cluster
> 
> To reproduce this DB connection check should be executed before any host
> command (BEFORE-ANY hook setups JDK as well)
> 
> **Result**  
> On the host with Oozie server (where DB connection check was executed) JDK is
> installed with incorrect permissions. Services will fail to start.
> 
> 
> 
> /usr/jdk64/jdk1.8.0_60/bin/java: error while loading shared libraries: 
> libjli.so: cannot open shared object file: No such file or directory
> 
> [root@c6406 ~]# ll /usr/jdk64/jdk1.8.0_60/jre/lib/amd64/jli/libjli.so
> -rwxr-x--- 1 vagrant vagrant 102352 Aug  4  2015 
> /usr/jdk64/jdk1.8.0_60/jre/lib/amd64/jli/libjli.so
> [root@c6406 ~]# ll /usr/jdk64/jdk1.8.0_60/lib/amd64/jli/libjli.so
> -rwxr-x--- 1 vagrant vagrant 102990 Aug  4  2015 
> /usr/jdk64/jdk1.8.0_60/lib/amd64/jli/libjli.so
> 
> 
> Diffs
> -
> 
>   ambari-server/src/main/resources/custom_actions/scripts/check_host.py 
> 9b79290 
> 
> Diff: https://reviews.apache.org/r/45105/diff/
> 
> 
> Testing
> ---
> 
> mvn clean test
> 
> 
> Thanks,
> 
> Andrew Onischuk
> 
>

Review Request 45105: DB connection check installs JDK with incorrect permissions. umask 027, non root

2016-03-21 Thread Andrew Onischuk


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45105/
---

Review request for Ambari and Dmitro Lisnichenko.


Bugs: AMBARI-15490
https://issues.apache.org/jira/browse/AMBARI-15490


Repository: ambari


Description
---

**STR**  
1) Setup umask 027, non root agent  
2) During the wizard set Oozie server and Ambari-server on different hosts  
3) On service configuration step set Oozie db as "Existing MySQL Database" and
hit "Test Connection" button  
4) Deploy the cluster

To reproduce this DB connection check should be executed before any host
command (BEFORE-ANY hook setups JDK as well)

**Result**  
On the host with Oozie server (where DB connection check was executed) JDK is
installed with incorrect permissions. Services will fail to start.



/usr/jdk64/jdk1.8.0_60/bin/java: error while loading shared libraries: 
libjli.so: cannot open shared object file: No such file or directory

[root@c6406 ~]# ll /usr/jdk64/jdk1.8.0_60/jre/lib/amd64/jli/libjli.so
-rwxr-x--- 1 vagrant vagrant 102352 Aug  4  2015 
/usr/jdk64/jdk1.8.0_60/jre/lib/amd64/jli/libjli.so
[root@c6406 ~]# ll /usr/jdk64/jdk1.8.0_60/lib/amd64/jli/libjli.so
-rwxr-x--- 1 vagrant vagrant 102990 Aug  4  2015 
/usr/jdk64/jdk1.8.0_60/lib/amd64/jli/libjli.so


Diffs
-

  ambari-server/src/main/resources/custom_actions/scripts/check_host.py 9b79290 

Diff: https://reviews.apache.org/r/45105/diff/


Testing
---

mvn clean test


Thanks,

Andrew Onischuk

Re: Review Request 43832: AMBARI-14627: Ability to automate setup-security and setup-ldap/sync-ldap

2016-03-21 Thread Daniel Gergely


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/43832/#review124541
---




ambari-server/src/main/python/ambari_server/userInput.py (lines 99 - 102)


As I see when validator fails for a parameter that value is set from an 
argument, it is asked interactively again.
Is this intentional? When using command line parameters I would expect 
non-interactive behaviour. (e.g. the command is assembled and run by a script)
What do you think of terminating execution with an exit code when 
validation fails for a value that is set as an argument.



ambari-server/src/main/python/ambari_server/userInput.py (lines 111 - 114)


See my comment above


- Daniel Gergely


On márc. 16, 2016, 5:14 du, Oliver Szabo wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/43832/
> ---
> 
> (Updated márc. 16, 2016, 5:14 du)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Andrew Onischuk, Robert 
> Levas, Sumit Mohanty, and Sebastian Toader.
> 
> 
> Bugs: AMBARI-14627
> https://issues.apache.org/jira/browse/AMBARI-14627
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Added ability to automate setup-security/setup-ldap and sync-ldap. Ambari 
> uses '--' flags in order to replace user inputs. (if one of the flag is 
> missing, ambari will ask for user input)
> Example usage: 
> 
> 1.) LDAP setup: 
>   ambari-server setup-ldap \
>   --ldap-url="ldap.hortonworks.com:389" \
>   --ldap-secondary-url="" \
>   --ldap-ssl="false" \
>   --ldap-user-class="person" \
>   --ldap-user-attr="sAMAccountName" \
>   --ldap-group-class="group" \
>   --ldap-group-attr="cn" \
>   --ldap-member-attr="member" \
>   --ldap-dn="distunguishedName" \
>   --ldap-base-dn="dc=hdp01,dc=local" \
>   --ldap-referral="" \
>   --ldap-bind-anonym=false \
>   --ldap-manager-dn="cn=hdfs,ou=hdp,dc=hdp01,dc=local" \
>   --ldap-manager-password="myldappassword" \
>   --ldap-save-settings \
>   --truststore-type="jks" \
>   --truststore-path="/var/lib/ambari-server/keys/jkskeystore.jks" \
>   --truststore-password="mypass"
> 
> 2.) Ldap sync:
> ambari-server sync-ldap --groups=groups.txt --ldap-sync-admin-name=admin 
> --ldap-sync-admin-password=admin
> 
> 3.) Setup Https:
>   ambari-server setup-security \ 
> --security-option=setup-https \
> --security-keys_dir=/var/lib/ambari-server/keys \
> --api-ssl=true --client-api-ssl-port=8443 \ 
> --import-cert-path=/var/lib/ambari-server/keys/my.crt \ 
> --import-key-path=/var/lib/ambari-server/keys/my.key \
> --pem-password=password
> 4.) Encrypt passwords:
>   ambari-server setup-security --security-option=encrypt-password 
> --master-key=masterkey --master-key-persist=true
> 
> 5.) Setup Kerberos JAAS:
>   ambari-server setup-security --security-option=setup-kerberos-jaas 
> --jaas-principal="amb...@example.com" 
> --jaas-keytab="/etc/security/keytabs/ambari.keytab"
> 
> 6.) Setup TrustStore:
> ambari-server setup-security \
>   --security-option=setup-truststore \ 
>   --truststore-path=/var/lib/ambari-server/keys/keystore.p12 \
>   --truststore-type=pkcs12 \ 
>   --truststore-password=password \
>   --truststore-reconfigure // not needed if not configured - also, this 
> option is not available on branch-2.2 
> 7.) Import certificate to TrustStore:
> ambari-server setup-security \ 
>   --security-option=import-certificate \ 
>   --truststore-path=/var/lib/ambari-server/keys/keystore.p12 \ 
>   --truststore-type=pkcs12 \ 
>   --truststore-password=password \ 
>   --import-cert-path=/var/lib/ambari-server/oleewere.crt \ 
>   --import-cert-alias=myalias \ 
>   --truststore-reconfigure // not needed if not configured - also, this 
> option is not available on branch-2.2
> 
> 
> Diffs
> -
> 
>   ambari-server/src/main/python/ambari-server.py bc86d32 
>   ambari-server/src/main/python/ambari_server/dbConfiguration.py 5519a3d 
>   ambari-server/src/main/python/ambari_server/dbConfiguration_linux.py 
> 59c5d85 
>   ambari-server/src/main/python/ambari_server/dbConfiguration_windows.py 
> 96cd823 
>   ambari-server/src/main/python/ambari_server/serverConfiguration.py 0f58c0e 
>   ambari-server/src/main/python/ambari_server/serverSetup.py 7f6a7e3 
>   ambari-server/src/main/python/ambari_server/setupHttps.py 5e293fb 
>   ambari-server/src/main/python/ambari_server/setupSecurity.py b0ea491 
>   ambari-server/src/main/python/ambari_server/userInput.py 247ebec 
>   ambari-server/src/test/python/TestAmbariServer.py 305ad1a 
> 
> Diff: https://reviews.apache.org/r/43832/diff/
> 
> 
>

Re: Review Request 44986: AMBARI-15474: Listen for changes to auto-start configuration and send them to the agent during heartbeats

2016-03-21 Thread Jonathan Hurley


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44986/#review124538
---


Fix it, then Ship it!




Nice use of the events. Just an FYI; remember that the AmbariEventPublisher 
(although it's asynchronous) is probably single-threaded so that events being 
pushed onto the bus are delivered to listeners one-at-a-time. Just wanted to 
make sure you knew that and didn't rely on concurrency here for any type of 
performance.


ambari-agent/src/main/python/ambari_agent/RecoveryManager.py (lines 581 - 583)


a no-op function?



ambari-agent/src/main/python/ambari_agent/RecoveryManager.py (lines 586 - 590)


Can you doc the params here; there are a lot of them.



ambari-server/src/main/java/org/apache/ambari/server/agent/HeartBeat.java (line 
71)


Doc.



ambari-server/src/main/java/org/apache/ambari/server/agent/HeartBeat.java (line 
75)


Doc.



ambari-server/src/main/java/org/apache/ambari/server/agent/HeartBeatHandler.java
 (lines 272 - 275)


Well, for now :)



ambari-server/src/main/java/org/apache/ambari/server/agent/HeartBeatResponse.java
 (line 115)


Doc.



ambari-server/src/main/java/org/apache/ambari/server/agent/HeartBeatResponse.java
 (line 119)


Doc.



ambari-server/src/main/java/org/apache/ambari/server/agent/RecoveryConfig.java 
(line 104)


Doc.



ambari-server/src/main/java/org/apache/ambari/server/agent/RecoveryConfig.java 
(line 108)


Doc.



ambari-server/src/main/java/org/apache/ambari/server/agent/RecoveryConfigHelper.java
 (line 115)


Doc how this is being calculated.



ambari-server/src/main/java/org/apache/ambari/server/agent/RecoveryConfigHelper.java
 (line 208)


Doc.


- Jonathan Hurley


On March 18, 2016, 2:07 p.m., Nahappan Somasundaram wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44986/
> ---
> 
> (Updated March 18, 2016, 2:07 p.m.)
> 
> 
> Review request for Ambari, Jonathan Hurley, Nate Cole, Sumit Mohanty, and Sid 
> Wagle.
> 
> 
> Bugs: AMBARI-15474
> https://issues.apache.org/jira/browse/AMBARI-15474
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> AMBARI-15474: Listen for changes to auto-start configuration and send them to 
> the agent during heartbeats.
> 
> In HeartbeatHandler::handleHeartBeat(), send the recovery configuration to 
> the agent. Instead of pulling the recovery configuration from the application 
> cache, implement change events and get notified whenever there are changes to 
> the recovery configuration.
> 
> **Changes**:
> 1. Instead of sending the recovery configuration to the agent during every 
> heartbeat (10 seconds interval), send the configuration only when changes are 
> detected.
> 2. Used AmbariEventPublisher to publish the changes. RecoveryConfigHelper is 
> the subscriber which listens to changes to the configuration.
> 3. RecoveryConfigHelper maintains a map of Cluster-->Host-->Timestamp which 
> is the last updated timestamp of the recovery configuration. This timestamp 
> is used as a token and is sent to the agent with the configuration. The agent 
> provides this timestamp during each heartbeat. If the timestamp from the 
> heartbeat is the same as the one on the server side, no configuration is sent 
> to the agent.
> 4. Whenever changes are detected, the timestamp is invalidated. During the 
> next heartbeat, the recovery configuration is read from the DB and sent to 
> the agent along with the latest timestamp.
> 5. Added new unit tests to test each configuration change event.
> 
> 
> Diffs
> -
> 
>   ambari-agent/src/main/python/ambari_agent/Controller.py 
> c1c16ac97cca0d2677da2e26eee7b4949a4bf15c 
>   ambari-agent/src/main/python/ambari_agent/Heartbeat.py 
> b28644c834a2387d2fb0ad17d104224e830a0245 
>   ambari-agent/src/main/python/ambari_agent/RecoveryManager.py 
> ed537ca6928ce376c5f3e906f7a47c3f1919e309 
>   ambari-server/src/main/java/org/apache/ambari/server/agent/HeartBeat.java 
> 1430aa20cf68db071be85c4ad2ebb9acdaccecc0 
>   
> ambari-server/src/main/java/org/apache/ambari/server/agent/HeartBeatHandler.java
>  3a80803724bcb6ca9e4ec875aa17c9ac1c66fbe8 
>   
>

Re: Review Request 45150: AMBARI-15476 : [AMS / Grafana] Rate calculation with sum() is wrong

Re: Review Request 45150: AMBARI-15476 : [AMS / Grafana] Rate calculation with sum() is wrong

Re: Review Request 45141: Apply the stack featurization prototype detailed on AMBARI-13364 to ZOOKEEPER service.

Re: Review Request 44933: Apply the stack featurization prototype detailed on AMBARI-13364 to ZOOKEEPER service.

Re: Review Request 45140: Hive interactive component should be created/deleted when interactive query is enabled/disabled

Re: Review Request 44986: AMBARI-15474: Listen for changes to auto-start configuration and send them to the agent during heartbeats

Review Request 45141: Apply the stack featurization prototype detailed on AMBARI-13364 to ZOOKEEPER service.

Re: Review Request 44933: Apply the stack featurization prototype detailed on AMBARI-13364 to ZOOKEEPER service.

Review Request 45140: Hive interactive component should be created/deleted when interactive query is enabled/disabled

Re: Review Request 44986: AMBARI-15474: Listen for changes to auto-start configuration and send them to the agent during heartbeats

Re: Review Request 45133: AMBARI-15501. HiveServerInteractive. Change the 'role command order' so that 'Hive Metastore' start comes before 'Hive Interactive Server'.

Re: Review Request 45125: 'ambari-server check-database' has errors after upgrade to ambari 2.2.2.0 [grafana errors]

Re: Review Request 45110: 'ambari-server check-database' reports error for spark-thrift-fairscheduler config post EU

Re: Review Request 44966: In IE, logging out of Ambari Admin View doesn't log user out of web console.

Re: Review Request 45101: Configuration with tag 'TOPOLOGY_RESOLVED' exists for 'cluster-env' error when creating Kerberized cluster with Blueprints

Re: Review Request 45035: Restarting HDFS Before Upgrade Finalizing Does Not Supply the rollingUpgrade Flag

Re: Review Request 45035: Restarting HDFS Before Upgrade Finalizing Does Not Supply the rollingUpgrade Flag

Re: Review Request 45101: Configuration with tag 'TOPOLOGY_RESOLVED' exists for 'cluster-env' error when creating Kerberized cluster with Blueprints

Re: Review Request 45101: Configuration with tag 'TOPOLOGY_RESOLVED' exists for 'cluster-env' error when creating Kerberized cluster with Blueprints

Re: Review Request 45101: Configuration with tag 'TOPOLOGY_RESOLVED' exists for 'cluster-env' error when creating Kerberized cluster with Blueprints

Re: Review Request 45106: Non NameNode-HA properties still in hdfs-site.xml after enabling HA (scripts refer to non-existent property values)

Re: Review Request 45105: DB connection check installs JDK with incorrect permissions. umask 027, non root

Review Request 45105: DB connection check installs JDK with incorrect permissions. umask 027, non root

Re: Review Request 43832: AMBARI-14627: Ability to automate setup-security and setup-ldap/sync-ldap

Re: Review Request 44986: AMBARI-15474: Listen for changes to auto-start configuration and send them to the agent during heartbeats

25 matches

Site Navigation

Mail list logo

Footer information