-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44986/
-----------------------------------------------------------
Review request for Ambari, Jonathan Hurley, Nate Cole, Sumit Mohanty, and Sid
Wagle.
Bugs: AMBARI-15474
https://issues.apache.org/jira/browse/AMBARI-15474
Repository: ambari
Description
-------
AMBARI-15474: Listen for changes to auto-start configuration and send them to
the agent during heartbeats.
In HeartbeatHandler::handleHeartBeat(), send the recovery configuration to the
agent. Instead of pulling the recovery configuration from the application
cache, implement change events and get notified whenever there are changes to
the recovery configuration.
**Changes**:
1. Instead of sending the recovery configuration to the agent during every
heartbeat (10 seconds interval), send the configuration only when changes are
detected.
2. Used AmbariEventPublisher to publish the changes. RecoveryConfigHelper is
the subscriber which listens to changes to the configuration.
3. RecoveryConfigHelper maintains a map of Cluster-->Host-->Timestamp which is
the last updated timestamp of the recovery configuration. This timestamp is
used as a token and is sent to the agent with the configuration. The agent
provides this timestamp during each heartbeat. If the timestamp from the
heartbeat is the same as the one on the server side, no configuration is sent
to the agent.
4. Whenever changes are detected, the timestamp is invalidated. During the next
heartbeat, the recovery configuration is read from the DB and sent to the agent
along with the latest timestamp.
5. Added new unit tests to test each configuration change event.
Diffs
-----
ambari-agent/src/main/python/ambari_agent/Controller.py
c1c16ac97cca0d2677da2e26eee7b4949a4bf15c
ambari-agent/src/main/python/ambari_agent/Heartbeat.py
b28644c834a2387d2fb0ad17d104224e830a0245
ambari-agent/src/main/python/ambari_agent/RecoveryManager.py
ed537ca6928ce376c5f3e906f7a47c3f1919e309
ambari-server/src/main/java/org/apache/ambari/server/agent/HeartBeat.java
1430aa20cf68db071be85c4ad2ebb9acdaccecc0
ambari-server/src/main/java/org/apache/ambari/server/agent/HeartBeatHandler.java
3a80803724bcb6ca9e4ec875aa17c9ac1c66fbe8
ambari-server/src/main/java/org/apache/ambari/server/agent/HeartBeatResponse.java
617b04b5b8768ba28a3c8ecb6e5e00601f153396
ambari-server/src/main/java/org/apache/ambari/server/agent/RecoveryConfig.java
c2c18462c586292caaf40bdb6a25a8fe9a39d76c
ambari-server/src/main/java/org/apache/ambari/server/agent/RecoveryConfigHelper.java
92a622117c71ddfc3bd2562c04ee525491bd6ffd
ambari-server/src/main/java/org/apache/ambari/server/events/AmbariEvent.java
78731854fe80c3d6087a73ebf00b0ffe9e287354
ambari-server/src/main/java/org/apache/ambari/server/events/ClusterConfigChangedEvent.java
PRE-CREATION
ambari-server/src/main/java/org/apache/ambari/server/events/ServiceComponentInstalledEvent.java
2ce01236b3bb1ab4934fe456e5669fa474987426
ambari-server/src/main/java/org/apache/ambari/server/events/ServiceComponentRecoveryChangeEvent.java
PRE-CREATION
ambari-server/src/main/java/org/apache/ambari/server/events/ServiceComponentUninstalledEvent.java
19712cdb18592816f82898cd7aa08a3600c7e1f4
ambari-server/src/main/java/org/apache/ambari/server/state/ConfigImpl.java
2cc3d0013140c9c7d3453d7c5e05c3aa02bc5794
ambari-server/src/main/java/org/apache/ambari/server/state/ServiceComponentImpl.java
4be1c21bf41416681be836716ce38305cc3f287e
ambari-server/src/main/java/org/apache/ambari/server/state/svccomphost/ServiceComponentHostImpl.java
3a0075e3a3c90088ca6bd2d9eca48ef50db9fdb3
ambari-server/src/test/java/org/apache/ambari/server/agent/TestHeartbeatHandler.java
b1f94e38f97550123dd366984438b0bbf9c2826c
ambari-server/src/test/java/org/apache/ambari/server/configuration/RecoveryConfigHelperTest.java
c86b95a7cd10b0720f0a3b9676f65a8bd68dd6c9
ambari-server/src/test/java/org/apache/ambari/server/events/listeners/upgrade/HostVersionOutOfSyncListenerTest.java
1bf9d8336533bc677384314f998f10aa2f10504e
Diff: https://reviews.apache.org/r/44986/diff/
Testing
-------
**1. mvn clean install **
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Ambari Main ....................................... SUCCESS [5.864s]
[INFO] Apache Ambari Project POM ......................... SUCCESS [0.037s]
[INFO] Ambari Web ........................................ SUCCESS [23.683s]
[INFO] Ambari Views ...................................... SUCCESS [1.069s]
[INFO] Ambari Admin View ................................. SUCCESS [5.949s]
[INFO] ambari-metrics .................................... SUCCESS [0.534s]
[INFO] Ambari Metrics Common ............................. SUCCESS [0.431s]
[INFO] Ambari Metrics Hadoop Sink ........................ SUCCESS [1.060s]
[INFO] Ambari Metrics Flume Sink ......................... SUCCESS [0.534s]
[INFO] Ambari Metrics Kafka Sink ......................... SUCCESS [0.584s]
[INFO] Ambari Metrics Storm Sink ......................... SUCCESS [1.472s]
[INFO] Ambari Metrics Collector .......................... SUCCESS [5.981s]
[INFO] Ambari Metrics Monitor ............................ SUCCESS [1.696s]
[INFO] Ambari Metrics Grafana ............................ SUCCESS [0.819s]
[INFO] Ambari Metrics Assembly ........................... SUCCESS [1:13.281s]
[INFO] Ambari Server ..................................... SUCCESS [2:11.632s]
[INFO] Ambari Functional Tests ........................... SUCCESS [0.907s]
[INFO] Ambari Agent ...................................... SUCCESS [21.659s]
[INFO] Ambari Client ..................................... SUCCESS [0.043s]
[INFO] Ambari Python Client .............................. SUCCESS [0.829s]
[INFO] Ambari Groovy Client .............................. SUCCESS [2.078s]
[INFO] Ambari Shell ...................................... SUCCESS [0.039s]
[INFO] Ambari Python Shell ............................... SUCCESS [0.554s]
[INFO] Ambari Groovy Shell ............................... SUCCESS [0.811s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 4:42.176s
[INFO] Finished at: Thu Mar 17 15:52:41 PDT 2016
[INFO] Final Memory: 261M/1122M
[INFO] ------------------------------------------------------------------------
**2. mvn test -DskipSurefireTests **
Ran 253 tests in 6.633s
OK
----------------------------------------------------------------------
Total run:926
Total errors:0
Total failures:0
OK
INFO: AMBARI_SERVER_LIB is not set, using default /usr/lib/ambari-server
INFO: Return code from stack upgrade command, retcode = 0
StackAdvisor implementation for stack HDP1, version 2.0.6 was not found
Returning DefaultStackAdvisor implementation
StackAdvisor implementation for stack XYZ, version 1.0.0 was loaded
StackAdvisor implementation for stack XYZ, version 1.0.1 was loaded
Returning XYZ101StackAdvisor implementation
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 52.037s
[INFO] Finished at: Thu Mar 17 15:11:40 PDT 2016
[INFO] Final Memory: 57M/1008M
[INFO] ------------------------------------------------------------------------
**3. mvn test
-Dtest=TestHeartbeatHandler,RecoveryConfigHelperTest,HostVersionOutOfSyncListenerTest
-DskipPythonTests **
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Picked up _JAVA_OPTIONS: -Xmx2048m -XX:MaxPermSize=512m -Djava.awt.headless=true
Running org.apache.ambari.server.agent.TestHeartbeatHandler
Tests run: 24, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 55.137 sec -
in org.apache.ambari.server.agent.TestHeartbeatHandler
Picked up _JAVA_OPTIONS: -Xmx2048m -XX:MaxPermSize=512m -Djava.awt.headless=true
Running org.apache.ambari.server.configuration.RecoveryConfigHelperTest
Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 22.704 sec - in
org.apache.ambari.server.configuration.RecoveryConfigHelperTest
Picked up _JAVA_OPTIONS: -Xmx2048m -XX:MaxPermSize=512m -Djava.awt.headless=true
Running
org.apache.ambari.server.events.listeners.upgrade.HostVersionOutOfSyncListenerTest
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 19.333 sec - in
org.apache.ambari.server.events.listeners.upgrade.HostVersionOutOfSyncListenerTest
Results :
Tests run: 36, Failures: 0, Errors: 0, Skipped: 0
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 1:59.372s
[INFO] Finished at: Thu Mar 17 15:55:50 PDT 2016
[INFO] Final Memory: 55M/1018M
[INFO] ------------------------------------------------------------------------
**4. Manual tests **
*Setup*
* Deployed a VM with the latest **trunk** bits and replaced
**ambari-server.jar** from the development build.
* Replaced **agent python** files from development.
*Tests*
curl -u admin:admin -H "X-Requested-By: ambari" -X PUT
http://localhost:8080/api/v1/clusters/c1/components/METRICS_COLLECTOR -d
'{"ServiceComponentInfo":{"recovery_enabled":"false"}}'
In the Ambari Web UI, METRICS_COLLECTOR was set to Maintenance Mode.
*Verification using the debugger*
Before these tests were run, it was verified that the recovery configuration
was sent only during registration and not during the heartbeats since nothing
was modified.
For both the tests above, it was verified that events were published for the
above changes and that RecoveryConfigHelper captured those events and
invalidated the timestamp. During the next heartbeat, the new recovery
configuration information was pulled from the DB and send to the agent. It was
also verified that for subsequent heartbeats, the recovery configuration was
not sent.
Thanks,
Nahappan Somasundaram
