-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27396/
-----------------------------------------------------------
(Updated Nov. 5, 2014, 3:49 p.m.)
Review request for Ambari, Alejandro Fernandez, Jonathan Hurley, John Speidel,
Mahadev Konar, Nate Cole, and Yusaku Sako.
Changes
-------
Fixed as per Nate's review comments.
Bugs: AMBARI-1522
https://issues.apache.org/jira/browse/AMBARI-1522
Repository: ambari
Description
-------
The idea is to run a scheduler which computes diffs in a component log and
pushes the diff (only if available) to HDFS at fixed interval.
1. Each component will have its own monitor
2. The apscheduler threadpool controls the monitor execution.
3. If the log directory is not available if the component (or service) is not
running, it will skip the processing.
4. It saves the last read line index and last modified time for next iteration
5. It uses HDFS shell utility to push log patch to HDFS.
6. Right now, the component log directory is configured in a json file. In the
next iteration, it will try to automatically detected from the stack config.
Diffs (updated)
-----
ambari-agent/src/main/python/ambari_agent/AmbariConfig.py ca2e80c
ambari-agent/src/main/python/ambari_agent/Controller.py dc3a1cf
ambari-agent/src/main/python/ambari_agent/LogSchedulerHandler.py PRE-CREATION
ambari-agent/src/main/python/ambari_agent/loghandler/__init__.py PRE-CREATION
ambari-agent/src/main/python/ambari_agent/loghandler/config.py PRE-CREATION
ambari-agent/src/main/python/ambari_agent/loghandler/hdfsapi.py PRE-CREATION
ambari-agent/src/main/python/ambari_agent/loghandler/monitor.py PRE-CREATION
ambari-agent/src/main/python/ambari_agent/loghandler/util.py PRE-CREATION
ambari-agent/src/test/python/ambari_agent/TestHdfsApi.py PRE-CREATION
ambari-agent/src/test/python/ambari_agent/TestLogHandler.py PRE-CREATION
ambari-agent/src/test/python/ambari_agent/dummy_files/log_handler_config.json
PRE-CREATION
Diff: https://reviews.apache.org/r/27396/diff/
Testing
-------
- Ran the ambari-agent with the patch and verified
1. Started ambari-agent with the patch
2. The scheduler started successfully
3. The logs files are getting pushed to HDFS successfully. Verified using HDFS
shell utility.
- Ran the tests successfully
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building Ambari Agent 1.3.0-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO]
:
----------------------------------------------------------------------
Ran 324 tests in 8.282s
OK
:
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 14.759s
[INFO] Finished at: Thu Oct 30 14:58:21 PDT 2014
[INFO] Final Memory: 10M/4079M
[INFO] ------------------------------------------------------------------------
Thanks,
Cabir Zounaidou
