----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/27396/#review59451 -----------------------------------------------------------
ambari-agent/src/main/python/ambari_agent/LogSchedulerHandler.py <https://reviews.apache.org/r/27396/#comment100733> I believe that this will create your own scheduler instance (along with its own locks, pools, and jobstore). However, we might want to confirm this. At the very least, maybe the alert and log schedulers should define a jobstore aside from using the default name? ambari-agent/src/main/python/ambari_agent/LogSchedulerHandler.py <https://reviews.apache.org/r/27396/#comment100732> make_cachedir seems to always be True; is it supposed to be configurable? ambari-agent/src/main/python/ambari_agent/LogSchedulerHandler.py <https://reviews.apache.org/r/27396/#comment100735> Why instantiate a new Scheduler here? start(self) also creates a new Scheduler. Maybe this should set Scheduler to None and then have start(self) check for None to instantiate one? ambari-agent/src/main/python/ambari_agent/LogSchedulerHandler.py <https://reviews.apache.org/r/27396/#comment100736> I know the documentation says that you can give the job a name on creation; I wasn't able to get it to work; I'm guessing you weren't either. ambari-agent/src/main/python/ambari_agent/LogSchedulerHandler.py <https://reviews.apache.org/r/27396/#comment100734> This will produce data in ambari-agent.out - I think we'd want to keep this ambari-agent.log by using logger.exception() Same goes for other areas where you have traceback ambari-agent/src/main/python/ambari_agent/loghandler/config.py <https://reviews.apache.org/r/27396/#comment100737> Perhaps log what component name you were looking for? Same for other areas where an Exception is constructed due to lack of a required parameter. ambari-agent/src/main/python/ambari_agent/loghandler/hdfsapi.py <https://reviews.apache.org/r/27396/#comment100738> I'm not sure this command will work for all deployments. In some cases, the "hdfs" user is the only one that can run these commands; and in security-enabled environments, that means using keytabs. Can you verify that this command will work in the above scenarios? Same goes for other hdfs commands. - Jonathan Hurley On Oct. 31, 2014, 7:50 p.m., Cabir Zounaidou wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/27396/ > ----------------------------------------------------------- > > (Updated Oct. 31, 2014, 7:50 p.m.) > > > Review request for Ambari, Alejandro Fernandez, Jonathan Hurley, John > Speidel, Mahadev Konar, Nate Cole, and Yusaku Sako. > > > Bugs: AMBARI-1522 > https://issues.apache.org/jira/browse/AMBARI-1522 > > > Repository: ambari > > > Description > ------- > > The idea is to run a scheduler which computes diffs in a component log and > pushes the diff (only if available) to HDFS at fixed interval. > > 1. Each component will have its own monitor > 2. The apscheduler threadpool controls the monitor execution. > 3. If the log directory is not available if the component (or service) is not > running, it will skip the processing. > 4. It saves the last read line index and last modified time for next iteration > 5. It uses HDFS shell utility to push log patch to HDFS. > 6. Right now, the component log directory is configured in a json file. In > the next iteration, it will try to automatically detected from the stack > config. > > > Diffs > ----- > > ambari-agent/src/main/python/ambari_agent/Controller.py dc3a1cf > ambari-agent/src/main/python/ambari_agent/LogSchedulerHandler.py > PRE-CREATION > ambari-agent/src/main/python/ambari_agent/loghandler/__init__.py > PRE-CREATION > ambari-agent/src/main/python/ambari_agent/loghandler/config.py PRE-CREATION > ambari-agent/src/main/python/ambari_agent/loghandler/hdfsapi.py > PRE-CREATION > ambari-agent/src/main/python/ambari_agent/loghandler/monitor.py > PRE-CREATION > ambari-agent/src/main/python/ambari_agent/loghandler/util.py PRE-CREATION > ambari-agent/src/test/python/ambari_agent/TestLogHandler.py PRE-CREATION > > ambari-agent/src/test/python/ambari_agent/dummy_files/log_handler_config.json > PRE-CREATION > > Diff: https://reviews.apache.org/r/27396/diff/ > > > Testing > ------- > > - Ran the ambari-agent with the patch and verified > 1. Started ambari-agent with the patch > 2. The scheduler started successfully > 3. The logs files are getting pushed to HDFS successfully. Verified using > HDFS shell utility. > - Ran the tests successfully > [INFO] > [INFO] > ------------------------------------------------------------------------ > [INFO] Building Ambari Agent 1.3.0-SNAPSHOT > [INFO] > ------------------------------------------------------------------------ > [INFO] > : > ---------------------------------------------------------------------- > Ran 324 tests in 8.282s > > OK > : > [INFO] > ------------------------------------------------------------------------ > [INFO] BUILD SUCCESS > [INFO] > ------------------------------------------------------------------------ > [INFO] Total time: 14.759s > [INFO] Finished at: Thu Oct 30 14:58:21 PDT 2014 > [INFO] Final Memory: 10M/4079M > [INFO] > ------------------------------------------------------------------------ > > > Thanks, > > Cabir Zounaidou > >
