> On Nov. 1, 2014, 2:34 a.m., Jonathan Hurley wrote: > >
Jonathan, thanks very much for the review. > On Nov. 1, 2014, 2:34 a.m., Jonathan Hurley wrote: > > ambari-agent/src/main/python/ambari_agent/LogSchedulerHandler.py, line 52 > > <https://reviews.apache.org/r/27396/diff/2/?file=746519#file746519line52> > > > > I believe that this will create your own scheduler instance (along with > > its own locks, pools, and jobstore). However, we might want to confirm > > this. At the very least, maybe the alert and log schedulers should define a > > jobstore aside from using the default name? Yes, verified that each scheduler instance creates it own locks, thread pools and jobstore. Not sure we need to explicitly specify jobstore, verified that it creates a new instance of jobstore for each instance of scheduler. > On Nov. 1, 2014, 2:34 a.m., Jonathan Hurley wrote: > > ambari-agent/src/main/python/ambari_agent/LogSchedulerHandler.py, line 167 > > <https://reviews.apache.org/r/27396/diff/2/?file=746519#file746519line167> > > > > I know the documentation says that you can give the job a name on > > creation; I wasn't able to get it to work; I'm guessing you weren't either. i am able to get the job name to work. Will fix the code to use name options parameter. scheduler.add_interval_job(callback, seconds=self.__interval, name=jobname) > On Nov. 1, 2014, 2:34 a.m., Jonathan Hurley wrote: > > ambari-agent/src/main/python/ambari_agent/loghandler/config.py, line 61 > > <https://reviews.apache.org/r/27396/diff/2/?file=746521#file746521line61> > > > > Perhaps log what component name you were looking for? Same for other > > areas where an Exception is constructed due to lack of a required parameter. The exception is raised because the component name is missing. Changed the message to 'component name required' > On Nov. 1, 2014, 2:34 a.m., Jonathan Hurley wrote: > > ambari-agent/src/main/python/ambari_agent/LogSchedulerHandler.py, line 179 > > <https://reviews.apache.org/r/27396/diff/2/?file=746519#file746519line179> > > > > This will produce data in ambari-agent.out - I think we'd want to keep > > this ambari-agent.log by using logger.exception() > > > > Same goes for other areas where you have traceback Changed to log info message, because this file need not have to exists and will be automatically generated by the execution command. > On Nov. 1, 2014, 2:34 a.m., Jonathan Hurley wrote: > > ambari-agent/src/main/python/ambari_agent/loghandler/hdfsapi.py, line 31 > > <https://reviews.apache.org/r/27396/diff/2/?file=746522#file746522line31> > > > > I'm not sure this command will work for all deployments. In some cases, > > the "hdfs" user is the only one that can run these commands; and in > > security-enabled environments, that means using keytabs. > > > > Can you verify that this command will work in the above scenarios? > > > > Same goes for other hdfs commands. Good point. Verified that it is an issue for 'ambari' user running hdfs command in secured mode. I will check using 'hdfs' user is okay for this use case. > On Nov. 1, 2014, 2:34 a.m., Jonathan Hurley wrote: > > ambari-agent/src/main/python/ambari_agent/LogSchedulerHandler.py, line 54 > > <https://reviews.apache.org/r/27396/diff/2/?file=746519#file746519line54> > > > > make_cachedir seems to always be True; is it supposed to be > > configurable? Made it as in input parameter. - Cabir ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/27396/#review59451 ----------------------------------------------------------- On Oct. 31, 2014, 11:50 p.m., Cabir Zounaidou wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/27396/ > ----------------------------------------------------------- > > (Updated Oct. 31, 2014, 11:50 p.m.) > > > Review request for Ambari, Alejandro Fernandez, Jonathan Hurley, John > Speidel, Mahadev Konar, Nate Cole, and Yusaku Sako. > > > Bugs: AMBARI-1522 > https://issues.apache.org/jira/browse/AMBARI-1522 > > > Repository: ambari > > > Description > ------- > > The idea is to run a scheduler which computes diffs in a component log and > pushes the diff (only if available) to HDFS at fixed interval. > > 1. Each component will have its own monitor > 2. The apscheduler threadpool controls the monitor execution. > 3. If the log directory is not available if the component (or service) is not > running, it will skip the processing. > 4. It saves the last read line index and last modified time for next iteration > 5. It uses HDFS shell utility to push log patch to HDFS. > 6. Right now, the component log directory is configured in a json file. In > the next iteration, it will try to automatically detected from the stack > config. > > > Diffs > ----- > > ambari-agent/src/main/python/ambari_agent/Controller.py dc3a1cf > ambari-agent/src/main/python/ambari_agent/LogSchedulerHandler.py > PRE-CREATION > ambari-agent/src/main/python/ambari_agent/loghandler/__init__.py > PRE-CREATION > ambari-agent/src/main/python/ambari_agent/loghandler/config.py PRE-CREATION > ambari-agent/src/main/python/ambari_agent/loghandler/hdfsapi.py > PRE-CREATION > ambari-agent/src/main/python/ambari_agent/loghandler/monitor.py > PRE-CREATION > ambari-agent/src/main/python/ambari_agent/loghandler/util.py PRE-CREATION > ambari-agent/src/test/python/ambari_agent/TestLogHandler.py PRE-CREATION > > ambari-agent/src/test/python/ambari_agent/dummy_files/log_handler_config.json > PRE-CREATION > > Diff: https://reviews.apache.org/r/27396/diff/ > > > Testing > ------- > > - Ran the ambari-agent with the patch and verified > 1. Started ambari-agent with the patch > 2. The scheduler started successfully > 3. The logs files are getting pushed to HDFS successfully. Verified using > HDFS shell utility. > - Ran the tests successfully > [INFO] > [INFO] > ------------------------------------------------------------------------ > [INFO] Building Ambari Agent 1.3.0-SNAPSHOT > [INFO] > ------------------------------------------------------------------------ > [INFO] > : > ---------------------------------------------------------------------- > Ran 324 tests in 8.282s > > OK > : > [INFO] > ------------------------------------------------------------------------ > [INFO] BUILD SUCCESS > [INFO] > ------------------------------------------------------------------------ > [INFO] Total time: 14.759s > [INFO] Finished at: Thu Oct 30 14:58:21 PDT 2014 > [INFO] Final Memory: 10M/4079M > [INFO] > ------------------------------------------------------------------------ > > > Thanks, > > Cabir Zounaidou > >
