Re: Review Request 51969: AMBARI-18413: RecoveryManager may be logging too often
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/51969/#review149359 --- Ship it! Ship It! - Sebastian Toader On Sept. 16, 2016, 11:37 p.m., Nahappan Somasundaram wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/51969/ > --- > > (Updated Sept. 16, 2016, 11:37 p.m.) > > > Review request for Ambari, Sumit Mohanty and Sebastian Toader. > > > Bugs: AMBARI-18413 > https://issues.apache.org/jira/browse/AMBARI-18413 > > > Repository: ambari > > > Description > --- > > AMBARI-18413: RecoveryManager may be logging too often > > **Issue: ** > Heartbeat interval in Controller.py is inversely proportional to the number > of nodes in the cluster. For a small cluster, with 1-9 nodes, the frequency > of heartbeat is 1 second. This causes get_recovery_commands() to be called > every second thereby logging every second in > RecoveryManager::requires_recovery(component). > > **Fix: ** > In the heartbeat loop in Controller.py, call get_recovery_commands() every > HEARTBEAT_IDLE_INTERVAL_DEFAULT_MAX_SEC seconds which is currently set to 10 > seconds. > > > Diffs > - > > ambari-agent/src/main/python/ambari_agent/Controller.py > a05011a70aeb02e7336e796c8481ace21c095d7d > > Diff: https://reviews.apache.org/r/51969/diff/ > > > Testing > --- > > ** 1. mvn clean install -DskipTests ** > [INFO] > > [INFO] Reactor Summary: > [INFO] > [INFO] Ambari Main ... SUCCESS [7.938s] > [INFO] Apache Ambari Project POM . SUCCESS [0.038s] > [INFO] Ambari Web SUCCESS [49.323s] > [INFO] Ambari Views .. SUCCESS [1.238s] > [INFO] Ambari Admin View . SUCCESS [8.422s] > [INFO] ambari-metrics SUCCESS [0.674s] > [INFO] Ambari Metrics Common . SUCCESS [3.808s] > [INFO] Ambari Metrics Hadoop Sink SUCCESS [1.957s] > [INFO] Ambari Metrics Flume Sink . SUCCESS [1.151s] > [INFO] Ambari Metrics Kafka Sink . SUCCESS [1.343s] > [INFO] Ambari Metrics Storm Sink . SUCCESS [3.463s] > [INFO] Ambari Metrics Storm Sink (Legacy) SUCCESS [1.550s] > [INFO] Ambari Metrics Collector .. SUCCESS [11.210s] > [INFO] Ambari Metrics Monitor SUCCESS [2.094s] > [INFO] Ambari Metrics Grafana SUCCESS [0.992s] > [INFO] Ambari Metrics Assembly ... SUCCESS [1:20.958s] > [INFO] Ambari Server . SUCCESS [3:20.499s] > [INFO] Ambari Functional Tests ... SUCCESS [2.272s] > [INFO] Ambari Agent .. SUCCESS [28.364s] > [INFO] Ambari Client . SUCCESS [0.068s] > [INFO] Ambari Python Client .. SUCCESS [0.976s] > [INFO] Ambari Groovy Client .. SUCCESS [2.604s] > [INFO] Ambari Shell .. SUCCESS [0.077s] > [INFO] Ambari Python Shell ... SUCCESS [0.858s] > [INFO] Ambari Groovy Shell ... SUCCESS [1.145s] > [INFO] ambari-logsearch .. SUCCESS [0.270s] > [INFO] Ambari Logsearch Appender . SUCCESS [0.281s] > [INFO] Ambari Logsearch Solr Client .. SUCCESS [1.407s] > [INFO] Ambari Logsearch Portal ... SUCCESS [7.756s] > [INFO] Ambari Logsearch Log Feeder ... SUCCESS [4.872s] > [INFO] Ambari Logsearch Assembly . SUCCESS [0.103s] > [INFO] > > [INFO] BUILD SUCCESS > [INFO] > > [INFO] Total time: 7:09.767s > [INFO] Finished at: Fri Sep 16 12:46:20 PDT 2016 > [INFO] Final Memory: 304M/1196M > [INFO] > > > ** 2. mvn test -DskipSurefireTests ** > > -- > Ran 444 tests in 23.052s > > OK > > [INFO] Rat check: Summary of files. Unapproved: 0 unknown: 0 generated: 0 > approved: 148 licence. > [INFO] > > [INFO] BUILD SUCCESS > [INFO] >
Re: Review Request 51969: AMBARI-18413: RecoveryManager may be logging too often
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/51969/#review149289 --- Ship it! Ship It! - Sumit Mohanty On Sept. 16, 2016, 9:37 p.m., Nahappan Somasundaram wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/51969/ > --- > > (Updated Sept. 16, 2016, 9:37 p.m.) > > > Review request for Ambari, Sumit Mohanty and Sebastian Toader. > > > Bugs: AMBARI-18413 > https://issues.apache.org/jira/browse/AMBARI-18413 > > > Repository: ambari > > > Description > --- > > AMBARI-18413: RecoveryManager may be logging too often > > **Issue: ** > Heartbeat interval in Controller.py is inversely proportional to the number > of nodes in the cluster. For a small cluster, with 1-9 nodes, the frequency > of heartbeat is 1 second. This causes get_recovery_commands() to be called > every second thereby logging every second in > RecoveryManager::requires_recovery(component). > > **Fix: ** > In the heartbeat loop in Controller.py, call get_recovery_commands() every > HEARTBEAT_IDLE_INTERVAL_DEFAULT_MAX_SEC seconds which is currently set to 10 > seconds. > > > Diffs > - > > ambari-agent/src/main/python/ambari_agent/Controller.py > a05011a70aeb02e7336e796c8481ace21c095d7d > > Diff: https://reviews.apache.org/r/51969/diff/ > > > Testing > --- > > ** 1. mvn clean install -DskipTests ** > [INFO] > > [INFO] Reactor Summary: > [INFO] > [INFO] Ambari Main ... SUCCESS [7.938s] > [INFO] Apache Ambari Project POM . SUCCESS [0.038s] > [INFO] Ambari Web SUCCESS [49.323s] > [INFO] Ambari Views .. SUCCESS [1.238s] > [INFO] Ambari Admin View . SUCCESS [8.422s] > [INFO] ambari-metrics SUCCESS [0.674s] > [INFO] Ambari Metrics Common . SUCCESS [3.808s] > [INFO] Ambari Metrics Hadoop Sink SUCCESS [1.957s] > [INFO] Ambari Metrics Flume Sink . SUCCESS [1.151s] > [INFO] Ambari Metrics Kafka Sink . SUCCESS [1.343s] > [INFO] Ambari Metrics Storm Sink . SUCCESS [3.463s] > [INFO] Ambari Metrics Storm Sink (Legacy) SUCCESS [1.550s] > [INFO] Ambari Metrics Collector .. SUCCESS [11.210s] > [INFO] Ambari Metrics Monitor SUCCESS [2.094s] > [INFO] Ambari Metrics Grafana SUCCESS [0.992s] > [INFO] Ambari Metrics Assembly ... SUCCESS [1:20.958s] > [INFO] Ambari Server . SUCCESS [3:20.499s] > [INFO] Ambari Functional Tests ... SUCCESS [2.272s] > [INFO] Ambari Agent .. SUCCESS [28.364s] > [INFO] Ambari Client . SUCCESS [0.068s] > [INFO] Ambari Python Client .. SUCCESS [0.976s] > [INFO] Ambari Groovy Client .. SUCCESS [2.604s] > [INFO] Ambari Shell .. SUCCESS [0.077s] > [INFO] Ambari Python Shell ... SUCCESS [0.858s] > [INFO] Ambari Groovy Shell ... SUCCESS [1.145s] > [INFO] ambari-logsearch .. SUCCESS [0.270s] > [INFO] Ambari Logsearch Appender . SUCCESS [0.281s] > [INFO] Ambari Logsearch Solr Client .. SUCCESS [1.407s] > [INFO] Ambari Logsearch Portal ... SUCCESS [7.756s] > [INFO] Ambari Logsearch Log Feeder ... SUCCESS [4.872s] > [INFO] Ambari Logsearch Assembly . SUCCESS [0.103s] > [INFO] > > [INFO] BUILD SUCCESS > [INFO] > > [INFO] Total time: 7:09.767s > [INFO] Finished at: Fri Sep 16 12:46:20 PDT 2016 > [INFO] Final Memory: 304M/1196M > [INFO] > > > ** 2. mvn test -DskipSurefireTests ** > > -- > Ran 444 tests in 23.052s > > OK > > [INFO] Rat check: Summary of files. Unapproved: 0 unknown: 0 generated: 0 > approved: 148 licence. > [INFO] > > [INFO] BUILD SUCCESS > [INFO] > -
Re: Review Request 51969: AMBARI-18413: RecoveryManager may be logging too often
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/51969/ --- (Updated Sept. 16, 2016, 2:37 p.m.) Review request for Ambari, Sumit Mohanty and Sebastian Toader. Changes --- 1. Added comments to the change. 2. Simplified the fix by removing the flag to call get_recovery_commands() Bugs: AMBARI-18413 https://issues.apache.org/jira/browse/AMBARI-18413 Repository: ambari Description --- AMBARI-18413: RecoveryManager may be logging too often **Issue: ** Heartbeat interval in Controller.py is inversely proportional to the number of nodes in the cluster. For a small cluster, with 1-9 nodes, the frequency of heartbeat is 1 second. This causes get_recovery_commands() to be called every second thereby logging every second in RecoveryManager::requires_recovery(component). **Fix: ** In the heartbeat loop in Controller.py, call get_recovery_commands() every HEARTBEAT_IDLE_INTERVAL_DEFAULT_MAX_SEC seconds which is currently set to 10 seconds. Diffs (updated) - ambari-agent/src/main/python/ambari_agent/Controller.py a05011a70aeb02e7336e796c8481ace21c095d7d Diff: https://reviews.apache.org/r/51969/diff/ Testing (updated) --- ** 1. mvn clean install -DskipTests ** [INFO] [INFO] Reactor Summary: [INFO] [INFO] Ambari Main ... SUCCESS [7.938s] [INFO] Apache Ambari Project POM . SUCCESS [0.038s] [INFO] Ambari Web SUCCESS [49.323s] [INFO] Ambari Views .. SUCCESS [1.238s] [INFO] Ambari Admin View . SUCCESS [8.422s] [INFO] ambari-metrics SUCCESS [0.674s] [INFO] Ambari Metrics Common . SUCCESS [3.808s] [INFO] Ambari Metrics Hadoop Sink SUCCESS [1.957s] [INFO] Ambari Metrics Flume Sink . SUCCESS [1.151s] [INFO] Ambari Metrics Kafka Sink . SUCCESS [1.343s] [INFO] Ambari Metrics Storm Sink . SUCCESS [3.463s] [INFO] Ambari Metrics Storm Sink (Legacy) SUCCESS [1.550s] [INFO] Ambari Metrics Collector .. SUCCESS [11.210s] [INFO] Ambari Metrics Monitor SUCCESS [2.094s] [INFO] Ambari Metrics Grafana SUCCESS [0.992s] [INFO] Ambari Metrics Assembly ... SUCCESS [1:20.958s] [INFO] Ambari Server . SUCCESS [3:20.499s] [INFO] Ambari Functional Tests ... SUCCESS [2.272s] [INFO] Ambari Agent .. SUCCESS [28.364s] [INFO] Ambari Client . SUCCESS [0.068s] [INFO] Ambari Python Client .. SUCCESS [0.976s] [INFO] Ambari Groovy Client .. SUCCESS [2.604s] [INFO] Ambari Shell .. SUCCESS [0.077s] [INFO] Ambari Python Shell ... SUCCESS [0.858s] [INFO] Ambari Groovy Shell ... SUCCESS [1.145s] [INFO] ambari-logsearch .. SUCCESS [0.270s] [INFO] Ambari Logsearch Appender . SUCCESS [0.281s] [INFO] Ambari Logsearch Solr Client .. SUCCESS [1.407s] [INFO] Ambari Logsearch Portal ... SUCCESS [7.756s] [INFO] Ambari Logsearch Log Feeder ... SUCCESS [4.872s] [INFO] Ambari Logsearch Assembly . SUCCESS [0.103s] [INFO] [INFO] BUILD SUCCESS [INFO] [INFO] Total time: 7:09.767s [INFO] Finished at: Fri Sep 16 12:46:20 PDT 2016 [INFO] Final Memory: 304M/1196M [INFO] ** 2. mvn test -DskipSurefireTests ** -- Ran 444 tests in 23.052s OK [INFO] Rat check: Summary of files. Unapproved: 0 unknown: 0 generated: 0 approved: 148 licence. [INFO] [INFO] BUILD SUCCESS [INFO] [INFO] Total time: 26.392s [INFO] Finished at: Fri Sep 16 14:33:33 PDT 2016 [INFO] Final Memory: 11M/245M [INFO] ** 3. Manual tests ** * Setup a VM with the latest trunk build and replaced ** /usr/lib/python2.6/site-packages/ambari_agent/Controller.py ** with the fix. * Enabled auto
Review Request 51969: AMBARI-18413: RecoveryManager may be logging too often
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/51969/ --- Review request for Ambari, Sumit Mohanty and Sebastian Toader. Bugs: AMBARI-18413 https://issues.apache.org/jira/browse/AMBARI-18413 Repository: ambari Description --- AMBARI-18413: RecoveryManager may be logging too often **Issue: ** Heartbeat interval in Controller.py is inversely proportional to the number of nodes in the cluster. For a small cluster, with 1-9 nodes, the frequency of heartbeat is 1 second. This causes get_recovery_commands() to be called every second thereby logging every second in RecoveryManager::requires_recovery(component). **Fix: ** In the heartbeat loop in Controller.py, call get_recovery_commands() every HEARTBEAT_IDLE_INTERVAL_DEFAULT_MAX_SEC seconds which is currently set to 10 seconds. Diffs - ambari-agent/src/main/python/ambari_agent/Controller.py a05011a70aeb02e7336e796c8481ace21c095d7d Diff: https://reviews.apache.org/r/51969/diff/ Testing --- ** 1. mvn clean install -DskipTests ** [INFO] [INFO] Reactor Summary: [INFO] [INFO] Ambari Main ... SUCCESS [7.938s] [INFO] Apache Ambari Project POM . SUCCESS [0.038s] [INFO] Ambari Web SUCCESS [49.323s] [INFO] Ambari Views .. SUCCESS [1.238s] [INFO] Ambari Admin View . SUCCESS [8.422s] [INFO] ambari-metrics SUCCESS [0.674s] [INFO] Ambari Metrics Common . SUCCESS [3.808s] [INFO] Ambari Metrics Hadoop Sink SUCCESS [1.957s] [INFO] Ambari Metrics Flume Sink . SUCCESS [1.151s] [INFO] Ambari Metrics Kafka Sink . SUCCESS [1.343s] [INFO] Ambari Metrics Storm Sink . SUCCESS [3.463s] [INFO] Ambari Metrics Storm Sink (Legacy) SUCCESS [1.550s] [INFO] Ambari Metrics Collector .. SUCCESS [11.210s] [INFO] Ambari Metrics Monitor SUCCESS [2.094s] [INFO] Ambari Metrics Grafana SUCCESS [0.992s] [INFO] Ambari Metrics Assembly ... SUCCESS [1:20.958s] [INFO] Ambari Server . SUCCESS [3:20.499s] [INFO] Ambari Functional Tests ... SUCCESS [2.272s] [INFO] Ambari Agent .. SUCCESS [28.364s] [INFO] Ambari Client . SUCCESS [0.068s] [INFO] Ambari Python Client .. SUCCESS [0.976s] [INFO] Ambari Groovy Client .. SUCCESS [2.604s] [INFO] Ambari Shell .. SUCCESS [0.077s] [INFO] Ambari Python Shell ... SUCCESS [0.858s] [INFO] Ambari Groovy Shell ... SUCCESS [1.145s] [INFO] ambari-logsearch .. SUCCESS [0.270s] [INFO] Ambari Logsearch Appender . SUCCESS [0.281s] [INFO] Ambari Logsearch Solr Client .. SUCCESS [1.407s] [INFO] Ambari Logsearch Portal ... SUCCESS [7.756s] [INFO] Ambari Logsearch Log Feeder ... SUCCESS [4.872s] [INFO] Ambari Logsearch Assembly . SUCCESS [0.103s] [INFO] [INFO] BUILD SUCCESS [INFO] [INFO] Total time: 7:09.767s [INFO] Finished at: Fri Sep 16 12:46:20 PDT 2016 [INFO] Final Memory: 304M/1196M [INFO] ** 2. mvn test -DskipSurefireTests ** -- Ran 268 tests in 6.688s OK -- Total run:1126 Total errors:0 Total failures:0 OK [INFO] [INFO] BUILD SUCCESS [INFO] [INFO] Total time: 1:09.146s [INFO] Finished at: Fri Sep 16 13:21:43 PDT 2016 [INFO] Final Memory: 47M/821M [INFO] ** 3. Manual tests ** * Setup a VM with the latest trunk build and replaced ** /usr/lib/python2.6/site-packages/ambari_agent/Controller.py ** with the fix. * Enabled auto-start for all services * Deployed a few services and observed that ** tail -f /var/log/ambari-agent/ambari-agent.log | grep RecoveryManager ** spewed lot less l
