Re: Review Request 63793: Log Feeder: cleanup checkpoint files periodically
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/63793/#review190965 --- Ship it! Ship It! - Robert Nettleton On Nov. 14, 2017, 1:07 p.m., Oliver Szabo wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/63793/ > --- > > (Updated Nov. 14, 2017, 1:07 p.m.) > > > Review request for Ambari, Krisztian Kasa and Robert Nettleton. > > > Bugs: AMBARI-22395 > https://issues.apache.org/jira/browse/AMBARI-22395 > > > Repository: ambari > > > Description > --- > > - added new config: "max_age_min" (similar as detach_time_min) to input > block. that stored in checkpoint files, during cleanup, based on that the > checkpoint file can be deleted. > - based on "max_age_min" value, during bootstrap, its possible logfeeder wont > even monitor a file in the future. > - created a new thread which periodically run the checkpoint cleanup (now in > every 2000 minutes ... its ok to run that rearly) > - fix metrics sink, as its called if some properties are missing > - add new input test files (testing wildcard pattern) > - fix when fatal error thrown if no any files exist based on a wildcard > pattern > > > Diffs > - > > > ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/AbstractInputFile.java > 8548a20 > > ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/Input.java > 96320e9 > > ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/InputManager.java > 8e70850 > > ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/monitor/AbstractLogFileMonitor.java > 3910b9b > > ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/monitor/CheckpointCleanupMonitor.java > PRE-CREATION > > ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/monitor/LogFileDetachMonitor.java > 322a56d > > ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/monitor/LogFilePathUpdateMonitor.java > cc5d664 > > ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/metrics/LogFeederAMSClient.java > 2d1bf40 > > ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/metrics/MetricsManager.java > 1432c87 > > ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/util/FileUtil.java > 843ae6b > > ambari-logsearch/ambari-logsearch-logfeeder/src/test/java/org/apache/ambari/logfeeder/metrics/MetrcisManagerTest.java > 8ee6d00 > ambari-logsearch/docker/test-config/logfeeder/logfeeder.properties 068bc3a > > ambari-logsearch/docker/test-config/logfeeder/shipper-conf/input.config-storm.json > PRE-CREATION > > ambari-logsearch/docker/test-logs/storm/worker-logs/streamline-1-TestAgg-2-3/6701/worker.log > PRE-CREATION > > > Diff: https://reviews.apache.org/r/63793/diff/3/ > > > Testing > --- > > done, manually, added new input file for docker env. > > > Thanks, > > Oliver Szabo > >
Re: Review Request 63793: Log Feeder: cleanup checkpoint files periodically
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/63793/ --- (Updated Nov. 14, 2017, 1:07 p.m.) Review request for Ambari, Krisztian Kasa and Robert Nettleton. Changes --- use waitIntervalMin as parameter Bugs: AMBARI-22395 https://issues.apache.org/jira/browse/AMBARI-22395 Repository: ambari Description --- - added new config: "max_age_min" (similar as detach_time_min) to input block. that stored in checkpoint files, during cleanup, based on that the checkpoint file can be deleted. - based on "max_age_min" value, during bootstrap, its possible logfeeder wont even monitor a file in the future. - created a new thread which periodically run the checkpoint cleanup (now in every 2000 minutes ... its ok to run that rearly) - fix metrics sink, as its called if some properties are missing - add new input test files (testing wildcard pattern) - fix when fatal error thrown if no any files exist based on a wildcard pattern Diffs (updated) - ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/AbstractInputFile.java 8548a20 ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/Input.java 96320e9 ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/InputManager.java 8e70850 ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/monitor/AbstractLogFileMonitor.java 3910b9b ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/monitor/CheckpointCleanupMonitor.java PRE-CREATION ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/monitor/LogFileDetachMonitor.java 322a56d ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/monitor/LogFilePathUpdateMonitor.java cc5d664 ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/metrics/LogFeederAMSClient.java 2d1bf40 ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/metrics/MetricsManager.java 1432c87 ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/util/FileUtil.java 843ae6b ambari-logsearch/ambari-logsearch-logfeeder/src/test/java/org/apache/ambari/logfeeder/metrics/MetrcisManagerTest.java 8ee6d00 ambari-logsearch/docker/test-config/logfeeder/logfeeder.properties 068bc3a ambari-logsearch/docker/test-config/logfeeder/shipper-conf/input.config-storm.json PRE-CREATION ambari-logsearch/docker/test-logs/storm/worker-logs/streamline-1-TestAgg-2-3/6701/worker.log PRE-CREATION Diff: https://reviews.apache.org/r/63793/diff/3/ Changes: https://reviews.apache.org/r/63793/diff/2-3/ Testing --- done, manually, added new input file for docker env. Thanks, Oliver Szabo
Re: Review Request 63793: Log Feeder: cleanup checkpoint files periodically
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/63793/#review190941 --- Ship it! Ship It! - Krisztian Kasa On Nov. 14, 2017, 12:50 p.m., Oliver Szabo wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/63793/ > --- > > (Updated Nov. 14, 2017, 12:50 p.m.) > > > Review request for Ambari, Krisztian Kasa and Robert Nettleton. > > > Bugs: AMBARI-22395 > https://issues.apache.org/jira/browse/AMBARI-22395 > > > Repository: ambari > > > Description > --- > > - added new config: "max_age_min" (similar as detach_time_min) to input > block. that stored in checkpoint files, during cleanup, based on that the > checkpoint file can be deleted. > - based on "max_age_min" value, during bootstrap, its possible logfeeder wont > even monitor a file in the future. > - created a new thread which periodically run the checkpoint cleanup (now in > every 2000 minutes ... its ok to run that rearly) > - fix metrics sink, as its called if some properties are missing > - add new input test files (testing wildcard pattern) > - fix when fatal error thrown if no any files exist based on a wildcard > pattern > > > Diffs > - > > > ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/AbstractInputFile.java > 8548a20 > > ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/Input.java > 96320e9 > > ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/InputManager.java > 8e70850 > > ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/monitor/AbstractLogFileMonitor.java > 3910b9b > > ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/monitor/CheckpointCleanupMonitor.java > PRE-CREATION > > ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/monitor/LogFileDetachMonitor.java > 322a56d > > ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/monitor/LogFilePathUpdateMonitor.java > cc5d664 > > ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/metrics/LogFeederAMSClient.java > 2d1bf40 > > ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/metrics/MetricsManager.java > 1432c87 > > ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/util/FileUtil.java > 843ae6b > > ambari-logsearch/ambari-logsearch-logfeeder/src/test/java/org/apache/ambari/logfeeder/metrics/MetrcisManagerTest.java > 8ee6d00 > ambari-logsearch/docker/test-config/logfeeder/logfeeder.properties 068bc3a > > ambari-logsearch/docker/test-config/logfeeder/shipper-conf/input.config-storm.json > PRE-CREATION > > ambari-logsearch/docker/test-logs/storm/worker-logs/streamline-1-TestAgg-2-3/6701/worker.log > PRE-CREATION > > > Diff: https://reviews.apache.org/r/63793/diff/2/ > > > Testing > --- > > done, manually, added new input file for docker env. > > > Thanks, > > Oliver Szabo > >
Re: Review Request 63793: Log Feeder: cleanup checkpoint files periodically
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/63793/#review190940 --- waitInterval -> waitIntervalMin - Krisztian Kasa On Nov. 14, 2017, 12:50 p.m., Oliver Szabo wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/63793/ > --- > > (Updated Nov. 14, 2017, 12:50 p.m.) > > > Review request for Ambari, Krisztian Kasa and Robert Nettleton. > > > Bugs: AMBARI-22395 > https://issues.apache.org/jira/browse/AMBARI-22395 > > > Repository: ambari > > > Description > --- > > - added new config: "max_age_min" (similar as detach_time_min) to input > block. that stored in checkpoint files, during cleanup, based on that the > checkpoint file can be deleted. > - based on "max_age_min" value, during bootstrap, its possible logfeeder wont > even monitor a file in the future. > - created a new thread which periodically run the checkpoint cleanup (now in > every 2000 minutes ... its ok to run that rearly) > - fix metrics sink, as its called if some properties are missing > - add new input test files (testing wildcard pattern) > - fix when fatal error thrown if no any files exist based on a wildcard > pattern > > > Diffs > - > > > ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/AbstractInputFile.java > 8548a20 > > ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/Input.java > 96320e9 > > ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/InputManager.java > 8e70850 > > ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/monitor/AbstractLogFileMonitor.java > 3910b9b > > ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/monitor/CheckpointCleanupMonitor.java > PRE-CREATION > > ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/monitor/LogFileDetachMonitor.java > 322a56d > > ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/monitor/LogFilePathUpdateMonitor.java > cc5d664 > > ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/metrics/LogFeederAMSClient.java > 2d1bf40 > > ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/metrics/MetricsManager.java > 1432c87 > > ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/util/FileUtil.java > 843ae6b > > ambari-logsearch/ambari-logsearch-logfeeder/src/test/java/org/apache/ambari/logfeeder/metrics/MetrcisManagerTest.java > 8ee6d00 > ambari-logsearch/docker/test-config/logfeeder/logfeeder.properties 068bc3a > > ambari-logsearch/docker/test-config/logfeeder/shipper-conf/input.config-storm.json > PRE-CREATION > > ambari-logsearch/docker/test-logs/storm/worker-logs/streamline-1-TestAgg-2-3/6701/worker.log > PRE-CREATION > > > Diff: https://reviews.apache.org/r/63793/diff/2/ > > > Testing > --- > > done, manually, added new input file for docker env. > > > Thanks, > > Oliver Szabo > >
Re: Review Request 63793: Log Feeder: cleanup checkpoint files periodically
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/63793/ --- (Updated Nov. 14, 2017, 12:50 p.m.) Review request for Ambari, Krisztian Kasa and Robert Nettleton. Changes --- handle that if maxAgeMin value set to zero Bugs: AMBARI-22395 https://issues.apache.org/jira/browse/AMBARI-22395 Repository: ambari Description --- - added new config: "max_age_min" (similar as detach_time_min) to input block. that stored in checkpoint files, during cleanup, based on that the checkpoint file can be deleted. - based on "max_age_min" value, during bootstrap, its possible logfeeder wont even monitor a file in the future. - created a new thread which periodically run the checkpoint cleanup (now in every 2000 minutes ... its ok to run that rearly) - fix metrics sink, as its called if some properties are missing - add new input test files (testing wildcard pattern) - fix when fatal error thrown if no any files exist based on a wildcard pattern Diffs (updated) - ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/AbstractInputFile.java 8548a20 ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/Input.java 96320e9 ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/InputManager.java 8e70850 ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/monitor/AbstractLogFileMonitor.java 3910b9b ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/monitor/CheckpointCleanupMonitor.java PRE-CREATION ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/monitor/LogFileDetachMonitor.java 322a56d ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/monitor/LogFilePathUpdateMonitor.java cc5d664 ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/metrics/LogFeederAMSClient.java 2d1bf40 ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/metrics/MetricsManager.java 1432c87 ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/util/FileUtil.java 843ae6b ambari-logsearch/ambari-logsearch-logfeeder/src/test/java/org/apache/ambari/logfeeder/metrics/MetrcisManagerTest.java 8ee6d00 ambari-logsearch/docker/test-config/logfeeder/logfeeder.properties 068bc3a ambari-logsearch/docker/test-config/logfeeder/shipper-conf/input.config-storm.json PRE-CREATION ambari-logsearch/docker/test-logs/storm/worker-logs/streamline-1-TestAgg-2-3/6701/worker.log PRE-CREATION Diff: https://reviews.apache.org/r/63793/diff/2/ Changes: https://reviews.apache.org/r/63793/diff/1-2/ Testing --- done, manually, added new input file for docker env. Thanks, Oliver Szabo
Review Request 63793: Log Feeder: cleanup checkpoint files periodically
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/63793/ --- Review request for Ambari, Krisztian Kasa and Robert Nettleton. Bugs: AMBARI-22395 https://issues.apache.org/jira/browse/AMBARI-22395 Repository: ambari Description --- - added new config: "max_age_min" (similar as detach_time_min) to input block. that stored in checkpoint files, during cleanup, based on that the checkpoint file can be deleted. - based on "max_age_min" value, during bootstrap, its possible logfeeder wont even monitor a file in the future. - created a new thread which periodically run the checkpoint cleanup (now in every 2000 minutes ... its ok to run that rearly) - fix metrics sink, as its called if some properties are missing - add new input test files (testing wildcard pattern) - fix when fatal error thrown if no any files exist based on a wildcard pattern Diffs - ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/AbstractInputFile.java 8548a20 ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/Input.java 96320e9 ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/InputManager.java 8e70850 ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/monitor/AbstractLogFileMonitor.java 3910b9b ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/monitor/CheckpointCleanupMonitor.java PRE-CREATION ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/monitor/LogFileDetachMonitor.java 322a56d ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/monitor/LogFilePathUpdateMonitor.java cc5d664 ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/metrics/LogFeederAMSClient.java 2d1bf40 ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/metrics/MetricsManager.java 1432c87 ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/util/FileUtil.java 843ae6b ambari-logsearch/ambari-logsearch-logfeeder/src/test/java/org/apache/ambari/logfeeder/metrics/MetrcisManagerTest.java 8ee6d00 ambari-logsearch/docker/test-config/logfeeder/logfeeder.properties 068bc3a ambari-logsearch/docker/test-config/logfeeder/shipper-conf/input.config-storm.json PRE-CREATION ambari-logsearch/docker/test-logs/storm/worker-logs/streamline-1-TestAgg-2-3/6701/worker.log PRE-CREATION Diff: https://reviews.apache.org/r/63793/diff/1/ Testing --- done, manually, added new input file for docker env. Thanks, Oliver Szabo
