-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50884/
-----------------------------------------------------------

Review request for Ambari, Hayat Behlim, Dharmesh Makwana, Miklos Gergely, and 
Oliver Szabo.


Bugs: AMBARI-17785
    https://issues.apache.org/jira/browse/AMBARI-17785


Repository: ambari


Description
-------

This patch adds support for periodically uploading logs to S3 from logfeeder. 
The set of changes included are:

* Supports one at a time processing of log events in the OutputS3File case. 
These are spooled locally and uploaded periodically to S3.
* Supports upload based on two criteria - file size threshold, and time based 
threshold.
* Refactors code to achieve the above, while not duplicating any existing 
functions - for e.g. the code path to upload files all at once is still 
retained and uses the same helper classes like S3Uploader etc.
* Unit tests added for all new code.

What is **not** part of this patch, which I plan to cover in forthcoming 
patches:

* Support for other security models as applicable (for e.g. using IAM role 
based security for instances running on EC2)
* Better resilience to errors and fault tolerance.
* Copying configs, if required, to the S3 buckets.


Diffs
-----

  
ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/InputMarker.java
 8def4b9 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/OutputS3File.java
 f42195c 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/S3LogPathResolver.java
 PRE-CREATION 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/S3OutputConfiguration.java
 PRE-CREATION 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/S3Uploader.java
 PRE-CREATION 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/spool/LogSpooler.java
 306326a 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/s3/S3Util.java
 ced2b5c 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/test/java/org/apache/ambari/logfeeder/output/OutputS3FileTest.java
 PRE-CREATION 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/test/java/org/apache/ambari/logfeeder/output/S3LogPathResolverTest.java
 PRE-CREATION 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/test/java/org/apache/ambari/logfeeder/output/S3UploaderTest.java
 PRE-CREATION 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/test/java/org/apache/ambari/logfeeder/output/spool/LogSpoolerTest.java
 7d9d78a 

Diff: https://reviews.apache.org/r/50884/diff/


Testing
-------

* New UTs added for the new functionality
* Have manually tested that uploads to S3 occur based on reaching the file size 
threshold and time based threshold both.


Thanks,

Hemanth Yamijala

Reply via email to