Re: Review Request 64059: Initial implementation of Schedulable document deletion & archiving for Infra Solr

2017-11-28 Thread Attila Magyar 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64059/#review192018
---


Ship it!




Ship It!

- Attila Magyar


On Nov. 27, 2017, 3:39 p.m., Krisztian Kasa wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64059/
> ---
> 
> (Updated Nov. 27, 2017, 3:39 p.m.)
> 
> 
> Review request for Ambari, Attila Magyar, Oliver Szabo, and Robert Nettleton.
> 
> 
> Bugs: AMBARI-22514
> https://issues.apache.org/jira/browse/AMBARI-22514
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Initial implementation of Schedulable document deletion & archiving for Infra 
> Solr
> ==
> 
> Spring bacth job for
> 
> * exporting solr documents from hadoop_logs collection
> * saving the documents to filesystem. All document is converted to json and 
> written into new line.
> * compress each file using tar.gz
> * job can be started using ambari-infra-manager api and parameters can be 
> overwritten
> * s3 upload support (not used yet)
> * job can be configurad to export from any collection: solr query can be 
> defined in properties
> 
> 
> Diffs
> -
> 
>   ambari-infra/ambari-infra-manager/pom.xml aa86da82b2 
>   
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/CompositeFileAction.java
>  PRE-CREATION 
>   
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/Document.java
>  PRE-CREATION 
>   
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentDestination.java
>  PRE-CREATION 
>   
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportConfiguration.java
>  PRE-CREATION 
>   
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportJobListener.java
>  PRE-CREATION 
>   
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportProperties.java
>  PRE-CREATION 
>   
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportStepListener.java
>  PRE-CREATION 
>   
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExporter.java
>  PRE-CREATION 
>   
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentItemReader.java
>  PRE-CREATION 
>   
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentItemWriter.java
>  PRE-CREATION 
>   
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentIterator.java
>  PRE-CREATION 
>   
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentSource.java
>  PRE-CREATION 
>   
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/FileAction.java
>  PRE-CREATION 
>   
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/LocalDocumentItemWriter.java
>  PRE-CREATION 
>   
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/S3Properties.java
>  PRE-CREATION 
>   
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/S3Uploader.java
>  PRE-CREATION 
>   
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/SolrDocumentIterator.java
>  PRE-CREATION 
>   
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/SolrDocumentSource.java
>  PRE-CREATION 
>   
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/SolrQueryBuilder.java
>  PRE-CREATION 
>   
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/SolrQueryProperties.java
>  PRE-CREATION 
>   
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/TarGzCompressor.java
>  PRE-CREATION 
>   
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/manager/JobManager.java
>  fc0a4f71db 
>   
> ambari-infra/ambari-infra-manager/src/main/resources/infra-manager.properties 
> 81623764da 
>   ambari-infra/ambari-infra-manager/src/main/resources/log4j2.xml ad1adcdcba 
>   
> ambari-infra/ambari-infra-manager/src/test/java/org/apache/ambari/infra/job/archive/DocumentExporterTest.java
>  PRE-CREATION 
>   
> ambari-infra/ambari-infra-manager/src/test/java/org/apache/ambari/infra/job/archive/DocumentItemReaderTest.java
>  PRE-CREATION 
>   
> ambari-infra/ambari-infra-manager/src/test/java/org/apache/ambari/infra/job/archive/LocalDocumentItemWriterTest.java
>

Re: Review Request 64059: Initial implementation of Schedulable document deletion & archiving for Infra Solr

2017-11-27 Thread Oliver Szabo 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64059/#review191897
---


Ship it!




Ship It!

- Oliver Szabo


On Nov. 27, 2017, 3:39 p.m., Krisztian Kasa wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64059/
> ---
> 
> (Updated Nov. 27, 2017, 3:39 p.m.)
> 
> 
> Review request for Ambari, Attila Magyar, Oliver Szabo, and Robert Nettleton.
> 
> 
> Bugs: AMBARI-22514
> https://issues.apache.org/jira/browse/AMBARI-22514
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Initial implementation of Schedulable document deletion & archiving for Infra 
> Solr
> ==
> 
> Spring bacth job for
> 
> * exporting solr documents from hadoop_logs collection
> * saving the documents to filesystem. All document is converted to json and 
> written into new line.
> * compress each file using tar.gz
> * job can be started using ambari-infra-manager api and parameters can be 
> overwritten
> * s3 upload support (not used yet)
> * job can be configurad to export from any collection: solr query can be 
> defined in properties
> 
> 
> Diffs
> -
> 
>   ambari-infra/ambari-infra-manager/pom.xml aa86da82b2 
>   
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/CompositeFileAction.java
>  PRE-CREATION 
>   
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/Document.java
>  PRE-CREATION 
>   
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentDestination.java
>  PRE-CREATION 
>   
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportConfiguration.java
>  PRE-CREATION 
>   
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportJobListener.java
>  PRE-CREATION 
>   
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportProperties.java
>  PRE-CREATION 
>   
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportStepListener.java
>  PRE-CREATION 
>   
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExporter.java
>  PRE-CREATION 
>   
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentItemReader.java
>  PRE-CREATION 
>   
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentItemWriter.java
>  PRE-CREATION 
>   
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentIterator.java
>  PRE-CREATION 
>   
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentSource.java
>  PRE-CREATION 
>   
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/FileAction.java
>  PRE-CREATION 
>   
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/LocalDocumentItemWriter.java
>  PRE-CREATION 
>   
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/S3Properties.java
>  PRE-CREATION 
>   
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/S3Uploader.java
>  PRE-CREATION 
>   
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/SolrDocumentIterator.java
>  PRE-CREATION 
>   
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/SolrDocumentSource.java
>  PRE-CREATION 
>   
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/SolrQueryBuilder.java
>  PRE-CREATION 
>   
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/SolrQueryProperties.java
>  PRE-CREATION 
>   
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/TarGzCompressor.java
>  PRE-CREATION 
>   
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/manager/JobManager.java
>  fc0a4f71db 
>   
> ambari-infra/ambari-infra-manager/src/main/resources/infra-manager.properties 
> 81623764da 
>   ambari-infra/ambari-infra-manager/src/main/resources/log4j2.xml ad1adcdcba 
>   
> ambari-infra/ambari-infra-manager/src/test/java/org/apache/ambari/infra/job/archive/DocumentExporterTest.java
>  PRE-CREATION 
>   
> ambari-infra/ambari-infra-manager/src/test/java/org/apache/ambari/infra/job/archive/DocumentItemReaderTest.java
>  PRE-CREATION 
>   
> ambari-infra/ambari-infra-manager/src/test/java/org/apache/ambari/infra/job/archive/LocalDocumentItemWriterTest.java
>

Re: Review Request 64059: Initial implementation of Schedulable document deletion & archiving for Infra Solr

2017-11-27 Thread Krisztian Kasa 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64059/
---

(Updated Nov. 27, 2017, 3:39 p.m.)


Review request for Ambari, Attila Magyar, Oliver Szabo, and Robert Nettleton.


Bugs: AMBARI-22514
https://issues.apache.org/jira/browse/AMBARI-22514


Repository: ambari


Description (updated)
---

Initial implementation of Schedulable document deletion & archiving for Infra 
Solr
==

Spring bacth job for

* exporting solr documents from hadoop_logs collection
* saving the documents to filesystem. All document is converted to json and 
written into new line.
* compress each file using tar.gz
* job can be started using ambari-infra-manager api and parameters can be 
overwritten
* s3 upload support (not used yet)
* job can be configurad to export from any collection: solr query can be 
defined in properties


Diffs (updated)
-

  ambari-infra/ambari-infra-manager/pom.xml aa86da82b2 
  
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/CompositeFileAction.java
 PRE-CREATION 
  
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/Document.java
 PRE-CREATION 
  
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentDestination.java
 PRE-CREATION 
  
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportConfiguration.java
 PRE-CREATION 
  
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportJobListener.java
 PRE-CREATION 
  
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportProperties.java
 PRE-CREATION 
  
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportStepListener.java
 PRE-CREATION 
  
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExporter.java
 PRE-CREATION 
  
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentItemReader.java
 PRE-CREATION 
  
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentItemWriter.java
 PRE-CREATION 
  
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentIterator.java
 PRE-CREATION 
  
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentSource.java
 PRE-CREATION 
  
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/FileAction.java
 PRE-CREATION 
  
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/LocalDocumentItemWriter.java
 PRE-CREATION 
  
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/S3Properties.java
 PRE-CREATION 
  
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/S3Uploader.java
 PRE-CREATION 
  
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/SolrDocumentIterator.java
 PRE-CREATION 
  
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/SolrDocumentSource.java
 PRE-CREATION 
  
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/SolrQueryBuilder.java
 PRE-CREATION 
  
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/SolrQueryProperties.java
 PRE-CREATION 
  
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/TarGzCompressor.java
 PRE-CREATION 
  
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/manager/JobManager.java
 fc0a4f71db 
  ambari-infra/ambari-infra-manager/src/main/resources/infra-manager.properties 
81623764da 
  ambari-infra/ambari-infra-manager/src/main/resources/log4j2.xml ad1adcdcba 
  
ambari-infra/ambari-infra-manager/src/test/java/org/apache/ambari/infra/job/archive/DocumentExporterTest.java
 PRE-CREATION 
  
ambari-infra/ambari-infra-manager/src/test/java/org/apache/ambari/infra/job/archive/DocumentItemReaderTest.java
 PRE-CREATION 
  
ambari-infra/ambari-infra-manager/src/test/java/org/apache/ambari/infra/job/archive/LocalDocumentItemWriterTest.java
 PRE-CREATION 
  
ambari-infra/ambari-infra-manager/src/test/java/org/apache/ambari/infra/job/archive/SolrQueryBuilderTest.java
 PRE-CREATION 
  ambari-logsearch/docker/test-config/logfeeder/logfeeder.properties fb7ddf2845 


Diff: https://reviews.apache.org/r/64059/diff/3/

Changes: https://reviews.apache.org/r/64059/diff/2-3/


Testing
---

* unit tests
* manual test using docker-compose


Thanks,

Krisztian Kasa

Re: Review Request 64059: Initial implementation of Schedulable document deletion & archiving for Infra Solr

2017-11-24 Thread Oliver Szabo 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64059/#review191833
---




ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/HadoopLogSource.java
Lines 27 (patched)


it can be service log source



ambari-infra/pom.xml
Lines 163 (patched)


is this really needed? is it becase of you reached the limit of the log 
file size for long running and it started to rotate? also it is too specific so 
it can be * * / * . log - * (i just use spaces to the editor not format it)


- Oliver Szabo


On Nov. 24, 2017, 9:31 a.m., Krisztian Kasa wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64059/
> ---
> 
> (Updated Nov. 24, 2017, 9:31 a.m.)
> 
> 
> Review request for Ambari, Attila Magyar, Oliver Szabo, and Robert Nettleton.
> 
> 
> Bugs: AMBARI-22514
> https://issues.apache.org/jira/browse/AMBARI-22514
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Initial implementation of Schedulable document deletion & archiving for Infra 
> Solr
> ==
> 
> Spring bacth job for
> 
> * exporting solr documents from hadoop_logs collection
> * saving the documents to filesystem. All document is converted to json and 
> written into new line.
> * compress each file using tar.gz
> * job can be started using ambari-infra-manager api and parameters can be 
> overwritten
> * s3 upload support (not used yet)
> 
> 
> Diffs
> -
> 
>   ambari-infra/ambari-infra-manager/pom.xml aa86da82b2 
>   
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/CompositeFileAction.java
>  PRE-CREATION 
>   
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/Document.java
>  PRE-CREATION 
>   
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentDestination.java
>  PRE-CREATION 
>   
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportConfiguration.java
>  PRE-CREATION 
>   
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportJobListener.java
>  PRE-CREATION 
>   
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportProperties.java
>  PRE-CREATION 
>   
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportStepListener.java
>  PRE-CREATION 
>   
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExporter.java
>  PRE-CREATION 
>   
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentItemReader.java
>  PRE-CREATION 
>   
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentItemWriter.java
>  PRE-CREATION 
>   
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentIterator.java
>  PRE-CREATION 
>   
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentSource.java
>  PRE-CREATION 
>   
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/FileAction.java
>  PRE-CREATION 
>   
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/HadoopLogSource.java
>  PRE-CREATION 
>   
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/LocalDocumentItemWriter.java
>  PRE-CREATION 
>   
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/S3Properties.java
>  PRE-CREATION 
>   
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/S3Uploader.java
>  PRE-CREATION 
>   
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/SolrDocumentIterator.java
>  PRE-CREATION 
>   
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/SolrDocumentSource.java
>  PRE-CREATION 
>   
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/TarGzCompressor.java
>  PRE-CREATION 
>   
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/manager/JobManager.java
>  fc0a4f71db 
>   
> ambari-infra/ambari-infra-manager/src/main/resources/infra-manager.properties 
> 81623764da 
>   
> ambari-infra/ambari-infra-manager/src/test/java/org/apache/ambari/infra/job/archive/DocumentExporterTest.java
>  PRE-CREATION 
>   
>

Re: Review Request 64059: Initial implementation of Schedulable document deletion & archiving for Infra Solr

2017-11-24 Thread Attila Magyar 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64059/#review191827
---




ambari-infra/ambari-infra-manager/src/test/java/org/apache/ambari/infra/job/archive/LocalDocumentItemWriterTest.java
Lines 85 (patched)


you can use org.apache.commons.io.FileUtils>>readLines if you have 
commons-io as a dependency


- Attila Magyar


On Nov. 24, 2017, 9:31 a.m., Krisztian Kasa wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64059/
> ---
> 
> (Updated Nov. 24, 2017, 9:31 a.m.)
> 
> 
> Review request for Ambari, Attila Magyar, Oliver Szabo, and Robert Nettleton.
> 
> 
> Bugs: AMBARI-22514
> https://issues.apache.org/jira/browse/AMBARI-22514
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Initial implementation of Schedulable document deletion & archiving for Infra 
> Solr
> ==
> 
> Spring bacth job for
> 
> * exporting solr documents from hadoop_logs collection
> * saving the documents to filesystem. All document is converted to json and 
> written into new line.
> * compress each file using tar.gz
> * job can be started using ambari-infra-manager api and parameters can be 
> overwritten
> * s3 upload support (not used yet)
> 
> 
> Diffs
> -
> 
>   ambari-infra/ambari-infra-manager/pom.xml aa86da82b2 
>   
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/CompositeFileAction.java
>  PRE-CREATION 
>   
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/Document.java
>  PRE-CREATION 
>   
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentDestination.java
>  PRE-CREATION 
>   
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportConfiguration.java
>  PRE-CREATION 
>   
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportJobListener.java
>  PRE-CREATION 
>   
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportProperties.java
>  PRE-CREATION 
>   
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportStepListener.java
>  PRE-CREATION 
>   
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExporter.java
>  PRE-CREATION 
>   
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentItemReader.java
>  PRE-CREATION 
>   
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentItemWriter.java
>  PRE-CREATION 
>   
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentIterator.java
>  PRE-CREATION 
>   
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentSource.java
>  PRE-CREATION 
>   
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/FileAction.java
>  PRE-CREATION 
>   
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/HadoopLogSource.java
>  PRE-CREATION 
>   
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/LocalDocumentItemWriter.java
>  PRE-CREATION 
>   
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/S3Properties.java
>  PRE-CREATION 
>   
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/S3Uploader.java
>  PRE-CREATION 
>   
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/SolrDocumentIterator.java
>  PRE-CREATION 
>   
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/SolrDocumentSource.java
>  PRE-CREATION 
>   
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/TarGzCompressor.java
>  PRE-CREATION 
>   
> ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/manager/JobManager.java
>  fc0a4f71db 
>   
> ambari-infra/ambari-infra-manager/src/main/resources/infra-manager.properties 
> 81623764da 
>   
> ambari-infra/ambari-infra-manager/src/test/java/org/apache/ambari/infra/job/archive/DocumentExporterTest.java
>  PRE-CREATION 
>   
> ambari-infra/ambari-infra-manager/src/test/java/org/apache/ambari/infra/job/archive/DocumentItemReaderTest.java
>  PRE-CREATION 
>   
> ambari-infra/ambari-infra-manager/src/test/java/org/apache/ambari/infra/job/archive/LocalDocumentItemWriterTest.java
>  PRE-CREATION 
>   ambari-infra/pom.xml e4b6a7601f

Re: Review Request 64059: Initial implementation of Schedulable document deletion & archiving for Infra Solr

2017-11-24 Thread Krisztian Kasa 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64059/
---

(Updated Nov. 24, 2017, 9:31 a.m.)


Review request for Ambari, Attila Magyar, Oliver Szabo, and Robert Nettleton.


Bugs: AMBARI-22514
https://issues.apache.org/jira/browse/AMBARI-22514


Repository: ambari


Description
---

Initial implementation of Schedulable document deletion & archiving for Infra 
Solr
==

Spring bacth job for

* exporting solr documents from hadoop_logs collection
* saving the documents to filesystem. All document is converted to json and 
written into new line.
* compress each file using tar.gz
* job can be started using ambari-infra-manager api and parameters can be 
overwritten
* s3 upload support (not used yet)


Diffs (updated)
-

  ambari-infra/ambari-infra-manager/pom.xml aa86da82b2 
  
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/CompositeFileAction.java
 PRE-CREATION 
  
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/Document.java
 PRE-CREATION 
  
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentDestination.java
 PRE-CREATION 
  
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportConfiguration.java
 PRE-CREATION 
  
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportJobListener.java
 PRE-CREATION 
  
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportProperties.java
 PRE-CREATION 
  
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportStepListener.java
 PRE-CREATION 
  
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExporter.java
 PRE-CREATION 
  
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentItemReader.java
 PRE-CREATION 
  
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentItemWriter.java
 PRE-CREATION 
  
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentIterator.java
 PRE-CREATION 
  
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentSource.java
 PRE-CREATION 
  
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/FileAction.java
 PRE-CREATION 
  
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/HadoopLogSource.java
 PRE-CREATION 
  
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/LocalDocumentItemWriter.java
 PRE-CREATION 
  
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/S3Properties.java
 PRE-CREATION 
  
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/S3Uploader.java
 PRE-CREATION 
  
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/SolrDocumentIterator.java
 PRE-CREATION 
  
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/SolrDocumentSource.java
 PRE-CREATION 
  
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/TarGzCompressor.java
 PRE-CREATION 
  
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/manager/JobManager.java
 fc0a4f71db 
  ambari-infra/ambari-infra-manager/src/main/resources/infra-manager.properties 
81623764da 
  
ambari-infra/ambari-infra-manager/src/test/java/org/apache/ambari/infra/job/archive/DocumentExporterTest.java
 PRE-CREATION 
  
ambari-infra/ambari-infra-manager/src/test/java/org/apache/ambari/infra/job/archive/DocumentItemReaderTest.java
 PRE-CREATION 
  
ambari-infra/ambari-infra-manager/src/test/java/org/apache/ambari/infra/job/archive/LocalDocumentItemWriterTest.java
 PRE-CREATION 
  ambari-infra/pom.xml e4b6a7601f 
  ambari-logsearch/docker/test-config/logfeeder/logfeeder.properties fb7ddf2845 


Diff: https://reviews.apache.org/r/64059/diff/2/

Changes: https://reviews.apache.org/r/64059/diff/1-2/


Testing
---

* unit tests
* manual test using docker-compose


Thanks,

Krisztian Kasa

Review Request 64059: Initial implementation of Schedulable document deletion & archiving for Infra Solr

2017-11-24 Thread Krisztian Kasa 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64059/
---

Review request for Ambari, Attila Magyar, Oliver Szabo, and Robert Nettleton.


Bugs: AMBARI-22514
https://issues.apache.org/jira/browse/AMBARI-22514


Repository: ambari


Description
---

Initial implementation of Schedulable document deletion & archiving for Infra 
Solr
==

Spring bacth job for

* exporting solr documents from hadoop_logs collection
* saving the documents to filesystem. All document is converted to json and 
written into new line.
* compress each file using tar.gz
* job can be started using ambari-infra-manager api and parameters can be 
overwritten
* s3 upload support (not used yet)


Diffs
-

  ambari-infra/ambari-infra-manager/pom.xml aa86da82b2 
  
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/CompositeFileAction.java
 PRE-CREATION 
  
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/Document.java
 PRE-CREATION 
  
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentDestination.java
 PRE-CREATION 
  
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportConfiguration.java
 PRE-CREATION 
  
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportJobListener.java
 PRE-CREATION 
  
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportProperties.java
 PRE-CREATION 
  
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportStepListener.java
 PRE-CREATION 
  
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExporter.java
 PRE-CREATION 
  
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentItemReader.java
 PRE-CREATION 
  
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentItemWriter.java
 PRE-CREATION 
  
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentIterator.java
 PRE-CREATION 
  
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentSource.java
 PRE-CREATION 
  
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/FileAction.java
 PRE-CREATION 
  
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/HadoopLogSource.java
 PRE-CREATION 
  
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/LocalDocumentItemWriter.java
 PRE-CREATION 
  
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/S3Properties.java
 PRE-CREATION 
  
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/S3Uploader.java
 PRE-CREATION 
  
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/SolrDocumentIterator.java
 PRE-CREATION 
  
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/SolrDocumentSource.java
 PRE-CREATION 
  
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/TarGzCompressor.java
 PRE-CREATION 
  
ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/manager/JobManager.java
 fc0a4f71db 
  ambari-infra/ambari-infra-manager/src/main/resources/infra-manager.properties 
81623764da 
  
ambari-infra/ambari-infra-manager/src/test/java/org/apache/ambari/infra/job/archive/DocumentExporterTest.java
 PRE-CREATION 
  
ambari-infra/ambari-infra-manager/src/test/java/org/apache/ambari/infra/job/archive/DocumentItemReaderTest.java
 PRE-CREATION 
  
ambari-infra/ambari-infra-manager/src/test/java/org/apache/ambari/infra/job/archive/LocalDocumentItemWriterTest.java
 PRE-CREATION 
  ambari-infra/pom.xml e4b6a7601f 
  ambari-logsearch/docker/test-config/logfeeder/logfeeder.properties fb7ddf2845 


Diff: https://reviews.apache.org/r/64059/diff/1/


Testing
---

* unit tests
* manual test using docker-compose


Thanks,

Krisztian Kasa