---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57495/#review168686
---
Ship it!
Ship It!
- Madhan Neethiraj
On March 10, 2017, 4:09 a.m., Ashutosh Mestry wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/57495/
> ---
>
> (Updated March 10, 2017, 4:09 a.m.)
>
>
> Review request for atlas, Madhan Neethiraj and Sarath Subramanian.
>
>
> Bugs: ATLAS-1503
> https://issues.apache.org/jira/browse/ATLAS-1503
>
>
> Repository: atlas
>
>
> Description
> ---
>
> **Background**
> Existing implementation of Export REST API uses *ByteArrayOutputStream* to
> during output zip file creation. This puts pressure on memory when handling
> large data. Also, the data transfer does not start until entire export is
> done. This situation is less than ideal for performance.
>
> **Solution**
> - Passing *ServletOutputStream* to *ZipSink*.
> - This improves memory usage as memory does not get held up by
> *ByteArrayOutputStream*.
> - Reduces additional copy from *ByteArrayOutputStream* to
> *ServletOutputSream*.
> - Simplifies *ZipSink*.
> - Clear internal data structures after operation completion.
> - This aids, though not much, when freeing up memory used. There is some
> improvement in large transfers.
> - *ExportService.ExportContext.guidsToProcess* removed sequential lookup from
> *List* to *Set*.
> - Data transfer from server to client starts much sooner. Client is able to
> interrupt the progress if needed.
>
>
> Diffs
> -
>
> intg/src/main/java/org/apache/atlas/model/impexp/AtlasExportResult.java
> e6a967e
> webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java
> 31a4cf9
> webapp/src/main/java/org/apache/atlas/web/resources/ExportService.java
> c1891e0
> webapp/src/main/java/org/apache/atlas/web/resources/ZipSink.java 2e4cb01
>
>
> Diff: https://reviews.apache.org/r/57495/diff/2/
>
>
> Testing
> ---
>
> Profiled using *jmap* & *Eclipse MAT*, verified using *YourKit*.
>
> Verified: *FetchTypes* viz. *full* and *connected*.
>
> Memory usage: Stays constant on prolonged use. Verified ~3 hrs of continuous
> runs using medium and large database exports.
>
> Performance improvement:
> Date | File Size | No. of Entities | Duration (in mins)|
> -|---|-|---|
> 3/02 | 180 MB | 202930 |29 mins|
> 3/08 | 180 MB | 202930 |22 mins|
> 3/09 | 180 MB | 202930 |19 mins|
>
> About 15% improvement with list & set combined data structures.
> About 30% improvement by eliminating use of *ByteArrayOutputStream*.
>
>
> Thanks,
>
> Ashutosh Mestry
>
>
