----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/68678/#review208499 -----------------------------------------------------------
repository/src/main/java/org/apache/atlas/repository/impexp/IncrementalExportEntityProvider.java Lines 94 (patched) <https://reviews.apache.org/r/68678/#comment292465> Perhaps it will be efficient to retrieve 'vertices' from the query, instead of creating a HashMap for each guid. Also, the vertices retrieved here can help avoid having to retrieve vertex for each guid returned from here. - Madhan Neethiraj On Sept. 10, 2018, 8:55 p.m., Ashutosh Mestry wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/68678/ > ----------------------------------------------------------- > > (Updated Sept. 10, 2018, 8:55 p.m.) > > > Review request for atlas, Madhan Neethiraj and Nixon Rodrigues. > > > Bugs: ATLAS-2864 > https://issues.apache.org/jira/browse/ATLAS-2864 > > > Repository: atlas > > > Description > ------- > > **Approach** > - New class _IncrementalExportEntityProvider_ encapsulates queries specific > to the requested database. > - Modified the deep traversal flow. > > > **CURL** > > Incremental with _changeMarker_ set to 0: > ``` > { > "itemsToExport": [ > { "typeName": "hive_db", "uniqueAttributes": { "qualifiedName": > "largedb@cl1" }} > ], > "options": { > "fetchType": "incremental", > "skipLineage": "true", > "replicatedTo": "clx", > "changeMarker": 0 > } > } > ``` > > Incremental with _changeMarker_ set to a value: > ``` > { > "itemsToExport": [ > { "typeName": "hive_db", "uniqueAttributes": { "qualifiedName": > "largedb@cl1" }} > ], > "options": { > "fetchType": "incremental", > "skipLineage": "true", > "replicatedTo": "clx", > "changeMarker": 1536048035652 > } > } > ``` > > > Diffs > ----- > > intg/src/main/java/org/apache/atlas/model/impexp/AtlasExportRequest.java > 106a4a02a > > repository/src/main/java/org/apache/atlas/repository/impexp/ExportService.java > aded67caf > > repository/src/main/java/org/apache/atlas/repository/impexp/IncrementalExportEntityProvider.java > PRE-CREATION > repository/src/main/java/org/apache/atlas/repository/util/UniqueList.java > 9148ce09a > > repository/src/test/java/org/apache/atlas/repository/impexp/ExportIncrementalQueryTest.java > PRE-CREATION > > > Diff: https://reviews.apache.org/r/68678/diff/1/ > > > Testing > ------- > > **Unit tests** > - Additional unit tests to verify this scenario. > > > **Volume test** > > Database: HiveDB, 50 tables, 50 views, 50 CTAS tables, each with 88 columns. > Full export is where _changeMarker_ is set to 0. > Increment export is where some modifications have been done to the database > and _changeMarker_ is set to some value. > > +-------------+-----------+ > | Operation | Duration | > +-------------+-----------+ > | Full Export | 40 secs| > +-------------+-----------+ > | Increment | 5 secs| > +-------------+-----------+ > > > Thanks, > > Ashutosh Mestry > >
