> On Sept. 12, 2018, 5:19 a.m., Madhan Neethiraj wrote: > > graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/Titan0Graph.java > > Lines 244 (patched) > > <https://reviews.apache.org/r/68678/diff/3/?file=2088302#file2088302line244> > > > > What is the advantage of returning iterator from here, instead of > > List<AtlasVertex> (line #237)? Here is a reference to such usage: > > ClassificationSearchProcessor.filter().
Based on what I have read, iterator brings less data in-memory when compared to List. - Ashutosh ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/68678/#review208546 ----------------------------------------------------------- On Sept. 11, 2018, 8:07 p.m., Ashutosh Mestry wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/68678/ > ----------------------------------------------------------- > > (Updated Sept. 11, 2018, 8:07 p.m.) > > > Review request for atlas, Apoorv Naik, Madhan Neethiraj, Nixon Rodrigues, and > Sarath Subramanian. > > > Bugs: ATLAS-2864 > https://issues.apache.org/jira/browse/ATLAS-2864 > > > Repository: atlas > > > Description > ------- > > **Approach** > - New class _IncrementalExportEntityProvider_ encapsulates queries specific > to the requested database. > - Modified the deep traversal flow. > > > **CURL** > > Incremental with _changeMarker_ set to 0: > ``` > { > "itemsToExport": [ > { "typeName": "hive_db", "uniqueAttributes": { "qualifiedName": > "largedb@cl1" }} > ], > "options": { > "fetchType": "incremental", > "skipLineage": "true", > "replicatedTo": "clx", > "changeMarker": 0 > } > } > ``` > > Incremental with _changeMarker_ set to a value: > ``` > { > "itemsToExport": [ > { "typeName": "hive_db", "uniqueAttributes": { "qualifiedName": > "largedb@cl1" }} > ], > "options": { > "fetchType": "incremental", > "skipLineage": "true", > "replicatedTo": "clx", > "changeMarker": 1536048035652 > } > } > ``` > > > Diffs > ----- > > > graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java > dded76f84 > > graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/Titan0Graph.java > 2408287bf > > graphdb/titan1/src/main/java/org/apache/atlas/repository/graphdb/titan1/Titan1Graph.java > 622c90019 > intg/src/main/java/org/apache/atlas/model/impexp/AtlasExportRequest.java > 106a4a02a > > repository/src/main/java/org/apache/atlas/repository/impexp/ExportService.java > aded67caf > > repository/src/main/java/org/apache/atlas/repository/impexp/IncrementalExportEntityProvider.java > PRE-CREATION > repository/src/main/java/org/apache/atlas/repository/impexp/Iterators.java > PRE-CREATION > repository/src/main/java/org/apache/atlas/repository/util/UniqueList.java > 9148ce09a > > repository/src/test/java/org/apache/atlas/repository/impexp/IncrementalExportEntityProviderTest.java > PRE-CREATION > > repository/src/test/java/org/apache/atlas/repository/impexp/IteratorsTest.java > PRE-CREATION > > repository/src/test/resources/json/stocksDB-Entities/export-incremental.json > fdd3b018d > > > Diff: https://reviews.apache.org/r/68678/diff/3/ > > > Testing > ------- > > **Pre-commit build** > https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/658/ > > **Unit tests** > - Additional unit tests to verify this scenario. > > > **Volume test** > > Database: HiveDB, 50 tables, 50 views, 50 CTAS tables, each with 88 columns. > Full export is where _changeMarker_ is set to 0. > Increment export is where some modifications have been done to the database > and _changeMarker_ is set to some value. > > +-------------+-----------+ > | Operation | Duration | > +-------------+-----------+ > | Full Export | 37 secs| > +-------------+-----------+ > | Increment | 5 secs| > +-------------+-----------+ > | Increment | 10 secs| > | (25 count) | . | > +-------------+-----------+ > > > Thanks, > > Ashutosh Mestry > >
