-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68678/
-----------------------------------------------------------
(Updated Sept. 11, 2018, 8:07 p.m.)
Review request for atlas, Madhan Neethiraj and Nixon Rodrigues.
Changes
-------
Updates include: Additional optimization to fetch mechanism.
Bugs: ATLAS-2864
https://issues.apache.org/jira/browse/ATLAS-2864
Repository: atlas
Description
-------
**Approach**
- New class _IncrementalExportEntityProvider_ encapsulates queries specific to
the requested database.
- Modified the deep traversal flow.
**CURL**
Incremental with _changeMarker_ set to 0:
```
{
"itemsToExport": [
{ "typeName": "hive_db", "uniqueAttributes": { "qualifiedName":
"largedb@cl1" }}
],
"options": {
"fetchType": "incremental",
"skipLineage": "true",
"replicatedTo": "clx",
"changeMarker": 0
}
}
```
Incremental with _changeMarker_ set to a value:
```
{
"itemsToExport": [
{ "typeName": "hive_db", "uniqueAttributes": { "qualifiedName":
"largedb@cl1" }}
],
"options": {
"fetchType": "incremental",
"skipLineage": "true",
"replicatedTo": "clx",
"changeMarker": 1536048035652
}
}
```
Diffs (updated)
-----
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java
dded76f84
graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/Titan0Graph.java
2408287bf
graphdb/titan1/src/main/java/org/apache/atlas/repository/graphdb/titan1/Titan1Graph.java
622c90019
intg/src/main/java/org/apache/atlas/model/impexp/AtlasExportRequest.java
106a4a02a
repository/src/main/java/org/apache/atlas/repository/impexp/ExportService.java
aded67caf
repository/src/main/java/org/apache/atlas/repository/impexp/IncrementalExportEntityProvider.java
PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/impexp/Iterators.java
PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/util/UniqueList.java
9148ce09a
repository/src/test/java/org/apache/atlas/repository/impexp/IncrementalExportEntityProviderTest.java
PRE-CREATION
repository/src/test/java/org/apache/atlas/repository/impexp/IteratorsTest.java
PRE-CREATION
repository/src/test/resources/json/stocksDB-Entities/export-incremental.json
fdd3b018d
Diff: https://reviews.apache.org/r/68678/diff/3/
Changes: https://reviews.apache.org/r/68678/diff/2-3/
Testing (updated)
-------
**Unit tests**
- Additional unit tests to verify this scenario.
**Volume test**
Database: HiveDB, 50 tables, 50 views, 50 CTAS tables, each with 88 columns.
Full export is where _changeMarker_ is set to 0.
Increment export is where some modifications have been done to the database and
_changeMarker_ is set to some value.
+-------------+-----------+
| Operation | Duration |
+-------------+-----------+
| Full Export | 37 secs|
+-------------+-----------+
| Increment | 5 secs|
+-------------+-----------+
| Increment | 10 secs|
| (25 count) | . |
+-------------+-----------+
Thanks,
Ashutosh Mestry
