-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68678/
-----------------------------------------------------------
Review request for atlas, Madhan Neethiraj and Nixon Rodrigues.
Bugs: ATLAS-2864
https://issues.apache.org/jira/browse/ATLAS-2864
Repository: atlas
Description
-------
**Approach**
- New class _IncrementalExportEntityProvider_ encapsulates queries specific to
the requested database.
- Modified the deep traversal flow.
**CURL**
Incremental with _changeMarker_ set to 0:
```
{
"itemsToExport": [
{ "typeName": "hive_db", "uniqueAttributes": { "qualifiedName":
"largedb@cl1" }}
],
"options": {
"fetchType": "incremental",
"skipLineage": "true",
"replicatedTo": "clx",
"changeMarker": 0
}
}
```
Incremental with _changeMarker_ set to a value:
```
{
"itemsToExport": [
{ "typeName": "hive_db", "uniqueAttributes": { "qualifiedName":
"largedb@cl1" }}
],
"options": {
"fetchType": "incremental",
"skipLineage": "true",
"replicatedTo": "clx",
"changeMarker": 1536048035652
}
}
```
Diffs
-----
intg/src/main/java/org/apache/atlas/model/impexp/AtlasExportRequest.java
106a4a02a
repository/src/main/java/org/apache/atlas/repository/impexp/ExportService.java
aded67caf
repository/src/main/java/org/apache/atlas/repository/impexp/IncrementalExportEntityProvider.java
PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/util/UniqueList.java
9148ce09a
repository/src/test/java/org/apache/atlas/repository/impexp/ExportIncrementalQueryTest.java
PRE-CREATION
Diff: https://reviews.apache.org/r/68678/diff/1/
Testing
-------
**Unit tests**
- Additional unit tests to verify this scenario.
**Volume test**
Database: HiveDB, 50 tables, 50 views, 50 CTAS tables, each with 88 columns.
Full export is where _changeMarker_ is set to 0.
Increment export is where some modifications have been done to the database and
_changeMarker_ is set to some value.
+-------------+-----------+
| Operation | Duration |
+-------------+-----------+
| Full Export | 40 secs|
+-------------+-----------+
| Increment | 5 secs|
+-------------+-----------+
Thanks,
Ashutosh Mestry
