-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68678/
-----------------------------------------------------------
(Updated Sept. 13, 2018, 4:52 a.m.)
Review request for atlas, Apoorv Naik, Madhan Neethiraj, Nixon Rodrigues, and
Sarath Subramanian.
Changes
-------
Updates include:
- Addressed review comments.
- Refactoring.
Bugs: ATLAS-2864
https://issues.apache.org/jira/browse/ATLAS-2864
Repository: atlas
Description
-------
**Approach**
- New class _IncrementalExportEntityProvider_ encapsulates queries specific to
the requested database.
- Modified the deep traversal flow.
**CURL**
Incremental with _changeMarker_ set to 0:
```
{
"itemsToExport": [
{ "typeName": "hive_db", "uniqueAttributes": { "qualifiedName":
"largedb@cl1" }}
],
"options": {
"fetchType": "incremental",
"skipLineage": "true",
"replicatedTo": "clx",
"changeMarker": 0
}
}
```
Incremental with _changeMarker_ set to a value:
```
{
"itemsToExport": [
{ "typeName": "hive_db", "uniqueAttributes": { "qualifiedName":
"largedb@cl1" }}
],
"options": {
"fetchType": "incremental",
"skipLineage": "true",
"replicatedTo": "clx",
"changeMarker": 1536048035652
}
}
```
Diffs (updated)
-----
intg/src/main/java/org/apache/atlas/model/impexp/AtlasExportRequest.java
106a4a02a
repository/src/main/java/org/apache/atlas/repository/impexp/ExportService.java
612549d64
repository/src/main/java/org/apache/atlas/repository/impexp/IncrementalExportEntityProvider.java
PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/util/UniqueList.java
9148ce09a
repository/src/test/java/org/apache/atlas/repository/impexp/IncrementalExportEntityProviderTest.java
PRE-CREATION
repository/src/test/resources/json/stocksDB-Entities/export-incremental.json
fdd3b018d
Diff: https://reviews.apache.org/r/68678/diff/4/
Changes: https://reviews.apache.org/r/68678/diff/3-4/
Testing
-------
**Pre-commit build**
https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/658/
**Unit tests**
- Additional unit tests to verify this scenario.
**Volume test**
Database: HiveDB, 50 tables, 50 views, 50 CTAS tables, each with 88 columns.
Full export is where _changeMarker_ is set to 0.
Increment export is where some modifications have been done to the database and
_changeMarker_ is set to some value.
+-------------+-----------+
| Operation | Duration |
+-------------+-----------+
| Full Export | 37 secs|
+-------------+-----------+
| Increment | 5 secs|
+-------------+-----------+
| Increment | 10 secs|
| (25 count) | . |
+-------------+-----------+
Thanks,
Ashutosh Mestry
