----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73012/ -----------------------------------------------------------
(Updated Jan. 20, 2021, 10:13 p.m.)
Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, Sarath
Subramanian, and Sidharth Mishra.
Changes
-------
Incorporated suggestions given by Madhan.
Bugs: Atlas-4019
https://issues.apache.org/jira/browse/Atlas-4019
Repository: atlas
Description
-------
For any entity operation after the entity is created - e.g. add a column,
update an attribute. - the entire updated entity is saved in Hbase audits (for
each operation). Only the delta between two entity states needs to be stored.
Update routine could be called for an individual entity or for a list of
entities. Update-request may contain a full entity object or partial
object(with only updated attributes or relations). There could be updates in
entity core attributes, relationship attributes, custom attributes, or
associated classifications. In all the cases today we store the entire updated
entity in Hbase audits. Improvement is required so that only the updated
information is stored in Hbase audits instead of the entire entity object.
Solution explanation
All sorts of update requests go through the createOrUpdate routine of
AtlasEntityStoreV2. In this routine we compare all entities submitted for
updates with their stored copy to make sure there are valid changes in the
submitted entities. If valid changes are not found we discard them from the
update entity list.
Solution for this task is to modify this routine in such a way that while
comparing the objects for difference we also copy the difference in dummy
differential entity for each submitted updated-entity. The differential entity
will have only the information which has changed. Changed information could be
regular attributes, essential attributes, relationship attributes, custom
attributes, or classifications.
Today audit information is created using the updated entity which has all the
entity information new and old. For this task, audits are generated from the
differential entities and hence it has only the information which got updated.
Functionality is behind the application flag. Inorder to enable it set
atlas.update.audits.regular=false
Diffs (updated)
-----
intg/src/main/java/org/apache/atlas/AtlasConfiguration.java ea9f26d47
intg/src/test/java/org/apache/atlas/TestUtilsV2.java 2b9cf6e7f
repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditListenerV2.java
0043f1ce4
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityComparator.java
PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java
244072289
repository/src/test/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityComparatorTest.java
PRE-CREATION
repository/src/test/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2Test.java
c9f491296
server-api/src/main/java/org/apache/atlas/RequestContext.java 216ba08d3
Diff: https://reviews.apache.org/r/73012/diff/5/
Changes: https://reviews.apache.org/r/73012/diff/4-5/
Testing
-------
Performance impact Summary
Each number in the following tables is an average time in millisecond of 50
events of the same type. For example, the first row in the following table
shows that the average event processing time of 50 create-events was 1862
milliseconds.
Base Line (No changes) ->
eventType eventTime preCreateOrUpdate
checkForUnchanged createOrUpdate graphCommit
----- -----
----- ----- -----
create 1862 333
0 905 947
create 1978 338
0 984 986
create 1930 316
0 960 964
create 1938 350
0 977 955
create 1932 319
0 944 980
----- -----
----- ----- -----
UpdateComment 295 85
51 157 132
UpdateComment 338 128
54 196 136
UpdateComment 245 75
52 140 100
UpdateComment 245 90
49 152 88
UpdateComment 253 90
50 153 94
----- -----
----- ----- -----
UpdateColumns 762 132
49 374 381
UpdateColumns 818 146
63 425 388
UpdateColumns 768 133
52 366 395
UpdateColumns 772 149
50 392 374
UpdateColumns 764 136
49 374 384
----- -----
----- ----- -----
With changes Without flag -> No performance impact if compared to
above unmbers.
eventType eventTime preCreateOrUpdate
checkForUnchanged createOrUpdate graphCommit
----- -----
----- ----- -----
create 1866 349
0 911 944
create 1906 346
0 955 942
create 1896 340
0 943 944
create 1915 321
0 931 976
create 1920 314
0 927 985
----- -----
----- ----- -----
UpdateComment 265 90
63 169 89
UpdateComment 250 91
53 158 85
UpdateComment 229 83
51 148 76
UpdateComment 241 87
48 149 86
UpdateComment 239 78
48 139 95
----- -----
----- ----- -----
UpdateColumns 767 133
50 384 376
UpdateColumns 728 145
48 376 345
UpdateColumns 745 143
56 384 354
UpdateColumns 740 134
56 384 350
UpdateColumns 771 135
47 365 401
----- -----
----- ----- -----
With changes With flag(atlas.update.audits.regular=false) ->
eventType eventTime preCreateOrUpdate
checkForUnchanged createOrUpdate graphCommit
----- -----
----- ----- -----
create 1856 330
0 884 960
create 1901 327
0 935 958
create 1877 325
0 937 933
create 1913 309
0 931 975
create 1915 316
0 932 976
----- -----
----- ----- -----
UpdateComment 277 81
65 156 115
UpdateComment 250 82
63 154 90
UpdateComment 281 89
54 152 124
UpdateComment 241 74
59 143 93
UpdateComment 255 77
62 148 102
----- -----
----- ----- -----
UpdateColumns 778 147
64 393 378
UpdateColumns 734 128
62 363 364
UpdateColumns 741 139
63 383 352
UpdateColumns 711 130
59 364 342
UpdateColumns 753 122
58 355 392
----- -----
----- ----- -----
Thanks,
Deep Singh
