-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70463/#review214670
-----------------------------------------------------------




intg/src/main/java/org/apache/atlas/model/patches/AtlasPatch.java
Line 53 (original), 53 (patched)
<https://reviews.apache.org/r/70463/#comment300856>

    APPLIED_PARTIALLY is never used, consider removing it.



repository/src/main/java/org/apache/atlas/repository/patches/DataPatchRegistry.java
Lines 57 (patched)
<https://reviews.apache.org/r/70463/#comment300865>

    patchNameStatusMap => patchRegistry



repository/src/main/java/org/apache/atlas/repository/patches/DataPatchRegistry.java
Lines 62 (patched)
<https://reviews.apache.org/r/70463/#comment300857>

    graph is already initialized in #61, we don't need to pass it as parameter 
for method - getPatchNameStatusForAllRegistered()



repository/src/main/java/org/apache/atlas/repository/patches/DataPatchRegistry.java
Lines 65 (patched)
<https://reviews.apache.org/r/70463/#comment300864>

    incomingId => patchId



repository/src/main/java/org/apache/atlas/repository/patches/DataPatchRegistry.java
Lines 136 (patched)
<https://reviews.apache.org/r/70463/#comment300866>

    getPatchNameStatusForAllRegistered => initPatchRegistry()



repository/src/main/java/org/apache/atlas/repository/patches/DataPatchRegistry.java
Lines 138 (patched)
<https://reviews.apache.org/r/70463/#comment300858>

    remove graph param, since its already initialized in constructor.



repository/src/main/java/org/apache/atlas/repository/patches/DataPatchRegistry.java
Lines 172 (patched)
<https://reviews.apache.org/r/70463/#comment300867>

    consider changing WARN to INFO, when atlas starts for the first time 
getAllPatches() will be empty first time.



repository/src/main/java/org/apache/atlas/repository/patches/NewPropertyDataPatch.java
Lines 84 (patched)
<https://reviews.apache.org/r/70463/#comment300871>

    move line 79 and 80 inside while loop:
    
    if (manager == null) {
      manager = new WorkItemManager<>(new ConsumerBuilder(graph), BATCH_SIZE, 
NUM_WORKERS);
      LOG.info("Processing: Started...");
    }



repository/src/main/java/org/apache/atlas/repository/patches/NewPropertyDataPatch.java
Lines 206 (patched)
<https://reviews.apache.org/r/70463/#comment300854>

    Do we need to change the logger to WARN instead of ERROR. Also consider 
adding propertyKey name in the error message.



repository/src/main/java/org/apache/atlas/repository/patches/TypeNameAttributeCache.java
Lines 28 (patched)
<https://reviews.apache.org/r/70463/#comment300873>

    we don't need this caching of typeName and unique attributes map. 
AtlasTypeRegistry already has this information.
    
    In NewPropertyDataPatch.apply(), you can have a helper method to request 
for uniqueattributes given a typeName.



repository/src/main/java/org/apache/atlas/repository/patches/UniqueAttributePatch.java
Lines 45 (patched)
<https://reviews.apache.org/r/70463/#comment300868>

    since this is very specific to UniqueAttributePatch consider renaming to 
UniqueAttributePatchImpl



repository/src/main/java/org/apache/atlas/repository/patches/UniqueAttributePatch.java
Lines 70 (patched)
<https://reviews.apache.org/r/70463/#comment300853>

    atlasGraph and indexer are already initialized in the constructor #52, you 
don't need to pass these params in the method. Consider removing it.



repository/src/main/java/org/apache/atlas/repository/patches/UniqueAttributePatch.java
Lines 75 (patched)
<https://reviews.apache.org/r/70463/#comment300870>

    registerUniqueAttributeForType => createIndexForUniqueAttributes()



repository/src/main/java/org/apache/atlas/repository/store/bootstrap/AtlasTypeDefStoreInitializer.java
Line 465 (original), 419 (patched)
<https://reviews.apache.org/r/70463/#comment300859>

    initialize dataPatchRegistry in the constructor #102 and not in every call 
to applyTypePatches()



repository/src/main/java/org/apache/atlas/repository/store/bootstrap/AtlasTypeDefStoreInitializer.java
Line 537 (original), 482 (patched)
<https://reviews.apache.org/r/70463/#comment300863>

    consider making register() take only patch argument => 
    patch.setStatus(status);
    
    dataPatchRegistry.register(patch);



repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasGraphUtilsV2.java
Line 569 (original), 472 (patched)
<https://reviews.apache.org/r/70463/#comment300855>

    this method is never used. consider removing it.


- Sarath Subramanian


On April 11, 2019, 10:43 p.m., Ashutosh Mestry wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/70463/
> -----------------------------------------------------------
> 
> (Updated April 11, 2019, 10:43 p.m.)
> 
> 
> Review request for atlas, Kapildeo Nayak, Madhan Neethiraj, Nikhil Bonte, 
> Nixon Rodrigues, and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3132
>     https://issues.apache.org/jira/browse/ATLAS-3132
> 
> 
> Repository: atlas
> 
> 
> Description
> -------
> 
> **Approach**
> - Refactored existing implementation for new design.
> - Renamed 'Java Patch Framework' to 'Data Patch Framework', rationale being 
> that this is essentially to modify structure of existing data.
> - New _DataPatchService_: Modified order in which services are called. 
> _DataPatchService_ will be called before other services are invoked, thereby 
> giving chance for it to complete before entertaining new data.
> - New _DataPatchRegistry_: Data access (CRUD) operation for data patches.
> - New _UniqueAttributePatchHandler_: Current implementation for adding the 
> new property to data vertices. Implemented rudimentary caching to precent 
> repetitive look-ups.
> - New REST Endpoint to query status of patches.
> - Duplicates entities are detected during the patch application process. (See 
> below.)
> 
> **Performance**
> Since the data patching operation is high-volume operation, it has been 
> treated with priority. 
> - New _NewPropertyDataHandler_ uses database in bulk loading mode for rapid 
> processing. This scales with resources. Additional properties:
> - _atlas.processing.batchSize_: Size of batch.
> - _atlas.processing.numWorkers_: Number of worker threads to be employed. 
> - Leverages existing PC framework.
> 
> Processing speed:
> - 300K vertices: ~5 mins (8 threads, batch size: 3000)
> - 4.2 M entities: ~45 mins (from: 2019-04-12 04:44:50 to 2019-04-12 
> 05:29:04), (4 threads, batch size: 300)
> 
> **Duplicates Detection**
> Once the patch is run, user can do a fgrep on the application.log and get a 
> dump of all the duplicates detected in the process:
> _fgrep "Duplicates detected" /var/log/atlas/application.log_
> 
> **Memory & CPU**
> Higher the memory, more the threads that can be spawned.
> 
> 
> Diffs
> -----
> 
>   intg/src/main/java/org/apache/atlas/model/patches/AtlasPatch.java ae0255f9b 
>   intg/src/main/java/org/apache/atlas/pc/WorkItemConsumer.java b7eb4d89c 
>   intg/src/main/java/org/apache/atlas/pc/WorkItemManager.java 0e7d3f22d 
>   notification/src/main/java/org/apache/atlas/kafka/EmbeddedKafkaServer.java 
> 32b597fb6 
>   notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 
> 1d0a2734b 
>   
> repository/src/main/java/org/apache/atlas/repository/patches/AtlasJavaPatchHandler.java
>  9153d497b 
>   
> repository/src/main/java/org/apache/atlas/repository/patches/DataPatchHandler.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/patches/DataPatchManager.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/patches/DataPatchRegistry.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/patches/DataPatchService.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/patches/NewPropertyDataPatch.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/patches/PatchContext.java
>  a60422b80 
>   
> repository/src/main/java/org/apache/atlas/repository/patches/TypeNameAttributeCache.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/patches/UniqueAttributePatch.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/patches/UniqueAttributePatchHandler.java
>  f2238f1b0 
>   
> repository/src/main/java/org/apache/atlas/repository/store/bootstrap/AtlasTypeDefStoreInitializer.java
>  78f3faf99 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasGraphUtilsV2.java
>  80141b4f1 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java
>  5aa6c8f0e 
>   
> repository/src/test/java/org/apache/atlas/patches/DataPatchRegistryTest.java 
> PRE-CREATION 
>   
> webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java
>  ce2d76f11 
>   webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 
> c5ceb9d6d 
>   webapp/src/test/java/org/apache/atlas/web/resources/AdminResourceTest.java 
> 223a90a9c 
> 
> 
> Diff: https://reviews.apache.org/r/70463/diff/2/
> 
> 
> Testing
> -------
> 
> **Unit tests**
> Additional tests added.
> 
> **Volume tests**
> Verification with large datasets: 
> - 4M entities
> - 3.2M entities
> - 16K entities.
> 
> **Performance tests**
> CPU usage, memory usage and disk IO.
> 
> **Pre-commit build**
> https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1031/
> 
> **Gremlin Queries for Verification**
> Check entities that do not have the new attribute:
> ```
> g.V().has('__typeName', 
> within('hive_db','hive_table','hive_column')).hasNot('Referenceable.__u_qualifiedName').valueMap('__guid')
> ```
> 
> Drop entities with new attribute:
> ```
> g.V().has('__typeName', 
> within('hive_db','hive_table','hive_column')).has('Referenceable.__u_qualifiedName').properties('Referenceable.__u_qualifiedName').drop()
> ```
> 
> Re-run patch:
> ```
> g.V().has('__patch.id', 
> 'JAVA_PATCH_0000_001').property('__patch.state','FAILED');
> ```
> 
> 
> Thanks,
> 
> Ashutosh Mestry
> 
>

Reply via email to