-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70463/#review214622
-----------------------------------------------------------




graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java
Lines 28 (patched)
<https://reviews.apache.org/r/70463/#comment300798>

    this 'import' is the only change in the file. Please revert.



graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/DataPatchGraphDBHandler.java
Lines 27 (patched)
<https://reviews.apache.org/r/70463/#comment300799>

    Its not clear what 'apply()' would do with 'typeNameAttributeMap'. Consider 
a better name that coveys the operation performed.



graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/VertexIterator.java
Lines 42 (patched)
<https://reviews.apache.org/r/70463/#comment300800>

    next() method is creating a vertex?
    
    This class, 'VertexIterator', seems to be to covert Gremlin 'Vertext' to 
'AtlasVertex'. Perhaps a better naming or abstraction?
    
    And this class doesn't seem to be used at all. If not needed, please remove.



repository/src/main/java/org/apache/atlas/repository/patches/DataPatchHandler.java
Lines 24 (patched)
<https://reviews.apache.org/r/70463/#comment300828>

    I think JavaPatchHandler is better, as the patch can be used to make 
changes to types and data as well. The name JavaPatch is to differentiate 
typedef-patching mechanism used during bootstrap.
    
    I suggest to revert this rename of classes.



repository/src/main/java/org/apache/atlas/repository/patches/DataPatchService.java
Lines 30 (patched)
<https://reviews.apache.org/r/70463/#comment300833>

    Use of @Component for DataPatchService seems unnecessary. This is needed 
only during startup; we should keep these references around for the life of the 
server. Please review and update.



repository/src/main/java/org/apache/atlas/repository/patches/UniqueAttributePatch.java
Lines 39 (patched)
<https://reviews.apache.org/r/70463/#comment300832>

    Use of @Component for patches seems unnecessary. These patch handlers are 
needed only during startup, that too only if the corresponding patch hasn't 
already been applied. Please review and revert.



repository/src/main/java/org/apache/atlas/repository/store/bootstrap/AtlasTypeDefStoreInitializer.java
Line 160 (original)
<https://reviews.apache.org/r/70463/#comment300831>

    Why was PatchContext abstraction removed?



repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasGraphUtilsV2.java
Lines 393 (patched)
<https://reviews.apache.org/r/70463/#comment300830>

    findAllByTypeAndUniquePropertyName => 
findActiveEntityVerticesByTypeAndPropertyName


- Madhan Neethiraj


On April 12, 2019, 5:43 a.m., Ashutosh Mestry wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/70463/
> -----------------------------------------------------------
> 
> (Updated April 12, 2019, 5:43 a.m.)
> 
> 
> Review request for atlas, Kapildeo Nayak, Madhan Neethiraj, Nikhil Bonte, 
> Nixon Rodrigues, and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3132
>     https://issues.apache.org/jira/browse/ATLAS-3132
> 
> 
> Repository: atlas
> 
> 
> Description
> -------
> 
> **Approach**
> - Refactored existing implementation for new design.
> - Renamed 'Java Patch Framework' to 'Data Patch Framework', rationale being 
> that this is essentially to modify structure of existing data.
> - New _DataPatchService_: Modified order in which services are called. 
> _DataPatchService_ will be called before other services are invoked, thereby 
> giving chance for it to complete before entertaining new data.
> - New _DataPatchRegistry_: Data access (CRUD) operation for data patches.
> - New _UniqueAttributePatchHandler_: Current implementation for adding the 
> new property to data vertices. Implemented rudimentary caching to precent 
> repetitive look-ups.
> - New REST Endpoint to query status of patches.
> 
> **Performance**
> Since the data patching operation is high-volume operation, it has been 
> treated with priority. 
> - New _NewPropertyDataHandler_ uses database in bulk loading mode for rapid 
> processing. This scales with resources. Additional properties:
> - _atlas.processing.batchSize_: Size of batch.
> - _atlas.processing.numWorkers_: Number of worker threads to be employed. 
> - Leverages existing PC framework.
> 
> Processing speed:
> - 300K vertices: ~5 mins (8 threads, batch size: 3000)
> - 4.2 M entities: ~45 mins (from: 2019-04-12 04:44:50 to 2019-04-12 
> 05:29:04), (4 threads, batch size: 300)
> 
> 
> Diffs
> -----
> 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 
> d282c9966 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/DataPatchGraphDBHandler.java
>  PRE-CREATION 
>   
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/VertexIterator.java
>  PRE-CREATION 
>   
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/patches/NewPropertyDataPatch.java
>  PRE-CREATION 
>   intg/src/main/java/org/apache/atlas/pc/WorkItemConsumer.java b7eb4d89c 
>   intg/src/main/java/org/apache/atlas/pc/WorkItemManager.java 0e7d3f22d 
>   notification/src/main/java/org/apache/atlas/kafka/EmbeddedKafkaServer.java 
> 32b597fb6 
>   notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 
> 1d0a2734b 
>   
> repository/src/main/java/org/apache/atlas/repository/patches/AtlasJavaPatchHandler.java
>  9153d497b 
>   
> repository/src/main/java/org/apache/atlas/repository/patches/DataPatchHandler.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/patches/DataPatchManager.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/patches/DataPatchRegistry.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/patches/DataPatchService.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/patches/PatchContext.java
>  a60422b80 
>   
> repository/src/main/java/org/apache/atlas/repository/patches/TypeNameAttributeCache.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/patches/UniqueAttributePatch.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/patches/UniqueAttributePatchHandler.java
>  f2238f1b0 
>   
> repository/src/main/java/org/apache/atlas/repository/store/bootstrap/AtlasTypeDefStoreInitializer.java
>  78f3faf99 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasGraphUtilsV2.java
>  80141b4f1 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java
>  5aa6c8f0e 
>   
> repository/src/test/java/org/apache/atlas/patches/DataPatchRegistryTest.java 
> PRE-CREATION 
>   
> webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java
>  ce2d76f11 
>   webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 
> c5ceb9d6d 
>   webapp/src/test/java/org/apache/atlas/web/resources/AdminResourceTest.java 
> 223a90a9c 
> 
> 
> Diff: https://reviews.apache.org/r/70463/diff/1/
> 
> 
> Testing
> -------
> 
> **Unit tests**
> Additional tests added.
> 
> **Volume tests**
> Verification with large datasets: 
> - 4M entities
> - 3.2M entities
> - 16K entities.
> 
> **Performance tests**
> CPU usage, memory usage and disk IO.
> 
> **Pre-commit build**
> https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1031/
> 
> 
> Thanks,
> 
> Ashutosh Mestry
> 
>

Reply via email to