----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/70463/#review214670 -----------------------------------------------------------
intg/src/main/java/org/apache/atlas/model/patches/AtlasPatch.java Line 53 (original), 53 (patched) <https://reviews.apache.org/r/70463/#comment300856> APPLIED_PARTIALLY is never used, consider removing it. repository/src/main/java/org/apache/atlas/repository/patches/DataPatchRegistry.java Lines 57 (patched) <https://reviews.apache.org/r/70463/#comment300865> patchNameStatusMap => patchRegistry repository/src/main/java/org/apache/atlas/repository/patches/DataPatchRegistry.java Lines 62 (patched) <https://reviews.apache.org/r/70463/#comment300857> graph is already initialized in #61, we don't need to pass it as parameter for method - getPatchNameStatusForAllRegistered() repository/src/main/java/org/apache/atlas/repository/patches/DataPatchRegistry.java Lines 65 (patched) <https://reviews.apache.org/r/70463/#comment300864> incomingId => patchId repository/src/main/java/org/apache/atlas/repository/patches/DataPatchRegistry.java Lines 136 (patched) <https://reviews.apache.org/r/70463/#comment300866> getPatchNameStatusForAllRegistered => initPatchRegistry() repository/src/main/java/org/apache/atlas/repository/patches/DataPatchRegistry.java Lines 138 (patched) <https://reviews.apache.org/r/70463/#comment300858> remove graph param, since its already initialized in constructor. repository/src/main/java/org/apache/atlas/repository/patches/DataPatchRegistry.java Lines 172 (patched) <https://reviews.apache.org/r/70463/#comment300867> consider changing WARN to INFO, when atlas starts for the first time getAllPatches() will be empty first time. repository/src/main/java/org/apache/atlas/repository/patches/NewPropertyDataPatch.java Lines 84 (patched) <https://reviews.apache.org/r/70463/#comment300871> move line 79 and 80 inside while loop: if (manager == null) { manager = new WorkItemManager<>(new ConsumerBuilder(graph), BATCH_SIZE, NUM_WORKERS); LOG.info("Processing: Started..."); } repository/src/main/java/org/apache/atlas/repository/patches/NewPropertyDataPatch.java Lines 206 (patched) <https://reviews.apache.org/r/70463/#comment300854> Do we need to change the logger to WARN instead of ERROR. Also consider adding propertyKey name in the error message. repository/src/main/java/org/apache/atlas/repository/patches/TypeNameAttributeCache.java Lines 28 (patched) <https://reviews.apache.org/r/70463/#comment300873> we don't need this caching of typeName and unique attributes map. AtlasTypeRegistry already has this information. In NewPropertyDataPatch.apply(), you can have a helper method to request for uniqueattributes given a typeName. repository/src/main/java/org/apache/atlas/repository/patches/UniqueAttributePatch.java Lines 45 (patched) <https://reviews.apache.org/r/70463/#comment300868> since this is very specific to UniqueAttributePatch consider renaming to UniqueAttributePatchImpl repository/src/main/java/org/apache/atlas/repository/patches/UniqueAttributePatch.java Lines 70 (patched) <https://reviews.apache.org/r/70463/#comment300853> atlasGraph and indexer are already initialized in the constructor #52, you don't need to pass these params in the method. Consider removing it. repository/src/main/java/org/apache/atlas/repository/patches/UniqueAttributePatch.java Lines 75 (patched) <https://reviews.apache.org/r/70463/#comment300870> registerUniqueAttributeForType => createIndexForUniqueAttributes() repository/src/main/java/org/apache/atlas/repository/store/bootstrap/AtlasTypeDefStoreInitializer.java Line 465 (original), 419 (patched) <https://reviews.apache.org/r/70463/#comment300859> initialize dataPatchRegistry in the constructor #102 and not in every call to applyTypePatches() repository/src/main/java/org/apache/atlas/repository/store/bootstrap/AtlasTypeDefStoreInitializer.java Line 537 (original), 482 (patched) <https://reviews.apache.org/r/70463/#comment300863> consider making register() take only patch argument => patch.setStatus(status); dataPatchRegistry.register(patch); repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasGraphUtilsV2.java Line 569 (original), 472 (patched) <https://reviews.apache.org/r/70463/#comment300855> this method is never used. consider removing it. - Sarath Subramanian On April 11, 2019, 10:43 p.m., Ashutosh Mestry wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/70463/ > ----------------------------------------------------------- > > (Updated April 11, 2019, 10:43 p.m.) > > > Review request for atlas, Kapildeo Nayak, Madhan Neethiraj, Nikhil Bonte, > Nixon Rodrigues, and Sarath Subramanian. > > > Bugs: ATLAS-3132 > https://issues.apache.org/jira/browse/ATLAS-3132 > > > Repository: atlas > > > Description > ------- > > **Approach** > - Refactored existing implementation for new design. > - Renamed 'Java Patch Framework' to 'Data Patch Framework', rationale being > that this is essentially to modify structure of existing data. > - New _DataPatchService_: Modified order in which services are called. > _DataPatchService_ will be called before other services are invoked, thereby > giving chance for it to complete before entertaining new data. > - New _DataPatchRegistry_: Data access (CRUD) operation for data patches. > - New _UniqueAttributePatchHandler_: Current implementation for adding the > new property to data vertices. Implemented rudimentary caching to precent > repetitive look-ups. > - New REST Endpoint to query status of patches. > - Duplicates entities are detected during the patch application process. (See > below.) > > **Performance** > Since the data patching operation is high-volume operation, it has been > treated with priority. > - New _NewPropertyDataHandler_ uses database in bulk loading mode for rapid > processing. This scales with resources. Additional properties: > - _atlas.processing.batchSize_: Size of batch. > - _atlas.processing.numWorkers_: Number of worker threads to be employed. > - Leverages existing PC framework. > > Processing speed: > - 300K vertices: ~5 mins (8 threads, batch size: 3000) > - 4.2 M entities: ~45 mins (from: 2019-04-12 04:44:50 to 2019-04-12 > 05:29:04), (4 threads, batch size: 300) > > **Duplicates Detection** > Once the patch is run, user can do a fgrep on the application.log and get a > dump of all the duplicates detected in the process: > _fgrep "Duplicates detected" /var/log/atlas/application.log_ > > **Memory & CPU** > Higher the memory, more the threads that can be spawned. > > > Diffs > ----- > > intg/src/main/java/org/apache/atlas/model/patches/AtlasPatch.java ae0255f9b > intg/src/main/java/org/apache/atlas/pc/WorkItemConsumer.java b7eb4d89c > intg/src/main/java/org/apache/atlas/pc/WorkItemManager.java 0e7d3f22d > notification/src/main/java/org/apache/atlas/kafka/EmbeddedKafkaServer.java > 32b597fb6 > notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java > 1d0a2734b > > repository/src/main/java/org/apache/atlas/repository/patches/AtlasJavaPatchHandler.java > 9153d497b > > repository/src/main/java/org/apache/atlas/repository/patches/DataPatchHandler.java > PRE-CREATION > > repository/src/main/java/org/apache/atlas/repository/patches/DataPatchManager.java > PRE-CREATION > > repository/src/main/java/org/apache/atlas/repository/patches/DataPatchRegistry.java > PRE-CREATION > > repository/src/main/java/org/apache/atlas/repository/patches/DataPatchService.java > PRE-CREATION > > repository/src/main/java/org/apache/atlas/repository/patches/NewPropertyDataPatch.java > PRE-CREATION > > repository/src/main/java/org/apache/atlas/repository/patches/PatchContext.java > a60422b80 > > repository/src/main/java/org/apache/atlas/repository/patches/TypeNameAttributeCache.java > PRE-CREATION > > repository/src/main/java/org/apache/atlas/repository/patches/UniqueAttributePatch.java > PRE-CREATION > > repository/src/main/java/org/apache/atlas/repository/patches/UniqueAttributePatchHandler.java > f2238f1b0 > > repository/src/main/java/org/apache/atlas/repository/store/bootstrap/AtlasTypeDefStoreInitializer.java > 78f3faf99 > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasGraphUtilsV2.java > 80141b4f1 > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java > 5aa6c8f0e > > repository/src/test/java/org/apache/atlas/patches/DataPatchRegistryTest.java > PRE-CREATION > > webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java > ce2d76f11 > webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java > c5ceb9d6d > webapp/src/test/java/org/apache/atlas/web/resources/AdminResourceTest.java > 223a90a9c > > > Diff: https://reviews.apache.org/r/70463/diff/2/ > > > Testing > ------- > > **Unit tests** > Additional tests added. > > **Volume tests** > Verification with large datasets: > - 4M entities > - 3.2M entities > - 16K entities. > > **Performance tests** > CPU usage, memory usage and disk IO. > > **Pre-commit build** > https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1031/ > > **Gremlin Queries for Verification** > Check entities that do not have the new attribute: > ``` > g.V().has('__typeName', > within('hive_db','hive_table','hive_column')).hasNot('Referenceable.__u_qualifiedName').valueMap('__guid') > ``` > > Drop entities with new attribute: > ``` > g.V().has('__typeName', > within('hive_db','hive_table','hive_column')).has('Referenceable.__u_qualifiedName').properties('Referenceable.__u_qualifiedName').drop() > ``` > > Re-run patch: > ``` > g.V().has('__patch.id', > 'JAVA_PATCH_0000_001').property('__patch.state','FAILED'); > ``` > > > Thanks, > > Ashutosh Mestry > >
