-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/75223/
-----------------------------------------------------------
(Updated Jan. 8, 2025, 11:08 a.m.)
Review request for atlas, Jayendra Parab, Mandar Ambawane, and Pinal Shah.
Bugs: ATLAS-4907
https://issues.apache.org/jira/browse/ATLAS-4907
Repository: atlas
Description
-------
Whenever Migration starts and vertex is created for the respective file in
progress
whenever Migration status is attempted to update it fails with nullpointer
exception
Due to this because the position was not updated in db hence when migration was
interrupted and restarted it would start from position 0
2024-09-18 10:29:04,153 WARN - [zipFileBasedMigrationImporter:]
~ Error updating status. Please rely on log messages.
(DataMigrationStatusService$MigrationStatusVertexManagement:202)
java.lang.NullPointerException
at
org.apache.atlas.repository.store.graph.v2.AtlasGraphUtilsV2.setProperty(AtlasGraphUtilsV2.java:250)
at
org.apache.atlas.repository.store.graph.v2.AtlasGraphUtilsV2.setEncodedProperty(AtlasGraphUtilsV2.java:238)
at
org.apache.atlas.repository.migration.DataMigrationStatusService$MigrationStatusVertexManagement.updateVertexPartialStatus(DataMigrationStatusService.java:200)
at
org.apache.atlas.repository.migration.DataMigrationStatusService.setStatus(DataMigrationStatusService.java:119)
at
org.apache.atlas.repository.store.graph.v2.bulkimport.pc.EntityCreationManager.read(EntityCreationManager.java:57)
at
org.apache.atlas.repository.store.graph.v2.bulkimport.MigrationImport.run(MigrationImport.java:79)
at
org.apache.atlas.repository.store.graph.v2.BulkImporterImpl.bulkImport(BulkImporterImpl.java:77)
at
org.apache.atlas.repository.impexp.ImportService.processEntities(ImportService.java:232)
at
org.apache.atlas.repository.impexp.ImportService.run(ImportService.java:119)
at
org.apache.atlas.repository.impexp.ImportService.run(ImportService.java:96)
at
org.apache.atlas.repository.migration.ZipFileMigrationImporter.performImport(ZipFileMigrationImporter.java:255)
at
org.apache.atlas.repository.migration.ZipFileMigrationImporter.run(ZipFileMigrationImporter.java:97)
at java.lang.Thread.run(Thread.java:748)
With this patch adds file hash instead of filename whenever it will try to
update vertex with status and position it will use file hash
Diffs (updated)
-----
repository/src/main/java/org/apache/atlas/repository/migration/DataMigrationStatusService.java
5b22f9cd5
Diff: https://reviews.apache.org/r/75223/diff/3/
Changes: https://reviews.apache.org/r/75223/diff/2-3/
Testing
-------
Before patch even after migration was completed position was never updated due
to above nullpointer exception
gremlin> g.V().has("__guid","b4fa0dcf2b0acac65e75dfda4f6dedd9").valueMap()
==>[__guid:[b4fa0dcf2b0acac65e75dfda4f6dedd9],__migration.startTime:[1726486589197],__migration.size:[1],__migration.position:[0],__migration.status:[DONE]]
2024-10-02 14:16:53,204 INFO - [zipFileBasedMigrationImporter:] ~ Migration
Import: Size: 0: Done! (MigrationImport:87)
After applying patch when interrupted in between it fetches last position of
entities processed:
==>[__guid:[b4fa0dcf2b0acac65e75dfda4f6dedd9],__migration.startTime:[1727887269568],__migration.size:[1],__migration.position:[6],__migration.status:[IN_PROGRESS]]
After completing migration:
gremlin> g.V().has("__guid","b4fa0dcf2b0acac65e75dfda4f6dedd9").valueMap()
==>[__guid:[b4fa0dcf2b0acac65e75dfda4f6dedd9],__migration.startTime:[1727950058639],__migration.size:[1],__migration.position:[3553],__migration.status:[DONE]]
It fetches correct position
2024-10-03 15:44:08,163 INFO - [zipFileBasedMigrationImporter:] ~ Migration
Import: Size: 3555: Done! (MigrationImport:88)
Thanks,
chaitali
