subject:"Re\: Review Request 66928\: Data Migration\: Import\: Infer Types that Store Edge Ids"

Re: Review Request 66928: Data Migration: Import: Infer Types that Store Edge Ids

2018-05-22 Thread Madhan Neethiraj


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66928/#review203643
---


Ship it!




Ship It!

- Madhan Neethiraj


On May 23, 2018, 1:30 a.m., Ashutosh Mestry wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66928/
> ---
> 
> (Updated May 23, 2018, 1:30 a.m.)
> 
> 
> Review request for atlas, Madhan Neethiraj, Ruchi Solani, and Sarath 
> Subramanian.
> 
> 
> Bugs: ATLAS-2637
> https://issues.apache.org/jira/browse/ATLAS-2637
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> **Approach**
> New class: _TypesStoringEdgeIds_: Navigates through all the types in the 
> _typeRegistry_, returns map of entity type and properties that store edge ids.
> Modified: _DataMigrationService_: Uses output from class above and passes it 
> down to migration.
> Modified: _PostProcessManager_: Uses the map generated above and uses it for 
> post processing.
> Modified: _GraphSONUtility_: Improvement to check for vertex of type. This 
> avoids potential exeception when a non-existent property is checked for 
> presence.
> 
> Added PostProcess framework.
> Added logic for handling new Array and Map representation.
> 
> 
> Diffs
> -
> 
>   common/src/main/java/org/apache/atlas/AtlasConstants.java 2b92e6ed6 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 
> 607baf664 
>   
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java
>  c0b9c1741 
>   
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphDatabase.java
>  16aecd5e2 
>   
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/AtlasGraphSONReader.java
>  ae119b0bc 
>   
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/ElementProcessors.java
>  PRE-CREATION 
>   
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtility.java
>  ec320b03e 
>   
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/JsonNodeProcessManager.java
>  e4b6ee2e9 
>   
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/MappedElementCache.java
>  cca72ad11 
>   
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessManager.java
>  d0a65f7b1 
>   
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/RelationshipCacheGenerator.java
>  PRE-CREATION 
>   
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/RelationshipTypeCache.java
>  e4e82649b 
>   
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/TypesWithCollectionsFinder.java
>  PRE-CREATION 
>   
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/postProcess/PostProcessListProperty.java
>  PRE-CREATION 
>   
> graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/BaseUtils.java
>  e863d9fae 
>   
> graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtilityPostProcessTest.java
>  4d73c78ef 
>   
> graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtilityTest.java
>  794b5471e 
>   
> graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/JsonNodeParsersTest.java
>  170b1af60 
>   
> graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/MappedElementCacheTest.java
>  cac09d229 
>   
> graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessListPropertyTest.java
>  PRE-CREATION 
>   graphdb/janus/src/test/resources/col-2-legacy.json PRE-CREATION 
>   graphdb/janus/src/test/resources/col-3-legacy.json PRE-CREATION 
>   graphdb/janus/src/test/resources/edge-legacy-col.json PRE-CREATION 
>   graphdb/janus/src/test/resources/edge-legacy-col2.json PRE-CREATION 
>   graphdb/janus/src/test/resources/edge-legacy-col3.json PRE-CREATION 
>   graphdb/janus/src/test/resources/edge-legacy-col4.json PRE-CREATION 
>   graphdb/janus/src/test/resources/edge-legacy-process.json PRE-CREATION 
>   graphdb/janus/src/test/resources/edge-legacy-tag.json PRE-CREATION 
>   graphdb/janus/src/test/resources/lineage-v-98312.json PRE-CREATION 
>   graphdb/janus/src/test/resources/table-v-147504.json 898dce5df 
>   graphdb/janus/src/test/resources/tag-163856752.json PRE-CREATION 
>   intg/src/main/java/org/apache/atlas/store/AtlasTypeDefStore.java b05754f4b 
>   intg/src/test/java/org/apache/atlas/TestUtilsV2.java 886ce77f5 
>   
> repository/src/main/java/org/apache/atlas/repository/migration/DataMigrationService.java
>  22cd55217

Re: Review Request 66928: Data Migration: Import: Infer Types that Store Edge Ids

2018-05-22 Thread Ashutosh Mestry


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66928/
---

(Updated May 23, 2018, 1:30 a.m.)


Review request for atlas, Madhan Neethiraj, Ruchi Solani, and Sarath 
Subramanian.


Changes
---

Updates include:
- Addressed exception 'IllegalStateException' during edge creation.
- Fixed bug: Where lineage did not have correct propagate direction.
- Clean up of unit tests.
- Minor updates to unit tests.


Bugs: ATLAS-2637
https://issues.apache.org/jira/browse/ATLAS-2637


Repository: atlas


Description
---

**Approach**
New class: _TypesStoringEdgeIds_: Navigates through all the types in the 
_typeRegistry_, returns map of entity type and properties that store edge ids.
Modified: _DataMigrationService_: Uses output from class above and passes it 
down to migration.
Modified: _PostProcessManager_: Uses the map generated above and uses it for 
post processing.
Modified: _GraphSONUtility_: Improvement to check for vertex of type. This 
avoids potential exeception when a non-existent property is checked for 
presence.

Added PostProcess framework.
Added logic for handling new Array and Map representation.


Diffs (updated)
-

  common/src/main/java/org/apache/atlas/AtlasConstants.java 2b92e6ed6 
  graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 
607baf664 
  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java
 c0b9c1741 
  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphDatabase.java
 16aecd5e2 
  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/AtlasGraphSONReader.java
 ae119b0bc 
  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/ElementProcessors.java
 PRE-CREATION 
  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtility.java
 ec320b03e 
  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/JsonNodeProcessManager.java
 e4b6ee2e9 
  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/MappedElementCache.java
 cca72ad11 
  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessManager.java
 d0a65f7b1 
  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/RelationshipCacheGenerator.java
 PRE-CREATION 
  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/RelationshipTypeCache.java
 e4e82649b 
  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/TypesWithCollectionsFinder.java
 PRE-CREATION 
  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/postProcess/PostProcessListProperty.java
 PRE-CREATION 
  
graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/BaseUtils.java
 e863d9fae 
  
graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtilityPostProcessTest.java
 4d73c78ef 
  
graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtilityTest.java
 794b5471e 
  
graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/JsonNodeParsersTest.java
 170b1af60 
  
graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/MappedElementCacheTest.java
 cac09d229 
  
graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessListPropertyTest.java
 PRE-CREATION 
  graphdb/janus/src/test/resources/col-2-legacy.json PRE-CREATION 
  graphdb/janus/src/test/resources/col-3-legacy.json PRE-CREATION 
  graphdb/janus/src/test/resources/edge-legacy-col.json PRE-CREATION 
  graphdb/janus/src/test/resources/edge-legacy-col2.json PRE-CREATION 
  graphdb/janus/src/test/resources/edge-legacy-col3.json PRE-CREATION 
  graphdb/janus/src/test/resources/edge-legacy-col4.json PRE-CREATION 
  graphdb/janus/src/test/resources/edge-legacy-process.json PRE-CREATION 
  graphdb/janus/src/test/resources/edge-legacy-tag.json PRE-CREATION 
  graphdb/janus/src/test/resources/lineage-v-98312.json PRE-CREATION 
  graphdb/janus/src/test/resources/table-v-147504.json 898dce5df 
  graphdb/janus/src/test/resources/tag-163856752.json PRE-CREATION 
  intg/src/main/java/org/apache/atlas/store/AtlasTypeDefStore.java b05754f4b 
  intg/src/test/java/org/apache/atlas/TestUtilsV2.java 886ce77f5 
  
repository/src/main/java/org/apache/atlas/repository/migration/DataMigrationService.java
 22cd55217 
  
repository/src/main/java/org/apache/atlas/repository/migration/RelationshipCacheGenerator.java
 2550dafc1 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasTypeDefGraphStoreV1.java
 3c84e3c22 
  
repository/src/test/java/org/apache/atlas/repository/migration/ComplexAttributesTest.java

Re: Review Request 66928: Data Migration: Import: Infer Types that Store Edge Ids

2018-05-22 Thread Ashutosh Mestry


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66928/
---

(Updated May 22, 2018, 10:03 p.m.)


Review request for atlas, Madhan Neethiraj, Ruchi Solani, and Sarath 
Subramanian.


Changes
---

Updates include: 
- Default propagation value now uses the relationship value.
- Unit test refactoring.


Bugs: ATLAS-2637
https://issues.apache.org/jira/browse/ATLAS-2637


Repository: atlas


Description
---

**Approach**
New class: _TypesStoringEdgeIds_: Navigates through all the types in the 
_typeRegistry_, returns map of entity type and properties that store edge ids.
Modified: _DataMigrationService_: Uses output from class above and passes it 
down to migration.
Modified: _PostProcessManager_: Uses the map generated above and uses it for 
post processing.
Modified: _GraphSONUtility_: Improvement to check for vertex of type. This 
avoids potential exeception when a non-existent property is checked for 
presence.

Added PostProcess framework.
Added logic for handling new Array and Map representation.


Diffs (updated)
-

  common/src/main/java/org/apache/atlas/AtlasConstants.java 2b92e6ed6 
  graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 
607baf664 
  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java
 c0b9c1741 
  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphDatabase.java
 16aecd5e2 
  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/AtlasGraphSONReader.java
 ae119b0bc 
  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/ElementProcessors.java
 PRE-CREATION 
  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtility.java
 ec320b03e 
  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/MappedElementCache.java
 cca72ad11 
  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessManager.java
 d0a65f7b1 
  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/RelationshipCacheGenerator.java
 PRE-CREATION 
  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/RelationshipTypeCache.java
 e4e82649b 
  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/TypesWithCollectionsFinder.java
 PRE-CREATION 
  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/postProcess/PostProcessListProperty.java
 PRE-CREATION 
  
graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/BaseUtils.java
 e863d9fae 
  
graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtilityPostProcessTest.java
 4d73c78ef 
  
graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtilityTest.java
 794b5471e 
  
graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/JsonNodeParsersTest.java
 170b1af60 
  
graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/MappedElementCacheTest.java
 cac09d229 
  
graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessListPropertyTest.java
 PRE-CREATION 
  graphdb/janus/src/test/resources/col-2-legacy.json PRE-CREATION 
  graphdb/janus/src/test/resources/col-3-legacy.json PRE-CREATION 
  graphdb/janus/src/test/resources/edge-legacy-col.json PRE-CREATION 
  graphdb/janus/src/test/resources/edge-legacy-col2.json PRE-CREATION 
  graphdb/janus/src/test/resources/edge-legacy-col3.json PRE-CREATION 
  graphdb/janus/src/test/resources/edge-legacy-col4.json PRE-CREATION 
  graphdb/janus/src/test/resources/edge-legacy-process.json PRE-CREATION 
  graphdb/janus/src/test/resources/edge-legacy-tag.json PRE-CREATION 
  graphdb/janus/src/test/resources/lineage-v-98312.json PRE-CREATION 
  graphdb/janus/src/test/resources/table-v-147504.json 898dce5df 
  graphdb/janus/src/test/resources/tag-163856752.json PRE-CREATION 
  intg/src/main/java/org/apache/atlas/store/AtlasTypeDefStore.java b05754f4b 
  intg/src/test/java/org/apache/atlas/TestUtilsV2.java 886ce77f5 
  
repository/src/main/java/org/apache/atlas/repository/migration/DataMigrationService.java
 22cd55217 
  
repository/src/main/java/org/apache/atlas/repository/migration/RelationshipCacheGenerator.java
 2550dafc1 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasTypeDefGraphStoreV1.java
 3c84e3c22 
  
repository/src/test/java/org/apache/atlas/repository/migration/ComplexAttributesTest.java
 PRE-CREATION 
  
repository/src/test/java/org/apache/atlas/repository/migration/HiveParititionTest.java
 ac0b79d38 
  
repository/src/test/java/org/apache/atlas/repository/migration/HiveStocksTest.java
 ffbf3200b 
  
repository/

Re: Review Request 66928: Data Migration: Import: Infer Types that Store Edge Ids

2018-05-21 Thread Ashutosh Mestry


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66928/
---

(Updated May 21, 2018, 11:49 p.m.)


Review request for atlas, Madhan Neethiraj, Ruchi Solani, and Sarath 
Subramanian.


Changes
---

Updates include:
- Default propagation set to ONE_TO_TWO for lineages edges.
- Refactoring to use _AtlasTypeRegistry_ instead of complex maps.
- Refactoring to move logic to _RelationshipTypeCache_ from _GraphSONUtility_.


Bugs: ATLAS-2637
https://issues.apache.org/jira/browse/ATLAS-2637


Repository: atlas


Description
---

**Approach**
New class: _TypesStoringEdgeIds_: Navigates through all the types in the 
_typeRegistry_, returns map of entity type and properties that store edge ids.
Modified: _DataMigrationService_: Uses output from class above and passes it 
down to migration.
Modified: _PostProcessManager_: Uses the map generated above and uses it for 
post processing.
Modified: _GraphSONUtility_: Improvement to check for vertex of type. This 
avoids potential exeception when a non-existent property is checked for 
presence.

Added PostProcess framework.
Added logic for handling new Array and Map representation.


Diffs (updated)
-

  common/src/main/java/org/apache/atlas/AtlasConstants.java 2b92e6ed6 
  graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 
607baf664 
  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java
 c0b9c1741 
  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphDatabase.java
 16aecd5e2 
  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/AtlasGraphSONReader.java
 ae119b0bc 
  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtility.java
 ec320b03e 
  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/MappedElementCache.java
 cca72ad11 
  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessManager.java
 d0a65f7b1 
  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/RelationshipTypeCache.java
 e4e82649b 
  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/TypesWithCollectionsFinder.java
 PRE-CREATION 
  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/postProcess/PostProcessListProperty.java
 PRE-CREATION 
  
graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/BaseUtils.java
 e863d9fae 
  
graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtilityPostProcessTest.java
 4d73c78ef 
  
graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtilityTest.java
 794b5471e 
  
graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/MappedElementCacheTest.java
 cac09d229 
  
graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessListPropertyTest.java
 PRE-CREATION 
  graphdb/janus/src/test/resources/col-2-legacy.json PRE-CREATION 
  graphdb/janus/src/test/resources/col-3-legacy.json PRE-CREATION 
  graphdb/janus/src/test/resources/edge-legacy-Process.json PRE-CREATION 
  graphdb/janus/src/test/resources/edge-legacy-col.json PRE-CREATION 
  graphdb/janus/src/test/resources/edge-legacy-col2.json PRE-CREATION 
  graphdb/janus/src/test/resources/edge-legacy-col3.json PRE-CREATION 
  graphdb/janus/src/test/resources/edge-legacy-col4.json PRE-CREATION 
  graphdb/janus/src/test/resources/edge-legacy-tag.json PRE-CREATION 
  graphdb/janus/src/test/resources/lineage-v-98312.json PRE-CREATION 
  graphdb/janus/src/test/resources/table-v-147504.json 898dce5df 
  graphdb/janus/src/test/resources/tag-163856752.json PRE-CREATION 
  intg/src/main/java/org/apache/atlas/store/AtlasTypeDefStore.java b05754f4b 
  intg/src/test/java/org/apache/atlas/TestUtilsV2.java 886ce77f5 
  
repository/src/main/java/org/apache/atlas/repository/migration/DataMigrationService.java
 22cd55217 
  
repository/src/main/java/org/apache/atlas/repository/migration/RelationshipCacheGenerator.java
 2550dafc1 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasTypeDefGraphStoreV1.java
 3c84e3c22 
  
repository/src/test/java/org/apache/atlas/repository/migration/ComplexAttributesTest.java
 PRE-CREATION 
  
repository/src/test/java/org/apache/atlas/repository/migration/HiveParititionTest.java
 ac0b79d38 
  
repository/src/test/java/org/apache/atlas/repository/migration/HiveStocksTest.java
 ffbf3200b 
  
repository/src/test/java/org/apache/atlas/repository/migration/MigrationBaseAsserts.java
 ec6e64a25 
  
repository/src/test/java/org/apache/atlas/repository/migration/RelationshipMappingTest.java
 48e539100 
  
repository/src/test/java/org/apache/atlas/repository/

Re: Review Request 66928: Data Migration: Import: Infer Types that Store Edge Ids

2018-05-20 Thread Ashutosh Mestry


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66928/
---

(Updated May 21, 2018, 2:15 a.m.)


Review request for atlas, Madhan Neethiraj, Ruchi Solani, and Sarath 
Subramanian.


Changes
---

Updates include: 
- Optimization for post processing.
- Refactoring of _RelationshipTypeCache_.
- Addressed review comments.


Bugs: ATLAS-2637
https://issues.apache.org/jira/browse/ATLAS-2637


Repository: atlas


Description
---

**Approach**
New class: _TypesStoringEdgeIds_: Navigates through all the types in the 
_typeRegistry_, returns map of entity type and properties that store edge ids.
Modified: _DataMigrationService_: Uses output from class above and passes it 
down to migration.
Modified: _PostProcessManager_: Uses the map generated above and uses it for 
post processing.
Modified: _GraphSONUtility_: Improvement to check for vertex of type. This 
avoids potential exeception when a non-existent property is checked for 
presence.

Added PostProcess framework.
Added logic for handling new Array and Map representation.


Diffs (updated)
-

  graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 
607baf664 
  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java
 c0b9c1741 
  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphDatabase.java
 16aecd5e2 
  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/AtlasGraphSONReader.java
 ae119b0bc 
  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtility.java
 ec320b03e 
  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessManager.java
 d0a65f7b1 
  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/RelationshipTypeCache.java
 e4e82649b 
  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/postProcess/PostProcessListProperty.java
 PRE-CREATION 
  
graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/BaseUtils.java
 e863d9fae 
  
graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtilityPostProcessTest.java
 4d73c78ef 
  
graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtilityTest.java
 794b5471e 
  
graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/MappedElementCacheTest.java
 cac09d229 
  
graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessListPropertyTest.java
 PRE-CREATION 
  graphdb/janus/src/test/resources/col-2-legacy.json PRE-CREATION 
  graphdb/janus/src/test/resources/edge-legacy-col.json PRE-CREATION 
  graphdb/janus/src/test/resources/edge-legacy-col2.json PRE-CREATION 
  graphdb/janus/src/test/resources/edge-legacy-col3.json PRE-CREATION 
  graphdb/janus/src/test/resources/edge-legacy-col4.json PRE-CREATION 
  graphdb/janus/src/test/resources/table-v-147504.json 898dce5df 
  intg/src/main/java/org/apache/atlas/store/AtlasTypeDefStore.java b05754f4b 
  intg/src/test/java/org/apache/atlas/TestUtilsV2.java 886ce77f5 
  
repository/src/main/java/org/apache/atlas/repository/migration/DataMigrationService.java
 22cd55217 
  
repository/src/main/java/org/apache/atlas/repository/migration/TypesWithCollectionsFinder.java
 PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasTypeDefGraphStoreV1.java
 3c84e3c22 
  
repository/src/test/java/org/apache/atlas/repository/migration/ComplexAttributesTest.java
 PRE-CREATION 
  
repository/src/test/java/org/apache/atlas/repository/migration/HiveParititionTest.java
 ac0b79d38 
  
repository/src/test/java/org/apache/atlas/repository/migration/HiveStocksTest.java
 ffbf3200b 
  
repository/src/test/java/org/apache/atlas/repository/migration/MigrationBaseAsserts.java
 ec6e64a25 
  
repository/src/test/java/org/apache/atlas/repository/migration/TypesWithCollectionsFinderTest.java
 PRE-CREATION 
  
repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityTestBase.java
 d810a1166 
  repository/src/test/resources/complex-attr_db/atlas-migration-data.json 
PRE-CREATION 
  repository/src/test/resources/complex-attr_db/atlas-migration-typesdef.json 
PRE-CREATION 
  repository/src/test/resources/parts_db/atlas-migration-data.json 1414ea160 


Diff: https://reviews.apache.org/r/66928/diff/18/

Changes: https://reviews.apache.org/r/66928/diff/17-18/


Testing
---

**Unit tests**

Additional tests added.

**Functional tests**

Regular flow verified.

**[Pre-commit 
build](https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/423/)**


Thanks,

Ashutosh Mestry

Re: Review Request 66928: Data Migration: Import: Infer Types that Store Edge Ids

2018-05-20 Thread Ashutosh Mestry


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66928/
---

(Updated May 20, 2018, 4:30 p.m.)


Review request for atlas, Madhan Neethiraj, Ruchi Solani, and Sarath 
Subramanian.


Changes
---

Updates include: 
- Addressed review comments.
- Updated pre-commit build #.


Bugs: ATLAS-2637
https://issues.apache.org/jira/browse/ATLAS-2637


Repository: atlas


Description
---

**Approach**
New class: _TypesStoringEdgeIds_: Navigates through all the types in the 
_typeRegistry_, returns map of entity type and properties that store edge ids.
Modified: _DataMigrationService_: Uses output from class above and passes it 
down to migration.
Modified: _PostProcessManager_: Uses the map generated above and uses it for 
post processing.
Modified: _GraphSONUtility_: Improvement to check for vertex of type. This 
avoids potential exeception when a non-existent property is checked for 
presence.

Added PostProcess framework.
Added logic for handling new Array and Map representation.


Diffs (updated)
-

  graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 
607baf664 
  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java
 c0b9c1741 
  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphDatabase.java
 16aecd5e2 
  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/AtlasGraphSONReader.java
 ae119b0bc 
  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtility.java
 ec320b03e 
  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessManager.java
 d0a65f7b1 
  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/RelationshipTypeCache.java
 e4e82649b 
  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/postProcess/PostProcessListProperty.java
 PRE-CREATION 
  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/postProcess/PostProcessor.java
 PRE-CREATION 
  
graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/BaseUtils.java
 e863d9fae 
  
graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtilityPostProcessTest.java
 4d73c78ef 
  
graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtilityTest.java
 794b5471e 
  
graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/MappedElementCacheTest.java
 cac09d229 
  
graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessListPropertyTest.java
 PRE-CREATION 
  graphdb/janus/src/test/resources/col-2-legacy.json PRE-CREATION 
  graphdb/janus/src/test/resources/edge-legacy-col.json PRE-CREATION 
  graphdb/janus/src/test/resources/edge-legacy-col2.json PRE-CREATION 
  graphdb/janus/src/test/resources/edge-legacy-col3.json PRE-CREATION 
  graphdb/janus/src/test/resources/edge-legacy-col4.json PRE-CREATION 
  graphdb/janus/src/test/resources/table-v-147504.json 898dce5df 
  intg/src/main/java/org/apache/atlas/store/AtlasTypeDefStore.java b05754f4b 
  intg/src/test/java/org/apache/atlas/TestUtilsV2.java 886ce77f5 
  
repository/src/main/java/org/apache/atlas/repository/migration/DataMigrationService.java
 22cd55217 
  
repository/src/main/java/org/apache/atlas/repository/migration/TypesWithCollectionsFinder.java
 PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasTypeDefGraphStoreV1.java
 3c84e3c22 
  
repository/src/test/java/org/apache/atlas/repository/migration/ComplexAttributesTest.java
 PRE-CREATION 
  
repository/src/test/java/org/apache/atlas/repository/migration/HiveParititionTest.java
 ac0b79d38 
  
repository/src/test/java/org/apache/atlas/repository/migration/HiveStocksTest.java
 ffbf3200b 
  
repository/src/test/java/org/apache/atlas/repository/migration/MigrationBaseAsserts.java
 ec6e64a25 
  
repository/src/test/java/org/apache/atlas/repository/migration/TypesWithCollectionsFinderTest.java
 PRE-CREATION 
  
repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityTestBase.java
 d810a1166 
  repository/src/test/resources/complex-attr_db/atlas-migration-data.json 
PRE-CREATION 
  repository/src/test/resources/complex-attr_db/atlas-migration-typesdef.json 
PRE-CREATION 
  repository/src/test/resources/parts_db/atlas-migration-data.json 1414ea160 


Diff: https://reviews.apache.org/r/66928/diff/17/

Changes: https://reviews.apache.org/r/66928/diff/16-17/


Testing (updated)
---

**Unit tests**

Additional tests added.

**Functional tests**

Regular flow verified.

**[Pre-commit 
build](https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/423/)**


Thanks,

Ashutosh Mestry

Re: Review Request 66928: Data Migration: Import: Infer Types that Store Edge Ids

2018-05-20 Thread Ashutosh Mestry



> On May 19, 2018, 12:36 a.m., Madhan Neethiraj wrote:
> > graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtility.java
> > Lines 152 (patched)
> > 
> >
> > edges for all array elements will have __index set to 0. Does this get 
> > updated in subsequent processing?

Yes. This is just initialized here.


- Ashutosh


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66928/#review203462
---


On May 17, 2018, 8:48 p.m., Ashutosh Mestry wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66928/
> ---
> 
> (Updated May 17, 2018, 8:48 p.m.)
> 
> 
> Review request for atlas, Madhan Neethiraj, Ruchi Solani, and Sarath 
> Subramanian.
> 
> 
> Bugs: ATLAS-2637
> https://issues.apache.org/jira/browse/ATLAS-2637
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> **Approach**
> New class: _TypesStoringEdgeIds_: Navigates through all the types in the 
> _typeRegistry_, returns map of entity type and properties that store edge ids.
> Modified: _DataMigrationService_: Uses output from class above and passes it 
> down to migration.
> Modified: _PostProcessManager_: Uses the map generated above and uses it for 
> post processing.
> Modified: _GraphSONUtility_: Improvement to check for vertex of type. This 
> avoids potential exeception when a non-existent property is checked for 
> presence.
> 
> Added PostProcess framework.
> Added logic for handling new Array and Map representation.
> 
> 
> Diffs
> -
> 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 
> 607baf664 
>   
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java
>  c0b9c1741 
>   
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphDatabase.java
>  16aecd5e2 
>   
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/AtlasGraphSONReader.java
>  ae119b0bc 
>   
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtility.java
>  ec320b03e 
>   
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessManager.java
>  d0a65f7b1 
>   
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/RelationshipTypeCache.java
>  e4e82649b 
>   
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/postProcess/PostProcessListProperty.java
>  PRE-CREATION 
>   
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/postProcess/PostProcessor.java
>  PRE-CREATION 
>   
> graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/BaseUtils.java
>  e863d9fae 
>   
> graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtilityPostProcessTest.java
>  4d73c78ef 
>   
> graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtilityTest.java
>  794b5471e 
>   
> graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/MappedElementCacheTest.java
>  cac09d229 
>   
> graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessListPropertyTest.java
>  PRE-CREATION 
>   graphdb/janus/src/test/resources/col-2-legacy.json PRE-CREATION 
>   graphdb/janus/src/test/resources/edge-legacy-col.json PRE-CREATION 
>   graphdb/janus/src/test/resources/edge-legacy-col2.json PRE-CREATION 
>   graphdb/janus/src/test/resources/edge-legacy-col3.json PRE-CREATION 
>   graphdb/janus/src/test/resources/edge-legacy-col4.json PRE-CREATION 
>   graphdb/janus/src/test/resources/table-v-147504.json 898dce5df 
>   intg/src/main/java/org/apache/atlas/store/AtlasTypeDefStore.java b05754f4b 
>   intg/src/test/java/org/apache/atlas/TestUtilsV2.java 886ce77f5 
>   
> repository/src/main/java/org/apache/atlas/repository/migration/DataMigrationService.java
>  22cd55217 
>   
> repository/src/main/java/org/apache/atlas/repository/migration/TypesWithCollectionsFinder.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasTypeDefGraphStoreV1.java
>  3c84e3c22 
>   
> repository/src/test/java/org/apache/atlas/repository/migration/ComplexAttributesTest.java
>  PRE-CREATION 
>   
> repository/src/test/java/org/apache/atlas/repository/migration/HiveParititionTest.java
>  ac0b79d38 
>   
> repository/src/test/java/org/apache/atlas/repository/migration/HiveStocksTest.java
>  ffbf3200b 
>   
> repository/src/test/java/org/apache/atlas/repository/migration/MigrationBaseAsserts.java
>  ec6e64a25 
>

Re: Review Request 66928: Data Migration: Import: Infer Types that Store Edge Ids

2018-05-20 Thread Ashutosh Mestry



> On May 19, 2018, 12:36 a.m., Madhan Neethiraj wrote:
> > graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtility.java
> > Line 67 (original), 74 (patched)
> > 
> >
> > This one handles removal/updte of properties for map type attributes. 
> > Aren't similar updates needed for array type attributes as well 
> > (primitive/struct/entity-ref)?

Yes, only that the indexes are known only after all edges are created. I 
considered doing this during creation itself, it eded up becoming messy and 
number of computations creases drastically. There is an option that is possible 
if it is some how possible to store a temporary map of edge ids to indexes.


- Ashutosh


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66928/#review203462
---


On May 17, 2018, 8:48 p.m., Ashutosh Mestry wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66928/
> ---
> 
> (Updated May 17, 2018, 8:48 p.m.)
> 
> 
> Review request for atlas, Madhan Neethiraj, Ruchi Solani, and Sarath 
> Subramanian.
> 
> 
> Bugs: ATLAS-2637
> https://issues.apache.org/jira/browse/ATLAS-2637
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> **Approach**
> New class: _TypesStoringEdgeIds_: Navigates through all the types in the 
> _typeRegistry_, returns map of entity type and properties that store edge ids.
> Modified: _DataMigrationService_: Uses output from class above and passes it 
> down to migration.
> Modified: _PostProcessManager_: Uses the map generated above and uses it for 
> post processing.
> Modified: _GraphSONUtility_: Improvement to check for vertex of type. This 
> avoids potential exeception when a non-existent property is checked for 
> presence.
> 
> Added PostProcess framework.
> Added logic for handling new Array and Map representation.
> 
> 
> Diffs
> -
> 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 
> 607baf664 
>   
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java
>  c0b9c1741 
>   
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphDatabase.java
>  16aecd5e2 
>   
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/AtlasGraphSONReader.java
>  ae119b0bc 
>   
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtility.java
>  ec320b03e 
>   
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessManager.java
>  d0a65f7b1 
>   
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/RelationshipTypeCache.java
>  e4e82649b 
>   
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/postProcess/PostProcessListProperty.java
>  PRE-CREATION 
>   
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/postProcess/PostProcessor.java
>  PRE-CREATION 
>   
> graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/BaseUtils.java
>  e863d9fae 
>   
> graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtilityPostProcessTest.java
>  4d73c78ef 
>   
> graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtilityTest.java
>  794b5471e 
>   
> graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/MappedElementCacheTest.java
>  cac09d229 
>   
> graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessListPropertyTest.java
>  PRE-CREATION 
>   graphdb/janus/src/test/resources/col-2-legacy.json PRE-CREATION 
>   graphdb/janus/src/test/resources/edge-legacy-col.json PRE-CREATION 
>   graphdb/janus/src/test/resources/edge-legacy-col2.json PRE-CREATION 
>   graphdb/janus/src/test/resources/edge-legacy-col3.json PRE-CREATION 
>   graphdb/janus/src/test/resources/edge-legacy-col4.json PRE-CREATION 
>   graphdb/janus/src/test/resources/table-v-147504.json 898dce5df 
>   intg/src/main/java/org/apache/atlas/store/AtlasTypeDefStore.java b05754f4b 
>   intg/src/test/java/org/apache/atlas/TestUtilsV2.java 886ce77f5 
>   
> repository/src/main/java/org/apache/atlas/repository/migration/DataMigrationService.java
>  22cd55217 
>   
> repository/src/main/java/org/apache/atlas/repository/migration/TypesWithCollectionsFinder.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasTypeDefGraphStoreV1.java
>  3c84e3c22 
>   
> repository/src/test/java/org/apache/atlas/repository/migration/ComplexAttributesTes

Re: Review Request 66928: Data Migration: Import: Infer Types that Store Edge Ids

2018-05-18 Thread Madhan Neethiraj


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66928/#review203462
---




graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/AtlasGraphSONReader.java
Line 315 (original), 332 (patched)


shouldn't this return 'this.propertiesToPostProcess', instead of creating a 
new instance? If this method is used only to initialize, either rename it or 
avoid this method by directly initializing in line #252.



graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtility.java
Line 67 (original), 74 (patched)


This one handles removal/updte of properties for map type attributes. 
Aren't similar updates needed for array type attributes as well 
(primitive/struct/entity-ref)?



graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtility.java
Lines 152 (patched)


edges for all array elements will have __index set to 0. Does this get 
updated in subsequent processing?



graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtility.java
Lines 340 (patched)


Method replaceReferencedEdgeIdForList() and postProcessListProperty don't 
seem to be used. Please review and remove.



graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/postProcess/PostProcessListProperty.java
Lines 43 (patched)


Should newIdPositionMap be cleared here?



graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/postProcess/PostProcessor.java
Lines 80 (patched)


doesNotHaveProperty() - the name doesn't seem to match the condition 
"return v.property(Constants.TYPENAME_PROPERTY_KEY).isPresent()", which will 
return true for all entity/classification vertices. Please review.



graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/postProcess/PostProcessor.java
Lines 84 (patched)


isVertexOfType() ==> isInstanceVertexOfType()


- Madhan Neethiraj


On May 17, 2018, 8:48 p.m., Ashutosh Mestry wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66928/
> ---
> 
> (Updated May 17, 2018, 8:48 p.m.)
> 
> 
> Review request for atlas, Madhan Neethiraj, Ruchi Solani, and Sarath 
> Subramanian.
> 
> 
> Bugs: ATLAS-2637
> https://issues.apache.org/jira/browse/ATLAS-2637
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> **Approach**
> New class: _TypesStoringEdgeIds_: Navigates through all the types in the 
> _typeRegistry_, returns map of entity type and properties that store edge ids.
> Modified: _DataMigrationService_: Uses output from class above and passes it 
> down to migration.
> Modified: _PostProcessManager_: Uses the map generated above and uses it for 
> post processing.
> Modified: _GraphSONUtility_: Improvement to check for vertex of type. This 
> avoids potential exeception when a non-existent property is checked for 
> presence.
> 
> Added PostProcess framework.
> Added logic for handling new Array and Map representation.
> 
> 
> Diffs
> -
> 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 
> 607baf664 
>   
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java
>  c0b9c1741 
>   
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphDatabase.java
>  16aecd5e2 
>   
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/AtlasGraphSONReader.java
>  ae119b0bc 
>   
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtility.java
>  ec320b03e 
>   
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessManager.java
>  d0a65f7b1 
>   
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/RelationshipTypeCache.java
>  e4e82649b 
>   
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/postProcess/PostProcessListProperty.java
>  PRE-CREATION 
>   
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/postProcess/PostProcessor.java
>  PRE-CREATION 
>   
> graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/BaseUtils.java
>  e863d9fae 
>   
> graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb

Re: Review Request 66928: Data Migration: Import: Infer Types that Store Edge Ids

2018-05-17 Thread Sarath Subramanian


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66928/#review203388
---


Ship it!




Ship It!

- Sarath Subramanian


On May 17, 2018, 1:48 p.m., Ashutosh Mestry wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66928/
> ---
> 
> (Updated May 17, 2018, 1:48 p.m.)
> 
> 
> Review request for atlas, Madhan Neethiraj, Ruchi Solani, and Sarath 
> Subramanian.
> 
> 
> Bugs: ATLAS-2637
> https://issues.apache.org/jira/browse/ATLAS-2637
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> **Approach**
> New class: _TypesStoringEdgeIds_: Navigates through all the types in the 
> _typeRegistry_, returns map of entity type and properties that store edge ids.
> Modified: _DataMigrationService_: Uses output from class above and passes it 
> down to migration.
> Modified: _PostProcessManager_: Uses the map generated above and uses it for 
> post processing.
> Modified: _GraphSONUtility_: Improvement to check for vertex of type. This 
> avoids potential exeception when a non-existent property is checked for 
> presence.
> 
> Added PostProcess framework.
> Added logic for handling new Array and Map representation.
> 
> 
> Diffs
> -
> 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 
> 607baf664 
>   
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java
>  c0b9c1741 
>   
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphDatabase.java
>  16aecd5e2 
>   
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/AtlasGraphSONReader.java
>  ae119b0bc 
>   
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtility.java
>  ec320b03e 
>   
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessManager.java
>  d0a65f7b1 
>   
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/RelationshipTypeCache.java
>  e4e82649b 
>   
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/postProcess/PostProcessListProperty.java
>  PRE-CREATION 
>   
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/postProcess/PostProcessor.java
>  PRE-CREATION 
>   
> graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/BaseUtils.java
>  e863d9fae 
>   
> graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtilityPostProcessTest.java
>  4d73c78ef 
>   
> graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtilityTest.java
>  794b5471e 
>   
> graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/MappedElementCacheTest.java
>  cac09d229 
>   
> graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessListPropertyTest.java
>  PRE-CREATION 
>   graphdb/janus/src/test/resources/col-2-legacy.json PRE-CREATION 
>   graphdb/janus/src/test/resources/edge-legacy-col.json PRE-CREATION 
>   graphdb/janus/src/test/resources/edge-legacy-col2.json PRE-CREATION 
>   graphdb/janus/src/test/resources/edge-legacy-col3.json PRE-CREATION 
>   graphdb/janus/src/test/resources/edge-legacy-col4.json PRE-CREATION 
>   graphdb/janus/src/test/resources/table-v-147504.json 898dce5df 
>   intg/src/main/java/org/apache/atlas/store/AtlasTypeDefStore.java b05754f4b 
>   intg/src/test/java/org/apache/atlas/TestUtilsV2.java 886ce77f5 
>   
> repository/src/main/java/org/apache/atlas/repository/migration/DataMigrationService.java
>  22cd55217 
>   
> repository/src/main/java/org/apache/atlas/repository/migration/TypesWithCollectionsFinder.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasTypeDefGraphStoreV1.java
>  3c84e3c22 
>   
> repository/src/test/java/org/apache/atlas/repository/migration/ComplexAttributesTest.java
>  PRE-CREATION 
>   
> repository/src/test/java/org/apache/atlas/repository/migration/HiveParititionTest.java
>  ac0b79d38 
>   
> repository/src/test/java/org/apache/atlas/repository/migration/HiveStocksTest.java
>  ffbf3200b 
>   
> repository/src/test/java/org/apache/atlas/repository/migration/MigrationBaseAsserts.java
>  ec6e64a25 
>   
> repository/src/test/java/org/apache/atlas/repository/migration/TypesWithCollectionsFinderTest.java
>  PRE-CREATION 
>   
> repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityTestBase.java
>  d810a1166 
>   repository/src/test/resources/complex-attr_db/atlas-migration-data.json 
> PRE-CREATION 
>   repository/src/test/resources/complex-attr_db/atlas-migration-ty

Re: Review Request 66928: Data Migration: Import: Infer Types that Store Edge Ids

2018-05-17 Thread Ashutosh Mestry


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66928/
---

(Updated May 17, 2018, 8:48 p.m.)


Review request for atlas, Madhan Neethiraj, Ruchi Solani, and Sarath 
Subramanian.


Changes
---

Updates include: Addressed review comments.


Bugs: ATLAS-2637
https://issues.apache.org/jira/browse/ATLAS-2637


Repository: atlas


Description
---

**Approach**
New class: _TypesStoringEdgeIds_: Navigates through all the types in the 
_typeRegistry_, returns map of entity type and properties that store edge ids.
Modified: _DataMigrationService_: Uses output from class above and passes it 
down to migration.
Modified: _PostProcessManager_: Uses the map generated above and uses it for 
post processing.
Modified: _GraphSONUtility_: Improvement to check for vertex of type. This 
avoids potential exeception when a non-existent property is checked for 
presence.

Added PostProcess framework.
Added logic for handling new Array and Map representation.


Diffs (updated)
-

  graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 
607baf664 
  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java
 c0b9c1741 
  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphDatabase.java
 16aecd5e2 
  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/AtlasGraphSONReader.java
 ae119b0bc 
  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtility.java
 ec320b03e 
  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessManager.java
 d0a65f7b1 
  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/RelationshipTypeCache.java
 e4e82649b 
  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/postProcess/PostProcessListProperty.java
 PRE-CREATION 
  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/postProcess/PostProcessor.java
 PRE-CREATION 
  
graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/BaseUtils.java
 e863d9fae 
  
graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtilityPostProcessTest.java
 4d73c78ef 
  
graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtilityTest.java
 794b5471e 
  
graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/MappedElementCacheTest.java
 cac09d229 
  
graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessListPropertyTest.java
 PRE-CREATION 
  graphdb/janus/src/test/resources/col-2-legacy.json PRE-CREATION 
  graphdb/janus/src/test/resources/edge-legacy-col.json PRE-CREATION 
  graphdb/janus/src/test/resources/edge-legacy-col2.json PRE-CREATION 
  graphdb/janus/src/test/resources/edge-legacy-col3.json PRE-CREATION 
  graphdb/janus/src/test/resources/edge-legacy-col4.json PRE-CREATION 
  graphdb/janus/src/test/resources/table-v-147504.json 898dce5df 
  intg/src/main/java/org/apache/atlas/store/AtlasTypeDefStore.java b05754f4b 
  intg/src/test/java/org/apache/atlas/TestUtilsV2.java 886ce77f5 
  
repository/src/main/java/org/apache/atlas/repository/migration/DataMigrationService.java
 22cd55217 
  
repository/src/main/java/org/apache/atlas/repository/migration/TypesWithCollectionsFinder.java
 PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasTypeDefGraphStoreV1.java
 3c84e3c22 
  
repository/src/test/java/org/apache/atlas/repository/migration/ComplexAttributesTest.java
 PRE-CREATION 
  
repository/src/test/java/org/apache/atlas/repository/migration/HiveParititionTest.java
 ac0b79d38 
  
repository/src/test/java/org/apache/atlas/repository/migration/HiveStocksTest.java
 ffbf3200b 
  
repository/src/test/java/org/apache/atlas/repository/migration/MigrationBaseAsserts.java
 ec6e64a25 
  
repository/src/test/java/org/apache/atlas/repository/migration/TypesWithCollectionsFinderTest.java
 PRE-CREATION 
  
repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityTestBase.java
 d810a1166 
  repository/src/test/resources/complex-attr_db/atlas-migration-data.json 
PRE-CREATION 
  repository/src/test/resources/complex-attr_db/atlas-migration-typesdef.json 
PRE-CREATION 
  repository/src/test/resources/parts_db/atlas-migration-data.json 1414ea160 


Diff: https://reviews.apache.org/r/66928/diff/15/

Changes: https://reviews.apache.org/r/66928/diff/14-15/


Testing
---

**Unit tests**

Additional tests added.

**Functional tests**

Regular flow verified.

**[Pre-commit 
build](https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/416/)**


Thanks,

Ashutosh Mestry

Re: Review Request 66928: Data Migration: Import: Infer Types that Store Edge Ids

2018-05-17 Thread Sarath Subramanian


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66928/#review203360
---




repository/src/main/java/org/apache/atlas/repository/migration/TypesWithCollectionsFinder.java
Lines 66 (patched)


getProperties() method in 66 can be replaced with one in line 82. 
AtlasStructDef is supertype of AtlasEntityDef.



repository/src/main/java/org/apache/atlas/repository/migration/TypesWithCollectionsFinder.java
Lines 109 (patched)


rename 109 and 113 with getArrayElementType and getMapValueType for better 
readability.



repository/src/main/java/org/apache/atlas/repository/migration/TypesWithCollectionsFinder.java
Lines 117 (patched)


update() and updatePrimitive() looks duplicate of each other with only 
"_PRIMTITIVE" string difference, can be refactored into a single method.


- Sarath Subramanian


On May 15, 2018, 10:58 p.m., Ashutosh Mestry wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66928/
> ---
> 
> (Updated May 15, 2018, 10:58 p.m.)
> 
> 
> Review request for atlas, Madhan Neethiraj, Ruchi Solani, and Sarath 
> Subramanian.
> 
> 
> Bugs: ATLAS-2637
> https://issues.apache.org/jira/browse/ATLAS-2637
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> **Approach**
> New class: _TypesStoringEdgeIds_: Navigates through all the types in the 
> _typeRegistry_, returns map of entity type and properties that store edge ids.
> Modified: _DataMigrationService_: Uses output from class above and passes it 
> down to migration.
> Modified: _PostProcessManager_: Uses the map generated above and uses it for 
> post processing.
> Modified: _GraphSONUtility_: Improvement to check for vertex of type. This 
> avoids potential exeception when a non-existent property is checked for 
> presence.
> 
> Added PostProcess framework.
> Added logic for handling new Array and Map representation.
> 
> 
> Diffs
> -
> 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 
> 607baf664 
>   
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java
>  c0b9c1741 
>   
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphDatabase.java
>  16aecd5e2 
>   
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/AtlasGraphSONReader.java
>  ae119b0bc 
>   
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtility.java
>  ec320b03e 
>   
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessManager.java
>  d0a65f7b1 
>   
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/RelationshipTypeCache.java
>  e4e82649b 
>   
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/postProcess/PostProcessListProperty.java
>  PRE-CREATION 
>   
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/postProcess/PostProcessor.java
>  PRE-CREATION 
>   
> graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/BaseUtils.java
>  e863d9fae 
>   
> graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtilityPostProcessTest.java
>  4d73c78ef 
>   
> graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtilityTest.java
>  794b5471e 
>   
> graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/MappedElementCacheTest.java
>  cac09d229 
>   
> graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessListPropertyTest.java
>  PRE-CREATION 
>   graphdb/janus/src/test/resources/col-2-legacy.json PRE-CREATION 
>   graphdb/janus/src/test/resources/edge-legacy-col.json PRE-CREATION 
>   graphdb/janus/src/test/resources/edge-legacy-col2.json PRE-CREATION 
>   graphdb/janus/src/test/resources/edge-legacy-col3.json PRE-CREATION 
>   graphdb/janus/src/test/resources/edge-legacy-col4.json PRE-CREATION 
>   graphdb/janus/src/test/resources/table-v-147504.json 898dce5df 
>   intg/src/main/java/org/apache/atlas/store/AtlasTypeDefStore.java b05754f4b 
>   intg/src/test/java/org/apache/atlas/TestUtilsV2.java 886ce77f5 
>   
> repository/src/main/java/org/apache/atlas/repository/migration/DataMigrationService.java
>  22cd55217 
>   
> repository/src/main/java/org/apache/atlas/repository/migration/TypesWithCollectionsFinder.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasTypeDefGraphStoreV1.j

Re: Review Request 66928: Data Migration: Import: Infer Types that Store Edge Ids

2018-05-15 Thread Ashutosh Mestry


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66928/
---

(Updated May 16, 2018, 5:58 a.m.)


Review request for atlas, Madhan Neethiraj, Ruchi Solani, and Sarath 
Subramanian.


Changes
---

Updates include:
- Removed commented code block from _GraphBackedIndexer_.
- Updated logic for _TypeStoringEdgeIds_.


Bugs: ATLAS-2637
https://issues.apache.org/jira/browse/ATLAS-2637


Repository: atlas


Description
---

**Approach**
New class: _TypesStoringEdgeIds_: Navigates through all the types in the 
_typeRegistry_, returns map of entity type and properties that store edge ids.
Modified: _DataMigrationService_: Uses output from class above and passes it 
down to migration.
Modified: _PostProcessManager_: Uses the map generated above and uses it for 
post processing.
Modified: _GraphSONUtility_: Improvement to check for vertex of type. This 
avoids potential exeception when a non-existent property is checked for 
presence.

Added PostProcess framework.
Added logic for handling new Array and Map representation.


Diffs (updated)
-

  graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 
607baf664 
  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java
 c0b9c1741 
  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphDatabase.java
 16aecd5e2 
  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/AtlasGraphSONReader.java
 ae119b0bc 
  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtility.java
 ec320b03e 
  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessManager.java
 d0a65f7b1 
  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/RelationshipTypeCache.java
 e4e82649b 
  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/postProcess/PostProcessListProperty.java
 PRE-CREATION 
  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/postProcess/PostProcessMapProperty.java
 PRE-CREATION 
  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/postProcess/PostProcessor.java
 PRE-CREATION 
  
graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/BaseUtils.java
 e863d9fae 
  
graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtilityPostProcessTest.java
 4d73c78ef 
  
graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtilityTest.java
 794b5471e 
  
graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/MappedElementCacheTest.java
 cac09d229 
  
graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessListPropertyTest.java
 PRE-CREATION 
  
graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessMapPropertyTest.java
 PRE-CREATION 
  graphdb/janus/src/test/resources/col-2-legacy.json PRE-CREATION 
  graphdb/janus/src/test/resources/edge-legacy-col.json PRE-CREATION 
  graphdb/janus/src/test/resources/edge-legacy-col2.json PRE-CREATION 
  graphdb/janus/src/test/resources/table-v-147504.json 898dce5df 
  intg/src/main/java/org/apache/atlas/store/AtlasTypeDefStore.java b05754f4b 
  intg/src/test/java/org/apache/atlas/TestUtilsV2.java 886ce77f5 
  
repository/src/main/java/org/apache/atlas/repository/migration/DataMigrationService.java
 22cd55217 
  
repository/src/main/java/org/apache/atlas/repository/migration/TypesStoringEdgeIds.java
 PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasTypeDefGraphStoreV1.java
 3c84e3c22 
  
repository/src/test/java/org/apache/atlas/repository/migration/MigrationBaseAsserts.java
 ec6e64a25 
  
repository/src/test/java/org/apache/atlas/repository/migration/TypesStoringEdgeIdsTest.java
 PRE-CREATION 
  repository/src/test/resources/parts_db/atlas-migration-data.json 1414ea160 


Diff: https://reviews.apache.org/r/66928/diff/10/

Changes: https://reviews.apache.org/r/66928/diff/9-10/


Testing
---

**Unit tests**

Additional tests added.

**Functional tests**

Regular flow verified.

**[Pre-commit 
build](https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/404/)**


Thanks,

Ashutosh Mestry

Re: Review Request 66928: Data Migration: Import: Infer Types that Store Edge Ids

2018-05-14 Thread Ashutosh Mestry


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66928/
---

(Updated May 15, 2018, 4:18 a.m.)


Review request for atlas, Madhan Neethiraj, Ruchi Solani, and Sarath 
Subramanian.


Changes
---

Updates include:
- Handling primitive and non-primitive Map.
- Handling non-primitive Arrays.
- Refactoring.
- Fixed failing UTs.


Bugs: ATLAS-2637
https://issues.apache.org/jira/browse/ATLAS-2637


Repository: atlas


Description
---

**Approach**
New class: _TypesStoringEdgeIds_: Navigates through all the types in the 
_typeRegistry_, returns map of entity type and properties that store edge ids.
Modified: _DataMigrationService_: Uses output from class above and passes it 
down to migration.
Modified: _PostProcessManager_: Uses the map generated above and uses it for 
post processing.
Modified: _GraphSONUtility_: Improvement to check for vertex of type. This 
avoids potential exeception when a non-existent property is checked for 
presence.

Added PostProcess framework.
Added logic for handling new Array and Map representation.


Diffs (updated)
-

  graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 
607baf664 
  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java
 c0b9c1741 
  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphDatabase.java
 16aecd5e2 
  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/AtlasGraphSONReader.java
 aa0b6375d 
  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtility.java
 ec320b03e 
  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessManager.java
 d0a65f7b1 
  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/RelationshipTypeCache.java
 e4e82649b 
  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/postProcess/PostProcessListProperty.java
 PRE-CREATION 
  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/postProcess/PostProcessMapProperty.java
 PRE-CREATION 
  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/postProcess/PostProcessor.java
 PRE-CREATION 
  
graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/BaseUtils.java
 de43e0f41 
  
graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtilityPostProcessTest.java
 4d73c78ef 
  
graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtilityTest.java
 794b5471e 
  
graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/MappedElementCacheTest.java
 cac09d229 
  
graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessListPropertyTest.java
 PRE-CREATION 
  
graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessMapPropertyTest.java
 PRE-CREATION 
  graphdb/janus/src/test/resources/col-2-legacy.json PRE-CREATION 
  graphdb/janus/src/test/resources/edge-legacy-col.json PRE-CREATION 
  graphdb/janus/src/test/resources/edge-legacy-col2.json PRE-CREATION 
  graphdb/janus/src/test/resources/table-v-147504.json 898dce5df 
  intg/src/main/java/org/apache/atlas/store/AtlasTypeDefStore.java b05754f4b 
  intg/src/test/java/org/apache/atlas/TestUtilsV2.java 886ce77f5 
  
repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java
 f289365f5 
  
repository/src/main/java/org/apache/atlas/repository/migration/DataMigrationService.java
 22cd55217 
  
repository/src/main/java/org/apache/atlas/repository/migration/TypesStoringEdgeIds.java
 PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasTypeDefGraphStoreV1.java
 3c84e3c22 
  
repository/src/test/java/org/apache/atlas/repository/migration/MigrationBaseAsserts.java
 ec6e64a25 
  
repository/src/test/java/org/apache/atlas/repository/migration/TypesStoringEdgeIdsTest.java
 PRE-CREATION 
  repository/src/test/resources/parts_db/atlas-migration-data.json 1414ea160 


Diff: https://reviews.apache.org/r/66928/diff/9/

Changes: https://reviews.apache.org/r/66928/diff/8-9/


Testing (updated)
---

**Unit tests**

Additional tests added.

**Functional tests**

Regular flow verified.

**[Pre-commit 
build](https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/404/)**


Thanks,

Ashutosh Mestry

Re: Review Request 66928: Data Migration: Import: Infer Types that Store Edge Ids

2018-05-11 Thread Ashutosh Mestry


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66928/
---

(Updated May 11, 2018, 10:27 p.m.)


Review request for atlas, Madhan Neethiraj, Ruchi Solani, and Sarath 
Subramanian.


Changes
---

Updates include:
- Refactoring.
- Minor modification to post processing.
- Fix to new attribute assignment.


Bugs: ATLAS-2637
https://issues.apache.org/jira/browse/ATLAS-2637


Repository: atlas


Description
---

**Approach**
New class: _TypesStoringEdgeIds_: Navigates through all the types in the 
_typeRegistry_, returns map of entity type and properties that store edge ids.
Modified: _DataMigrationService_: Uses output from class above and passes it 
down to migration.
Modified: _PostProcessManager_: Uses the map generated above and uses it for 
post processing.
Modified: _GraphSONUtility_: Improvement to check for vertex of type. This 
avoids potential exeception when a non-existent property is checked for 
presence.

Added PostProcess framework.
Added logic for handling new Array and Map representation.


Diffs (updated)
-

  graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 
607baf664 
  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java
 c0b9c1741 
  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphDatabase.java
 3636ca65a 
  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/AtlasGraphSONReader.java
 aa0b6375d 
  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtility.java
 ec320b03e 
  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessManager.java
 d0a65f7b1 
  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/RelationshipTypeCache.java
 e4e82649b 
  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/postProcess/PostProcessListProperty.java
 PRE-CREATION 
  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/postProcess/PostProcessMapProperty.java
 PRE-CREATION 
  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/postProcess/PostProcessor.java
 PRE-CREATION 
  
graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/BaseUtils.java
 de43e0f41 
  
graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtilityPostProcessTest.java
 4d73c78ef 
  
graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtilityTest.java
 794b5471e 
  
graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/MappedElementCacheTest.java
 cac09d229 
  
graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessListPropertyTest.java
 PRE-CREATION 
  
graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessMapPropertyTest.java
 PRE-CREATION 
  graphdb/janus/src/test/resources/col-2-legacy.json PRE-CREATION 
  graphdb/janus/src/test/resources/edge-legacy-col.json PRE-CREATION 
  graphdb/janus/src/test/resources/edge-legacy-col2.json PRE-CREATION 
  graphdb/janus/src/test/resources/table-v-147504.json 898dce5df 
  intg/src/main/java/org/apache/atlas/store/AtlasTypeDefStore.java b05754f4b 
  
repository/src/main/java/org/apache/atlas/repository/migration/DataMigrationService.java
 22cd55217 
  
repository/src/main/java/org/apache/atlas/repository/migration/TypesStoringEdgeIds.java
 PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasTypeDefGraphStoreV1.java
 3c84e3c22 
  
repository/src/test/java/org/apache/atlas/repository/migration/MigrationBaseAsserts.java
 ec6e64a25 
  
repository/src/test/java/org/apache/atlas/repository/migration/TypesStoringEdgeIdsTest.java
 PRE-CREATION 


Diff: https://reviews.apache.org/r/66928/diff/8/

Changes: https://reviews.apache.org/r/66928/diff/7-8/


Testing
---

**Unit tests**

Additional tests added.

**Functional tests**

Regular flow verified.

**[Pre-commit 
build](https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/382/)**


Thanks,

Ashutosh Mestry

Re: Review Request 66928: Data Migration: Import: Infer Types that Store Edge Ids

2018-05-10 Thread Ashutosh Mestry


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66928/
---

(Updated May 11, 2018, 1:50 a.m.)


Review request for atlas, Madhan Neethiraj, Ruchi Solani, and Sarath 
Subramanian.


Changes
---

Updates include: Fixes based on testing.


Bugs: ATLAS-2637
https://issues.apache.org/jira/browse/ATLAS-2637


Repository: atlas


Description
---

**Approach**
New class: _TypesStoringEdgeIds_: Navigates through all the types in the 
_typeRegistry_, returns map of entity type and properties that store edge ids.
Modified: _DataMigrationService_: Uses output from class above and passes it 
down to migration.
Modified: _PostProcessManager_: Uses the map generated above and uses it for 
post processing.
Modified: _GraphSONUtility_: Improvement to check for vertex of type. This 
avoids potential exeception when a non-existent property is checked for 
presence.

Added PostProcess framework.
Added logic for handling new Array and Map representation.


Diffs (updated)
-

  graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 
607baf664 
  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java
 c0b9c1741 
  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphDatabase.java
 3636ca65a 
  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/AtlasGraphSONReader.java
 aa0b6375d 
  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtility.java
 ec320b03e 
  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessManager.java
 d0a65f7b1 
  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/postProcess/PostProcessListProperty.java
 PRE-CREATION 
  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/postProcess/PostProcessMapProperty.java
 PRE-CREATION 
  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/postProcess/PostProcessor.java
 PRE-CREATION 
  
graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/BaseUtils.java
 de43e0f41 
  
graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtilityPostProcessTest.java
 4d73c78ef 
  
graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/MappedElementCacheTest.java
 cac09d229 
  
graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessListPropertyTest.java
 PRE-CREATION 
  
graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessMapPropertyTest.java
 PRE-CREATION 
  graphdb/janus/src/test/resources/col-2-legacy.json PRE-CREATION 
  graphdb/janus/src/test/resources/edge-legacy-col.json PRE-CREATION 
  graphdb/janus/src/test/resources/edge-legacy-col2.json PRE-CREATION 
  graphdb/janus/src/test/resources/table-v-147504.json 898dce5df 
  intg/src/main/java/org/apache/atlas/store/AtlasTypeDefStore.java b05754f4b 
  
repository/src/main/java/org/apache/atlas/repository/migration/DataMigrationService.java
 22cd55217 
  
repository/src/main/java/org/apache/atlas/repository/migration/TypesStoringEdgeIds.java
 PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasTypeDefGraphStoreV1.java
 3c84e3c22 
  
repository/src/test/java/org/apache/atlas/repository/migration/MigrationBaseAsserts.java
 ec6e64a25 
  
repository/src/test/java/org/apache/atlas/repository/migration/TypesStoringEdgeIdsTest.java
 PRE-CREATION 


Diff: https://reviews.apache.org/r/66928/diff/7/

Changes: https://reviews.apache.org/r/66928/diff/6-7/


Testing
---

**Unit tests**

Additional tests added.

**Functional tests**

Regular flow verified.


Thanks,

Ashutosh Mestry

Re: Review Request 66928: Data Migration: Import: Infer Types that Store Edge Ids

2018-05-10 Thread Ashutosh Mestry


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66928/
---

(Updated May 11, 2018, 12:40 a.m.)


Review request for atlas, Madhan Neethiraj, Ruchi Solani, and Sarath 
Subramanian.


Changes
---

Updates include: Array and Map type handling.


Bugs: ATLAS-2637
https://issues.apache.org/jira/browse/ATLAS-2637


Repository: atlas


Description (updated)
---

**Approach**
New class: _TypesStoringEdgeIds_: Navigates through all the types in the 
_typeRegistry_, returns map of entity type and properties that store edge ids.
Modified: _DataMigrationService_: Uses output from class above and passes it 
down to migration.
Modified: _PostProcessManager_: Uses the map generated above and uses it for 
post processing.
Modified: _GraphSONUtility_: Improvement to check for vertex of type. This 
avoids potential exeception when a non-existent property is checked for 
presence.

Added PostProcess framework.
Added logic for handling new Array and Map representation.


Diffs (updated)
-

  graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 
607baf664 
  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java
 c0b9c1741 
  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphDatabase.java
 3636ca65a 
  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/AtlasGraphSONReader.java
 aa0b6375d 
  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtility.java
 ec320b03e 
  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessManager.java
 d0a65f7b1 
  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/postProcess/PostProcessListProperty.java
 PRE-CREATION 
  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/postProcess/PostProcessMapProperty.java
 PRE-CREATION 
  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/postProcess/PostProcessor.java
 PRE-CREATION 
  
graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/BaseUtils.java
 de43e0f41 
  
graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtilityPostProcessTest.java
 4d73c78ef 
  
graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessListPropertyTest.java
 PRE-CREATION 
  
graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessMapPropertyTest.java
 PRE-CREATION 
  graphdb/janus/src/test/resources/col-2-legacy.json PRE-CREATION 
  graphdb/janus/src/test/resources/edge-legacy-col.json PRE-CREATION 
  graphdb/janus/src/test/resources/edge-legacy-col2.json PRE-CREATION 
  graphdb/janus/src/test/resources/table-v-147504.json 898dce5df 
  intg/src/main/java/org/apache/atlas/store/AtlasTypeDefStore.java b05754f4b 
  
repository/src/main/java/org/apache/atlas/repository/migration/DataMigrationService.java
 22cd55217 
  
repository/src/main/java/org/apache/atlas/repository/migration/TypesStoringEdgeIds.java
 PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasTypeDefGraphStoreV1.java
 3c84e3c22 
  
repository/src/test/java/org/apache/atlas/repository/migration/MigrationBaseAsserts.java
 ec6e64a25 
  
repository/src/test/java/org/apache/atlas/repository/migration/TypesStoringEdgeIdsTest.java
 PRE-CREATION 


Diff: https://reviews.apache.org/r/66928/diff/5/

Changes: https://reviews.apache.org/r/66928/diff/4-5/


Testing
---

**Unit tests**

Additional tests added.

**Functional tests**

Regular flow verified.


Thanks,

Ashutosh Mestry

Re: Review Request 66928: Data Migration: Import: Infer Types that Store Edge Ids

2018-05-02 Thread Ashutosh Mestry


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66928/
---

(Updated May 3, 2018, 5:39 a.m.)


Review request for atlas, Madhan Neethiraj and Ruchi Solani.


Changes
---

Updates include:
- Refactored _AtlasGraphSONReader.Builder.setDefaults_.


Bugs: ATLAS-2637
https://issues.apache.org/jira/browse/ATLAS-2637


Repository: atlas


Description
---

**Approach**
New class: _TypesStoringEdgeIds_: Navigates through all the types in the 
_typeRegistry_, returns map of entity type and properties that store edge ids.
Modified: _DataMigrationService_: Uses output from class above and passes it 
down to migration.
Modified: _PostProcessManager_: Uses the map generated above and uses it for 
post processing.
Modified: _GraphSONUtility_: Improvement to check for vertex of type. This 
avoids potential exeception when a non-existent property is checked for 
presence.


Diffs (updated)
-

  graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 
607baf664 
  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java
 c0b9c1741 
  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphDatabase.java
 86cd29900 
  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/AtlasGraphSONReader.java
 aa0b6375d 
  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtility.java
 ec320b03e 
  
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessManager.java
 d0a65f7b1 
  
graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtilityPostProcessTest.java
 4d73c78ef 
  
graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/Titan0Graph.java
 d191b5549 
  intg/src/main/java/org/apache/atlas/store/AtlasTypeDefStore.java b05754f4b 
  
repository/src/main/java/org/apache/atlas/repository/migration/DataMigrationService.java
 22cd55217 
  
repository/src/main/java/org/apache/atlas/repository/migration/TypesStoringEdgeIds.java
 PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasTypeDefGraphStoreV1.java
 3c84e3c22 
  
repository/src/test/java/org/apache/atlas/repository/migration/MigrationBaseAsserts.java
 ec6e64a25 
  
repository/src/test/java/org/apache/atlas/repository/migration/TypesStoringEdgeIdsTest.java
 PRE-CREATION 


Diff: https://reviews.apache.org/r/66928/diff/2/

Changes: https://reviews.apache.org/r/66928/diff/1-2/


Testing
---

**Unit tests**

Additional tests added.

**Functional tests**

Regular flow verified.


Thanks,

Ashutosh Mestry

Re: Review Request 66928: Data Migration: Import: Infer Types that Store Edge Ids

Re: Review Request 66928: Data Migration: Import: Infer Types that Store Edge Ids

Re: Review Request 66928: Data Migration: Import: Infer Types that Store Edge Ids

Re: Review Request 66928: Data Migration: Import: Infer Types that Store Edge Ids

Re: Review Request 66928: Data Migration: Import: Infer Types that Store Edge Ids

Re: Review Request 66928: Data Migration: Import: Infer Types that Store Edge Ids

Re: Review Request 66928: Data Migration: Import: Infer Types that Store Edge Ids

Re: Review Request 66928: Data Migration: Import: Infer Types that Store Edge Ids

Re: Review Request 66928: Data Migration: Import: Infer Types that Store Edge Ids

Re: Review Request 66928: Data Migration: Import: Infer Types that Store Edge Ids

Re: Review Request 66928: Data Migration: Import: Infer Types that Store Edge Ids

Re: Review Request 66928: Data Migration: Import: Infer Types that Store Edge Ids

Re: Review Request 66928: Data Migration: Import: Infer Types that Store Edge Ids

Re: Review Request 66928: Data Migration: Import: Infer Types that Store Edge Ids

Re: Review Request 66928: Data Migration: Import: Infer Types that Store Edge Ids

Re: Review Request 66928: Data Migration: Import: Infer Types that Store Edge Ids

Re: Review Request 66928: Data Migration: Import: Infer Types that Store Edge Ids

Re: Review Request 66928: Data Migration: Import: Infer Types that Store Edge Ids

18 matches

Site Navigation

Mail list logo

Footer information