subject:"Re\: Review Request 47810\: ATLAS\-694\: Update Atlas to use Graph DB abstraction layer"


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47810/#review151403
---




repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java
 (line 356)


can remove propertyname and get it from PropertyKey everywhere?


- Suma Shivaprasad


On Oct. 4, 2016, 1:28 a.m., Jeff Hagelberg wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47810/
> ---
> 
> (Updated Oct. 4, 2016, 1:28 a.m.)
> 
> 
> Review request for atlas, David Kantor and Neeru Gupta.
> 
> 
> Bugs: ATLAS-694
> https://issues.apache.org/jira/browse/ATLAS-694
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> ATLAS-694: Update Atlas to use abstraction layer.  All of the Atlas code 
> (with the exception of the catalog, which was only updated minimally) has 
> been updated to use the graph database abstraction layer.  In addition, there 
> is now an optional Atlas configuration property that specifies the class of 
> the abstraction layer database to use.  I basically put all of the changes in 
> here with the exception of the actual Titan 1 implementation of code.  This 
> includes the changes to support Tinkerpop 3 syntax.  This is mostly to 
> expedite getting the changes into Atlas.  Originally the TP3 changes were 
> going to be brought in as part of the Titan 1 implementation task.
> 
> Change Summary:
> 
>- change Atlas classes to use AtlasGraph,AtlasVertex,AtlasEdge, etc 
> instead of TitanGraph/Vertex/Edge, etc
>- compile time dependency on titan 0.5.4/TP 2 removed (except in Catalog, 
> which was only changed to use AtlasGraphProvider/AtlasGraph) - see 
> repository\pom.xml, other pom.xmls
>- updated DSL translation to generate Gremlin that is compliant with TP3 
> when TP3 is being used.  See GremlinQuery.scala, 
> GraphPersistenceStrategies.scala
>- TitanGraphProvider replaced by AtlasGraphProvider.  Graph database 
> implementation is determined from a new optional configuration property
>- HiveTitanSample is no longer used by tests.  It has been replaced by 
> hive-instances.json (which uses normal Atlas json syntax).  The data is saved 
> with a new JSONImporter class.  This was needed because the graphson syntax 
> used by HiveTitanSample is not compatible with TP3.  
> 
> Last rebase: 10/3/2016
> 
> 
> Diffs
> -
> 
>   .gitignore e10adbc4457f6297600f0feb01eb54718b8ec406 
>   addons/falcon-bridge/pom.xml 1365bd05a388dc92f7a56c7f7427b5b85f97c7da 
>   addons/hdfs-model/pom.xml 492f39cea085c6e69781e17bcbdbc3a231806df3 
>   
> addons/hdfs-model/src/test/java/org/apache/atlas/fs/model/HDFSModelTest.java 
> ac60294e328835ba0340e150799ddfb348ccdb52 
>   addons/hive-bridge/pom.xml 6993bdb938a6095ca24482e290393eeeb3911bcb 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
>  ad7a4a5d09d8542a841701dfe04981f65f767c14 
>   addons/sqoop-bridge/pom.xml 8c9d278d43b5979ea1743d10845905c13249f8a6 
>   addons/storm-bridge/pom.xml 12c1208b448d456a923bd7309601174ddb561ba5 
>   catalog/pom.xml 2f58a8f0748de65ab78eab35df6abd2fe7c336af 
>   catalog/src/main/java/org/apache/atlas/catalog/query/BaseQuery.java 
> e7bb505075983371ca12d9bc1d8c6eb240c3d134 
>   distro/src/conf/atlas-application.properties 
> d334600dc5534840409b586157799ef3abf9abf2 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasEdge.java 
> dd4b7e614cdd9bf30f957fb6a839d8c60f3e1701 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasElement.java
>  1bc0fc38c0802897f32260520770a16795474d04 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 
> 995c5457ac7f807172f367cc8e3348b3a98dd6f3 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphIndex.java
>  41194d34f079842db0d95c73a8b099459f76ff2f 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphManagement.java
>  c8cd2842ca3090b6bbd384c773b4eb45aff149ce 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphQuery.java
>  93447495bcf18e9f19df9df68fd1cbe1427fc462 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasIndexQuery.java
>  e719d306ffe9f68e3ac6f7406baaf60a12390c34 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasPropertyKey.java
>  315ecddb861e1a1be6e0ab9b36fe4c0a52486ae8 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasVertex.java
>  fff6fb79247e7d0615ce83c4cbbd93d1bf8cf29c 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasVertexQuery.java
>  366cfe131512f36c02f59ad7e969ff94dfb0e12c 
>   
>

Re: Review Request 47810: ATLAS-694: Update Atlas to use Graph DB abstraction layer


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47810/#review151399
---




repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java 
(line 295)


why is this needed? We fixed this earlier by allowing keywords which are 
escaped by quotes ``


- Suma Shivaprasad


On Oct. 4, 2016, 1:28 a.m., Jeff Hagelberg wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47810/
> ---
> 
> (Updated Oct. 4, 2016, 1:28 a.m.)
> 
> 
> Review request for atlas, David Kantor and Neeru Gupta.
> 
> 
> Bugs: ATLAS-694
> https://issues.apache.org/jira/browse/ATLAS-694
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> ATLAS-694: Update Atlas to use abstraction layer.  All of the Atlas code 
> (with the exception of the catalog, which was only updated minimally) has 
> been updated to use the graph database abstraction layer.  In addition, there 
> is now an optional Atlas configuration property that specifies the class of 
> the abstraction layer database to use.  I basically put all of the changes in 
> here with the exception of the actual Titan 1 implementation of code.  This 
> includes the changes to support Tinkerpop 3 syntax.  This is mostly to 
> expedite getting the changes into Atlas.  Originally the TP3 changes were 
> going to be brought in as part of the Titan 1 implementation task.
> 
> Change Summary:
> 
>- change Atlas classes to use AtlasGraph,AtlasVertex,AtlasEdge, etc 
> instead of TitanGraph/Vertex/Edge, etc
>- compile time dependency on titan 0.5.4/TP 2 removed (except in Catalog, 
> which was only changed to use AtlasGraphProvider/AtlasGraph) - see 
> repository\pom.xml, other pom.xmls
>- updated DSL translation to generate Gremlin that is compliant with TP3 
> when TP3 is being used.  See GremlinQuery.scala, 
> GraphPersistenceStrategies.scala
>- TitanGraphProvider replaced by AtlasGraphProvider.  Graph database 
> implementation is determined from a new optional configuration property
>- HiveTitanSample is no longer used by tests.  It has been replaced by 
> hive-instances.json (which uses normal Atlas json syntax).  The data is saved 
> with a new JSONImporter class.  This was needed because the graphson syntax 
> used by HiveTitanSample is not compatible with TP3.  
> 
> Last rebase: 10/3/2016
> 
> 
> Diffs
> -
> 
>   .gitignore e10adbc4457f6297600f0feb01eb54718b8ec406 
>   addons/falcon-bridge/pom.xml 1365bd05a388dc92f7a56c7f7427b5b85f97c7da 
>   addons/hdfs-model/pom.xml 492f39cea085c6e69781e17bcbdbc3a231806df3 
>   
> addons/hdfs-model/src/test/java/org/apache/atlas/fs/model/HDFSModelTest.java 
> ac60294e328835ba0340e150799ddfb348ccdb52 
>   addons/hive-bridge/pom.xml 6993bdb938a6095ca24482e290393eeeb3911bcb 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
>  ad7a4a5d09d8542a841701dfe04981f65f767c14 
>   addons/sqoop-bridge/pom.xml 8c9d278d43b5979ea1743d10845905c13249f8a6 
>   addons/storm-bridge/pom.xml 12c1208b448d456a923bd7309601174ddb561ba5 
>   catalog/pom.xml 2f58a8f0748de65ab78eab35df6abd2fe7c336af 
>   catalog/src/main/java/org/apache/atlas/catalog/query/BaseQuery.java 
> e7bb505075983371ca12d9bc1d8c6eb240c3d134 
>   distro/src/conf/atlas-application.properties 
> d334600dc5534840409b586157799ef3abf9abf2 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasEdge.java 
> dd4b7e614cdd9bf30f957fb6a839d8c60f3e1701 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasElement.java
>  1bc0fc38c0802897f32260520770a16795474d04 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 
> 995c5457ac7f807172f367cc8e3348b3a98dd6f3 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphIndex.java
>  41194d34f079842db0d95c73a8b099459f76ff2f 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphManagement.java
>  c8cd2842ca3090b6bbd384c773b4eb45aff149ce 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphQuery.java
>  93447495bcf18e9f19df9df68fd1cbe1427fc462 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasIndexQuery.java
>  e719d306ffe9f68e3ac6f7406baaf60a12390c34 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasPropertyKey.java
>  315ecddb861e1a1be6e0ab9b36fe4c0a52486ae8 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasVertex.java
>  fff6fb79247e7d0615ce83c4cbbd93d1bf8cf29c 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasVertexQuery.java
>

Re: Review Request 47810: ATLAS-694: Update Atlas to use Graph DB abstraction layer



> On Oct. 4, 2016, 9 p.m., Suma Shivaprasad wrote:
> > graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphManagement.java,
> >  line 130
> > 
> >
> > Does the index here refer to the edgeLabel we are trying to create an 
> > idex for? Do we have everything we need in this interface to index an edge 
> > label?

I see that the index here refers to the indexName and not the edgeLabel. We 
need to provide for indexing edge labels as well even though we dont use that 
currently


- Suma


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47810/#review151393
---


On Oct. 4, 2016, 1:28 a.m., Jeff Hagelberg wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47810/
> ---
> 
> (Updated Oct. 4, 2016, 1:28 a.m.)
> 
> 
> Review request for atlas, David Kantor and Neeru Gupta.
> 
> 
> Bugs: ATLAS-694
> https://issues.apache.org/jira/browse/ATLAS-694
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> ATLAS-694: Update Atlas to use abstraction layer.  All of the Atlas code 
> (with the exception of the catalog, which was only updated minimally) has 
> been updated to use the graph database abstraction layer.  In addition, there 
> is now an optional Atlas configuration property that specifies the class of 
> the abstraction layer database to use.  I basically put all of the changes in 
> here with the exception of the actual Titan 1 implementation of code.  This 
> includes the changes to support Tinkerpop 3 syntax.  This is mostly to 
> expedite getting the changes into Atlas.  Originally the TP3 changes were 
> going to be brought in as part of the Titan 1 implementation task.
> 
> Change Summary:
> 
>- change Atlas classes to use AtlasGraph,AtlasVertex,AtlasEdge, etc 
> instead of TitanGraph/Vertex/Edge, etc
>- compile time dependency on titan 0.5.4/TP 2 removed (except in Catalog, 
> which was only changed to use AtlasGraphProvider/AtlasGraph) - see 
> repository\pom.xml, other pom.xmls
>- updated DSL translation to generate Gremlin that is compliant with TP3 
> when TP3 is being used.  See GremlinQuery.scala, 
> GraphPersistenceStrategies.scala
>- TitanGraphProvider replaced by AtlasGraphProvider.  Graph database 
> implementation is determined from a new optional configuration property
>- HiveTitanSample is no longer used by tests.  It has been replaced by 
> hive-instances.json (which uses normal Atlas json syntax).  The data is saved 
> with a new JSONImporter class.  This was needed because the graphson syntax 
> used by HiveTitanSample is not compatible with TP3.  
> 
> Last rebase: 10/3/2016
> 
> 
> Diffs
> -
> 
>   .gitignore e10adbc4457f6297600f0feb01eb54718b8ec406 
>   addons/falcon-bridge/pom.xml 1365bd05a388dc92f7a56c7f7427b5b85f97c7da 
>   addons/hdfs-model/pom.xml 492f39cea085c6e69781e17bcbdbc3a231806df3 
>   
> addons/hdfs-model/src/test/java/org/apache/atlas/fs/model/HDFSModelTest.java 
> ac60294e328835ba0340e150799ddfb348ccdb52 
>   addons/hive-bridge/pom.xml 6993bdb938a6095ca24482e290393eeeb3911bcb 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
>  ad7a4a5d09d8542a841701dfe04981f65f767c14 
>   addons/sqoop-bridge/pom.xml 8c9d278d43b5979ea1743d10845905c13249f8a6 
>   addons/storm-bridge/pom.xml 12c1208b448d456a923bd7309601174ddb561ba5 
>   catalog/pom.xml 2f58a8f0748de65ab78eab35df6abd2fe7c336af 
>   catalog/src/main/java/org/apache/atlas/catalog/query/BaseQuery.java 
> e7bb505075983371ca12d9bc1d8c6eb240c3d134 
>   distro/src/conf/atlas-application.properties 
> d334600dc5534840409b586157799ef3abf9abf2 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasEdge.java 
> dd4b7e614cdd9bf30f957fb6a839d8c60f3e1701 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasElement.java
>  1bc0fc38c0802897f32260520770a16795474d04 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 
> 995c5457ac7f807172f367cc8e3348b3a98dd6f3 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphIndex.java
>  41194d34f079842db0d95c73a8b099459f76ff2f 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphManagement.java
>  c8cd2842ca3090b6bbd384c773b4eb45aff149ce 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphQuery.java
>  93447495bcf18e9f19df9df68fd1cbe1427fc462 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasIndexQuery.java
>  e719d306ffe9f68e3ac6f7406baaf60a12390c34 
>   
>

Re: Review Request 47810: ATLAS-694: Update Atlas to use Graph DB abstraction layer


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47810/#review151396
---




graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphManagement.java
 (line 78)


can we change the vararg to a List everywhee to keep it consistent?


- Suma Shivaprasad


On Oct. 4, 2016, 1:28 a.m., Jeff Hagelberg wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47810/
> ---
> 
> (Updated Oct. 4, 2016, 1:28 a.m.)
> 
> 
> Review request for atlas, David Kantor and Neeru Gupta.
> 
> 
> Bugs: ATLAS-694
> https://issues.apache.org/jira/browse/ATLAS-694
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> ATLAS-694: Update Atlas to use abstraction layer.  All of the Atlas code 
> (with the exception of the catalog, which was only updated minimally) has 
> been updated to use the graph database abstraction layer.  In addition, there 
> is now an optional Atlas configuration property that specifies the class of 
> the abstraction layer database to use.  I basically put all of the changes in 
> here with the exception of the actual Titan 1 implementation of code.  This 
> includes the changes to support Tinkerpop 3 syntax.  This is mostly to 
> expedite getting the changes into Atlas.  Originally the TP3 changes were 
> going to be brought in as part of the Titan 1 implementation task.
> 
> Change Summary:
> 
>- change Atlas classes to use AtlasGraph,AtlasVertex,AtlasEdge, etc 
> instead of TitanGraph/Vertex/Edge, etc
>- compile time dependency on titan 0.5.4/TP 2 removed (except in Catalog, 
> which was only changed to use AtlasGraphProvider/AtlasGraph) - see 
> repository\pom.xml, other pom.xmls
>- updated DSL translation to generate Gremlin that is compliant with TP3 
> when TP3 is being used.  See GremlinQuery.scala, 
> GraphPersistenceStrategies.scala
>- TitanGraphProvider replaced by AtlasGraphProvider.  Graph database 
> implementation is determined from a new optional configuration property
>- HiveTitanSample is no longer used by tests.  It has been replaced by 
> hive-instances.json (which uses normal Atlas json syntax).  The data is saved 
> with a new JSONImporter class.  This was needed because the graphson syntax 
> used by HiveTitanSample is not compatible with TP3.  
> 
> Last rebase: 10/3/2016
> 
> 
> Diffs
> -
> 
>   .gitignore e10adbc4457f6297600f0feb01eb54718b8ec406 
>   addons/falcon-bridge/pom.xml 1365bd05a388dc92f7a56c7f7427b5b85f97c7da 
>   addons/hdfs-model/pom.xml 492f39cea085c6e69781e17bcbdbc3a231806df3 
>   
> addons/hdfs-model/src/test/java/org/apache/atlas/fs/model/HDFSModelTest.java 
> ac60294e328835ba0340e150799ddfb348ccdb52 
>   addons/hive-bridge/pom.xml 6993bdb938a6095ca24482e290393eeeb3911bcb 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
>  ad7a4a5d09d8542a841701dfe04981f65f767c14 
>   addons/sqoop-bridge/pom.xml 8c9d278d43b5979ea1743d10845905c13249f8a6 
>   addons/storm-bridge/pom.xml 12c1208b448d456a923bd7309601174ddb561ba5 
>   catalog/pom.xml 2f58a8f0748de65ab78eab35df6abd2fe7c336af 
>   catalog/src/main/java/org/apache/atlas/catalog/query/BaseQuery.java 
> e7bb505075983371ca12d9bc1d8c6eb240c3d134 
>   distro/src/conf/atlas-application.properties 
> d334600dc5534840409b586157799ef3abf9abf2 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasEdge.java 
> dd4b7e614cdd9bf30f957fb6a839d8c60f3e1701 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasElement.java
>  1bc0fc38c0802897f32260520770a16795474d04 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 
> 995c5457ac7f807172f367cc8e3348b3a98dd6f3 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphIndex.java
>  41194d34f079842db0d95c73a8b099459f76ff2f 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphManagement.java
>  c8cd2842ca3090b6bbd384c773b4eb45aff149ce 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphQuery.java
>  93447495bcf18e9f19df9df68fd1cbe1427fc462 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasIndexQuery.java
>  e719d306ffe9f68e3ac6f7406baaf60a12390c34 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasPropertyKey.java
>  315ecddb861e1a1be6e0ab9b36fe4c0a52486ae8 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasVertex.java
>  fff6fb79247e7d0615ce83c4cbbd93d1bf8cf29c 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasVertexQuery.java
>  366cfe131512f36c02f59ad7e969ff94dfb0e12c 
>   
>

Re: Review Request 47810: ATLAS-694: Update Atlas to use Graph DB abstraction layer



> On Sept. 23, 2016, 8:36 p.m., Suma Shivaprasad wrote:
> > graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/Titan0Database.java,
> >  line 188
> > 
> >
> > can rename this to clear instead of removeTestGraph?
> > Also initializeTestGraph can be removed for now?
> 
> Jeff Hagelberg wrote:
> I suppose we could remove it.  I was trying to make the interfaces as 
> close as possible to what we have in our fork, and this method is used by the 
> other implementations.  I guess I missed this comment in my last round of 
> testing and rebasing.  Would cleanupTestGraph be ok as a name?  The issue is 
> that the cleanup mechanism is actually specific to the graph database.  Titan 
> clears the graph.  However, in IBM Graph, clearing the graph is not allowed, 
> so we actually remove it, and initializeTestGraph sets things up so that a 
> new graph gets created for each test class.  Let's see what comments my 
> latest changes get.  I'll put this in while addressing those, assuming there 
> are some :-).
> 
> Jeff Hagelberg wrote:
> or maybe just cleanup()

yes am okay with cleanup. Could we have an initiliaze method for all graphs 
instead that can be used for tests to setup the graph and for non-tests/prod 
deployments may be empty impl if nothing is required?


- Suma


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47810/#review150231
---


On Oct. 4, 2016, 1:28 a.m., Jeff Hagelberg wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47810/
> ---
> 
> (Updated Oct. 4, 2016, 1:28 a.m.)
> 
> 
> Review request for atlas, David Kantor and Neeru Gupta.
> 
> 
> Bugs: ATLAS-694
> https://issues.apache.org/jira/browse/ATLAS-694
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> ATLAS-694: Update Atlas to use abstraction layer.  All of the Atlas code 
> (with the exception of the catalog, which was only updated minimally) has 
> been updated to use the graph database abstraction layer.  In addition, there 
> is now an optional Atlas configuration property that specifies the class of 
> the abstraction layer database to use.  I basically put all of the changes in 
> here with the exception of the actual Titan 1 implementation of code.  This 
> includes the changes to support Tinkerpop 3 syntax.  This is mostly to 
> expedite getting the changes into Atlas.  Originally the TP3 changes were 
> going to be brought in as part of the Titan 1 implementation task.
> 
> Change Summary:
> 
>- change Atlas classes to use AtlasGraph,AtlasVertex,AtlasEdge, etc 
> instead of TitanGraph/Vertex/Edge, etc
>- compile time dependency on titan 0.5.4/TP 2 removed (except in Catalog, 
> which was only changed to use AtlasGraphProvider/AtlasGraph) - see 
> repository\pom.xml, other pom.xmls
>- updated DSL translation to generate Gremlin that is compliant with TP3 
> when TP3 is being used.  See GremlinQuery.scala, 
> GraphPersistenceStrategies.scala
>- TitanGraphProvider replaced by AtlasGraphProvider.  Graph database 
> implementation is determined from a new optional configuration property
>- HiveTitanSample is no longer used by tests.  It has been replaced by 
> hive-instances.json (which uses normal Atlas json syntax).  The data is saved 
> with a new JSONImporter class.  This was needed because the graphson syntax 
> used by HiveTitanSample is not compatible with TP3.  
> 
> Last rebase: 10/3/2016
> 
> 
> Diffs
> -
> 
>   .gitignore e10adbc4457f6297600f0feb01eb54718b8ec406 
>   addons/falcon-bridge/pom.xml 1365bd05a388dc92f7a56c7f7427b5b85f97c7da 
>   addons/hdfs-model/pom.xml 492f39cea085c6e69781e17bcbdbc3a231806df3 
>   
> addons/hdfs-model/src/test/java/org/apache/atlas/fs/model/HDFSModelTest.java 
> ac60294e328835ba0340e150799ddfb348ccdb52 
>   addons/hive-bridge/pom.xml 6993bdb938a6095ca24482e290393eeeb3911bcb 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
>  ad7a4a5d09d8542a841701dfe04981f65f767c14 
>   addons/sqoop-bridge/pom.xml 8c9d278d43b5979ea1743d10845905c13249f8a6 
>   addons/storm-bridge/pom.xml 12c1208b448d456a923bd7309601174ddb561ba5 
>   catalog/pom.xml 2f58a8f0748de65ab78eab35df6abd2fe7c336af 
>   catalog/src/main/java/org/apache/atlas/catalog/query/BaseQuery.java 
> e7bb505075983371ca12d9bc1d8c6eb240c3d134 
>   distro/src/conf/atlas-application.properties 
> d334600dc5534840409b586157799ef3abf9abf2 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasEdge.java 
> dd4b7e614cdd9bf30f957fb6a839d8c60f3e1701 
>   
>

Re: Review Request 47810: ATLAS-694: Update Atlas to use Graph DB abstraction layer


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47810/#review151393
---




graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphManagement.java
 (line 111)


Does the index here refer to the edgeLabel we are trying to create an idex 
for? Do we have everything we need in this interface to index an edge label?


- Suma Shivaprasad


On Oct. 4, 2016, 1:28 a.m., Jeff Hagelberg wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47810/
> ---
> 
> (Updated Oct. 4, 2016, 1:28 a.m.)
> 
> 
> Review request for atlas, David Kantor and Neeru Gupta.
> 
> 
> Bugs: ATLAS-694
> https://issues.apache.org/jira/browse/ATLAS-694
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> ATLAS-694: Update Atlas to use abstraction layer.  All of the Atlas code 
> (with the exception of the catalog, which was only updated minimally) has 
> been updated to use the graph database abstraction layer.  In addition, there 
> is now an optional Atlas configuration property that specifies the class of 
> the abstraction layer database to use.  I basically put all of the changes in 
> here with the exception of the actual Titan 1 implementation of code.  This 
> includes the changes to support Tinkerpop 3 syntax.  This is mostly to 
> expedite getting the changes into Atlas.  Originally the TP3 changes were 
> going to be brought in as part of the Titan 1 implementation task.
> 
> Change Summary:
> 
>- change Atlas classes to use AtlasGraph,AtlasVertex,AtlasEdge, etc 
> instead of TitanGraph/Vertex/Edge, etc
>- compile time dependency on titan 0.5.4/TP 2 removed (except in Catalog, 
> which was only changed to use AtlasGraphProvider/AtlasGraph) - see 
> repository\pom.xml, other pom.xmls
>- updated DSL translation to generate Gremlin that is compliant with TP3 
> when TP3 is being used.  See GremlinQuery.scala, 
> GraphPersistenceStrategies.scala
>- TitanGraphProvider replaced by AtlasGraphProvider.  Graph database 
> implementation is determined from a new optional configuration property
>- HiveTitanSample is no longer used by tests.  It has been replaced by 
> hive-instances.json (which uses normal Atlas json syntax).  The data is saved 
> with a new JSONImporter class.  This was needed because the graphson syntax 
> used by HiveTitanSample is not compatible with TP3.  
> 
> Last rebase: 10/3/2016
> 
> 
> Diffs
> -
> 
>   .gitignore e10adbc4457f6297600f0feb01eb54718b8ec406 
>   addons/falcon-bridge/pom.xml 1365bd05a388dc92f7a56c7f7427b5b85f97c7da 
>   addons/hdfs-model/pom.xml 492f39cea085c6e69781e17bcbdbc3a231806df3 
>   
> addons/hdfs-model/src/test/java/org/apache/atlas/fs/model/HDFSModelTest.java 
> ac60294e328835ba0340e150799ddfb348ccdb52 
>   addons/hive-bridge/pom.xml 6993bdb938a6095ca24482e290393eeeb3911bcb 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
>  ad7a4a5d09d8542a841701dfe04981f65f767c14 
>   addons/sqoop-bridge/pom.xml 8c9d278d43b5979ea1743d10845905c13249f8a6 
>   addons/storm-bridge/pom.xml 12c1208b448d456a923bd7309601174ddb561ba5 
>   catalog/pom.xml 2f58a8f0748de65ab78eab35df6abd2fe7c336af 
>   catalog/src/main/java/org/apache/atlas/catalog/query/BaseQuery.java 
> e7bb505075983371ca12d9bc1d8c6eb240c3d134 
>   distro/src/conf/atlas-application.properties 
> d334600dc5534840409b586157799ef3abf9abf2 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasEdge.java 
> dd4b7e614cdd9bf30f957fb6a839d8c60f3e1701 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasElement.java
>  1bc0fc38c0802897f32260520770a16795474d04 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 
> 995c5457ac7f807172f367cc8e3348b3a98dd6f3 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphIndex.java
>  41194d34f079842db0d95c73a8b099459f76ff2f 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphManagement.java
>  c8cd2842ca3090b6bbd384c773b4eb45aff149ce 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphQuery.java
>  93447495bcf18e9f19df9df68fd1cbe1427fc462 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasIndexQuery.java
>  e719d306ffe9f68e3ac6f7406baaf60a12390c34 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasPropertyKey.java
>  315ecddb861e1a1be6e0ab9b36fe4c0a52486ae8 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasVertex.java
>  fff6fb79247e7d0615ce83c4cbbd93d1bf8cf29c 
>   
>

Re: Review Request 47810: ATLAS-694: Update Atlas to use Graph DB abstraction layer



> On Sept. 23, 2016, 8:36 p.m., Suma Shivaprasad wrote:
> > graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/Titan0Database.java,
> >  line 188
> > 
> >
> > can rename this to clear instead of removeTestGraph?
> > Also initializeTestGraph can be removed for now?
> 
> Jeff Hagelberg wrote:
> I suppose we could remove it.  I was trying to make the interfaces as 
> close as possible to what we have in our fork, and this method is used by the 
> other implementations.  I guess I missed this comment in my last round of 
> testing and rebasing.  Would cleanupTestGraph be ok as a name?  The issue is 
> that the cleanup mechanism is actually specific to the graph database.  Titan 
> clears the graph.  However, in IBM Graph, clearing the graph is not allowed, 
> so we actually remove it, and initializeTestGraph sets things up so that a 
> new graph gets created for each test class.  Let's see what comments my 
> latest changes get.  I'll put this in while addressing those, assuming there 
> are some :-).

or maybe just cleanup()


- Jeff


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47810/#review150231
---


On Oct. 4, 2016, 1:28 a.m., Jeff Hagelberg wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47810/
> ---
> 
> (Updated Oct. 4, 2016, 1:28 a.m.)
> 
> 
> Review request for atlas, David Kantor and Neeru Gupta.
> 
> 
> Bugs: ATLAS-694
> https://issues.apache.org/jira/browse/ATLAS-694
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> ATLAS-694: Update Atlas to use abstraction layer.  All of the Atlas code 
> (with the exception of the catalog, which was only updated minimally) has 
> been updated to use the graph database abstraction layer.  In addition, there 
> is now an optional Atlas configuration property that specifies the class of 
> the abstraction layer database to use.  I basically put all of the changes in 
> here with the exception of the actual Titan 1 implementation of code.  This 
> includes the changes to support Tinkerpop 3 syntax.  This is mostly to 
> expedite getting the changes into Atlas.  Originally the TP3 changes were 
> going to be brought in as part of the Titan 1 implementation task.
> 
> Change Summary:
> 
>- change Atlas classes to use AtlasGraph,AtlasVertex,AtlasEdge, etc 
> instead of TitanGraph/Vertex/Edge, etc
>- compile time dependency on titan 0.5.4/TP 2 removed (except in Catalog, 
> which was only changed to use AtlasGraphProvider/AtlasGraph) - see 
> repository\pom.xml, other pom.xmls
>- updated DSL translation to generate Gremlin that is compliant with TP3 
> when TP3 is being used.  See GremlinQuery.scala, 
> GraphPersistenceStrategies.scala
>- TitanGraphProvider replaced by AtlasGraphProvider.  Graph database 
> implementation is determined from a new optional configuration property
>- HiveTitanSample is no longer used by tests.  It has been replaced by 
> hive-instances.json (which uses normal Atlas json syntax).  The data is saved 
> with a new JSONImporter class.  This was needed because the graphson syntax 
> used by HiveTitanSample is not compatible with TP3.  
> 
> Last rebase: 10/3/2016
> 
> 
> Diffs
> -
> 
>   .gitignore e10adbc4457f6297600f0feb01eb54718b8ec406 
>   addons/falcon-bridge/pom.xml 1365bd05a388dc92f7a56c7f7427b5b85f97c7da 
>   addons/hdfs-model/pom.xml 492f39cea085c6e69781e17bcbdbc3a231806df3 
>   
> addons/hdfs-model/src/test/java/org/apache/atlas/fs/model/HDFSModelTest.java 
> ac60294e328835ba0340e150799ddfb348ccdb52 
>   addons/hive-bridge/pom.xml 6993bdb938a6095ca24482e290393eeeb3911bcb 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
>  ad7a4a5d09d8542a841701dfe04981f65f767c14 
>   addons/sqoop-bridge/pom.xml 8c9d278d43b5979ea1743d10845905c13249f8a6 
>   addons/storm-bridge/pom.xml 12c1208b448d456a923bd7309601174ddb561ba5 
>   catalog/pom.xml 2f58a8f0748de65ab78eab35df6abd2fe7c336af 
>   catalog/src/main/java/org/apache/atlas/catalog/query/BaseQuery.java 
> e7bb505075983371ca12d9bc1d8c6eb240c3d134 
>   distro/src/conf/atlas-application.properties 
> d334600dc5534840409b586157799ef3abf9abf2 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasEdge.java 
> dd4b7e614cdd9bf30f957fb6a839d8c60f3e1701 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasElement.java
>  1bc0fc38c0802897f32260520770a16795474d04 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 
> 995c5457ac7f807172f367cc8e3348b3a98dd6f3 
>   
>

Re: Review Request 47810: ATLAS-694: Update Atlas to use Graph DB abstraction layer



> On Sept. 23, 2016, 8:36 p.m., Suma Shivaprasad wrote:
> > graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/Titan0Database.java,
> >  line 188
> > 
> >
> > can rename this to clear instead of removeTestGraph?
> > Also initializeTestGraph can be removed for now?

I suppose we could remove it.  I was trying to make the interfaces as close as 
possible to what we have in our fork, and this method is used by the other 
implementations.  I guess I missed this comment in my last round of testing and 
rebasing.  Would cleanupTestGraph be ok as a name?  The issue is that the 
cleanup mechanism is actually specific to the graph database.  Titan clears the 
graph.  However, in IBM Graph, clearing the graph is not allowed, so we 
actually remove it, and initializeTestGraph sets things up so that a new graph 
gets created for each test class.  Let's see what comments my latest changes 
get.  I'll put this in while addressing those, assuming there are some :-).


- Jeff


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47810/#review150231
---


On Oct. 4, 2016, 1:28 a.m., Jeff Hagelberg wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47810/
> ---
> 
> (Updated Oct. 4, 2016, 1:28 a.m.)
> 
> 
> Review request for atlas, David Kantor and Neeru Gupta.
> 
> 
> Bugs: ATLAS-694
> https://issues.apache.org/jira/browse/ATLAS-694
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> ATLAS-694: Update Atlas to use abstraction layer.  All of the Atlas code 
> (with the exception of the catalog, which was only updated minimally) has 
> been updated to use the graph database abstraction layer.  In addition, there 
> is now an optional Atlas configuration property that specifies the class of 
> the abstraction layer database to use.  I basically put all of the changes in 
> here with the exception of the actual Titan 1 implementation of code.  This 
> includes the changes to support Tinkerpop 3 syntax.  This is mostly to 
> expedite getting the changes into Atlas.  Originally the TP3 changes were 
> going to be brought in as part of the Titan 1 implementation task.
> 
> Change Summary:
> 
>- change Atlas classes to use AtlasGraph,AtlasVertex,AtlasEdge, etc 
> instead of TitanGraph/Vertex/Edge, etc
>- compile time dependency on titan 0.5.4/TP 2 removed (except in Catalog, 
> which was only changed to use AtlasGraphProvider/AtlasGraph) - see 
> repository\pom.xml, other pom.xmls
>- updated DSL translation to generate Gremlin that is compliant with TP3 
> when TP3 is being used.  See GremlinQuery.scala, 
> GraphPersistenceStrategies.scala
>- TitanGraphProvider replaced by AtlasGraphProvider.  Graph database 
> implementation is determined from a new optional configuration property
>- HiveTitanSample is no longer used by tests.  It has been replaced by 
> hive-instances.json (which uses normal Atlas json syntax).  The data is saved 
> with a new JSONImporter class.  This was needed because the graphson syntax 
> used by HiveTitanSample is not compatible with TP3.  
> 
> Last rebase: 10/3/2016
> 
> 
> Diffs
> -
> 
>   .gitignore e10adbc4457f6297600f0feb01eb54718b8ec406 
>   addons/falcon-bridge/pom.xml 1365bd05a388dc92f7a56c7f7427b5b85f97c7da 
>   addons/hdfs-model/pom.xml 492f39cea085c6e69781e17bcbdbc3a231806df3 
>   
> addons/hdfs-model/src/test/java/org/apache/atlas/fs/model/HDFSModelTest.java 
> ac60294e328835ba0340e150799ddfb348ccdb52 
>   addons/hive-bridge/pom.xml 6993bdb938a6095ca24482e290393eeeb3911bcb 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
>  ad7a4a5d09d8542a841701dfe04981f65f767c14 
>   addons/sqoop-bridge/pom.xml 8c9d278d43b5979ea1743d10845905c13249f8a6 
>   addons/storm-bridge/pom.xml 12c1208b448d456a923bd7309601174ddb561ba5 
>   catalog/pom.xml 2f58a8f0748de65ab78eab35df6abd2fe7c336af 
>   catalog/src/main/java/org/apache/atlas/catalog/query/BaseQuery.java 
> e7bb505075983371ca12d9bc1d8c6eb240c3d134 
>   distro/src/conf/atlas-application.properties 
> d334600dc5534840409b586157799ef3abf9abf2 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasEdge.java 
> dd4b7e614cdd9bf30f957fb6a839d8c60f3e1701 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasElement.java
>  1bc0fc38c0802897f32260520770a16795474d04 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 
> 995c5457ac7f807172f367cc8e3348b3a98dd6f3 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphIndex.java
>

Re: Review Request 47810: ATLAS-694: Update Atlas to use Graph DB abstraction layer


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47810/
---

(Updated Oct. 4, 2016, 1:28 a.m.)


Review request for atlas, David Kantor and Neeru Gupta.


Bugs: ATLAS-694
https://issues.apache.org/jira/browse/ATLAS-694


Repository: atlas


Description (updated)
---

ATLAS-694: Update Atlas to use abstraction layer.  All of the Atlas code (with 
the exception of the catalog, which was only updated minimally) has been 
updated to use the graph database abstraction layer.  In addition, there is now 
an optional Atlas configuration property that specifies the class of the 
abstraction layer database to use.  I basically put all of the changes in here 
with the exception of the actual Titan 1 implementation of code.  This includes 
the changes to support Tinkerpop 3 syntax.  This is mostly to expedite getting 
the changes into Atlas.  Originally the TP3 changes were going to be brought in 
as part of the Titan 1 implementation task.

Change Summary:

   - change Atlas classes to use AtlasGraph,AtlasVertex,AtlasEdge, etc instead 
of TitanGraph/Vertex/Edge, etc
   - compile time dependency on titan 0.5.4/TP 2 removed (except in Catalog, 
which was only changed to use AtlasGraphProvider/AtlasGraph) - see 
repository\pom.xml, other pom.xmls
   - updated DSL translation to generate Gremlin that is compliant with TP3 
when TP3 is being used.  See GremlinQuery.scala, 
GraphPersistenceStrategies.scala
   - TitanGraphProvider replaced by AtlasGraphProvider.  Graph database 
implementation is determined from a new optional configuration property
   - HiveTitanSample is no longer used by tests.  It has been replaced by 
hive-instances.json (which uses normal Atlas json syntax).  The data is saved 
with a new JSONImporter class.  This was needed because the graphson syntax 
used by HiveTitanSample is not compatible with TP3.  

Last rebase: 10/3/2016


Diffs
-

  .gitignore e10adbc4457f6297600f0feb01eb54718b8ec406 
  addons/falcon-bridge/pom.xml 1365bd05a388dc92f7a56c7f7427b5b85f97c7da 
  addons/hdfs-model/pom.xml 492f39cea085c6e69781e17bcbdbc3a231806df3 
  addons/hdfs-model/src/test/java/org/apache/atlas/fs/model/HDFSModelTest.java 
ac60294e328835ba0340e150799ddfb348ccdb52 
  addons/hive-bridge/pom.xml 6993bdb938a6095ca24482e290393eeeb3911bcb 
  
addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
 ad7a4a5d09d8542a841701dfe04981f65f767c14 
  addons/sqoop-bridge/pom.xml 8c9d278d43b5979ea1743d10845905c13249f8a6 
  addons/storm-bridge/pom.xml 12c1208b448d456a923bd7309601174ddb561ba5 
  catalog/pom.xml 2f58a8f0748de65ab78eab35df6abd2fe7c336af 
  catalog/src/main/java/org/apache/atlas/catalog/query/BaseQuery.java 
e7bb505075983371ca12d9bc1d8c6eb240c3d134 
  distro/src/conf/atlas-application.properties 
d334600dc5534840409b586157799ef3abf9abf2 
  graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasEdge.java 
dd4b7e614cdd9bf30f957fb6a839d8c60f3e1701 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasElement.java 
1bc0fc38c0802897f32260520770a16795474d04 
  graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 
995c5457ac7f807172f367cc8e3348b3a98dd6f3 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphIndex.java
 41194d34f079842db0d95c73a8b099459f76ff2f 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphManagement.java
 c8cd2842ca3090b6bbd384c773b4eb45aff149ce 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphQuery.java
 93447495bcf18e9f19df9df68fd1cbe1427fc462 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasIndexQuery.java
 e719d306ffe9f68e3ac6f7406baaf60a12390c34 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasPropertyKey.java
 315ecddb861e1a1be6e0ab9b36fe4c0a52486ae8 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasVertex.java 
fff6fb79247e7d0615ce83c4cbbd93d1bf8cf29c 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasVertexQuery.java
 366cfe131512f36c02f59ad7e969ff94dfb0e12c 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/GraphDatabase.java
 37acda5f6ddb883ee364d08a31251b071e6a89a5 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/GremlinVersion.java
 f328d3cb9d2b695bf0eb9981d7046bb67d0eac7d 
  
graphdb/common/src/main/java/org/apache/atlas/repository/graphdb/titan/query/NativeTitanGraphQuery.java
 b79bf7964ae9c830cfa28b9f2f1a1a1059d67223 
  
graphdb/common/src/main/java/org/apache/atlas/repository/graphdb/titan/query/NativeTitanQueryFactory.java
 5c654119a4292136e8ff935c34dd6e39741bffc1 
  
graphdb/common/src/main/java/org/apache/atlas/repository/graphdb/titan/query/TitanGraphQuery.java
 f23847a2f75c351ad307680d0642bc1da49fc59f

Re: Review Request 47810: ATLAS-694: Update Atlas to use Graph DB abstraction layer


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47810/
---

(Updated Oct. 4, 2016, 1:22 a.m.)


Review request for atlas, David Kantor and Neeru Gupta.


Changes
---

Addressed review comments.  Changed hive instance sample to programatically 
create entities.  Refactored AtlasGraphManagement, GraphBackedSearchIndexer to 
make methods less titan-centric.  Rebased.


Bugs: ATLAS-694
https://issues.apache.org/jira/browse/ATLAS-694


Repository: atlas


Description
---

ATLAS-694: Update Atlas to use abstraction layer.  All of the Atlas code (with 
the exception of the catalog, which was only updated minimally) has been 
updated to use the graph database abstraction layer.  In addition, there is now 
an optional Atlas configuration property that specifies the class of the 
abstraction layer database to use.  I basically put all of the changes in here 
with the exception of the actual Titan 1 implementation of code.  This includes 
the changes to support Tinkerpop 3 syntax.  This is mostly to expedite getting 
the changes into Atlas.  Originally the TP3 changes were going to be brought in 
as part of the Titan 1 implementation task.

Change Summary:

   - change Atlas classes to use AtlasGraph,AtlasVertex,AtlasEdge, etc instead 
of TitanGraph/Vertex/Edge, etc
   - compile time dependency on titan 0.5.4/TP 2 removed (except in Catalog, 
which was only changed to use AtlasGraphProvider/AtlasGraph) - see 
repository\pom.xml, other pom.xmls
   - updated DSL translation to generate Gremlin that is compliant with TP3 
when TP3 is being used.  See GremlinQuery.scala, 
GraphPersistenceStrategies.scala
   - TitanGraphProvider replaced by AtlasGraphProvider.  Graph database 
implementation is determined from a new optional configuration property
   - HiveTitanSample is no longer used by tests.  It has been replaced by 
hive-instances.json (which uses normal Atlas json syntax).  The data is saved 
with a new JSONImporter class.  This was needed because the graphson syntax 
used by HiveTitanSample is not compatible with TP3.  

Last rebase: 9/22/2016


Diffs (updated)
-

  .gitignore e10adbc4457f6297600f0feb01eb54718b8ec406 
  addons/falcon-bridge/pom.xml 1365bd05a388dc92f7a56c7f7427b5b85f97c7da 
  addons/hdfs-model/pom.xml 492f39cea085c6e69781e17bcbdbc3a231806df3 
  addons/hdfs-model/src/test/java/org/apache/atlas/fs/model/HDFSModelTest.java 
ac60294e328835ba0340e150799ddfb348ccdb52 
  addons/hive-bridge/pom.xml 6993bdb938a6095ca24482e290393eeeb3911bcb 
  
addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
 ad7a4a5d09d8542a841701dfe04981f65f767c14 
  addons/sqoop-bridge/pom.xml 8c9d278d43b5979ea1743d10845905c13249f8a6 
  addons/storm-bridge/pom.xml 12c1208b448d456a923bd7309601174ddb561ba5 
  catalog/pom.xml 2f58a8f0748de65ab78eab35df6abd2fe7c336af 
  catalog/src/main/java/org/apache/atlas/catalog/query/BaseQuery.java 
e7bb505075983371ca12d9bc1d8c6eb240c3d134 
  distro/src/conf/atlas-application.properties 
d334600dc5534840409b586157799ef3abf9abf2 
  graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasEdge.java 
dd4b7e614cdd9bf30f957fb6a839d8c60f3e1701 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasElement.java 
1bc0fc38c0802897f32260520770a16795474d04 
  graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 
995c5457ac7f807172f367cc8e3348b3a98dd6f3 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphIndex.java
 41194d34f079842db0d95c73a8b099459f76ff2f 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphManagement.java
 c8cd2842ca3090b6bbd384c773b4eb45aff149ce 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphQuery.java
 93447495bcf18e9f19df9df68fd1cbe1427fc462 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasIndexQuery.java
 e719d306ffe9f68e3ac6f7406baaf60a12390c34 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasPropertyKey.java
 315ecddb861e1a1be6e0ab9b36fe4c0a52486ae8 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasVertex.java 
fff6fb79247e7d0615ce83c4cbbd93d1bf8cf29c 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasVertexQuery.java
 366cfe131512f36c02f59ad7e969ff94dfb0e12c 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/GraphDatabase.java
 37acda5f6ddb883ee364d08a31251b071e6a89a5 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/GremlinVersion.java
 f328d3cb9d2b695bf0eb9981d7046bb67d0eac7d 
  
graphdb/common/src/main/java/org/apache/atlas/repository/graphdb/titan/query/NativeTitanGraphQuery.java
 b79bf7964ae9c830cfa28b9f2f1a1a1059d67223 
  
graphdb/common/src/main/java/org/apache/atlas/repository/graphdb/titan/query/NativeTitanQueryFactory.java

Re: Review Request 47810: ATLAS-694: Update Atlas to use Graph DB abstraction layer

2016-09-28 Thread Shwetha GS


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47810/#review150691
---




repository/src/test/resources/hive-instances.json (line 2)


Modifying this json because of changes in model, 
serialisation/deserialisation is tough. Can we just generate the entities from 
test code?


- Shwetha GS


On Sept. 23, 2016, 3:29 a.m., Jeff Hagelberg wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47810/
> ---
> 
> (Updated Sept. 23, 2016, 3:29 a.m.)
> 
> 
> Review request for atlas, David Kantor and Neeru Gupta.
> 
> 
> Bugs: ATLAS-694
> https://issues.apache.org/jira/browse/ATLAS-694
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> ATLAS-694: Update Atlas to use abstraction layer.  All of the Atlas code 
> (with the exception of the catalog, which was only updated minimally) has 
> been updated to use the graph database abstraction layer.  In addition, there 
> is now an optional Atlas configuration property that specifies the class of 
> the abstraction layer database to use.  I basically put all of the changes in 
> here with the exception of the actual Titan 1 implementation of code.  This 
> includes the changes to support Tinkerpop 3 syntax.  This is mostly to 
> expedite getting the changes into Atlas.  Originally the TP3 changes were 
> going to be brought in as part of the Titan 1 implementation task.
> 
> Change Summary:
> 
>- change Atlas classes to use AtlasGraph,AtlasVertex,AtlasEdge, etc 
> instead of TitanGraph/Vertex/Edge, etc
>- compile time dependency on titan 0.5.4/TP 2 removed (except in Catalog, 
> which was only changed to use AtlasGraphProvider/AtlasGraph) - see 
> repository\pom.xml, other pom.xmls
>- updated DSL translation to generate Gremlin that is compliant with TP3 
> when TP3 is being used.  See GremlinQuery.scala, 
> GraphPersistenceStrategies.scala
>- TitanGraphProvider replaced by AtlasGraphProvider.  Graph database 
> implementation is determined from a new optional configuration property
>- HiveTitanSample is no longer used by tests.  It has been replaced by 
> hive-instances.json (which uses normal Atlas json syntax).  The data is saved 
> with a new JSONImporter class.  This was needed because the graphson syntax 
> used by HiveTitanSample is not compatible with TP3.  
> 
> Last rebase: 9/22/2016
> 
> 
> Diffs
> -
> 
>   .gitignore e10adbc4457f6297600f0feb01eb54718b8ec406 
>   addons/falcon-bridge/pom.xml 1365bd05a388dc92f7a56c7f7427b5b85f97c7da 
>   addons/hdfs-model/pom.xml 492f39cea085c6e69781e17bcbdbc3a231806df3 
>   
> addons/hdfs-model/src/test/java/org/apache/atlas/fs/model/HDFSModelTest.java 
> ac60294e328835ba0340e150799ddfb348ccdb52 
>   addons/hive-bridge/pom.xml 6993bdb938a6095ca24482e290393eeeb3911bcb 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
>  ad7a4a5d09d8542a841701dfe04981f65f767c14 
>   addons/sqoop-bridge/pom.xml 8c9d278d43b5979ea1743d10845905c13249f8a6 
>   addons/storm-bridge/pom.xml 12c1208b448d456a923bd7309601174ddb561ba5 
>   catalog/pom.xml 2f58a8f0748de65ab78eab35df6abd2fe7c336af 
>   catalog/src/main/java/org/apache/atlas/catalog/query/BaseQuery.java 
> e7bb505075983371ca12d9bc1d8c6eb240c3d134 
>   distro/src/conf/atlas-application.properties 
> d334600dc5534840409b586157799ef3abf9abf2 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasEdge.java 
> dd4b7e614cdd9bf30f957fb6a839d8c60f3e1701 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasElement.java
>  1bc0fc38c0802897f32260520770a16795474d04 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 
> 995c5457ac7f807172f367cc8e3348b3a98dd6f3 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphIndex.java
>  41194d34f079842db0d95c73a8b099459f76ff2f 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphManagement.java
>  c8cd2842ca3090b6bbd384c773b4eb45aff149ce 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphQuery.java
>  93447495bcf18e9f19df9df68fd1cbe1427fc462 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasIndexQuery.java
>  e719d306ffe9f68e3ac6f7406baaf60a12390c34 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasPropertyKey.java
>  315ecddb861e1a1be6e0ab9b36fe4c0a52486ae8 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasVertex.java
>  fff6fb79247e7d0615ce83c4cbbd93d1bf8cf29c 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasVertexQuery.java
>

Re: Review Request 47810: ATLAS-694: Update Atlas to use Graph DB abstraction layer

2016-09-26 Thread Jeff Hagelberg



> On Sept. 23, 2016, 8:13 p.m., Suma Shivaprasad wrote:
> > repository/src/main/scala/org/apache/atlas/query/GremlinQuery.scala, line 
> > 575
> > 
> >
> > Is this change, fixing a bug in current code? if so, can you pls 
> > explain?

This isn't really a bug fix.  It is part of a performance optimization that was 
put in for IBM Graph support.  For that implementation, it replaces all 
vertices in the query result with a structure that contains both the vertex id 
and the list of outgoing edges for the vertex.  Previous to that, only the id 
was coming back, and we had to make additional REST api calls to get the other 
information about the vertices.  For titan0/titan1, 
getOutputTransformationPredicate() returns an empty String.


- Jeff


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47810/#review150230
---


On Sept. 23, 2016, 3:29 a.m., Jeff Hagelberg wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47810/
> ---
> 
> (Updated Sept. 23, 2016, 3:29 a.m.)
> 
> 
> Review request for atlas, David Kantor and Neeru Gupta.
> 
> 
> Bugs: ATLAS-694
> https://issues.apache.org/jira/browse/ATLAS-694
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> ATLAS-694: Update Atlas to use abstraction layer.  All of the Atlas code 
> (with the exception of the catalog, which was only updated minimally) has 
> been updated to use the graph database abstraction layer.  In addition, there 
> is now an optional Atlas configuration property that specifies the class of 
> the abstraction layer database to use.  I basically put all of the changes in 
> here with the exception of the actual Titan 1 implementation of code.  This 
> includes the changes to support Tinkerpop 3 syntax.  This is mostly to 
> expedite getting the changes into Atlas.  Originally the TP3 changes were 
> going to be brought in as part of the Titan 1 implementation task.
> 
> Change Summary:
> 
>- change Atlas classes to use AtlasGraph,AtlasVertex,AtlasEdge, etc 
> instead of TitanGraph/Vertex/Edge, etc
>- compile time dependency on titan 0.5.4/TP 2 removed (except in Catalog, 
> which was only changed to use AtlasGraphProvider/AtlasGraph) - see 
> repository\pom.xml, other pom.xmls
>- updated DSL translation to generate Gremlin that is compliant with TP3 
> when TP3 is being used.  See GremlinQuery.scala, 
> GraphPersistenceStrategies.scala
>- TitanGraphProvider replaced by AtlasGraphProvider.  Graph database 
> implementation is determined from a new optional configuration property
>- HiveTitanSample is no longer used by tests.  It has been replaced by 
> hive-instances.json (which uses normal Atlas json syntax).  The data is saved 
> with a new JSONImporter class.  This was needed because the graphson syntax 
> used by HiveTitanSample is not compatible with TP3.  
> 
> Last rebase: 9/22/2016
> 
> 
> Diffs
> -
> 
>   .gitignore e10adbc4457f6297600f0feb01eb54718b8ec406 
>   addons/falcon-bridge/pom.xml 1365bd05a388dc92f7a56c7f7427b5b85f97c7da 
>   addons/hdfs-model/pom.xml 492f39cea085c6e69781e17bcbdbc3a231806df3 
>   
> addons/hdfs-model/src/test/java/org/apache/atlas/fs/model/HDFSModelTest.java 
> ac60294e328835ba0340e150799ddfb348ccdb52 
>   addons/hive-bridge/pom.xml 6993bdb938a6095ca24482e290393eeeb3911bcb 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
>  ad7a4a5d09d8542a841701dfe04981f65f767c14 
>   addons/sqoop-bridge/pom.xml 8c9d278d43b5979ea1743d10845905c13249f8a6 
>   addons/storm-bridge/pom.xml 12c1208b448d456a923bd7309601174ddb561ba5 
>   catalog/pom.xml 2f58a8f0748de65ab78eab35df6abd2fe7c336af 
>   catalog/src/main/java/org/apache/atlas/catalog/query/BaseQuery.java 
> e7bb505075983371ca12d9bc1d8c6eb240c3d134 
>   distro/src/conf/atlas-application.properties 
> d334600dc5534840409b586157799ef3abf9abf2 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasEdge.java 
> dd4b7e614cdd9bf30f957fb6a839d8c60f3e1701 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasElement.java
>  1bc0fc38c0802897f32260520770a16795474d04 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 
> 995c5457ac7f807172f367cc8e3348b3a98dd6f3 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphIndex.java
>  41194d34f079842db0d95c73a8b099459f76ff2f 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphManagement.java
>  c8cd2842ca3090b6bbd384c773b4eb45aff149ce 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphQuery.java
>

Re: Review Request 47810: ATLAS-694: Update Atlas to use Graph DB abstraction layer

2016-09-26 Thread Jeff Hagelberg



> On Sept. 23, 2016, 7:58 p.m., Suma Shivaprasad wrote:
> > repository/src/main/scala/org/apache/atlas/query/GremlinQuery.scala, line 
> > 341
> > 
> >
> > Can you pls raise a jira for this?

I have created ATLAS-1196 for this.


- Jeff


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47810/#review150229
---


On Sept. 23, 2016, 3:29 a.m., Jeff Hagelberg wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47810/
> ---
> 
> (Updated Sept. 23, 2016, 3:29 a.m.)
> 
> 
> Review request for atlas, David Kantor and Neeru Gupta.
> 
> 
> Bugs: ATLAS-694
> https://issues.apache.org/jira/browse/ATLAS-694
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> ATLAS-694: Update Atlas to use abstraction layer.  All of the Atlas code 
> (with the exception of the catalog, which was only updated minimally) has 
> been updated to use the graph database abstraction layer.  In addition, there 
> is now an optional Atlas configuration property that specifies the class of 
> the abstraction layer database to use.  I basically put all of the changes in 
> here with the exception of the actual Titan 1 implementation of code.  This 
> includes the changes to support Tinkerpop 3 syntax.  This is mostly to 
> expedite getting the changes into Atlas.  Originally the TP3 changes were 
> going to be brought in as part of the Titan 1 implementation task.
> 
> Change Summary:
> 
>- change Atlas classes to use AtlasGraph,AtlasVertex,AtlasEdge, etc 
> instead of TitanGraph/Vertex/Edge, etc
>- compile time dependency on titan 0.5.4/TP 2 removed (except in Catalog, 
> which was only changed to use AtlasGraphProvider/AtlasGraph) - see 
> repository\pom.xml, other pom.xmls
>- updated DSL translation to generate Gremlin that is compliant with TP3 
> when TP3 is being used.  See GremlinQuery.scala, 
> GraphPersistenceStrategies.scala
>- TitanGraphProvider replaced by AtlasGraphProvider.  Graph database 
> implementation is determined from a new optional configuration property
>- HiveTitanSample is no longer used by tests.  It has been replaced by 
> hive-instances.json (which uses normal Atlas json syntax).  The data is saved 
> with a new JSONImporter class.  This was needed because the graphson syntax 
> used by HiveTitanSample is not compatible with TP3.  
> 
> Last rebase: 9/22/2016
> 
> 
> Diffs
> -
> 
>   .gitignore e10adbc4457f6297600f0feb01eb54718b8ec406 
>   addons/falcon-bridge/pom.xml 1365bd05a388dc92f7a56c7f7427b5b85f97c7da 
>   addons/hdfs-model/pom.xml 492f39cea085c6e69781e17bcbdbc3a231806df3 
>   
> addons/hdfs-model/src/test/java/org/apache/atlas/fs/model/HDFSModelTest.java 
> ac60294e328835ba0340e150799ddfb348ccdb52 
>   addons/hive-bridge/pom.xml 6993bdb938a6095ca24482e290393eeeb3911bcb 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
>  ad7a4a5d09d8542a841701dfe04981f65f767c14 
>   addons/sqoop-bridge/pom.xml 8c9d278d43b5979ea1743d10845905c13249f8a6 
>   addons/storm-bridge/pom.xml 12c1208b448d456a923bd7309601174ddb561ba5 
>   catalog/pom.xml 2f58a8f0748de65ab78eab35df6abd2fe7c336af 
>   catalog/src/main/java/org/apache/atlas/catalog/query/BaseQuery.java 
> e7bb505075983371ca12d9bc1d8c6eb240c3d134 
>   distro/src/conf/atlas-application.properties 
> d334600dc5534840409b586157799ef3abf9abf2 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasEdge.java 
> dd4b7e614cdd9bf30f957fb6a839d8c60f3e1701 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasElement.java
>  1bc0fc38c0802897f32260520770a16795474d04 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 
> 995c5457ac7f807172f367cc8e3348b3a98dd6f3 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphIndex.java
>  41194d34f079842db0d95c73a8b099459f76ff2f 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphManagement.java
>  c8cd2842ca3090b6bbd384c773b4eb45aff149ce 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphQuery.java
>  93447495bcf18e9f19df9df68fd1cbe1427fc462 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasIndexQuery.java
>  e719d306ffe9f68e3ac6f7406baaf60a12390c34 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasPropertyKey.java
>  315ecddb861e1a1be6e0ab9b36fe4c0a52486ae8 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasVertex.java
>  fff6fb79247e7d0615ce83c4cbbd93d1bf8cf29c 
>   
>

Re: Review Request 47810: ATLAS-694: Update Atlas to use Graph DB abstraction layer



On Sept. 22, 2016, 11:12 a.m., Jeff Hagelberg wrote:
> > Enable checkstyle on new modules with this property in pom.xml - 
> > true
> > 
> > With patch:
> > Running org.apache.atlas.repository.graph.GraphRepoMapperScaleTest
> > Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 113.27 sec 
> > - in org.apache.atlas.repository.graph.GraphRepoMapperScaleTest
> > 
> > Without patch:
> > Running org.apache.atlas.repository.graph.GraphRepoMapperScaleTest
> > Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 89.008 sec 
> > - in org.apache.atlas.repository.graph.GraphRepoMapperScaleTest
> > 
> > With the patch, the test takes longer to run. This implies some performance 
> > issue with the patch
> > 
> > Whats the plan on including titan 1.x implementation - will it be a profile 
> > at compile time such that only one implementation(titan 0.x or titan 1.x) 
> > will be part of the package. Or are both implementations part of the 
> > package and the implementation is chosen from the runtime configuration in 
> > application properties? If its compile time binding, then we shouldn't 
> > expose the conf atlas.graphdb.backend?
> 
> Jeff Hagelberg wrote:
> The plan is for all of the Atlas code to just depend on the abstraction 
> layer.  There will be profiles in incubator-atlas\pom.xml for each graph 
> backend.  The ones we have in our fork look like this:
> 
> 
> titan1
> 
> 
> org.apache.atlas.repository.graphdb.titan1.Titan1Database
> 
> 
> 
> 
> org.apache.atlas
> atlas-graphdb-impls
> ${project.version}
> pom
> 
> 
> org.apache.atlas
> 
> atlas-graphdb-titan0
> 
> 
> com.ibm.analytics
> 
> atlas-graphdb-ibm-graph
> 
> 
> 
> 
> 
> 
> 
> 
> ibm-graph
> 
> 
> com.ibm.analytics.atlas.ibmgraph.IBMGraphDatabase
> 
> org.apache.atlas.repository.typestore.StoreBackedTenantTypeCache
> 4000
> 
> 
> 
> 
> org.apache.atlas
> atlas-graphdb-impls
> ${project.version}
> pom
> 
> 
> org.apache.atlas
> 
> atlas-graphdb-titan0
> 
> 
> org.apache.atlas
> 
> atlas-graphdb-titan1
> 
> 
> 
> 
> 
> 
> 
> 
> These configure the atlas-graphdb-impls project to just pull in the jar 
> from the implementation that is being used.  Right now, webapp has a 
> dependency on this project, and will bundle the graph database implementation 
> uber jar file corresponding to the profile that was selected.  Unfortunately, 
> it is not possible for the titan0 and titan1 implementations to co-exist in 
> the war without the use of a custom classloader.  titan0 and titan1 define 
> many classes with the same name.  In addition, the two require different 
> contents in META-INF/services/javax.script.ScriptEngineFactory.  Our plan has 
> been to have the implementations available in the filesystem and have the 
> correct loaded on startup, most likely using a custom classloader.  We have 
> not really investigated that very much. It's possible that the 
> plugin-classloader that was recently added might be able to help with this.  
> Right now, there is no way to switch between implementations without 
> modifying that atlas war file.  Th
 is is probably ok for now, since we have not contributed the titan1 
implementation yet, but it is definitely something we will need to look at.  I 
am pretty sure that we created a JIRA for that.
> 
> I will do profiling of GraphRepoMapperScaleTest with and without the 
> changes to see if I can identify and fix the performance regression in 
> GraphRepoMapperScaleTest.
> 
> Jeff Hagelberg wrote:
> I've identified and fixed the performance regression.  The main culprit 
> was the logic for looking up whether a property is a

Re: Review Request 47810: ATLAS-694: Update Atlas to use Graph DB abstraction layer


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47810/#review150230
---




repository/src/main/scala/org/apache/atlas/query/GremlinQuery.scala (line 549)


Is this change, fixing a bug in current code? if so, can you pls explain?


- Suma Shivaprasad


On Sept. 23, 2016, 3:29 a.m., Jeff Hagelberg wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47810/
> ---
> 
> (Updated Sept. 23, 2016, 3:29 a.m.)
> 
> 
> Review request for atlas, David Kantor and Neeru Gupta.
> 
> 
> Bugs: ATLAS-694
> https://issues.apache.org/jira/browse/ATLAS-694
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> ATLAS-694: Update Atlas to use abstraction layer.  All of the Atlas code 
> (with the exception of the catalog, which was only updated minimally) has 
> been updated to use the graph database abstraction layer.  In addition, there 
> is now an optional Atlas configuration property that specifies the class of 
> the abstraction layer database to use.  I basically put all of the changes in 
> here with the exception of the actual Titan 1 implementation of code.  This 
> includes the changes to support Tinkerpop 3 syntax.  This is mostly to 
> expedite getting the changes into Atlas.  Originally the TP3 changes were 
> going to be brought in as part of the Titan 1 implementation task.
> 
> Change Summary:
> 
>- change Atlas classes to use AtlasGraph,AtlasVertex,AtlasEdge, etc 
> instead of TitanGraph/Vertex/Edge, etc
>- compile time dependency on titan 0.5.4/TP 2 removed (except in Catalog, 
> which was only changed to use AtlasGraphProvider/AtlasGraph) - see 
> repository\pom.xml, other pom.xmls
>- updated DSL translation to generate Gremlin that is compliant with TP3 
> when TP3 is being used.  See GremlinQuery.scala, 
> GraphPersistenceStrategies.scala
>- TitanGraphProvider replaced by AtlasGraphProvider.  Graph database 
> implementation is determined from a new optional configuration property
>- HiveTitanSample is no longer used by tests.  It has been replaced by 
> hive-instances.json (which uses normal Atlas json syntax).  The data is saved 
> with a new JSONImporter class.  This was needed because the graphson syntax 
> used by HiveTitanSample is not compatible with TP3.  
> 
> Last rebase: 9/22/2016
> 
> 
> Diffs
> -
> 
>   .gitignore e10adbc4457f6297600f0feb01eb54718b8ec406 
>   addons/falcon-bridge/pom.xml 1365bd05a388dc92f7a56c7f7427b5b85f97c7da 
>   addons/hdfs-model/pom.xml 492f39cea085c6e69781e17bcbdbc3a231806df3 
>   
> addons/hdfs-model/src/test/java/org/apache/atlas/fs/model/HDFSModelTest.java 
> ac60294e328835ba0340e150799ddfb348ccdb52 
>   addons/hive-bridge/pom.xml 6993bdb938a6095ca24482e290393eeeb3911bcb 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
>  ad7a4a5d09d8542a841701dfe04981f65f767c14 
>   addons/sqoop-bridge/pom.xml 8c9d278d43b5979ea1743d10845905c13249f8a6 
>   addons/storm-bridge/pom.xml 12c1208b448d456a923bd7309601174ddb561ba5 
>   catalog/pom.xml 2f58a8f0748de65ab78eab35df6abd2fe7c336af 
>   catalog/src/main/java/org/apache/atlas/catalog/query/BaseQuery.java 
> e7bb505075983371ca12d9bc1d8c6eb240c3d134 
>   distro/src/conf/atlas-application.properties 
> d334600dc5534840409b586157799ef3abf9abf2 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasEdge.java 
> dd4b7e614cdd9bf30f957fb6a839d8c60f3e1701 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasElement.java
>  1bc0fc38c0802897f32260520770a16795474d04 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 
> 995c5457ac7f807172f367cc8e3348b3a98dd6f3 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphIndex.java
>  41194d34f079842db0d95c73a8b099459f76ff2f 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphManagement.java
>  c8cd2842ca3090b6bbd384c773b4eb45aff149ce 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphQuery.java
>  93447495bcf18e9f19df9df68fd1cbe1427fc462 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasIndexQuery.java
>  e719d306ffe9f68e3ac6f7406baaf60a12390c34 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasPropertyKey.java
>  315ecddb861e1a1be6e0ab9b36fe4c0a52486ae8 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasVertex.java
>  fff6fb79247e7d0615ce83c4cbbd93d1bf8cf29c 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasVertexQuery.java
>  366cfe131512f36c02f59ad7e969ff94dfb0e12c 
>   
>

Re: Review Request 47810: ATLAS-694: Update Atlas to use Graph DB abstraction layer


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47810/#review150229
---




repository/src/main/scala/org/apache/atlas/query/GremlinQuery.scala (line 324)


Can you pls raise a jira for this?


- Suma Shivaprasad


On Sept. 23, 2016, 3:29 a.m., Jeff Hagelberg wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47810/
> ---
> 
> (Updated Sept. 23, 2016, 3:29 a.m.)
> 
> 
> Review request for atlas, David Kantor and Neeru Gupta.
> 
> 
> Bugs: ATLAS-694
> https://issues.apache.org/jira/browse/ATLAS-694
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> ATLAS-694: Update Atlas to use abstraction layer.  All of the Atlas code 
> (with the exception of the catalog, which was only updated minimally) has 
> been updated to use the graph database abstraction layer.  In addition, there 
> is now an optional Atlas configuration property that specifies the class of 
> the abstraction layer database to use.  I basically put all of the changes in 
> here with the exception of the actual Titan 1 implementation of code.  This 
> includes the changes to support Tinkerpop 3 syntax.  This is mostly to 
> expedite getting the changes into Atlas.  Originally the TP3 changes were 
> going to be brought in as part of the Titan 1 implementation task.
> 
> Change Summary:
> 
>- change Atlas classes to use AtlasGraph,AtlasVertex,AtlasEdge, etc 
> instead of TitanGraph/Vertex/Edge, etc
>- compile time dependency on titan 0.5.4/TP 2 removed (except in Catalog, 
> which was only changed to use AtlasGraphProvider/AtlasGraph) - see 
> repository\pom.xml, other pom.xmls
>- updated DSL translation to generate Gremlin that is compliant with TP3 
> when TP3 is being used.  See GremlinQuery.scala, 
> GraphPersistenceStrategies.scala
>- TitanGraphProvider replaced by AtlasGraphProvider.  Graph database 
> implementation is determined from a new optional configuration property
>- HiveTitanSample is no longer used by tests.  It has been replaced by 
> hive-instances.json (which uses normal Atlas json syntax).  The data is saved 
> with a new JSONImporter class.  This was needed because the graphson syntax 
> used by HiveTitanSample is not compatible with TP3.  
> 
> Last rebase: 9/22/2016
> 
> 
> Diffs
> -
> 
>   .gitignore e10adbc4457f6297600f0feb01eb54718b8ec406 
>   addons/falcon-bridge/pom.xml 1365bd05a388dc92f7a56c7f7427b5b85f97c7da 
>   addons/hdfs-model/pom.xml 492f39cea085c6e69781e17bcbdbc3a231806df3 
>   
> addons/hdfs-model/src/test/java/org/apache/atlas/fs/model/HDFSModelTest.java 
> ac60294e328835ba0340e150799ddfb348ccdb52 
>   addons/hive-bridge/pom.xml 6993bdb938a6095ca24482e290393eeeb3911bcb 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
>  ad7a4a5d09d8542a841701dfe04981f65f767c14 
>   addons/sqoop-bridge/pom.xml 8c9d278d43b5979ea1743d10845905c13249f8a6 
>   addons/storm-bridge/pom.xml 12c1208b448d456a923bd7309601174ddb561ba5 
>   catalog/pom.xml 2f58a8f0748de65ab78eab35df6abd2fe7c336af 
>   catalog/src/main/java/org/apache/atlas/catalog/query/BaseQuery.java 
> e7bb505075983371ca12d9bc1d8c6eb240c3d134 
>   distro/src/conf/atlas-application.properties 
> d334600dc5534840409b586157799ef3abf9abf2 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasEdge.java 
> dd4b7e614cdd9bf30f957fb6a839d8c60f3e1701 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasElement.java
>  1bc0fc38c0802897f32260520770a16795474d04 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 
> 995c5457ac7f807172f367cc8e3348b3a98dd6f3 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphIndex.java
>  41194d34f079842db0d95c73a8b099459f76ff2f 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphManagement.java
>  c8cd2842ca3090b6bbd384c773b4eb45aff149ce 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphQuery.java
>  93447495bcf18e9f19df9df68fd1cbe1427fc462 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasIndexQuery.java
>  e719d306ffe9f68e3ac6f7406baaf60a12390c34 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasPropertyKey.java
>  315ecddb861e1a1be6e0ab9b36fe4c0a52486ae8 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasVertex.java
>  fff6fb79247e7d0615ce83c4cbbd93d1bf8cf29c 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasVertexQuery.java
>  366cfe131512f36c02f59ad7e969ff94dfb0e12c 
>   
>

Re: Review Request 47810: ATLAS-694: Update Atlas to use Graph DB abstraction layer


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47810/#review150227
---




repository/src/main/scala/org/apache/atlas/query/GremlinQuery.scala (line 489)


filter in Titan 0.5.4 doesnt use any index. Is it the same for Titan 1.0.0 
? Which types does this apply to - map, etc?


- Suma Shivaprasad


On Sept. 23, 2016, 3:29 a.m., Jeff Hagelberg wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47810/
> ---
> 
> (Updated Sept. 23, 2016, 3:29 a.m.)
> 
> 
> Review request for atlas, David Kantor and Neeru Gupta.
> 
> 
> Bugs: ATLAS-694
> https://issues.apache.org/jira/browse/ATLAS-694
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> ATLAS-694: Update Atlas to use abstraction layer.  All of the Atlas code 
> (with the exception of the catalog, which was only updated minimally) has 
> been updated to use the graph database abstraction layer.  In addition, there 
> is now an optional Atlas configuration property that specifies the class of 
> the abstraction layer database to use.  I basically put all of the changes in 
> here with the exception of the actual Titan 1 implementation of code.  This 
> includes the changes to support Tinkerpop 3 syntax.  This is mostly to 
> expedite getting the changes into Atlas.  Originally the TP3 changes were 
> going to be brought in as part of the Titan 1 implementation task.
> 
> Change Summary:
> 
>- change Atlas classes to use AtlasGraph,AtlasVertex,AtlasEdge, etc 
> instead of TitanGraph/Vertex/Edge, etc
>- compile time dependency on titan 0.5.4/TP 2 removed (except in Catalog, 
> which was only changed to use AtlasGraphProvider/AtlasGraph) - see 
> repository\pom.xml, other pom.xmls
>- updated DSL translation to generate Gremlin that is compliant with TP3 
> when TP3 is being used.  See GremlinQuery.scala, 
> GraphPersistenceStrategies.scala
>- TitanGraphProvider replaced by AtlasGraphProvider.  Graph database 
> implementation is determined from a new optional configuration property
>- HiveTitanSample is no longer used by tests.  It has been replaced by 
> hive-instances.json (which uses normal Atlas json syntax).  The data is saved 
> with a new JSONImporter class.  This was needed because the graphson syntax 
> used by HiveTitanSample is not compatible with TP3.  
> 
> Last rebase: 9/22/2016
> 
> 
> Diffs
> -
> 
>   .gitignore e10adbc4457f6297600f0feb01eb54718b8ec406 
>   addons/falcon-bridge/pom.xml 1365bd05a388dc92f7a56c7f7427b5b85f97c7da 
>   addons/hdfs-model/pom.xml 492f39cea085c6e69781e17bcbdbc3a231806df3 
>   
> addons/hdfs-model/src/test/java/org/apache/atlas/fs/model/HDFSModelTest.java 
> ac60294e328835ba0340e150799ddfb348ccdb52 
>   addons/hive-bridge/pom.xml 6993bdb938a6095ca24482e290393eeeb3911bcb 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
>  ad7a4a5d09d8542a841701dfe04981f65f767c14 
>   addons/sqoop-bridge/pom.xml 8c9d278d43b5979ea1743d10845905c13249f8a6 
>   addons/storm-bridge/pom.xml 12c1208b448d456a923bd7309601174ddb561ba5 
>   catalog/pom.xml 2f58a8f0748de65ab78eab35df6abd2fe7c336af 
>   catalog/src/main/java/org/apache/atlas/catalog/query/BaseQuery.java 
> e7bb505075983371ca12d9bc1d8c6eb240c3d134 
>   distro/src/conf/atlas-application.properties 
> d334600dc5534840409b586157799ef3abf9abf2 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasEdge.java 
> dd4b7e614cdd9bf30f957fb6a839d8c60f3e1701 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasElement.java
>  1bc0fc38c0802897f32260520770a16795474d04 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 
> 995c5457ac7f807172f367cc8e3348b3a98dd6f3 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphIndex.java
>  41194d34f079842db0d95c73a8b099459f76ff2f 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphManagement.java
>  c8cd2842ca3090b6bbd384c773b4eb45aff149ce 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphQuery.java
>  93447495bcf18e9f19df9df68fd1cbe1427fc462 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasIndexQuery.java
>  e719d306ffe9f68e3ac6f7406baaf60a12390c34 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasPropertyKey.java
>  315ecddb861e1a1be6e0ab9b36fe4c0a52486ae8 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasVertex.java
>  fff6fb79247e7d0615ce83c4cbbd93d1bf8cf29c 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasVertexQuery.java
>

Re: Review Request 47810: ATLAS-694: Update Atlas to use Graph DB abstraction layer


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47810/#review150218
---




graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphManagement.java
 (line 50)


the mixed and composite index interfaces are too specific to Titan. Can we 
abstract this interface as just two functions indexFullText and indexExactMatch 
and the rest could be handled by the implemenations for Titan0, Titan1, any 
other property graph db . the API could be changed to something like 
indexFullText(indexname, Map params) and the implemenation for 
Titan0 would have the backing index in alongother required params cardinality 
etc.


- Suma Shivaprasad


On Sept. 23, 2016, 3:29 a.m., Jeff Hagelberg wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47810/
> ---
> 
> (Updated Sept. 23, 2016, 3:29 a.m.)
> 
> 
> Review request for atlas, David Kantor and Neeru Gupta.
> 
> 
> Bugs: ATLAS-694
> https://issues.apache.org/jira/browse/ATLAS-694
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> ATLAS-694: Update Atlas to use abstraction layer.  All of the Atlas code 
> (with the exception of the catalog, which was only updated minimally) has 
> been updated to use the graph database abstraction layer.  In addition, there 
> is now an optional Atlas configuration property that specifies the class of 
> the abstraction layer database to use.  I basically put all of the changes in 
> here with the exception of the actual Titan 1 implementation of code.  This 
> includes the changes to support Tinkerpop 3 syntax.  This is mostly to 
> expedite getting the changes into Atlas.  Originally the TP3 changes were 
> going to be brought in as part of the Titan 1 implementation task.
> 
> Change Summary:
> 
>- change Atlas classes to use AtlasGraph,AtlasVertex,AtlasEdge, etc 
> instead of TitanGraph/Vertex/Edge, etc
>- compile time dependency on titan 0.5.4/TP 2 removed (except in Catalog, 
> which was only changed to use AtlasGraphProvider/AtlasGraph) - see 
> repository\pom.xml, other pom.xmls
>- updated DSL translation to generate Gremlin that is compliant with TP3 
> when TP3 is being used.  See GremlinQuery.scala, 
> GraphPersistenceStrategies.scala
>- TitanGraphProvider replaced by AtlasGraphProvider.  Graph database 
> implementation is determined from a new optional configuration property
>- HiveTitanSample is no longer used by tests.  It has been replaced by 
> hive-instances.json (which uses normal Atlas json syntax).  The data is saved 
> with a new JSONImporter class.  This was needed because the graphson syntax 
> used by HiveTitanSample is not compatible with TP3.  
> 
> Last rebase: 9/22/2016
> 
> 
> Diffs
> -
> 
>   .gitignore e10adbc4457f6297600f0feb01eb54718b8ec406 
>   addons/falcon-bridge/pom.xml 1365bd05a388dc92f7a56c7f7427b5b85f97c7da 
>   addons/hdfs-model/pom.xml 492f39cea085c6e69781e17bcbdbc3a231806df3 
>   
> addons/hdfs-model/src/test/java/org/apache/atlas/fs/model/HDFSModelTest.java 
> ac60294e328835ba0340e150799ddfb348ccdb52 
>   addons/hive-bridge/pom.xml 6993bdb938a6095ca24482e290393eeeb3911bcb 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
>  ad7a4a5d09d8542a841701dfe04981f65f767c14 
>   addons/sqoop-bridge/pom.xml 8c9d278d43b5979ea1743d10845905c13249f8a6 
>   addons/storm-bridge/pom.xml 12c1208b448d456a923bd7309601174ddb561ba5 
>   catalog/pom.xml 2f58a8f0748de65ab78eab35df6abd2fe7c336af 
>   catalog/src/main/java/org/apache/atlas/catalog/query/BaseQuery.java 
> e7bb505075983371ca12d9bc1d8c6eb240c3d134 
>   distro/src/conf/atlas-application.properties 
> d334600dc5534840409b586157799ef3abf9abf2 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasEdge.java 
> dd4b7e614cdd9bf30f957fb6a839d8c60f3e1701 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasElement.java
>  1bc0fc38c0802897f32260520770a16795474d04 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 
> 995c5457ac7f807172f367cc8e3348b3a98dd6f3 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphIndex.java
>  41194d34f079842db0d95c73a8b099459f76ff2f 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphManagement.java
>  c8cd2842ca3090b6bbd384c773b4eb45aff149ce 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphQuery.java
>  93447495bcf18e9f19df9df68fd1cbe1427fc462 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasIndexQuery.java
>  e719d306ffe9f68e3ac6f7406baaf60a12390c34 
>   
>

Re: Review Request 47810: ATLAS-694: Update Atlas to use Graph DB abstraction layer

2016-09-23 Thread Jeff Hagelberg



> On Sept. 22, 2016, 11:12 a.m., Shwetha GS wrote:
> > catalog/pom.xml, line 67
> > 
> >
> > for now, catalog can only be used with Titan 0.5.4 - why?
> 
> Jeff Hagelberg wrote:
> A good portion of the catalog project, especially the ones in the query 
> package, make direct use of TP2-specific classes such as GremlinPipeline.  
> TP3 uses a different set of packages.  It would take a good deal of work to 
> refactor the catalog project to generate gremlin that is compatible with both 
> TP2 and TP3.  To reduce the scope, I did not tackle that in this initial 
> implementation.  The query generation logic here will be need to use some 
> abstraction that is independent of the TP version.  If done right, this could 
> also be used to help clean up the DSL translation logic as well (See my 
> response to Suma's comment about that).  At this point, I would really like 
> to focus on getting the general framework in place, which these changes 
> covers.  I think things making catalog work with TP3 should be treated as a 
> separate JIRA.

I've created JIRA ATLAS-1194 to add TP3 support to Catalog.


- Jeff


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47810/#review149947
---


On Sept. 23, 2016, 3:29 a.m., Jeff Hagelberg wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47810/
> ---
> 
> (Updated Sept. 23, 2016, 3:29 a.m.)
> 
> 
> Review request for atlas, David Kantor and Neeru Gupta.
> 
> 
> Bugs: ATLAS-694
> https://issues.apache.org/jira/browse/ATLAS-694
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> ATLAS-694: Update Atlas to use abstraction layer.  All of the Atlas code 
> (with the exception of the catalog, which was only updated minimally) has 
> been updated to use the graph database abstraction layer.  In addition, there 
> is now an optional Atlas configuration property that specifies the class of 
> the abstraction layer database to use.  I basically put all of the changes in 
> here with the exception of the actual Titan 1 implementation of code.  This 
> includes the changes to support Tinkerpop 3 syntax.  This is mostly to 
> expedite getting the changes into Atlas.  Originally the TP3 changes were 
> going to be brought in as part of the Titan 1 implementation task.
> 
> Change Summary:
> 
>- change Atlas classes to use AtlasGraph,AtlasVertex,AtlasEdge, etc 
> instead of TitanGraph/Vertex/Edge, etc
>- compile time dependency on titan 0.5.4/TP 2 removed (except in Catalog, 
> which was only changed to use AtlasGraphProvider/AtlasGraph) - see 
> repository\pom.xml, other pom.xmls
>- updated DSL translation to generate Gremlin that is compliant with TP3 
> when TP3 is being used.  See GremlinQuery.scala, 
> GraphPersistenceStrategies.scala
>- TitanGraphProvider replaced by AtlasGraphProvider.  Graph database 
> implementation is determined from a new optional configuration property
>- HiveTitanSample is no longer used by tests.  It has been replaced by 
> hive-instances.json (which uses normal Atlas json syntax).  The data is saved 
> with a new JSONImporter class.  This was needed because the graphson syntax 
> used by HiveTitanSample is not compatible with TP3.  
> 
> Last rebase: 9/22/2016
> 
> 
> Diffs
> -
> 
>   .gitignore e10adbc4457f6297600f0feb01eb54718b8ec406 
>   addons/falcon-bridge/pom.xml 1365bd05a388dc92f7a56c7f7427b5b85f97c7da 
>   addons/hdfs-model/pom.xml 492f39cea085c6e69781e17bcbdbc3a231806df3 
>   
> addons/hdfs-model/src/test/java/org/apache/atlas/fs/model/HDFSModelTest.java 
> ac60294e328835ba0340e150799ddfb348ccdb52 
>   addons/hive-bridge/pom.xml 6993bdb938a6095ca24482e290393eeeb3911bcb 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
>  ad7a4a5d09d8542a841701dfe04981f65f767c14 
>   addons/sqoop-bridge/pom.xml 8c9d278d43b5979ea1743d10845905c13249f8a6 
>   addons/storm-bridge/pom.xml 12c1208b448d456a923bd7309601174ddb561ba5 
>   catalog/pom.xml 2f58a8f0748de65ab78eab35df6abd2fe7c336af 
>   catalog/src/main/java/org/apache/atlas/catalog/query/BaseQuery.java 
> e7bb505075983371ca12d9bc1d8c6eb240c3d134 
>   distro/src/conf/atlas-application.properties 
> d334600dc5534840409b586157799ef3abf9abf2 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasEdge.java 
> dd4b7e614cdd9bf30f957fb6a839d8c60f3e1701 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasElement.java
>  1bc0fc38c0802897f32260520770a16795474d04 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 
> 995c5457ac7f807172f367cc8e3348b3a98dd6f3 
>

Re: Review Request 47810: ATLAS-694: Update Atlas to use Graph DB abstraction layer

2016-09-23 Thread Jeff Hagelberg



> On Sept. 21, 2016, 11:47 p.m., Suma Shivaprasad wrote:
> > repository/src/main/scala/org/apache/atlas/query/GremlinQuery.scala, line 
> > 279
> > 
> >
> > Instead of having multiple checks its better to abstract out the 
> > Gremlin2.0 and Gremlin3.0 specific implementations which are referred to 
> > from a base installation?
> 
> Jeff Hagelberg wrote:
> Yes, I definitely agree.  The DSL translation should be done by 
> translating the DSL into intermedate classes such as GremlinSelectExpression, 
> GremlnHasExpression, etc.  These would have subclasses specific to gremlin 
> 2/3 where appropriate, and the instance would be served up by some factory.  
> However, doing this would require a substantial rewrite of the DSL 
> translator.  I can do this if you feel it is important, but it was something 
> I was hoping to avoid.  I do think this is where we want to end up, though.  
> I'll start looking at this.  It will be a non-trivial exercise though.  Could 
> it be moved into a follow-on JIRA?
> 
> Apoorv Naik wrote:
> Can the potential DSL translator be written in Java ? (Just a thought)

I've created JIRA ATLAS-1195 to clean up the DSL translation.  Apoorv, I'll 
take that into consideration.  It largely depends on what the community thinks. 
 Most likely, any new classes I implement would be in Java.  I'm hoping to be 
able to leverage the expression classes that already exist, though (which are 
in Scala).  It's too early to say anything very concrete, I'm  still 
experimenting with different options for this.


- Jeff


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47810/#review149922
---


On Sept. 23, 2016, 3:29 a.m., Jeff Hagelberg wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47810/
> ---
> 
> (Updated Sept. 23, 2016, 3:29 a.m.)
> 
> 
> Review request for atlas, David Kantor and Neeru Gupta.
> 
> 
> Bugs: ATLAS-694
> https://issues.apache.org/jira/browse/ATLAS-694
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> ATLAS-694: Update Atlas to use abstraction layer.  All of the Atlas code 
> (with the exception of the catalog, which was only updated minimally) has 
> been updated to use the graph database abstraction layer.  In addition, there 
> is now an optional Atlas configuration property that specifies the class of 
> the abstraction layer database to use.  I basically put all of the changes in 
> here with the exception of the actual Titan 1 implementation of code.  This 
> includes the changes to support Tinkerpop 3 syntax.  This is mostly to 
> expedite getting the changes into Atlas.  Originally the TP3 changes were 
> going to be brought in as part of the Titan 1 implementation task.
> 
> Change Summary:
> 
>- change Atlas classes to use AtlasGraph,AtlasVertex,AtlasEdge, etc 
> instead of TitanGraph/Vertex/Edge, etc
>- compile time dependency on titan 0.5.4/TP 2 removed (except in Catalog, 
> which was only changed to use AtlasGraphProvider/AtlasGraph) - see 
> repository\pom.xml, other pom.xmls
>- updated DSL translation to generate Gremlin that is compliant with TP3 
> when TP3 is being used.  See GremlinQuery.scala, 
> GraphPersistenceStrategies.scala
>- TitanGraphProvider replaced by AtlasGraphProvider.  Graph database 
> implementation is determined from a new optional configuration property
>- HiveTitanSample is no longer used by tests.  It has been replaced by 
> hive-instances.json (which uses normal Atlas json syntax).  The data is saved 
> with a new JSONImporter class.  This was needed because the graphson syntax 
> used by HiveTitanSample is not compatible with TP3.  
> 
> Last rebase: 9/22/2016
> 
> 
> Diffs
> -
> 
>   .gitignore e10adbc4457f6297600f0feb01eb54718b8ec406 
>   addons/falcon-bridge/pom.xml 1365bd05a388dc92f7a56c7f7427b5b85f97c7da 
>   addons/hdfs-model/pom.xml 492f39cea085c6e69781e17bcbdbc3a231806df3 
>   
> addons/hdfs-model/src/test/java/org/apache/atlas/fs/model/HDFSModelTest.java 
> ac60294e328835ba0340e150799ddfb348ccdb52 
>   addons/hive-bridge/pom.xml 6993bdb938a6095ca24482e290393eeeb3911bcb 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
>  ad7a4a5d09d8542a841701dfe04981f65f767c14 
>   addons/sqoop-bridge/pom.xml 8c9d278d43b5979ea1743d10845905c13249f8a6 
>   addons/storm-bridge/pom.xml 12c1208b448d456a923bd7309601174ddb561ba5 
>   catalog/pom.xml 2f58a8f0748de65ab78eab35df6abd2fe7c336af 
>   catalog/src/main/java/org/apache/atlas/catalog/query/BaseQuery.java 
> e7bb505075983371ca12d9bc1d8c6eb240c3d134 
>   distro/src/conf/atlas-application.properties

Re: Review Request 47810: ATLAS-694: Update Atlas to use Graph DB abstraction layer



On Sept. 22, 2016, 11:12 a.m., Jeff Hagelberg wrote:
> > Enable checkstyle on new modules with this property in pom.xml - 
> > true
> > 
> > With patch:
> > Running org.apache.atlas.repository.graph.GraphRepoMapperScaleTest
> > Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 113.27 sec 
> > - in org.apache.atlas.repository.graph.GraphRepoMapperScaleTest
> > 
> > Without patch:
> > Running org.apache.atlas.repository.graph.GraphRepoMapperScaleTest
> > Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 89.008 sec 
> > - in org.apache.atlas.repository.graph.GraphRepoMapperScaleTest
> > 
> > With the patch, the test takes longer to run. This implies some performance 
> > issue with the patch
> > 
> > Whats the plan on including titan 1.x implementation - will it be a profile 
> > at compile time such that only one implementation(titan 0.x or titan 1.x) 
> > will be part of the package. Or are both implementations part of the 
> > package and the implementation is chosen from the runtime configuration in 
> > application properties? If its compile time binding, then we shouldn't 
> > expose the conf atlas.graphdb.backend?
> 
> Jeff Hagelberg wrote:
> The plan is for all of the Atlas code to just depend on the abstraction 
> layer.  There will be profiles in incubator-atlas\pom.xml for each graph 
> backend.  The ones we have in our fork look like this:
> 
> 
> titan1
> 
> 
> org.apache.atlas.repository.graphdb.titan1.Titan1Database
> 
> 
> 
> 
> org.apache.atlas
> atlas-graphdb-impls
> ${project.version}
> pom
> 
> 
> org.apache.atlas
> 
> atlas-graphdb-titan0
> 
> 
> com.ibm.analytics
> 
> atlas-graphdb-ibm-graph
> 
> 
> 
> 
> 
> 
> 
> 
> ibm-graph
> 
> 
> com.ibm.analytics.atlas.ibmgraph.IBMGraphDatabase
> 
> org.apache.atlas.repository.typestore.StoreBackedTenantTypeCache
> 4000
> 
> 
> 
> 
> org.apache.atlas
> atlas-graphdb-impls
> ${project.version}
> pom
> 
> 
> org.apache.atlas
> 
> atlas-graphdb-titan0
> 
> 
> org.apache.atlas
> 
> atlas-graphdb-titan1
> 
> 
> 
> 
> 
> 
> 
> 
> These configure the atlas-graphdb-impls project to just pull in the jar 
> from the implementation that is being used.  Right now, webapp has a 
> dependency on this project, and will bundle the graph database implementation 
> uber jar file corresponding to the profile that was selected.  Unfortunately, 
> it is not possible for the titan0 and titan1 implementations to co-exist in 
> the war without the use of a custom classloader.  titan0 and titan1 define 
> many classes with the same name.  In addition, the two require different 
> contents in META-INF/services/javax.script.ScriptEngineFactory.  Our plan has 
> been to have the implementations available in the filesystem and have the 
> correct loaded on startup, most likely using a custom classloader.  We have 
> not really investigated that very much. It's possible that the 
> plugin-classloader that was recently added might be able to help with this.  
> Right now, there is no way to switch between implementations without 
> modifying that atlas war file.  Th
 is is probably ok for now, since we have not contributed the titan1 
implementation yet, but it is definitely something we will need to look at.  I 
am pretty sure that we created a JIRA for that.
> 
> I will do profiling of GraphRepoMapperScaleTest with and without the 
> changes to see if I can identify and fix the performance regression in 
> GraphRepoMapperScaleTest.

I've identified and fixed the performance regression.  The main culprit was the 
logic for looking up whether a property is a multi-property.  That checked has

Re: Review Request 47810: ATLAS-694: Update Atlas to use Graph DB abstraction layer


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47810/
---

(Updated Sept. 23, 2016, 3:29 a.m.)


Review request for atlas, David Kantor and Neeru Gupta.


Bugs: ATLAS-694
https://issues.apache.org/jira/browse/ATLAS-694


Repository: atlas


Description (updated)
---

ATLAS-694: Update Atlas to use abstraction layer.  All of the Atlas code (with 
the exception of the catalog, which was only updated minimally) has been 
updated to use the graph database abstraction layer.  In addition, there is now 
an optional Atlas configuration property that specifies the class of the 
abstraction layer database to use.  I basically put all of the changes in here 
with the exception of the actual Titan 1 implementation of code.  This includes 
the changes to support Tinkerpop 3 syntax.  This is mostly to expedite getting 
the changes into Atlas.  Originally the TP3 changes were going to be brought in 
as part of the Titan 1 implementation task.

Change Summary:

   - change Atlas classes to use AtlasGraph,AtlasVertex,AtlasEdge, etc instead 
of TitanGraph/Vertex/Edge, etc
   - compile time dependency on titan 0.5.4/TP 2 removed (except in Catalog, 
which was only changed to use AtlasGraphProvider/AtlasGraph) - see 
repository\pom.xml, other pom.xmls
   - updated DSL translation to generate Gremlin that is compliant with TP3 
when TP3 is being used.  See GremlinQuery.scala, 
GraphPersistenceStrategies.scala
   - TitanGraphProvider replaced by AtlasGraphProvider.  Graph database 
implementation is determined from a new optional configuration property
   - HiveTitanSample is no longer used by tests.  It has been replaced by 
hive-instances.json (which uses normal Atlas json syntax).  The data is saved 
with a new JSONImporter class.  This was needed because the graphson syntax 
used by HiveTitanSample is not compatible with TP3.  

Last rebase: 9/22/2016


Diffs
-

  .gitignore e10adbc4457f6297600f0feb01eb54718b8ec406 
  addons/falcon-bridge/pom.xml 1365bd05a388dc92f7a56c7f7427b5b85f97c7da 
  addons/hdfs-model/pom.xml 492f39cea085c6e69781e17bcbdbc3a231806df3 
  addons/hdfs-model/src/test/java/org/apache/atlas/fs/model/HDFSModelTest.java 
ac60294e328835ba0340e150799ddfb348ccdb52 
  addons/hive-bridge/pom.xml 6993bdb938a6095ca24482e290393eeeb3911bcb 
  
addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
 ad7a4a5d09d8542a841701dfe04981f65f767c14 
  addons/sqoop-bridge/pom.xml 8c9d278d43b5979ea1743d10845905c13249f8a6 
  addons/storm-bridge/pom.xml 12c1208b448d456a923bd7309601174ddb561ba5 
  catalog/pom.xml 2f58a8f0748de65ab78eab35df6abd2fe7c336af 
  catalog/src/main/java/org/apache/atlas/catalog/query/BaseQuery.java 
e7bb505075983371ca12d9bc1d8c6eb240c3d134 
  distro/src/conf/atlas-application.properties 
d334600dc5534840409b586157799ef3abf9abf2 
  graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasEdge.java 
dd4b7e614cdd9bf30f957fb6a839d8c60f3e1701 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasElement.java 
1bc0fc38c0802897f32260520770a16795474d04 
  graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 
995c5457ac7f807172f367cc8e3348b3a98dd6f3 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphIndex.java
 41194d34f079842db0d95c73a8b099459f76ff2f 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphManagement.java
 c8cd2842ca3090b6bbd384c773b4eb45aff149ce 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphQuery.java
 93447495bcf18e9f19df9df68fd1cbe1427fc462 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasIndexQuery.java
 e719d306ffe9f68e3ac6f7406baaf60a12390c34 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasPropertyKey.java
 315ecddb861e1a1be6e0ab9b36fe4c0a52486ae8 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasVertex.java 
fff6fb79247e7d0615ce83c4cbbd93d1bf8cf29c 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasVertexQuery.java
 366cfe131512f36c02f59ad7e969ff94dfb0e12c 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/GraphDatabase.java
 37acda5f6ddb883ee364d08a31251b071e6a89a5 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/GremlinVersion.java
 f328d3cb9d2b695bf0eb9981d7046bb67d0eac7d 
  
graphdb/common/src/main/java/org/apache/atlas/repository/graphdb/titan/query/NativeTitanGraphQuery.java
 b79bf7964ae9c830cfa28b9f2f1a1a1059d67223 
  
graphdb/common/src/main/java/org/apache/atlas/repository/graphdb/titan/query/NativeTitanQueryFactory.java
 5c654119a4292136e8ff935c34dd6e39741bffc1 
  
graphdb/common/src/main/java/org/apache/atlas/repository/graphdb/titan/query/TitanGraphQuery.java
 f23847a2f75c351ad307680d0642bc1da49fc59f

Re: Review Request 47810: ATLAS-694: Update Atlas to use Graph DB abstraction layer



> On Sept. 22, 2016, 11:12 a.m., Shwetha GS wrote:
> > repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java,
> >  line 422
> > 
> >
> > Isn't this also in Titan0Vertex?

Yes, I'll move this logic into a method in AtlasGraph so it can be reused in 
both places.


- Jeff


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47810/#review149947
---


On Sept. 23, 2016, 3:21 a.m., Jeff Hagelberg wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47810/
> ---
> 
> (Updated Sept. 23, 2016, 3:21 a.m.)
> 
> 
> Review request for atlas, David Kantor and Neeru Gupta.
> 
> 
> Bugs: ATLAS-694
> https://issues.apache.org/jira/browse/ATLAS-694
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> ATLAS-694: Update Atlas to use abstraction layer.  All of the Atlas code 
> (with the exception of the catalog, which was only updated minimally) has 
> been updated to use the graph database abstraction layer.  In addition, there 
> is now an optional Atlas configuration property that specifies the class of 
> the abstraction layer database to use.  I basically put all of the changes in 
> here with the exception of the actual Titan 1 implementation of code.  This 
> includes the changes to support Tinkerpop 3 syntax.  This is mostly to 
> expedite getting the changes into Atlas.  Originally the TP3 changes were 
> going to be brought in as part of the Titan 1 implementation task.
> 
> Change Summary:
> 
>- change Atlas classes to use AtlasGraph,AtlasVertex,AtlasEdge, etc 
> instead of TitanGraph/Vertex/Edge, etc
>- compile time dependency on titan 0.5.4/TP 2 removed (except in Catalog, 
> which was only changed to use AtlasGraphProvider/AtlasGraph) - see 
> repository\pom.xml, other pom.xmls
>- updated DSL translation to generate Gremlin that is compliant with TP3 
> when TP3 is being used.  See GremlinQuery.scala, 
> GraphPersistenceStrategies.scala
>- TitanGraphProvider replaced by AtlasGraphProvider.  Graph database 
> implementation is determined from a new optional configuration property
>- HiveTitanSample is no longer used by tests.  It has been replaced by 
> hive-instances.json (which uses normal Atlas json syntax).  The data is saved 
> with a new JSONImporter class.  This was needed because the graphson syntax 
> used by HiveTitanSample is not compatible with TP3.  
> 
> Last rebase: 9/21/2016
> 
> 
> Diffs
> -
> 
>   .gitignore e10adbc4457f6297600f0feb01eb54718b8ec406 
>   addons/falcon-bridge/pom.xml 1365bd05a388dc92f7a56c7f7427b5b85f97c7da 
>   addons/hdfs-model/pom.xml 492f39cea085c6e69781e17bcbdbc3a231806df3 
>   
> addons/hdfs-model/src/test/java/org/apache/atlas/fs/model/HDFSModelTest.java 
> ac60294e328835ba0340e150799ddfb348ccdb52 
>   addons/hive-bridge/pom.xml 6993bdb938a6095ca24482e290393eeeb3911bcb 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
>  ad7a4a5d09d8542a841701dfe04981f65f767c14 
>   addons/sqoop-bridge/pom.xml 8c9d278d43b5979ea1743d10845905c13249f8a6 
>   addons/storm-bridge/pom.xml 12c1208b448d456a923bd7309601174ddb561ba5 
>   catalog/pom.xml 2f58a8f0748de65ab78eab35df6abd2fe7c336af 
>   catalog/src/main/java/org/apache/atlas/catalog/query/BaseQuery.java 
> e7bb505075983371ca12d9bc1d8c6eb240c3d134 
>   distro/src/conf/atlas-application.properties 
> d334600dc5534840409b586157799ef3abf9abf2 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasEdge.java 
> dd4b7e614cdd9bf30f957fb6a839d8c60f3e1701 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasElement.java
>  1bc0fc38c0802897f32260520770a16795474d04 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 
> 995c5457ac7f807172f367cc8e3348b3a98dd6f3 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphIndex.java
>  41194d34f079842db0d95c73a8b099459f76ff2f 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphManagement.java
>  c8cd2842ca3090b6bbd384c773b4eb45aff149ce 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphQuery.java
>  93447495bcf18e9f19df9df68fd1cbe1427fc462 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasIndexQuery.java
>  e719d306ffe9f68e3ac6f7406baaf60a12390c34 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasPropertyKey.java
>  315ecddb861e1a1be6e0ab9b36fe4c0a52486ae8 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasVertex.java
>  fff6fb79247e7d0615ce83c4cbbd93d1bf8cf29c 
>   
>

Re: Review Request 47810: ATLAS-694: Update Atlas to use Graph DB abstraction layer


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47810/
---

(Updated Sept. 23, 2016, 3:21 a.m.)


Review request for atlas, David Kantor and Neeru Gupta.


Changes
---

Addressed code review comments.  Fixed performance issue in 
GraphRepoMapperScaleTest.  The main issue there was extra processing done 
during set vertices.  That has been optimized, now the performance is 
comparable to what it was before the changes (actually, it was slightly better 
in my test runs).  Enabled checkstyle in new projects, addressed violations.  
Added synchronization check to AtlasGraphProvider.  Added @VisibleForTesting as 
requested.  Refactored/opimized check for multi-properties, now it is defined 
as a method in AtlasGraph and called from both places where it was used.  
Updated atlas-application.properties in typesystem to make use of the 
atlas.graphdb.backend property.  Reran all tests, no issues found.


Bugs: ATLAS-694
https://issues.apache.org/jira/browse/ATLAS-694


Repository: atlas


Description
---

ATLAS-694: Update Atlas to use abstraction layer.  All of the Atlas code (with 
the exception of the catalog, which was only updated minimally) has been 
updated to use the graph database abstraction layer.  In addition, there is now 
an optional Atlas configuration property that specifies the class of the 
abstraction layer database to use.  I basically put all of the changes in here 
with the exception of the actual Titan 1 implementation of code.  This includes 
the changes to support Tinkerpop 3 syntax.  This is mostly to expedite getting 
the changes into Atlas.  Originally the TP3 changes were going to be brought in 
as part of the Titan 1 implementation task.

Change Summary:

   - change Atlas classes to use AtlasGraph,AtlasVertex,AtlasEdge, etc instead 
of TitanGraph/Vertex/Edge, etc
   - compile time dependency on titan 0.5.4/TP 2 removed (except in Catalog, 
which was only changed to use AtlasGraphProvider/AtlasGraph) - see 
repository\pom.xml, other pom.xmls
   - updated DSL translation to generate Gremlin that is compliant with TP3 
when TP3 is being used.  See GremlinQuery.scala, 
GraphPersistenceStrategies.scala
   - TitanGraphProvider replaced by AtlasGraphProvider.  Graph database 
implementation is determined from a new optional configuration property
   - HiveTitanSample is no longer used by tests.  It has been replaced by 
hive-instances.json (which uses normal Atlas json syntax).  The data is saved 
with a new JSONImporter class.  This was needed because the graphson syntax 
used by HiveTitanSample is not compatible with TP3.  

Last rebase: 9/21/2016


Diffs (updated)
-

  .gitignore e10adbc4457f6297600f0feb01eb54718b8ec406 
  addons/falcon-bridge/pom.xml 1365bd05a388dc92f7a56c7f7427b5b85f97c7da 
  addons/hdfs-model/pom.xml 492f39cea085c6e69781e17bcbdbc3a231806df3 
  addons/hdfs-model/src/test/java/org/apache/atlas/fs/model/HDFSModelTest.java 
ac60294e328835ba0340e150799ddfb348ccdb52 
  addons/hive-bridge/pom.xml 6993bdb938a6095ca24482e290393eeeb3911bcb 
  
addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
 ad7a4a5d09d8542a841701dfe04981f65f767c14 
  addons/sqoop-bridge/pom.xml 8c9d278d43b5979ea1743d10845905c13249f8a6 
  addons/storm-bridge/pom.xml 12c1208b448d456a923bd7309601174ddb561ba5 
  catalog/pom.xml 2f58a8f0748de65ab78eab35df6abd2fe7c336af 
  catalog/src/main/java/org/apache/atlas/catalog/query/BaseQuery.java 
e7bb505075983371ca12d9bc1d8c6eb240c3d134 
  distro/src/conf/atlas-application.properties 
d334600dc5534840409b586157799ef3abf9abf2 
  graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasEdge.java 
dd4b7e614cdd9bf30f957fb6a839d8c60f3e1701 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasElement.java 
1bc0fc38c0802897f32260520770a16795474d04 
  graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 
995c5457ac7f807172f367cc8e3348b3a98dd6f3 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphIndex.java
 41194d34f079842db0d95c73a8b099459f76ff2f 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphManagement.java
 c8cd2842ca3090b6bbd384c773b4eb45aff149ce 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphQuery.java
 93447495bcf18e9f19df9df68fd1cbe1427fc462 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasIndexQuery.java
 e719d306ffe9f68e3ac6f7406baaf60a12390c34 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasPropertyKey.java
 315ecddb861e1a1be6e0ab9b36fe4c0a52486ae8 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasVertex.java 
fff6fb79247e7d0615ce83c4cbbd93d1bf8cf29c 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasVertexQuery.java

Re: Review Request 47810: ATLAS-694: Update Atlas to use Graph DB abstraction layer

> On Sept. 22, 2016, 11:12 a.m., Shwetha GS wrote:
> > catalog/pom.xml, line 67
> > 
> >
> > for now, catalog can only be used with Titan 0.5.4 - why?

A good portion of the catalog project, especially the ones in the query 
package, make direct use of TP2-specific classes such as GremlinPipeline.  TP3 
uses a different set of packages.  It would take a good deal of work to 
refactor the catalog project to generate gremlin that is compatible with both 
TP2 and TP3.  To reduce the scope, I did not tackle that in this initial 
implementation.  The query generation logic here will be need to use some 
abstraction that is independent of the TP version.  If done right, this could 
also be used to help clean up the DSL translation logic as well (See my 
response to Suma's comment about that).  At this point, I would really like to 
focus on getting the general framework in place, which these changes covers.  I 
think things making catalog work with TP3 should be treated as a separate JIRA.

> On Sept. 22, 2016, 11:12 a.m., Shwetha GS wrote:
> > graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasPropertyKey.java,
> >  line 33
> > 
> >
> > There is AtlasPropertyKey.Cardinality and AtlasCardinality. Whats the 
> > difference?

There isn't a difference.  I'll consolidate them.

> On Sept. 22, 2016, 11:12 a.m., Shwetha GS wrote:
> > repository/src/main/java/org/apache/atlas/discovery/graph/DefaultGraphPersistenceStrategy.java,
> >  line 148
> > 
> >
> >

Fixed to use GraphHelper.

> On Sept. 22, 2016, 11:12 a.m., Shwetha GS wrote:
> > repository/src/main/java/org/apache/atlas/repository/graph/AtlasGraphProvider.java,
> >  line 57
> > 
> >
> > add visible for testing annotation. Will be good if you can move to 
> > some test class instead

Annotations have been added.  I found it useful to have the methods in this 
class because they are used by many different test projects.

> On Sept. 22, 2016, 11:12 a.m., Shwetha GS wrote:
> > pom.xml, line 457
> > 
> >
> > where is this used? git grep returns just this result

I guess that at this point it is not needed. since Titan0Database (now 
Titan0GraphDatabase) is the default.  Once support for other implementations is 
merged in, that will be needed to control which implementation class ends up in 
the test atlas-application.properties file in typesystem.  I did include the 
changes to those property files though.  I guess I'll go ahead and add them in.

On Sept. 22, 2016, 11:12 a.m., Jeff Hagelberg wrote:
> > Enable checkstyle on new modules with this property in pom.xml - 
> > true
> > 
> > With patch:
> > Running org.apache.atlas.repository.graph.GraphRepoMapperScaleTest
> > Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 113.27 sec 
> > - in org.apache.atlas.repository.graph.GraphRepoMapperScaleTest
> > 
> > Without patch:
> > Running org.apache.atlas.repository.graph.GraphRepoMapperScaleTest
> > Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 89.008 sec 
> > - in org.apache.atlas.repository.graph.GraphRepoMapperScaleTest
> > 
> > With the patch, the test takes longer to run. This implies some performance 
> > issue with the patch
> > 
> > Whats the plan on including titan 1.x implementation - will it be a profile 
> > at compile time such that only one implementation(titan 0.x or titan 1.x) 
> > will be part of the package. Or are both implementations part of the 
> > package and the implementation is chosen from the runtime configuration in 
> > application properties? If its compile time binding, then we shouldn't 
> > expose the conf atlas.graphdb.backend?

The plan is for all of the Atlas code to just depend on the abstraction layer.  
There will be profiles in incubator-atlas\pom.xml for each graph backend.  The 
ones we have in our fork look like this:

titan1

org.apache.atlas.repository.graphdb.titan1.Titan1Database

org.apache.atlas
atlas-graphdb-impls
${project.version}
pom

org.apache.atlas
atlas-graphdb-titan0

com.ibm.analytics
atlas-graphdb-ibm-graph

Re: Review Request 47810: ATLAS-694: Update Atlas to use Graph DB abstraction layer

2016-09-22 Thread Shwetha GS


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47810/#review149947
---




catalog/pom.xml (line 67)


for now, catalog can only be used with Titan 0.5.4 - why?



graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasPropertyKey.java
 (line 33)


There is AtlasPropertyKey.Cardinality and AtlasCardinality. Whats the 
difference?



graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/GraphDbObjectFactory.java
 (line 36)


The javadoc params are out of date



graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/Titan0DatabaseManager.java
 (line 40)


Rename to Titan0GraphManagement? In general, rename Database to Graph for 
all new classes?



pom.xml (line 457)


where is this used? git grep returns just this result



repository/src/main/java/org/apache/atlas/discovery/graph/DefaultGraphPersistenceStrategy.java
 (line 148)






repository/src/main/java/org/apache/atlas/repository/graph/AtlasGraphProvider.java
 (line 45)


You will need synchronized locking here - 
http://www.cs.umd.edu/~pugh/java/memoryModel/DoubleCheckedLocking.html



repository/src/main/java/org/apache/atlas/repository/graph/AtlasGraphProvider.java
 (line 57)


add visible for testing annotation. Will be good if you can move to some 
test class instead



repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java
 (line 82)


add visiblefortesting annotation



repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java 
(line 414)


Isn't this also in Titan0Vertex?


Enable checkstyle on new modules with this property in pom.xml - 
true

With patch:
Running org.apache.atlas.repository.graph.GraphRepoMapperScaleTest
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 113.27 sec - in 
org.apache.atlas.repository.graph.GraphRepoMapperScaleTest

Without patch:
Running org.apache.atlas.repository.graph.GraphRepoMapperScaleTest
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 89.008 sec - in 
org.apache.atlas.repository.graph.GraphRepoMapperScaleTest

With the patch, the test takes longer to run. This implies some performance 
issue with the patch

Whats the plan on including titan 1.x implementation - will it be a profile at 
compile time such that only one implementation(titan 0.x or titan 1.x) will be 
part of the package. Or are both implementations part of the package and the 
implementation is chosen from the runtime configuration in application 
properties? If its compile time binding, then we shouldn't expose the conf 
atlas.graphdb.backend?

- Shwetha GS


On Sept. 21, 2016, 1:49 p.m., Jeff Hagelberg wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47810/
> ---
> 
> (Updated Sept. 21, 2016, 1:49 p.m.)
> 
> 
> Review request for atlas, David Kantor and Neeru Gupta.
> 
> 
> Bugs: ATLAS-694
> https://issues.apache.org/jira/browse/ATLAS-694
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> ATLAS-694: Update Atlas to use abstraction layer.  All of the Atlas code 
> (with the exception of the catalog, which was only updated minimally) has 
> been updated to use the graph database abstraction layer.  In addition, there 
> is now an optional Atlas configuration property that specifies the class of 
> the abstraction layer database to use.  I basically put all of the changes in 
> here with the exception of the actual Titan 1 implementation of code.  This 
> includes the changes to support Tinkerpop 3 syntax.  This is mostly to 
> expedite getting the changes into Atlas.  Originally the TP3 changes were 
> going to be brought in as part of the Titan 1 implementation task.
> 
> Change Summary:
> 
>- change Atlas classes to use AtlasGraph,AtlasVertex,AtlasEdge, etc 
> instead of TitanGraph/Vertex/Edge, etc
>- compile time dependency on titan 0.5.4/TP 2 removed (except in Catalog, 
> which was only changed to use AtlasGraphProvider/AtlasGraph) - see 
> repository\pom.xml, other pom.xmls
>- updated DSL translation to generate Gremlin that is compliant with TP3 
> when TP3 is being used.  See GremlinQuery.scala, 
>

Re: Review Request 47810: ATLAS-694: Update Atlas to use Graph DB abstraction layer

2016-09-21 Thread Apoorv Naik



> On Sept. 21, 2016, 11:47 p.m., Suma Shivaprasad wrote:
> > repository/src/main/scala/org/apache/atlas/query/GremlinQuery.scala, line 
> > 279
> > 
> >
> > Instead of having multiple checks its better to abstract out the 
> > Gremlin2.0 and Gremlin3.0 specific implementations which are referred to 
> > from a base installation?
> 
> Jeff Hagelberg wrote:
> Yes, I definitely agree.  The DSL translation should be done by 
> translating the DSL into intermedate classes such as GremlinSelectExpression, 
> GremlnHasExpression, etc.  These would have subclasses specific to gremlin 
> 2/3 where appropriate, and the instance would be served up by some factory.  
> However, doing this would require a substantial rewrite of the DSL 
> translator.  I can do this if you feel it is important, but it was something 
> I was hoping to avoid.  I do think this is where we want to end up, though.  
> I'll start looking at this.  It will be a non-trivial exercise though.  Could 
> it be moved into a follow-on JIRA?

Can the potential DSL translator be written in Java ? (Just a thought)


- Apoorv


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47810/#review149922
---


On Sept. 21, 2016, 1:49 p.m., Jeff Hagelberg wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47810/
> ---
> 
> (Updated Sept. 21, 2016, 1:49 p.m.)
> 
> 
> Review request for atlas, David Kantor and Neeru Gupta.
> 
> 
> Bugs: ATLAS-694
> https://issues.apache.org/jira/browse/ATLAS-694
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> ATLAS-694: Update Atlas to use abstraction layer.  All of the Atlas code 
> (with the exception of the catalog, which was only updated minimally) has 
> been updated to use the graph database abstraction layer.  In addition, there 
> is now an optional Atlas configuration property that specifies the class of 
> the abstraction layer database to use.  I basically put all of the changes in 
> here with the exception of the actual Titan 1 implementation of code.  This 
> includes the changes to support Tinkerpop 3 syntax.  This is mostly to 
> expedite getting the changes into Atlas.  Originally the TP3 changes were 
> going to be brought in as part of the Titan 1 implementation task.
> 
> Change Summary:
> 
>- change Atlas classes to use AtlasGraph,AtlasVertex,AtlasEdge, etc 
> instead of TitanGraph/Vertex/Edge, etc
>- compile time dependency on titan 0.5.4/TP 2 removed (except in Catalog, 
> which was only changed to use AtlasGraphProvider/AtlasGraph) - see 
> repository\pom.xml, other pom.xmls
>- updated DSL translation to generate Gremlin that is compliant with TP3 
> when TP3 is being used.  See GremlinQuery.scala, 
> GraphPersistenceStrategies.scala
>- TitanGraphProvider replaced by AtlasGraphProvider.  Graph database 
> implementation is determined from a new optional configuration property
>- HiveTitanSample is no longer used by tests.  It has been replaced by 
> hive-instances.json (which uses normal Atlas json syntax).  The data is saved 
> with a new JSONImporter class.  This was needed because the graphson syntax 
> used by HiveTitanSample is not compatible with TP3.  
> 
> Last rebase: 9/21/2016
> 
> 
> Diffs
> -
> 
>   .gitignore e10adbc4457f6297600f0feb01eb54718b8ec406 
>   addons/falcon-bridge/pom.xml 1365bd05a388dc92f7a56c7f7427b5b85f97c7da 
>   addons/hdfs-model/pom.xml 492f39cea085c6e69781e17bcbdbc3a231806df3 
>   
> addons/hdfs-model/src/test/java/org/apache/atlas/fs/model/HDFSModelTest.java 
> ac60294e328835ba0340e150799ddfb348ccdb52 
>   addons/hive-bridge/pom.xml 6993bdb938a6095ca24482e290393eeeb3911bcb 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
>  ad7a4a5d09d8542a841701dfe04981f65f767c14 
>   addons/sqoop-bridge/pom.xml 8c9d278d43b5979ea1743d10845905c13249f8a6 
>   addons/storm-bridge/pom.xml 12c1208b448d456a923bd7309601174ddb561ba5 
>   catalog/pom.xml 2f58a8f0748de65ab78eab35df6abd2fe7c336af 
>   catalog/src/main/java/org/apache/atlas/catalog/query/BaseQuery.java 
> e7bb505075983371ca12d9bc1d8c6eb240c3d134 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphManagement.java
>  c8cd2842ca3090b6bbd384c773b4eb45aff149ce 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasPropertyKey.java
>  315ecddb861e1a1be6e0ab9b36fe4c0a52486ae8 
>   graphdb/graphdb-impls/pom.xml PRE-CREATION 
>   graphdb/pom.xml ad3461741d42ae83b9d3306bfa4f632ecfc06f3b 
>   
> graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/GraphDbObjectFactory.java
>

Re: Review Request 47810: ATLAS-694: Update Atlas to use Graph DB abstraction layer



> On Sept. 21, 2016, 1:54 p.m., David Radley wrote:
> > I have just did git checkout master and applied the latest diff. I get 
> > compile errors. I my workspace it errors in the pom files as it is not 
> > recognising the new dependancy   
> > 
> > org.apache.atlas
> > atlas-graphdb-impls
> > pom
> > test
> > 
> > 
> > Is there another patch I need to apply as well?
> 
> Jeff Hagelberg wrote:
> Hmm, that project should get built with the rest of Atlas.  I'll 
> investigate.

Did you build the entire Atlas source tree?


- Jeff


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47810/#review149815
---


On Sept. 21, 2016, 1:49 p.m., Jeff Hagelberg wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47810/
> ---
> 
> (Updated Sept. 21, 2016, 1:49 p.m.)
> 
> 
> Review request for atlas, David Kantor and Neeru Gupta.
> 
> 
> Bugs: ATLAS-694
> https://issues.apache.org/jira/browse/ATLAS-694
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> ATLAS-694: Update Atlas to use abstraction layer.  All of the Atlas code 
> (with the exception of the catalog, which was only updated minimally) has 
> been updated to use the graph database abstraction layer.  In addition, there 
> is now an optional Atlas configuration property that specifies the class of 
> the abstraction layer database to use.  I basically put all of the changes in 
> here with the exception of the actual Titan 1 implementation of code.  This 
> includes the changes to support Tinkerpop 3 syntax.  This is mostly to 
> expedite getting the changes into Atlas.  Originally the TP3 changes were 
> going to be brought in as part of the Titan 1 implementation task.
> 
> Change Summary:
> 
>- change Atlas classes to use AtlasGraph,AtlasVertex,AtlasEdge, etc 
> instead of TitanGraph/Vertex/Edge, etc
>- compile time dependency on titan 0.5.4/TP 2 removed (except in Catalog, 
> which was only changed to use AtlasGraphProvider/AtlasGraph) - see 
> repository\pom.xml, other pom.xmls
>- updated DSL translation to generate Gremlin that is compliant with TP3 
> when TP3 is being used.  See GremlinQuery.scala, 
> GraphPersistenceStrategies.scala
>- TitanGraphProvider replaced by AtlasGraphProvider.  Graph database 
> implementation is determined from a new optional configuration property
>- HiveTitanSample is no longer used by tests.  It has been replaced by 
> hive-instances.json (which uses normal Atlas json syntax).  The data is saved 
> with a new JSONImporter class.  This was needed because the graphson syntax 
> used by HiveTitanSample is not compatible with TP3.  
> 
> Last rebase: 9/21/2016
> 
> 
> Diffs
> -
> 
>   .gitignore e10adbc4457f6297600f0feb01eb54718b8ec406 
>   addons/falcon-bridge/pom.xml 1365bd05a388dc92f7a56c7f7427b5b85f97c7da 
>   addons/hdfs-model/pom.xml 492f39cea085c6e69781e17bcbdbc3a231806df3 
>   
> addons/hdfs-model/src/test/java/org/apache/atlas/fs/model/HDFSModelTest.java 
> ac60294e328835ba0340e150799ddfb348ccdb52 
>   addons/hive-bridge/pom.xml 6993bdb938a6095ca24482e290393eeeb3911bcb 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
>  ad7a4a5d09d8542a841701dfe04981f65f767c14 
>   addons/sqoop-bridge/pom.xml 8c9d278d43b5979ea1743d10845905c13249f8a6 
>   addons/storm-bridge/pom.xml 12c1208b448d456a923bd7309601174ddb561ba5 
>   catalog/pom.xml 2f58a8f0748de65ab78eab35df6abd2fe7c336af 
>   catalog/src/main/java/org/apache/atlas/catalog/query/BaseQuery.java 
> e7bb505075983371ca12d9bc1d8c6eb240c3d134 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphManagement.java
>  c8cd2842ca3090b6bbd384c773b4eb45aff149ce 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasPropertyKey.java
>  315ecddb861e1a1be6e0ab9b36fe4c0a52486ae8 
>   graphdb/graphdb-impls/pom.xml PRE-CREATION 
>   graphdb/pom.xml ad3461741d42ae83b9d3306bfa4f632ecfc06f3b 
>   
> graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/GraphDbObjectFactory.java
>  89de23d225c738bcb7f4f502315525af8fa26188 
>   
> graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/Titan0DatabaseManager.java
>  b4234d7321d43c8ff7fc247e895226848b6e256d 
>   
> graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/Titan0PropertyKey.java
>  1f9f6ef786e38a66528189c47d5b147b13461859 
>   
> graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/Titan0Vertex.java
>  b26ff040886c777f1892beb15f09a177f54ea279 
>   
> graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/TitanObjectFactory.java
>

Re: Review Request 47810: ATLAS-694: Update Atlas to use Graph DB abstraction layer



> On Sept. 21, 2016, 1:54 p.m., David Radley wrote:
> > I have just did git checkout master and applied the latest diff. I get 
> > compile errors. I my workspace it errors in the pom files as it is not 
> > recognising the new dependancy   
> > 
> > org.apache.atlas
> > atlas-graphdb-impls
> > pom
> > test
> > 
> > 
> > Is there another patch I need to apply as well?

Hmm, that project should get built with the rest of Atlas.  I'll investigate.


- Jeff


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47810/#review149815
---


On Sept. 21, 2016, 1:49 p.m., Jeff Hagelberg wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47810/
> ---
> 
> (Updated Sept. 21, 2016, 1:49 p.m.)
> 
> 
> Review request for atlas, David Kantor and Neeru Gupta.
> 
> 
> Bugs: ATLAS-694
> https://issues.apache.org/jira/browse/ATLAS-694
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> ATLAS-694: Update Atlas to use abstraction layer.  All of the Atlas code 
> (with the exception of the catalog, which was only updated minimally) has 
> been updated to use the graph database abstraction layer.  In addition, there 
> is now an optional Atlas configuration property that specifies the class of 
> the abstraction layer database to use.  I basically put all of the changes in 
> here with the exception of the actual Titan 1 implementation of code.  This 
> includes the changes to support Tinkerpop 3 syntax.  This is mostly to 
> expedite getting the changes into Atlas.  Originally the TP3 changes were 
> going to be brought in as part of the Titan 1 implementation task.
> 
> Change Summary:
> 
>- change Atlas classes to use AtlasGraph,AtlasVertex,AtlasEdge, etc 
> instead of TitanGraph/Vertex/Edge, etc
>- compile time dependency on titan 0.5.4/TP 2 removed (except in Catalog, 
> which was only changed to use AtlasGraphProvider/AtlasGraph) - see 
> repository\pom.xml, other pom.xmls
>- updated DSL translation to generate Gremlin that is compliant with TP3 
> when TP3 is being used.  See GremlinQuery.scala, 
> GraphPersistenceStrategies.scala
>- TitanGraphProvider replaced by AtlasGraphProvider.  Graph database 
> implementation is determined from a new optional configuration property
>- HiveTitanSample is no longer used by tests.  It has been replaced by 
> hive-instances.json (which uses normal Atlas json syntax).  The data is saved 
> with a new JSONImporter class.  This was needed because the graphson syntax 
> used by HiveTitanSample is not compatible with TP3.  
> 
> Last rebase: 9/21/2016
> 
> 
> Diffs
> -
> 
>   .gitignore e10adbc4457f6297600f0feb01eb54718b8ec406 
>   addons/falcon-bridge/pom.xml 1365bd05a388dc92f7a56c7f7427b5b85f97c7da 
>   addons/hdfs-model/pom.xml 492f39cea085c6e69781e17bcbdbc3a231806df3 
>   
> addons/hdfs-model/src/test/java/org/apache/atlas/fs/model/HDFSModelTest.java 
> ac60294e328835ba0340e150799ddfb348ccdb52 
>   addons/hive-bridge/pom.xml 6993bdb938a6095ca24482e290393eeeb3911bcb 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
>  ad7a4a5d09d8542a841701dfe04981f65f767c14 
>   addons/sqoop-bridge/pom.xml 8c9d278d43b5979ea1743d10845905c13249f8a6 
>   addons/storm-bridge/pom.xml 12c1208b448d456a923bd7309601174ddb561ba5 
>   catalog/pom.xml 2f58a8f0748de65ab78eab35df6abd2fe7c336af 
>   catalog/src/main/java/org/apache/atlas/catalog/query/BaseQuery.java 
> e7bb505075983371ca12d9bc1d8c6eb240c3d134 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphManagement.java
>  c8cd2842ca3090b6bbd384c773b4eb45aff149ce 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasPropertyKey.java
>  315ecddb861e1a1be6e0ab9b36fe4c0a52486ae8 
>   graphdb/graphdb-impls/pom.xml PRE-CREATION 
>   graphdb/pom.xml ad3461741d42ae83b9d3306bfa4f632ecfc06f3b 
>   
> graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/GraphDbObjectFactory.java
>  89de23d225c738bcb7f4f502315525af8fa26188 
>   
> graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/Titan0DatabaseManager.java
>  b4234d7321d43c8ff7fc247e895226848b6e256d 
>   
> graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/Titan0PropertyKey.java
>  1f9f6ef786e38a66528189c47d5b147b13461859 
>   
> graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/Titan0Vertex.java
>  b26ff040886c777f1892beb15f09a177f54ea279 
>   
> graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/TitanObjectFactory.java
>  ab0e798d24a2890e60109193f86944b069ff0046 
>   
>

Re: Review Request 47810: ATLAS-694: Update Atlas to use Graph DB abstraction layer

2016-09-21 Thread David Radley


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47810/#review149815
---



I have just did git checkout master and applied the latest diff. I get compile 
errors. I my workspace it errors in the pom files as it is not recognising the 
new dependancy   

org.apache.atlas
atlas-graphdb-impls
pom
test


Is there another patch I need to apply as well?

- David Radley


On Sept. 21, 2016, 1:49 p.m., Jeff Hagelberg wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47810/
> ---
> 
> (Updated Sept. 21, 2016, 1:49 p.m.)
> 
> 
> Review request for atlas, David Kantor and Neeru Gupta.
> 
> 
> Bugs: ATLAS-694
> https://issues.apache.org/jira/browse/ATLAS-694
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> ATLAS-694: Update Atlas to use abstraction layer.  All of the Atlas code 
> (with the exception of the catalog, which was only updated minimally) has 
> been updated to use the graph database abstraction layer.  In addition, there 
> is now an optional Atlas configuration property that specifies the class of 
> the abstraction layer database to use.  I basically put all of the changes in 
> here with the exception of the actual Titan 1 implementation of code.  This 
> includes the changes to support Tinkerpop 3 syntax.  This is mostly to 
> expedite getting the changes into Atlas.  Originally the TP3 changes were 
> going to be brought in as part of the Titan 1 implementation task.
> 
> Change Summary:
> 
>- change Atlas classes to use AtlasGraph,AtlasVertex,AtlasEdge, etc 
> instead of TitanGraph/Vertex/Edge, etc
>- compile time dependency on titan 0.5.4/TP 2 removed (except in Catalog, 
> which was only changed to use AtlasGraphProvider/AtlasGraph) - see 
> repository\pom.xml, other pom.xmls
>- updated DSL translation to generate Gremlin that is compliant with TP3 
> when TP3 is being used.  See GremlinQuery.scala, 
> GraphPersistenceStrategies.scala
>- TitanGraphProvider replaced by AtlasGraphProvider.  Graph database 
> implementation is determined from a new optional configuration property
>- HiveTitanSample is no longer used by tests.  It has been replaced by 
> hive-instances.json (which uses normal Atlas json syntax).  The data is saved 
> with a new JSONImporter class.  This was needed because the graphson syntax 
> used by HiveTitanSample is not compatible with TP3.  
> 
> Last rebase: 9/21/2016
> 
> 
> Diffs
> -
> 
>   .gitignore e10adbc4457f6297600f0feb01eb54718b8ec406 
>   addons/falcon-bridge/pom.xml 1365bd05a388dc92f7a56c7f7427b5b85f97c7da 
>   addons/hdfs-model/pom.xml 492f39cea085c6e69781e17bcbdbc3a231806df3 
>   
> addons/hdfs-model/src/test/java/org/apache/atlas/fs/model/HDFSModelTest.java 
> ac60294e328835ba0340e150799ddfb348ccdb52 
>   addons/hive-bridge/pom.xml 6993bdb938a6095ca24482e290393eeeb3911bcb 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
>  ad7a4a5d09d8542a841701dfe04981f65f767c14 
>   addons/sqoop-bridge/pom.xml 8c9d278d43b5979ea1743d10845905c13249f8a6 
>   addons/storm-bridge/pom.xml 12c1208b448d456a923bd7309601174ddb561ba5 
>   catalog/pom.xml 2f58a8f0748de65ab78eab35df6abd2fe7c336af 
>   catalog/src/main/java/org/apache/atlas/catalog/query/BaseQuery.java 
> e7bb505075983371ca12d9bc1d8c6eb240c3d134 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphManagement.java
>  c8cd2842ca3090b6bbd384c773b4eb45aff149ce 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasPropertyKey.java
>  315ecddb861e1a1be6e0ab9b36fe4c0a52486ae8 
>   graphdb/graphdb-impls/pom.xml PRE-CREATION 
>   graphdb/pom.xml ad3461741d42ae83b9d3306bfa4f632ecfc06f3b 
>   
> graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/GraphDbObjectFactory.java
>  89de23d225c738bcb7f4f502315525af8fa26188 
>   
> graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/Titan0DatabaseManager.java
>  b4234d7321d43c8ff7fc247e895226848b6e256d 
>   
> graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/Titan0PropertyKey.java
>  1f9f6ef786e38a66528189c47d5b147b13461859 
>   
> graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/Titan0Vertex.java
>  b26ff040886c777f1892beb15f09a177f54ea279 
>   
> graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/TitanObjectFactory.java
>  ab0e798d24a2890e60109193f86944b069ff0046 
>   
> graphdb/titan0/src/test/java/org/apache/atlas/repository/graphdb/titan0/AbstractGraphDatabaseTest.java
>  35735e335fd8f8d47d211d813817e19d9f6a9552 
>   
>

Re: Review Request 47810: ATLAS-694: Update Atlas to use Graph DB abstraction layer


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47810/
---

(Updated Sept. 21, 2016, 1:49 p.m.)


Review request for atlas, David Kantor and Neeru Gupta.


Bugs: ATLAS-694
https://issues.apache.org/jira/browse/ATLAS-694


Repository: atlas


Description (updated)
---

ATLAS-694: Update Atlas to use abstraction layer.  All of the Atlas code (with 
the exception of the catalog, which was only updated minimally) has been 
updated to use the graph database abstraction layer.  In addition, there is now 
an optional Atlas configuration property that specifies the class of the 
abstraction layer database to use.  I basically put all of the changes in here 
with the exception of the actual Titan 1 implementation of code.  This includes 
the changes to support Tinkerpop 3 syntax.  This is mostly to expedite getting 
the changes into Atlas.  Originally the TP3 changes were going to be brought in 
as part of the Titan 1 implementation task.

Change Summary:

   - change Atlas classes to use AtlasGraph,AtlasVertex,AtlasEdge, etc instead 
of TitanGraph/Vertex/Edge, etc
   - compile time dependency on titan 0.5.4/TP 2 removed (except in Catalog, 
which was only changed to use AtlasGraphProvider/AtlasGraph) - see 
repository\pom.xml, other pom.xmls
   - updated DSL translation to generate Gremlin that is compliant with TP3 
when TP3 is being used.  See GremlinQuery.scala, 
GraphPersistenceStrategies.scala
   - TitanGraphProvider replaced by AtlasGraphProvider.  Graph database 
implementation is determined from a new optional configuration property
   - HiveTitanSample is no longer used by tests.  It has been replaced by 
hive-instances.json (which uses normal Atlas json syntax).  The data is saved 
with a new JSONImporter class.  This was needed because the graphson syntax 
used by HiveTitanSample is not compatible with TP3.  

Last rebase: 9/21/2016


Diffs
-

  .gitignore e10adbc4457f6297600f0feb01eb54718b8ec406 
  addons/falcon-bridge/pom.xml 1365bd05a388dc92f7a56c7f7427b5b85f97c7da 
  addons/hdfs-model/pom.xml 492f39cea085c6e69781e17bcbdbc3a231806df3 
  addons/hdfs-model/src/test/java/org/apache/atlas/fs/model/HDFSModelTest.java 
ac60294e328835ba0340e150799ddfb348ccdb52 
  addons/hive-bridge/pom.xml 6993bdb938a6095ca24482e290393eeeb3911bcb 
  
addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
 ad7a4a5d09d8542a841701dfe04981f65f767c14 
  addons/sqoop-bridge/pom.xml 8c9d278d43b5979ea1743d10845905c13249f8a6 
  addons/storm-bridge/pom.xml 12c1208b448d456a923bd7309601174ddb561ba5 
  catalog/pom.xml 2f58a8f0748de65ab78eab35df6abd2fe7c336af 
  catalog/src/main/java/org/apache/atlas/catalog/query/BaseQuery.java 
e7bb505075983371ca12d9bc1d8c6eb240c3d134 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphManagement.java
 c8cd2842ca3090b6bbd384c773b4eb45aff149ce 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasPropertyKey.java
 315ecddb861e1a1be6e0ab9b36fe4c0a52486ae8 
  graphdb/graphdb-impls/pom.xml PRE-CREATION 
  graphdb/pom.xml ad3461741d42ae83b9d3306bfa4f632ecfc06f3b 
  
graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/GraphDbObjectFactory.java
 89de23d225c738bcb7f4f502315525af8fa26188 
  
graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/Titan0DatabaseManager.java
 b4234d7321d43c8ff7fc247e895226848b6e256d 
  
graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/Titan0PropertyKey.java
 1f9f6ef786e38a66528189c47d5b147b13461859 
  
graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/Titan0Vertex.java
 b26ff040886c777f1892beb15f09a177f54ea279 
  
graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/TitanObjectFactory.java
 ab0e798d24a2890e60109193f86944b069ff0046 
  
graphdb/titan0/src/test/java/org/apache/atlas/repository/graphdb/titan0/AbstractGraphDatabaseTest.java
 35735e335fd8f8d47d211d813817e19d9f6a9552 
  
graphdb/titan0/src/test/java/org/apache/atlas/repository/graphdb/titan0/Titan0DatabaseTest.java
 6c2ea263a309c6f71c9eb39fdffed39c8e4895a2 
  pom.xml ac5b0425bc7816261e7c35caad99131c79e75872 
  repository/pom.xml 663ac874518f0942c6f647ea9ee982503410492d 
  repository/src/main/java/org/apache/atlas/GraphTransactionInterceptor.java 
fff8925ebf3b4a7498565f4b9e33885dbb5bc61a 
  repository/src/main/java/org/apache/atlas/RepositoryMetadataModule.java 
f1ef1404a45b10ee4c1c229417565e711624957f 
  
repository/src/main/java/org/apache/atlas/discovery/DataSetLineageService.java 
c216469ceb1d89b5f6a578f9bd96f01c09cccd06 
  
repository/src/main/java/org/apache/atlas/discovery/graph/DefaultGraphPersistenceStrategy.java
 b17eec7e55f478bf4403a61cef7585cf06a2d9d9 
  
repository/src/main/java/org/apache/atlas/discovery/graph/GraphBackedDiscoveryService.java

Re: Review Request 47810: ATLAS-694: Update Atlas to use Graph DB abstraction layer

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47810/
---

(Updated Sept. 21, 2016, 12:28 p.m.)

Review request for atlas, David Kantor and Neeru Gupta.

Changes
---

Changes rebased.

Bugs: ATLAS-694
https://issues.apache.org/jira/browse/ATLAS-694

Repository: atlas

Description
---

ATLAS-694: Update Atlas to use abstraction layer. All of the Atlas code (with
the exception of the catalog, which was only updated minimally) has been
updated to use the graph database abstraction layer. In addition, there is now
an optional Atlas configuration property that specifies the class of the
abstraction layer database to use. I basically put all of the changes in here
with the exception of the actual Titan 1 implementation of code. This includes
the changes to support Tinkerpop 3 syntax. This is mostly to expedite getting
the changes into Atlas. Originally the TP3 changes were going to be brought in
as part of the Titan 1 implementation task.

Change Summary:

- change Atlas classes to use AtlasGraph,AtlasVertex,AtlasEdge, etc instead
of TitanGraph/Vertex/Edge, etc
- compile time dependency on titan 0.5.4/TP 2 removed (except in Catalog,
which was only changed to use AtlasGraphProvider/AtlasGraph) - see
repository\pom.xml, other pom.xmls
- updated DSL translation to generate Gremlin that is compliant with TP3
when TP3 is being used. See GremlinQuery.scala,
GraphPersistenceStrategies.scala
- TitanGraphProvider replaced by AtlasGraphProvider. Graph database
implementation is determined from a new optional configuration property
- HiveTitanSample is no longer used by tests. It has been replaced by
hive-instances.json (which uses normal Atlas json syntax). The data is saved
with a new JSONImporter class. This was needed because the graphson syntax
used by HiveTitanSample is not compatible with TP3.
- RepositoryMetaModule - GraphBackedSearchIndexer changed to an eager
singleton to force index creation to happen before we execute any gremlin
queries (which happens during the type system restoration triggered by the
DefaultMetadataService constructor)
- atlas_config.py - minor change to allow Atlas to use a non-default JVM.
It now supports having the java home in ATLAS_JAVA_HOME. If that is not set,
we fall back to the previous behavior looking for JAVA_HOME and then checking
the path.
- performance : add index on __state property, fix resulting test failure -
see GraphBackedSearchIndexer, GraphHelper.getVertexForProperty
- performance - consolidate creation of built-in types into single call.
See DefaultMetadataService.

Note - the diff here is cumulative and includes the changes from ATLAS-693. I
could not figure out how to remove those. You can, for the most part, ignore
the new files under graphdb\titan0. There are a few minor changes that have
been put in since ATLAS-693, but nothing very significant. Just changes to
implement a couple new methods in the abstraction layer.

Last rebase: 6/7/2016

Diffs (updated)
-

.gitignore e10adbc4457f6297600f0feb01eb54718b8ec406
addons/falcon-bridge/pom.xml 1365bd05a388dc92f7a56c7f7427b5b85f97c7da
addons/hdfs-model/pom.xml 492f39cea085c6e69781e17bcbdbc3a231806df3
addons/hdfs-model/src/test/java/org/apache/atlas/fs/model/HDFSModelTest.java
ac60294e328835ba0340e150799ddfb348ccdb52
addons/hive-bridge/pom.xml 6993bdb938a6095ca24482e290393eeeb3911bcb

addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
ad7a4a5d09d8542a841701dfe04981f65f767c14
addons/sqoop-bridge/pom.xml 8c9d278d43b5979ea1743d10845905c13249f8a6
addons/storm-bridge/pom.xml 12c1208b448d456a923bd7309601174ddb561ba5
catalog/pom.xml 2f58a8f0748de65ab78eab35df6abd2fe7c336af
catalog/src/main/java/org/apache/atlas/catalog/query/BaseQuery.java
e7bb505075983371ca12d9bc1d8c6eb240c3d134

graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphManagement.java
c8cd2842ca3090b6bbd384c773b4eb45aff149ce

graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasPropertyKey.java
315ecddb861e1a1be6e0ab9b36fe4c0a52486ae8
graphdb/graphdb-impls/pom.xml PRE-CREATION
graphdb/pom.xml ad3461741d42ae83b9d3306bfa4f632ecfc06f3b

graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/GraphDbObjectFactory.java
89de23d225c738bcb7f4f502315525af8fa26188

graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/Titan0DatabaseManager.java
b4234d7321d43c8ff7fc247e895226848b6e256d

graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/Titan0PropertyKey.java
1f9f6ef786e38a66528189c47d5b147b13461859

graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/Titan0Vertex.java
b26ff040886c777f1892beb15f09a177f54ea279

Re: Review Request 47810: ATLAS-694: Update Atlas to use Graph DB abstraction layer

2016-06-08 Thread Jeff Hagelberg


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47810/
---

(Updated June 8, 2016, 2:17 p.m.)


Review request for atlas, David Kantor and Neeru Gupta.


Bugs: ATLAS-694
https://issues.apache.org/jira/browse/ATLAS-694


Repository: atlas


Description (updated)
---

ATLAS-694: Update Atlas to use abstraction layer.  All of the Atlas code (with 
the exception of the catalog, which was only updated minimally) has been 
updated to use the graph database abstraction layer.  In addition, there is now 
an optional Atlas configuration property that specifies the class of the 
abstraction layer database to use.  I basically put all of the changes in here 
with the exception of the actual Titan 1 implementation of code.  This includes 
the changes to support Tinkerpop 3 syntax.  This is mostly to expedite getting 
the changes into Atlas.  Originally the TP3 changes were going to be brought in 
as part of the Titan 1 implementation task.

Change Summary:

   - change Atlas classes to use AtlasGraph,AtlasVertex,AtlasEdge, etc instead 
of TitanGraph/Vertex/Edge, etc
   - compile time dependency on titan 0.5.4/TP 2 removed (except in Catalog, 
which was only changed to use AtlasGraphProvider/AtlasGraph) - see 
repository\pom.xml, other pom.xmls
   - updated DSL translation to generate Gremlin that is compliant with TP3 
when TP3 is being used.  See GremlinQuery.scala, 
GraphPersistenceStrategies.scala
   - TitanGraphProvider replaced by AtlasGraphProvider.  Graph database 
implementation is determined from a new optional configuration property
   - HiveTitanSample is no longer used by tests.  It has been replaced by 
hive-instances.json (which uses normal Atlas json syntax).  The data is saved 
with a new JSONImporter class.  This was needed because the graphson syntax 
used by HiveTitanSample is not compatible with TP3.  
   - RepositoryMetaModule - GraphBackedSearchIndexer changed to an eager 
singleton to force index creation to happen before we execute any gremlin 
queries (which happens during the type system restoration triggered by the 
DefaultMetadataService constructor)
   - atlas_config.py - minor change to allow Atlas to use a non-default JVM.  
It now supports having the java home in ATLAS_JAVA_HOME.  If that is not set, 
we fall back to the previous behavior looking for JAVA_HOME and then checking 
the path.
   - performance : add index on __state property, fix resulting test failure - 
see GraphBackedSearchIndexer, GraphHelper.getVertexForProperty
   - performance - consolidate creation of built-in types into single call.  
See DefaultMetadataService.
   
   
Note - the diff here is cumulative and includes the changes from ATLAS-693.  I 
could not figure out how to remove those.  You can, for the most part, ignore 
the new files under graphdb\titan0.  There are a few minor changes that have 
been put in since ATLAS-693, but nothing very significant.  Just changes to 
implement a couple new methods in the abstraction layer.

Last rebase: 6/7/2016


Diffs
-

  addons/falcon-bridge/pom.xml 14c60901a6f194497ca0fb7929a032d2261f85fa 
  addons/hdfs-model/pom.xml f4033bce0fe3ea5a33cfd32ebccd47c01207dc35 
  addons/hdfs-model/src/test/java/org/apache/atlas/fs/model/HDFSModelTest.java 
ac60294e328835ba0340e150799ddfb348ccdb52 
  addons/hive-bridge/pom.xml 47e72e814a421f03b00a9b7c6a13163e1b67ca14 
  addons/sqoop-bridge/pom.xml 343bb4eeae93031ab24cd48ed7b535769150d6c7 
  addons/storm-bridge/pom.xml b1a7a9b1a88442d9e65db4a01b8388c8d3522bfe 
  authorization/pom.xml 74c433bf740a346d1d907a868605e9ab457151bb 
  catalog/pom.xml 8a49d3d673a5936613d45e53479aa99a5a14517d 
  catalog/src/main/java/org/apache/atlas/catalog/DefaultTypeSystem.java 
a28a32b692e61377479cdd432f39ba3c60bd5238 
  catalog/src/main/java/org/apache/atlas/catalog/query/BaseQuery.java 
ba8e0e712769afad72f7f8254ec626a1d17136e8 
  
catalog/src/test/java/org/apache/atlas/catalog/query/AtlasEntityQueryTest.java 
149134c8a79c802fe625d64c5e04af0d88174cfa 
  distro/src/bin/atlas_config.py fab4046345e05e0b7749ca75834c8bdcb0d01752 
  distro/src/conf/atlas-application.properties 
1cdd424d8e126a6bdbf99db8bf4e5d4329e806bb 
  graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasEdge.java 
71b577b5c2e8aee8fec816b9224672ebc2cad5b3 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasEdgeDirection.java
 e7da1c9a5075004973ba68b8a36a2c95756e0e62 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasElement.java 
3c41693e531eb523f04a5c5781f1f123ab38ef07 
  graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 
f312117af9cb6ae8de21d565bfcf3b10d9bdfcc6 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphIndex.java
 PRE-CREATION

Re: Review Request 47810: ATLAS-694: Update Atlas to use Graph DB abstraction layer


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47810/
---

(Updated June 7, 2016, 2:44 a.m.)


Review request for atlas, David Kantor and Neeru Gupta.


Changes
---

Fixed merge-related test failure.


Bugs: ATLAS-694
https://issues.apache.org/jira/browse/ATLAS-694


Repository: atlas


Description
---

ATLAS-694: Update Atlas to use abstraction layer.  All of the Atlas code (with 
the exception of the catalog, which was only updated minimally) has been 
updated to use the graph database abstraction layer.  In addition, there is now 
an optional Atlas configuration property that specifies the class of the 
abstraction layer database to use.  I basically put all of the changes in here 
with the exception of the actual Titan 1 implementation of code.  This includes 
the changes to support Tinkerpop 3 syntax.  This is mostly to expedite getting 
the changes into Atlas.  Originally the TP3 changes were going to be brought in 
as part of the Titan 1 implementation task.

Change Summary:

   - change Atlas classes to use AtlasGraph,AtlasVertex,AtlasEdge, etc instead 
of TitanGraph/Vertex/Edge, etc
   - compile time dependency on titan 0.5.4/TP 2 removed (except in Catalog, 
which was only changed to use AtlasGraphProvider/AtlasGraph)
   - updated DSL translation to generate Gremlin that is compliant with TP3 
when TP3 is being used
   - TitanGraphProvider replaced by AtlasGraphProvider.  Graph database 
implementation is determined from a new optional configuration property
   - search indexer enhanced to wait for all indices to be enabled
   - HiveTitanSample is no longer used by tests.  It has been replaced by 
hive-instances.json (which uses normal Atlas json syntax).  The data is saved 
with a new JSONImporter class.  This was needed because the graphson syntax 
used by HiveTitanSample is not compatible with TP3.  
   - RepositoryMetaModule - GraphBackedSearchIndexer changed to an eager 
singleton to force index creation to happen before we execute any gremlin 
queries (which happens during the type system restoration triggered by the 
DefaultMetadataService constructor)
   - atlas_config.py - minor change to allow Atlas to use a non-default JVM.  
It now supports having the java home in ATLAS_JAVA_HOME.  If that is not set, 
we fall back to the previous behavior looking for JAVA_HOME and then checking 
the path.
   
   
Note - the diff here is cumulative and includes the changes from ATLAS-693.  I 
could not figure out how to remove those.  You can, for the most part, ignore 
the new files under graphdb\titan0.  There are a few minor changes that have 
been put in since ATLAS-693, but nothing very significant.  Just changes to 
implement a couple new methods in the abstraction layer.

Last rebase: 6/6/2016


Diffs (updated)
-

  addons/falcon-bridge/pom.xml 14c60901a6f194497ca0fb7929a032d2261f85fa 
  addons/hdfs-model/pom.xml f4033bce0fe3ea5a33cfd32ebccd47c01207dc35 
  addons/hdfs-model/src/test/java/org/apache/atlas/fs/model/HDFSModelTest.java 
ac60294e328835ba0340e150799ddfb348ccdb52 
  addons/hive-bridge/pom.xml 47e72e814a421f03b00a9b7c6a13163e1b67ca14 
  addons/sqoop-bridge/pom.xml 343bb4eeae93031ab24cd48ed7b535769150d6c7 
  addons/storm-bridge/pom.xml b1a7a9b1a88442d9e65db4a01b8388c8d3522bfe 
  authorization/pom.xml 74c433bf740a346d1d907a868605e9ab457151bb 
  catalog/pom.xml 8a49d3d673a5936613d45e53479aa99a5a14517d 
  catalog/src/main/java/org/apache/atlas/catalog/DefaultTypeSystem.java 
a28a32b692e61377479cdd432f39ba3c60bd5238 
  catalog/src/main/java/org/apache/atlas/catalog/query/BaseQuery.java 
ba8e0e712769afad72f7f8254ec626a1d17136e8 
  
catalog/src/test/java/org/apache/atlas/catalog/query/AtlasEntityQueryTest.java 
149134c8a79c802fe625d64c5e04af0d88174cfa 
  distro/src/bin/atlas_config.py fab4046345e05e0b7749ca75834c8bdcb0d01752 
  distro/src/conf/atlas-application.properties 
1cdd424d8e126a6bdbf99db8bf4e5d4329e806bb 
  graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasEdge.java 
71b577b5c2e8aee8fec816b9224672ebc2cad5b3 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasEdgeDirection.java
 e7da1c9a5075004973ba68b8a36a2c95756e0e62 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasElement.java 
3c41693e531eb523f04a5c5781f1f123ab38ef07 
  graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 
f312117af9cb6ae8de21d565bfcf3b10d9bdfcc6 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphIndex.java
 PRE-CREATION 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphManagement.java
 2776b2242c4469dcae344616ccfe100633a79432 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphQuery.java
 f0dca9a28eebd8996d55d71dcc3254cce9cd4098

Re: Review Request 47810: ATLAS-694: Update Atlas to use Graph DB abstraction layer


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47810/
---

(Updated June 6, 2016, 8:44 p.m.)


Review request for atlas, David Kantor and Neeru Gupta.


Changes
---

Removed @author tag


Bugs: ATLAS-694
https://issues.apache.org/jira/browse/ATLAS-694


Repository: atlas


Description
---

ATLAS-694: Update Atlas to use abstraction layer.  All of the Atlas code (with 
the exception of the catalog, which was only updated minimally) has been 
updated to use the graph database abstraction layer.  In addition, there is now 
an optional Atlas configuration property that specifies the class of the 
abstraction layer database to use.  I basically put all of the changes in here 
with the exception of the actual Titan 1 implementation of code.  This includes 
the changes to support Tinkerpop 3 syntax.  This is mostly to expedite getting 
the changes into Atlas.  Originally the TP3 changes were going to be brought in 
as part of the Titan 1 implementation task.

Change Summary:

   - change Atlas classes to use AtlasGraph,AtlasVertex,AtlasEdge, etc instead 
of TitanGraph/Vertex/Edge, etc
   - compile time dependency on titan 0.5.4/TP 2 removed (except in Catalog, 
which was only changed to use AtlasGraphProvider/AtlasGraph)
   - updated DSL translation to generate Gremlin that is compliant with TP3 
when TP3 is being used
   - TitanGraphProvider replaced by AtlasGraphProvider.  Graph database 
implementation is determined from a new optional configuration property
   - search indexer enhanced to wait for all indices to be enabled
   - HiveTitanSample is no longer used by tests.  It has been replaced by 
hive-instances.json (which uses normal Atlas json syntax).  The data is saved 
with a new JSONImporter class.  This was needed because the graphson syntax 
used by HiveTitanSample is not compatible with TP3.  
   - RepositoryMetaModule - GraphBackedSearchIndexer changed to an eager 
singleton to force index creation to happen before we execute any gremlin 
queries (which happens during the type system restoration triggered by the 
DefaultMetadataService constructor)
   - atlas_config.py - minor change to allow Atlas to use a non-default JVM.  
It now supports having the java home in ATLAS_JAVA_HOME.  If that is not set, 
we fall back to the previous behavior looking for JAVA_HOME and then checking 
the path.
   
   
Note - the diff here is cumulative and includes the changes from ATLAS-693.  I 
could not figure out how to remove those.  You can, for the most part, ignore 
the new files under graphdb\titan0.  There are a few minor changes that have 
been put in since ATLAS-693, but nothing very significant.  Just changes to 
implement a couple new methods in the abstraction layer.

Last rebase: 6/6/2016


Diffs (updated)
-

  addons/falcon-bridge/pom.xml 14c60901a6f194497ca0fb7929a032d2261f85fa 
  addons/hdfs-model/pom.xml f4033bce0fe3ea5a33cfd32ebccd47c01207dc35 
  addons/hdfs-model/src/test/java/org/apache/atlas/fs/model/HDFSModelTest.java 
ac60294e328835ba0340e150799ddfb348ccdb52 
  addons/hive-bridge/pom.xml 47e72e814a421f03b00a9b7c6a13163e1b67ca14 
  addons/sqoop-bridge/pom.xml 343bb4eeae93031ab24cd48ed7b535769150d6c7 
  addons/storm-bridge/pom.xml b1a7a9b1a88442d9e65db4a01b8388c8d3522bfe 
  authorization/pom.xml 74c433bf740a346d1d907a868605e9ab457151bb 
  catalog/pom.xml 8a49d3d673a5936613d45e53479aa99a5a14517d 
  catalog/src/main/java/org/apache/atlas/catalog/DefaultTypeSystem.java 
a28a32b692e61377479cdd432f39ba3c60bd5238 
  catalog/src/main/java/org/apache/atlas/catalog/query/BaseQuery.java 
ba8e0e712769afad72f7f8254ec626a1d17136e8 
  
catalog/src/test/java/org/apache/atlas/catalog/query/AtlasEntityQueryTest.java 
149134c8a79c802fe625d64c5e04af0d88174cfa 
  distro/src/bin/atlas_config.py fab4046345e05e0b7749ca75834c8bdcb0d01752 
  distro/src/conf/atlas-application.properties 
1cdd424d8e126a6bdbf99db8bf4e5d4329e806bb 
  graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasEdge.java 
71b577b5c2e8aee8fec816b9224672ebc2cad5b3 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasEdgeDirection.java
 e7da1c9a5075004973ba68b8a36a2c95756e0e62 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasElement.java 
3c41693e531eb523f04a5c5781f1f123ab38ef07 
  graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 
f312117af9cb6ae8de21d565bfcf3b10d9bdfcc6 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphIndex.java
 PRE-CREATION 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphManagement.java
 2776b2242c4469dcae344616ccfe100633a79432 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphQuery.java
 f0dca9a28eebd8996d55d71dcc3254cce9cd4098

Re: Review Request 47810: ATLAS-694: Update Atlas to use Graph DB abstraction layer


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47810/
---

(Updated June 6, 2016, 8:35 p.m.)


Review request for atlas, David Kantor and Neeru Gupta.


Changes
---

Fix issue in rat license check.


Bugs: ATLAS-694
https://issues.apache.org/jira/browse/ATLAS-694


Repository: atlas


Description
---

ATLAS-694: Update Atlas to use abstraction layer.  All of the Atlas code (with 
the exception of the catalog, which was only updated minimally) has been 
updated to use the graph database abstraction layer.  In addition, there is now 
an optional Atlas configuration property that specifies the class of the 
abstraction layer database to use.  I basically put all of the changes in here 
with the exception of the actual Titan 1 implementation of code.  This includes 
the changes to support Tinkerpop 3 syntax.  This is mostly to expedite getting 
the changes into Atlas.  Originally the TP3 changes were going to be brought in 
as part of the Titan 1 implementation task.

Change Summary:

   - change Atlas classes to use AtlasGraph,AtlasVertex,AtlasEdge, etc instead 
of TitanGraph/Vertex/Edge, etc
   - compile time dependency on titan 0.5.4/TP 2 removed (except in Catalog, 
which was only changed to use AtlasGraphProvider/AtlasGraph)
   - updated DSL translation to generate Gremlin that is compliant with TP3 
when TP3 is being used
   - TitanGraphProvider replaced by AtlasGraphProvider.  Graph database 
implementation is determined from a new optional configuration property
   - search indexer enhanced to wait for all indices to be enabled
   - HiveTitanSample is no longer used by tests.  It has been replaced by 
hive-instances.json (which uses normal Atlas json syntax).  The data is saved 
with a new JSONImporter class.  This was needed because the graphson syntax 
used by HiveTitanSample is not compatible with TP3.  
   - RepositoryMetaModule - GraphBackedSearchIndexer changed to an eager 
singleton to force index creation to happen before we execute any gremlin 
queries (which happens during the type system restoration triggered by the 
DefaultMetadataService constructor)
   - atlas_config.py - minor change to allow Atlas to use a non-default JVM.  
It now supports having the java home in ATLAS_JAVA_HOME.  If that is not set, 
we fall back to the previous behavior looking for JAVA_HOME and then checking 
the path.
   
   
Note - the diff here is cumulative and includes the changes from ATLAS-693.  I 
could not figure out how to remove those.  You can, for the most part, ignore 
the new files under graphdb\titan0.  There are a few minor changes that have 
been put in since ATLAS-693, but nothing very significant.  Just changes to 
implement a couple new methods in the abstraction layer.

Last rebase: 6/6/2016


Diffs (updated)
-

  addons/falcon-bridge/pom.xml 14c60901a6f194497ca0fb7929a032d2261f85fa 
  addons/hdfs-model/pom.xml f4033bce0fe3ea5a33cfd32ebccd47c01207dc35 
  addons/hdfs-model/src/test/java/org/apache/atlas/fs/model/HDFSModelTest.java 
ac60294e328835ba0340e150799ddfb348ccdb52 
  addons/hive-bridge/pom.xml 47e72e814a421f03b00a9b7c6a13163e1b67ca14 
  addons/sqoop-bridge/pom.xml 343bb4eeae93031ab24cd48ed7b535769150d6c7 
  addons/storm-bridge/pom.xml b1a7a9b1a88442d9e65db4a01b8388c8d3522bfe 
  authorization/pom.xml 74c433bf740a346d1d907a868605e9ab457151bb 
  catalog/pom.xml 8a49d3d673a5936613d45e53479aa99a5a14517d 
  catalog/src/main/java/org/apache/atlas/catalog/DefaultTypeSystem.java 
a28a32b692e61377479cdd432f39ba3c60bd5238 
  catalog/src/main/java/org/apache/atlas/catalog/query/BaseQuery.java 
ba8e0e712769afad72f7f8254ec626a1d17136e8 
  
catalog/src/test/java/org/apache/atlas/catalog/query/AtlasEntityQueryTest.java 
149134c8a79c802fe625d64c5e04af0d88174cfa 
  distro/src/bin/atlas_config.py fab4046345e05e0b7749ca75834c8bdcb0d01752 
  distro/src/conf/atlas-application.properties 
1cdd424d8e126a6bdbf99db8bf4e5d4329e806bb 
  graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasEdge.java 
71b577b5c2e8aee8fec816b9224672ebc2cad5b3 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasEdgeDirection.java
 e7da1c9a5075004973ba68b8a36a2c95756e0e62 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasElement.java 
3c41693e531eb523f04a5c5781f1f123ab38ef07 
  graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 
f312117af9cb6ae8de21d565bfcf3b10d9bdfcc6 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphIndex.java
 PRE-CREATION 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphManagement.java
 2776b2242c4469dcae344616ccfe100633a79432 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphQuery.java
 f0dca9a28eebd8996d55d71dcc3254cce9cd4098

Re: Review Request 47810: ATLAS-694: Update Atlas to use Graph DB abstraction layer


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47810/
---

(Updated June 6, 2016, 7:19 p.m.)


Review request for atlas, David Kantor and Neeru Gupta.


Changes
---

Rebase, apply latest changes to graph database abstraction layer.


Bugs: ATLAS-694
https://issues.apache.org/jira/browse/ATLAS-694


Repository: atlas


Description
---

ATLAS-694: Update Atlas to use abstraction layer.  All of the Atlas code (with 
the exception of the catalog, which was only updated minimally) has been 
updated to use the graph database abstraction layer.  In addition, there is now 
a required Atlas configuration property that specifies the class of the 
abstraction layer database to use.  I basically put all of the changes in here 
with the exception of the actual Titan 1 implementation of code.  This includes 
the changes to support Tinkerpop 3 syntax.  This is mostly to expedite getting 
the changes into Atlas.  Originally the TP3 changes were going to be brought in 
as part of the Titan 1 implementation task.

Change Summary:

   - change Atlas classes to use AtlasGraph,AtlasVertex,AtlasEdge, etc instead 
of TitanGraph/Vertex/Edge, etc
   - compile time dependency on titan 0.5.4/TP 2 removed (except in Catalog, 
which was only changed to use AtlasGraphProvider/AtlasGraph)
   - updated DSL translation to generate Gremlin that is compliant with TP3 
when TP3 is being used
   - TitanGraphProvider replaced by AtlasGraphProvider.  Graph database 
implementation is determined from a new required configuration property
   - search indexer enhanced to wait for all indices to be enabled
   - HiveTitanSample is no longer used by tests.  It has been replaced by 
hive-instances.json (which uses normal Atlas json syntax).  The data is saved 
with a new JSONImporter class.  This was needed because the graphson syntax 
used by HiveTitanSample is not compatible with TP3.  
   - RepositoryMetaModule - GraphBackedSearchIndexer changed to an eager 
singleton to force index creation to happen before we execute any gremlin 
queries (which happens during the type system restoration triggered by the 
DefaultMetadataService constructor)
   - atlas_config.py - minor change to allow Atlas to use a non-default JVM.  
It now supports having the java home in ATLAS_JAVA_HOME.  If that is not set, 
we fall back to the previous behavior looking for JAVA_HOME and then checking 
the path.
   
Note - the diff here is cumulative and includes the changes from ATLAS-693.  I 
could not figure out how to remove those.

Last rebase: 5/27/2016


Diffs (updated)
-

  addons/falcon-bridge/pom.xml 14c60901a6f194497ca0fb7929a032d2261f85fa 
  addons/hdfs-model/pom.xml f4033bce0fe3ea5a33cfd32ebccd47c01207dc35 
  addons/hdfs-model/src/test/java/org/apache/atlas/fs/model/HDFSModelTest.java 
ac60294e328835ba0340e150799ddfb348ccdb52 
  addons/hive-bridge/pom.xml 47e72e814a421f03b00a9b7c6a13163e1b67ca14 
  addons/sqoop-bridge/pom.xml 343bb4eeae93031ab24cd48ed7b535769150d6c7 
  addons/storm-bridge/pom.xml b1a7a9b1a88442d9e65db4a01b8388c8d3522bfe 
  authorization/pom.xml 74c433bf740a346d1d907a868605e9ab457151bb 
  catalog/pom.xml 8a49d3d673a5936613d45e53479aa99a5a14517d 
  catalog/src/main/java/org/apache/atlas/catalog/DefaultTypeSystem.java 
a28a32b692e61377479cdd432f39ba3c60bd5238 
  catalog/src/main/java/org/apache/atlas/catalog/query/BaseQuery.java 
ba8e0e712769afad72f7f8254ec626a1d17136e8 
  
catalog/src/test/java/org/apache/atlas/catalog/query/AtlasEntityQueryTest.java 
149134c8a79c802fe625d64c5e04af0d88174cfa 
  distro/src/bin/atlas_config.py fab4046345e05e0b7749ca75834c8bdcb0d01752 
  distro/src/conf/atlas-application.properties 
1cdd424d8e126a6bdbf99db8bf4e5d4329e806bb 
  graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasEdge.java 
71b577b5c2e8aee8fec816b9224672ebc2cad5b3 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasEdgeDirection.java
 e7da1c9a5075004973ba68b8a36a2c95756e0e62 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasElement.java 
3c41693e531eb523f04a5c5781f1f123ab38ef07 
  graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 
f312117af9cb6ae8de21d565bfcf3b10d9bdfcc6 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphIndex.java
 PRE-CREATION 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphManagement.java
 2776b2242c4469dcae344616ccfe100633a79432 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphQuery.java
 f0dca9a28eebd8996d55d71dcc3254cce9cd4098 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasIndexQuery.java
 7ee6b2831a7610e7d5b1d36ae69282a9315dc06b 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasPropertyKey.java
 PRE-CREATION

Re: Review Request 47810: ATLAS-694: Update Atlas to use Graph DB abstraction layer


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47810/
---

(Updated May 27, 2016, 6:20 p.m.)


Review request for atlas, David Kantor and Neeru Gupta.


Changes
---

Fixed checkstyle issues.


Bugs: ATLAS-694
https://issues.apache.org/jira/browse/ATLAS-694


Repository: atlas


Description
---

ATLAS-694: Update Atlas to use abstraction layer.  All of the Atlas code (with 
the exception of the catalog, which was only updated minimally) has been 
updated to use the graph database abstraction layer.  In addition, there is now 
a required Atlas configuration property that specifies the class of the 
abstraction layer database to use.  I basically put all of the changes in here 
with the exception of the actual Titan 1 implementation of code.  This includes 
the changes to support Tinkerpop 3 syntax.  This is mostly to expedite getting 
the changes into Atlas.  Originally the TP3 changes were going to be brought in 
as part of the Titan 1 implementation task.

Change Summary:

   - change Atlas classes to use AtlasGraph,AtlasVertex,AtlasEdge, etc instead 
of TitanGraph/Vertex/Edge, etc
   - compile time dependency on titan 0.5.4/TP 2 removed (except in Catalog, 
which was only changed to use AtlasGraphProvider/AtlasGraph)
   - updated DSL translation to generate Gremlin that is compliant with TP3 
when TP3 is being used
   - TitanGraphProvider replaced by AtlasGraphProvider.  Graph database 
implementation is determined from a new required configuration property
   - search indexer enhanced to wait for all indices to be enabled
   - HiveTitanSample is no longer used by tests.  It has been replaced by 
hive-instances.json (which uses normal Atlas json syntax).  The data is saved 
with a new JSONImporter class.  This was needed because the graphson syntax 
used by HiveTitanSample is not compatible with TP3.  
   - RepositoryMetaModule - GraphBackedSearchIndexer changed to an eager 
singleton to force index creation to happen before we execute any gremlin 
queries (which happens during the type system restoration triggered by the 
DefaultMetadataService constructor)
   - atlas_config.py - minor change to allow Atlas to use a non-default JVM.  
It now supports having the java home in ATLAS_JAVA_HOME.  If that is not set, 
we fall back to the previous behavior looking for JAVA_HOME and then checking 
the path.
   
Note - the diff here is cumulative and includes the changes from ATLAS-693.  I 
could not figure out how to remove those.

Last rebase: 5/27/2016


Diffs (updated)
-

  addons/falcon-bridge/pom.xml 14c60901a6f194497ca0fb7929a032d2261f85fa 
  addons/hdfs-model/pom.xml f4033bce0fe3ea5a33cfd32ebccd47c01207dc35 
  addons/hdfs-model/src/test/java/org/apache/atlas/fs/model/HDFSModelTest.java 
ac60294e328835ba0340e150799ddfb348ccdb52 
  addons/hive-bridge/pom.xml 47e72e814a421f03b00a9b7c6a13163e1b67ca14 
  addons/sqoop-bridge/pom.xml 343bb4eeae93031ab24cd48ed7b535769150d6c7 
  addons/storm-bridge/pom.xml b1a7a9b1a88442d9e65db4a01b8388c8d3522bfe 
  authorization/pom.xml 74c433bf740a346d1d907a868605e9ab457151bb 
  catalog/pom.xml 8a49d3d673a5936613d45e53479aa99a5a14517d 
  catalog/src/main/java/org/apache/atlas/catalog/query/BaseQuery.java 
c2eabf70de57f2893a4c30a045fa8f10aaf0b837 
  
catalog/src/test/java/org/apache/atlas/catalog/query/AtlasEntityQueryTest.java 
838d4c16579efcc2519d668149610edfe78f83c5 
  distro/pom.xml 22f70ae970d3ee4ce3d56d15f0e9df5a3fa21b94 
  distro/src/bin/atlas_config.py fab4046345e05e0b7749ca75834c8bdcb0d01752 
  distro/src/conf/atlas-application.properties 
bb12538197ea47aad0cad013529c6821e1b1a766 
  graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasEdge.java 
71b577b5c2e8aee8fec816b9224672ebc2cad5b3 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasEdgeDirection.java
 e7da1c9a5075004973ba68b8a36a2c95756e0e62 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasElement.java 
3c41693e531eb523f04a5c5781f1f123ab38ef07 
  graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 
f312117af9cb6ae8de21d565bfcf3b10d9bdfcc6 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphIndex.java
 PRE-CREATION 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphManagement.java
 2776b2242c4469dcae344616ccfe100633a79432 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphQuery.java
 f0dca9a28eebd8996d55d71dcc3254cce9cd4098 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasIndexQuery.java
 7ee6b2831a7610e7d5b1d36ae69282a9315dc06b 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasPropertyKey.java
 PRE-CREATION 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasSchemaViolationException.java
 fda83b807f3640c04a660001941376f41be6d9be

Re: Review Request 47810: ATLAS-694: Update Atlas to use Graph DB abstraction layer


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47810/
---

(Updated May 27, 2016, 4:07 p.m.)


Review request for atlas, David Kantor and Neeru Gupta.


Bugs: ATLAS-694
https://issues.apache.org/jira/browse/ATLAS-694


Repository: atlas


Description (updated)
---

ATLAS-694: Update Atlas to use abstraction layer.  All of the Atlas code (with 
the exception of the catalog, which was only updated minimally) has been 
updated to use the graph database abstraction layer.  In addition, there is now 
a required Atlas configuration property that specifies the class of the 
abstraction layer database to use.  I basically put all of the changes in here 
with the exception of the actual Titan 1 implementation of code.  This includes 
the changes to support Tinkerpop 3 syntax.  This is mostly to expedite getting 
the changes into Atlas.  Originally the TP3 changes were going to be brought in 
as part of the Titan 1 implementation task.

Change Summary:

   - change Atlas classes to use AtlasGraph,AtlasVertex,AtlasEdge, etc instead 
of TitanGraph/Vertex/Edge, etc
   - compile time dependency on titan 0.5.4/TP 2 removed (except in Catalog, 
which was only changed to use AtlasGraphProvider/AtlasGraph)
   - updated DSL translation to generate Gremlin that is compliant with TP3 
when TP3 is being used
   - TitanGraphProvider replaced by AtlasGraphProvider.  Graph database 
implementation is determined from a new required configuration property
   - search indexer enhanced to wait for all indices to be enabled
   - HiveTitanSample is no longer used by tests.  It has been replaced by 
hive-instances.json (which uses normal Atlas json syntax).  The data is saved 
with a new JSONImporter class.  This was needed because the graphson syntax 
used by HiveTitanSample is not compatible with TP3.  
   - RepositoryMetaModule - GraphBackedSearchIndexer changed to an eager 
singleton to force index creation to happen before we execute any gremlin 
queries (which happens during the type system restoration triggered by the 
DefaultMetadataService constructor)
   - atlas_config.py - minor change to allow Atlas to use a non-default JVM.  
It now supports having the java home in ATLAS_JAVA_HOME.  If that is not set, 
we fall back to the previous behavior looking for JAVA_HOME and then checking 
the path.
   
Note - the diff here is cumulative and includes the changes from ATLAS-693.  I 
could not figure out how to remove those.

Last rebase: 5/27/2016


Diffs
-

  addons/falcon-bridge/pom.xml 14c60901a6f194497ca0fb7929a032d2261f85fa 
  addons/hdfs-model/pom.xml f4033bce0fe3ea5a33cfd32ebccd47c01207dc35 
  addons/hdfs-model/src/test/java/org/apache/atlas/fs/model/HDFSModelTest.java 
ac60294e328835ba0340e150799ddfb348ccdb52 
  addons/hive-bridge/pom.xml 47e72e814a421f03b00a9b7c6a13163e1b67ca14 
  addons/sqoop-bridge/pom.xml 343bb4eeae93031ab24cd48ed7b535769150d6c7 
  addons/storm-bridge/pom.xml b1a7a9b1a88442d9e65db4a01b8388c8d3522bfe 
  authorization/pom.xml 74c433bf740a346d1d907a868605e9ab457151bb 
  catalog/pom.xml 8a49d3d673a5936613d45e53479aa99a5a14517d 
  catalog/src/main/java/org/apache/atlas/catalog/query/BaseQuery.java 
c2eabf70de57f2893a4c30a045fa8f10aaf0b837 
  
catalog/src/test/java/org/apache/atlas/catalog/query/AtlasEntityQueryTest.java 
838d4c16579efcc2519d668149610edfe78f83c5 
  distro/pom.xml 22f70ae970d3ee4ce3d56d15f0e9df5a3fa21b94 
  distro/src/bin/atlas_config.py fab4046345e05e0b7749ca75834c8bdcb0d01752 
  distro/src/conf/atlas-application.properties 
bb12538197ea47aad0cad013529c6821e1b1a766 
  graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasEdge.java 
71b577b5c2e8aee8fec816b9224672ebc2cad5b3 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasEdgeDirection.java
 e7da1c9a5075004973ba68b8a36a2c95756e0e62 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasElement.java 
3c41693e531eb523f04a5c5781f1f123ab38ef07 
  graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 
f312117af9cb6ae8de21d565bfcf3b10d9bdfcc6 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphIndex.java
 PRE-CREATION 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphManagement.java
 2776b2242c4469dcae344616ccfe100633a79432 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphQuery.java
 f0dca9a28eebd8996d55d71dcc3254cce9cd4098 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasIndexQuery.java
 7ee6b2831a7610e7d5b1d36ae69282a9315dc06b 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasPropertyKey.java
 PRE-CREATION 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasSchemaViolationException.java
 fda83b807f3640c04a660001941376f41be6d9be

Re: Review Request 47810: ATLAS-694: Update Atlas to use Graph DB abstraction layer


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47810/
---

(Updated May 27, 2016, 4:04 p.m.)


Review request for atlas, David Kantor and Neeru Gupta.


Bugs: ATLAS-694
https://issues.apache.org/jira/browse/ATLAS-694


Repository: atlas


Description (updated)
---

ATLAS-694: Update Atlas to use abstraction layer.  All of the Atlas code (with 
the exception of the catalog, which was only updated minimally) has been 
updated to use the graph database abstraction layer.  In addition, there is now 
a required Atlas configuration property that specifies the class of the 
abstraction layer database to use.  I basically put all of the changes in here 
with the exception of the actual Titan 1 implementation of code.  This includes 
the changes to support Tinkerpop 3 syntax.  This is mostly to expedite getting 
the changes into Atlas.  Originally the TP3 changes were going to be brought in 
as part of the Titan 1 implementation task.

Change Summary:

   - change Atlas classes to use AtlasGraph,AtlasVertex,AtlasEdge, etc instead 
of TitanGraph/Vertex/Edge, etc
   - compile time dependency on titan 0.5.4/TP 2 removed (except in Catalog, 
which was only changed to use AtlasGraphProvider/AtlasGraph)
   - updated DSL translation to generate Gremlin that is compliant with TP3 
when TP3 is being used
   - TitanGraphProvider replaced by AtlasGraphProvider.  Graph database 
implementation is determined from a new required configuration property
   - search indexer enhanced to wait for all indices to be enabled
   - HiveTitanSample is no longer used by tests.  It has been replaced by 
hive-instances.json (which uses normal Atlas json syntax).  The data is saved 
with a new JSONImporter class.  This was needed because the graphson syntax 
used by HiveTitanSample is not compatible with TP3.  
   - RepositoryMetaModule - GraphBackedSearchIndexer changed to an eager 
singleton to force index creation to happen before we execute any gremlin 
queries (which happens during the type system restoration triggered by the 
DefaultMetadataService constructor)
   - atlas_config.py - minor change to allow Atlas to use a non-default JVM.  
It now supports having the java home in ATLAS_JAVA_HOME.  If that is not set, 
we fall back to the previous behavior looking for JAVA_HOME and then checking 
the path.
   
Note - the diff here is cumulative and includes the changes from ATLAS-693.  I 
could not figure out how to remove those.


Diffs
-

  addons/falcon-bridge/pom.xml 14c60901a6f194497ca0fb7929a032d2261f85fa 
  addons/hdfs-model/pom.xml f4033bce0fe3ea5a33cfd32ebccd47c01207dc35 
  addons/hdfs-model/src/test/java/org/apache/atlas/fs/model/HDFSModelTest.java 
ac60294e328835ba0340e150799ddfb348ccdb52 
  addons/hive-bridge/pom.xml 47e72e814a421f03b00a9b7c6a13163e1b67ca14 
  addons/sqoop-bridge/pom.xml 343bb4eeae93031ab24cd48ed7b535769150d6c7 
  addons/storm-bridge/pom.xml b1a7a9b1a88442d9e65db4a01b8388c8d3522bfe 
  authorization/pom.xml 74c433bf740a346d1d907a868605e9ab457151bb 
  catalog/pom.xml 8a49d3d673a5936613d45e53479aa99a5a14517d 
  catalog/src/main/java/org/apache/atlas/catalog/query/BaseQuery.java 
c2eabf70de57f2893a4c30a045fa8f10aaf0b837 
  
catalog/src/test/java/org/apache/atlas/catalog/query/AtlasEntityQueryTest.java 
838d4c16579efcc2519d668149610edfe78f83c5 
  distro/pom.xml 22f70ae970d3ee4ce3d56d15f0e9df5a3fa21b94 
  distro/src/bin/atlas_config.py fab4046345e05e0b7749ca75834c8bdcb0d01752 
  distro/src/conf/atlas-application.properties 
bb12538197ea47aad0cad013529c6821e1b1a766 
  graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasEdge.java 
71b577b5c2e8aee8fec816b9224672ebc2cad5b3 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasEdgeDirection.java
 e7da1c9a5075004973ba68b8a36a2c95756e0e62 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasElement.java 
3c41693e531eb523f04a5c5781f1f123ab38ef07 
  graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 
f312117af9cb6ae8de21d565bfcf3b10d9bdfcc6 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphIndex.java
 PRE-CREATION 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphManagement.java
 2776b2242c4469dcae344616ccfe100633a79432 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphQuery.java
 f0dca9a28eebd8996d55d71dcc3254cce9cd4098 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasIndexQuery.java
 7ee6b2831a7610e7d5b1d36ae69282a9315dc06b 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasPropertyKey.java
 PRE-CREATION 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasSchemaViolationException.java
 fda83b807f3640c04a660001941376f41be6d9be

Re: Review Request 47810: ATLAS-694: Update Atlas to use Graph DB abstraction layer


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47810/
---

(Updated May 27, 2016, 4:01 p.m.)


Review request for atlas, David Kantor and Neeru Gupta.


Changes
---

Fix checkstyle violations from ATLAS-693.


Bugs: ATLAS-694
https://issues.apache.org/jira/browse/ATLAS-694


Repository: atlas


Description
---

ATLAS-694: Update Atlas to use abstraction layer.  All of the Atlas code (with 
the exception of the catalog, which was only updated minimally) has been 
updated to use the graph database abstraction layer.  In addition, there is now 
a required Atlas configuration property that specifies the class of the 
abstraction layer database to use.  I basically put all of the changes in here 
with the exception of the actual Titan 1 implementation of code.  This includes 
the changes to support Tinkerpop 3 syntax.  This is mostly to expedite getting 
the changes into Atlas.  Originally the TP3 changes were going to be brought in 
as part of the Titan 1 implementation task.

Change Summary:

   - change Atlas classes to use AtlasGraph,AtlasVertex,AtlasEdge, etc instead 
of TitanGraph/Vertex/Edge, etc
   - compile time dependency on titan 0.5.4/TP 2 removed (except in Catalog, 
which was only changed to use AtlasGraphProvider/AtlasGraph)
   - updated DSL translation to generate Gremlin that is compliant with TP3 
when TP3 is being used
   - TitanGraphProvider replaced by AtlasGraphProvider.  Graph database 
implementation is determined from a new required configuration property
   - search indexer enhanced to wait for all indices to be enabled
   - HiveTitanSample is no longer used by tests.  It has been replaced by 
hive-instances.json (which uses normal Atlas json syntax).  The data is saved 
with a new JSONImporter class.  This was needed because the graphson syntax 
used by HiveTitanSample is not compatible with TP3.  
   - RepositoryMetaModule - GraphBackedSearchIndexer changed to an eager 
singleton to force index creation to happen before we execute any gremlin 
queries (which happens during the type system restoration triggered by the 
DefaultMetadataService constructor)
   - atlas_config.py - minor change to allow Atlas to use a non-default JVM.  
It now supports having the java home in ATLAS_JAVA_HOME.  If that is not set, 
we fall back to the previous behavior looking for JAVA_HOME and then checking 
the path.
   -
Note - I did remove the spurious whitespace.  So far I have not figured out how 
to update the diff.  It's kind of trickly since I only want the commits since 
the last diff generated for ATLAS-693.


Diffs (updated)
-

  addons/falcon-bridge/pom.xml 14c60901a6f194497ca0fb7929a032d2261f85fa 
  addons/hdfs-model/pom.xml f4033bce0fe3ea5a33cfd32ebccd47c01207dc35 
  addons/hdfs-model/src/test/java/org/apache/atlas/fs/model/HDFSModelTest.java 
ac60294e328835ba0340e150799ddfb348ccdb52 
  addons/hive-bridge/pom.xml 47e72e814a421f03b00a9b7c6a13163e1b67ca14 
  addons/sqoop-bridge/pom.xml 343bb4eeae93031ab24cd48ed7b535769150d6c7 
  addons/storm-bridge/pom.xml b1a7a9b1a88442d9e65db4a01b8388c8d3522bfe 
  authorization/pom.xml 74c433bf740a346d1d907a868605e9ab457151bb 
  catalog/pom.xml 8a49d3d673a5936613d45e53479aa99a5a14517d 
  catalog/src/main/java/org/apache/atlas/catalog/query/BaseQuery.java 
c2eabf70de57f2893a4c30a045fa8f10aaf0b837 
  
catalog/src/test/java/org/apache/atlas/catalog/query/AtlasEntityQueryTest.java 
838d4c16579efcc2519d668149610edfe78f83c5 
  distro/pom.xml 22f70ae970d3ee4ce3d56d15f0e9df5a3fa21b94 
  distro/src/bin/atlas_config.py fab4046345e05e0b7749ca75834c8bdcb0d01752 
  distro/src/conf/atlas-application.properties 
bb12538197ea47aad0cad013529c6821e1b1a766 
  graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasEdge.java 
71b577b5c2e8aee8fec816b9224672ebc2cad5b3 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasEdgeDirection.java
 e7da1c9a5075004973ba68b8a36a2c95756e0e62 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasElement.java 
3c41693e531eb523f04a5c5781f1f123ab38ef07 
  graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 
f312117af9cb6ae8de21d565bfcf3b10d9bdfcc6 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphIndex.java
 PRE-CREATION 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphManagement.java
 2776b2242c4469dcae344616ccfe100633a79432 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphQuery.java
 f0dca9a28eebd8996d55d71dcc3254cce9cd4098 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasIndexQuery.java
 7ee6b2831a7610e7d5b1d36ae69282a9315dc06b 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasPropertyKey.java
 PRE-CREATION

Re: Review Request 47810: ATLAS-694: Update Atlas to use Graph DB abstraction layer


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47810/
---

(Updated May 27, 2016, 3:48 p.m.)


Review request for atlas, David Kantor and Neeru Gupta.


Bugs: ATLAS-694
https://issues.apache.org/jira/browse/ATLAS-694


Repository: atlas


Description
---

ATLAS-694: Update Atlas to use abstraction layer.  All of the Atlas code (with 
the exception of the catalog, which was only updated minimally) has been 
updated to use the graph database abstraction layer.  In addition, there is now 
a required Atlas configuration property that specifies the class of the 
abstraction layer database to use.  I basically put all of the changes in here 
with the exception of the actual Titan 1 implementation of code.  This includes 
the changes to support Tinkerpop 3 syntax.  This is mostly to expedite getting 
the changes into Atlas.  Originally the TP3 changes were going to be brought in 
as part of the Titan 1 implementation task.

Change Summary:

   - change Atlas classes to use AtlasGraph,AtlasVertex,AtlasEdge, etc instead 
of TitanGraph/Vertex/Edge, etc
   - compile time dependency on titan 0.5.4/TP 2 removed (except in Catalog, 
which was only changed to use AtlasGraphProvider/AtlasGraph)
   - updated DSL translation to generate Gremlin that is compliant with TP3 
when TP3 is being used
   - TitanGraphProvider replaced by AtlasGraphProvider.  Graph database 
implementation is determined from a new required configuration property
   - search indexer enhanced to wait for all indices to be enabled
   - HiveTitanSample is no longer used by tests.  It has been replaced by 
hive-instances.json (which uses normal Atlas json syntax).  The data is saved 
with a new JSONImporter class.  This was needed because the graphson syntax 
used by HiveTitanSample is not compatible with TP3.  
   - RepositoryMetaModule - GraphBackedSearchIndexer changed to an eager 
singleton to force index creation to happen before we execute any gremlin 
queries (which happens during the type system restoration triggered by the 
DefaultMetadataService constructor)
   - atlas_config.py - minor change to allow Atlas to use a non-default JVM.  
It now supports having the java home in ATLAS_JAVA_HOME.  If that is not set, 
we fall back to the previous behavior looking for JAVA_HOME and then checking 
the path.
   -
Note - I did remove the spurious whitespace.  So far I have not figured out how 
to update the diff.  It's kind of trickly since I only want the commits since 
the last diff generated for ATLAS-693.


Diffs (updated)
-

  addons/falcon-bridge/pom.xml 14c60901a6f194497ca0fb7929a032d2261f85fa 
  addons/hdfs-model/pom.xml f4033bce0fe3ea5a33cfd32ebccd47c01207dc35 
  addons/hdfs-model/src/test/java/org/apache/atlas/fs/model/HDFSModelTest.java 
ac60294e328835ba0340e150799ddfb348ccdb52 
  addons/hive-bridge/pom.xml 47e72e814a421f03b00a9b7c6a13163e1b67ca14 
  addons/sqoop-bridge/pom.xml 343bb4eeae93031ab24cd48ed7b535769150d6c7 
  addons/storm-bridge/pom.xml b1a7a9b1a88442d9e65db4a01b8388c8d3522bfe 
  authorization/pom.xml 74c433bf740a346d1d907a868605e9ab457151bb 
  catalog/pom.xml 8a49d3d673a5936613d45e53479aa99a5a14517d 
  catalog/src/main/java/org/apache/atlas/catalog/query/BaseQuery.java 
c2eabf70de57f2893a4c30a045fa8f10aaf0b837 
  
catalog/src/test/java/org/apache/atlas/catalog/query/AtlasEntityQueryTest.java 
838d4c16579efcc2519d668149610edfe78f83c5 
  distro/pom.xml 22f70ae970d3ee4ce3d56d15f0e9df5a3fa21b94 
  distro/src/bin/atlas_config.py fab4046345e05e0b7749ca75834c8bdcb0d01752 
  distro/src/conf/atlas-application.properties 
bb12538197ea47aad0cad013529c6821e1b1a766 
  graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasEdge.java 
71b577b5c2e8aee8fec816b9224672ebc2cad5b3 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasEdgeDirection.java
 e7da1c9a5075004973ba68b8a36a2c95756e0e62 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasElement.java 
3c41693e531eb523f04a5c5781f1f123ab38ef07 
  graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 
f312117af9cb6ae8de21d565bfcf3b10d9bdfcc6 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphIndex.java
 PRE-CREATION 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphManagement.java
 2776b2242c4469dcae344616ccfe100633a79432 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphQuery.java
 f0dca9a28eebd8996d55d71dcc3254cce9cd4098 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasIndexQuery.java
 7ee6b2831a7610e7d5b1d36ae69282a9315dc06b 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasPropertyKey.java
 PRE-CREATION 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasSchemaViolationException.java

Re: Review Request 47810: ATLAS-694: Update Atlas to use Graph DB abstraction layer



> On May 25, 2016, 5:03 p.m., David Kantor wrote:
> > distro/src/bin/atlas_config.py, line 162
> > 
> >
> > Perhaps add comments here explaining the need/motivation for 
> > ATLAS_JAVA_HOME.

I added an ATLAS_JAVA_HOME as a convenient way to configure only Atlas to use a 
specific JVM.  Many environments have multiple JVMs installed, and different 
software has different JVM requirements.  In environments where the default JVM 
is not compatible with Atlas, this provides a way to configure Atlas to use a 
specific JVM without affecting other applications.  This is also a convenient 
way for developers to test Atlas with a specific JVM as well.


> On May 25, 2016, 5:03 p.m., David Kantor wrote:
> > repository/src/main/java/org/apache/atlas/repository/graph/AtlasGraphProvider.java,
> >  line 18
> > 
> >
> > Please rename this, as plugin is an overloaded term.

plugin_ was renamed to graph_


> On May 25, 2016, 5:03 p.m., David Kantor wrote:
> > repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java,
> >  line 470
> > 
> >
> > What is the use case for this method?

This was refactored out of DefaultMetadataService.  It is used when converting 
the json received from clients into ITypedReferenceableInstances.  I refactored 
it so that it could be reused by the JSONImporter class that I added.


- Jeff


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47810/#review134773
---


On May 27, 2016, 3:30 p.m., Jeff Hagelberg wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47810/
> ---
> 
> (Updated May 27, 2016, 3:30 p.m.)
> 
> 
> Review request for atlas, David Kantor and Neeru Gupta.
> 
> 
> Bugs: ATLAS-694
> https://issues.apache.org/jira/browse/ATLAS-694
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> ATLAS-694: Update Atlas to use abstraction layer.  All of the Atlas code 
> (with the exception of the catalog, which was only updated minimally) has 
> been updated to use the graph database abstraction layer.  In addition, there 
> is now a required Atlas configuration property that specifies the class of 
> the abstraction layer database to use.  I basically put all of the changes in 
> here with the exception of the actual Titan 1 implementation of code.  This 
> includes the changes to support Tinkerpop 3 syntax.  This is mostly to 
> expedite getting the changes into Atlas.  Originally the TP3 changes were 
> going to be brought in as part of the Titan 1 implementation task.
> 
> Change Summary:
> 
>- change Atlas classes to use AtlasGraph,AtlasVertex,AtlasEdge, etc 
> instead of TitanGraph/Vertex/Edge, etc
>- compile time dependency on titan 0.5.4/TP 2 removed (except in Catalog, 
> which was only changed to use AtlasGraphProvider/AtlasGraph)
>- updated DSL translation to generate Gremlin that is compliant with TP3 
> when TP3 is being used
>- TitanGraphProvider replaced by AtlasGraphProvider.  Graph database 
> implementation is determined from a new required configuration property
>- search indexer enhanced to wait for all indices to be enabled
>- HiveTitanSample is no longer used by tests.  It has been replaced by 
> hive-instances.json (which uses normal Atlas json syntax).  The data is saved 
> with a new JSONImporter class.  This was needed because the graphson syntax 
> used by HiveTitanSample is not compatible with TP3.  
>- RepositoryMetaModule - GraphBackedSearchIndexer changed to an eager 
> singleton to force index creation to happen before we execute any gremlin 
> queries (which happens during the type system restoration triggered by the 
> DefaultMetadataService constructor)
>- atlas_config.py - minor change to allow Atlas to use a non-default JVM.  
> It now supports having the java home in ATLAS_JAVA_HOME.  If that is not set, 
> we fall back to the previous behavior looking for JAVA_HOME and then checking 
> the path.
>-
> Note - I did remove the spurious whitespace.  So far I have not figured out 
> how to update the diff.  It's kind of trickly since I only want the commits 
> since the last diff generated for ATLAS-693.
> 
> 
> Diffs
> -
> 
>   addons/falcon-bridge/pom.xml 14c60901a6f194497ca0fb7929a032d2261f85fa 
>   addons/hdfs-model/pom.xml f4033bce0fe3ea5a33cfd32ebccd47c01207dc35 
>   
> addons/hdfs-model/src/test/java/org/apache/atlas/fs/model/HDFSModelTest.java 
> ac60294e328835ba0340e150799ddfb348ccdb52 
>   addons/hive-bridge/pom.xml

Re: Review Request 47810: ATLAS-694: Update Atlas to use Graph DB abstraction layer