Re: Review Request 52077: Column level lineage in Hive

[Shwetha GS]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52077/#review149804
---




addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/ColumnLineageUtils.java
 (line 73)


Don't add toString() to arguments



addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java (line 
634)


Why have you changed Entity update request to entity create request? It 
should be update



addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java (line 
819)


e.key() is the output column name? e.key doesn't contain cluster name, so 
might get de-duped across other lineage processes.

Qualified name for lineage process should be
'qualified name of hive command process : column name'.



addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java (line 
1044)


Use syntax: LOG.debug("Column Lineage Map  - {}", 
this.lineageInfo.entrySet()); so that toString() is not evaluated if debug log 
level is not enabled



addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 
(line 1121)


rename to testColumnLevelLineage



addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 
(line 1143)


Also assert on lineage API response on columns


- Shwetha GS


On Sept. 20, 2016, 9:07 a.m., Vimal Sharma wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/52077/
> ---
> 
> (Updated Sept. 20, 2016, 9:07 a.m.)
> 
> 
> Review request for atlas.
> 
> 
> Bugs: ATLAS-247
> https://issues.apache.org/jira/browse/ATLAS-247
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> After a CTAS query, lineage relationship between source columns and 
> destination column can be captured. This information can be used to create a 
> column lineage process.
> 
> 
> Diffs
> -
> 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/ColumnLineageUtils.java
>  PRE-CREATION 
>   addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
> a3464a0 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/model/HiveDataModelGenerator.java
>  45f0bc9 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/model/HiveDataTypes.java
>  e094cb6 
>   addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 
> a5838b4 
> 
> Diff: https://reviews.apache.org/r/52077/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Vimal Sharma
> 
>

[jira] [Created] (ATLAS-1183) UI: help link should point to atlas website

[Shwetha G S (JIRA)]

Shwetha G S created ATLAS-1183:
--

 Summary: UI: help link should point to atlas website
 Key: ATLAS-1183
 URL: https://issues.apache.org/jira/browse/ATLAS-1183
 Project: Atlas
  Issue Type: Bug
Reporter: Shwetha G S
 Fix For: 0.8-incubating


Help link points to 
https://cwiki.apache.org/confluence/pages/viewpage.action?spaceKey=ATLAS=Atlas+Home
 which requires login. Instead, link to http://atlas.incubator.apache.org/



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (ATLAS-1184) ReservedTypesRegistrar checks for existence of 1st class type

[Shwetha G S (JIRA)]

Shwetha G S created ATLAS-1184:
--

 Summary: ReservedTypesRegistrar checks for existence of 1st class 
type
 Key: ATLAS-1184
 URL: https://issues.apache.org/jira/browse/ATLAS-1184
 Project: Atlas
  Issue Type: Bug
Reporter: Shwetha G S


ReservedTypesRegistrar registers the models for supported integrations like 
hive, falcon, sqoop and storm. ReservedTypesRegistrar reads the model files and 
registers the types defined in the model file. The check thats done to verify 
if the model is already registered - checks for existence of first class type. 
This will not work if we add more class types to the existing model or modify 
the existing type to add optional attributes. 

ReservedTypesRegistrar should check for every type in the model file. Also, 
ReservedTypesRegistrar should do type update which does create if not exists, 
and update if exists



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (ATLAS-1184) ReservedTypesRegistrar checks for existence of 1st class type

[Vimal Sharma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-1184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vimal Sharma reassigned ATLAS-1184:
---

Assignee: Vimal Sharma

> ReservedTypesRegistrar checks for existence of 1st class type
> -
>
> Key: ATLAS-1184
> URL: https://issues.apache.org/jira/browse/ATLAS-1184
> Project: Atlas
>  Issue Type: Bug
>Reporter: Shwetha G S
>Assignee: Vimal Sharma
>
> ReservedTypesRegistrar registers the models for supported integrations like 
> hive, falcon, sqoop and storm. ReservedTypesRegistrar reads the model files 
> and registers the types defined in the model file. The check thats done to 
> verify if the model is already registered - checks for existence of first 
> class type. This will not work if we add more class types to the existing 
> model or modify the existing type to add optional attributes. 
> ReservedTypesRegistrar should check for every type in the model file. Also, 
> ReservedTypesRegistrar should do type update which does create if not exists, 
> and update if exists



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (ATLAS-1185) UI: columns attribute displayed for all entities

[Shwetha G S (JIRA)]

Shwetha G S created ATLAS-1185:
--

 Summary: UI: columns attribute displayed for all entities
 Key: ATLAS-1185
 URL: https://issues.apache.org/jira/browse/ATLAS-1185
 Project: Atlas
  Issue Type: Bug
Reporter: Shwetha G S
 Attachments: Screen Shot 2016-09-21 at 5.06.29 PM.png

In the entity details page, columns attribute is displayed without any value. 
Looks like this is shown for all entity types



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (ATLAS-1185) UI: columns attribute displayed for all entities

[Shwetha G S (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-1185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shwetha G S updated ATLAS-1185:
---
Attachment: Screen Shot 2016-09-21 at 5.06.29 PM.png

> UI: columns attribute displayed for all entities
> 
>
> Key: ATLAS-1185
> URL: https://issues.apache.org/jira/browse/ATLAS-1185
> Project: Atlas
>  Issue Type: Bug
>Reporter: Shwetha G S
> Attachments: Screen Shot 2016-09-21 at 5.06.29 PM.png
>
>
> In the entity details page, columns attribute is displayed without any value. 
> Looks like this is shown for all entity types



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (ATLAS-1186) Add Business Category

[David Radley (JIRA)]

David Radley created ATLAS-1186:
---

 Summary: Add Business Category
 Key: ATLAS-1186
 URL: https://issues.apache.org/jira/browse/ATLAS-1186
 Project: Atlas
  Issue Type: New Feature
Reporter: David Radley


Add Business Category, which would have a name and description and be hung off 
a taxonomy. The category would contain (composition type) sub categories. 
Categories can contain terms.  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 51939: Framework to apply updates to types in the type-system

[David Radley]

> On Sept. 20, 2016, 1:11 p.m., David Radley wrote:
> > It looks like this is a way of updating the versions of types using a local 
> > file on the Atlas server. I think we should be SaaS and expose this sort of 
> > functionality primarily as a REST API, otherwise this could become a 
> > barrier to integration.
> 
> Sarath Kumar Subramanian wrote:
> David, the goal of this framework is to apply patches during upgrade 
> scenarios - when user has changed the shape of a type (added new attributes, 
> new supertypes) or to update an existing type. The patch files will be 
> checked and applied during atlas startup, this will avoid dealing with 
> multiple patch scripts for upgrading multiple types. Currrently a type can be 
> updated using REST (PUT - /api/atlas/types) by including the type definition 
> in request payload. We can add only optional attributes, so instances 
> depending on old type will not be broken.
> 
> I can add a dedicated REST layer to this once we have ATLAS-1171 ready.

Hi Sumar, Thank you for explaining. I assume the patch files are used once for 
the upgrade. This is a batch orientated way of changing the shapes of types - 
which I can see could be of use in the short term, especially due to the 
exising REST restrictions . My concern is that this approach, which introduces 
versioning, might make introducing the SaaS runtime REST approach more 
difficult. Also if we put effort into a batch process that we use to upgrade 
type shapes - we may not put effort into the REST API that products integrating 
with Atlas will use at runtime.  I assume the REST Implmentaiton would be to 
relax the restriction to only allow updates of optional attributes and to add 
the concept of versioning to the API. Are you planning to do a dedicated REST 
implementation soon? all the best, David.


- David


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51939/#review149652
---


On Sept. 21, 2016, 7:51 a.m., Sarath Kumar Subramanian wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51939/
> ---
> 
> (Updated Sept. 21, 2016, 7:51 a.m.)
> 
> 
> Review request for atlas, Madhan Neethiraj, Shwetha GS, and Suma Shivaprasad.
> 
> 
> Bugs: ATLAS-1174
> https://issues.apache.org/jira/browse/ATLAS-1174
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> 1. Introduce "version" attribute to all types in the type-system, this helps 
> to track changes made to the default types (hive, sqoop, falcon and storm 
> types) and user created types. If version is not mentioned during creation of 
> a type, default version "1.0" is assigned (optional attribute).
> 2. Using the version attributed for types, introduce a patch framework for 
> type system. This framework applies patches to a type using its version 
> number and can be used during upgrade - add new attributes to an existing 
> types and it will be run during atlas startup.
> The sequence of steps:
> a. During atlas startup, check $ATLAS_HOME/models/patches directory for any 
> available patch files (json files). If there any patch files handle them.
> b. Sample patch json file looks like:
> {
> "patches": [
> { 
> "action": "ADD_ATTRIBUTE",
> "typeName": "hive_column",
> "applyToVersion": "1.0",
> "updateToVersion": "2.0",
> "actionParams": [
> { "name": "position", "dataTypeName": "int", "multiplicity": "optional", 
> "isComposite": false, "isUnique": false, "isIndexable": false, 
> "reverseAttributeName": null }
> ]
> } ]
> }
> c. The framework updates the type in "typeName" for the matching version 
> number and after applying the patch, update the version to the one mentioned 
> in "updateToVersion"
> d. The json file can have more than one action (array of actions).
> e. There can be multiple patch json files in the directory and are applied in 
> the sort order of the filename. eg:
> 001-hive_column_add_position.json
> 002-hive_column_add_anotherattribute.json
> 
> 
> Diffs
> -
> 
>   common/src/main/java/org/apache/atlas/AtlasConstants.java 17ffbd7 
>   common/src/main/java/org/apache/atlas/repository/Constants.java d7f9c89 
>   
> repository/src/main/java/org/apache/atlas/repository/typestore/GraphBackedTypeStore.java
>  a94d157 
>   
> repository/src/main/java/org/apache/atlas/services/AtlasTypeAttributePatch.java
>  PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/services/AtlasTypePatch.java 
> PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java
>  6a937f4 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/AbstractDataType.java
>  fad091d 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/ClassType.java 
> 

[jira] [Updated] (ATLAS-694) Update Atlas code to use abstraction layer

[Jeffrey Hagelberg (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeffrey Hagelberg updated ATLAS-694:

Attachment: rb47810.patch

Changes rebased.

> Update Atlas code to use abstraction layer
> --
>
> Key: ATLAS-694
> URL: https://issues.apache.org/jira/browse/ATLAS-694
> Project: Atlas
>  Issue Type: Sub-task
>Reporter: Jeffrey Hagelberg
>Assignee: Jeffrey Hagelberg
> Attachments: rb47810.patch
>
>
> In this task, we will update Atlas to replace the direct usage of titan 
> classes with usage of the equivalent interfaces in the abstraction layer.  At 
> this point, there will be no options to configure the abstraction layer 
> implementation that gets used.  Atlas will be hard-coded to use the titan 
> 0.5.4 implementation.  It will also continue to have a compile time 
> dependency on the titan 0.5.4 graph database implementation.  When this task 
> is complete, Atlas will only be accessing titan through the abstraction layer.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 47810: ATLAS-694: Update Atlas to use Graph DB abstraction layer

[Jeff Hagelberg]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47810/
---

(Updated Sept. 21, 2016, 1:49 p.m.)


Review request for atlas, David Kantor and Neeru Gupta.


Bugs: ATLAS-694
https://issues.apache.org/jira/browse/ATLAS-694


Repository: atlas


Description (updated)
---

ATLAS-694: Update Atlas to use abstraction layer.  All of the Atlas code (with 
the exception of the catalog, which was only updated minimally) has been 
updated to use the graph database abstraction layer.  In addition, there is now 
an optional Atlas configuration property that specifies the class of the 
abstraction layer database to use.  I basically put all of the changes in here 
with the exception of the actual Titan 1 implementation of code.  This includes 
the changes to support Tinkerpop 3 syntax.  This is mostly to expedite getting 
the changes into Atlas.  Originally the TP3 changes were going to be brought in 
as part of the Titan 1 implementation task.

Change Summary:

   - change Atlas classes to use AtlasGraph,AtlasVertex,AtlasEdge, etc instead 
of TitanGraph/Vertex/Edge, etc
   - compile time dependency on titan 0.5.4/TP 2 removed (except in Catalog, 
which was only changed to use AtlasGraphProvider/AtlasGraph) - see 
repository\pom.xml, other pom.xmls
   - updated DSL translation to generate Gremlin that is compliant with TP3 
when TP3 is being used.  See GremlinQuery.scala, 
GraphPersistenceStrategies.scala
   - TitanGraphProvider replaced by AtlasGraphProvider.  Graph database 
implementation is determined from a new optional configuration property
   - HiveTitanSample is no longer used by tests.  It has been replaced by 
hive-instances.json (which uses normal Atlas json syntax).  The data is saved 
with a new JSONImporter class.  This was needed because the graphson syntax 
used by HiveTitanSample is not compatible with TP3.  

Last rebase: 9/21/2016


Diffs
-

  .gitignore e10adbc4457f6297600f0feb01eb54718b8ec406 
  addons/falcon-bridge/pom.xml 1365bd05a388dc92f7a56c7f7427b5b85f97c7da 
  addons/hdfs-model/pom.xml 492f39cea085c6e69781e17bcbdbc3a231806df3 
  addons/hdfs-model/src/test/java/org/apache/atlas/fs/model/HDFSModelTest.java 
ac60294e328835ba0340e150799ddfb348ccdb52 
  addons/hive-bridge/pom.xml 6993bdb938a6095ca24482e290393eeeb3911bcb 
  
addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
 ad7a4a5d09d8542a841701dfe04981f65f767c14 
  addons/sqoop-bridge/pom.xml 8c9d278d43b5979ea1743d10845905c13249f8a6 
  addons/storm-bridge/pom.xml 12c1208b448d456a923bd7309601174ddb561ba5 
  catalog/pom.xml 2f58a8f0748de65ab78eab35df6abd2fe7c336af 
  catalog/src/main/java/org/apache/atlas/catalog/query/BaseQuery.java 
e7bb505075983371ca12d9bc1d8c6eb240c3d134 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphManagement.java
 c8cd2842ca3090b6bbd384c773b4eb45aff149ce 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasPropertyKey.java
 315ecddb861e1a1be6e0ab9b36fe4c0a52486ae8 
  graphdb/graphdb-impls/pom.xml PRE-CREATION 
  graphdb/pom.xml ad3461741d42ae83b9d3306bfa4f632ecfc06f3b 
  
graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/GraphDbObjectFactory.java
 89de23d225c738bcb7f4f502315525af8fa26188 
  
graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/Titan0DatabaseManager.java
 b4234d7321d43c8ff7fc247e895226848b6e256d 
  
graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/Titan0PropertyKey.java
 1f9f6ef786e38a66528189c47d5b147b13461859 
  
graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/Titan0Vertex.java
 b26ff040886c777f1892beb15f09a177f54ea279 
  
graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/TitanObjectFactory.java
 ab0e798d24a2890e60109193f86944b069ff0046 
  
graphdb/titan0/src/test/java/org/apache/atlas/repository/graphdb/titan0/AbstractGraphDatabaseTest.java
 35735e335fd8f8d47d211d813817e19d9f6a9552 
  
graphdb/titan0/src/test/java/org/apache/atlas/repository/graphdb/titan0/Titan0DatabaseTest.java
 6c2ea263a309c6f71c9eb39fdffed39c8e4895a2 
  pom.xml ac5b0425bc7816261e7c35caad99131c79e75872 
  repository/pom.xml 663ac874518f0942c6f647ea9ee982503410492d 
  repository/src/main/java/org/apache/atlas/GraphTransactionInterceptor.java 
fff8925ebf3b4a7498565f4b9e33885dbb5bc61a 
  repository/src/main/java/org/apache/atlas/RepositoryMetadataModule.java 
f1ef1404a45b10ee4c1c229417565e711624957f 
  
repository/src/main/java/org/apache/atlas/discovery/DataSetLineageService.java 
c216469ceb1d89b5f6a578f9bd96f01c09cccd06 
  
repository/src/main/java/org/apache/atlas/discovery/graph/DefaultGraphPersistenceStrategy.java
 b17eec7e55f478bf4403a61cef7585cf06a2d9d9 
  
repository/src/main/java/org/apache/atlas/discovery/graph/GraphBackedDiscoveryService.java
 

Re: Review Request 47810: ATLAS-694: Update Atlas to use Graph DB abstraction layer

[Jeff Hagelberg]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47810/
---

(Updated Sept. 21, 2016, 12:28 p.m.)


Review request for atlas, David Kantor and Neeru Gupta.


Changes
---

Changes rebased.


Bugs: ATLAS-694
https://issues.apache.org/jira/browse/ATLAS-694


Repository: atlas


Description
---

ATLAS-694: Update Atlas to use abstraction layer.  All of the Atlas code (with 
the exception of the catalog, which was only updated minimally) has been 
updated to use the graph database abstraction layer.  In addition, there is now 
an optional Atlas configuration property that specifies the class of the 
abstraction layer database to use.  I basically put all of the changes in here 
with the exception of the actual Titan 1 implementation of code.  This includes 
the changes to support Tinkerpop 3 syntax.  This is mostly to expedite getting 
the changes into Atlas.  Originally the TP3 changes were going to be brought in 
as part of the Titan 1 implementation task.

Change Summary:

   - change Atlas classes to use AtlasGraph,AtlasVertex,AtlasEdge, etc instead 
of TitanGraph/Vertex/Edge, etc
   - compile time dependency on titan 0.5.4/TP 2 removed (except in Catalog, 
which was only changed to use AtlasGraphProvider/AtlasGraph) - see 
repository\pom.xml, other pom.xmls
   - updated DSL translation to generate Gremlin that is compliant with TP3 
when TP3 is being used.  See GremlinQuery.scala, 
GraphPersistenceStrategies.scala
   - TitanGraphProvider replaced by AtlasGraphProvider.  Graph database 
implementation is determined from a new optional configuration property
   - HiveTitanSample is no longer used by tests.  It has been replaced by 
hive-instances.json (which uses normal Atlas json syntax).  The data is saved 
with a new JSONImporter class.  This was needed because the graphson syntax 
used by HiveTitanSample is not compatible with TP3.  
   - RepositoryMetaModule - GraphBackedSearchIndexer changed to an eager 
singleton to force index creation to happen before we execute any gremlin 
queries (which happens during the type system restoration triggered by the 
DefaultMetadataService constructor)
   - atlas_config.py - minor change to allow Atlas to use a non-default JVM.  
It now supports having the java home in ATLAS_JAVA_HOME.  If that is not set, 
we fall back to the previous behavior looking for JAVA_HOME and then checking 
the path.
   - performance : add index on __state property, fix resulting test failure - 
see GraphBackedSearchIndexer, GraphHelper.getVertexForProperty
   - performance - consolidate creation of built-in types into single call.  
See DefaultMetadataService.
   
   
Note - the diff here is cumulative and includes the changes from ATLAS-693.  I 
could not figure out how to remove those.  You can, for the most part, ignore 
the new files under graphdb\titan0.  There are a few minor changes that have 
been put in since ATLAS-693, but nothing very significant.  Just changes to 
implement a couple new methods in the abstraction layer.

Last rebase: 6/7/2016


Diffs (updated)
-

  .gitignore e10adbc4457f6297600f0feb01eb54718b8ec406 
  addons/falcon-bridge/pom.xml 1365bd05a388dc92f7a56c7f7427b5b85f97c7da 
  addons/hdfs-model/pom.xml 492f39cea085c6e69781e17bcbdbc3a231806df3 
  addons/hdfs-model/src/test/java/org/apache/atlas/fs/model/HDFSModelTest.java 
ac60294e328835ba0340e150799ddfb348ccdb52 
  addons/hive-bridge/pom.xml 6993bdb938a6095ca24482e290393eeeb3911bcb 
  
addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
 ad7a4a5d09d8542a841701dfe04981f65f767c14 
  addons/sqoop-bridge/pom.xml 8c9d278d43b5979ea1743d10845905c13249f8a6 
  addons/storm-bridge/pom.xml 12c1208b448d456a923bd7309601174ddb561ba5 
  catalog/pom.xml 2f58a8f0748de65ab78eab35df6abd2fe7c336af 
  catalog/src/main/java/org/apache/atlas/catalog/query/BaseQuery.java 
e7bb505075983371ca12d9bc1d8c6eb240c3d134 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphManagement.java
 c8cd2842ca3090b6bbd384c773b4eb45aff149ce 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasPropertyKey.java
 315ecddb861e1a1be6e0ab9b36fe4c0a52486ae8 
  graphdb/graphdb-impls/pom.xml PRE-CREATION 
  graphdb/pom.xml ad3461741d42ae83b9d3306bfa4f632ecfc06f3b 
  
graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/GraphDbObjectFactory.java
 89de23d225c738bcb7f4f502315525af8fa26188 
  
graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/Titan0DatabaseManager.java
 b4234d7321d43c8ff7fc247e895226848b6e256d 
  
graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/Titan0PropertyKey.java
 1f9f6ef786e38a66528189c47d5b147b13461859 
  
graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/Titan0Vertex.java
 b26ff040886c777f1892beb15f09a177f54ea279 
 

Re: Thoughts on the first step to understand the Atlas data model internals

[David Radley]

Hi Shwetha,
OK Thanks - what is your thinking around possible changes to 
EntityResourceDefinition?
 all the best, avid. 




From: 
To: "dev@atlas.incubator.apache.org" 
Date:   20/09/2016 06:05
Subject:Re: Thoughts on the first step to understand the Atlas 
data model internals



David,

EntityResourceDefinition was added as part of business catalog, and this
might change. 

org.apache.atlas.web.resources.EntityResource and
org.apache.atlas.web.resources.TypesResource are the initial APIs added,
these are used by AtlasClient and Atlas UI and is also documented in
Hemanth¹s doc

Regards,
Shwetha






On 15/09/16, 8:27 PM, "David Radley"  wrote:

>Hi Hermanth,
>Thanks for your response. I agree we should evolve the existing
>documentation. I was aware of the guide you pointed me to. This appears
>to 
>be useful for a user of the REST API. This sort of document is sometimes
>known as an application programming guide. The motivation of this persona
>is to effectively use the Atlas REST APIs to manage their metadata.
>
>I am interested in documenting the internal data model and motivation for
>the separations of concerns in the source code. This is for different
>personas, namely to document design consensus around the data model by
>and 
>for the Atlas contributers and committers.
> all the best David.
>
>
>
>From:   Hemanth Yamijala 
>To: "dev@atlas.incubator.apache.org" 
>Date:   15/09/2016 14:28
>Subject:Re: Thoughts on the first step to understand the Atlas
>data model internals
>
>
>
>David,
>
>This might not directly address all the points you mention below, but
>regarding learning about Atlas from a core understanding perspective, you
>may want to go through the document here:
>http://atlas.incubator.apache.org/AtlasTechnicalUserGuide.pdf. This is
>something we created just after the 0.7 release timeframe and feel it
>represents core concepts to a reasonable extent.
>
>There are several improvements that it needs:
>Likely, it is not complete and missing pieces.
>Further, being in PDF form makes it hard to edit, so moving it to a more
>editable format and specifically adding it to source code itself should
>be 
>a goal.
>
>Etc...
>
>But if it forms a starting point, then it would be great to base further
>improvements on top of it.
>
>Thanks
>Hemanth
>
>From: David Radley 
>Sent: Thursday, September 15, 2016 6:26 PM
>To: atlas
>Subject: Thoughts on the first step to understand the Atlas data model
>internals
>
>Hi,
>I am looking to understand the important pieces in the Atlas architecture
>and the order that is useful to think about them. I wanted to check my
>understanding and questions around the top level data model concept as
>someone relatively new to the project. then update the docs. I am
>interested in feedback / thoughts / things I may have misunderstood. I
>think the next areas for a person looking to understand the internals to
>understand is the type system and providers and maybe then tracking how
>the code flows from the web to the graph.
>
>It seems that the data model of Atlas is the where to start; the
>fundamental interface is interface ResourceDefinition and the base object
>is BaseResourceDefinition. As the top level data object, I think this
>needs to be simple and intuitive and have a defined purpose.
>I would expect top level metadata objects should have the ability to have
>relationships and attributes, which it seems to have and a way to 
identify
>them (names and guid - which I do not see in this object)
>
>I do not think the validate*** request call methods should live in this
>interface. I would separate out request logic from the core data model
>object; as they are different concerns.
>
>The baseResourceDefinition has :
> protected static final TypeSystem typeSystem = 
TypeSystem.getInstance
>();
>
>protected final Set instanceProperties = new HashSet<>();
>protected final Set collectionProperties = new HashSet<>();
>protected Map propertyDefs = new
>HashMap<>();
>protected Map properties = new HashMap<>();
>
>protected final Map projections = new 
HashMap<>();
>protected final Map relations = new HashMap<>();
>
>protected final PropertyMapper propertyMapper;
>protected final Map
>propertyValueFormatters = new HashMap<>();
>
>
>There is an implied concept of  Property here in the naming of these
>fields. If this is important then I suggest we have a Property class /
>Interface defining what we mean by it.
>I see there are projections and relationships. do we need both of these
>concepts in the top level object as the only implementation of a
>Projection is a RelationshipProjection.
>I see 

[jira] [Commented] (ATLAS-1184) ReservedTypesRegistrar checks for existence of 1st class type

[Shwetha G S (JIRA)]

[ 
https://issues.apache.org/jira/browse/ATLAS-1184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15509686#comment-15509686
 ] 

Shwetha G S commented on ATLAS-1184:


Looks like ReservedTypesRegistrar uses internal APIs directly. It should just 
be a utility which uses rest APIs so that its audited

> ReservedTypesRegistrar checks for existence of 1st class type
> -
>
> Key: ATLAS-1184
> URL: https://issues.apache.org/jira/browse/ATLAS-1184
> Project: Atlas
>  Issue Type: Bug
>Reporter: Shwetha G S
>Assignee: Vimal Sharma
>
> ReservedTypesRegistrar registers the models for supported integrations like 
> hive, falcon, sqoop and storm. ReservedTypesRegistrar reads the model files 
> and registers the types defined in the model file. The check thats done to 
> verify if the model is already registered - checks for existence of first 
> class type. This will not work if we add more class types to the existing 
> model or modify the existing type to add optional attributes. 
> ReservedTypesRegistrar should check for every type in the model file. Also, 
> ReservedTypesRegistrar should do type update which does create if not exists, 
> and update if exists



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (ATLAS-694) Update Atlas code to use abstraction layer

[Jeffrey Hagelberg (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeffrey Hagelberg updated ATLAS-694:

Attachment: (was: rb47810.patch)

> Update Atlas code to use abstraction layer
> --
>
> Key: ATLAS-694
> URL: https://issues.apache.org/jira/browse/ATLAS-694
> Project: Atlas
>  Issue Type: Sub-task
>Reporter: Jeffrey Hagelberg
>Assignee: Jeffrey Hagelberg
>
> In this task, we will update Atlas to replace the direct usage of titan 
> classes with usage of the equivalent interfaces in the abstraction layer.  At 
> this point, there will be no options to configure the abstraction layer 
> implementation that gets used.  Atlas will be hard-coded to use the titan 
> 0.5.4 implementation.  It will also continue to have a compile time 
> dependency on the titan 0.5.4 graph database implementation.  When this task 
> is complete, Atlas will only be accessing titan through the abstraction layer.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (ATLAS-1187) Rename Trait to Classification

[David Radley (JIRA)]

David Radley created ATLAS-1187:
---

 Summary: Rename Trait to Classification
 Key: ATLAS-1187
 URL: https://issues.apache.org/jira/browse/ATLAS-1187
 Project: Atlas
  Issue Type: Improvement
Reporter: David Radley


Classification would be more descriptive of the capability currently called 
trait in Atlas (which appear to have come from Ranger terminology).  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 47810: ATLAS-694: Update Atlas to use Graph DB abstraction layer

[David Radley]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47810/#review149815
---



I have just did git checkout master and applied the latest diff. I get compile 
errors. I my workspace it errors in the pom files as it is not recognising the 
new dependancy   

org.apache.atlas
atlas-graphdb-impls
pom
test


Is there another patch I need to apply as well?

- David Radley


On Sept. 21, 2016, 1:49 p.m., Jeff Hagelberg wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47810/
> ---
> 
> (Updated Sept. 21, 2016, 1:49 p.m.)
> 
> 
> Review request for atlas, David Kantor and Neeru Gupta.
> 
> 
> Bugs: ATLAS-694
> https://issues.apache.org/jira/browse/ATLAS-694
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> ATLAS-694: Update Atlas to use abstraction layer.  All of the Atlas code 
> (with the exception of the catalog, which was only updated minimally) has 
> been updated to use the graph database abstraction layer.  In addition, there 
> is now an optional Atlas configuration property that specifies the class of 
> the abstraction layer database to use.  I basically put all of the changes in 
> here with the exception of the actual Titan 1 implementation of code.  This 
> includes the changes to support Tinkerpop 3 syntax.  This is mostly to 
> expedite getting the changes into Atlas.  Originally the TP3 changes were 
> going to be brought in as part of the Titan 1 implementation task.
> 
> Change Summary:
> 
>- change Atlas classes to use AtlasGraph,AtlasVertex,AtlasEdge, etc 
> instead of TitanGraph/Vertex/Edge, etc
>- compile time dependency on titan 0.5.4/TP 2 removed (except in Catalog, 
> which was only changed to use AtlasGraphProvider/AtlasGraph) - see 
> repository\pom.xml, other pom.xmls
>- updated DSL translation to generate Gremlin that is compliant with TP3 
> when TP3 is being used.  See GremlinQuery.scala, 
> GraphPersistenceStrategies.scala
>- TitanGraphProvider replaced by AtlasGraphProvider.  Graph database 
> implementation is determined from a new optional configuration property
>- HiveTitanSample is no longer used by tests.  It has been replaced by 
> hive-instances.json (which uses normal Atlas json syntax).  The data is saved 
> with a new JSONImporter class.  This was needed because the graphson syntax 
> used by HiveTitanSample is not compatible with TP3.  
> 
> Last rebase: 9/21/2016
> 
> 
> Diffs
> -
> 
>   .gitignore e10adbc4457f6297600f0feb01eb54718b8ec406 
>   addons/falcon-bridge/pom.xml 1365bd05a388dc92f7a56c7f7427b5b85f97c7da 
>   addons/hdfs-model/pom.xml 492f39cea085c6e69781e17bcbdbc3a231806df3 
>   
> addons/hdfs-model/src/test/java/org/apache/atlas/fs/model/HDFSModelTest.java 
> ac60294e328835ba0340e150799ddfb348ccdb52 
>   addons/hive-bridge/pom.xml 6993bdb938a6095ca24482e290393eeeb3911bcb 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
>  ad7a4a5d09d8542a841701dfe04981f65f767c14 
>   addons/sqoop-bridge/pom.xml 8c9d278d43b5979ea1743d10845905c13249f8a6 
>   addons/storm-bridge/pom.xml 12c1208b448d456a923bd7309601174ddb561ba5 
>   catalog/pom.xml 2f58a8f0748de65ab78eab35df6abd2fe7c336af 
>   catalog/src/main/java/org/apache/atlas/catalog/query/BaseQuery.java 
> e7bb505075983371ca12d9bc1d8c6eb240c3d134 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphManagement.java
>  c8cd2842ca3090b6bbd384c773b4eb45aff149ce 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasPropertyKey.java
>  315ecddb861e1a1be6e0ab9b36fe4c0a52486ae8 
>   graphdb/graphdb-impls/pom.xml PRE-CREATION 
>   graphdb/pom.xml ad3461741d42ae83b9d3306bfa4f632ecfc06f3b 
>   
> graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/GraphDbObjectFactory.java
>  89de23d225c738bcb7f4f502315525af8fa26188 
>   
> graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/Titan0DatabaseManager.java
>  b4234d7321d43c8ff7fc247e895226848b6e256d 
>   
> graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/Titan0PropertyKey.java
>  1f9f6ef786e38a66528189c47d5b147b13461859 
>   
> graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/Titan0Vertex.java
>  b26ff040886c777f1892beb15f09a177f54ea279 
>   
> graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/TitanObjectFactory.java
>  ab0e798d24a2890e60109193f86944b069ff0046 
>   
> graphdb/titan0/src/test/java/org/apache/atlas/repository/graphdb/titan0/AbstractGraphDatabaseTest.java
>  35735e335fd8f8d47d211d813817e19d9f6a9552 
>   
> 

Re: Review Request 47810: ATLAS-694: Update Atlas to use Graph DB abstraction layer

[Jeff Hagelberg]

> On Sept. 21, 2016, 1:54 p.m., David Radley wrote:
> > I have just did git checkout master and applied the latest diff. I get 
> > compile errors. I my workspace it errors in the pom files as it is not 
> > recognising the new dependancy   
> > 
> > org.apache.atlas
> > atlas-graphdb-impls
> > pom
> > test
> > 
> > 
> > Is there another patch I need to apply as well?

Hmm, that project should get built with the rest of Atlas.  I'll investigate.


- Jeff


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47810/#review149815
---


On Sept. 21, 2016, 1:49 p.m., Jeff Hagelberg wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47810/
> ---
> 
> (Updated Sept. 21, 2016, 1:49 p.m.)
> 
> 
> Review request for atlas, David Kantor and Neeru Gupta.
> 
> 
> Bugs: ATLAS-694
> https://issues.apache.org/jira/browse/ATLAS-694
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> ATLAS-694: Update Atlas to use abstraction layer.  All of the Atlas code 
> (with the exception of the catalog, which was only updated minimally) has 
> been updated to use the graph database abstraction layer.  In addition, there 
> is now an optional Atlas configuration property that specifies the class of 
> the abstraction layer database to use.  I basically put all of the changes in 
> here with the exception of the actual Titan 1 implementation of code.  This 
> includes the changes to support Tinkerpop 3 syntax.  This is mostly to 
> expedite getting the changes into Atlas.  Originally the TP3 changes were 
> going to be brought in as part of the Titan 1 implementation task.
> 
> Change Summary:
> 
>- change Atlas classes to use AtlasGraph,AtlasVertex,AtlasEdge, etc 
> instead of TitanGraph/Vertex/Edge, etc
>- compile time dependency on titan 0.5.4/TP 2 removed (except in Catalog, 
> which was only changed to use AtlasGraphProvider/AtlasGraph) - see 
> repository\pom.xml, other pom.xmls
>- updated DSL translation to generate Gremlin that is compliant with TP3 
> when TP3 is being used.  See GremlinQuery.scala, 
> GraphPersistenceStrategies.scala
>- TitanGraphProvider replaced by AtlasGraphProvider.  Graph database 
> implementation is determined from a new optional configuration property
>- HiveTitanSample is no longer used by tests.  It has been replaced by 
> hive-instances.json (which uses normal Atlas json syntax).  The data is saved 
> with a new JSONImporter class.  This was needed because the graphson syntax 
> used by HiveTitanSample is not compatible with TP3.  
> 
> Last rebase: 9/21/2016
> 
> 
> Diffs
> -
> 
>   .gitignore e10adbc4457f6297600f0feb01eb54718b8ec406 
>   addons/falcon-bridge/pom.xml 1365bd05a388dc92f7a56c7f7427b5b85f97c7da 
>   addons/hdfs-model/pom.xml 492f39cea085c6e69781e17bcbdbc3a231806df3 
>   
> addons/hdfs-model/src/test/java/org/apache/atlas/fs/model/HDFSModelTest.java 
> ac60294e328835ba0340e150799ddfb348ccdb52 
>   addons/hive-bridge/pom.xml 6993bdb938a6095ca24482e290393eeeb3911bcb 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
>  ad7a4a5d09d8542a841701dfe04981f65f767c14 
>   addons/sqoop-bridge/pom.xml 8c9d278d43b5979ea1743d10845905c13249f8a6 
>   addons/storm-bridge/pom.xml 12c1208b448d456a923bd7309601174ddb561ba5 
>   catalog/pom.xml 2f58a8f0748de65ab78eab35df6abd2fe7c336af 
>   catalog/src/main/java/org/apache/atlas/catalog/query/BaseQuery.java 
> e7bb505075983371ca12d9bc1d8c6eb240c3d134 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphManagement.java
>  c8cd2842ca3090b6bbd384c773b4eb45aff149ce 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasPropertyKey.java
>  315ecddb861e1a1be6e0ab9b36fe4c0a52486ae8 
>   graphdb/graphdb-impls/pom.xml PRE-CREATION 
>   graphdb/pom.xml ad3461741d42ae83b9d3306bfa4f632ecfc06f3b 
>   
> graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/GraphDbObjectFactory.java
>  89de23d225c738bcb7f4f502315525af8fa26188 
>   
> graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/Titan0DatabaseManager.java
>  b4234d7321d43c8ff7fc247e895226848b6e256d 
>   
> graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/Titan0PropertyKey.java
>  1f9f6ef786e38a66528189c47d5b147b13461859 
>   
> graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/Titan0Vertex.java
>  b26ff040886c777f1892beb15f09a177f54ea279 
>   
> graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/TitanObjectFactory.java
>  ab0e798d24a2890e60109193f86944b069ff0046 
>   
> 

Re: Review Request 51939: Framework to apply updates to types in the type-system

[Shwetha GS]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51939/#review149811
---




repository/src/main/java/org/apache/atlas/services/AtlasTypeAttributePatch.java 
(line 42)


This class works on jsons directly which is error prone. Use POJOs instead 
- HierarchicalTypeDefinition, StructTypeDefinition



repository/src/main/java/org/apache/atlas/services/AtlasTypeAttributePatch.java 
(line 154)


== doesn't work on strings



repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java 
(line 177)


Move this to AtlasTypeAttributePatch


Add tests

1. Currently, the model files(like hive_model.json) are auto generated from 
model definitions defined in java(like HiveDataModelGenerator). The patch files 
in this case has to be hand coded which is error prone
2. For completeness, readability and debuggability, the type update has to be 
done in the corresponding model definitions like HiveDataModelGenerator. So, 
same data will be in two places and the model definitions and the patch files 
can go out of sync
3. Since the model definitions(like HiveDataModelGenerator) will be updated 
anyways, if we modify ReservedTypesRegistrar to do type update instead of type 
create, the type updates will automatically be taken care with the same model 
json. So, model update patches are not necessary then
3. This jira doesn't implement type versioning - doesn't have support for 
storing multiple versions of the type. But it maintains the version of the 
latest type definition which I think is useful for debugging, to know the 
version of type that the server knows. Can we maintain this info in 
HiveDataModelGenerator itself, and hence will be part of hive_model.json

- Shwetha GS


On Sept. 21, 2016, 7:51 a.m., Sarath Kumar Subramanian wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51939/
> ---
> 
> (Updated Sept. 21, 2016, 7:51 a.m.)
> 
> 
> Review request for atlas, Madhan Neethiraj, Shwetha GS, and Suma Shivaprasad.
> 
> 
> Bugs: ATLAS-1174
> https://issues.apache.org/jira/browse/ATLAS-1174
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> 1. Introduce "version" attribute to all types in the type-system, this helps 
> to track changes made to the default types (hive, sqoop, falcon and storm 
> types) and user created types. If version is not mentioned during creation of 
> a type, default version "1.0" is assigned (optional attribute).
> 2. Using the version attributed for types, introduce a patch framework for 
> type system. This framework applies patches to a type using its version 
> number and can be used during upgrade - add new attributes to an existing 
> types and it will be run during atlas startup.
> The sequence of steps:
> a. During atlas startup, check $ATLAS_HOME/models/patches directory for any 
> available patch files (json files). If there any patch files handle them.
> b. Sample patch json file looks like:
> {
> "patches": [
> { 
> "action": "ADD_ATTRIBUTE",
> "typeName": "hive_column",
> "applyToVersion": "1.0",
> "updateToVersion": "2.0",
> "actionParams": [
> { "name": "position", "dataTypeName": "int", "multiplicity": "optional", 
> "isComposite": false, "isUnique": false, "isIndexable": false, 
> "reverseAttributeName": null }
> ]
> } ]
> }
> c. The framework updates the type in "typeName" for the matching version 
> number and after applying the patch, update the version to the one mentioned 
> in "updateToVersion"
> d. The json file can have more than one action (array of actions).
> e. There can be multiple patch json files in the directory and are applied in 
> the sort order of the filename. eg:
> 001-hive_column_add_position.json
> 002-hive_column_add_anotherattribute.json
> 
> 
> Diffs
> -
> 
>   common/src/main/java/org/apache/atlas/AtlasConstants.java 17ffbd7 
>   common/src/main/java/org/apache/atlas/repository/Constants.java d7f9c89 
>   
> repository/src/main/java/org/apache/atlas/repository/typestore/GraphBackedTypeStore.java
>  a94d157 
>   
> repository/src/main/java/org/apache/atlas/services/AtlasTypeAttributePatch.java
>  PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/services/AtlasTypePatch.java 
> PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java
>  6a937f4 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/AbstractDataType.java
>  fad091d 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/ClassType.java 
> c56987a 
>   

Re: Review Request 47810: ATLAS-694: Update Atlas to use Graph DB abstraction layer

[Jeff Hagelberg]

> On Sept. 21, 2016, 1:54 p.m., David Radley wrote:
> > I have just did git checkout master and applied the latest diff. I get 
> > compile errors. I my workspace it errors in the pom files as it is not 
> > recognising the new dependancy   
> > 
> > org.apache.atlas
> > atlas-graphdb-impls
> > pom
> > test
> > 
> > 
> > Is there another patch I need to apply as well?
> 
> Jeff Hagelberg wrote:
> Hmm, that project should get built with the rest of Atlas.  I'll 
> investigate.

Did you build the entire Atlas source tree?


- Jeff


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47810/#review149815
---


On Sept. 21, 2016, 1:49 p.m., Jeff Hagelberg wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47810/
> ---
> 
> (Updated Sept. 21, 2016, 1:49 p.m.)
> 
> 
> Review request for atlas, David Kantor and Neeru Gupta.
> 
> 
> Bugs: ATLAS-694
> https://issues.apache.org/jira/browse/ATLAS-694
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> ATLAS-694: Update Atlas to use abstraction layer.  All of the Atlas code 
> (with the exception of the catalog, which was only updated minimally) has 
> been updated to use the graph database abstraction layer.  In addition, there 
> is now an optional Atlas configuration property that specifies the class of 
> the abstraction layer database to use.  I basically put all of the changes in 
> here with the exception of the actual Titan 1 implementation of code.  This 
> includes the changes to support Tinkerpop 3 syntax.  This is mostly to 
> expedite getting the changes into Atlas.  Originally the TP3 changes were 
> going to be brought in as part of the Titan 1 implementation task.
> 
> Change Summary:
> 
>- change Atlas classes to use AtlasGraph,AtlasVertex,AtlasEdge, etc 
> instead of TitanGraph/Vertex/Edge, etc
>- compile time dependency on titan 0.5.4/TP 2 removed (except in Catalog, 
> which was only changed to use AtlasGraphProvider/AtlasGraph) - see 
> repository\pom.xml, other pom.xmls
>- updated DSL translation to generate Gremlin that is compliant with TP3 
> when TP3 is being used.  See GremlinQuery.scala, 
> GraphPersistenceStrategies.scala
>- TitanGraphProvider replaced by AtlasGraphProvider.  Graph database 
> implementation is determined from a new optional configuration property
>- HiveTitanSample is no longer used by tests.  It has been replaced by 
> hive-instances.json (which uses normal Atlas json syntax).  The data is saved 
> with a new JSONImporter class.  This was needed because the graphson syntax 
> used by HiveTitanSample is not compatible with TP3.  
> 
> Last rebase: 9/21/2016
> 
> 
> Diffs
> -
> 
>   .gitignore e10adbc4457f6297600f0feb01eb54718b8ec406 
>   addons/falcon-bridge/pom.xml 1365bd05a388dc92f7a56c7f7427b5b85f97c7da 
>   addons/hdfs-model/pom.xml 492f39cea085c6e69781e17bcbdbc3a231806df3 
>   
> addons/hdfs-model/src/test/java/org/apache/atlas/fs/model/HDFSModelTest.java 
> ac60294e328835ba0340e150799ddfb348ccdb52 
>   addons/hive-bridge/pom.xml 6993bdb938a6095ca24482e290393eeeb3911bcb 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
>  ad7a4a5d09d8542a841701dfe04981f65f767c14 
>   addons/sqoop-bridge/pom.xml 8c9d278d43b5979ea1743d10845905c13249f8a6 
>   addons/storm-bridge/pom.xml 12c1208b448d456a923bd7309601174ddb561ba5 
>   catalog/pom.xml 2f58a8f0748de65ab78eab35df6abd2fe7c336af 
>   catalog/src/main/java/org/apache/atlas/catalog/query/BaseQuery.java 
> e7bb505075983371ca12d9bc1d8c6eb240c3d134 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphManagement.java
>  c8cd2842ca3090b6bbd384c773b4eb45aff149ce 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasPropertyKey.java
>  315ecddb861e1a1be6e0ab9b36fe4c0a52486ae8 
>   graphdb/graphdb-impls/pom.xml PRE-CREATION 
>   graphdb/pom.xml ad3461741d42ae83b9d3306bfa4f632ecfc06f3b 
>   
> graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/GraphDbObjectFactory.java
>  89de23d225c738bcb7f4f502315525af8fa26188 
>   
> graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/Titan0DatabaseManager.java
>  b4234d7321d43c8ff7fc247e895226848b6e256d 
>   
> graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/Titan0PropertyKey.java
>  1f9f6ef786e38a66528189c47d5b147b13461859 
>   
> graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/Titan0Vertex.java
>  b26ff040886c777f1892beb15f09a177f54ea279 
>   
> graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/TitanObjectFactory.java
>  

Re: Review Request 51939: Framework to apply updates to types in the type-system

[Shwetha GS]

> On Sept. 20, 2016, 1:11 p.m., David Radley wrote:
> > It looks like this is a way of updating the versions of types using a local 
> > file on the Atlas server. I think we should be SaaS and expose this sort of 
> > functionality primarily as a REST API, otherwise this could become a 
> > barrier to integration.
> 
> Sarath Kumar Subramanian wrote:
> David, the goal of this framework is to apply patches during upgrade 
> scenarios - when user has changed the shape of a type (added new attributes, 
> new supertypes) or to update an existing type. The patch files will be 
> checked and applied during atlas startup, this will avoid dealing with 
> multiple patch scripts for upgrading multiple types. Currrently a type can be 
> updated using REST (PUT - /api/atlas/types) by including the type definition 
> in request payload. We can add only optional attributes, so instances 
> depending on old type will not be broken.
> 
> I can add a dedicated REST layer to this once we have ATLAS-1171 ready.
> 
> David Radley wrote:
> Hi Sumar, Thank you for explaining. I assume the patch files are used 
> once for the upgrade. This is a batch orientated way of changing the shapes 
> of types - which I can see could be of use in the short term, especially due 
> to the exising REST restrictions . My concern is that this approach, which 
> introduces versioning, might make introducing the SaaS runtime REST approach 
> more difficult. Also if we put effort into a batch process that we use to 
> upgrade type shapes - we may not put effort into the REST API that products 
> integrating with Atlas will use at runtime.  I assume the REST Implmentaiton 
> would be to relax the restriction to only allow updates of optional 
> attributes and to add the concept of versioning to the API. Are you planning 
> to do a dedicated REST implementation soon? all the best, David.

Type update is already available through REST API. The API takes the complete 
new type definition and automatically figures out what has changed and 
validates the change


- Shwetha


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51939/#review149652
---


On Sept. 21, 2016, 7:51 a.m., Sarath Kumar Subramanian wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51939/
> ---
> 
> (Updated Sept. 21, 2016, 7:51 a.m.)
> 
> 
> Review request for atlas, Madhan Neethiraj, Shwetha GS, and Suma Shivaprasad.
> 
> 
> Bugs: ATLAS-1174
> https://issues.apache.org/jira/browse/ATLAS-1174
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> 1. Introduce "version" attribute to all types in the type-system, this helps 
> to track changes made to the default types (hive, sqoop, falcon and storm 
> types) and user created types. If version is not mentioned during creation of 
> a type, default version "1.0" is assigned (optional attribute).
> 2. Using the version attributed for types, introduce a patch framework for 
> type system. This framework applies patches to a type using its version 
> number and can be used during upgrade - add new attributes to an existing 
> types and it will be run during atlas startup.
> The sequence of steps:
> a. During atlas startup, check $ATLAS_HOME/models/patches directory for any 
> available patch files (json files). If there any patch files handle them.
> b. Sample patch json file looks like:
> {
> "patches": [
> { 
> "action": "ADD_ATTRIBUTE",
> "typeName": "hive_column",
> "applyToVersion": "1.0",
> "updateToVersion": "2.0",
> "actionParams": [
> { "name": "position", "dataTypeName": "int", "multiplicity": "optional", 
> "isComposite": false, "isUnique": false, "isIndexable": false, 
> "reverseAttributeName": null }
> ]
> } ]
> }
> c. The framework updates the type in "typeName" for the matching version 
> number and after applying the patch, update the version to the one mentioned 
> in "updateToVersion"
> d. The json file can have more than one action (array of actions).
> e. There can be multiple patch json files in the directory and are applied in 
> the sort order of the filename. eg:
> 001-hive_column_add_position.json
> 002-hive_column_add_anotherattribute.json
> 
> 
> Diffs
> -
> 
>   common/src/main/java/org/apache/atlas/AtlasConstants.java 17ffbd7 
>   common/src/main/java/org/apache/atlas/repository/Constants.java d7f9c89 
>   
> repository/src/main/java/org/apache/atlas/repository/typestore/GraphBackedTypeStore.java
>  a94d157 
>   
> repository/src/main/java/org/apache/atlas/services/AtlasTypeAttributePatch.java
>  PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/services/AtlasTypePatch.java 
> PRE-CREATION 
>   
> 

Gremlin console with underlying Titan DB

[Apoorv Naik]

Has anyone ever attempted this ? I followed  the titandb guides and was able to 
connect to the hbase storage backend with the table name “apache_atlas_titan” 
but the traversal is showing zero vertex.

Here’s how I (supposedly) connected to the graph via the console
graph = TitanFactory.build().set("storage.backend", 
"hbase").set("storage.hbase.ext.zookeeper.znode.parent","/hbase-unsecure").set("storage.hbase.table",
 "apache_atlas_titan").set("storage.hostname", "127.0.0.1").open()

Re: Review Request 47810: ATLAS-694: Update Atlas to use Graph DB abstraction layer

[Apoorv Naik]

> On Sept. 21, 2016, 11:47 p.m., Suma Shivaprasad wrote:
> > repository/src/main/scala/org/apache/atlas/query/GremlinQuery.scala, line 
> > 279
> > 
> >
> > Instead of having multiple checks its better to abstract out the 
> > Gremlin2.0 and Gremlin3.0 specific implementations which are referred to 
> > from a base installation?
> 
> Jeff Hagelberg wrote:
> Yes, I definitely agree.  The DSL translation should be done by 
> translating the DSL into intermedate classes such as GremlinSelectExpression, 
> GremlnHasExpression, etc.  These would have subclasses specific to gremlin 
> 2/3 where appropriate, and the instance would be served up by some factory.  
> However, doing this would require a substantial rewrite of the DSL 
> translator.  I can do this if you feel it is important, but it was something 
> I was hoping to avoid.  I do think this is where we want to end up, though.  
> I'll start looking at this.  It will be a non-trivial exercise though.  Could 
> it be moved into a follow-on JIRA?

Can the potential DSL translator be written in Java ? (Just a thought)


- Apoorv


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47810/#review149922
---


On Sept. 21, 2016, 1:49 p.m., Jeff Hagelberg wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47810/
> ---
> 
> (Updated Sept. 21, 2016, 1:49 p.m.)
> 
> 
> Review request for atlas, David Kantor and Neeru Gupta.
> 
> 
> Bugs: ATLAS-694
> https://issues.apache.org/jira/browse/ATLAS-694
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> ATLAS-694: Update Atlas to use abstraction layer.  All of the Atlas code 
> (with the exception of the catalog, which was only updated minimally) has 
> been updated to use the graph database abstraction layer.  In addition, there 
> is now an optional Atlas configuration property that specifies the class of 
> the abstraction layer database to use.  I basically put all of the changes in 
> here with the exception of the actual Titan 1 implementation of code.  This 
> includes the changes to support Tinkerpop 3 syntax.  This is mostly to 
> expedite getting the changes into Atlas.  Originally the TP3 changes were 
> going to be brought in as part of the Titan 1 implementation task.
> 
> Change Summary:
> 
>- change Atlas classes to use AtlasGraph,AtlasVertex,AtlasEdge, etc 
> instead of TitanGraph/Vertex/Edge, etc
>- compile time dependency on titan 0.5.4/TP 2 removed (except in Catalog, 
> which was only changed to use AtlasGraphProvider/AtlasGraph) - see 
> repository\pom.xml, other pom.xmls
>- updated DSL translation to generate Gremlin that is compliant with TP3 
> when TP3 is being used.  See GremlinQuery.scala, 
> GraphPersistenceStrategies.scala
>- TitanGraphProvider replaced by AtlasGraphProvider.  Graph database 
> implementation is determined from a new optional configuration property
>- HiveTitanSample is no longer used by tests.  It has been replaced by 
> hive-instances.json (which uses normal Atlas json syntax).  The data is saved 
> with a new JSONImporter class.  This was needed because the graphson syntax 
> used by HiveTitanSample is not compatible with TP3.  
> 
> Last rebase: 9/21/2016
> 
> 
> Diffs
> -
> 
>   .gitignore e10adbc4457f6297600f0feb01eb54718b8ec406 
>   addons/falcon-bridge/pom.xml 1365bd05a388dc92f7a56c7f7427b5b85f97c7da 
>   addons/hdfs-model/pom.xml 492f39cea085c6e69781e17bcbdbc3a231806df3 
>   
> addons/hdfs-model/src/test/java/org/apache/atlas/fs/model/HDFSModelTest.java 
> ac60294e328835ba0340e150799ddfb348ccdb52 
>   addons/hive-bridge/pom.xml 6993bdb938a6095ca24482e290393eeeb3911bcb 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
>  ad7a4a5d09d8542a841701dfe04981f65f767c14 
>   addons/sqoop-bridge/pom.xml 8c9d278d43b5979ea1743d10845905c13249f8a6 
>   addons/storm-bridge/pom.xml 12c1208b448d456a923bd7309601174ddb561ba5 
>   catalog/pom.xml 2f58a8f0748de65ab78eab35df6abd2fe7c336af 
>   catalog/src/main/java/org/apache/atlas/catalog/query/BaseQuery.java 
> e7bb505075983371ca12d9bc1d8c6eb240c3d134 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphManagement.java
>  c8cd2842ca3090b6bbd384c773b4eb45aff149ce 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasPropertyKey.java
>  315ecddb861e1a1be6e0ab9b36fe4c0a52486ae8 
>   graphdb/graphdb-impls/pom.xml PRE-CREATION 
>   graphdb/pom.xml ad3461741d42ae83b9d3306bfa4f632ecfc06f3b 
>   
> graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/GraphDbObjectFactory.java
>  

[jira] [Commented] (ATLAS-1175) Type updates should allow removal of optional attributes

[Shwetha G S (JIRA)]

[ 
https://issues.apache.org/jira/browse/ATLAS-1175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15509085#comment-15509085
 ] 

Shwetha G S commented on ATLAS-1175:


Removing the optional attribute is not an issue as it doesn't surface in entity 
definition. The issue is if the user adds the removed attribute again - in this 
case, the entities which had the values for old attribute will re-surface 
again, which is not right.

> Type updates should allow removal of optional attributes
> 
>
> Key: ATLAS-1175
> URL: https://issues.apache.org/jira/browse/ATLAS-1175
> Project: Atlas
>  Issue Type: Bug
>Affects Versions: 0.7-incubating, 0.8-incubating
>Reporter: Suma Shivaprasad
>Assignee: Sarath Subramanian
>
> Currently optional attributes are not allowed to be removed from a given 
> type. This should be allowed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 51939: Framework to apply updates to types in the type-system

[Sarath Kumar Subramanian]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51939/
---

(Updated Sept. 21, 2016, 7:51 a.m.)


Review request for atlas, Madhan Neethiraj, Shwetha GS, and Suma Shivaprasad.


Changes
---

please ignore the previous diff(included patch file in the commit). Added 
support to update and delete attributes in addition to add, addressed suma's 
code review suggestions.


Bugs: ATLAS-1174
https://issues.apache.org/jira/browse/ATLAS-1174


Repository: atlas


Description
---

1. Introduce "version" attribute to all types in the type-system, this helps to 
track changes made to the default types (hive, sqoop, falcon and storm types) 
and user created types. If version is not mentioned during creation of a type, 
default version "1.0" is assigned (optional attribute).
2. Using the version attributed for types, introduce a patch framework for type 
system. This framework applies patches to a type using its version number and 
can be used during upgrade - add new attributes to an existing types and it 
will be run during atlas startup.
The sequence of steps:
a. During atlas startup, check $ATLAS_HOME/models/patches directory for any 
available patch files (json files). If there any patch files handle them.
b. Sample patch json file looks like:
{
"patches": [
{ 
"action": "ADD_ATTRIBUTE",
"typeName": "hive_column",
"applyToVersion": "1.0",
"updateToVersion": "2.0",
"actionParams": [
{ "name": "position", "dataTypeName": "int", "multiplicity": "optional", 
"isComposite": false, "isUnique": false, "isIndexable": false, 
"reverseAttributeName": null }
]
} ]
}
c. The framework updates the type in "typeName" for the matching version number 
and after applying the patch, update the version to the one mentioned in 
"updateToVersion"
d. The json file can have more than one action (array of actions).
e. There can be multiple patch json files in the directory and are applied in 
the sort order of the filename. eg:
001-hive_column_add_position.json
002-hive_column_add_anotherattribute.json


Diffs (updated)
-

  common/src/main/java/org/apache/atlas/AtlasConstants.java 17ffbd7 
  common/src/main/java/org/apache/atlas/repository/Constants.java d7f9c89 
  
repository/src/main/java/org/apache/atlas/repository/typestore/GraphBackedTypeStore.java
 a94d157 
  
repository/src/main/java/org/apache/atlas/services/AtlasTypeAttributePatch.java 
PRE-CREATION 
  repository/src/main/java/org/apache/atlas/services/AtlasTypePatch.java 
PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java 
6a937f4 
  
typesystem/src/main/java/org/apache/atlas/typesystem/types/AbstractDataType.java
 fad091d 
  typesystem/src/main/java/org/apache/atlas/typesystem/types/ClassType.java 
c56987a 
  typesystem/src/main/java/org/apache/atlas/typesystem/types/EnumType.java 
bdd0a13 
  
typesystem/src/main/java/org/apache/atlas/typesystem/types/EnumTypeDefinition.java
 6340615 
  
typesystem/src/main/java/org/apache/atlas/typesystem/types/HierarchicalType.java
 7224699 
  
typesystem/src/main/java/org/apache/atlas/typesystem/types/HierarchicalTypeDefinition.java
 9a299f0 
  typesystem/src/main/java/org/apache/atlas/typesystem/types/IDataType.java 
85ddee7 
  typesystem/src/main/java/org/apache/atlas/typesystem/types/StructType.java 
6f40c1d 
  
typesystem/src/main/java/org/apache/atlas/typesystem/types/StructTypeDefinition.java
 f1ce1b7 
  typesystem/src/main/java/org/apache/atlas/typesystem/types/TraitType.java 
f23bf5b 
  typesystem/src/main/java/org/apache/atlas/typesystem/types/TypeSystem.java 
70ba89b 
  
typesystem/src/main/java/org/apache/atlas/typesystem/types/utils/TypesUtil.java 
ef8448d 
  
typesystem/src/main/scala/org/apache/atlas/typesystem/json/TypesSerialization.scala
 5618938 

Diff: https://reviews.apache.org/r/51939/diff/


Testing
---


Thanks,

Sarath Kumar Subramanian

Re: Review Request 51939: Framework to apply updates to types in the type-system

[Sarath Kumar Subramanian]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51939/
---

(Updated Sept. 21, 2016, 7:21 a.m.)


Review request for atlas, Madhan Neethiraj, Shwetha GS, and Suma Shivaprasad.


Changes
---

added support to update and delete attributes in addition to add, addressed 
suma's code review suggestions.


Bugs: ATLAS-1174
https://issues.apache.org/jira/browse/ATLAS-1174


Repository: atlas


Description
---

1. Introduce "version" attribute to all types in the type-system, this helps to 
track changes made to the default types (hive, sqoop, falcon and storm types) 
and user created types. If version is not mentioned during creation of a type, 
default version "1.0" is assigned (optional attribute).
2. Using the version attributed for types, introduce a patch framework for type 
system. This framework applies patches to a type using its version number and 
can be used during upgrade - add new attributes to an existing types and it 
will be run during atlas startup.
The sequence of steps:
a. During atlas startup, check $ATLAS_HOME/models/patches directory for any 
available patch files (json files). If there any patch files handle them.
b. Sample patch json file looks like:
{
"patches": [
{ 
"action": "ADD_ATTRIBUTE",
"typeName": "hive_column",
"applyToVersion": "1.0",
"updateToVersion": "2.0",
"actionParams": [
{ "name": "position", "dataTypeName": "int", "multiplicity": "optional", 
"isComposite": false, "isUnique": false, "isIndexable": false, 
"reverseAttributeName": null }
]
} ]
}
c. The framework updates the type in "typeName" for the matching version number 
and after applying the patch, update the version to the one mentioned in 
"updateToVersion"
d. The json file can have more than one action (array of actions).
e. There can be multiple patch json files in the directory and are applied in 
the sort order of the filename. eg:
001-hive_column_add_position.json
002-hive_column_add_anotherattribute.json


Diffs (updated)
-

  ATLAS-1171.7.patch PRE-CREATION 
  common/src/main/java/org/apache/atlas/AtlasConstants.java 17ffbd7 
  common/src/main/java/org/apache/atlas/repository/Constants.java d7f9c89 
  
repository/src/main/java/org/apache/atlas/repository/typestore/GraphBackedTypeStore.java
 a94d157 
  
repository/src/main/java/org/apache/atlas/services/AtlasTypeAttributePatch.java 
PRE-CREATION 
  repository/src/main/java/org/apache/atlas/services/AtlasTypePatch.java 
PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java 
6a937f4 
  
typesystem/src/main/java/org/apache/atlas/typesystem/types/AbstractDataType.java
 fad091d 
  typesystem/src/main/java/org/apache/atlas/typesystem/types/ClassType.java 
c56987a 
  typesystem/src/main/java/org/apache/atlas/typesystem/types/EnumType.java 
bdd0a13 
  
typesystem/src/main/java/org/apache/atlas/typesystem/types/EnumTypeDefinition.java
 6340615 
  
typesystem/src/main/java/org/apache/atlas/typesystem/types/HierarchicalType.java
 7224699 
  
typesystem/src/main/java/org/apache/atlas/typesystem/types/HierarchicalTypeDefinition.java
 9a299f0 
  typesystem/src/main/java/org/apache/atlas/typesystem/types/IDataType.java 
85ddee7 
  typesystem/src/main/java/org/apache/atlas/typesystem/types/StructType.java 
6f40c1d 
  
typesystem/src/main/java/org/apache/atlas/typesystem/types/StructTypeDefinition.java
 f1ce1b7 
  typesystem/src/main/java/org/apache/atlas/typesystem/types/TraitType.java 
f23bf5b 
  typesystem/src/main/java/org/apache/atlas/typesystem/types/TypeSystem.java 
70ba89b 
  typesystem/src/main/java/org/apache/atlas/typesystem/types/TypeUtils.java 
f5c2ce9 
  
typesystem/src/main/java/org/apache/atlas/typesystem/types/utils/TypesUtil.java 
ef8448d 
  
typesystem/src/main/scala/org/apache/atlas/typesystem/json/TypesSerialization.scala
 5618938 

Diff: https://reviews.apache.org/r/51939/diff/


Testing
---


Thanks,

Sarath Kumar Subramanian

Re: Review Request 51939: Framework to apply updates to types in the type-system

[Madhan Neethiraj]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51939/#review149854
---




repository/src/main/java/org/apache/atlas/services/AtlasTypeAttributePatch.java 
(line 61)


"attributeDefinitions" is an array in structType and classType. Lets keep 
this consistent here. Please refer to 
./addons/hive-bridge/target/models/hive_model.json.



repository/src/main/java/org/apache/atlas/services/AtlasTypeAttributePatch.java 
(line 87)


Consider adding "currentVersion == null ||".



repository/src/main/java/org/apache/atlas/services/AtlasTypeAttributePatch.java 
(line 103)


Consider returning PatchResult() with the error message from exception 
blocks.

It will be a good practice to minimize the number of returns from a method. 
I generally define a variable named "ret" at the entry of the method; set it to 
appropriate value in the body; and return from the end of the method. Make it 
easier to read and add debug tracing.



repository/src/main/java/org/apache/atlas/services/AtlasTypeAttributePatch.java 
(line 122)


Since this loop iterates for each attribute  in the patch, should this call 
only check for presense of the current patch attribute?



repository/src/main/java/org/apache/atlas/services/AtlasTypeAttributePatch.java 
(line 125)


Consider rewritting this 'if' as below, for better readability:

if(! typeContainsPatchAttrs) {
  if(action == PatchAction.ADD_ATTRIBUTES) {
// add the atttribute
  } else {
LOG.warn(action + ": invalid action for non-existing attribute. 
Ignored.");
  }
} else {
  if(action == PatchAction.UPDATE_ATTRIBUTES) {
// update the atttribute
  } else if(action == PatchAction.DELETE_ATTRIBUTES) {
// delete the atttribute
  } else {
LOG.warn(action + ": invalid action for already existing attribute. 
Ignored.");
  }
}



repository/src/main/java/org/apache/atlas/services/AtlasTypeAttributePatch.java 
(line 133)


"to the type" ==> "in type"



repository/src/main/java/org/apache/atlas/services/AtlasTypeAttributePatch.java 
(line 137)


"to the type" ==> "from type"



repository/src/main/java/org/apache/atlas/services/AtlasTypeAttributePatch.java 
(line 162)


Consider updating the error messagea to include more details (like 
typename): 

"error in getting type '" + patchData.getTypeName() + "' from typesystem"

Please review other error messages as well.



repository/src/main/java/org/apache/atlas/services/AtlasTypeAttributePatch.java 
(line 186)


Is it necessary to convert to JSON? Wouldn't iterating on currentAttributes 
work?

if(currentAttributes != null) {
  for(AttributeDefintion attributeDef : currentAttributes) {
s.add(attributeDef.name);
  }
}



repository/src/main/java/org/apache/atlas/services/AtlasTypePatch.java (line 77)


Consider adding attributeDefinitions as a first class attribute - to save 
from having to tranform from/to JSON in many places. When we add support for 
other type of patches (like types..), appropraite members can be added here.

class PatchData {
  private String   action;
  private String   typeName;
  ..
  private List attributeDefinitions;
  private Map  params;
}



repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java 
(line 171)


checkForPatches() call will be missed in HA deployments. Make sure to call 
from instanceIsActive() - just as restoreTypeSystem() is called.



repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java 
(line 190)


patchFile can be null. Please update to handle this condition.



repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java 
(line 220)


Supported actions: ADD_ATTRIBUTE, UPDATE_ATTRIBUTE, DELETE_ATTRIBUTE



repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java 
(line 225)


result could be null. Please update to handle this condition.

Re: Review Request 52077: Column level lineage in Hive

[Suma Shivaprasad]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52077/#review149896
---




addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/ColumnLineageUtils.java
 (line 94)


Use constant for "columns". Also can remove .getValuesMap .get should work 
on Referenceable



addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/ColumnLineageUtils.java
 (line 97)


why is column qualifiedName different from the convention we are using for 
hive_column instances which are referred to from the table. Why is clustername 
removed?


- Suma Shivaprasad


On Sept. 20, 2016, 9:07 a.m., Vimal Sharma wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/52077/
> ---
> 
> (Updated Sept. 20, 2016, 9:07 a.m.)
> 
> 
> Review request for atlas.
> 
> 
> Bugs: ATLAS-247
> https://issues.apache.org/jira/browse/ATLAS-247
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> After a CTAS query, lineage relationship between source columns and 
> destination column can be captured. This information can be used to create a 
> column lineage process.
> 
> 
> Diffs
> -
> 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/ColumnLineageUtils.java
>  PRE-CREATION 
>   addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
> a3464a0 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/model/HiveDataModelGenerator.java
>  45f0bc9 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/model/HiveDataTypes.java
>  e094cb6 
>   addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 
> a5838b4 
> 
> Diff: https://reviews.apache.org/r/52077/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Vimal Sharma
> 
>