Re: Review Request 51939: Framework to apply updates to types in the type-system

[Sarath Kumar Subramanian]

On Sept. 21, 2016, 7:46 a.m., Sarath Kumar Subramanian wrote:
> > Add tests
> > 
> > 1. Currently, the model files(like hive_model.json) are auto generated from 
> > model definitions defined in java(like HiveDataModelGenerator). The patch 
> > files in this case has to be hand coded which is error prone
> > 2. For completeness, readability and debuggability, the type update has to 
> > be done in the corresponding model definitions like HiveDataModelGenerator. 
> > So, same data will be in two places and the model definitions and the patch 
> > files can go out of sync
> > 3. Since the model definitions(like HiveDataModelGenerator) will be updated 
> > anyways, if we modify ReservedTypesRegistrar to do type update instead of 
> > type create, the type updates will automatically be taken care with the 
> > same model json. So, model update patches are not necessary then
> > 3. This jira doesn't implement type versioning - doesn't have support for 
> > storing multiple versions of the type. But it maintains the version of the 
> > latest type definition which I think is useful for debugging, to know the 
> > version of type that the server knows. Can we maintain this info in 
> > HiveDataModelGenerator itself, and hence will be part of hive_model.json
> 
> Sarath Kumar Subramanian wrote:
> 1. patch json genarator can be addressed in a separate jiira
> 2. this is only for upgrade scenarios
> 3. lets address this in a separate jiira.
> 4. we dont want to maintain multiple versions of the same type in the 
> typesystem, will confuse users on which version to use to create entities.
> 
> Shwetha GS wrote:
> What benefit does this patch framework add when the type update can be 
> done with the existing functionality itself (Type versioning is useful and 
> can be done without the patch framework). The patch framework adds more 
> overhead, and I don't see the necessity
> 
> David Radley wrote:
> I agree - I think this capability would be better placed in separate 
> tooling that uses the Atlas REST APIs.
> 
> Madhan Neethiraj wrote:
> @Shwetha, @David - the rational for the patch framework is exactly same 
> as the need for reading contents of "models" directory during startup and 
> initializing the typesystem. This just makes it easier to deal with 
> updates/additions to typesystem during software upgrade/patch.

The existing system to update types (using REST) has these limitations:

1. Clients might add new attributes to a type and when we add a set of new 
attributes in model.json -  the rest api model overwrites customer changes.

2. More granular type updates cannot be applied, if you add a new attribute in 
hive_model for eg.,  During atlas startup it doesn't do a type by model 
comparison and apply the diff. This patch can update to a specific type based 
on its version - condition based update.


- Sarath Kumar


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51939/#review149811
---


On Sept. 22, 2016, 12:34 a.m., Sarath Kumar Subramanian wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51939/
> ---
> 
> (Updated Sept. 22, 2016, 12:34 a.m.)
> 
> 
> Review request for atlas, Madhan Neethiraj, Shwetha GS, and Suma Shivaprasad.
> 
> 
> Bugs: ATLAS-1174
> https://issues.apache.org/jira/browse/ATLAS-1174
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> 1. Introduce "version" attribute to all types in the type-system, this helps 
> to track changes made to the default types (hive, sqoop, falcon and storm 
> types) and user created types. If version is not mentioned during creation of 
> a type, default version "1.0" is assigned (optional attribute).
> 2. Using the version attributed for types, introduce a patch framework for 
> type system. This framework applies patches to a type using its version 
> number and can be used during upgrade - add new attributes to an existing 
> types and it will be run during atlas startup.
> The sequence of steps:
> a. During atlas startup, check $ATLAS_HOME/models/patches directory for any 
> available patch files (json files). If there any patch files handle them.
> b. Sample patch json file looks like:
> {
> "patches": [
> { 
> "action": "ADD_ATTRIBUTE",
> "typeName": "hive_column",
> "applyToVersion": "1.0",
> "updateToVersion": "2.0",
> "actionParams": [
> { "name": "position", "dataTypeName": "int", "multiplicity": "optional", 
> "isComposite": false, "isUnique": false, "isIndexable": false, 
> "reverseAttributeName": null }
> ]
> } ]
> }
> c. The framework updates the type in "typeName" for the matching version 
> number and after applying the patch, update the version to the one mentioned 
> in 

Re: Review Request 51939: Framework to apply updates to types in the type-system

[Sarath Kumar Subramanian]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51939/
---

(Updated Sept. 22, 2016, 8:02 p.m.)


Review request for atlas, Madhan Neethiraj, Shwetha GS, and Suma Shivaprasad.


Changes
---

moved checkForPatches() method to a seperate class which can handle all patch 
updates


Bugs: ATLAS-1174
https://issues.apache.org/jira/browse/ATLAS-1174


Repository: atlas


Description
---

1. Introduce "version" attribute to all types in the type-system, this helps to 
track changes made to the default types (hive, sqoop, falcon and storm types) 
and user created types. If version is not mentioned during creation of a type, 
default version "1.0" is assigned (optional attribute).
2. Using the version attributed for types, introduce a patch framework for type 
system. This framework applies patches to a type using its version number and 
can be used during upgrade - add new attributes to an existing types and it 
will be run during atlas startup.
The sequence of steps:
a. During atlas startup, check $ATLAS_HOME/models/patches directory for any 
available patch files (json files). If there any patch files handle them.
b. Sample patch json file looks like:
{
"patches": [
{ 
"action": "ADD_ATTRIBUTE",
"typeName": "hive_column",
"applyToVersion": "1.0",
"updateToVersion": "2.0",
"actionParams": [
{ "name": "position", "dataTypeName": "int", "multiplicity": "optional", 
"isComposite": false, "isUnique": false, "isIndexable": false, 
"reverseAttributeName": null }
]
} ]
}
c. The framework updates the type in "typeName" for the matching version number 
and after applying the patch, update the version to the one mentioned in 
"updateToVersion"
d. The json file can have more than one action (array of actions).
e. There can be multiple patch json files in the directory and are applied in 
the sort order of the filename. eg:
001-hive_column_add_position.json
002-hive_column_add_anotherattribute.json


Diffs (updated)
-

  common/src/main/java/org/apache/atlas/AtlasConstants.java 17ffbd7 
  common/src/main/java/org/apache/atlas/repository/Constants.java d7f9c89 
  
repository/src/main/java/org/apache/atlas/repository/typestore/GraphBackedTypeStore.java
 a94d157 
  repository/src/main/java/org/apache/atlas/services/AtlasPatchHandler.java 
PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/services/AtlasTypeAttributePatch.java 
PRE-CREATION 
  repository/src/main/java/org/apache/atlas/services/AtlasTypePatch.java 
PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java 
6a937f4 
  
typesystem/src/main/java/org/apache/atlas/typesystem/types/AbstractDataType.java
 fad091d 
  typesystem/src/main/java/org/apache/atlas/typesystem/types/ClassType.java 
c56987a 
  typesystem/src/main/java/org/apache/atlas/typesystem/types/EnumType.java 
bdd0a13 
  
typesystem/src/main/java/org/apache/atlas/typesystem/types/EnumTypeDefinition.java
 6340615 
  
typesystem/src/main/java/org/apache/atlas/typesystem/types/HierarchicalType.java
 7224699 
  
typesystem/src/main/java/org/apache/atlas/typesystem/types/HierarchicalTypeDefinition.java
 9a299f0 
  typesystem/src/main/java/org/apache/atlas/typesystem/types/IDataType.java 
85ddee7 
  typesystem/src/main/java/org/apache/atlas/typesystem/types/StructType.java 
6f40c1d 
  
typesystem/src/main/java/org/apache/atlas/typesystem/types/StructTypeDefinition.java
 f1ce1b7 
  typesystem/src/main/java/org/apache/atlas/typesystem/types/TraitType.java 
f23bf5b 
  typesystem/src/main/java/org/apache/atlas/typesystem/types/TypeSystem.java 
70ba89b 
  
typesystem/src/main/java/org/apache/atlas/typesystem/types/utils/TypesUtil.java 
ef8448d 
  
typesystem/src/main/scala/org/apache/atlas/typesystem/json/TypesSerialization.scala
 5618938 

Diff: https://reviews.apache.org/r/51939/diff/


Testing
---


Thanks,

Sarath Kumar Subramanian

[jira] [Updated] (ATLAS-694) Update Atlas code to use abstraction layer

[Jeffrey Hagelberg (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeffrey Hagelberg updated ATLAS-694:

Attachment: rb47810.patch

Addressed code review comments and performance issue.

> Update Atlas code to use abstraction layer
> --
>
> Key: ATLAS-694
> URL: https://issues.apache.org/jira/browse/ATLAS-694
> Project: Atlas
>  Issue Type: Sub-task
>Reporter: Jeffrey Hagelberg
>Assignee: Jeffrey Hagelberg
> Attachments: rb47810.patch
>
>
> In this task, we will update Atlas to replace the direct usage of titan 
> classes with usage of the equivalent interfaces in the abstraction layer.  At 
> this point, there will be no options to configure the abstraction layer 
> implementation that gets used.  Atlas will be hard-coded to use the titan 
> 0.5.4 implementation.  It will also continue to have a compile time 
> dependency on the titan 0.5.4 graph database implementation.  When this task 
> is complete, Atlas will only be accessing titan through the abstraction layer.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (ATLAS-694) Update Atlas code to use abstraction layer

[Jeffrey Hagelberg (JIRA)]

[ 
https://issues.apache.org/jira/browse/ATLAS-694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15515256#comment-15515256
 ] 

Jeffrey Hagelberg edited comment on ATLAS-694 at 9/23/16 3:27 AM:
--

Addressed code review comments and performance regression.


was (Author: jnhagelb):
Addressed code review comments and performance issue.

> Update Atlas code to use abstraction layer
> --
>
> Key: ATLAS-694
> URL: https://issues.apache.org/jira/browse/ATLAS-694
> Project: Atlas
>  Issue Type: Sub-task
>Reporter: Jeffrey Hagelberg
>Assignee: Jeffrey Hagelberg
> Attachments: rb47810.patch
>
>
> In this task, we will update Atlas to replace the direct usage of titan 
> classes with usage of the equivalent interfaces in the abstraction layer.  At 
> this point, there will be no options to configure the abstraction layer 
> implementation that gets used.  Atlas will be hard-coded to use the titan 
> 0.5.4 implementation.  It will also continue to have a compile time 
> dependency on the titan 0.5.4 graph database implementation.  When this task 
> is complete, Atlas will only be accessing titan through the abstraction layer.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 47810: ATLAS-694: Update Atlas to use Graph DB abstraction layer

[Jeff Hagelberg]

On Sept. 22, 2016, 11:12 a.m., Jeff Hagelberg wrote:
> > Enable checkstyle on new modules with this property in pom.xml - 
> > true
> > 
> > With patch:
> > Running org.apache.atlas.repository.graph.GraphRepoMapperScaleTest
> > Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 113.27 sec 
> > - in org.apache.atlas.repository.graph.GraphRepoMapperScaleTest
> > 
> > Without patch:
> > Running org.apache.atlas.repository.graph.GraphRepoMapperScaleTest
> > Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 89.008 sec 
> > - in org.apache.atlas.repository.graph.GraphRepoMapperScaleTest
> > 
> > With the patch, the test takes longer to run. This implies some performance 
> > issue with the patch
> > 
> > Whats the plan on including titan 1.x implementation - will it be a profile 
> > at compile time such that only one implementation(titan 0.x or titan 1.x) 
> > will be part of the package. Or are both implementations part of the 
> > package and the implementation is chosen from the runtime configuration in 
> > application properties? If its compile time binding, then we shouldn't 
> > expose the conf atlas.graphdb.backend?
> 
> Jeff Hagelberg wrote:
> The plan is for all of the Atlas code to just depend on the abstraction 
> layer.  There will be profiles in incubator-atlas\pom.xml for each graph 
> backend.  The ones we have in our fork look like this:
> 
> 
> titan1
> 
> 
> org.apache.atlas.repository.graphdb.titan1.Titan1Database
> 
> 
> 
> 
> org.apache.atlas
> atlas-graphdb-impls
> ${project.version}
> pom
> 
> 
> org.apache.atlas
> 
> atlas-graphdb-titan0
> 
> 
> com.ibm.analytics
> 
> atlas-graphdb-ibm-graph
> 
> 
> 
> 
> 
> 
> 
> 
> ibm-graph
> 
> 
> com.ibm.analytics.atlas.ibmgraph.IBMGraphDatabase
> 
> org.apache.atlas.repository.typestore.StoreBackedTenantTypeCache
> 4000
> 
> 
> 
> 
> org.apache.atlas
> atlas-graphdb-impls
> ${project.version}
> pom
> 
> 
> org.apache.atlas
> 
> atlas-graphdb-titan0
> 
> 
> org.apache.atlas
> 
> atlas-graphdb-titan1
> 
> 
> 
> 
> 
> 
> 
> 
> These configure the atlas-graphdb-impls project to just pull in the jar 
> from the implementation that is being used.  Right now, webapp has a 
> dependency on this project, and will bundle the graph database implementation 
> uber jar file corresponding to the profile that was selected.  Unfortunately, 
> it is not possible for the titan0 and titan1 implementations to co-exist in 
> the war without the use of a custom classloader.  titan0 and titan1 define 
> many classes with the same name.  In addition, the two require different 
> contents in META-INF/services/javax.script.ScriptEngineFactory.  Our plan has 
> been to have the implementations available in the filesystem and have the 
> correct loaded on startup, most likely using a custom classloader.  We have 
> not really investigated that very much. It's possible that the 
> plugin-classloader that was recently added might be able to help with this.  
> Right now, there is no way to switch between implementations without 
> modifying that atlas war file.  Th
 is is probably ok for now, since we have not contributed the titan1 
implementation yet, but it is definitely something we will need to look at.  I 
am pretty sure that we created a JIRA for that.
> 
> I will do profiling of GraphRepoMapperScaleTest with and without the 
> changes to see if I can identify and fix the performance regression in 
> GraphRepoMapperScaleTest.

I've identified and fixed the performance regression.  The main culprit was the 
logic for looking up whether a property is a multi-property.  That checked has 

[jira] [Resolved] (ATLAS-1189) Server Signature validation on client failed while using only CA in the client truststore

[Hemanth Yamijala (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-1189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hemanth Yamijala resolved ATLAS-1189.
-
Resolution: Invalid

I am suspecting this is not an Apache Atlas project issue, but maybe some other 
project. Could you please confirm? If yes, please file a JIRA against the right 
component. If it is related to Atlas, please reopen the issue and provide some 
more information in the report. Thanks!

> Server Signature validation on client failed while using only CA in the 
> client truststore
> -
>
> Key: ATLAS-1189
> URL: https://issues.apache.org/jira/browse/ATLAS-1189
> Project: Atlas
>  Issue Type: Bug
>Reporter: Libois Claude
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (ATLAS-1191) Falcon cluster entity name and the actual cluster name(maintained by Ambari) should be same - otherwise the entity created via falconHook and hiveHook wont match and this

[Ayub Khan (JIRA)]

Ayub Khan created ATLAS-1191:


 Summary: Falcon cluster entity name and the actual cluster 
name(maintained by Ambari) should be same - otherwise the entity created via 
falconHook and hiveHook wont match and this will lead to lineage break
 Key: ATLAS-1191
 URL: https://issues.apache.org/jira/browse/ATLAS-1191
 Project: Atlas
  Issue Type: Bug
Affects Versions: trunk
Reporter: Ayub Khan
Priority: Critical
 Fix For: trunk


Falcon cluster entity name and the actual cluster name(maintained by Ambari) 
should be same - otherwise the entity created via falconHook and hiveHook wont 
match and this will lead to lineage break.

This is a limitation from Atlas side - which is not a right expectation from 
customer point of view.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (ATLAS-1190) Atlas UI compatibility with different browsers

[Ayub Khan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-1190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ayub Khan updated ATLAS-1190:
-
Assignee: Keval Bhatt

> Atlas UI compatibility with different browsers
> --
>
> Key: ATLAS-1190
> URL: https://issues.apache.org/jira/browse/ATLAS-1190
> Project: Atlas
>  Issue Type: Bug
>Affects Versions: trunk
>Reporter: Ayub Khan
>Assignee: Keval Bhatt
>Priority: Critical
> Fix For: trunk
>
>
> Test Atlas UI compatibility with different browsers and make sure it works 
> with all supported browsers.
> Off late, many customers are accessing Atlas UI via IE/edge and facing issues.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 47810: ATLAS-694: Update Atlas to use Graph DB abstraction layer

[Jeff Hagelberg]

> On Sept. 22, 2016, 11:12 a.m., Shwetha GS wrote:
> > repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java,
> >  line 422
> > 
> >
> > Isn't this also in Titan0Vertex?

Yes, I'll move this logic into a method in AtlasGraph so it can be reused in 
both places.


- Jeff


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47810/#review149947
---


On Sept. 23, 2016, 3:21 a.m., Jeff Hagelberg wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47810/
> ---
> 
> (Updated Sept. 23, 2016, 3:21 a.m.)
> 
> 
> Review request for atlas, David Kantor and Neeru Gupta.
> 
> 
> Bugs: ATLAS-694
> https://issues.apache.org/jira/browse/ATLAS-694
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> ATLAS-694: Update Atlas to use abstraction layer.  All of the Atlas code 
> (with the exception of the catalog, which was only updated minimally) has 
> been updated to use the graph database abstraction layer.  In addition, there 
> is now an optional Atlas configuration property that specifies the class of 
> the abstraction layer database to use.  I basically put all of the changes in 
> here with the exception of the actual Titan 1 implementation of code.  This 
> includes the changes to support Tinkerpop 3 syntax.  This is mostly to 
> expedite getting the changes into Atlas.  Originally the TP3 changes were 
> going to be brought in as part of the Titan 1 implementation task.
> 
> Change Summary:
> 
>- change Atlas classes to use AtlasGraph,AtlasVertex,AtlasEdge, etc 
> instead of TitanGraph/Vertex/Edge, etc
>- compile time dependency on titan 0.5.4/TP 2 removed (except in Catalog, 
> which was only changed to use AtlasGraphProvider/AtlasGraph) - see 
> repository\pom.xml, other pom.xmls
>- updated DSL translation to generate Gremlin that is compliant with TP3 
> when TP3 is being used.  See GremlinQuery.scala, 
> GraphPersistenceStrategies.scala
>- TitanGraphProvider replaced by AtlasGraphProvider.  Graph database 
> implementation is determined from a new optional configuration property
>- HiveTitanSample is no longer used by tests.  It has been replaced by 
> hive-instances.json (which uses normal Atlas json syntax).  The data is saved 
> with a new JSONImporter class.  This was needed because the graphson syntax 
> used by HiveTitanSample is not compatible with TP3.  
> 
> Last rebase: 9/21/2016
> 
> 
> Diffs
> -
> 
>   .gitignore e10adbc4457f6297600f0feb01eb54718b8ec406 
>   addons/falcon-bridge/pom.xml 1365bd05a388dc92f7a56c7f7427b5b85f97c7da 
>   addons/hdfs-model/pom.xml 492f39cea085c6e69781e17bcbdbc3a231806df3 
>   
> addons/hdfs-model/src/test/java/org/apache/atlas/fs/model/HDFSModelTest.java 
> ac60294e328835ba0340e150799ddfb348ccdb52 
>   addons/hive-bridge/pom.xml 6993bdb938a6095ca24482e290393eeeb3911bcb 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
>  ad7a4a5d09d8542a841701dfe04981f65f767c14 
>   addons/sqoop-bridge/pom.xml 8c9d278d43b5979ea1743d10845905c13249f8a6 
>   addons/storm-bridge/pom.xml 12c1208b448d456a923bd7309601174ddb561ba5 
>   catalog/pom.xml 2f58a8f0748de65ab78eab35df6abd2fe7c336af 
>   catalog/src/main/java/org/apache/atlas/catalog/query/BaseQuery.java 
> e7bb505075983371ca12d9bc1d8c6eb240c3d134 
>   distro/src/conf/atlas-application.properties 
> d334600dc5534840409b586157799ef3abf9abf2 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasEdge.java 
> dd4b7e614cdd9bf30f957fb6a839d8c60f3e1701 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasElement.java
>  1bc0fc38c0802897f32260520770a16795474d04 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 
> 995c5457ac7f807172f367cc8e3348b3a98dd6f3 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphIndex.java
>  41194d34f079842db0d95c73a8b099459f76ff2f 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphManagement.java
>  c8cd2842ca3090b6bbd384c773b4eb45aff149ce 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphQuery.java
>  93447495bcf18e9f19df9df68fd1cbe1427fc462 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasIndexQuery.java
>  e719d306ffe9f68e3ac6f7406baaf60a12390c34 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasPropertyKey.java
>  315ecddb861e1a1be6e0ab9b36fe4c0a52486ae8 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasVertex.java
>  fff6fb79247e7d0615ce83c4cbbd93d1bf8cf29c 
>   
> 

Re: Review Request 47810: ATLAS-694: Update Atlas to use Graph DB abstraction layer

[Jeff Hagelberg]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47810/
---

(Updated Sept. 23, 2016, 3:21 a.m.)


Review request for atlas, David Kantor and Neeru Gupta.


Changes
---

Addressed code review comments.  Fixed performance issue in 
GraphRepoMapperScaleTest.  The main issue there was extra processing done 
during set vertices.  That has been optimized, now the performance is 
comparable to what it was before the changes (actually, it was slightly better 
in my test runs).  Enabled checkstyle in new projects, addressed violations.  
Added synchronization check to AtlasGraphProvider.  Added @VisibleForTesting as 
requested.  Refactored/opimized check for multi-properties, now it is defined 
as a method in AtlasGraph and called from both places where it was used.  
Updated atlas-application.properties in typesystem to make use of the 
atlas.graphdb.backend property.  Reran all tests, no issues found.


Bugs: ATLAS-694
https://issues.apache.org/jira/browse/ATLAS-694


Repository: atlas


Description
---

ATLAS-694: Update Atlas to use abstraction layer.  All of the Atlas code (with 
the exception of the catalog, which was only updated minimally) has been 
updated to use the graph database abstraction layer.  In addition, there is now 
an optional Atlas configuration property that specifies the class of the 
abstraction layer database to use.  I basically put all of the changes in here 
with the exception of the actual Titan 1 implementation of code.  This includes 
the changes to support Tinkerpop 3 syntax.  This is mostly to expedite getting 
the changes into Atlas.  Originally the TP3 changes were going to be brought in 
as part of the Titan 1 implementation task.

Change Summary:

   - change Atlas classes to use AtlasGraph,AtlasVertex,AtlasEdge, etc instead 
of TitanGraph/Vertex/Edge, etc
   - compile time dependency on titan 0.5.4/TP 2 removed (except in Catalog, 
which was only changed to use AtlasGraphProvider/AtlasGraph) - see 
repository\pom.xml, other pom.xmls
   - updated DSL translation to generate Gremlin that is compliant with TP3 
when TP3 is being used.  See GremlinQuery.scala, 
GraphPersistenceStrategies.scala
   - TitanGraphProvider replaced by AtlasGraphProvider.  Graph database 
implementation is determined from a new optional configuration property
   - HiveTitanSample is no longer used by tests.  It has been replaced by 
hive-instances.json (which uses normal Atlas json syntax).  The data is saved 
with a new JSONImporter class.  This was needed because the graphson syntax 
used by HiveTitanSample is not compatible with TP3.  

Last rebase: 9/21/2016


Diffs (updated)
-

  .gitignore e10adbc4457f6297600f0feb01eb54718b8ec406 
  addons/falcon-bridge/pom.xml 1365bd05a388dc92f7a56c7f7427b5b85f97c7da 
  addons/hdfs-model/pom.xml 492f39cea085c6e69781e17bcbdbc3a231806df3 
  addons/hdfs-model/src/test/java/org/apache/atlas/fs/model/HDFSModelTest.java 
ac60294e328835ba0340e150799ddfb348ccdb52 
  addons/hive-bridge/pom.xml 6993bdb938a6095ca24482e290393eeeb3911bcb 
  
addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
 ad7a4a5d09d8542a841701dfe04981f65f767c14 
  addons/sqoop-bridge/pom.xml 8c9d278d43b5979ea1743d10845905c13249f8a6 
  addons/storm-bridge/pom.xml 12c1208b448d456a923bd7309601174ddb561ba5 
  catalog/pom.xml 2f58a8f0748de65ab78eab35df6abd2fe7c336af 
  catalog/src/main/java/org/apache/atlas/catalog/query/BaseQuery.java 
e7bb505075983371ca12d9bc1d8c6eb240c3d134 
  distro/src/conf/atlas-application.properties 
d334600dc5534840409b586157799ef3abf9abf2 
  graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasEdge.java 
dd4b7e614cdd9bf30f957fb6a839d8c60f3e1701 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasElement.java 
1bc0fc38c0802897f32260520770a16795474d04 
  graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 
995c5457ac7f807172f367cc8e3348b3a98dd6f3 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphIndex.java
 41194d34f079842db0d95c73a8b099459f76ff2f 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphManagement.java
 c8cd2842ca3090b6bbd384c773b4eb45aff149ce 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphQuery.java
 93447495bcf18e9f19df9df68fd1cbe1427fc462 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasIndexQuery.java
 e719d306ffe9f68e3ac6f7406baaf60a12390c34 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasPropertyKey.java
 315ecddb861e1a1be6e0ab9b36fe4c0a52486ae8 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasVertex.java 
fff6fb79247e7d0615ce83c4cbbd93d1bf8cf29c 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasVertexQuery.java
 

Re: Review Request 47810: ATLAS-694: Update Atlas to use Graph DB abstraction layer

[Jeff Hagelberg]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47810/
---

(Updated Sept. 23, 2016, 3:29 a.m.)


Review request for atlas, David Kantor and Neeru Gupta.


Bugs: ATLAS-694
https://issues.apache.org/jira/browse/ATLAS-694


Repository: atlas


Description (updated)
---

ATLAS-694: Update Atlas to use abstraction layer.  All of the Atlas code (with 
the exception of the catalog, which was only updated minimally) has been 
updated to use the graph database abstraction layer.  In addition, there is now 
an optional Atlas configuration property that specifies the class of the 
abstraction layer database to use.  I basically put all of the changes in here 
with the exception of the actual Titan 1 implementation of code.  This includes 
the changes to support Tinkerpop 3 syntax.  This is mostly to expedite getting 
the changes into Atlas.  Originally the TP3 changes were going to be brought in 
as part of the Titan 1 implementation task.

Change Summary:

   - change Atlas classes to use AtlasGraph,AtlasVertex,AtlasEdge, etc instead 
of TitanGraph/Vertex/Edge, etc
   - compile time dependency on titan 0.5.4/TP 2 removed (except in Catalog, 
which was only changed to use AtlasGraphProvider/AtlasGraph) - see 
repository\pom.xml, other pom.xmls
   - updated DSL translation to generate Gremlin that is compliant with TP3 
when TP3 is being used.  See GremlinQuery.scala, 
GraphPersistenceStrategies.scala
   - TitanGraphProvider replaced by AtlasGraphProvider.  Graph database 
implementation is determined from a new optional configuration property
   - HiveTitanSample is no longer used by tests.  It has been replaced by 
hive-instances.json (which uses normal Atlas json syntax).  The data is saved 
with a new JSONImporter class.  This was needed because the graphson syntax 
used by HiveTitanSample is not compatible with TP3.  

Last rebase: 9/22/2016


Diffs
-

  .gitignore e10adbc4457f6297600f0feb01eb54718b8ec406 
  addons/falcon-bridge/pom.xml 1365bd05a388dc92f7a56c7f7427b5b85f97c7da 
  addons/hdfs-model/pom.xml 492f39cea085c6e69781e17bcbdbc3a231806df3 
  addons/hdfs-model/src/test/java/org/apache/atlas/fs/model/HDFSModelTest.java 
ac60294e328835ba0340e150799ddfb348ccdb52 
  addons/hive-bridge/pom.xml 6993bdb938a6095ca24482e290393eeeb3911bcb 
  
addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
 ad7a4a5d09d8542a841701dfe04981f65f767c14 
  addons/sqoop-bridge/pom.xml 8c9d278d43b5979ea1743d10845905c13249f8a6 
  addons/storm-bridge/pom.xml 12c1208b448d456a923bd7309601174ddb561ba5 
  catalog/pom.xml 2f58a8f0748de65ab78eab35df6abd2fe7c336af 
  catalog/src/main/java/org/apache/atlas/catalog/query/BaseQuery.java 
e7bb505075983371ca12d9bc1d8c6eb240c3d134 
  distro/src/conf/atlas-application.properties 
d334600dc5534840409b586157799ef3abf9abf2 
  graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasEdge.java 
dd4b7e614cdd9bf30f957fb6a839d8c60f3e1701 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasElement.java 
1bc0fc38c0802897f32260520770a16795474d04 
  graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 
995c5457ac7f807172f367cc8e3348b3a98dd6f3 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphIndex.java
 41194d34f079842db0d95c73a8b099459f76ff2f 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphManagement.java
 c8cd2842ca3090b6bbd384c773b4eb45aff149ce 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphQuery.java
 93447495bcf18e9f19df9df68fd1cbe1427fc462 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasIndexQuery.java
 e719d306ffe9f68e3ac6f7406baaf60a12390c34 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasPropertyKey.java
 315ecddb861e1a1be6e0ab9b36fe4c0a52486ae8 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasVertex.java 
fff6fb79247e7d0615ce83c4cbbd93d1bf8cf29c 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasVertexQuery.java
 366cfe131512f36c02f59ad7e969ff94dfb0e12c 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/GraphDatabase.java
 37acda5f6ddb883ee364d08a31251b071e6a89a5 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/GremlinVersion.java
 f328d3cb9d2b695bf0eb9981d7046bb67d0eac7d 
  
graphdb/common/src/main/java/org/apache/atlas/repository/graphdb/titan/query/NativeTitanGraphQuery.java
 b79bf7964ae9c830cfa28b9f2f1a1a1059d67223 
  
graphdb/common/src/main/java/org/apache/atlas/repository/graphdb/titan/query/NativeTitanQueryFactory.java
 5c654119a4292136e8ff935c34dd6e39741bffc1 
  
graphdb/common/src/main/java/org/apache/atlas/repository/graphdb/titan/query/TitanGraphQuery.java
 f23847a2f75c351ad307680d0642bc1da49fc59f 
  

[jira] [Comment Edited] (ATLAS-694) Update Atlas code to use abstraction layer

[Jeffrey Hagelberg (JIRA)]

[ 
https://issues.apache.org/jira/browse/ATLAS-694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15515256#comment-15515256
 ] 

Jeffrey Hagelberg edited comment on ATLAS-694 at 9/23/16 3:28 AM:
--

New patch uploaded.  Addressed code review comments and performance regression.


was (Author: jnhagelb):
Addressed code review comments and performance regression.

> Update Atlas code to use abstraction layer
> --
>
> Key: ATLAS-694
> URL: https://issues.apache.org/jira/browse/ATLAS-694
> Project: Atlas
>  Issue Type: Sub-task
>Reporter: Jeffrey Hagelberg
>Assignee: Jeffrey Hagelberg
> Attachments: rb47810.patch
>
>
> In this task, we will update Atlas to replace the direct usage of titan 
> classes with usage of the equivalent interfaces in the abstraction layer.  At 
> this point, there will be no options to configure the abstraction layer 
> implementation that gets used.  Atlas will be hard-coded to use the titan 
> 0.5.4 implementation.  It will also continue to have a compile time 
> dependency on the titan 0.5.4 graph database implementation.  When this task 
> is complete, Atlas will only be accessing titan through the abstraction layer.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (ATLAS-1189) Server Signature validation on client failed while using only CA in the client truststore

[Libois Claude (JIRA)]

Libois Claude created ATLAS-1189:


 Summary: Server Signature validation on client failed while using 
only CA in the client truststore
 Key: ATLAS-1189
 URL: https://issues.apache.org/jira/browse/ATLAS-1189
 Project: Atlas
  Issue Type: Bug
Reporter: Libois Claude






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 51939: Framework to apply updates to types in the type-system

[Sarath Kumar Subramanian]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51939/
---

(Updated Sept. 22, 2016, 12:34 a.m.)


Review request for atlas, Madhan Neethiraj, Shwetha GS, and Suma Shivaprasad.


Changes
---

removed all json operations to change typedef and used appropriate pojo 
classes, moved check for patches method in AtlasTypeAttributePatch, 
incorporated madhan and shwetha's comments.


Bugs: ATLAS-1174
https://issues.apache.org/jira/browse/ATLAS-1174


Repository: atlas


Description
---

1. Introduce "version" attribute to all types in the type-system, this helps to 
track changes made to the default types (hive, sqoop, falcon and storm types) 
and user created types. If version is not mentioned during creation of a type, 
default version "1.0" is assigned (optional attribute).
2. Using the version attributed for types, introduce a patch framework for type 
system. This framework applies patches to a type using its version number and 
can be used during upgrade - add new attributes to an existing types and it 
will be run during atlas startup.
The sequence of steps:
a. During atlas startup, check $ATLAS_HOME/models/patches directory for any 
available patch files (json files). If there any patch files handle them.
b. Sample patch json file looks like:
{
"patches": [
{ 
"action": "ADD_ATTRIBUTE",
"typeName": "hive_column",
"applyToVersion": "1.0",
"updateToVersion": "2.0",
"actionParams": [
{ "name": "position", "dataTypeName": "int", "multiplicity": "optional", 
"isComposite": false, "isUnique": false, "isIndexable": false, 
"reverseAttributeName": null }
]
} ]
}
c. The framework updates the type in "typeName" for the matching version number 
and after applying the patch, update the version to the one mentioned in 
"updateToVersion"
d. The json file can have more than one action (array of actions).
e. There can be multiple patch json files in the directory and are applied in 
the sort order of the filename. eg:
001-hive_column_add_position.json
002-hive_column_add_anotherattribute.json


Diffs (updated)
-

  common/src/main/java/org/apache/atlas/AtlasConstants.java 17ffbd7 
  common/src/main/java/org/apache/atlas/repository/Constants.java d7f9c89 
  
repository/src/main/java/org/apache/atlas/repository/typestore/GraphBackedTypeStore.java
 a94d157 
  
repository/src/main/java/org/apache/atlas/services/AtlasTypeAttributePatch.java 
PRE-CREATION 
  repository/src/main/java/org/apache/atlas/services/AtlasTypePatch.java 
PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java 
6a937f4 
  
typesystem/src/main/java/org/apache/atlas/typesystem/types/AbstractDataType.java
 fad091d 
  typesystem/src/main/java/org/apache/atlas/typesystem/types/ClassType.java 
c56987a 
  typesystem/src/main/java/org/apache/atlas/typesystem/types/EnumType.java 
bdd0a13 
  
typesystem/src/main/java/org/apache/atlas/typesystem/types/EnumTypeDefinition.java
 6340615 
  
typesystem/src/main/java/org/apache/atlas/typesystem/types/HierarchicalType.java
 7224699 
  
typesystem/src/main/java/org/apache/atlas/typesystem/types/HierarchicalTypeDefinition.java
 9a299f0 
  typesystem/src/main/java/org/apache/atlas/typesystem/types/IDataType.java 
85ddee7 
  typesystem/src/main/java/org/apache/atlas/typesystem/types/StructType.java 
6f40c1d 
  
typesystem/src/main/java/org/apache/atlas/typesystem/types/StructTypeDefinition.java
 f1ce1b7 
  typesystem/src/main/java/org/apache/atlas/typesystem/types/TraitType.java 
f23bf5b 
  typesystem/src/main/java/org/apache/atlas/typesystem/types/TypeSystem.java 
70ba89b 
  
typesystem/src/main/java/org/apache/atlas/typesystem/types/utils/TypesUtil.java 
ef8448d 
  
typesystem/src/main/scala/org/apache/atlas/typesystem/json/TypesSerialization.scala
 5618938 

Diff: https://reviews.apache.org/r/51939/diff/


Testing
---


Thanks,

Sarath Kumar Subramanian

Re: Review Request 51939: Framework to apply updates to types in the type-system

[Sarath Kumar Subramanian]

> On Sept. 21, 2016, 1:55 p.m., Madhan Neethiraj wrote:
> > repository/src/main/java/org/apache/atlas/services/AtlasTypePatch.java, 
> > line 77
> > 
> >
> > Consider adding attributeDefinitions as a first class attribute - to 
> > save from having to tranform from/to JSON in many places. When we add 
> > support for other type of patches (like types..), appropraite members can 
> > be added here.
> > 
> > class PatchData {
> >   private String   action;
> >   private String   typeName;
> >   ..
> >   private List attributeDefinitions;
> >   private Map  params;
> > }

removed all json operations and now using pojo classes


> On Sept. 21, 2016, 1:55 p.m., Madhan Neethiraj wrote:
> > repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java,
> >  line 172
> > 
> >
> > checkForPatches() call will be missed in HA deployments. Make sure to 
> > call from instanceIsActive() - just as restoreTypeSystem() is called.

instanceIsActive() is not the right method for invoking checkPatches() method. 
Moved the method outside the non HA validation check.


- Sarath Kumar


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51939/#review149854
---


On Sept. 22, 2016, 12:34 a.m., Sarath Kumar Subramanian wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51939/
> ---
> 
> (Updated Sept. 22, 2016, 12:34 a.m.)
> 
> 
> Review request for atlas, Madhan Neethiraj, Shwetha GS, and Suma Shivaprasad.
> 
> 
> Bugs: ATLAS-1174
> https://issues.apache.org/jira/browse/ATLAS-1174
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> 1. Introduce "version" attribute to all types in the type-system, this helps 
> to track changes made to the default types (hive, sqoop, falcon and storm 
> types) and user created types. If version is not mentioned during creation of 
> a type, default version "1.0" is assigned (optional attribute).
> 2. Using the version attributed for types, introduce a patch framework for 
> type system. This framework applies patches to a type using its version 
> number and can be used during upgrade - add new attributes to an existing 
> types and it will be run during atlas startup.
> The sequence of steps:
> a. During atlas startup, check $ATLAS_HOME/models/patches directory for any 
> available patch files (json files). If there any patch files handle them.
> b. Sample patch json file looks like:
> {
> "patches": [
> { 
> "action": "ADD_ATTRIBUTE",
> "typeName": "hive_column",
> "applyToVersion": "1.0",
> "updateToVersion": "2.0",
> "actionParams": [
> { "name": "position", "dataTypeName": "int", "multiplicity": "optional", 
> "isComposite": false, "isUnique": false, "isIndexable": false, 
> "reverseAttributeName": null }
> ]
> } ]
> }
> c. The framework updates the type in "typeName" for the matching version 
> number and after applying the patch, update the version to the one mentioned 
> in "updateToVersion"
> d. The json file can have more than one action (array of actions).
> e. There can be multiple patch json files in the directory and are applied in 
> the sort order of the filename. eg:
> 001-hive_column_add_position.json
> 002-hive_column_add_anotherattribute.json
> 
> 
> Diffs
> -
> 
>   common/src/main/java/org/apache/atlas/AtlasConstants.java 17ffbd7 
>   common/src/main/java/org/apache/atlas/repository/Constants.java d7f9c89 
>   
> repository/src/main/java/org/apache/atlas/repository/typestore/GraphBackedTypeStore.java
>  a94d157 
>   
> repository/src/main/java/org/apache/atlas/services/AtlasTypeAttributePatch.java
>  PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/services/AtlasTypePatch.java 
> PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java
>  6a937f4 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/AbstractDataType.java
>  fad091d 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/ClassType.java 
> c56987a 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/EnumType.java 
> bdd0a13 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/EnumTypeDefinition.java
>  6340615 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/HierarchicalType.java
>  7224699 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/HierarchicalTypeDefinition.java
>  9a299f0 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/IDataType.java 
> 85ddee7 
>   

Re: Review Request 51939: Framework to apply updates to types in the type-system

[Sarath Kumar Subramanian]

On Sept. 21, 2016, 7:46 a.m., Sarath Kumar Subramanian wrote:
> > Add tests
> > 
> > 1. Currently, the model files(like hive_model.json) are auto generated from 
> > model definitions defined in java(like HiveDataModelGenerator). The patch 
> > files in this case has to be hand coded which is error prone
> > 2. For completeness, readability and debuggability, the type update has to 
> > be done in the corresponding model definitions like HiveDataModelGenerator. 
> > So, same data will be in two places and the model definitions and the patch 
> > files can go out of sync
> > 3. Since the model definitions(like HiveDataModelGenerator) will be updated 
> > anyways, if we modify ReservedTypesRegistrar to do type update instead of 
> > type create, the type updates will automatically be taken care with the 
> > same model json. So, model update patches are not necessary then
> > 3. This jira doesn't implement type versioning - doesn't have support for 
> > storing multiple versions of the type. But it maintains the version of the 
> > latest type definition which I think is useful for debugging, to know the 
> > version of type that the server knows. Can we maintain this info in 
> > HiveDataModelGenerator itself, and hence will be part of hive_model.json

1. patch json genarator can be addressed in a separate jiira
2. this is only for upgrade scenarios
3. lets address this in a separate jiira.
4. we dont want to maintain multiple versions of the same type in the 
typesystem, will confuse users on which version to use to create entities.


- Sarath Kumar


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51939/#review149811
---


On Sept. 22, 2016, 12:34 a.m., Sarath Kumar Subramanian wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51939/
> ---
> 
> (Updated Sept. 22, 2016, 12:34 a.m.)
> 
> 
> Review request for atlas, Madhan Neethiraj, Shwetha GS, and Suma Shivaprasad.
> 
> 
> Bugs: ATLAS-1174
> https://issues.apache.org/jira/browse/ATLAS-1174
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> 1. Introduce "version" attribute to all types in the type-system, this helps 
> to track changes made to the default types (hive, sqoop, falcon and storm 
> types) and user created types. If version is not mentioned during creation of 
> a type, default version "1.0" is assigned (optional attribute).
> 2. Using the version attributed for types, introduce a patch framework for 
> type system. This framework applies patches to a type using its version 
> number and can be used during upgrade - add new attributes to an existing 
> types and it will be run during atlas startup.
> The sequence of steps:
> a. During atlas startup, check $ATLAS_HOME/models/patches directory for any 
> available patch files (json files). If there any patch files handle them.
> b. Sample patch json file looks like:
> {
> "patches": [
> { 
> "action": "ADD_ATTRIBUTE",
> "typeName": "hive_column",
> "applyToVersion": "1.0",
> "updateToVersion": "2.0",
> "actionParams": [
> { "name": "position", "dataTypeName": "int", "multiplicity": "optional", 
> "isComposite": false, "isUnique": false, "isIndexable": false, 
> "reverseAttributeName": null }
> ]
> } ]
> }
> c. The framework updates the type in "typeName" for the matching version 
> number and after applying the patch, update the version to the one mentioned 
> in "updateToVersion"
> d. The json file can have more than one action (array of actions).
> e. There can be multiple patch json files in the directory and are applied in 
> the sort order of the filename. eg:
> 001-hive_column_add_position.json
> 002-hive_column_add_anotherattribute.json
> 
> 
> Diffs
> -
> 
>   common/src/main/java/org/apache/atlas/AtlasConstants.java 17ffbd7 
>   common/src/main/java/org/apache/atlas/repository/Constants.java d7f9c89 
>   
> repository/src/main/java/org/apache/atlas/repository/typestore/GraphBackedTypeStore.java
>  a94d157 
>   
> repository/src/main/java/org/apache/atlas/services/AtlasTypeAttributePatch.java
>  PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/services/AtlasTypePatch.java 
> PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java
>  6a937f4 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/AbstractDataType.java
>  fad091d 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/ClassType.java 
> c56987a 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/EnumType.java 
> bdd0a13 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/EnumTypeDefinition.java
>  6340615 
>   
> 

[jira] [Created] (ATLAS-1188) cannot create vertex_index edge_index fulltext_index

[zhangqiang2 (JIRA)]

zhangqiang2 created ATLAS-1188:
--

 Summary: cannot create  vertex_index edge_index fulltext_index
 Key: ATLAS-1188
 URL: https://issues.apache.org/jira/browse/ATLAS-1188
 Project: Atlas
  Issue Type: Bug
Affects Versions: 0.7-incubating
 Environment: 
hadoop2.6.0+hbase1.1.5+solr5.5.2+zookeeper3.4.8+hive1.2.1+redhat6.5
Reporter: zhangqiang2


hi ,  when i run the shell  atlas_start.py,   i find it cannot work well ,when 
i look the application.log i find it does not have automication create file( 
vertex_index ,edge_index , fulltext_index) 
 so i try create three files use solr shell ,then i  find it can work well . 
later i looked  atlas_start.py ,  i find in this bash it include the shell to 
create ( vertex_index ,edge_index , fulltext_index).   however it really doesnt 
work well。   i am a newer for  atlas but i really want to know more about this  
projecct 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (ATLAS-1181) UI changes for column level lineage

[Keval Bhatt (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-1181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Keval Bhatt reassigned ATLAS-1181:
--

Assignee: Keval Bhatt

> UI changes for column level lineage
> ---
>
> Key: ATLAS-1181
> URL: https://issues.apache.org/jira/browse/ATLAS-1181
> Project: Atlas
>  Issue Type: Improvement
>Reporter: Vimal Sharma
>Assignee: Keval Bhatt
>
> While showing column level lineage, qualifiedName attribute of the column 
> should be used instead of the name attribute. There are chances that the 
> columns with the same name are present in different tables.
> The lineage diagram would be ambiguous if such columns are part of a column 
> lineage process.
> Further, dependencyType attribute can be shown along with the CTAS query in  
> lineage process details



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (ATLAS-1183) UI: help link should point to atlas website

[Keval Bhatt (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-1183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Keval Bhatt reassigned ATLAS-1183:
--

Assignee: Keval Bhatt

> UI: help link should point to atlas website
> ---
>
> Key: ATLAS-1183
> URL: https://issues.apache.org/jira/browse/ATLAS-1183
> Project: Atlas
>  Issue Type: Bug
>Reporter: Shwetha G S
>Assignee: Keval Bhatt
> Fix For: 0.8-incubating
>
>
> Help link points to 
> https://cwiki.apache.org/confluence/pages/viewpage.action?spaceKey=ATLAS=Atlas+Home
>  which requires login. Instead, link to http://atlas.incubator.apache.org/



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Review Request 52166: Allow optional attributes to be removed when updating type

[Sarath Kumar Subramanian]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52166/
---

Review request for atlas, Madhan Neethiraj, shweta agarwal, and Suma 
Shivaprasad.


Bugs: ATLAS-1175
https://issues.apache.org/jira/browse/ATLAS-1175


Repository: atlas


Description
---

we restrict optional attributes to be removed from the type system. This needs 
to be allowed


Diffs
-

  typesystem/src/main/java/org/apache/atlas/typesystem/types/TypeUtils.java 
f5c2ce9 

Diff: https://reviews.apache.org/r/52166/diff/


Testing
---

tested using REST calls to delete optional and required attributes and only 
optional attributes were removed.


Thanks,

Sarath Kumar Subramanian

[jira] [Commented] (ATLAS-1175) Type updates should allow removal of optional attributes

[Sarath Subramanian (JIRA)]

[ 
https://issues.apache.org/jira/browse/ATLAS-1175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15514065#comment-15514065
 ] 

Sarath Subramanian commented on ATLAS-1175:
---

It is a corner case -  this needs to be addressed and we should not be 
surfacing values for deleted attributes when you re-add the same attribute. We 
may need to include a flag when removing attribute to check for all entities of 
the type to safely remove attribute value from titan. This work requires a 
separate jiira to handle safe removal of optional attributes and clean entity 
in titan.

> Type updates should allow removal of optional attributes
> 
>
> Key: ATLAS-1175
> URL: https://issues.apache.org/jira/browse/ATLAS-1175
> Project: Atlas
>  Issue Type: Bug
>Affects Versions: 0.7-incubating, 0.8-incubating
>Reporter: Suma Shivaprasad
>Assignee: Sarath Subramanian
>
> Currently optional attributes are not allowed to be removed from a given 
> type. This should be allowed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 51939: Framework to apply updates to types in the type-system

[Madhan Neethiraj]

On Sept. 21, 2016, 2:46 p.m., Sarath Kumar Subramanian wrote:
> > Add tests
> > 
> > 1. Currently, the model files(like hive_model.json) are auto generated from 
> > model definitions defined in java(like HiveDataModelGenerator). The patch 
> > files in this case has to be hand coded which is error prone
> > 2. For completeness, readability and debuggability, the type update has to 
> > be done in the corresponding model definitions like HiveDataModelGenerator. 
> > So, same data will be in two places and the model definitions and the patch 
> > files can go out of sync
> > 3. Since the model definitions(like HiveDataModelGenerator) will be updated 
> > anyways, if we modify ReservedTypesRegistrar to do type update instead of 
> > type create, the type updates will automatically be taken care with the 
> > same model json. So, model update patches are not necessary then
> > 3. This jira doesn't implement type versioning - doesn't have support for 
> > storing multiple versions of the type. But it maintains the version of the 
> > latest type definition which I think is useful for debugging, to know the 
> > version of type that the server knows. Can we maintain this info in 
> > HiveDataModelGenerator itself, and hence will be part of hive_model.json
> 
> Sarath Kumar Subramanian wrote:
> 1. patch json genarator can be addressed in a separate jiira
> 2. this is only for upgrade scenarios
> 3. lets address this in a separate jiira.
> 4. we dont want to maintain multiple versions of the same type in the 
> typesystem, will confuse users on which version to use to create entities.
> 
> Shwetha GS wrote:
> What benefit does this patch framework add when the type update can be 
> done with the existing functionality itself (Type versioning is useful and 
> can be done without the patch framework). The patch framework adds more 
> overhead, and I don't see the necessity
> 
> David Radley wrote:
> I agree - I think this capability would be better placed in separate 
> tooling that uses the Atlas REST APIs.

@Shwetha, @David - the rational for the patch framework is exactly same as the 
need for reading contents of "models" directory during startup and initializing 
the typesystem. This just makes it easier to deal with updates/additions to 
typesystem during software upgrade/patch.


- Madhan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51939/#review149811
---


On Sept. 22, 2016, 7:34 a.m., Sarath Kumar Subramanian wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51939/
> ---
> 
> (Updated Sept. 22, 2016, 7:34 a.m.)
> 
> 
> Review request for atlas, Madhan Neethiraj, Shwetha GS, and Suma Shivaprasad.
> 
> 
> Bugs: ATLAS-1174
> https://issues.apache.org/jira/browse/ATLAS-1174
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> 1. Introduce "version" attribute to all types in the type-system, this helps 
> to track changes made to the default types (hive, sqoop, falcon and storm 
> types) and user created types. If version is not mentioned during creation of 
> a type, default version "1.0" is assigned (optional attribute).
> 2. Using the version attributed for types, introduce a patch framework for 
> type system. This framework applies patches to a type using its version 
> number and can be used during upgrade - add new attributes to an existing 
> types and it will be run during atlas startup.
> The sequence of steps:
> a. During atlas startup, check $ATLAS_HOME/models/patches directory for any 
> available patch files (json files). If there any patch files handle them.
> b. Sample patch json file looks like:
> {
> "patches": [
> { 
> "action": "ADD_ATTRIBUTE",
> "typeName": "hive_column",
> "applyToVersion": "1.0",
> "updateToVersion": "2.0",
> "actionParams": [
> { "name": "position", "dataTypeName": "int", "multiplicity": "optional", 
> "isComposite": false, "isUnique": false, "isIndexable": false, 
> "reverseAttributeName": null }
> ]
> } ]
> }
> c. The framework updates the type in "typeName" for the matching version 
> number and after applying the patch, update the version to the one mentioned 
> in "updateToVersion"
> d. The json file can have more than one action (array of actions).
> e. There can be multiple patch json files in the directory and are applied in 
> the sort order of the filename. eg:
> 001-hive_column_add_position.json
> 002-hive_column_add_anotherattribute.json
> 
> 
> Diffs
> -
> 
>   common/src/main/java/org/apache/atlas/AtlasConstants.java 17ffbd7 
>   common/src/main/java/org/apache/atlas/repository/Constants.java d7f9c89 
>   
> 

Re: Review Request 47810: ATLAS-694: Update Atlas to use Graph DB abstraction layer

[Shwetha GS]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47810/#review149947
---




catalog/pom.xml (line 67)


for now, catalog can only be used with Titan 0.5.4 - why?



graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasPropertyKey.java
 (line 33)


There is AtlasPropertyKey.Cardinality and AtlasCardinality. Whats the 
difference?



graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/GraphDbObjectFactory.java
 (line 36)


The javadoc params are out of date



graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/Titan0DatabaseManager.java
 (line 40)


Rename to Titan0GraphManagement? In general, rename Database to Graph for 
all new classes?



pom.xml (line 457)


where is this used? git grep returns just this result



repository/src/main/java/org/apache/atlas/discovery/graph/DefaultGraphPersistenceStrategy.java
 (line 148)






repository/src/main/java/org/apache/atlas/repository/graph/AtlasGraphProvider.java
 (line 45)


You will need synchronized locking here - 
http://www.cs.umd.edu/~pugh/java/memoryModel/DoubleCheckedLocking.html



repository/src/main/java/org/apache/atlas/repository/graph/AtlasGraphProvider.java
 (line 57)


add visible for testing annotation. Will be good if you can move to some 
test class instead



repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java
 (line 82)


add visiblefortesting annotation



repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java 
(line 414)


Isn't this also in Titan0Vertex?


Enable checkstyle on new modules with this property in pom.xml - 
true

With patch:
Running org.apache.atlas.repository.graph.GraphRepoMapperScaleTest
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 113.27 sec - in 
org.apache.atlas.repository.graph.GraphRepoMapperScaleTest

Without patch:
Running org.apache.atlas.repository.graph.GraphRepoMapperScaleTest
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 89.008 sec - in 
org.apache.atlas.repository.graph.GraphRepoMapperScaleTest

With the patch, the test takes longer to run. This implies some performance 
issue with the patch

Whats the plan on including titan 1.x implementation - will it be a profile at 
compile time such that only one implementation(titan 0.x or titan 1.x) will be 
part of the package. Or are both implementations part of the package and the 
implementation is chosen from the runtime configuration in application 
properties? If its compile time binding, then we shouldn't expose the conf 
atlas.graphdb.backend?

- Shwetha GS


On Sept. 21, 2016, 1:49 p.m., Jeff Hagelberg wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47810/
> ---
> 
> (Updated Sept. 21, 2016, 1:49 p.m.)
> 
> 
> Review request for atlas, David Kantor and Neeru Gupta.
> 
> 
> Bugs: ATLAS-694
> https://issues.apache.org/jira/browse/ATLAS-694
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> ATLAS-694: Update Atlas to use abstraction layer.  All of the Atlas code 
> (with the exception of the catalog, which was only updated minimally) has 
> been updated to use the graph database abstraction layer.  In addition, there 
> is now an optional Atlas configuration property that specifies the class of 
> the abstraction layer database to use.  I basically put all of the changes in 
> here with the exception of the actual Titan 1 implementation of code.  This 
> includes the changes to support Tinkerpop 3 syntax.  This is mostly to 
> expedite getting the changes into Atlas.  Originally the TP3 changes were 
> going to be brought in as part of the Titan 1 implementation task.
> 
> Change Summary:
> 
>- change Atlas classes to use AtlasGraph,AtlasVertex,AtlasEdge, etc 
> instead of TitanGraph/Vertex/Edge, etc
>- compile time dependency on titan 0.5.4/TP 2 removed (except in Catalog, 
> which was only changed to use AtlasGraphProvider/AtlasGraph) - see 
> repository\pom.xml, other pom.xmls
>- updated DSL translation to generate Gremlin that is compliant with TP3 
> when TP3 is being used.  See GremlinQuery.scala, 
> 

Re: Review Request 52077: Column level lineage in Hive

[Vimal Sharma]

> On Sept. 21, 2016, 9:01 p.m., Suma Shivaprasad wrote:
> > addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java, 
> > line 628
> > 
> >
> > shouldnt the lineage process refer to the same set of column instances 
> > which are already part of a table reference. Why are we recreating them?

Column Referenceable instances are not re-created. Column Referenceable 
instances which are part of table reference are stored in Map columnQNameToRef 
and then this map is used in function createColumnLineageObjects to set column 
lineage process.

The name createColumnLineageObjects is misleading though. I have changed it to 
createColumnLineageProcessInstances


- Vimal


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52077/#review149897
---


On Sept. 20, 2016, 9:07 a.m., Vimal Sharma wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/52077/
> ---
> 
> (Updated Sept. 20, 2016, 9:07 a.m.)
> 
> 
> Review request for atlas.
> 
> 
> Bugs: ATLAS-247
> https://issues.apache.org/jira/browse/ATLAS-247
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> After a CTAS query, lineage relationship between source columns and 
> destination column can be captured. This information can be used to create a 
> column lineage process.
> 
> 
> Diffs
> -
> 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/ColumnLineageUtils.java
>  PRE-CREATION 
>   addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
> a3464a0 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/model/HiveDataModelGenerator.java
>  45f0bc9 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/model/HiveDataTypes.java
>  e094cb6 
>   addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 
> a5838b4 
> 
> Diff: https://reviews.apache.org/r/52077/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Vimal Sharma
> 
>

Re: Review Request 52077: Column level lineage in Hive

[Vimal Sharma]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52077/
---

(Updated Sept. 22, 2016, 11:53 a.m.)


Review request for atlas.


Changes
---

Fixed review comments from Shwetha and Suma. Added try catch to make sure that 
Atlas doesn't fail to register CTAS process when Column lineage can't be set 
due to absence of lineage data from Hive Hook.


Bugs: ATLAS-247
https://issues.apache.org/jira/browse/ATLAS-247


Repository: atlas


Description
---

After a CTAS query, lineage relationship between source columns and destination 
column can be captured. This information can be used to create a column lineage 
process.


Diffs (updated)
-

  
addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/ColumnLineageUtils.java
 PRE-CREATION 
  addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
a3464a0 
  
addons/hive-bridge/src/main/java/org/apache/atlas/hive/model/HiveDataModelGenerator.java
 45f0bc9 
  
addons/hive-bridge/src/main/java/org/apache/atlas/hive/model/HiveDataTypes.java 
e094cb6 
  addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 
a5838b4 

Diff: https://reviews.apache.org/r/52077/diff/


Testing
---


Thanks,

Vimal Sharma

[jira] [Updated] (ATLAS-1183) UI: help link should point to atlas website

[Keval Bhatt (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-1183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Keval Bhatt updated ATLAS-1183:
---
Attachment: ATLAS-1183.patch

> UI: help link should point to atlas website
> ---
>
> Key: ATLAS-1183
> URL: https://issues.apache.org/jira/browse/ATLAS-1183
> Project: Atlas
>  Issue Type: Bug
>Reporter: Shwetha G S
>Assignee: Keval Bhatt
> Fix For: 0.8-incubating
>
> Attachments: ATLAS-1183.patch
>
>
> Help link points to 
> https://cwiki.apache.org/confluence/pages/viewpage.action?spaceKey=ATLAS=Atlas+Home
>  which requires login. Instead, link to http://atlas.incubator.apache.org/



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 52077: Column level lineage in Hive

[Vimal Sharma]

> On Sept. 21, 2016, 8:58 p.m., Suma Shivaprasad wrote:
> > addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/ColumnLineageUtils.java,
> >  line 97
> > 
> >
> > why is column qualifiedName different from the convention we are using 
> > for hive_column instances which are referred to from the table. Why is 
> > clustername removed?

Cluster information is not available in Lineage information provided by Hive. 
Further, qualifiedName used in this patch is used only while setting column 
lineage and is not used for communication with rest of Atlas codebase.


- Vimal


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52077/#review149896
---


On Sept. 20, 2016, 9:07 a.m., Vimal Sharma wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/52077/
> ---
> 
> (Updated Sept. 20, 2016, 9:07 a.m.)
> 
> 
> Review request for atlas.
> 
> 
> Bugs: ATLAS-247
> https://issues.apache.org/jira/browse/ATLAS-247
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> After a CTAS query, lineage relationship between source columns and 
> destination column can be captured. This information can be used to create a 
> column lineage process.
> 
> 
> Diffs
> -
> 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/ColumnLineageUtils.java
>  PRE-CREATION 
>   addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
> a3464a0 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/model/HiveDataModelGenerator.java
>  45f0bc9 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/model/HiveDataTypes.java
>  e094cb6 
>   addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 
> a5838b4 
> 
> Diff: https://reviews.apache.org/r/52077/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Vimal Sharma
> 
>

Re: Review Request 47810: ATLAS-694: Update Atlas to use Graph DB abstraction layer

[Jeff Hagelberg]

> On Sept. 22, 2016, 11:12 a.m., Shwetha GS wrote:
> > catalog/pom.xml, line 67
> > 
> >
> > for now, catalog can only be used with Titan 0.5.4 - why?

A good portion of the catalog project, especially the ones in the query 
package, make direct use of TP2-specific classes such as GremlinPipeline.  TP3 
uses a different set of packages.  It would take a good deal of work to 
refactor the catalog project to generate gremlin that is compatible with both 
TP2 and TP3.  To reduce the scope, I did not tackle that in this initial 
implementation.  The query generation logic here will be need to use some 
abstraction that is independent of the TP version.  If done right, this could 
also be used to help clean up the DSL translation logic as well (See my 
response to Suma's comment about that).  At this point, I would really like to 
focus on getting the general framework in place, which these changes covers.  I 
think things making catalog work with TP3 should be treated as a separate JIRA.


> On Sept. 22, 2016, 11:12 a.m., Shwetha GS wrote:
> > graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasPropertyKey.java,
> >  line 33
> > 
> >
> > There is AtlasPropertyKey.Cardinality and AtlasCardinality. Whats the 
> > difference?

There isn't a difference.  I'll consolidate them.


> On Sept. 22, 2016, 11:12 a.m., Shwetha GS wrote:
> > repository/src/main/java/org/apache/atlas/discovery/graph/DefaultGraphPersistenceStrategy.java,
> >  line 148
> > 
> >
> >

Fixed to use GraphHelper.


> On Sept. 22, 2016, 11:12 a.m., Shwetha GS wrote:
> > repository/src/main/java/org/apache/atlas/repository/graph/AtlasGraphProvider.java,
> >  line 57
> > 
> >
> > add visible for testing annotation. Will be good if you can move to 
> > some test class instead

Annotations have been added.  I found it useful to have the methods in this 
class because they are used by many different test projects.


> On Sept. 22, 2016, 11:12 a.m., Shwetha GS wrote:
> > pom.xml, line 457
> > 
> >
> > where is this used? git grep returns just this result

I guess that at this point it is not needed. since Titan0Database (now 
Titan0GraphDatabase) is the default.  Once support for other implementations is 
merged in, that will be needed to control which implementation class ends up in 
the test atlas-application.properties file in typesystem.  I did include the 
changes to those property files though.  I guess I'll go ahead and add them in.


On Sept. 22, 2016, 11:12 a.m., Jeff Hagelberg wrote:
> > Enable checkstyle on new modules with this property in pom.xml - 
> > true
> > 
> > With patch:
> > Running org.apache.atlas.repository.graph.GraphRepoMapperScaleTest
> > Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 113.27 sec 
> > - in org.apache.atlas.repository.graph.GraphRepoMapperScaleTest
> > 
> > Without patch:
> > Running org.apache.atlas.repository.graph.GraphRepoMapperScaleTest
> > Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 89.008 sec 
> > - in org.apache.atlas.repository.graph.GraphRepoMapperScaleTest
> > 
> > With the patch, the test takes longer to run. This implies some performance 
> > issue with the patch
> > 
> > Whats the plan on including titan 1.x implementation - will it be a profile 
> > at compile time such that only one implementation(titan 0.x or titan 1.x) 
> > will be part of the package. Or are both implementations part of the 
> > package and the implementation is chosen from the runtime configuration in 
> > application properties? If its compile time binding, then we shouldn't 
> > expose the conf atlas.graphdb.backend?

The plan is for all of the Atlas code to just depend on the abstraction layer.  
There will be profiles in incubator-atlas\pom.xml for each graph backend.  The 
ones we have in our fork look like this:


titan1


org.apache.atlas.repository.graphdb.titan1.Titan1Database




org.apache.atlas
atlas-graphdb-impls
${project.version}
pom


org.apache.atlas
atlas-graphdb-titan0


com.ibm.analytics
atlas-graphdb-ibm-graph

Re: Review Request 51939: Framework to apply updates to types in the type-system

[David Radley]

On Sept. 21, 2016, 2:46 p.m., Sarath Kumar Subramanian wrote:
> > Add tests
> > 
> > 1. Currently, the model files(like hive_model.json) are auto generated from 
> > model definitions defined in java(like HiveDataModelGenerator). The patch 
> > files in this case has to be hand coded which is error prone
> > 2. For completeness, readability and debuggability, the type update has to 
> > be done in the corresponding model definitions like HiveDataModelGenerator. 
> > So, same data will be in two places and the model definitions and the patch 
> > files can go out of sync
> > 3. Since the model definitions(like HiveDataModelGenerator) will be updated 
> > anyways, if we modify ReservedTypesRegistrar to do type update instead of 
> > type create, the type updates will automatically be taken care with the 
> > same model json. So, model update patches are not necessary then
> > 3. This jira doesn't implement type versioning - doesn't have support for 
> > storing multiple versions of the type. But it maintains the version of the 
> > latest type definition which I think is useful for debugging, to know the 
> > version of type that the server knows. Can we maintain this info in 
> > HiveDataModelGenerator itself, and hence will be part of hive_model.json
> 
> Sarath Kumar Subramanian wrote:
> 1. patch json genarator can be addressed in a separate jiira
> 2. this is only for upgrade scenarios
> 3. lets address this in a separate jiira.
> 4. we dont want to maintain multiple versions of the same type in the 
> typesystem, will confuse users on which version to use to create entities.
> 
> Shwetha GS wrote:
> What benefit does this patch framework add when the type update can be 
> done with the existing functionality itself (Type versioning is useful and 
> can be done without the patch framework). The patch framework adds more 
> overhead, and I don't see the necessity

I agree - I think this capability would be better placed in separate tooling 
that uses the Atlas REST APIs.


- David


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51939/#review149811
---


On Sept. 22, 2016, 7:34 a.m., Sarath Kumar Subramanian wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51939/
> ---
> 
> (Updated Sept. 22, 2016, 7:34 a.m.)
> 
> 
> Review request for atlas, Madhan Neethiraj, Shwetha GS, and Suma Shivaprasad.
> 
> 
> Bugs: ATLAS-1174
> https://issues.apache.org/jira/browse/ATLAS-1174
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> 1. Introduce "version" attribute to all types in the type-system, this helps 
> to track changes made to the default types (hive, sqoop, falcon and storm 
> types) and user created types. If version is not mentioned during creation of 
> a type, default version "1.0" is assigned (optional attribute).
> 2. Using the version attributed for types, introduce a patch framework for 
> type system. This framework applies patches to a type using its version 
> number and can be used during upgrade - add new attributes to an existing 
> types and it will be run during atlas startup.
> The sequence of steps:
> a. During atlas startup, check $ATLAS_HOME/models/patches directory for any 
> available patch files (json files). If there any patch files handle them.
> b. Sample patch json file looks like:
> {
> "patches": [
> { 
> "action": "ADD_ATTRIBUTE",
> "typeName": "hive_column",
> "applyToVersion": "1.0",
> "updateToVersion": "2.0",
> "actionParams": [
> { "name": "position", "dataTypeName": "int", "multiplicity": "optional", 
> "isComposite": false, "isUnique": false, "isIndexable": false, 
> "reverseAttributeName": null }
> ]
> } ]
> }
> c. The framework updates the type in "typeName" for the matching version 
> number and after applying the patch, update the version to the one mentioned 
> in "updateToVersion"
> d. The json file can have more than one action (array of actions).
> e. There can be multiple patch json files in the directory and are applied in 
> the sort order of the filename. eg:
> 001-hive_column_add_position.json
> 002-hive_column_add_anotherattribute.json
> 
> 
> Diffs
> -
> 
>   common/src/main/java/org/apache/atlas/AtlasConstants.java 17ffbd7 
>   common/src/main/java/org/apache/atlas/repository/Constants.java d7f9c89 
>   
> repository/src/main/java/org/apache/atlas/repository/typestore/GraphBackedTypeStore.java
>  a94d157 
>   
> repository/src/main/java/org/apache/atlas/services/AtlasTypeAttributePatch.java
>  PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/services/AtlasTypePatch.java 
> PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java
>  6a937f4 

[jira] [Updated] (ATLAS-247) Hive Column level lineage

[Vimal Sharma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vimal Sharma updated ATLAS-247:
---
Attachment: ATLAS-247-v5.patch

> Hive Column level lineage
> -
>
> Key: ATLAS-247
> URL: https://issues.apache.org/jira/browse/ATLAS-247
> Project: Atlas
>  Issue Type: New Feature
>Affects Versions: 0.5-incubating
>Reporter: Herman Yu
>Assignee: Harish Butani
> Fix For: 0.8-incubating
>
> Attachments: ATLAS-247-v4.patch, ATLAS-247-v5.patch, 
> ATLAS-247.2.patch, ATLAS-247.patch
>
>
> hive_column is not inherited from DataSet, thus can't be using hive_process 
> to track column level lineages
> Is there specific reason that hive_column is not inheriting from Data Set? 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)