[jira] [Comment Edited] (ATLAS-1175) Type updates should allow removal of optional attributes

[Sarath Subramanian (JIRA)]

[ 
https://issues.apache.org/jira/browse/ATLAS-1175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15508049#comment-15508049
 ] 

Sarath Subramanian edited comment on ATLAS-1175 at 9/20/16 10:55 PM:
-

updating a type by removing optional attributes doesn't affect entities based 
on the old type, the schema of the type is changed and the value of the removed 
attribute still exists in titan db for older entities, but its not surfaced in 
REST API and UI. 

when you create a new entity and include value for removed attribute, it will 
be ignored by atlas and the attribute value will not be stored in titan.


was (Author: sarath.ku...@gmail.com):
updating a type by removing optional attributes doesn't affect entities based 
on the old type, the schema of the type is changed and the value of the removed 
attribute still exists in titan db for older entities, but its not surfaced in 
REST API and UI.

> Type updates should allow removal of optional attributes
> 
>
> Key: ATLAS-1175
> URL: https://issues.apache.org/jira/browse/ATLAS-1175
> Project: Atlas
>  Issue Type: Bug
>Affects Versions: 0.7-incubating, 0.8-incubating
>Reporter: Suma Shivaprasad
>Assignee: Sarath Subramanian
>
> Currently optional attributes are not allowed to be removed from a given 
> type. This should be allowed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 51939: Framework to apply updates to types in the type-system

[Suma Shivaprasad]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51939/#review149727
---




typesystem/src/main/java/org/apache/atlas/typesystem/types/TypeSystem.java 
(line 314)


We would need to audit types as well to 
know what was the original type definition that was updated and the patch 
that was applied.


- Suma Shivaprasad


On Sept. 19, 2016, 6:10 p.m., Sarath Kumar Subramanian wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51939/
> ---
> 
> (Updated Sept. 19, 2016, 6:10 p.m.)
> 
> 
> Review request for atlas, Madhan Neethiraj, Shwetha GS, and Suma Shivaprasad.
> 
> 
> Bugs: ATLAS-1174
> https://issues.apache.org/jira/browse/ATLAS-1174
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> 1. Introduce "version" attribute to all types in the type-system, this helps 
> to track changes made to the default types (hive, sqoop, falcon and storm 
> types) and user created types. If version is not mentioned during creation of 
> a type, default version "1.0" is assigned (optional attribute).
> 2. Using the version attributed for types, introduce a patch framework for 
> type system. This framework applies patches to a type using its version 
> number and can be used during upgrade - add new attributes to an existing 
> types and it will be run during atlas startup.
> The sequence of steps:
> a. During atlas startup, check $ATLAS_HOME/models/patches directory for any 
> available patch files (json files). If there any patch files handle them.
> b. Sample patch json file looks like:
> {
> "patches": [
> { 
> "action": "ADD_ATTRIBUTE",
> "typeName": "hive_column",
> "applyToVersion": "1.0",
> "updateToVersion": "2.0",
> "actionParams": [
> { "name": "position", "dataTypeName": "int", "multiplicity": "optional", 
> "isComposite": false, "isUnique": false, "isIndexable": false, 
> "reverseAttributeName": null }
> ]
> } ]
> }
> c. The framework updates the type in "typeName" for the matching version 
> number and after applying the patch, update the version to the one mentioned 
> in "updateToVersion"
> d. The json file can have more than one action (array of actions).
> e. There can be multiple patch json files in the directory and are applied in 
> the sort order of the filename. eg:
> 001-hive_column_add_position.json
> 002-hive_column_add_anotherattribute.json
> 
> 
> Diffs
> -
> 
>   common/src/main/java/org/apache/atlas/repository/Constants.java d7f9c89 
>   
> repository/src/main/java/org/apache/atlas/repository/typestore/GraphBackedTypeStore.java
>  a94d157 
>   repository/src/main/java/org/apache/atlas/services/AddAttributePatch.java 
> PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/services/AtlasTypePatch.java 
> PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java
>  6a937f4 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/AbstractDataType.java
>  fad091d 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/ClassType.java 
> c56987a 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/EnumType.java 
> bdd0a13 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/EnumTypeDefinition.java
>  6340615 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/HierarchicalType.java
>  7224699 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/HierarchicalTypeDefinition.java
>  9a299f0 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/IDataType.java 
> 85ddee7 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/StructType.java 
> 6f40c1d 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/StructTypeDefinition.java
>  f1ce1b7 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/TraitType.java 
> f23bf5b 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/TypeSystem.java 
> 70ba89b 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/utils/TypesUtil.java
>  ef8448d 
>   
> typesystem/src/main/scala/org/apache/atlas/typesystem/json/TypesSerialization.scala
>  5618938 
> 
> Diff: https://reviews.apache.org/r/51939/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sarath Kumar Subramanian
> 
>

Re: Review Request 51896: ATLAS-1171: structured, high-level APIs

[Jeff Hagelberg]

> On Sept. 19, 2016, 11:13 p.m., Jeff Hagelberg wrote:
> > common/src/main/java/org/apache/atlas/model/AtlasAttributeDef.java, line 61
> > 
> >
> > Why is this needed?  It seems like if 'valuesMaxCount' > 1, it is 
> > automatically multi-valued.
> 
> Madhan Neethiraj wrote:
> Please see my earlier commit. I think the use of explicit attributes like 
> isMultiValued make it easier to read/understand.

I think that makes sense, but what happens if someone sets isMultiValued=true 
and valuesMaxCount=0.  Would a configuration like that be allowed?  It seems 
like the parameters should be defined in a way that does not allow invalid 
configurations to be specified.


- Jeff


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51896/#review149121
---


On Sept. 20, 2016, 8:06 a.m., Madhan Neethiraj wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51896/
> ---
> 
> (Updated Sept. 20, 2016, 8:06 a.m.)
> 
> 
> Review request for atlas.
> 
> 
> Bugs: ATLAS-1171
> https://issues.apache.org/jira/browse/ATLAS-1171
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> first-cut API for review
> 
> 
> Diffs
> -
> 
>   common/pom.xml e3b6465 
>   common/src/main/java/org/apache/atlas/api/AtlasApiEntities.java 
> PRE-CREATION 
>   common/src/main/java/org/apache/atlas/api/AtlasApiTypes.java PRE-CREATION 
>   common/src/main/java/org/apache/atlas/api/PList.java PRE-CREATION 
>   common/src/main/java/org/apache/atlas/api/SearchFilter.java PRE-CREATION 
>   common/src/main/java/org/apache/atlas/model/instances/AtlasEntity.java 
> PRE-CREATION 
>   common/src/main/java/org/apache/atlas/model/instances/AtlasObjectId.java 
> PRE-CREATION 
>   common/src/main/java/org/apache/atlas/model/instances/AtlasProcess.java 
> PRE-CREATION 
>   common/src/main/java/org/apache/atlas/model/instances/AtlasStruct.java 
> PRE-CREATION 
>   common/src/main/java/org/apache/atlas/model/instances/AtlasTrait.java 
> PRE-CREATION 
>   common/src/main/java/org/apache/atlas/model/types/AtlasBaseTypeDef.java 
> PRE-CREATION 
>   common/src/main/java/org/apache/atlas/model/types/AtlasEntityDef.java 
> PRE-CREATION 
>   common/src/main/java/org/apache/atlas/model/types/AtlasEnumDef.java 
> PRE-CREATION 
>   common/src/main/java/org/apache/atlas/model/types/AtlasStructDef.java 
> PRE-CREATION 
>   common/src/main/java/org/apache/atlas/model/types/AtlasTraitDef.java 
> PRE-CREATION 
>   common/src/main/java/org/apache/atlas/typesystem/AtlasArrayType.java 
> PRE-CREATION 
>   common/src/main/java/org/apache/atlas/typesystem/AtlasBuiltInTypes.java 
> PRE-CREATION 
>   common/src/main/java/org/apache/atlas/typesystem/AtlasDataType.java 
> PRE-CREATION 
>   common/src/main/java/org/apache/atlas/typesystem/AtlasEntityType.java 
> PRE-CREATION 
>   common/src/main/java/org/apache/atlas/typesystem/AtlasEnumType.java 
> PRE-CREATION 
>   common/src/main/java/org/apache/atlas/typesystem/AtlasMapType.java 
> PRE-CREATION 
>   common/src/main/java/org/apache/atlas/typesystem/AtlasStructType.java 
> PRE-CREATION 
>   common/src/main/java/org/apache/atlas/typesystem/AtlasTraitType.java 
> PRE-CREATION 
>   common/src/main/java/org/apache/atlas/typesystem/AtlasTypeRegistry.java 
> PRE-CREATION 
>   common/src/test/java/org/apache/atlas/model/ModelTestUtil.java PRE-CREATION 
>   common/src/test/java/org/apache/atlas/model/instances/TestAtlasEntity.java 
> PRE-CREATION 
>   common/src/test/java/org/apache/atlas/model/types/TestAtlasEntityDef.java 
> PRE-CREATION 
>   common/src/test/java/org/apache/atlas/model/types/TestAtlasEnumDef.java 
> PRE-CREATION 
>   common/src/test/java/org/apache/atlas/model/types/TestAtlasStructDef.java 
> PRE-CREATION 
>   common/src/test/java/org/apache/atlas/model/types/TestAtlasTraitDef.java 
> PRE-CREATION 
>   pom.xml ac5b042 
>   webapp/src/main/java/org/apache/atlas/web/rest/TypesREST.java PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/51896/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Madhan Neethiraj
> 
>

Re: Review Request 51939: Framework to apply updates to types in the type-system

[Sarath Kumar Subramanian]

> On Sept. 20, 2016, 4:15 a.m., Suma Shivaprasad wrote:
> > repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java,
> >  line 191
> > 
> >
> > it would make more sense to sort by date instead of file name?

Suma, if the patch files are sorted my name (each patch file starting with 
001-xxx, 002-yyy), we can be sure the patch will be applied in order. Sorting 
by date can mess up the order if the files are not laid in order.


- Sarath Kumar


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51939/#review149608
---


On Sept. 19, 2016, 6:10 p.m., Sarath Kumar Subramanian wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51939/
> ---
> 
> (Updated Sept. 19, 2016, 6:10 p.m.)
> 
> 
> Review request for atlas, Madhan Neethiraj, Shwetha GS, and Suma Shivaprasad.
> 
> 
> Bugs: ATLAS-1174
> https://issues.apache.org/jira/browse/ATLAS-1174
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> 1. Introduce "version" attribute to all types in the type-system, this helps 
> to track changes made to the default types (hive, sqoop, falcon and storm 
> types) and user created types. If version is not mentioned during creation of 
> a type, default version "1.0" is assigned (optional attribute).
> 2. Using the version attributed for types, introduce a patch framework for 
> type system. This framework applies patches to a type using its version 
> number and can be used during upgrade - add new attributes to an existing 
> types and it will be run during atlas startup.
> The sequence of steps:
> a. During atlas startup, check $ATLAS_HOME/models/patches directory for any 
> available patch files (json files). If there any patch files handle them.
> b. Sample patch json file looks like:
> {
> "patches": [
> { 
> "action": "ADD_ATTRIBUTE",
> "typeName": "hive_column",
> "applyToVersion": "1.0",
> "updateToVersion": "2.0",
> "actionParams": [
> { "name": "position", "dataTypeName": "int", "multiplicity": "optional", 
> "isComposite": false, "isUnique": false, "isIndexable": false, 
> "reverseAttributeName": null }
> ]
> } ]
> }
> c. The framework updates the type in "typeName" for the matching version 
> number and after applying the patch, update the version to the one mentioned 
> in "updateToVersion"
> d. The json file can have more than one action (array of actions).
> e. There can be multiple patch json files in the directory and are applied in 
> the sort order of the filename. eg:
> 001-hive_column_add_position.json
> 002-hive_column_add_anotherattribute.json
> 
> 
> Diffs
> -
> 
>   common/src/main/java/org/apache/atlas/repository/Constants.java d7f9c89 
>   
> repository/src/main/java/org/apache/atlas/repository/typestore/GraphBackedTypeStore.java
>  a94d157 
>   repository/src/main/java/org/apache/atlas/services/AddAttributePatch.java 
> PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/services/AtlasTypePatch.java 
> PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java
>  6a937f4 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/AbstractDataType.java
>  fad091d 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/ClassType.java 
> c56987a 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/EnumType.java 
> bdd0a13 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/EnumTypeDefinition.java
>  6340615 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/HierarchicalType.java
>  7224699 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/HierarchicalTypeDefinition.java
>  9a299f0 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/IDataType.java 
> 85ddee7 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/StructType.java 
> 6f40c1d 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/StructTypeDefinition.java
>  f1ce1b7 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/TraitType.java 
> f23bf5b 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/TypeSystem.java 
> 70ba89b 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/utils/TypesUtil.java
>  ef8448d 
>   
> typesystem/src/main/scala/org/apache/atlas/typesystem/json/TypesSerialization.scala
>  5618938 
> 
> Diff: https://reviews.apache.org/r/51939/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sarath Kumar Subramanian
> 
>

Re: Review Request 51939: Framework to apply updates to types in the type-system

[Suma Shivaprasad]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51939/#review149703
---




typesystem/src/main/java/org/apache/atlas/typesystem/types/HierarchicalType.java
 (line 62)


DEFAULT_TYPE_VERSION seems to be defined in multiple places. should be 
defined at a single place  and could go to AtlasConstants


- Suma Shivaprasad


On Sept. 19, 2016, 6:10 p.m., Sarath Kumar Subramanian wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51939/
> ---
> 
> (Updated Sept. 19, 2016, 6:10 p.m.)
> 
> 
> Review request for atlas, Madhan Neethiraj, Shwetha GS, and Suma Shivaprasad.
> 
> 
> Bugs: ATLAS-1174
> https://issues.apache.org/jira/browse/ATLAS-1174
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> 1. Introduce "version" attribute to all types in the type-system, this helps 
> to track changes made to the default types (hive, sqoop, falcon and storm 
> types) and user created types. If version is not mentioned during creation of 
> a type, default version "1.0" is assigned (optional attribute).
> 2. Using the version attributed for types, introduce a patch framework for 
> type system. This framework applies patches to a type using its version 
> number and can be used during upgrade - add new attributes to an existing 
> types and it will be run during atlas startup.
> The sequence of steps:
> a. During atlas startup, check $ATLAS_HOME/models/patches directory for any 
> available patch files (json files). If there any patch files handle them.
> b. Sample patch json file looks like:
> {
> "patches": [
> { 
> "action": "ADD_ATTRIBUTE",
> "typeName": "hive_column",
> "applyToVersion": "1.0",
> "updateToVersion": "2.0",
> "actionParams": [
> { "name": "position", "dataTypeName": "int", "multiplicity": "optional", 
> "isComposite": false, "isUnique": false, "isIndexable": false, 
> "reverseAttributeName": null }
> ]
> } ]
> }
> c. The framework updates the type in "typeName" for the matching version 
> number and after applying the patch, update the version to the one mentioned 
> in "updateToVersion"
> d. The json file can have more than one action (array of actions).
> e. There can be multiple patch json files in the directory and are applied in 
> the sort order of the filename. eg:
> 001-hive_column_add_position.json
> 002-hive_column_add_anotherattribute.json
> 
> 
> Diffs
> -
> 
>   common/src/main/java/org/apache/atlas/repository/Constants.java d7f9c89 
>   
> repository/src/main/java/org/apache/atlas/repository/typestore/GraphBackedTypeStore.java
>  a94d157 
>   repository/src/main/java/org/apache/atlas/services/AddAttributePatch.java 
> PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/services/AtlasTypePatch.java 
> PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java
>  6a937f4 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/AbstractDataType.java
>  fad091d 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/ClassType.java 
> c56987a 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/EnumType.java 
> bdd0a13 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/EnumTypeDefinition.java
>  6340615 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/HierarchicalType.java
>  7224699 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/HierarchicalTypeDefinition.java
>  9a299f0 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/IDataType.java 
> 85ddee7 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/StructType.java 
> 6f40c1d 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/StructTypeDefinition.java
>  f1ce1b7 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/TraitType.java 
> f23bf5b 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/TypeSystem.java 
> 70ba89b 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/utils/TypesUtil.java
>  ef8448d 
>   
> typesystem/src/main/scala/org/apache/atlas/typesystem/json/TypesSerialization.scala
>  5618938 
> 
> Diff: https://reviews.apache.org/r/51939/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sarath Kumar Subramanian
> 
>

Re: Review Request 51939: Framework to apply updates to types in the type-system

[Suma Shivaprasad]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51939/#review149702
---




repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java 
(line 234)


we should log how many patch files were suceesfully applied out of the 
total count


- Suma Shivaprasad


On Sept. 19, 2016, 6:10 p.m., Sarath Kumar Subramanian wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51939/
> ---
> 
> (Updated Sept. 19, 2016, 6:10 p.m.)
> 
> 
> Review request for atlas, Madhan Neethiraj, Shwetha GS, and Suma Shivaprasad.
> 
> 
> Bugs: ATLAS-1174
> https://issues.apache.org/jira/browse/ATLAS-1174
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> 1. Introduce "version" attribute to all types in the type-system, this helps 
> to track changes made to the default types (hive, sqoop, falcon and storm 
> types) and user created types. If version is not mentioned during creation of 
> a type, default version "1.0" is assigned (optional attribute).
> 2. Using the version attributed for types, introduce a patch framework for 
> type system. This framework applies patches to a type using its version 
> number and can be used during upgrade - add new attributes to an existing 
> types and it will be run during atlas startup.
> The sequence of steps:
> a. During atlas startup, check $ATLAS_HOME/models/patches directory for any 
> available patch files (json files). If there any patch files handle them.
> b. Sample patch json file looks like:
> {
> "patches": [
> { 
> "action": "ADD_ATTRIBUTE",
> "typeName": "hive_column",
> "applyToVersion": "1.0",
> "updateToVersion": "2.0",
> "actionParams": [
> { "name": "position", "dataTypeName": "int", "multiplicity": "optional", 
> "isComposite": false, "isUnique": false, "isIndexable": false, 
> "reverseAttributeName": null }
> ]
> } ]
> }
> c. The framework updates the type in "typeName" for the matching version 
> number and after applying the patch, update the version to the one mentioned 
> in "updateToVersion"
> d. The json file can have more than one action (array of actions).
> e. There can be multiple patch json files in the directory and are applied in 
> the sort order of the filename. eg:
> 001-hive_column_add_position.json
> 002-hive_column_add_anotherattribute.json
> 
> 
> Diffs
> -
> 
>   common/src/main/java/org/apache/atlas/repository/Constants.java d7f9c89 
>   
> repository/src/main/java/org/apache/atlas/repository/typestore/GraphBackedTypeStore.java
>  a94d157 
>   repository/src/main/java/org/apache/atlas/services/AddAttributePatch.java 
> PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/services/AtlasTypePatch.java 
> PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java
>  6a937f4 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/AbstractDataType.java
>  fad091d 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/ClassType.java 
> c56987a 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/EnumType.java 
> bdd0a13 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/EnumTypeDefinition.java
>  6340615 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/HierarchicalType.java
>  7224699 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/HierarchicalTypeDefinition.java
>  9a299f0 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/IDataType.java 
> 85ddee7 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/StructType.java 
> 6f40c1d 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/StructTypeDefinition.java
>  f1ce1b7 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/TraitType.java 
> f23bf5b 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/TypeSystem.java 
> 70ba89b 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/utils/TypesUtil.java
>  ef8448d 
>   
> typesystem/src/main/scala/org/apache/atlas/typesystem/json/TypesSerialization.scala
>  5618938 
> 
> Diff: https://reviews.apache.org/r/51939/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sarath Kumar Subramanian
> 
>

Re: Review Request 51939: Framework to apply updates to types in the type-system

[Suma Shivaprasad]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51939/#review149700
---




repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java 
(line 231)


this should be ERROR level


- Suma Shivaprasad


On Sept. 19, 2016, 6:10 p.m., Sarath Kumar Subramanian wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51939/
> ---
> 
> (Updated Sept. 19, 2016, 6:10 p.m.)
> 
> 
> Review request for atlas, Madhan Neethiraj, Shwetha GS, and Suma Shivaprasad.
> 
> 
> Bugs: ATLAS-1174
> https://issues.apache.org/jira/browse/ATLAS-1174
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> 1. Introduce "version" attribute to all types in the type-system, this helps 
> to track changes made to the default types (hive, sqoop, falcon and storm 
> types) and user created types. If version is not mentioned during creation of 
> a type, default version "1.0" is assigned (optional attribute).
> 2. Using the version attributed for types, introduce a patch framework for 
> type system. This framework applies patches to a type using its version 
> number and can be used during upgrade - add new attributes to an existing 
> types and it will be run during atlas startup.
> The sequence of steps:
> a. During atlas startup, check $ATLAS_HOME/models/patches directory for any 
> available patch files (json files). If there any patch files handle them.
> b. Sample patch json file looks like:
> {
> "patches": [
> { 
> "action": "ADD_ATTRIBUTE",
> "typeName": "hive_column",
> "applyToVersion": "1.0",
> "updateToVersion": "2.0",
> "actionParams": [
> { "name": "position", "dataTypeName": "int", "multiplicity": "optional", 
> "isComposite": false, "isUnique": false, "isIndexable": false, 
> "reverseAttributeName": null }
> ]
> } ]
> }
> c. The framework updates the type in "typeName" for the matching version 
> number and after applying the patch, update the version to the one mentioned 
> in "updateToVersion"
> d. The json file can have more than one action (array of actions).
> e. There can be multiple patch json files in the directory and are applied in 
> the sort order of the filename. eg:
> 001-hive_column_add_position.json
> 002-hive_column_add_anotherattribute.json
> 
> 
> Diffs
> -
> 
>   common/src/main/java/org/apache/atlas/repository/Constants.java d7f9c89 
>   
> repository/src/main/java/org/apache/atlas/repository/typestore/GraphBackedTypeStore.java
>  a94d157 
>   repository/src/main/java/org/apache/atlas/services/AddAttributePatch.java 
> PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/services/AtlasTypePatch.java 
> PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java
>  6a937f4 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/AbstractDataType.java
>  fad091d 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/ClassType.java 
> c56987a 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/EnumType.java 
> bdd0a13 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/EnumTypeDefinition.java
>  6340615 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/HierarchicalType.java
>  7224699 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/HierarchicalTypeDefinition.java
>  9a299f0 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/IDataType.java 
> 85ddee7 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/StructType.java 
> 6f40c1d 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/StructTypeDefinition.java
>  f1ce1b7 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/TraitType.java 
> f23bf5b 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/TypeSystem.java 
> 70ba89b 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/utils/TypesUtil.java
>  ef8448d 
>   
> typesystem/src/main/scala/org/apache/atlas/typesystem/json/TypesSerialization.scala
>  5618938 
> 
> Diff: https://reviews.apache.org/r/51939/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sarath Kumar Subramanian
> 
>

Re: Review Request 51939: Framework to apply updates to types in the type-system

[Sarath Kumar Subramanian]

> On Sept. 20, 2016, 1:11 p.m., David Radley wrote:
> > It looks like this is a way of updating the versions of types using a local 
> > file on the Atlas server. I think we should be SaaS and expose this sort of 
> > functionality primarily as a REST API, otherwise this could become a 
> > barrier to integration.

David, the goal of this framework is to apply patches during upgrade scenarios 
- when user has changed the shape of a type (added new attributes, new 
supertypes) or to update an existing type. The patch files will be checked and 
applied during atlas startup, this will avoid dealing with multiple patch 
scripts for upgrading multiple types. Currrently a type can be updated using 
REST (PUT - /api/atlas/types) by including the type definition in request 
payload. We can add only optional attributes, so instances depending on old 
type will not be broken.

I can add a dedicated REST layer to this once we have ATLAS-1171 ready.


- Sarath Kumar


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51939/#review149652
---


On Sept. 19, 2016, 6:10 p.m., Sarath Kumar Subramanian wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51939/
> ---
> 
> (Updated Sept. 19, 2016, 6:10 p.m.)
> 
> 
> Review request for atlas, Madhan Neethiraj, Shwetha GS, and Suma Shivaprasad.
> 
> 
> Bugs: ATLAS-1174
> https://issues.apache.org/jira/browse/ATLAS-1174
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> 1. Introduce "version" attribute to all types in the type-system, this helps 
> to track changes made to the default types (hive, sqoop, falcon and storm 
> types) and user created types. If version is not mentioned during creation of 
> a type, default version "1.0" is assigned (optional attribute).
> 2. Using the version attributed for types, introduce a patch framework for 
> type system. This framework applies patches to a type using its version 
> number and can be used during upgrade - add new attributes to an existing 
> types and it will be run during atlas startup.
> The sequence of steps:
> a. During atlas startup, check $ATLAS_HOME/models/patches directory for any 
> available patch files (json files). If there any patch files handle them.
> b. Sample patch json file looks like:
> {
> "patches": [
> { 
> "action": "ADD_ATTRIBUTE",
> "typeName": "hive_column",
> "applyToVersion": "1.0",
> "updateToVersion": "2.0",
> "actionParams": [
> { "name": "position", "dataTypeName": "int", "multiplicity": "optional", 
> "isComposite": false, "isUnique": false, "isIndexable": false, 
> "reverseAttributeName": null }
> ]
> } ]
> }
> c. The framework updates the type in "typeName" for the matching version 
> number and after applying the patch, update the version to the one mentioned 
> in "updateToVersion"
> d. The json file can have more than one action (array of actions).
> e. There can be multiple patch json files in the directory and are applied in 
> the sort order of the filename. eg:
> 001-hive_column_add_position.json
> 002-hive_column_add_anotherattribute.json
> 
> 
> Diffs
> -
> 
>   common/src/main/java/org/apache/atlas/repository/Constants.java d7f9c89 
>   
> repository/src/main/java/org/apache/atlas/repository/typestore/GraphBackedTypeStore.java
>  a94d157 
>   repository/src/main/java/org/apache/atlas/services/AddAttributePatch.java 
> PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/services/AtlasTypePatch.java 
> PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java
>  6a937f4 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/AbstractDataType.java
>  fad091d 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/ClassType.java 
> c56987a 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/EnumType.java 
> bdd0a13 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/EnumTypeDefinition.java
>  6340615 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/HierarchicalType.java
>  7224699 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/HierarchicalTypeDefinition.java
>  9a299f0 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/IDataType.java 
> 85ddee7 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/StructType.java 
> 6f40c1d 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/StructTypeDefinition.java
>  f1ce1b7 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/TraitType.java 
> f23bf5b 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/TypeSystem.java 
> 70ba89b 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/utils/TypesUtil.java
>  

[jira] [Comment Edited] (ATLAS-1145) Storm hook with kafka topic and hive bolt picks wrong hive table name

[Ayub Khan (JIRA)]

[ 
https://issues.apache.org/jira/browse/ATLAS-1145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15507236#comment-15507236
 ] 

Ayub Khan edited comment on ATLAS-1145 at 9/20/16 5:41 PM:
---

Sure, I can verify this as part of my testing and update this thread with my 
findings.


was (Author: ayubkhan):
Sure, I can verify this as part of my testing and update this thread.

> Storm hook with kafka topic and hive bolt picks wrong hive table name 
> --
>
> Key: ATLAS-1145
> URL: https://issues.apache.org/jira/browse/ATLAS-1145
> Project: Atlas
>  Issue Type: Bug
>Reporter: Shwetha G S
>Assignee: Sarath Subramanian
>
> Storm hook with kafka topic and hive bolt picks kafka topic name as hive 
> table name



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (ATLAS-1145) Storm hook with kafka topic and hive bolt picks wrong hive table name

[Ayub Khan (JIRA)]

[ 
https://issues.apache.org/jira/browse/ATLAS-1145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15507236#comment-15507236
 ] 

Ayub Khan commented on ATLAS-1145:
--

Sure, I can verify this as part of my testing and update this thread.

> Storm hook with kafka topic and hive bolt picks wrong hive table name 
> --
>
> Key: ATLAS-1145
> URL: https://issues.apache.org/jira/browse/ATLAS-1145
> Project: Atlas
>  Issue Type: Bug
>Reporter: Shwetha G S
>Assignee: Sarath Subramanian
>
> Storm hook with kafka topic and hive bolt picks kafka topic name as hive 
> table name



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (ATLAS-1145) Storm hook with kafka topic and hive bolt picks wrong hive table name

[Sarath Subramanian (JIRA)]

[ 
https://issues.apache.org/jira/browse/ATLAS-1145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15507157#comment-15507157
 ] 

Sarath Subramanian commented on ATLAS-1145:
---

[~ayubkhan] can you confirm this, since you are writing an automation script to 
create these storm topologies and verify them in atlas.

> Storm hook with kafka topic and hive bolt picks wrong hive table name 
> --
>
> Key: ATLAS-1145
> URL: https://issues.apache.org/jira/browse/ATLAS-1145
> Project: Atlas
>  Issue Type: Bug
>Reporter: Shwetha G S
>Assignee: Sarath Subramanian
>
> Storm hook with kafka topic and hive bolt picks kafka topic name as hive 
> table name



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (ATLAS-1145) Storm hook with kafka topic and hive bolt picks wrong hive table name

[Sarath Subramanian (JIRA)]

[ 
https://issues.apache.org/jira/browse/ATLAS-1145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15507148#comment-15507148
 ] 

Sarath Subramanian edited comment on ATLAS-1145 at 9/20/16 5:12 PM:


Thanks for the samples [~shwethags], [~yhemanth] and [~ayubkhan]. I was able to 
create a storm topology with kafka spout/hive bolt using the samples from 
Ayub's link. I'm not able to reproduce this issue. Below is the kafka topic 
entity definition created by the storm topology, the kafka topic entity in 
atlas gets assigned the right topic name and not the hive table name. 

{
  "requestId": "qtp1448525331-188 - f7904223-bac3-489b-b593-ffbb3100debd",
  "definition": {
"jsonClass": 
"org.apache.atlas.typesystem.json.InstanceSerialization$_Reference",
"id": {
  "jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Id",
  "id": "5669f2ba-8397-44c5-bf0d-0544533979a4",
  "version": 0,
  "typeName": "kafka_topic",
  "state": "ACTIVE"
},
"typeName": "kafka_topic",
"values": {
  "name": "kafka_topic_7",
  "description": null,
  "uri": "localhost:2181",
  "qualifiedName": "kafka_topic_7@ErieCluster",
  "topic": "kafka_topic_7",
  "owner": "storm"
},
"traitNames": [],
"traits": {}
  }
}


was (Author: sarath.ku...@gmail.com):
Thanks for the samples [~shwethags], [~yhemanth] and [~ayubkhan]. I was able to 
create a storm topology with kafka spout/hive bolt using the samples from 
Ayub's link. I'm not able to reproduce this issue. Attaching the entity 
definition from REST, the kafka topic entity in atlas gets assigned the right 
topic name and not the hive table name. Attaching the kafka topic entity 
definition created by the storm topology.

{
  "requestId": "qtp1448525331-188 - f7904223-bac3-489b-b593-ffbb3100debd",
  "definition": {
"jsonClass": 
"org.apache.atlas.typesystem.json.InstanceSerialization$_Reference",
"id": {
  "jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Id",
  "id": "5669f2ba-8397-44c5-bf0d-0544533979a4",
  "version": 0,
  "typeName": "kafka_topic",
  "state": "ACTIVE"
},
"typeName": "kafka_topic",
"values": {
  "name": "kafka_topic_7",
  "description": null,
  "uri": "localhost:2181",
  "qualifiedName": "kafka_topic_7@ErieCluster",
  "topic": "kafka_topic_7",
  "owner": "storm"
},
"traitNames": [],
"traits": {}
  }
}

> Storm hook with kafka topic and hive bolt picks wrong hive table name 
> --
>
> Key: ATLAS-1145
> URL: https://issues.apache.org/jira/browse/ATLAS-1145
> Project: Atlas
>  Issue Type: Bug
>Reporter: Shwetha G S
>Assignee: Sarath Subramanian
>
> Storm hook with kafka topic and hive bolt picks kafka topic name as hive 
> table name



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (ATLAS-1145) Storm hook with kafka topic and hive bolt picks wrong hive table name

[Sarath Subramanian (JIRA)]

[ 
https://issues.apache.org/jira/browse/ATLAS-1145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15507148#comment-15507148
 ] 

Sarath Subramanian commented on ATLAS-1145:
---

Thanks for the samples [~shwethags], [~yhemanth] and [~ayubkhan]. I was able to 
create a storm topology with kafka spout/hive bolt using the samples from 
Ayub's link. I'm not able to reproduce this issue. Attaching the entity 
definition from REST, the kafka topic entity in atlas gets assigned the right 
topic name and not the hive table name. Attaching the kafka topic entity 
definition created by the storm topology.

{
  "requestId": "qtp1448525331-188 - f7904223-bac3-489b-b593-ffbb3100debd",
  "definition": {
"jsonClass": 
"org.apache.atlas.typesystem.json.InstanceSerialization$_Reference",
"id": {
  "jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Id",
  "id": "5669f2ba-8397-44c5-bf0d-0544533979a4",
  "version": 0,
  "typeName": "kafka_topic",
  "state": "ACTIVE"
},
"typeName": "kafka_topic",
"values": {
  "name": "kafka_topic_7",
  "description": null,
  "uri": "localhost:2181",
  "qualifiedName": "kafka_topic_7@ErieCluster",
  "topic": "kafka_topic_7",
  "owner": "storm"
},
"traitNames": [],
"traits": {}
  }
}

> Storm hook with kafka topic and hive bolt picks wrong hive table name 
> --
>
> Key: ATLAS-1145
> URL: https://issues.apache.org/jira/browse/ATLAS-1145
> Project: Atlas
>  Issue Type: Bug
>Reporter: Shwetha G S
>Assignee: Sarath Subramanian
>
> Storm hook with kafka topic and hive bolt picks kafka topic name as hive 
> table name



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (ATLAS-1182) Hive Column level lineage docs

[Vimal Sharma (JIRA)]

Vimal Sharma created ATLAS-1182:
---

 Summary: Hive Column level lineage docs
 Key: ATLAS-1182
 URL: https://issues.apache.org/jira/browse/ATLAS-1182
 Project: Atlas
  Issue Type: Improvement
Reporter: Vimal Sharma


Add column level lineage details to Atlas documentation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (ATLAS-1181) UI changes for column level lineage

[Vimal Sharma (JIRA)]

Vimal Sharma created ATLAS-1181:
---

 Summary: UI changes for column level lineage
 Key: ATLAS-1181
 URL: https://issues.apache.org/jira/browse/ATLAS-1181
 Project: Atlas
  Issue Type: Improvement
Reporter: Vimal Sharma


While showing column level lineage, qualifiedName attribute of the column 
should be used instead of the name attribute. There are chances that the 
columns with the same name are present in different tables.
The lineage diagram would be ambiguous if such columns are part of a column 
lineage process.

Further, dependencyType attribute can be shown along with the CTAS query in  
lineage process details



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 51939: Framework to apply updates to types in the type-system

[David Radley]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51939/#review149652
---



It looks like this is a way of updating the versions of types using a local 
file on the Atlas server. I think we should be SaaS and expose this sort of 
functionality primarily as a REST API, otherwise this could become a barrier to 
integration.

- David Radley


On Sept. 19, 2016, 6:10 p.m., Sarath Kumar Subramanian wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51939/
> ---
> 
> (Updated Sept. 19, 2016, 6:10 p.m.)
> 
> 
> Review request for atlas, Madhan Neethiraj, Shwetha GS, and Suma Shivaprasad.
> 
> 
> Bugs: ATLAS-1174
> https://issues.apache.org/jira/browse/ATLAS-1174
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> 1. Introduce "version" attribute to all types in the type-system, this helps 
> to track changes made to the default types (hive, sqoop, falcon and storm 
> types) and user created types. If version is not mentioned during creation of 
> a type, default version "1.0" is assigned (optional attribute).
> 2. Using the version attributed for types, introduce a patch framework for 
> type system. This framework applies patches to a type using its version 
> number and can be used during upgrade - add new attributes to an existing 
> types and it will be run during atlas startup.
> The sequence of steps:
> a. During atlas startup, check $ATLAS_HOME/models/patches directory for any 
> available patch files (json files). If there any patch files handle them.
> b. Sample patch json file looks like:
> {
> "patches": [
> { 
> "action": "ADD_ATTRIBUTE",
> "typeName": "hive_column",
> "applyToVersion": "1.0",
> "updateToVersion": "2.0",
> "actionParams": [
> { "name": "position", "dataTypeName": "int", "multiplicity": "optional", 
> "isComposite": false, "isUnique": false, "isIndexable": false, 
> "reverseAttributeName": null }
> ]
> } ]
> }
> c. The framework updates the type in "typeName" for the matching version 
> number and after applying the patch, update the version to the one mentioned 
> in "updateToVersion"
> d. The json file can have more than one action (array of actions).
> e. There can be multiple patch json files in the directory and are applied in 
> the sort order of the filename. eg:
> 001-hive_column_add_position.json
> 002-hive_column_add_anotherattribute.json
> 
> 
> Diffs
> -
> 
>   common/src/main/java/org/apache/atlas/repository/Constants.java d7f9c89 
>   
> repository/src/main/java/org/apache/atlas/repository/typestore/GraphBackedTypeStore.java
>  a94d157 
>   repository/src/main/java/org/apache/atlas/services/AddAttributePatch.java 
> PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/services/AtlasTypePatch.java 
> PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java
>  6a937f4 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/AbstractDataType.java
>  fad091d 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/ClassType.java 
> c56987a 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/EnumType.java 
> bdd0a13 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/EnumTypeDefinition.java
>  6340615 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/HierarchicalType.java
>  7224699 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/HierarchicalTypeDefinition.java
>  9a299f0 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/IDataType.java 
> 85ddee7 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/StructType.java 
> 6f40c1d 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/StructTypeDefinition.java
>  f1ce1b7 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/TraitType.java 
> f23bf5b 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/TypeSystem.java 
> 70ba89b 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/utils/TypesUtil.java
>  ef8448d 
>   
> typesystem/src/main/scala/org/apache/atlas/typesystem/json/TypesSerialization.scala
>  5618938 
> 
> Diff: https://reviews.apache.org/r/51939/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sarath Kumar Subramanian
> 
>

Re: Review Request 52077: Column level lineage in Hive

[Vimal Sharma]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52077/
---

(Updated Sept. 20, 2016, 9:07 a.m.)


Review request for atlas.


Changes
---

Newly added file was not available in the previous patch. Updating the patch


Bugs: ATLAS-247
https://issues.apache.org/jira/browse/ATLAS-247


Repository: atlas


Description
---

After a CTAS query, lineage relationship between source columns and destination 
column can be captured. This information can be used to create a column lineage 
process.


Diffs (updated)
-

  
addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/ColumnLineageUtils.java
 PRE-CREATION 
  addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
a3464a0 
  
addons/hive-bridge/src/main/java/org/apache/atlas/hive/model/HiveDataModelGenerator.java
 45f0bc9 
  
addons/hive-bridge/src/main/java/org/apache/atlas/hive/model/HiveDataTypes.java 
e094cb6 
  addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 
a5838b4 

Diff: https://reviews.apache.org/r/52077/diff/


Testing
---


Thanks,

Vimal Sharma

[jira] [Commented] (ATLAS-247) hive_column leve lineage

[Vimal Sharma (JIRA)]

[ 
https://issues.apache.org/jira/browse/ATLAS-247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15506095#comment-15506095
 ] 

Vimal Sharma commented on ATLAS-247:


Deleted the older patch as it didn't have the newly added file.

> hive_column leve lineage
> 
>
> Key: ATLAS-247
> URL: https://issues.apache.org/jira/browse/ATLAS-247
> Project: Atlas
>  Issue Type: New Feature
>Affects Versions: 0.5-incubating
>Reporter: Herman Yu
>Assignee: Harish Butani
> Fix For: 0.8-incubating
>
> Attachments: ATLAS-247-v4.patch, ATLAS-247.2.patch, ATLAS-247.patch
>
>
> hive_column is not inherited from DataSet, thus can't be using hive_process 
> to track column level lineages
> Is there specific reason that hive_column is not inheriting from Data Set? 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (ATLAS-247) hive_column leve lineage

[Vimal Sharma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vimal Sharma updated ATLAS-247:
---
Attachment: ATLAS-247-v4.patch

> hive_column leve lineage
> 
>
> Key: ATLAS-247
> URL: https://issues.apache.org/jira/browse/ATLAS-247
> Project: Atlas
>  Issue Type: New Feature
>Affects Versions: 0.5-incubating
>Reporter: Herman Yu
>Assignee: Harish Butani
> Fix For: 0.8-incubating
>
> Attachments: ATLAS-247-v4.patch, ATLAS-247.2.patch, ATLAS-247.patch
>
>
> hive_column is not inherited from DataSet, thus can't be using hive_process 
> to track column level lineages
> Is there specific reason that hive_column is not inheriting from Data Set? 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (ATLAS-247) hive_column leve lineage

[Vimal Sharma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vimal Sharma updated ATLAS-247:
---
Attachment: (was: ATLAS-247-v3.patch)

> hive_column leve lineage
> 
>
> Key: ATLAS-247
> URL: https://issues.apache.org/jira/browse/ATLAS-247
> Project: Atlas
>  Issue Type: New Feature
>Affects Versions: 0.5-incubating
>Reporter: Herman Yu
>Assignee: Harish Butani
> Fix For: 0.8-incubating
>
> Attachments: ATLAS-247-v4.patch, ATLAS-247.2.patch, ATLAS-247.patch
>
>
> hive_column is not inherited from DataSet, thus can't be using hive_process 
> to track column level lineages
> Is there specific reason that hive_column is not inheriting from Data Set? 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 51896: ATLAS-1171: structured, high-level APIs

[Madhan Neethiraj]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51896/
---

(Updated Sept. 20, 2016, 8:06 a.m.)


Review request for atlas.


Changes
---

- added TypesREST.java, the REST API to work with type-defs 
(enum/struct/trait/entity). This should provide the necessary details for 
integration. A similar interface will shorlty be added for instances - to deal 
with creation of entity/trait/process/add-remove-trait/lineage. Implementation 
of these interfaces will be added later.
- Jeff - thanks for the review and comments. Addressed few of these comments in 
this update.


Bugs: ATLAS-1171
https://issues.apache.org/jira/browse/ATLAS-1171


Repository: atlas


Description
---

first-cut API for review


Diffs (updated)
-

  common/pom.xml e3b6465 
  common/src/main/java/org/apache/atlas/api/AtlasApiEntities.java PRE-CREATION 
  common/src/main/java/org/apache/atlas/api/AtlasApiTypes.java PRE-CREATION 
  common/src/main/java/org/apache/atlas/api/PList.java PRE-CREATION 
  common/src/main/java/org/apache/atlas/api/SearchFilter.java PRE-CREATION 
  common/src/main/java/org/apache/atlas/model/instances/AtlasEntity.java 
PRE-CREATION 
  common/src/main/java/org/apache/atlas/model/instances/AtlasObjectId.java 
PRE-CREATION 
  common/src/main/java/org/apache/atlas/model/instances/AtlasProcess.java 
PRE-CREATION 
  common/src/main/java/org/apache/atlas/model/instances/AtlasStruct.java 
PRE-CREATION 
  common/src/main/java/org/apache/atlas/model/instances/AtlasTrait.java 
PRE-CREATION 
  common/src/main/java/org/apache/atlas/model/types/AtlasBaseTypeDef.java 
PRE-CREATION 
  common/src/main/java/org/apache/atlas/model/types/AtlasEntityDef.java 
PRE-CREATION 
  common/src/main/java/org/apache/atlas/model/types/AtlasEnumDef.java 
PRE-CREATION 
  common/src/main/java/org/apache/atlas/model/types/AtlasStructDef.java 
PRE-CREATION 
  common/src/main/java/org/apache/atlas/model/types/AtlasTraitDef.java 
PRE-CREATION 
  common/src/main/java/org/apache/atlas/typesystem/AtlasArrayType.java 
PRE-CREATION 
  common/src/main/java/org/apache/atlas/typesystem/AtlasBuiltInTypes.java 
PRE-CREATION 
  common/src/main/java/org/apache/atlas/typesystem/AtlasDataType.java 
PRE-CREATION 
  common/src/main/java/org/apache/atlas/typesystem/AtlasEntityType.java 
PRE-CREATION 
  common/src/main/java/org/apache/atlas/typesystem/AtlasEnumType.java 
PRE-CREATION 
  common/src/main/java/org/apache/atlas/typesystem/AtlasMapType.java 
PRE-CREATION 
  common/src/main/java/org/apache/atlas/typesystem/AtlasStructType.java 
PRE-CREATION 
  common/src/main/java/org/apache/atlas/typesystem/AtlasTraitType.java 
PRE-CREATION 
  common/src/main/java/org/apache/atlas/typesystem/AtlasTypeRegistry.java 
PRE-CREATION 
  common/src/test/java/org/apache/atlas/model/ModelTestUtil.java PRE-CREATION 
  common/src/test/java/org/apache/atlas/model/instances/TestAtlasEntity.java 
PRE-CREATION 
  common/src/test/java/org/apache/atlas/model/types/TestAtlasEntityDef.java 
PRE-CREATION 
  common/src/test/java/org/apache/atlas/model/types/TestAtlasEnumDef.java 
PRE-CREATION 
  common/src/test/java/org/apache/atlas/model/types/TestAtlasStructDef.java 
PRE-CREATION 
  common/src/test/java/org/apache/atlas/model/types/TestAtlasTraitDef.java 
PRE-CREATION 
  pom.xml ac5b042 
  webapp/src/main/java/org/apache/atlas/web/rest/TypesREST.java PRE-CREATION 

Diff: https://reviews.apache.org/r/51896/diff/


Testing
---


Thanks,

Madhan Neethiraj

Re: Review Request 51896: ATLAS-1171: structured, high-level APIs

[Madhan Neethiraj]

> On Sept. 19, 2016, 11:13 p.m., Jeff Hagelberg wrote:
> > common/src/main/java/org/apache/atlas/model/AtlasAttributeDef.java, line 58
> > 
> >
> > This seems redundant.  Isn't it automatically optional if 
> > valuesMinCount = 0?

Following attributes replace current use of type 'Multiplicity' - since this 
type does not seem very intutive. I think use of simply named attributes make 
it intutive.
 - isOptional, isMultiValued, valuesMinCount, valuesMaxCount, areValuesUnique


> On Sept. 19, 2016, 11:13 p.m., Jeff Hagelberg wrote:
> > common/src/main/java/org/apache/atlas/model/AtlasAttributeDef.java, line 61
> > 
> >
> > Why is this needed?  It seems like if 'valuesMaxCount' > 1, it is 
> > automatically multi-valued.

Please see my earlier commit. I think the use of explicit attributes like 
isMultiValued make it easier to read/understand.


> On Sept. 19, 2016, 11:13 p.m., Jeff Hagelberg wrote:
> > common/src/main/java/org/apache/atlas/model/AtlasAttributeDef.java, line 62
> > 
> >
> > Should valuesMinCount/valuesMaxCount be changed to 
> > lowerBound/upperBound?  Those seem like more intuitive names.

"bound" might mean the value of the attribute, ranther than the number of 
values in a multi-value attribute. Hence the names valuesMinCount and 
valuesMaxCount.


> On Sept. 19, 2016, 11:13 p.m., Jeff Hagelberg wrote:
> > common/src/main/java/org/apache/atlas/model/AtlasAttributeDef.java, line 64
> > 
> >
> > How is this different from isUnique?

isUnique is to specify if the attribute value should be unique across all 
entities.
areValuesUnique is specific to multi-values attributes (isMultiValued=true), to 
specify if the values of an attribute instance should be unique.


> On Sept. 19, 2016, 11:13 p.m., Jeff Hagelberg wrote:
> > common/src/main/java/org/apache/atlas/model/AtlasBaseModelObject.java, line 
> > 47
> > 
> >
> > It's not clear what "disabled" is and how it is different from 
> > "deleted".  I also tend to think that the status should be an internal 
> > attribute that is not exposed.  It seems like by default, users should be 
> > able to assume that any object that comes back from a query has not been 
> > deleted.  I know that is not currently the case though.  The query behavior 
> > really should not depend on whether hard or soft delete is being used, 
> > though, it should be transparent.  I would prefer to have a separate set of 
> > apis for managing the entity status where you can query for deleted objects 
> > and change their state back to ACTIVE if desired.

- removed "disabled".
- "status" field will be an internal field, which will only be set in 
create/delete calls. If explicit status change is necessary, we can add 
separate APIs.


> On Sept. 19, 2016, 11:13 p.m., Jeff Hagelberg wrote:
> > common/src/main/java/org/apache/atlas/model/AtlasBuiltInDatatypes.java, 
> > line 22
> > 
> >
> > It seems like constants like this should be declared once in the code 
> > and referred to everyplace else.  For example, AtlasClient.DataSet, 
> > AtlasClient.Process, BooleanType.name (which could be made public).

- moved ATLAS_TYPE_* constants to AtlasBaseTypeDefs
- added types Process, DataSet, Asset, Infrastructure, Referenceable to this 
built-in types. Yet to add classes for many of these though.


> On Sept. 19, 2016, 11:13 p.m., Jeff Hagelberg wrote:
> > common/src/main/java/org/apache/atlas/model/AtlasBuiltInDatatypes.java, 
> > line 33
> > 
> >
> > I'm not sure what this is.  Is it a data type?

This type is used to reference to another entity. I think it might be easier to 
continue with the current approach of directly using the entity-type name as 
the dataTypeName - like hive_table having 'dbName' attribute of type 
'hive_database'. I will remove 'objectid' type shortly.


- Madhan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51896/#review149121
---


On Sept. 19, 2016, 6:51 p.m., Madhan Neethiraj wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51896/
> ---
> 
> (Updated Sept. 19, 2016, 6:51 p.m.)
> 
> 
> Review request for atlas.
> 
> 
> Bugs: ATLAS-1171
> 

Review Request 52077: Column level lineage in Hive

[Vimal Sharma]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52077/
---

Review request for atlas.


Repository: atlas


Description
---

After a CTAS query, lineage relationship between source columns and destination 
column can be captured. This information can be used to create a column lineage 
process.


Diffs
-

  addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
a3464a0 
  
addons/hive-bridge/src/main/java/org/apache/atlas/hive/model/HiveDataModelGenerator.java
 45f0bc9 
  
addons/hive-bridge/src/main/java/org/apache/atlas/hive/model/HiveDataTypes.java 
e094cb6 
  addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 
a5838b4 

Diff: https://reviews.apache.org/r/52077/diff/


Testing
---


Thanks,

Vimal Sharma

[jira] [Commented] (ATLAS-1145) Storm hook with kafka topic and hive bolt picks wrong hive table name

[Ayub Khan (JIRA)]

[ 
https://issues.apache.org/jira/browse/ATLAS-1145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15505762#comment-15505762
 ] 

Ayub Khan commented on ATLAS-1145:
--

This has some hive-kafka examples: 
https://github.com/sakserv/storm-topology-examples

> Storm hook with kafka topic and hive bolt picks wrong hive table name 
> --
>
> Key: ATLAS-1145
> URL: https://issues.apache.org/jira/browse/ATLAS-1145
> Project: Atlas
>  Issue Type: Bug
>Reporter: Shwetha G S
>Assignee: Sarath Subramanian
>
> Storm hook with kafka topic and hive bolt picks kafka topic name as hive 
> table name



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)