-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73862/
-----------------------------------------------------------
Review request for atlas, Jayendra Parab and Pinal Shah.
Bugs: ATLAS-3682
https://issues.apache.org/jira/browse/ATLAS-3682
Repository: atlas
Description
-------
Problem : Multivalued attributes like arrayList, HashMap are not searchable.
On clicking on atlas UI basic search, attribute like arrayList and hashmap are
not listed in attribute filter list because on atlas server end there is no
proper mapping of atlas typeName "ArrayList" and "HashMap" with respect to
janusgraph. Janusgraph does not support arrayList or HashMap typeName.
Current Behaviour : Atlas server has hardcoding of Cardinality as SINGLE for
attribute typeName = ArrayList.
Approach : To support typeName arrayList<String>, janusgraph expects typeName
as String, cardinality as LIST/SET. hence we need to map internal Atlas Types
accordingly
Changes Needed:
- Changes in creation of attribute of type
arrayList
- Changes in setting the values to the property
- Changes in getting the values of the property
- Java Patch to be added to update the
cardinality of the attribute whose typeName is ArrayList to SINGLE in the
typeDefinition because internally for ArrayList it is hardcoded to SINGLE
- TypeName hashmap attribute will get stored in
"||" double pipe format.
- Changes in filter search for atlas typeName
"ArrayList" and "HashMap" from UI or POST request.
Diffs
-----
common/src/main/java/org/apache/atlas/repository/Constants.java 2669c8a7e
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasElement.java
ef8bd4265
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusElement.java
66ee4e96e
graphdb/janus/src/main/java/org/janusgraph/diskstorage/solr/Solr6Index.java
81ae202f1
intg/src/main/java/org/apache/atlas/type/AtlasStructType.java 21ce23657
repository/src/main/java/org/apache/atlas/discovery/SearchProcessor.java
f69dc4215
repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java
d65bb1a2a
repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java
d5112bbf6
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasGraphUtilsV2.java
2fce123f8
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java
6d8305a32
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java
794847555
repository/src/main/java/org/apache/atlas/util/SearchPredicateUtil.java
9c2975645
Diff: https://reviews.apache.org/r/73862/diff/1/
Testing
-------
Tested for above mentioned scenario, working as expected.
Step to follow for testing this scenario,
-mvn clean install
-make cardinality = single changes in 1030-hive_model.json file wherein
entity=hive_table and attributeDefs name=aliases
-atlas_start
-quick_start
1) Create an attributeDefs for hive_table entity
PUT http method -> URL api/atlas/v2/types/typedefs
{ "name": "dummylist",
"typeName": "array<string>",
"isOptional": true,
"cardinality": "SET",
"valuesMinCount": 0,
"valuesMaxCount": 2147483647,
"isUnique": false,
"isIndexable": true,
"includeInNotification": false,
"searchWeight": -1
},
{
"name": "dummyMap",
"typeName": "map<string,string>",
"isOptional": true,
"cardinality": "SET",
"valuesMinCount": 0,
"valuesMaxCount": 1,
"isUnique": false,
"isIndexable": true,
"includeInNotification": false,
"searchWeight": -1
},
2) create an hive_table entity from UI or POST request
POST http method -> URL api/atlas/v2/entity
{
"entity": {
"guid": "-1",
"isProxy": false,
"isIncomplete": false,
"provenanceType": 0,
"version": 0,
"relationshipAttributes": {
"columns": [],
"ddlQueries": [],
"inputToProcesses": [],
"meanings": [],
"outputFromProcesses": [],
"partitionKeys": [],
"schema": []
},
"typeName": "hive_table",
"attributes": {
"aliases": [
"X",
"Y",
"Z",
"A"
],
"createTime": 1638210600000,
"lastAccessTime": 1638210600000,
"name": "A1",
"dummyMap": {
"name": "dummy"
},
"dummyList": [
"X",
"Y",
"Z",
"A"
],
"qualifiedName": "A1",
"replicatedFrom": [],
"replicatedTo": []
}
},
"referredEntities": {}
}
3) Search list through POST API
POST http method -> api/atlas/v2/search/basic
SEARCH OPERATOR SUPPORT :- =, IN, isNull, notNull, notEmpty
LIST SEARCH API :-
{
"excludeDeletedEntities": true,
"includeSubClassifications": true,
"includeSubTypes": true,
"includeClassificationAttributes": true,
"entityFilters": {
"condition": "AND",
"criterion": [
{
"attributeName": "dummyList",
"operator": "=",
"attributeValue": "x"
}
]
},
"tagFilters": null,
"attributes": [
"dummyList"
],
"limit": 25,
"offset": 0,
"typeName": "hive_table",
"classification": null,
"termName": null
}
------------------------------------------------------------------------------------
MAP SEARCH API :-
{
"excludeDeletedEntities": true,
"includeSubClassifications": true,
"includeSubTypes": true,
"includeClassificationAttributes": true,
"entityFilters": {
"condition": "AND",
"criterion": [
{
"attributeName": "dummyMap
"operator": "=",
"attributeValue": "name=dummy"
}
]
},
"tagFilters": null,
"attributes": [
"dummyMap"
],
"limit": 25,
"offset": 0,
"typeName": "hive_table",
"classification": null,
"termName": null
}
Thanks,
Paresh Devalia
