[atlas] branch branch-2.0 updated: ATLAS-4536: The newline character in attribues will fail simple auth check

[sarath]

This is an automated email from the ASF dual-hosted git repository.

sarath pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/atlas.git


The following commit(s) were added to refs/heads/branch-2.0 by this push:
 new 3a2378f  ATLAS-4536: The newline character in attribues will fail 
simple auth check
3a2378f is described below

commit 3a2378f2adc6b796a838cde411f2f3edd77311ea
Author: Garry Easop 
AuthorDate: Thu Jan 20 11:50:57 2022 -0800

ATLAS-4536: The newline character in attribues will fail simple auth check

Signed-off-by: Sarath Subramanian 
(cherry picked from commit fd55c6fe2695c62338c0cfa751d1e019a523e81f)
---
 .../java/org/apache/atlas/authorize/simple/AtlasSimpleAuthorizer.java  | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git 
a/authorization/src/main/java/org/apache/atlas/authorize/simple/AtlasSimpleAuthorizer.java
 
b/authorization/src/main/java/org/apache/atlas/authorize/simple/AtlasSimpleAuthorizer.java
old mode 100644
new mode 100755
index 7b99e97..d80cab4
--- 
a/authorization/src/main/java/org/apache/atlas/authorize/simple/AtlasSimpleAuthorizer.java
+++ 
b/authorization/src/main/java/org/apache/atlas/authorize/simple/AtlasSimpleAuthorizer.java
@@ -64,6 +64,7 @@ public final class AtlasSimpleAuthorizer implements 
AtlasAuthorizer {
 
 private AtlasSimpleAuthzPolicy authzPolicy;
 
+private final static String REGEX_MATCHALL = ".*";
 
 public AtlasSimpleAuthorizer() {
 }
@@ -466,7 +467,7 @@ public final class AtlasSimpleAuthorizer implements 
AtlasAuthorizer {
 if (value == null) {
 ret = true;
 } else {
-ret = StringUtils.equalsIgnoreCase(value, pattern) || 
value.matches(pattern);
+ret = pattern.equals(REGEX_MATCHALL) || 
StringUtils.equalsIgnoreCase(value, pattern) || value.matches(pattern);
 }
 
 return ret;

[atlas] branch master updated: ATLAS-4536: The newline character in attribues will fail simple auth check

[sarath]

This is an automated email from the ASF dual-hosted git repository.

sarath pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/atlas.git


The following commit(s) were added to refs/heads/master by this push:
 new fd55c6f  ATLAS-4536: The newline character in attribues will fail 
simple auth check
fd55c6f is described below

commit fd55c6fe2695c62338c0cfa751d1e019a523e81f
Author: Garry Easop 
AuthorDate: Thu Jan 20 11:50:57 2022 -0800

ATLAS-4536: The newline character in attribues will fail simple auth check

Signed-off-by: Sarath Subramanian 
---
 .../java/org/apache/atlas/authorize/simple/AtlasSimpleAuthorizer.java  | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git 
a/authorization/src/main/java/org/apache/atlas/authorize/simple/AtlasSimpleAuthorizer.java
 
b/authorization/src/main/java/org/apache/atlas/authorize/simple/AtlasSimpleAuthorizer.java
old mode 100644
new mode 100755
index 7b99e97..d80cab4
--- 
a/authorization/src/main/java/org/apache/atlas/authorize/simple/AtlasSimpleAuthorizer.java
+++ 
b/authorization/src/main/java/org/apache/atlas/authorize/simple/AtlasSimpleAuthorizer.java
@@ -64,6 +64,7 @@ public final class AtlasSimpleAuthorizer implements 
AtlasAuthorizer {
 
 private AtlasSimpleAuthzPolicy authzPolicy;
 
+private final static String REGEX_MATCHALL = ".*";
 
 public AtlasSimpleAuthorizer() {
 }
@@ -466,7 +467,7 @@ public final class AtlasSimpleAuthorizer implements 
AtlasAuthorizer {
 if (value == null) {
 ret = true;
 } else {
-ret = StringUtils.equalsIgnoreCase(value, pattern) || 
value.matches(pattern);
+ret = pattern.equals(REGEX_MATCHALL) || 
StringUtils.equalsIgnoreCase(value, pattern) || value.matches(pattern);
 }
 
 return ret;

[atlas] branch branch-2.0 updated: ATLAS-4522: Updating typedef with new supertype should be allowed only if attributes are unique compared to other existing supertypes

[sarath]

This is an automated email from the ASF dual-hosted git repository.

sarath pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/atlas.git


The following commit(s) were added to refs/heads/branch-2.0 by this push:
 new 1d026fa  ATLAS-4522: Updating typedef with new supertype should be 
allowed only if attributes are unique compared to other existing supertypes
1d026fa is described below

commit 1d026fa463bbda95ac78b4a8db377412989077b6
Author: Radhika Kundam 
AuthorDate: Wed Jan 5 17:38:52 2022 -0500

ATLAS-4522: Updating typedef with new supertype should be allowed only if 
attributes are unique compared to other existing supertypes

Signed-off-by: Sarath Subramanian 
(cherry picked from commit 1b0dff89e26ff06f7b14456369975fe696f05ebc)
---
 .../main/java/org/apache/atlas/AtlasErrorCode.java |  1 +
 .../org/apache/atlas/type/AtlasEntityType.java | 18 --
 .../apache/atlas/type/TestAtlasTypeRegistry.java   | 66 ++
 .../store/graph/v2/EntityGraphRetriever.java   |  2 +
 .../store/graph/v2/AtlasEntityStoreV2Test.java |  5 +-
 .../web/integration/TypedefsJerseyResourceIT.java  |  3 +-
 .../web/integration/TypesJerseyResourceIT.java |  3 +-
 7 files changed, 90 insertions(+), 8 deletions(-)

diff --git a/intg/src/main/java/org/apache/atlas/AtlasErrorCode.java 
b/intg/src/main/java/org/apache/atlas/AtlasErrorCode.java
index 9d11820..c7e6f3a 100644
--- a/intg/src/main/java/org/apache/atlas/AtlasErrorCode.java
+++ b/intg/src/main/java/org/apache/atlas/AtlasErrorCode.java
@@ -173,6 +173,7 @@ public enum AtlasErrorCode {
 NO_DATA_FOUND(400, "ATLAS-400-00-09B", "No data found in the uploaded 
file"),
 NOT_VALID_FILE(400, "ATLAS-400-00-09C", "Invalid {0} file"),
 ATTRIBUTE_NAME_ALREADY_EXISTS_IN_PARENT_TYPE(400, "ATLAS-400-00-09D", 
"Invalid attribute name: {0}.{1}. Attribute already exists in parent type: 
{2}"),
+ATTRIBUTE_NAME_ALREADY_EXISTS_IN_ANOTHER_PARENT_TYPE(400, 
"ATLAS-400-00-09E", "Invalid attribute name: {0}.{1}. Attribute already exists 
in another parent type: {2}"),
 UNAUTHORIZED_ACCESS(403, "ATLAS-403-00-001", "{0} is not authorized to 
perform {1}"),
 
 // All Not found enums go here
diff --git a/intg/src/main/java/org/apache/atlas/type/AtlasEntityType.java 
b/intg/src/main/java/org/apache/atlas/type/AtlasEntityType.java
index 6d83599..76bee36 100644
--- a/intg/src/main/java/org/apache/atlas/type/AtlasEntityType.java
+++ b/intg/src/main/java/org/apache/atlas/type/AtlasEntityType.java
@@ -843,9 +843,10 @@ public class AtlasEntityType extends AtlasStructType {
 private void getTypeHierarchyInfo(AtlasTypeRegistry typeRegistry,
   Set allSuperTypeNames,
   Map 
allAttributes) throws AtlasBaseException {
-List visitedTypes = new ArrayList<>();
+ListvisitedTypes = new ArrayList<>();
+Map attributeToEntityNameMap = new HashMap<>();
 
-collectTypeHierarchyInfo(typeRegistry, allSuperTypeNames, 
allAttributes, visitedTypes);
+collectTypeHierarchyInfo(typeRegistry, allSuperTypeNames, 
allAttributes, attributeToEntityNameMap, visitedTypes);
 }
 
 
@@ -858,6 +859,7 @@ public class AtlasEntityType extends AtlasStructType {
 private void collectTypeHierarchyInfo(AtlasTypeRegistry typeRegistry,
   Set allSuperTypeNames,
   Map 
allAttributes,
+  Map 
attributeToEntityNameMap,
   List visitedTypes) throws 
AtlasBaseException {
 if (visitedTypes.contains(entityDef.getName())) {
 throw new AtlasBaseException(AtlasErrorCode.CIRCULAR_REFERENCE, 
entityDef.getName(),
@@ -870,7 +872,7 @@ public class AtlasEntityType extends AtlasStructType {
 AtlasEntityType superType = 
typeRegistry.getEntityTypeByName(superTypeName);
 
 if (superType != null) {
-superType.collectTypeHierarchyInfo(typeRegistry, 
allSuperTypeNames, allAttributes, visitedTypes);
+superType.collectTypeHierarchyInfo(typeRegistry, 
allSuperTypeNames, allAttributes, attributeToEntityNameMap, visitedTypes);
 }
 }
 visitedTypes.remove(entityDef.getName());
@@ -879,9 +881,15 @@ public class AtlasEntityType extends AtlasStructType {
 
 if (CollectionUtils.isNotEmpty(entityDef.getAttributeDefs())) {
 for (AtlasAttributeDef attributeDef : 
entityDef.getAttributeDefs()) {
+AtlasType type  = 
typeRegistry.getType(attributeDef.getTypeName());
+StringattributeName = attributeDef.getName();
 
-AtlasType type = 
typeRegist

[atlas] branch master updated: ATLAS-4522: Updating typedef with new supertype should be allowed only if attributes are unique compared to other existing supertypes

[sarath]

This is an automated email from the ASF dual-hosted git repository.

sarath pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/atlas.git


The following commit(s) were added to refs/heads/master by this push:
 new 1b0dff8  ATLAS-4522: Updating typedef with new supertype should be 
allowed only if attributes are unique compared to other existing supertypes
1b0dff8 is described below

commit 1b0dff89e26ff06f7b14456369975fe696f05ebc
Author: Radhika Kundam 
AuthorDate: Wed Jan 5 17:38:52 2022 -0500

ATLAS-4522: Updating typedef with new supertype should be allowed only if 
attributes are unique compared to other existing supertypes

Signed-off-by: Sarath Subramanian 
---
 .../main/java/org/apache/atlas/AtlasErrorCode.java |  1 +
 .../org/apache/atlas/type/AtlasEntityType.java | 18 --
 .../apache/atlas/type/TestAtlasTypeRegistry.java   | 66 ++
 .../store/graph/v2/EntityGraphRetriever.java   |  2 +
 .../store/graph/v2/AtlasEntityStoreV2Test.java |  5 +-
 .../web/integration/TypedefsJerseyResourceIT.java  |  3 +-
 .../web/integration/TypesJerseyResourceIT.java |  3 +-
 7 files changed, 90 insertions(+), 8 deletions(-)

diff --git a/intg/src/main/java/org/apache/atlas/AtlasErrorCode.java 
b/intg/src/main/java/org/apache/atlas/AtlasErrorCode.java
index 9d11820..c7e6f3a 100644
--- a/intg/src/main/java/org/apache/atlas/AtlasErrorCode.java
+++ b/intg/src/main/java/org/apache/atlas/AtlasErrorCode.java
@@ -173,6 +173,7 @@ public enum AtlasErrorCode {
 NO_DATA_FOUND(400, "ATLAS-400-00-09B", "No data found in the uploaded 
file"),
 NOT_VALID_FILE(400, "ATLAS-400-00-09C", "Invalid {0} file"),
 ATTRIBUTE_NAME_ALREADY_EXISTS_IN_PARENT_TYPE(400, "ATLAS-400-00-09D", 
"Invalid attribute name: {0}.{1}. Attribute already exists in parent type: 
{2}"),
+ATTRIBUTE_NAME_ALREADY_EXISTS_IN_ANOTHER_PARENT_TYPE(400, 
"ATLAS-400-00-09E", "Invalid attribute name: {0}.{1}. Attribute already exists 
in another parent type: {2}"),
 UNAUTHORIZED_ACCESS(403, "ATLAS-403-00-001", "{0} is not authorized to 
perform {1}"),
 
 // All Not found enums go here
diff --git a/intg/src/main/java/org/apache/atlas/type/AtlasEntityType.java 
b/intg/src/main/java/org/apache/atlas/type/AtlasEntityType.java
index 6d83599..76bee36 100644
--- a/intg/src/main/java/org/apache/atlas/type/AtlasEntityType.java
+++ b/intg/src/main/java/org/apache/atlas/type/AtlasEntityType.java
@@ -843,9 +843,10 @@ public class AtlasEntityType extends AtlasStructType {
 private void getTypeHierarchyInfo(AtlasTypeRegistry typeRegistry,
   Set allSuperTypeNames,
   Map 
allAttributes) throws AtlasBaseException {
-List visitedTypes = new ArrayList<>();
+ListvisitedTypes = new ArrayList<>();
+Map attributeToEntityNameMap = new HashMap<>();
 
-collectTypeHierarchyInfo(typeRegistry, allSuperTypeNames, 
allAttributes, visitedTypes);
+collectTypeHierarchyInfo(typeRegistry, allSuperTypeNames, 
allAttributes, attributeToEntityNameMap, visitedTypes);
 }
 
 
@@ -858,6 +859,7 @@ public class AtlasEntityType extends AtlasStructType {
 private void collectTypeHierarchyInfo(AtlasTypeRegistry typeRegistry,
   Set allSuperTypeNames,
   Map 
allAttributes,
+  Map 
attributeToEntityNameMap,
   List visitedTypes) throws 
AtlasBaseException {
 if (visitedTypes.contains(entityDef.getName())) {
 throw new AtlasBaseException(AtlasErrorCode.CIRCULAR_REFERENCE, 
entityDef.getName(),
@@ -870,7 +872,7 @@ public class AtlasEntityType extends AtlasStructType {
 AtlasEntityType superType = 
typeRegistry.getEntityTypeByName(superTypeName);
 
 if (superType != null) {
-superType.collectTypeHierarchyInfo(typeRegistry, 
allSuperTypeNames, allAttributes, visitedTypes);
+superType.collectTypeHierarchyInfo(typeRegistry, 
allSuperTypeNames, allAttributes, attributeToEntityNameMap, visitedTypes);
 }
 }
 visitedTypes.remove(entityDef.getName());
@@ -879,9 +881,15 @@ public class AtlasEntityType extends AtlasStructType {
 
 if (CollectionUtils.isNotEmpty(entityDef.getAttributeDefs())) {
 for (AtlasAttributeDef attributeDef : 
entityDef.getAttributeDefs()) {
+AtlasType type  = 
typeRegistry.getType(attributeDef.getTypeName());
+StringattributeName = attributeDef.getName();
 
-AtlasType type = 
typeRegistry.getType(attributeDef.getTypeName());
-allAttributes.put(attributeDe

[atlas] 01/02: ATLAS-4370: Persist Metrics for user to retrieve Metrics info at a past timestamp

[sarath]

This is an automated email from the ASF dual-hosted git repository.

sarath pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/atlas.git

commit 447efb8e61b6604c2c4ba34991a18e78c0906432
Author: JP Li 
AuthorDate: Thu Dec 2 01:15:53 2021 -0500

ATLAS-4370: Persist Metrics for user to retrieve Metrics info at a past 
timestamp

Co-authored-by: Disha Talreja 

Signed-off-by: Sarath Subramanian 
(cherry picked from commit a4ee6c05ad6e8759cf2945024684bfbd1e7af8e1)
---
 addons/models/-Area0/0010-base_model.json  |  42 
 .../java/org/apache/atlas/AtlasConfiguration.java  |   4 +-
 .../main/java/org/apache/atlas/AtlasErrorCode.java |   1 +
 .../model/metrics/AtlasMetricsMapToChart.java  |  62 +
 .../atlas/model/metrics/AtlasMetricsStat.java  | 159 +
 .../org/apache/atlas/utils/AtlasEntityUtil.java|  50 
 .../repository/graph/GraphBackedSearchIndexer.java |   7 +
 .../ogm/metrics/AtlasMetricsStatDTO.java   | 153 
 .../org/apache/atlas/services/MetricsService.java  | 264 -
 .../test/java/org/apache/atlas/TestModules.java|   2 +
 .../apache/atlas/services/MetricsServiceTest.java  |  78 +-
 .../resources/solr/core-template/solrconfig.xml|   2 +-
 .../apache/atlas/web/resources/AdminResource.java  | 196 ++-
 13 files changed, 1001 insertions(+), 19 deletions(-)

diff --git a/addons/models/-Area0/0010-base_model.json 
b/addons/models/-Area0/0010-base_model.json
index 769d885..a4a9248 100644
--- a/addons/models/-Area0/0010-base_model.json
+++ b/addons/models/-Area0/0010-base_model.json
@@ -371,6 +371,48 @@
   ]
 },
 {
+  "name": "__AtlasMetricsStat",
+  "superTypes": [
+"__internal"
+  ],
+  "serviceType": "atlas_core",
+  "typeVersion": "1.0",
+  "attributeDefs": [
+{
+  "name": "metricsId",
+  "typeName": "string",
+  "isOptional": false,
+  "cardinality": "SINGLE",
+  "isUnique": true,
+  "isIndexable": true
+},
+{
+  "name": "metrics",
+  "typeName": "string",
+  "isOptional": true,
+  "cardinality": "SINGLE",
+  "isUnique": false,
+  "isIndexable": false
+},
+{
+  "name": "collectionTime",
+  "typeName": "long",
+  "isOptional": true,
+  "cardinality": "SINGLE",
+  "isUnique": false,
+  "isIndexable": true
+},
+{
+  "name": "timeToLiveMillis",
+  "typeName": "long",
+  "isOptional": true,
+  "cardinality": "SINGLE",
+  "isUnique": false,
+  "isIndexable": true
+}
+  ]
+},
+{
   "name": "__ExportImportAuditEntry",
   "serviceType": "atlas_core",
   "typeVersion": "1.0",
diff --git a/intg/src/main/java/org/apache/atlas/AtlasConfiguration.java 
b/intg/src/main/java/org/apache/atlas/AtlasConfiguration.java
index b63fab7..28fb68a 100644
--- a/intg/src/main/java/org/apache/atlas/AtlasConfiguration.java
+++ b/intg/src/main/java/org/apache/atlas/AtlasConfiguration.java
@@ -82,8 +82,8 @@ public enum AtlasConfiguration {
 DEBUG_METRICS_ENABLED("atlas.debug.metrics.enabled", false),
 TASKS_USE_ENABLED("atlas.tasks.enabled", true),
 SESSION_TIMEOUT_SECS("atlas.session.timeout.secs", -1),
-UPDATE_COMPOSITE_INDEX_STATUS("atlas.update.composite.index.status", true);
-
+UPDATE_COMPOSITE_INDEX_STATUS("atlas.update.composite.index.status", true),
+METRICS_TIME_TO_LIVE_HOURS( "atlas.metrics.ttl.hours", 336); // 14 days 
default
 
 private static final Configuration APPLICATION_PROPERTIES;
 
diff --git a/intg/src/main/java/org/apache/atlas/AtlasErrorCode.java 
b/intg/src/main/java/org/apache/atlas/AtlasErrorCode.java
index 7d09261..8bc7996 100644
--- a/intg/src/main/java/org/apache/atlas/AtlasErrorCode.java
+++ b/intg/src/main/java/org/apache/atlas/AtlasErrorCode.java
@@ -211,6 +211,7 @@ public enum AtlasErrorCode {
 GLOSSARY_TERM_ALREADY_EXISTS(409, "ATLAS-409-00-009", "Glossary term with 
qualifiedName {0} already exists"),
 GLOSSARY_CATEGORY_ALREADY_EXISTS(409, "ATLAS-409-00-00A", "Glossary 
category with qualifiedName {0} already exists"),
 GLOSSARY_IMPORT_FAILED(409, "ATLAS-409-00-

[atlas] branch branch-2.0 updated (8810c44 -> ee0d37b)

[sarath]

This is an automated email from the ASF dual-hosted git repository.

sarath pushed a change to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/atlas.git.


from 8810c44  ATLAS-4473: GlossaryTerms Bulk Create Performance Improvement
 new 447efb8  ATLAS-4370: Persist Metrics for user to retrieve Metrics info 
at a past timestamp
 new ee0d37b  ATLAS-4451:UI: #2 Chart representation of Metrics data on 
Statistics dialog box

The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 addons/models/-Area0/0010-base_model.json  |42 +
 dashboardv2/public/css/nv.d3.min.css   |18 +
 dashboardv2/public/css/scss/stats.scss |77 +
 dashboardv2/public/index.html.tpl  | 1 +
 dashboardv2/public/js/router/Router.js | 2 +-
 ...ics_tmpl.html => Statistics_Timeline_tmlp.html} |38 +-
 .../public/js/templates/site/entity_tmpl.html  | 2 +-
 dashboardv2/public/js/utils/UrlLinks.js| 9 +
 .../{Statistics.js => StatisticsTimelineView.js}   |   354 +-
 dashboardv2/public/js/views/site/d3.v3.min.js  |23 +
 dashboardv2/public/js/views/site/nv.d3.js  | 18144 +++
 dashboardv3/public/css/nv.d3.min.css   |18 +
 dashboardv3/public/css/scss/stats.scss |77 +
 dashboardv3/public/index.html.tpl  | 1 +
 dashboardv3/public/js/router/Router.js | 2 +-
 .../templates/site/Statistics_Timeline_tmlp.html   |38 +-
 .../public/js/templates/site/entity_tmpl.html  | 2 +-
 dashboardv3/public/js/utils/UrlLinks.js| 9 +
 .../public/js/views/site/StatisticsTimelineView.js |   354 +-
 dashboardv3/public/js/views/site/d3.v3.min.js  |23 +
 dashboardv3/public/js/views/site/nv.d3.js  | 18144 +++
 .../java/org/apache/atlas/AtlasConfiguration.java  | 4 +-
 .../main/java/org/apache/atlas/AtlasErrorCode.java | 1 +
 .../AtlasMetricsMapToChart.java}   |48 +-
 .../atlas/model/metrics/AtlasMetricsStat.java  |   159 +
 .../org/apache/atlas/utils/AtlasEntityUtil.java|50 +
 .../repository/graph/GraphBackedSearchIndexer.java | 7 +
 .../ogm/metrics/AtlasMetricsStatDTO.java   |   153 +
 .../org/apache/atlas/services/MetricsService.java  |   264 +-
 .../test/java/org/apache/atlas/TestModules.java| 2 +
 .../apache/atlas/services/MetricsServiceTest.java  |78 +-
 .../resources/solr/core-template/solrconfig.xml| 2 +-
 .../apache/atlas/web/resources/AdminResource.java  |   196 +-
 33 files changed, 38245 insertions(+), 97 deletions(-)
 create mode 100644 dashboardv2/public/css/nv.d3.min.css
 copy dashboardv2/public/js/templates/site/{Statistics_tmpl.html => 
Statistics_Timeline_tmlp.html} (80%)
 copy dashboardv2/public/js/views/site/{Statistics.js => 
StatisticsTimelineView.js} (57%)
 create mode 100644 dashboardv2/public/js/views/site/d3.v3.min.js
 create mode 100644 dashboardv2/public/js/views/site/nv.d3.js
 create mode 100644 dashboardv3/public/css/nv.d3.min.css
 copy dashboardv2/public/js/templates/site/Statistics_tmpl.html => 
dashboardv3/public/js/templates/site/Statistics_Timeline_tmlp.html (80%)
 copy dashboardv2/public/js/views/site/Statistics.js => 
dashboardv3/public/js/views/site/StatisticsTimelineView.js (57%)
 create mode 100644 dashboardv3/public/js/views/site/d3.v3.min.js
 create mode 100644 dashboardv3/public/js/views/site/nv.d3.js
 copy 
intg/src/main/java/org/apache/atlas/model/{discovery/AtlasSuggestionsResult.java
 => metrics/AtlasMetricsMapToChart.java} (57%)
 create mode 100644 
intg/src/main/java/org/apache/atlas/model/metrics/AtlasMetricsStat.java
 create mode 100644 
repository/src/main/java/org/apache/atlas/repository/ogm/metrics/AtlasMetricsStatDTO.java

[atlas] branch master updated: ATLAS-4370: Persist Metrics for user to retrieve Metrics info at a past timestamp

[sarath]

This is an automated email from the ASF dual-hosted git repository.

sarath pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/atlas.git


The following commit(s) were added to refs/heads/master by this push:
 new a4ee6c0  ATLAS-4370: Persist Metrics for user to retrieve Metrics info 
at a past timestamp
a4ee6c0 is described below

commit a4ee6c05ad6e8759cf2945024684bfbd1e7af8e1
Author: JP Li 
AuthorDate: Thu Dec 2 01:15:53 2021 -0500

ATLAS-4370: Persist Metrics for user to retrieve Metrics info at a past 
timestamp

Co-authored-by: Disha Talreja 

Signed-off-by: Sarath Subramanian 
---
 addons/models/-Area0/0010-base_model.json  |  42 
 .../java/org/apache/atlas/AtlasConfiguration.java  |   4 +-
 .../main/java/org/apache/atlas/AtlasErrorCode.java |   1 +
 .../model/metrics/AtlasMetricsMapToChart.java  |  62 +
 .../atlas/model/metrics/AtlasMetricsStat.java  | 159 +
 .../org/apache/atlas/utils/AtlasEntityUtil.java|  50 
 .../repository/graph/GraphBackedSearchIndexer.java |   7 +
 .../ogm/metrics/AtlasMetricsStatDTO.java   | 153 
 .../org/apache/atlas/services/MetricsService.java  | 264 -
 .../test/java/org/apache/atlas/TestModules.java|   2 +
 .../apache/atlas/services/MetricsServiceTest.java  |  78 +-
 .../resources/solr/core-template/solrconfig.xml|   2 +-
 .../apache/atlas/web/resources/AdminResource.java  | 196 ++-
 13 files changed, 1001 insertions(+), 19 deletions(-)

diff --git a/addons/models/-Area0/0010-base_model.json 
b/addons/models/-Area0/0010-base_model.json
index 769d885..a4a9248 100644
--- a/addons/models/-Area0/0010-base_model.json
+++ b/addons/models/-Area0/0010-base_model.json
@@ -371,6 +371,48 @@
   ]
 },
 {
+  "name": "__AtlasMetricsStat",
+  "superTypes": [
+"__internal"
+  ],
+  "serviceType": "atlas_core",
+  "typeVersion": "1.0",
+  "attributeDefs": [
+{
+  "name": "metricsId",
+  "typeName": "string",
+  "isOptional": false,
+  "cardinality": "SINGLE",
+  "isUnique": true,
+  "isIndexable": true
+},
+{
+  "name": "metrics",
+  "typeName": "string",
+  "isOptional": true,
+  "cardinality": "SINGLE",
+  "isUnique": false,
+  "isIndexable": false
+},
+{
+  "name": "collectionTime",
+  "typeName": "long",
+  "isOptional": true,
+  "cardinality": "SINGLE",
+  "isUnique": false,
+  "isIndexable": true
+},
+{
+  "name": "timeToLiveMillis",
+  "typeName": "long",
+  "isOptional": true,
+  "cardinality": "SINGLE",
+  "isUnique": false,
+  "isIndexable": true
+}
+  ]
+},
+{
   "name": "__ExportImportAuditEntry",
   "serviceType": "atlas_core",
   "typeVersion": "1.0",
diff --git a/intg/src/main/java/org/apache/atlas/AtlasConfiguration.java 
b/intg/src/main/java/org/apache/atlas/AtlasConfiguration.java
index b63fab7..28fb68a 100644
--- a/intg/src/main/java/org/apache/atlas/AtlasConfiguration.java
+++ b/intg/src/main/java/org/apache/atlas/AtlasConfiguration.java
@@ -82,8 +82,8 @@ public enum AtlasConfiguration {
 DEBUG_METRICS_ENABLED("atlas.debug.metrics.enabled", false),
 TASKS_USE_ENABLED("atlas.tasks.enabled", true),
 SESSION_TIMEOUT_SECS("atlas.session.timeout.secs", -1),
-UPDATE_COMPOSITE_INDEX_STATUS("atlas.update.composite.index.status", true);
-
+UPDATE_COMPOSITE_INDEX_STATUS("atlas.update.composite.index.status", true),
+METRICS_TIME_TO_LIVE_HOURS( "atlas.metrics.ttl.hours", 336); // 14 days 
default
 
 private static final Configuration APPLICATION_PROPERTIES;
 
diff --git a/intg/src/main/java/org/apache/atlas/AtlasErrorCode.java 
b/intg/src/main/java/org/apache/atlas/AtlasErrorCode.java
index 7d09261..8bc7996 100644
--- a/intg/src/main/java/org/apache/atlas/AtlasErrorCode.java
+++ b/intg/src/main/java/org/apache/atlas/AtlasErrorCode.java
@@ -211,6 +211,7 @@ public enum AtlasErrorCode {
 GLOSSARY_TERM_ALREADY_EXISTS(409, "ATLAS-409-00-009", "Glossary term with 
qualifiedName {0} already exists"),
 GLOSSARY_CATEGORY_ALREADY_EXISTS(409, "ATLAS-409-00-00A", "

[atlas] branch branch-2.0 updated: ATLAS-4431: Random NPE when retrieving tasks - #2 - added additional null check

[sarath]

This is an automated email from the ASF dual-hosted git repository.

sarath pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/atlas.git


The following commit(s) were added to refs/heads/branch-2.0 by this push:
 new ee456fe  ATLAS-4431: Random NPE when retrieving tasks - #2 - added 
additional null check
ee456fe is described below

commit ee456fed753ddc6cdff6cf6dd587a4a764dcb71f
Author: Disha Talreja 
AuthorDate: Mon Nov 8 18:47:42 2021 -0500

ATLAS-4431: Random NPE when retrieving tasks - #2 - added additional null 
check

Signed-off-by: Sarath Subramanian 
(cherry picked from commit 2be96918c89e54da15ce399f774a1938ab07f3db)
---
 .../java/org/apache/atlas/tasks/TaskRegistry.java  | 38 ++
 1 file changed, 31 insertions(+), 7 deletions(-)

diff --git a/repository/src/main/java/org/apache/atlas/tasks/TaskRegistry.java 
b/repository/src/main/java/org/apache/atlas/tasks/TaskRegistry.java
index 32e0ad9..6f770ed 100644
--- a/repository/src/main/java/org/apache/atlas/tasks/TaskRegistry.java
+++ b/repository/src/main/java/org/apache/atlas/tasks/TaskRegistry.java
@@ -181,10 +181,25 @@ public class TaskRegistry {
 private AtlasTask toAtlasTask(AtlasVertex v) {
 AtlasTask ret = new AtlasTask();
 
-ret.setGuid(v.getProperty(Constants.TASK_GUID, String.class));
-ret.setType(v.getProperty(Constants.TASK_TYPE, String.class));
-ret.setStatus(v.getProperty(Constants.TASK_STATUS, String.class));
-ret.setCreatedBy(v.getProperty(Constants.TASK_CREATED_BY, 
String.class));
+String guid = v.getProperty(Constants.TASK_GUID, String.class);
+if (guid != null) {
+ret.setGuid(guid);
+}
+
+String type = v.getProperty(Constants.TASK_TYPE, String.class);
+if (type != null) {
+ret.setType(type);
+}
+
+String status = v.getProperty(Constants.TASK_STATUS, String.class);
+if (status != null) {
+ret.setStatus(status);
+}
+
+String createdBy = v.getProperty(Constants.TASK_CREATED_BY, 
String.class);
+if (createdBy != null) {
+ret.setCreatedBy(createdBy);
+}
 
 Long createdTime = v.getProperty(Constants.TASK_CREATED_TIME, 
Long.class);
 if (createdTime != null) {
@@ -207,10 +222,19 @@ public class TaskRegistry {
 }
 
 String parametersJson = v.getProperty(Constants.TASK_PARAMETERS, 
String.class);
-ret.setParameters(AtlasType.fromJson(parametersJson, Map.class));
+if (parametersJson != null) {
+ret.setParameters(AtlasType.fromJson(parametersJson, Map.class));
+}
 
-ret.setAttemptCount(v.getProperty(Constants.TASK_ATTEMPT_COUNT, 
Integer.class));
-ret.setErrorMessage(v.getProperty(Constants.TASK_ERROR_MESSAGE, 
String.class));
+Integer attemptCount = v.getProperty(Constants.TASK_ATTEMPT_COUNT, 
Integer.class);
+if (attemptCount != null) {
+ret.setAttemptCount(attemptCount);
+}
+
+String errorMessage = v.getProperty(Constants.TASK_ERROR_MESSAGE, 
String.class);
+if (errorMessage != null) {
+ret.setErrorMessage(errorMessage);
+}
 
 return ret;
 }

[atlas] branch master updated: ATLAS-4431: Random NPE when retrieving tasks - #2 - added additional null check

[sarath]

This is an automated email from the ASF dual-hosted git repository.

sarath pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/atlas.git


The following commit(s) were added to refs/heads/master by this push:
 new 2be9691  ATLAS-4431: Random NPE when retrieving tasks - #2 - added 
additional null check
2be9691 is described below

commit 2be96918c89e54da15ce399f774a1938ab07f3db
Author: Disha Talreja 
AuthorDate: Mon Nov 8 18:47:42 2021 -0500

ATLAS-4431: Random NPE when retrieving tasks - #2 - added additional null 
check

Signed-off-by: Sarath Subramanian 
---
 .../java/org/apache/atlas/tasks/TaskRegistry.java  | 38 ++
 1 file changed, 31 insertions(+), 7 deletions(-)

diff --git a/repository/src/main/java/org/apache/atlas/tasks/TaskRegistry.java 
b/repository/src/main/java/org/apache/atlas/tasks/TaskRegistry.java
index 32e0ad9..6f770ed 100644
--- a/repository/src/main/java/org/apache/atlas/tasks/TaskRegistry.java
+++ b/repository/src/main/java/org/apache/atlas/tasks/TaskRegistry.java
@@ -181,10 +181,25 @@ public class TaskRegistry {
 private AtlasTask toAtlasTask(AtlasVertex v) {
 AtlasTask ret = new AtlasTask();
 
-ret.setGuid(v.getProperty(Constants.TASK_GUID, String.class));
-ret.setType(v.getProperty(Constants.TASK_TYPE, String.class));
-ret.setStatus(v.getProperty(Constants.TASK_STATUS, String.class));
-ret.setCreatedBy(v.getProperty(Constants.TASK_CREATED_BY, 
String.class));
+String guid = v.getProperty(Constants.TASK_GUID, String.class);
+if (guid != null) {
+ret.setGuid(guid);
+}
+
+String type = v.getProperty(Constants.TASK_TYPE, String.class);
+if (type != null) {
+ret.setType(type);
+}
+
+String status = v.getProperty(Constants.TASK_STATUS, String.class);
+if (status != null) {
+ret.setStatus(status);
+}
+
+String createdBy = v.getProperty(Constants.TASK_CREATED_BY, 
String.class);
+if (createdBy != null) {
+ret.setCreatedBy(createdBy);
+}
 
 Long createdTime = v.getProperty(Constants.TASK_CREATED_TIME, 
Long.class);
 if (createdTime != null) {
@@ -207,10 +222,19 @@ public class TaskRegistry {
 }
 
 String parametersJson = v.getProperty(Constants.TASK_PARAMETERS, 
String.class);
-ret.setParameters(AtlasType.fromJson(parametersJson, Map.class));
+if (parametersJson != null) {
+ret.setParameters(AtlasType.fromJson(parametersJson, Map.class));
+}
 
-ret.setAttemptCount(v.getProperty(Constants.TASK_ATTEMPT_COUNT, 
Integer.class));
-ret.setErrorMessage(v.getProperty(Constants.TASK_ERROR_MESSAGE, 
String.class));
+Integer attemptCount = v.getProperty(Constants.TASK_ATTEMPT_COUNT, 
Integer.class);
+if (attemptCount != null) {
+ret.setAttemptCount(attemptCount);
+}
+
+String errorMessage = v.getProperty(Constants.TASK_ERROR_MESSAGE, 
String.class);
+if (errorMessage != null) {
+ret.setErrorMessage(errorMessage);
+}
 
 return ret;
 }

[atlas] branch branch-2.0 updated: ATLAS-4408: Dynamic handling of failure in updating index

[sarath]

This is an automated email from the ASF dual-hosted git repository.

sarath pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/atlas.git


The following commit(s) were added to refs/heads/branch-2.0 by this push:
 new 9021859  ATLAS-4408: Dynamic handling of failure in updating index
9021859 is described below

commit 9021859cd90ec56e770c8e68907441807951e894
Author: Radhika Kundam 
AuthorDate: Sun Oct 10 21:50:29 2021 -0700

ATLAS-4408: Dynamic handling of failure in updating index

(cherry picked from commit 261331bbd1fa8e50517421ab834f01f89f4997e1)
---
 .../org/apache/atlas/repository/Constants.java |   8 +
 .../repository/graphdb/AtlasGraphIndexClient.java  |   6 +
 .../repository/graphdb/AtlasGraphManagement.java   |  19 ++
 .../graphdb/janus/AtlasJanusGraphDatabase.java |  74 +-
 .../graphdb/janus/AtlasJanusGraphIndexClient.java  |  55 +++-
 .../graphdb/janus/AtlasJanusGraphManagement.java   |  85 ++-
 .../org/apache/atlas/ApplicationProperties.java|   2 +
 .../repository/graph/GraphBackedSearchIndexer.java |   3 +
 .../repository/graph/IndexRecoveryService.java | 283 +
 .../graph/RecoveryInfoManagementTest.java  |  65 +
 .../atlas/listener/ActiveStateChangeHandler.java   |   3 +-
 11 files changed, 575 insertions(+), 28 deletions(-)

diff --git a/common/src/main/java/org/apache/atlas/repository/Constants.java 
b/common/src/main/java/org/apache/atlas/repository/Constants.java
index aea0c13..2669c8a 100644
--- a/common/src/main/java/org/apache/atlas/repository/Constants.java
+++ b/common/src/main/java/org/apache/atlas/repository/Constants.java
@@ -228,6 +228,14 @@ public final class Constants {
 public static final String TASK_START_TIME= 
encodePropertyKey(TASK_PREFIX + "startTime");
 public static final String TASK_END_TIME  = 
encodePropertyKey(TASK_PREFIX + "endTime");
 
+/**
+ * Index Recovery vertex property keys.
+ */
+public static final String INDEX_RECOVERY_PREFIX  = 
INTERNAL_PROPERTY_KEY_PREFIX + "idxRecovery_";
+public static final String PROPERTY_KEY_INDEX_RECOVERY_NAME   = 
encodePropertyKey(INDEX_RECOVERY_PREFIX + "name");
+public static final String PROPERTY_KEY_INDEX_RECOVERY_START_TIME = 
encodePropertyKey(INDEX_RECOVERY_PREFIX + "startTime");
+public static final String PROPERTY_KEY_INDEX_RECOVERY_PREV_TIME  = 
encodePropertyKey(INDEX_RECOVERY_PREFIX + "prevTime");
+
 /*
  * All supported file-format extensions for Bulk Imports through file 
upload
  */
diff --git 
a/graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphIndexClient.java
 
b/graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphIndexClient.java
index 9960d89..54cf5f6 100644
--- 
a/graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphIndexClient.java
+++ 
b/graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphIndexClient.java
@@ -54,4 +54,10 @@ public interface AtlasGraphIndexClient {
  * @param suggestionProperties the list of suggestion properties.
  */
 void applySuggestionFields(String collectionName, List 
suggestionProperties);
+
+/**
+ * Returns status of index client
+ * @return returns true if index client is active
+ */
+boolean isHealthy();
 }
diff --git 
a/graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphManagement.java
 
b/graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphManagement.java
index 7e3b2f4..50d17a2 100644
--- 
a/graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphManagement.java
+++ 
b/graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphManagement.java
@@ -180,4 +180,23 @@ public interface AtlasGraphManagement {
  * @throws Exception
  */
 void reindex(String indexName, List elements) throws 
Exception;
+
+/**
+ * Starts recovering indices from the specified recovery time and returns 
TransactionRecovery
+ * @param startTime
+ * @return transactionRecoveryObject
+ */
+Object startIndexRecovery(long startTime);
+
+/**
+ * Stop index recovery.
+ * @param txRecoveryObject
+ */
+void stopIndexRecovery(Object txRecoveryObject);
+
+/**
+ * Print index recovery stats.
+ * @param txRecoveryObject
+ */
+void printIndexRecoveryStats(Object txRecoveryObject);
 }
diff --git 
a/graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphDatabase.java
 
b/graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphDatabase.java
index 11267c4..0d47e38 100644
--- 
a/graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphDatabase.java
+++ 
b/graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb

[atlas] branch master updated: ATLAS-4408: Dynamic handling of failure in updating index

[sarath]

This is an automated email from the ASF dual-hosted git repository.

sarath pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/atlas.git


The following commit(s) were added to refs/heads/master by this push:
 new 261331b  ATLAS-4408: Dynamic handling of failure in updating index
261331b is described below

commit 261331bbd1fa8e50517421ab834f01f89f4997e1
Author: Radhika Kundam 
AuthorDate: Sun Oct 10 21:50:29 2021 -0700

ATLAS-4408: Dynamic handling of failure in updating index
---
 .../org/apache/atlas/repository/Constants.java |   8 +
 .../repository/graphdb/AtlasGraphIndexClient.java  |   6 +
 .../repository/graphdb/AtlasGraphManagement.java   |  19 ++
 .../graphdb/janus/AtlasJanusGraphDatabase.java |  74 +-
 .../graphdb/janus/AtlasJanusGraphIndexClient.java  |  55 +++-
 .../graphdb/janus/AtlasJanusGraphManagement.java   |  85 ++-
 .../org/apache/atlas/ApplicationProperties.java|   2 +
 .../repository/graph/GraphBackedSearchIndexer.java |   3 +
 .../repository/graph/IndexRecoveryService.java | 283 +
 .../graph/RecoveryInfoManagementTest.java  |  65 +
 .../atlas/listener/ActiveStateChangeHandler.java   |   3 +-
 11 files changed, 575 insertions(+), 28 deletions(-)

diff --git a/common/src/main/java/org/apache/atlas/repository/Constants.java 
b/common/src/main/java/org/apache/atlas/repository/Constants.java
index aea0c13..2669c8a 100644
--- a/common/src/main/java/org/apache/atlas/repository/Constants.java
+++ b/common/src/main/java/org/apache/atlas/repository/Constants.java
@@ -228,6 +228,14 @@ public final class Constants {
 public static final String TASK_START_TIME= 
encodePropertyKey(TASK_PREFIX + "startTime");
 public static final String TASK_END_TIME  = 
encodePropertyKey(TASK_PREFIX + "endTime");
 
+/**
+ * Index Recovery vertex property keys.
+ */
+public static final String INDEX_RECOVERY_PREFIX  = 
INTERNAL_PROPERTY_KEY_PREFIX + "idxRecovery_";
+public static final String PROPERTY_KEY_INDEX_RECOVERY_NAME   = 
encodePropertyKey(INDEX_RECOVERY_PREFIX + "name");
+public static final String PROPERTY_KEY_INDEX_RECOVERY_START_TIME = 
encodePropertyKey(INDEX_RECOVERY_PREFIX + "startTime");
+public static final String PROPERTY_KEY_INDEX_RECOVERY_PREV_TIME  = 
encodePropertyKey(INDEX_RECOVERY_PREFIX + "prevTime");
+
 /*
  * All supported file-format extensions for Bulk Imports through file 
upload
  */
diff --git 
a/graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphIndexClient.java
 
b/graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphIndexClient.java
index 9960d89..54cf5f6 100644
--- 
a/graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphIndexClient.java
+++ 
b/graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphIndexClient.java
@@ -54,4 +54,10 @@ public interface AtlasGraphIndexClient {
  * @param suggestionProperties the list of suggestion properties.
  */
 void applySuggestionFields(String collectionName, List 
suggestionProperties);
+
+/**
+ * Returns status of index client
+ * @return returns true if index client is active
+ */
+boolean isHealthy();
 }
diff --git 
a/graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphManagement.java
 
b/graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphManagement.java
index 7e3b2f4..50d17a2 100644
--- 
a/graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphManagement.java
+++ 
b/graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphManagement.java
@@ -180,4 +180,23 @@ public interface AtlasGraphManagement {
  * @throws Exception
  */
 void reindex(String indexName, List elements) throws 
Exception;
+
+/**
+ * Starts recovering indices from the specified recovery time and returns 
TransactionRecovery
+ * @param startTime
+ * @return transactionRecoveryObject
+ */
+Object startIndexRecovery(long startTime);
+
+/**
+ * Stop index recovery.
+ * @param txRecoveryObject
+ */
+void stopIndexRecovery(Object txRecoveryObject);
+
+/**
+ * Print index recovery stats.
+ * @param txRecoveryObject
+ */
+void printIndexRecoveryStats(Object txRecoveryObject);
 }
diff --git 
a/graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphDatabase.java
 
b/graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphDatabase.java
index 11267c4..0d47e38 100644
--- 
a/graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphDatabase.java
+++ 
b/graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphDatabase.java
@@ -28,7 +28,6 @@ import 
org.apache.atlas.repo

[atlas] branch branch-2.0 updated: ATLAS-4421: [Atlas: Hive Import] When import-hive is run with incorrect input the information is not conveyed to the user

[sarath]

This is an automated email from the ASF dual-hosted git repository.

sarath pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/atlas.git


The following commit(s) were added to refs/heads/branch-2.0 by this push:
 new 8556495  ATLAS-4421: [Atlas: Hive Import] When import-hive is run with 
incorrect input the information is not conveyed to the user
8556495 is described below

commit 855649508d5b0e9ab69afb6cf97996ba7393d47c
Author: Disha Talreja 
AuthorDate: Tue Sep 28 02:06:16 2021 -0400

ATLAS-4421: [Atlas: Hive Import] When import-hive is run with incorrect 
input the information is not conveyed to the user

Signed-off-by: Sarath Subramanian 
(cherry picked from commit 0aa45c8657a071bd4ee9e698abbeba81e1e542c6)
---
 addons/hive-bridge/src/bin/import-hive.sh| 4 ++--
 .../java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java   | 9 ++---
 2 files changed, 8 insertions(+), 5 deletions(-)

diff --git a/addons/hive-bridge/src/bin/import-hive.sh 
b/addons/hive-bridge/src/bin/import-hive.sh
index fde486f..c353937 100755
--- a/addons/hive-bridge/src/bin/import-hive.sh
+++ b/addons/hive-bridge/src/bin/import-hive.sh
@@ -156,8 +156,8 @@ echo "Log file for import is $LOGFILE"
 "${JAVA_BIN}" ${JAVA_PROPERTIES} -cp "${CP}" 
org.apache.atlas.hive.bridge.HiveMetaStoreBridge $IMPORT_ARGS
 
 RETVAL=$?
-[ $RETVAL -eq 0 ] && echo Hive Meta Data imported successfully!!!
-[ $RETVAL -ne 0 ] && echo Failed to import Hive Meta Data!!!
+[ $RETVAL -eq 0 ] && echo Hive metadata imported successfully!
+[ $RETVAL -ne 0 ] && echo Failed to import Hive metadata! Check logs at: 
$LOGFILE for details.
 
 exit $RETVAL
 
diff --git 
a/addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
 
b/addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
index 08471b4..c361ac6 100755
--- 
a/addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
+++ 
b/addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
@@ -161,6 +161,7 @@ public class HiveMetaStoreBridge {
 
 if (deleteNonExisting) {
 
hiveMetaStoreBridge.deleteEntitiesForNonExistingHiveMetadata(failOnError);
+exitCode = EXIT_CODE_SUCCESS;
 } else if (StringUtils.isNotEmpty(fileToImport)) {
 File f = new File(fileToImport);
 
@@ -187,12 +188,13 @@ public class HiveMetaStoreBridge {
 exitCode = EXIT_CODE_SUCCESS;
 } else {
 LOG.error("Failed to read the input file: " + 
fileToImport);
+exitCode = EXIT_CODE_FAILED;
 }
 } else {
 hiveMetaStoreBridge.importHiveMetadata(databaseToImport, 
tableToImport, failOnError);
+exitCode = EXIT_CODE_SUCCESS;
 }
 
-exitCode = EXIT_CODE_SUCCESS;
 } catch(ParseException e) {
 LOG.error("Failed to parse arguments. Error: ", e.getMessage());
 printUsage();
@@ -326,7 +328,8 @@ public class HiveMetaStoreBridge {
 }
 }
 } else {
-LOG.info("No database found");
+LOG.error("No database found");
+System.exit(EXIT_CODE_FAILED);
 }
 }
 
@@ -365,7 +368,7 @@ public class HiveMetaStoreBridge {
 }
 }
 } else {
-LOG.info("No tables to import in database {}", databaseName);
+LOG.error("No tables to import in database {}", databaseName);
 }
 
 return tablesImported;

[atlas] branch master updated: ATLAS-4421: [Atlas: Hive Import] When import-hive is run with incorrect input the information is not conveyed to the user

[sarath]

This is an automated email from the ASF dual-hosted git repository.

sarath pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/atlas.git


The following commit(s) were added to refs/heads/master by this push:
 new 0aa45c8  ATLAS-4421: [Atlas: Hive Import] When import-hive is run with 
incorrect input the information is not conveyed to the user
0aa45c8 is described below

commit 0aa45c8657a071bd4ee9e698abbeba81e1e542c6
Author: Disha Talreja 
AuthorDate: Tue Sep 28 02:06:16 2021 -0400

ATLAS-4421: [Atlas: Hive Import] When import-hive is run with incorrect 
input the information is not conveyed to the user

Signed-off-by: Sarath Subramanian 
---
 addons/hive-bridge/src/bin/import-hive.sh| 4 ++--
 .../java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java   | 9 ++---
 2 files changed, 8 insertions(+), 5 deletions(-)

diff --git a/addons/hive-bridge/src/bin/import-hive.sh 
b/addons/hive-bridge/src/bin/import-hive.sh
index fde486f..c353937 100755
--- a/addons/hive-bridge/src/bin/import-hive.sh
+++ b/addons/hive-bridge/src/bin/import-hive.sh
@@ -156,8 +156,8 @@ echo "Log file for import is $LOGFILE"
 "${JAVA_BIN}" ${JAVA_PROPERTIES} -cp "${CP}" 
org.apache.atlas.hive.bridge.HiveMetaStoreBridge $IMPORT_ARGS
 
 RETVAL=$?
-[ $RETVAL -eq 0 ] && echo Hive Meta Data imported successfully!!!
-[ $RETVAL -ne 0 ] && echo Failed to import Hive Meta Data!!!
+[ $RETVAL -eq 0 ] && echo Hive metadata imported successfully!
+[ $RETVAL -ne 0 ] && echo Failed to import Hive metadata! Check logs at: 
$LOGFILE for details.
 
 exit $RETVAL
 
diff --git 
a/addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
 
b/addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
index 08471b4..c361ac6 100755
--- 
a/addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
+++ 
b/addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
@@ -161,6 +161,7 @@ public class HiveMetaStoreBridge {
 
 if (deleteNonExisting) {
 
hiveMetaStoreBridge.deleteEntitiesForNonExistingHiveMetadata(failOnError);
+exitCode = EXIT_CODE_SUCCESS;
 } else if (StringUtils.isNotEmpty(fileToImport)) {
 File f = new File(fileToImport);
 
@@ -187,12 +188,13 @@ public class HiveMetaStoreBridge {
 exitCode = EXIT_CODE_SUCCESS;
 } else {
 LOG.error("Failed to read the input file: " + 
fileToImport);
+exitCode = EXIT_CODE_FAILED;
 }
 } else {
 hiveMetaStoreBridge.importHiveMetadata(databaseToImport, 
tableToImport, failOnError);
+exitCode = EXIT_CODE_SUCCESS;
 }
 
-exitCode = EXIT_CODE_SUCCESS;
 } catch(ParseException e) {
 LOG.error("Failed to parse arguments. Error: ", e.getMessage());
 printUsage();
@@ -326,7 +328,8 @@ public class HiveMetaStoreBridge {
 }
 }
 } else {
-LOG.info("No database found");
+LOG.error("No database found");
+System.exit(EXIT_CODE_FAILED);
 }
 }
 
@@ -365,7 +368,7 @@ public class HiveMetaStoreBridge {
 }
 }
 } else {
-LOG.info("No tables to import in database {}", databaseName);
+LOG.error("No tables to import in database {}", databaseName);
 }
 
 return tablesImported;

[atlas] branch branch-2.0 updated: ATLAS-4431: Add additional null check when retrieving tasks from vertex

[sarath]

This is an automated email from the ASF dual-hosted git repository.

sarath pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/atlas.git


The following commit(s) were added to refs/heads/branch-2.0 by this push:
 new 48b106c  ATLAS-4431: Add additional null check when retrieving tasks 
from vertex
48b106c is described below

commit 48b106c422ca3d7b655c5c9bd4b97e55a6e73598
Author: Disha Talreja 
AuthorDate: Mon Sep 20 16:25:21 2021 -0400

ATLAS-4431: Add additional null check when retrieving tasks from vertex

Signed-off-by: Sarath Subramanian 
(cherry picked from commit e57204bd55d27f258c19e7334c10cc4802cf14c7)
---
 .../src/main/java/org/apache/atlas/tasks/TaskRegistry.java   | 12 ++--
 1 file changed, 10 insertions(+), 2 deletions(-)

diff --git a/repository/src/main/java/org/apache/atlas/tasks/TaskRegistry.java 
b/repository/src/main/java/org/apache/atlas/tasks/TaskRegistry.java
index fae8a4f..32e0ad9 100644
--- a/repository/src/main/java/org/apache/atlas/tasks/TaskRegistry.java
+++ b/repository/src/main/java/org/apache/atlas/tasks/TaskRegistry.java
@@ -185,8 +185,16 @@ public class TaskRegistry {
 ret.setType(v.getProperty(Constants.TASK_TYPE, String.class));
 ret.setStatus(v.getProperty(Constants.TASK_STATUS, String.class));
 ret.setCreatedBy(v.getProperty(Constants.TASK_CREATED_BY, 
String.class));
-ret.setCreatedTime(new Date(v.getProperty(Constants.TASK_CREATED_TIME, 
Long.class)));
-ret.setUpdatedTime(new Date(v.getProperty(Constants.TASK_UPDATED_TIME, 
Long.class)));
+
+Long createdTime = v.getProperty(Constants.TASK_CREATED_TIME, 
Long.class);
+if (createdTime != null) {
+ret.setCreatedTime(new Date(createdTime));
+}
+
+Long updatedTime = v.getProperty(Constants.TASK_UPDATED_TIME, 
Long.class);
+if (updatedTime != null) {
+ret.setUpdatedTime(new Date(updatedTime));
+}
 
 Long startTime = v.getProperty(Constants.TASK_START_TIME, Long.class);
 if (startTime != null) {

[atlas] branch master updated: ATLAS-4431: Add additional null check when retrieving tasks from vertex

[sarath]

This is an automated email from the ASF dual-hosted git repository.

sarath pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/atlas.git


The following commit(s) were added to refs/heads/master by this push:
 new e57204b  ATLAS-4431: Add additional null check when retrieving tasks 
from vertex
e57204b is described below

commit e57204bd55d27f258c19e7334c10cc4802cf14c7
Author: Disha Talreja 
AuthorDate: Mon Sep 20 16:25:21 2021 -0400

ATLAS-4431: Add additional null check when retrieving tasks from vertex

Signed-off-by: Sarath Subramanian 
---
 .../src/main/java/org/apache/atlas/tasks/TaskRegistry.java   | 12 ++--
 1 file changed, 10 insertions(+), 2 deletions(-)

diff --git a/repository/src/main/java/org/apache/atlas/tasks/TaskRegistry.java 
b/repository/src/main/java/org/apache/atlas/tasks/TaskRegistry.java
index fae8a4f..32e0ad9 100644
--- a/repository/src/main/java/org/apache/atlas/tasks/TaskRegistry.java
+++ b/repository/src/main/java/org/apache/atlas/tasks/TaskRegistry.java
@@ -185,8 +185,16 @@ public class TaskRegistry {
 ret.setType(v.getProperty(Constants.TASK_TYPE, String.class));
 ret.setStatus(v.getProperty(Constants.TASK_STATUS, String.class));
 ret.setCreatedBy(v.getProperty(Constants.TASK_CREATED_BY, 
String.class));
-ret.setCreatedTime(new Date(v.getProperty(Constants.TASK_CREATED_TIME, 
Long.class)));
-ret.setUpdatedTime(new Date(v.getProperty(Constants.TASK_UPDATED_TIME, 
Long.class)));
+
+Long createdTime = v.getProperty(Constants.TASK_CREATED_TIME, 
Long.class);
+if (createdTime != null) {
+ret.setCreatedTime(new Date(createdTime));
+}
+
+Long updatedTime = v.getProperty(Constants.TASK_UPDATED_TIME, 
Long.class);
+if (updatedTime != null) {
+ret.setUpdatedTime(new Date(updatedTime));
+}
 
 Long startTime = v.getProperty(Constants.TASK_START_TIME, Long.class);
 if (startTime != null) {

[atlas] branch branch-2.0 updated: ATLAS-4369: import-hive.sh does not write logs to /var/log/atlas/import-hive.log file

[sarath]

This is an automated email from the ASF dual-hosted git repository.

sarath pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/atlas.git


The following commit(s) were added to refs/heads/branch-2.0 by this push:
 new 2520116  ATLAS-4369: import-hive.sh does not write logs to 
/var/log/atlas/import-hive.log file
2520116 is described below

commit 25201164f770480737a2802be842115741685432
Author: Disha Talreja 
AuthorDate: Mon Sep 13 13:44:34 2021 -0400

ATLAS-4369: import-hive.sh does not write logs to 
/var/log/atlas/import-hive.log file

Signed-off-by: Sarath Subramanian 
(cherry picked from commit 6e4d369fd8b1f1529af14261d084c877109b58c6)
---
 addons/hive-bridge/src/bin/import-hive.sh | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/addons/hive-bridge/src/bin/import-hive.sh 
b/addons/hive-bridge/src/bin/import-hive.sh
index 693a79b..fde486f 100755
--- a/addons/hive-bridge/src/bin/import-hive.sh
+++ b/addons/hive-bridge/src/bin/import-hive.sh
@@ -93,7 +93,8 @@ fi
 HIVE_CP="${HIVE_CONF}"
 # Multiple jars in HIVE_CP_EXCLUDE_LIST can be added using "\|" separator
 # Ex: HIVE_CP_EXCLUDE_LIST="javax.ws.rs-api\|jersey-multipart"
-HIVE_CP_EXCLUDE_LIST="javax.ws.rs-api"
+# exclude log4j libs from hive classpath to avoid conflict
+HIVE_CP_EXCLUDE_LIST="javax.ws.rs-api\|log4j-slf4j-impl\|log4j-1.2-api\|log4j-api\|log4j-core\|log4j-web"
 
 for i in $(find "${HIVE_HOME}/lib/" -name  "*.jar" | grep -v 
"$HIVE_CP_EXCLUDE_LIST"); do
 HIVE_CP="${HIVE_CP}:$i"

[atlas] branch master updated: ATLAS-4369: import-hive.sh does not write logs to /var/log/atlas/import-hive.log file

[sarath]

This is an automated email from the ASF dual-hosted git repository.

sarath pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/atlas.git


The following commit(s) were added to refs/heads/master by this push:
 new 6e4d369  ATLAS-4369: import-hive.sh does not write logs to 
/var/log/atlas/import-hive.log file
6e4d369 is described below

commit 6e4d369fd8b1f1529af14261d084c877109b58c6
Author: Disha Talreja 
AuthorDate: Mon Sep 13 13:44:34 2021 -0400

ATLAS-4369: import-hive.sh does not write logs to 
/var/log/atlas/import-hive.log file

Signed-off-by: Sarath Subramanian 
---
 addons/hive-bridge/src/bin/import-hive.sh | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/addons/hive-bridge/src/bin/import-hive.sh 
b/addons/hive-bridge/src/bin/import-hive.sh
index 693a79b..fde486f 100755
--- a/addons/hive-bridge/src/bin/import-hive.sh
+++ b/addons/hive-bridge/src/bin/import-hive.sh
@@ -93,7 +93,8 @@ fi
 HIVE_CP="${HIVE_CONF}"
 # Multiple jars in HIVE_CP_EXCLUDE_LIST can be added using "\|" separator
 # Ex: HIVE_CP_EXCLUDE_LIST="javax.ws.rs-api\|jersey-multipart"
-HIVE_CP_EXCLUDE_LIST="javax.ws.rs-api"
+# exclude log4j libs from hive classpath to avoid conflict
+HIVE_CP_EXCLUDE_LIST="javax.ws.rs-api\|log4j-slf4j-impl\|log4j-1.2-api\|log4j-api\|log4j-core\|log4j-web"
 
 for i in $(find "${HIVE_HOME}/lib/" -name  "*.jar" | grep -v 
"$HIVE_CP_EXCLUDE_LIST"); do
 HIVE_CP="${HIVE_CP}:$i"

[atlas] branch branch-2.0 updated: ATLAS-4403: Improve find entity by unique attributes logic - when multiple unique attributes exist for entity type

[sarath]

This is an automated email from the ASF dual-hosted git repository.

sarath pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/atlas.git


The following commit(s) were added to refs/heads/branch-2.0 by this push:
 new 1a2ca74  ATLAS-4403: Improve find entity by unique attributes logic - 
when multiple unique attributes exist for entity type
1a2ca74 is described below

commit 1a2ca741115c7957c994e562f6a0fb2bc75da8dd
Author: Sarath Subramanian 
AuthorDate: Fri Aug 27 14:11:52 2021 -0700

ATLAS-4403: Improve find entity by unique attributes logic - when multiple 
unique attributes exist for entity type

(cherry picked from commit 097f50eae35e927411af3593fa401a335e61a0e3)
---
 .../store/graph/v2/AtlasGraphUtilsV2.java  | 201 -
 1 file changed, 80 insertions(+), 121 deletions(-)

diff --git 
a/repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasGraphUtilsV2.java
 
b/repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasGraphUtilsV2.java
index e73f084..2fce123 100644
--- 
a/repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasGraphUtilsV2.java
+++ 
b/repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasGraphUtilsV2.java
@@ -59,6 +59,7 @@ import java.util.ArrayList;
 import java.util.Arrays;
 import java.util.Collections;
 import java.util.Date;
+import java.util.HashMap;
 import java.util.Iterator;
 import java.util.List;
 import java.util.Map;
@@ -70,6 +71,7 @@ import static 
org.apache.atlas.repository.Constants.INDEX_SEARCH_VERTEX_PREFIX_D
 import static 
org.apache.atlas.repository.Constants.INDEX_SEARCH_VERTEX_PREFIX_PROPERTY;
 import static 
org.apache.atlas.repository.Constants.PROPAGATED_CLASSIFICATION_NAMES_KEY;
 import static org.apache.atlas.repository.Constants.STATE_PROPERTY_KEY;
+import static org.apache.atlas.repository.Constants.SUPER_TYPES_PROPERTY_KEY;
 import static org.apache.atlas.repository.Constants.TYPENAME_PROPERTY_KEY;
 import static org.apache.atlas.repository.Constants.TYPE_NAME_PROPERTY_KEY;
 import static 
org.apache.atlas.repository.graph.AtlasGraphProvider.getGraphInstance;
@@ -316,48 +318,29 @@ public class AtlasGraphUtilsV2 {
 }
 
 public static AtlasVertex findByUniqueAttributes(AtlasGraph graph, 
AtlasEntityType entityType, Map attrValues) {
-MetricRecorder metric = 
RequestContext.get().startMetricRecord("findByUniqueAttributes");
-
-AtlasVertex vertex = null;
-
+MetricRecordermetric   = 
RequestContext.get().startMetricRecord("findByUniqueAttributes");
+AtlasVertex   vertex   = null;
 final Map uniqueAttributes = 
entityType.getUniqAttributes();
 
 if (MapUtils.isNotEmpty(uniqueAttributes) && 
MapUtils.isNotEmpty(attrValues)) {
-for (AtlasAttribute attribute : uniqueAttributes.values()) {
-Object attrValue = attrValues.get(attribute.getName());
+Map uniqAttrValues = 
populateUniqueAttributesMap(uniqueAttributes, attrValues);
+Map attrNameValues = 
populateAttributesMap(uniqueAttributes, attrValues);
+String  typeName   = entityType.getTypeName();
+Set entitySubTypes = entityType.getAllSubTypes();
 
-if (attrValue == null) {
-continue;
-}
-
-if (canUseIndexQuery(graph, entityType, attribute.getName())) {
-vertex = 
AtlasGraphUtilsV2.getAtlasVertexFromIndexQuery(graph, entityType, attribute, 
attrValue);
-} else {
-if (USE_UNIQUE_INDEX_PROPERTY_TO_FIND_ENTITY && 
attribute.getVertexUniquePropertyName() != null) {
-vertex = 
AtlasGraphUtilsV2.findByTypeAndUniquePropertyName(graph, 
entityType.getTypeName(), attribute.getVertexUniquePropertyName(), attrValue);
-
-// if no instance of given typeName is found, try to 
find an instance of type's sub-type
-if (vertex == null && 
!entityType.getAllSubTypes().isEmpty()) {
-vertex = 
AtlasGraphUtilsV2.findBySuperTypeAndUniquePropertyName(graph, 
entityType.getTypeName(), attribute.getVertexUniquePropertyName(), attrValue);
-}
-} else {
-vertex = 
AtlasGraphUtilsV2.findByTypeAndPropertyName(graph, entityType.getTypeName(), 
attribute.getVertexPropertyName(), attrValue);
-
-// if no instance of given typeName is found, try to 
find an instance of type's sub-type
-if (vertex == null && 
!entityType.getAllSubTypes().isEmpty()) {
-vertex = 
AtlasGraphUtilsV2.findBySuperTypeAndPropertyName(graph, 
entityType.getType

[atlas] branch master updated: ATLAS-4403: Improve find entity by unique attributes logic - when multiple unique attributes exist for entity type

[sarath]

This is an automated email from the ASF dual-hosted git repository.

sarath pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/atlas.git


The following commit(s) were added to refs/heads/master by this push:
 new 097f50e  ATLAS-4403: Improve find entity by unique attributes logic - 
when multiple unique attributes exist for entity type
097f50e is described below

commit 097f50eae35e927411af3593fa401a335e61a0e3
Author: Sarath Subramanian 
AuthorDate: Fri Aug 27 14:11:52 2021 -0700

ATLAS-4403: Improve find entity by unique attributes logic - when multiple 
unique attributes exist for entity type
---
 .../store/graph/v2/AtlasGraphUtilsV2.java  | 201 -
 1 file changed, 80 insertions(+), 121 deletions(-)

diff --git 
a/repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasGraphUtilsV2.java
 
b/repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasGraphUtilsV2.java
index e73f084..2fce123 100644
--- 
a/repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasGraphUtilsV2.java
+++ 
b/repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasGraphUtilsV2.java
@@ -59,6 +59,7 @@ import java.util.ArrayList;
 import java.util.Arrays;
 import java.util.Collections;
 import java.util.Date;
+import java.util.HashMap;
 import java.util.Iterator;
 import java.util.List;
 import java.util.Map;
@@ -70,6 +71,7 @@ import static 
org.apache.atlas.repository.Constants.INDEX_SEARCH_VERTEX_PREFIX_D
 import static 
org.apache.atlas.repository.Constants.INDEX_SEARCH_VERTEX_PREFIX_PROPERTY;
 import static 
org.apache.atlas.repository.Constants.PROPAGATED_CLASSIFICATION_NAMES_KEY;
 import static org.apache.atlas.repository.Constants.STATE_PROPERTY_KEY;
+import static org.apache.atlas.repository.Constants.SUPER_TYPES_PROPERTY_KEY;
 import static org.apache.atlas.repository.Constants.TYPENAME_PROPERTY_KEY;
 import static org.apache.atlas.repository.Constants.TYPE_NAME_PROPERTY_KEY;
 import static 
org.apache.atlas.repository.graph.AtlasGraphProvider.getGraphInstance;
@@ -316,48 +318,29 @@ public class AtlasGraphUtilsV2 {
 }
 
 public static AtlasVertex findByUniqueAttributes(AtlasGraph graph, 
AtlasEntityType entityType, Map attrValues) {
-MetricRecorder metric = 
RequestContext.get().startMetricRecord("findByUniqueAttributes");
-
-AtlasVertex vertex = null;
-
+MetricRecordermetric   = 
RequestContext.get().startMetricRecord("findByUniqueAttributes");
+AtlasVertex   vertex   = null;
 final Map uniqueAttributes = 
entityType.getUniqAttributes();
 
 if (MapUtils.isNotEmpty(uniqueAttributes) && 
MapUtils.isNotEmpty(attrValues)) {
-for (AtlasAttribute attribute : uniqueAttributes.values()) {
-Object attrValue = attrValues.get(attribute.getName());
+Map uniqAttrValues = 
populateUniqueAttributesMap(uniqueAttributes, attrValues);
+Map attrNameValues = 
populateAttributesMap(uniqueAttributes, attrValues);
+String  typeName   = entityType.getTypeName();
+Set entitySubTypes = entityType.getAllSubTypes();
 
-if (attrValue == null) {
-continue;
-}
-
-if (canUseIndexQuery(graph, entityType, attribute.getName())) {
-vertex = 
AtlasGraphUtilsV2.getAtlasVertexFromIndexQuery(graph, entityType, attribute, 
attrValue);
-} else {
-if (USE_UNIQUE_INDEX_PROPERTY_TO_FIND_ENTITY && 
attribute.getVertexUniquePropertyName() != null) {
-vertex = 
AtlasGraphUtilsV2.findByTypeAndUniquePropertyName(graph, 
entityType.getTypeName(), attribute.getVertexUniquePropertyName(), attrValue);
-
-// if no instance of given typeName is found, try to 
find an instance of type's sub-type
-if (vertex == null && 
!entityType.getAllSubTypes().isEmpty()) {
-vertex = 
AtlasGraphUtilsV2.findBySuperTypeAndUniquePropertyName(graph, 
entityType.getTypeName(), attribute.getVertexUniquePropertyName(), attrValue);
-}
-} else {
-vertex = 
AtlasGraphUtilsV2.findByTypeAndPropertyName(graph, entityType.getTypeName(), 
attribute.getVertexPropertyName(), attrValue);
-
-// if no instance of given typeName is found, try to 
find an instance of type's sub-type
-if (vertex == null && 
!entityType.getAllSubTypes().isEmpty()) {
-vertex = 
AtlasGraphUtilsV2.findBySuperTypeAndPropertyName(graph, 
entityType.getTypeName(), attribute.getVertexPropertyName(), attrValue);
-}
-}
+if (USE_UNIQUE_I

[atlas] branch branch-2.0 updated: ATLAS-4398: Add additional path types to Hive DDL filter

[sarath]

This is an automated email from the ASF dual-hosted git repository.

sarath pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/atlas.git


The following commit(s) were added to refs/heads/branch-2.0 by this push:
 new 51485ec  ATLAS-4398: Add additional path types to Hive DDL filter
51485ec is described below

commit 51485ece56cc45d78862de77ad09a52aa72d2d0c
Author: Radhika Kundam 
AuthorDate: Fri Aug 20 09:19:52 2021 -0700

ATLAS-4398: Add additional path types to Hive DDL filter

Signed-off-by: Sarath Subramanian 
(cherry picked from commit b612a9458004ee6601a6ad58d7ec88f8b322a49e)
---
 .../java/org/apache/atlas/hive/hook/utils/HiveDDLEntityFilter.java| 4 
 1 file changed, 4 insertions(+)

diff --git 
a/addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/utils/HiveDDLEntityFilter.java
 
b/addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/utils/HiveDDLEntityFilter.java
index 25a5085..0f9aa45 100644
--- 
a/addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/utils/HiveDDLEntityFilter.java
+++ 
b/addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/utils/HiveDDLEntityFilter.java
@@ -43,8 +43,12 @@ public class HiveDDLEntityFilter implements EntityFilter {
 
 private static final Set defaultPathTypesToRetain = new 
HashSet() {{
 add(AtlasPathExtractorUtil.HDFS_TYPE_PATH);
+add(AtlasPathExtractorUtil.ADLS_GEN2_ACCOUNT);
+add(AtlasPathExtractorUtil.ADLS_GEN2_CONTAINER);
 add(AtlasPathExtractorUtil.ADLS_GEN2_DIRECTORY);
 add(AtlasPathExtractorUtil.GCS_VIRTUAL_DIR);
+add(AtlasPathExtractorUtil.AWS_S3_BUCKET);
+add(AtlasPathExtractorUtil.AWS_S3_V2_BUCKET);
 add(AtlasPathExtractorUtil.AWS_S3_V2_PSEUDO_DIR);
 add(AtlasPathExtractorUtil.AWS_S3_PSEUDO_DIR);
 add(AtlasPathExtractorUtil.OZONE_KEY);

[atlas] branch master updated: ATLAS-4398: Add additional path types to Hive DDL filter

[sarath]

This is an automated email from the ASF dual-hosted git repository.

sarath pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/atlas.git


The following commit(s) were added to refs/heads/master by this push:
 new b612a94  ATLAS-4398: Add additional path types to Hive DDL filter
b612a94 is described below

commit b612a9458004ee6601a6ad58d7ec88f8b322a49e
Author: Radhika Kundam 
AuthorDate: Fri Aug 20 09:19:52 2021 -0700

ATLAS-4398: Add additional path types to Hive DDL filter

Signed-off-by: Sarath Subramanian 
---
 .../java/org/apache/atlas/hive/hook/utils/HiveDDLEntityFilter.java| 4 
 1 file changed, 4 insertions(+)

diff --git 
a/addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/utils/HiveDDLEntityFilter.java
 
b/addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/utils/HiveDDLEntityFilter.java
index 25a5085..0f9aa45 100644
--- 
a/addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/utils/HiveDDLEntityFilter.java
+++ 
b/addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/utils/HiveDDLEntityFilter.java
@@ -43,8 +43,12 @@ public class HiveDDLEntityFilter implements EntityFilter {
 
 private static final Set defaultPathTypesToRetain = new 
HashSet() {{
 add(AtlasPathExtractorUtil.HDFS_TYPE_PATH);
+add(AtlasPathExtractorUtil.ADLS_GEN2_ACCOUNT);
+add(AtlasPathExtractorUtil.ADLS_GEN2_CONTAINER);
 add(AtlasPathExtractorUtil.ADLS_GEN2_DIRECTORY);
 add(AtlasPathExtractorUtil.GCS_VIRTUAL_DIR);
+add(AtlasPathExtractorUtil.AWS_S3_BUCKET);
+add(AtlasPathExtractorUtil.AWS_S3_V2_BUCKET);
 add(AtlasPathExtractorUtil.AWS_S3_V2_PSEUDO_DIR);
 add(AtlasPathExtractorUtil.AWS_S3_PSEUDO_DIR);
 add(AtlasPathExtractorUtil.OZONE_KEY);

[atlas] branch branch-2.0 updated: ATLAS-4340: Set Solr wait-searcher property to false by default to make Solr commits async

[sarath]

This is an automated email from the ASF dual-hosted git repository.

sarath pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/atlas.git


The following commit(s) were added to refs/heads/branch-2.0 by this push:
 new f08a407  ATLAS-4340: Set Solr wait-searcher property to false by 
default to make Solr commits async
f08a407 is described below

commit f08a4078cb30aaf0c34ea376f6c00bb2e3cac94c
Author: Sarath Subramanian 
AuthorDate: Tue Jul 27 14:59:48 2021 -0700

ATLAS-4340: Set Solr wait-searcher property to false by default to make 
Solr commits async

(cherry picked from commit a7ac64d54954064be6b0f56c04f05bbe128772d5)
---
 .../src/test/resources/atlas-application.properties|  1 +
 .../src/test/resources/atlas-application.properties|  1 +
 .../src/test/resources/atlas-application.properties|  1 +
 .../src/test/resources/atlas-application.properties|  3 +--
 distro/pom.xml |  8 
 distro/src/bin/atlas_config.py |  6 +++---
 distro/src/bin/atlas_start.py  | 18 +-
 distro/src/test/python/scripts/TestMetadata.py |  6 +++---
 .../src/test/resources/atlas-application.properties|  1 +
 .../java/org/apache/atlas/ApplicationProperties.java   | 14 --
 intg/src/test/resources/atlas-application.properties   |  3 +--
 webapp/src/test/resources/atlas-application.properties |  2 +-
 12 files changed, 38 insertions(+), 26 deletions(-)

diff --git 
a/addons/falcon-bridge/src/test/resources/atlas-application.properties 
b/addons/falcon-bridge/src/test/resources/atlas-application.properties
index 898b69c..3b12e5f 100644
--- a/addons/falcon-bridge/src/test/resources/atlas-application.properties
+++ b/addons/falcon-bridge/src/test/resources/atlas-application.properties
@@ -24,6 +24,7 @@ atlas.rest.address=http://localhost:31000
 
 # Graph database implementation.  Value inserted by maven.
 
atlas.graphdb.backend=org.apache.atlas.repository.graphdb.janus.AtlasJanusGraphDatabase
+atlas.graph.index.search.solr.wait-searcher=true
 
 # Graph Storage
 atlas.graph.storage.backend=berkeleyje
diff --git 
a/addons/hbase-bridge/src/test/resources/atlas-application.properties 
b/addons/hbase-bridge/src/test/resources/atlas-application.properties
index 898b69c..3b12e5f 100644
--- a/addons/hbase-bridge/src/test/resources/atlas-application.properties
+++ b/addons/hbase-bridge/src/test/resources/atlas-application.properties
@@ -24,6 +24,7 @@ atlas.rest.address=http://localhost:31000
 
 # Graph database implementation.  Value inserted by maven.
 
atlas.graphdb.backend=org.apache.atlas.repository.graphdb.janus.AtlasJanusGraphDatabase
+atlas.graph.index.search.solr.wait-searcher=true
 
 # Graph Storage
 atlas.graph.storage.backend=berkeleyje
diff --git 
a/addons/kafka-bridge/src/test/resources/atlas-application.properties 
b/addons/kafka-bridge/src/test/resources/atlas-application.properties
index 91fd8b0..4a12cf6 100644
--- a/addons/kafka-bridge/src/test/resources/atlas-application.properties
+++ b/addons/kafka-bridge/src/test/resources/atlas-application.properties
@@ -24,6 +24,7 @@ atlas.rest.address=http://localhost:31000
 
 # Graph database implementation.  Value inserted by maven.
 
atlas.graphdb.backend=org.apache.atlas.repository.graphdb.janus.AtlasJanusGraphDatabase
+atlas.graph.index.search.solr.wait-searcher=true
 
 # Graph Storage
 atlas.graph.storage.backend=berkeleyje
diff --git a/authorization/src/test/resources/atlas-application.properties 
b/authorization/src/test/resources/atlas-application.properties
index 2e02678..d735900 100644
--- a/authorization/src/test/resources/atlas-application.properties
+++ b/authorization/src/test/resources/atlas-application.properties
@@ -18,8 +18,7 @@
 
 #system property
 atlas.data=${sys:user.dir}/target/data
-
-
+atlas.graph.index.search.solr.wait-searcher=true
 
 #re-use existing property
 atlas.graph.data=${atlas.data}/graph
diff --git a/distro/pom.xml b/distro/pom.xml
index d71c436..33684df 100644
--- a/distro/pom.xml
+++ b/distro/pom.xml
@@ -41,7 +41,7 @@ atlas.graph.index.search.solr.mode=cloud
 atlas.graph.index.search.solr.zookeeper-url=
 atlas.graph.index.search.solr.zookeeper-connect-timeout=6
 atlas.graph.index.search.solr.zookeeper-session-timeout=6
-atlas.graph.index.search.solr.wait-searcher=true
+atlas.graph.index.search.solr.wait-searcher=false
 
 #Solr http mode properties
 #atlas.graph.index.search.solr.mode=http
@@ -162,7 +162,7 @@ atlas.graph.index.search.solr.mode=cloud
 atlas.graph.index.search.solr.zookeeper-url=localhost:2181
 atlas.graph.index.search.solr.zookeeper-connect-timeout=6
 atlas.graph.index.search.solr.zookeeper-session-timeout=6
-atlas.graph.index.search.solr.wait-searcher=true
+atlas.graph.index.search.solr.wait-searcher=false
 
 #Solr http mode properties
 #atlas.graph.index.search.solr.mode=http
@@ -321,7 +321,7

[atlas] branch master updated: ATLAS-4340: Set Solr wait-searcher property to false by default to make Solr commits async

[sarath]

This is an automated email from the ASF dual-hosted git repository.

sarath pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/atlas.git


The following commit(s) were added to refs/heads/master by this push:
 new a7ac64d  ATLAS-4340: Set Solr wait-searcher property to false by 
default to make Solr commits async
a7ac64d is described below

commit a7ac64d54954064be6b0f56c04f05bbe128772d5
Author: Sarath Subramanian 
AuthorDate: Tue Jul 27 14:59:48 2021 -0700

ATLAS-4340: Set Solr wait-searcher property to false by default to make 
Solr commits async
---
 .../src/test/resources/atlas-application.properties|  1 +
 .../src/test/resources/atlas-application.properties|  1 +
 .../src/test/resources/atlas-application.properties|  1 +
 .../src/test/resources/atlas-application.properties|  3 +--
 distro/pom.xml |  8 
 distro/src/bin/atlas_config.py |  6 +++---
 distro/src/bin/atlas_start.py  | 18 +-
 distro/src/test/python/scripts/TestMetadata.py |  6 +++---
 .../src/test/resources/atlas-application.properties|  1 +
 .../java/org/apache/atlas/ApplicationProperties.java   | 14 --
 intg/src/test/resources/atlas-application.properties   |  3 +--
 webapp/src/test/resources/atlas-application.properties |  2 +-
 12 files changed, 38 insertions(+), 26 deletions(-)

diff --git 
a/addons/falcon-bridge/src/test/resources/atlas-application.properties 
b/addons/falcon-bridge/src/test/resources/atlas-application.properties
index 898b69c..3b12e5f 100644
--- a/addons/falcon-bridge/src/test/resources/atlas-application.properties
+++ b/addons/falcon-bridge/src/test/resources/atlas-application.properties
@@ -24,6 +24,7 @@ atlas.rest.address=http://localhost:31000
 
 # Graph database implementation.  Value inserted by maven.
 
atlas.graphdb.backend=org.apache.atlas.repository.graphdb.janus.AtlasJanusGraphDatabase
+atlas.graph.index.search.solr.wait-searcher=true
 
 # Graph Storage
 atlas.graph.storage.backend=berkeleyje
diff --git 
a/addons/hbase-bridge/src/test/resources/atlas-application.properties 
b/addons/hbase-bridge/src/test/resources/atlas-application.properties
index 898b69c..3b12e5f 100644
--- a/addons/hbase-bridge/src/test/resources/atlas-application.properties
+++ b/addons/hbase-bridge/src/test/resources/atlas-application.properties
@@ -24,6 +24,7 @@ atlas.rest.address=http://localhost:31000
 
 # Graph database implementation.  Value inserted by maven.
 
atlas.graphdb.backend=org.apache.atlas.repository.graphdb.janus.AtlasJanusGraphDatabase
+atlas.graph.index.search.solr.wait-searcher=true
 
 # Graph Storage
 atlas.graph.storage.backend=berkeleyje
diff --git 
a/addons/kafka-bridge/src/test/resources/atlas-application.properties 
b/addons/kafka-bridge/src/test/resources/atlas-application.properties
index 91fd8b0..4a12cf6 100644
--- a/addons/kafka-bridge/src/test/resources/atlas-application.properties
+++ b/addons/kafka-bridge/src/test/resources/atlas-application.properties
@@ -24,6 +24,7 @@ atlas.rest.address=http://localhost:31000
 
 # Graph database implementation.  Value inserted by maven.
 
atlas.graphdb.backend=org.apache.atlas.repository.graphdb.janus.AtlasJanusGraphDatabase
+atlas.graph.index.search.solr.wait-searcher=true
 
 # Graph Storage
 atlas.graph.storage.backend=berkeleyje
diff --git a/authorization/src/test/resources/atlas-application.properties 
b/authorization/src/test/resources/atlas-application.properties
index 2e02678..d735900 100644
--- a/authorization/src/test/resources/atlas-application.properties
+++ b/authorization/src/test/resources/atlas-application.properties
@@ -18,8 +18,7 @@
 
 #system property
 atlas.data=${sys:user.dir}/target/data
-
-
+atlas.graph.index.search.solr.wait-searcher=true
 
 #re-use existing property
 atlas.graph.data=${atlas.data}/graph
diff --git a/distro/pom.xml b/distro/pom.xml
index d84f5e7..078bb2e 100644
--- a/distro/pom.xml
+++ b/distro/pom.xml
@@ -41,7 +41,7 @@ atlas.graph.index.search.solr.mode=cloud
 atlas.graph.index.search.solr.zookeeper-url=
 atlas.graph.index.search.solr.zookeeper-connect-timeout=6
 atlas.graph.index.search.solr.zookeeper-session-timeout=6
-atlas.graph.index.search.solr.wait-searcher=true
+atlas.graph.index.search.solr.wait-searcher=false
 
 #Solr http mode properties
 #atlas.graph.index.search.solr.mode=http
@@ -162,7 +162,7 @@ atlas.graph.index.search.solr.mode=cloud
 atlas.graph.index.search.solr.zookeeper-url=localhost:2181
 atlas.graph.index.search.solr.zookeeper-connect-timeout=6
 atlas.graph.index.search.solr.zookeeper-session-timeout=6
-atlas.graph.index.search.solr.wait-searcher=true
+atlas.graph.index.search.solr.wait-searcher=false
 
 #Solr http mode properties
 #atlas.graph.index.search.solr.mode=http
@@ -321,7 +321,7 @@ atlas.graph.index.search.solr.mode=cloud
 atlas.graph.index.search.solr.zookeeper-url=localhost:2181

[atlas] branch branch-2.0 updated: ATLAS-4324: FS entity created for load data inpath is created as shell entity - adding ozone_key

[sarath]

This is an automated email from the ASF dual-hosted git repository.

sarath pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/atlas.git


The following commit(s) were added to refs/heads/branch-2.0 by this push:
 new 0cc6ae8  ATLAS-4324: FS entity created for load data inpath is created 
as shell entity - adding ozone_key
0cc6ae8 is described below

commit 0cc6ae84a759886486ef2446a87aa4f63d11d6db
Author: Radhika Kundam 
AuthorDate: Tue Jul 27 09:12:06 2021 -0700

ATLAS-4324: FS entity created for load data inpath is created as shell 
entity - adding ozone_key

Signed-off-by: Sarath Subramanian 
(cherry picked from commit 98c6cea0140d64761542e603532d13127a0524b4)
---
 .../java/org/apache/atlas/hive/hook/utils/HiveDDLEntityFilter.java | 3 +++
 1 file changed, 3 insertions(+)

diff --git 
a/addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/utils/HiveDDLEntityFilter.java
 
b/addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/utils/HiveDDLEntityFilter.java
index 3ce3942..25a5085 100644
--- 
a/addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/utils/HiveDDLEntityFilter.java
+++ 
b/addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/utils/HiveDDLEntityFilter.java
@@ -47,6 +47,9 @@ public class HiveDDLEntityFilter implements EntityFilter {
 add(AtlasPathExtractorUtil.GCS_VIRTUAL_DIR);
 add(AtlasPathExtractorUtil.AWS_S3_V2_PSEUDO_DIR);
 add(AtlasPathExtractorUtil.AWS_S3_PSEUDO_DIR);
+add(AtlasPathExtractorUtil.OZONE_KEY);
+add(AtlasPathExtractorUtil.OZONE_BUCKET);
+add(AtlasPathExtractorUtil.OZONE_VOLUME);
 }};
 
 private static final Set typesToRetain = new HashSet() {{

[atlas] branch master updated: ATLAS-4324: FS entity created for load data inpath is created as shell entity - adding ozone_key

[sarath]

This is an automated email from the ASF dual-hosted git repository.

sarath pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/atlas.git


The following commit(s) were added to refs/heads/master by this push:
 new 98c6cea  ATLAS-4324: FS entity created for load data inpath is created 
as shell entity - adding ozone_key
98c6cea is described below

commit 98c6cea0140d64761542e603532d13127a0524b4
Author: Radhika Kundam 
AuthorDate: Tue Jul 27 09:12:06 2021 -0700

ATLAS-4324: FS entity created for load data inpath is created as shell 
entity - adding ozone_key

Signed-off-by: Sarath Subramanian 
---
 .../java/org/apache/atlas/hive/hook/utils/HiveDDLEntityFilter.java | 3 +++
 1 file changed, 3 insertions(+)

diff --git 
a/addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/utils/HiveDDLEntityFilter.java
 
b/addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/utils/HiveDDLEntityFilter.java
index 3ce3942..25a5085 100644
--- 
a/addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/utils/HiveDDLEntityFilter.java
+++ 
b/addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/utils/HiveDDLEntityFilter.java
@@ -47,6 +47,9 @@ public class HiveDDLEntityFilter implements EntityFilter {
 add(AtlasPathExtractorUtil.GCS_VIRTUAL_DIR);
 add(AtlasPathExtractorUtil.AWS_S3_V2_PSEUDO_DIR);
 add(AtlasPathExtractorUtil.AWS_S3_PSEUDO_DIR);
+add(AtlasPathExtractorUtil.OZONE_KEY);
+add(AtlasPathExtractorUtil.OZONE_BUCKET);
+add(AtlasPathExtractorUtil.OZONE_VOLUME);
 }};
 
 private static final Set typesToRetain = new HashSet() {{

[atlas] branch branch-2.0 updated: ATLAS-4364: Update Java Patch Id of ProcessNamePatch

[sarath]

This is an automated email from the ASF dual-hosted git repository.

sarath pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/atlas.git


The following commit(s) were added to refs/heads/branch-2.0 by this push:
 new eac8ca2  ATLAS-4364: Update Java Patch Id of ProcessNamePatch
eac8ca2 is described below

commit eac8ca2dfec8873f263eb1861b06eb3bcd64db50
Author: Radhika Kundam 
AuthorDate: Tue Jul 20 15:09:44 2021 -0700

ATLAS-4364: Update Java Patch Id of ProcessNamePatch

Signed-off-by: Sarath Subramanian 
(cherry picked from commit 217067210012798f88067582efbf8432474758ca)
---
 .../main/java/org/apache/atlas/repository/patches/ProcessNamePatch.java | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git 
a/repository/src/main/java/org/apache/atlas/repository/patches/ProcessNamePatch.java
 
b/repository/src/main/java/org/apache/atlas/repository/patches/ProcessNamePatch.java
index 2efb747..9889d46 100644
--- 
a/repository/src/main/java/org/apache/atlas/repository/patches/ProcessNamePatch.java
+++ 
b/repository/src/main/java/org/apache/atlas/repository/patches/ProcessNamePatch.java
@@ -48,7 +48,7 @@ import static 
org.apache.atlas.repository.store.graph.v2.AtlasGraphUtilsV2.getId
 public class ProcessNamePatch extends AtlasPatchHandler {
 private static final Logger LOG = 
LoggerFactory.getLogger(ProcessNamePatch.class);
 
-private static final String PATCH_ID  = "JAVA_PATCH__007";
+private static final String PATCH_ID  = "JAVA_PATCH__009";
 private static final String PATCH_DESCRIPTION = "Set name to 
qualifiedName.";
 
 private final PatchContext context;

[atlas] branch master updated: ATLAS-4364: Update Java Patch Id of ProcessNamePatch

[sarath]

This is an automated email from the ASF dual-hosted git repository.

sarath pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/atlas.git


The following commit(s) were added to refs/heads/master by this push:
 new 2170672  ATLAS-4364: Update Java Patch Id of ProcessNamePatch
2170672 is described below

commit 217067210012798f88067582efbf8432474758ca
Author: Radhika Kundam 
AuthorDate: Tue Jul 20 15:09:44 2021 -0700

ATLAS-4364: Update Java Patch Id of ProcessNamePatch

Signed-off-by: Sarath Subramanian 
---
 .../main/java/org/apache/atlas/repository/patches/ProcessNamePatch.java | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git 
a/repository/src/main/java/org/apache/atlas/repository/patches/ProcessNamePatch.java
 
b/repository/src/main/java/org/apache/atlas/repository/patches/ProcessNamePatch.java
index 2efb747..9889d46 100644
--- 
a/repository/src/main/java/org/apache/atlas/repository/patches/ProcessNamePatch.java
+++ 
b/repository/src/main/java/org/apache/atlas/repository/patches/ProcessNamePatch.java
@@ -48,7 +48,7 @@ import static 
org.apache.atlas.repository.store.graph.v2.AtlasGraphUtilsV2.getId
 public class ProcessNamePatch extends AtlasPatchHandler {
 private static final Logger LOG = 
LoggerFactory.getLogger(ProcessNamePatch.class);
 
-private static final String PATCH_ID  = "JAVA_PATCH__007";
+private static final String PATCH_ID  = "JAVA_PATCH__009";
 private static final String PATCH_DESCRIPTION = "Set name to 
qualifiedName.";
 
 private final PatchContext context;

[atlas] branch branch-2.0 updated: ATLAS-4348: Atlas-Kafka Hook : When a producer publishes messages to multiple topics, the latest relationship is marked ACTIVE , rest are marked DELETED

[sarath]

This is an automated email from the ASF dual-hosted git repository.

sarath pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/atlas.git


The following commit(s) were added to refs/heads/branch-2.0 by this push:
 new 8fb29d3  ATLAS-4348: Atlas-Kafka Hook : When a producer publishes 
messages to multiple topics, the latest relationship is marked ACTIVE , rest 
are marked DELETED
8fb29d3 is described below

commit 8fb29d30baf9328687916cb6cdf4981a7915d805
Author: Radhika Kundam 
AuthorDate: Wed Jul 14 22:12:31 2021 -0700

ATLAS-4348: Atlas-Kafka Hook : When a producer publishes messages to 
multiple topics, the latest relationship is marked ACTIVE , rest are marked 
DELETED

Signed-off-by: Sarath Subramanian 
(cherry picked from commit 8725f31af9821907ced99a829b9e0dbbf3f6de2b)
---
 .../019-kafka_producer_lineage_add_options.json| 15 +++
 .../org/apache/atlas/type/AtlasEntityType.java | 28 ++---
 .../org/apache/atlas/TestRelationshipUtilsV2.java  |  2 +
 .../store/graph/v2/EntityGraphMapper.java  | 49 +++---
 .../store/graph/v2/AtlasEntityTestBase.java|  2 +-
 .../graph/v2/AtlasRelationshipStoreV2Test.java | 33 +++
 6 files changed, 117 insertions(+), 12 deletions(-)

diff --git 
a/addons/models/1000-Hadoop/patches/019-kafka_producer_lineage_add_options.json 
b/addons/models/1000-Hadoop/patches/019-kafka_producer_lineage_add_options.json
new file mode 100644
index 000..bff4bd5
--- /dev/null
+++ 
b/addons/models/1000-Hadoop/patches/019-kafka_producer_lineage_add_options.json
@@ -0,0 +1,15 @@
+{
+  "patches": [
+{
+  "id": "TYPEDEF_PATCH_1000_019_001",
+  "description": "Add 'appendRelationshipsOnPartialUpdate' typeDefOptions 
to kafka_producer_lineage",
+  "action": "UPDATE_TYPEDEF_OPTIONS",
+  "typeName": "kafka_producer_lineage",
+  "applyToVersion": "1.0",
+  "updateToVersion": "1.1",
+  "typeDefOptions": {
+"appendRelationshipsOnPartialUpdate": "[\"outputs\"]"
+  }
+}
+  ]
+}
\ No newline at end of file
diff --git a/intg/src/main/java/org/apache/atlas/type/AtlasEntityType.java 
b/intg/src/main/java/org/apache/atlas/type/AtlasEntityType.java
index ded6d63..6d83599 100644
--- a/intg/src/main/java/org/apache/atlas/type/AtlasEntityType.java
+++ b/intg/src/main/java/org/apache/atlas/type/AtlasEntityType.java
@@ -65,11 +65,12 @@ public class AtlasEntityType extends AtlasStructType {
 private static final String OPTION_SCHEMA_ATTRIBUTES = 
"schemaAttributes";
 private static final String INTERNAL_TYPENAME= "__internal";
 
-private static final char   NS_ATTRIBUTE_NAME_SEPARATOR  = '.';
+public static final String OPTION_APPEND_RELATIONSHIPS_ON_PARTIAL_UPDATE = 
"appendRelationshipsOnPartialUpdate";
 
-private static final char DYN_ATTRIBUTE_NAME_SEPARATOR   = '.';
-private static final char DYN_ATTRIBUTE_OPEN_DELIM   = '{';
-private static final char DYN_ATTRIBUTE_CLOSE_DELIM  = '}';
+private static final char NS_ATTRIBUTE_NAME_SEPARATOR  = '.';
+private static final char DYN_ATTRIBUTE_NAME_SEPARATOR = '.';
+private static final char DYN_ATTRIBUTE_OPEN_DELIM = '{';
+private static final char DYN_ATTRIBUTE_CLOSE_DELIM= '}';
 
 private static final String[] ENTITY_HEADER_ATTRIBUTES = new 
String[]{NAME, DESCRIPTION, OWNER, CREATE_TIME};
 private static final String   ENTITY_ROOT_NAME = "__ENTITY_ROOT";
@@ -326,7 +327,11 @@ public class AtlasEntityType extends AtlasStructType {
 String relationshipType = relationsEntry.getKey();
 AtlasAttribute relationshipAttr = relationsEntry.getValue();
 
-relationshipAttrDefs.add(new 
AtlasRelationshipAttributeDef(relationshipType, 
relationshipAttr.isLegacyAttribute(), relationshipAttr.getAttributeDef()));
+AtlasRelationshipAttributeDef relationshipAttributeDef = new 
AtlasRelationshipAttributeDef(relationshipType, 
relationshipAttr.isLegacyAttribute(), relationshipAttr.getAttributeDef());
+
+
updateRelationshipAttrDefForPartialUpdate(relationshipAttributeDef, entityDef);
+
+relationshipAttrDefs.add(relationshipAttributeDef);
 }
 }
 
@@ -369,6 +374,19 @@ public class AtlasEntityType extends AtlasStructType {
 }
 }
 
+private void 
updateRelationshipAttrDefForPartialUpdate(AtlasRelationshipAttributeDef 
relationshipAttributeDef, AtlasEntityDef entityDef) {
+String appendRelationshipsOnPartialUpdate = 
entityDef.getOption(OPTION_APPEND_RELATIONSHIPS_ON_PARTIAL_UPDATE);
+String relationshipAttributeName  = 
relationshipA

[atlas] branch master updated: ATLAS-4348: Atlas-Kafka Hook : When a producer publishes messages to multiple topics, the latest relationship is marked ACTIVE , rest are marked DELETED

[sarath]

This is an automated email from the ASF dual-hosted git repository.

sarath pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/atlas.git


The following commit(s) were added to refs/heads/master by this push:
 new 8725f31  ATLAS-4348: Atlas-Kafka Hook : When a producer publishes 
messages to multiple topics, the latest relationship is marked ACTIVE , rest 
are marked DELETED
8725f31 is described below

commit 8725f31af9821907ced99a829b9e0dbbf3f6de2b
Author: Radhika Kundam 
AuthorDate: Wed Jul 14 22:12:31 2021 -0700

ATLAS-4348: Atlas-Kafka Hook : When a producer publishes messages to 
multiple topics, the latest relationship is marked ACTIVE , rest are marked 
DELETED

Signed-off-by: Sarath Subramanian 
---
 .../019-kafka_producer_lineage_add_options.json| 15 +++
 .../org/apache/atlas/type/AtlasEntityType.java | 28 ++---
 .../org/apache/atlas/TestRelationshipUtilsV2.java  |  2 +
 .../store/graph/v2/EntityGraphMapper.java  | 49 +++---
 .../store/graph/v2/AtlasEntityTestBase.java|  2 +-
 .../graph/v2/AtlasRelationshipStoreV2Test.java | 33 +++
 6 files changed, 117 insertions(+), 12 deletions(-)

diff --git 
a/addons/models/1000-Hadoop/patches/019-kafka_producer_lineage_add_options.json 
b/addons/models/1000-Hadoop/patches/019-kafka_producer_lineage_add_options.json
new file mode 100644
index 000..bff4bd5
--- /dev/null
+++ 
b/addons/models/1000-Hadoop/patches/019-kafka_producer_lineage_add_options.json
@@ -0,0 +1,15 @@
+{
+  "patches": [
+{
+  "id": "TYPEDEF_PATCH_1000_019_001",
+  "description": "Add 'appendRelationshipsOnPartialUpdate' typeDefOptions 
to kafka_producer_lineage",
+  "action": "UPDATE_TYPEDEF_OPTIONS",
+  "typeName": "kafka_producer_lineage",
+  "applyToVersion": "1.0",
+  "updateToVersion": "1.1",
+  "typeDefOptions": {
+"appendRelationshipsOnPartialUpdate": "[\"outputs\"]"
+  }
+}
+  ]
+}
\ No newline at end of file
diff --git a/intg/src/main/java/org/apache/atlas/type/AtlasEntityType.java 
b/intg/src/main/java/org/apache/atlas/type/AtlasEntityType.java
index ded6d63..6d83599 100644
--- a/intg/src/main/java/org/apache/atlas/type/AtlasEntityType.java
+++ b/intg/src/main/java/org/apache/atlas/type/AtlasEntityType.java
@@ -65,11 +65,12 @@ public class AtlasEntityType extends AtlasStructType {
 private static final String OPTION_SCHEMA_ATTRIBUTES = 
"schemaAttributes";
 private static final String INTERNAL_TYPENAME= "__internal";
 
-private static final char   NS_ATTRIBUTE_NAME_SEPARATOR  = '.';
+public static final String OPTION_APPEND_RELATIONSHIPS_ON_PARTIAL_UPDATE = 
"appendRelationshipsOnPartialUpdate";
 
-private static final char DYN_ATTRIBUTE_NAME_SEPARATOR   = '.';
-private static final char DYN_ATTRIBUTE_OPEN_DELIM   = '{';
-private static final char DYN_ATTRIBUTE_CLOSE_DELIM  = '}';
+private static final char NS_ATTRIBUTE_NAME_SEPARATOR  = '.';
+private static final char DYN_ATTRIBUTE_NAME_SEPARATOR = '.';
+private static final char DYN_ATTRIBUTE_OPEN_DELIM = '{';
+private static final char DYN_ATTRIBUTE_CLOSE_DELIM= '}';
 
 private static final String[] ENTITY_HEADER_ATTRIBUTES = new 
String[]{NAME, DESCRIPTION, OWNER, CREATE_TIME};
 private static final String   ENTITY_ROOT_NAME = "__ENTITY_ROOT";
@@ -326,7 +327,11 @@ public class AtlasEntityType extends AtlasStructType {
 String relationshipType = relationsEntry.getKey();
 AtlasAttribute relationshipAttr = relationsEntry.getValue();
 
-relationshipAttrDefs.add(new 
AtlasRelationshipAttributeDef(relationshipType, 
relationshipAttr.isLegacyAttribute(), relationshipAttr.getAttributeDef()));
+AtlasRelationshipAttributeDef relationshipAttributeDef = new 
AtlasRelationshipAttributeDef(relationshipType, 
relationshipAttr.isLegacyAttribute(), relationshipAttr.getAttributeDef());
+
+
updateRelationshipAttrDefForPartialUpdate(relationshipAttributeDef, entityDef);
+
+relationshipAttrDefs.add(relationshipAttributeDef);
 }
 }
 
@@ -369,6 +374,19 @@ public class AtlasEntityType extends AtlasStructType {
 }
 }
 
+private void 
updateRelationshipAttrDefForPartialUpdate(AtlasRelationshipAttributeDef 
relationshipAttributeDef, AtlasEntityDef entityDef) {
+String appendRelationshipsOnPartialUpdate = 
entityDef.getOption(OPTION_APPEND_RELATIONSHIPS_ON_PARTIAL_UPDATE);
+String relationshipAttributeName  = 
relationshipAttributeDef.getName();
+
+if (StringUtils.isNotEmpty(appendRelatio

[atlas] branch branch-2.0 updated: ATLAS-4339: Atlas should support skip temporary tables using config property

[sarath]

This is an automated email from the ASF dual-hosted git repository.

sarath pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/atlas.git


The following commit(s) were added to refs/heads/branch-2.0 by this push:
 new 756873e  ATLAS-4339: Atlas should support skip temporary tables using 
config property
756873e is described below

commit 756873eabdd5fa8dbc47682c045b0dff381d4e52
Author: Radhika Kundam 
AuthorDate: Wed Jun 23 14:18:28 2021 -0700

ATLAS-4339: Atlas should support skip temporary tables using config property

Signed-off-by: Sarath Subramanian 
(cherry picked from commit 77b03aa1407ec6d78cc0fcaf9a06654191d42d73)
---
 .../org/apache/atlas/hive/hook/AtlasHiveHookContext.java | 16 +++-
 .../main/java/org/apache/atlas/hive/hook/HiveHook.java   | 15 +++
 .../apache/atlas/hive/hook/HiveMetastoreHookImpl.java|  4 ++--
 .../org/apache/atlas/hive/hook/events/AlterTable.java|  2 +-
 .../org/apache/atlas/hive/hook/events/BaseHiveEvent.java |  4 +++-
 .../apache/atlas/hive/hook/events/CreateHiveProcess.java |  5 +
 .../org/apache/atlas/hive/hook/events/CreateTable.java   |  5 +
 7 files changed, 30 insertions(+), 21 deletions(-)

diff --git 
a/addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/AtlasHiveHookContext.java
 
b/addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/AtlasHiveHookContext.java
index ab8e183..14cc2f2 100644
--- 
a/addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/AtlasHiveHookContext.java
+++ 
b/addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/AtlasHiveHookContext.java
@@ -58,19 +58,20 @@ public class AtlasHiveHookContext {
 
 private boolean isSkippedInputEntity;
 private boolean isSkippedOutputEntity;
+private boolean skipTempTables;
 
 public AtlasHiveHookContext(HiveHook hook, HiveOperation hiveOperation, 
HookContext hiveContext,
-HiveHookObjectNamesCache knownObjects) throws 
Exception {
-this(hook, hiveOperation, hiveContext, knownObjects, null, null);
+HiveHookObjectNamesCache knownObjects, boolean 
skipTempTables) throws Exception {
+this(hook, hiveOperation, hiveContext, knownObjects, null, null, 
skipTempTables);
 }
 
 public AtlasHiveHookContext(HiveHook hook, HiveOperation hiveOperation, 
HiveHookObjectNamesCache knownObjects,
-HiveMetastoreHook metastoreHook, ListenerEvent 
listenerEvent) throws Exception {
-this(hook, hiveOperation, null, knownObjects, metastoreHook, 
listenerEvent);
+HiveMetastoreHook metastoreHook, ListenerEvent 
listenerEvent, boolean skipTempTables) throws Exception {
+this(hook, hiveOperation, null, knownObjects, metastoreHook, 
listenerEvent, skipTempTables);
 }
 
 public AtlasHiveHookContext(HiveHook hook, HiveOperation hiveOperation, 
HookContext hiveContext, HiveHookObjectNamesCache knownObjects,
-HiveMetastoreHook metastoreHook, ListenerEvent 
listenerEvent) throws Exception {
+HiveMetastoreHook metastoreHook, ListenerEvent 
listenerEvent, boolean skipTempTables) throws Exception {
 this.hook = hook;
 this.hiveOperation= hiveOperation;
 this.hiveContext  = hiveContext;
@@ -79,6 +80,7 @@ public class AtlasHiveHookContext {
 this.metastoreHook= metastoreHook;
 this.metastoreEvent   = listenerEvent;
 this.metastoreHandler = (listenerEvent != null) ? 
metastoreEvent.getIHMSHandler() : null;
+this.skipTempTables   = skipTempTables;
 
 init();
 }
@@ -131,6 +133,10 @@ public class AtlasHiveHookContext {
 }
 }
 
+public boolean isSkipTempTables() {
+return skipTempTables;
+}
+
 public LineageInfo getLineageInfo() {
 return hiveContext != null ? hiveContext.getLinfo() : null;
 }
diff --git 
a/addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
b/addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java
index bf500ca..2db970a 100644
--- a/addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java
+++ b/addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java
@@ -70,7 +70,8 @@ public class HiveHook extends AtlasHook implements 
ExecuteWithHookContext {
 public static final String HOOK_HIVE_TABLE_PRUNE_PATTERN   
  = CONF_PREFIX + "hive_table.prune.pattern";
 public static final String HOOK_HIVE_TABLE_CACHE_SIZE  
  = CONF_PREFIX + "hive_table.cache.size";
 public static final String HOOK_HIVE_IGNORE_DDL_OPERATIONS 
  = CONF_PREFIX + "hs2.ignore.ddl.operations";
-public static final String 
HOOK_HIVE_FILTER_ENTITY_ADDITIONAL_TYPES_T

[atlas] branch master updated: ATLAS-4339: Atlas should support skip temporary tables using config property

[sarath]

This is an automated email from the ASF dual-hosted git repository.

sarath pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/atlas.git


The following commit(s) were added to refs/heads/master by this push:
 new 77b03aa  ATLAS-4339: Atlas should support skip temporary tables using 
config property
77b03aa is described below

commit 77b03aa1407ec6d78cc0fcaf9a06654191d42d73
Author: Radhika Kundam 
AuthorDate: Wed Jun 23 14:18:28 2021 -0700

ATLAS-4339: Atlas should support skip temporary tables using config property

Signed-off-by: Sarath Subramanian 
---
 .../org/apache/atlas/hive/hook/AtlasHiveHookContext.java | 16 +++-
 .../main/java/org/apache/atlas/hive/hook/HiveHook.java   | 15 +++
 .../apache/atlas/hive/hook/HiveMetastoreHookImpl.java|  4 ++--
 .../org/apache/atlas/hive/hook/events/AlterTable.java|  2 +-
 .../org/apache/atlas/hive/hook/events/BaseHiveEvent.java |  4 +++-
 .../apache/atlas/hive/hook/events/CreateHiveProcess.java |  5 +
 .../org/apache/atlas/hive/hook/events/CreateTable.java   |  5 +
 7 files changed, 30 insertions(+), 21 deletions(-)

diff --git 
a/addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/AtlasHiveHookContext.java
 
b/addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/AtlasHiveHookContext.java
index ab8e183..14cc2f2 100644
--- 
a/addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/AtlasHiveHookContext.java
+++ 
b/addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/AtlasHiveHookContext.java
@@ -58,19 +58,20 @@ public class AtlasHiveHookContext {
 
 private boolean isSkippedInputEntity;
 private boolean isSkippedOutputEntity;
+private boolean skipTempTables;
 
 public AtlasHiveHookContext(HiveHook hook, HiveOperation hiveOperation, 
HookContext hiveContext,
-HiveHookObjectNamesCache knownObjects) throws 
Exception {
-this(hook, hiveOperation, hiveContext, knownObjects, null, null);
+HiveHookObjectNamesCache knownObjects, boolean 
skipTempTables) throws Exception {
+this(hook, hiveOperation, hiveContext, knownObjects, null, null, 
skipTempTables);
 }
 
 public AtlasHiveHookContext(HiveHook hook, HiveOperation hiveOperation, 
HiveHookObjectNamesCache knownObjects,
-HiveMetastoreHook metastoreHook, ListenerEvent 
listenerEvent) throws Exception {
-this(hook, hiveOperation, null, knownObjects, metastoreHook, 
listenerEvent);
+HiveMetastoreHook metastoreHook, ListenerEvent 
listenerEvent, boolean skipTempTables) throws Exception {
+this(hook, hiveOperation, null, knownObjects, metastoreHook, 
listenerEvent, skipTempTables);
 }
 
 public AtlasHiveHookContext(HiveHook hook, HiveOperation hiveOperation, 
HookContext hiveContext, HiveHookObjectNamesCache knownObjects,
-HiveMetastoreHook metastoreHook, ListenerEvent 
listenerEvent) throws Exception {
+HiveMetastoreHook metastoreHook, ListenerEvent 
listenerEvent, boolean skipTempTables) throws Exception {
 this.hook = hook;
 this.hiveOperation= hiveOperation;
 this.hiveContext  = hiveContext;
@@ -79,6 +80,7 @@ public class AtlasHiveHookContext {
 this.metastoreHook= metastoreHook;
 this.metastoreEvent   = listenerEvent;
 this.metastoreHandler = (listenerEvent != null) ? 
metastoreEvent.getIHMSHandler() : null;
+this.skipTempTables   = skipTempTables;
 
 init();
 }
@@ -131,6 +133,10 @@ public class AtlasHiveHookContext {
 }
 }
 
+public boolean isSkipTempTables() {
+return skipTempTables;
+}
+
 public LineageInfo getLineageInfo() {
 return hiveContext != null ? hiveContext.getLinfo() : null;
 }
diff --git 
a/addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
b/addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java
index 37084d5..3cc7b3b 100644
--- a/addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java
+++ b/addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java
@@ -70,7 +70,8 @@ public class HiveHook extends AtlasHook implements 
ExecuteWithHookContext {
 public static final String HOOK_HIVE_TABLE_PRUNE_PATTERN   
  = CONF_PREFIX + "hive_table.prune.pattern";
 public static final String HOOK_HIVE_TABLE_CACHE_SIZE  
  = CONF_PREFIX + "hive_table.cache.size";
 public static final String HOOK_HIVE_IGNORE_DDL_OPERATIONS 
  = CONF_PREFIX + "hs2.ignore.ddl.operations";
-public static final String 
HOOK_HIVE_FILTER_ENTITY_ADDITIONAL_TYPES_TO_RETAIN = CONF_PREFIX + 
"hs2.filter.entity.additional.types.to.retai

[atlas] branch branch-2.0 updated: ATLAS-4329: Update Kafka version to 2.5

[sarath]

This is an automated email from the ASF dual-hosted git repository.

sarath pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/atlas.git


The following commit(s) were added to refs/heads/branch-2.0 by this push:
 new 75ee653  ATLAS-4329: Update Kafka version to 2.5
75ee653 is described below

commit 75ee653ae47529b1b2f77d9c316a7c768c7ca5f3
Author: Sarath Subramanian 
AuthorDate: Mon Jun 7 10:31:10 2021 -0700

ATLAS-4329: Update Kafka version to 2.5

(cherry picked from commit 8db2ccc5ec05c1b2eaeae29bed7951d53fd87443)
---
 pom.xml | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/pom.xml b/pom.xml
index a52932f..0c20600 100644
--- a/pom.xml
+++ b/pom.xml
@@ -718,8 +718,8 @@
 3.2.11
 1.1
 4.13
-2.11
-2.0.0
+2.12
+2.5.0
 6.0.1
 1.2.17
 2.13.3

[atlas] branch master updated: ATLAS-4329: Update Kafka version to 2.5

[sarath]

This is an automated email from the ASF dual-hosted git repository.

sarath pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/atlas.git


The following commit(s) were added to refs/heads/master by this push:
 new 8db2ccc  ATLAS-4329: Update Kafka version to 2.5
8db2ccc is described below

commit 8db2ccc5ec05c1b2eaeae29bed7951d53fd87443
Author: Sarath Subramanian 
AuthorDate: Mon Jun 7 10:31:10 2021 -0700

ATLAS-4329: Update Kafka version to 2.5
---
 pom.xml | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/pom.xml b/pom.xml
index 91c23ee..91176e3 100644
--- a/pom.xml
+++ b/pom.xml
@@ -729,8 +729,8 @@
 3.2.11
 1.1
 4.13
-2.11
-2.0.0
+2.12
+2.5.0
 6.0.1
 1.2.17
 2.13.3

[atlas] branch branch-2.0 updated: ATLAS-4324: FS entity created for load data inpath is created as shell entity

[sarath]

This is an automated email from the ASF dual-hosted git repository.

sarath pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/atlas.git


The following commit(s) were added to refs/heads/branch-2.0 by this push:
 new 3c60783  ATLAS-4324: FS entity created for load data inpath is created 
as shell entity
3c60783 is described below

commit 3c607836173472dfa05e60ff102c50133e02b709
Author: Radhika Kundam 
AuthorDate: Thu Jun 3 00:10:12 2021 -0700

ATLAS-4324: FS entity created for load data inpath is created as shell 
entity

Signed-off-by: Sarath Subramanian 
(cherry picked from commit a9e528bca9277707a1847d110c33288188a8fff5)
---
 .../java/org/apache/atlas/hive/hook/HiveHook.java  |  1 +
 .../atlas/hive/hook/utils/HiveDDLEntityFilter.java | 47 +++---
 .../hive/hook/utils/ActiveEntityFilterTest.java| 34 +-
 .../src/test/resources/json/hs2-create-db-v2.json  | 24 +++
 ...eate-db-with-no-pathentities-to-retain-v2.json} |  0
 ...2-create-db-with-no-pathentities-to-retain.json | 73 ++
 .../test/resources/json/hs2-create-table-v2.json   | 17 -
 .../test/resources/json/hs2-load-inpath-v2.json| 18 ++
 ...d-inpath-with-no-pathentities-to-retain-v2.json |  4 ++
 ...load-inpath-with-no-pathentities-to-retain.json | 18 ++
 .../src/test/resources/json/hs2-load-inpath.json   | 18 ++
 11 files changed, 240 insertions(+), 14 deletions(-)

diff --git 
a/addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
b/addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java
index 5e6785a..9548eda 100644
--- a/addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java
+++ b/addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java
@@ -70,6 +70,7 @@ public class HiveHook extends AtlasHook implements 
ExecuteWithHookContext {
 public static final String HOOK_HIVE_TABLE_PRUNE_PATTERN   
  = CONF_PREFIX + "hive_table.prune.pattern";
 public static final String HOOK_HIVE_TABLE_CACHE_SIZE  
  = CONF_PREFIX + "hive_table.cache.size";
 public static final String HOOK_HIVE_IGNORE_DDL_OPERATIONS 
  = CONF_PREFIX + "hs2.ignore.ddl.operations";
+public static final String HOOK_HIVE_FILTER_ENTITY_TYPES_TO_RETAIN 
  = CONF_PREFIX + "hs2.filter.entity.types.to.retain";
 public static final String DEFAULT_HOST_NAME = "localhost";
 
 private static final Map OPERATION_MAP = new 
HashMap<>();
diff --git 
a/addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/utils/HiveDDLEntityFilter.java
 
b/addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/utils/HiveDDLEntityFilter.java
index 2b39e81..9163c47 100644
--- 
a/addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/utils/HiveDDLEntityFilter.java
+++ 
b/addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/utils/HiveDDLEntityFilter.java
@@ -18,15 +18,20 @@
 package org.apache.atlas.hive.hook.utils;
 
 import com.google.common.annotations.VisibleForTesting;
+import org.apache.atlas.ApplicationProperties;
+import org.apache.atlas.hive.hook.HiveHook;
 import org.apache.atlas.hive.hook.events.BaseHiveEvent;
 import org.apache.atlas.model.instance.AtlasEntity;
 import org.apache.atlas.model.instance.AtlasObjectId;
-import org.apache.atlas.model.instance.AtlasRelatedObjectId;
 import org.apache.atlas.model.notification.HookNotification;
+import org.apache.atlas.utils.AtlasPathExtractorUtil;
 import org.apache.commons.collections.CollectionUtils;
 import org.apache.commons.collections.MapUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
 
 import java.util.ArrayList;
+import java.util.Arrays;
 import java.util.HashSet;
 import java.util.LinkedHashMap;
 import java.util.List;
@@ -36,14 +41,38 @@ import java.util.function.Function;
 import java.util.stream.Collectors;
 
 public class HiveDDLEntityFilter implements EntityFilter {
+private static final Logger LOG = 
LoggerFactory.getLogger(HiveDDLEntityFilter.class);
+
+private static final Set defaultPathTypes = new HashSet() 
{{
+add(AtlasPathExtractorUtil.HDFS_TYPE_PATH);
+add(AtlasPathExtractorUtil.ADLS_GEN2_DIRECTORY);
+add(AtlasPathExtractorUtil.GCS_VIRTUAL_DIR);
+add(AtlasPathExtractorUtil.AWS_S3_V2_PSEUDO_DIR);
+add(AtlasPathExtractorUtil.AWS_S3_PSEUDO_DIR);
+}};
+
 private static final Set typesToRetain = new HashSet() {{
 add(BaseHiveEvent.HIVE_TYPE_PROCESS);
 add(BaseHiveEvent.HIVE_TYPE_PROCESS_EXECUTION);
 add(BaseHiveEvent.HIVE_TYPE_COLUMN_LINEAGE);
 add(BaseHiveEvent.HIVE_DB_DDL);
 add(BaseHiveEvent.HIVE_TABLE_DDL);
+addAll(defaultPathTypes);
+addAll(getConfiguredTypesToRetain());
 }};
 
+private static List getConfiguredTypesToRet

[atlas] branch master updated: ATLAS-4324: FS entity created for load data inpath is created as shell entity

[sarath]

This is an automated email from the ASF dual-hosted git repository.

sarath pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/atlas.git


The following commit(s) were added to refs/heads/master by this push:
 new a9e528b  ATLAS-4324: FS entity created for load data inpath is created 
as shell entity
a9e528b is described below

commit a9e528bca9277707a1847d110c33288188a8fff5
Author: Radhika Kundam 
AuthorDate: Thu Jun 3 00:10:12 2021 -0700

ATLAS-4324: FS entity created for load data inpath is created as shell 
entity

Signed-off-by: Sarath Subramanian 
---
 .../java/org/apache/atlas/hive/hook/HiveHook.java  |  1 +
 .../atlas/hive/hook/utils/HiveDDLEntityFilter.java | 47 +++---
 .../hive/hook/utils/ActiveEntityFilterTest.java| 34 +-
 .../src/test/resources/json/hs2-create-db-v2.json  | 24 +++
 ...eate-db-with-no-pathentities-to-retain-v2.json} |  0
 ...2-create-db-with-no-pathentities-to-retain.json | 73 ++
 .../test/resources/json/hs2-create-table-v2.json   | 17 -
 .../test/resources/json/hs2-load-inpath-v2.json| 18 ++
 ...d-inpath-with-no-pathentities-to-retain-v2.json |  4 ++
 ...load-inpath-with-no-pathentities-to-retain.json | 18 ++
 .../src/test/resources/json/hs2-load-inpath.json   | 18 ++
 11 files changed, 240 insertions(+), 14 deletions(-)

diff --git 
a/addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
b/addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java
index cf918ef..94ef225 100644
--- a/addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java
+++ b/addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java
@@ -70,6 +70,7 @@ public class HiveHook extends AtlasHook implements 
ExecuteWithHookContext {
 public static final String HOOK_HIVE_TABLE_PRUNE_PATTERN   
  = CONF_PREFIX + "hive_table.prune.pattern";
 public static final String HOOK_HIVE_TABLE_CACHE_SIZE  
  = CONF_PREFIX + "hive_table.cache.size";
 public static final String HOOK_HIVE_IGNORE_DDL_OPERATIONS 
  = CONF_PREFIX + "hs2.ignore.ddl.operations";
+public static final String HOOK_HIVE_FILTER_ENTITY_TYPES_TO_RETAIN 
  = CONF_PREFIX + "hs2.filter.entity.types.to.retain";
 public static final String DEFAULT_HOST_NAME = "localhost";
 
 private static final Map OPERATION_MAP = new 
HashMap<>();
diff --git 
a/addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/utils/HiveDDLEntityFilter.java
 
b/addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/utils/HiveDDLEntityFilter.java
index 2b39e81..9163c47 100644
--- 
a/addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/utils/HiveDDLEntityFilter.java
+++ 
b/addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/utils/HiveDDLEntityFilter.java
@@ -18,15 +18,20 @@
 package org.apache.atlas.hive.hook.utils;
 
 import com.google.common.annotations.VisibleForTesting;
+import org.apache.atlas.ApplicationProperties;
+import org.apache.atlas.hive.hook.HiveHook;
 import org.apache.atlas.hive.hook.events.BaseHiveEvent;
 import org.apache.atlas.model.instance.AtlasEntity;
 import org.apache.atlas.model.instance.AtlasObjectId;
-import org.apache.atlas.model.instance.AtlasRelatedObjectId;
 import org.apache.atlas.model.notification.HookNotification;
+import org.apache.atlas.utils.AtlasPathExtractorUtil;
 import org.apache.commons.collections.CollectionUtils;
 import org.apache.commons.collections.MapUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
 
 import java.util.ArrayList;
+import java.util.Arrays;
 import java.util.HashSet;
 import java.util.LinkedHashMap;
 import java.util.List;
@@ -36,14 +41,38 @@ import java.util.function.Function;
 import java.util.stream.Collectors;
 
 public class HiveDDLEntityFilter implements EntityFilter {
+private static final Logger LOG = 
LoggerFactory.getLogger(HiveDDLEntityFilter.class);
+
+private static final Set defaultPathTypes = new HashSet() 
{{
+add(AtlasPathExtractorUtil.HDFS_TYPE_PATH);
+add(AtlasPathExtractorUtil.ADLS_GEN2_DIRECTORY);
+add(AtlasPathExtractorUtil.GCS_VIRTUAL_DIR);
+add(AtlasPathExtractorUtil.AWS_S3_V2_PSEUDO_DIR);
+add(AtlasPathExtractorUtil.AWS_S3_PSEUDO_DIR);
+}};
+
 private static final Set typesToRetain = new HashSet() {{
 add(BaseHiveEvent.HIVE_TYPE_PROCESS);
 add(BaseHiveEvent.HIVE_TYPE_PROCESS_EXECUTION);
 add(BaseHiveEvent.HIVE_TYPE_COLUMN_LINEAGE);
 add(BaseHiveEvent.HIVE_DB_DDL);
 add(BaseHiveEvent.HIVE_TABLE_DDL);
+addAll(defaultPathTypes);
+addAll(getConfiguredTypesToRetain());
 }};
 
+private static List getConfiguredTypesToRetain() {
+String[]configuredTypesToRetain = nul

[atlas] branch branch-2.0 updated: ATLAS-4301 Handle Integration Test Failures on HiveHook module

[sarath]

This is an automated email from the ASF dual-hosted git repository.

sarath pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/atlas.git


The following commit(s) were added to refs/heads/branch-2.0 by this push:
 new d24ce80  ATLAS-4301 Handle Integration Test Failures on HiveHook module
d24ce80 is described below

commit d24ce80914594831a03aa18e5e8cc7d129590d88
Author: Mandar Ambawane 
AuthorDate: Thu Jun 3 14:35:22 2021 +0530

ATLAS-4301 Handle Integration Test Failures on HiveHook module

Signed-off-by: Sarath Subramanian 
(cherry picked from commit e993f85d7b2c5e11914a328f4b3aee4392c4e106)
---
 .../org/apache/atlas/hive/hook/events/BaseHiveEvent.java |  7 +--
 .../src/test/java/org/apache/atlas/hive/HiveITBase.java  |  2 +-
 .../src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java | 12 ++--
 3 files changed, 12 insertions(+), 9 deletions(-)

diff --git 
a/addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/BaseHiveEvent.java
 
b/addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/BaseHiveEvent.java
index a69d5ce..aefe9e9 100644
--- 
a/addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/BaseHiveEvent.java
+++ 
b/addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/BaseHiveEvent.java
@@ -233,7 +233,8 @@ public abstract class BaseHiveEvent {
 switch(entity.getType()) {
 case TABLE:
 case PARTITION:
-case DFS_DIR: {
+case DFS_DIR:
+case LOCAL_DIR: {
 ret = toAtlasEntity(entity, entityExtInfo, skipTempTables);
 }
 break;
@@ -281,7 +282,8 @@ public abstract class BaseHiveEvent {
 }
 break;
 
-case DFS_DIR: {
+case DFS_DIR:
+case LOCAL_DIR: {
 URI location = entity.getLocation();
 
 if (location != null) {
@@ -822,6 +824,7 @@ public abstract class BaseHiveEvent {
 return getQualifiedName(entity.getTable());
 
 case DFS_DIR:
+case LOCAL_DIR:
 return getQualifiedName(entity.getLocation());
 }
 
diff --git 
a/addons/hive-bridge/src/test/java/org/apache/atlas/hive/HiveITBase.java 
b/addons/hive-bridge/src/test/java/org/apache/atlas/hive/HiveITBase.java
index 7a11ec3..0875afa 100644
--- a/addons/hive-bridge/src/test/java/org/apache/atlas/hive/HiveITBase.java
+++ b/addons/hive-bridge/src/test/java/org/apache/atlas/hive/HiveITBase.java
@@ -175,7 +175,7 @@ public class HiveITBase {
 }
 
 protected String createTestDFSPath(String path) throws Exception {
-return "pfile://" + mkdir(path);
+return "file://" + mkdir(path);
 }
 
 protected String file(String tag) throws Exception {
diff --git 
a/addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 
b/addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java
index 03bbd60..eee5a75 100755
--- 
a/addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java
+++ 
b/addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java
@@ -286,7 +286,7 @@ public class HiveHookIT extends HiveITBase {
 private Set getInputs(String inputName, Entity.Type 
entityType) throws HiveException {
 final ReadEntity entity;
 
-if (Entity.Type.DFS_DIR.equals(entityType)) {
+if (Entity.Type.DFS_DIR.equals(entityType) || 
Entity.Type.LOCAL_DIR.equals(entityType)) {
 entity = new TestReadEntity(lower(new Path(inputName).toString()), 
entityType);
 } else {
 entity = new TestReadEntity(getQualifiedTblName(inputName), 
entityType);
@@ -606,7 +606,7 @@ public class HiveHookIT extends HiveITBase {
 Assert.assertNotNull(ddlQueries);
 Assert.assertEquals(ddlQueries.size(), 1);
 
-assertProcessIsRegistered(constructEvent(query, HiveOperation.LOAD, 
null, getOutputs(tableName, Entity.Type.TABLE)));
+assertProcessIsRegistered(constructEvent(query, HiveOperation.LOAD, 
getInputs("file://" + loadFile, Entity.Type.LOCAL_DIR), getOutputs(tableName, 
Entity.Type.TABLE)));
 }
 
 @Test
@@ -844,7 +844,7 @@ public class HiveHookIT extends HiveITBase {
 runCommand(query);
 
 HiveEventContext event = constructEvent(query,  HiveOperation.QUERY,
-getInputs(tableName, Entity.Type.TABLE), null);
+getInputs(tableName, Entity.Type.TABLE), 
getOutputs(randomLocalPath, Entity.Type.LOCAL_DIR));
 AtlasEntity hiveProcess = validateProcess(event);
 AtlasEntity hiveProcessExecution = 
validateProcessExecution(hiveProcess, event);
 AtlasObjectId process = 
toAtlasObjectId(hiveProcessExecution.getRelationshipAttribute(
@@ -1061,7 +1061,7 @@ public class HiveHookIT extends HiveITBase {
 
 String tblId = assertTableIsRe

[atlas] branch master updated: ATLAS-4301 Handle Integration Test Failures on HiveHook module

[sarath]

This is an automated email from the ASF dual-hosted git repository.

sarath pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/atlas.git


The following commit(s) were added to refs/heads/master by this push:
 new e993f85  ATLAS-4301 Handle Integration Test Failures on HiveHook module
e993f85 is described below

commit e993f85d7b2c5e11914a328f4b3aee4392c4e106
Author: Mandar Ambawane 
AuthorDate: Thu Jun 3 14:35:22 2021 +0530

ATLAS-4301 Handle Integration Test Failures on HiveHook module

Signed-off-by: Sarath Subramanian 
---
 .../org/apache/atlas/hive/hook/events/BaseHiveEvent.java |  7 +--
 .../src/test/java/org/apache/atlas/hive/HiveITBase.java  |  2 +-
 .../src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java | 12 ++--
 3 files changed, 12 insertions(+), 9 deletions(-)

diff --git 
a/addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/BaseHiveEvent.java
 
b/addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/BaseHiveEvent.java
index 21c0ae3..2e8237b 100644
--- 
a/addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/BaseHiveEvent.java
+++ 
b/addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/BaseHiveEvent.java
@@ -230,7 +230,8 @@ public abstract class BaseHiveEvent {
 switch(entity.getType()) {
 case TABLE:
 case PARTITION:
-case DFS_DIR: {
+case DFS_DIR:
+case LOCAL_DIR: {
 ret = toAtlasEntity(entity, entityExtInfo, skipTempTables);
 }
 break;
@@ -278,7 +279,8 @@ public abstract class BaseHiveEvent {
 }
 break;
 
-case DFS_DIR: {
+case DFS_DIR:
+case LOCAL_DIR: {
 URI location = entity.getLocation();
 
 if (location != null) {
@@ -819,6 +821,7 @@ public abstract class BaseHiveEvent {
 return getQualifiedName(entity.getTable());
 
 case DFS_DIR:
+case LOCAL_DIR:
 return getQualifiedName(entity.getLocation());
 }
 
diff --git 
a/addons/hive-bridge/src/test/java/org/apache/atlas/hive/HiveITBase.java 
b/addons/hive-bridge/src/test/java/org/apache/atlas/hive/HiveITBase.java
index 7a11ec3..0875afa 100644
--- a/addons/hive-bridge/src/test/java/org/apache/atlas/hive/HiveITBase.java
+++ b/addons/hive-bridge/src/test/java/org/apache/atlas/hive/HiveITBase.java
@@ -175,7 +175,7 @@ public class HiveITBase {
 }
 
 protected String createTestDFSPath(String path) throws Exception {
-return "pfile://" + mkdir(path);
+return "file://" + mkdir(path);
 }
 
 protected String file(String tag) throws Exception {
diff --git 
a/addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 
b/addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java
index 52684e7..1db73e5 100755
--- 
a/addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java
+++ 
b/addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java
@@ -282,7 +282,7 @@ public class HiveHookIT extends HiveITBase {
 private Set getInputs(String inputName, Entity.Type 
entityType) throws HiveException {
 final ReadEntity entity;
 
-if (Entity.Type.DFS_DIR.equals(entityType)) {
+if (Entity.Type.DFS_DIR.equals(entityType) || 
Entity.Type.LOCAL_DIR.equals(entityType)) {
 entity = new TestReadEntity(lower(new Path(inputName).toString()), 
entityType);
 } else {
 entity = new TestReadEntity(getQualifiedTblName(inputName), 
entityType);
@@ -602,7 +602,7 @@ public class HiveHookIT extends HiveITBase {
 Assert.assertNotNull(ddlQueries);
 Assert.assertEquals(ddlQueries.size(), 1);
 
-assertProcessIsRegistered(constructEvent(query, HiveOperation.LOAD, 
null, getOutputs(tableName, Entity.Type.TABLE)));
+assertProcessIsRegistered(constructEvent(query, HiveOperation.LOAD, 
getInputs("file://" + loadFile, Entity.Type.LOCAL_DIR), getOutputs(tableName, 
Entity.Type.TABLE)));
 }
 
 @Test
@@ -840,7 +840,7 @@ public class HiveHookIT extends HiveITBase {
 runCommand(query);
 
 HiveEventContext event = constructEvent(query,  HiveOperation.QUERY,
-getInputs(tableName, Entity.Type.TABLE), null);
+getInputs(tableName, Entity.Type.TABLE), 
getOutputs(randomLocalPath, Entity.Type.LOCAL_DIR));
 AtlasEntity hiveProcess = validateProcess(event);
 AtlasEntity hiveProcessExecution = 
validateProcessExecution(hiveProcess, event);
 AtlasObjectId process = 
toAtlasObjectId(hiveProcessExecution.getRelationshipAttribute(
@@ -1057,7 +1057,7 @@ public class HiveHookIT extends HiveITBase {
 
 String tblId = assertTableIsRegistered(DEFAULT_DB, tableName);
 
-String filename = "pfile://" + mkdir

[atlas] branch branch-2.0 updated: ATLAS-4285: Multiple propagations with intersecting lineage.

[sarath]

This is an automated email from the ASF dual-hosted git repository.

sarath pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/atlas.git


The following commit(s) were added to refs/heads/branch-2.0 by this push:
 new 3016ba6  ATLAS-4285: Multiple propagations with intersecting lineage.
3016ba6 is described below

commit 3016ba6fb16f2f4b598eefc48e97ac2c50f16a84
Author: Ashutosh Mestry 
AuthorDate: Thu May 13 17:28:26 2021 -0700

ATLAS-4285: Multiple propagations with intersecting lineage.

Signed-off-by: Sarath Subramanian 
(cherry picked from commit aba97b35393f1732eb30858e69fd5f489634afdc)
---
 .../store/graph/v2/EntityGraphMapper.java  | 79 +++---
 .../tasks/ClassificationPropagateTaskFactory.java  |  5 --
 .../v2/tasks/ClassificationPropagationTasks.java   | 15 
 .../apache/atlas/tasks/TaskFactoryRegistry.java| 23 ++-
 .../org/apache/atlas/tasks/TaskManagement.java | 11 +++
 .../ClassificationPropagationWithTasksTest.java|  6 --
 6 files changed, 43 insertions(+), 96 deletions(-)

diff --git 
a/repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java
 
b/repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java
index 7984a34..5baff33 100644
--- 
a/repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java
+++ 
b/repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java
@@ -2041,10 +2041,7 @@ public class EntityGraphMapper {
 return null;
 }
 
-
GraphTransactionInterceptor.lockObjectAndReleasePostCommit(entityGuid);
-
 AtlasVertex entityVertex = 
graphHelper.getVertexForGUID(entityGuid);
-
 if (entityVertex == null) {
 LOG.warn("propagateClassification(entityGuid={}, 
classificationVertexId={}): entity vertex not found", entityGuid, 
classificationVertexId);
 
@@ -2052,7 +2049,6 @@ public class EntityGraphMapper {
 }
 
 AtlasVertex classificationVertex = 
graph.getVertex(classificationVertexId);
-
 if (classificationVertex == null) {
 LOG.warn("propagateClassification(entityGuid={}, 
classificationVertexId={}): classification vertex not found", entityGuid, 
classificationVertexId);
 
@@ -2060,13 +2056,15 @@ public class EntityGraphMapper {
 }
 
 List impactedVertices = 
entityRetriever.getIncludedImpactedVerticesV2(entityVertex, relationshipGuid, 
classificationVertexId);
-
 if (CollectionUtils.isEmpty(impactedVertices)) {
 LOG.debug("propagateClassification(entityGuid={}, 
classificationVertexId={}): found no entities to propagate the classification", 
entityGuid, classificationVertexId);
 
 return null;
 }
 
+List impactedVerticesGuidsToLock = 
impactedVertices.stream().map(x -> 
GraphHelper.getGuid(x)).collect(Collectors.toList());
+
GraphTransactionInterceptor.lockObjectAndReleasePostCommit(impactedVerticesGuidsToLock);
+
 AtlasClassification classification   = 
entityRetriever.toAtlasClassification(classificationVertex);
 List   entitiesPropagatedTo = 
deleteDelegate.getHandler().addTagPropagation(classificationVertex, 
impactedVertices);
 
@@ -2453,63 +2451,6 @@ public class EntityGraphMapper {
 AtlasPerfTracer.log(perf);
 }
 
-@GraphTransaction
-public List updateClassificationsPropagation(String entityGuid, 
String classificationVertexId, String relationshipGuid) throws 
AtlasBaseException {
-try {
-if (StringUtils.isEmpty(entityGuid) || 
StringUtils.isEmpty(classificationVertexId)) {
-LOG.warn("updateClassificationsPropagation(entityGuid={}, 
classificationVertexId={}): entityGuid and/or classification vertex id is 
empty", entityGuid, classificationVertexId);
-return null;
-}
-
-AtlasVertex entityVertex = 
graphHelper.getVertexForGUID(entityGuid);
-
-if (entityVertex == null) {
-LOG.warn("updateClassificationsPropagation(entityGuid={}, 
classificationVertexId={}): entity vertex not found", entityGuid, 
classificationVertexId);
-return null;
-}
-
-AtlasVertex classificationVertex = 
graph.getVertex(classificationVertexId);
-
-if (classificationVertex == null) {
-LOG.warn("updateClassificationsPropagation(entityGuid={}, 
classificationVertexId={}): classification vertex not found", entityGuid, 
classificationVertexId);
-return null;
-}
-
-List entitiesToPropagateTo = 
entityRetriever.getImpactedVerticesV2(entityVertex, relationshipGuid, 
classificationVertex.getIdForDisplay());
-
-if (CollectionUti

[atlas] branch master updated: ATLAS-4285: Multiple propagations with intersecting lineage.

[sarath]

This is an automated email from the ASF dual-hosted git repository.

sarath pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/atlas.git


The following commit(s) were added to refs/heads/master by this push:
 new aba97b3  ATLAS-4285: Multiple propagations with intersecting lineage.
aba97b3 is described below

commit aba97b35393f1732eb30858e69fd5f489634afdc
Author: Ashutosh Mestry 
AuthorDate: Thu May 13 17:28:26 2021 -0700

ATLAS-4285: Multiple propagations with intersecting lineage.

Signed-off-by: Sarath Subramanian 
---
 .../store/graph/v2/EntityGraphMapper.java  | 79 +++---
 .../tasks/ClassificationPropagateTaskFactory.java  |  5 --
 .../v2/tasks/ClassificationPropagationTasks.java   | 15 
 .../apache/atlas/tasks/TaskFactoryRegistry.java| 23 ++-
 .../org/apache/atlas/tasks/TaskManagement.java | 11 +++
 .../ClassificationPropagationWithTasksTest.java|  6 --
 6 files changed, 43 insertions(+), 96 deletions(-)

diff --git 
a/repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java
 
b/repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java
index 7984a34..5baff33 100644
--- 
a/repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java
+++ 
b/repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java
@@ -2041,10 +2041,7 @@ public class EntityGraphMapper {
 return null;
 }
 
-
GraphTransactionInterceptor.lockObjectAndReleasePostCommit(entityGuid);
-
 AtlasVertex entityVertex = 
graphHelper.getVertexForGUID(entityGuid);
-
 if (entityVertex == null) {
 LOG.warn("propagateClassification(entityGuid={}, 
classificationVertexId={}): entity vertex not found", entityGuid, 
classificationVertexId);
 
@@ -2052,7 +2049,6 @@ public class EntityGraphMapper {
 }
 
 AtlasVertex classificationVertex = 
graph.getVertex(classificationVertexId);
-
 if (classificationVertex == null) {
 LOG.warn("propagateClassification(entityGuid={}, 
classificationVertexId={}): classification vertex not found", entityGuid, 
classificationVertexId);
 
@@ -2060,13 +2056,15 @@ public class EntityGraphMapper {
 }
 
 List impactedVertices = 
entityRetriever.getIncludedImpactedVerticesV2(entityVertex, relationshipGuid, 
classificationVertexId);
-
 if (CollectionUtils.isEmpty(impactedVertices)) {
 LOG.debug("propagateClassification(entityGuid={}, 
classificationVertexId={}): found no entities to propagate the classification", 
entityGuid, classificationVertexId);
 
 return null;
 }
 
+List impactedVerticesGuidsToLock = 
impactedVertices.stream().map(x -> 
GraphHelper.getGuid(x)).collect(Collectors.toList());
+
GraphTransactionInterceptor.lockObjectAndReleasePostCommit(impactedVerticesGuidsToLock);
+
 AtlasClassification classification   = 
entityRetriever.toAtlasClassification(classificationVertex);
 List   entitiesPropagatedTo = 
deleteDelegate.getHandler().addTagPropagation(classificationVertex, 
impactedVertices);
 
@@ -2453,63 +2451,6 @@ public class EntityGraphMapper {
 AtlasPerfTracer.log(perf);
 }
 
-@GraphTransaction
-public List updateClassificationsPropagation(String entityGuid, 
String classificationVertexId, String relationshipGuid) throws 
AtlasBaseException {
-try {
-if (StringUtils.isEmpty(entityGuid) || 
StringUtils.isEmpty(classificationVertexId)) {
-LOG.warn("updateClassificationsPropagation(entityGuid={}, 
classificationVertexId={}): entityGuid and/or classification vertex id is 
empty", entityGuid, classificationVertexId);
-return null;
-}
-
-AtlasVertex entityVertex = 
graphHelper.getVertexForGUID(entityGuid);
-
-if (entityVertex == null) {
-LOG.warn("updateClassificationsPropagation(entityGuid={}, 
classificationVertexId={}): entity vertex not found", entityGuid, 
classificationVertexId);
-return null;
-}
-
-AtlasVertex classificationVertex = 
graph.getVertex(classificationVertexId);
-
-if (classificationVertex == null) {
-LOG.warn("updateClassificationsPropagation(entityGuid={}, 
classificationVertexId={}): classification vertex not found", entityGuid, 
classificationVertexId);
-return null;
-}
-
-List entitiesToPropagateTo = 
entityRetriever.getImpactedVerticesV2(entityVertex, relationshipGuid, 
classificationVertex.getIdForDisplay());
-
-if (CollectionUtils.isEmpty(entitiesToPropagateTo)) {
-LOG.debug("updateClassification

[atlas] branch branch-2.0 updated: ATLAS-4274: Non matching relation are created via bulk import

[sarath]

This is an automated email from the ASF dual-hosted git repository.

sarath pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/atlas.git


The following commit(s) were added to refs/heads/branch-2.0 by this push:
 new 1310598  ATLAS-4274: Non matching relation are created via bulk import
1310598 is described below

commit 1310598af79ea3a815972cde18d50be252337856
Author: sidmishra 
AuthorDate: Sat May 8 09:52:23 2021 -0700

ATLAS-4274: Non matching relation are created via bulk import

Signed-off-by: Sarath Subramanian 
(cherry picked from commit 821b4c441153505db7f0ba9c5670623c527af989)
---
 .../org/apache/atlas/glossary/GlossaryService.java |   3 +
 .../apache/atlas/glossary/GlossaryTermUtils.java   | 205 ++---
 2 files changed, 185 insertions(+), 23 deletions(-)

diff --git 
a/repository/src/main/java/org/apache/atlas/glossary/GlossaryService.java 
b/repository/src/main/java/org/apache/atlas/glossary/GlossaryService.java
index 73217de..9c84598 100644
--- a/repository/src/main/java/org/apache/atlas/glossary/GlossaryService.java
+++ b/repository/src/main/java/org/apache/atlas/glossary/GlossaryService.java
@@ -1118,6 +1118,7 @@ public class GlossaryService {
 }
 }
 
+@GraphTransaction
 public BulkImportResponse importGlossaryData(InputStream inputStream, 
String fileName) throws AtlasBaseException {
 BulkImportResponse ret = new BulkImportResponse();
 
@@ -1161,6 +1162,8 @@ public class GlossaryService {
 
 private void updateGlossaryTermsRelation(List 
glossaryTerms, BulkImportResponse bulkImportResponse) {
 for (AtlasGlossaryTerm glossaryTerm : glossaryTerms) {
+glossaryTermUtils.updateGlossaryTermRelations(glossaryTerm);
+
 if (glossaryTerm.hasTerms()) {
 String glossaryTermName = glossaryTerm.getName();
 String glossaryName = getGlossaryName(glossaryTerm);
diff --git 
a/repository/src/main/java/org/apache/atlas/glossary/GlossaryTermUtils.java 
b/repository/src/main/java/org/apache/atlas/glossary/GlossaryTermUtils.java
index 553d3d0..fa02c8e 100644
--- a/repository/src/main/java/org/apache/atlas/glossary/GlossaryTermUtils.java
+++ b/repository/src/main/java/org/apache/atlas/glossary/GlossaryTermUtils.java
@@ -66,7 +66,9 @@ public class GlossaryTermUtils extends GlossaryUtils {
 private static final int INDEX_FOR_GLOSSARY_AT_RECORD = 0;
 private static final int INDEX_FOR_TERM_AT_RECORD = 1;
 
-private static final ThreadLocal> 
glossaryNameGuidCacheForImport = ThreadLocal.withInitial(() -> new 
LinkedHashMap<>());
+private static final ThreadLocal>  
glossaryNameGuidCache  = ThreadLocal.withInitial(() -> new 
LinkedHashMap<>());
+private static final ThreadLocal> 
glossaryTermOrderCache = ThreadLocal.withInitial(() -> new HashMap<>());
+private static final ThreadLocal>  
glossaryTermQNameGuidCache = ThreadLocal.withInitial(() -> new HashMap<>());
 
 protected GlossaryTermUtils(AtlasRelationshipStore relationshipStore, 
AtlasTypeRegistry typeRegistry, DataAccess dataAccess) {
 super(relationshipStore, typeRegistry, dataAccess);
@@ -150,7 +152,9 @@ public class GlossaryTermUtils extends GlossaryUtils {
 }
 
 public void clearImportCache() {
-glossaryNameGuidCacheForImport.get().clear();
+glossaryNameGuidCache.get().clear();
+glossaryTermOrderCache.get().clear();
+glossaryTermQNameGuidCache.get().clear();
 }
 
 private boolean isRelationshipGuidSame(AtlasRelatedObjectId storeObject, 
AtlasRelatedObjectId relatedObjectId) {
@@ -560,7 +564,7 @@ public class GlossaryTermUtils extends GlossaryUtils {
 } else {
 glossaryName = record[INDEX_FOR_GLOSSARY_AT_RECORD];
 
-String glossaryGuid = 
glossaryNameGuidCacheForImport.get().get(glossaryName);
+String glossaryGuid = 
glossaryNameGuidCache.get().get(glossaryName);
 
 if (StringUtils.isEmpty(glossaryGuid)) {
 glossaryGuid = getGlossaryGUIDFromGraphDB(glossaryName);
@@ -569,13 +573,15 @@ public class GlossaryTermUtils extends GlossaryUtils {
 glossaryGuid = createGlossary(glossaryName, 
failedTermMsgs);
 }
 
-glossaryNameGuidCacheForImport.get().put(glossaryName, 
glossaryGuid);
+glossaryNameGuidCache.get().put(glossaryName, 
glossaryGuid);
 }
 
 if (StringUtils.isNotEmpty(glossaryGuid)) {
 glossaryTerm = populateGlossaryTermObject(failedTermMsgs, 
record, glossaryGuid, false);
 
-
glossaryTerm.setQualifiedName(getGlossaryTermQualifiedName(glossaryTerm.getName(),
 glossaryGuid));
+
glossaryTerm.setQualifiedName(getGlossaryTermQualifiedName(glossaryTerm.getName(),
 glossaryName));
+

[atlas] branch master updated: ATLAS-4274: Non matching relation are created via bulk import

[sarath]

This is an automated email from the ASF dual-hosted git repository.

sarath pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/atlas.git


The following commit(s) were added to refs/heads/master by this push:
 new 821b4c4  ATLAS-4274: Non matching relation are created via bulk import
821b4c4 is described below

commit 821b4c441153505db7f0ba9c5670623c527af989
Author: sidmishra 
AuthorDate: Sat May 8 09:52:23 2021 -0700

ATLAS-4274: Non matching relation are created via bulk import

Signed-off-by: Sarath Subramanian 
---
 .../org/apache/atlas/glossary/GlossaryService.java |   3 +
 .../apache/atlas/glossary/GlossaryTermUtils.java   | 205 ++---
 2 files changed, 185 insertions(+), 23 deletions(-)

diff --git 
a/repository/src/main/java/org/apache/atlas/glossary/GlossaryService.java 
b/repository/src/main/java/org/apache/atlas/glossary/GlossaryService.java
index 73217de..9c84598 100644
--- a/repository/src/main/java/org/apache/atlas/glossary/GlossaryService.java
+++ b/repository/src/main/java/org/apache/atlas/glossary/GlossaryService.java
@@ -1118,6 +1118,7 @@ public class GlossaryService {
 }
 }
 
+@GraphTransaction
 public BulkImportResponse importGlossaryData(InputStream inputStream, 
String fileName) throws AtlasBaseException {
 BulkImportResponse ret = new BulkImportResponse();
 
@@ -1161,6 +1162,8 @@ public class GlossaryService {
 
 private void updateGlossaryTermsRelation(List 
glossaryTerms, BulkImportResponse bulkImportResponse) {
 for (AtlasGlossaryTerm glossaryTerm : glossaryTerms) {
+glossaryTermUtils.updateGlossaryTermRelations(glossaryTerm);
+
 if (glossaryTerm.hasTerms()) {
 String glossaryTermName = glossaryTerm.getName();
 String glossaryName = getGlossaryName(glossaryTerm);
diff --git 
a/repository/src/main/java/org/apache/atlas/glossary/GlossaryTermUtils.java 
b/repository/src/main/java/org/apache/atlas/glossary/GlossaryTermUtils.java
index 553d3d0..fa02c8e 100644
--- a/repository/src/main/java/org/apache/atlas/glossary/GlossaryTermUtils.java
+++ b/repository/src/main/java/org/apache/atlas/glossary/GlossaryTermUtils.java
@@ -66,7 +66,9 @@ public class GlossaryTermUtils extends GlossaryUtils {
 private static final int INDEX_FOR_GLOSSARY_AT_RECORD = 0;
 private static final int INDEX_FOR_TERM_AT_RECORD = 1;
 
-private static final ThreadLocal> 
glossaryNameGuidCacheForImport = ThreadLocal.withInitial(() -> new 
LinkedHashMap<>());
+private static final ThreadLocal>  
glossaryNameGuidCache  = ThreadLocal.withInitial(() -> new 
LinkedHashMap<>());
+private static final ThreadLocal> 
glossaryTermOrderCache = ThreadLocal.withInitial(() -> new HashMap<>());
+private static final ThreadLocal>  
glossaryTermQNameGuidCache = ThreadLocal.withInitial(() -> new HashMap<>());
 
 protected GlossaryTermUtils(AtlasRelationshipStore relationshipStore, 
AtlasTypeRegistry typeRegistry, DataAccess dataAccess) {
 super(relationshipStore, typeRegistry, dataAccess);
@@ -150,7 +152,9 @@ public class GlossaryTermUtils extends GlossaryUtils {
 }
 
 public void clearImportCache() {
-glossaryNameGuidCacheForImport.get().clear();
+glossaryNameGuidCache.get().clear();
+glossaryTermOrderCache.get().clear();
+glossaryTermQNameGuidCache.get().clear();
 }
 
 private boolean isRelationshipGuidSame(AtlasRelatedObjectId storeObject, 
AtlasRelatedObjectId relatedObjectId) {
@@ -560,7 +564,7 @@ public class GlossaryTermUtils extends GlossaryUtils {
 } else {
 glossaryName = record[INDEX_FOR_GLOSSARY_AT_RECORD];
 
-String glossaryGuid = 
glossaryNameGuidCacheForImport.get().get(glossaryName);
+String glossaryGuid = 
glossaryNameGuidCache.get().get(glossaryName);
 
 if (StringUtils.isEmpty(glossaryGuid)) {
 glossaryGuid = getGlossaryGUIDFromGraphDB(glossaryName);
@@ -569,13 +573,15 @@ public class GlossaryTermUtils extends GlossaryUtils {
 glossaryGuid = createGlossary(glossaryName, 
failedTermMsgs);
 }
 
-glossaryNameGuidCacheForImport.get().put(glossaryName, 
glossaryGuid);
+glossaryNameGuidCache.get().put(glossaryName, 
glossaryGuid);
 }
 
 if (StringUtils.isNotEmpty(glossaryGuid)) {
 glossaryTerm = populateGlossaryTermObject(failedTermMsgs, 
record, glossaryGuid, false);
 
-
glossaryTerm.setQualifiedName(getGlossaryTermQualifiedName(glossaryTerm.getName(),
 glossaryGuid));
+
glossaryTerm.setQualifiedName(getGlossaryTermQualifiedName(glossaryTerm.getName(),
 glossaryName));
+
+
glossaryTermOrderCache.get

[atlas] branch branch-2.0 updated: ATLAS-4284: Hive pruned tables are ignored and not pruned

[sarath]

This is an automated email from the ASF dual-hosted git repository.

sarath pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/atlas.git


The following commit(s) were added to refs/heads/branch-2.0 by this push:
 new 6f45f5f  ATLAS-4284: Hive pruned tables are ignored and not pruned
6f45f5f is described below

commit 6f45f5f63e3628e3e17b6c94e48ed60d45ad2e9d
Author: Radhika Kundam 
AuthorDate: Wed May 12 13:55:08 2021 -0700

ATLAS-4284: Hive pruned tables are ignored and not pruned

Signed-off-by: Sarath Subramanian 
(cherry picked from commit 81888a9c7772b83437d009071e91fbd2a13cc831)
---
 .../org/apache/atlas/notification/preprocessor/HivePreprocessor.java  | 4 
 1 file changed, 4 insertions(+)

diff --git 
a/webapp/src/main/java/org/apache/atlas/notification/preprocessor/HivePreprocessor.java
 
b/webapp/src/main/java/org/apache/atlas/notification/preprocessor/HivePreprocessor.java
index e69d63e..86e3384 100644
--- 
a/webapp/src/main/java/org/apache/atlas/notification/preprocessor/HivePreprocessor.java
+++ 
b/webapp/src/main/java/org/apache/atlas/notification/preprocessor/HivePreprocessor.java
@@ -81,6 +81,10 @@ public class HivePreprocessor {
 entity.setAttribute(ATTRIBUTE_SD, null);
 entity.setAttribute(ATTRIBUTE_COLUMNS, null);
 entity.setAttribute(ATTRIBUTE_PARTITION_KEYS, null);
+
+entity.setRelationshipAttribute(ATTRIBUTE_SD, null);
+entity.setRelationshipAttribute(ATTRIBUTE_COLUMNS, null);
+entity.setRelationshipAttribute(ATTRIBUTE_PARTITION_KEYS, 
null);
 } else if (context.getHiveTypesRemoveOwnedRefAttrs()) {
 context.removeRefAttributeAndRegisterToMove(entity, 
ATTRIBUTE_SD, RELATIONSHIP_TYPE_HIVE_TABLE_STORAGEDESC, ATTRIBUTE_TABLE);
 context.removeRefAttributeAndRegisterToMove(entity, 
ATTRIBUTE_COLUMNS, RELATIONSHIP_TYPE_HIVE_TABLE_COLUMNS, ATTRIBUTE_TABLE);

[atlas] branch master updated: ATLAS-4284: Hive pruned tables are ignored and not pruned

[sarath]

This is an automated email from the ASF dual-hosted git repository.

sarath pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/atlas.git


The following commit(s) were added to refs/heads/master by this push:
 new 81888a9  ATLAS-4284: Hive pruned tables are ignored and not pruned
81888a9 is described below

commit 81888a9c7772b83437d009071e91fbd2a13cc831
Author: Radhika Kundam 
AuthorDate: Wed May 12 13:55:08 2021 -0700

ATLAS-4284: Hive pruned tables are ignored and not pruned

Signed-off-by: Sarath Subramanian 
---
 .../org/apache/atlas/notification/preprocessor/HivePreprocessor.java  | 4 
 1 file changed, 4 insertions(+)

diff --git 
a/webapp/src/main/java/org/apache/atlas/notification/preprocessor/HivePreprocessor.java
 
b/webapp/src/main/java/org/apache/atlas/notification/preprocessor/HivePreprocessor.java
index e69d63e..86e3384 100644
--- 
a/webapp/src/main/java/org/apache/atlas/notification/preprocessor/HivePreprocessor.java
+++ 
b/webapp/src/main/java/org/apache/atlas/notification/preprocessor/HivePreprocessor.java
@@ -81,6 +81,10 @@ public class HivePreprocessor {
 entity.setAttribute(ATTRIBUTE_SD, null);
 entity.setAttribute(ATTRIBUTE_COLUMNS, null);
 entity.setAttribute(ATTRIBUTE_PARTITION_KEYS, null);
+
+entity.setRelationshipAttribute(ATTRIBUTE_SD, null);
+entity.setRelationshipAttribute(ATTRIBUTE_COLUMNS, null);
+entity.setRelationshipAttribute(ATTRIBUTE_PARTITION_KEYS, 
null);
 } else if (context.getHiveTypesRemoveOwnedRefAttrs()) {
 context.removeRefAttributeAndRegisterToMove(entity, 
ATTRIBUTE_SD, RELATIONSHIP_TYPE_HIVE_TABLE_STORAGEDESC, ATTRIBUTE_TABLE);
 context.removeRefAttributeAndRegisterToMove(entity, 
ATTRIBUTE_COLUMNS, RELATIONSHIP_TYPE_HIVE_TABLE_COLUMNS, ATTRIBUTE_TABLE);

[atlas] branch master updated: ATLAS-4164: [Atlas: Spooling] Tables created after spooling are created before the spooled tables when there is multiple frequent restart in kafka brokers

[sarath]

This is an automated email from the ASF dual-hosted git repository.

sarath pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/atlas.git


The following commit(s) were added to refs/heads/master by this push:
 new 4100684  ATLAS-4164: [Atlas: Spooling] Tables created after spooling 
are created before the spooled tables when there is multiple frequent restart 
in kafka brokers
4100684 is described below

commit 4100684fa3f63cb2a6267ab24051002ce38de017
Author: Radhika Kundam 
AuthorDate: Tue May 11 18:00:49 2021 -0700

ATLAS-4164: [Atlas: Spooling] Tables created after spooling are created 
before the spooled tables when there is multiple frequent restart in kafka 
brokers

Signed-off-by: Sarath Subramanian 
---
 .../org/apache/atlas/notification/spool/IndexManagement.java   | 10 --
 1 file changed, 8 insertions(+), 2 deletions(-)

diff --git 
a/notification/src/main/java/org/apache/atlas/notification/spool/IndexManagement.java
 
b/notification/src/main/java/org/apache/atlas/notification/spool/IndexManagement.java
index b3a586b..f018983 100644
--- 
a/notification/src/main/java/org/apache/atlas/notification/spool/IndexManagement.java
+++ 
b/notification/src/main/java/org/apache/atlas/notification/spool/IndexManagement.java
@@ -98,8 +98,9 @@ public class IndexManagement {
 }
 
 public boolean isPending() {
-return !indexReader.isEmpty() ||
-(indexWriter.getCurrent() != null && 
indexWriter.getCurrent().getLine() > 0);
+return !indexReader.isEmpty()
+|| (indexWriter.getCurrent() != null && 
indexWriter.getCurrent().isStatusWriteInProgress())
+|| (indexReader.currentIndexRecord != null && 
indexReader.currentIndexRecord.getStatus() == 
IndexRecord.STATUS_READ_IN_PROGRESS);
 }
 
 public synchronized DataOutput getSpoolWriter() throws IOException {
@@ -146,6 +147,8 @@ public class IndexManagement {
 
 public void update(IndexRecord record) {
 this.indexFileManager.updateIndex(record);
+
+LOG.info("this.indexFileManager.updateIndex: {}", record.getLine());
 }
 
 public void flushSpoolWriter() throws IOException {
@@ -349,6 +352,9 @@ public class IndexManagement {
 
 public IndexRecord next() throws InterruptedException {
 this.currentIndexRecord = blockingQueue.poll(retryDestinationMS, 
TimeUnit.MILLISECONDS);
+if (this.currentIndexRecord != null) {
+
this.currentIndexRecord.setStatus(IndexRecord.STATUS_READ_IN_PROGRESS);
+}
 
 return this.currentIndexRecord;
 }

[atlas] branch branch-2.0 updated: ATLAS-4183: Upgrade deployment descriptor for webapp project

[sarath]

This is an automated email from the ASF dual-hosted git repository.

sarath pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/atlas.git


The following commit(s) were added to refs/heads/branch-2.0 by this push:
 new 97689b3  ATLAS-4183: Upgrade deployment descriptor for webapp project
97689b3 is described below

commit 97689b34df747a8ce2a6a6d141340970d746ecbf
Author: Deep Singh 
AuthorDate: Tue May 11 10:09:51 2021 -0500

ATLAS-4183: Upgrade deployment descriptor for webapp project

Signed-off-by: Sarath Subramanian 
(cherry picked from commit c5ad5810856e465efa0cbb2d33c1614d52ff3b00)
---
 webapp/src/main/webapp/WEB-INF/web.xml | 9 +
 1 file changed, 5 insertions(+), 4 deletions(-)

diff --git a/webapp/src/main/webapp/WEB-INF/web.xml 
b/webapp/src/main/webapp/WEB-INF/web.xml
index 2595a15..7fb56ef 100755
--- a/webapp/src/main/webapp/WEB-INF/web.xml
+++ b/webapp/src/main/webapp/WEB-INF/web.xml
@@ -17,10 +17,11 @@
   ~ limitations under the License.
   -->
 
-http://java.sun.com/dtd/web-app_2_3.dtd;>
-
-
+http://java.sun.com/xml/ns/javaee;
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
+ xsi:schemaLocation="http://java.sun.com/xml/ns/javaee 
http://java.sun.com/xml/ns/javaee/web-app_2_5.xsd;
+ version="2.5"
+>
 Apache Atlas
 Metadata Management and Data Governance Platform over 
Hadoop
 

[atlas] branch master updated: ATLAS-4183: Upgrade deployment descriptor for webapp project

[sarath]

This is an automated email from the ASF dual-hosted git repository.

sarath pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/atlas.git


The following commit(s) were added to refs/heads/master by this push:
 new c5ad581  ATLAS-4183: Upgrade deployment descriptor for webapp project
c5ad581 is described below

commit c5ad5810856e465efa0cbb2d33c1614d52ff3b00
Author: Deep Singh 
AuthorDate: Tue May 11 10:09:51 2021 -0500

ATLAS-4183: Upgrade deployment descriptor for webapp project

Signed-off-by: Sarath Subramanian 
---
 webapp/src/main/webapp/WEB-INF/web.xml | 9 +
 1 file changed, 5 insertions(+), 4 deletions(-)

diff --git a/webapp/src/main/webapp/WEB-INF/web.xml 
b/webapp/src/main/webapp/WEB-INF/web.xml
index 2595a15..7fb56ef 100755
--- a/webapp/src/main/webapp/WEB-INF/web.xml
+++ b/webapp/src/main/webapp/WEB-INF/web.xml
@@ -17,10 +17,11 @@
   ~ limitations under the License.
   -->
 
-http://java.sun.com/dtd/web-app_2_3.dtd;>
-
-
+http://java.sun.com/xml/ns/javaee;
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
+ xsi:schemaLocation="http://java.sun.com/xml/ns/javaee 
http://java.sun.com/xml/ns/javaee/web-app_2_5.xsd;
+ version="2.5"
+>
 Apache Atlas
 Metadata Management and Data Governance Platform over 
Hadoop
 

[atlas] branch master updated: ATLAS-4153: [Atlas: Spooling] The order of the entities created in atlas is not same as the order created in hive

[sarath]

This is an automated email from the ASF dual-hosted git repository.

sarath pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/atlas.git


The following commit(s) were added to refs/heads/master by this push:
 new e6f78f2  ATLAS-4153: [Atlas: Spooling] The order of the entities 
created in atlas is not same as the order created in hive
e6f78f2 is described below

commit e6f78f28c8167c0099c1493fee51aac39abeebea
Author: Radhika Kundam 
AuthorDate: Mon May 10 18:11:43 2021 -0700

ATLAS-4153: [Atlas: Spooling] The order of the entities created in atlas is 
not same as the order created in hive

Signed-off-by: Sarath Subramanian 
---
 .../main/java/org/apache/atlas/notification/spool/Publisher.java  | 8 
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git 
a/notification/src/main/java/org/apache/atlas/notification/spool/Publisher.java 
b/notification/src/main/java/org/apache/atlas/notification/spool/Publisher.java
index 2947a21..22242c9 100644
--- 
a/notification/src/main/java/org/apache/atlas/notification/spool/Publisher.java
+++ 
b/notification/src/main/java/org/apache/atlas/notification/spool/Publisher.java
@@ -134,9 +134,9 @@ public class Publisher implements Runnable {
 FileLockedReadWrite fileLockedRead = new 
FileLockedReadWrite(source);
 
 try {
-DataInput dataInput = fileLockedRead.getInput(new 
File(record.getPath()));
-int lineInSpoolFile = 0;
-List messages = new ArrayList<>();
+DataInputdataInput   = fileLockedRead.getInput(new 
File(record.getPath()));
+int  lineInSpoolFile = 0;
+List messages= new ArrayList<>();
 
 for (String message = dataInput.readLine(); message != null; 
message = dataInput.readLine()) {
 lineInSpoolFile++;
@@ -147,7 +147,7 @@ public class Publisher implements Runnable {
 
 messages.add(message);
 
-if (messages.size() == messageBatchSize) {
+if ((isDestDown && messages.size() == 1) || 
messages.size() == messageBatchSize) {
 dispatch(record, lineInSpoolFile, messages);
 }
 }

[atlas] branch branch-2.0 updated: ATLAS-4269: Deferred Actions : When a tag is propagated from an entity via 2 processes , blocking 1 process removes tag propagated from another process

[sarath]

This is an automated email from the ASF dual-hosted git repository.

sarath pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/atlas.git


The following commit(s) were added to refs/heads/branch-2.0 by this push:
 new 06dbe0f  ATLAS-4269: Deferred Actions : When a tag is propagated from 
an entity via 2 processes , blocking 1 process removes tag propagated from 
another process
06dbe0f is described below

commit 06dbe0f0feda648bae41ab170984bf7c4fb94da5
Author: Sarath Subramanian 
AuthorDate: Thu Apr 29 21:00:44 2021 -0700

ATLAS-4269: Deferred Actions : When a tag is propagated from an entity via 
2 processes , blocking 1 process removes tag propagated from another process

(cherry picked from commit 85e3c5cf64747432bd434bbf1573224988c37bf6)
---
 .../repository/store/graph/v2/EntityGraphRetriever.java| 14 +-
 1 file changed, 9 insertions(+), 5 deletions(-)

diff --git 
a/repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java
 
b/repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java
index 9abcf64..7948475 100644
--- 
a/repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java
+++ 
b/repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java
@@ -531,9 +531,9 @@ public class EntityGraphRetriever {
 
 private void traverseImpactedVertices(final AtlasVertex entityVertexStart, 
final String relationshipGuidToExclude,
   final String classificationId, final 
List result) {
-Set visitedVertices = new HashSet<>();
-
-Queue queue = new ArrayDeque() {{ 
add(entityVertexStart); }};
+Set  visitedVertices = new HashSet<>();
+Queue   queue   = new 
ArrayDeque() {{ add(entityVertexStart); }};
+Map resultsMap  = new HashMap<>();
 
 while (!queue.isEmpty()) {
 AtlasVertex entityVertex   = queue.poll();
@@ -555,6 +555,7 @@ public class EntityGraphRetriever {
 }
 
 Iterable propagationEdges = 
entityVertex.getEdges(AtlasEdgeDirection.BOTH, tagPropagationEdges);
+
 for (AtlasEdge propagationEdge : propagationEdges) {
 if (getEdgeStatus(propagationEdge) != ACTIVE) {
 continue;
@@ -582,6 +583,7 @@ public class EntityGraphRetriever {
 
 if (classificationId != null) {
 List blockedClassificationIds = 
getBlockedClassificationIds(propagationEdge);
+
 if (CollectionUtils.isNotEmpty(blockedClassificationIds) 
&& blockedClassificationIds.contains(classificationId)) {
 continue;
 }
@@ -590,13 +592,15 @@ public class EntityGraphRetriever {
 AtlasVertex adjacentVertex = 
getOtherVertex(propagationEdge, entityVertex);
 String  adjacentVertexIdForDisplay = 
adjacentVertex.getIdForDisplay();
 
-if (!visitedVertices.contains(adjacentVertexIdForDisplay)) {
-result.add(adjacentVertex);
+if (!visitedVertices.contains(adjacentVertexIdForDisplay) && 
!resultsMap.containsKey(adjacentVertexIdForDisplay)) {
+resultsMap.put(adjacentVertexIdForDisplay, adjacentVertex);
 
 queue.add(adjacentVertex);
 }
 }
 }
+
+result.addAll(resultsMap.values());
 }
 
 private boolean isOutVertex(AtlasVertex vertex, AtlasEdge edge) {

[atlas] branch master updated: ATLAS-4269: Deferred Actions : When a tag is propagated from an entity via 2 processes , blocking 1 process removes tag propagated from another process

[sarath]

This is an automated email from the ASF dual-hosted git repository.

sarath pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/atlas.git


The following commit(s) were added to refs/heads/master by this push:
 new 85e3c5c  ATLAS-4269: Deferred Actions : When a tag is propagated from 
an entity via 2 processes , blocking 1 process removes tag propagated from 
another process
85e3c5c is described below

commit 85e3c5cf64747432bd434bbf1573224988c37bf6
Author: Sarath Subramanian 
AuthorDate: Thu Apr 29 21:00:44 2021 -0700

ATLAS-4269: Deferred Actions : When a tag is propagated from an entity via 
2 processes , blocking 1 process removes tag propagated from another process
---
 .../repository/store/graph/v2/EntityGraphRetriever.java| 14 +-
 1 file changed, 9 insertions(+), 5 deletions(-)

diff --git 
a/repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java
 
b/repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java
index 9abcf64..7948475 100644
--- 
a/repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java
+++ 
b/repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java
@@ -531,9 +531,9 @@ public class EntityGraphRetriever {
 
 private void traverseImpactedVertices(final AtlasVertex entityVertexStart, 
final String relationshipGuidToExclude,
   final String classificationId, final 
List result) {
-Set visitedVertices = new HashSet<>();
-
-Queue queue = new ArrayDeque() {{ 
add(entityVertexStart); }};
+Set  visitedVertices = new HashSet<>();
+Queue   queue   = new 
ArrayDeque() {{ add(entityVertexStart); }};
+Map resultsMap  = new HashMap<>();
 
 while (!queue.isEmpty()) {
 AtlasVertex entityVertex   = queue.poll();
@@ -555,6 +555,7 @@ public class EntityGraphRetriever {
 }
 
 Iterable propagationEdges = 
entityVertex.getEdges(AtlasEdgeDirection.BOTH, tagPropagationEdges);
+
 for (AtlasEdge propagationEdge : propagationEdges) {
 if (getEdgeStatus(propagationEdge) != ACTIVE) {
 continue;
@@ -582,6 +583,7 @@ public class EntityGraphRetriever {
 
 if (classificationId != null) {
 List blockedClassificationIds = 
getBlockedClassificationIds(propagationEdge);
+
 if (CollectionUtils.isNotEmpty(blockedClassificationIds) 
&& blockedClassificationIds.contains(classificationId)) {
 continue;
 }
@@ -590,13 +592,15 @@ public class EntityGraphRetriever {
 AtlasVertex adjacentVertex = 
getOtherVertex(propagationEdge, entityVertex);
 String  adjacentVertexIdForDisplay = 
adjacentVertex.getIdForDisplay();
 
-if (!visitedVertices.contains(adjacentVertexIdForDisplay)) {
-result.add(adjacentVertex);
+if (!visitedVertices.contains(adjacentVertexIdForDisplay) && 
!resultsMap.containsKey(adjacentVertexIdForDisplay)) {
+resultsMap.put(adjacentVertexIdForDisplay, adjacentVertex);
 
 queue.add(adjacentVertex);
 }
 }
 }
+
+result.addAll(resultsMap.values());
 }
 
 private boolean isOutVertex(AtlasVertex vertex, AtlasEdge edge) {

[atlas] branch branch-2.0 updated: ATLAS-4268: Deferred Actions : When a tag is already associated to a term and when the term is assigned to the entity, tag propagation doesn't happen

[sarath]

This is an automated email from the ASF dual-hosted git repository.

sarath pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/atlas.git


The following commit(s) were added to refs/heads/branch-2.0 by this push:
 new 2bb3199  ATLAS-4268: Deferred Actions : When a tag is already 
associated to a term and when the term is assigned to the entity, tag 
propagation doesn't happen
2bb3199 is described below

commit 2bb3199f528952cbea3bbde9f25b57f98217550c
Author: Sarath Subramanian 
AuthorDate: Thu Apr 29 12:33:03 2021 -0700

ATLAS-4268: Deferred Actions : When a tag is already associated to a term 
and when the term is assigned to the entity, tag propagation doesn't happen

(cherry picked from commit 7e73fc2591f8177630f1ffd82e774b490832650a)
---
 .../apache/atlas/repository/store/graph/v2/EntityGraphMapper.java   | 2 +-
 .../atlas/repository/store/graph/v2/EntityGraphRetriever.java   | 6 +-
 2 files changed, 6 insertions(+), 2 deletions(-)

diff --git 
a/repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java
 
b/repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java
index 2a71e34..02ba5ef 100644
--- 
a/repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java
+++ 
b/repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java
@@ -2059,7 +2059,7 @@ public class EntityGraphMapper {
 return null;
 }
 
-List impactedVertices = 
entityRetriever.getImpactedVerticesV2(entityVertex, relationshipGuid, 
classificationVertexId);
+List impactedVertices = 
entityRetriever.getIncludedImpactedVerticesV2(entityVertex, relationshipGuid, 
classificationVertexId);
 
 if (CollectionUtils.isEmpty(impactedVertices)) {
 LOG.debug("propagateClassification(entityGuid={}, 
classificationVertexId={}): found no entities to propagate the classification", 
entityGuid, classificationVertexId);
diff --git 
a/repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java
 
b/repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java
index 2e0f39a..9abcf64 100644
--- 
a/repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java
+++ 
b/repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java
@@ -510,9 +510,13 @@ public class EntityGraphRetriever {
 }
 
 public List getIncludedImpactedVerticesV2(AtlasVertex 
entityVertex, String relationshipGuidToExclude) {
+return getIncludedImpactedVerticesV2(entityVertex, 
relationshipGuidToExclude, null);
+}
+
+public List getIncludedImpactedVerticesV2(AtlasVertex 
entityVertex, String relationshipGuidToExclude, String classificationId) {
 List ret = new ArrayList<>(Arrays.asList(entityVertex));
 
-traverseImpactedVertices(entityVertex, relationshipGuidToExclude, 
null, ret);
+traverseImpactedVertices(entityVertex, relationshipGuidToExclude, 
classificationId, ret);
 
 return ret;
 }

[atlas] branch master updated: ATLAS-4268: Deferred Actions : When a tag is already associated to a term and when the term is assigned to the entity, tag propagation doesn't happen

[sarath]

This is an automated email from the ASF dual-hosted git repository.

sarath pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/atlas.git


The following commit(s) were added to refs/heads/master by this push:
 new 7e73fc2  ATLAS-4268: Deferred Actions : When a tag is already 
associated to a term and when the term is assigned to the entity, tag 
propagation doesn't happen
7e73fc2 is described below

commit 7e73fc2591f8177630f1ffd82e774b490832650a
Author: Sarath Subramanian 
AuthorDate: Thu Apr 29 12:33:03 2021 -0700

ATLAS-4268: Deferred Actions : When a tag is already associated to a term 
and when the term is assigned to the entity, tag propagation doesn't happen
---
 .../apache/atlas/repository/store/graph/v2/EntityGraphMapper.java   | 2 +-
 .../atlas/repository/store/graph/v2/EntityGraphRetriever.java   | 6 +-
 2 files changed, 6 insertions(+), 2 deletions(-)

diff --git 
a/repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java
 
b/repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java
index 2a71e34..02ba5ef 100644
--- 
a/repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java
+++ 
b/repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java
@@ -2059,7 +2059,7 @@ public class EntityGraphMapper {
 return null;
 }
 
-List impactedVertices = 
entityRetriever.getImpactedVerticesV2(entityVertex, relationshipGuid, 
classificationVertexId);
+List impactedVertices = 
entityRetriever.getIncludedImpactedVerticesV2(entityVertex, relationshipGuid, 
classificationVertexId);
 
 if (CollectionUtils.isEmpty(impactedVertices)) {
 LOG.debug("propagateClassification(entityGuid={}, 
classificationVertexId={}): found no entities to propagate the classification", 
entityGuid, classificationVertexId);
diff --git 
a/repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java
 
b/repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java
index 2e0f39a..9abcf64 100644
--- 
a/repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java
+++ 
b/repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java
@@ -510,9 +510,13 @@ public class EntityGraphRetriever {
 }
 
 public List getIncludedImpactedVerticesV2(AtlasVertex 
entityVertex, String relationshipGuidToExclude) {
+return getIncludedImpactedVerticesV2(entityVertex, 
relationshipGuidToExclude, null);
+}
+
+public List getIncludedImpactedVerticesV2(AtlasVertex 
entityVertex, String relationshipGuidToExclude, String classificationId) {
 List ret = new ArrayList<>(Arrays.asList(entityVertex));
 
-traverseImpactedVertices(entityVertex, relationshipGuidToExclude, 
null, ret);
+traverseImpactedVertices(entityVertex, relationshipGuidToExclude, 
classificationId, ret);
 
 return ret;
 }

[atlas] branch branch-2.0 updated: ATLAS-4261: Changed the AtlasGlossaryTermHeader glossaryGUID to termGUID

[sarath]

This is an automated email from the ASF dual-hosted git repository.

sarath pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/atlas.git


The following commit(s) were added to refs/heads/branch-2.0 by this push:
 new 00f3054  ATLAS-4261: Changed the AtlasGlossaryTermHeader glossaryGUID 
to termGUID
00f3054 is described below

commit 00f30545d75a7456bf4119b44effe26f384eae03
Author: sidmishra 
AuthorDate: Thu Apr 29 10:36:09 2021 -0700

ATLAS-4261: Changed the AtlasGlossaryTermHeader glossaryGUID to termGUID

Signed-off-by: Sarath Subramanian 
(cherry picked from commit 5673d72287d59aaf2685030a727998f040abdd72)
---
 .../model/glossary/AtlasGlossaryTermHeader.java| 24 +++---
 1 file changed, 12 insertions(+), 12 deletions(-)

diff --git 
a/intg/src/main/java/org/apache/atlas/model/glossary/AtlasGlossaryTermHeader.java
 
b/intg/src/main/java/org/apache/atlas/model/glossary/AtlasGlossaryTermHeader.java
index 65a5ce0..f807ba2 100644
--- 
a/intg/src/main/java/org/apache/atlas/model/glossary/AtlasGlossaryTermHeader.java
+++ 
b/intg/src/main/java/org/apache/atlas/model/glossary/AtlasGlossaryTermHeader.java
@@ -23,22 +23,22 @@ import java.util.Objects;
 
 @AtlasJSON
 public class AtlasGlossaryTermHeader {
-private String glossaryGuid;
+private String termGuid;
 private String qualifiedName;
 
-public AtlasGlossaryTermHeader(String glossaryGuid) {
-this.glossaryGuid = glossaryGuid;
+public AtlasGlossaryTermHeader(String termGuid) {
+this.termGuid = termGuid;
 }
 
-public AtlasGlossaryTermHeader(String glossaryGuid, String qualifiedName) {
-this.glossaryGuid = glossaryGuid;
+public AtlasGlossaryTermHeader(String termGuid, String qualifiedName) {
+this.termGuid = termGuid;
 this.qualifiedName = qualifiedName;
 }
 
 @Override
 public String toString() {
 final StringBuilder sb = new StringBuilder("AtlasGlossaryTermHeader{");
-sb.append("glossaryGuid='").append(glossaryGuid).append('\'');
+sb.append("termGuid='").append(termGuid).append('\'');
 sb.append(", qualifiedName='").append(qualifiedName).append('\'');
 sb.append('}');
 return sb.toString();
@@ -47,12 +47,12 @@ public class AtlasGlossaryTermHeader {
 public AtlasGlossaryTermHeader() {
 }
 
-public String getGlossaryGuid() {
-return glossaryGuid;
+public String getTermGuid() {
+return termGuid;
 }
 
-public void setGlossaryGuid(final String glossaryGuid) {
-this.glossaryGuid = glossaryGuid;
+public void setTermGuid(final String termGuid) {
+this.termGuid = termGuid;
 }
 
 public String getQualifiedName() {
@@ -68,13 +68,13 @@ public class AtlasGlossaryTermHeader {
 if (this == o) return true;
 if (!(o instanceof 
org.apache.atlas.model.glossary.AtlasGlossaryTermHeader)) return false;
 final org.apache.atlas.model.glossary.AtlasGlossaryTermHeader that = 
(org.apache.atlas.model.glossary.AtlasGlossaryTermHeader) o;
-return Objects.equals(glossaryGuid, that.glossaryGuid) &&
+return Objects.equals(termGuid, that.termGuid) &&
 Objects.equals(qualifiedName, that.qualifiedName);
 }
 
 @Override
 public int hashCode() {
-return Objects.hash(glossaryGuid, qualifiedName);
+return Objects.hash(termGuid, qualifiedName);
 }
 
 }

[atlas] branch master updated: ATLAS-4261: Changed the AtlasGlossaryTermHeader glossaryGUID to termGUID

[sarath]

This is an automated email from the ASF dual-hosted git repository.

sarath pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/atlas.git


The following commit(s) were added to refs/heads/master by this push:
 new 5673d72  ATLAS-4261: Changed the AtlasGlossaryTermHeader glossaryGUID 
to termGUID
5673d72 is described below

commit 5673d72287d59aaf2685030a727998f040abdd72
Author: sidmishra 
AuthorDate: Thu Apr 29 10:36:09 2021 -0700

ATLAS-4261: Changed the AtlasGlossaryTermHeader glossaryGUID to termGUID

Signed-off-by: Sarath Subramanian 
---
 .../model/glossary/AtlasGlossaryTermHeader.java| 24 +++---
 1 file changed, 12 insertions(+), 12 deletions(-)

diff --git 
a/intg/src/main/java/org/apache/atlas/model/glossary/AtlasGlossaryTermHeader.java
 
b/intg/src/main/java/org/apache/atlas/model/glossary/AtlasGlossaryTermHeader.java
index 65a5ce0..f807ba2 100644
--- 
a/intg/src/main/java/org/apache/atlas/model/glossary/AtlasGlossaryTermHeader.java
+++ 
b/intg/src/main/java/org/apache/atlas/model/glossary/AtlasGlossaryTermHeader.java
@@ -23,22 +23,22 @@ import java.util.Objects;
 
 @AtlasJSON
 public class AtlasGlossaryTermHeader {
-private String glossaryGuid;
+private String termGuid;
 private String qualifiedName;
 
-public AtlasGlossaryTermHeader(String glossaryGuid) {
-this.glossaryGuid = glossaryGuid;
+public AtlasGlossaryTermHeader(String termGuid) {
+this.termGuid = termGuid;
 }
 
-public AtlasGlossaryTermHeader(String glossaryGuid, String qualifiedName) {
-this.glossaryGuid = glossaryGuid;
+public AtlasGlossaryTermHeader(String termGuid, String qualifiedName) {
+this.termGuid = termGuid;
 this.qualifiedName = qualifiedName;
 }
 
 @Override
 public String toString() {
 final StringBuilder sb = new StringBuilder("AtlasGlossaryTermHeader{");
-sb.append("glossaryGuid='").append(glossaryGuid).append('\'');
+sb.append("termGuid='").append(termGuid).append('\'');
 sb.append(", qualifiedName='").append(qualifiedName).append('\'');
 sb.append('}');
 return sb.toString();
@@ -47,12 +47,12 @@ public class AtlasGlossaryTermHeader {
 public AtlasGlossaryTermHeader() {
 }
 
-public String getGlossaryGuid() {
-return glossaryGuid;
+public String getTermGuid() {
+return termGuid;
 }
 
-public void setGlossaryGuid(final String glossaryGuid) {
-this.glossaryGuid = glossaryGuid;
+public void setTermGuid(final String termGuid) {
+this.termGuid = termGuid;
 }
 
 public String getQualifiedName() {
@@ -68,13 +68,13 @@ public class AtlasGlossaryTermHeader {
 if (this == o) return true;
 if (!(o instanceof 
org.apache.atlas.model.glossary.AtlasGlossaryTermHeader)) return false;
 final org.apache.atlas.model.glossary.AtlasGlossaryTermHeader that = 
(org.apache.atlas.model.glossary.AtlasGlossaryTermHeader) o;
-return Objects.equals(glossaryGuid, that.glossaryGuid) &&
+return Objects.equals(termGuid, that.termGuid) &&
 Objects.equals(qualifiedName, that.qualifiedName);
 }
 
 @Override
 public int hashCode() {
-return Objects.hash(glossaryGuid, qualifiedName);
+return Objects.hash(termGuid, qualifiedName);
 }
 
 }

[atlas] branch branch-2.0 updated: ATLAS-4261: Bulk Glossary Import Response and Failed Error Message Improvements

[sarath]

This is an automated email from the ASF dual-hosted git repository.

sarath pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/atlas.git


The following commit(s) were added to refs/heads/branch-2.0 by this push:
 new 6a02504  ATLAS-4261: Bulk Glossary Import Response and Failed Error 
Message Improvements
6a02504 is described below

commit 6a025043d2aac4d8e44bf0b3f5598bb3c766a38b
Author: sidmishra 
AuthorDate: Wed Apr 28 23:09:09 2021 -0700

ATLAS-4261: Bulk Glossary Import Response and Failed Error Message 
Improvements

Signed-off-by: Sarath Subramanian 
(cherry picked from commit 7d7d1f3e1b1be9ea585398eb337208250730bda3)
---
 .../main/java/org/apache/atlas/AtlasErrorCode.java |   4 +-
 .../atlas/model/glossary/AtlasGlossaryTerm.java|  80 ++
 .../model/glossary/AtlasGlossaryTermHeader.java|  80 ++
 .../org/apache/atlas/glossary/GlossaryService.java |  24 ++-
 .../apache/atlas/glossary/GlossaryTermUtils.java   | 171 +
 5 files changed, 262 insertions(+), 97 deletions(-)

diff --git a/intg/src/main/java/org/apache/atlas/AtlasErrorCode.java 
b/intg/src/main/java/org/apache/atlas/AtlasErrorCode.java
index 5ef62d3..2febff4 100644
--- a/intg/src/main/java/org/apache/atlas/AtlasErrorCode.java
+++ b/intg/src/main/java/org/apache/atlas/AtlasErrorCode.java
@@ -208,6 +208,7 @@ public enum AtlasErrorCode {
 GLOSSARY_ALREADY_EXISTS(409, "ATLAS-409-00-007", "Glossary with name {0} 
already exists"),
 GLOSSARY_TERM_ALREADY_EXISTS(409, "ATLAS-409-00-009", "Glossary term with 
qualifiedName {0} already exists"),
 GLOSSARY_CATEGORY_ALREADY_EXISTS(409, "ATLAS-409-00-00A", "Glossary 
category with qualifiedName {0} already exists"),
+GLOSSARY_IMPORT_FAILED(409, "ATLAS-409-00-011", "Glossary import failed"),
 
 // All internal errors go here
 INTERNAL_ERROR(500, "ATLAS-500-00-001", "Internal server error {0}"),
@@ -230,7 +231,8 @@ public enum AtlasErrorCode {
 DATA_ACCESS_LOAD_FAILED(500, "ATLAS-500-00-013", "Load failed: {0}"),
 ENTITY_NOTIFICATION_FAILED(500, "ATLAS-500-00-014", "Notification failed 
for operation: {0} : {1}"),
 FAILED_TO_UPLOAD(500, "ATLAS-500-00-015", "Error occurred while uploading 
the file: {0}"),
-FAILED_TO_CREATE_GLOSSARY_TERM(500, "ATLAS-500-00-016", "Error occurred 
while creating glossary term: {0}");
+FAILED_TO_CREATE_GLOSSARY_TERM(500, "ATLAS-500-00-016", "Error occurred 
while creating glossary term: {0}"),
+FAILED_TO_UPDATE_GLOSSARY_TERM(500, "ATLAS-500-00-017", "Error occurred 
while updating glossary term: {0}");
 
 private String errorCode;
 private String errorMessage;
diff --git 
a/intg/src/main/java/org/apache/atlas/model/glossary/AtlasGlossaryTerm.java 
b/intg/src/main/java/org/apache/atlas/model/glossary/AtlasGlossaryTerm.java
index 4fa1538..ee306c6 100644
--- a/intg/src/main/java/org/apache/atlas/model/glossary/AtlasGlossaryTerm.java
+++ b/intg/src/main/java/org/apache/atlas/model/glossary/AtlasGlossaryTerm.java
@@ -176,7 +176,10 @@ public class AtlasGlossaryTerm extends 
AtlasGlossaryBaseObject {
 
 public void setSeeAlso(final Set seeAlso) {
 this.seeAlso = seeAlso;
-hasTerms = true;
+
+if (CollectionUtils.isNotEmpty(seeAlso)) {
+hasTerms = true;
+}
 }
 
 public Set getSynonyms() {
@@ -185,7 +188,10 @@ public class AtlasGlossaryTerm extends 
AtlasGlossaryBaseObject {
 
 public void setSynonyms(final Set synonyms) {
 this.synonyms = synonyms;
-hasTerms = true;
+
+if (CollectionUtils.isNotEmpty(synonyms)) {
+hasTerms = true;
+}
 }
 
 public Set getAntonyms() {
@@ -194,7 +200,10 @@ public class AtlasGlossaryTerm extends 
AtlasGlossaryBaseObject {
 
 public void setAntonyms(final Set antonyms) {
 this.antonyms = antonyms;
-hasTerms = true;
+
+if (CollectionUtils.isNotEmpty(antonyms)) {
+hasTerms = true;
+}
 }
 
 public Set getPreferredTerms() {
@@ -203,7 +212,10 @@ public class AtlasGlossaryTerm extends 
AtlasGlossaryBaseObject {
 
 public void setPreferredTerms(final Set 
preferredTerms) {
 this.preferredTerms = preferredTerms;
-hasTerms = true;
+
+if (CollectionUtils.isNotEmpty(preferredTerms)) {
+hasTerms = true;
+}
 }
 
 public Set getPreferredToTerms() {
@@ -220,7 +232,10 @@ public class AtlasGlossaryTerm extends 
AtlasGlossaryBaseObject {
 
 public void setReplacementTerms(final Set 
replacementTerms) {
 this.replacementTerms = replacementTerms;
-hasTerms = true;
+
+if (CollectionUtils.isNotEmpty(replacementTerms)) {
+hasTerms = true;
+}

[atlas] branch master updated: ATLAS-4261: Bulk Glossary Import Response and Failed Error Message Improvements

[sarath]

This is an automated email from the ASF dual-hosted git repository.

sarath pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/atlas.git


The following commit(s) were added to refs/heads/master by this push:
 new 7d7d1f3  ATLAS-4261: Bulk Glossary Import Response and Failed Error 
Message Improvements
7d7d1f3 is described below

commit 7d7d1f3e1b1be9ea585398eb337208250730bda3
Author: sidmishra 
AuthorDate: Wed Apr 28 23:09:09 2021 -0700

ATLAS-4261: Bulk Glossary Import Response and Failed Error Message 
Improvements

Signed-off-by: Sarath Subramanian 
---
 .../main/java/org/apache/atlas/AtlasErrorCode.java |   4 +-
 .../atlas/model/glossary/AtlasGlossaryTerm.java|  80 ++
 .../model/glossary/AtlasGlossaryTermHeader.java|  80 ++
 .../org/apache/atlas/glossary/GlossaryService.java |  24 ++-
 .../apache/atlas/glossary/GlossaryTermUtils.java   | 171 +
 5 files changed, 262 insertions(+), 97 deletions(-)

diff --git a/intg/src/main/java/org/apache/atlas/AtlasErrorCode.java 
b/intg/src/main/java/org/apache/atlas/AtlasErrorCode.java
index 5ef62d3..2febff4 100644
--- a/intg/src/main/java/org/apache/atlas/AtlasErrorCode.java
+++ b/intg/src/main/java/org/apache/atlas/AtlasErrorCode.java
@@ -208,6 +208,7 @@ public enum AtlasErrorCode {
 GLOSSARY_ALREADY_EXISTS(409, "ATLAS-409-00-007", "Glossary with name {0} 
already exists"),
 GLOSSARY_TERM_ALREADY_EXISTS(409, "ATLAS-409-00-009", "Glossary term with 
qualifiedName {0} already exists"),
 GLOSSARY_CATEGORY_ALREADY_EXISTS(409, "ATLAS-409-00-00A", "Glossary 
category with qualifiedName {0} already exists"),
+GLOSSARY_IMPORT_FAILED(409, "ATLAS-409-00-011", "Glossary import failed"),
 
 // All internal errors go here
 INTERNAL_ERROR(500, "ATLAS-500-00-001", "Internal server error {0}"),
@@ -230,7 +231,8 @@ public enum AtlasErrorCode {
 DATA_ACCESS_LOAD_FAILED(500, "ATLAS-500-00-013", "Load failed: {0}"),
 ENTITY_NOTIFICATION_FAILED(500, "ATLAS-500-00-014", "Notification failed 
for operation: {0} : {1}"),
 FAILED_TO_UPLOAD(500, "ATLAS-500-00-015", "Error occurred while uploading 
the file: {0}"),
-FAILED_TO_CREATE_GLOSSARY_TERM(500, "ATLAS-500-00-016", "Error occurred 
while creating glossary term: {0}");
+FAILED_TO_CREATE_GLOSSARY_TERM(500, "ATLAS-500-00-016", "Error occurred 
while creating glossary term: {0}"),
+FAILED_TO_UPDATE_GLOSSARY_TERM(500, "ATLAS-500-00-017", "Error occurred 
while updating glossary term: {0}");
 
 private String errorCode;
 private String errorMessage;
diff --git 
a/intg/src/main/java/org/apache/atlas/model/glossary/AtlasGlossaryTerm.java 
b/intg/src/main/java/org/apache/atlas/model/glossary/AtlasGlossaryTerm.java
index 4fa1538..ee306c6 100644
--- a/intg/src/main/java/org/apache/atlas/model/glossary/AtlasGlossaryTerm.java
+++ b/intg/src/main/java/org/apache/atlas/model/glossary/AtlasGlossaryTerm.java
@@ -176,7 +176,10 @@ public class AtlasGlossaryTerm extends 
AtlasGlossaryBaseObject {
 
 public void setSeeAlso(final Set seeAlso) {
 this.seeAlso = seeAlso;
-hasTerms = true;
+
+if (CollectionUtils.isNotEmpty(seeAlso)) {
+hasTerms = true;
+}
 }
 
 public Set getSynonyms() {
@@ -185,7 +188,10 @@ public class AtlasGlossaryTerm extends 
AtlasGlossaryBaseObject {
 
 public void setSynonyms(final Set synonyms) {
 this.synonyms = synonyms;
-hasTerms = true;
+
+if (CollectionUtils.isNotEmpty(synonyms)) {
+hasTerms = true;
+}
 }
 
 public Set getAntonyms() {
@@ -194,7 +200,10 @@ public class AtlasGlossaryTerm extends 
AtlasGlossaryBaseObject {
 
 public void setAntonyms(final Set antonyms) {
 this.antonyms = antonyms;
-hasTerms = true;
+
+if (CollectionUtils.isNotEmpty(antonyms)) {
+hasTerms = true;
+}
 }
 
 public Set getPreferredTerms() {
@@ -203,7 +212,10 @@ public class AtlasGlossaryTerm extends 
AtlasGlossaryBaseObject {
 
 public void setPreferredTerms(final Set 
preferredTerms) {
 this.preferredTerms = preferredTerms;
-hasTerms = true;
+
+if (CollectionUtils.isNotEmpty(preferredTerms)) {
+hasTerms = true;
+}
 }
 
 public Set getPreferredToTerms() {
@@ -220,7 +232,10 @@ public class AtlasGlossaryTerm extends 
AtlasGlossaryBaseObject {
 
 public void setReplacementTerms(final Set 
replacementTerms) {
 this.replacementTerms = replacementTerms;
-hasTerms = true;
+
+if (CollectionUtils.isNotEmpty(replacementTerms)) {
+hasTerms = true;
+}
 }
 
 public Set getReplacedBy() {
@@ -229,7 +244,10 @@ public class AtlasGl

[atlas] branch branch-2.0 updated: ATLAS-4256, ATLAS-4258: AtlasTasks - Elegant handling of Failover Scenarios

[sarath]

This is an automated email from the ASF dual-hosted git repository.

sarath pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/atlas.git


The following commit(s) were added to refs/heads/branch-2.0 by this push:
 new af1990c  ATLAS-4256, ATLAS-4258: AtlasTasks - Elegant handling of 
Failover Scenarios
af1990c is described below

commit af1990cfa5e31858798e85ee9425c9b0247f0bb1
Author: Ashutosh Mestry 
AuthorDate: Mon Apr 26 22:02:02 2021 -0700

ATLAS-4256, ATLAS-4258: AtlasTasks - Elegant handling of Failover Scenarios

Signed-off-by: Sarath Subramanian 
(cherry picked from commit eddda699def908895f17384735bf90e08f02d6bf)
---
 .../org/apache/atlas/repository/Constants.java |  2 +-
 .../repository/graph/GraphBackedSearchIndexer.java |  1 +
 .../repository/store/graph/v1/DeleteHandlerV1.java |  2 +-
 .../store/graph/v2/AtlasGraphUtilsV2.java  | 26 ++
 .../store/graph/v2/EntityGraphMapper.java  | 42 --
 .../store/graph/v2/EntityGraphRetriever.java   |  6 +++-
 .../tasks/ClassificationPropagateTaskFactory.java  |  4 +--
 .../v2/tasks/ClassificationPropagationTasks.java   |  3 +-
 .../store/graph/v2/tasks/ClassificationTask.java   | 26 --
 .../apache/atlas/tasks/TaskFactoryRegistry.java| 39 
 .../org/apache/atlas/tasks/TaskManagement.java |  6 ++--
 .../ClassificationPropagationWithTasksTest.java|  4 +--
 .../apache/atlas/web/resources/AdminResource.java  |  5 ---
 13 files changed, 137 insertions(+), 29 deletions(-)

diff --git a/common/src/main/java/org/apache/atlas/repository/Constants.java 
b/common/src/main/java/org/apache/atlas/repository/Constants.java
index 4df38a5..ffcec97 100644
--- a/common/src/main/java/org/apache/atlas/repository/Constants.java
+++ b/common/src/main/java/org/apache/atlas/repository/Constants.java
@@ -94,7 +94,7 @@ public final class Constants {
 public static final String PROPAGATED_CLASSIFICATION_NAMES_KEY  = 
encodePropertyKey(INTERNAL_PROPERTY_KEY_PREFIX + 
"propagatedClassificationNames");
 public static final String CUSTOM_ATTRIBUTES_PROPERTY_KEY   = 
encodePropertyKey(INTERNAL_PROPERTY_KEY_PREFIX + "customAttributes");
 public static final String LABELS_PROPERTY_KEY  = 
encodePropertyKey(INTERNAL_PROPERTY_KEY_PREFIX + "labels");
-public static final String PENDING_TASKS_PROPERTY_KEY   = 
encodePropertyKey(INTERNAL_PROPERTY_KEY_PREFIX + "pendingTasks");
+public static final String EDGE_PENDING_TASKS_PROPERTY_KEY  = 
encodePropertyKey(RELATIONSHIP_PROPERTY_KEY_PREFIX + "__pendingTasks");
 
 /**
  * Patch vertices property keys.
diff --git 
a/repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java
 
b/repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java
index 47c8b14..c73449e 100755
--- 
a/repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java
+++ 
b/repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java
@@ -65,6 +65,7 @@ import static 
org.apache.atlas.repository.store.graph.v2.AtlasGraphUtilsV2.isRef
 import static 
org.apache.atlas.type.AtlasStructType.UNIQUE_ATTRIBUTE_SHADE_PROPERTY_PREFIX;
 import static org.apache.atlas.type.AtlasTypeUtil.isArrayType;
 import static org.apache.atlas.type.AtlasTypeUtil.isMapType;
+import static org.apache.atlas.type.Constants.PENDING_TASKS_PROPERTY_KEY;
 
 
 /**
diff --git 
a/repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerV1.java
 
b/repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerV1.java
index 20d5e6f..f118ae6 100644
--- 
a/repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerV1.java
+++ 
b/repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerV1.java
@@ -1203,7 +1203,7 @@ public abstract class DeleteHandlerV1 {
 Map taskParams = 
ClassificationTask.toParameters(relationshipEdgeId, relationship);
 AtlasTask   task   = 
taskManagement.createTask(taskType, currentUser, taskParams);
 
-AtlasGraphUtilsV2.addEncodedProperty(relationshipEdge, 
PENDING_TASKS_PROPERTY_KEY, task.getGuid());
+AtlasGraphUtilsV2.addItemToListProperty(relationshipEdge, 
EDGE_PENDING_TASKS_PROPERTY_KEY, task.getGuid());
 
 RequestContext.get().queueTask(task);
 }
diff --git 
a/repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasGraphUtilsV2.java
 
b/repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasGraphUtilsV2.java
index 8d4fdf3..0a94708 100644
--- 
a/repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasGraphUtilsV2.java
+++ 
b/repository/src/main/java/org/apache/atlas/repository/store/graph/

[atlas] branch master updated: ATLAS-4256, ATLAS-4258: AtlasTasks - Elegant handling of Failover Scenarios

[sarath]

This is an automated email from the ASF dual-hosted git repository.

sarath pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/atlas.git


The following commit(s) were added to refs/heads/master by this push:
 new eddda69  ATLAS-4256, ATLAS-4258: AtlasTasks - Elegant handling of 
Failover Scenarios
eddda69 is described below

commit eddda699def908895f17384735bf90e08f02d6bf
Author: Ashutosh Mestry 
AuthorDate: Mon Apr 26 22:02:02 2021 -0700

ATLAS-4256, ATLAS-4258: AtlasTasks - Elegant handling of Failover Scenarios

Signed-off-by: Sarath Subramanian 
---
 .../org/apache/atlas/repository/Constants.java |  2 +-
 .../repository/graph/GraphBackedSearchIndexer.java |  1 +
 .../repository/store/graph/v1/DeleteHandlerV1.java |  2 +-
 .../store/graph/v2/AtlasGraphUtilsV2.java  | 26 ++
 .../store/graph/v2/EntityGraphMapper.java  | 42 --
 .../store/graph/v2/EntityGraphRetriever.java   |  6 +++-
 .../tasks/ClassificationPropagateTaskFactory.java  |  4 +--
 .../v2/tasks/ClassificationPropagationTasks.java   |  3 +-
 .../store/graph/v2/tasks/ClassificationTask.java   | 26 --
 .../apache/atlas/tasks/TaskFactoryRegistry.java| 39 
 .../org/apache/atlas/tasks/TaskManagement.java |  6 ++--
 .../ClassificationPropagationWithTasksTest.java|  4 +--
 .../apache/atlas/web/resources/AdminResource.java  |  5 ---
 13 files changed, 137 insertions(+), 29 deletions(-)

diff --git a/common/src/main/java/org/apache/atlas/repository/Constants.java 
b/common/src/main/java/org/apache/atlas/repository/Constants.java
index 4df38a5..ffcec97 100644
--- a/common/src/main/java/org/apache/atlas/repository/Constants.java
+++ b/common/src/main/java/org/apache/atlas/repository/Constants.java
@@ -94,7 +94,7 @@ public final class Constants {
 public static final String PROPAGATED_CLASSIFICATION_NAMES_KEY  = 
encodePropertyKey(INTERNAL_PROPERTY_KEY_PREFIX + 
"propagatedClassificationNames");
 public static final String CUSTOM_ATTRIBUTES_PROPERTY_KEY   = 
encodePropertyKey(INTERNAL_PROPERTY_KEY_PREFIX + "customAttributes");
 public static final String LABELS_PROPERTY_KEY  = 
encodePropertyKey(INTERNAL_PROPERTY_KEY_PREFIX + "labels");
-public static final String PENDING_TASKS_PROPERTY_KEY   = 
encodePropertyKey(INTERNAL_PROPERTY_KEY_PREFIX + "pendingTasks");
+public static final String EDGE_PENDING_TASKS_PROPERTY_KEY  = 
encodePropertyKey(RELATIONSHIP_PROPERTY_KEY_PREFIX + "__pendingTasks");
 
 /**
  * Patch vertices property keys.
diff --git 
a/repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java
 
b/repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java
index 276343e..cc727c6 100755
--- 
a/repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java
+++ 
b/repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java
@@ -65,6 +65,7 @@ import static 
org.apache.atlas.repository.store.graph.v2.AtlasGraphUtilsV2.isRef
 import static 
org.apache.atlas.type.AtlasStructType.UNIQUE_ATTRIBUTE_SHADE_PROPERTY_PREFIX;
 import static org.apache.atlas.type.AtlasTypeUtil.isArrayType;
 import static org.apache.atlas.type.AtlasTypeUtil.isMapType;
+import static org.apache.atlas.type.Constants.PENDING_TASKS_PROPERTY_KEY;
 
 
 /**
diff --git 
a/repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerV1.java
 
b/repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerV1.java
index 20d5e6f..f118ae6 100644
--- 
a/repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerV1.java
+++ 
b/repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerV1.java
@@ -1203,7 +1203,7 @@ public abstract class DeleteHandlerV1 {
 Map taskParams = 
ClassificationTask.toParameters(relationshipEdgeId, relationship);
 AtlasTask   task   = 
taskManagement.createTask(taskType, currentUser, taskParams);
 
-AtlasGraphUtilsV2.addEncodedProperty(relationshipEdge, 
PENDING_TASKS_PROPERTY_KEY, task.getGuid());
+AtlasGraphUtilsV2.addItemToListProperty(relationshipEdge, 
EDGE_PENDING_TASKS_PROPERTY_KEY, task.getGuid());
 
 RequestContext.get().queueTask(task);
 }
diff --git 
a/repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasGraphUtilsV2.java
 
b/repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasGraphUtilsV2.java
index 8d4fdf3..0a94708 100644
--- 
a/repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasGraphUtilsV2.java
+++ 
b/repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasGraphUtilsV2.java
@@ -47,6 +47,7 @@ import org.apache.atlas.type.AtlasType

[atlas] branch branch-2.0 updated: ATLAS-4195: Entity comparator doesn't account for inactive relations

[sarath]

This is an automated email from the ASF dual-hosted git repository.

sarath pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/atlas.git


The following commit(s) were added to refs/heads/branch-2.0 by this push:
 new c108ed4  ATLAS-4195: Entity comparator doesn't account for inactive 
relations
c108ed4 is described below

commit c108ed49dab88d7270158fc763239c128d9b6794
Author: Deep Singh 
AuthorDate: Mon Apr 26 11:26:28 2021 -0500

ATLAS-4195: Entity comparator doesn't account for inactive relations

Signed-off-by: Sarath Subramanian 
(cherry picked from commit 683e2a91db50850014aa92a2abfa668d14624f1a)
---
 .../store/graph/v2/AtlasEntityStoreV2.java |   8 +-
 .../store/graph/v2/EntityGraphRetriever.java   |  19 +++-
 .../store/graph/v2/DifferentialAuditsTest.java |  90 
 .../src/test/resources/json/entities/db01.json |  23 
 .../test/resources/json/entities/tbl01-2cols.json  | 116 +
 .../src/test/resources/json/entities/tbl01.json|  94 +
 6 files changed, 346 insertions(+), 4 deletions(-)

diff --git 
a/repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java
 
b/repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java
index c133920..65b92ab 100644
--- 
a/repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java
+++ 
b/repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java
@@ -18,6 +18,7 @@
 package org.apache.atlas.repository.store.graph.v2;
 
 
+import com.google.common.annotations.VisibleForTesting;
 import org.apache.atlas.AtlasErrorCode;
 import org.apache.atlas.DeleteType;
 import org.apache.atlas.GraphTransactionInterceptor;
@@ -109,7 +110,7 @@ public class AtlasEntityStoreV2 implements AtlasEntityStore 
{
 private final IAtlasEntityChangeNotifier entityChangeNotifier;
 private final EntityGraphMapper  entityGraphMapper;
 private final EntityGraphRetriever   entityRetriever;
-private final booleanstoreDifferentialAudits;
+private   booleanstoreDifferentialAudits;
 
 @Inject
 public AtlasEntityStoreV2(AtlasGraph graph, DeleteHandlerDelegate 
deleteDelegate, AtlasTypeRegistry typeRegistry,
@@ -123,6 +124,11 @@ public class AtlasEntityStoreV2 implements 
AtlasEntityStore {
 this.storeDifferentialAudits = STORE_DIFFERENTIAL_AUDITS.getBoolean();
 }
 
+@VisibleForTesting
+public void setStoreDifferentialAudits(boolean val) {
+this.storeDifferentialAudits = val;
+}
+
 @Override
 @GraphTransaction
 public List getEntityGUIDS(final String typename) throws 
AtlasBaseException {
diff --git 
a/repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java
 
b/repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java
index b790023..b6f1ef7 100644
--- 
a/repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java
+++ 
b/repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java
@@ -271,7 +271,7 @@ public class EntityGraphRetriever {
 Object ret = null;
 
 try {
-ret = getVertexAttribute(entityVertex, attribute);
+ret = getVertexAttributeIgnoreInactive(entityVertex, attribute);
 } catch (AtlasBaseException excp) {
 // ignore
 }
@@ -966,6 +966,10 @@ public class EntityGraphRetriever {
 }
 
 private Object mapVertexToAttribute(AtlasVertex entityVertex, 
AtlasAttribute attribute, AtlasEntityExtInfo entityExtInfo, final boolean 
isMinExtInfo, boolean includeReferences) throws AtlasBaseException {
+return mapVertexToAttribute(entityVertex, attribute, entityExtInfo, 
isMinExtInfo, includeReferences, false);
+}
+
+private Object mapVertexToAttribute(AtlasVertex entityVertex, 
AtlasAttribute attribute, AtlasEntityExtInfo entityExtInfo, final boolean 
isMinExtInfo, boolean includeReferences, boolean ignoreInactive) throws 
AtlasBaseException {
 Objectret= null;
 AtlasType attrType   = attribute.getAttributeType();
 StringedgeLabel  = attribute.getRelationshipEdgeLabel();
@@ -1011,7 +1015,7 @@ public class EntityGraphRetriever {
 if (attribute.getAttributeDef().isSoftReferenced()) {
 ret = mapVertexToArrayForSoftRef(entityVertex, 
attribute, entityExtInfo, isMinExtInfo);
 } else {
-ret = mapVertexToArray(entityVertex, entityExtInfo, 
isOwnedAttribute, attribute, isMinExtInfo, includeReferences);
+ret = mapVertexToArray(entityVertex, entityExtInfo, 
isOwnedAttribute, attribute, isMinExtInfo, includeReferences, ignoreInactive

[atlas] branch master updated: ATLAS-4195: Entity comparator doesn't account for inactive relations

[sarath]

This is an automated email from the ASF dual-hosted git repository.

sarath pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/atlas.git


The following commit(s) were added to refs/heads/master by this push:
 new 683e2a9  ATLAS-4195: Entity comparator doesn't account for inactive 
relations
683e2a9 is described below

commit 683e2a91db50850014aa92a2abfa668d14624f1a
Author: Deep Singh 
AuthorDate: Mon Apr 26 11:26:28 2021 -0500

ATLAS-4195: Entity comparator doesn't account for inactive relations

Signed-off-by: Sarath Subramanian 
---
 .../store/graph/v2/AtlasEntityStoreV2.java |   8 +-
 .../store/graph/v2/EntityGraphRetriever.java   |  19 +++-
 .../store/graph/v2/DifferentialAuditsTest.java |  90 
 .../src/test/resources/json/entities/db01.json |  23 
 .../test/resources/json/entities/tbl01-2cols.json  | 116 +
 .../src/test/resources/json/entities/tbl01.json|  94 +
 6 files changed, 346 insertions(+), 4 deletions(-)

diff --git 
a/repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java
 
b/repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java
index c133920..65b92ab 100644
--- 
a/repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java
+++ 
b/repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java
@@ -18,6 +18,7 @@
 package org.apache.atlas.repository.store.graph.v2;
 
 
+import com.google.common.annotations.VisibleForTesting;
 import org.apache.atlas.AtlasErrorCode;
 import org.apache.atlas.DeleteType;
 import org.apache.atlas.GraphTransactionInterceptor;
@@ -109,7 +110,7 @@ public class AtlasEntityStoreV2 implements AtlasEntityStore 
{
 private final IAtlasEntityChangeNotifier entityChangeNotifier;
 private final EntityGraphMapper  entityGraphMapper;
 private final EntityGraphRetriever   entityRetriever;
-private final booleanstoreDifferentialAudits;
+private   booleanstoreDifferentialAudits;
 
 @Inject
 public AtlasEntityStoreV2(AtlasGraph graph, DeleteHandlerDelegate 
deleteDelegate, AtlasTypeRegistry typeRegistry,
@@ -123,6 +124,11 @@ public class AtlasEntityStoreV2 implements 
AtlasEntityStore {
 this.storeDifferentialAudits = STORE_DIFFERENTIAL_AUDITS.getBoolean();
 }
 
+@VisibleForTesting
+public void setStoreDifferentialAudits(boolean val) {
+this.storeDifferentialAudits = val;
+}
+
 @Override
 @GraphTransaction
 public List getEntityGUIDS(final String typename) throws 
AtlasBaseException {
diff --git 
a/repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java
 
b/repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java
index b790023..b6f1ef7 100644
--- 
a/repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java
+++ 
b/repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java
@@ -271,7 +271,7 @@ public class EntityGraphRetriever {
 Object ret = null;
 
 try {
-ret = getVertexAttribute(entityVertex, attribute);
+ret = getVertexAttributeIgnoreInactive(entityVertex, attribute);
 } catch (AtlasBaseException excp) {
 // ignore
 }
@@ -966,6 +966,10 @@ public class EntityGraphRetriever {
 }
 
 private Object mapVertexToAttribute(AtlasVertex entityVertex, 
AtlasAttribute attribute, AtlasEntityExtInfo entityExtInfo, final boolean 
isMinExtInfo, boolean includeReferences) throws AtlasBaseException {
+return mapVertexToAttribute(entityVertex, attribute, entityExtInfo, 
isMinExtInfo, includeReferences, false);
+}
+
+private Object mapVertexToAttribute(AtlasVertex entityVertex, 
AtlasAttribute attribute, AtlasEntityExtInfo entityExtInfo, final boolean 
isMinExtInfo, boolean includeReferences, boolean ignoreInactive) throws 
AtlasBaseException {
 Objectret= null;
 AtlasType attrType   = attribute.getAttributeType();
 StringedgeLabel  = attribute.getRelationshipEdgeLabel();
@@ -1011,7 +1015,7 @@ public class EntityGraphRetriever {
 if (attribute.getAttributeDef().isSoftReferenced()) {
 ret = mapVertexToArrayForSoftRef(entityVertex, 
attribute, entityExtInfo, isMinExtInfo);
 } else {
-ret = mapVertexToArray(entityVertex, entityExtInfo, 
isOwnedAttribute, attribute, isMinExtInfo, includeReferences);
+ret = mapVertexToArray(entityVertex, entityExtInfo, 
isOwnedAttribute, attribute, isMinExtInfo, includeReferences, ignoreInactive);
 }
 }
 }
@@ -1158,7 +1162,8

[atlas] branch branch-2.0 updated: ATLAS-4257: Upgrade groovy version to 2.5.14

[sarath]

This is an automated email from the ASF dual-hosted git repository.

sarath pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/atlas.git


The following commit(s) were added to refs/heads/branch-2.0 by this push:
 new 68d44e2  ATLAS-4257: Upgrade groovy version to 2.5.14
68d44e2 is described below

commit 68d44e2fb7ae2df669a8e5bd37e0ff92287fecb4
Author: Radhika Kundam 
AuthorDate: Thu Apr 22 15:30:30 2021 -0700

ATLAS-4257: Upgrade groovy version to 2.5.14

Signed-off-by: Sarath Subramanian 
(cherry picked from commit 3a0612f58ad2c49c00541305de9b71b650907420)
---
 pom.xml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/pom.xml b/pom.xml
index 8e3e32e..55cad6e 100644
--- a/pom.xml
+++ b/pom.xml
@@ -756,7 +756,7 @@
 2C
 2.18.1
 6.9.4
-3.4.8
+3.4.10
 5.0.3
 3.5.7
 

[atlas] branch master updated: ATLAS-4257: Upgrade groovy version to 2.5.14

[sarath]

This is an automated email from the ASF dual-hosted git repository.

sarath pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/atlas.git


The following commit(s) were added to refs/heads/master by this push:
 new 3a0612f  ATLAS-4257: Upgrade groovy version to 2.5.14
3a0612f is described below

commit 3a0612f58ad2c49c00541305de9b71b650907420
Author: Radhika Kundam 
AuthorDate: Thu Apr 22 15:30:30 2021 -0700

ATLAS-4257: Upgrade groovy version to 2.5.14

Signed-off-by: Sarath Subramanian 
---
 pom.xml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/pom.xml b/pom.xml
index 092ba09..9719c1b 100644
--- a/pom.xml
+++ b/pom.xml
@@ -756,7 +756,7 @@
 2C
 2.18.1
 6.9.4
-3.4.8
+3.4.10
 5.0.3
 3.5.7
 

[atlas] branch branch-2.0 updated: ATLAS-4231: UI Create icons for new entity types for Google Cloud Storage

[sarath]

This is an automated email from the ASF dual-hosted git repository.

sarath pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/atlas.git


The following commit(s) were added to refs/heads/branch-2.0 by this push:
 new a2a0296  ATLAS-4231: UI Create icons for new entity types for Google 
Cloud Storage
a2a0296 is described below

commit a2a02963bb48594f00f94cf80433e164167e7bfd
Author: prasad pawar 
AuthorDate: Thu Apr 1 12:48:27 2021 +0530

ATLAS-4231: UI Create icons for new entity types for Google Cloud Storage

Signed-off-by: Sarath Subramanian 
(cherry picked from commit 404f8a1a2d1c6d965b66d4d25f7c84543dde8626)
---
 dashboardv2/public/img/entity-icon/disabled/gcp.png | Bin 0 -> 3977 bytes
 dashboardv2/public/img/entity-icon/gcp.png  | Bin 0 -> 4649 bytes
 dashboardv3/public/img/entity-icon/disabled/gcp.png | Bin 0 -> 3977 bytes
 dashboardv3/public/img/entity-icon/gcp.png  | Bin 0 -> 4649 bytes
 4 files changed, 0 insertions(+), 0 deletions(-)

diff --git a/dashboardv2/public/img/entity-icon/disabled/gcp.png 
b/dashboardv2/public/img/entity-icon/disabled/gcp.png
new file mode 100644
index 000..e7d83fe
Binary files /dev/null and 
b/dashboardv2/public/img/entity-icon/disabled/gcp.png differ
diff --git a/dashboardv2/public/img/entity-icon/gcp.png 
b/dashboardv2/public/img/entity-icon/gcp.png
new file mode 100644
index 000..4740e8d
Binary files /dev/null and b/dashboardv2/public/img/entity-icon/gcp.png differ
diff --git a/dashboardv3/public/img/entity-icon/disabled/gcp.png 
b/dashboardv3/public/img/entity-icon/disabled/gcp.png
new file mode 100644
index 000..e7d83fe
Binary files /dev/null and 
b/dashboardv3/public/img/entity-icon/disabled/gcp.png differ
diff --git a/dashboardv3/public/img/entity-icon/gcp.png 
b/dashboardv3/public/img/entity-icon/gcp.png
new file mode 100644
index 000..4740e8d
Binary files /dev/null and b/dashboardv3/public/img/entity-icon/gcp.png differ

[atlas] branch master updated: ATLAS-4231: UI Create icons for new entity types for Google Cloud Storage

[sarath]

This is an automated email from the ASF dual-hosted git repository.

sarath pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/atlas.git


The following commit(s) were added to refs/heads/master by this push:
 new 404f8a1  ATLAS-4231: UI Create icons for new entity types for Google 
Cloud Storage
404f8a1 is described below

commit 404f8a1a2d1c6d965b66d4d25f7c84543dde8626
Author: prasad pawar 
AuthorDate: Thu Apr 1 12:48:27 2021 +0530

ATLAS-4231: UI Create icons for new entity types for Google Cloud Storage

Signed-off-by: Sarath Subramanian 
---
 dashboardv2/public/img/entity-icon/disabled/gcp.png | Bin 0 -> 3977 bytes
 dashboardv2/public/img/entity-icon/gcp.png  | Bin 0 -> 4649 bytes
 dashboardv3/public/img/entity-icon/disabled/gcp.png | Bin 0 -> 3977 bytes
 dashboardv3/public/img/entity-icon/gcp.png  | Bin 0 -> 4649 bytes
 4 files changed, 0 insertions(+), 0 deletions(-)

diff --git a/dashboardv2/public/img/entity-icon/disabled/gcp.png 
b/dashboardv2/public/img/entity-icon/disabled/gcp.png
new file mode 100644
index 000..e7d83fe
Binary files /dev/null and 
b/dashboardv2/public/img/entity-icon/disabled/gcp.png differ
diff --git a/dashboardv2/public/img/entity-icon/gcp.png 
b/dashboardv2/public/img/entity-icon/gcp.png
new file mode 100644
index 000..4740e8d
Binary files /dev/null and b/dashboardv2/public/img/entity-icon/gcp.png differ
diff --git a/dashboardv3/public/img/entity-icon/disabled/gcp.png 
b/dashboardv3/public/img/entity-icon/disabled/gcp.png
new file mode 100644
index 000..e7d83fe
Binary files /dev/null and 
b/dashboardv3/public/img/entity-icon/disabled/gcp.png differ
diff --git a/dashboardv3/public/img/entity-icon/gcp.png 
b/dashboardv3/public/img/entity-icon/gcp.png
new file mode 100644
index 000..4740e8d
Binary files /dev/null and b/dashboardv3/public/img/entity-icon/gcp.png differ

[atlas] branch branch-2.0 updated: ATLAS-4191: Improve Bulk Glossary Import to support import with relations

[sarath]

This is an automated email from the ASF dual-hosted git repository.

sarath pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/atlas.git


The following commit(s) were added to refs/heads/branch-2.0 by this push:
 new 55723a3  ATLAS-4191: Improve Bulk Glossary Import to support import 
with relations
55723a3 is described below

commit 55723a3b749afb5f98ce5cac972a51368d63dc37
Author: sidmishra 
AuthorDate: Wed Apr 7 09:57:50 2021 -0700

ATLAS-4191: Improve Bulk Glossary Import to support import with relations

Signed-off-by: Sarath Subramanian 
(cherry picked from commit ff662427e54f675200646f52eb13db6084ef4ef3)
---
 .../main/java/org/apache/atlas/AtlasErrorCode.java |   1 +
 .../atlas/model/glossary/AtlasGlossaryTerm.java|  16 +++
 .../org/apache/atlas/glossary/GlossaryService.java |  95 ++
 .../apache/atlas/glossary/GlossaryTermUtils.java   | 109 -
 .../store/graph/v2/AtlasEntityStoreV2.java |  34 +++
 .../main/java/org/apache/atlas/util/FileUtils.java |  44 +
 .../apache/atlas/glossary/GlossaryServiceTest.java |  29 +-
 .../template_with_circular_relationship.csv|   4 +
 .../src/test/resources/excelFiles/invalid_xls.xls  |   5 +
 9 files changed, 205 insertions(+), 132 deletions(-)

diff --git a/intg/src/main/java/org/apache/atlas/AtlasErrorCode.java 
b/intg/src/main/java/org/apache/atlas/AtlasErrorCode.java
index 884f81f..773fae2 100644
--- a/intg/src/main/java/org/apache/atlas/AtlasErrorCode.java
+++ b/intg/src/main/java/org/apache/atlas/AtlasErrorCode.java
@@ -171,6 +171,7 @@ public enum AtlasErrorCode {
 INVALID_BUSINESS_ATTRIBUTES_IMPORT_DATA(400, "ATLAS-400-00-099","The 
uploaded file was not processed due to following errors : {0}"),
 ATTRIBUTE_NAME_INVALID_CHARS(400, "ATLAS-400-00-09A", "{0}: invalid name. 
Attribute names must begin with a letter followed by a sequence of letters, 
numbers, or '_' characters"),
 NO_DATA_FOUND(400, "ATLAS-400-00-09B", "No data found in the uploaded 
file"),
+NOT_VALID_FILE(400, "ATLAS-400-00-09C", "Invalid {0} file"),
 UNAUTHORIZED_ACCESS(403, "ATLAS-403-00-001", "{0} is not authorized to 
perform {1}"),
 
 // All Not found enums go here
diff --git 
a/intg/src/main/java/org/apache/atlas/model/glossary/AtlasGlossaryTerm.java 
b/intg/src/main/java/org/apache/atlas/model/glossary/AtlasGlossaryTerm.java
index 8d0b7c5..4fa1538 100644
--- a/intg/src/main/java/org/apache/atlas/model/glossary/AtlasGlossaryTerm.java
+++ b/intg/src/main/java/org/apache/atlas/model/glossary/AtlasGlossaryTerm.java
@@ -286,6 +286,22 @@ public class AtlasGlossaryTerm extends 
AtlasGlossaryBaseObject {
 hasTerms = true;
 }
 
+public boolean containAnyRelation() {
+return (CollectionUtils.isNotEmpty(getTranslationTerms()) ||
+CollectionUtils.isNotEmpty(getValidValuesFor()) ||
+CollectionUtils.isNotEmpty(getSynonyms()) ||
+CollectionUtils.isNotEmpty(getReplacedBy()) ||
+CollectionUtils.isNotEmpty(getValidValues()) ||
+CollectionUtils.isNotEmpty(getReplacementTerms()) ||
+CollectionUtils.isNotEmpty(getSeeAlso()) ||
+CollectionUtils.isNotEmpty(getTranslatedTerms()) ||
+CollectionUtils.isNotEmpty(getIsA()) ||
+CollectionUtils.isNotEmpty(getAntonyms()) ||
+CollectionUtils.isNotEmpty(getClassifies()) ||
+CollectionUtils.isNotEmpty(getPreferredToTerms()) ||
+CollectionUtils.isNotEmpty(getPreferredTerms()));
+}
+
 @JsonIgnore
 public String toAuditString() {
 AtlasGlossaryTerm t = new AtlasGlossaryTerm();
diff --git 
a/repository/src/main/java/org/apache/atlas/glossary/GlossaryService.java 
b/repository/src/main/java/org/apache/atlas/glossary/GlossaryService.java
index d156700..b12c0b8 100644
--- a/repository/src/main/java/org/apache/atlas/glossary/GlossaryService.java
+++ b/repository/src/main/java/org/apache/atlas/glossary/GlossaryService.java
@@ -44,7 +44,6 @@ import org.slf4j.LoggerFactory;
 import org.springframework.stereotype.Service;
 
 import javax.inject.Inject;
-import java.io.IOException;
 import java.io.InputStream;
 import java.util.ArrayList;
 import java.util.Collection;
@@ -394,8 +393,12 @@ public class GlossaryService {
 return ret;
 }
 
-@GraphTransaction
 public AtlasGlossaryTerm updateTerm(AtlasGlossaryTerm atlasGlossaryTerm) 
throws AtlasBaseException {
+return updateTerm(atlasGlossaryTerm, true);
+}
+
+@GraphTransaction
+public AtlasGlossaryTerm updateTerm(AtlasGlossaryTerm atlasGlossaryTerm, 
boolean ignoreUpdateIfTermExists) throws AtlasBaseException {
 if (DEBUG_ENABLED) {
 LOG.debug("==> GlossaryService.updateTerm({})

[atlas] branch master updated: ATLAS-4191: Improve Bulk Glossary Import to support import with relations

[sarath]

This is an automated email from the ASF dual-hosted git repository.

sarath pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/atlas.git


The following commit(s) were added to refs/heads/master by this push:
 new ff66242  ATLAS-4191: Improve Bulk Glossary Import to support import 
with relations
ff66242 is described below

commit ff662427e54f675200646f52eb13db6084ef4ef3
Author: sidmishra 
AuthorDate: Wed Apr 7 09:57:50 2021 -0700

ATLAS-4191: Improve Bulk Glossary Import to support import with relations

Signed-off-by: Sarath Subramanian 
---
 .../main/java/org/apache/atlas/AtlasErrorCode.java |   1 +
 .../atlas/model/glossary/AtlasGlossaryTerm.java|  16 +++
 .../org/apache/atlas/glossary/GlossaryService.java |  95 ++
 .../apache/atlas/glossary/GlossaryTermUtils.java   | 109 -
 .../store/graph/v2/AtlasEntityStoreV2.java |  34 +++
 .../main/java/org/apache/atlas/util/FileUtils.java |  44 +
 .../apache/atlas/glossary/GlossaryServiceTest.java |  29 +-
 .../template_with_circular_relationship.csv|   4 +
 .../src/test/resources/excelFiles/invalid_xls.xls  |   5 +
 9 files changed, 205 insertions(+), 132 deletions(-)

diff --git a/intg/src/main/java/org/apache/atlas/AtlasErrorCode.java 
b/intg/src/main/java/org/apache/atlas/AtlasErrorCode.java
index 884f81f..773fae2 100644
--- a/intg/src/main/java/org/apache/atlas/AtlasErrorCode.java
+++ b/intg/src/main/java/org/apache/atlas/AtlasErrorCode.java
@@ -171,6 +171,7 @@ public enum AtlasErrorCode {
 INVALID_BUSINESS_ATTRIBUTES_IMPORT_DATA(400, "ATLAS-400-00-099","The 
uploaded file was not processed due to following errors : {0}"),
 ATTRIBUTE_NAME_INVALID_CHARS(400, "ATLAS-400-00-09A", "{0}: invalid name. 
Attribute names must begin with a letter followed by a sequence of letters, 
numbers, or '_' characters"),
 NO_DATA_FOUND(400, "ATLAS-400-00-09B", "No data found in the uploaded 
file"),
+NOT_VALID_FILE(400, "ATLAS-400-00-09C", "Invalid {0} file"),
 UNAUTHORIZED_ACCESS(403, "ATLAS-403-00-001", "{0} is not authorized to 
perform {1}"),
 
 // All Not found enums go here
diff --git 
a/intg/src/main/java/org/apache/atlas/model/glossary/AtlasGlossaryTerm.java 
b/intg/src/main/java/org/apache/atlas/model/glossary/AtlasGlossaryTerm.java
index 8d0b7c5..4fa1538 100644
--- a/intg/src/main/java/org/apache/atlas/model/glossary/AtlasGlossaryTerm.java
+++ b/intg/src/main/java/org/apache/atlas/model/glossary/AtlasGlossaryTerm.java
@@ -286,6 +286,22 @@ public class AtlasGlossaryTerm extends 
AtlasGlossaryBaseObject {
 hasTerms = true;
 }
 
+public boolean containAnyRelation() {
+return (CollectionUtils.isNotEmpty(getTranslationTerms()) ||
+CollectionUtils.isNotEmpty(getValidValuesFor()) ||
+CollectionUtils.isNotEmpty(getSynonyms()) ||
+CollectionUtils.isNotEmpty(getReplacedBy()) ||
+CollectionUtils.isNotEmpty(getValidValues()) ||
+CollectionUtils.isNotEmpty(getReplacementTerms()) ||
+CollectionUtils.isNotEmpty(getSeeAlso()) ||
+CollectionUtils.isNotEmpty(getTranslatedTerms()) ||
+CollectionUtils.isNotEmpty(getIsA()) ||
+CollectionUtils.isNotEmpty(getAntonyms()) ||
+CollectionUtils.isNotEmpty(getClassifies()) ||
+CollectionUtils.isNotEmpty(getPreferredToTerms()) ||
+CollectionUtils.isNotEmpty(getPreferredTerms()));
+}
+
 @JsonIgnore
 public String toAuditString() {
 AtlasGlossaryTerm t = new AtlasGlossaryTerm();
diff --git 
a/repository/src/main/java/org/apache/atlas/glossary/GlossaryService.java 
b/repository/src/main/java/org/apache/atlas/glossary/GlossaryService.java
index d156700..b12c0b8 100644
--- a/repository/src/main/java/org/apache/atlas/glossary/GlossaryService.java
+++ b/repository/src/main/java/org/apache/atlas/glossary/GlossaryService.java
@@ -44,7 +44,6 @@ import org.slf4j.LoggerFactory;
 import org.springframework.stereotype.Service;
 
 import javax.inject.Inject;
-import java.io.IOException;
 import java.io.InputStream;
 import java.util.ArrayList;
 import java.util.Collection;
@@ -394,8 +393,12 @@ public class GlossaryService {
 return ret;
 }
 
-@GraphTransaction
 public AtlasGlossaryTerm updateTerm(AtlasGlossaryTerm atlasGlossaryTerm) 
throws AtlasBaseException {
+return updateTerm(atlasGlossaryTerm, true);
+}
+
+@GraphTransaction
+public AtlasGlossaryTerm updateTerm(AtlasGlossaryTerm atlasGlossaryTerm, 
boolean ignoreUpdateIfTermExists) throws AtlasBaseException {
 if (DEBUG_ENABLED) {
 LOG.debug("==> GlossaryService.updateTerm({})", atlasGlossaryTerm);
 }
@@ -412,9 +415,12 @@ public class GlossaryServi

[atlas] branch branch-2.0 updated: ATLAS-4129: Glossary Bulk Import Bugs consolidated fix (ATLAS-4128, ATLAS-4131, ATLAS-4160, ATLAS-4130, ATLAS-4129)

[sarath]

This is an automated email from the ASF dual-hosted git repository.

sarath pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/atlas.git


The following commit(s) were added to refs/heads/branch-2.0 by this push:
 new b5b6ca0  ATLAS-4129: Glossary Bulk Import Bugs consolidated fix 
(ATLAS-4128, ATLAS-4131, ATLAS-4160, ATLAS-4130, ATLAS-4129)
b5b6ca0 is described below

commit b5b6ca0e8d09fae070aa3555373f2946892057bd
Author: mayanknj 
AuthorDate: Sat Mar 27 00:57:40 2021 +0530

ATLAS-4129: Glossary Bulk Import Bugs consolidated fix (ATLAS-4128, 
ATLAS-4131, ATLAS-4160, ATLAS-4130, ATLAS-4129)

Signed-off-by: Sarath Subramanian 
(cherry picked from commit 842890d27bf739a0949e82b58d9db1f1dd6111ea)
---
 .../main/java/org/apache/atlas/AtlasClientV2.java  |  4 +-
 .../atlas/bulkimport/BulkImportResponse.java   | 50 +++---
 .../org/apache/atlas/glossary/GlossaryService.java | 44 +++
 .../apache/atlas/glossary/GlossaryTermUtils.java   | 28 
 .../store/graph/v2/AtlasEntityStoreV2.java | 26 +--
 .../apache/atlas/glossary/GlossaryServiceTest.java | 32 --
 .../org/apache/atlas/web/rest/GlossaryREST.java|  5 ++-
 .../atlas/web/integration/GlossaryClientV2IT.java  | 20 +++--
 8 files changed, 125 insertions(+), 84 deletions(-)

diff --git a/client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java 
b/client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java
index 7bf5023..eb0e630 100644
--- a/client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java
+++ b/client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java
@@ -1014,10 +1014,10 @@ public class AtlasClientV2 extends AtlasBaseClient {
 return readStreamContents(inputStream);
 }
 
-public List importGlossary(String fileName) throws 
AtlasServiceException {
+public BulkImportResponse importGlossary(String fileName) throws 
AtlasServiceException {
 MultiPart multipartEntity = getMultiPartData(fileName);
 
-return callAPI(API_V2.IMPORT_GLOSSARY, List.class, multipartEntity);
+return callAPI(API_V2.IMPORT_GLOSSARY, BulkImportResponse.class, 
multipartEntity);
 }
 
 
diff --git 
a/intg/src/main/java/org/apache/atlas/bulkimport/BulkImportResponse.java 
b/intg/src/main/java/org/apache/atlas/bulkimport/BulkImportResponse.java
index 047d497..5b95047 100644
--- a/intg/src/main/java/org/apache/atlas/bulkimport/BulkImportResponse.java
+++ b/intg/src/main/java/org/apache/atlas/bulkimport/BulkImportResponse.java
@@ -17,13 +17,18 @@
  */
 package org.apache.atlas.bulkimport;
 
+import org.apache.atlas.model.annotation.AtlasJSON;
+
+import java.io.Serializable;
 import java.util.ArrayList;
 import java.util.List;
 
-public class BulkImportResponse {
+import static 
org.apache.atlas.bulkimport.BulkImportResponse.ImportStatus.SUCCESS;
 
-private List failedImportInfoList = new 
ArrayList();
-private List successImportInfoList = new 
ArrayList();
+@AtlasJSON
+public class BulkImportResponse implements Serializable {
+private List failedImportInfoList = new ArrayList<>();
+private List successImportInfoList = new ArrayList<>();
 
 public BulkImportResponse() {}
 
@@ -31,36 +36,32 @@ public class BulkImportResponse {
 return failedImportInfoList;
 }
 
-public void setFailedImportInfoList(List failedImportInfoList){
+public void setFailedImportInfoList(List failedImportInfoList) 
{
 this.failedImportInfoList = failedImportInfoList;
 }
 
-public void setFailedImportInfoList(ImportInfo importInfo){
-List failedImportInfoList = this.failedImportInfoList;
-
-if (failedImportInfoList == null) {
-failedImportInfoList = new ArrayList<>();
+public void addToFailedImportInfoList(ImportInfo importInfo) {
+if (this.failedImportInfoList == null) {
+this.failedImportInfoList = new ArrayList<>();
 }
-failedImportInfoList.add(importInfo);
-setFailedImportInfoList(failedImportInfoList);
+
+this.failedImportInfoList.add(importInfo);
 }
 
 public List getSuccessImportInfoList() {
 return successImportInfoList;
 }
 
-public void setSuccessImportInfoList(List 
successImportInfoList){
+public void setSuccessImportInfoList(List 
successImportInfoList) {
 this.successImportInfoList = successImportInfoList;
 }
 
-public void setSuccessImportInfoList(ImportInfo importInfo){
-List successImportInfoList = this.successImportInfoList;
-
+public void addToSuccessImportInfoList(ImportInfo importInfo) {
 if (successImportInfoList == null) {
 successImportInfoList = new ArrayList<>();
 }
+
 successImportInfoList.add(importInfo);
-setSuccessImportInfoList(successImportInfoList);
 }
 
 public enum ImportStatus {
@@

[atlas] branch master updated: ATLAS-4129: Glossary Bulk Import Bugs consolidated fix (ATLAS-4128, ATLAS-4131, ATLAS-4160, ATLAS-4130, ATLAS-4129)

[sarath]

This is an automated email from the ASF dual-hosted git repository.

sarath pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/atlas.git


The following commit(s) were added to refs/heads/master by this push:
 new 842890d  ATLAS-4129: Glossary Bulk Import Bugs consolidated fix 
(ATLAS-4128, ATLAS-4131, ATLAS-4160, ATLAS-4130, ATLAS-4129)
842890d is described below

commit 842890d27bf739a0949e82b58d9db1f1dd6111ea
Author: mayanknj 
AuthorDate: Sat Mar 27 00:57:40 2021 +0530

ATLAS-4129: Glossary Bulk Import Bugs consolidated fix (ATLAS-4128, 
ATLAS-4131, ATLAS-4160, ATLAS-4130, ATLAS-4129)

Signed-off-by: Sarath Subramanian 
---
 .../main/java/org/apache/atlas/AtlasClientV2.java  |  4 +-
 .../atlas/bulkimport/BulkImportResponse.java   | 50 +++---
 .../org/apache/atlas/glossary/GlossaryService.java | 44 +++
 .../apache/atlas/glossary/GlossaryTermUtils.java   | 28 
 .../store/graph/v2/AtlasEntityStoreV2.java | 26 +--
 .../apache/atlas/glossary/GlossaryServiceTest.java | 32 --
 .../org/apache/atlas/web/rest/GlossaryREST.java|  5 ++-
 .../atlas/web/integration/GlossaryClientV2IT.java  | 20 +++--
 8 files changed, 125 insertions(+), 84 deletions(-)

diff --git a/client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java 
b/client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java
index 7bf5023..eb0e630 100644
--- a/client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java
+++ b/client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java
@@ -1014,10 +1014,10 @@ public class AtlasClientV2 extends AtlasBaseClient {
 return readStreamContents(inputStream);
 }
 
-public List importGlossary(String fileName) throws 
AtlasServiceException {
+public BulkImportResponse importGlossary(String fileName) throws 
AtlasServiceException {
 MultiPart multipartEntity = getMultiPartData(fileName);
 
-return callAPI(API_V2.IMPORT_GLOSSARY, List.class, multipartEntity);
+return callAPI(API_V2.IMPORT_GLOSSARY, BulkImportResponse.class, 
multipartEntity);
 }
 
 
diff --git 
a/intg/src/main/java/org/apache/atlas/bulkimport/BulkImportResponse.java 
b/intg/src/main/java/org/apache/atlas/bulkimport/BulkImportResponse.java
index 047d497..5b95047 100644
--- a/intg/src/main/java/org/apache/atlas/bulkimport/BulkImportResponse.java
+++ b/intg/src/main/java/org/apache/atlas/bulkimport/BulkImportResponse.java
@@ -17,13 +17,18 @@
  */
 package org.apache.atlas.bulkimport;
 
+import org.apache.atlas.model.annotation.AtlasJSON;
+
+import java.io.Serializable;
 import java.util.ArrayList;
 import java.util.List;
 
-public class BulkImportResponse {
+import static 
org.apache.atlas.bulkimport.BulkImportResponse.ImportStatus.SUCCESS;
 
-private List failedImportInfoList = new 
ArrayList();
-private List successImportInfoList = new 
ArrayList();
+@AtlasJSON
+public class BulkImportResponse implements Serializable {
+private List failedImportInfoList = new ArrayList<>();
+private List successImportInfoList = new ArrayList<>();
 
 public BulkImportResponse() {}
 
@@ -31,36 +36,32 @@ public class BulkImportResponse {
 return failedImportInfoList;
 }
 
-public void setFailedImportInfoList(List failedImportInfoList){
+public void setFailedImportInfoList(List failedImportInfoList) 
{
 this.failedImportInfoList = failedImportInfoList;
 }
 
-public void setFailedImportInfoList(ImportInfo importInfo){
-List failedImportInfoList = this.failedImportInfoList;
-
-if (failedImportInfoList == null) {
-failedImportInfoList = new ArrayList<>();
+public void addToFailedImportInfoList(ImportInfo importInfo) {
+if (this.failedImportInfoList == null) {
+this.failedImportInfoList = new ArrayList<>();
 }
-failedImportInfoList.add(importInfo);
-setFailedImportInfoList(failedImportInfoList);
+
+this.failedImportInfoList.add(importInfo);
 }
 
 public List getSuccessImportInfoList() {
 return successImportInfoList;
 }
 
-public void setSuccessImportInfoList(List 
successImportInfoList){
+public void setSuccessImportInfoList(List 
successImportInfoList) {
 this.successImportInfoList = successImportInfoList;
 }
 
-public void setSuccessImportInfoList(ImportInfo importInfo){
-List successImportInfoList = this.successImportInfoList;
-
+public void addToSuccessImportInfoList(ImportInfo importInfo) {
 if (successImportInfoList == null) {
 successImportInfoList = new ArrayList<>();
 }
+
 successImportInfoList.add(importInfo);
-setSuccessImportInfoList(successImportInfoList);
 }
 
 public enum ImportStatus {
@@ -75,14 +76,16 @@ public class BulkImportResponse {
 '}';
 }
 
-

[atlas] branch branch-2.0 updated: ATLAS-4230: Add support for Google Cloud Storage Path Entity creation

[sarath]

This is an automated email from the ASF dual-hosted git repository.

sarath pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/atlas.git


The following commit(s) were added to refs/heads/branch-2.0 by this push:
 new 32b25aa  ATLAS-4230: Add support for Google Cloud Storage Path Entity 
creation
32b25aa is described below

commit 32b25aa6e3f1751c302b2a741b771e5d102316ef
Author: sidmishra 
AuthorDate: Tue Mar 30 16:20:40 2021 -0700

ATLAS-4230: Add support for Google Cloud Storage Path Entity creation

Signed-off-by: Sarath Subramanian 
(cherry picked from commit c03ee72ca557eab49f2e315fc17aefcce03176ef)
---
 .../apache/atlas/utils/AtlasPathExtractorUtil.java | 93 +-
 .../atlas/utils/AtlasPathExtractorUtilTest.java| 66 ++-
 2 files changed, 155 insertions(+), 4 deletions(-)

diff --git 
a/common/src/main/java/org/apache/atlas/utils/AtlasPathExtractorUtil.java 
b/common/src/main/java/org/apache/atlas/utils/AtlasPathExtractorUtil.java
index 81f847e..a9f2e50 100644
--- a/common/src/main/java/org/apache/atlas/utils/AtlasPathExtractorUtil.java
+++ b/common/src/main/java/org/apache/atlas/utils/AtlasPathExtractorUtil.java
@@ -82,6 +82,13 @@ public class AtlasPathExtractorUtil {
 public static final String RELATIONSHIP_OZONE_VOLUME_BUCKET = 
"ozone_volume_buckets";
 public static final String RELATIONSHIP_OZONE_PARENT_CHILDREN   = 
"ozone_parent_children";
 
+//Google Cloud Storage
+public static final String GCS_SCHEME   = "gs" + 
SCHEME_SEPARATOR;
+public static final String GCS_BUCKET   = 
"gcp_storage_bucket";
+public static final String GCS_VIRTUAL_DIR  = 
"gcp_storage_virtual_directory";
+public static final String ATTRIBUTE_GCS_PARENT = "parent";
+public static final String RELATIONSHIP_GCS_PARENT_CHILDREN = 
"gcp_storage_parent_children";
+
 public static AtlasEntityWithExtInfo getPathEntity(Path path, 
PathExtractorContext context) {
 AtlasEntityWithExtInfo entityWithExtInfo = new 
AtlasEntityWithExtInfo();
 AtlasEntity ret;
@@ -98,9 +105,12 @@ public class AtlasPathExtractorUtil {
 ret = addAbfsPathEntity(path, entityWithExtInfo, context);
 } else if (isOzonePath(strPath)) {
 ret = addOzonePathEntity(path, entityWithExtInfo, context);
+} else if (isGCSPath(strPath)) {
+ret = addGCSPathEntity(path, entityWithExtInfo, context);
 } else {
 ret = addHDFSPathEntity(path, context);
 }
+
 entityWithExtInfo.setEntity(ret);
 
 return entityWithExtInfo;
@@ -123,6 +133,10 @@ public class AtlasPathExtractorUtil {
 return strPath != null && (strPath.startsWith(OZONE_SCHEME) || 
strPath.startsWith(OZONE_3_SCHEME));
 }
 
+private static boolean isGCSPath(String strPath) {
+return strPath != null && strPath.startsWith(GCS_SCHEME);
+}
+
 private static AtlasEntity addS3PathEntityV1(Path path, AtlasEntityExtInfo 
extInfo, PathExtractorContext context) {
 String strPath = path.toString();
 
@@ -217,7 +231,7 @@ public class AtlasPathExtractorUtil {
 ret = new AtlasEntity(AWS_S3_V2_PSEUDO_DIR);
 
 ret.setRelationshipAttribute(ATTRIBUTE_CONTAINER, parentObjId);
-ret.setAttribute(ATTRIBUTE_OBJECT_PREFIX, subDirPath);
+ret.setAttribute(ATTRIBUTE_OBJECT_PREFIX, parentPath);
 ret.setAttribute(ATTRIBUTE_QUALIFIED_NAME, 
subDirQualifiedName);
 ret.setAttribute(ATTRIBUTE_NAME, subDirName);
 
@@ -442,6 +456,83 @@ public class AtlasPathExtractorUtil {
 return ret;
 }
 
+private static AtlasEntity addGCSPathEntity(Path path, AtlasEntityExtInfo 
extInfo, PathExtractorContext context) {
+String strPath = path.toString();
+
+if (LOG.isDebugEnabled()) {
+LOG.debug("==> addGCSPathEntity(strPath={})", strPath);
+}
+
+String  metadataNamespace = context.getMetadataNamespace();
+String  pathQualifiedName = strPath + QNAME_SEP_METADATA_NAMESPACE 
+ metadataNamespace;
+AtlasEntity ret   = context.getEntity(pathQualifiedName);
+
+if (ret == null) {
+String  bucketName  = path.toUri().getAuthority();
+String  schemeAndBucketName = (path.toUri().getScheme() + 
SCHEME_SEPARATOR + bucketName).toLowerCase();
+String  bucketQualifiedName = schemeAndBucketName + 
QNAME_SEP_METADATA_NAMESPACE + metadataNamespace;
+AtlasEntity bucketEntity= 
context.getEntity(bucketQualifiedName);
+
+if (bucketEntity == null) {
+bucketEntity = new AtlasEntity(GCS_BUCKET);
+
+bucketEntity.setAttribute(ATTRIBU

[atlas] branch master updated: ATLAS-4230: Add support for Google Cloud Storage Path Entity creation

[sarath]

This is an automated email from the ASF dual-hosted git repository.

sarath pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/atlas.git


The following commit(s) were added to refs/heads/master by this push:
 new c03ee72  ATLAS-4230: Add support for Google Cloud Storage Path Entity 
creation
c03ee72 is described below

commit c03ee72ca557eab49f2e315fc17aefcce03176ef
Author: sidmishra 
AuthorDate: Tue Mar 30 16:20:40 2021 -0700

ATLAS-4230: Add support for Google Cloud Storage Path Entity creation

Signed-off-by: Sarath Subramanian 
---
 .../apache/atlas/utils/AtlasPathExtractorUtil.java | 93 +-
 .../atlas/utils/AtlasPathExtractorUtilTest.java| 66 ++-
 2 files changed, 155 insertions(+), 4 deletions(-)

diff --git 
a/common/src/main/java/org/apache/atlas/utils/AtlasPathExtractorUtil.java 
b/common/src/main/java/org/apache/atlas/utils/AtlasPathExtractorUtil.java
index 81f847e..a9f2e50 100644
--- a/common/src/main/java/org/apache/atlas/utils/AtlasPathExtractorUtil.java
+++ b/common/src/main/java/org/apache/atlas/utils/AtlasPathExtractorUtil.java
@@ -82,6 +82,13 @@ public class AtlasPathExtractorUtil {
 public static final String RELATIONSHIP_OZONE_VOLUME_BUCKET = 
"ozone_volume_buckets";
 public static final String RELATIONSHIP_OZONE_PARENT_CHILDREN   = 
"ozone_parent_children";
 
+//Google Cloud Storage
+public static final String GCS_SCHEME   = "gs" + 
SCHEME_SEPARATOR;
+public static final String GCS_BUCKET   = 
"gcp_storage_bucket";
+public static final String GCS_VIRTUAL_DIR  = 
"gcp_storage_virtual_directory";
+public static final String ATTRIBUTE_GCS_PARENT = "parent";
+public static final String RELATIONSHIP_GCS_PARENT_CHILDREN = 
"gcp_storage_parent_children";
+
 public static AtlasEntityWithExtInfo getPathEntity(Path path, 
PathExtractorContext context) {
 AtlasEntityWithExtInfo entityWithExtInfo = new 
AtlasEntityWithExtInfo();
 AtlasEntity ret;
@@ -98,9 +105,12 @@ public class AtlasPathExtractorUtil {
 ret = addAbfsPathEntity(path, entityWithExtInfo, context);
 } else if (isOzonePath(strPath)) {
 ret = addOzonePathEntity(path, entityWithExtInfo, context);
+} else if (isGCSPath(strPath)) {
+ret = addGCSPathEntity(path, entityWithExtInfo, context);
 } else {
 ret = addHDFSPathEntity(path, context);
 }
+
 entityWithExtInfo.setEntity(ret);
 
 return entityWithExtInfo;
@@ -123,6 +133,10 @@ public class AtlasPathExtractorUtil {
 return strPath != null && (strPath.startsWith(OZONE_SCHEME) || 
strPath.startsWith(OZONE_3_SCHEME));
 }
 
+private static boolean isGCSPath(String strPath) {
+return strPath != null && strPath.startsWith(GCS_SCHEME);
+}
+
 private static AtlasEntity addS3PathEntityV1(Path path, AtlasEntityExtInfo 
extInfo, PathExtractorContext context) {
 String strPath = path.toString();
 
@@ -217,7 +231,7 @@ public class AtlasPathExtractorUtil {
 ret = new AtlasEntity(AWS_S3_V2_PSEUDO_DIR);
 
 ret.setRelationshipAttribute(ATTRIBUTE_CONTAINER, parentObjId);
-ret.setAttribute(ATTRIBUTE_OBJECT_PREFIX, subDirPath);
+ret.setAttribute(ATTRIBUTE_OBJECT_PREFIX, parentPath);
 ret.setAttribute(ATTRIBUTE_QUALIFIED_NAME, 
subDirQualifiedName);
 ret.setAttribute(ATTRIBUTE_NAME, subDirName);
 
@@ -442,6 +456,83 @@ public class AtlasPathExtractorUtil {
 return ret;
 }
 
+private static AtlasEntity addGCSPathEntity(Path path, AtlasEntityExtInfo 
extInfo, PathExtractorContext context) {
+String strPath = path.toString();
+
+if (LOG.isDebugEnabled()) {
+LOG.debug("==> addGCSPathEntity(strPath={})", strPath);
+}
+
+String  metadataNamespace = context.getMetadataNamespace();
+String  pathQualifiedName = strPath + QNAME_SEP_METADATA_NAMESPACE 
+ metadataNamespace;
+AtlasEntity ret   = context.getEntity(pathQualifiedName);
+
+if (ret == null) {
+String  bucketName  = path.toUri().getAuthority();
+String  schemeAndBucketName = (path.toUri().getScheme() + 
SCHEME_SEPARATOR + bucketName).toLowerCase();
+String  bucketQualifiedName = schemeAndBucketName + 
QNAME_SEP_METADATA_NAMESPACE + metadataNamespace;
+AtlasEntity bucketEntity= 
context.getEntity(bucketQualifiedName);
+
+if (bucketEntity == null) {
+bucketEntity = new AtlasEntity(GCS_BUCKET);
+
+bucketEntity.setAttribute(ATTRIBUTE_QUALIFIED_NAME, 
bucketQualifiedName);
+ 

[atlas] branch master updated: ATLAS-4226: Add new model types to support Google Cloud Platform

[sarath]

This is an automated email from the ASF dual-hosted git repository.

sarath pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/atlas.git


The following commit(s) were added to refs/heads/master by this push:
 new 2aa448c  ATLAS-4226: Add new model types to support Google Cloud 
Platform
2aa448c is described below

commit 2aa448c110a175f74aa8c8e3bb338fefc71ae8b1
Author: sidmishra 
AuthorDate: Tue Mar 30 17:08:31 2021 -0700

ATLAS-4226: Add new model types to support Google Cloud Platform

Signed-off-by: Sarath Subramanian 
---
 addons/models/3000-Cloud/3060-gcp_typedefs.json | 232 
 1 file changed, 232 insertions(+)

diff --git a/addons/models/3000-Cloud/3060-gcp_typedefs.json 
b/addons/models/3000-Cloud/3060-gcp_typedefs.json
new file mode 100644
index 000..fbaed79
--- /dev/null
+++ b/addons/models/3000-Cloud/3060-gcp_typedefs.json
@@ -0,0 +1,232 @@
+{
+  "enumDefs": [],
+  "structDefs": [],
+  "classificationDefs": [],
+  "entityDefs": [
+{
+  "name":"gcp_storage_parent",
+  "description": "Atlas entity-type representing parent types (bucket, 
vitual directory) that can hold other objects in Google Cloud Storage",
+  "superTypes":  [
+  ],
+  "serviceType": "gcp",
+  "typeVersion": "1.0",
+  "attributeDefs": [
+  ]
+},
+{
+  "name":"gcp_storage_child",
+  "description": "Atlas entity-type representing child types (virtual 
directory, blob) in Google Cloud Storage",
+  "superTypes":  [
+  ],
+  "serviceType": "gcp",
+  "typeVersion": "1.0",
+  "attributeDefs": [
+  ]
+},
+{
+  "name":"gcp_storage_base",
+  "description": "Atlas Type representing base Type for all Google Cloud 
Storage Data Types",
+  "superTypes":  [
+"DataSet"
+  ],
+  "serviceType": "gcp",
+  "typeVersion": "1.0",
+  "attributeDefs": [
+{
+  "name":"creationTime",
+  "typeName":"date",
+  "cardinality": "SINGLE",
+  "isIndexable": true,
+  "isOptional":  true,
+  "isUnique":false
+},
+{
+  "name":"lastModifiedTime",
+  "typeName":"date",
+  "cardinality": "SINGLE",
+  "isIndexable": true,
+  "isOptional":  true,
+  "isUnique":false
+},
+{
+  "name":"storageClass",
+  "typeName":"string",
+  "cardinality": "SINGLE",
+  "isIndexable": true,
+  "isOptional":  true,
+  "isUnique":false
+},
+{
+  "name":"encryption",
+  "typeName":"string",
+  "cardinality": "SINGLE",
+  "isIndexable": false,
+  "isOptional":  true,
+  "isUnique":false
+},
+{
+  "name":"publicAccess",
+  "typeName":"string",
+  "cardinality": "SINGLE",
+  "isIndexable": true,
+  "isOptional":  true,
+  "isUnique":false
+}
+  ]
+},
+{
+  "name":"gcp_storage_bucket",
+  "description": "Atlas Type representing a Bucket in Google Cloud 
Storage",
+  "superTypes":  [
+"gcp_storage_parent",
+"gcp_storage_base"
+  ],
+  "serviceType": "gcp",
+  "typeVersion": "1.0",
+  "attributeDefs": [
+{
+  "name":"location",
+  "typeName":"string",
+  "cardinality": "SINGLE",
+  "isIndexable": true,
+  "isOptional":  true,
+  "isUnique":false
+},
+{
+  "name":"locationType",
+  "typeName":"string",
+  "cardinality": "SINGLE",
+  "isIndexable": true,
+  "isOptional":  true,
+  "isUnique":fals

[atlas] branch branch-2.0 updated: ATLAS-4227: Update objectPrefix isUnique for aws_s3_v2_directory to false

[sarath]

This is an automated email from the ASF dual-hosted git repository.

sarath pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/atlas.git


The following commit(s) were added to refs/heads/branch-2.0 by this push:
 new a07fe33  ATLAS-4227: Update objectPrefix isUnique for 
aws_s3_v2_directory to false
a07fe33 is described below

commit a07fe33ea3e4a488336b7ca022ee4707b33f66d4
Author: Ameer Assi 
AuthorDate: Mon Mar 29 15:12:56 2021 -0700

ATLAS-4227: Update objectPrefix isUnique for aws_s3_v2_directory to false

Signed-off-by: Sarath Subramanian 
(cherry picked from commit 3a59132a5fb05516d590f3061fd10c8af32a86f9)
---
 .../models/3000-Cloud/3030-aws_s3_v2_typedefs.json |  4 ++--
 .../patches/007-s3_v2_update_attributes.json   | 23 ++
 2 files changed, 25 insertions(+), 2 deletions(-)

diff --git a/addons/models/3000-Cloud/3030-aws_s3_v2_typedefs.json 
b/addons/models/3000-Cloud/3030-aws_s3_v2_typedefs.json
index e656c31..977c066 100644
--- a/addons/models/3000-Cloud/3030-aws_s3_v2_typedefs.json
+++ b/addons/models/3000-Cloud/3030-aws_s3_v2_typedefs.json
@@ -150,7 +150,7 @@
 "Path"
   ],
   "serviceType": "aws",
-  "typeVersion": "1.1",
+  "typeVersion": "1.2",
   "attributeDefs": [
 {
   "name":"objectPrefix",
@@ -158,7 +158,7 @@
   "cardinality": "SINGLE",
   "isIndexable": true,
   "isOptional":  false,
-  "isUnique":true
+  "isUnique":false
 },
 {
   "name":"subdomain",
diff --git a/addons/models/3000-Cloud/patches/007-s3_v2_update_attributes.json 
b/addons/models/3000-Cloud/patches/007-s3_v2_update_attributes.json
new file mode 100644
index 000..f29c5bf
--- /dev/null
+++ b/addons/models/3000-Cloud/patches/007-s3_v2_update_attributes.json
@@ -0,0 +1,23 @@
+{
+  "patches": [
+{
+  "id": "TYPEDEF_PATCH_3000_007_001",
+  "description": "Update objectPrefix isUnique for aws_s3_v2_directory to 
false",
+  "action": "UPDATE_ATTRIBUTE",
+  "typeName": "aws_s3_v2_directory",
+  "applyToVersion": "1.1",
+  "updateToVersion": "1.2",
+  "params": null,
+  "attributeDefs": [
+{
+  "name":"objectPrefix",
+  "typeName":"string",
+  "cardinality": "SINGLE",
+  "isIndexable": true,
+  "isOptional":  false,
+  "isUnique":false
+}
+  ]
+}
+  ]
+}
\ No newline at end of file

[atlas] branch master updated: ATLAS-4227: Update objectPrefix isUnique for aws_s3_v2_directory to false

[sarath]

This is an automated email from the ASF dual-hosted git repository.

sarath pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/atlas.git


The following commit(s) were added to refs/heads/master by this push:
 new 3a59132  ATLAS-4227: Update objectPrefix isUnique for 
aws_s3_v2_directory to false
3a59132 is described below

commit 3a59132a5fb05516d590f3061fd10c8af32a86f9
Author: Ameer Assi 
AuthorDate: Mon Mar 29 15:12:56 2021 -0700

ATLAS-4227: Update objectPrefix isUnique for aws_s3_v2_directory to false

Signed-off-by: Sarath Subramanian 
---
 .../models/3000-Cloud/3030-aws_s3_v2_typedefs.json |  4 ++--
 .../patches/007-s3_v2_update_attributes.json   | 23 ++
 2 files changed, 25 insertions(+), 2 deletions(-)

diff --git a/addons/models/3000-Cloud/3030-aws_s3_v2_typedefs.json 
b/addons/models/3000-Cloud/3030-aws_s3_v2_typedefs.json
index e656c31..977c066 100644
--- a/addons/models/3000-Cloud/3030-aws_s3_v2_typedefs.json
+++ b/addons/models/3000-Cloud/3030-aws_s3_v2_typedefs.json
@@ -150,7 +150,7 @@
 "Path"
   ],
   "serviceType": "aws",
-  "typeVersion": "1.1",
+  "typeVersion": "1.2",
   "attributeDefs": [
 {
   "name":"objectPrefix",
@@ -158,7 +158,7 @@
   "cardinality": "SINGLE",
   "isIndexable": true,
   "isOptional":  false,
-  "isUnique":true
+  "isUnique":false
 },
 {
   "name":"subdomain",
diff --git a/addons/models/3000-Cloud/patches/007-s3_v2_update_attributes.json 
b/addons/models/3000-Cloud/patches/007-s3_v2_update_attributes.json
new file mode 100644
index 000..f29c5bf
--- /dev/null
+++ b/addons/models/3000-Cloud/patches/007-s3_v2_update_attributes.json
@@ -0,0 +1,23 @@
+{
+  "patches": [
+{
+  "id": "TYPEDEF_PATCH_3000_007_001",
+  "description": "Update objectPrefix isUnique for aws_s3_v2_directory to 
false",
+  "action": "UPDATE_ATTRIBUTE",
+  "typeName": "aws_s3_v2_directory",
+  "applyToVersion": "1.1",
+  "updateToVersion": "1.2",
+  "params": null,
+  "attributeDefs": [
+{
+  "name":"objectPrefix",
+  "typeName":"string",
+  "cardinality": "SINGLE",
+  "isIndexable": true,
+  "isOptional":  false,
+  "isUnique":false
+}
+  ]
+}
+  ]
+}
\ No newline at end of file

[atlas] branch branch-2.0 updated: ATLAS-4180: [Atlas: Glossary Term Bulk Import] Update error message thrown while using @ in glossary/term name

[sarath]

This is an automated email from the ASF dual-hosted git repository.

sarath pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/atlas.git


The following commit(s) were added to refs/heads/branch-2.0 by this push:
 new e1181ae  ATLAS-4180: [Atlas: Glossary Term Bulk Import] Update error 
message thrown while using @ in glossary/term name
e1181ae is described below

commit e1181ae85a7389bd199297558635f1744bcb14d6
Author: chaitali borole 
AuthorDate: Mon Mar 15 16:35:01 2021 +0530

ATLAS-4180: [Atlas: Glossary Term Bulk Import] Update error message thrown 
while using @ in glossary/term name

Signed-off-by: Sarath Subramanian 
(cherry picked from commit 49576ba7dffa3403a2917d7a8fc0d477cd31bc1f)
---
 .../src/main/java/org/apache/atlas/glossary/GlossaryService.java  | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git 
a/repository/src/main/java/org/apache/atlas/glossary/GlossaryService.java 
b/repository/src/main/java/org/apache/atlas/glossary/GlossaryService.java
index df5fcbd..e526e1a 100644
--- a/repository/src/main/java/org/apache/atlas/glossary/GlossaryService.java
+++ b/repository/src/main/java/org/apache/atlas/glossary/GlossaryService.java
@@ -1120,7 +1120,7 @@ public class GlossaryService {
 ret = glossaryTermUtils.getGlossaryTermDataList(fileData, 
failedTermMsgs);
 ret = createGlossaryTerms(ret);
 } catch (IOException e) {
-throw new AtlasBaseException(AtlasErrorCode.FAILED_TO_UPLOAD, e);
+throw new AtlasBaseException(AtlasErrorCode.FAILED_TO_UPLOAD, 
fileName);
 }
 
 return ret;
@@ -1134,7 +1134,7 @@ public class GlossaryService {
 ret.add(createTerm(glossaryTerm));
 } catch (AtlasBaseException e) {
 if 
(!e.getAtlasErrorCode().equals(AtlasErrorCode.GLOSSARY_TERM_ALREADY_EXISTS)) {
-throw new 
AtlasBaseException(AtlasErrorCode.FAILED_TO_CREATE_GLOSSARY_TERM, e);
+throw new 
AtlasBaseException(AtlasErrorCode.FAILED_TO_CREATE_GLOSSARY_TERM, 
glossaryTerm.getName());
 }
 }
 }

[atlas] branch master updated: ATLAS-4180: [Atlas: Glossary Term Bulk Import] Update error message thrown while using @ in glossary/term name

[sarath]

This is an automated email from the ASF dual-hosted git repository.

sarath pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/atlas.git


The following commit(s) were added to refs/heads/master by this push:
 new 49576ba  ATLAS-4180: [Atlas: Glossary Term Bulk Import] Update error 
message thrown while using @ in glossary/term name
49576ba is described below

commit 49576ba7dffa3403a2917d7a8fc0d477cd31bc1f
Author: chaitali borole 
AuthorDate: Mon Mar 15 16:35:01 2021 +0530

ATLAS-4180: [Atlas: Glossary Term Bulk Import] Update error message thrown 
while using @ in glossary/term name

Signed-off-by: Sarath Subramanian 
---
 .../src/main/java/org/apache/atlas/glossary/GlossaryService.java  | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git 
a/repository/src/main/java/org/apache/atlas/glossary/GlossaryService.java 
b/repository/src/main/java/org/apache/atlas/glossary/GlossaryService.java
index df5fcbd..e526e1a 100644
--- a/repository/src/main/java/org/apache/atlas/glossary/GlossaryService.java
+++ b/repository/src/main/java/org/apache/atlas/glossary/GlossaryService.java
@@ -1120,7 +1120,7 @@ public class GlossaryService {
 ret = glossaryTermUtils.getGlossaryTermDataList(fileData, 
failedTermMsgs);
 ret = createGlossaryTerms(ret);
 } catch (IOException e) {
-throw new AtlasBaseException(AtlasErrorCode.FAILED_TO_UPLOAD, e);
+throw new AtlasBaseException(AtlasErrorCode.FAILED_TO_UPLOAD, 
fileName);
 }
 
 return ret;
@@ -1134,7 +1134,7 @@ public class GlossaryService {
 ret.add(createTerm(glossaryTerm));
 } catch (AtlasBaseException e) {
 if 
(!e.getAtlasErrorCode().equals(AtlasErrorCode.GLOSSARY_TERM_ALREADY_EXISTS)) {
-throw new 
AtlasBaseException(AtlasErrorCode.FAILED_TO_CREATE_GLOSSARY_TERM, e);
+throw new 
AtlasBaseException(AtlasErrorCode.FAILED_TO_CREATE_GLOSSARY_TERM, 
glossaryTerm.getName());
 }
 }
 }

[atlas] branch branch-2.0 updated: ATLAS-4219: Basic Search - Search with Glossary term returning approximate count as -1

[sarath]

This is an automated email from the ASF dual-hosted git repository.

sarath pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/atlas.git


The following commit(s) were added to refs/heads/branch-2.0 by this push:
 new 6d6a281  ATLAS-4219: Basic Search - Search with Glossary term 
returning approximate count as -1
6d6a281 is described below

commit 6d6a281160ed837b9559a6d41ac75e319ca51bf7
Author: Pinal 
AuthorDate: Sun Mar 21 21:48:04 2021 +0530

ATLAS-4219: Basic Search - Search with Glossary term returning approximate 
count as -1

Signed-off-by: Sarath Subramanian 
(cherry picked from commit bc2c21898805da9e3c749a108b319c18f777ff2b)
---
 .../src/main/java/org/apache/atlas/discovery/TermSearchProcessor.java   | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git 
a/repository/src/main/java/org/apache/atlas/discovery/TermSearchProcessor.java 
b/repository/src/main/java/org/apache/atlas/discovery/TermSearchProcessor.java
index d9db9bf..45a8158 100644
--- 
a/repository/src/main/java/org/apache/atlas/discovery/TermSearchProcessor.java
+++ 
b/repository/src/main/java/org/apache/atlas/discovery/TermSearchProcessor.java
@@ -114,6 +114,6 @@ public class TermSearchProcessor extends SearchProcessor {
 
 @Override
 public long getResultCount() {
-return -1;
+return assignedEntities != null ? assignedEntities.size() : -1;
 }
 }

[atlas] branch master updated: ATLAS-4219: Basic Search - Search with Glossary term returning approximate count as -1

[sarath]

This is an automated email from the ASF dual-hosted git repository.

sarath pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/atlas.git


The following commit(s) were added to refs/heads/master by this push:
 new bc2c218  ATLAS-4219: Basic Search - Search with Glossary term 
returning approximate count as -1
bc2c218 is described below

commit bc2c21898805da9e3c749a108b319c18f777ff2b
Author: Pinal 
AuthorDate: Sun Mar 21 21:48:04 2021 +0530

ATLAS-4219: Basic Search - Search with Glossary term returning approximate 
count as -1

Signed-off-by: Sarath Subramanian 
---
 .../src/main/java/org/apache/atlas/discovery/TermSearchProcessor.java   | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git 
a/repository/src/main/java/org/apache/atlas/discovery/TermSearchProcessor.java 
b/repository/src/main/java/org/apache/atlas/discovery/TermSearchProcessor.java
index d9db9bf..45a8158 100644
--- 
a/repository/src/main/java/org/apache/atlas/discovery/TermSearchProcessor.java
+++ 
b/repository/src/main/java/org/apache/atlas/discovery/TermSearchProcessor.java
@@ -114,6 +114,6 @@ public class TermSearchProcessor extends SearchProcessor {
 
 @Override
 public long getResultCount() {
-return -1;
+return assignedEntities != null ? assignedEntities.size() : -1;
 }
 }

[atlas] branch branch-2.0 updated: ATLAS-4211: DSL Search: Regression: When fetching string attribute with null value returns null , earlier versions returned empty

[sarath]

This is an automated email from the ASF dual-hosted git repository.

sarath pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/atlas.git


The following commit(s) were added to refs/heads/branch-2.0 by this push:
 new 947f4bd  ATLAS-4211: DSL Search: Regression: When fetching string 
attribute with null value returns null , earlier versions returned empty
947f4bd is described below

commit 947f4bd60603fb4fed65201587e685e3391ffbdd
Author: Pinal 
AuthorDate: Tue Mar 16 21:54:54 2021 +0530

ATLAS-4211: DSL Search: Regression: When fetching string attribute with 
null value returns null , earlier versions returned empty

Signed-off-by: Sarath Subramanian 
(cherry picked from commit 83aa1f45b9aebea50c9e6c0847d16a821978ddd5)
---
 .../org/apache/atlas/query/executors/SelectClauseProjections.java| 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git 
a/repository/src/main/java/org/apache/atlas/query/executors/SelectClauseProjections.java
 
b/repository/src/main/java/org/apache/atlas/query/executors/SelectClauseProjections.java
index 7018c15..8305477 100644
--- 
a/repository/src/main/java/org/apache/atlas/query/executors/SelectClauseProjections.java
+++ 
b/repository/src/main/java/org/apache/atlas/query/executors/SelectClauseProjections.java
@@ -24,6 +24,7 @@ import org.apache.atlas.query.GremlinQuery;
 import org.apache.atlas.query.SelectClauseComposer;
 import org.apache.atlas.repository.graphdb.AtlasVertex;
 import org.apache.atlas.repository.store.graph.v2.EntityGraphRetriever;
+import org.apache.commons.lang.StringUtils;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
@@ -130,8 +131,8 @@ public class SelectClauseProjections {
 } else {
 if (selectClauseComposer.isPrimitiveAttribute(idx)) {
 String propertyName = 
selectClauseComposer.getAttribute(idx);
-
-row.add(vertex.getProperty(propertyName, 
Object.class));
+Object value = vertex.getProperty(propertyName, 
Object.class);
+row.add(value != null ? value : StringUtils.EMPTY);
 } else {
 
row.add(entityRetriever.toAtlasEntityHeaderWithClassifications(vertex));
 }

[atlas] branch master updated: ATLAS-4211: DSL Search: Regression: When fetching string attribute with null value returns null , earlier versions returned empty

[sarath]

This is an automated email from the ASF dual-hosted git repository.

sarath pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/atlas.git


The following commit(s) were added to refs/heads/master by this push:
 new 83aa1f4  ATLAS-4211: DSL Search: Regression: When fetching string 
attribute with null value returns null , earlier versions returned empty
83aa1f4 is described below

commit 83aa1f45b9aebea50c9e6c0847d16a821978ddd5
Author: Pinal 
AuthorDate: Tue Mar 16 21:54:54 2021 +0530

ATLAS-4211: DSL Search: Regression: When fetching string attribute with 
null value returns null , earlier versions returned empty

Signed-off-by: Sarath Subramanian 
---
 .../org/apache/atlas/query/executors/SelectClauseProjections.java| 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git 
a/repository/src/main/java/org/apache/atlas/query/executors/SelectClauseProjections.java
 
b/repository/src/main/java/org/apache/atlas/query/executors/SelectClauseProjections.java
index 7018c15..8305477 100644
--- 
a/repository/src/main/java/org/apache/atlas/query/executors/SelectClauseProjections.java
+++ 
b/repository/src/main/java/org/apache/atlas/query/executors/SelectClauseProjections.java
@@ -24,6 +24,7 @@ import org.apache.atlas.query.GremlinQuery;
 import org.apache.atlas.query.SelectClauseComposer;
 import org.apache.atlas.repository.graphdb.AtlasVertex;
 import org.apache.atlas.repository.store.graph.v2.EntityGraphRetriever;
+import org.apache.commons.lang.StringUtils;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
@@ -130,8 +131,8 @@ public class SelectClauseProjections {
 } else {
 if (selectClauseComposer.isPrimitiveAttribute(idx)) {
 String propertyName = 
selectClauseComposer.getAttribute(idx);
-
-row.add(vertex.getProperty(propertyName, 
Object.class));
+Object value = vertex.getProperty(propertyName, 
Object.class);
+row.add(value != null ? value : StringUtils.EMPTY);
 } else {
 
row.add(entityRetriever.toAtlasEntityHeaderWithClassifications(vertex));
 }

[atlas] branch branch-2.0 updated: ATLAS-4169: Add Hive Location Path entities to HMS Hook

[sarath]

This is an automated email from the ASF dual-hosted git repository.

sarath pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/atlas.git


The following commit(s) were added to refs/heads/branch-2.0 by this push:
 new 8c41fc0  ATLAS-4169: Add Hive Location Path entities to HMS Hook
8c41fc0 is described below

commit 8c41fc0d76dbeab72af529162f8d911f592b11cc
Author: Radhika Kundam 
AuthorDate: Thu Feb 18 16:26:41 2021 -0800

ATLAS-4169: Add Hive Location Path entities to HMS Hook

Signed-off-by: Sarath Subramanian 
(cherry picked from commit 47c18b9419f825fd6a15bbf9f22196b4ef49aefe)
---
 .../org/apache/atlas/hive/hook/events/AlterDatabase.java  |  2 ++
 .../org/apache/atlas/hive/hook/events/CreateDatabase.java | 15 +++
 2 files changed, 13 insertions(+), 4 deletions(-)

diff --git 
a/addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/AlterDatabase.java
 
b/addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/AlterDatabase.java
index 6b01c4e..d2623b3 100644
--- 
a/addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/AlterDatabase.java
+++ 
b/addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/AlterDatabase.java
@@ -61,6 +61,8 @@ public class AlterDatabase extends CreateDatabase {
 AtlasEntity dbEntity = toDbEntity(newDb);
 
 ret.addEntity(dbEntity);
+
+addLocationEntities(dbEntity, ret);
 } else {
 LOG.error("AlterDatabase.getEntities(): failed to retrieve db");
 }
diff --git 
a/addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/CreateDatabase.java
 
b/addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/CreateDatabase.java
index 8305a44..bf5f562 100644
--- 
a/addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/CreateDatabase.java
+++ 
b/addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/CreateDatabase.java
@@ -68,6 +68,8 @@ public class CreateDatabase extends BaseHiveEvent {
 AtlasEntity dbEntity = toDbEntity(db);
 
 ret.addEntity(dbEntity);
+
+addLocationEntities(dbEntity, ret);
 } else {
 LOG.error("CreateDatabase.getEntities(): failed to retrieve db");
 }
@@ -98,10 +100,7 @@ public class CreateDatabase extends BaseHiveEvent {
 ret.addEntity(dbDDLEntity);
 }
 
-AtlasEntity dbLocationEntity = 
createHiveLocationEntity(dbEntity, ret);
-if (dbLocationEntity != null) {
-ret.addEntity(dbLocationEntity);
-}
+addLocationEntities(dbEntity, ret);
 } else {
 LOG.error("CreateDatabase.getEntities(): failed to 
retrieve db");
 }
@@ -112,4 +111,12 @@ public class CreateDatabase extends BaseHiveEvent {
 
 return ret;
 }
+
+public void addLocationEntities(AtlasEntity dbEntity, 
AtlasEntitiesWithExtInfo ret) {
+AtlasEntity dbLocationEntity = createHiveLocationEntity(dbEntity, ret);
+
+if (dbLocationEntity != null) {
+ret.addEntity(dbLocationEntity);
+}
+}
 }
\ No newline at end of file

[atlas] branch master updated: ATLAS-4169: Add Hive Location Path entities to HMS Hook

[sarath]

This is an automated email from the ASF dual-hosted git repository.

sarath pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/atlas.git


The following commit(s) were added to refs/heads/master by this push:
 new 47c18b9  ATLAS-4169: Add Hive Location Path entities to HMS Hook
47c18b9 is described below

commit 47c18b9419f825fd6a15bbf9f22196b4ef49aefe
Author: Radhika Kundam 
AuthorDate: Thu Feb 18 16:26:41 2021 -0800

ATLAS-4169: Add Hive Location Path entities to HMS Hook

Signed-off-by: Sarath Subramanian 
---
 .../org/apache/atlas/hive/hook/events/AlterDatabase.java  |  2 ++
 .../org/apache/atlas/hive/hook/events/CreateDatabase.java | 15 +++
 2 files changed, 13 insertions(+), 4 deletions(-)

diff --git 
a/addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/AlterDatabase.java
 
b/addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/AlterDatabase.java
index 6b01c4e..d2623b3 100644
--- 
a/addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/AlterDatabase.java
+++ 
b/addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/AlterDatabase.java
@@ -61,6 +61,8 @@ public class AlterDatabase extends CreateDatabase {
 AtlasEntity dbEntity = toDbEntity(newDb);
 
 ret.addEntity(dbEntity);
+
+addLocationEntities(dbEntity, ret);
 } else {
 LOG.error("AlterDatabase.getEntities(): failed to retrieve db");
 }
diff --git 
a/addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/CreateDatabase.java
 
b/addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/CreateDatabase.java
index 8305a44..bf5f562 100644
--- 
a/addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/CreateDatabase.java
+++ 
b/addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/CreateDatabase.java
@@ -68,6 +68,8 @@ public class CreateDatabase extends BaseHiveEvent {
 AtlasEntity dbEntity = toDbEntity(db);
 
 ret.addEntity(dbEntity);
+
+addLocationEntities(dbEntity, ret);
 } else {
 LOG.error("CreateDatabase.getEntities(): failed to retrieve db");
 }
@@ -98,10 +100,7 @@ public class CreateDatabase extends BaseHiveEvent {
 ret.addEntity(dbDDLEntity);
 }
 
-AtlasEntity dbLocationEntity = 
createHiveLocationEntity(dbEntity, ret);
-if (dbLocationEntity != null) {
-ret.addEntity(dbLocationEntity);
-}
+addLocationEntities(dbEntity, ret);
 } else {
 LOG.error("CreateDatabase.getEntities(): failed to 
retrieve db");
 }
@@ -112,4 +111,12 @@ public class CreateDatabase extends BaseHiveEvent {
 
 return ret;
 }
+
+public void addLocationEntities(AtlasEntity dbEntity, 
AtlasEntitiesWithExtInfo ret) {
+AtlasEntity dbLocationEntity = createHiveLocationEntity(dbEntity, ret);
+
+if (dbLocationEntity != null) {
+ret.addEntity(dbLocationEntity);
+}
+}
 }
\ No newline at end of file

[atlas] branch branch-2.0 updated: ATLAS-4170: Fixed Bulk Entity GET API to skip unauthorised entities when skipFailedEntities is passed as True

[sarath]

This is an automated email from the ASF dual-hosted git repository.

sarath pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/atlas.git


The following commit(s) were added to refs/heads/branch-2.0 by this push:
 new 9434bb7  ATLAS-4170: Fixed Bulk Entity GET API to skip unauthorised 
entities when skipFailedEntities is passed as True
9434bb7 is described below

commit 9434bb7aef454070e1f5453b4f6a4964c69ae73f
Author: sidmishra 
AuthorDate: Thu Feb 18 18:00:54 2021 -0800

ATLAS-4170: Fixed Bulk Entity GET API to skip unauthorised entities when 
skipFailedEntities is passed as True

Signed-off-by: Sarath Subramanian 
(cherry picked from commit 928d86864b6e4c9244cd77c18c1d3d54d6a6a836)
---
 .../apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java  | 6 --
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git 
a/repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java
 
b/repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java
index ce58e9a..e6fe306 100644
--- 
a/repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java
+++ 
b/repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java
@@ -214,9 +214,8 @@ public class AtlasEntityStoreV2 implements AtlasEntityStore 
{
 
 if(ret != null){
 for(String guid : guids) {
+AtlasEntity entity = ret.getEntity(guid);
 try {
-AtlasEntity entity = ret.getEntity(guid);
-
 AtlasAuthorizationUtils.verifyAccess(new 
AtlasEntityAccessRequest(typeRegistry, AtlasPrivilege.ENTITY_READ, new 
AtlasEntityHeader(entity)), "read entity: guid=", guid);
 } catch (AtlasBaseException e) {
 if (RequestContext.get().isSkipFailedEntities()) {
@@ -224,7 +223,10 @@ public class AtlasEntityStoreV2 implements 
AtlasEntityStore {
 LOG.debug("getByIds(): ignoring failure for entity 
{}: error code={}, message={}", guid, e.getAtlasErrorCode(), e.getMessage());
 }
 
+//Remove from referred entities
 ret.removeEntity(guid);
+//Remove from entities
+ret.removeEntity(entity);
 
 continue;
 }

[atlas] branch master updated: ATLAS-4170: Fixed Bulk Entity GET API to skip unauthorised entities when skipFailedEntities is passed as True

[sarath]

This is an automated email from the ASF dual-hosted git repository.

sarath pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/atlas.git


The following commit(s) were added to refs/heads/master by this push:
 new 928d868  ATLAS-4170: Fixed Bulk Entity GET API to skip unauthorised 
entities when skipFailedEntities is passed as True
928d868 is described below

commit 928d86864b6e4c9244cd77c18c1d3d54d6a6a836
Author: sidmishra 
AuthorDate: Thu Feb 18 18:00:54 2021 -0800

ATLAS-4170: Fixed Bulk Entity GET API to skip unauthorised entities when 
skipFailedEntities is passed as True

Signed-off-by: Sarath Subramanian 
---
 .../apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java  | 6 --
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git 
a/repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java
 
b/repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java
index ce58e9a..e6fe306 100644
--- 
a/repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java
+++ 
b/repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java
@@ -214,9 +214,8 @@ public class AtlasEntityStoreV2 implements AtlasEntityStore 
{
 
 if(ret != null){
 for(String guid : guids) {
+AtlasEntity entity = ret.getEntity(guid);
 try {
-AtlasEntity entity = ret.getEntity(guid);
-
 AtlasAuthorizationUtils.verifyAccess(new 
AtlasEntityAccessRequest(typeRegistry, AtlasPrivilege.ENTITY_READ, new 
AtlasEntityHeader(entity)), "read entity: guid=", guid);
 } catch (AtlasBaseException e) {
 if (RequestContext.get().isSkipFailedEntities()) {
@@ -224,7 +223,10 @@ public class AtlasEntityStoreV2 implements 
AtlasEntityStore {
 LOG.debug("getByIds(): ignoring failure for entity 
{}: error code={}, message={}", guid, e.getAtlasErrorCode(), e.getMessage());
 }
 
+//Remove from referred entities
 ret.removeEntity(guid);
+//Remove from entities
+ret.removeEntity(entity);
 
 continue;
 }

[atlas] branch branch-2.0 updated: ATLAS-4127: Optimize Hive table calls in Atlas Hook

[sarath]

This is an automated email from the ASF dual-hosted git repository.

sarath pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/atlas.git


The following commit(s) were added to refs/heads/branch-2.0 by this push:
 new 5b3b0da  ATLAS-4127: Optimize Hive table calls in Atlas Hook
5b3b0da is described below

commit 5b3b0da9e95830c8cf6330bb218993717de9e0ff
Author: Deep Singh 
AuthorDate: Wed Feb 3 20:05:28 2021 -0600

ATLAS-4127: Optimize Hive table calls in Atlas Hook

Signed-off-by: Sarath Subramanian 
(cherry picked from commit f848c9761bd340c9cbb790254420506d97c8e6b4)
---
 .../java/org/apache/atlas/hive/hook/events/BaseHiveEvent.java| 9 -
 1 file changed, 8 insertions(+), 1 deletion(-)

diff --git 
a/addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/BaseHiveEvent.java
 
b/addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/BaseHiveEvent.java
index dd1c41d..111b065 100644
--- 
a/addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/BaseHiveEvent.java
+++ 
b/addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/BaseHiveEvent.java
@@ -1038,6 +1038,7 @@ public abstract class BaseHiveEvent {
 Collections.sort(sortedEntities, entityComparator);
 
 Set dataSetsProcessed = new HashSet<>();
+Map tableMap   = new HashMap<>();
 
 for (Entity entity : sortedEntities) {
 if (ignoreHDFSPaths && 
(Entity.Type.DFS_DIR.equals(entity.getType()) || 
Entity.Type.LOCAL_DIR.equals(entity.getType( {
@@ -1049,8 +1050,14 @@ public abstract class BaseHiveEvent {
 
 try {
 if (entity.getType() == Entity.Type.PARTITION || 
entity.getType() == Entity.Type.TABLE) {
-Table table = 
getHive().getTable(entity.getTable().getDbName(), 
entity.getTable().getTableName());
+String tableKey = entity.getTable().getDbName() + "." + 
entity.getTable().getTableName();
+Table  table= tableMap.get(tableKey);
 
+if (table == null) {
+table = 
getHive().getTable(entity.getTable().getDbName(), 
entity.getTable().getTableName());
+
+tableMap.put(tableKey, table); //since there could be 
several partitions in a table, store it to avoid hive calls.
+}
 if (table != null) {
 createTime= getTableCreateTime(table);
 qualifiedName = getQualifiedName(table);

[atlas] branch master updated: ATLAS-4127: Optimize Hive table calls in Atlas Hook

[sarath]

This is an automated email from the ASF dual-hosted git repository.

sarath pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/atlas.git


The following commit(s) were added to refs/heads/master by this push:
 new f848c97  ATLAS-4127: Optimize Hive table calls in Atlas Hook
f848c97 is described below

commit f848c9761bd340c9cbb790254420506d97c8e6b4
Author: Deep Singh 
AuthorDate: Wed Feb 3 20:05:28 2021 -0600

ATLAS-4127: Optimize Hive table calls in Atlas Hook

Signed-off-by: Sarath Subramanian 
---
 .../java/org/apache/atlas/hive/hook/events/BaseHiveEvent.java| 9 -
 1 file changed, 8 insertions(+), 1 deletion(-)

diff --git 
a/addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/BaseHiveEvent.java
 
b/addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/BaseHiveEvent.java
index 3b2bbe0..7c269ce 100644
--- 
a/addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/BaseHiveEvent.java
+++ 
b/addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/BaseHiveEvent.java
@@ -1035,6 +1035,7 @@ public abstract class BaseHiveEvent {
 Collections.sort(sortedEntities, entityComparator);
 
 Set dataSetsProcessed = new HashSet<>();
+Map tableMap   = new HashMap<>();
 
 for (Entity entity : sortedEntities) {
 if (ignoreHDFSPaths && 
(Entity.Type.DFS_DIR.equals(entity.getType()) || 
Entity.Type.LOCAL_DIR.equals(entity.getType( {
@@ -1046,8 +1047,14 @@ public abstract class BaseHiveEvent {
 
 try {
 if (entity.getType() == Entity.Type.PARTITION || 
entity.getType() == Entity.Type.TABLE) {
-Table table = 
getHive().getTable(entity.getTable().getDbName(), 
entity.getTable().getTableName());
+String tableKey = entity.getTable().getDbName() + "." + 
entity.getTable().getTableName();
+Table  table= tableMap.get(tableKey);
 
+if (table == null) {
+table = 
getHive().getTable(entity.getTable().getDbName(), 
entity.getTable().getTableName());
+
+tableMap.put(tableKey, table); //since there could be 
several partitions in a table, store it to avoid hive calls.
+}
 if (table != null) {
 createTime= getTableCreateTime(table);
 qualifiedName = getQualifiedName(table);

[atlas] branch branch-2.0 updated: ATLAS-4125: JavaPatch to add new supertypes to existing entities #2 fix Unit tests

[sarath]

This is an automated email from the ASF dual-hosted git repository.

sarath pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/atlas.git


The following commit(s) were added to refs/heads/branch-2.0 by this push:
 new f32f735  ATLAS-4125: JavaPatch to add new supertypes to existing 
entities #2 fix Unit tests
f32f735 is described below

commit f32f7352f9b1c5871f48e9ff91af4db61872f055
Author: Radhika Kundam 
AuthorDate: Tue Feb 9 15:48:12 2021 -0800

ATLAS-4125: JavaPatch to add new supertypes to existing entities #2 fix 
Unit tests

Signed-off-by: Sarath Subramanian 
(cherry picked from commit a531e8c685a0758e13d2862a98284110bc8faf22)
---
 .../org/apache/atlas/hive/hook/HiveHookIT.java | 39 --
 1 file changed, 21 insertions(+), 18 deletions(-)

diff --git 
a/addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 
b/addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java
index 5083bcc..03bbd60 100755
--- 
a/addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java
+++ 
b/addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java
@@ -71,7 +71,8 @@ import static org.testng.AssertJUnit.assertEquals;
 public class HiveHookIT extends HiveITBase {
 private static final Logger LOG = 
LoggerFactory.getLogger(HiveHookIT.class);
 
-private static final String PART_FILE  = "2015-01-01";
+private static final String PART_FILE  = "2015-01-01";
+private static final String PATH_TYPE_NAME = "Path";
 
 private Driver driverWithNoHook;
 
@@ -142,16 +143,6 @@ public class HiveHookIT extends HiveITBase {
 String hdfsLocation = 
"hdfs://localhost:8020/warehouse/tablespace/external/hive/reports.db";
 alterDatabaseLocation(dbName, hdfsLocation);
 assertDatabaseLocationRelationship(dbId);
-
-//AWS location
-String s3Location = 
"s3://localhost:8020/warehouse/tablespace/external/hive/reports.db";
-alterDatabaseLocation(dbName, s3Location);
-assertDatabaseLocationRelationship(dbId);
-
-//ABFS location
-String abfsLocation = 
"abfs://localhost:8020/warehouse/tablespace/external/hive/reports.db";
-alterDatabaseLocation(dbName, abfsLocation);
-assertDatabaseLocationRelationship(dbId);
 }
 
 //alter database location
@@ -162,20 +153,32 @@ public class HiveHookIT extends HiveITBase {
 }
 
 public void assertDatabaseLocationRelationship(String dbId) throws 
Exception {
-AtlasEntity dbEntity = atlasClientV2.getEntityByGuid(dbId).getEntity();
-AtlasEntityDef pathEntityDef = 
atlasClientV2.getEntityDefByName("Path");
+AtlasEntitydbEntity  = 
atlasClientV2.getEntityByGuid(dbId).getEntity();
+AtlasEntityDef pathEntityDef = getPathEntityDefWithAllSubTypes();
 
-//Check if dbEntity has location attribute
 assertTrue(dbEntity.hasAttribute(ATTRIBUTE_LOCATION));
-//Check if dbEntity has value for location attribute
+
 assertNotNull(dbEntity.getAttribute(ATTRIBUTE_LOCATION));
-//Check if dbEntity has location relationship attribute
-assertEquals(((List) 
dbEntity.getRelationshipAttribute(ATTRIBUTE_LOCATION_PATH)).size(), 1);
+
+
assertNotNull(dbEntity.getRelationshipAttribute(ATTRIBUTE_LOCATION_PATH));
+
 AtlasObjectId locationEntityObject = 
toAtlasObjectId(dbEntity.getRelationshipAttribute(ATTRIBUTE_LOCATION_PATH));
-//Check if location relationship attribute is subtype of "Path"
 
assertTrue(pathEntityDef.getSubTypes().contains(locationEntityObject.getTypeName()));
 }
 
+public AtlasEntityDef getPathEntityDefWithAllSubTypes() throws Exception {
+Set possiblePathSubTypes = new 
HashSet<>(Arrays.asList("fs_path", "hdfs_path", "aws_s3_pseudo_dir", 
"aws_s3_v2_directory", "adls_gen2_directory"));
+AtlasEntityDef  pathEntityDef= 
atlasClientV2.getEntityDefByName(PATH_TYPE_NAME);
+
+if(pathEntityDef == null) {
+pathEntityDef = new AtlasEntityDef(PATH_TYPE_NAME);
+}
+
+pathEntityDef.setSubTypes(possiblePathSubTypes);
+
+return pathEntityDef;
+}
+
 @Test
 public void testCreateTable() throws Exception {
 String tableName = tableName();

[atlas] branch master updated: ATLAS-4125: JavaPatch to add new supertypes to existing entities #2 fix Unit tests

[sarath]

This is an automated email from the ASF dual-hosted git repository.

sarath pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/atlas.git


The following commit(s) were added to refs/heads/master by this push:
 new a531e8c  ATLAS-4125: JavaPatch to add new supertypes to existing 
entities #2 fix Unit tests
a531e8c is described below

commit a531e8c685a0758e13d2862a98284110bc8faf22
Author: Radhika Kundam 
AuthorDate: Tue Feb 9 15:48:12 2021 -0800

ATLAS-4125: JavaPatch to add new supertypes to existing entities #2 fix 
Unit tests

Signed-off-by: Sarath Subramanian 
---
 .../org/apache/atlas/hive/hook/HiveHookIT.java | 39 --
 1 file changed, 21 insertions(+), 18 deletions(-)

diff --git 
a/addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 
b/addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java
index fccfc48..52684e7 100755
--- 
a/addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java
+++ 
b/addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java
@@ -67,7 +67,8 @@ import static org.testng.AssertJUnit.assertEquals;
 public class HiveHookIT extends HiveITBase {
 private static final Logger LOG = 
LoggerFactory.getLogger(HiveHookIT.class);
 
-private static final String PART_FILE  = "2015-01-01";
+private static final String PART_FILE  = "2015-01-01";
+private static final String PATH_TYPE_NAME = "Path";
 
 private Driver driverWithNoHook;
 
@@ -138,16 +139,6 @@ public class HiveHookIT extends HiveITBase {
 String hdfsLocation = 
"hdfs://localhost:8020/warehouse/tablespace/external/hive/reports.db";
 alterDatabaseLocation(dbName, hdfsLocation);
 assertDatabaseLocationRelationship(dbId);
-
-//AWS location
-String s3Location = 
"s3://localhost:8020/warehouse/tablespace/external/hive/reports.db";
-alterDatabaseLocation(dbName, s3Location);
-assertDatabaseLocationRelationship(dbId);
-
-//ABFS location
-String abfsLocation = 
"abfs://localhost:8020/warehouse/tablespace/external/hive/reports.db";
-alterDatabaseLocation(dbName, abfsLocation);
-assertDatabaseLocationRelationship(dbId);
 }
 
 //alter database location
@@ -158,20 +149,32 @@ public class HiveHookIT extends HiveITBase {
 }
 
 public void assertDatabaseLocationRelationship(String dbId) throws 
Exception {
-AtlasEntity dbEntity = atlasClientV2.getEntityByGuid(dbId).getEntity();
-AtlasEntityDef pathEntityDef = 
atlasClientV2.getEntityDefByName("Path");
+AtlasEntitydbEntity  = 
atlasClientV2.getEntityByGuid(dbId).getEntity();
+AtlasEntityDef pathEntityDef = getPathEntityDefWithAllSubTypes();
 
-//Check if dbEntity has location attribute
 assertTrue(dbEntity.hasAttribute(ATTRIBUTE_LOCATION));
-//Check if dbEntity has value for location attribute
+
 assertNotNull(dbEntity.getAttribute(ATTRIBUTE_LOCATION));
-//Check if dbEntity has location relationship attribute
-assertEquals(((List) 
dbEntity.getRelationshipAttribute(ATTRIBUTE_LOCATION_PATH)).size(), 1);
+
+
assertNotNull(dbEntity.getRelationshipAttribute(ATTRIBUTE_LOCATION_PATH));
+
 AtlasObjectId locationEntityObject = 
toAtlasObjectId(dbEntity.getRelationshipAttribute(ATTRIBUTE_LOCATION_PATH));
-//Check if location relationship attribute is subtype of "Path"
 
assertTrue(pathEntityDef.getSubTypes().contains(locationEntityObject.getTypeName()));
 }
 
+public AtlasEntityDef getPathEntityDefWithAllSubTypes() throws Exception {
+Set possiblePathSubTypes = new 
HashSet<>(Arrays.asList("fs_path", "hdfs_path", "aws_s3_pseudo_dir", 
"aws_s3_v2_directory", "adls_gen2_directory"));
+AtlasEntityDef  pathEntityDef= 
atlasClientV2.getEntityDefByName(PATH_TYPE_NAME);
+
+if(pathEntityDef == null) {
+pathEntityDef = new AtlasEntityDef(PATH_TYPE_NAME);
+}
+
+pathEntityDef.setSubTypes(possiblePathSubTypes);
+
+return pathEntityDef;
+}
+
 @Test
 public void testCreateTable() throws Exception {
 String tableName = tableName();

[atlas] branch branch-2.0 updated: ATLAS-4112 : Basic Search : Attribute search of QualifiedName beginswith operator not returning results when the value ends with a digit+dot

[sarath]

This is an automated email from the ASF dual-hosted git repository.

sarath pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/atlas.git


The following commit(s) were added to refs/heads/branch-2.0 by this push:
 new 0296ad6  ATLAS-4112 : Basic Search : Attribute search of QualifiedName 
beginswith operator not returning results when the value ends with a digit+dot
0296ad6 is described below

commit 0296ad6f4acbc00b66811655aa5729cff00769d0
Author: Pinal 
AuthorDate: Wed Jan 27 10:24:24 2021 +0530

ATLAS-4112 : Basic Search : Attribute search of QualifiedName beginswith 
operator not returning results when the value ends with a digit+dot

Signed-off-by: Sarath Subramanian 
(cherry picked from commit e873515f5c9bd4cbc6eda9609034779bcddaf3a4)
---
 .../org/apache/atlas/type/AtlasStructType.java |  48 ++-
 .../apache/atlas/discovery/SearchProcessor.java|  66 ++--
 .../atlas/discovery/AtlasDiscoveryServiceTest.java | 352 -
 3 files changed, 439 insertions(+), 27 deletions(-)

diff --git a/intg/src/main/java/org/apache/atlas/type/AtlasStructType.java 
b/intg/src/main/java/org/apache/atlas/type/AtlasStructType.java
index 0509809..d89aca2 100644
--- a/intg/src/main/java/org/apache/atlas/type/AtlasStructType.java
+++ b/intg/src/main/java/org/apache/atlas/type/AtlasStructType.java
@@ -960,10 +960,14 @@ public class AtlasStructType extends AtlasType {
 }
 
 public static String escapeIndexQueryValue(String value) {
-return escapeIndexQueryValue(value, false);
+return escapeIndexQueryValue(value, false, true);
 }
 
 public static String escapeIndexQueryValue(String value, boolean 
allowWildcard) {
+return escapeIndexQueryValue(value, allowWildcard, true);
+}
+
+public static String escapeIndexQueryValue(String value, boolean 
allowWildcard, boolean shouldQuote) {
 String  ret= value;
 boolean quoteValue = false;
 
@@ -977,7 +981,7 @@ public class AtlasStructType extends AtlasType {
 sb.append('\\');
 }
 
-if (!quoteValue) {
+if (shouldQuote && !quoteValue) {
 quoteValue = shouldQuoteIndexQueryForChar(c);
 }
 
@@ -987,7 +991,7 @@ public class AtlasStructType extends AtlasType {
 ret = sb.toString();
 } else if (value != null) {
 for (int i = 0; i < value.length(); i++) {
-if (shouldQuoteIndexQueryForChar(value.charAt(i))) {
+if (shouldQuote && 
shouldQuoteIndexQueryForChar(value.charAt(i))) {
 quoteValue = true;
 
 break;
@@ -1047,12 +1051,50 @@ public class AtlasStructType extends AtlasType {
 case ':':
 case '\\':
 case '/':
+case ' ':
 return true;
 }
 
 return false;
 }
 
+public static boolean hastokenizeChar(String value) {
+if (value != null) {
+for (int i = 0; i < value.length(); i++) {
+if (hastokenizeChar(value, i)) {
+return true;
+}
+}
+}
+
+return false;
+}
+
+
+private static boolean hastokenizeChar(String value, int i) {
+char c = value.charAt(i);
+if (!Character.isLetterOrDigit(c)) {
+switch (c) {
+case '_':
+return false;
+case '.':
+case ':':
+case '\'':
+if (i > 0 && !Character.isAlphabetic(value.charAt(i - 
1))) {
+return true;
+}
+if (i < value.length() - 1 && 
!Character.isAlphabetic(value.charAt(i + 1))) {
+return true;
+}
+return false;
+}
+
+return true;
+}
+
+return false;
+}
+
 private static boolean shouldQuoteIndexQueryForChar(char c) {
 switch (c) {
 case '@':
diff --git 
a/repository/src/main/java/org/apache/atlas/discovery/SearchProcessor.java 
b/repository/src/main/java/org/apache/atlas/discovery/SearchProcessor.java
index 01daf53..275fc78 100644
--- a/repository/src/main/java/org/apache/atlas/discovery/SearchProcessor.java
+++ b/repository/src/main/java/org/apache/atlas/discovery/SearchProcessor.java
@@ -26,6 +26,7 @@ import org.apache.atlas.model.discovery.SearchParameters;
 import org.apache.atlas.model.discovery.SearchParameters.FilterCriteria;
 import 
org.apache.atlas.model.discovery.Search

[atlas] branch master updated: ATLAS-4112 : Basic Search : Attribute search of QualifiedName beginswith operator not returning results when the value ends with a digit+dot

[sarath]

This is an automated email from the ASF dual-hosted git repository.

sarath pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/atlas.git


The following commit(s) were added to refs/heads/master by this push:
 new e873515  ATLAS-4112 : Basic Search : Attribute search of QualifiedName 
beginswith operator not returning results when the value ends with a digit+dot
e873515 is described below

commit e873515f5c9bd4cbc6eda9609034779bcddaf3a4
Author: Pinal 
AuthorDate: Wed Jan 27 10:24:24 2021 +0530

ATLAS-4112 : Basic Search : Attribute search of QualifiedName beginswith 
operator not returning results when the value ends with a digit+dot

Signed-off-by: Sarath Subramanian 
---
 .../org/apache/atlas/type/AtlasStructType.java |  48 ++-
 .../apache/atlas/discovery/SearchProcessor.java|  66 ++--
 .../atlas/discovery/AtlasDiscoveryServiceTest.java | 352 -
 3 files changed, 439 insertions(+), 27 deletions(-)

diff --git a/intg/src/main/java/org/apache/atlas/type/AtlasStructType.java 
b/intg/src/main/java/org/apache/atlas/type/AtlasStructType.java
index 0509809..d89aca2 100644
--- a/intg/src/main/java/org/apache/atlas/type/AtlasStructType.java
+++ b/intg/src/main/java/org/apache/atlas/type/AtlasStructType.java
@@ -960,10 +960,14 @@ public class AtlasStructType extends AtlasType {
 }
 
 public static String escapeIndexQueryValue(String value) {
-return escapeIndexQueryValue(value, false);
+return escapeIndexQueryValue(value, false, true);
 }
 
 public static String escapeIndexQueryValue(String value, boolean 
allowWildcard) {
+return escapeIndexQueryValue(value, allowWildcard, true);
+}
+
+public static String escapeIndexQueryValue(String value, boolean 
allowWildcard, boolean shouldQuote) {
 String  ret= value;
 boolean quoteValue = false;
 
@@ -977,7 +981,7 @@ public class AtlasStructType extends AtlasType {
 sb.append('\\');
 }
 
-if (!quoteValue) {
+if (shouldQuote && !quoteValue) {
 quoteValue = shouldQuoteIndexQueryForChar(c);
 }
 
@@ -987,7 +991,7 @@ public class AtlasStructType extends AtlasType {
 ret = sb.toString();
 } else if (value != null) {
 for (int i = 0; i < value.length(); i++) {
-if (shouldQuoteIndexQueryForChar(value.charAt(i))) {
+if (shouldQuote && 
shouldQuoteIndexQueryForChar(value.charAt(i))) {
 quoteValue = true;
 
 break;
@@ -1047,12 +1051,50 @@ public class AtlasStructType extends AtlasType {
 case ':':
 case '\\':
 case '/':
+case ' ':
 return true;
 }
 
 return false;
 }
 
+public static boolean hastokenizeChar(String value) {
+if (value != null) {
+for (int i = 0; i < value.length(); i++) {
+if (hastokenizeChar(value, i)) {
+return true;
+}
+}
+}
+
+return false;
+}
+
+
+private static boolean hastokenizeChar(String value, int i) {
+char c = value.charAt(i);
+if (!Character.isLetterOrDigit(c)) {
+switch (c) {
+case '_':
+return false;
+case '.':
+case ':':
+case '\'':
+if (i > 0 && !Character.isAlphabetic(value.charAt(i - 
1))) {
+return true;
+}
+if (i < value.length() - 1 && 
!Character.isAlphabetic(value.charAt(i + 1))) {
+return true;
+}
+return false;
+}
+
+return true;
+}
+
+return false;
+}
+
 private static boolean shouldQuoteIndexQueryForChar(char c) {
 switch (c) {
 case '@':
diff --git 
a/repository/src/main/java/org/apache/atlas/discovery/SearchProcessor.java 
b/repository/src/main/java/org/apache/atlas/discovery/SearchProcessor.java
index 01daf53..275fc78 100644
--- a/repository/src/main/java/org/apache/atlas/discovery/SearchProcessor.java
+++ b/repository/src/main/java/org/apache/atlas/discovery/SearchProcessor.java
@@ -26,6 +26,7 @@ import org.apache.atlas.model.discovery.SearchParameters;
 import org.apache.atlas.model.discovery.SearchParameters.FilterCriteria;
 import 
org.apache.atlas.model.discovery.SearchParameters.FilterCriteria.Condition;
 import org.apache.a

[atlas] branch branch-2.0 updated: ATLAS-4123: Handle create/drop materialized view queries in hive hook

[sarath]

This is an automated email from the ASF dual-hosted git repository.

sarath pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/atlas.git


The following commit(s) were added to refs/heads/branch-2.0 by this push:
 new d3e3697  ATLAS-4123: Handle create/drop materialized view queries in 
hive hook
d3e3697 is described below

commit d3e36976a3d24d0ffe8f3f2ed4198423c2c3cfac
Author: sidmishra 
AuthorDate: Tue Feb 2 16:59:21 2021 -0800

ATLAS-4123: Handle create/drop materialized view queries in hive hook

Signed-off-by: Sarath Subramanian 
(cherry picked from commit 0b02bd9f21eab4354b7d8479e05a38db75e5e1a0)
---
 .../hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java  | 2 ++
 1 file changed, 2 insertions(+)

diff --git 
a/addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
b/addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java
index 6513234..3b0ee6e 100644
--- a/addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java
+++ b/addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java
@@ -196,10 +196,12 @@ public class HiveHook extends AtlasHook implements 
ExecuteWithHookContext {
 
 case DROPTABLE:
 case DROPVIEW:
+case DROP_MATERIALIZED_VIEW:
 event = new DropTable(context);
 break;
 
 case CREATETABLE_AS_SELECT:
+case CREATE_MATERIALIZED_VIEW:
 case CREATEVIEW:
 case ALTERVIEW_AS:
 case LOAD:

[atlas] branch master updated: ATLAS-4123: Handle create/drop materialized view queries in hive hook

[sarath]

This is an automated email from the ASF dual-hosted git repository.

sarath pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/atlas.git


The following commit(s) were added to refs/heads/master by this push:
 new 0b02bd9  ATLAS-4123: Handle create/drop materialized view queries in 
hive hook
0b02bd9 is described below

commit 0b02bd9f21eab4354b7d8479e05a38db75e5e1a0
Author: sidmishra 
AuthorDate: Tue Feb 2 16:59:21 2021 -0800

ATLAS-4123: Handle create/drop materialized view queries in hive hook

Signed-off-by: Sarath Subramanian 
---
 .../hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java  | 2 ++
 1 file changed, 2 insertions(+)

diff --git 
a/addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
b/addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java
index e48967d..79e87c7 100644
--- a/addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java
+++ b/addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java
@@ -201,10 +201,12 @@ public class HiveHook extends AtlasHook implements 
ExecuteWithHookContext {
 
 case DROPTABLE:
 case DROPVIEW:
+case DROP_MATERIALIZED_VIEW:
 event = new DropTable(context);
 break;
 
 case CREATETABLE_AS_SELECT:
+case CREATE_MATERIALIZED_VIEW:
 case CREATEVIEW:
 case ALTERVIEW_AS:
 case LOAD:

[atlas] branch branch-2.0 updated: ATLAS-4052: Upgrade elasticsearch version to 6.8.3

[sarath]

This is an automated email from the ASF dual-hosted git repository.

sarath pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/atlas.git


The following commit(s) were added to refs/heads/branch-2.0 by this push:
 new fb28e83  ATLAS-4052: Upgrade elasticsearch version to 6.8.3
fb28e83 is described below

commit fb28e8318815e24f1c9c8e48021cea982e32a658
Author: chaitali borole 
AuthorDate: Tue Jan 19 20:05:39 2021 +0530

ATLAS-4052: Upgrade elasticsearch version to 6.8.3

Signed-off-by: Sarath Subramanian 
(cherry picked from commit f0bab76e8ec18ccbdf98c1d2fa8cc765ffb346e7)
---
 pom.xml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/pom.xml b/pom.xml
index e179c7d..46eedd1 100644
--- a/pom.xml
+++ b/pom.xml
@@ -686,7 +686,7 @@
 4.0.1
 1.8
 3.2.2
-6.6.0
+6.8.3
 
org.apache.atlas.repository.audit.InMemoryEntityAuditRepository
 2.13.2
 2.18.1

[atlas] branch master updated: ATLAS-4052: Upgrade elasticsearch version to 6.8.3

[sarath]

This is an automated email from the ASF dual-hosted git repository.

sarath pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/atlas.git


The following commit(s) were added to refs/heads/master by this push:
 new f0bab76  ATLAS-4052: Upgrade elasticsearch version to 6.8.3
f0bab76 is described below

commit f0bab76e8ec18ccbdf98c1d2fa8cc765ffb346e7
Author: chaitali borole 
AuthorDate: Tue Jan 19 20:05:39 2021 +0530

ATLAS-4052: Upgrade elasticsearch version to 6.8.3

Signed-off-by: Sarath Subramanian 
---
 pom.xml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/pom.xml b/pom.xml
index 7bfd68b..df4dec5 100644
--- a/pom.xml
+++ b/pom.xml
@@ -686,7 +686,7 @@
 4.0.1
 1.8
 3.2.2
-6.6.0
+6.8.3
 
org.apache.atlas.repository.audit.InMemoryEntityAuditRepository
 2.13.2
 2.18.1

[atlas] branch branch-2.0 updated: ATLAS-4107: Atlas not picking the ldap bind password from the correct jceks file #2 Unit test fix

[sarath]

This is an automated email from the ASF dual-hosted git repository.

sarath pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/atlas.git


The following commit(s) were added to refs/heads/branch-2.0 by this push:
 new 8dd04d2  ATLAS-4107: Atlas not picking the ldap bind password from the 
correct jceks file #2 Unit test fix
8dd04d2 is described below

commit 8dd04d27e332b882868e3d5b281ff771f4795862
Author: Sarath Subramanian 
AuthorDate: Thu Jan 28 15:23:36 2021 -0800

ATLAS-4107: Atlas not picking the ldap bind password from the correct jceks 
file #2 Unit test fix

(cherry picked from commit 39a067a215dc247ff787567e8f25ae52b83e010d)
---
 .../java/org/apache/atlas/web/service/SecureEmbeddedServerTestBase.java | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git 
a/webapp/src/test/java/org/apache/atlas/web/service/SecureEmbeddedServerTestBase.java
 
b/webapp/src/test/java/org/apache/atlas/web/service/SecureEmbeddedServerTestBase.java
index 9eff636..57807f3 100755
--- 
a/webapp/src/test/java/org/apache/atlas/web/service/SecureEmbeddedServerTestBase.java
+++ 
b/webapp/src/test/java/org/apache/atlas/web/service/SecureEmbeddedServerTestBase.java
@@ -111,7 +111,7 @@ public class SecureEmbeddedServerTestBase {
 Assert.fail("Should have thrown an exception");
 } catch (IOException e) {
 Assert.assertEquals(e.getMessage(),
-"No credential provider path configured for storage of 
certificate store passwords");
+"No credential provider path 
cert.stores.credential.provider.path configured for storage of certificate 
store passwords");
 } finally {
 if (secureEmbeddedServer != null) {
 secureEmbeddedServer.server.stop();

[atlas] branch master updated: ATLAS-4107: Atlas not picking the ldap bind password from the correct jceks file #2 Unit test fix

[sarath]

This is an automated email from the ASF dual-hosted git repository.

sarath pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/atlas.git


The following commit(s) were added to refs/heads/master by this push:
 new 39a067a  ATLAS-4107: Atlas not picking the ldap bind password from the 
correct jceks file #2 Unit test fix
39a067a is described below

commit 39a067a215dc247ff787567e8f25ae52b83e010d
Author: Sarath Subramanian 
AuthorDate: Thu Jan 28 15:23:36 2021 -0800

ATLAS-4107: Atlas not picking the ldap bind password from the correct jceks 
file #2 Unit test fix
---
 .../java/org/apache/atlas/web/service/SecureEmbeddedServerTestBase.java | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git 
a/webapp/src/test/java/org/apache/atlas/web/service/SecureEmbeddedServerTestBase.java
 
b/webapp/src/test/java/org/apache/atlas/web/service/SecureEmbeddedServerTestBase.java
index 9eff636..57807f3 100755
--- 
a/webapp/src/test/java/org/apache/atlas/web/service/SecureEmbeddedServerTestBase.java
+++ 
b/webapp/src/test/java/org/apache/atlas/web/service/SecureEmbeddedServerTestBase.java
@@ -111,7 +111,7 @@ public class SecureEmbeddedServerTestBase {
 Assert.fail("Should have thrown an exception");
 } catch (IOException e) {
 Assert.assertEquals(e.getMessage(),
-"No credential provider path configured for storage of 
certificate store passwords");
+"No credential provider path 
cert.stores.credential.provider.path configured for storage of certificate 
store passwords");
 } finally {
 if (secureEmbeddedServer != null) {
 secureEmbeddedServer.server.stop();

[atlas] branch branch-2.0 updated: ATLAS-4107: Atlas not picking the ldap bind password from the correct jceks file.

[sarath]

This is an automated email from the ASF dual-hosted git repository.

sarath pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/atlas.git


The following commit(s) were added to refs/heads/branch-2.0 by this push:
 new 634db60  ATLAS-4107: Atlas not picking the ldap bind password from the 
correct jceks file.
634db60 is described below

commit 634db601bec844584875350865e61dd010515ebf
Author: nixonrodrigues 
AuthorDate: Thu Jan 28 20:04:28 2021 +0530

ATLAS-4107: Atlas not picking the ldap bind password from the correct jceks 
file.

Change-Id: I8f457b63f3170c2b1313ab365223d18af6023f87
Signed-off-by: Sarath Subramanian 
(cherry picked from commit bb745b9e90c517517f88cd66ccca845bf73dfeca)
---
 .../org/apache/atlas/ApplicationProperties.java | 11 +++
 .../apache/atlas/security/SecurityProperties.java   |  1 +
 .../org/apache/atlas/security/SecurityUtil.java | 21 ++---
 3 files changed, 26 insertions(+), 7 deletions(-)

diff --git a/intg/src/main/java/org/apache/atlas/ApplicationProperties.java 
b/intg/src/main/java/org/apache/atlas/ApplicationProperties.java
index e662c8f..bf97ab1 100644
--- a/intg/src/main/java/org/apache/atlas/ApplicationProperties.java
+++ b/intg/src/main/java/org/apache/atlas/ApplicationProperties.java
@@ -34,6 +34,7 @@ import java.net.URL;
 import java.util.AbstractMap.SimpleEntry;
 import java.util.Iterator;
 import java.util.Properties;
+import static 
org.apache.atlas.security.SecurityProperties.HADOOP_SECURITY_CREDENTIAL_PROVIDER_PATH;
 
 /**
  * Application properties used by Atlas.
@@ -57,6 +58,8 @@ public final class ApplicationProperties extends 
PropertiesConfiguration {
 public static final String  STORAGE_BACKEND_HBASE2  = "hbase2";
 public static final String  INDEX_BACKEND_SOLR  = "solr";
 public static final String  LDAP_TYPE   =  
"atlas.authentication.method.ldap.type";
+public static final String  LDAP=  "LDAP";
+public static final String  AD  =  "AD";
 public static final String  LDAP_AD_BIND_PASSWORD   =  
"atlas.authentication.method.ldap.ad.bind.password";
 public static final String  LDAP_BIND_PASSWORD  =  
"atlas.authentication.method.ldap.bind.password";
 public static final String  MASK_LDAP_PASSWORD  =  "";
@@ -278,17 +281,17 @@ public final class ApplicationProperties extends 
PropertiesConfiguration {
 
 if (StringUtils.isNotEmpty(ldapType)) {
 try {
-if (ldapType.equalsIgnoreCase("ldap")) {
+if (ldapType.equalsIgnoreCase(LDAP)) {
 String maskPasssword = 
configuration.getString(LDAP_BIND_PASSWORD);
 if (MASK_LDAP_PASSWORD.equals(maskPasssword)) {
-String password = 
SecurityUtil.getPassword(configuration, LDAP_BIND_PASSWORD);
+String password = 
SecurityUtil.getPassword(configuration, LDAP_BIND_PASSWORD, 
HADOOP_SECURITY_CREDENTIAL_PROVIDER_PATH);
 configuration.clearProperty(LDAP_BIND_PASSWORD);
 configuration.addProperty(LDAP_BIND_PASSWORD, 
password);
 }
-} else if (ldapType.equalsIgnoreCase("ad")) {
+} else if (ldapType.equalsIgnoreCase(AD)) {
 String maskPasssword = 
configuration.getString(LDAP_AD_BIND_PASSWORD);
 if (MASK_LDAP_PASSWORD.equals(maskPasssword)) {
-String password = 
SecurityUtil.getPassword(configuration, LDAP_AD_BIND_PASSWORD);
+String password = 
SecurityUtil.getPassword(configuration, LDAP_AD_BIND_PASSWORD, 
HADOOP_SECURITY_CREDENTIAL_PROVIDER_PATH);
 configuration.clearProperty(LDAP_AD_BIND_PASSWORD);
 configuration.addProperty(LDAP_AD_BIND_PASSWORD, 
password);
 }
diff --git 
a/intg/src/main/java/org/apache/atlas/security/SecurityProperties.java 
b/intg/src/main/java/org/apache/atlas/security/SecurityProperties.java
index 2147cd1..0d94986 100644
--- a/intg/src/main/java/org/apache/atlas/security/SecurityProperties.java
+++ b/intg/src/main/java/org/apache/atlas/security/SecurityProperties.java
@@ -40,6 +40,7 @@ public final class SecurityProperties {
 public static final String SERVER_CERT_PASSWORD_KEY = "password";
 public static final String CLIENT_AUTH_KEY = "client.auth.enabled";
 public static final String CERT_STORES_CREDENTIAL_PROVIDER_PATH = 
"cert.stores.credential.provider.path";
+public static final String HADOOP_SECURITY_CREDENTIAL_PROVIDER_PATH = 
"hadoop.security.credential.provider.path";
 public static final String SS

[atlas] branch master updated: ATLAS-4107: Atlas not picking the ldap bind password from the correct jceks file.

[sarath]

This is an automated email from the ASF dual-hosted git repository.

sarath pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/atlas.git


The following commit(s) were added to refs/heads/master by this push:
 new bb745b9  ATLAS-4107: Atlas not picking the ldap bind password from the 
correct jceks file.
bb745b9 is described below

commit bb745b9e90c517517f88cd66ccca845bf73dfeca
Author: nixonrodrigues 
AuthorDate: Thu Jan 28 20:04:28 2021 +0530

ATLAS-4107: Atlas not picking the ldap bind password from the correct jceks 
file.

Change-Id: I8f457b63f3170c2b1313ab365223d18af6023f87
Signed-off-by: Sarath Subramanian 
---
 .../org/apache/atlas/ApplicationProperties.java | 11 +++
 .../apache/atlas/security/SecurityProperties.java   |  1 +
 .../org/apache/atlas/security/SecurityUtil.java | 21 ++---
 3 files changed, 26 insertions(+), 7 deletions(-)

diff --git a/intg/src/main/java/org/apache/atlas/ApplicationProperties.java 
b/intg/src/main/java/org/apache/atlas/ApplicationProperties.java
index e662c8f..bf97ab1 100644
--- a/intg/src/main/java/org/apache/atlas/ApplicationProperties.java
+++ b/intg/src/main/java/org/apache/atlas/ApplicationProperties.java
@@ -34,6 +34,7 @@ import java.net.URL;
 import java.util.AbstractMap.SimpleEntry;
 import java.util.Iterator;
 import java.util.Properties;
+import static 
org.apache.atlas.security.SecurityProperties.HADOOP_SECURITY_CREDENTIAL_PROVIDER_PATH;
 
 /**
  * Application properties used by Atlas.
@@ -57,6 +58,8 @@ public final class ApplicationProperties extends 
PropertiesConfiguration {
 public static final String  STORAGE_BACKEND_HBASE2  = "hbase2";
 public static final String  INDEX_BACKEND_SOLR  = "solr";
 public static final String  LDAP_TYPE   =  
"atlas.authentication.method.ldap.type";
+public static final String  LDAP=  "LDAP";
+public static final String  AD  =  "AD";
 public static final String  LDAP_AD_BIND_PASSWORD   =  
"atlas.authentication.method.ldap.ad.bind.password";
 public static final String  LDAP_BIND_PASSWORD  =  
"atlas.authentication.method.ldap.bind.password";
 public static final String  MASK_LDAP_PASSWORD  =  "";
@@ -278,17 +281,17 @@ public final class ApplicationProperties extends 
PropertiesConfiguration {
 
 if (StringUtils.isNotEmpty(ldapType)) {
 try {
-if (ldapType.equalsIgnoreCase("ldap")) {
+if (ldapType.equalsIgnoreCase(LDAP)) {
 String maskPasssword = 
configuration.getString(LDAP_BIND_PASSWORD);
 if (MASK_LDAP_PASSWORD.equals(maskPasssword)) {
-String password = 
SecurityUtil.getPassword(configuration, LDAP_BIND_PASSWORD);
+String password = 
SecurityUtil.getPassword(configuration, LDAP_BIND_PASSWORD, 
HADOOP_SECURITY_CREDENTIAL_PROVIDER_PATH);
 configuration.clearProperty(LDAP_BIND_PASSWORD);
 configuration.addProperty(LDAP_BIND_PASSWORD, 
password);
 }
-} else if (ldapType.equalsIgnoreCase("ad")) {
+} else if (ldapType.equalsIgnoreCase(AD)) {
 String maskPasssword = 
configuration.getString(LDAP_AD_BIND_PASSWORD);
 if (MASK_LDAP_PASSWORD.equals(maskPasssword)) {
-String password = 
SecurityUtil.getPassword(configuration, LDAP_AD_BIND_PASSWORD);
+String password = 
SecurityUtil.getPassword(configuration, LDAP_AD_BIND_PASSWORD, 
HADOOP_SECURITY_CREDENTIAL_PROVIDER_PATH);
 configuration.clearProperty(LDAP_AD_BIND_PASSWORD);
 configuration.addProperty(LDAP_AD_BIND_PASSWORD, 
password);
 }
diff --git 
a/intg/src/main/java/org/apache/atlas/security/SecurityProperties.java 
b/intg/src/main/java/org/apache/atlas/security/SecurityProperties.java
index 2147cd1..0d94986 100644
--- a/intg/src/main/java/org/apache/atlas/security/SecurityProperties.java
+++ b/intg/src/main/java/org/apache/atlas/security/SecurityProperties.java
@@ -40,6 +40,7 @@ public final class SecurityProperties {
 public static final String SERVER_CERT_PASSWORD_KEY = "password";
 public static final String CLIENT_AUTH_KEY = "client.auth.enabled";
 public static final String CERT_STORES_CREDENTIAL_PROVIDER_PATH = 
"cert.stores.credential.provider.path";
+public static final String HADOOP_SECURITY_CREDENTIAL_PROVIDER_PATH = 
"hadoop.security.credential.provider.path";
 public static final String SSL_CLIENT_PROPERTIES = "ssl-client.xml";
 public static final 

[atlas] branch branch-2.0 updated: ATLAS-4111: Add 'replicationFactor' attribute to kafka_topic entity type + formatting changes

[sarath]

This is an automated email from the ASF dual-hosted git repository.

sarath pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/atlas.git


The following commit(s) were added to refs/heads/branch-2.0 by this push:
 new ade75fe  ATLAS-4111: Add 'replicationFactor' attribute to kafka_topic 
entity type + formatting changes
ade75fe is described below

commit ade75fe55f88ef3b25be8da18acfbab3a8d293e2
Author: Barnabas Maidics 
AuthorDate: Mon Jan 25 14:26:45 2021 +0100

ATLAS-4111: Add 'replicationFactor' attribute to kafka_topic entity type + 
formatting changes

Signed-off-by: Sarath Subramanian 
(cherry picked from commit ce1342dd6dd7cbc4d1e35bea3b26d08dc24bd67e)
---
 .../org/apache/atlas/kafka/bridge/KafkaBridge.java |  66 
 .../patches/018-kafka_topic_add_rf_attribute.json  |  23 +++
 .../java/org/apache/atlas/utils/KafkaUtils.java| 173 ++---
 3 files changed, 174 insertions(+), 88 deletions(-)

diff --git 
a/addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/KafkaBridge.java
 
b/addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/KafkaBridge.java
index d22010d..bf74c67 100644
--- 
a/addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/KafkaBridge.java
+++ 
b/addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/KafkaBridge.java
@@ -54,24 +54,23 @@ import java.util.concurrent.ExecutionException;
 import java.util.regex.Pattern;
 
 public class KafkaBridge {
-private static final Logger LOG = 
LoggerFactory.getLogger(KafkaBridge.class);
-
-private static final intEXIT_CODE_SUCCESS  = 0;
-private static final intEXIT_CODE_FAILED   = 1;
-private static final String ATLAS_ENDPOINT = 
"atlas.rest.address";
-private static final String DEFAULT_ATLAS_URL  = 
"http://localhost:21000/;;
-private static final String CLUSTER_NAME_KEY   = 
"atlas.cluster.name";
-private static final String KAFKA_METADATA_NAMESPACE   = 
"atlas.metadata.namespace";
-private static final String DEFAULT_CLUSTER_NAME   = "primary";
-private static final String ATTRIBUTE_QUALIFIED_NAME   = "qualifiedName";
-private static final String DESCRIPTION_ATTR   = "description";
-private static final String PARTITION_COUNT= "partitionCount";
-private static final String NAME   = "name";
-private static final String URI= "uri";
-private static final String CLUSTERNAME= "clusterName";
-private static final String TOPIC  = "topic";
-
-private static final String FORMAT_KAKFA_TOPIC_QUALIFIED_NAME   = 
"%s@%s";
+private static final Logger LOG   = 
LoggerFactory.getLogger(KafkaBridge.class);
+private static final intEXIT_CODE_SUCCESS = 0;
+private static final intEXIT_CODE_FAILED  = 1;
+private static final String ATLAS_ENDPOINT= 
"atlas.rest.address";
+private static final String DEFAULT_ATLAS_URL = 
"http://localhost:21000/;;
+private static final String CLUSTER_NAME_KEY  = 
"atlas.cluster.name";
+private static final String KAFKA_METADATA_NAMESPACE  = 
"atlas.metadata.namespace";
+private static final String DEFAULT_CLUSTER_NAME  = "primary";
+private static final String ATTRIBUTE_QUALIFIED_NAME  = 
"qualifiedName";
+private static final String DESCRIPTION_ATTR  = 
"description";
+private static final String PARTITION_COUNT   = 
"partitionCount";
+private static final String REPLICATION_FACTOR= 
"replicationFactor";
+private static final String NAME  = "name";
+private static final String URI   = "uri";
+private static final String CLUSTERNAME   = 
"clusterName";
+private static final String TOPIC = "topic";
+private static final String FORMAT_KAKFA_TOPIC_QUALIFIED_NAME = "%s@%s";
 
 private final List  availableTopics;
 private final StringmetadataNamespace;
@@ -80,9 +79,9 @@ public class KafkaBridge {
 
 
 public static void main(String[] args) {
-int exitCode = EXIT_CODE_FAILED;
+int   exitCode  = EXIT_CODE_FAILED;
 AtlasClientV2 atlasClientV2 = null;
-KafkaUtils kafkaUtils = null;
+KafkaUtilskafkaUtils= null;
 
 try {
 Options options = new Options();
@@ -114,14 +113,15 @@ public 

[atlas] branch master updated: ATLAS-4111: Add 'replicationFactor' attribute to kafka_topic entity type + formatting changes

[sarath]

This is an automated email from the ASF dual-hosted git repository.

sarath pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/atlas.git


The following commit(s) were added to refs/heads/master by this push:
 new ce1342d  ATLAS-4111: Add 'replicationFactor' attribute to kafka_topic 
entity type + formatting changes
ce1342d is described below

commit ce1342dd6dd7cbc4d1e35bea3b26d08dc24bd67e
Author: Barnabas Maidics 
AuthorDate: Mon Jan 25 14:26:45 2021 +0100

ATLAS-4111: Add 'replicationFactor' attribute to kafka_topic entity type + 
formatting changes

Signed-off-by: Sarath Subramanian 
---
 .../org/apache/atlas/kafka/bridge/KafkaBridge.java |  66 
 .../patches/018-kafka_topic_add_rf_attribute.json  |  23 +++
 .../java/org/apache/atlas/utils/KafkaUtils.java| 173 ++---
 3 files changed, 174 insertions(+), 88 deletions(-)

diff --git 
a/addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/KafkaBridge.java
 
b/addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/KafkaBridge.java
index d22010d..bf74c67 100644
--- 
a/addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/KafkaBridge.java
+++ 
b/addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/KafkaBridge.java
@@ -54,24 +54,23 @@ import java.util.concurrent.ExecutionException;
 import java.util.regex.Pattern;
 
 public class KafkaBridge {
-private static final Logger LOG = 
LoggerFactory.getLogger(KafkaBridge.class);
-
-private static final intEXIT_CODE_SUCCESS  = 0;
-private static final intEXIT_CODE_FAILED   = 1;
-private static final String ATLAS_ENDPOINT = 
"atlas.rest.address";
-private static final String DEFAULT_ATLAS_URL  = 
"http://localhost:21000/;;
-private static final String CLUSTER_NAME_KEY   = 
"atlas.cluster.name";
-private static final String KAFKA_METADATA_NAMESPACE   = 
"atlas.metadata.namespace";
-private static final String DEFAULT_CLUSTER_NAME   = "primary";
-private static final String ATTRIBUTE_QUALIFIED_NAME   = "qualifiedName";
-private static final String DESCRIPTION_ATTR   = "description";
-private static final String PARTITION_COUNT= "partitionCount";
-private static final String NAME   = "name";
-private static final String URI= "uri";
-private static final String CLUSTERNAME= "clusterName";
-private static final String TOPIC  = "topic";
-
-private static final String FORMAT_KAKFA_TOPIC_QUALIFIED_NAME   = 
"%s@%s";
+private static final Logger LOG   = 
LoggerFactory.getLogger(KafkaBridge.class);
+private static final intEXIT_CODE_SUCCESS = 0;
+private static final intEXIT_CODE_FAILED  = 1;
+private static final String ATLAS_ENDPOINT= 
"atlas.rest.address";
+private static final String DEFAULT_ATLAS_URL = 
"http://localhost:21000/;;
+private static final String CLUSTER_NAME_KEY  = 
"atlas.cluster.name";
+private static final String KAFKA_METADATA_NAMESPACE  = 
"atlas.metadata.namespace";
+private static final String DEFAULT_CLUSTER_NAME  = "primary";
+private static final String ATTRIBUTE_QUALIFIED_NAME  = 
"qualifiedName";
+private static final String DESCRIPTION_ATTR  = 
"description";
+private static final String PARTITION_COUNT   = 
"partitionCount";
+private static final String REPLICATION_FACTOR= 
"replicationFactor";
+private static final String NAME  = "name";
+private static final String URI   = "uri";
+private static final String CLUSTERNAME   = 
"clusterName";
+private static final String TOPIC = "topic";
+private static final String FORMAT_KAKFA_TOPIC_QUALIFIED_NAME = "%s@%s";
 
 private final List  availableTopics;
 private final StringmetadataNamespace;
@@ -80,9 +79,9 @@ public class KafkaBridge {
 
 
 public static void main(String[] args) {
-int exitCode = EXIT_CODE_FAILED;
+int   exitCode  = EXIT_CODE_FAILED;
 AtlasClientV2 atlasClientV2 = null;
-KafkaUtils kafkaUtils = null;
+KafkaUtilskafkaUtils= null;
 
 try {
 Options options = new Options();
@@ -114,14 +113,15 @@ public class KafkaBridge {
 kafkaUtils = new KafkaUtils(a

[atlas] 01/02: ATLAS-4110: updated handling of neq operator for recent changes in JanusGraph

[sarath]

This is an automated email from the ASF dual-hosted git repository.

sarath pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/atlas.git

commit 63ef2d8b623c6bfb6da64d9a922c8222fd405ae9
Author: Deep Singh 
AuthorDate: Thu Jan 21 17:14:12 2021 -0600

ATLAS-4110: updated handling of neq operator for recent changes in 
JanusGraph

Signed-off-by: Madhan Neethiraj 
(cherry picked from commit c48e4c393b747b25f1c399a2c37175e5004d1526)
---
 repository/src/main/java/org/apache/atlas/query/GremlinClause.java  | 1 +
 .../src/main/java/org/apache/atlas/query/GremlinQueryComposer.java  | 4 
 .../test/java/org/apache/atlas/query/GremlinQueryComposerTest.java  | 6 ++
 3 files changed, 11 insertions(+)

diff --git a/repository/src/main/java/org/apache/atlas/query/GremlinClause.java 
b/repository/src/main/java/org/apache/atlas/query/GremlinClause.java
index 7a98ddd..b7e5330 100644
--- a/repository/src/main/java/org/apache/atlas/query/GremlinClause.java
+++ b/repository/src/main/java/org/apache/atlas/query/GremlinClause.java
@@ -25,6 +25,7 @@ public enum GremlinClause {
 GROUP_BY("group().by('%s')"),
 HAS("has('%s', %s)"),
 HAS_OPERATOR("has('%s', %s(%s))"),
+HAS_NOT_OPERATOR("or(__.has('%s', neq(%s)), __.hasNot('%s'))"),
 HAS_PROPERTY("has('%s')"),
 WHERE("where(%s)"),
 HAS_NOT_PROPERTY("hasNot('%s')"),
diff --git 
a/repository/src/main/java/org/apache/atlas/query/GremlinQueryComposer.java 
b/repository/src/main/java/org/apache/atlas/query/GremlinQueryComposer.java
index c5c53fc..c7d01cf 100644
--- a/repository/src/main/java/org/apache/atlas/query/GremlinQueryComposer.java
+++ b/repository/src/main/java/org/apache/atlas/query/GremlinQueryComposer.java
@@ -219,6 +219,10 @@ public class GremlinQueryComposer {
 }
 } else if (op == SearchParameters.Operator.IN) {
 add(GremlinClause.HAS_OPERATOR, getPropertyForClause(lhsI), 
"within", rhs);
+} else if (op == SearchParameters.Operator.NEQ) {
+String propertyName = getPropertyForClause(lhsI);
+
+add(GremlinClause.HAS_NOT_OPERATOR, propertyName, rhs, 
propertyName);
 } else {
 Object normalizedRhs = getNormalizedAttrVal(lhsI, 
IdentifierHelper.removeQuotes(rhs));
 
diff --git 
a/repository/src/test/java/org/apache/atlas/query/GremlinQueryComposerTest.java 
b/repository/src/test/java/org/apache/atlas/query/GremlinQueryComposerTest.java
index 96e2840..0e8af18 100644
--- 
a/repository/src/test/java/org/apache/atlas/query/GremlinQueryComposerTest.java
+++ 
b/repository/src/test/java/org/apache/atlas/query/GremlinQueryComposerTest.java
@@ -385,6 +385,12 @@ public class GremlinQueryComposerTest {
 }
 
 @Test
+public void whereClauseWithNEQCondition() {
+verify("Table where owner != 'random'",
+"g.V().has('__typeName', 'Table').or(__.has('Table.owner', 
neq('random')), __.hasNot('Table.owner')).dedup().limit(25).toList()");
+}
+
+@Test
 public void invalidQueries() {
 verify("hdfs_path like h1", "");
 }

[atlas] branch branch-2.0 updated (9fcdf0f -> 0dc87b8)

[sarath]

This is an automated email from the ASF dual-hosted git repository.

sarath pushed a change to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/atlas.git.


from 9fcdf0f  ATLAS-4108: updated free-text search processor to support 
Elasticsearch
 new 63ef2d8  ATLAS-4110: updated handling of neq operator for recent 
changes in JanusGraph
 new 0dc87b8  ATLAS-4110: Inconsistency in the result of DSL query and 
basic query

The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../src/main/java/org/apache/atlas/query/GremlinClause.java |  1 +
 .../java/org/apache/atlas/query/GremlinQueryComposer.java   | 13 +++--
 .../java/org/apache/atlas/query/RegistryBasedLookup.java|  1 +
 .../query/executors/GremlinClauseToTraversalTranslator.java |  6 ++
 .../src/test/java/org/apache/atlas/BasicTestSetup.java  |  2 +-
 .../test/java/org/apache/atlas/query/DSLQueriesTest.java|  5 -
 .../org/apache/atlas/query/GremlinQueryComposerTest.java|  6 ++
 7 files changed, 30 insertions(+), 4 deletions(-)