atlas git commit: ATLAS-2232: update enunciate verision from 2.8.0 to 2.10.1. Set source, target versions to 1.8

2017-11-17 Thread madhan
Repository: atlas
Updated Branches:
  refs/heads/master a74f1b335 -> 12d83b5bd


ATLAS-2232: update enunciate verision from 2.8.0 to 2.10.1. Set source,target 
versions to 1.8


Project: http://git-wip-us.apache.org/repos/asf/atlas/repo
Commit: http://git-wip-us.apache.org/repos/asf/atlas/commit/12d83b5b
Tree: http://git-wip-us.apache.org/repos/asf/atlas/tree/12d83b5b
Diff: http://git-wip-us.apache.org/repos/asf/atlas/diff/12d83b5b

Branch: refs/heads/master
Commit: 12d83b5bd52fe1314004188475b59cc945bcd6d1
Parents: a74f1b3
Author: Madhan Neethiraj <mad...@apache.org>
Authored: Fri Nov 17 09:41:43 2017 -0800
Committer: Madhan Neethiraj <mad...@apache.org>
Committed: Fri Nov 17 13:54:45 2017 -0800

--
 pom.xml| 5 +++--
 webapp/pom.xml | 5 -
 2 files changed, 7 insertions(+), 3 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/atlas/blob/12d83b5b/pom.xml
--
diff --git a/pom.xml b/pom.xml
index 0fd07e2..3720c1f 100644
--- a/pom.xml
+++ b/pom.xml
@@ -563,6 +563,7 @@
 1.3.7
 2.7
 0.8
+2.10.1
 
 64m
 512m
@@ -1488,7 +1489,7 @@
 
 com.webcohesion.enunciate
 enunciate-core-annotations
-2.8.0
+${enunciate-maven-plugin.version}
 
 
 
@@ -1749,7 +1750,7 @@
 
 com.webcohesion.enunciate
 enunciate-maven-plugin
-2.8.0
+${enunciate-maven-plugin.version}
 
 
 

http://git-wip-us.apache.org/repos/asf/atlas/blob/12d83b5b/webapp/pom.xml
--
diff --git a/webapp/pom.xml b/webapp/pom.xml
index bcb5a16..3e11b21 100755
--- a/webapp/pom.xml
+++ b/webapp/pom.xml
@@ -392,7 +392,7 @@
 test
 
 
-   
+
 org.apache.atlas
 atlas-repository
 tests
@@ -681,10 +681,13 @@
 
 com.webcohesion.enunciate
 enunciate-maven-plugin
+${enunciate-maven-plugin.version}
 
 
${project.parent.basedir}/build-tools/src/main/resources/enunciate.xml
 
 ${project.build.directory}/api/v2/
+1.8
+1.8
 ${skipEnunciate}
 
 



atlas git commit: ATLAS-2269: Error in the Area 0 and 3 model files

2017-11-20 Thread madhan
Repository: atlas
Updated Branches:
  refs/heads/master 0765d5796 -> 4a8777b40


ATLAS-2269: Error in the Area 0 and 3 model files

Signed-off-by: Madhan Neethiraj <mad...@apache.org>


Project: http://git-wip-us.apache.org/repos/asf/atlas/repo
Commit: http://git-wip-us.apache.org/repos/asf/atlas/commit/4a8777b4
Tree: http://git-wip-us.apache.org/repos/asf/atlas/tree/4a8777b4
Diff: http://git-wip-us.apache.org/repos/asf/atlas/diff/4a8777b4

Branch: refs/heads/master
Commit: 4a8777b40e641584c088e6e9bc14145f72f681c0
Parents: 0765d57
Author: David Radley <david_rad...@uk.ibm.com>
Authored: Mon Nov 20 15:38:17 2017 +0000
Committer: Madhan Neethiraj <mad...@apache.org>
Committed: Mon Nov 20 09:58:29 2017 -0800

--
 addons/models/-Area0/0090-CloudPlatformsAndServices.json  | 2 +-
 addons/models/0300-Area3-SubjectArea/0320-CategoryHierarchy.json  | 1 -
 addons/models/0300-Area3-SubjectArea/0330-Terms.json  | 1 -
 addons/models/0300-Area3-SubjectArea/0350-RelatedTerms.json   | 2 +-
 addons/models/0300-Area3-SubjectArea/0370-SemanticAssignment.json | 3 +--
 5 files changed, 3 insertions(+), 6 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/atlas/blob/4a8777b4/addons/models/-Area0/0090-CloudPlatformsAndServices.json
--
diff --git a/addons/models/-Area0/0090-CloudPlatformsAndServices.json 
b/addons/models/-Area0/0090-CloudPlatformsAndServices.json
index 803c0d1..de5fb36 100644
--- a/addons/models/-Area0/0090-CloudPlatformsAndServices.json
+++ b/addons/models/-Area0/0090-CloudPlatformsAndServices.json
@@ -21,7 +21,7 @@
 },
 {
   "name": "CloudPlatform",
-  "descriptiojn":"If the cloud provider is offering platform as a service 
(PaaS), an application may deploy server capability onto the cloud platform.",
+  "description":"If the cloud provider is offering platform as a service 
(PaaS), an application may deploy server capability onto the cloud platform.",
   "superTypes": [],
   "entityTypes":["Server"],
   "typeVersion": "1.0",

http://git-wip-us.apache.org/repos/asf/atlas/blob/4a8777b4/addons/models/0300-Area3-SubjectArea/0320-CategoryHierarchy.json
--
diff --git a/addons/models/0300-Area3-SubjectArea/0320-CategoryHierarchy.json 
b/addons/models/0300-Area3-SubjectArea/0320-CategoryHierarchy.json
index 804be9e..1063797 100644
--- a/addons/models/0300-Area3-SubjectArea/0320-CategoryHierarchy.json
+++ b/addons/models/0300-Area3-SubjectArea/0320-CategoryHierarchy.json
@@ -86,7 +86,6 @@
 },
 {
   "name": "LibraryCategoryReference",
-  "superTypes": [],
   "typeVersion": "1.0",
   "description": "LibraryCategoryReference provides reference information 
for how this category corresponds to a category in an external glossary.",
   "endDef1": {

http://git-wip-us.apache.org/repos/asf/atlas/blob/4a8777b4/addons/models/0300-Area3-SubjectArea/0330-Terms.json
--
diff --git a/addons/models/0300-Area3-SubjectArea/0330-Terms.json 
b/addons/models/0300-Area3-SubjectArea/0330-Terms.json
index c267fb2..73315ab 100644
--- a/addons/models/0300-Area3-SubjectArea/0330-Terms.json
+++ b/addons/models/0300-Area3-SubjectArea/0330-Terms.json
@@ -153,7 +153,6 @@
 },
 {
   "name": "LibraryTermReference",
-  "superTypes": [],
   "typeVersion": "1.0",
   "description": "Library Term Reference provides reference information 
for how this term corresponds to a term in an external glossary.",
   "endDef1": {

http://git-wip-us.apache.org/repos/asf/atlas/blob/4a8777b4/addons/models/0300-Area3-SubjectArea/0350-RelatedTerms.json
--
diff --git a/addons/models/0300-Area3-SubjectArea/0350-RelatedTerms.json 
b/addons/models/0300-Area3-SubjectArea/0350-RelatedTerms.json
index 37e1141..d88f57c 100644
--- a/addons/models/0300-Area3-SubjectArea/0350-RelatedTerms.json
+++ b/addons/models/0300-Area3-SubjectArea/0350-RelatedTerms.json
@@ -289,7 +289,7 @@
   "typeVersion": "1.0",
   "description": "Translation is a relationship that defines that the 
related terms represent the same meaning, but each are written in a different 
language. Hence one is a translation of the other. The language of each term is 
defined in the Glos

[05/42] atlas git commit: ATLAS-2251: Remove TypeSystem and related implementation, to avoid unncessary duplicate of type details in cache

2017-11-12 Thread madhan
http://git-wip-us.apache.org/repos/asf/atlas/blob/435fe3fb/typesystem/src/test/scala/org/apache/atlas/typesystem/builders/MultiplicityTest.scala
--
diff --git 
a/typesystem/src/test/scala/org/apache/atlas/typesystem/builders/MultiplicityTest.scala
 
b/typesystem/src/test/scala/org/apache/atlas/typesystem/builders/MultiplicityTest.scala
deleted file mode 100644
index 91e72c7..000
--- 
a/typesystem/src/test/scala/org/apache/atlas/typesystem/builders/MultiplicityTest.scala
+++ /dev/null
@@ -1,124 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.atlas.typesystem.builders
-
-import org.apache.atlas.AtlasException
-import org.apache.atlas.typesystem.types.{ClassType, Multiplicity, TypeSystem}
-import org.testng.annotations.{BeforeMethod,Test}
-
-class MultiplicityTest {
-
-  @BeforeMethod
-  def beforeAll {
-TypeSystem.getInstance().reset()
-
-val b = new TypesBuilder
-import b._
-
-val tDef = types {
-
-  _trait("Dimension") {}
-  _trait("PII") {}
-  _trait("Metric") {}
-  _trait("ETL") {}
-  _trait("JdbcAccess") {}
-
-  _class("DB") {
-"name" ~ (string, required, indexed, unique)
-"owner" ~ (string)
-"createTime" ~ (int)
-  }
-
-  _class("StorageDesc") {
-"inputFormat" ~ (string, required)
-"outputFormat" ~ (string, required)
-  }
-
-  _class("Column") {
-"name" ~ (string, required)
-"dataType" ~ (string, required)
-"sd" ~ ("StorageDesc", required)
-  }
-
-  _class("Table", List()) {
-"name" ~ (string,  required,  indexed)
-"db" ~ ("DB", required)
-"sd" ~ ("StorageDesc", required)
-  }
-
-  _class("LoadProcess") {
-"name" ~ (string, required)
-"inputTables" ~ (array("Table"), collection)
-"outputTable" ~ ("Table", required)
-
-  }
-
-  _class("View") {
-"name" ~ (string, required)
-"inputTables" ~ (array("Table"), collection)
-  }
-
-  _class("AT") {
-"name" ~ (string, required)
-"stringSet" ~ (array("string"), multiplicty(0, Int.MaxValue, true))
-  }
-}
-
-TypeSystem.getInstance().defineTypes(tDef)
-  }
-
-  @Test
-  def test1 {
-
-val b = new InstanceBuilder
-import b._
-
-val instances = b create {
-  val a = instance("AT") {  // use instance to create Referenceables. use 
closure to
-// set attributes of instance
-'name ~ "A1"  // use '~' to set attributes. Use a 
Symbol (names starting with ') for
-'stringSet ~ Seq("a", "a")
-  }
-}
-
-val ts = TypeSystem.getInstance()
-import scala.collection.JavaConversions._
-val typedInstances = instances.map { i =>
-  val iTyp = ts.getDataType(classOf[ClassType], i.getTypeName)
-  iTyp.convert(i, Multiplicity.REQUIRED)
-}
-
-typedInstances.foreach { i =>
-  println(i)
-}
-  }
-
-  @Test(expectedExceptions =  Array(classOf[AtlasException]) , 
expectedExceptionsMessageRegExp = "A multiplicty of more than one requires a 
collection type for attribute 'stringSet'")
-  def WrongMultiplicity {
-val b = new TypesBuilder
-import b._
-val tDef = types {
-  _class("Wrong") {
-"name" ~ (string, required)
-"stringSet" ~ (string, multiplicty(0, Int.MaxValue, true))
-  }
-}
-TypeSystem.getInstance().defineTypes(tDef)
-  }
-}

http://git-wip-us.apache.org/repos/asf/atlas/blob/435fe3fb/typesystem/src/test/scala/org/apache/atlas/typesystem/builders/TypesBuilderTest.scala
--
diff --git 
a/typesystem/src/test/scala/org/apache/atlas/typesystem/builders/TypesBuilderTest.scala
 
b/typesystem/src/test/scala/org/apache/atlas/typesystem/builders/TypesBuilderTest.scala
deleted file mode 100644
index d01adb4..000
--- 
a/typesystem/src/test/scala/org/apache/atlas/typesystem/builders/TypesBuilderTest.scala
+++ /dev/null
@@ -1,33 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed 

[21/42] atlas git commit: ATLAS-2251: Remove TypeSystem and related implementation, to avoid unncessary duplicate of type details in cache

2017-11-12 Thread madhan
http://git-wip-us.apache.org/repos/asf/atlas/blob/435fe3fb/repository/src/test/java/org/apache/atlas/query/QueryProcessorTest.java
--
diff --git 
a/repository/src/test/java/org/apache/atlas/query/QueryProcessorTest.java 
b/repository/src/test/java/org/apache/atlas/query/QueryProcessorTest.java
deleted file mode 100644
index 57bc74f..000
--- a/repository/src/test/java/org/apache/atlas/query/QueryProcessorTest.java
+++ /dev/null
@@ -1,157 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.atlas.query;
-
-import com.google.common.collect.ImmutableSet;
-import org.apache.atlas.AtlasException;
-import org.apache.atlas.typesystem.types.ClassType;
-import org.apache.atlas.typesystem.types.DataTypes;
-import org.apache.atlas.typesystem.types.DataTypes.TypeCategory;
-import org.apache.atlas.typesystem.types.HierarchicalTypeDefinition;
-import org.apache.atlas.typesystem.types.IDataType;
-import org.apache.atlas.typesystem.types.TypeSystem;
-import org.apache.atlas.typesystem.types.cache.DefaultTypeCache;
-import org.testng.annotations.Test;
-import scala.util.Either;
-import scala.util.parsing.combinator.Parsers;
-
-import java.util.HashSet;
-import java.util.Set;
-
-import static 
org.apache.atlas.typesystem.types.utils.TypesUtil.createClassTypeDef;
-import static 
org.apache.atlas.typesystem.types.utils.TypesUtil.createRequiredAttrDef;
-import static org.testng.Assert.assertFalse;
-import static org.testng.Assert.assertTrue;
-
-/**
- * Tests the logic for skipping type cache lookup for things that
- * cannot be types.
- *
- */
-public class QueryProcessorTest {
-
-
-@Test
-public void testAliasesNotTreatedAsTypes() throws Exception {
-
-ValidatingTypeCache tc = findTypeLookupsDuringQueryParsing("hive_db as 
inst where inst.name=\"Reporting\" select inst as id, inst.name");
-assertTrue(tc.wasTypeRequested("hive_db"));
-assertFalse(tc.wasTypeRequested("inst"));
-assertFalse(tc.wasTypeRequested("name"));
-
-}
-
-
-@Test
-public void testFieldInComparisionNotTreatedAsType() throws Exception {
-
-//test when the IdExpression is on the left, on the right, and on both 
sides of the ComparsionExpression
-ValidatingTypeCache tc = findTypeLookupsDuringQueryParsing("hive_db 
where name=\"Reporting\" or \"Reporting\" = name or name=name");
-assertTrue(tc.wasTypeRequested("hive_db"));
-assertFalse(tc.wasTypeRequested("name"));
-
-}
-
-
-@Test
-public void testFieldInArithmeticExprNotTreatedAsType() throws Exception {
-
-//test when the IdExpression is on the left, on the right, and on both 
sides of the ArithmeticExpression
-ValidatingTypeCache tc = findTypeLookupsDuringQueryParsing("hive_db 
where (tableCount + 3) > (tableCount + tableCount) select (3 + tableCount) as 
updatedCount");
-
-assertTrue(tc.wasTypeRequested("hive_db"));
-assertFalse(tc.wasTypeRequested("tableCount"));
-assertFalse(tc.wasTypeRequested("updatedCount"));
-
-}
-
-@Test
-public void testFieldInSelectListWithAlasNotTreatedAsType() throws 
Exception {
-
-ValidatingTypeCache tc = findTypeLookupsDuringQueryParsing("hive_db 
select name as theName");
-assertTrue(tc.wasTypeRequested("hive_db"));
-assertFalse(tc.wasTypeRequested("theName"));
-assertFalse(tc.wasTypeRequested("name"));
-
-}
-
-@Test
-public void testFieldInSelectListNotTreatedAsType() throws Exception {
-
-
-ValidatingTypeCache tc = findTypeLookupsDuringQueryParsing("hive_db 
select name");
-assertTrue(tc.wasTypeRequested("hive_db"));
-assertFalse(tc.wasTypeRequested("name"));
-
-}
-
-private ValidatingTypeCache findTypeLookupsDuringQueryParsing(String 
query) throws AtlasException {
-TypeSystem typeSystem = TypeSystem.getInstance();
-ValidatingTypeCache result = new ValidatingTypeCache();
-typeSystem.setTypeCache(result);
-typeSystem.reset();
-HierarchicalTypeDefinition hiveTypeDef = 
createClassTypeDef("hive_db", "", ImmutableSet.of(),
-

[06/42] atlas git commit: ATLAS-2251: Remove TypeSystem and related implementation, to avoid unncessary duplicate of type details in cache

2017-11-12 Thread madhan
http://git-wip-us.apache.org/repos/asf/atlas/blob/435fe3fb/typesystem/src/test/java/org/apache/atlas/typesystem/types/TypeInheritanceTest.java
--
diff --git 
a/typesystem/src/test/java/org/apache/atlas/typesystem/types/TypeInheritanceTest.java
 
b/typesystem/src/test/java/org/apache/atlas/typesystem/types/TypeInheritanceTest.java
deleted file mode 100644
index c13ef3a..000
--- 
a/typesystem/src/test/java/org/apache/atlas/typesystem/types/TypeInheritanceTest.java
+++ /dev/null
@@ -1,252 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.atlas.typesystem.types;
-
-import com.google.common.collect.ImmutableSet;
-
-import org.apache.atlas.AtlasException;
-import org.apache.atlas.typesystem.IStruct;
-import org.apache.atlas.typesystem.ITypedInstance;
-import org.apache.atlas.typesystem.ITypedStruct;
-import org.apache.atlas.typesystem.Struct;
-import org.testng.Assert;
-import org.testng.annotations.BeforeMethod;
-import org.testng.annotations.Test;
-
-import static 
org.apache.atlas.typesystem.types.utils.TypesUtil.createClassTypeDef;
-import static 
org.apache.atlas.typesystem.types.utils.TypesUtil.createOptionalAttrDef;
-import static 
org.apache.atlas.typesystem.types.utils.TypesUtil.createRequiredAttrDef;
-import static 
org.apache.atlas.typesystem.types.utils.TypesUtil.createTraitTypeDef;
-
-/**
- * Unit tests for type inheritance.
- */
-public class TypeInheritanceTest extends BaseTest {
-
-@BeforeMethod
-public void setup() throws Exception {
-TypeSystem.getInstance().reset();
-super.setup();
-}
-
-/*
- * Type Hierarchy is:
- *   A(a)
- *   B(b) extends A
- */
-@Test
-public void testSimpleInheritance() throws AtlasException {
-HierarchicalTypeDefinition A = createClassTypeDef("A", null, 
createRequiredAttrDef("a", DataTypes.INT_TYPE));
-
-HierarchicalTypeDefinition B =
-createClassTypeDef("B", ImmutableSet.of("A"), 
createOptionalAttrDef("b", DataTypes.BOOLEAN_TYPE));
-
-defineClasses(A, B);
-
-ClassType BType = getTypeSystem().getDataType(ClassType.class, "B");
-
-Struct s1 = new Struct("B");
-s1.set("b", true);
-s1.set("a", 1);
-
-ITypedInstance ts = BType.convert(s1, Multiplicity.REQUIRED);
-Assert.assertEquals(ts.toString(), "{\n" +
-"\tid : (type: B, id: )\n" +
-"\tb : \ttrue\n" +
-"\ta : \t1\n" +
-"}");
-}
-
-/*
- * Type Hierarchy is:
- *   A(a, b)
- *   B(b) extends A
- */
-@Test
-public void testSimpleInheritanceWithOverrides() throws AtlasException {
-HierarchicalTypeDefinition A = createClassTypeDef("A", null, 
createRequiredAttrDef("a", DataTypes.INT_TYPE),
-createRequiredAttrDef("b", DataTypes.BOOLEAN_TYPE));
-
-HierarchicalTypeDefinition B =
-createClassTypeDef("B", ImmutableSet.of("A"), 
createOptionalAttrDef("b", DataTypes.BOOLEAN_TYPE));
-
-defineClasses(A, B);
-
-ClassType BType = getTypeSystem().getDataType(ClassType.class, "B");
-
-Struct s1 = new Struct("B");
-s1.set("b", true);
-s1.set("a", 1);
-s1.set("A.B.b", false);
-
-ITypedInstance ts = BType.convert(s1, Multiplicity.REQUIRED);
-Assert.assertEquals(ts.toString(), "{\n" +
-"\tid : (type: B, id: )\n" +
-"\tb : \ttrue\n" +
-"\ta : \t1\n" +
-"\tA.B.b : \tfalse\n" +
-"}");
-}
-
-/*
- * Type Hierarchy is:
- *   A(a)
- *   B(b) extends A
- *   C(c) extends B
- *   D(d) extends C
- */
-@Test
-public void testMultiLevelInheritance() throws AtlasException {
-HierarchicalTypeDefinition A = createClassTypeDef("A", null, 
createRequiredAttrDef("a", DataTypes.INT_TYPE));
-
-HierarchicalTypeDefinition B =
-createClassTypeDef("B", ImmutableSet.of("A"), 
createOptionalAttrDef("b", DataTypes.BOOLEAN_TYPE));
-
-HierarchicalTypeDefinition C =
-createClassTypeDef("C", 

[14/42] atlas git commit: ATLAS-2251: Remove TypeSystem and related implementation, to avoid unncessary duplicate of type details in cache

2017-11-12 Thread madhan
http://git-wip-us.apache.org/repos/asf/atlas/blob/435fe3fb/repository/src/test/scala/org/apache/atlas/query/ExpressionTest.scala
--
diff --git 
a/repository/src/test/scala/org/apache/atlas/query/ExpressionTest.scala 
b/repository/src/test/scala/org/apache/atlas/query/ExpressionTest.scala
deleted file mode 100755
index 918f327..000
--- a/repository/src/test/scala/org/apache/atlas/query/ExpressionTest.scala
+++ /dev/null
@@ -1,172 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.atlas.query
-
-import org.apache.atlas.DBSandboxer
-import org.apache.atlas.query.Expressions._
-import org.apache.atlas.repository.BaseTest
-import org.testng.annotations.{BeforeMethod, Listeners, Test}
-
-class ExpressionTest extends BaseTest {
-
-@BeforeMethod
-override def setup {
-super.setup
-
-QueryTestsUtils.setupTypes
-
-}
-
-@Test def testClass: Unit = {
-val e = QueryProcessor.validate(_class("DB"))
-println(e)
-}
-
-@Test def testFilter: Unit = {
-val e = 
QueryProcessor.validate(_class("DB").where(id("name").`=`(string("Reporting"
-println(e)
-}
-
-@Test def testSelect: Unit = {
-val e = 
QueryProcessor.validate(_class("DB").where(id("name").`=`(string("Reporting"))).
-select(id("name"), id("owner")))
-println(e)
-}
-
-@Test def testNegTypeTest: Unit = {
-try {
-val e = QueryProcessor.validate(_class("DB").where(id("name")))
-println(e)
-} catch {
-case e: ExpressionException if e.getMessage.endsWith("expression: 
DB where name") => ()
-}
-}
-
-@Test def testIsTrait: Unit = {
-val e = 
QueryProcessor.validate(_class("DB").where(isTrait("JdbcAccess")))
-println(e)
-}
-
-@Test def testIsTraitNegative: Unit = {
-try {
-val e = QueryProcessor.validate(_class("DB").where(isTrait("Jdb")))
-println(e)
-} catch {
-case e: ExpressionException if e.getMessage.endsWith("not a 
TraitType, expression:  is Jdb") => ()
-}
-}
-
-@Test def testhasField: Unit = {
-val e = QueryProcessor.validate(_class("DB").where(hasField("name")))
-println(e)
-}
-
-@Test def testHasFieldNegative: Unit = {
-try {
-val e = 
QueryProcessor.validate(_class("DB").where(hasField("nam")))
-println(e)
-} catch {
-case e: ExpressionException if e.getMessage.endsWith("not a 
TraitType, expression:  is Jdb") => ()
-}
-}
-
-@Test def testFieldReference: Unit = {
-val e = QueryProcessor.validate(_class("DB").field("Table"))
-println(e)
-}
-
-@Test def testNegFieldReference: Unit = {
-try {
-val e = 
QueryProcessor.validate(_class("DB").where(_class("LoadProcess").hasField("name")))
-println(e)
-} catch {
-case e: ExpressionException
-if e.getMessage.endsWith("srcType of field doesn't match input 
type, expression: LoadProcess has name") => ()
-}
-}
-
-@Test def testFieldReferenceRedundant: Unit = {
-val e = 
QueryProcessor.validate(_class("DB").where(_class("DB").hasField("name")))
-println(e)
-}
-
-@Test def testBackReference: Unit = {
-val e = QueryProcessor.validate(
-
_class("DB").as("db1").field("Table").where(id("db1").field("name").`=`(string("Reporting"
-println(e)
-}
-
-@Test def testArith: Unit = {
-val e = 
QueryProcessor.validate(_class("DB").where(id("name").`=`(string("Reporting"))).
-select(id("name"), id("createTime") + int(1)))
-println(e)
-}
-
-@Test def testComparisonLogical: Unit = {
-val e = 
QueryProcessor.validate(_class("DB").where(id("name").`=`(string("Reporting")).
-and(id("createTime") + int(1) > int(0
-println(e)
-}
-
-@Test def testJoinAndSelect1: Unit = {
-val e = QueryProcessor.validate(
-

[03/42] atlas git commit: ATLAS-2251: Remove TypeSystem and related implementation, to avoid unncessary duplicate of type details in cache

2017-11-12 Thread madhan
http://git-wip-us.apache.org/repos/asf/atlas/blob/435fe3fb/webapp/src/main/java/org/apache/atlas/web/resources/MetadataDiscoveryResource.java
--
diff --git 
a/webapp/src/main/java/org/apache/atlas/web/resources/MetadataDiscoveryResource.java
 
b/webapp/src/main/java/org/apache/atlas/web/resources/MetadataDiscoveryResource.java
index d7c66d3..ff1751d 100755
--- 
a/webapp/src/main/java/org/apache/atlas/web/resources/MetadataDiscoveryResource.java
+++ 
b/webapp/src/main/java/org/apache/atlas/web/resources/MetadataDiscoveryResource.java
@@ -22,8 +22,6 @@ import com.google.common.base.Preconditions;
 import org.apache.atlas.AtlasClient;
 import org.apache.atlas.AtlasConfiguration;
 import org.apache.atlas.classification.InterfaceAudience;
-import org.apache.atlas.discovery.DiscoveryException;
-import org.apache.atlas.discovery.DiscoveryService;
 import org.apache.atlas.query.QueryParams;
 import org.apache.atlas.utils.AtlasPerfTracer;
 import org.apache.atlas.utils.ParamChecker;
@@ -46,6 +44,7 @@ import javax.ws.rs.Produces;
 import javax.ws.rs.QueryParam;
 import javax.ws.rs.WebApplicationException;
 import javax.ws.rs.core.Response;
+import java.util.ArrayList;
 import java.util.List;
 import java.util.Map;
 
@@ -65,8 +64,6 @@ public class MetadataDiscoveryResource {
 private static final String QUERY_TYPE_FULLTEXT = "full-text";
 private static final String LIMIT_OFFSET_DEFAULT = "-1";
 
-private final DiscoveryService discoveryService;
-
 private final  boolean   gremlinSearchEnabled;
 private static Configuration applicationProperties  = null;
 private static final String  ENABLE_GREMLIN_SEARCH_PROPERTY = 
"atlas.search.gremlin.enable";
@@ -75,11 +72,10 @@ public class MetadataDiscoveryResource {
  * Created by the Guice ServletModule and injected with the
  * configured DiscoveryService.
  *
- * @param discoveryService metadata service handle
+ * @param configuration configuration
  */
 @Inject
-public MetadataDiscoveryResource(DiscoveryService discoveryService, 
Configuration configuration) {
-this.discoveryService  = discoveryService;
+public MetadataDiscoveryResource(Configuration configuration) {
 applicationProperties  = configuration;
 gremlinSearchEnabled   = applicationProperties != null && 
applicationProperties.getBoolean(ENABLE_GREMLIN_SEARCH_PROPERTY, false);
 }
@@ -152,12 +148,12 @@ public class MetadataDiscoveryResource {
 
 dslQuery = ParamChecker.notEmpty(dslQuery, "dslQuery cannot be 
null");
 QueryParams queryParams = validateQueryParams(limit, offset);
-final String jsonResultStr = 
discoveryService.searchByDSL(dslQuery, queryParams);
+final String jsonResultStr = ""; // TODO-typeSystem-removal: 
discoveryService.searchByDSL(dslQuery, queryParams);
 
 JSONObject response = new 
DSLJSONResponseBuilder().results(jsonResultStr).query(dslQuery).build();
 
 return Response.ok(response).build();
-} catch (DiscoveryException | IllegalArgumentException e) {
+} catch (IllegalArgumentException e) {
 LOG.error("Unable to get entity list for dslQuery {}", dslQuery, 
e);
 throw new WebApplicationException(Servlets.getErrorResponse(e, 
Response.Status.BAD_REQUEST));
 } catch (WebApplicationException e) {
@@ -220,11 +216,11 @@ public class MetadataDiscoveryResource {
 }
 
 if (!gremlinSearchEnabled) {
-throw new DiscoveryException("Gremlin search is not enabled.");
+throw new Exception("Gremlin search is not enabled.");
 }
 
 gremlinQuery = ParamChecker.notEmpty(gremlinQuery, "gremlinQuery 
cannot be null or empty");
-final List> results = 
discoveryService.searchByGremlin(gremlinQuery);
+final List> results = new ArrayList<>(); // 
TODO-typeSystem-removal: discoveryService.searchByGremlin(gremlinQuery);
 
 JSONObject response = new JSONObject();
 response.put(AtlasClient.REQUEST_ID, Servlets.getRequestId());
@@ -239,7 +235,7 @@ public class MetadataDiscoveryResource {
 response.put(AtlasClient.COUNT, list.length());
 
 return Response.ok(response).build();
-} catch (DiscoveryException | IllegalArgumentException e) {
+} catch (IllegalArgumentException e) {
 LOG.error("Unable to get entity list for gremlinQuery {}", 
gremlinQuery, e);
 throw new WebApplicationException(Servlets.getErrorResponse(e, 
Response.Status.BAD_REQUEST));
 } catch (WebApplicationException e) {
@@ -284,12 +280,12 @@ public class MetadataDiscoveryResource {
 
 query = ParamChecker.notEmpty(query, "query cannot be null or 
empty");
 QueryParams queryParams = validateQueryParams(limit, 

[13/42] atlas git commit: ATLAS-2251: Remove TypeSystem and related implementation, to avoid unncessary duplicate of type details in cache

2017-11-12 Thread madhan
http://git-wip-us.apache.org/repos/asf/atlas/blob/435fe3fb/repository/src/test/scala/org/apache/atlas/query/LineageQueryTest.scala
--
diff --git 
a/repository/src/test/scala/org/apache/atlas/query/LineageQueryTest.scala 
b/repository/src/test/scala/org/apache/atlas/query/LineageQueryTest.scala
deleted file mode 100755
index b01cbf8..000
--- a/repository/src/test/scala/org/apache/atlas/query/LineageQueryTest.scala
+++ /dev/null
@@ -1,541 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.atlas.query
-
-import org.apache.atlas.{DBSandboxer, TestUtils}
-import org.apache.atlas.discovery.graph.DefaultGraphPersistenceStrategy
-import org.apache.atlas.query.Expressions._class
-import org.apache.atlas.query.Expressions.id
-import org.apache.atlas.query.Expressions.int
-import org.apache.atlas.repository.graph.AtlasGraphProvider
-import org.apache.atlas.repository.graph.GraphBackedMetadataRepository
-import org.apache.atlas.repository.graphdb.AtlasGraph
-import org.apache.atlas.typesystem.types.TypeSystem
-import org.testng.annotations._
-
-class LineageQueryTest extends BaseGremlinTest {
-
-var g: AtlasGraph[_,_] = null
-var gp:GraphPersistenceStrategies = null;
-
-@BeforeMethod
-def resetRequestContext() {
-TestUtils.resetRequestContext()
-}
-
-
-@BeforeClass
-def beforeAll() {
-TypeSystem.getInstance().reset()
-var repo = new GraphBackedMetadataRepository(null, new 
AtlasGraphProvider().get());
-TestUtils.setupGraphProvider(repo);
-//force graph to be initialized first
-AtlasGraphProvider.getGraphInstance();
-  
-//create types and indices up front.  Without this, some of the 
property keys (particularly __traitNames and __superTypes)
-//get ended up created implicitly with some graph backends with the 
wrong multiplicity.  This also makes the queries  
-//we execute perform better :-)
-   QueryTestsUtils.setupTypesAndIndices()
-
-   gp = new DefaultGraphPersistenceStrategy(repo);
-   g = QueryTestsUtils.setupTestGraph(repo)
-}
-
-@AfterClass
-def afterAll() {
-AtlasGraphProvider.cleanup()
-}
-
-val PREFIX_SPACES_REGEX = ("\\n\\s*").r
-
-  @Test def testInputTables {
-val r = 
QueryProcessor.evaluate(_class("LoadProcess").field("inputTables"), g, gp)
-val x = r.toJson
-validateJson(r,"""{
- |  "query":"LoadProcess inputTables",
- |  "dataType":{
- |"superTypes":[
- |
- |],
- |
"hierarchicalMetaTypeName":"org.apache.atlas.typesystem.types.ClassType",
- |"typeName":"Table",
- |"attributeDefinitions":[
- |  {
- |"name":"name",
- |"dataTypeName":"string",
- |"multiplicity":{
- |  "lower":0,
- |  "upper":1,
- |  "isUnique":false
- |},
- |"isComposite":false,
- |"isUnique":false,
- |"isIndexable":false,
- |"reverseAttributeName":null
- |  },
- |  {
- |"name":"db",
- |"dataTypeName":"DB",
- |"multiplicity":{
- |  "lower":1,
- |  "upper":1,
- |  "isUnique":false
- |},
- |"isComposite":false,
- |"isUnique":false,
- |"isIndexable":false,
- |"reverseAttributeName":null
- |  },
-  

[01/42] atlas git commit: ATLAS-2251: Remove TypeSystem and related implementation, to avoid unncessary duplicate of type details in cache

2017-11-12 Thread madhan
Repository: atlas
Updated Branches:
  refs/heads/master 0abf84caa -> 435fe3fba


http://git-wip-us.apache.org/repos/asf/atlas/blob/435fe3fb/webapp/src/test/java/org/apache/atlas/web/integration/MetadataDiscoveryJerseyResourceIT.java
--
diff --git 
a/webapp/src/test/java/org/apache/atlas/web/integration/MetadataDiscoveryJerseyResourceIT.java
 
b/webapp/src/test/java/org/apache/atlas/web/integration/MetadataDiscoveryJerseyResourceIT.java
index a13f069..f25fc5f 100755
--- 
a/webapp/src/test/java/org/apache/atlas/web/integration/MetadataDiscoveryJerseyResourceIT.java
+++ 
b/webapp/src/test/java/org/apache/atlas/web/integration/MetadataDiscoveryJerseyResourceIT.java
@@ -18,23 +18,16 @@
 
 package org.apache.atlas.web.integration;
 
-import com.google.common.collect.ImmutableList;
-import com.google.common.collect.ImmutableSet;
 import com.sun.jersey.api.client.ClientResponse;
 import com.sun.jersey.core.util.MultivaluedMapImpl;
 import org.apache.atlas.AtlasClient;
 import org.apache.atlas.AtlasServiceException;
-import org.apache.atlas.typesystem.Referenceable;
-import org.apache.atlas.typesystem.Struct;
-import org.apache.atlas.typesystem.TypesDef;
-import org.apache.atlas.typesystem.persistence.Id;
-import org.apache.atlas.typesystem.types.ClassType;
-import org.apache.atlas.typesystem.types.DataTypes;
-import org.apache.atlas.typesystem.types.EnumTypeDefinition;
-import org.apache.atlas.typesystem.types.HierarchicalTypeDefinition;
-import org.apache.atlas.typesystem.types.StructTypeDefinition;
-import org.apache.atlas.typesystem.types.TraitType;
-import org.apache.atlas.typesystem.types.utils.TypesUtil;
+import org.apache.atlas.model.typedef.AtlasBaseTypeDef;
+import org.apache.atlas.v1.model.instance.Id;
+import org.apache.atlas.v1.model.instance.Referenceable;
+import org.apache.atlas.v1.model.instance.Struct;
+import org.apache.atlas.v1.model.typedef.*;
+import org.apache.atlas.v1.typesystem.types.utils.TypesUtil;
 import org.codehaus.jettison.json.JSONArray;
 import org.codehaus.jettison.json.JSONObject;
 import org.testng.Assert;
@@ -42,6 +35,7 @@ import org.testng.annotations.BeforeClass;
 import org.testng.annotations.Test;
 
 import javax.ws.rs.core.MultivaluedMap;
+import java.util.Collections;
 import java.util.List;
 
 import static org.testng.Assert.assertEquals;
@@ -236,16 +230,16 @@ public class MetadataDiscoveryJerseyResourceIT extends 
BaseResourceIT {
 private void createTypes() throws Exception {
 createTypeDefinitionsV1();
 
-HierarchicalTypeDefinition dslTestTypeDefinition = TypesUtil
-.createClassTypeDef("dsl_test_type", ImmutableSet.of(),
-TypesUtil.createUniqueRequiredAttrDef("name", 
DataTypes.STRING_TYPE),
-TypesUtil.createRequiredAttrDef("description", 
DataTypes.STRING_TYPE));
+ClassTypeDefinition dslTestTypeDefinition = TypesUtil
+.createClassTypeDef("dsl_test_type", null, 
Collections.emptySet(),
+TypesUtil.createUniqueRequiredAttrDef("name", 
AtlasBaseTypeDef.ATLAS_TYPE_STRING),
+TypesUtil.createRequiredAttrDef("description", 
AtlasBaseTypeDef.ATLAS_TYPE_STRING));
 
-HierarchicalTypeDefinition classificationTraitDefinition = 
TypesUtil
-.createTraitTypeDef("Classification", 
ImmutableSet.of(),
-TypesUtil.createRequiredAttrDef("tag", 
DataTypes.STRING_TYPE));
-TypesDef typesDef = 
TypesUtil.getTypesDef(ImmutableList.of(), 
ImmutableList.of(),
-ImmutableList.of(classificationTraitDefinition), 
ImmutableList.of(dslTestTypeDefinition));
+TraitTypeDefinition classificationTraitDefinition = TypesUtil
+.createTraitTypeDef("Classification", null, 
Collections.emptySet(),
+TypesUtil.createRequiredAttrDef("tag", 
AtlasBaseTypeDef.ATLAS_TYPE_STRING));
+TypesDef typesDef = new 
TypesDef(Collections.emptyList(), 
Collections.emptyList(),
+Collections.singletonList(classificationTraitDefinition), 
Collections.singletonList(dslTestTypeDefinition));
 createType(typesDef);
 }
 
@@ -259,7 +253,7 @@ public class MetadataDiscoveryJerseyResourceIT extends 
BaseResourceIT {
 tagName = randomString();
 traitInstance.set("tag", tagName);
 
-List traits = entityInstance.getTraits();
+List traits = entityInstance.getTraitNames();
 assertEquals(traits.size(), 1);
 
 return createInstance(entityInstance);

http://git-wip-us.apache.org/repos/asf/atlas/blob/435fe3fb/webapp/src/test/java/org/apache/atlas/web/integration/TypedefsJerseyResourceIT.java
--
diff --git 
a/webapp/src/test/java/org/apache/atlas/web/integration/TypedefsJerseyResourceIT.java
 

[09/42] atlas git commit: ATLAS-2251: Remove TypeSystem and related implementation, to avoid unncessary duplicate of type details in cache

2017-11-12 Thread madhan
http://git-wip-us.apache.org/repos/asf/atlas/blob/435fe3fb/typesystem/src/main/java/org/apache/atlas/typesystem/types/ObjectGraphWalker.java
--
diff --git 
a/typesystem/src/main/java/org/apache/atlas/typesystem/types/ObjectGraphWalker.java
 
b/typesystem/src/main/java/org/apache/atlas/typesystem/types/ObjectGraphWalker.java
deleted file mode 100755
index 036d18d..000
--- 
a/typesystem/src/main/java/org/apache/atlas/typesystem/types/ObjectGraphWalker.java
+++ /dev/null
@@ -1,226 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.atlas.typesystem.types;
-
-import com.google.common.collect.ImmutableList;
-import com.google.common.collect.ImmutableMap;
-import org.apache.atlas.AtlasException;
-import org.apache.atlas.typesystem.IReferenceableInstance;
-import org.apache.atlas.typesystem.IStruct;
-import org.apache.atlas.typesystem.persistence.Id;
-
-import java.util.Collection;
-import java.util.HashSet;
-import java.util.Iterator;
-import java.util.LinkedList;
-import java.util.List;
-import java.util.Map;
-import java.util.Queue;
-import java.util.Set;
-
-/**
- * Given a IReferenceableInstance, a Walker will traverse the Object Graph
- * reachable form the instance. It will invoke the process call on the 
provided NodeProcessor
- * for each non-primitive attribute (Structs, Traits, References, Arrays of 
Non-Primitives, Maps
- * of Non-Primitives)
- */
-public class ObjectGraphWalker {
-
-final Queue queue;
-final TypeSystem typeSystem;
-final NodeProcessor nodeProcessor;
-Set processedIds;
-
-public ObjectGraphWalker(TypeSystem typeSystem, NodeProcessor 
nodeProcessor) throws AtlasException {
-this(typeSystem, nodeProcessor, (IReferenceableInstance) null);
-}
-
-public ObjectGraphWalker(TypeSystem typeSystem, NodeProcessor 
nodeProcessor, IReferenceableInstance start)
-throws AtlasException {
-this.typeSystem = typeSystem;
-this.nodeProcessor = nodeProcessor;
-queue = new LinkedList<>();
-processedIds = new HashSet<>();
-if (start != null) {
-visitReferenceableInstance(start);
-}
-}
-
-public ObjectGraphWalker(TypeSystem typeSystem, NodeProcessor 
nodeProcessor,
-List roots) throws 
AtlasException {
-this.typeSystem = typeSystem;
-this.nodeProcessor = nodeProcessor;
-queue = new LinkedList<>();
-processedIds = new HashSet<>();
-for (IReferenceableInstance r : roots) {
-visitReferenceableInstance(r);
-}
-}
-
-public void walk() throws AtlasException {
-while (!queue.isEmpty()) {
-IReferenceableInstance r = queue.poll();
-if(r != null) {
-processReferenceableInstance(r);
-}
-}
-}
-
-public void addRoot(IReferenceableInstance root) {
-visitReferenceableInstance(root);
-}
-
-void traverseValue(IDataType dT, Object val) throws AtlasException {
-if (val != null) {
-if (dT.getTypeCategory() == DataTypes.TypeCategory.ARRAY) {
-IDataType elemType = ((DataTypes.ArrayType) dT).getElemType();
-visitCollection(elemType, val);
-} else if (dT.getTypeCategory() == DataTypes.TypeCategory.MAP) {
-IDataType keyType = ((DataTypes.MapType) dT).getKeyType();
-IDataType valueType = ((DataTypes.MapType) dT).getValueType();
-visitMap(keyType, valueType, val);
-} else if (dT.getTypeCategory() == DataTypes.TypeCategory.STRUCT
-|| dT.getTypeCategory() == DataTypes.TypeCategory.TRAIT) {
-visitStruct(val);
-} else if (dT.getTypeCategory() == DataTypes.TypeCategory.CLASS) {
-visitReferenceableInstance(val);
-}
-}
-}
-
-void visitMap(IDataType keyType, IDataType valueType, Object val) throws 
AtlasException {
-if (keyType.getTypeCategory() == DataTypes.TypeCategory.PRIMITIVE
-&& valueType.getTypeCategory() == 
DataTypes.TypeCategory.PRIMITIVE) {
-return;
-   

[23/42] atlas git commit: ATLAS-2251: Remove TypeSystem and related implementation, to avoid unncessary duplicate of type details in cache

2017-11-12 Thread madhan
http://git-wip-us.apache.org/repos/asf/atlas/blob/435fe3fb/repository/src/test/java/org/apache/atlas/discovery/DataSetLineageServiceTest.java
--
diff --git 
a/repository/src/test/java/org/apache/atlas/discovery/DataSetLineageServiceTest.java
 
b/repository/src/test/java/org/apache/atlas/discovery/DataSetLineageServiceTest.java
deleted file mode 100644
index 3db58fe..000
--- 
a/repository/src/test/java/org/apache/atlas/discovery/DataSetLineageServiceTest.java
+++ /dev/null
@@ -1,497 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.atlas.discovery;
-
-import com.google.common.collect.ImmutableList;
-import org.apache.atlas.AtlasException;
-import org.apache.atlas.BaseRepositoryTest;
-import org.apache.atlas.TestModules;
-import org.apache.atlas.TestUtils;
-import org.apache.atlas.model.legacy.EntityResult;
-import org.apache.atlas.query.QueryParams;
-import org.apache.atlas.typesystem.ITypedReferenceableInstance;
-import org.apache.atlas.typesystem.Referenceable;
-import org.apache.atlas.typesystem.Struct;
-import org.apache.atlas.typesystem.exception.EntityNotFoundException;
-import org.apache.atlas.typesystem.json.InstanceSerialization;
-import org.apache.atlas.typesystem.persistence.Id;
-import org.apache.commons.collections.ArrayStack;
-import org.codehaus.jettison.json.JSONArray;
-import org.codehaus.jettison.json.JSONException;
-import org.codehaus.jettison.json.JSONObject;
-import org.testng.Assert;
-import org.testng.annotations.AfterClass;
-import org.testng.annotations.BeforeClass;
-import org.testng.annotations.DataProvider;
-import org.testng.annotations.Guice;
-import org.testng.annotations.Test;
-
-import javax.inject.Inject;
-import java.util.Arrays;
-import java.util.List;
-import java.util.Map;
-
-import static org.testng.Assert.assertEquals;
-import static org.testng.Assert.assertNotNull;
-import static org.testng.Assert.assertTrue;
-import static org.testng.Assert.fail;
-
-/**
- * Unit tests for Hive LineageService.
- */
-@Guice(modules = TestModules.TestOnlyModule.class)
-public class DataSetLineageServiceTest extends BaseRepositoryTest {
-
-@Inject
-private DiscoveryService discoveryService;
-
-@Inject
-private DataSetLineageService lineageService;
-
-@BeforeClass
-public void setUp() throws Exception {
-super.setUp();
-}
-
-@AfterClass
-public void tearDown() throws Exception {
-super.tearDown();
-}
-
-@DataProvider(name = "dslQueriesProvider")
-private Object[][] createDSLQueries() {
-return new String[][]{
-// joins
-{"hive_table where name=\"sales_fact\", columns"},
-{"hive_table where name=\"sales_fact\", columns select name, 
dataType, comment"},
-{"hive_table where name=\"sales_fact\", columns as c select 
c.name, c.dataType, c.comment"},
-//{"hive_db as db where (db.name=\"Reporting\"), 
hive_table as table select db.name,
-// table.name"},
-{"from hive_db"}, {"hive_db"}, {"hive_db where 
hive_db.name=\"Reporting\""},
-{"hive_db hive_db.name = \"Reporting\""},
-{"hive_db where hive_db.name=\"Reporting\" select name, owner"}, 
{"hive_db has name"},
-//{"hive_db, hive_table"},
-//{"hive_db, hive_process has name"},
-//{"hive_db as db1, hive_table where db1.name = 
\"Reporting\""},
-//{"hive_db where hive_db.name=\"Reporting\" and 
hive_db.createTime < " + System
-// .currentTimeMillis()},
-{"from hive_table"}, {"hive_table"}, {"hive_table is Dimension"},
-{"hive_column where hive_column isa PII"},
-//{"hive_column where hive_column isa PII select 
hive_column.name"},
-{"hive_column select hive_column.name"}, {"hive_column select 
name"},
-{"hive_column where hive_column.name=\"customer_id\""}, {"from 
hive_table select hive_table.name"},
-{"hive_db where (name = \"Reporting\")"},
-{"hive_db where (name = \"Reporting\") select name as 

[04/42] atlas git commit: ATLAS-2251: Remove TypeSystem and related implementation, to avoid unncessary duplicate of type details in cache

2017-11-12 Thread madhan
http://git-wip-us.apache.org/repos/asf/atlas/blob/435fe3fb/webapp/src/main/java/org/apache/atlas/notification/NotificationEntityChangeListener.java
--
diff --git 
a/webapp/src/main/java/org/apache/atlas/notification/NotificationEntityChangeListener.java
 
b/webapp/src/main/java/org/apache/atlas/notification/NotificationEntityChangeListener.java
index 53acf56..4633de9 100644
--- 
a/webapp/src/main/java/org/apache/atlas/notification/NotificationEntityChangeListener.java
+++ 
b/webapp/src/main/java/org/apache/atlas/notification/NotificationEntityChangeListener.java
@@ -21,44 +21,34 @@ import com.google.common.annotations.VisibleForTesting;
 import org.apache.atlas.ApplicationProperties;
 import org.apache.atlas.AtlasException;
 import org.apache.atlas.listener.EntityChangeListener;
-import org.apache.atlas.notification.entity.EntityNotification;
-import org.apache.atlas.notification.entity.EntityNotificationImpl;
+import org.apache.atlas.notification.NotificationInterface.NotificationType;
+import org.apache.atlas.v1.model.instance.Referenceable;
+import org.apache.atlas.v1.model.instance.Struct;
+import org.apache.atlas.v1.model.notification.EntityNotificationV1;
+import 
org.apache.atlas.v1.model.notification.EntityNotificationV1.OperationType;
 import org.apache.atlas.repository.graph.GraphHelper;
-import org.apache.atlas.typesystem.IReferenceableInstance;
-import org.apache.atlas.typesystem.IStruct;
-import org.apache.atlas.typesystem.ITypedReferenceableInstance;
-import org.apache.atlas.typesystem.Referenceable;
-import org.apache.atlas.typesystem.Struct;
-import org.apache.atlas.typesystem.types.FieldMapping;
-import org.apache.atlas.typesystem.types.TraitType;
-import org.apache.atlas.typesystem.types.TypeSystem;
+import org.apache.atlas.type.AtlasClassificationType;
+import org.apache.atlas.type.AtlasTypeRegistry;
 import org.apache.commons.collections.CollectionUtils;
 import org.apache.commons.collections.MapUtils;
 import org.apache.commons.configuration.Configuration;
 import org.springframework.stereotype.Component;
 
 import javax.inject.Inject;
-import java.util.Arrays;
-import java.util.Collection;
-import java.util.Collections;
-import java.util.HashMap;
-import java.util.LinkedList;
-import java.util.List;
-import java.util.Map;
-import java.util.Set;
+import java.util.*;
 
 /**
  * Listen to the repository for entity changes and produce entity change 
notifications.
  */
 @Component
 public class NotificationEntityChangeListener implements EntityChangeListener {
+private static final String ATLAS_ENTITY_NOTIFICATION_PROPERTY = 
"atlas.notification.entity";
 
-private final NotificationInterface notificationInterface;
-private final TypeSystem typeSystem;
+private final NotificationInterface notificationInterface;
+private final AtlasTypeRegistry typeRegistry;
+private final Map notificationAttributesCache = new 
HashMap<>();
 
-private Map notificationAttributesCache = new 
HashMap<>();
-private static final String ATLAS_ENTITY_NOTIFICATION_PROPERTY = 
"atlas.notification.entity";
-static Configuration APPLICATION_PROPERTIES = null;
+private static Configuration APPLICATION_PROPERTIES = null;
 
 
 
@@ -68,45 +58,45 @@ public class NotificationEntityChangeListener implements 
EntityChangeListener {
  * Construct a NotificationEntityChangeListener.
  *
  * @param notificationInterface the notification framework interface
- * @param typeSystem the Atlas type system
+ * @param typeRegistry the Atlas type system
  */
 @Inject
-public NotificationEntityChangeListener(NotificationInterface 
notificationInterface, TypeSystem typeSystem) {
+public NotificationEntityChangeListener(NotificationInterface 
notificationInterface, AtlasTypeRegistry typeRegistry) {
 this.notificationInterface = notificationInterface;
-this.typeSystem = typeSystem;
+this.typeRegistry  = typeRegistry;
 }
 
 
 // - EntityChangeListener 
--
 
 @Override
-public void onEntitiesAdded(Collection 
entities, boolean isImport) throws AtlasException {
-notifyOfEntityEvent(entities, 
EntityNotification.OperationType.ENTITY_CREATE);
+public void onEntitiesAdded(Collection entities, boolean 
isImport) throws AtlasException {
+notifyOfEntityEvent(entities, OperationType.ENTITY_CREATE);
 }
 
 @Override
-public void onEntitiesUpdated(Collection 
entities, boolean isImport) throws AtlasException {
-notifyOfEntityEvent(entities, 
EntityNotification.OperationType.ENTITY_UPDATE);
+public void onEntitiesUpdated(Collection entities, boolean 
isImport) throws AtlasException {
+notifyOfEntityEvent(entities, OperationType.ENTITY_UPDATE);
 }
 
 @Override
-public void onTraitsAdded(ITypedReferenceableInstance 

[37/42] atlas git commit: ATLAS-2251: Remove TypeSystem and related implementation, to avoid unncessary duplicate of type details in cache

2017-11-12 Thread madhan
http://git-wip-us.apache.org/repos/asf/atlas/blob/435fe3fb/notification/src/main/java/org/apache/atlas/notification/AtlasNotificationBaseMessage.java
--
diff --git 
a/notification/src/main/java/org/apache/atlas/notification/AtlasNotificationBaseMessage.java
 
b/notification/src/main/java/org/apache/atlas/notification/AtlasNotificationBaseMessage.java
deleted file mode 100644
index 3b377de..000
--- 
a/notification/src/main/java/org/apache/atlas/notification/AtlasNotificationBaseMessage.java
+++ /dev/null
@@ -1,194 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.atlas.notification;
-
-
-import org.apache.atlas.AtlasConfiguration;
-import org.apache.commons.codec.binary.Base64;
-import org.apache.commons.codec.binary.StringUtils;
-import org.apache.commons.compress.utils.IOUtils;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-
-import java.io.ByteArrayInputStream;
-import java.io.ByteArrayOutputStream;
-import java.io.IOException;
-import java.util.zip.GZIPInputStream;
-import java.util.zip.GZIPOutputStream;
-
-
-public class AtlasNotificationBaseMessage {
-private static final Logger LOG = 
LoggerFactory.getLogger(AtlasNotificationBaseMessage.class);
-
-public static final int MESSAGE_MAX_LENGTH_BYTES= 
AtlasConfiguration.NOTIFICATION_MESSAGE_MAX_LENGTH_BYTES.getInt() - 512; // 512 
bytes for envelop;
-public static final boolean MESSAGE_COMPRESSION_ENABLED = 
AtlasConfiguration.NOTIFICATION_MESSAGE_COMPRESSION_ENABLED.getBoolean();
-
-public enum CompressionKind { NONE, GZIP };
-
-private MessageVersion  version= null;
-private String  msgId  = null;
-private CompressionKind msgCompressionKind = CompressionKind.NONE;
-private int msgSplitIdx= 1;
-private int msgSplitCount  = 1;
-
-
-public AtlasNotificationBaseMessage() {
-}
-
-public AtlasNotificationBaseMessage(MessageVersion version) {
-this(version, null, CompressionKind.NONE);
-}
-
-public AtlasNotificationBaseMessage(MessageVersion version, String msgId, 
CompressionKind msgCompressionKind) {
-this.version= version;
-this.msgId  = msgId;
-this.msgCompressionKind = msgCompressionKind;
-}
-
-public AtlasNotificationBaseMessage(MessageVersion version, String msgId, 
CompressionKind msgCompressionKind, int msgSplitIdx, int msgSplitCount) {
-this.version= version;
-this.msgId  = msgId;
-this.msgCompressionKind = msgCompressionKind;
-this.msgSplitIdx= msgSplitIdx;
-this.msgSplitCount  = msgSplitCount;
-}
-
-public void setVersion(MessageVersion version) {
-this.version = version;
-}
-
-public MessageVersion getVersion() {
-return version;
-}
-
-public String getMsgId() {
-return msgId;
-}
-
-public void setMsgId(String msgId) {
-this.msgId = msgId;
-}
-
-public CompressionKind getMsgCompressionKind() {
-return msgCompressionKind;
-}
-
-public void setMsgCompressed(CompressionKind msgCompressionKind) {
-this.msgCompressionKind = msgCompressionKind;
-}
-
-public int getMsgSplitIdx() {
-return msgSplitIdx;
-}
-
-public void setMsgSplitIdx(int msgSplitIdx) {
-this.msgSplitIdx = msgSplitIdx;
-}
-
-public int getMsgSplitCount() {
-return msgSplitCount;
-}
-
-public void setMsgSplitCount(int msgSplitCount) {
-this.msgSplitCount = msgSplitCount;
-}
-
-/**
- * Compare the version of this message with the given version.
- *
- * @param compareToVersion  the version to compare to
- *
- * @return a negative integer, zero, or a positive integer as this 
message's version is less than, equal to,
- * or greater than the given version.
- */
-public int compareVersion(MessageVersion compareToVersion) {
-return version.compareTo(compareToVersion);
-}
-
-
-public static byte[] getBytesUtf8(String str) {
-

[29/42] atlas git commit: ATLAS-2251: Remove TypeSystem and related implementation, to avoid unncessary duplicate of type details in cache

2017-11-12 Thread madhan
http://git-wip-us.apache.org/repos/asf/atlas/blob/435fe3fb/repository/src/main/java/org/apache/atlas/repository/graph/TypedInstanceToGraphMapper.java
--
diff --git 
a/repository/src/main/java/org/apache/atlas/repository/graph/TypedInstanceToGraphMapper.java
 
b/repository/src/main/java/org/apache/atlas/repository/graph/TypedInstanceToGraphMapper.java
deleted file mode 100644
index 1fb4ee9..000
--- 
a/repository/src/main/java/org/apache/atlas/repository/graph/TypedInstanceToGraphMapper.java
+++ /dev/null
@@ -1,941 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.atlas.repository.graph;
-
-import com.google.common.base.Function;
-import com.google.common.collect.Lists;
-import org.apache.atlas.AtlasException;
-import org.apache.atlas.RequestContext;
-import org.apache.atlas.model.instance.GuidMapping;
-import org.apache.atlas.repository.Constants;
-import org.apache.atlas.repository.RepositoryException;
-import org.apache.atlas.repository.graphdb.AtlasEdge;
-import org.apache.atlas.repository.graphdb.AtlasSchemaViolationException;
-import org.apache.atlas.repository.graphdb.AtlasVertex;
-import org.apache.atlas.typesystem.IReferenceableInstance;
-import org.apache.atlas.typesystem.ITypedInstance;
-import org.apache.atlas.typesystem.ITypedReferenceableInstance;
-import org.apache.atlas.typesystem.ITypedStruct;
-import org.apache.atlas.typesystem.exception.EntityExistsException;
-import org.apache.atlas.typesystem.exception.EntityNotFoundException;
-import org.apache.atlas.typesystem.persistence.Id;
-import org.apache.atlas.typesystem.persistence.ReferenceableInstance;
-import org.apache.atlas.typesystem.types.*;
-import org.apache.atlas.typesystem.types.DataTypes.TypeCategory;
-import org.apache.atlas.typesystem.types.utils.TypesUtil;
-import org.apache.atlas.util.AtlasRepositoryConfiguration;
-import org.apache.atlas.utils.SHA256Utils;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-import org.springframework.stereotype.Component;
-
-import javax.inject.Inject;
-import java.security.MessageDigest;
-import java.util.ArrayList;
-import java.util.Collection;
-import java.util.Collections;
-import java.util.Date;
-import java.util.HashMap;
-import java.util.HashSet;
-import java.util.List;
-import java.util.Map;
-import java.util.Set;
-
-import static org.apache.atlas.repository.graph.GraphHelper.string;
-
-@Component
-@Deprecated
-public final class TypedInstanceToGraphMapper {
-
-private static final Logger LOG = 
LoggerFactory.getLogger(TypedInstanceToGraphMapper.class);
-private final Map idToVertexMap = new HashMap<>();
-private final TypeSystem typeSystem = TypeSystem.getInstance();
-private static final GraphHelper graphHelper = GraphHelper.getInstance();
-
-private DeleteHandler deleteHandler;
-private GraphToTypedInstanceMapper graphToTypedInstanceMapper;
-
-@Inject
-public TypedInstanceToGraphMapper(GraphToTypedInstanceMapper 
graphToTypedInstanceMapper, DeleteHandler deleteHandler) {
-this.graphToTypedInstanceMapper = graphToTypedInstanceMapper;
-this.deleteHandler = deleteHandler;
-}
-
-private final String SIGNATURE_HASH_PROPERTY_KEY = 
Constants.INTERNAL_PROPERTY_KEY_PREFIX + "signature";
-
-public enum Operation {
-CREATE,
-UPDATE_PARTIAL,
-UPDATE_FULL
-}
-
-void mapTypedInstanceToGraph(Operation operation, 
ITypedReferenceableInstance... typedInstances)
-throws AtlasException {
-
-RequestContext requestContext = RequestContext.get();
-Collection allNewInstances = new ArrayList<>();
-for (ITypedReferenceableInstance typedInstance : typedInstances) {
-allNewInstances.addAll(walkClassInstances(typedInstance));
-}
-
-TypeUtils.Pair instancesPair =
-createVerticesAndDiscoverInstances(allNewInstances);
-
-List entitiesToCreate = 
instancesPair.left;
-List entitiesToUpdate = 
instancesPair.right;
-
-FullTextMapper fulltextMapper = new FullTextMapper(this, 
graphToTypedInstanceMapper);
-switch 

[38/42] atlas git commit: ATLAS-2251: Remove TypeSystem and related implementation, to avoid unncessary duplicate of type details in cache

2017-11-12 Thread madhan
http://git-wip-us.apache.org/repos/asf/atlas/blob/435fe3fb/intg/src/test/java/org/apache/atlas/TestUtilsV2.java
--
diff --git a/intg/src/test/java/org/apache/atlas/TestUtilsV2.java 
b/intg/src/test/java/org/apache/atlas/TestUtilsV2.java
index 91fdb47..bbccf77 100755
--- a/intg/src/test/java/org/apache/atlas/TestUtilsV2.java
+++ b/intg/src/test/java/org/apache/atlas/TestUtilsV2.java
@@ -18,8 +18,6 @@
 
 package org.apache.atlas;
 
-import com.google.common.collect.ImmutableList;
-import com.google.common.collect.ImmutableSet;
 import org.apache.atlas.model.instance.AtlasEntity;
 import org.apache.atlas.model.instance.AtlasEntity.AtlasEntitiesWithExtInfo;
 import org.apache.atlas.model.instance.AtlasEntity.AtlasEntityWithExtInfo;
@@ -30,7 +28,6 @@ import org.apache.atlas.model.typedef.AtlasClassificationDef;
 import org.apache.atlas.model.typedef.AtlasEntityDef;
 import org.apache.atlas.model.typedef.AtlasEnumDef;
 import org.apache.atlas.model.typedef.AtlasEnumDef.AtlasEnumElementDef;
-import org.apache.atlas.model.typedef.AtlasRelationshipDef;
 import org.apache.atlas.model.typedef.AtlasStructDef;
 import org.apache.atlas.model.typedef.AtlasStructDef.AtlasAttributeDef;
 import 
org.apache.atlas.model.typedef.AtlasStructDef.AtlasAttributeDef.Cardinality;
@@ -92,7 +89,7 @@ public final class TestUtilsV2 {
 AtlasTypeUtil.createRequiredAttrDef("city", "string"));
 
 AtlasEntityDef deptTypeDef =
-AtlasTypeUtil.createClassTypeDef(DEPARTMENT_TYPE, 
"Department"+_description, ImmutableSet.of(),
+AtlasTypeUtil.createClassTypeDef(DEPARTMENT_TYPE, 
"Department"+_description, Collections.emptySet(),
 AtlasTypeUtil.createUniqueRequiredAttrDef("name", 
"string"),
 new AtlasAttributeDef("employees", 
String.format("array<%s>", "Employee"), true,
 AtlasAttributeDef.Cardinality.SINGLE, 0, 1, 
false, false,
@@ -100,7 +97,7 @@ public final class TestUtilsV2 {
 add(new 
AtlasStructDef.AtlasConstraintDef(AtlasConstraintDef.CONSTRAINT_TYPE_OWNED_REF));
 }}));
 
-AtlasEntityDef personTypeDef = 
AtlasTypeUtil.createClassTypeDef("Person", "Person"+_description, 
ImmutableSet.of(),
+AtlasEntityDef personTypeDef = 
AtlasTypeUtil.createClassTypeDef("Person", "Person"+_description, 
Collections.emptySet(),
 AtlasTypeUtil.createUniqueRequiredAttrDef("name", "string"),
 AtlasTypeUtil.createOptionalAttrDef("address", "Address"),
 AtlasTypeUtil.createOptionalAttrDef("birthday", "date"),
@@ -113,7 +110,7 @@ public final class TestUtilsV2 {
 AtlasTypeUtil.createOptionalAttrDef("approximationOfPi", 
"bigdecimal")
 );
 
-AtlasEntityDef employeeTypeDef = 
AtlasTypeUtil.createClassTypeDef("Employee", "Employee"+_description, 
ImmutableSet.of("Person"),
+AtlasEntityDef employeeTypeDef = 
AtlasTypeUtil.createClassTypeDef("Employee", "Employee"+_description, 
Collections.singleton("Person"),
 AtlasTypeUtil.createOptionalAttrDef("orgLevel", "OrgLevel"),
 new AtlasAttributeDef("department", "Department", false,
 AtlasAttributeDef.Cardinality.SINGLE, 1, 1,
@@ -142,18 +139,18 @@ public final class TestUtilsV2 {
 put(AtlasConstraintDef.CONSTRAINT_PARAM_ATTRIBUTE, 
"employees");
 }}));
 
-AtlasEntityDef managerTypeDef = 
AtlasTypeUtil.createClassTypeDef("Manager", "Manager"+_description, 
ImmutableSet.of("Employee"),
+AtlasEntityDef managerTypeDef = 
AtlasTypeUtil.createClassTypeDef("Manager", "Manager"+_description, 
Collections.singleton("Employee"),
 new AtlasAttributeDef("subordinates", 
String.format("array<%s>", "Employee"), false, 
AtlasAttributeDef.Cardinality.SET,
 1, 10, false, false,
 Collections.emptyList()));
 
 AtlasClassificationDef securityClearanceTypeDef =
-AtlasTypeUtil.createTraitTypeDef("SecurityClearance", 
"SecurityClearance"+_description, ImmutableSet.of(),
+AtlasTypeUtil.createTraitTypeDef("SecurityClearance", 
"SecurityClearance"+_description, Collections.emptySet(),
 AtlasTypeUtil.createRequiredAttrDef("level", "int"));
 
-AtlasTypesDef ret = new AtlasTypesDef(ImmutableList.of(orgLevelEnum), 
ImmutableList.of(addressDetails),
-ImmutableList.of(securityClearanceTypeDef),
-ImmutableList.of(deptTypeDef, personTypeDef, employeeTypeDef, 
managerTypeDef));
+AtlasTypesDef ret = new 
AtlasTypesDef(Collections.singletonList(orgLevelEnum), 
Collections.singletonList(addressDetails),
+Collections.singletonList(securityClearanceTypeDef),
+Arrays.asList(deptTypeDef, 

[19/42] atlas git commit: ATLAS-2251: Remove TypeSystem and related implementation, to avoid unncessary duplicate of type details in cache

2017-11-12 Thread madhan
http://git-wip-us.apache.org/repos/asf/atlas/blob/435fe3fb/repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepositoryTest.java
--
diff --git 
a/repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepositoryTest.java
 
b/repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepositoryTest.java
deleted file mode 100755
index 952a644..000
--- 
a/repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepositoryTest.java
+++ /dev/null
@@ -1,864 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.atlas.repository.graph;
-
-import com.google.common.collect.ImmutableList;
-import com.google.common.collect.ImmutableSet;
-import org.apache.atlas.AtlasException;
-import org.apache.atlas.CreateUpdateEntitiesResult;
-import org.apache.atlas.RequestContext;
-import org.apache.atlas.TestModules;
-import org.apache.atlas.TestUtils;
-import org.apache.atlas.annotation.GraphTransaction;
-import org.apache.atlas.discovery.graph.GraphBackedDiscoveryService;
-import org.apache.atlas.query.QueryParams;
-import org.apache.atlas.repository.Constants;
-import org.apache.atlas.repository.MetadataRepository;
-import org.apache.atlas.repository.RepositoryException;
-import org.apache.atlas.repository.graphdb.AtlasEdge;
-import org.apache.atlas.repository.graphdb.AtlasEdgeDirection;
-import org.apache.atlas.repository.graphdb.AtlasGraph;
-import org.apache.atlas.repository.graphdb.AtlasGraphQuery;
-import org.apache.atlas.repository.graphdb.AtlasGraphQuery.ComparisionOperator;
-import org.apache.atlas.repository.graphdb.AtlasVertex;
-import org.apache.atlas.type.AtlasTypeRegistry;
-import org.apache.atlas.typesystem.IStruct;
-import org.apache.atlas.typesystem.ITypedReferenceableInstance;
-import org.apache.atlas.typesystem.ITypedStruct;
-import org.apache.atlas.typesystem.Referenceable;
-import org.apache.atlas.typesystem.Struct;
-import org.apache.atlas.typesystem.exception.EntityNotFoundException;
-import org.apache.atlas.typesystem.exception.TraitNotFoundException;
-import org.apache.atlas.typesystem.persistence.AtlasSystemAttributes;
-import org.apache.atlas.typesystem.persistence.Id;
-import org.apache.atlas.typesystem.types.AttributeDefinition;
-import org.apache.atlas.typesystem.types.ClassType;
-import org.apache.atlas.typesystem.types.DataTypes;
-import org.apache.atlas.typesystem.types.HierarchicalTypeDefinition;
-import org.apache.atlas.typesystem.types.Multiplicity;
-import org.apache.atlas.typesystem.types.TraitType;
-import org.apache.atlas.typesystem.types.TypeSystem;
-import org.apache.atlas.typesystem.types.utils.TypesUtil;
-import org.apache.commons.lang.RandomStringUtils;
-import org.codehaus.jettison.json.JSONArray;
-import org.codehaus.jettison.json.JSONObject;
-import org.testng.Assert;
-import org.testng.annotations.AfterClass;
-import org.testng.annotations.BeforeClass;
-import org.testng.annotations.BeforeMethod;
-import org.testng.annotations.Guice;
-import org.testng.annotations.Test;
-
-import javax.inject.Inject;
-import java.util.ArrayList;
-import java.util.Arrays;
-import java.util.Date;
-import java.util.HashMap;
-import java.util.Iterator;
-import java.util.List;
-import java.util.Map;
-import java.util.UUID;
-import java.util.concurrent.Callable;
-import java.util.concurrent.ExecutorService;
-import java.util.concurrent.Executors;
-import java.util.concurrent.Future;
-
-import static 
org.apache.atlas.typesystem.types.utils.TypesUtil.createClassTypeDef;
-import static 
org.apache.atlas.typesystem.types.utils.TypesUtil.createUniqueRequiredAttrDef;
-import static org.testng.Assert.assertEquals;
-import static org.testng.Assert.assertNotEquals;
-import static org.testng.Assert.assertNotNull;
-import static org.testng.Assert.assertTrue;
-
-/**
- * GraphBackedMetadataRepository test
- *
- * Guice loads the dependencies and injects the necessary objects
- *
- */
-@Guice(modules = TestModules.TestOnlyModule.class)
-public class GraphBackedMetadataRepositoryTest {
-
-@Inject
-private MetadataRepository repositoryService;
-
-@Inject
-

[28/42] atlas git commit: ATLAS-2251: Remove TypeSystem and related implementation, to avoid unncessary duplicate of type details in cache

2017-11-12 Thread madhan
http://git-wip-us.apache.org/repos/asf/atlas/blob/435fe3fb/repository/src/main/java/org/apache/atlas/repository/memory/MemRepository.java
--
diff --git 
a/repository/src/main/java/org/apache/atlas/repository/memory/MemRepository.java
 
b/repository/src/main/java/org/apache/atlas/repository/memory/MemRepository.java
deleted file mode 100755
index aef06a4..000
--- 
a/repository/src/main/java/org/apache/atlas/repository/memory/MemRepository.java
+++ /dev/null
@@ -1,299 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.atlas.repository.memory;
-
-import org.apache.atlas.AtlasException;
-import org.apache.atlas.repository.DiscoverInstances;
-import org.apache.atlas.repository.IRepository;
-import org.apache.atlas.repository.RepositoryException;
-import org.apache.atlas.typesystem.IReferenceableInstance;
-import org.apache.atlas.typesystem.ITypedReferenceableInstance;
-import org.apache.atlas.typesystem.persistence.Id;
-import org.apache.atlas.typesystem.persistence.MapIds;
-import org.apache.atlas.typesystem.persistence.ReferenceableInstance;
-import org.apache.atlas.typesystem.types.ClassType;
-import org.apache.atlas.typesystem.types.DataTypes;
-import org.apache.atlas.typesystem.types.HierarchicalType;
-import org.apache.atlas.typesystem.types.HierarchicalTypeDependencySorter;
-import org.apache.atlas.typesystem.types.Multiplicity;
-import org.apache.atlas.typesystem.types.ObjectGraphWalker;
-import org.apache.atlas.typesystem.types.TraitType;
-import org.apache.atlas.typesystem.types.TypeSystem;
-
-import java.util.ArrayList;
-import java.util.HashMap;
-import java.util.List;
-import java.util.Map;
-import java.util.Set;
-import java.util.TreeSet;
-import java.util.concurrent.atomic.AtomicInteger;
-
-@Deprecated
-public class MemRepository implements IRepository {
-
-final TypeSystem typeSystem;
-/*
- * A Store for each Class and Trait.
- */
-final Map typeStores;
-final AtomicInteger ID_SEQ = new AtomicInteger(0);
-
-public MemRepository(TypeSystem typeSystem) {
-this.typeSystem = typeSystem;
-this.typeStores = new HashMap<>();
-}
-
-@Override
-public Id newId(String typeName) {
-return new Id("" + ID_SEQ.incrementAndGet(), 0, typeName);
-}
-
-/**
- * 1. traverse the Object Graph from  i and create idToNewIdMap : Map[Id, 
Id],
- *also create old Id to Instance Map: oldIdToInstance : Map[Id, 
IInstance]
- *   - traverse reference Attributes, List[ClassType], Maps where 
Key/value is ClassType
- *   - traverse Structs
- *   - traverse Traits.
- * 1b. Ensure that every newId has an associated Instance.
- * 2. Traverse oldIdToInstance map create newInstances : 
List[ITypedReferenceableInstance]
- *- create a ITypedReferenceableInstance.
- *  replace any old References ( ids or object references) with new 
Ids.
- * 3. Traverse over newInstances
- *- ask ClassStore to assign a position to the Id.
- *  - for Instances with Traits, assign a position for each Trait
- *- invoke store on the nwInstance.
- *
- * Recovery:
- * - on each newInstance, invoke releaseId and delete on its ClassStore 
and Traits' Stores.
- *
- * @param i
- * @return
- * @throws org.apache.atlas.repository.RepositoryException
- */
-public ITypedReferenceableInstance create(IReferenceableInstance i) throws 
RepositoryException {
-
-DiscoverInstances discoverInstances = new DiscoverInstances(this);
-
-/*
- * Step 1: traverse the Object Graph from  i and create idToNewIdMap : 
Map[Id, Id],
- *also create old Id to Instance Map: oldIdToInstance : Map[Id, 
IInstance]
- *   - traverse reference Attributes, List[ClassType], Maps where 
Key/value is ClassType
- *   - traverse Structs
- *   - traverse Traits.
- */
-try {
-new ObjectGraphWalker(typeSystem, discoverInstances, i).walk();
-} catch (AtlasException me) {
-throw new RepositoryException("TypeSystem error when 

[39/42] atlas git commit: ATLAS-2251: Remove TypeSystem and related implementation, to avoid unncessary duplicate of type details in cache

2017-11-12 Thread madhan
http://git-wip-us.apache.org/repos/asf/atlas/blob/435fe3fb/intg/src/main/java/org/apache/atlas/v1/model/notification/HookNotificationV1.java
--
diff --git 
a/intg/src/main/java/org/apache/atlas/v1/model/notification/HookNotificationV1.java
 
b/intg/src/main/java/org/apache/atlas/v1/model/notification/HookNotificationV1.java
new file mode 100644
index 000..c70e7d0
--- /dev/null
+++ 
b/intg/src/main/java/org/apache/atlas/v1/model/notification/HookNotificationV1.java
@@ -0,0 +1,357 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.atlas.v1.model.notification;
+
+import org.apache.atlas.model.notification.HookNotification;
+import org.apache.atlas.model.typedef.AtlasBaseTypeDef;
+import org.apache.atlas.v1.model.instance.Referenceable;
+import org.apache.atlas.v1.model.typedef.TypesDef;
+import org.codehaus.jackson.annotate.JsonAutoDetect;
+import org.codehaus.jackson.annotate.JsonIgnoreProperties;
+import org.codehaus.jackson.map.annotate.JsonSerialize;
+
+import javax.xml.bind.annotation.XmlAccessType;
+import javax.xml.bind.annotation.XmlAccessorType;
+import javax.xml.bind.annotation.XmlRootElement;
+import java.io.Serializable;
+import java.util.Arrays;
+import java.util.List;
+
+import static org.codehaus.jackson.annotate.JsonAutoDetect.Visibility.NONE;
+import static 
org.codehaus.jackson.annotate.JsonAutoDetect.Visibility.PUBLIC_ONLY;
+
+/**
+ * Contains the structure of messages transferred from hooks to atlas.
+ */
+public class HookNotificationV1 {
+
+/**
+ * Hook message for create type definitions.
+ */
+@JsonAutoDetect(getterVisibility=PUBLIC_ONLY, 
setterVisibility=PUBLIC_ONLY, fieldVisibility=NONE)
+@JsonSerialize(include=JsonSerialize.Inclusion.ALWAYS)
+@JsonIgnoreProperties(ignoreUnknown=true)
+@XmlRootElement
+@XmlAccessorType(XmlAccessType.PROPERTY)
+public static class TypeRequest extends HookNotification implements 
Serializable {
+private static final long serialVersionUID = 1L;
+
+private TypesDef typesDef;
+
+public TypeRequest() {
+}
+
+public TypeRequest(HookNotificationType type, TypesDef typesDef, 
String user) {
+super(type, user);
+this.typesDef = typesDef;
+}
+
+public TypesDef getTypesDef() {
+return typesDef;
+}
+
+public void setTypesDef(TypesDef typesDef) {
+this.typesDef = typesDef;
+}
+
+@Override
+public StringBuilder toString(StringBuilder sb) {
+if (sb == null) {
+sb = new StringBuilder();
+}
+
+sb.append("TypeRequest{");
+super.toString(sb);
+sb.append("typesDef=");
+if (typesDef != null) {
+typesDef.toString(sb);
+}
+sb.append("}");
+
+return sb;
+}
+}
+
+/**
+ * Hook message for creating new entities.
+ */
+@JsonAutoDetect(getterVisibility=PUBLIC_ONLY, 
setterVisibility=PUBLIC_ONLY, fieldVisibility=NONE)
+@JsonSerialize(include=JsonSerialize.Inclusion.ALWAYS)
+@JsonIgnoreProperties(ignoreUnknown=true)
+@XmlRootElement
+@XmlAccessorType(XmlAccessType.PROPERTY)
+public static class EntityCreateRequest extends HookNotification 
implements Serializable {
+private static final long serialVersionUID = 1L;
+
+private List entities;
+
+public EntityCreateRequest() {
+}
+
+public EntityCreateRequest(String user, Referenceable... entities) {
+this(HookNotificationType.ENTITY_CREATE, Arrays.asList(entities), 
user);
+}
+
+public EntityCreateRequest(String user, List entities) {
+this(HookNotificationType.ENTITY_CREATE, entities, user);
+}
+
+protected EntityCreateRequest(HookNotificationType type, 
List entities, String user) {
+super(type, user);
+
+this.entities = entities;
+}
+
+public List getEntities() {
+return entities;
+}
+
+public void setEntities(List entities) {
+this.entities = 

[10/42] atlas git commit: ATLAS-2251: Remove TypeSystem and related implementation, to avoid unncessary duplicate of type details in cache

2017-11-12 Thread madhan
http://git-wip-us.apache.org/repos/asf/atlas/blob/435fe3fb/typesystem/src/main/java/org/apache/atlas/typesystem/types/DataTypes.java
--
diff --git 
a/typesystem/src/main/java/org/apache/atlas/typesystem/types/DataTypes.java 
b/typesystem/src/main/java/org/apache/atlas/typesystem/types/DataTypes.java
deleted file mode 100755
index f9f4abe..000
--- a/typesystem/src/main/java/org/apache/atlas/typesystem/types/DataTypes.java
+++ /dev/null
@@ -1,655 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.atlas.typesystem.types;
-
-import com.google.common.collect.ImmutableCollection;
-import com.google.common.collect.ImmutableList;
-import com.google.common.collect.ImmutableMap;
-import com.google.common.collect.ImmutableSet;
-import org.apache.atlas.AtlasException;
-import org.apache.atlas.typesystem.IReferenceableInstance;
-import org.apache.atlas.typesystem.persistence.Id;
-import org.apache.commons.lang3.StringUtils;
-import org.joda.time.DateTime;
-import org.joda.time.DateTimeZone;
-import org.joda.time.format.DateTimeFormatter;
-import org.joda.time.format.ISODateTimeFormat;
-
-import java.math.BigDecimal;
-import java.math.BigInteger;
-import java.nio.charset.Charset;
-import java.security.MessageDigest;
-import java.util.Collection;
-import java.util.Date;
-import java.util.Iterator;
-import java.util.List;
-import java.util.Map;
-import java.util.Set;
-
-public class DataTypes {
-
-public static BooleanType BOOLEAN_TYPE = new BooleanType();
-public static ByteType BYTE_TYPE = new ByteType();
-public static ShortType SHORT_TYPE = new ShortType();
-public static IntType INT_TYPE = new IntType();
-public static LongType LONG_TYPE = new LongType();
-public static FloatType FLOAT_TYPE = new FloatType();
-public static DoubleType DOUBLE_TYPE = new DoubleType();
-public static BigIntegerType BIGINTEGER_TYPE = new BigIntegerType();
-public static BigDecimalType BIGDECIMAL_TYPE = new BigDecimalType();
-public static DateType DATE_TYPE = new DateType();
-public static StringType STRING_TYPE = new StringType();
-public static String ARRAY_TYPE_PREFIX = "array<";
-static String ARRAY_TYPE_SUFFIX = ">";
-public static String MAP_TYPE_PREFIX = "map<";
-static String MAP_TYPE_SUFFIX = ">";
-
-public static String arrayTypeName(String elemTypeName) {
-assert elemTypeName != null;
-return String.format("%s%s%s", ARRAY_TYPE_PREFIX, elemTypeName, 
ARRAY_TYPE_SUFFIX);
-}
-
-public static String arrayTypeName(IDataType elemType) {
-return arrayTypeName(elemType.getName());
-}
-
-public static String mapTypeName(String keyTypeName, String valueTypeName) 
{
-return String.format("%s%s,%s%s", MAP_TYPE_PREFIX, keyTypeName, 
valueTypeName, MAP_TYPE_SUFFIX);
-}
-
-public static String mapTypeName(IDataType keyType, IDataType valueType) {
-assert keyType != null;
-assert valueType != null;
-return mapTypeName(keyType.getName(), valueType.getName());
-}
-
-public enum TypeCategory {
-PRIMITIVE,
-ENUM,
-ARRAY,
-MAP,
-STRUCT,
-TRAIT,
-CLASS,
-RELATIONSHIP
-}
-
-public static abstract class PrimitiveType extends AbstractDataType {
-public PrimitiveType(String name, String description) {
-super(name, description);
-}
-
-@Override
-public TypeCategory getTypeCategory() {
-return TypeCategory.PRIMITIVE;
-}
-
-public abstract T nullValue();
-
-@Override
-protected T convertNull(Multiplicity m) throws AtlasException {
-if (!m.nullAllowed()) {
-throw new ValueConversionException.NullConversionException(m);
-}
-
-return nullValue();
-}
-
-@Override
-public void updateSignatureHash(MessageDigest digester, Object val) 
throws AtlasException {
-if ( val != null ) {
-
digester.update(val.toString().getBytes(Charset.forName("UTF-8")));
-}
-}
-
-}
-
-

[35/42] atlas git commit: ATLAS-2251: Remove TypeSystem and related implementation, to avoid unncessary duplicate of type details in cache

2017-11-12 Thread madhan
http://git-wip-us.apache.org/repos/asf/atlas/blob/435fe3fb/repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java
--
diff --git 
a/repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java
 
b/repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java
index 7f24d5a..03a86f4 100644
--- 
a/repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java
+++ 
b/repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java
@@ -24,7 +24,6 @@ import org.apache.atlas.AtlasErrorCode;
 import org.apache.atlas.AtlasException;
 import org.apache.atlas.SortOrder;
 import org.apache.atlas.annotation.GraphTransaction;
-import org.apache.atlas.discovery.graph.DefaultGraphPersistenceStrategy;
 import org.apache.atlas.exception.AtlasBaseException;
 import org.apache.atlas.model.discovery.AtlasSearchResult;
 import org.apache.atlas.model.discovery.AtlasSearchResult.AtlasFullTextResult;
@@ -44,7 +43,6 @@ import org.apache.atlas.query.QueryParser;
 import org.apache.atlas.query.QueryProcessor;
 import org.apache.atlas.query.SelectExpressionHelper;
 import org.apache.atlas.repository.Constants;
-import org.apache.atlas.repository.MetadataRepository;
 import org.apache.atlas.repository.graph.GraphBackedSearchIndexer;
 import org.apache.atlas.repository.graph.GraphHelper;
 import org.apache.atlas.repository.graphdb.AtlasGraph;
@@ -70,9 +68,6 @@ import org.apache.commons.lang.StringUtils;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 import org.springframework.stereotype.Component;
-import scala.Option;
-import scala.util.Either;
-import scala.util.parsing.combinator.Parsers.NoSuccess;
 
 import javax.inject.Inject;
 import javax.script.Bindings;
@@ -104,7 +99,6 @@ public class EntityDiscoveryService implements 
AtlasDiscoveryService {
 private static final String DEFAULT_SORT_ATTRIBUTE_NAME = "name";
 
 private final AtlasGraph  graph;
-private final DefaultGraphPersistenceStrategy graphPersistenceStrategy;
 private final EntityGraphRetrieverentityRetriever;
 private final AtlasGremlinQueryProvider   gremlinQueryProvider;
 private final AtlasTypeRegistry   typeRegistry;
@@ -116,11 +110,10 @@ public class EntityDiscoveryService implements 
AtlasDiscoveryService {
 private final UserProfileService  userProfileService;
 
 @Inject
-EntityDiscoveryService(MetadataRepository metadataRepository, 
AtlasTypeRegistry typeRegistry,
+EntityDiscoveryService(AtlasTypeRegistry typeRegistry,
AtlasGraph graph, GraphBackedSearchIndexer indexer, 
SearchTracker searchTracker,
UserProfileService userProfileService) throws 
AtlasException {
 this.graph= graph;
-this.graphPersistenceStrategy = new 
DefaultGraphPersistenceStrategy(metadataRepository);
 this.entityRetriever  = new EntityGraphRetriever(typeRegistry);
 this.indexer  = indexer;
 this.searchTracker= searchTracker;
@@ -685,15 +678,14 @@ public class EntityDiscoveryService implements 
AtlasDiscoveryService {
 
 private GremlinQuery toGremlinQuery(String query, int limit, int offset) 
throws AtlasBaseException {
 QueryParams params = validateSearchParams(limit, offset);
-Either either = QueryParser.apply(query, 
params);
+Expression expression = QueryParser.apply(query, params);
 
-if (either.isLeft()) {
+if (expression == null) {
 throw new AtlasBaseException(DISCOVERY_QUERY_FAILED, query);
 }
 
-Expression   expression  = either.right().get();
 Expression   validExpression = QueryProcessor.validate(expression);
-GremlinQuery gremlinQuery= new GremlinTranslator(validExpression, 
graphPersistenceStrategy).translate();
+GremlinQuery gremlinQuery= new 
GremlinTranslator(validExpression).translate();
 
 if (LOG.isDebugEnabled()) {
 LOG.debug("Translated Gremlin Query: {}", gremlinQuery.queryStr());
@@ -730,9 +722,9 @@ public class EntityDiscoveryService implements 
AtlasDiscoveryService {
 List values = new ArrayList<>();
 
 // extract select attributes from gremlin query
-Option selectExpr = 
SelectExpressionHelper.extractSelectExpression(query.expr());
-if (selectExpr.isDefined()) {
-List aliases = selectExpr.get().toJavaList();
+SelectExpression selectExpr = 
SelectExpressionHelper.extractSelectExpression(query.expr());
+if (selectExpr != null) {
+List aliases = selectExpr.toJavaList();
 
 if (CollectionUtils.isNotEmpty(aliases)) {
 for (AliasExpression alias : aliases) {


[18/42] atlas git commit: ATLAS-2251: Remove TypeSystem and related implementation, to avoid unncessary duplicate of type details in cache

2017-11-12 Thread madhan
http://git-wip-us.apache.org/repos/asf/atlas/blob/435fe3fb/repository/src/test/java/org/apache/atlas/repository/graph/GraphRepoMapperScaleTest.java
--
diff --git 
a/repository/src/test/java/org/apache/atlas/repository/graph/GraphRepoMapperScaleTest.java
 
b/repository/src/test/java/org/apache/atlas/repository/graph/GraphRepoMapperScaleTest.java
deleted file mode 100755
index 1f077ba..000
--- 
a/repository/src/test/java/org/apache/atlas/repository/graph/GraphRepoMapperScaleTest.java
+++ /dev/null
@@ -1,241 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- * 
- * http://www.apache.org/licenses/LICENSE-2.0
- * 
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.atlas.repository.graph;
-
-import org.apache.atlas.ApplicationProperties;
-import org.apache.atlas.CreateUpdateEntitiesResult;
-import org.apache.atlas.TestModules;
-import org.apache.atlas.TestUtils;
-import org.apache.atlas.annotation.GraphTransaction;
-import org.apache.atlas.repository.Constants;
-import org.apache.atlas.repository.RepositoryException;
-import org.apache.atlas.repository.graphdb.AtlasGraph;
-import org.apache.atlas.repository.graphdb.AtlasGraphQuery;
-import org.apache.atlas.repository.graphdb.AtlasGraphQuery.ComparisionOperator;
-import org.apache.atlas.repository.graphdb.AtlasIndexQuery;
-import org.apache.atlas.repository.graphdb.AtlasVertex;
-import org.apache.atlas.type.AtlasTypeRegistry;
-import org.apache.atlas.typesystem.ITypedReferenceableInstance;
-import org.apache.atlas.typesystem.Referenceable;
-import org.apache.atlas.typesystem.Struct;
-import org.apache.atlas.typesystem.exception.EntityExistsException;
-import org.apache.atlas.typesystem.persistence.Id;
-import org.apache.atlas.typesystem.types.ClassType;
-import org.apache.atlas.typesystem.types.IDataType;
-import org.apache.atlas.typesystem.types.Multiplicity;
-import org.apache.atlas.typesystem.types.TypeSystem;
-import org.testng.Assert;
-import org.testng.annotations.AfterClass;
-import org.testng.annotations.BeforeClass;
-import org.testng.annotations.BeforeMethod;
-import org.testng.annotations.Guice;
-import org.testng.annotations.Test;
-
-import javax.inject.Inject;
-import java.util.ArrayList;
-import java.util.Collection;
-import java.util.Date;
-import java.util.Iterator;
-
-@Test
-@Guice(modules = TestModules.TestOnlyModule.class)
-public class GraphRepoMapperScaleTest {
-
-private static final String DATABASE_NAME = "foo";
-private static final String TABLE_NAME = "bar";
-
-@Inject
-private GraphBackedMetadataRepository repositoryService;
-
-@Inject
-private GraphBackedSearchIndexer searchIndexer;
-
-private TypeSystem typeSystem = TypeSystem.getInstance();
-
-private String dbGUID;
-
-@BeforeClass
-@GraphTransaction
-public void setUp() throws Exception {
-//force up front graph initialization
-TestUtils.getGraph();
-searchIndexer = new GraphBackedSearchIndexer(new AtlasGraphProvider(), 
ApplicationProperties.get(), new AtlasTypeRegistry());
-//Make sure we can cleanup the index directory
-Collection typesAdded = 
TestUtils.createHiveTypes(typeSystem);
-searchIndexer.onAdd(typesAdded);
-}
-
-@BeforeMethod
-public void setupContext() {
-TestUtils.resetRequestContext();
-}
-
-@AfterClass
-public void tearDown() throws Exception {
-TypeSystem.getInstance().reset();
-//AtlasGraphProvider.cleanup();
-}
-
-@Test
-public void testSubmitEntity() throws Exception {
-Referenceable databaseInstance = new 
Referenceable(TestUtils.DATABASE_TYPE);
-databaseInstance.set("name", DATABASE_NAME);
-databaseInstance.set("description", "foo database");
-// System.out.println("databaseInstance = " + databaseInstance);
-
-ClassType dbType = typeSystem.getDataType(ClassType.class, 
TestUtils.DATABASE_TYPE);
-ITypedReferenceableInstance db = dbType.convert(databaseInstance, 
Multiplicity.REQUIRED);
-
-dbGUID = result(db).getCreatedEntities().get(0);
-
-Referenceable dbInstance = new Referenceable(dbGUID, 
TestUtils.DATABASE_TYPE, databaseInstance.getValuesMap());
-
-for (int index = 0; index < 1000; 

[26/42] atlas git commit: ATLAS-2251: Remove TypeSystem and related implementation, to avoid unncessary duplicate of type details in cache

2017-11-12 Thread madhan
http://git-wip-us.apache.org/repos/asf/atlas/blob/435fe3fb/repository/src/main/scala/org/apache/atlas/query/Expressions.scala
--
diff --git a/repository/src/main/scala/org/apache/atlas/query/Expressions.scala 
b/repository/src/main/scala/org/apache/atlas/query/Expressions.scala
deleted file mode 100644
index bf9efd2..000
--- a/repository/src/main/scala/org/apache/atlas/query/Expressions.scala
+++ /dev/null
@@ -1,923 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.atlas.query
-
-import com.google.common.collect.ImmutableCollection
-import org.apache.atlas.AtlasException
-import org.apache.atlas.typesystem.types.DataTypes.{ArrayType, PrimitiveType, 
TypeCategory}
-import org.apache.atlas.typesystem.types._
-import scala.collection.JavaConverters._
-
-object Expressions {
-
-import TypeUtils._
-
-class ExpressionException(val e: Expression, message: String, cause: 
Throwable, enableSuppression: Boolean,
-  writableStackTrace: Boolean)
-extends AtlasException(message, cause, enableSuppression, 
writableStackTrace) {
-
-def this(e: Expression, message: String) {
-this(e, message, null, false, true)
-}
-
-def this(e: Expression, message: String, cause: Throwable) {
-this(e, message, cause, false, true)
-}
-
-def this(e: Expression, cause: Throwable) {
-this(e, null, cause, false, true)
-}
-
-override def getMessage: String = {
-val eString = e.toString
-s"${super.getMessage}, expression:${if (eString contains "\n") 
"\n" else " "}$e"
-}
-
-}
-
-class UnresolvedException(expr: Expression, function: String) extends
-ExpressionException(expr, s"Unresolved $function")
-
-def attachExpression[A](e: Expression, msg: String = "")(f: => A): A = {
-try f catch {
-case eex: ExpressionException => throw eex
-case ex: Exception => throw new ExpressionException(e, msg, ex)
-}
-}
-
-trait Expression {
-self: Product =>
-def isAggregator = false
-def children: Seq[Expression]
-
-/**
- * Returns `true` if the schema for this expression and all its 
children have been resolved.
- * The default logic is that an Expression is resolve if all its 
children are resolved.
- */
-lazy val resolved: Boolean = childrenResolved
-
-/**
- * Returns the output [[IDataType[_]] of this expression.  Expressions 
that are unresolved will
- * throw if this method is invoked.
- */
-def dataType: IDataType[_]
-
-/**
- * Returns true if  all the children have been resolved.
- */
-def childrenResolved = !children.exists(!_.resolved)
-
-
-/**
- * the aliases that are present in this Expression Tree
- */
-def namedExpressions: Map[String, Expression] = Map()
-
-def fastEquals(other: Expression): Boolean = {
-this.eq(other) || this == other
-}
-
-def makeCopy(newArgs: Array[AnyRef]): this.type = 
attachExpression(this, "makeCopy") {
-try {
-val defaultCtor = 
getClass.getConstructors.find(_.getParameterTypes.size != 0).head
-defaultCtor.newInstance(newArgs: _*).asInstanceOf[this.type]
-} catch {
-case e: java.lang.IllegalArgumentException =>
-throw new ExpressionException(
-this, s"Failed to copy node. Reason: ${e.getMessage}.")
-}
-}
-
-def transformChildrenDown(rule: PartialFunction[Expression, 
Expression]): this.type = {
-var changed = false
-val newArgs = productIterator.map {
-case arg: Expression if children contains arg =>
-val newChild = 
arg.asInstanceOf[Expression].transformDown(rule)
-if (!(newChild fastEquals arg)) {
-changed = true
-newChild
-} else {
-  

[08/42] atlas git commit: ATLAS-2251: Remove TypeSystem and related implementation, to avoid unncessary duplicate of type details in cache

2017-11-12 Thread madhan
http://git-wip-us.apache.org/repos/asf/atlas/blob/435fe3fb/typesystem/src/main/java/org/apache/atlas/typesystem/types/cache/DefaultTypeCache.java
--
diff --git 
a/typesystem/src/main/java/org/apache/atlas/typesystem/types/cache/DefaultTypeCache.java
 
b/typesystem/src/main/java/org/apache/atlas/typesystem/types/cache/DefaultTypeCache.java
deleted file mode 100644
index 0d86474..000
--- 
a/typesystem/src/main/java/org/apache/atlas/typesystem/types/cache/DefaultTypeCache.java
+++ /dev/null
@@ -1,301 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.atlas.typesystem.types.cache;
-
-import org.apache.atlas.AtlasException;
-import org.apache.atlas.annotation.ConditionalOnAtlasProperty;
-import org.apache.atlas.typesystem.types.ClassType;
-import org.apache.atlas.typesystem.types.DataTypes.TypeCategory;
-import org.apache.atlas.typesystem.types.EnumType;
-import org.apache.atlas.typesystem.types.HierarchicalType;
-import org.apache.atlas.typesystem.types.IDataType;
-import org.apache.atlas.typesystem.types.StructType;
-import org.apache.atlas.typesystem.types.TraitType;
-import org.apache.commons.lang.StringUtils;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-import org.springframework.stereotype.Component;
-
-import javax.inject.Singleton;
-import java.util.ArrayList;
-import java.util.Arrays;
-import java.util.Collection;
-import java.util.List;
-import java.util.Map;
-import java.util.Map.Entry;
-import java.util.concurrent.ConcurrentHashMap;
-
-/**
- * Caches the types in-memory within the same process space.
- */
-@SuppressWarnings("rawtypes")
-@Singleton
-@Component
-@ConditionalOnAtlasProperty(property = "atlas.TypeCache.impl", isDefault = 
true)
-public class DefaultTypeCache implements TypeCache {
-private static final Logger LOG = 
LoggerFactory.getLogger(DefaultTypeCache.class);
-
-private Map types_ = new ConcurrentHashMap<>();
-private static final List validTypeFilterCategories =
-Arrays.asList(TypeCategory.CLASS, TypeCategory.TRAIT, 
TypeCategory.ENUM, TypeCategory.STRUCT);
-private static final List validSupertypeFilterCategories =
-Arrays.asList(TypeCategory.CLASS, TypeCategory.TRAIT);
-
-/*
- * (non-Javadoc)
- * @see
- * org.apache.atlas.typesystem.types.cache.TypeCache#has(java.lang
- * .String)
- */
-@Override
-public boolean has(String typeName) throws AtlasException {
-
-return types_.containsKey(typeName);
-}
-
-/* (non-Javadoc)
- * @see org.apache.atlas.typesystem.types.cache.TypeCache#has(org.
- * apache.atlas.typesystem.types.DataTypes.TypeCategory, java.lang.String)
- */
-@Override
-public boolean has(TypeCategory typeCategory, String typeName)
-throws AtlasException {
-
-assertValidTypeCategory(typeCategory);
-return has(typeName);
-}
-
-private void assertValidTypeCategory(String typeCategory) {
-assertValidTypeCategory(TypeCategory.valueOf(typeCategory));
-}
-
-private void assertValidTypeCategory(TypeCategory typeCategory) {
-// there might no need of 'typeCategory' in this implementation for
-// certain API, but for a distributed cache, it might help for the
-// implementers to partition the types per their category
-// while persisting so that look can be efficient
-
-if (typeCategory == null) {
-throw new IllegalArgumentException("Category of the types to be 
filtered is null.");
-}
-
-if (!validTypeFilterCategories.contains(typeCategory)) {
-throw new IllegalArgumentException("Category of the types should 
be one of " +
-StringUtils.join(validTypeFilterCategories, ", "));
-}
-}
-
-/*
- * (non-Javadoc)
- * @see
- * org.apache.atlas.typesystem.types.cache.TypeCache#get(java.lang
- * .String)
- */
-@Override
-public IDataType get(String typeName) throws AtlasException {
-
-return types_.get(typeName);
-}
-
-/* (non-Javadoc)
- * @see 

[27/42] atlas git commit: ATLAS-2251: Remove TypeSystem and related implementation, to avoid unncessary duplicate of type details in cache

2017-11-12 Thread madhan
http://git-wip-us.apache.org/repos/asf/atlas/blob/435fe3fb/repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java
--
diff --git 
a/repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java
 
b/repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java
deleted file mode 100755
index 9eb695c..000
--- 
a/repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java
+++ /dev/null
@@ -1,821 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.atlas.services;
-
-import com.google.common.base.Preconditions;
-import com.google.common.collect.ImmutableList;
-import org.apache.atlas.AtlasClient;
-import org.apache.atlas.AtlasErrorCode;
-import org.apache.atlas.AtlasException;
-import org.apache.atlas.CreateUpdateEntitiesResult;
-import org.apache.atlas.EntityAuditEvent;
-import org.apache.atlas.RequestContext;
-import org.apache.atlas.exception.AtlasBaseException;
-import org.apache.atlas.ha.HAConfiguration;
-import org.apache.atlas.listener.ActiveStateChangeHandler;
-import org.apache.atlas.listener.ChangedTypeDefs;
-import org.apache.atlas.listener.EntityChangeListener;
-import org.apache.atlas.listener.TypeDefChangeListener;
-import org.apache.atlas.listener.TypesChangeListener;
-import org.apache.atlas.model.legacy.EntityResult;
-import org.apache.atlas.repository.MetadataRepository;
-import org.apache.atlas.repository.RepositoryException;
-import org.apache.atlas.repository.audit.EntityAuditRepository;
-import org.apache.atlas.repository.graph.GraphHelper;
-import org.apache.atlas.repository.typestore.ITypeStore;
-import org.apache.atlas.type.AtlasTypeUtil;
-import org.apache.atlas.typesystem.IReferenceableInstance;
-import org.apache.atlas.typesystem.IStruct;
-import org.apache.atlas.typesystem.ITypedReferenceableInstance;
-import org.apache.atlas.typesystem.ITypedStruct;
-import org.apache.atlas.typesystem.Referenceable;
-import org.apache.atlas.typesystem.Struct;
-import org.apache.atlas.typesystem.TypesDef;
-import org.apache.atlas.typesystem.exception.EntityNotFoundException;
-import org.apache.atlas.typesystem.exception.TypeNotFoundException;
-import org.apache.atlas.typesystem.json.InstanceSerialization;
-import org.apache.atlas.typesystem.json.TypesSerialization;
-import org.apache.atlas.typesystem.persistence.Id;
-import org.apache.atlas.typesystem.persistence.ReferenceableInstance;
-import org.apache.atlas.typesystem.types.*;
-import org.apache.atlas.typesystem.types.cache.TypeCache;
-import org.apache.atlas.utils.ParamChecker;
-import org.apache.commons.configuration.Configuration;
-import org.codehaus.jettison.json.JSONException;
-import org.codehaus.jettison.json.JSONObject;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-import org.springframework.stereotype.Component;
-
-import javax.inject.Inject;
-import javax.inject.Singleton;
-import java.util.ArrayList;
-import java.util.Collection;
-import java.util.Collections;
-import java.util.LinkedHashSet;
-import java.util.List;
-import java.util.Map;
-import java.util.Set;
-
-
-
-/**
- * Simple wrapper over TypeSystem and MetadataRepository services with hooks
- * for listening to changes to the repository.
- */
-@Singleton
-@Component
-@Deprecated
-public class DefaultMetadataService implements MetadataService, 
ActiveStateChangeHandler, TypeDefChangeListener {
-private enum OperationType {
-CREATE, UPDATE, DELETE
-};
-
-private static final Logger LOG = 
LoggerFactory.getLogger(DefaultMetadataService.class);
-private final short maxAuditResults;
-private static final String CONFIG_MAX_AUDIT_RESULTS = 
"atlas.audit.maxResults";
-private static final short DEFAULT_MAX_AUDIT_RESULTS = 1000;
-
-private final TypeSystem typeSystem;
-private final MetadataRepository repository;
-private final ITypeStore typeStore;
-
-private final Collection typeChangeListeners = new 
LinkedHashSet<>();
-private final Collection entityChangeListeners = new 
LinkedHashSet<>();
-
-private EntityAuditRepository auditRepository;
-
-@Inject
-public 

[02/42] atlas git commit: ATLAS-2251: Remove TypeSystem and related implementation, to avoid unncessary duplicate of type details in cache

2017-11-12 Thread madhan
http://git-wip-us.apache.org/repos/asf/atlas/blob/435fe3fb/webapp/src/test/java/org/apache/atlas/web/adapters/TestEntitiesREST.java
--
diff --git 
a/webapp/src/test/java/org/apache/atlas/web/adapters/TestEntitiesREST.java 
b/webapp/src/test/java/org/apache/atlas/web/adapters/TestEntitiesREST.java
index e753881..98df12b 100644
--- a/webapp/src/test/java/org/apache/atlas/web/adapters/TestEntitiesREST.java
+++ b/webapp/src/test/java/org/apache/atlas/web/adapters/TestEntitiesREST.java
@@ -18,7 +18,6 @@
 package org.apache.atlas.web.adapters;
 
 import org.apache.atlas.AtlasClient;
-import org.apache.atlas.RequestContext;
 import org.apache.atlas.RequestContextV1;
 import org.apache.atlas.TestModules;
 import org.apache.atlas.TestUtilsV2;
@@ -99,7 +98,6 @@ public class TestEntitiesREST {
 
 @AfterMethod
 public void cleanup() throws Exception {
-RequestContext.clear();
 RequestContextV1.clear();
 }
 

http://git-wip-us.apache.org/repos/asf/atlas/blob/435fe3fb/webapp/src/test/java/org/apache/atlas/web/adapters/TestEntityREST.java
--
diff --git 
a/webapp/src/test/java/org/apache/atlas/web/adapters/TestEntityREST.java 
b/webapp/src/test/java/org/apache/atlas/web/adapters/TestEntityREST.java
index b90ea64..ea6fe31 100644
--- a/webapp/src/test/java/org/apache/atlas/web/adapters/TestEntityREST.java
+++ b/webapp/src/test/java/org/apache/atlas/web/adapters/TestEntityREST.java
@@ -18,7 +18,6 @@
 package org.apache.atlas.web.adapters;
 
 import org.apache.atlas.TestModules;
-import org.apache.atlas.RequestContext;
 import org.apache.atlas.RequestContextV1;
 import org.apache.atlas.TestUtilsV2;
 import org.apache.atlas.model.instance.AtlasClassification;
@@ -77,7 +76,6 @@ public class TestEntityREST {
 
 @AfterMethod
 public void cleanup() throws Exception {
-RequestContext.clear();
 RequestContextV1.clear();
 }
 

http://git-wip-us.apache.org/repos/asf/atlas/blob/435fe3fb/webapp/src/test/java/org/apache/atlas/web/filters/AtlasAuthenticationKerberosFilterTest.java
--
diff --git 
a/webapp/src/test/java/org/apache/atlas/web/filters/AtlasAuthenticationKerberosFilterTest.java
 
b/webapp/src/test/java/org/apache/atlas/web/filters/AtlasAuthenticationKerberosFilterTest.java
index 5628b17..a3f5601 100644
--- 
a/webapp/src/test/java/org/apache/atlas/web/filters/AtlasAuthenticationKerberosFilterTest.java
+++ 
b/webapp/src/test/java/org/apache/atlas/web/filters/AtlasAuthenticationKerberosFilterTest.java
@@ -16,7 +16,7 @@
  */
 package org.apache.atlas.web.filters;
 
-import org.apache.atlas.RequestContext;
+import org.apache.atlas.RequestContextV1;
 import org.apache.atlas.web.security.BaseSecurityTest;
 import org.apache.atlas.web.service.EmbeddedServer;
 import org.apache.commons.configuration.PropertiesConfiguration;
@@ -111,7 +111,7 @@ public class AtlasAuthenticationKerberosFilterTest extends 
BaseSecurityTest {
 connection.connect();
 
 assertEquals(connection.getResponseCode(), 200);
-assertEquals(RequestContext.get().getUser(), TESTUSER);
+assertEquals(RequestContextV1.get().getUser(), TESTUSER);
 return null;
 }
 });

http://git-wip-us.apache.org/repos/asf/atlas/blob/435fe3fb/webapp/src/test/java/org/apache/atlas/web/integration/BaseResourceIT.java
--
diff --git 
a/webapp/src/test/java/org/apache/atlas/web/integration/BaseResourceIT.java 
b/webapp/src/test/java/org/apache/atlas/web/integration/BaseResourceIT.java
index 512750f..ab27612 100755
--- a/webapp/src/test/java/org/apache/atlas/web/integration/BaseResourceIT.java
+++ b/webapp/src/test/java/org/apache/atlas/web/integration/BaseResourceIT.java
@@ -19,8 +19,6 @@
 package org.apache.atlas.web.integration;
 
 import com.google.common.base.Preconditions;
-import com.google.common.collect.ImmutableList;
-import com.google.common.collect.ImmutableSet;
 import org.apache.atlas.ApplicationProperties;
 import org.apache.atlas.AtlasClient;
 import org.apache.atlas.AtlasClientV2;
@@ -32,26 +30,21 @@ import org.apache.atlas.model.instance.AtlasEntityHeader;
 import org.apache.atlas.model.instance.AtlasStruct;
 import org.apache.atlas.model.instance.EntityMutationResponse;
 import org.apache.atlas.model.instance.EntityMutations;
-import org.apache.atlas.model.typedef.AtlasClassificationDef;
-import org.apache.atlas.model.typedef.AtlasEntityDef;
-import org.apache.atlas.model.typedef.AtlasEnumDef;
-import org.apache.atlas.model.typedef.AtlasStructDef;
+import org.apache.atlas.model.typedef.*;
 import org.apache.atlas.model.typedef.AtlasStructDef.AtlasAttributeDef;
 import 

[36/42] atlas git commit: ATLAS-2251: Remove TypeSystem and related implementation, to avoid unncessary duplicate of type details in cache

2017-11-12 Thread madhan
http://git-wip-us.apache.org/repos/asf/atlas/blob/435fe3fb/notification/src/test/java/org/apache/atlas/notification/AbstractNotificationConsumerTest.java
--
diff --git 
a/notification/src/test/java/org/apache/atlas/notification/AbstractNotificationConsumerTest.java
 
b/notification/src/test/java/org/apache/atlas/notification/AbstractNotificationConsumerTest.java
index f313ddc..caa72ce 100644
--- 
a/notification/src/test/java/org/apache/atlas/notification/AbstractNotificationConsumerTest.java
+++ 
b/notification/src/test/java/org/apache/atlas/notification/AbstractNotificationConsumerTest.java
@@ -18,13 +18,15 @@
 
 package org.apache.atlas.notification;
 
-import com.google.gson.Gson;
-import com.google.gson.reflect.TypeToken;
 import org.apache.atlas.kafka.AtlasKafkaMessage;
+import org.apache.atlas.model.notification.AtlasNotificationMessage;
+import org.apache.atlas.type.AtlasType;
+import org.apache.atlas.model.notification.MessageVersion;
+import org.codehaus.jackson.type.TypeReference;
 import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
 import org.testng.annotations.Test;
 
-import java.lang.reflect.Type;
 import java.util.ArrayList;
 import java.util.LinkedList;
 import java.util.List;
@@ -41,8 +43,6 @@ import org.apache.kafka.common.TopicPartition;
  */
 public class AbstractNotificationConsumerTest {
 
-private static final Gson GSON = new Gson();
-
 @Test
 public void testReceive() throws Exception {
 Logger logger = mock(Logger.class);
@@ -54,27 +54,24 @@ public class AbstractNotificationConsumerTest {
 
 List jsonList = new LinkedList<>();
 
-jsonList.add(GSON.toJson(new AtlasNotificationMessage<>(new 
MessageVersion("1.0.0"), testMessage1)));
-jsonList.add(GSON.toJson(new AtlasNotificationMessage<>(new 
MessageVersion("1.0.0"), testMessage2)));
-jsonList.add(GSON.toJson(new AtlasNotificationMessage<>(new 
MessageVersion("1.0.0"), testMessage3)));
-jsonList.add(GSON.toJson(new AtlasNotificationMessage<>(new 
MessageVersion("1.0.0"), testMessage4)));
-
-Type notificationMessageType = new 
TypeToken(){}.getType();
+jsonList.add(AtlasType.toV1Json(new AtlasNotificationMessage<>(new 
MessageVersion("1.0.0"), testMessage1)));
+jsonList.add(AtlasType.toV1Json(new AtlasNotificationMessage<>(new 
MessageVersion("1.0.0"), testMessage2)));
+jsonList.add(AtlasType.toV1Json(new AtlasNotificationMessage<>(new 
MessageVersion("1.0.0"), testMessage3)));
+jsonList.add(AtlasType.toV1Json(new AtlasNotificationMessage<>(new 
MessageVersion("1.0.0"), testMessage4)));
 
-NotificationConsumer consumer =
-new TestNotificationConsumer<>(notificationMessageType, 
jsonList, logger);
+NotificationConsumer consumer = new 
TestNotificationConsumer(jsonList, logger);
 
 List messageList = consumer.receive();
 
 assertFalse(messageList.isEmpty());
 
-assertEquals(testMessage1, messageList.get(0).getMessage());
+assertEquals(messageList.get(0).getMessage(), testMessage1);
 
-assertEquals(testMessage2, messageList.get(1).getMessage());
+assertEquals(messageList.get(1).getMessage(), testMessage2);
 
-assertEquals(testMessage3, messageList.get(2).getMessage());
+assertEquals(messageList.get(2).getMessage(), testMessage3);
 
-assertEquals(testMessage4, messageList.get(3).getMessage());
+assertEquals(messageList.get(3).getMessage(), testMessage4);
 }
 
 @Test
@@ -88,20 +85,17 @@ public class AbstractNotificationConsumerTest {
 
 List jsonList = new LinkedList<>();
 
-String json1 = GSON.toJson(new AtlasNotificationMessage<>(new 
MessageVersion("1.0.0"), testMessage1));
-String json2 = GSON.toJson(new AtlasNotificationMessage<>(new 
MessageVersion("0.0.5"), testMessage2));
-String json3 = GSON.toJson(new AtlasNotificationMessage<>(new 
MessageVersion("0.5.0"), testMessage3));
-String json4 = GSON.toJson(testMessage4);
+String json1 = AtlasType.toV1Json(new AtlasNotificationMessage<>(new 
MessageVersion("1.0.0"), testMessage1));
+String json2 = AtlasType.toV1Json(new AtlasNotificationMessage<>(new 
MessageVersion("0.0.5"), testMessage2));
+String json3 = AtlasType.toV1Json(new AtlasNotificationMessage<>(new 
MessageVersion("0.5.0"), testMessage3));
+String json4 = AtlasType.toV1Json(testMessage4);
 
 jsonList.add(json1);
 jsonList.add(json2);
 jsonList.add(json3);
 jsonList.add(json4);
 
-Type notificationMessageType = new 
TypeToken(){}.getType();
-
-NotificationConsumer consumer =
-new TestNotificationConsumer<>(notificationMessageType, jsonList, 
logger);
+NotificationConsumer consumer = new 
TestNotificationConsumer(jsonList, 

[32/42] atlas git commit: ATLAS-2251: Remove TypeSystem and related implementation, to avoid unncessary duplicate of type details in cache

2017-11-12 Thread madhan
http://git-wip-us.apache.org/repos/asf/atlas/blob/435fe3fb/repository/src/main/java/org/apache/atlas/repository/converters/AtlasEntityFormatConverter.java
--
diff --git 
a/repository/src/main/java/org/apache/atlas/repository/converters/AtlasEntityFormatConverter.java
 
b/repository/src/main/java/org/apache/atlas/repository/converters/AtlasEntityFormatConverter.java
index 1ce6168..a529dc1 100644
--- 
a/repository/src/main/java/org/apache/atlas/repository/converters/AtlasEntityFormatConverter.java
+++ 
b/repository/src/main/java/org/apache/atlas/repository/converters/AtlasEntityFormatConverter.java
@@ -18,21 +18,20 @@
 package org.apache.atlas.repository.converters;
 
 import org.apache.atlas.AtlasErrorCode;
-import org.apache.atlas.AtlasException;
 import org.apache.atlas.exception.AtlasBaseException;
 import org.apache.atlas.model.TypeCategory;
 import org.apache.atlas.model.instance.AtlasClassification;
 import org.apache.atlas.model.instance.AtlasEntity;
 import org.apache.atlas.model.instance.AtlasEntity.Status;
 import org.apache.atlas.model.instance.AtlasObjectId;
+import org.apache.atlas.v1.model.instance.AtlasSystemAttributes;
+import org.apache.atlas.v1.model.instance.Id;
+import org.apache.atlas.v1.model.instance.Referenceable;
+import org.apache.atlas.v1.model.instance.Struct;
+import org.apache.atlas.type.AtlasClassificationType;
 import org.apache.atlas.type.AtlasEntityType;
 import org.apache.atlas.type.AtlasType;
 import org.apache.atlas.type.AtlasTypeRegistry;
-import org.apache.atlas.typesystem.IReferenceableInstance;
-import org.apache.atlas.typesystem.IStruct;
-import org.apache.atlas.typesystem.Referenceable;
-import org.apache.atlas.typesystem.persistence.Id;
-import org.apache.atlas.typesystem.persistence.Id.EntityState;
 import org.apache.commons.collections.CollectionUtils;
 import org.apache.commons.collections.MapUtils;
 import org.apache.commons.lang3.StringUtils;
@@ -57,37 +56,29 @@ public class AtlasEntityFormatConverter extends 
AtlasStructFormatConverter {
 if (v1Obj != null) {
 AtlasEntityType entityType = (AtlasEntityType) type;
 
-if (v1Obj instanceof IReferenceableInstance) {
-IReferenceableInstance entRef = (IReferenceableInstance) v1Obj;
+if (v1Obj instanceof Referenceable) {
+Referenceable entRef = (Referenceable)v1Obj;
 
-String guid = entRef.getId()._getId();
+String guid = entRef.getId().getId();
 
 if (!context.entityExists(guid)) {
-Map v1Attribs = null;
+entity = new AtlasEntity(entRef.getTypeName(), 
super.fromV1ToV2(entityType, entRef.getValues(), context));
 
-try {
-v1Attribs = entRef.getValuesMap();
-} catch (AtlasException excp) {
-LOG.error("IReferenceableInstance.getValuesMap() 
failed", excp);
-}
-
-entity = new AtlasEntity(entRef.getTypeName(),
- 
super.fromV1ToV2(entityType, v1Attribs, context));
-entity.setGuid(entRef.getId()._getId());
+entity.setGuid(entRef.getId().getId());
 entity.setStatus(convertState(entRef.getId().getState()));
-
entity.setCreatedBy(entRef.getSystemAttributes().createdBy);
-
entity.setCreateTime(entRef.getSystemAttributes().createdTime);
-
entity.setUpdatedBy(entRef.getSystemAttributes().modifiedBy);
-
entity.setUpdateTime(entRef.getSystemAttributes().modifiedTime);
-entity.setVersion((long) entRef.getId().version);
+
entity.setCreatedBy(entRef.getSystemAttributes().getCreatedBy());
+
entity.setCreateTime(entRef.getSystemAttributes().getCreatedTime());
+
entity.setUpdatedBy(entRef.getSystemAttributes().getModifiedBy());
+
entity.setUpdateTime(entRef.getSystemAttributes().getModifiedTime());
+entity.setVersion((long) entRef.getId().getVersion());
 
-if (CollectionUtils.isNotEmpty(entRef.getTraits())) {
+if (CollectionUtils.isNotEmpty(entRef.getTraitNames())) {
 List classifications = new 
ArrayList<>();
-AtlasFormatConverter traitConverter = 
converterRegistry.getConverter(TypeCategory.CLASSIFICATION);
+AtlasFormatConverter  traitConverter  = 
converterRegistry.getConverter(TypeCategory.CLASSIFICATION);
 
-for (String traitName : entRef.getTraits()) {
-IStruct trait = entRef.getTrait(traitName);
-AtlasType classifiType = 
typeRegistry.getType(traitName);
+  

[07/42] atlas git commit: ATLAS-2251: Remove TypeSystem and related implementation, to avoid unncessary duplicate of type details in cache

2017-11-12 Thread madhan
http://git-wip-us.apache.org/repos/asf/atlas/blob/435fe3fb/typesystem/src/main/scala/org/apache/atlas/typesystem/json/TypesSerialization.scala
--
diff --git 
a/typesystem/src/main/scala/org/apache/atlas/typesystem/json/TypesSerialization.scala
 
b/typesystem/src/main/scala/org/apache/atlas/typesystem/json/TypesSerialization.scala
deleted file mode 100755
index 4478a44..000
--- 
a/typesystem/src/main/scala/org/apache/atlas/typesystem/json/TypesSerialization.scala
+++ /dev/null
@@ -1,270 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.atlas.typesystem.json
-
-import java.text.SimpleDateFormat
-import com.google.common.collect.ImmutableList
-import org.apache.atlas.AtlasException
-import org.apache.atlas.AtlasConstants
-import org.apache.atlas.typesystem.TypesDef
-import org.apache.atlas.typesystem.types.DataTypes.{ArrayType, MapType, 
TypeCategory}
-import org.apache.atlas.typesystem.types._
-import org.json4s.JsonAST.JString
-import org.json4s._
-import org.json4s.native.Serialization._
-import com.google.common.collect.ImmutableSet
-
-
-/**
- * Module for serializing to/from Json.
- *
- * @example {{{
- *val j = TypesSerialization.toJson(typeSystem, "Employee", 
"Person", "Department", "SecurityClearance")
- *
- *val typesDef = TypesSerialization.fromJson(jsonStr)
- *typesDef.enumTypes.foreach( typeSystem.defineEnumType(_))
-
-  typeSystem.defineTypes(ImmutableList.copyOf(typesDef.structTypes.toArray),
-ImmutableList.copyOf(typesDef.traitTypes.toArray),
-ImmutableList.copyOf(typesDef.classTypes.toArray)
-  )
- * }}}
- *
- * @todo doesn't traverse includes directives. Includes are parsed into
- *   [[org.apache.atlas.tools.thrift.IncludeDef IncludeDef]] structures
- *   but are not traversed.
- * @todo mixing in [[scala.util.parsing.combinator.PackratParsers 
PackratParsers]] is a placeholder. Need to
- *   change specific grammar rules to `lazy val` and `Parser[Elem]` to 
`PackratParser[Elem]`. Will do based on
- *   performance analysis.
- * @todo Error reporting
- */
-object TypesSerialization {
-
-def toJsonValue(typ: IDataType[_])(implicit formats: Formats): JValue = {
-typ.getTypeCategory match {
-case TypeCategory.CLASS => {
-
Extraction.decompose(convertClassTypeToHierarchicalTypeDefinition(typ.asInstanceOf[ClassType]))
-}
-case TypeCategory.STRUCT => {
-
Extraction.decompose(convertStructTypeToStructDef(typ.asInstanceOf[StructType]))
-}
-case TypeCategory.TRAIT => {
-
Extraction.decompose(convertTraitTypeToHierarchicalTypeDefinition(typ.asInstanceOf[TraitType]))
-}
-case TypeCategory.ENUM => {
-
Extraction.decompose(convertEnumTypeToEnumTypeDef(typ.asInstanceOf[EnumType]))
-}
-case _ => JString(s"${typ.getName}")
-}
-}
-
-def toJson(ts: TypeSystem, typName: String): String = {
-toJson(ts, List(typName): _*)
-}
-
-def toJson(ts: TypeSystem, typNames: String*): String = {
-toJson(ts, (typ: IDataType[_]) => typNames.contains(typ.getName))
-}
-
-import scala.collection.JavaConversions._
-
-def toJson(ts: TypeSystem, typNames: java.util.List[String]): String = {
-toJson(ts, typNames.toIndexedSeq: _*)
-}
-
-val _formats = new DefaultFormats {
-override val dateFormatter = 
TypeSystem.getInstance().getDateFormat.asInstanceOf[SimpleDateFormat]
-override val typeHints = NoTypeHints
-}
-
-def toJson(ts: TypeSystem, export: IDataType[_] => Boolean): String = {
-implicit val formats = _formats + new MultiplicitySerializer
-
-val typsDef = convertToTypesDef(ts, export)
-
-writePretty(typsDef)
-}
-
-def fromJson(jsonStr: String): TypesDef = {
-implicit val formats = _formats + new MultiplicitySerializer
-
-read[TypesDef](jsonStr)
-}
-
-def toJson(typesDef : TypesDef) : String = {
-  implicit val formats = _formats + new MultiplicitySerializer
-  

[20/42] atlas git commit: ATLAS-2251: Remove TypeSystem and related implementation, to avoid unncessary duplicate of type details in cache

2017-11-12 Thread madhan
http://git-wip-us.apache.org/repos/asf/atlas/blob/435fe3fb/repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepositoryDeleteTestBase.java
--
diff --git 
a/repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepositoryDeleteTestBase.java
 
b/repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepositoryDeleteTestBase.java
deleted file mode 100644
index 0834601..000
--- 
a/repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepositoryDeleteTestBase.java
+++ /dev/null
@@ -1,1205 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.atlas.repository.graph;
-
-import com.google.common.collect.ImmutableList;
-import com.google.common.collect.ImmutableSet;
-import org.apache.atlas.AtlasClient;
-import org.apache.atlas.AtlasException;
-import org.apache.atlas.CreateUpdateEntitiesResult;
-import org.apache.atlas.RequestContext;
-import org.apache.atlas.TestModules;
-import org.apache.atlas.TestUtils;
-import org.apache.atlas.model.legacy.EntityResult;
-import org.apache.atlas.repository.Constants;
-import org.apache.atlas.repository.MetadataRepository;
-import org.apache.atlas.repository.RepositoryException;
-import org.apache.atlas.repository.graphdb.AtlasGraph;
-import org.apache.atlas.repository.graphdb.AtlasVertex;
-import org.apache.atlas.type.AtlasTypeRegistry;
-import org.apache.atlas.typesystem.IReferenceableInstance;
-import org.apache.atlas.typesystem.IStruct;
-import org.apache.atlas.typesystem.ITypedReferenceableInstance;
-import org.apache.atlas.typesystem.ITypedStruct;
-import org.apache.atlas.typesystem.Referenceable;
-import org.apache.atlas.typesystem.Struct;
-import org.apache.atlas.typesystem.TypesDef;
-import org.apache.atlas.typesystem.exception.EntityExistsException;
-import org.apache.atlas.typesystem.exception.EntityNotFoundException;
-import org.apache.atlas.typesystem.exception.NullRequiredAttributeException;
-import org.apache.atlas.typesystem.persistence.Id;
-import org.apache.atlas.typesystem.types.*;
-import org.apache.atlas.typesystem.types.utils.TypesUtil;
-import org.testng.Assert;
-import org.testng.annotations.AfterClass;
-import org.testng.annotations.BeforeClass;
-import org.testng.annotations.BeforeMethod;
-import org.testng.annotations.Guice;
-import org.testng.annotations.Test;
-
-import javax.inject.Inject;
-import java.util.ArrayList;
-import java.util.Arrays;
-import java.util.Collections;
-import java.util.HashMap;
-import java.util.List;
-import java.util.Map;
-
-import static org.apache.atlas.TestUtils.*;
-import static org.testng.Assert.assertEquals;
-import static org.testng.Assert.assertNotEquals;
-import static org.testng.Assert.assertTrue;
-import static org.testng.Assert.fail;
-
-/**
- * Test for GraphBackedMetadataRepository.deleteEntities
- *
- * Guice loads the dependencies and injects the necessary objects
- *
- */
-@Guice(modules = TestModules.TestOnlyModule.class)
-public abstract class GraphBackedMetadataRepositoryDeleteTestBase {
-
-protected MetadataRepository repositoryService;
-
-private TypeSystem typeSystem;
-
-private ClassType compositeMapOwnerType;
-
-private ClassType compositeMapValueType;
-
-@Inject
-AtlasGraph atlasGraph;
-
-@BeforeClass
-public void setUp() throws Exception {
-
-typeSystem = TypeSystem.getInstance();
-typeSystem.reset();
-
-new GraphBackedSearchIndexer(new AtlasTypeRegistry());
-final GraphBackedMetadataRepository delegate = new 
GraphBackedMetadataRepository(getDeleteHandler(typeSystem), atlasGraph);
-
-repositoryService = TestUtils.addTransactionWrapper(delegate);
-
-TestUtils.defineDeptEmployeeTypes(typeSystem);
-TestUtils.createHiveTypes(typeSystem);
-
-// Define type for map value.
-HierarchicalTypeDefinition mapValueDef = 
TypesUtil.createClassTypeDef("CompositeMapValue",
-ImmutableSet.of(),
-TypesUtil.createUniqueRequiredAttrDef(NAME, 
DataTypes.STRING_TYPE));
-
-// Define type with map where the value is a composite class 

[34/42] atlas git commit: ATLAS-2251: Remove TypeSystem and related implementation, to avoid unncessary duplicate of type details in cache

2017-11-12 Thread madhan
http://git-wip-us.apache.org/repos/asf/atlas/blob/435fe3fb/repository/src/main/java/org/apache/atlas/gremlin/GremlinExpressionFactory.java
--
diff --git 
a/repository/src/main/java/org/apache/atlas/gremlin/GremlinExpressionFactory.java
 
b/repository/src/main/java/org/apache/atlas/gremlin/GremlinExpressionFactory.java
deleted file mode 100644
index d603150..000
--- 
a/repository/src/main/java/org/apache/atlas/gremlin/GremlinExpressionFactory.java
+++ /dev/null
@@ -1,658 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS_METHOD IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.atlas.gremlin;
-
-import java.util.ArrayList;
-import java.util.Collections;
-import java.util.HashMap;
-import java.util.List;
-import java.util.Map;
-
-import org.apache.atlas.AtlasException;
-import org.apache.atlas.groovy.AbstractFunctionExpression;
-import org.apache.atlas.groovy.ArithmeticExpression;
-import org.apache.atlas.groovy.ArithmeticExpression.ArithmeticOperator;
-import org.apache.atlas.groovy.CastExpression;
-import org.apache.atlas.groovy.ClosureExpression;
-import org.apache.atlas.groovy.FieldExpression;
-import org.apache.atlas.groovy.FunctionCallExpression;
-import org.apache.atlas.groovy.GroovyExpression;
-import org.apache.atlas.groovy.IdentifierExpression;
-import org.apache.atlas.groovy.ListExpression;
-import org.apache.atlas.groovy.LiteralExpression;
-import org.apache.atlas.groovy.TraversalStepType;
-import org.apache.atlas.groovy.TypeCoersionExpression;
-import org.apache.atlas.groovy.VariableAssignmentExpression;
-import org.apache.atlas.query.GraphPersistenceStrategies;
-import org.apache.atlas.query.IntSequence;
-import org.apache.atlas.query.TypeUtils.FieldInfo;
-import org.apache.atlas.repository.graph.AtlasGraphProvider;
-import org.apache.atlas.repository.graphdb.AtlasEdgeDirection;
-import org.apache.atlas.repository.graphdb.GremlinVersion;
-import org.apache.atlas.typesystem.types.IDataType;
-import org.apache.atlas.typesystem.types.TypeSystem;
-import org.apache.atlas.typesystem.types.cache.TypeCache.TYPE_FILTER;
-import org.apache.atlas.util.AtlasRepositoryConfiguration;
-
-import com.google.common.collect.ImmutableList;
-
-/**
- * Factory to generate Groovy expressions representing Gremlin syntax that that
- * are independent of the specific version of Gremlin that is being used.
- *
- */
-public abstract class GremlinExpressionFactory {
-
-private static final String G_VARIABLE = "g";
-private static final String IT_VARIABLE = "it";
-
-protected static final String SET_CLASS = "Set";
-
-
-private static final String OBJECT_FIELD = "object";
-
-protected static final String V_METHOD = "V";
-protected static final String FILTER_METHOD = "filter";
-private static final String PATH_METHOD = "path";
-private static final String AS_METHOD = "as";
-private static final String IN_OPERATOR = "in";
-protected static final String HAS_METHOD = "has";
-protected static final String TO_LOWER_CASE_METHOD = "toLowerCase";
-protected static final String SELECT_METHOD = "select";
-protected static final String ORDER_METHOD = "order";
-protected static final String FILL_METHOD = "fill";
-protected static final String MATCHES = "matches";
-
-public static final GremlinExpressionFactory INSTANCE = 
AtlasGraphProvider.getGraphInstance()
-.getSupportedGremlinVersion() == GremlinVersion.THREE ? new 
Gremlin3ExpressionFactory()
-: new Gremlin2ExpressionFactory();
-
-/**
- * Returns the unqualified name of the class used in this version of 
gremlin to
- * represent Gremlin queries as they are being generated.
- * @return
- */
-public abstract String getTraversalExpressionClass();
-
-/**
- * Gets the expression to use as the parent when translating the loop
- * expression in a loop
- *
- * @param inputQry
- *the
- * @return
- */
-public abstract GroovyExpression getLoopExpressionParent(GroovyExpression 
inputQry);
-
-/**
- * Generates a loop expression.
- *
- * @param parent
- *the parent of the loop expression
-

[33/42] atlas git commit: ATLAS-2251: Remove TypeSystem and related implementation, to avoid unncessary duplicate of type details in cache

2017-11-12 Thread madhan
http://git-wip-us.apache.org/repos/asf/atlas/blob/435fe3fb/repository/src/main/java/org/apache/atlas/gremlin/optimizer/GremlinQueryOptimizer.java
--
diff --git 
a/repository/src/main/java/org/apache/atlas/gremlin/optimizer/GremlinQueryOptimizer.java
 
b/repository/src/main/java/org/apache/atlas/gremlin/optimizer/GremlinQueryOptimizer.java
deleted file mode 100644
index a0c08fd..000
--- 
a/repository/src/main/java/org/apache/atlas/gremlin/optimizer/GremlinQueryOptimizer.java
+++ /dev/null
@@ -1,262 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.atlas.gremlin.optimizer;
-
-import java.util.ArrayList;
-import java.util.List;
-
-import org.apache.atlas.gremlin.GremlinExpressionFactory;
-import org.apache.atlas.groovy.AbstractFunctionExpression;
-import org.apache.atlas.groovy.GroovyExpression;
-import org.apache.atlas.groovy.StatementListExpression;
-import org.apache.atlas.groovy.TraversalStepType;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-
-import com.google.common.annotations.VisibleForTesting;
-
-
-
-/**
- * Optimizer for gremlin queries.  This class provides a framework for 
applying optimizations
- * to gremlin queries.  Each optimization is implemented as a class that 
implements {@link GremlinOptimization}.
- *
- * The GremlinQueryOptimizer is the entry point for applying these 
optimizations.
- *
- *
- */
-public final class GremlinQueryOptimizer {
-
-private static final Logger LOGGER = 
LoggerFactory.getLogger(GremlinQueryOptimizer.class);
-
-
-private final List optimizations = new ArrayList<>();
-
-//Allows expression factory to be substituted in unit tests.
-private static volatile GremlinExpressionFactory FACTORY = 
GremlinExpressionFactory.INSTANCE;
-
-private static volatile GremlinQueryOptimizer INSTANCE = null;
-
-private GremlinQueryOptimizer() {
-
-}
-
-private void addOptimization(GremlinOptimization opt) {
-optimizations.add(opt);
-}
-
-public static GremlinQueryOptimizer getInstance() {
-if(INSTANCE == null) {
-synchronized(GremlinQueryOptimizer.class) {
-if(INSTANCE == null) {
-GremlinQueryOptimizer createdInstance = new 
GremlinQueryOptimizer();
-//The order here is important.  If there is an "or" nested 
within an "and",
-//that will not be found if ExpandOrsOptimization runs 
before ExpandAndsOptimization.
-createdInstance.addOptimization(new 
ExpandAndsOptimization(FACTORY));
-createdInstance.addOptimization(new 
ExpandOrsOptimization(FACTORY));
-INSTANCE = createdInstance;
-}
-}
-}
-return INSTANCE;
-}
-
-/**
- * For testing only
- */
-@VisibleForTesting
-public static void setExpressionFactory(GremlinExpressionFactory factory) {
-GremlinQueryOptimizer.FACTORY = factory;
-}
-
-/**
- * For testing only
- */
-@VisibleForTesting
-public static void reset() {
-INSTANCE = null;
-}
-
-/**
- * Optimizes the provided groovy expression.  Note that the optimization
- * is a destructive process.  The source GroovyExpression will be
- * modified as part of the optimization process.  This is done to avoid
- * expensive copying operations where possible.
- *
- * @param source what to optimize
- * @return the optimized query
- */
-public GroovyExpression optimize(GroovyExpression source) {
-LOGGER.debug("Optimizing gremlin query: " + source);
-OptimizationContext context = new OptimizationContext();
-GroovyExpression updatedExpression = source;
-for (GremlinOptimization opt : optimizations) {
-updatedExpression = optimize(updatedExpression, opt, context);
-LOGGER.debug("After "+ opt.getClass().getSimpleName() + ", query = 
" + updatedExpression);
-}
-
-StatementListExpression result = new StatementListExpression();
-result.addStatements(context.getInitialStatements());
-

[24/42] atlas git commit: ATLAS-2251: Remove TypeSystem and related implementation, to avoid unncessary duplicate of type details in cache

2017-11-12 Thread madhan
http://git-wip-us.apache.org/repos/asf/atlas/blob/435fe3fb/repository/src/test/java/org/apache/atlas/BaseRepositoryTest.java
--
diff --git a/repository/src/test/java/org/apache/atlas/BaseRepositoryTest.java 
b/repository/src/test/java/org/apache/atlas/BaseRepositoryTest.java
deleted file mode 100644
index 7c6b60b..000
--- a/repository/src/test/java/org/apache/atlas/BaseRepositoryTest.java
+++ /dev/null
@@ -1,428 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.atlas;
-
-import com.google.common.base.Preconditions;
-import com.google.common.collect.ImmutableList;
-import com.google.common.collect.ImmutableSet;
-import org.apache.atlas.repository.MetadataRepository;
-import org.apache.atlas.repository.graph.GraphBackedSearchIndexer;
-import org.apache.atlas.services.MetadataService;
-import org.apache.atlas.type.AtlasTypeRegistry;
-import org.apache.atlas.typesystem.ITypedReferenceableInstance;
-import org.apache.atlas.typesystem.Referenceable;
-import org.apache.atlas.typesystem.TypesDef;
-import org.apache.atlas.typesystem.json.TypesSerialization;
-import org.apache.atlas.typesystem.persistence.Id;
-import org.apache.atlas.typesystem.types.*;
-import org.apache.atlas.typesystem.types.utils.TypesUtil;
-import org.testng.annotations.Guice;
-
-import javax.inject.Inject;
-import java.util.ArrayList;
-import java.util.Date;
-import java.util.List;
-
-import static org.apache.atlas.AtlasClient.PROCESS_ATTRIBUTE_INPUTS;
-import static org.apache.atlas.AtlasClient.PROCESS_ATTRIBUTE_OUTPUTS;
-
-/**
- *  Base Class to set up hive types and instances for tests
- */
-@Guice(modules = TestModules.TestOnlyModule.class)
-public class BaseRepositoryTest {
-
-@Inject
-protected MetadataService metadataService;
-
-@Inject
-protected MetadataRepository repository;
-
-
-protected void setUp() throws Exception {
-//force graph initialization / built in type registration
-TestUtils.getGraph();
-setUpDefaultTypes();
-setUpTypes();
-TestUtils.getGraph().commit();
-new GraphBackedSearchIndexer(new AtlasTypeRegistry());
-TestUtils.resetRequestContext();
-setupInstances();
-TestUtils.getGraph().commit();
-TestUtils.dumpGraph(TestUtils.getGraph());
-}
-
-protected void tearDown() throws Exception {
-TypeSystem.getInstance().reset();
-}
-
-private void setUpTypes() throws Exception {
-TypesDef typesDef = createTypeDefinitions();
-String typesAsJSON = TypesSerialization.toJson(typesDef);
-metadataService.createType(typesAsJSON);
-}
-
-protected static final String DATABASE_TYPE = "hive_db";
-protected static final String HIVE_TABLE_TYPE = "hive_table";
-private static final String COLUMN_TYPE = "hive_column";
-private static final String HIVE_PROCESS_TYPE = "hive_process";
-private static final String STORAGE_DESC_TYPE = "StorageDesc";
-private static final String VIEW_TYPE = "View";
-private static final String PARTITION_TYPE = "hive_partition";
-protected static final String DATASET_SUBTYPE = "dataset_subtype";
-
-TypesDef createTypeDefinitions() {
-HierarchicalTypeDefinition dbClsDef = TypesUtil
-.createClassTypeDef(DATABASE_TYPE, null,
-TypesUtil.createUniqueRequiredAttrDef("name", 
DataTypes.STRING_TYPE),
-attrDef("description", DataTypes.STRING_TYPE), 
attrDef("locationUri", DataTypes.STRING_TYPE),
-attrDef("owner", DataTypes.STRING_TYPE), attrDef("createTime", 
DataTypes.LONG_TYPE));
-
-HierarchicalTypeDefinition columnClsDef = TypesUtil
-.createClassTypeDef(COLUMN_TYPE, null, attrDef("name", 
DataTypes.STRING_TYPE),
-attrDef("dataType", DataTypes.STRING_TYPE), attrDef("comment", 
DataTypes.STRING_TYPE));
-
-HierarchicalTypeDefinition storageDescClsDef = TypesUtil
-.createClassTypeDef(STORAGE_DESC_TYPE, null,
-attrDef("location", DataTypes.STRING_TYPE),
-attrDef("inputFormat", DataTypes.STRING_TYPE), 
attrDef("outputFormat", 

[30/42] atlas git commit: ATLAS-2251: Remove TypeSystem and related implementation, to avoid unncessary duplicate of type details in cache

2017-11-12 Thread madhan
http://git-wip-us.apache.org/repos/asf/atlas/blob/435fe3fb/repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java
--
diff --git 
a/repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java 
b/repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java
index ffe859b..3e60243 100755
--- 
a/repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java
+++ 
b/repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java
@@ -24,12 +24,15 @@ import com.google.common.collect.HashBiMap;
 import org.apache.atlas.ApplicationProperties;
 import org.apache.atlas.AtlasErrorCode;
 import org.apache.atlas.AtlasException;
-import org.apache.atlas.RequestContext;
+import org.apache.atlas.RequestContextV1;
 import org.apache.atlas.exception.AtlasBaseException;
 import org.apache.atlas.model.instance.AtlasEntity.Status;
 import org.apache.atlas.model.instance.AtlasObjectId;
 import org.apache.atlas.model.instance.AtlasRelationship;
 import org.apache.atlas.model.typedef.AtlasRelationshipDef;
+import org.apache.atlas.v1.model.instance.Id;
+import org.apache.atlas.v1.model.instance.Referenceable;
+import org.apache.atlas.type.AtlasStructType.AtlasAttribute;
 import 
org.apache.atlas.type.AtlasStructType.AtlasAttribute.AtlasRelationshipEdgeDirection;
 import org.apache.atlas.repository.Constants;
 import org.apache.atlas.repository.RepositoryException;
@@ -43,31 +46,11 @@ import 
org.apache.atlas.repository.store.graph.v1.AtlasGraphUtilsV1;
 import org.apache.atlas.type.AtlasEntityType;
 import org.apache.atlas.type.AtlasRelationshipType;
 import org.apache.atlas.type.AtlasType;
-import org.apache.atlas.typesystem.IReferenceableInstance;
-import org.apache.atlas.typesystem.ITypedInstance;
-import org.apache.atlas.typesystem.ITypedReferenceableInstance;
-import org.apache.atlas.typesystem.Referenceable;
-import org.apache.atlas.typesystem.exception.EntityNotFoundException;
-import org.apache.atlas.typesystem.exception.TypeNotFoundException;
-import org.apache.atlas.typesystem.json.InstanceSerialization;
-import org.apache.atlas.typesystem.persistence.Id;
-import org.apache.atlas.typesystem.persistence.ReferenceableInstance;
-import org.apache.atlas.typesystem.types.AttributeInfo;
-import org.apache.atlas.typesystem.types.ClassType;
-import org.apache.atlas.typesystem.types.DataTypes;
-import org.apache.atlas.typesystem.types.DataTypes.TypeCategory;
-import org.apache.atlas.typesystem.types.HierarchicalType;
-import org.apache.atlas.typesystem.types.IDataType;
-import org.apache.atlas.typesystem.types.Multiplicity;
-import org.apache.atlas.typesystem.types.TypeSystem;
-import org.apache.atlas.typesystem.types.ValueConversionException;
-import org.apache.atlas.typesystem.types.utils.TypesUtil;
+import org.apache.atlas.exception.EntityNotFoundException;
 import org.apache.atlas.util.AttributeValueMap;
 import org.apache.atlas.util.IndexedInstance;
-import org.apache.atlas.utils.ParamChecker;
 import org.apache.commons.collections.CollectionUtils;
 import org.apache.commons.lang.StringUtils;
-import org.codehaus.jettison.json.JSONArray;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
@@ -82,7 +65,6 @@ import java.util.List;
 import java.util.Map;
 import java.util.Objects;
 import java.util.Set;
-import java.util.Stack;
 import java.util.UUID;
 
 import static 
org.apache.atlas.type.AtlasStructType.AtlasAttribute.AtlasRelationshipEdgeDirection.BOTH;
@@ -97,8 +79,6 @@ public final class GraphHelper {
 private static final Logger LOG = 
LoggerFactory.getLogger(GraphHelper.class);
 public static final String EDGE_LABEL_PREFIX = "__";
 
-private static final TypeSystem typeSystem = TypeSystem.getInstance();
-
 public static final String RETRY_COUNT = "atlas.graph.storage.num.retries";
 public static final String RETRY_DELAY = 
"atlas.graph.storage.retry.sleeptime.ms";
 
@@ -143,7 +123,7 @@ public final class GraphHelper {
 }
 
 
-public AtlasVertex createVertexWithIdentity(ITypedReferenceableInstance 
typedInstance, Set superTypeNames) {
+public AtlasVertex createVertexWithIdentity(Referenceable typedInstance, 
Set superTypeNames) {
 final String guid = UUID.randomUUID().toString();
 
 final AtlasVertex vertexWithIdentity = 
createVertexWithoutIdentity(typedInstance.getTypeName(),
@@ -153,7 +133,7 @@ public final class GraphHelper {
 setProperty(vertexWithIdentity, Constants.GUID_PROPERTY_KEY, guid);
 
 // add version information
-setProperty(vertexWithIdentity, Constants.VERSION_PROPERTY_KEY, 
Long.valueOf(typedInstance.getId().version));
+setProperty(vertexWithIdentity, Constants.VERSION_PROPERTY_KEY, 
Long.valueOf(typedInstance.getId().getVersion()));
 
 return vertexWithIdentity;
 }
@@ -179,12 +159,12 @@ public final class GraphHelper {
 setProperty(vertexWithoutIdentity, 

[41/42] atlas git commit: ATLAS-2251: Remove TypeSystem and related implementation, to avoid unncessary duplicate of type details in cache

2017-11-12 Thread madhan
http://git-wip-us.apache.org/repos/asf/atlas/blob/435fe3fb/addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java
--
diff --git 
a/addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 
b/addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java
index f7404ae..773a1e8 100755
--- 
a/addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java
+++ 
b/addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java
@@ -19,18 +19,16 @@
 package org.apache.atlas.hive.hook;
 
 import com.google.common.base.Joiner;
-import com.google.common.collect.ImmutableList;
 import com.sun.jersey.api.client.ClientResponse;
 import org.apache.atlas.AtlasClient;
 import org.apache.atlas.AtlasServiceException;
 import org.apache.atlas.hive.HiveITBase;
 import org.apache.atlas.hive.bridge.HiveMetaStoreBridge;
 import org.apache.atlas.hive.model.HiveDataTypes;
-import org.apache.atlas.typesystem.Referenceable;
-import org.apache.atlas.typesystem.Struct;
-import org.apache.atlas.typesystem.persistence.Id;
-import org.apache.atlas.typesystem.types.TypeSystem;
-import org.apache.commons.lang.RandomStringUtils;
+import org.apache.atlas.model.typedef.AtlasBaseTypeDef;
+import org.apache.atlas.v1.model.instance.Id;
+import org.apache.atlas.v1.model.instance.Struct;
+import org.apache.atlas.v1.model.instance.Referenceable;
 import org.apache.commons.lang.StringUtils;
 import org.apache.hadoop.fs.Path;
 import org.apache.hadoop.hive.metastore.TableType;
@@ -182,7 +180,7 @@ public class HiveHookIT extends HiveITBase {
 private void verifyTimestamps(Referenceable ref, String property, long 
expectedTime) throws ParseException {
 //Verify timestamps.
 String createTimeStr = (String) ref.get(property);
-Date createDate = 
TypeSystem.getInstance().getDateFormat().parse(createTimeStr);
+Date createDate = AtlasBaseTypeDef.DATE_FORMATTER.parse(createTimeStr);
 Assert.assertNotNull(createTimeStr);
 
 if (expectedTime > 0) {
@@ -1262,7 +1260,7 @@ public class HiveHookIT extends HiveITBase {
 String guid2 = 
assertColumnIsRegistered(HiveMetaStoreBridge.getColumnQualifiedName(tbqn, 
"id_new"));
 assertEquals(guid2, guid);
 
-assertTrue(atlasClient.getEntity(guid2).getTraits().contains(trait));
+
assertTrue(atlasClient.getEntity(guid2).getTraitNames().contains(trait));
 }
 
 @Test
@@ -1349,15 +1347,15 @@ public class HiveHookIT extends HiveITBase {
 @Test
 public void testAlterTableBucketingClusterSort() throws Exception {
 String tableName = createTable();
-ImmutableList cols = ImmutableList.of("id");
+List cols = Collections.singletonList("id");
 runBucketSortQuery(tableName, 5, cols, cols);
 
-cols = ImmutableList.of("id", NAME);
+cols = Arrays.asList("id", NAME);
 runBucketSortQuery(tableName, 2, cols, cols);
 }
 
-private void runBucketSortQuery(String tableName, final int numBuckets,  
final ImmutableList bucketCols,
-final ImmutableList sortCols) 
throws Exception {
+private void runBucketSortQuery(String tableName, final int numBuckets,  
final List bucketCols,
+final List sortCols) throws 
Exception {
 final String fmtQuery = "alter table %s CLUSTERED BY (%s) SORTED BY 
(%s) INTO %s BUCKETS";
 String query = String.format(fmtQuery, tableName, 
stripListBrackets(bucketCols.toString()),
 stripListBrackets(sortCols.toString()), numBuckets);
@@ -1375,8 +1373,8 @@ public class HiveHookIT extends HiveITBase {
 }
 
 private void verifyBucketSortingProperties(Referenceable tableRef, int 
numBuckets,
-   ImmutableList 
bucketColNames,
-   ImmutableList  
sortcolNames) throws Exception {
+   List bucketColNames,
+   List  sortcolNames) 
throws Exception {
 Referenceable sdRef = (Referenceable) 
tableRef.get(HiveMetaStoreBridge.STORAGE_DESC);
 Assert.assertEquals(((scala.math.BigInt) 
sdRef.get(HiveMetaStoreBridge.STORAGE_NUM_BUCKETS)).intValue(),
 numBuckets);

http://git-wip-us.apache.org/repos/asf/atlas/blob/435fe3fb/addons/sqoop-bridge/pom.xml
--
diff --git a/addons/sqoop-bridge/pom.xml b/addons/sqoop-bridge/pom.xml
index d46d8a9..5a9305e 100644
--- a/addons/sqoop-bridge/pom.xml
+++ b/addons/sqoop-bridge/pom.xml
@@ -94,11 +94,6 @@
 compile
 
 
-   
-org.apache.atlas
-atlas-typesystem
-
-
 
 org.apache.atlas
 atlas-client-v1
@@ -155,13 +150,6 @@
 pom
  

[15/42] atlas git commit: ATLAS-2251: Remove TypeSystem and related implementation, to avoid unncessary duplicate of type details in cache

2017-11-12 Thread madhan
http://git-wip-us.apache.org/repos/asf/atlas/blob/435fe3fb/repository/src/test/java/org/apache/atlas/service/DefaultMetadataServiceTest.java
--
diff --git 
a/repository/src/test/java/org/apache/atlas/service/DefaultMetadataServiceTest.java
 
b/repository/src/test/java/org/apache/atlas/service/DefaultMetadataServiceTest.java
deleted file mode 100644
index e885b8c..000
--- 
a/repository/src/test/java/org/apache/atlas/service/DefaultMetadataServiceTest.java
+++ /dev/null
@@ -1,1381 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- * 
- * http://www.apache.org/licenses/LICENSE-2.0
- * 
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.atlas.service;
-
-import com.google.common.collect.ImmutableList;
-import com.google.common.collect.ImmutableSet;
-import com.google.inject.Inject;
-import org.apache.atlas.AtlasClient;
-import org.apache.atlas.AtlasException;
-import org.apache.atlas.CreateUpdateEntitiesResult;
-import org.apache.atlas.EntityAuditEvent;
-import org.apache.atlas.RequestContext;
-import org.apache.atlas.TestModules;
-import org.apache.atlas.TestUtils;
-import org.apache.atlas.discovery.graph.GraphBackedDiscoveryService;
-import org.apache.atlas.exception.AtlasBaseException;
-import org.apache.atlas.listener.ChangedTypeDefs;
-import org.apache.atlas.listener.EntityChangeListener;
-import org.apache.atlas.listener.TypeDefChangeListener;
-import org.apache.atlas.model.legacy.EntityResult;
-import org.apache.atlas.query.QueryParams;
-import org.apache.atlas.repository.Constants;
-import org.apache.atlas.repository.audit.EntityAuditRepository;
-import org.apache.atlas.repository.audit.HBaseBasedAuditRepository;
-import org.apache.atlas.repository.audit.HBaseTestUtils;
-import org.apache.atlas.repository.graph.AtlasGraphProvider;
-import org.apache.atlas.repository.graphdb.AtlasEdge;
-import org.apache.atlas.repository.graphdb.AtlasEdgeDirection;
-import org.apache.atlas.repository.graphdb.AtlasGraph;
-import org.apache.atlas.repository.graphdb.AtlasVertex;
-import org.apache.atlas.services.DefaultMetadataService;
-import org.apache.atlas.services.MetadataService;
-import org.apache.atlas.type.AtlasTypeUtil;
-import org.apache.atlas.typesystem.IReferenceableInstance;
-import org.apache.atlas.typesystem.IStruct;
-import org.apache.atlas.typesystem.ITypedReferenceableInstance;
-import org.apache.atlas.typesystem.Referenceable;
-import org.apache.atlas.typesystem.Struct;
-import org.apache.atlas.typesystem.TypesDef;
-import org.apache.atlas.typesystem.exception.EntityNotFoundException;
-import org.apache.atlas.typesystem.exception.TypeNotFoundException;
-import org.apache.atlas.typesystem.json.InstanceSerialization;
-import org.apache.atlas.typesystem.json.TypesSerialization;
-import org.apache.atlas.typesystem.persistence.Id;
-import org.apache.atlas.typesystem.types.AttributeDefinition;
-import org.apache.atlas.typesystem.types.ClassType;
-import org.apache.atlas.typesystem.types.DataTypes;
-import org.apache.atlas.typesystem.types.EnumValue;
-import org.apache.atlas.typesystem.types.HierarchicalTypeDefinition;
-import org.apache.atlas.typesystem.types.Multiplicity;
-import org.apache.atlas.typesystem.types.TypeSystem;
-import org.apache.atlas.typesystem.types.ValueConversionException;
-import org.apache.atlas.typesystem.types.cache.TypeCache;
-import org.apache.atlas.typesystem.types.utils.TypesUtil;
-import org.apache.atlas.utils.ParamChecker;
-import org.codehaus.jettison.json.JSONArray;
-import org.codehaus.jettison.json.JSONException;
-import org.codehaus.jettison.json.JSONObject;
-import org.testng.Assert;
-import org.testng.annotations.AfterTest;
-import org.testng.annotations.BeforeTest;
-import org.testng.annotations.Guice;
-import org.testng.annotations.Test;
-
-import java.util.ArrayList;
-import java.util.Arrays;
-import java.util.Collection;
-import java.util.Collections;
-import java.util.HashMap;
-import java.util.List;
-import java.util.Map;
-import java.util.concurrent.Callable;
-import java.util.concurrent.ExecutionException;
-import java.util.concurrent.ExecutorService;
-import java.util.concurrent.Executors;
-import java.util.concurrent.Future;
-
-import static org.apache.atlas.TestUtils.*;
-import static 

[42/42] atlas git commit: ATLAS-2251: Remove TypeSystem and related implementation, to avoid unncessary duplicate of type details in cache

2017-11-12 Thread madhan
ATLAS-2251: Remove TypeSystem and related implementation, to avoid unncessary 
duplicate of type details in cache


Project: http://git-wip-us.apache.org/repos/asf/atlas/repo
Commit: http://git-wip-us.apache.org/repos/asf/atlas/commit/435fe3fb
Tree: http://git-wip-us.apache.org/repos/asf/atlas/tree/435fe3fb
Diff: http://git-wip-us.apache.org/repos/asf/atlas/diff/435fe3fb

Branch: refs/heads/master
Commit: 435fe3fbad0dbb41297cc7af7986d3062704d7ee
Parents: 0abf84c
Author: Madhan Neethiraj <mad...@apache.org>
Authored: Fri Nov 3 22:45:05 2017 -0700
Committer: Madhan Neethiraj <mad...@apache.org>
Committed: Sun Nov 12 10:08:25 2017 -0800

--
 addons/falcon-bridge/pom.xml|   61 +-
 .../atlas/falcon/bridge/FalconBridge.java   |4 +-
 .../apache/atlas/falcon/hook/FalconHook.java|9 +-
 .../apache/atlas/falcon/hook/FalconHookIT.java  |   20 +-
 addons/hbase-bridge/pom.xml |   97 +-
 .../atlas/hbase/bridge/HBaseAtlasHook.java  |   47 +-
 .../hbase/model/HBaseOperationContext.java  |8 +-
 .../apache/atlas/hbase/HBaseAtlasHookIT.java|2 +-
 addons/hdfs-model/pom.xml   |   66 -
 addons/hive-bridge/pom.xml  |   56 +-
 .../atlas/hive/bridge/ColumnLineageUtils.java   |2 +-
 .../atlas/hive/bridge/HiveMetaStoreBridge.java  |   18 +-
 .../org/apache/atlas/hive/hook/HiveHook.java|   29 +-
 .../java/org/apache/atlas/hive/HiveITBase.java  |4 +-
 .../hive/bridge/HiveMetaStoreBridgeTest.java|4 +-
 .../hive/bridge/HiveMetastoreBridgeIT.java  |8 +-
 .../org/apache/atlas/hive/hook/HiveHookIT.java  |   26 +-
 addons/sqoop-bridge/pom.xml |   61 +-
 .../org/apache/atlas/sqoop/hook/SqoopHook.java  |9 +-
 addons/storm-bridge/pom.xml |  115 +-
 .../apache/atlas/storm/hook/StormAtlasHook.java |2 +-
 .../atlas/storm/hook/StormAtlasHookIT.java  |2 +-
 authorization/pom.xml   |7 +
 client/client-v1/pom.xml|9 +-
 .../main/java/org/apache/atlas/AtlasClient.java |   54 +-
 .../java/org/apache/atlas/EntityAuditEvent.java |   19 +-
 .../src/main/java/org/apache/atlas/SerDe.java   |   80 -
 .../java/org/apache/atlas/AtlasClientTest.java  |6 +-
 client/pom.xml  |   12 +-
 common/pom.xml  |1 +
 .../org/apache/atlas/AtlasConfiguration.java|   83 --
 .../org/apache/atlas/repository/Constants.java  |7 +
 graphdb/api/pom.xml |6 +-
 .../atlas/repository/graphdb/AtlasGraph.java|   12 +-
 graphdb/common/pom.xml  |6 +
 .../graphdb/janus/AtlasJanusGraph.java  |6 +-
 .../repository/graphdb/titan0/Titan0Graph.java  |6 +-
 .../repository/graphdb/titan1/Titan1Graph.java  |6 +-
 intg/pom.xml|   26 +-
 .../org/apache/atlas/AtlasConfiguration.java|   83 ++
 .../AtlasNotificationBaseMessage.java   |  208 +++
 .../notification/AtlasNotificationMessage.java  |  101 ++
 .../AtlasNotificationStringMessage.java |   82 ++
 .../model/notification/EntityNotification.java  |   86 ++
 .../model/notification/HookNotification.java|  103 ++
 .../model/notification/MessageVersion.java  |  170 +++
 .../java/org/apache/atlas/type/AtlasType.java   |  167 ++-
 .../apache/atlas/type/AtlasTypeRegistry.java|4 +-
 .../org/apache/atlas/type/AtlasTypeUtil.java|  157 +-
 .../atlas/typesystem/types/DataTypes.java   |   35 +
 .../model/instance/AtlasSystemAttributes.java   |  156 ++
 .../org/apache/atlas/v1/model/instance/Id.java  |  270 
 .../atlas/v1/model/instance/Referenceable.java  |  252 
 .../apache/atlas/v1/model/instance/Struct.java  |  211 +++
 .../model/lineage/DataSetLineageResponse.java   |   65 +
 .../atlas/v1/model/lineage/LineageResponse.java |   75 +
 .../atlas/v1/model/lineage/SchemaResponse.java  |  116 ++
 .../notification/EntityNotificationV1.java  |  231 +++
 .../model/notification/HookNotificationV1.java  |  357 +
 .../v1/model/typedef/AttributeDefinition.java   |  179 +++
 .../v1/model/typedef/ClassTypeDefinition.java   |   57 +
 .../v1/model/typedef/EnumTypeDefinition.java|  174 +++
 .../typedef/HierarchicalTypeDefinition.java |   96 ++
 .../atlas/v1/model/typedef/Multiplicity.java|  167 +++
 .../v1/model/typedef/StructTypeDefinition.java  |  119 ++
 .../v1/model/typedef/TraitTypeDefinition.java   |   51 +
 .../apache/atlas/v1/model/typedef/TypesDef.java |  118 ++
 .../v1/typesystem/types/utils/TypesUtil.java|  112 ++
 .../apache/atlas/TestRelationshipUtilsV2.java   |   26 +-
 .../test/java/org/apache/atlas/TestUtilsV2.java |  144 +-
 .../atlas/type/TestAtlasRelationshipType.java   |   14 +-
 .../test/resources/atlas-application.properties |  145 ++
 intg/src/test/reso

[17/42] atlas git commit: ATLAS-2251: Remove TypeSystem and related implementation, to avoid unncessary duplicate of type details in cache

2017-11-12 Thread madhan
http://git-wip-us.apache.org/repos/asf/atlas/blob/435fe3fb/repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasDeleteHandlerV1Test.java
--
diff --git 
a/repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasDeleteHandlerV1Test.java
 
b/repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasDeleteHandlerV1Test.java
deleted file mode 100644
index 718538a..000
--- 
a/repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasDeleteHandlerV1Test.java
+++ /dev/null
@@ -1,1141 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- * 
- * http://www.apache.org/licenses/LICENSE-2.0
- * 
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.atlas.repository.store.graph.v1;
-
-import com.google.common.collect.ImmutableList;
-import com.google.common.collect.ImmutableSet;
-import org.apache.atlas.AtlasException;
-import org.apache.atlas.RequestContextV1;
-import org.apache.atlas.TestUtils;
-import org.apache.atlas.TestUtilsV2;
-import org.apache.atlas.exception.AtlasBaseException;
-import org.apache.atlas.model.instance.AtlasEntity;
-import org.apache.atlas.model.instance.AtlasEntityHeader;
-import org.apache.atlas.model.instance.AtlasObjectId;
-import org.apache.atlas.model.instance.AtlasStruct;
-import org.apache.atlas.model.instance.EntityMutationResponse;
-import org.apache.atlas.model.instance.EntityMutations;
-import org.apache.atlas.model.typedef.AtlasClassificationDef;
-import org.apache.atlas.model.typedef.AtlasEntityDef;
-import org.apache.atlas.model.typedef.AtlasEnumDef;
-import org.apache.atlas.model.typedef.AtlasStructDef;
-import org.apache.atlas.model.typedef.AtlasTypesDef;
-import org.apache.atlas.repository.Constants;
-import org.apache.atlas.repository.graph.AtlasEdgeLabel;
-import org.apache.atlas.repository.graph.AtlasGraphProvider;
-import org.apache.atlas.repository.graph.GraphHelper;
-import org.apache.atlas.repository.graphdb.AtlasGraph;
-import org.apache.atlas.repository.graphdb.AtlasVertex;
-import 
org.apache.atlas.repository.store.bootstrap.AtlasTypeDefStoreInitializer;
-import org.apache.atlas.repository.store.graph.AtlasEntityStore;
-import org.apache.atlas.services.MetadataService;
-import org.apache.atlas.store.AtlasTypeDefStore;
-import org.apache.atlas.type.AtlasEntityType;
-import org.apache.atlas.type.AtlasTypeRegistry;
-import org.apache.atlas.type.AtlasTypeUtil;
-import org.apache.atlas.typesystem.IReferenceableInstance;
-import org.apache.atlas.typesystem.IStruct;
-import org.apache.atlas.typesystem.ITypedReferenceableInstance;
-import org.apache.atlas.typesystem.ITypedStruct;
-import org.apache.atlas.typesystem.Struct;
-import org.apache.atlas.typesystem.exception.EntityNotFoundException;
-import org.apache.atlas.typesystem.persistence.Id;
-import org.apache.atlas.typesystem.types.Multiplicity;
-import org.apache.atlas.typesystem.types.TraitType;
-import org.apache.atlas.typesystem.types.TypeSystem;
-import org.testng.Assert;
-import org.testng.annotations.AfterClass;
-import org.testng.annotations.BeforeClass;
-import org.testng.annotations.BeforeTest;
-import org.testng.annotations.Test;
-
-import javax.inject.Inject;
-import java.util.ArrayList;
-import java.util.Arrays;
-import java.util.Collections;
-import java.util.HashMap;
-import java.util.List;
-import java.util.Map;
-
-import static org.apache.atlas.TestUtils.*;
-import static org.mockito.Mockito.mock;
-import static org.testng.Assert.assertEquals;
-import static org.testng.Assert.assertNotEquals;
-import static org.testng.Assert.assertTrue;
-import static org.testng.Assert.fail;
-
-public abstract class AtlasDeleteHandlerV1Test {
-
-@Inject
-AtlasTypeRegistry typeRegistry;
-
-@Inject
-AtlasTypeDefStore typeDefStore;
-
-@Inject
-AtlasEntityStore entityStore;
-
-@Inject
-MetadataService metadataService;
-
-private AtlasEntityType compositeMapOwnerType;
-
-private AtlasEntityType compositeMapValueType;
-
-private TypeSystem typeSystem = TypeSystem.getInstance();
-
-@BeforeClass
-public void setUp() throws Exception {
-RequestContextV1.clear();
-RequestContextV1.get().setUser(TestUtilsV2.TEST_USER);
-
-metadataService = 

[16/42] atlas git commit: ATLAS-2251: Remove TypeSystem and related implementation, to avoid unncessary duplicate of type details in cache

2017-11-12 Thread madhan
http://git-wip-us.apache.org/repos/asf/atlas/blob/435fe3fb/repository/src/test/java/org/apache/atlas/repository/store/graph/v1/InverseReferenceUpdateV1Test.java
--
diff --git 
a/repository/src/test/java/org/apache/atlas/repository/store/graph/v1/InverseReferenceUpdateV1Test.java
 
b/repository/src/test/java/org/apache/atlas/repository/store/graph/v1/InverseReferenceUpdateV1Test.java
index 6475639..2c21638 100644
--- 
a/repository/src/test/java/org/apache/atlas/repository/store/graph/v1/InverseReferenceUpdateV1Test.java
+++ 
b/repository/src/test/java/org/apache/atlas/repository/store/graph/v1/InverseReferenceUpdateV1Test.java
@@ -20,7 +20,6 @@ package org.apache.atlas.repository.store.graph.v1;
 import com.google.common.collect.ImmutableList;
 import com.google.common.collect.ImmutableMap;
 import org.apache.atlas.RequestContextV1;
-import org.apache.atlas.TestUtils;
 import org.apache.atlas.TestUtilsV2;
 import org.apache.atlas.model.instance.AtlasEntity;
 import org.apache.atlas.model.instance.AtlasEntity.AtlasEntitiesWithExtInfo;
@@ -32,7 +31,6 @@ import org.apache.atlas.model.typedef.AtlasTypesDef;
 import org.apache.atlas.repository.graph.AtlasGraphProvider;
 import 
org.apache.atlas.repository.store.bootstrap.AtlasTypeDefStoreInitializer;
 import org.apache.atlas.repository.store.graph.AtlasEntityStore;
-import org.apache.atlas.services.MetadataService;
 import org.apache.atlas.store.AtlasTypeDefStore;
 import org.apache.atlas.type.AtlasEntityType;
 import org.apache.atlas.type.AtlasTypeRegistry;
@@ -48,11 +46,12 @@ import java.util.HashMap;
 import java.util.List;
 import java.util.Map;
 
-import static org.apache.atlas.TestUtils.NAME;
 import static org.testng.Assert.assertEquals;
 import static org.testng.Assert.assertNull;
 import static org.testng.Assert.assertTrue;
 
+import static org.apache.atlas.TestUtilsV2.NAME;
+
 /**
  * Test automatic inverse reference updating in V1 (V2?) code path.
  *
@@ -67,9 +66,6 @@ public abstract class InverseReferenceUpdateV1Test {
 @Inject
 AtlasEntityStore entityStore;
 
-@Inject
-MetadataService metadataService;
-
 private AtlasEntitiesWithExtInfo deptEntity;
 
 protected Map nameIdMap = new HashMap<>();
@@ -79,8 +75,6 @@ public abstract class InverseReferenceUpdateV1Test {
 RequestContextV1.clear();
 RequestContextV1.get().setUser(TestUtilsV2.TEST_USER);
 
-metadataService = TestUtils.addSessionCleanupWrapper(metadataService);
-
 AtlasTypesDef[] testTypesDefs = new AtlasTypesDef[] { 
TestUtilsV2.defineDeptEmployeeTypes(),
   
TestUtilsV2.defineInverseReferenceTestTypes()
 };
@@ -147,14 +141,14 @@ public abstract class InverseReferenceUpdateV1Test {
 public void testInverseReferenceAutoUpdate_NonCompositeManyToOne() throws 
Exception {
 AtlasEntityType bType = typeRegistry.getEntityTypeByName("B");
 AtlasEntity a1 = new AtlasEntity("A");
-a1.setAttribute(NAME, TestUtils.randomString());
+a1.setAttribute(NAME, TestUtilsV2.randomString());
 AtlasEntity a2 = new AtlasEntity("A");
-a2.setAttribute(NAME, TestUtils.randomString());
+a2.setAttribute(NAME, TestUtilsV2.randomString());
 AtlasEntity a3 = new AtlasEntity("A");
-a3.setAttribute(NAME, TestUtils.randomString());
+a3.setAttribute(NAME, TestUtilsV2.randomString());
 AtlasEntity b = new AtlasEntity("B");
 
-b.setAttribute(NAME, TestUtils.randomString());
+b.setAttribute(NAME, TestUtilsV2.randomString());
 AtlasEntitiesWithExtInfo atlasEntitiesWithExtInfo = new 
AtlasEntitiesWithExtInfo();
 atlasEntitiesWithExtInfo.addEntity(a1);
 atlasEntitiesWithExtInfo.addEntity(a2);
@@ -208,11 +202,11 @@ public abstract class InverseReferenceUpdateV1Test {
 public void testInverseReferenceAutoUpdate_NonComposite_OneToOne() throws 
Exception {
 AtlasEntityType bType = typeRegistry.getEntityTypeByName("B");
 AtlasEntity a1 = new AtlasEntity("A");
-a1.setAttribute(NAME, TestUtils.randomString());
+a1.setAttribute(NAME, TestUtilsV2.randomString());
 AtlasEntity a2 = new AtlasEntity("A");
-a2.setAttribute(NAME, TestUtils.randomString());
+a2.setAttribute(NAME, TestUtilsV2.randomString());
 AtlasEntity b = new AtlasEntity("B");
-b.setAttribute(NAME, TestUtils.randomString());
+b.setAttribute(NAME, TestUtilsV2.randomString());
 AtlasEntitiesWithExtInfo atlasEntitiesWithExtInfo = new 
AtlasEntitiesWithExtInfo();
 atlasEntitiesWithExtInfo.addEntity(a1);
 atlasEntitiesWithExtInfo.addEntity(a2);
@@ -261,15 +255,15 @@ public abstract class InverseReferenceUpdateV1Test {
 public void 

[25/42] atlas git commit: ATLAS-2251: Remove TypeSystem and related implementation, to avoid unncessary duplicate of type details in cache

2017-11-12 Thread madhan
http://git-wip-us.apache.org/repos/asf/atlas/blob/435fe3fb/repository/src/main/scala/org/apache/atlas/query/GremlinQuery.scala
--
diff --git 
a/repository/src/main/scala/org/apache/atlas/query/GremlinQuery.scala 
b/repository/src/main/scala/org/apache/atlas/query/GremlinQuery.scala
deleted file mode 100644
index 37015d8..000
--- a/repository/src/main/scala/org/apache/atlas/query/GremlinQuery.scala
+++ /dev/null
@@ -1,806 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.atlas.query
-
-import java.lang.Boolean
-import java.lang.Byte
-import java.lang.Double
-import java.lang.Float
-import java.lang.Integer
-import java.lang.Long
-import java.lang.Short
-import java.util.ArrayList
-
-import scala.collection.JavaConversions.asScalaBuffer
-import scala.collection.JavaConversions.bufferAsJavaList
-import scala.collection.mutable
-import scala.collection.mutable.ArrayBuffer
-
-
-import org.apache.atlas.gremlin.GremlinExpressionFactory
-import org.apache.atlas.gremlin.optimizer.GremlinQueryOptimizer
-import org.apache.atlas.groovy.CastExpression
-import org.apache.atlas.groovy.ClosureExpression
-import org.apache.atlas.groovy.LabeledExpression
-import org.apache.atlas.groovy.FunctionCallExpression
-import org.apache.atlas.groovy.GroovyExpression
-import org.apache.atlas.groovy.GroovyGenerationContext
-import org.apache.atlas.groovy.IdentifierExpression
-import org.apache.atlas.groovy.ListExpression
-import org.apache.atlas.groovy.LiteralExpression
-import org.apache.atlas.groovy.TraversalStepType
-import org.apache.atlas.query.Expressions.AliasExpression
-import org.apache.atlas.query.Expressions.ArithmeticExpression
-import org.apache.atlas.query.Expressions.BackReference
-import org.apache.atlas.query.Expressions.ClassExpression
-import org.apache.atlas.query.Expressions.ComparisonExpression
-import org.apache.atlas.query.Expressions.Expression
-import org.apache.atlas.query.Expressions.ExpressionException
-import org.apache.atlas.query.Expressions.FieldExpression
-import org.apache.atlas.query.Expressions.FilterExpression
-import org.apache.atlas.query.Expressions.InstanceExpression
-import org.apache.atlas.query.Expressions.LimitExpression
-import org.apache.atlas.query.Expressions.ListLiteral
-import org.apache.atlas.query.Expressions.Literal
-import org.apache.atlas.query.Expressions.LogicalExpression
-import org.apache.atlas.query.Expressions.LoopExpression
-import org.apache.atlas.query.Expressions.OrderExpression
-import org.apache.atlas.query.Expressions.PathExpression
-import org.apache.atlas.query.Expressions.SelectExpression
-import org.apache.atlas.query.Expressions.TraitExpression
-import org.apache.atlas.query.Expressions.TraitInstanceExpression
-import org.apache.atlas.query.Expressions.hasFieldLeafExpression
-import org.apache.atlas.query.Expressions.hasFieldUnaryExpression
-import org.apache.atlas.query.Expressions.id
-import org.apache.atlas.query.Expressions.isTraitLeafExpression
-import org.apache.atlas.query.Expressions.isTraitUnaryExpression
-import org.apache.atlas.repository.RepositoryException
-import org.apache.atlas.repository.graphdb.AtlasEdgeDirection
-import org.apache.atlas.typesystem.types.DataTypes
-import org.apache.atlas.typesystem.types.DataTypes.TypeCategory
-import org.apache.atlas.typesystem.types.IDataType
-import org.apache.atlas.typesystem.types.TypeSystem
-import org.joda.time.format.ISODateTimeFormat
-import org.apache.atlas.query.Expressions.GroupByExpression
-import org.apache.atlas.query.Expressions.MaxExpression
-import org.apache.atlas.query.Expressions.MinExpression
-import org.apache.atlas.query.Expressions.SumExpression
-import org.apache.atlas.query.Expressions.CountExpression
-
-import org.apache.atlas.util.AtlasRepositoryConfiguration
-import java.util.HashSet
-
-trait IntSequence {
-def next: Int
-}
-
-case class GremlinQuery(expr: Expression, queryStr: String, resultMaping: 
Map[String, (String, Int)]) {
-
-def hasSelectList = resultMaping != null
-
-def isPathExpression = expr.isInstanceOf[PathExpression]
-
-def isGroupBy = expr.isInstanceOf[GroupByExpression]
-}
-
-
-trait 

[2/2] atlas git commit: ATLAS-2251: fixed NPE in V1 to V2 entity conversion

2017-11-12 Thread madhan
ATLAS-2251: fixed NPE in V1 to V2 entity conversion


Project: http://git-wip-us.apache.org/repos/asf/atlas/repo
Commit: http://git-wip-us.apache.org/repos/asf/atlas/commit/6a1c4f4d
Tree: http://git-wip-us.apache.org/repos/asf/atlas/tree/6a1c4f4d
Diff: http://git-wip-us.apache.org/repos/asf/atlas/diff/6a1c4f4d

Branch: refs/heads/master
Commit: 6a1c4f4df2b906673a8cedd6abbd20761fcfb183
Parents: 41e5404
Author: Madhan Neethiraj <mad...@apache.org>
Authored: Sun Nov 12 14:39:13 2017 -0800
Committer: Madhan Neethiraj <mad...@apache.org>
Committed: Sun Nov 12 14:39:13 2017 -0800

--
 .../converters/AtlasEntityFormatConverter.java  | 12 
 1 file changed, 8 insertions(+), 4 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/atlas/blob/6a1c4f4d/repository/src/main/java/org/apache/atlas/repository/converters/AtlasEntityFormatConverter.java
--
diff --git 
a/repository/src/main/java/org/apache/atlas/repository/converters/AtlasEntityFormatConverter.java
 
b/repository/src/main/java/org/apache/atlas/repository/converters/AtlasEntityFormatConverter.java
index a529dc1..2229ece 100644
--- 
a/repository/src/main/java/org/apache/atlas/repository/converters/AtlasEntityFormatConverter.java
+++ 
b/repository/src/main/java/org/apache/atlas/repository/converters/AtlasEntityFormatConverter.java
@@ -66,10 +66,14 @@ public class AtlasEntityFormatConverter extends 
AtlasStructFormatConverter {
 
 entity.setGuid(entRef.getId().getId());
 entity.setStatus(convertState(entRef.getId().getState()));
-
entity.setCreatedBy(entRef.getSystemAttributes().getCreatedBy());
-
entity.setCreateTime(entRef.getSystemAttributes().getCreatedTime());
-
entity.setUpdatedBy(entRef.getSystemAttributes().getModifiedBy());
-
entity.setUpdateTime(entRef.getSystemAttributes().getModifiedTime());
+
+if (entRef.getSystemAttributes() != null) {
+
entity.setCreatedBy(entRef.getSystemAttributes().getCreatedBy());
+
entity.setCreateTime(entRef.getSystemAttributes().getCreatedTime());
+
entity.setUpdatedBy(entRef.getSystemAttributes().getModifiedBy());
+
entity.setUpdateTime(entRef.getSystemAttributes().getModifiedTime());
+}
+
 entity.setVersion((long) entRef.getId().getVersion());
 
 if (CollectionUtils.isNotEmpty(entRef.getTraitNames())) {



[1/2] atlas git commit: ATLAS-1836: fixed incorrect json in Area0 model

2017-11-12 Thread madhan
Repository: atlas
Updated Branches:
  refs/heads/master 435fe3fba -> 6a1c4f4df


ATLAS-1836: fixed incorrect json in Area0 model


Project: http://git-wip-us.apache.org/repos/asf/atlas/repo
Commit: http://git-wip-us.apache.org/repos/asf/atlas/commit/41e5404f
Tree: http://git-wip-us.apache.org/repos/asf/atlas/tree/41e5404f
Diff: http://git-wip-us.apache.org/repos/asf/atlas/diff/41e5404f

Branch: refs/heads/master
Commit: 41e5404f96395c8fc3f1f8045ba921f283a49cce
Parents: 435fe3f
Author: Madhan Neethiraj <mad...@apache.org>
Authored: Sun Nov 12 14:37:40 2017 -0800
Committer: Madhan Neethiraj <mad...@apache.org>
Committed: Sun Nov 12 14:37:40 2017 -0800

--
 addons/models/-Area0/0020-PropertyFacets_model.json | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/atlas/blob/41e5404f/addons/models/-Area0/0020-PropertyFacets_model.json
--
diff --git a/addons/models/-Area0/0020-PropertyFacets_model.json 
b/addons/models/-Area0/0020-PropertyFacets_model.json
index 0acdf48..fe92e26 100644
--- a/addons/models/-Area0/0020-PropertyFacets_model.json
+++ b/addons/models/-Area0/0020-PropertyFacets_model.json
@@ -70,7 +70,8 @@
   "isIndexable": true,
   "isOptional": true,
   "isUnique": false
-},
+}
+  ]
 }
   ]
 }



atlas git commit: ATLAS-2265: upgrade Jackson library version from 1.9.13 to 2.9.2, to be in sync with dependent Hadoop projects (#2)

2017-11-17 Thread madhan
Repository: atlas
Updated Branches:
  refs/heads/master 12d83b5bd -> 6dea2e4f6


ATLAS-2265: upgrade Jackson library version from 1.9.13 to 2.9.2, to be in sync 
with dependent Hadoop projects (#2)


Project: http://git-wip-us.apache.org/repos/asf/atlas/repo
Commit: http://git-wip-us.apache.org/repos/asf/atlas/commit/6dea2e4f
Tree: http://git-wip-us.apache.org/repos/asf/atlas/tree/6dea2e4f
Diff: http://git-wip-us.apache.org/repos/asf/atlas/diff/6dea2e4f

Branch: refs/heads/master
Commit: 6dea2e4f6318911328ff5d1edc52f66521826d6f
Parents: 12d83b5
Author: Madhan Neethiraj <mad...@apache.org>
Authored: Fri Nov 17 16:35:49 2017 -0800
Committer: Madhan Neethiraj <mad...@apache.org>
Committed: Fri Nov 17 16:35:49 2017 -0800

--
 graphdb/titan0/pom.xml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/atlas/blob/6dea2e4f/graphdb/titan0/pom.xml
--
diff --git a/graphdb/titan0/pom.xml b/graphdb/titan0/pom.xml
index 40535fe..28d051b 100644
--- a/graphdb/titan0/pom.xml
+++ b/graphdb/titan0/pom.xml
@@ -274,7 +274,7 @@
 META-INF/*.SF
 META-INF/*.DSA
 META-INF/*.RSA
-
org/codehaus/jackson/map/**
+
org/codehaus/jackson/**
 
com/fasterxml/jackson/**
 
 



[15/40] atlas git commit: ATLAS-2251: Remove TypeSystem and related implementation, to avoid unncessary duplicate of type details in cache

2017-11-05 Thread madhan
http://git-wip-us.apache.org/repos/asf/atlas/blob/0877e47c/repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasRelationshipStoreV1Test.java
--
diff --git 
a/repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasRelationshipStoreV1Test.java
 
b/repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasRelationshipStoreV1Test.java
deleted file mode 100644
index 8c58a92..000
--- 
a/repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasRelationshipStoreV1Test.java
+++ /dev/null
@@ -1,623 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.atlas.repository.store.graph.v1;
-
-import com.google.common.collect.ImmutableList;
-import org.apache.atlas.RequestContextV1;
-import org.apache.atlas.TestModules;
-import org.apache.atlas.TestUtilsV2;
-import org.apache.atlas.exception.AtlasBaseException;
-import org.apache.atlas.model.instance.AtlasEntity;
-import org.apache.atlas.model.instance.AtlasEntity.AtlasEntitiesWithExtInfo;
-import org.apache.atlas.model.instance.AtlasEntity.AtlasEntityWithExtInfo;
-import org.apache.atlas.model.instance.AtlasEntityHeader;
-import org.apache.atlas.model.instance.AtlasObjectId;
-import org.apache.atlas.model.instance.AtlasRelatedObjectId;
-import org.apache.atlas.model.instance.EntityMutationResponse;
-import org.apache.atlas.model.typedef.AtlasTypesDef;
-import org.apache.atlas.repository.graph.AtlasGraphProvider;
-import org.apache.atlas.repository.graph.GraphBackedSearchIndexer;
-import 
org.apache.atlas.repository.store.bootstrap.AtlasTypeDefStoreInitializer;
-import org.apache.atlas.repository.store.graph.AtlasEntityStore;
-import org.apache.atlas.repository.store.graph.AtlasRelationshipStore;
-import org.apache.atlas.store.AtlasTypeDefStore;
-import org.apache.atlas.type.AtlasEntityType;
-import org.apache.atlas.type.AtlasTypeRegistry;
-import org.apache.commons.collections.CollectionUtils;
-import org.testng.annotations.AfterClass;
-import org.testng.annotations.BeforeClass;
-import org.testng.annotations.BeforeTest;
-import org.testng.annotations.Guice;
-import org.testng.annotations.Test;
-
-import javax.inject.Inject;
-import java.util.ArrayList;
-import java.util.Collections;
-import java.util.HashMap;
-import java.util.List;
-import java.util.Map;
-
-import static org.apache.atlas.TestRelationshipUtilsV2.EMPLOYEE_TYPE;
-import static 
org.apache.atlas.TestRelationshipUtilsV2.getDepartmentEmployeeInstances;
-import static 
org.apache.atlas.TestRelationshipUtilsV2.getDepartmentEmployeeTypes;
-import static 
org.apache.atlas.TestRelationshipUtilsV2.getInverseReferenceTestTypes;
-import static org.apache.atlas.TestUtils.NAME;
-import static org.apache.atlas.type.AtlasTypeUtil.getAtlasObjectId;
-import static org.mockito.Mockito.mock;
-import static org.testng.Assert.assertEquals;
-import static org.testng.Assert.assertNotNull;
-import static org.testng.Assert.assertNull;
-import static org.testng.Assert.assertTrue;
-
-@Guice(modules = TestModules.TestOnlyModule.class)
-public abstract class AtlasRelationshipStoreV1Test {
-
-@Inject
-AtlasTypeRegistry typeRegistry;
-
-@Inject
-AtlasTypeDefStore typeDefStore;
-
-@Inject
-DeleteHandlerV1   deleteHandler;
-
-@Inject
-EntityGraphMapper graphMapper;
-
-AtlasEntityStore  entityStore;
-AtlasRelationshipStorerelationshipStore;
-AtlasEntityChangeNotifier mockChangeNotifier = 
mock(AtlasEntityChangeNotifier.class);
-
-protected Map employeeNameIdMap = new HashMap<>();
-
-@BeforeClass
-public void setUp() throws Exception {
-new GraphBackedSearchIndexer(typeRegistry);
-
-// create employee relationship types
-AtlasTypesDef employeeTypes = getDepartmentEmployeeTypes();
-typeDefStore.createTypesDef(employeeTypes);
-
-AtlasEntitiesWithExtInfo employeeInstances = 
getDepartmentEmployeeInstances();
-EntityMutationResponse response = entityStore.createOrUpdate(new 
AtlasEntityStream(employeeInstances), false);
-
-for (AtlasEntityHeader entityHeader : 

[02/40] atlas git commit: ATLAS-2251: Remove TypeSystem and related implementation, to avoid unncessary duplicate of type details in cache

2017-11-05 Thread madhan
http://git-wip-us.apache.org/repos/asf/atlas/blob/0877e47c/webapp/src/test/java/org/apache/atlas/notification/EntityNotificationIT.java
--
diff --git 
a/webapp/src/test/java/org/apache/atlas/notification/EntityNotificationIT.java 
b/webapp/src/test/java/org/apache/atlas/notification/EntityNotificationIT.java
index 7e94330..295d9ea 100644
--- 
a/webapp/src/test/java/org/apache/atlas/notification/EntityNotificationIT.java
+++ 
b/webapp/src/test/java/org/apache/atlas/notification/EntityNotificationIT.java
@@ -21,16 +21,12 @@ package org.apache.atlas.notification;
 import com.google.common.collect.ImmutableSet;
 import org.apache.atlas.AtlasClient;
 import org.apache.atlas.kafka.NotificationProvider;
+import org.apache.atlas.model.v1.instance.Id;
+import org.apache.atlas.model.v1.instance.Referenceable;
+import org.apache.atlas.model.v1.instance.Struct;
+import org.apache.atlas.model.v1.typedef.TraitTypeDefinition;
 import org.apache.atlas.notification.entity.EntityNotification;
-import org.apache.atlas.typesystem.IReferenceableInstance;
-import org.apache.atlas.typesystem.IStruct;
-import org.apache.atlas.typesystem.Referenceable;
-import org.apache.atlas.typesystem.Struct;
-import org.apache.atlas.typesystem.json.InstanceSerialization;
-import org.apache.atlas.typesystem.json.TypesSerialization$;
-import org.apache.atlas.typesystem.persistence.Id;
-import org.apache.atlas.typesystem.types.HierarchicalTypeDefinition;
-import org.apache.atlas.typesystem.types.TraitType;
+import org.apache.atlas.type.AtlasType;
 import org.apache.atlas.typesystem.types.utils.TypesUtil;
 import org.apache.atlas.web.integration.BaseResourceIT;
 import org.testng.annotations.BeforeClass;
@@ -119,7 +115,7 @@ public class EntityNotificationIT extends BaseResourceIT {
 createTrait(traitName, superTraitName);
 
 Struct traitInstance = new Struct(traitName);
-String traitInstanceJSON = InstanceSerialization.toJson(traitInstance, 
true);
+String traitInstanceJSON = AtlasType.toV1Json(traitInstance);
 LOG.debug("Trait instance = {}", traitInstanceJSON);
 
 final String guid = tableId._getId();
@@ -129,13 +125,13 @@ public class EntityNotificationIT extends BaseResourceIT {
 EntityNotification entityNotification = 
waitForNotification(notificationConsumer, MAX_WAIT_TIME,
 
newNotificationPredicate(EntityNotification.OperationType.TRAIT_ADD, 
HIVE_TABLE_TYPE_BUILTIN, guid));
 
-IReferenceableInstance entity = entityNotification.getEntity();
-assertTrue(entity.getTraits().contains(traitName));
+Referenceable entity = entityNotification.getEntity();
+assertTrue(entity.getTraitNames().contains(traitName));
 
-List allTraits = entityNotification.getAllTraits();
+List allTraits = entityNotification.getAllTraits();
 List allTraitNames = new LinkedList<>();
 
-for (IStruct struct : allTraits) {
+for (Struct struct : allTraits) {
 allTraitNames.add(struct.getTypeName());
 }
 assertTrue(allTraitNames.contains(traitName));
@@ -146,7 +142,7 @@ public class EntityNotificationIT extends BaseResourceIT {
 createTrait(anotherTraitName, superTraitName);
 
 traitInstance = new Struct(anotherTraitName);
-traitInstanceJSON = InstanceSerialization.toJson(traitInstance, true);
+traitInstanceJSON = AtlasType.toV1Json(traitInstance);
 LOG.debug("Trait instance = {}", traitInstanceJSON);
 
 atlasClientV1.addTrait(guid, traitInstance);
@@ -157,7 +153,7 @@ public class EntityNotificationIT extends BaseResourceIT {
 allTraits = entityNotification.getAllTraits();
 allTraitNames = new LinkedList<>();
 
-for (IStruct struct : allTraits) {
+for (Struct struct : allTraits) {
 allTraitNames.add(struct.getTypeName());
 }
 assertTrue(allTraitNames.contains(traitName));
@@ -174,17 +170,17 @@ public class EntityNotificationIT extends BaseResourceIT {
 EntityNotification entityNotification = 
waitForNotification(notificationConsumer, MAX_WAIT_TIME,
 
newNotificationPredicate(EntityNotification.OperationType.TRAIT_DELETE, 
HIVE_TABLE_TYPE_BUILTIN, guid));
 
-
assertFalse(entityNotification.getEntity().getTraits().contains(traitName));
+
assertFalse(entityNotification.getEntity().getTraitNames().contains(traitName));
 }
 
 
 // - helper methods ---
 
 private void createTrait(String traitName, String ... superTraitNames) 
throws Exception {
-HierarchicalTypeDefinition trait =
-TypesUtil.createTraitTypeDef(traitName, 
ImmutableSet.copyOf(superTraitNames));
+TraitTypeDefinition trait =
+TypesUtil.createTraitTypeDef(traitName, null, 
ImmutableSet.copyOf(superTraitNames));
 
-String 

[16/40] atlas git commit: ATLAS-2251: Remove TypeSystem and related implementation, to avoid unncessary duplicate of type details in cache

2017-11-05 Thread madhan
http://git-wip-us.apache.org/repos/asf/atlas/blob/0877e47c/repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStoreV1Test.java
--
diff --git 
a/repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStoreV1Test.java
 
b/repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStoreV1Test.java
deleted file mode 100644
index ef8bd2f..000
--- 
a/repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStoreV1Test.java
+++ /dev/null
@@ -1,1049 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.atlas.repository.store.graph.v1;
-
-import com.google.common.collect.ImmutableSet;
-import org.apache.atlas.AtlasErrorCode;
-import org.apache.atlas.AtlasException;
-import org.apache.atlas.TestModules;
-import org.apache.atlas.RequestContextV1;
-import org.apache.atlas.TestUtils;
-import org.apache.atlas.TestUtilsV2;
-import org.apache.atlas.exception.AtlasBaseException;
-import org.apache.atlas.model.instance.AtlasClassification;
-import org.apache.atlas.model.instance.AtlasEntity;
-import org.apache.atlas.model.instance.AtlasEntity.AtlasEntitiesWithExtInfo;
-import org.apache.atlas.model.instance.AtlasEntity.AtlasEntityExtInfo;
-import org.apache.atlas.model.instance.AtlasEntity.AtlasEntityWithExtInfo;
-import org.apache.atlas.model.instance.AtlasEntityHeader;
-import org.apache.atlas.model.instance.AtlasObjectId;
-import org.apache.atlas.model.instance.AtlasStruct;
-import org.apache.atlas.model.instance.EntityMutationResponse;
-import org.apache.atlas.model.instance.EntityMutations;
-import org.apache.atlas.model.instance.EntityMutations.EntityOperation;
-import org.apache.atlas.model.typedef.AtlasClassificationDef;
-import org.apache.atlas.model.typedef.AtlasEntityDef;
-import org.apache.atlas.model.typedef.AtlasStructDef;
-import org.apache.atlas.model.typedef.AtlasStructDef.AtlasAttributeDef;
-import org.apache.atlas.model.typedef.AtlasTypesDef;
-import org.apache.atlas.repository.graph.AtlasGraphProvider;
-import org.apache.atlas.repository.graph.GraphBackedSearchIndexer;
-import 
org.apache.atlas.repository.store.bootstrap.AtlasTypeDefStoreInitializer;
-import org.apache.atlas.repository.store.graph.AtlasEntityStore;
-import org.apache.atlas.services.MetadataService;
-import org.apache.atlas.store.AtlasTypeDefStore;
-import org.apache.atlas.type.AtlasArrayType;
-import org.apache.atlas.type.AtlasMapType;
-import org.apache.atlas.type.AtlasStructType;
-import org.apache.atlas.type.AtlasType;
-import org.apache.atlas.type.AtlasTypeRegistry;
-import org.apache.atlas.type.AtlasTypeUtil;
-import org.apache.commons.collections.CollectionUtils;
-import org.apache.commons.collections.MapUtils;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-import org.testng.Assert;
-import org.testng.annotations.AfterClass;
-import org.testng.annotations.BeforeClass;
-import org.testng.annotations.BeforeTest;
-import org.testng.annotations.Guice;
-import org.testng.annotations.Test;
-
-import javax.inject.Inject;
-import java.util.ArrayList;
-import java.util.Arrays;
-import java.util.Collections;
-import java.util.HashMap;
-import java.util.List;
-import java.util.Map;
-
-import static org.apache.atlas.TestUtils.COLUMNS_ATTR_NAME;
-import static org.apache.atlas.TestUtils.COLUMN_TYPE;
-import static org.apache.atlas.TestUtils.NAME;
-import static org.apache.atlas.TestUtils.randomString;
-import static org.apache.atlas.TestUtilsV2.STORAGE_DESC_TYPE;
-import static org.apache.atlas.TestUtilsV2.TABLE_TYPE;
-import static org.mockito.Mockito.mock;
-import static org.testng.Assert.assertEquals;
-import static org.testng.Assert.assertTrue;
-import static org.testng.Assert.fail;
-
-@Guice(modules = TestModules.TestOnlyModule.class)
-public class AtlasEntityStoreV1Test {
-private static final Logger LOG = 
LoggerFactory.getLogger(AtlasEntityStoreV1Test.class);
-
-@Inject
-AtlasTypeRegistry typeRegistry;
-
-@Inject
-AtlasTypeDefStore typeDefStore;
-
-AtlasEntityStore entityStore;
-
-@Inject
-MetadataService metadataService;
-
-@Inject
-

[36/40] atlas git commit: ATLAS-2251: Remove TypeSystem and related implementation, to avoid unncessary duplicate of type details in cache

2017-11-05 Thread madhan
http://git-wip-us.apache.org/repos/asf/atlas/blob/0877e47c/repository/src/main/java/org/apache/atlas/gremlin/Gremlin3ExpressionFactory.java
--
diff --git 
a/repository/src/main/java/org/apache/atlas/gremlin/Gremlin3ExpressionFactory.java
 
b/repository/src/main/java/org/apache/atlas/gremlin/Gremlin3ExpressionFactory.java
deleted file mode 100644
index b936695..000
--- 
a/repository/src/main/java/org/apache/atlas/gremlin/Gremlin3ExpressionFactory.java
+++ /dev/null
@@ -1,485 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.atlas.gremlin;
-
-import java.util.ArrayList;
-import java.util.Collections;
-import java.util.List;
-
-import org.apache.atlas.AtlasException;
-import org.apache.atlas.groovy.AbstractFunctionExpression;
-import org.apache.atlas.groovy.CastExpression;
-import org.apache.atlas.groovy.ClosureExpression;
-import org.apache.atlas.groovy.ComparisonExpression;
-import org.apache.atlas.groovy.ComparisonExpression.ComparisonOperator;
-import org.apache.atlas.groovy.ComparisonOperatorExpression;
-import org.apache.atlas.groovy.FieldExpression;
-import org.apache.atlas.groovy.FunctionCallExpression;
-import org.apache.atlas.groovy.GroovyExpression;
-import org.apache.atlas.groovy.IdentifierExpression;
-import org.apache.atlas.groovy.LiteralExpression;
-import org.apache.atlas.groovy.LogicalExpression;
-import org.apache.atlas.groovy.LogicalExpression.LogicalOperator;
-import org.apache.atlas.groovy.TernaryOperatorExpression;
-import org.apache.atlas.groovy.TraversalStepType;
-import org.apache.atlas.groovy.TypeCoersionExpression;
-import org.apache.atlas.query.GraphPersistenceStrategies;
-import org.apache.atlas.query.TypeUtils.FieldInfo;
-import org.apache.atlas.repository.graph.AtlasGraphProvider;
-import org.apache.atlas.repository.graphdb.AtlasGraph;
-import org.apache.atlas.typesystem.types.AttributeInfo;
-import org.apache.atlas.typesystem.types.IDataType;
-
-/**
- * Generates gremlin query expressions using Gremlin 3 syntax.
- *
- */
-public class Gremlin3ExpressionFactory extends GremlinExpressionFactory {
-
-
-
-private static final String VERTEX_LIST_CLASS = "List";
-private static final String VERTEX_ARRAY_CLASS = "Vertex[]";
-private static final String OBJECT_ARRAY_CLASS = "Object[]";
-private static final String VERTEX_CLASS = "Vertex";
-private static final String FUNCTION_CLASS = "Function";
-
-private static final String VALUE_METHOD = "value";
-private static final String IS_PRESENT_METHOD = "isPresent";
-private static final String MAP_METHOD = "map";
-private static final String VALUES_METHOD = "values";
-private static final String GET_METHOD = "get";
-private static final String OR_ELSE_METHOD = "orElse";
-private static final String PROPERTY_METHOD = "property";
-private static final String BY_METHOD = "by";
-private static final String EQ_METHOD = "eq";
-private static final String EMIT_METHOD = "emit";
-private static final String TIMES_METHOD = "times";
-private static final String REPEAT_METHOD = "repeat";
-private static final String RANGE_METHOD = "range";
-private static final String LAST_METHOD = "last";
-private static final String TO_STRING_METHOD = "toString";
-
-private static final GroovyExpression EMPTY_STRING_EXPRESSION = new 
LiteralExpression("");
-
-@Override
-public GroovyExpression generateLogicalExpression(GroovyExpression parent, 
String operator,
-  List 
operands) {
-return new FunctionCallExpression(TraversalStepType.FILTER, parent, 
operator, operands);
-}
-
-@Override
-public GroovyExpression generateBackReferenceExpression(GroovyExpression 
parent, boolean inSelect, String alias) {
-if (inSelect) {
-return getFieldInSelect();
-} else {
-return new 
FunctionCallExpression(TraversalStepType.MAP_TO_ELEMENT, parent, SELECT_METHOD, 
new LiteralExpression(alias));
-}
-}
-
-@Override
-public GroovyExpression typeTestExpression(GraphPersistenceStrategies s, 
String typeName, 

[14/40] atlas git commit: ATLAS-2251: Remove TypeSystem and related implementation, to avoid unncessary duplicate of type details in cache

2017-11-05 Thread madhan
http://git-wip-us.apache.org/repos/asf/atlas/blob/0877e47c/repository/src/test/java/org/apache/atlas/repository/typestore/GraphBackedTypeStoreTest.java
--
diff --git 
a/repository/src/test/java/org/apache/atlas/repository/typestore/GraphBackedTypeStoreTest.java
 
b/repository/src/test/java/org/apache/atlas/repository/typestore/GraphBackedTypeStoreTest.java
deleted file mode 100755
index 6e3dabb..000
--- 
a/repository/src/test/java/org/apache/atlas/repository/typestore/GraphBackedTypeStoreTest.java
+++ /dev/null
@@ -1,256 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.atlas.repository.typestore;
-
-import com.google.common.collect.ImmutableList;
-import com.google.common.collect.ImmutableSet;
-import org.apache.atlas.AtlasException;
-import org.apache.atlas.TestModules;
-import org.apache.atlas.TestUtils;
-import org.apache.atlas.repository.RepositoryException;
-import org.apache.atlas.repository.graph.GraphHelper;
-import org.apache.atlas.repository.graphdb.AtlasEdge;
-import org.apache.atlas.repository.graphdb.AtlasEdgeDirection;
-import org.apache.atlas.repository.graphdb.AtlasGraph;
-import org.apache.atlas.repository.graphdb.AtlasVertex;
-import org.apache.atlas.typesystem.TypesDef;
-import org.apache.atlas.typesystem.types.*;
-import org.apache.atlas.typesystem.types.utils.TypesUtil;
-import org.testng.Assert;
-import org.testng.annotations.AfterClass;
-import org.testng.annotations.BeforeClass;
-import org.testng.annotations.Guice;
-import org.testng.annotations.Test;
-
-import javax.inject.Inject;
-import java.util.Collections;
-import java.util.Iterator;
-import java.util.List;
-import java.util.Map;
-
-import static 
org.apache.atlas.typesystem.types.utils.TypesUtil.createClassTypeDef;
-import static 
org.apache.atlas.typesystem.types.utils.TypesUtil.createOptionalAttrDef;
-import static 
org.apache.atlas.typesystem.types.utils.TypesUtil.createRequiredAttrDef;
-import static 
org.apache.atlas.typesystem.types.utils.TypesUtil.createStructTypeDef;
-
-@Guice(modules = TestModules.TestOnlyModule.class)
-public class GraphBackedTypeStoreTest {
-
-private static final String DESCRIPTION = "_description";
-
-@Inject
-private ITypeStore typeStore;
-
-private TypeSystem ts;
-
-@BeforeClass
-public void setUp() throws Exception {
-ts = TypeSystem.getInstance();
-ts.reset();
-TestUtils.defineDeptEmployeeTypes(ts);
-}
-
-@AfterClass
-public void tearDown() throws Exception {
-ts.reset();
-//AtlasGraphProvider.cleanup();
-}
-
-
-@Test
-public void testStore() throws AtlasException {
-ImmutableList typeNames = ts.getTypeNames();
-typeStore.store(ts, typeNames);
-dumpGraph();
-}
-
-@Test(dependsOnMethods = "testStore")
-public void testRestoreType() throws Exception {
-TypesDef typesDef = typeStore.restoreType("Manager");
-verifyRestoredClassType(typesDef, "Manager");
-}
-
-private void dumpGraph() {
-AtlasGraph graph = TestUtils.getGraph();
-for (AtlasVertex v : graph.getVertices()) {
-System.out.println("v = " + GraphHelper.vertexString(v));
-for (AtlasEdge e : v.getEdges(AtlasEdgeDirection.OUT)) {
-System.out.println("e = " + GraphHelper.edgeString(e));
-}
-}
-}
-
-@Test(dependsOnMethods = "testStore")
-public void testRestore() throws Exception {
-TypesDef types = typeStore.restore();
-
-//validate enum
-List enumTypes = types.enumTypesAsJavaList();
-Assert.assertEquals(1, enumTypes.size());
-EnumTypeDefinition orgLevel = enumTypes.get(0);
-Assert.assertEquals(orgLevel.name, "OrgLevel");
-Assert.assertEquals(orgLevel.description, "OrgLevel"+DESCRIPTION);
-Assert.assertEquals(orgLevel.enumValues.length, 2);
-EnumValue enumValue = orgLevel.enumValues[0];
-Assert.assertEquals(enumValue.value, "L1");
-Assert.assertEquals(enumValue.ordinal, 1);
-
-//validate class
-List structTypes = 

[1/5] atlas git commit: Restoring removed legacy REST APIs

2017-11-05 Thread madhan
Repository: atlas
Updated Branches:
  refs/heads/ATLAS-2251 10c19eac7 -> 3f44770da


Restoring removed legacy REST APIs


Project: http://git-wip-us.apache.org/repos/asf/atlas/repo
Commit: http://git-wip-us.apache.org/repos/asf/atlas/commit/da4cf2dd
Tree: http://git-wip-us.apache.org/repos/asf/atlas/tree/da4cf2dd
Diff: http://git-wip-us.apache.org/repos/asf/atlas/diff/da4cf2dd

Branch: refs/heads/ATLAS-2251
Commit: da4cf2dd485f90717fbc535c1fa1aa95030586fa
Parents: 0877e47
Author: Madhan Neethiraj <mad...@apache.org>
Authored: Sun Nov 5 12:29:56 2017 -0800
Committer: Madhan Neethiraj <mad...@apache.org>
Committed: Sun Nov 5 12:31:45 2017 -0800

--
 .../web/resources/DataSetLineageResource.java   | 185 +
 .../atlas/web/resources/LineageResource.java| 207 ++
 .../resources/MetadataDiscoveryResource.java| 405 +++
 3 files changed, 797 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/atlas/blob/da4cf2dd/webapp/src/main/java/org/apache/atlas/web/resources/DataSetLineageResource.java
--
diff --git 
a/webapp/src/main/java/org/apache/atlas/web/resources/DataSetLineageResource.java
 
b/webapp/src/main/java/org/apache/atlas/web/resources/DataSetLineageResource.java
new file mode 100644
index 000..278d7ca
--- /dev/null
+++ 
b/webapp/src/main/java/org/apache/atlas/web/resources/DataSetLineageResource.java
@@ -0,0 +1,185 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.atlas.web.resources;
+
+import org.apache.atlas.AtlasClient;
+import org.apache.atlas.utils.AtlasPerfTracer;
+import org.apache.atlas.web.util.Servlets;
+import org.codehaus.jettison.json.JSONObject;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.springframework.stereotype.Service;
+
+import javax.inject.Inject;
+import javax.inject.Singleton;
+import javax.servlet.http.HttpServletRequest;
+import javax.ws.rs.*;
+import javax.ws.rs.core.Context;
+import javax.ws.rs.core.Response;
+
+/**
+ * Jersey Resource for Hive Table Lineage.
+ */
+@Path("lineage/hive")
+@Singleton
+@Service
+@Deprecated
+public class DataSetLineageResource {
+
+private static final Logger LOG = 
LoggerFactory.getLogger(DataSetLineageResource.class);
+private static final Logger PERF_LOG = 
AtlasPerfTracer.getPerfLogger("rest.DataSetLineageResource");
+
+/**
+ * Created by the Guice ServletModule and injected with the
+ * configured LineageService.
+ *
+ */
+@Inject
+public DataSetLineageResource() {
+}
+
+/**
+ * Returns the inputs graph for a given entity.
+ *
+ * @param tableName table name
+ */
+@GET
+@Path("table/{tableName}/inputs/graph")
+@Consumes(Servlets.JSON_MEDIA_TYPE)
+@Produces(Servlets.JSON_MEDIA_TYPE)
+public Response inputsGraph(@Context HttpServletRequest request, 
@PathParam("tableName") String tableName) {
+if (LOG.isDebugEnabled()) {
+LOG.debug("==> DataSetLineageResource.inputsGraph({})", tableName);
+}
+
+AtlasPerfTracer perf = null;
+
+try {
+if (AtlasPerfTracer.isPerfTraceEnabled(PERF_LOG)) {
+perf = AtlasPerfTracer.getPerfTracer(PERF_LOG, 
"DataSetLineageResource.inputsGraph(tableName=" + tableName + ")");
+}
+
+final String jsonResult = ""; // TODO-typeSystem-removal: 
lineageService.getInputsGraph(tableName);
+
+JSONObject response = new JSONObject();
+response.put(AtlasClient.REQUEST_ID, Servlets.getRequestId());
+response.put("tableName", tableName);
+response.put(AtlasClient.RESULTS, new JSONObject(jsonResult));
+
+return Response.ok(response).build();
+} catch (IllegalArgumentException e) {
+LOG.error("Unable to get lineage inputs graph for table {}", 
tableName, e);
+throw new WebApplicationEx

[3/5] atlas git commit: ATLAS-2251: relocated legacy classes in intg module

2017-11-05 Thread madhan
http://git-wip-us.apache.org/repos/asf/atlas/blob/eacf8513/intg/src/main/java/org/apache/atlas/v1/model/instance/AtlasSystemAttributes.java
--
diff --git 
a/intg/src/main/java/org/apache/atlas/v1/model/instance/AtlasSystemAttributes.java
 
b/intg/src/main/java/org/apache/atlas/v1/model/instance/AtlasSystemAttributes.java
new file mode 100644
index 000..43eca0b
--- /dev/null
+++ 
b/intg/src/main/java/org/apache/atlas/v1/model/instance/AtlasSystemAttributes.java
@@ -0,0 +1,123 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.atlas.v1.model.instance;
+
+import org.codehaus.jackson.annotate.JsonAutoDetect;
+import org.codehaus.jackson.annotate.JsonIgnoreProperties;
+import org.codehaus.jackson.map.annotate.JsonSerialize;
+
+import javax.xml.bind.annotation.XmlAccessType;
+import javax.xml.bind.annotation.XmlAccessorType;
+import javax.xml.bind.annotation.XmlRootElement;
+import java.io.Serializable;
+import java.util.Date;
+import java.util.Objects;
+
+import static org.codehaus.jackson.annotate.JsonAutoDetect.Visibility.NONE;
+import static 
org.codehaus.jackson.annotate.JsonAutoDetect.Visibility.PUBLIC_ONLY;
+
+
+@JsonAutoDetect(getterVisibility=PUBLIC_ONLY, setterVisibility=PUBLIC_ONLY, 
fieldVisibility=NONE)
+@JsonSerialize(include=JsonSerialize.Inclusion.NON_NULL)
+@JsonIgnoreProperties(ignoreUnknown=true)
+@XmlRootElement
+@XmlAccessorType(XmlAccessType.PROPERTY)
+public class AtlasSystemAttributes implements Serializable {
+private static final long serialVersionUID = 1L;
+
+private String createdBy;
+private String modifiedBy;
+private Date   createdTime;
+private Date   modifiedTime;
+
+
+public AtlasSystemAttributes() {
+}
+
+public AtlasSystemAttributes(AtlasSystemAttributes that) {
+if (that != null) {
+this.createdBy= that.createdBy;
+this.modifiedBy   = that.modifiedBy;
+this.createdTime  = that.createdTime;
+this.modifiedTime = that.modifiedTime;
+}
+}
+
+public AtlasSystemAttributes(String createdBy, String modifiedBy, Date 
createdTime, Date modifiedTime){
+this.createdBy= createdBy;
+this.modifiedBy   = modifiedBy;
+this.createdTime  = createdTime;
+this.modifiedTime = modifiedTime;
+}
+
+public String getCreatedBy(){
+return createdBy;
+}
+
+public void setCreatedBy(String createdBy) {
+this.createdBy = createdBy;
+}
+
+public String getModifiedBy(){
+return modifiedBy;
+}
+
+public void setModifiedBy(String modifiedBy) {
+this.modifiedBy = modifiedBy;
+}
+
+public Date getCreatedTime(){
+return createdTime;
+}
+
+public void setCreatedTime(Date createdTime) {
+this.createdTime = createdTime;
+}
+
+public Date getModifiedTime(){
+return modifiedTime;
+}
+
+public void setModifiedTime(Date modifiedTime) {
+this.modifiedTime = modifiedTime;
+}
+
+
+@Override
+public boolean equals(Object o) {
+if (this == o) {
+return true;
+}
+if (o == null || getClass() != o.getClass()) {
+return false;
+}
+
+AtlasSystemAttributes obj = (AtlasSystemAttributes) o;
+
+return Objects.equals(createdBy, obj.createdBy) &&
+   Objects.equals(modifiedBy, obj.modifiedBy) &&
+   Objects.equals(createdTime, obj.createdTime) &&
+   Objects.equals(modifiedTime, obj.modifiedTime);
+}
+
+@Override
+public int hashCode() {
+return Objects.hash(createdBy, modifiedBy, createdTime, modifiedTime);
+}
+}

http://git-wip-us.apache.org/repos/asf/atlas/blob/eacf8513/intg/src/main/java/org/apache/atlas/v1/model/instance/Id.java
--
diff --git a/intg/src/main/java/org/apache/atlas/v1/model/instance/Id.java 
b/intg/src/main/java/org/apache/atlas/v1/model/instance/Id.java
new file mode 100644
index 000..1b250f0
--- /dev/null
+++ b/intg/src/main/java/org/apache/atlas/v1/model/instance/Id.java
@@ -0,0 

[5/5] atlas git commit: Merge branch 'ATLAS-2251' of https://git-wip-us.apache.org/repos/asf/atlas into ATLAS-2251

2017-11-05 Thread madhan
Merge branch 'ATLAS-2251' of https://git-wip-us.apache.org/repos/asf/atlas into 
ATLAS-2251


Project: http://git-wip-us.apache.org/repos/asf/atlas/repo
Commit: http://git-wip-us.apache.org/repos/asf/atlas/commit/3f44770d
Tree: http://git-wip-us.apache.org/repos/asf/atlas/tree/3f44770d
Diff: http://git-wip-us.apache.org/repos/asf/atlas/diff/3f44770d

Branch: refs/heads/ATLAS-2251
Commit: 3f44770da188f270330230594344e9a263b5a5d4
Parents: eacf851 10c19ea
Author: Madhan Neethiraj <mad...@apache.org>
Authored: Sun Nov 5 12:41:54 2017 -0800
Committer: Madhan Neethiraj <mad...@apache.org>
Committed: Sun Nov 5 12:41:54 2017 -0800

--

--




[32/40] atlas git commit: ATLAS-2251: Remove TypeSystem and related implementation, to avoid unncessary duplicate of type details in cache

2017-11-05 Thread madhan
http://git-wip-us.apache.org/repos/asf/atlas/blob/0877e47c/repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepository.java
--
diff --git 
a/repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepository.java
 
b/repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepository.java
deleted file mode 100755
index 74886b5..000
--- 
a/repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepository.java
+++ /dev/null
@@ -1,505 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.atlas.repository.graph;
-
-import com.google.common.base.Preconditions;
-import org.apache.atlas.AtlasException;
-import org.apache.atlas.CreateUpdateEntitiesResult;
-import org.apache.atlas.GraphTransactionInterceptor;
-import org.apache.atlas.RequestContext;
-import org.apache.atlas.annotation.GraphTransaction;
-import org.apache.atlas.model.instance.GuidMapping;
-import org.apache.atlas.model.legacy.EntityResult;
-import org.apache.atlas.repository.Constants;
-import org.apache.atlas.repository.MetadataRepository;
-import org.apache.atlas.repository.RepositoryException;
-import org.apache.atlas.repository.graphdb.AtlasEdge;
-import org.apache.atlas.repository.graphdb.AtlasGraph;
-import org.apache.atlas.repository.graphdb.AtlasGraphQuery;
-import org.apache.atlas.repository.graphdb.AtlasVertex;
-import org.apache.atlas.typesystem.ITypedReferenceableInstance;
-import org.apache.atlas.typesystem.ITypedStruct;
-import org.apache.atlas.typesystem.exception.EntityExistsException;
-import org.apache.atlas.typesystem.exception.EntityNotFoundException;
-import org.apache.atlas.typesystem.exception.TraitNotFoundException;
-import org.apache.atlas.typesystem.persistence.Id;
-import org.apache.atlas.typesystem.types.AttributeInfo;
-import org.apache.atlas.typesystem.types.ClassType;
-import org.apache.atlas.typesystem.types.DataTypes;
-import org.apache.atlas.typesystem.types.IDataType;
-import org.apache.atlas.typesystem.types.TypeSystem;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-import org.springframework.stereotype.Component;
-
-import javax.inject.Inject;
-import javax.inject.Singleton;
-import java.util.*;
-
-/**
- * An implementation backed by a Graph database provided
- * as a Graph Service.
- */
-@Singleton
-@Component
-@Deprecated
-public class GraphBackedMetadataRepository implements MetadataRepository {
-
-private static final Logger LOG = 
LoggerFactory.getLogger(GraphBackedMetadataRepository.class);
-
-private static TypeSystem typeSystem = TypeSystem.getInstance();
-
-private static final GraphHelper graphHelper = GraphHelper.getInstance();
-
-private DeleteHandler deleteHandler;
-
-private final AtlasGraph atlasGraph;
-private final GraphToTypedInstanceMapper graphToInstanceMapper;
-
-@Inject
-public GraphBackedMetadataRepository(DeleteHandler deleteHandler, 
AtlasGraph atlasGraph) {
-this.atlasGraph = atlasGraph;
-this.graphToInstanceMapper = new 
GraphToTypedInstanceMapper(atlasGraph);
-this.deleteHandler = deleteHandler;
-}
-
-public GraphToTypedInstanceMapper getGraphToInstanceMapper() {
-return graphToInstanceMapper;
-}
-
-@Override
-public String getTypeAttributeName() {
-return Constants.ENTITY_TYPE_PROPERTY_KEY;
-}
-
-@Override
-public String getStateAttributeName() {
-return Constants.STATE_PROPERTY_KEY;
-}
-
-/**
- * Returns the property key used to store super type names.
- *
- * @return property key used to store super type names.
- */
-@Override
-public String getSuperTypeAttributeName() {
-return Constants.SUPER_TYPES_PROPERTY_KEY;
-}
-
-public String getIdAttributeName() {
-return Constants.GUID_PROPERTY_KEY;
-}
-
-@Override
-public String getVersionAttributeName() {
-return Constants.VERSION_PROPERTY_KEY;
-}
-
-@Override
-public String getTraitLabel(IDataType dataType, String traitName) {
-return 

[04/40] atlas git commit: ATLAS-2251: Remove TypeSystem and related implementation, to avoid unncessary duplicate of type details in cache

2017-11-05 Thread madhan
http://git-wip-us.apache.org/repos/asf/atlas/blob/0877e47c/typesystem/src/test/java/org/apache/atlas/typesystem/types/cache/DefaultTypeCacheTest.java
--
diff --git 
a/typesystem/src/test/java/org/apache/atlas/typesystem/types/cache/DefaultTypeCacheTest.java
 
b/typesystem/src/test/java/org/apache/atlas/typesystem/types/cache/DefaultTypeCacheTest.java
deleted file mode 100644
index 5c397dd..000
--- 
a/typesystem/src/test/java/org/apache/atlas/typesystem/types/cache/DefaultTypeCacheTest.java
+++ /dev/null
@@ -1,450 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.atlas.typesystem.types.cache;
-
-import com.google.common.collect.ImmutableList;
-import com.google.common.collect.ImmutableSet;
-import org.apache.atlas.AtlasException;
-import org.apache.atlas.typesystem.types.ClassType;
-import org.apache.atlas.typesystem.types.DataTypes;
-import org.apache.atlas.typesystem.types.DataTypes.TypeCategory;
-import org.apache.atlas.typesystem.types.EnumType;
-import org.apache.atlas.typesystem.types.EnumValue;
-import org.apache.atlas.typesystem.types.IDataType;
-import org.apache.atlas.typesystem.types.StructType;
-import org.apache.atlas.typesystem.types.TraitType;
-import org.apache.atlas.typesystem.types.TypeSystem;
-import org.apache.atlas.typesystem.types.utils.TypesUtil;
-import org.testng.annotations.BeforeClass;
-import org.testng.annotations.BeforeMethod;
-import org.testng.annotations.Test;
-
-import java.util.ArrayList;
-import java.util.Arrays;
-import java.util.Collection;
-import java.util.Collections;
-import java.util.HashMap;
-import java.util.List;
-
-import static org.testng.Assert.assertEquals;
-import static org.testng.Assert.assertFalse;
-import static org.testng.Assert.assertNotNull;
-import static org.testng.Assert.assertNull;
-import static org.testng.Assert.assertTrue;
-import static org.testng.Assert.fail;
-
-/**
- * Tests functional behavior of {@link DefaultTypeCache}
- */
-@SuppressWarnings("rawtypes")
-public class DefaultTypeCacheTest {
-
-private String CLASSTYPE_CUSTOMER = "Customer";
-private String STRUCTTYPE_ADDRESS = "Address";
-private String TRAITTYPE_PRIVILEGED = "Privileged";
-private String ENUMTYPE_SHIPPING = "Shipping";
-
-private String UNKNOWN_TYPE = "UndefinedType";
-
-private ClassType customerType;
-private StructType addressType;
-private TraitType privilegedTrait;
-private EnumType shippingEnum;
-
-private DefaultTypeCache cache;
-
-@BeforeClass
-public void onetimeSetup() throws Exception {
-
-// init TypeSystem
-TypeSystem ts = TypeSystem.getInstance().reset();
-
-// Customer ClassType
-customerType = ts.defineClassType(TypesUtil
-.createClassTypeDef(CLASSTYPE_CUSTOMER, ImmutableSet.of(),
-TypesUtil.createRequiredAttrDef("name", DataTypes.STRING_TYPE),
-TypesUtil.createRequiredAttrDef("id", DataTypes.LONG_TYPE)));
-
-// Address StructType
-addressType = ts.defineStructType(STRUCTTYPE_ADDRESS, true,
-TypesUtil.createRequiredAttrDef("first line", 
DataTypes.STRING_TYPE),
-TypesUtil.createOptionalAttrDef("second line", 
DataTypes.STRING_TYPE),
-TypesUtil.createRequiredAttrDef("city", DataTypes.STRING_TYPE),
-TypesUtil.createRequiredAttrDef("pincode", 
DataTypes.INT_TYPE));
-
-// Privileged TraitType
-privilegedTrait = ts.defineTraitType(TypesUtil
-.createTraitTypeDef(TRAITTYPE_PRIVILEGED, 
ImmutableSet.of(),
-TypesUtil.createRequiredAttrDef("category", 
DataTypes.INT_TYPE)));
-
-// Shipping EnumType
-shippingEnum = 
ts.defineEnumType(TypesUtil.createEnumTypeDef(ENUMTYPE_SHIPPING,
-new EnumValue("Domestic", 1), new EnumValue("International", 2)));
-}
-
-@BeforeMethod
-public void eachTestSetup() throws Exception {
-
-cache = new DefaultTypeCache();
-
-cache.put(customerType);
-cache.put(addressType);
-cache.put(privilegedTrait);
-cache.put(shippingEnum);
-}
-
-@Test

[23/40] atlas git commit: ATLAS-2251: Remove TypeSystem and related implementation, to avoid unncessary duplicate of type details in cache

2017-11-05 Thread madhan
http://git-wip-us.apache.org/repos/asf/atlas/blob/0877e47c/repository/src/test/java/org/apache/atlas/query/QueryProcessorTest.java
--
diff --git 
a/repository/src/test/java/org/apache/atlas/query/QueryProcessorTest.java 
b/repository/src/test/java/org/apache/atlas/query/QueryProcessorTest.java
deleted file mode 100644
index 57bc74f..000
--- a/repository/src/test/java/org/apache/atlas/query/QueryProcessorTest.java
+++ /dev/null
@@ -1,157 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.atlas.query;
-
-import com.google.common.collect.ImmutableSet;
-import org.apache.atlas.AtlasException;
-import org.apache.atlas.typesystem.types.ClassType;
-import org.apache.atlas.typesystem.types.DataTypes;
-import org.apache.atlas.typesystem.types.DataTypes.TypeCategory;
-import org.apache.atlas.typesystem.types.HierarchicalTypeDefinition;
-import org.apache.atlas.typesystem.types.IDataType;
-import org.apache.atlas.typesystem.types.TypeSystem;
-import org.apache.atlas.typesystem.types.cache.DefaultTypeCache;
-import org.testng.annotations.Test;
-import scala.util.Either;
-import scala.util.parsing.combinator.Parsers;
-
-import java.util.HashSet;
-import java.util.Set;
-
-import static 
org.apache.atlas.typesystem.types.utils.TypesUtil.createClassTypeDef;
-import static 
org.apache.atlas.typesystem.types.utils.TypesUtil.createRequiredAttrDef;
-import static org.testng.Assert.assertFalse;
-import static org.testng.Assert.assertTrue;
-
-/**
- * Tests the logic for skipping type cache lookup for things that
- * cannot be types.
- *
- */
-public class QueryProcessorTest {
-
-
-@Test
-public void testAliasesNotTreatedAsTypes() throws Exception {
-
-ValidatingTypeCache tc = findTypeLookupsDuringQueryParsing("hive_db as 
inst where inst.name=\"Reporting\" select inst as id, inst.name");
-assertTrue(tc.wasTypeRequested("hive_db"));
-assertFalse(tc.wasTypeRequested("inst"));
-assertFalse(tc.wasTypeRequested("name"));
-
-}
-
-
-@Test
-public void testFieldInComparisionNotTreatedAsType() throws Exception {
-
-//test when the IdExpression is on the left, on the right, and on both 
sides of the ComparsionExpression
-ValidatingTypeCache tc = findTypeLookupsDuringQueryParsing("hive_db 
where name=\"Reporting\" or \"Reporting\" = name or name=name");
-assertTrue(tc.wasTypeRequested("hive_db"));
-assertFalse(tc.wasTypeRequested("name"));
-
-}
-
-
-@Test
-public void testFieldInArithmeticExprNotTreatedAsType() throws Exception {
-
-//test when the IdExpression is on the left, on the right, and on both 
sides of the ArithmeticExpression
-ValidatingTypeCache tc = findTypeLookupsDuringQueryParsing("hive_db 
where (tableCount + 3) > (tableCount + tableCount) select (3 + tableCount) as 
updatedCount");
-
-assertTrue(tc.wasTypeRequested("hive_db"));
-assertFalse(tc.wasTypeRequested("tableCount"));
-assertFalse(tc.wasTypeRequested("updatedCount"));
-
-}
-
-@Test
-public void testFieldInSelectListWithAlasNotTreatedAsType() throws 
Exception {
-
-ValidatingTypeCache tc = findTypeLookupsDuringQueryParsing("hive_db 
select name as theName");
-assertTrue(tc.wasTypeRequested("hive_db"));
-assertFalse(tc.wasTypeRequested("theName"));
-assertFalse(tc.wasTypeRequested("name"));
-
-}
-
-@Test
-public void testFieldInSelectListNotTreatedAsType() throws Exception {
-
-
-ValidatingTypeCache tc = findTypeLookupsDuringQueryParsing("hive_db 
select name");
-assertTrue(tc.wasTypeRequested("hive_db"));
-assertFalse(tc.wasTypeRequested("name"));
-
-}
-
-private ValidatingTypeCache findTypeLookupsDuringQueryParsing(String 
query) throws AtlasException {
-TypeSystem typeSystem = TypeSystem.getInstance();
-ValidatingTypeCache result = new ValidatingTypeCache();
-typeSystem.setTypeCache(result);
-typeSystem.reset();
-HierarchicalTypeDefinition hiveTypeDef = 
createClassTypeDef("hive_db", "", ImmutableSet.of(),
-

[20/40] atlas git commit: ATLAS-2251: Remove TypeSystem and related implementation, to avoid unncessary duplicate of type details in cache

2017-11-05 Thread madhan
http://git-wip-us.apache.org/repos/asf/atlas/blob/0877e47c/repository/src/test/java/org/apache/atlas/repository/graph/GraphHelperTest.java
--
diff --git 
a/repository/src/test/java/org/apache/atlas/repository/graph/GraphHelperTest.java
 
b/repository/src/test/java/org/apache/atlas/repository/graph/GraphHelperTest.java
deleted file mode 100644
index e64c2c8..000
--- 
a/repository/src/test/java/org/apache/atlas/repository/graph/GraphHelperTest.java
+++ /dev/null
@@ -1,218 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- * 
- * http://www.apache.org/licenses/LICENSE-2.0
- * 
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.atlas.repository.graph;
-
-import org.apache.atlas.AtlasException;
-import org.apache.atlas.TestModules;
-import org.apache.atlas.TestUtils;
-import org.apache.atlas.repository.graph.GraphHelper.VertexInfo;
-import org.apache.atlas.repository.graphdb.AtlasEdge;
-import org.apache.atlas.repository.graphdb.AtlasGraph;
-import org.apache.atlas.repository.graphdb.AtlasVertex;
-import org.apache.atlas.services.MetadataService;
-import org.apache.atlas.type.AtlasTypeRegistry;
-import org.apache.atlas.typesystem.ITypedReferenceableInstance;
-import org.apache.atlas.typesystem.Referenceable;
-import org.apache.atlas.typesystem.TypesDef;
-import org.apache.atlas.typesystem.exception.TypeNotFoundException;
-import org.apache.atlas.typesystem.json.InstanceSerialization;
-import org.apache.atlas.typesystem.json.TypesSerialization;
-import org.apache.atlas.typesystem.types.ClassType;
-import org.apache.atlas.typesystem.types.Multiplicity;
-import org.apache.atlas.typesystem.types.TypeSystem;
-import org.codehaus.jettison.json.JSONArray;
-import org.testng.Assert;
-import org.testng.annotations.AfterClass;
-import org.testng.annotations.BeforeClass;
-import org.testng.annotations.DataProvider;
-import org.testng.annotations.Guice;
-import org.testng.annotations.Test;
-
-import javax.inject.Inject;
-import java.util.ArrayList;
-import java.util.Arrays;
-import java.util.HashMap;
-import java.util.HashSet;
-import java.util.Iterator;
-import java.util.List;
-import java.util.Map;
-import java.util.Set;
-
-import static org.testng.Assert.*;
-
-@Guice(modules = TestModules.TestOnlyModule.class)
-public class GraphHelperTest {
-
-
-@DataProvider(name = "encodeDecodeTestData")
-private Object[][] createTestData() {
-return new Object[][]{
-{"hivedb$", "hivedb_d"},
-{"hivedb", "hivedb"},
-{"{hivedb}", "_ohivedb_c"},
-{"%hivedb}", "_phivedb_c"},
-{"\"hivedb\"", "_qhivedb_q"},
-{"\"$%{}", "_q_d_p_o_c"},
-{"", ""},
-{"  ", "  "},
-{"\n\r", "\n\r"},
-{null, null}
-};
-}
-
-@Inject
-private MetadataService metadataService;
-
-@Inject
-private GraphBackedMetadataRepository repositoryService;
-
-private TypeSystem typeSystem;
-
-@Inject
-private AtlasTypeRegistry typeRegistry;
-
-@BeforeClass
-public void setUp() throws Exception {
-typeSystem = TypeSystem.getInstance();
-typeSystem.reset();
-
-new GraphBackedSearchIndexer(typeRegistry);
-TypesDef typesDef = TestUtils.defineHiveTypes();
-try {
-metadataService.getTypeDefinition(TestUtils.TABLE_TYPE);
-} catch (TypeNotFoundException e) {
-metadataService.createType(TypesSerialization.toJson(typesDef));
-}
-TestUtils.defineDeptEmployeeTypes(typeSystem);
-}
-
-@AfterClass
-public void tearDown() {
-//AtlasGraphProvider.cleanup();
-}
-
-@Test
-public void testGetInstancesByUniqueAttributes() throws Exception {
-
-GraphHelper helper = GraphHelper.getInstance();
-List instances =  new ArrayList<>();
-List guids = new ArrayList<>();
-TypeSystem ts = TypeSystem.getInstance();
-ClassType dbType = ts.getDataType(ClassType.class, 
TestUtils.DATABASE_TYPE);
-
-for(int i = 0; i < 10; i++) {
-Referenceable db = TestUtils.createDBEntity();
-String guid = createInstance(db);
-ITypedReferenceableInstance 

[34/40] atlas git commit: ATLAS-2251: Remove TypeSystem and related implementation, to avoid unncessary duplicate of type details in cache

2017-11-05 Thread madhan
http://git-wip-us.apache.org/repos/asf/atlas/blob/0877e47c/repository/src/main/java/org/apache/atlas/query/Expressions.java
--
diff --git a/repository/src/main/java/org/apache/atlas/query/Expressions.java 
b/repository/src/main/java/org/apache/atlas/query/Expressions.java
new file mode 100644
index 000..9e93ce4
--- /dev/null
+++ b/repository/src/main/java/org/apache/atlas/query/Expressions.java
@@ -0,0 +1,45 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.atlas.query;
+
+
+import java.util.List;
+
+public class Expressions {
+public static class Expression {
+
+}
+
+public static class AliasExpression {
+public String alias() {
+String ret = null;
+
+return ret;
+}
+
+}
+
+public static class SelectExpression {
+public List toJavaList() {
+List ret = null;
+
+return ret;
+}
+
+}
+}

http://git-wip-us.apache.org/repos/asf/atlas/blob/0877e47c/repository/src/main/java/org/apache/atlas/query/GremlinQuery.java
--
diff --git a/repository/src/main/java/org/apache/atlas/query/GremlinQuery.java 
b/repository/src/main/java/org/apache/atlas/query/GremlinQuery.java
new file mode 100644
index 000..fcb1f48
--- /dev/null
+++ b/repository/src/main/java/org/apache/atlas/query/GremlinQuery.java
@@ -0,0 +1,42 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.atlas.query;
+
+import org.apache.atlas.query.Expressions.Expression;
+
+
+public class GremlinQuery {
+
+public boolean hasSelectList() {
+boolean ret = false;
+
+return ret;
+}
+
+public String queryStr() {
+String ret = null;
+
+return ret;
+}
+
+public Expression expr() {
+Expression ret = null;
+
+return ret;
+}
+}

http://git-wip-us.apache.org/repos/asf/atlas/blob/0877e47c/repository/src/main/java/org/apache/atlas/query/GremlinTranslator.java
--
diff --git 
a/repository/src/main/java/org/apache/atlas/query/GremlinTranslator.java 
b/repository/src/main/java/org/apache/atlas/query/GremlinTranslator.java
new file mode 100644
index 000..5395ddd
--- /dev/null
+++ b/repository/src/main/java/org/apache/atlas/query/GremlinTranslator.java
@@ -0,0 +1,34 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.atlas.query;
+
+import org.apache.atlas.query.Expressions.Expression;
+
+public class GremlinTranslator {
+private Expression expression;
+

[13/40] atlas git commit: ATLAS-2251: Remove TypeSystem and related implementation, to avoid unncessary duplicate of type details in cache

2017-11-05 Thread madhan
http://git-wip-us.apache.org/repos/asf/atlas/blob/0877e47c/repository/src/test/java/org/apache/atlas/service/DefaultMetadataServiceTest.java
--
diff --git 
a/repository/src/test/java/org/apache/atlas/service/DefaultMetadataServiceTest.java
 
b/repository/src/test/java/org/apache/atlas/service/DefaultMetadataServiceTest.java
deleted file mode 100644
index e885b8c..000
--- 
a/repository/src/test/java/org/apache/atlas/service/DefaultMetadataServiceTest.java
+++ /dev/null
@@ -1,1381 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- * 
- * http://www.apache.org/licenses/LICENSE-2.0
- * 
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.atlas.service;
-
-import com.google.common.collect.ImmutableList;
-import com.google.common.collect.ImmutableSet;
-import com.google.inject.Inject;
-import org.apache.atlas.AtlasClient;
-import org.apache.atlas.AtlasException;
-import org.apache.atlas.CreateUpdateEntitiesResult;
-import org.apache.atlas.EntityAuditEvent;
-import org.apache.atlas.RequestContext;
-import org.apache.atlas.TestModules;
-import org.apache.atlas.TestUtils;
-import org.apache.atlas.discovery.graph.GraphBackedDiscoveryService;
-import org.apache.atlas.exception.AtlasBaseException;
-import org.apache.atlas.listener.ChangedTypeDefs;
-import org.apache.atlas.listener.EntityChangeListener;
-import org.apache.atlas.listener.TypeDefChangeListener;
-import org.apache.atlas.model.legacy.EntityResult;
-import org.apache.atlas.query.QueryParams;
-import org.apache.atlas.repository.Constants;
-import org.apache.atlas.repository.audit.EntityAuditRepository;
-import org.apache.atlas.repository.audit.HBaseBasedAuditRepository;
-import org.apache.atlas.repository.audit.HBaseTestUtils;
-import org.apache.atlas.repository.graph.AtlasGraphProvider;
-import org.apache.atlas.repository.graphdb.AtlasEdge;
-import org.apache.atlas.repository.graphdb.AtlasEdgeDirection;
-import org.apache.atlas.repository.graphdb.AtlasGraph;
-import org.apache.atlas.repository.graphdb.AtlasVertex;
-import org.apache.atlas.services.DefaultMetadataService;
-import org.apache.atlas.services.MetadataService;
-import org.apache.atlas.type.AtlasTypeUtil;
-import org.apache.atlas.typesystem.IReferenceableInstance;
-import org.apache.atlas.typesystem.IStruct;
-import org.apache.atlas.typesystem.ITypedReferenceableInstance;
-import org.apache.atlas.typesystem.Referenceable;
-import org.apache.atlas.typesystem.Struct;
-import org.apache.atlas.typesystem.TypesDef;
-import org.apache.atlas.typesystem.exception.EntityNotFoundException;
-import org.apache.atlas.typesystem.exception.TypeNotFoundException;
-import org.apache.atlas.typesystem.json.InstanceSerialization;
-import org.apache.atlas.typesystem.json.TypesSerialization;
-import org.apache.atlas.typesystem.persistence.Id;
-import org.apache.atlas.typesystem.types.AttributeDefinition;
-import org.apache.atlas.typesystem.types.ClassType;
-import org.apache.atlas.typesystem.types.DataTypes;
-import org.apache.atlas.typesystem.types.EnumValue;
-import org.apache.atlas.typesystem.types.HierarchicalTypeDefinition;
-import org.apache.atlas.typesystem.types.Multiplicity;
-import org.apache.atlas.typesystem.types.TypeSystem;
-import org.apache.atlas.typesystem.types.ValueConversionException;
-import org.apache.atlas.typesystem.types.cache.TypeCache;
-import org.apache.atlas.typesystem.types.utils.TypesUtil;
-import org.apache.atlas.utils.ParamChecker;
-import org.codehaus.jettison.json.JSONArray;
-import org.codehaus.jettison.json.JSONException;
-import org.codehaus.jettison.json.JSONObject;
-import org.testng.Assert;
-import org.testng.annotations.AfterTest;
-import org.testng.annotations.BeforeTest;
-import org.testng.annotations.Guice;
-import org.testng.annotations.Test;
-
-import java.util.ArrayList;
-import java.util.Arrays;
-import java.util.Collection;
-import java.util.Collections;
-import java.util.HashMap;
-import java.util.List;
-import java.util.Map;
-import java.util.concurrent.Callable;
-import java.util.concurrent.ExecutionException;
-import java.util.concurrent.ExecutorService;
-import java.util.concurrent.Executors;
-import java.util.concurrent.Future;
-
-import static org.apache.atlas.TestUtils.*;
-import static 

[10/40] atlas git commit: ATLAS-2251: Remove TypeSystem and related implementation, to avoid unncessary duplicate of type details in cache

2017-11-05 Thread madhan
http://git-wip-us.apache.org/repos/asf/atlas/blob/0877e47c/typesystem/src/main/java/org/apache/atlas/typesystem/IStruct.java
--
diff --git a/typesystem/src/main/java/org/apache/atlas/typesystem/IStruct.java 
b/typesystem/src/main/java/org/apache/atlas/typesystem/IStruct.java
deleted file mode 100755
index e0f8576..000
--- a/typesystem/src/main/java/org/apache/atlas/typesystem/IStruct.java
+++ /dev/null
@@ -1,26 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.atlas.typesystem;
-
-/**
- * A marker interface for StructType and TraitType instances.
- */
-public interface IStruct extends IInstance {
-
-}

http://git-wip-us.apache.org/repos/asf/atlas/blob/0877e47c/typesystem/src/main/java/org/apache/atlas/typesystem/ITypedInstance.java
--
diff --git 
a/typesystem/src/main/java/org/apache/atlas/typesystem/ITypedInstance.java 
b/typesystem/src/main/java/org/apache/atlas/typesystem/ITypedInstance.java
deleted file mode 100755
index c3bb9af..000
--- a/typesystem/src/main/java/org/apache/atlas/typesystem/ITypedInstance.java
+++ /dev/null
@@ -1,88 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.atlas.typesystem;
-
-import org.apache.atlas.AtlasException;
-import org.apache.atlas.typesystem.types.FieldMapping;
-
-import java.math.BigDecimal;
-import java.math.BigInteger;
-import java.security.MessageDigest;
-import java.util.Date;
-
-/**
- * An instance whose structure is associated with a IDataType.
- * This is obtained by a call to 'createInstance' or the result of a Query.
- * A ITypedInstance can only contain information on attributes of the 
associated Type.
- * Instance can still be invalid because of missing required fields or 
incorrect multiplicity.
- * But user can only get/set on a known field of the associated type. Type 
values have to match
- * the IDataType of the associated attribute.
- */
-public interface ITypedInstance extends IInstance {
-
-FieldMapping fieldMapping();
-
-boolean getBoolean(String attrName) throws AtlasException;
-
-byte getByte(String attrName) throws AtlasException;
-
-short getShort(String attrName) throws AtlasException;
-
-int getInt(String attrName) throws AtlasException;
-
-long getLong(String attrName) throws AtlasException;
-
-float getFloat(String attrName) throws AtlasException;
-
-double getDouble(String attrName) throws AtlasException;
-
-BigInteger getBigInt(String attrName) throws AtlasException;
-
-BigDecimal getBigDecimal(String attrName) throws AtlasException;
-
-Date getDate(String attrName) throws AtlasException;
-
-String getString(String attrName) throws AtlasException;
-
-void setBoolean(String attrName, boolean val) throws AtlasException;
-
-void setByte(String attrName, byte val) throws AtlasException;
-
-void setShort(String attrName, short val) throws AtlasException;
-
-void setInt(String attrName, int val) throws AtlasException;
-
-void setLong(String attrName, long val) throws AtlasException;
-
-void setFloat(String attrName, float val) throws AtlasException;
-
-void setDouble(String attrName, double val) throws AtlasException;
-
-void setBigInt(String attrName, BigInteger val) throws AtlasException;
-
-void setBigDecimal(String attrName, BigDecimal 

[01/40] atlas git commit: ATLAS-2251: Remove TypeSystem and related implementation, to avoid unncessary duplicate of type details in cache

2017-11-05 Thread madhan
Repository: atlas
Updated Branches:
  refs/heads/ATLAS-2251 [created] 10c19eac7


http://git-wip-us.apache.org/repos/asf/atlas/blob/0877e47c/webapp/src/test/java/org/apache/atlas/web/integration/TypesJerseyResourceIT.java
--
diff --git 
a/webapp/src/test/java/org/apache/atlas/web/integration/TypesJerseyResourceIT.java
 
b/webapp/src/test/java/org/apache/atlas/web/integration/TypesJerseyResourceIT.java
index 351e5ae..ab39955 100755
--- 
a/webapp/src/test/java/org/apache/atlas/web/integration/TypesJerseyResourceIT.java
+++ 
b/webapp/src/test/java/org/apache/atlas/web/integration/TypesJerseyResourceIT.java
@@ -23,17 +23,10 @@ import com.google.common.collect.ImmutableSet;
 import com.sun.jersey.core.util.MultivaluedMapImpl;
 import org.apache.atlas.AtlasClient;
 import org.apache.atlas.AtlasServiceException;
-import org.apache.atlas.typesystem.TypesDef;
-import org.apache.atlas.typesystem.json.TypesSerialization;
-import org.apache.atlas.typesystem.json.TypesSerialization$;
-import org.apache.atlas.typesystem.types.AttributeDefinition;
-import org.apache.atlas.typesystem.types.ClassType;
+import org.apache.atlas.model.typedef.AtlasBaseTypeDef;
+import org.apache.atlas.model.v1.typedef.*;
+import org.apache.atlas.type.AtlasType;
 import org.apache.atlas.typesystem.types.DataTypes;
-import org.apache.atlas.typesystem.types.EnumTypeDefinition;
-import org.apache.atlas.typesystem.types.HierarchicalTypeDefinition;
-import org.apache.atlas.typesystem.types.Multiplicity;
-import org.apache.atlas.typesystem.types.StructTypeDefinition;
-import org.apache.atlas.typesystem.types.TraitType;
 import org.apache.atlas.typesystem.types.utils.TypesUtil;
 import org.codehaus.jettison.json.JSONArray;
 import org.codehaus.jettison.json.JSONObject;
@@ -76,9 +69,9 @@ public class TypesJerseyResourceIT extends BaseResourceIT {
 public void testSubmit() throws Exception {
 for (HierarchicalTypeDefinition typeDefinition : typeDefinitions) {
 try{
-atlasClientV1.getType(typeDefinition.typeName);
+atlasClientV1.getType(typeDefinition.getTypeName());
 } catch (AtlasServiceException ase){
-String typesAsJSON = TypesSerialization.toJson(typeDefinition, 
false);
+String typesAsJSON = AtlasType.toV1Json(typeDefinition);
 System.out.println("typesAsJSON = " + typesAsJSON);
 
 JSONObject response = 
atlasClientV1.callAPIWithBody(AtlasClient.API_V1.CREATE_TYPE, typesAsJSON);
@@ -87,18 +80,18 @@ public class TypesJerseyResourceIT extends BaseResourceIT {
 
 JSONArray typesAdded = 
response.getJSONArray(AtlasClient.TYPES);
 assertEquals(typesAdded.length(), 1);
-assertEquals(typesAdded.getJSONObject(0).getString(NAME), 
typeDefinition.typeName);
+assertEquals(typesAdded.getJSONObject(0).getString(NAME), 
typeDefinition.getTypeName());
 Assert.assertNotNull(response.get(AtlasClient.REQUEST_ID));}
 }
 }
 
 @Test
 public void testDuplicateSubmit() throws Exception {
-HierarchicalTypeDefinition type = 
TypesUtil.createClassTypeDef(randomString(),
-ImmutableSet.of(), 
TypesUtil.createUniqueRequiredAttrDef(NAME, DataTypes.STRING_TYPE));
+ClassTypeDefinition type = 
TypesUtil.createClassTypeDef(randomString(), null,
+ImmutableSet.of(), 
TypesUtil.createUniqueRequiredAttrDef(NAME, 
AtlasBaseTypeDef.ATLAS_TYPE_STRING));
 TypesDef typesDef =
-TypesUtil.getTypesDef(ImmutableList.of(), 
ImmutableList.of(),
-
ImmutableList.of(), 
ImmutableList.of(type));
+new TypesDef(ImmutableList.of(), 
ImmutableList.of(),
+ImmutableList.of(), 
ImmutableList.of(type));
 atlasClientV1.createType(typesDef);
 
 try {
@@ -111,51 +104,51 @@ public class TypesJerseyResourceIT extends BaseResourceIT 
{
 
 @Test
 public void testUpdate() throws Exception {
-HierarchicalTypeDefinition typeDefinition = TypesUtil
+ClassTypeDefinition typeDefinition = TypesUtil
 .createClassTypeDef(randomString(), null, "1.0", 
ImmutableSet.of(),
-TypesUtil.createUniqueRequiredAttrDef(NAME, 
DataTypes.STRING_TYPE));
-List typesCreated = 
atlasClientV1.createType(TypesSerialization.toJson(typeDefinition, false));
+TypesUtil.createUniqueRequiredAttrDef(NAME, 
AtlasBaseTypeDef.ATLAS_TYPE_STRING));
+List typesCreated = 
atlasClientV1.createType(AtlasType.toV1Json(typeDefinition));
 assertEquals(typesCreated.size(), 1);
-assertEquals(typesCreated.get(0), typeDefinition.typeName);
+assertEquals(typesCreated.get(0), typeDefinition.getTypeName());
 
 //Add attribute description
-typeDefinition = 

[06/40] atlas git commit: ATLAS-2251: Remove TypeSystem and related implementation, to avoid unncessary duplicate of type details in cache

2017-11-05 Thread madhan
http://git-wip-us.apache.org/repos/asf/atlas/blob/0877e47c/typesystem/src/main/scala/org/apache/atlas/typesystem/builders/TypesBuilder.scala
--
diff --git 
a/typesystem/src/main/scala/org/apache/atlas/typesystem/builders/TypesBuilder.scala
 
b/typesystem/src/main/scala/org/apache/atlas/typesystem/builders/TypesBuilder.scala
deleted file mode 100644
index 5ea345f..000
--- 
a/typesystem/src/main/scala/org/apache/atlas/typesystem/builders/TypesBuilder.scala
+++ /dev/null
@@ -1,188 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.atlas.typesystem.builders
-
-import com.google.common.collect.ImmutableList
-import org.apache.atlas.typesystem.TypesDef
-import org.apache.atlas.typesystem.types._
-import org.apache.atlas.typesystem.types.utils.TypesUtil
-import scala.collection.mutable.ArrayBuffer
-import scala.language.{dynamics, implicitConversions, postfixOps}
-import scala.util.DynamicVariable
-import com.google.common.collect.ImmutableSet
-
-object TypesBuilder {
-
-  case class Context(enums : ArrayBuffer[EnumTypeDefinition],
-  structs : ArrayBuffer[StructTypeDefinition],
-  classes : 
ArrayBuffer[HierarchicalTypeDefinition[ClassType]],
-  traits : 
ArrayBuffer[HierarchicalTypeDefinition[TraitType]],
-  currentTypeAttrs : ArrayBuffer[Attr] = null)
-
-  class AttrOption()
-  class ReverseAttributeName(val rNm : String) extends AttrOption
-  class MultiplicityOption(val lower: Int, val upper: Int, val isUnique: 
Boolean) extends AttrOption
-
-  val required = new AttrOption()
-  val optional = new AttrOption()
-  val collection = new AttrOption()
-  val set = new AttrOption()
-  val composite = new AttrOption()
-  val unique = new AttrOption()
-  val indexed = new AttrOption()
-  def reverseAttributeName(rNm : String) = new ReverseAttributeName(rNm)
-  def multiplicty(lower: Int, upper: Int, isUnique: Boolean) = new 
MultiplicityOption(lower, upper, isUnique)
-
-  val boolean = DataTypes.BOOLEAN_TYPE.getName
-  val byte = DataTypes.BYTE_TYPE.getName
-  val short = DataTypes.SHORT_TYPE.getName
-  val int = DataTypes.INT_TYPE.getName
-  val long = DataTypes.LONG_TYPE.getName
-  val float = DataTypes.FLOAT_TYPE.getName
-
-  val double = DataTypes.DOUBLE_TYPE.getName
-  val bigint = DataTypes.BIGINTEGER_TYPE.getName
-  val bigdecimal = DataTypes.BIGDECIMAL_TYPE.getName
-  val date = DataTypes.DATE_TYPE.getName
-  val string = DataTypes.STRING_TYPE.getName
-
-  def array(t : String) : String = {
-DataTypes.arrayTypeName(t)
-  }
-
-  def map(kt : String, vt : String) : String = {
-DataTypes.mapTypeName(kt, vt)
-  }
-
-  class Attr(ctx : Context, val name : String) {
-
-private var dataTypeName : String = DataTypes.BOOLEAN_TYPE.getName
-private var multiplicity: Multiplicity = Multiplicity.OPTIONAL
-private var isComposite: Boolean = false
-private var reverseAttributeName: String = null
-private var isUnique: Boolean = false
-private var isIndexable: Boolean = false
-
-ctx.currentTypeAttrs += this
-
-def getDef : AttributeDefinition =
-  new AttributeDefinition(name, dataTypeName,
-multiplicity, isComposite, isUnique, isIndexable, reverseAttributeName)
-
-def `~`(dT : String, options : AttrOption*) : Attr = {
-  dataTypeName = dT
-  options.foreach { o =>
-o match {
-  case `required` => {multiplicity = Multiplicity.REQUIRED}
-  case `optional` => {multiplicity = Multiplicity.OPTIONAL}
-  case `collection` => {multiplicity = Multiplicity.COLLECTION}
-  case `set` => {multiplicity = Multiplicity.SET}
-  case `composite` => {isComposite = true}
-  case `unique` => {isUnique = true}
-  case `indexed` => {isIndexable = true}
-  case m : MultiplicityOption => {multiplicity = new 
Multiplicity(m.lower, m.upper, m.isUnique)}
-  case r : ReverseAttributeName => {reverseAttributeName = r.rNm}
-  case _ => ()
-}
-  }
-  this
-}
-
-  }
-
-}
-
-class TypesBuilder {
-
-  import 

[27/40] atlas git commit: ATLAS-2251: Remove TypeSystem and related implementation, to avoid unncessary duplicate of type details in cache

2017-11-05 Thread madhan
http://git-wip-us.apache.org/repos/asf/atlas/blob/0877e47c/repository/src/main/scala/org/apache/atlas/query/GremlinQuery.scala
--
diff --git 
a/repository/src/main/scala/org/apache/atlas/query/GremlinQuery.scala 
b/repository/src/main/scala/org/apache/atlas/query/GremlinQuery.scala
deleted file mode 100644
index 37015d8..000
--- a/repository/src/main/scala/org/apache/atlas/query/GremlinQuery.scala
+++ /dev/null
@@ -1,806 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.atlas.query
-
-import java.lang.Boolean
-import java.lang.Byte
-import java.lang.Double
-import java.lang.Float
-import java.lang.Integer
-import java.lang.Long
-import java.lang.Short
-import java.util.ArrayList
-
-import scala.collection.JavaConversions.asScalaBuffer
-import scala.collection.JavaConversions.bufferAsJavaList
-import scala.collection.mutable
-import scala.collection.mutable.ArrayBuffer
-
-
-import org.apache.atlas.gremlin.GremlinExpressionFactory
-import org.apache.atlas.gremlin.optimizer.GremlinQueryOptimizer
-import org.apache.atlas.groovy.CastExpression
-import org.apache.atlas.groovy.ClosureExpression
-import org.apache.atlas.groovy.LabeledExpression
-import org.apache.atlas.groovy.FunctionCallExpression
-import org.apache.atlas.groovy.GroovyExpression
-import org.apache.atlas.groovy.GroovyGenerationContext
-import org.apache.atlas.groovy.IdentifierExpression
-import org.apache.atlas.groovy.ListExpression
-import org.apache.atlas.groovy.LiteralExpression
-import org.apache.atlas.groovy.TraversalStepType
-import org.apache.atlas.query.Expressions.AliasExpression
-import org.apache.atlas.query.Expressions.ArithmeticExpression
-import org.apache.atlas.query.Expressions.BackReference
-import org.apache.atlas.query.Expressions.ClassExpression
-import org.apache.atlas.query.Expressions.ComparisonExpression
-import org.apache.atlas.query.Expressions.Expression
-import org.apache.atlas.query.Expressions.ExpressionException
-import org.apache.atlas.query.Expressions.FieldExpression
-import org.apache.atlas.query.Expressions.FilterExpression
-import org.apache.atlas.query.Expressions.InstanceExpression
-import org.apache.atlas.query.Expressions.LimitExpression
-import org.apache.atlas.query.Expressions.ListLiteral
-import org.apache.atlas.query.Expressions.Literal
-import org.apache.atlas.query.Expressions.LogicalExpression
-import org.apache.atlas.query.Expressions.LoopExpression
-import org.apache.atlas.query.Expressions.OrderExpression
-import org.apache.atlas.query.Expressions.PathExpression
-import org.apache.atlas.query.Expressions.SelectExpression
-import org.apache.atlas.query.Expressions.TraitExpression
-import org.apache.atlas.query.Expressions.TraitInstanceExpression
-import org.apache.atlas.query.Expressions.hasFieldLeafExpression
-import org.apache.atlas.query.Expressions.hasFieldUnaryExpression
-import org.apache.atlas.query.Expressions.id
-import org.apache.atlas.query.Expressions.isTraitLeafExpression
-import org.apache.atlas.query.Expressions.isTraitUnaryExpression
-import org.apache.atlas.repository.RepositoryException
-import org.apache.atlas.repository.graphdb.AtlasEdgeDirection
-import org.apache.atlas.typesystem.types.DataTypes
-import org.apache.atlas.typesystem.types.DataTypes.TypeCategory
-import org.apache.atlas.typesystem.types.IDataType
-import org.apache.atlas.typesystem.types.TypeSystem
-import org.joda.time.format.ISODateTimeFormat
-import org.apache.atlas.query.Expressions.GroupByExpression
-import org.apache.atlas.query.Expressions.MaxExpression
-import org.apache.atlas.query.Expressions.MinExpression
-import org.apache.atlas.query.Expressions.SumExpression
-import org.apache.atlas.query.Expressions.CountExpression
-
-import org.apache.atlas.util.AtlasRepositoryConfiguration
-import java.util.HashSet
-
-trait IntSequence {
-def next: Int
-}
-
-case class GremlinQuery(expr: Expression, queryStr: String, resultMaping: 
Map[String, (String, Int)]) {
-
-def hasSelectList = resultMaping != null
-
-def isPathExpression = expr.isInstanceOf[PathExpression]
-
-def isGroupBy = expr.isInstanceOf[GroupByExpression]
-}
-
-
-trait 

[29/40] atlas git commit: ATLAS-2251: Remove TypeSystem and related implementation, to avoid unncessary duplicate of type details in cache

2017-11-05 Thread madhan
http://git-wip-us.apache.org/repos/asf/atlas/blob/0877e47c/repository/src/main/java/org/apache/atlas/repository/typestore/TypeVertexInfo.java
--
diff --git 
a/repository/src/main/java/org/apache/atlas/repository/typestore/TypeVertexInfo.java
 
b/repository/src/main/java/org/apache/atlas/repository/typestore/TypeVertexInfo.java
deleted file mode 100644
index 32a9a19..000
--- 
a/repository/src/main/java/org/apache/atlas/repository/typestore/TypeVertexInfo.java
+++ /dev/null
@@ -1,94 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.atlas.repository.typestore;
-
-import java.util.Objects;
-
-import org.apache.atlas.typesystem.types.DataTypes;
-import org.apache.atlas.typesystem.types.DataTypes.TypeCategory;
-
-/**
- * Records the information needed to create a particular type vertex.
- */
-public class TypeVertexInfo {
-
-private DataTypes.TypeCategory category;
-private String typeName;
-private String typeDescription;
-
-public TypeVertexInfo(TypeCategory category, String typeName, String 
typeDescription) {
-super();
-this.category = category;
-this.typeName = typeName;
-this.typeDescription = typeDescription;
-}
-
-public DataTypes.TypeCategory getCategory() {
-return category;
-}
-
-public void setCategory(DataTypes.TypeCategory category) {
-this.category = category;
-}
-
-public String getTypeName() {
-return typeName;
-}
-
-public void setTypeName(String typeName) {
-this.typeName = typeName;
-}
-
-public String getTypeDescription() {
-return typeDescription;
-}
-
-public void setTypeDescription(String typeDescription) {
-this.typeDescription = typeDescription;
-}
-
-@Override
-public int hashCode() {
-return Objects.hash(category, typeName);
-}
-
-@Override
-public boolean equals(Object obj) {
-
-if (this == obj) {
-return true;
-}
-
-if (getClass() != obj.getClass()) {
-return false;
-}
-
-TypeVertexInfo other = (TypeVertexInfo)obj;
-if(! Objects.equals(category, other.category)) {
-return false;
-}
-
-if(! Objects.equals(typeName, other.typeName)) {
-return false;
-}
-
-return true;
-}
-
-}
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/atlas/blob/0877e47c/repository/src/main/java/org/apache/atlas/repository/typestore/TypeVisitor.java
--
diff --git 
a/repository/src/main/java/org/apache/atlas/repository/typestore/TypeVisitor.java
 
b/repository/src/main/java/org/apache/atlas/repository/typestore/TypeVisitor.java
deleted file mode 100644
index a6e353c..000
--- 
a/repository/src/main/java/org/apache/atlas/repository/typestore/TypeVisitor.java
+++ /dev/null
@@ -1,96 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.atlas.repository.typestore;
-
-import java.util.List;
-
-import org.apache.atlas.AtlasException;
-import org.apache.atlas.repository.RepositoryException;
-import org.apache.atlas.typesystem.types.AttributeInfo;
-import org.apache.atlas.typesystem.types.DataTypes.TypeCategory;
-import org.apache.atlas.typesystem.types.EnumType;
-import 

[08/40] atlas git commit: ATLAS-2251: Remove TypeSystem and related implementation, to avoid unncessary duplicate of type details in cache

2017-11-05 Thread madhan
http://git-wip-us.apache.org/repos/asf/atlas/blob/0877e47c/typesystem/src/main/java/org/apache/atlas/typesystem/types/HierarchicalTypeDefinition.java
--
diff --git 
a/typesystem/src/main/java/org/apache/atlas/typesystem/types/HierarchicalTypeDefinition.java
 
b/typesystem/src/main/java/org/apache/atlas/typesystem/types/HierarchicalTypeDefinition.java
deleted file mode 100755
index ab63fea..000
--- 
a/typesystem/src/main/java/org/apache/atlas/typesystem/types/HierarchicalTypeDefinition.java
+++ /dev/null
@@ -1,74 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.atlas.typesystem.types;
-
-import com.google.common.collect.ImmutableSet;
-import org.apache.atlas.AtlasConstants;
-
-import java.util.Objects;
-
-public class HierarchicalTypeDefinition extends 
StructTypeDefinition {
-
-public final ImmutableSet superTypes;
-public final String hierarchicalMetaTypeName;
-
-public HierarchicalTypeDefinition(Class hierarchicalMetaType, String 
typeName, String typeDescription, ImmutableSet superTypes,
-AttributeDefinition[] attributeDefinitions) {
-this(hierarchicalMetaType, typeName, typeDescription, 
AtlasConstants.DEFAULT_TYPE_VERSION, superTypes,
-attributeDefinitions);
-}
-
-// Used only for de-serializing JSON String to typedef.
-public HierarchicalTypeDefinition( String hierarchicalMetaTypeName, String 
typeName, String typeDescription, String typeVersion, String[] superTypes, 
AttributeDefinition[] attributeDefinitions) throws ClassNotFoundException {
-this((Class) Class.forName(hierarchicalMetaTypeName), typeName, 
typeDescription, typeVersion, ImmutableSet.copyOf(superTypes), 
attributeDefinitions);
-}
-// Used only for de-serializing JSON String to typedef (no typeVersion).
-public HierarchicalTypeDefinition( String hierarchicalMetaTypeName, String 
typeName, String typeDescription, String[] superTypes, AttributeDefinition[] 
attributeDefinitions) throws ClassNotFoundException {
-this((Class) Class.forName(hierarchicalMetaTypeName), typeName, 
typeDescription, AtlasConstants.DEFAULT_TYPE_VERSION, 
ImmutableSet.copyOf(superTypes), attributeDefinitions);
-}
-// Used only for serializing typedef to JSON String.
-public HierarchicalTypeDefinition( String hierarchicalMetaTypeName, String 
typeName, String typeDescription, String typeVersion, ImmutableSet 
superTypes, AttributeDefinition[] attributeDefinitions, String typeDef) throws 
ClassNotFoundException {
-this((Class) Class.forName(hierarchicalMetaTypeName), typeName, 
typeDescription, typeVersion, superTypes, attributeDefinitions);
-}
-// Used only for serializing typedef to JSON String (no typeVersion).
-public HierarchicalTypeDefinition( String hierarchicalMetaTypeName, String 
typeName, String typeDescription, ImmutableSet superTypes, 
AttributeDefinition[] attributeDefinitions, String typeDef) throws 
ClassNotFoundException {
-this((Class) Class.forName(hierarchicalMetaTypeName), typeName, 
typeDescription, AtlasConstants.DEFAULT_TYPE_VERSION, superTypes, 
attributeDefinitions);
-}
-
-public HierarchicalTypeDefinition(Class hierarchicalMetaType, String 
typeName, String typeDescription, String typeVersion, ImmutableSet 
superTypes, AttributeDefinition[] attributeDefinitions) {
-super(typeName, typeDescription, typeVersion, false, 
attributeDefinitions);
-this.hierarchicalMetaTypeName = hierarchicalMetaType.getName();
-this.superTypes = superTypes == null ? ImmutableSet.of() : 
superTypes;
-}
-
-@Override
-public boolean equals(Object o) {
-if (this == o) return true;
-if (o == null || getClass() != o.getClass()) return false;
-if (!super.equals(o)) return false;
-HierarchicalTypeDefinition that = (HierarchicalTypeDefinition) o;
-return Objects.equals(superTypes, that.superTypes) &&
-Objects.equals(hierarchicalMetaTypeName, 
that.hierarchicalMetaTypeName);
-}
-
-@Override
-public int hashCode() {
-return Objects.hash(super.hashCode(), superTypes, 

[37/40] atlas git commit: ATLAS-2251: Remove TypeSystem and related implementation, to avoid unncessary duplicate of type details in cache

2017-11-05 Thread madhan
http://git-wip-us.apache.org/repos/asf/atlas/blob/0877e47c/notification/src/test/java/org/apache/atlas/notification/hook/HookMessageDeserializerTest.java
--
diff --git 
a/notification/src/test/java/org/apache/atlas/notification/hook/HookMessageDeserializerTest.java
 
b/notification/src/test/java/org/apache/atlas/notification/hook/HookMessageDeserializerTest.java
index 49b877b..3e1c3dd 100644
--- 
a/notification/src/test/java/org/apache/atlas/notification/hook/HookMessageDeserializerTest.java
+++ 
b/notification/src/test/java/org/apache/atlas/notification/hook/HookMessageDeserializerTest.java
@@ -18,12 +18,13 @@
 
 package org.apache.atlas.notification.hook;
 
+import org.apache.atlas.model.v1.instance.Referenceable;
+import org.apache.atlas.model.v1.instance.Struct;
 import org.apache.atlas.notification.AbstractNotification;
 import org.apache.atlas.notification.entity.EntityNotificationImplTest;
 import org.apache.atlas.notification.hook.HookNotification.EntityUpdateRequest;
 import 
org.apache.atlas.notification.hook.HookNotification.HookNotificationMessage;
-import org.apache.atlas.typesystem.Referenceable;
-import org.apache.atlas.typesystem.Struct;
+import org.apache.atlas.type.AtlasType;
 import org.apache.commons.lang3.RandomStringUtils;
 import org.testng.annotations.Test;
 
@@ -61,7 +62,7 @@ public class HookMessageDeserializerTest {
 Referenceable   entity  = generateEntityWithTrait();
 EntityUpdateRequest message = new EntityUpdateRequest("user1", entity);
 
-String  jsonMsg = 
AbstractNotification.GSON.toJson(message);
+String  jsonMsg = 
AtlasType.toV1Json(message);
 HookNotificationMessage deserializedMessage = 
deserializer.deserialize(jsonMsg);
 
 assertEqualMessage(deserializedMessage, message);
@@ -79,7 +80,7 @@ public class HookMessageDeserializerTest {
 assertTrue(jsonMsgList.size() == 1);
 
 String compressedMsg   = jsonMsgList.get(0);
-String uncompressedMsg = AbstractNotification.GSON.toJson(message);
+String uncompressedMsg = AtlasType.toV1Json(message);
 
 assertTrue(compressedMsg.length() < uncompressedMsg.length(), 
"Compressed message (" + compressedMsg.length() + ") should be shorter than 
uncompressed message (" + uncompressedMsg.length() + ")");
 
@@ -134,7 +135,7 @@ public class HookMessageDeserializerTest {
 EntityUpdateRequest deserializedEntityUpdateRequest = 
(EntityUpdateRequest) deserializedMessage;
 Referenceable   deserializedEntity  = 
deserializedEntityUpdateRequest.getEntities().get(0);
 Referenceable   entity  = 
message.getEntities().get(0);
-String  traitName   = 
entity.getTraits().get(0);
+String  traitName   = 
entity.getTraitNames().get(0);
 
 assertEquals(deserializedEntity.getId(), entity.getId());
 assertEquals(deserializedEntity.getTypeName(), entity.getTypeName());

http://git-wip-us.apache.org/repos/asf/atlas/blob/0877e47c/notification/src/test/java/org/apache/atlas/notification/hook/HookNotificationTest.java
--
diff --git 
a/notification/src/test/java/org/apache/atlas/notification/hook/HookNotificationTest.java
 
b/notification/src/test/java/org/apache/atlas/notification/hook/HookNotificationTest.java
index dd3257e..786fbfe 100644
--- 
a/notification/src/test/java/org/apache/atlas/notification/hook/HookNotificationTest.java
+++ 
b/notification/src/test/java/org/apache/atlas/notification/hook/HookNotificationTest.java
@@ -17,8 +17,8 @@
  */
 package org.apache.atlas.notification.hook;
 
-import org.apache.atlas.notification.AbstractNotification;
-import org.apache.atlas.typesystem.Referenceable;
+import org.apache.atlas.model.v1.instance.Referenceable;
+import org.apache.atlas.type.AtlasType;
 import org.testng.annotations.Test;
 
 import static org.testng.Assert.assertEquals;
@@ -37,7 +37,7 @@ public class HookNotificationTest {
 String user = "user";
 HookNotification.EntityCreateRequest request = new 
HookNotification.EntityCreateRequest(user, entity1, entity2);
 
-String notificationJson = AbstractNotification.GSON.toJson(request);
+String notificationJson = AtlasType.toV1Json(request);
 HookNotification.HookNotificationMessage actualNotification =
 HOOK_MESSAGE_DESERIALIZER.deserialize(notificationJson);
 
@@ -60,7 +60,7 @@ public class HookNotificationTest {
 entity.set("attr", "value");
 HookNotification.EntityCreateRequest request = new 
HookNotification.EntityCreateRequest(null, entity);
 
-String notificationJsonFromCode = 
AbstractNotification.GSON.toJson(request);
+String notificationJsonFromCode = 

[24/40] atlas git commit: ATLAS-2251: Remove TypeSystem and related implementation, to avoid unncessary duplicate of type details in cache

2017-11-05 Thread madhan
http://git-wip-us.apache.org/repos/asf/atlas/blob/0877e47c/repository/src/test/java/org/apache/atlas/discovery/GraphBackedDiscoveryServiceTest.java
--
diff --git 
a/repository/src/test/java/org/apache/atlas/discovery/GraphBackedDiscoveryServiceTest.java
 
b/repository/src/test/java/org/apache/atlas/discovery/GraphBackedDiscoveryServiceTest.java
deleted file mode 100755
index bd730e4..000
--- 
a/repository/src/test/java/org/apache/atlas/discovery/GraphBackedDiscoveryServiceTest.java
+++ /dev/null
@@ -1,1334 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.atlas.discovery;
-
-import com.google.common.collect.ImmutableSet;
-import org.apache.atlas.AtlasException;
-import org.apache.atlas.BaseRepositoryTest;
-import org.apache.atlas.RequestContext;
-import org.apache.atlas.TestModules;
-import org.apache.atlas.TestUtils;
-import org.apache.atlas.discovery.graph.GraphBackedDiscoveryService;
-import org.apache.atlas.query.QueryParams;
-import org.apache.atlas.repository.Constants;
-import org.apache.atlas.repository.MetadataRepository;
-import org.apache.atlas.repository.graph.AtlasGraphProvider;
-import org.apache.atlas.repository.graph.GraphBackedSearchIndexer;
-import org.apache.atlas.repository.graphdb.GremlinVersion;
-import org.apache.atlas.util.AtlasGremlinQueryProvider;
-import org.apache.atlas.util.AtlasGremlinQueryProvider.AtlasGremlinQuery;
-import org.apache.atlas.type.AtlasTypeRegistry;
-import org.apache.atlas.typesystem.ITypedReferenceableInstance;
-import org.apache.atlas.typesystem.Referenceable;
-import org.apache.atlas.typesystem.persistence.Id;
-import org.apache.atlas.typesystem.types.ClassType;
-import org.apache.atlas.typesystem.types.DataTypes;
-import org.apache.atlas.typesystem.types.HierarchicalTypeDefinition;
-import org.apache.atlas.typesystem.types.IDataType;
-import org.apache.atlas.typesystem.types.Multiplicity;
-import org.apache.atlas.typesystem.types.TypeSystem;
-import org.codehaus.jettison.json.JSONArray;
-import org.codehaus.jettison.json.JSONException;
-import org.codehaus.jettison.json.JSONObject;
-import org.testng.Assert;
-import org.testng.annotations.AfterClass;
-import org.testng.annotations.BeforeClass;
-import org.testng.annotations.BeforeMethod;
-import org.testng.annotations.DataProvider;
-import org.testng.annotations.Guice;
-import org.testng.annotations.Test;
-
-import javax.inject.Inject;
-import java.text.SimpleDateFormat;
-import java.util.*;
-
-import static 
org.apache.atlas.typesystem.types.utils.TypesUtil.createClassTypeDef;
-import static 
org.apache.atlas.typesystem.types.utils.TypesUtil.createOptionalAttrDef;
-import static 
org.apache.atlas.typesystem.types.utils.TypesUtil.createRequiredAttrDef;
-import static org.testng.Assert.assertEquals;
-import static org.testng.Assert.assertNotNull;
-
-@Guice(modules = TestModules.TestOnlyModule.class)
-public class GraphBackedDiscoveryServiceTest extends BaseRepositoryTest {
-
-@Inject
-private MetadataRepository repositoryService;
-
-@Inject
-private GraphBackedDiscoveryService discoveryService;
-private QueryParams queryParams = new QueryParams(40, 0);
-private static final String idType = "idType";
-@Override
-@BeforeClass
-public void setUp() throws Exception {
-super.setUp();
-
-repositoryService = TestUtils.addTransactionWrapper(repositoryService);
-final TypeSystem typeSystem = TypeSystem.getInstance();
-Collection oldTypeNames = new HashSet<>();
-oldTypeNames.addAll(typeSystem.getTypeNames());
-
-TestUtils.defineDeptEmployeeTypes(typeSystem);
-
-addIndexesForNewTypes(oldTypeNames, typeSystem);
-
-ITypedReferenceableInstance hrDept = 
TestUtils.createDeptEg1(typeSystem);
-repositoryService.createEntities(hrDept);
-
-ITypedReferenceableInstance jane = 
repositoryService.getEntityDefinition("Manager", "name", "Jane");
-Id janeGuid = jane.getId();
-ClassType personType = typeSystem.getDataType(ClassType.class, 
"Person");
-ITypedReferenceableInstance instance = 
personType.createInstance(janeGuid);
-

[18/40] atlas git commit: ATLAS-2251: Remove TypeSystem and related implementation, to avoid unncessary duplicate of type details in cache

2017-11-05 Thread madhan
http://git-wip-us.apache.org/repos/asf/atlas/blob/0877e47c/repository/src/test/java/org/apache/atlas/repository/impexp/ZipSourceTest.java
--
diff --git 
a/repository/src/test/java/org/apache/atlas/repository/impexp/ZipSourceTest.java
 
b/repository/src/test/java/org/apache/atlas/repository/impexp/ZipSourceTest.java
deleted file mode 100644
index 1c1c68f..000
--- 
a/repository/src/test/java/org/apache/atlas/repository/impexp/ZipSourceTest.java
+++ /dev/null
@@ -1,189 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.atlas.repository.impexp;
-
-import org.apache.atlas.exception.AtlasBaseException;
-import org.apache.atlas.model.instance.AtlasEntity;
-import org.apache.atlas.model.typedef.AtlasTypesDef;
-import org.testng.Assert;
-import org.testng.ITestContext;
-import org.testng.annotations.DataProvider;
-import org.testng.annotations.Test;
-
-import java.io.ByteArrayInputStream;
-import java.io.FileInputStream;
-import java.io.IOException;
-import java.util.List;
-
-import static 
org.apache.atlas.repository.impexp.ZipFileResourceTestUtils.getZipSource;
-import static org.testng.Assert.assertEquals;
-import static org.testng.Assert.assertFalse;
-import static org.testng.Assert.assertNotNull;
-import static org.testng.AssertJUnit.assertTrue;
-
-public class ZipSourceTest {
-@DataProvider(name = "zipFileStocks")
-public static Object[][] getDataFromZipFile() throws IOException {
-FileInputStream fs = 
ZipFileResourceTestUtils.getFileInputStream("stocks.zip");
-
-return new Object[][] {{ new ZipSource(fs) }};
-}
-
-@DataProvider(name = "zipFileStocksFloat")
-public static Object[][] getDataFromZipFileWithLongFloats() throws 
IOException {
-FileInputStream fs = 
ZipFileResourceTestUtils.getFileInputStream("stocks-float.zip");
-
-return new Object[][] {{ new ZipSource(fs) }};
-}
-
-@DataProvider(name = "sales")
-public static Object[][] getDataFromQuickStart_v1_Sales(ITestContext 
context) throws IOException {
-return getZipSource("sales-v1-full.zip");
-}
-
-@Test
-public void improperInit_ReturnsNullCreationOrder() throws IOException, 
AtlasBaseException {
-byte bytes[] = new byte[10];
-ByteArrayInputStream bais = new ByteArrayInputStream(bytes);
-ZipSource zs = new ZipSource(bais);
-List s = zs.getCreationOrder();
-Assert.assertNull(s);
-}
-
-@Test(dataProvider = "zipFileStocks")
-public void examineContents_BehavesAsExpected(ZipSource zipSource) throws 
IOException, AtlasBaseException {
-List creationOrder = zipSource.getCreationOrder();
-
-assertNotNull(creationOrder);
-assertEquals(creationOrder.size(), 4);
-
-AtlasTypesDef typesDef = zipSource.getTypesDef();
-assertNotNull(typesDef);
-assertEquals(typesDef.getEntityDefs().size(), 6);
-
-useCreationOrderToFetchEntitiesWithExtInfo(zipSource, creationOrder);
-useCreationOrderToFetchEntities(zipSource, creationOrder);
-attemptToFetchNonExistentGuid_ReturnsNull(zipSource, 
"non-existent-guid");
-verifyGuidRemovalOnImportComplete(zipSource, creationOrder.get(0));
-}
-
-private void useCreationOrderToFetchEntities(ZipSource zipSource, 
List creationOrder) {
-for (String guid : creationOrder) {
-AtlasEntity e = zipSource.getByGuid(guid);
-assertNotNull(e);
-}
-}
-
-private void verifyGuidRemovalOnImportComplete(ZipSource zipSource, String 
guid) {
-AtlasEntity e = zipSource.getByGuid(guid);
-assertNotNull(e);
-
-zipSource.onImportComplete(guid);
-
-e = zipSource.getByGuid(guid);
-Assert.assertNull(e);
-}
-
-private void attemptToFetchNonExistentGuid_ReturnsNull(ZipSource 
zipSource, String guid) {
-AtlasEntity e = zipSource.getByGuid(guid);
-Assert.assertNull(e);
-}
-
-private void useCreationOrderToFetchEntitiesWithExtInfo(ZipSource 
zipSource, List creationOrder) throws AtlasBaseException {
-for (String guid : creationOrder) {
-

[26/40] atlas git commit: ATLAS-2251: Remove TypeSystem and related implementation, to avoid unncessary duplicate of type details in cache

2017-11-05 Thread madhan
http://git-wip-us.apache.org/repos/asf/atlas/blob/0877e47c/repository/src/test/java/org/apache/atlas/BaseRepositoryTest.java
--
diff --git a/repository/src/test/java/org/apache/atlas/BaseRepositoryTest.java 
b/repository/src/test/java/org/apache/atlas/BaseRepositoryTest.java
deleted file mode 100644
index 7c6b60b..000
--- a/repository/src/test/java/org/apache/atlas/BaseRepositoryTest.java
+++ /dev/null
@@ -1,428 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.atlas;
-
-import com.google.common.base.Preconditions;
-import com.google.common.collect.ImmutableList;
-import com.google.common.collect.ImmutableSet;
-import org.apache.atlas.repository.MetadataRepository;
-import org.apache.atlas.repository.graph.GraphBackedSearchIndexer;
-import org.apache.atlas.services.MetadataService;
-import org.apache.atlas.type.AtlasTypeRegistry;
-import org.apache.atlas.typesystem.ITypedReferenceableInstance;
-import org.apache.atlas.typesystem.Referenceable;
-import org.apache.atlas.typesystem.TypesDef;
-import org.apache.atlas.typesystem.json.TypesSerialization;
-import org.apache.atlas.typesystem.persistence.Id;
-import org.apache.atlas.typesystem.types.*;
-import org.apache.atlas.typesystem.types.utils.TypesUtil;
-import org.testng.annotations.Guice;
-
-import javax.inject.Inject;
-import java.util.ArrayList;
-import java.util.Date;
-import java.util.List;
-
-import static org.apache.atlas.AtlasClient.PROCESS_ATTRIBUTE_INPUTS;
-import static org.apache.atlas.AtlasClient.PROCESS_ATTRIBUTE_OUTPUTS;
-
-/**
- *  Base Class to set up hive types and instances for tests
- */
-@Guice(modules = TestModules.TestOnlyModule.class)
-public class BaseRepositoryTest {
-
-@Inject
-protected MetadataService metadataService;
-
-@Inject
-protected MetadataRepository repository;
-
-
-protected void setUp() throws Exception {
-//force graph initialization / built in type registration
-TestUtils.getGraph();
-setUpDefaultTypes();
-setUpTypes();
-TestUtils.getGraph().commit();
-new GraphBackedSearchIndexer(new AtlasTypeRegistry());
-TestUtils.resetRequestContext();
-setupInstances();
-TestUtils.getGraph().commit();
-TestUtils.dumpGraph(TestUtils.getGraph());
-}
-
-protected void tearDown() throws Exception {
-TypeSystem.getInstance().reset();
-}
-
-private void setUpTypes() throws Exception {
-TypesDef typesDef = createTypeDefinitions();
-String typesAsJSON = TypesSerialization.toJson(typesDef);
-metadataService.createType(typesAsJSON);
-}
-
-protected static final String DATABASE_TYPE = "hive_db";
-protected static final String HIVE_TABLE_TYPE = "hive_table";
-private static final String COLUMN_TYPE = "hive_column";
-private static final String HIVE_PROCESS_TYPE = "hive_process";
-private static final String STORAGE_DESC_TYPE = "StorageDesc";
-private static final String VIEW_TYPE = "View";
-private static final String PARTITION_TYPE = "hive_partition";
-protected static final String DATASET_SUBTYPE = "dataset_subtype";
-
-TypesDef createTypeDefinitions() {
-HierarchicalTypeDefinition dbClsDef = TypesUtil
-.createClassTypeDef(DATABASE_TYPE, null,
-TypesUtil.createUniqueRequiredAttrDef("name", 
DataTypes.STRING_TYPE),
-attrDef("description", DataTypes.STRING_TYPE), 
attrDef("locationUri", DataTypes.STRING_TYPE),
-attrDef("owner", DataTypes.STRING_TYPE), attrDef("createTime", 
DataTypes.LONG_TYPE));
-
-HierarchicalTypeDefinition columnClsDef = TypesUtil
-.createClassTypeDef(COLUMN_TYPE, null, attrDef("name", 
DataTypes.STRING_TYPE),
-attrDef("dataType", DataTypes.STRING_TYPE), attrDef("comment", 
DataTypes.STRING_TYPE));
-
-HierarchicalTypeDefinition storageDescClsDef = TypesUtil
-.createClassTypeDef(STORAGE_DESC_TYPE, null,
-attrDef("location", DataTypes.STRING_TYPE),
-attrDef("inputFormat", DataTypes.STRING_TYPE), 
attrDef("outputFormat", 

[33/40] atlas git commit: ATLAS-2251: Remove TypeSystem and related implementation, to avoid unncessary duplicate of type details in cache

2017-11-05 Thread madhan
http://git-wip-us.apache.org/repos/asf/atlas/blob/0877e47c/repository/src/main/java/org/apache/atlas/repository/converters/AtlasStructFormatConverter.java
--
diff --git 
a/repository/src/main/java/org/apache/atlas/repository/converters/AtlasStructFormatConverter.java
 
b/repository/src/main/java/org/apache/atlas/repository/converters/AtlasStructFormatConverter.java
index 6b6ee01..b03eda3 100644
--- 
a/repository/src/main/java/org/apache/atlas/repository/converters/AtlasStructFormatConverter.java
+++ 
b/repository/src/main/java/org/apache/atlas/repository/converters/AtlasStructFormatConverter.java
@@ -18,21 +18,24 @@
 package org.apache.atlas.repository.converters;
 
 import org.apache.atlas.AtlasErrorCode;
-import org.apache.atlas.AtlasException;
 import org.apache.atlas.exception.AtlasBaseException;
 import org.apache.atlas.model.TypeCategory;
+import org.apache.atlas.model.instance.AtlasEntity;
+import org.apache.atlas.model.instance.AtlasObjectId;
 import org.apache.atlas.model.instance.AtlasStruct;
-import org.apache.atlas.type.AtlasStructType;
+import org.apache.atlas.model.v1.instance.Struct;
+import org.apache.atlas.type.*;
+import org.apache.atlas.type.AtlasBuiltInTypes.AtlasObjectIdType;
 import org.apache.atlas.type.AtlasStructType.AtlasAttribute;
-import org.apache.atlas.type.AtlasType;
-import org.apache.atlas.type.AtlasTypeRegistry;
-import org.apache.atlas.typesystem.IStruct;
-import org.apache.atlas.typesystem.Struct;
 import org.apache.commons.collections.MapUtils;
+import org.apache.commons.lang3.StringUtils;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
+import java.util.ArrayList;
+import java.util.Collection;
 import java.util.HashMap;
+import java.util.List;
 import java.util.Map;
 
 public class AtlasStructFormatConverter extends AtlasAbstractFormatConverter {
@@ -64,19 +67,12 @@ public class AtlasStructFormatConverter extends 
AtlasAbstractFormatConverter {
 } else {
 ret = new AtlasStruct(type.getTypeName());
 }
-} else if (v1Obj instanceof IStruct) {
-IStruct struct= (IStruct) v1Obj;
-Map v1Attribs = null;
-
-try {
-v1Attribs = struct.getValuesMap();
-} catch (AtlasException excp) {
-LOG.error("IStruct.getValuesMap() failed", excp);
-}
+} else if (v1Obj instanceof Struct) {
+Struct struct = (Struct) v1Obj;
 
-ret = new AtlasStruct(type.getTypeName(), 
fromV1ToV2(structType, v1Attribs, converterContext));
+ret = new AtlasStruct(type.getTypeName(), 
fromV1ToV2(structType, struct.getValues(), converterContext));
 } else {
-throw new AtlasBaseException(AtlasErrorCode.UNEXPECTED_TYPE, 
"Map or IStruct", v1Obj.getClass().getCanonicalName());
+throw new AtlasBaseException(AtlasErrorCode.UNEXPECTED_TYPE, 
"Map or Struct", v1Obj.getClass().getCanonicalName());
 }
 }
 
@@ -118,7 +114,8 @@ public class AtlasStructFormatConverter extends 
AtlasAbstractFormatConverter {
 }
 
 protected Map fromV2ToV1(AtlasStructType structType, 
Map attributes, ConverterContext context) throws 
AtlasBaseException {
-Map ret = null;
+Map ret  = null;
+boolean isEntityType = structType instanceof 
AtlasEntityType;
 
 if (MapUtils.isNotEmpty(attributes)) {
 ret = new HashMap<>();
@@ -132,13 +129,80 @@ public class AtlasStructFormatConverter extends 
AtlasAbstractFormatConverter {
 continue;
 }
 
-AtlasType attrType = attr.getAttributeType();
+AtlasTypeattrType  = attr.getAttributeType();
+AtlasFormatConverter attrConverter = 
converterRegistry.getConverter(attrType.getTypeCategory());
+Object   v2Value   = 
attributes.get(attr.getName());
+
+if (v2Value != null && isEntityType && attr.isOwnedRef()) {
+if (LOG.isDebugEnabled()) {
+LOG.debug("{}: is ownedRef, attrType={}", 
attr.getQualifiedName(), attrType.getTypeName());
+}
+
+if (attrType instanceof AtlasArrayType) {
+AtlasArrayType  arrayType  = (AtlasArrayType) attrType;
+AtlasType   elemType   = 
arrayType.getElementType();
+String  elemTypeName;
+
+if (elemType instanceof AtlasObjectIdType) {
+elemTypeName = ((AtlasObjectIdType) 
elemType).getObjectType();
+} else {
+elemTypeName = 

[19/40] atlas git commit: ATLAS-2251: Remove TypeSystem and related implementation, to avoid unncessary duplicate of type details in cache

2017-11-05 Thread madhan
http://git-wip-us.apache.org/repos/asf/atlas/blob/0877e47c/repository/src/test/java/org/apache/atlas/repository/impexp/AtlasImportRequestTest.java
--
diff --git 
a/repository/src/test/java/org/apache/atlas/repository/impexp/AtlasImportRequestTest.java
 
b/repository/src/test/java/org/apache/atlas/repository/impexp/AtlasImportRequestTest.java
deleted file mode 100644
index b70b181..000
--- 
a/repository/src/test/java/org/apache/atlas/repository/impexp/AtlasImportRequestTest.java
+++ /dev/null
@@ -1,145 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- * 
- * http://www.apache.org/licenses/LICENSE-2.0
- * 
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.atlas.repository.impexp;
-
-import org.apache.atlas.model.impexp.AtlasImportRequest;
-import org.apache.atlas.type.AtlasType;
-import org.testng.annotations.Test;
-
-import static org.testng.Assert.*;
-
-public class AtlasImportRequestTest {
-@Test
-public void serializeAtlasImportRequstFromJsonWithEmptyOptions() {
-String jsonData = "{ \"options\": {} }";
-
-AtlasImportRequest request = AtlasType.fromJson(jsonData, 
AtlasImportRequest.class);
-
-assertNotNull(request);
-assertNotNull(request.getOptions());
-
assertNull(request.getOptions().get(AtlasImportRequest.TRANSFORMS_KEY));
-
-ImportTransforms tr = 
ImportTransforms.fromJson(request.getOptions().get(AtlasImportRequest.TRANSFORMS_KEY));
-
-assertNull(tr);
-}
-
-@Test
-public void serializeOptions_VerifyAccessors() {
-String guid = "\"abcd\"";
-String pos = "\"1\"";
-String trueVal = "\"true\"";
-
-String jsonData = "{ \"options\": " +
-"   {" +
-"\"startGuid\":" + guid + "," +
-"\"startPosition\":" + pos + "," +
-"\"updateTypeDefinition\":" + trueVal +
-"}" +
-"}";
-
-AtlasImportRequest request = AtlasType.fromJson(jsonData, 
AtlasImportRequest.class);
-
-assertNotNull(request);
-assertNotNull(request.getStartGuid());
-assertNotNull(request.getStartPosition());
-assertNotNull(request.getUpdateTypeDefs());
-
-assertEquals(request.getStartGuid(), guid.replace("\"", ""));
-assertEquals(request.getStartPosition(), pos.replace("\"", ""));
-assertEquals(request.getUpdateTypeDefs(), trueVal.replace("\"", ""));
-}
-
-@Test
-public void optionsDefaultsTest() {
-String jsonData = "{ \"options\": " +
-"   {" +
-"}" +
-"}";
-
-AtlasImportRequest request = AtlasType.fromJson(jsonData, 
AtlasImportRequest.class);
-
-assertNotNull(request);
-assertNull(request.getStartGuid());
-assertNull(request.getStartPosition());
-assertNull(request.getUpdateTypeDefs());
-}
-
-@Test
-public void serializeAtlasImportRequstFromJsonWithEmptyTransforms() {
-String jsonData = "{ \"options\": { \"transforms\": \"{ }\" } }";
-
-AtlasImportRequest request = AtlasType.fromJson(jsonData, 
AtlasImportRequest.class);
-
-assertNotNull(request);
-assertNotNull(request.getOptions());
-
assertNotNull(request.getOptions().get(AtlasImportRequest.TRANSFORMS_KEY));
-
-ImportTransforms tr = 
ImportTransforms.fromJson(request.getOptions().get(AtlasImportRequest.TRANSFORMS_KEY));
-
-assertNotNull(tr);
-assertNotNull(tr.getTransforms());
-assertEquals(tr.getTransforms().size(), 0);
-}
-
-@Test
-public void serializeAtlasImportRequstFromJsonWith1Transform() {
-String jsonData = "{ \"options\": { \"transforms\": \"{ 
\\\"hive_db\\\": { \\\"qualifiedName\\\": [ \\\"replace:@cl1:@cl2\\\" ] } }\" } 
}";
-
-AtlasImportRequest request = AtlasType.fromJson(jsonData, 
AtlasImportRequest.class);
-
-assertNotNull(request);
-assertNotNull(request.getOptions());
-
assertNotNull(request.getOptions().get(AtlasImportRequest.TRANSFORMS_KEY));
-
-ImportTransforms tr = 
ImportTransforms.fromJson(request.getOptions().get(AtlasImportRequest.TRANSFORMS_KEY));
-
-

[35/40] atlas git commit: ATLAS-2251: Remove TypeSystem and related implementation, to avoid unncessary duplicate of type details in cache

2017-11-05 Thread madhan
http://git-wip-us.apache.org/repos/asf/atlas/blob/0877e47c/repository/src/main/java/org/apache/atlas/gremlin/optimizer/ExpandOrsOptimization.java
--
diff --git 
a/repository/src/main/java/org/apache/atlas/gremlin/optimizer/ExpandOrsOptimization.java
 
b/repository/src/main/java/org/apache/atlas/gremlin/optimizer/ExpandOrsOptimization.java
deleted file mode 100644
index a48a007..000
--- 
a/repository/src/main/java/org/apache/atlas/gremlin/optimizer/ExpandOrsOptimization.java
+++ /dev/null
@@ -1,588 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.atlas.gremlin.optimizer;
-
-import java.util.ArrayList;
-import java.util.Collections;
-import java.util.Iterator;
-import java.util.List;
-
-import org.apache.atlas.gremlin.GremlinExpressionFactory;
-import org.apache.atlas.groovy.AbstractFunctionExpression;
-import org.apache.atlas.groovy.ClosureExpression;
-import org.apache.atlas.groovy.FunctionCallExpression;
-import org.apache.atlas.groovy.GroovyExpression;
-import org.apache.atlas.groovy.LiteralExpression;
-import org.apache.atlas.groovy.StatementListExpression;
-import org.apache.atlas.groovy.TraversalStepType;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-
-import com.google.common.collect.Lists;
-
-
-
-/**
- * Optimization that removes 'or' expressions from a graph traversal when 
possible
- * and replaces them with separate calls that are combined using a logical 
union operation.
- * Unfortunately, Titan does not use indices when executing the child graph 
traversals associated
- * with an 'or' call.  In order to make the index be used, we split queries 
with
- * or expressions into multiple queries.  These queries are executed 
individually,
- * using indices, and then the results are combined back together.  Here is a
- * simple example to illustrate this:
- *
- * Original Query
- *
- * 
- * g.V().or(has('name','Fred'),has('age','17'))
- * 
- *
- *Optimized Query
- *
- * 
- * def r = [] as Set;
- * g.V().has('name','Fred').fill(r);
- * g.V().has('age','17').fill(r);
- * r;
- * 
- *
- * Here, we introduce an intermediate variable "r" which is declared as a Set. 
 The Set is performing
- * the union for us.  If there are vertices that happen to both have "Fred" as 
the name and "17" as the age,
- * the Set will prevent the second query execution from adding a duplicate 
vertex to the result.  Recall that
- * in Groovy scripts, the last expression is the one that will be returned 
back to the caller.  We refer to
- * that expression is the "result expression".  For this example, the result 
expression is simply "r", which
- * contains the vertices that matched the query.
- * 
- * If the query does any kind of transformation of the vertices to produce the 
query result, that needs
- * to be done in the result expression.  To understand why that is, let's take 
a look at another example:
- *
- * Original Query
- *
- * 
- * 
g.V().or(has('name','Fred'),has('age','17')).as('person').select('person').by('gender')
- * 
- *
- * Incorrect Optimized Query
- *
- * 
- * def r = [] as Set;
- * g.V().has('name','Fred').as('person').select('person').by('gender').fill(r)
- * g.V().has('age','17').as('person').select('person').by('gender').fill(r)
- * r;
- * 
- *
- * The problem with this query is that now 'r' contains Strings (the gender of 
the person).  Suppose
- * that there is one person named Fred and there are 3 people whose age is 17 
(let's say Fred's age is 16).
- * The original query would have produced 4 rows, one corresponding to each of 
those people.  The new
- * query would produce at most 2 rows - one for 'male' and one for 'female'.  
This is happening because
- * we are now performing the union on the Strings, not on the vertices.  To 
fix this, we need to split
- * the original query and put the end portion into the result expression:
- *
- * Correct Optimized Query
- *
- * 
- * def r = [] as Set;
- * g.V().has('name','Fred').fill(r)
- * g.V().has('age','17').fill(r)
- * __.inject(r as Object[]).as('person').select('person').by('gender')
- * 
- *
- * The logic for doing this splitting is described in more detail in
- * {@link 

[17/40] atlas git commit: ATLAS-2251: Remove TypeSystem and related implementation, to avoid unncessary duplicate of type details in cache

2017-11-05 Thread madhan
http://git-wip-us.apache.org/repos/asf/atlas/blob/0877e47c/repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasDeleteHandlerV1Test.java
--
diff --git 
a/repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasDeleteHandlerV1Test.java
 
b/repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasDeleteHandlerV1Test.java
deleted file mode 100644
index 718538a..000
--- 
a/repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasDeleteHandlerV1Test.java
+++ /dev/null
@@ -1,1141 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- * 
- * http://www.apache.org/licenses/LICENSE-2.0
- * 
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.atlas.repository.store.graph.v1;
-
-import com.google.common.collect.ImmutableList;
-import com.google.common.collect.ImmutableSet;
-import org.apache.atlas.AtlasException;
-import org.apache.atlas.RequestContextV1;
-import org.apache.atlas.TestUtils;
-import org.apache.atlas.TestUtilsV2;
-import org.apache.atlas.exception.AtlasBaseException;
-import org.apache.atlas.model.instance.AtlasEntity;
-import org.apache.atlas.model.instance.AtlasEntityHeader;
-import org.apache.atlas.model.instance.AtlasObjectId;
-import org.apache.atlas.model.instance.AtlasStruct;
-import org.apache.atlas.model.instance.EntityMutationResponse;
-import org.apache.atlas.model.instance.EntityMutations;
-import org.apache.atlas.model.typedef.AtlasClassificationDef;
-import org.apache.atlas.model.typedef.AtlasEntityDef;
-import org.apache.atlas.model.typedef.AtlasEnumDef;
-import org.apache.atlas.model.typedef.AtlasStructDef;
-import org.apache.atlas.model.typedef.AtlasTypesDef;
-import org.apache.atlas.repository.Constants;
-import org.apache.atlas.repository.graph.AtlasEdgeLabel;
-import org.apache.atlas.repository.graph.AtlasGraphProvider;
-import org.apache.atlas.repository.graph.GraphHelper;
-import org.apache.atlas.repository.graphdb.AtlasGraph;
-import org.apache.atlas.repository.graphdb.AtlasVertex;
-import 
org.apache.atlas.repository.store.bootstrap.AtlasTypeDefStoreInitializer;
-import org.apache.atlas.repository.store.graph.AtlasEntityStore;
-import org.apache.atlas.services.MetadataService;
-import org.apache.atlas.store.AtlasTypeDefStore;
-import org.apache.atlas.type.AtlasEntityType;
-import org.apache.atlas.type.AtlasTypeRegistry;
-import org.apache.atlas.type.AtlasTypeUtil;
-import org.apache.atlas.typesystem.IReferenceableInstance;
-import org.apache.atlas.typesystem.IStruct;
-import org.apache.atlas.typesystem.ITypedReferenceableInstance;
-import org.apache.atlas.typesystem.ITypedStruct;
-import org.apache.atlas.typesystem.Struct;
-import org.apache.atlas.typesystem.exception.EntityNotFoundException;
-import org.apache.atlas.typesystem.persistence.Id;
-import org.apache.atlas.typesystem.types.Multiplicity;
-import org.apache.atlas.typesystem.types.TraitType;
-import org.apache.atlas.typesystem.types.TypeSystem;
-import org.testng.Assert;
-import org.testng.annotations.AfterClass;
-import org.testng.annotations.BeforeClass;
-import org.testng.annotations.BeforeTest;
-import org.testng.annotations.Test;
-
-import javax.inject.Inject;
-import java.util.ArrayList;
-import java.util.Arrays;
-import java.util.Collections;
-import java.util.HashMap;
-import java.util.List;
-import java.util.Map;
-
-import static org.apache.atlas.TestUtils.*;
-import static org.mockito.Mockito.mock;
-import static org.testng.Assert.assertEquals;
-import static org.testng.Assert.assertNotEquals;
-import static org.testng.Assert.assertTrue;
-import static org.testng.Assert.fail;
-
-public abstract class AtlasDeleteHandlerV1Test {
-
-@Inject
-AtlasTypeRegistry typeRegistry;
-
-@Inject
-AtlasTypeDefStore typeDefStore;
-
-@Inject
-AtlasEntityStore entityStore;
-
-@Inject
-MetadataService metadataService;
-
-private AtlasEntityType compositeMapOwnerType;
-
-private AtlasEntityType compositeMapValueType;
-
-private TypeSystem typeSystem = TypeSystem.getInstance();
-
-@BeforeClass
-public void setUp() throws Exception {
-RequestContextV1.clear();
-RequestContextV1.get().setUser(TestUtilsV2.TEST_USER);
-
-metadataService = 

[39/40] atlas git commit: ATLAS-2251: Remove TypeSystem and related implementation, to avoid unncessary duplicate of type details in cache

2017-11-05 Thread madhan
ATLAS-2251: Remove TypeSystem and related implementation, to avoid unncessary 
duplicate of type details in cache


Project: http://git-wip-us.apache.org/repos/asf/atlas/repo
Commit: http://git-wip-us.apache.org/repos/asf/atlas/commit/0877e47c
Tree: http://git-wip-us.apache.org/repos/asf/atlas/tree/0877e47c
Diff: http://git-wip-us.apache.org/repos/asf/atlas/diff/0877e47c

Branch: refs/heads/ATLAS-2251
Commit: 0877e47cc4fdf79befb9075e6db0464d1601061d
Parents: 37be53b
Author: Madhan Neethiraj <mad...@apache.org>
Authored: Fri Nov 3 22:45:05 2017 -0700
Committer: Madhan Neethiraj <mad...@apache.org>
Committed: Sun Nov 5 11:55:26 2017 -0800

--
 .../atlas/falcon/bridge/FalconBridge.java   |4 +-
 .../apache/atlas/falcon/hook/FalconHook.java|2 +-
 .../apache/atlas/falcon/hook/FalconHookIT.java  |   20 +-
 .../atlas/hbase/bridge/HBaseAtlasHook.java  |2 +-
 .../apache/atlas/hbase/HBaseAtlasHookIT.java|2 +-
 .../atlas/hive/bridge/ColumnLineageUtils.java   |2 +-
 .../atlas/hive/bridge/HiveMetaStoreBridge.java  |   18 +-
 .../org/apache/atlas/hive/hook/HiveHook.java|2 +-
 .../java/org/apache/atlas/hive/HiveITBase.java  |4 +-
 .../hive/bridge/HiveMetaStoreBridgeTest.java|2 +-
 .../hive/bridge/HiveMetastoreBridgeIT.java  |8 +-
 .../org/apache/atlas/hive/hook/HiveHookIT.java  |   13 +-
 .../org/apache/atlas/sqoop/hook/SqoopHook.java  |2 +-
 .../apache/atlas/storm/hook/StormAtlasHook.java |2 +-
 .../atlas/storm/hook/StormAtlasHookIT.java  |2 +-
 .../main/java/org/apache/atlas/AtlasClient.java |   54 +-
 .../java/org/apache/atlas/EntityAuditEvent.java |   19 +-
 .../src/main/java/org/apache/atlas/SerDe.java   |   80 -
 .../java/org/apache/atlas/AtlasClientTest.java  |6 +-
 .../org/apache/atlas/repository/Constants.java  |7 +
 .../atlas/repository/graphdb/AtlasGraph.java|   12 +-
 .../graphdb/janus/AtlasJanusGraph.java  |6 +-
 .../repository/graphdb/titan0/Titan0Graph.java  |6 +-
 .../repository/graphdb/titan1/Titan1Graph.java  |6 +-
 .../v1/instance/AtlasSystemAttributes.java  |  123 ++
 .../org/apache/atlas/model/v1/instance/Id.java  |  171 +++
 .../atlas/model/v1/instance/Referenceable.java  |  201 +++
 .../apache/atlas/model/v1/instance/Struct.java  |  141 ++
 .../model/v1/typedef/AttributeDefinition.java   |  159 ++
 .../model/v1/typedef/ClassTypeDefinition.java   |   51 +
 .../model/v1/typedef/EnumTypeDefinition.java|  174 +++
 .../v1/typedef/HierarchicalTypeDefinition.java  |   96 ++
 .../atlas/model/v1/typedef/Multiplicity.java|  113 ++
 .../model/v1/typedef/StructTypeDefinition.java  |  119 ++
 .../model/v1/typedef/TraitTypeDefinition.java   |   51 +
 .../apache/atlas/model/v1/typedef/TypesDef.java |   91 ++
 .../java/org/apache/atlas/type/AtlasType.java   |  114 +-
 .../atlas/typesystem/types/DataTypes.java   |   33 +
 .../atlas/typesystem/types/utils/TypesUtil.java |  112 ++
 .../java/org/apache/atlas/hook/AtlasHook.java   |6 +-
 .../AbstractMessageDeserializer.java|   21 +-
 .../notification/AbstractNotification.java  |   12 +-
 .../notification/entity/EntityNotification.java |9 +-
 .../entity/EntityNotificationImpl.java  |   83 +-
 .../notification/hook/HookNotification.java |8 +-
 .../apache/atlas/kafka/KafkaConsumerTest.java   |   14 +-
 .../atlas/kafka/KafkaNotificationTest.java  |3 +-
 .../entity/EntityMessageDeserializerTest.java   |9 +-
 .../entity/EntityNotificationImplTest.java  |   50 +-
 .../hook/HookMessageDeserializerTest.java   |   11 +-
 .../notification/hook/HookNotificationTest.java |8 +-
 .../apache/atlas/GraphTransactionAdvisor.java   |   63 -
 .../atlas/discovery/DataSetLineageService.java  |  233 ---
 .../atlas/discovery/DiscoveryService.java   |   59 -
 .../atlas/discovery/EntityDiscoveryService.java |8 +-
 .../apache/atlas/discovery/SearchIndexer.java   |4 +-
 .../graph/DefaultGraphPersistenceStrategy.java  |  292 
 .../graph/GraphBackedDiscoveryService.java  |  269 
 .../gremlin/Gremlin2ExpressionFactory.java  |  379 -
 .../gremlin/Gremlin3ExpressionFactory.java  |  485 --
 .../atlas/gremlin/GremlinExpressionFactory.java |  658 -
 .../atlas/gremlin/optimizer/AliasFinder.java|  103 --
 .../gremlin/optimizer/CallHierarchyVisitor.java |   62 -
 .../optimizer/ExpandAndsOptimization.java   |  127 --
 .../optimizer/ExpandOrsOptimization.java|  588 
 .../gremlin/optimizer/ExpressionFinder.java |   69 -
 .../gremlin/optimizer/FunctionGenerator.java|  326 -
 .../gremlin/optimizer/GremlinOptimization.java  |   48 -
 .../optimizer/GremlinQueryOptimizer.java|  262 
 .../gremlin/optimizer/HasForbiddenType.java |   52 -
 .../apache/atlas/gremlin/optimizer/IsOr.java|   48 -
 .../atlas/gremlin/optimizer/IsOrParent.java |   60 -

[12/40] atlas git commit: ATLAS-2251: Remove TypeSystem and related implementation, to avoid unncessary duplicate of type details in cache

2017-11-05 Thread madhan
http://git-wip-us.apache.org/repos/asf/atlas/blob/0877e47c/repository/src/test/java/org/apache/atlas/utils/HiveModel.java
--
diff --git a/repository/src/test/java/org/apache/atlas/utils/HiveModel.java 
b/repository/src/test/java/org/apache/atlas/utils/HiveModel.java
deleted file mode 100644
index 4fc0473..000
--- a/repository/src/test/java/org/apache/atlas/utils/HiveModel.java
+++ /dev/null
@@ -1,303 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.atlas.utils;
-
-import org.apache.atlas.TestUtils;
-import org.apache.atlas.typesystem.ITypedReferenceableInstance;
-import org.apache.atlas.typesystem.Referenceable;
-import org.apache.atlas.typesystem.Struct;
-import org.apache.atlas.typesystem.persistence.Id;
-import org.apache.atlas.typesystem.types.ClassType;
-import org.apache.atlas.typesystem.types.Multiplicity;
-import org.apache.atlas.typesystem.types.TypeSystem;
-
-import java.lang.reflect.Field;
-import java.lang.reflect.Modifier;
-import java.util.ArrayList;
-import java.util.Collections;
-import java.util.Date;
-import java.util.List;
-
-/**
- * Allows easy creation of entities for classes in the hive test model.
- *
- */
-public class HiveModel {
-
-public static class StructInstance {
-
-public String getTypeName() {
-return getClass().getSimpleName();
-}
-
-public Struct toStruct() throws Exception {
-
-Struct entity = new Struct(getTypeName());
-addDeclaredFields(getClass(), entity);
-return entity;
-}
-
-protected void addDeclaredFields(Class clazz, Struct r) throws 
Exception {
-
-for (Field f : clazz.getDeclaredFields()) {
-
-if (Modifier.isTransient(f.getModifiers())) {
-continue;
-}
-String fieldName = f.getName();
-
-f.setAccessible(true);
-Object value = f.get(this);
-
-if (value instanceof List) {
-
-List listValue = (List) value;
-List toSet = new ArrayList(listValue.size());
-for (Object listItem : listValue) {
-Object toAdd = null;
-toAdd = convertValue(listItem);
-toSet.add(toAdd);
-}
-r.set(fieldName, toSet);
-} else {
-
-Object converted = convertValue(value);
-r.set(fieldName, converted);
-}
-}
-
-if (clazz != StructInstance.class) {
-addDeclaredFields(clazz.getSuperclass(), r);
-}
-}
-
-private Object convertValue(Object toConvert) throws Exception {
-
-if (toConvert instanceof ClassInstance) {
-return ((ClassInstance) toConvert).toReferenceable();
-}
-if (toConvert instanceof StructInstance) {
-return ((StructInstance) toConvert).toStruct();
-} else {
-return toConvert;
-}
-}
-}
-
-public static class ClassInstance extends StructInstance {
-
-private transient final Id guid;
-private transient List traits = new ArrayList();
-
-public T withTrait(String name) {
-traits.add(name);
-return getInstance();
-}
-
-public T withTraits(List names) {
-traits.addAll(names);
-return getInstance();
-}
-
-public T getInstance() {
-return (T) this;
-}
-
-public ClassInstance() {
-guid = new Id(getTypeName());
-}
-
-public Referenceable toReferenceable() throws Exception {
-
-String[] traitArray = new String[traits.size()];
-traitArray = traits.toArray(traitArray);
-Referenceable entity = new Referenceable(getTypeName(), 
traitArray);
-entity.replaceWithNewId(guid);
-

[38/40] atlas git commit: ATLAS-2251: Remove TypeSystem and related implementation, to avoid unncessary duplicate of type details in cache

2017-11-05 Thread madhan
http://git-wip-us.apache.org/repos/asf/atlas/blob/0877e47c/intg/src/main/java/org/apache/atlas/model/v1/instance/Referenceable.java
--
diff --git 
a/intg/src/main/java/org/apache/atlas/model/v1/instance/Referenceable.java 
b/intg/src/main/java/org/apache/atlas/model/v1/instance/Referenceable.java
new file mode 100644
index 000..df541b7
--- /dev/null
+++ b/intg/src/main/java/org/apache/atlas/model/v1/instance/Referenceable.java
@@ -0,0 +1,201 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.atlas.model.v1.instance;
+
+
+
+import org.codehaus.jackson.annotate.JsonAutoDetect;
+import org.codehaus.jackson.annotate.JsonIgnore;
+import org.codehaus.jackson.annotate.JsonIgnoreProperties;
+import org.codehaus.jackson.map.annotate.JsonSerialize;
+
+import javax.xml.bind.annotation.XmlAccessType;
+import javax.xml.bind.annotation.XmlAccessorType;
+import javax.xml.bind.annotation.XmlRootElement;
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+
+import static org.codehaus.jackson.annotate.JsonAutoDetect.Visibility.NONE;
+import static 
org.codehaus.jackson.annotate.JsonAutoDetect.Visibility.PUBLIC_ONLY;
+
+
+@JsonAutoDetect(getterVisibility=PUBLIC_ONLY, setterVisibility=PUBLIC_ONLY, 
fieldVisibility=NONE)
+@JsonSerialize(include=JsonSerialize.Inclusion.NON_NULL)
+@JsonIgnoreProperties(ignoreUnknown=true)
+@XmlRootElement
+@XmlAccessorType(XmlAccessType.PROPERTY)
+public class Referenceable extends Struct implements Serializable {
+private static final long serialVersionUID = 1L;
+
+private Idid;
+private Map   traits = new HashMap<>();
+private List  traitNames = new ArrayList<>();
+private AtlasSystemAttributes systemAttributes;
+
+
+public Referenceable() {
+super();
+}
+
+public Referenceable(Referenceable that) {
+super(that);
+
+if (that != null) {
+this.id = new Id(that.id);
+
+if (that.traits != null) {
+this.traits.putAll(that.traits);
+}
+
+if (that.traitNames != null) {
+this.traitNames.addAll(that.traitNames);
+}
+
+this.systemAttributes = new 
AtlasSystemAttributes(that.systemAttributes);
+}
+}
+
+public Referenceable(String typeName, String... traitNames) {
+super(typeName);
+
+this.id   = new Id(typeName);
+this.systemAttributes = null;
+
+if (traitNames != null) {
+for (String traitName : traitNames) {
+this.traitNames.add(traitName);
+this.traits.put(traitName, new Struct(traitName));
+}
+}
+}
+
+public Referenceable(String typeName, Map values) {
+this(new Id(typeName), typeName, values, null, null);
+}
+
+public Referenceable(String guid, String typeName, Map 
values) {
+this(new Id(guid, 0, typeName), typeName, values, null, null, null);
+}
+
+public Referenceable(String guid, String typeName, Map 
values, AtlasSystemAttributes systemAttributes) {
+this(new Id(guid, 0, typeName), typeName, values, systemAttributes, 
null, null);
+}
+
+public Referenceable(String guid, String typeName, Map 
values, AtlasSystemAttributes systemAttributes, List traitNames, 
Map traits) {
+this(new Id(guid, 0, typeName), typeName, values, systemAttributes, 
traitNames, traits);
+}
+
+public Referenceable(String guid, String typeName, Map 
values, List traitNames, Map traits) {
+this(new Id(guid, 0, typeName), typeName, values, null, traitNames, 
traits);
+}
+
+public Referenceable(Id id, String typeName, Map values, 
List traitNames, Map traits) {
+this(id, typeName, values, null, traitNames, traits);
+}
+
+public Referenceable(Id id, String typeName, Map values, 

[03/40] atlas git commit: ATLAS-2251: Remove TypeSystem and related implementation, to avoid unncessary duplicate of type details in cache

2017-11-05 Thread madhan
http://git-wip-us.apache.org/repos/asf/atlas/blob/0877e47c/webapp/src/main/java/org/apache/atlas/notification/NotificationEntityChangeListener.java
--
diff --git 
a/webapp/src/main/java/org/apache/atlas/notification/NotificationEntityChangeListener.java
 
b/webapp/src/main/java/org/apache/atlas/notification/NotificationEntityChangeListener.java
index 53acf56..308f18d 100644
--- 
a/webapp/src/main/java/org/apache/atlas/notification/NotificationEntityChangeListener.java
+++ 
b/webapp/src/main/java/org/apache/atlas/notification/NotificationEntityChangeListener.java
@@ -21,31 +21,20 @@ import com.google.common.annotations.VisibleForTesting;
 import org.apache.atlas.ApplicationProperties;
 import org.apache.atlas.AtlasException;
 import org.apache.atlas.listener.EntityChangeListener;
+import org.apache.atlas.model.v1.instance.Referenceable;
+import org.apache.atlas.model.v1.instance.Struct;
 import org.apache.atlas.notification.entity.EntityNotification;
 import org.apache.atlas.notification.entity.EntityNotificationImpl;
 import org.apache.atlas.repository.graph.GraphHelper;
-import org.apache.atlas.typesystem.IReferenceableInstance;
-import org.apache.atlas.typesystem.IStruct;
-import org.apache.atlas.typesystem.ITypedReferenceableInstance;
-import org.apache.atlas.typesystem.Referenceable;
-import org.apache.atlas.typesystem.Struct;
-import org.apache.atlas.typesystem.types.FieldMapping;
-import org.apache.atlas.typesystem.types.TraitType;
-import org.apache.atlas.typesystem.types.TypeSystem;
+import org.apache.atlas.type.AtlasClassificationType;
+import org.apache.atlas.type.AtlasTypeRegistry;
 import org.apache.commons.collections.CollectionUtils;
 import org.apache.commons.collections.MapUtils;
 import org.apache.commons.configuration.Configuration;
 import org.springframework.stereotype.Component;
 
 import javax.inject.Inject;
-import java.util.Arrays;
-import java.util.Collection;
-import java.util.Collections;
-import java.util.HashMap;
-import java.util.LinkedList;
-import java.util.List;
-import java.util.Map;
-import java.util.Set;
+import java.util.*;
 
 /**
  * Listen to the repository for entity changes and produce entity change 
notifications.
@@ -54,7 +43,7 @@ import java.util.Set;
 public class NotificationEntityChangeListener implements EntityChangeListener {
 
 private final NotificationInterface notificationInterface;
-private final TypeSystem typeSystem;
+private final AtlasTypeRegistrytypeRegistry;
 
 private Map notificationAttributesCache = new 
HashMap<>();
 private static final String ATLAS_ENTITY_NOTIFICATION_PROPERTY = 
"atlas.notification.entity";
@@ -68,44 +57,44 @@ public class NotificationEntityChangeListener implements 
EntityChangeListener {
  * Construct a NotificationEntityChangeListener.
  *
  * @param notificationInterface the notification framework interface
- * @param typeSystem the Atlas type system
+ * @param typeRegistry the Atlas type system
  */
 @Inject
-public NotificationEntityChangeListener(NotificationInterface 
notificationInterface, TypeSystem typeSystem) {
+public NotificationEntityChangeListener(NotificationInterface 
notificationInterface, AtlasTypeRegistry typeRegistry) {
 this.notificationInterface = notificationInterface;
-this.typeSystem = typeSystem;
+this.typeRegistry  = typeRegistry;
 }
 
 
 // - EntityChangeListener 
--
 
 @Override
-public void onEntitiesAdded(Collection 
entities, boolean isImport) throws AtlasException {
+public void onEntitiesAdded(Collection entities, boolean 
isImport) throws AtlasException {
 notifyOfEntityEvent(entities, 
EntityNotification.OperationType.ENTITY_CREATE);
 }
 
 @Override
-public void onEntitiesUpdated(Collection 
entities, boolean isImport) throws AtlasException {
+public void onEntitiesUpdated(Collection entities, boolean 
isImport) throws AtlasException {
 notifyOfEntityEvent(entities, 
EntityNotification.OperationType.ENTITY_UPDATE);
 }
 
 @Override
-public void onTraitsAdded(ITypedReferenceableInstance entity, Collection traits) throws AtlasException {
+public void onTraitsAdded(Referenceable entity, Collection traits) throws AtlasException {
 notifyOfEntityEvent(Collections.singleton(entity), 
EntityNotification.OperationType.TRAIT_ADD);
 }
 
 @Override
-public void onTraitsDeleted(ITypedReferenceableInstance entity, 
Collection traitNames) throws AtlasException {
+public void onTraitsDeleted(Referenceable entity, Collection 
traitNames) throws AtlasException {
 notifyOfEntityEvent(Collections.singleton(entity), 
EntityNotification.OperationType.TRAIT_DELETE);
 }
 
 @Override
-public void onTraitsUpdated(ITypedReferenceableInstance entity, 
Collection traits) throws 

[40/40] atlas git commit: misc

2017-11-05 Thread madhan
misc


Project: http://git-wip-us.apache.org/repos/asf/atlas/repo
Commit: http://git-wip-us.apache.org/repos/asf/atlas/commit/10c19eac
Tree: http://git-wip-us.apache.org/repos/asf/atlas/tree/10c19eac
Diff: http://git-wip-us.apache.org/repos/asf/atlas/diff/10c19eac

Branch: refs/heads/ATLAS-2251
Commit: 10c19eac7d3e8ee57122dc5b26dcfbd8d8548250
Parents: 0877e47
Author: Madhan Neethiraj <mad...@apache.org>
Authored: Sun Nov 5 12:29:56 2017 -0800
Committer: Madhan Neethiraj <mad...@apache.org>
Committed: Sun Nov 5 12:29:56 2017 -0800

--
 .../web/resources/DataSetLineageResource.java   | 185 +
 .../atlas/web/resources/LineageResource.java| 207 ++
 .../resources/MetadataDiscoveryResource.java| 405 +++
 3 files changed, 797 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/atlas/blob/10c19eac/webapp/src/main/java/org/apache/atlas/web/resources/DataSetLineageResource.java
--
diff --git 
a/webapp/src/main/java/org/apache/atlas/web/resources/DataSetLineageResource.java
 
b/webapp/src/main/java/org/apache/atlas/web/resources/DataSetLineageResource.java
new file mode 100644
index 000..278d7ca
--- /dev/null
+++ 
b/webapp/src/main/java/org/apache/atlas/web/resources/DataSetLineageResource.java
@@ -0,0 +1,185 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.atlas.web.resources;
+
+import org.apache.atlas.AtlasClient;
+import org.apache.atlas.utils.AtlasPerfTracer;
+import org.apache.atlas.web.util.Servlets;
+import org.codehaus.jettison.json.JSONObject;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.springframework.stereotype.Service;
+
+import javax.inject.Inject;
+import javax.inject.Singleton;
+import javax.servlet.http.HttpServletRequest;
+import javax.ws.rs.*;
+import javax.ws.rs.core.Context;
+import javax.ws.rs.core.Response;
+
+/**
+ * Jersey Resource for Hive Table Lineage.
+ */
+@Path("lineage/hive")
+@Singleton
+@Service
+@Deprecated
+public class DataSetLineageResource {
+
+private static final Logger LOG = 
LoggerFactory.getLogger(DataSetLineageResource.class);
+private static final Logger PERF_LOG = 
AtlasPerfTracer.getPerfLogger("rest.DataSetLineageResource");
+
+/**
+ * Created by the Guice ServletModule and injected with the
+ * configured LineageService.
+ *
+ */
+@Inject
+public DataSetLineageResource() {
+}
+
+/**
+ * Returns the inputs graph for a given entity.
+ *
+ * @param tableName table name
+ */
+@GET
+@Path("table/{tableName}/inputs/graph")
+@Consumes(Servlets.JSON_MEDIA_TYPE)
+@Produces(Servlets.JSON_MEDIA_TYPE)
+public Response inputsGraph(@Context HttpServletRequest request, 
@PathParam("tableName") String tableName) {
+if (LOG.isDebugEnabled()) {
+LOG.debug("==> DataSetLineageResource.inputsGraph({})", tableName);
+}
+
+AtlasPerfTracer perf = null;
+
+try {
+if (AtlasPerfTracer.isPerfTraceEnabled(PERF_LOG)) {
+perf = AtlasPerfTracer.getPerfTracer(PERF_LOG, 
"DataSetLineageResource.inputsGraph(tableName=" + tableName + ")");
+}
+
+final String jsonResult = ""; // TODO-typeSystem-removal: 
lineageService.getInputsGraph(tableName);
+
+JSONObject response = new JSONObject();
+response.put(AtlasClient.REQUEST_ID, Servlets.getRequestId());
+response.put("tableName", tableName);
+response.put(AtlasClient.RESULTS, new JSONObject(jsonResult));
+
+return Response.ok(response).build();
+} catch (IllegalArgumentException e) {
+LOG.error("Unable to get lineage inputs graph for table {}", 
tableName, e);
+throw new WebApplicationException(Servlets.getErrorResponse(e, 
Response.Status.BAD_REQUEST));
+} catch (WebApplicationExceptio

[31/40] atlas git commit: ATLAS-2251: Remove TypeSystem and related implementation, to avoid unncessary duplicate of type details in cache

2017-11-05 Thread madhan
http://git-wip-us.apache.org/repos/asf/atlas/blob/0877e47c/repository/src/main/java/org/apache/atlas/repository/graph/SoftDeleteHandler.java
--
diff --git 
a/repository/src/main/java/org/apache/atlas/repository/graph/SoftDeleteHandler.java
 
b/repository/src/main/java/org/apache/atlas/repository/graph/SoftDeleteHandler.java
deleted file mode 100644
index b37fe75..000
--- 
a/repository/src/main/java/org/apache/atlas/repository/graph/SoftDeleteHandler.java
+++ /dev/null
@@ -1,74 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- * 
- * http://www.apache.org/licenses/LICENSE-2.0
- * 
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.atlas.repository.graph;
-
-import org.apache.atlas.AtlasException;
-import org.apache.atlas.RequestContext;
-import org.apache.atlas.annotation.ConditionalOnAtlasProperty;
-import org.apache.atlas.repository.graphdb.AtlasEdge;
-import org.apache.atlas.repository.graphdb.AtlasVertex;
-import org.apache.atlas.typesystem.persistence.Id;
-import org.apache.atlas.typesystem.types.TypeSystem;
-import org.springframework.stereotype.Component;
-
-import javax.inject.Inject;
-
-import static 
org.apache.atlas.repository.Constants.MODIFICATION_TIMESTAMP_PROPERTY_KEY;
-import static org.apache.atlas.repository.Constants.MODIFIED_BY_KEY;
-import static org.apache.atlas.repository.Constants.STATE_PROPERTY_KEY;
-
-@Component
-@ConditionalOnAtlasProperty(property = "atlas.DeleteHandler.impl", isDefault = 
true)
-public class SoftDeleteHandler extends DeleteHandler {
-
-@Inject
-public SoftDeleteHandler(TypeSystem typeSystem) {
-super(typeSystem, false, true);
-}
-
-@Override
-protected void _deleteVertex(AtlasVertex instanceVertex, boolean force) {
-if (force) {
-graphHelper.removeVertex(instanceVertex);
-} else {
-Id.EntityState state = GraphHelper.getState(instanceVertex);
-if (state != Id.EntityState.DELETED) {
-GraphHelper.setProperty(instanceVertex, STATE_PROPERTY_KEY, 
Id.EntityState.DELETED.name());
-GraphHelper.setProperty(instanceVertex, 
MODIFICATION_TIMESTAMP_PROPERTY_KEY,
-RequestContext.get().getRequestTime());
-GraphHelper.setProperty(instanceVertex, MODIFIED_BY_KEY, 
RequestContext.get().getUser());
-}
-}
-}
-
-@Override
-protected void deleteEdge(AtlasEdge edge, boolean force) throws 
AtlasException {
-if (force) {
-graphHelper.removeEdge(edge);
-} else {
-Id.EntityState state = GraphHelper.getState(edge);
-if (state != Id.EntityState.DELETED) {
-GraphHelper.setProperty(edge, STATE_PROPERTY_KEY, 
Id.EntityState.DELETED.name());
-GraphHelper
-.setProperty(edge, 
MODIFICATION_TIMESTAMP_PROPERTY_KEY, RequestContext.get().getRequestTime());
-GraphHelper.setProperty(edge, MODIFIED_BY_KEY, 
RequestContext.get().getUser());
-}
-}
-}
-}

http://git-wip-us.apache.org/repos/asf/atlas/blob/0877e47c/repository/src/main/java/org/apache/atlas/repository/graph/TypedInstanceToGraphMapper.java
--
diff --git 
a/repository/src/main/java/org/apache/atlas/repository/graph/TypedInstanceToGraphMapper.java
 
b/repository/src/main/java/org/apache/atlas/repository/graph/TypedInstanceToGraphMapper.java
deleted file mode 100644
index 1fb4ee9..000
--- 
a/repository/src/main/java/org/apache/atlas/repository/graph/TypedInstanceToGraphMapper.java
+++ /dev/null
@@ -1,941 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the 

[28/40] atlas git commit: ATLAS-2251: Remove TypeSystem and related implementation, to avoid unncessary duplicate of type details in cache

2017-11-05 Thread madhan
http://git-wip-us.apache.org/repos/asf/atlas/blob/0877e47c/repository/src/main/scala/org/apache/atlas/query/Expressions.scala
--
diff --git a/repository/src/main/scala/org/apache/atlas/query/Expressions.scala 
b/repository/src/main/scala/org/apache/atlas/query/Expressions.scala
deleted file mode 100644
index bf9efd2..000
--- a/repository/src/main/scala/org/apache/atlas/query/Expressions.scala
+++ /dev/null
@@ -1,923 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.atlas.query
-
-import com.google.common.collect.ImmutableCollection
-import org.apache.atlas.AtlasException
-import org.apache.atlas.typesystem.types.DataTypes.{ArrayType, PrimitiveType, 
TypeCategory}
-import org.apache.atlas.typesystem.types._
-import scala.collection.JavaConverters._
-
-object Expressions {
-
-import TypeUtils._
-
-class ExpressionException(val e: Expression, message: String, cause: 
Throwable, enableSuppression: Boolean,
-  writableStackTrace: Boolean)
-extends AtlasException(message, cause, enableSuppression, 
writableStackTrace) {
-
-def this(e: Expression, message: String) {
-this(e, message, null, false, true)
-}
-
-def this(e: Expression, message: String, cause: Throwable) {
-this(e, message, cause, false, true)
-}
-
-def this(e: Expression, cause: Throwable) {
-this(e, null, cause, false, true)
-}
-
-override def getMessage: String = {
-val eString = e.toString
-s"${super.getMessage}, expression:${if (eString contains "\n") 
"\n" else " "}$e"
-}
-
-}
-
-class UnresolvedException(expr: Expression, function: String) extends
-ExpressionException(expr, s"Unresolved $function")
-
-def attachExpression[A](e: Expression, msg: String = "")(f: => A): A = {
-try f catch {
-case eex: ExpressionException => throw eex
-case ex: Exception => throw new ExpressionException(e, msg, ex)
-}
-}
-
-trait Expression {
-self: Product =>
-def isAggregator = false
-def children: Seq[Expression]
-
-/**
- * Returns `true` if the schema for this expression and all its 
children have been resolved.
- * The default logic is that an Expression is resolve if all its 
children are resolved.
- */
-lazy val resolved: Boolean = childrenResolved
-
-/**
- * Returns the output [[IDataType[_]] of this expression.  Expressions 
that are unresolved will
- * throw if this method is invoked.
- */
-def dataType: IDataType[_]
-
-/**
- * Returns true if  all the children have been resolved.
- */
-def childrenResolved = !children.exists(!_.resolved)
-
-
-/**
- * the aliases that are present in this Expression Tree
- */
-def namedExpressions: Map[String, Expression] = Map()
-
-def fastEquals(other: Expression): Boolean = {
-this.eq(other) || this == other
-}
-
-def makeCopy(newArgs: Array[AnyRef]): this.type = 
attachExpression(this, "makeCopy") {
-try {
-val defaultCtor = 
getClass.getConstructors.find(_.getParameterTypes.size != 0).head
-defaultCtor.newInstance(newArgs: _*).asInstanceOf[this.type]
-} catch {
-case e: java.lang.IllegalArgumentException =>
-throw new ExpressionException(
-this, s"Failed to copy node. Reason: ${e.getMessage}.")
-}
-}
-
-def transformChildrenDown(rule: PartialFunction[Expression, 
Expression]): this.type = {
-var changed = false
-val newArgs = productIterator.map {
-case arg: Expression if children contains arg =>
-val newChild = 
arg.asInstanceOf[Expression].transformDown(rule)
-if (!(newChild fastEquals arg)) {
-changed = true
-newChild
-} else {
-  

[25/40] atlas git commit: ATLAS-2251: Remove TypeSystem and related implementation, to avoid unncessary duplicate of type details in cache

2017-11-05 Thread madhan
http://git-wip-us.apache.org/repos/asf/atlas/blob/0877e47c/repository/src/test/java/org/apache/atlas/discovery/DataSetLineageServiceTest.java
--
diff --git 
a/repository/src/test/java/org/apache/atlas/discovery/DataSetLineageServiceTest.java
 
b/repository/src/test/java/org/apache/atlas/discovery/DataSetLineageServiceTest.java
deleted file mode 100644
index 3db58fe..000
--- 
a/repository/src/test/java/org/apache/atlas/discovery/DataSetLineageServiceTest.java
+++ /dev/null
@@ -1,497 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.atlas.discovery;
-
-import com.google.common.collect.ImmutableList;
-import org.apache.atlas.AtlasException;
-import org.apache.atlas.BaseRepositoryTest;
-import org.apache.atlas.TestModules;
-import org.apache.atlas.TestUtils;
-import org.apache.atlas.model.legacy.EntityResult;
-import org.apache.atlas.query.QueryParams;
-import org.apache.atlas.typesystem.ITypedReferenceableInstance;
-import org.apache.atlas.typesystem.Referenceable;
-import org.apache.atlas.typesystem.Struct;
-import org.apache.atlas.typesystem.exception.EntityNotFoundException;
-import org.apache.atlas.typesystem.json.InstanceSerialization;
-import org.apache.atlas.typesystem.persistence.Id;
-import org.apache.commons.collections.ArrayStack;
-import org.codehaus.jettison.json.JSONArray;
-import org.codehaus.jettison.json.JSONException;
-import org.codehaus.jettison.json.JSONObject;
-import org.testng.Assert;
-import org.testng.annotations.AfterClass;
-import org.testng.annotations.BeforeClass;
-import org.testng.annotations.DataProvider;
-import org.testng.annotations.Guice;
-import org.testng.annotations.Test;
-
-import javax.inject.Inject;
-import java.util.Arrays;
-import java.util.List;
-import java.util.Map;
-
-import static org.testng.Assert.assertEquals;
-import static org.testng.Assert.assertNotNull;
-import static org.testng.Assert.assertTrue;
-import static org.testng.Assert.fail;
-
-/**
- * Unit tests for Hive LineageService.
- */
-@Guice(modules = TestModules.TestOnlyModule.class)
-public class DataSetLineageServiceTest extends BaseRepositoryTest {
-
-@Inject
-private DiscoveryService discoveryService;
-
-@Inject
-private DataSetLineageService lineageService;
-
-@BeforeClass
-public void setUp() throws Exception {
-super.setUp();
-}
-
-@AfterClass
-public void tearDown() throws Exception {
-super.tearDown();
-}
-
-@DataProvider(name = "dslQueriesProvider")
-private Object[][] createDSLQueries() {
-return new String[][]{
-// joins
-{"hive_table where name=\"sales_fact\", columns"},
-{"hive_table where name=\"sales_fact\", columns select name, 
dataType, comment"},
-{"hive_table where name=\"sales_fact\", columns as c select 
c.name, c.dataType, c.comment"},
-//{"hive_db as db where (db.name=\"Reporting\"), 
hive_table as table select db.name,
-// table.name"},
-{"from hive_db"}, {"hive_db"}, {"hive_db where 
hive_db.name=\"Reporting\""},
-{"hive_db hive_db.name = \"Reporting\""},
-{"hive_db where hive_db.name=\"Reporting\" select name, owner"}, 
{"hive_db has name"},
-//{"hive_db, hive_table"},
-//{"hive_db, hive_process has name"},
-//{"hive_db as db1, hive_table where db1.name = 
\"Reporting\""},
-//{"hive_db where hive_db.name=\"Reporting\" and 
hive_db.createTime < " + System
-// .currentTimeMillis()},
-{"from hive_table"}, {"hive_table"}, {"hive_table is Dimension"},
-{"hive_column where hive_column isa PII"},
-//{"hive_column where hive_column isa PII select 
hive_column.name"},
-{"hive_column select hive_column.name"}, {"hive_column select 
name"},
-{"hive_column where hive_column.name=\"customer_id\""}, {"from 
hive_table select hive_table.name"},
-{"hive_db where (name = \"Reporting\")"},
-{"hive_db where (name = \"Reporting\") select name as 

[30/40] atlas git commit: ATLAS-2251: Remove TypeSystem and related implementation, to avoid unncessary duplicate of type details in cache

2017-11-05 Thread madhan
http://git-wip-us.apache.org/repos/asf/atlas/blob/0877e47c/repository/src/main/java/org/apache/atlas/repository/memory/HierarchicalTypeStore.java
--
diff --git 
a/repository/src/main/java/org/apache/atlas/repository/memory/HierarchicalTypeStore.java
 
b/repository/src/main/java/org/apache/atlas/repository/memory/HierarchicalTypeStore.java
deleted file mode 100755
index 429730c..000
--- 
a/repository/src/main/java/org/apache/atlas/repository/memory/HierarchicalTypeStore.java
+++ /dev/null
@@ -1,207 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.atlas.repository.memory;
-
-import com.google.common.collect.ImmutableBiMap;
-import com.google.common.collect.ImmutableList;
-import com.google.common.collect.ImmutableMap;
-import com.google.common.collect.Lists;
-import org.apache.atlas.repository.RepositoryException;
-import org.apache.atlas.typesystem.persistence.Id;
-import org.apache.atlas.typesystem.persistence.ReferenceableInstance;
-import org.apache.atlas.typesystem.persistence.StructInstance;
-import org.apache.atlas.typesystem.types.AttributeInfo;
-import org.apache.atlas.typesystem.types.HierarchicalType;
-import org.apache.atlas.typesystem.types.IConstructableType;
-
-import java.util.ArrayList;
-import java.util.HashMap;
-import java.util.List;
-import java.util.Map;
-import java.util.Set;
-import java.util.concurrent.locks.ReentrantReadWriteLock;
-
-@Deprecated
-public abstract class HierarchicalTypeStore {
-
-final MemRepository repository;
-final IConstructableType hierarchicalType;
-final ArrayList typeNameList;
-final ImmutableMap attrStores;
-final ImmutableList superTypeStores;
-
-
-/**
- * Map Id to position in storage lists.
- */
-Map idPosMap;
-
-List freePositions;
-
-int nextPos;
-
-/**
- * Lock for each Class/Trait.
- */
-ReentrantReadWriteLock lock;
-
-HierarchicalTypeStore(MemRepository repository, HierarchicalType 
hierarchicalType) throws RepositoryException {
-this.hierarchicalType = (IConstructableType) hierarchicalType;
-this.repository = repository;
-ImmutableMap.Builder b =
-new ImmutableBiMap.Builder<>();
-typeNameList = Lists.newArrayList((String) null);
-ImmutableList l = hierarchicalType.immediateAttrs;
-for (AttributeInfo i : l) {
-b.put(i, AttributeStores.createStore(i));
-}
-attrStores = b.build();
-
-ImmutableList.Builder b1 = new 
ImmutableList.Builder<>();
-Set allSuperTypeNames = 
hierarchicalType.getAllSuperTypeNames();
-for (String s : allSuperTypeNames) {
-b1.add(repository.getStore(s));
-}
-superTypeStores = b1.build();
-
-nextPos = 0;
-idPosMap = new HashMap<>();
-freePositions = new ArrayList<>();
-
-lock = new ReentrantReadWriteLock();
-}
-
-/**
- * Assign a storage position to an Id.
- * - try to assign from freePositions
- * - ensure storage capacity.
- * - add entry in idPosMap.
- * @param id
- * @return
- * @throws RepositoryException
- */
-int assignPosition(Id id) throws RepositoryException {
-
-int pos = -1;
-if (!freePositions.isEmpty()) {
-pos = freePositions.remove(0);
-} else {
-pos = nextPos++;
-ensureCapacity(pos);
-}
-
-idPosMap.put(id, pos);
-
-for (HierarchicalTypeStore s : superTypeStores) {
-s.assignPosition(id);
-}
-
-return pos;
-}
-
-/**
- * - remove from idPosMap
- * - add to freePositions.
- * @throws RepositoryException
- */
-void releaseId(Id id) {
-
-Integer pos = idPosMap.get(id);
-if (pos != null) {
-idPosMap.remove(id);
-freePositions.add(pos);
-
-for (HierarchicalTypeStore s : superTypeStores) {
-s.releaseId(id);
-}
-}
-}
-
-void acquireReadLock() {
-lock.readLock().lock();
-  

[2/3] atlas git commit: ATLAS-2251: notification module updates

2017-11-06 Thread madhan
http://git-wip-us.apache.org/repos/asf/atlas/blob/64e739da/notification/src/main/java/org/apache/atlas/notification/AbstractNotification.java
--
diff --git 
a/notification/src/main/java/org/apache/atlas/notification/AbstractNotification.java
 
b/notification/src/main/java/org/apache/atlas/notification/AbstractNotification.java
index 988d98a..8bc7cb4 100644
--- 
a/notification/src/main/java/org/apache/atlas/notification/AbstractNotification.java
+++ 
b/notification/src/main/java/org/apache/atlas/notification/AbstractNotification.java
@@ -18,17 +18,19 @@
 package org.apache.atlas.notification;
 
 import com.google.common.annotations.VisibleForTesting;
-import com.google.gson.Gson;
-import com.google.gson.GsonBuilder;
 import com.google.gson.JsonElement;
 import com.google.gson.JsonParser;
 import com.google.gson.JsonSerializationContext;
 import com.google.gson.JsonSerializer;
 import org.apache.atlas.AtlasException;
 import org.apache.atlas.ha.HAConfiguration;
+import org.apache.atlas.model.notification.AtlasNotificationBaseMessage;
+import org.apache.atlas.model.notification.AtlasNotificationMessage;
+import org.apache.atlas.model.notification.AtlasNotificationStringMessage;
 import org.apache.atlas.v1.model.instance.Referenceable;
-import 
org.apache.atlas.notification.AtlasNotificationBaseMessage.CompressionKind;
+import 
org.apache.atlas.model.notification.AtlasNotificationBaseMessage.CompressionKind;
 import org.apache.atlas.type.AtlasType;
+import org.apache.atlas.model.notification.MessageVersion;
 import org.apache.commons.configuration.Configuration;
 import org.apache.commons.lang.StringUtils;
 import org.codehaus.jettison.json.JSONArray;
@@ -44,8 +46,8 @@ import java.util.List;
 import java.util.UUID;
 import java.util.concurrent.atomic.AtomicInteger;
 
-import static 
org.apache.atlas.notification.AtlasNotificationBaseMessage.MESSAGE_COMPRESSION_ENABLED;
-import static 
org.apache.atlas.notification.AtlasNotificationBaseMessage.MESSAGE_MAX_LENGTH_BYTES;
+import static 
org.apache.atlas.model.notification.AtlasNotificationBaseMessage.MESSAGE_COMPRESSION_ENABLED;
+import static 
org.apache.atlas.model.notification.AtlasNotificationBaseMessage.MESSAGE_MAX_LENGTH_BYTES;
 
 /**
  * Abstract notification interface implementation.
@@ -78,14 +80,6 @@ public abstract class AbstractNotification implements 
NotificationInterface {
 private final boolean embedded;
 private final boolean isHAEnabled;
 
-/**
- * Used for message serialization.
- */
-public static final Gson GSON = new GsonBuilder().
-registerTypeAdapter(Referenceable.class, new 
ReferenceableSerializer()).
-registerTypeAdapter(JSONArray.class, new JSONArraySerializer()).
-create();
-
 // - Constructors 
 
 public AbstractNotification(Configuration applicationProperties) throws 
AtlasException {
@@ -158,7 +152,7 @@ public abstract class AbstractNotification implements 
NotificationInterface {
 public static String getMessageJson(Object message) {
 AtlasNotificationMessage notificationMsg = new 
AtlasNotificationMessage<>(CURRENT_MESSAGE_VERSION, message);
 
-return GSON.toJson(notificationMsg);
+return AtlasType.toV1Json(notificationMsg);
 }
 
 private static String getHostAddress() {
@@ -188,7 +182,7 @@ public abstract class AbstractNotification implements 
NotificationInterface {
  */
 public static void createNotificationMessages(Object message, List 
msgJsonList) {
 AtlasNotificationMessage notificationMsg = new 
AtlasNotificationMessage<>(CURRENT_MESSAGE_VERSION, message, getHostAddress(), 
getCurrentUser());
-String  msgJson = 
GSON.toJson(notificationMsg);
+String  msgJson = 
AtlasType.toV1Json(notificationMsg);
 
 boolean msgLengthExceedsLimit = (msgJson.length() * 
MAX_BYTES_PER_CHAR) > MESSAGE_MAX_LENGTH_BYTES;
 
@@ -213,7 +207,7 @@ public abstract class AbstractNotification implements 
NotificationInterface {
 if (!msgLengthExceedsLimit) { // no need to split
 AtlasNotificationStringMessage compressedMsg = new 
AtlasNotificationStringMessage(encodedBytes, msgId, compressionKind);
 
-msgJson  = GSON.toJson(compressedMsg); // msgJson will 
not have multi-byte characters here, due to use of encodeBase64() above
+msgJson  = AtlasType.toV1Json(compressedMsg); // 
msgJson will not have multi-byte characters here, due to use of encodeBase64() 
above
 msgBytes = null; // not used after this point
 } else { // encodedBytes will be split
 msgJson  = null; // not used after this point
@@ -239,7 +233,7 @@ public abstract class AbstractNotification implements 
NotificationInterface 

[3/3] atlas git commit: ATLAS-2251: notification module updates

2017-11-06 Thread madhan
ATLAS-2251: notification module updates


Project: http://git-wip-us.apache.org/repos/asf/atlas/repo
Commit: http://git-wip-us.apache.org/repos/asf/atlas/commit/64e739da
Tree: http://git-wip-us.apache.org/repos/asf/atlas/tree/64e739da
Diff: http://git-wip-us.apache.org/repos/asf/atlas/diff/64e739da

Branch: refs/heads/ATLAS-2251
Commit: 64e739da7e3bb36398c5ceb966016b435ae76a00
Parents: 3f44770
Author: Madhan Neethiraj <mad...@apache.org>
Authored: Sun Nov 5 15:05:12 2017 -0800
Committer: Madhan Neethiraj <mad...@apache.org>
Committed: Mon Nov 6 14:49:49 2017 -0800

--
 .../apache/atlas/falcon/hook/FalconHook.java|   2 +-
 .../atlas/hbase/bridge/HBaseAtlasHook.java  |   2 +-
 .../hbase/model/HBaseOperationContext.java  |   2 +-
 .../org/apache/atlas/hive/hook/HiveHook.java|   2 +-
 .../org/apache/atlas/sqoop/hook/SqoopHook.java  |   2 +-
 .../org/apache/atlas/AtlasConfiguration.java|  83 
 .../org/apache/atlas/AtlasConfiguration.java|  83 
 .../AtlasNotificationBaseMessage.java   | 208 ++
 .../notification/AtlasNotificationMessage.java  | 101 +
 .../AtlasNotificationStringMessage.java |  82 
 .../model/notification/MessageVersion.java  | 170 
 .../java/org/apache/atlas/type/AtlasType.java   |  62 ++-
 .../atlas/typesystem/types/DataTypes.java   |   4 +-
 .../model/instance/AtlasSystemAttributes.java   |  21 +
 .../org/apache/atlas/v1/model/instance/Id.java  |  19 +
 .../atlas/v1/model/instance/Referenceable.java  |  27 ++
 .../apache/atlas/v1/model/instance/Struct.java  |  21 +
 .../model/notification/EntityNotification.java  | 218 ++
 .../v1/model/notification/HookNotification.java | 415 +++
 .../apache/atlas/v1/model/typedef/TypesDef.java |  27 ++
 .../java/org/apache/atlas/hook/AtlasHook.java   |   2 +-
 .../apache/atlas/kafka/AtlasKafkaConsumer.java  |  18 +-
 .../apache/atlas/kafka/KafkaNotification.java   |   5 +-
 .../AbstractMessageDeserializer.java| 129 +-
 .../notification/AbstractNotification.java  |  28 +-
 .../AbstractNotificationConsumer.java   |  19 +-
 .../AtlasNotificationBaseMessage.java   | 194 -
 .../notification/AtlasNotificationMessage.java  |  87 
 .../AtlasNotificationMessageDeserializer.java   |  59 +--
 .../AtlasNotificationStringMessage.java |  66 ---
 .../atlas/notification/MessageVersion.java  | 141 ---
 .../notification/NotificationInterface.java |  39 +-
 .../notification/SplitMessageAggregator.java|   2 +
 .../entity/EntityMessageDeserializer.java   |  38 +-
 .../notification/entity/EntityNotification.java |  66 ---
 .../entity/EntityNotificationImpl.java  | 157 ---
 .../hook/HookMessageDeserializer.java   |  19 +-
 .../notification/hook/HookNotification.java | 275 
 .../org/apache/atlas/hook/AtlasHookTest.java|   2 +-
 .../apache/atlas/kafka/KafkaConsumerTest.java   |  17 +-
 .../atlas/kafka/KafkaNotificationTest.java  |   4 +-
 .../AbstractNotificationConsumerTest.java   | 103 +++--
 .../notification/AbstractNotificationTest.java  |   7 +-
 .../AtlasNotificationMessageTest.java   |   2 +
 .../atlas/notification/MessageVersionTest.java  |   1 +
 .../SplitMessageAggregatorTest.java |   3 +-
 .../entity/EntityMessageDeserializerTest.java   |  12 +-
 .../entity/EntityNotificationImplTest.java  | 147 ---
 .../entity/EntityNotificationTest.java  | 148 +++
 .../hook/HookMessageDeserializerTest.java   |  16 +-
 .../notification/hook/HookNotificationTest.java |  12 +-
 .../test/java/org/apache/atlas/DBSandboxer.java |  53 +++
 .../NotificationEntityChangeListener.java   |  27 +-
 .../notification/NotificationHookConsumer.java  |  16 +-
 .../notification/EntityNotificationIT.java  |   2 +-
 .../NotificationHookConsumerIT.java |  12 +-
 .../NotificationHookConsumerKafkaTest.java  |   4 +-
 .../NotificationHookConsumerTest.java   |   2 +-
 .../atlas/web/integration/BaseResourceIT.java   |   2 +-
 59 files changed, 1893 insertions(+), 1594 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/atlas/blob/64e739da/addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/hook/FalconHook.java
--
diff --git 
a/addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/hook/FalconHook.java
 
b/addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/hook/FalconHook.java
index e9b9765..5912cb0 100644
--- 
a/addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/hook/FalconHook.java
+++ 
b/addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/hook/FalconHook.java
@@ -26,7 +26,7 @@ import org.apache.atlas.falcon.publisher.FalconEventPublis

[1/3] atlas git commit: ATLAS-2251: notification module updates

2017-11-06 Thread madhan
Repository: atlas
Updated Branches:
  refs/heads/ATLAS-2251 3f44770da -> 64e739da7


http://git-wip-us.apache.org/repos/asf/atlas/blob/64e739da/notification/src/test/java/org/apache/atlas/notification/entity/EntityNotificationImplTest.java
--
diff --git 
a/notification/src/test/java/org/apache/atlas/notification/entity/EntityNotificationImplTest.java
 
b/notification/src/test/java/org/apache/atlas/notification/entity/EntityNotificationImplTest.java
deleted file mode 100644
index 81a0784..000
--- 
a/notification/src/test/java/org/apache/atlas/notification/entity/EntityNotificationImplTest.java
+++ /dev/null
@@ -1,147 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.atlas.notification.entity;
-
-import org.apache.atlas.v1.model.instance.Referenceable;
-import org.apache.atlas.v1.model.instance.Struct;
-import org.apache.atlas.type.AtlasClassificationType;
-import org.apache.atlas.type.AtlasTypeRegistry;
-import org.testng.annotations.Test;
-
-import java.util.Collections;
-import java.util.HashMap;
-import java.util.LinkedList;
-import java.util.List;
-import java.util.Map;
-import java.util.Set;
-
-import static org.mockito.Mockito.mock;
-import static org.mockito.Mockito.when;
-import static org.testng.Assert.assertEquals;
-import static org.testng.Assert.assertTrue;
-
-/**
- * EntityNotificationImpl tests.
- */
-public class EntityNotificationImplTest {
-
-@Test
-public void testGetEntity() throws Exception {
-Referenceable entity = getEntity("id");
-
-EntityNotificationImpl entityNotification =
-new EntityNotificationImpl(entity, 
EntityNotification.OperationType.ENTITY_CREATE,
-Collections.emptyList());
-
-assertEquals(entity, entityNotification.getEntity());
-}
-
-@Test
-public void testGetOperationType() throws Exception {
-Referenceable entity = getEntity("id");
-
-EntityNotificationImpl entityNotification =
-new EntityNotificationImpl(entity, 
EntityNotification.OperationType.ENTITY_CREATE,
-Collections.emptyList());
-
-assertEquals(EntityNotification.OperationType.ENTITY_CREATE, 
entityNotification.getOperationType());
-}
-
-@Test
-public void testGetAllTraits() throws Exception {
-Referenceable entity = getEntity("id");
-String traitName = "MyTrait";
-List traitInfo = new LinkedList<>();
-Struct trait = new Struct(traitName, Collections.emptyMap());
-traitInfo.add(trait);
-
-EntityNotificationImpl entityNotification =
-new EntityNotificationImpl(entity, 
EntityNotification.OperationType.TRAIT_ADD, traitInfo);
-
-assertEquals(traitInfo, entityNotification.getAllTraits());
-}
-
-@Test
-public void testGetAllTraitsSuperTraits() throws Exception {
-AtlasTypeRegistry typeRegistry = mock(AtlasTypeRegistry.class);
-
-String traitName = "MyTrait";
-Struct myTrait = new Struct(traitName);
-
-String superTraitName = "MySuperTrait";
-
-AtlasClassificationType traitType = 
mock(AtlasClassificationType.class);
-Set superTypeNames = Collections.singleton(superTraitName);
-
-AtlasClassificationType superTraitType = 
mock(AtlasClassificationType.class);
-Set superSuperTypeNames = Collections.emptySet();
-
-Referenceable entity = getEntity("id", myTrait);
-
-
when(typeRegistry.getClassificationTypeByName(traitName)).thenReturn(traitType);
-
when(typeRegistry.getClassificationTypeByName(superTraitName)).thenReturn(superTraitType);
-
-when(traitType.getAllSuperTypes()).thenReturn(superTypeNames);
-
when(superTraitType.getAllSuperTypes()).thenReturn(superSuperTypeNames);
-
-EntityNotificationImpl entityNotification =
-new EntityNotificationImpl(entity, 
EntityNotification.OperationType.TRAIT_ADD, typeRegistry);
-
-List allTraits = entityNotification.getAllTraits();
-
-assertEquals(2, allTraits.size());
-
-for (Struct trait : allTraits) {
-String typeName = 

[1/4] atlas git commit: ATLAS-2251: notification module updates (#4)

2017-11-07 Thread madhan
Repository: atlas
Updated Branches:
  refs/heads/ATLAS-2251 84f1349df -> f01e46d73


http://git-wip-us.apache.org/repos/asf/atlas/blob/f01e46d7/webapp/src/test/java/org/apache/atlas/web/integration/BaseResourceIT.java
--
diff --git 
a/webapp/src/test/java/org/apache/atlas/web/integration/BaseResourceIT.java 
b/webapp/src/test/java/org/apache/atlas/web/integration/BaseResourceIT.java
index e63099d..ab27612 100755
--- a/webapp/src/test/java/org/apache/atlas/web/integration/BaseResourceIT.java
+++ b/webapp/src/test/java/org/apache/atlas/web/integration/BaseResourceIT.java
@@ -19,8 +19,6 @@
 package org.apache.atlas.web.integration;
 
 import com.google.common.base.Preconditions;
-import com.google.common.collect.ImmutableList;
-import com.google.common.collect.ImmutableSet;
 import org.apache.atlas.ApplicationProperties;
 import org.apache.atlas.AtlasClient;
 import org.apache.atlas.AtlasClientV2;
@@ -43,7 +41,7 @@ import org.apache.atlas.v1.model.typedef.*;
 import org.apache.atlas.v1.model.typedef.EnumTypeDefinition.EnumValue;
 import org.apache.atlas.notification.NotificationConsumer;
 import org.apache.atlas.kafka.*;
-import org.apache.atlas.v1.model.notification.EntityNotification;
+import org.apache.atlas.v1.model.notification.EntityNotificationV1;
 import org.apache.atlas.type.AtlasType;
 import org.apache.atlas.type.AtlasTypeUtil;
 import org.apache.atlas.v1.typesystem.types.utils.TypesUtil;
@@ -75,31 +73,30 @@ import static org.testng.Assert.assertTrue;
  * Sets up the web resource and has helper methods to created type and entity.
  */
 public abstract class BaseResourceIT {
+public static final Logger LOG = 
LoggerFactory.getLogger(BaseResourceIT.class);
 
 public static final String ATLAS_REST_ADDRESS = "atlas.rest.address";
-public static final String NAME = "name";
-public static final String QUALIFIED_NAME = "qualifiedName";
-public static final String CLUSTER_NAME = "clusterName";
-public static final String DESCRIPTION = "description";
-public static final String PII_TAG = "pii_Tag";
-public static final String PHI_TAG = "phi_Tag";
-public static final String PCI_TAG = "pci_Tag";
-public static final String SOX_TAG = "sox_Tag";
-public static final String SEC_TAG = "sec_Tag";
-public static final String FINANCE_TAG = "finance_Tag";
-public static final String CLASSIFICATION = "classification";
+public static final String NAME   = "name";
+public static final String QUALIFIED_NAME = "qualifiedName";
+public static final String CLUSTER_NAME   = "clusterName";
+public static final String DESCRIPTION= "description";
+public static final String PII_TAG= "pii_Tag";
+public static final String PHI_TAG= "phi_Tag";
+public static final String PCI_TAG= "pci_Tag";
+public static final String SOX_TAG= "sox_Tag";
+public static final String SEC_TAG= "sec_Tag";
+public static final String FINANCE_TAG= "finance_Tag";
+public static final String CLASSIFICATION = "classification";
+
+protected static final int MAX_WAIT_TIME = 6;
 
 // All service clients
-protected AtlasClient atlasClientV1;
+protected AtlasClient   atlasClientV1;
 protected AtlasClientV2 atlasClientV2;
-
-public static final Logger LOG = 
LoggerFactory.getLogger(BaseResourceIT.class);
-protected static final int MAX_WAIT_TIME = 6;
-protected String[] atlasUrls;
+protected String[]  atlasUrls;
 
 @BeforeClass
 public void setUp() throws Exception {
-
 //set high timeouts so that tests do not fail due to read timeouts 
while you
 //are stepping through the code in a debugger
 
ApplicationProperties.get().setProperty("atlas.client.readTimeoutMSecs", 
"1");
@@ -107,6 +104,7 @@ public abstract class BaseResourceIT {
 
 
 Configuration configuration = ApplicationProperties.get();
+
 atlasUrls = configuration.getStringArray(ATLAS_REST_ADDRESS);
 
 if (atlasUrls == null || atlasUrls.length == 0) {
@@ -124,6 +122,7 @@ public abstract class BaseResourceIT {
 
 protected void batchCreateTypes(AtlasTypesDef typesDef) throws 
AtlasServiceException {
 AtlasTypesDef toCreate = new AtlasTypesDef();
+
 for (AtlasEnumDef enumDef : typesDef.getEnumDefs()) {
 if (atlasClientV2.typeWithNameExists(enumDef.getName())) {
 LOG.warn("Type with name {} already exists. Skipping", 
enumDef.getName());
@@ -165,10 +164,10 @@ public abstract class BaseResourceIT {
 }
 
 protected List createType(TypesDef typesDef) throws Exception {
-List enumTypes = new ArrayList<>();
+List   enumTypes   = new ArrayList<>();
 List structTypes = new ArrayList<>();
-List traitTypes = new ArrayList<>();
-List classTypes = new 

[3/4] atlas git commit: ATLAS-2251: notification module updates (#4)

2017-11-07 Thread madhan
http://git-wip-us.apache.org/repos/asf/atlas/blob/f01e46d7/notification/src/main/java/org/apache/atlas/hook/AtlasHook.java
--
diff --git a/notification/src/main/java/org/apache/atlas/hook/AtlasHook.java 
b/notification/src/main/java/org/apache/atlas/hook/AtlasHook.java
index 3bc4fba..bf6a36c 100644
--- a/notification/src/main/java/org/apache/atlas/hook/AtlasHook.java
+++ b/notification/src/main/java/org/apache/atlas/hook/AtlasHook.java
@@ -21,22 +21,20 @@ package org.apache.atlas.hook;
 import com.google.common.annotations.VisibleForTesting;
 import org.apache.atlas.ApplicationProperties;
 import org.apache.atlas.kafka.NotificationProvider;
+import org.apache.atlas.model.notification.HookNotification;
 import org.apache.atlas.v1.model.instance.Referenceable;
 import org.apache.atlas.notification.NotificationException;
 import org.apache.atlas.notification.NotificationInterface;
-import org.apache.atlas.v1.model.notification.HookNotification;
+import 
org.apache.atlas.v1.model.notification.HookNotificationV1.EntityCreateRequest;
 import org.apache.atlas.security.InMemoryJAASConfiguration;
-import org.apache.atlas.type.AtlasType;
 import org.apache.commons.configuration.Configuration;
 import org.apache.commons.lang.StringUtils;
 import org.apache.hadoop.security.UserGroupInformation;
-import org.codehaus.jettison.json.JSONArray;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
 import java.io.IOException;
 import java.util.ArrayList;
-import java.util.Collection;
 import java.util.List;
 
 
@@ -102,9 +100,9 @@ public abstract class AtlasHook {
 protected abstract String getNumberOfRetriesPropertyKey();
 
 protected void notifyEntities(String user, List entities) {
-List 
hookNotificationMessages = new ArrayList<>();
-hookNotificationMessages.add(new 
HookNotification.EntityCreateRequest(user, entities));
-notifyEntities(hookNotificationMessages);
+List hookNotifications = new ArrayList<>();
+hookNotifications.add(new EntityCreateRequest(user, entities));
+notifyEntities(hookNotifications);
 }
 
 /**
@@ -116,12 +114,12 @@ public abstract class AtlasHook {
  * @param messages   hook notification messages
  * @param maxRetries maximum number of retries while sending message to 
messaging system
  */
-public static void 
notifyEntities(List messages, int 
maxRetries) {
+public static void notifyEntities(List messages, int 
maxRetries) {
 notifyEntitiesInternal(messages, maxRetries, notificationInterface, 
logFailedMessages, failedMessagesLogger);
 }
 
 @VisibleForTesting
-static void 
notifyEntitiesInternal(List messages, 
int maxRetries,
+static void notifyEntitiesInternal(List messages, int 
maxRetries,
NotificationInterface 
notificationInterface,
boolean shouldLogFailedMessages, 
FailedMessagesLogger logger) {
 if (messages == null || messages.isEmpty()) {
@@ -168,7 +166,7 @@ public abstract class AtlasHook {
  *
  * @param messages hook notification messages
  */
-protected void 
notifyEntities(List messages) {
+protected void notifyEntities(List messages) {
 final int maxRetries = 
atlasProperties.getInt(getNumberOfRetriesPropertyKey(), 3);
 notifyEntities(messages, maxRetries);
 }

http://git-wip-us.apache.org/repos/asf/atlas/blob/f01e46d7/notification/src/main/java/org/apache/atlas/notification/NotificationInterface.java
--
diff --git 
a/notification/src/main/java/org/apache/atlas/notification/NotificationInterface.java
 
b/notification/src/main/java/org/apache/atlas/notification/NotificationInterface.java
index 975967d..6caf7e2 100644
--- 
a/notification/src/main/java/org/apache/atlas/notification/NotificationInterface.java
+++ 
b/notification/src/main/java/org/apache/atlas/notification/NotificationInterface.java
@@ -17,16 +17,9 @@
  */
 package org.apache.atlas.notification;
 
-import com.google.gson.reflect.TypeToken;
-import org.apache.atlas.model.notification.AtlasNotificationMessage;
 import org.apache.atlas.notification.entity.EntityMessageDeserializer;
 import org.apache.atlas.notification.hook.HookMessageDeserializer;
-import org.apache.atlas.v1.model.notification.EntityNotification;
-import org.apache.atlas.v1.model.notification.HookNotification;
-import org.codehaus.jackson.type.TypeReference;
-import scala.reflect.internal.Types;
 
-import java.lang.reflect.Type;
 import java.util.List;
 
 /**
@@ -46,19 +39,6 @@ public interface NotificationInterface {
 String PROPERTY_PREFIX = "atlas.notification";
 
 /**
- * Notification message class types.
- */
-Class HOOK_NOTIFICATION_CLASS =
-HookNotification.HookNotificationMessage.class;
-
-Class ENTITY_NOTIFICATION_CLASS = 

[2/4] atlas git commit: ATLAS-2251: notification module updates (#4)

2017-11-07 Thread madhan
http://git-wip-us.apache.org/repos/asf/atlas/blob/f01e46d7/webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java
--
diff --git 
a/webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java
 
b/webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java
index 779298a..456a778 100644
--- 
a/webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java
+++ 
b/webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java
@@ -29,13 +29,14 @@ import org.apache.atlas.RequestContextV1;
 import org.apache.atlas.ha.HAConfiguration;
 import org.apache.atlas.kafka.AtlasKafkaMessage;
 import org.apache.atlas.listener.ActiveStateChangeHandler;
-import org.apache.atlas.model.instance.AtlasEntity;
+import org.apache.atlas.model.instance.AtlasEntity.AtlasEntitiesWithExtInfo;
+import org.apache.atlas.model.notification.HookNotification;
+import org.apache.atlas.notification.NotificationInterface.NotificationType;
 import org.apache.atlas.v1.model.instance.Referenceable;
-import 
org.apache.atlas.v1.model.notification.HookNotification.EntityCreateRequest;
-import 
org.apache.atlas.v1.model.notification.HookNotification.EntityDeleteRequest;
-import 
org.apache.atlas.v1.model.notification.HookNotification.EntityPartialUpdateRequest;
-import 
org.apache.atlas.v1.model.notification.HookNotification.EntityUpdateRequest;
-import 
org.apache.atlas.v1.model.notification.HookNotification.HookNotificationMessage;
+import 
org.apache.atlas.v1.model.notification.HookNotificationV1.EntityCreateRequest;
+import 
org.apache.atlas.v1.model.notification.HookNotificationV1.EntityDeleteRequest;
+import 
org.apache.atlas.v1.model.notification.HookNotificationV1.EntityPartialUpdateRequest;
+import 
org.apache.atlas.v1.model.notification.HookNotificationV1.EntityUpdateRequest;
 import org.apache.atlas.repository.converters.AtlasInstanceConverter;
 import org.apache.atlas.repository.store.graph.AtlasEntityStore;
 import org.apache.atlas.repository.store.graph.v1.AtlasEntityStream;
@@ -56,10 +57,7 @@ import org.springframework.core.annotation.Order;
 import org.springframework.stereotype.Component;
 
 import javax.inject.Inject;
-import java.util.ArrayList;
-import java.util.Date;
-import java.util.HashMap;
-import java.util.List;
+import java.util.*;
 import java.util.concurrent.ExecutorService;
 import java.util.concurrent.Executors;
 import java.util.concurrent.TimeUnit;
@@ -76,37 +74,37 @@ import static 
org.apache.atlas.AtlasClientV2.API_V2.UPDATE_ENTITY_BY_ATTRIBUTE;
 @Order(4)
 @DependsOn(value = {"atlasTypeDefStoreInitializer", 
"atlasTypeDefGraphStoreV1"})
 public class NotificationHookConsumer implements Service, 
ActiveStateChangeHandler {
-private static final Logger LOG = 
LoggerFactory.getLogger(NotificationHookConsumer.class);
-private static final Logger PERF_LOG = 
AtlasPerfTracer.getPerfLogger(NotificationHookConsumer.class);
-private static final String LOCALHOST = "localhost";
-private static Logger FAILED_LOG = LoggerFactory.getLogger("FAILED");
+private static final Logger LOG= 
LoggerFactory.getLogger(NotificationHookConsumer.class);
+private static final Logger PERF_LOG   = 
AtlasPerfTracer.getPerfLogger(NotificationHookConsumer.class);
+private static final Logger FAILED_LOG = LoggerFactory.getLogger("FAILED");
 
+private static final String LOCALHOST = "localhost";
 private static final String THREADNAME_PREFIX = 
NotificationHookConsumer.class.getSimpleName();
 
-public static final String CONSUMER_THREADS_PROPERTY = 
"atlas.notification.hook.numthreads";
-public static final String CONSUMER_RETRIES_PROPERTY = 
"atlas.notification.hook.maxretries";
+public static final String CONSUMER_THREADS_PROPERTY = 
"atlas.notification.hook.numthreads";
+public static final String CONSUMER_RETRIES_PROPERTY = 
"atlas.notification.hook.maxretries";
 public static final String CONSUMER_FAILEDCACHESIZE_PROPERTY = 
"atlas.notification.hook.failedcachesize";
-public static final String CONSUMER_RETRY_INTERVAL = 
"atlas.notification.consumer.retry.interval";
-public static final String CONSUMER_MIN_RETRY_INTERVAL = 
"atlas.notification.consumer.min.retry.interval";
-public static final String CONSUMER_MAX_RETRY_INTERVAL = 
"atlas.notification.consumer.max.retry.interval";
-
+public static final String CONSUMER_RETRY_INTERVAL   = 
"atlas.notification.consumer.retry.interval";
+public static final String CONSUMER_MIN_RETRY_INTERVAL   = 
"atlas.notification.consumer.min.retry.interval";
+public static final String CONSUMER_MAX_RETRY_INTERVAL   = 
"atlas.notification.consumer.max.retry.interval";
 
 public static final int SERVER_READY_WAIT_TIME_MS = 1000;
-private final AtlasEntityStore atlasEntityStore;
-private final ServiceState 

[3/3] atlas git commit: Merge branch 'master' into ATLAS-2251

2017-11-07 Thread madhan
Merge branch 'master' into ATLAS-2251


Project: http://git-wip-us.apache.org/repos/asf/atlas/repo
Commit: http://git-wip-us.apache.org/repos/asf/atlas/commit/d511b288
Tree: http://git-wip-us.apache.org/repos/asf/atlas/tree/d511b288
Diff: http://git-wip-us.apache.org/repos/asf/atlas/diff/d511b288

Branch: refs/heads/ATLAS-2251
Commit: d511b28887346db37fa8e0b7db244ad4a9aa3c55
Parents: f01e46d 549310e
Author: Madhan Neethiraj <mad...@apache.org>
Authored: Tue Nov 7 17:23:08 2017 -0800
Committer: Madhan Neethiraj <mad...@apache.org>
Committed: Tue Nov 7 17:23:08 2017 -0800

--
 3party-licenses/backgrid-columnmanager-LICENSE  |  22 +
 LICENSE |   3 +
 dashboardv2/gruntfile.js| 308 +++---
 dashboardv2/package.json|  32 +-
 dashboardv2/public/index.html.tpl   |   2 +-
 .../css/Backgrid.ColumnManager.css  | 114 +++
 .../js/Backgrid.ColumnManager.js| 960 +++
 dashboardv2/public/js/main.js   |   2 +-
 8 files changed, 1276 insertions(+), 167 deletions(-)
--




[1/3] atlas git commit: ATLAS-2250: Upgrade JQuery version to 3.*

2017-11-07 Thread madhan
Repository: atlas
Updated Branches:
  refs/heads/ATLAS-2251 f01e46d73 -> d511b2888


ATLAS-2250: Upgrade JQuery version to 3.*


Project: http://git-wip-us.apache.org/repos/asf/atlas/repo
Commit: http://git-wip-us.apache.org/repos/asf/atlas/commit/af51fbcf
Tree: http://git-wip-us.apache.org/repos/asf/atlas/tree/af51fbcf
Diff: http://git-wip-us.apache.org/repos/asf/atlas/diff/af51fbcf

Branch: refs/heads/ATLAS-2251
Commit: af51fbcf2d8c94659ed454d78f1b5551c56d052a
Parents: 37be53b
Author: kevalbhatt 
Authored: Mon Nov 6 19:00:44 2017 +0530
Committer: kevalbhatt 
Committed: Tue Nov 7 17:11:49 2017 +0530

--
 3party-licenses/backgrid-columnmanager-LICENSE  |  22 +
 LICENSE |   3 +
 dashboardv2/gruntfile.js| 308 +++---
 dashboardv2/package.json|  32 +-
 dashboardv2/public/index.html.tpl   |   2 +-
 .../css/Backgrid.ColumnManager.css  | 114 +++
 .../js/Backgrid.ColumnManager.js| 960 +++
 dashboardv2/public/js/main.js   |   2 +-
 8 files changed, 1276 insertions(+), 167 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/atlas/blob/af51fbcf/3party-licenses/backgrid-columnmanager-LICENSE
--
diff --git a/3party-licenses/backgrid-columnmanager-LICENSE 
b/3party-licenses/backgrid-columnmanager-LICENSE
new file mode 100644
index 000..0750984
--- /dev/null
+++ b/3party-licenses/backgrid-columnmanager-LICENSE
@@ -0,0 +1,22 @@
+The MIT License (MIT)
+
+Copyright (c) 2015 Wilbert van de Ridder
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
+

http://git-wip-us.apache.org/repos/asf/atlas/blob/af51fbcf/LICENSE
--
diff --git a/LICENSE b/LICENSE
index 435e340..38ae8ab 100755
--- a/LICENSE
+++ b/LICENSE
@@ -256,4 +256,7 @@ MIT License.  For details, see 
3party-licenses/bootstrap-daterangepicker-LICENSE
 
 This product bundles jQuery QueryBuilder 2.4.3, which is available under
 MIT License.  For details, see 3party-licenses/jQuery-QueryBuilder-LICENSE
+
+This product bundles backgrid-columnmanager 0.2.4, which is available under
+MIT License.  For details, see 3party-licenses/backgrid-columnmanager-LICENSE
 ===

http://git-wip-us.apache.org/repos/asf/atlas/blob/af51fbcf/dashboardv2/gruntfile.js
--
diff --git a/dashboardv2/gruntfile.js b/dashboardv2/gruntfile.js
index ebe665b..ba9ea0f 100644
--- a/dashboardv2/gruntfile.js
+++ b/dashboardv2/gruntfile.js
@@ -17,16 +17,12 @@
  */
 
 'use strict';
-
-var git = require('git-rev');
 module.exports = function(grunt) {
-var classPathSep = (process.platform === "win32") ? ';' : ':',
-gitHash = '',
-pkg = grunt.file.readJSON('package.json'),
-buildTime = new Date().getTime(),
-distPath = 'dist',
-libPath = distPath + '/js/libs',
+var buildTime = new Date().getTime(),
+distPath = './dist/',
+libPath = distPath + 'js/libs/',
 isDashboardDirectory = grunt.file.isDir('public'),
+nodeModulePath = './node_modules/',
 modulesPath = 'public/';
 if (!isDashboardDirectory) {
 modulesPath = '../public/'
@@ -58,135 +54,124 @@ module.exports = function(grunt) {
 base: distPath,
 // change this to '0.0.0.0' to access the server from 
outside
 hostname: '0.0.0.0',
-middleware: function(connect, options, defaultMiddleware) {
-var proxy = 
require('grunt-connect-proxy/lib/utils').proxyRequest;
-

[2/3] atlas git commit: ATLAS-2240: remove duplicate test testConcurrentCalls()

2017-11-07 Thread madhan
ATLAS-2240: remove duplicate test testConcurrentCalls()


Project: http://git-wip-us.apache.org/repos/asf/atlas/repo
Commit: http://git-wip-us.apache.org/repos/asf/atlas/commit/549310e3
Tree: http://git-wip-us.apache.org/repos/asf/atlas/tree/549310e3
Diff: http://git-wip-us.apache.org/repos/asf/atlas/diff/549310e3

Branch: refs/heads/ATLAS-2251
Commit: 549310e3fbb73fdd969fe771821ef857246cc385
Parents: af51fbc
Author: Sarath Subramanian 
Authored: Tue Nov 7 11:57:35 2017 -0800
Committer: Sarath Subramanian 
Committed: Tue Nov 7 11:57:35 2017 -0800

--
 .../GraphBackedMetadataRepositoryTest.java  | 51 
 1 file changed, 51 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/atlas/blob/549310e3/repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepositoryTest.java
--
diff --git 
a/repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepositoryTest.java
 
b/repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepositoryTest.java
index da6fa75..952a644 100755
--- 
a/repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepositoryTest.java
+++ 
b/repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepositoryTest.java
@@ -130,57 +130,6 @@ public class GraphBackedMetadataRepositoryTest {
 }
 
 @Test
-//In some cases of parallel APIs, the edge is added, but get edge by label 
doesn't return the edge. ATLAS-1104
-public void testConcurrentCalls() throws Exception {
-final HierarchicalTypeDefinition refType =
-createClassTypeDef(randomString(), ImmutableSet.of());
-HierarchicalTypeDefinition type =
-createClassTypeDef(randomString(), ImmutableSet.of(),
-new AttributeDefinition("ref", refType.typeName, 
Multiplicity.OPTIONAL, true, null));
-typeSystem.defineClassType(refType);
-typeSystem.defineClassType(type);
-
-String refId1 = createEntity(new 
Referenceable(refType.typeName)).get(0);
-String refId2 = createEntity(new 
Referenceable(refType.typeName)).get(0);
-
-final Referenceable instance1 = new Referenceable(type.typeName);
-instance1.set("ref", new Referenceable(refId1, refType.typeName, 
null));
-
-final Referenceable instance2 = new Referenceable(type.typeName);
-instance2.set("ref", new Referenceable(refId2, refType.typeName, 
null));
-
-ExecutorService executor = Executors.newFixedThreadPool(3);
-List futures = new ArrayList<>();
-futures.add(executor.submit(new Callable() {
-@Override
-public Object call() throws Exception {
-return createEntity(instance1).get(0);
-}
-}));
-futures.add(executor.submit(new Callable() {
-@Override
-public Object call() throws Exception {
-return createEntity(instance2).get(0);
-}
-}));
-futures.add(executor.submit(new Callable() {
-@Override
-public Object call() throws Exception {
-return discoveryService.searchByDSL(TestUtils.TABLE_TYPE, new 
QueryParams(10, 0));
-}
-}));
-
-String id1 = (String) futures.get(0).get();
-String id2 = (String) futures.get(1).get();
-futures.get(2).get();
-executor.shutdown();
-
-boolean validated1 = assertEdge(id1, type.typeName);
-boolean validated2 = assertEdge(id2, type.typeName);
-assertTrue(validated1 | validated2);
-}
-
-@Test
 public void testSubmitEntity() throws Exception {
 ITypedReferenceableInstance hrDept = 
TestUtils.createDeptEg1(typeSystem);
 



atlas git commit: ATLAS-2256: Make JanusGraph as the default graph-provider in atlas

2017-11-09 Thread madhan
Repository: atlas
Updated Branches:
  refs/heads/ATLAS-2251 2e24da428 -> c2b9846dd


ATLAS-2256: Make JanusGraph as the default graph-provider in atlas


Project: http://git-wip-us.apache.org/repos/asf/atlas/repo
Commit: http://git-wip-us.apache.org/repos/asf/atlas/commit/c2b9846d
Tree: http://git-wip-us.apache.org/repos/asf/atlas/tree/c2b9846d
Diff: http://git-wip-us.apache.org/repos/asf/atlas/diff/c2b9846d

Branch: refs/heads/ATLAS-2251
Commit: c2b9846dd5c59c938a5eefb2e825584381338e13
Parents: 2e24da4
Author: Sarath Subramanian <ssubraman...@hortonworks.com>
Authored: Thu Nov 9 13:51:38 2017 -0800
Committer: Madhan Neethiraj <mad...@apache.org>
Committed: Thu Nov 9 17:21:39 2017 -0800

--
 distro/pom.xml   | 2 +-
 distro/src/bin/atlas_config.py   | 2 +-
 distro/src/conf/atlas-application.properties | 4 ++--
 pom.xml  | 4 ++--
 .../java/org/apache/atlas/util/AtlasRepositoryConfiguration.java | 2 +-
 5 files changed, 7 insertions(+), 7 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/atlas/blob/c2b9846d/distro/pom.xml
--
diff --git a/distro/pom.xml b/distro/pom.xml
index 9db742d..0c4e302 100644
--- a/distro/pom.xml
+++ b/distro/pom.xml
@@ -40,7 +40,7 @@ atlas.graph.storage.hostname=
 atlas.graph.storage.hbase.regions-per-server=1
 atlas.graph.storage.lock.wait-time=1
 
-solr5
+solr
 #Solr
 #Solr cloud mode properties
 atlas.graph.index.search.solr.mode=cloud

http://git-wip-us.apache.org/repos/asf/atlas/blob/c2b9846d/distro/src/bin/atlas_config.py
--
diff --git a/distro/src/bin/atlas_config.py b/distro/src/bin/atlas_config.py
index 4d13b30..e6415cf 100755
--- a/distro/src/bin/atlas_config.py
+++ b/distro/src/bin/atlas_config.py
@@ -62,7 +62,7 @@ ON_POSIX = 'posix' in sys.builtin_module_names
 CONF_FILE="atlas-application.properties"
 HBASE_STORAGE_CONF_ENTRY="atlas.graph.storage.backend\s*=\s*hbase"
 HBASE_STORAGE_LOCAL_CONF_ENTRY="atlas.graph.storage.hostname\s*=\s*localhost"
-SOLR_INDEX_CONF_ENTRY="atlas.graph.index.search.backend\s*=\s*solr5"
+SOLR_INDEX_CONF_ENTRY="atlas.graph.index.search.backend\s*=\s*solr"
 
SOLR_INDEX_LOCAL_CONF_ENTRY="atlas.graph.index.search.solr.zookeeper-url\s*=\s*localhost"
 SOLR_INDEX_ZK_URL="atlas.graph.index.search.solr.zookeeper-url"
 TOPICS_TO_CREATE="atlas.notification.topics"

http://git-wip-us.apache.org/repos/asf/atlas/blob/c2b9846d/distro/src/conf/atlas-application.properties
--
diff --git a/distro/src/conf/atlas-application.properties 
b/distro/src/conf/atlas-application.properties
index b61c2bc..1f38705 100755
--- a/distro/src/conf/atlas-application.properties
+++ b/distro/src/conf/atlas-application.properties
@@ -20,8 +20,8 @@
 
 # Graph Database
 
-#Configures the graph database to use.  Defaults to Titan 0.5.4.
-#atlas.graphdb.backend=org.apache.atlas.repository.graphdb.titan0.Titan0GraphDatabase
+#Configures the graph database to use.  Defaults to JanusGraph 0.1.1
+#atlas.graphdb.backend=org.apache.atlas.repository.graphdb.janus.AtlasJanusGraphDatabase
 
 # Graph Storage
 atlas.graph.storage.backend=${graph.storage.backend}

http://git-wip-us.apache.org/repos/asf/atlas/blob/c2b9846d/pom.xml
--
diff --git a/pom.xml b/pom.xml
index fdf42a7..8277414 100644
--- a/pom.xml
+++ b/pom.xml
@@ -649,9 +649,9 @@
 
 
 org.apache.atlas
-atlas-graphdb-titan0
+atlas-graphdb-janus
 false
-
org.apache.atlas.repository.graphdb.titan0.Titan0GraphDatabase
+
org.apache.atlas.repository.graphdb.janus.AtlasJanusGraphDatabase
 
 
 

http://git-wip-us.apache.org/repos/asf/atlas/blob/c2b9846d/repository/src/main/java/org/apache/atlas/util/AtlasRepositoryConfiguration.java
--
diff --git 
a/repository/src/main/java/org/apache/atlas/util/AtlasRepositoryConfiguration.java
 
b/repository/src/main/java/org/apache/atlas/util/AtlasRepositoryConfiguration.java
index f0c7500..0e1e5b6 100644
--- 
a/repository/src/main/java/org/apache/atlas/util/AtlasRepositoryConfiguration.java
+++ 
b/repository/src/main/java/org/apache/atlas/util/AtlasRepositoryConfiguration.java
@@ -122,7 +122,7 @@ public class AtlasRepositoryConfiguration {
 }
 
 private static final String GRAP

[3/5] atlas git commit: ATLAS-2251: restored bunch of deleted tests

2017-11-09 Thread madhan
http://git-wip-us.apache.org/repos/asf/atlas/blob/042fc557/repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStoreV1Test.java
--
diff --git 
a/repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStoreV1Test.java
 
b/repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStoreV1Test.java
new file mode 100644
index 000..fd1b6db
--- /dev/null
+++ 
b/repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStoreV1Test.java
@@ -0,0 +1,1043 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.atlas.repository.store.graph.v1;
+
+import com.google.common.collect.ImmutableSet;
+import org.apache.atlas.AtlasErrorCode;
+import org.apache.atlas.AtlasException;
+import org.apache.atlas.TestModules;
+import org.apache.atlas.RequestContextV1;
+import org.apache.atlas.TestUtilsV2;
+import org.apache.atlas.exception.AtlasBaseException;
+import org.apache.atlas.model.instance.AtlasClassification;
+import org.apache.atlas.model.instance.AtlasEntity;
+import org.apache.atlas.model.instance.AtlasEntity.AtlasEntitiesWithExtInfo;
+import org.apache.atlas.model.instance.AtlasEntity.AtlasEntityExtInfo;
+import org.apache.atlas.model.instance.AtlasEntity.AtlasEntityWithExtInfo;
+import org.apache.atlas.model.instance.AtlasEntityHeader;
+import org.apache.atlas.model.instance.AtlasObjectId;
+import org.apache.atlas.model.instance.AtlasStruct;
+import org.apache.atlas.model.instance.EntityMutationResponse;
+import org.apache.atlas.model.instance.EntityMutations;
+import org.apache.atlas.model.instance.EntityMutations.EntityOperation;
+import org.apache.atlas.model.typedef.AtlasClassificationDef;
+import org.apache.atlas.model.typedef.AtlasEntityDef;
+import org.apache.atlas.model.typedef.AtlasStructDef;
+import org.apache.atlas.model.typedef.AtlasStructDef.AtlasAttributeDef;
+import org.apache.atlas.model.typedef.AtlasTypesDef;
+import org.apache.atlas.repository.graph.AtlasGraphProvider;
+import org.apache.atlas.repository.graph.GraphBackedSearchIndexer;
+import 
org.apache.atlas.repository.store.bootstrap.AtlasTypeDefStoreInitializer;
+import org.apache.atlas.repository.store.graph.AtlasEntityStore;
+import org.apache.atlas.store.AtlasTypeDefStore;
+import org.apache.atlas.type.AtlasArrayType;
+import org.apache.atlas.type.AtlasMapType;
+import org.apache.atlas.type.AtlasStructType;
+import org.apache.atlas.type.AtlasType;
+import org.apache.atlas.type.AtlasTypeRegistry;
+import org.apache.atlas.type.AtlasTypeUtil;
+import org.apache.commons.collections.CollectionUtils;
+import org.apache.commons.collections.MapUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.testng.Assert;
+import org.testng.annotations.AfterClass;
+import org.testng.annotations.BeforeClass;
+import org.testng.annotations.BeforeTest;
+import org.testng.annotations.Guice;
+import org.testng.annotations.Test;
+
+import javax.inject.Inject;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import static org.apache.atlas.TestUtilsV2.COLUMNS_ATTR_NAME;
+import static org.apache.atlas.TestUtilsV2.COLUMN_TYPE;
+import static org.apache.atlas.TestUtilsV2.NAME;
+import static org.apache.atlas.TestUtilsV2.randomString;
+import static org.apache.atlas.TestUtilsV2.STORAGE_DESC_TYPE;
+import static org.apache.atlas.TestUtilsV2.TABLE_TYPE;
+import static org.mockito.Mockito.mock;
+import static org.testng.Assert.assertEquals;
+import static org.testng.Assert.assertTrue;
+import static org.testng.Assert.fail;
+
+@Guice(modules = TestModules.TestOnlyModule.class)
+public class AtlasEntityStoreV1Test {
+private static final Logger LOG = 
LoggerFactory.getLogger(AtlasEntityStoreV1Test.class);
+
+@Inject
+AtlasTypeRegistry typeRegistry;
+
+@Inject
+AtlasTypeDefStore typeDefStore;
+
+AtlasEntityStore entityStore;
+
+@Inject
+DeleteHandlerV1 deleteHandler;
+
+private AtlasEntitiesWithExtInfo deptEntity;
+private AtlasEntityWithExtInfo   dbEntity;
+

[5/5] atlas git commit: ATLAS-2251: restored bunch of deleted tests

2017-11-09 Thread madhan
ATLAS-2251: restored bunch of deleted tests


Project: http://git-wip-us.apache.org/repos/asf/atlas/repo
Commit: http://git-wip-us.apache.org/repos/asf/atlas/commit/042fc557
Tree: http://git-wip-us.apache.org/repos/asf/atlas/tree/042fc557
Diff: http://git-wip-us.apache.org/repos/asf/atlas/diff/042fc557

Branch: refs/heads/ATLAS-2251
Commit: 042fc55755e17b3b0112a7a08d30c296300b95d9
Parents: 5f40541
Author: Madhan Neethiraj <mad...@apache.org>
Authored: Thu Nov 9 21:44:26 2017 -0800
Committer: Madhan Neethiraj <mad...@apache.org>
Committed: Thu Nov 9 21:44:26 2017 -0800

--
 .../org/apache/atlas/query/QueryParser.java |   11 +-
 .../impexp/AtlasImportRequestTest.java  |  145 +++
 .../repository/impexp/ExportServiceTest.java|  358 ++
 .../repository/impexp/ImportServiceTest.java|  224 
 .../impexp/ImportServiceTestUtils.java  |  116 ++
 .../impexp/ImportTransformerJSONTest.java   |   47 +
 .../impexp/ImportTransformerTest.java   |  131 +++
 .../repository/impexp/ImportTransformsTest.java |  151 +++
 .../impexp/TypeAttributeDifferenceTest.java |  150 +++
 .../atlas/repository/impexp/UniqueListTest.java |   69 ++
 .../impexp/ZipFileResourceTestUtils.java|  220 
 .../atlas/repository/impexp/ZipSinkTest.java|  210 
 .../atlas/repository/impexp/ZipSourceTest.java  |  189 
 .../store/graph/AtlasTypeDefGraphStoreTest.java |  644 +++
 .../graph/v1/AtlasEntityDefStoreV1Test.java |   73 ++
 ...AtlasEntityStoreV1BulkImportPercentTest.java |  167 +++
 .../store/graph/v1/AtlasEntityStoreV1Test.java  | 1043 ++
 .../v1/AtlasRelationshipDefStoreV1Test.java |  330 ++
 .../AtlasRelationshipStoreHardDeleteV1Test.java |  112 ++
 .../AtlasRelationshipStoreSoftDeleteV1Test.java |  116 ++
 .../graph/v1/AtlasRelationshipStoreV1Test.java  |  623 +++
 .../InverseReferenceUpdateHardDeleteV1Test.java |   75 ++
 .../InverseReferenceUpdateSoftDeleteV1Test.java |   78 ++
 .../graph/v1/InverseReferenceUpdateV1Test.java  |  374 +++
 .../userprofile/UserProfileServiceTest.java |  280 +
 .../atlas/services/MetricsServiceTest.java  |  123 +++
 .../utils/ObjectUpdateSynchronizerTest.java |  218 
 .../org/apache/atlas/util/RestUtilsTest.java|  195 
 28 files changed, 6471 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/atlas/blob/042fc557/repository/src/main/java/org/apache/atlas/query/QueryParser.java
--
diff --git a/repository/src/main/java/org/apache/atlas/query/QueryParser.java 
b/repository/src/main/java/org/apache/atlas/query/QueryParser.java
index be2d9b1..1e5e5ff 100644
--- a/repository/src/main/java/org/apache/atlas/query/QueryParser.java
+++ b/repository/src/main/java/org/apache/atlas/query/QueryParser.java
@@ -19,11 +19,20 @@ package org.apache.atlas.query;
 
 import org.apache.atlas.query.Expressions.Expression;
 
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.Set;
 
 
 public class QueryParser {
+private static final Set RESERVED_KEYWORDS =
+new HashSet<>(Arrays.asList("[", "]", "(", ")", "=", "<", ">", 
"!=", "<=", ">=", ",", "and", "or", "+", "-",
+"*", "/", ".", "select", "from", 
"where", "groupby", "loop", "isa", "is", "has",
+"as", "times", "withPath", "limit", 
"offset", "orderby", "count", "max", "min",
+"sum", "by", "order", "like"));
+
 public static boolean isKeyword(String word) {
-return false; // TODO:
+return RESERVED_KEYWORDS.contains(word);
 }
 
 public static Expression apply(String queryStr, QueryParams params) {

http://git-wip-us.apache.org/repos/asf/atlas/blob/042fc557/repository/src/test/java/org/apache/atlas/repository/impexp/AtlasImportRequestTest.java
--
diff --git 
a/repository/src/test/java/org/apache/atlas/repository/impexp/AtlasImportRequestTest.java
 
b/repository/src/test/java/org/apache/atlas/repository/impexp/AtlasImportRequestTest.java
new file mode 100644
index 000..b70b181
--- /dev/null
+++ 
b/repository/src/test/java/org/apache/atlas/repository/impexp/AtlasImportRequestTest.java
@@ -0,0 +1,145 @@
+/**
+ * Licensed to the Apache Software Foundation (

[1/5] atlas git commit: ATLAS-2251: restored bunch of deleted tests

2017-11-09 Thread madhan
Repository: atlas
Updated Branches:
  refs/heads/ATLAS-2251 5f40541b5 -> 042fc5575


http://git-wip-us.apache.org/repos/asf/atlas/blob/042fc557/webapp/src/test/java/org/apache/atlas/util/RestUtilsTest.java
--
diff --git a/webapp/src/test/java/org/apache/atlas/util/RestUtilsTest.java 
b/webapp/src/test/java/org/apache/atlas/util/RestUtilsTest.java
new file mode 100644
index 000..0d4af1e
--- /dev/null
+++ b/webapp/src/test/java/org/apache/atlas/util/RestUtilsTest.java
@@ -0,0 +1,195 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.atlas.util;
+
+import static org.mockito.Matchers.any;
+import static org.mockito.Matchers.eq;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+import java.util.*;
+
+import org.apache.atlas.exception.AtlasBaseException;
+import org.apache.atlas.model.typedef.AtlasBaseTypeDef;
+import org.apache.atlas.model.typedef.AtlasEntityDef;
+import org.apache.atlas.model.typedef.AtlasStructDef;
+import org.apache.atlas.model.typedef.AtlasStructDef.AtlasAttributeDef;
+import org.apache.atlas.model.typedef.AtlasTypesDef;
+import org.apache.atlas.repository.Constants;
+import org.apache.atlas.repository.converters.TypeConverterUtil;
+import org.apache.atlas.repository.graphdb.AtlasVertex;
+import org.apache.atlas.repository.store.graph.v1.AtlasGraphUtilsV1;
+import org.apache.atlas.repository.store.graph.v1.AtlasStructDefStoreV1;
+import org.apache.atlas.repository.store.graph.v1.AtlasTypeDefGraphStoreV1;
+import org.apache.atlas.type.AtlasEntityType;
+import org.apache.atlas.type.AtlasStructType;
+import org.apache.atlas.type.AtlasStructType.AtlasAttribute;
+import org.apache.atlas.type.AtlasType;
+import org.apache.atlas.type.AtlasTypeRegistry;
+import org.apache.atlas.type.AtlasTypeRegistry.AtlasTransientTypeRegistry;
+import org.apache.atlas.typesystem.types.DataTypes.TypeCategory;
+import org.apache.atlas.v1.model.typedef.*;
+import org.apache.atlas.v1.typesystem.types.utils.TypesUtil;
+import org.testng.Assert;
+import org.testng.annotations.Test;
+
+/**
+ * Validates that conversion from V1 to legacy types (and back) is consistent. 
 This also tests
+ * that the conversion logic in AtlasStructDefStoreV1 is consistent with the 
conversion logic
+ * in RestUtils.  This tests particularly focuses on composite attributes, 
since a defect was
+ * found in that area.
+ */
+public class RestUtilsTest {
+
+@Test(enabled=false)
+// FIXME: On conversion back to V1, reverse attribute name
+// "containingDatabase"
+// in tables attribute in "database" type is lost.  See ATLAS-1528.
+public void testBidirectonalCompositeMappingConsistent() throws 
AtlasBaseException {
+
+ClassTypeDefinition dbV1Type = 
TypesUtil.createClassTypeDef("database", "", Collections.emptySet(),
+   new AttributeDefinition("tables", 
AtlasBaseTypeDef.getArrayTypeName("table"),
+   
Multiplicity.OPTIONAL, true, "containingDatabase"));
+
+ClassTypeDefinition tableV1Type = 
TypesUtil.createClassTypeDef("table", "", Collections.emptySet(),
+new 
AttributeDefinition("containingDatabase", "database",
+
Multiplicity.OPTIONAL, false, "tables"));
+
+testV1toV2toV1Conversion(Arrays.asList(dbV1Type, tableV1Type), new 
boolean[] { true, false });
+}
+
+@Test(enabled=false)
+// FIXME: On conversion back to V1, reverse attribute name
+// "containingDatabase" is lost
+// in "table" attribute in "database".  See ATLAS-1528.
+public void testBidirectonalNonCompositeMappingConsistent() throws 
AtlasBaseException {
+ClassTypeDefinition dbV1Type = 
TypesUtil.createClassTypeDef("database", "", Collections.emptySet(),
+new AttributeDefinition("tables", 
AtlasBaseTypeDef.getArrayTypeName("table"),
+
Multiplicity.OPTIONAL, false, "containingDatabase"));
+
+

[4/5] atlas git commit: ATLAS-2251: restored bunch of deleted tests

2017-11-09 Thread madhan
http://git-wip-us.apache.org/repos/asf/atlas/blob/042fc557/repository/src/test/java/org/apache/atlas/repository/impexp/ZipSourceTest.java
--
diff --git 
a/repository/src/test/java/org/apache/atlas/repository/impexp/ZipSourceTest.java
 
b/repository/src/test/java/org/apache/atlas/repository/impexp/ZipSourceTest.java
new file mode 100644
index 000..1c1c68f
--- /dev/null
+++ 
b/repository/src/test/java/org/apache/atlas/repository/impexp/ZipSourceTest.java
@@ -0,0 +1,189 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.atlas.repository.impexp;
+
+import org.apache.atlas.exception.AtlasBaseException;
+import org.apache.atlas.model.instance.AtlasEntity;
+import org.apache.atlas.model.typedef.AtlasTypesDef;
+import org.testng.Assert;
+import org.testng.ITestContext;
+import org.testng.annotations.DataProvider;
+import org.testng.annotations.Test;
+
+import java.io.ByteArrayInputStream;
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.util.List;
+
+import static 
org.apache.atlas.repository.impexp.ZipFileResourceTestUtils.getZipSource;
+import static org.testng.Assert.assertEquals;
+import static org.testng.Assert.assertFalse;
+import static org.testng.Assert.assertNotNull;
+import static org.testng.AssertJUnit.assertTrue;
+
+public class ZipSourceTest {
+@DataProvider(name = "zipFileStocks")
+public static Object[][] getDataFromZipFile() throws IOException {
+FileInputStream fs = 
ZipFileResourceTestUtils.getFileInputStream("stocks.zip");
+
+return new Object[][] {{ new ZipSource(fs) }};
+}
+
+@DataProvider(name = "zipFileStocksFloat")
+public static Object[][] getDataFromZipFileWithLongFloats() throws 
IOException {
+FileInputStream fs = 
ZipFileResourceTestUtils.getFileInputStream("stocks-float.zip");
+
+return new Object[][] {{ new ZipSource(fs) }};
+}
+
+@DataProvider(name = "sales")
+public static Object[][] getDataFromQuickStart_v1_Sales(ITestContext 
context) throws IOException {
+return getZipSource("sales-v1-full.zip");
+}
+
+@Test
+public void improperInit_ReturnsNullCreationOrder() throws IOException, 
AtlasBaseException {
+byte bytes[] = new byte[10];
+ByteArrayInputStream bais = new ByteArrayInputStream(bytes);
+ZipSource zs = new ZipSource(bais);
+List s = zs.getCreationOrder();
+Assert.assertNull(s);
+}
+
+@Test(dataProvider = "zipFileStocks")
+public void examineContents_BehavesAsExpected(ZipSource zipSource) throws 
IOException, AtlasBaseException {
+List creationOrder = zipSource.getCreationOrder();
+
+assertNotNull(creationOrder);
+assertEquals(creationOrder.size(), 4);
+
+AtlasTypesDef typesDef = zipSource.getTypesDef();
+assertNotNull(typesDef);
+assertEquals(typesDef.getEntityDefs().size(), 6);
+
+useCreationOrderToFetchEntitiesWithExtInfo(zipSource, creationOrder);
+useCreationOrderToFetchEntities(zipSource, creationOrder);
+attemptToFetchNonExistentGuid_ReturnsNull(zipSource, 
"non-existent-guid");
+verifyGuidRemovalOnImportComplete(zipSource, creationOrder.get(0));
+}
+
+private void useCreationOrderToFetchEntities(ZipSource zipSource, 
List creationOrder) {
+for (String guid : creationOrder) {
+AtlasEntity e = zipSource.getByGuid(guid);
+assertNotNull(e);
+}
+}
+
+private void verifyGuidRemovalOnImportComplete(ZipSource zipSource, String 
guid) {
+AtlasEntity e = zipSource.getByGuid(guid);
+assertNotNull(e);
+
+zipSource.onImportComplete(guid);
+
+e = zipSource.getByGuid(guid);
+Assert.assertNull(e);
+}
+
+private void attemptToFetchNonExistentGuid_ReturnsNull(ZipSource 
zipSource, String guid) {
+AtlasEntity e = zipSource.getByGuid(guid);
+Assert.assertNull(e);
+}
+
+private void useCreationOrderToFetchEntitiesWithExtInfo(ZipSource 
zipSource, List creationOrder) throws AtlasBaseException {
+for (String guid : creationOrder) {
+

[2/5] atlas git commit: ATLAS-2251: restored bunch of deleted tests

2017-11-09 Thread madhan
http://git-wip-us.apache.org/repos/asf/atlas/blob/042fc557/repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasRelationshipStoreV1Test.java
--
diff --git 
a/repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasRelationshipStoreV1Test.java
 
b/repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasRelationshipStoreV1Test.java
new file mode 100644
index 000..d207a69
--- /dev/null
+++ 
b/repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasRelationshipStoreV1Test.java
@@ -0,0 +1,623 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.atlas.repository.store.graph.v1;
+
+import com.google.common.collect.ImmutableList;
+import org.apache.atlas.RequestContextV1;
+import org.apache.atlas.TestModules;
+import org.apache.atlas.TestUtilsV2;
+import org.apache.atlas.exception.AtlasBaseException;
+import org.apache.atlas.model.instance.AtlasEntity;
+import org.apache.atlas.model.instance.AtlasEntity.AtlasEntitiesWithExtInfo;
+import org.apache.atlas.model.instance.AtlasEntity.AtlasEntityWithExtInfo;
+import org.apache.atlas.model.instance.AtlasEntityHeader;
+import org.apache.atlas.model.instance.AtlasObjectId;
+import org.apache.atlas.model.instance.AtlasRelatedObjectId;
+import org.apache.atlas.model.instance.EntityMutationResponse;
+import org.apache.atlas.model.typedef.AtlasTypesDef;
+import org.apache.atlas.repository.graph.AtlasGraphProvider;
+import org.apache.atlas.repository.graph.GraphBackedSearchIndexer;
+import 
org.apache.atlas.repository.store.bootstrap.AtlasTypeDefStoreInitializer;
+import org.apache.atlas.repository.store.graph.AtlasEntityStore;
+import org.apache.atlas.repository.store.graph.AtlasRelationshipStore;
+import org.apache.atlas.store.AtlasTypeDefStore;
+import org.apache.atlas.type.AtlasEntityType;
+import org.apache.atlas.type.AtlasTypeRegistry;
+import org.apache.commons.collections.CollectionUtils;
+import org.testng.annotations.AfterClass;
+import org.testng.annotations.BeforeClass;
+import org.testng.annotations.BeforeTest;
+import org.testng.annotations.Guice;
+import org.testng.annotations.Test;
+
+import javax.inject.Inject;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import static org.apache.atlas.TestRelationshipUtilsV2.EMPLOYEE_TYPE;
+import static 
org.apache.atlas.TestRelationshipUtilsV2.getDepartmentEmployeeInstances;
+import static 
org.apache.atlas.TestRelationshipUtilsV2.getDepartmentEmployeeTypes;
+import static 
org.apache.atlas.TestRelationshipUtilsV2.getInverseReferenceTestTypes;
+import static org.apache.atlas.TestUtilsV2.NAME;
+import static org.apache.atlas.type.AtlasTypeUtil.getAtlasObjectId;
+import static org.mockito.Mockito.mock;
+import static org.testng.Assert.assertEquals;
+import static org.testng.Assert.assertNotNull;
+import static org.testng.Assert.assertNull;
+import static org.testng.Assert.assertTrue;
+
+@Guice(modules = TestModules.TestOnlyModule.class)
+public abstract class AtlasRelationshipStoreV1Test {
+
+@Inject
+AtlasTypeRegistry typeRegistry;
+
+@Inject
+AtlasTypeDefStore typeDefStore;
+
+@Inject
+DeleteHandlerV1   deleteHandler;
+
+@Inject
+EntityGraphMapper graphMapper;
+
+AtlasEntityStore  entityStore;
+AtlasRelationshipStorerelationshipStore;
+AtlasEntityChangeNotifier mockChangeNotifier = 
mock(AtlasEntityChangeNotifier.class);
+
+protected Map employeeNameIdMap = new HashMap<>();
+
+@BeforeClass
+public void setUp() throws Exception {
+new GraphBackedSearchIndexer(typeRegistry);
+
+// create employee relationship types
+AtlasTypesDef employeeTypes = getDepartmentEmployeeTypes();
+typeDefStore.createTypesDef(employeeTypes);
+
+AtlasEntitiesWithExtInfo employeeInstances = 
getDepartmentEmployeeInstances();
+EntityMutationResponse response = entityStore.createOrUpdate(new 
AtlasEntityStream(employeeInstances), false);
+
+for (AtlasEntityHeader entityHeader : 

atlas git commit: ATLAS-2251: restoring deleted InMemory and Noop EntityAuditRepository

2017-11-09 Thread madhan
Repository: atlas
Updated Branches:
  refs/heads/ATLAS-2251 c2b9846dd -> 5f40541b5


ATLAS-2251: restoring deleted InMemory and Noop EntityAuditRepository


Project: http://git-wip-us.apache.org/repos/asf/atlas/repo
Commit: http://git-wip-us.apache.org/repos/asf/atlas/commit/5f40541b
Tree: http://git-wip-us.apache.org/repos/asf/atlas/tree/5f40541b
Diff: http://git-wip-us.apache.org/repos/asf/atlas/diff/5f40541b

Branch: refs/heads/ATLAS-2251
Commit: 5f40541b5345c94804b19d4a085e30e73c8ee83e
Parents: c2b9846
Author: Madhan Neethiraj <mad...@apache.org>
Authored: Thu Nov 9 19:41:51 2017 -0800
Committer: Madhan Neethiraj <mad...@apache.org>
Committed: Thu Nov 9 19:41:51 2017 -0800

--
 .../test/java/org/apache/atlas/TestUtilsV2.java |   4 +
 .../audit/InMemoryEntityAuditRepository.java|  84 +++
 .../audit/NoopEntityAuditRepository.java|  63 +++
 .../audit/AuditRepositoryTestBase.java  | 107 +++
 .../audit/InMemoryAuditRepositoryTest.java  |  28 +
 5 files changed, 286 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/atlas/blob/5f40541b/intg/src/test/java/org/apache/atlas/TestUtilsV2.java
--
diff --git a/intg/src/test/java/org/apache/atlas/TestUtilsV2.java 
b/intg/src/test/java/org/apache/atlas/TestUtilsV2.java
index 5564955..bbccf77 100755
--- a/intg/src/test/java/org/apache/atlas/TestUtilsV2.java
+++ b/intg/src/test/java/org/apache/atlas/TestUtilsV2.java
@@ -836,6 +836,10 @@ public final class TestUtilsV2 {
 return RandomStringUtils.randomAlphanumeric(10);
 }
 
+public static final String randomString(int count) {
+return RandomStringUtils.randomAlphanumeric(count);
+}
+
 public static AtlasEntity createDBEntity() {
 String dbName = RandomStringUtils.randomAlphanumeric(10);
 return createDBEntity(dbName);

http://git-wip-us.apache.org/repos/asf/atlas/blob/5f40541b/repository/src/main/java/org/apache/atlas/repository/audit/InMemoryEntityAuditRepository.java
--
diff --git 
a/repository/src/main/java/org/apache/atlas/repository/audit/InMemoryEntityAuditRepository.java
 
b/repository/src/main/java/org/apache/atlas/repository/audit/InMemoryEntityAuditRepository.java
new file mode 100644
index 000..22d2a81
--- /dev/null
+++ 
b/repository/src/main/java/org/apache/atlas/repository/audit/InMemoryEntityAuditRepository.java
@@ -0,0 +1,84 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.atlas.repository.audit;
+
+import org.apache.atlas.AtlasException;
+import org.apache.atlas.EntityAuditEvent;
+import org.apache.atlas.annotation.ConditionalOnAtlasProperty;
+import org.springframework.stereotype.Component;
+
+import javax.inject.Singleton;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.SortedMap;
+import java.util.TreeMap;
+
+/**
+ * Entity audit repository where audit events are stored in-memory. Used only 
for integration tests
+ */
+@Singleton
+@Component
+@ConditionalOnAtlasProperty(property = "atlas.EntityAuditRepository.impl")
+public class InMemoryEntityAuditRepository implements EntityAuditRepository {
+private TreeMap<String, EntityAuditEvent> auditEvents = new TreeMap<>();
+
+@Override
+public void putEvents(EntityAuditEvent... events) throws AtlasException {
+putEvents(Arrays.asList(events));
+}
+
+@Override
+public synchronized void putEvents(List events) throws 
AtlasException {
+for (EntityAuditEvent event : events) {
+String rowKey = event.getEntityId() + (Long.MAX_VALUE - 
event.getTimestamp());
+event.setEventKey(rowKey);
+auditEvents.put(rowKey, event);
+}
+}
+
+//synchronized to avoid concurrent modification exception that occurs if 
events are added
+//while we are iterating through the map
+@Override
+public synchronized

atlas git commit: ATLAS-2253: Remove shading of janus graph-provider module

2017-11-08 Thread madhan
Repository: atlas
Updated Branches:
  refs/heads/ATLAS-2251 f4e37f0a7 -> 663e4f58a


ATLAS-2253: Remove shading of janus graph-provider module


Project: http://git-wip-us.apache.org/repos/asf/atlas/repo
Commit: http://git-wip-us.apache.org/repos/asf/atlas/commit/663e4f58
Tree: http://git-wip-us.apache.org/repos/asf/atlas/tree/663e4f58
Diff: http://git-wip-us.apache.org/repos/asf/atlas/diff/663e4f58

Branch: refs/heads/ATLAS-2251
Commit: 663e4f58a91eeb3db2d6ee102834a0f49b9f3fea
Parents: f4e37f0
Author: Sarath Subramanian <ssubraman...@hortonworks.com>
Authored: Wed Nov 8 17:28:20 2017 -0800
Committer: Madhan Neethiraj <mad...@apache.org>
Committed: Wed Nov 8 21:31:39 2017 -0800

--
 graphdb/janus/pom.xml  | 44 ++--
 graphdb/titan0/pom.xml |  6 ++
 graphdb/titan1/pom.xml |  6 ++
 pom.xml|  6 --
 4 files changed, 14 insertions(+), 48 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/atlas/blob/663e4f58/graphdb/janus/pom.xml
--
diff --git a/graphdb/janus/pom.xml b/graphdb/janus/pom.xml
index e8a0a61..ad96db3 100644
--- a/graphdb/janus/pom.xml
+++ b/graphdb/janus/pom.xml
@@ -121,10 +121,11 @@
 ${janus.version}
 
 
+
 
 org.apache.solr
 solr-solrj
-6.1.0
+5.2.1
 
 
 
@@ -193,47 +194,6 @@
 
 
 
-
-
-org.apache.maven.plugins
-maven-shade-plugin
-2.4.3
-
-
-package
-
-shade
-
-
-
false
-
-
-
-org.slf4j:*
-
-
-
-
-
-
-
-*:*
-
-META-INF/*.SF
-META-INF/*.DSA
-META-INF/*.RSA
-
-
-
-
-
-
-
-
 
 
 

http://git-wip-us.apache.org/repos/asf/atlas/blob/663e4f58/graphdb/titan0/pom.xml
--
diff --git a/graphdb/titan0/pom.xml b/graphdb/titan0/pom.xml
index d1674a9..2eec260 100644
--- a/graphdb/titan0/pom.xml
+++ b/graphdb/titan0/pom.xml
@@ -187,6 +187,12 @@
 test
 
 
+
+org.apache.solr
+solr-solrj
+${solr.version}
+
+
 
 
 

http://git-wip-us.apache.org/repos/asf/atlas/blob/663e4f58/graphdb/titan1/pom.xml
--
diff --git a/graphdb/titan1/pom.xml b/graphdb/titan1/pom.xml
index 74596bd..c729caa 100644
--- a/graphdb/titan1/pom.xml
+++ b/graphdb/titan1/pom.xml
@@ -144,6 +144,12 @@
 
 
 
+org.apache.solr
+solr-solrj
+${solr.version}
+
+
+
 org.testng
 testng
 

http://git-wip-us.apache.org/repos/asf/atlas/blob/663e4f58/pom.xml
--
diff --git a/pom.xml b/pom.xml
index 3c14501..35b0fec 100644
--- a/pom.xml
+++ b/pom.xml
@@ -1249,12 +1249,6 @@
 
 
 
-org.apache.solr
-solr-solrj
-${solr.version}
-
-
-
 javax.servlet
 javax.servlet-api
 ${javax.servlet.version}



[3/4] atlas git commit: ATLAS-2251: removed typesystem module and scala, json4s, gson libraries

2017-11-08 Thread madhan
http://git-wip-us.apache.org/repos/asf/atlas/blob/0a00dcb1/intg/src/test/java/org/apache/atlas/TestUtilsV2.java
--
diff --git a/intg/src/test/java/org/apache/atlas/TestUtilsV2.java 
b/intg/src/test/java/org/apache/atlas/TestUtilsV2.java
index 91fdb47..5564955 100755
--- a/intg/src/test/java/org/apache/atlas/TestUtilsV2.java
+++ b/intg/src/test/java/org/apache/atlas/TestUtilsV2.java
@@ -18,8 +18,6 @@
 
 package org.apache.atlas;
 
-import com.google.common.collect.ImmutableList;
-import com.google.common.collect.ImmutableSet;
 import org.apache.atlas.model.instance.AtlasEntity;
 import org.apache.atlas.model.instance.AtlasEntity.AtlasEntitiesWithExtInfo;
 import org.apache.atlas.model.instance.AtlasEntity.AtlasEntityWithExtInfo;
@@ -30,7 +28,6 @@ import org.apache.atlas.model.typedef.AtlasClassificationDef;
 import org.apache.atlas.model.typedef.AtlasEntityDef;
 import org.apache.atlas.model.typedef.AtlasEnumDef;
 import org.apache.atlas.model.typedef.AtlasEnumDef.AtlasEnumElementDef;
-import org.apache.atlas.model.typedef.AtlasRelationshipDef;
 import org.apache.atlas.model.typedef.AtlasStructDef;
 import org.apache.atlas.model.typedef.AtlasStructDef.AtlasAttributeDef;
 import 
org.apache.atlas.model.typedef.AtlasStructDef.AtlasAttributeDef.Cardinality;
@@ -92,7 +89,7 @@ public final class TestUtilsV2 {
 AtlasTypeUtil.createRequiredAttrDef("city", "string"));
 
 AtlasEntityDef deptTypeDef =
-AtlasTypeUtil.createClassTypeDef(DEPARTMENT_TYPE, 
"Department"+_description, ImmutableSet.of(),
+AtlasTypeUtil.createClassTypeDef(DEPARTMENT_TYPE, 
"Department"+_description, Collections.emptySet(),
 AtlasTypeUtil.createUniqueRequiredAttrDef("name", 
"string"),
 new AtlasAttributeDef("employees", 
String.format("array<%s>", "Employee"), true,
 AtlasAttributeDef.Cardinality.SINGLE, 0, 1, 
false, false,
@@ -100,7 +97,7 @@ public final class TestUtilsV2 {
 add(new 
AtlasStructDef.AtlasConstraintDef(AtlasConstraintDef.CONSTRAINT_TYPE_OWNED_REF));
 }}));
 
-AtlasEntityDef personTypeDef = 
AtlasTypeUtil.createClassTypeDef("Person", "Person"+_description, 
ImmutableSet.of(),
+AtlasEntityDef personTypeDef = 
AtlasTypeUtil.createClassTypeDef("Person", "Person"+_description, 
Collections.emptySet(),
 AtlasTypeUtil.createUniqueRequiredAttrDef("name", "string"),
 AtlasTypeUtil.createOptionalAttrDef("address", "Address"),
 AtlasTypeUtil.createOptionalAttrDef("birthday", "date"),
@@ -113,7 +110,7 @@ public final class TestUtilsV2 {
 AtlasTypeUtil.createOptionalAttrDef("approximationOfPi", 
"bigdecimal")
 );
 
-AtlasEntityDef employeeTypeDef = 
AtlasTypeUtil.createClassTypeDef("Employee", "Employee"+_description, 
ImmutableSet.of("Person"),
+AtlasEntityDef employeeTypeDef = 
AtlasTypeUtil.createClassTypeDef("Employee", "Employee"+_description, 
Collections.singleton("Person"),
 AtlasTypeUtil.createOptionalAttrDef("orgLevel", "OrgLevel"),
 new AtlasAttributeDef("department", "Department", false,
 AtlasAttributeDef.Cardinality.SINGLE, 1, 1,
@@ -142,18 +139,18 @@ public final class TestUtilsV2 {
 put(AtlasConstraintDef.CONSTRAINT_PARAM_ATTRIBUTE, 
"employees");
 }}));
 
-AtlasEntityDef managerTypeDef = 
AtlasTypeUtil.createClassTypeDef("Manager", "Manager"+_description, 
ImmutableSet.of("Employee"),
+AtlasEntityDef managerTypeDef = 
AtlasTypeUtil.createClassTypeDef("Manager", "Manager"+_description, 
Collections.singleton("Employee"),
 new AtlasAttributeDef("subordinates", 
String.format("array<%s>", "Employee"), false, 
AtlasAttributeDef.Cardinality.SET,
 1, 10, false, false,
 Collections.emptyList()));
 
 AtlasClassificationDef securityClearanceTypeDef =
-AtlasTypeUtil.createTraitTypeDef("SecurityClearance", 
"SecurityClearance"+_description, ImmutableSet.of(),
+AtlasTypeUtil.createTraitTypeDef("SecurityClearance", 
"SecurityClearance"+_description, Collections.emptySet(),
 AtlasTypeUtil.createRequiredAttrDef("level", "int"));
 
-AtlasTypesDef ret = new AtlasTypesDef(ImmutableList.of(orgLevelEnum), 
ImmutableList.of(addressDetails),
-ImmutableList.of(securityClearanceTypeDef),
-ImmutableList.of(deptTypeDef, personTypeDef, employeeTypeDef, 
managerTypeDef));
+AtlasTypesDef ret = new 
AtlasTypesDef(Collections.singletonList(orgLevelEnum), 
Collections.singletonList(addressDetails),
+Collections.singletonList(securityClearanceTypeDef),
+Arrays.asList(deptTypeDef, 

[1/4] atlas git commit: ATLAS-2251: removed typesystem module and scala, json4s, gson libraries

2017-11-08 Thread madhan
Repository: atlas
Updated Branches:
  refs/heads/ATLAS-2251 d511b2888 -> 0a00dcb1a


http://git-wip-us.apache.org/repos/asf/atlas/blob/0a00dcb1/typesystem/src/test/resources/users-credentials.properties
--
diff --git a/typesystem/src/test/resources/users-credentials.properties 
b/typesystem/src/test/resources/users-credentials.properties
deleted file mode 100644
index 3fc3bb1..000
--- a/typesystem/src/test/resources/users-credentials.properties
+++ /dev/null
@@ -1,3 +0,0 @@
-#username=group::sha256-password
-admin=ADMIN::8c6976e5b5410415bde908bd4dee15dfb167a9c873fc4bb8a81f6f2ab448a918
-rangertagsync=RANGER_TAG_SYNC::e3f67240f5117d1753c940dae9eea772d36ed5fe9bd9c94a300e40413f1afb9d

http://git-wip-us.apache.org/repos/asf/atlas/blob/0a00dcb1/webapp/pom.xml
--
diff --git a/webapp/pom.xml b/webapp/pom.xml
index 782c280..569e06f 100755
--- a/webapp/pom.xml
+++ b/webapp/pom.xml
@@ -88,11 +88,6 @@
 
 
 org.apache.atlas
-atlas-typesystem
-
-
-
-org.apache.atlas
 atlas-repository
 
 
@@ -262,11 +257,6 @@
 
 
 
-com.google.code.gson
-gson
-
-
-
 org.eclipse.jetty
 jetty-jsp
 
@@ -390,13 +380,6 @@
 
 
 org.apache.atlas
-atlas-typesystem
-tests
-test
-
-
-
-org.apache.atlas
 atlas-intg
 tests
 test
@@ -638,7 +621,7 @@
 
 
 atlas.conf
-
${project.build.directory}/../../typesystem/target/test-classes
+
${project.build.directory}/../../intg/target/test-classes
 
 
 atlas-stop
@@ -665,11 +648,6 @@
 
 
 
-
-net.alchim31.maven
-scala-maven-plugin
-3.2.0
-

 org.apache.maven.plugins
 maven-resources-plugin

http://git-wip-us.apache.org/repos/asf/atlas/blob/0a00dcb1/webapp/src/main/java/org/apache/atlas/classification/InterfaceAudience.java
--
diff --git 
a/webapp/src/main/java/org/apache/atlas/classification/InterfaceAudience.java 
b/webapp/src/main/java/org/apache/atlas/classification/InterfaceAudience.java
new file mode 100755
index 000..ae162ac
--- /dev/null
+++ 
b/webapp/src/main/java/org/apache/atlas/classification/InterfaceAudience.java
@@ -0,0 +1,48 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.atlas.classification;
+
+import java.lang.annotation.Documented;
+import java.lang.annotation.Retention;
+import java.lang.annotation.RetentionPolicy;
+
+/**
+ * Annotation to mark methods for consumption.
+ */
+@InterfaceAudience.Public
+public class InterfaceAudience {
+private InterfaceAudience() {
+}
+
+@Documented
+@Retention(RetentionPolicy.RUNTIME)
+public @interface Private {
+}
+
+@Documented
+@Retention(RetentionPolicy.RUNTIME)
+public @interface LimitedPrivate {
+String[] value();
+}
+
+@Documented
+@Retention(RetentionPolicy.RUNTIME)
+public @interface Public {
+}
+}

http://git-wip-us.apache.org/repos/asf/atlas/blob/0a00dcb1/webapp/src/main/java/org/apache/atlas/examples/QuickStart.java
--
diff --git a/webapp/src/main/java/org/apache/atlas/examples/QuickStart.java 
b/webapp/src/main/java/org/apache/atlas/examples/QuickStart.java
index 5362084..0e2d32d 100755
--- a/webapp/src/main/java/org/apache/atlas/examples/QuickStart.java
+++ b/webapp/src/main/java/org/apache/atlas/examples/QuickStart.java
@@ -20,8 +20,6 @@ package org.apache.atlas.examples;
 
 import com.google.common.annotations.VisibleForTesting;
 import com.google.common.base.Preconditions;

[4/4] atlas git commit: ATLAS-2251: removed typesystem module and scala, json4s, gson libraries

2017-11-08 Thread madhan
ATLAS-2251: removed typesystem module and scala, json4s, gson libraries


Project: http://git-wip-us.apache.org/repos/asf/atlas/repo
Commit: http://git-wip-us.apache.org/repos/asf/atlas/commit/0a00dcb1
Tree: http://git-wip-us.apache.org/repos/asf/atlas/tree/0a00dcb1
Diff: http://git-wip-us.apache.org/repos/asf/atlas/diff/0a00dcb1

Branch: refs/heads/ATLAS-2251
Commit: 0a00dcb1a504f9959617f95942f4bca3c581
Parents: d511b28
Author: Madhan Neethiraj <mad...@apache.org>
Authored: Wed Nov 8 16:36:17 2017 -0800
Committer: Madhan Neethiraj <mad...@apache.org>
Committed: Wed Nov 8 20:50:43 2017 -0800

--
 addons/falcon-bridge/pom.xml|  61 +-
 addons/hbase-bridge/pom.xml |  97 +--
 addons/hdfs-model/pom.xml   |  66 --
 addons/hive-bridge/pom.xml  |  56 +-
 .../hive/bridge/HiveMetaStoreBridgeTest.java|   2 +-
 .../org/apache/atlas/hive/hook/HiveHookIT.java  |  13 +-
 addons/sqoop-bridge/pom.xml |  61 +-
 addons/storm-bridge/pom.xml | 115 +---
 authorization/pom.xml   |   7 +
 client/client-v1/pom.xml|   9 +-
 client/pom.xml  |  12 +-
 common/pom.xml  |   1 +
 graphdb/api/pom.xml |   6 +-
 graphdb/common/pom.xml  |   6 +
 intg/pom.xml|  18 +-
 .../apache/atlas/TestRelationshipUtilsV2.java   |  26 +-
 .../test/java/org/apache/atlas/TestUtilsV2.java | 140 ++--
 .../atlas/type/TestAtlasRelationshipType.java   |  14 +-
 .../test/resources/atlas-application.properties | 145 +
 intg/src/test/resources/policy-store.txt|   9 +
 intg/src/test/resources/sampleInstance.json |  72 +++
 intg/src/test/resources/sampleTypes.json| 633 +++
 .../test/resources/users-credentials.properties |   3 +
 notification/pom.xml|  33 +-
 .../atlas/kafka/KafkaNotificationMockTest.java  |   2 +-
 pom.xml | 137 +---
 repository/pom.xml  |  85 +--
 .../atlas/discovery/EntityDiscoveryService.java |  14 +-
 .../org/apache/atlas/query/QueryParser.java |   6 +-
 .../atlas/query/SelectExpressionHelper.java |   6 +-
 server-api/pom.xml  |   5 -
 typesystem/.gitignore   |  35 -
 typesystem/LICENSE.md   | 201 --
 typesystem/doc/IDataType.uml| 190 --
 typesystem/doc/Instance.uml | 126 
 typesystem/doc/QueryDSL.org | 266 
 typesystem/doc/notes.MD |  78 ---
 typesystem/pom.xml  | 165 -
 .../atlas/classification/InterfaceAudience.java |  48 --
 .../exception/TypeExistsException.java  |  27 -
 .../exception/TypeNotFoundException.java|  46 --
 .../atlas/typesystem/types/package-info.java| 125 
 typesystem/src/main/resources/atlas-log4j.xml   | 105 ---
 .../test/resources/atlas-application.properties | 145 -
 typesystem/src/test/resources/policy-store.txt  |   9 -
 .../src/test/resources/sampleInstance.json  |  72 ---
 typesystem/src/test/resources/sampleTypes.json  | 633 ---
 .../test/resources/users-credentials.properties |   3 -
 webapp/pom.xml  |  24 +-
 .../atlas/classification/InterfaceAudience.java |  48 ++
 .../org/apache/atlas/examples/QuickStart.java   |  38 +-
 .../org/apache/atlas/examples/QuickStartV2.java |  66 +-
 .../atlas/web/service/ActiveInstanceState.java  |   2 +-
 .../notification/EntityNotificationIT.java  |   8 +-
 .../atlas/web/integration/BasicSearchIT.java|   4 +-
 .../DataSetLineageJerseyResourceIT.java |  34 +-
 .../EntityDiscoveryJerseyResourceIT.java|  11 +-
 .../web/integration/EntityJerseyResourceIT.java |  43 +-
 .../EntityLineageJerseyResourceIT.java  |  32 +-
 .../integration/EntityV2JerseyResourceIT.java   |  35 +-
 .../MetadataDiscoveryJerseyResourceIT.java  |  11 +-
 .../integration/TypedefsJerseyResourceIT.java   |  23 +-
 .../web/integration/TypesJerseyResourceIT.java  |  36 +-
 .../web/service/ActiveInstanceStateTest.java|   2 +-
 64 files changed, 1225 insertions(+), 3326 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/atlas/blob/0a00dcb1/addons/falcon-bridge/pom.xml
--
diff --git a/addons/falcon-bridge/pom.xml b/addons/falcon-bridge/pom.xml
index a732e5d..f5b0b45 100644
--- a/addons/falcon-bridge/pom.xml
+++ b/addons/falcon-bridge/pom.xml
@@ -48,11 +48,6 @@
 
 
 org.apache.atlas
- 

[2/4] atlas git commit: ATLAS-2251: removed typesystem module and scala, json4s, gson libraries

2017-11-08 Thread madhan
http://git-wip-us.apache.org/repos/asf/atlas/blob/0a00dcb1/typesystem/LICENSE.md
--
diff --git a/typesystem/LICENSE.md b/typesystem/LICENSE.md
deleted file mode 100755
index 5c304d1..000
--- a/typesystem/LICENSE.md
+++ /dev/null
@@ -1,201 +0,0 @@
-Apache License
-   Version 2.0, January 2004
-http://www.apache.org/licenses/
-
-   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
-
-   1. Definitions.
-
-  "License" shall mean the terms and conditions for use, reproduction,
-  and distribution as defined by Sections 1 through 9 of this document.
-
-  "Licensor" shall mean the copyright owner or entity authorized by
-  the copyright owner that is granting the License.
-
-  "Legal Entity" shall mean the union of the acting entity and all
-  other entities that control, are controlled by, or are under common
-  control with that entity. For the purposes of this definition,
-  "control" means (i) the power, direct or indirect, to cause the
-  direction or management of such entity, whether by contract or
-  otherwise, or (ii) ownership of fifty percent (50%) or more of the
-  outstanding shares, or (iii) beneficial ownership of such entity.
-
-  "You" (or "Your") shall mean an individual or Legal Entity
-  exercising permissions granted by this License.
-
-  "Source" form shall mean the preferred form for making modifications,
-  including but not limited to software source code, documentation
-  source, and configuration files.
-
-  "Object" form shall mean any form resulting from mechanical
-  transformation or translation of a Source form, including but
-  not limited to compiled object code, generated documentation,
-  and conversions to other media types.
-
-  "Work" shall mean the work of authorship, whether in Source or
-  Object form, made available under the License, as indicated by a
-  copyright notice that is included in or attached to the work
-  (an example is provided in the Appendix below).
-
-  "Derivative Works" shall mean any work, whether in Source or Object
-  form, that is based on (or derived from) the Work and for which the
-  editorial revisions, annotations, elaborations, or other modifications
-  represent, as a whole, an original work of authorship. For the purposes
-  of this License, Derivative Works shall not include works that remain
-  separable from, or merely link (or bind by name) to the interfaces of,
-  the Work and Derivative Works thereof.
-
-  "Contribution" shall mean any work of authorship, including
-  the original version of the Work and any modifications or additions
-  to that Work or Derivative Works thereof, that is intentionally
-  submitted to Licensor for inclusion in the Work by the copyright owner
-  or by an individual or Legal Entity authorized to submit on behalf of
-  the copyright owner. For the purposes of this definition, "submitted"
-  means any form of electronic, verbal, or written communication sent
-  to the Licensor or its representatives, including but not limited to
-  communication on electronic mailing lists, source code control systems,
-  and issue tracking systems that are managed by, or on behalf of, the
-  Licensor for the purpose of discussing and improving the Work, but
-  excluding communication that is conspicuously marked or otherwise
-  designated in writing by the copyright owner as "Not a Contribution."
-
-  "Contributor" shall mean Licensor and any individual or Legal Entity
-  on behalf of whom a Contribution has been received by Licensor and
-  subsequently incorporated within the Work.
-
-   2. Grant of Copyright License. Subject to the terms and conditions of
-  this License, each Contributor hereby grants to You a perpetual,
-  worldwide, non-exclusive, no-charge, royalty-free, irrevocable
-  copyright license to reproduce, prepare Derivative Works of,
-  publicly display, publicly perform, sublicense, and distribute the
-  Work and such Derivative Works in Source or Object form.
-
-   3. Grant of Patent License. Subject to the terms and conditions of
-  this License, each Contributor hereby grants to You a perpetual,
-  worldwide, non-exclusive, no-charge, royalty-free, irrevocable
-  (except as stated in this section) patent license to make, have made,
-  use, offer to sell, sell, import, and otherwise transfer the Work,
-  where such license applies only to those patent claims licensable
-  by such Contributor that are necessarily infringed by their
-  Contribution(s) alone or by combination of their Contribution(s)
-  with the Work to which such Contribution(s) was submitted. If You
-  institute patent litigation against any entity (including a
- 

<    1   2   3   4   5   6   7   8   9   10   >