Re: Review Request 36935: ATLAS-86 Jenkins build failing as of build #41

[Suma Shivaprasad]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36935/#review93572
---



src/conf/application.properties (line 43)
https://reviews.apache.org/r/36935/#comment147957

{ is missing


- Suma Shivaprasad


On July 30, 2015, 9:33 a.m., Shwetha GS wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/36935/
 ---
 
 (Updated July 30, 2015, 9:33 a.m.)
 
 
 Review request for atlas.
 
 
 Bugs: ATLAS-86
 https://issues.apache.org/jira/browse/ATLAS-86
 
 
 Repository: atlas
 
 
 Description
 ---
 
 Main changes:
 1. Moved security tests from hive-bridge module to webapp
 2. In security tests - moved common code out to base classes, changed the 
 keystore/war paths to absolute paths, writing application/client properties 
 to random target
 3. commons-configuration 1.10 already supports variables. So, removed manual 
 handling of variables
 
 
 Diffs
 -
 
   addons/hive-bridge/pom.xml 117d76c 
   
 addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/BaseSSLAndKerberosTest.java
  11163c8 
   
 addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/NegativeSSLAndKerberosHiveHookIT.java
  891e06c 
   
 addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/SSLAndKerberosHiveHookIT.java
  7471680 
   
 addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/SSLHiveHookIT.java
  b114d5a 
   client/pom.xml be19a71 
   client/src/main/java/org/apache/atlas/ApplicationProperties.java ad87d8d 
   client/src/main/java/org/apache/atlas/security/SecureClientUtils.java 
 b70c5ce 
   client/src/test/resources/application.properties 9672d1e 
   pom.xml c201b4c 
   src/conf/application.properties 51f2529 
   webapp/pom.xml 86e1ebf 
   webapp/src/main/java/org/apache/atlas/web/service/SecureEmbeddedServer.java 
 ca71293 
   webapp/src/test/java/org/apache/atlas/web/TestUtils.java PRE-CREATION 
   
 webapp/src/test/java/org/apache/atlas/web/security/BaseSSLAndKerberosTest.java
  PRE-CREATION 
   webapp/src/test/java/org/apache/atlas/web/security/BaseSecurityTest.java 
 67f3901 
   
 webapp/src/test/java/org/apache/atlas/web/security/NegativeSSLAndKerberosIT.java
  PRE-CREATION 
   webapp/src/test/java/org/apache/atlas/web/security/SSLAndKerberosIT.java 
 PRE-CREATION 
   webapp/src/test/java/org/apache/atlas/web/security/SSLIT.java PRE-CREATION 
 
 Diff: https://reviews.apache.org/r/36935/diff/
 
 
 Testing
 ---
 
 UTs
 
 
 Thanks,
 
 Shwetha GS
 

Re: Review Request 36935: ATLAS-86 Jenkins build failing as of build #41

[Suma Shivaprasad]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36935/#review93574
---


Can we run tests in distributed mode once?

- Suma Shivaprasad


On July 30, 2015, 9:33 a.m., Shwetha GS wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/36935/
 ---
 
 (Updated July 30, 2015, 9:33 a.m.)
 
 
 Review request for atlas.
 
 
 Bugs: ATLAS-86
 https://issues.apache.org/jira/browse/ATLAS-86
 
 
 Repository: atlas
 
 
 Description
 ---
 
 Main changes:
 1. Moved security tests from hive-bridge module to webapp
 2. In security tests - moved common code out to base classes, changed the 
 keystore/war paths to absolute paths, writing application/client properties 
 to random target
 3. commons-configuration 1.10 already supports variables. So, removed manual 
 handling of variables
 
 
 Diffs
 -
 
   addons/hive-bridge/pom.xml 117d76c 
   
 addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/BaseSSLAndKerberosTest.java
  11163c8 
   
 addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/NegativeSSLAndKerberosHiveHookIT.java
  891e06c 
   
 addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/SSLAndKerberosHiveHookIT.java
  7471680 
   
 addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/SSLHiveHookIT.java
  b114d5a 
   client/pom.xml be19a71 
   client/src/main/java/org/apache/atlas/ApplicationProperties.java ad87d8d 
   client/src/main/java/org/apache/atlas/security/SecureClientUtils.java 
 b70c5ce 
   client/src/test/resources/application.properties 9672d1e 
   pom.xml c201b4c 
   src/conf/application.properties 51f2529 
   webapp/pom.xml 86e1ebf 
   webapp/src/main/java/org/apache/atlas/web/service/SecureEmbeddedServer.java 
 ca71293 
   webapp/src/test/java/org/apache/atlas/web/TestUtils.java PRE-CREATION 
   
 webapp/src/test/java/org/apache/atlas/web/security/BaseSSLAndKerberosTest.java
  PRE-CREATION 
   webapp/src/test/java/org/apache/atlas/web/security/BaseSecurityTest.java 
 67f3901 
   
 webapp/src/test/java/org/apache/atlas/web/security/NegativeSSLAndKerberosIT.java
  PRE-CREATION 
   webapp/src/test/java/org/apache/atlas/web/security/SSLAndKerberosIT.java 
 PRE-CREATION 
   webapp/src/test/java/org/apache/atlas/web/security/SSLIT.java PRE-CREATION 
 
 Diff: https://reviews.apache.org/r/36935/diff/
 
 
 Testing
 ---
 
 UTs
 
 
 Thanks,
 
 Shwetha GS
 

Review Request 37036: Configuration files and scripts for solr deployment

[Suma Shivaprasad]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/37036/
---

Review request for atlas and Shwetha GS.


Bugs: ATLAS-91
https://issues.apache.org/jira/browse/ATLAS-91


Repository: atlas


Description
---

Configuration files and scripts for solr deployment


Diffs
-

  src/bin/solr/solr.in.sh PRE-CREATION 
  src/bin/solr/start_solr.sh.template PRE-CREATION 
  src/bin/solr/stop_solr.sh.template PRE-CREATION 
  src/conf/solr/edge_index/conf/currency.xml PRE-CREATION 
  src/conf/solr/edge_index/conf/lang/stopwords_en.txt PRE-CREATION 
  src/conf/solr/edge_index/conf/protwords.txt PRE-CREATION 
  src/conf/solr/edge_index/conf/schema.xml PRE-CREATION 
  src/conf/solr/edge_index/conf/solrconfig.xml PRE-CREATION 
  src/conf/solr/edge_index/conf/stopwords.txt PRE-CREATION 
  src/conf/solr/edge_index/conf/synonyms.txt PRE-CREATION 
  src/conf/solr/edge_index/core.properties PRE-CREATION 
  src/conf/solr/edge_index/core.properties.template PRE-CREATION 
  src/conf/solr/fulltext_index/conf/currency.xml PRE-CREATION 
  src/conf/solr/fulltext_index/conf/lang/stopwords_en.txt PRE-CREATION 
  src/conf/solr/fulltext_index/conf/protwords.txt PRE-CREATION 
  src/conf/solr/fulltext_index/conf/schema.xml PRE-CREATION 
  src/conf/solr/fulltext_index/conf/solrconfig.xml PRE-CREATION 
  src/conf/solr/fulltext_index/conf/stopwords.txt PRE-CREATION 
  src/conf/solr/fulltext_index/conf/synonyms.txt PRE-CREATION 
  src/conf/solr/fulltext_index/core.properties PRE-CREATION 
  src/conf/solr/fulltext_index/core.properties.template PRE-CREATION 
  src/conf/solr/solr.xml PRE-CREATION 
  src/conf/solr/vertex_index/conf/currency.xml PRE-CREATION 
  src/conf/solr/vertex_index/conf/lang/stopwords_en.txt PRE-CREATION 
  src/conf/solr/vertex_index/conf/protwords.txt PRE-CREATION 
  src/conf/solr/vertex_index/conf/schema.xml PRE-CREATION 
  src/conf/solr/vertex_index/conf/solrconfig.xml PRE-CREATION 
  src/conf/solr/vertex_index/conf/stopwords.txt PRE-CREATION 
  src/conf/solr/vertex_index/conf/synonyms.txt PRE-CREATION 
  src/conf/solr/vertex_index/core.properties PRE-CREATION 
  src/conf/solr/vertex_index/core.properties.template PRE-CREATION 
  src/conf/solr/zoo.cfg PRE-CREATION 

Diff: https://reviews.apache.org/r/37036/diff/


Testing
---

NA


Thanks,

Suma Shivaprasad

Re: Review Request 37285: solr conf files and changes to make it work with upgraded titan

[Suma Shivaprasad]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/37285/
---

(Updated Aug. 10, 2015, 9:12 a.m.)


Review request for atlas and Shwetha GS.


Changes
---

Fixed review comments


Repository: atlas


Description
---

changes for titan upgraded to solr 5


Diffs (updated)
-

  client/pom.xml d393b3a 
  docs/src/site/twiki/InstallationSteps.twiki 0391c2d 
  pom.xml facd539 
  repository/pom.xml a2f8e08 
  src/conf/solr/atlasConfig/conf/currency.xml PRE-CREATION 
  src/conf/solr/atlasConfig/conf/lang/stopwords_en.txt PRE-CREATION 
  src/conf/solr/atlasConfig/conf/protwords.txt PRE-CREATION 
  src/conf/solr/atlasConfig/conf/schema.xml PRE-CREATION 
  src/conf/solr/atlasConfig/conf/solrconfig.xml PRE-CREATION 
  src/conf/solr/atlasConfig/conf/stopwords.txt PRE-CREATION 
  src/conf/solr/atlasConfig/conf/synonyms.txt PRE-CREATION 

Diff: https://reviews.apache.org/r/37285/diff/


Testing
---

All tests pass . No new tests added


Thanks,

Suma Shivaprasad

Re: Review Request 37285: solr conf files and changes to make it work with upgraded titan

[Suma Shivaprasad]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/37285/
---

(Updated Aug. 10, 2015, 11:16 a.m.)


Review request for atlas and Shwetha GS.


Changes
---

Fixed doc review comments


Repository: atlas


Description
---

changes for titan upgraded to solr 5


Diffs (updated)
-

  client/pom.xml d393b3a 
  docs/src/site/twiki/InstallationSteps.twiki 0391c2d 
  pom.xml facd539 
  repository/pom.xml a2f8e08 
  src/conf/solr/atlasConfig/conf/currency.xml PRE-CREATION 
  src/conf/solr/atlasConfig/conf/lang/stopwords_en.txt PRE-CREATION 
  src/conf/solr/atlasConfig/conf/protwords.txt PRE-CREATION 
  src/conf/solr/atlasConfig/conf/schema.xml PRE-CREATION 
  src/conf/solr/atlasConfig/conf/solrconfig.xml PRE-CREATION 
  src/conf/solr/atlasConfig/conf/stopwords.txt PRE-CREATION 
  src/conf/solr/atlasConfig/conf/synonyms.txt PRE-CREATION 

Diff: https://reviews.apache.org/r/37285/diff/


Testing
---

All tests pass . No new tests added


Thanks,

Suma Shivaprasad

Review Request 37285: solr conf files and changes to make it work with upgraded titan

[Suma Shivaprasad]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/37285/
---

Review request for atlas and Shwetha GS.


Repository: atlas


Description
---

changes for titan upgraded to solr 5


Diffs
-

  client/pom.xml d393b3a 
  pom.xml facd539 
  repository/pom.xml a2f8e08 
  src/conf/solr/atlasConfig/conf/currency.xml PRE-CREATION 
  src/conf/solr/atlasConfig/conf/lang/stopwords_en.txt PRE-CREATION 
  src/conf/solr/atlasConfig/conf/protwords.txt PRE-CREATION 
  src/conf/solr/atlasConfig/conf/schema.xml PRE-CREATION 
  src/conf/solr/atlasConfig/conf/solrconfig.xml PRE-CREATION 
  src/conf/solr/atlasConfig/conf/stopwords.txt PRE-CREATION 
  src/conf/solr/atlasConfig/conf/synonyms.txt PRE-CREATION 
  src/conf/solr/solr.xml PRE-CREATION 
  src/conf/solr/zoo.cfg PRE-CREATION 

Diff: https://reviews.apache.org/r/37285/diff/


Testing
---

All tests pass . No new tests added


Thanks,

Suma Shivaprasad

Re: Review Request 37181: ATLAS-59 De-duping of entities

[Suma Shivaprasad]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/37181/#review94406
---



repository/src/main/java/org/apache/atlas/repository/MetadataRepository.java 
(line 83)
https://reviews.apache.org/r/37181/#comment148994

throws RepositoryException, EntityExistsException should be more clear?



repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java 
(line 117)
https://reviews.apache.org/r/37181/#comment148992

Nice +1


- Suma Shivaprasad


On Aug. 6, 2015, 11:59 a.m., Shwetha GS wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/37181/
 ---
 
 (Updated Aug. 6, 2015, 11:59 a.m.)
 
 
 Review request for atlas.
 
 
 Bugs: ATLAS-59
 https://issues.apache.org/jira/browse/ATLAS-59
 
 
 Repository: atlas
 
 
 Description
 ---
 
 Exposes qualifiedName that can be set on Referenceable. In the backend, it 
 creates unique index for this property for all class types and createEntity 
 fails if there is another entity of the same type with the same qualified name
 
 
 Diffs
 -
 
   repository/src/main/java/org/apache/atlas/repository/Constants.java d5d3b3c 
   
 repository/src/main/java/org/apache/atlas/repository/MetadataRepository.java 
 1b0faa5 
   
 repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepository.java
  cc10ea3 
   
 repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java
  10babed 
   repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java 
 ceb8a65 
   
 typesystem/src/main/java/org/apache/atlas/typesystem/IReferenceableInstance.java
  9285014 
   typesystem/src/main/java/org/apache/atlas/typesystem/Referenceable.java 
 213e46c 
   typesystem/src/main/java/org/apache/atlas/typesystem/persistence/Id.java 
 641146a 
   
 typesystem/src/main/java/org/apache/atlas/typesystem/persistence/ReferenceableInstance.java
  911a5f4 
   typesystem/src/main/java/org/apache/atlas/typesystem/types/ClassType.java 
 cdfbf07 
   
 typesystem/src/main/scala/org/apache/atlas/typesystem/json/InstanceSerialization.scala
  f792858 
   webapp/src/main/java/org/apache/atlas/web/resources/EntityResource.java 
 dcb460d 
   
 webapp/src/test/java/org/apache/atlas/web/resources/EntityJerseyResourceIT.java
  e6a7325 
 
 Diff: https://reviews.apache.org/r/37181/diff/
 
 
 Testing
 ---
 
 UT and IT
 
 
 Thanks,
 
 Shwetha GS
 

Re: Review Request 36724: ATLAS-80 Support for variables in application properties

[Suma Shivaprasad]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36724/#review93087
---



client/src/main/java/org/apache/atlas/ApplicationProperties.java (line 99)
https://reviews.apache.org/r/36724/#comment147345

+1. LGTM. Minor nit. Please correct indentation in the substituteVars 
function


- Suma Shivaprasad


On July 24, 2015, 5:52 a.m., Shwetha GS wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/36724/
 ---
 
 (Updated July 24, 2015, 5:52 a.m.)
 
 
 Review request for atlas.
 
 
 Bugs: ATLAS-80
 https://issues.apache.org/jira/browse/ATLAS-80
 
 
 Repository: atlas
 
 
 Description
 ---
 
 Adds support for system variables in properties file. It also adds support 
 for re-using already defined variable. For example:
 atlas.data=/var/data/${user.name}/atlas
 atlas.graph.data=${atlas.data}/graph
 
 Also made sure application.properties is loaded just once
 
 
 Diffs
 -
 
   
 addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/NegativeSSLAndKerberosHiveHookIT.java
  072c36b 
   
 addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/SSLAndKerberosHiveHookIT.java
  6fab2ee 
   client/src/main/java/org/apache/atlas/ApplicationProperties.java 
 PRE-CREATION 
   client/src/main/java/org/apache/atlas/AtlasClient.java ff03bc1 
   client/src/main/java/org/apache/atlas/PropertiesUtil.java bc56cbf 
   client/src/main/java/org/apache/atlas/security/SecureClientUtils.java 
 eb55b4a 
   client/src/main/java/org/apache/atlas/security/SecurityProperties.java 
 e03ffcc 
   client/src/test/java/org/apache/atlas/ApplicationPropertiesTest.java 
 PRE-CREATION 
   client/src/test/resources/application.properties PRE-CREATION 
   docs/pom.xml 55f5f8d 
   pom.xml a8afcc3 
   repository/src/main/java/org/apache/atlas/discovery/HiveLineageService.java 
 76d94a3 
   
 repository/src/main/java/org/apache/atlas/repository/graph/TitanGraphProvider.java
  d7ed1f1 
   src/bin/atlas_start.py 0ad188c 
   src/conf/application.properties 5c8c59a 
   src/test/python/scripts/TestMetadata.py 9ca346d 
   webapp/src/main/java/org/apache/atlas/Main.java 6cdeca8 
   
 webapp/src/main/java/org/apache/atlas/web/filters/AtlasAuthenticationFilter.java
  80d586b 
   webapp/src/main/java/org/apache/atlas/web/listeners/GuiceServletConfig.java 
 b643f13 
   webapp/src/main/java/org/apache/atlas/web/listeners/LoginProcessor.java 
 5f180f5 
   webapp/src/main/java/org/apache/atlas/web/service/EmbeddedServer.java 
 16591d8 
   webapp/src/test/java/org/apache/atlas/web/listeners/LoginProcessorIT.java 
 2148fe2 
 
 Diff: https://reviews.apache.org/r/36724/diff/
 
 
 Testing
 ---
 
 UTs and end to end test
 
 
 Thanks,
 
 Shwetha GS
 

Re: Falcon

[Suma Shivaprasad]

Hi Nigel,

Yes the Falcon plugin was removed since it was not yet mature. Only HIve
plugin is available as of now.There is work in progress to make these
plugins reliable - https://issues.apache.org/jira/browse/ATLAS-58
The plan is to work on Storm and Kafka plugins next. Falcon is not yet
planned for the near term.

Have raised an  issue for Falcon plugin -
https://issues.apache.org/jira/browse/ATLAS-69. You are welcome to take up
if you are interested.

Thanks
Suma


On Tue, Jul 14, 2015 at 8:53 PM, Nigel Jones jon...@uk.ibm.com wrote:

 In the current atlas code I see Hive plugins, but the Falcon plugin that
 was present in the 0.1 code preview is no longer present.

 Is only Hive available for now?
 Is there an expectation that Falcon will reappear in the near term?
 Are any further plugins being tracked in Jira (I couldn't see any)?

 Many thanks
 Nigel.

Re: Review Request 36359: Create configuration properties for HTTP and HTTPS server ports

[Suma Shivaprasad]

 On July 10, 2015, 5:05 p.m., Suma Shivaprasad wrote:
  webapp/src/main/java/org/apache/atlas/Main.java, line 103
  https://reviews.apache.org/r/36359/diff/1/?file=1003745#file1003745line103
 
  Could remove protected to make it package private. Dont get what a 
  protected static method means.
 
 Jonathan Maron wrote:
 I'm not sure about the qualifier either, but making it protected enabled 
 a better testing approach and didn't seem to expose the internals in a 
 detrimental way.
 
 Jonathan Maron wrote:
 Actually, I imagine protected means that the method can be statically 
 invoked from child classes or from within the same package.

Since we need the method only in MainIT, the test which is in same package, we 
can drop the protected and make it 
static int getApplicationPort(CommandLine cmd, IMO. No need to have 
protected.


- Suma


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36359/#review91320
---


On July 9, 2015, 6:06 p.m., Jonathan Maron wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/36359/
 ---
 
 (Updated July 9, 2015, 6:06 p.m.)
 
 
 Review request for atlas, Arpit Gupta and Seetharam Venkatesh.
 
 
 Bugs: ATLAS-36
 https://issues.apache.org/jira/browse/ATLAS-36
 
 
 Repository: atlas
 
 
 Description
 ---
 
 Distinct HTTP and HTTPS persisted port properties will make the configuration 
 clearer and allow for a better experience from management interfaces such as 
 Ambari.
 
 
 Diffs
 -
 
   src/conf/application.properties 6c4c7d2 
   webapp/src/main/java/org/apache/atlas/Main.java 1dd17b4 
   webapp/src/test/java/org/apache/atlas/MainIT.java PRE-CREATION 
 
 Diff: https://reviews.apache.org/r/36359/diff/
 
 
 Testing
 ---
 
 Installed a local server and observed the configuration properties working.
 Added a unit test for port selection logic (MainIT).
 
 
 Thanks,
 
 Jonathan Maron
 

Review Request 36550: Integration with Hbase and solr

[Suma Shivaprasad]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36550/
---

Review request for atlas, Jonathan Maron, Shwetha GS, and Seetharam Venkatesh.


Bugs: ATLAS-37
https://issues.apache.org/jira/browse/ATLAS-37


Repository: atlas


Description
---

Fixed dependencies and code, tests for Titan to work with Hbase and solr


Diffs
-

  addons/hive-bridge/pom.xml 2617aeb 
  
addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/BaseSSLAndKerberosTest.java
 5ae6fe2 
  
addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/NegativeSSLAndKerberosHiveHookIT.java
 629a9f3 
  
addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/SSLAndKerberosHiveHookIT.java
 b8f29aa 
  
addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/SSLHiveHookIT.java 
54bfaf6 
  addons/hive-bridge/src/test/resources/application.properties 50eefed 
  client/src/test/java/org/apache/atlas/security/BaseSecurityTest.java 598f56b 
  pom.xml 6e7c10c 
  repository/pom.xml b23c289 
  repository/src/main/java/org/apache/atlas/RepositoryMetadataModule.java 
e7353f9 
  repository/src/main/java/org/apache/atlas/repository/graph/GraphProvider.java 
5472081 
  
repository/src/main/java/org/apache/atlas/repository/graph/TitanGraphProvider.java
 3c1d151 
  
repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java 
9556272 
  repository/src/test/java/org/apache/atlas/RepositoryServiceLoadingTest.java 
4195955 
  
repository/src/test/java/org/apache/atlas/discovery/GraphBackedDiscoveryServiceTest.java
 67cdd78 
  
repository/src/test/java/org/apache/atlas/discovery/HiveLineageServiceTest.java 
ce00f8c 
  
repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepositoryTest.java
 fb697d0 
  
repository/src/test/java/org/apache/atlas/repository/typestore/GraphBackedTypeStoreTest.java
 c774123 
  repository/src/test/resources/application.properties 20c0c51 
  src/conf/application.properties c6b16cc 
  webapp/pom.xml 0ae4a05 
  webapp/src/main/java/org/apache/atlas/web/listeners/GuiceServletConfig.java 
18ce17c 
  webapp/src/main/java/org/apache/atlas/web/service/EmbeddedServer.java 88200f0 
  webapp/src/main/java/org/apache/atlas/web/service/SecureEmbeddedServer.java 
174dafb 
  webapp/src/main/resources/application.properties c9b8408 
  
webapp/src/test/java/org/apache/atlas/web/filters/MetadataAuthenticationKerberosFilterIT.java
 95acc57 
  
webapp/src/test/java/org/apache/atlas/web/filters/MetadataAuthenticationSimpleFilterIT.java
 5ea2e15 
  webapp/src/test/java/org/apache/atlas/web/listeners/LoginProcessorIT.java 
54a7a7b 
  
webapp/src/test/java/org/apache/atlas/web/listeners/TestGuiceServletConfig.java 
PRE-CREATION 
  webapp/src/test/java/org/apache/atlas/web/security/BaseSecurityTest.java 
PRE-CREATION 
  webapp/src/test/java/org/apache/atlas/web/service/SecureEmbeddedServerIT.java 
3e27532 
  
webapp/src/test/java/org/apache/atlas/web/service/SecureEmbeddedServerITBase.java
 db110ca 
  webapp/src/test/webapp/WEB-INF/web.xml PRE-CREATION 

Diff: https://reviews.apache.org/r/36550/diff/


Testing
---

Ran all UTs and ITS. Except the hive hook security Its, all other tests succeed.


Thanks,

Suma Shivaprasad

Re: Review Request 36550: Integration with Hbase and solr

[Suma Shivaprasad]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36550/
---

(Updated July 17, 2015, 1:12 p.m.)


Review request for atlas, Jonathan Maron, Shwetha GS, and Seetharam Venkatesh.


Changes
---

Embeeded mode tests fixed


Bugs: ATLAS-37
https://issues.apache.org/jira/browse/ATLAS-37


Repository: atlas


Description
---

Fixed dependencies and code, tests for Titan to work with Hbase and solr


Diffs (updated)
-

  addons/hive-bridge/pom.xml 2617aeb 
  
addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/BaseSSLAndKerberosTest.java
 5ae6fe2 
  
addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/NegativeSSLAndKerberosHiveHookIT.java
 629a9f3 
  
addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/SSLAndKerberosHiveHookIT.java
 b8f29aa 
  
addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/SSLHiveHookIT.java 
54bfaf6 
  addons/hive-bridge/src/test/resources/application.properties 50eefed 
  client/src/test/java/org/apache/atlas/security/BaseSecurityTest.java 598f56b 
  pom.xml 6e7c10c 
  repository/pom.xml b23c289 
  repository/src/main/java/org/apache/atlas/RepositoryMetadataModule.java 
e7353f9 
  repository/src/main/java/org/apache/atlas/repository/graph/GraphProvider.java 
5472081 
  
repository/src/main/java/org/apache/atlas/repository/graph/TitanGraphProvider.java
 3c1d151 
  
repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java 
9556272 
  repository/src/test/java/org/apache/atlas/RepositoryServiceLoadingTest.java 
4195955 
  
repository/src/test/java/org/apache/atlas/discovery/GraphBackedDiscoveryServiceTest.java
 67cdd78 
  
repository/src/test/java/org/apache/atlas/discovery/HiveLineageServiceTest.java 
ce00f8c 
  
repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepositoryTest.java
 fb697d0 
  
repository/src/test/java/org/apache/atlas/repository/typestore/GraphBackedTypeStoreTest.java
 c774123 
  repository/src/test/resources/application.properties 20c0c51 
  src/conf/application.properties c6b16cc 
  webapp/pom.xml 0ae4a05 
  webapp/src/main/java/org/apache/atlas/web/listeners/GuiceServletConfig.java 
18ce17c 
  webapp/src/main/java/org/apache/atlas/web/service/EmbeddedServer.java 88200f0 
  webapp/src/main/java/org/apache/atlas/web/service/SecureEmbeddedServer.java 
174dafb 
  webapp/src/main/resources/application.properties c9b8408 
  
webapp/src/test/java/org/apache/atlas/web/filters/MetadataAuthenticationKerberosFilterIT.java
 95acc57 
  
webapp/src/test/java/org/apache/atlas/web/filters/MetadataAuthenticationSimpleFilterIT.java
 5ea2e15 
  webapp/src/test/java/org/apache/atlas/web/listeners/LoginProcessorIT.java 
54a7a7b 
  
webapp/src/test/java/org/apache/atlas/web/listeners/TestGuiceServletConfig.java 
PRE-CREATION 
  webapp/src/test/java/org/apache/atlas/web/security/BaseSecurityTest.java 
PRE-CREATION 
  webapp/src/test/java/org/apache/atlas/web/service/SecureEmbeddedServerIT.java 
3e27532 
  
webapp/src/test/java/org/apache/atlas/web/service/SecureEmbeddedServerITBase.java
 db110ca 
  webapp/src/test/webapp/WEB-INF/web.xml PRE-CREATION 

Diff: https://reviews.apache.org/r/36550/diff/


Testing
---

Ran all UTs and ITS. Except the hive hook security Its, all other tests succeed.


Thanks,

Suma Shivaprasad

Re: Review Request 36550: Integration with Hbase and solr

[Suma Shivaprasad]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36550/
---

(Updated July 20, 2015, 11:19 a.m.)


Review request for atlas, Jonathan Maron, Shwetha GS, and Seetharam Venkatesh.


Changes
---

Fixed review comments


Bugs: ATLAS-37
https://issues.apache.org/jira/browse/ATLAS-37


Repository: atlas


Description
---

Fixed dependencies and code, tests for Titan to work with Hbase and solr


Diffs (updated)
-

  addons/hive-bridge/pom.xml 2617aeb 
  
addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/BaseSSLAndKerberosTest.java
 5ae6fe2 
  
addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/NegativeSSLAndKerberosHiveHookIT.java
 629a9f3 
  
addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/SSLAndKerberosHiveHookIT.java
 b8f29aa 
  
addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/SSLHiveHookIT.java 
54bfaf6 
  addons/hive-bridge/src/test/resources/application.properties 50eefed 
  client/src/test/java/org/apache/atlas/security/BaseSecurityTest.java 598f56b 
  pom.xml 6e7c10c 
  repository/pom.xml b23c289 
  repository/src/main/java/org/apache/atlas/RepositoryMetadataModule.java 
e7353f9 
  repository/src/main/java/org/apache/atlas/repository/graph/GraphProvider.java 
5472081 
  
repository/src/main/java/org/apache/atlas/repository/graph/TitanGraphProvider.java
 3c1d151 
  
repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java 
9556272 
  repository/src/test/java/org/apache/atlas/RepositoryServiceLoadingTest.java 
4195955 
  
repository/src/test/java/org/apache/atlas/discovery/GraphBackedDiscoveryServiceTest.java
 67cdd78 
  
repository/src/test/java/org/apache/atlas/discovery/HiveLineageServiceTest.java 
ce00f8c 
  
repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepositoryTest.java
 fb697d0 
  
repository/src/test/java/org/apache/atlas/repository/typestore/GraphBackedTypeStoreTest.java
 c774123 
  repository/src/test/resources/application.properties 20c0c51 
  src/conf/application.properties c6b16cc 
  webapp/pom.xml 0ae4a05 
  webapp/src/main/java/org/apache/atlas/web/listeners/GuiceServletConfig.java 
18ce17c 
  webapp/src/main/java/org/apache/atlas/web/service/EmbeddedServer.java 88200f0 
  webapp/src/main/java/org/apache/atlas/web/service/SecureEmbeddedServer.java 
174dafb 
  webapp/src/main/resources/application.properties c9b8408 
  
webapp/src/test/java/org/apache/atlas/web/filters/MetadataAuthenticationKerberosFilterIT.java
 95acc57 
  
webapp/src/test/java/org/apache/atlas/web/filters/MetadataAuthenticationSimpleFilterIT.java
 5ea2e15 
  webapp/src/test/java/org/apache/atlas/web/listeners/LoginProcessorIT.java 
54a7a7b 
  
webapp/src/test/java/org/apache/atlas/web/listeners/TestGuiceServletConfig.java 
PRE-CREATION 
  webapp/src/test/java/org/apache/atlas/web/security/BaseSecurityTest.java 
PRE-CREATION 
  webapp/src/test/java/org/apache/atlas/web/service/SecureEmbeddedServerIT.java 
3e27532 
  
webapp/src/test/java/org/apache/atlas/web/service/SecureEmbeddedServerITBase.java
 db110ca 
  webapp/src/test/webapp/WEB-INF/web.xml PRE-CREATION 

Diff: https://reviews.apache.org/r/36550/diff/


Testing
---

Ran all UTs and ITS. Except the hive hook security Its, all other tests succeed.


Thanks,

Suma Shivaprasad

Re: Unable to build Atlas

[Suma Shivaprasad]

HI Rishabh,

Can you pls do a clean checkout and try to build. There are some runtime
files generated in dahsboard which was part of the older build process but
the rat ignore is removed for them

Thanks
Suma



On Mon, Oct 12, 2015 at 10:33 AM, Rishabh Bhardwaj 
wrote:

> Hi,
> I am new to atlas and trying to build it.
> But I am getting the following error.
>
> [ERROR] Failed to execute goal org.apache.rat:apache-rat-plugin:0.7:check
> (rat-check) on project apache-atlas: Too many unapproved licenses: 442 ->
> [Help 1]
>
> Please Help.
>
> Regards,
> Rishabh.
>

Re: Review Request 39157: Configure Consumer Groups for Notifications

[Suma Shivaprasad]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39157/#review102184
---

Ship it!


Ship It!

- Suma Shivaprasad


On Oct. 9, 2015, 1:40 a.m., Tom Beerbower wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39157/
> ---
> 
> (Updated Oct. 9, 2015, 1:40 a.m.)
> 
> 
> Review request for atlas, John Speidel and Shwetha GS.
> 
> 
> Bugs: ATLAS-215
> https://issues.apache.org/jira/browse/ATLAS-215
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Currently with the notification framework we set the properties so that all 
> consumers are created in a single group...
> 
> //todo take group id as argument to allow multiple consumers??
> properties.put(ConsumerConfig.GROUP_ID_CONFIG, ATLAS_GROUP);
> 
> Having Kafka consumers for both the HOOK and ENTITIES topics in a single 
> group causes this issue ...
> 
> java.lang.IllegalArgumentException: requirement failed: Round-robin 
> assignment is allowed only if all consumers in the group subscribe to the 
> same topics, AND if the stream counts across topics are identical for a given 
> consumer instance.
> Topic ATLAS_ENTITIES has the following available consumer streams: 
> Set(atlas_c6401.ambari.apache.org-1443714948705-e518b690-0)
> Topic ATLAS_HOOK has the following available consumer streams: 
> Set(atlas_c6401.ambari.apache.org-1443714214755-8fd9a5f8-0)
> 
> We shouldn't have HOOK consumers and ENTITIES consumers in the same group.
> 
> 
> Diffs
> -
> 
>   notification/pom.xml 2e12520 
>   notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 
> 7b3cf89 
>   
> notification/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java
>  29194a4 
>   
> notification/src/main/java/org/apache/atlas/notification/NotificationInterface.java
>  d4be07b 
>   
> notification/src/test/java/org/apache/atlas/kafka/KafkaNotificationTest.java 
> 735655c 
> 
> Diff: https://reviews.apache.org/r/39157/diff/
> 
> 
> Testing
> ---
> 
> new unit test added
> 
> mvn clean test
> 
> [INFO] 
> 
> [INFO] BUILD SUCCESS
> [INFO] 
> 
> [INFO] Total time: 12.246 s
> [INFO] Finished at: 2015-10-08T21:37:55-04:00
> [INFO] Final Memory: 40M/605M
> [INFO] 
> 
> 
> 
> Thanks,
> 
> Tom Beerbower
> 
>

Re: Review Request 38179: ATLAS-128 - Equality matches on lists

[Suma Shivaprasad]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38179/
---

(Updated Oct. 5, 2015, 9:02 a.m.)


Review request for atlas and Harish Butani.


Changes
---

Fixed all review comments


Bugs: ATLAS-128
https://issues.apache.org/jira/browse/ATLAS-128


Repository: atlas


Description
---

1. Supporting equality matches on array type as array_type_col_name = [ List of 
values ] where list of values is a comma separated list of literals which are 
of primitive type like string, int, etc

2. Changed DSL tests - Gremlin, Expression, GraphBackedDiscoveryServiceTest to 
use the DefaultGraphPersistenceStrategy instead of the inmemory one so that the 
real code execution path is tested
3. Fixed ATLAS-30 - Fixed tests in GraphBackedDiscoveryServiceTest to validate 
expected test results
4. Fixed "typename has fieldname" queries where fieldname was not getting fully 
qualified with typeName.
5. 5. Fixed "type is trait" queries - trait label did not have __

Could not complete load of list types due to issues with validation in DSL. 
Will be covering this in a separate jira.


Diffs (updated)
-

  addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 
589ef16 
  
repository/src/main/java/org/apache/atlas/discovery/graph/DefaultGraphPersistenceStrategy.java
 c446d15 
  repository/src/main/java/org/apache/atlas/repository/MetadataRepository.java 
97d1fe2 
  
repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepository.java
 b44ce60 
  
repository/src/main/java/org/apache/atlas/repository/graph/TitanGraphProvider.java
 6605ae7 
  repository/src/main/scala/org/apache/atlas/query/Expressions.scala 24fa917 
  
repository/src/main/scala/org/apache/atlas/query/GraphPersistenceStrategies.scala
 2cec503 
  repository/src/main/scala/org/apache/atlas/query/GremlinQuery.scala 75fa916 
  repository/src/main/scala/org/apache/atlas/query/QueryParser.scala 5b129fb 
  repository/src/main/scala/org/apache/atlas/query/Resolver.scala 142ba8d 
  repository/src/test/java/org/apache/atlas/BaseHiveRepositoryTest.java 
PRE-CREATION 
  
repository/src/test/java/org/apache/atlas/discovery/GraphBackedDiscoveryServiceTest.java
 c72ffb2 
  
repository/src/test/java/org/apache/atlas/discovery/HiveLineageServiceTest.java 
db51ae5 
  
repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepositoryTest.java
 bec3067 
  
repository/src/test/java/org/apache/atlas/repository/graph/GraphRepoMapperScaleTest.java
 20781db 
  repository/src/test/scala/org/apache/atlas/query/GremlinTest.scala 957a566 
  repository/src/test/scala/org/apache/atlas/query/GremlinTest2.scala ea0b9bb 
  repository/src/test/scala/org/apache/atlas/query/HiveTitanSample.scala 
c662847 
  repository/src/test/scala/org/apache/atlas/query/LineageQueryTest.scala 
0b4dd2b 
  repository/src/test/scala/org/apache/atlas/query/ParserTest.scala 602b2b5 
  repository/src/test/scala/org/apache/atlas/query/QueryTestsUtils.scala 
3c66da0 
  typesystem/src/main/java/org/apache/atlas/typesystem/types/DataTypes.java 
e35453e 

Diff: https://reviews.apache.org/r/38179/diff/


Testing
---

Tests added in Parser, GremlinTest,  GraphBackedDiscoveryTest


Thanks,

Suma Shivaprasad

Re: Review Request 39157: Configure Consumer Groups for Notifications

[Suma Shivaprasad]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39157/#review102064
---



notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java (line 
223)
<https://reviews.apache.org/r/39157/#comment159590>

consumerConnector need to be shutdown


- Suma Shivaprasad


On Oct. 9, 2015, 1:40 a.m., Tom Beerbower wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39157/
> ---
> 
> (Updated Oct. 9, 2015, 1:40 a.m.)
> 
> 
> Review request for atlas, John Speidel and Shwetha GS.
> 
> 
> Bugs: ATLAS-215
> https://issues.apache.org/jira/browse/ATLAS-215
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Currently with the notification framework we set the properties so that all 
> consumers are created in a single group...
> 
> //todo take group id as argument to allow multiple consumers??
> properties.put(ConsumerConfig.GROUP_ID_CONFIG, ATLAS_GROUP);
> 
> Having Kafka consumers for both the HOOK and ENTITIES topics in a single 
> group causes this issue ...
> 
> java.lang.IllegalArgumentException: requirement failed: Round-robin 
> assignment is allowed only if all consumers in the group subscribe to the 
> same topics, AND if the stream counts across topics are identical for a given 
> consumer instance.
> Topic ATLAS_ENTITIES has the following available consumer streams: 
> Set(atlas_c6401.ambari.apache.org-1443714948705-e518b690-0)
> Topic ATLAS_HOOK has the following available consumer streams: 
> Set(atlas_c6401.ambari.apache.org-1443714214755-8fd9a5f8-0)
> 
> We shouldn't have HOOK consumers and ENTITIES consumers in the same group.
> 
> 
> Diffs
> -
> 
>   notification/pom.xml 2e12520 
>   notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 
> 7b3cf89 
>   
> notification/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java
>  29194a4 
>   
> notification/src/main/java/org/apache/atlas/notification/NotificationInterface.java
>  d4be07b 
>   
> notification/src/test/java/org/apache/atlas/kafka/KafkaNotificationTest.java 
> 735655c 
> 
> Diff: https://reviews.apache.org/r/39157/diff/
> 
> 
> Testing
> ---
> 
> new unit test added
> 
> mvn clean test
> 
> [INFO] 
> 
> [INFO] BUILD SUCCESS
> [INFO] 
> 
> [INFO] Total time: 12.246 s
> [INFO] Finished at: 2015-10-08T21:37:55-04:00
> [INFO] Final Memory: 40M/605M
> [INFO] 
> 
> 
> 
> Thanks,
> 
> Tom Beerbower
> 
>

Re: Review Request 39157: Configure Consumer Groups for Notifications

[Suma Shivaprasad]

> On Oct. 9, 2015, 4 p.m., Suma Shivaprasad wrote:
> > notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java, 
> > line 226
> > <https://reviews.apache.org/r/39157/diff/1/?file=1093468#file1093468line226>
> >
> > consumerConnector need to be shutdown

This seems to be taken care of in close() already


- Suma


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39157/#review102064
---


On Oct. 9, 2015, 1:40 a.m., Tom Beerbower wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39157/
> ---
> 
> (Updated Oct. 9, 2015, 1:40 a.m.)
> 
> 
> Review request for atlas, John Speidel and Shwetha GS.
> 
> 
> Bugs: ATLAS-215
> https://issues.apache.org/jira/browse/ATLAS-215
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Currently with the notification framework we set the properties so that all 
> consumers are created in a single group...
> 
> //todo take group id as argument to allow multiple consumers??
> properties.put(ConsumerConfig.GROUP_ID_CONFIG, ATLAS_GROUP);
> 
> Having Kafka consumers for both the HOOK and ENTITIES topics in a single 
> group causes this issue ...
> 
> java.lang.IllegalArgumentException: requirement failed: Round-robin 
> assignment is allowed only if all consumers in the group subscribe to the 
> same topics, AND if the stream counts across topics are identical for a given 
> consumer instance.
> Topic ATLAS_ENTITIES has the following available consumer streams: 
> Set(atlas_c6401.ambari.apache.org-1443714948705-e518b690-0)
> Topic ATLAS_HOOK has the following available consumer streams: 
> Set(atlas_c6401.ambari.apache.org-1443714214755-8fd9a5f8-0)
> 
> We shouldn't have HOOK consumers and ENTITIES consumers in the same group.
> 
> 
> Diffs
> -
> 
>   notification/pom.xml 2e12520 
>   notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 
> 7b3cf89 
>   
> notification/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java
>  29194a4 
>   
> notification/src/main/java/org/apache/atlas/notification/NotificationInterface.java
>  d4be07b 
>   
> notification/src/test/java/org/apache/atlas/kafka/KafkaNotificationTest.java 
> 735655c 
> 
> Diff: https://reviews.apache.org/r/39157/diff/
> 
> 
> Testing
> ---
> 
> new unit test added
> 
> mvn clean test
> 
> [INFO] 
> 
> [INFO] BUILD SUCCESS
> [INFO] 
> 
> [INFO] Total time: 12.246 s
> [INFO] Finished at: 2015-10-08T21:37:55-04:00
> [INFO] Final Memory: 40M/605M
> [INFO] 
> 
> 
> 
> Thanks,
> 
> Tom Beerbower
> 
>

Re: Titan: configuration in atlas

[Suma Shivaprasad]

Hi Nigel,

You should be able to point to the graph created in ATLAS through Rexster
by following the steps at
http://s3.thinkaurelius.com/docs/titan/current/server.html#_downloading_rexster_and_titan_separately.
I was able to get the Rexster instance up this way but got some issue while
loading the vertices though the edges are seen. Need to dig into this
further.

Thanks
Suma


On Thu, Sep 3, 2015 at 8:47 PM, Nigel Jones  wrote:

>
>
> I'm trying to understand more about the setup/configuration - for example
> I'd like to improve my understanding of atlas by:
>   - Use the titan rexster api to view/understand the underlying graph
>   - using titans doghouse UI to poke around
>   - use the gremlin shell
>
> In the Atlas Configuration documentation i can see reference to modifying
> the underlying titan storage backend & a few other aspects of config.
>
> Atlas comes with the 0.5.4 titan jars, but not the full distribution
>
> Before I start trying to add in rexster etc I was wondering if anyone else
> has done this & what the appropriate approach might be
>
> Thanks
> Nigel.
>
>
>
>
>

Re: Titan: configuration in atlas

[Suma Shivaprasad]

I think I figured out the reason why the vertices were not getting loaded.
The atlas jars also need to be copied into REXHOME/ext/titan to make it
work.

ATLAS as you might be ware is using berkeley DB and ES as the default
storage and index backends . You need to configure that in the rextser.xml
conf pointed out in the above link and point to the directory where the
data for them resides.

Connecting through the gremlin shell also requires a .properties file with
all these properties set to load the graph stored by ATLAS.

Thanks
Suma

On Fri, Sep 4, 2015 at 5:47 PM, Suma Shivaprasad <
sumasai.shivapra...@gmail.com> wrote:

> Hi Nigel,
>
> You should be able to point to the graph created in ATLAS through Rexster
> by following the steps at
> http://s3.thinkaurelius.com/docs/titan/current/server.html#_downloading_rexster_and_titan_separately.
> I was able to get the Rexster instance up this way but got some issue while
> loading the vertices though the edges are seen. Need to dig into this
> further.
>
> Thanks
> Suma
>
>
> On Thu, Sep 3, 2015 at 8:47 PM, Nigel Jones <jon...@uk.ibm.com> wrote:
>
>>
>>
>> I'm trying to understand more about the setup/configuration - for example
>> I'd like to improve my understanding of atlas by:
>>   - Use the titan rexster api to view/understand the underlying graph
>>   - using titans doghouse UI to poke around
>>   - use the gremlin shell
>>
>> In the Atlas Configuration documentation i can see reference to modifying
>> the underlying titan storage backend & a few other aspects of config.
>>
>> Atlas comes with the 0.5.4 titan jars, but not the full distribution
>>
>> Before I start trying to add in rexster etc I was wondering if anyone
>> else has done this & what the appropriate approach might be
>>
>> Thanks
>> Nigel.
>>
>>
>>
>>
>>
>

Review Request 38179: ATLAS-128 - Equality matches on lists

[Suma Shivaprasad]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38179/
---

Review request for atlas and Harish Butani.


Bugs: ATLAS-128
https://issues.apache.org/jira/browse/ATLAS-128


Repository: atlas


Description
---

Supporting equality matches on array type as array_type_col_name = [ List of 
values ] where list of values is a comma separated list of literals which are 
of primitive type like string, int, etc

TODO - supporting loading of array of class types. Add tests for array of 
struct, class type loading


Diffs
-

  addons/hive-bridge/pom.xml 914d8c6 
  addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 
2bf3aba 
  pom.xml a2b0b26 
  
repository/src/main/java/org/apache/atlas/discovery/graph/DefaultGraphPersistenceStrategy.java
 c446d15 
  repository/src/main/java/org/apache/atlas/repository/MetadataRepository.java 
1b0faa5 
  
repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepository.java
 cc10ea3 
  repository/src/main/scala/org/apache/atlas/query/Expressions.scala 24fa917 
  
repository/src/main/scala/org/apache/atlas/query/GraphPersistenceStrategies.scala
 2cec503 
  repository/src/main/scala/org/apache/atlas/query/GremlinQuery.scala 75fa916 
  repository/src/main/scala/org/apache/atlas/query/QueryParser.scala 5b129fb 
  repository/src/test/scala/org/apache/atlas/query/GremlinTest.scala 957a566 
  repository/src/test/scala/org/apache/atlas/query/HiveTitanSample.scala 
eef13df 
  repository/src/test/scala/org/apache/atlas/query/ParserTest.scala 602b2b5 
  repository/src/test/scala/org/apache/atlas/query/QueryTestsUtils.scala 
ec21886 
  src/main/assemblies/src-package.xml ca03c7a 
  src/main/assemblies/standalone-package.xml 8c48b22 
  typesystem/src/main/java/org/apache/atlas/typesystem/types/DataTypes.java 
e35453e 
  webapp/pom.xml 65c7bde 

Diff: https://reviews.apache.org/r/38179/diff/


Testing
---

Tests added in GremlinTest


Thanks,

Suma Shivaprasad

Re: Review Request 38341: Provide Entity Change Notification

[Suma Shivaprasad]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38341/#review98993
---



notification/src/main/java/org/apache/atlas/notification/entity/Entity.java 
(line 26)
<https://reviews.apache.org/r/38341/#comment155779>

Already exposed by Referenceable like Shwetha mentioned



notification/src/main/java/org/apache/atlas/notification/entity/Trait.java 
(line 25)
<https://reviews.apache.org/r/38341/#comment155780>

THis can also be removed to use existing interfaces


- Suma Shivaprasad


On Sept. 14, 2015, 4:08 a.m., Tom Beerbower wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38341/
> ---
> 
> (Updated Sept. 14, 2015, 4:08 a.m.)
> 
> 
> Review request for atlas, John Speidel and Shwetha GS.
> 
> 
> Bugs: ATLAS-158
> https://issues.apache.org/jira/browse/ATLAS-158
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Add entity change notification to Atlas based using the existing 
> atlas-notification module.
> 
> First cut at a patch for the Atlas entity change notification.  Note that at 
> a minimum additional unit tests are required.  I'm putting up the review to 
> get some initial feedback.
> 
> 
> Diffs
> -
> 
>   notification/src/main/java/org/apache/atlas/notification/entity/Entity.java 
> PRE-CREATION 
>   
> notification/src/main/java/org/apache/atlas/notification/entity/EntityChangeConsumer.java
>  PRE-CREATION 
>   
> notification/src/main/java/org/apache/atlas/notification/entity/EntityChangeListener.java
>  PRE-CREATION 
>   
> notification/src/main/java/org/apache/atlas/notification/entity/EntityImpl.java
>  PRE-CREATION 
>   
> notification/src/main/java/org/apache/atlas/notification/entity/EntityNotification.java
>  PRE-CREATION 
>   notification/src/main/java/org/apache/atlas/notification/entity/Trait.java 
> PRE-CREATION 
>   
> notification/src/main/java/org/apache/atlas/notification/entity/TraitImpl.java
>  PRE-CREATION 
>   
> notification/src/test/java/org/apache/atlas/notification/entity/EntityImplTest.java
>  PRE-CREATION 
>   
> notification/src/test/java/org/apache/atlas/notification/entity/TraitImplTest.java
>  PRE-CREATION 
>   repository/pom.xml 8e4d0f3 
>   repository/src/main/java/org/apache/atlas/RepositoryMetadataModule.java 
> fbd01de 
>   
> repository/src/main/java/org/apache/atlas/listener/EntityChangeListener.java 
> f58d6de 
>   
> repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java
>  56168db 
> 
> Diff: https://reviews.apache.org/r/38341/diff/
> 
> 
> Testing
> ---
> 
> mvn clean test
> 
> All existing tests pass.
> 
> New unit tests added (more required).
> 
> 
> Thanks,
> 
> Tom Beerbower
> 
>

Re: Search syntax

[Suma Shivaprasad]

= new HashSet<>();
> >>> [ERROR] ^
> >>> [ERROR]
> >>>
> >>>/teeup/source_repo/atlas/typesystem/src/main/java/org/apache/atlas/types
> >>>ys
> >>> tem/types/TypeSystem.java:505: illegal start of type
> >>> [ERROR] List traitTypes = new ArrayList<>();
> >>> [ERROR]^
> >>> [ERROR]
> >>>
> >>>/teeup/source_repo/atlas/typesystem/src/main/java/org/apache/atlas/types
> >>>ys
> >>> tem/types/TypeSystem.java:511: illegal start of type
> >>> [ERROR] List classTypes = new ArrayList<>();
> >>> [ERROR]^
> >>> [ERROR]
> >>>
> >>>/teeup/source_repo/atlas/typesystem/src/main/java/org/apache/atlas/types
> >>>ys
> >>> tem/types/TypeSystem.java:564: illegal start of type
> >>> [ERROR] Map<String, IDataType> newTypes = new HashMap<>();
> >>> [ERROR]   ^
> >>> [ERROR]
> >>>
> >>>/teeup/source_repo/atlas/typesystem/src/main/java/org/apache/atlas/types
> >>>ys
> >>> tem/types/utils/TypesUtil.java:66: illegal start of type
> >>> [ERROR] return new
> >>>HierarchicalTypeDefinition<>(TraitType.class,
> >>> name, superTypes, attrDefs);
> >>> [ERROR]   ^
> >>> [ERROR]
> >>>
> >>>/teeup/source_repo/atlas/typesystem/src/main/java/org/apache/atlas/types
> >>>ys
> >>> tem/types/utils/TypesUtil.java:75: illegal start of type
> >>> [ERROR] return new
> >>>HierarchicalTypeDefinition<>(ClassType.class,
> >>> name, superTypes, attrDefs);
> >>> [ERROR]   ^
> >>> [ERROR] 24 errors
> >>> [INFO]
> >>>
> >>>
> >>> [INFO] Reactor Summary:
> >>> [INFO]
> >>> [INFO] apache-atlas ... SUCCESS
> >>> [03:00 min]
> >>> [INFO] Apache Atlas Typesystem  FAILURE
> >>> [03:48 min]
> >>> [INFO] Apache Atlas Common  SKIPPED
> >>> [INFO] Apache Atlas Client  SKIPPED
> >>> [INFO] Apache Atlas Notification .. SKIPPED
> >>> [INFO] Apache Atlas Repository  SKIPPED
> >>> [INFO] Apache Atlas Web Application ... SKIPPED
> >>> [INFO] Apache Atlas Documentation . SKIPPED
> >>> [INFO] Apache Atlas Hive Bridge ... SKIPPED
> >>> [INFO] Apache Atlas Distribution .. SKIPPED
> >>> [INFO]
> >>>
> >>>
> >>> [INFO] BUILD FAILURE
> >>> [INFO]
> >>>
> >>>
> >>> [INFO] Total time: 07:38 min
> >>> [INFO] Finished at: 2015-09-24T11:31:51-04:00
> >>> [INFO] Final Memory: 42M/200M
> >>> [INFO]
> >>>
> >>>
> >>> [ERROR] Failed to execute goal
> >>> net.alchim31.maven:scala-maven-plugin:3.2.0:compile
> >>>(scala-compile-first)
> >>> on project atlas-typesystem: Execution scala-compile-first of goal
> >>> net.alchim31.maven:scala-maven-plugin:3.2.0:compile failed.
> >>>CompileFailed
> >>> -> [Help 1]
> >>>
> >>>
> >>>
> >>>
> >>> On Sep 23, 2015, at 12:16 PM, Suma Shivaprasad
> >>> <sumasai.shivapra...@gmail.com> wrote:
> >>>
> >>>> Could be an issue with 0.5. I fixed an issue while querying for
> >>>>entities
> >>>> with traits as part of https://reviews.apache.org/r/38179/. . Should
> >>>>be
> >>>> committed soon.
> >>>>
> >>>> Thanks
> >>>> Suma
> >>>>
> >>>>
> >>>> On Wed, Sep 23, 2015 at 8:40 PM, Shwetha Shivalingamurthy <
> >>>> sshivalingamur...@hortonworks.com> wrote:
> >>>>
> >>>>> 'hive_table is ¹ should work. Here is the doc on DSL
> >>>>>syntax:
> >>>>> http://atlas.incubator.apache.org/Search.html
> >>>>>
> >>>>> Regards,
> >>>>> Shwetha
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> On 23/09/15 8:30 pm, "Herman Yu" <herman...@teeupdata.com> wrote:
> >>>>>
> >>>>>> Hi Suma,
> >>>>>> Thanks for the help. I tried as you suggested, ³hive_table is
> >>>>>> collection², however "0 results matching your search query
> >>>>>>hive_table
> >>>>>> is
> >>>>>> collection were found².
> >>>>>> ³collection² is a trait I defined, if I just search with
> >>>>>>³collection²
> >>>>>> in
> >>>>>> the UI, I have two hive_table entities returned.
> >>>>>> I also tried with those types and traits defined with the
> >>>>>> quick_start.py,
> >>>>>> (e.g. "Table is Fact" didn¹t return anything.
> >>>>>> Is this a bug with v0.5? or I used wrong search syntax? Is there a
> >>>>>> document with all valid search syntax?
> >>>>>>
> >>>>>> thanks
> >>>>>> Herman.
> >>>>>>
> >>>>>>
> >>>>>> On Sep 21, 2015, at 10:54 PM, Herman Yu <herman...@teeupdata.com>
> >>>>>> wrote:
> >>>>>>
> >>>>>> Hi Herman,
> >>>>>>
> >>>>>> You can fire a DSL query in the UI as "hive_table  is "trait_name"
> >>>>>>
> >>>>>> Thanks
> >>>>>> Suma
> >>>>>>> Hi,
> >>>>>>>
> >>>>>>> With version v0.5, is search based on combination of a Trait and a
> >>>>>>> type
> >>>>>>> supported?  I have both hive_tables and hive_columns associated
> >>>>>>>with
> >>>>>>> a
> >>>>>>> Trait. I can search base on the Trait, or search based on the
> >>>>>>> type(hive_table or hive_colume). How to combine them together, for
> >>>>>>> example, search the trait but only want to return hive_table?
> >>>>>>>
> >>>>>>> I tried  and ,  and
> >>>>>>> typeName= in both UI and REST API, it doesn¹t work.
> >>>>>>>
> >>>>>>> This is my first time posting here, sorry if this is not the right
> >>>>>>> place asking questions like this. If so, what would be the correct
> >>>>>>> channel?
> >>>>>>>
> >>>>>>> Thanks
> >>>>>>> Herman.
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>
> >>
> >
>
>

Re: Search syntax

[Suma Shivaprasad]

 Could be an issue with 0.5. I fixed an issue while querying for entities
with traits as part of https://reviews.apache.org/r/38179/. . Should be
committed soon.

Thanks
Suma


On Wed, Sep 23, 2015 at 8:40 PM, Shwetha Shivalingamurthy <
sshivalingamur...@hortonworks.com> wrote:

> 'hive_table is ¹ should work. Here is the doc on DSL syntax:
> http://atlas.incubator.apache.org/Search.html
>
> Regards,
> Shwetha
>
>
>
>
>
>
> On 23/09/15 8:30 pm, "Herman Yu"  wrote:
>
> >Hi Suma,
> >Thanks for the help. I tried as you suggested, ³hive_table is
> >collection², however "0 results matching your search query hive_table is
> >collection were found².
> >³collection² is a trait I defined, if I just search with ³collection² in
> >the UI, I have two hive_table entities returned.
> >I also tried with those types and traits defined with the quick_start.py,
> >(e.g. "Table is Fact" didn¹t return anything.
> >Is this a bug with v0.5? or I used wrong search syntax? Is there a
> >document with all valid search syntax?
> >
> >thanks
> >Herman.
> >
> >
> >On Sep 21, 2015, at 10:54 PM, Herman Yu  wrote:
> >
> >Hi Herman,
> >
> >You can fire a DSL query in the UI as "hive_table  is "trait_name"
> >
> >Thanks
> >Suma
> >> Hi,
> >>
> >> With version v0.5, is search based on combination of a Trait and a type
> >>supported?  I have both hive_tables and hive_columns associated with a
> >>Trait. I can search base on the Trait, or search based on the
> >>type(hive_table or hive_colume). How to combine them together, for
> >>example, search the trait but only want to return hive_table?
> >>
> >> I tried  and ,  and
> >>typeName= in both UI and REST API, it doesn¹t work.
> >>
> >> This is my first time posting here, sorry if this is not the right
> >>place asking questions like this. If so, what would be the correct
> >>channel?
> >>
> >> Thanks
> >> Herman.
> >>
> >>
> >
> >
>
>

Re: Search syntax

[Suma Shivaprasad]

Hi Herman,

You can fire a DSL query in the UI as "hive_table  is "trait_name"

Thanks
Suma

On Tue, Sep 22, 2015 at 8:24 AM, Herman Yu  wrote:

> Hi,
>
> With version v0.5, is search based on combination of a Trait and a type
> supported?  I have both hive_tables and hive_columns associated with a
> Trait. I can search base on the Trait, or search based on the
> type(hive_table or hive_colume). How to combine them together, for example,
> search the trait but only want to return hive_table?
>
> I tried  and ,  and
> typeName= in both UI and REST API, it doesn’t work.
>
> This is my first time posting here, sorry if this is not the right place
> asking questions like this. If so, what would be the correct channel?
>
> Thanks
> Herman.
>
>
>

Re: Review Request 38393: ATLAS-58 Make hive hook reliable

[Suma Shivaprasad]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38393/#review99954
---

Ship it!


Ship It!

- Suma Shivaprasad


On Sept. 22, 2015, 9:09 a.m., Shwetha GS wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38393/
> ---
> 
> (Updated Sept. 22, 2015, 9:09 a.m.)
> 
> 
> Review request for atlas.
> 
> 
> Bugs: ATLAS-58
> https://issues.apache.org/jira/browse/ATLAS-58
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Hive hook sends notification messages (list of entities). The Notification 
> consumer on server side consumes these messages and registers the entities. 
> The server handles de-duping of entities based on the unique attribute of the 
> entity
> Big changes:
> 1. Concept of service that are started and stopped at atlas start and stop
> 2. De-duping of entities on server based on any unique attribute for the 
> entity. If entity doesn't have any unique attribute, de-duping is not done 
> and new entity is created
> 3. Changed entity submit API to take list of entities instead of just 1 
> entity (required for hive hook) - backward incompatible
> 4. Moved submit and list from EntityResource to EntitiesResource - backward 
> incompatible 
> 5. Moved security tests from integration tests to unit tests - as they were 
> creating issues with server start as jetty already starts another server for 
> integration tests
> 6. Removed some duplicate tests from repository module (the same tests exist 
> in typesystem module as well)
> 7. In webapp ITs, re-used the types defined
> 8. Hive hook now sends notifications instead of registering entities. Sending 
> notification is done synchronously. So, this adds to hive command execution 
> delay. But this also makes it reliable
> 
> Pending:
> 1. Entity updates like alter table commands are not handlded. Will create 
> another jira for this
> 2. Webapp jetty plugin doesn't shutdown embedded kafka at the end of 
> integration tests. So, hive bridge ITs fail. Hive bridge ITs pass if run on 
> their own. Still checking on this
> 
> 
> Diffs
> -
> 
>   addons/hive-bridge/pom.xml 914d8c6 
>   addons/hive-bridge/src/bin/import-hive.sh 99300e8 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
>  4d1af4e 
>   addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
> fe26446 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/model/HiveDataModelGenerator.java
>  cb8359e 
>   addons/hive-bridge/src/site/twiki/Bridge-Hive.twiki 647af27 
>   addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 
> 2bf3aba 
>   client/pom.xml 2e27930 
>   client/src/main/java/org/apache/atlas/ApplicationProperties.java 738ec53 
>   client/src/main/java/org/apache/atlas/AtlasClient.java 1c7d62a 
>   client/src/test/java/org/apache/atlas/ApplicationPropertiesTest.java 
> 4acb3e1 
>   client/src/test/resources/application.properties dbd6002 
>   common/pom.xml PRE-CREATION 
>   common/src/main/java/org/apache/atlas/service/Service.java PRE-CREATION 
>   common/src/main/java/org/apache/atlas/service/Services.java PRE-CREATION 
>   dashboard/public/modules/details/detailsResource.js af2f249 
>   dashboard/public/modules/search/searchController.js b5e6c37 
>   distro/pom.xml d42d78f 
>   distro/src/bin/atlas_start.py 7680f04 
>   distro/src/conf/application.properties bf323a7 
>   distro/src/main/assemblies/standalone-package.xml 625396a 
>   distro/src/test/python/scripts/TestMetadata.py 57f247f 
>   docs/pom.xml 4535fe0 
>   docs/src/site/resources/architecture.png 826df37 
>   docs/src/site/resources/data-types.png 3aa1904 
>   docs/src/site/resources/guide-class-diagram.png ca51239 
>   docs/src/site/resources/guide-instance-graph.png a2c8f82 
>   docs/src/site/resources/images/twiki/architecture.png PRE-CREATION 
>   docs/src/site/resources/images/twiki/data-types.png PRE-CREATION 
>   docs/src/site/resources/images/twiki/guide-class-diagram.png PRE-CREATION 
>   docs/src/site/resources/images/twiki/guide-instance-graph.png PRE-CREATION 
>   docs/src/site/resources/images/twiki/notification.png PRE-CREATION 
>   docs/src/site/resources/images/twiki/types-instance.png PRE-CREATION 
>   docs/src/site/resources/types-instance.png 6afca21 
>   docs/src/site/site.xml b412569 
>   docs/src/site/twiki/Architecture.twiki 5f10fde 
>   docs/src/site/twiki/Bridge-Hive

Re: [Atlas] A source code question

[Suma Shivaprasad]

attrname is same as i.name in this case. So shoudnt be an issue

Thanks
Suma

On Wed, Sep 16, 2015 at 1:20 PM, ltf  wrote:

> Hello,
> In the following code, for ENUM, the key for fieldPos is i.name, but
> others are attrName.
> Is it a issue??? Thanks.
>
>
>
>
> file : org/apache/atlas/typesystem/types/HierarchicalType.java
> protected Pair>
> constructFieldMapping(ImmutableList superTypes, AttributeInfo...
> fields) throws AtlasException { ... String attrName = i.name; if
> (fieldsMap.containsKey(attrName)) { attrName =
> currentPath.addOverrideAttr(attrName); } attributeNameToType.put(attrName,
> superType.getName()); fieldsMap.put(attrName, i);
> fieldNullPos.put(attrName, fieldNullPos.size()); if (i.dataType() ==
> DataTypes.BOOLEAN_TYPE) { fieldPos.put(attrName, numBools); numBools++; }
> else if (i.dataType() == DataTypes.BYTE_TYPE) { fieldPos.put(attrName,
> numBytes); numBytes++; } else if (i.dataType() == DataTypes.SHORT_TYPE) {
> fieldPos.put(attrName, numShorts); numShorts++; } else if (i.dataType() ==
> DataTypes.INT_TYPE) { fieldPos.put(attrName, numInts); numInts++; } else if
> (i.dataType() == DataTypes.LONG_TYPE) { fieldPos.put(attrName, numLongs);
> numLongs++; } else if (i.dataType() == DataTypes.FLOAT_TYPE) {
> fieldPos.put(attrName, numFloats); numFloats++; } else if (i.dataType() ==
> DataTypes.DOUBLE_TYPE) { fieldPos.put(attrName, numDoubles); numDoubles++;
> } else if (i.dataType() == DataTypes.BIGINTEGER_TYPE) {
> fieldPos.put(attrName, numBigInts); numBigInts++; } else if (i.dataType()
> == DataTypes.BIGDECIMAL_TYPE) { fieldPos.put(attrName, numBigDecimals);
> numBigDecimals++; } else if (i.dataType() == DataTypes.DATE_TYPE) {
> fieldPos.put(attrName, numDates); numDates++; } else if (i.dataType() ==
> DataTypes.STRING_TYPE) { fieldPos.put(attrName, numStrings); numStrings++;
> } else if (i.dataType().getTypeCategory() == DataTypes.TypeCategory.ENUM) {
> fieldPos.put(i.name, numInts); numInts++; } else if
> (i.dataType().getTypeCategory() == DataTypes.TypeCategory.ARRAY) {
> fieldPos.put(attrName, numArrays); numArrays++;
>
>
>
>
>
>
>
>
> Best, liutongfeng
>
>
>
>
>

Re: Review Request 38393: ATLAS-58 Make hive hook reliable

[Suma Shivaprasad]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38393/#review99747
---



repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepository.java
 (line 647)
<https://reviews.apache.org/r/38393/#comment156726>

we should rename this function to make the intention clear



repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java 
(line 197)
<https://reviews.apache.org/r/38393/#comment156738>

we should include the typedefinition as well in the exception?



repository/src/main/java/org/apache/atlas/services/MetadataService.java (line 
83)
<https://reviews.apache.org/r/38393/#comment156739>

pls add java doc for th args to make it clear


- Suma Shivaprasad


On Sept. 21, 2015, 8:13 a.m., Shwetha GS wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38393/
> ---
> 
> (Updated Sept. 21, 2015, 8:13 a.m.)
> 
> 
> Review request for atlas.
> 
> 
> Bugs: ATLAS-58
> https://issues.apache.org/jira/browse/ATLAS-58
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Hive hook sends notification messages (list of entities). The Notification 
> consumer on server side consumes these messages and registers the entities. 
> The server handles de-duping of entities based on the unique attribute of the 
> entity
> Big changes:
> 1. Concept of service that are started and stopped at atlas start and stop
> 2. De-duping of entities on server based on any unique attribute for the 
> entity. If entity doesn't have any unique attribute, de-duping is not done 
> and new entity is created
> 3. Changed entity submit API to take list of entities instead of just 1 
> entity (required for hive hook) - backward incompatible
> 4. Moved submit and list from EntityResource to EntitiesResource - backward 
> incompatible 
> 5. Moved security tests from integration tests to unit tests - as they were 
> creating issues with server start as jetty already starts another server for 
> integration tests
> 6. Removed some duplicate tests from repository module (the same tests exist 
> in typesystem module as well)
> 7. In webapp ITs, re-used the types defined
> 8. Hive hook now sends notifications instead of registering entities. Sending 
> notification is done synchronously. So, this adds to hive command execution 
> delay. But this also makes it reliable
> 
> Pending:
> 1. Entity updates like alter table commands are not handlded. Will create 
> another jira for this
> 2. Webapp jetty plugin doesn't shutdown embedded kafka at the end of 
> integration tests. So, hive bridge ITs fail. Hive bridge ITs pass if run on 
> their own. Still checking on this
> 
> 
> Diffs
> -
> 
>   addons/hive-bridge/pom.xml 914d8c6 
>   addons/hive-bridge/src/bin/import-hive.sh 99300e8 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
>  4d1af4e 
>   addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
> fe26446 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/model/HiveDataModelGenerator.java
>  cb8359e 
>   addons/hive-bridge/src/site/twiki/Bridge-Hive.twiki 647af27 
>   addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 
> 2bf3aba 
>   client/pom.xml 2e27930 
>   client/src/main/java/org/apache/atlas/ApplicationProperties.java 738ec53 
>   client/src/main/java/org/apache/atlas/AtlasClient.java 1c7d62a 
>   client/src/test/java/org/apache/atlas/ApplicationPropertiesTest.java 
> 4acb3e1 
>   client/src/test/resources/application.properties dbd6002 
>   common/pom.xml PRE-CREATION 
>   common/src/main/java/org/apache/atlas/service/Service.java PRE-CREATION 
>   common/src/main/java/org/apache/atlas/service/Services.java PRE-CREATION 
>   dashboard/public/modules/details/detailsResource.js af2f249 
>   dashboard/public/modules/search/searchController.js b5e6c37 
>   distro/pom.xml d42d78f 
>   distro/src/bin/atlas_start.py 7680f04 
>   distro/src/conf/application.properties bf323a7 
>   distro/src/main/assemblies/standalone-package.xml 625396a 
>   docs/pom.xml 4535fe0 
>   docs/src/site/resources/architecture.png 826df37 
>   docs/src/site/resources/data-types.png 3aa1904 
>   docs/src/site/resources/guide-class-diagram.png ca51239 
>   docs/src/site/resources/guide-instance-graph.png a2c8f82 
>   docs/src/site/resources/images/twiki/architecture.png PRE-CREATION 
>   docs/src/site/

Re: Review Request 38393: ATLAS-58 Make hive hook reliable

[Suma Shivaprasad]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38393/#review99753
---



docs/src/site/twiki/TypeSystem.twiki (line 8)
<https://reviews.apache.org/r/38393/#comment156745>

pls add docs for modelling unique entities by qualifiedName



typesystem/src/main/resources/application.properties (line 59)
<https://reviews.apache.org/r/38393/#comment156744>

pls add comments for the configs


- Suma Shivaprasad


On Sept. 21, 2015, 8:13 a.m., Shwetha GS wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38393/
> ---
> 
> (Updated Sept. 21, 2015, 8:13 a.m.)
> 
> 
> Review request for atlas.
> 
> 
> Bugs: ATLAS-58
> https://issues.apache.org/jira/browse/ATLAS-58
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Hive hook sends notification messages (list of entities). The Notification 
> consumer on server side consumes these messages and registers the entities. 
> The server handles de-duping of entities based on the unique attribute of the 
> entity
> Big changes:
> 1. Concept of service that are started and stopped at atlas start and stop
> 2. De-duping of entities on server based on any unique attribute for the 
> entity. If entity doesn't have any unique attribute, de-duping is not done 
> and new entity is created
> 3. Changed entity submit API to take list of entities instead of just 1 
> entity (required for hive hook) - backward incompatible
> 4. Moved submit and list from EntityResource to EntitiesResource - backward 
> incompatible 
> 5. Moved security tests from integration tests to unit tests - as they were 
> creating issues with server start as jetty already starts another server for 
> integration tests
> 6. Removed some duplicate tests from repository module (the same tests exist 
> in typesystem module as well)
> 7. In webapp ITs, re-used the types defined
> 8. Hive hook now sends notifications instead of registering entities. Sending 
> notification is done synchronously. So, this adds to hive command execution 
> delay. But this also makes it reliable
> 
> Pending:
> 1. Entity updates like alter table commands are not handlded. Will create 
> another jira for this
> 2. Webapp jetty plugin doesn't shutdown embedded kafka at the end of 
> integration tests. So, hive bridge ITs fail. Hive bridge ITs pass if run on 
> their own. Still checking on this
> 
> 
> Diffs
> -
> 
>   addons/hive-bridge/pom.xml 914d8c6 
>   addons/hive-bridge/src/bin/import-hive.sh 99300e8 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
>  4d1af4e 
>   addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
> fe26446 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/model/HiveDataModelGenerator.java
>  cb8359e 
>   addons/hive-bridge/src/site/twiki/Bridge-Hive.twiki 647af27 
>   addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 
> 2bf3aba 
>   client/pom.xml 2e27930 
>   client/src/main/java/org/apache/atlas/ApplicationProperties.java 738ec53 
>   client/src/main/java/org/apache/atlas/AtlasClient.java 1c7d62a 
>   client/src/test/java/org/apache/atlas/ApplicationPropertiesTest.java 
> 4acb3e1 
>   client/src/test/resources/application.properties dbd6002 
>   common/pom.xml PRE-CREATION 
>   common/src/main/java/org/apache/atlas/service/Service.java PRE-CREATION 
>   common/src/main/java/org/apache/atlas/service/Services.java PRE-CREATION 
>   dashboard/public/modules/details/detailsResource.js af2f249 
>   dashboard/public/modules/search/searchController.js b5e6c37 
>   distro/pom.xml d42d78f 
>   distro/src/bin/atlas_start.py 7680f04 
>   distro/src/conf/application.properties bf323a7 
>   distro/src/main/assemblies/standalone-package.xml 625396a 
>   docs/pom.xml 4535fe0 
>   docs/src/site/resources/architecture.png 826df37 
>   docs/src/site/resources/data-types.png 3aa1904 
>   docs/src/site/resources/guide-class-diagram.png ca51239 
>   docs/src/site/resources/guide-instance-graph.png a2c8f82 
>   docs/src/site/resources/images/twiki/architecture.png PRE-CREATION 
>   docs/src/site/resources/images/twiki/data-types.png PRE-CREATION 
>   docs/src/site/resources/images/twiki/guide-class-diagram.png PRE-CREATION 
>   docs/src/site/resources/images/twiki/guide-instance-graph.png PRE-CREATION 
>   docs/src/site/resources/images/twiki/notification.png PRE-CREATION 
>

Re: Review Request 38393: ATLAS-58 Make hive hook reliable

[Suma Shivaprasad]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38393/#review99752
---



typesystem/src/main/java/org/apache/atlas/ApplicationProperties.java (line 32)
<https://reviews.apache.org/r/38393/#comment156740>

have we moved this from client to ypesystem?



typesystem/src/main/java/org/apache/atlas/typesystem/persistence/StructInstance.java
 (line 121)
<https://reviews.apache.org/r/38393/#comment156741>

if possible, can you pls fix the error message in other places where 
ValueCnversionException is being thrown curently :) Right now, we dont ge to 
know which attribute is causing the issue


- Suma Shivaprasad


On Sept. 21, 2015, 8:13 a.m., Shwetha GS wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38393/
> ---
> 
> (Updated Sept. 21, 2015, 8:13 a.m.)
> 
> 
> Review request for atlas.
> 
> 
> Bugs: ATLAS-58
> https://issues.apache.org/jira/browse/ATLAS-58
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Hive hook sends notification messages (list of entities). The Notification 
> consumer on server side consumes these messages and registers the entities. 
> The server handles de-duping of entities based on the unique attribute of the 
> entity
> Big changes:
> 1. Concept of service that are started and stopped at atlas start and stop
> 2. De-duping of entities on server based on any unique attribute for the 
> entity. If entity doesn't have any unique attribute, de-duping is not done 
> and new entity is created
> 3. Changed entity submit API to take list of entities instead of just 1 
> entity (required for hive hook) - backward incompatible
> 4. Moved submit and list from EntityResource to EntitiesResource - backward 
> incompatible 
> 5. Moved security tests from integration tests to unit tests - as they were 
> creating issues with server start as jetty already starts another server for 
> integration tests
> 6. Removed some duplicate tests from repository module (the same tests exist 
> in typesystem module as well)
> 7. In webapp ITs, re-used the types defined
> 8. Hive hook now sends notifications instead of registering entities. Sending 
> notification is done synchronously. So, this adds to hive command execution 
> delay. But this also makes it reliable
> 
> Pending:
> 1. Entity updates like alter table commands are not handlded. Will create 
> another jira for this
> 2. Webapp jetty plugin doesn't shutdown embedded kafka at the end of 
> integration tests. So, hive bridge ITs fail. Hive bridge ITs pass if run on 
> their own. Still checking on this
> 
> 
> Diffs
> -
> 
>   addons/hive-bridge/pom.xml 914d8c6 
>   addons/hive-bridge/src/bin/import-hive.sh 99300e8 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
>  4d1af4e 
>   addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
> fe26446 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/model/HiveDataModelGenerator.java
>  cb8359e 
>   addons/hive-bridge/src/site/twiki/Bridge-Hive.twiki 647af27 
>   addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 
> 2bf3aba 
>   client/pom.xml 2e27930 
>   client/src/main/java/org/apache/atlas/ApplicationProperties.java 738ec53 
>   client/src/main/java/org/apache/atlas/AtlasClient.java 1c7d62a 
>   client/src/test/java/org/apache/atlas/ApplicationPropertiesTest.java 
> 4acb3e1 
>   client/src/test/resources/application.properties dbd6002 
>   common/pom.xml PRE-CREATION 
>   common/src/main/java/org/apache/atlas/service/Service.java PRE-CREATION 
>   common/src/main/java/org/apache/atlas/service/Services.java PRE-CREATION 
>   dashboard/public/modules/details/detailsResource.js af2f249 
>   dashboard/public/modules/search/searchController.js b5e6c37 
>   distro/pom.xml d42d78f 
>   distro/src/bin/atlas_start.py 7680f04 
>   distro/src/conf/application.properties bf323a7 
>   distro/src/main/assemblies/standalone-package.xml 625396a 
>   docs/pom.xml 4535fe0 
>   docs/src/site/resources/architecture.png 826df37 
>   docs/src/site/resources/data-types.png 3aa1904 
>   docs/src/site/resources/guide-class-diagram.png ca51239 
>   docs/src/site/resources/guide-instance-graph.png a2c8f82 
>   docs/src/site/resources/images/twiki/architecture.png PRE-CREATION 
>   docs/src/site/resources/images/twiki/data-types.png PRE-CREATION 
>   docs/src/site/resources/images/twiki/guide-cla

Re: Review Request 38393: ATLAS-58 Make hive hook reliable

[Suma Shivaprasad]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38393/#review99486
---



addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
 (line 277)
<https://reviews.apache.org/r/38393/#comment156363>

is sdQualifiedName supposed to be same as tableQualifiedName ?


- Suma Shivaprasad


On Sept. 15, 2015, 6:56 a.m., Shwetha GS wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38393/
> ---
> 
> (Updated Sept. 15, 2015, 6:56 a.m.)
> 
> 
> Review request for atlas.
> 
> 
> Bugs: ATLAS-58
> https://issues.apache.org/jira/browse/ATLAS-58
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Hive hook sends notification messages (list of entities). The Notification 
> consumer on server side consumes these messages and registers the entities. 
> The server handles de-duping of entities based on the unique attribute of the 
> entity
> Big changes:
> 1. Concept of service that are started and stopped at atlas start and stop
> 2. De-duping of entities on server based on any unique attribute for the 
> entity. If entity doesn't have any unique attribute, de-duping is not done 
> and new entity is created
> 3. Changed entity submit API to take list of entities instead of just 1 
> entity (required for hive hook) - backward incompatible
> 4. Moved submit and list from EntityResource to EntitiesResource - backward 
> incompatible 
> 5. Moved security tests from integration tests to unit tests - as they were 
> creating issues with server start as jetty already starts another server for 
> integration tests
> 6. Removed some duplicate tests from repository module (the same tests exist 
> in typesystem module as well)
> 7. In webapp ITs, re-used the types defined
> 8. Hive hook now sends notifications instead of registering entities. Sending 
> notification is done synchronously. So, this adds to hive command execution 
> delay. But this also makes it reliable
> 
> Pending:
> 1. Entity updates like alter table commands are not handlded. Will create 
> another jira for this
> 2. Webapp jetty plugin doesn't shutdown embedded kafka at the end of 
> integration tests. So, hive bridge ITs fail. Hive bridge ITs pass if run on 
> their own. Still checking on this
> 
> 
> Diffs
> -
> 
>   addons/hive-bridge/pom.xml 914d8c6 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
>  4d1af4e 
>   addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
> fe26446 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/model/HiveDataModelGenerator.java
>  cb8359e 
>   addons/hive-bridge/src/site/twiki/Bridge-Hive.twiki 647af27 
>   addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 
> 2bf3aba 
>   client/pom.xml 2e27930 
>   client/src/main/java/org/apache/atlas/ApplicationProperties.java 738ec53 
>   client/src/main/java/org/apache/atlas/AtlasClient.java 1c7d62a 
>   client/src/test/java/org/apache/atlas/ApplicationPropertiesTest.java 
> 4acb3e1 
>   client/src/test/resources/application.properties dbd6002 
>   common/pom.xml PRE-CREATION 
>   common/src/main/java/org/apache/atlas/service/Service.java PRE-CREATION 
>   common/src/main/java/org/apache/atlas/service/Services.java PRE-CREATION 
>   dashboard/public/modules/details/detailsResource.js af2f249 
>   distro/pom.xml d42d78f 
>   distro/src/conf/application.properties bf323a7 
>   docs/src/site/twiki/InstallationSteps.twiki 3d0351c 
>   notification/pom.xml b036855 
>   notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 
> 9978275 
>   
> notification/src/main/java/org/apache/atlas/kafka/KafkaNotificationProvider.java
>  PRE-CREATION 
>   
> notification/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java
>  36a62f0 
>   
> notification/src/main/java/org/apache/atlas/notification/NotificationInterface.java
>  0951124 
>   
> notification/src/main/java/org/apache/atlas/notification/NotificationModule.java
>  db17e35 
>   
> notification/src/test/java/org/apache/atlas/kafka/KafkaNotificationTest.java 
> 02752dc 
>   pom.xml e8d18a2 
>   repository/src/main/java/org/apache/atlas/GraphTransactionInterceptor.java 
> 3d47293 
>   repository/src/main/java/org/apache/atlas/RepositoryMetadataModule.java 
> fbd01de 
>   
> repository/src/main/java/org/a

Re: Review Request 38393: ATLAS-58 Make hive hook reliable

[Suma Shivaprasad]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38393/#review99487
---



addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java (line 
267)
<https://reviews.apache.org/r/38393/#comment156364>

Can we make this info ? This migght be the case of delays in synchronous 
calls and would be good to have visibility.



client/src/main/java/org/apache/atlas/AtlasClient.java (line 109)
<https://reviews.apache.org/r/38393/#comment156374>

we should make rge readTimeout configurable and also add connectTimeout



common/src/main/java/org/apache/atlas/service/Services.java (line 38)
<https://reviews.apache.org/r/38393/#comment156375>

should we add a register and unregister service method here?


- Suma Shivaprasad


On Sept. 15, 2015, 6:56 a.m., Shwetha GS wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38393/
> ---
> 
> (Updated Sept. 15, 2015, 6:56 a.m.)
> 
> 
> Review request for atlas.
> 
> 
> Bugs: ATLAS-58
> https://issues.apache.org/jira/browse/ATLAS-58
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Hive hook sends notification messages (list of entities). The Notification 
> consumer on server side consumes these messages and registers the entities. 
> The server handles de-duping of entities based on the unique attribute of the 
> entity
> Big changes:
> 1. Concept of service that are started and stopped at atlas start and stop
> 2. De-duping of entities on server based on any unique attribute for the 
> entity. If entity doesn't have any unique attribute, de-duping is not done 
> and new entity is created
> 3. Changed entity submit API to take list of entities instead of just 1 
> entity (required for hive hook) - backward incompatible
> 4. Moved submit and list from EntityResource to EntitiesResource - backward 
> incompatible 
> 5. Moved security tests from integration tests to unit tests - as they were 
> creating issues with server start as jetty already starts another server for 
> integration tests
> 6. Removed some duplicate tests from repository module (the same tests exist 
> in typesystem module as well)
> 7. In webapp ITs, re-used the types defined
> 8. Hive hook now sends notifications instead of registering entities. Sending 
> notification is done synchronously. So, this adds to hive command execution 
> delay. But this also makes it reliable
> 
> Pending:
> 1. Entity updates like alter table commands are not handlded. Will create 
> another jira for this
> 2. Webapp jetty plugin doesn't shutdown embedded kafka at the end of 
> integration tests. So, hive bridge ITs fail. Hive bridge ITs pass if run on 
> their own. Still checking on this
> 
> 
> Diffs
> -
> 
>   addons/hive-bridge/pom.xml 914d8c6 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
>  4d1af4e 
>   addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
> fe26446 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/model/HiveDataModelGenerator.java
>  cb8359e 
>   addons/hive-bridge/src/site/twiki/Bridge-Hive.twiki 647af27 
>   addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 
> 2bf3aba 
>   client/pom.xml 2e27930 
>   client/src/main/java/org/apache/atlas/ApplicationProperties.java 738ec53 
>   client/src/main/java/org/apache/atlas/AtlasClient.java 1c7d62a 
>   client/src/test/java/org/apache/atlas/ApplicationPropertiesTest.java 
> 4acb3e1 
>   client/src/test/resources/application.properties dbd6002 
>   common/pom.xml PRE-CREATION 
>   common/src/main/java/org/apache/atlas/service/Service.java PRE-CREATION 
>   common/src/main/java/org/apache/atlas/service/Services.java PRE-CREATION 
>   dashboard/public/modules/details/detailsResource.js af2f249 
>   distro/pom.xml d42d78f 
>   distro/src/conf/application.properties bf323a7 
>   docs/src/site/twiki/InstallationSteps.twiki 3d0351c 
>   notification/pom.xml b036855 
>   notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 
> 9978275 
>   
> notification/src/main/java/org/apache/atlas/kafka/KafkaNotificationProvider.java
>  PRE-CREATION 
>   
> notification/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java
>  36a62f0 
>   
> notification/src/main/java/org/apache/atlas/notification/NotificationInterface.java
>  0951124 
>   
> notification/src/main/java/org

Re: Review Request 38393: ATLAS-58 Make hive hook reliable

[Suma Shivaprasad]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38393/#review99499
---



notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java (line 
131)
<https://reviews.apache.org/r/38393/#comment156387>

should we throw and exceptiopn instead?


- Suma Shivaprasad


On Sept. 15, 2015, 6:56 a.m., Shwetha GS wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38393/
> ---
> 
> (Updated Sept. 15, 2015, 6:56 a.m.)
> 
> 
> Review request for atlas.
> 
> 
> Bugs: ATLAS-58
> https://issues.apache.org/jira/browse/ATLAS-58
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Hive hook sends notification messages (list of entities). The Notification 
> consumer on server side consumes these messages and registers the entities. 
> The server handles de-duping of entities based on the unique attribute of the 
> entity
> Big changes:
> 1. Concept of service that are started and stopped at atlas start and stop
> 2. De-duping of entities on server based on any unique attribute for the 
> entity. If entity doesn't have any unique attribute, de-duping is not done 
> and new entity is created
> 3. Changed entity submit API to take list of entities instead of just 1 
> entity (required for hive hook) - backward incompatible
> 4. Moved submit and list from EntityResource to EntitiesResource - backward 
> incompatible 
> 5. Moved security tests from integration tests to unit tests - as they were 
> creating issues with server start as jetty already starts another server for 
> integration tests
> 6. Removed some duplicate tests from repository module (the same tests exist 
> in typesystem module as well)
> 7. In webapp ITs, re-used the types defined
> 8. Hive hook now sends notifications instead of registering entities. Sending 
> notification is done synchronously. So, this adds to hive command execution 
> delay. But this also makes it reliable
> 
> Pending:
> 1. Entity updates like alter table commands are not handlded. Will create 
> another jira for this
> 2. Webapp jetty plugin doesn't shutdown embedded kafka at the end of 
> integration tests. So, hive bridge ITs fail. Hive bridge ITs pass if run on 
> their own. Still checking on this
> 
> 
> Diffs
> -
> 
>   addons/hive-bridge/pom.xml 914d8c6 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
>  4d1af4e 
>   addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
> fe26446 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/model/HiveDataModelGenerator.java
>  cb8359e 
>   addons/hive-bridge/src/site/twiki/Bridge-Hive.twiki 647af27 
>   addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 
> 2bf3aba 
>   client/pom.xml 2e27930 
>   client/src/main/java/org/apache/atlas/ApplicationProperties.java 738ec53 
>   client/src/main/java/org/apache/atlas/AtlasClient.java 1c7d62a 
>   client/src/test/java/org/apache/atlas/ApplicationPropertiesTest.java 
> 4acb3e1 
>   client/src/test/resources/application.properties dbd6002 
>   common/pom.xml PRE-CREATION 
>   common/src/main/java/org/apache/atlas/service/Service.java PRE-CREATION 
>   common/src/main/java/org/apache/atlas/service/Services.java PRE-CREATION 
>   dashboard/public/modules/details/detailsResource.js af2f249 
>   distro/pom.xml d42d78f 
>   distro/src/conf/application.properties bf323a7 
>   docs/src/site/twiki/InstallationSteps.twiki 3d0351c 
>   notification/pom.xml b036855 
>   notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 
> 9978275 
>   
> notification/src/main/java/org/apache/atlas/kafka/KafkaNotificationProvider.java
>  PRE-CREATION 
>   
> notification/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java
>  36a62f0 
>   
> notification/src/main/java/org/apache/atlas/notification/NotificationInterface.java
>  0951124 
>   
> notification/src/main/java/org/apache/atlas/notification/NotificationModule.java
>  db17e35 
>   
> notification/src/test/java/org/apache/atlas/kafka/KafkaNotificationTest.java 
> 02752dc 
>   pom.xml e8d18a2 
>   repository/src/main/java/org/apache/atlas/GraphTransactionInterceptor.java 
> 3d47293 
>   repository/src/main/java/org/apache/atlas/RepositoryMetadataModule.java 
> fbd01de 
>   
> repository/src/main/java/org/apache/atlas/listener/EntityChangeListener.java

Re: Review Request 38179: ATLAS-128 - Equality matches on lists

[Suma Shivaprasad]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38179/
---

(Updated Sept. 23, 2015, 4:40 p.m.)


Review request for atlas and Harish Butani.


Bugs: ATLAS-128
https://issues.apache.org/jira/browse/ATLAS-128


Repository: atlas


Description (updated)
---

1. Supporting equality matches on array type as array_type_col_name = [ List of 
values ] where list of values is a comma separated list of literals which are 
of primitive type like string, int, etc

2. Changed DSL tests - Gremlin, Expression, GraphBackedDiscoveryServiceTest to 
use the DefaultGraphPersistenceStrategy instead of the inmemory one so that the 
real code execution path is tested
3. Fixed ATLAS-30 - Fixed tests in GraphBackedDiscoveryServiceTest to validate 
expected test results
4. Fixed "typename has fieldname" queries where fieldname was not getting fully 
qualified with typeName.
5. 5. Fixed "type is trait" queries - trait label did not have __

Could not complete load of list types due to issues with validation in DSL. 
Will be covering this in a separate jira.


Diffs
-

  addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 
589ef16 
  
repository/src/main/java/org/apache/atlas/discovery/graph/DefaultGraphPersistenceStrategy.java
 c446d15 
  repository/src/main/java/org/apache/atlas/repository/MetadataRepository.java 
97d1fe2 
  
repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepository.java
 b44ce60 
  
repository/src/main/java/org/apache/atlas/repository/graph/TitanGraphProvider.java
 6605ae7 
  repository/src/main/scala/org/apache/atlas/query/Expressions.scala 24fa917 
  
repository/src/main/scala/org/apache/atlas/query/GraphPersistenceStrategies.scala
 2cec503 
  repository/src/main/scala/org/apache/atlas/query/GremlinQuery.scala 75fa916 
  repository/src/main/scala/org/apache/atlas/query/QueryParser.scala 5b129fb 
  repository/src/main/scala/org/apache/atlas/query/Resolver.scala 142ba8d 
  repository/src/test/java/org/apache/atlas/BaseHiveRepositoryTest.java 
PRE-CREATION 
  
repository/src/test/java/org/apache/atlas/discovery/GraphBackedDiscoveryServiceTest.java
 c72ffb2 
  
repository/src/test/java/org/apache/atlas/discovery/HiveLineageServiceTest.java 
db51ae5 
  
repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepositoryTest.java
 bec3067 
  
repository/src/test/java/org/apache/atlas/repository/graph/GraphRepoMapperScaleTest.java
 20781db 
  repository/src/test/scala/org/apache/atlas/query/GremlinTest.scala 957a566 
  repository/src/test/scala/org/apache/atlas/query/GremlinTest2.scala ea0b9bb 
  repository/src/test/scala/org/apache/atlas/query/HiveTitanSample.scala 
c662847 
  repository/src/test/scala/org/apache/atlas/query/LineageQueryTest.scala 
0b4dd2b 
  repository/src/test/scala/org/apache/atlas/query/ParserTest.scala 602b2b5 
  repository/src/test/scala/org/apache/atlas/query/QueryTestsUtils.scala 
3c66da0 
  typesystem/src/main/java/org/apache/atlas/typesystem/types/DataTypes.java 
e35453e 

Diff: https://reviews.apache.org/r/38179/diff/


Testing
---

Tests added in Parser, GremlinTest,  GraphBackedDiscoveryTest


Thanks,

Suma Shivaprasad

[VOTE] Release Apache Atlas version 0.6-incubating

[Suma Shivaprasad]

Hi Folks,

I have created a build for Apache Atlas 0.6-incubating, release candidate
0. Thanks to everyone who have contributed to this release and also tested
this.

The source tarball (*.tar.gz), signature (*.asc), checksum (*.md5, *.sha):

>
https://dist.apache.org/repos/dist/dev/incubator/atlas/0.6.0-incubating-rc0

The SHA512 checksum of the archive is
1D86B6B3 C89DAC70 31ACAB27 3D77F0AD 078A6080 694F9B66 668029A1 8289ECCC
48FE013F
 84069800 768D9C11 2B4119DA 85BB210D A02188D1 A09BE223 399A92DF

The commit id (adb6c76db6c262fd538cb81be2e0538ab24ae4ce) to be voted upon:

>
https://git-wip-us.apache.org/repos/asf?p=incubator-atlas.git;a=commit;h=adb6c76db6c262fd538cb81be2e0538ab24ae4ce

The tag to be voted upon:

>
https://git-wip-us.apache.org/repos/asf?p=incubator-atlas.git;a=tag;h=refs/tags/release-0.6-rc0

The list of fixed issues:

>
https://git-wip-us.apache.org/repos/asf?p=incubator-atlas.git;a=blob;f=release-log.txt;h=1dae6fb46376cbd0369784d9d4e2a29c99dec025;hb=51656991fc008e57936bfc12afa51b9e372ae6a5

Keys to verify the signature of the release artifact are available at:
http://www.apache.org/dist/incubator/atlas/KEYS
PGP release keys:
http://pgp.mit.edu/pks/lookup?op=vindex=0x2DBD4D9B

Note that this is a source only release and we are voting on the source
(release-0.6-incubating-rc0 / Commit:
adb6c76db6c262fd538cb81be2e0538ab24ae4ce / SHA512:
1D86B6B3 C89DAC70 31ACAB27 3D77F0AD 078A6080 694F9B66 668029A1 8289ECCC
48FE013F
 84069800 768D9C11 2B4119DA 85BB210D A02188D1 A09BE223 399A92DF).

Please download, test, and try it out.

Vote will be open for at least 72 hours - (6:30 AM PST/ 8 PM IST/ 9:30 AM
IST)  (until the required number of IPMC votes are obtained).

[ ] +1 approve
[ ] +0 no opinion
[ ] -1 disapprove (and reason why)

+1 from my side for the release.

For folks not familiar with vetting a release, please refer to
http://incubator.apache.org/guides/releasemanagement.html#check-list

Regards,
Suma

[CANCEL][VOTE] Release Apache Atlas version 0.6-incubating Release candidate-0

[Suma Shivaprasad]

Canceling release candidate-0 of Apache Atlas 0.6-incubating  as there were
issues with the source tar ball.

Thanks
Suma

Re: [VOTE] Release Apache Atlas version 0.6-incubating

[Suma Shivaprasad]

Thanks Bryan for pointing this out and taking t he time to validate the
release ! There was a bug with the source artifact generation which I have
fixed as part of https://issues.apache.org/jira/browse/ATLAS-380.

Will cancel this vote and raise a new VOTE thread.

Thanks
Suma

On Wed, Dec 9, 2015 at 6:51 AM, Bryan Bende <bbe...@gmail.com> wrote:

> Completely new to Atlas so could be way off here, but the source tarball
> [1] appears to only contain the atlas-distro module. Should it contain the
> entire source tree?
>
> [1]
>
> https://dist.apache.org/repos/dist/dev/incubator/atlas/0.6.0-incubating-rc0/apache-atlas-0.6-incubating-sources.tar.gz
>
>
> On Tue, Dec 8, 2015 at 9:26 AM, Suma Shivaprasad <suma...@apache.org>
> wrote:
>
> > Hi Folks,
> >
> > I have created a build for Apache Atlas 0.6-incubating, release candidate
> > 0. Thanks to everyone who have contributed to this release and also
> tested
> > this.
> >
> > The source tarball (*.tar.gz), signature (*.asc), checksum (*.md5,
> *.sha):
> >
> > >
> >
> https://dist.apache.org/repos/dist/dev/incubator/atlas/0.6.0-incubating-rc0
> >
> > The SHA512 checksum of the archive is
> > 1D86B6B3 C89DAC70 31ACAB27 3D77F0AD 078A6080 694F9B66 668029A1 8289ECCC
> > 48FE013F
> >  84069800 768D9C11 2B4119DA 85BB210D A02188D1 A09BE223 399A92DF
> >
> > The commit id (adb6c76db6c262fd538cb81be2e0538ab24ae4ce) to be voted
> upon:
> >
> > >
> >
> >
> https://git-wip-us.apache.org/repos/asf?p=incubator-atlas.git;a=commit;h=adb6c76db6c262fd538cb81be2e0538ab24ae4ce
> >
> > The tag to be voted upon:
> >
> > >
> >
> >
> https://git-wip-us.apache.org/repos/asf?p=incubator-atlas.git;a=tag;h=refs/tags/release-0.6-rc0
> >
> > The list of fixed issues:
> >
> > >
> >
> >
> https://git-wip-us.apache.org/repos/asf?p=incubator-atlas.git;a=blob;f=release-log.txt;h=1dae6fb46376cbd0369784d9d4e2a29c99dec025;hb=51656991fc008e57936bfc12afa51b9e372ae6a5
> >
> > Keys to verify the signature of the release artifact are available at:
> > http://www.apache.org/dist/incubator/atlas/KEYS
> > PGP release keys:
> > http://pgp.mit.edu/pks/lookup?op=vindex=0x2DBD4D9B
> >
> > Note that this is a source only release and we are voting on the source
> > (release-0.6-incubating-rc0 / Commit:
> > adb6c76db6c262fd538cb81be2e0538ab24ae4ce / SHA512:
> > 1D86B6B3 C89DAC70 31ACAB27 3D77F0AD 078A6080 694F9B66 668029A1 8289ECCC
> > 48FE013F
> >  84069800 768D9C11 2B4119DA 85BB210D A02188D1 A09BE223 399A92DF).
> >
> > Please download, test, and try it out.
> >
> > Vote will be open for at least 72 hours - (6:30 AM PST/ 8 PM IST/ 9:30 AM
> > IST)  (until the required number of IPMC votes are obtained).
> >
> > [ ] +1 approve
> > [ ] +0 no opinion
> > [ ] -1 disapprove (and reason why)
> >
> > +1 from my side for the release.
> >
> > For folks not familiar with vetting a release, please refer to
> > http://incubator.apache.org/guides/releasemanagement.html#check-list
> >
> > Regards,
> > Suma
> >
>

Re: [VOTE] Release Apache Atlas version 0.6.0-incubating (Release Candidate 1)

[Suma Shivaprasad]

Srikanth,

I tried to reproduce the jar dependency issue for "org.pentaho:pentaho-
aggdesigner-algorithm:jar:5.1.5-jhyde" by moving the maven repo and doing a
clean build but unable to reproduce it. It was able to download from the
"conjars" repo with a clean maven repo and no settings.xml and the build
went through successfully.

Downloading:
http://conjars.org/repo/org/pentaho/pentaho-aggdesigner-algorithm/5.1.5-jhyde/pentaho-aggdesigner-al.
..
Downloaded:
http://conjars.org/repo/org/pentaho/pentaho-aggdesigner-algorithm/5.1.5-jhyde/pentaho-aggdesigner-al...
(48 KB at 27.3 KB/sec)

[INFO]

[INFO] Reactor Summary:
[INFO]
[INFO] apache-atlas ... SUCCESS [07:28
min]
[INFO] Apache Atlas Common  SUCCESS [05:49
min]
[INFO] Apache Atlas Typesystem  SUCCESS [01:28
min]
[INFO] Apache Atlas Server API  SUCCESS [
 6.273 s]
[INFO] Apache Atlas Client  SUCCESS [02:16
min]
[INFO] Apache Atlas Notification .. SUCCESS [01:30
min]
[INFO] Apache Atlas Titan . SUCCESS [07:00
min]
[INFO] Apache Atlas Repository  SUCCESS [04:30
min]
[INFO] Apache Atlas UI  SUCCESS [02:31
min]
[INFO] Apache Atlas Web Application ... SUCCESS [04:32
min]
[INFO] Apache Atlas Documentation . SUCCESS [01:06
min]
[INFO] Apache Atlas Hive Bridge ... SUCCESS [04:08
min]
[INFO] Apache Atlas Distribution .. SUCCESS [
47.208 s]
[INFO]

[INFO] BUILD SUCCESS
[INFO]

[INFO] Total time: 45:21 min
[INFO] Finished at: 2015-12-10T14:03:20+05:30
[INFO] Final Memory: 173M/726M
[INFO]


Would request someone else to try the same and let me know. If its working
fine, request you to reconsider the vote.

Thanks
Suma









On Thu, Dec 10, 2015 at 1:58 PM, Suma Shivaprasad <
sumasai.shivapra...@gmail.com> wrote:

> Hi all,
>
> Thanks for your inputs and taking time to validate the release.
>
> Srikanth,
>
> Will fix the dependency issue that you mentioned as part of the POM and
> raise another vote/cancel this vote thread.
>
> Have raised an Infra ticket for the release candidate archive removal -
> https://issues.apache.org/jira/browse/INFRA-10932
>
> Thanks
> Suma
>
>
>
> On Thu, Dec 10, 2015 at 12:45 PM, Srikanth Sundarrajan <srik...@gmail.com>
> wrote:
>
>> Looks like org.pentaho:pentaho-aggdesigner-algorithm:jar:5.1.5-jhyde isn't
>> available on a maven repo that Atlas refers to. My vote would be "0" on
>> this RC. If am successfully able to build, will change my vote to +1.
>>
>> License +1
>> Signature +1
>> Checksums +1
>>
>> Build fails: -1
>>
>> [ERROR] Failed to execute goal on project hive-bridge: Could not resolve
>> dependencies for project org.apache.atlas:hive-bridge:jar:0.6-incubating:
>> Could not find artifact
>> org.pentaho:pentaho-aggdesigner-algorithm:jar:5.1.5-jhyde in central (
>> http://repo1.maven.org/maven2) -> [Help 1]
>> org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute
>> goal on project hive-bridge: Could not resolve dependencies for project
>> org.apache.atlas:hive-bridge:jar:0.6-incubating: Could not find artifact
>> org.pentaho:pentaho-aggdesigner-algorithm:jar:5.1.5-jhyde in central (
>> http://repo1.maven.org/maven2)
>>
>> I had to add the following repository to my maven settings to get it to
>> build (it can be added to pom as well)
>>
>> 
>> conjars
>> conjars
>> http://www.conjars.org/repo/
>> 
>> true
>> 
>> 
>> false
>> 
>> 
>>
>> This can be provided as an instruction in README or pom can be modified to
>> include this.
>>
>> Also few other observations:
>> REMOVE (until the required number of IPMC votes are obtained) from the
>> vote
>> text. - You need to keep the vote running at least for 72 hours regardless
>> of whether you have the minimum IPMC votes or not.
>>
>> https://dist.apache.org/repos/dist/release/incubator/atlas/ contains
>> 0.6-RC0. This nee

Re: [VOTE] Release Apache Atlas version 0.6.0-incubating (Release Candidate 1)

[Suma Shivaprasad]

Hi all,

https://issues.apache.org/jira/browse/ATLAS-383 has been identified as a
blocker for this release candidate. Hence will be canceling the vote for
this Release Candidate and will start a new vote thread once the fix is in
place and tested.

Thanks
Suma

On Thu, Dec 10, 2015 at 9:08 PM, Srikanth Sundarrajan <srik...@gmail.com>
wrote:

> Am able to build successfully on another machine, so will change my vote to
> +1 purely from a release sanity point of view.
>
> But would recommend that we look into the functional issue that Hemanth has
> pointed out, particularly if it is blocker and not a corner case.
>
> +1 (binding)
>
> Regards
> Srikanth Sundarrajan
>
> On Thu, Dec 10, 2015 at 7:27 PM, Hemanth Yamijala <yhema...@gmail.com>
> wrote:
>
> > Hi,
> >
> > I tried to validate the release. While I don't see the compile errors, I
> do
> > see hive-import failing for me. This is occurring on trunk as well.
> >
> > This could be an environment issue, but I suspect some recent changes
> could
> > be the cause as well. I filed
> > https://issues.apache.org/jira/browse/ATLAS-383 to investigate. Until we
> > identify I hope we can hold on to this RC.
> >
> > I am a -1 for now. Will revert the vote if it turns out to be my
> > environment issue.
> >
> > Thanks
> > hemanth
> >
> > On Thu, Dec 10, 2015 at 4:50 PM, Srikanth Sundarrajan <srik...@gmail.com
> >
> > wrote:
> >
> > > Looks like many of you are able to build without issues and may be the
> > > issue is local to my setup. Will try on another host and revert back.
> > >
> > > Regards
> > > Srikanth Sundarrajan
> > >
> > > On Thu, Dec 10, 2015 at 4:31 PM, Sandeep Samudrala <
> sandys...@gmail.com>
> > > wrote:
> > >
> > > > +1.(non-binding)
> > > > Verified build and signatures.
> > > >
> > > > On Thu, Dec 10, 2015 at 3:09 PM, Shwetha G S <shweth...@gmail.com>
> > > wrote:
> > > >
> > > > > It worked for me as well
> > > > >
> > > > > Regards,
> > > > > Shwetha
> > > > >
> > > > > > On Dec 10, 2015, at 2:46 PM, Suma Shivaprasad <
> > > > > sumasai.shivapra...@gmail.com> wrote:
> > > > > >
> > > > > > Srikanth,
> > > > > >
> > > > > > I tried to reproduce the jar dependency issue for
> > > "org.pentaho:pentaho-
> > > > > > aggdesigner-algorithm:jar:5.1.5-jhyde" by moving the maven repo
> and
> > > > > doing a
> > > > > > clean build but unable to reproduce it. It was able to download
> > from
> > > > the
> > > > > > "conjars" repo with a clean maven repo and no settings.xml and
> the
> > > > build
> > > > > > went through successfully.
> > > > > >
> > > > > > Downloading:
> > > > > >
> > > > >
> > > >
> > >
> >
> http://conjars.org/repo/org/pentaho/pentaho-aggdesigner-algorithm/5.1.5-jhyde/pentaho-aggdesigner-al
> > > > > .
> > > > > > ..
> > > > > > Downloaded:
> > > > > >
> > > > >
> > > >
> > >
> >
> http://conjars.org/repo/org/pentaho/pentaho-aggdesigner-algorithm/5.1.5-jhyde/pentaho-aggdesigner-al
> > > > .
> > > > > ..
> > > > > > (48 KB at 27.3 KB/sec)
> > > > > >
> > > > > > [INFO]
> > > > > >
> > > >
> > 
> > > > > > [INFO] Reactor Summary:
> > > > > > [INFO]
> > > > > > [INFO] apache-atlas ...
> SUCCESS
> > > > > [07:28
> > > > > > min]
> > > > > > [INFO] Apache Atlas Common 
> SUCCESS
> > > > > [05:49
> > > > > > min]
> > > > > > [INFO] Apache Atlas Typesystem 
> SUCCESS
> > > > > [01:28
> > > > > > min]
> > > > > > [INFO] Apache Atlas Server API 
> > SUCCESS [
> > > > > > 6.273 s]
> > > > > > [INFO] Apache Atlas Client 
> SUCCESS
> > > > > [02:16
> 

[ANNOUNCE] Apache Atlas 0.6-incubating released

[Suma Shivaprasad]

Hi folks,

The Apache Atlas team is happy to announce the release of Apache Atlas -
version 0.6-incubating.

Atlas is a scalable and extensible set of core foundational governance
services – enabling enterprises to effectively and efficiently meet their
compliance requirements within Hadoop and allows integration with the whole
enterprise data ecosystem.

The release artifacts are available at:
*http://www.apache.org/dyn/closer.cgi/incubator/atlas/0.6.0-incubating/
*

Release notes available at:
*https://git-wip-us.apache.org/repos/asf?p=incubator-atlas.git;a=blob;f=release-log.txt;h=c778ff0014b355b88ede8296a6c849a19f8202d3;hb=refs/heads/branch-0.6-incubating
*

To use these artifacts, please use the following documentation:
*http://atlas.incubator.apache.org/0.6.0-incubating/index.html
*

More details on Apache Atlas can be found at:
http://atlas.incubator.apache.org

We would like to thank the many contributors who contributed to this
release and made this release possible.

Thanks,
The Apache Atlas team


DISCLAIMER

Apache Atlas is an effort undergoing incubation at the Apache
Software Foundation (ASF), sponsored by the Apache Incubator PMC.

Incubation is required of all newly accepted projects until a further
review indicates that the infrastructure, communications, and decision
making process have stabilized in a manner consistent with other
successful ASF projects.

While incubation status is not necessarily a reflection of the
completeness or stability of the code, it does indicate that the
project has yet to be fully endorsed by the ASF.

Re: [VOTE] Release Apache Atlas version 0.6-incubating

[Suma Shivaprasad]

Hi folks,

Thanks for taking time to review especially during the holidays and vote on
this release !  The vote passes
with four binding +1's, 3 non-binding +1s .

Binding:
Arun C Murthy
Justin Mclean
Seetharam Venkatesh
John D Ament

Non-binding:
Harish Butani
Shwetha Shivalingamurthy
Suma Shivaprasad

I'll start working on the release artifacts and then announce the release
to this list.

Thanks!

Regards,
Suma

On Sun, Dec 27, 2015 at 11:14 PM, Harish Butani <rhbut...@apache.org> wrote:

> +1
>
> On Sat, Dec 26, 2015 at 1:03 PM, John D. Ament <john.d.am...@gmail.com>
> wrote:
>
> > +1 looks good to me.
> > On Dec 24, 2015 22:07, "Suma Shivaprasad" <suma...@apache.org> wrote:
> >
> > > Hello folks,
> > >
> > > This is a call for a vote on the Apache Atlas 0.6 incubating release.
> > >
> > > A vote was held on developer mailing list and it passed with five +1's.
> > >
> > > Vote thread: http://s.apache.org/nxl
> > > Results thread: http://s.apache.org/HyG
> > >
> > > The source tarball (*.tar.gz), signature (*.asc), checksum (*.md5,
> > *.sha):
> > >
> > > > *
> > >
> >
> https://dist.apache.org/repos/dist/dev/incubator/atlas/0.6.0-incubating-rc2/
> > > <
> > >
> >
> https://dist.apache.org/repos/dist/dev/incubator/atlas/0.6.0-incubating-rc2/
> > > >*
> > >
> > > The SHA512 checksum of the archive is
> > >
> > > CD7441A1 1790B2DA 52DA09D6 1ED4CF10 2B202D56 109DE7AE D86D1026 8EDE3A11
> > > CF9AD7BF 9FAEDF55 E5175EB2 137A01B6 A311ECA1 EEA6EBFE 3D6CFA7F 23BE6CC8
> > > The commit id (c8109a0177058d4a788d476a74d425a7593e7a9b) to be voted
> > upon:
> > >
> > > >
> > >
> > > *
> > >
> >
> https://git-wip-us.apache.org/repos/asf?p=incubator-atlas.git;a=commit;h=c8109a0177058d4a788d476a74d425a7593e7a9b
> > > <
> > >
> >
> https://git-wip-us.apache.org/repos/asf?p=incubator-atlas.git;a=commit;h=c8109a0177058d4a788d476a74d425a7593e7a9b
> > > >*
> > > The tag to be voted upon:
> > >
> > > > *
> > >
> >
> https://git-wip-us.apache.org/repos/asf?p=incubator-atlas.git;a=tag;h=refs/tags/release-0.6-rc2
> > > <
> > >
> >
> https://git-wip-us.apache.org/repos/asf?p=incubator-atlas.git;a=tag;h=refs/tags/release-0.6-rc2
> > > >*
> > >
> > > The list of fixed issues:
> > >
> > > > *
> > >
> >
> https://git-wip-us.apache.org/repos/asf?p=incubator-atlas.git;a=blob;f=release-log.txt;h=c778ff0014b355b88ede8296a6c849a19f8202d3;hb=refs/heads/branch-0.6-incubating
> > > <
> > >
> >
> https://git-wip-us.apache.org/repos/asf?p=incubator-atlas.git;a=blob;f=release-log.txt;h=c778ff0014b355b88ede8296a6c849a19f8202d3;hb=refs/heads/branch-0.6-incubating
> > > >*
> > >
> > > Keys to verify the signature of the release artifact are available at:
> > > http://www.apache.org/dist/incubator/atlas/KEYS
> > > PGP release keys:
> > > http://pgp.mit.edu/pks/lookup?op=vindex=0x2DBD4D9B
> > >
> > > Note that this is a source only release and we are voting on the source
> > > (release-0.6-incubating-rc2 /
> > >
> > > Commit:
> > > c8109a0177058d4a788d476a74d425a7593e7a9b /
> > >
> > > Checksums:
> > > SHA512:
> > > CD7441A1 1790B2DA 52DA09D6 1ED4CF10 2B202D56 109DE7AE D86D1026 8EDE3A11
> > > CF9AD7BF 9FAEDF55 E5175EB2 137A01B6 A311ECA1 EEA6EBFE 3D6CFA7F 23BE6CC8
> > > MD5 : 32 A1 13 D6 E8 7A 8E C9  34 1C D3 45 E8 4D 90 E1).
> > >
> > > Vote will be open for atleast 72 hours ( atleast till 30th Dec 7:30 PM
> > PST/
> > > 10:30 PM EST /31st Dec 9 AM IST).
> > >
> > > [ ] +1 approve
> > > [ ] +0 no opinion
> > > [ ] -1 disapprove (and reason why)
> > >
> > > Thanks!
> > >
> > > Regards,
> > > Suma
> > >
> >
>

Re: [VOTE] Release Apache Atlas version 0.6.0-incubating (Release Candidate 2)

[Suma Shivaprasad]

Hi folks,

Thanks for taking time to review and vote on this release.  The vote passes
with five binding +1's and one 0, no -1s.

Binding:
IPMC - Arun Murthy, Jitendra Pandey
PPMC - Harish Butani, Shwetha GS, Suma Shivaprasad

I'll start voting on Incubator general.

Thanks!

Regards,
Suma

On Tue, Dec 22, 2015 at 8:28 PM, Harish Butani <rhbut...@gmail.com> wrote:

> +1
>
> verified the signatures, did a build, ran the tests.
>
> On Mon, Dec 21, 2015 at 3:14 PM, Arun C. Murthy <acmur...@gmail.com>
> wrote:
>
> > +1 (binding)
> >
> > Arun
> >
> >
> >
> > > On Dec 21, 2015, at 2:55 AM, Suma Shivaprasad <suma...@apache.org>
> > wrote:
> > >
> > > Hi Folks,
> > >
> > > I have created a build for Apache Atlas 0.6-incubating, release
> > candidate 2.
> > > Thanks to everyone who have contributed to this release and also tested
> > > this.
> > >
> > > The source tarball (*.tar.gz), signature (*.asc), checksum (*.md5,
> > *.sha):
> > >
> > >> *
> >
> https://dist.apache.org/repos/dist/dev/incubator/atlas/0.6.0-incubating-rc2/
> > > <
> >
> https://dist.apache.org/repos/dist/dev/incubator/atlas/0.6.0-incubating-rc2/
> > >*
> > >
> > > The SHA512 checksum of the archive is
> > >
> > > CD7441A1 1790B2DA 52DA09D6 1ED4CF10 2B202D56 109DE7AE D86D1026 8EDE3A11
> > > CF9AD7BF 9FAEDF55 E5175EB2 137A01B6 A311ECA1 EEA6EBFE 3D6CFA7F 23BE6CC8
> > > The commit id (c8109a0177058d4a788d476a74d425a7593e7a9b) to be voted
> > upon:
> > >
> > >
> > > *
> >
> https://git-wip-us.apache.org/repos/asf?p=incubator-atlas.git;a=commit;h=c8109a0177058d4a788d476a74d425a7593e7a9b
> > > <
> >
> https://git-wip-us.apache.org/repos/asf?p=incubator-atlas.git;a=commit;h=c8109a0177058d4a788d476a74d425a7593e7a9b
> > >*
> > > The tag to be voted upon:
> > >
> > >> *
> >
> https://git-wip-us.apache.org/repos/asf?p=incubator-atlas.git;a=tag;h=refs/tags/release-0.6-rc2
> > > <
> >
> https://git-wip-us.apache.org/repos/asf?p=incubator-atlas.git;a=tag;h=refs/tags/release-0.6-rc2
> > >*
> > >
> > > The list of fixed issues:
> > >
> > >> *
> >
> https://git-wip-us.apache.org/repos/asf?p=incubator-atlas.git;a=blob;f=release-log.txt;h=c778ff0014b355b88ede8296a6c849a19f8202d3;hb=refs/heads/branch-0.6-incubating
> > > <
> >
> https://git-wip-us.apache.org/repos/asf?p=incubator-atlas.git;a=blob;f=release-log.txt;h=c778ff0014b355b88ede8296a6c849a19f8202d3;hb=refs/heads/branch-0.6-incubating
> > >*
> > >
> > > Keys to verify the signature of the release artifact are available at:
> > > http://www.apache.org/dist/incubator/atlas/KEYS
> > > PGP release keys:
> > > http://pgp.mit.edu/pks/lookup?op=vindex=0x2DBD4D9B
> > >
> > > Note that this is a source only release and we are voting on the source
> > > (release-0.6-incubating-rc2 / Commit:
> > > c8109a0177058d4a788d476a74d425a7593e7a9b / SHA512:
> > > CD7441A1 1790B2DA 52DA09D6 1ED4CF10 2B202D56 109DE7AE D86D1026 8EDE3A11
> > > CF9AD7BF 9FAEDF55 E5175EB2 137A01B6 A311ECA1 EEA6EBFE 3D6CFA7F
> 23BE6CC8).
> > >
> > > Please download, test, and try it out.
> > >
> > > Vote will be open for at least 72 hours - (Dec 24 - 9 AM PST /  12 PM
> > > EST/ 10:30
> > > PM IST)
> > >
> > > [ ] +1 approve
> > > [ ] +0 no opinion
> > > [ ] -1 disapprove (and reason why)
> > >
> > > +1 from my side for the release.
> > >
> > > For folks not familiar with vetting a release, please refer to
> > > http://incubator.apache.org/guides/releasemanagement.html#check-list
> > >
> > > Regards,
> > > Suma
> >
>

[VOTE] Release Apache Atlas version 0.6-incubating

[Suma Shivaprasad]

Hello folks,

This is a call for a vote on the Apache Atlas 0.6 incubating release.

A vote was held on developer mailing list and it passed with five +1's.

Vote thread: http://s.apache.org/nxl
Results thread: http://s.apache.org/HyG

The source tarball (*.tar.gz), signature (*.asc), checksum (*.md5, *.sha):

> *https://dist.apache.org/repos/dist/dev/incubator/atlas/0.6.0-incubating-rc2/
*

The SHA512 checksum of the archive is

CD7441A1 1790B2DA 52DA09D6 1ED4CF10 2B202D56 109DE7AE D86D1026 8EDE3A11
CF9AD7BF 9FAEDF55 E5175EB2 137A01B6 A311ECA1 EEA6EBFE 3D6CFA7F 23BE6CC8
The commit id (c8109a0177058d4a788d476a74d425a7593e7a9b) to be voted upon:

>

*https://git-wip-us.apache.org/repos/asf?p=incubator-atlas.git;a=commit;h=c8109a0177058d4a788d476a74d425a7593e7a9b
*
The tag to be voted upon:

> *https://git-wip-us.apache.org/repos/asf?p=incubator-atlas.git;a=tag;h=refs/tags/release-0.6-rc2
*

The list of fixed issues:

> *https://git-wip-us.apache.org/repos/asf?p=incubator-atlas.git;a=blob;f=release-log.txt;h=c778ff0014b355b88ede8296a6c849a19f8202d3;hb=refs/heads/branch-0.6-incubating
*

Keys to verify the signature of the release artifact are available at:
http://www.apache.org/dist/incubator/atlas/KEYS
PGP release keys:
http://pgp.mit.edu/pks/lookup?op=vindex=0x2DBD4D9B

Note that this is a source only release and we are voting on the source
(release-0.6-incubating-rc2 /

Commit:
c8109a0177058d4a788d476a74d425a7593e7a9b /

Checksums:
SHA512:
CD7441A1 1790B2DA 52DA09D6 1ED4CF10 2B202D56 109DE7AE D86D1026 8EDE3A11
CF9AD7BF 9FAEDF55 E5175EB2 137A01B6 A311ECA1 EEA6EBFE 3D6CFA7F 23BE6CC8
MD5 : 32 A1 13 D6 E8 7A 8E C9  34 1C D3 45 E8 4D 90 E1).

Vote will be open for atleast 72 hours ( atleast till 30th Dec 7:30 PM PST/
10:30 PM EST /31st Dec 9 AM IST).

[ ] +1 approve
[ ] +0 no opinion
[ ] -1 disapprove (and reason why)

Thanks!

Regards,
Suma

Re: Review Request 47658: ATLAS-499: Update reverse references

[Suma Shivaprasad]

> On June 3, 2016, 11:29 p.m., Suma Shivaprasad wrote:
> > repository/src/main/java/org/apache/atlas/repository/graph/TypedInstanceToGraphMapper.java,
> >  line 443
> > <https://reviews.apache.org/r/47658/diff/5/?file=1405688#file1405688line443>
> >
> > for map types, can we check if the reverseAttributeName is set on the 
> > map type and remove it if set else ignore?
> 
> David Kantor wrote:
> I'm thinking it may be better to remove the reverse attribute name when 
> the type is registered and possibly issue a warning that bi-directional 
> references are not supported for map types, rather then doing it every time 
> there is an update operation involving that type.  Thoughts?

yeah that seems like a better option


- Suma


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47658/#review136133
---


On June 3, 2016, 1:55 p.m., David Kantor wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47658/
> ---
> 
> (Updated June 3, 2016, 1:55 p.m.)
> 
> 
> Review request for atlas and Shwetha GS.
> 
> 
> Bugs: ATLAS-499
> https://issues.apache.org/jira/browse/ATLAS-499
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> ATLAS-499: Automatically update inverse references to prevent repository 
> corruption from unbalanced references.
> 
> 
> Diffs
> -
> 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/DeleteHandler.java 
> 91f9bd008a6939dfe78656f5c1845637a30ee9eb 
>   repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java 
> 4f6d0118072380342f5ea302d0aaf7902784849a 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/TypedInstanceToGraphMapper.java
>  4c1f5591f4adead41c3336e64b0bc956836f7edb 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepositoryDeleteTestBase.java
>  449e066a167ba4546b118a77c8e3de5fd99f077b 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/ReverseReferenceUpdateHardDeleteTest.java
>  PRE-CREATION 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/ReverseReferenceUpdateSoftDeleteTest.java
>  PRE-CREATION 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/ReverseReferenceUpdateTestBase.java
>  PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/47658/diff/
> 
> 
> Testing
> ---
> 
> Ran all unit and integration tests with no regressions.
> 
> 
> Thanks,
> 
> David Kantor
> 
>

Re: Review Request 47658: ATLAS-499: Update reverse references

[Suma Shivaprasad]

> On June 3, 2016, 11:25 p.m., Suma Shivaprasad wrote:
> > repository/src/main/java/org/apache/atlas/repository/graph/TypedInstanceToGraphMapper.java,
> >  line 785
> > <https://reviews.apache.org/r/47658/diff/5/?file=1405688#file1405688line785>
> >
> > do we need the HashSet since contains and all will both O(n)
> 
> David Kantor wrote:
> HashSet.contains is O(1).  I am using the HashSet to avoid the O(n) 
> performance of ArrayList.contains.  For a very small data set, the difference 
> between HashSet.contains and ArrayList.contains is probably negligible.  But 
> in my experience doing Java performance analysis on our legacy metadata 
> repository, ArrayList.contains() often ended up being a major hot spot when 
> enforcing uniqueness for high cardinality references.

I was just saying that adding all the list elemnts to the Hashset is itself 
O(n) so it might be better to skip creating the hashset. Leave it up to you not 
a major issue to fix.


- Suma


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47658/#review136132
---


On June 3, 2016, 1:55 p.m., David Kantor wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47658/
> ---
> 
> (Updated June 3, 2016, 1:55 p.m.)
> 
> 
> Review request for atlas and Shwetha GS.
> 
> 
> Bugs: ATLAS-499
> https://issues.apache.org/jira/browse/ATLAS-499
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> ATLAS-499: Automatically update inverse references to prevent repository 
> corruption from unbalanced references.
> 
> 
> Diffs
> -
> 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/DeleteHandler.java 
> 91f9bd008a6939dfe78656f5c1845637a30ee9eb 
>   repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java 
> 4f6d0118072380342f5ea302d0aaf7902784849a 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/TypedInstanceToGraphMapper.java
>  4c1f5591f4adead41c3336e64b0bc956836f7edb 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepositoryDeleteTestBase.java
>  449e066a167ba4546b118a77c8e3de5fd99f077b 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/ReverseReferenceUpdateHardDeleteTest.java
>  PRE-CREATION 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/ReverseReferenceUpdateSoftDeleteTest.java
>  PRE-CREATION 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/ReverseReferenceUpdateTestBase.java
>  PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/47658/diff/
> 
> 
> Testing
> ---
> 
> Ran all unit and integration tests with no regressions.
> 
> 
> Thanks,
> 
> David Kantor
> 
>

ATLAS-542 Backward incompatible changes

[Suma Shivaprasad]

Hi all,

As part of changes in ATLAS-542 to make attribute names consistent across
Datasets and Process inherited types, "name" will represent shorter name
and "qualifiedName" will represent the fully qualified unique name which
uniquely idenitifies the entity.   One of the attribute names that
primarily changes as part of this is for hive_table type in which "name"
now represents the shorter table name and "qualifiedName" represents fully
qualified table name which was earlier modelled as "tableName" and "name"
respectively.

Please get back to me if you have further questions.

Thanks
Suma

Re: Review Request 48280: ATLAS-515: Ability to initialize Kafka topics with more than 1 replica

[Suma Shivaprasad]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48280/#review136932
---


Ship it!




Pls add documentation on how to use the scripts. The changes looks good to me.

- Suma Shivaprasad


On June 9, 2016, 10:30 a.m., Hemanth Yamijala wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/48280/
> ---
> 
> (Updated June 9, 2016, 10:30 a.m.)
> 
> 
> Review request for atlas.
> 
> 
> Bugs: ATLAS-515
> https://issues.apache.org/jira/browse/ATLAS-515
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Attached patch provides a method to automatically create Kafka topics 
> ATLAS_HOOK and ATLAS_ENTITIES with configured number of replicas.
> 
> 
> Diffs
> -
> 
>   distro/src/bin/atlas_client_cmdline.py f109ad3 
>   distro/src/bin/atlas_config.py fab4046 
>   distro/src/bin/atlas_kafka_setup.py PRE-CREATION 
>   distro/src/bin/atlas_kafka_setup_hook.py PRE-CREATION 
>   distro/src/conf/atlas-application.properties 873c1e0 
>   distro/src/main/assemblies/standalone-package.xml 1c7b2c5 
>   notification/pom.xml b3738db 
>   notification/src/main/java/org/apache/atlas/hook/AtlasTopicCreator.java 
> PRE-CREATION 
>   notification/src/test/java/org/apache/atlas/hook/AtlasTopicCreatorTest.java 
> PRE-CREATION 
>   pom.xml eac95c8 
> 
> Diff: https://reviews.apache.org/r/48280/diff/
> 
> 
> Testing
> ---
> 
> * Started multiple Kafka brokers on single machine, configured number of 
> replicas to be 2, started Atlas server and verified they got created 
> correctly.
> * Started multiple Kafka brokers on single machine, configured number of 
> replicas to be 2, started Hive hook and verified it created ATLAS_HOOK 
> correctly. Then started Atlas server and verified it by-passed ATLAS_HOOK 
> creation, but did ATLAS_ENTITIES creation.
> * Started one Kafka broker, onfigured number of replicas to be 2, started 
> Atlas server and verified it received an exception as expected. But topic 
> could still get auto-created and continued.
> * Started 2 instances of Atlas server concurrently and verified topic 
> creation is working fine.
> * Existing UTs/ITs/new UTs pass.
> 
> 
> Thanks,
> 
> Hemanth Yamijala
> 
>

Re: Review Request 47658: ATLAS-499: Update reverse references

[Suma Shivaprasad]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47658/#review136132
---




repository/src/main/java/org/apache/atlas/repository/graph/TypedInstanceToGraphMapper.java
 (line 783)
<https://reviews.apache.org/r/47658/#comment201148>

do we need the HashSet since contains and all will both O(n)


- Suma Shivaprasad


On June 3, 2016, 1:55 p.m., David Kantor wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47658/
> ---
> 
> (Updated June 3, 2016, 1:55 p.m.)
> 
> 
> Review request for atlas and Shwetha GS.
> 
> 
> Bugs: ATLAS-499
> https://issues.apache.org/jira/browse/ATLAS-499
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> ATLAS-499: Automatically update inverse references to prevent repository 
> corruption from unbalanced references.
> 
> 
> Diffs
> -
> 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/DeleteHandler.java 
> 91f9bd008a6939dfe78656f5c1845637a30ee9eb 
>   repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java 
> 4f6d0118072380342f5ea302d0aaf7902784849a 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/TypedInstanceToGraphMapper.java
>  4c1f5591f4adead41c3336e64b0bc956836f7edb 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepositoryDeleteTestBase.java
>  449e066a167ba4546b118a77c8e3de5fd99f077b 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/ReverseReferenceUpdateHardDeleteTest.java
>  PRE-CREATION 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/ReverseReferenceUpdateSoftDeleteTest.java
>  PRE-CREATION 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/ReverseReferenceUpdateTestBase.java
>  PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/47658/diff/
> 
> 
> Testing
> ---
> 
> Ran all unit and integration tests with no regressions.
> 
> 
> Thanks,
> 
> David Kantor
> 
>

Re: Review Request 47658: ATLAS-499: Update reverse references

[Suma Shivaprasad]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47658/#review136133
---




repository/src/main/java/org/apache/atlas/repository/graph/TypedInstanceToGraphMapper.java
 (line 441)
<https://reviews.apache.org/r/47658/#comment201149>

for map types, can we check if the reverseAttributeName is set on the map 
type and remove it if set else ignore?


- Suma Shivaprasad


On June 3, 2016, 1:55 p.m., David Kantor wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47658/
> ---
> 
> (Updated June 3, 2016, 1:55 p.m.)
> 
> 
> Review request for atlas and Shwetha GS.
> 
> 
> Bugs: ATLAS-499
> https://issues.apache.org/jira/browse/ATLAS-499
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> ATLAS-499: Automatically update inverse references to prevent repository 
> corruption from unbalanced references.
> 
> 
> Diffs
> -
> 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/DeleteHandler.java 
> 91f9bd008a6939dfe78656f5c1845637a30ee9eb 
>   repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java 
> 4f6d0118072380342f5ea302d0aaf7902784849a 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/TypedInstanceToGraphMapper.java
>  4c1f5591f4adead41c3336e64b0bc956836f7edb 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepositoryDeleteTestBase.java
>  449e066a167ba4546b118a77c8e3de5fd99f077b 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/ReverseReferenceUpdateHardDeleteTest.java
>  PRE-CREATION 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/ReverseReferenceUpdateSoftDeleteTest.java
>  PRE-CREATION 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/ReverseReferenceUpdateTestBase.java
>  PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/47658/diff/
> 
> 
> Testing
> ---
> 
> Ran all unit and integration tests with no regressions.
> 
> 
> Thanks,
> 
> David Kantor
> 
>

Re: Review Request 47841: ATLAS-826 Add support for singleton traits

[Suma Shivaprasad]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47841/#review134887
---




repository/src/main/java/org/apache/atlas/repository/graph/TypedInstanceToGraphMapper.java
 (line 661)
<https://reviews.apache.org/r/47841/#comment199831>

if there any way to support trait attribute updates by its type for 
singleton traits ?


- Suma Shivaprasad


On May 25, 2016, 6:54 p.m., Shwetha GS wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47841/
> ---
> 
> (Updated May 25, 2016, 6:54 p.m.)
> 
> 
> Review request for atlas.
> 
> 
> Bugs: ATLAS-826
> https://issues.apache.org/jira/browse/ATLAS-826
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Adds support for singleton traits:
> 1. while defining trait type, trait can be marked as singleton
> 2. The first time the trait is associated with an entity, the trait instance 
> is created
> 3. When the singleton trait is associated to another entity, the old instance 
> of the trait is used
> 4. When the singleton trait is deleted for an entity, just the edge reference 
> is removed(singleton trait instance is never deleted)
> 5. Modified UI to pass the singleton attribute in trait type definition - but 
> doesn't seem to work, will check tomorrow
> 6. Updated type store to store the new attribute 'singleton'
> 
> Note:
> 1. trait type definition json adds new attribute called 'singleton' and is 
> not backward compatible. Lets handle the backward compatibity in another 
> jira, need to fix it for description as well
> 2. There is no way to directly create singleton trait instance(its created 
> the first time the singleton trait is associated with the entity). Can be 
> addressed in another jira again
> 3. When the singleton trait is associated with entity the second time, the 
> trait attribute values are not used. This may be confusing for the user, but 
> can't think of any clean way of enabling this
> 
> 
> Diffs
> -
> 
>   
> addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/model/FalconDataModelGenerator.java
>  397dea4 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/model/HiveDataModelGenerator.java
>  347405c 
>   
> addons/sqoop-bridge/src/main/java/org/apache/atlas/sqoop/model/SqoopDataModelGenerator.java
>  c3bdbfa 
>   catalog/src/main/java/org/apache/atlas/catalog/DefaultTypeSystem.java 
> f71c061 
>   dashboardv2/public/js/views/tag/TagLayoutView.js 03adfbc 
>   repository/src/main/java/org/apache/atlas/repository/Constants.java 893f1b6 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/DeleteHandler.java 
> 91f9bd0 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepository.java
>  0d82d90 
>   repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java 
> 4f6d011 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/HardDeleteHandler.java
>  3636791 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/SoftDeleteHandler.java
>  25aa7c5 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/TypedInstanceToGraphMapper.java
>  4c1f559 
>   
> repository/src/main/java/org/apache/atlas/repository/typestore/GraphBackedTypeStore.java
>  5ed9e02 
>   repository/src/test/java/org/apache/atlas/BaseRepositoryTest.java d1f9430 
>   repository/src/test/java/org/apache/atlas/TestUtils.java 345e874 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepositoryTest.java
>  2d1c33a 
>   
> repository/src/test/java/org/apache/atlas/repository/typestore/GraphBackedTypeStoreTest.java
>  961442b 
>   repository/src/test/scala/org/apache/atlas/query/QueryTestsUtils.scala 
> b5faaf3 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/HierarchicalTypeDefinition.java
>  9a299f0 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/TraitType.java 
> f23bf5b 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/TypeSystem.java 
> 14f1968 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/utils/TypesUtil.java
>  ef8448d 
>   
> typesystem/src/main/scala/org/apache/atlas/typesystem/json/TypesSerialization.scala
>  5618938 
>   
> typesystem/src/test/java/org/apache/atlas/typesystem/json/SerializationJavaTest.java
>  eb1a15a 
>   typesystem/src/test/java/org/apache/atlas/typesystem/types/ClassTest.java 
> daecdd7 
>   typesystem/src/test/java/org/apache/atlas/typesystem/types/TraitTest.java 
> 31bfb2c 
>   
> typesystem/src/test/java/org/apache/atlas/typesystem/types/TypeSystemTest.java
>  574e0f9 
> 
> Diff: https://reviews.apache.org/r/47841/diff/
> 
> 
> Testing
> ---
> 
> Added UTs
> 
> 
> Thanks,
> 
> Shwetha GS
> 
>

Re: Review Request 48280: ATLAS-515: Ability to initialize Kafka topics with more than 1 replica

[Suma Shivaprasad]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48280/#review137355
---


Ship it!




Ship It!

- Suma Shivaprasad


On June 13, 2016, 4:35 p.m., Hemanth Yamijala wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/48280/
> ---
> 
> (Updated June 13, 2016, 4:35 p.m.)
> 
> 
> Review request for atlas.
> 
> 
> Bugs: ATLAS-515
> https://issues.apache.org/jira/browse/ATLAS-515
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Attached patch provides a method to automatically create Kafka topics 
> ATLAS_HOOK and ATLAS_ENTITIES with configured number of replicas.
> 
> 
> Diffs
> -
> 
>   common/src/main/java/org/apache/atlas/utils/AuthenticationUtil.java 3dbab17 
>   distro/src/bin/atlas_client_cmdline.py f109ad3 
>   distro/src/bin/atlas_config.py fab4046 
>   distro/src/bin/atlas_kafka_setup.py PRE-CREATION 
>   distro/src/bin/atlas_kafka_setup_hook.py PRE-CREATION 
>   distro/src/conf/atlas-application.properties 2b90086 
>   distro/src/main/assemblies/standalone-package.xml 1c7b2c5 
>   docs/src/site/twiki/Configuration.twiki 7150483 
>   docs/src/site/twiki/InstallationSteps.twiki 518c380 
>   notification/pom.xml b3738db 
>   notification/src/main/java/org/apache/atlas/hook/AtlasTopicCreator.java 
> PRE-CREATION 
>   notification/src/test/java/org/apache/atlas/hook/AtlasTopicCreatorTest.java 
> PRE-CREATION 
>   pom.xml e13345e 
> 
> Diff: https://reviews.apache.org/r/48280/diff/
> 
> 
> Testing
> ---
> 
> * Started multiple Kafka brokers on single machine, configured number of 
> replicas to be 2, started Atlas server and verified they got created 
> correctly.
> * Started multiple Kafka brokers on single machine, configured number of 
> replicas to be 2, started Hive hook and verified it created ATLAS_HOOK 
> correctly. Then started Atlas server and verified it by-passed ATLAS_HOOK 
> creation, but did ATLAS_ENTITIES creation.
> * Started one Kafka broker, onfigured number of replicas to be 2, started 
> Atlas server and verified it received an exception as expected. But topic 
> could still get auto-created and continued.
> * Started 2 instances of Atlas server concurrently and verified topic 
> creation is working fine.
> * Existing UTs/ITs/new UTs pass.
> 
> 
> Thanks,
> 
> Hemanth Yamijala
> 
>

Re: Review Request 48452: ATLAS-885: optimize HBaseStoreManager to avoid expensive HTable instantiation every 5 seconds

[Suma Shivaprasad]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48452/#review136754
---


Ship it!




Ship It!

- Suma Shivaprasad


On June 9, 2016, 12:41 a.m., Madhan Neethiraj wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/48452/
> ---
> 
> (Updated June 9, 2016, 12:41 a.m.)
> 
> 
> Review request for atlas, enis and Abhay Kulkarni.
> 
> 
> Bugs: ATLAS-885
> https://issues.apache.org/jira/browse/ATLAS-885
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> optimize HBaseStoreManager to avoid expensive HTable instantiation every 5 
> seconds
> 
> 
> Diffs
> -
> 
>   
> titan/src/main/java/com/thinkaurelius/titan/diskstorage/hbase/HBaseStoreManager.java
>  52f28af 
>   
> titan/src/main/java/com/thinkaurelius/titan/diskstorage/hbase/HTable0_98.java 
> 4ddb2f0 
>   
> titan/src/main/java/com/thinkaurelius/titan/diskstorage/hbase/HTable1_0.java 
> 5085abb 
>   
> titan/src/main/java/com/thinkaurelius/titan/diskstorage/hbase/TableMask.java 
> dd3d61e 
> 
> Diff: https://reviews.apache.org/r/48452/diff/
> 
> 
> Testing
> ---
> 
> Verified that the logs generated during HTable instantiation are no more 
> present after this fix.
> Verified that basic scenarios like entity creation (via Hive hook), tag 
> creation, tag assocation, UI interaction work well.
> 
> 
> Thanks,
> 
> Madhan Neethiraj
> 
>

Re: Review Request 48050: ATLAS-503: Not all Hive tables are not imported into Atlas when interrupted with search queries while importing.

[Suma Shivaprasad]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48050/#review135675
---


Ship it!




Ship It!

- Suma Shivaprasad


On May 30, 2016, 6:07 p.m., Hemanth Yamijala wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/48050/
> ---
> 
> (Updated May 30, 2016, 6:07 p.m.)
> 
> 
> Review request for atlas and Suma Shivaprasad.
> 
> 
> Bugs: ATLAS-503
> https://issues.apache.org/jira/browse/ATLAS-503
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> The patch implements retries in HBaseKeyColumnValueStore while acquiring 
> locks within a JVM using the LocalLockMediator. The number of retries is 
> configured using atlas.graph.storage.lock.retries. As described above, the 
> value for this property should be equal to the desired level of parallelism. 
> For e.g. if this is set to 10, it means that 10 concurrent operations can 
> proceed to acquire locks on the same store, key and column.
> 
> 
> Diffs
> -
> 
>   
> titan/src/main/java/com/thinkaurelius/titan/diskstorage/hbase/HBaseKeyColumnValueStore.java
>  b4dc12e 
>   
> titan/src/test/java/com/thinkaurelius/titan/diskstorage/hbase/HBaseKeyColumnValueStoreTest.java
>  PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/48050/diff/
> 
> 
> Testing
> ---
> 
> * Existing UTs and new UTs added here.
> * Also set this configuration to 10 retries and verified that 10 concurrent 
> users from JMeter can create traits in parallel.
> 
> 
> Thanks,
> 
> Hemanth Yamijala
> 
>

Re: Review Request 47841: ATLAS-826 Add support for singleton traits

[Suma Shivaprasad]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47841/#review134884
---




repository/src/main/java/org/apache/atlas/repository/graph/DeleteHandler.java 
(line 196)
<https://reviews.apache.org/r/47841/#comment199827>

with this is there any way to delete singleton traits if no references 
exist to that trait?


- Suma Shivaprasad


On May 25, 2016, 6:54 p.m., Shwetha GS wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47841/
> ---
> 
> (Updated May 25, 2016, 6:54 p.m.)
> 
> 
> Review request for atlas.
> 
> 
> Bugs: ATLAS-826
> https://issues.apache.org/jira/browse/ATLAS-826
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Adds support for singleton traits:
> 1. while defining trait type, trait can be marked as singleton
> 2. The first time the trait is associated with an entity, the trait instance 
> is created
> 3. When the singleton trait is associated to another entity, the old instance 
> of the trait is used
> 4. When the singleton trait is deleted for an entity, just the edge reference 
> is removed(singleton trait instance is never deleted)
> 5. Modified UI to pass the singleton attribute in trait type definition - but 
> doesn't seem to work, will check tomorrow
> 6. Updated type store to store the new attribute 'singleton'
> 
> Note:
> 1. trait type definition json adds new attribute called 'singleton' and is 
> not backward compatible. Lets handle the backward compatibity in another 
> jira, need to fix it for description as well
> 2. There is no way to directly create singleton trait instance(its created 
> the first time the singleton trait is associated with the entity). Can be 
> addressed in another jira again
> 3. When the singleton trait is associated with entity the second time, the 
> trait attribute values are not used. This may be confusing for the user, but 
> can't think of any clean way of enabling this
> 
> 
> Diffs
> -
> 
>   
> addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/model/FalconDataModelGenerator.java
>  397dea4 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/model/HiveDataModelGenerator.java
>  347405c 
>   
> addons/sqoop-bridge/src/main/java/org/apache/atlas/sqoop/model/SqoopDataModelGenerator.java
>  c3bdbfa 
>   catalog/src/main/java/org/apache/atlas/catalog/DefaultTypeSystem.java 
> f71c061 
>   dashboardv2/public/js/views/tag/TagLayoutView.js 03adfbc 
>   repository/src/main/java/org/apache/atlas/repository/Constants.java 893f1b6 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/DeleteHandler.java 
> 91f9bd0 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepository.java
>  0d82d90 
>   repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java 
> 4f6d011 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/HardDeleteHandler.java
>  3636791 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/SoftDeleteHandler.java
>  25aa7c5 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/TypedInstanceToGraphMapper.java
>  4c1f559 
>   
> repository/src/main/java/org/apache/atlas/repository/typestore/GraphBackedTypeStore.java
>  5ed9e02 
>   repository/src/test/java/org/apache/atlas/BaseRepositoryTest.java d1f9430 
>   repository/src/test/java/org/apache/atlas/TestUtils.java 345e874 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepositoryTest.java
>  2d1c33a 
>   
> repository/src/test/java/org/apache/atlas/repository/typestore/GraphBackedTypeStoreTest.java
>  961442b 
>   repository/src/test/scala/org/apache/atlas/query/QueryTestsUtils.scala 
> b5faaf3 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/HierarchicalTypeDefinition.java
>  9a299f0 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/TraitType.java 
> f23bf5b 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/TypeSystem.java 
> 14f1968 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/utils/TypesUtil.java
>  ef8448d 
>   
> typesystem/src/main/scala/org/apache/atlas/typesystem/json/TypesSerialization.scala
>  5618938 
>   
> typesystem/src/test/java/org/apache/atlas/typesystem/json/SerializationJavaTest.java
>  eb1a15a 
>   typesystem/src/test/java/org/apache/atlas/typesystem/types/ClassTest.java 
> daecdd7 
>   typesystem/src/test/java/org/apache/atlas/typesystem/types/TraitTest.java 
> 31bfb2c 
>   
> typesystem/src/test/java/org/apache/atlas/typesystem/types/TypeSystemTest.java
>  574e0f9 
> 
> Diff: https://reviews.apache.org/r/47841/diff/
> 
> 
> Testing
> ---
> 
> Added UTs
> 
> 
> Thanks,
> 
> Shwetha GS
> 
>

Re: Review Request 47638: ATLAS-752 Column rename in Hive should retain traits/tags associated with it

[Suma Shivaprasad]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47638/#review134911
---




addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java (line 
345)
<https://reviews.apache.org/r/47638/#comment199860>

its better to load the newTable from metastore here instead of directly 
passing it. The reason is some flags and properties on the table are 
incorrect/incomplete when they are not reloaded from hive metastore? Any reason 
this was done?


- Suma Shivaprasad


On May 25, 2016, 6:41 p.m., Vimal Sharma wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47638/
> ---
> 
> (Updated May 25, 2016, 6:41 p.m.)
> 
> 
> Review request for atlas and Shwetha GS.
> 
> 
> Bugs: ATLAS-752
> https://issues.apache.org/jira/browse/ATLAS-752
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> ATLAS-752 When a column is renamed, the traits associted with it are lost. 
> The traits should be retained with the new column
> 
> 
> Diffs
> -
> 
>   addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
> 418e755 
>   addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 
> 70100f1 
> 
> Diff: https://reviews.apache.org/r/47638/diff/
> 
> 
> Testing
> ---
> 
> 
> File Attachments
> 
> 
> Revised Patch
>   
> https://reviews.apache.org/media/uploaded/files/2016/05/20/a3ab9b8c-4ffc-4ec6-92d7-1f84db246c6a__ATLAS-752-v3.patch
> 
> 
> Thanks,
> 
> Vimal Sharma
> 
>

Re: Review Request 47638: ATLAS-752 Column rename in Hive should retain traits/tags associated with it

[Suma Shivaprasad]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47638/#review135160
---


Ship it!




Ship It!

- Suma Shivaprasad


On May 26, 2016, 6:04 p.m., Vimal Sharma wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47638/
> ---
> 
> (Updated May 26, 2016, 6:04 p.m.)
> 
> 
> Review request for atlas and Shwetha GS.
> 
> 
> Bugs: ATLAS-752
> https://issues.apache.org/jira/browse/ATLAS-752
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> ATLAS-752 When a column is renamed, the traits associted with it are lost. 
> The traits should be retained with the new column
> 
> 
> Diffs
> -
> 
>   addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
> 418e755 
>   addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 
> 70100f1 
> 
> Diff: https://reviews.apache.org/r/47638/diff/
> 
> 
> Testing
> ---
> 
> 
> File Attachments
> 
> 
> Revised Patch
>   
> https://reviews.apache.org/media/uploaded/files/2016/05/20/a3ab9b8c-4ffc-4ec6-92d7-1f84db246c6a__ATLAS-752-v3.patch
> 
> 
> Thanks,
> 
> Vimal Sharma
> 
>

Re: [VOTE] Release Apache Atlas version 0.7-incubating

[Suma Shivaprasad]

Hi all,

Verified signatures, commit id and tag

Compiled and built source tarball and verified that all tests passed.
Please note that you need to export MAVEN_OPTS at have atleast -Xmx1536m
for the build to succeed

Ran quick start and verified entities created in the UI
Verified that add, delete tag in the UI works
Created a business taxonomy and subterms and was able to associate terms
with assets, search for terms
Verified with both DSL and fulltext search
Verified schema and audit by altering columns through alter table commands
Ran a few hive commands and tested hive hook functionality and that changes
are notified to ATLAS_entities topic

Hence a +1 for the release (binding)

Thanks for taking care of this Shwetha

Suma

On Wed, Jun 22, 2016 at 5:37 AM, Hemanth Yamijala  wrote:

> Hi,
>
> I took this for a spin.
>
> * Downloaded & verified the md5 and sha checksums of the source tarball.
> * Verified git commit ID.
> * Verified tag
> * Compiled the source to create binary tarballs. Tests passed.
> * Installed tarball on a single node and tested with external Hbase, Solr,
> Kafka configuration and verified the following functions:
>   * Ran quick start, browsed entities created in the UI.
>   * Ran hive hook and created tables, also using CTAS queries, browsed the
> same in UI and verified lineage, schema views.
>   * Created tag, associated and disassociated to entities, searched using
> DSL. Worked as expected.
>   * Verified entity audit events are captured.
>   * Verified entity change events are notified to Kafka topic
> ATLAS_ENTITIES correctly.
> * I then installed another instance on the same machine (different port)
> and configured both instances in HA mode.
>   * Brought up the instances - verified one was ACTIVE and the other
> PASSIVE
>   * Failed over couple of times and verified automatic failover was working
>   * Verified UI redirection from passive to active instance
>   * Verified Kafka messages are picked up by the current active instance
> correctly.
>
> Based on the above, I am +1 for the release (binding).
>
> Thanks, Shwetha, for putting this together!
>
> Thanks
> hemanth
>
>
>
> On 6/22/16, 4:03 PM, "Shwetha Shivalingamurthy" <
> sshivalingamur...@hortonworks.com> wrote:
>
> >Hi All,
> >
> >I have created a build for Apache Atlas 0.7-incubating, release candidate
> 0. Thanks to everyone who have contributed to this release and also tested
> this.
> >
> >The source tarball (*.tar.gz), signature (*.asc), checksum (*.md5, *.sha):
> ><
> https://dist.apache.org/repos/dist/dev/incubator/atlas/0.7.0-incubating-rc0/
> >
> https://dist.apache.org/repos/dist/dev/incubator/atlas/0.7.0-incubating-rc0/
> >
> >The SHA512 checksum of the archive is
> >1B1F16A6 5F7A4EAE 057F410B DC4753B1 F1C0D49B 588B5C91 C8641634 77F0CDF8
> 15F46252
> > 863D6CEA C98863B2 3853F156 B18D212E 30159C47 ACB1E7BA C6F8DB91
> >
> >The commit id (ca9fea712d9a920e5949f16c32a759acd9222f19) to be voted upon:
> >
> https://git-wip-us.apache.org/repos/asf?p=incubator-atlas.git;a=commit;h=ca9fea712d9a920e5949f16c32a759acd9222f19
> >
> >The tag to be voted upon:
> >
> https://git-wip-us.apache.org/repos/asf?p=incubator-atlas.git;a=log;h=refs/tags/release-0.7-rc0
> >
> >The list of fixed issues:
> >
> https://git-wip-us.apache.org/repos/asf?p=incubator-atlas.git;a=blob;f=release-log.txt;h=5d72620f1db75c3e032745c3f7c6cefa85b9505c;hb=refs/heads/0.7-incubating
> >
> >Keys to verify the signature of the release artifact are available at:
> >http://www.apache.org/dist/incubator/atlas/KEYS
> >PGP release keys:
> >http://pgp.mit.edu/pks/lookup?op=vindex=0xA0E6F9F5D96BF0FD
> >
> >Note that this is a source only release and we are voting on the source
> release-0.7-incubating-rc0.
> >
> >Please download, test, and try it out.
> >
> >Vote will be open for at least 72 hours (until the required number of
> IPMC votes are obtained) till 26th June, 2016 10 AM PDT.
> >
> >[ ] +1 approve
> >[ ] +0 no opinion
> >[ ] -1 disapprove (and reason why)
> >
> >+1 from my side for the release.
> >
> >For folks not familiar with vetting a release, please refer to
> >http://incubator.apache.org/guides/releasemanagement.html#check-list
> >
> >Regards,
> >Shwetha
>

Re: Review Request 48939: ATLAS-904 Handle process qualified name per Hive Operation

[Suma Shivaprasad]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48939/
---

(Updated June 20, 2016, 5:27 p.m.)


Review request for atlas, Shwetha GS and Hemanth Yamijala.


Changes
---

Thanks for reviewing Hemanth. Fixed review comments. Please reopen any issue 
which I have dropped if you feel it should be addressed or if you have any more 
questions.


Bugs: ATLAS-904
https://issues.apache.org/jira/browse/ATLAS-904


Repository: atlas


Description
---

1. Process qualified name = HiveOperation.name + sorted inputs + sorted outputs
2. HiveOperation.name doesnt provide identifiers for identiifying INSERT, 
INSERT_OVERWRITE, UPDATE, DELETE etc separately . Hence adding 
WriteEntity.WriteType as well which exhibits the following behaviour
a. If there are multiple outputs, for each output, adds the query 
type(WriteType)
b. if query being run if is type INSERT [into/overwrite] TABLE [PARTITION], 
WriteType is INSERT/INSERT_OVERWRITE
b. If query is of type INSERT OVERWRITE hdfs_path, adds WriteType as PATH_WRITE
c. If query is of type UPDATE/DELETE, adds type as UPDATE/DELETE [ Note - 
linage is not available for this since this is single table operation]
3.When input is of type local dir or hdfs path currently, it doesnt add it to 
qualified name. The reason is that partition based paths cause a lot of 
processes to be created in this case instead of updating the same process.
Pending:
Address Shwetha G S suggestion to add hdfs paths to process qualified name only 
in case of non-partition based queries. This needs to be done per HiveOperation 
type
1. if HiveOperation = LOAD, IMPORT, EXPORT - detect if the current query 
context is dealing with partitions and do not add if it is partition based.
2. If HiveOperation = INSERT OVERWRITE DFS_PATH/LOCAL_PATH , then detect if the 
query context is dealing with a partitioned table in inputs and decide if we 
need to add or not.


Diffs (updated)
-

  
addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
 c956a32 
  addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
5d9950f 
  addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 
5a175e7 
  webapp/src/main/java/org/apache/atlas/web/resources/EntityResource.java 
0713d30 

Diff: https://reviews.apache.org/r/48939/diff/


Testing
---

Existing tests modified to query with new qualified name. Need to add tests for 
INSERT INTO TABLE


Thanks,

Suma Shivaprasad

Re: Review Request 48939: ATLAS-904 Handle process qualified name per Hive Operation

[Suma Shivaprasad]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48939/
---

(Updated June 20, 2016, 6:22 p.m.)


Review request for atlas, Shwetha GS and Hemanth Yamijala.


Changes
---

Fixed test failures due to lower case change


Bugs: ATLAS-904
https://issues.apache.org/jira/browse/ATLAS-904


Repository: atlas


Description
---

1. Process qualified name = HiveOperation.name + sorted inputs + sorted outputs
2. HiveOperation.name doesnt provide identifiers for identiifying INSERT, 
INSERT_OVERWRITE, UPDATE, DELETE etc separately . Hence adding 
WriteEntity.WriteType as well which exhibits the following behaviour
a. If there are multiple outputs, for each output, adds the query 
type(WriteType)
b. if query being run if is type INSERT [into/overwrite] TABLE [PARTITION], 
WriteType is INSERT/INSERT_OVERWRITE
b. If query is of type INSERT OVERWRITE hdfs_path, adds WriteType as PATH_WRITE
c. If query is of type UPDATE/DELETE, adds type as UPDATE/DELETE [ Note - 
linage is not available for this since this is single table operation]
3.When input is of type local dir or hdfs path currently, it doesnt add it to 
qualified name. The reason is that partition based paths cause a lot of 
processes to be created in this case instead of updating the same process.
Pending:
Address Shwetha G S suggestion to add hdfs paths to process qualified name only 
in case of non-partition based queries. This needs to be done per HiveOperation 
type
1. if HiveOperation = LOAD, IMPORT, EXPORT - detect if the current query 
context is dealing with partitions and do not add if it is partition based.
2. If HiveOperation = INSERT OVERWRITE DFS_PATH/LOCAL_PATH , then detect if the 
query context is dealing with a partitioned table in inputs and decide if we 
need to add or not.


Diffs (updated)
-

  
addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
 c956a32 
  addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
5d9950f 
  addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 
5a175e7 
  webapp/src/main/java/org/apache/atlas/web/resources/EntityResource.java 
0713d30 

Diff: https://reviews.apache.org/r/48939/diff/


Testing
---

Existing tests modified to query with new qualified name. Need to add tests for 
INSERT INTO TABLE


Thanks,

Suma Shivaprasad

Re: Review Request 48939: ATLAS-904 Handle process qualified name per Hive Operation

[Suma Shivaprasad]

> On June 20, 2016, 9:26 a.m., Hemanth Yamijala wrote:
> > addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java, 
> > line 763
> > <https://reviews.apache.org/r/48939/diff/2/?file=1423788#file1423788line763>
> >
> > Do we need a separator between the input set and output set?

This is already being taken care of within the if checks and is added before 
and output dataset entry is added to the buffer.


- Suma


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48939/#review138565
---


On June 20, 2016, 4 a.m., Suma Shivaprasad wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/48939/
> ---
> 
> (Updated June 20, 2016, 4 a.m.)
> 
> 
> Review request for atlas, Shwetha GS and Hemanth Yamijala.
> 
> 
> Bugs: ATLAS-904
> https://issues.apache.org/jira/browse/ATLAS-904
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> 1. Process qualified name = HiveOperation.name + sorted inputs + sorted 
> outputs
> 2. HiveOperation.name doesnt provide identifiers for identiifying INSERT, 
> INSERT_OVERWRITE, UPDATE, DELETE etc separately . Hence adding 
> WriteEntity.WriteType as well which exhibits the following behaviour
> a. If there are multiple outputs, for each output, adds the query 
> type(WriteType)
> b. if query being run if is type INSERT [into/overwrite] TABLE [PARTITION], 
> WriteType is INSERT/INSERT_OVERWRITE
> b. If query is of type INSERT OVERWRITE hdfs_path, adds WriteType as 
> PATH_WRITE
> c. If query is of type UPDATE/DELETE, adds type as UPDATE/DELETE [ Note - 
> linage is not available for this since this is single table operation]
> 3.When input is of type local dir or hdfs path currently, it doesnt add it to 
> qualified name. The reason is that partition based paths cause a lot of 
> processes to be created in this case instead of updating the same process.
> Pending:
> Address Shwetha G S suggestion to add hdfs paths to process qualified name 
> only in case of non-partition based queries. This needs to be done per 
> HiveOperation type
> 1. if HiveOperation = LOAD, IMPORT, EXPORT - detect if the current query 
> context is dealing with partitions and do not add if it is partition based.
> 2. If HiveOperation = INSERT OVERWRITE DFS_PATH/LOCAL_PATH , then detect if 
> the query context is dealing with a partitioned table in inputs and decide if 
> we need to add or not.
> 
> 
> Diffs
> -
> 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
>  c956a32 
>   addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
> 23c82df 
>   addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 
> e7fbf71 
>   webapp/src/main/java/org/apache/atlas/web/resources/EntityResource.java 
> 0713d30 
> 
> Diff: https://reviews.apache.org/r/48939/diff/
> 
> 
> Testing
> ---
> 
> Existing tests modified to query with new qualified name. Need to add tests 
> for INSERT INTO TABLE
> 
> 
> Thanks,
> 
> Suma Shivaprasad
> 
>

Re: Review Request 48945: ATLAs-819 All user defined types should have a set of common attributes

[Suma Shivaprasad]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48945/#review138658
---


Ship it!




Ship It!

- Suma Shivaprasad


On June 20, 2016, 12:01 p.m., Shwetha GS wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/48945/
> ---
> 
> (Updated June 20, 2016, 12:01 p.m.)
> 
> 
> Review request for atlas.
> 
> 
> Bugs: ATLAS-819
> https://issues.apache.org/jira/browse/ATLAS-819
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Added  base type with the attributes name, owner and description that any 
> type can extend and the users can set these attributes.
> 
> 
> Diffs
> -
> 
>   
> addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/Util/EventUtil.java
>  c1ccd05 
>   
> addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/bridge/FalconBridge.java
>  1621d95 
>   
> addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/event/FalconEvent.java
>  37df6da 
>   
> addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/hook/FalconHook.java
>  95f255e 
>   
> addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/model/FalconDataModelGenerator.java
>  81cd5e0 
>   
> addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/service/AtlasService.java
>  c92bd43 
>   
> addons/falcon-bridge/src/test/java/org/apache/atlas/falcon/hook/FalconHookIT.java
>  8a5736a 
>   
> addons/hdfs-model/src/main/scala/org/apache/atlas/fs/model/FSDataModel.scala 
> cf81406 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
>  c956a32 
>   addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
> 5d9950f 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/model/HiveDataModelGenerator.java
>  3686fa8 
>   addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 
> 5a175e7 
>   
> addons/sqoop-bridge/src/main/java/org/apache/atlas/sqoop/hook/SqoopHook.java 
> 9db8180 
>   
> addons/sqoop-bridge/src/main/java/org/apache/atlas/sqoop/model/SqoopDataModelGenerator.java
>  c3bdbfa 
>   
> addons/storm-bridge/src/main/java/org/apache/atlas/storm/hook/StormAtlasHook.java
>  b2171c2 
>   
> addons/storm-bridge/src/main/scala/org/apache/atlas/storm/model/StormDataModel.scala
>  bae4c1a 
>   client/src/main/java/org/apache/atlas/AtlasClient.java 4c88da9 
>   distro/src/conf/atlas-log4j.xml e14afa3 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepository.java
>  dd7dd74 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java
>  c385df3 
>   
> repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java
>  d9e6cb9 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/StructTypeDefinition.java
>  9dc93ae 
>   webapp/src/main/java/org/apache/atlas/web/resources/EntityResource.java 
> 0713d30 
>   webapp/src/main/java/org/apache/atlas/web/resources/TypesResource.java 
> 74d89ce 
> 
> Diff: https://reviews.apache.org/r/48945/diff/
> 
> 
> Testing
> ---
> 
> UTs
> 
> 
> Thanks,
> 
> Shwetha GS
> 
>

Re: Review Request 48945: ATLAs-819 All user defined types should have a set of common attributes

[Suma Shivaprasad]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48945/#review138656
---


Ship it!




Ship It!

- Suma Shivaprasad


On June 20, 2016, 12:01 p.m., Shwetha GS wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/48945/
> ---
> 
> (Updated June 20, 2016, 12:01 p.m.)
> 
> 
> Review request for atlas.
> 
> 
> Bugs: ATLAS-819
> https://issues.apache.org/jira/browse/ATLAS-819
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Added  base type with the attributes name, owner and description that any 
> type can extend and the users can set these attributes.
> 
> 
> Diffs
> -
> 
>   
> addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/Util/EventUtil.java
>  c1ccd05 
>   
> addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/bridge/FalconBridge.java
>  1621d95 
>   
> addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/event/FalconEvent.java
>  37df6da 
>   
> addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/hook/FalconHook.java
>  95f255e 
>   
> addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/model/FalconDataModelGenerator.java
>  81cd5e0 
>   
> addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/service/AtlasService.java
>  c92bd43 
>   
> addons/falcon-bridge/src/test/java/org/apache/atlas/falcon/hook/FalconHookIT.java
>  8a5736a 
>   
> addons/hdfs-model/src/main/scala/org/apache/atlas/fs/model/FSDataModel.scala 
> cf81406 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
>  c956a32 
>   addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
> 5d9950f 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/model/HiveDataModelGenerator.java
>  3686fa8 
>   addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 
> 5a175e7 
>   
> addons/sqoop-bridge/src/main/java/org/apache/atlas/sqoop/hook/SqoopHook.java 
> 9db8180 
>   
> addons/sqoop-bridge/src/main/java/org/apache/atlas/sqoop/model/SqoopDataModelGenerator.java
>  c3bdbfa 
>   
> addons/storm-bridge/src/main/java/org/apache/atlas/storm/hook/StormAtlasHook.java
>  b2171c2 
>   
> addons/storm-bridge/src/main/scala/org/apache/atlas/storm/model/StormDataModel.scala
>  bae4c1a 
>   client/src/main/java/org/apache/atlas/AtlasClient.java 4c88da9 
>   distro/src/conf/atlas-log4j.xml e14afa3 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepository.java
>  dd7dd74 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java
>  c385df3 
>   
> repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java
>  d9e6cb9 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/StructTypeDefinition.java
>  9dc93ae 
>   webapp/src/main/java/org/apache/atlas/web/resources/EntityResource.java 
> 0713d30 
>   webapp/src/main/java/org/apache/atlas/web/resources/TypesResource.java 
> 74d89ce 
> 
> Diff: https://reviews.apache.org/r/48945/diff/
> 
> 
> Testing
> ---
> 
> UTs
> 
> 
> Thanks,
> 
> Shwetha GS
> 
>

Re: Review Request 49191: ATLAS-947 Return state information in inputs and outputs lineage API

[Suma Shivaprasad]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49191/#review139387
---


Ship it!




Ship It!

- Suma Shivaprasad


On June 24, 2016, 10:12 a.m., Shwetha GS wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/49191/
> ---
> 
> (Updated June 24, 2016, 10:12 a.m.)
> 
> 
> Review request for atlas.
> 
> 
> Bugs: ATLAS-947
> https://issues.apache.org/jira/browse/ATLAS-947
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> State is returned in lineage APIs
> 
> 
> Diffs
> -
> 
>   addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 
> 2464646 
>   
> repository/src/main/java/org/apache/atlas/discovery/DataSetLineageService.java
>  a2f47f0 
>   
> repository/src/main/java/org/apache/atlas/discovery/graph/DefaultGraphPersistenceStrategy.java
>  90718ed 
>   
> repository/src/main/java/org/apache/atlas/repository/MetadataRepository.java 
> 43e9f85 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepository.java
>  a4bdef7 
>   
> repository/src/main/scala/org/apache/atlas/query/GraphPersistenceStrategies.scala
>  34d101a 
>   
> repository/src/test/java/org/apache/atlas/discovery/DataSetLineageServiceTest.java
>  3028dde 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/TypeSystem.java 
> 90b5815 
>   typesystem/src/test/java/org/apache/atlas/ApplicationPropertiesTest.java 
> 6d43359 
>   typesystem/src/test/resources/atlas-application.properties 28d8af3 
>   
> webapp/src/test/java/org/apache/atlas/web/resources/DataSetLineageJerseyResourceIT.java
>  bc4f239 
> 
> Diff: https://reviews.apache.org/r/49191/diff/
> 
> 
> Testing
> ---
> 
> UTs
> 
> 
> Thanks,
> 
> Shwetha GS
> 
>

Re: Review Request 45784: Hve Hook - Support tracking lineage for External Tables( Create/alter) , Load, import, export

[Suma Shivaprasad]

> On April 6, 2016, 11:21 a.m., Shwetha GS wrote:
> > addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java,
> >  line 480
> > <https://reviews.apache.org/r/45784/diff/1/?file=1327184#file1327184line480>
> >
> > We need to fix the clusterName mess later - can't pickup hdfs 
> > clustername from hive conf

Have removed it for now since we dont know the right clusterName


- Suma


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45784/#review127310
---


On April 6, 2016, 7:08 p.m., Suma Shivaprasad wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/45784/
> ---
> 
> (Updated April 6, 2016, 7:08 p.m.)
> 
> 
> Review request for atlas.
> 
> 
> Bugs: ATLAS-527
> https://issues.apache.org/jira/browse/ATLAS-527
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Added support to track lineage between HDFS Paths and hive tables  in 
> 
> a. LOAD( at table, partition level) - input is a HDFS path and output is 
> table( even though we dont create partition entities, we are still tracking 
> the lineage at table level for partitions. This could be an issue if there 
> are large number of partition queries which is not being addressed in this 
> jira - https://issues.apache.org/jira/browse/ATLAS-619) . refer 
> https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DML
> b. IMPORT, EXPORT to and from hdfs paths - Refer 
> https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DML
> c. CREATE EXTERNAL TABLE - input is hdfs path and o/p is table
> d. ALTER TABLE LOCATION for an external table - input is the new hdfs path 
> and o/p is the table.
> 
> Also changed the ordering of model registration by sorting them by 
> modifiedTime to ensure they are registered in correct order
> 
> 
> Diffs
> -
> 
>   
> addons/hdfs-model/src/main/java/org/apache/atlas/fs/model/FSDataModelGenerator.java
>  555d565 
>   
> addons/hdfs-model/src/main/scala/org/apache/atlas/fs/model/FSDataModel.scala 
> c964f73 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
>  3a802d7 
>   addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
> 68e32ff 
>   addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 
> e17afb8 
>   
> addons/storm-bridge/src/main/java/org/apache/atlas/storm/hook/StormAtlasHook.java
>  5665856 
>   client/src/main/java/org/apache/atlas/AtlasClient.java c3b4ba9 
>   
> repository/src/main/java/org/apache/atlas/services/ReservedTypesRegistrar.java
>  430bb6b 
> 
> Diff: https://reviews.apache.org/r/45784/diff/
> 
> 
> Testing
> ---
> 
> Added tests in HiveHookIT
> 
> 
> Thanks,
> 
> Suma Shivaprasad
> 
>

Re: Review Request 45784: Hve Hook - Support tracking lineage for External Tables( Create/alter) , Load, import, export

[Suma Shivaprasad]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45784/
---

(Updated April 6, 2016, 5:44 p.m.)


Review request for atlas.


Changes
---

Fixed review comments


Bugs: ATLAS-527
https://issues.apache.org/jira/browse/ATLAS-527


Repository: atlas


Description
---

Added support to track lineage between HDFS Paths and hive tables  in 

a. LOAD( at table, partition level) - input is a HDFS path and output is table( 
even though we dont create partition entities, we are still tracking the 
lineage at table level for partitions. This could be an issue if there are 
large number of partition queries which is not being addressed in this jira - 
https://issues.apache.org/jira/browse/ATLAS-619) . refer 
https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DML
b. IMPORT, EXPORT to and from hdfs paths - Refer 
https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DML
c. CREATE EXTERNAL TABLE - input is hdfs path and o/p is table
d. ALTER TABLE LOCATION for an external table - input is the new hdfs path and 
o/p is the table.

Also changed the ordering of model registration by sorting them by modifiedTime 
to ensure they are registered in correct order


Diffs (updated)
-

  
addons/hdfs-model/src/main/java/org/apache/atlas/fs/model/FSDataModelGenerator.java
 555d565 
  addons/hdfs-model/src/main/scala/org/apache/atlas/fs/model/FSDataModel.scala 
c964f73 
  
addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
 3a802d7 
  addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
68e32ff 
  addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 
e17afb8 
  
addons/storm-bridge/src/main/java/org/apache/atlas/storm/hook/StormAtlasHook.java
 5665856 
  client/src/main/java/org/apache/atlas/AtlasClient.java c3b4ba9 
  
repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java 
0a04c5f 
  
repository/src/main/java/org/apache/atlas/services/ReservedTypesRegistrar.java 
430bb6b 

Diff: https://reviews.apache.org/r/45784/diff/


Testing
---

Added tests in HiveHookIT


Thanks,

Suma Shivaprasad

Review Request 45843: Support for drop table, view

[Suma Shivaprasad]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45843/
---

Review request for atlas.


Bugs: ATLAS-528
https://issues.apache.org/jira/browse/ATLAS-528


Repository: atlas


Description
---

Also added support for 
a. truncate table - has o/p as the same table in lineage. Not setting anything 
in inputs since it might be confusing. Felt it was important to capture this 
from a audit/governance perspective
b. alter table partition column type - which can be used to change a partition 
keys's column type since table has reference to partition keys


Diffs
-

  
addons/hdfs-model/src/main/java/org/apache/atlas/fs/model/FSDataModelGenerator.java
 555d565 
  addons/hdfs-model/src/main/scala/org/apache/atlas/fs/model/FSDataModel.scala 
c964f73 
  
addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
 3a802d7 
  addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
68e32ff 
  addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 
e17afb8 
  
addons/storm-bridge/src/main/java/org/apache/atlas/storm/hook/StormAtlasHook.java
 5665856 
  client/src/main/java/org/apache/atlas/AtlasClient.java c3b4ba9 
  
repository/src/main/java/org/apache/atlas/services/ReservedTypesRegistrar.java 
430bb6b 

Diff: https://reviews.apache.org/r/45843/diff/


Testing
---

Added test in HiveHookIT


Thanks,

Suma Shivaprasad

Re: Review Request 46011: ATLAS-622 Introduce soft delete

[Suma Shivaprasad]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46011/#review128505
---




repository/src/main/java/org/apache/atlas/repository/graph/DeleteHandler.java 
(line 244)
<https://reviews.apache.org/r/46011/#comment191919>

Please enable this check. Raising this to track so that we dont miss


- Suma Shivaprasad


On April 12, 2016, 3:27 p.m., Shwetha GS wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/46011/
> ---
> 
> (Updated April 12, 2016, 3:27 p.m.)
> 
> 
> Review request for atlas.
> 
> 
> Bugs: ATLAS-622
> https://issues.apache.org/jira/browse/ATLAS-622
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Added soft delete which is enabled by default. Hard delete can be configured 
> using guice. Had to refactor entity deletes. 
> 
> Pending:
> 1. 
> GraphBackedMetadataRepositoryDeleteEntitiesTest.testDisconnectUnidirectionalArrayReferenceFromStructAndTraitTypes
>  is disabled, need to debug
> 2. Have to write more tests for soft deletes
> 3. hive hook tests - have fixed some tests, 13/40 are still failing
> 
> 
> Diffs
> -
> 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
>  6b348e2 
>   addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
> b947a8c 
>   addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 
> 2f0c71f 
>   client/src/main/java/org/apache/atlas/AtlasClient.java 938a0a3 
>   repository/src/main/java/org/apache/atlas/RepositoryMetadataModule.java 
> ce1bdfb 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/AtlasEdgeLabel.java
>  da2ad9a 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/DeleteHandler.java 
> PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepository.java
>  b94ff5a 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java
>  e7e8fb9 
>   repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java 
> 895f9df 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/GraphToTypedInstanceMapper.java
>  1d682bb 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/HardDeleteHandler.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/SoftDeleteHandler.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/TypedInstanceToGraphMapper.java
>  7011fdf 
>   
> repository/src/main/java/org/apache/atlas/repository/typestore/GraphBackedTypeStore.java
>  71e9028 
>   repository/src/test/java/org/apache/atlas/BaseHiveRepositoryTest.java 
> d07f89d 
>   repository/src/test/java/org/apache/atlas/TestUtils.java a3cf929 
>   
> repository/src/test/java/org/apache/atlas/discovery/GraphBackedDiscoveryServiceTest.java
>  ea93cbf 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepositoryDeleteEntitiesTest.java
>  975d530 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepositoryDeleteTestBase.java
>  PRE-CREATION 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepositoryTest.java
>  b5440c2 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedReposiotrySoftDeleteTest.java
>  PRE-CREATION 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedRepositoryHardDeleteTest.java
>  PRE-CREATION 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/GraphRepoMapperScaleTest.java
>  3f71eb3 
>   
> repository/src/test/java/org/apache/atlas/repository/typestore/GraphBackedTypeStoreTest.java
>  8ca3132 
>   
> repository/src/test/java/org/apache/atlas/service/DefaultMetadataServiceTest.java
>  156eb3d 
>   repository/src/test/scala/org/apache/atlas/query/GremlinTest.scala 0289b1a 
>   repository/src/test/scala/org/apache/atlas/query/GremlinTest2.scala 79ebfae 
>   repository/src/test/scala/org/apache/atlas/query/LineageQueryTest.scala 
> cd1c424 
>   server-api/src/main/java/org/apache/atlas/RequestContext.java 943e4b8 
>   typesystem/src/main/java/org/apache/atlas/typesystem/IInstance.java ffe40a7 
>   typesystem/src/main/java/org/apache/atlas/typesystem/Referenceable.java 
> 7857483 
>   typesystem/src/main/java/org/apache/

Re: Review Request 46011: ATLAS-622 Introduce soft delete

[Suma Shivaprasad]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46011/#review128447
---




repository/src/main/java/org/apache/atlas/repository/graph/AtlasEdgeLabel.java 
<https://reviews.apache.org/r/46011/#comment191869>

why is this removed?


- Suma Shivaprasad


On April 12, 2016, 3:27 p.m., Shwetha GS wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/46011/
> ---
> 
> (Updated April 12, 2016, 3:27 p.m.)
> 
> 
> Review request for atlas.
> 
> 
> Bugs: ATLAS-622
> https://issues.apache.org/jira/browse/ATLAS-622
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Added soft delete which is enabled by default. Hard delete can be configured 
> using guice. Had to refactor entity deletes. 
> 
> Pending:
> 1. 
> GraphBackedMetadataRepositoryDeleteEntitiesTest.testDisconnectUnidirectionalArrayReferenceFromStructAndTraitTypes
>  is disabled, need to debug
> 2. Have to write more tests for soft deletes
> 3. hive hook tests - have fixed some tests, 13/40 are still failing
> 
> 
> Diffs
> -
> 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
>  6b348e2 
>   addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
> b947a8c 
>   addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 
> 2f0c71f 
>   client/src/main/java/org/apache/atlas/AtlasClient.java 938a0a3 
>   repository/src/main/java/org/apache/atlas/RepositoryMetadataModule.java 
> ce1bdfb 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/AtlasEdgeLabel.java
>  da2ad9a 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/DeleteHandler.java 
> PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepository.java
>  b94ff5a 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java
>  e7e8fb9 
>   repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java 
> 895f9df 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/GraphToTypedInstanceMapper.java
>  1d682bb 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/HardDeleteHandler.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/SoftDeleteHandler.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/TypedInstanceToGraphMapper.java
>  7011fdf 
>   
> repository/src/main/java/org/apache/atlas/repository/typestore/GraphBackedTypeStore.java
>  71e9028 
>   repository/src/test/java/org/apache/atlas/BaseHiveRepositoryTest.java 
> d07f89d 
>   repository/src/test/java/org/apache/atlas/TestUtils.java a3cf929 
>   
> repository/src/test/java/org/apache/atlas/discovery/GraphBackedDiscoveryServiceTest.java
>  ea93cbf 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepositoryDeleteEntitiesTest.java
>  975d530 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepositoryDeleteTestBase.java
>  PRE-CREATION 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepositoryTest.java
>  b5440c2 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedReposiotrySoftDeleteTest.java
>  PRE-CREATION 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedRepositoryHardDeleteTest.java
>  PRE-CREATION 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/GraphRepoMapperScaleTest.java
>  3f71eb3 
>   
> repository/src/test/java/org/apache/atlas/repository/typestore/GraphBackedTypeStoreTest.java
>  8ca3132 
>   
> repository/src/test/java/org/apache/atlas/service/DefaultMetadataServiceTest.java
>  156eb3d 
>   repository/src/test/scala/org/apache/atlas/query/GremlinTest.scala 0289b1a 
>   repository/src/test/scala/org/apache/atlas/query/GremlinTest2.scala 79ebfae 
>   repository/src/test/scala/org/apache/atlas/query/LineageQueryTest.scala 
> cd1c424 
>   server-api/src/main/java/org/apache/atlas/RequestContext.java 943e4b8 
>   typesystem/src/main/java/org/apache/atlas/typesystem/IInstance.java ffe40a7 
>   typesystem/src/main/java/org/apache/atlas/typesystem/Referenceable.java 
> 7857483 
>   typesystem/src/main/java/org/apache/atlas/typesystem/Struct.java 70deab2 
>   
> typesystem

Re: Review Request 46011: ATLAS-622 Introduce soft delete

[Suma Shivaprasad]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46011/#review128312
---




addons/hdfs-model/src/main/scala/org/apache/atlas/fs/model/FSDataModel.scala 
(line 42)
<https://reviews.apache.org/r/46011/#comment191740>

Had kept this non-unique since local FS Paths could be modelled as just 
non-unique paths. Also need to move out the REFERENCE_SUPER_TYPE inheritance 
out of this. Will take care of that in another jira. Pls revert this for now.


- Suma Shivaprasad


On April 11, 2016, 8:55 a.m., Shwetha GS wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/46011/
> ---
> 
> (Updated April 11, 2016, 8:55 a.m.)
> 
> 
> Review request for atlas.
> 
> 
> Bugs: ATLAS-622
> https://issues.apache.org/jira/browse/ATLAS-622
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Added soft delete which is enabled by default. Hard delete can be configured 
> using guice. Had to refactor entity deletes. 
> 
> Pending:
> 1. 
> GraphBackedMetadataRepositoryDeleteEntitiesTest.testDisconnectUnidirectionalArrayReferenceFromStructAndTraitTypes
>  is disabled, need to debug
> 2. Have to write more tests for soft deletes
> 3. hive hook tests - have fixed some tests, 13/40 are still failing
> 
> 
> Diffs
> -
> 
>   
> addons/hdfs-model/src/main/scala/org/apache/atlas/fs/model/FSDataModel.scala 
> c380a92 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
>  6b348e2 
>   addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 
> 2f0c71f 
>   repository/src/main/java/org/apache/atlas/RepositoryMetadataModule.java 
> ce1bdfb 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/DeleteHandler.java 
> PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepository.java
>  b94ff5a 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java
>  e7e8fb9 
>   repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java 
> 895f9df 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/GraphToTypedInstanceMapper.java
>  1d682bb 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/HardDeleteHandler.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/SoftDeleteHandler.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/TypedInstanceToGraphMapper.java
>  7011fdf 
>   
> repository/src/main/java/org/apache/atlas/repository/typestore/GraphBackedTypeStore.java
>  71e9028 
>   repository/src/test/java/org/apache/atlas/BaseHiveRepositoryTest.java 
> d07f89d 
>   repository/src/test/java/org/apache/atlas/TestUtils.java a3cf929 
>   
> repository/src/test/java/org/apache/atlas/discovery/GraphBackedDiscoveryServiceTest.java
>  ea93cbf 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepositoryDeleteEntitiesTest.java
>  975d530 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepositoryTest.java
>  b5440c2 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/GraphRepoMapperScaleTest.java
>  3f71eb3 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/SoftDeleteHandlerTest.java
>  PRE-CREATION 
>   
> repository/src/test/java/org/apache/atlas/repository/typestore/GraphBackedTypeStoreTest.java
>  8ca3132 
>   
> repository/src/test/java/org/apache/atlas/service/DefaultMetadataServiceTest.java
>  156eb3d 
>   repository/src/test/scala/org/apache/atlas/query/GremlinTest.scala 0289b1a 
>   repository/src/test/scala/org/apache/atlas/query/GremlinTest2.scala 79ebfae 
>   repository/src/test/scala/org/apache/atlas/query/LineageQueryTest.scala 
> cd1c424 
>   server-api/src/main/java/org/apache/atlas/RequestContext.java 943e4b8 
>   typesystem/src/main/java/org/apache/atlas/typesystem/IInstance.java ffe40a7 
>   typesystem/src/main/java/org/apache/atlas/typesystem/Referenceable.java 
> 7857483 
>   typesystem/src/main/java/org/apache/atlas/typesystem/Struct.java 70deab2 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/persistence/DownCastStructInstance.java
>  d3b9a33 
>   typesystem/src/main/java/org/apache/atlas/typesystem/persistence/Id.java 
> d742bb7 
>   
> typesystem/src/main/

Re: Review Request 45666: ATLAS-621 Introduce entity state in Id object

[Suma Shivaprasad]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45666/#review128018
---


Ship it!




Ship It!

- Suma Shivaprasad


On April 9, 2016, 4:13 p.m., Shwetha GS wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/45666/
> ---
> 
> (Updated April 9, 2016, 4:13 p.m.)
> 
> 
> Review request for atlas.
> 
> 
> Bugs: ATLAS-621
> https://issues.apache.org/jira/browse/ATLAS-621
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Added entity state with ACTIVE/DELETED in 
> org.apache.atlas.typesystem.persistence.Id which is the id object used in 
> entity definition. At entity create, added new property for state and is set 
> to ACTIVE for entities. This is set only for vertices with id, which implies 
> there is no state for structs and traits. I will revisit this when modifying 
> hard delete to soft delete.
> 
> Also, modified Instance serialisations/deserialisations used in entity 
> definition and search. However, lineage doesn't return back entity state
> 
> 
> Diffs
> -
> 
>   
> notification/src/test/java/org/apache/atlas/notification/hook/HookNotificationTest.java
>  11b7a53 
>   repository/src/main/java/org/apache/atlas/repository/Constants.java aa3326e 
>   repository/src/main/java/org/apache/atlas/repository/DiscoverInstances.java 
> 69b4a00 
>   repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java 
> 2eb4d5b 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/GraphToTypedInstanceMapper.java
>  1fabafa 
>   
> repository/src/test/java/org/apache/atlas/discovery/GraphBackedDiscoveryServiceTest.java
>  b4a9cb2 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepositoryTest.java
>  f1dd604 
>   typesystem/src/main/java/org/apache/atlas/typesystem/Referenceable.java 
> aa1736d 
>   typesystem/src/main/java/org/apache/atlas/typesystem/persistence/Id.java 
> 20c2f91 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/ClassType.java 
> 475ac9f 
>   
> typesystem/src/main/scala/org/apache/atlas/typesystem/json/InstanceSerialization.scala
>  b8fa4ea 
>   
> typesystem/src/main/scala/org/apache/atlas/typesystem/json/Serialization.scala
>  c5aa6e8 
>   typesystem/src/test/java/org/apache/atlas/typesystem/types/BaseTest.java 
> e3352b0 
>   
> typesystem/src/test/scala/org/apache/atlas/typesystem/json/SerializationTest.scala
>  f98ca93 
> 
> Diff: https://reviews.apache.org/r/45666/diff/
> 
> 
> Testing
> ---
> 
> UTs
> 
> 
> Thanks,
> 
> Shwetha GS
> 
>

Review Request 45784: Hve Hook - Support tracking lineage for External Tables( Create/alter) , Load, import, export

[Suma Shivaprasad]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45784/
---

Review request for atlas.


Bugs: ATLAS-527
https://issues.apache.org/jira/browse/ATLAS-527


Repository: atlas


Description
---

Added support to track lineage between HDFS Paths and hive tables  in 

a. LOAD( at table, partition level) - input is a HDFS path and output is table( 
even though we dont create partition entities, we are still tracking the 
lineage at table level for partitions. This could be an issue if there are 
large number of partition queries which is not being addressed in this jira - 
https://issues.apache.org/jira/browse/ATLAS-619) . refer 
https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DML
b. IMPORT, EXPORT to and from hdfs paths - Refer 
https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DML
c. CREATE EXTERNAL TABLE - input is hdfs path and o/p is table
d. ALTER TABLE LOCATION for an external table - input is the new hdfs path and 
o/p is the table.


Diffs
-

  
addons/hdfs-model/src/main/java/org/apache/atlas/fs/model/FSDataModelGenerator.java
 555d565 
  addons/hdfs-model/src/main/scala/org/apache/atlas/fs/model/FSDataModel.scala 
c964f73 
  addons/hive-bridge/pom.xml e125f18 
  
addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
 3a802d7 
  addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
68e32ff 
  addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 
e17afb8 
  
addons/storm-bridge/src/main/java/org/apache/atlas/storm/hook/StormAtlasHook.java
 5665856 
  
repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java 
0a04c5f 
  
repository/src/main/java/org/apache/atlas/services/ReservedTypesRegistrar.java 
430bb6b 

Diff: https://reviews.apache.org/r/45784/diff/


Testing
---


Thanks,

Suma Shivaprasad

Re: Review Request 45784: Hve Hook - Support tracking lineage for External Tables( Create/alter) , Load, import, export

[Suma Shivaprasad]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45784/
---

(Updated April 5, 2016, 11:54 p.m.)


Review request for atlas.


Bugs: ATLAS-527
https://issues.apache.org/jira/browse/ATLAS-527


Repository: atlas


Description
---

Added support to track lineage between HDFS Paths and hive tables  in 

a. LOAD( at table, partition level) - input is a HDFS path and output is table( 
even though we dont create partition entities, we are still tracking the 
lineage at table level for partitions. This could be an issue if there are 
large number of partition queries which is not being addressed in this jira - 
https://issues.apache.org/jira/browse/ATLAS-619) . refer 
https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DML
b. IMPORT, EXPORT to and from hdfs paths - Refer 
https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DML
c. CREATE EXTERNAL TABLE - input is hdfs path and o/p is table
d. ALTER TABLE LOCATION for an external table - input is the new hdfs path and 
o/p is the table.


Diffs
-

  
addons/hdfs-model/src/main/java/org/apache/atlas/fs/model/FSDataModelGenerator.java
 555d565 
  addons/hdfs-model/src/main/scala/org/apache/atlas/fs/model/FSDataModel.scala 
c964f73 
  addons/hive-bridge/pom.xml e125f18 
  
addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
 3a802d7 
  addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
68e32ff 
  addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 
e17afb8 
  
addons/storm-bridge/src/main/java/org/apache/atlas/storm/hook/StormAtlasHook.java
 5665856 
  
repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java 
0a04c5f 
  
repository/src/main/java/org/apache/atlas/services/ReservedTypesRegistrar.java 
430bb6b 

Diff: https://reviews.apache.org/r/45784/diff/


Testing (updated)
---

Added tests in HiveHookIT


Thanks,

Suma Shivaprasad

Re: Review Request 45784: Hve Hook - Support tracking lineage for External Tables( Create/alter) , Load, import, export

[Suma Shivaprasad]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45784/
---

(Updated April 5, 2016, 11:58 p.m.)


Review request for atlas.


Bugs: ATLAS-527
https://issues.apache.org/jira/browse/ATLAS-527


Repository: atlas


Description (updated)
---

Added support to track lineage between HDFS Paths and hive tables  in 

a. LOAD( at table, partition level) - input is a HDFS path and output is table( 
even though we dont create partition entities, we are still tracking the 
lineage at table level for partitions. This could be an issue if there are 
large number of partition queries which is not being addressed in this jira - 
https://issues.apache.org/jira/browse/ATLAS-619) . refer 
https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DML
b. IMPORT, EXPORT to and from hdfs paths - Refer 
https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DML
c. CREATE EXTERNAL TABLE - input is hdfs path and o/p is table
d. ALTER TABLE LOCATION for an external table - input is the new hdfs path and 
o/p is the table.

Also changed the ordering of model registration by sorting them by modifiedTime 
to ensure they are registered in correct order


Diffs
-

  
addons/hdfs-model/src/main/java/org/apache/atlas/fs/model/FSDataModelGenerator.java
 555d565 
  addons/hdfs-model/src/main/scala/org/apache/atlas/fs/model/FSDataModel.scala 
c964f73 
  addons/hive-bridge/pom.xml e125f18 
  
addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
 3a802d7 
  addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
68e32ff 
  addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 
e17afb8 
  
addons/storm-bridge/src/main/java/org/apache/atlas/storm/hook/StormAtlasHook.java
 5665856 
  
repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java 
0a04c5f 
  
repository/src/main/java/org/apache/atlas/services/ReservedTypesRegistrar.java 
430bb6b 

Diff: https://reviews.apache.org/r/45784/diff/


Testing
---

Added tests in HiveHookIT


Thanks,

Suma Shivaprasad

Re: Review Request 45784: Hve Hook - Support tracking lineage for External Tables( Create/alter) , Load, import, export

[Suma Shivaprasad]

> On April 6, 2016, 11:21 a.m., Shwetha GS wrote:
> > addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java, 
> > line 519
> > <https://reviews.apache.org/r/45784/diff/1/?file=1327185#file1327185line519>
> >
> > Aren't there cases where input/output is local fs, for example load 
> > from local path?

I am filtering out the cases where it is LOCAL_DIR  by checking getType = 
DFS_DIR and theres also test case for LOAD local DIR and INSERT into local dir 
which confirms that this case is addressed. You are suggesting we ignore local 
dirs right?


> On April 6, 2016, 11:21 a.m., Shwetha GS wrote:
> > addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java, 
> > line 558
> > <https://reviews.apache.org/r/45784/diff/1/?file=1327185#file1327185line558>
> >
> > This should be part of HiveMetaStoreBridge and should be used in 
> > import-hive as well?
> > 
> > Because this lineage will be created in import-hive, process name 
> > should be just tablename for create table so that its created just once.

Initially this was my thought too. However not sure how to get the query for 
the create table itself. I checked how show create table constructs this  and 
it is on the fly and it does not store in metadata. Also, if we dont address 
this, tt will look different from the other lineages where we will always hav 
the query in the process . So did nto want to address this now till we figure 
out how we can construct the query itself. Created a separate issue to track 
this - https://issues.apache.org/jira/browse/ATLAS-642


- Suma


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45784/#review127310
---


On April 5, 2016, 11:58 p.m., Suma Shivaprasad wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/45784/
> ---
> 
> (Updated April 5, 2016, 11:58 p.m.)
> 
> 
> Review request for atlas.
> 
> 
> Bugs: ATLAS-527
> https://issues.apache.org/jira/browse/ATLAS-527
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Added support to track lineage between HDFS Paths and hive tables  in 
> 
> a. LOAD( at table, partition level) - input is a HDFS path and output is 
> table( even though we dont create partition entities, we are still tracking 
> the lineage at table level for partitions. This could be an issue if there 
> are large number of partition queries which is not being addressed in this 
> jira - https://issues.apache.org/jira/browse/ATLAS-619) . refer 
> https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DML
> b. IMPORT, EXPORT to and from hdfs paths - Refer 
> https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DML
> c. CREATE EXTERNAL TABLE - input is hdfs path and o/p is table
> d. ALTER TABLE LOCATION for an external table - input is the new hdfs path 
> and o/p is the table.
> 
> Also changed the ordering of model registration by sorting them by 
> modifiedTime to ensure they are registered in correct order
> 
> 
> Diffs
> -
> 
>   
> addons/hdfs-model/src/main/java/org/apache/atlas/fs/model/FSDataModelGenerator.java
>  555d565 
>   
> addons/hdfs-model/src/main/scala/org/apache/atlas/fs/model/FSDataModel.scala 
> c964f73 
>   addons/hive-bridge/pom.xml e125f18 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
>  3a802d7 
>   addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
> 68e32ff 
>   addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 
> e17afb8 
>   
> addons/storm-bridge/src/main/java/org/apache/atlas/storm/hook/StormAtlasHook.java
>  5665856 
>   
> repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java
>  0a04c5f 
>   
> repository/src/main/java/org/apache/atlas/services/ReservedTypesRegistrar.java
>  430bb6b 
> 
> Diff: https://reviews.apache.org/r/45784/diff/
> 
> 
> Testing
> ---
> 
> Added tests in HiveHookIT
> 
> 
> Thanks,
> 
> Suma Shivaprasad
> 
>

Re: Review Request 45784: Hve Hook - Support tracking lineage for External Tables( Create/alter) , Load, import, export

[Suma Shivaprasad]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45784/
---

(Updated April 6, 2016, 5:03 p.m.)


Review request for atlas.


Bugs: ATLAS-527
https://issues.apache.org/jira/browse/ATLAS-527


Repository: atlas


Description
---

Added support to track lineage between HDFS Paths and hive tables  in 

a. LOAD( at table, partition level) - input is a HDFS path and output is table( 
even though we dont create partition entities, we are still tracking the 
lineage at table level for partitions. This could be an issue if there are 
large number of partition queries which is not being addressed in this jira - 
https://issues.apache.org/jira/browse/ATLAS-619) . refer 
https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DML
b. IMPORT, EXPORT to and from hdfs paths - Refer 
https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DML
c. CREATE EXTERNAL TABLE - input is hdfs path and o/p is table
d. ALTER TABLE LOCATION for an external table - input is the new hdfs path and 
o/p is the table.

Also changed the ordering of model registration by sorting them by modifiedTime 
to ensure they are registered in correct order


Diffs (updated)
-

  
addons/hdfs-model/src/main/java/org/apache/atlas/fs/model/FSDataModelGenerator.java
 555d565 
  addons/hdfs-model/src/main/scala/org/apache/atlas/fs/model/FSDataModel.scala 
c964f73 
  
addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
 3a802d7 
  addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
68e32ff 
  addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 
e17afb8 
  
addons/storm-bridge/src/main/java/org/apache/atlas/storm/hook/StormAtlasHook.java
 5665856 
  client/src/main/java/org/apache/atlas/AtlasClient.java c3b4ba9 
  
repository/src/main/java/org/apache/atlas/services/ReservedTypesRegistrar.java 
430bb6b 

Diff: https://reviews.apache.org/r/45784/diff/


Testing
---

Added tests in HiveHookIT


Thanks,

Suma Shivaprasad

Re: Review Request 46011: ATLAS-622 Introduce soft delete

[Suma Shivaprasad]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46011/#review128270
---




addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 
(line 1219)
<https://reviews.apache.org/r/46011/#comment191689>

return false is missing



repository/src/main/java/org/apache/atlas/RepositoryMetadataModule.java (line 
90)
<https://reviews.apache.org/r/46011/#comment191688>

How are we planing to make this configurable? This is configurable at 
system or type level?



repository/src/main/java/org/apache/atlas/repository/graph/DeleteHandler.java 
(line 147)
<https://reviews.apache.org/r/46011/#comment191690>

Can we move this edge Label to a function in TYpedInstanceToGraphMapper so 
that there is one single place for all the CRUD ops.


- Suma Shivaprasad


On April 11, 2016, 8:55 a.m., Shwetha GS wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/46011/
> ---
> 
> (Updated April 11, 2016, 8:55 a.m.)
> 
> 
> Review request for atlas.
> 
> 
> Bugs: ATLAS-622
> https://issues.apache.org/jira/browse/ATLAS-622
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Added soft delete which is enabled by default. Hard delete can be configured 
> using guice. Had to refactor entity deletes. 
> 
> Pending:
> 1. 
> GraphBackedMetadataRepositoryDeleteEntitiesTest.testDisconnectUnidirectionalArrayReferenceFromStructAndTraitTypes
>  is disabled, need to debug
> 2. Have to write more tests for soft deletes
> 3. hive hook tests - have fixed some tests, 13/40 are still failing
> 
> 
> Diffs
> -
> 
>   
> addons/hdfs-model/src/main/scala/org/apache/atlas/fs/model/FSDataModel.scala 
> c380a92 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
>  6b348e2 
>   addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 
> 2f0c71f 
>   repository/src/main/java/org/apache/atlas/RepositoryMetadataModule.java 
> ce1bdfb 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/DeleteHandler.java 
> PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepository.java
>  b94ff5a 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java
>  e7e8fb9 
>   repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java 
> 895f9df 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/GraphToTypedInstanceMapper.java
>  1d682bb 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/HardDeleteHandler.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/SoftDeleteHandler.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/TypedInstanceToGraphMapper.java
>  7011fdf 
>   
> repository/src/main/java/org/apache/atlas/repository/typestore/GraphBackedTypeStore.java
>  71e9028 
>   repository/src/test/java/org/apache/atlas/BaseHiveRepositoryTest.java 
> d07f89d 
>   repository/src/test/java/org/apache/atlas/TestUtils.java a3cf929 
>   
> repository/src/test/java/org/apache/atlas/discovery/GraphBackedDiscoveryServiceTest.java
>  ea93cbf 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepositoryDeleteEntitiesTest.java
>  975d530 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepositoryTest.java
>  b5440c2 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/GraphRepoMapperScaleTest.java
>  3f71eb3 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/SoftDeleteHandlerTest.java
>  PRE-CREATION 
>   
> repository/src/test/java/org/apache/atlas/repository/typestore/GraphBackedTypeStoreTest.java
>  8ca3132 
>   
> repository/src/test/java/org/apache/atlas/service/DefaultMetadataServiceTest.java
>  156eb3d 
>   repository/src/test/scala/org/apache/atlas/query/GremlinTest.scala 0289b1a 
>   repository/src/test/scala/org/apache/atlas/query/GremlinTest2.scala 79ebfae 
>   repository/src/test/scala/org/apache/atlas/query/LineageQueryTest.scala 
> cd1c424 
>   server-api/src/main/java/org/apache/atlas/RequestContext.java 943e4b8 
>   typesystem/src/main/java/org/apache/atlas/typesystem/IInstance.java ffe40a7 
>   typesystem/src/main/java/org/apache/atlas/typesystem/Referenceable.java 
> 7857483 
>   typesystem/sr

Re: Review Request 45499: ATLAS-436, ATLAS-435 order by and limit clause for DSL queries

[Suma Shivaprasad]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45499/#review130247
---




docs/src/site/twiki/Search.twiki (line 31)
<https://reviews.apache.org/r/45499/#comment193970>

should be sortOrder


- Suma Shivaprasad


On April 20, 2016, 9:22 p.m., Neeru Gupta wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/45499/
> ---
> 
> (Updated April 20, 2016, 9:22 p.m.)
> 
> 
> Review request for atlas, David Kantor and Jeff Hagelberg.
> 
> 
> Bugs: ATLAS-435 and ATLAS-436
> https://issues.apache.org/jira/browse/ATLAS-435
> https://issues.apache.org/jira/browse/ATLAS-436
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> ATLAS-435, 436 order by and limit clause in DSL
> 
> 
> Diffs
> -
> 
>   docs/src/site/twiki/Search.twiki 3a1d00f0f21fa8ed70b8c5308f57049122078278 
>   repository/src/main/scala/org/apache/atlas/query/Expressions.scala 
> a5dfa9f409c0d810be7449413f5c2d53d5103dce 
>   repository/src/main/scala/org/apache/atlas/query/GremlinEvaluator.scala 
> edb190db599d43eb98eca13c7907c73e7d77ae34 
>   repository/src/main/scala/org/apache/atlas/query/GremlinQuery.scala 
> f1590a884eda007182dad7541a2bc8eb97d68fbc 
>   repository/src/main/scala/org/apache/atlas/query/QueryParser.scala 
> b6bbbd31a91dd2d0ff3e9082abafff3539765a01 
>   repository/src/main/scala/org/apache/atlas/query/Resolver.scala 
> c7e1e81a555d57be1d7f63cb84a1c8f550ac37ab 
>   
> repository/src/test/java/org/apache/atlas/discovery/GraphBackedDiscoveryServiceTest.java
>  ea93cbf0e6071a961aa5d030f0340358e79e446b 
> 
> Diff: https://reviews.apache.org/r/45499/diff/
> 
> 
> Testing
> ---
> 
> Manual and unit testing done for various DSL queries. Have added test cases 
> in GraphBackedDiscoveryServiceTest for orderby and limit clause for various 
> queries.
> 
> Order by clause is limited to specifying one criterian only. Comparison is 
> case insensitive.
> 
> 
> Thanks,
> 
> Neeru Gupta
> 
>

Re: Review Request 46731: ATLAS-716 Entity update/delete notifications

[Suma Shivaprasad]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46731/#review134305
---




repository/src/main/java/org/apache/atlas/repository/graph/DeleteHandler.java 
(line 312)
<https://reviews.apache.org/r/46731/#comment199116>

why is the check for isComposite removed? Can you pls explain?


- Suma Shivaprasad


On May 19, 2016, 5:41 a.m., Shwetha GS wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/46731/
> ---
> 
> (Updated May 19, 2016, 5:41 a.m.)
> 
> 
> Review request for atlas.
> 
> 
> Bugs: ATLAS-716
> https://issues.apache.org/jira/browse/ATLAS-716
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> 1. Update/delete entity APIs return list of entities created, updated and 
> deleted
> 2. Listeners are called for all the affected entities
> 
> 
> Diffs
> -
> 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
>  254e150 
>   addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 
> 70100f1 
>   client/src/main/java/org/apache/atlas/AtlasClient.java b3ec95c 
>   client/src/main/java/org/apache/atlas/EntityAuditEvent.java 460f708 
>   client/src/main/java/org/apache/atlas/SerDe.java PRE-CREATION 
>   client/src/test/java/org/apache/atlas/AtlasClientTest.java 0e80573 
>   
> notification/src/main/java/org/apache/atlas/notification/MessageVersion.java 
> 3f16a9a 
>   
> notification/src/main/java/org/apache/atlas/notification/NotificationInterface.java
>  384f383 
>   
> notification/src/main/java/org/apache/atlas/notification/VersionedMessageDeserializer.java
>  290be59 
>   
> notification/src/main/java/org/apache/atlas/notification/entity/NotificationEntityChangeListener.java
>  300cbb5 
>   
> notification/src/test/java/org/apache/atlas/notification/AbstractNotificationConsumerTest.java
>  e8b55ef 
>   
> repository/src/main/java/org/apache/atlas/repository/MetadataRepository.java 
> 540c308 
>   
> repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditListener.java
>  5b4bdbf 
>   
> repository/src/main/java/org/apache/atlas/repository/audit/HBaseBasedAuditRepository.java
>  8f11322 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/DeleteHandler.java 
> a9e4f39 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepository.java
>  3604277 
>   repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java 
> cccafc2 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/GraphToTypedInstanceMapper.java
>  df28ab3 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/HardDeleteHandler.java
>  f8bbf73 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/SoftDeleteHandler.java
>  aa78582 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/TypedInstanceToGraphMapper.java
>  a017536 
>   
> repository/src/main/java/org/apache/atlas/repository/typestore/GraphBackedTypeStore.java
>  3fb128c 
>   
> repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java
>  5195cbe 
>   
> repository/src/test/java/org/apache/atlas/repository/audit/AuditRepositoryTestBase.java
>  be407a5 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepositoryDeleteTestBase.java
>  1aeedb5 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedRepositoryHardDeleteTest.java
>  d2109d3 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedRepositorySoftDeleteTest.java
>  d9e3ec9 
>   
> repository/src/test/java/org/apache/atlas/service/DefaultMetadataServiceTest.java
>  1f906ed 
>   server-api/src/main/java/org/apache/atlas/RequestContext.java b1d87ea 
>   server-api/src/main/java/org/apache/atlas/services/MetadataService.java 
> 13d20d8 
>   typesystem/src/main/java/org/apache/atlas/typesystem/Referenceable.java 
> 31f157e 
>   webapp/src/main/java/org/apache/atlas/LocalAtlasClient.java c6ed85d 
>   
> webapp/src/main/java/org/apache/atlas/notification/NotificationEntityChangeListener.java
>  PRE-CREATION 
>   webapp/src/main/java/org/apache/atlas/web/resources/EntityResource.java 
> 709fec5 
>   webapp/src/main/java/org/apache/atlas/web/service/ServiceModule.java 
> 0f8bcb1 
>   webapp/src/test/java/org/apache/atlas/LocalAtlasClientTest.java 7f2065

Re: Review Request 46731: ATLAS-716 Entity update/delete notifications

[Suma Shivaprasad]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46731/#review134304
---




repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java 
(line 427)
<https://reviews.apache.org/r/46731/#comment199115>

if this needs to be fixed, pls raise a jira


- Suma Shivaprasad


On May 19, 2016, 5:41 a.m., Shwetha GS wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/46731/
> ---
> 
> (Updated May 19, 2016, 5:41 a.m.)
> 
> 
> Review request for atlas.
> 
> 
> Bugs: ATLAS-716
> https://issues.apache.org/jira/browse/ATLAS-716
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> 1. Update/delete entity APIs return list of entities created, updated and 
> deleted
> 2. Listeners are called for all the affected entities
> 
> 
> Diffs
> -
> 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
>  254e150 
>   addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 
> 70100f1 
>   client/src/main/java/org/apache/atlas/AtlasClient.java b3ec95c 
>   client/src/main/java/org/apache/atlas/EntityAuditEvent.java 460f708 
>   client/src/main/java/org/apache/atlas/SerDe.java PRE-CREATION 
>   client/src/test/java/org/apache/atlas/AtlasClientTest.java 0e80573 
>   
> notification/src/main/java/org/apache/atlas/notification/MessageVersion.java 
> 3f16a9a 
>   
> notification/src/main/java/org/apache/atlas/notification/NotificationInterface.java
>  384f383 
>   
> notification/src/main/java/org/apache/atlas/notification/VersionedMessageDeserializer.java
>  290be59 
>   
> notification/src/main/java/org/apache/atlas/notification/entity/NotificationEntityChangeListener.java
>  300cbb5 
>   
> notification/src/test/java/org/apache/atlas/notification/AbstractNotificationConsumerTest.java
>  e8b55ef 
>   
> repository/src/main/java/org/apache/atlas/repository/MetadataRepository.java 
> 540c308 
>   
> repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditListener.java
>  5b4bdbf 
>   
> repository/src/main/java/org/apache/atlas/repository/audit/HBaseBasedAuditRepository.java
>  8f11322 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/DeleteHandler.java 
> a9e4f39 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepository.java
>  3604277 
>   repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java 
> cccafc2 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/GraphToTypedInstanceMapper.java
>  df28ab3 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/HardDeleteHandler.java
>  f8bbf73 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/SoftDeleteHandler.java
>  aa78582 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/TypedInstanceToGraphMapper.java
>  a017536 
>   
> repository/src/main/java/org/apache/atlas/repository/typestore/GraphBackedTypeStore.java
>  3fb128c 
>   
> repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java
>  5195cbe 
>   
> repository/src/test/java/org/apache/atlas/repository/audit/AuditRepositoryTestBase.java
>  be407a5 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepositoryDeleteTestBase.java
>  1aeedb5 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedRepositoryHardDeleteTest.java
>  d2109d3 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedRepositorySoftDeleteTest.java
>  d9e3ec9 
>   
> repository/src/test/java/org/apache/atlas/service/DefaultMetadataServiceTest.java
>  1f906ed 
>   server-api/src/main/java/org/apache/atlas/RequestContext.java b1d87ea 
>   server-api/src/main/java/org/apache/atlas/services/MetadataService.java 
> 13d20d8 
>   typesystem/src/main/java/org/apache/atlas/typesystem/Referenceable.java 
> 31f157e 
>   webapp/src/main/java/org/apache/atlas/LocalAtlasClient.java c6ed85d 
>   
> webapp/src/main/java/org/apache/atlas/notification/NotificationEntityChangeListener.java
>  PRE-CREATION 
>   webapp/src/main/java/org/apache/atlas/web/resources/EntityResource.java 
> 709fec5 
>   webapp/src/main/java/org/apache/atlas/web/service/ServiceModule.java 
> 0f8bcb1 
>   webapp/src/test/java/org/apache/atlas/LocalAtlasClientTest.java 7f20652 
> 

Re: Review Request 46731: ATLAS-716 Entity update/delete notifications

[Suma Shivaprasad]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46731/#review134307
---




webapp/src/test/java/org/apache/atlas/web/resources/EntityJerseyResourceIT.java 
(line 74)
<https://reviews.apache.org/r/46731/#comment199118>

pls add asserts for update test cases as well for updated entities


- Suma Shivaprasad


On May 19, 2016, 5:41 a.m., Shwetha GS wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/46731/
> ---
> 
> (Updated May 19, 2016, 5:41 a.m.)
> 
> 
> Review request for atlas.
> 
> 
> Bugs: ATLAS-716
> https://issues.apache.org/jira/browse/ATLAS-716
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> 1. Update/delete entity APIs return list of entities created, updated and 
> deleted
> 2. Listeners are called for all the affected entities
> 
> 
> Diffs
> -
> 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
>  254e150 
>   addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 
> 70100f1 
>   client/src/main/java/org/apache/atlas/AtlasClient.java b3ec95c 
>   client/src/main/java/org/apache/atlas/EntityAuditEvent.java 460f708 
>   client/src/main/java/org/apache/atlas/SerDe.java PRE-CREATION 
>   client/src/test/java/org/apache/atlas/AtlasClientTest.java 0e80573 
>   
> notification/src/main/java/org/apache/atlas/notification/MessageVersion.java 
> 3f16a9a 
>   
> notification/src/main/java/org/apache/atlas/notification/NotificationInterface.java
>  384f383 
>   
> notification/src/main/java/org/apache/atlas/notification/VersionedMessageDeserializer.java
>  290be59 
>   
> notification/src/main/java/org/apache/atlas/notification/entity/NotificationEntityChangeListener.java
>  300cbb5 
>   
> notification/src/test/java/org/apache/atlas/notification/AbstractNotificationConsumerTest.java
>  e8b55ef 
>   
> repository/src/main/java/org/apache/atlas/repository/MetadataRepository.java 
> 540c308 
>   
> repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditListener.java
>  5b4bdbf 
>   
> repository/src/main/java/org/apache/atlas/repository/audit/HBaseBasedAuditRepository.java
>  8f11322 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/DeleteHandler.java 
> a9e4f39 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepository.java
>  3604277 
>   repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java 
> cccafc2 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/GraphToTypedInstanceMapper.java
>  df28ab3 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/HardDeleteHandler.java
>  f8bbf73 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/SoftDeleteHandler.java
>  aa78582 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/TypedInstanceToGraphMapper.java
>  a017536 
>   
> repository/src/main/java/org/apache/atlas/repository/typestore/GraphBackedTypeStore.java
>  3fb128c 
>   
> repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java
>  5195cbe 
>   
> repository/src/test/java/org/apache/atlas/repository/audit/AuditRepositoryTestBase.java
>  be407a5 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepositoryDeleteTestBase.java
>  1aeedb5 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedRepositoryHardDeleteTest.java
>  d2109d3 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedRepositorySoftDeleteTest.java
>  d9e3ec9 
>   
> repository/src/test/java/org/apache/atlas/service/DefaultMetadataServiceTest.java
>  1f906ed 
>   server-api/src/main/java/org/apache/atlas/RequestContext.java b1d87ea 
>   server-api/src/main/java/org/apache/atlas/services/MetadataService.java 
> 13d20d8 
>   typesystem/src/main/java/org/apache/atlas/typesystem/Referenceable.java 
> 31f157e 
>   webapp/src/main/java/org/apache/atlas/LocalAtlasClient.java c6ed85d 
>   
> webapp/src/main/java/org/apache/atlas/notification/NotificationEntityChangeListener.java
>  PRE-CREATION 
>   webapp/src/main/java/org/apache/atlas/web/resources/EntityResource.java 
> 709fec5 
>   webapp/src/main/java/org/apache/atlas/web/service/ServiceModule.java 
> 0f8bcb1 
>   webapp/src/test/java/org/apache/atlas/LocalAtlasClientTest.java 7f

Re: Review Request 46731: ATLAS-716 Entity update/delete notifications

[Suma Shivaprasad]

> On May 23, 2016, 5:02 a.m., Suma Shivaprasad wrote:
> > repository/src/main/java/org/apache/atlas/repository/graph/DeleteHandler.java,
> >  line 318
> > <https://reviews.apache.org/r/46731/diff/3/?file=1387706#file1387706line318>
> >
> > why is the check for isComposite removed? Can you pls explain?
> 
> Shwetha GS wrote:
> For example, in case of soft delete, if the column is deleted, we don't 
> update the table's columns attribute. So, table definition returns the list 
> of columns that includes deleted columns as well. 
> 
> For hive_process, when table is deleted, the reference is not removed. To 
> be consistent, for hive_table, when column is deleted, reference is not 
> removed as well

Wouldnt this be an issue in case of hard deletes?


- Suma


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46731/#review134305
---


On May 19, 2016, 5:41 a.m., Shwetha GS wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/46731/
> ---
> 
> (Updated May 19, 2016, 5:41 a.m.)
> 
> 
> Review request for atlas.
> 
> 
> Bugs: ATLAS-716
> https://issues.apache.org/jira/browse/ATLAS-716
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> 1. Update/delete entity APIs return list of entities created, updated and 
> deleted
> 2. Listeners are called for all the affected entities
> 
> 
> Diffs
> -
> 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
>  254e150 
>   addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 
> 70100f1 
>   client/src/main/java/org/apache/atlas/AtlasClient.java b3ec95c 
>   client/src/main/java/org/apache/atlas/EntityAuditEvent.java 460f708 
>   client/src/main/java/org/apache/atlas/SerDe.java PRE-CREATION 
>   client/src/test/java/org/apache/atlas/AtlasClientTest.java 0e80573 
>   
> notification/src/main/java/org/apache/atlas/notification/MessageVersion.java 
> 3f16a9a 
>   
> notification/src/main/java/org/apache/atlas/notification/NotificationInterface.java
>  384f383 
>   
> notification/src/main/java/org/apache/atlas/notification/VersionedMessageDeserializer.java
>  290be59 
>   
> notification/src/main/java/org/apache/atlas/notification/entity/NotificationEntityChangeListener.java
>  300cbb5 
>   
> notification/src/test/java/org/apache/atlas/notification/AbstractNotificationConsumerTest.java
>  e8b55ef 
>   
> repository/src/main/java/org/apache/atlas/repository/MetadataRepository.java 
> 540c308 
>   
> repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditListener.java
>  5b4bdbf 
>   
> repository/src/main/java/org/apache/atlas/repository/audit/HBaseBasedAuditRepository.java
>  8f11322 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/DeleteHandler.java 
> a9e4f39 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepository.java
>  3604277 
>   repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java 
> cccafc2 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/GraphToTypedInstanceMapper.java
>  df28ab3 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/HardDeleteHandler.java
>  f8bbf73 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/SoftDeleteHandler.java
>  aa78582 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/TypedInstanceToGraphMapper.java
>  a017536 
>   
> repository/src/main/java/org/apache/atlas/repository/typestore/GraphBackedTypeStore.java
>  3fb128c 
>   
> repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java
>  5195cbe 
>   
> repository/src/test/java/org/apache/atlas/repository/audit/AuditRepositoryTestBase.java
>  be407a5 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepositoryDeleteTestBase.java
>  1aeedb5 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedRepositoryHardDeleteTest.java
>  d2109d3 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedRepositorySoftDeleteTest.java
>  d9e3ec9 
>   
> repository/src/test/java/org/apache/atlas/service/DefaultMetadataServiceTest.java
>  1f906ed 
>   server-api/src/main/java/org/apache/atlas/RequestContext.java b1d87ea 
>   server-api/src/main/java/org/

Re: Review Request 47638: ATLAS-752 Column rename in Hive should retain traits/tags associated with it

[Suma Shivaprasad]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47638/#review134157
---




addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java (line 
311)
<https://reviews.apache.org/r/47638/#comment198814>

pls change var name from nameChanged to nameNotModified.


- Suma Shivaprasad


On May 20, 2016, 6:42 a.m., Vimal Sharma wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47638/
> ---
> 
> (Updated May 20, 2016, 6:42 a.m.)
> 
> 
> Review request for atlas.
> 
> 
> Bugs: ATLAS-752
> https://issues.apache.org/jira/browse/ATLAS-752
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> ATLAS-752 When a column is renamed, the traits associted with it are lost. 
> The traits should be retained with the new column
> 
> 
> Diffs
> -
> 
>   addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
> 418e755 
>   addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 
> 70100f1 
> 
> Diff: https://reviews.apache.org/r/47638/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Vimal Sharma
> 
>

Re: Review Request 46922: Temporary tables should be ignored during table creation

[Suma Shivaprasad]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46922/
---

(Updated May 9, 2016, 6:17 a.m.)


Review request for atlas.


Bugs: ATLAS-626
https://issues.apache.org/jira/browse/ATLAS-626


Repository: atlas


Description
---

Capturing temp table info only when lineage is involved.


Diffs (updated)
-

  
addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
 d4212a1 
  addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
813177f 
  
addons/hive-bridge/src/test/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridgeTest.java
 dac5a18 
  addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 
317d636 

Diff: https://reviews.apache.org/r/46922/diff/


Testing
---

Added tests in HIveHookIT.testInsertIntoTempTable.


Thanks,

Suma Shivaprasad

Re: Review Request 47111: ATLAS-738 Add query ability on system properties like guid, state, createdtime etc

[Suma Shivaprasad]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47111/#review132328
---




repository/src/main/java/org/apache/atlas/repository/Constants.java (line 70)
<https://reviews.apache.org/r/47111/#comment196509>

rename to getInternalAttribute?


- Suma Shivaprasad


On May 9, 2016, 8:02 a.m., Shwetha GS wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47111/
> ---
> 
> (Updated May 9, 2016, 8:02 a.m.)
> 
> 
> Review request for atlas.
> 
> 
> Bugs: ATLAS-738
> https://issues.apache.org/jira/browse/ATLAS-738
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Add query ability on system properties like guid, state, createdtime etc
> 
> 
> Diffs
> -
> 
>   repository/src/main/java/org/apache/atlas/repository/Constants.java 0832c81 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepository.java
>  de29e86 
>   repository/src/main/scala/org/apache/atlas/query/TypeUtils.scala 552cc32 
>   
> repository/src/test/java/org/apache/atlas/discovery/GraphBackedDiscoveryServiceTest.java
>  9cc501f 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/AttributeInfo.java 
> 330d1cb 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/utils/TypesUtil.java
>  2ea90a3 
> 
> Diff: https://reviews.apache.org/r/47111/diff/
> 
> 
> Testing
> ---
> 
> UTs
> 
> 
> Thanks,
> 
> Shwetha GS
> 
>

Re: Review Request 47111: ATLAS-738 Add query ability on system properties like guid, state, createdtime etc

[Suma Shivaprasad]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47111/#review132329
---




repository/src/main/java/org/apache/atlas/repository/Constants.java (line 69)
<https://reviews.apache.org/r/47111/#comment196510>

move this as well to TypesUtil?


- Suma Shivaprasad


On May 9, 2016, 8:02 a.m., Shwetha GS wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47111/
> ---
> 
> (Updated May 9, 2016, 8:02 a.m.)
> 
> 
> Review request for atlas.
> 
> 
> Bugs: ATLAS-738
> https://issues.apache.org/jira/browse/ATLAS-738
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Add query ability on system properties like guid, state, createdtime etc
> 
> 
> Diffs
> -
> 
>   repository/src/main/java/org/apache/atlas/repository/Constants.java 0832c81 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepository.java
>  de29e86 
>   repository/src/main/scala/org/apache/atlas/query/TypeUtils.scala 552cc32 
>   
> repository/src/test/java/org/apache/atlas/discovery/GraphBackedDiscoveryServiceTest.java
>  9cc501f 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/AttributeInfo.java 
> 330d1cb 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/utils/TypesUtil.java
>  2ea90a3 
> 
> Diff: https://reviews.apache.org/r/47111/diff/
> 
> 
> Testing
> ---
> 
> UTs
> 
> 
> Thanks,
> 
> Shwetha GS
> 
>

Re: Review Request 47111: ATLAS-738 Add query ability on system properties like guid, state, createdtime etc

[Suma Shivaprasad]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47111/#review132331
---


Ship it!




Ship It!

- Suma Shivaprasad


On May 9, 2016, 8:02 a.m., Shwetha GS wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47111/
> ---
> 
> (Updated May 9, 2016, 8:02 a.m.)
> 
> 
> Review request for atlas.
> 
> 
> Bugs: ATLAS-738
> https://issues.apache.org/jira/browse/ATLAS-738
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Add query ability on system properties like guid, state, createdtime etc
> 
> 
> Diffs
> -
> 
>   repository/src/main/java/org/apache/atlas/repository/Constants.java 0832c81 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepository.java
>  de29e86 
>   repository/src/main/scala/org/apache/atlas/query/TypeUtils.scala 552cc32 
>   
> repository/src/test/java/org/apache/atlas/discovery/GraphBackedDiscoveryServiceTest.java
>  9cc501f 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/AttributeInfo.java 
> 330d1cb 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/utils/TypesUtil.java
>  2ea90a3 
> 
> Diff: https://reviews.apache.org/r/47111/diff/
> 
> 
> Testing
> ---
> 
> UTs
> 
> 
> Thanks,
> 
> Shwetha GS
> 
>

Re: Review Request 46731: ATLAS-716 Entity update/delete notifications

[Suma Shivaprasad]

> On May 5, 2016, 9:56 p.m., Suma Shivaprasad wrote:
> > client/src/main/java/org/apache/atlas/AtlasClient.java, line 579
> > <https://reviews.apache.org/r/46731/diff/2/?file=1363301#file1363301line579>
> >
> > would be better to keep the result consistent for all CRUD operations 
> > and standardize as EntityResult

The reason I thought its better is we could add optional attributes the json 
after this and this is more flexible than changing the API later on. Makes 
sense?


- Suma


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46731/#review131952
---


On April 27, 2016, 1:05 p.m., Shwetha GS wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/46731/
> ---
> 
> (Updated April 27, 2016, 1:05 p.m.)
> 
> 
> Review request for atlas.
> 
> 
> Bugs: ATLAS-716
> https://issues.apache.org/jira/browse/ATLAS-716
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> 1. Update/delete entity APIs return list of entities created, updated and 
> deleted
> 2. Listeners are called for all the affected entities
> 
> 
> Diffs
> -
> 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
>  104c0c5 
>   addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
> 813177f 
>   
> addons/sqoop-bridge/src/main/java/org/apache/atlas/sqoop/hook/SqoopHook.java 
> ab7e6ee 
>   client/src/main/java/org/apache/atlas/AtlasClient.java be34802 
>   client/src/main/java/org/apache/atlas/EntityAuditEvent.java 460f708 
>   client/src/main/java/org/apache/atlas/SerDe.java PRE-CREATION 
>   client/src/main/java/org/apache/atlas/notification/EntityNotification.java 
> PRE-CREATION 
>   
> client/src/main/java/org/apache/atlas/notification/EntityNotificationImpl.java
>  PRE-CREATION 
>   client/src/main/java/org/apache/atlas/notification/HookNotification.java 
> PRE-CREATION 
>   client/src/test/java/org/apache/atlas/AtlasClientTest.java 8911bf5 
>   
> client/src/test/java/org/apache/atlas/notification/EntityNotificationImplTest.java
>  PRE-CREATION 
>   
> client/src/test/java/org/apache/atlas/notification/HookNotificationTest.java 
> PRE-CREATION 
>   notification/src/main/java/org/apache/atlas/hook/AtlasHook.java 68a868f 
>   
> notification/src/main/java/org/apache/atlas/notification/AbstractNotification.java
>  596f988 
>   
> notification/src/main/java/org/apache/atlas/notification/AbstractNotificationConsumer.java
>  1cadb99 
>   
> notification/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java
>  1f2df3e 
>   
> notification/src/main/java/org/apache/atlas/notification/NotificationInterface.java
>  ac285aa 
>   
> notification/src/main/java/org/apache/atlas/notification/entity/EntityNotification.java
>  82a1100 
>   
> notification/src/main/java/org/apache/atlas/notification/entity/EntityNotificationImpl.java
>  fda588e 
>   
> notification/src/main/java/org/apache/atlas/notification/entity/NotificationEntityChangeListener.java
>  300cbb5 
>   
> notification/src/main/java/org/apache/atlas/notification/hook/HookNotification.java
>  80c96fa 
>   
> notification/src/test/java/org/apache/atlas/notification/NotificationEntityChangeListenerTest.java
>  PRE-CREATION 
>   
> notification/src/test/java/org/apache/atlas/notification/entity/EntityNotificationImplTest.java
>  385c41f 
>   
> notification/src/test/java/org/apache/atlas/notification/hook/HookNotificationTest.java
>  57b0eea 
>   pom.xml 729b178 
>   repository/pom.xml 533e48b 
>   
> repository/src/main/java/org/apache/atlas/repository/MetadataRepository.java 
> 540c308 
>   
> repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditListener.java
>  5b4bdbf 
>   
> repository/src/main/java/org/apache/atlas/repository/audit/HBaseBasedAuditRepository.java
>  8f11322 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/DeleteHandler.java 
> 369a5d5 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepository.java
>  de29e86 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/TypedInstanceToGraphMapper.java
>  a3dc7e5 
>   
> repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java
>  9f69940 
>   
> repository/src/test/java/org/apache/atlas/repository/audit/AuditRepositoryTestBase.java
&g

Re: Review Request 46731: ATLAS-716 Entity update/delete notifications

[Suma Shivaprasad]

> On May 5, 2016, 10:05 p.m., Suma Shivaprasad wrote:
> > client/src/main/java/org/apache/atlas/EntityAuditEvent.java, line 66
> > <https://reviews.apache.org/r/46731/diff/2/?file=1363302#file1363302line66>
> >
> > why do  we need String comparision here and not direct quals which 
> > would be faster?

any reason to still keep this as toString comparision instead of direct ones? 
Can you pls explain?


- Suma


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46731/#review131956
---


On April 27, 2016, 1:05 p.m., Shwetha GS wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/46731/
> ---
> 
> (Updated April 27, 2016, 1:05 p.m.)
> 
> 
> Review request for atlas.
> 
> 
> Bugs: ATLAS-716
> https://issues.apache.org/jira/browse/ATLAS-716
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> 1. Update/delete entity APIs return list of entities created, updated and 
> deleted
> 2. Listeners are called for all the affected entities
> 
> 
> Diffs
> -
> 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java
>  104c0c5 
>   addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
> 813177f 
>   
> addons/sqoop-bridge/src/main/java/org/apache/atlas/sqoop/hook/SqoopHook.java 
> ab7e6ee 
>   client/src/main/java/org/apache/atlas/AtlasClient.java be34802 
>   client/src/main/java/org/apache/atlas/EntityAuditEvent.java 460f708 
>   client/src/main/java/org/apache/atlas/SerDe.java PRE-CREATION 
>   client/src/main/java/org/apache/atlas/notification/EntityNotification.java 
> PRE-CREATION 
>   
> client/src/main/java/org/apache/atlas/notification/EntityNotificationImpl.java
>  PRE-CREATION 
>   client/src/main/java/org/apache/atlas/notification/HookNotification.java 
> PRE-CREATION 
>   client/src/test/java/org/apache/atlas/AtlasClientTest.java 8911bf5 
>   
> client/src/test/java/org/apache/atlas/notification/EntityNotificationImplTest.java
>  PRE-CREATION 
>   
> client/src/test/java/org/apache/atlas/notification/HookNotificationTest.java 
> PRE-CREATION 
>   notification/src/main/java/org/apache/atlas/hook/AtlasHook.java 68a868f 
>   
> notification/src/main/java/org/apache/atlas/notification/AbstractNotification.java
>  596f988 
>   
> notification/src/main/java/org/apache/atlas/notification/AbstractNotificationConsumer.java
>  1cadb99 
>   
> notification/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java
>  1f2df3e 
>   
> notification/src/main/java/org/apache/atlas/notification/NotificationInterface.java
>  ac285aa 
>   
> notification/src/main/java/org/apache/atlas/notification/entity/EntityNotification.java
>  82a1100 
>   
> notification/src/main/java/org/apache/atlas/notification/entity/EntityNotificationImpl.java
>  fda588e 
>   
> notification/src/main/java/org/apache/atlas/notification/entity/NotificationEntityChangeListener.java
>  300cbb5 
>   
> notification/src/main/java/org/apache/atlas/notification/hook/HookNotification.java
>  80c96fa 
>   
> notification/src/test/java/org/apache/atlas/notification/NotificationEntityChangeListenerTest.java
>  PRE-CREATION 
>   
> notification/src/test/java/org/apache/atlas/notification/entity/EntityNotificationImplTest.java
>  385c41f 
>   
> notification/src/test/java/org/apache/atlas/notification/hook/HookNotificationTest.java
>  57b0eea 
>   pom.xml 729b178 
>   repository/pom.xml 533e48b 
>   
> repository/src/main/java/org/apache/atlas/repository/MetadataRepository.java 
> 540c308 
>   
> repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditListener.java
>  5b4bdbf 
>   
> repository/src/main/java/org/apache/atlas/repository/audit/HBaseBasedAuditRepository.java
>  8f11322 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/DeleteHandler.java 
> 369a5d5 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepository.java
>  de29e86 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/TypedInstanceToGraphMapper.java
>  a3dc7e5 
>   
> repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java
>  9f69940 
>   
> repository/src/test/java/org/apache/atlas/repository/audit/AuditRepositoryTestBase.java
>  be407a5 
>   
> repository/src/test/java/org/apache/atlas/reposit

Re: Review Request 47135: Install Setup: SOLR

[Suma Shivaprasad]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47135/#review132508
---




distro/src/bin/atlas_config.py (line 102)
<https://reviews.apache.org/r/47135/#comment196727>

if this config is changed by a user through Ambari, we need to handle the 
case where we need to support resharding after confirming from the user. I am 
not sure if this is something that can be done instantaneously or requires down 
time. Can we dig deeper and see how we can handle this. The same goes for 
replica


- Suma Shivaprasad


On May 9, 2016, 9:32 p.m., Tom Beerbower wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47135/
> ---
> 
> (Updated May 9, 2016, 9:32 p.m.)
> 
> 
> Review request for atlas.
> 
> 
> Bugs: ATLAS-663 and ATLAS-673
> https://issues.apache.org/jira/browse/ATLAS-663
> https://issues.apache.org/jira/browse/ATLAS-673
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Install standalone SOLR by default.
> 
> SOLR indexes need to be setup properly.
> 
> Fix location of standalone solr and hbase data dirs.
> 
> 
> Diffs
> -
> 
>   .gitignore b5a1c74 
>   distro/pom.xml ee9add0 
>   distro/src/bin/atlas_config.py dbe1191 
>   distro/src/bin/atlas_start.py 73b8920 
>   distro/src/bin/atlas_stop.py 265084e 
>   distro/src/conf/atlas-application.properties 290105f 
>   distro/src/conf/hbase/hbase-site.xml.template f5c4aad 
>   distro/src/main/assemblies/standalone-package.xml 527f604 
>   distro/src/test/python/scripts/TestMetadata.py 9e55869 
>   docs/src/site/twiki/Configuration.twiki 0793c82 
>   docs/src/site/twiki/InstallationSteps.twiki 270d1c4 
> 
> Diff: https://reviews.apache.org/r/47135/diff/
> 
> 
> Testing
> ---
> 
> Manual testing of atlas_start / atlas_start with standalone solr.
> 
> mvn clean test
> 
> 
> Thanks,
> 
> Tom Beerbower
> 
>

Re: Review Request 46905: ATLAS-667: Enforce lower bound when disconnecting references during entity deletion

[Suma Shivaprasad]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46905/#review132559
---




repository/src/main/java/org/apache/atlas/repository/graph/DeleteHandler.java 
(line 284)
<https://reviews.apache.org/r/46905/#comment196773>

the same case should be taken care of for maps?


- Suma Shivaprasad


On May 3, 2016, 8:26 p.m., David Kantor wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/46905/
> ---
> 
> (Updated May 3, 2016, 8:26 p.m.)
> 
> 
> Review request for atlas.
> 
> 
> Bugs: ATLAS-667
> https://issues.apache.org/jira/browse/ATLAS-667
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> ATLAS-667: Enforce lower bound on references when deleting entities.  If the 
> deletion candidate is the target of a reference with a non-zero lower bound, 
> disallow the delete if disconnecting the reference would reduce the reference 
> cardinality to zero.
> 
> 
> Diffs
> -
> 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/DeleteHandler.java 
> 369a5d58147f59773c25064375f767adf6b7 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepositoryDeleteTestBase.java
>  ae215f99080047c6fe45b55ae0c5ad7c1ba600e2 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedRepositoryHardDeleteTest.java
>  8d2961ef5f9bae5ff40b03a9c00d5c1c51bf384b 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedRepositorySoftDeleteTest.java
>  5c59c8a7be68987738b23168ba92843cf229d61d 
>   
> server-api/src/main/java/org/apache/atlas/typesystem/exception/NullRequiredAttributeException.java
>  PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/46905/diff/
> 
> 
> Testing
> ---
> 
> Ran all unit and integration tests with no regressions.
> 
> 
> Thanks,
> 
> David Kantor
> 
>

Re: Review Request 47107: ATLAS_619 Normalize literals for hive queries

[Suma Shivaprasad]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47107/
---

(Updated May 12, 2016, 7:01 a.m.)


Review request for atlas.


Changes
---

Fixed lineage for hdfs paths


Bugs: ATLAS-619
https://issues.apache.org/jira/browse/ATLAS-619


Repository: atlas


Description
---

Normlized all literals in a query and replacing them CONSTANT literals. Storing 
recently run queries in  a new attribute for hive_process - recentQueries


Diffs (updated)
-

  
addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/hook/FalconHook.java 
8fced05 
  
addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/model/FalconDataModelGenerator.java
 2494675 
  
addons/falcon-bridge/src/test/java/org/apache/atlas/falcon/hook/FalconHookIT.java
 4e2a06f 
  addons/hive-bridge/pom.xml eeb2aa4 
  addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
ffd5d22 
  
addons/hive-bridge/src/main/java/org/apache/atlas/hive/model/HiveDataModelGenerator.java
 6418770 
  
addons/hive-bridge/src/main/java/org/apache/atlas/hive/rewrite/ASTRewriter.java 
PRE-CREATION 
  
addons/hive-bridge/src/main/java/org/apache/atlas/hive/rewrite/HiveASTRewriter.java
 PRE-CREATION 
  
addons/hive-bridge/src/main/java/org/apache/atlas/hive/rewrite/LiteralRewriter.java
 PRE-CREATION 
  
addons/hive-bridge/src/main/java/org/apache/atlas/hive/rewrite/RewriteContext.java
 PRE-CREATION 
  
addons/hive-bridge/src/main/java/org/apache/atlas/hive/rewrite/RewriteException.java
 PRE-CREATION 
  
addons/hive-bridge/src/test/java/org/apache/atlas/hive/bridge/HiveLiteralRewriterTest.java
 PRE-CREATION 
  addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 
da21195 
  addons/hive-bridge/src/test/resources/hive-site.xml f1facb8 
  addons/sqoop-bridge/src/main/java/org/apache/atlas/sqoop/hook/SqoopHook.java 
ab7e6ee 
  
addons/sqoop-bridge/src/test/java/org/apache/atlas/sqoop/hook/SqoopHookIT.java 
2820169 
  
addons/storm-bridge/src/main/java/org/apache/atlas/storm/hook/StormAtlasHook.java
 267e228 
  
addons/storm-bridge/src/main/scala/org/apache/atlas/storm/model/StormDataModel.scala
 de67c39 
  
repository/src/main/java/org/apache/atlas/repository/graph/TypedInstanceToGraphMapper.java
 a017536 
  
repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java 
9f69940 
  repository/src/test/java/org/apache/atlas/BaseHiveRepositoryTest.java 66e1365 
  
typesystem/src/main/scala/org/apache/atlas/typesystem/json/InstanceSerialization.scala
 73b3526 
  webapp/src/main/java/org/apache/atlas/examples/QuickStart.java 70dce6b 
  webapp/src/test/java/org/apache/atlas/examples/QuickStartIT.java cdf6049 
  
webapp/src/test/java/org/apache/atlas/web/resources/HiveLineageJerseyResourceIT.java
 bc02f90 

Diff: https://reviews.apache.org/r/47107/diff/


Testing
---

Added UT - HiveAStREwriteTest and IT asserttions in HiveHookIT


Thanks,

Suma Shivaprasad

Re: Review Request 46905: ATLAS-667: Enforce lower bound when disconnecting references during entity deletion

[Suma Shivaprasad]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46905/#review132979
---


Ship it!




Ship It!

- Suma Shivaprasad


On May 11, 2016, 11:15 p.m., David Kantor wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/46905/
> ---
> 
> (Updated May 11, 2016, 11:15 p.m.)
> 
> 
> Review request for atlas.
> 
> 
> Bugs: ATLAS-667
> https://issues.apache.org/jira/browse/ATLAS-667
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> ATLAS-667: Enforce lower bound on references when deleting entities.  If the 
> deletion candidate is the target of a reference with a non-zero lower bound, 
> disallow the delete if disconnecting the reference would reduce the reference 
> cardinality to zero.
> 
> 
> Diffs
> -
> 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/DeleteHandler.java 
> 369a5d58147f59773c25064375f767adf6b7 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepositoryDeleteTestBase.java
>  ae215f99080047c6fe45b55ae0c5ad7c1ba600e2 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedRepositoryHardDeleteTest.java
>  8d2961ef5f9bae5ff40b03a9c00d5c1c51bf384b 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedRepositorySoftDeleteTest.java
>  5c59c8a7be68987738b23168ba92843cf229d61d 
>   server-api/src/main/java/org/apache/atlas/RequestContext.java 
> fa94763c6f298df6b66d392e909bdbf5323fb06c 
>   
> server-api/src/main/java/org/apache/atlas/typesystem/exception/NullRequiredAttributeException.java
>  PRE-CREATION 
>   
> webapp/src/test/java/org/apache/atlas/notification/EntityNotificationIT.java 
> d6199ab09049a4609d57dbd1cab3a716184ff52c 
> 
> Diff: https://reviews.apache.org/r/46905/diff/
> 
> 
> Testing
> ---
> 
> Ran all unit and integration tests with no regressions.
> 
> 
> Thanks,
> 
> David Kantor
> 
>

Re: Review Request 47225: ATLAS-713 Atlas UI is still showing the deleted column of a hive table (Entity response of hive table does not have it). - with browser cache disabled.

[Suma Shivaprasad]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47225/#review132977
---




repository/src/main/java/org/apache/atlas/discovery/HiveLineageService.java 
(lines 65 - 66)
<https://reviews.apache.org/r/47225/#comment197270>

while we are anyways doing this, should we rename HiveLineageService to 
DataSetLineageService


- Suma Shivaprasad


On May 11, 2016, 9:21 a.m., Shwetha GS wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47225/
> ---
> 
> (Updated May 11, 2016, 9:21 a.m.)
> 
> 
> Review request for atlas.
> 
> 
> Bugs: ATLAS-713
> https://issues.apache.org/jira/browse/ATLAS-713
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Current lineage APIs work on the dataset name. With soft delete, there can be 
> more than one entities for the given dataset name. 
> 1. Added new lineage APIs(input graph, output graph, schema) which work on 
> entity id
> 2. Modified current lineage APIs to work on the active entity for the given 
> dataset name
> 3. Modified UI to use the new APIs
> 
> 
> Diffs
> -
> 
>   client/src/main/java/org/apache/atlas/AtlasClient.java b3ec95c 
>   dashboardv2/public/js/models/VLineage.js e33488a 
>   dashboardv2/public/js/models/VSchema.js 1f8e0bb 
>   dashboardv2/public/js/views/detail_page/DetailPageLayoutView.js 87adec0 
>   dashboardv2/public/js/views/graph/LineageLayoutView.js 973d091 
>   dashboardv2/public/js/views/schema/SchemaLayoutView.js 35487a5 
>   distro/src/conf/atlas-application.properties 119865d 
>   repository/src/main/java/org/apache/atlas/discovery/HiveLineageService.java 
> 00905d7 
>   repository/src/main/scala/org/apache/atlas/query/ClosureQuery.scala 05dc6a4 
>   repository/src/test/java/org/apache/atlas/BaseHiveRepositoryTest.java 
> 66e1365 
>   
> repository/src/test/java/org/apache/atlas/discovery/HiveLineageServiceTest.java
>  6d5a15a 
>   repository/src/test/scala/org/apache/atlas/query/GremlinTest2.scala 2fd8bb9 
>   server-api/src/main/java/org/apache/atlas/discovery/LineageService.java 
> 8dc36cd 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/AttributeInfo.java 
> 330d1cb 
>   typesystem/src/main/resources/atlas-application.properties a343a20 
>   
> webapp/src/main/java/org/apache/atlas/web/resources/HiveLineageResource.java 
> 9b3fbc9 
>   webapp/src/main/java/org/apache/atlas/web/resources/LineageResource.java 
> PRE-CREATION 
>   
> webapp/src/test/java/org/apache/atlas/web/resources/HiveLineageJerseyResourceIT.java
>  bc02f90 
> 
> Diff: https://reviews.apache.org/r/47225/diff/
> 
> 
> Testing
> ---
> 
> UTs, ITs and end to end test with hive cli
> 
> 
> Thanks,
> 
> Shwetha GS
> 
>

Re: Review Request 47135: Install Setup: SOLR

[Suma Shivaprasad]

> On May 10, 2016, 6:11 p.m., Suma Shivaprasad wrote:
> > distro/src/bin/atlas_config.py, line 102
> > <https://reviews.apache.org/r/47135/diff/1/?file=1377049#file1377049line102>
> >
> > if this config is changed by a user through Ambari, we need to handle 
> > the case where we need to support resharding after confirming from the 
> > user. I am not sure if this is something that can be done instantaneously 
> > or requires down time. Can we dig deeper and see how we can handle this. 
> > The same goes for replica
> 
> Tom Beerbower wrote:
> It can be set by Ambari.  I'll deal with it in the related Ambari patch.
> 
> The resharding seems like an issue with or without this patch.  Can it be 
> handled in a separate Jira?

Yes we can take it up in a separate jira


- Suma


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47135/#review132508
---


On May 12, 2016, 1:19 a.m., Tom Beerbower wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47135/
> ---
> 
> (Updated May 12, 2016, 1:19 a.m.)
> 
> 
> Review request for atlas.
> 
> 
> Bugs: ATLAS-663 and ATLAS-673
> https://issues.apache.org/jira/browse/ATLAS-663
> https://issues.apache.org/jira/browse/ATLAS-673
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Install standalone SOLR by default.
> 
> SOLR indexes need to be setup properly.
> 
> Fix location of standalone solr and hbase data dirs.
> 
> 
> Diffs
> -
> 
>   .gitignore 20db282 
>   distro/pom.xml ee9add0 
>   distro/src/bin/atlas_config.py dbe1191 
>   distro/src/bin/atlas_start.py 73b8920 
>   distro/src/bin/atlas_stop.py 265084e 
>   distro/src/conf/atlas-application.properties 119865d 
>   distro/src/conf/hbase/hbase-site.xml.template f5c4aad 
>   distro/src/main/assemblies/standalone-package.xml 527f604 
>   distro/src/test/python/scripts/TestMetadata.py 9e55869 
>   docs/src/site/twiki/Configuration.twiki 0793c82 
>   docs/src/site/twiki/InstallationSteps.twiki 270d1c4 
> 
> Diff: https://reviews.apache.org/r/47135/diff/
> 
> 
> Testing
> ---
> 
> Manual testing of atlas_start / atlas_start with standalone solr.
> 
> mvn clean test
> 
> 
> Thanks,
> 
> Tom Beerbower
> 
>

Re: Review Request 47135: Install Setup: SOLR

[Suma Shivaprasad]

> On May 10, 2016, 6:11 p.m., Suma Shivaprasad wrote:
> > distro/src/bin/atlas_config.py, line 102
> > <https://reviews.apache.org/r/47135/diff/1/?file=1377049#file1377049line102>
> >
> > if this config is changed by a user through Ambari, we need to handle 
> > the case where we need to support resharding after confirming from the 
> > user. I am not sure if this is something that can be done instantaneously 
> > or requires down time. Can we dig deeper and see how we can handle this. 
> > The same goes for replica
> 
> Tom Beerbower wrote:
> It can be set by Ambari.  I'll deal with it in the related Ambari patch.
> 
> The resharding seems like an issue with or without this patch.  Can it be 
> handled in a separate Jira?
> 
> Suma Shivaprasad wrote:
> Yes we can take it up in a separate jira

If the user changes these values and expects it to be reflected with the bew 
values in the setup, it can lead to confusion. Are we planning to address this 
jira for Erie?


- Suma


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47135/#review132508
---


On May 12, 2016, 1:19 a.m., Tom Beerbower wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47135/
> ---
> 
> (Updated May 12, 2016, 1:19 a.m.)
> 
> 
> Review request for atlas.
> 
> 
> Bugs: ATLAS-663 and ATLAS-673
> https://issues.apache.org/jira/browse/ATLAS-663
> https://issues.apache.org/jira/browse/ATLAS-673
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Install standalone SOLR by default.
> 
> SOLR indexes need to be setup properly.
> 
> Fix location of standalone solr and hbase data dirs.
> 
> 
> Diffs
> -
> 
>   .gitignore 20db282 
>   distro/pom.xml ee9add0 
>   distro/src/bin/atlas_config.py dbe1191 
>   distro/src/bin/atlas_start.py 73b8920 
>   distro/src/bin/atlas_stop.py 265084e 
>   distro/src/conf/atlas-application.properties 119865d 
>   distro/src/conf/hbase/hbase-site.xml.template f5c4aad 
>   distro/src/main/assemblies/standalone-package.xml 527f604 
>   distro/src/test/python/scripts/TestMetadata.py 9e55869 
>   docs/src/site/twiki/Configuration.twiki 0793c82 
>   docs/src/site/twiki/InstallationSteps.twiki 270d1c4 
> 
> Diff: https://reviews.apache.org/r/47135/diff/
> 
> 
> Testing
> ---
> 
> Manual testing of atlas_start / atlas_start with standalone solr.
> 
> mvn clean test
> 
> 
> Thanks,
> 
> Tom Beerbower
> 
>

Re: Review Request 47225: ATLAS-713 Atlas UI is still showing the deleted column of a hive table (Entity response of hive table does not have it). - with browser cache disabled.

[Suma Shivaprasad]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47225/#review132978
---




repository/src/main/java/org/apache/atlas/discovery/HiveLineageService.java 
(line 146)
<https://reviews.apache.org/r/47225/#comment197271>

rename to DataSetLineageQuery and remove all hive specific references ?


- Suma Shivaprasad


On May 11, 2016, 9:21 a.m., Shwetha GS wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47225/
> ---
> 
> (Updated May 11, 2016, 9:21 a.m.)
> 
> 
> Review request for atlas.
> 
> 
> Bugs: ATLAS-713
> https://issues.apache.org/jira/browse/ATLAS-713
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Current lineage APIs work on the dataset name. With soft delete, there can be 
> more than one entities for the given dataset name. 
> 1. Added new lineage APIs(input graph, output graph, schema) which work on 
> entity id
> 2. Modified current lineage APIs to work on the active entity for the given 
> dataset name
> 3. Modified UI to use the new APIs
> 
> 
> Diffs
> -
> 
>   client/src/main/java/org/apache/atlas/AtlasClient.java b3ec95c 
>   dashboardv2/public/js/models/VLineage.js e33488a 
>   dashboardv2/public/js/models/VSchema.js 1f8e0bb 
>   dashboardv2/public/js/views/detail_page/DetailPageLayoutView.js 87adec0 
>   dashboardv2/public/js/views/graph/LineageLayoutView.js 973d091 
>   dashboardv2/public/js/views/schema/SchemaLayoutView.js 35487a5 
>   distro/src/conf/atlas-application.properties 119865d 
>   repository/src/main/java/org/apache/atlas/discovery/HiveLineageService.java 
> 00905d7 
>   repository/src/main/scala/org/apache/atlas/query/ClosureQuery.scala 05dc6a4 
>   repository/src/test/java/org/apache/atlas/BaseHiveRepositoryTest.java 
> 66e1365 
>   
> repository/src/test/java/org/apache/atlas/discovery/HiveLineageServiceTest.java
>  6d5a15a 
>   repository/src/test/scala/org/apache/atlas/query/GremlinTest2.scala 2fd8bb9 
>   server-api/src/main/java/org/apache/atlas/discovery/LineageService.java 
> 8dc36cd 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/AttributeInfo.java 
> 330d1cb 
>   typesystem/src/main/resources/atlas-application.properties a343a20 
>   
> webapp/src/main/java/org/apache/atlas/web/resources/HiveLineageResource.java 
> 9b3fbc9 
>   webapp/src/main/java/org/apache/atlas/web/resources/LineageResource.java 
> PRE-CREATION 
>   
> webapp/src/test/java/org/apache/atlas/web/resources/HiveLineageJerseyResourceIT.java
>  bc02f90 
> 
> Diff: https://reviews.apache.org/r/47225/diff/
> 
> 
> Testing
> ---
> 
> UTs, ITs and end to end test with hive cli
> 
> 
> Thanks,
> 
> Shwetha GS
> 
>

Re: Review Request 47225: ATLAS-713 Atlas UI is still showing the deleted column of a hive table (Entity response of hive table does not have it). - with browser cache disabled.

[Suma Shivaprasad]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47225/#review133548
---


Ship it!




Ship It!

- Suma Shivaprasad


On May 17, 2016, 8:15 a.m., Shwetha GS wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47225/
> ---
> 
> (Updated May 17, 2016, 8:15 a.m.)
> 
> 
> Review request for atlas.
> 
> 
> Bugs: ATLAS-713
> https://issues.apache.org/jira/browse/ATLAS-713
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Current lineage APIs work on the dataset name. With soft delete, there can be 
> more than one entities for the given dataset name. 
> 1. Added new lineage APIs(input graph, output graph, schema) which work on 
> entity id
> 2. Modified current lineage APIs to work on the active entity for the given 
> dataset name
> 3. Modified UI to use the new APIs
> 
> 
> Diffs
> -
> 
>   client/src/main/java/org/apache/atlas/AtlasClient.java b3ec95c 
>   dashboardv2/public/js/models/VLineage.js e33488a 
>   dashboardv2/public/js/models/VSchema.js 1f8e0bb 
>   dashboardv2/public/js/views/detail_page/DetailPageLayoutView.js 87adec0 
>   dashboardv2/public/js/views/graph/LineageLayoutView.js 973d091 
>   dashboardv2/public/js/views/schema/SchemaLayoutView.js 35487a5 
>   distro/src/conf/atlas-application.properties 68a0021 
>   repository/src/main/java/org/apache/atlas/RepositoryMetadataModule.java 
> 8dae968 
>   
> repository/src/main/java/org/apache/atlas/discovery/DataSetLineageService.java
>  PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/discovery/HiveLineageService.java 
> 00905d7 
>   repository/src/main/scala/org/apache/atlas/query/ClosureQuery.scala 05dc6a4 
>   repository/src/test/java/org/apache/atlas/BaseHiveRepositoryTest.java 
> 66e1365 
>   repository/src/test/java/org/apache/atlas/BaseRepositoryTest.java 
> PRE-CREATION 
>   
> repository/src/test/java/org/apache/atlas/discovery/DataSetLineageServiceTest.java
>  PRE-CREATION 
>   
> repository/src/test/java/org/apache/atlas/discovery/GraphBackedDiscoveryServiceTest.java
>  3b50dfb 
>   
> repository/src/test/java/org/apache/atlas/discovery/HiveLineageServiceTest.java
>  6d5a15a 
>   repository/src/test/scala/org/apache/atlas/query/GremlinTest2.scala 2fd8bb9 
>   server-api/src/main/java/org/apache/atlas/discovery/LineageService.java 
> 8dc36cd 
>   typesystem/src/main/resources/atlas-application.properties aafad0f 
>   
> webapp/src/main/java/org/apache/atlas/web/resources/DataSetLineageResource.java
>  PRE-CREATION 
>   
> webapp/src/main/java/org/apache/atlas/web/resources/HiveLineageResource.java 
> 9b3fbc9 
>   webapp/src/main/java/org/apache/atlas/web/resources/LineageResource.java 
> PRE-CREATION 
>   
> webapp/src/test/java/org/apache/atlas/web/resources/DataSetLineageJerseyResourceIT.java
>  PRE-CREATION 
>   
> webapp/src/test/java/org/apache/atlas/web/resources/HiveLineageJerseyResourceIT.java
>  bc02f90 
> 
> Diff: https://reviews.apache.org/r/47225/diff/
> 
> 
> Testing
> ---
> 
> UTs, ITs and end to end test with hive cli
> 
> 
> Thanks,
> 
> Shwetha GS
> 
>

Re: Review Request 47107: ATLAS_619 Normalize literals for hive queries

[Suma Shivaprasad]

> On May 12, 2016, 9:21 a.m., Shwetha GS wrote:
> > addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java, 
> > line 726
> > <https://reviews.apache.org/r/47107/diff/2/?file=1380615#file1380615line726>
> >
> > Can you file a bug to handle hive_process update - to update 
> > recentQueries

ATLAS-782


- Suma


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47107/#review132845
---


On May 13, 2016, 7:20 a.m., Suma Shivaprasad wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47107/
> ---
> 
> (Updated May 13, 2016, 7:20 a.m.)
> 
> 
> Review request for atlas.
> 
> 
> Bugs: ATLAS-619
> https://issues.apache.org/jira/browse/ATLAS-619
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Normlized all literals in a query and replacing them CONSTANT literals. 
> Storing recently run queries in  a new attribute for hive_process - 
> recentQueries
> 
> 
> Diffs
> -
> 
>   
> addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/hook/FalconHook.java
>  8fced05 
>   
> addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/model/FalconDataModelGenerator.java
>  2494675 
>   
> addons/falcon-bridge/src/test/java/org/apache/atlas/falcon/hook/FalconHookIT.java
>  4e2a06f 
>   addons/hive-bridge/pom.xml eeb2aa4 
>   addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
> ffd5d22 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/model/HiveDataModelGenerator.java
>  6418770 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/rewrite/ASTRewriter.java
>  PRE-CREATION 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/rewrite/HiveASTRewriter.java
>  PRE-CREATION 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/rewrite/LiteralRewriter.java
>  PRE-CREATION 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/rewrite/RewriteContext.java
>  PRE-CREATION 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/rewrite/RewriteException.java
>  PRE-CREATION 
>   
> addons/hive-bridge/src/test/java/org/apache/atlas/hive/bridge/HiveLiteralRewriterTest.java
>  PRE-CREATION 
>   addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 
> da21195 
>   addons/hive-bridge/src/test/resources/hive-site.xml f1facb8 
>   
> addons/sqoop-bridge/src/main/java/org/apache/atlas/sqoop/hook/SqoopHook.java 
> ab7e6ee 
>   
> addons/sqoop-bridge/src/test/java/org/apache/atlas/sqoop/hook/SqoopHookIT.java
>  2820169 
>   
> addons/storm-bridge/src/main/java/org/apache/atlas/storm/hook/StormAtlasHook.java
>  267e228 
>   
> addons/storm-bridge/src/main/scala/org/apache/atlas/storm/model/StormDataModel.scala
>  de67c39 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/TypedInstanceToGraphMapper.java
>  a017536 
>   
> repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java
>  9f69940 
>   repository/src/test/java/org/apache/atlas/BaseHiveRepositoryTest.java 
> 66e1365 
>   webapp/src/main/java/org/apache/atlas/examples/QuickStart.java 70dce6b 
>   webapp/src/test/java/org/apache/atlas/examples/QuickStartIT.java cdf6049 
>   
> webapp/src/test/java/org/apache/atlas/web/resources/HiveLineageJerseyResourceIT.java
>  bc02f90 
> 
> Diff: https://reviews.apache.org/r/47107/diff/
> 
> 
> Testing
> ---
> 
> Added UT - HiveAStREwriteTest and IT asserttions in HiveHookIT
> 
> 
> Thanks,
> 
> Suma Shivaprasad
> 
>

Re: Review Request 47107: ATLAS_619 Normalize literals for hive queries

[Suma Shivaprasad]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47107/
---

(Updated May 13, 2016, 7:20 a.m.)


Review request for atlas.


Bugs: ATLAS-619
https://issues.apache.org/jira/browse/ATLAS-619


Repository: atlas


Description
---

Normlized all literals in a query and replacing them CONSTANT literals. Storing 
recently run queries in  a new attribute for hive_process - recentQueries


Diffs (updated)
-

  
addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/hook/FalconHook.java 
8fced05 
  
addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/model/FalconDataModelGenerator.java
 2494675 
  
addons/falcon-bridge/src/test/java/org/apache/atlas/falcon/hook/FalconHookIT.java
 4e2a06f 
  addons/hive-bridge/pom.xml eeb2aa4 
  addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
ffd5d22 
  
addons/hive-bridge/src/main/java/org/apache/atlas/hive/model/HiveDataModelGenerator.java
 6418770 
  
addons/hive-bridge/src/main/java/org/apache/atlas/hive/rewrite/ASTRewriter.java 
PRE-CREATION 
  
addons/hive-bridge/src/main/java/org/apache/atlas/hive/rewrite/HiveASTRewriter.java
 PRE-CREATION 
  
addons/hive-bridge/src/main/java/org/apache/atlas/hive/rewrite/LiteralRewriter.java
 PRE-CREATION 
  
addons/hive-bridge/src/main/java/org/apache/atlas/hive/rewrite/RewriteContext.java
 PRE-CREATION 
  
addons/hive-bridge/src/main/java/org/apache/atlas/hive/rewrite/RewriteException.java
 PRE-CREATION 
  
addons/hive-bridge/src/test/java/org/apache/atlas/hive/bridge/HiveLiteralRewriterTest.java
 PRE-CREATION 
  addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 
da21195 
  addons/hive-bridge/src/test/resources/hive-site.xml f1facb8 
  addons/sqoop-bridge/src/main/java/org/apache/atlas/sqoop/hook/SqoopHook.java 
ab7e6ee 
  
addons/sqoop-bridge/src/test/java/org/apache/atlas/sqoop/hook/SqoopHookIT.java 
2820169 
  
addons/storm-bridge/src/main/java/org/apache/atlas/storm/hook/StormAtlasHook.java
 267e228 
  
addons/storm-bridge/src/main/scala/org/apache/atlas/storm/model/StormDataModel.scala
 de67c39 
  
repository/src/main/java/org/apache/atlas/repository/graph/TypedInstanceToGraphMapper.java
 a017536 
  
repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java 
9f69940 
  repository/src/test/java/org/apache/atlas/BaseHiveRepositoryTest.java 66e1365 
  webapp/src/main/java/org/apache/atlas/examples/QuickStart.java 70dce6b 
  webapp/src/test/java/org/apache/atlas/examples/QuickStartIT.java cdf6049 
  
webapp/src/test/java/org/apache/atlas/web/resources/HiveLineageJerseyResourceIT.java
 bc02f90 

Diff: https://reviews.apache.org/r/47107/diff/


Testing
---

Added UT - HiveAStREwriteTest and IT asserttions in HiveHookIT


Thanks,

Suma Shivaprasad

Re: Review Request 47107: ATLAS_619 Normalize literals for hive queries

[Suma Shivaprasad]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47107/
---

(Updated May 13, 2016, 7:11 a.m.)


Review request for atlas.


Changes
---

Fixed review comments


Bugs: ATLAS-619
https://issues.apache.org/jira/browse/ATLAS-619


Repository: atlas


Description
---

Normlized all literals in a query and replacing them CONSTANT literals. Storing 
recently run queries in  a new attribute for hive_process - recentQueries


Diffs (updated)
-

  
addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/hook/FalconHook.java 
8fced05 
  
addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/model/FalconDataModelGenerator.java
 2494675 
  
addons/falcon-bridge/src/test/java/org/apache/atlas/falcon/hook/FalconHookIT.java
 4e2a06f 
  addons/hive-bridge/pom.xml eeb2aa4 
  addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
ffd5d22 
  
addons/hive-bridge/src/main/java/org/apache/atlas/hive/model/HiveDataModelGenerator.java
 6418770 
  
addons/hive-bridge/src/main/java/org/apache/atlas/hive/rewrite/ASTRewriter.java 
PRE-CREATION 
  
addons/hive-bridge/src/main/java/org/apache/atlas/hive/rewrite/HiveASTRewriter.java
 PRE-CREATION 
  
addons/hive-bridge/src/main/java/org/apache/atlas/hive/rewrite/LiteralRewriter.java
 PRE-CREATION 
  
addons/hive-bridge/src/main/java/org/apache/atlas/hive/rewrite/RewriteContext.java
 PRE-CREATION 
  
addons/hive-bridge/src/main/java/org/apache/atlas/hive/rewrite/RewriteException.java
 PRE-CREATION 
  
addons/hive-bridge/src/test/java/org/apache/atlas/hive/bridge/HiveLiteralRewriterTest.java
 PRE-CREATION 
  addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 
da21195 
  addons/hive-bridge/src/test/resources/hive-site.xml f1facb8 
  addons/sqoop-bridge/src/main/java/org/apache/atlas/sqoop/hook/SqoopHook.java 
ab7e6ee 
  
addons/sqoop-bridge/src/test/java/org/apache/atlas/sqoop/hook/SqoopHookIT.java 
2820169 
  
addons/storm-bridge/src/main/java/org/apache/atlas/storm/hook/StormAtlasHook.java
 267e228 
  
addons/storm-bridge/src/main/scala/org/apache/atlas/storm/model/StormDataModel.scala
 de67c39 
  
repository/src/main/java/org/apache/atlas/repository/graph/TypedInstanceToGraphMapper.java
 a017536 
  
repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java 
9f69940 
  repository/src/test/java/org/apache/atlas/BaseHiveRepositoryTest.java 66e1365 
  webapp/src/main/java/org/apache/atlas/examples/QuickStart.java 70dce6b 
  webapp/src/test/java/org/apache/atlas/examples/QuickStartIT.java cdf6049 
  
webapp/src/test/java/org/apache/atlas/web/resources/HiveLineageJerseyResourceIT.java
 bc02f90 

Diff: https://reviews.apache.org/r/47107/diff/


Testing
---

Added UT - HiveAStREwriteTest and IT asserttions in HiveHookIT


Thanks,

Suma Shivaprasad