[jira] [Created] (ATLAS-866) Atlas allows re-creation of tags with same name

[Sharmadha Sainath (JIRA)]

Sharmadha Sainath created ATLAS-866:
---

 Summary: Atlas allows re-creation of tags with same name
 Key: ATLAS-866
 URL: https://issues.apache.org/jira/browse/ATLAS-866
 Project: Atlas
  Issue Type: Bug
 Environment: Atlas commit id : 59268875d256c92f748198cea999ed5057c6bf18

Reporter: Sharmadha Sainath


Started Atlas and created an already existing Tag DataProducer . The new tag 
created successfully and replaced old tag.
In previous versions of Atlas, creating tags with same name would throw error. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (ATLAS-852) Change Default landing page to taxonomy

[Hemanth Yamijala (JIRA)]

[ 
https://issues.apache.org/jira/browse/ATLAS-852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15317915#comment-15317915
 ] 

Hemanth Yamijala commented on ATLAS-852:


Looks good to me. Tested with master, and see that the taxonomy tab comes first 
in focus. +1.

> Change Default landing page to taxonomy
> ---
>
> Key: ATLAS-852
> URL: https://issues.apache.org/jira/browse/ATLAS-852
> Project: Atlas
>  Issue Type: Improvement
>Reporter: Keval Bhatt
>Assignee: Keval Bhatt
> Attachments: ATLAS-852.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (ATLAS-694) Update Atlas code to use abstraction layer

[Jeffrey Hagelberg (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeffrey Hagelberg updated ATLAS-694:

Attachment: rb47810.patch

Resolved new test failures.

> Update Atlas code to use abstraction layer
> --
>
> Key: ATLAS-694
> URL: https://issues.apache.org/jira/browse/ATLAS-694
> Project: Atlas
>  Issue Type: Sub-task
>Reporter: Jeffrey Hagelberg
>Assignee: Jeffrey Hagelberg
> Attachments: rb47810.patch
>
>
> In this task, we will update Atlas to replace the direct usage of titan 
> classes with usage of the equivalent interfaces in the abstraction layer.  At 
> this point, there will be no options to configure the abstraction layer 
> implementation that gets used.  Atlas will be hard-coded to use the titan 
> 0.5.4 implementation.  It will also continue to have a compile time 
> dependency on the titan 0.5.4 graph database implementation.  When this task 
> is complete, Atlas will only be accessing titan through the abstraction layer.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 47810: ATLAS-694: Update Atlas to use Graph DB abstraction layer

[Jeff Hagelberg]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47810/
---

(Updated June 7, 2016, 2:44 a.m.)


Review request for atlas, David Kantor and Neeru Gupta.


Changes
---

Fixed merge-related test failure.


Bugs: ATLAS-694
https://issues.apache.org/jira/browse/ATLAS-694


Repository: atlas


Description
---

ATLAS-694: Update Atlas to use abstraction layer.  All of the Atlas code (with 
the exception of the catalog, which was only updated minimally) has been 
updated to use the graph database abstraction layer.  In addition, there is now 
an optional Atlas configuration property that specifies the class of the 
abstraction layer database to use.  I basically put all of the changes in here 
with the exception of the actual Titan 1 implementation of code.  This includes 
the changes to support Tinkerpop 3 syntax.  This is mostly to expedite getting 
the changes into Atlas.  Originally the TP3 changes were going to be brought in 
as part of the Titan 1 implementation task.

Change Summary:

   - change Atlas classes to use AtlasGraph,AtlasVertex,AtlasEdge, etc instead 
of TitanGraph/Vertex/Edge, etc
   - compile time dependency on titan 0.5.4/TP 2 removed (except in Catalog, 
which was only changed to use AtlasGraphProvider/AtlasGraph)
   - updated DSL translation to generate Gremlin that is compliant with TP3 
when TP3 is being used
   - TitanGraphProvider replaced by AtlasGraphProvider.  Graph database 
implementation is determined from a new optional configuration property
   - search indexer enhanced to wait for all indices to be enabled
   - HiveTitanSample is no longer used by tests.  It has been replaced by 
hive-instances.json (which uses normal Atlas json syntax).  The data is saved 
with a new JSONImporter class.  This was needed because the graphson syntax 
used by HiveTitanSample is not compatible with TP3.  
   - RepositoryMetaModule - GraphBackedSearchIndexer changed to an eager 
singleton to force index creation to happen before we execute any gremlin 
queries (which happens during the type system restoration triggered by the 
DefaultMetadataService constructor)
   - atlas_config.py - minor change to allow Atlas to use a non-default JVM.  
It now supports having the java home in ATLAS_JAVA_HOME.  If that is not set, 
we fall back to the previous behavior looking for JAVA_HOME and then checking 
the path.
   
   
Note - the diff here is cumulative and includes the changes from ATLAS-693.  I 
could not figure out how to remove those.  You can, for the most part, ignore 
the new files under graphdb\titan0.  There are a few minor changes that have 
been put in since ATLAS-693, but nothing very significant.  Just changes to 
implement a couple new methods in the abstraction layer.

Last rebase: 6/6/2016


Diffs (updated)
-

  addons/falcon-bridge/pom.xml 14c60901a6f194497ca0fb7929a032d2261f85fa 
  addons/hdfs-model/pom.xml f4033bce0fe3ea5a33cfd32ebccd47c01207dc35 
  addons/hdfs-model/src/test/java/org/apache/atlas/fs/model/HDFSModelTest.java 
ac60294e328835ba0340e150799ddfb348ccdb52 
  addons/hive-bridge/pom.xml 47e72e814a421f03b00a9b7c6a13163e1b67ca14 
  addons/sqoop-bridge/pom.xml 343bb4eeae93031ab24cd48ed7b535769150d6c7 
  addons/storm-bridge/pom.xml b1a7a9b1a88442d9e65db4a01b8388c8d3522bfe 
  authorization/pom.xml 74c433bf740a346d1d907a868605e9ab457151bb 
  catalog/pom.xml 8a49d3d673a5936613d45e53479aa99a5a14517d 
  catalog/src/main/java/org/apache/atlas/catalog/DefaultTypeSystem.java 
a28a32b692e61377479cdd432f39ba3c60bd5238 
  catalog/src/main/java/org/apache/atlas/catalog/query/BaseQuery.java 
ba8e0e712769afad72f7f8254ec626a1d17136e8 
  
catalog/src/test/java/org/apache/atlas/catalog/query/AtlasEntityQueryTest.java 
149134c8a79c802fe625d64c5e04af0d88174cfa 
  distro/src/bin/atlas_config.py fab4046345e05e0b7749ca75834c8bdcb0d01752 
  distro/src/conf/atlas-application.properties 
1cdd424d8e126a6bdbf99db8bf4e5d4329e806bb 
  graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasEdge.java 
71b577b5c2e8aee8fec816b9224672ebc2cad5b3 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasEdgeDirection.java
 e7da1c9a5075004973ba68b8a36a2c95756e0e62 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasElement.java 
3c41693e531eb523f04a5c5781f1f123ab38ef07 
  graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 
f312117af9cb6ae8de21d565bfcf3b10d9bdfcc6 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphIndex.java
 PRE-CREATION 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphManagement.java
 2776b2242c4469dcae344616ccfe100633a79432 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphQuery.java
 f0dca9a28eebd8996d55d71dcc3254cce9cd4098 
  

[jira] [Updated] (ATLAS-694) Update Atlas code to use abstraction layer

[Jeffrey Hagelberg (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeffrey Hagelberg updated ATLAS-694:

Attachment: (was: rb47810.patch)

> Update Atlas code to use abstraction layer
> --
>
> Key: ATLAS-694
> URL: https://issues.apache.org/jira/browse/ATLAS-694
> Project: Atlas
>  Issue Type: Sub-task
>Reporter: Jeffrey Hagelberg
>Assignee: Jeffrey Hagelberg
>
> In this task, we will update Atlas to replace the direct usage of titan 
> classes with usage of the equivalent interfaces in the abstraction layer.  At 
> this point, there will be no options to configure the abstraction layer 
> implementation that gets used.  Atlas will be hard-coded to use the titan 
> 0.5.4 implementation.  It will also continue to have a compile time 
> dependency on the titan 0.5.4 graph database implementation.  When this task 
> is complete, Atlas will only be accessing titan through the abstraction layer.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (ATLAS-694) Update Atlas code to use abstraction layer

[Jeffrey Hagelberg (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeffrey Hagelberg updated ATLAS-694:

Attachment: rb47810.patch

> Update Atlas code to use abstraction layer
> --
>
> Key: ATLAS-694
> URL: https://issues.apache.org/jira/browse/ATLAS-694
> Project: Atlas
>  Issue Type: Sub-task
>Reporter: Jeffrey Hagelberg
>Assignee: Jeffrey Hagelberg
> Attachments: rb47810.patch
>
>
> In this task, we will update Atlas to replace the direct usage of titan 
> classes with usage of the equivalent interfaces in the abstraction layer.  At 
> this point, there will be no options to configure the abstraction layer 
> implementation that gets used.  Atlas will be hard-coded to use the titan 
> 0.5.4 implementation.  It will also continue to have a compile time 
> dependency on the titan 0.5.4 graph database implementation.  When this task 
> is complete, Atlas will only be accessing titan through the abstraction layer.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (ATLAS-694) Update Atlas code to use abstraction layer

[Jeffrey Hagelberg (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeffrey Hagelberg updated ATLAS-694:

Attachment: (was: rb47810.patch)

> Update Atlas code to use abstraction layer
> --
>
> Key: ATLAS-694
> URL: https://issues.apache.org/jira/browse/ATLAS-694
> Project: Atlas
>  Issue Type: Sub-task
>Reporter: Jeffrey Hagelberg
>Assignee: Jeffrey Hagelberg
> Attachments: rb47810.patch
>
>
> In this task, we will update Atlas to replace the direct usage of titan 
> classes with usage of the equivalent interfaces in the abstraction layer.  At 
> this point, there will be no options to configure the abstraction layer 
> implementation that gets used.  Atlas will be hard-coded to use the titan 
> 0.5.4 implementation.  It will also continue to have a compile time 
> dependency on the titan 0.5.4 graph database implementation.  When this task 
> is complete, Atlas will only be accessing titan through the abstraction layer.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Review Request 48270: ATLAS-852 : Change Default landing page to taxonomy

[keval bhatt]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48270/
---

Review request for atlas, Gautam Borad, Madhan Neethiraj, Shwetha GS, 
Selvamohan Neethiraj, and Hemanth Yamijala.


Bugs: ATLAS-852
https://issues.apache.org/jira/browse/ATLAS-852


Repository: atlas


Description
---

Changing default landing page to taxonomy.


Diffs
-

  dashboardv2/public/js/router/Router.js 4d89ab8 
  dashboardv2/public/js/templates/business_catalog/SideNavLayoutView_tmpl.html 
a59fb58 
  dashboardv2/public/js/views/business_catalog/BusinessCatalogLayoutView.js 
3e88b6d 
  dashboardv2/public/js/views/tag/TagLayoutView.js 03adfbc 

Diff: https://reviews.apache.org/r/48270/diff/


Testing
---

Verified default landing page.

Note: Unable to access apache jira site so i have not attached patch on apache 
JIRA.


Thanks,

keval bhatt

[jira] [Updated] (ATLAS-735) UI showing incorrect results if two queries are executed.(first query succeeds but second should fail)

[Erik Bergenholtz (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Erik Bergenholtz updated ATLAS-735:
---
Assignee: Keval Bhatt  (was: Erik Bergenholtz)

> UI showing incorrect results if two queries are executed.(first query 
> succeeds but second should fail)
> --
>
> Key: ATLAS-735
> URL: https://issues.apache.org/jira/browse/ATLAS-735
> Project: Atlas
>  Issue Type: Bug
>Affects Versions: 0.7-incubating
>Reporter: Ayub Khan
>Assignee: Keval Bhatt
> Attachments: DSL-query.gif
>
>
> Steps to repro:
> 1. Through search tab, fire a valid dsl search query(eg: hive_table) and make 
> sure this query response takes time(~30 seconds)
> 2. Before the response comes for the first query, search for another invalid 
> query.(this will fail with 400 error).
> 3. Now when the response for first query returns, the UI still renders the 
> first query's response as if it is a response to second query which seems 
> incorrect. And this might lead to a confusion from a customer's perspective.
> This can be an issue from a customer's perspective and the probability of 
> happening in production is pretty high.
> If the above steps are not clear, I am attaching a GIF which captures the 
> steps to repro. Please take a look.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (ATLAS-740) UI: "References" DSL query response is not handled properly

[Erik Bergenholtz (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Erik Bergenholtz updated ATLAS-740:
---
Assignee: Keval Bhatt  (was: Erik Bergenholtz)

> UI: "References" DSL query response is not handled properly
> ---
>
> Key: ATLAS-740
> URL: https://issues.apache.org/jira/browse/ATLAS-740
> Project: Atlas
>  Issue Type: Bug
>Affects Versions: 0.7-incubating
>Reporter: Ayub Khan
>Assignee: Keval Bhatt
> Fix For: 0.7-incubating
>
>
> Issue snapshot: https://monosnap.com/file/NbjjIAjZfsiWChA2U5B1KZiEBYXF7X



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (ATLAS-865) Edit description functionality for Tags

[Keval Bhatt (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Keval Bhatt reassigned ATLAS-865:
-

Assignee: Keval Bhatt

> Edit description functionality for Tags
> ---
>
> Key: ATLAS-865
> URL: https://issues.apache.org/jira/browse/ATLAS-865
> Project: Atlas
>  Issue Type: Bug
>Reporter: Keval Bhatt
>Assignee: Keval Bhatt
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (ATLAS-852) Change Default landing page to taxonomy

[ATLAS QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/ATLAS-852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15316475#comment-15316475
 ] 

ATLAS QA commented on ATLAS-852:


{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment
  http://issues.apache.org/jira/secure/attachment/12808369/ATLAS-852.patch
  against master revision d661964.

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:red}-1 tests included{color}.  The patch doesn't appear to include 
any new or modified tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

+1 checkstyle.  The patch generated 0 code style errors.

{color:red}-1 findbugs{color}.  The patch appears to introduce 378 new 
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:red}-1 core tests{color}.  The patch failed these unit tests in :
   
org.apache.atlas.repository.typestore.GraphBackedTypeStoreTest

Test results: 
https://builds.apache.org/job/PreCommit-ATLAS-Build/288//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-ATLAS-Build/288//artifact/patchprocess/newPatchFindbugsWarningsrepository.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-ATLAS-Build/288//artifact/patchprocess/newPatchFindbugsWarningscommon.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-ATLAS-Build/288//artifact/patchprocess/newPatchFindbugsWarningsnotification.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-ATLAS-Build/288//artifact/patchprocess/newPatchFindbugsWarningswebapp.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-ATLAS-Build/288//artifact/patchprocess/newPatchFindbugsWarningsclient.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-ATLAS-Build/288//artifact/patchprocess/newPatchFindbugsWarningstypesystem.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-ATLAS-Build/288//artifact/patchprocess/newPatchFindbugsWarningsfalcon-bridge.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-ATLAS-Build/288//artifact/patchprocess/newPatchFindbugsWarningssqoop-bridge.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-ATLAS-Build/288//artifact/patchprocess/newPatchFindbugsWarningshive-bridge.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-ATLAS-Build/288//artifact/patchprocess/newPatchFindbugsWarningsstorm-bridge.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-ATLAS-Build/288//artifact/patchprocess/newPatchFindbugsWarningshdfs-model.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-ATLAS-Build/288//artifact/patchprocess/newPatchFindbugsWarningsauthorization.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-ATLAS-Build/288//artifact/patchprocess/newPatchFindbugsWarningscatalog.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-ATLAS-Build/288//artifact/patchprocess/newPatchFindbugsWarningstitan.html
Console output: https://builds.apache.org/job/PreCommit-ATLAS-Build/288//console

This message is automatically generated.

> Change Default landing page to taxonomy
> ---
>
> Key: ATLAS-852
> URL: https://issues.apache.org/jira/browse/ATLAS-852
> Project: Atlas
>  Issue Type: Improvement
>Reporter: Keval Bhatt
>Assignee: Keval Bhatt
> Attachments: ATLAS-852.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (ATLAS-761) UI: is ignoring table parameters

[Erik Bergenholtz (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Erik Bergenholtz updated ATLAS-761:
---
Assignee: Keval Bhatt  (was: Erik Bergenholtz)

> UI: is ignoring table parameters
> 
>
> Key: ATLAS-761
> URL: https://issues.apache.org/jira/browse/ATLAS-761
> Project: Atlas
>  Issue Type: Bug
>Affects Versions: 0.7-incubating
>Reporter: Ayub Khan
>Assignee: Keval Bhatt
> Fix For: 0.7-incubating
>
>
> Entity api response showing table parameters which are ignored by UI.
> {noformat}
> curl 
> http://localhost:21000/api/atlas/entities/b17b2c93-dbf7-4592-b59e-4614c6e3d37e
>  | python -m json.tool
>   % Total% Received % Xferd  Average Speed   TimeTime Time  
> Current
>  Dload  Upload   Total   SpentLeft  Speed
> 100  65080  65080 0   4152  0 --:--:--  0:00:01 --:--:--  4150
> {
> "GUID": "b17b2c93-dbf7-4592-b59e-4614c6e3d37e",
> "definition": {
> "id": {
> "id": "b17b2c93-dbf7-4592-b59e-4614c6e3d37e",
> "jsonClass": 
> "org.apache.atlas.typesystem.json.InstanceSerialization$_Id",
> "state": "ACTIVE",
> "typeName": "hive_table",
> "version": 0
> },
> "jsonClass": 
> "org.apache.atlas.typesystem.json.InstanceSerialization$_Reference",
> "traitNames": [],
> "traits": {},
> "typeName": "hive_table",
> "values": {
> "columns": [
> {
> "id": {
> "id": "f118ccf4-bd8b-4a0e-af35-17a5c5018e06",
> "jsonClass": 
> "org.apache.atlas.typesystem.json.InstanceSerialization$_Id",
> "state": "ACTIVE",
> "typeName": "hive_column",
> "version": 0
> },
> "jsonClass": 
> "org.apache.atlas.typesystem.json.InstanceSerialization$_Reference",
> "traitNames": [],
> "traits": {},
> "typeName": "hive_column",
> "values": {
> "comment": null,
> "name": "viewtime",
> "qualifiedName": 
> "default.table3_s1vxgkt0bh.viewtime@primary",
> "table": {
> "id": "b17b2c93-dbf7-4592-b59e-4614c6e3d37e",
> "jsonClass": 
> "org.apache.atlas.typesystem.json.InstanceSerialization$_Id",
> "state": "ACTIVE",
> "typeName": "hive_table",
> "version": 0
> },
> "type": "int"
> }
> },
> {
> "id": {
> "id": "4b0d47e1-16ba-4cdd-85a5-6f35a968c0a3",
> "jsonClass": 
> "org.apache.atlas.typesystem.json.InstanceSerialization$_Id",
> "state": "ACTIVE",
> "typeName": "hive_column",
> "version": 0
> },
> "jsonClass": 
> "org.apache.atlas.typesystem.json.InstanceSerialization$_Reference",
> "traitNames": [],
> "traits": {},
> "typeName": "hive_column",
> "values": {
> "comment": null,
> "name": "userid",
> "qualifiedName": 
> "default.table3_s1vxgkt0bh.userid@primary",
> "table": {
> "id": "b17b2c93-dbf7-4592-b59e-4614c6e3d37e",
> "jsonClass": 
> "org.apache.atlas.typesystem.json.InstanceSerialization$_Id",
> "state": "ACTIVE",
> "typeName": "hive_table",
> "version": 0
> },
> "type": "bigint"
> }
> },
> {
> "id": {
> "id": "ca652a37-a5b8-4563-be27-4c6adb20f2d9",
> "jsonClass": 
> "org.apache.atlas.typesystem.json.InstanceSerialization$_Id",
> "state": "ACTIVE",
> "typeName": "hive_column",
> "version": 0
> },
> "jsonClass": 
> "org.apache.atlas.typesystem.json.InstanceSerialization$_Reference",
> "traitNames": [],
> "traits": {},
> "typeName": "hive_column",
> "values": {
> "comment": 

[jira] [Updated] (ATLAS-755) UI: User defined tableproperties are ignored/not displayed.

[Erik Bergenholtz (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Erik Bergenholtz updated ATLAS-755:
---
Assignee: Keval Bhatt  (was: Erik Bergenholtz)

> UI: User defined tableproperties are ignored/not displayed.
> ---
>
> Key: ATLAS-755
> URL: https://issues.apache.org/jira/browse/ATLAS-755
> Project: Atlas
>  Issue Type: Bug
>Affects Versions: 0.7-incubating
>Reporter: Ayub Khan
>Assignee: Keval Bhatt
> Fix For: 0.7-incubating
>
>
> Add hive table props: ALTER TABLE test_table set TBLPROPERTIES 
> ('key1'='value1','comment'='test comment')
> Hive entity udpate message reflecting the table property changes..
> {noformat}
> {
> "entities": [
> {
> "jsonClass": 
> "org.apache.atlas.typesystem.json.InstanceSerialization$_Reference",
> "id": {
> "jsonClass": 
> "org.apache.atlas.typesystem.json.InstanceSerialization$_Id",
> "id": "-85198870717493",
> "version": 0,
> "typeName": "hive_db",
> "state": "ACTIVE"
> },
> "typeName": "hive_db",
> "values": {
> "name": "default",
> "description": "Default Hive database",
> "ownerType": 2,
> "qualifiedName": "default@cl1",
> "locationUri": 
> "hdfs://os-r6-apathan-atlas-erie-nosec-2.openstacklocal:8020/apps/hive/warehouse",
> "owner": "public",
> "clusterName": "cl1",
> "parameters": {}
> },
> "traitNames": [],
> "traits": {}
> },
> {
> "jsonClass": 
> "org.apache.atlas.typesystem.json.InstanceSerialization$_Reference",
> "id": {
> "jsonClass": 
> "org.apache.atlas.typesystem.json.InstanceSerialization$_Id",
> "id": "-85198897495080",
> "version": 0,
> "typeName": "hive_table",
> "state": "ACTIVE"
> },
> "typeName": "hive_table",
> "values": {
> "tableType": "MANAGED_TABLE",
> "name": "default.test_table@cl1",
> "createTime": "2016-05-06T10:00:52.000Z",
> "temporary": false,
> "db": {
> "jsonClass": 
> "org.apache.atlas.typesystem.json.InstanceSerialization$_Reference",
> "id": {
> "jsonClass": 
> "org.apache.atlas.typesystem.json.InstanceSerialization$_Id",
> "id": "-85198870717493",
> "version": 0,
> "typeName": "hive_db",
> "state": "ACTIVE"
> },
> "typeName": "hive_db",
> "values": {
> "name": "default",
> "description": "Default Hive database",
> "ownerType": 2,
> "qualifiedName": "default@cl1",
> "locationUri": 
> "hdfs://os-r6-apathan-atlas-erie-nosec-2.openstacklocal:8020/apps/hive/warehouse",
> "owner": "public",
> "clusterName": "cl1",
> "parameters": {}
> },
> "traitNames": [],
> "traits": {}
> },
> "retention": 0,
> "tableName": "test_table",
> "columns": [
> {
> "jsonClass": 
> "org.apache.atlas.typesystem.json.InstanceSerialization$_Reference",
> "id": {
> "jsonClass": 
> "org.apache.atlas.typesystem.json.InstanceSerialization$_Id",
> "id": "-85198897676876",
> "version": 0,
> "typeName": "hive_column",
> "state": "ACTIVE"
> },
> "typeName": "hive_column",
> "values": {
> "name": "id1",
> "qualifiedName": "default.test_table.id1@cl1",
> "type": "int",
> "table": {
> "jsonClass": 
> "org.apache.atlas.typesystem.json.InstanceSerialization$_Id",
> "id": "-85198897495080",
> "version": 0,
> "typeName": "hive_table",
> "state": "ACTIVE"
> }
> },
>  

[jira] [Updated] (ATLAS-623) Atlas startup showing new tags(DataProducer and DataConsumer) added by default

[Erik Bergenholtz (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Erik Bergenholtz updated ATLAS-623:
---
Assignee: Hemanth Yamijala

> Atlas startup showing new tags(DataProducer and DataConsumer) added by default
> --
>
> Key: ATLAS-623
> URL: https://issues.apache.org/jira/browse/ATLAS-623
> Project: Atlas
>  Issue Type: Bug
>Affects Versions: trunk
>Reporter: Ayub Khan
>Assignee: Hemanth Yamijala
> Fix For: trunk
>
> Attachments: Screen Shot 2016-04-04 at 11.25.51 AM.png
>
>
> Two tags are added(DataProducer and DataConsumer) to atlas startup by default.
> This seems to be part of storm data model registration which happens in atlas 
> startup phase.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (ATLAS-865) Edit description functionality for Tags

[Keval Bhatt (JIRA)]

Keval Bhatt created ATLAS-865:
-

 Summary: Edit description functionality for Tags
 Key: ATLAS-865
 URL: https://issues.apache.org/jira/browse/ATLAS-865
 Project: Atlas
  Issue Type: Bug
Reporter: Keval Bhatt






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (ATLAS-680) Test With External HBase

[Erik Bergenholtz (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Erik Bergenholtz resolved ATLAS-680.

Resolution: Done

> Test With External HBase
> 
>
> Key: ATLAS-680
> URL: https://issues.apache.org/jira/browse/ATLAS-680
> Project: Atlas
>  Issue Type: Task
>Affects Versions: 0.7-incubating
>Reporter: Erik Bergenholtz
> Fix For: 0.7-incubating
>
>
> Now that we have made the change to incorporate Standalone HBase we need to 
> test that Atlas functions properly with a non Ambari provisioned HBase 
> with/without Kerberos.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (ATLAS-680) Test With External HBase

[Erik Bergenholtz (JIRA)]

[ 
https://issues.apache.org/jira/browse/ATLAS-680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15316455#comment-15316455
 ] 

Erik Bergenholtz commented on ATLAS-680:


I am resolving this issue since Atlas does work with an external HBase instance 
in both secure and non-secure mode.

> Test With External HBase
> 
>
> Key: ATLAS-680
> URL: https://issues.apache.org/jira/browse/ATLAS-680
> Project: Atlas
>  Issue Type: Task
>Affects Versions: 0.7-incubating
>Reporter: Erik Bergenholtz
> Fix For: 0.7-incubating
>
>
> Now that we have made the change to incorporate Standalone HBase we need to 
> test that Atlas functions properly with a non Ambari provisioned HBase 
> with/without Kerberos.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 48270: ATLAS-852 : Change Default landing page to taxonomy

[keval bhatt]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48270/
---

(Updated June 6, 2016, 12:18 p.m.)


Review request for atlas, Gautam Borad, Madhan Neethiraj, Shwetha GS, 
Selvamohan Neethiraj, and Hemanth Yamijala.


Bugs: ATLAS-852
https://issues.apache.org/jira/browse/ATLAS-852


Repository: atlas


Description
---

Changing default landing page to taxonomy.


Diffs
-

  dashboardv2/public/js/router/Router.js 4d89ab8 
  dashboardv2/public/js/templates/business_catalog/SideNavLayoutView_tmpl.html 
a59fb58 
  dashboardv2/public/js/views/business_catalog/BusinessCatalogLayoutView.js 
3e88b6d 
  dashboardv2/public/js/views/tag/TagLayoutView.js 03adfbc 

Diff: https://reviews.apache.org/r/48270/diff/


Testing (updated)
---

Verified default landing page.


Thanks,

keval bhatt

[jira] [Commented] (ATLAS-813) Maintain attributes inherited from parent Tags

[Keval Bhatt (JIRA)]

[ 
https://issues.apache.org/jira/browse/ATLAS-813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15316410#comment-15316410
 ] 

Keval Bhatt commented on ATLAS-813:
---

 Already handled in ATLAS-802 bug id

> Maintain attributes inherited from parent Tags
> --
>
> Key: ATLAS-813
> URL: https://issues.apache.org/jira/browse/ATLAS-813
> Project: Atlas
>  Issue Type: Bug
>Affects Versions: 0.7-incubating
>Reporter: Keval Bhatt
>Assignee: Keval Bhatt
> Fix For: 0.7-incubating
>
>
> As part of ATLAS-802 new UI layout has been created, however features which 
> were working previously needs to work in new UI. 
> If a Tag is created having any parent tag, then it should inherit all 
> attributes of parent tag.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (ATLAS-813) Maintain attributes inherited from parent Tags

[Keval Bhatt (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Keval Bhatt resolved ATLAS-813.
---
Resolution: Fixed

> Maintain attributes inherited from parent Tags
> --
>
> Key: ATLAS-813
> URL: https://issues.apache.org/jira/browse/ATLAS-813
> Project: Atlas
>  Issue Type: Bug
>Affects Versions: 0.7-incubating
>Reporter: Keval Bhatt
>Assignee: Keval Bhatt
> Fix For: 0.7-incubating
>
>
> As part of ATLAS-802 new UI layout has been created, however features which 
> were working previously needs to work in new UI. 
> If a Tag is created having any parent tag, then it should inherit all 
> attributes of parent tag.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (ATLAS-624) UI: Clicking a tag hyperlink should always result in DSL search. In some cases, results in full-text search.

[Erik Bergenholtz (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Erik Bergenholtz updated ATLAS-624:
---
Assignee: Keval Bhatt

> UI: Clicking a tag hyperlink should always result in DSL search. In some 
> cases, results in full-text search.
> 
>
> Key: ATLAS-624
> URL: https://issues.apache.org/jira/browse/ATLAS-624
> Project: Atlas
>  Issue Type: Bug
>Affects Versions: trunk
>Reporter: Ayub Khan
>Assignee: Keval Bhatt
> Fix For: 0.7-incubating
>
>
> UI: Clicking a tag hyperlink should always result in DSL search.
> 1. Create a hive table with a column, Attach a tag to the column.
> 2. Now navigate to the schema tab of hive_table search result.
> 3. Click on the attached tag - This will result in full-text query search of 
> that tag, which will always return empty response. This should be ideally a 
> DSL search query which will show all the entities associated with that tag.
> Ideally, this should be true in all cases. i.e; Whenever a tag hyperlink is 
> clicked on atlas UI, this should result in DSL search only.
> UI snapshot for reference: 
> https://monosnap.com/file/QKWcuSeG85UI3NDXakvR7iD9LJd1RC



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (ATLAS-852) Change Default landing page to taxonomy

[Keval Bhatt (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Keval Bhatt updated ATLAS-852:
--
Attachment: ATLAS-852.patch

> Change Default landing page to taxonomy
> ---
>
> Key: ATLAS-852
> URL: https://issues.apache.org/jira/browse/ATLAS-852
> Project: Atlas
>  Issue Type: Improvement
>Reporter: Keval Bhatt
>Assignee: Keval Bhatt
> Attachments: ATLAS-852.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (ATLAS-744) UI: HDFS_PATH DSL query results are showing href links for qualifiedName and Path

[Erik Bergenholtz (JIRA)]

[ 
https://issues.apache.org/jira/browse/ATLAS-744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15316445#comment-15316445
 ] 

Erik Bergenholtz commented on ATLAS-744:


This may already be resolved.

> UI: HDFS_PATH DSL query results are showing href links for qualifiedName and 
> Path
> -
>
> Key: ATLAS-744
> URL: https://issues.apache.org/jira/browse/ATLAS-744
> Project: Atlas
>  Issue Type: Bug
>Affects Versions: 0.7-incubating
>Reporter: Ayub Khan
>Assignee: Keval Bhatt
> Fix For: 0.7-incubating
>
>
> snapshot link: https://monosnap.com/file/TkmOEu6m77pWiia55wKXr9t5hcDfkR



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (ATLAS-744) UI: HDFS_PATH DSL query results are showing href links for qualifiedName and Path

[Erik Bergenholtz (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Erik Bergenholtz updated ATLAS-744:
---
Assignee: Keval Bhatt  (was: Erik Bergenholtz)

> UI: HDFS_PATH DSL query results are showing href links for qualifiedName and 
> Path
> -
>
> Key: ATLAS-744
> URL: https://issues.apache.org/jira/browse/ATLAS-744
> Project: Atlas
>  Issue Type: Bug
>Affects Versions: 0.7-incubating
>Reporter: Ayub Khan
>Assignee: Keval Bhatt
> Fix For: 0.7-incubating
>
>
> snapshot link: https://monosnap.com/file/TkmOEu6m77pWiia55wKXr9t5hcDfkR



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (ATLAS-753) UI: showing "viewOriginalText/viewExpandedText" field which is not part of hive table metadata message and the value will be always be null

[Erik Bergenholtz (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Erik Bergenholtz updated ATLAS-753:
---
Assignee: Keval Bhatt  (was: Erik Bergenholtz)

> UI: showing "viewOriginalText/viewExpandedText" field which is not part of 
> hive table metadata message and the value will be always be null
> ---
>
> Key: ATLAS-753
> URL: https://issues.apache.org/jira/browse/ATLAS-753
> Project: Atlas
>  Issue Type: Bug
>Affects Versions: 0.7-incubating
>Reporter: Ayub Khan
>Assignee: Keval Bhatt
>Priority: Minor
> Fix For: 0.7-incubating
>
>
> UI showing "viewOriginalText" field for hive_table which is not part of hive 
> metadata.
> {noformat}
> 0: jdbc:hive2://os-r6-apathan-atlas-erie-nose> describe formatted test_table;
> +---+-+---+--+
> |   col_name|  
> data_type  |comment|
> +---+-+---+--+
> | # col_name| data_type   
> | comment   |
> |   | NULL
> | NULL  |
> | abc   | string  
> |   |
> |   | NULL
> | NULL  |
> | # Partition Information   | NULL
> | NULL  |
> | # col_name| data_type   
> | comment   |
> |   | NULL
> | NULL  |
> | dt| date
> |   |
> |   | NULL
> | NULL  |
> | # Detailed Table Information  | NULL
> | NULL  |
> | Database: | default 
> | NULL  |
> | Owner:| hiveuser
> | NULL  |
> | CreateTime:   | Fri May 06 05:44:25 UTC 2016
> | NULL  |
> | LastAccessTime:   | UNKNOWN 
> | NULL  |
> | Protect Mode: | None
> | NULL  |
> | Retention:| 0   
> | NULL  |
> | Location: | 
> hdfs://os-r6-apathan-atlas-erie-nosec-2.openstacklocal:8020/apps/hive/warehouse/test_table
>   | NULL  |
> | Table Type:   | MANAGED_TABLE   
> | NULL  |
> | Table Parameters: | NULL
> | NULL  |
> |   | transient_lastDdlTime   
> | 1462513465|
> |   | NULL
> | NULL  |
> | # Storage Information | NULL
> | NULL  |
> | 

[jira] [Updated] (ATLAS-680) Test With External HBase

[Madhan Neethiraj (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Madhan Neethiraj updated ATLAS-680:
---
Assignee: Tom Beerbower

> Test With External HBase
> 
>
> Key: ATLAS-680
> URL: https://issues.apache.org/jira/browse/ATLAS-680
> Project: Atlas
>  Issue Type: Task
>Affects Versions: 0.7-incubating
>Reporter: Erik Bergenholtz
>Assignee: Tom Beerbower
> Fix For: 0.7-incubating
>
>
> Now that we have made the change to incorporate Standalone HBase we need to 
> test that Atlas functions properly with a non Ambari provisioned HBase 
> with/without Kerberos.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Review Request 48280: ATLAS-515: Ability to initialize Kafka topics with more than 1 replica

[Hemanth Yamijala]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48280/
---

Review request for atlas.


Bugs: ATLAS-515
https://issues.apache.org/jira/browse/ATLAS-515


Repository: atlas


Description
---

Attached patch provides a method to automatically create Kafka topics 
ATLAS_HOOK and ATLAS_ENTITIES with configured number of replicas.


Diffs
-

  addons/falcon-bridge/pom.xml 14c6090 
  addons/hive-bridge/pom.xml 47e72e8 
  addons/storm-bridge/pom.xml b1a7a9b 
  distro/src/conf/atlas-application.properties 1cdd424 
  notification/pom.xml b3738db 
  notification/src/main/java/org/apache/atlas/hook/AtlasHook.java 71029b0 
  notification/src/main/java/org/apache/atlas/hook/AtlasTopicCreator.java 
PRE-CREATION 
  notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 
1ee62d2 
  
notification/src/main/java/org/apache/atlas/notification/AbstractNotification.java
 cb44fc6 
  notification/src/test/java/org/apache/atlas/hook/AtlasTopicCreatorTest.java 
PRE-CREATION 
  pom.xml eac95c8 

Diff: https://reviews.apache.org/r/48280/diff/


Testing
---

* Started multiple Kafka brokers on single machine, configured number of 
replicas to be 2, started Atlas server and verified they got created correctly.
* Started multiple Kafka brokers on single machine, configured number of 
replicas to be 2, started Hive hook and verified it created ATLAS_HOOK 
correctly. Then started Atlas server and verified it by-passed ATLAS_HOOK 
creation, but did ATLAS_ENTITIES creation.
* Started one Kafka broker, onfigured number of replicas to be 2, started Atlas 
server and verified it received an exception as expected. But topic could still 
get auto-created and continued.
* Started 2 instances of Atlas server concurrently and verified topic creation 
is working fine.
* Existing UTs/ITs/new UTs pass.


Thanks,

Hemanth Yamijala

[jira] [Commented] (ATLAS-515) Ability to initialize Kafka topics with more than 1 replica

[ATLAS QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/ATLAS-515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15316660#comment-15316660
 ] 

ATLAS QA commented on ATLAS-515:


{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment
  http://issues.apache.org/jira/secure/attachment/12808398/ATLAS-515.patch
  against master revision d661964.

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 1 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

+1 checkstyle.  The patch generated 0 code style errors.

{color:red}-1 findbugs{color}.  The patch appears to introduce 378 new 
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:red}-1 core tests{color}.  The patch failed these unit tests in :
   
org.apache.atlas.repository.typestore.GraphBackedTypeStoreTest

Test results: 
https://builds.apache.org/job/PreCommit-ATLAS-Build/289//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-ATLAS-Build/289//artifact/patchprocess/newPatchFindbugsWarningsrepository.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-ATLAS-Build/289//artifact/patchprocess/newPatchFindbugsWarningscommon.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-ATLAS-Build/289//artifact/patchprocess/newPatchFindbugsWarningsnotification.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-ATLAS-Build/289//artifact/patchprocess/newPatchFindbugsWarningswebapp.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-ATLAS-Build/289//artifact/patchprocess/newPatchFindbugsWarningsclient.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-ATLAS-Build/289//artifact/patchprocess/newPatchFindbugsWarningstypesystem.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-ATLAS-Build/289//artifact/patchprocess/newPatchFindbugsWarningsfalcon-bridge.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-ATLAS-Build/289//artifact/patchprocess/newPatchFindbugsWarningssqoop-bridge.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-ATLAS-Build/289//artifact/patchprocess/newPatchFindbugsWarningshive-bridge.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-ATLAS-Build/289//artifact/patchprocess/newPatchFindbugsWarningsstorm-bridge.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-ATLAS-Build/289//artifact/patchprocess/newPatchFindbugsWarningshdfs-model.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-ATLAS-Build/289//artifact/patchprocess/newPatchFindbugsWarningsauthorization.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-ATLAS-Build/289//artifact/patchprocess/newPatchFindbugsWarningscatalog.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-ATLAS-Build/289//artifact/patchprocess/newPatchFindbugsWarningstitan.html
Console output: https://builds.apache.org/job/PreCommit-ATLAS-Build/289//console

This message is automatically generated.

> Ability to initialize Kafka topics with more than 1 replica
> ---
>
> Key: ATLAS-515
> URL: https://issues.apache.org/jira/browse/ATLAS-515
> Project: Atlas
>  Issue Type: Sub-task
>Reporter: Hemanth Yamijala
>Assignee: Hemanth Yamijala
> Fix For: 0.7-incubating
>
> Attachments: ATLAS-515-poc.patch, ATLAS-515.patch
>
>
> Atlas uses two Kafka topics - ATLAS_HOOK and ATLAS_ENTITIES for communicating 
> with its upstream and downstream dependencies. If not explicitly created, 
> these topics are setup with a single replica. From a HA perspective, this 
> becomes a problem. This JIRA is to request for a mechanism for creating 
> topics with more than 1 replica. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 48130: ATLAS-820- Kerberized env: Authentication failing

[Nixon Rodrigues]

> On June 2, 2016, 2:12 p.m., Hemanth Yamijala wrote:
> > webapp/src/main/java/org/apache/atlas/web/filters/AtlasAuthenticationFilter.java,
> >  line 287
> > 
> >
> > This function is too deeply nested and difficult to read. Can you 
> > please refactor it and make it more readable?

I am trying regex to simply this method. Its working in local program but not 
when integrated on server. will provide fix for this tommorow.


- Nixon


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48130/#review135929
---


On June 2, 2016, 8:59 a.m., Nixon Rodrigues wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/48130/
> ---
> 
> (Updated June 2, 2016, 8:59 a.m.)
> 
> 
> Review request for atlas, Gautam Borad, Madhan Neethiraj, Mehul Parikh, 
> Shwetha GS, and Hemanth Yamijala.
> 
> 
> Bugs: ATLAS-820
> https://issues.apache.org/jira/browse/ATLAS-820
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Kerberos-Authentication-related-changes.
> 
> Changes includes.
> 
> * Adding/Configaration of AtlasAuthentication fitler into spring's filter.
> * Refactoring of authentication related properties.
> 
> 
> Diffs
> -
> 
>   common/src/main/java/org/apache/atlas/utils/AuthenticationUtil.java f8e22f0 
>   distro/src/conf/atlas-application.properties bfa40e8 
>   distro/src/conf/policy-store.txt 339f014 
>   webapp/src/main/java/org/apache/atlas/web/dao/UserDao.java f1ceee2 
>   
> webapp/src/main/java/org/apache/atlas/web/filters/AtlasAuthenticationFilter.java
>  2d84b10 
>   
> webapp/src/main/java/org/apache/atlas/web/filters/KerberosAuthenticationFilter.java
>  PRE-CREATION 
>   webapp/src/main/java/org/apache/atlas/web/filters/MockServletContext.java 
> PRE-CREATION 
>   webapp/src/main/java/org/apache/atlas/web/listeners/GuiceServletConfig.java 
> 010fa2a 
>   webapp/src/main/java/org/apache/atlas/web/listeners/LoginProcessor.java 
> b7943e7 
>   
> webapp/src/main/java/org/apache/atlas/web/security/AtlasAuthenticationProvider.java
>  389a609 
>   webapp/src/main/resources/spring-security.xml bba054d 
>   
> webapp/src/test/java/org/apache/atlas/web/security/FileAuthenticationTest.java
>  a07874a 
> 
> Diff: https://reviews.apache.org/r/48130/diff/
> 
> 
> Testing
> ---
> 
> * Tested Quick Started in both kerberos and normal env.
> * mvn clean install.
> * Tested atlas UI in kerberized and non kerberized browser.
> * Executed curl commands with and without kinit.
> 
> kinit -kt /etc/security/keytabs/atlas.service.keytab 
> atlas/mp-atls-495-1.openstacklo...@example.com
> 
> curl url  curl -v -u admin:admin 
> http://mp-atls-495-1.openstacklocal:21000/api/atlas/types
> curl with –negotiate  curl -k -v  --negotiate -u :  
> http://mp-atls-495-1.openstacklocal:21000/api/atlas/types
> curl with without negotiate   curl -k -v   -u :  
> http://mp-atls-495-1.openstacklocal:21000/api/atlas/types
> curl with -u admin:admin  curl -v -u admin:admin 
> http://mp-atls-495-1.openstacklocal:21000/api/atlas/types
> with browser  google-chrome 
> –auth-server-whitelist="mp-atls-495-1.openstacklocal"
> 
> 
> Thanks,
> 
> Nixon Rodrigues
> 
>

Re: Review Request 47658: ATLAS-499: Update reverse references

[David Kantor]

> On June 3, 2016, 11:29 p.m., Suma Shivaprasad wrote:
> > repository/src/main/java/org/apache/atlas/repository/graph/TypedInstanceToGraphMapper.java,
> >  line 443
> > 
> >
> > for map types, can we check if the reverseAttributeName is set on the 
> > map type and remove it if set else ignore?

I'm thinking it may be better to remove the reverse attribute name when the 
type is registered and possibly issue a warning that bi-directional references 
are not supported for map types, rather then doing it every time there is an 
update operation involving that type.  Thoughts?


- David


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47658/#review136133
---


On June 3, 2016, 1:55 p.m., David Kantor wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47658/
> ---
> 
> (Updated June 3, 2016, 1:55 p.m.)
> 
> 
> Review request for atlas and Shwetha GS.
> 
> 
> Bugs: ATLAS-499
> https://issues.apache.org/jira/browse/ATLAS-499
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> ATLAS-499: Automatically update inverse references to prevent repository 
> corruption from unbalanced references.
> 
> 
> Diffs
> -
> 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/DeleteHandler.java 
> 91f9bd008a6939dfe78656f5c1845637a30ee9eb 
>   repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java 
> 4f6d0118072380342f5ea302d0aaf7902784849a 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/TypedInstanceToGraphMapper.java
>  4c1f5591f4adead41c3336e64b0bc956836f7edb 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepositoryDeleteTestBase.java
>  449e066a167ba4546b118a77c8e3de5fd99f077b 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/ReverseReferenceUpdateHardDeleteTest.java
>  PRE-CREATION 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/ReverseReferenceUpdateSoftDeleteTest.java
>  PRE-CREATION 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/ReverseReferenceUpdateTestBase.java
>  PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/47658/diff/
> 
> 
> Testing
> ---
> 
> Ran all unit and integration tests with no regressions.
> 
> 
> Thanks,
> 
> David Kantor
> 
>

Re: Review Request 48130: ATLAS-820- Kerberized env: Authentication failing

[Nixon Rodrigues]

> On June 2, 2016, 1:14 p.m., Hemanth Yamijala wrote:
> > distro/src/conf/policy-store.txt, line 8
> > 
> >
> > What is this policy for specifically? It seems like it is to grant 
> > access to members of the 'hadoop' group. Does this have any special 
> > significance?

Yes, its just to grant privilegde to hadoop user group. It can be removed after 
testing is over.


- Nixon


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48130/#review135923
---


On June 2, 2016, 8:59 a.m., Nixon Rodrigues wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/48130/
> ---
> 
> (Updated June 2, 2016, 8:59 a.m.)
> 
> 
> Review request for atlas, Gautam Borad, Madhan Neethiraj, Mehul Parikh, 
> Shwetha GS, and Hemanth Yamijala.
> 
> 
> Bugs: ATLAS-820
> https://issues.apache.org/jira/browse/ATLAS-820
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Kerberos-Authentication-related-changes.
> 
> Changes includes.
> 
> * Adding/Configaration of AtlasAuthentication fitler into spring's filter.
> * Refactoring of authentication related properties.
> 
> 
> Diffs
> -
> 
>   common/src/main/java/org/apache/atlas/utils/AuthenticationUtil.java f8e22f0 
>   distro/src/conf/atlas-application.properties bfa40e8 
>   distro/src/conf/policy-store.txt 339f014 
>   webapp/src/main/java/org/apache/atlas/web/dao/UserDao.java f1ceee2 
>   
> webapp/src/main/java/org/apache/atlas/web/filters/AtlasAuthenticationFilter.java
>  2d84b10 
>   
> webapp/src/main/java/org/apache/atlas/web/filters/KerberosAuthenticationFilter.java
>  PRE-CREATION 
>   webapp/src/main/java/org/apache/atlas/web/filters/MockServletContext.java 
> PRE-CREATION 
>   webapp/src/main/java/org/apache/atlas/web/listeners/GuiceServletConfig.java 
> 010fa2a 
>   webapp/src/main/java/org/apache/atlas/web/listeners/LoginProcessor.java 
> b7943e7 
>   
> webapp/src/main/java/org/apache/atlas/web/security/AtlasAuthenticationProvider.java
>  389a609 
>   webapp/src/main/resources/spring-security.xml bba054d 
>   
> webapp/src/test/java/org/apache/atlas/web/security/FileAuthenticationTest.java
>  a07874a 
> 
> Diff: https://reviews.apache.org/r/48130/diff/
> 
> 
> Testing
> ---
> 
> * Tested Quick Started in both kerberos and normal env.
> * mvn clean install.
> * Tested atlas UI in kerberized and non kerberized browser.
> * Executed curl commands with and without kinit.
> 
> kinit -kt /etc/security/keytabs/atlas.service.keytab 
> atlas/mp-atls-495-1.openstacklo...@example.com
> 
> curl url  curl -v -u admin:admin 
> http://mp-atls-495-1.openstacklocal:21000/api/atlas/types
> curl with –negotiate  curl -k -v  --negotiate -u :  
> http://mp-atls-495-1.openstacklocal:21000/api/atlas/types
> curl with without negotiate   curl -k -v   -u :  
> http://mp-atls-495-1.openstacklocal:21000/api/atlas/types
> curl with -u admin:admin  curl -v -u admin:admin 
> http://mp-atls-495-1.openstacklocal:21000/api/atlas/types
> with browser  google-chrome 
> –auth-server-whitelist="mp-atls-495-1.openstacklocal"
> 
> 
> Thanks,
> 
> Nixon Rodrigues
> 
>

Re: Review Request 48130: ATLAS-820- Kerberized env: Authentication failing

[Nixon Rodrigues]

> On June 4, 2016, 12:32 a.m., Madhan Neethiraj wrote:
> > webapp/src/main/java/org/apache/atlas/web/filters/AtlasAuthenticationFilter.java,
> >  line 155
> > 
> >
> > Should "simple" authentication be supported now?
> > 
> > Shouldn't the flow be:
> >  1. if kerberos authn is enabled
> >  - try kerberos auth
> >  2. if unauthenticated && ldap authn is enabled
> >  - try ldap authn
> >  3. if unauthenticated && file authn is enabled
> >  - try file authn
> >  4. if unauthenticated
> >  - fail authentication

This filter is dedicated for for Kerberos and Simple authentication and for its 
initialization requires anyone of the type.
For non kerberos it will be initialized with simple type but it will only send 
authentication request only in case of Kerberos auth enabled.
This Filter is at top and will handle kerberos authentication, the filters 
below in chain will take of authentication for form based and basic auth.


> On June 4, 2016, 12:32 a.m., Madhan Neethiraj wrote:
> > webapp/src/main/java/org/apache/atlas/web/listeners/LoginProcessor.java, 
> > line 103
> > 
> >
> > Should "simple" authentication be supported now?
> > 
> > Shouldn't the flow be:
> >  1. if kerberos authn is enabled
> >  - try kerberos auth
> >  2. if unauthenticated && ldap authn is enabled
> >  - try ldap authn
> >  3. if unauthenticated && file authn is enabled
> >  - try file authn
> >  4. if unauthenticated
> >  - fail authentication

This LoginProcessor is called on initialization of Atlas Server and it does a 
doServiceLogin, not sure whether it is required . Again here Simple Or Kerberos 
type are required for as auth type by design. Should we call this methos only 
for Kerberos type ?.


- Nixon


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48130/#review136143
---


On June 2, 2016, 8:59 a.m., Nixon Rodrigues wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/48130/
> ---
> 
> (Updated June 2, 2016, 8:59 a.m.)
> 
> 
> Review request for atlas, Gautam Borad, Madhan Neethiraj, Mehul Parikh, 
> Shwetha GS, and Hemanth Yamijala.
> 
> 
> Bugs: ATLAS-820
> https://issues.apache.org/jira/browse/ATLAS-820
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Kerberos-Authentication-related-changes.
> 
> Changes includes.
> 
> * Adding/Configaration of AtlasAuthentication fitler into spring's filter.
> * Refactoring of authentication related properties.
> 
> 
> Diffs
> -
> 
>   common/src/main/java/org/apache/atlas/utils/AuthenticationUtil.java f8e22f0 
>   distro/src/conf/atlas-application.properties bfa40e8 
>   distro/src/conf/policy-store.txt 339f014 
>   webapp/src/main/java/org/apache/atlas/web/dao/UserDao.java f1ceee2 
>   
> webapp/src/main/java/org/apache/atlas/web/filters/AtlasAuthenticationFilter.java
>  2d84b10 
>   
> webapp/src/main/java/org/apache/atlas/web/filters/KerberosAuthenticationFilter.java
>  PRE-CREATION 
>   webapp/src/main/java/org/apache/atlas/web/filters/MockServletContext.java 
> PRE-CREATION 
>   webapp/src/main/java/org/apache/atlas/web/listeners/GuiceServletConfig.java 
> 010fa2a 
>   webapp/src/main/java/org/apache/atlas/web/listeners/LoginProcessor.java 
> b7943e7 
>   
> webapp/src/main/java/org/apache/atlas/web/security/AtlasAuthenticationProvider.java
>  389a609 
>   webapp/src/main/resources/spring-security.xml bba054d 
>   
> webapp/src/test/java/org/apache/atlas/web/security/FileAuthenticationTest.java
>  a07874a 
> 
> Diff: https://reviews.apache.org/r/48130/diff/
> 
> 
> Testing
> ---
> 
> * Tested Quick Started in both kerberos and normal env.
> * mvn clean install.
> * Tested atlas UI in kerberized and non kerberized browser.
> * Executed curl commands with and without kinit.
> 
> kinit -kt /etc/security/keytabs/atlas.service.keytab 
> atlas/mp-atls-495-1.openstacklo...@example.com
> 
> curl url  curl -v -u admin:admin 
> http://mp-atls-495-1.openstacklocal:21000/api/atlas/types
> curl with –negotiate  curl -k -v  --negotiate -u :  
> http://mp-atls-495-1.openstacklocal:21000/api/atlas/types
> curl with without negotiate   curl -k -v   -u :  
> http://mp-atls-495-1.openstacklocal:21000/api/atlas/types
> curl with -u admin:admin  curl -v -u admin:admin 
> http://mp-atls-495-1.openstacklocal:21000/api/atlas/types
> with browser  google-chrome 
> –auth-server-whitelist="mp-atls-495-1.openstacklocal"
> 
> 
> Thanks,
> 
> Nixon Rodrigues
> 
>

Re: Review Request 47658: ATLAS-499: Update reverse references

[David Kantor]

> On June 3, 2016, 11:25 p.m., Suma Shivaprasad wrote:
> > repository/src/main/java/org/apache/atlas/repository/graph/TypedInstanceToGraphMapper.java,
> >  line 785
> > 
> >
> > do we need the HashSet since contains and all will both O(n)

HashSet.contains is O(1).  I am using the HashSet to avoid the O(n) performance 
of ArrayList.contains.  For a very small data set, the difference between 
HashSet.contains and ArrayList.contains is probably negligible.  But in my 
experience doing Java performance analysis on our legacy metadata repository, 
ArrayList.contains() often ended up being a major hot spot when enforcing 
uniqueness for high cardinality references.


- David


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47658/#review136132
---


On June 3, 2016, 1:55 p.m., David Kantor wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47658/
> ---
> 
> (Updated June 3, 2016, 1:55 p.m.)
> 
> 
> Review request for atlas and Shwetha GS.
> 
> 
> Bugs: ATLAS-499
> https://issues.apache.org/jira/browse/ATLAS-499
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> ATLAS-499: Automatically update inverse references to prevent repository 
> corruption from unbalanced references.
> 
> 
> Diffs
> -
> 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/DeleteHandler.java 
> 91f9bd008a6939dfe78656f5c1845637a30ee9eb 
>   repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java 
> 4f6d0118072380342f5ea302d0aaf7902784849a 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/TypedInstanceToGraphMapper.java
>  4c1f5591f4adead41c3336e64b0bc956836f7edb 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepositoryDeleteTestBase.java
>  449e066a167ba4546b118a77c8e3de5fd99f077b 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/ReverseReferenceUpdateHardDeleteTest.java
>  PRE-CREATION 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/ReverseReferenceUpdateSoftDeleteTest.java
>  PRE-CREATION 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/ReverseReferenceUpdateTestBase.java
>  PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/47658/diff/
> 
> 
> Testing
> ---
> 
> Ran all unit and integration tests with no regressions.
> 
> 
> Thanks,
> 
> David Kantor
> 
>

[jira] [Resolved] (ATLAS-863) 1 table out of 50,000 tables is left unimported throwing exception during deserialization

[Sharmadha Sainath (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sharmadha Sainath resolved ATLAS-863.
-
Resolution: Duplicate

Duplicate of ATLAS-861 created due to JIRA connectivity issues.

> 1 table out of 50,000 tables is left unimported throwing exception during 
> deserialization
> -
>
> Key: ATLAS-863
> URL: https://issues.apache.org/jira/browse/ATLAS-863
> Project: Atlas
>  Issue Type: Bug
> Environment: Atlas commit id : 
> 59268875d256c92f748198cea999ed5057c6bf18
> Cluster setup :
> Machine 1 : Atlas, Solr
> Machine 2 : HBase, Kafka
> Machine 3 : Client
> atlas.notification.hook.numthreads=5
> ATLAS_HOOK topic created with 5 partitions
>Reporter: Sharmadha Sainath
>
> On importing 50,000 tables using hiveserver2, 1 table is not imported 
> (table_7770). Below is the exception stack trace :
> java.lang.NumberFormatException: For input string: ""
>   at 
> java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
>   at java.lang.Long.parseLong(Long.java:453)
>   at java.lang.Long.parseLong(Long.java:483)
>   at java.text.DigitList.getLong(DigitList.java:194)
>   at java.text.DecimalFormat.parse(DecimalFormat.java:1316)
>   at java.text.SimpleDateFormat.subParse(SimpleDateFormat.java:1793)
>   at java.text.SimpleDateFormat.parse(SimpleDateFormat.java:1455)
>   at java.text.DateFormat.parse(DateFormat.java:355)
>   at 
> org.apache.atlas.typesystem.types.DataTypes$DateType.convert(DataTypes.java:415)
>   at 
> org.apache.atlas.typesystem.types.DataTypes$DateType.convert(DataTypes.java:400)
>   at 
> org.apache.atlas.typesystem.persistence.StructInstance.set(StructInstance.java:121)
>   at 
> org.apache.atlas.typesystem.types.ClassType.convert(ClassType.java:143)
>   at 
> org.apache.atlas.services.DefaultMetadataService.getTypedReferenceableInstance(DefaultMetadataService.java:356)
>   at 
> org.apache.atlas.services.DefaultMetadataService.deserializeClassInstances(DefaultMetadataService.java:332)
>   at 
> org.apache.atlas.services.DefaultMetadataService.updateEntities(DefaultMetadataService.java:426)
>   at 
> org.apache.atlas.web.resources.EntityResource.updateEntities(EntityResource.java:167)
>   at org.apache.atlas.LocalAtlasClient$2.invoke(LocalAtlasClient.java:97)
>   at 
> org.apache.atlas.LocalAtlasClient$EntityOperation.run(LocalAtlasClient.java:118)
>   at 
> org.apache.atlas.LocalAtlasClient.updateEntities(LocalAtlasClient.java:100)
>   at org.apache.atlas.AtlasClient.updateEntities(AtlasClient.java:684)
>   at 
> org.apache.atlas.notification.NotificationHookConsumer$HookConsumer.handleMessage(NotificationHookConsumer.java:225)
>   at 
> org.apache.atlas.notification.NotificationHookConsumer$HookConsumer.run(NotificationHookConsumer.java:187)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:745)
> 2016-06-03 05:59:28,049 ERROR - [NotificationHookConsumer thread-1:] ~ Unable 
> to persist entity instance (EntityResource:177)
> java.lang.IllegalArgumentException: Unable to deserialize json
>   at 
> org.apache.atlas.services.DefaultMetadataService.deserializeClassInstances(DefaultMetadataService.java:340)
>   at 
> org.apache.atlas.services.DefaultMetadataService.updateEntities(DefaultMetadataService.java:426)
>   at 
> org.apache.atlas.web.resources.EntityResource.updateEntities(EntityResource.java:167)
>   at org.apache.atlas.LocalAtlasClient$2.invoke(LocalAtlasClient.java:97)
>   at 
> org.apache.atlas.LocalAtlasClient$EntityOperation.run(LocalAtlasClient.java:118)
>   at 
> org.apache.atlas.LocalAtlasClient.updateEntities(LocalAtlasClient.java:100)
>   at org.apache.atlas.AtlasClient.updateEntities(AtlasClient.java:684)
>   at 
> org.apache.atlas.notification.NotificationHookConsumer$HookConsumer.handleMessage(NotificationHookConsumer.java:225)
>   at 
> org.apache.atlas.notification.NotificationHookConsumer$HookConsumer.run(NotificationHookConsumer.java:187)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:745)
> Caused 

Re: Review Request 47810: ATLAS-694: Update Atlas to use Graph DB abstraction layer

[Jeff Hagelberg]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47810/
---

(Updated June 6, 2016, 7:19 p.m.)


Review request for atlas, David Kantor and Neeru Gupta.


Changes
---

Rebase, apply latest changes to graph database abstraction layer.


Bugs: ATLAS-694
https://issues.apache.org/jira/browse/ATLAS-694


Repository: atlas


Description
---

ATLAS-694: Update Atlas to use abstraction layer.  All of the Atlas code (with 
the exception of the catalog, which was only updated minimally) has been 
updated to use the graph database abstraction layer.  In addition, there is now 
a required Atlas configuration property that specifies the class of the 
abstraction layer database to use.  I basically put all of the changes in here 
with the exception of the actual Titan 1 implementation of code.  This includes 
the changes to support Tinkerpop 3 syntax.  This is mostly to expedite getting 
the changes into Atlas.  Originally the TP3 changes were going to be brought in 
as part of the Titan 1 implementation task.

Change Summary:

   - change Atlas classes to use AtlasGraph,AtlasVertex,AtlasEdge, etc instead 
of TitanGraph/Vertex/Edge, etc
   - compile time dependency on titan 0.5.4/TP 2 removed (except in Catalog, 
which was only changed to use AtlasGraphProvider/AtlasGraph)
   - updated DSL translation to generate Gremlin that is compliant with TP3 
when TP3 is being used
   - TitanGraphProvider replaced by AtlasGraphProvider.  Graph database 
implementation is determined from a new required configuration property
   - search indexer enhanced to wait for all indices to be enabled
   - HiveTitanSample is no longer used by tests.  It has been replaced by 
hive-instances.json (which uses normal Atlas json syntax).  The data is saved 
with a new JSONImporter class.  This was needed because the graphson syntax 
used by HiveTitanSample is not compatible with TP3.  
   - RepositoryMetaModule - GraphBackedSearchIndexer changed to an eager 
singleton to force index creation to happen before we execute any gremlin 
queries (which happens during the type system restoration triggered by the 
DefaultMetadataService constructor)
   - atlas_config.py - minor change to allow Atlas to use a non-default JVM.  
It now supports having the java home in ATLAS_JAVA_HOME.  If that is not set, 
we fall back to the previous behavior looking for JAVA_HOME and then checking 
the path.
   
Note - the diff here is cumulative and includes the changes from ATLAS-693.  I 
could not figure out how to remove those.

Last rebase: 5/27/2016


Diffs (updated)
-

  addons/falcon-bridge/pom.xml 14c60901a6f194497ca0fb7929a032d2261f85fa 
  addons/hdfs-model/pom.xml f4033bce0fe3ea5a33cfd32ebccd47c01207dc35 
  addons/hdfs-model/src/test/java/org/apache/atlas/fs/model/HDFSModelTest.java 
ac60294e328835ba0340e150799ddfb348ccdb52 
  addons/hive-bridge/pom.xml 47e72e814a421f03b00a9b7c6a13163e1b67ca14 
  addons/sqoop-bridge/pom.xml 343bb4eeae93031ab24cd48ed7b535769150d6c7 
  addons/storm-bridge/pom.xml b1a7a9b1a88442d9e65db4a01b8388c8d3522bfe 
  authorization/pom.xml 74c433bf740a346d1d907a868605e9ab457151bb 
  catalog/pom.xml 8a49d3d673a5936613d45e53479aa99a5a14517d 
  catalog/src/main/java/org/apache/atlas/catalog/DefaultTypeSystem.java 
a28a32b692e61377479cdd432f39ba3c60bd5238 
  catalog/src/main/java/org/apache/atlas/catalog/query/BaseQuery.java 
ba8e0e712769afad72f7f8254ec626a1d17136e8 
  
catalog/src/test/java/org/apache/atlas/catalog/query/AtlasEntityQueryTest.java 
149134c8a79c802fe625d64c5e04af0d88174cfa 
  distro/src/bin/atlas_config.py fab4046345e05e0b7749ca75834c8bdcb0d01752 
  distro/src/conf/atlas-application.properties 
1cdd424d8e126a6bdbf99db8bf4e5d4329e806bb 
  graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasEdge.java 
71b577b5c2e8aee8fec816b9224672ebc2cad5b3 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasEdgeDirection.java
 e7da1c9a5075004973ba68b8a36a2c95756e0e62 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasElement.java 
3c41693e531eb523f04a5c5781f1f123ab38ef07 
  graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 
f312117af9cb6ae8de21d565bfcf3b10d9bdfcc6 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphIndex.java
 PRE-CREATION 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphManagement.java
 2776b2242c4469dcae344616ccfe100633a79432 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphQuery.java
 f0dca9a28eebd8996d55d71dcc3254cce9cd4098 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasIndexQuery.java
 7ee6b2831a7610e7d5b1d36ae69282a9315dc06b 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasPropertyKey.java
 PRE-CREATION 
  

[jira] [Updated] (ATLAS-861) 1 table out of 50,000 tables is left unimported throwing exception during deserialization

[Sharmadha Sainath (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sharmadha Sainath updated ATLAS-861:

Attachment: Kakfa Message table_7770.rtf

Attached the message sent to Atlas topic and the Deserialization error.

> 1 table out of 50,000 tables is left unimported throwing exception during 
> deserialization
> -
>
> Key: ATLAS-861
> URL: https://issues.apache.org/jira/browse/ATLAS-861
> Project: Atlas
>  Issue Type: Bug
> Environment: Atlas commit id : 
> 59268875d256c92f748198cea999ed5057c6bf18
> Cluster setup :
> Machine 1 : Atlas, Solr
> Machine 2 : HBase, Kafka
> Machine 3 : Client
> atlas.notification.hook.numthreads=5
> ATLAS_HOOK topic created with 5 partitions
>Reporter: Sharmadha Sainath
> Attachments: Kakfa Message table_7770.rtf
>
>
> On importing 50,000 tables using hiveserver2, 1 table is not imported 
> (table_7770). Below is the exception stack trace :
> java.lang.NumberFormatException: For input string: ""
>   at 
> java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
>   at java.lang.Long.parseLong(Long.java:453)
>   at java.lang.Long.parseLong(Long.java:483)
>   at java.text.DigitList.getLong(DigitList.java:194)
>   at java.text.DecimalFormat.parse(DecimalFormat.java:1316)
>   at java.text.SimpleDateFormat.subParse(SimpleDateFormat.java:1793)
>   at java.text.SimpleDateFormat.parse(SimpleDateFormat.java:1455)
>   at java.text.DateFormat.parse(DateFormat.java:355)
>   at 
> org.apache.atlas.typesystem.types.DataTypes$DateType.convert(DataTypes.java:415)
>   at 
> org.apache.atlas.typesystem.types.DataTypes$DateType.convert(DataTypes.java:400)
>   at 
> org.apache.atlas.typesystem.persistence.StructInstance.set(StructInstance.java:121)
>   at 
> org.apache.atlas.typesystem.types.ClassType.convert(ClassType.java:143)
>   at 
> org.apache.atlas.services.DefaultMetadataService.getTypedReferenceableInstance(DefaultMetadataService.java:356)
>   at 
> org.apache.atlas.services.DefaultMetadataService.deserializeClassInstances(DefaultMetadataService.java:332)
>   at 
> org.apache.atlas.services.DefaultMetadataService.updateEntities(DefaultMetadataService.java:426)
>   at 
> org.apache.atlas.web.resources.EntityResource.updateEntities(EntityResource.java:167)
>   at org.apache.atlas.LocalAtlasClient$2.invoke(LocalAtlasClient.java:97)
>   at 
> org.apache.atlas.LocalAtlasClient$EntityOperation.run(LocalAtlasClient.java:118)
>   at 
> org.apache.atlas.LocalAtlasClient.updateEntities(LocalAtlasClient.java:100)
>   at org.apache.atlas.AtlasClient.updateEntities(AtlasClient.java:684)
>   at 
> org.apache.atlas.notification.NotificationHookConsumer$HookConsumer.handleMessage(NotificationHookConsumer.java:225)
>   at 
> org.apache.atlas.notification.NotificationHookConsumer$HookConsumer.run(NotificationHookConsumer.java:187)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:745)
> 2016-06-03 05:59:28,049 ERROR - [NotificationHookConsumer thread-1:] ~ Unable 
> to persist entity instance (EntityResource:177)
> java.lang.IllegalArgumentException: Unable to deserialize json
>   at 
> org.apache.atlas.services.DefaultMetadataService.deserializeClassInstances(DefaultMetadataService.java:340)
>   at 
> org.apache.atlas.services.DefaultMetadataService.updateEntities(DefaultMetadataService.java:426)
>   at 
> org.apache.atlas.web.resources.EntityResource.updateEntities(EntityResource.java:167)
>   at org.apache.atlas.LocalAtlasClient$2.invoke(LocalAtlasClient.java:97)
>   at 
> org.apache.atlas.LocalAtlasClient$EntityOperation.run(LocalAtlasClient.java:118)
>   at 
> org.apache.atlas.LocalAtlasClient.updateEntities(LocalAtlasClient.java:100)
>   at org.apache.atlas.AtlasClient.updateEntities(AtlasClient.java:684)
>   at 
> org.apache.atlas.notification.NotificationHookConsumer$HookConsumer.handleMessage(NotificationHookConsumer.java:225)
>   at 
> org.apache.atlas.notification.NotificationHookConsumer$HookConsumer.run(NotificationHookConsumer.java:187)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> 

[jira] [Updated] (ATLAS-694) Update Atlas code to use abstraction layer

[Jeffrey Hagelberg (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeffrey Hagelberg updated ATLAS-694:

Attachment: (was: rb47810.patch)

> Update Atlas code to use abstraction layer
> --
>
> Key: ATLAS-694
> URL: https://issues.apache.org/jira/browse/ATLAS-694
> Project: Atlas
>  Issue Type: Sub-task
>Reporter: Jeffrey Hagelberg
>Assignee: Jeffrey Hagelberg
>
> In this task, we will update Atlas to replace the direct usage of titan 
> classes with usage of the equivalent interfaces in the abstraction layer.  At 
> this point, there will be no options to configure the abstraction layer 
> implementation that gets used.  Atlas will be hard-coded to use the titan 
> 0.5.4 implementation.  It will also continue to have a compile time 
> dependency on the titan 0.5.4 graph database implementation.  When this task 
> is complete, Atlas will only be accessing titan through the abstraction layer.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (ATLAS-694) Update Atlas code to use abstraction layer

[Jeffrey Hagelberg (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeffrey Hagelberg updated ATLAS-694:

Attachment: (was: rb47810.patch)

> Update Atlas code to use abstraction layer
> --
>
> Key: ATLAS-694
> URL: https://issues.apache.org/jira/browse/ATLAS-694
> Project: Atlas
>  Issue Type: Sub-task
>Reporter: Jeffrey Hagelberg
>Assignee: Jeffrey Hagelberg
> Attachments: rb47810.patch
>
>
> In this task, we will update Atlas to replace the direct usage of titan 
> classes with usage of the equivalent interfaces in the abstraction layer.  At 
> this point, there will be no options to configure the abstraction layer 
> implementation that gets used.  Atlas will be hard-coded to use the titan 
> 0.5.4 implementation.  It will also continue to have a compile time 
> dependency on the titan 0.5.4 graph database implementation.  When this task 
> is complete, Atlas will only be accessing titan through the abstraction layer.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (ATLAS-694) Update Atlas code to use abstraction layer

[Jeffrey Hagelberg (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeffrey Hagelberg updated ATLAS-694:

Attachment: rb47810.patch

> Update Atlas code to use abstraction layer
> --
>
> Key: ATLAS-694
> URL: https://issues.apache.org/jira/browse/ATLAS-694
> Project: Atlas
>  Issue Type: Sub-task
>Reporter: Jeffrey Hagelberg
>Assignee: Jeffrey Hagelberg
> Attachments: rb47810.patch
>
>
> In this task, we will update Atlas to replace the direct usage of titan 
> classes with usage of the equivalent interfaces in the abstraction layer.  At 
> this point, there will be no options to configure the abstraction layer 
> implementation that gets used.  Atlas will be hard-coded to use the titan 
> 0.5.4 implementation.  It will also continue to have a compile time 
> dependency on the titan 0.5.4 graph database implementation.  When this task 
> is complete, Atlas will only be accessing titan through the abstraction layer.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (ATLAS-862) 1 table out of 50,000 tables is left unimported throwing exception during deserialization

[Sharmadha Sainath (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sharmadha Sainath resolved ATLAS-862.
-
Resolution: Duplicate

Duplicate of ATLAS-861 created due to JIRA connectivity issues.

> 1 table out of 50,000 tables is left unimported throwing exception during 
> deserialization
> -
>
> Key: ATLAS-862
> URL: https://issues.apache.org/jira/browse/ATLAS-862
> Project: Atlas
>  Issue Type: Bug
> Environment: Atlas commit id : 
> 59268875d256c92f748198cea999ed5057c6bf18
> Cluster setup :
> Machine 1 : Atlas, Solr
> Machine 2 : HBase, Kafka
> Machine 3 : Client
> atlas.notification.hook.numthreads=5
> ATLAS_HOOK topic created with 5 partitions
>Reporter: Sharmadha Sainath
>
> On importing 50,000 tables using hiveserver2, 1 table is not imported 
> (table_7770). Below is the exception stack trace :
> java.lang.NumberFormatException: For input string: ""
>   at 
> java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
>   at java.lang.Long.parseLong(Long.java:453)
>   at java.lang.Long.parseLong(Long.java:483)
>   at java.text.DigitList.getLong(DigitList.java:194)
>   at java.text.DecimalFormat.parse(DecimalFormat.java:1316)
>   at java.text.SimpleDateFormat.subParse(SimpleDateFormat.java:1793)
>   at java.text.SimpleDateFormat.parse(SimpleDateFormat.java:1455)
>   at java.text.DateFormat.parse(DateFormat.java:355)
>   at 
> org.apache.atlas.typesystem.types.DataTypes$DateType.convert(DataTypes.java:415)
>   at 
> org.apache.atlas.typesystem.types.DataTypes$DateType.convert(DataTypes.java:400)
>   at 
> org.apache.atlas.typesystem.persistence.StructInstance.set(StructInstance.java:121)
>   at 
> org.apache.atlas.typesystem.types.ClassType.convert(ClassType.java:143)
>   at 
> org.apache.atlas.services.DefaultMetadataService.getTypedReferenceableInstance(DefaultMetadataService.java:356)
>   at 
> org.apache.atlas.services.DefaultMetadataService.deserializeClassInstances(DefaultMetadataService.java:332)
>   at 
> org.apache.atlas.services.DefaultMetadataService.updateEntities(DefaultMetadataService.java:426)
>   at 
> org.apache.atlas.web.resources.EntityResource.updateEntities(EntityResource.java:167)
>   at org.apache.atlas.LocalAtlasClient$2.invoke(LocalAtlasClient.java:97)
>   at 
> org.apache.atlas.LocalAtlasClient$EntityOperation.run(LocalAtlasClient.java:118)
>   at 
> org.apache.atlas.LocalAtlasClient.updateEntities(LocalAtlasClient.java:100)
>   at org.apache.atlas.AtlasClient.updateEntities(AtlasClient.java:684)
>   at 
> org.apache.atlas.notification.NotificationHookConsumer$HookConsumer.handleMessage(NotificationHookConsumer.java:225)
>   at 
> org.apache.atlas.notification.NotificationHookConsumer$HookConsumer.run(NotificationHookConsumer.java:187)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:745)
> 2016-06-03 05:59:28,049 ERROR - [NotificationHookConsumer thread-1:] ~ Unable 
> to persist entity instance (EntityResource:177)
> java.lang.IllegalArgumentException: Unable to deserialize json
>   at 
> org.apache.atlas.services.DefaultMetadataService.deserializeClassInstances(DefaultMetadataService.java:340)
>   at 
> org.apache.atlas.services.DefaultMetadataService.updateEntities(DefaultMetadataService.java:426)
>   at 
> org.apache.atlas.web.resources.EntityResource.updateEntities(EntityResource.java:167)
>   at org.apache.atlas.LocalAtlasClient$2.invoke(LocalAtlasClient.java:97)
>   at 
> org.apache.atlas.LocalAtlasClient$EntityOperation.run(LocalAtlasClient.java:118)
>   at 
> org.apache.atlas.LocalAtlasClient.updateEntities(LocalAtlasClient.java:100)
>   at org.apache.atlas.AtlasClient.updateEntities(AtlasClient.java:684)
>   at 
> org.apache.atlas.notification.NotificationHookConsumer$HookConsumer.handleMessage(NotificationHookConsumer.java:225)
>   at 
> org.apache.atlas.notification.NotificationHookConsumer$HookConsumer.run(NotificationHookConsumer.java:187)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:745)
> Caused 

[jira] [Updated] (ATLAS-694) Update Atlas code to use abstraction layer

[Jeffrey Hagelberg (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeffrey Hagelberg updated ATLAS-694:

Attachment: rb47810.patch

Update patch

> Update Atlas code to use abstraction layer
> --
>
> Key: ATLAS-694
> URL: https://issues.apache.org/jira/browse/ATLAS-694
> Project: Atlas
>  Issue Type: Sub-task
>Reporter: Jeffrey Hagelberg
>Assignee: Jeffrey Hagelberg
> Attachments: rb47810.patch
>
>
> In this task, we will update Atlas to replace the direct usage of titan 
> classes with usage of the equivalent interfaces in the abstraction layer.  At 
> this point, there will be no options to configure the abstraction layer 
> implementation that gets used.  Atlas will be hard-coded to use the titan 
> 0.5.4 implementation.  It will also continue to have a compile time 
> dependency on the titan 0.5.4 graph database implementation.  When this task 
> is complete, Atlas will only be accessing titan through the abstraction layer.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (ATLAS-858) Unable to delete terms via API which are 3 or more levels deep

[John Speidel (JIRA)]

[ 
https://issues.apache.org/jira/browse/ATLAS-858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15316952#comment-15316952
 ] 

John Speidel commented on ATLAS-858:


Yes, that also works, and is an acceptable use (shorter url's) but don't 
provide for an explicit hierarchy. The url's with the explicit hierarchy are 
the "official" url's which are returned by the api to the user.

> Unable to delete terms via API which are 3 or more levels deep
> --
>
> Key: ATLAS-858
> URL: https://issues.apache.org/jira/browse/ATLAS-858
> Project: Atlas
>  Issue Type: Bug
>Affects Versions: 0.7-incubating
>Reporter: John Speidel
>Assignee: John Speidel
> Fix For: 0.7-incubating
>
> Attachments: ATLAS-858.patch
>
>
> Attempting to delete a term that is 3 or more levels deep results in an error 
> response: 
> DELETE 
> api/atlas/v1/taxonomies/taxonomy1/terms/term1/terms/term11/terms/term111
> 405 - Method Not Allowed
> This is because I for got to add a service method for deleting sub terms.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (ATLAS-694) Update Atlas code to use abstraction layer

[ATLAS QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/ATLAS-694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15317140#comment-15317140
 ] 

ATLAS QA commented on ATLAS-694:


{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment
  http://issues.apache.org/jira/secure/attachment/12808451/rb47810.patch
  against master revision d661964.

{color:red}-1 @author{color}.  The patch appears to contain 1 @author tags 
which the Atlas community has agreed to not allow in code contributions.

{color:green}+1 tests included{color}.  The patch appears to include 35 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

+1 checkstyle.  The patch generated 0 code style errors.

{color:red}-1 findbugs{color}.  The patch appears to introduce 383 new 
Findbugs (version 2.0.3) warnings.

{color:red}-1 release audit{color}.  The applied patch generated 3 
release audit warnings.

{color:red}-1 core tests{color}.  The test build failed in  

Test results: 
https://builds.apache.org/job/PreCommit-ATLAS-Build/290//testReport/
Release audit warnings: 
https://builds.apache.org/job/PreCommit-ATLAS-Build/290//artifact/patchprocess/patchReleaseAuditProblems.txt
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-ATLAS-Build/290//artifact/patchprocess/newPatchFindbugsWarningswebapp.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-ATLAS-Build/290//artifact/patchprocess/newPatchFindbugsWarningsauthorization.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-ATLAS-Build/290//artifact/patchprocess/newPatchFindbugsWarningscommon.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-ATLAS-Build/290//artifact/patchprocess/newPatchFindbugsWarningstitan0.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-ATLAS-Build/290//artifact/patchprocess/newPatchFindbugsWarningssqoop-bridge.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-ATLAS-Build/290//artifact/patchprocess/newPatchFindbugsWarningshdfs-model.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-ATLAS-Build/290//artifact/patchprocess/newPatchFindbugsWarningsstorm-bridge.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-ATLAS-Build/290//artifact/patchprocess/newPatchFindbugsWarningsfalcon-bridge.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-ATLAS-Build/290//artifact/patchprocess/newPatchFindbugsWarningshive-bridge.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-ATLAS-Build/290//artifact/patchprocess/newPatchFindbugsWarningsrepository.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-ATLAS-Build/290//artifact/patchprocess/newPatchFindbugsWarningstypesystem.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-ATLAS-Build/290//artifact/patchprocess/newPatchFindbugsWarningscatalog.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-ATLAS-Build/290//artifact/patchprocess/newPatchFindbugsWarningsclient.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-ATLAS-Build/290//artifact/patchprocess/newPatchFindbugsWarningsnotification.html
Console output: https://builds.apache.org/job/PreCommit-ATLAS-Build/290//console

This message is automatically generated.

> Update Atlas code to use abstraction layer
> --
>
> Key: ATLAS-694
> URL: https://issues.apache.org/jira/browse/ATLAS-694
> Project: Atlas
>  Issue Type: Sub-task
>Reporter: Jeffrey Hagelberg
>Assignee: Jeffrey Hagelberg
> Attachments: rb47810.patch
>
>
> In this task, we will update Atlas to replace the direct usage of titan 
> classes with usage of the equivalent interfaces in the abstraction layer.  At 
> this point, there will be no options to configure the abstraction layer 
> implementation that gets used.  Atlas will be hard-coded to use the titan 
> 0.5.4 implementation.  It will also continue to have a compile time 
> dependency on the titan 0.5.4 graph database implementation.  When this task 
> is complete, Atlas will only be accessing titan through the abstraction layer.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 47810: ATLAS-694: Update Atlas to use Graph DB abstraction layer

[Jeff Hagelberg]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47810/
---

(Updated June 6, 2016, 8:35 p.m.)


Review request for atlas, David Kantor and Neeru Gupta.


Changes
---

Fix issue in rat license check.


Bugs: ATLAS-694
https://issues.apache.org/jira/browse/ATLAS-694


Repository: atlas


Description
---

ATLAS-694: Update Atlas to use abstraction layer.  All of the Atlas code (with 
the exception of the catalog, which was only updated minimally) has been 
updated to use the graph database abstraction layer.  In addition, there is now 
an optional Atlas configuration property that specifies the class of the 
abstraction layer database to use.  I basically put all of the changes in here 
with the exception of the actual Titan 1 implementation of code.  This includes 
the changes to support Tinkerpop 3 syntax.  This is mostly to expedite getting 
the changes into Atlas.  Originally the TP3 changes were going to be brought in 
as part of the Titan 1 implementation task.

Change Summary:

   - change Atlas classes to use AtlasGraph,AtlasVertex,AtlasEdge, etc instead 
of TitanGraph/Vertex/Edge, etc
   - compile time dependency on titan 0.5.4/TP 2 removed (except in Catalog, 
which was only changed to use AtlasGraphProvider/AtlasGraph)
   - updated DSL translation to generate Gremlin that is compliant with TP3 
when TP3 is being used
   - TitanGraphProvider replaced by AtlasGraphProvider.  Graph database 
implementation is determined from a new optional configuration property
   - search indexer enhanced to wait for all indices to be enabled
   - HiveTitanSample is no longer used by tests.  It has been replaced by 
hive-instances.json (which uses normal Atlas json syntax).  The data is saved 
with a new JSONImporter class.  This was needed because the graphson syntax 
used by HiveTitanSample is not compatible with TP3.  
   - RepositoryMetaModule - GraphBackedSearchIndexer changed to an eager 
singleton to force index creation to happen before we execute any gremlin 
queries (which happens during the type system restoration triggered by the 
DefaultMetadataService constructor)
   - atlas_config.py - minor change to allow Atlas to use a non-default JVM.  
It now supports having the java home in ATLAS_JAVA_HOME.  If that is not set, 
we fall back to the previous behavior looking for JAVA_HOME and then checking 
the path.
   
   
Note - the diff here is cumulative and includes the changes from ATLAS-693.  I 
could not figure out how to remove those.  You can, for the most part, ignore 
the new files under graphdb\titan0.  There are a few minor changes that have 
been put in since ATLAS-693, but nothing very significant.  Just changes to 
implement a couple new methods in the abstraction layer.

Last rebase: 6/6/2016


Diffs (updated)
-

  addons/falcon-bridge/pom.xml 14c60901a6f194497ca0fb7929a032d2261f85fa 
  addons/hdfs-model/pom.xml f4033bce0fe3ea5a33cfd32ebccd47c01207dc35 
  addons/hdfs-model/src/test/java/org/apache/atlas/fs/model/HDFSModelTest.java 
ac60294e328835ba0340e150799ddfb348ccdb52 
  addons/hive-bridge/pom.xml 47e72e814a421f03b00a9b7c6a13163e1b67ca14 
  addons/sqoop-bridge/pom.xml 343bb4eeae93031ab24cd48ed7b535769150d6c7 
  addons/storm-bridge/pom.xml b1a7a9b1a88442d9e65db4a01b8388c8d3522bfe 
  authorization/pom.xml 74c433bf740a346d1d907a868605e9ab457151bb 
  catalog/pom.xml 8a49d3d673a5936613d45e53479aa99a5a14517d 
  catalog/src/main/java/org/apache/atlas/catalog/DefaultTypeSystem.java 
a28a32b692e61377479cdd432f39ba3c60bd5238 
  catalog/src/main/java/org/apache/atlas/catalog/query/BaseQuery.java 
ba8e0e712769afad72f7f8254ec626a1d17136e8 
  
catalog/src/test/java/org/apache/atlas/catalog/query/AtlasEntityQueryTest.java 
149134c8a79c802fe625d64c5e04af0d88174cfa 
  distro/src/bin/atlas_config.py fab4046345e05e0b7749ca75834c8bdcb0d01752 
  distro/src/conf/atlas-application.properties 
1cdd424d8e126a6bdbf99db8bf4e5d4329e806bb 
  graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasEdge.java 
71b577b5c2e8aee8fec816b9224672ebc2cad5b3 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasEdgeDirection.java
 e7da1c9a5075004973ba68b8a36a2c95756e0e62 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasElement.java 
3c41693e531eb523f04a5c5781f1f123ab38ef07 
  graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 
f312117af9cb6ae8de21d565bfcf3b10d9bdfcc6 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphIndex.java
 PRE-CREATION 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphManagement.java
 2776b2242c4469dcae344616ccfe100633a79432 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphQuery.java
 f0dca9a28eebd8996d55d71dcc3254cce9cd4098 
  

[jira] [Commented] (ATLAS-694) Update Atlas code to use abstraction layer

[ATLAS QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/ATLAS-694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15317230#comment-15317230
 ] 

ATLAS QA commented on ATLAS-694:


{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment
  http://issues.apache.org/jira/secure/attachment/12808454/rb47810.patch
  against master revision d661964.

{color:red}-1 @author{color}.  The patch appears to contain 1 @author tags 
which the Atlas community has agreed to not allow in code contributions.

{color:green}+1 tests included{color}.  The patch appears to include 35 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

+1 checkstyle.  The patch generated 0 code style errors.

{color:red}-1 findbugs{color}.  The patch appears to introduce 383 new 
Findbugs (version 2.0.3) warnings.

{color:red}-1 release audit{color}.  The applied patch generated 3 
release audit warnings.

{color:red}-1 core tests{color}.  The test build failed in  

Test results: 
https://builds.apache.org/job/PreCommit-ATLAS-Build/291//testReport/
Release audit warnings: 
https://builds.apache.org/job/PreCommit-ATLAS-Build/291//artifact/patchprocess/patchReleaseAuditProblems.txt
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-ATLAS-Build/291//artifact/patchprocess/newPatchFindbugsWarningswebapp.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-ATLAS-Build/291//artifact/patchprocess/newPatchFindbugsWarningsauthorization.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-ATLAS-Build/291//artifact/patchprocess/newPatchFindbugsWarningscommon.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-ATLAS-Build/291//artifact/patchprocess/newPatchFindbugsWarningstitan0.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-ATLAS-Build/291//artifact/patchprocess/newPatchFindbugsWarningssqoop-bridge.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-ATLAS-Build/291//artifact/patchprocess/newPatchFindbugsWarningshdfs-model.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-ATLAS-Build/291//artifact/patchprocess/newPatchFindbugsWarningsstorm-bridge.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-ATLAS-Build/291//artifact/patchprocess/newPatchFindbugsWarningsfalcon-bridge.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-ATLAS-Build/291//artifact/patchprocess/newPatchFindbugsWarningshive-bridge.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-ATLAS-Build/291//artifact/patchprocess/newPatchFindbugsWarningsrepository.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-ATLAS-Build/291//artifact/patchprocess/newPatchFindbugsWarningstypesystem.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-ATLAS-Build/291//artifact/patchprocess/newPatchFindbugsWarningscatalog.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-ATLAS-Build/291//artifact/patchprocess/newPatchFindbugsWarningsclient.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-ATLAS-Build/291//artifact/patchprocess/newPatchFindbugsWarningsnotification.html
Console output: https://builds.apache.org/job/PreCommit-ATLAS-Build/291//console

This message is automatically generated.

> Update Atlas code to use abstraction layer
> --
>
> Key: ATLAS-694
> URL: https://issues.apache.org/jira/browse/ATLAS-694
> Project: Atlas
>  Issue Type: Sub-task
>Reporter: Jeffrey Hagelberg
>Assignee: Jeffrey Hagelberg
> Attachments: rb47810.patch
>
>
> In this task, we will update Atlas to replace the direct usage of titan 
> classes with usage of the equivalent interfaces in the abstraction layer.  At 
> this point, there will be no options to configure the abstraction layer 
> implementation that gets used.  Atlas will be hard-coded to use the titan 
> 0.5.4 implementation.  It will also continue to have a compile time 
> dependency on the titan 0.5.4 graph database implementation.  When this task 
> is complete, Atlas will only be accessing titan through the abstraction layer.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 47658: ATLAS-499: Update reverse references

[Suma Shivaprasad]

> On June 3, 2016, 11:29 p.m., Suma Shivaprasad wrote:
> > repository/src/main/java/org/apache/atlas/repository/graph/TypedInstanceToGraphMapper.java,
> >  line 443
> > 
> >
> > for map types, can we check if the reverseAttributeName is set on the 
> > map type and remove it if set else ignore?
> 
> David Kantor wrote:
> I'm thinking it may be better to remove the reverse attribute name when 
> the type is registered and possibly issue a warning that bi-directional 
> references are not supported for map types, rather then doing it every time 
> there is an update operation involving that type.  Thoughts?

yeah that seems like a better option


- Suma


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47658/#review136133
---


On June 3, 2016, 1:55 p.m., David Kantor wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47658/
> ---
> 
> (Updated June 3, 2016, 1:55 p.m.)
> 
> 
> Review request for atlas and Shwetha GS.
> 
> 
> Bugs: ATLAS-499
> https://issues.apache.org/jira/browse/ATLAS-499
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> ATLAS-499: Automatically update inverse references to prevent repository 
> corruption from unbalanced references.
> 
> 
> Diffs
> -
> 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/DeleteHandler.java 
> 91f9bd008a6939dfe78656f5c1845637a30ee9eb 
>   repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java 
> 4f6d0118072380342f5ea302d0aaf7902784849a 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/TypedInstanceToGraphMapper.java
>  4c1f5591f4adead41c3336e64b0bc956836f7edb 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepositoryDeleteTestBase.java
>  449e066a167ba4546b118a77c8e3de5fd99f077b 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/ReverseReferenceUpdateHardDeleteTest.java
>  PRE-CREATION 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/ReverseReferenceUpdateSoftDeleteTest.java
>  PRE-CREATION 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/ReverseReferenceUpdateTestBase.java
>  PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/47658/diff/
> 
> 
> Testing
> ---
> 
> Ran all unit and integration tests with no regressions.
> 
> 
> Thanks,
> 
> David Kantor
> 
>

[jira] [Updated] (ATLAS-694) Update Atlas code to use abstraction layer

[Jeffrey Hagelberg (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeffrey Hagelberg updated ATLAS-694:

Attachment: (was: rb47810.patch)

> Update Atlas code to use abstraction layer
> --
>
> Key: ATLAS-694
> URL: https://issues.apache.org/jira/browse/ATLAS-694
> Project: Atlas
>  Issue Type: Sub-task
>Reporter: Jeffrey Hagelberg
>Assignee: Jeffrey Hagelberg
> Attachments: rb47810.patch
>
>
> In this task, we will update Atlas to replace the direct usage of titan 
> classes with usage of the equivalent interfaces in the abstraction layer.  At 
> this point, there will be no options to configure the abstraction layer 
> implementation that gets used.  Atlas will be hard-coded to use the titan 
> 0.5.4 implementation.  It will also continue to have a compile time 
> dependency on the titan 0.5.4 graph database implementation.  When this task 
> is complete, Atlas will only be accessing titan through the abstraction layer.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (ATLAS-694) Update Atlas code to use abstraction layer

[Jeffrey Hagelberg (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeffrey Hagelberg updated ATLAS-694:

Attachment: rb47810.patch

Fix rat license check issue that caused the build to fail.

> Update Atlas code to use abstraction layer
> --
>
> Key: ATLAS-694
> URL: https://issues.apache.org/jira/browse/ATLAS-694
> Project: Atlas
>  Issue Type: Sub-task
>Reporter: Jeffrey Hagelberg
>Assignee: Jeffrey Hagelberg
> Attachments: rb47810.patch
>
>
> In this task, we will update Atlas to replace the direct usage of titan 
> classes with usage of the equivalent interfaces in the abstraction layer.  At 
> this point, there will be no options to configure the abstraction layer 
> implementation that gets used.  Atlas will be hard-coded to use the titan 
> 0.5.4 implementation.  It will also continue to have a compile time 
> dependency on the titan 0.5.4 graph database implementation.  When this task 
> is complete, Atlas will only be accessing titan through the abstraction layer.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 47658: ATLAS-499: Update reverse references

[Suma Shivaprasad]

> On June 3, 2016, 11:25 p.m., Suma Shivaprasad wrote:
> > repository/src/main/java/org/apache/atlas/repository/graph/TypedInstanceToGraphMapper.java,
> >  line 785
> > 
> >
> > do we need the HashSet since contains and all will both O(n)
> 
> David Kantor wrote:
> HashSet.contains is O(1).  I am using the HashSet to avoid the O(n) 
> performance of ArrayList.contains.  For a very small data set, the difference 
> between HashSet.contains and ArrayList.contains is probably negligible.  But 
> in my experience doing Java performance analysis on our legacy metadata 
> repository, ArrayList.contains() often ended up being a major hot spot when 
> enforcing uniqueness for high cardinality references.

I was just saying that adding all the list elemnts to the Hashset is itself 
O(n) so it might be better to skip creating the hashset. Leave it up to you not 
a major issue to fix.


- Suma


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47658/#review136132
---


On June 3, 2016, 1:55 p.m., David Kantor wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47658/
> ---
> 
> (Updated June 3, 2016, 1:55 p.m.)
> 
> 
> Review request for atlas and Shwetha GS.
> 
> 
> Bugs: ATLAS-499
> https://issues.apache.org/jira/browse/ATLAS-499
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> ATLAS-499: Automatically update inverse references to prevent repository 
> corruption from unbalanced references.
> 
> 
> Diffs
> -
> 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/DeleteHandler.java 
> 91f9bd008a6939dfe78656f5c1845637a30ee9eb 
>   repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java 
> 4f6d0118072380342f5ea302d0aaf7902784849a 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/TypedInstanceToGraphMapper.java
>  4c1f5591f4adead41c3336e64b0bc956836f7edb 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepositoryDeleteTestBase.java
>  449e066a167ba4546b118a77c8e3de5fd99f077b 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/ReverseReferenceUpdateHardDeleteTest.java
>  PRE-CREATION 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/ReverseReferenceUpdateSoftDeleteTest.java
>  PRE-CREATION 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/ReverseReferenceUpdateTestBase.java
>  PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/47658/diff/
> 
> 
> Testing
> ---
> 
> Ran all unit and integration tests with no regressions.
> 
> 
> Thanks,
> 
> David Kantor
> 
>

Re: Review Request 47810: ATLAS-694: Update Atlas to use Graph DB abstraction layer

[Jeff Hagelberg]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47810/
---

(Updated June 6, 2016, 8:44 p.m.)


Review request for atlas, David Kantor and Neeru Gupta.


Changes
---

Removed @author tag


Bugs: ATLAS-694
https://issues.apache.org/jira/browse/ATLAS-694


Repository: atlas


Description
---

ATLAS-694: Update Atlas to use abstraction layer.  All of the Atlas code (with 
the exception of the catalog, which was only updated minimally) has been 
updated to use the graph database abstraction layer.  In addition, there is now 
an optional Atlas configuration property that specifies the class of the 
abstraction layer database to use.  I basically put all of the changes in here 
with the exception of the actual Titan 1 implementation of code.  This includes 
the changes to support Tinkerpop 3 syntax.  This is mostly to expedite getting 
the changes into Atlas.  Originally the TP3 changes were going to be brought in 
as part of the Titan 1 implementation task.

Change Summary:

   - change Atlas classes to use AtlasGraph,AtlasVertex,AtlasEdge, etc instead 
of TitanGraph/Vertex/Edge, etc
   - compile time dependency on titan 0.5.4/TP 2 removed (except in Catalog, 
which was only changed to use AtlasGraphProvider/AtlasGraph)
   - updated DSL translation to generate Gremlin that is compliant with TP3 
when TP3 is being used
   - TitanGraphProvider replaced by AtlasGraphProvider.  Graph database 
implementation is determined from a new optional configuration property
   - search indexer enhanced to wait for all indices to be enabled
   - HiveTitanSample is no longer used by tests.  It has been replaced by 
hive-instances.json (which uses normal Atlas json syntax).  The data is saved 
with a new JSONImporter class.  This was needed because the graphson syntax 
used by HiveTitanSample is not compatible with TP3.  
   - RepositoryMetaModule - GraphBackedSearchIndexer changed to an eager 
singleton to force index creation to happen before we execute any gremlin 
queries (which happens during the type system restoration triggered by the 
DefaultMetadataService constructor)
   - atlas_config.py - minor change to allow Atlas to use a non-default JVM.  
It now supports having the java home in ATLAS_JAVA_HOME.  If that is not set, 
we fall back to the previous behavior looking for JAVA_HOME and then checking 
the path.
   
   
Note - the diff here is cumulative and includes the changes from ATLAS-693.  I 
could not figure out how to remove those.  You can, for the most part, ignore 
the new files under graphdb\titan0.  There are a few minor changes that have 
been put in since ATLAS-693, but nothing very significant.  Just changes to 
implement a couple new methods in the abstraction layer.

Last rebase: 6/6/2016


Diffs (updated)
-

  addons/falcon-bridge/pom.xml 14c60901a6f194497ca0fb7929a032d2261f85fa 
  addons/hdfs-model/pom.xml f4033bce0fe3ea5a33cfd32ebccd47c01207dc35 
  addons/hdfs-model/src/test/java/org/apache/atlas/fs/model/HDFSModelTest.java 
ac60294e328835ba0340e150799ddfb348ccdb52 
  addons/hive-bridge/pom.xml 47e72e814a421f03b00a9b7c6a13163e1b67ca14 
  addons/sqoop-bridge/pom.xml 343bb4eeae93031ab24cd48ed7b535769150d6c7 
  addons/storm-bridge/pom.xml b1a7a9b1a88442d9e65db4a01b8388c8d3522bfe 
  authorization/pom.xml 74c433bf740a346d1d907a868605e9ab457151bb 
  catalog/pom.xml 8a49d3d673a5936613d45e53479aa99a5a14517d 
  catalog/src/main/java/org/apache/atlas/catalog/DefaultTypeSystem.java 
a28a32b692e61377479cdd432f39ba3c60bd5238 
  catalog/src/main/java/org/apache/atlas/catalog/query/BaseQuery.java 
ba8e0e712769afad72f7f8254ec626a1d17136e8 
  
catalog/src/test/java/org/apache/atlas/catalog/query/AtlasEntityQueryTest.java 
149134c8a79c802fe625d64c5e04af0d88174cfa 
  distro/src/bin/atlas_config.py fab4046345e05e0b7749ca75834c8bdcb0d01752 
  distro/src/conf/atlas-application.properties 
1cdd424d8e126a6bdbf99db8bf4e5d4329e806bb 
  graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasEdge.java 
71b577b5c2e8aee8fec816b9224672ebc2cad5b3 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasEdgeDirection.java
 e7da1c9a5075004973ba68b8a36a2c95756e0e62 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasElement.java 
3c41693e531eb523f04a5c5781f1f123ab38ef07 
  graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 
f312117af9cb6ae8de21d565bfcf3b10d9bdfcc6 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphIndex.java
 PRE-CREATION 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphManagement.java
 2776b2242c4469dcae344616ccfe100633a79432 
  
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphQuery.java
 f0dca9a28eebd8996d55d71dcc3254cce9cd4098 
  

[jira] [Updated] (ATLAS-694) Update Atlas code to use abstraction layer

[Jeffrey Hagelberg (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeffrey Hagelberg updated ATLAS-694:

Attachment: (was: rb47810.patch)

> Update Atlas code to use abstraction layer
> --
>
> Key: ATLAS-694
> URL: https://issues.apache.org/jira/browse/ATLAS-694
> Project: Atlas
>  Issue Type: Sub-task
>Reporter: Jeffrey Hagelberg
>Assignee: Jeffrey Hagelberg
>
> In this task, we will update Atlas to replace the direct usage of titan 
> classes with usage of the equivalent interfaces in the abstraction layer.  At 
> this point, there will be no options to configure the abstraction layer 
> implementation that gets used.  Atlas will be hard-coded to use the titan 
> 0.5.4 implementation.  It will also continue to have a compile time 
> dependency on the titan 0.5.4 graph database implementation.  When this task 
> is complete, Atlas will only be accessing titan through the abstraction layer.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 47658: ATLAS-499: Update reverse references

[David Kantor]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47658/
---

(Updated June 6, 2016, 10:53 p.m.)


Review request for atlas and Shwetha GS.


Changes
---

Addressed Suma's comment about HashSet usage.


Bugs: ATLAS-499
https://issues.apache.org/jira/browse/ATLAS-499


Repository: atlas


Description
---

ATLAS-499: Automatically update inverse references to prevent repository 
corruption from unbalanced references.


Diffs (updated)
-

  repository/src/main/java/org/apache/atlas/repository/graph/DeleteHandler.java 
91f9bd008a6939dfe78656f5c1845637a30ee9eb 
  repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java 
4f6d0118072380342f5ea302d0aaf7902784849a 
  
repository/src/main/java/org/apache/atlas/repository/graph/TypedInstanceToGraphMapper.java
 4c1f5591f4adead41c3336e64b0bc956836f7edb 
  
repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepositoryDeleteTestBase.java
 449e066a167ba4546b118a77c8e3de5fd99f077b 
  
repository/src/test/java/org/apache/atlas/repository/graph/ReverseReferenceUpdateHardDeleteTest.java
 PRE-CREATION 
  
repository/src/test/java/org/apache/atlas/repository/graph/ReverseReferenceUpdateSoftDeleteTest.java
 PRE-CREATION 
  
repository/src/test/java/org/apache/atlas/repository/graph/ReverseReferenceUpdateTestBase.java
 PRE-CREATION 

Diff: https://reviews.apache.org/r/47658/diff/


Testing
---

Ran all unit and integration tests with no regressions.


Thanks,

David Kantor

Re: Review Request 47658: ATLAS-499: Update reverse references

[David Kantor]

> On June 3, 2016, 11:25 p.m., Suma Shivaprasad wrote:
> > repository/src/main/java/org/apache/atlas/repository/graph/TypedInstanceToGraphMapper.java,
> >  line 785
> > 
> >
> > do we need the HashSet since contains and all will both O(n)
> 
> David Kantor wrote:
> HashSet.contains is O(1).  I am using the HashSet to avoid the O(n) 
> performance of ArrayList.contains.  For a very small data set, the difference 
> between HashSet.contains and ArrayList.contains is probably negligible.  But 
> in my experience doing Java performance analysis on our legacy metadata 
> repository, ArrayList.contains() often ended up being a major hot spot when 
> enforcing uniqueness for high cardinality references.
> 
> Suma Shivaprasad wrote:
> I was just saying that adding all the list elemnts to the Hashset is 
> itself O(n) so it might be better to skip creating the hashset. Leave it up 
> to you not a major issue to fix.

Thanks very much for clarifying that, I agree and have addressed the comment.


- David


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47658/#review136132
---


On June 6, 2016, 10:53 p.m., David Kantor wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47658/
> ---
> 
> (Updated June 6, 2016, 10:53 p.m.)
> 
> 
> Review request for atlas and Shwetha GS.
> 
> 
> Bugs: ATLAS-499
> https://issues.apache.org/jira/browse/ATLAS-499
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> ATLAS-499: Automatically update inverse references to prevent repository 
> corruption from unbalanced references.
> 
> 
> Diffs
> -
> 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/DeleteHandler.java 
> 91f9bd008a6939dfe78656f5c1845637a30ee9eb 
>   repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java 
> 4f6d0118072380342f5ea302d0aaf7902784849a 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/TypedInstanceToGraphMapper.java
>  4c1f5591f4adead41c3336e64b0bc956836f7edb 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepositoryDeleteTestBase.java
>  449e066a167ba4546b118a77c8e3de5fd99f077b 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/ReverseReferenceUpdateHardDeleteTest.java
>  PRE-CREATION 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/ReverseReferenceUpdateSoftDeleteTest.java
>  PRE-CREATION 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/ReverseReferenceUpdateTestBase.java
>  PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/47658/diff/
> 
> 
> Testing
> ---
> 
> Ran all unit and integration tests with no regressions.
> 
> 
> Thanks,
> 
> David Kantor
> 
>

Re: Review Request 47658: ATLAS-499: Update reverse references

[David Kantor]

> On June 3, 2016, 11:29 p.m., Suma Shivaprasad wrote:
> > repository/src/main/java/org/apache/atlas/repository/graph/TypedInstanceToGraphMapper.java,
> >  line 443
> > 
> >
> > for map types, can we check if the reverseAttributeName is set on the 
> > map type and remove it if set else ignore?
> 
> David Kantor wrote:
> I'm thinking it may be better to remove the reverse attribute name when 
> the type is registered and possibly issue a warning that bi-directional 
> references are not supported for map types, rather then doing it every time 
> there is an update operation involving that type.  Thoughts?
> 
> Suma Shivaprasad wrote:
> yeah that seems like a better option

In thinking more about this, perhaps it is beyond the scope of this JIRA to 
make such a significant change to the type system functionality.  You could 
argue that having a bi-directional reference where one of the ends is a map 
reference is OK, it's just that in the specific case where the user just 
updates the non-map end of the reference, we don't know what to use for the map 
key when updating the reverse map reference so we can't support auto-updating 
in that use case.  It doesn't necessarily mean that the type system should 
ignore/remove the reverseAttributeName on a map reference at registration time. 
 Modelers may have a valid use case for bi-directional map references and if we 
suddenly change the type system to ignore or remove the reverseAttributeName, 
that could be a breaking change to existing applications.  Perhaps your point 
in the original comment is that if we don't support auto-updating when just the 
non-map reference is updated, we should be consistent and not suppor
 t automatic reverse reference updating if *either* end of the reference is a 
map.


- David


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47658/#review136133
---


On June 3, 2016, 1:55 p.m., David Kantor wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47658/
> ---
> 
> (Updated June 3, 2016, 1:55 p.m.)
> 
> 
> Review request for atlas and Shwetha GS.
> 
> 
> Bugs: ATLAS-499
> https://issues.apache.org/jira/browse/ATLAS-499
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> ATLAS-499: Automatically update inverse references to prevent repository 
> corruption from unbalanced references.
> 
> 
> Diffs
> -
> 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/DeleteHandler.java 
> 91f9bd008a6939dfe78656f5c1845637a30ee9eb 
>   repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java 
> 4f6d0118072380342f5ea302d0aaf7902784849a 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/TypedInstanceToGraphMapper.java
>  4c1f5591f4adead41c3336e64b0bc956836f7edb 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepositoryDeleteTestBase.java
>  449e066a167ba4546b118a77c8e3de5fd99f077b 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/ReverseReferenceUpdateHardDeleteTest.java
>  PRE-CREATION 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/ReverseReferenceUpdateSoftDeleteTest.java
>  PRE-CREATION 
>   
> repository/src/test/java/org/apache/atlas/repository/graph/ReverseReferenceUpdateTestBase.java
>  PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/47658/diff/
> 
> 
> Testing
> ---
> 
> Ran all unit and integration tests with no regressions.
> 
> 
> Thanks,
> 
> David Kantor
> 
>

[jira] [Commented] (ATLAS-801) Atlas hooks would lose messages if Kafka is down for extended period of time

[Shwetha G S (JIRA)]

[ 
https://issues.apache.org/jira/browse/ATLAS-801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15316226#comment-15316226
 ] 

Shwetha G S commented on ATLAS-801:
---

Agree with the short term solution for now.
1. Ensure multiple replicas for Kafka  - the number of consumer threads will 
still be 1, right? Else, we might end up with some message handling failures
2. Add retries when sending messages to Kafka - Doesn't all hooks have 
configurable retries already, with default 3
3. Build some way of alerting admins of errors in communicating to Kafka - we 
probably need to hook into alerting mechanism of each component?

Additionally, we can also write failed messages to separate file using log4j 
and use log4j to roll the file based on size

> Atlas hooks would lose messages if Kafka is down for extended period of time
> 
>
> Key: ATLAS-801
> URL: https://issues.apache.org/jira/browse/ATLAS-801
> Project: Atlas
>  Issue Type: Improvement
>Reporter: Hemanth Yamijala
>Assignee: Hemanth Yamijala
>
> All integration hooks in Atlas write messages to Kafka which are picked up by 
> the Atlas server. If communication to Kafka breaks, then this results in loss 
> of metadata messages. This can be mitigated to some extent using multiple 
> replicas for Kafka topics (see ATLAS-515). This JIRA is to see if we can make 
> this even more robust and have some form of store and forward mechanism for 
> increased fault tolerance.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (ATLAS-836) Add timeout for kafka notifications

[Shwetha G S (JIRA)]

[ 
https://issues.apache.org/jira/browse/ATLAS-836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15316237#comment-15316237
 ] 

Shwetha G S commented on ATLAS-836:
---

30 secs is too much, we should set a lower value?

> Add timeout for kafka notifications
> ---
>
> Key: ATLAS-836
> URL: https://issues.apache.org/jira/browse/ATLAS-836
> Project: Atlas
>  Issue Type: Bug
>Reporter: Shwetha G S
> Fix For: 0.7-incubating
>
>
> In atlas conf, we don't have timeout for calls to kafka and have seen cases 
> where the kafka calls get stuck for a long time. We should have reasonable 
> timeouts



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (ATLAS-864) 1 table out of 50,000 tables is left unimported throwing exception during deserialization

[Sharmadha Sainath (JIRA)]

Sharmadha Sainath created ATLAS-864:
---

 Summary: 1 table out of 50,000 tables is left unimported throwing 
exception during deserialization
 Key: ATLAS-864
 URL: https://issues.apache.org/jira/browse/ATLAS-864
 Project: Atlas
  Issue Type: Bug
 Environment: Atlas commit id : 59268875d256c92f748198cea999ed5057c6bf18
Cluster setup :
Machine 1 : Atlas, Solr
Machine 2 : HBase, Kafka
Machine 3 : Client
atlas.notification.hook.numthreads=5
ATLAS_HOOK topic created with 5 partitions
Reporter: Sharmadha Sainath


On importing 50,000 tables using hiveserver2, 1 table is not imported 
(table_7770). Below is the exception stack trace :
java.lang.NumberFormatException: For input string: ""
at 
java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
at java.lang.Long.parseLong(Long.java:453)
at java.lang.Long.parseLong(Long.java:483)
at java.text.DigitList.getLong(DigitList.java:194)
at java.text.DecimalFormat.parse(DecimalFormat.java:1316)
at java.text.SimpleDateFormat.subParse(SimpleDateFormat.java:1793)
at java.text.SimpleDateFormat.parse(SimpleDateFormat.java:1455)
at java.text.DateFormat.parse(DateFormat.java:355)
at 
org.apache.atlas.typesystem.types.DataTypes$DateType.convert(DataTypes.java:415)
at 
org.apache.atlas.typesystem.types.DataTypes$DateType.convert(DataTypes.java:400)
at 
org.apache.atlas.typesystem.persistence.StructInstance.set(StructInstance.java:121)
at 
org.apache.atlas.typesystem.types.ClassType.convert(ClassType.java:143)
at 
org.apache.atlas.services.DefaultMetadataService.getTypedReferenceableInstance(DefaultMetadataService.java:356)
at 
org.apache.atlas.services.DefaultMetadataService.deserializeClassInstances(DefaultMetadataService.java:332)
at 
org.apache.atlas.services.DefaultMetadataService.updateEntities(DefaultMetadataService.java:426)
at 
org.apache.atlas.web.resources.EntityResource.updateEntities(EntityResource.java:167)
at org.apache.atlas.LocalAtlasClient$2.invoke(LocalAtlasClient.java:97)
at 
org.apache.atlas.LocalAtlasClient$EntityOperation.run(LocalAtlasClient.java:118)
at 
org.apache.atlas.LocalAtlasClient.updateEntities(LocalAtlasClient.java:100)
at org.apache.atlas.AtlasClient.updateEntities(AtlasClient.java:684)
at 
org.apache.atlas.notification.NotificationHookConsumer$HookConsumer.handleMessage(NotificationHookConsumer.java:225)
at 
org.apache.atlas.notification.NotificationHookConsumer$HookConsumer.run(NotificationHookConsumer.java:187)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2016-06-03 05:59:28,049 ERROR - [NotificationHookConsumer thread-1:] ~ Unable 
to persist entity instance (EntityResource:177)
java.lang.IllegalArgumentException: Unable to deserialize json
at 
org.apache.atlas.services.DefaultMetadataService.deserializeClassInstances(DefaultMetadataService.java:340)
at 
org.apache.atlas.services.DefaultMetadataService.updateEntities(DefaultMetadataService.java:426)
at 
org.apache.atlas.web.resources.EntityResource.updateEntities(EntityResource.java:167)
at org.apache.atlas.LocalAtlasClient$2.invoke(LocalAtlasClient.java:97)
at 
org.apache.atlas.LocalAtlasClient$EntityOperation.run(LocalAtlasClient.java:118)
at 
org.apache.atlas.LocalAtlasClient.updateEntities(LocalAtlasClient.java:100)
at org.apache.atlas.AtlasClient.updateEntities(AtlasClient.java:684)
at 
org.apache.atlas.notification.NotificationHookConsumer$HookConsumer.handleMessage(NotificationHookConsumer.java:225)
at 
org.apache.atlas.notification.NotificationHookConsumer$HookConsumer.run(NotificationHookConsumer.java:187)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NumberFormatException: For input string: ""




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (ATLAS-861) 1 table out of 50,000 tables is left unimported throwing exception during deserialization

[Sharmadha Sainath (JIRA)]

Sharmadha Sainath created ATLAS-861:
---

 Summary: 1 table out of 50,000 tables is left unimported throwing 
exception during deserialization
 Key: ATLAS-861
 URL: https://issues.apache.org/jira/browse/ATLAS-861
 Project: Atlas
  Issue Type: Bug
 Environment: Atlas commit id : 59268875d256c92f748198cea999ed5057c6bf18
Cluster setup :
Machine 1 : Atlas, Solr
Machine 2 : HBase, Kafka
Machine 3 : Client
atlas.notification.hook.numthreads=5
ATLAS_HOOK topic created with 5 partitions
Reporter: Sharmadha Sainath


On importing 50,000 tables using hiveserver2, 1 table is not imported 
(table_7770). Below is the exception stack trace :
java.lang.NumberFormatException: For input string: ""
at 
java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
at java.lang.Long.parseLong(Long.java:453)
at java.lang.Long.parseLong(Long.java:483)
at java.text.DigitList.getLong(DigitList.java:194)
at java.text.DecimalFormat.parse(DecimalFormat.java:1316)
at java.text.SimpleDateFormat.subParse(SimpleDateFormat.java:1793)
at java.text.SimpleDateFormat.parse(SimpleDateFormat.java:1455)
at java.text.DateFormat.parse(DateFormat.java:355)
at 
org.apache.atlas.typesystem.types.DataTypes$DateType.convert(DataTypes.java:415)
at 
org.apache.atlas.typesystem.types.DataTypes$DateType.convert(DataTypes.java:400)
at 
org.apache.atlas.typesystem.persistence.StructInstance.set(StructInstance.java:121)
at 
org.apache.atlas.typesystem.types.ClassType.convert(ClassType.java:143)
at 
org.apache.atlas.services.DefaultMetadataService.getTypedReferenceableInstance(DefaultMetadataService.java:356)
at 
org.apache.atlas.services.DefaultMetadataService.deserializeClassInstances(DefaultMetadataService.java:332)
at 
org.apache.atlas.services.DefaultMetadataService.updateEntities(DefaultMetadataService.java:426)
at 
org.apache.atlas.web.resources.EntityResource.updateEntities(EntityResource.java:167)
at org.apache.atlas.LocalAtlasClient$2.invoke(LocalAtlasClient.java:97)
at 
org.apache.atlas.LocalAtlasClient$EntityOperation.run(LocalAtlasClient.java:118)
at 
org.apache.atlas.LocalAtlasClient.updateEntities(LocalAtlasClient.java:100)
at org.apache.atlas.AtlasClient.updateEntities(AtlasClient.java:684)
at 
org.apache.atlas.notification.NotificationHookConsumer$HookConsumer.handleMessage(NotificationHookConsumer.java:225)
at 
org.apache.atlas.notification.NotificationHookConsumer$HookConsumer.run(NotificationHookConsumer.java:187)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2016-06-03 05:59:28,049 ERROR - [NotificationHookConsumer thread-1:] ~ Unable 
to persist entity instance (EntityResource:177)
java.lang.IllegalArgumentException: Unable to deserialize json
at 
org.apache.atlas.services.DefaultMetadataService.deserializeClassInstances(DefaultMetadataService.java:340)
at 
org.apache.atlas.services.DefaultMetadataService.updateEntities(DefaultMetadataService.java:426)
at 
org.apache.atlas.web.resources.EntityResource.updateEntities(EntityResource.java:167)
at org.apache.atlas.LocalAtlasClient$2.invoke(LocalAtlasClient.java:97)
at 
org.apache.atlas.LocalAtlasClient$EntityOperation.run(LocalAtlasClient.java:118)
at 
org.apache.atlas.LocalAtlasClient.updateEntities(LocalAtlasClient.java:100)
at org.apache.atlas.AtlasClient.updateEntities(AtlasClient.java:684)
at 
org.apache.atlas.notification.NotificationHookConsumer$HookConsumer.handleMessage(NotificationHookConsumer.java:225)
at 
org.apache.atlas.notification.NotificationHookConsumer$HookConsumer.run(NotificationHookConsumer.java:187)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NumberFormatException: For input string: ""




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (ATLAS-862) 1 table out of 50,000 tables is left unimported throwing exception during deserialization

[Sharmadha Sainath (JIRA)]

Sharmadha Sainath created ATLAS-862:
---

 Summary: 1 table out of 50,000 tables is left unimported throwing 
exception during deserialization
 Key: ATLAS-862
 URL: https://issues.apache.org/jira/browse/ATLAS-862
 Project: Atlas
  Issue Type: Bug
 Environment: Atlas commit id : 59268875d256c92f748198cea999ed5057c6bf18
Cluster setup :
Machine 1 : Atlas, Solr
Machine 2 : HBase, Kafka
Machine 3 : Client
atlas.notification.hook.numthreads=5
ATLAS_HOOK topic created with 5 partitions
Reporter: Sharmadha Sainath


On importing 50,000 tables using hiveserver2, 1 table is not imported 
(table_7770). Below is the exception stack trace :
java.lang.NumberFormatException: For input string: ""
at 
java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
at java.lang.Long.parseLong(Long.java:453)
at java.lang.Long.parseLong(Long.java:483)
at java.text.DigitList.getLong(DigitList.java:194)
at java.text.DecimalFormat.parse(DecimalFormat.java:1316)
at java.text.SimpleDateFormat.subParse(SimpleDateFormat.java:1793)
at java.text.SimpleDateFormat.parse(SimpleDateFormat.java:1455)
at java.text.DateFormat.parse(DateFormat.java:355)
at 
org.apache.atlas.typesystem.types.DataTypes$DateType.convert(DataTypes.java:415)
at 
org.apache.atlas.typesystem.types.DataTypes$DateType.convert(DataTypes.java:400)
at 
org.apache.atlas.typesystem.persistence.StructInstance.set(StructInstance.java:121)
at 
org.apache.atlas.typesystem.types.ClassType.convert(ClassType.java:143)
at 
org.apache.atlas.services.DefaultMetadataService.getTypedReferenceableInstance(DefaultMetadataService.java:356)
at 
org.apache.atlas.services.DefaultMetadataService.deserializeClassInstances(DefaultMetadataService.java:332)
at 
org.apache.atlas.services.DefaultMetadataService.updateEntities(DefaultMetadataService.java:426)
at 
org.apache.atlas.web.resources.EntityResource.updateEntities(EntityResource.java:167)
at org.apache.atlas.LocalAtlasClient$2.invoke(LocalAtlasClient.java:97)
at 
org.apache.atlas.LocalAtlasClient$EntityOperation.run(LocalAtlasClient.java:118)
at 
org.apache.atlas.LocalAtlasClient.updateEntities(LocalAtlasClient.java:100)
at org.apache.atlas.AtlasClient.updateEntities(AtlasClient.java:684)
at 
org.apache.atlas.notification.NotificationHookConsumer$HookConsumer.handleMessage(NotificationHookConsumer.java:225)
at 
org.apache.atlas.notification.NotificationHookConsumer$HookConsumer.run(NotificationHookConsumer.java:187)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2016-06-03 05:59:28,049 ERROR - [NotificationHookConsumer thread-1:] ~ Unable 
to persist entity instance (EntityResource:177)
java.lang.IllegalArgumentException: Unable to deserialize json
at 
org.apache.atlas.services.DefaultMetadataService.deserializeClassInstances(DefaultMetadataService.java:340)
at 
org.apache.atlas.services.DefaultMetadataService.updateEntities(DefaultMetadataService.java:426)
at 
org.apache.atlas.web.resources.EntityResource.updateEntities(EntityResource.java:167)
at org.apache.atlas.LocalAtlasClient$2.invoke(LocalAtlasClient.java:97)
at 
org.apache.atlas.LocalAtlasClient$EntityOperation.run(LocalAtlasClient.java:118)
at 
org.apache.atlas.LocalAtlasClient.updateEntities(LocalAtlasClient.java:100)
at org.apache.atlas.AtlasClient.updateEntities(AtlasClient.java:684)
at 
org.apache.atlas.notification.NotificationHookConsumer$HookConsumer.handleMessage(NotificationHookConsumer.java:225)
at 
org.apache.atlas.notification.NotificationHookConsumer$HookConsumer.run(NotificationHookConsumer.java:187)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NumberFormatException: For input string: ""




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)