Review Request 72257: ATLAS-3583 Use Audit framework to generate audit entries for Classification CREATE, UPDATE and DELETE
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/72257/ --- Review request for atlas, Ashutosh Mestry, Jayendra Parab, Madhan Neethiraj, Nixon Rodrigues, and Sarath Subramanian. Bugs: ATLAS-3583 https://issues.apache.org/jira/browse/ATLAS-3583 Repository: atlas Description --- Added listener to listen create/update/delete operations of all type defs. Diffs - addons/models/-Area0/0010-base_model.json 351e07b intg/src/main/java/org/apache/atlas/model/audit/AtlasAuditEntry.java f2a8b3e repository/src/main/java/org/apache/atlas/repository/store/graph/AtlasTypeDefGraphStore.java 79f5270 repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AuditChangeListener.java PRE-CREATION Diff: https://reviews.apache.org/r/72257/diff/1/ Testing --- Pre-commit build: https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/1747/console Manual testing is done. Thanks, Mandar Ambawane
Review Request 72259: ATLAS-3675 Enable quick search and suggestions for business metadata attributes
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/72259/ --- Review request for atlas, Ashutosh Mestry, Jayendra Parab, Madhan Neethiraj, Nixon Rodrigues, and Sarath Subramanian. Bugs: ATLAS-3675 https://issues.apache.org/jira/browse/ATLAS-3675 Repository: atlas Description --- Added solr index for business metadata attributes. >From UI, need to add searchWeight with metadata attributes to support >suggestions. Diffs - intg/src/main/java/org/apache/atlas/listener/ChangedTypeDefs.java af00f6c repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java 7c55130 repository/src/main/java/org/apache/atlas/repository/graph/SolrIndexHelper.java 4ec0086 Diff: https://reviews.apache.org/r/72259/diff/1/ Testing --- Pre-commit build: https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/1746/console Basic testing is done. Thanks, Mandar Ambawane
[jira] [Updated] (ATLAS-3423) Import Glossary Terms CSV into a Glossary
[ https://issues.apache.org/jira/browse/ATLAS-3423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian updated ATLAS-3423: -- Component/s: atlas-core > Import Glossary Terms CSV into a Glossary > - > > Key: ATLAS-3423 > URL: https://issues.apache.org/jira/browse/ATLAS-3423 > Project: Atlas > Issue Type: New Feature > Components: atlas-core >Affects Versions: 3.0.0 >Reporter: Nixon Rodrigues >Assignee: Mayank Jain >Priority: Major > Fix For: 2.1.0, 3.0.0 > > Attachments: ATLAS-3423-19.patch > > > As a data steward, I would like to be able to glossary terms in bulk from an > appropriately formatted CSV/XLS template (which includes all the relevant > properties pertaining to a glossary term) and import it into Atlas directly > from Atlas UI. > > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (ATLAS-3675) Enable quick search and suggestions for business metadata attributes
Mandar Ambawane created ATLAS-3675: -- Summary: Enable quick search and suggestions for business metadata attributes Key: ATLAS-3675 URL: https://issues.apache.org/jira/browse/ATLAS-3675 Project: Atlas Issue Type: Bug Reporter: Mandar Ambawane Assignee: Mandar Ambawane -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (ATLAS-342) Atlas is sending an ENTITY_CREATE event to the ATLAS_ENTITIES topic even if the entity exists already.
[
https://issues.apache.org/jira/browse/ATLAS-342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17063796#comment-17063796
]
ASF subversion and git services commented on ATLAS-342:
---
Commit 0eb4562dca581e77508fb04dfb5a13b331939800 in atlas's branch
refs/heads/branch-2.0 from mayanknj
[ https://gitbox.apache.org/repos/asf?p=atlas.git;h=0eb4562 ]
ATLAS-342: Import Glossary Terms from csv/excel file into Glossary
Signed-off-by: Sarath Subramanian
(cherry picked from commit 784b606ddfa2149535b437483051a68859689a9d)
> Atlas is sending an ENTITY_CREATE event to the ATLAS_ENTITIES topic even if
> the entity exists already.
> --
>
> Key: ATLAS-342
> URL: https://issues.apache.org/jira/browse/ATLAS-342
> Project: Atlas
> Issue Type: Bug
>Reporter: Hemanth Yamijala
>Assignee: Shwetha GS
>Priority: Major
> Fix For: 0.6-incubating
>
> Attachments: ATLAS-342.patch
>
>
> * Configure Hive hook in Hive.
> * Configure Atlas to talk to Kafka topics on a separately set Kafka instance.
> * Run a script to create tables in Hive multiple times, for e.g.
> {code}
> for i in `seq 1 10`; do ./bin/hive -e "create table tbl$i (column${i}1
> string, column${i}2 int)"; done
> {code}
> * After the script completes, check the number of entities & list the actual
> entities in the ATLAS_ENTITIES topic.
> We can see one ENTITY_CREATE event for the Hive database for every table
> created in Hive. For e.g.:
> {code}
> {"entity":{"jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Reference","id":{"jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Id","id":"e7ed4ad9-1fba-47ad-a089-1ff7e715c1ad","version":0,"typeName":"hive_db"},"typeName":"hive_db","values":{"name":"default","description":"Default
> Hive
> database","ownerType":{"value":"ROLE","ordinal":2},"qualifiedName":"primary.default","locationUri":"hdfs://localhost:9000/user/hive/warehouse","ownerName":"public","clusterName":"primary"},"traitNames":[],"traits":{}},"operationType":"ENTITY_CREATE","traits":[]}
> {"entity":{"jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Reference","id":{"jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Id","id":"e7ed4ad9-1fba-47ad-a089-1ff7e715c1ad","version":0,"typeName":"hive_db"},"typeName":"hive_db","values":{"name":"default","description":"Default
> Hive
> database","ownerType":{"value":"ROLE","ordinal":2},"qualifiedName":"primary.default","locationUri":"hdfs://localhost:9000/user/hive/warehouse","ownerName":"public","clusterName":"primary"},"traitNames":[],"traits":{}},"operationType":"ENTITY_CREATE","traits":[]}
> {code}
> The expectation is that we don't have these extraneous events as there is no
> change to the entity.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (ATLAS-342) Atlas is sending an ENTITY_CREATE event to the ATLAS_ENTITIES topic even if the entity exists already.
[
https://issues.apache.org/jira/browse/ATLAS-342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17063795#comment-17063795
]
ASF subversion and git services commented on ATLAS-342:
---
Commit 784b606ddfa2149535b437483051a68859689a9d in atlas's branch
refs/heads/master from mayanknj
[ https://gitbox.apache.org/repos/asf?p=atlas.git;h=784b606 ]
ATLAS-342: Import Glossary Terms from csv/excel file into Glossary
Signed-off-by: Sarath Subramanian
> Atlas is sending an ENTITY_CREATE event to the ATLAS_ENTITIES topic even if
> the entity exists already.
> --
>
> Key: ATLAS-342
> URL: https://issues.apache.org/jira/browse/ATLAS-342
> Project: Atlas
> Issue Type: Bug
>Reporter: Hemanth Yamijala
>Assignee: Shwetha GS
>Priority: Major
> Fix For: 0.6-incubating
>
> Attachments: ATLAS-342.patch
>
>
> * Configure Hive hook in Hive.
> * Configure Atlas to talk to Kafka topics on a separately set Kafka instance.
> * Run a script to create tables in Hive multiple times, for e.g.
> {code}
> for i in `seq 1 10`; do ./bin/hive -e "create table tbl$i (column${i}1
> string, column${i}2 int)"; done
> {code}
> * After the script completes, check the number of entities & list the actual
> entities in the ATLAS_ENTITIES topic.
> We can see one ENTITY_CREATE event for the Hive database for every table
> created in Hive. For e.g.:
> {code}
> {"entity":{"jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Reference","id":{"jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Id","id":"e7ed4ad9-1fba-47ad-a089-1ff7e715c1ad","version":0,"typeName":"hive_db"},"typeName":"hive_db","values":{"name":"default","description":"Default
> Hive
> database","ownerType":{"value":"ROLE","ordinal":2},"qualifiedName":"primary.default","locationUri":"hdfs://localhost:9000/user/hive/warehouse","ownerName":"public","clusterName":"primary"},"traitNames":[],"traits":{}},"operationType":"ENTITY_CREATE","traits":[]}
> {"entity":{"jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Reference","id":{"jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Id","id":"e7ed4ad9-1fba-47ad-a089-1ff7e715c1ad","version":0,"typeName":"hive_db"},"typeName":"hive_db","values":{"name":"default","description":"Default
> Hive
> database","ownerType":{"value":"ROLE","ordinal":2},"qualifiedName":"primary.default","locationUri":"hdfs://localhost:9000/user/hive/warehouse","ownerName":"public","clusterName":"primary"},"traitNames":[],"traits":{}},"operationType":"ENTITY_CREATE","traits":[]}
> {code}
> The expectation is that we don't have these extraneous events as there is no
> change to the entity.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Re: Review Request 71482: ATLAS-3423:-Import Glossary Terms CSV into a Glossary
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71482/#review220053
---
Ship it!
Ship It!
- Sarath Subramanian
On March 20, 2020, 7:06 a.m., mayank jain wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71482/
> ---
>
> (Updated March 20, 2020, 7:06 a.m.)
>
>
> Review request for atlas, Ashutosh Mestry, keval bhatt, Madhan Neethiraj,
> Nixon Rodrigues, and Sarath Subramanian.
>
>
> Bugs: ATLAS-3423
> https://issues.apache.org/jira/browse/ATLAS-3423
>
>
> Repository: atlas
>
>
> Description
> ---
>
> This patch consists implementation for 2 end points first for template
> download and other for csv file upload with term details also the Unit Test
> cases for both the end points.
>
> * The 1st endpoint {glossary/template} return template file this would be
> type of format of data that shows how the data needs to be populated by user
> in the file.
>
> http://localhost:21000/api/atlas/v2/glossary/import/template
>
> Template structure:-
>
> GlossaryName, TermName, ShortDescription, LongDescription, Examples,
> Abbreviation, Usage, AdditionalAttributes, TranslationTerms, ValidValuesFor,
> Synonyms, ReplacedBy, ValidValues, ReplacementTerms, SeeAlso,
> TranslatedTerms, IsA, Antonyms, Classifies, PreferredToTerms, PreferredTerms
> Fruits,Apple5,SD4,LD4,"EXAMPLE","ABBREVIATION","USAGE",,"Footwear:B4","Footwear:B4","Footwear:B4","Footwear:B4","Footwear:B4","Footwear:B4","Footwear:B4","Footwear:B4","Footwear:B4","Footwear:B4","Footwear:B4","Footwear:B4","Footwear:B4"
>
>
> * The 2nd endpoint {glossary/importGlossaryData} (file upload) would actually
> parse the Data into AtlasObjects and further create the AtlasGlossaryTerms
> inside Glossary.
>
> curl -v -g POST -u admin:admin -H "Content-Type: multipart/form-data" -H
> "Cache-Control: no-cache" -F file=@template_6.csv
> "http://localhost:21000/api/atlas/v2/glossary/import
>
>
> Note:-
>
> While populating the data in the csv file each record should be maintained
> in single Line (enter command within the record would result in parsing the
> second line as a new record).
>
> The downloaded template needs to be saved as whateverTheFileNameIs.csv
> explicitly.
>
> If the file is been succefully uploaded then the AtlasGlossaryTerm would be
> returned or else List of Errors would returned for user to rectify them
> further.
>
>
> Diffs
> -
>
> common/src/main/java/org/apache/atlas/repository/Constants.java 7c0fd56
> intg/src/main/java/org/apache/atlas/AtlasErrorCode.java 1bd2fd0
>
> intg/src/main/java/org/apache/atlas/model/glossary/relations/AtlasGlossaryHeader.java
> 660514b
> pom.xml a6da9b8
> repository/pom.xml 802d587
> repository/src/main/java/org/apache/atlas/glossary/GlossaryService.java
> 9229d2d
> repository/src/main/java/org/apache/atlas/glossary/GlossaryTermUtils.java
> cdc3f07
> repository/src/main/java/org/apache/atlas/glossary/GlossaryUtils.java
> 9625f94
> repository/src/main/java/org/apache/atlas/util/FileUtils.java PRE-CREATION
> repository/src/test/java/org/apache/atlas/glossary/GlossaryServiceTest.java
> 759dcdf
> repository/src/test/resources/csvFiles/empty.csv PRE-CREATION
> repository/src/test/resources/csvFiles/incorrectFile.csv PRE-CREATION
> repository/src/test/resources/csvFiles/template_1.csv PRE-CREATION
> repository/src/test/resources/excelFiles/template_1.xlsx PRE-CREATION
> webapp/src/main/java/org/apache/atlas/web/rest/GlossaryREST.java 151aa6b
>
>
> Diff: https://reviews.apache.org/r/71482/diff/20/
>
>
> Testing
> ---
>
> Tested both the endpoint with curl call.
>
> Tested upload term csv with around 100 records from curl.
>
>
> Thanks,
>
> mayank jain
>
>
[jira] [Commented] (ATLAS-3665) Add 'queryText' attribute to the 'spark_process' type
[ https://issues.apache.org/jira/browse/ATLAS-3665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17063972#comment-17063972 ] ASF subversion and git services commented on ATLAS-3665: Commit 02df5c87f0bdee03142bcdc4e5cfb44edf0ad26f in atlas's branch refs/heads/branch-2.0 from Vlad Glinsky [ https://gitbox.apache.org/repos/asf?p=atlas.git;h=02df5c8 ] ATLAS-3665 :- Add 'queryText' attribute to the 'spark_process' type (#94) (cherry picked from commit a30c6d4e9c3855c15978bf1e50f287831bbae8bc) > Add 'queryText' attribute to the 'spark_process' type > - > > Key: ATLAS-3665 > URL: https://issues.apache.org/jira/browse/ATLAS-3665 > Project: Atlas > Issue Type: Task >Reporter: Vladislav Glinskiy >Priority: Major > Fix For: 2.1.0, 3.0.0 > > Time Spent: 2h > Remaining Estimate: 0h > > Add 'queryText' attribute to the 'spark_process' type in order to make > `spark_process` more readable by the user. The `queryText` attribute stores > exact SQL query that are executed within the Spark session as a Spark process. -- This message was sent by Atlassian Jira (v8.3.4#803005)
Enable support for solr in http mode.
Hi everyone, For testing purposes in our Atlas dev deployments In ING WBAA, it was required to run embedded solr with external Hbase. It was enabled by changes in the start scripts as well as in the way Solr client is configured to avoid setting up zookeeper by using solr in http mode. Additional benefit is full-integration with JanusGraph configuration. Here is propose solution to enable support of Solr in http(standalone) mode: https://issues.apache.org/jira/browse/ATLAS-3654 https://github.com/apache/atlas/pull/90 As it is my first PR to Atlas please let me know if I missed anything here. Damian
Re: Review Request 72259: ATLAS-3675 Enable quick search and suggestions for business metadata attributes
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/72259/#review220054 --- Ship it! Ship It! - Madhan Neethiraj On March 21, 2020, 8:41 a.m., Mandar Ambawane wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/72259/ > --- > > (Updated March 21, 2020, 8:41 a.m.) > > > Review request for atlas, Ashutosh Mestry, Jayendra Parab, Madhan Neethiraj, > Nixon Rodrigues, and Sarath Subramanian. > > > Bugs: ATLAS-3675 > https://issues.apache.org/jira/browse/ATLAS-3675 > > > Repository: atlas > > > Description > --- > > Added solr index for business metadata attributes. > > From UI, need to add searchWeight with metadata attributes to support > suggestions. > > > Diffs > - > > intg/src/main/java/org/apache/atlas/listener/ChangedTypeDefs.java af00f6c > > repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java > 7c55130 > > repository/src/main/java/org/apache/atlas/repository/graph/SolrIndexHelper.java > 4ec0086 > > > Diff: https://reviews.apache.org/r/72259/diff/1/ > > > Testing > --- > > Pre-commit build: > https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/1746/console > > Basic testing is done. > > > Thanks, > > Mandar Ambawane > >
