[jira] [Updated] (ATLAS-3631) Make Server name header configurable and Basic auth improvement.
[ https://issues.apache.org/jira/browse/ATLAS-3631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chaitali borole updated ATLAS-3631: --- Fix Version/s: 3.0.0 > Make Server name header configurable and Basic auth improvement. > > > Key: ATLAS-3631 > URL: https://issues.apache.org/jira/browse/ATLAS-3631 > Project: Atlas > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: Nixon Rodrigues >Assignee: chaitali borole >Priority: Major > Fix For: 2.1.0, 3.0.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: Review Request 71791: ATLAS-3518: Create Audit Framework for Atlas Purge Entity
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71791/ --- (Updated Feb. 26, 2020, 9:21 p.m.) Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, and Sarath Subramanian. Bugs: ATLAS-3518 https://issues.apache.org/jira/browse/ATLAS-3518 Repository: atlas Description --- Right now for purge, delete entity, etc. at Atlas we create audit entry at HBase. User can go to entity and click on audit to see the audit information for that particular entity. But if user purge one entity there will not be any vertex at Janus graph and user will not able to get the purge audit which is important for governance purpose. As part of fix, implemented Atlas Audit Framework which is being used for Purge operation and can be extended for other operations. In Future once we change the AtlasAudit type as super type for Import export as well we can use this framework for all Atlas operations. admin/audit rest has been added which will return all the Atlas audit operations including import export at Atlas. Diffs (updated) - addons/models/-Area0/0010-base_model.json 6bdd2f7ba client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java 8c0a6407c dashboardv2/public/js/utils/Enums.js 82e11ca6c dashboardv3/public/js/utils/Enums.js 82e11ca6c intg/src/main/java/org/apache/atlas/model/audit/AtlasAuditEntry.java PRE-CREATION intg/src/main/java/org/apache/atlas/model/audit/AuditSearchParameters.java PRE-CREATION intg/src/main/java/org/apache/atlas/model/instance/EntityMutationResponse.java b448d510b repository/src/main/java/org/apache/atlas/repository/audit/AtlasAuditService.java PRE-CREATION repository/src/main/java/org/apache/atlas/repository/ogm/AtlasAuditEntryDTO.java PRE-CREATION repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerV1.java a87e2b57f repository/src/test/java/org/apache/atlas/TestModules.java 94b08a187 repository/src/test/java/org/apache/atlas/glossary/GlossaryServiceTest.java be1698fb4 repository/src/test/java/org/apache/atlas/query/BasicTestSetup.java 02f78b369 repository/src/test/java/org/apache/atlas/repository/audit/AdminPurgeTest.java PRE-CREATION repository/src/test/java/org/apache/atlas/repository/audit/AtlasAuditServiceTest.java PRE-CREATION repository/src/test/java/org/apache/atlas/repository/impexp/AtlasServerServiceTest.java 78865c569 repository/src/test/java/org/apache/atlas/repository/impexp/ExportImportAuditServiceTest.java ba7a8a04b repository/src/test/java/org/apache/atlas/repository/impexp/ExportImportTestBase.java 925b2a007 repository/src/test/java/org/apache/atlas/repository/impexp/ExportIncrementalTest.java 4d438525a repository/src/test/java/org/apache/atlas/repository/impexp/ExportServiceTest.java db8c6c83c repository/src/test/java/org/apache/atlas/repository/impexp/ExportSkipLineageTest.java 25e0a5362 repository/src/test/java/org/apache/atlas/repository/impexp/HdfsPathEntityCreatorTest.java 1863b8d74 repository/src/test/java/org/apache/atlas/repository/impexp/ImportReactivateTableTest.java d0c06a107 repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java c14850f43 repository/src/test/java/org/apache/atlas/repository/impexp/ImportTransformsShaperTest.java 78fdaca8d repository/src/test/java/org/apache/atlas/repository/impexp/IncrementalExportEntityProviderTest.java 42b63535b repository/src/test/java/org/apache/atlas/repository/impexp/RelationshipAttributesExtractorTest.java a1b512fb9 repository/src/test/java/org/apache/atlas/repository/impexp/ReplicationEntityAttributeTest.java 7a1ed18cb repository/src/test/java/org/apache/atlas/repository/impexp/StartEntityFetchByExportRequestTest.java 48b958271 repository/src/test/java/org/apache/atlas/repository/impexp/TableReplicationRequestProcessorTest.java c9bb11c56 repository/src/test/java/org/apache/atlas/repository/impexp/ZipFileResourceTestUtils.java 0ffc3d595 repository/src/test/java/org/apache/atlas/repository/migration/MigrationBaseAsserts.java 18e950e8c repository/src/test/java/org/apache/atlas/repository/migration/RelationshipCacheGeneratorTest.java 050bda3ae repository/src/test/java/org/apache/atlas/repository/store/graph/v2/AtlasNamespaceDefStoreV2Test.java cbac18c8c repository/src/test/java/org/apache/atlas/repository/tagpropagation/ClassificationPropagationTest.java 6269d3f74 repository/src/test/java/org/apache/atlas/repository/userprofile/UserProfileServiceTest.java e56e48308 repository/src/test/java/org/apache/atlas/services/MetricsServiceTest.java baeafd4b7 repository/src/test/java/org/apache/atlas/utils/TestLoadModelUtils.java PRE-CREATION
[jira] [Updated] (ATLAS-3624) Beta UI : Allow user to add multi-valued attributes in namespaces
[ https://issues.apache.org/jira/browse/ATLAS-3624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Keval Bhatt updated ATLAS-3624: --- Attachment: ATLAS-3624-3.patch > Beta UI : Allow user to add multi-valued attributes in namespaces > - > > Key: ATLAS-3624 > URL: https://issues.apache.org/jira/browse/ATLAS-3624 > Project: Atlas > Issue Type: Improvement >Reporter: Keval Bhatt >Assignee: Keval Bhatt >Priority: Major > Fix For: 2.1.0, 3.0.0 > > Attachments: ATLAS-3624-1.patch, ATLAS-3624-2.patch, > ATLAS-3624-3.patch, ATLAS-3624.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-3634) Betal UI: Add refresh button for basic search / Advance Search
[ https://issues.apache.org/jira/browse/ATLAS-3634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Keval Bhatt updated ATLAS-3634: --- Attachment: ATLAS-3634-1.patch > Betal UI: Add refresh button for basic search / Advance Search > -- > > Key: ATLAS-3634 > URL: https://issues.apache.org/jira/browse/ATLAS-3634 > Project: Atlas > Issue Type: Improvement >Reporter: Keval Bhatt >Assignee: Keval Bhatt >Priority: Major > Fix For: 2.1.0, 3.0.0 > > Attachments: ATLAS-3634-1.patch, ATLAS-3634.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-3615) Basic Search : Search with _ALL_ENTITY_TYPES and classification doesn't not fetch any entity associated to that classification
[
https://issues.apache.org/jira/browse/ATLAS-3615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pinal updated ATLAS-3615:
-
Attachment: (was: ATLAS-3615.2.patch)
> Basic Search : Search with _ALL_ENTITY_TYPES and classification doesn't not
> fetch any entity associated to that classification
> --
>
> Key: ATLAS-3615
> URL: https://issues.apache.org/jira/browse/ATLAS-3615
> Project: Atlas
> Issue Type: Bug
> Components: atlas-core
>Reporter: Pinal
>Assignee: Pinal
>Priority: Major
> Fix For: 2.1.0, 3.0.0
>
>
> 1. classification : tag1 returns all entities associated to tag1
> 2. typename : _ALL_ENTITY_TYPES returns all entities under all types.
> 3. classification : tag1 typename : hive_table returns all hive_table
> entities associated to tag1
> 4. classification : tag1 typename : _ALL_ENTITY_TYPES doesn't return any
> result. Expected it would return same results as query1.
> Basic search payload :
> {code:java}
> {
> "excludeDeletedEntities":true,
> "includeSubClassifications":true,
> "includeSubTypes":true,
> "includeClassificationAttributes":true,
> "entityFilters":null,
> "tagFilters":null,
> "attributes":[
> ],
> "limit":25,
> "offset":0,
> "typeName":"_ALL_ENTITY_TYPES",
> "classification":"tag1",
> "termName":null
> }
> {code}
> 5. classification : tag1, typename : _ALL_ENTITY_TYPES, Search by text, this
> combination doesn't return any result
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (ATLAS-3639) Enable to select BigQuery as a GraphStorageBackend instead of HBase
Masaru Uchida created ATLAS-3639:
Summary: Enable to select BigQuery as a GraphStorageBackend
instead of HBase
Key: ATLAS-3639
URL: https://issues.apache.org/jira/browse/ATLAS-3639
Project: Atlas
Issue Type: Wish
Reporter: Masaru Uchida
Hi, I'm developing an ETL infrastructure and researching Atlas as part of a
metadata management infrastructure replacement.
h2. Overview
I want to be able to use Bigquery for graph storage backend.
h2. Why
* Building hbase on your own is very painful.
* Introducing managed services lowers installation costs.
h2. What did and did not work
By replacing Hbase of {{HbaseStoreManager}} (atlas-janusgraph-hbase2) with
{{BigtableConfiguration}} , operation such as startup and registration of
Entity could be confirmed.
{code:java}
-this.hconf = HBaseConfiguration.create();
+String PROJECT_ID = "my-gcp-project-id";
+String INSTANCE_ID = "my-bigtable-instance-id";
+String APP_PROFILE_ID = "default";
+org.apache.hadoop.conf.Configuration conf =
BigtableConfiguration.configure(PROJECT_ID, INSTANCE_ID);
+Credentials credentials =
CredentialFactory.getApplicationDefaultCredential();
+
+this.hconf = BigtableConfiguration.withCredentials(conf, credentials);
+hconf.set(BigtableOptionsFactory.APP_PROFILE_ID_KEY, APP_PROFILE_ID);
{code}
https://github.com/usadamasa/atlas/blob/e2c2420aebe9dd6ebbd8ef7a41667a76ed621657/graphdb/janus-hbase2/src/main/java/org/janusgraph/diskstorage/hbase2/HBaseStoreManager.java#L295-L303
https://github.com/usadamasa/atlas/blob/e2c2420aebe9dd6ebbd8ef7a41667a76ed621657/graphdb/janus-hbase2/src/main/java/org/janusgraph/diskstorage/hbase2/HBaseCompat2_0.java#L48
(this branch from {{origin/branch-2.0}} )
However, {{Scan().SetReversed(true)}} calling in {{HBaseBasedAuditRepository}}
fails in audit log search due to API not supported by Bigtable.
bq. Reverse scans are not supported. You cannot call the method
org.apache.hadoop.hbase.client.Scan#setReversed(boolean reversed).
https://cloud.google.com/bigtable/docs/hbase-differences#gets_and_scans
h2. Proposal
* Enable to change the implementation of {{HBaseStoreManager}} if bigtable is
specified in {{atlas.graph.storage.backend}} of atlas-application.properties.
* It would be nice if Solr or elasticsearch could be specified in
{{atlas.EntityAuditRepository.impl}} .
Sorry in poor english. thank you.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (ATLAS-3320) Import Service: Support Concurrent Ingest
[ https://issues.apache.org/jira/browse/ATLAS-3320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17045295#comment-17045295 ] ASF subversion and git services commented on ATLAS-3320: Commit 024f5d52d0a972616362ac9b15ed7a417384026f in atlas's branch refs/heads/master from Nixon Rodrigues [ https://gitbox.apache.org/repos/asf?p=atlas.git;h=024f5d5 ] Revert "ATLAS-3320: Import Service. Support concurrent ingest." This reverts commit a2ccfb9f3577e911103041d8d4b91c169697f6a1. > Import Service: Support Concurrent Ingest > - > > Key: ATLAS-3320 > URL: https://issues.apache.org/jira/browse/ATLAS-3320 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Affects Versions: trunk >Reporter: Ashutosh Mestry >Assignee: Ashutosh Mestry >Priority: Major > Fix For: trunk > > Attachments: ATLAS-3320-ImportService-Concurrent-ingest.patch > > > *Background* > Existing implementation of import supports single entity creation. This > greatly limits the rate of ingest. > *Solution Approach* > * Use existing producer-consumer framework to allow for configurable > consumers of entities from the input stream. > * Modify _BulkImporterImpl_ to implement the consumer. > * Modify _AtlasImportRequest_ to specify the batch size and number of > workers. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (ATLAS-3601) No LABEL_DELETE event is audited when last label is deleted
[ https://issues.apache.org/jira/browse/ATLAS-3601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nixon Rodrigues resolved ATLAS-3601. Fix Version/s: 3.0.0 Resolution: Fixed > No LABEL_DELETE event is audited when last label is deleted > > > Key: ATLAS-3601 > URL: https://issues.apache.org/jira/browse/ATLAS-3601 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Affects Versions: 2.0.0 >Reporter: Pinal >Assignee: Pinal >Priority: Major > Fix For: 2.1.0, 3.0.0 > > Attachments: ATLAS-3601.patch > > > When 1 label among multiple labels is removed , LABEL_DELETE is audited. But > when all labels are deleted or the last label is deleted , no LABEL_DELETE is > audited. > Also , we see "One or more existing labels were removed successfully" popup > when there is no LABEL_DELETE audit. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (ATLAS-3615) Basic Search : Search with _ALL_ENTITY_TYPES and classification doesn't not fetch any entity associated to that classification
[
https://issues.apache.org/jira/browse/ATLAS-3615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17045292#comment-17045292
]
ASF subversion and git services commented on ATLAS-3615:
Commit bc685f22cd9ede632f955d34558e42de0044e247 in atlas's branch
refs/heads/branch-2.0 from Pinal Shah
[ https://gitbox.apache.org/repos/asf?p=atlas.git;h=bc685f2 ]
ATLAS-3615 :- Basic Search : Search with _ALL_ENTITY_TYPES and classification
doesn't not fetch any entity associated to that classification
Signed-off-by: nixonrodrigues
(cherry picked from commit 75e678da7595420a729226e9ce2b575facb7bbb5)
> Basic Search : Search with _ALL_ENTITY_TYPES and classification doesn't not
> fetch any entity associated to that classification
> --
>
> Key: ATLAS-3615
> URL: https://issues.apache.org/jira/browse/ATLAS-3615
> Project: Atlas
> Issue Type: Bug
> Components: atlas-core
>Reporter: Pinal
>Assignee: Pinal
>Priority: Major
> Attachments: ATLAS-3615.2.patch
>
>
> 1. classification : tag1 returns all entities associated to tag1
> 2. typename : _ALL_ENTITY_TYPES returns all entities under all types.
> 3. classification : tag1 typename : hive_table returns all hive_table
> entities associated to tag1
> 4. classification : tag1 typename : _ALL_ENTITY_TYPES doesn't return any
> result. Expected it would return same results as query1.
> Basic search payload :
> {code:java}
> {
> "excludeDeletedEntities":true,
> "includeSubClassifications":true,
> "includeSubTypes":true,
> "includeClassificationAttributes":true,
> "entityFilters":null,
> "tagFilters":null,
> "attributes":[
> ],
> "limit":25,
> "offset":0,
> "typeName":"_ALL_ENTITY_TYPES",
> "classification":"tag1",
> "termName":null
> }
> {code}
> 5. classification : tag1, typename : _ALL_ENTITY_TYPES, Search by text, this
> combination doesn't return any result
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (ATLAS-3601) No LABEL_DELETE event is audited when last label is deleted
[ https://issues.apache.org/jira/browse/ATLAS-3601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17045293#comment-17045293 ] ASF subversion and git services commented on ATLAS-3601: Commit b5c6e5dc4fda061b7d70660d36e2ba29e99b8d86 in atlas's branch refs/heads/branch-2.0 from Pinal Shah [ https://gitbox.apache.org/repos/asf?p=atlas.git;h=b5c6e5d ] ATLAS-3601 :- No LABEL_DELETE event is audited when last label is deleted. (cherry picked from commit c99e15b3711e0aebb4c1db086553f915ecd471d1) > No LABEL_DELETE event is audited when last label is deleted > > > Key: ATLAS-3601 > URL: https://issues.apache.org/jira/browse/ATLAS-3601 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Affects Versions: 2.0.0 >Reporter: Pinal >Assignee: Pinal >Priority: Major > Fix For: 2.1.0 > > Attachments: ATLAS-3601.patch > > > When 1 label among multiple labels is removed , LABEL_DELETE is audited. But > when all labels are deleted or the last label is deleted , no LABEL_DELETE is > audited. > Also , we see "One or more existing labels were removed successfully" popup > when there is no LABEL_DELETE audit. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (ATLAS-3601) No LABEL_DELETE event is audited when last label is deleted
[ https://issues.apache.org/jira/browse/ATLAS-3601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17045287#comment-17045287 ] ASF subversion and git services commented on ATLAS-3601: Commit c99e15b3711e0aebb4c1db086553f915ecd471d1 in atlas's branch refs/heads/master from Pinal Shah [ https://gitbox.apache.org/repos/asf?p=atlas.git;h=c99e15b ] ATLAS-3601 :- No LABEL_DELETE event is audited when last label is deleted. > No LABEL_DELETE event is audited when last label is deleted > > > Key: ATLAS-3601 > URL: https://issues.apache.org/jira/browse/ATLAS-3601 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Affects Versions: 2.0.0 >Reporter: Pinal >Assignee: Pinal >Priority: Major > Fix For: 2.1.0 > > Attachments: ATLAS-3601.patch > > > When 1 label among multiple labels is removed , LABEL_DELETE is audited. But > when all labels are deleted or the last label is deleted , no LABEL_DELETE is > audited. > Also , we see "One or more existing labels were removed successfully" popup > when there is no LABEL_DELETE audit. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (ATLAS-3615) Basic Search : Search with _ALL_ENTITY_TYPES and classification doesn't not fetch any entity associated to that classification
[
https://issues.apache.org/jira/browse/ATLAS-3615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17045286#comment-17045286
]
ASF subversion and git services commented on ATLAS-3615:
Commit 75e678da7595420a729226e9ce2b575facb7bbb5 in atlas's branch
refs/heads/master from Pinal Shah
[ https://gitbox.apache.org/repos/asf?p=atlas.git;h=75e678d ]
ATLAS-3615 :- Basic Search : Search with _ALL_ENTITY_TYPES and classification
doesn't not fetch any entity associated to that classification
Signed-off-by: nixonrodrigues
> Basic Search : Search with _ALL_ENTITY_TYPES and classification doesn't not
> fetch any entity associated to that classification
> --
>
> Key: ATLAS-3615
> URL: https://issues.apache.org/jira/browse/ATLAS-3615
> Project: Atlas
> Issue Type: Bug
> Components: atlas-core
>Reporter: Pinal
>Assignee: Pinal
>Priority: Major
> Attachments: ATLAS-3615.2.patch
>
>
> 1. classification : tag1 returns all entities associated to tag1
> 2. typename : _ALL_ENTITY_TYPES returns all entities under all types.
> 3. classification : tag1 typename : hive_table returns all hive_table
> entities associated to tag1
> 4. classification : tag1 typename : _ALL_ENTITY_TYPES doesn't return any
> result. Expected it would return same results as query1.
> Basic search payload :
> {code:java}
> {
> "excludeDeletedEntities":true,
> "includeSubClassifications":true,
> "includeSubTypes":true,
> "includeClassificationAttributes":true,
> "entityFilters":null,
> "tagFilters":null,
> "attributes":[
> ],
> "limit":25,
> "offset":0,
> "typeName":"_ALL_ENTITY_TYPES",
> "classification":"tag1",
> "termName":null
> }
> {code}
> 5. classification : tag1, typename : _ALL_ENTITY_TYPES, Search by text, this
> combination doesn't return any result
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
