Review Request 73418: ATLAS-4335: Hook Notifications through Rest Interface
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73418/ --- Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, Radhika Kundam, Sarath Subramanian, Sidharth Mishra, and Umesh Padashetty. Bugs: ATLAS-4335 https://issues.apache.org/jira/browse/ATLAS-4335 Repository: atlas Description --- There are 4 parts to this project. - The first part is Notification Rest Service. In this part a new post API (api/atlas/v2/notification/topic/\) is created. NotificationREST.java is the entry point having a method handleNotifications. It does verify the authorization of the incoming requests. It further checks if the provided topic-name is among valid supported topic names. After validations it handover incoming messages to the Kafka notifier which in turn does the job of bumping messages in Kafka topic. An Integration test(NotificationRestIT.java) is also created to cover this new API Impacted files are listed below. webapp/src/main/java/org/apache/atlas/web/rest/NotificationREST.java authorization/src/main/java/org/apache/atlas/authorize/AtlasPrivilege.java intg/src/main/java/org/apache/atlas/AtlasErrorCode.java webapp/src/test/java/org/apache/atlas/web/integration/NotificationRestIT.java webapp/src/test/resources/json/notifications/create-db-ddl.json webapp/src/test/resources/json/notifications/create-db.json webapp/src/test/resources/json/notifications/delete-db.json --- The second part is Rest Notifier In this part, a new Notifier(RestNotification.java) is created. It extends the AbstractNotification interface and works on the same line as KafkaNotification and AtlasFileSpool, it just that it uses AtlasClientV2 to send the notifications to the abovementioned rest API. A new API endpoint is added in the AtlasClientV2. NotificationProvider is modified to instantiate the new rest notifier based on the configuration flag(atlas.hook.rest.notification.enabled) Unit test(RestNotificationTest.java) has been added to cover basic flows. AtlasBaseClient is modified to make one of the static variables available to the mentioned unit test. Impacted files are listed below. client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java notification/src/main/java/org/apache/atlas/kafka/NotificationProvider.java notification/src/main/java/org/apache/atlas/notification/rest/RestNotification.java intg/src/main/java/org/apache/atlas/AtlasConfiguration.java notification/src/test/java/org/apache/atlas/notification/RestNotificationTest.java client/common/src/main/java/org/apache/atlas/AtlasBaseClient.java - The third part is Buffered Kafka consumer (tumbling window solution) This part addresses the reordering of out or order notifications. With the introduction of a new notification delivery mechanism based on rest(RestNotification), it was observed that notifications were landing on the Kafka, not in the same order as they were produced at the services. Particularly when two different services generate events by the same trigger(HMS and HS2). This problem was there with the KafkaNotification delivery mechanism as well, but it never surfaces so frequently as there is a delay in the HS2 events, and the Kafka client delivers messages almost instantaneously to Kafka. With the rest route, the delivery time is not instantaneous, especially in a non-kerbarised or non-token-based authentication setup. To cover this AtlasKafkaBufferedConsumer is created. It’s a tumbling window solution on the stream of events. Here while consuming events, instead of reading events one by one, it loads events from Kafka in batches(size = 10 events by default). It has 2 such batches in memory, the current batch, and the lookup batch. In a cycle, it processes the current batch, but before processing, it pulls into the current batch all events in the lookup batch having timestamp lower than the greatest event timestamp in the current batch. It also sorts events in the current batch by event timestamp, thus fixes the order of messages. AtlasKafkaBufferedConsumer's constructor can throw AtlasException therefore KafkaNotification and NotificationHookConsumer are modified to cover for the exception. Unit test(AtlasKafkaBufferedConsumer) was added to cover the flow extensively. Impacted files are listed below. notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaBufferedConsumer.java notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerKafkaTest.java notification/src/test/java/org/apache/a
[jira] [Commented] (ATLAS-4324) FS entity created for load data inpath is created as shell entity
[ https://issues.apache.org/jira/browse/ATLAS-4324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17361923#comment-17361923 ] ASF subversion and git services commented on ATLAS-4324: Commit 1ecc6099b0a51d4c944741230ee0f99d0f592ca1 in atlas's branch refs/heads/branch-2.0 from Ashutosh Mestry [ https://gitbox.apache.org/repos/asf?p=atlas.git;h=1ecc609 ] ATLAS-4324: FS entity created for 'load data inpath' Part 2 > FS entity created for load data inpath is created as shell entity > - > > Key: ATLAS-4324 > URL: https://issues.apache.org/jira/browse/ATLAS-4324 > Project: Atlas > Issue Type: Bug > Components: hive-integration >Affects Versions: 2.1.0 >Reporter: Radhika Kundam >Assignee: Radhika Kundam >Priority: Major > Labels: hive-hooks > Fix For: 3.0.0, 2.2.0 > > Attachments: load_data_shell_entity.png > > > beeline > > create external table default.hive_table_cloud_load_data_in_path_123 > (student_roll int, student_name string, student_dob date) ROW FORMAT > DELIMITED FIELDS TERMINATED BY ' ' STORED AS TEXTFILE location > 'hdfs://ns1/tmp/hive_table_cloud_load_data_in_path_123' > > load data inpath 'hdfs://ns1/tmp/data123.txt' into table > hive_table_cloud_load_data_in_path_123; > > Creates 'hdfs://ns1/tmp/data123.txt' as shell entity. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (ATLAS-4324) FS entity created for load data inpath is created as shell entity
[ https://issues.apache.org/jira/browse/ATLAS-4324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17361911#comment-17361911 ] ASF subversion and git services commented on ATLAS-4324: Commit 0ac4b12be7c38ba34951da6bdbe82d23968cce13 in atlas's branch refs/heads/master from Ashutosh Mestry [ https://gitbox.apache.org/repos/asf?p=atlas.git;h=0ac4b12 ] ATLAS-4324: FS entity created for 'load data inpath' Part 2 > FS entity created for load data inpath is created as shell entity > - > > Key: ATLAS-4324 > URL: https://issues.apache.org/jira/browse/ATLAS-4324 > Project: Atlas > Issue Type: Bug > Components: hive-integration >Affects Versions: 2.1.0 >Reporter: Radhika Kundam >Assignee: Radhika Kundam >Priority: Major > Labels: hive-hooks > Fix For: 3.0.0, 2.2.0 > > Attachments: load_data_shell_entity.png > > > beeline > > create external table default.hive_table_cloud_load_data_in_path_123 > (student_roll int, student_name string, student_dob date) ROW FORMAT > DELIMITED FIELDS TERMINATED BY ' ' STORED AS TEXTFILE location > 'hdfs://ns1/tmp/hive_table_cloud_load_data_in_path_123' > > load data inpath 'hdfs://ns1/tmp/data123.txt' into table > hive_table_cloud_load_data_in_path_123; > > Creates 'hdfs://ns1/tmp/data123.txt' as shell entity. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (ATLAS-4327) UI: Deleting all glossaries and then switching to new UI results in unexpected behaviour
[ https://issues.apache.org/jira/browse/ATLAS-4327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17361771#comment-17361771 ] ASF subversion and git services commented on ATLAS-4327: Commit 817afa0a33d31a32f5ed94e30a8e6584eccf77a9 in atlas's branch refs/heads/master from prasad pawar [ https://gitbox.apache.org/repos/asf?p=atlas.git;h=817afa0 ] ATLAS-4327: UI: Deleting all glossaries and then switching to new UI results in unexpected behaviour. Signed-off-by: nixonrodrigues > UI: Deleting all glossaries and then switching to new UI results in > unexpected behaviour > > > Key: ATLAS-4327 > URL: https://issues.apache.org/jira/browse/ATLAS-4327 > Project: Atlas > Issue Type: Bug >Reporter: Rahul Kurup >Assignee: Prasad P. Pawar >Priority: Major > Attachments: > 0001-ATLAS-4327-1-UI-Deleting-all-glossaries-and-then-swi.patch > > > As seen in the below video url, if the user deletes all glossaries from old > UI and then switches over to new UI, the Glossary section is missing and a > endless loader animation keeps playing. > https://drive.google.com/file/d/1lXoFEGNNkC_49Zhc8Vb4rT47GT_YYPSq/view?usp=sharing -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (ATLAS-4327) UI: Deleting all glossaries and then switching to new UI results in unexpected behaviour
[ https://issues.apache.org/jira/browse/ATLAS-4327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17361773#comment-17361773 ] ASF subversion and git services commented on ATLAS-4327: Commit b8763b996e9430235e4d44539a46c3d56a6d11ac in atlas's branch refs/heads/branch-2.0 from prasad pawar [ https://gitbox.apache.org/repos/asf?p=atlas.git;h=b8763b9 ] ATLAS-4327: UI: Deleting all glossaries and then switching to new UI results in unexpected behaviour. Signed-off-by: nixonrodrigues (cherry picked from commit 817afa0a33d31a32f5ed94e30a8e6584eccf77a9) > UI: Deleting all glossaries and then switching to new UI results in > unexpected behaviour > > > Key: ATLAS-4327 > URL: https://issues.apache.org/jira/browse/ATLAS-4327 > Project: Atlas > Issue Type: Bug >Reporter: Rahul Kurup >Assignee: Prasad P. Pawar >Priority: Major > Attachments: > 0001-ATLAS-4327-1-UI-Deleting-all-glossaries-and-then-swi.patch > > > As seen in the below video url, if the user deletes all glossaries from old > UI and then switches over to new UI, the Glossary section is missing and a > endless loader animation keeps playing. > https://drive.google.com/file/d/1lXoFEGNNkC_49Zhc8Vb4rT47GT_YYPSq/view?usp=sharing -- This message was sent by Atlassian Jira (v8.3.4#803005)
Review Request 73416: ATLAS-4336 Restrict attributes of type to be created starting with double underscore
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73416/ --- Review request for atlas, Ashutosh Mestry, Jayendra Parab, Nixon Rodrigues, and Sarath Subramanian. Bugs: ATLAS-4336 https://issues.apache.org/jira/browse/ATLAS-4336 Repository: atlas Description --- Currently type can be updated with an attribute whose name starts with "__" like __guid. This gets mixed up with system attributes. Hence attribute names should not be allowed to be started with double underscore or following attribute names (system attribute names) should be restricted when a type is created or updated : __classificationNames __modifiedBy __createdBy __state __typeName __modificationTimestamp __propagatedClassificationNames __customAttributes __isIncomplete __guid __timestamp __labels Diffs - repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasAbstractDefStoreV2.java 3dab120 Diff: https://reviews.apache.org/r/73416/diff/1/ Testing --- Thanks, Mandar Ambawane
[jira] [Assigned] (ATLAS-4336) Restrict attributes of type to be created starting with double underscore
[ https://issues.apache.org/jira/browse/ATLAS-4336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mandar Ambawane reassigned ATLAS-4336: -- Assignee: Mandar Ambawane > Restrict attributes of type to be created starting with double underscore > - > > Key: ATLAS-4336 > URL: https://issues.apache.org/jira/browse/ATLAS-4336 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Reporter: Sharmadha S >Assignee: Mandar Ambawane >Priority: Major > > Currently type can be updated with an attribute whose name starts with "__" > like __guid. > This gets mixed up with system attributes. > Hence attribute names should not be allowed to be started with double > underscore or following attribute names (system attribute names) should be > restricted when a type is created or updated : > __classificationNames > __modifiedBy > __createdBy > __state > __typeName > __modificationTimestamp > __propagatedClassificationNames > __customAttributes > __isIncomplete > __guid > __timestamp > __labels -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (ATLAS-4328) Add flink favicon
[ https://issues.apache.org/jira/browse/ATLAS-4328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17361682#comment-17361682 ] ASF subversion and git services commented on ATLAS-4328: Commit 2733758bb2935aea96793e4adbb9793043f074f2 in atlas's branch refs/heads/branch-2.0 from prasad pawar [ https://gitbox.apache.org/repos/asf?p=atlas.git;h=2733758 ] ATLAS-4328: UI: Add flink type icons / images. via (Josh Yeh ) Signed-off-by: nixonrodrigues (cherry picked from commit cf925b6821854a6d6accd1875a3b67805cae110c) > Add flink favicon > - > > Key: ATLAS-4328 > URL: https://issues.apache.org/jira/browse/ATLAS-4328 > Project: Atlas > Issue Type: Task > Components: atlas-intg >Reporter: Josh Yeh >Assignee: Prasad P. Pawar >Priority: Trivial > Attachments: > 0001-ATLAS-4328-UI-Add-flink-favicon-images-fixed-1.patch, flink.png, > flink_process.png, image-2021-06-07-08-41-37-559.png > > > While testing ATLAS-3812, I noticed flink_application and flink_process icons > are missing. File this Jira to track. > !image-2021-06-07-08-41-37-559.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (ATLAS-4336) Restrict attributes of type to be created starting with double underscore
Sharmadha S created ATLAS-4336: -- Summary: Restrict attributes of type to be created starting with double underscore Key: ATLAS-4336 URL: https://issues.apache.org/jira/browse/ATLAS-4336 Project: Atlas Issue Type: Bug Components: atlas-core Reporter: Sharmadha S Currently type can be updated with an attribute whose name starts with "__" like __guid. This gets mixed up with system attributes. Hence attribute names should not be allowed to be started with double underscore or following attribute names (system attribute names) should be restricted when a type is created or updated : __classificationNames __modifiedBy __createdBy __state __typeName __modificationTimestamp __propagatedClassificationNames __customAttributes __isIncomplete __guid __timestamp __labels -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (ATLAS-4328) Add flink favicon
[ https://issues.apache.org/jira/browse/ATLAS-4328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17361677#comment-17361677 ] ASF subversion and git services commented on ATLAS-4328: Commit cf925b6821854a6d6accd1875a3b67805cae110c in atlas's branch refs/heads/master from prasad pawar [ https://gitbox.apache.org/repos/asf?p=atlas.git;h=cf925b6 ] ATLAS-4328: UI: Add flink type icons / images. via (Josh Yeh ) Signed-off-by: nixonrodrigues > Add flink favicon > - > > Key: ATLAS-4328 > URL: https://issues.apache.org/jira/browse/ATLAS-4328 > Project: Atlas > Issue Type: Task > Components: atlas-intg >Reporter: Josh Yeh >Assignee: Prasad P. Pawar >Priority: Trivial > Attachments: > 0001-ATLAS-4328-UI-Add-flink-favicon-images-fixed-1.patch, flink.png, > flink_process.png, image-2021-06-07-08-41-37-559.png > > > While testing ATLAS-3812, I noticed flink_application and flink_process icons > are missing. File this Jira to track. > !image-2021-06-07-08-41-37-559.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-4332) DSL Query : query with like operator and "/" in search text throws 500
[ https://issues.apache.org/jira/browse/ATLAS-4332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pinal updated ATLAS-4332: - Fix Version/s: 3.0.0 > DSL Query : query with like operator and "/" in search text throws 500 > -- > > Key: ATLAS-4332 > URL: https://issues.apache.org/jira/browse/ATLAS-4332 > Project: Atlas > Issue Type: Bug > Components: atlas-core >Reporter: Pinal Shah >Assignee: Pinal >Priority: Major > Labels: AdvancedSearch > Fix For: 2.1.0, 3.0.0 > > > hdfs_path name="/warehouse/tablespace/external/hive/db_fdchj.db" returns > right entity. > hdfs_path name like "/warehouse/*" throws following exception: -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (ATLAS-4327) UI: Deleting all glossaries and then switching to new UI results in unexpected behaviour
[ https://issues.apache.org/jira/browse/ATLAS-4327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17361516#comment-17361516 ] Rahul Kurup commented on ATLAS-4327: +1 > UI: Deleting all glossaries and then switching to new UI results in > unexpected behaviour > > > Key: ATLAS-4327 > URL: https://issues.apache.org/jira/browse/ATLAS-4327 > Project: Atlas > Issue Type: Bug >Reporter: Rahul Kurup >Assignee: Prasad P. Pawar >Priority: Major > Attachments: > 0001-ATLAS-4327-1-UI-Deleting-all-glossaries-and-then-swi.patch > > > As seen in the below video url, if the user deletes all glossaries from old > UI and then switches over to new UI, the Glossary section is missing and a > endless loader animation keeps playing. > https://drive.google.com/file/d/1lXoFEGNNkC_49Zhc8Vb4rT47GT_YYPSq/view?usp=sharing -- This message was sent by Atlassian Jira (v8.3.4#803005)
