[jira] [Updated] (ATLAS-4464) Ingest: Improve Rate of Ingest
[ https://issues.apache.org/jira/browse/ATLAS-4464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Mestry updated ATLAS-4464: --- Attachment: (was: ATLAS-4464-concurrent-notification-processing-obj-sync-v3.patch) > Ingest: Improve Rate of Ingest > -- > > Key: ATLAS-4464 > URL: https://issues.apache.org/jira/browse/ATLAS-4464 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Affects Versions: trunk >Reporter: Ashutosh Mestry >Assignee: Ashutosh Mestry >Priority: Major > Fix For: trunk > > > *Background* > Existing implementation of _NotificationHookConsumer_ has linear complexity > for ingestion. This has several impacts: > * Authorization policies will take longer to get enforced. > * Unpredictable wait times for metadata showing up in Atlas. > *Solution* > Implement a mechanism for processing messages such that: > * Determine dependencies within incoming messages. > * Dependent messages should get processed serially. > * Messages without dependencies are processed concurrently. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-4464) Ingest: Improve Rate of Ingest
[ https://issues.apache.org/jira/browse/ATLAS-4464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Mestry updated ATLAS-4464: --- Attachment: ATLAS-4464-concurrent-notification-processing-obj-sync-v3.patch > Ingest: Improve Rate of Ingest > -- > > Key: ATLAS-4464 > URL: https://issues.apache.org/jira/browse/ATLAS-4464 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Affects Versions: trunk >Reporter: Ashutosh Mestry >Assignee: Ashutosh Mestry >Priority: Major > Fix For: trunk > > Attachments: > ATLAS-4464-concurrent-notification-processing-obj-sync-v3.patch > > > *Background* > Existing implementation of _NotificationHookConsumer_ has linear complexity > for ingestion. This has several impacts: > * Authorization policies will take longer to get enforced. > * Unpredictable wait times for metadata showing up in Atlas. > *Solution* > Implement a mechanism for processing messages such that: > * Determine dependencies within incoming messages. > * Dependent messages should get processed serially. > * Messages without dependencies are processed concurrently. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
