[ 
https://issues.apache.org/jira/browse/ATLAS-4922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Disha Talreja updated ATLAS-4922:
---------------------------------
    Summary: Atlas Async Import using Kafka  (was: Atlas Async Import 
Functionality using Kafka)

> Atlas Async Import using Kafka
> ------------------------------
>
>                 Key: ATLAS-4922
>                 URL: https://issues.apache.org/jira/browse/ATLAS-4922
>             Project: Atlas
>          Issue Type: New Feature
>          Components:  atlas-core
>            Reporter: Disha Talreja
>            Assignee: Disha Talreja
>            Priority: Major
>
> Develop an import process flow in Apache Atlas that leverages Kafka to manage 
> parallel import requests. This solution will use a dedicated Kafka topic 
> (ATLAS_IMPORT) to ensure import requests are processed efficiently and 
> without conflicts. The ImportService will be enhanced to consume and process 
> each message from the topic, enabling seamless behind-the-scenes import 
> execution.
> Users will be able to submit new import requests after the staging process, 
> which prepares the file stream for Kafka, is complete. This approach supports 
> concurrent imports and enhances the handling of import operations in Apache 
> Atlas.
> Additionally, this feature will introduce two new API endpoints for tracking 
> the current status of imports. These endpoints will allow users to check both 
> the overall progress and detailed status of each import request at any stage 
> of the import lifecycle (e.g., "PROCESSING," "PROCESSED," "COMPLETE") using 
> the unique request ID. Users will also gain access to request metadata, 
> providing insights into the status and progress of their imports.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Reply via email to