[ https://issues.apache.org/jira/browse/ATLAS-4922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Disha Talreja updated ATLAS-4922: --------------------------------- Summary: Atlas Async Import using Kafka (was: Atlas Async Import Functionality using Kafka) > Atlas Async Import using Kafka > ------------------------------ > > Key: ATLAS-4922 > URL: https://issues.apache.org/jira/browse/ATLAS-4922 > Project: Atlas > Issue Type: New Feature > Components: atlas-core > Reporter: Disha Talreja > Assignee: Disha Talreja > Priority: Major > > Develop an import process flow in Apache Atlas that leverages Kafka to manage > parallel import requests. This solution will use a dedicated Kafka topic > (ATLAS_IMPORT) to ensure import requests are processed efficiently and > without conflicts. The ImportService will be enhanced to consume and process > each message from the topic, enabling seamless behind-the-scenes import > execution. > Users will be able to submit new import requests after the staging process, > which prepares the file stream for Kafka, is complete. This approach supports > concurrent imports and enhances the handling of import operations in Apache > Atlas. > Additionally, this feature will introduce two new API endpoints for tracking > the current status of imports. These endpoints will allow users to check both > the overall progress and detailed status of each import request at any stage > of the import lifecycle (e.g., "PROCESSING," "PROCESSED," "COMPLETE") using > the unique request ID. Users will also gain access to request metadata, > providing insights into the status and progress of their imports. -- This message was sent by Atlassian Jira (v8.20.10#820010)