Hi Rajat....... sorry!
Sent from IBM Verse
me --- Re: Regarding pipeline of multiple processes and ATLAS-1236 ---
From:"Ernie Ostic" <[email protected]>To:[email protected]:Thu, Aug
24, 2017 5:04 PMSubject:Re: Regarding pipeline of multiple processes and
ATLAS-1236
Hi Goel. Welcome. Great thoughts and use cases. I will let others chime
in here, but one question would be "what is your "process"? ". For example,
is it HiveQL? SQOOP? Something for which there exists today a supported
"hook"? (see the atlas.apache.org page with the high level specs, api, and
hooks list). ....and are your datasets known to Atlas (hive, for example), or
something external to Atlas? This would help outline what can work today
directly, or possibly need custom work...There is great work being done by the
team here to further enhance the underlying models (as you can see thru recent
jira's)....that will facilitate even wider creative use cases. Looking forward
to the details on your "process" and dataset object types.ErnieSent from IBM
Verse Rajat Goel --- Regarding pipeline of multiple processes and ATLAS-1236
--- From:"Rajat Goel"
<[email protected]>To:[email protected]:Thu, Aug 24, 2017 2:06
PMSubject:Regarding pipeline of multiple processes and ATLAS-1236 Hi,I am
new user and exploring Apache Atlas for metadata management. I have ause case
where I want to track lineage across a pipeline of processingfunctions i.e.
DataSet 1 -> Function/Process1 -> Process2 -> Process 3 ->DataSet 2.Another use
case is that each of the above Processes/Functions themselvescould be some
named pipelines. Eg. DataSet1 ->Process1 (which is pipeline of SubProcess1 ->
SubProcess2) -> Process2 ->DataSet2.Can Apache Atlas support these use cases
for metadata management andlineage?If yes, please suggest how. If not, is there
any plan to support these infuture?I found one Jira improvement ticket
ATLAS-1236 which looks to be relevantto the above use cases. Is there any plan
to resolve it in upcomingreleases?Thanks & Regards,Rajat Goel
