[
https://issues.apache.org/jira/browse/FALCON-287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13920953#comment-13920953
]
Shaik Idris Ali edited comment on FALCON-287 at 3/5/14 3:20 PM:
----------------------------------------------------------------
May be I missed the actual use-case of lineage, why do we need to persist it
and who are the consumers of this information.
Secondly, instead of FalconPostProcessing storing this data on HDFS, which
might further slowdown user workflow.
A service can subscribe to FALCON.ENTITY.TOPIC and record the details in HDFS
or anywhere.
I think we discussed about this doing in ProcessSubscriberService.
{quote}
There seems to be redundant work in the current form.
>This is the beginning and as I had said previously, I'll add more data for
>each feed.
{quote}
>> May be I got the intent of storing this, but what all additional data we
>> require for each feed.
+1 for moving from ARG to JSON.
Can we discuss this over a chat? 11 PM IST.
was (Author: shaik.idris):
May be I missed the actual use-case of lineage, why do we need to persist it
and who are the consumers of this information.
Secondly, instead of FalconPostProcessing storing this data on HDFS, which
might further slowdown user workflow.
A service can subscribe to FALCON.ENTITY.TOPIC and record the details in HDFS
or anywhere.
I think we discussed about this doing in ProcessSubscriberService.
Can we discuss this over a chat? 11 PM IST.
> Record lineage information in post processing
> ---------------------------------------------
>
> Key: FALCON-287
> URL: https://issues.apache.org/jira/browse/FALCON-287
> Project: Falcon
> Issue Type: Sub-task
> Affects Versions: 0.5
> Reporter: Venkatesh Seetharam
> Assignee: Venkatesh Seetharam
> Labels: lineage
> Attachments: FALCON-287-v1.patch, FALCON-287-v2.patch,
> FALCON-287.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.2#6252)
