ng at the interfaces. If I am reading it correctly,there
>> is
>> > >>> one relationship between the source and sink and this relationship
>> > >>> represents the operational lineage. Lineage is usually represented
>> as asset
>> > >>> -
cess - > asset – see for example
> > >>>
> https://egeria-project.org/features/lineage-management/overview/#the-lineage-graph
> > >>>
> > >>> Maybe I am missing it, but it seems to be that it would be useful to
> > >>> store the
neage-graph
> >>>
> >>> Maybe I am missing it, but it seems to be that it would be useful to
> >>> store the process in the lineage graph.
> >>>
> >>> It is useful to have the top level lineage as source -> Flink job ->
> >>&g
uld be useful to
>>> store the process in the lineage graph.
>>>
>>> It is useful to have the top level lineage as source -> Flink job ->
>>> sink. Where the Flink job is the process, but also to have this asset ->
>>> process -> asset pattern f
e the Flink job is the process, but also to have this asset ->
>> process -> asset pattern for each of the steps in the job. If this is
>> present, please could you point me to it,
>>
>> Kind regards, David.
>>
>>
>>
>>
>>
>> From: David R
gt;
>
>
>
> From: David Radley
> Date: Tuesday, 19 September 2023 at 16:11
> To: dev@flink.apache.org
> Subject: [EXTERNAL] RE: [DISCUSS] FLIP-314: Support Customized Job Lineage
> Listener
> Hi,
> I notice that there is an experimental lineage integration for Flink w
m: XTransfer
Date: Tuesday, 19 September 2023 at 15:47
To: dev@flink.apache.org
Subject: [EXTERNAL] Re: [DISCUSS] FLIP-314: Support Customized Job Lineage
Listener
Thanks Shammon for this proposal.
That’s helpful for collecting the lineage of Flink tasks.
Looking forward to its implementat
2023 at 15:47
To: dev@flink.apache.org
Subject: [EXTERNAL] Re: [DISCUSS] FLIP-314: Support Customized Job Lineage
Listener
Thanks Shammon for this proposal.
That’s helpful for collecting the lineage of Flink tasks.
Looking forward to its implementation.
Best,
Jiabao
> 2023年9月18日 20:56，Leonard
Thanks Shammon for this proposal.
That’s helpful for collecting the lineage of Flink tasks.
Looking forward to its implementation.
Best,
Jiabao
> 2023年9月18日 20:56，Leonard Xu 写道：
>
> Thanks Shammon for the informations, the comment makes the lifecycle clearer.
> +1
>
>
> Best,
> Leonard
>
Thanks Shammon for the informations, the comment makes the lifecycle clearer.
+1
Best,
Leonard
> On Sep 18, 2023, at 7:54 PM, Shammon FY wrote:
>
> Hi devs,
>
> After discussing with @Qingsheng, I fixed a minor issue of the lineage
> lifecycle in `StreamExecutionEnvironment`. I have
Hi devs,
After discussing with @Qingsheng, I fixed a minor issue of the lineage
lifecycle in `StreamExecutionEnvironment`. I have added the comment to
explain that the lineage information in `StreamExecutionEnvironment` will
be consistent with that of transformations. When users clear the
Hi devs,
Thanks for all the valuable feedback. If there are no more concerns about
FLIP-314[1], I would like to start voting later, thanks.
[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-314%3A+Support+Customized+Job+Lineage+Listener
Best,
Shammon FY
On Wed, Jul 12, 2023 at 11:18
Thanks for the valuable feedback, Leonard.
I have discussed with Leonard off-line. We have reached some conclusions
about these issues and I have updated the FLIP as follows:
1. Simplify the `LineageEdge` interface by creating an edge from one source
vertex to sink vertex.
2. Remove the
> Best,
>>> Shammon FY
>>>
>>>
>>> On Sun, Jun 25, 2023 at 4:17 PM Yun Tang wrote:
>>>
>>>> Hi Shammon,
>>>>
>>>> I like the idea in general and it will help to analysis the job
>> lineages
>>&g
gt; >
> > > > > > By the way, if you want to focus on job lineage instead of data
> > > column
> > > > > lineage in this FLIP, why we must introduce so many column-lineage
> > > > related
> > > > > interface here?
>
column
> > > > lineage. Although SQL jobs currently do not support column lineage,
> we
> > > > would like to support this in the next step. So we have
> comprehensively
> > > > considered the table lineage and column lineage interfaces here, and
> > > > defined these two i
gt; > Best,
> > > Shammon FY
> > >
> > >
> > > On Sun, Jun 25, 2023 at 4:17 PM Yun Tang wrote:
> > >
> > > > Hi Shammon,
> > > >
> > > > I like the idea in general and it will help to analysis the job
> > lineages
ther clearly
>> >
>> >
>> > [1]
>> >
>> >
>> https://github.com/FangYongs/flink/commit/d4bfe57e7a5315b790e79b8acef8b11e82c9187c
>> >
>> > Best,
>> > Shammon FY
>> >
>> >
>> > On Sun, Jun 25,
not easy to understand for
> > users.
> > >
> > > I have one more question on the lookup-join dim tables, it seems this
> > FLIP
> > > does not touch them, and will them become part of the
> List
> > > sources() or adding another interface?
&g
re question on the lookup-join dim tables, it seems this
> FLIP
> > does not touch them, and will them become part of the List
> > sources() or adding another interface?
> >
> > By the way, if you want to focus on job lineage instead of data column
> > lineage in th
environments.
> > >
> > > For Qingsheng's concern, I'd like the name of JobType more than
> > > RuntimeExecutionMode, as the latter one is not easy to understand for
> > users.
> > >
> > > I have one more question on the lookup-join dim table
t; FLIP
> > does not touch them, and will them become part of the List
> > sources() or adding another interface?
> >
> > By the way, if you want to focus on job lineage instead of data column
> > lineage in this FLIP, why we must introduce so many column-lineage
s FLIP, why we must introduce so many column-lineage related
> interface here?
>
>
> Best
> Yun Tang
>
> From: Shammon FY
> Sent: Sunday, June 25, 2023 16:13
> To: dev@flink.apache.org
> Subject: Re: [DISCUSS] FLIP-314: Support Custom
column-lineage related interface
here?
Best
Yun Tang
From: Shammon FY
Sent: Sunday, June 25, 2023 16:13
To: dev@flink.apache.org
Subject: Re: [DISCUSS] FLIP-314: Support Customized Job Lineage Listener
Hi Qingsheng,
Thanks for your valuable feedback.
>
Hi Qingsheng,
Thanks for your valuable feedback.
> 1. Is there any specific use case to expose the batch / streaming info to
listeners or meta services?
I agree with you that Flink is evolving towards batch-streaming
unification, but the lifecycle of them is different. If a job processes a
tter about the interfaces.
Best regards,
Yuxia
- 原始邮件 -
发件人: "Qingsheng Ren"
收件人: "dev"
抄送: "Shammon FY"
发送时间: 星期二, 2023年 6 月 20日 下午 6:19:10
主题: Re: [DISCUSS] FLIP-314: Support Customized Job Lineage Listener
Hi Shammon,
Thanks for starting this FLIP! Data line
Hi Shammon,
Thanks for starting this FLIP! Data lineage is a very important topic,
which has been missing for a long time in Flink. I have some questions
about the FLIP.
About events and listeners:
1. I’m not sure if it is necessary to expose JobType to in JobCreatedEvent.
This is an internal
Hi devs,
Is there any comment or feedback for this FLIP? Hope to hear from you,
thanks
Best,
Shammon FY
On Tue, Jun 6, 2023 at 8:22 PM Shammon FY wrote:
> Hi devs,
>
> I would like to start a discussion on FLIP-314: Support Customized Job
> Lineage Listener[1] which is the next stage of
Hi devs,
I would like to start a discussion on FLIP-314: Support Customized Job
Lineage Listener[1] which is the next stage of FLIP-294 [2]. Flink
streaming and batch jobs create lineage dependency between source and sink,
users can manage their data and jobs according to this lineage
29 matches
Mail list logo
