--
From:liwei li
Send Time:2022 Oct. 10 (Mon.) 13:22
To:Steven Wu
Cc:Krzysztof Chmielewski ; Yun Gao
; dev
Subject:Re: Sink V2 interface replacement for GlobalCommitter
Thanks for the discussion.
Favor schemes that allow for custom committer parallelism
(parallelism = 1)`.
> >>
> >> Do you think this would be reasonable?
> >>
> >> Also @Krzysztof I have the same question with Steven that is it possible
> >> directly write to
> >> the formal files and skip the step of renaming? Since before the meta is
&g
directly write to
>> the formal files and skip the step of renaming? Since before the meta is
>> written to the Delta Log
>> I guess the files are not visible to the users, thus it is safe to
>> directly write to the formal files?
>>
>> Best,
>> Yun Gao
>>
>>
>> [1] ht
[1] https://issues.apache.org/jira/browse/FLINK-29459
>
>
>
> ----------
> From:Steven Wu
> Send Time:2022 Sep. 14 (Wed.) 21:33
> To:Krzysztof Chmielewski
> Cc:dev ; Yun Gao ; hililiwei <
> hilili...@gmail.com&g
ra/browse/FLINK-29459
<https://issues.apache.org/jira/browse/FLINK-29459 >
--
From:Steven Wu
Send Time:2022 Sep. 14 (Wed.) 21:33
To:Krzysztof Chmielewski
Cc:dev ; Yun Gao ; hililiwei
Subject:Re: Sink V2 interface replacemen
all the records could be committed at the end of bounded
>>>> streaming job, we have to desert the support for the cascade commits,
>>>> which makes the cascade commit of `committer -> global committer` not
>>>> work
>>>> in all cases.
>>
olves
>>> the issue in the current scenarios? I'll also have a double check with if
>>> it could be implemented and the failed tests Krzysztof met.
>>> Best,
>>> Yun
>>> --
>>> F
he issue in the current scenarios? I'll also have a double check with if
>> it could be implemented and the failed tests Krzysztof met.
>> Best,
>> Yun
>> ------------------
>> From:Steven Wu
>> Send Time:2022 Sep. 10 (Sat.
lemented and the failed tests Krzysztof met.
> Best,
> Yun
> --
> From:Steven Wu
> Send Time:2022 Sep. 10 (Sat.) 11:31
> To:dev
> Cc:Yun Gao ; hililiwei
> Subject:Re: Sink V2 interface replacement for GlobalCommitter
> Martjin, thanks a lot for chiming in!
> Her
dev
Cc:Yun Gao ; hililiwei
Subject:Re: Sink V2 interface replacement for GlobalCommitter
Martjin, thanks a lot for chiming in!
Here are my concerns with adding GlobalCommitter in the PostCommitTopology
1. when we use TwoPhaseCommittingSink. We would need to create a noop/dummy
committer. Act
> interface
> > > > > onto an implementation of `WithPostCommitTopology`.
> > > > > Since `WithPostCommitTopology` supports arbitrary subgraph, thus It
> > > seems
> > > > > to
> > > > > me it could support both global committer and small file
> compaction?
> > We
&
> > > > might
> > > > have an `WithPostCommitTopology` implementation like
> > > > DataStream ds = add global committer;
> > > > if (enable file compaction) {
> > > > build the compaction subgraph from ds
> > > > }
> > > &g
est,
> > > Yun
> > > [1]
> > >
> >
> https://github.com/apache/flink/blob/a8ca381c57788cd1a1527e4ebdc19bdbcd132fc4/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/transformations/SinkV1Adapter.java#L365
> > > <
> > >
> >
> https://github.com/apache/flink/blob/a8ca381c57788cd1a1527e4ebdc19bdbcd132fc4/flink-str
ache/flink/streaming/api/transformations/SinkV1Adapter.java#L365
> > <
> >
> https://github.com/apache/flink/blob/a8ca381c57788cd1a1527e4ebdc19bdbcd132fc4/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/transformations/SinkV1Adapter.java#L365
> > >
&
-
> From:Steven Wu
> Send Time:2022 Aug. 17 (Wed.) 07:30
> To:dev ; hililiwei
> Subject:Re: Sink V2 interface replacement for GlobalCommitter
> > Plus, it will disable the future capability of small file compaction
> stage post commit.
> I should clarify this comment. if w
iliwei
Subject:Re: Sink V2 interface replacement for GlobalCommitter
> Plus, it will disable the future capability of small file compaction
stage post commit.
I should clarify this comment. if we are using the `WithPostCommitTopology`
for global committer, we would lose the capability of using
> Plus, it will disable the future capability of small file compaction
stage post commit.
I should clarify this comment. if we are using the `WithPostCommitTopology`
for global committer, we would lose the capability of using the post commit
stage for small files compaction.
On Tue, Aug 16,
In the V1 sink interface, there is a GlobalCommitter for Iceberg. With the
V2 sink interface, GlobalCommitter has been deprecated by
WithPostCommitTopology. I thought the post commit stage is mainly for async
maintenance (like compaction).
Are we supposed to do sth similar to the
18 matches
Mail list logo
