IP-217? Can not Iceberg
> just
> >> >> emit
> >> >>>>> all splits and let FLIP-182/FLIP-217 handle the watermark
> alignment
> >> >> and
> >> >>>>> block the splits that are too much into the future? I can see this
> >> >>
t; >>>>> issue if the existence of too many blocked splits is occupying too
>> >> many
>> >>>>> resources.
>> >>>>>
>> >>>>> If that's the case, indeed SourceCoordinator/SplitEnumerator would
>
s how many and which splits to assign in what
> >> order. But
> >>>>> in that case I'm not sure how much you could use from FLIP-182 and
> >>>>> FLIP-217. They seem somehow orthogonal to me, operating on different
> >>>>> levels. FLIP-18
ts have
>> already
>>>>> been generated and assigned. You could leverage FLIP-182 and FLIP-217
>> and
>>>>> take care of only the problem to limit the number of parallel active
>>>>> splits. And here I'm not sure if it would be worth generalis
lated comment sometime ago
> >> > about it [1]. It sounds to me like you also need to solve this
> problem,
> >> > otherwise Iceberg users will encounter late records in case of some
> race
> >> > conditions between assigning new splits and completions of older.
&
ceberg users will encounter late records in case of some race
>> > conditions between assigning new splits and completions of older.
>> >
>> > Best,
>> > Piotrek
>> >
>> > [1]
>> > https://issues.apache.org/jira/browse/FLINK-21871?focusedCommentId=1
ader. So it seems useful to expose the
> > WatermarkAlignmentEvent information to the
> > SplitEnumerator as well.
>
> It seems like a valid potential use case. But do we have a good enough
> motivation to work on it right now?
>
> Piotrek
>
> czw., 5 maj 2
d potential use case. But do we have a good enough
motivation to work on it right now?
Piotrek
czw., 5 maj 2022 o 16:21 Steven Wu napisał(a):
> Piotr,
>
> With FLIP-27, Iceberg source already implemented alignment by tracking
> watermark and holding back split assignment when
Piotr,
With FLIP-27, Iceberg source already implemented alignment by tracking
watermark and holding back split assignment when necessary.
The purpose of this discussion is to see if Iceberg source can leverage
some of the watermark alignment work from Flink framework.
Thanks,
Steven
On Thu
like you also need to solve this problem,
>> > otherwise Iceberg users will encounter late records in case of some race
>> > conditions between assigning new splits and completions of older.
>> >
>> > Best,
>> > Piotrek
>> >
>> > [1]
>> >
>&g
gt; [1]
> >
> https://issues.apache.org/jira/browse/FLINK-21871?focusedCommentId=17495545=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17495545
> >
> > pon., 2 maj 2022 o 04:26 Steven Wu napisał(a):
> >
> >> add dev@ group t
;
>> Arvid,
>>
>> The scenario 3 (Dynamic assignment + temporary no split) in the FLIP-180
>> (idleness) can happen to Iceberg source alignment, as readers can be
>> temporarily starved due to the holdback by the enumerator when assigning
>> new splits upon
ted
>
> Arvid,
>
> The scenario 3 (Dynamic assignment + temporary no split) in the FLIP-180
> (idleness) can happen to Iceberg source alignment, as readers can be
> temporarily starved due to the holdback by the enumerator when assigning
> new splits upon request.
>
> Totally
add dev@ group to the thread as Thomas suggested
Arvid,
The scenario 3 (Dynamic assignment + temporary no split) in the FLIP-180
(idleness) can happen to Iceberg source alignment, as readers can be
temporarily starved due to the holdback by the enumerator when assigning
new splits upon request
14 matches
Mail list logo
