Thank you for your efforts!
I think that Qingsheng reported my concerns about discrepancies between event
time from the external system and current system time in the Flink cluster in a
more concrete fashion. Thank you for the catch!
The FLIP definitely makes sense and would love to see your
Thanks Qingsheng for the patient guidance! And sorry to everyone for
wasting your time.
I suddenly realised that my implementation is wrong.
In my implementation the FetchTime and EventTime do not come from the same
record and I am making this stupid mistake :( .
I need some time to think about
Hi Jialiang,
Thanks for the FLIP! Here're some thoughts of mine.
- For currentFetchEventTimeLag:
The problem of currentFetchEventTimeLag is: FetchTime is determined in
SplitReader driven by SplitFetcher thread, while EventTime is calculated at
the output of SourceOperator driven by task's main
Thanks and glad I saw your reply.
> I could not find the precise comment in the thread you are referring to
unfortunately. Do you have some specific point in the discussion in mind?
Sorry for the misdirection, are you worried about the time zone and the NTP
problem?
Here are some of my personal
Sorry for some mal-informed questions, and thanks for adding context here and
to the FLIP.
About 1:
> I think this has been discussed in the FLIP-33 lists thread[2].
I could not find the precise comment in the thread you are referring to
unfortunately. Do you have some specific point in the
Hi lorenzo, thanks for your feedback!
> There can be major discrepancies between the absolute time got by the
> TaskManagers (if clocks are not synchronized via ntp for example), and the
> results of the metric might be quite distant for different TMs.
I don't know much about this. can more
Hello jialiang tan, and thank you for your contribution!
Here are my questions:
1 - I don't know if exposing this as metric to the user could create some harm.
There can be major discrepancies between the absolute time got by the
TaskManagers (if clocks are not synchronized via ntp for
Sorry all, it seems bad formatting in my email message, now I send it again
gently and hope it work.
I would like to start a discussion about FLIP-XXX:
SupportcurrentFetchEventTimeLag and processingLag metrics [1].
The main motivation for this change was that I had some difficulties
