Re: Re: [ANNOUNCE] New PMC member: Xintong Song

2021-06-16 Thread Yun Gao
Congratulations, Xintong! Best, Yun -- Sender:Jingsong Li Date:2021/06/17 10:41:22 Recipient:dev Theme:Re: [ANNOUNCE] New PMC member: Xintong Song Congratulations, Xintong! Best, Jingsong On Thu, Jun 17, 2021 at 10:26 AM Yun

Re: Re: [ANNOUNCE] New PMC member: Arvid Heise

2021-06-16 Thread Yun Gao
Congratulations, Arvid! Best, Yun -- Sender:Jingsong Li Date:2021/06/17 10:41:29 Recipient:dev Theme:Re: [ANNOUNCE] New PMC member: Arvid Heise Congratulations, Arvid! Best, Jingsong On Thu, Jun 17, 2021 at 6:52 AM Matthias J.

Re: Re: [DISCUSS] FLIP-169: DataStream API for Fine-Grained Resource Requirements

2021-06-11 Thread Yun Gao
Hi, Very thanks @Yangze for bringing up this discuss. Overall +1 for exposing the fine-grained resource requirements in the DataStream API. One similar issue as Arvid has pointed out is that users may also creating different SlotSharingGroup objects, with different names but with different

Re: Re: [DISCUSS] FLIP-147: Support Checkpoints After Tasks Finished

2021-06-10 Thread Yun Gao
makes sense to add `flush`, as most of them > >>> > shouldn't buffer any data. Apart from Sinks, it's usually an operator > >>> that > >>> > is buffering the data (that holds true for AsyncFunction, > >>> ReduceFunction, > >>

[jira] [Created] (FLINK-22929) Change the default failover strategy to FixDelayRestartStrategy

2021-06-08 Thread Yun Gao (Jira)
Yun Gao created FLINK-22929: --- Summary: Change the default failover strategy to FixDelayRestartStrategy Key: FLINK-22929 URL: https://issues.apache.org/jira/browse/FLINK-22929 Project: Flink Issue

[jira] [Created] (FLINK-22928) Unexpected exception happens in RecordWriter when stopping-with-savepoint

2021-06-08 Thread Yun Gao (Jira)
Yun Gao created FLINK-22928: --- Summary: Unexpected exception happens in RecordWriter when stopping-with-savepoint Key: FLINK-22928 URL: https://issues.apache.org/jira/browse/FLINK-22928 Project: Flink

Re: Re: Add control mode for flink

2021-06-07 Thread Yun Gao
; his colleagues at Kuaishou maintain an internal version of Flink. One of their custom features is allowing dynamically changing operator behaviors via the REST APIs. He's willing to contribute this feature to the community, and came to Yun Gao and me for suggestions. After discussion, we feel

Re: [DISCUSS] FLIP-147: Support Checkpoints After Tasks Finished

2021-06-04 Thread Yun Gao
input. Very thanks for all the deep insights and discussions! Best, Yun -- From:Dawid Wysakowicz Send Time:2021 Jun. 3 (Thu.) 21:21 To:dev ; Till Rohrmann ; Yun Gao Cc:Piotr Nowojski ; Guowei Ma ; Stephan Ewen Subject:Re: [D

Re: Can we use ZK as HA for flink JM cluster(deployed in k8s standalone session mode)

2021-05-05 Thread Yun Gao
Hi, Logically using zk HA on k8s cluster is supported. Attaching files is not support by apache mail server, could you update the files into some file server and paste the connections ? Best, Yun -- Sender:bhagi@R

Re: Re: [ANNOUNCE] New Apache Flink Committer - Rui Li

2021-04-21 Thread Yun Gao
Congratulations Rui! Best, Yun -- Sender:Nicholas Jiang Date:2021/04/22 11:26:05 Recipient: Theme:Re: [ANNOUNCE] New Apache Flink Committer - Rui Li Congrats, Rui! Best, Nicholas Jiang -- Sent from:

[jira] [Created] (FLINK-22301) Statebackend and CheckpointStorage type is not shown in the Web UI

2021-04-16 Thread Yun Gao (Jira)
Yun Gao created FLINK-22301: --- Summary: Statebackend and CheckpointStorage type is not shown in the Web UI Key: FLINK-22301 URL: https://issues.apache.org/jira/browse/FLINK-22301 Project: Flink

[jira] [Created] (FLINK-22117) Not print stack trace for checkpoint trigger failure if not all tasks are started.

2021-04-06 Thread Yun Gao (Jira)
Yun Gao created FLINK-22117: --- Summary: Not print stack trace for checkpoint trigger failure if not all tasks are started. Key: FLINK-22117 URL: https://issues.apache.org/jira/browse/FLINK-22117 Project

Re: Zigzag shape in TM JVM used memory

2021-04-04 Thread Yun Gao
Hi Lu, The image seems not be able to shown due to the mail server limitation, could you upload it somewhere and paste the link here ? Logically, I think zigzag usually due to there are some small object get created and eliminated soon in the heap. Are you running a SQL job or a DataStream

[jira] [Created] (FLINK-22088) CheckpointCoordinator might not be able to abort triggering checkpoint if failover happens during triggering

2021-04-01 Thread Yun Gao (Jira)
Yun Gao created FLINK-22088: --- Summary: CheckpointCoordinator might not be able to abort triggering checkpoint if failover happens during triggering Key: FLINK-22088 URL: https://issues.apache.org/jira/browse/FLINK

[jira] [Created] (FLINK-22087) CheckpointPlanCalculator should abort checkpoint if tasks is in canceling/canceled and so on

2021-04-01 Thread Yun Gao (Jira)
Yun Gao created FLINK-22087: --- Summary: CheckpointPlanCalculator should abort checkpoint if tasks is in canceling/canceled and so on Key: FLINK-22087 URL: https://issues.apache.org/jira/browse/FLINK-22087

[jira] [Created] (FLINK-21967) Add document on the operation of blocking result partition

2021-03-24 Thread Yun Gao (Jira)
Yun Gao created FLINK-21967: --- Summary: Add document on the operation of blocking result partition Key: FLINK-21967 URL: https://issues.apache.org/jira/browse/FLINK-21967 Project: Flink Issue Type

[jira] [Created] (FLINK-21952) Make all the "Connection reset by peer" exception wrapped as RemoteTransportException

2021-03-24 Thread Yun Gao (Jira)
Yun Gao created FLINK-21952: --- Summary: Make all the "Connection reset by peer" exception wrapped as RemoteTransportException Key: FLINK-21952 URL: https://issues.apache.org/jira/browse/FLINK-21952

Re: Re: Re: [DISCUSS] FLIP-147: Support Checkpoints After Tasks Finished

2021-03-04 Thread Yun Gao
totally agree with that the two are separate and we could think on them distinctly. Regarding the order, I would still tend to we support the ordered case, since the sinks' implementation seem to depend on this functionality. Best, Yun ------

Re: Re: Re: [DISCUSS] FLIP-147: Support Checkpoints After Tasks Finished

2021-03-04 Thread Yun Gao
i Send Time:2021 Mar. 4 (Thu.) 17:16 To:Kezhu Wang Cc:dev ; Yun Gao ; jingsongl...@gmail.com ; Guowei Ma ; Till Rohrmann Subject:Re: Re: Re: [DISCUSS] FLIP-147: Support Checkpoints After Tasks Finished Hi Kezhu, What do you mean by “end-flushing”? I was suggesting to just keep `endOfInput()

Re: Re: Re: [DISCUSS] FLIP-147: Support Checkpoints After Tasks Finished

2021-02-28 Thread Yun Gao
the job finish again, it would re-emit the MAX_WATERMARK? Best, Yun -- From:Kezhu Wang Send Time:2021 Mar. 1 (Mon.) 01:26 To:Till Rohrmann Cc:Piotr Nowojski ; Guowei Ma ; dev ; Yun Gao ; jingsongl...@gmail.com Subject:

Re: Re: Re: [DISCUSS] FLIP-147: Support Checkpoints After Tasks Finished

2021-02-28 Thread Yun Gao
From:Kezhu Wang Send Time:2021 Mar. 1 (Mon.) 01:26 To:Till Rohrmann Cc:Piotr Nowojski ; Guowei Ma ; dev ; Yun Gao ; jingsongl...@gmail.com Subject:Re: Re: Re: [DISCUSS] FLIP-147: Support Checkpoints After Tasks Finished In “stop-with-savepoint —drain”, MAX_WATERMARK is not an issue.

Re: Re: Re: [DISCUSS] FLIP-147: Support Checkpoints After Tasks Finished

2021-02-25 Thread Yun Gao
hu Wang On February 24, 2021 at 23:47:36, Piotr Nowojski (piotr.nowoj...@gmail.com) wrote: Thanks for the reponses Guowei and Yun, Could you elaborate more/remind me, what does it mean to replace emitting results from the `notifyCheckpointComplete` with `OperatorCoordinator` approach

Re: Re: [DISCUSS] FLIP-147: Support Checkpoints After Tasks Finished

2021-02-24 Thread Yun Gao
`OperatorCoordinator` to avoid sending the element to the downstream > > operator. > > But I agree we could not limit the users not to emit records in the > > `notiyCheckpointComplete`. > > > > Best, > > Guowei > > > > > > On Tue

Re: Re: [ANNOUNCE] New Apache Flink Committers - Wei Zhong and Xingbo Huang

2021-02-22 Thread Yun Gao
Congratulations Wei and Xingbo! Best, Yun --Original Mail -- Sender:Roman Khachatryan Send Date:Tue Feb 23 00:59:22 2021 Recipients:dev Subject:Re: [ANNOUNCE] New Apache Flink Committers - Wei Zhong and Xingbo Huang Congratulations! Regards, Roman On Mon,

Re: [DISCUSS] FLIP-147: Support Checkpoints After Tasks Finished

2021-02-15 Thread Yun Gao
sh`, operators would be forbidden from emitting more records, so that > we can replace step 4. with this `flush` call, and then having a single > checkpoint to finish 2pc for all of the operators inside the chain, doesn't > work. Sheer fact of adding this `flush` method and changing the con

Re: Re: [ANNOUNCE] Welcome Roman Khachatryan a new Apache Flink Committer

2021-02-10 Thread Yun Gao
Congratulations Roman! Best, Yun --Original Mail -- Sender:Till Rohrmann Send Date:Wed Feb 10 20:53:21 2021 Recipients:dev CC:Khachatryan Roman , Roman Khachatryan Subject:Re: [ANNOUNCE] Welcome Roman Khachatryan a new Apache Flink Committer Congratulations

[jira] [Created] (FLINK-21246) Decline Checkpoint if some tasks finished before get triggered

2021-02-02 Thread Yun Gao (Jira)
Yun Gao created FLINK-21246: --- Summary: Decline Checkpoint if some tasks finished before get triggered Key: FLINK-21246 URL: https://issues.apache.org/jira/browse/FLINK-21246 Project: Flink Issue

[jira] [Created] (FLINK-21127) Stores finished status for fully finished operators

2021-01-25 Thread Yun Gao (Jira)
Yun Gao created FLINK-21127: --- Summary: Stores finished status for fully finished operators Key: FLINK-21127 URL: https://issues.apache.org/jira/browse/FLINK-21127 Project: Flink Issue Type: Sub

[jira] [Created] (FLINK-21088) CheckpointCoordinator pass the flag about whether a operator is fully finished on recovery

2021-01-21 Thread Yun Gao (Jira)
Yun Gao created FLINK-21088: --- Summary: CheckpointCoordinator pass the flag about whether a operator is fully finished on recovery Key: FLINK-21088 URL: https://issues.apache.org/jira/browse/FLINK-21088

[jira] [Created] (FLINK-21087) StreamTask waits for all the asynchronous step of pending checkpoint to finish

2021-01-21 Thread Yun Gao (Jira)
Yun Gao created FLINK-21087: --- Summary: StreamTask waits for all the asynchronous step of pending checkpoint to finish Key: FLINK-21087 URL: https://issues.apache.org/jira/browse/FLINK-21087 Project: Flink

[jira] [Created] (FLINK-21086) CheckpointBarrierHandler Insert barriers into channels received EndOfPartition

2021-01-21 Thread Yun Gao (Jira)
Yun Gao created FLINK-21086: --- Summary: CheckpointBarrierHandler Insert barriers into channels received EndOfPartition Key: FLINK-21086 URL: https://issues.apache.org/jira/browse/FLINK-21086 Project: Flink

[jira] [Created] (FLINK-21085) Refactor StreamTask hierarchy to support triggering checkpoint via RPC for non-source tasks

2021-01-21 Thread Yun Gao (Jira)
Yun Gao created FLINK-21085: --- Summary: Refactor StreamTask hierarchy to support triggering checkpoint via RPC for non-source tasks Key: FLINK-21085 URL: https://issues.apache.org/jira/browse/FLINK-21085

[jira] [Created] (FLINK-21084) Allows tasks to report operator subtask finish state

2021-01-21 Thread Yun Gao (Jira)
Yun Gao created FLINK-21084: --- Summary: Allows tasks to report operator subtask finish state Key: FLINK-21084 URL: https://issues.apache.org/jira/browse/FLINK-21084 Project: Flink Issue Type: Sub

[jira] [Created] (FLINK-21081) Re-compute tasks to trigger when tasks get triggered before finished

2021-01-21 Thread Yun Gao (Jira)
Yun Gao created FLINK-21081: --- Summary: Re-compute tasks to trigger when tasks get triggered before finished Key: FLINK-21081 URL: https://issues.apache.org/jira/browse/FLINK-21081 Project: Flink

[jira] [Created] (FLINK-21080) Identify JobVertex containing legacy source operators and abort checkpoint with legacy source operators partially finished

2021-01-21 Thread Yun Gao (Jira)
Yun Gao created FLINK-21080: --- Summary: Identify JobVertex containing legacy source operators and abort checkpoint with legacy source operators partially finished Key: FLINK-21080 URL: https://issues.apache.org/jira

[jira] [Created] (FLINK-21067) Modify the logic of computing tasks to trigger/wait/commit to consider finished tasks

2021-01-21 Thread Yun Gao (Jira)
Yun Gao created FLINK-21067: --- Summary: Modify the logic of computing tasks to trigger/wait/commit to consider finished tasks Key: FLINK-21067 URL: https://issues.apache.org/jira/browse/FLINK-21067 Project

[jira] [Created] (FLINK-21066) Refactor the CheckpointCoordinator to compute the tasks to trigger/wait/commit dynamically

2021-01-21 Thread Yun Gao (Jira)
Yun Gao created FLINK-21066: --- Summary: Refactor the CheckpointCoordinator to compute the tasks to trigger/wait/commit dynamically Key: FLINK-21066 URL: https://issues.apache.org/jira/browse/FLINK-21066

Re: Re: [VOTE] FLIP-147: Support Checkpoint After Tasks Finished

2021-01-20 Thread Yun Gao
(binding) Best, Arvid On Wed, Jan 20, 2021 at 12:13 PM Aljoscha Krettek wrote: > +1 (binding) > > Best, > Aljoscha > > On 2021/01/15 22:55, Yun Gao wrote: > > > >Hi all, > > > >I would like to start the vote for FLIP-147[1], which propose to sup

Re: Re: [ANNOUNCE] Welcome Guowei Ma as a new Apache Flink Committer

2021-01-19 Thread Yun Gao
Congratulations Guowei! Best, Yun-- Sender:Yangze Guo Date:2021/01/20 13:48:52 Recipient:dev Theme:Re: [ANNOUNCE] Welcome Guowei Ma as a new Apache Flink Committer Congratulations, Guowei! Well deserved. Best, Yangze Guo On Wed,

Re: Re: [DISCUSS] Dealing with deprecated and legacy code in Flink

2021-01-18 Thread Yun Gao
Hi, Very thanks for @Timo to initiate the discussion! I would also +1 for providing some informations to users via annotations or documents in advanced to not suprise users before we actually remove the legacy code. If we finally decide to change one functionality that user could sense,

[VOTE] FLIP-147: Support Checkpoint After Tasks Finished

2021-01-15 Thread Yun Gao
Hi all, I would like to start the vote for FLIP-147[1], which propose to support checkpoints after tasks finished and is discussed in [2]. The vote will last at least 72 hours (Jan 20th due to weekend), following the consensus voting process. thanks, Yun [1]

Re: [DISCUSS] FLIP-147: Support Checkpoints After Tasks Finished

2021-01-15 Thread Yun Gao
: [DISCUSS] FLIP-147: Support Checkpoints After Tasks Finished Thanks for the summary! I think we can now move towards a [VOTE] thread, right? On 2021/01/15 13:43, Yun Gao wrote: >1) For the problem that the "new" root task coincidently finished >before getting triggered succe

Re: Re: [DISCUSS] FLIP-147: Support Checkpoints After Tasks Finished

2021-01-14 Thread Yun Gao
Hi all, We have some offline discussion together with @Arvid, @Roman and @Aljoscha and I'd like to post some points we discussed: 1) For the problem that the "new" root task coincidently finished before getting triggered successfully, we have listed two options in the FLIP-147[1], for the

Re: Re: [DISCUSS] FLIP-147: Support Checkpoints After Tasks Finished

2021-01-13 Thread Yun Gao
Hi all, I updated the FLIP[1] to reflect the major discussed points in the ML thread: 1) For the "new" root tasks finished before it received trigger message, previously we proposed to let JM re-compute and re-trigger the descendant tasks, but after the discussion we realized that it might

Re: [ANNOUNCE] Welcome Danny Cranmer as a new Apache Flink Committer

2021-01-12 Thread Yun Gao
Congratulations, Danny! Best, Yun -- From:Xintong Song Send Time:2021 Jan. 13 (Wed.) 15:29 To:dev Subject:Re: [ANNOUNCE] Welcome Danny Cranmer as a new Apache Flink Committer Congratulations, Danny. Welcome aboard. Thank you~

Re: Re: [DISCUSS] FLIP-147: Support Checkpoints After Tasks Finished

2021-01-11 Thread Yun Gao
Hi Roman, Very thanks for the feedbacks and suggestions! > I think UC will be the common case with multiple sources each with DoP > 1. > IIUC, waiting for EoP will be needed on each subtask each time one of it's source subtask finishes. Yes, waiting for

Re: Re: [DISCUSS] FLIP-147: Support Checkpoints After Tasks Finished

2021-01-11 Thread Yun Gao
Hi Roman, Very thanks for the feedbacks ! > Probably it would be simpler to just decline the RPC-triggered checkpoint > if not all inputs of this task are finished (with CHECKPOINT_DECLINED_TASK_NOT_READY). > But I wonder how significantly this waiting

Re: Re: [DISCUSS] FLIP-147: Support Checkpoints After Tasks Finished

2021-01-07 Thread Yun Gao
Hi Roman, Very thanks for the feedbacks! I'll try to answer the issues inline: > 1. Option 1 is said to be not preferable because it wastes resources and adds > complexity (new event). > However, the resources would be wasted for a relatively short time until the > job finishes completely.

[jira] [Created] (FLINK-20888) ContinuousFileReaderOperator should not close the output on close()

2021-01-07 Thread Yun Gao (Jira)
Yun Gao created FLINK-20888: --- Summary: ContinuousFileReaderOperator should not close the output on close() Key: FLINK-20888 URL: https://issues.apache.org/jira/browse/FLINK-20888 Project: Flink

Re: Re: [DISCUSS] FLIP-147: Support Checkpoints After Tasks Finished

2021-01-06 Thread Yun Gao
. Best, Yun --Original Mail -- Sender:Arvid Heise Send Date:Thu Jan 7 00:52:27 2021 Recipients:Aljoscha Krettek CC:dev , Yun Gao Subject:Re: [DISCUSS] FLIP-147: Support Checkpoints After Tasks Finished Okay then at least you guys are in sync ;) (Although I'

Re: Re: [DISCUSS] FLIP-147: Support Checkpoints After Tasks Finished

2021-01-06 Thread Yun Gao
Hi Arvid, Very thanks for the feedbacks! I'll try to answer the questions inline: > I'm also concerned about the notion of a final checkpoint. What happens > when this final checkpoint times out (checkpoint timeout > async timeout) > or fails for a different reason? I'm currently more inclined

Re: [DISCUSS] FLIP-147: Support Checkpoints After Tasks Finished

2021-01-06 Thread Yun Gao
xamples if you go beyond chained operators and fully connected exchanges. Think of any fan-in, let's assume you have source S1...S4, with S1+S2->M1, and S3+S4->M2. If S1 is finished, S2 and M1 is still running. Or I didn't get your question ;). On Tue, Jan 5, 2021 at 5:00 PM Yun G

Re: [DISCUSS] FLIP-147: Support Checkpoints After Tasks Finished

2021-01-05 Thread Yun Gao
Hi Aljoscha, Very thanks for the feedbacks! For the second issue, I'm indeed thinking the race condition between deciding to trigger and operator get finished. And for this point, > One thought here is this: will there ever be intermediate operators that > should be

Re: [DISCUSS] FLIP-147: Support Checkpoints After Tasks Finished

2021-01-05 Thread Yun Gao
rained on OperatorSubtaskState. Maybe we can store the flag inside managed or raw state without changing the format? On Fri, Dec 25, 2020 at 8:39 AM Yun Gao wrote: Hi all, I tested the previous PoC with the current tests and I found some new issues that might cause divergence, and

Re: [DISCUSS] FLIP-147: Support Checkpoints After Tasks Finished

2020-12-24 Thread Yun Gao
he thorough update! I'll answer inline. On 14.12.20 16:33, Yun Gao wrote: > 1. To include EndOfPartition into consideration for barrier alignment at > the TM side, we now tend to decouple the logic for EndOfPartition with the > normal alignment behaviors to avoid the complex interference (whic

Re: [DISCUSS] FLIP-147: Support Checkpoints After Tasks Finished

2020-12-15 Thread Yun Gao
d Time:2020 Dec. 15 (Tue.) 18:11 To:dev Subject:Re: [DISCUSS] FLIP-147: Support Checkpoints After Tasks Finished Thanks for the thorough update! I'll answer inline. On 14.12.20 16:33, Yun Gao wrote: > 1. To include EndOfPartition into consideration for barrier alignment at > t

Re: [DISCUSS] FLIP-147: Support Checkpoints After Tasks Finished

2020-12-14 Thread Yun Gao
Hi all, I would like to resume this discussion for supporting checkpoints after tasks Finished :) Based on the previous discussion, we now implement a version of PoC [1] to try the idea. During the PoC we also met with some possible issues: 1. To include EndOfPartition into

[jira] [Created] (FLINK-20315) Optimize the compaction section for the document of table filesystem sink.

2020-11-23 Thread Yun Gao (Jira)
Yun Gao created FLINK-20315: --- Summary: Optimize the compaction section for the document of table filesystem sink. Key: FLINK-20315 URL: https://issues.apache.org/jira/browse/FLINK-20315 Project: Flink

[jira] [Created] (FLINK-20295) File Source lost data when reading from directories created by FileSystemTableSink with JSON format

2020-11-23 Thread Yun Gao (Jira)
Yun Gao created FLINK-20295: --- Summary: File Source lost data when reading from directories created by FileSystemTableSink with JSON format Key: FLINK-20295 URL: https://issues.apache.org/jira/browse/FLINK-20295

[jira] [Created] (FLINK-20292) Avro format conflicts with Parquet and Orc

2020-11-23 Thread Yun Gao (Jira)
Yun Gao created FLINK-20292: --- Summary: Avro format conflicts with Parquet and Orc Key: FLINK-20292 URL: https://issues.apache.org/jira/browse/FLINK-20292 Project: Flink Issue Type: Bug

[jira] [Created] (FLINK-20291) Optimize the exception message of FileSystemTableSink when missing format dependencies

2020-11-23 Thread Yun Gao (Jira)
Yun Gao created FLINK-20291: --- Summary: Optimize the exception message of FileSystemTableSink when missing format dependencies Key: FLINK-20291 URL: https://issues.apache.org/jira/browse/FLINK-20291 Project

[jira] [Created] (FLINK-20180) Translation the FileSink Document into Chinese

2020-11-16 Thread Yun Gao (Jira)
Yun Gao created FLINK-20180: --- Summary: Translation the FileSink Document into Chinese Key: FLINK-20180 URL: https://issues.apache.org/jira/browse/FLINK-20180 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-20172) Unify the behavior for file committers when the source or dest files are not exists

2020-11-16 Thread Yun Gao (Jira)
Yun Gao created FLINK-20172: --- Summary: Unify the behavior for file committers when the source or dest files are not exists Key: FLINK-20172 URL: https://issues.apache.org/jira/browse/FLINK-20172 Project

[jira] [Created] (FLINK-20032) Make FileSink support processing time

2020-11-06 Thread Yun Gao (Jira)
Yun Gao created FLINK-20032: --- Summary: Make FileSink support processing time Key: FLINK-20032 URL: https://issues.apache.org/jira/browse/FLINK-20032 Project: Flink Issue Type: Sub-task

Re: Re: [ANNOUNCE] New Apache Flink Committer - Congxian Qiu

2020-10-29 Thread Yun Gao
Congratulations Congxian ! Best, Yun -- Sender:Leonard Xu Date:2020/10/29 21:19:56 Recipient:dev Theme:Re: [ANNOUNCE] New Apache Flink Committer - Congxian Qiu Congratulations! Congxian Best, Leonard > 在 2020年10月29日,20:55,Stephan

[jira] [Created] (FLINK-19850) Add E2E tests for the new streaming case of the new FileSink

2020-10-28 Thread Yun Gao (Jira)
Yun Gao created FLINK-19850: --- Summary: Add E2E tests for the new streaming case of the new FileSink Key: FLINK-19850 URL: https://issues.apache.org/jira/browse/FLINK-19850 Project: Flink Issue

[jira] [Created] (FLINK-19758) Implement a new unified File Sink based on the new Sink API

2020-10-21 Thread Yun Gao (Jira)
Yun Gao created FLINK-19758: --- Summary: Implement a new unified File Sink based on the new Sink API Key: FLINK-19758 URL: https://issues.apache.org/jira/browse/FLINK-19758 Project: Flink Issue

[jira] [Created] (FLINK-19701) Unaligned Checkpoint might misuse the number of buffers to persist from the previous barrier

2020-10-18 Thread Yun Gao (Jira)
Yun Gao created FLINK-19701: --- Summary: Unaligned Checkpoint might misuse the number of buffers to persist from the previous barrier Key: FLINK-19701 URL: https://issues.apache.org/jira/browse/FLINK-19701

[jira] [Created] (FLINK-19653) HiveCatalogITCase fails on azure

2020-10-14 Thread Yun Gao (Jira)
Yun Gao created FLINK-19653: --- Summary: HiveCatalogITCase fails on azure Key: FLINK-19653 URL: https://issues.apache.org/jira/browse/FLINK-19653 Project: Flink Issue Type: Improvement

Re: [DISCUSS] FLIP-147: Support Checkpoints After Tasks Finished

2020-10-14 Thread Yun Gao
nd Time:2020 Oct. 13 (Tue.) 17:25 To:Yun Gao Cc:Arvid Heise ; Flink Dev ; User-Flink Subject:Re: [DISCUSS] FLIP-147: Support Checkpoints After Tasks Finished Thanks for starting this discussion Yun Gao, I have three comments/questions: 1) When restarting all tasks independent of the status at ch

Re: [DISCUSS] FLIP-147: Support Checkpoints After Tasks Finished

2020-10-13 Thread Yun Gao
ore the problem on how to store and restore output buffers of a completed task (also important for the next point). 5) I think we are on the same page and I completely agree that for the MVP/first version, it's completely fine to start and immediately stop. A tad better would be even to

Re: [DISCUSS] FLIP-147: Support Checkpoints After Tasks Finished

2020-10-12 Thread Yun Gao
first time that we recover an incomplete DAG. Afaik the subtasks are deployed before the state is recovered, so at some point, the subtasks either need to be removed again or maybe we could even avoid them being created in the first place. [1] https://issues.apache.org/jira/browse/FLINK-

Re: [DISCUSS] FLIP-147: Support Checkpoints After Tasks Finished

2020-10-09 Thread Yun Gao
Hi, devs & users Very sorry for the spoiled formats, I resent the discussion as follows. As discussed in FLIP-131[1], Flink will make DataStream the unified API for processing bounded and unbounded data in both streaming and blocking modes. However, one long-standing problem for the streaming

[DISCUSS] FLIP-147: Support Checkpoints After Tasks Finished

2020-10-09 Thread Yun Gao
Hi, devs & users As discussed in FLIP-131 [1], Flink will make DataStream the unified API for processing bounded and unbounded data in both streaming and blocking modes. However, one long-standing problem for the streaming mode is that currently Flink does not support checkpoints after some

Re: Re: [ANNOUNCE] New PMC member: Zhu Zhu

2020-10-05 Thread Yun Gao
Congratulations Zhu! Best, Yun-- Sender:Arvid Heise Date:2020/10/05 15:53:01 Recipient:dev Cc: Theme:Re: [ANNOUNCE] New PMC member: Zhu Zhu Congratulations Zhu Zhu! On Mon, Oct 5, 2020 at 9:28 AM Till Rohrmann wrote: > Congrats

Re: Re: [ANNOUNCE] New Apache Flink Committer - Godfrey He

2020-09-18 Thread Yun Gao
Congratulations Godfrey! Best, Yun --Original Mail -- Sender:Dawid Wysakowicz Send Date:Thu Sep 17 14:45:55 2020 Recipients:Flink Dev , 贺小令 Subject:Re: [ANNOUNCE] New Apache Flink Committer - Godfrey He Congratulations! On 16/09/2020 06:19, Jark Wu wrote: >

Re: Re: [ANNOUNCE] Apache Flink 1.11.2 released

2020-09-18 Thread Yun Gao
Great! Very thanks @ZhuZhu for driving this and thanks for all contributed to the release! Best, Yun --Original Mail -- Sender:Jingsong Li Send Date:Thu Sep 17 13:31:41 2020 Recipients:user-zh CC:dev , user , Apache Announce List Subject:Re: [ANNOUNCE]

Re: Re: [VOTE] FLIP-134: Batch execution for the DataStream API

2020-09-16 Thread Yun Gao
+1 (non-binding) Very thanks for bring this up! And the FLIP is indeed necessary for stream & batch unification. -- Sender:Dawid Wysakowicz Date:2020/09/16 15:01:08 Recipient:; Aljoscha Krettek Theme:Re: [VOTE] FLIP-134: Batch

Re: Re: [ANNOUNCE] New Apache Flink Committer - Igal Shilman

2020-09-15 Thread Yun Gao
Congratulations Igal! Best, Yun -- Sender:Stephan Ewen Date:2020/09/15 22:48:30 Recipient:dev Theme:Re: [ANNOUNCE] New Apache Flink Committer - Igal Shilman Welcome, Igal! On Tue, Sep 15, 2020 at 3:18 PM Seth Wiesman wrote:

Re: Re: [ANNOUNCE] New Apache Flink Committer - Yun Tang

2020-09-15 Thread Yun Gao
Congratulations Yun Tang! Best, Yun Gao -- Sender:Zhu Zhu Date:2020/09/15 18:49:24 Recipient:dev Cc:; Yun Tang Theme:Re: [ANNOUNCE] New Apache Flink Committer - Yun Tang Congratulations! Thanks, Zhu Yu Li 于2020年9月15日周二 下午6:19写

Re: Re: [VOTE] FLIP-140: Introduce bounded style execution for keyed streams

2020-09-15 Thread Yun Gao
Very thanks for bring this up! +1 (non-binding) Best, Yun -- Sender:Seth Wiesman Date:2020/09/14 21:56:55 Recipient:dev Theme:Re: [VOTE] FLIP-140: Introduce bounded style execution for keyed streams +1 (binding) Seth On Thu,

Re: Re: [ANNOUNCE] New Apache Flink Committer - Niels Basjes

2020-09-15 Thread Yun Gao
Congratulations Niels! Best, Yun -- Sender:Congxian Qiu Date:2020/09/15 13:33:31 Recipient:dev@flink.apache.org Theme:Re: [ANNOUNCE] New Apache Flink Committer - Niels Basjes Congratulations Best, Congxian Yang Wang

Re: Re: [ANNOUNCE] New Apache Flink Committer - Arvid Heise

2020-09-15 Thread Yun Gao
Congratulations Arvid ! Best, Yun --Original Mail -- Sender:Yangze Guo Send Date:Tue Sep 15 13:39:56 2020 Recipients:dev Subject:Re: [ANNOUNCE] New Apache Flink Committer - Arvid Heise Congrats! Arvid. Best, Yangze Guo On Tue, Sep 15, 2020 at 1:31 PM

Re: [VOTE] FLIP-131: Consolidate the user-facing Dataflow SDKs/APIs (and deprecate the DataSet API)

2020-09-03 Thread Yun Gao
Very thanks for bring this up! +1 for deprecating the DataSet API and providing a unified streaming/batch programming model to users. Best, Yun -- Sender:Aljoscha Krettek Date:2020/09/02 19:22:51 Recipient:Flink Dev

Re: Re: [ANNOUNCE] New PMC member: Dian Fu

2020-08-27 Thread Yun Gao
Congratulations Dian ! Best Yun -- Sender:Marta Paes Moreira Date:2020/08/27 17:42:34 Recipient:Yuan Mei Cc:Xingbo Huang; jincheng sun; dev; Dian Fu; user; user-zh Theme:Re: [ANNOUNCE] New PMC member: Dian Fu Congrats, Dian!

Re: SDK vs Connectors

2020-08-22 Thread Yun Gao
Hi Prasanna, 1) Semantically both a) and b) would be Ok. If the Custom sink could be chained with the map operator (I assume the map operator is the "Processing" in the graph), there should be also no much difference physically, if they could not chain, then writting a custom sink would

Re: Re: [DISCUSS] FLIP-134: DataStream Semantics for Bounded Input

2020-08-17 Thread Yun Gao
Hi, Very thanks for bringing up this discussion! One more question is that does the BATCH and STREAMING mode also decides the shuffle types and operators? I'm asking so because that even for blocking mode, it should also benefit from keeping some edges to be pipeline if the resources

Re: Re: [DISCUSS] Removing deprecated methods from DataStream API

2020-08-17 Thread Yun Gao
+1 for removing the methods that are deprecated for a while & have alternative methods. One specific thing is that if we remove the DataStream#split, do we consider enabling side-output in more operators in the future ? Currently it should be only available in ProcessFunctions, but not

Re: Re: [ANNOUNCE] New PMC member: Piotr Nowojski

2020-07-06 Thread Yun Gao
Congratulations Piotr ! Best, Yun -- Sender:Austin Bennett Date:2020/07/07 10:06:15 Recipient: Theme:Re: [ANNOUNCE] New PMC member: Piotr Nowojski Thanks for what you do, Piotr! On Mon, Jul 6, 2020, 7:02 PM Matt Wang wrote: >

Re: adding s3 object metadata while using StreamFileSink

2020-06-21 Thread Yun Gao
Hi Dhurandar, With my understand I think what you need is to get notified when a file is written successfully (committed) on the S3 FileSystem. However, currently there is no public API for the listener and there an issue tracking it [1]. With the current version, one possible method

Re: Re: [ANNOUNCE] Yu Li is now part of the Flink PMC

2020-06-16 Thread Yun Gao
Congratulations Yu! Best, Yun -- Sender:Zhijiang Date:2020/06/17 11:18:35 Recipient:Dian Fu; dev Cc:Haibo Sun; user; user-zh Theme:Re: [ANNOUNCE] Yu Li is now part of the Flink PMC Congratulations Yu! Well deserved! Best,

Re: Re: [DISCUSS] Update our EditorConfig file

2020-06-13 Thread Yun Gao
+1 for have a editorconfig file that is more consistent with Flink's code sytle. From my own experience, whenever I open the Flink repo for the first time, I have to manually change the continuation indent to 4 and disable the wildcard imports. I also think that it should be more nice to new

[jira] [Created] (FLINK-18269) ContinuousFileReaderOperator also supports monitoring subdirectories

2020-06-11 Thread Yun Gao (Jira)
Yun Gao created FLINK-18269: --- Summary: ContinuousFileReaderOperator also supports monitoring subdirectories Key: FLINK-18269 URL: https://issues.apache.org/jira/browse/FLINK-18269 Project: Flink

Re: Re: [ANNOUNCE] New Flink Committer: Benchao Li

2020-06-10 Thread Yun Gao
Congratulations, Benchao! Best, Yun -- Sender:Danny Chan Date:2020/06/10 20:01:01 Recipient: Theme:Re: [ANNOUNCE] New Flink Committer: Benchao Li Congrats Benchao! Best, Danny Chan 在 2020年6月10日 +0800

Re: [ANNOUNCE] New Apache Flink Committer - Xintong Song

2020-06-04 Thread Yun Gao
Congratulates Xintong ! Best Yun -- Sender:Leonard Xu Date:2020/06/05 12:51:20 Recipient:dev Cc:Xintong Song Theme:Re: [ANNOUNCE] New Apache Flink Committer - Xintong Song Congratulates Xintong ! Best, Leonard Xu > 在

[jira] [Created] (FLINK-18110) Bucket Listener in StreamingFileSink should notify for buckets detected to be inactive at recovery

2020-06-03 Thread Yun Gao (Jira)
Yun Gao created FLINK-18110: --- Summary: Bucket Listener in StreamingFileSink should notify for buckets detected to be inactive at recovery Key: FLINK-18110 URL: https://issues.apache.org/jira/browse/FLINK-18110

[jira] [Created] (FLINK-18089) Add the zero-copy test into the azure E2E pipeline

2020-06-03 Thread Yun Gao (Jira)
Yun Gao created FLINK-18089: --- Summary: Add the zero-copy test into the azure E2E pipeline Key: FLINK-18089 URL: https://issues.apache.org/jira/browse/FLINK-18089 Project: Flink Issue Type: Test

[jira] [Created] (FLINK-18056) Hive file sink throws exception when the target in-progress file exists.

2020-06-01 Thread Yun Gao (Jira)
Yun Gao created FLINK-18056: --- Summary: Hive file sink throws exception when the target in-progress file exists. Key: FLINK-18056 URL: https://issues.apache.org/jira/browse/FLINK-18056 Project: Flink

[jira] [Created] (FLINK-17872) Update StreamingFileSink documents to add avro formats

2020-05-21 Thread Yun Gao (Jira)
Yun Gao created FLINK-17872: --- Summary: Update StreamingFileSink documents to add avro formats Key: FLINK-17872 URL: https://issues.apache.org/jira/browse/FLINK-17872 Project: Flink Issue Type

[jira] [Created] (FLINK-17711) Support hadoop OutputCommitter based file committer

2020-05-14 Thread Yun Gao (Jira)
Yun Gao created FLINK-17711: --- Summary: Support hadoop OutputCommitter based file committer Key: FLINK-17711 URL: https://issues.apache.org/jira/browse/FLINK-17711 Project: Flink Issue Type: Sub

<    1   2   3   4   5   6   >