Re:Re: Re: Task scheduling of Flink

2021-01-11 Thread penguin.
Hi Lasantha, Thanks for your reply.I also found the method call chain you said. Next, I'm going to study these codes to see if I can modify them to schedule tasks to a slot in the specified taskManager. But I think it's a very difficult thing for me to read the source code, especially in the

[jira] [Created] (FLINK-20936) Improvements in custom partition extractor

2021-01-11 Thread zhoukui (Jira)
zhoukui created FLINK-20936: --- Summary: Improvements in custom partition extractor Key: FLINK-20936 URL: https://issues.apache.org/jira/browse/FLINK-20936 Project: Flink Issue Type: Improvement

[jira] [Created] (FLINK-20935) can't write flink configuration to tmp file and add it to local resource in yarn session mode

2021-01-11 Thread yuemeng (Jira)
yuemeng created FLINK-20935: --- Summary: can't write flink configuration to tmp file and add it to local resource in yarn session mode Key: FLINK-20935 URL: https://issues.apache.org/jira/browse/FLINK-20935

[jira] [Created] (FLINK-20934) CsvFileCompactionITCase.testSingleParallelism test failure

2021-01-11 Thread Piotr Nowojski (Jira)
Piotr Nowojski created FLINK-20934: -- Summary: CsvFileCompactionITCase.testSingleParallelism test failure Key: FLINK-20934 URL: https://issues.apache.org/jira/browse/FLINK-20934 Project: Flink

[jira] [Created] (FLINK-20933) Config Python Operator Use Managed Memory In Python DataStream

2021-01-11 Thread Huang Xingbo (Jira)
Huang Xingbo created FLINK-20933: Summary: Config Python Operator Use Managed Memory In Python DataStream Key: FLINK-20933 URL: https://issues.apache.org/jira/browse/FLINK-20933 Project: Flink

Re: [VOTE] FLIP-153: Support state access in Python DataStream API

2021-01-11 Thread Shuiqiang Chen
Hi everyone, The voting time for FLIP-153 is up. I'm going to finalize the vote. Up till now, there are 6 +1 votes, 4 of which are binding: - Dian Fu (binding) - Xingbo Huang (non-binding) - Wei Zhong (non-binding) - Yun Tang (binding) - Becket Qin (binding) - Yu Li (binding) And there is no

Re: Re: [DISCUSS] FLIP-147: Support Checkpoints After Tasks Finished

2021-01-11 Thread Yun Gao
Hi Roman, Very thanks for the feedbacks and suggestions! > I think UC will be the common case with multiple sources each with DoP > 1. > IIUC, waiting for EoP will be needed on each subtask each time one of it's source subtask finishes. Yes, waiting for

[jira] [Created] (FLINK-20932) sql client.blink streaming to make it checkpoint enabled [HOw |supported now?]

2021-01-11 Thread Norman He (Jira)
Norman He created FLINK-20932: - Summary: sql client.blink streaming to make it checkpoint enabled [HOw |supported now?] Key: FLINK-20932 URL: https://issues.apache.org/jira/browse/FLINK-20932 Project:

[jira] [Created] (FLINK-20931) Remove globalModVersion from ExecutionGraph

2021-01-11 Thread Zhu Zhu (Jira)
Zhu Zhu created FLINK-20931: --- Summary: Remove globalModVersion from ExecutionGraph Key: FLINK-20931 URL: https://issues.apache.org/jira/browse/FLINK-20931 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-20930) Remove AbstractExecutionSlotAllocator

2021-01-11 Thread Zhu Zhu (Jira)
Zhu Zhu created FLINK-20930: --- Summary: Remove AbstractExecutionSlotAllocator Key: FLINK-20930 URL: https://issues.apache.org/jira/browse/FLINK-20930 Project: Flink Issue Type: Sub-task

Re: Task manger isn’t initiating with defined values in Flink 1.11 version as part of EMR 6.1.0

2021-01-11 Thread Till Rohrmann
Hi Deep, Could you take a look at the logs of the TM which was lost? They might help us to debug the root cause. My guess would be that we are exceeding some memory threshold and that the kernel kills the process. Cheers, Till On Mon, Jan 11, 2021 at 1:46 PM DEEP NARAYAN Singh wrote: > Hi

[jira] [Created] (FLINK-20929) Add automated deployment of nightly docker images

2021-01-11 Thread Robert Metzger (Jira)
Robert Metzger created FLINK-20929: -- Summary: Add automated deployment of nightly docker images Key: FLINK-20929 URL: https://issues.apache.org/jira/browse/FLINK-20929 Project: Flink Issue

Re: Task manger isn’t initiating with defined values in Flink 1.11 version as part of EMR 6.1.0

2021-01-11 Thread DEEP NARAYAN Singh
Hi Till, Thanks for your support on task manager, in continuation to above email when we increased the TM and JM memory to run the job with increased parallelism but the job is getting failed in *one day* with below exception. Logs : *org.apache.flink.client.program.ProgramInvocationException:

Re: [VOTE] Release 1.12.1, release candidate #2

2021-01-11 Thread Xintong Song
Thanks Xingbo for bringing up the concern, and thanks for Arvid and Roman for the clarification. Thank you~ Xintong Song On Mon, Jan 11, 2021 at 7:34 PM Xingbo Huang wrote: > Thanks for Avrid and Roman's clarification. > > Best, > Xingbo > > Khachatryan Roman 于2021年1月11日周一 下午6:44写道: > > >

Re: [VOTE] Release 1.12.1, release candidate #2

2021-01-11 Thread Xingbo Huang
Thanks for Avrid and Roman's clarification. Best, Xingbo Khachatryan Roman 于2021年1月11日周一 下午6:44写道: > Hi, > I'm not sure whether it's > https://issues.apache.org/jira/browse/FLINK-20654 or > a new issue but I agree it shouldn't block 1.12.1. > > Regards, > Roman > > > On Mon, Jan 11, 2021 at

[jira] [Created] (FLINK-20928) KafkaSourceReaderTest.testOffsetCommitOnCheckpointComplete:189->pollUntil:270 » Timeout

2021-01-11 Thread Robert Metzger (Jira)
Robert Metzger created FLINK-20928: -- Summary: KafkaSourceReaderTest.testOffsetCommitOnCheckpointComplete:189->pollUntil:270 » Timeout Key: FLINK-20928 URL: https://issues.apache.org/jira/browse/FLINK-20928

Re: Re: [DISCUSS] FLIP-147: Support Checkpoints After Tasks Finished

2021-01-11 Thread Khachatryan Roman
Hi Yun, > b) With unaligned checkpoint enabled, the slower cases might happen if the downstream task processes very slowly. I think UC will be the common case with multiple sources each with DoP > 1. IIUC, waiting for EoP will be needed on each subtask each time one of it's source subtask

[jira] [Created] (FLINK-20927) Update configuration option in YarnConfigOptions class

2021-01-11 Thread Ruguo Yu (Jira)
Ruguo Yu created FLINK-20927: Summary: Update configuration option in YarnConfigOptions class Key: FLINK-20927 URL: https://issues.apache.org/jira/browse/FLINK-20927 Project: Flink Issue Type:

[jira] [Created] (FLINK-20926) Allow to read metadata for maxwell-json format

2021-01-11 Thread Q Kang (Jira)
Q Kang created FLINK-20926: -- Summary: Allow to read metadata for maxwell-json format Key: FLINK-20926 URL: https://issues.apache.org/jira/browse/FLINK-20926 Project: Flink Issue Type: Sub-task

Re: [VOTE] Release 1.12.1, release candidate #2

2021-01-11 Thread Khachatryan Roman
Hi, I'm not sure whether it's https://issues.apache.org/jira/browse/FLINK-20654 or a new issue but I agree it shouldn't block 1.12.1. Regards, Roman On Mon, Jan 11, 2021 at 10:30 AM Arvid Heise wrote: > Hi Xingbo, > > This ticket is actually about >

Re: Roadmap for Execution Mode (Batch/Streaming) and interaction with Table/SQL APIs

2021-01-11 Thread Aljoscha Krettek
Also cc'ing dev@flink.apache.org On 2021/01/06 09:19, burkaygur wrote: 1) How do these changes impact the Table and SQL APIs? Are they completely orthogonal or can we get the benefits of the new Batch Mode with Flink SQL as well? The answer here is a bit complicated. The Table API/SQL already

Re: Re: [DISCUSS] FLIP-147: Support Checkpoints After Tasks Finished

2021-01-11 Thread Yun Gao
Hi Roman, Very thanks for the feedbacks ! > Probably it would be simpler to just decline the RPC-triggered checkpoint > if not all inputs of this task are finished (with CHECKPOINT_DECLINED_TASK_NOT_READY). > But I wonder how significantly this waiting

Re: [VOTE] FLIP-153: Support state access in Python DataStream API

2021-01-11 Thread Yu Li
+1 (binding) Thanks for driving the FLIP! Best Regards, Yu On Mon, 11 Jan 2021 at 14:30, Becket Qin wrote: > +1 (binding) > > Thanks for driving the FLIP, Shuiqiang. > > Jiangjie (Becket) Qin > > On Wed, Jan 6, 2021 at 2:58 PM Yun Tang wrote: > > > +1 (binding) > > > > Best > > Yun Tang > >

[jira] [Created] (FLINK-20925) Separate the implementation of StreamExecLookup and BatchExecLookup

2021-01-11 Thread Wenlong Lyu (Jira)
Wenlong Lyu created FLINK-20925: --- Summary: Separate the implementation of StreamExecLookup and BatchExecLookup Key: FLINK-20925 URL: https://issues.apache.org/jira/browse/FLINK-20925 Project: Flink

Re: [VOTE] Release 1.12.1, release candidate #2

2021-01-11 Thread Arvid Heise
Hi Xingbo, This ticket is actually about https://issues.apache.org/jira/browse/FLINK-20654 where we added a major fix last week. At this point, we are not sure what additionally causes this issue and cannot give an estimate about the impact of the fix nor the time needed to fix it. It may also

[jira] [Created] (FLINK-20924) Port StreamExecPythonOverAggregate and BatchExecPythonOverAggregate to Java

2021-01-11 Thread godfrey he (Jira)
godfrey he created FLINK-20924: -- Summary: Port StreamExecPythonOverAggregate and BatchExecPythonOverAggregate to Java Key: FLINK-20924 URL: https://issues.apache.org/jira/browse/FLINK-20924 Project:

[jira] [Created] (FLINK-20923) insert hive failed using old planner

2021-01-11 Thread Spongebob (Jira)
Spongebob created FLINK-20923: - Summary: insert hive failed using old planner Key: FLINK-20923 URL: https://issues.apache.org/jira/browse/FLINK-20923 Project: Flink Issue Type: Bug