[jira] [Created] (FLINK-19891) ScalarOperatorGens should generate a not-null type for IS NULL and IS NOT NULL

2020-10-29 Thread Rui Li (Jira)
Rui Li created FLINK-19891: -- Summary: ScalarOperatorGens should generate a not-null type for IS NULL and IS NOT NULL Key: FLINK-19891 URL: https://issues.apache.org/jira/browse/FLINK-19891 Project: Flink

[jira] [Created] (FLINK-19890) Introduce LimitableBulkFormat to wrap BulkFormat when limit pushed down

2020-10-29 Thread Jingsong Lee (Jira)
Jingsong Lee created FLINK-19890: Summary: Introduce LimitableBulkFormat to wrap BulkFormat when limit pushed down Key: FLINK-19890 URL: https://issues.apache.org/jira/browse/FLINK-19890 Project:

[jira] [Created] (FLINK-19889) Supports nested projection pushdown for filesystem connector of columnar format

2020-10-29 Thread Jark Wu (Jira)
Jark Wu created FLINK-19889: --- Summary: Supports nested projection pushdown for filesystem connector of columnar format Key: FLINK-19889 URL: https://issues.apache.org/jira/browse/FLINK-19889 Project: Flink

Re: [DISCUSS] 1.12 feature freeze date += one week ?

2020-10-29 Thread Yu Li
+1 for extending the feature freeze, it would be helpful for more thorough testing of the frocksdb upgrade work. Thanks for the consideration and starting the discussion Stephan! Best Regards, Yu On Thu, 29 Oct 2020 at 16:21, Xintong Song wrote: > +1 for one week extension. > > The native

[jira] [Created] (FLINK-19888) Migrate Hive source to FLIP-27 source interface for streaming

2020-10-29 Thread Rui Li (Jira)
Rui Li created FLINK-19888: -- Summary: Migrate Hive source to FLIP-27 source interface for streaming Key: FLINK-19888 URL: https://issues.apache.org/jira/browse/FLINK-19888 Project: Flink Issue

Re: [ANNOUNCE] New Apache Flink Committer - Congxian Qiu

2020-10-29 Thread Zou Dan
Congratulatioins Congxian! Best, Dan Zou > 2020年10月29日 下午9:20,Yun Gao 写道: > > Congratulations Congxian ! > > Best, > Yun > -- > Sender:Leonard Xu > Date:2020/10/29 21:19:56 > Recipient:dev > Theme:Re: [ANNOUNCE] New Apache Flink

[jira] [Created] (FLINK-19887) flink-1.11.2 - 使用包对象下的样例类,会导致table api程序编译失败

2020-10-29 Thread Jira
谢波 created FLINK-19887: -- Summary: flink-1.11.2 - 使用包对象下的样例类,会导致table api程序编译失败 Key: FLINK-19887 URL: https://issues.apache.org/jira/browse/FLINK-19887 Project: Flink Issue Type: Bug

[jira] [Created] (FLINK-19886) Integrate file compaction to Hive connector

2020-10-29 Thread Jingsong Lee (Jira)
Jingsong Lee created FLINK-19886: Summary: Integrate file compaction to Hive connector Key: FLINK-19886 URL: https://issues.apache.org/jira/browse/FLINK-19886 Project: Flink Issue Type:

Re: [DISCUSS][docker] Adopt Jemalloc as default memory allocator for debian based Flink docker image

2020-10-29 Thread Xie Billy
After long time running test application(>96 hours) , glibc used less memory than jemalloc , almost half of jemalloc. Can refer this test case. Best Regards! Billy xie(谢志民) On Fri, Oct 30, 2020 at 12:38 AM 王刚 wrote: > we also met glic memory leak .it appeared many 64 MB memory segment when

Question about processing a 3-level List data type in parquet

2020-10-29 Thread Naehee Kim
Hi Flink Dev Community, I've found RowConverter.java in flink-parquet module doesn't support reading a 3-level list type in parquet though it is able to process a 2-level list type. 3-level optional group my_list (LIST) { repeated group element { required binary str (UTF8); }; }

Re: Is there any way to change operator level parallelism dynamically?

2020-10-29 Thread DEEP NARAYAN Singh
Hi Till, Got it.thanks for the prompt reply. -Deep On Thu, 29 Oct, 2020, 1:22 PM Till Rohrmann, wrote: > Hi Deep, > > at the moment the only way to change the parallelism is to take a > savepoint, stop the job and then resume the job with a changed parallelism > from the savepoint. The

回复: [DISCUSS][docker] Adopt Jemalloc as default memory allocator for debian based Flink docker image

2020-10-29 Thread 王刚
we also met glic memory leak .it appeared many 64 MB memory segment when using pmap command . i am +1 on setting jemalloc as default 发送自autohome 发件人: Yu Li 发送时间: 2020-10-30 00:15:12 收件人: dev 抄送: n...@ververica.com 主题: Re: [DISCUSS][docker] Adopt Jemalloc as

Re: [DISCUSS][docker] Adopt Jemalloc as default memory allocator for debian based Flink docker image

2020-10-29 Thread Yu Li
Thanks for sharing more thoughts guys. My experience with malloc libraries are also limited, and as mentioned in my previous email [1], my major concern comes from some online information [2] which indicates in some cases jemalloc will consume as much as twice the memory than glibc, which is a

Re: Re: [ANNOUNCE] New Apache Flink Committer - Congxian Qiu

2020-10-29 Thread Yun Gao
Congratulations Congxian ! Best, Yun -- Sender:Leonard Xu Date:2020/10/29 21:19:56 Recipient:dev Theme:Re: [ANNOUNCE] New Apache Flink Committer - Congxian Qiu Congratulations! Congxian Best, Leonard > 在 2020年10月29日,20:55,Stephan

Re: [ANNOUNCE] New Apache Flink Committer - Congxian Qiu

2020-10-29 Thread Leonard Xu
Congratulations! Congxian Best, Leonard > 在 2020年10月29日,20:55,Stephan Ewen 写道: > > Congrats, Congxian! > > On Thu, Oct 29, 2020 at 12:21 PM Shawn Huang wrote: > >> Congratulations! >> >> Best, >> Shawn Huang >> >> >> hailongwang <18868816...@163.com> 于2020年10月29日周四 下午7:11写道: >> >>>

[jira] [Created] (FLINK-19885) Extend the JobManager's web UI in accordance to FLIP-104

2020-10-29 Thread Matthias (Jira)
Matthias created FLINK-19885: Summary: Extend the JobManager's web UI in accordance to FLIP-104 Key: FLINK-19885 URL: https://issues.apache.org/jira/browse/FLINK-19885 Project: Flink Issue Type:

Re: [DISCUSS][docker] Adopt Jemalloc as default memory allocator for debian based Flink docker image

2020-10-29 Thread Stephan Ewen
I think Till has a good point. The default should work well, because very few users will ever change it. Even on OOMs. So I am slightly in favor of going with jemalloc. @Yun Tang - Do you see a noticeable performance difference between the two? On Thu, Oct 29, 2020 at 12:32 PM Yun Tang wrote:

Re: Re: [ANNOUNCE] New Apache Flink Committer - Congxian Qiu

2020-10-29 Thread Stephan Ewen
Congrats, Congxian! On Thu, Oct 29, 2020 at 12:21 PM Shawn Huang wrote: > Congratulations! > > Best, > Shawn Huang > > > hailongwang <18868816...@163.com> 于2020年10月29日周四 下午7:11写道: > > > Congratulations, Congxian! > > > > Best, > > Hailong Wang > > At 2020-10-29 16:45:19, "Yun Tang" wrote: > >

Re: [VOTE] Remove flink-connector-filesystem module.

2020-10-29 Thread Stephan Ewen
+1 On Thu, Oct 29, 2020 at 1:29 PM Dawid Wysakowicz wrote: > +1 > > On 29/10/2020 09:18, Kostas Kloudas wrote: > > Hi all, > > > > Following the discussion in [1], I would like to start a vote on > > removing the flink-connector-filesystem module which includes the > > BucketingSink. > > > >

Re: [VOTE] Remove flink-connector-filesystem module.

2020-10-29 Thread Dawid Wysakowicz
+1 On 29/10/2020 09:18, Kostas Kloudas wrote: > Hi all, > > Following the discussion in [1], I would like to start a vote on > removing the flink-connector-filesystem module which includes the > BucketingSink. > > The vote will be open till November 3rd (72h, excluding the weekend) > unless there

Re: [VOTE] Remove flink-connector-filesystem module.

2020-10-29 Thread Jingsong Li
+1 On Thu, Oct 29, 2020 at 7:59 PM Chesnay Schepler wrote: > +1 > > On 10/29/2020 9:18 AM, Kostas Kloudas wrote: > > Hi all, > > > > Following the discussion in [1], I would like to start a vote on > > removing the flink-connector-filesystem module which includes the > > BucketingSink. > > > >

Re: [VOTE] Remove flink-connector-filesystem module.

2020-10-29 Thread Chesnay Schepler
+1 On 10/29/2020 9:18 AM, Kostas Kloudas wrote: Hi all, Following the discussion in [1], I would like to start a vote on removing the flink-connector-filesystem module which includes the BucketingSink. The vote will be open till November 3rd (72h, excluding the weekend) unless there is an

Re: [DISCUSS][docker] Adopt Jemalloc as default memory allocator for debian based Flink docker image

2020-10-29 Thread Yun Tang
Hi I think Till's view deserves wider discussion. I want to give my two cents when I debug with Nico on his reported RocksDB OOM problem. Jemalloc has the mechanism to profile memory allocation [1] which is widely used to analysis memory leak. Once we set jemalloc as default memory allocator,

Re: Re: [ANNOUNCE] New Apache Flink Committer - Congxian Qiu

2020-10-29 Thread Shawn Huang
Congratulations! Best, Shawn Huang hailongwang <18868816...@163.com> 于2020年10月29日周四 下午7:11写道: > Congratulations, Congxian! > > Best, > Hailong Wang > At 2020-10-29 16:45:19, "Yun Tang" wrote: > >Congratulations, Congxian! > > > >Best > >Yun Tang > > > >From:

Re:Re: [ANNOUNCE] New Apache Flink Committer - Congxian Qiu

2020-10-29 Thread hailongwang
Congratulations, Congxian! Best, Hailong Wang At 2020-10-29 16:45:19, "Yun Tang" wrote: >Congratulations, Congxian! > >Best >Yun Tang > >From: Xintong Song >Sent: Thursday, October 29, 2020 16:16 >To: dev >Cc: Congxian Qiu ; klio...@apache.org > >Subject: Re:

Re: [DISCUSS] Releasing Apache Flink 1.11.3

2020-10-29 Thread Xintong Song
Hi folks, Just to provide some updates concerning the status on the test instabilities. Currently, we have 30 unresolved tickets labeled with `Affects Version` 1.11.x.

Re: [DISCUSS][docker] Adopt Jemalloc as default memory allocator for debian based Flink docker image

2020-10-29 Thread Till Rohrmann
Hi Yu, I see your point why you are in favour of continuing using glibc. I honestly don't have a lot of experience with malloc libraries so I can only argue from a general perspective: We know that glibc has some problems wrt to memory fragmentation which can cause processes to exceed its memory

[jira] [Created] (FLINK-19883) Support "IF EXISTS" in DDL for ALTER TABLE

2020-10-29 Thread Jira
Ingo Bürk created FLINK-19883: - Summary: Support "IF EXISTS" in DDL for ALTER TABLE Key: FLINK-19883 URL: https://issues.apache.org/jira/browse/FLINK-19883 Project: Flink Issue Type: Improvement

[jira] [Created] (FLINK-19882) E2E: SQLClientHBaseITCase crash

2020-10-29 Thread Jingsong Lee (Jira)
Jingsong Lee created FLINK-19882: Summary: E2E: SQLClientHBaseITCase crash Key: FLINK-19882 URL: https://issues.apache.org/jira/browse/FLINK-19882 Project: Flink Issue Type: Bug

[jira] [Created] (FLINK-19881) Optimize temporal join with upsert-Source(upsert-kafka)

2020-10-29 Thread Leonard Xu (Jira)
Leonard Xu created FLINK-19881: -- Summary: Optimize temporal join with upsert-Source(upsert-kafka) Key: FLINK-19881 URL: https://issues.apache.org/jira/browse/FLINK-19881 Project: Flink Issue

Re: [ANNOUNCE] New Apache Flink Committer - Congxian Qiu

2020-10-29 Thread Matt Wang
Congratulations! -- Best, Matt Wang On 10/29/2020 17:48,Niels Basjes wrote: Congratulations! On Thu, Oct 29, 2020 at 10:45 AM Yun Tang wrote: Congratulations, Congxian! Best Yun Tang From: Xintong Song Sent: Thursday, October 29, 2020 16:16 To: dev Cc:

Re: [ANNOUNCE] New Apache Flink Committer - Congxian Qiu

2020-10-29 Thread Niels Basjes
Congratulations! On Thu, Oct 29, 2020 at 10:45 AM Yun Tang wrote: > Congratulations, Congxian! > > Best > Yun Tang > > From: Xintong Song > Sent: Thursday, October 29, 2020 16:16 > To: dev > Cc: Congxian Qiu ; klio...@apache.org < > klio...@apache.org> >

Re: [ANNOUNCE] New Apache Flink Committer - Congxian Qiu

2020-10-29 Thread Yun Tang
Congratulations, Congxian! Best Yun Tang From: Xintong Song Sent: Thursday, October 29, 2020 16:16 To: dev Cc: Congxian Qiu ; klio...@apache.org Subject: Re: [ANNOUNCE] New Apache Flink Committer - Congxian Qiu Congratulations, Congxian. Thank you~ Xintong

[jira] [Created] (FLINK-19880) flink-json The validation and incoming parameters do not match

2020-10-29 Thread MingZhangOk (Jira)
MingZhangOk created FLINK-19880: --- Summary: flink-json The validation and incoming parameters do not match Key: FLINK-19880 URL: https://issues.apache.org/jira/browse/FLINK-19880 Project: Flink

[jira] [Created] (FLINK-19878) Improve watermark ChangelogNormalize for upsertSource

2020-10-29 Thread Leonard Xu (Jira)
Leonard Xu created FLINK-19878: -- Summary: Improve watermark ChangelogNormalize for upsertSource Key: FLINK-19878 URL: https://issues.apache.org/jira/browse/FLINK-19878 Project: Flink Issue

[jira] [Created] (FLINK-19879) Support "IF NOT EXISTS" in DDL for CREATE TABLE

2020-10-29 Thread Jira
Ingo Bürk created FLINK-19879: - Summary: Support "IF NOT EXISTS" in DDL for CREATE TABLE Key: FLINK-19879 URL: https://issues.apache.org/jira/browse/FLINK-19879 Project: Flink Issue Type:

[jira] [Created] (FLINK-19877) AlterDatabaseOperation is missing ignoreIfNotExists

2020-10-29 Thread Jira
Ingo Bürk created FLINK-19877: - Summary: AlterDatabaseOperation is missing ignoreIfNotExists Key: FLINK-19877 URL: https://issues.apache.org/jira/browse/FLINK-19877 Project: Flink Issue Type:

[jira] [Created] (FLINK-19876) Latency metrics can merge chain task latency metrics to reduce the number of latency metrics

2020-10-29 Thread zlzhang0122 (Jira)
zlzhang0122 created FLINK-19876: --- Summary: Latency metrics can merge chain task latency metrics to reduce the number of latency metrics Key: FLINK-19876 URL: https://issues.apache.org/jira/browse/FLINK-19876

Re: [DISCUSS] 1.12 feature freeze date += one week ?

2020-10-29 Thread Xintong Song
+1 for one week extension. The native Kubernetes HA could also benefit from it. Thank you~ Xintong Song On Thu, Oct 29, 2020 at 3:59 PM Till Rohrmann wrote: > Thanks for starting this discussion Stephan. Some of the features I help > with could also benefit from an extension. Hence, +1 for

[VOTE] Remove flink-connector-filesystem module.

2020-10-29 Thread Kostas Kloudas
Hi all, Following the discussion in [1], I would like to start a vote on removing the flink-connector-filesystem module which includes the BucketingSink. The vote will be open till November 3rd (72h, excluding the weekend) unless there is an objection or not enough votes. Cheers, Kostas [1]

Re: [ANNOUNCE] New Apache Flink Committer - Congxian Qiu

2020-10-29 Thread Xintong Song
Congratulations, Congxian. Thank you~ Xintong Song On Thu, Oct 29, 2020 at 3:57 PM Jingsong Li wrote: > Congratulations! > > On Thu, Oct 29, 2020 at 3:56 PM Yuan Mei wrote: > > > Congratulations! > > > > On Thu, Oct 29, 2020 at 3:53 PM Till Rohrmann > > wrote: > > > > > Congratulations

Re: [DISCUSS] 1.12 feature freeze date += one week ?

2020-10-29 Thread Till Rohrmann
Thanks for starting this discussion Stephan. Some of the features I help with could also benefit from an extension. Hence, +1 for moving the feature freeze to the 8th of November. Cheers, Till On Thu, Oct 29, 2020 at 7:16 AM Robert Metzger wrote: > I have no objections against an extension, if

Re: [ANNOUNCE] New Apache Flink Committer - Congxian Qiu

2020-10-29 Thread Jingsong Li
Congratulations! On Thu, Oct 29, 2020 at 3:56 PM Yuan Mei wrote: > Congratulations! > > On Thu, Oct 29, 2020 at 3:53 PM Till Rohrmann > wrote: > > > Congratulations Congxian! Great to have you as a committer now :-) > > > > Cheers, > > Till > > > > On Thu, Oct 29, 2020 at 8:33 AM Benchao Li

Re: [ANNOUNCE] New Apache Flink Committer - Congxian Qiu

2020-10-29 Thread Yuan Mei
Congratulations! On Thu, Oct 29, 2020 at 3:53 PM Till Rohrmann wrote: > Congratulations Congxian! Great to have you as a committer now :-) > > Cheers, > Till > > On Thu, Oct 29, 2020 at 8:33 AM Benchao Li wrote: > > > Congratulations! > > > > Biao Liu 于2020年10月29日周四 下午3:20写道: > > > > >

Re: [ANNOUNCE] New Apache Flink Committer - Congxian Qiu

2020-10-29 Thread Till Rohrmann
Congratulations Congxian! Great to have you as a committer now :-) Cheers, Till On Thu, Oct 29, 2020 at 8:33 AM Benchao Li wrote: > Congratulations! > > Biao Liu 于2020年10月29日周四 下午3:20写道: > > > Congrads! > > > > Thanks, > > Biao /'bɪ.aʊ/ > > > > > > > > On Thu, 29 Oct 2020 at 15:14, Xingbo

Re: Is there any way to change operator level parallelism dynamically?

2020-10-29 Thread Till Rohrmann
Hi Deep, at the moment the only way to change the parallelism is to take a savepoint, stop the job and then resume the job with a changed parallelism from the savepoint. The community works on a more automated way for this problem. However, in the first versions it will use the same approach with

[jira] [Created] (FLINK-19875) Integrate file compaction to connectors

2020-10-29 Thread Jingsong Lee (Jira)
Jingsong Lee created FLINK-19875: Summary: Integrate file compaction to connectors Key: FLINK-19875 URL: https://issues.apache.org/jira/browse/FLINK-19875 Project: Flink Issue Type: Sub-task

Re: [ANNOUNCE] New Apache Flink Committer - Congxian Qiu

2020-10-29 Thread Benchao Li
Congratulations! Biao Liu 于2020年10月29日周四 下午3:20写道: > Congrads! > > Thanks, > Biao /'bɪ.aʊ/ > > > > On Thu, 29 Oct 2020 at 15:14, Xingbo Huang wrote: > > > Congratulations Congxian. > > > > Best, > > Xingbo > > > > Dian Fu 于2020年10月29日周四 下午3:05写道: > > > > > Congratulations Congxian! > > > > >

Re: [ANNOUNCE] New Apache Flink Committer - Congxian Qiu

2020-10-29 Thread Biao Liu
Congrads! Thanks, Biao /'bɪ.aʊ/ On Thu, 29 Oct 2020 at 15:14, Xingbo Huang wrote: > Congratulations Congxian. > > Best, > Xingbo > > Dian Fu 于2020年10月29日周四 下午3:05写道: > > > Congratulations Congxian! > > > > Regards, > > Dian > > > > > 在 2020年10月29日,下午2:35,Yangze Guo 写道: > > > > > >

Re: [ANNOUNCE] New Apache Flink Committer - Congxian Qiu

2020-10-29 Thread Xingbo Huang
Congratulations Congxian. Best, Xingbo Dian Fu 于2020年10月29日周四 下午3:05写道: > Congratulations Congxian! > > Regards, > Dian > > > 在 2020年10月29日,下午2:35,Yangze Guo 写道: > > > > Congratulations! > > > > Best, > > Yangze Guo > > > > On Thu, Oct 29, 2020 at 2:31 PM Jark Wu wrote: > >> > >> Congrats

[jira] [Created] (FLINK-19873) Skip DDL change events for Canal data

2020-10-29 Thread Jark Wu (Jira)
Jark Wu created FLINK-19873: --- Summary: Skip DDL change events for Canal data Key: FLINK-19873 URL: https://issues.apache.org/jira/browse/FLINK-19873 Project: Flink Issue Type: Sub-task

Re: [ANNOUNCE] New Apache Flink Committer - Congxian Qiu

2020-10-29 Thread Dian Fu
Congratulations Congxian! Regards, Dian > 在 2020年10月29日,下午2:35,Yangze Guo 写道: > > Congratulations! > > Best, > Yangze Guo > > On Thu, Oct 29, 2020 at 2:31 PM Jark Wu wrote: >> >> Congrats Congxian! >> >> Best, >> Jark >> >> On Thu, 29 Oct 2020 at 14:28, Yu Li wrote: >> >>> Hi all, >>>

Re: [ANNOUNCE] New Apache Flink Committer - Congxian Qiu

2020-10-29 Thread Yangze Guo
Congratulations! Best, Yangze Guo On Thu, Oct 29, 2020 at 2:31 PM Jark Wu wrote: > > Congrats Congxian! > > Best, > Jark > > On Thu, 29 Oct 2020 at 14:28, Yu Li wrote: > > > Hi all, > > > > On behalf of the PMC, I’m very happy to announce Congxian Qiu as a new > > Flink committer. > > > >

Re: [ANNOUNCE] New Apache Flink Committer - Congxian Qiu

2020-10-29 Thread Jark Wu
Congrats Congxian! Best, Jark On Thu, 29 Oct 2020 at 14:28, Yu Li wrote: > Hi all, > > On behalf of the PMC, I’m very happy to announce Congxian Qiu as a new > Flink committer. > > Congxian has been an active contributor for more than two years, with 226 > contributions including 76 commits

[jira] [Created] (FLINK-19872) Support to parse millisecond for TIME type in CSV format

2020-10-29 Thread Fangliang Liu (Jira)
Fangliang Liu created FLINK-19872: - Summary: Support to parse millisecond for TIME type in CSV format Key: FLINK-19872 URL: https://issues.apache.org/jira/browse/FLINK-19872 Project: Flink

[ANNOUNCE] New Apache Flink Committer - Congxian Qiu

2020-10-29 Thread Yu Li
Hi all, On behalf of the PMC, I’m very happy to announce Congxian Qiu as a new Flink committer. Congxian has been an active contributor for more than two years, with 226 contributions including 76 commits and many PR reviews. Congxian mainly works on state backend and checkpoint modules,

[jira] [Created] (FLINK-19871) add hash(primary key) to kafka partition sink mapping

2020-10-29 Thread Jira
底限 created FLINK-19871: -- Summary: add hash(primary key) to kafka partition sink mapping Key: FLINK-19871 URL: https://issues.apache.org/jira/browse/FLINK-19871 Project: Flink Issue Type: Improvement

Re: [DISCUSS] 1.12 feature freeze date += one week ?

2020-10-29 Thread Robert Metzger
I have no objections against an extension, if it helps delivering more, and stable features in the release. My proposal would hence be to set the 1.12 cutoff date to the weekend of > the 7th/8th and fork the release-1.12 branch on Monday Nov. 9th. Side comment: In the last feature freeze

Re: [DISCUSS] 1.12 feature freeze date += one week ?

2020-10-29 Thread Zhijiang
Thanks for the consideration, stephan! +1 for one week extension of feature freeze, it would be really helpful for ongoing batch shuffle improvement. Best, Zhijiang -- From:Yuan Mei Send Time:2020年10月29日(星期四) 11:03 To:dev