Re: [DISCUSS] Releasing Apache Flink 1.11.3

2020-11-03 Thread Tzu-Li (Gordon) Tai
Thanks for the efforts so far with pushing for 1.11.3. I'd like to provide a mid-week update on how we're looking with all the pending blockers and backports: *Blockers**:* Currently all blockers either have an open PR, or have been merged. I'll highlight below the blockers *that still require

[jira] [Created] (FLINK-19961) Bash e2e utility run_test_with_timeout is (sometimes?) not properly stopping it's watchdog

2020-11-03 Thread Robert Metzger (Jira)
Robert Metzger created FLINK-19961: -- Summary: Bash e2e utility run_test_with_timeout is (sometimes?) not properly stopping it's watchdog Key: FLINK-19961 URL: https://issues.apache.org/jira/browse/FLINK-19961

Re: [DISCUSS][docker] Adopt Jemalloc as default memory allocator for debian based Flink docker image

2020-11-03 Thread Yun Tang
Hi @ Billy I found there existed many benchmark case existed in the two repos, which benchmark case did you run? Best Yun Tang From: Xie Billy Sent: Tuesday, November 3, 2020 22:08 To: dev@flink.apache.org Subject: Re: [DISCUSS][docker] Adopt Jemalloc as

Re: [DISCUSS] Releasing Apache Flink 1.11.3

2020-11-03 Thread Tzu-Li (Gordon) Tai
> The collect() utils can be picked back, I see no issue with that (it is isolated utilities). Just checking on all the requested backports mentioned in this thread, and figuring out which ones seem to still be unassigned / open. Is someone working on backporting

[jira] [Created] (FLINK-19960) Introduce PartitionFieldExtractor to extract partition field from split

2020-11-03 Thread Jingsong Lee (Jira)
Jingsong Lee created FLINK-19960: Summary: Introduce PartitionFieldExtractor to extract partition field from split Key: FLINK-19960 URL: https://issues.apache.org/jira/browse/FLINK-19960 Project:

[jira] [Created] (FLINK-19959) Multiple input creation algorithm will deduce an incorrect input order if the inputs are related under PIPELINED shuffle mode

2020-11-03 Thread Caizhi Weng (Jira)
Caizhi Weng created FLINK-19959: --- Summary: Multiple input creation algorithm will deduce an incorrect input order if the inputs are related under PIPELINED shuffle mode Key: FLINK-19959 URL:

[jira] [Created] (FLINK-19958) Unified exception signature in Sink API

2020-11-03 Thread Guowei Ma (Jira)
Guowei Ma created FLINK-19958: - Summary: Unified exception signature in Sink API Key: FLINK-19958 URL: https://issues.apache.org/jira/browse/FLINK-19958 Project: Flink Issue Type: Sub-task

Re: [ANNOUNCE] New Apache Flink Committer - Congxian Qiu

2020-11-03 Thread Guowei Ma
Congratulations! Congxian Best, Guowei On Wed, Nov 4, 2020 at 10:36 AM godfrey he wrote: > Congratulations! Congxian > > Best, > Godfrey > > Fabian Hueske 于2020年11月2日周一 下午7:00写道: > > > Congrats Congxian! > > > > Cheers, Fabian > > > > Am Mo., 2. Nov. 2020 um 10:33 Uhr schrieb Yang Wang < >

Re: [ANNOUNCE] New Apache Flink Committer - Congxian Qiu

2020-11-03 Thread godfrey he
Congratulations! Congxian Best, Godfrey Fabian Hueske 于2020年11月2日周一 下午7:00写道: > Congrats Congxian! > > Cheers, Fabian > > Am Mo., 2. Nov. 2020 um 10:33 Uhr schrieb Yang Wang >: > > > Congratulations Congxian! > > > > Best, > > Yang > > > > Zhu Zhu 于2020年11月2日周一 下午5:14写道: > > > > > Congrats

[jira] [Created] (FLINK-19957) flink-sql-connector-hive incompatible with cdh6

2020-11-03 Thread Cheng Pan (Jira)
Cheng Pan created FLINK-19957: - Summary: flink-sql-connector-hive incompatible with cdh6 Key: FLINK-19957 URL: https://issues.apache.org/jira/browse/FLINK-19957 Project: Flink Issue Type: Bug

[jira] [Created] (FLINK-19956) $ does not work on variables without being qualified in Scala

2020-11-03 Thread Rex Remind (Jira)
Rex Remind created FLINK-19956: -- Summary: $ does not work on variables without being qualified in Scala Key: FLINK-19956 URL: https://issues.apache.org/jira/browse/FLINK-19956 Project: Flink

[DISCUSS] FLIP-151: Incremental snapshots for heap-based state backend

2020-11-03 Thread Khachatryan Roman
Hi devs, I'd like to start a discussion of FLIP-151: Incremental snapshots for heap-based state backend [1] Heap backend, while being limited state sizes fitting into memory, also has some advantages compared to RocksDB backend: 1. Serialization once per checkpoint, not per state modification.

Re: [DISCUSS] Releasing Apache Flink 1.11.3

2020-11-03 Thread Steven Wu
@Stephan Ewen yeah, we can do that. don't worry about it. your earlier email had the perfect explanation on why file source shouldn't be backported. On Tue, Nov 3, 2020 at 3:37 AM Stephan Ewen wrote: > @Steven would it be possible to initially copy some of the code into the > iceberg source

[jira] [Created] (FLINK-19955) Document cross-version compatibility

2020-11-03 Thread Robert Metzger (Jira)
Robert Metzger created FLINK-19955: -- Summary: Document cross-version compatibility Key: FLINK-19955 URL: https://issues.apache.org/jira/browse/FLINK-19955 Project: Flink Issue Type:

Re: [DISCUSS] Releasing StateFun hotfix version 2.2.1

2020-11-03 Thread Robert Metzger
Hi Gordon, thanks a lot for this clarification. In this case I would vote for releasing StateFun 2.2.1 asap and not wait for 1.11.3. Thanks a lot for your efforts! On Tue, Nov 3, 2020 at 3:38 PM Tzu-Li (Gordon) Tai wrote: > Hi Robert, > > So far we've only seen a single user report the

Re: [DISCUSS] Releasing StateFun hotfix version 2.2.1

2020-11-03 Thread Tzu-Li (Gordon) Tai
Hi Robert, So far we've only seen a single user report the issue, but the severity of FLINK-19692 is actually pretty huge. TL;DR: If a checkpoint / savepoint that contains feedback events (which is considered normal under typical StateFun operations) is attempted to be restored from, the restore

Re: [DISCUSS][docker] Adopt Jemalloc as default memory allocator for debian based Flink docker image

2020-11-03 Thread Xie Billy
Hi guys: refer: https://stackoverflow.com/questions/13027475/cpu-and-memory-usage-of-jemalloc-as-compared-to-glibc-malloc code: https://github.com/jemalloc/jemalloc-experiments https://code.woboq.org/userspace/glibc/benchtests/ Best Regards! Billy xie(谢志民) On Fri,

Re: [DISCUSS] Releasing StateFun hotfix version 2.2.1

2020-11-03 Thread Robert Metzger
Thanks a lot for starting this thread. How many users are affected by the problem? Is it somebody else besides the initial issue reporter? If it is just one person, I would suggest to rather help pushing the 1.11.3 release over the line or work on more StateFun features ;) On Tue, Nov 3, 2020 at

[jira] [Created] (FLINK-19954) Move execution deployment tracking logic from legacy EG code to SchedulerNG

2020-11-03 Thread Andrey Zagrebin (Jira)
Andrey Zagrebin created FLINK-19954: --- Summary: Move execution deployment tracking logic from legacy EG code to SchedulerNG Key: FLINK-19954 URL: https://issues.apache.org/jira/browse/FLINK-19954

[jira] [Created] (FLINK-19953) Translation in docs/ops/memory/mem_setup.zh.md

2020-11-03 Thread Matthias (Jira)
Matthias created FLINK-19953: Summary: Translation in docs/ops/memory/mem_setup.zh.md Key: FLINK-19953 URL: https://issues.apache.org/jira/browse/FLINK-19953 Project: Flink Issue Type:

Re: [DISCUSS] Releasing Apache Flink 1.11.3

2020-11-03 Thread Stephan Ewen
@Steven would it be possible to initially copy some of the code into the iceberg source and later replace it by a dependency on the Flink file source? On Mon, Nov 2, 2020 at 8:33 PM Steven Wu wrote: > Stephan, thanks a lot for explaining the file connector. that makes sense. > > I was asking

Re: [DISCUSS] Releasing StateFun hotfix version 2.2.1

2020-11-03 Thread Igal Shilman
Hi Gordon, Thanks for driving this discussion! I would go with the second suggestion - having two consecutive StateFun releases 2.2.1 and 2.2.2, since the Flink-1.11.3 release might take a while, and this hot-fix release is important enough to get out as early as possible. Cheers, Igal. On

[jira] [Created] (FLINK-19952) flink SecurityOptions.class use depracted method so mant times

2020-11-03 Thread jackylau (Jira)
jackylau created FLINK-19952: Summary: flink SecurityOptions.class use depracted method so mant times Key: FLINK-19952 URL: https://issues.apache.org/jira/browse/FLINK-19952 Project: Flink

[jira] [Created] (FLINK-19951) PyFlink end-to-end test stuck in "Reading kafka messages"

2020-11-03 Thread Robert Metzger (Jira)
Robert Metzger created FLINK-19951: -- Summary: PyFlink end-to-end test stuck in "Reading kafka messages" Key: FLINK-19951 URL: https://issues.apache.org/jira/browse/FLINK-19951 Project: Flink

[DISCUSS]FLIP-150: Introduce Hybrid Source

2020-11-03 Thread Nicholas Jiang
Hi devs, I'd like to start a new FLIP to introduce the Hybrid Source. The hybrid source is a source that contains a list of concrete sources. The hybrid source reads from each contained source in the defined order. It switches from source A to the next source B when source A finishes. In

[jira] [Created] (FLINK-19950) LookupJoin can not support view or subquery and so on. o

2020-11-03 Thread jackylau (Jira)
jackylau created FLINK-19950: Summary: LookupJoin can not support view or subquery and so on. o Key: FLINK-19950 URL: https://issues.apache.org/jira/browse/FLINK-19950 Project: Flink Issue Type:

Re: Dependency injection and flink.

2020-11-03 Thread Arvid Heise
I'm answering your user ML mail. Please only ask your question once. Dev ML is only used to coordinate Flink development, which kinda makes sense in your case after we verified in your first thread that indeed Flink should provide such things. On Tue, Nov 3, 2020 at 3:14 AM santhosh venkat

[jira] [Created] (FLINK-19949) Unescape CSV format line delimiter character

2020-11-03 Thread Danny Chen (Jira)
Danny Chen created FLINK-19949: -- Summary: Unescape CSV format line delimiter character Key: FLINK-19949 URL: https://issues.apache.org/jira/browse/FLINK-19949 Project: Flink Issue Type:

[jira] [Created] (FLINK-19948) CAST(now() as bigint) throws compile exception

2020-11-03 Thread Jark Wu (Jira)
Jark Wu created FLINK-19948: --- Summary: CAST(now() as bigint) throws compile exception Key: FLINK-19948 URL: https://issues.apache.org/jira/browse/FLINK-19948 Project: Flink Issue Type: Bug