Re: [DISCUSS] FLIP-142: Disentangle StateBackends from Checkpointing

It seems my questions are misunderstood to be about details on implementations, but actually my concerns are from the users' view, especially how to understand the new APIs and how to choose/use them. To better express my concern, let's list the new APIs this FLIP proposes to introduce:

[jira] [Created] (FLINK-19335) Automatically get udf's resource files from hdfs when running a job that uses hive-udf

Husky Zeng created FLINK-19335: -- Summary: Automatically get udf's resource files from hdfs when running a job that uses hive-udf Key: FLINK-19335 URL: https://issues.apache.org/jira/browse/FLINK-19335

[jira] [Created] (FLINK-19338) New source interface cannot unregister unregistered source

Arvid Heise created FLINK-19338: --- Summary: New source interface cannot unregister unregistered source Key: FLINK-19338 URL: https://issues.apache.org/jira/browse/FLINK-19338 Project: Flink

[jira] [Created] (FLINK-19332) Special characters issue using Kinesis Data Analytics for Apache Flink

Zekun Yu created FLINK-19332: Summary: Special characters issue using Kinesis Data Analytics for Apache Flink Key: FLINK-19332 URL: https://issues.apache.org/jira/browse/FLINK-19332 Project: Flink

[jira] [Created] (FLINK-19336) EncodingUtils#encodeObjectToString should propagate inner exception

Benchao Li created FLINK-19336: -- Summary: EncodingUtils#encodeObjectToString should propagate inner exception Key: FLINK-19336 URL: https://issues.apache.org/jira/browse/FLINK-19336 Project: Flink

[jira] [Created] (FLINK-19333) Introduce BatchArrowPythonOverWindowAggregateFunctionOperator

Huang Xingbo created FLINK-19333: Summary: Introduce BatchArrowPythonOverWindowAggregateFunctionOperator Key: FLINK-19333 URL: https://issues.apache.org/jira/browse/FLINK-19333 Project: Flink

Automatically get udf's resource files from hdfs when running a job that uses hive-udf

http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Is-there-a-way-to-avoid-submit-hive-udf-s-resources-when-we-submit-a-job-td38204.html As the mail say，I plan to resolve this problem in my production environment , is there any suggestion ？ -- Sent from:

[jira] [Created] (FLINK-19334) Translate page 'Streaming File Sink' into Chinese

hailong wang created FLINK-19334: Summary: Translate page 'Streaming File Sink' into Chinese Key: FLINK-19334 URL: https://issues.apache.org/jira/browse/FLINK-19334 Project: Flink Issue

Re: Automatically get udf's resource files from hdfs when running a job that uses hive-udf

Hi Husky, Module is a mechanism to support built-in functions which should always be in the classpath. So I'm afraid it may conflict with the current mechanism to support dynamic loading for modules. IIUC, what you want is the `CREATE FUNCTION ... USING JAR` which is discussed in FLINK-14055

Re: [DISCUSS] FLIP-143: Unified Sink API

Aljoscha/Guowei, I think we are pretty close with aligning on the Iceberg sink requirements. This new sink API can really benefit and simplify Iceberg sink implementation. Looking forward to the initial scope with 1.12 release. > CommitResult commit(GlobalCommittableT); I like the

[jira] [Created] (FLINK-19337) Make a little bit improvement for PyFlink package structure and class name.

sunjincheng created FLINK-19337: --- Summary: Make a little bit improvement for PyFlink package structure and class name. Key: FLINK-19337 URL: https://issues.apache.org/jira/browse/FLINK-19337 Project:

Re: [DISCUSS] FLIP-142: Disentangle StateBackends from Checkpointing

To me, the simplifications made by Seth sound good and do make a lot of sense. We should really break this down to a few orthogonal guides, then it is easy for users: - Metadata always goes through the JobManager, no matter what CheckpointStorage. - The JobManagerCheckpointStorage has the

[jira] [Created] (FLINK-19330) Recovery with async operations fails due to unitialized runtimeContext

Igal Shilman created FLINK-19330: Summary: Recovery with async operations fails due to unitialized runtimeContext Key: FLINK-19330 URL: https://issues.apache.org/jira/browse/FLINK-19330 Project:

Re: [DISCUSS] FLIP-143: Unified Sink API

Hi all, I'll try and summarize my thoughts after Guowei, Yun, Kostas, Dawid, and me had an offline discussion about this. Also, I would like to give credit to Guowei for initially coming up with the idea of a topology sink in the context of this discussion. I think it's a good idea and we

Re: [VOTE] Apache Flink Stateful Functions 2.2.0, release candidate #1

Hi everyone, This vote is canceled. We have found few blocking issues [FLINK-19330], [FLINK-19327] a new RC would be out tomorrow. Thanks, Igal. On Mon, Sep 21, 2020 at 8:56 AM Tzu-Li (Gordon) Tai wrote: > Hi everyone, > > Please review and vote on the release candidate #1 for the version

[jira] [Created] (FLINK-19331) State processor api has native resouce leak when working with RocksDB

Seth Wiesman created FLINK-19331: Summary: State processor api has native resouce leak when working with RocksDB Key: FLINK-19331 URL: https://issues.apache.org/jira/browse/FLINK-19331 Project: Flink

[jira] [Created] (FLINK-19309) Add TaskExecutorAllocator

Chesnay Schepler created FLINK-19309: Summary: Add TaskExecutorAllocator Key: FLINK-19309 URL: https://issues.apache.org/jira/browse/FLINK-19309 Project: Flink Issue Type: Task

[jira] [Created] (FLINK-19319) Deprecate StreamExecutionEnvironment.setStreamTimeCharacteristic()

Aljoscha Krettek created FLINK-19319: Summary: Deprecate StreamExecutionEnvironment.setStreamTimeCharacteristic() Key: FLINK-19319 URL: https://issues.apache.org/jira/browse/FLINK-19319 Project:

[jira] [Created] (FLINK-19320) Remove clearBuffers from the public interfaces of RecordWriter

Yingjie Cao created FLINK-19320: --- Summary: Remove clearBuffers from the public interfaces of RecordWriter Key: FLINK-19320 URL: https://issues.apache.org/jira/browse/FLINK-19320 Project: Flink

[jira] [Created] (FLINK-19311) Add ResourceRequirement(s)

Chesnay Schepler created FLINK-19311: Summary: Add ResourceRequirement(s) Key: FLINK-19311 URL: https://issues.apache.org/jira/browse/FLINK-19311 Project: Flink Issue Type: Sub-task

Re: [VOTE] FLIP-33: Standardize connector metrics

+1 On Mon, Sep 21, 2020 at 11:31 AM Becket Qin wrote: > Hi all, > > I would like to start the voting thread for FLIP-33 which proposes to > standardize the metrics of Flink connectors. > > In short, we would like to provide a convention and guidance of Flink > connector metrics. It will help

[jira] [Created] (FLINK-19322) Specified parallelism in create table operation

YufeiLiu created FLINK-19322: Summary: Specified parallelism in create table operation Key: FLINK-19322 URL: https://issues.apache.org/jira/browse/FLINK-19322 Project: Flink Issue Type:

[jira] [Created] (FLINK-19305) Setup Azure build for running core and flink-tests

Chesnay Schepler created FLINK-19305: Summary: Setup Azure build for running core and flink-tests Key: FLINK-19305 URL: https://issues.apache.org/jira/browse/FLINK-19305 Project: Flink

[jira] [Created] (FLINK-19304) Add feature toggle

Chesnay Schepler created FLINK-19304: Summary: Add feature toggle Key: FLINK-19304 URL: https://issues.apache.org/jira/browse/FLINK-19304 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-19310) Add new TaskManagerSlot implementation

Chesnay Schepler created FLINK-19310: Summary: Add new TaskManagerSlot implementation Key: FLINK-19310 URL: https://issues.apache.org/jira/browse/FLINK-19310 Project: Flink Issue Type:

[jira] [Created] (FLINK-19312) Introduce BufferWritingResultPartition which wraps the logic of writing buffers to ResultSubpartition

Yingjie Cao created FLINK-19312: --- Summary: Introduce BufferWritingResultPartition which wraps the logic of writing buffers to ResultSubpartition Key: FLINK-19312 URL:

[jira] [Created] (FLINK-19321) CollectSinkFunction does not define serialVersionUID

Fabian Hueske created FLINK-19321: - Summary: CollectSinkFunction does not define serialVersionUID Key: FLINK-19321 URL: https://issues.apache.org/jira/browse/FLINK-19321 Project: Flink Issue

Re: [VOTE] FLIP-33: Standardize connector metrics

+1， Thanks for driving this, Becket! Best, Jincheng Becket Qin 于2020年9月21日周一 上午11:31写道： > Hi all, > > I would like to start the voting thread for FLIP-33 which proposes to > standardize the metrics of Flink connectors. > > In short, we would like to provide a convention and guidance of Flink

Re: [VOTE] FLIP-33: Standardize connector metrics

+1 I could see this is a well written document after a long and thorough discussion [1]. Thanks for driving this all along, Becket! Best Regards, Yu [1] http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-33-Standardize-connector-metrics-td26869.html On Mon, 21 Sep

[jira] [Created] (FLINK-19324) Map requested/allocated containers with priority on YARN

Xintong Song created FLINK-19324: Summary: Map requested/allocated containers with priority on YARN Key: FLINK-19324 URL: https://issues.apache.org/jira/browse/FLINK-19324 Project: Flink

[jira] [Created] (FLINK-19313) Implement declarative SlotPool

Chesnay Schepler created FLINK-19313: Summary: Implement declarative SlotPool Key: FLINK-19313 URL: https://issues.apache.org/jira/browse/FLINK-19313 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-19315) Move slot-bookkeeping into separate component

Chesnay Schepler created FLINK-19315: Summary: Move slot-bookkeeping into separate component Key: FLINK-19315 URL: https://issues.apache.org/jira/browse/FLINK-19315 Project: Flink Issue

[jira] [Created] (FLINK-19323) Small optimization of network layer record serialization

Yingjie Cao created FLINK-19323: --- Summary: Small optimization of network layer record serialization Key: FLINK-19323 URL: https://issues.apache.org/jira/browse/FLINK-19323 Project: Flink Issue

[jira] [Created] (FLINK-19325) Optimize the consumed time for checkpoint completion

Congxian Qiu(klion26) created FLINK-19325: - Summary: Optimize the consumed time for checkpoint completion Key: FLINK-19325 URL: https://issues.apache.org/jira/browse/FLINK-19325 Project: Flink

[jira] [Created] (FLINK-19306) Implement DeclarativeSlotManager

Chesnay Schepler created FLINK-19306: Summary: Implement DeclarativeSlotManager Key: FLINK-19306 URL: https://issues.apache.org/jira/browse/FLINK-19306 Project: Flink Issue Type:

Re: [DISCUSS] FLIP-143: Unified Sink API

Hi Guowei, > I believe that we could support such an async sink writer > very easily in the future. What do you think? How would you see the expansion in the future? Do you mean just adding `isAvailable()` method with a default implementation later on? Piotrek pon., 21 wrz 2020 o 02:39 Steven

[jira] [Created] (FLINK-19307) Add RequirementsTracker

Chesnay Schepler created FLINK-19307: Summary: Add RequirementsTracker Key: FLINK-19307 URL: https://issues.apache.org/jira/browse/FLINK-19307 Project: Flink Issue Type: Task

[jira] [Created] (FLINK-19317) Make EventTime the default StreamTimeCharacteristic

Aljoscha Krettek created FLINK-19317: Summary: Make EventTime the default StreamTimeCharacteristic Key: FLINK-19317 URL: https://issues.apache.org/jira/browse/FLINK-19317 Project: Flink

[jira] [Created] (FLINK-19308) Add SlotTracker

Chesnay Schepler created FLINK-19308: Summary: Add SlotTracker Key: FLINK-19308 URL: https://issues.apache.org/jira/browse/FLINK-19308 Project: Flink Issue Type: Task

[jira] [Created] (FLINK-19314) Add compatibility layer

Chesnay Schepler created FLINK-19314: Summary: Add compatibility layer Key: FLINK-19314 URL: https://issues.apache.org/jira/browse/FLINK-19314 Project: Flink Issue Type: Task

[jira] [Created] (FLINK-19316) FLIP-134: Batch execution for the DataStream API

Aljoscha Krettek created FLINK-19316: Summary: FLIP-134: Batch execution for the DataStream API Key: FLINK-19316 URL: https://issues.apache.org/jira/browse/FLINK-19316 Project: Flink

[jira] [Created] (FLINK-19318) Deprecate timeWindow() operations in DataStream API

Aljoscha Krettek created FLINK-19318: Summary: Deprecate timeWindow() operations in DataStream API Key: FLINK-19318 URL: https://issues.apache.org/jira/browse/FLINK-19318 Project: Flink

[jira] [Created] (FLINK-19326) Allow explicitly configuring time behaviour on CEP PatternStream

Aljoscha Krettek created FLINK-19326: Summary: Allow explicitly configuring time behaviour on CEP PatternStream Key: FLINK-19326 URL: https://issues.apache.org/jira/browse/FLINK-19326 Project:

Re: [DISCUSS][Release 1.12] Stale blockers and build instabilities

Hi all, An update on the release status: 1. We have 35 days = *5 weeks left until feature freeze* 2. There are currently 2 blockers for Flink , all making progress 3. We have 72 test instabilities

[jira] [Created] (FLINK-19301) Improve the package structure of Python DataStream API

Dian Fu created FLINK-19301: --- Summary: Improve the package structure of Python DataStream API Key: FLINK-19301 URL: https://issues.apache.org/jira/browse/FLINK-19301 Project: Flink Issue Type:

Re: [ANNOUNCE] Apache Flink 1.11.2 released

Thanks Zhu Zhu for being our release manager and everyone else who made the release possible! Best Regards, Yu On Thu, 17 Sep 2020 at 13:29, Zhu Zhu wrote: > The Apache Flink community is very happy to announce the release of Apache > Flink 1.11.2, which is the second bugfix release for the

[RESULT][VOTE] FLIP-138: Declarative Resource Management

FLIP-138 has been accepted with 3 binding votes and no disapproving  votes: - zhu zhu - xintong - chesnay On 9/17/2020 10:59 AM, Chesnay Schepler wrote: +1 On 9/4/2020 9:40 AM, Zhu Zhu wrote: +1 Thanks, Zhu Xintong Song 于2020年9月4日周五 下午1:26写道： Thanks for starting this vote. +1 from my

Re: [DISCUSS] FLIP-142: Disentangle StateBackends from Checkpointing

Thanks for the update Seth, and let me further clarify my comments / concerns around the new `CheckpointStorage`. 1. In the existing `MemoryStateBackend`, there's a `maxStateSize` field which limits the maximal state size sent to JM from one single memory backend, with the default size of 5MB.

[VOTE] Apache Flink Stateful Functions 2.2.0, release candidate #1

Hi everyone, Please review and vote on the release candidate #1 for the version 2.2.0 of Apache Flink Stateful Functions, as follows: [ ] +1, Approve the release [ ] -1, Do not approve the release (please provide specific comments) ***Testing Guideline*** You can find here [1] a page in the

[jira] [Created] (FLINK-19303) Disable WAL in RocksDB recovery

Juha Mynttinen created FLINK-19303: -- Summary: Disable WAL in RocksDB recovery Key: FLINK-19303 URL: https://issues.apache.org/jira/browse/FLINK-19303 Project: Flink Issue Type: Improvement

[jira] [Created] (FLINK-19302) Flushing of BoundedBlockingResultPartition should finish current BufferBuilder

Yingjie Cao created FLINK-19302: --- Summary: Flushing of BoundedBlockingResultPartition should finish current BufferBuilder Key: FLINK-19302 URL: https://issues.apache.org/jira/browse/FLINK-19302

Re: Timed out patterns handling using MATCH_RECOGNIZE

Hi Jark, Thank you for your e-mail. I agree, let's engage all interested parties in this discussion - I'm writing this e-mail to both Flink and Calcite dev mailing lists. I'll repeat myself to present the proposal to the Calcite community. I would like to propose an enrichment of existing

[RESULT][VOTE] FLIP-134: Batch execution for the DataStream API

Hi all, The voting time for FLIP-134 [1] has passed. I'm closing the vote now. Including my implicit vote, there were 7 + 1 votes, 4 of which are binding: - Dawid Wysakowicz (binding) - Gao Yun - Ma Guowei - David Anderson (binding) - Kostas Kloudas (binding) - Peter Huang - Aljoscha Krettek

Re: Can you unify the language ?

Hi, you are right. Having two languages in the code base doesn't make our lives easier. But Flink is a big project with a long history, multiple design shifts, and many contributors. It is naturally that the bigger a code base gets, the messier it looks like. It must be a continuous effort

Re: [DISCUSS] Drop Scala 2.11

We use a Cloudera 6.3 cluster in prod. I'd guess that it's still widely used in prod as those cloudera upgrades for major versions are planned long time ahead and take a significant amount of resources in big data lakes. On that 6.3. cluster, if I open spark-shell, I still see scala 2.11 in

[jira] [Created] (FLINK-19329) FunctionGroupOperator#dispose might throw an NPE

Igal Shilman created FLINK-19329: Summary: FunctionGroupOperator#dispose might throw an NPE Key: FLINK-19329 URL: https://issues.apache.org/jira/browse/FLINK-19329 Project: Flink Issue

[jira] [Created] (FLINK-19328) flink sql

cellen created FLINK-19328: -- Summary: flink sql Key: FLINK-19328 URL: https://issues.apache.org/jira/browse/FLINK-19328 Project: Flink Issue Type: Bug Components: Table SQL / API

[jira] [Created] (FLINK-19327) Helm charts specify a too low value for the job manager's heap size

Igal Shilman created FLINK-19327: Summary: Helm charts specify a too low value for the job manager's heap size Key: FLINK-19327 URL: https://issues.apache.org/jira/browse/FLINK-19327 Project: Flink

Re: [DISCUSS] FLIP-142: Disentangle StateBackends from Checkpointing

Hi Yu, Let me address your comments one at a time. I think I can address comments one and two with a single answer. This FLIP does not change any runtime data structures or implementations. As such, it only provides new user-facing factory classes for those components. StateBackend (the