[jira] [Created] (FLINK-19597) Introduce SortBuffer and its implementation PartitionSortedBuffer for sort-merge based blocking shuffle

Yingjie Cao created FLINK-19597: --- Summary: Introduce SortBuffer and its implementation PartitionSortedBuffer for sort-merge based blocking shuffle Key: FLINK-19597 URL:

Re: Need help in creating Flink Streaming s3 Job for multiple path reader one by one

Thanks, I'll check it out. On Mon, Oct 12, 2020 at 1:29 PM Chesnay Schepler wrote: > 1) There's no mechanism in the API to restrict the number of number of > readers across several sources. I can't quite think of a way to achieve > this; maybe Kostas has an idea. > > 2) You're mixing up the

[jira] [Created] (FLINK-19596) Do not recover CompletedCheckpointStore on each failover

Jiayi Liao created FLINK-19596: -- Summary: Do not recover CompletedCheckpointStore on each failover Key: FLINK-19596 URL: https://issues.apache.org/jira/browse/FLINK-19596 Project: Flink Issue

[jira] [Created] (FLINK-19595) Flink SQL support S3 select

liuxiaolong created FLINK-19595: --- Summary: Flink SQL support S3 select Key: FLINK-19595 URL: https://issues.apache.org/jira/browse/FLINK-19595 Project: Flink Issue Type: Improvement

[jira] [Created] (FLINK-19594) SubTasks start index don't unified and may confuse users

zlzhang0122 created FLINK-19594: --- Summary: SubTasks start index don't unified and may confuse users Key: FLINK-19594 URL: https://issues.apache.org/jira/browse/FLINK-19594 Project: Flink Issue

[jira] [Created] (FLINK-19593) FlinkKafkaInternalProducer should be @Internal annotation ?

Kenzyme Le created FLINK-19593: -- Summary: FlinkKafkaInternalProducer should be @Internal annotation ? Key: FLINK-19593 URL: https://issues.apache.org/jira/browse/FLINK-19593 Project: Flink

Re: [DISCUSS]Why not emit messages to prevent too early state eviction in MiniBatchGroupAggFunction?

Hi Benchao, Thank you for your confirmation. I created an issue for this bug as Flink-19592 (https://issues.apache.org/jira/browse/FLINK-19592) - Best wishes. Smile -- Sent from: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/

[jira] [Created] (FLINK-19592) MiniBatchGroupAggFunction should emit messages to prevent too early state eviction of downstream operators

Smile created FLINK-19592: - Summary: MiniBatchGroupAggFunction should emit messages to prevent too early state eviction of downstream operators Key: FLINK-19592 URL: https://issues.apache.org/jira/browse/FLINK-19592

[jira] [Created] (FLINK-19591) Missing @Internal annotation SimpleJdbcConnectionProvider

Kenzyme Le created FLINK-19591: -- Summary: Missing @Internal annotation SimpleJdbcConnectionProvider Key: FLINK-19591 URL: https://issues.apache.org/jira/browse/FLINK-19591 Project: Flink Issue

[jira] [Created] (FLINK-19590) flink-streaming-java module failed to compile

Dian Fu created FLINK-19590: --- Summary: flink-streaming-java module failed to compile Key: FLINK-19590 URL: https://issues.apache.org/jira/browse/FLINK-19590 Project: Flink Issue Type: Bug

[jira] [Created] (FLINK-19589) Expose extra options in the S3 FileSystem

Padarn Wilson created FLINK-19589: - Summary: Expose extra options in the S3 FileSystem Key: FLINK-19589 URL: https://issues.apache.org/jira/browse/FLINK-19589 Project: Flink Issue Type:

Re: [DISCUSS] FLIP-147: Support Checkpoints After Tasks Finished

Hi Yun, 4) Yes, the interaction is not trivial and also I have not completely thought it through. But in general, I'm currently at the point where I think that we also need non-checkpoint related events in unaligned checkpoints. So just keep that in mind, that we might converge anyhow at this

Re: [DISCUSS] Remove flink-connector-filesystem module.

Is there a way for us to change the module (in a reasonable way) that would allow users to continue using it? Is it an API problem, or one of semantics? On 10/12/2020 4:57 PM, Kostas Kloudas wrote: Hi Chesnay, Unfortunately not from what I can see in the code. This is the reason why I am

Re: [DISCUSS][Release 1.12] Stale blockers and build instabilities

Hi all! According to the plan discussed earlier in the release cycle, the feature freeze is expected to happen in the week of October 26th. That's in 2.5 weeks from now. I believe now is the time to discuss if we want to postpone

[jira] [Created] (FLINK-19588) HBase zookeeper connection not released in hbase batch table source while flink job failover

zouyunhe created FLINK-19588: Summary: HBase zookeeper connection not released in hbase batch table source while flink job failover Key: FLINK-19588 URL: https://issues.apache.org/jira/browse/FLINK-19588

Re: [DISCUSS] FLIP-147: Support Checkpoints After Tasks Finished

Hi Arvid, Very thanks for the insightful comments! I added the responses for this issue under the quota: >> 1) You call the tasks that get the barriers injected leaf nodes, which would >> make the > sinks the root nodes. That is very similar to how graphs in >> relational algebra are labeled.

Re:Re: [VOTE] Add Translation Specification for Stateful Functions

+1 Best, Hailong Wang At 2020-10-12 17:00:34, "Xintong Song" wrote: >+1 > >Thank you~ > >Xintong Song > > > >On Mon, Oct 12, 2020 at 5:59 PM Jark Wu wrote: > >> +1 >> >> On Mon, 12 Oct 2020 at 17:14, Yu Li wrote: >> >> > +1 >> > >> > Best Regards, >> > Yu >> > >> > >> > On Mon, 12 Oct 2020 at

[jira] [Created] (FLINK-19587) Error result when casting binary type as varchar

hailong wang created FLINK-19587: Summary: Error result when casting binary type as varchar Key: FLINK-19587 URL: https://issues.apache.org/jira/browse/FLINK-19587 Project: Flink Issue Type:

[jira] [Created] (FLINK-19586) Implement the stream committer operator

Guowei Ma created FLINK-19586: - Summary: Implement the stream committer operator Key: FLINK-19586 URL: https://issues.apache.org/jira/browse/FLINK-19586 Project: Flink Issue Type: Sub-task

Re: [DISCUSS] Remove flink-connector-filesystem module.

Hi Chesnay, Unfortunately not from what I can see in the code. This is the reason why I am opening a discussion. I think that if we supported backwards compatibility, this would have been an easier process. Kostas On Mon, Oct 12, 2020 at 4:32 PM Chesnay Schepler wrote: > > Are older versions

[jira] [Created] (FLINK-19585) UnalignedCheckpointCompatibilityITCase.test:97->runAndTakeSavepoint

Robert Metzger created FLINK-19585: -- Summary: UnalignedCheckpointCompatibilityITCase.test:97->runAndTakeSavepoint Key: FLINK-19585 URL: https://issues.apache.org/jira/browse/FLINK-19585 Project:

[jira] [Created] (FLINK-19584) HBaseSinkFunction no needs to create thread to flush when bufferFlushMaxMutations = 1

hailong wang created FLINK-19584: Summary: HBaseSinkFunction no needs to create thread to flush when bufferFlushMaxMutations = 1 Key: FLINK-19584 URL: https://issues.apache.org/jira/browse/FLINK-19584

Re: [DISCUSS] Remove flink-connector-filesystem module.

Are older versions of the module compatible with 1.12+? On 10/12/2020 4:30 PM, Kostas Kloudas wrote: Hi all, As the title suggests, this thread is to discuss the removal of the flink-connector-filesystem module which contains (only) the deprecated BucketingSink. The BucketingSin is deprecated

[DISCUSS] Remove flink-connector-filesystem module.

Hi all, As the title suggests, this thread is to discuss the removal of the flink-connector-filesystem module which contains (only) the deprecated BucketingSink. The BucketingSin is deprecated since FLINK 1.9 [1] in favor of the relatively recently introduced StreamingFileSink. For the sake of a

[jira] [Created] (FLINK-19583) Expose the execution.runtime-mode to the users

Kostas Kloudas created FLINK-19583: -- Summary: Expose the execution.runtime-mode to the users Key: FLINK-19583 URL: https://issues.apache.org/jira/browse/FLINK-19583 Project: Flink Issue

[jira] [Created] (FLINK-19582) Introduce sort-merge based blocking shuffle to Flink

Yingjie Cao created FLINK-19582: --- Summary: Introduce sort-merge based blocking shuffle to Flink Key: FLINK-19582 URL: https://issues.apache.org/jira/browse/FLINK-19582 Project: Flink Issue

Re: [DISCUSS]Why not emit messages to prevent too early state eviction in MiniBatchGroupAggFunction?

Hi Smile, Thanks for bringing up this discussion. I think you are right, it's an bug, MiniBatchGroupAggFunction should also do this. You can open a jira issue for this, and offer to fix it if you wish. Even more, current MiniBatchGroupAggFunction has some more severe bug, it's not cleaning

Re: [VOTE] Add Translation Specification for Stateful Functions

+1 > 在 2020年10月12日，下午6:31，Smile 写道： > > +1 > > A specification really matters for the translation quality. > > > > -- > Sent from: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/

[jira] [Created] (FLINK-19581) Introduce OrcColumnarRowInputFormat

Jingsong Lee created FLINK-19581: Summary: Introduce OrcColumnarRowInputFormat Key: FLINK-19581 URL: https://issues.apache.org/jira/browse/FLINK-19581 Project: Flink Issue Type: Sub-task

Re: [VOTE] Add Translation Specification for Stateful Functions

+1 A specification really matters for the translation quality. -- Sent from: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/

[DISCUSS]Why not emit messages to prevent too early state eviction in MiniBatchGroupAggFunction?

When using SELECT DISTINCT in Flink SQL I found that when a new message with the same key arrives, it will emit a retract and a new insert message. According to JIRA-FLINK-8566 and JIRA-FLINK-8564

Re: [VOTE] Add Translation Specification for Stateful Functions

+1 Thank you~ Xintong Song On Mon, Oct 12, 2020 at 5:59 PM Jark Wu wrote: > +1 > > On Mon, 12 Oct 2020 at 17:14, Yu Li wrote: > > > +1 > > > > Best Regards, > > Yu > > > > > > On Mon, 12 Oct 2020 at 14:41, Congxian Qiu > wrote: > > > > > I would like to start a voting thread for adding

[jira] [Created] (FLINK-19580) Creating PackagedProgram overwrites an actual job name with default JobName

Murtaza H Zaveri created FLINK-19580: Summary: Creating PackagedProgram overwrites an actual job name with default JobName Key: FLINK-19580 URL: https://issues.apache.org/jira/browse/FLINK-19580

Re: [VOTE] Add Translation Specification for Stateful Functions

+1 On Mon, 12 Oct 2020 at 17:14, Yu Li wrote: > +1 > > Best Regards, > Yu > > > On Mon, 12 Oct 2020 at 14:41, Congxian Qiu wrote: > > > I would like to start a voting thread for adding translation > specification > > for Stateful Functions, which we’ve reached consensus in [1]. > > > > > >

Re: Wrapping a Flink Function

Could you maybe outline how you want to extend the wrapped sink functionality? A better approach might be to add an operation "in front" of the sink. Best, Aljoscha On 08.10.20 11:32, Lorenzo Pirazzini wrote: Hello, I'm having trouble finding a way to add logic to an existing SinkFunction.

Re: [VOTE] Add Translation Specification for Stateful Functions

+1 Best Regards, Yu On Mon, 12 Oct 2020 at 14:41, Congxian Qiu wrote: > I would like to start a voting thread for adding translation specification > for Stateful Functions, which we’ve reached consensus in [1]. > > > This voting will be open for a minimum 3 days till 3:00 pm UTC, Oct 15. > >

[jira] [Created] (FLINK-19579) Create the SideOutputTransformation translator

Kostas Kloudas created FLINK-19579: -- Summary: Create the SideOutputTransformation translator Key: FLINK-19579 URL: https://issues.apache.org/jira/browse/FLINK-19579 Project: Flink Issue

[jira] [Created] (FLINK-19577) Create the UnionTransformation translator

Kostas Kloudas created FLINK-19577: -- Summary: Create the UnionTransformation translator Key: FLINK-19577 URL: https://issues.apache.org/jira/browse/FLINK-19577 Project: Flink Issue Type:

[jira] [Created] (FLINK-19578) Create the PartitionTransformation translator

Kostas Kloudas created FLINK-19578: -- Summary: Create the PartitionTransformation translator Key: FLINK-19578 URL: https://issues.apache.org/jira/browse/FLINK-19578 Project: Flink Issue

[jira] [Created] (FLINK-19575) Create the LegacySourceTransformation translator

Kostas Kloudas created FLINK-19575: -- Summary: Create the LegacySourceTransformation translator Key: FLINK-19575 URL: https://issues.apache.org/jira/browse/FLINK-19575 Project: Flink Issue

[jira] [Created] (FLINK-19576) Create the SinkTransformation translator

Kostas Kloudas created FLINK-19576: -- Summary: Create the SinkTransformation translator Key: FLINK-19576 URL: https://issues.apache.org/jira/browse/FLINK-19576 Project: Flink Issue Type:

[jira] [Created] (FLINK-19573) Create the MultipleInputTransformation translator

Kostas Kloudas created FLINK-19573: -- Summary: Create the MultipleInputTransformation translator Key: FLINK-19573 URL: https://issues.apache.org/jira/browse/FLINK-19573 Project: Flink Issue

[jira] [Created] (FLINK-19574) Create the SourceTransformation translator

Kostas Kloudas created FLINK-19574: -- Summary: Create the SourceTransformation translator Key: FLINK-19574 URL: https://issues.apache.org/jira/browse/FLINK-19574 Project: Flink Issue Type:

[jira] [Created] (FLINK-19572) Create the TwoInputTransformation translator

Kostas Kloudas created FLINK-19572: -- Summary: Create the TwoInputTransformation translator Key: FLINK-19572 URL: https://issues.apache.org/jira/browse/FLINK-19572 Project: Flink Issue Type:

[jira] [Created] (FLINK-19571) Port Transformation translation logic to StreamGraphTranslator framework

Kostas Kloudas created FLINK-19571: -- Summary: Port Transformation translation logic to StreamGraphTranslator framework Key: FLINK-19571 URL: https://issues.apache.org/jira/browse/FLINK-19571

[jira] [Created] (FLINK-19570) Execution graph related tests are possibly broken due to registering duplicated ExecutionAttemptID

Zhu Zhu created FLINK-19570: --- Summary: Execution graph related tests are possibly broken due to registering duplicated ExecutionAttemptID Key: FLINK-19570 URL: https://issues.apache.org/jira/browse/FLINK-19570

[jira] [Created] (FLINK-19569) Upgrade ICU4J to 67.1+

Chesnay Schepler created FLINK-19569: Summary: Upgrade ICU4J to 67.1+ Key: FLINK-19569 URL: https://issues.apache.org/jira/browse/FLINK-19569 Project: Flink Issue Type: Improvement

Re: Need help in creating Flink Streaming s3 Job for multiple path reader one by one

1) There's no mechanism in the API to restrict the number of  number of readers across several sources. I can't quite think of a way to achieve this; maybe Kostas has an idea. 2) You're mixing  up the Java Streams and Finks DataStream API. Try this: s3PathList.stream() .map(...) .reduce(...)

Re: [DISCUSS] FLIP-147: Support Checkpoints After Tasks Finished

Hi Yun, Thank you for starting the discussion. This will solve one of the long-standing issues [1] that confuse users. I'm also a big fan of option 3. It is also a bit closer to Chandy-Lamport again. A couple of comments: 1) You call the tasks that get the barriers injected leaf nodes, which

[VOTE] Add Translation Specification for Stateful Functions

I would like to start a voting thread for adding translation specification for Stateful Functions, which we’ve reached consensus in [1]. This voting will be open for a minimum 3 days till 3:00 pm UTC, Oct 15. [1]

[jira] [Created] (FLINK-19568) Offload creating TM launch contexts to the IO executor

Xintong Song created FLINK-19568: Summary: Offload creating TM launch contexts to the IO executor Key: FLINK-19568 URL: https://issues.apache.org/jira/browse/FLINK-19568 Project: Flink Issue

Re: [DISCUSS] Add Translation Specification for Stateful Functions

Thanks all for the positive feedback. merging the proposed translation specification into the existing document is better. In that form, people can find the specification on one page, it's more convenient。 I'll start a separate email for the vote. Best, Congxian Tzu-Li (Gordon) Tai