Why checkpointing is synchronized with data processing?

Hi, flink dev team: I'm reading flink's source code and it seems that the process of saving checkpointing data to external storage system will block data processing, and I wonder why this has to be designed like this? Isn't is a potential risk that may increase latency of processing? Is the

Re: [DISCUSSION] Timely function interface and timer params

Hi Gyula, if you look at the internal API you'll notice that it is pretty much like your second proposal. Just for reference, the interface is roughly this: public interface InternalTimerService { long currentProcessingTime(); long currentWatermark(); void registerProcessingTimeTimer(N

RE: trouble with type casting

I agree with you, Fabian. I will add a small hint in exception and cast bigDecimal manually in test cases -Original Message- From: Fabian Hueske [mailto:fhue...@gmail.com] Sent: Friday, October 28, 2016 7:17 PM To: dev@flink.apache.org Subject: Re: trouble with type casting Hi Anton,

[jira] [Created] (FLINK-4963) Tabulate edge direction for directed VertexMetrics

Greg Hogan created FLINK-4963: - Summary: Tabulate edge direction for directed VertexMetrics Key: FLINK-4963 URL: https://issues.apache.org/jira/browse/FLINK-4963 Project: Flink Issue Type:

Re: trouble with type casting

Hi Anton, it seems that the Table API validation is more strict than Calcite's SQL validator (maybe because it is not aware of the actual implementation). In principle, it is correct to prevent the auto-casting from BigDecimal to double. I think it is fine to request an explicit cast from users

Re: [DISCUSS] FLIP-11: Table API Stream Aggregations

Thanks for bringing up this point Stephan. I think it is a good decision to have a standard compliant SQL interface that SQL developers and tools can use. The SQL proposal [1] I posted a few days ago follows this approach. However, SQL was not designed with stream processing in mind and many

[jira] [Created] (FLINK-4962) Improve exception information about operand type inference failed in UDF SQL validation

Jark Wu created FLINK-4962: -- Summary: Improve exception information about operand type inference failed in UDF SQL validation Key: FLINK-4962 URL: https://issues.apache.org/jira/browse/FLINK-4962 Project:

[DISCUSSION] Timely function interface and timer params

Hello, I was thinking about the methods provided by the timely functions and the timerservice and I am wondering if it makes sense to change them a little so they can cover a wider set of use case. Maybe I missed something completely obvious so please shoot me down in that case :) Currently the

[jira] [Created] (FLINK-4961) SGD for Matrix Factorization

Gábor Hermann created FLINK-4961: Summary: SGD for Matrix Factorization Key: FLINK-4961 URL: https://issues.apache.org/jira/browse/FLINK-4961 Project: Flink Issue Type: New Feature

[jira] [Created] (FLINK-4960) Enable testing of different re-scaling scenarios in the AbstractStreamOperatorTestHarness to

Kostas Kloudas created FLINK-4960: - Summary: Enable testing of different re-scaling scenarios in the AbstractStreamOperatorTestHarness to Key: FLINK-4960 URL: https://issues.apache.org/jira/browse/FLINK-4960

Re: [DISCUSS] Schedule and Scope for Flink 1.2

Hello, As Aljoshca said, the TriggerDSL depends on another PR, before it can be merged. In addition, this will change also the whole structure of how we test window/trigger related functionality, so some refactoring will be required. Kostas > On Oct 28, 2016, at 3:06 PM, Aljoscha Krettek

Re: [HEADS UP] Introducing ConfigOptions

+1 for the ConfigOptions and the automatic configuration description generation once we've migrated all config keys. Cheers, Till On Fri, Oct 28, 2016 at 2:18 PM, Robert Metzger wrote: > You are right Stephan. The doc migration will be easier when we use the new > logic

Re: [DISCUSS] Schedule and Scope for Flink 1.2

@Till: For the TriggerDSL we have to merge my open PR [1] for refactoring the WindowOperator/Trigger tests and then refactor the tests of the TriggerDSL to use the new test facilities introduced there. Plus then maybe some fixups. On Fri, 28 Oct 2016 at 11:17 Jark Wu

[jira] [Created] (FLINK-4959) Write Documentation for Timely FlatMap Functions

Aljoscha Krettek created FLINK-4959: --- Summary: Write Documentation for Timely FlatMap Functions Key: FLINK-4959 URL: https://issues.apache.org/jira/browse/FLINK-4959 Project: Flink Issue

[jira] [Created] (FLINK-4958) Send slot report to RM when registering

Till Rohrmann created FLINK-4958: Summary: Send slot report to RM when registering Key: FLINK-4958 URL: https://issues.apache.org/jira/browse/FLINK-4958 Project: Flink Issue Type: Sub-task

Re: [DISCUSS] Releasing Flink 1.1.4

Thanks for all your feedback. If there are no objections, I would like to stick to the mentioned issues in this thread and create RC1 as soon as they are all addressed. This will probably not be this week though, but it looks good for next week. DONE = - FLINK-4619: Answer client if

[jira] [Created] (FLINK-4957) Provide API for TimelyCoFlatMapFunction

Aljoscha Krettek created FLINK-4957: --- Summary: Provide API for TimelyCoFlatMapFunction Key: FLINK-4957 URL: https://issues.apache.org/jira/browse/FLINK-4957 Project: Flink Issue Type: Test

Re: [HEADS UP] Introducing ConfigOptions

You are right Stephan. The doc migration will be easier when we use the new logic for all keys. On Fri, Oct 28, 2016 at 4:27 AM, Jark Wu wrote: > +1 > > The automatic configuration page generating sounds really cool. > > - Jark Wu > > > 在 2016年10月28日，上午12:23，Stephan

[jira] [Created] (FLINK-4956) Introduce convenient methods for testing checkpoint/restore in StreamTaskTestHarness

Stefan Richter created FLINK-4956: - Summary: Introduce convenient methods for testing checkpoint/restore in StreamTaskTestHarness Key: FLINK-4956 URL: https://issues.apache.org/jira/browse/FLINK-4956

[jira] [Created] (FLINK-4955) Add Translations Tests for KeyedStream.flatMap(TimelyFlatMapFunction)

Aljoscha Krettek created FLINK-4955: --- Summary: Add Translations Tests for KeyedStream.flatMap(TimelyFlatMapFunction) Key: FLINK-4955 URL: https://issues.apache.org/jira/browse/FLINK-4955 Project:

[jira] [Created] (FLINK-4954) Remove stashing from AkkaRpcActor

Till Rohrmann created FLINK-4954: Summary: Remove stashing from AkkaRpcActor Key: FLINK-4954 URL: https://issues.apache.org/jira/browse/FLINK-4954 Project: Flink Issue Type: Sub-task

Re: [DISCUSS] Add Side Input/Broadcast Set For Streaming API

Hello, I find this feature really cool because it would allow people to tackle scenarios requiring a more advanced "join" on multiple streams as already stated above. Here you can find my few

trouble with type casting

I'm working on flink-4743 issue I tried write tests as Fabian suggested me https://github.com/apache/flink/pull/2686#discussion_r85218584 and figure out that there is no implicit conversion of bigDecimal to double in table api. "power" function accept only double as operands. all is okay when

Re: [DISCUSS] Schedule and Scope for Flink 1.2

+1 to the schedule I hope to add 1) Table API: User defined table function (FLINK-4469 ) 2) Table API: Move Table API to org.apache.flink.table (FLINK-4704 ) The 1) has been discussed for a

Re: [DISCUSS] FLIP-11: Table API Stream Aggregations

Hi , I agree that Table API should be a SQL-like API, that do not need to be strictly consistent with SQL. > - Not put the window definition into the groupBy clause. +1 Putting the window clause out of groupBy, will be easy to learn and easy to discover for users. > I like the idea of

Re: Apache Flink and Kudu integration

Hi Ruben, I am currently not aware of such an effort, but I definitely do agree that it is an interesting pattern to investigate. As a motivation you could have a look at the Spark connector implementations to see the Kudu APIs in use. For that I would recommend the DataSource API implementation

[jira] [Created] (FLINK-4953) Allow querying time in window functions

Manu Zhang created FLINK-4953: - Summary: Allow querying time in window functions Key: FLINK-4953 URL: https://issues.apache.org/jira/browse/FLINK-4953 Project: Flink Issue Type: Improvement

Apache Flink and Kudu integration

Hi all, Is there any PoC about reading/writing from/to Kudu? I think the flow kafka-flink-kudu is an interesting pattern. I would like to evaluate it so please let me know if there is any existing attempt to avoid starting from scratch. Advices are welcomed :) Best