Re: [DISCUSS] Support Interactive Programming in Flink Table API

Thanks for the feedback, Fabian. As you mentioned, cache() method itself does not imply any implementation detail. In fact, we plan to implement a default table service which is locality aware, so the default table service hopefully will be satisfactory in most cases. We could also explore more

Re: [DISCUSS]Enhancing flink scheduler by implementing blacklist mechanism

This is a quite useful feature for production use. I once encountered such a case in production cluster and the Storm jobs used 2 hours to stabilize. After that, we implemented similar blacklist solution for storm. The design doc looks good to me. Some minor suggestions about blacklist removal:

Re: [DISCUSS]Enhancing flink scheduler by implementing blacklist mechanism

thanks yingjie to share this doc and I think this is very important feature for production. As you mentioned in your document, an unhealthy node can cause a TM startup failure but cluster management may offer the same node for some reason. (I have encountered such a scenario in our production

Re: [DISCUSS]Enhancing flink scheduler by implementing blacklist mechanism

Hi yingjie, Thanks for proposing the blacklist! I agree with that black list is important for job maintenance, since some jobs may not be able to failover automatically if some tasks are always scheduled to the problematic hosts or TMs. This will increase the burden of the operators

Apply for contributor permission

Hi! I find some bug-fix PRs I created weeks ago are not getting reviewed, and I’m wondering if I need to assign the jira ticket to myself to get the corresponding PR reviewed? If so, please kindly give me contributor permission and my jira id is `Paul Lin`. Thank you! Best, Paul Lam

Apply for flink contributor permission

Hi Flink team, Could you pls give me the contributor permission, as I’m working on JIRA issue [1]. My Jira ID is *Qi* (luoqi...@gmail.com ) [1] https://issues.apache.org/jira/browse/FLINK-10941 Thanks, Qi

Re: Apply for flink contributor permission

Thanks, Fabian! On Tue, Nov 27, 2018 at 10:20 PM Fabian Hueske wrote: > Hi, > > I gave you contributor permissions. > > Best, Fabian > > Am Di., 27. Nov. 2018 um 10:44 Uhr schrieb Jinhu Wu < > jinhu.wu@gmail.com > >: > > > Hi Flink team > > Could you pls give me the contributor permission,

Re: [DISCUSS] Long-term goal of making flink-table Scala-free

Hi Timo and Vino, I agree that table is very active and there is no guarantee for not producing any conflicts if you decide to develop based on community version. I think this part is the risk what we can imagine in the first place. But massively language replacing is something you can not

Re: [DISCUSS] Flink SQL DDL Design

Thanks a lot, Timo and Xuefu. Yes, I think we can finalize the design doc first and start implementation w/o the unified connector API ready by skipping some featue. Xuefu, I like the idea of making Flink specific properties into generic key-value pairs, so that it will make integration with Hive

Re: [DISCUSS] Flink backward compatibility

Some scenarios that come to mind: Flink client binary compatibility with remote cluster: This would include RemoteStreamEnvironment, RESTClusterClient etc. - User should be able to submit the job built with 1.6.x using the 1.6.x binaries to the remote Flink 1.7.x or later cluster. The use case

[jira] [Created] (FLINK-11015) Remove deprecated methods and classes from all the Kafka connectors

vinoyang created FLINK-11015: Summary: Remove deprecated methods and classes from all the Kafka connectors Key: FLINK-11015 URL: https://issues.apache.org/jira/browse/FLINK-11015 Project: Flink

Re: [DISCUSS] Flink SQL DDL Design

Yeah! I agree with Timo that DDL can actually proceed w/o being blocked by connector API. We can leave the unknown out while defining the basic syntax. @Shuyi As commented in the doc, I think we can probably stick with simple syntax with general properties, without extending the syntax too

Re: [DISCUSS] Flink backward compatibility

Hi, I think this is a very good discussion to have. Flink is becoming part of more and more production deployments and more tools are built around it. The question is do we want to (or can we) make parts of the control/maintenance/monitoring API stable such that external systems/frameworks can

Re: Apply for flink contributor permission

Hi, I gave you contributor permissions. Best, Fabian Am Di., 27. Nov. 2018 um 10:44 Uhr schrieb Jinhu Wu : > Hi Flink team > Could you pls give me the contributor permission, I could not assign Jira > ticket to myself. My Jira ID is *wujinhu* > > > Thanks >

Re: [DISCUSS] Flink backward compatibility

I think this discussion needs specific examples as to what should be possible as it otherwise is to vague / open to interpretation. For example, "job submission" may refer to CLI invocations continuing to work (i.e. CLI arguments), or being able to use a 1.6 client against a 1.7 cluster,

[jira] [Created] (FLINK-11014) Relocate flink-metrics-graphite's dropwizard dependencies

Till Rohrmann created FLINK-11014: - Summary: Relocate flink-metrics-graphite's dropwizard dependencies Key: FLINK-11014 URL: https://issues.apache.org/jira/browse/FLINK-11014 Project: Flink

[jira] [Created] (FLINK-11013) Fix distinct aggregates for group window in Table API

Dian Fu created FLINK-11013: --- Summary: Fix distinct aggregates for group window in Table API Key: FLINK-11013 URL: https://issues.apache.org/jira/browse/FLINK-11013 Project: Flink Issue Type: Bug

Re: [DISCUSS] Long-term goal of making flink-table Scala-free

Hi Kurt, I understand your concerns. However, there is no concrete roadmap for Flink 2.0 and (as Vino said) the flink-table is developed very actively. Major refactorings happened in the past and will also happen with or without Scala migration. A good example, is the proper catalog support

Re: [DISCUSS] Long-term goal of making flink-table Scala-free

Hi Kurt, Currently, there is still a long time to go from flink 2.0. Considering that the flink-table is one of the most active modules in the current flink project, each version has a number of changes and features added. I think that refactoring faster will reduce subsequent complexity and

Re: [DISCUSS] Long-term goal of making flink-table Scala-free

Hi Timo, Thanks for writing up the document. I'm +1 for reorganizing the module structure and make table scala free. But I have a little concern abount the timing. Is it more appropriate to get this done when Flink decide to bump to next big version, like 2.x. It's true you can keep all the

Re: [DISCUSS] Table API Enhancement Outline

I don't think we came to a conclusion yet. Are you suggesting that 'w would disappear if we do the following: val tabX = tab.window(Tumble ... as 'w) .groupBy('w, 'k1, 'k2) .flatAgg(tableAgg('a)) .select('*) Given that tableAgg returns [col1, col2], what would the schema of tabX be?

Re: [DISCUSS] FLIP-27: Refactor Source Interface

Hi Biao, Thanks for the answer! So given the multi-threaded readers, now we have as open questions: 1) How do we let the checkpoints pass through our multi-threaded reader operator? 2) Do we have separate reader and source operators or not? In the strategy that has a separate source, the

Re: [DISCUSS] Long-term goal of making flink-table Scala-free

Hi Hequn, thanks for your feedback. Yes, migrating the test cases is another issue that is not represented in the document but should naturally go along with the migration. I agree that we should migrate the main API classes quickly within this 1.8 release after the module split has been

Re: [DISCUSS] Table API Enhancement Outline

Thanks Fabian, if we enforcing select, as i said before user should using 'w.start, 'w.end, 'w.rowtime, 'w.proctime, 'w.XXX' etc. In this way we should not defined the type of 'w, we can keep the current way of using 'w. I'll file the JIRA. and open the PR ASAP. Thanks, Jincheng Fabian Hueske

Apply for flink contributor permission

Hi Flink team Could you pls give me the contributor permission, I could not assign Jira ticket to myself. My Jira ID is *wujinhu* Thanks

[jira] [Created] (FLINK-11012) Introduce abstract superclass for filesystem IT cases

Stefan Richter created FLINK-11012: -- Summary: Introduce abstract superclass for filesystem IT cases Key: FLINK-11012 URL: https://issues.apache.org/jira/browse/FLINK-11012 Project: Flink

[jira] [Created] (FLINK-11011) Elasticsearch 6 sink end-to-end test unstable

Timo Walther created FLINK-11011: Summary: Elasticsearch 6 sink end-to-end test unstable Key: FLINK-11011 URL: https://issues.apache.org/jira/browse/FLINK-11011 Project: Flink Issue Type:

Re: [DISCUSS] Table API Enhancement Outline

I thought about it again. Enforcing select won't help us with the choice of how to represent 'w. select('*) and select('a, 'b, 'w). would still be valid expressions and we need to decide how 'w is represented. As I said before, Tuple, Row, and Map have disadvantages because the syntax 'w.rowtime

[jira] [Created] (FLINK-11010) Flink SQL timestamp is inconsistent with currentProcessingTime()

lamber-ken created FLINK-11010: -- Summary: Flink SQL timestamp is inconsistent with currentProcessingTime() Key: FLINK-11010 URL: https://issues.apache.org/jira/browse/FLINK-11010 Project: Flink

Re: [DISCUSS] Flink SQL DDL Design

Thanks for offering your help here, Xuefu. It would be great to move these efforts forward. I agree that the DDL is somehow releated to the unified connector API design but we can also start with the basic functionality now and evolve the DDL during this release and next releases. For