Re: [DISCUSS] Flink 1.6 features

2018-06-05 Thread Ben Yan
Hi Stephan, Will [ https://issues.apache.org/jira/browse/FLINK-5479 ] (Per-partition watermarks in FlinkKafkaConsumer should consider idle partitions) be included in 1.6? As we are seeing more users with this issue on the mailing lists. Thanks. Ben 2018-06-05 5:29 GMT+08:00 Che Lui Shum : >

Re: Ask for SQL using kafka in Flink

2018-06-05 Thread Will Du
Yes, I am also looking for examples for Kafka avro table examples in java and command line. Also, Kafka avro table sink is still missing. In addition, once we have Kafka topic, the API should read the schema directly from schema file or schema registry. The way of current API supporting lacks

[jira] [Created] (FLINK-9526) BucketingSink end-to-end test failed on Travis

2018-06-05 Thread Chesnay Schepler (JIRA)
Chesnay Schepler created FLINK-9526: --- Summary: BucketingSink end-to-end test failed on Travis Key: FLINK-9526 URL: https://issues.apache.org/jira/browse/FLINK-9526 Project: Flink Issue

[jira] [Created] (FLINK-9525) Missing META-INF/services/*FileSystemFactory in flink-hadoop-fs module

2018-06-05 Thread Hai Zhou (JIRA)
Hai Zhou created FLINK-9525: --- Summary: Missing META-INF/services/*FileSystemFactory in flink-hadoop-fs module Key: FLINK-9525 URL: https://issues.apache.org/jira/browse/FLINK-9525 Project: Flink

[jira] [Created] (FLINK-9527) Paths initialize with URI are not normalized

2018-06-05 Thread Chesnay Schepler (JIRA)
Chesnay Schepler created FLINK-9527: --- Summary: Paths initialize with URI are not normalized Key: FLINK-9527 URL: https://issues.apache.org/jira/browse/FLINK-9527 Project: Flink Issue Type:

Re: Ask for SQL using kafka in Flink

2018-06-05 Thread Timo Walther
@Shuyi: Yes, a more advanced table example would be helpful anyway and combining it with Kafka/Avro end-to-end would be even better. @Will: I totally agree that the current connector ecosystem could be improved. This is also on mid-term roadmap. Contributors that could help here are very

Re: [DISCUSS] Adding new interfaces in [Stream]ExecutionEnvironment

2018-06-05 Thread Fabian Hueske
We could also offer a feature that users can request classloaders with additional jars. This could work as follows: 1) Users register jar files in the ExecutionEnvironment (similar to cached files) with a name, e.g., env.registerJarFile("~/myJar.jar", "myName"); 2) In a function, the user can

[jira] [Created] (FLINK-9529) Do not expose keyed methods on ProcessFunction.Context

2018-06-05 Thread Till Rohrmann (JIRA)
Till Rohrmann created FLINK-9529: Summary: Do not expose keyed methods on ProcessFunction.Context Key: FLINK-9529 URL: https://issues.apache.org/jira/browse/FLINK-9529 Project: Flink Issue

[jira] [Created] (FLINK-9530) Task numRecords metrics broken for chains

2018-06-05 Thread Chesnay Schepler (JIRA)
Chesnay Schepler created FLINK-9530: --- Summary: Task numRecords metrics broken for chains Key: FLINK-9530 URL: https://issues.apache.org/jira/browse/FLINK-9530 Project: Flink Issue Type:

[jira] [Created] (FLINK-9528) Incorrect results: Filter does not treat Upsert messages correctly.

2018-06-05 Thread Fabian Hueske (JIRA)
Fabian Hueske created FLINK-9528: Summary: Incorrect results: Filter does not treat Upsert messages correctly. Key: FLINK-9528 URL: https://issues.apache.org/jira/browse/FLINK-9528 Project: Flink

[WEBSITE] Proposal to rework the Flink website

2018-06-05 Thread Fabian Hueske
Hi everybody, I've opened a PR [1] that reworks parts of the Flink website ( flink.apache.org). My goal is to improve the structure of the website and provide more valuable information about the project and the community. A visitor (who doesn't know Flink yet) should be able to easily find

Re: [DISCUSS] FLIP-6 Problems

2018-06-05 Thread Till Rohrmann
Hi Renjie, 1) you're right that the Flink session mode does not give you proper job isolation. It is the same as with Flink 1.4 session mode. If this is a strong requirement for you, then I recommend using the per job mode. 2) At the moment it is also not possible to define per job resource

Re: Need help regarding MongoDB oplog tailing using flink.

2018-06-05 Thread Chesnay Schepler
Please ask a more specific question. What have you tried so far, where exactly are you stuck etc. . Note that the existing connectors are always a good source for inspiration: https://github.com/apache/flink/tree/master/flink-connectors On 05.06.2018 10:58, Amol S - iProgrammer wrote: Hello

Need help regarding MongoDB oplog tailing using flink.

2018-06-05 Thread Amol S - iProgrammer
Hello Ezmlm, I have gone through flink documentation and found it quit interesting but I am stuck in one task i.e. mongodb streaming oplog using flink. Can you help me to figure out this? --- *Amol Suryawanshi* Java Developer am...@iprogrammer.com

Re: [TABLE][SQL] Unify UniqueKeyExtractor and DataStreamRetractionRules

2018-06-05 Thread Fabian Hueske
Hi, I think the proposed refactoring is a good idea. It should simplify the logic to determine which update mode to use. We could also try to make some of the method and field names more intuitive and extend the internal documentation a bit. @Hequn, It would be good to get your thoughts on this

Re: [TABLE][SQL] Unify UniqueKeyExtractor and DataStreamRetractionRules

2018-06-05 Thread Rong Rong
+1 on the refactoring. I spent some time a while back trying to get a better understanding on the several rules mentioned here. Correct me if I were wrong by I was under the impression that the reason why the rules are split was because AccMode and UpdateMode are the ones that we care about and

Re: [VOTE] Release flink-shaded 4.0, release candidate #2

2018-06-05 Thread Timo Walther
+1 - Checked the commits that went into this release - Build from source I found one minor thing. There are two "create_release_branch.sh" scripts in the tools directory. Regards, Timo Am 01.06.18 um 08:32 schrieb Tzu-Li (Gordon) Tai: +1 - Checked signatures and hashes - Source builds

Re: [DISCUSS] FLIP-6 Problems

2018-06-05 Thread Stephan Ewen
Hi Renjie, When you suggest to have TaskManager isolation in session mode, do you mean to have a shared JobManager / Dispatcher, but job-specific TaskManagers? If this mainly to reduce the overhead of the per-job JobManager? One assumption so far was that if TaskManager isolation is required,

Re: [DISCUSS] FLIP-6 Problems

2018-06-05 Thread Renjie Liu
Hi, Till: 1. Does the community has any plan to add task manager isolation into the session mode? 2. Is there any issues to track this feature? I want to help contribute. 3. Thanks for the knowledge but it can't help if task manager isolation is not present. On Tue, Jun 5, 2018

[jira] [Created] (FLINK-9531) Improve logging during type extraction of the Row type

2018-06-05 Thread Timo Walther (JIRA)
Timo Walther created FLINK-9531: --- Summary: Improve logging during type extraction of the Row type Key: FLINK-9531 URL: https://issues.apache.org/jira/browse/FLINK-9531 Project: Flink Issue

Re: [WEBSITE] Proposal to rework the Flink website

2018-06-05 Thread Stephan Ewen
Big +1 to this! I would like to contribute to this effort and help strengthen the way Flink presents itself. On Tue, Jun 5, 2018 at 11:56 AM, Fabian Hueske wrote: > Hi everybody, > > I've opened a PR [1] that reworks parts of the Flink website ( > flink.apache.org). > > My goal is to improve

Re: [WEBSITE] Proposal to rework the Flink website

2018-06-05 Thread Ted Yu
+1 On Tue, Jun 5, 2018 at 9:46 AM, Stephan Ewen wrote: > Big +1 to this! > > I would like to contribute to this effort and help strengthen the way Flink > presents itself. > > > On Tue, Jun 5, 2018 at 11:56 AM, Fabian Hueske wrote: > > > Hi everybody, > > > > I've opened a PR [1] that reworks

Re: [WEBSITE] Proposal to rework the Flink website

2018-06-05 Thread Ken Krugler
Along these lines, it would help to add a sitemap (and the robots.txt required to reference it) for flink.apache.org and ci.apache.org (for /projects/flink) You can see what Tomcat did along these lines - http://tomcat.apache.org/robots.txt references http://tomcat.apache.org/sitemap.xml,

[jira] [Created] (FLINK-9534) CEP

2018-06-05 Thread aitozi (JIRA)
aitozi created FLINK-9534: - Summary: CEP Key: FLINK-9534 URL: https://issues.apache.org/jira/browse/FLINK-9534 Project: Flink Issue Type: Bug Reporter: aitozi -- This message was

[jira] [Created] (FLINK-9533) CEP

2018-06-05 Thread aitozi (JIRA)
aitozi created FLINK-9533: - Summary: CEP Key: FLINK-9533 URL: https://issues.apache.org/jira/browse/FLINK-9533 Project: Flink Issue Type: Bug Reporter: aitozi -- This message was

Re: [TABLE][SQL] Unify UniqueKeyExtractor and DataStreamRetractionRules

2018-06-05 Thread Hequn Cheng
Hi, thanks for bringing up this discussion. I agree to unify the UniqueKeyExtractor and DataStreamRetractionRules, however I am not sure if it is a good idea to implement it with RelShuttle. Theoretically, retraction rules and other rules may depend on each other. So, by using a RelShuttle

[jira] [Created] (FLINK-9532) Flink Overview of Jobs Documentation Incorrect

2018-06-05 Thread Abdul Qadeer (JIRA)
Abdul Qadeer created FLINK-9532: --- Summary: Flink Overview of Jobs Documentation Incorrect Key: FLINK-9532 URL: https://issues.apache.org/jira/browse/FLINK-9532 Project: Flink Issue Type: Bug

Re: [DISCUSS] FLIP-6 Problems

2018-06-05 Thread Renjie Liu
Hi, Stephan: Yes that's what I mean. In fact the most import thing is to share the dispatcher so that we can have *a centralized gateway for flink job management and submission. The problem with per job cluster is that we can't have a centralized gateway.* I didn't realize that job manager also