Re: Flip23

2018-11-05 Thread Fabian Hueske
Hi Boris, Thanks for sharing the code that you'd like to contribute for FLIP-23. I have a quick look at the repository and collected some stats to estimate the reviewing effort for the contribution. There are approx 1900 lines of Java and 2000 lines of Scala code. This is a reasonable size that s

Re: Flink SQL Client Kafka keyed serialization

2018-10-23 Thread Fabian Hueske
Hi Dom, I think support for Kafka keys would be covered by Timo's proposal for improvements of the source / sink connectors [1]. See the section on "Concat multiple formats for accessing connector-specific properties" in the proposal document [2]. Best, Fabian [1] https://lists.apache.org/thread

Re: [VOTE] Release 1.6.2, release candidate #1

2018-10-19 Thread Fabian Hueske
n 2 years. > > Given that no listed issue is new in this release I would not cancel the > RC. > > On 18.10.2018 23:16, Fabian Hueske wrote: > > Hi, > > > > Thanks Chesnay for preparing the RC1 for Flink 1.6.2. > > > > I checked a few things, but there

Re: [VOTE] Release 1.5.5, release candidate #1

2018-10-19 Thread Fabian Hueske
k-table tests aren't run, and hence > the bridge never being loaded. > > The ES5 issue is an OS-incompatibility caused by ES. This issue should > not be new since we didn't modify the ES5 version in 2 years. > > Given that neither issue is new in this release I would

Re: [VOTE] Release 1.5.5, release candidate #1

2018-10-19 Thread Fabian Hueske
Yes, exactly Am Fr., 19. Okt. 2018 um 13:49 Uhr schrieb Chesnay Schepler < ches...@apache.org>: > When you went past hbase, which was the other module you failed? I would > guess you also failed in the ES5 module here like for 1.6.2? > > On 18.10.2018 23:55, Fabian Hu

Re: [VOTE] Release 1.6.2, release candidate #1

2018-10-19 Thread Fabian Hueske
0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) Am Fr., 19. Okt. 2018 um 12:17 Uhr schrieb Fabian Hueske : > Thanks Chesnay, > I'll check again on the error that failed the build. > > > > > > Am Fr., 19. Okt. 2018 um 12:13 Uhr schrieb Chesn

Re: [VOTE] Release 1.6.2, release candidate #1

2018-10-19 Thread Fabian Hueske
lock the release on it. > > On 19.10.2018 10:44, Chesnay Schepler wrote: > > I've opened https://issues.apache.org/jira/browse/FLINK-10608 for the > > RAT issue. > > > > On 19.10.2018 10:38, Fabian Hueske wrote: > >> It's these two: > >>

Re: [VOTE] Release 1.6.2, release candidate #1

2018-10-19 Thread Fabian Hueske
se > header? > > On 18.10.2018 23:16, Fabian Hueske wrote: > > Hi, > > > > Thanks Chesnay for preparing the RC1 for Flink 1.6.2. > > > > I checked a few things, but there seem to be some issues with the release > > candidate. > > > > *

Re: [VOTE] Release 1.5.5, release candidate #1

2018-10-18 Thread Fabian Hueske
Hi, I checked the following things: * no dependencies added or changed since Flink 1.5.4 * compiling the source distribution without tests succeeds * compiling the source distribution with tests fails (see exception appended below). When I restart the compilation, it goes past flink-hbase but fai

Re: [VOTE] Release 1.6.2, release candidate #1

2018-10-18 Thread Fabian Hueske
Hi, Thanks Chesnay for preparing the RC1 for Flink 1.6.2. I checked a few things, but there seem to be some issues with the release candidate. * no dependencies added or changed since Flink 1.6.1 * building the source distribution without tests succeeds, however, a second build fails due to gene

Re: [DISCUSS] [Contributing] (3) - Review Tooling

2018-10-18 Thread Fabian Hueske
flink/pull/6873 Am Do., 18. Okt. 2018 um 07:34 Uhr schrieb jincheng sun < sunjincheng...@gmail.com>: > I like @Fabian Hueske 's proposal, currently design > the > template is pretty good idea. Because the template is convenient for > contributors to follow the norms

Re: [DISCUSS] [Contributing] (2) - Review Steps

2018-10-18 Thread Fabian Hueske
Hi, I merged the PR. The review process is documented at [1]. Best, Fabian [1] https://flink.apache.org/reviewing-prs.html Am Mi., 10. Okt. 2018 um 17:48 Uhr schrieb Fabian Hueske : > Hi all, > > I opened a PR [1] to add the PR review guide to the Flink website. > > Cheers

Re: [DISCUSS] [Contributing] (3) - Review Tooling

2018-10-16 Thread Fabian Hueske
repository + contributors can learn about the review process before opening a PR On the downside, the template grows a bit at the end. What do you think? Best, Fabian Am Mo., 24. Sep. 2018 um 15:51 Uhr schrieb Fabian Hueske : > Hi, > > Coming back to the original topic of the threa

Re: Creating a slide set for a Flink intro talk

2018-10-16 Thread Fabian Hueske
ggestions. > > Best, Hequn > > On Sat, Oct 13, 2018 at 10:12 AM jincheng sun > wrote: > >> @Fabian Hueske Thanks for create the slide. >> I think it is very important for the construction of the flink ecosystem. >> The content of the slide outline is very comprehe

Re: Add checkpoint hooks to window triggers

2018-10-16 Thread Fabian Hueske
chieve that with the current API? > > Thank you very much! > > Best, > Paul Lam > > > 在 2018年10月15日,19:45,Fabian Hueske 写道: > > Hi Paul, > > I think this would be very tricky to implement and interfere with many > parts of the system like state backends,

Re: Add checkpoint hooks to window triggers

2018-10-15 Thread Fabian Hueske
e providing a lower isolation level. > > Thanks a lot! > > Best, > Paul Lam > > > 在 2018年10月15日,15:47,Fabian Hueske 写道: > > > > Hi Paul, > > > > If I got your proposal right, you'd like to fire a Trigger right before a > > checkpoi

Re: Add checkpoint hooks to window triggers

2018-10-15 Thread Fabian Hueske
Hi Paul, If I got your proposal right, you'd like to fire a Trigger right before a checkpoint is taken, correct? So, before taking a checkpoint, a Trigger would fire and the operator would process and emit some intermediate results. This approach would not completely solve the consistency issue b

[jira] [Created] (FLINK-10543) Leverage efficient timer deletion in relational operators

2018-10-14 Thread Fabian Hueske (JIRA)
Fabian Hueske created FLINK-10543: - Summary: Leverage efficient timer deletion in relational operators Key: FLINK-10543 URL: https://issues.apache.org/jira/browse/FLINK-10543 Project: Flink

Creating a slide set for a Flink intro talk

2018-10-12 Thread Fabian Hueske
Hi everybody, I'm currently creating a slide set for a Flink intro talk [1]. The content will be mostly based on pages of the recently updated website * Main page [2] * What is Apache Flink? [3] * Use cases [4] * Powered By [5] The idea is to have a good set of slides that anybody can use to giv

Re: Become a contributor

2018-10-12 Thread Fabian Hueske
Hi Xuefu, I gave (hopefully) your Jira user (xuefuz) Contributor permissions for Flink's Jira. You can now assign issues to yourself. Best, Fabian Am Fr., 12. Okt. 2018 um 01:18 Uhr schrieb Zhang, Xuefu < xuef...@alibaba-inc.com>: > Hi there, > > Could anyone kindly add me as a contributor to F

[jira] [Created] (FLINK-10528) Remove deprecated APIs from Table API for Flink 1.7.0

2018-10-11 Thread Fabian Hueske (JIRA)
Fabian Hueske created FLINK-10528: - Summary: Remove deprecated APIs from Table API for Flink 1.7.0 Key: FLINK-10528 URL: https://issues.apache.org/jira/browse/FLINK-10528 Project: Flink

Re: Sharing state between subtasks

2018-10-10 Thread Fabian Hueske
I think the new source interface would be designed to be able to leverage shared state to achieve time alignment. I don't think this would be possible without some kind of shared state. The problem of tasks that are far ahead in time cannot be solved with back-pressure. That's because a task canno

Re: [DISCUSS] [Contributing] (2) - Review Steps

2018-10-10 Thread Fabian Hueske
>>> Peter Huang 于2018年10月9日周二 下午1:54写道: > >>> > >>>> +1 > >>>> > >>>> On Mon, Oct 8, 2018 at 7:47 PM Thomas Weise wrote: > >>>> > >>>>> +1 > >>>>> > >>>>> > >

Re: [DISCUSS] Integrate Flink SQL well with Hive ecosystem

2018-10-10 Thread Fabian Hueske
Hi Xuefu, Welcome to the Flink community and thanks for starting this discussion! Better Hive integration would be really great! Can you go into details of what you are proposing? I can think of a couple ways to improve Flink in that regard: * Support for Hive UDFs * Support for Hive metadata cat

Re: Sharing state between subtasks

2018-10-09 Thread Fabian Hueske
Hi, I think watermark / event-time skew is a problem that many users are struggling with. A built-in primitive to align event-time would be a great feature! However, there are also some cases when it would be useful for different streams to have diverging event-time, such as an interval join [1]

Re: [DISCUSS] Dropping flink-storm?

2018-10-09 Thread Fabian Hueske
Yes, let's do it this way. The wrapper classes are probably not too complex and can be easily tested. We have the same for the Hadoop interfaces, although I think only the Input- and OutputFormatWrappers are actually used. Am Di., 9. Okt. 2018 um 09:46 Uhr schrieb Chesnay Schepler < ches...@apach

Re: Request for Contributor permissions

2018-10-08 Thread Fabian Hueske
Hi Rafi, Welcome to the Flink community! I gave you contributor permissions for JIRA. Best, Fabian Am Mo., 8. Okt. 2018 um 21:18 Uhr schrieb Rafi Aroch : > Hi, > > I would like to assign an issue to myself. Could someone assign contributor > permissions to my user? > My username in JIRA is: *ar

Re: [DISCUSS] [Contributing] (2) - Review Steps

2018-10-08 Thread Fabian Hueske
at? > > > > > > Thanks, > > > Thomas > > > > > > > > > > > > On Tue, Sep 25, 2018 at 6:17 AM Tzu-Li Chen > > wrote: > > > > > >> Hi Fabian, > > >> > > >> You convinced me. I miss the advantag

Re: TPCH/TPCDS benchmark

2018-10-04 Thread Fabian Hueske
Hi, I have a branch in my Github repository to test the TPC-H queries [1] [2]. All queries are supported (four need to be slightly rewritten). When checking the results of the benchmark, please keep in mind that so far we focused our efforts on extending the functionality and unified semantics fo

Re: MapState support for operator state

2018-10-04 Thread Fabian Hueske
One challenge would be duplicate keys in this context. Am Do., 4. Okt. 2018 um 10:17 Uhr schrieb Till Rohrmann < trohrm...@apache.org>: > Hi Daniel, > > I don't think that there is a fundamental problem of having MapState > available for operator state. First, there are some questions to be > ans

[jira] [Created] (FLINK-10488) Add DISTINCT operator for streaming tables that leverages time attributes

2018-10-04 Thread Fabian Hueske (JIRA)
Fabian Hueske created FLINK-10488: - Summary: Add DISTINCT operator for streaming tables that leverages time attributes Key: FLINK-10488 URL: https://issues.apache.org/jira/browse/FLINK-10488 Project

Re: [DISCUSS] Move hadoop 2.4 test profiles to cron jobs

2018-10-02 Thread Fabian Hueske
+1 Am Di., 2. Okt. 2018 um 14:50 Uhr schrieb Till Rohrmann < trohrm...@apache.org>: > Great idea Chesnay. +1 for running Hadoop 2.4 in a cron job. This will help > us to cut down our Travis time by almost 2. > > Cheers, > Till > > On Tue, Oct 2, 2018 at 1:49 PM Chesnay Schepler > wrote: > > > He

Re: [DISCUSS] Improvements to the Unified SQL Connector API

2018-10-02 Thread Fabian Hueske
Thanks for the proposal Timo! I've done a pass and added some comments (mostly asking for clarification, details). Overall, this is going into a very good direction. I think the tables which are stored in different systems and using a format definition to define other formats require some more dis

Re: [DISCUSS] FLIP-23 Model Serving

2018-10-01 Thread Fabian Hueske
Am Mo., 5. Feb. 2018 um 13:11 Uhr schrieb Stavros Kontopoulos < st.kontopou...@gmail.com>: > Thanx @Fabian. I will update the document accordingly wrt metrics. > I agree there are pros and cons. > > Best, > Stavros > > > On Wed, Jan 31, 2018 at 1:07 AM, Fabian Hueske w

[jira] [Created] (FLINK-10474) Don't translate IN to JOIN with VALUES for streaming queries

2018-10-01 Thread Fabian Hueske (JIRA)
Fabian Hueske created FLINK-10474: - Summary: Don't translate IN to JOIN with VALUES for streaming queries Key: FLINK-10474 URL: https://issues.apache.org/jira/browse/FLINK-10474 Project:

Re: [DISCUSS] Dropping flink-storm?

2018-10-01 Thread Fabian Hueske
+1 to drop it. Thanks, Fabian Am Sa., 29. Sep. 2018 um 12:05 Uhr schrieb Niels Basjes : > I would drop it. > > Niels Basjes > > On Sat, 29 Sep 2018, 10:38 Kostas Kloudas, > wrote: > > > +1 to drop it as nobody seems to be willing to maintain it and it also > > stands in the way for future deve

Re: [DISCUSS] [Contributing] (2) - Review Steps

2018-09-25 Thread Fabian Hueske
Hi, I think questions about Flink should be posted on the public mailing lists instead of asking just a single expert. There's many reasons for that: * usually more than one person can answer the question (what if the expert is not available?) * non-committers can join the discussion and contribu

Re: [DISCUSS] [Contributing] (3) - Review Tooling

2018-09-24 Thread Fabian Hueske
Hi, Coming back to the original topic of the thread: How to implement the guided review process. I am in favor of starting with a low-tech solution. We design a review template with a checkbox for the five questions (see [1]) and a link to the detailed description of the review process ([1] will

Re: [DISCUSS] [Contributing] (2) - Review Steps

2018-09-24 Thread Fabian Hueske
Thanks for separating the threads Stephan! (1) Do we agree on the five basic steps below?* +1 to the five steps and making the third question in the proposal the first. (2) How do we understand that consensus is reached about adding the feature? +1 to lazy consensus with one committer's +1 (3) T

Re: [VOTE] Release 1.5.4, release candidate #1

2018-09-19 Thread Fabian Hueske
+1 binding * I checked the diffs and did not find any added dependencies or updated dependency versions. * I checked the sha hash and signatures of all release artifacts. Best, Fabian 2018-09-15 23:26 GMT+02:00 Till Rohrmann : > Hi everyone, > Please review and vote on the release candidate #1

Re: [VOTE] Release 1.6.1, release candidate #1

2018-09-19 Thread Fabian Hueske
+1 binding * I checked the diffs and did not find any added dependencies or updated dependency versions. * I checked the sha hash and signatures of all release artifacts. Best, Fabian 2018-09-19 11:43 GMT+02:00 Gary Yao : > +1 (non-binding) > > Ran test suite from the flink-jepsen project on AW

Re: [PROPOSAL] [community] A more structured approach to reviews and contributions

2018-09-19 Thread Fabian Hueske
Hi, I'd like to suggest that we keep this thread focused on discussing Stephan's proposal, i.e., introducing a structured PR review process. Tison and Piotr raised some good points related to PR reviews that are definitely worth discussing, but I think we should do that on different threads and mo

[jira] [Created] (FLINK-10359) Scala example in DataSet docs is broken

2018-09-17 Thread Fabian Hueske (JIRA)
Fabian Hueske created FLINK-10359: - Summary: Scala example in DataSet docs is broken Key: FLINK-10359 URL: https://issues.apache.org/jira/browse/FLINK-10359 Project: Flink Issue Type: Bug

Re: CEP & checkpoints/savepoints

2018-09-17 Thread Fabian Hueske
Hi Ron, The CEP library is built on top of the DataStream / ProcessFunction API and holds all necessary state (the state of the pattern matching state machine) in regular keyed MapState. Hence, CEP does not require a dedicated configuration for checkpoints and savepoints, besides the regular appli

Re: [ANNOUNCE] New committer Gary Yao

2018-09-07 Thread Fabian Hueske
Congratulations Gary! 2018-09-07 16:29 GMT+02:00 Thomas Weise : > Congrats, Gary! > > On Fri, Sep 7, 2018 at 4:17 PM Dawid Wysakowicz > wrote: > > > Congratulations Gary! Well deserved! > > > > On 07/09/18 16:00, zhangmingleihe wrote: > > > Congrats Gary! > > > > > > Cheers > > > Minglei > > > >

[jira] [Created] (FLINK-10263) User-defined function with LITERAL paramters yields CompileException

2018-08-30 Thread Fabian Hueske (JIRA)
Fabian Hueske created FLINK-10263: - Summary: User-defined function with LITERAL paramters yields CompileException Key: FLINK-10263 URL: https://issues.apache.org/jira/browse/FLINK-10263 Project

[jira] [Created] (FLINK-10259) Key validation for GroupWindowAggregate is broken

2018-08-30 Thread Fabian Hueske (JIRA)
Fabian Hueske created FLINK-10259: - Summary: Key validation for GroupWindowAggregate is broken Key: FLINK-10259 URL: https://issues.apache.org/jira/browse/FLINK-10259 Project: Flink Issue

Re: [DISCUSS][TABLE] How to handle empty delete for UpsertSource

2018-08-29 Thread Fabian Hueske
ubtask about UpsertSink later. > > Thanks again for all the suggestions. It really helps me a lot. > Best, Hequn. > > > On Tue, Aug 28, 2018 at 9:47 PM Fabian Hueske wrote: > > > Hi Hequn, hi Piotr, > > > > Thanks for pushing this discussion forward and sorry for n

Re: [DISCUSS][TABLE] How to handle empty delete for UpsertSource

2018-08-28 Thread Fabian Hueske
here because user started Flink in a middle of a > stream, the result is still undefined (the join case that you mentioned), > since Flink could skip or ingest extra any number of messages (deletes or > not).* > > I think the result is clear if we clearly define that the upsert sourc

Re: [DISCUSS] Remove the slides under "Community & Project Info"

2018-08-27 Thread Fabian Hueske
I agree to remove the slides section. A lot of the content is out-dated and hence not only useless but might sometimes even cause confusion. Best, Fabian Am Mo., 27. Aug. 2018 um 08:29 Uhr schrieb Renjie Liu < liurenjie2...@gmail.com>: > Hi, Stephan: > Can we put project wiki in some place? I

[jira] [Created] (FLINK-10192) SQL Client table visualization mode does not update correctly

2018-08-21 Thread Fabian Hueske (JIRA)
Fabian Hueske created FLINK-10192: - Summary: SQL Client table visualization mode does not update correctly Key: FLINK-10192 URL: https://issues.apache.org/jira/browse/FLINK-10192 Project: Flink

Re: [DISCUSS][TABLE] How to handle empty delete for UpsertSource

2018-08-21 Thread Fabian Hueske
Hi, Thanks fort starting this discussion Hequn! These are a tricky questions. 1) Handling empty deletes in UpsertSource. I think forwarding empty deletes would only have a semantic difference if the output is persisted in a non-empty external table, e.g., a Cassandra table with entries. If we wou

Re: SQL Client Limitations

2018-08-21 Thread Fabian Hueske
Hi Dominik, The SQL Client supports the same subset of SQL that you get with Java / Scala embedded queries. The documentation [1] covers all supported operations. There are some limitations because certain operators require special time attributes (row time or processing time attributes) which ar

Re: [DISCUSS] Rust add adapter for parquet

2018-08-20 Thread Fabian Hueske
Hi Renjie, Did you intend to send this mail to dev@arrow.a.o instead of dev@flink.a.o? Best, Fabian 2018-08-20 4:39 GMT+02:00 Renjie Liu : > cc:Sunchao and Any > > -- Forwarded message - > From: Uwe L. Korn > Date: Sun, Aug 19, 2018 at 5:08 PM > Subject: Re: [DISCUSS] Rust add

Re: [DISCUSS] Improve Tutorials section of documentation

2018-08-15 Thread Fabian Hueske
02:00 Fabian Hueske : > Hi, > > Thanks for the feedback. > I've had a look at the structure that was derived from my initial message > and adjusted it to my current plans (see below). > > The main ideas are: > * add a new Tutorials section and move all existing tutor

[jira] [Created] (FLINK-10156) Drop the Table.writeToSink() method

2018-08-15 Thread Fabian Hueske (JIRA)
Fabian Hueske created FLINK-10156: - Summary: Drop the Table.writeToSink() method Key: FLINK-10156 URL: https://issues.apache.org/jira/browse/FLINK-10156 Project: Flink Issue Type

[jira] [Created] (FLINK-10153) Add tutorial section to documentation

2018-08-15 Thread Fabian Hueske (JIRA)
Fabian Hueske created FLINK-10153: - Summary: Add tutorial section to documentation Key: FLINK-10153 URL: https://issues.apache.org/jira/browse/FLINK-10153 Project: Flink Issue Type

Re: [Discuss] Outer join support and timestamp assignment for IntervalJoin

2018-08-14 Thread Fabian Hueske
Thanks for starting the discussion Florian. I'm also in favor of both A options. Option A for the outer joins is also is closest to the join syntax of the DataSet API. Thanks, Fabian 2018-08-13 20:50 GMT+02:00 Elias Levy : > As a developer, while not quite a succinct, I feel that option A in

Re: [DISCUSS] Improve Tutorials section of documentation

2018-08-10 Thread Fabian Hueske
; - Depolyment & Operations > > > - ... > > > - Debugging & Monitoring > > > - ... > > > > > > - Internals > > > - ... > > > ``` > > > > > > Aljoscha Krettek 于2018年8月9日周四 下午11:29写道: > > > > &g

[DISCUSS] Improve Tutorials section of documentation

2018-08-09 Thread Fabian Hueske
Hi everyone, I'd like to discuss a proposal to improve the tutorials / quickstart guides of Flink's documentation. I think the current tutorials have a few issues that should be fix in order to help our (future) users getting started with Flink. I propose to add a single "Tutorials" section to th

[jira] [Created] (FLINK-10100) Optimizer pushes partitioning past Null-Filter

2018-08-08 Thread Fabian Hueske (JIRA)
Fabian Hueske created FLINK-10100: - Summary: Optimizer pushes partitioning past Null-Filter Key: FLINK-10100 URL: https://issues.apache.org/jira/browse/FLINK-10100 Project: Flink Issue Type

Re: [apache/flink-web] One of your dependencies may have a security vulnerability

2018-08-02 Thread Fabian Hueske
I've filed this under FLINK-10007 [1]. Cheers, Fabian [1] https://issues.apache.org/jira/browse/FLINK-10007 2018-08-02 11:10 GMT+02:00 Ufuk Celebi : > We fixed this for the Flink docs a while back in > https://github.com/apache/flink/pull/5395, but didn't think of the > flink-web repo which use

[jira] [Created] (FLINK-10007) Security vulnerability in website build infrastructure

2018-07-31 Thread Fabian Hueske (JIRA)
Fabian Hueske created FLINK-10007: - Summary: Security vulnerability in website build infrastructure Key: FLINK-10007 URL: https://issues.apache.org/jira/browse/FLINK-10007 Project: Flink

Re: [RESULT][VOTE] Release 1.5.2, release candidate #2

2018-07-31 Thread Fabian Hueske
Thanks Chesnay! 2018-07-31 10:59 GMT+02:00 vino yang : > Thanks for releasing Flink 1.5.2, Chesnay! > > Thanks. > Vino. > > 2018-07-31 16:49 GMT+08:00 Till Rohrmann : > > > Thanks Chesnay for being our release manager and thanks to the community > > for all the work! > > > > Cheers, > > Till > >

Re: Consuming data from dynamoDB streams to flink

2018-07-31 Thread Fabian Hueske
t; > > > https://issues.apache.org/jira/browse/FLINK-4582 > > > > > > On Mon, Jul 30, 2018 at 1:25 AM Fabian Hueske wrote: > > > > > Hi Ying, > > > > > > Thanks for considering to contribute the connector! > > > > > >

Re: Questions about join and exactly-once

2018-07-31 Thread Fabian Hueske
Hi Minglei, 1. Not sure if you are asking for a specific problem, but IMO the main challenge is that there are many different ways (and meanings) to join two streams. The required semantics always depend on the concrete use case. If you want to perform an simple equality join with SQL semantics, y

Re: Consuming data from dynamoDB streams to flink

2018-07-30 Thread Fabian Hueske
re of. In particular, I am wondering > >> what's > >> > the *"resharding > >> > > behavior"* mentioned in FLINK-4582. > >> > > > >> > > Thanks a lot! > >> > > > >> > > - > >> > >

Re: Intro

2018-07-25 Thread Fabian Hueske
Hi Kevin, Welcome to the Flink community! The documentation is located in a folder in the regular code repository [1] and written in Markdown format. You can contribute to the documentation by opening pull requests against the repository. The contribution guidelines give a bit more info on the pr

[ANNOUNCE] Program for Flink Forward Berlin 2018 has been announced

2018-07-17 Thread Fabian Hueske
Hi everyone, I'd like to announce the program for Flink Forward Berlin 2018. The program committee [1] assembled a program of about 50 talks on use cases, operations, ecosystem, tech deep dive, and research topics. The conference will host speakers from Airbnb, Amazon, Google, ING, Lyft, Microsof

Re: Flink memory management in table api

2018-07-04 Thread Fabian Hueske
ording to above conversation flink will persist state forever for non > windowed operations. I want to know how flink persiat the state i.e. > Database or file system or in memory etc. > > On Wed, 4 Jul 2018 at 2:12 PM, Fabian Hueske wrote: > >> Hi Amol, >> >> The memory

Re: Flink memory management in table api

2018-07-04 Thread Fabian Hueske
Hi Amol, The memory consumption depends on the query/operation that you are doing. Time-based operations like group-window-aggregations, over-window-aggregations, or window-joins can automatically clean up their state once data is not no longer needed. Operations such as non-windowed aggregations

Re: Kafka Avro Table Source

2018-07-03 Thread Fabian Hueske
Hi Will, The community is currently working on improving the Kafka Avro integration for Flink SQL. There's a PR [1]. If you like, you could try it out and give some feedback. Timo (in CC) has been working Kafka Avro and should be able to help with any specific questions. Best, Fabian [1] https:

Re: [DISCUSS] Long-term goal of making flink-table Scala-free

2018-07-02 Thread Fabian Hueske
years > writing new code in Scala (and increasing technological debt), because > nobody have found a time to rewrite some non important and not actively > developed part of some module. > > > > Piotrek > > > >> On 14 Jun 2018, at 15:34, Fabian Hueske wrote: > &g

Re: Flink table api

2018-07-02 Thread Fabian Hueske
> > > > > *Office 103, 104, 1st Floor Pride Portal,Shivaji Housing Society, > > Bahiratwadi,Near Hotel JW Marriott, Off Senapati Bapat Road, Pune - > 411016, > > MH, INDIA.**Phone: +91 9689077510 | Skype: amols_iprogrammer* > > www.iprogrammer.com > > --

Re: Flink table api

2018-07-02 Thread Fabian Hueske
t. Ltd.* > > > > *Office 103, 104, 1st Floor Pride Portal,Shivaji Housing Society, > Bahiratwadi,Near Hotel JW Marriott, Off Senapati Bapat Road, Pune - 411016, > MH, INDIA.**Phone: +91 9689077510 | Skype: amols_iprogrammer* > www.iprogrammer.com > ------

Re: Flink table api

2018-07-02 Thread Fabian Hueske
Hi Amol, These are the requirements for POJOs [1] that are fully supported by Flink. Best, Fabian [1] https://ci.apache.org/projects/flink/flink-docs-release-1.5/dev/api_concepts.html#pojos 2018-07-02 12:19 GMT+02:00 Amol S - iProgrammer : > Hello Xingcan > > As mentioned in above mail thread

Re: Flink table api

2018-07-02 Thread Fabian Hueske
Hi, It looks like the type of master is not known to Flink. What's the output of customerMISMaster.printSchema(); ? Best, Fabian 2018-07-02 11:33 GMT+02:00 Amol S - iProgrammer : > Hello Xingcan > > DataStream streamSource = env > .addSource(kafkaConsumer) > .setParallelism(

[jira] [Created] (FLINK-9673) Improve State efficiency of bounded OVER window operators

2018-06-27 Thread Fabian Hueske (JIRA)
Fabian Hueske created FLINK-9673: Summary: Improve State efficiency of bounded OVER window operators Key: FLINK-9673 URL: https://issues.apache.org/jira/browse/FLINK-9673 Project: Flink

Re: Consuming data from dynamoDB streams to flink

2018-06-27 Thread Fabian Hueske
Hi Ying, I'm not aware of any effort for this issue. You could check with the assigned contributor in Jira if there is some previous work. Best, Fabian 2018-06-26 9:46 GMT+02:00 Ying Xu : > Hello Flink dev: > > We have a number of use cases which involves pulling data from DynamoDB > streams in

Re: [Flink-9407] Question about proposed ORC Sink !

2018-06-27 Thread Fabian Hueske
Hi Sagar, That's more a question for the ORC community, but AFAIK, the top-level type is always a struct because it needs to wrap the fields, e.g., struct(name:string, age:int) Best, Fabian 2018-06-26 22:38 GMT+02:00 sagar loke : > @zhangminglei, > > Question about the schema for ORC format: >

Re: [DISCUSS] Adding new interfaces in [Stream]ExecutionEnvironment

2018-06-26 Thread Fabian Hueske
will work and approach 2) will be more complicated, but with some nice > > benefit as stated above. However, the implementation choice should be > > transparent to the end user. Also, I am wondering outside of the SQL DDL, > > will these new functionality/interface be helpful in othe

Re: [ANNOUNCE] New committer Piotr Nowojski

2018-06-25 Thread Fabian Hueske
Congratulations Piotr! Good to have you on board :-) Cheers, Fabian 2018-06-23 19:25 GMT+02:00 Ufuk Celebi : > Congrats and welcome Piotr! :-) > > – Ufuk > > > On Sat, Jun 23, 2018 at 3:54 AM, zhangminglei <18717838...@163.com> wrote: > > Congrats Piotr! > > > > Cheers > > Minglei > >> 在 2018年6

Re: [ANNOUNCE] New committer: Sihua Zhou

2018-06-22 Thread Fabian Hueske
Congratulations! :-) Cheers, Fabian 2018-06-22 16:07 GMT+02:00 Stefan Richter : > Congratulations Sihua! That is great news. > > Best, > Stefan > > > Am 22.06.2018 um 15:17 schrieb Till Rohrmann : > > > > Hi everybody, > > > > On behalf of the PMC I am delighted to announce Sihua Zhou as a new F

Re: [WEBSITE] Proposal to rework the Flink website

2018-06-21 Thread Fabian Hueske
Hi, I've merged the proposed changes for the website. As usual, we can incrementally refine and improve it. Best, Fabian 2018-06-15 16:16 GMT+02:00 Fabian Hueske : > Hi, > > I'm planning to put the reworked website online next week. > You can have a look at PR #109 [1] to

Re: DataStreamCalcRule$1802" grows beyond 64 KB when execute long sql.

2018-06-19 Thread Fabian Hueske
nt. Is it hard to implement ? I am a new to flink table api & sql. > > Best Minglei. > > 在 2018年6月19日,下午10:36,Fabian Hueske 写道: > > Hi, > > Which version are you using? We fixed a similar issue for Flink 1.5.0. > If you can't upgrade yet, you can also implement

Re: DataStreamCalcRule$1802" grows beyond 64 KB when execute long sql.

2018-06-19 Thread Fabian Hueske
Hi, Which version are you using? We fixed a similar issue for Flink 1.5.0. If you can't upgrade yet, you can also implement a user-defined function that evaluates the big CASE WHEN statement. Best, Fabian 2018-06-19 16:27 GMT+02:00 zhangminglei <18717838...@163.com>: > Hi, friends. > > When I e

Re: [WEBSITE] Proposal to rework the Flink website

2018-06-15 Thread Fabian Hueske
[2] https://issues.apache.org/jira/browse/FLINK-9541 2018-06-06 12:57 GMT+02:00 Fabian Hueske : > Thanks for the feedback so far. > > +1 for adding a sitemap.xml and robots.txt to the website. > > I think we can make this a separate issue. I'll create a JIRA for that. > > Any ot

[ANNOUNCE] Registration for Flink Forward Berlin is open

2018-06-14 Thread Fabian Hueske
Hi everyone, *Flink Forward Berlin 2018 will take place from September 3rd to 5th.* The conference will start with one day of training and continue with two days of keynotes and talks. *The registration for Flink Forward Berlin 2018 is now open!* A limited amount of early-bird passes is available

Re: [DISCUSS] Long-term goal of making flink-table Scala-free

2018-06-14 Thread Fabian Hueske
Hi, In general, I think this is a good effort. However, it won't be easy and I think we have to plan this well. I don't like the idea of having the whole code base fragmented into Java and Scala code for too long. I think we should do this one step at a time and focus on migrating one module at a

Re: [PROPOSAL] Introduce Elastic Bloom Filter For Flink

2018-06-12 Thread Fabian Hueske
cientbloomfilters-jea.pdf> by > F. Putze, P. Sanders, and J. Singler. See section 4. > > > > We should discuss which exact implementation of bloom filters are the best > fit. > @Fabian: There are also implementations of bloom filters that use counting > and therefore supp

Re: [Proposal] Code Reviewer List

2018-06-09 Thread Fabian Hueske
Hi Yazdan, First of all, thank you for contributing to Flink! As Chesnay said, many of the Flink committers try to review pull requests in a timely manner. However, everybody has to put time aside to review a PR. Many committers are either working on features themselves or review PRs in their fre

[jira] [Created] (FLINK-9541) Add robots.txt and sitemap.xml to Flink website

2018-06-06 Thread Fabian Hueske (JIRA)
Fabian Hueske created FLINK-9541: Summary: Add robots.txt and sitemap.xml to Flink website Key: FLINK-9541 URL: https://issues.apache.org/jira/browse/FLINK-9541 Project: Flink Issue Type

Re: [WEBSITE] Proposal to rework the Flink website

2018-06-06 Thread Fabian Hueske
gt; > Thoughts? > > > > — Ken > > > >> On Jun 5, 2018, at 9:46 AM, Stephan Ewen wrote: > >> > >> Big +1 to this! > >> > >> I would like to contribute to this effort and help strengthen the way > Flink > >> presents itself

[WEBSITE] Proposal to rework the Flink website

2018-06-05 Thread Fabian Hueske
Hi everybody, I've opened a PR [1] that reworks parts of the Flink website ( flink.apache.org). My goal is to improve the structure of the website and provide more valuable information about the project and the community. A visitor (who doesn't know Flink yet) should be able to easily find answe

Re: [TABLE][SQL] Unify UniqueKeyExtractor and DataStreamRetractionRules

2018-06-05 Thread Fabian Hueske
Hi, I think the proposed refactoring is a good idea. It should simplify the logic to determine which update mode to use. We could also try to make some of the method and field names more intuitive and extend the internal documentation a bit. @Hequn, It would be good to get your thoughts on this i

[jira] [Created] (FLINK-9528) Incorrect results: Filter does not treat Upsert messages correctly.

2018-06-05 Thread Fabian Hueske (JIRA)
Fabian Hueske created FLINK-9528: Summary: Incorrect results: Filter does not treat Upsert messages correctly. Key: FLINK-9528 URL: https://issues.apache.org/jira/browse/FLINK-9528 Project: Flink

Re: [DISCUSS] Adding new interfaces in [Stream]ExecutionEnvironment

2018-06-05 Thread Fabian Hueske
ssing details here regarding the "distribution of > > external > > > libraries in runtime" part. Maybe you can share more example of this > use > > > case. Would this be included in the design doc @Timo? > > > > > > -- > > >

[jira] [Created] (FLINK-9522) Rework Flink website

2018-06-04 Thread Fabian Hueske (JIRA)
Fabian Hueske created FLINK-9522: Summary: Rework Flink website Key: FLINK-9522 URL: https://issues.apache.org/jira/browse/FLINK-9522 Project: Flink Issue Type: Improvement

Re: [Discuss] Proposing FLIP-25 - Support User State TTL Natively in Flink

2018-06-04 Thread Fabian Hueske
uncomfortable when the state backend is based on RocksDB, because the > timers are living together with the other states in the same RocksDB > instance, which means that with using TTL, the amount of the records in > RocksDB has to be doubled, I'm afraid this may hurt its performance.

[ANNOUNCE] Flink Forward Berlin 2018 - Call for Presentations extended until June 11

2018-06-01 Thread Fabian Hueske
Hi everybody, Due to popular demand, we've extended the Call for Presentations for Flink Forward Berlin 2018 by one week. The call will close on *Monday, June 11* (11:59pm CEST). Please submit a proposal to present your Flink and Stream Processing use case, experiences, and best practices in Berl

<    1   2   3   4   5   6   7   8   9   10   >