Re: [Discuss] FLIP-43: Savepoint Connector

2019-05-30 Thread Steven Wu
this is an awesome feature. > The name "Savepoint Connector" might indeed be not that good, as it doesn't point out the fact that with the current design, all kinds of snapshots (savepoint / full or incremental checkpoints) can be read. @Gordon can you add the above clarification to the FLIP

[jira] [Created] (FLINK-12686) StreamGraph Supports Blink Batch Mode

2019-05-30 Thread Biao Liu (JIRA)
Biao Liu created FLINK-12686: Summary: StreamGraph Supports Blink Batch Mode Key: FLINK-12686 URL: https://issues.apache.org/jira/browse/FLINK-12686 Project: Flink Issue Type: Improvement

[ANNOUNCE] Apache Flink-shaded 7.0 released

2019-05-30 Thread jincheng sun
Hi all, The Apache Flink community is very happy to announce the release of Apache Flink-shaded 7.0. The flink-shaded project contains a number of shaded dependencies for Apache Flink. Apache Flink® is an open-source stream processing framework for distributed, high-performing,

Re: Flink internals

2019-05-30 Thread Rong Rong
Hi Fan, John, The flink internal link[1] seems to be not updated in the last years. I found out some higher level pages in the official documentation here[2] however they are still very limited. Is there any other well maintained, internal documentations for contributors? I think it might be a

[jira] [Created] (FLINK-12688) Make serializer lazy initialization thread safe in StateDescriptor

2019-05-30 Thread Yu Li (JIRA)
Yu Li created FLINK-12688: - Summary: Make serializer lazy initialization thread safe in StateDescriptor Key: FLINK-12688 URL: https://issues.apache.org/jira/browse/FLINK-12688 Project: Flink Issue

[jira] [Created] (FLINK-12687) ByteHashSet is always in dense mode

2019-05-30 Thread Liya Fan (JIRA)
Liya Fan created FLINK-12687: Summary: ByteHashSet is always in dense mode Key: FLINK-12687 URL: https://issues.apache.org/jira/browse/FLINK-12687 Project: Flink Issue Type: Improvement

[jira] [Created] (FLINK-12685) Supports UNNEST query in blink planner

2019-05-30 Thread godfrey he (JIRA)
godfrey he created FLINK-12685: -- Summary: Supports UNNEST query in blink planner Key: FLINK-12685 URL: https://issues.apache.org/jira/browse/FLINK-12685 Project: Flink Issue Type: New Feature

[jira] [Created] (FLINK-12684) User defined configuration for per job clusters

2019-05-30 Thread Richard Moorhead (JIRA)
Richard Moorhead created FLINK-12684: Summary: User defined configuration for per job clusters Key: FLINK-12684 URL: https://issues.apache.org/jira/browse/FLINK-12684 Project: Flink

Re: [Discuss] FLIP-43: Savepoint Connector

2019-05-30 Thread Tzu-Li (Gordon) Tai
The name "Savepoint Connector" might indeed be not that good, as it doesn't point out the fact that with the current design, all kinds of snapshots (savepoint / full or incremental checkpoints) can be read. @Paul That would be a very valid requirement. Querying the list of existing operator ids

[jira] [Created] (FLINK-12682) StringWriter support custom row delimiter

2019-05-30 Thread Xiaowei Wang (JIRA)
Xiaowei Wang created FLINK-12682: Summary: StringWriter support custom row delimiter Key: FLINK-12682 URL: https://issues.apache.org/jira/browse/FLINK-12682 Project: Flink Issue Type: New

Re: [DISCUSS] Proposal to support disk spilling in HeapKeyedStateBackend

2019-05-30 Thread Yu Li
Thanks all for the feedbacks here. And thanks for the reminder about the plan of reworking abstractions of snapshot strategies @Gordon, will definitely watch it and make sure of the collaboration of the two works. Since this discussion thread has been open for some time and we all get a

[jira] [Created] (FLINK-12683) Provide task manager's location information for checkpoint acknowledge message to improve the checkpoint log message

2019-05-30 Thread vinoyang (JIRA)
vinoyang created FLINK-12683: Summary: Provide task manager's location information for checkpoint acknowledge message to improve the checkpoint log message Key: FLINK-12683 URL:

Re: [DISCUSS] FLIP-33: Standardize connector metrics

2019-05-30 Thread Becket Qin
Hi all, Thanks a lot for all the feedback and comments. I'd like to continue the discussion on this FLIP. I updated the FLIP-33 wiki to remove all the histogram metrics from the first version of metric convention due to the performance concern. The plan is to introduce them later when we have a

Re: [Discuss] FLIP-43: Savepoint Connector

2019-05-30 Thread Louis
+1 from my size. I think it will be a good feature. Best -- Louis Email:xu_soft39211...@163.com > On 30 May 2019, at 15:57, Tzu-Li (Gordon) Tai wrote: > > The name "Savepoint Connector" might indeed be not that good, as it doesn't > point out the fact that with the current design, all kinds

Re: [DISCUSS] FLIP-33: Standardize connector metrics

2019-05-30 Thread Becket Qin
Hi Piotr, Thanks a lot for the feedback. 1a) I guess you are referring to the part that "original system specific metrics should also be reported". The performance impact depends on the implementation. An efficient implementation would only record the metric once, but report them with two

Flink internals

2019-05-30 Thread John Tipper
Hi all, Is there a guide somewhere for the internals of Flink for developers wanting to get involved in core development? I'm particularly interested in any notes on how the codebase is put together so that it's possible to learn how it works internally. I'm particularly interested in what

Re: [VOTE] Release flink-shaded 7.0, release candidate 3

2019-05-30 Thread jincheng sun
Hi, All Thanks to everyone for checking the release and voting! I’ll tally the results in a separate email. :) Best, Jincheng jincheng sun 于2019年5月29日周三 下午4:52写道: > > +1 (non-binding) > > @Nico, Great thanks for your detail check! > @Chesnay Great thanks for your explanation! > > I also do

[RESULT] [VOTE] Release flink-shaded 7.0, release candidate #3

2019-05-30 Thread jincheng sun
Hi all, I'm happy to announce that we have unanimously approved this release. There are 6 approving votes, 3 of which are binding: * Chesnay * Timo * Hequn * Till * Nico * Jincheng There are no disapproving votes. Thanks everyone! Cheers, Jincheng

Re: [DISCUSS] Proposal to support disk spilling in HeapKeyedStateBackend

2019-05-30 Thread Tzu-Li (Gordon) Tai
Hi Yu, +1 to move forward with efforts to add this feature. As mentioned in the document as well as some offline discussions, from my side the only comments I have are related to how we snapshot the off-heap key groups. I think a recent discussion I posted about savepoint format unification for

Apply for contributor permission

2019-05-30 Thread Piyush Goyal
Hi, Could someone add me as a contributor? My JIRA username is pgoyal. Thanks!

Re: [Discuss] FLIP-43: Savepoint Connector

2019-05-30 Thread Paul Lam
Hi, @Gordon @Seth Thanks a lot for your inputs! In general, I agree with you. The metadata querying feature is a nice-to-have but not a must-have, and it’s reasonable to make it as a follow up since it requires some extra work. Best, Paul Lam > 在 2019年5月30日,19:22,Seth Wiesman 写道: > > @Paul

Re: Flink internals

2019-05-30 Thread Fan Liya
You can find some articles here: https://cwiki.apache.org/confluence/display/FLINK/Flink+Internals Best, Liya Fan On Thu, May 30, 2019 at 6:29 PM John Tipper wrote: > Hi all, > > Is there a guide somewhere for the internals of Flink for developers > wanting to get involved in core development?

Re: [DISCUSS] Releasing Flink 1.8.1

2019-05-30 Thread Jark Wu
Hi Jingcheng, Thanks for coordinating the work to release 1.8.1. +1 for 1.8.1 On Wed, 29 May 2019 at 19:48, Hequn Cheng wrote: > Hi Jincheng, > > Thanks for putting these together with a nice document. > +1 to release 1.8.1. I think it would be nice if we can have a new release > with so many

Re: [Discuss] FLIP-43: Savepoint Connector

2019-05-30 Thread Seth Wiesman
@Paul I agree with Gordon that those are useful features. The only thing I’d like to add is that I don’t believe listing operator ids will be useful to most users, they want to see UIDs which would also require changes to the Savepoint metadata file. I think that would be a good follow up but