Big +1 on this. Some of these topics are not only for contributors, but would also be super useful for advance users. One topic I can think of in addition is: Security/Kerberos.
Echo on Both Seth's idea, we could have both wiki and PR submission: As Robert mentioned - wiki submission would make the experience more frictionless. I was having concerns splitting the blog posts in two places, but I also think adding the banner/blog-series of "Flink Engine Room" would help readers distinct between the two. -- Rong On Tue, Mar 3, 2020 at 8:39 AM Dian Fu <dian0511...@gmail.com> wrote: > Big +1 on this idea. It will benefit both the developers and users a lot. > > Regarding to the place to hold these blogs, my preference is 3) as I > notice that there are already a few high quality blogs on flink web-site[1] > and I guess that may be a good place to start with. We just need to figure > out a way to let contributors clearly mark the audience of their articles > and also help users to easily determine whether the content is what they > want. > > Regards, > Dian > > [1] https://flink.apache.org/blog/ <https://flink.apache.org/blog/> > > 在 2020年3月3日，下午11:14，Yadong Xie <vthink...@gmail.com> 写道： > > > > Hi all > > > > maybe we can use markdown & GitHub to make the submission easy to review > > I have set up a similar blog for Flink-china blog before(deprecated), > glad > > to offer help if needed > > > > here is the link: https://github.com/flink-china/doc > > > > Seth Wiesman <sjwies...@gmail.com> 于2020年3月3日周二 下午10:51写道： > > > >> For lack of a better way to put this, I think the location depends on > the > >> level of effort you want to put into writing these articles. > >> > >> If they are informal design documents then I think the wiki is the way > to > >> go. > >> > >> If you want to have them be more polished then the existing blog. This > >> means going through a PR on the flink website, thinking about language, > >> etc. If we go this route we can distinguish them with a series title > like > >> "Flink Engine Room" and a disclaimer at the top. > >> > >> "Flink Engine Room: Plugins" > >> > >> "Flink Engine Room is a series of blog posts covering ongoing > development > >> on Apache Flink internals, why decisions were made, and how they will > >> impact future development. The information described in this post is not > >> required to successfully write and deploy Flink applications in > >> production." > >> > >> Seth > >> > >> > >> On Tue, Mar 3, 2020 at 8:29 AM Arvid Heise <ar...@ververica.com> wrote: > >> > >>> I think there is enough to positive to start setting it up. That begs > the > >>> question: in which format. > >>> > >>> Following possibilities exist: > >>> 1) Use wiki as Robert pointed out. > >>> 2) Add new blog. > >>> 3) Use existing blog and separate by tags #user, #expert, #dev (can be > >>> mixed). Start page could filter on #user by default. > >>> 4) ??? > >>> > >>> I'm assuming only few have a strong opinion, so I'd be happy if you'd > >> just > >>> drop your numbers in order of highest to lowest preference. > >>> > >>> On Tue, Mar 3, 2020 at 2:48 PM Piotr Nowojski <pi...@ververica.com> > >> wrote: > >>> > >>>> +1 for the idea :) And fully agree to clearly separate them. > >>>> > >>>> I think the original idea was writing about some recent changes in the > >>>> Flink’s code base, that could affect other Flink developers > >>>> (contributors/committers). Like for example some new ideas/future > >>>> directions that we want to follow. Especially if they are work in > >>> progress > >>>> and there is lots of old code not adhering to those new ideas. In some > >>>> later responses, it seemed like people are more thinking about > >> presenting > >>>> some more advanced features, like a deep tech dive for power users. > >>>> > >>>> I’m not opposing the deep tech dives, but I just wanted to note that > >> is a > >>>> different target audience. I think the dev blogs could cover both of > >>> them. > >>>> At least initially. Later on we can decide to put more emphasis on > >> power > >>>> users or Flink devs, or split them, or whatever. > >>>> > >>>> Piotrek > >>>> > >>>>> On 3 Mar 2020, at 12:37, Jingsong Li <jingsongl...@gmail.com> wrote: > >>>>> > >>>>> +1 for this proposal. I have a lot of desired topics in table and > >>> batch. > >>>>> > >>>>> I also second Seth and Stephan 's comment separate this in a clear > >> way. > >>>>> Have concerns that maybe easy to confuse new users. > >>>>> If I am a beginner and find a bunch of deep documents, I need to > >>> further > >>>>> distinguish which is effective and which is invalid for me, which may > >>>> cause > >>>>> me a lot of trouble. > >>>>> > >>>>> Best, > >>>>> Jingsong Lee > >>>>> > >>>>> On Tue, Mar 3, 2020 at 6:36 PM Flavio Pompermaier < > >>> pomperma...@okkam.it> > >>>>> wrote: > >>>>> > >>>>>> Big +1 from my side. I'd be very interested in what Jeff proposed, > >> in > >>>>>> particular everything related to client part (job submission, > >> workflow > >>>>>> management, callbacks on submission/success/failure, etc). > >>>>>> Something I can't find anywhere is also how to query Flink > >>>> states..would it > >>>>>> be possible to have something like the Presto UI [1]? Does Flink > >>>> implement > >>>>>> some sort of query queuing? I heard about a query proxy server but I > >>>> don't > >>>>>> know if there's a will to push in that direction. > >>>>>> For Stateful Functions it would be nice to deeply compare the taxi > >>>> driver > >>>>>> solution with a more common implementation (i.e. using a database to > >>>>>> persist the legal data..is it safe to keep them as a Flink state?). > >>>>>> [1] > >>>> https://www.tutorialspoint.com/apache_presto/images/web_interface.jpg > >>>>>> > >>>>>> Best, > >>>>>> Flavio > >>>>>> > >>>>>> On Tue, Mar 3, 2020 at 10:47 AM Jeff Zhang <zjf...@gmail.com> > >> wrote: > >>>>>> > >>>>>>> +1 for this proposal. I am preparing some articles for how to use > >>>> Flink > >>>>>> on > >>>>>>> Zeppelin, although it is not closely related with this topic, but > >>>> should > >>>>>> be > >>>>>>> helpful for users to get started with Flink. > >>>>>>> > >>>>>>> Till Rohrmann <trohrm...@apache.org> 于2020年3月3日周二 下午5:39写道： > >>>>>>> > >>>>>>>> I like the idea. +1 from my side. > >>>>>>>> > >>>>>>>> Potential topics: > >>>>>>>> - Scheduling > >>>>>>>> - Cluster partitions > >>>>>>>> - Memory configuration > >>>>>>>> - Recovery > >>>>>>>> > >>>>>>>> Cheers, > >>>>>>>> Till > >>>>>>>> > >>>>>>>> On Tue, Mar 3, 2020 at 3:56 AM Xintong Song < > >> tonysong...@gmail.com> > >>>>>>> wrote: > >>>>>>>> > >>>>>>>>> Big +1. Thanks for the idea, Arvid. > >>>>>>>>> > >>>>>>>>> I'd be excited to read such blogs. > >>>>>>>>> > >>>>>>>>> And we would also be happy to contribute some contents on the > >>> newest > >>>>>>>>> efforts from our team. > >>>>>>>>> Potential topics: > >>>>>>>>> - Memory configuration > >>>>>>>>> - Active Kubernetes integration > >>>>>>>>> - GPU support > >>>>>>>>> - Pluggable (dynamic) slot allocation > >>>>>>>>> > >>>>>>>>> Thank you~ > >>>>>>>>> > >>>>>>>>> Xintong Song > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> On Tue, Mar 3, 2020 at 9:59 AM Benchao Li <libenc...@gmail.com> > >>>>>> wrote: > >>>>>>>>> > >>>>>>>>>> +1 for this proposal. As a contributor, it would be very helpful > >>> to > >>>>>>>> have > >>>>>>>>>> such blogs for us to understand status and future of Flink. > >>>>>>>>>> > >>>>>>>>>> Robert Metzger <rmetz...@apache.org> 于2020年3月3日周二 上午6:00写道： > >>>>>>>>>> > >>>>>>>>>>> I would be excited to read such a blog (can I request topics? > >> :) > >>>>>> ) > >>>>>>>>>>> > >>>>>>>>>>> We could start very low key by using our wiki's blog feature: > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>> > >>>>>> > >>>> > >>> > >> > https://cwiki.apache.org/confluence/pages/viewrecentblogposts.action?key=FLINK > >>>>>>>>>>> > >>>>>>>>>>> On Mon, Mar 2, 2020 at 8:26 PM Stephan Ewen <se...@apache.org> > >>>>>>>> wrote: > >>>>>>>>>>> > >>>>>>>>>>>> Great idea, but I also second Seth's comment to separate this > >>>>>> in > >>>>>>> a > >>>>>>>>>> clear > >>>>>>>>>>>> way. It's easy to confuse new / potential users. > >>>>>>>>>>>> > >>>>>>>>>>>> On Mon, Mar 2, 2020 at 8:15 PM Seth Wiesman < > >>>>>> sjwies...@gmail.com > >>>>>>>> > >>>>>>>>>> wrote: > >>>>>>>>>>>> > >>>>>>>>>>>>> +1 on the idea. > >>>>>>>>>>>>> > >>>>>>>>>>>>> My only request would be they are clearly marked as being > >>>>>> about > >>>>>>>>>>>> internals / > >>>>>>>>>>>>> for advanced users to not give typical users the wrong > >>>>>>> impression > >>>>>>>>>> about > >>>>>>>>>>>> how > >>>>>>>>>>>>> much they need to understand to use Flink. Nico's network > >>>>>> stack > >>>>>>>>> blog > >>>>>>>>>>> post > >>>>>>>>>>>>> does this well[1]. > >>>>>>>>>>>>> > >>>>>>>>>>>>> Seth > >>>>>>>>>>>>> > >>>>>>>>>>>>> [1] > >>>>>>> https://flink.apache.org/2019/06/05/flink-network-stack.html > >>>>>>>>>>>>> > >>>>>>>>>>>>> On Mon, Mar 2, 2020 at 10:39 AM Ufuk Celebi <u...@apache.org> > >>>>>>>>> wrote: > >>>>>>>>>>>>> > >>>>>>>>>>>>>> I'd be happy to read such a blog. Big +1 as a potential > >>>>>>> reader. > >>>>>>>>> ;-) > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> – Ufuk > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> On Mon, Mar 2, 2020 at 11:53 AM Arvid Heise < > >>>>>>>> ar...@ververica.com > >>>>>>>>>> > >>>>>>>>>>>> wrote: > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Dear devs, > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> development speed of Flink has steadily increased. Lots > >>>>>> of > >>>>>>>> new > >>>>>>>>>>>> concepts > >>>>>>>>>>>>>> are > >>>>>>>>>>>>>>> introduced and technical debt removed. However, it's hard > >>>>>>> to > >>>>>>>>> keep > >>>>>>>>>>>> track > >>>>>>>>>>>>>> of > >>>>>>>>>>>>>>> these things if you are not directly involved. Especially > >>>>>>> for > >>>>>>>>> new > >>>>>>>>>>>>>>> contributors, it's often not easy to know what the best > >>>>>>>>> practices > >>>>>>>>>>> are > >>>>>>>>>>>>> or > >>>>>>>>>>>>>> if > >>>>>>>>>>>>>>> there are related work streams going on. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> In the runtime team, we had the idea to set up a dev blog > >>>>>>>> where > >>>>>>>>>> we > >>>>>>>>>>>>> could > >>>>>>>>>>>>>>> introduce newest developments. The scope should be expert > >>>>>>>> users > >>>>>>>>>>> that > >>>>>>>>>>>>>>> contribute to the project. Of course, some articles may > >>>>>>> have > >>>>>>>> a > >>>>>>>>>>>> broader > >>>>>>>>>>>>>>> scope and even be linked from release notes. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Examples from our team to give a more specific idea: > >>>>>>>>>>>>>>> * Deprecated checkpoint lock and mailbox model > >>>>>>>>>>>>>>> * Revised interface for two phase commit sinks and new > >>>>>> JDBC > >>>>>>>>> sink > >>>>>>>>>>>>>>> * N-ary input operators > >>>>>>>>>>>>>>> * Unaligned checkpoints > >>>>>>>>>>>>>>> * Operator factories > >>>>>>>>>>>>>>> * Plugins > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> These articles would be less formal than a FLIP (but > >>>>>> could > >>>>>>>> link > >>>>>>>>>> to > >>>>>>>>>>>> them > >>>>>>>>>>>>>> to > >>>>>>>>>>>>>>> avoid redundancy) and focus more on how other developers > >>>>>>> are > >>>>>>>>>>> actually > >>>>>>>>>>>>>>> impacted by the changes. It can also be used to share > >>>>>>>>> experiences > >>>>>>>>>>>>> during > >>>>>>>>>>>>>>> the implementation. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> We would like to ask the other teams if they see a > >>>>>> benefit > >>>>>>> of > >>>>>>>>>> such > >>>>>>>>>>> a > >>>>>>>>>>>>> blog > >>>>>>>>>>>>>>> and would like to contribute. Bonus points if you could > >>>>>>>> provide > >>>>>>>>>>>>> examples > >>>>>>>>>>>>>> of > >>>>>>>>>>>>>>> your topics. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Best, > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Arvid > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> -- > >>>>>>>>>> > >>>>>>>>>> Benchao Li > >>>>>>>>>> School of Electronics Engineering and Computer Science, Peking > >>>>>>>> University > >>>>>>>>>> Tel:+86-15650713730 > >>>>>>>>>> Email: libenc...@gmail.com; libenc...@pku.edu.cn > >>>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> -- > >>>>>>> Best Regards > >>>>>>> > >>>>>>> Jeff Zhang > >>>>>>> > >>>>>> > >>>>> > >>>>> > >>>>> -- > >>>>> Best, Jingsong Lee > >>>> > >>>> > >>> > >> > >
