Oh. I didn't know this. The last mail from Gyula sounded like the Streaming part is ready2release.
My personal goal was to prepare a release candiate today so that we have a reference point against which we can test (which would also mean forking of a 0.7-release branch and basically doing a feature freeze). I also though that it might be a good idea to have a release candidate by tomorrow because it could be of use for the Hackathon tomorrow in Stockholm. But since the Scala POJO changes and the streaming examples are both outstanding, we can probably scratch that. Doing a fork, release candidate and a feature freeze doesn't mean we can not supply bugfixes anymore for the 0.7-release ;) What is your best-case scenario for FLINK-1103? (as far as I can see the only outstanding streaming feature for the release) Best, Robert On Tue, Oct 7, 2014 at 7:07 PM, Márton Balassi <[email protected]> wrote: > As for the streaming side we would like to push some bugfix commits and the > resolution of the FLINK-1103 JIRA issue. > > These are more or less ready, hopefully will be available at the end of > this week. > > On Tue, Oct 7, 2014 at 6:50 PM, Robert Metzger <[email protected]> > wrote: > > > I've merged the record api deprecation already. > > I'll merge #141 once Aljoscha provided his Scala changes. > > > > We certainly should merge #136 and #143 as well. > > > > On Tue, Oct 7, 2014 at 5:20 PM, Fabian Hueske <[email protected]> > wrote: > > > > > So which PRs will be included in the candidate? > > > > > > #141 POJOs > > > #144 Deprecate Record API > > > #136 Fixed example quickstart > > > #143(?) Hadoop Compat: Documentation + Hadoop function wrappers > (includes > > > PR #131) > > > > > > 2014-10-07 16:26 GMT+02:00 Stephan Ewen <[email protected]>: > > > > > > > Great news, looking forward to seeing this in the master! > > > > > > > > On Tue, Oct 7, 2014 at 1:53 PM, Robert Metzger <[email protected]> > > > > wrote: > > > > > > > > > As an update for everyone: My POJO feature is finished, including > > > > > documentation. > > > > > Aljoscha is currently adopting the Scala API to have support for > > > (nested) > > > > > POJOs as well. > > > > > Once that is done, I'll merge everything and create a first > candidate > > > > that > > > > > we can use for testing. > > > > > > > > > > On Tue, Sep 30, 2014 at 10:49 PM, Fabian Hueske < > [email protected]> > > > > > wrote: > > > > > > > > > > > I just checked the "Run Example" quickstart and it needs a bit of > > > work. > > > > > > > > > > > > 2014-09-30 22:41 GMT+02:00 Robert Metzger <[email protected]>: > > > > > > > > > > > > > I'm working hard on getting the POJOs ready. > > > > > > > > > > > > > > We also should do a pass over our documentation, the > quickstarts > > > and > > > > > the > > > > > > > website to see if everything is in a good state (for example > the > > > > > > > collection-based execution needs documentation as well). We > > should > > > > also > > > > > > > finally document the hadoop-input format wrappers (I think Timo > > is > > > > > > working > > > > > > > on a pull request for that). > > > > > > > This page mentions the LocalDistributedExecutor and contains > some > > > > (most > > > > > > > likely outdated) scala code: > > > > > > > > > > > > > > > > > > > > > > > > > > > > http://flink.incubator.apache.org/docs/0.7-incubating/local_execution.html > > > > > > > > > > > > > > We also need to deprecate the old record api ( > > > > > > > https://issues.apache.org/jira/browse/FLINK-1106). > > > > > > > > > > > > > > On Tue, Sep 30, 2014 at 9:17 PM, Stephan Ewen < > [email protected]> > > > > > wrote: > > > > > > > > > > > > > > > I think we are approaching ready state. > > > > > > > > Last issues are going in and we started working on > dependencies > > > and > > > > > > test > > > > > > > > platform diversity in order to make stabilizing phase. > > > > > > > > > > > > > > > > We should have an official feature freeze soon and fork the > > > > > 0.7-release > > > > > > > > branch. I personally vote to include the POJO support (I > think > > > > Robert > > > > > > is > > > > > > > > sorting that one out and is close to completion), and I want > to > > > add > > > > > the > > > > > > > > collection-based execution (today or tomorrow). Till has the > > BLOB > > > > > > manager > > > > > > > > ready, which would be good to include (better support large > > > > libraries > > > > > > or > > > > > > > > fat jars). > > > > > > > > > > > > > > > > After that, I vote to freeze. > > > > > > > > > > > > > > > > On Tue, Sep 30, 2014 at 9:12 PM, Gyula Fora < > [email protected] > > > > > > > > wrote: > > > > > > > > > > > > > > > > > Hey, > > > > > > > > > > > > > > > > > > So what is the current decision regarding the time of the > > > > upcoming > > > > > > > > release? > > > > > > > > > > > > > > > > > > As for the streaming component, we included all the > features > > we > > > > > > wanted, > > > > > > > > we > > > > > > > > > will start to test everything tomorrow, making sure that > all > > > > works > > > > > as > > > > > > > > > intended. > > > > > > > > > We are also almost finished with cleaning up the connector > > > > > > dependencies > > > > > > > > > that Stephan pointed out, should be finished by tomorrow. > > > > > > > > > > > > > > > > > > Gyula > > > > > > > > > > > > > > > > > > On 26 Sep 2014, at 10:49, Fabian Hueske < > [email protected]> > > > > > wrote: > > > > > > > > > > > > > > > > > > > +1 to manage this on JIRA (if possible) > > > > > > > > > > > > > > > > > > > > 2014-09-26 10:40 GMT+02:00 Aljoscha Krettek < > > > > [email protected] > > > > > >: > > > > > > > > > > > > > > > > > > > >> Can we not manage this stuff on Jira? > > > > > > > > > >> > > > > > > > > > >> On Fri, Sep 26, 2014 at 10:16 AM, Stephan Ewen < > > > > > [email protected]> > > > > > > > > > wrote: > > > > > > > > > >>> I personally like the idea of SOFT time-based feature > > > > freezes. > > > > > > > > > Otherwise, > > > > > > > > > >>> releases will get delayed again and again, because of > > > > features > > > > > > that > > > > > > > > we > > > > > > > > > >> try > > > > > > > > > >>> to squeeze in. > > > > > > > > > >>> > > > > > > > > > >>> Having reached the freeze point already, we could still > > > > include > > > > > > the > > > > > > > > > >>> features that are pending ready state in the next days > > > > > > (streaming, > > > > > > > > blob > > > > > > > > > >>> Manager, POJOs), but otherwise head for a release > state. > > > > > > > > > >>> > > > > > > > > > >>> We had a mail listing the issues to include into 0.7, > > but a > > > > > wiki > > > > > > > page > > > > > > > > > >> would > > > > > > > > > >>> probably be better. In that sense, we could start > > > collecting > > > > > the > > > > > > > > issues > > > > > > > > > >> for > > > > > > > > > >>> the next release from now on. > > > > > > > > > >>> Am 26.09.2014 09:17 schrieb "Daniel Warneke" < > > > > > [email protected] > > > > > > >: > > > > > > > > > >>> > > > > > > > > > >>>> Hi, > > > > > > > > > >>>> > > > > > > > > > >>>> I like Fabian's idea. Is there a wiki page (or > something > > > > > > similar) > > > > > > > > > where > > > > > > > > > >> we > > > > > > > > > >>>> can collect the proposed JIRAs? > > > > > > > > > >>>> > > > > > > > > > >>>> Best regards, > > > > > > > > > >>>> > > > > > > > > > >>>> Daniel > > > > > > > > > >>>> > > > > > > > > > >>>> Am 24.09.2014 23:03, schrieb Fabian Hueske: > > > > > > > > > >>>> > > > > > > > > > >>>>> I agree, a hard feature stop deadline might not be > the > > > best > > > > > > > > practice. > > > > > > > > > >>>>> > > > > > > > > > >>>>> How about the following procedure: > > > > > > > > > >>>>> We decide two (or three) weeks before a targeted > > release > > > > date > > > > > > > about > > > > > > > > > >> which > > > > > > > > > >>>>> JIRAs to include. JIRAs that are selected for a > release > > > > > should > > > > > > be > > > > > > > > > >>>>> completed > > > > > > > > > >>>>> or really close to completion (via progress estimates > > in > > > > > JIRA). > > > > > > > > > >>>>> After we decided which JIRAs to include in a release, > > we > > > > can > > > > > > use > > > > > > > > JIRA > > > > > > > > > >> to > > > > > > > > > >>>>> track the progress and dedicate another week > > exclusively > > > > for > > > > > > > > testing > > > > > > > > > >> after > > > > > > > > > >>>>> the last feature was completed. > > > > > > > > > >>>>> > > > > > > > > > >>>>> > > > > > > > > > >>>>> 2014-09-24 19:10 GMT+02:00 Márton Balassi < > > > > > > > > [email protected] > > > > > > > > > >: > > > > > > > > > >>>>> > > > > > > > > > >>>>> As for the streaming team we're also getting ready > for > > > the > > > > > > > release, > > > > > > > > > >> but a > > > > > > > > > >>>>>> couple of days will be needed to finish the features > > > that > > > > we > > > > > > > would > > > > > > > > > >> like > > > > > > > > > >>>>>> to > > > > > > > > > >>>>>> include. > > > > > > > > > >>>>>> > > > > > > > > > >>>>>> - A little work is still needed for reduce > > operations > > > > and > > > > > > > > > >>>>>> groups/connected streams (any comment on Gyula's > > > recent > > > > > > > e-mail > > > > > > > > is > > > > > > > > > >>>>>> really > > > > > > > > > >>>>>> appreciated :) ) > > > > > > > > > >>>>>> - The examples are being updated to match the > > > standard, > > > > > > check > > > > > > > > out > > > > > > > > > >> the > > > > > > > > > >>>>>> WordCount. ( > > > > > > > > > >>>>>> > > > > > > > > > >>>>>> https://github.com/mbalassi/incubator-flink/blob/ > > > > > > > > > >>>>>> streaming-new/flink-addons/flink-streaming/flink- > > > > > > > > > >>>>>> streaming-examples/src/main/java/org/apache/flink/ > > > > > > > > > >>>>>> streaming/examples/wordcount/WordCount.java > > > > > > > > > >>>>>> ) > > > > > > > > > >>>>>> Hopefully it gives you some deja vu. :) > > > > > > > > > >>>>>> > > > > > > > > > >>>>>> > > > > > > > > > >>>>>> On Wed, Sep 24, 2014 at 6:53 PM, Ufuk Celebi < > > > > > [email protected]> > > > > > > > > > wrote: > > > > > > > > > >>>>>> > > > > > > > > > >>>>>> On 24 Sep 2014, at 18:37, Robert Metzger < > > > > > [email protected] > > > > > > > > > > > > > > > > >> wrote: > > > > > > > > > >>>>>>> > > > > > > > > > >>>>>>> Hey guys, > > > > > > > > > >>>>>>>> > > > > > > > > > >>>>>>>> exactly 3 weeks ago, we discussed to do a feature > > > freeze > > > > > for > > > > > > > the > > > > > > > > > >>>>>>>> 0.7-incubating release today. > > > > > > > > > >>>>>>>> > > > > > > > > > >>>>>>>> From our initial feature list: > > > > > > > > > >>>>>>>> - *Flink Streaming* "Beta Preview". I would > suggest > > to > > > > > ship > > > > > > > the > > > > > > > > > >>>>>>>> > > > > > > > > > >>>>>>> streaming, > > > > > > > > > >>>>>>> > > > > > > > > > >>>>>>>> but clearly mark it as a preview in the > > documentation. > > > > > > > > > >>>>>>>> -* Java API Pojo improvements*: Code generation, > key > > > > > > selection > > > > > > > > > >> using a > > > > > > > > > >>>>>>>> string-expression: > > > > > > > > > https://issues.apache.org/jira/browse/FLINK-1032 > > > > > > > > > >>>>>>>> - *Reworked Scala API*. Bring the Scala API in > sync > > > > with > > > > > > the > > > > > > > > > >> latest > > > > > > > > > >>>>>>>> developments in the Java API: > > > > > > > > > >>>>>>>> https://issues.apache.org/jira/browse/FLINK-641 > > > > > > > > > >>>>>>>> -* Akka-based RPC service*: > > > > > > > > > >>>>>>>> https://issues.apache.org/jira/browse/FLINK-1019 > > > > > > > > > >>>>>>>> - *Kryo-based serialization*. This feature has > been > > > > > > requested > > > > > > > > by > > > > > > > > > >> many > > > > > > > > > >>>>>>>> users. Mostly because they wanted to use > Collections > > > > > inside > > > > > > > > POJOs: > > > > > > > > > >>>>>>>> https://issues.apache.org/jira/browse/FLINK-610 > > > > > > > > > >>>>>>>> - Rework JobManager internals to support > incremental > > > > > program > > > > > > > > > >> rollout & > > > > > > > > > >>>>>>>> execution > > > > > > > > > >>>>>>>> - First parts of dynamic memory assignments > > > > > > > > > >>>>>>>> > > > > > > > > > >>>>>>>> The following features are in the master, as of > > today: > > > > > > > > > >>>>>>>> - *Flink Streaming* > > > > > > > > > >>>>>>>> - *Reworked Scala API* > > > > > > > > > >>>>>>>> -* New Scheduler* > > > > > > > > > >>>>>>>> > > > > > > > > > >>>>>>>> We certainly need some days to test everything > until > > > we > > > > > can > > > > > > > > start > > > > > > > > > >> the > > > > > > > > > >>>>>>>> > > > > > > > > > >>>>>>> vote. > > > > > > > > > >>>>>>> > > > > > > > > > >>>>>>>> Based on our experience with the last major > > release, I > > > > > would > > > > > > > > > really > > > > > > > > > >>>>>>>> > > > > > > > > > >>>>>>> like > > > > > > > > > >>>>>> > > > > > > > > > >>>>>>> to > > > > > > > > > >>>>>>> > > > > > > > > > >>>>>>>> do the testing and bugfixing BEFORE the first > > release > > > > > > > candidate. > > > > > > > > > For > > > > > > > > > >>>>>>>> > > > > > > > > > >>>>>>> the > > > > > > > > > >>>>>> > > > > > > > > > >>>>>>> 0.6-incubating release, we had 6 candidates) > > > > > > > > > >>>>>>>> > > > > > > > > > >>>>>>>> How do you guys feel about this? Should we wait a > > few > > > > more > > > > > > > days > > > > > > > > > for > > > > > > > > > >> the > > > > > > > > > >>>>>>>> release so that a few more features make it into > the > > > > > > release? > > > > > > > > > >>>>>>>> > > > > > > > > > >>>>>>>> I'm undecided on this. On the one hand, its really > > > nice > > > > to > > > > > > > > release > > > > > > > > > >> on a > > > > > > > > > >>>>>>>> regular schedule, but it also eats up some time > and > > > > causes > > > > > > > > > overhead > > > > > > > > > >>>>>>>> (different branches etc.). > > > > > > > > > >>>>>>>> I would really like to have the Java API Pojo > > > > improvements > > > > > > in > > > > > > > > the > > > > > > > > > >>>>>>>> > > > > > > > > > >>>>>>> release. > > > > > > > > > >>>>>>> > > > > > > > > > >>>>>>>> I think I can finish it until end of this week. > > > > > > > > > >>>>>>>> > > > > > > > > > >>>>>>>> Opinions? > > > > > > > > > >>>>>>>> > > > > > > > > > >>>>>>> I agree that the finished features (especially the > > > Scala > > > > > API) > > > > > > > are > > > > > > > > > >> nice > > > > > > > > > >>>>>>> > > > > > > > > > >>>>>> for > > > > > > > > > >>>>>> > > > > > > > > > >>>>>>> a new release, but still I would like to wait a few > > > more > > > > > > days. > > > > > > > > > >>>>>>> > > > > > > > > > >>>>>>> Some of the missing features are on the brink of > > being > > > > > > finished > > > > > > > > > (e.g. > > > > > > > > > >>>>>>> the > > > > > > > > > >>>>>>> Pojo improvements). I wouldn't want to invest a > week > > in > > > > bug > > > > > > > > fixing > > > > > > > > > >> and > > > > > > > > > >>>>>>> doing the release vote, when the new features are > > > likely > > > > to > > > > > > be > > > > > > > > > >> finished > > > > > > > > > >>>>>>> just a few days afterwards. And the upcoming > features > > > > will > > > > > > > > > >> definitely be > > > > > > > > > >>>>>>> worth a release, so users can work with them. ;) > > > > > > > > > >>>>>>> > > > > > > > > > >>>>>> > > > > > > > > > >>>> > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
