We can add it and keep it initially undocumented (experimental) until further tests...
On Tue, Oct 7, 2014 at 10:40 PM, Robert Metzger <[email protected]> wrote: > I would add #142. > > On Tue, Oct 7, 2014 at 10:29 PM, Stephan Ewen <[email protected]> wrote: > > > I agree, having a release candidate out would be nice. > > > > What is your opinion on issue #142? Fault tolerance is inactive by > default, > > but can be activated through the configuration. > > > > On Tue, Oct 7, 2014 at 7:18 PM, Robert Metzger <[email protected]> > > wrote: > > > > > Oh. I didn't know this. The last mail from Gyula sounded like the > > Streaming > > > part is ready2release. > > > > > > My personal goal was to prepare a release candiate today so that we > have > > a > > > reference point against which we can test (which would also mean > forking > > of > > > a 0.7-release branch and basically doing a feature freeze). > > > I also though that it might be a good idea to have a release candidate > by > > > tomorrow because it could be of use for the Hackathon tomorrow in > > > Stockholm. > > > But since the Scala POJO changes and the streaming examples are both > > > outstanding, we can probably scratch that. > > > > > > Doing a fork, release candidate and a feature freeze doesn't mean we > can > > > not supply bugfixes anymore for the 0.7-release ;) > > > > > > What is your best-case scenario for FLINK-1103? (as far as I can see > the > > > only outstanding streaming feature for the release) > > > > > > Best, > > > Robert > > > > > > > > > On Tue, Oct 7, 2014 at 7:07 PM, Márton Balassi < > [email protected] > > > > > > wrote: > > > > > > > As for the streaming side we would like to push some bugfix commits > and > > > the > > > > resolution of the FLINK-1103 JIRA issue. > > > > > > > > These are more or less ready, hopefully will be available at the end > of > > > > this week. > > > > > > > > On Tue, Oct 7, 2014 at 6:50 PM, Robert Metzger <[email protected]> > > > > wrote: > > > > > > > > > I've merged the record api deprecation already. > > > > > I'll merge #141 once Aljoscha provided his Scala changes. > > > > > > > > > > We certainly should merge #136 and #143 as well. > > > > > > > > > > On Tue, Oct 7, 2014 at 5:20 PM, Fabian Hueske <[email protected]> > > > > wrote: > > > > > > > > > > > So which PRs will be included in the candidate? > > > > > > > > > > > > #141 POJOs > > > > > > #144 Deprecate Record API > > > > > > #136 Fixed example quickstart > > > > > > #143(?) Hadoop Compat: Documentation + Hadoop function wrappers > > > > (includes > > > > > > PR #131) > > > > > > > > > > > > 2014-10-07 16:26 GMT+02:00 Stephan Ewen <[email protected]>: > > > > > > > > > > > > > Great news, looking forward to seeing this in the master! > > > > > > > > > > > > > > On Tue, Oct 7, 2014 at 1:53 PM, Robert Metzger < > > > [email protected]> > > > > > > > wrote: > > > > > > > > > > > > > > > As an update for everyone: My POJO feature is finished, > > including > > > > > > > > documentation. > > > > > > > > Aljoscha is currently adopting the Scala API to have support > > for > > > > > > (nested) > > > > > > > > POJOs as well. > > > > > > > > Once that is done, I'll merge everything and create a first > > > > candidate > > > > > > > that > > > > > > > > we can use for testing. > > > > > > > > > > > > > > > > On Tue, Sep 30, 2014 at 10:49 PM, Fabian Hueske < > > > > [email protected]> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > I just checked the "Run Example" quickstart and it needs a > > bit > > > of > > > > > > work. > > > > > > > > > > > > > > > > > > 2014-09-30 22:41 GMT+02:00 Robert Metzger < > > [email protected] > > > >: > > > > > > > > > > > > > > > > > > > I'm working hard on getting the POJOs ready. > > > > > > > > > > > > > > > > > > > > We also should do a pass over our documentation, the > > > > quickstarts > > > > > > and > > > > > > > > the > > > > > > > > > > website to see if everything is in a good state (for > > example > > > > the > > > > > > > > > > collection-based execution needs documentation as well). > We > > > > > should > > > > > > > also > > > > > > > > > > finally document the hadoop-input format wrappers (I > think > > > Timo > > > > > is > > > > > > > > > working > > > > > > > > > > on a pull request for that). > > > > > > > > > > This page mentions the LocalDistributedExecutor and > > contains > > > > some > > > > > > > (most > > > > > > > > > > likely outdated) scala code: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > http://flink.incubator.apache.org/docs/0.7-incubating/local_execution.html > > > > > > > > > > > > > > > > > > > > We also need to deprecate the old record api ( > > > > > > > > > > https://issues.apache.org/jira/browse/FLINK-1106). > > > > > > > > > > > > > > > > > > > > On Tue, Sep 30, 2014 at 9:17 PM, Stephan Ewen < > > > > [email protected]> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > I think we are approaching ready state. > > > > > > > > > > > Last issues are going in and we started working on > > > > dependencies > > > > > > and > > > > > > > > > test > > > > > > > > > > > platform diversity in order to make stabilizing phase. > > > > > > > > > > > > > > > > > > > > > > We should have an official feature freeze soon and fork > > the > > > > > > > > 0.7-release > > > > > > > > > > > branch. I personally vote to include the POJO support > (I > > > > think > > > > > > > Robert > > > > > > > > > is > > > > > > > > > > > sorting that one out and is close to completion), and I > > > want > > > > to > > > > > > add > > > > > > > > the > > > > > > > > > > > collection-based execution (today or tomorrow). Till > has > > > the > > > > > BLOB > > > > > > > > > manager > > > > > > > > > > > ready, which would be good to include (better support > > large > > > > > > > libraries > > > > > > > > > or > > > > > > > > > > > fat jars). > > > > > > > > > > > > > > > > > > > > > > After that, I vote to freeze. > > > > > > > > > > > > > > > > > > > > > > On Tue, Sep 30, 2014 at 9:12 PM, Gyula Fora < > > > > [email protected] > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > Hey, > > > > > > > > > > > > > > > > > > > > > > > > So what is the current decision regarding the time of > > the > > > > > > > upcoming > > > > > > > > > > > release? > > > > > > > > > > > > > > > > > > > > > > > > As for the streaming component, we included all the > > > > features > > > > > we > > > > > > > > > wanted, > > > > > > > > > > > we > > > > > > > > > > > > will start to test everything tomorrow, making sure > > that > > > > all > > > > > > > works > > > > > > > > as > > > > > > > > > > > > intended. > > > > > > > > > > > > We are also almost finished with cleaning up the > > > connector > > > > > > > > > dependencies > > > > > > > > > > > > that Stephan pointed out, should be finished by > > tomorrow. > > > > > > > > > > > > > > > > > > > > > > > > Gyula > > > > > > > > > > > > > > > > > > > > > > > > On 26 Sep 2014, at 10:49, Fabian Hueske < > > > > [email protected]> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > +1 to manage this on JIRA (if possible) > > > > > > > > > > > > > > > > > > > > > > > > > > 2014-09-26 10:40 GMT+02:00 Aljoscha Krettek < > > > > > > > [email protected] > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > > > > > >> Can we not manage this stuff on Jira? > > > > > > > > > > > > >> > > > > > > > > > > > > >> On Fri, Sep 26, 2014 at 10:16 AM, Stephan Ewen < > > > > > > > > [email protected]> > > > > > > > > > > > > wrote: > > > > > > > > > > > > >>> I personally like the idea of SOFT time-based > > feature > > > > > > > freezes. > > > > > > > > > > > > Otherwise, > > > > > > > > > > > > >>> releases will get delayed again and again, > because > > of > > > > > > > features > > > > > > > > > that > > > > > > > > > > > we > > > > > > > > > > > > >> try > > > > > > > > > > > > >>> to squeeze in. > > > > > > > > > > > > >>> > > > > > > > > > > > > >>> Having reached the freeze point already, we could > > > still > > > > > > > include > > > > > > > > > the > > > > > > > > > > > > >>> features that are pending ready state in the next > > > days > > > > > > > > > (streaming, > > > > > > > > > > > blob > > > > > > > > > > > > >>> Manager, POJOs), but otherwise head for a release > > > > state. > > > > > > > > > > > > >>> > > > > > > > > > > > > >>> We had a mail listing the issues to include into > > 0.7, > > > > > but a > > > > > > > > wiki > > > > > > > > > > page > > > > > > > > > > > > >> would > > > > > > > > > > > > >>> probably be better. In that sense, we could start > > > > > > collecting > > > > > > > > the > > > > > > > > > > > issues > > > > > > > > > > > > >> for > > > > > > > > > > > > >>> the next release from now on. > > > > > > > > > > > > >>> Am 26.09.2014 09:17 schrieb "Daniel Warneke" < > > > > > > > > [email protected] > > > > > > > > > >: > > > > > > > > > > > > >>> > > > > > > > > > > > > >>>> Hi, > > > > > > > > > > > > >>>> > > > > > > > > > > > > >>>> I like Fabian's idea. Is there a wiki page (or > > > > something > > > > > > > > > similar) > > > > > > > > > > > > where > > > > > > > > > > > > >> we > > > > > > > > > > > > >>>> can collect the proposed JIRAs? > > > > > > > > > > > > >>>> > > > > > > > > > > > > >>>> Best regards, > > > > > > > > > > > > >>>> > > > > > > > > > > > > >>>> Daniel > > > > > > > > > > > > >>>> > > > > > > > > > > > > >>>> Am 24.09.2014 23:03, schrieb Fabian Hueske: > > > > > > > > > > > > >>>> > > > > > > > > > > > > >>>>> I agree, a hard feature stop deadline might not > > be > > > > the > > > > > > best > > > > > > > > > > > practice. > > > > > > > > > > > > >>>>> > > > > > > > > > > > > >>>>> How about the following procedure: > > > > > > > > > > > > >>>>> We decide two (or three) weeks before a > targeted > > > > > release > > > > > > > date > > > > > > > > > > about > > > > > > > > > > > > >> which > > > > > > > > > > > > >>>>> JIRAs to include. JIRAs that are selected for a > > > > release > > > > > > > > should > > > > > > > > > be > > > > > > > > > > > > >>>>> completed > > > > > > > > > > > > >>>>> or really close to completion (via progress > > > estimates > > > > > in > > > > > > > > JIRA). > > > > > > > > > > > > >>>>> After we decided which JIRAs to include in a > > > release, > > > > > we > > > > > > > can > > > > > > > > > use > > > > > > > > > > > JIRA > > > > > > > > > > > > >> to > > > > > > > > > > > > >>>>> track the progress and dedicate another week > > > > > exclusively > > > > > > > for > > > > > > > > > > > testing > > > > > > > > > > > > >> after > > > > > > > > > > > > >>>>> the last feature was completed. > > > > > > > > > > > > >>>>> > > > > > > > > > > > > >>>>> > > > > > > > > > > > > >>>>> 2014-09-24 19:10 GMT+02:00 Márton Balassi < > > > > > > > > > > > [email protected] > > > > > > > > > > > > >: > > > > > > > > > > > > >>>>> > > > > > > > > > > > > >>>>> As for the streaming team we're also getting > > ready > > > > for > > > > > > the > > > > > > > > > > release, > > > > > > > > > > > > >> but a > > > > > > > > > > > > >>>>>> couple of days will be needed to finish the > > > features > > > > > > that > > > > > > > we > > > > > > > > > > would > > > > > > > > > > > > >> like > > > > > > > > > > > > >>>>>> to > > > > > > > > > > > > >>>>>> include. > > > > > > > > > > > > >>>>>> > > > > > > > > > > > > >>>>>> - A little work is still needed for reduce > > > > > operations > > > > > > > and > > > > > > > > > > > > >>>>>> groups/connected streams (any comment on > > > Gyula's > > > > > > recent > > > > > > > > > > e-mail > > > > > > > > > > > is > > > > > > > > > > > > >>>>>> really > > > > > > > > > > > > >>>>>> appreciated :) ) > > > > > > > > > > > > >>>>>> - The examples are being updated to match > the > > > > > > standard, > > > > > > > > > check > > > > > > > > > > > out > > > > > > > > > > > > >> the > > > > > > > > > > > > >>>>>> WordCount. ( > > > > > > > > > > > > >>>>>> > > > > > > > > > > > > >>>>>> > > https://github.com/mbalassi/incubator-flink/blob/ > > > > > > > > > > > > >>>>>> > > streaming-new/flink-addons/flink-streaming/flink- > > > > > > > > > > > > >>>>>> > > streaming-examples/src/main/java/org/apache/flink/ > > > > > > > > > > > > >>>>>> streaming/examples/wordcount/WordCount.java > > > > > > > > > > > > >>>>>> ) > > > > > > > > > > > > >>>>>> Hopefully it gives you some deja vu. :) > > > > > > > > > > > > >>>>>> > > > > > > > > > > > > >>>>>> > > > > > > > > > > > > >>>>>> On Wed, Sep 24, 2014 at 6:53 PM, Ufuk Celebi < > > > > > > > > [email protected]> > > > > > > > > > > > > wrote: > > > > > > > > > > > > >>>>>> > > > > > > > > > > > > >>>>>> On 24 Sep 2014, at 18:37, Robert Metzger < > > > > > > > > [email protected] > > > > > > > > > > > > > > > > > > > > > > >> wrote: > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > >>>>>>> Hey guys, > > > > > > > > > > > > >>>>>>>> > > > > > > > > > > > > >>>>>>>> exactly 3 weeks ago, we discussed to do a > > > feature > > > > > > freeze > > > > > > > > for > > > > > > > > > > the > > > > > > > > > > > > >>>>>>>> 0.7-incubating release today. > > > > > > > > > > > > >>>>>>>> > > > > > > > > > > > > >>>>>>>> From our initial feature list: > > > > > > > > > > > > >>>>>>>> - *Flink Streaming* "Beta Preview". I would > > > > suggest > > > > > to > > > > > > > > ship > > > > > > > > > > the > > > > > > > > > > > > >>>>>>>> > > > > > > > > > > > > >>>>>>> streaming, > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > >>>>>>>> but clearly mark it as a preview in the > > > > > documentation. > > > > > > > > > > > > >>>>>>>> -* Java API Pojo improvements*: Code > > generation, > > > > key > > > > > > > > > selection > > > > > > > > > > > > >> using a > > > > > > > > > > > > >>>>>>>> string-expression: > > > > > > > > > > > > https://issues.apache.org/jira/browse/FLINK-1032 > > > > > > > > > > > > >>>>>>>> - *Reworked Scala API*. Bring the Scala API > > in > > > > sync > > > > > > > with > > > > > > > > > the > > > > > > > > > > > > >> latest > > > > > > > > > > > > >>>>>>>> developments in the Java API: > > > > > > > > > > > > >>>>>>>> > > https://issues.apache.org/jira/browse/FLINK-641 > > > > > > > > > > > > >>>>>>>> -* Akka-based RPC service*: > > > > > > > > > > > > >>>>>>>> > > > https://issues.apache.org/jira/browse/FLINK-1019 > > > > > > > > > > > > >>>>>>>> - *Kryo-based serialization*. This feature > > has > > > > been > > > > > > > > > requested > > > > > > > > > > > by > > > > > > > > > > > > >> many > > > > > > > > > > > > >>>>>>>> users. Mostly because they wanted to use > > > > Collections > > > > > > > > inside > > > > > > > > > > > POJOs: > > > > > > > > > > > > >>>>>>>> > > https://issues.apache.org/jira/browse/FLINK-610 > > > > > > > > > > > > >>>>>>>> - Rework JobManager internals to support > > > > incremental > > > > > > > > program > > > > > > > > > > > > >> rollout & > > > > > > > > > > > > >>>>>>>> execution > > > > > > > > > > > > >>>>>>>> - First parts of dynamic memory assignments > > > > > > > > > > > > >>>>>>>> > > > > > > > > > > > > >>>>>>>> The following features are in the master, as > > of > > > > > today: > > > > > > > > > > > > >>>>>>>> - *Flink Streaming* > > > > > > > > > > > > >>>>>>>> - *Reworked Scala API* > > > > > > > > > > > > >>>>>>>> -* New Scheduler* > > > > > > > > > > > > >>>>>>>> > > > > > > > > > > > > >>>>>>>> We certainly need some days to test > everything > > > > until > > > > > > we > > > > > > > > can > > > > > > > > > > > start > > > > > > > > > > > > >> the > > > > > > > > > > > > >>>>>>>> > > > > > > > > > > > > >>>>>>> vote. > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > >>>>>>>> Based on our experience with the last major > > > > > release, I > > > > > > > > would > > > > > > > > > > > > really > > > > > > > > > > > > >>>>>>>> > > > > > > > > > > > > >>>>>>> like > > > > > > > > > > > > >>>>>> > > > > > > > > > > > > >>>>>>> to > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > >>>>>>>> do the testing and bugfixing BEFORE the > first > > > > > release > > > > > > > > > > candidate. > > > > > > > > > > > > For > > > > > > > > > > > > >>>>>>>> > > > > > > > > > > > > >>>>>>> the > > > > > > > > > > > > >>>>>> > > > > > > > > > > > > >>>>>>> 0.6-incubating release, we had 6 candidates) > > > > > > > > > > > > >>>>>>>> > > > > > > > > > > > > >>>>>>>> How do you guys feel about this? Should we > > wait > > > a > > > > > few > > > > > > > more > > > > > > > > > > days > > > > > > > > > > > > for > > > > > > > > > > > > >> the > > > > > > > > > > > > >>>>>>>> release so that a few more features make it > > into > > > > the > > > > > > > > > release? > > > > > > > > > > > > >>>>>>>> > > > > > > > > > > > > >>>>>>>> I'm undecided on this. On the one hand, its > > > really > > > > > > nice > > > > > > > to > > > > > > > > > > > release > > > > > > > > > > > > >> on a > > > > > > > > > > > > >>>>>>>> regular schedule, but it also eats up some > > time > > > > and > > > > > > > causes > > > > > > > > > > > > overhead > > > > > > > > > > > > >>>>>>>> (different branches etc.). > > > > > > > > > > > > >>>>>>>> I would really like to have the Java API > Pojo > > > > > > > improvements > > > > > > > > > in > > > > > > > > > > > the > > > > > > > > > > > > >>>>>>>> > > > > > > > > > > > > >>>>>>> release. > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > >>>>>>>> I think I can finish it until end of this > > week. > > > > > > > > > > > > >>>>>>>> > > > > > > > > > > > > >>>>>>>> Opinions? > > > > > > > > > > > > >>>>>>>> > > > > > > > > > > > > >>>>>>> I agree that the finished features > (especially > > > the > > > > > > Scala > > > > > > > > API) > > > > > > > > > > are > > > > > > > > > > > > >> nice > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > >>>>>> for > > > > > > > > > > > > >>>>>> > > > > > > > > > > > > >>>>>>> a new release, but still I would like to > wait a > > > few > > > > > > more > > > > > > > > > days. > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > >>>>>>> Some of the missing features are on the brink > > of > > > > > being > > > > > > > > > finished > > > > > > > > > > > > (e.g. > > > > > > > > > > > > >>>>>>> the > > > > > > > > > > > > >>>>>>> Pojo improvements). I wouldn't want to > invest a > > > > week > > > > > in > > > > > > > bug > > > > > > > > > > > fixing > > > > > > > > > > > > >> and > > > > > > > > > > > > >>>>>>> doing the release vote, when the new features > > are > > > > > > likely > > > > > > > to > > > > > > > > > be > > > > > > > > > > > > >> finished > > > > > > > > > > > > >>>>>>> just a few days afterwards. And the upcoming > > > > features > > > > > > > will > > > > > > > > > > > > >> definitely be > > > > > > > > > > > > >>>>>>> worth a release, so users can work with them. > > ;) > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > >>>>>> > > > > > > > > > > > > >>>> > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
