Re: [DISCUSS] Releasing Flink 1.1.0

2016-07-07 Thread Ufuk Celebi
I've now created a preview RC0 in a separate thread. As soon as we resolve the remaining open issues and anything we the community discovers with RC0, we can start with the first official RC1. – Ufuk On Tue, Jul 5, 2016 at 6:18 PM, Flavio Pompermaier wrote: > it's just because Fabian said that

Re: [DISCUSS] Releasing Flink 1.1.0

2016-07-05 Thread Flavio Pompermaier
it's just because Fabian said that it's better to not mix java and scala (as you can see in the comments of that PR) On 5 Jul 2016 18:53, "Aljoscha Krettek" wrote: > I think it's not strictly required that all code be in Scala. There is > already some Java code in there so we shouldn't force peop

Re: [DISCUSS] Releasing Flink 1.1.0

2016-07-05 Thread Aljoscha Krettek
I think it's not strictly required that all code be in Scala. There is already some Java code in there so we shouldn't force people to write Scala code if they make a valuable contribution in Java. On Tue, 5 Jul 2016 at 17:33 Flavio Pompermaier wrote: > Hi to all, > if Flink 1.1 will introduce u

Re: [DISCUSS] Releasing Flink 1.1.0

2016-07-05 Thread Flavio Pompermaier
Hi to all, if Flink 1.1 will introduce ufficially the Table API, do you think someone could take care of rewriting in scala the necessary java code of my PR about reading CSV as Rows instead of tuples[1]? For our use cases, and many new users approaching to Flink IMHO, that will be definitely usef

Re: [DISCUSS] Releasing Flink 1.1.0

2016-07-05 Thread Aljoscha Krettek
Btw, another blocking issue, IMHO: https://issues.apache.org/jira/browse/FLINK-4149 I'm working on a fix. On Tue, 5 Jul 2016 at 17:08 Till Rohrmann wrote: > I found another critical issue [1]. The murmur hash correction introduced > between Flink 1.0 and 1.1 breaks the backwards compatibility

Re: [DISCUSS] Releasing Flink 1.1.0

2016-07-05 Thread Till Rohrmann
I found another critical issue [1]. The murmur hash correction introduced between Flink 1.0 and 1.1 breaks the backwards compatibility with respect to savepoints. I think we have to fix this for the release. @Ufuk, I'm not sure whether I find time this week to work on FLINK-4150. I could make it a

Re: [DISCUSS] Releasing Flink 1.1.0

2016-07-05 Thread Greg Hogan
Hi Ufuk, The old sort-based combine is still the default. The user calls .setCombineHint(CombineHint) to make a selection (I think this was originally overloaded on DataSet and it looks like the pr1517 documentation update does not reflect the new usage). I'd be glad to merge this in but I didn't

Re: [DISCUSS] Releasing Flink 1.1.0

2016-07-05 Thread Ufuk Celebi
Great that we are all on the same page :-) Thanks for pointing out the two issues Aljoscha and Till. I agree with you and I've updated them to blockers ;-) The FsStateBackend looks like it will be done soon. @Till: do you have time to look into FLINK-4150 this week? I can also do it after I've add

Re: [DISCUSS] Releasing Flink 1.1.0

2016-07-05 Thread Till Rohrmann
Stefan Richter found the following problem with HA: https://issues.apache.org/jira/browse/FLINK-4150 I think we should fix it for the 1.1 release. On Mon, Jul 4, 2016 at 9:05 PM, Robert Metzger wrote: > +1 to do a RC0 this week, but the master-forking with RC1. I would like to > reduce the time

Re: [DISCUSS] Releasing Flink 1.1.0

2016-07-04 Thread Robert Metzger
+1 to do a RC0 this week, but the master-forking with RC1. I would like to reduce the time we need to apply patches to multiple branches. @Aljoscha: I was running into the same issue on EMR when I used Flink w/ RocksDB recently, so I agree ;) On Mon, Jul 4, 2016 at 3:35 PM, Aljoscha Krettek wr

Re: [DISCUSS] Releasing Flink 1.1.0

2016-07-04 Thread Aljoscha Krettek
IMHO, the fix for this should also go in: https://issues.apache.org/jira/browse/FLINK-4115. This is blocking for users that want to use the RocksDB backend or FsStateBackend on Amazon EMR with S3. There is already an open PR that I'm hoping to get in this week. On Mon, 4 Jul 2016 at 13:48 Ufuk Ce

Re: [DISCUSS] Releasing Flink 1.1.0

2016-07-04 Thread Ufuk Celebi
Thanks for the feedback. I would like to create a preview RC0 this week like we did for the last releases, too. In past major releases, we needed to create many release candidates, often for fixing just some small issues. I would like to speed up the release process by collecting as many issues as

Re: [DISCUSS] Releasing Flink 1.1.0

2016-07-01 Thread Maximilian Michels
Yet another one for the release: FLINK-4144 https://github.com/apache/flink/pull/2191 On Fri, Jul 1, 2016 at 5:18 PM, Maximilian Michels wrote: > There is also FLINK-4141. We just found this during testing. PR is > waiting to be merged here: https://github.com/apache/flink/pull/2190 > > On Fri, J

Re: [DISCUSS] Releasing Flink 1.1.0

2016-07-01 Thread Maximilian Michels
There is also FLINK-4141. We just found this during testing. PR is waiting to be merged here: https://github.com/apache/flink/pull/2190 On Fri, Jul 1, 2016 at 3:26 PM, Maximilian Michels wrote: > FLINK-3904 is not Yarn related. Not pressing to fix for this release > and actually a bit tricky to f

Re: [DISCUSS] Releasing Flink 1.1.0

2016-07-01 Thread Maximilian Michels
FLINK-3904 is not Yarn related. Not pressing to fix for this release and actually a bit tricky to fix. I've addressed the other issues and merged all pending pull requests. Good to go from my side. On Fri, Jul 1, 2016 at 11:29 AM, Chesnay Schepler wrote: > There are also 2 fixes for Cassandra tha

Re: [DISCUSS] Releasing Flink 1.1.0

2016-07-01 Thread Chesnay Schepler
There are also 2 fixes for Cassandra that should be included: https://github.com/apache/flink/pull/2167 https://github.com/apache/flink/pull/2183 We should also include the documentation for the metrics stuff (hopefully merged today) https://github.com/apache/flink/pull/2158 In regards to met

Re: [DISCUSS] Releasing Flink 1.1.0

2016-06-30 Thread Greg Hogan
It would be great if hash-based combine (FLINK-3477) could make it in to be tested for this release. We've seen impressive improvements in performance (though, admittedly, some sort-based enhancements are yet to be worked on). This PR looks to be ripe. Also, as we tidy up a few things with Gelly a

Re: [DISCUSS] Releasing Flink 1.1.0

2016-06-27 Thread Robert Metzger
Sure Ufuk! Thanks a lot for taking care of the release management. I'll be on vacation in three weeks, for three weeks and I'm not sure if we get the release done until then. On Mon, Jun 27, 2016 at 12:08 PM, Ufuk Celebi wrote: > I would like to do it if that's OK with you Robert. I would follow

Re: [DISCUSS] Releasing Flink 1.1.0

2016-06-27 Thread Ufuk Celebi
I would like to do it if that's OK with you Robert. I would follow your suggestion and wait a few days until the following important fixes are in: - Savepoint headers and proper disposal (FLINK-4067 and https://github.com/apache/flink/pull/2083) - Metrics (https://github.com/apache/flink/pull/2146)

Re: [DISCUSS] Releasing Flink 1.1.0

2016-06-23 Thread Robert Metzger
Hi, it doesn't seem that there are volunteers for the RM, so I'll probably do it. I try to do the first release candidate (mostly for testing) next week (it depends on the JIRAs fixed by then) On Thu, Jun 16, 2016 at 10:56 PM, Henry Saputra wrote: > Thanks for the reply, @Max. I was not aware i

Re: [DISCUSS] Releasing Flink 1.1.0

2016-06-16 Thread Henry Saputra
Thanks for the reply, @Max. I was not aware it was about dynamic scaling, which I think also asked for YARN support. I agree to list all related half merge JIRA for the ResourceManager. Looking forward for the Apache Mesos integration design for sure =) - Henry On Thu, Jun 16, 2016 at 2:12 AM, M

Re: [DISCUSS] Releasing Flink 1.1.0

2016-06-16 Thread Andrea Sella
Hi Robert, Yes, it works in the current master. Thank you. Cheers, Andrea 2016-06-16 14:03 GMT+02:00 Robert Metzger : > @Andrea: Do I understand correctly that the scala-shell works with HA in > the current master? If so, it will be included automatically into the > release. > > I created a Wik

Re: [DISCUSS] Releasing Flink 1.1.0

2016-06-16 Thread Robert Metzger
@Andrea: Do I understand correctly that the scala-shell works with HA in the current master? If so, it will be included automatically into the release. I created a Wiki page for the 1.1 release: https://cwiki.apache.org/confluence/display/FLINK/1.1+Release I don't think it makes sense to copy a lo

Re: [DISCUSS] Releasing Flink 1.1.0

2016-06-16 Thread Vishnu Viswanath
Ok, Makes sense. Thanks Fabian! On Thu, Jun 16, 2016 at 7:27 AM, Fabian Hueske wrote: > Hi Vishnu, > > no, queryable state would not be part of the 1.1 release according to > Robert's proposal. > > IMO, adding a major feature like queryable state shortly before a release > is not a good idea and

Re: [DISCUSS] Releasing Flink 1.1.0

2016-06-16 Thread Fabian Hueske
Hi Vishnu, no, queryable state would not be part of the 1.1 release according to Robert's proposal. IMO, adding a major feature like queryable state shortly before a release is not a good idea and likely to delay the release or cause problems later. Major features need a bit of exposure in the ma

Re: [DISCUSS] Releasing Flink 1.1.0

2016-06-16 Thread Vishnu Viswanath
Hi, Will queryable state available in 1.1.0? Thanks, Vishnu On Thursday, 16 June 2016, Aljoscha Krettek wrote: > Hi, > +1 for releasing what we have now and making it rock solid. > > The support for merging windows (session windows) is also new, btw. :-) > > Cheers, > Aljoscha > > On Thu, 16 J

Re: [DISCUSS] Releasing Flink 1.1.0

2016-06-16 Thread Aljoscha Krettek
Hi, +1 for releasing what we have now and making it rock solid. The support for merging windows (session windows) is also new, btw. :-) Cheers, Aljoscha On Thu, 16 Jun 2016 at 11:12 Maximilian Michels wrote: > Hi Robert, hi Henry, > > +1 for a 1.1.0 release soon! We have enough new features th

Re: [DISCUSS] Releasing Flink 1.1.0

2016-06-16 Thread Maximilian Michels
Hi Robert, hi Henry, +1 for a 1.1.0 release soon! We have enough new features that justify a major release. @Henry We have plans to extend the ResourceManager to interact with the Scheduler which will be a prerequisite for dynamic scaling. I think this is out of scope for 1.1.0. The upcoming Meso

Re: [DISCUSS] Releasing Flink 1.1.0

2016-06-16 Thread Andrea Sella
Hi Robert, Another interesting features to add in 1.1.0 should be the scala-shell integration with YARN in HA[1]. Thank you, Andrea [1] http://mail-archives.apache.org/mod_mbox/flink-dev/201604.mbox/%3CCABsXpV1Qu7v8cwxx-DkQ4JRNBvGNh-EF2MxdjzyYjsWUpQn01g%40mail.gmail.com%3E 2016-06-15 22:50 GMT+

Re: [DISCUSS] Releasing Flink 1.1.0

2016-06-15 Thread Henry Saputra
Hi Robert, Thanks for staying the discussion. Do you know if there any open tasks for the Resource Manager left? That is probably needed for Mesos integration? - Henry On Wed, Jun 15, 2016 at 12:55 PM, Robert Metzger wrote: > Hi, > > Flink 1.0.0 was released early March, so three months have

[DISCUSS] Releasing Flink 1.1.0

2016-06-15 Thread Robert Metzger
Hi, Flink 1.0.0 was released early March, so three months have passed and I think we should start discussing the scope of the next major release (1.1.0). >From a high level point of view, we've added the following new features: in master: - Table API Refactoring, SQL, StreamSQL - The metrics syst