I have a fix for this issue: https://issues.apache.org/jira/browse/FLINK-1378?jql=project%20%3D%20FLINK and I think this should also make it into the 0.8.0 release.
What do you think? On Fri, Jan 9, 2015 at 3:28 PM, Márton Balassi <balassi.mar...@gmail.com> wrote: > Sure, thanks Ufuk. > > On Fri, Jan 9, 2015 at 3:15 PM, Ufuk Celebi <u...@apache.org> wrote: > >> Marton, could you also cherry-pick 7f659f6 and 7e08fa1 for the next RC? >> It's a minor update to the README describing the IDE setup. >> >> I will closed the respective issue FLINK-1109. >> >> On 08 Jan 2015, at 23:50, Henry Saputra <henry.sapu...@gmail.com> wrote: >> >> > Marton, could you close this VOTE thread by replying to the original >> > email and append [CANCEL] in the subject line. >> > >> > - Henry >> > >> > On Thu, Jan 8, 2015 at 9:35 AM, Márton Balassi <balassi.mar...@gmail.com> >> wrote: >> >> Cherry-picked and tested: found no duplicate dependencies in lib, yarn >> >> uberjar build goes without the mentioned warns. >> >> Travis tests are passing, pushing soon. >> >> >> >> On Thu, Jan 8, 2015 at 4:57 PM, Stephan Ewen <se...@apache.org> wrote: >> >> >> >>> Nice. >> >>> >> >>> @Marton: As soon as as you are done, I make a pass over the licenses... >> >>> >> >>> Stephan >> >>> >> >>> >> >>> On Thu, Jan 8, 2015 at 4:42 PM, Robert Metzger <rmetz...@apache.org> >> >>> wrote: >> >>> >> >>>> Allright. The travis tests are green and I tested it again with >> Tachyon >> >>> on >> >>>> a cluster. >> >>>> >> >>>> My pull request also fixes some of the issues mentioned earlier in >> this >> >>>> thread by Stephan (the warnings from shading regarding duplicate >> >>> classes). >> >>>> I can not fix all of them because some projects like hadoop just have >> the >> >>>> "conflicts" in their dependencies. >> >>>> >> >>>> I'll merge the pull request to master. >> >>>> Marton can merge it to "release-0.8". >> >>>> >> >>>> On Thu, Jan 8, 2015 at 2:42 PM, Stephan Ewen <se...@apache.org> >> wrote: >> >>>> >> >>>>> Robert, can you let us know once the PR is merged? >> >>>>> >> >>>>> Then we can start with the next steps... >> >>>>> >> >>>>> >> >>>>> >> >>>>> On Thu, Jan 8, 2015 at 2:07 PM, Robert Metzger <rmetz...@apache.org> >> >>>>> wrote: >> >>>>> >> >>>>>> Please wait with merging 268. There is still a minor open issue. >> >>>>>> >> >>>>>> On Thu, Jan 8, 2015 at 1:28 PM, Márton Balassi < >> >>>> balassi.mar...@gmail.com >> >>>>>> >> >>>>>> wrote: >> >>>>>> >> >>>>>>> +1 for merging PR 268, volunteering for the merge. It should also >> >>>> solve >> >>>>>> the >> >>>>>>> issue mentioned by Stephan with the yarn uberjar. >> >>>>>>> >> >>>>>>> Updating the release scripts and doing a double-check on the >> >>>> cosmetics. >> >>>>>>> >> >>>>>>> On Thu, Jan 8, 2015 at 12:55 PM, Stephan Ewen <se...@apache.org> >> >>>>> wrote: >> >>>>>>> >> >>>>>>>> We just found and fixed another highly critical bug. >> >>>>>>>> >> >>>>>>>> I vote to also cherry pick commit >> >>>>>>>> 19066b520435528e104a69ccf372f56811123ee3 ([FLINK-1371] >> >>>>>>>> [runtime] Fix KryoSerializer to not swallow EOFExceptions) >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> On Thu, Jan 8, 2015 at 11:55 AM, Robert Metzger < >> >>>> rmetz...@apache.org >> >>>>>> >> >>>>>>>> wrote: >> >>>>>>>> >> >>>>>>>>> +1 for merging pull request 268. It makes Flink even more >> >>>>> compatible >> >>>>>>> with >> >>>>>>>>> hadoop. >> >>>>>>>>> >> >>>>>>>>> >> >>>>>>>>> regarding the documentation: I already updated the build >> >>>>>> instructions. >> >>>>>>>> But >> >>>>>>>>> we should double check everything again. >> >>>>>>>>> I probably won't have time to look into this today however. >> >>>>>>>>> >> >>>>>>>>> On Thu, Jan 8, 2015 at 11:50 AM, Stephan Ewen < >> >>> se...@apache.org> >> >>>>>>> wrote: >> >>>>>>>>> >> >>>>>>>>>> After the dependencies are cleaned up (Robert's pull request >> >>>>>>>>>> https://github.com/apache/flink/pull/268 solves that), we >> >>> need >> >>>>> to >> >>>>>>> also >> >>>>>>>>> do >> >>>>>>>>>> the following: >> >>>>>>>>>> >> >>>>>>>>>> - Make a pass over dependencies to ensure that NOTICE and >> >>>>> LICENSE >> >>>>>>>> files >> >>>>>>>>>> are up to date (both source and bin distribution) >> >>>>>>>>>> - Adjust the variables in the docs to reflect the fact that >> >>>>>> hadoop2 >> >>>>>>> is >> >>>>>>>>> now >> >>>>>>>>>> the default profile >> >>>>>>>>>> >> >>>>>>>>>> I would volunteer for the first task. >> >>>>>>>>>> >> >>>>>>>>>> Stephan >> >>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>>> On Wed, Jan 7, 2015 at 10:01 PM, Stephan Ewen < >> >>>> se...@apache.org> >> >>>>>>>> wrote: >> >>>>>>>>>> >> >>>>>>>>>>> I also see a warning when building the yarn fat jar. >> >>>>>>>>>>> >> >>>>>>>>>>> This is a consequence of the duplicate dependencies. >> >>>>>>>>>>> >> >>>>>>>>>>> [WARNING] servlet-api-3.0.20100224.jar, >> >>>>>>> javax.servlet-api-3.0.1.jar, >> >>>>>>>>>>> javax.servlet-3.1.jar, servlet-api-2.5.jar define 42 >> >>>>> overlappping >> >>>>>>>>>> classes: >> >>>>>>>>>>> [WARNING] - javax.servlet.http.Cookie >> >>>>>>>>>>> [WARNING] - javax.servlet.http.HttpSessionBindingEvent >> >>>>>>>>>>> [WARNING] - javax.servlet.http.NoBodyResponse >> >>>>>>>>>>> [WARNING] - javax.servlet.ServletContext >> >>>>>>>>>>> [WARNING] - javax.servlet.ServletOutputStream >> >>>>>>>>>>> [WARNING] - javax.servlet.http.HttpSessionListener >> >>>>>>>>>>> [WARNING] - javax.servlet.http.HttpSessionContext >> >>>>>>>>>>> [WARNING] - javax.servlet.FilterChain >> >>>>>>>>>>> [WARNING] - javax.servlet.GenericServlet >> >>>>>>>>>>> [WARNING] - javax.servlet.http.HttpServletRequestWrapper >> >>>>>>>>>>> [WARNING] - 32 more... >> >>>>>>>>>>> [WARNING] hadoop-yarn-api-2.2.0.jar, >> >>>>> hadoop-yarn-common-2.2.0.jar >> >>>>>>>>> define >> >>>>>>>>>> 3 >> >>>>>>>>>>> overlappping classes: >> >>>>>>>>>>> [WARNING] - org.apache.hadoop.yarn.util.package-info >> >>>>>>>>>>> [WARNING] - org.apache.hadoop.yarn.factories.package-info >> >>>>>>>>>>> [WARNING] - >> >>>>>> org.apache.hadoop.yarn.factory.providers.package-info >> >>>>>>>>>>> [WARNING] asm-4.0.jar, asm-3.1.jar define 21 overlappping >> >>>>>> classes: >> >>>>>>>>>>> [WARNING] - org.objectweb.asm.Type >> >>>>>>>>>>> [WARNING] - org.objectweb.asm.AnnotationVisitor >> >>>>>>>>>>> [WARNING] - org.objectweb.asm.MethodVisitor >> >>>>>>>>>>> [WARNING] - org.objectweb.asm.Attribute >> >>>>>>>>>>> [WARNING] - org.objectweb.asm.FieldWriter >> >>>>>>>>>>> [WARNING] - org.objectweb.asm.signature.SignatureWriter >> >>>>>>>>>>> [WARNING] - org.objectweb.asm.MethodWriter >> >>>>>>>>>>> [WARNING] - org.objectweb.asm.Edge >> >>>>>>>>>>> [WARNING] - org.objectweb.asm.Handler >> >>>>>>>>>>> [WARNING] - org.objectweb.asm.ByteVector >> >>>>>>>>>>> [WARNING] - 11 more... >> >>>>>>>>>>> [WARNING] javax.servlet-api-3.0.1.jar, >> >>> javax.servlet-3.1.jar, >> >>>>>>>>>>> servlet-api-3.0.20100224.jar define 32 overlappping >> >>> classes: >> >>>>>>>>>>> [WARNING] - javax.servlet.annotation.HttpConstraint >> >>>>>>>>>>> [WARNING] - javax.servlet.DispatcherType >> >>>>>>>>>>> [WARNING] - >> >>>>> javax.servlet.descriptor.JspPropertyGroupDescriptor >> >>>>>>>>>>> [WARNING] - javax.servlet.Registration >> >>>>>>>>>>> [WARNING] - javax.servlet.SessionTrackingMode >> >>>>>>>>>>> [WARNING] - >> >>>>>>>>> javax.servlet.annotation.ServletSecurity$EmptyRoleSemantic >> >>>>>>>>>>> [WARNING] - javax.servlet.annotation.HandlesTypes >> >>>>>>>>>>> [WARNING] - javax.servlet.ServletRegistration >> >>>>>>>>>>> [WARNING] - javax.servlet.annotation.ServletSecurity >> >>>>>>>>>>> [WARNING] - javax.servlet.ServletContainerInitializer >> >>>>>>>>>>> [WARNING] - 22 more... >> >>>>>>>>>>> [WARNING] jasper-compiler-5.5.23.jar, >> >>>> jasper-runtime-5.5.23.jar >> >>>>>>>> define >> >>>>>>>>> 1 >> >>>>>>>>>>> overlappping classes: >> >>>>>>>>>>> [WARNING] - org.apache.jasper.compiler.Localizer >> >>>>>>>>>>> [WARNING] commons-collections-3.2.1.jar, >> >>>>>>> commons-beanutils-1.7.0.jar, >> >>>>>>>>>>> commons-beanutils-core-1.8.0.jar define 10 overlappping >> >>>>> classes: >> >>>>>>>>>>> [WARNING] - >> >>>>> org.apache.commons.collections.FastHashMap$EntrySet >> >>>>>>>>>>> [WARNING] - >> >>>> org.apache.commons.collections.FastHashMap$KeySet >> >>>>>>>>>>> [WARNING] - org.apache.commons.collections.ArrayStack >> >>>>>>>>>>> [WARNING] - >> >>>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>> >> >>>>>>>> >> >>>>>>> >> >>>>>> >> >>>>> >> >>>> >> >>> >> org.apache.commons.collections.FastHashMap$CollectionView$CollectionViewIterator >> >>>>>>>>>>> [WARNING] - >> >>>> org.apache.commons.collections.FastHashMap$Values >> >>>>>>>>>>> [WARNING] - >> >>>>>>>> org.apache.commons.collections.FastHashMap$CollectionView >> >>>>>>>>>>> [WARNING] - org.apache.commons.collections.FastHashMap$1 >> >>>>>>>>>>> [WARNING] - org.apache.commons.collections.Buffer >> >>>>>>>>>>> [WARNING] - org.apache.commons.collections.FastHashMap >> >>>>>>>>>>> [WARNING] - >> >>>>>>> org.apache.commons.collections.BufferUnderflowException >> >>>>>>>>>>> [WARNING] commons-beanutils-1.7.0.jar, >> >>>>>>>> commons-beanutils-core-1.8.0.jar >> >>>>>>>>>>> define 82 overlappping classes: >> >>>>>>>>>>> [WARNING] - org.apache.commons.beanutils.ConvertUtilsBean >> >>>>>>>>>>> [WARNING] - >> >>>>>>>> org.apache.commons.beanutils.converters.SqlTimeConverter >> >>>>>>>>>>> [WARNING] - org.apache.commons.beanutils.Converter >> >>>>>>>>>>> [WARNING] - >> >>>>>>>>> org.apache.commons.beanutils.converters.FloatArrayConverter >> >>>>>>>>>>> [WARNING] - >> >>>> org.apache.commons.beanutils.NestedNullException >> >>>>>>>>>>> [WARNING] - >> >>>>> org.apache.commons.beanutils.ConvertingWrapDynaBean >> >>>>>>>>>>> [WARNING] - >> >>>>>>>>> org.apache.commons.beanutils.converters.LongArrayConverter >> >>>>>>>>>>> [WARNING] - >> >>>>>>>> org.apache.commons.beanutils.converters.SqlDateConverter >> >>>>>>>>>>> [WARNING] - >> >>>>>>>>>> org.apache.commons.beanutils.converters.BooleanArrayConverter >> >>>>>>>>>>> [WARNING] - >> >>>>>>> org.apache.commons.beanutils.converters.StringConverter >> >>>>>>>>>>> [WARNING] - 72 more... >> >>>>>>>>>>> >> >>>>>>>>>>> >> >>>>>>>>>>> On Wed, Jan 7, 2015 at 9:08 PM, Robert Metzger < >> >>>>>>> rmetz...@apache.org> >> >>>>>>>>>>> wrote: >> >>>>>>>>>>> >> >>>>>>>>>>>> ... I did some more checks: >> >>>>>>>>>>>> >> >>>>>>>>>>>> The hadoop200alpha version has some issues with the hadoop >> >>>>>>>> dependency >> >>>>>>>>>>>> exclusions. I fixed the issues in this pull request: >> >>>>>>>>>>>> https://github.com/apache/flink/pull/268. The dependency >> >>>>>>> exclusions >> >>>>>>>>> for >> >>>>>>>>>>>> hadoop are basically disabled because they are overwritten >> >>>> by >> >>>>>> the >> >>>>>>>>>>>> hadoop200alpha profile. This causes issues like two asm >> >>>>> versions >> >>>>>>>> (4.0 >> >>>>>>>>>> and >> >>>>>>>>>>>> 3.2) in the lib/ folder. >> >>>>>>>>>>>> >> >>>>>>>>>>>> Very cosmetic: >> >>>>>>>>>>>> - We can update the urls in the README.txt / README.md of >> >>>> the >> >>>>>>> binary >> >>>>>>>>>>>> releases. They contain >> >>>>>>>>>>>> http://flink.incubator.apache.org >> >>>>>>>>>>>> https://github.com/apache/incubator-flink >> >>>>>>>>>>>> u...@flink.incubator.apache.org >> >>>>>>>>>>>> d...@flink.incubator.apache.org >> >>>>>>>>>>>> Which all don't need the "incubator" anymore (mailinglists >> >>>>>> should >> >>>>>>>> work >> >>>>>>>>>> as >> >>>>>>>>>>>> well, but we need to verify) >> >>>>>>>>>>>> >> >>>>>>>>>>>> >> >>>>>>>>>>>> Positive checks: >> >>>>>>>>>>>> - No incubator notices >> >>>>>>>>>>>> - Example plan preview is working for all examples >> >>>>>>>>>>>> - All examples are running with build-in data, submitted >> >>>> from >> >>>>>> the >> >>>>>>>> web >> >>>>>>>>>>>> interface >> >>>>>>>>>>>> - scala examples (some) >> >>>>>>>>>>>> - streaming examples (some) >> >>>>>>>>>>>> - No exceptions or WARN log messages occurred after a >> >>> "start >> >>>>> jm >> >>>>>> / >> >>>>>>>> web >> >>>>>>>>>> --> >> >>>>>>>>>>>> all java batch examples --> stop everything" cycle. >> >>>>>>>>>>>> - ./bin/flink also works for the wordcount example ;) >> >>>>>>>>>>>> - no binaries in the src archive >> >>>>>>>>>>>> - the source archive builds (mvn clean install) (including >> >>>> rat >> >>>>>>>> license >> >>>>>>>>>>>> header check, checkstyle and all tests) >> >>>>>>>>>>>> - gpg signatures are valid: gpg --verify >> >>>>>> flink-0.8.0-src.tgz.asc >> >>>>>>>>>>>> flink-0.8.0-src.tgz >> >>>>>>>>>>>> >> >>>>>>>>>>>> >> >>>>>>>>>>>> On Wed, Jan 7, 2015 at 7:55 PM, Robert Metzger < >> >>>>>>> rmetz...@apache.org >> >>>>>>>>> >> >>>>>>>>>>>> wrote: >> >>>>>>>>>>>> >> >>>>>>>>>>>>> -1 >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> There is a major issue with the hadoop1 and hadoop2 >> >>>>> versions: >> >>>>>> We >> >>>>>>>>> made >> >>>>>>>>>>>>> hadoop2 the default profile, so the 0.8.0 version will >> >>>> pull >> >>>>>> the >> >>>>>>>>>> hadoop2 >> >>>>>>>>>>>>> dependencies. >> >>>>>>>>>>>>> We basically need an explicit 0.8.0-hadoop1 version now. >> >>>>>>>>>>>>> The release candidate contains a 0.8.0 version which has >> >>>>>> hadoop2 >> >>>>>>>>>>>> activated >> >>>>>>>>>>>>> by default, the 0.8.0-hadoop2 version also has hadoop2 >> >>>>>> activated >> >>>>>>>> by >> >>>>>>>>>>>>> default. In this release candidate, there is no way for >> >>>>> users >> >>>>>> to >> >>>>>>>>>> depend >> >>>>>>>>>>>> on >> >>>>>>>>>>>>> a hadoop1 flink dependency. >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> In general, we should warn users that the default hadoop >> >>>>>>>>> dependencies >> >>>>>>>>>>>> have >> >>>>>>>>>>>>> changed with the release! >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> I suggest to wait a bit with the next RC to collect more >> >>>>>>> feedback >> >>>>>>>> on >> >>>>>>>>>> the >> >>>>>>>>>>>>> release candidate. >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> On Wed, Jan 7, 2015 at 7:37 PM, Márton Balassi < >> >>>>>>>> mbala...@apache.org >> >>>>>>>>>> >> >>>>>>>>>>>>> wrote: >> >>>>>>>>>>>>> >> >>>>>>>>>>>>>> Please vote on releasing the following candidate as >> >>>> Apache >> >>>>>>> Flink >> >>>>>>>>>>>> version >> >>>>>>>>>>>>>> 0.8.0 >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> This release will be the first major release for Flink >> >>>> as a >> >>>>>> top >> >>>>>>>>> level >> >>>>>>>>>>>>>> project. >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> >> >>>>> ------------------------------------------------------------- >> >>>>>>>>>>>>>> The commit to be voted on is in the branch >> >>>>>> "release-0.8.0-rc1" >> >>>>>>>>>>>>>> (commit 8c30f60): >> >>>>>>>>>>>>>> >> >>>>> https://git-wip-us.apache.org/repos/asf/flink/commit/8c30f60 >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> The release artifacts to be voted on can be found at: >> >>>>>>>>>>>>>> http://people.apache.org/~mbalassi/flink-0.8.0-rc1/ >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> Release artifacts are signed with the following key: >> >>>>>>>>>>>>>> https://people.apache.org/keys/committer/mbalassi.asc >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> The staging repository for this release can be found >> >>> at: >> >>>>>>>>>>>>>> >> >>>>>>>>>> >> >>>>>>> >> >>>> >> https://repository.apache.org/content/repositories/orgapacheflink-1021 >> >>>>>>>>>>>>>> >> >>>>> ------------------------------------------------------------- >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> Please vote on releasing this package as Apache Flink >> >>>>> 0.8.0. >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> The vote is open for the next 72 hours and passes if a >> >>>>>> majority >> >>>>>>>> of >> >>>>>>>>> at >> >>>>>>>>>>>>>> least >> >>>>>>>>>>>>>> three +1 PMC votes are cast. >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> [ ] +1 Release this package as Apache Flink 0.8.0 >> >>>>>>>>>>>>>> [ ] -1 Do not release this package because ... >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> >> >>>>>>>>>>>> >> >>>>>>>>>>> >> >>>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>> >> >>>>>>>> >> >>>>>>> >> >>>>>> >> >>>>> >> >>>> >> >>> >> >>
