Thanks for the update! Just to be clear, you're proposing going forward with the "simple fix" of reverting FLINK-7294?
> On 1. Dec 2017, at 18:39, Eron Wright <eronwri...@gmail.com> wrote: > > Update on reported Mesos issue (FLINK-8174): > > TLDR; a PR will be ready within 24 hours that will undo reservation support. > > A couple of months ago, a fix (FLINK-7294) was merged related to how Flink > accepts Mesos resource offers. The intention was to allow Flink to make > use of so-called +reserved+ resources, a Mesos feature which makes it > possible to reserve hosts for use by a specific framework/role. The fix > inadvertently regressed the ability to use +unreserved+ resources. This is > a serious regression because unreserved resources are the common case. > > The simple solution is to revert the earlier fix, deferring support for > reservations to another release. We are spending some time to find a fix > that works for all scenarios, but seems unlikely at this time. I am > reaching out to the original contributor to get their feedback. > > In the course of the investigation, a related flaw was discovered in Fenzo > that causes Flink to misinterpret offers that contain a mix of reserved and > unreserved resources. I believe that a small fix is possible purely > within Flink; an update to Fenzo does not appear necessary. > > Going forward, we will contribute an improved integration test suite with > which to test Flink under diverse Mesos conditions (e.g. reservations). > > Thanks, > Eron > > On Thu, Nov 30, 2017 at 9:47 PM, Tzu-Li (Gordon) Tai <tzuli...@apache.org> > wrote: > >> Hi, >> >> I’ve noticed a behavioral regression in the Kafka producer, that should >> also be considered a blocker: https://issues.apache.org/ >> jira/browse/FLINK-8181 >> There’s already a PR for the issue here: https://github.com/ >> apache/flink/pull/5108 >> >> Best, >> Gordon >> >> On 30 November 2017 at 5:27:22 PM, Fabian Hueske (fhue...@gmail.com) >> wrote: >> >> I've created a JIRA issue for the the Hadoop 2.9.0 build problem [1]. >> >> Best, Fabian >> >> [1] https://issues.apache.org/jira/browse/FLINK-8177 >> >> 2017-11-30 4:35 GMT+01:00 Eron Wright <eronwri...@gmail.com>: >> >>> Unfortunately we've identified a blocker bug for Flink on Mesos - >>> FLINK-8174. We'll have a patch ready on Thursday. >>> >>> Thanks, >>> Eron >>> >>> On Wed, Nov 29, 2017 at 3:40 PM, Eron Wright <eronwri...@gmail.com> >> wrote: >>> >>>> On Dell EMC side, we're testing the RC2 on DCOS 1.10.0. Seeing a >>>> potential issue with offer acceptance and we'll update the thread with >> a >>> +1 >>>> or with a more concrete issue within 24 hours. >>>> >>>> Thanks, >>>> Eron >>>> >>>> On Wed, Nov 29, 2017 at 6:54 AM, Chesnay Schepler <ches...@apache.org> >>>> wrote: >>>> >>>>> I don't think anyone has taken a look yet, nor was there a discussion >> as >>>>> to postponing it. >>>>> >>>>> It just slipped through the cracks i guess... >>>>> >>>>> >>>>> On 29.11.2017 15:47, Gyula Fóra wrote: >>>>> >>>>>> Hi guys, >>>>>> I ran into this again while playing with savepoint/restore >> parallelism: >>>>>> >>>>>> https://issues.apache.org/jira/browse/FLINK-7595 >>>>>> https://github.com/apache/flink/pull/4651 >>>>>> >>>>>> Anyone has some idea about the status of this PR or were we planning >> to >>>>>> postpone this to 1.5? >>>>>> >>>>>> Thanks, >>>>>> Gyula >>>>>> >>>>>> >>>>>> Fabian Hueske <fhue...@gmail.com> ezt írta (időpont: 2017. nov. 29., >>>>>> Sze, >>>>>> 13:10): >>>>>> >>>>>> OK, the situation is the following: >>>>>>> >>>>>>> The test class (org.apache.flink.yarn.UtilsTest) implements a >> Hadoop >>>>>>> interface (Container) that was extended in Hadoop 2.9.0 by a getter >>> and >>>>>>> setter. >>>>>>> By adding the methods, we can compile Flink for Hadoop 2.9.0. >> However, >>>>>>> the >>>>>>> getter/setter add a dependency on a class that was also added in >>> Hadoop >>>>>>> 2.9.0. >>>>>>> Therefore, the implementation is not backwards compatible with >> Hadoop >>>>>>> versions < 2.9.0. >>>>>>> >>>>>>> Not sure how we can fix the problem. We would need two version of >> the >>>>>>> class >>>>>>> that are chosen based on the Hadoop version. Do we have something >> like >>>>>>> that >>>>>>> somewhere else? >>>>>>> >>>>>>> Since this is only a problem in a test class, Flink 1.4.0 might >> still >>>>>>> work >>>>>>> very well with Hadoop 2.9.0. >>>>>>> However, this has not been tested AFAIK. >>>>>>> >>>>>>> Cheers, Fabian >>>>>>> >>>>>>> 2017-11-29 12:47 GMT+01:00 Fabian Hueske <fhue...@gmail.com>: >>>>>>> >>>>>>> I just tried to build the release-1.4 branch for Hadoop 2.9.0 >>> (released >>>>>>>> a >>>>>>>> few days ago) and got a compilation failure in a test class. >>>>>>>> >>>>>>>> Right now, I'm assessing how much we need to fix to support Hadoop >>>>>>>> 2.9.0. >>>>>>>> I'll report later. >>>>>>>> >>>>>>>> Best, Fabian >>>>>>>> >>>>>>>> 2017-11-29 11:16 GMT+01:00 Aljoscha Krettek <aljos...@apache.org>: >>>>>>>> >>>>>>>> Agreed, this is a regression compared to the previous >> functionality. >>> I >>>>>>>>> updated the issue to "Blocker". >>>>>>>>> >>>>>>>>> On 29. Nov 2017, at 10:01, Gyula Fóra <gyula.f...@gmail.com> >> wrote: >>>>>>>>>> >>>>>>>>>> Hi all, >>>>>>>>>> >>>>>>>>>> I have found the following issue: >>>>>>>>>> https://issues.apache.org/jira/browse/FLINK-8165 >>>>>>>>>> >>>>>>>>>> I would say this is a blocker (I personally pass the >> ParameterTool >>>>>>>>>> all >>>>>>>>>> >>>>>>>>> over >>>>>>>>> >>>>>>>>>> the place in my production apps), but a pretty trivial issue to >>> fix, >>>>>>>>>> >>>>>>>>> we >>>>>>> >>>>>>>> can >>>>>>>>> >>>>>>>>>> wait a little to find other potential problems. >>>>>>>>>> >>>>>>>>>> I can submit a fix in a little bit. >>>>>>>>>> >>>>>>>>>> Cheers, >>>>>>>>>> Gyula >>>>>>>>>> >>>>>>>>>> Tzu-Li (Gordon) Tai <tzuli...@apache.org> ezt írta (időpont: >> 2017. >>>>>>>>>> >>>>>>>>> nov. >>>>>>> >>>>>>>> 29., Sze, 9:23): >>>>>>>>>> >>>>>>>>>> +1 >>>>>>>>>>> >>>>>>>>>>> Verified: >>>>>>>>>>> - No missing release Maven artifacts >>>>>>>>>>> - Staged Apache source & binary convenience releases looks good >>>>>>>>>>> - NOTICE / LICENSE is correct, README is sane >>>>>>>>>>> - Built from source (macOS, Scala 2.11, Hadoop-free & Hadoop >> 2.8) >>>>>>>>>>> - Cluster testing on AWS EMR (see release-testing-doc for >>>>>>>>>>> >>>>>>>>>> configuration >>>>>>> >>>>>>>> details) >>>>>>>>>>> - Tested Kinesis / Elasticsearch connector (no dependency >> clashes >>> on >>>>>>>>>>> cluster execution, works locally in IDE) >>>>>>>>>>> >>>>>>>>>>> Thanks a lot for managing the release Aljoscha! >>>>>>>>>>> >>>>>>>>>>> Cheers, >>>>>>>>>>> Gordon >>>>>>>>>>> >>>>>>>>>>> On 28 November 2017 at 8:32:42 PM, Stefan Richter ( >>>>>>>>>>> s.rich...@data-artisans.com) wrote: >>>>>>>>>>> >>>>>>>>>>> +1 (non-binding) >>>>>>>>>>> >>>>>>>>>>> I tested Flink in a cluster setup on Google Cloud, YARN-per-job, >>>>>>>>>>> >>>>>>>>>> checked >>>>>>>>> >>>>>>>>>> that for all backends that HA, recovery, at-least-once, >> end-to-end >>>>>>>>>>> >>>>>>>>>> exactly >>>>>>>>> >>>>>>>>>> once (with Kafka11 Producer), savepoints, externalized >> checkpoints, >>>>>>>>>>> >>>>>>>>>> and >>>>>>> >>>>>>>> rescaling work correctly. >>>>>>>>>>> >>>>>>>>>>> Am 28.11.2017 um 11:47 schrieb Aljoscha Krettek < >>>>>>>>>>>> >>>>>>>>>>> aljos...@apache.org >>>>>>> >>>>>>>> : >>>>>>>>>> >>>>>>>>>>> +1 >>>>>>>>>>>> >>>>>>>>>>>> Verified: >>>>>>>>>>>> - NOTICE and LICENSE are correct >>>>>>>>>>>> - source doesn't contain binaries >>>>>>>>>>>> - verified signatures >>>>>>>>>>>> - verified hashes >>>>>>>>>>>> - cluster testing on AWS and Cloudera VM (with Kerberos) (see >>>>>>>>>>>> >>>>>>>>>>> release-testing doc) >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On 28. Nov 2017, at 11:20, Aljoscha Krettek < >> aljos...@apache.org >>>> >>>>>>>>>>>>> >>>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> Phew, thanks for the update! >>>>>>>>>>>>> >>>>>>>>>>>>> On 28. Nov 2017, at 11:19, Gyula Fóra <gyf...@apache.org> >>> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> Ok seems like I had to remove the snappy jar as it was >>> corrupted >>>>>>>>>>>>>> >>>>>>>>>>>>> (makes >>>>>>>>> >>>>>>>>>> total sense) :P >>>>>>>>>>>>>> >>>>>>>>>>>>>> Gyula Fóra <gyf...@apache.org> ezt írta (időpont: 2017. nov. >>>>>>>>>>>>>> 28., >>>>>>>>>>>>>> >>>>>>>>>>>>> K, >>>>>>>>> >>>>>>>>>> 11:13): >>>>>>>>>>> >>>>>>>>>>>> Hi Aljoscha, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thanks for the release candidate. I am having a hard time >>>>>>>>>>>>>>> >>>>>>>>>>>>>> building >>>>>>> >>>>>>>> the rc, >>>>>>>>>>> >>>>>>>>>>>> I seem to get this error no matter what I do: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> [ERROR] Failed to execute goal >>>>>>>>>>>>>>> org.apache.maven.plugins:maven-shade-plugin:2.4.1:shade >>>>>>>>>>>>>>> >>>>>>>>>>>>>> (shade-hadoop) on >>>>>>>>>>> >>>>>>>>>>>> project flink-shaded-hadoop2-uber: Error creating shaded jar: >>>>>>>>>>>>>>> >>>>>>>>>>>>>> invalid >>>>>>>>> >>>>>>>>>> LOC >>>>>>>>>>> >>>>>>>>>>>> header (bad signature) -> [Help 1] >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> (Apache Maven 3.3.9) >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Any idea what I am missing? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>> Gyula >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Aljoscha Krettek <aljos...@apache.org> ezt írta (időpont: >>> 2017. >>>>>>>>>>>>>>> >>>>>>>>>>>>>> nov. >>>>>>>>> >>>>>>>>>> 27., >>>>>>>>>>> >>>>>>>>>>>> H, 19:35): >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi everyone, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Please review and vote on release candidate #2 for the >>> version >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> 1.4.0, as >>>>>>>>>>> >>>>>>>>>>>> follows: >>>>>>>>>>>>>>>> [ ] +1, Approve the release >>>>>>>>>>>>>>>> [ ] -1, Do not approve the release (please provide specific >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> comments) >>>>>>>>> >>>>>>>>>> >>>>>>>>>>>>>>>> The complete staging area is available for your review, >> which >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> includes: >>>>>>>>>>> >>>>>>>>>>>> * JIRA release notes [1], >>>>>>>>>>>>>>>> * the official Apache source release and binary convenience >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> releases >>>>>>>>> >>>>>>>>>> to >>>>>>>>>>> >>>>>>>>>>>> be deployed to dist.apache.org[2], which are signed with the >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> key >>>>>>> >>>>>>>> with >>>>>>>>>>> >>>>>>>>>>>> fingerprint F2A67A8047499BBB3908D17AA8F4FD97121D7293 [3], >>>>>>>>>>>>>>>> * all artifacts to be deployed to the Maven Central >>> Repository >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> [4], >>>>>>>>> >>>>>>>>>> * source code tag "release-1.4.0-rc1" [5], >>>>>>>>>>>>>>>> * website pull request listing the new release [6]. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Please have a careful look at the website PR because I >>> changed >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> some >>>>>>>>> >>>>>>>>>> wording and we're now also releasing a binary without Hadoop >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> dependencies. >>>>>>>>>>> >>>>>>>>>>>> Please use this document for coordinating testing efforts: [7] >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> The only change between RC1 and this RC2 is that the source >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> release >>>>>>>>> >>>>>>>>>> package does not include the erroneously included binary Ruby >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> dependencies >>>>>>>>>>> >>>>>>>>>>>> of the documentation anymore. Because of this I would like to >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> propose a >>>>>>>>>>> >>>>>>>>>>>> shorter voting time and close the vote around the time that RC1 >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> would have >>>>>>>>>>> >>>>>>>>>>>> closed. This would mean closing by end of Wednesday. Please let >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> me >>>>>>> >>>>>>>> know if >>>>>>>>>>> >>>>>>>>>>>> you disagree with this. The vote is adopted by majority >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> approval, >>>>>>> >>>>>>>> with at >>>>>>>>>>> >>>>>>>>>>>> least 3 PMC affirmative votes. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>> Your friendly Release Manager >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> [1] >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> https://issues.apache.org/jira/secure/ReleaseNote.jspa? >> proje >>>>>>>>>>> >>>>>>>>>> ctId=12315522&version=12340533 >>>>>>>>> >>>>>>>>>> [2] http://people.apache.org/~aljoscha/flink-1.4.0-rc2/ >>>>>>>>>>>>>>>> [3] https://dist.apache.org/repos/dist/release/flink/KEYS >>>>>>>>>>>>>>>> [4] >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> https://repository.apache.org/ >> content/repositories/orgapache >>>>>>> flink-1140 >>>>>>> >>>>>>>> [5] >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> https://git-wip-us.apache.org/ >> repos/asf?p=flink.git;a=tag;h= >>>>>>>>>>> >>>>>>>>>> ea751b7b23b23446ed3fcdeed564bbe8bf4adf9c >>>>>>>>> >>>>>>>>>> [6] https://github.com/apache/flink-web/pull/95 >>>>>>>>>>>>>>>> [7] >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> https://docs.google.com/document/d/ >> 1HqYyrNoMSXwo8zBpZj7s39Uz >>>>>>>>>>> >>>>>>>>>> UdlFcFO8TRpHNZ_cl44/edit?usp=sharing >>>>>>>>> >>>>>>>>>> Pro-tip: you can create a settings.xml file with these contents: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> <settings> >>>>>>>>>>>>>>>> <activeProfiles> >>>>>>>>>>>>>>>> <activeProfile>flink-1.4.0</activeProfile> >>>>>>>>>>>>>>>> </activeProfiles> >>>>>>>>>>>>>>>> <profiles> >>>>>>>>>>>>>>>> <profile> >>>>>>>>>>>>>>>> <id>flink-1.4.0</id> >>>>>>>>>>>>>>>> <repositories> >>>>>>>>>>>>>>>> <repository> >>>>>>>>>>>>>>>> <id>flink-1.4.0</id> >>>>>>>>>>>>>>>> <url> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> https://repository.apache.org/ >> content/repositories/orgapache >>>>>>>>>>> >>>>>>>>>> flink-1140/ >>>>>>>>> >>>>>>>>>> </url> >>>>>>>>>>>>>>>> </repository> >>>>>>>>>>>>>>>> <repository> >>>>>>>>>>>>>>>> <id>archetype</id> >>>>>>>>>>>>>>>> <url> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> https://repository.apache.org/ >> content/repositories/orgapache >>>>>>>>>>> >>>>>>>>>> flink-1140/ >>>>>>>>> >>>>>>>>>> </url> >>>>>>>>>>>>>>>> </repository> >>>>>>>>>>>>>>>> </repositories> >>>>>>>>>>>>>>>> </profile> >>>>>>>>>>>>>>>> </profiles> >>>>>>>>>>>>>>>> </settings> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> And reference that in you maven commands via --settings >>>>>>>>>>>>>>>> path/to/settings.xml. This is useful for creating a >>> quickstart >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> based >>>>>>>>> >>>>>>>>>> on the >>>>>>>>>>> >>>>>>>>>>>> staged release and for building against the staged jars. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>> >>>>> >>>> >>> >>
