The issue FLINK-4904 (Add a limit for how much data may be spilled in checkpoint alignments) is doen for master and I am currently backporting it. Hope to finish that this week...
Stephan On Wed, Nov 2, 2016 at 5:03 PM, Till Rohrmann <till.rohrm...@gmail.com> wrote: > It might make sense to backport > > - [FLINK-4944] Replace Akka's death watch with own heartbeat on the TM > side: https://github.com/apache/flink/pull/2742 > > as well. This will allow us to activate the quarantine monitoring per > default in 1.1.4 without risking to kill all TMs in case of a JM failure. > > Cheers, > Till > > On Wed, Nov 2, 2016 at 11:43 AM, Ufuk Celebi <u...@apache.org> wrote: > > > As a quick update: the "pending review" issues have all been resolved. > > > > The open issues are still open: > > > > - FLINK-4904: Add a limit for how much data may be spilled in > > checkpoint alignments => fix pending > > - FLINK-4910: Introduce safety net for closing file system streams > > > > Any updates here? > > > > – Ufuk > > > > > > On Fri, Oct 28, 2016 at 5:45 PM, Stefan Richter > > <s.rich...@data-artisans.com> wrote: > > > Benefit of a backport, as I see it, is increased stability. The danger > > is potentially breaking some code that was casting FileSystems to > subtypes > > like LocalFileSytem. I don’t know how common that would be in user code. > > > > > >> Am 28.10.2016 um 14:27 schrieb Ufuk Celebi <u...@apache.org>: > > >> > > >> Thanks for all your feedback. > > >> > > >> If there are no objections, I would like to stick to the mentioned > > >> issues in this thread and create RC1 as soon as they are all > > >> addressed. This will probably not be this week though, but it looks > > >> good for next week. > > >> > > >> DONE > > >> ===== > > >> - FLINK-4619: Answer client if savepoint restore fails > > >> - FLINK-4715: Safety net for stuck task cancellation > > >> - FLINK-4510: Always create CheckpointCoordinator > > >> - FLINK-4894: Don't block on buffer request after broadcast event > > >> - FLINK-4298: Add proper repository for Closure dependencies > > >> - FLINK-4218: Do not fail checkpoints when state size cannot be > > determined > > >> - FLINK-3347: TaskManager (or its ActorSystem) need to restart in case > > >> they notice quarantine > > >> - FLINK-4875: Use correct operator name > > >> - FLINK-4913: Include user jars in system class loader > > >> > > >> PENDING REVIEW > > >> =============== > > >> - FLINK-4445: Add option to ignore unmatched state when restoring from > > >> savepoint => https://github.com/apache/flink/pull/2713 > > >> - FLINK-4932: Don't let ExecutionGraph fail when in state Restarting > > >> => https://github.com/apache/flink/pull/2711 > > >> - FLINK-4933: ExecutionGraph.scheduleOrUpdateConsumers can fail the > > >> ExecutionGraph => https://github.com/apache/flink/pull/2701 > > >> > > >> OPEN > > >> ===== > > >> - FLINK-4904: Add a limit for how much data may be spilled in > > >> checkpoint alignments => fix pending > > >> - FLINK-4910: Introduce safety net for closing file system streams => > > >> @Stephan, Stefan: What's the conclusion of your discussion whether to > > >> backport this or not? > > >> > > >> > > >> On Wed, Oct 26, 2016 at 9:57 PM, dan bress <danbr...@gmail.com> > wrote: > > >>> +1 for this release, > > >>> also +1 to Chesnay's suggesting for including this: [FLINK-4875] > > [metrics] > > >>> Use correct operator name > > >>> > > >>> Dan > > >>> > > >>> On Wed, Oct 26, 2016 at 5:06 AM Till Rohrmann <trohrm...@apache.org> > > wrote: > > >>> > > >>>> I'll work on FLINK-3347. Additionally I would like to get in > > >>>> > > >>>> - https://issues.apache.org/jira/browse/FLINK-4932: Don't let > > >>>> ExecutionGraph fail when in state Restarting > > >>>> - https://issues.apache.org/jira/browse/FLINK-4933: > > >>>> ExecutionGraph.scheduleOrUpdateConsumers > > >>>> can fail the ExecutionGraph > > >>>> > > >>>> Cheers, > > >>>> Till > > >>>> > > >>>> On Wed, Oct 26, 2016 at 1:02 PM, Stephan Ewen <se...@apache.org> > > wrote: > > >>>> > > >>>>> Concerning backporting the "I/O streams safety net" - we need to > make > > >>>> sure > > >>>>> that this does not change any behavior that users may implicitly > > expect. > > >>>>> > > >>>>> > > >>>>> On Wed, Oct 26, 2016 at 11:21 AM, Maximilian Michels < > m...@apache.org > > > > > >>>>> wrote: > > >>>>> > > >>>>>> +1 for a 1.1.4 release > > >>>>>> > > >>>>>> We could backport putting user jars into the system class loader > for > > >>>>>> per-job Yarn clusters: https://github.com/apache/flink/pull/2692 > > >>>>>> Arguably, this is somewhat a new feature but it gets rid of > > duplicate > > >>>>>> class loading issues users experienced in practice. > > >>>>>> > > >>>>>> We already have the following commits on the release-1.1 branch: > > >>>>>> > > >>>>>> 05a5f46 [FLINK-4862] fix Timer register in > > ContinuousEventTimeTrigger > > >>>>>> 5731672 [FLINK-4581] [table] Fix Table API throwing "No suitable > > driver > > >>>>>> found for jdbc:calcite" > > >>>>>> 9c87f92 [FLINK-4586] [core] Broken AverageAccumulator > > >>>>>> 210230c [FLINK-4829] snapshot accumulators on a best-effort basis > > >>>>>> c1d6b24 [FLINK-4829] protect user accumulators against concurrent > > >>>> updates > > >>>>>> fe464b4 [FLINK-4709] [core] Fix resource leak in > > >>>>> InputStreamFSInputWrapper > > >>>>>> 9f72698 [FLINK-4108] [scala] Respect ResultTypeQueryable for > > >>>>> InputFormats. > > >>>>>> 9591d50 [FLINK-4506] [DataSet] Fix documentation of > CsvOutputFormat > > >>>> about > > >>>>>> incorrect default of allowNullValues > > >>>>>> c9433bf [FLINK-3706] Fix YARN test instability > > >>>>>> 2203f74 [FLINK-4778] [docs] Fix WordCount parameters in CLI > > examples. > > >>>>>> > > >>>>>> -Max > > >>>>>> > > >>>>>> > > >>>>>> On Wed, Oct 26, 2016 at 7:05 AM, Jean-Baptiste Onofré < > > j...@nanthrax.net > > >>>>> > > >>>>>> wrote: > > >>>>>>> +1 > > >>>>>>> > > >>>>>>> Looking forward this release ! > > >>>>>>> > > >>>>>>> Regards > > >>>>>>> JB > > >>>>>>> > > >>>>>>> > > >>>>>>> > > >>>>>>> On Oct 25, 2016, 14:43, at 14:43, Robert Metzger < > > >>>> rmetz...@apache.org> > > >>>>>> wrote: > > >>>>>>>> +1 for a bugfix release soon. > > >>>>>>>> > > >>>>>>>> On Tue, Oct 25, 2016 at 10:53 AM, Stephan Ewen < > se...@apache.org> > > >>>>>>>> wrote: > > >>>>>>>> > > >>>>>>>>> Thanks fort starting this Ufuk. > > >>>>>>>>> > > >>>>>>>>> I would like to add the following issues to 1.1.4: > > >>>>>>>>> > > >>>>>>>>> Build errors due to Storm dependencies *(fix pending)* > > >>>>>>>>> - [FLINK-4298] [storm compatibility] Add proper repository > for > > >>>>>>>> Closure > > >>>>>>>>> dependencies. > > >>>>>>>>> > > >>>>>>>>> Stability on S3 considering eventual consistency *(fix > pending)* > > >>>>>>>>> - [FLINK-4218] [checkpoints] Do not fail checkpoints when > > state > > >>>>>>>> size > > >>>>>>>>> cannot be determined > > >>>>>>>>> > > >>>>>>>>> Avoiding Zombie TaskManagers *(still needs to be done)* > > >>>>>>>>> - [FLINK-3347] [akka] TaskManager (or its ActorSystem) need > to > > >>>>>>>> restart > > >>>>>>>>> in case they notice quarantine > > >>>>>>>>> > > >>>>>>>>> Adding a limit to the amount of data spilled during checkpoint > > >>>>>>>> alignments > > >>>>>>>>> *(fix > > >>>>>>>>> is work in progress)* > > >>>>>>>>> - [FLINK-4904] [checkpoints] Add a limit for how much data > may > > >>>> be > > >>>>>>>>> spilled in checkpoint alignments > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> I can push the first two fixes to the 1.1.4 branch in a bit, > the > > >>>>>>>> fourth one > > >>>>>>>>> later today. > > >>>>>>>>> The third one (akka) is still pending. > > >>>>>>>>> > > >>>>>>>>> Best, > > >>>>>>>>> Stephan > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> On Mon, Oct 24, 2016 at 3:32 PM, Ufuk Celebi <u...@apache.org> > > >>>> wrote: > > >>>>>>>>> > > >>>>>>>>>> Hey all, > > >>>>>>>>>> > > >>>>>>>>>> I would like to start the discussion for kicking off the next > > bug > > >>>>>>>> fix > > >>>>>>>>>> release, Flink 1.1.4. What do you think about aiming for a RC > by > > >>>>>>>> end > > >>>>>>>>>> of this week? > > >>>>>>>>>> > > >>>>>>>>>> Users reported some instabilities/inconveniences that would be > > >>>> good > > >>>>>>>> to > > >>>>>>>>> fix. > > >>>>>>>>>> > > >>>>>>>>>> Personally, I would like to backport the following fixes: > > >>>>>>>>>> > > >>>>>>>>>> (1) https://issues.apache.org/jira/browse/FLINK-4619: Answer > > >>>>> client > > >>>>>>>> if > > >>>>>>>>>> savepoint restore fails (Already merged for master, needs > > minimal > > >>>>>>>>>> adjustment for 1.1) > > >>>>>>>>>> (2) https://issues.apache.org/jira/browse/FLINK-4715: Safety > > net > > >>>>>>>> for > > >>>>>>>>>> stuck task cancellation (Already reviewed for master, waiting > > for > > >>>>>>>>>> tests to finish of backport) > > >>>>>>>>>> (3) https://issues.apache.org/jira/browse/FLINK-4510: Always > > >>>>> create > > >>>>>>>>>> CheckpointCoordinator (Already merged for master, needs > minimal > > >>>>>>>>>> adjustments for 1.1) > > >>>>>>>>>> > > >>>>>>>>>> Furthermore, I would like to address the following: > > >>>>>>>>>> > > >>>>>>>>>> (4) https://issues.apache.org/jira/browse/FLINK-4445: Add > > option > > >>>>> to > > >>>>>>>>>> ignore unmatched state when restoring from savepoint > > >>>>>>>>>> (5) https://issues.apache.org/jira/browse/FLINK-4894: Don't > > >>>> block > > >>>>>>>> on > > >>>>>>>>>> buffer request after broadcast event > > >>>>>>>>>> > > >>>>>>>>>> Strictly speaking, the (4) is not a bug fix. But given that it > > >>>>>>>> would > > >>>>>>>>>> only add an optional flag to savepoint restoring and should > have > > >>>>>>>> been > > >>>>>>>>>> addressed for 1.1.0 already, I would like to get it in. > > >>>>>>>>>> > > >>>>>>>>> > > >>>>>> > > >>>>> > > >>>> > > > > > >
