[jira] [Created] (FLINK-20469) Enable TaskManager start and terminate in MiniCluster

[Andrey Zagrebin (Jira)]

Andrey Zagrebin created FLINK-20469:
---

 Summary: Enable TaskManager start and terminate in MiniCluster
 Key: FLINK-20469
 URL: https://issues.apache.org/jira/browse/FLINK-20469
 Project: Flink
  Issue Type: Improvement
  Components: Runtime / Coordination
Reporter: Andrey Zagrebin
Assignee: Andrey Zagrebin
 Fix For: 1.12.0


Currently we expose startTaskManager/terminateTaskManager only in internal 
TestingMiniCluster. Nonetheless, they are useful methods to implement IT cases 
similar to E2E tests.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-20468) Enable leadership control in MiniCluster to test JM failover

[Andrey Zagrebin (Jira)]

Andrey Zagrebin created FLINK-20468:
---

 Summary: Enable leadership control in MiniCluster to test JM 
failover
 Key: FLINK-20468
 URL: https://issues.apache.org/jira/browse/FLINK-20468
 Project: Flink
  Issue Type: Improvement
  Components: Runtime / Coordination
Reporter: Andrey Zagrebin
Assignee: Andrey Zagrebin
 Fix For: 1.12.0






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-20290) Duplicated output in FileSource continuous ITCase with TM failover

[Andrey Zagrebin (Jira)]

Andrey Zagrebin created FLINK-20290:
---

 Summary: Duplicated output in FileSource continuous ITCase with TM 
failover
 Key: FLINK-20290
 URL: https://issues.apache.org/jira/browse/FLINK-20290
 Project: Flink
  Issue Type: Bug
  Components: Connectors / FileSystem
Affects Versions: 1.12.0
Reporter: Andrey Zagrebin


If FileSourceTextLinesITCase::testContinuousTextFileSource includes TM restarts 
(after failing TM with TestingMiniCluster::terminateTaskExecutor, see 
testContinuousTextFileSourceWithTaskManagerFailover in 
[branch|https://github.com/azagrebin/flink/tree/FLINK-20118-it]) then sometimes 
I observe duplicated lines in the output after running the test suite 5-10 
times in IDE:
{code:java}
Test 
testContinuousTextFileSourceWithTaskManagerFailover(org.apache.flink.connector.file.src.FileSourceTextLinesITCase)
 failed with:
java.lang.AssertionError: 
Expected: ["And by opposing end them?--To die,--to sleep,--", "And enterprises 
of great pith and moment,", "And lose the name of action.--Soft you now!", "And 
makes us rather bear those ills we have", "And thus the native hue of 
resolution", "Be all my sins remember'd.", "But that the dread of something 
after death,--", "Devoutly to be wish'd. To die,--to sleep;--", "For in that 
sleep of death what dreams may come,", "For who would bear the whips and scorns 
of time,", "Is sicklied o'er with the pale cast of thought;", "Must give us 
pause: there's the respect", "No more; and by a sleep to say we end", "No 
traveller returns,--puzzles the will,", "Or to take arms against a sea of 
troubles,", "Than fly to others that we know not of?", "That flesh is heir 
to,--'tis a consummation", "That makes calamity of so long life;", "That 
patient merit of the unworthy takes,", "The fair Ophelia!--Nymph, in thy 
orisons", "The heartache, and the thousand natural shocks", "The insolence of 
office, and the spurns", "The oppressor's wrong, the proud man's contumely,", 
"The pangs of despis'd love, the law's delay,", "The slings and arrows of 
outrageous fortune", "The undiscover'd country, from whose bourn", "Thus 
conscience does make cowards of us all;", "To be, or not to be,--that is the 
question:--", "To grunt and sweat under a weary life,", "To sleep! perchance to 
dream:--ay, there's the rub;", "When he himself might his quietus make", "When 
we have shuffled off this mortal coil,", "Whether 'tis nobler in the mind to 
suffer", "With a bare bodkin? who would these fardels bear,", "With this 
regard, their currents turn awry,"]
 but: was ["And by opposing end them?--To die,--to sleep,--", "And 
enterprises of great pith and moment,", "And lose the name of action.--Soft you 
now!", "And makes us rather bear those ills we have", "And thus the native hue 
of resolution", "Be all my sins remember'd.", "But that the dread of something 
after death,--", "Devoutly to be wish'd. To die,--to sleep;--", "Devoutly to be 
wish'd. To die,--to sleep;--", "For in that sleep of death what dreams may 
come,", "For who would bear the whips and scorns of time,", "Is sicklied o'er 
with the pale cast of thought;", "Must give us pause: there's the respect", "No 
more; and by a sleep to say we end", "No more; and by a sleep to say we end", 
"No traveller returns,--puzzles the will,", "Or to take arms against a sea of 
troubles,", "Than fly to others that we know not of?", "That flesh is heir 
to,--'tis a consummation", "That flesh is heir to,--'tis a consummation", "That 
makes calamity of so long life;", "The fair Ophelia!--Nymph, in thy orisons", 
"The heartache, and the thousand natural shocks", "The heartache, and the 
thousand natural shocks", "The slings and arrows of outrageous fortune", "The 
undiscover'd country, from whose bourn", "Thus conscience does make cowards of 
us all;", "To be, or not to be,--that is the question:--", "To grunt and sweat 
under a weary life,", "To sleep! perchance to dream:--ay, there's the rub;", 
"To sleep! perchance to dream:--ay, there's the rub;", "When we have shuffled 
off this mortal coil,", "Whether 'tis nobler in the mind to suffer", "With a 
bare bodkin? who would these fardels bear,", "With this regard, their curre

[jira] [Created] (FLINK-20261) Uncaught exception in ExecutorNotifier due to split assignment broken by failed task

[Andrey Zagrebin (Jira)]

Andrey Zagrebin created FLINK-20261:
---

 Summary: Uncaught exception in ExecutorNotifier due to split 
assignment broken by failed task
 Key: FLINK-20261
 URL: https://issues.apache.org/jira/browse/FLINK-20261
 Project: Flink
  Issue Type: Bug
  Components: Connectors / FileSystem
Affects Versions: 1.12.0
Reporter: Andrey Zagrebin


While trying to extend FileSourceTextLinesITCase::testContinuousTextFileSource 
with recovery test after TM failure (TestingMiniCluster::terminateTaskExecutor, 
[branch|https://github.com/azagrebin/flink/tree/FLINK-20118-it]), I encountered 
the following case:
* SourceCoordinatorContext::assignSplits schedules async assignment (all reader 
tasks alive)
* call TestingMiniCluster::terminateTaskExecutor while doing writeFile in a 
loop of testContinuousTextFileSource
* causes graceful TaskExecutor::onStop shutdown
* causes TM/RM disconnect and failing slot allocations in JM by RM
* eventually causes SourceCoordinatorContext::unregisterSourceReader
* actual assignment starts (SourceCoordinatorContext::assignSplits: 
callInCoordinatorThread)
* registeredReaders.containsKey(subtaskId) check fails with 
IllegalArgumentException which is uncaught in single thread executor
* forces ThreadPool to recreate the single thread
* calls CoordinatorExecutorThreadFactory::newThread
* fails expected condition of single thread creation with IllegalStateException 
which is uncaught
* calls FatalExitExceptionHandler and exits JVM abruptly



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-20171) Improve error message for Flink process memory configuration

[Andrey Zagrebin (Jira)]

Andrey Zagrebin created FLINK-20171:
---

 Summary: Improve error message for Flink process memory 
configuration
 Key: FLINK-20171
 URL: https://issues.apache.org/jira/browse/FLINK-20171
 Project: Flink
  Issue Type: Improvement
  Components: Runtime / Coordination
Affects Versions: 1.11.0
Reporter: Andrey Zagrebin
Assignee: Andrey Zagrebin
 Fix For: 1.12.0, 1.11.0


Currently, all configuration failures will result in 
IllegalConfigurationException from JobManagerProcessUtils and 
TaskExecutorProcessUtils. The exception error messages do not refer to the 
process type (JM or TM), it can only become clear from the stack trace.

We can wrap main configuration calls with extra try/catch 
(TaskExecutorProcessUtils::processSpecFromConfig and 
JobManagerProcessUtils::processSpecFromConfigWithNewOptionToInterpretLegacyHeap)
 where IllegalConfigurationException is wrapped into another one which states 
type of the process (JM or TM).



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-20078) Factor out an ExecutionGraph factory method for DefaultExecutionTopology

[Andrey Zagrebin (Jira)]

Andrey Zagrebin created FLINK-20078:
---

 Summary: Factor out an ExecutionGraph factory method for 
DefaultExecutionTopology
 Key: FLINK-20078
 URL: https://issues.apache.org/jira/browse/FLINK-20078
 Project: Flink
  Issue Type: Improvement
  Components: Runtime / Coordination
Reporter: Andrey Zagrebin


Based on [this PR 
discussion|https://github.com/apache/flink/pull/13958#discussion_r519676104].



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-19954) Move execution deployment tracking logic from legacy EG code to SchedulerNG

[Andrey Zagrebin (Jira)]

Andrey Zagrebin created FLINK-19954:
---

 Summary: Move execution deployment tracking logic from legacy EG 
code to SchedulerNG
 Key: FLINK-19954
 URL: https://issues.apache.org/jira/browse/FLINK-19954
 Project: Flink
  Issue Type: Task
  Components: Runtime / Coordination
Affects Versions: 1.12.0
Reporter: Andrey Zagrebin


FLINK-17075 introduced the execution state reconciliation between TM and JM. 
The reconciliation requires tracking of the execution deployment state. The 
tracking logic was added to the legacy code of EG state handling which is 
partially inactive as discussed in FLINK-19927. The recent state handling logic 
resides in the new SchedulerNG, currently DefaultScheduler.

We could reconsider how the execution tracking for reconciliation is integrated 
with the scheduling. I think the tracking logic could be moved from 
Execution#deploy and EG#notifyExecutionChange to either 
SchedulerNG#updateTaskExecutionState or DefaultScheduler#deployTaskSafe. The 
latter looks to me currently more natural. ExecutionVertexOperations.deploy 
could return submission future for deployment completion in 
ExecutionDeploymentTracker and Execution#getTerminalFuture to stop the 
tracking. This would be also easier to unit test.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-19923) Remove BulkSlotProvider, its implementation and tests

[Andrey Zagrebin (Jira)]

Andrey Zagrebin created FLINK-19923:
---

 Summary: Remove BulkSlotProvider, its implementation and tests
 Key: FLINK-19923
 URL: https://issues.apache.org/jira/browse/FLINK-19923
 Project: Flink
  Issue Type: Task
  Components: Runtime / Coordination
Affects Versions: 1.12.0
Reporter: Andrey Zagrebin
Assignee: Andrey Zagrebin
 Fix For: 1.12.0


BulkSlotProvider is not used any more because it was introduced for the removed 
OneSlotAllocator replaced by SlotSharingExecutionSlotAllocator for the 
PipelinedRegionSchedulingStrategy.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-19918) RocksIncrementalCheckpointRescalingTest.testScalingDown fails on Windows

[Andrey Zagrebin (Jira)]

Andrey Zagrebin created FLINK-19918:
---

 Summary: RocksIncrementalCheckpointRescalingTest.testScalingDown 
fails on Windows
 Key: FLINK-19918
 URL: https://issues.apache.org/jira/browse/FLINK-19918
 Project: Flink
  Issue Type: Bug
  Components: Runtime / State Backends
Reporter: Andrey Zagrebin


{code:java}
java.lang.NullPointerExceptionjava.lang.NullPointerException at 
org.apache.flink.streaming.util.AbstractStreamOperatorTestHarness.close(AbstractStreamOperatorTestHarness.java:656)
 at 
org.apache.flink.contrib.streaming.state.RocksIncrementalCheckpointRescalingTest.closeHarness(RocksIncrementalCheckpointRescalingTest.java:357)
 at 
org.apache.flink.contrib.streaming.state.RocksIncrementalCheckpointRescalingTest.testScalingDown(RocksIncrementalCheckpointRescalingTest.java:276)
{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-19917) RocksDBInitTest.testTempLibFolderDeletedOnFail fails on Windows

[Andrey Zagrebin (Jira)]

Andrey Zagrebin created FLINK-19917:
---

 Summary:  RocksDBInitTest.testTempLibFolderDeletedOnFail fails on 
Windows
 Key: FLINK-19917
 URL: https://issues.apache.org/jira/browse/FLINK-19917
 Project: Flink
  Issue Type: Bug
  Components: Runtime / State Backends
Reporter: Andrey Zagrebin


{code:java}
java.lang.AssertionError: 
Expected :0
Actual   :2{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-19860) Consider skipping restart and traverse regions which are already being restarted in RestartPipelinedRegionFailoverStrategy

[Andrey Zagrebin (Jira)]

Andrey Zagrebin created FLINK-19860:
---

 Summary: Consider skipping restart and traverse regions which are 
already being restarted in RestartPipelinedRegionFailoverStrategy
 Key: FLINK-19860
 URL: https://issues.apache.org/jira/browse/FLINK-19860
 Project: Flink
  Issue Type: Improvement
  Components: Runtime / Coordination
Reporter: Andrey Zagrebin


Original 
[discussion|https://github.com/apache/flink/pull/13749#pullrequestreview-516385846].
 Follow-up for FLINK-19712.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-19832) Improve handling of immediately failed physical slot in SlotSharingExecutionSlotAllocator

[Andrey Zagrebin (Jira)]

Andrey Zagrebin created FLINK-19832:
---

 Summary: Improve handling of immediately failed physical slot in 
SlotSharingExecutionSlotAllocator
 Key: FLINK-19832
 URL: https://issues.apache.org/jira/browse/FLINK-19832
 Project: Flink
  Issue Type: Improvement
  Components: Runtime / Coordination
Affects Versions: 1.12.0
Reporter: Andrey Zagrebin
Assignee: Andrey Zagrebin


Improve handling of immediately failed physical slot in 
SlotSharingExecutionSlotAllocator

If a physical slot future the immediately fails for a new SharedSlot in 
SlotSharingExecutionSlotAllocator#getOrAllocateSharedSlot but we continue to 
add logical slots to this SharedSlot, eventually, the logical slot also fails 
and gets removed from {{the SharedSlot}} which gets released (state RELEASED). 
The subsequent logical slot addings in the loop of 
{{allocateLogicalSlotsFromSharedSlots}} will fail the scheduling
with the ALLOCATED state check because it will be RELEASED.

The subsequent bulk timeout check will also not find the SharedSlot and fail 
with NPE.

Hence, such SharedSlot with the immediately failed physical slot future should 
not be kept in the SlotSharingExecutionSlotAllocator and the logical slot 
requests depending on it can be immediately returned failed. The bulk timeout 
check does not need to be started because if some physical (and its logical) 
slot requests failed then the whole bulk will be canceled by scheduler.

If the last assumption is not true for the future scheduling, this bulk failure 
might need additional explicit pending requests cancelation. We expect to 
refactor it for the declarative scheduling anyways.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] NEW FLIP-102: Add More Metrics to TaskManager

[Andrey Zagrebin]

Thanks a lot for this nice UI guys!

+1 and for closed issues that is just because many steps have been
already done.

Best,
Andrey

On Fri, Oct 23, 2020 at 11:12 AM Till Rohrmann  wrote:

> Thanks for reviving this Flip Yadong! The changes look good to me and the
> new memory UI looks awesome :-)
>
> I think the reason why the REST issues are closed is because they are
> already done. In that sense some of the work already finished.
>
> +1 for adopting this FLIP and moving forward with updating the web UI
> accordingly.
>
> Cheers,
> Till
>
> On Fri, Oct 23, 2020 at 8:58 AM Jark Wu  wrote:
>
> > +1
> >
> > Thanks for the work.
> >
> > Best,
> > Jark
> >
> > On Fri, 23 Oct 2020 at 10:13, Xintong Song 
> wrote:
> >
> > > Thanks Yadong, Mattias and Lining for reviving this FLIP.
> > >
> > > I've seen so many users confused by the current webui page of task
> > manager
> > > metrics. This FLIP should definitely help them understand the memory
> > > footprints and tune the configurations for task managers.
> > >
> > > The design part of this proposal looks really good to me. The UI is
> clear
> > > and easy to understand. The metrics look correct to me.
> > >
> > > KIND REMINDER: I think the section `Implementation Proposal` in the
> FLIP
> > > doc needs to be updated, so that we can vote on this FLIP. Currently,
> all
> > > the tickets listed are closed.
> > >
> > > Thank you~
> > >
> > > Xintong Song
> > >
> > >
> > >
> > > On Thu, Oct 22, 2020 at 5:53 PM Yadong Xie 
> wrote:
> > >
> > > > Hi all
> > > >
> > > > I want to start a new vote for FLIP-102, which proposes to add more
> > > metrics
> > > > to the task manager in web UI.
> > > >
> > > > The new FLIP-102 was revisited and adapted following the old ML
> > > discussion
> > > > <
> > > >
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/VOTE-FLIP-102-Add-More-Metrics-to-TaskManager-td37898.html
> > > > >
> > > > .
> > > >
> > > > Thanks to Matthias and Lining's effort, more metrics are available.
> We
> > > can
> > > > match most of the effective configuration to the metrics just as
> Flink
> > > Doc
> > > > <
> > > >
> > >
> >
> https://ci.apache.org/projects/flink/flink-docs-release-1.11/ops/memory/mem_setup_tm.html#detailed-memory-model
> > > > >
> > > > describes now.
> > > >
> > > > The vote will last for at least 72 hours, following the consensus
> > voting.
> > > >
> > > >
> > > > FLIP-102 wiki:
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-102%3A+Add+More+Metrics+to+TaskManager
> > > >
> > > >
> > > > Discussion thread:
> > > >
> > > >
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/VOTE-FLIP-102-Add-More-Metrics-to-TaskManager-td37898.html
> > > >
> > > >
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-75-Flink-Web-UI-Improvement-Proposal-td33540.html
> > > >
> > > > Thanks,
> > > >
> > > > Yadong
> > > >
> > >
> >
>

Re: [VOTE] FLIP-141: Intra-Slot Managed Memory Sharing

[Andrey Zagrebin]

For the option name, maybe:
*flink.main*
or
*flink.managed* (this may be a bit confusing for existing users as we said
that the overall managed memory is managed by Flink)

On Wed, Sep 9, 2020 at 9:56 AM Andrey Zagrebin  wrote:

> +1
>
> Best,
> Andrey
>
> On Tue, Sep 8, 2020 at 2:16 PM Yu Li  wrote:
>
>> +1
>>
>> Best Regards,
>> Yu
>>
>>
>> On Tue, 8 Sep 2020 at 17:03, Aljoscha Krettek 
>> wrote:
>>
>> > +1
>> >
>> > We just need to make sure to find a good name before the release but
>> > shouldn't block any work on this.
>> >
>> > Aljoscha
>> >
>> > On 08.09.20 07:59, Xintong Song wrote:
>> > > Thanks for the vote, @Jincheng.
>> > >
>> > >
>> > > Concerning the namings, the original idea was, as you suggested, to
>> have
>> > > separate configuration names for batch and rocksdb while only one of
>> them
>> > > will take effect at a time.
>> > >
>> > >
>> > > It was then in the discussion thread [1] that @Stepahn suggested to
>> > combine
>> > > these two.
>> > >
>> > >>  We never have batch algos and RocksDB mixed, having this as
>> > separate
>> > >> options is confusing as it suggests this can be combined
>> arbitrarily. I
>> > >> also think that a slim possibility that we may ever combine this in
>> the
>> > >> future is not enough reason to make it more complex/confusing.
>> > >>
>> > >
>> > > This suggestion was also supported by others in the discussion thread.
>> > > That's why we are trying to come up with a name that covers both batch
>> > and
>> > > rocksdb memory consumers.
>> > >
>> > >
>> > > Thank you~
>> > >
>> > > Xintong Song
>> > >
>> > >
>> > > [1]
>> > >
>> >
>> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-141-Intra-Slot-Managed-Memory-Sharing-tp44146p44253.html
>> > >
>> > > On Tue, Sep 8, 2020 at 1:37 PM jincheng sun > >
>> > > wrote:
>> > >
>> > >> +1 for the proposal!
>> > >>
>> > >> Regarding the name of `BATCH_OP/ROCKSDB`, we can reserve the
>> > configuration
>> > >> names for batch and rocksdb respectively, ` batch_ OP` for batch,
>> > "ROCKSDB"
>> > >> for roockdb. and the default value as follows:
>> > >>
>> > >> {
>> > >>  BATCH_OP: 70,
>> > >>  ROCKSDB : 70,
>> > >>  PYTHON : 30
>> > >> }
>> > >>
>> > >> Only one of `BATCH_ OP` and `ROCKSDB` will work. What do you think?
>> > >>
>> > >> Best,
>> > >> Jincheng
>> > >>
>> > >>
>> > >> Xintong Song  于2020年9月7日周一 下午1:46写道：
>> > >>
>> > >>> Thanks for the votes.
>> > >>>
>> > >>> Concerning the name for batch/RocksDB memory consumer, how about
>> > >> "execution
>> > >>> memory"?
>> > >>> We can further explain in docs and config option description that
>> this
>> > is
>> > >>> used for job execution, which is currently dedicated to rocksdb in
>> > >>> streaming and batch algorithms in batch.
>> > >>>
>> > >>> Thank you~
>> > >>>
>> > >>> Xintong Song
>> > >>>
>> > >>>
>> > >>>
>> > >>> On Mon, Sep 7, 2020 at 11:43 AM Yangze Guo 
>> wrote:
>> > >>>
>> > >>>> +1
>> > >>>>
>> > >>>> Best,
>> > >>>> Yangze Guo
>> > >>>>
>> > >>>> On Mon, Sep 7, 2020 at 10:54 AM Zhu Zhu  wrote:
>> > >>>>>
>> > >>>>> +1
>> > >>>>>
>> > >>>>> Thanks,
>> > >>>>> Zhu
>> > >>>>>
>> > >>>>> Dian Fu  于2020年9月7日周一 上午10:34写道：
>> > >>>>>
>> > >>>>>> +1
>> > >>>>>>
>> > >>>>>>> 在 2020年9月3日，下午8:46，Till Rohrmann  写道：
>> > >>>>>>>
>> > >>>>>>> Hi Xintong,
>> > >>>>>>>
>> > >>>>>>> thanks for starting the vote.
>> > >>>>>>>
>> > >>>>>>> +1 for the proposal given that we find a proper name for the
>> > >>>>>>> different memory consumers (specifically the batch/RocksDB
>> > >>> consumer)
>> > >>>> and
>> > >>>>>>> their corresponding weights.
>> > >>>>>>>
>> > >>>>>>> Cheers,
>> > >>>>>>> Till
>> > >>>>>>>
>> > >>>>>>> On Thu, Sep 3, 2020 at 12:43 PM Xintong Song <
>> > >>> tonysong...@gmail.com>
>> > >>>>>> wrote:
>> > >>>>>>>
>> > >>>>>>>> Hi devs,
>> > >>>>>>>>
>> > >>>>>>>> I'd like to start a voting thread on FLIP-141[1], which
>> proposes
>> > >>> how
>> > >>>>>>>> managed memory should be shared by various use cases within a
>> > >>> slot.
>> > >>>> The
>> > >>>>>>>> proposal has been discussed in [2].
>> > >>>>>>>>
>> > >>>>>>>> The vote will be open for at least 72h + weekends. I'll try to
>> > >>>> close it
>> > >>>>>> on
>> > >>>>>>>> September 8, unless there is an objection or not enough votes.
>> > >>>>>>>>
>> > >>>>>>>> Thank you~
>> > >>>>>>>>
>> > >>>>>>>> Xintong Song
>> > >>>>>>>>
>> > >>>>>>>>
>> > >>>>>>>> [1]
>> > >>>>>>>>
>> > >>>>>>>>
>> > >>>>>>
>> > >>>>
>> > >>>
>> > >>
>> >
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-141%3A+Intra-Slot+Managed+Memory+Sharing#FLIP141:IntraSlotManagedMemorySharing-compatibility
>> > >>>>>>>>
>> > >>>>>>>> [2]
>> > >>>>>>>>
>> > >>>>>>>>
>> > >>>>>>
>> > >>>>
>> > >>>
>> > >>
>> >
>> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-141-Intra-Slot-Managed-Memory-Sharing-td44146.html
>> > >>>>>>>>
>> > >>>>>>
>> > >>>>>>
>> > >>>>
>> > >>>
>> > >>
>> > >
>> >
>> >
>>
>

Re: [VOTE] FLIP-141: Intra-Slot Managed Memory Sharing

[Andrey Zagrebin]

+1

Best,
Andrey

On Tue, Sep 8, 2020 at 2:16 PM Yu Li  wrote:

> +1
>
> Best Regards,
> Yu
>
>
> On Tue, 8 Sep 2020 at 17:03, Aljoscha Krettek  wrote:
>
> > +1
> >
> > We just need to make sure to find a good name before the release but
> > shouldn't block any work on this.
> >
> > Aljoscha
> >
> > On 08.09.20 07:59, Xintong Song wrote:
> > > Thanks for the vote, @Jincheng.
> > >
> > >
> > > Concerning the namings, the original idea was, as you suggested, to
> have
> > > separate configuration names for batch and rocksdb while only one of
> them
> > > will take effect at a time.
> > >
> > >
> > > It was then in the discussion thread [1] that @Stepahn suggested to
> > combine
> > > these two.
> > >
> > >>  We never have batch algos and RocksDB mixed, having this as
> > separate
> > >> options is confusing as it suggests this can be combined arbitrarily.
> I
> > >> also think that a slim possibility that we may ever combine this in
> the
> > >> future is not enough reason to make it more complex/confusing.
> > >>
> > >
> > > This suggestion was also supported by others in the discussion thread.
> > > That's why we are trying to come up with a name that covers both batch
> > and
> > > rocksdb memory consumers.
> > >
> > >
> > > Thank you~
> > >
> > > Xintong Song
> > >
> > >
> > > [1]
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-141-Intra-Slot-Managed-Memory-Sharing-tp44146p44253.html
> > >
> > > On Tue, Sep 8, 2020 at 1:37 PM jincheng sun 
> > > wrote:
> > >
> > >> +1 for the proposal!
> > >>
> > >> Regarding the name of `BATCH_OP/ROCKSDB`, we can reserve the
> > configuration
> > >> names for batch and rocksdb respectively, ` batch_ OP` for batch,
> > "ROCKSDB"
> > >> for roockdb. and the default value as follows:
> > >>
> > >> {
> > >>  BATCH_OP: 70,
> > >>  ROCKSDB : 70,
> > >>  PYTHON : 30
> > >> }
> > >>
> > >> Only one of `BATCH_ OP` and `ROCKSDB` will work. What do you think?
> > >>
> > >> Best,
> > >> Jincheng
> > >>
> > >>
> > >> Xintong Song  于2020年9月7日周一 下午1:46写道：
> > >>
> > >>> Thanks for the votes.
> > >>>
> > >>> Concerning the name for batch/RocksDB memory consumer, how about
> > >> "execution
> > >>> memory"?
> > >>> We can further explain in docs and config option description that
> this
> > is
> > >>> used for job execution, which is currently dedicated to rocksdb in
> > >>> streaming and batch algorithms in batch.
> > >>>
> > >>> Thank you~
> > >>>
> > >>> Xintong Song
> > >>>
> > >>>
> > >>>
> > >>> On Mon, Sep 7, 2020 at 11:43 AM Yangze Guo 
> wrote:
> > >>>
> >  +1
> > 
> >  Best,
> >  Yangze Guo
> > 
> >  On Mon, Sep 7, 2020 at 10:54 AM Zhu Zhu  wrote:
> > >
> > > +1
> > >
> > > Thanks,
> > > Zhu
> > >
> > > Dian Fu  于2020年9月7日周一 上午10:34写道：
> > >
> > >> +1
> > >>
> > >>> 在 2020年9月3日，下午8:46，Till Rohrmann  写道：
> > >>>
> > >>> Hi Xintong,
> > >>>
> > >>> thanks for starting the vote.
> > >>>
> > >>> +1 for the proposal given that we find a proper name for the
> > >>> different memory consumers (specifically the batch/RocksDB
> > >>> consumer)
> >  and
> > >>> their corresponding weights.
> > >>>
> > >>> Cheers,
> > >>> Till
> > >>>
> > >>> On Thu, Sep 3, 2020 at 12:43 PM Xintong Song <
> > >>> tonysong...@gmail.com>
> > >> wrote:
> > >>>
> >  Hi devs,
> > 
> >  I'd like to start a voting thread on FLIP-141[1], which proposes
> > >>> how
> >  managed memory should be shared by various use cases within a
> > >>> slot.
> >  The
> >  proposal has been discussed in [2].
> > 
> >  The vote will be open for at least 72h + weekends. I'll try to
> >  close it
> > >> on
> >  September 8, unless there is an objection or not enough votes.
> > 
> >  Thank you~
> > 
> >  Xintong Song
> > 
> > 
> >  [1]
> > 
> > 
> > >>
> > 
> > >>>
> > >>
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-141%3A+Intra-Slot+Managed+Memory+Sharing#FLIP141:IntraSlotManagedMemorySharing-compatibility
> > 
> >  [2]
> > 
> > 
> > >>
> > 
> > >>>
> > >>
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-141-Intra-Slot-Managed-Memory-Sharing-td44146.html
> > 
> > >>
> > >>
> > 
> > >>>
> > >>
> > >
> >
> >
>

[jira] [Created] (FLINK-19142) Investigate slot hijacking from preceding pipelined regions after failover

[Andrey Zagrebin (Jira)]

Andrey Zagrebin created FLINK-19142:
---

 Summary: Investigate slot hijacking from preceding pipelined 
regions after failover
 Key: FLINK-19142
 URL: https://issues.apache.org/jira/browse/FLINK-19142
 Project: Flink
  Issue Type: Improvement
Reporter: Andrey Zagrebin


The ticket originates from [this PR 
discussion|https://github.com/apache/flink/pull/13181#discussion_r481087221].

The previous AllocationIDs are used by PreviousAllocationSlotSelectionStrategy 
to schedule subtasks into the slot where they were previously executed before a 
failover. If the previous slot (AllocationID) is not available, we do not want 
subtasks to take previous slots (AllocationIDs) of other subtasks.

The MergingSharedSlotProfileRetriever gets all previous AllocationIDs of the 
bulk from SlotSharingExecutionSlotAllocator but only from the current bulk. The 
previous AllocationIDs of other bulks stay unknown. Therefore, the current bulk 
can potentially hijack the previous slots from the preceding bulks. On the 
other hand the previous AllocationIDs of other tasks should be taken if the 
other tasks are not going to run at the same time, e.g. not enough resources 
after failover or other bulks are done.

One way to do it may be to give to MergingSharedSlotProfileRetriever all 
previous AllocationIDs of bulks which are going to run at the same time.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: How jobmanager and task manager communicates with each other ?

[Andrey Zagrebin]

Hi Sidhant,

(1) If we are not using Flink's HA services then how we can dynamically
> configure task manager nodes to connect to job manager? Any suggestions or
> best practices?

Not sure what you mean by 'dynamically'.
I think you have to restart the task manager with the new configuration
to connect to another job manager.

(2) Which and how flink's HA service can be used for the service discovery
> of job manager ?

You can check the docs for the zookeeper implementation of the HA in Flink
[1]

Best,
Andrey

[1]
https://ci.apache.org/projects/flink/flink-docs-release-1.11/ops/jobmanager_high_availability.html

On Tue, Aug 25, 2020 at 5:45 PM sidhant gupta  wrote:

> Hi Till,
>
> Thanks for the reply.
>
> (1) If we are not using Flink's HA services then how we can dynamically
> configure task manager nodes to connect to job manager? Any suggestions or
> best practices?
>
> (2) Which and how flink's HA service can be used for the service discovery
> of job manager ?
>
> Regards
> Sidhant Gupta
>
>
> On Tue, Aug 25, 2020, 11:51 AM Till Rohrmann  wrote:
>
>> Hi Sidhant,
>>
>> the cluster components use tcp to communicate with each other. If you are
>> not using Flink's HA services, then the TaskManager nodes need to be
>> configured with the JobManager's address to connect to them. If you are
>> using HA services, then the service discovery happens through the HA
>> services. One requirement for Flink to work is that the different cluster
>> nodes on which a Flink process is started can communicate with each other.
>>
>> Cheers,
>> Till
>>
>> On Mon, Aug 24, 2020 at 6:26 PM sidhant gupta 
>> wrote:
>>
>>> ++dev@flink.apache.org
>>>
>>> On Mon, Aug 24, 2020, 7:31 PM sidhant gupta  wrote:
>>>
>>> > Hi User
>>> >
>>> > How jobmanager and task manager communicates with each other ? How to
>>> set
>>> > connection between jobmanager and task manager running in
>>> different/same
>>> > ec2 instance ? Is it http or tcp ? How the service discovery works ?
>>> >
>>> > Thanks
>>> > Sidhant Gupta
>>> >
>>>
>>

Re: [VOTE] FLIP-102: Add More Metrics to TaskManager

[Andrey Zagrebin]

Hi All,

Thanks for reviving the discussion, Matthias!

This would mean that we could adapt the current proposal to replace the
> Nonheap usage pane by a pane displaying the Metaspace usage.
>
I do not know the value of having the Nonheap usage in metrics. I can see
that the metaspace metric can be interesting for the users to debug OOMs.
We had the Nonheap usage before, so as discussed, I would be a bit careful
removing. I believe it deserves a separate poll in user ML
whether the Nonheap usage is useless or not.
As a current solution, we could keep both or merge them into one box with a
slash, like Metaspace/Nonheap -> 5Mb/10Mb, if the majority agrees that this
is not confusing and clear that the metaspace is a part of Nonheap.

Btw, the "Nonheap" in the configuration box of the current FLIP-102 is
probably incorrect or confusing as it does not one-to-one correspond to the
Nonheap JVM metric.

The only issue I see is that JVM Overhead would still not be represented in
> the memory usage
> overview.

My understanding is that we do not need a usage metric for JVM Overhead as
it is a virtual unmanaged component which is more about configuring the max
total process memory.

Is there a reason for us to introduce a nested structure
> TaskManagerMetricsInfo in the response object? I would rather keep it
> consistent in a flat structure instead, i.e. having all the members of
> TaskManagerResourceInfo being members of TaskManagerMetricsInfo

I would suggest introducing a separate REST call for
TaskManagerResourceInfo.
Semantically, TaskManagerResourceInfo is more about the TM configuration
and it is not directly related to the usage metrics.
In future, I would avoid having calls with many responsibilities and maybe
consider splitting the 'TM details' call into metrics etc unless there is a
concern for having to do more calls instead of one from UI.

Alternatively, one could think of grouping the metrics collecting the
> different values (i.e. max, used, committed) per metric in a JSON object.
> But this would apply for all the other metrics of TaskManagerMetricsInfo
> as
> well.

I would personally prefer this for metrics but I am not pushing for this.

metrics.resource.managedMemory and metrics.resource.networkMemory have
> counterparts in metrics.networkMemory[Used|Total] and
> metrics.managedMemory[Used|Total]: Is this redundant data or do they have
> different semantics?

As I understand, they have different semantics. The later is about
configuration, the former is about current usage metrics.

Is metrics.resource.totalProcessMemory a basic sum over all provided
> values?

this is again about configuration, I do not think it makes sense to come up
with a usage metric for the totalProcessMemory component.

Best,
Andrey


On Thu, Aug 20, 2020 at 9:06 AM Matthias  wrote:

> Hi Jing,
> I recently joined Ververica and started looking into FLIP-102. I'm trying
> to
> figure out how we would implement the proposal on the backend side.
> I looked into the proposal for the REST API response and a few questions
> popped up:
> - Is there a reason for us to introduce a nested structure
> TaskManagerMetricsInfo in the response object? I would rather keep it
> consistent in a flat structure instead, i.e. having all the members of
> TaskManagerResourceInfo being members of TaskManagerMetricsInfo.
>   Alternatively, one could think of grouping the metrics collecting the
> different values (i.e. max, used, committed) per metric in a JSON object.
> But this would apply for all the other metrics of TaskManagerMetricsInfo as
> well.
> - metrics.resource.managedMemory and metrics.resource.networkMemory have
> counterparts in metrics.networkMemory[Used|Total] and
> metrics.managedMemory[Used|Total]: Is this redundant data or do they have
> different semantics?
> - Is metrics.resource.totalProcessMemory a basic sum over all provided
> values? I see the necessity to have this member if we decide to not provide
> the memory usage for all memory pools (e.g. providing Metaspace but leaving
> Code Cache and Compressed Class Space as Non-Heap pools out of the
> response). Otherwise, would it be worth it to remove this member from the
> response for simplicity reasons since we could sum up the memory on the
> frontend side?
>
> Best,
> Matthias
>
>
>
> --
> Sent from: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/
>

[jira] [Created] (FLINK-18957) Implement bulk fulfil-ability timeout tracking for shared slots

[Andrey Zagrebin (Jira)]

Andrey Zagrebin created FLINK-18957:
---

 Summary: Implement bulk fulfil-ability timeout tracking for shared 
slots
 Key: FLINK-18957
 URL: https://issues.apache.org/jira/browse/FLINK-18957
 Project: Flink
  Issue Type: Sub-task
  Components: Runtime / Coordination
Reporter: Andrey Zagrebin
Assignee: Andrey Zagrebin
 Fix For: 1.12.0


Track fulfil-ability of required physical slots for all SharedSlot(s) (no 
matter whether they are created at this bulk or not) with timeout. This ensures 
we will not wait indefinitely if the required slots for this bulk cannot be 
fully fulfilled at the same time.
 # Create a LogicalSlotRequestBulk to track all physical requests and logical 
slot requests (logical slot requests only which belong to the bulk)
 # Mark physical slot request fulfilled in LogicalSlotRequestBulk, once its 
future is done
 # If any physical slot request fails then clear the LogicalSlotRequestBulk to 
stop the fulfil-ability check
 # Schedule a fulfil-ability check in LogicalSlotRequestBulkChecker for the 
LogicalSlotRequestBulk
 # In case of timeout:
 # cancel/fail the logical slot futures of the bulk in SharedSlot(s)
 # remove



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Adding a new "Docker Images" component to Jira

[Andrey Zagrebin]

+1 for the consolidation

Best,
Andrey

On Fri, Aug 7, 2020 at 3:38 PM Till Rohrmann  wrote:

> +1 for unifying Deployment / Docker, Dockerfiles and Release System /
> Docker into Docker.
>
> Cheers,
> Till
>
> On Fri, Aug 7, 2020 at 12:18 PM Robert Metzger 
> wrote:
>
> > Hi all,
> >
> > we now have 3 components containing the word "docker":
> > - Deployment / Docker
> > <
> >
> https://issues.apache.org/jira/issues/?jql=project+%3D+FLINK+AND+component+%3D+%22Deployment+%2F+Docker%22
> > >
> > (63
> > issues)
> > - Dockerfiles
> > <
> >
> https://issues.apache.org/jira/issues/?jql=project+%3D+FLINK+AND+component+%3D+Dockerfiles
> > >
> > (12
> > issues)
> > - Release System / Docker
> > <
> >
> https://issues.apache.org/jira/issues/?jql=project+%3D+FLINK+AND+component+%3D+%22Release+System+%2F+Docker%22
> > >
> > (3
> > issues)
> >
> > I would suggest consolidating these three components into one, as there
> are
> > not that many tickets for this aspect of Flink.
> > Maybe we should just rename "Deployment / Docker" to "flink-docker", and
> > merge the two other components into it?
> >
> >
> > On Fri, Feb 21, 2020 at 11:47 AM Patrick Lucas 
> > wrote:
> >
> > > Thanks, Chesnay!
> > >
> > > On Fri, Feb 21, 2020 at 11:26 AM Chesnay Schepler 
> > > wrote:
> > >
> > > > I've added a "Release System / Docker" component.
> > > >
> > > > On 21/02/2020 11:19, Patrick Lucas wrote:
> > > > > Hi,
> > > > >
> > > > > Could someone with permissions add a new component to the FLINK
> > project
> > > > in
> > > > > Jira for the Docker images  >?
> > > > >
> > > > > There is already a "Deployment / Docker" component, but that's not
> > > quite
> > > > > the same as maintenance/improvements on the flink-docker images.
> > > > >
> > > > > Either top-level "Docker Images" or perhaps "Release / Docker
> Images"
> > > > would
> > > > > be fine.
> > > > >
> > > > > Thanks,
> > > > > Patrick
> > > > >
> > > >
> > > >
> > >
> >
>

[jira] [Created] (FLINK-18751) Implement SlotSharingExecutionSlotAllocator

[Andrey Zagrebin (Jira)]

Andrey Zagrebin created FLINK-18751:
---

 Summary: Implement SlotSharingExecutionSlotAllocator
 Key: FLINK-18751
 URL: https://issues.apache.org/jira/browse/FLINK-18751
 Project: Flink
  Issue Type: Sub-task
  Components: Runtime / Coordination
Reporter: Andrey Zagrebin
Assignee: Andrey Zagrebin


SlotSharingExecutionSlotAllocator maintains a SharedSlot for each 
ExecutionSlotSharingGroup. SlotSharingExecutionSlotAllocator allocates physical 
slots for SharedSlot(s) and then allocates logical slots from it for scheduled 
tasks. In this way, the slot sharing hints can be respected in the 
ExecutionSlotAllocator. And we no longer need to rely on the SlotProvider to do 
the slot sharing matching. Co-location constraints will be respected since 
co-located subtasks will be in the same ExecutionSlotSharingGroup.

The physical slot will be lazily allocated for a SharedSlot, upon any hosted 
subtask asking for the SharedSlot. Each subsequent sharing subtask allocates a 
logical slot from the SharedSlot. The SharedSlot/physical slot can be released 
only if all the requested logical slots are released or canceled.
h4. Slot Allocation Process

When SlotSharingExecutionSlotAllocator receives a set of tasks to allocate 
slots for, it should do the following:
 # Map the tasks to ExecutionSlotSharingGroup(s)
 # Check which ExecutionSlotSharingGroup(s) _already have_ SharedSlot(s)
 # For all involved ExecutionSlotSharingGroup(s) _which do not have a 
SharedSlot_ yet.
 # Create a SlotProfile future by MergingSharedSlotProfileRetriever and then
 # Allocate a physical slot from the PhysicalSlotProvider
 # Create SharedSlot based on the returned physical slot futures
 # If physical slot future fails, remove the SharedSlot


 # Allocate logical slot futures for the tasks from all corresponding 
SharedSlot(s).
 # Generates SlotExecutionVertexAssignment(s)  based on the logical slot 
futures and returns the results.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-18739) Implement MergingSharedSlotProfileRetriever

[Andrey Zagrebin (Jira)]

Andrey Zagrebin created FLINK-18739:
---

 Summary: Implement MergingSharedSlotProfileRetriever
 Key: FLINK-18739
 URL: https://issues.apache.org/jira/browse/FLINK-18739
 Project: Flink
  Issue Type: Sub-task
  Components: Runtime / Coordination
Reporter: Andrey Zagrebin
Assignee: Andrey Zagrebin


Input location preferences will be considered for each SharedSlot when 
allocating a physical slot for it. Input location preferences of a SharedSlot 
will be the merge of input location preferences of all the tasks to run in this 
SharedSlot.

Inter-ExecutionSlotSharingGroup input location preferences can be respected in 
this way for ExecutionSlotSharingGroups belonging to different bulks. If 
ExecutionSlotSharingGroups belong to the same bulk, the input location 
preferences are ignored because of possible cyclic dependencies. Later, we can 
optimise this case when the declarative resource management for reactive mode 
is ready.

Intra-ExecutionSlotSharingGroup input location preferences will also be 
respected when creating ExecutionSlotSharingGroup(s) in 
DefaultSlotSharingStrategy.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-18709) Implement PhysicalSlotProvider

[Andrey Zagrebin (Jira)]

Andrey Zagrebin created FLINK-18709:
---

 Summary: Implement PhysicalSlotProvider
 Key: FLINK-18709
 URL: https://issues.apache.org/jira/browse/FLINK-18709
 Project: Flink
  Issue Type: Sub-task
  Components: Runtime / Coordination
Reporter: Andrey Zagrebin
Assignee: Andrey Zagrebin


PhysicalSlotProviderImpl tries to allocate a physical slot from the available 
idle cached slots in SlotPool. If it is not possible, it requests a new slot 
from the SlotPool.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-18689) Deterministic Slot Sharing

[Andrey Zagrebin (Jira)]

Andrey Zagrebin created FLINK-18689:
---

 Summary: Deterministic Slot Sharing
 Key: FLINK-18689
 URL: https://issues.apache.org/jira/browse/FLINK-18689
 Project: Flink
  Issue Type: New Feature
  Components: Runtime / Coordination
Reporter: Andrey Zagrebin


[Design 
doc|https://docs.google.com/document/d/10CbCVDBJWafaFOovIXrR8nAr2BnZ_RAGFtUeTHiJplw]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-18690) Implement LocalInputPreferredSlotSharingStrategy

[Andrey Zagrebin (Jira)]

Andrey Zagrebin created FLINK-18690:
---

 Summary: Implement LocalInputPreferredSlotSharingStrategy
 Key: FLINK-18690
 URL: https://issues.apache.org/jira/browse/FLINK-18690
 Project: Flink
  Issue Type: Sub-task
Reporter: Andrey Zagrebin
Assignee: Zhu Zhu


Implement ExecutionSlotSharingGroup, SlotSharingStrategy and default 
LocalInputPreferredSlotSharingStrategy.

The default strategy would be LocalInputPreferredSlotSharingStrategy. It will 
try to reduce remote data exchanges. Subtasks, which are connected and belong 
to the same SlotSharingGroup, tend to be put in the same 
ExecutionSlotSharingGroup.

See [design 
doc|https://docs.google.com/document/d/10CbCVDBJWafaFOovIXrR8nAr2BnZ_RAGFtUeTHiJplw/edit#heading=h.t4vfmm4atqoy]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[REMINDER] Use 'starter' labels for Jira issues where it makes sense

[Andrey Zagrebin]

Hi Flink Devs,

I would like to remind you that we have a 'starter' label [1] to annotate
Jira issues which need a contribution and which are not very
complicated for the new contributors. The starter issues can be a good
opportunity for the new contributors who want to learn about Flink but do
not know where to start [2].

When you open a Jira issue, please, pay attention to whether it can be a
starter task.
Let's try to help on-boarding new contributors!

Cheers,
Andrey

[1]
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=154995514
 (Labels)
[2]
https://flink.apache.org/contributing/contribute-code.html#looking-for-what-to-contribute

[jira] [Created] (FLINK-18646) Managed memory released check can block RPC thread

[Andrey Zagrebin (Jira)]

Andrey Zagrebin created FLINK-18646:
---

 Summary: Managed memory released check can block RPC thread
 Key: FLINK-18646
 URL: https://issues.apache.org/jira/browse/FLINK-18646
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Task
Affects Versions: 1.11.0
Reporter: Andrey Zagrebin


UnsafeMemoryBudget#verifyEmpty, called on slot freeing, needs time to wait on 
GC of all allocated/released managed memory. If there are a lot of segments to 
GC then it can take time to finish the check. If slot freeing happens in RPC 
thread, the GC waiting can block it and TM risks to miss its heartbeat.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-18467) Document what can be reconfigured for state with TTL between job restarts

[Andrey Zagrebin (Jira)]

Andrey Zagrebin created FLINK-18467:
---

 Summary: Document what can be reconfigured for state with TTL 
between job restarts
 Key: FLINK-18467
 URL: https://issues.apache.org/jira/browse/FLINK-18467
 Project: Flink
  Issue Type: Task
  Components: Runtime / State Backends
Affects Versions: 1.8.4
Reporter: Andrey Zagrebin


changing whether the state has TTL or not is not easy as it requires migration
but changing how to treat the expiration timestamp is possible, e.g. value of 
TTL or when to update/remove it



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-18454) Add a code contribution section about how to look for what to contribute

[Andrey Zagrebin (Jira)]

Andrey Zagrebin created FLINK-18454:
---

 Summary: Add a code contribution section about how to look for 
what to contribute
 Key: FLINK-18454
 URL: https://issues.apache.org/jira/browse/FLINK-18454
 Project: Flink
  Issue Type: Task
  Components: Project Website
Reporter: Andrey Zagrebin
Assignee: Andrey Zagrebin


This section is to give general advices about browsing open Jira issues and 
starter tasks.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-18309) Recommend avoiding uppercase to emphasise statements in doc style

[Andrey Zagrebin (Jira)]

Andrey Zagrebin created FLINK-18309:
---

 Summary: Recommend avoiding uppercase to emphasise statements in 
doc style
 Key: FLINK-18309
 URL: https://issues.apache.org/jira/browse/FLINK-18309
 Project: Flink
  Issue Type: Improvement
  Components: Project Website
Reporter: Andrey Zagrebin
Assignee: Andrey Zagrebin


Some contributions tend to use uppercase in user docs to highlight and/or 
emphasise statements. For example: "you MUST use the latest version". This 
style may appear somewhat aggressive to users.

Therefore, I suggest to add a recommendation to not use uppercase in user docs. 
We could highlight this statements as note paragraphs or with less 'shooting' 
style, e.g. italics to draw user attention.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-18308) KafkaProducerTestBase->Kafka011ProducerExactlyOnceITCase. testExactlyOnceCustomOperator hangs in Azure

[Andrey Zagrebin (Jira)]

Andrey Zagrebin created FLINK-18308:
---

 Summary: KafkaProducerTestBase->Kafka011ProducerExactlyOnceITCase. 
testExactlyOnceCustomOperator hangs in Azure
 Key: FLINK-18308
 URL: https://issues.apache.org/jira/browse/FLINK-18308
 Project: Flink
  Issue Type: Bug
  Components: Connectors / Kafka
Affects Versions: 1.11.0
Reporter: Andrey Zagrebin


[https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=3267=logs=d44f43ce-542c-597d-bf94-b0718c71e5e8=03dca39c-73e8-5aaf-601d-328ae5c35f20]

For last 3.5 hours, the test log ends with about 5000 entries like this:
{code:java}
2020-06-11T11:04:54.3299945Z 11:04:54,328 [FailingIdentityMapper Status 
Printer] INFO  
org.apache.flink.streaming.connectors.kafka.testutils.FailingIdentityMapper [] 
- > Failing mapper  0: count=690, 
totalCount=1000{code}
The problem was observed not on master but in [this 
PR|https://github.com/apache/flink/pull/12596]. The PR is simple fatal error 
handling refactoring in TM. Therefore, the PR looks unrelated. Another run of 
this PR in My Azure CI 
[passes|https://dev.azure.com/azagrebin/azagrebin/_build/results?buildId=214=results].



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-18250) Enrich OOM error messages with more details in ClusterEntrypoint

[Andrey Zagrebin (Jira)]

Andrey Zagrebin created FLINK-18250:
---

 Summary: Enrich OOM error messages with more details in 
ClusterEntrypoint
 Key: FLINK-18250
 URL: https://issues.apache.org/jira/browse/FLINK-18250
 Project: Flink
  Issue Type: Improvement
  Components: Runtime / Coordination
Reporter: Andrey Zagrebin
Assignee: Andrey Zagrebin
 Fix For: 1.11.0


Similar to what we added for TM in [https://github.com/apache/flink/pull/11408],

we should add more information and hints about out-of-memory failures to JM.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [ANNOUNCE] New Apache Flink Committer - Xintong Song

[Andrey Zagrebin]

Welcome to committers and congrats, Xintong!

Cheers,
Andrey

On Fri, Jun 5, 2020 at 4:22 PM Till Rohrmann  wrote:

> Congratulations!
>
> Cheers,
> Till
>
> On Fri, Jun 5, 2020 at 10:00 AM Dawid Wysakowicz 
> wrote:
>
> > Congratulations!
> >
> > Best,
> >
> > Dawid
> >
> > On 05/06/2020 09:10, tison wrote:
> > > Congrats, Xintong!
> > >
> > > Best,
> > > tison.
> > >
> > >
> > > Jark Wu  于2020年6月5日周五 下午3:00写道：
> > >
> > >> Congratulations Xintong!
> > >>
> > >> Best,
> > >> Jark
> > >>
> > >> On Fri, 5 Jun 2020 at 14:32, Danny Chan  wrote:
> > >>
> > >>> Congratulations Xintong !
> > >>>
> > >>> Best,
> > >>> Danny Chan
> > >>> 在 2020年6月5日 +0800 PM2:20，dev@flink.apache.org，写道：
> >  Congratulations Xintong
> >
>

[jira] [Created] (FLINK-17811) Update docker hub Flink page

[Andrey Zagrebin (Jira)]

Andrey Zagrebin created FLINK-17811:
---

 Summary: Update docker hub Flink page
 Key: FLINK-17811
 URL: https://issues.apache.org/jira/browse/FLINK-17811
 Project: Flink
  Issue Type: Task
  Components: Deployment / Docker
Reporter: Andrey Zagrebin


In FLINK-17161, we refactored the Flink docker images docs. We should also 
update and possibly link the related Flink docs about docker integration in 
[docker hub Flink image 
description|https://hub.docker.com/_/flink?tab=description].



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-17740) Remove flink-container/kubernetes

[Andrey Zagrebin (Jira)]

Andrey Zagrebin created FLINK-17740:
---

 Summary: Remove flink-container/kubernetes
 Key: FLINK-17740
 URL: https://issues.apache.org/jira/browse/FLINK-17740
 Project: Flink
  Issue Type: Sub-task
  Components: Deployment / Docker, Deployment / Kubernetes
Reporter: Andrey Zagrebin
Assignee: Chesnay Schepler
 Fix For: 1.11.0


FLINK-17161 added Kubernetes integration examples for Job Cluster.
FLINK-17656 copies job service yaml from flink-container/kubernetes to e2e 
Kubernetes Job Cluster test making them independent.
Therefore, we do not need flink-container/kubernetes and it can be removed.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-17652) Legacy JM heap options should fallback to new JVM_HEAP_MEMORY in standalone

[Andrey Zagrebin (Jira)]

Andrey Zagrebin created FLINK-17652:
---

 Summary: Legacy JM heap options should fallback to new 
JVM_HEAP_MEMORY in standalone
 Key: FLINK-17652
 URL: https://issues.apache.org/jira/browse/FLINK-17652
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Configuration, Runtime / Coordination
Reporter: Andrey Zagrebin
Assignee: Andrey Zagrebin
 Fix For: 1.11.0


FLINK-16742 states that the legacy JM heap options should fallback to 
JobManagerOptions.JVM_HEAP_MEMORY in standalone scripts. 
BashJavaUtils#getJmResourceParams has been implemented to fallback to 
JobManagerOptions.TOTAL_FLINK_MEMORY. This should be fixed.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-17546) Consider setting the number of TM CPU cores to the actual number of cores

[Andrey Zagrebin (Jira)]

Andrey Zagrebin created FLINK-17546:
---

 Summary: Consider setting the number of TM CPU cores to the actual 
number of cores
 Key: FLINK-17546
 URL: https://issues.apache.org/jira/browse/FLINK-17546
 Project: Flink
  Issue Type: Improvement
  Components: Runtime / Configuration
Reporter: Andrey Zagrebin


So far we do not use CPU cores resource in TaskExecutorResourceSpec. It was a 
preparation for dynamic slot/resource allocation (FLINK-14187). It is not fully 
clear how Flink or users would define the number of cores. We could consider 
setting the number of TM CPU cores to the actual number of cores by default, 
e.g. got somehow from OS in standalone or container configuration.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Re: The use of state ttl incremental cleanup strategy in sql deduplication resulting in significant performance degradation

[Andrey Zagrebin]

Hi lsyldliu,

You can try to tune the StateTtlConfig. As the documentation suggests [1]
the TTL incremental cleanup can decrease the per record performance. This
is the price of the automatic cleanup.
If the only thing, which happens mostly in your operator, is working with
state then even checking one additional record to cleanup is two times more
actions to do.
Timer approach was discussed in TTL feature design. It needs an additional
implementation and keeps more state but performs only one cleanup action
exactly when needed so it is a performance/storage trade-off.

Anyways, 20x degradation looks indeed a lot.
As a first step, I would suggest to configure the incremental cleanup
explicitly in `StateTtlConfigUtil#createTtlConfig` with a less entries to
check, e.g. 1 because processFirstRow/processLastRow already access the
state twice and do cleanup:

.cleanupIncrementally(1, false)


Also not sure but depending on the input data, finishBundle can happen
mostly during the snapshotting which slows down taking the checkpoint.
Could this fail the checkpoint accumulating the backpressure and slowing
down the pipeline?

Not sure why to keep the deduplication data in a Java map and in Flink
state at the same time, why not to keep it only in Flink state and
deduplicate on each incoming record?

Best,
Andrey

[1] note 2 in
https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/state/state.html#incremental-cleanup

On Wed, Apr 29, 2020 at 11:53 AM 刘大龙  wrote:

>
>
>
> > -原始邮件-
> > 发件人: "Jark Wu" 
> > 发送时间: 2020-04-29 14:09:44 (星期三)
> > 收件人: dev , "Yu Li" ,
> myas...@live.com
> > 抄送: azagre...@apache.org
> > 主题: Re: The use of state ttl incremental cleanup strategy in sql
> deduplication resulting in significant performance degradation
> >
> > Hi lsyldliu,
> >
> > Thanks for investigating this.
> >
> > First of all, if you are using mini-batch deduplication, it doesn't
> support
> > state ttl in 1.9. That's why the tps looks the same with 1.11 disable
> state
> > ttl.
> > We just introduce state ttl for mini-batch deduplication recently.
> >
> > Regarding to the performance regression, it looks very surprise to me.
> The
> > performance is reduced by 19x when StateTtlConfig is enabled in 1.11.
> > I don't have much experience of the underlying of StateTtlConfig. So I
> loop
> > in @Yu Li  @YunTang in CC who may have more insights
> on
> > this.
> >
> > For more information, we use the following StateTtlConfig [1] in blink
> > planner:
> >
> > StateTtlConfig
> >   .newBuilder(Time.milliseconds(retentionTime))
> >   .setUpdateType(StateTtlConfig.UpdateType.OnCreateAndWrite)
> >   .setStateVisibility(StateTtlConfig.StateVisibility.NeverReturnExpired)
> >   .build();
> >
> >
> > Best,
> > Jark
> >
> >
> > [1]:
> >
> https://github.com/apache/flink/blob/master/flink-table/flink-table-runtime-blink/src/main/java/org/apache/flink/table/runtime/util/StateTtlConfigUtil.java#L27
> >
> >
> >
> >
> >
> > On Wed, 29 Apr 2020 at 11:53, 刘大龙  wrote:
> >
> > > Hi, all!
> > >
> > > At flink master branch, we have supported state ttl  for sql mini-batch
> > > deduplication using incremental cleanup strategy on heap backend,
> refer to
> > > FLINK-16581. Because I want to test the performance of this feature,
> so I
> > > compile master branch code and deploy the jar to production
> > > environment，then run three types of tests, respectively:
> > >
> > >
> > >
> > >
> > > flink 1.9.0 release version enable state ttl
> > > flink 1.11-snapshot version disable state ttl
> > > flink 1.11-snapshot version enable state ttl
> > >
> > >
> > >
> > >
> > > The test query sql as follows:
> > >
> > > select order_date,
> > > sum(price * amount - goods_all_fav_amt - virtual_money_amt +
> > > goods_carriage_amt) as saleP,
> > > sum(amount) as saleN,
> > > count(distinct parent_sn) as orderN,
> > > count(distinct user_id) as cusN
> > >from(
> > > select order_date, user_id,
> > > order_type, order_status, terminal, last_update_time,
> > > goods_all_fav_amt,
> > > goods_carriage_amt, virtual_money_amt, price, amount,
> > > order_quality, quality_goods_cnt, acture_goods_amt
> > > from (select *, row_number() over(partition by order_id,
> > > order_goods_id order by proctime desc) as rownum from
> dm_trd_order_goods)
> > > where rownum=1
> > > and (order_type in (1,2,3,4,5) or order_status = 70)
> > > and terminal = 'shop' and price > 0)
> > > group by order_date
> > >
> > >
> > > At runtime, this query will generate two operators which include
> > > Deduplication and GroupAgg. In the test, the configuration is same,
> > > parallelism is 20, set kafka consumer from the earliest, and disable
> > > mini-batch function, The test results as follows:
> > >
> > > flink 1.9.0 enable state ttl：this test lasted 44m, flink receive 1374w
> > > records, average tps at 5200/s, Flink UI picture link back pressure,
> > > checkpoint
> > > flink 

[jira] [Created] (FLINK-17465) Update Chinese user documentation for job manager memory model

[Andrey Zagrebin (Jira)]

Andrey Zagrebin created FLINK-17465:
---

 Summary: Update Chinese user documentation for job manager memory 
model
 Key: FLINK-17465
 URL: https://issues.apache.org/jira/browse/FLINK-17465
 Project: Flink
  Issue Type: Sub-task
Reporter: Andrey Zagrebin
 Fix For: 1.11.0


This is a follow-up for FLINK-16946.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-17344) RecordWriterTest.testIdleTime possibly deadlocks on Travis

[Andrey Zagrebin (Jira)]

Andrey Zagrebin created FLINK-17344:
---

 Summary: RecordWriterTest.testIdleTime possibly deadlocks on Travis
 Key: FLINK-17344
 URL: https://issues.apache.org/jira/browse/FLINK-17344
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Network
Affects Versions: 1.11.0
Reporter: Andrey Zagrebin
 Fix For: 1.11.0


https://travis-ci.org/github/apache/flink/jobs/678193214
The test was introduced in 
[FLINK-16864|https://jira.apache.org/jira/browse/FLINK-16864].
It may be an instability as it passed 2 times (core and core-scala) and failed 
in core-hadoop:
https://travis-ci.org/github/apache/flink/builds/678193199



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-17167) Extend entry point script and docs with history server mode

[Andrey Zagrebin (Jira)]

Andrey Zagrebin created FLINK-17167:
---

 Summary: Extend entry point script and docs with history server 
mode
 Key: FLINK-17167
 URL: https://issues.apache.org/jira/browse/FLINK-17167
 Project: Flink
  Issue Type: Sub-task
  Components: Deployment / Docker
Reporter: Andrey Zagrebin
Assignee: Sebastian J.
 Fix For: 1.11.0






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-17166) Modify the log4j-console.properties to also output logs into the files for WebUI

[Andrey Zagrebin (Jira)]

Andrey Zagrebin created FLINK-17166:
---

 Summary: Modify the log4j-console.properties to also output logs 
into the files for WebUI
 Key: FLINK-17166
 URL: https://issues.apache.org/jira/browse/FLINK-17166
 Project: Flink
  Issue Type: Sub-task
  Components: Runtime / Configuration
Reporter: Andrey Zagrebin
 Fix For: 1.11.0






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-17165) Remove flink-container/docker

[Andrey Zagrebin (Jira)]

Andrey Zagrebin created FLINK-17165:
---

 Summary: Remove flink-container/docker
 Key: FLINK-17165
 URL: https://issues.apache.org/jira/browse/FLINK-17165
 Project: Flink
  Issue Type: Sub-task
  Components: Deployment / Docker
Reporter: Andrey Zagrebin
 Fix For: 1.11.0






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-17164) Extend entry point script and docs with job cluster mode and user job artefacts

[Andrey Zagrebin (Jira)]

Andrey Zagrebin created FLINK-17164:
---

 Summary: Extend entry point script and docs with job cluster mode 
and user job artefacts
 Key: FLINK-17164
 URL: https://issues.apache.org/jira/browse/FLINK-17164
 Project: Flink
  Issue Type: Sub-task
  Components: Deployment / Docker
Reporter: Andrey Zagrebin
 Fix For: 1.11.0






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-17163) Remove flink-contrib/docker-flink

[Andrey Zagrebin (Jira)]

Andrey Zagrebin created FLINK-17163:
---

 Summary: Remove flink-contrib/docker-flink
 Key: FLINK-17163
 URL: https://issues.apache.org/jira/browse/FLINK-17163
 Project: Flink
  Issue Type: Sub-task
  Components: Deployment / Docker
Reporter: Andrey Zagrebin
 Fix For: 1.11.0






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-17162) Document examples of how to extend the official docker hub image

[Andrey Zagrebin (Jira)]

Andrey Zagrebin created FLINK-17162:
---

 Summary: Document examples of how to extend the official docker 
hub image
 Key: FLINK-17162
 URL: https://issues.apache.org/jira/browse/FLINK-17162
 Project: Flink
  Issue Type: Sub-task
  Components: Deployment / Docker, Documentation
Reporter: Andrey Zagrebin
 Fix For: 1.11.0






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-17161) Document the official docker hub image and examples of how to run

[Andrey Zagrebin (Jira)]

Andrey Zagrebin created FLINK-17161:
---

 Summary: Document the official docker hub image and examples of 
how to run
 Key: FLINK-17161
 URL: https://issues.apache.org/jira/browse/FLINK-17161
 Project: Flink
  Issue Type: Sub-task
  Components: Deployment / Docker, Documentation
Reporter: Andrey Zagrebin
 Fix For: 1.11.0






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-17160) FLIP-111: Docker image unification

[Andrey Zagrebin (Jira)]

Andrey Zagrebin created FLINK-17160:
---

 Summary: FLIP-111: Docker image unification
 Key: FLINK-17160
 URL: https://issues.apache.org/jira/browse/FLINK-17160
 Project: Flink
  Issue Type: Improvement
  Components: Deployment / Docker, Dockerfiles
Reporter: Andrey Zagrebin
Assignee: Andrey Zagrebin
 Fix For: 1.11.0


This an umbrella issue for 
[FLIP-111.|https://cwiki.apache.org/confluence/display/FLINK/FLIP-111%3A+Docker+image+unification]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] FLIP-111: Docker image unification

[Andrey Zagrebin]

Hi all,

My vote is +1 (binding).

Thanks for participating and giving your votes.

Hereby, the vote is closed and the FLIP is accepted with no -1/vetos.

+1's:
3 binding (Ufuk, Till, Andrey)
4 non-binding (Yang, Canbin, Ismaël, Yangze)

Thanks,
Andrey

On Fri, Apr 10, 2020 at 8:40 AM Yangze Guo  wrote:

> +1 (non-binding)
> Thanks for driving this, Andrey.
>
> Best,
> Yangze Guo
>
> On Thu, Apr 9, 2020 at 11:33 PM Ismaël Mejía  wrote:
> >
> > +1 (non-binding)
> > Great work Andrey, pretty excited about this happening!
> >
> >
> > On Wed, Apr 8, 2020 at 4:20 AM Canbin Zheng 
> wrote:
> > >
> > > Thanks for the FLIP Andrey.
> > >
> > > +1 (non-binding) from my side.
> > >
> > > Regards,
> > > Canbin Zheng
> > >
> > > Yang Wang  于2020年4月8日周三 上午9:57写道：
> > >
> > > > Thanks Andrey for the efforts of docker image unification.
> > > >
> > > > +1 (non-binding)
> > > >
> > > > Best,
> > > > Yang
> > > >
> > > > Till Rohrmann  于2020年4月7日周二 下午11:04写道：
> > > >
> > > > > Thanks for driving this effort Andrey.
> > > > >
> > > > > +1 (binding)
> > > > >
> > > > > Cheers,
> > > > > Till
> > > > >
> > > > > On Tue, Apr 7, 2020 at 4:48 PM Ufuk Celebi  wrote:
> > > > >
> > > > > > Thanks for starting this FLIP.
> > > > > >
> > > > > > +1
> > > > > >
> > > > > > On Tue, Apr 7, 2020 at 11:29 AM Andrey Zagrebin <
> azagre...@apache.org>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi All,
> > > > > > >
> > > > > > > As discussed in these threads [1] and [2],
> > > > > > > we suggest to unify the docker topic in Flink for users [3].
> > > > > > >
> > > > > > > This mainly means refactoring of the existing code and
> introducing
> > > > more
> > > > > > > docs as a first step.
> > > > > > > The effort should enable further improvements and follow-ups
> for the
> > > > > > docker
> > > > > > > integration with Flink.
> > > > > > >
> > > > > > > The integration with docker in Flink is currently addressed in
> many
> > > > > > places
> > > > > > > which often intersect, repeat each other or apply different
> > > > approaches.
> > > > > > > This makes it really hard to follow the whole topic for users
> and
> > > > > > > maintainers. This FLIP suggests how to unify this topic. It
> means
> > > > > having
> > > > > > > one place which has the *Dockerfile*, all necessary scripts
> and docs
> > > > > > > following each other in a consistent way without any
> repetitions or
> > > > > > > contradictions.
> > > > > > >
> > > > > > > The idea is to keep all docker related resources in
> > > > apache/flink-docker
> > > > > > > <https://github.com/apache/flink-docker>. It already has a
> detailed
> > > > > > > Dockerfile which is well suited for common use cases or at
> least
> > > > serves
> > > > > > as
> > > > > > > a good starting point. The suggestion is to make it extensible
> for
> > > > > other
> > > > > > > concerns which are currently addressed in other places.
> > > > > > >
> > > > > > > Best,
> > > > > > > Andrey
> > > > > > >
> > > > > > > [1]
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-111-Docker-image-unification-td38444.html#a39822
> > > > > > > [2]
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-111-Docker-image-unification-td38444i20.html#a39950
> > > > > > > [3]
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-111%3A+Docker+image+unification
> > > > > > >
> > > > > >
> > > > >
> > > >
>

[jira] [Created] (FLINK-17048) Add memory related JVM args to Mesos JM startup scripts

[Andrey Zagrebin (Jira)]

Andrey Zagrebin created FLINK-17048:
---

 Summary: Add memory related JVM args to Mesos JM startup scripts
 Key: FLINK-17048
 URL: https://issues.apache.org/jira/browse/FLINK-17048
 Project: Flink
  Issue Type: Sub-task
  Components: Runtime / Configuration, Runtime / Coordination
Reporter: Andrey Zagrebin


It looks like we never respected memory configuration in Mesos JM startup 
scripts:
mesos-appmaster.sh
mesos-appmaster-job.sh

Now we have a chance to adopt FLIP-116 here as well, similar to what we are 
doing with standalone scripts in FLINK-16742



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[VOTE] FLIP-111: Docker image unification

[Andrey Zagrebin]

Hi All,

As discussed in these threads [1] and [2],
we suggest to unify the docker topic in Flink for users [3].

This mainly means refactoring of the existing code and introducing more
docs as a first step.
The effort should enable further improvements and follow-ups for the docker
integration with Flink.

The integration with docker in Flink is currently addressed in many places
which often intersect, repeat each other or apply different approaches.
This makes it really hard to follow the whole topic for users and
maintainers. This FLIP suggests how to unify this topic. It means having
one place which has the *Dockerfile*, all necessary scripts and docs
following each other in a consistent way without any repetitions or
contradictions.

The idea is to keep all docker related resources in apache/flink-docker
. It already has a detailed
Dockerfile which is well suited for common use cases or at least serves as
a good starting point. The suggestion is to make it extensible for other
concerns which are currently addressed in other places.

Best,
Andrey

[1]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-111-Docker-image-unification-td38444.html#a39822
[2]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-111-Docker-image-unification-td38444i20.html#a39950
[3]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-111%3A+Docker+image+unification

Re: [DISCUSS] FLIP-111: Docker image unification

[Andrey Zagrebin]

 will be overlap with things you already have taken
>> > into account.
>> >
>> >1. No more 'flink:latest' docker image tag.
>> >Related to https://issues.apache.org/jira/browse/FLINK-15794
>> >What I have learned is that the 'latest' version of a docker image
>> only
>> >makes sense IFF this is an almost standalone thing.
>> >So if I have a servlet that does something in isolation (like my
>> hobby
>> >project https://hub.docker.com/r/nielsbasjes/yauaa ) then 'latest'
>> > makes
>> >sense.
>> >With Flink you have the application code and all nodes in the cluster
>> >that are depending on each other and as such must run the exact same
>> >versions of the base software.
>> >So if you run flink in a cluster (local/yarn/k8s/mesos/swarm/...)
>> where
>> >the application and the nodes inter communicate and closely depend on
>> > each
>> >other then 'latest' is a bad idea.
>> >   1. Assume I have an application built against the Flink N api and
>> the
>> >   cluster downloads the latest which is also Flink N.
>> >   Then a week later Flink N+1 is released and the API I use changes
>> >   (Deprecated)
>> >   and a while later Flink N+2 is released and the deprecated API is
>> >   removed: Then my application no longer works even though I have
>> > not changed
>> >   anything.
>> >   So I want my application to be 'pinned' to the exact version I
>> built
>> >   it with.
>> >   2. I have a running cluster with my application and cluster
>> running
>> >   Flink N.
>> >   I add some additional nodes and the new nodes pick up the Flink
>> N+1
>> >   image ... now I have a cluster with mixed versions.
>> >   3. The version of flink is really the "Flink+Scala" version pair.
>> >   If you have the right flink but the wrong scala you get really
>> nasty
>> >   errors: https://issues.apache.org/jira/browse/FLINK-16289
>> >
>> >   2. Deploy SNAPSHOT docker images (i.e. something like
>> >*flink:1.11-SNAPSHOT_2.12*) .
>> >More and more use cases will be running on the code delivered via
>> Docker
>> >images instead of bare jar files.
>> >So if a "SNAPSHOT" is released and deployed into a 'staging' maven
>> repo
>> >(which may be locally on the developers workstation) then it is my
>> > opinion
>> >that at the same moment a "SNAPSHOT" docker image should be
>> >created/deployed.
>> >Each time a "SNAPSHOT" docker image is released this will overwrite
>> the
>> >previous "SNAPSHOT".
>> >If the final version is released the SNAPSHOTs of that version
>> >can/should be removed.
>> >This will make testing in clusters a lot easier.
>> >Also building a local fix and then running it locally will work
>> without
>> >additional modifications to the code.
>> >
>> >3. Support for a 'single application cluster'
>> >I've been playing around with the S3 plugin and what I have found is
>> >that this essentially requires all nodes to have full access to the
>> >credentials needed to connect to S3.
>> >This essentially means that a multi-tenant setup is not possible in
>> >these cases.
>> >So I think the single application cluster should be a feature
>> available
>> >in all cases.
>> >
>> >4. I would like a native-kubernetes-single-application base image.
>> >I can then create a derived image where I only add the jar of my
>> >application.
>> >My desire is that I can then create a k8s yaml file for kubectl
>> >that adds the needed configs/secrets/arguments/environment variables
>> and
>> >starts the cluster and application.
>> >Because the native kubernetes support makes it automatically scale
>> based
>> >on the application this should 'just work'.
>> >
>> > Additional note:
>> >
>> >1. Job/Task attempt logging instead of task manager logging.
>> >*I realize this has nothing to do with the docker images*
>> >    I found something "hard to work with" while running some tests last
>> > week.
>> >The logging is done to a single log for the task manager.
>> >So if I ha

Re: [VOTE] FLIP-102: Add More Metrics to TaskManager

[Andrey Zagrebin]

Hi guys,

Thanks for more details Zhijiang.
It also looks to me that mapped memory size is mostly driven by OS limits
and bit-ness of JVM (32/64).

Thinking more about the 'Metrics' tab layout, couple of more things have
come into my mind.

# 'Metrics' tab -> 'Memory': 'Metrics' and 'Configuration' tabs

It contains only memory specific things and the design suggests not only
metrics but configuration as well.
Moreover, there are other metrics on top which are not in the metrics tab.
Therefore, I would name it 'Memory' and then add sub-tabs: e.g. 'Metrics'
and 'Configuration' tab.
Alternatively, one could consider splitting 'Metrics' into 'Metrics' and
'Configuration' tabs.

# Metrics (a bit different structure)

I would put memory metrics into 4 groups:
- JVM Memory
- Managed
- Network
- Garbage collection

Alternatively, one could consider:
- Managed by JVM (same as JVM Memory)
- Managed by Flink (Managed Segments and Network buffers)
- Garbage collection

## Total memory (remove from metrics)

As mentioned in the discussions before, it is hard to measure the total
memory usage.
Therefore, I would put into the configuration tab, see below.

## JVM Memory

Here we can have Heap, Non-Heap, Direct and mapped because they are all
managed by JVM.
Heap and direct can stay as they are.

### Non-Heap (could stay for now)

I think it is ok to keep Non-Heap for now because we had it also before.
This metric does not correlate explicitly with FLIP-49 but it is exposed by
JVM.
Once, we find better things to show (related only to JVM, e.g. Metaspace
etc), we can reconsider this as a follow-up.

### Mapped (looks still valuable)

As I understand at the moment, this can have a value for users to monitor
spilling of batch partitions.

### Metaspace (new, sub-component of Non-Heap, follow-up)

We have never had anything for the Metaspace. The recent experience shows
that it can be useful.
I would put it on road map as a follow-up though, because it also needs
some research and preparation on server side [1].

# Configuration (see Flink user docs picture)

We already have a picture in the docs representing memory components in
Flink [2].
The layout in this picture can be also used in this FLIP to depict the
actual configuration.
This would be more clear for users to see the same as we have in docs.

The configuration can also depict size of the total process and total Flink
memory according to docs.

As mentioned above, I also suggest to put it into a separate tab.

Best,
Andrey

[1]
https://kb.novaordis.com/index.php/Memory_Monitoring_and_Management_Platform_MBeans#Metaspace
[2]
https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/memory/mem_detail.html#overview


On Wed, Apr 1, 2020 at 8:03 PM Zhijiang 
wrote:

> Thanks for the FLIP, Yadong. In general I think this work is valuable for
> users to better understand the Flink's memory usages in different
> dimensions.
>
> Sorry for not going through every detailed discussions below, and I try to
> do that later if possible. Firstly I try to answer some Andrey's concerns
> with mmap.
>
> > - I do not know how the mapped memory works. Is it meant for the new
> spilled partitions? If the mapped memory also pulls from the direct
> > memory limit then this is something we do not account in our network
> buffers as I understand. In this case, this metric may be useful for tuning
> to understand
> > how much the mapped memory uses from the direct memory limit to set e.g.
> framework off-heap limit correctly and avoid direct OOM.
> > It could be something to discuss with Zhijiang. e.g. is the direct
> memory used there to buffer fetched regions of partition files or what for?
>
> Yes, the mapped memory is used in bounded blocking partition for batch
> jobs now, but not the default mode.
>
>  AIK it is not related and limited to the setting of `MaxDirectMemory`, so
> we do not need to worry about the current direct memory setting and the
> potential OOM issue.
> It is up to the address space to determine the mapped file size, and in 64
> bit system we can regard the limitless size in theory.
>
> Regarding the size of mapped buffer pool from MXBean, it only indicates
> how much file size were already mapped before, even it is unchanged to not
> reflect the real
> physical memory use. E.g. when the file was mapped 100GB region at the
> beginning, the mapped buffer pool from MXBean would be 100GB. But how many
> physical
> memories are really consumed is up to the specific read or write
> operations in practice, and also controlled by the operator system. E.g
> some unused regions might be
> exchanged into SWAP virtual memory when physical memory is limited.
>
> From this point, I guess it is no meaningful to show the size of mapped
> buffer pool for users who may be more concerned with how many physical
> memories are really
> used.
>
> Best,
> Zhijiang
>
>
> --
> From:Andrey Zagrebin 
> Send Time:2020 Mar. 

Re: [DISCUSS] FLIP-111: Docker image unification

[Andrey Zagrebin]

Hi everyone,

Patrick and Ufuk, thanks a lot for more ideas and suggestions!

I have updated the FLIP according to the current state of discussion.
Now it also contains the implementation steps and future follow-ups.
Please, review if there are any concerns.
The order of the steps aims for keeping Flink releasable at any point if
something does not have enough time to get in.

It looks that we are reaching mostly a consensus for the open questions.
There is also a list of items, which have been discussed in this thread,
and short summary below.
As soon as there are no concerns, I will create a voting thread.

I also added some thoughts for further customising logging setup. This may
be an optional follow-up
which is additional to the default logging into files for Web UI.

# FLIP scope
The focus is users of the official releases.
Create docs for how to use the official docker image.
Remove other Dockerfiles in Flink repo.
Rely on running the official docker image in different modes (JM/TM).
Customise running the official image with env vars (This should minimise
manual manipulating of local files and creation of a custom image).

# Base oficial image

## Java versions
There is a separate effort for this:
https://github.com/apache/flink-docker/pull/9

# Run image

## Entry point modes
JM session, JM job, TM

## Entry point config
We use env vars for this, e.g. FLINK_PROPERTIES and ENABLE_BUILT_IN_PLUGINS

## Flink config options
We document the existing FLINK_PROPERTIES env var to override config
options in flink-conf.yaml.
Then later, we do not need to expose and handle any other special env vars
for config options (address, port etc).
The future plan is to make Flink process configurable by env vars, e.g.
'some.yaml.option: val' -> FLINK_SOME_YAML_OPTION=val

## Extra files: jars, custom logging properties
We can provide env vars to point to custom locations, e.g. in mounted
volumes.

# Extend image

## Python/hadoop versions, activating certain libs/plugins
Users can install extra dependencies and change configs in their custom
image which extends our base image.

# Logging

## Web UI
Modify the *log4j-console.properties* to also output logs into the files
for WebUI. Limit log file size.

## Container output
Separate effort for proper split of Flink process stdout and stderr into
files and container output
(idea with tee command: `program start-foreground &2>1 | tee
flink-user-taskexecutor.out`)

# Docker bash utils
We are not going to expose it to users as an API.
They should be able either to configure and run the standard entry point
or the documentation should give short examples about how to extend and
customise the base image.

During the implementation, we will see if it makes sense to factor out
certain bash procedures
to reuse them e.g. in custom dev versions of docker image.

# Dockerfile / image for developers
We keep it on our future roadmap. This effort should help to understand
what we can reuse there.

Best,
Andrey


On Fri, Apr 3, 2020 at 12:57 PM Till Rohrmann  wrote:

> Hi everyone,
>
> just a small inline comment.
>
> On Fri, Apr 3, 2020 at 11:42 AM Ufuk Celebi  wrote:
>
> > Hey Yang,
> >
> > thanks! See inline answers.
> >
> > On Fri, Apr 3, 2020 at 5:11 AM Yang Wang  wrote:
> >
> > > Hi Ufuk,
> > >
> > > Thanks for make the conclusion and directly point out what need to be
> > done
> > > in
> > > FLIP-111. I agree with you that we should narrow down the scope and
> focus
> > > the
> > > most important and basic part about docker image unification.
> > >
> > > (1) Extend the entrypoint script in apache/flink-docker to start the
> job
> > >> cluster entry point
> > >
> > > I want to add a small requirement for the entry point script.
> Currently,
> > > for the native
> > > K8s integration, we are using the apache/flink-docker image, but with
> > > different entry
> > > point("kubernetes-entry.sh"). Generate the java cmd in KubernetesUtils
> > and
> > > run it
> > > in the entry point. I really hope it could merge to apache/flink-docker
> > > "docker-entrypoint.sh".
> > >
> >
> > The script [1] only adds the FLINK_CLASSPATH env var which seems
> generally
> > reasonable to me. But since principled classpath and entrypoint
> > configuration is somewhat related to the follow-up improvement
> proposals, I
> > could also see this being done after FLIP-111.
> >
> >
> > > (2) Extend the example log4j-console configuration
> > >> => support log retrieval from the Flink UI out of the box
> > >
> > > If you mean to update the "flink-dist/conf/log4j-console.properties" to
> > > support console and
> > > local log files. I will say "+1". But we need to find a proper way to
> > make
> > > stdout/stderr output
> > > both available for console and log files. Maybe till's proposal could
> > help
> > > to solve this.
> > > "`program &2>1 | tee flink-user-taskexecutor.out`"
> > >
> >
> > I think we can simply add a rolling file appender with a limit on the log
> > size.
> >
> > I think this won't solve 

[jira] [Created] (FLINK-16946) Update user documentation for job manager memory model

[Andrey Zagrebin (Jira)]

Andrey Zagrebin created FLINK-16946:
---

 Summary: Update user documentation for job manager memory model
 Key: FLINK-16946
 URL: https://issues.apache.org/jira/browse/FLINK-16946
 Project: Flink
  Issue Type: Sub-task
  Components: Documentation, Runtime / Configuration, Runtime / 
Coordination
Reporter: Andrey Zagrebin
 Fix For: 1.11.0






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] FLIP 116: Unified Memory Configuration for Job Managers

[Andrey Zagrebin]

Thanks everybody for the voting.
I also vote
+1 (binding)

Hereby the vote is closed and the FLIP-116 is accepted

3 binding votes:
@Till Rohrmann 
@g...@apache.org 
@azagre...@apache.org  (me)

2 non-binding votes:
@Xintong Song 
@Yang Wang 

no vetos/-1s

Best,
Andrey

On Wed, Mar 25, 2020 at 6:16 PM Gary Yao  wrote:

> +1 (binding)
>
> Best,
> Gary
>
> On Wed, Mar 18, 2020 at 3:16 PM Andrey Zagrebin 
> wrote:
>
> > Hi All,
> >
> > The discussion for FLIP-116 looks to be resolved [1].
> > Therefore, I start the vote for it.
> > The vote will end at 6pm CET on Monday, 23 March.
> >
> > Best,
> > Andrey
> >
> > [1]
> >
> >
> http://mail-archives.apache.org/mod_mbox/flink-dev/202003.mbox/%3CCAJNyZN7AJAU_RUVhnWa7r%2B%2BtXpmUqWFH%2BG0hfoLVBzgRMmAO2w%40mail.gmail.com%3E
> >
>

[jira] [Created] (FLINK-16754) Consider refactoring of ProcessMemoryUtilsTestBase to avoid inheritance

[Andrey Zagrebin (Jira)]

Andrey Zagrebin created FLINK-16754:
---

 Summary: Consider refactoring of ProcessMemoryUtilsTestBase to 
avoid inheritance
 Key: FLINK-16754
 URL: https://issues.apache.org/jira/browse/FLINK-16754
 Project: Flink
  Issue Type: Sub-task
  Components: Runtime / Configuration, Runtime / Coordination
Reporter: Andrey Zagrebin
Assignee: Andrey Zagrebin
 Fix For: 1.11.0


After FLINK-16615 we have class structure of memory utils with isolation of 
responsibilities, mostly through composition. We should consider to refactor 
the tests as well to get more abstraction targeted tests with better isolation 
and w/o implicit test inheritance contracts.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-16742) Extend and use BashJavaUtils to start JM JVM process and pass JVM memory args

[Andrey Zagrebin (Jira)]

Andrey Zagrebin created FLINK-16742:
---

 Summary: Extend and use BashJavaUtils to start JM JVM process and 
pass JVM memory args
 Key: FLINK-16742
 URL: https://issues.apache.org/jira/browse/FLINK-16742
 Project: Flink
  Issue Type: Sub-task
  Components: Deployment / Scripts, Runtime / Configuration, Runtime / 
Coordination
Reporter: Andrey Zagrebin


Currently, the legacy options `_jobmanager.heap.size_` (or 
`_jobmanager.heap.mb_`) is used in JM standalone bash scripts to pass JVM heap 
size arg and start JM JVM process.

BashJavaUtils should be extended to get JVM memory arg list string from Flink 
configuration. BashJavaUtils can use 
JobManagerProcessUtils#processSpecFromConfig to obtain JobManagerProcessSpec. 
JobManagerProcessSpec can be passed to 
ProcessMemoryUtils#generateJvmParametersStr to get JVM memory arg list string.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-16746) Deprecate/remove legacy memory options for JM and expose the new ones

[Andrey Zagrebin (Jira)]

Andrey Zagrebin created FLINK-16746:
---

 Summary: Deprecate/remove legacy memory options for JM and expose 
the new ones
 Key: FLINK-16746
 URL: https://issues.apache.org/jira/browse/FLINK-16746
 Project: Flink
  Issue Type: Sub-task
  Components: Runtime / Configuration, Runtime / Coordination
Reporter: Andrey Zagrebin
 Fix For: 1.11.0


Deprecate legacy heap options: `_jobmanager.heap.size_` (and update 
`_jobmanager.heap.mb_`)

Remove container cut-off options: `_containerized.heap-cutoff-ratio_` and 
`_containerized.heap-cutoff-min_`



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-16745) Use JobManagerProcessUtils to start JM container and pass JVM memory args

[Andrey Zagrebin (Jira)]

Andrey Zagrebin created FLINK-16745:
---

 Summary: Use JobManagerProcessUtils to start JM container and pass 
JVM memory args
 Key: FLINK-16745
 URL: https://issues.apache.org/jira/browse/FLINK-16745
 Project: Flink
  Issue Type: Sub-task
  Components: Deployment / Kubernetes, Deployment / Mesos, Deployment / 
YARN, Runtime / Configuration, Runtime / Coordination
Reporter: Andrey Zagrebin
 Fix For: 1.11.0


JobManagerProcessUtils#processSpecFromConfig should be used to get 
JobManagerProcessSpec. Then JobManagerProcessSpec can be passed to 
ProcessMemoryUtils#generateJvmParametersStr to get JVM memory arg list string.

The configuration should be fixed to fallback to 
JobManagerOptions.TOTAL_PROCESS_MEMORY if a legacy option is set 
(JobManagerProcessUtils#getConfigurationWithLegacyHeapSizeMappedToNewConfigOption)
 before passing it to JobManagerProcessUtils#processSpecFromConfig.

Then the JVM memory arg list can be used to start the JM container in 
Yarn/Mesos/Kubernetes active RMs instead of using the existing legacy heap 
options: `_jobmanager.heap.size_` (or `_jobmanager.heap.mb_`).



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] FLIP-111: Docker image unification

[Andrey Zagrebin]

 Otherwise it
> will only add more maintenance burden.
>
> Long story short, with the existing configuration options (envsubts,
> FLINK_PROPERTIES) we can already configure the Flink process and Flink
> itself. Since maintaining backwards compatibility is important, we could
> rely on these mechanisms until we have proper env variable configuration
> and don't have to introduce a new way to change the configuration.
>
> # Logging & Stdout/err
>
> ## Logging
>
> I think Konstantin is right and we should provide a log4j.properties file
> which, per default, specifies the file and console appender. We could add a
> special log4j.properties file to apache/flink-docker which we include in
> the Dockerfile.
>
> This approach will give users the most flexibility w/o relying on magic
> (e.g. tailing the log files after starting the process in the background).
>
> ## Stdout/err
>
> I think for printing the stdout/err output to STDOUT/ERR and to capture it
> in a file there are solutions. For example, one could use `program &2>1 |
> tee flink-user-taskexecutor.out` to achieve this.
>
> # Java version
>
> I agree that it would be nice to also offer a Java 11 Dockerfile. For the
> sake of limiting the scope of this proposal I would suggest to do this as a
> follow up issue.
>
> # Dev version
>
> Tooling to create a Docker image from the current Flink repository is
> indeed very nice for development. As Andrey suggested, I think this would
> be a good follow up for this proposal. I don't think that Andrey's current
> proposal would block any future developments in this direction.
>
> # Scripts
>
> At the moment, I would be in favour of placing the Dockerfile scripts
> under apache/flink-docker since they belong more to the Dockerfile than to
> Flink's binary distribution. If we see that we might be able to reuse them
> for the developer Dockerfile, then we can still move them to the Flink
> repository.
>
> I would refrain from offering special commands to set individual
> configuration options (e.g., flink_docker_utils set_web_ui_port 8081). It
> should be fine enough to do it via flink_docker-utils conifgure rest.port
> 8081 if we cannot solve it via the general configuration mechanism.
>
> Cheers,
> Till
>
> On Wed, Mar 18, 2020 at 6:38 AM Yangze Guo  wrote:
>
>> I second Thomas that we can support both Java 8 and 11.
>>
>> Best,
>> Yangze Guo
>>
>> On Wed, Mar 18, 2020 at 12:12 PM Thomas Weise  wrote:
>> >
>> > -->
>> >
>> > On Mon, Mar 16, 2020 at 1:58 AM Andrey Zagrebin 
>> wrote:
>> >>
>> >> Thanks for the further feedback Thomas and Yangze.
>> >>
>> >> > A generic, dynamic configuration mechanism based on environment
>> variables
>> >> is essential and it is already supported via envsubst and an
>> environment
>> >> variable that can supply a configuration fragment
>> >>
>> >> True, we already have this. As I understand this was introduced for
>> >> flexibility to template a custom flink-conf.yaml with env vars, put it
>> into
>> >> the FLINK_PROPERTIES and merge it with the default one.
>> >> Could we achieve the same with the dynamic properties
>> (-Drpc.port=1234),
>> >> passed as image args to run it, instead of FLINK_PROPERTIES?
>> >> They could be also parametrised with env vars. This would require
>> >> jobmanager.sh to properly propagate them to
>> >> the StandaloneSessionClusterEntrypoint though:
>> >>
>> https://github.com/docker-flink/docker-flink/pull/82#issuecomment-525285552
>> >> cc @Till
>> >> This would provide a unified configuration approach.
>> >>
>> >
>> > How would that look like for the various use cases? The k8s operator
>> would need to generate the -Dabc .. -Dxyz entry point command instead of
>> setting the FLINK_PROPERTIES environment variable? Potentially that
>> introduces additional complexity for little gain. Do most deployment
>> platforms that support Docker containers handle the command line route
>> well? Backward compatibility may also be a concern.
>> >
>> >>
>> >> > On the flip side, attempting to support a fixed subset of
>> configuration
>> >> options is brittle and will probably lead to compatibility issues down
>> the
>> >> road
>> >>
>> >> I agree with it. The idea was to have just some shortcut scripted
>> functions
>> >> to set options in flink-conf.yaml for a custom Dockerfile or entry
>> poi

[jira] [Created] (FLINK-16686) [State TTL] Make user class loader available in native RocksDB compaction thread

[Andrey Zagrebin (Jira)]

Andrey Zagrebin created FLINK-16686:
---

 Summary: [State TTL] Make user class loader available in native 
RocksDB compaction thread
 Key: FLINK-16686
 URL: https://issues.apache.org/jira/browse/FLINK-16686
 Project: Flink
  Issue Type: Bug
  Components: Runtime / State Backends
Affects Versions: 1.8.0
Reporter: Andrey Zagrebin


The issue is initially reported 
[here|https://stackoverflow.com/questions/60745711/flink-kryo-serializer-because-chill-serializer-couldnt-be-found].

The problem is that the java code of Flink compaction filter is called from 
RocksDB native C++ code. It is called in the context of the native compaction 
thread. RocksDB has utilities to create java Thread context for the Flink java 
callback. Presumably, the Java thread context class loader is not set at all 
and if it is queried then it produces NullPointerException.

The provided report enabled a list state with TTL. The compaction filter has to 
deserialise elements to check expiration. The deserialiser relies on Kryo which 
queries the thread context class loader which is expected to be the user class 
loader of the task but turns out to be null.

We should investigate how to pass the user class loader to the compaction 
thread of the list state with TTL.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] FLIP 116: Unified Memory Configuration for Job Managers

[Andrey Zagrebin]

Alright, thanks for the feedback. I also agree with it. Then this is resolved.

> On 19 Mar 2020, at 14:14, Till Rohrmann  wrote:
> 
> I agree with Xintong's proposal. If we see that many users run into this
> problem, then one could think about escalating the warning message into a
> failure.
> 
> Cheers,
> Till
> 
> On Thu, Mar 19, 2020 at 4:23 AM Xintong Song  wrote:
> 
>> I think recommend a minimum value in docs and throw a warning if the heap
>> size is too small should be good enough.
>> Not sure about failing job if the min heap is not fulfilled. As already
>> mentioned, it would be hard to determine the min heap size. And if we make
>> the min heap configurable, then in any case that users need to configure
>> the min heap, they can configure the heap size directly.
>> 
>> Thank you~
>> 
>> Xintong Song
>> 
>> 
>> 
>> On Wed, Mar 18, 2020 at 10:55 PM Andrey Zagrebin 
>> wrote:
>> 
>>> Hi all,
>>> 
>>> One thing more thing to mention, the current calculations can lead to
>>> arbitrary small JVM Heap, maybe even zero.
>>> I suggest to introduce a check where we at least recommend to set the JVM
>>> heap to e.g. 128Mb.
>>> 
>>> Additionally, we can demand some minimum value to function and fail if it
>>> is not fulfilled.
>>> We could experiment with what is the working minimum but It is hard to
>> come
>>> up with this limit because it again can depend on the job and
>> environment.
>>> 
>>> Best,
>>> Andrey
>>> 
>>> On Wed, Mar 18, 2020 at 5:03 PM Andrey Zagrebin 
>>> wrote:
>>> 
>>>> Hi all,
>>>> 
>>>> Thanks for the feedback, Xintong and Till.
>>>> 
>>>>> rename jobmanager.memory.direct.size into
>>> jobmanager.memory.off-heap.size
>>>> 
>>>> I am ok with that to align it with TM and avoid further complications
>> for
>>>> users.
>>>> I will adjust the FLIP.
>>>> 
>>>>> change the default value of JM Metaspace size to 256 MB
>>>> 
>>>> Indeed, no reason to assume that the user code would need less
>> Metaspace
>>>> in JM.
>>>> I will change it unless a better argument is reported for another
>> value.
>>>> 
>>>> I think all concerns has been resolved so I am starting the voting in a
>>>> separate thread.
>>>> 
>>>> Best,
>>>> Andrey
>>>> 
>>>> On Tue, Mar 17, 2020 at 6:16 PM Till Rohrmann 
>>>> wrote:
>>>> 
>>>>> Thanks for creating this FLIP Andrey.
>>>>> 
>>>>> I agree with Xintong that we should rename
>> jobmanager.memory.direct.size
>>>>> into jobmanager.memory.off-heap.size which accounts for native and
>>> direct
>>>>> memory usage. I think it should be good enough and is easier to
>>> understand
>>>>> for the user.
>>>>> 
>>>>> Concerning the default value for the metaspace size. Did we take the
>>>>> lessons learned from the TM metaspace size into account? IIRC we are
>>> about
>>>>> to change the default value to 256 MB.
>>>>> 
>>>>> Feel free to start a vote once these last two questions have been
>>>>> resolved.
>>>>> 
>>>>> Cheers,
>>>>> Till
>>>>> 
>>>>> On Thu, Mar 12, 2020 at 4:25 AM Xintong Song 
>>>>> wrote:
>>>>> 
>>>>>> Thanks Andrey for kicking this discussion off.
>>>>>> 
>>>>>> Regarding "direct" vs. "off-heap", I'm personally in favor of
>> renaming
>>>>> the
>>>>>> "direct" memory in the current FLIP-116[1] to "off-heap" memory, and
>>>>> making
>>>>>> it also account for user native memory usage.
>>>>>> 
>>>>>> On one hand, I think it would be good that JM & TM provide
>> consistent
>>>>>> concepts and terminologies to users. IIUC, this is exactly the
>> purpose
>>>>> of
>>>>>> this FLIP. For TMs, we already have "off-heap" memory accounting for
>>>>> both
>>>>>> direct and native memory usages, and we did this so that users do
>> not
>>>>> need
>>>>>> to understand 

Re: [DISCUSS] FLIP 116: Unified Memory Configuration for Job Managers

[Andrey Zagrebin]

Hi all,

One thing more thing to mention, the current calculations can lead to
arbitrary small JVM Heap, maybe even zero.
I suggest to introduce a check where we at least recommend to set the JVM
heap to e.g. 128Mb.

Additionally, we can demand some minimum value to function and fail if it
is not fulfilled.
We could experiment with what is the working minimum but It is hard to come
up with this limit because it again can depend on the job and environment.

Best,
Andrey

On Wed, Mar 18, 2020 at 5:03 PM Andrey Zagrebin 
wrote:

> Hi all,
>
> Thanks for the feedback, Xintong and Till.
>
> > rename jobmanager.memory.direct.size into jobmanager.memory.off-heap.size
>
> I am ok with that to align it with TM and avoid further complications for
> users.
> I will adjust the FLIP.
>
> > change the default value of JM Metaspace size to 256 MB
>
> Indeed, no reason to assume that the user code would need less Metaspace
> in JM.
> I will change it unless a better argument is reported for another value.
>
> I think all concerns has been resolved so I am starting the voting in a
> separate thread.
>
> Best,
> Andrey
>
> On Tue, Mar 17, 2020 at 6:16 PM Till Rohrmann 
> wrote:
>
>> Thanks for creating this FLIP Andrey.
>>
>> I agree with Xintong that we should rename jobmanager.memory.direct.size
>> into jobmanager.memory.off-heap.size which accounts for native and direct
>> memory usage. I think it should be good enough and is easier to understand
>> for the user.
>>
>> Concerning the default value for the metaspace size. Did we take the
>> lessons learned from the TM metaspace size into account? IIRC we are about
>> to change the default value to 256 MB.
>>
>> Feel free to start a vote once these last two questions have been
>> resolved.
>>
>> Cheers,
>> Till
>>
>> On Thu, Mar 12, 2020 at 4:25 AM Xintong Song 
>> wrote:
>>
>> > Thanks Andrey for kicking this discussion off.
>> >
>> > Regarding "direct" vs. "off-heap", I'm personally in favor of renaming
>> the
>> > "direct" memory in the current FLIP-116[1] to "off-heap" memory, and
>> making
>> > it also account for user native memory usage.
>> >
>> > On one hand, I think it would be good that JM & TM provide consistent
>> > concepts and terminologies to users. IIUC, this is exactly the purpose
>> of
>> > this FLIP. For TMs, we already have "off-heap" memory accounting for
>> both
>> > direct and native memory usages, and we did this so that users do not
>> need
>> > to understand the differences between the two kinds.
>> >
>> > On the other hand, while for TMs it is hard to tell which kind of
>> memory is
>> > needed mostly due to variety of applications, I believe for JM the major
>> > memory consumption is heap memory in most cases. That means we probably
>> can
>> > rely on the heap activities to trigger GC in most cases, and the max
>> direct
>> > memory limit can act as a safe net. Moreover, I think the cases should
>> be
>> > very rare that we need native memory for user codes. Therefore, we
>> probably
>> > should not break the JM/TM consistency for potential risks in such rare
>> > cases.
>> >
>> > WDYT?
>> >
>> > Thank you~
>> >
>> > Xintong Song
>> >
>> >
>> > [1]
>> >
>> >
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP+116%3A+Unified+Memory+Configuration+for+Job+Managers
>> >
>> > On Wed, Mar 11, 2020 at 8:56 PM Andrey Zagrebin 
>> > wrote:
>> >
>> > > Hi All,
>> > >
>> > > As you may have noticed, 1.10 release included an extensive
>> improvements
>> > to
>> > > memory management and configuration of Task Managers, FLIP-49: [1].
>> The
>> > > memory configuration of Job Managers has not been touched in 1.10.
>> > >
>> > > Although, Job Manager's memory model does not look so sophisticated as
>> > > for Task Managers, It makes to align Job Manager memory model and
>> > settings
>> > > with Task Managers. Therefore, we propose to reconsider it as well in
>> > 1.11
>> > > and I prepared a FLIP 116 [2] for that.
>> > >
>> > > Any feedback is appreciated.
>> > >
>> > > So far, there is one discussion point about how to address native
>> > > non-direct memory usage of user code. The user code can 

[VOTE] FLIP 116: Unified Memory Configuration for Job Managers

[Andrey Zagrebin]

Hi All,

The discussion for FLIP-116 looks to be resolved [1].
Therefore, I start the vote for it.
The vote will end at 6pm CET on Monday, 23 March.

Best,
Andrey

[1]
http://mail-archives.apache.org/mod_mbox/flink-dev/202003.mbox/%3CCAJNyZN7AJAU_RUVhnWa7r%2B%2BtXpmUqWFH%2BG0hfoLVBzgRMmAO2w%40mail.gmail.com%3E

Re: [DISCUSS] FLIP 116: Unified Memory Configuration for Job Managers

[Andrey Zagrebin]

Hi all,

Thanks for the feedback, Xintong and Till.

> rename jobmanager.memory.direct.size into jobmanager.memory.off-heap.size

I am ok with that to align it with TM and avoid further complications for
users.
I will adjust the FLIP.

> change the default value of JM Metaspace size to 256 MB

Indeed, no reason to assume that the user code would need less Metaspace in
JM.
I will change it unless a better argument is reported for another value.

I think all concerns has been resolved so I am starting the voting in a
separate thread.

Best,
Andrey

On Tue, Mar 17, 2020 at 6:16 PM Till Rohrmann  wrote:

> Thanks for creating this FLIP Andrey.
>
> I agree with Xintong that we should rename jobmanager.memory.direct.size
> into jobmanager.memory.off-heap.size which accounts for native and direct
> memory usage. I think it should be good enough and is easier to understand
> for the user.
>
> Concerning the default value for the metaspace size. Did we take the
> lessons learned from the TM metaspace size into account? IIRC we are about
> to change the default value to 256 MB.
>
> Feel free to start a vote once these last two questions have been resolved.
>
> Cheers,
> Till
>
> On Thu, Mar 12, 2020 at 4:25 AM Xintong Song 
> wrote:
>
> > Thanks Andrey for kicking this discussion off.
> >
> > Regarding "direct" vs. "off-heap", I'm personally in favor of renaming
> the
> > "direct" memory in the current FLIP-116[1] to "off-heap" memory, and
> making
> > it also account for user native memory usage.
> >
> > On one hand, I think it would be good that JM & TM provide consistent
> > concepts and terminologies to users. IIUC, this is exactly the purpose of
> > this FLIP. For TMs, we already have "off-heap" memory accounting for both
> > direct and native memory usages, and we did this so that users do not
> need
> > to understand the differences between the two kinds.
> >
> > On the other hand, while for TMs it is hard to tell which kind of memory
> is
> > needed mostly due to variety of applications, I believe for JM the major
> > memory consumption is heap memory in most cases. That means we probably
> can
> > rely on the heap activities to trigger GC in most cases, and the max
> direct
> > memory limit can act as a safe net. Moreover, I think the cases should be
> > very rare that we need native memory for user codes. Therefore, we
> probably
> > should not break the JM/TM consistency for potential risks in such rare
> > cases.
> >
> > WDYT?
> >
> > Thank you~
> >
> > Xintong Song
> >
> >
> > [1]
> >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP+116%3A+Unified+Memory+Configuration+for+Job+Managers
> >
> > On Wed, Mar 11, 2020 at 8:56 PM Andrey Zagrebin 
> > wrote:
> >
> > > Hi All,
> > >
> > > As you may have noticed, 1.10 release included an extensive
> improvements
> > to
> > > memory management and configuration of Task Managers, FLIP-49: [1]. The
> > > memory configuration of Job Managers has not been touched in 1.10.
> > >
> > > Although, Job Manager's memory model does not look so sophisticated as
> > > for Task Managers, It makes to align Job Manager memory model and
> > settings
> > > with Task Managers. Therefore, we propose to reconsider it as well in
> > 1.11
> > > and I prepared a FLIP 116 [2] for that.
> > >
> > > Any feedback is appreciated.
> > >
> > > So far, there is one discussion point about how to address native
> > > non-direct memory usage of user code. The user code can be run e.g. in
> > > certain job submission scenarios within the JM process. For simplicity,
> > > FLIP suggests only an option for direct memory which is translated into
> > the
> > > setting of the JVM direct memory limit.
> > > Although, we documented for TM that the similar parameters can also
> > > address native non-direct memory usage [3], this can lead to wrong
> > > functioning of the JVM direct memory limit. The direct memory option in
> > JM
> > > could be also named in more general way, e.g. off-heap memory but this
> > > naming would somewhat hide its nature of JVM direct memory limit.
> > > On the other hand, JVM Overhead does not suffer from this problem and
> > > affects only the container/worker memory size which is the most
> important
> > > matter to address for the native non-direct memory consumption. The
> > caveat
> > > here is that JVM Overhead was not supposed to be used by any Flink or
> > user
> > > components.
> > >
> > > Thanks,
> > > Andrey
> > >
> > > [1]
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-49%3A+Unified+Memory+Configuration+for+TaskExecutors
> > > [2]
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP+116%3A+Unified+Memory+Configuration+for+Job+Managers
> > > [3]
> > >
> > >
> >
> https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/memory/mem_detail.html#overview
> > >
> >
>

Re: Question about RocksDBStateBackend Compaction Filter state cleanup

[Andrey Zagrebin]

Hi Lake,

When the Flink doc mentions a state entry in RocksDB, we mean one key/value
pair stored by user code over any keyed state API
(keyed context in keyed operators obtained e.g. from keyBy()
transformation).
In case of map or list, the doc means map key/value and list element.

- value/aggregating/folding/reducing state: key -> value
- map state: key -> map key -> value
- list state: key -> list -> element in some position

Best,
Andrey

On Tue, Mar 17, 2020 at 11:04 AM Yun Tang  wrote:

> Hi Lake
>
> Flink leverage RocksDB's background compaction mechanism to filter
> out-of-TTL entries (by comparing with current timestamp provided from
> RocksDB's time_provider) to not let them stay in newly compacted data.
>
> This would iterator over data entries with FlinkCompactionFilter::FilterV2
> [1], and the parameter 'queryTimeAfterNumEntries' in Flink indicates the
> threshold 'query_time_after_num_entries_' in FrocksDB  [2]. Once RocksDB
> iterator more than several entries .e.g 1000, it would call time_provider
> to update current timestamp to let the process of cleaning up more eagerly
> and accurately.
>
> [1]
> https://github.com/dataArtisans/frocksdb/blob/49bc897d5d768026f1eb816d960c1f2383396ef4/utilities/flink/flink_compaction_filter.cc#L107
> [2]
> https://github.com/dataArtisans/frocksdb/blob/49bc897d5d768026f1eb816d960c1f2383396ef4/utilities/flink/flink_compaction_filter.h#L140
>
> Best
> Yun Tang
>
> --
> *From:* LakeShen 
> *Sent:* Tuesday, March 17, 2020 15:30
> *To:* dev ; user-zh ;
> user 
> *Subject:* Question about RocksDBStateBackend Compaction Filter state
> cleanup
>
> Hi community ,
>
> I see the flink RocksDBStateBackend state cleanup,now the code like this :
>
> StateTtlConfig ttlConfig = StateTtlConfig
> .newBuilder(Time.seconds(1))
> .cleanupInRocksdbCompactFilter(1000)
> .build();
>
>
>
> The default background cleanup for RocksDB backend queries the current
> timestamp each time 1000 entries have been processed.
>
>
> What's the meaning of  1000 entries? 1000 different key ?
>
> Thanks to your reply.
>
> Best regards,
> LakeShen
>

[jira] [Created] (FLINK-16615) Introduce data structures and utilities to calculate Job Manager memory components

[Andrey Zagrebin (Jira)]

Andrey Zagrebin created FLINK-16615:
---

 Summary: Introduce data structures and utilities to calculate Job 
Manager memory components
 Key: FLINK-16615
 URL: https://issues.apache.org/jira/browse/FLINK-16615
 Project: Flink
  Issue Type: Sub-task
  Components: Runtime / Configuration, Runtime / Coordination
Reporter: Andrey Zagrebin
Assignee: Andrey Zagrebin
 Fix For: 1.11.0






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-16614) FLIP-116 Unified Memory Configuration for Job Manager

[Andrey Zagrebin (Jira)]

Andrey Zagrebin created FLINK-16614:
---

 Summary: FLIP-116 Unified Memory Configuration for Job Manager
 Key: FLINK-16614
 URL: https://issues.apache.org/jira/browse/FLINK-16614
 Project: Flink
  Issue Type: Improvement
  Components: Runtime / Configuration, Runtime / Coordination
Reporter: Andrey Zagrebin
Assignee: Andrey Zagrebin
 Fix For: 1.11.0


This is the umbrella issue of [FLIP-116: Unified Memory Configuration for Job 
Managers|https://cwiki.apache.org/confluence/display/FLINK/FLIP+116%3A+Unified+Memory+Configuration+for+Job+Managers].



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] FLIP-111: Docker image unification

[Andrey Zagrebin]

ook at it.
>
> Regarding supporting JAVA 11:
> - Not sure if it is necessary to ship JAVA. Maybe we could just change
> the base image from openjdk:8-jre to openjdk:11-jre in template docker
> file[1]. Correct me if I understand incorrectly. Also, I agree to move
> this out of the scope of this FLIP if it indeed takes much extra
> effort.
>
> Regarding the custom configuration, the mechanism that Thomas mentioned
> LGTM.
>
> [1]
> https://github.com/apache/flink-docker/blob/master/Dockerfile-debian.template
>
> Best,
> Yangze Guo
>
> On Wed, Mar 11, 2020 at 5:52 AM Thomas Weise  wrote:
> >
> > Thanks for working on improvements to the Flink Docker container images.
> This will be important as more and more users are looking to adopt
> Kubernetes and other deployment tooling that relies on Docker images.
> >
> > A generic, dynamic configuration mechanism based on environment
> variables is essential and it is already supported via envsubst and an
> environment variable that can supply a configuration fragment:
> >
> >
> https://github.com/apache/flink-docker/blob/09adf2dcd99abfb6180e1e2b5b917b288e0c01f6/docker-entrypoint.sh#L88
> >
> https://github.com/apache/flink-docker/blob/09adf2dcd99abfb6180e1e2b5b917b288e0c01f6/docker-entrypoint.sh#L85
> >
> > This gives the necessary control for infrastructure use cases that aim
> to supply deployment tooling other users. An example in this category this
> is the FlinkK8sOperator:
> >
> > https://github.com/lyft/flinkk8soperator/tree/master/examples/wordcount
> >
> > On the flip side, attempting to support a fixed subset of configuration
> options is brittle and will probably lead to compatibility issues down the
> road:
> >
> >
> https://github.com/apache/flink-docker/blob/09adf2dcd99abfb6180e1e2b5b917b288e0c01f6/docker-entrypoint.sh#L97
> >
> > Besides the configuration, it may be worthwhile to see in which other
> ways the base Docker images can provide more flexibility to incentivize
> wider adoption.
> >
> > I would second that it is desirable to support Java 11 and in general
> use a base image that allows the (straightforward) use of more recent
> versions of other software (Python etc.)
> >
> >
> https://github.com/apache/flink-docker/blob/d3416e720377e9b4c07a2d0f4591965264ac74c5/Dockerfile-debian.template#L19
> >
> > Thanks,
> > Thomas
> >
> > On Tue, Mar 10, 2020 at 12:26 PM Andrey Zagrebin 
> wrote:
> >>
> >> Hi All,
> >>
> >> Thanks a lot for the feedback!
> >>
> >> *@Yangze Guo*
> >>
> >> - Regarding the flink_docker_utils#install_flink function, I think it
> >> > should also support build from local dist and build from a
> >> > user-defined archive.
> >>
> >> I suppose you bring this up mostly for development purpose or powerful
> >> users.
> >> Most of normal users are usually interested in mainstream released
> versions
> >> of Flink.
> >> Although, you are bring a valid concern, my idea was to keep scope of
> this
> >> FLIP mostly for those normal users.
> >> The powerful users are usually capable to design a completely
> >> custom Dockerfile themselves.
> >> At the moment, we already have custom Dockerfiles e.g. for tests in
> >>
> flink-end-to-end-tests/test-scripts/docker-hadoop-secure-cluster/Dockerfile.
> >> We can add something similar for development purposes and maybe
> introduce a
> >> special maven goal. There is a maven docker plugin, afaik.
> >> I will add this to FLIP as next step.
> >>
> >> - It seems that the install_shaded_hadoop could be an option of
> >> > install_flink
> >>
> >> I woud rather think about this as a separate independent optional step.
> >>
> >> - Should we support JAVA 11? Currently, most of the docker file based on
> >> > JAVA 8.
> >>
> >> Indeed, it is a valid concern. Java version is a fundamental property of
> >> the docker image.
> >> To customise this in the current mainstream image is difficult, this
> would
> >> require to ship it w/o Java at all.
> >> Or this is a separate discussion whether we want to distribute docker
> hub
> >> images with different Java versions or just bump it to Java 11.
> >> This should be easy in a custom Dockerfile for development purposes
> though
> >> as mentioned before.
> >>
> >> - I do not understand how to set config options through
> >>
> >> "flink_docker_utils configure"? Does th

Re: [DISCUSS] Releasing Flink 1.10.1

[Andrey Zagrebin]

ely to cause problem.
> > > >
> > > > So basically only people have small 'process.size' in custom config
> > file
> > > > are really affected. I'm not sure what is the proportion of such use
> > > cases
> > > > though. (From questions asked on the user ML, probably not much).
> > > >
> > > > Thank you~
> > > >
> > > > Xintong Song
> > > >
> > > >
> > > >
> > > > On Thu, Mar 12, 2020 at 10:09 PM Stephan Ewen 
> > wrote:
> > > >
> > > > > No need to revert it now - I am not saying it should not go into
> > > 1.10.1,
> > > > I
> > > > > am just saying this is not clear to me yet.
> > > > >
> > > > > We have to trade off the fact that we may break some deployments
> with
> > > the
> > > > > fact that we will "safe" a lot of new deployments.
> > > > > I simply lack the data points / insight at the moment to understand
> > how
> > > > > significant both cases are, meaning how many users would be
> affected
> > > and
> > > > > how badly.
> > > > >
> > > > > Independent of that, improving the error message is always helpful.
> > > > >
> > > > > On Thu, Mar 12, 2020 at 1:22 PM Andrey Zagrebin <
> > > > > azagrebin.apa...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > >   - For 1.10.1 I am not completely sure, because users expect
> to
> > > > > upgrade
> > > > > > > that without config adjustments. That might not be possible
> with
> > > that
> > > > > > > change.
> > > > > >
> > > > > > Ok, makes sense, I will revert it for 1.10 and only try to
> improve
> > > > error
> > > > > > message and docs.
> > > > > >
> > > > > > > On 12 Mar 2020, at 13:15, Stephan Ewen 
> wrote:
> > > > > > >
> > > > > > > @Andrey about the increase in metaspace size
> > > > > > >   - I have no concerns for 1.11.0.
> > > > > > >   - For 1.10.1 I am not completely sure, because users expect
> to
> > > > > upgrade
> > > > > > > that without config adjustments. That might not be possible
> with
> > > that
> > > > > > > change.
> > > > > > >
> > > > > > > On Thu, Mar 12, 2020 at 12:55 PM Andrey Zagrebin <
> > > > > > azagrebin.apa...@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > >>
> > > > > > >>> About "FLINK-16142 Memory Leak causes Metaspace OOM error on
> > > > repeated
> > > > > > >> job”
> > > > > > >>
> > > > > > >> My understanding that the issue is basically covered by:
> > > > > > >>
> > > > > > >> - [FLINK-16225] Metaspace Out Of Memory should be handled as
> > Fatal
> > > > > Error
> > > > > > >> in TaskManager
> > > > > > >>   no full consensus there but improving error message for
> > existing
> > > > > task
> > > > > > >> thread fatal handling could be done at least
> > > > > > >>
> > > > > > >> - [FLINK-16406] Increase default value for JVM Metaspace to
> > > minimise
> > > > > its
> > > > > > >> OutOfMemoryError
> > > > > > >>   see further
> > > > > > >>
> > > > > > >> - [FLINK-16246] Exclude "SdkMBeanRegistrySupport" from
> > dynamically
> > > > > > loaded
> > > > > > >> AWS connectors
> > > > > > >>  not sure whether this is a blocker but looks close to be
> > resolved
> > > > > > >>
> > > > > > >>> About "FLINK-16406 Increase default value for JVM Metaspace"
> > > > > > >>> - Have we consensus that this is okay for a bugfix release?
> It
> > > > > changes
> > > > > > >>> setups, takes away memory from heap / managed memory on
> > existing
> > > > > setups
> > > >

Re: [DISCUSS] Releasing Flink 1.10.1

[Andrey Zagrebin]

>   - For 1.10.1 I am not completely sure, because users expect to upgrade
> that without config adjustments. That might not be possible with that
> change.

Ok, makes sense, I will revert it for 1.10 and only try to improve error 
message and docs.

> On 12 Mar 2020, at 13:15, Stephan Ewen  wrote:
> 
> @Andrey about the increase in metaspace size
>   - I have no concerns for 1.11.0.
>   - For 1.10.1 I am not completely sure, because users expect to upgrade
> that without config adjustments. That might not be possible with that
> change.
> 
> On Thu, Mar 12, 2020 at 12:55 PM Andrey Zagrebin 
> wrote:
> 
>> 
>>> About "FLINK-16142 Memory Leak causes Metaspace OOM error on repeated
>> job”
>> 
>> My understanding that the issue is basically covered by:
>> 
>> - [FLINK-16225] Metaspace Out Of Memory should be handled as Fatal Error
>> in TaskManager
>>   no full consensus there but improving error message for existing task
>> thread fatal handling could be done at least
>> 
>> - [FLINK-16406] Increase default value for JVM Metaspace to minimise its
>> OutOfMemoryError
>>   see further
>> 
>> - [FLINK-16246] Exclude "SdkMBeanRegistrySupport" from dynamically loaded
>> AWS connectors
>>  not sure whether this is a blocker but looks close to be resolved
>> 
>>> About "FLINK-16406 Increase default value for JVM Metaspace"
>>> - Have we consensus that this is okay for a bugfix release? It changes
>>> setups, takes away memory from heap / managed memory on existing setups
>>> that keep their flink-conf.yaml.
>> 
>> My understanding was that increasing to 256m resolved the reported problems
>> and we decided to make the change so I have merged it today as there were
>> no more concerns.
>> If there are concerns I can revert it.
>> 
>> On the other hand, I think improving the message error with reference to
>> the metaspace option should help the most
>> because user would not have to read all docs to fix it
>> then maybe this change is not even needed.
>> 
>> Best,
>> Andrey
>> 
>>> On 12 Mar 2020, at 12:28, Stephan Ewen  wrote:
>>> 
>>> Good idea to go ahead with 1.10.1
>>> 
>>> About "FLINK-16142 Memory Leak causes Metaspace OOM error on repeated
>> job"
>>> - I don't think we have consensus on the exact solution, yet, and some
>> of
>>> the changes might also have side effects that are hard to predict, so I
>> am
>>> not sure we should rush this in.
>>> 
>>> About "FLINK-16406 Increase default value for JVM Metaspace"
>>> - Have we consensus that this is okay for a bugfix release? It changes
>>> setups, takes away memory from heap / managed memory on existing setups
>>> that keep their flink-conf.yaml.
>>> 
>>> We may need to unblock the release form these two issues and think about
>>> having 1.10.2 in the near future.
>>> 
>>> On Thu, Mar 12, 2020 at 7:15 AM Yu Li  wrote:
>>> 
>>>> Thanks for the reminder Jark. Will keep an eye on these two.
>>>> 
>>>> Best Regards,
>>>> Yu
>>>> 
>>>> 
>>>> On Thu, 12 Mar 2020 at 12:32, Jark Wu  wrote:
>>>> 
>>>>> Thanks for driving this release, Yu!
>>>>> +1 to start 1.10.1 release cycle.
>>>>> 
>>>>> From the Table SQL module, I think we should also try to get in the
>>>>> following issues:
>>>>> - FLINK-16441: Allow users to override flink-conf parameters from SQL
>> CLI
>>>>> environment
>>>>> this allows users to set e.g. statebackend, watermark interval,
>>>>> exactly-once/at-least-once, in the SQL CLI
>>>>> - FLINK-16217: SQL Client crashed when any uncatched exception is
>> thrown
>>>>> this will improve much experience when using SQL CLI
>>>>> 
>>>>> Best,
>>>>> Jark
>>>>> 
>>>>> 
>>>>> On Wed, 11 Mar 2020 at 20:37, Yu Li  wrote:
>>>>> 
>>>>>> Thanks for the suggestion Andrey! I've added 1.10.1 into FLINK-16225
>>>> fix
>>>>>> versions and promoted its priority to Critical. Will also watch the
>>>>>> progress of FLINK-16108/FLINK-16408.
>>>>>> 
>>>>>> Best Regards,
>>>>>> Yu
>>>>>> 
>>>>>> 
>>>>>&g

Re: [DISCUSS] Releasing Flink 1.10.1

[Andrey Zagrebin]

> About "FLINK-16142 Memory Leak causes Metaspace OOM error on repeated job”

My understanding that the issue is basically covered by:

- [FLINK-16225] Metaspace Out Of Memory should be handled as Fatal Error in 
TaskManager
   no full consensus there but improving error message for existing task thread 
fatal handling could be done at least

- [FLINK-16406] Increase default value for JVM Metaspace to minimise its 
OutOfMemoryError
   see further

- [FLINK-16246] Exclude "SdkMBeanRegistrySupport" from dynamically loaded AWS 
connectors
  not sure whether this is a blocker but looks close to be resolved 

> About "FLINK-16406 Increase default value for JVM Metaspace"
>  - Have we consensus that this is okay for a bugfix release? It changes
> setups, takes away memory from heap / managed memory on existing setups
> that keep their flink-conf.yaml.

My understanding was that increasing to 256m resolved the reported problems
and we decided to make the change so I have merged it today as there were no 
more concerns.
If there are concerns I can revert it.

On the other hand, I think improving the message error with reference to the 
metaspace option should help the most
because user would not have to read all docs to fix it
then maybe this change is not even needed.

Best,
Andrey

> On 12 Mar 2020, at 12:28, Stephan Ewen  wrote:
> 
> Good idea to go ahead with 1.10.1
> 
> About "FLINK-16142 Memory Leak causes Metaspace OOM error on repeated job"
>  - I don't think we have consensus on the exact solution, yet, and some of
> the changes might also have side effects that are hard to predict, so I am
> not sure we should rush this in.
> 
> About "FLINK-16406 Increase default value for JVM Metaspace"
>  - Have we consensus that this is okay for a bugfix release? It changes
> setups, takes away memory from heap / managed memory on existing setups
> that keep their flink-conf.yaml.
> 
> We may need to unblock the release form these two issues and think about
> having 1.10.2 in the near future.
> 
> On Thu, Mar 12, 2020 at 7:15 AM Yu Li  wrote:
> 
>> Thanks for the reminder Jark. Will keep an eye on these two.
>> 
>> Best Regards,
>> Yu
>> 
>> 
>> On Thu, 12 Mar 2020 at 12:32, Jark Wu  wrote:
>> 
>>> Thanks for driving this release, Yu!
>>> +1 to start 1.10.1 release cycle.
>>> 
>>> From the Table SQL module, I think we should also try to get in the
>>> following issues:
>>> - FLINK-16441: Allow users to override flink-conf parameters from SQL CLI
>>> environment
>>>  this allows users to set e.g. statebackend, watermark interval,
>>> exactly-once/at-least-once, in the SQL CLI
>>> - FLINK-16217: SQL Client crashed when any uncatched exception is thrown
>>>  this will improve much experience when using SQL CLI
>>> 
>>> Best,
>>> Jark
>>> 
>>> 
>>> On Wed, 11 Mar 2020 at 20:37, Yu Li  wrote:
>>> 
>>>> Thanks for the suggestion Andrey! I've added 1.10.1 into FLINK-16225
>> fix
>>>> versions and promoted its priority to Critical. Will also watch the
>>>> progress of FLINK-16108/FLINK-16408.
>>>> 
>>>> Best Regards,
>>>> Yu
>>>> 
>>>> 
>>>> On Wed, 11 Mar 2020 at 18:18, Andrey Zagrebin 
>>>> wrote:
>>>> 
>>>>> Hi Yu,
>>>>> 
>>>>> Thanks for kicking off the 1.10.1 release discussion!
>>>>> 
>>>>> Apart from
>>>>> - FLINK-16406 Increase default value for JVM Metaspace to minimise
>> its
>>>>> OutOfMemoryError
>>>>> which should be merged soon
>>>>> 
>>>>> I think we should also try to get in the following issues:
>>>>> 
>>>>> - [FLINK-16225] Metaspace Out Of Memory should be handled as Fatal
>>> Error
>>>> in
>>>>> TaskManager
>>>>> This should solve the Metaspace problem even in a better way because
>>> OOM
>>>>> failure should point users to the docs immediately
>>>>> 
>>>>> - [FLINK-16408] Bind user code class loader to lifetime of a slot
>>>>> This should give a better protection against class loading leaks
>>>>> 
>>>>> - [FLINK-16018] Improve error reporting when submitting batch job
>>>> (instead
>>>>> of AskTimeoutException)
>>>>> This problem has recently happened for multiple users
>>>>> 
>>>>> Best,
>>>>> 

[DISCUSS] FLIP 116: Unified Memory Configuration for Job Managers

[Andrey Zagrebin]

Hi All,

As you may have noticed, 1.10 release included an extensive improvements to
memory management and configuration of Task Managers, FLIP-49: [1]. The
memory configuration of Job Managers has not been touched in 1.10.

Although, Job Manager's memory model does not look so sophisticated as
for Task Managers, It makes to align Job Manager memory model and settings
with Task Managers. Therefore, we propose to reconsider it as well in 1.11
and I prepared a FLIP 116 [2] for that.

Any feedback is appreciated.

So far, there is one discussion point about how to address native
non-direct memory usage of user code. The user code can be run e.g. in
certain job submission scenarios within the JM process. For simplicity,
FLIP suggests only an option for direct memory which is translated into the
setting of the JVM direct memory limit.
Although, we documented for TM that the similar parameters can also
address native non-direct memory usage [3], this can lead to wrong
functioning of the JVM direct memory limit. The direct memory option in JM
could be also named in more general way, e.g. off-heap memory but this
naming would somewhat hide its nature of JVM direct memory limit.
On the other hand, JVM Overhead does not suffer from this problem and
affects only the container/worker memory size which is the most important
matter to address for the native non-direct memory consumption. The caveat
here is that JVM Overhead was not supposed to be used by any Flink or user
components.

Thanks,
Andrey

[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-49%3A+Unified+Memory+Configuration+for+TaskExecutors
[2]
https://cwiki.apache.org/confluence/display/FLINK/FLIP+116%3A+Unified+Memory+Configuration+for+Job+Managers
[3]
https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/memory/mem_detail.html#overview

Re: [DISCUSS] Releasing Flink 1.10.1

[Andrey Zagrebin]

Hi Yu,

Thanks for kicking off the 1.10.1 release discussion!

Apart from
- FLINK-16406 Increase default value for JVM Metaspace to minimise its
OutOfMemoryError
which should be merged soon

I think we should also try to get in the following issues:

- [FLINK-16225] Metaspace Out Of Memory should be handled as Fatal Error in
TaskManager
This should solve the Metaspace problem even in a better way because OOM
failure should point users to the docs immediately

- [FLINK-16408] Bind user code class loader to lifetime of a slot
This should give a better protection against class loading leaks

- [FLINK-16018] Improve error reporting when submitting batch job (instead
of AskTimeoutException)
This problem has recently happened for multiple users

Best,
Andrey


On Wed, Mar 11, 2020 at 8:46 AM Jingsong Li  wrote:

> Thanks for driving. Yu. +1 for starting the 1.10.1 release.
>
> Some issues are very important, Users are looking forward to them.
>
> Best,
> Jingsong Lee
>
> On Wed, Mar 11, 2020 at 2:52 PM Yangze Guo  wrote:
>
> > Thanks for driving this release, Yu!
> >
> > +1 for starting the 1.10.1 release cycle.
> >
> > Best,
> > Yangze Guo
> >
> > On Wed, Mar 11, 2020 at 1:42 PM Xintong Song 
> > wrote:
> > >
> > > Yu,
> > > Thanks for the explanation.
> > > I've no concerns. I was just trying to get some inputs for prioritizing
> > > tasks on my side, and ~1month sounds good to me.
> > >
> > >
> > > Thank you~
> > >
> > > Xintong Song
> > >
> > >
> > >
> > > On Wed, Mar 11, 2020 at 12:15 PM Yu Li  wrote:
> > >
> > > > bq. what is the time plan for 1.10.1?
> > > >
> > > > According to the history, the first patch release of a major version
> > will
> > > > take ~1month from discussion started, depending on the speed of
> blocker
> > > > issue resolving:
> > > > * 1.8.1: started discussion on May 28th [1], released on Jul 3rd [2]
> > > > * 1.9.1: started discussion on Sep 23rd [3], released on Oct 19th [4]
> > > >
> > > > We won't rush to match the history of course, but could use it as a
> > > > reference. And please feel free to let me know if any concerns
> Xintong.
> > > > Thanks.
> > > >
> > > > Best Regards,
> > > > Yu
> > > >
> > > > [1]
> > > >
> > > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Releasing-Flink-1-8-1-td29154.html
> > > > [2]
> > > >
> > > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/ANNOUNCE-Apache-Flink-1-8-1-released-td30124.html
> > > > [3]
> > > >
> > > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Releasing-Flink-1-9-1-td33343.html
> > > > [4]
> > > >
> > > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/ANNOUNCE-Apache-Flink-1-9-1-released-td34170.html
> > > >
> > > >
> > > > On Wed, 11 Mar 2020 at 11:54, Xintong Song 
> > wrote:
> > > >
> > > > > Thanks Yu, for the kick off and volunteering to be the release
> > manager.
> > > > >
> > > > > +1 for the proposal.
> > > > >
> > > > >
> > > > > One quick question, what is the time plan for 1.10.1?
> > > > >
> > > > >
> > > > > Thank you~
> > > > >
> > > > > Xintong Song
> > > > >
> > > > >
> > > > >
> > > > > On Wed, Mar 11, 2020 at 11:51 AM Zhijiang
> > > > >  wrote:
> > > > >
> > > > > > Thanks for driving this release, Yu!
> > > > > > +1 on my side
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > Best,
> > > > > > Zhijiang
> > > > > >
> > > > > >
> > > > > >
> --
> > > > > > From:Yu Li 
> > > > > > Send Time:2020 Mar. 10 (Tue.) 20:25
> > > > > > To:dev 
> > > > > > Subject:Re: [DISCUSS] Releasing Flink 1.10.1
> > > > > >
> > > > > > Thanks for the supplement Hequn. Yes will also keep an eye on
> these
> > > > > > existing blocker issues.
> > > > > >
> > > > > > Best Regards,
> > > > > > Yu
> > > > > >
> > > > > >
> > > > > > On Tue, 10 Mar 2020 at 19:10, Hequn Cheng 
> > wrote:
> > > > > >
> > > > > > > Hi Yu,
> > > > > > >
> > > > > > > Thanks a lot for raising the discussion and volunteer as the
> > release
> > > > > > > manager!
> > > > > > >
> > > > > > > I found there are some other issues[1] which are marked as a
> > blocker:
> > > > > > > - FLINK-16454 Update the copyright year in NOTICE files
> > > > > > > - FLINK-16262 Class loader problem with
> > > > > > > FlinkKafkaProducer.Semantic.EXACTLY_ONCE and usrlib directory
> > > > > > > - FLINK-16170 SearchTemplateRequest ClassNotFoundException when
> > use
> > > > > > > flink-sql-connector-elasticsearch7
> > > > > > > - FLINK-16018 Improve error reporting when submitting batch job
> > > > > (instead
> > > > > > of
> > > > > > > AskTimeoutException)
> > > > > > >
> > > > > > > These may also need to be resolved in 1.10.1.
> > > > > > >
> > > > > > > Best,
> > > > > > > Hequn
> > > > > > >
> > > > > > > [1]
> > https://issues.apache.org/jira/projects/FLINK/versions/12346891
> > > > > > >
> > > > > > >
> > > > > > > On Tue, Mar 10, 2020 at 6:48 PM Yu Li 
> 

Re: [DISCUSS] FLIP-111: Docker image unification

[Andrey Zagrebin]

more flexible, e.g. it makes it very easy to
> pass values of Kubernetes Secrets into the Flink configuration.

This is indeed an interesting option to pass arguments to the entry point
in general.
For the config options, the dynamic args can be a better option as
mentioned above.

With respect to logging, I would opt to keep this very basic and to only
> support logging to the console (maybe with a fix for the web user
> interface). For everything else, users can easily build their own images
> based on library/flink (provide the dependencies, change the logging
> configuration).

agree

Thanks,
Andrey

On Sun, Mar 8, 2020 at 8:55 PM Konstantin Knauf 
wrote:

> Hi Andrey,
>
> thanks a lot for this proposal. The variety of Docker files in the project
> has been causing quite some confusion.
>
> For the entrypoint, have you considered to also allow setting
> configuration via environment variables as in "docker run -e
> FLINK_REST_BIN_PORT=8081 ..."? This is quite common and more flexible, e.g.
> it makes it very easy to pass values of Kubernetes Secrets into the Flink
> configuration.
>
> With respect to logging, I would opt to keep this very basic and to only
> support logging to the console (maybe with a fix for the web user
> interface). For everything else, users can easily build their own images
> based on library/flink (provide the dependencies, change the logging
> configuration).
>
> Cheers,
>
> Konstantin
>
>
> On Thu, Mar 5, 2020 at 11:01 AM Yang Wang  wrote:
>
>> Hi Andrey,
>>
>>
>> Thanks for driving this significant FLIP. From the user ML, we could also
>> know there are
>> many users running Flink in container environment. Then the docker image
>> will be the
>> very basic requirement. Just as you say, we should provide a unified
>> place for all various
>> usage(e.g. session, job, native k8s, swarm, etc.).
>>
>>
>> > About docker utils
>>
>> I really like the idea to provide some utils for the docker file and
>> entry point. The
>> `flink_docker_utils` will help to build the image easier. I am not sure
>> about the
>> `flink_docker_utils start_jobmaster`. Do you mean when we build a docker
>> image, we
>> need to add `RUN flink_docker_utils start_jobmaster` in the docker file?
>> Why do we need this?
>>
>>
>> > About docker entry point
>>
>> I agree with you that the docker entry point could more powerful with
>> more functionality.
>> Mostly, it is about to override the config options. If we support dynamic
>> properties, i think
>> it is more convenient for users without any learning curve.
>> `docker run flink session_jobmanager -D rest.bind-port=8081`
>>
>>
>> > About the logging
>>
>> Updating the `log4j-console.properties` to support multiple appender is a
>> better option.
>> Currently, the native K8s is suggesting users to debug the logs in this
>> way[1]. However,
>> there is also some problems. The stderr and stdout of JM/TM processes
>> could not be
>> forwarded to the docker container console.
>>
>>
>> [1].
>> https://ci.apache.org/projects/flink/flink-docs-master/ops/deployment/native_kubernetes.html#log-files
>>
>>
>> Best,
>> Yang
>>
>>
>>
>>
>> Andrey Zagrebin  于2020年3月4日周三 下午5:34写道：
>>
>>> Hi All,
>>>
>>> If you have ever touched the docker topic in Flink, you
>>> probably noticed that we have multiple places in docs and repos which
>>> address its various concerns.
>>>
>>> We have prepared a FLIP [1] to simplify the perception of docker topic in
>>> Flink by users. It mostly advocates for an approach of extending official
>>> Flink image from the docker hub. For convenience, it can come with a set
>>> of
>>> bash utilities and documented examples of their usage. The utilities
>>> allow
>>> to:
>>>
>>>- run the docker image in various modes (single job, session master,
>>>task manager etc)
>>>- customise the extending Dockerfile
>>>- and its entry point
>>>
>>> Eventually, the FLIP suggests to remove all other user facing Dockerfiles
>>> and building scripts from Flink repo, move all docker docs to
>>> apache/flink-docker and adjust existing docker use cases to refer to this
>>> new approach (mostly Kubernetes now).
>>>
>>> The first contributed version of Flink docker integration also contained
>>> example and docs for the integration with Bluemix in IBM cloud. We also
>>> suggest to maintain it outside of Flink repository (cc Markus Müller).
>>>
>>> Thanks,
>>> Andrey
>>>
>>> [1]
>>>
>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-111%3A+Docker+image+unification
>>>
>>
>
> --
>
> Konstantin Knauf | Head of Product
>
> +49 160 91394525
>
>
> Follow us @VervericaData Ververica <https://www.ververica.com/>
>
>
> --
>
> Join Flink Forward <https://flink-forward.org/> - The Apache Flink
> Conference
>
> Stream Processing | Event Driven | Real Time
>
> --
>
> Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany
>
> --
> Ververica GmbH
> Registered at Amtsgericht Charlottenburg: HRB 158244 B
> Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason, Ji
> (Tony) Cheng
>

[DISCUSS] FLIP-111: Docker image unification

[Andrey Zagrebin]

Hi All,

If you have ever touched the docker topic in Flink, you
probably noticed that we have multiple places in docs and repos which
address its various concerns.

We have prepared a FLIP [1] to simplify the perception of docker topic in
Flink by users. It mostly advocates for an approach of extending official
Flink image from the docker hub. For convenience, it can come with a set of
bash utilities and documented examples of their usage. The utilities allow
to:

   - run the docker image in various modes (single job, session master,
   task manager etc)
   - customise the extending Dockerfile
   - and its entry point

Eventually, the FLIP suggests to remove all other user facing Dockerfiles
and building scripts from Flink repo, move all docker docs to
apache/flink-docker and adjust existing docker use cases to refer to this
new approach (mostly Kubernetes now).

The first contributed version of Flink docker integration also contained
example and docs for the integration with Bluemix in IBM cloud. We also
suggest to maintain it outside of Flink repository (cc Markus Müller).

Thanks,
Andrey

[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-111%3A+Docker+image+unification

[jira] [Created] (FLINK-16406) Increase default value for JVM Metaspace to minimise its OutOfMemoryError

[Andrey Zagrebin (Jira)]

Andrey Zagrebin created FLINK-16406:
---

 Summary: Increase default value for JVM Metaspace to minimise its 
OutOfMemoryError
 Key: FLINK-16406
 URL: https://issues.apache.org/jira/browse/FLINK-16406
 Project: Flink
  Issue Type: Improvement
  Components: Runtime / Configuration, Runtime / Task
Affects Versions: 1.10.0
Reporter: Andrey Zagrebin
Assignee: Andrey Zagrebin
 Fix For: 1.10.1, 1.11.0


With FLIP-49 ([FLINK-13980|https://issues.apache.org/jira/browse/FLINK-13980]), 
we introduced a limit for JVM Metaspace 
('taskmanager.memory.jvm-metaspace.size') when TM JVM process is started. It 
caused '_OutOfMemoryError: Metaspace_' for some use cases after upgrading to 
the latest 1.10 version. In some cases, a real class loading leak has been 
discovered, like in 
[FLINK-16142|https://issues.apache.org/jira/browse/FLINK-16142]. Some users had 
to increase the default value to accommodate for their use cases (mostly from 
96Mb to 256Mb).

While this limit was introduced to properly plan Flink resources, especially 
for container environment, and to detect class loading leaks, the user 
experience should be as smooth as possible. One way is provide good 
documentation for this change 
([FLINK-16278|https://issues.apache.org/jira/browse/FLINK-16278]).

Another question is the sanity of the default value. It is still arguable what 
the default value should be (currently 96Mb). In general, the size depends on 
the use case (job user code, how many jobs are deployed in the cluster etc).

This issue tries to tackle this problem by firstly increasing it to 256Mb. We 
also want to poll which Metaspace setting resolved the _OutOfMemoryError_. 
Please, if you encountered this problem, report here any relevant specifics of 
your job and your Metaspace size if there was no class loading leak.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-16198) FileUtilsTest fails on Mac OS

[Andrey Zagrebin (Jira)]

Andrey Zagrebin created FLINK-16198:
---

 Summary: FileUtilsTest fails on Mac OS
 Key: FLINK-16198
 URL: https://issues.apache.org/jira/browse/FLINK-16198
 Project: Flink
  Issue Type: Bug
  Components: FileSystems, Tests
Reporter: Andrey Zagrebin


The following tests fail if run on Mac OS (IDE/maven).

 

FileUtilsTest.testCompressionOnRelativePath:

 
{code:java}
java.nio.file.NoSuchFileException: 
../../../../../var/folders/67/v4yp_42d21j6_n8k1h556h0cgn/T/junit6496651678375117676/compressDir/rootDirjava.nio.file.NoSuchFileException:
 
../../../../../var/folders/67/v4yp_42d21j6_n8k1h556h0cgn/T/junit6496651678375117676/compressDir/rootDir
 at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86) at 
sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) at 
sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) at 
sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:384)
 at java.nio.file.Files.createDirectory(Files.java:674) at 
org.apache.flink.util.FileUtilsTest.verifyDirectoryCompression(FileUtilsTest.java:440)
 at 
org.apache.flink.util.FileUtilsTest.testCompressionOnRelativePath(FileUtilsTest.java:261)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:498) at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
 at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
 at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
 at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
 at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) at 
org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) at 
org.junit.rules.RunRules.evaluate(RunRules.java:20) at 
org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
 at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
 at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at 
org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at 
org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at 
org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at 
org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at 
org.junit.runners.ParentRunner.run(ParentRunner.java:363) at 
org.junit.runner.JUnitCore.run(JUnitCore.java:137) at 
com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
 at 
com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47)
 at 
com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242)
 at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
{code}
 

FileUtilsTest.testDeleteDirectoryConcurrently

 

 
{code:java}
java.nio.file.FileSystemException: 
/var/folders/67/v4yp_42d21j6_n8k1h556h0cgn/T/junit7558825557740784886/junit3566161583262218465/ab1fa0bde8b22cad58b717508c7a7300/121fdf5f7b057183843ed2e1298f9b66/6598025f390d3084d69c98b36e542fe2/8db7cd9c063396a19a86f5b63ce53f66:
 Invalid argument at 
sun.nio.fs.UnixException.translateToIOException(UnixException.java:91)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
at 
sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:244)
at 
sun.nio.fs.AbstractFileSystemProvider.deleteIfExists(AbstractFileSystemProvider.java:108)
at java.nio.file.Files.deleteIfExists(Files.java:1165)
at 
org.apache.flink.util.FileUtils.deleteFileOrDirectoryInternal(FileUtils.java:324)
at org.apache.flink.util.FileUtils.guardIfWindows(FileUtils.java:391)
at 
org.apache.flink.util.FileUtils.deleteFileOrDirectory(FileUtils.java:258)
at 
org.apache.flink.util.FileUtils.cleanDirectoryInternal(FileUtils.java:376)
at 
org.apache.flink.util.FileUtils.deleteDirectoryInternal(FileUtils.java:335)
at 
org.apache.flink.util.FileUtils.deleteFileOrDirectoryInternal(FileUtils.java:320)
at org.apache.flink.util.FileUtils.guardIfWindows(FileUtils.java:391)
at 
org.apache.flink.util.FileUtils.deleteFileOrDirectory(FileUtils.java:258)
at 
org.apache.flink.util.FileUtils.cleanDirectoryInternal(FileUtils.java:376)
at 
org.apache.flink.util.FileUtils.deleteDirectoryInternal(FileUtils.java:335

[jira] [Created] (FLINK-15991) Create Chinese documentation for FLIP-49 TM memory model

[Andrey Zagrebin (Jira)]

Andrey Zagrebin created FLINK-15991:
---

 Summary: Create Chinese documentation for FLIP-49 TM memory model
 Key: FLINK-15991
 URL: https://issues.apache.org/jira/browse/FLINK-15991
 Project: Flink
  Issue Type: Task
  Components: Documentation
Reporter: Andrey Zagrebin
Assignee: Xintong Song
 Fix For: 1.10.0, 1.11.0


Chinese translation of FLINK-15143



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] Release 1.10.0, release candidate #3

[Andrey Zagrebin]

Hi,

@Jingsong Lee
Regarding "OutOfMemoryError: Direct buffer memory" in
FileChannelBoundedData$FileBufferReader
I saw you created a specific issue issue:
https://issues.apache.org/jira/browse/FLINK-15981

In general, I think we could rewrap this error
in MemorySegmentFactory#allocateUnpooledOffHeapMemory,
e.g. suggesting to increase off heap memory option:
https://issues.apache.org/jira/browse/FLINK-15989
It can always happen independently from Flink if user code over-allocates
the direct memory somewhere else.

Thanks,
Andrey

On Tue, Feb 11, 2020 at 4:12 AM Yangze Guo  wrote:

> +1 (non-binding)
>
> - Build from source
> - Run mesos e2e tests(including unmerged heap state backend and rocks
> state backend case)
>
>
> Best,
> Yangze Guo
>
> On Tue, Feb 11, 2020 at 10:08 AM Yu Li  wrote:
> >
> > Thanks for the reminder Patrick! According to the release process [1] we
> > will publish the Dockerfiles *after* the RC voting passed, to finalize
> the
> > release.
> >
> > I have created FLINK-15978 [2] and prepared a PR [3] for it, will follow
> up
> > after we conclude our RC vote. Thanks.
> >
> > Best Regards,
> > Yu
> >
> > [1]
> >
> https://cwiki.apache.org/confluence/display/FLINK/Creating+a+Flink+Release
> > [2] https://issues.apache.org/jira/browse/FLINK-15978
> > [3] https://github.com/apache/flink-docker/pull/6
> >
> >
> > On Mon, 10 Feb 2020 at 20:57, Patrick Lucas 
> wrote:
> >
> > > Now that [FLINK-15828] Integrate docker-flink/docker-flink into Flink
> > > release process  is
> > > complete, the Dockerfiles for 1.10.0 can be published as part of the
> > > release process.
> > >
> > > @Gary/@Yu: please let me know if you have any questions regarding the
> > > workflow or its documentation.
> > >
> > > --
> > > Patrick
> > >
> > > On Mon, Feb 10, 2020 at 1:29 PM Benchao Li 
> wrote:
> > >
> > > > +1 (non-binding)
> > > >
> > > > - build from source
> > > > - start standalone cluster, and run some examples
> > > > - played with sql-client with some simple sql
> > > > - run tests in IDE
> > > > - run some sqls running in 1.9 internal version with 1.10.0-rc3,
> seems
> > > 1.10
> > > > behaves well.
> > > >
> > > > Xintong Song  于2020年2月10日周一 下午8:13写道：
> > > >
> > > > > +1 (non-binding)
> > > > >
> > > > > - build from source (with tests)
> > > > > - run nightly e2e tests
> > > > > - run example jobs in local/standalone/yarn setups
> > > > > - play around with memory configurations on local/standalone/yarn
> > > setups
> > > > >
> > > > > Thank you~
> > > > >
> > > > > Xintong Song
> > > > >
> > > > >
> > > > >
> > > > > On Mon, Feb 10, 2020 at 7:55 PM Jark Wu  wrote:
> > > > >
> > > > > > +1 (binding)
> > > > > >
> > > > > > - build the source release with Scala 2.12 and Scala 2.11
> > > successfully
> > > > > > - checked/verified signatures and hashes
> > > > > > - started cluster for both Scala 2.11 and 2.12, ran examples,
> > > verified
> > > > > web
> > > > > > ui and log output, nothing unexpected
> > > > > > - started cluster and run some e2e sql queries, all of them works
> > > well
> > > > > and
> > > > > > the results are as expected:
> > > > > >   - read from kafka source, aggregate, write into mysql
> > > > > >   - read from kafka source with watermark defined in ddl, window
> > > > > aggregate,
> > > > > > write into mysql
> > > > > >   - read from kafka with computed column defined in ddl, temporal
> > > join
> > > > > with
> > > > > > a mysql table, write into kafka
> > > > > >
> > > > > > Cheers,
> > > > > > Jark
> > > > > >
> > > > > >
> > > > > > On Mon, 10 Feb 2020 at 19:23, Kurt Young 
> wrote:
> > > > > >
> > > > > > > +1 (binding)
> > > > > > >
> > > > > > > - verified signatures and checksums
> > > > > > > - start local cluster, run some examples, randomly play some
> sql
> > > with
> > > > > sql
> > > > > > > client, no suspicious error/warn log found in log files
> > > > > > > - repeat above operation with both scala 2.11 and 2.12 binary
> > > > > > >
> > > > > > > Best,
> > > > > > > Kurt
> > > > > > >
> > > > > > >
> > > > > > > On Mon, Feb 10, 2020 at 6:38 PM Yang Wang <
> danrtsey...@gmail.com>
> > > > > wrote:
> > > > > > >
> > > > > > > >  +1 non-binding
> > > > > > > >
> > > > > > > >
> > > > > > > > - Building from source with all tests skipped
> > > > > > > > - Build a custom image with 1.10-rc3
> > > > > > > > - K8s tests
> > > > > > > > * Deploy a standalone session cluster on K8s and submit
> > > > multiple
> > > > > > jobs
> > > > > > > > * Deploy a standalone per-job cluster
> > > > > > > > * Deploy a native session cluster on K8s with/without HA
> > > > > > configured,
> > > > > > > > kill TM and jobs could recover successfully
> > > > > > > >
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Yang
> > > > > > > >
> > > > > > > > Jingsong Li  于2020年2月10日周一 下午4:29写道：
> > > > > > > >
> > > > > > > > > Hi,
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > +1 (non-binding) Thanks for driving this, 

[jira] [Created] (FLINK-15989) Rewrap OutOfMemoryError in allocateUnpooledOffHeap with better message

[Andrey Zagrebin (Jira)]

Andrey Zagrebin created FLINK-15989:
---

 Summary: Rewrap OutOfMemoryError in allocateUnpooledOffHeap with 
better message
 Key: FLINK-15989
 URL: https://issues.apache.org/jira/browse/FLINK-15989
 Project: Flink
  Issue Type: Improvement
Reporter: Andrey Zagrebin
 Fix For: 1.10.1, 1.11.0


Now if Flink allocates direct memory in 
MemorySegmentFactory#allocateUnpooledOffHeapMemory and its limit is exceeded 
for any reason, e.g. user code over-allocated direct memory, 
ByteBuffer#allocateDirect will throw a generic "OutOfMemoryError: Direct buffer 
memory". We can catch it and add a message which provides more explanation and 
points to an option taskmanager.memory.task.off-heap.size to increase as a 
possible solution.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-15946) Task manager Kubernetes pods take long time to terminate

[Andrey Zagrebin (Jira)]

Andrey Zagrebin created FLINK-15946:
---

 Summary: Task manager Kubernetes pods take long time to terminate
 Key: FLINK-15946
 URL: https://issues.apache.org/jira/browse/FLINK-15946
 Project: Flink
  Issue Type: Bug
  Components: Deployment / Kubernetes, Runtime / Coordination
Affects Versions: 1.10.0
Reporter: Andrey Zagrebin


The problem is initially described in this [ML 
thread|https://mail-archives.apache.org/mod_mbox/flink-user/202002.mbox/browser].

We should investigate whether and if yes, why the TM pod killing/shutdown is 
delayed by reconnecting to the terminated JM.

cc [~fly_in_gis]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-15942) Improve logging of infinite resource profile

[Andrey Zagrebin (Jira)]

Andrey Zagrebin created FLINK-15942:
---

 Summary: Improve logging of infinite resource profile
 Key: FLINK-15942
 URL: https://issues.apache.org/jira/browse/FLINK-15942
 Project: Flink
  Issue Type: Improvement
  Components: Runtime / Configuration, Runtime / Task
Affects Versions: 1.10.0
Reporter: Andrey Zagrebin


After we set task memory and CPU to infinity in FLINK-15763, it spoiled the 
logs:
{code:java}
00:23:49,442 INFO org.apache.flink.runtime.taskexecutor.slot.TaskSlotTableImpl 
- Free slot TaskSlot(index:0, state:ACTIVE, resource profile: 
ResourceProfile{cpuCores=44942328371557892500.,
 taskHeapMemory=2097152.000tb (2305843009213693951 bytes), 
taskOffHeapMemory=2097152.000tb (2305843009213693951 bytes), 
managedMemory=20.000mb (20971520 bytes), networkMemory=16.000mb (16777216 
bytes)}, allocationId: 349dacfbf1ac4d0b44a2d11e1976d264, jobId: 
689a0cf24b40f16b6f45157f78754c46).
{code}
We should treat the infinity as a special case and print it accordingly



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] Release 1.10.0, release candidate #2

[Andrey Zagrebin]

alright, thanks for confirming this Benchao!

On Thu, Feb 6, 2020 at 6:36 PM Benchao Li  wrote:

> Hi Andrey,
>
> I noticed that 1.10 has changed to enabling background cleanup by default
> just after I posted to this email.
> So it won't affect 1.10 any more, just affect 1.9.x. We can move to the
> Jira ticket to discuss further more.
>
> Andrey Zagrebin  于2020年2月6日周四 下午11:30写道：
>
> > Hi Benchao,
> >
> > Do you observe this issue FLINK-15938 with 1.9 or 1.10?
> > If with 1.9, I suggest to check with 1.10.
> >
> > Thanks,
> > Andrey
> >
> > On Thu, Feb 6, 2020 at 4:07 PM Benchao Li  wrote:
> >
> > > Hi all,
> > >
> > > I found another issue[1], I don't know if it should be a blocker. But
> it
> > > does affects joins without window in blink planner.
> > >
> > > [1] https://issues.apache.org/jira/browse/FLINK-15938
> > >
> > > Jeff Zhang  于2020年2月6日周四 下午5:05写道：
> > >
> > > > Hi Jingsong,
> > > >
> > > > Thanks for the suggestion. It works for running it in IDE, but for
> > > > downstream project like Zeppelin where I will include flink jars in
> > > > classpath.
> > > > it only works when I specify the jars one by one explicitly in
> > classpath,
> > > > using * doesn't work.
> > > >
> > > > e.g.
> > > >
> > > > The following command where I use * to specify classpath doesn't
> work,
> > > > jzhang 4794 1.2 6.3 8408904 1048828 s007 S 4:30PM 0:33.90
> > > >
> > /Library/Java/JavaVirtualMachines/jdk1.8.0_211.jdk/Contents/Home/bin/java
> > > > -Dfile.encoding=UTF-8
> > > >
> > > >
> > >
> >
> -Dlog4j.configuration=file:///Users/jzhang/github/zeppelin/conf/log4j.properties
> > > >
> > > >
> > >
> >
> -Dzeppelin.log.file=/Users/jzhang/github/zeppelin/logs/zeppelin-interpreter-flink-shared_process-jzhang-JeffdeMacBook-Pro.local.log
> > > > -Xms1024m -Xmx2048m -XX:MaxPermSize=512m -cp
> > > >
> > > >
> > >
> >
> *:/Users/jzhang/github/flink/build-target/lib/**:/Users/jzhang/github/zeppelin/interpreter/flink/*:/Users/jzhang/github/zeppelin/zeppelin-interpreter-api/target/*:/Users/jzhang/github/zeppelin/zeppelin-zengine/target/test-classes/*::/Users/jzhang/github/zeppelin/interpreter/zeppelin-interpreter-api-0.9.0-SNAPSHOT.jar:/Users/jzhang/github/zeppelin/zeppelin-interpreter/target/classes:/Users/jzhang/github/zeppelin/zeppelin-zengine/target/test-classes:/Users/jzhang/github/flink/build-target/opt/flink-python_2.11-1.10-SNAPSHOT.jar
> > > > org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer
> 0.0.0.0
> > > > 52395 flink-shared_process :
> > > >
> > > >
> > > > While this command where I specify jar one by one explicitly in
> > classpath
> > > > works
> > > >
> > > > jzhang5660 205.2  6.1  8399420 1015036 s005  R 4:43PM
> > > > 0:24.82
> > > >
> > /Library/Java/JavaVirtualMachines/jdk1.8.0_211.jdk/Contents/Home/bin/java
> > > > -Dfile.encoding=UTF-8
> > > >
> > > >
> > >
> >
> -Dlog4j.configuration=file:///Users/jzhang/github/zeppelin/conf/log4j.properties
> > > >
> > > >
> > >
> >
> -Dzeppelin.log.file=/Users/jzhang/github/zeppelin/logs/zeppelin-interpreter-flink-shared_process-jzhang-JeffdeMacBook-Pro.local.log
> > > > -Xms1024m -Xmx2048m -XX:MaxPermSize=512m -cp
> > > >
> > > >
> > >
> >
> *:/Users/jzhang/github/flink/build-target/lib/slf4j-log4j12-1.7.15.jar:/Users/jzhang/github/flink/build-target/lib/flink-connector-hive_2.11-1.10-SNAPSHOT.jar:/Users/jzhang/github/flink/build-target/lib/hive-exec-2.3.4.jar:/Users/jzhang/github/flink/build-target/lib/flink-shaded-hadoop-2-uber-2.7.5-7.0.jar:/Users/jzhang/github/flink/build-target/lib/log4j-1.2.17.jar:/Users/jzhang/github/flink/build-target/lib/flink-table-blink_2.11-1.10-SNAPSHOT.jar:/Users/jzhang/github/flink/build-target/lib/flink-table_2.11-1.10-SNAPSHOT.jar:/Users/jzhang/github/flink/build-target/lib/flink-dist_2.11-1.10-SNAPSHOT.jar:/Users/jzhang/github/flink/build-target/lib/flink-python_2.11-1.10-SNAPSHOT.jar:/Users/jzhang/github/flink/build-target/lib/flink-hadoop-compatibility_2.11-1.9.1.jar*:/Users/jzhang/github/zeppelin/interpreter/flink/*:/Users/jzhang/github/zeppelin/zeppelin-interpreter-api/target/*:/Users/jzhang/github/zeppelin/zeppelin-zengine/target/test-classes/*::/Users/jzhang/github/zeppelin/interpreter/zeppelin-interpreter-api-0.9.0-SNAPSHOT.jar:/U

Re: [VOTE] Release 1.10.0, release candidate #2

[Andrey Zagrebin]

Hi Benchao,

Do you observe this issue FLINK-15938 with 1.9 or 1.10?
If with 1.9, I suggest to check with 1.10.

Thanks,
Andrey

On Thu, Feb 6, 2020 at 4:07 PM Benchao Li  wrote:

> Hi all,
>
> I found another issue[1], I don't know if it should be a blocker. But it
> does affects joins without window in blink planner.
>
> [1] https://issues.apache.org/jira/browse/FLINK-15938
>
> Jeff Zhang  于2020年2月6日周四 下午5:05写道：
>
> > Hi Jingsong,
> >
> > Thanks for the suggestion. It works for running it in IDE, but for
> > downstream project like Zeppelin where I will include flink jars in
> > classpath.
> > it only works when I specify the jars one by one explicitly in classpath,
> > using * doesn't work.
> >
> > e.g.
> >
> > The following command where I use * to specify classpath doesn't work,
> > jzhang 4794 1.2 6.3 8408904 1048828 s007 S 4:30PM 0:33.90
> > /Library/Java/JavaVirtualMachines/jdk1.8.0_211.jdk/Contents/Home/bin/java
> > -Dfile.encoding=UTF-8
> >
> >
> -Dlog4j.configuration=file:///Users/jzhang/github/zeppelin/conf/log4j.properties
> >
> >
> -Dzeppelin.log.file=/Users/jzhang/github/zeppelin/logs/zeppelin-interpreter-flink-shared_process-jzhang-JeffdeMacBook-Pro.local.log
> > -Xms1024m -Xmx2048m -XX:MaxPermSize=512m -cp
> >
> >
> *:/Users/jzhang/github/flink/build-target/lib/**:/Users/jzhang/github/zeppelin/interpreter/flink/*:/Users/jzhang/github/zeppelin/zeppelin-interpreter-api/target/*:/Users/jzhang/github/zeppelin/zeppelin-zengine/target/test-classes/*::/Users/jzhang/github/zeppelin/interpreter/zeppelin-interpreter-api-0.9.0-SNAPSHOT.jar:/Users/jzhang/github/zeppelin/zeppelin-interpreter/target/classes:/Users/jzhang/github/zeppelin/zeppelin-zengine/target/test-classes:/Users/jzhang/github/flink/build-target/opt/flink-python_2.11-1.10-SNAPSHOT.jar
> > org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer 0.0.0.0
> > 52395 flink-shared_process :
> >
> >
> > While this command where I specify jar one by one explicitly in classpath
> > works
> >
> > jzhang5660 205.2  6.1  8399420 1015036 s005  R 4:43PM
> > 0:24.82
> > /Library/Java/JavaVirtualMachines/jdk1.8.0_211.jdk/Contents/Home/bin/java
> > -Dfile.encoding=UTF-8
> >
> >
> -Dlog4j.configuration=file:///Users/jzhang/github/zeppelin/conf/log4j.properties
> >
> >
> -Dzeppelin.log.file=/Users/jzhang/github/zeppelin/logs/zeppelin-interpreter-flink-shared_process-jzhang-JeffdeMacBook-Pro.local.log
> > -Xms1024m -Xmx2048m -XX:MaxPermSize=512m -cp
> >
> >
> *:/Users/jzhang/github/flink/build-target/lib/slf4j-log4j12-1.7.15.jar:/Users/jzhang/github/flink/build-target/lib/flink-connector-hive_2.11-1.10-SNAPSHOT.jar:/Users/jzhang/github/flink/build-target/lib/hive-exec-2.3.4.jar:/Users/jzhang/github/flink/build-target/lib/flink-shaded-hadoop-2-uber-2.7.5-7.0.jar:/Users/jzhang/github/flink/build-target/lib/log4j-1.2.17.jar:/Users/jzhang/github/flink/build-target/lib/flink-table-blink_2.11-1.10-SNAPSHOT.jar:/Users/jzhang/github/flink/build-target/lib/flink-table_2.11-1.10-SNAPSHOT.jar:/Users/jzhang/github/flink/build-target/lib/flink-dist_2.11-1.10-SNAPSHOT.jar:/Users/jzhang/github/flink/build-target/lib/flink-python_2.11-1.10-SNAPSHOT.jar:/Users/jzhang/github/flink/build-target/lib/flink-hadoop-compatibility_2.11-1.9.1.jar*:/Users/jzhang/github/zeppelin/interpreter/flink/*:/Users/jzhang/github/zeppelin/zeppelin-interpreter-api/target/*:/Users/jzhang/github/zeppelin/zeppelin-zengine/target/test-classes/*::/Users/jzhang/github/zeppelin/interpreter/zeppelin-interpreter-api-0.9.0-SNAPSHOT.jar:/Users/jzhang/github/zeppelin/zeppelin-interpreter/target/classes:/Users/jzhang/github/zeppelin/zeppelin-zengine/target/test-classes:/Users/jzhang/github/flink/build-target/opt/flink-python_2.11-1.10-SNAPSHOT.jar
> >
> >
> /Users/jzhang/github/flink/build-target/opt/tmp/flink-python_2.11-1.10-SNAPSHOT.jar
> > org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer 0.0.0.0
> > 52603 flink-shared_process :
> >
> >
> > Jingsong Li  于2020年2月6日周四 下午4:10写道：
> >
> > > Hi Jeff,
> > >
> > >
> > > For FLINK-15935 [1],
> > >
> > > I try to think of it as a non blocker. But it's really an important
> > issue.
> > >
> > >
> > > The problem is the class loading order. We want to load the class in
> the
> > > blink-planner.jar, but actually load the class in the
> flink-planner.jar.
> > >
> > >
> > > First of all, the order of class loading is based on the order of
> > > classpath.
> > >
> > >
> > > I just tried, the order of classpath of the folder is depends on the
> > order
> > > of file names.
> > >
> > > -That is to say, our order is OK now: because
> > > flink-table-blink_2.11-1.11-SNAPSHOT.jar is before the name order of
> > > flink-table_2.11-1.11-snapshot.jar.
> > >
> > > -But once I change the name of flink-table_2.11-1.11-SNAPSHOT.jar to
> > > aflink-table_2.11-1.11-SNAPSHOT.jar, an error will be reported.
> > >
> > >
> > > The order of classpaths should be influenced by the ls of Linux. By
> > default
> > > the ls command is listing the files in 

[jira] [Created] (FLINK-15774) Consider adding an explicit network memory size config option

[Andrey Zagrebin (Jira)]

Andrey Zagrebin created FLINK-15774:
---

 Summary: Consider adding an explicit network memory size  config 
option
 Key: FLINK-15774
 URL: https://issues.apache.org/jira/browse/FLINK-15774
 Project: Flink
  Issue Type: Improvement
  Components: Runtime / Configuration, Runtime / Task
Reporter: Andrey Zagrebin


See [PR 
discussion|[https://github.com/apache/flink/pull/10946#discussion_r371169251]]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-15763) Set necessary resource configuration options to defaults for local execution ignoring FLIP-49

[Andrey Zagrebin (Jira)]

Andrey Zagrebin created FLINK-15763:
---

 Summary: Set necessary resource configuration options to defaults 
for local execution ignoring FLIP-49
 Key: FLINK-15763
 URL: https://issues.apache.org/jira/browse/FLINK-15763
 Project: Flink
  Issue Type: Improvement
  Components: Runtime / Task
Reporter: Andrey Zagrebin
Assignee: Andrey Zagrebin
 Fix For: 1.10.0






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-15758) Investigate potential out-of-memory problems due to managed unsafe memory allocation

[Andrey Zagrebin (Jira)]

Andrey Zagrebin created FLINK-15758:
---

 Summary: Investigate potential out-of-memory problems due to 
managed unsafe memory allocation
 Key: FLINK-15758
 URL: https://issues.apache.org/jira/browse/FLINK-15758
 Project: Flink
  Issue Type: Task
  Components: Runtime / Task
Reporter: Andrey Zagrebin
 Fix For: 1.11.0






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-15741) Fix TTL docs after enabling RocksDB compaction filter by default

[Andrey Zagrebin (Jira)]

Andrey Zagrebin created FLINK-15741:
---

 Summary: Fix TTL docs after enabling RocksDB compaction filter by 
default
 Key: FLINK-15741
 URL: https://issues.apache.org/jira/browse/FLINK-15741
 Project: Flink
  Issue Type: Task
  Components: Runtime / State Backends
Reporter: Andrey Zagrebin
Assignee: Andrey Zagrebin
 Fix For: 1.10.0


RocksDB compaction filter is always enabled by default after [FLINK-15506 
|https://issues.apache.org/jira/browse/FLINK-15506]and we deprecated its 
disabling. The docs should not refer to its enabling/disabling.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] Change default for RocksDB timers: Java Heap => in RocksDB

[Andrey Zagrebin]

Hi Stephan,

Thanks for starting this discussion. I am +1 for this change.
In general, number of timer state keys can have the same order as number of
main state keys.
So if RocksDB is used for main state for scalability, it makes sense to
have timers there as well
unless timers are used for only very limited subset of keys which fits into
memory.

Best,
Andrey

On Thu, Jan 16, 2020 at 4:27 PM Stephan Ewen  wrote:

> Hi all!
>
> I would suggest a change of the current default for timers. A bit of
> background:
>
>   - Timers (for windows, process functions, etc.) are state that is
> managed and checkpointed as well.
>   - When using the MemoryStateBackend and the FsStateBackend, timers are
> kept on the JVM heap, like regular state.
>   - When using the RocksDBStateBackend, timers can be kept in RocksDB
> (like other state) or on the JVM heap. The JVM heap is the default though!
>
> I find this a bit un-intuitive and would propose to change this to let the
> RocksDBStateBackend store all state in RocksDB by default.
> The rationale being that if there is a tradeoff (like here), safe and
> scalable should be the default and unsafe performance be an explicit choice.
>
> This sentiment seems to be shared by various users as well, see
> https://twitter.com/StephanEwen/status/1214590846168903680 and
> https://twitter.com/StephanEwen/status/1214594273565388801
> We would of course keep the switch and mention in the performance tuning
> section that this is an option.
>
> # RocksDB State Backend Timers on Heap
>   - Pro: faster
>   - Con: not memory safe, GC overhead, longer synchronous checkpoint time,
> no incremental checkpoints
>
> #  RocksDB State Backend Timers on in RocksDB
>   - Pro: safe and scalable, asynchronously and incrementally checkpointed
>   - Con: performance overhead.
>
> Please chime in and let me know what you think.
>
> Best,
> Stephan
>
>

[jira] [Created] (FLINK-15621) State TTL: Remove deprecated option and method to disable TTL compaction filter

[Andrey Zagrebin (Jira)]

Andrey Zagrebin created FLINK-15621:
---

 Summary: State TTL: Remove deprecated option and method to disable 
TTL compaction filter
 Key: FLINK-15621
 URL: https://issues.apache.org/jira/browse/FLINK-15621
 Project: Flink
  Issue Type: Task
Reporter: Andrey Zagrebin
 Fix For: 1.11.0


Follow-up for FLINK-15506.
 * Remove RocksDBOptions#TTL_COMPACT_FILTER_ENABLED
 * Remove 
RocksDBStateBackend#enableTtlCompactionFilter/isTtlCompactionFilterEnabled/disableTtlCompactionFilter,
 also in python API
 * Cleanup code from this flag and tests, also in python API
 * Check any related code in 
[frocksdb|[https://github.com/dataArtisans/frocksdb]] if any



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-15620) State TTL: Remove deprecated enable default background cleanup

[Andrey Zagrebin (Jira)]

Andrey Zagrebin created FLINK-15620:
---

 Summary: State TTL: Remove deprecated enable default background 
cleanup
 Key: FLINK-15620
 URL: https://issues.apache.org/jira/browse/FLINK-15620
 Project: Flink
  Issue Type: Task
  Components: Runtime / State Backends
Reporter: Andrey Zagrebin
 Fix For: 1.11.0






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-15606) Deprecate enable default background cleanup of state with TTL

[Andrey Zagrebin (Jira)]

Andrey Zagrebin created FLINK-15606:
---

 Summary: Deprecate enable default background cleanup of state with 
TTL
 Key: FLINK-15606
 URL: https://issues.apache.org/jira/browse/FLINK-15606
 Project: Flink
  Issue Type: Improvement
  Components: Runtime / State Backends
Reporter: Andrey Zagrebin
Assignee: Andrey Zagrebin
 Fix For: 1.10.0


Follow-up for FLINK-14898.

Enabling TTL without any background cleanup does not make too much
 sense. So we can keep it always enabled, just cleanup settings can be
 tweaked for particular backends.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-15605) Remove deprecated in 1.9 StateTtlConfig.TimeCharacteristic

[Andrey Zagrebin (Jira)]

Andrey Zagrebin created FLINK-15605:
---

 Summary: Remove deprecated in 1.9 StateTtlConfig.TimeCharacteristic
 Key: FLINK-15605
 URL: https://issues.apache.org/jira/browse/FLINK-15605
 Project: Flink
  Issue Type: Improvement
  Components: Runtime / State Backends
Reporter: Andrey Zagrebin
Assignee: Andrey Zagrebin
 Fix For: 1.10.0






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] decentralized scheduling strategy is needed

[Andrey Zagrebin]

HI HuWeihua,

I think your issue should resolve with 1.9.2 and 1.10 (not released but in
progress).
You can check the related Jira ticket [1].

Best,
Andrey

[1] https://jira.apache.org/jira/browse/FLINK-12122

On Wed, Jan 15, 2020 at 10:08 AM HuWeihua  wrote:

> Hi, All
> We encountered some problems during the upgrade from Flink 1.5 to Flink
> 1.9. Flink's scheduling strategy has changed. Flink 1.9 prefers centralized
> scheduling, while Flink 1.5 prefers decentralized scheduling. This change
> has caused resources imbalance and blocked our upgrade plan. We have
> thousands of jobs that need to be upgraded.
>
> For example,
> There is a job with 10 sources and 100 sinks. Each source need 1 core and
> each sink need 0.1 core.
> Try to run this job on Yarn, configure the numberOfTaskSlots is 10,
> yarn.containers.vcores is 2.
>
> When using Flink-1.5:
> Each TaskManager will run 1 source and 9 sinks, they need 1.9 cores
> totally. So the job with this configuration works very well. The schedule
> results is shown in Figure 1.
> When using Flink-1.9:
> The 10 sources will be scheduled to one TaskManager  and the 100 sinks
> will scheduled to other 10 TaskManagers.  The schedule results is shown
> in Figure 2.
> In this scenario, the TaskManager which run sources need 10 cores, other
> TaskManagers need 1 cores. But TaskManager must be configured the same, So
> we need 11 TaskManager with 10 cores.
> This situation waste (10-2)*11 = 88 cores more than Flink 1.5.
>
> In addition to the waste of resources, we also encountered other problems
> caused by centralized scheduling strategy.
>
>1. Network bandwidth. Tasks of the same type are scheduled to the one
>TaskManager, causing too much network traffic on the machine.
>
>
>1. Some jobs need to sink to the local agent. After centralized
>scheduling, the insufficient processing capacity of the single machine
>causes a backlog of consumption.
>
>
> In summary, we think a decentralized scheduling strategy is necessary.
>
>
> Figure 1. Flink 1.5 schedule results
>
> Figure 2. Flink 1.9 schedule results
>
>
>
> Best
> Weihua Hu
>
>

[jira] [Created] (FLINK-15597) Relax sanity check of JVM memory overhead to be within its min/max

[Andrey Zagrebin (Jira)]

Andrey Zagrebin created FLINK-15597:
---

 Summary: Relax sanity check of JVM memory overhead to be within 
its min/max
 Key: FLINK-15597
 URL: https://issues.apache.org/jira/browse/FLINK-15597
 Project: Flink
  Issue Type: Improvement
  Components: Runtime / Configuration, Runtime / Task
Reporter: Andrey Zagrebin
Assignee: Xintong Song
 Fix For: 1.10.0


When the explicitly configured process and Flink memory sizes are verified with 
the JVM meta space and overhead, JVM overhead does not have to be the exact 
fraction.
It can be just within its min/max range, similar to how it is now for 
network/shuffle memory check after FLINK-15300.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [Discuss] Tuning FLIP-49 configuration default values.

[Andrey Zagrebin]

Hi all,

Stephan, Till and me had another offline discussion today. Here is the
outcome of our brainstorm.

*managed fraction 0.4*
just confirmed what we already discussed here.

*process.size = 1536Mb (1,5Gb)*
We agreed to have process.size in the default settings with the explanation
of flink.size alternative in the comment.
The suggestion is to increase it from 1024 to 1536mb. As you can see in the
earlier provided calculation spreadsheet,
it will result in bigger JVM Heap and managed memory (both ~0.5Gb) for all
new setups.
This should provide good enough experience for trying out Flink.

*JVM overhead min 196 -> 192Mb (128 + 64)*
small correction for better power 2 alignment of sizes

*meta space at least 96Mb?*
There is still a concern about JVM metaspace being just 64Mb.
We should confirm that it is not a problem by trying to test it also with
the SQL jobs, Blink planner.
Also, by running tpc-ds e2e Flink tests with this setting. Basically, where
more classes are generated/loaded.
We can look into this tomorrow.

*sanity check of JVM overhead*
When the explicitly configured process and flink memory sizes are verified
with the JVM meta space and overhead,
JVM overhead does not have to be the exact fraction.
It can be just within its min/max range, similar to how it is now for
network/shuffle memory check after FLINK-15300.

Best,Andrey

On Tue, Jan 14, 2020 at 4:30 PM Stephan Ewen  wrote:

> I like the idea of having a larger default "flink.size" in the config.yaml.
> Maybe we don't need to double it, but something like 1280m would be okay?
>
> On Tue, Jan 14, 2020 at 3:47 PM Andrey Zagrebin 
> wrote:
>
> > Hi all!
> >
> > Great that we have already tried out new FLIP-49 with the bigger jobs.
> >
> > I am also +1 for the JVM metaspace and overhead changes.
> >
> > Regarding 0.3 vs 0.4 for managed memory, +1 for having more managed
> memory
> > for Rocksdb limiting case.
> >
> > In general, this looks mostly to be about memory distribution between JVM
> > heap and managed off-heap.
> > Comparing to the previous default setup, the JVM heap dropped (especially
> > for standalone) mostly due to moving managed from heap to off-heap and
> then
> > also adding framework off-heap.
> > In general, this can be the most important consequence for beginners and
> > those who rely on the default configuration.
> > Especially the legacy default configuration in standalone with falling
> back
> > heap.size to flink.size but there it seems we cannot do too much now.
> >
> > I prepared a spreadsheet
> > <
> >
> https://docs.google.com/spreadsheets/d/1mJaMkMPfDJJ-w6nMXALYmTc4XxiV30P5U7DzgwLkSoE
> > >
> > to play with numbers for the mentioned in the report setups.
> >
> > One idea would be to set process size (or smaller flink size
> respectively)
> > to a bigger default number, like 2048.
> > In this case, the abs derived default JVM heap and managed memory are
> close
> > to the previous defaults, especially for managed fraction 0.3.
> > This should align the defaults with the previous standalone try-out
> > experience where the increased off-heap memory is not strictly controlled
> > by the environment anyways.
> > The consequence for container users who relied on and updated the default
> > configuration is that the containers will be requested with the double
> > size.
> >
> > Best,
> > Andrey
> >
> >
> > On Tue, Jan 14, 2020 at 11:20 AM Till Rohrmann 
> > wrote:
> >
> > > +1 for the JVM metaspace and overhead changes.
> > >
> > > On Tue, Jan 14, 2020 at 11:19 AM Till Rohrmann 
> > > wrote:
> > >
> > >> I guess one of the most important results of this experiment is to
> have
> > a
> > >> good tuning guide available for users who are past the initial try-out
> > >> phase because the default settings will be kind of a compromise. I
> > assume
> > >> that this is part of the outstanding FLIP-49 documentation task.
> > >>
> > >> If we limit RocksDB's memory consumption by default, then I believe
> that
> > >> 0.4 would give the better all-round experience as it leaves a bit more
> > >> memory for RocksDB. However, I'm a bit sceptical whether we should
> > optimize
> > >> the default settings for a configuration where the user still needs to
> > >> activate the strict memory limiting for RocksDB. In this case, I would
> > >> expect that the user could also adapt the managed memory fraction.
> > >>
> > >> Cheers,
> > >> Till
> > >>
> > >> O

Re: [DISCUSS] Some feedback after trying out the new FLIP-49 memory configurations

[Andrey Zagrebin]

Hi all,

Stephan, Till and me had another offline discussion today. Here is the
outcome of our brainstorm.

We agreed to have process.size in the default settings with the explanation
of flink.size alternative in the comment.
This way we keep -tm as a shortcut to process.size only and any
inconsistencies fail fast as if configured in yaml.
I will also follow-up on the thread "[Discuss] Tuning FLIP-49 configuration
default values" with a bit more details.

If no further objections, we can conclude this last point in this
discussion.

Best,
Andrey

On Tue, Jan 14, 2020 at 4:28 PM Stephan Ewen  wrote:

> I think that problem exists anyways and is independent of the "-tm" option.
>
> You can have a combination of `task.heap.size` and `managed.size` and
> `framework.heap.size` that conflicts with `flink.size`. In that case, we
> get an exception during the startup (incompatible memory configuration)?
> That is the price for having these "shortcut" options (`flink.size` and
> `process.size`). But it is a fair price, because the shortcuts are very
> valuable to most users.
>
> That is added with the "-tm" setting is that this is an easy way to get the
> shortcut setting added, even if it was not in the config. So where a config
> worked in standalone, it can now fail in a container setup when "-tm" is
> used.
> But that is expected, I guess, when you start manually tune different
> memory types and end up defining an inconsistent combination. And it never
> conflicts with the default setup, only with manually tuned regions.
>
> But this example shows why we need to have log output for the logic that
> validates and configures the memory settings, so that users understand what
> is happening.
>
> Best,
> Stephan
>
>
> On Tue, Jan 14, 2020 at 2:54 PM Till Rohrmann 
> wrote:
>
> > Clearing the `flink.size` option and setting `process.size` could indeed
> be
> > a solution. The thing I'm wondering is what would happen if the user has
> > configured `task.heap.size` and `managed.size` instead of `flink.size`?
> > Would we also ignore these settings? If not, then we risk to run into the
> > situation that TaskExecutorResourceUtils fails because the memory does
> not
> > add up. I guess the thing which bugs me a bit is the special casing which
> > could lead easily into inconsistent behaviour if we don't cover all
> cases.
> > The consequence of using slightly different concepts (`flink.size`,
> > `process.size`) in standalone vs. container/Yarn/Mesos mode in order to
> > keep the status quo is an increased maintenance overhead which we should
> be
> > aware of.
> >
> > Cheers,
> > Till
> >
> > On Tue, Jan 14, 2020 at 3:59 AM Xintong Song 
> > wrote:
> >
> > > True, even we have "process.size" rather than "flink.size" in the
> default
> > > config file, user can still have "flink.size" in their custom config
> > file.
> > > If we consider "-tm" as a shortcut for users to override the TM memory
> > > size, then it makes less sense to require users to remove "flink.size"
> > from
> > > their config file whenever then want to use "-tm".
> > >
> > > I'm convinced that ignoring "flink.size" might not be a bad idea.
> > > And I think we should also update the description of "-tm" (in
> > > "FlinkYarnSessionCli") to explicitly mention that it would overwrite
> > > "flink.size" and "process.size".
> > >
> > > Thank you~
> > >
> > > Xintong Song
> > >
> > >
> > >
> > > On Tue, Jan 14, 2020 at 2:24 AM Stephan Ewen  wrote:
> > >
> > > > Would be good to hear the thoughts of some more Yarn users, though.
> > > >
> > > > On Mon, Jan 13, 2020 at 7:23 PM Stephan Ewen 
> wrote:
> > > >
> > > > > I think we need an interpretation of "-tm" regardless of what is in
> > the
> > > > > default configuration, because we can always have a modified
> > > > configuration
> > > > > and then a user passes the "-tm" flag.
> > > > >
> > > > > I kind of like the first proposal: Interpret "-tm" as "override
> > memory
> > > > > size config and set the Yarn TM container size." It would mean
> > exactly
> > > > > ignoring "taskmanager.memory.flink.size" and using the "-tm" value
> > as "
> > > > > "taskmanager.memory.process.size&q

Re: [Discuss] Tuning FLIP-49 configuration default values.

[Andrey Zagrebin]

Hi all!

Great that we have already tried out new FLIP-49 with the bigger jobs.

I am also +1 for the JVM metaspace and overhead changes.

Regarding 0.3 vs 0.4 for managed memory, +1 for having more managed memory
for Rocksdb limiting case.

In general, this looks mostly to be about memory distribution between JVM
heap and managed off-heap.
Comparing to the previous default setup, the JVM heap dropped (especially
for standalone) mostly due to moving managed from heap to off-heap and then
also adding framework off-heap.
In general, this can be the most important consequence for beginners and
those who rely on the default configuration.
Especially the legacy default configuration in standalone with falling back
heap.size to flink.size but there it seems we cannot do too much now.

I prepared a spreadsheet

to play with numbers for the mentioned in the report setups.

One idea would be to set process size (or smaller flink size respectively)
to a bigger default number, like 2048.
In this case, the abs derived default JVM heap and managed memory are close
to the previous defaults, especially for managed fraction 0.3.
This should align the defaults with the previous standalone try-out
experience where the increased off-heap memory is not strictly controlled
by the environment anyways.
The consequence for container users who relied on and updated the default
configuration is that the containers will be requested with the double size.

Best,
Andrey


On Tue, Jan 14, 2020 at 11:20 AM Till Rohrmann  wrote:

> +1 for the JVM metaspace and overhead changes.
>
> On Tue, Jan 14, 2020 at 11:19 AM Till Rohrmann 
> wrote:
>
>> I guess one of the most important results of this experiment is to have a
>> good tuning guide available for users who are past the initial try-out
>> phase because the default settings will be kind of a compromise. I assume
>> that this is part of the outstanding FLIP-49 documentation task.
>>
>> If we limit RocksDB's memory consumption by default, then I believe that
>> 0.4 would give the better all-round experience as it leaves a bit more
>> memory for RocksDB. However, I'm a bit sceptical whether we should optimize
>> the default settings for a configuration where the user still needs to
>> activate the strict memory limiting for RocksDB. In this case, I would
>> expect that the user could also adapt the managed memory fraction.
>>
>> Cheers,
>> Till
>>
>> On Tue, Jan 14, 2020 at 3:39 AM Xintong Song 
>> wrote:
>>
>>> Thanks for the feedback, Stephan and Kurt.
>>>
>>> @Stephan
>>>
>>> Regarding managed memory fraction,
>>> - It makes sense to keep the default value 0.4, if we assume rocksdb
>>> memory is limited by default.
>>> - AFAIK, currently rocksdb by default does not limit its memory usage.
>>> And I'm positive to change it.
>>> - Personally, I don't like the idea that we the out-of-box experience
>>> (for which we set the default fraction) relies on that users will manually
>>> turn another switch on.
>>>
>>> Regarding framework heap memory,
>>> - The major reason we set it by default is, as you mentioned, that to
>>> have a safe net of minimal JVM heap size.
>>> - Also, considering the in progress FLIP-56 (dynamic slot allocation),
>>> we want to reserve some heap memory that will not go into the slot
>>> profiles. That's why we decide the default value according to the heap
>>> memory usage of an empty task executor.
>>>
>>> @Kurt
>>> Regarding metaspace,
>>> - This config option ("taskmanager.memory.jvm-metaspace") only takes
>>> effect on TMs. Currently we do not set metaspace size for JM.
>>> - If we have the same metaspace problem on TMs, then yes, changing it
>>> from 128M to 64M will make it worse. However, IMO 10T tpc-ds benchmark
>>> should not be considered as out-of-box experience and it makes sense to
>>> tune the configurations for it. I think the smaller metaspace size would be
>>> a better choice for the first trying-out, where a job should not be too
>>> complicated, the TM size could be relative small (e.g. 1g).
>>>
>>> Thank you~
>>>
>>> Xintong Song
>>>
>>>
>>>
>>> On Tue, Jan 14, 2020 at 9:38 AM Kurt Young  wrote:
>>>
 HI Xingtong,

 IIRC during our tpc-ds 10T benchmark, we have suffered by JM's
 metaspace size and full gc which
 caused by lots of classloadings of source input split. Could you check
 whether changing the default
 value from 128MB to 64MB will make it worse?

 Correct me if I misunderstood anything, also cc @Jingsong

 Best,
 Kurt


 On Tue, Jan 14, 2020 at 3:44 AM Stephan Ewen  wrote:

> Hi all!
>
> Thanks a lot, Xintong, for this thorough analysis. Based on your
> analysis,
> here are some thoughts:
>
> +1 to change default JVM metaspace size from 128MB to 64MB
> +1 to change default JVM overhead min size from 128MB to 196MB
>
> Concerning the managed memory