[jira] [Created] (FLINK-18283) [Javdoc] Update outdated Javadoc for clear method of ProcessWindowFunction
Abhijit Shandilya created FLINK-18283:
-
Summary: [Javdoc] Update outdated Javadoc for clear method of
ProcessWindowFunction
Key: FLINK-18283
URL: https://issues.apache.org/jira/browse/FLINK-18283
Project: Flink
Issue Type: Improvement
Components: API / Core
Reporter: Abhijit Shandilya
+*Summary:*+
Javadoc for ProcessWindowFunction is has incorrect (outdated) information.
+*Description:*+
The current javadoc for
[ProcessWindowFunction|https://ci.apache.org/projects/flink/flink-docs-release-1.10/api/java/org/apache/flink/streaming/api/functions/windowing/ProcessWindowFunction.html]
[clear|https://ci.apache.org/projects/flink/flink-docs-release-1.10/api/java/org/apache/flink/streaming/api/functions/windowing/ProcessWindowFunction.html#clear-org.apache.flink.streaming.api.functions.windowing.ProcessWindowFunction.Context-]
method says
{quote}Deletes any state in the Context when the Window is purged.{quote}
But, this is not true anymore. This behavior was changed in FLINK-4994.
Before FLINK-4994, when Trigger.PURGE was called, it would invoke
ProcessWindowFunction's clear( ) method to clean up all keyed per-window state.
But after FLINK-4994, ProcessWindowFunction's clear is called only when the
window expires, which is to say the window reaches its [window.maxTimestamp()
|https://ci.apache.org/projects/flink/flink-docs-release-1.10/api/java/org/apache/flink/streaming/api/windowing/windows/Window.html#maxTimestamp--]
This change in behavior comes from
[this|https://github.com/apache/flink/commit/0b331a421267a541d91e94f2713534704ed32bed#diff-408a499e1a35840c52e29b7ccab866b1R461-R464]
code change (repeated in a few other places) in FLINK-4994.
+*Proposed change:*+
I think we should change the description to say
{quote}Deletes any state in the Context when the Window expires (reaches its
[maxTimestamp|https://ci.apache.org/projects/flink/flink-docs-release-1.10/api/java/org/apache/flink/streaming/api/windowing/windows/Window.html#maxTimestamp--]).{quote}
+*Why this can be helpful:*+
The current documentation could be misleading. Developers may expect to get the
_keyed per-window state_ cleared when a PURGE happens. But this will not
happen. I myself had to go through flink source code to identify the
disconnect. Updating the javadoc can help future users to avoid such confusions.
+*Links to lines of code that will need updating:*+ #
[ProcessWindowFunction.scala|https://github.com/apache/flink/blob/8674b69964eae50cad024f2c5caf92a71bf21a09/flink-streaming-scala/src/main/scala/org/apache/flink/streaming/api/scala/function/ProcessWindowFunction.scala#L54]
#
[ProcessAllWindowFunction.scala|https://github.com/apache/flink/blob/8674b69964eae50cad024f2c5caf92a71bf21a09/flink-streaming-scala/src/main/scala/org/apache/flink/streaming/api/scala/function/ProcessAllWindowFunction.scala#L51]
#
[ProcessWindowFunction.java|https://github.com/apache/flink/blob/8674b69964eae50cad024f2c5caf92a71bf21a09/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/windowing/ProcessWindowFunction.java#L55]
#
[ProcessAllWindowFunction.java|https://github.com/apache/flink/blob/8674b69964eae50cad024f2c5caf92a71bf21a09/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/windowing/ProcessAllWindowFunction.java#L53]
#
[InternalWindowFunction.java|https://github.com/apache/flink/blob/8674b69964eae50cad024f2c5caf92a71bf21a09/flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/windowing/functions/InternalWindowFunction.java#L47]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-18282) retranslate the documentation home page
David Anderson created FLINK-18282: -- Summary: retranslate the documentation home page Key: FLINK-18282 URL: https://issues.apache.org/jira/browse/FLINK-18282 Project: Flink Issue Type: Improvement Components: chinese-translation, Documentation Reporter: David Anderson Fix For: 1.11.0 FLINK-17981 was a complete rewrite of the documentation home page. The chinese translation should be updated along the same lines. docs/index.zh.md -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-18281) Add WindowStagger into all Tumbling and Sliding Windows
Teng Hu created FLINK-18281: --- Summary: Add WindowStagger into all Tumbling and Sliding Windows Key: FLINK-18281 URL: https://issues.apache.org/jira/browse/FLINK-18281 Project: Flink Issue Type: New Feature Components: API / DataStream Reporter: Teng Hu Adding the window staggering functionality into *TumblingEventTimeWindows*, *SlidingProcessingTimeWindows* and *SlidingEventTimeWindows*. This is a follow-up issue of [FLINK-12855|https://issues.apache.org/jira/browse/FLINK-12855] -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: [DISCUSS] Add Japanese translation of the flink.apache.org website
Thank you all for your comments! It seems that we don't have anybody in our community that speaks Japanese, which will make the review for us difficult anyways. In particular initially, it is probably important to make sure we are using the right words for certain concepts in Flink, similar to Chinese [1]. Secondly, it seems that this thread has uncovered a lot of issues around our current translation process. It sounds like maintaining the docs in two languages is really painful, and we need to revisit our tooling. For the tooling, the following ideas were brought up: a) Xintong's idea to have some custom tooling tracking changes based on a commit sha b) Jark's list of tools (out of which Crowdin seems interesting, in particular because it has been implemented by another Apache project already) I want to throw in c) We look for a new documentation generation tool? Jekyll seems very slow, and Jark raised concerns regarding its compatibility with Crowdin. As next steps, I propose a) we need to find somebody in the community who wants to invest some time to researching translation tools (or a new docs generation tool). *Who wants to look into that?* b) a new discussion thread to follow up. For the proposed Japanese translation, I will summarize this discussion in the PR, and put the PR on hold. Best, Robert [1] https://cwiki.apache.org/confluence/display/FLINK/Flink+Translation+Specifications On Thu, Jun 11, 2020 at 4:07 AM Congxian Qiu wrote: > Share some experience for the current translation(and review) process from > my side. for documentation review, we can pull the patch locally and set up > a local server[1][2], after that, we can reference the rendered > documentation in `localhost:4000` > > [1] > > https://flink.apache.org/contributing/contribute-documentation.html#update-or-extend-the-documentation > [2] > > https://flink.apache.org/contributing/improve-website.html#update-or-extend-the-documentation > > Best, > Congxian > > > Yun Tang 于2020年6月9日周二 下午10:34写道： > > > Hi > > > > I think supporting website with another language should be okay. However, > > I'm afraid Flink community lacks of resources > > to maintain technical documentations of another languages currently. > > > > From my experiences of translating documentation to Chinese since the > past > > year, I found > > Chinese documentation is easy to be out-of-date and containing broken > > links. Some developers > > might only update English version and forget to update the related > Chinese > > version. > > > > Since Jark talked about re-investigating tools for translating, I just > > want to share some painful experiences > > when translating documentations. > > First of all, I think code review should be totally different from doc > > review. Github lacks such kind of power > > to make the markdown review more readable, we often need to read a long > > line without wrapping and give advice for > > just some of the characters. > > Secondly, user who wants to contributes cannot leverage any given > powerful > > tool. > > > > I think Crodwin might be a good choice. > > > > 1. Many popular projects already use this: node.js [1], gitlab [2], > > Minecraft [3]. > > 2. This tool is free for open-source project [4] > > > > [1] https://crowdin.com/project/nodejs-website > > [2] https://crowdin.com/project/gitlab-ee > > [3] https://crowdin.com/project/minecraft > > [4] https://crowdin.com/page/open-source-project-setup-request > > > > Best > > Yun Tang > > > > > > From: Marta Paes Moreira > > Sent: Tuesday, June 9, 2020 22:13 > > To: dev > > Subject: Re: [DISCUSS] Add Japanese translation of the flink.apache.org > > website > > > > I had a second look at the PR and maybe it'd be important to clarify the > > scope. > > > > It seems to only target the website and, if that's the case, then > > synchronization might be less of an issue as the core of the website is > > pretty static. I'm not sure how useful it is to have a translated website > > without any other kind of support (e.g. mailing list, documentation) in > the > > same language, but the contributor might be able to shed some light on > > this. > > > > Just wanted to amend my original reply as it was pretty blindsided by the > > previous comments. It'd be worth understanding the end goal of the > > contribution first. > > > > Thanks, > > > > Marta > > > > On Tue, Jun 9, 2020 at 2:22 PM Congxian Qiu > > wrote: > > > > > Hi > > > > > > Thanks for bringing this up, Robert. > > > > > > I agree we may need to investigate better translation/synchronization > > tools > > > again. > > > > > > I'll share some experience when translating the documentation or > > reviewing > > > the translation. > > > 1. Translate documentation using the current procedure needs a lot of > > > resources to keep the translated version fresh, I'm glad that we have > so > > > many contributors who want to translate the documentation to Chinese. > > > 2. The
[jira] [Created] (FLINK-18280) Kotlin adapters for Flink types?
Marshall Pierce created FLINK-18280:
---
Summary: Kotlin adapters for Flink types?
Key: FLINK-18280
URL: https://issues.apache.org/jira/browse/FLINK-18280
Project: Flink
Issue Type: Wish
Affects Versions: 1.10.1
Reporter: Marshall Pierce
Currently, using a Kotlin lambda for, say, a {{KeySelector}} doesn't work – it
needs to be an {{object}} expression for the runtime type detection to work. At
my day job we have started building up a handful of wrappers, like this one for
{{KeySelector}}:
{code:kotlin}
inline fun keySelector(crossinline block: (T) -> K): KeySelector {
return object : KeySelector {
override fun getKey(value: T): K {
return block(value)
}
}
}
{code}
Usage looks like: {{keySelector { it.fooId }}}. Surely not the only way to
solve that problem, but it's been working smoothly for us so far.
Is there any interested in shipping these sorts of extensions as part of the
Flink project so users don't need to write them? It could be a wholly separate
artifact (or rather multiple artifacts, as there would probably be one for
flink core, one for flink streaming, etc).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-18279) Simplify table overview page
Seth Wiesman created FLINK-18279: Summary: Simplify table overview page Key: FLINK-18279 URL: https://issues.apache.org/jira/browse/FLINK-18279 Project: Flink Issue Type: Improvement Reporter: Seth Wiesman Assignee: Seth Wiesman Fix For: 1.11.0, 1.12.0 The table overview page contains an overwhelming amount of information. We should simplify the page so users quickly know: * What dependencies they need to add in their user code * Which planner to use -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-18278) Translate new documenation homepage
Seth Wiesman created FLINK-18278: Summary: Translate new documenation homepage Key: FLINK-18278 URL: https://issues.apache.org/jira/browse/FLINK-18278 Project: Flink Issue Type: Improvement Components: chinese-translation Reporter: Seth Wiesman Sync changes with FLINK-17981 -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: [DISCUSS] Update our EditorConfig file
It seems that nobody cares about the file, as it has been basically never touched after its creation in late 2015. I would be fine adding the changes you are proposing to the codebase. On Thu, Jun 11, 2020 at 4:47 AM tison wrote: > > is anyone actually using our .editorconfig file? > > I think IDEA already takes this file into consideration. So far it works > well for me. > > Best, > tison. > > > Jingsong Li 于2020年6月11日周四 上午10:26写道： > > > +1 looks more friendly to Flink newbies. > > > > Best, > > Jingsong Lee > > > > On Wed, Jun 10, 2020 at 8:38 PM Aljoscha Krettek > > wrote: > > > > > Hi, > > > > > > is anyone actually using our .editorconfig file? IntelliJ has a plugin > > > for this that is actually quite powerful. > > > > > > I managed to write a .editorconfig file that I quite like: > > > https://github.com/aljoscha/flink/commits/new-editorconfig. For me to > > > use that, we would either need to update our Flink file to what I did > > > there or remove the "root = true" part from the file to allow me to > > > place my custom .editorconfig in the directory above. > > > > > > It's probably a lost cause to find consensus on what settings we should > > > have in that file but it could be helpful if we all used the same > > > settings. For what it's worth, this will format code in such a way that > > > it pleases our (very lenient) checkstyle rules. > > > > > > What do you think? > > > > > > Best, > > > Aljoscha > > > > > > > > > -- > > Best, Jingsong Lee > > >
Re: [DISCUSS] Semantics of our JIRA fields
I agree with you Till -- changing the definition of the priorities should be a separate discussion. Piotrek, do you agree with my "affects version" explanation? I would like to bring this discussion to a conclusion. On Tue, May 26, 2020 at 4:51 PM Till Rohrmann wrote: > If we change the meaning of the priority levels, then I would suggest to > have a dedicated discussion for it. This would also be more visible than > compared to being hidden in some lengthy discussion thread. I think the > proposed definitions of priority levels differ slightly from how the > community worked in the past. > > Cheers, > Till > > On Tue, May 26, 2020 at 4:30 PM Robert Metzger > wrote: > > > Hi, > > > > 1. I'm okay with updating the definition of the priorities for the reason > > you've mentioned. > > > > 2. "Affects version" > > > > The reason why like to mark affects version on unreleased versions is to > > clearly indicate which branch is affected by a bug. Given the current > Flink > > release status, if there's a bug only in "release-1.11", but not in > > "master", there is no way of figuring that out, if we only allow released > > versions for "affects version" (In my proposal, you would set "affects > > version" to '1.11.0', '1.12.0' to indicate that). > > > > What we could do is introduce "1.12-SNAPSHOT" as version to mark issues > on > > unreleased versions. (But then people might accidentally set the "fix > > version" to a "-SNAPSHOT" version.) > > > > I'm still in favor of my proposal. I have never heard a report from a > > confused user about our Jira fields (I guess they usually check bugs for > > released versions only) > > > > > > On Tue, May 26, 2020 at 12:39 PM Piotr Nowojski > > wrote: > > > > > Hi, > > > > > > Sorry for a bit late response. I have two concerns: > > > > > > 1. Priority > > > > > > I would propose to stretch priorities that we are using to > differentiate > > > between things that must be fixed for given release: > > > > > > BLOCKER - drop anything you are doing, this issue must be fixed right > now > > > CRITICAL - release can not happen without fixing it, but can be fixed a > > > bit later (for example without context switching and dropping whatever > > I’m > > > doing right now) > > > MAJOR - default, nice to have > > > Anything below - meh > > > > > > We were already using this semantic for tracking test instabilities > > during > > > the 1.11 release cycle. Good examples: > > > > > > BLOCKER - master branch not compiling, very frequent test failures (for > > > example almost every build affected), … > > > CRITICAL - performance regression/bug that we introduced in some > feature, > > > but which is not affecting other developers as much > > > MAJOR - freshly discovered test instability with unknown > impact/frequency > > > (could be happening once a year), > > > > > > 2. Affects version > > > > > > If bug is only on the master branch, does it affect an unreleased > > version? > > > > > > So far I was assuming that it doesn’t - unreleased bugs would have > empty > > > “affects version” field. My reasoning was that this field should be > used > > > for Flink users, to check which RELEASED Flink versions are affected by > > > some bug, that user is searching for. Otherwise it might be a bit > > confusing > > > if there are lots of bugs with both affects version and fix version set > > to > > > the same value. > > > > > > Piotrek > > > > > > > On 25 May 2020, at 16:40, Robert Metzger > wrote: > > > > > > > > Hi all, > > > > thanks a lot for the feedback. The majority of responses are very > > > positive > > > > to my proposal. > > > > > > > > I have put my proposal into our wiki: > > > > > > > > > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=154995514 > > > > > > > > Regarding the comments so far: > > > > @Jark: I clarified this in the wiki. > > > > > > > > @Israel: I have not considered build changing all 3000 resolved > tickets > > > to > > > > closed yet, but after consideration I don't think it is necessary. If > > > > others in the community would like to change them, please speak up in > > > this > > > > thread :) > > > > > > > > @Flavio: I agree that we can not ask new or infrequent users to fully > > > > adhere to these definitions. I added a note in the Wiki. > > > > Using the resolved state for indicating "PR available" is problematic > > > > because there are plenty of cases where PRs are stale (and this > ticket > > > > would then appear as a "resolved"). The Apache tools are adding a > link > > to > > > > the PR, and some contributors are setting the ticket to "In > Progress". > > I > > > > don't see a problem that we need to solve here. > > > > > > > > @Yun: Thank you for your comment. I added an example clarifying how I > > > would > > > > handle such a case. It is slightly different from your proposal: You > > > > suggested using x.y.0 versions, I used "the next supported, > unreleased > > > > version", because that's how we've done it so far
[jira] [Created] (FLINK-18277) Elasticsearch6DynamicSink#asSummaryString() return identifier typo
Qishang Zhong created FLINK-18277:
-
Summary: Elasticsearch6DynamicSink#asSummaryString() return
identifier typo
Key: FLINK-18277
URL: https://issues.apache.org/jira/browse/FLINK-18277
Project: Flink
Issue Type: Bug
Components: Connectors / ElasticSearch
Reporter: Qishang Zhong
Fix For: 1.11.0
identifier Spelling mistakes
`org.apache.flink.streaming.connectors.elasticsearch.table.Elasticsearch6DynamicSink#asSummaryString`
{code:java}
@Override
public String asSummaryString() {
return "Elasticsearch7";
}
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Re: request create flip permission for flink es bounded source/lookup source connector
Hi, I gave you access to the Wiki! On Fri, Jun 12, 2020 at 11:50 AM Jacky Lau wrote: > Hi Jack: >Thank you so much. My wiki name is jackylau > > > Jark Wu-2 wrote > > Hi Jacky, > > > > What's your username in wiki? So that I can give the permission to you. > > > > Best, > > Jark > > > > On Fri, 12 Jun 2020 at 11:38, Jacky Lau > > > liuyongvs@ > > > wrote: > > > >> hi all: > >>After this simple discussion here > >> > >> > http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Discussion-flink-elasticsearch-connector-supports-td42082.html#a42106 > >> , > >>and i should create i flip127 to track this. But i don't have create > >> flip permision. > >> > >> > >> > >> -- > >> Sent from: > >> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/ > >> > > > Jark Wu-2 wrote > > Hi Jacky, > > > > What's your username in wiki? So that I can give the permission to you. > > > > Best, > > Jark > > > > On Fri, 12 Jun 2020 at 11:38, Jacky Lau > > > liuyongvs@ > > > wrote: > > > >> hi all: > >>After this simple discussion here > >> > >> > http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Discussion-flink-elasticsearch-connector-supports-td42082.html#a42106 > >> , > >>and i should create i flip127 to track this. But i don't have create > >> flip permision. > >> > >> > >> > >> -- > >> Sent from: > >> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/ > >> > > > > > > -- > Sent from: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/ >
Re: request create flip permission for flink es bounded source/lookup source connector
Hi Jack: Thank you so much. My wiki name is jackylau Jark Wu-2 wrote > Hi Jacky, > > What's your username in wiki? So that I can give the permission to you. > > Best, > Jark > > On Fri, 12 Jun 2020 at 11:38, Jacky Lau > liuyongvs@ > wrote: > >> hi all: >>After this simple discussion here >> >> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Discussion-flink-elasticsearch-connector-supports-td42082.html#a42106 >> , >>and i should create i flip127 to track this. But i don't have create >> flip permision. >> >> >> >> -- >> Sent from: >> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/ >> Jark Wu-2 wrote > Hi Jacky, > > What's your username in wiki? So that I can give the permission to you. > > Best, > Jark > > On Fri, 12 Jun 2020 at 11:38, Jacky Lau > liuyongvs@ > wrote: > >> hi all: >>After this simple discussion here >> >> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Discussion-flink-elasticsearch-connector-supports-td42082.html#a42106 >> , >>and i should create i flip127 to track this. But i don't have create >> flip permision. >> >> >> >> -- >> Sent from: >> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/ >> -- Sent from: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/
Re: Flink on Kubes -- issues
Usually that means the remote task manager, where the slot locates, is lost.
You will need to look into the log of that task manager to find out what's
wrong with it.
Thank you~
Xintong Song
On Fri, Jun 12, 2020 at 4:13 PM Ramya Ramamurthy wrote:
> Yes ... the image was on Heap Committed metrics.
> And i have not yet faced this issue now, post changing the memory.
>
> I seem to get one more frequent error:
> org.apache.flink.util.FlinkException:
> The assigned slot d9d4db5cc747bcbd374888d97e81945b_0 was removed.
>
> When are we likely to get this ??
>
> Thanks,
>
> ~Ramya.
>
>
> On Fri, Jun 12, 2020 at 12:03 PM Xintong Song
> wrote:
>
> > BTW, the image you previously attached cannot be displayed. So I assume
> you
> > are talking about the "Heap Committed" displayed on Flink's webui?
> >
> > Thank you~
> >
> > Xintong Song
> >
> >
> >
> > On Fri, Jun 12, 2020 at 2:30 PM Xintong Song
> > wrote:
> >
> > > Do you still run into the "java.lang.OutOfMemoryError: Java heap
> space"?
> > >
> > > If not, then you don't really need to worry about the committed memory.
> > >
> > > It is the maximum that really matters. The committed memory should
> > > increase automatically when it's needed.
> > >
> > > Thank you~
> > >
> > > Xintong Song
> > >
> > >
> > >
> > > On Fri, Jun 12, 2020 at 2:24 PM Ramya Ramamurthy
> > > wrote:
> > >
> > >> Hi Xintong,
> > >>
> > >> Thanks for the quick response.
> > >>
> > >> I have kept my task manager memory to be 1.5GB. But still seeing the
> > Heap
> > >> committed metric to be around 54MB or so. Why does this happen ?
> Should
> > I
> > >> configure any memory fraction configurations here ?
> > >>
> > >> Thanks.
> > >>
> > >> On Fri, Jun 12, 2020 at 10:58 AM Xintong Song
> > >> wrote:
> > >>
> > >> > Hi Ramya,
> > >> >
> > >> > Increasing the memory of your pod will not give you more JVM heap
> > space.
> > >> > You will need to configure Flink so it launches the JVM process with
> > >> more
> > >> > memory.
> > >> >
> > >> > In Flink 1.7, this could be achieved by configuring
> > >> 'jobmanager.heap.size'
> > >> > & 'taskmanager.heap.size' in your 'flink-conf.yaml'. Both of them
> are
> > by
> > >> > default 1024m.
> > >> >
> > >> > Please also note that, you should not configure these two options
> two
> > as
> > >> > large as your Kubernetes pod. Because Flink may also have some
> > off-heap
> > >> > memory overhead, so the total memory consumed by the Flink processes
> > >> might
> > >> > be larger than configured. This may cause your pods getting killed
> by
> > >> > Kubernetes due to memory exceeding.
> > >> >
> > >> > According to our experience, leaving around 20~25% of your pod
> memory
> > >> for
> > >> > such overhead might be a good practice. In your case, that means
> > >> > configuring 'taskmanager.heap.size' to 4GB. If RocksDB is used in
> your
> > >> > workload, you may need to further increase the off-heap memory size.
> > >> >
> > >> > Thank you~
> > >> >
> > >> > Xintong Song
> > >> >
> > >> >
> > >> >
> > >> > On Fri, Jun 12, 2020 at 1:11 PM Ramya Ramamurthy >
> > >> > wrote:
> > >> >
> > >> > > Thanks Till.
> > >> > > Actually, i have around 5GB pods for each TM, and each pod with
> only
> > >> one
> > >> > > slot.
> > >> > > But the metrics i have pulled is as below, which is slightly
> > >> confusing.
> > >> > > It says only ~50MB of Heap is committed for the tasks. Would you
> be
> > >> able
> > >> > > to point me to the right configuration to be set.
> > >> > >
> > >> > > Thanks
> > >> > > ~Ramya.
> > >> > >
> > >> > > [image: image.png]
> > >> > >
> > >> > > On Tue, Jun 9, 2020 at 3:12 PM Till Rohrmann <
> trohrm...@apache.org>
> > >> > wrote:
> > >> > >
> > >> > >> Hi Ramya,
> > >> > >>
> > >> > >> it looks as if you should give your Flink pods and also the Flink
> > >> > process
> > >> > >> a
> > >> > >> bit more memory as the process fails with an out of memory error.
> > You
> > >> > >> could
> > >> > >> also try Flink's latest version which comes with native
> Kubernetes
> > >> > >> support.
> > >> > >>
> > >> > >> Cheers,
> > >> > >> Till
> > >> > >>
> > >> > >> On Tue, Jun 9, 2020 at 8:45 AM Ramya Ramamurthy <
> hair...@gmail.com
> > >
> > >> > >> wrote:
> > >> > >>
> > >> > >> > Hi,
> > >> > >> >
> > >> > >> > My flink jobs are constantly going down beyond an hour with the
> > >> below
> > >> > >> > exception.
> > >> > >> > This is Flink 1.7 on kubes, with checkpoints to Google storage.
> > >> > >> >
> > >> > >> > AsynchronousException{java.lang.Exception: Could not
> materialize
> > >> > >> > checkpoint 21 for operator Source: Kafka011TableSource(sid,
> > >> _zpsbd3,
> > >> > >> > _zpsbd4, _zpsbd6, _zpsbd7, _zpsbd9, lvl_1, isBot, botcode,
> > ssresp,
> > >> > >> > reason, ts) -> from: (sid, _zpsbd3, _zpsbd6, ts) ->
> > >> > >> > Timestamps/Watermarks -> where: (<>(sid, _UTF-16LE'7759')),
> > select:
> > >> > >> > (sid, _zpsbd3, _zpsbd6, ts) -> time attribute: (ts) (5/6).}
> > >> > >> > at
> > >> > >> >
> > >> > >>
> > >> >
[jira] [Created] (FLINK-18276) NullPointerException when closing KafkaConsumer
Nico Kruber created FLINK-18276:
---
Summary: NullPointerException when closing KafkaConsumer
Key: FLINK-18276
URL: https://issues.apache.org/jira/browse/FLINK-18276
Project: Flink
Issue Type: Bug
Components: Connectors / Kafka
Affects Versions: 1.10.1, 1.9.3, 1.8.3, 1.11.0
Reporter: Nico Kruber
{code}
WARN org.apache.flink.streaming.connectors.kafka.internal.KafkaFetcher -
Error while closing Kafka consumer
java.lang.NullPointerException
at
org.apache.flink.streaming.connectors.kafka.internal.KafkaConsumerThread.run(KafkaConsumerThread.java:282)
{code}
{{KafkaConsumerThread#reassignPartitions}} is temporarily setting {{consumer}}
to {{null}} and if there is an exception (in this case, it was a timeout), the
{{finally}} block in {{KafkaConsumerThread.run}} would fail with an NPE. Even
more so, {{KafkaConsumerThread#reassignPartitions}} put the original consumer
into {{consumerTmp}} which is not closed now and may leak underlying (Kafka)
resources.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-18275) Register the DataStream as table how to write multiple fields
robert created FLINK-18275:
--
Summary: Register the DataStream as table how to write multiple
fields
Key: FLINK-18275
URL: https://issues.apache.org/jira/browse/FLINK-18275
Project: Flink
Issue Type: Improvement
Reporter: robert
Register the DataStream as table how to write multiple fields
Official wording：
{{tableEnv.registerDataStream("myTable2", stream, 'myLong, 'myString)}}
{{}}
{{This is a known field, if unknown, how to write}}
{{}}
{{}}
{{}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Re: Flink on Kubes -- issues
Yes ... the image was on Heap Committed metrics.
And i have not yet faced this issue now, post changing the memory.
I seem to get one more frequent error: org.apache.flink.util.FlinkException:
The assigned slot d9d4db5cc747bcbd374888d97e81945b_0 was removed.
When are we likely to get this ??
Thanks,
~Ramya.
On Fri, Jun 12, 2020 at 12:03 PM Xintong Song wrote:
> BTW, the image you previously attached cannot be displayed. So I assume you
> are talking about the "Heap Committed" displayed on Flink's webui?
>
> Thank you~
>
> Xintong Song
>
>
>
> On Fri, Jun 12, 2020 at 2:30 PM Xintong Song
> wrote:
>
> > Do you still run into the "java.lang.OutOfMemoryError: Java heap space"?
> >
> > If not, then you don't really need to worry about the committed memory.
> >
> > It is the maximum that really matters. The committed memory should
> > increase automatically when it's needed.
> >
> > Thank you~
> >
> > Xintong Song
> >
> >
> >
> > On Fri, Jun 12, 2020 at 2:24 PM Ramya Ramamurthy
> > wrote:
> >
> >> Hi Xintong,
> >>
> >> Thanks for the quick response.
> >>
> >> I have kept my task manager memory to be 1.5GB. But still seeing the
> Heap
> >> committed metric to be around 54MB or so. Why does this happen ? Should
> I
> >> configure any memory fraction configurations here ?
> >>
> >> Thanks.
> >>
> >> On Fri, Jun 12, 2020 at 10:58 AM Xintong Song
> >> wrote:
> >>
> >> > Hi Ramya,
> >> >
> >> > Increasing the memory of your pod will not give you more JVM heap
> space.
> >> > You will need to configure Flink so it launches the JVM process with
> >> more
> >> > memory.
> >> >
> >> > In Flink 1.7, this could be achieved by configuring
> >> 'jobmanager.heap.size'
> >> > & 'taskmanager.heap.size' in your 'flink-conf.yaml'. Both of them are
> by
> >> > default 1024m.
> >> >
> >> > Please also note that, you should not configure these two options two
> as
> >> > large as your Kubernetes pod. Because Flink may also have some
> off-heap
> >> > memory overhead, so the total memory consumed by the Flink processes
> >> might
> >> > be larger than configured. This may cause your pods getting killed by
> >> > Kubernetes due to memory exceeding.
> >> >
> >> > According to our experience, leaving around 20~25% of your pod memory
> >> for
> >> > such overhead might be a good practice. In your case, that means
> >> > configuring 'taskmanager.heap.size' to 4GB. If RocksDB is used in your
> >> > workload, you may need to further increase the off-heap memory size.
> >> >
> >> > Thank you~
> >> >
> >> > Xintong Song
> >> >
> >> >
> >> >
> >> > On Fri, Jun 12, 2020 at 1:11 PM Ramya Ramamurthy
> >> > wrote:
> >> >
> >> > > Thanks Till.
> >> > > Actually, i have around 5GB pods for each TM, and each pod with only
> >> one
> >> > > slot.
> >> > > But the metrics i have pulled is as below, which is slightly
> >> confusing.
> >> > > It says only ~50MB of Heap is committed for the tasks. Would you be
> >> able
> >> > > to point me to the right configuration to be set.
> >> > >
> >> > > Thanks
> >> > > ~Ramya.
> >> > >
> >> > > [image: image.png]
> >> > >
> >> > > On Tue, Jun 9, 2020 at 3:12 PM Till Rohrmann
> >> > wrote:
> >> > >
> >> > >> Hi Ramya,
> >> > >>
> >> > >> it looks as if you should give your Flink pods and also the Flink
> >> > process
> >> > >> a
> >> > >> bit more memory as the process fails with an out of memory error.
> You
> >> > >> could
> >> > >> also try Flink's latest version which comes with native Kubernetes
> >> > >> support.
> >> > >>
> >> > >> Cheers,
> >> > >> Till
> >> > >>
> >> > >> On Tue, Jun 9, 2020 at 8:45 AM Ramya Ramamurthy >
> >> > >> wrote:
> >> > >>
> >> > >> > Hi,
> >> > >> >
> >> > >> > My flink jobs are constantly going down beyond an hour with the
> >> below
> >> > >> > exception.
> >> > >> > This is Flink 1.7 on kubes, with checkpoints to Google storage.
> >> > >> >
> >> > >> > AsynchronousException{java.lang.Exception: Could not materialize
> >> > >> > checkpoint 21 for operator Source: Kafka011TableSource(sid,
> >> _zpsbd3,
> >> > >> > _zpsbd4, _zpsbd6, _zpsbd7, _zpsbd9, lvl_1, isBot, botcode,
> ssresp,
> >> > >> > reason, ts) -> from: (sid, _zpsbd3, _zpsbd6, ts) ->
> >> > >> > Timestamps/Watermarks -> where: (<>(sid, _UTF-16LE'7759')),
> select:
> >> > >> > (sid, _zpsbd3, _zpsbd6, ts) -> time attribute: (ts) (5/6).}
> >> > >> > at
> >> > >> >
> >> > >>
> >> >
> >>
> org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointExceptionHandler.tryHandleCheckpointException(StreamTask.java:1153)
> >> > >> > at
> >> > >> >
> >> > >>
> >> >
> >>
> org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointRunnable.handleExecutionException(StreamTask.java:947)
> >> > >> > at
> >> > >> >
> >> > >>
> >> >
> >>
> org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointRunnable.run(StreamTask.java:884)
> >> > >> > at
> >> > >> >
> >> >
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> >> > >> > at
>
[jira] [Created] (FLINK-18274) document example error
appleyuchi created FLINK-18274:
--
Summary: document example error
Key: FLINK-18274
URL: https://issues.apache.org/jira/browse/FLINK-18274
Project: Flink
Issue Type: Improvement
Reporter: appleyuchi
Attachments: 文档问题.png
could you change it?
[https://ci.apache.org/projects/flink/flink-docs-release-1.0/apis/batch/dataset_transformations.html#flatmap]
{{texLines}}->{{textLines}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-18273) Test and document remote cluster submission in SQL Client
Timo Walther created FLINK-18273:
Summary: Test and document remote cluster submission in SQL Client
Key: FLINK-18273
URL: https://issues.apache.org/jira/browse/FLINK-18273
Project: Flink
Issue Type: Improvement
Components: Table SQL / Client
Reporter: Timo Walther
The SQL Client YAML has a deployment section. One can use the regular {{flink
run}} options there and configure e.g. a YARN job session cluster. This is
neither tested nor documented but should work due to the architecture.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-18272) FileSystemLookupFunction can fail if the file gets updated/deleted while cache is reloaded
Rui Li created FLINK-18272: -- Summary: FileSystemLookupFunction can fail if the file gets updated/deleted while cache is reloaded Key: FLINK-18272 URL: https://issues.apache.org/jira/browse/FLINK-18272 Project: Flink Issue Type: Bug Components: FileSystems Reporter: Rui Li Fix For: 1.11.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: request create flip permission for flink es bounded source/lookup source connector
Hi Jacky, What's your username in wiki? So that I can give the permission to you. Best, Jark On Fri, 12 Jun 2020 at 11:38, Jacky Lau wrote: > hi all: >After this simple discussion here > > http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Discussion-flink-elasticsearch-connector-supports-td42082.html#a42106 > , >and i should create i flip127 to track this. But i don't have create > flip permision. > > > > -- > Sent from: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/ >
Re: Flink on Kubes -- issues
BTW, the image you previously attached cannot be displayed. So I assume you
are talking about the "Heap Committed" displayed on Flink's webui?
Thank you~
Xintong Song
On Fri, Jun 12, 2020 at 2:30 PM Xintong Song wrote:
> Do you still run into the "java.lang.OutOfMemoryError: Java heap space"?
>
> If not, then you don't really need to worry about the committed memory.
>
> It is the maximum that really matters. The committed memory should
> increase automatically when it's needed.
>
> Thank you~
>
> Xintong Song
>
>
>
> On Fri, Jun 12, 2020 at 2:24 PM Ramya Ramamurthy
> wrote:
>
>> Hi Xintong,
>>
>> Thanks for the quick response.
>>
>> I have kept my task manager memory to be 1.5GB. But still seeing the Heap
>> committed metric to be around 54MB or so. Why does this happen ? Should I
>> configure any memory fraction configurations here ?
>>
>> Thanks.
>>
>> On Fri, Jun 12, 2020 at 10:58 AM Xintong Song
>> wrote:
>>
>> > Hi Ramya,
>> >
>> > Increasing the memory of your pod will not give you more JVM heap space.
>> > You will need to configure Flink so it launches the JVM process with
>> more
>> > memory.
>> >
>> > In Flink 1.7, this could be achieved by configuring
>> 'jobmanager.heap.size'
>> > & 'taskmanager.heap.size' in your 'flink-conf.yaml'. Both of them are by
>> > default 1024m.
>> >
>> > Please also note that, you should not configure these two options two as
>> > large as your Kubernetes pod. Because Flink may also have some off-heap
>> > memory overhead, so the total memory consumed by the Flink processes
>> might
>> > be larger than configured. This may cause your pods getting killed by
>> > Kubernetes due to memory exceeding.
>> >
>> > According to our experience, leaving around 20~25% of your pod memory
>> for
>> > such overhead might be a good practice. In your case, that means
>> > configuring 'taskmanager.heap.size' to 4GB. If RocksDB is used in your
>> > workload, you may need to further increase the off-heap memory size.
>> >
>> > Thank you~
>> >
>> > Xintong Song
>> >
>> >
>> >
>> > On Fri, Jun 12, 2020 at 1:11 PM Ramya Ramamurthy
>> > wrote:
>> >
>> > > Thanks Till.
>> > > Actually, i have around 5GB pods for each TM, and each pod with only
>> one
>> > > slot.
>> > > But the metrics i have pulled is as below, which is slightly
>> confusing.
>> > > It says only ~50MB of Heap is committed for the tasks. Would you be
>> able
>> > > to point me to the right configuration to be set.
>> > >
>> > > Thanks
>> > > ~Ramya.
>> > >
>> > > [image: image.png]
>> > >
>> > > On Tue, Jun 9, 2020 at 3:12 PM Till Rohrmann
>> > wrote:
>> > >
>> > >> Hi Ramya,
>> > >>
>> > >> it looks as if you should give your Flink pods and also the Flink
>> > process
>> > >> a
>> > >> bit more memory as the process fails with an out of memory error. You
>> > >> could
>> > >> also try Flink's latest version which comes with native Kubernetes
>> > >> support.
>> > >>
>> > >> Cheers,
>> > >> Till
>> > >>
>> > >> On Tue, Jun 9, 2020 at 8:45 AM Ramya Ramamurthy
>> > >> wrote:
>> > >>
>> > >> > Hi,
>> > >> >
>> > >> > My flink jobs are constantly going down beyond an hour with the
>> below
>> > >> > exception.
>> > >> > This is Flink 1.7 on kubes, with checkpoints to Google storage.
>> > >> >
>> > >> > AsynchronousException{java.lang.Exception: Could not materialize
>> > >> > checkpoint 21 for operator Source: Kafka011TableSource(sid,
>> _zpsbd3,
>> > >> > _zpsbd4, _zpsbd6, _zpsbd7, _zpsbd9, lvl_1, isBot, botcode, ssresp,
>> > >> > reason, ts) -> from: (sid, _zpsbd3, _zpsbd6, ts) ->
>> > >> > Timestamps/Watermarks -> where: (<>(sid, _UTF-16LE'7759')), select:
>> > >> > (sid, _zpsbd3, _zpsbd6, ts) -> time attribute: (ts) (5/6).}
>> > >> > at
>> > >> >
>> > >>
>> >
>> org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointExceptionHandler.tryHandleCheckpointException(StreamTask.java:1153)
>> > >> > at
>> > >> >
>> > >>
>> >
>> org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointRunnable.handleExecutionException(StreamTask.java:947)
>> > >> > at
>> > >> >
>> > >>
>> >
>> org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointRunnable.run(StreamTask.java:884)
>> > >> > at
>> > >> >
>> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>> > >> > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>> > >> > at
>> > >> >
>> > >>
>> >
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>> > >> > at
>> > >> >
>> > >>
>> >
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>> > >> > at java.lang.Thread.run(Thread.java:748)
>> > >> > Caused by: java.lang.Exception: Could not materialize checkpoint 21
>> > >> > for operator Source: Kafka011TableSource(sid, _zpsbd3, _zpsbd4,
>> > >> > _zpsbd6, _zpsbd7, _zpsbd9, lvl_1, isBot, botcode, ssresp, reason,
>> ts)
>> > >> > -> from: (sid, _zpsbd3, _zpsbd6, ts) -> Timestamps/Watermarks ->
>> >
Re: Flink on Kubes -- issues
Do you still run into the "java.lang.OutOfMemoryError: Java heap space"?
If not, then you don't really need to worry about the committed memory.
It is the maximum that really matters. The committed memory should increase
automatically when it's needed.
Thank you~
Xintong Song
On Fri, Jun 12, 2020 at 2:24 PM Ramya Ramamurthy wrote:
> Hi Xintong,
>
> Thanks for the quick response.
>
> I have kept my task manager memory to be 1.5GB. But still seeing the Heap
> committed metric to be around 54MB or so. Why does this happen ? Should I
> configure any memory fraction configurations here ?
>
> Thanks.
>
> On Fri, Jun 12, 2020 at 10:58 AM Xintong Song
> wrote:
>
> > Hi Ramya,
> >
> > Increasing the memory of your pod will not give you more JVM heap space.
> > You will need to configure Flink so it launches the JVM process with more
> > memory.
> >
> > In Flink 1.7, this could be achieved by configuring
> 'jobmanager.heap.size'
> > & 'taskmanager.heap.size' in your 'flink-conf.yaml'. Both of them are by
> > default 1024m.
> >
> > Please also note that, you should not configure these two options two as
> > large as your Kubernetes pod. Because Flink may also have some off-heap
> > memory overhead, so the total memory consumed by the Flink processes
> might
> > be larger than configured. This may cause your pods getting killed by
> > Kubernetes due to memory exceeding.
> >
> > According to our experience, leaving around 20~25% of your pod memory for
> > such overhead might be a good practice. In your case, that means
> > configuring 'taskmanager.heap.size' to 4GB. If RocksDB is used in your
> > workload, you may need to further increase the off-heap memory size.
> >
> > Thank you~
> >
> > Xintong Song
> >
> >
> >
> > On Fri, Jun 12, 2020 at 1:11 PM Ramya Ramamurthy
> > wrote:
> >
> > > Thanks Till.
> > > Actually, i have around 5GB pods for each TM, and each pod with only
> one
> > > slot.
> > > But the metrics i have pulled is as below, which is slightly confusing.
> > > It says only ~50MB of Heap is committed for the tasks. Would you be
> able
> > > to point me to the right configuration to be set.
> > >
> > > Thanks
> > > ~Ramya.
> > >
> > > [image: image.png]
> > >
> > > On Tue, Jun 9, 2020 at 3:12 PM Till Rohrmann
> > wrote:
> > >
> > >> Hi Ramya,
> > >>
> > >> it looks as if you should give your Flink pods and also the Flink
> > process
> > >> a
> > >> bit more memory as the process fails with an out of memory error. You
> > >> could
> > >> also try Flink's latest version which comes with native Kubernetes
> > >> support.
> > >>
> > >> Cheers,
> > >> Till
> > >>
> > >> On Tue, Jun 9, 2020 at 8:45 AM Ramya Ramamurthy
> > >> wrote:
> > >>
> > >> > Hi,
> > >> >
> > >> > My flink jobs are constantly going down beyond an hour with the
> below
> > >> > exception.
> > >> > This is Flink 1.7 on kubes, with checkpoints to Google storage.
> > >> >
> > >> > AsynchronousException{java.lang.Exception: Could not materialize
> > >> > checkpoint 21 for operator Source: Kafka011TableSource(sid, _zpsbd3,
> > >> > _zpsbd4, _zpsbd6, _zpsbd7, _zpsbd9, lvl_1, isBot, botcode, ssresp,
> > >> > reason, ts) -> from: (sid, _zpsbd3, _zpsbd6, ts) ->
> > >> > Timestamps/Watermarks -> where: (<>(sid, _UTF-16LE'7759')), select:
> > >> > (sid, _zpsbd3, _zpsbd6, ts) -> time attribute: (ts) (5/6).}
> > >> > at
> > >> >
> > >>
> >
> org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointExceptionHandler.tryHandleCheckpointException(StreamTask.java:1153)
> > >> > at
> > >> >
> > >>
> >
> org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointRunnable.handleExecutionException(StreamTask.java:947)
> > >> > at
> > >> >
> > >>
> >
> org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointRunnable.run(StreamTask.java:884)
> > >> > at
> > >> >
> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> > >> > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > >> > at
> > >> >
> > >>
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> > >> > at
> > >> >
> > >>
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > >> > at java.lang.Thread.run(Thread.java:748)
> > >> > Caused by: java.lang.Exception: Could not materialize checkpoint 21
> > >> > for operator Source: Kafka011TableSource(sid, _zpsbd3, _zpsbd4,
> > >> > _zpsbd6, _zpsbd7, _zpsbd9, lvl_1, isBot, botcode, ssresp, reason,
> ts)
> > >> > -> from: (sid, _zpsbd3, _zpsbd6, ts) -> Timestamps/Watermarks ->
> > >> > where: (<>(sid, _UTF-16LE'7759')), select: (sid, _zpsbd3, _zpsbd6,
> ts)
> > >> > -> time attribute: (ts) (5/6).
> > >> > at
> > >> >
> > >>
> >
> org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointRunnable.handleExecutionException(StreamTask.java:942)
> > >> > ... 6 more
> > >> > Caused by: java.util.concurrent.ExecutionException:
> > >> >
Re: Flink on Kubes -- issues
Hi Xintong,
Thanks for the quick response.
I have kept my task manager memory to be 1.5GB. But still seeing the Heap
committed metric to be around 54MB or so. Why does this happen ? Should I
configure any memory fraction configurations here ?
Thanks.
On Fri, Jun 12, 2020 at 10:58 AM Xintong Song wrote:
> Hi Ramya,
>
> Increasing the memory of your pod will not give you more JVM heap space.
> You will need to configure Flink so it launches the JVM process with more
> memory.
>
> In Flink 1.7, this could be achieved by configuring 'jobmanager.heap.size'
> & 'taskmanager.heap.size' in your 'flink-conf.yaml'. Both of them are by
> default 1024m.
>
> Please also note that, you should not configure these two options two as
> large as your Kubernetes pod. Because Flink may also have some off-heap
> memory overhead, so the total memory consumed by the Flink processes might
> be larger than configured. This may cause your pods getting killed by
> Kubernetes due to memory exceeding.
>
> According to our experience, leaving around 20~25% of your pod memory for
> such overhead might be a good practice. In your case, that means
> configuring 'taskmanager.heap.size' to 4GB. If RocksDB is used in your
> workload, you may need to further increase the off-heap memory size.
>
> Thank you~
>
> Xintong Song
>
>
>
> On Fri, Jun 12, 2020 at 1:11 PM Ramya Ramamurthy
> wrote:
>
> > Thanks Till.
> > Actually, i have around 5GB pods for each TM, and each pod with only one
> > slot.
> > But the metrics i have pulled is as below, which is slightly confusing.
> > It says only ~50MB of Heap is committed for the tasks. Would you be able
> > to point me to the right configuration to be set.
> >
> > Thanks
> > ~Ramya.
> >
> > [image: image.png]
> >
> > On Tue, Jun 9, 2020 at 3:12 PM Till Rohrmann
> wrote:
> >
> >> Hi Ramya,
> >>
> >> it looks as if you should give your Flink pods and also the Flink
> process
> >> a
> >> bit more memory as the process fails with an out of memory error. You
> >> could
> >> also try Flink's latest version which comes with native Kubernetes
> >> support.
> >>
> >> Cheers,
> >> Till
> >>
> >> On Tue, Jun 9, 2020 at 8:45 AM Ramya Ramamurthy
> >> wrote:
> >>
> >> > Hi,
> >> >
> >> > My flink jobs are constantly going down beyond an hour with the below
> >> > exception.
> >> > This is Flink 1.7 on kubes, with checkpoints to Google storage.
> >> >
> >> > AsynchronousException{java.lang.Exception: Could not materialize
> >> > checkpoint 21 for operator Source: Kafka011TableSource(sid, _zpsbd3,
> >> > _zpsbd4, _zpsbd6, _zpsbd7, _zpsbd9, lvl_1, isBot, botcode, ssresp,
> >> > reason, ts) -> from: (sid, _zpsbd3, _zpsbd6, ts) ->
> >> > Timestamps/Watermarks -> where: (<>(sid, _UTF-16LE'7759')), select:
> >> > (sid, _zpsbd3, _zpsbd6, ts) -> time attribute: (ts) (5/6).}
> >> > at
> >> >
> >>
> org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointExceptionHandler.tryHandleCheckpointException(StreamTask.java:1153)
> >> > at
> >> >
> >>
> org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointRunnable.handleExecutionException(StreamTask.java:947)
> >> > at
> >> >
> >>
> org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointRunnable.run(StreamTask.java:884)
> >> > at
> >> >
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> >> > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> >> > at
> >> >
> >>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> >> > at
> >> >
> >>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> >> > at java.lang.Thread.run(Thread.java:748)
> >> > Caused by: java.lang.Exception: Could not materialize checkpoint 21
> >> > for operator Source: Kafka011TableSource(sid, _zpsbd3, _zpsbd4,
> >> > _zpsbd6, _zpsbd7, _zpsbd9, lvl_1, isBot, botcode, ssresp, reason, ts)
> >> > -> from: (sid, _zpsbd3, _zpsbd6, ts) -> Timestamps/Watermarks ->
> >> > where: (<>(sid, _UTF-16LE'7759')), select: (sid, _zpsbd3, _zpsbd6, ts)
> >> > -> time attribute: (ts) (5/6).
> >> > at
> >> >
> >>
> org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointRunnable.handleExecutionException(StreamTask.java:942)
> >> > ... 6 more
> >> > Caused by: java.util.concurrent.ExecutionException:
> >> > java.lang.OutOfMemoryError: Java heap space
> >> > at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> >> > at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> >> > at
> >> >
> org.apache.flink.util.FutureUtil.runIfNotDoneAndGet(FutureUtil.java:53)
> >> > at
> >> >
> >>
> org.apache.flink.streaming.api.operators.OperatorSnapshotFinalizer.(OperatorSnapshotFinalizer.java:53)
> >> > at
> >> >
> >>
> org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointRunnable.run(StreamTask.java:853)
> >> > ... 5 more
> >> > Caused by:
[jira] [Created] (FLINK-18271) Move run-nightly-tests.sh
Robert Metzger created FLINK-18271: -- Summary: Move run-nightly-tests.sh Key: FLINK-18271 URL: https://issues.apache.org/jira/browse/FLINK-18271 Project: Flink Issue Type: Task Reporter: Robert Metzger -- This message was sent by Atlassian Jira (v8.3.4#803005)
