你说的这种情况，只能是分析 jvm heap dump 看下内存都用在哪里了。 可以试下配置 env.java.opts 添加 -XX:+HeapDumpOnOutOfMemoryError。
Thank you~ Xintong Song On Mon, Apr 6, 2020 at 3:30 PM 欧阳苗 <[email protected]> wrote: > 你好， > 我看了下这篇文档，但是只给出了加大jvm的内存，但是我更想知道的是该如何排查哪一部分代码造成的内存溢出，请问有什么方法吗。 > 其次我这里的环境是，flink on k8s, 单机版。有一个服务是专门远程flink job,这个服务也是部署在k8s中 > > > > > ------------------ 原始邮件 ------------------ > 发件人: "Xintong Song"<[email protected]>; > 发送时间: 2020年4月6日(星期一) 下午2:56 > 收件人: "user-zh"<[email protected]>; > > 主题: Re: flink 内存溢出 如何排查 > > > > 你好， > > 请参考官方文档：部署与运维/内存配置/常见问题/OutOfMemoryError: Java heap space > > https://ci.apache.org/projects/flink/flink-docs-release-1.10/zh/ops/memory/mem_trouble.html#outofmemoryerror-java-heap-space > > Thank you~ > > Xintong Song > > > > On Mon, Apr 6, 2020 at 12:33 AM 欧阳苗 <[email protected]> wrote: > > > 2020-04-04 18:58:41.515 ERROR 1 --- [ent-IO-thread-4] > > org.apache.flink.runtime.rest.RestClient.parseResponse:393 : Received > > response was neither of the expected type ([simple type, class > > > org.apache.flink.runtime.rest.messages.job.JobExecutionResultResponseBody]) > > nor an error. > > > Response=JsonResponse{json={"status":{"id":"COMPLETED"},"job-execution-result":{"id":"95bfafbc9b8e31565d44ae0018a0af7d","application-status":"FAILED","accumulator-results":{},"net-runtime":3098,"failure-cause":{"class":"java.lang.OutOfMemoryError","stack-trace":"java.lang.OutOfMemoryError: > > Java heap > > > space\n","serialized-throwable":"rO0ABXNyAClvcmcuYXBhY2hlLmZsaW5rLnV0aWwuU2VyaWFsaXplZFRocm93YWJsZWUWnfUfpxPzAgADTAAZZnVsbFN0cmluZ2lmaWVkU3RhY2tUcmFjZXQAEkxqYXZhL2xhbmcvU3RyaW5nO0wAFm9yaWdpbmFsRXJyb3JDbGFzc05hbWVxAH4AAVsAE3NlcmlhbGl6ZWRFeGNlcHRpb250AAJbQnhyABNqYXZhLmxhbmcuRXhjZXB0aW9u0P0fPho7HMQCAAB4cgATamF2YS5sYW5nLlRocm93YWJsZdXGNSc5d7jLAwAETAAFY2F1c2V0ABVMamF2YS9sYW5nL1Rocm93YWJsZTtMAA1kZXRhaWxNZXNzYWdlcQB+AAFbAApzdGFja1RyYWNldAAeW0xqYXZhL2xhbmcvU3RhY2tUcmFjZUVsZW1lbnQ7TAAUc3VwcHJlc3NlZEV4Y2VwdGlvbnN0ABBMamF2YS91dGlsL0xpc3Q7eHBwdAAPSmF2YSBoZWFwIHNwYWNldXIAHltMamF2YS5sYW5nLlN0YWNrVHJhY2VFbGVtZW50OwJGKjw8/SI5AgAAeHAAAAAAc3IAJmphdmEudXRpbC5Db2xsZWN0aW9ucyRVbm1vZGlmaWFibGVMaXN0/A8lMbXsjhACAAFMAARsaXN0cQB+AAd4cgAsamF2YS51dGlsLkNvbGxlY3Rpb25zJFVubW9kaWZpYWJsZUNvbGxlY3Rpb24ZQgCAy173HgIAAUwAAWN0ABZMamF2YS91dGlsL0NvbGxlY3Rpb247eHBzcgATamF2YS51dGlsLkFycmF5TGlzdHiB0h2Zx2GdAwABSQAEc2l6ZXhwAAAAAHcEAAAAAHhxAH4AEXh0ACxqYXZhLmxhbmcuT3V0T2ZNZW1vcnlFcnJvcjogSmF2YSBoZWFwIHNwYWNlCnQAGmphdmEubGFuZy5PdXRPZk1lbW9yeUVycm9ydXIAAltCrPMX+AYIVOACAAB4cAAAAf6s7QAFc3IAGmphdmEubGFuZy5PdXRPZk1lbW9yeUVycm9ycjG7cIiI4xUCAAB4cgAdamF2YS5sYW5nLlZpcnR1YWxNYWNoaW5lRXJyb3I5wlZUgC8OHgIAAHhyAA9qYXZhLmxhbmcuRXJyb3JFHTZWi4IOVgIAAHhyABNqYXZhLmxhbmcuVGhyb3dhYmxl1cY1Jzl3uMsDAARMAAVjYXVzZXQAFUxqYXZhL2xhbmcvVGhyb3dhYmxlO0wADWRldGFpbE1lc3NhZ2V0ABJMamF2YS9sYW5nL1N0cmluZztbAApzdGFja1RyYWNldAAeW0xqYXZhL2xhbmcvU3RhY2tUcmFjZUVsZW1lbnQ7TAAUc3VwcHJlc3NlZEV4Y2VwdGlvbnN0ABBMamF2YS91dGlsL0xpc3Q7eHBwdAAPSmF2YSBoZWFwIHNwYWNldXIAHltMamF2YS5sYW5nLlN0YWNrVHJhY2VFbGVtZW50OwJGKjw8/SI5AgAAeHAAAAABc3IAG2phdmEubGFuZy5TdGFja1RyYWNlRWxlbWVudGEJxZomNt2FAgAESQAKbGluZU51bWJlckwADmRlY2xhcmluZ0NsYXNzcQB+AAVMAAhmaWxlTmFtZXEAfgAFTAAKbWV0aG9kTmFtZXEAfgAFeHCAAAAAdAAAcHEAfgAOcHg="}}}, > > httpResponseStatus=200 OK} > > > > > > > org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException: > > Unrecognized field "status" (class > > org.apache.flink.runtime.rest.messages.ErrorResponseBody), not marked > as > > ignorable (one known property: "errors"]) > > &nbsp;at [Source: N/A; line: -1, column: -1] (through reference > chain: > > org.apache.flink.runtime.rest.messages.ErrorResponseBody["status"]) > > at > > > org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException.from(UnrecognizedPropertyException.java:62) > > ~[flink-shaded-jackson-2.7.9-6.0.jar!/:2.7.9-6.0] > > at > > > org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.DeserializationContext.reportUnknownProperty(DeserializationContext.java:851) > > ~[flink-shaded-jackson-2.7.9-6.0.jar!/:2.7.9-6.0] > > at > > > org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.std.StdDeserializer.handleUnknownProperty(StdDeserializer.java:1085) > > ~[flink-shaded-jackson-2.7.9-6.0.jar!/:2.7.9-6.0] > > at > > > org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.BeanDeserializerBase.handleUnknownProperty(BeanDeserializerBase.java:1392) > > ~[flink-shaded-jackson-2.7.9-6.0.jar!/:2.7.9-6.0] > > at > > > org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.BeanDeserializerBase.handleUnknownProperties(BeanDeserializerBase.java:1346) > > ~[flink-shaded-jackson-2.7.9-6.0.jar!/:2.7.9-6.0] > > at > > > org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased(BeanDeserializer.java:455) > > ~[flink-shaded-jackson-2.7.9-6.0.jar!/:2.7.9-6.0] > > at > > > org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1127) > > ~[flink-shaded-jackson-2.7.9-6.0.jar!/:2.7.9-6.0] > > at > > > org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:298) > > ~[flink-shaded-jackson-2.7.9-6.0.jar!/:2.7.9-6.0] > > at > > > org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:133) > > ~[flink-shaded-jackson-2.7.9-6.0.jar!/:2.7.9-6.0] > > at > > > org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ObjectMapper._readValue(ObjectMapper.java:3779) > > ~[flink-shaded-jackson-2.7.9-6.0.jar!/:2.7.9-6.0] > > at > > > org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2050) > > ~[flink-shaded-jackson-2.7.9-6.0.jar!/:2.7.9-6.0] > > at > > > org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ObjectMapper.treeToValue(ObjectMapper.java:2547) > > ~[flink-shaded-jackson-2.7.9-6.0.jar!/:2.7.9-6.0] > > at > > > org.apache.flink.runtime.rest.RestClient.parseResponse(RestClient.java:388) > > ~[flink-runtime_2.12-1.8.0.jar!/:1.8.0] > > at > > > org.apache.flink.runtime.rest.RestClient.lambda$submitRequest$3(RestClient.java:373) > > ~[flink-runtime_2.12-1.8.0.jar!/:1.8.0] > > at > > > java.util.concurrent.CompletableFuture.uniCompose(CompletableFuture.java:952) > > [?:1.8.0_192] > > at > > > java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:926) > > [?:1.8.0_192] > > at > > > java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:442) > > [?:1.8.0_192] > > at > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > > [?:1.8.0_192] > > at > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > > [?:1.8.0_192] > > at > java.lang.Thread.run(Thread.java:748) [?:1.8.0_192]
