[ https://issues.apache.org/jira/browse/FLINK-37802 ]
jackray wang deleted comment on FLINK-37802:
--------------------------------------
was (Author: jackray):
!image-2025-08-11-19-56-32-416.png!
> files in directory job-result-store was incorrect
> -------------------------------------------------
>
> Key: FLINK-37802
> URL: https://issues.apache.org/jira/browse/FLINK-37802
> Project: Flink
> Issue Type: Bug
> Affects Versions: 1.17.1
> Reporter: yanggang
> Priority: Major
> Attachments: image-2025-05-14-16-32-05-505.png,
> image-2025-05-14-17-18-20-701.png, image-2025-05-14-17-18-32-785.png
>
>
> there parse json file , throw a exception when json is not correct
> !image-2025-05-14-17-18-32-785.png!
> file size about 3.3MB
> !image-2025-05-14-16-32-05-505.png!
> this is file content. business data and some trace was hidden:
> {
> "result": {
> "id": "fcae72c8faccbab418489585445c1969",
> "application-status": "FAILED",
> "accumulator-results": {},
> "net-runtime": 169179245,
> "failure-cause":
> { "class": "org.apache.flink.runtime.JobException", "stack-trace":
> "org.apache.flink.runtime.JobException: Recovery is suppressed by
> FixedDelayRestartBackoffTimeStrategy(maxNumberRestartAttempts=3,
> backoffTimeMS=30000)\n\tat
> org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.handleFailure(ExecutionFailureHandler.java:139)\n\tat
>
> org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.getFailureHandlingResult(ExecutionFailureHandler.java:83)\n\tat
>
> org.apache.flink.runtime.scheduler.DefaultScheduler.recordTaskFailure(DefaultScheduler.java:258)\n\tat
> org.apache.flink.runtime.scheduler.DefaultScheduler.handleTaskFailure ---
> hidden --- }
> \n\tat
> org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:839)\n\tat
>
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:788)\n\tat
>
> org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:952)\n\tat
>
> org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:931)\n\tat
> org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:745)\n\tat
> org.apache.flink.runtime.taskmanager.Task.run(Task.java:562)\n\tat
> java.lang.Thread.run(Thread.java:750)\nCaused by:
> java.util.concurrent.TimeoutException: Async function call has timed
> out.\n\t... 19 more\n"
>
>
> The above is the json content of the file, mybe missed }}}
>
>
> flink error log:
> 2025-05-13 22:30:19,082 ERROR [cluster-io-thread-1]
> [org.apache.flink.runtime.entrypoint.ClusterEntrypoint :548] - Fatal
> error occurred in the cluster entrypoint.
> java.util.concurrent.CompletionException:
> org.apache.flink.util.FlinkRuntimeException: Could not retrieve JobResults of
> globally-terminated jobs from JobResultStore
> at
> java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:273)
> ~[?:1.8.0_432]
> at
> java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:280)
> [?:1.8.0_432]
> at
> java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1606)
> [?:1.8.0_432]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> [?:1.8.0_432]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> [?:1.8.0_432]
> at java.lang.Thread.run(Thread.java:750) [?:1.8.0_432]
> Caused by: org.apache.flink.util.FlinkRuntimeException: Could not retrieve
> JobResults of globally-terminated jobs from JobResultStore
> at
> org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess.getDirtyJobResults(SessionDispatcherLeaderProcess.java:196)
> ~[performance-flink-job-24.0.2118.jar:?]
> at
> org.apache.flink.runtime.dispatcher.runner.AbstractDispatcherLeaderProcess.supplyUnsynchronizedIfRunning(AbstractDispatcherLeaderProcess.java:198)
> ~[performance-flink-job-24.0.2118.jar:?]
> at
> org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess.getDirtyJobResultsIfRunning(SessionDispatcherLeaderProcess.java:188)
> ~[performance-flink-job-24.0.2118.jar:?]
> at
> java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
> ~[?:1.8.0_432]
> ... 3 more
> Caused by:
> org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.JsonMappingException:
> Unexpected end-of-input: expected close marker for Object (start marker at
> [Source: (org.apache.flink.core.fs.local.LocalDataInputStream); line: 1,
> column: 147])
> at [Source: (org.apache.flink.core.fs.local.LocalDataInputStream); line: 1,
> column: 3223450] (through reference chain:
> org.apache.flink.runtime.highavailability.FileSystemJobResultStore$JsonJobResultEntry["result"])
> at
> org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:392)
> ~[performance-flink-job-24.0.2118.jar:?]
> at
> org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:351)
> ~[performance-flink-job-24.0.2118.jar:?]
> at
> org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.BeanDeserializerBase.wrapAndThrow(BeanDeserializerBase.java:1821)
> ~[performance-flink-job-24.0.2118.jar:?]
> at
> org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeWithErrorWrapping(BeanDeserializer.java:566)
> ~[performance-flink-job-24.0.2118.jar:?]
> at
> org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased(BeanDeserializer.java:439)
> ~[performance-flink-job-24.0.2118.jar:?]
> at
> org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1405)
> ~[performance-flink-job-24.0.2118.jar:?]
> at
> org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:352)
> ~[performance-flink-job-24.0.2118.jar:?]
> at
> org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:185)
> ~[performance-flink-job-24.0.2118.jar:?]
> at
> org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.DefaultDeserializationContext.readRootValue(DefaultDeserializationContext.java:323)
> ~[performance-flink-job-24.0.2118.jar:?]
> at
> org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4674)
> ~[performance-flink-job-24.0.2118.jar:?]
> at
> org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3666)
> ~[performance-flink-job-24.0.2118.jar:?]
> at
> org.apache.flink.runtime.highavailability.FileSystemJobResultStore.getDirtyResultsInternal(FileSystemJobResultStore.java:208)
> ~[performance-flink-job-24.0.2118.jar:?]
> at
> org.apache.flink.runtime.highavailability.AbstractThreadsafeJobResultStore.withReadLock(AbstractThreadsafeJobResultStore.java:118)
> ~[performance-flink-job-24.0.2118.jar:?]
> at
> org.apache.flink.runtime.highavailability.AbstractThreadsafeJobResultStore.getDirtyResults(AbstractThreadsafeJobResultStore.java:100)
> ~[performance-flink-job-24.0.2118.jar:?]
> at
> org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess.getDirtyJobResults(SessionDispatcherLeaderProcess.java:194)
> ~[performance-flink-job-24.0.2118.jar:?]
> at
> org.apache.flink.runtime.dispatcher.runner.AbstractDispatcherLeaderProcess.supplyUnsynchronizedIfRunning(AbstractDispatcherLeaderProcess.java:198)
> ~[performance-flink-job-24.0.2118.jar:?]
> at
> org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess.getDirtyJobResultsIfRunning(SessionDispatcherLeaderProcess.java:188)
> ~[performance-flink-job-24.0.2118.jar:?]
> at
> java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
> ~[?:1.8.0_432]
> ... 3 more
> Caused by:
> org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.io.JsonEOFException:
> Unexpected end-of-input: expected close marker for Object (start marker at
> [Source: (org.apache.flink.core.fs.local.LocalDataInputStream); line: 1,
> column: 147])
> at [Source: (org.apache.flink.core.fs.local.LocalDataInputStream); line: 1,
> column: 3223450]
> at
> org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.base.ParserMinimalBase._reportInvalidEOF(ParserMinimalBase.java:682)
> ~[performance-flink-job-24.0.2118.jar:?]
> at
> org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.base.ParserBase._handleEOF(ParserBase.java:494)
> ~[performance-flink-job-24.0.2118.jar:?]
> at
> org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.base.ParserBase._eofAsNextChar(ParserBase.java:511)
> ~[performance-flink-job-24.0.2118.jar:?]
> at
> org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.json.UTF8StreamJsonParser._skipWSOrEnd(UTF8StreamJsonParser.java:3039)
> ~[performance-flink-job-24.0.2118.jar:?]
> at
> org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.json.UTF8StreamJsonParser.nextFieldName(UTF8StreamJsonParser.java:1034)
> ~[performance-flink-job-24.0.2118.jar:?]
> at
> org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.std.BaseNodeDeserializer._deserializeContainerNoRecursion(JsonNodeDeserializer.java:440)
> ~[performance-flink-job-24.0.2118.jar:?]
> at
> org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.std.JsonNodeDeserializer.deserialize(JsonNodeDeserializer.java:84)
> ~[performance-flink-job-24.0.2118.jar:?]
> at
> org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.std.JsonNodeDeserializer.deserialize(JsonNodeDeserializer.java:20)
> ~[performance-flink-job-24.0.2118.jar:?]
> at
> org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.DefaultDeserializationContext.readRootValue(DefaultDeserializationContext.java:323)
> ~[performance-flink-job-24.0.2118.jar:?]
> at
> org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ObjectMapper._readValue(ObjectMapper.java:4650)
> ~[performance-flink-job-24.0.2118.jar:?]
> at
> org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ObjectMapper.readTree(ObjectMapper.java:2941)
> ~[performance-flink-job-24.0.2118.jar:?]
> at
> org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.JsonParser.readValueAsTree(JsonParser.java:2365)
> ~[performance-flink-job-24.0.2118.jar:?]
> at
> org.apache.flink.runtime.rest.messages.json.SerializedThrowableDeserializer.deserialize(SerializedThrowableDeserializer.java:45)
> ~[performance-flink-job-24.0.2118.jar:?]
> at
> org.apache.flink.runtime.rest.messages.json.JobResultDeserializer.deserialize(JobResultDeserializer.java:106)
> ~[performance-flink-job-24.0.2118.jar:?]
> at
> org.apache.flink.runtime.rest.messages.json.JobResultDeserializer.deserialize(JobResultDeserializer.java:50)
> ~[performance-flink-job-24.0.2118.jar:?]
> at
> org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.SettableBeanProperty.deserialize(SettableBeanProperty.java:542)
> ~[performance-flink-job-24.0.2118.jar:?]
> at
> org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeWithErrorWrapping(BeanDeserializer.java:564)
> ~[performance-flink-job-24.0.2118.jar:?]
> at
> org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased(BeanDeserializer.java:439)
> ~[performance-flink-job-24.0.2118.jar:?]
> at
> org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1405)
> ~[performance-flink-job-24.0.2118.jar:?]
> at
> org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:352)
> ~[performance-flink-job-24.0.2118.jar:?]
> at
> org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:185)
> ~[performance-flink-job-24.0.2118.jar:?]
> at
> org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.DefaultDeserializationContext.readRootValue(DefaultDeserializationContext.java:323)
> ~[performance-flink-job-24.0.2118.jar:?]
> at
> org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4674)
> ~[performance-flink-job-24.0.2118.jar:?]
> at
> org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3666)
> ~[performance-flink-job-24.0.2118.jar:?]
> at
> org.apache.flink.runtime.highavailability.FileSystemJobResultStore.getDirtyResultsInternal(FileSystemJobResultStore.java:208)
> ~[performance-flink-job-24.0.2118.jar:?]
> at
> org.apache.flink.runtime.highavailability.AbstractThreadsafeJobResultStore.withReadLock(AbstractThreadsafeJobResultStore.java:118)
> ~[performance-flink-job-24.0.2118.jar:?]
> at
> org.apache.flink.runtime.highavailability.AbstractThreadsafeJobResultStore.getDirtyResults(AbstractThreadsafeJobResultStore.java:100)
> ~[performance-flink-job-24.0.2118.jar:?]
> at
> org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess.getDirtyJobResults(SessionDispatcherLeaderProcess.java:194)
> ~[performance-flink-job-24.0.2118.jar:?]
> at
> org.apache.flink.runtime.dispatcher.runner.AbstractDispatcherLeaderProcess.supplyUnsynchronizedIfRunning(AbstractDispatcherLeaderProcess.java:198)
> ~[performance-flink-job-24.0.2118.jar:?]
> at
> org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess.getDirtyJobResultsIfRunning(SessionDispatcherLeaderProcess.java:188)
> ~[performance-flink-job-24.0.2118.jar:?]
> at
> java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
> ~[?:1.8.0_432]
> ... 3 more
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
