我昨天模拟下断电的情况 10个ha文件的日期是错开的5秒一个 chk-xxx也不是都损坏了，有的是可以启动的，这个我也试了 现在情况是 yarn集群停电重启首先会循环尝试从10个ha的文件中启动应用，ha文件记录的chk的相关原数据 1.如果ha文件都损坏了，即使chk没有损坏，flink应用也是拉不起来的
现在想的是让hdfs上存在至少1组个可用的的ha文件及其对应的chk 现在是5秒一个chk，保存了10个，也会出现损坏无法启动的问题 5秒*10 = 50秒，也想知道多长时间的存档才能保证存在一组没有损坏ha和chk呢。 在 2023-03-14 10:16:48，"Guojun Li" <gjli.schna...@gmail.com> 写道： >Hi > >确认一下这些 ha 文件的 last modification time 是一致的还是错开的？ > >另外，指定 chk- 恢复尝试了没有？可以恢复吗？ > >Best, >Guojun > >On Fri, Mar 10, 2023 at 11:56 AM guanyq <dlgua...@163.com> wrote: > >> flink ha路径为 /tmp/flink/ha/ >> flink chk路径为 /tmp/flink/checkpoint >> >> >> 我现在不确定是这个ha的文件损坏了，还是所有chk都损坏，但是这个需要模拟验证一下。 >> >> >> >> >> 会尝试从10个chk恢复，日志有打印 >> 2023-03-0718:37:43,703INFOorg.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore >> - Recovering checkpoints from ZooKeeper. >> 2023-03-0718:37:43,730INFOorg.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore >> - Found 10 checkpoints in ZooKeeper. >> 2023-03-0718:37:43,731INFOorg.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore >> - Trying to fetch 10 checkpoints from storage. >> 2023-03-0718:37:43,731INFOorg.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore >> - Trying to retrieve checkpoint 7079. >> 2023-03-0718:37:43,837INFOorg.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore >> - Trying to retrieve checkpoint 7080. >> 2023-03-0718:37:43,868INFOorg.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore >> - Trying to retrieve checkpoint 7081. >> 2023-03-0718:37:43,896INFOorg.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore >> - Trying to retrieve checkpoint 7082. >> 2023-03-0718:37:43,906INFOorg.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore >> - Trying to retrieve checkpoint 7083. >> 2023-03-0718:37:43,928INFOorg.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore >> - Trying to retrieve checkpoint 7084. >> 2023-03-0718:37:43,936INFOorg.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore >> - Trying to retrieve checkpoint 7085. >> 2023-03-0718:37:43,947INFOorg.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore >> - Trying to retrieve checkpoint 7086. >> >> >> >> 详细日志为，后面重复部分我给省略了，不同点就是尝试不同的/tmp/flink/ha/application_1678102326043_0007/completedCheckpointxxxxxx启动 >> 2023-03-0718:37:43,621INFOorg.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl >> - Starting the SlotManager. >> 2023-03-0718:37:43,621INFOorg.apache.flink.runtime.jobmaster.JobMaster - >> Successfully ran initialization on master in 0 ms. >> 2023-03-0718:37:43,660INFOorg.apache.flink.runtime.util.ZooKeeperUtils - >> Initialized ZooKeeperCompletedCheckpointStore in >> '/checkpoints/3844b96b002601d932e66233dd46899c'. >> 2023-03-0718:37:43,680INFOorg.apache.flink.runtime.jobmaster.JobMaster - >> Using application-defined state backend: File State Backend (checkpoints: >> 'hdfs:/tmp/flink/checkpoint', savepoints: 'null', asynchronous: UNDEFINED, >> fileStateThreshold: -1) >> 2023-03-0718:37:43,680INFOorg.apache.flink.runtime.jobmaster.JobMaster - >> Configuring application-defined state backend with job/cluster config >> 2023-03-0718:37:43,703INFOorg.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore >> - Recovering checkpoints from ZooKeeper. >> 2023-03-0718:37:43,730INFOorg.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore >> - Found 10 checkpoints in ZooKeeper. >> 2023-03-0718:37:43,731INFOorg.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore >> - Trying to fetch 10 checkpoints from storage. >> 2023-03-0718:37:43,731INFOorg.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore >> - Trying to retrieve checkpoint 7079. >> 2023-03-0718:37:43,837INFOorg.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore >> - Trying to retrieve checkpoint 7080. >> 2023-03-0718:37:43,868INFOorg.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore >> - Trying to retrieve checkpoint 7081. >> 2023-03-0718:37:43,896INFOorg.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore >> - Trying to retrieve checkpoint 7082. >> 2023-03-0718:37:43,906INFOorg.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore >> - Trying to retrieve checkpoint 7083. >> 2023-03-0718:37:43,928INFOorg.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore >> - Trying to retrieve checkpoint 7084. >> 2023-03-0718:37:43,936INFOorg.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore >> - Trying to retrieve checkpoint 7085. >> 2023-03-0718:37:43,947INFOorg.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore >> - Trying to retrieve checkpoint 7086. >> 2023-03-0718:37:43,979WARNorg.apache.hadoop.hdfs.BlockReaderFactory - I/O >> error constructing remote block reader. >> java.io.IOException: Got error, status message opReadBlock >> BP-1003103929-192.168.200.11-1668473836936:blk_1301252639_227512278 >> received exception >> org.apache.hadoop.hdfs.server.datanode.CorruptMetaHeaderException: >> The meta file length 0 is less than the expected length 7, for >> OP_READ_BLOCK, self=/192.168.200.23:45534, remote=/192.168.200.21:9866, >> for file >> /tmp/flink/ha/application_1678102326043_0007/completedCheckpoint58755403e33a, >> for pool BP-1003103929-192.168.200.11-1668473836936 block >> 1301252639_227512278 >> at >> org.apache.hadoop.hdfs.protocol.datatransfer.DataTransferProtoUtil.checkBlockOpStatus(DataTransferProtoUtil.java:142) >> at >> org.apache.hadoop.hdfs.RemoteBlockReader2.checkSuccess(RemoteBlockReader2.java:456) >> at >> org.apache.hadoop.hdfs.RemoteBlockReader2.newBlockReader(RemoteBlockReader2.java:424) >> at >> org.apache.hadoop.hdfs.BlockReaderFactory.getRemoteBlockReader(BlockReaderFactory.java:818) >> at >> org.apache.hadoop.hdfs.BlockReaderFactory.getRemoteBlockReaderFromTcp(BlockReaderFactory.java:697) >> at >> org.apache.hadoop.hdfs.BlockReaderFactory.build(BlockReaderFactory.java:355) >> at >> org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:665) >> at >> org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:874) >> at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:926) >> at java.io.DataInputStream.read(DataInputStream.java:149) >> at >> org.apache.flink.runtime.fs.hdfs.HadoopDataInputStream.read(HadoopDataInputStream.java:94) >> at >> java.io.ObjectInputStream$PeekInputStream.read(ObjectInputStream.java:2663) >> at >> java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2679) >> at >> java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:3156) >> at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:862) >> at java.io.ObjectInputStream.<init>(ObjectInputStream.java:358) >> at >> org.apache.flink.util.InstantiationUtil$ClassLoaderObjectInputStream.<init>(InstantiationUtil.java:69) >> at >> org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:572) >> at >> org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:555) >> at >> org.apache.flink.runtime.state.RetrievableStreamStateHandle.retrieveState(RetrievableStreamStateHandle.java:58) >> at >> org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore.retrieveCompletedCheckpoint(ZooKeeperCompletedCheckpointStore.java:339) >> at >> org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore.recover(ZooKeeperCompletedCheckpointStore.java:175) >> at >> org.apache.flink.runtime.checkpoint.CheckpointCoordinator.restoreLatestCheckpointedState(CheckpointCoordinator.java:1070) >> at >> org.apache.flink.runtime.scheduler.SchedulerBase.createAndRestoreExecutionGraph(SchedulerBase.java:234) >> at >> org.apache.flink.runtime.scheduler.SchedulerBase.<init>(SchedulerBase.java:216) >> at >> org.apache.flink.runtime.scheduler.DefaultScheduler.<init>(DefaultScheduler.java:120) >> at >> org.apache.flink.runtime.scheduler.DefaultSchedulerFactory.createInstance(DefaultSchedulerFactory.java:105) >> at >> org.apache.flink.runtime.jobmaster.JobMaster.createScheduler(JobMaster.java:278) >> at org.apache.flink.runtime.jobmaster.JobMaster.<init>(JobMaster.java:266) >> at >> org.apache.flink.runtime.jobmaster.factories.DefaultJobMasterServiceFactory.createJobMasterService(DefaultJobMasterServiceFactory.java:98) >> at >> org.apache.flink.runtime.jobmaster.factories.DefaultJobMasterServiceFactory.createJobMasterService(DefaultJobMasterServiceFactory.java:40) >> at >> org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl.<init>(JobManagerRunnerImpl.java:146) >> at >> org.apache.flink.runtime.dispatcher.DefaultJobManagerRunnerFactory.createJobManagerRunner(DefaultJobManagerRunnerFactory.java:84) >> at >> org.apache.flink.runtime.dispatcher.Dispatcher.lambda$createJobManagerRunner$6(Dispatcher.java:381) >> at >> org.apache.flink.util.function.CheckedSupplier.lambda$unchecked$0(CheckedSupplier.java:34) >> at >> java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590) >> at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40) >> at >> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:44) >> at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) >> at >> akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) >> at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) >> at >> akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> 在 2023-03-10 11:00:58，"Weihua Hu" <huweihua....@gmail.com> 写道： >> >Hi >> > >> >一般来说只是 YARN 集群异常停电不会影响已经完成的历史 Checkpoint（最后一次 Checkpoint 可能会写 hdfs 异常） >> > >> >有更详细的 JobManager 日志吗？可以先确认下 Flink 在恢复时检索到了多少个 completedCheckpoint >> >以及最终尝试从哪一次 cp 恢复的。 >> > >> >也可以尝试按照 Yanfei 所说指定历史的 cp 作为 savepoint 恢复 >> > >> > >> >Best, >> >Weihua >> > >> > >> >On Fri, Mar 10, 2023 at 10:38 AM guanyq <dlgua...@163.com> wrote: >> > >> >> 没有开启增量chk >> >> 文件损坏是看了启动日志，启动日志尝试从10个chk启动，但是都因为以下块损坏启动失败了 >> >> 错误日志为： >> >> >> >> java.io.IOException: Got error, status message opReadBlock >> >> BP-1003103929-192.168.200.11-1668473836936:blk_1301252639_227512278 >> >> received exception >> >> org.apache.hadoop.hdfs.server.datanode.CorruptMetaHeaderException: >> >> The meta file length 0 is less than the expected length 7, for >> >> OP_READ_BLOCK, self=/ip:45534, remote=/ip:9866, >> >> for file >> >> >> /tmp/flink/ha/application_1678102326043_0007/completedCheckpoint58755403e33a, >> >> for pool BP-1003103929-192.168.200.11-1668473836936 block >> >> 1301252639_227512278 >> >> at >> >> >> org.apache.hadoop.hdfs.protocol.datatransfer.DataTransferProtoUtil.checkBlockOpStatus(DataTransferProtoUtil.java:142) >> >> at >> >> >> org.apache.hadoop.hdfs.RemoteBlockReader2.checkSuccess(RemoteBlockReader2.java:456) >> >> at >> >> >> org.apache.hadoop.hdfs.RemoteBlockReader2.newBlockReader(RemoteBlockReader2.java:424) >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> 在 2023-03-10 10:26:11，"Yanfei Lei" <fredia...@gmail.com> 写道： >> >> >Hi 可以尝试去flink配置的checkpoint dir下面去找一找历史chk-x文件夹，如果能找到历史的chk-x，可以尝试手工指定 >> >> chk重启[1]。 >> >> > >> >> >> flink任务是10个checkpoint，每个checkpoint间隔5秒，如果突然停电，为什么所有的checkpoint都损坏了。 >> >> >> >> >> >请问作业开启增量checkpoint了吗？在开启了增量的情况下，如果是比较早的一个checkpoint的文件损坏了，会影响后续基于它进行增量的checkpoint。 >> >> > >> >> >> checkpoint落盘的机制，这个应该和hdfs写入有关系，flink任务checkpoint成功，但是hdfs却没有落盘。 >> >> >是观察到checkpoint dir下面没有文件吗？ >> >> > >> >> >[1] >> >> >> https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/ops/state/savepoints/#resuming-from-savepoints >> >> > >> >> >guanyq <dlgua...@163.com> 于2023年3月10日周五 08:58写道： >> >> >> >> >> >> 目前也想着用savepoint处理异常停电的问题 >> >> >> 但是我这面还有个疑问： >> >> >> flink任务是10个checkpoint，每个checkpoint间隔5秒，如果突然停电，为什么所有的checkpoint都损坏了。 >> >> >> 就很奇怪是不是10个checkpoint都没落盘导致的。 >> >> >> 想问下： >> >> >> checkpoint落盘的机制，这个应该和hdfs写入有关系，flink任务checkpoint成功，但是hdfs却没有落盘。 >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> 在 2023-03-10 08:47:11，"Shammon FY" <zjur...@gmail.com> 写道： >> >> >> >Hi >> >> >> > >> >> >> >我觉得Flink >> >> >> >> >> >> >作业恢复失败时，作业本身很难确定失败是checkpoint的文件块损坏之类的原因。如果你的作业做过savepoint，可以尝试从指定的savepoint恢复作业 >> >> >> > >> >> >> >Best, >> >> >> >Shammon >> >> >> > >> >> >> >On Thu, Mar 9, 2023 at 10:06 PM guanyq <dlgua...@163.com> wrote: >> >> >> > >> >> >> >> 前提 >> >> >> >> 1.flink配置了高可用 >> >> >> >> 2.flink配置checkpoint数为10 >> >> >> >> 3.yarn集群配置了任务恢复 >> >> >> >> 疑问 >> >> >> >> >> yarn集群停电重启后，恢复flink任务时，如果最近的checkpoint由于停电导致块损坏，是否会尝试从其他checkpoint启动 >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> > >> >> > >> >> > >> >> >-- >> >> >Best, >> >> >Yanfei >> >> >>
