[
https://issues.apache.org/jira/browse/NIFI-12468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17792997#comment-17792997
]
David Handermann commented on NIFI-12468:
-----------------------------------------
The Xodus Environment Implementation looks like it writes some additional log
statements related to transaction commit failures.
https://github.com/JetBrains/xodus/blob/5433bb950a38ebff5a086c359a80f0612c2df260/environment/src/main/java/jetbrains/exodus/env/EnvironmentImpl.java#L968
According to the message, closing and opening the environment starts a recovery
process. It could be possible to catch and evaluate exception types, but a poor
implementation could result in error loops.
If this type of scenario occurred in normal operations, restarting NiFi could
be the appropriate option. That may in fact be safer than attempting to retry
for a certain number of times.
> Xodus Flow Configuration History does not recover from exception
> ----------------------------------------------------------------
>
> Key: NIFI-12468
> URL: https://issues.apache.org/jira/browse/NIFI-12468
> Project: Apache NiFi
> Issue Type: Bug
> Affects Versions: 2.0.0-M1
> Reporter: Michael W Moser
> Priority: Major
>
> I caused Xodus to timeout writing a transation to flow configuration history.
> Thereafter, every operation that would update flow configuration history, or
> trying to view flow configuration history, continues to throw an exception.
> The UI shows "An unexpected error has occurred. Please check the logs for
> additional details" based on a 500 response from the REST API
> /nifi-api/flow/history
> I had to restart NiFi to recover. Hopefully, this code can be improved to
> detect exceptions and recover on its own.
> I caused the initial problem by setting a breakpoint while a debugger was
> attached to NiFi. I dragged a new Process Group onto the graph from a flow
> definition file and let it sit at the breakpoint for a while.
> {noformat}
> 2023-12-04 16:32:00,448 ERROR [NiFi Web Server-466]
> jetbrains.exodus.env.EnvironmentImpl Failed to flush transaction
> java.io.IOException: Stream Closed
> at java.base/java.io.RandomAccessFile.length0(Native Method)
> at java.base/java.io.RandomAccessFile.length(RandomAccessFile.java:653)
> at jetbrains.exodus.io.FileDataWriter.ensureFile(FileDataWriter.kt:172)
> at jetbrains.exodus.io.FileDataWriter.write(FileDataWriter.kt:59)
> at
> jetbrains.exodus.log.BufferedDataWriter.writePage(BufferedDataWriter.java:264)
> at
> jetbrains.exodus.log.BufferedDataWriter.flush(BufferedDataWriter.java:176)
> at jetbrains.exodus.log.Log.flush(Log.kt:631)
> at jetbrains.exodus.log.Log.flush$default(Log.kt:630)
> at jetbrains.exodus.log.Log.flush(Log.kt)
> at
> jetbrains.exodus.env.EnvironmentImpl.flushTransaction(EnvironmentImpl.java:731)
> at
> jetbrains.exodus.env.EnvironmentImpl.commitTransaction(EnvironmentImpl.java:684)
> at
> jetbrains.exodus.env.ReadWriteTransaction.commit(ReadWriteTransaction.java:96)
> at
> jetbrains.exodus.entitystore.PersistentStoreTransaction.doCommit(PersistentStoreTransaction.java:192)
> at
> jetbrains.exodus.entitystore.PersistentStoreTransaction.commit(PersistentStoreTransaction.java:180)
> at
> jetbrains.exodus.entitystore.PersistentEntityStoreImpl.executeInExclusiveTransaction(PersistentEntityStoreImpl.java:720)
> at
> org.apache.nifi.admin.service.EntityStoreAuditService.addActions(EntityStoreAuditService.java:116)
> at org.apache.nifi.audit.NiFiAuditor.saveActions(NiFiAuditor.java:66)
> at org.apache.nifi.audit.NiFiAuditor.saveAction(NiFiAuditor.java:53)
> at
> org.apache.nifi.audit.ProcessGroupAuditor.saveUpdateAction(ProcessGroupAuditor.java:359)
> at
> org.apache.nifi.audit.ProcessGroupAuditor.updateProcessGroupFlowAdvice(ProcessGroupAuditor.java:305)
> at
> java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
> {noformat}
> Afterwards, any attempt to read or write from Xodus produces an exception
> similar to this:
> {noformat}
> 2023-12-04 16:32:53,111 WARN [NiFi Web Server-371]
> o.apache.nifi.audit.ProcessGroupAuditor Unable to record actions:
> jetbrains.exodus.ExodusException: Environment is inoperative
> at
> jetbrains.exodus.ExodusException.toExodusException(ExodusException.java:49)
> at
> jetbrains.exodus.env.EnvironmentImpl.checkIsOperative(EnvironmentImpl.java:1069)
> at
> jetbrains.exodus.env.EnvironmentImpl.beginTransaction(EnvironmentImpl.java:613)
> at
> jetbrains.exodus.env.EnvironmentImpl.beginExclusiveTransaction(EnvironmentImpl.java:264)
> at
> jetbrains.exodus.entitystore.PersistentStoreTransaction.<init>(PersistentStoreTransaction.java:130)
> at
> jetbrains.exodus.entitystore.PersistentEntityStoreImpl.beginExclusiveTransaction(PersistentEntityStoreImpl.java:539)
> at
> jetbrains.exodus.entitystore.PersistentEntityStoreImpl.executeInExclusiveTransaction(PersistentEntityStoreImpl.java:717)
> at
> org.apache.nifi.admin.service.EntityStoreAuditService.addActions(EntityStoreAuditService.java:116)
> at org.apache.nifi.audit.NiFiAuditor.saveActions(NiFiAuditor.java:66)
> at org.apache.nifi.audit.NiFiAuditor.saveAction(NiFiAuditor.java:53)
> at
> org.apache.nifi.audit.ProcessGroupAuditor.saveUpdateAction(ProcessGroupAuditor.java:359)
> at
> org.apache.nifi.audit.ProcessGroupAuditor.updateProcessGroupFlowAdvice(ProcessGroupAuditor.java:305)
> at
> java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
> ...
> ...
> Caused by: java.io.IOException: Stream Closed
> at java.base/java.io.RandomAccessFile.length0(Native Method)
> at java.base/java.io.RandomAccessFile.length(RandomAccessFile.java:653)
> at jetbrains.exodus.io.FileDataWriter.ensureFile(FileDataWriter.kt:172)
> at jetbrains.exodus.io.FileDataWriter.write(FileDataWriter.kt:59)
> at
> jetbrains.exodus.log.BufferedDataWriter.writePage(BufferedDataWriter.java:264)
> at
> jetbrains.exodus.log.BufferedDataWriter.flush(BufferedDataWriter.java:176)
> at jetbrains.exodus.log.Log.flush(Log.kt:631)
> at jetbrains.exodus.log.Log.flush$default(Log.kt:630)
> at jetbrains.exodus.log.Log.flush(Log.kt)
> at
> jetbrains.exodus.env.EnvironmentImpl.flushTransaction(EnvironmentImpl.java:731)
> at
> jetbrains.exodus.env.EnvironmentImpl.commitTransaction(EnvironmentImpl.java:684)
> at
> jetbrains.exodus.env.ReadWriteTransaction.commit(ReadWriteTransaction.java:96)
> at
> jetbrains.exodus.entitystore.PersistentStoreTransaction.doCommit(PersistentStoreTransaction.java:192)
> at
> jetbrains.exodus.entitystore.PersistentStoreTransaction.commit(PersistentStoreTransaction.java:180)
> at
> jetbrains.exodus.entitystore.PersistentEntityStoreImpl.executeInExclusiveTransaction(PersistentEntityStoreImpl.java:720)
> at
> org.apache.nifi.admin.service.EntityStoreAuditService.addActions(EntityStoreAuditService.java:116)
> at org.apache.nifi.audit.NiFiAuditor.saveActions(NiFiAuditor.java:66)
> at org.apache.nifi.audit.NiFiAuditor.saveAction(NiFiAuditor.java:53)
> at
> org.apache.nifi.audit.ProcessGroupAuditor.saveUpdateAction(ProcessGroupAuditor.java:359)
> at
> org.apache.nifi.audit.ProcessGroupAuditor.updateProcessGroupFlowAdvice(ProcessGroupAuditor.java:305)
> at
> java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
