aglinxinyuan opened a new issue, #7907:
URL: https://github.com/apache/texera/issues/7907
### What happened?
Worker checkpoint persistence cannot succeed in the shipped configuration.
`FinalizeCheckpointHandler` always throws `NotSerializableException` at the
point it writes the checkpoint out, so no worker checkpoint is ever persisted.
The chain:
1. `FinalizeCheckpointHandler.scala:69` does `writer.writeRecord(chkpt)`,
where `chkpt` is a `CheckpointState`.
2. `SequentialRecordWriter` serializes through `AmberRuntime.serde`.
3. `common/config/src/main/resources/cluster.conf:44-51` sets
`allow-java-serialization = off` and binds only:
```
serialization-bindings {
"java.io.Serializable" = kryo
"java.lang.Throwable" = pekko-misc
}
```
4.
`amber/src/main/scala/org/apache/texera/amber/engine/common/CheckpointState.scala:24`
declares `class CheckpointState {` — it implements neither
`java.io.Serializable` nor `Throwable`, and no other binding matches it.
So there is no serializer for the type being written.
`AmberRuntime.serde.serialize(new CheckpointState())` returns a `Failure`,
which was confirmed by asserting it directly in a throwaway probe.
The consequence is that `FinalizeCheckpointHandler.scala:70-73` —
`writer.flush()`, `writer.close()`, the size log and the returned
`chkpt.size()` — are unreachable in production. The handler's estimate-only
branch is unaffected; it is only the real write path that cannot complete.
Note this is *already documented as a limitation* in
`FinalizeCheckpointHandlerSpec`'s scaladoc, which explains why that spec does
not assert the write. It appears never to have been filed as a defect, which is
why this issue exists.
### How to reproduce?
Two ways.
**Directly, from a scratch test in the `WorkflowExecutionService` project:**
```scala
import org.apache.texera.amber.engine.common.{AmberRuntime, CheckpointState}
assert(AmberRuntime.serde.serialize(new CheckpointState()).isFailure)
```
That assertion passes today.
**Through the handler**, which is how it manifests: drive
`prepareCheckpoint` with `estimationOnly = false` so a `CheckpointState` is
registered under the checkpoint id, then call `finalizeCheckpoint` with a real
`writeTo` destination. The call fails inside `writeRecord` rather than
returning a `FinalizeCheckpointResponse`.
A suggested fix is to give `CheckpointState` a serialization binding — the
smallest change being to have it extend `java.io.Serializable`, which the
existing kryo binding then covers. Whether that is the right choice depends on
whether kryo is intended to own this type; adding an explicit binding for
`CheckpointState` would be the more deliberate alternative. Either way it is a
production change, so it is reported here rather than worked around in tests.
Worth flagging for whoever picks this up: a test can reach the write path
today only by installing a `CheckpointState with java.io.Serializable` subclass
into `dp.ecmManager.checkpoints`. That double should not be mistaken for
evidence that the production path works, and a test asserting "the write
throws" would cement the defect rather than catch it.
### Version/Branch
main
### Commit Hash (Optional)
4786d6c661
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
