jjj-n opened a new issue, #12924:
URL: https://github.com/apache/seatunnel/issues/12924
### Search before asking
- [x] I searched the related open/closed issues and PRs before reporting. No
matching atomic-publication fix was found within those searches (2026-10-20).
### What happened
The E2E restore path copies connector JARs directly over the formal paths
referenced by an already-running SeaTunnel service. A concurrent `GET
/job-info/{jobId}` can deserialize a job DAG while the LocalFile connector
archive is only partially written. A controlled copy-window regression observed
HTTP 500 and a `LocalFileSink` class-loading failure, although the complete
input JAR contains the class and the job can subsequently restore.
Expected: the formal connector path should continue exposing the previous
complete archive until its complete replacement is available. A failed copy
should retain that previous archive. This report concerns E2E artifact
publication; no checkpoint barrier or production classloader change is proposed.
### SeaTunnel Version
Unreleased `3.0.1-SNAPSHOT`, fixed upstream dev commit
`1ce2d781566564d5bd603710e7442bd88c531414`. The same dev SHA was observed again
before filing this report. Released versions were not tested.
### SeaTunnel Config
Existing E2E fixture
`savepoint-restore/stream_checkpointable_sequence_to_localfile.conf` (uses the
E2E-only CheckpointableSequenceSource test plugin):
```hocon
env {
parallelism = 1
job.mode = "STREAMING"
checkpoint.interval = 3000
checkpoint.retain-after-job-cancelled = true
}
source {
CheckpointableSequenceSource {
start_offset = 0
end_offset = 1000000
split_num = 1
emit_interval_ms = 50
records_per_poll = 1
}
}
transform {
}
sink {
LocalFile {
path = "/tmp/seatunnel_mnt/savepoint-restore/sinkfile/"
row_delimiter = "\n"
file_name_expression = "offset_${transactionId}"
file_format_type = "text"
is_enable_transaction = true
}
}
```
### Running Command
The observed red/green runs used an external JUnit ConsoleLauncher in an
isolated Docker daemon with fixed runtime artifacts, Java 11, and freshly
compiled test sources. They did not use Maven Failsafe. Those experimental
launchers and logs remain local; the commands below are a suggested
repository-standard reproduction, not a claim of an executed Failsafe run.
The controlled sequence is: run the fixture, wait for a completed
checkpoint, create a savepoint, and enter the actual restore helper. Expose
only the first 64 KiB of the LocalFile connector at the copy target and issue a
job-info request before completing that copy. The old direct copy exposes a
truncated formal archive; staging at a unique same-directory non-JAR path
avoids that exposure.
The strengthened current regression checks the formal old SHA before
requesting REST. Applied alone to the unpatched base, it is expected to reject
direct overwrite at that SHA assertion before a new HTTP500 observation. That
proposed reproduction is distinct from the archived earlier HTTP500 run and has
not been newly executed through Failsafe.
### Error Exception
Observed in the archived red service log:
```text
com.hazelcast.nio.serialization.HazelcastSerializationException:
java.lang.ClassNotFoundException:
org.apache.seatunnel.connectors.seatunnel.file.local.sink.LocalFileSink
Caused by: java.lang.ClassNotFoundException:
org.apache.seatunnel.connectors.seatunnel.file.local.sink.LocalFileSink
```
### Zeta or Flink or Spark Version
Zeta built from the dev SHA above.
### Java or Scala Version
Java 11 for the isolated Linux service and IT runtime. Directed unit tests
were also validated on Windows Java 17.
### Suggested fix and local validation
Copy each selected connector into a unique same-directory temporary name
that does not end in `.jar`, then publish the completed file with checked `mv
-f`. Use POSIX container paths on every host. If copying or publication fails,
attempt to remove only that temporary file, retain the primary failure and
interrupt status, and leave the old formal file intact.
The independent local patch passes 10 directed UTs, an affected 66-module
quick verify, and five serial external-launcher rounds (25 tests: 10 recovery
and 15 local REST fault checks; zero failures/skips). All 587 job-info checks
during publication returned 200; the corresponding REST ERROR count was zero.
The original recovery/checkpoint/offset assertions are unchanged. These results
do not establish whole-repository coverage, Maven Failsafe coverage, remote CI,
or long-term stability.
Related context only: #12316 fixes source-to-barrier handoff, #12271 handles
task-generation classloader ownership, #11514 concerns NaN metric
serialization, and closed/unmerged #8095 reduces repeated connector copying by
filename. None is the issue this patch would close.
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of
Conduct](https://www.apache.org/foundation/policies/conduct).
### Affected code and reproduction outline
The affected helper is
[`ContainerUtil.copyConnectorJarToContainer`](https://github.com/apache/seatunnel/blob/1ce2d781566564d5bd603710e7442bd88c531414/seatunnel-e2e/seatunnel-e2e-common/src/test/java/org/apache/seatunnel/e2e/common/util/ContainerUtil.java#L93-L119).
Its `copyFileToContainer` destination is the formal connector path. The
existing recovery fixture is
[`SavepointRestoreIT`](https://github.com/apache/seatunnel/blob/1ce2d781566564d5bd603710e7442bd88c531414/seatunnel-e2e/seatunnel-engine-e2e/connector-seatunnel-e2e-base/src/test/java/org/apache/seatunnel/engine/e2e/SavepointRestoreIT.java).
To make the copy window deterministic in a test wrapper around the actual
helper:
1. Run the existing savepoint fixture, wait for a completed checkpoint,
save/cancel, and enter the restore path while the service stays alive.
2. Intercept the LocalFile connector copy and transfer only its first 64 KiB
to the destination requested by the helper. Pause before completing the
transfer. The original complete archive contains `LocalFileSink`.
3. While paused, query `GET /job-info/{jobId}`. With direct overwrite, the
requested destination is the formal JAR and can now be incomplete. The
historical controlled run returned HTTP 500 with the exception above.
4. Complete the copy and continue restoration. The recovery assertions can
pass even though a REST request failed during publication.
5. Repeat with a unique staged non-JAR destination followed by
same-directory publication. Assert the formal old SHA remains unchanged during
the partial copy and after an injected failed copy, then assert that successful
publication exposes the complete, different replacement SHA and that
REST/recovery pass.
An ordinary run may miss this short race; the controlled copy-window
interception is necessary for a deterministic regression. The regression
additions and fix are currently local and are not yet available in a PR.
Suggested Maven command after adding that regression (**not executed here**):
```bash
./mvnw -B -T 1 -pl :connector-seatunnel-e2e-base -am \
-Dit.test=SavepointRestoreIT -Dfailsafe.failIfNoSpecifiedTests=false \
-DskipTests=false -DskipUT=true -DskipIT=false \
-Dskip.ui=true -Dlicense.skipAddThirdParty=true verify
```
The planned supplemental Failsafe run was explicitly skipped; no Failsafe
success or remote CI result is claimed.
### Completed local external-launcher rounds
| Round | Tests | Failures/errors/skips | Publication-window job-info HTTP
200 | Corresponding REST ERROR lines |
| --- | ---: | --- | ---: | ---: |
| 1 | 5 | 0/0/0 | 53 | 0 |
| 2 | 5 | 0/0/0 | 66 | 0 |
| 3 | 5 | 0/0/0 | 68 | 0 |
| 4 | 5 | 0/0/0 | 55 | 0 |
| 5 | 5 | 0/0/0 | 345 | 0 |
Each round includes two recovery tests and three local REST fault tests.
These finite runs support the local patch's behavior for the recorded inputs;
they do not prove long-term stability or full-repository integration coverage.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
