goutamadwant opened a new issue, #12078:
URL: https://github.com/apache/seatunnel/issues/12078
### Search before asking
- [x] I searched existing issues and pull requests for strict checksum
fallback, stream content comparison, read-boundary differences, and
`IOUtils.contentEquals`. I found no issue or pull request implementing this fix.
### What happened
File update mode can copy an unchanged file when strict checksum comparison
falls back to comparing stream content.
Both the batch read strategy and the continuous file enumerator currently
call `read(byte[])` once on each stream and treat different returned byte
counts as different content. The `InputStream` contract does not require two
streams containing identical bytes to return the same number of bytes per read.
Filesystem wrappers, buffering, object stores, and network-backed streams can
therefore expose different chunk boundaries for identical files.
A deterministic reproduction uses identical `identical-content` bytes in
both files, makes the source stream return at most 3 bytes per read, makes the
target stream return at most 5 bytes per read, and returns no filesystem
checksum. The first reads return `3` and `5`, so the current comparator returns
`false` without comparing the complete byte sequence.
Observed production-path effects:
- The batch update path includes the unchanged source file in the copy list.
- The continuous enumerator enqueues an unnecessary split for the unchanged
file.
Expected behavior: content equality must depend on the complete byte
sequence, not on how each stream partitions that sequence across reads.
### SeaTunnel Version
`dev` at `9ebb27e8f92059dee76d71029adbfbaa1c099285`.
### SeaTunnel Config
The defect is reached when update mode uses strict checksum comparison and a
source or target filesystem cannot provide a checksum:
```conf
source {
LocalFile {
path = "/path/to/source"
file_format_type = "binary"
sync_mode = "update"
target_path = "/path/to/target"
update_strategy = "strict"
compare_mode = "checksum"
}
}
```
The same comparison logic is used by the continuous multiple-table file
enumerator.
### Running Command
The deterministic reproduction invokes the existing batch and continuous
production paths with checksum-unavailable filesystem proxies whose input
streams return different maximum chunk sizes.
```shell
./mvnw -pl seatunnel-connectors-v2/connector-file/connector-file-base \
-Dskip.spotless.check=true \
-Dtest=UpdateSyncModeTest,ContinuousMultipleTableFileSourceSplitEnumeratorTest
test
```
### Error Exception
No exception is raised. Identical content is silently classified as
different, so the file is copied or enqueued again.
Before the fix, identical bytes exposed as 3-byte and 5-byte read chunks
were classified as different in both batch and continuous production paths on
Java 8 and Java 11.
### Zeta or Flink or Spark Version
Not engine-specific. The defect is in `connector-file-base` update-mode
comparison shared by file connector paths.
### Java or Scala Version
- Java 8: `1.8.0_172`
- Java 11: `11.0.19`
### Proposed fix and compatibility contract
- Use the module's existing Apache Commons IO content-equality
implementation, which compares the complete streams independently of individual
read boundaries.
- Keep constant-memory streaming behavior and try-with-resources ownership.
- Preserve the current fail-open-to-copy behavior when content comparison
raises an I/O error.
- Add batch and continuous regressions for equal content with different
chunk sizes, unequal bytes, unequal lengths, and empty streams.
- Add no configuration, public API, or dependency.
### Are you willing to submit a PR?
- [x] Yes, I am willing to submit a PR.
### Code of Conduct
- [x] I agree to follow this project's [Code of
Conduct](https://www.apache.org/foundation/policies/conduct).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
