goutamadwant opened a new pull request, #12689:
URL: https://github.com/apache/seatunnel/pull/12689
### Purpose of this pull request
<!-- Describe the purpose of this pull request. -->
Related to #9870.
Concurrent SFTP writers can observe the same directory as missing and
attempt to create it. After one writer succeeds, another receives an SFTP
error, which currently fails directory creation and can propagate into an
aggregate commit failure.
This PR:
- Rechecks the target after a failed `mkdir` and accepts the result only
when it is a directory.
- Creates directories using absolute paths without changing the pooled SFTP
channel's working directory.
- Preserves the original SFTP exception when directory creation still fails.
- Adds deterministic unit coverage and a real-server concurrency regression
in the existing `SftpFileIT`.
- Adds the existing shaded Hadoop artifact as a test-scoped dependency for
the integration test; production dependencies are unchanged.
The directory-creation race was reproduced on current `dev`. The reporter's
complete MySQL workload was not reproduced, so this PR does not claim to
resolve every possible cause of the reported aggregate commit error.
### Does this PR introduce _any_ user-facing change?
<!--
Note that it means *any* user-facing change including all aspects such as
the documentation fix.
If yes, please clarify the previous behavior and the change this PR proposes.
-->
Yes, a bug fix:
- Before: a concurrent writer creating the same directory could cause an
otherwise valid SFTP write or commit to fail.
- After: directory creation succeeds when the target is confirmed to be a
directory, including when another writer created it concurrently.
- Conflicting files and failures where the directory remains absent still
fail.
No configuration, public API, authentication, filename, or overwrite-policy
changes. No migration is required. Existing documented usage remains unchanged.
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some
test cases that check the changes thoroughly including negative and positive
cases if possible.
If it was tested in a way different from regular unit tests, please clarify
how you tested step by step.
-->
- All 22 SFTP unit tests passed on Java 8 and Java 11.
- Added coverage for concurrent directory/file creation, permission
failures, existing directories/files, relative paths, and nested parent
creation.
- The new real-server integration regression failed before the fix and
passed after it on Java 8 and Java 11. It uses eight clients over three rounds
and verifies every output file's contents.
- A separate real-server probe with 20 clients reproduced 95/100
directory-creation failures and 57/60 aggregate-commit failures before the fix.
After the fix, both Java versions completed with zero failures and verified all
committed file contents.
- Formatting and scoped Maven verification passed.
- Broader shared-file tests: 324 passed; 12 errored because the bundled
Snappy library has no macOS ARM64 native binary. The full distributed-engine
test matrix was not run.
Run the connector unit tests:
```shell
./mvnw -pl seatunnel-connectors-v2/connector-file/connector-file-sftp \
-DskipIT test
```
After building/installing the required reactor dependencies, run the
real-server regression with Docker available:
```shell
./mvnw -pl seatunnel-e2e/seatunnel-connector-v2-e2e/connector-file-sftp-e2e \
-DskipUT -DskipIT=false \
'-Dit.test=SftpFileIT#testConcurrentDirectoryCreationAndWrites' verify
```
### Check list
* [x] If any new Jar binary package adding in your PR, please add License
Notice according
[New License
Guide](https://github.com/apache/seatunnel/blob/dev/docs/en/developer/new-license.md).
No new external binary package; the test reuses the existing shaded Hadoop
artifact.
* [x] If necessary, please update the documentation to describe the new
feature.
Not applicable: no new feature or configuration.
* [x] If necessary, please update `incompatible-changes.md` to describe the
incompatibility caused by this PR.
Not applicable: no incompatible change.
* [x] If you are contributing the connector code, please check that the
following files are updated:
1. `plugin-mapping.properties` — unchanged; existing connector.
2. `seatunnel-dist/pom.xml` — unchanged; no distribution change.
3. `label-scope-conf.yml` — unchanged; existing module coverage.
4. `seatunnel-e2e` — regression added to the existing SFTP integration
test.
5. Connector `plugin_config` — unchanged; no configuration change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
