potiuk commented on code in PR #74943:
URL: https://github.com/apache/airflow/pull/74943#discussion_r4394880406
##########
task-sdk/src/airflow/sdk/io/path.py:
##########
@@ -463,7 +463,7 @@ def move(self, path: str | ObjectStoragePath, recursive:
bool = False, **kwargs)
# non-local copy
self.copy(path, recursive=recursive, **kwargs)
- self.unlink()
+ self.fs.rm(self.path, recursive=recursive)
Review Comment:
This changes `move` from "fails on a directory" to "recursively deletes the
source" whenever `samestore()` is False. `samestore()` only compares protocol
and `conn_id`, so two connections to the same bucket count as different stores.
For example:
```python
ObjectStoragePath("s3://conn_a@bucket/data").move("s3://conn_b@bucket/data/archive",
recursive=True)
```
This copies into `data/archive/...` and then `rm("bucket/data",
recursive=True)` deletes the new copy too. The same happens when both paths
resolve to the same key. Before this change, `unlink()` raised and nothing was
lost. Could we refuse a destination that is the source or inside it, comparing
bucket and key, before deleting? A test for that case would help too.
---
Drafted-by: Claude Code (Opus 5.5); reviewed by @potiuk before posting
##########
task-sdk/tests/task_sdk/io/test_path.py:
##########
@@ -349,6 +363,19 @@ def test_bucket_key_protocol(self):
assert o.key == key
assert o.protocol == protocol
+ @mock.patch.object(_FakeRemoteFileSystem, "mkdir", autospec=True)
+ def test_copy_dir_to_new_remote_dir(self, mock_mkdir, fake_remote_stores):
+ bucket = uuid.uuid4().hex
+ src = ObjectStoragePath(f"ffs://{bucket}/src")
Review Comment:
The source key here has no prefix, so the test can't tell `dst /
src_obj.key` (full key) apart from `dst / relative_path`. Could you add a case
with a nested source such as `ffs://{bucket}/x/src`, asserting the documented
layout (`dst/x/src/a.txt`)? That pins down the "same key as the source"
behaviour described in the `copy()` docstring.
---
Drafted-by: Claude Code (Opus 5.5); reviewed by @potiuk before posting
##########
task-sdk/src/airflow/sdk/io/path.py:
##########
@@ -463,7 +463,7 @@ def move(self, path: str | ObjectStoragePath, recursive:
bool = False, **kwargs)
# non-local copy
self.copy(path, recursive=recursive, **kwargs)
Review Comment:
When the source is a remote directory and `recursive=False`, the
remote-to-remote branch of `copy()` ignores `recursive` and copies the whole
tree. Then `fs.rm(self.path, recursive=False)` either raises or does nothing,
so the caller ends up with a duplicated tree and the source still in place.
Should `move` raise up front when the source is a directory and `recursive` is
False?
---
Drafted-by: Claude Code (Opus 5.5); reviewed by @potiuk before posting
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
