juergbi commented on code in PR #1793:
URL: https://github.com/apache/buildstream/pull/1793#discussion_r1068453722
##########
src/buildstream/utils.py:
##########
@@ -357,6 +358,21 @@ def sha256sum(filename: str) -> str:
return h.hexdigest()
+def copy_file_range(src, dest):
+ if not _USE_CP_FILE_RANGE:
+ return False
+ with open(src, "rb") as src_file, open(dest, "wb") as dest_file:
+ num_bytes = os.fstat(src_file.fileno()).st_size
+ while num_bytes > 0:
+ try:
+ num_bytes -= os.copy_file_range(src_file.fileno(),
dest_file.fileno(), num_bytes)
+ except OSError as error:
+ if error.errno in (errno.ENOSYS, errno.EXDEV):
+ return False
Review Comment:
Should we set `_USE_CP_FILE_RANGE` to `False` (using `global`) to avoid the
overhead of repeatedly failing when copying a large number of files?
##########
src/buildstream/utils.py:
##########
@@ -357,6 +358,21 @@ def sha256sum(filename: str) -> str:
return h.hexdigest()
+def copy_file_range(src, dest):
+ if not _USE_CP_FILE_RANGE:
+ return False
+ with open(src, "rb") as src_file, open(dest, "wb") as dest_file:
+ num_bytes = os.fstat(src_file.fileno()).st_size
+ while num_bytes > 0:
+ try:
+ num_bytes -= os.copy_file_range(src_file.fileno(),
dest_file.fileno(), num_bytes)
Review Comment:
We should detect `os.copy_file_range()` returning zero, indicating end of
file. This can only happen if the source file was truncated but it would lead
to an infinite loop.
##########
src/buildstream/utils.py:
##########
@@ -381,9 +397,14 @@ def safe_copy(src: str, dest: str, *, copystat: bool =
True, result: Optional[Fi
raise UtilError("Failed to remove destination file '{}':
{}".format(dest, e)) from e
try:
- shutil.copyfile(src, dest)
+ ret = copy_file_range(src, dest)
except (OSError, shutil.Error) as e:
raise UtilError("Failed to copy '{} -> {}': {}".format(src, dest, e))
from e
+ if not ret:
+ try:
+ shutil.copyfile(src, dest)
+ except (OSError, shutil.Error) as e:
+ raise UtilError("Failed to copy '{} -> {}': {}".format(src, dest,
e)) from e
Review Comment:
Let's merge the two identical `except` clauses. I first wanted to suggest to
keep the separate `except` clauses but use different error messages. However,
if e.g. `open()` fails in `copy_file_range()`, raising a
`copy_file_range()`-specific error message may be confusing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
