[gentoo-dev] Packages up for grabs

[Michael Palimaka]

Due to a retiring proxied maintainer, these packages are now up for grabs:

dev-python/python-gammu
net-irc/inspircd

Re: [gentoo-portage-dev] [PATCH v2] movefile: support in-kernel file copying on Linux (bug 607868)

[Zac Medico]

On 03/03/2017 12:45 PM, Michał Górny wrote:
> W dniu 02.03.2017, czw o godzinie 17∶09 -0800, użytkownik Zac Medico
>> +def copyfile(src, dst):
>> +"""
>> +Copy the contents (no metadata) of the file named src to a file
>> +named dst.
>> +
>> +If possible, copying is done within the kernel, and uses
>> +"copy acceleration" techniques (such as reflinks). This also
>> +supports sparse files.
>> +
>> +@param src: path of source file
>> +@type src: str
>> +@param dst: path of destination file
>> +@type dst: str
>> +"""
>> +global _copyfile
>> +
>> +if _copyfile is None:
>> +if _file_copy is None:
>> +_copyfile = shutil.copyfile
>> +else:
>> +_copyfile = _optimized_copyfile
> 
> I dare say the caching logic takes more time than the actual logic.
> However, this could be made even simpler -- you could just check
> the condition in global scope and set _copyfile value there.

In v3 it's like this:

if _file_copy is None:
copyfile = shutil.copyfile
else:
copyfile = _optimized_copyfile

>> +static off_t
>> +do_lseek(int fd_out, int fd_in, loff_t *off_out) {
> 
> Maybe name it do_lseek_data() to make the goal clearer? Some function
> docs in comments would also be helpful.

Renamed to do_lseek_data v3.

>> +
>> +if (!(error && errno == EINTR && PyErr_CheckSignals() == 0))
>> +eintr_retry = 0;
> 
> To be honest, I can't do it but it'd be really a good idea if someone
> could analyze all the code paths where the code above could give 'error
> = 1' and ensure that:
> 
> a. there is no case when we error=1 and leave errno unmodified/unset
> (i.e. can accidentally leave earlier EINTR there),

In v3, I set error = errno immediately after every function call that
fails. That makes it really easy to see exactly which function calls the
errno value is picked up from.

> b. we can correctly recover from any EINTR, i.e. if EINTR causes some
> function to be interrupted in the middle, our next iteration starts
> safely.

The state is mostly tracked by the offset_out variable, which makes it
hard to screw up. We just have to watch that variable carefully, and
make sure we don't corrupt its state. I added in a couple of extra lseek
calls to ensure that the fd_in file offset is always in sync with the
offset_out variable when we resume from EINTR.

>> +}
>> +
>> +free(buf);
> 
> if (buf != NULL) ?

The free(3) documentation says we're allowed to pass in a null pointer,
but I've added a buf != NULL check for readability.
-- 
Thanks,
Zac

[gentoo-dev] Last rites: x11-misc/oroborus-desklaunch

[Michael Palimaka]

# Michael Palimaka  (03 Mar 2017)
# Segfaults at runtime. Dead upstream. Unmaintained.
# Masked for removal in 30 days. Bug #611406.
x11-misc/oroborus-desklaunch

[gentoo-dev] [warning] the bug queue has 81 bugs

[Alex Alexander]

Our bug queue has 81 bugs!

If you have some spare time, please help assign/sort a few bugs.

To view the bug queue, click here: http://bit.ly/m8PQS5

Thanks!

Re: [gentoo-dev] [PATCHES] python-r1, add integrity checks for redefined control vars

[Tim Harder]

I've attached another simple patch that I don't think was fixed in your
changeset to stop the 'impl' var from _python_obtain_impls() in
python-r1.eclass from leaking into the environment.

Thanks,
Tim
>From 0a6174036e5d31028e47fb5f477033fdb7b76aba Mon Sep 17 00:00:00 2001
From: Tim Harder 
Date: Fri, 3 Mar 2017 15:49:16 -0500
Subject: [PATCH] python-r1.eclass: localize variable to avoid leaking into the
 env

---
 eclass/python-r1.eclass | 1 +
 1 file changed, 1 insertion(+)

diff --git a/eclass/python-r1.eclass b/eclass/python-r1.eclass
index 929ec8fa8f..4d27881cd5 100644
--- a/eclass/python-r1.eclass
+++ b/eclass/python-r1.eclass
@@ -476,6 +476,7 @@ _python_obtain_impls() {
 
 	MULTIBUILD_VARIANTS=()
 
+	local impl
 	for impl in "${_PYTHON_SUPPORTED_IMPLS[@]}"; do
 		has "${impl}" "${PYTHON_COMPAT[@]}" && \
 		use "python_targets_${impl}" && MULTIBUILD_VARIANTS+=( "${impl}" )
-- 
2.12.0

Re: [gentoo-portage-dev] [PATCH v2] movefile: support in-kernel file copying on Linux (bug 607868)

[Michał Górny]

W dniu 02.03.2017, czw o godzinie 17∶09 -0800, użytkownik Zac Medico
napisał:
> Perform in-kernel file copying when possible, and also support
> reflinks and sparse files. If the optimized implementation
> fails at runtime, gracefully fallback to a plain read/write
> loop.
> 
> Compile-time and run-time fallbacks are implemented, so that
> any incompatiblities will be handled gracefully. For example,
> if the code is compiled on a system that supports the
> copy_file_range syscall, but at run-time an older kernel that
> does not support this syscall is detected, it will be handled
> gracefully. There are similar fallbacks for lack of lseek
> SEEK_DATA and sendfile support.
> 
> X-Gentoo-Bug: 607868
> X-Gentoo-Bug-Url: https://bugs.gentoo.org/show_bug.cgi?id=607868
> ---
> [PATCH v2] adds a native fallback that will work on any kernel,
> using a plain read/write loop.
> 
>  pym/portage/tests/util/file_copy/__init__.py  |   0
>  pym/portage/tests/util/file_copy/__test__.py  |   0
>  pym/portage/tests/util/file_copy/test_copyfile.py |  68 +
>  pym/portage/util/file_copy/__init__.py|  45 
>  pym/portage/util/movefile.py  |   4 +-
>  setup.py  |   9 +
>  src/portage_util_file_copy_reflink_linux.c| 298 
> ++
>  7 files changed, 423 insertions(+), 1 deletion(-)
>  create mode 100644 pym/portage/tests/util/file_copy/__init__.py
>  create mode 100644 pym/portage/tests/util/file_copy/__test__.py
>  create mode 100644 pym/portage/tests/util/file_copy/test_copyfile.py
>  create mode 100644 pym/portage/util/file_copy/__init__.py
>  create mode 100644 src/portage_util_file_copy_reflink_linux.c
> 
> diff --git a/pym/portage/tests/util/file_copy/__init__.py 
> b/pym/portage/tests/util/file_copy/__init__.py
> new file mode 100644
> index 000..e69de29
> diff --git a/pym/portage/tests/util/file_copy/__test__.py 
> b/pym/portage/tests/util/file_copy/__test__.py
> new file mode 100644
> index 000..e69de29
> diff --git a/pym/portage/tests/util/file_copy/test_copyfile.py 
> b/pym/portage/tests/util/file_copy/test_copyfile.py
> new file mode 100644
> index 000..94fb4d7
> --- /dev/null
> +++ b/pym/portage/tests/util/file_copy/test_copyfile.py
> @@ -0,0 +1,68 @@
> +# Copyright 2017 Gentoo Foundation
> +# Distributed under the terms of the GNU General Public License v2
> +
> +import shutil
> +import tempfile
> +
> +from portage import os
> +from portage.tests import TestCase
> +from portage.checksum import perform_md5
> +from portage.util.file_copy import copyfile
> +
> +
> +class CopyFileTestCase(TestCase):
> +
> + def testCopyFile(self):
> +
> + tempdir = tempfile.mkdtemp()
> + try:
> + src_path = os.path.join(tempdir, 'src')
> + dest_path = os.path.join(tempdir, 'dest')
> + content = b'foo'
> +
> + with open(src_path, 'wb') as f:
> + f.write(content)
> +
> + copyfile(src_path, dest_path)
> +
> + self.assertEqual(perform_md5(src_path), 
> perform_md5(dest_path))
> + finally:
> + shutil.rmtree(tempdir)
> +
> +
> +class CopyFileSparseTestCase(TestCase):
> +
> + def testCopyFileSparse(self):
> +
> + tempdir = tempfile.mkdtemp()
> + try:
> + src_path = os.path.join(tempdir, 'src')
> + dest_path = os.path.join(tempdir, 'dest')
> + content = b'foo'
> +
> + # Use seek to create some sparse blocks. Don't make 
> these
> + # files too big, in case the filesystem doesn't support
> + # sparse files.
> + with open(src_path, 'wb') as f:
> + f.write(content)
> + f.seek(2**18, 1)
> + f.write(content)
> + f.seek(2**19, 1)
> + f.write(content)
> +
> + copyfile(src_path, dest_path)
> +
> + self.assertEqual(perform_md5(src_path), 
> perform_md5(dest_path))
> +
> + # This last part of the test is expected to fail when 
> sparse
> + # copy is not implemented, so set the todo flag in order
> + # to tolerate failures.
> + self.todo = True
> +
> + # If sparse blocks were preserved, then both files 
> should
> + # consume the same number of blocks.
> + self.assertEqual(
> + os.stat(src_path).st_blocks,
> + os.stat(dest_path).st_blocks)
> + finally:
> + shutil.rmtree(tempdir)
> diff --git a/pym/portage/util/file_copy/__init__.py 
> 

Re: [gentoo-portage-dev] [PATCH] movefile: support in-kernel file copying on Linux (bug 607868)

[Zac Medico]

On 03/02/2017 08:24 AM, Michał Górny wrote:
> W dniu 01.03.2017, śro o godzinie 19∶03 -0800, użytkownik Zac Medico
> napisał:
>> +def _test_optimized_copyfile():
>> +"""
>> +Test if _optimized_copyfile works. It will fail for Linux versions
>> +from 2.6.0 to 2.6.32, because sendfile does not support writing
>> +to regular files. It will also fail for Linux versions less than
>> +3.1 if portage was compiled 3.1 or later, due to missing support
>> +for lseek SEEK_DATA/SEEK_HOLE.
> 
> I don't really like the idea of relying on one-time attempt to determine
> whether a complex mechanism lacking proper fallback will work. But I
> guess you're only aiming at catching corner cases here.

In v2, there's a complete fallback mechanism in the native code, and the
_test_optimized_copyfile function has been eliminated.

>> +/* Revert SEEK_HOLE offset change, since we're going
>> + * to copy the data that comes before the hole.
>> + */
>> +if (lseek(fd_in, offset_out, SEEK_SET) < 0) {
>> +error = 1;
>> +break;
>> +}
>> +
>> +copyfunc_ret = copyfunc(fd_out,
>> +fd_in,
>> +_out,
>> +offset_hole - offset_data);
>> +
>> +if (copyfunc_ret < 0) {
>> +if ((errno == EXDEV || errno == ENOSYS) &&
>> +copyfunc == cfr_wrapper) {
>> +/* Use sendfile instead of copy_file_range for
>> + * cross-device copies, or when the copy_file_range
>> + * syscall is not available (less than Linux 4.5).
>> + */
>> +copyfunc = sendfile;
>> +copyfunc_ret = copyfunc(fd_out,
>> +fd_in,
>> +_out,
>> +offset_hole - offset_data);
>> +
>> +if (copyfunc_ret < 0) {
> 
> I think you still should have a proper fallback for sendfile() refusing
> to do its job.

In the lseek loop, sendfile should always work if lseek SEEK_DATA works,
so there should be no need to fallback for sendfile failure here. In v2,
I've added a comment about this.

>> +while (eintr_retry) {
>> +
>> +error = 0;
>> +
>> +Py_BEGIN_ALLOW_THREADS
>> +
>> +if (!stat_acquired && fstat(fd_in, ) < 0) {
>> +error = 1;
>> +}
>> +else {
>> +stat_acquired = 1;
>> +while (offset_out < sb.st_size) {
>> +copyfunc_ret = sendfile(fd_out,
>> +fd_in,
>> +_out,
>> +sb.st_size - offset_out);
>> +
>> +if (copyfunc_ret < 0) {
> 
> Likewise, especially that old kernels may refuse file-to-file copy here.

In v2, it will fallback to a plain read/write loop.
-- 
Thanks,
Zac