Hello community, here is the log from the commit of package python-minio for openSUSE:Factory checked in at 2019-04-08 20:53:26 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Comparing /work/SRC/openSUSE:Factory/python-minio (Old) and /work/SRC/openSUSE:Factory/.python-minio.new.3908 (New) ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Package is "python-minio" Mon Apr 8 20:53:26 2019 rev:2 rq:691809 version:4.0.13 Changes: -------- --- /work/SRC/openSUSE:Factory/python-minio/python-minio.changes 2019-03-18 10:38:37.099485177 +0100 +++ /work/SRC/openSUSE:Factory/.python-minio.new.3908/python-minio.changes 2019-04-08 20:53:27.750565116 +0200 @@ -1,0 +2,6 @@ +Fri Apr 5 12:36:37 UTC 2019 - Tomáš Chvátal <[email protected]> + +- Update to 4.0.13: + * Add configurable part_size (#745) + +------------------------------------------------------------------- Old: ---- minio-4.0.12.tar.gz New: ---- minio-4.0.13.tar.gz ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Other differences: ------------------ ++++++ python-minio.spec ++++++ --- /var/tmp/diff_new_pack.iLPyY4/_old 2019-04-08 20:53:28.542565697 +0200 +++ /var/tmp/diff_new_pack.iLPyY4/_new 2019-04-08 20:53:28.542565697 +0200 @@ -12,20 +12,27 @@ # license that conforms to the Open Source Definition (Version 1.9) # published by the Open Source Initiative. -# Please submit bugfixes or comments via http://bugs.opensuse.org/ +# Please submit bugfixes or comments via https://bugs.opensuse.org/ +# %{?!python_module:%define python_module() python-%{**} python3-%{**}} Name: python-minio -Version: 4.0.12 +Version: 4.0.13 Release: 0 -License: Apache-2.0 Summary: Minio library for Amazon S3 compatible cloud storage -Url: https://github.com/minio/minio-py +License: Apache-2.0 Group: Development/Languages/Python +URL: https://github.com/minio/minio-py Source: https://files.pythonhosted.org/packages/source/m/minio/minio-%{version}.tar.gz -BuildRequires: python-rpm-macros BuildRequires: %{python_module setuptools} +BuildRequires: fdupes +BuildRequires: python-rpm-macros +Requires: python-certifi +Requires: python-python-dateutil +Requires: python-pytz +Requires: python-urllib3 +BuildArch: noarch # SECTION test requirements BuildRequires: %{python_module Faker} BuildRequires: %{python_module certifi} @@ -35,13 +42,6 @@ BuildRequires: %{python_module pytz} BuildRequires: %{python_module urllib3} # /SECTION -BuildRequires: fdupes -Requires: python-certifi -Requires: python-python-dateutil -Requires: python-pytz -Requires: python-urllib3 -BuildArch: noarch - %python_subpackages %description ++++++ minio-4.0.12.tar.gz -> minio-4.0.13.tar.gz ++++++ diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/minio-4.0.12/PKG-INFO new/minio-4.0.13/PKG-INFO --- old/minio-4.0.12/PKG-INFO 2019-03-13 22:31:45.000000000 +0100 +++ new/minio-4.0.13/PKG-INFO 2019-03-27 22:33:49.000000000 +0100 @@ -1,6 +1,6 @@ Metadata-Version: 1.1 Name: minio -Version: 4.0.12 +Version: 4.0.13 Summary: Minio Python Library for Amazon S3 Compatible Cloud Storage for Python Home-page: https://github.com/minio/minio-py Author: Minio, Inc. diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/minio-4.0.12/docs/API.md new/minio-4.0.13/docs/API.md --- old/minio-4.0.12/docs/API.md 2019-03-13 22:30:31.000000000 +0100 +++ new/minio-4.0.13/docs/API.md 2019-03-27 22:32:29.000000000 +0100 @@ -28,18 +28,18 @@ -|Bucket operations | Object operations| Presigned operations | Bucket policy/notification operations -|:---|:---|:---|:---| -| [`make_bucket`](#make_bucket) | [`get_object`](#get_object) | [`presigned_get_object`](#presigned_get_object) | [`get_bucket_policy`](#get_bucket_policy) | -| [`list_buckets`](#list_buckets) | [`put_object`](#put_object) | [`presigned_put_object`](#presigned_put_object) | [`set_bucket_policy`](#set_bucket_policy) | -| [`bucket_exists`](#bucket_exists) | [`copy_object`](#copy_object) | [`presigned_post_policy`](#presigned_post_policy) | [`get_bucket_notification`](#get_bucket_notification) | -| [`remove_bucket`](#remove_bucket) | [`stat_object`](#stat_object) | | [`set_bucket_notification`](#set_bucket_notification) | -| [`list_objects`](#list_objects) | [`remove_object`](#remove_object) | | [`remove_all_bucket_notification`](#remove_all_bucket_notification) | -| [`list_objects_v2`](#list_objects_v2) | [`remove_objects`](#remove_objects) | | [`listen_bucket_notification`](#listen_bucket_notification) | -| [`list_incomplete_uploads`](#list_incomplete_uploads) | [`remove_incomplete_upload`](#remove_incomplete_upload) | | | -| | [`fput_object`](#fput_object) | | | -| | [`fget_object`](#fget_object) | | | -| | [`get_partial_object`](#get_partial_object) | | | +| Bucket operations | Object operations | Presigned operations | Bucket policy/notification operations | +|:------------------------------------------------------|:--------------------------------------------------------|:--------------------------------------------------|:--------------------------------------------------------------------| +| [`make_bucket`](#make_bucket) | [`get_object`](#get_object) | [`presigned_get_object`](#presigned_get_object) | [`get_bucket_policy`](#get_bucket_policy) | +| [`list_buckets`](#list_buckets) | [`put_object`](#put_object) | [`presigned_put_object`](#presigned_put_object) | [`set_bucket_policy`](#set_bucket_policy) | +| [`bucket_exists`](#bucket_exists) | [`copy_object`](#copy_object) | [`presigned_post_policy`](#presigned_post_policy) | [`get_bucket_notification`](#get_bucket_notification) | +| [`remove_bucket`](#remove_bucket) | [`stat_object`](#stat_object) | | [`set_bucket_notification`](#set_bucket_notification) | +| [`list_objects`](#list_objects) | [`remove_object`](#remove_object) | | [`remove_all_bucket_notification`](#remove_all_bucket_notification) | +| [`list_objects_v2`](#list_objects_v2) | [`remove_objects`](#remove_objects) | | [`listen_bucket_notification`](#listen_bucket_notification) | +| [`list_incomplete_uploads`](#list_incomplete_uploads) | [`remove_incomplete_upload`](#remove_incomplete_upload) | | | +| | [`fput_object`](#fput_object) | | | +| | [`fget_object`](#fget_object) | | | +| | [`get_partial_object`](#get_partial_object) | | | ## 1. Constructor @@ -474,7 +474,7 @@ server/service, a `ResponseError` is thrown. If there are validation errors, `InvalidArgumentError` or `TypeError` may be thrown. The input configuration cannot be empty - to delete the notification -configuration on a bucket, use the `remove_all_bucket_notifications()` +configuration on a bucket, use the `remove_all_bucket_notification()` API. __Example__ @@ -544,8 +544,8 @@ print(err) ``` -<a name="remove_all_bucket_notifications"></a> -### remove_all_bucket_notifications(bucket_name) +<a name="remove_all_bucket_notification"></a> +### remove_all_bucket_notification(bucket_name) Remove all notifications configured on the bucket. @@ -563,7 +563,7 @@ ```py # Remove all the notifications config for a bucket. -minioClient.remove_all_bucket_notifications('mybucket') +minioClient.remove_all_bucket_notification('mybucket') ``` <a name="listen_bucket_notification"></a> @@ -763,29 +763,30 @@ ``` <a name="put_object"></a> -### put_object(bucket_name, object_name, data, length, content_type='application/octet-stream', metadata=None) +### put_object(bucket_name, object_name, data, length, content_type='application/octet-stream', metadata=None, progress=None, part_size=5*1024*1024) Add a new object to the object storage server. If provided metadata key is not one of the valid/supported metadata names, the metadata information is saved with prefix `X-Amz-Meta-` prepended to the original metadata key name. NOTE: Maximum object size supported by this API is 5TiB. __Parameters__ -|Param |Type |Description | -|:---|:---|:---| -|``bucket_name`` |_string_ |Name of the bucket. | -|``object_name`` |_string_ |Name of the object. | -|``data`` |_io.RawIOBase_ |Any python object implementing io.RawIOBase. | -|``length`` |_int_ |Total length of object. | -|``content_type`` |_string_ | Content type of the object. (optional, defaults to 'application/octet-stream'). | -|``metadata`` |_dict_ | Any additional metadata. (optional, defaults to None). | -|``sse`` |_dict_ |Server-Side Encryption headers (optional, defaults to None). | -|``progress`` |_subclass_of_threading_ |A progress object (optional, defaults to None). | +| Param | Type | Description | +|:-----------------|:------------------------|:--------------------------------------------------------------------------------| +| ``bucket_name`` | _string_ | Name of the bucket. | +| ``object_name`` | _string_ | Name of the object. | +| ``data`` | _io.RawIOBase_ | Any python object implementing io.RawIOBase. | +| ``length`` | _int_ | Total length of object. | +| ``content_type`` | _string_ | Content type of the object. (optional, defaults to 'application/octet-stream'). | +| ``metadata`` | _dict_ | Any additional metadata. (optional, defaults to None). | +| ``sse`` | _dict_ | Server-Side Encryption headers (optional, defaults to None). | +| ``progress`` | _subclass_of_threading_ | A progress object (optional, defaults to None). | +| ``part_size`` | _int_ | Multipart part size. | __Return Value__ -|Param |Type |Description | -|:---|:---|:---| -|``etag``|_string_ |Object etag computed by the server. | +| Param | Type | Description | +|:---------|:---------|:------------------------------------| +| ``etag`` | _string_ | Object etag computed by the server. | __Example__ @@ -814,26 +815,27 @@ ``` <a name="fput_object"></a> -### fput_object(bucket_name, object_name, file_path, content_type='application/octet-stream', metadata=None) +### fput_object(bucket_name, object_name, file_path, content_type='application/octet-stream', metadata=None, progress=None, part_size=5*1024*1024) Uploads contents from a file, `file_path`, to `object_name`. If provided metadata key is not one of the valid/supported metadata names, the metadata information is saved with prefix `X-Amz-Meta-` prepended to the original metadata key name. __Parameters__ -|Param |Type |Description | -|:---|:---|:---| -|``bucket_name`` |_string_ |Name of the bucket. | -|``object_name`` |_string_ |Name of the object. | -|``file_path`` |_string_ |Path on the local filesystem from which object data will be read. | -|``content_type`` |_string_ | Content type of the object (optional, defaults to 'application/octet-stream'). | -|``metadata`` |_dict_ | Any additional metadata (optional, defaults to None). | -|``sse`` |_dict_ |Server-Side Encryption headers (optional, defaults to None). | -|``progress`` |_subclass_of_threading_ |A progress object (optional, defaults to None). | +| Param | Type | Description | +|:-----------------|:------------------------|:-------------------------------------------------------------------------------| +| ``bucket_name`` | _string_ | Name of the bucket. | +| ``object_name`` | _string_ | Name of the object. | +| ``file_path`` | _string_ | Path on the local filesystem from which object data will be read. | +| ``content_type`` | _string_ | Content type of the object (optional, defaults to 'application/octet-stream'). | +| ``metadata`` | _dict_ | Any additional metadata (optional, defaults to None). | +| ``sse`` | _dict_ | Server-Side Encryption headers (optional, defaults to None). | +| ``progress`` | _subclass_of_threading_ | A progress object (optional, defaults to None). | +| ``part_size`` | _int_ | Multipart part size. | __Return Value__ -|Param |Type |Description | -|:---|:---|:---| -|``etag``|_string_ |Object etag computed by the server. | +| Param | Type | Description | +|:---------|:---------|:------------------------------------| +| ``etag`` | _string_ | Object etag computed by the server. | __Example__ diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/minio-4.0.12/docs/zh_CN/API.md new/minio-4.0.13/docs/zh_CN/API.md --- old/minio-4.0.12/docs/zh_CN/API.md 2019-03-13 22:30:31.000000000 +0100 +++ new/minio-4.0.13/docs/zh_CN/API.md 2019-03-27 22:32:29.000000000 +0100 @@ -439,7 +439,7 @@ * __Value__ (string) -- 指定规则适用的值。 -没有返回值。如果目标服务报错，会抛出`ResponseError`。如果有验证错误，会抛出`InvalidArgumentError`或者`TypeError`。输入参数的configuration不能为空 - 为了删除存储桶上的通知配置，参考`remove_all_bucket_notifications()` API。 +没有返回值。如果目标服务报错，会抛出`ResponseError`。如果有验证错误，会抛出`InvalidArgumentError`或者`TypeError`。输入参数的configuration不能为空 - 为了删除存储桶上的通知配置，参考`remove_all_bucket_notification()` API。 __示例__ @@ -508,8 +508,8 @@ print(err) ``` -<a name="remove_all_bucket_notifications"></a> -### remove_all_bucket_notifications(bucket_name) +<a name="remove_all_bucket_notification"></a> +### remove_all_bucket_notification(bucket_name) 删除存储桶上配置的所有通知。 @@ -526,7 +526,7 @@ ```py # Remove all the notifications config for a bucket. -minioClient.remove_all_bucket_notifications('mybucket') +minioClient.remove_all_bucket_notification('mybucket') ``` <a name="listen_bucket_notification"></a> diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/minio-4.0.12/examples/remove_all_bucket_notification.py new/minio-4.0.13/examples/remove_all_bucket_notification.py --- old/minio-4.0.12/examples/remove_all_bucket_notification.py 2019-03-13 22:30:31.000000000 +0100 +++ new/minio-4.0.13/examples/remove_all_bucket_notification.py 2019-03-27 22:32:29.000000000 +0100 @@ -26,6 +26,6 @@ try: # Remove all notification config for a bucket. - client.remove_all_bucket_notifications('my-bucketname') + client.remove_all_bucket_notification('my-bucketname') except ResponseError as err: print(err) diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/minio-4.0.12/minio/__init__.py new/minio-4.0.13/minio/__init__.py --- old/minio-4.0.12/minio/__init__.py 2019-03-13 22:30:31.000000000 +0100 +++ new/minio-4.0.13/minio/__init__.py 2019-03-27 22:32:29.000000000 +0100 @@ -29,7 +29,7 @@ __title__ = 'minio-py' __author__ = 'Minio, Inc.' -__version__ = '4.0.12' +__version__ = '4.0.13' __license__ = 'Apache 2.0' __copyright__ = 'Copyright 2015, 2016, 2017 Minio, Inc.' diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/minio-4.0.12/minio/api.py new/minio-4.0.13/minio/api.py --- old/minio-4.0.12/minio/api.py 2019-03-13 22:30:31.000000000 +0100 +++ new/minio-4.0.13/minio/api.py 2019-03-27 22:32:29.000000000 +0100 @@ -78,8 +78,11 @@ mkdir_p, dump_http, amzprefix_user_metadata, is_supported_header,is_amz_header) from .helpers import (MAX_MULTIPART_OBJECT_SIZE, + MAX_PART_SIZE, MAX_POOL_SIZE, - MIN_PART_SIZE) + MIN_PART_SIZE, + DEFAULT_PART_SIZE, + MAX_MULTIPART_COUNT) from .signer import (sign_v4, presign_v4, generate_credential_string, post_presign_signature) @@ -541,7 +544,8 @@ def fput_object(self, bucket_name, object_name, file_path, content_type='application/octet-stream', - metadata=None, sse=None, progress=None): + metadata=None, sse=None, progress=None, + part_size=DEFAULT_PART_SIZE): """ Add a new object to the cloud storage server. @@ -555,6 +559,7 @@ :param metadata: Any additional metadata to be uploaded along with your PUT request. :param progress: A progress object + :param part_size: Multipart part size :return: etag """ @@ -562,7 +567,8 @@ with open(file_path, 'rb') as file_data: file_size = os.stat(file_path).st_size return self.put_object(bucket_name, object_name, file_data, - file_size, content_type, metadata, sse, progress) + file_size, content_type, metadata, sse, + progress, part_size) def fget_object(self, bucket_name, object_name, file_path, request_headers=None, sse=None): """ @@ -751,7 +757,8 @@ def put_object(self, bucket_name, object_name, data, length, content_type='application/octet-stream', - metadata=None, sse=None, progress=None): + metadata=None, sse=None, progress=None, + part_size=DEFAULT_PART_SIZE): """ Add a new object to the cloud storage server. @@ -775,6 +782,7 @@ :param metadata: Any additional metadata to be uploaded along with your PUT request. :param progress: A progress object + :param part_size: Multipart part size :return: etag """ @@ -792,9 +800,18 @@ raise ValueError( 'Invalid input data does not implement a callable read() method') - if length > MAX_MULTIPART_OBJECT_SIZE: - raise InvalidArgumentError('Input content size is bigger ' - ' than allowed maximum of 5TiB.') + if length > (part_size * MAX_MULTIPART_COUNT): + raise InvalidArgumentError('Part size * max_parts(10000) is ' + ' lesser than input length.') + + if part_size < MIN_PART_SIZE: + raise InvalidArgumentError('Input part size is smaller ' + ' than allowed minimum of 5MiB.') + + if part_size > MAX_PART_SIZE: + raise InvalidArgumentError('Input part size is bigger ' + ' than allowed maximum of 5GiB.') + if not metadata: metadata = {} @@ -803,10 +820,11 @@ metadata['Content-Type'] = 'application/octet-stream' if \ not content_type else content_type - if length > MIN_PART_SIZE: + if length > part_size: return self._stream_put_object(bucket_name, object_name, data, length, metadata=metadata, - sse=sse, progress=progress) + sse=sse, progress=progress, + part_size=part_size) current_data = data.read(length) if len(current_data) != length: @@ -1550,7 +1568,8 @@ def _stream_put_object(self, bucket_name, object_name, data, content_size, - metadata=None, sse=None, progress=None): + metadata=None, sse=None, + progress=None, part_size=MIN_PART_SIZE): """ Streaming multipart upload operation. @@ -1562,6 +1581,7 @@ :param metadata: Any additional metadata to be uploaded along with your object. :param progress: A progress object + :param part_size: Multipart part size """ is_valid_bucket_name(bucket_name) is_non_empty_string(object_name) @@ -1579,7 +1599,7 @@ # Calculate optimal part info. total_parts_count, part_size, last_part_size = optimal_part_info( - content_size) + content_size, part_size) # Instantiate a thread pool with 3 worker threads pool = ThreadPool(_PARALLEL_UPLOADERS) diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/minio-4.0.12/minio/helpers.py new/minio-4.0.13/minio/helpers.py --- old/minio-4.0.12/minio/helpers.py 2019-03-13 22:30:31.000000000 +0100 +++ new/minio-4.0.13/minio/helpers.py 2019-03-27 22:32:29.000000000 +0100 @@ -46,8 +46,10 @@ # Constants MAX_MULTIPART_COUNT = 10000 # 10000 parts MAX_MULTIPART_OBJECT_SIZE = 5 * 1024 * 1024 * 1024 * 1024 # 5TiB +MAX_PART_SIZE = 5 * 1024 * 1024 * 1024 # 5GiB MAX_POOL_SIZE = 10 MIN_PART_SIZE = 5 * 1024 * 1024 # 5MiB +DEFAULT_PART_SIZE = MIN_PART_SIZE # Currently its 5MiB _VALID_BUCKETNAME_REGEX = re.compile('^[a-z0-9][a-z0-9\\.\\-]+[a-z0-9]$') _ALLOWED_HOSTNAME_REGEX = re.compile( @@ -616,7 +618,7 @@ return Hasher.md5(content).base64digest() -def optimal_part_info(length): +def optimal_part_info(length, part_size): """ Calculate optimal part size for multipart uploads. @@ -633,8 +635,8 @@ # Use floats for part size for all calculations to avoid # overflows during float64 to int64 conversions. part_size_float = math.ceil(length/MAX_MULTIPART_COUNT) - part_size_float = (math.ceil(part_size_float/MIN_PART_SIZE) - * MIN_PART_SIZE) + part_size_float = (math.ceil(part_size_float/part_size) + * part_size) # Total parts count. total_parts_count = int(math.ceil(length/part_size_float)) # Part size. diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/minio-4.0.12/minio.egg-info/PKG-INFO new/minio-4.0.13/minio.egg-info/PKG-INFO --- old/minio-4.0.12/minio.egg-info/PKG-INFO 2019-03-13 22:31:45.000000000 +0100 +++ new/minio-4.0.13/minio.egg-info/PKG-INFO 2019-03-27 22:33:49.000000000 +0100 @@ -1,6 +1,6 @@ Metadata-Version: 1.1 Name: minio -Version: 4.0.12 +Version: 4.0.13 Summary: Minio Python Library for Amazon S3 Compatible Cloud Storage for Python Home-page: https://github.com/minio/minio-py Author: Minio, Inc. diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/minio-4.0.12/tests/unit/optimal_part_test.py new/minio-4.0.13/tests/unit/optimal_part_test.py --- old/minio-4.0.12/tests/unit/optimal_part_test.py 2019-03-13 22:30:31.000000000 +0100 +++ new/minio-4.0.13/tests/unit/optimal_part_test.py 2019-03-27 22:32:29.000000000 +0100 @@ -17,28 +17,28 @@ from nose.tools import eq_, raises from unittest import TestCase -from minio.helpers import optimal_part_info, MAX_MULTIPART_OBJECT_SIZE +from minio.helpers import optimal_part_info, MAX_MULTIPART_OBJECT_SIZE, MIN_PART_SIZE from minio.error import InvalidArgumentError class TraceTest(TestCase): @raises(InvalidArgumentError) def test_input_size_wrong(self): - optimal_part_info(MAX_MULTIPART_OBJECT_SIZE + 1) + optimal_part_info(MAX_MULTIPART_OBJECT_SIZE + 1, MIN_PART_SIZE) def test_input_size_valid_maximum(self): - total_parts_count, part_size, last_part_size = optimal_part_info(MAX_MULTIPART_OBJECT_SIZE) + total_parts_count, part_size, last_part_size = optimal_part_info(MAX_MULTIPART_OBJECT_SIZE, MIN_PART_SIZE) eq_(total_parts_count, 9987) eq_(part_size, 550502400) eq_(last_part_size, 241172480) def test_input_size_valid(self): - total_parts_count, part_size, last_part_size = optimal_part_info(MAX_MULTIPART_OBJECT_SIZE/1024) + total_parts_count, part_size, last_part_size = optimal_part_info(MAX_MULTIPART_OBJECT_SIZE/1024, MIN_PART_SIZE) eq_(total_parts_count, 1024) eq_(part_size, 5242880) eq_(last_part_size, 5242880) def test_input_size_is_special_value(self): - total_parts_count, part_size, last_part_size = optimal_part_info(-1) + total_parts_count, part_size, last_part_size = optimal_part_info(-1, MIN_PART_SIZE) eq_(total_parts_count, 9987) eq_(part_size, 550502400) eq_(last_part_size, 241172480)
