Script 'mail_helper' called by obssrc
Hello community,
here is the log from the commit of package python-pymemcache for
openSUSE:Factory checked in at 2021-06-19 23:04:23
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Comparing /work/SRC/openSUSE:Factory/python-pymemcache (Old)
and /work/SRC/openSUSE:Factory/.python-pymemcache.new.2625 (New)
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Package is "python-pymemcache"
Sat Jun 19 23:04:23 2021 rev:12 rq:900866 version:3.4.4
Changes:
--------
--- /work/SRC/openSUSE:Factory/python-pymemcache/python-pymemcache.changes
2021-06-02 22:12:31.808110085 +0200
+++
/work/SRC/openSUSE:Factory/.python-pymemcache.new.2625/python-pymemcache.changes
2021-06-19 23:05:04.503859614 +0200
@@ -1,0 +2,21 @@
+Sat Jun 19 02:18:02 UTC 2021 - John Vandenberg <jay...@gmail.com>
+
+- Switch to using pytest
+- Add merged_pr_327.patch to fix mocked tests
+- Update to v3.4.4
+ * Idle connections will be removed from the pool after pool_idle_timeout
+- from v3.4.3
+ * Fix `HashClient.{get,set}_many()` with UNIX sockets.
+- from v3.4.2
+ * Remove trailing space for commands that don't take arguments, such
+ as `stats`. This was a violation of the memcached protocol.
+- from v3.4.1
+ * CAS operations will now raise MemcacheIllegalInputError when
+ None is given as the `cas` value.
+- from v3.4.0
+ * Added IPv6 support for TCP socket connections. Note that IPv6 may
+ be used in preference to IPv4 when passing a domain name as the
+ host if an IPv6 address can be resolved for that domain.
+ * `HashClient` now supports UNIX sockets.
+
+-------------------------------------------------------------------
Old:
----
pymemcache-3.3.0.tar.gz
New:
----
merged_pr_327.patch
pymemcache-3.4.4.tar.gz
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Other differences:
------------------
++++++ python-pymemcache.spec ++++++
--- /var/tmp/diff_new_pack.cNIIvG/_old 2021-06-19 23:05:04.951860304 +0200
+++ /var/tmp/diff_new_pack.cNIIvG/_new 2021-06-19 23:05:04.955860311 +0200
@@ -20,21 +20,28 @@
%{?!python_module:%define python_module() python-%{**} python3-%{**}}
%bcond_without python2
Name: python-pymemcache
-Version: 3.3.0
+Version: 3.4.4
Release: 0
Summary: A pure Python memcached client
License: Apache-2.0
Group: Development/Languages/Python
URL: https://github.com/Pinterest/pymemcache
Source:
https://files.pythonhosted.org/packages/source/p/pymemcache/pymemcache-%{version}.tar.gz
-BuildRequires: %{python_module mock}
-BuildRequires: %{python_module pytest}
+Patch0:
https://patch-diff.githubusercontent.com/raw/pinterest/pymemcache/pull/327.patch#/merged_pr_327.patch
BuildRequires: %{python_module setuptools}
-BuildRequires: %{python_module six}
BuildRequires: fdupes
+BuildRequires: memcached
BuildRequires: python-rpm-macros
Requires: python-six
BuildArch: noarch
+# SECTION test requirements
+BuildRequires: %{python_module gevent}
+BuildRequires: %{python_module mock}
+BuildRequires: %{python_module pylibmc}
+BuildRequires: %{python_module pytest}
+BuildRequires: %{python_module python-memcached}
+BuildRequires: %{python_module six}
+# /SECTION
%if %{with python2}
BuildRequires: python-future
%endif
@@ -56,6 +63,9 @@
%prep
%setup -q -n pymemcache-%{version}
+%patch0 -p1
+# Disable pytest-cov
+sed -i 's/tool:pytest/tool:ignore-pytest-cov/' setup.cfg
%build
%python_build
@@ -65,7 +75,11 @@
%python_expand %fdupes %{buildroot}%{$python_sitelib}
%check
-%pyunittest discover -v
+%{_sbindir}/memcached &
+# TLS tests depend on setting up a memcached equivalent to
+# https://github.com/scoriacorp/docker-tls-memcached
+%pytest -rs -k 'not tls'
+
%files %{python_files}
%license LICENSE.txt
++++++ merged_pr_327.patch ++++++
>From 742763658d0171598e631a72a53e331b47e281e1 Mon Sep 17 00:00:00 2001
From: Jon Parise <j...@pinterest.com>
Date: Fri, 18 Jun 2021 11:56:06 -0700
Subject: [PATCH] Provide a mock implementation of socket.getaddrinfo
9551dfd0 introduced a call to socket.getaddrinfo() to support IPv6, but
we never added an implementation of that function to MockSocketModule.
This resulted in some tests making "live" socket.getaddrinfo() calls
because of the default MockSocketModule.__getattr__ implementation
(which we need to forward other module attribute lookups).
---
pymemcache/test/test_client.py | 24 ++++++++++++++++++++----
1 file changed, 20 insertions(+), 4 deletions(-)
diff --git a/pymemcache/test/test_client.py b/pymemcache/test/test_client.py
index 24ecab2..93d327c 100644
--- a/pymemcache/test/test_client.py
+++ b/pymemcache/test/test_client.py
@@ -40,6 +40,11 @@
from pymemcache.test.utils import MockMemcacheClient
+# TODO: Use ipaddress module when dropping support for Python < 3.3
+def is_ipv6(address):
+ return re.match(r'^[0-9a-f:]+$', address)
+
+
class MockSocket(object):
def __init__(self, recv_bufs, connect_failure=None, close_failure=None):
self.recv_bufs = collections.deque(recv_bufs)
@@ -53,10 +58,8 @@ def __init__(self, recv_bufs, connect_failure=None,
close_failure=None):
@property
def family(self):
- # TODO: Use ipaddress module when dropping support for Python < 3.3
- ipv6_re = re.compile(r'^[0-9a-f:]+$')
- is_ipv6 = any(ipv6_re.match(c[0]) for c in self.connections)
- return socket.AF_INET6 if is_ipv6 else socket.AF_INET
+ any_ipv6 = any(is_ipv6(c[0]) for c in self.connections)
+ return socket.AF_INET6 if any_ipv6 else socket.AF_INET
def sendall(self, value):
self.send_bufs.append(value)
@@ -115,6 +118,19 @@ def socket(self, family, type, proto=0, fileno=None):
self.sockets.append(socket)
return socket
+ def getaddrinfo(self, host, port, family=0, type=0, proto=0, flags=0):
+ family = family or (
+ socket.AF_INET6 if is_ipv6(host) else socket.AF_INET
+ )
+ type = type or socket.SOCK_STREAM
+ proto = proto or socket.IPPROTO_TCP
+ sockaddr = (
+ ('::1', 11211, 0, 0)
+ if family == socket.AF_INET6
+ else ('127.0.0.1', 11211)
+ )
+ return [(family, type, proto, '', sockaddr)]
+
def __getattr__(self, name):
return getattr(socket, name)
++++++ pymemcache-3.3.0.tar.gz -> pymemcache-3.4.4.tar.gz ++++++
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/pymemcache-3.3.0/ChangeLog.rst
new/pymemcache-3.4.4/ChangeLog.rst
--- old/pymemcache-3.3.0/ChangeLog.rst 2020-08-18 01:20:42.000000000 +0200
+++ new/pymemcache-3.4.4/ChangeLog.rst 2021-06-02 19:41:32.000000000 +0200
@@ -1,6 +1,31 @@
Changelog
=========
+New in version 3.4.4
+--------------------
+* Idle connections will be removed from the pool after ``pool_idle_timeout``.
+
+New in version 3.4.3
+--------------------
+* Fix ``HashClient.{get,set}_many()`` with UNIX sockets.
+
+New in version 3.4.2
+--------------------
+* Remove trailing space for commands that don't take arguments, such as
+ ``stats``. This was a violation of the memcached protocol.
+
+New in version 3.4.1
+--------------------
+* CAS operations will now raise ``MemcacheIllegalInputError`` when ``None`` is
+ given as the ``cas`` value.
+
+New in version 3.4.0
+--------------------
+* Added IPv6 support for TCP socket connections. Note that IPv6 may be used in
+ preference to IPv4 when passing a domain name as the host if an IPv6 address
+ can be resolved for that domain.
+* ``HashClient`` now supports UNIX sockets.
+
New in version 3.3.0
--------------------
* ``HashClient`` can now be imported from the top-level ``pymemcache`` package
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/pymemcache-3.3.0/PKG-INFO
new/pymemcache-3.4.4/PKG-INFO
--- old/pymemcache-3.3.0/PKG-INFO 2020-08-19 22:07:45.773500400 +0200
+++ new/pymemcache-3.4.4/PKG-INFO 2021-06-02 19:42:03.184234600 +0200
@@ -1,7 +1,7 @@
Metadata-Version: 2.1
Name: pymemcache
-Version: 3.3.0
-Summary: "A comprehensive, fast, pure Python memcached client"
+Version: 3.4.4
+Summary: A comprehensive, fast, pure Python memcached client
Home-page: https://github.com/pinterest/pymemcache
Author: Jon Parise
Author-email: j...@pinterest.com
@@ -21,6 +21,7 @@
pymemcache supports the following features:
* Complete implementation of the memcached text protocol.
+ * Connections using UNIX sockets, or TCP over IPv4 or IPv6.
* Configurable timeouts for socket connect and send/recv calls.
* Access to the "noreply" flag, which can significantly increase the
speed of writes.
* Flexible, modular and simple approach to serialization and
deserialization.
@@ -134,6 +135,7 @@
* `Stephen Rosen <https://github.com/sirosen>`_
* `Feras Alazzeh <https://github.com/FerasAlazzeh>`_
* `Mois??s Guimar??es de Medeiros
<https://github.com/moisesguimaraes>`_
+ * `Nick Pope <https://github.com/pope1ni>`_
We're Hiring!
=============
@@ -143,6 +145,31 @@
Changelog
=========
+ New in version 3.4.4
+ --------------------
+ * Idle connections will be removed from the pool after
``pool_idle_timeout``.
+
+ New in version 3.4.3
+ --------------------
+ * Fix ``HashClient.{get,set}_many()`` with UNIX sockets.
+
+ New in version 3.4.2
+ --------------------
+ * Remove trailing space for commands that don't take arguments, such as
+ ``stats``. This was a violation of the memcached protocol.
+
+ New in version 3.4.1
+ --------------------
+ * CAS operations will now raise ``MemcacheIllegalInputError`` when
``None`` is
+ given as the ``cas`` value.
+
+ New in version 3.4.0
+ --------------------
+ * Added IPv6 support for TCP socket connections. Note that IPv6 may be
used in
+ preference to IPv4 when passing a domain name as the host if an IPv6
address
+ can be resolved for that domain.
+ * ``HashClient`` now supports UNIX sockets.
+
New in version 3.3.0
--------------------
* ``HashClient`` can now be imported from the top-level ``pymemcache``
package
@@ -316,10 +343,10 @@
Platform: UNKNOWN
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 2.7
-Classifier: Programming Language :: Python :: 3.5
Classifier: Programming Language :: Python :: 3.6
Classifier: Programming Language :: Python :: 3.7
Classifier: Programming Language :: Python :: 3.8
+Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: Implementation :: PyPy
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Topic :: Database
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/pymemcache-3.3.0/README.rst
new/pymemcache-3.4.4/README.rst
--- old/pymemcache-3.3.0/README.rst 2020-04-28 17:07:16.000000000 +0200
+++ new/pymemcache-3.4.4/README.rst 2020-09-15 17:14:13.000000000 +0200
@@ -13,6 +13,7 @@
pymemcache supports the following features:
* Complete implementation of the memcached text protocol.
+* Connections using UNIX sockets, or TCP over IPv4 or IPv6.
* Configurable timeouts for socket connect and send/recv calls.
* Access to the "noreply" flag, which can significantly increase the speed of
writes.
* Flexible, modular and simple approach to serialization and deserialization.
@@ -126,6 +127,7 @@
* `Stephen Rosen <https://github.com/sirosen>`_
* `Feras Alazzeh <https://github.com/FerasAlazzeh>`_
* `Mois??s Guimar??es de Medeiros <https://github.com/moisesguimaraes>`_
+* `Nick Pope <https://github.com/pope1ni>`_
We're Hiring!
=============
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/pymemcache-3.3.0/pymemcache/__init__.py
new/pymemcache-3.4.4/pymemcache/__init__.py
--- old/pymemcache-3.3.0/pymemcache/__init__.py 2020-08-18 01:20:42.000000000
+0200
+++ new/pymemcache-3.4.4/pymemcache/__init__.py 2021-06-02 19:41:32.000000000
+0200
@@ -1,4 +1,4 @@
-__version__ = '3.3.0'
+__version__ = '3.4.4'
from pymemcache.client.base import Client # noqa
from pymemcache.client.base import PooledClient # noqa
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/pymemcache-3.3.0/pymemcache/client/base.py
new/pymemcache-3.4.4/pymemcache/client/base.py
--- old/pymemcache-3.3.0/pymemcache/client/base.py 2020-08-18
01:10:28.000000000 +0200
+++ new/pymemcache-3.4.4/pymemcache/client/base.py 2021-06-02
19:25:06.000000000 +0200
@@ -37,7 +37,6 @@
b'prepend': (b'STORED', b'NOT_STORED'),
b'cas': (b'STORED', b'EXISTS', b'NOT_FOUND'),
}
-VALID_STRING_TYPES = (six.text_type, six.string_types)
# Some of the values returned by the "stats" command
@@ -87,9 +86,9 @@
if allow_unicode_keys:
if isinstance(key, six.text_type):
key = key.encode('utf8')
- elif isinstance(key, VALID_STRING_TYPES):
+ elif isinstance(key, six.string_types):
try:
- if isinstance(key, bytes):
+ if isinstance(key, six.binary_type):
key = key.decode().encode('ascii')
else:
key = key.encode('ascii')
@@ -110,6 +109,27 @@
return key
+def normalize_server_spec(server):
+ if isinstance(server, tuple) or server is None:
+ return server
+ if isinstance(server, list):
+ return tuple(server) # Assume [host, port] provided.
+ if not isinstance(server, six.string_types):
+ raise ValueError('Unknown server provided: %r' % server)
+ if server.startswith('unix:'):
+ return server[5:]
+ if server.startswith('/'):
+ return server
+ if ':' not in server or server.endswith(']'):
+ host, port = server, 11211
+ else:
+ host, port = server.rsplit(':', 1)
+ port = int(port)
+ if host.startswith('['):
+ host = host.strip('[]')
+ return (host, port)
+
+
class Client(object):
"""
A client for a single memcached server.
@@ -254,7 +274,7 @@
The constructor does not make a connection to memcached. The first
call to a method on the object will do that.
"""
- self.server = server
+ self.server = normalize_server_spec(server)
self.serde = serde or LegacyWrappingSerde(serializer, deserializer)
self.connect_timeout = connect_timeout
self.timeout = timeout
@@ -264,7 +284,7 @@
self.sock = None
if isinstance(key_prefix, six.text_type):
key_prefix = key_prefix.encode('ascii')
- if not isinstance(key_prefix, bytes):
+ if not isinstance(key_prefix, six.binary_type):
raise TypeError("key_prefix should be bytes.")
self.key_prefix = key_prefix
self.default_noreply = default_noreply
@@ -280,25 +300,41 @@
def _connect(self):
self.close()
- if isinstance(self.server, (list, tuple)):
- sock = self.socket_module.socket(self.socket_module.AF_INET,
- self.socket_module.SOCK_STREAM)
-
- if self.tls_context:
- sock = self.tls_context.wrap_socket(
- sock, server_hostname=self.server[0]
- )
+ s = self.socket_module
+
+ if not isinstance(self.server, tuple):
+ sockaddr = self.server
+ sock = s.socket(s.AF_UNIX, s.SOCK_STREAM)
+
else:
- sock = self.socket_module.socket(self.socket_module.AF_UNIX,
- self.socket_module.SOCK_STREAM)
+ sock = None
+ error = None
+ host, port = self.server
+ info = s.getaddrinfo(host, port, s.AF_UNSPEC, s.SOCK_STREAM,
+ s.IPPROTO_TCP)
+ for family, socktype, proto, _, sockaddr in info:
+ try:
+ sock = s.socket(family, socktype, proto)
+ if self.no_delay:
+ sock.setsockopt(s.IPPROTO_TCP, s.TCP_NODELAY, 1)
+ if self.tls_context:
+ context = self.tls_context
+ sock = context.wrap_socket(sock, server_hostname=host)
+ except Exception as e:
+ error = e
+ if sock is not None:
+ sock.close()
+ sock = None
+ else:
+ break
+
+ if error is not None:
+ raise error
+
try:
sock.settimeout(self.connect_timeout)
- sock.connect(self.server)
+ sock.connect(sockaddr)
sock.settimeout(self.timeout)
- if self.no_delay and sock.family == self.socket_module.AF_INET:
- sock.setsockopt(self.socket_module.IPPROTO_TCP,
- self.socket_module.TCP_NODELAY, 1)
-
except Exception:
sock.close()
raise
@@ -481,6 +517,7 @@
the key didn't exist, False if it existed but had a different cas
value and True if it existed and was changed.
"""
+ cas = self._check_cas(cas)
return self._store_cmd(b'cas', {key: value}, expire, noreply,
flags=flags, cas=cas)[key]
@@ -794,7 +831,7 @@
def _check_integer(self, value, name):
"""Check that a value is an integer and encode it as a binary string"""
- if not isinstance(value, six.integer_types): # includes "long" on py2
+ if not isinstance(value, six.integer_types):
raise MemcacheIllegalInputError(
'%s must be integer, got bad value: %r' % (name, value)
)
@@ -808,7 +845,7 @@
The value will be (re)encoded so that we can accept strings or bytes.
"""
# convert non-binary values to binary
- if isinstance(cas, (six.integer_types, VALID_STRING_TYPES)):
+ if isinstance(cas, (six.integer_types, six.string_types)):
try:
cas = six.text_type(cas).encode(self.encoding)
except UnicodeEncodeError:
@@ -832,7 +869,7 @@
"""
This function is abstracted from _fetch_cmd to support different ways
of value extraction. In order to use this feature, _extract_value needs
- to be overriden in the subclass.
+ to be overridden in the subclass.
"""
if expect_cas:
_, key, flags, size, cas = line.split()
@@ -856,7 +893,10 @@
remapped_keys = dict(zip(prefixed_keys, keys))
# It is important for all keys to be listed in their original order.
- cmd = name + b' ' + b' '.join(prefixed_keys) + b'\r\n'
+ cmd = name
+ if prefixed_keys:
+ cmd += b' ' + b' '.join(prefixed_keys)
+ cmd += b'\r\n'
try:
if self.sock is None:
@@ -897,7 +937,6 @@
extra = b''
if cas is not None:
- cas = self._check_cas(cas)
extra += b' ' + cas
if noreply:
extra += b' noreply'
@@ -1000,8 +1039,11 @@
max_pool_size: maximum pool size to use (going above this amount
triggers a runtime error), by default this is 2147483648L
when not provided (or none).
+ pool_idle_timeout: pooled connections are discarded if they have been
+ unused for this many seconds. A value of 0 indicates
+ that pooled connections are never discarded.
lock_generator: a callback/type that takes no arguments that will
- be called to create a lock or sempahore that can
+ be called to create a lock or semaphore that can
protect the pool from concurrent access (for example a
eventlet lock or semaphore could be used instead)
@@ -1026,12 +1068,13 @@
socket_module=socket,
key_prefix=b'',
max_pool_size=None,
+ pool_idle_timeout=0,
lock_generator=None,
default_noreply=True,
allow_unicode_keys=False,
encoding='ascii',
tls_context=None):
- self.server = server
+ self.server = normalize_server_spec(server)
self.serde = serde or LegacyWrappingSerde(serializer, deserializer)
self.connect_timeout = connect_timeout
self.timeout = timeout
@@ -1042,13 +1085,14 @@
self.allow_unicode_keys = allow_unicode_keys
if isinstance(key_prefix, six.text_type):
key_prefix = key_prefix.encode('ascii')
- if not isinstance(key_prefix, bytes):
+ if not isinstance(key_prefix, six.binary_type):
raise TypeError("key_prefix should be bytes.")
self.key_prefix = key_prefix
self.client_pool = pool.ObjectPool(
self._create_client,
after_remove=lambda client: client.close(),
max_size=max_pool_size,
+ idle_timeout=pool_idle_timeout,
lock_generator=lock_generator)
self.encoding = encoding
self.tls_context = tls_context
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/pymemcache-3.3.0/pymemcache/client/hash.py
new/pymemcache-3.4.4/pymemcache/client/hash.py
--- old/pymemcache-3.3.0/pymemcache/client/hash.py 2020-08-18
01:10:28.000000000 +0200
+++ new/pymemcache-3.4.4/pymemcache/client/hash.py 2021-06-02
19:25:06.000000000 +0200
@@ -4,7 +4,12 @@
import logging
import six
-from pymemcache.client.base import Client, PooledClient, check_key_helper
+from pymemcache.client.base import (
+ Client,
+ PooledClient,
+ check_key_helper,
+ normalize_server_spec,
+)
from pymemcache.client.rendezvous import RendezvousHash
from pymemcache.exceptions import MemcacheError
@@ -31,6 +36,7 @@
socket_module=socket,
key_prefix=b'',
max_pool_size=None,
+ pool_idle_timeout=0,
lock_generator=None,
retry_attempts=2,
retry_timeout=1,
@@ -46,7 +52,8 @@
Constructor.
Args:
- servers: list(tuple(hostname, port))
+ servers: list() of tuple(hostname, port) or string containing a UNIX
+ socket path.
hasher: optional class three functions ``get_node``, ``add_node``,
and ``remove_node``
defaults to Rendezvous (HRW) hash.
@@ -98,33 +105,49 @@
if use_pooling is True:
self.default_kwargs.update({
'max_pool_size': max_pool_size,
+ 'pool_idle_timeout': pool_idle_timeout,
'lock_generator': lock_generator
})
- for server, port in servers:
- self.add_server(server, port)
+ for server in servers:
+ self.add_server(normalize_server_spec(server))
self.encoding = encoding
self.tls_context = tls_context
- def add_server(self, server, port):
- key = '%s:%s' % (server, port)
+ def _make_client_key(self, server):
+ if isinstance(server, (list, tuple)) and len(server) == 2:
+ return '%s:%s' % server
+ return server
+
+ def add_server(self, server, port=None):
+ # To maintain backward compatibility, if a port is provided, assume
+ # that server wasn't provided as a (host, port) tuple.
+ if port is not None:
+ if not isinstance(server, six.string_types):
+ raise TypeError('Server must be a string when passing port.')
+ server = (server, port)
+ _class = PooledClient if self.use_pooling else self.client_class
+ client = _class(server, **self.default_kwargs)
if self.use_pooling:
- client = PooledClient(
- (server, port),
- **self.default_kwargs
- )
- else:
- client = self.client_class((server, port), **self.default_kwargs)
+ client.client_class = self.client_class
+ key = self._make_client_key(server)
self.clients[key] = client
self.hasher.add_node(key)
- def remove_server(self, server, port):
+ def remove_server(self, server, port=None):
+ # To maintain backward compatibility, if a port is provided, assume
+ # that server wasn't provided as a (host, port) tuple.
+ if port is not None:
+ if not isinstance(server, six.string_types):
+ raise TypeError('Server must be a string when passing port.')
+ server = (server, port)
+
+ key = self._make_client_key(server)
dead_time = time.time()
- self._failed_clients.pop((server, port))
- self._dead_clients[(server, port)] = dead_time
- key = '%s:%s' % (server, port)
+ self._failed_clients.pop(server)
+ self._dead_clients[server] = dead_time
self.hasher.remove_node(key)
def _retry_dead(self):
@@ -141,7 +164,7 @@
'bringing server back into rotation %s',
server
)
- self.add_server(*server)
+ self.add_server(server)
del self._dead_clients[server]
self._last_dead_check_time = current_time
@@ -157,8 +180,7 @@
return
raise MemcacheError('All servers seem to be down right now')
- client = self.clients[server]
- return client
+ return self.clients[server]
def _safely_run_func(self, client, func, default_val, *args, **kwargs):
try:
@@ -185,7 +207,7 @@
# We've reached our max retry attempts, we need to mark
# the sever as dead
logger.debug('marking server as dead: %s', client.server)
- self.remove_server(*client.server)
+ self.remove_server(client.server)
result = func(*args, **kwargs)
return result
@@ -239,7 +261,7 @@
# We've reached our max retry attempts, we need to mark
# the sever as dead
logger.debug('marking server as dead: %s', client.server)
- self.remove_server(*client.server)
+ self.remove_server(client.server)
succeeded, failed, err = self._set_many(
client, values, *args, **kwargs
@@ -289,7 +311,7 @@
'attempts': 0,
}
logger.debug("marking server as dead %s", server)
- self.remove_server(*server)
+ self.remove_server(server)
# This client has failed previously, we need to update the metadata
# to reflect that we have attempted it again
else:
@@ -316,15 +338,12 @@
succeeded = []
try:
- for key, value in six.iteritems(values):
- result = client.set(key, value, *args, **kwargs)
- if result:
- succeeded.append(key)
- else:
- failed.append(key)
+ failed = client.set_many(values, *args, **kwargs)
except Exception as e:
- return succeeded, failed, e
+ if not self.ignore_exc:
+ return succeeded, failed, e
+ succeeded = [key for key in six.iterkeys(values) if key not in failed]
return succeeded, failed, None
def close(self):
@@ -359,8 +378,7 @@
client_batches[client.server][key] = value
for server, values in client_batches.items():
- client = self.clients['%s:%s' % server]
-
+ client = self.clients[self._make_client_key(server)]
failed += self._safely_run_set_many(
client, values, *args, **kwargs
)
@@ -382,7 +400,7 @@
client_batches[client.server].append(key)
for server, keys in client_batches.items():
- client = self.clients['%s:%s' % server]
+ client = self.clients[self._make_client_key(server)]
new_args = list(args)
new_args.insert(0, keys)
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/pymemcache-3.3.0/pymemcache/fallback.py
new/pymemcache-3.4.4/pymemcache/fallback.py
--- old/pymemcache-3.3.0/pymemcache/fallback.py 2020-04-28 17:07:16.000000000
+0200
+++ new/pymemcache-3.4.4/pymemcache/fallback.py 2020-09-15 17:14:13.000000000
+0200
@@ -16,7 +16,7 @@
A client for falling back to older memcached servers when performing reads.
It is sometimes necessary to deploy memcached on new servers, or with a
-different configuration. In theses cases, it is undesirable to start up an
+different configuration. In these cases, it is undesirable to start up an
empty memcached server and point traffic to it, since the cache will be cold,
and the backing store will have a large increase in traffic.
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/pymemcache-3.3.0/pymemcache/pool.py
new/pymemcache-3.4.4/pymemcache/pool.py
--- old/pymemcache-3.3.0/pymemcache/pool.py 2020-04-28 17:07:16.000000000
+0200
+++ new/pymemcache-3.4.4/pymemcache/pool.py 2021-06-02 19:25:06.000000000
+0200
@@ -16,6 +16,7 @@
import contextlib
import sys
import threading
+import time
import six
@@ -25,6 +26,7 @@
def __init__(self, obj_creator,
after_remove=None, max_size=None,
+ idle_timeout=0,
lock_generator=None):
self._used_objs = collections.deque()
self._free_objs = collections.deque()
@@ -38,6 +40,8 @@
if not isinstance(max_size, six.integer_types) or max_size < 0:
raise ValueError('"max_size" must be a positive integer')
self.max_size = max_size
+ self.idle_timeout = idle_timeout
+ self._idle_clock = time.time if idle_timeout else int
@property
def used(self):
@@ -63,19 +67,27 @@
def get(self):
with self._lock:
- if not self._free_objs:
+ # Find a free object, removing any that have idled for too long.
+ now = self._idle_clock()
+ while self._free_objs:
+ obj = self._free_objs.popleft()
+ if now - obj._last_used <= self.idle_timeout:
+ break
+
+ if self._after_remove is not None:
+ self._after_remove(obj)
+ else:
+ # No free objects, create a new one.
curr_count = len(self._used_objs)
if curr_count >= self.max_size:
raise RuntimeError("Too many objects,"
" %s >= %s" % (curr_count,
self.max_size))
obj = self._obj_creator()
- self._used_objs.append(obj)
- return obj
- else:
- obj = self._free_objs.pop()
- self._used_objs.append(obj)
- return obj
+
+ self._used_objs.append(obj)
+ obj._last_used = now
+ return obj
def destroy(self, obj, silent=True):
was_dropped = False
@@ -94,6 +106,7 @@
try:
self._used_objs.remove(obj)
self._free_objs.append(obj)
+ obj._last_used = self._idle_clock()
except ValueError:
if not silent:
raise
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/pymemcache-3.3.0/pymemcache/test/test_client.py
new/pymemcache-3.4.4/pymemcache/test/test_client.py
--- old/pymemcache-3.3.0/pymemcache/test/test_client.py 2020-04-29
01:54:59.000000000 +0200
+++ new/pymemcache-3.4.4/pymemcache/test/test_client.py 2021-06-02
19:25:06.000000000 +0200
@@ -21,12 +21,13 @@
import json
import os
import mock
+import re
import socket
import unittest
import pytest
-from pymemcache.client.base import PooledClient, Client
+from pymemcache.client.base import PooledClient, Client, normalize_server_spec
from pymemcache.exceptions import (
MemcacheClientError,
MemcacheServerError,
@@ -52,7 +53,10 @@
@property
def family(self):
- return socket.AF_INET
+ # TODO: Use ipaddress module when dropping support for Python < 3.3
+ ipv6_re = re.compile(r'^[0-9a-f:]+$')
+ is_ipv6 = any(ipv6_re.match(c[0]) for c in self.connections)
+ return socket.AF_INET6 if is_ipv6 else socket.AF_INET
def sendall(self, value):
self.send_bufs.append(value)
@@ -103,7 +107,7 @@
self.close_failure = close_failure
self.sockets = []
- def socket(self, family, type):
+ def socket(self, family, type, proto=0, fileno=None):
socket = MockSocket(
[],
connect_failure=self.connect_failure,
@@ -482,6 +486,9 @@
def test_cas_malformed(self):
client = self.make_client([b'STORED\r\n'])
with pytest.raises(MemcacheIllegalInputError):
+ client.cas(b'key', b'value', None, noreply=False)
+
+ with pytest.raises(MemcacheIllegalInputError):
client.cas(b'key', b'value', 'nonintegerstring', noreply=False)
with pytest.raises(MemcacheIllegalInputError):
@@ -882,7 +889,7 @@
client = self.make_client([b'STAT fake_stats 1\r\n', b'END\r\n'])
result = client.stats()
assert client.sock.send_bufs == [
- b'stats \r\n'
+ b'stats\r\n'
]
assert result == {b'fake_stats': 1}
@@ -916,7 +923,7 @@
])
result = client.stats()
assert client.sock.send_bufs == [
- b'stats \r\n'
+ b'stats\r\n'
]
expected = {
b'cmd_get': 2519,
@@ -1075,12 +1082,39 @@
@pytest.mark.unit()
class TestClientSocketConnect(unittest.TestCase):
- def test_socket_connect(self):
- server = ("example.com", 11211)
+ def test_socket_connect_ipv4(self):
+ server = ('127.0.0.1', 11211)
client = Client(server, socket_module=MockSocketModule())
client._connect()
assert client.sock.connections == [server]
+ assert client.sock.family == socket.AF_INET
+
+ timeout = 2
+ connect_timeout = 3
+ client = Client(
+ server, connect_timeout=connect_timeout, timeout=timeout,
+ socket_module=MockSocketModule())
+ client._connect()
+ assert client.sock.timeouts == [connect_timeout, timeout]
+
+ client = Client(server, socket_module=MockSocketModule())
+ client._connect()
+ assert client.sock.socket_options == []
+
+ client = Client(
+ server, socket_module=MockSocketModule(), no_delay=True)
+ client._connect()
+ assert client.sock.socket_options == [(socket.IPPROTO_TCP,
+ socket.TCP_NODELAY, 1)]
+
+ def test_socket_connect_ipv6(self):
+ server = ('::1', 11211)
+
+ client = Client(server, socket_module=MockSocketModule())
+ client._connect()
+ assert client.sock.connections == [server + (0, 0)]
+ assert client.sock.family == socket.AF_INET6
timeout = 2
connect_timeout = 3
@@ -1221,6 +1255,38 @@
assert isinstance(client.client_pool.get(), MyClient)
+class TestPooledClientIdleTimeout(ClientTestMixin, unittest.TestCase):
+ def make_client(self, mock_socket_values, **kwargs):
+ mock_client = Client(None, **kwargs)
+ mock_client.sock = MockSocket(list(mock_socket_values))
+ client = PooledClient(None, pool_idle_timeout=60, **kwargs)
+ client.client_pool = pool.ObjectPool(lambda: mock_client)
+ return client
+
+ def test_free_idle(self):
+ class Counter(object):
+ count = 0
+
+ def increment(self, obj):
+ self.count += 1
+
+ removed = Counter()
+
+ client = self.make_client([b'VALUE key 0 5\r\nvalue\r\nEND\r\n']*2)
+ client.client_pool._after_remove = removed.increment
+ client.client_pool._idle_clock = lambda: 0
+
+ client.set(b'key', b'value')
+ assert removed.count == 0
+ client.get(b'key')
+ assert removed.count == 0
+
+ # Advance clock to beyond the idle timeout.
+ client.client_pool._idle_clock = lambda: 61
+ client.get(b'key')
+ assert removed.count == 1
+
+
class TestMockClient(ClientTestMixin, unittest.TestCase):
def make_client(self, mock_socket_values, **kwargs):
client = MockMemcacheClient(None, **kwargs)
@@ -1330,3 +1396,28 @@
b'ue1\r\nEND\r\n',
])
assert client[b'key1'] == b'value1'
+
+
+@pytest.mark.unit()
+class TestNormalizeServerSpec(unittest.TestCase):
+ def test_normalize_server_spec(self):
+ f = normalize_server_spec
+ assert f(None) is None
+ assert f(('127.0.0.1', 12345)) == ('127.0.0.1', 12345)
+ assert f(['127.0.0.1', 12345]) == ('127.0.0.1', 12345)
+ assert f('unix:/run/memcached/socket') == '/run/memcached/socket'
+ assert f('/run/memcached/socket') == '/run/memcached/socket'
+ assert f('localhost') == ('localhost', 11211)
+ assert f('localhost:12345') == ('localhost', 12345)
+ assert f('[::1]') == ('::1', 11211)
+ assert f('[::1]:12345') == ('::1', 12345)
+ assert f('127.0.0.1') == ('127.0.0.1', 11211)
+ assert f('127.0.0.1:12345') == ('127.0.0.1', 12345)
+
+ with pytest.raises(ValueError) as excinfo:
+ f({'host': 12345})
+ assert str(excinfo.value) == "Unknown server provided: {'host': 12345}"
+
+ with pytest.raises(ValueError) as excinfo:
+ f(12345)
+ assert str(excinfo.value) == "Unknown server provided: 12345"
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/pymemcache-3.3.0/pymemcache/test/test_client_hash.py
new/pymemcache-3.4.4/pymemcache/test/test_client_hash.py
--- old/pymemcache-3.3.0/pymemcache/test/test_client_hash.py 2020-08-18
01:10:28.000000000 +0200
+++ new/pymemcache-3.4.4/pymemcache/test/test_client_hash.py 2021-05-27
22:16:10.000000000 +0200
@@ -5,6 +5,7 @@
from .test_client import ClientTestMixin, MockSocket
import unittest
+import os
import pytest
import mock
import socket
@@ -38,17 +39,55 @@
return client
+ def make_unix_client(self, sockets, *mock_socket_values, **kwargs):
+ client = HashClient([], **kwargs)
+
+ for socket_, vals in zip(sockets, mock_socket_values):
+ c = self.make_client_pool(
+ socket_,
+ vals,
+ **kwargs
+ )
+ client.clients[socket_] = c
+ client.hasher.add_node(socket_)
+
+ return client
+
def test_setup_client_without_pooling(self):
client_class = 'pymemcache.client.hash.HashClient.client_class'
with mock.patch(client_class) as internal_client:
client = HashClient([], timeout=999, key_prefix='foo_bar_baz')
- client.add_server('127.0.0.1', '11211')
+ client.add_server(('127.0.0.1', '11211'))
assert internal_client.call_args[0][0] == ('127.0.0.1', '11211')
kwargs = internal_client.call_args[1]
assert kwargs['timeout'] == 999
assert kwargs['key_prefix'] == 'foo_bar_baz'
+ def test_get_many_unix(self):
+ pid = os.getpid()
+ sockets = [
+ '/tmp/pymemcache.1.%d' % pid,
+ '/tmp/pymemcache.2.%d' % pid,
+ ]
+ client = self.make_unix_client(sockets, *[
+ [b'STORED\r\n', b'VALUE key3 0 6\r\nvalue2\r\nEND\r\n', ],
+ [b'STORED\r\n', b'VALUE key1 0 6\r\nvalue1\r\nEND\r\n', ],
+ ])
+
+ def get_clients(key):
+ if key == b'key3':
+ return client.clients['/tmp/pymemcache.1.%d' % pid]
+ else:
+ return client.clients['/tmp/pymemcache.2.%d' % pid]
+
+ client._get_client = get_clients
+
+ result = client.set(b'key1', b'value1', noreply=False)
+ result = client.set(b'key3', b'value2', noreply=False)
+ result = client.get_many([b'key1', b'key3'])
+ assert result == {b'key1': b'value1', b'key3': b'value2'}
+
def test_get_many_all_found(self):
client = self.make_client(*[
[b'STORED\r\n', b'VALUE key3 0 6\r\nvalue2\r\nEND\r\n', ],
@@ -262,7 +301,7 @@
], ignore_exc=True)
result = client.set_many(values, noreply=False)
- assert len(result) == 2
+ assert len(result) == 0
def test_noreply_set_many(self):
values = {
@@ -283,6 +322,22 @@
result = client.set_many(values, noreply=True)
assert result == []
+ def test_set_many_unix(self):
+ values = {
+ 'key1': 'value1',
+ 'key2': 'value2',
+ 'key3': 'value3'
+ }
+
+ pid = os.getpid()
+ sockets = ['/tmp/pymemcache.%d' % pid]
+ client = self.make_unix_client(sockets, *[
+ [b'STORED\r\n', b'NOT_STORED\r\n', b'STORED\r\n'],
+ ])
+
+ result = client.set_many(values, noreply=False)
+ assert len(result) == 1
+
def test_server_encoding_pooled(self):
"""
test passed encoding from hash client to pooled clients
@@ -313,7 +368,7 @@
@mock.patch("pymemcache.client.hash.HashClient.client_class")
def test_dead_server_comes_back(self, client_patch):
client = HashClient([], dead_timeout=0, retry_attempts=0)
- client.add_server("127.0.0.1", 11211)
+ client.add_server(("127.0.0.1", 11211))
test_client = client_patch.return_value
test_client.server = ("127.0.0.1", 11211)
@@ -332,7 +387,7 @@
@mock.patch("pymemcache.client.hash.HashClient.client_class")
def test_failed_is_retried(self, client_patch):
client = HashClient([], retry_attempts=1, retry_timeout=0)
- client.add_server("127.0.0.1", 11211)
+ client.add_server(("127.0.0.1", 11211))
assert client_patch.call_count == 1
@@ -354,7 +409,55 @@
client = HashClient([])
client.client_class = MyClient
- client.add_server('host', 11211)
+ client.add_server(('host', 11211))
assert isinstance(client.clients['host:11211'], MyClient)
+ def test_custom_client_with_pooling(self):
+ class MyClient(Client):
+ pass
+
+ client = HashClient([], use_pooling=True)
+ client.client_class = MyClient
+ client.add_server(('host', 11211))
+ assert isinstance(client.clients['host:11211'], PooledClient)
+
+ pool = client.clients['host:11211'].client_pool
+ with pool.get_and_release(destroy_on_fail=True) as c:
+ assert isinstance(c, MyClient)
+
+ def test_mixed_inet_and_unix_sockets(self):
+ expected = {
+ '/tmp/pymemcache.{pid}'.format(pid=os.getpid()),
+ ('127.0.0.1', 11211),
+ ('::1', 11211),
+ }
+ client = HashClient([
+ '/tmp/pymemcache.{pid}'.format(pid=os.getpid()),
+ '127.0.0.1',
+ '127.0.0.1:11211',
+ '[::1]',
+ '[::1]:11211',
+ ('127.0.0.1', 11211),
+ ('::1', 11211),
+ ])
+ assert expected == {c.server for c in client.clients.values()}
+
+ def test_legacy_add_remove_server_signature(self):
+ server = ('127.0.0.1', 11211)
+ client = HashClient([])
+ assert client.clients == {}
+ client.add_server(*server) # Unpack (host, port) tuple.
+ assert ('%s:%s' % server) in client.clients
+ client._mark_failed_server(server)
+ assert server in client._failed_clients
+ client.remove_server(*server) # Unpack (host, port) tuple.
+ assert server in client._dead_clients
+ assert server not in client._failed_clients
+
+ # Ensure that server is a string if passing port argument:
+ with pytest.raises(TypeError):
+ client.add_server(server, server[-1])
+ with pytest.raises(TypeError):
+ client.remove_server(server, server[-1])
+
# TODO: Test failover logic
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/pymemcache-3.3.0/pymemcache.egg-info/PKG-INFO
new/pymemcache-3.4.4/pymemcache.egg-info/PKG-INFO
--- old/pymemcache-3.3.0/pymemcache.egg-info/PKG-INFO 2020-08-19
22:07:45.000000000 +0200
+++ new/pymemcache-3.4.4/pymemcache.egg-info/PKG-INFO 2021-06-02
19:42:02.000000000 +0200
@@ -1,7 +1,7 @@
Metadata-Version: 2.1
Name: pymemcache
-Version: 3.3.0
-Summary: "A comprehensive, fast, pure Python memcached client"
+Version: 3.4.4
+Summary: A comprehensive, fast, pure Python memcached client
Home-page: https://github.com/pinterest/pymemcache
Author: Jon Parise
Author-email: j...@pinterest.com
@@ -21,6 +21,7 @@
pymemcache supports the following features:
* Complete implementation of the memcached text protocol.
+ * Connections using UNIX sockets, or TCP over IPv4 or IPv6.
* Configurable timeouts for socket connect and send/recv calls.
* Access to the "noreply" flag, which can significantly increase the
speed of writes.
* Flexible, modular and simple approach to serialization and
deserialization.
@@ -134,6 +135,7 @@
* `Stephen Rosen <https://github.com/sirosen>`_
* `Feras Alazzeh <https://github.com/FerasAlazzeh>`_
* `Mois??s Guimar??es de Medeiros
<https://github.com/moisesguimaraes>`_
+ * `Nick Pope <https://github.com/pope1ni>`_
We're Hiring!
=============
@@ -143,6 +145,31 @@
Changelog
=========
+ New in version 3.4.4
+ --------------------
+ * Idle connections will be removed from the pool after
``pool_idle_timeout``.
+
+ New in version 3.4.3
+ --------------------
+ * Fix ``HashClient.{get,set}_many()`` with UNIX sockets.
+
+ New in version 3.4.2
+ --------------------
+ * Remove trailing space for commands that don't take arguments, such as
+ ``stats``. This was a violation of the memcached protocol.
+
+ New in version 3.4.1
+ --------------------
+ * CAS operations will now raise ``MemcacheIllegalInputError`` when
``None`` is
+ given as the ``cas`` value.
+
+ New in version 3.4.0
+ --------------------
+ * Added IPv6 support for TCP socket connections. Note that IPv6 may be
used in
+ preference to IPv4 when passing a domain name as the host if an IPv6
address
+ can be resolved for that domain.
+ * ``HashClient`` now supports UNIX sockets.
+
New in version 3.3.0
--------------------
* ``HashClient`` can now be imported from the top-level ``pymemcache``
package
@@ -316,10 +343,10 @@
Platform: UNKNOWN
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 2.7
-Classifier: Programming Language :: Python :: 3.5
Classifier: Programming Language :: Python :: 3.6
Classifier: Programming Language :: Python :: 3.7
Classifier: Programming Language :: Python :: 3.8
+Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: Implementation :: PyPy
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Topic :: Database
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/pymemcache-3.3.0/setup.cfg
new/pymemcache-3.4.4/setup.cfg
--- old/pymemcache-3.3.0/setup.cfg 2020-08-19 22:07:45.774235000 +0200
+++ new/pymemcache-3.4.4/setup.cfg 2021-06-02 19:42:03.185471500 +0200
@@ -3,7 +3,7 @@
version = attr: pymemcache.__version__
author = Jon Parise
author_email = j...@pinterest.com
-description = "A comprehensive, fast, pure Python memcached client"
+description = A comprehensive, fast, pure Python memcached client
long_description = file: README.rst, ChangeLog.rst
long_description_content_type = text/x-rst
license = Apache License 2.0
@@ -12,10 +12,10 @@
classifiers =
Programming Language :: Python
Programming Language :: Python :: 2.7
- Programming Language :: Python :: 3.5
Programming Language :: Python :: 3.6
Programming Language :: Python :: 3.7
Programming Language :: Python :: 3.8
+ Programming Language :: Python :: 3.9
Programming Language :: Python :: Implementation :: PyPy
License :: OSI Approved :: Apache Software License
Topic :: Database
