Script 'mail_helper' called by obssrc
Hello community,
here is the log from the commit of package python-django-cacheops for
openSUSE:Factory checked in at 2021-05-11 23:04:18
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Comparing /work/SRC/openSUSE:Factory/python-django-cacheops (Old)
and /work/SRC/openSUSE:Factory/.python-django-cacheops.new.2988 (New)
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Package is "python-django-cacheops"
Tue May 11 23:04:18 2021 rev:3 rq:892240 version:6.0
Changes:
--------
---
/work/SRC/openSUSE:Factory/python-django-cacheops/python-django-cacheops.changes
2020-10-29 14:53:02.265260023 +0100
+++
/work/SRC/openSUSE:Factory/.python-django-cacheops.new.2988/python-django-cacheops.changes
2021-05-11 23:04:26.608880202 +0200
@@ -1,0 +2,17 @@
+Sun May 9 23:34:35 UTC 2021 - Daniel Molkentin <[email protected]>
+
+- Update to v6.0
+ * support and test against Python 3.9 and Django 3.1/3.2
+ * added custom serializers support (thx to Arcady Usov)
+ * support callable extra in @cached_as() and friends
+ * made simple cache obey prefix
+ * skip JSONFields for purposes of invalidation
+ * configure skipped fields by internal types, classes still supported
+ * handle `DatabaseError` on transaction cleanup (Roman Gorbil)
+ * do not query old object if cacheops is disabled
+ * do not fetch deferred fields during invalidation, fixes #387
+ Backwards incompatible changes:
+ * callable `extra` param, including type, now behaves differently
+ * simple cache now uses prefix
+
+-------------------------------------------------------------------
Old:
----
django-cacheops-5.1.tar.gz
New:
----
django-cacheops-6.0.tar.gz
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Other differences:
------------------
++++++ python-django-cacheops.spec ++++++
--- /var/tmp/diff_new_pack.cWXJvb/_old 2021-05-11 23:04:27.044878214 +0200
+++ /var/tmp/diff_new_pack.cWXJvb/_new 2021-05-11 23:04:27.048878195 +0200
@@ -1,7 +1,7 @@
#
# spec file for package python-django-cacheops
#
-# Copyright (c) 2020 SUSE LLC
+# Copyright (c) 2021 SUSE LLC
#
# All modifications and additions to the file contributed by third parties
# remain the property of their copyright owners, unless otherwise agreed
@@ -19,7 +19,7 @@
%{?!python_module:%define python_module() python-%{**} python3-%{**}}
%define skip_python2 1
Name: python-django-cacheops
-Version: 5.1
+Version: 6.0
Release: 0
Summary: Django ORM cache with automatic granular event-driven
invalidation
License: BSD-3-Clause
++++++ django-cacheops-5.1.tar.gz -> django-cacheops-6.0.tar.gz ++++++
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/django-cacheops-5.1/CHANGELOG
new/django-cacheops-6.0/CHANGELOG
--- old/django-cacheops-5.1/CHANGELOG 2020-10-25 14:47:14.000000000 +0100
+++ new/django-cacheops-6.0/CHANGELOG 2021-05-03 11:08:24.000000000 +0200
@@ -1,3 +1,17 @@
+6.0
+- support and test against Python 3.9 and Django 3.1/3.2
+- added custom serializers support (thx to Arcady Usov)
+- support callable extra in @cached_as() and friends
+- made simple cache obey prefix
+- skip JSONFields for purposes of invalidation
+- configure skipped fields by internal types, classes still supported
+- handle `DatabaseError` on transaction cleanup (Roman Gorbil)
+- do not query old object if cacheops is disabled
+- do not fetch deferred fields during invalidation, fixes #387
+Backwards incompatible changes:
+- callable `extra` param, including type, now behaves differently
+- simple cache now uses prefix
+
5.1
- support subqueries in annotations (Jeremy Stretch)
- included tests into distro (John Vandenberg)
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/django-cacheops-5.1/PKG-INFO
new/django-cacheops-6.0/PKG-INFO
--- old/django-cacheops-5.1/PKG-INFO 2020-10-25 15:11:24.448593400 +0100
+++ new/django-cacheops-6.0/PKG-INFO 2021-05-03 13:09:39.016655400 +0200
@@ -1,6 +1,6 @@
Metadata-Version: 1.2
Name: django-cacheops
-Version: 5.1
+Version: 6.0
Summary: A slick ORM cache with automatic granular event-driven invalidation
for Django.
Home-page: http://github.com/Suor/django-cacheops
Author: Alexander Schepanovski
@@ -275,9 +275,18 @@
@cached_view_as(News)
def news_index(request):
# ...
- return HttpResponse(...)
+ return render(...)
+
+ You can pass ``timeout``, ``extra`` and several samples the same way
as to ``@cached_as()``. Note that you can pass a function as ``extra``:
- You can pass ``timeout``, ``extra`` and several samples the same way
as to ``@cached_as()``.
+ .. code:: python
+
+ @cached_view_as(News, extra=lambda req: req.user.is_staff)
+ def news_index(request):
+ # ... add extra things for staff
+ return render(...)
+
+ A function passed as ``extra`` receives the same arguments as the
cached function.
Class based views can also be cached:
@@ -286,7 +295,7 @@
class NewsIndex(ListView):
model = News
- news_index = cached_view_as(News)(NewsIndex.as_view())
+ news_index = cached_view_as(News, ...)(NewsIndex.as_view())
Invalidation
@@ -663,6 +672,27 @@
**NOTE:** prefix is not used in simple and file cache. This might
change in future cacheops.
+ Custom serialization
+ --------------------
+
+ Cacheops uses ``pickle`` by default, employing it's default protocol.
But you can specify your own
+ it might be any module or a class having `.dumps()` and `.loads()`
functions. For example you can use ``dill`` instead, which can serialize more
things like anonymous functions:
+
+ .. code:: python
+
+ CACHEOPS_SERIALIZER = 'dill'
+
+ One less obvious use is to fix pickle protocol, to use cacheops cache
across python versions:
+
+ .. code:: python
+
+ import pickle
+
+ class CACHEOPS_SERIALIZER:
+ dumps = lambda data: pickle.dumps(data, 3)
+ loads = pickle.loads
+
+
Using memory limit
------------------
@@ -804,10 +834,13 @@
Classifier: Programming Language :: Python :: 3.6
Classifier: Programming Language :: Python :: 3.7
Classifier: Programming Language :: Python :: 3.8
+Classifier: Programming Language :: Python :: 3.9
Classifier: Framework :: Django
Classifier: Framework :: Django :: 2.1
Classifier: Framework :: Django :: 2.2
Classifier: Framework :: Django :: 3.0
+Classifier: Framework :: Django :: 3.1
+Classifier: Framework :: Django :: 3.2
Classifier: Environment :: Web Environment
Classifier: Intended Audience :: Developers
Classifier: Topic :: Internet :: WWW/HTTP
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/django-cacheops-5.1/README.rst
new/django-cacheops-6.0/README.rst
--- old/django-cacheops-5.1/README.rst 2020-07-18 08:34:04.000000000 +0200
+++ new/django-cacheops-6.0/README.rst 2021-04-30 11:03:42.000000000 +0200
@@ -265,9 +265,18 @@
@cached_view_as(News)
def news_index(request):
# ...
- return HttpResponse(...)
+ return render(...)
+
+You can pass ``timeout``, ``extra`` and several samples the same way as to
``@cached_as()``. Note that you can pass a function as ``extra``:
-You can pass ``timeout``, ``extra`` and several samples the same way as to
``@cached_as()``.
+.. code:: python
+
+ @cached_view_as(News, extra=lambda req: req.user.is_staff)
+ def news_index(request):
+ # ... add extra things for staff
+ return render(...)
+
+A function passed as ``extra`` receives the same arguments as the cached
function.
Class based views can also be cached:
@@ -276,7 +285,7 @@
class NewsIndex(ListView):
model = News
- news_index = cached_view_as(News)(NewsIndex.as_view())
+ news_index = cached_view_as(News, ...)(NewsIndex.as_view())
Invalidation
@@ -653,6 +662,27 @@
**NOTE:** prefix is not used in simple and file cache. This might change in
future cacheops.
+Custom serialization
+--------------------
+
+Cacheops uses ``pickle`` by default, employing it's default protocol. But you
can specify your own
+it might be any module or a class having `.dumps()` and `.loads()` functions.
For example you can use ``dill`` instead, which can serialize more things like
anonymous functions:
+
+.. code:: python
+
+ CACHEOPS_SERIALIZER = 'dill'
+
+One less obvious use is to fix pickle protocol, to use cacheops cache across
python versions:
+
+.. code:: python
+
+ import pickle
+
+ class CACHEOPS_SERIALIZER:
+ dumps = lambda data: pickle.dumps(data, 3)
+ loads = pickle.loads
+
+
Using memory limit
------------------
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/django-cacheops-5.1/cacheops/conf.py
new/django-cacheops-6.0/cacheops/conf.py
--- old/django-cacheops-5.1/cacheops/conf.py 2020-06-21 08:00:30.000000000
+0200
+++ new/django-cacheops-6.0/cacheops/conf.py 2021-05-03 11:00:18.000000000
+0200
@@ -1,10 +1,9 @@
+from importlib import import_module
from funcy import memoize, merge
from django.conf import settings as base_settings
from django.core.exceptions import ImproperlyConfigured
from django.core.signals import setting_changed
-from django.db import models
-from django.utils.module_loading import import_string
ALL_OPS = {'get', 'fetch', 'count', 'aggregate', 'exists'}
@@ -22,8 +21,9 @@
CACHEOPS_SENTINEL = {}
# NOTE: we don't use this fields in invalidator conditions since their
values could be very long
# and one should not filter by their equality anyway.
- CACHEOPS_SKIP_FIELDS = models.FileField, models.TextField,
models.BinaryField
+ CACHEOPS_SKIP_FIELDS = "FileField", "TextField", "BinaryField", "JSONField"
CACHEOPS_LONG_DISJUNCTION = 8
+ CACHEOPS_SERIALIZER = 'pickle'
FILE_CACHE_DIR = '/tmp/cacheops_file_cache'
FILE_CACHE_TIMEOUT = 60*60*24*30
@@ -32,8 +32,13 @@
class Settings(object):
def __getattr__(self, name):
res = getattr(base_settings, name, getattr(Defaults, name))
- if name == 'CACHEOPS_PREFIX':
- res = res if callable(res) else import_string(res)
+ if name in ['CACHEOPS_PREFIX', 'CACHEOPS_SERIALIZER']:
+ res = import_string(res) if isinstance(res, str) else res
+
+ # Convert old list of classes to list of strings
+ if name == 'CACHEOPS_SKIP_FIELDS':
+ res = [f if isinstance(f, str) else f.get_internal_type(res) for f
in res]
+
# Save to dict to speed up next access, __getattr__ won't be called
self.__dict__[name] = res
return res
@@ -42,6 +47,14 @@
setting_changed.connect(lambda setting, **kw: settings.__dict__.pop(setting,
None), weak=False)
+def import_string(path):
+ if "." in path:
+ module, attr = path.rsplit(".", 1)
+ return getattr(import_module(module), attr)
+ else:
+ return import_module(path)
+
+
@memoize
def prepare_profiles():
"""
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/django-cacheops-5.1/cacheops/invalidation.py
new/django-cacheops-6.0/cacheops/invalidation.py
--- old/django-cacheops-5.1/cacheops/invalidation.py 2020-05-15
12:03:28.000000000 +0200
+++ new/django-cacheops-6.0/cacheops/invalidation.py 2021-02-03
04:56:58.000000000 +0100
@@ -1,6 +1,6 @@
import json
import threading
-from funcy import memoize, post_processing, ContextDecorator
+from funcy import memoize, post_processing, ContextDecorator, decorator
from django.db import DEFAULT_DB_ALIAS
from django.db.models.expressions import F, Expression
@@ -14,6 +14,14 @@
__all__ = ('invalidate_obj', 'invalidate_model', 'invalidate_all',
'no_invalidation')
+@decorator
+def skip_on_no_invalidation(call):
+ if not settings.CACHEOPS_ENABLED or no_invalidation.active:
+ return
+ return call()
+
+
+@skip_on_no_invalidation
@queue_when_in_transaction
@handle_connection_failure
def invalidate_dict(model, obj_dict, using=DEFAULT_DB_ALIAS):
@@ -28,6 +36,7 @@
cache_invalidated.send(sender=model, obj_dict=obj_dict)
+@skip_on_no_invalidation
def invalidate_obj(obj, using=DEFAULT_DB_ALIAS):
"""
Invalidates caches that can possibly be influenced by object
@@ -36,6 +45,7 @@
invalidate_dict(model, get_obj_dict(model, obj), using=using)
+@skip_on_no_invalidation
@queue_when_in_transaction
@handle_connection_failure
def invalidate_model(model, using=DEFAULT_DB_ALIAS):
@@ -44,8 +54,6 @@
NOTE: This is a heavy artillery which uses redis KEYS request,
which could be relatively slow on large datasets.
"""
- if no_invalidation.active or not settings.CACHEOPS_ENABLED:
- return
model = model._meta.concrete_model
# NOTE: if we use sharding dependent on DNF then this will fail,
# which is ok, since it's hard/impossible to predict all the shards
@@ -58,10 +66,9 @@
cache_invalidated.send(sender=model, obj_dict=None)
+@skip_on_no_invalidation
@handle_connection_failure
def invalidate_all():
- if no_invalidation.active or not settings.CACHEOPS_ENABLED:
- return
redis_client.flushdb()
cache_invalidated.send(sender=None, obj_dict=None)
@@ -90,12 +97,17 @@
@memoize
def serializable_fields(model):
- return tuple(f for f in model._meta.fields
- if not isinstance(f, settings.CACHEOPS_SKIP_FIELDS))
+ return {f for f in model._meta.fields
+ if f.get_internal_type() not in settings.CACHEOPS_SKIP_FIELDS}
@post_processing(dict)
def get_obj_dict(model, obj):
for field in serializable_fields(model):
+ # Skip deferred fields, in post_delete trying to fetch them results in
error anyway.
+ # In post_save we rely on deferred values be the same as in pre_save.
+ if field.attname not in obj.__dict__:
+ continue
+
value = getattr(obj, field.attname)
if value is None:
yield field.attname, None
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/django-cacheops-5.1/cacheops/query.py
new/django-cacheops-6.0/cacheops/query.py
--- old/django-cacheops-5.1/cacheops/query.py 2020-09-26 10:56:10.000000000
+0200
+++ new/django-cacheops-6.0/cacheops/query.py 2021-04-30 11:04:51.000000000
+0200
@@ -1,13 +1,12 @@
import sys
import json
import threading
-import pickle
from random import random
from funcy import select_keys, cached_property, once, once_per, monkey, wraps,
walk, chain
from funcy import lmap, map, lcat, join_with
-from django.utils.encoding import smart_str, force_text
+from django.utils.encoding import force_str
from django.core.exceptions import ImproperlyConfigured, EmptyResultSet
from django.db import DEFAULT_DB_ALIAS
from django.db import models
@@ -23,12 +22,12 @@
MAX_GET_RESULTS = None
from .conf import model_profile, settings, ALL_OPS
-from .utils import monkey_mix, stamp_fields, func_cache_key, cached_view_fab,
family_has_profile
+from .utils import monkey_mix, stamp_fields, get_cache_key, cached_view_fab,
family_has_profile
from .utils import md5
from .sharding import get_prefix
from .redis import redis_client, handle_connection_failure, load_script
from .tree import dnfs
-from .invalidation import invalidate_obj, invalidate_dict, no_invalidation
+from .invalidation import invalidate_obj, invalidate_dict,
skip_on_no_invalidation
from .transaction import transaction_states
from .signals import cache_read
@@ -52,15 +51,14 @@
load_script('cache_thing', settings.CACHEOPS_LRU)(
keys=[prefix, cache_key, precall_key],
args=[
- pickle.dumps(data, -1),
+ settings.CACHEOPS_SERIALIZER.dumps(data),
json.dumps(cond_dnfs, default=str),
timeout
]
)
-def cached_as(*samples, timeout=None, extra=None, lock=None, keep_fresh=False,
- key_func=func_cache_key):
+def cached_as(*samples, timeout=None, extra=None, lock=None, keep_fresh=False):
"""
Caches results of a function and invalidates them same way as given
queryset(s).
NOTE: Ignores queryset cached ops settings, always caches.
@@ -92,8 +90,7 @@
querysets = lmap(_get_queryset, samples)
dbs = list({qs.db for qs in querysets})
cond_dnfs = join_with(lcat, map(dnfs, querysets))
- key_extra = [qs._cache_key(prefix=False) for qs in querysets]
- key_extra.append(extra)
+ qs_keys = [qs._cache_key(prefix=False) for qs in querysets]
if timeout is None:
timeout = min(qs._cacheprofile['timeout'] for qs in querysets)
if lock is None:
@@ -106,12 +103,13 @@
return func(*args, **kwargs)
prefix = get_prefix(func=func, _cond_dnfs=cond_dnfs, dbs=dbs)
- cache_key = prefix + 'as:' + key_func(func, args, kwargs,
key_extra)
+ extra_val = extra(*args, **kwargs) if callable(extra) else extra
+ cache_key = prefix + 'as:' + get_cache_key(func, args, kwargs,
qs_keys, extra_val)
with redis_client.getting(cache_key, lock=lock) as cache_data:
cache_read.send(sender=None, func=func, hit=cache_data is not
None)
if cache_data is not None:
- return pickle.loads(cache_data)
+ return settings.CACHEOPS_SERIALIZER.loads(cache_data)
else:
if keep_fresh:
# We call this "asp" for "as precall" because this key
is
@@ -119,7 +117,7 @@
# the key to prevent falsely thinking the key was not
# invalidated when in fact it was invalidated and the
# function was called again in another process.
- suffix = key_func(func, args, kwargs, key_extra +
[random()])
+ suffix = get_cache_key(func, args, kwargs, qs_keys,
extra_val, random())
precall_key = prefix + 'asp:' + suffix
# Cache a precall_key to watch for invalidation during
# the function call. Its value does not matter. If and
@@ -176,8 +174,8 @@
try:
sql_str = sql % params
except UnicodeDecodeError:
- sql_str = sql % walk(force_text, params)
- md.update(smart_str(sql_str))
+ sql_str = sql % walk(force_str, params)
+ md.update(force_str(sql_str))
except EmptyResultSet:
pass
# If query results differ depending on database
@@ -278,7 +276,7 @@
with redis_client.getting(cache_key, lock=lock) as cache_data:
cache_read.send(sender=self.model, func=None, hit=cache_data is
not None)
if cache_data is not None:
- self._result_cache = pickle.loads(cache_data)
+ self._result_cache =
settings.CACHEOPS_SERIALIZER.loads(cache_data)
else:
self._result_cache = list(self._iterable_class(self))
self._cache_results(cache_key, self._result_cache)
@@ -428,18 +426,18 @@
if cls.__module__ != '__fake__' and family_has_profile(cls):
self._install_cacheops(cls)
+ @skip_on_no_invalidation
def _pre_save(self, sender, instance, using, **kwargs):
- if not (instance.pk is None or instance._state.adding or
no_invalidation.active):
+ if instance.pk is not None and not instance._state.adding:
try:
+ # TODO: do not fetch non-serializable fields
_old_objs.__dict__[sender, instance.pk] \
= sender.objects.using(using).get(pk=instance.pk)
except sender.DoesNotExist:
pass
+ @skip_on_no_invalidation
def _post_save(self, sender, instance, using, **kwargs):
- if not settings.CACHEOPS_ENABLED or no_invalidation.active:
- return
-
# Invoke invalidations for both old and new versions of saved object
old = _old_objs.__dict__.pop((sender, instance.pk), None)
if old:
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/django-cacheops-5.1/cacheops/serializers.py
new/django-cacheops-6.0/cacheops/serializers.py
--- old/django-cacheops-5.1/cacheops/serializers.py 1970-01-01
01:00:00.000000000 +0100
+++ new/django-cacheops-6.0/cacheops/serializers.py 2021-04-30
10:44:54.000000000 +0200
@@ -0,0 +1,11 @@
+import pickle
+
+
+class PickleSerializer:
+ # properties
+ PickleError = pickle.PickleError
+ HIGHEST_PROTOCOL = pickle.HIGHEST_PROTOCOL
+
+ # methods
+ dumps = pickle.dumps
+ loads = pickle.loads
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/django-cacheops-5.1/cacheops/signals.py
new/django-cacheops-6.0/cacheops/signals.py
--- old/django-cacheops-5.1/cacheops/signals.py 2017-06-04 05:41:28.000000000
+0200
+++ new/django-cacheops-6.0/cacheops/signals.py 2021-02-19 04:50:58.000000000
+0100
@@ -1,4 +1,4 @@
import django.dispatch
-cache_read = django.dispatch.Signal(providing_args=["func", "hit"])
-cache_invalidated = django.dispatch.Signal(providing_args=["obj_dict"])
+cache_read = django.dispatch.Signal() # args: func, hit
+cache_invalidated = django.dispatch.Signal() # args: obj_dict
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/django-cacheops-5.1/cacheops/simple.py
new/django-cacheops-6.0/cacheops/simple.py
--- old/django-cacheops-5.1/cacheops/simple.py 2020-01-05 04:46:06.000000000
+0100
+++ new/django-cacheops-6.0/cacheops/simple.py 2021-05-01 11:57:11.000000000
+0200
@@ -1,12 +1,12 @@
import os
-import pickle
import time
from funcy import wraps
from .conf import settings
-from .utils import func_cache_key, cached_view_fab, md5hex
+from .utils import get_cache_key, cached_view_fab, md5hex
from .redis import redis_client, handle_connection_failure
+from .sharding import get_prefix
__all__ = ('cache', 'cached', 'cached_view', 'file_cache', 'CacheMiss',
'FileCache', 'RedisCache')
@@ -24,25 +24,29 @@
return self
def get(self):
- self.cache.get(self)
+ self.cache._get(self)
def set(self, value):
- self.cache.set(self, value, self.timeout)
+ self.cache._set(self, value, self.timeout)
def delete(self):
- self.cache.delete(self)
+ self.cache._delete(self)
class BaseCache(object):
"""
Simple cache with time-based invalidation
"""
- def cached(self, timeout=None, extra=None, key_func=func_cache_key):
+ def cached(self, timeout=None, extra=None):
"""
A decorator for caching function calls
"""
# Support @cached (without parentheses) form
if callable(timeout):
- return self.cached(key_func=key_func)(timeout)
+ return self.cached()(timeout)
+
+ def _get_key(func, args, kwargs):
+ extra_val = extra(*args, **kwargs) if callable(extra) else extra
+ return get_prefix(func=func) + 'c:' + get_cache_key(func, args,
kwargs, extra_val)
def decorator(func):
@wraps(func)
@@ -50,23 +54,21 @@
if not settings.CACHEOPS_ENABLED:
return func(*args, **kwargs)
- cache_key = 'c:' + key_func(func, args, kwargs, extra)
+ cache_key = _get_key(func, args, kwargs)
try:
- result = self.get(cache_key)
+ result = self._get(cache_key)
except CacheMiss:
result = func(*args, **kwargs)
- self.set(cache_key, result, timeout)
+ self._set(cache_key, result, timeout)
return result
def invalidate(*args, **kwargs):
- cache_key = 'c:' + key_func(func, args, kwargs, extra)
- self.delete(cache_key)
+ self._delete(_get_key(func, args, kwargs))
wrapper.invalidate = invalidate
def key(*args, **kwargs):
- cache_key = 'c:' + key_func(func, args, kwargs, extra)
- return CacheKey.make(cache_key, cache=self, timeout=timeout)
+ return CacheKey.make(_get_key(func, args, kwargs), cache=self,
timeout=timeout)
wrapper.key = key
return wrapper
@@ -77,27 +79,36 @@
return self.cached_view()(timeout)
return cached_view_fab(self.cached)(timeout=timeout, extra=extra)
+ def get(self, cache_key):
+ return self._get(get_prefix() + cache_key)
+
+ def set(self, cache_key, data, timeout=None):
+ self._set(get_prefix() + cache_key, data, timeout)
+
+ def delete(self, cache_key):
+ self._delete(get_prefix() + cache_key)
+
class RedisCache(BaseCache):
def __init__(self, conn):
self.conn = conn
- def get(self, cache_key):
+ def _get(self, cache_key):
data = self.conn.get(cache_key)
if data is None:
raise CacheMiss
- return pickle.loads(data)
+ return settings.CACHEOPS_SERIALIZER.loads(data)
@handle_connection_failure
- def set(self, cache_key, data, timeout=None):
- pickled_data = pickle.dumps(data, -1)
+ def _set(self, cache_key, data, timeout=None):
+ pickled_data = settings.CACHEOPS_SERIALIZER.dumps(data)
if timeout is not None:
self.conn.setex(cache_key, timeout, pickled_data)
else:
self.conn.set(cache_key, pickled_data)
@handle_connection_failure
- def delete(self, cache_key):
+ def _delete(self, cache_key):
self.conn.delete(cache_key)
cache = RedisCache(redis_client)
@@ -122,7 +133,7 @@
digest = md5hex(key)
return os.path.join(self._dir, digest[-2:], digest[:-2])
- def get(self, key):
+ def _get(self, key):
filename = self._key_to_filename(key)
try:
# Remove file if it's stale
@@ -131,11 +142,11 @@
raise CacheMiss
with open(filename, 'rb') as f:
- return pickle.load(f)
- except (IOError, OSError, EOFError, pickle.PickleError):
+ return settings.CACHEOPS_SERIALIZER.load(f)
+ except (IOError, OSError, EOFError):
raise CacheMiss
- def set(self, key, data, timeout=None):
+ def _set(self, key, data, timeout=None):
filename = self._key_to_filename(key)
dirname = os.path.dirname(filename)
@@ -149,7 +160,7 @@
# Use open with exclusive rights to prevent data corruption
f = os.open(filename, os.O_EXCL | os.O_WRONLY | os.O_CREAT)
try:
- os.write(f, pickle.dumps(data, pickle.HIGHEST_PROTOCOL))
+ os.write(f, settings.CACHEOPS_SERIALIZER.dumps(data))
finally:
os.close(f)
@@ -158,7 +169,7 @@
except (IOError, OSError):
pass
- def delete(self, fname):
+ def _delete(self, fname):
try:
os.remove(fname)
# Trying to remove directory in case it's empty
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/django-cacheops-5.1/cacheops/transaction.py
new/django-cacheops-6.0/cacheops/transaction.py
--- old/django-cacheops-5.1/cacheops/transaction.py 2020-01-05
04:24:04.000000000 +0100
+++ new/django-cacheops-6.0/cacheops/transaction.py 2021-04-30
06:08:02.000000000 +0200
@@ -3,7 +3,7 @@
from funcy import once, decorator
-from django.db import DEFAULT_DB_ALIAS
+from django.db import DEFAULT_DB_ALIAS, DatabaseError
from django.db.backends.utils import CursorWrapper
from django.db.transaction import Atomic, get_connection, on_commit
@@ -73,13 +73,17 @@
def __exit__(self, exc_type, exc_value, traceback):
connection = get_connection(self.using)
- self._no_monkey.__exit__(self, exc_type, exc_value, traceback)
- if not connection.closed_in_transaction and exc_type is None and \
- not connection.needs_rollback:
- if transaction_states[self.using]:
- transaction_states[self.using].commit()
- else:
+ try:
+ self._no_monkey.__exit__(self, exc_type, exc_value, traceback)
+ except DatabaseError:
transaction_states[self.using].rollback()
+ else:
+ if not connection.closed_in_transaction and exc_type is None and \
+ not connection.needs_rollback:
+ if transaction_states[self.using]:
+ transaction_states[self.using].commit()
+ else:
+ transaction_states[self.using].rollback()
class CursorWrapperMixin(object):
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/django-cacheops-5.1/cacheops/tree.py
new/django-cacheops-6.0/cacheops/tree.py
--- old/django-cacheops-5.1/cacheops/tree.py 2020-09-26 11:01:39.000000000
+0200
+++ new/django-cacheops-6.0/cacheops/tree.py 2021-02-03 04:56:09.000000000
+0100
@@ -10,6 +10,7 @@
from django.db.models.expressions import BaseExpression, Exists
from .conf import settings
+from .invalidation import serializable_fields
def dnfs(qs):
@@ -44,7 +45,7 @@
if isinstance(where.rhs, (QuerySet, Query, BaseExpression)):
return SOME_TREE
# Skip conditions on non-serialized fields
- if isinstance(where.lhs.target, settings.CACHEOPS_SKIP_FIELDS):
+ if where.lhs.target not in
serializable_fields(where.lhs.target.model):
return SOME_TREE
attname = where.lhs.target.attname
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/django-cacheops-5.1/cacheops/utils.py
new/django-cacheops-6.0/cacheops/utils.py
--- old/django-cacheops-5.1/cacheops/utils.py 2020-06-21 07:49:11.000000000
+0200
+++ new/django-cacheops-6.0/cacheops/utils.py 2021-04-23 06:10:03.000000000
+0200
@@ -77,6 +77,8 @@
def obj_key(obj):
if isinstance(obj, models.Model):
return '%s.%s.%s' % (obj._meta.app_label, obj._meta.model_name, obj.pk)
+ elif hasattr(obj, 'build_absolute_uri'):
+ return obj.build_absolute_uri() # Only vary HttpRequest by uri
elif inspect.isfunction(obj):
factors = [obj.__module__, obj.__name__]
# Really useful to ignore this while code still in development
@@ -86,24 +88,9 @@
else:
return str(obj)
-def func_cache_key(func, args, kwargs, extra=None):
- """
- Calculate cache key based on func and arguments
- """
- factors = [func, args, kwargs, extra]
+def get_cache_key(*factors):
return md5hex(json.dumps(factors, sort_keys=True, default=obj_key))
-def view_cache_key(func, args, kwargs, extra=None):
- """
- Calculate cache key for view func.
- Use url instead of not properly serializable request argument.
- """
- if hasattr(args[0], 'build_absolute_uri'):
- uri = args[0].build_absolute_uri()
- else:
- uri = args[0]
- return 'v:' + func_cache_key(func, args[1:], kwargs, extra=(uri, extra))
-
def cached_view_fab(_cached):
def force_render(response):
if hasattr(response, 'render') and callable(response.render):
@@ -112,7 +99,6 @@
def cached_view(*dargs, **dkwargs):
def decorator(func):
- dkwargs['key_func'] = view_cache_key
cached_func = _cached(*dargs, **dkwargs)(compose(force_render,
func))
@wraps(func)
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore'
old/django-cacheops-5.1/django_cacheops.egg-info/PKG-INFO
new/django-cacheops-6.0/django_cacheops.egg-info/PKG-INFO
--- old/django-cacheops-5.1/django_cacheops.egg-info/PKG-INFO 2020-10-25
15:11:24.000000000 +0100
+++ new/django-cacheops-6.0/django_cacheops.egg-info/PKG-INFO 2021-05-03
13:09:38.000000000 +0200
@@ -1,6 +1,6 @@
Metadata-Version: 1.2
Name: django-cacheops
-Version: 5.1
+Version: 6.0
Summary: A slick ORM cache with automatic granular event-driven invalidation
for Django.
Home-page: http://github.com/Suor/django-cacheops
Author: Alexander Schepanovski
@@ -275,9 +275,18 @@
@cached_view_as(News)
def news_index(request):
# ...
- return HttpResponse(...)
+ return render(...)
+
+ You can pass ``timeout``, ``extra`` and several samples the same way
as to ``@cached_as()``. Note that you can pass a function as ``extra``:
- You can pass ``timeout``, ``extra`` and several samples the same way
as to ``@cached_as()``.
+ .. code:: python
+
+ @cached_view_as(News, extra=lambda req: req.user.is_staff)
+ def news_index(request):
+ # ... add extra things for staff
+ return render(...)
+
+ A function passed as ``extra`` receives the same arguments as the
cached function.
Class based views can also be cached:
@@ -286,7 +295,7 @@
class NewsIndex(ListView):
model = News
- news_index = cached_view_as(News)(NewsIndex.as_view())
+ news_index = cached_view_as(News, ...)(NewsIndex.as_view())
Invalidation
@@ -663,6 +672,27 @@
**NOTE:** prefix is not used in simple and file cache. This might
change in future cacheops.
+ Custom serialization
+ --------------------
+
+ Cacheops uses ``pickle`` by default, employing it's default protocol.
But you can specify your own
+ it might be any module or a class having `.dumps()` and `.loads()`
functions. For example you can use ``dill`` instead, which can serialize more
things like anonymous functions:
+
+ .. code:: python
+
+ CACHEOPS_SERIALIZER = 'dill'
+
+ One less obvious use is to fix pickle protocol, to use cacheops cache
across python versions:
+
+ .. code:: python
+
+ import pickle
+
+ class CACHEOPS_SERIALIZER:
+ dumps = lambda data: pickle.dumps(data, 3)
+ loads = pickle.loads
+
+
Using memory limit
------------------
@@ -804,10 +834,13 @@
Classifier: Programming Language :: Python :: 3.6
Classifier: Programming Language :: Python :: 3.7
Classifier: Programming Language :: Python :: 3.8
+Classifier: Programming Language :: Python :: 3.9
Classifier: Framework :: Django
Classifier: Framework :: Django :: 2.1
Classifier: Framework :: Django :: 2.2
Classifier: Framework :: Django :: 3.0
+Classifier: Framework :: Django :: 3.1
+Classifier: Framework :: Django :: 3.2
Classifier: Environment :: Web Environment
Classifier: Intended Audience :: Developers
Classifier: Topic :: Internet :: WWW/HTTP
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore'
old/django-cacheops-5.1/django_cacheops.egg-info/SOURCES.txt
new/django-cacheops-6.0/django_cacheops.egg-info/SOURCES.txt
--- old/django-cacheops-5.1/django_cacheops.egg-info/SOURCES.txt
2020-10-25 15:11:24.000000000 +0100
+++ new/django-cacheops-6.0/django_cacheops.egg-info/SOURCES.txt
2021-05-03 13:09:39.000000000 +0200
@@ -15,6 +15,7 @@
cacheops/jinja2.py
cacheops/query.py
cacheops/redis.py
+cacheops/serializers.py
cacheops/sharding.py
cacheops/signals.py
cacheops/simple.py
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/django-cacheops-5.1/requirements-test.txt
new/django-cacheops-6.0/requirements-test.txt
--- old/django-cacheops-5.1/requirements-test.txt 2020-08-06
09:58:02.000000000 +0200
+++ new/django-cacheops-6.0/requirements-test.txt 2021-04-30
12:57:36.000000000 +0200
@@ -4,3 +4,4 @@
six>=1.4.0
before_after==1.0.0
jinja2>=2.10
+dill
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/django-cacheops-5.1/setup.py
new/django-cacheops-6.0/setup.py
--- old/django-cacheops-5.1/setup.py 2020-10-25 14:48:47.000000000 +0100
+++ new/django-cacheops-6.0/setup.py 2021-05-03 11:10:18.000000000 +0200
@@ -10,7 +10,7 @@
setup(
name='django-cacheops',
- version='5.1',
+ version='6.0',
author='Alexander Schepanovski',
author_email='[email protected]',
@@ -41,10 +41,13 @@
'Programming Language :: Python :: 3.6',
'Programming Language :: Python :: 3.7',
'Programming Language :: Python :: 3.8',
+ 'Programming Language :: Python :: 3.9',
'Framework :: Django',
'Framework :: Django :: 2.1',
'Framework :: Django :: 2.2',
'Framework :: Django :: 3.0',
+ 'Framework :: Django :: 3.1',
+ 'Framework :: Django :: 3.2',
'Environment :: Web Environment',
'Intended Audience :: Developers',
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/django-cacheops-5.1/tests/bench.py
new/django-cacheops-6.0/tests/bench.py
--- old/django-cacheops-5.1/tests/bench.py 2020-01-05 04:34:45.000000000
+0100
+++ new/django-cacheops-6.0/tests/bench.py 2021-04-30 11:06:03.000000000
+0200
@@ -1,6 +1,5 @@
-import pickle
-
from cacheops import invalidate_obj, invalidate_model
+from cacheops.conf import settings
from cacheops.redis import redis_client
from cacheops.tree import dnfs
@@ -8,13 +7,13 @@
posts = list(Post.objects.cache().all())
-posts_pickle = pickle.dumps(posts, -1)
+posts_pickle = settings.CACHEOPS_SERIALIZER.dumps(posts)
def do_pickle():
- pickle.dumps(posts, -1)
+ settings.CACHEOPS_SERIALIZER.dumps(posts)
def do_unpickle():
- pickle.loads(posts_pickle)
+ settings.CACHEOPS_SERIALIZER.loads(posts_pickle)
get_key = Category.objects.filter(pk=1).order_by()._cache_key()
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/django-cacheops-5.1/tests/models.py
new/django-cacheops-6.0/tests/models.py
--- old/django-cacheops-5.1/tests/models.py 2020-06-19 08:21:05.000000000
+0200
+++ new/django-cacheops-6.0/tests/models.py 2021-04-30 10:44:54.000000000
+0200
@@ -106,14 +106,17 @@
# TODO: check other new fields:
-# - PostgreSQL ones: ArrayField, HStoreField, RangeFields, unaccent
+# - PostgreSQL ones: HStoreField, RangeFields, unaccent
# - Other: DurationField
if os.environ.get('CACHEOPS_DB') in {'postgresql', 'postgis'}:
from django.contrib.postgres.fields import ArrayField
try:
- from django.contrib.postgres.fields import JSONField
+ from django.db.models import JSONField
except ImportError:
- JSONField = None
+ try:
+ from django.contrib.postgres.fields import JSONField # Used
before Django 3.1
+ except ImportError:
+ JSONField = None
class TaggedPost(models.Model):
name = models.CharField(max_length=200)
@@ -267,3 +270,19 @@
blank=True,
null=True
)
+
+
+# 385
+class Client(models.Model):
+ def __init__(self, *args, **kwargs):
+ # copied from Django 2.1.5 (not exists in Django 3.1.5 installed by
current requirements)
+ def curry(_curried_func, *args, **kwargs):
+ def _curried(*moreargs, **morekwargs):
+ return _curried_func(*args, *moreargs, **{**kwargs,
**morekwargs})
+
+ return _curried
+
+ super().__init__(*args, **kwargs)
+ setattr(self, '_get_private_data', curry(sum, [1, 2, 3, 4]))
+
+ name = models.CharField(max_length=255)
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/django-cacheops-5.1/tests/settings.py
new/django-cacheops-6.0/tests/settings.py
--- old/django-cacheops-5.1/tests/settings.py 2020-07-09 11:12:51.000000000
+0200
+++ new/django-cacheops-6.0/tests/settings.py 2021-04-23 08:08:01.000000000
+0200
@@ -13,6 +13,8 @@
AUTH_PROFILE_MODULE = 'tests.UserProfile'
+DEFAULT_AUTO_FIELD = 'django.db.models.AutoField'
+
# Django replaces this, but it still wants it. *shrugs*
DATABASE_ENGINE = 'django.db.backends.sqlite3',
if os.environ.get('CACHEOPS_DB') == 'postgresql':
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/django-cacheops-5.1/tests/tests.py
new/django-cacheops-6.0/tests/tests.py
--- old/django-cacheops-5.1/tests/tests.py 2020-10-25 14:15:12.000000000
+0100
+++ new/django-cacheops-6.0/tests/tests.py 2021-05-01 11:57:55.000000000
+0200
@@ -1,5 +1,6 @@
from contextlib import contextmanager
import re
+import platform
import unittest
import mock
@@ -638,6 +639,27 @@
categories =
Category.objects.cache().annotate(newest_post=Subquery(newest_post[:1]))
self.assertEqual(categories[0].newest_post, post.pk)
+ @unittest.skipIf(platform.python_implementation() == "PyPy", "dill doesn't
do that in PyPy")
+ def test_385(self):
+ Client.objects.create(name='Client Name')
+
+ with self.assertRaises(AttributeError) as e:
+ Client.objects.filter(name='Client Name').cache().first()
+ self.assertEqual(
+ str(e.exception),
+ "Can't pickle local object
'Client.__init__.<locals>.curry.<locals>._curried'")
+
+ invalidate_model(Client)
+
+ with override_settings(CACHEOPS_SERIALIZER='dill'):
+ with self.assertNumQueries(1):
+ Client.objects.filter(name='Client Name').cache().first()
+ Client.objects.filter(name='Client Name').cache().first()
+
+ def test_387(self):
+ post = Post.objects.defer("visible").last()
+ post.delete()
+
class RelatedTests(BaseTestCase):
fixtures = ['basic']
@@ -912,12 +934,7 @@
class SimpleCacheTests(BaseTestCase):
def test_cached(self):
- calls = [0]
-
- @cached(timeout=100)
- def get_calls(_):
- calls[0] += 1
- return calls[0]
+ get_calls = make_inc(cached(timeout=100))
self.assertEqual(get_calls(1), 1)
self.assertEqual(get_calls(1), 1)
@@ -932,12 +949,7 @@
self.assertEqual(get_calls(2), 42)
def test_cached_view(self):
- calls = [0]
-
- @cached_view(timeout=100)
- def get_calls(request):
- calls[0] += 1
- return calls[0]
+ get_calls = make_inc(cached_view(timeout=100))
factory = RequestFactory()
r1 = factory.get('/hi')
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/django-cacheops-5.1/tests/tests_sharding.py
new/django-cacheops-6.0/tests/tests_sharding.py
--- old/django-cacheops-5.1/tests/tests_sharding.py 2020-01-05
04:48:16.000000000 +0100
+++ new/django-cacheops-6.0/tests/tests_sharding.py 2021-05-03
05:26:29.000000000 +0200
@@ -1,6 +1,7 @@
from django.core.exceptions import ImproperlyConfigured
from django.test import override_settings
+from cacheops import cache, CacheMiss
from .models import Category, Post, Extra
from .utils import BaseTestCase
@@ -41,3 +42,13 @@
def test_union_tables(self):
qs = Post.objects.filter(pk=1).union(Post.objects.filter(pk=2)).cache()
list(qs)
+
+
+class SimpleCacheTests(BaseTestCase):
+ def test_prefix(self):
+ with override_settings(CACHEOPS_PREFIX=lambda _: 'a'):
+ cache.set("key", "value")
+ self.assertEqual(cache.get("key"), "value")
+
+ with self.assertRaises(CacheMiss):
+ cache.get("key")
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/django-cacheops-5.1/tests/tests_transactions.py
new/django-cacheops-6.0/tests/tests_transactions.py
--- old/django-cacheops-5.1/tests/tests_transactions.py 2020-01-05
04:22:19.000000000 +0100
+++ new/django-cacheops-6.0/tests/tests_transactions.py 2021-04-30
06:08:02.000000000 +0200
@@ -1,10 +1,10 @@
-from django.db import connection
+from django.db import connection, IntegrityError
from django.db.transaction import atomic
from django.test import TransactionTestCase
from cacheops.transaction import queue_when_in_transaction
-from .models import Category
+from .models import Category, Post
from .utils import run_in_thread
@@ -91,6 +91,22 @@
with self.assertNumQueries(1):
get_category()
+ def test_rollback_during_integrity_error(self):
+ # store category in cache
+ get_category()
+
+ # Make current DB be "dirty" by write
+ try:
+ with atomic():
+ Post.objects.create(category_id=-1, title='')
+ except IntegrityError:
+ # however, this write should be rolled back and current DB should
+ # not be "dirty"
+ pass
+
+ with self.assertNumQueries(0):
+ get_category()
+
def test_call_cacheops_cbs_before_on_commit_cbs(self):
calls = []
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/django-cacheops-5.1/tox.ini
new/django-cacheops-6.0/tox.ini
--- old/django-cacheops-5.1/tox.ini 2020-06-19 15:43:01.000000000 +0200
+++ new/django-cacheops-6.0/tox.ini 2021-05-01 11:48:33.000000000 +0200
@@ -2,11 +2,11 @@
minversion = 2.7
envlist =
lint,
- py35-dj{21,22},
- py36-dj{21,22,30},
- py37-dj{21,22,30,master},
- py38-dj{21,22,30,master},
- pypy3-dj{21,22,30}
+ py36-dj{30,31},
+ py37-dj{30,31,32},
+ py38-dj{30,31,32,master},
+ py39-dj{30,31,32,master},
+ pypy3-dj{30,31,32}
[travis:env]
@@ -14,6 +14,8 @@
2.1: dj21
2.2: dj22
3.0: dj30
+ 3.1: dj31
+ 3.2: dj32
master: djmaster
@@ -27,13 +29,16 @@
dj21: Django>=2.1,<2.2
dj22: Django>=2.2,<2.3
dj30: Django>=3.0a1,<3.1
+ dj31: Django>=3.1,<3.2
+ dj32: Django>=3.2,<3.3
djmaster: git+https://github.com/django/django
mysqlclient
- py{35,36,37,38}: psycopg2
+ py{35,36,37,38,39}: psycopg2
; gdal=={env:GDAL_VERSION:2.4}
pypy3: psycopg2cffi>=2.7.6
before_after==1.0.0
jinja2>=2.10
+ dill
commands =
./run_tests.py []
env CACHEOPS_PREFIX=1 ./run_tests.py []
