Filippo Giunchedi has uploaded a new change for review.
https://gerrit.wikimedia.org/r/272455
Change subject: swiftrepl: fix destination container listing limit
......................................................................
swiftrepl: fix destination container listing limit
while testing the latest version of swiftrepl I came across an http error from
swift "412 Precondition failed". As it turns out swiftrepl was asking container
listings with a limit of 20000 but default maximum is 10000.
My understanding of what happened is this: 'limit' variable assignment was
removed in I148b0c209 from the function, however a global 'limit' variable was
introduced in I464b6a68. In I66a04cef1 there's been a cleanup of variables but
'limit' was re-introduced outside the 'while True' loop, not inside.
Change-Id: I17614830f79e7806b1d253df8209473e89f9fb41
---
M swiftrepl/swiftrepl.py
1 file changed, 5 insertions(+), 5 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/operations/software
refs/changes/55/272455/1
diff --git a/swiftrepl/swiftrepl.py b/swiftrepl/swiftrepl.py
index 10b8a29..e1152e8 100644
--- a/swiftrepl/swiftrepl.py
+++ b/swiftrepl/swiftrepl.py
@@ -25,7 +25,7 @@
copy_headers = re.compile(r'^X-Content-Duration$', flags=re.IGNORECASE)
NOBJECT = 1000
-PER_PAGE = 10000
+LIMIT_MAX = 10000
src = {}
dst = {}
@@ -302,15 +302,15 @@
dstconn = None
dstobjects = None
- limit = PER_PAGE
while True:
srcobjects = get_container_objects(srccontainer, limit=NOBJECT,
marker=last, connpool=srcconnpool)
+ limit = NOBJECT
while dstobjects is None or (len(dstobjects) >= limit and
dstobjects[-1].name < srcobjects[-1].name):
dstobjects = get_container_objects(dstcontainer, limit=limit,
marker=last, connpool=dstconnpool)
if len(dstobjects) == limit:
limit *= 2
- if limit > 10000:
+ if limit > LIMIT_MAX:
dstobjects = None
break
@@ -568,12 +568,12 @@
last = ''
while True:
- page = srcconn.get_all_containers(limit=PER_PAGE, marker=last)
+ page = srcconn.get_all_containers(limit=LIMIT_MAX, marker=last)
if len(page) == 0:
break
last = page[-1].name.encode("utf-8")
containers.extend(page)
- if len(page) < PER_PAGE:
+ if len(page) < LIMIT_MAX:
break
containerlist = [container for container in containers
--
To view, visit https://gerrit.wikimedia.org/r/272455
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I17614830f79e7806b1d253df8209473e89f9fb41
Gerrit-PatchSet: 1
Gerrit-Project: operations/software
Gerrit-Branch: master
Gerrit-Owner: Filippo Giunchedi <[email protected]>
_______________________________________________
MediaWiki-commits mailing list
[email protected]
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
