EBernhardson has uploaded a new change for review.
https://gerrit.wikimedia.org/r/292955
Change subject: Handle es 2.x document missing exceptions
......................................................................
Handle es 2.x document missing exceptions
ES 2.x made error responses more structured. This is good because it
means we can more explicitly test for errors, but it broke the existing
code. Will need to re-run this weeks popularity score export after
deploying this patch.
Change-Id: Ia8f469a28c1b657c4cc44f27bf4f8edabfa4c685
---
M oozie/transfer_to_es/transferToES.py
1 file changed, 7 insertions(+), 1 deletion(-)
git pull ssh://gerrit.wikimedia.org:29418/wikimedia/discovery/analytics
refs/changes/55/292955/1
diff --git a/oozie/transfer_to_es/transferToES.py
b/oozie/transfer_to_es/transferToES.py
index 73b3656..eb0328e 100644
--- a/oozie/transfer_to_es/transferToES.py
+++ b/oozie/transfer_to_es/transferToES.py
@@ -91,6 +91,12 @@
updateCounter.add(1)
return True
+ def justDocumentMissingError(item):
+ for error in item['update']['error']:
+ if error['type'] != 'document_missing_exception':
+ return False
+ return True
+
def parseResponse(resp):
respData = resp.json()
for item in respData.get('items', {}):
@@ -99,7 +105,7 @@
if 'status' not in item['update']:
continue
if item['update']['status'] != 200:
- if 'error' in item['update'] and item['update']['error'][0:24]
== 'DocumentMissingException':
+ if 'error' in item['update'] and
justDocumentMissingError(item):
continue
failedDocumentCounter.add(1)
--
To view, visit https://gerrit.wikimedia.org/r/292955
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ia8f469a28c1b657c4cc44f27bf4f8edabfa4c685
Gerrit-PatchSet: 1
Gerrit-Project: wikimedia/discovery/analytics
Gerrit-Branch: master
Gerrit-Owner: EBernhardson <[email protected]>
_______________________________________________
MediaWiki-commits mailing list
[email protected]
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
