Reviewed: https://reviews.mahara.org/4831 Committed: https://git.nzoss.org.nz/mahara/mahara/commit/f7194ce1c0c47c9da1b69026e61b71f9024b1a86 Submitter: Robert Lyon ([email protected]) Branch: master
commit f7194ce1c0c47c9da1b69026e61b71f9024b1a86 Author: Aaron Wells <[email protected]> Date: Tue Jun 9 18:00:47 2015 +1200 More robust handling of Elasticsearch bulk operations Bug 1457712. This patch accomplishes four main things: 1. Sets a limit on the number on the number of documents per Elasticsearch bulk request. 2. Doesn't delete records from the queue table until after they have been successfully sent 3. If a bulk request fails, later retries the records individually 4. Performs deletion in bulk Change-Id: I9ac5e3a33b473e256fdf331800dc60101c126dcc -- You received this bug notification because you are a member of Mahara Contributors, which is subscribed to Mahara. Matching subscriptions: Subscription for all Mahara Contributors -- please ask on #mahara-dev or mahara.org forum before editing or unsubscribing it! https://bugs.launchpad.net/bugs/1457712 Title: More robust Elasticsearch data uploading Status in Mahara ePortfolio: Fix Committed Bug description: The search/elasticsearch plugin has a cron job that loads data into the Elasticsearch server. Currently, this cron job has a couple of problems: 1. It sends all the data from search_elasticsearch_queue in just one /_bulk operation. This means that if you've got a proxy server in front of your Elasticsearch server, it's quite likely to reject the request because it'll be larger than the allowed request size. 2. It deletes the records from the queue table *before* attempting to upload them to Elasticsearch. If the upload fails, all of the records that were part of that bulk operation are basically lost and never get indexed. To manage notifications about this bug go to: https://bugs.launchpad.net/mahara/+bug/1457712/+subscriptions _______________________________________________ Mailing list: https://launchpad.net/~mahara-contributors Post to : [email protected] Unsubscribe : https://launchpad.net/~mahara-contributors More help : https://help.launchpad.net/ListHelp
