Hi Tim, you can use promises to handle the async requests, or simply use synchronous requests. No definite answers here but I find using synchronous requests easier to work with and user interface responsiveness is not an issue for server side ad-hoc scripts like this.
The urllib-sync library will wrap urllib and provide sync behavior: https://www.npmjs.com/package/urllib-sync Install with npm: npm install urllib-sync Import: var urlsync = require('urllib-sync'); Then do synchronous requests/responses like this: var response = urlsync.request(url, postOptions); Lars On Tue, Aug 25, 2015 at 10:35 AM, Mark Polak <[email protected]> wrote: > Hey Timothy, > > Is there a reason why you would not want to wait for the response? You > have figured out you're sort of flooding the server with a bunch of > requests. > > You can still send them in batches of say 5, wait till those are done then > send the next 5, etc. > > Node now supports Promises so you could transform your requests into > promises and handle them in batches by using Promise.all(). > > The code below i just wrote up quickly without running it so you might > need to see if it works and adjust it, but it was more to give you an idea. > > Regards, > > Mark > > function doRequests(dataElementsToUpdate) { > //Map over the items > return dataElementsToUpdate.map(function (dataElementToUpdate) { > //Create a promise for each item and call resolve when that > request is done. > return new Promise(function (resolve, reject) { > > var postData = JSON.stringify(dataElementsToUpdate[i]); > serverConfigPOST.dhis2server.path = > "/dhis/api/sharing?type=dataElement&id=" + dataElementsToUpdate[i]. > object.id; > serverConfigPOST.dhis2server.headers = { > 'Content-Type': 'application/json', > 'Content-Length': postData.length > }; > console.log(serverConfigPOST); > > var req = https.request(serverConfigPOST.dhis2server, > function(res) { > console.log('STATUS: ' + res.statusCode); > console.log('HEADERS: ' + JSON.stringify(res.headers)); > res.setEncoding('utf8'); > res.on('data', function (chunk) { > console.log('BODY: ' + chunk); > }); > res.on('end', function () { > resolve({item: dataElementsToUpdate[i].object, > success: true); > }); > }); > > req.on('error', function(e) { > //console.log('problem with request: ' + e.message); > resolve({item: dataElementsToUpdate[i].object, > success: false, message: e.message); > }); > }); > }); > } > > function updateSharingInBatches() { > //Pick 5 elements to process next > var itemsToProcess = dataElementsToUpdate.splice(0, 5); > > //Only continue if there are more elements > if (itemsToProcess.length) { > //Wait til the passed batch is done and excute the next after > return Promise.all(doRequests(itemsToProcess)) > .then(function (responses) { > updateSharingInBatches(dataElementsToUpdate); > }) > //If you want to > .catch(function (responses) { > console.log('One or more did not work'); > //If you wish to continue with the batches you can just > call > //updateSharingInBatches(dataElementsToUpdate); > //Or you can try run the failed one again. :) > }) > } > } > > //Run the whole thing > updateSharingInBatches(dataElementsToUpdate); > > > > On Mon, Aug 24, 2015 at 6:45 PM, Timothy Harding <[email protected]> > wrote: > >> Thanks Pierre, >> >> Not sure how I would set up the recursive loop in javascript, I had heard >> about async from stack overflow as well, I was thinking of trying that, but >> I kind of like how quickly this works, not needing to wait for the response >> from one request before sending the next. The DHIS2 hardware I am working >> with just doesn't seem to keep up after a certain number of requests >> however, hence my attempt to slow them down. >> >> Timothy Harding >> RPCV Vanuatu >> Skype: [email protected] >> +1 (541) 632-6623 >> >> On Mon, Aug 24, 2015 at 9:39 AM, Pierre Dane <[email protected]> wrote: >> >>> might be best to try a recursive loop using setInterval or >>> setTimeout with a configurable value (I.e. a method calling itself) . or >>> you could look at the async library to make things synchronous . >>> >>> >>> On Monday, August 24, 2015, Timothy Harding <[email protected]> wrote: >>> >>>> Hello DHIS2 Devs, >>>> >>>> I've got a script that works pretty well for updating a bunch of >>>> sharing settings for various objects using node.js. The only problem is, I >>>> am getting timeouts when the object update counts get too high (500+). >>>> >>>> I am a bit new to the asynchronous nature of node.js, and I like that I >>>> can bundle a lot of updates into one without needing to worry that I need a >>>> response for one before sending the request for the next. The downside is, >>>> I have no idea where to put the sleep timer to slow down the volume of >>>> requests. Everywhere I've put it so far increases the timeouts, rather than >>>> decreases them. I know there are some node.js experts out there on the >>>> DHIS2 dev list so, perhaps someone has run into something similar: >>>> >>>> var serverConfigPOST = serverConfig; >>>> serverConfigPOST.dhis2server.method = "POST"; >>>> var i=0; >>>> for(i=0; i < dataElementsToUpdate.length; i++){ >>>> >>>> Tried here >>>> var postData = JSON.stringify(dataElementsToUpdate[i]); >>>> serverConfigPOST.dhis2server.path = >>>> "/dhis/api/sharing?type=dataElement&id=" + dataElementsToUpdate[i]. >>>> object.id; >>>> serverConfigPOST.dhis2server.headers = { >>>> 'Content-Type': 'application/json', >>>> 'Content-Length': postData.length >>>> }; >>>> console.log(serverConfigPOST); >>>> var req = https.request(serverConfigPOST.dhis2server, function(res) { >>>> console.log('STATUS: ' + res.statusCode); >>>> console.log('HEADERS: ' + JSON.stringify(res.headers)); >>>> res.setEncoding('utf8'); >>>> res.on('data', function (chunk) { >>>> console.log('BODY: ' + chunk); >>>> }); >>>> }); >>>> req.on('error', function(e) { >>>> //console.log('problem with request: ' + e.message); >>>> }); >>>> // write data to request body >>>> req.write(postData); >>>> req.end(); >>>> >>>> Tried here >>>> // if(i==20){ >>>> // break; >>>> // } >>>> } >>>> >>>> Not quite sure how to slow this one down, I may just have to run/rerun >>>> it with different starting and stopping i values in the mean time. >>>> >>>> Timothy Harding >>>> RPCV Vanuatu >>>> Skype: [email protected] >>>> +1 (541) 632-6623 >>>> >>> >>> >>> -- >>> Pierre Dane >>> >>> Development Manager >>> Jembi Health Systems >>> >>> >> >> _______________________________________________ >> Mailing list: https://launchpad.net/~dhis2-devs >> Post to : [email protected] >> Unsubscribe : https://launchpad.net/~dhis2-devs >> More help : https://help.launchpad.net/ListHelp >> >> > > _______________________________________________ > Mailing list: https://launchpad.net/~dhis2-devs > Post to : [email protected] > Unsubscribe : https://launchpad.net/~dhis2-devs > More help : https://help.launchpad.net/ListHelp > > -- Lars Helge Øverland Lead developer, DHIS 2 University of Oslo Skype: larshelgeoverland http://www.dhis2.org <https://www.dhis2.org>
_______________________________________________ Mailing list: https://launchpad.net/~dhis2-devs Post to : [email protected] Unsubscribe : https://launchpad.net/~dhis2-devs More help : https://help.launchpad.net/ListHelp
