Re: How to deal with outbound http calls

[Dave Watts]

> I have a web app that at times can make many outbound http calls.  For 
> example I have a page that collects various rss feeds
> from other sites and then parses them and finally displays them all on one 
> page for the user.  I have a few questions.:
>
> Is there a limit to the number of outbound http calls CFMX 9 can make?

Not specifically, but as Russ and Cameron have mentioned, there is a
limit to the number of threads that can run. And running serial HTTP
requests is going to be sloow. You could increase the thread
count, then use CFTHREAD to run some of these concurrently if you
really need to be doing them at serve time.

> Is there a way I can see the total number of outbound http calls going on?

I'm not aware of anything in CF that would specifically let you see
the number of outbound HTTP requests, but you could certainly see what
scripts are being executed by what threads, so if you had a concurrent
CFTHREAD approach, you could get a pretty accurate count. You could
monitor network connections outside of CF to see how many outbound
HTTP requests the machine is making, of course.

> It would seem to me that this is a poor design under load conditions.  Is 
> there a better way to fetch off site content?

Yes, you should ideally do it either before (via caching) or after
(via asynchronous processing) the user wants this content, instead of
exactly when the user wants it.

Dave Watts, CTO, Fig Leaf Software
http://www.figleaf.com/
http://training.figleaf.com/

Fig Leaf Software is a Veteran-Owned Small Business (VOSB) on
GSA Schedule, and provides the highest caliber vendor-authorized
instruction at our training centers, online, or onsite.

~|
Order the Adobe Coldfusion Anthology now!
http://www.amazon.com/Adobe-Coldfusion-Anthology/dp/1430272155/?tag=houseoffusion
Archive: 
http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:353240
Subscription: http://www.houseoffusion.com/groups/cf-talk/subscribe.cfm
Unsubscribe: http://www.houseoffusion.com/groups/cf-talk/unsubscribe.cfm

RE: How to deal with outbound http calls

[DeMarco, Alex]

Yes this is happening real time.  Unfortunately I cannot make code changes only 
advise those who can.

Thanks for all the comments!

- Alex 

-Original Message-
From: Cameron Childress [mailto:camer...@gmail.com] 
Sent: Sunday, November 18, 2012 4:43 PM
To: cf-talk
Subject: Re: How to deal with outbound http calls


On Sun, Nov 18, 2012 at 7:48 AM, DeMarco, Alex wrote:

> I have a web app that at times can make many outbound http calls.  For 
> example I have a page that collects various rss feeds from other sites 
> and then parses them and finally displays them all on one page for the 
> user.  I have a few questions.:
>
> Is there a limit to the number of outbound http calls CFMX 9 can make?
>

Assuming you aren't in a cluster, some creative use of CFLOCK would do the 
trick. If you aren't familiar with CFLOCK, check the documentation. You could 
also reduce the number of threads as Russ suggests, however I would recommend 
putting some code in place to do something when that max is reached or else 
you're just creating a new problem from the old one.


> Is there a way I can see the total number of outbound http calls going on?
>

Outside of CF there would be a number of network monitoring tools you could use 
for this. You could also do some creative application or session scope 
monitoring of start and stop counts for certain activities.


> It would seem to me that this is a poor design under load conditions.  
> Is there a better way to fetch off site content?


Are you doing it in realtime when a user's requests a page? Yes, that is poor 
design.

Try loading the content using a background process of some sort (ie:
scheduled task) and then staging it all in a table. That way when the user 
requests the page you aren't triggering a bunch of CFHTTP calls.

-Cameron

--
Cameron Childress
--
p:   678.637.5072
im: cameroncf
facebook <http://www.facebook.com/cameroncf> | 
twitter<http://twitter.com/cameronc> |
google+ <https://profiles.google.com/u/0/117829379451708140985>




~|
Order the Adobe Coldfusion Anthology now!
http://www.amazon.com/Adobe-Coldfusion-Anthology/dp/1430272155/?tag=houseoffusion
Archive: 
http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:353237
Subscription: http://www.houseoffusion.com/groups/cf-talk/subscribe.cfm
Unsubscribe: http://www.houseoffusion.com/groups/cf-talk/unsubscribe.cfm

Re: How to deal with outbound http calls

[Cameron Childress]

On Sun, Nov 18, 2012 at 7:48 AM, DeMarco, Alex wrote:

> I have a web app that at times can make many outbound http calls.  For
> example I have a page that collects various rss feeds from other sites and
> then parses them and finally displays them all on one page for the user.  I
> have a few questions.:
>
> Is there a limit to the number of outbound http calls CFMX 9 can make?
>

Assuming you aren't in a cluster, some creative use of CFLOCK would do the
trick. If you aren't familiar with CFLOCK, check the documentation. You
could also reduce the number of threads as Russ suggests, however I would
recommend putting some code in place to do something when that max is
reached or else you're just creating a new problem from the old one.


> Is there a way I can see the total number of outbound http calls going on?
>

Outside of CF there would be a number of network monitoring tools you could
use for this. You could also do some creative application or session scope
monitoring of start and stop counts for certain activities.


> It would seem to me that this is a poor design under load conditions.  Is
> there a better way to fetch off site content?


Are you doing it in realtime when a user's requests a page? Yes, that is
poor design.

Try loading the content using a background process of some sort (ie:
scheduled task) and then staging it all in a table. That way when the user
requests the page you aren't triggering a bunch of CFHTTP calls.

-Cameron

-- 
Cameron Childress
--
p:   678.637.5072
im: cameroncf
facebook  |
twitter |
google+ 


~|
Order the Adobe Coldfusion Anthology now!
http://www.amazon.com/Adobe-Coldfusion-Anthology/dp/1430272155/?tag=houseoffusion
Archive: 
http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:353234
Subscription: http://www.houseoffusion.com/groups/cf-talk/subscribe.cfm
Unsubscribe: http://www.houseoffusion.com/groups/cf-talk/unsubscribe.cfm

Re: How to deal with outbound http calls

[Russ Michaels]

Outgoing http calls is still a cf thread so is governed by your settings in
the cfadmin and the number of cpus you have.
If the content does not need to be live then why not do it on a schedule
and store it in the db or in static files.
You can use the built in server monitor to see current running requests.

Regards
Russ Michaels
www.michaels.me.uk
www.cfmldeveloper.com - Free CFML hosting for developers
www.cfsearch.com - CF search engine
On Nov 18, 2012 8:00 PM, "DeMarco, Alex"  wrote:

>
> Good Morning,
>
> I have a web app that at times can make many outbound http calls.  For
> example I have a page that collects various rss feeds from other sites and
> then parses them and finally displays them all on one page for the user.  I
> have a few questions.:
>
> Is there a limit to the number of outbound http calls CFMX 9 can make?
>
> Is there a way I can see the total number of outbound http calls going on?
>
> It would seem to me that this is a poor design under load conditions.  Is
> there a better way to fetch off site content?
>
> Thanks!
>
> [circle]
>
> Alex DeMarco
> Manager of Technical Services
> The State University of New York
> State University Plaza - Albany, New York 12246
> Tel: 518.320.1398Fax: 518.320.1550
> Be a part of Generation SUNY: Facebook<
> http://www.facebook.com/generationsuny> - Twitter<
> http://www.twitter.com/generationsuny> - YouTube<
> http://www.youtube.com/generationsuny>
>
>
>
>
>
> 

~|
Order the Adobe Coldfusion Anthology now!
http://www.amazon.com/Adobe-Coldfusion-Anthology/dp/1430272155/?tag=houseoffusion
Archive: 
http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:353233
Subscription: http://www.houseoffusion.com/groups/cf-talk/subscribe.cfm
Unsubscribe: http://www.houseoffusion.com/groups/cf-talk/unsubscribe.cfm