The one provision ( or gotcha ) that I would add to this discussion is
log files
You should insure that log files to not become too large, especially
if they are using an XML format that is very fluffy.
I just had a client decide to program his ping schedule into my system
for once every 3 seconds instead of the design of 50 per hour. One of
my log files expanded to 42 Mb and caused a connection timeout
( greater than 10 seconds ). I now have taken protective action to
protect my dedicated server, but the shared hosting part of the
network cannot be changed ( the admin company is not willing to do
this )
Most log file logic is to add new transactions to the end of an
existing file, thus 2 Mb requires more time than 100K. Some servers
start new log files every calendar day, but others may have a much
lower frequency.
You may not be able to control this frequency, but you might be able
to switch modes.
Sometimes all events are logged and designated 'verbose' or 'debugging
mode'.
If you are on a shared host, you may not have the permissions to edit
or delete these files. On your own server, you should investigate if
the log files could expand quickly, especially if you are doing a week
of intense testing and programming.
On Aug 6, 2010, at 9:55 AM, Mark Talluto wrote:
On Aug 4, 2010, at 7:53 AM, Richard Gaskin wrote:
I've been experimenting with spidering, data mining, and analytics,
and like any processor-intensive tasks it would never occur to me
to put them on a shared host.
Like many hosts, the one I'm using offers dedicated servers for
less than $70/mo, but being a cheapskate I've gone one step further
during this experimental phase: I bought a nettop off Ebay for
just $150, set it up with Ubuntu and Rev, and that does all the
heavy lifting 24/7, posting only the output from those process to
my servers periodically as needed.
I never run into the CPU cycle limits most hosts have on their
servers, and I don't even slow down my own web server from its
tasks of serving pages to my visitors and handling their purchases.
When the workflow expands to required tighter integration between
the processing and the output, I can move the system from my office
to a dedicated server with multiple redundant fat-pipe connections
for just a few bucks a month.
There are a million ways to create robust scalable infrastructures
to handle any load. Many are cheap and easy to do, and for most of
those tasks you can do them all in one fun language.
We have been using this technique for years. We even posted the
application we use to do this task in RevNet. I believe I need to
update that file now that I think of it. But in short, we use our
ISP to gather orders. Our client software sends a request for a
key. Our local computer in the lab just pings the directory on the
ISP every 4 seconds and downloads all the orders in that given
directory. The heavy lifting and database work is done on a
computer in the lab. The key is then sent back up to the ISP where
the client computer is checking in for the result of that work every
4 seconds. The whole thing works out nicely and we keep our CPU
usage low.
Mark Talluto
Jim Ault
Las Vegas
_______________________________________________
use-revolution mailing list
[email protected]
Please visit this url to subscribe, unsubscribe and manage your subscription
preferences:
http://lists.runrev.com/mailman/listinfo/use-revolution
