Hi Robert,
I'm not ready to promise to maintain your project, but I'm
interested. Currently I use IM for thumbnail generation via a shell
script.
I'm really intrigued by your download comparisons for the
thumbnails. I didn't see any difference because by inbound pipe was
the limiting factor.
So your results say that thumbtest.taf is 3x slower than
thumbtest.php? Wow! I guess I'll have to run my own tests on my
download.taf vs. a download.php. But I notice that your php server
has keep alive on, while the witango apache server doesn't. Does
this make a difference?
But, it seems to me that for high performance, we should ditch php/
taf and instead use mod_rewrite with an external rewriting program.
The program, which is only loaded once, takes the uid from the $ENV
and performs the sql query directly against the db. Have you looked
into this approach?
bill
On Apr 26, 2006, at 2:53 AM, Robert Garcia wrote:
I have some to contribute, but more cuz I am tired of dealing with
it than anything. As you know, I have asked many times for
documentation on the bean handler in witango. I have never got it,
and have asked numerous times. We use external actions for some
imagemagick stuff, which has been fine for us, cuz we don't use it
that often. However, I found myself in a situation, where we may
need to use it heavily. I realized the only way to do this right in
witango was in a bean.
It is very frustrating, unless you are just passing some simple int
or string parameters and coming back out. There are text encoding
issues, some datatypes just blow up, even booleans. So I went
through the trial and error, and got jmagick running on my g5, and
built a bean with one method at first. The method takes 4
parameters straight from witango, it takes an inpath, outpath, and
width and height. The method takes them, and reads the inpath in
jmagick, and outputs to the outpath, as a thumbnail. JMagick is a
java wrapper for the imagemagick C api. The C API is IMO more
powerful than the command line, and handles blobs and does some
other cool stuff. But the IM API can be quite funky, and if you
don't have a lot of experience with it, yoiu may pull some hair.
But I have been using for 2 years, and know its ins and outs, so
building the methods is just like my C methods, for the most part.
The method worked great. It took 800ms to thumb a 6megapixel jpeg
to 150x150, and the external action about a second. This may seem
like little difference, but the main difference, is the bean
instantiates in a few ms, and then if you load the object in a
global var, it loads in 1ms. The external action, always takes
about 30-50ms to just load, and then you have to deal with the env
vars and all that. I will rarely use 6megapixel images, there will
usually be less than 1megapixel. So a 40ms method, that takes 1ms
of overhead, vs 30-50ms of overhead is very nice.
So with that success, and testing on my mac with tiger, and testing
on our windows test server, I moved on to what I really wanted to
do. I wanted to read an image from a db blob, pass the blob to the
bean, and then manipulate the image as a blob, and return the blob
back to witango, to either serve direct to browser, for on the fly
manipulation, or put back in db, without ever writing to disk, for
maximum efficiency. So I wrote the method. First problem, you can
escape binary data in a var in witango, and you can't bind more
than 32k. So you have to base64 the blob, in and out, because there
are also text encoding issues. This is ok, cuz I figured with the
efficiency of ZERO diks IO, and the bean almost ZERO overhead, i
would still be in good shape. Well the method worked, but it took
about 3 times as long, as having witango write the blob to disk,
and using the first method, and then read it back in from disk. It
wasn't the base64, witango is fast at that, and java even faster,
with the opensource base64 class I included in my code. After much
trial and error, it is just that the witango bean handler takes
forever to pass data from witango back and forth to the bean. All
that work for nothing. I hate working with trial and error methods,
due to lack of documentation and support.
Anyway, I have the Mac XCode 2.2 project for this bean, and if
someone promises to keep it public, make a opensource project or
something, they can have it. Just contact me. I was planning to
write methods for rotate, flp, flop, shadow, and several other IM
methods I have, but I am done. At least if someone else uses it, it
won't feel like such a waste.
PHP vs Witango ( The Pepsi Challenge? )
Now, we are moving to php, and that is not a secret, but over this
year. With this issue, I did some extra homework, and setup a few
Fedora Core 5 Apache 2.2/php5 servers. I did some tests, and well,
it screams. I can use the GD graphics library in php, to do
whatever I need, and it is VERY fast, as fast or faster than IM. So
I got Ben, and the two of us built some apples to apples tests. As
apples to apples as we could. First, php beat witango on anything
image manipulation wise, cuz witango just can't do it without
external help, and external methods have huge limitations, or very
buggy. But what about just serving data? We needed to setup pure
image servers, with some on the fly image transformation. So we
setup 1 server, AMD2800, 2G ram, intel gigabit, running windows
2003 web edition, IIS 6, and witango 5.5.009. And php server,
running Fedora Core 5, apache2.2/php5 with the same hardware. Now,
we have tweaked our witango configs to the nth degree over years of
witango experience. The linux server is the default install. ZERO
performance tweaks, zero INI tweaks. The database server is our
production mysql 5 server, with millions of thumbnails and images.
The mysql server is tweaked, but both test servers are hitting it,
so thats fair. All 3 machines are connected to the same gigabit
switch with intel MT 1000 cards. We have some tafs, in use for many
moons that serve thumbnails and images. We duplicated the
functionality in PHP.
Again, this test was not set to prove anything, it was done solely
to make the right decision as to how we were going to build our new
image servers in the short term.
To keep as fair as possible, we made a page, served from our
witango production servers, that made a table of 100 thumbnails,
each of the thumbnail is a blob in the mysql db, and served by one
of the above test servers. So the main page is hosted by a bank of
witnago servers, but the images come from the witango test server
above, or the php test server above, depending on which test you
run. Well we were shocked by the results, PHP will fill your pipe
as fast as your connection will allow. It is up to 3x faster than
witango. Thats right, 3x.
http://ws.bighead.net/test/serveimage.taf
(be nice, if it gets hammered, I will have to pull it down.)
on most broadband connections, you will see a 15-25% or so speed
increase with php. But if you run from the same network as the
servers, so your connection is not the bottleneck, witango takes
about 3000ms to load all the thumbnails, and php about 1000ms. Ben
and I and my other devs were floored. We did NOTHING to the php
test server, I loaded FC5, and dis yum install php php-mysql php-
gd, and started it with apache. I can't wait to see what it will do
as we tweak.
For the record, I don't think IIS apache is the issue. I have bench
tested them against each other many times, and IIS will fill your
pipe, it is very fast. Also, I do think linux makes some of this
difference, cuz I know php doesn't run as fast on windows. But I do
know witango is highly optimized for windows, and I chose windows
for witango, because it has been historically the best platform for
witango.
One more thing, witango is taking full advantage of persistent
connections for this test. However, php is not. We are using the
mysqli API for php, which does not support persistant connections.
So php is making a connection for each thumbnail and closing it.
That blows me away.
Well, we are now making the move to php a huge priority. Still
looking for at least one more witango developer with php experience.
Also, if someone wants to question the testing we did, I would be
happy to supply the code we used for a legitimate testing.
Oh man, and don't get me or Ben started on how nice the Zend/PHP
studio is.
--
Robert Garcia
President - BigHead Technology
VP Application Development - eventpix.com
13653 West Park Dr
Magalia, Ca 95954
ph: 530.645.4040 x222 fax: 530.645.4040
[EMAIL PROTECTED] - [EMAIL PROTECTED]
http://bighead.net/ - http://eventpix.com/
--
Robert Garcia
President - BigHead Technology
VP Application Development - eventpix.com
13653 West Park Dr
Magalia, Ca 95954
ph: 530.645.4040 x222 fax: 530.645.4040
[EMAIL PROTECTED] - [EMAIL PROTECTED]
http://bighead.net/ - http://eventpix.com/
______________________________________________________________________
__ TO UNSUBSCRIBE: Go to http://www.witango.com/developer/maillist.taf
Bill
William M. Conlon, P.E., Ph.D.
To the Point
345 California Avenue Suite 2
Palo Alto, CA 94306
vox: 650.327.2175 (direct)
fax: 650.329.8335
mobile: 650.906.9929
e-mail: mailto:[EMAIL PROTECTED]
web: http://www.tothept.com
________________________________________________________________________
TO UNSUBSCRIBE: Go to http://www.witango.com/developer/maillist.taf
