My primary pain points in decreasing order of severity:

1. For years I've been battling alignment issues in virtual machines with no
   success. I can't take the performance hit or invest the time in twiddling
   alignment anymore. TCP overhead is nothing compared to extra writes due to
   disk alignment. iSCSI straight out of the box doesn't have this issue.

2. Zero distribution (CentOS/RedHat) integration. Having to maintain my own
   init scripts and making the system unload the aoe module before the bonding
   module on shutdown, start/stop the vblades, and various other bits of
   trickery has become a real hassle.

3. Despite claims that AoE use is accelerating I've never run into another
   person who uses it or even really knows what it is. I've never seen/heard
   about it in any trade magazine aside from a Linux Journal article years ago.
   Nobody stops in the AoE IRC channel except for disappointed "Age of Empires"
   players. Traffic on this list is nearly dead. The blog you set up is quiet.
   I've had an AoE google alert setup for years. There is nearly no chatter
   about it. I have had an RSS search feed for AoE on ServerFault for quite
   some time. It very rarely comes up. Currently, it hasn't been mentioned
   since July:

   And look at the comments. Ouch.

   There is a lot of support and training material for iSCSI and it is part of
   the RHCE exam. I have had to do all of my own in-house training for AoE and
   it scares customers that it is such a niche technology with so few people on
   earth knowing how to use it, despite its on-the-wire simplicity. It actually
   hurts my business for people to know I use AoE. At one point I thought I
   could sell it as a competitive advantage. Not only is it an operational
   hassle but it has turned into a competitive disadvantage.

4. While I have always advocated AoE on the basis of its simplicity, the actual
   number of steps needed to get iSCSI working on RedHat/CentOS are far fewer
   than AoE which in practice makes AoE not simple. I have now
   scripted/cookbooked the steps to getting iSCSI up and running and my junior
   admin can do it. Mounting an iSCSI volume is a minor task on the RHCE exam
   taken by relatively newbie Linux admins. The issues involved in getting AoE
   working reliably and especially the troubleshooting are beyond many
   administrators. I have puppet scripts to distribute kernel modules (have to
   distribute it and compile it per kernel if you aren't running the exact same
   version everywhere, and deal with a recompile when you yum update the
   kernel), manage overly complicated init scripts, tweak sysctls, manage
   vblade configs, etc. Using the builtin distro iSCSI none of this is

I could perhaps resolve all of this myself by getting involved in Fedora and
contributing patches to make AoE work as smoothly as iSCSI in RHEL.  But so far
I've been completely technically incapable of solving the alignment issue and
as for the rest: Why should I invest so much time and effort duplicating the
excellent work of the iSCSI people? 

Anne Hathaway of Coraid actually contacted me last year about working for
Coraid. I was very tempted but simply unable to move away from San Diego. And
the first thing I would have wanted to do would be to make AoE work well in
RHEL as target and initiator which I really doubt Coraid would be willing to
pay me to do...

I have often had the feeling that AoE's lack of distro integration has been
because Coraid would prefer we all spend tens of thousands on Coraid hardware
and if the free version worked well enough there would be less incentive for
people to do so. I understand that every business needs to make money. But I'm
not sure I can get behind that as a business model. Last I looked (it's been a
while) a Coraid box was all Supermicro hardware which I could assemble myself
from Newegg. I seem to recall it was Plan 9 for the OS, a strange choice. I
suspect it's something to do with licensing, which makes me somewhat
uncomfortable. Again, they need to make money but I'm not interested in helping
them to work against my own interest.

I've got a business to run too, can't afford to purchase Coraid, and just
wanted a decent ethernet SAN which, given today's level of commodity
technology, is quite possible for a very reasonable price. I'm afraid I've hurt
myself spending so much time on making AoE work. I wish AoE well and maybe some
day I'll come back to it but for six years it has cost me way too much time
(and therefore money) and now I'm a bit embarrassed by the whole thing.

On Fri, Oct 19, 2012 at 06:19:13AM PDT, Ed Cashin spake thusly:
> AoE use is accelerating rapidly, and iSCSI persists because even though it's 
> not the best fit for same-LAN data storage, folks can still get it to work if 
> they are willing and able to deal with the complexity, sacrifice the 
> performance, and don't need the kind of scaling that virtualization and cloud 
> deployments require.
> Although a lot of the expansion of AoE use lately has been by users of the 
> Coraid HBA, there are still a lot of first-time AoE users using the 
> Linux initiator and also the 
> initiator.  Recently there have been a lot of patches from Coraid to the 
> Linux Kernel Mailing List for bringing the driver up 
> to date.  For example,
> We also fixed a regression in the Linux kernel's network layer that affected 
> AoE performance:
> I don't have much time to participate on the aoetools-discuss mailing list 
> right now, partly so that I can keep generating those patches, but I think 
> pessimism is especially inappropriate today, when the use of AoE is 
> accelerating along with the development of its associated technologies, 
> including open source technologies.
> -- 
>   Ed Cashin

Tracy Reed

Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics
Download AppDynamics Lite for free today:
Aoetools-discuss mailing list

Reply via email to