On Apr 25, 2015, at 5:10 AM, Neil Horman <nhorman at tuxdriver.com> wrote:
> I'm more focused on why that level of participation is not higher
Hi Neal,
This mail is probably way too long, but here is what I saw about participation,
in my case I used DPDK on two projects so far:
1) proprietary project for a L4-L7 stateful replay DPI firewall performance
tester at a large network test and measurement corporation using the old
original 6WIND DPDK 1.X under NDA before it became available to a wider channel
2) open-source self-created threat intelligence sensor project using DPDK
1.7.X, under development in my spare time
I think the things that making more DPDK contributions are ironically more
technical and social in nature than legal or bureaucratic in nature as one
would normally suspect, and as you theorized in your original mail. Let me go
through some things and see what people think.
At the social level, there are not very many people in the world who are
familiar and comfortable with the LKML style embedded coding workflow (heavy
mailing list usage, sending and reviewing patches in emails, putting specific
people in To or Cc to get patches approved and ACKed in subtrees, etc.) I
happen to know some if not most of this tribally, because I used Linux since
1997, but very few developers have any clue about this stuff. However I never
participated in the actual LKML flow any further than tester / bug reporter,
and I actually use DPDK very deliberately, to avoid fighting with the headaches
of the linux-net code and flamewars.
This is why I was proposing that we try to find a way to allow contributions
via Github or Bitbucket... their fork-and-pull model is much simpler for
outsiders to comprehend and make quick patches when they find little bugs or
issues as they integrate with the library... given we are BSD licensed, the low
barrier to entry is the ultimate way to keep the patch velocity as high as
possible, and keep the community going.
At the technical level, I see two or three difficulties:
1) A lot of the various performance enhancements one can use are kind of
"magical" or "jigsaw puzzle" and not presented in a unified way, where I can
methodically go through a checklist and enable everything my app should use
even though I have no clue what they all are. For example, 1) let's talk about
hashing... there is RSS hashing (symmetric or asymmetric), JHASH, CRC hash, ...
not sure how many different ones. 2) Let's talk about CPU models... SSE, SSE4,
SSE4.1, SSE4.2, etc. I don't know what I have myself, much less what my users
will have, much less what I actually need or should use, without guidance from
some processor people. 3) Let's talk about PCIe bus... there is DCA, some other
non-DCA acceleration that's faster if you are on the same NUMA node as the PCIe
slot, but fails to work if you aren't... etc.
A lot of the people from Intel, 6WIND, and the kernel people are just thinking
"this stuff is obvious... we've used it since 200X and it's 2015!" That's true
if you are a processor / kernel hacker... but if you spent your whole career on
packet processing or network security like a lot of us app developers might
have done, that's very orthogonal to Intel-specific and compiler-specific and
hardware-specific performance hacks... so a lot of us have no real clue how to
configure and test them all, much less enhance them and make some patches to
it. We just blindly trust 6WIND and Intel to get it right, because as far as we
can see, all the DPDK code is pretty clean and readable, and we're pretty sure
we don't know anything better than what's already coded and put into the
repository, if we don't have some checklists to follow to enable and test every
combination, and find any more bugfixes to suggest.
2) A lot of the network adapters DPDK uses, especially when you begin using the
more crazy accelerations, are either hard to obtain or expensive... for example
from what I saw in my jobs, the 10 gigabit boards were a minimum ~$250 USD in
manufacturing quantities. The 2-port gigabit latest-gen Intel board I got was
$100 USD... I think a lot of higher-ed students and overseas people from less
developed nations might have a hard time paying for some of this stuff to start
hacking... some kind of program to get these sort of people some sample gear
might help.
I also hit difficulties with the virtio-net driver... it doesn't work with the
virtio-net adapter in Virtualbox... which makes it harder for me to use cool,
convenient environments configured by the tool Vagrant, to make very simple dev
VM environments to quickly get new hackers up to speed on my open source
running just a command or two, and by extension harder for me to show them why
they should use DPDK for all their cool new network ${GADGET}s.
The same difficulty comes into place if I wanted to do some performance
patches... I don't have the money to buy the VTune profiler for my spare time
project, that you would want if you're going to get in there and start dumping
out all the CPU time and cache counters, so I could help you guys to squash the
performance bugs more quickly by testing the behavior in a real packet
processing app and not just some random microbenchmark that doesn't do any real
work on real stuff.
3) The DPDK library itself only covers Layer 2 which makes it pretty esoteric
for average Joes. This meant that I had to implement ARP, NDP, IPv4, IPv6, UDP,
and TCP myself, which was very educational but probably not super efficient for
attracting community attention. All except TCP weren't too bad... TCP was a ton
of work... I had to go to the university library downtown in my city and get
hold of Tanenbaum and some of the other networking textbooks and write quite a
lot of paper notes, just to implement enough of the server half of TCP to
process the traffic and pull out the data stream... and as Jay Rolette can
attest since he gave me pointers... I cheated like hell to do it, totally
ignoring reordering and retransmissions... but I am proud to say it seems to
work great for how half-baked of a job I did coding it. I would gladly trade a
first-born child for a better TCP implementation available with the DPDK. ;)
But if I look at the world of coders out there... a lot of guys aren't going to
have any idea what to do with an L2-only network stack that doesn't support the
two most popular Transport protocols out of the box... they only know about
what to do with TCP and UDP, not how to redo those themselves from the ground
up. In my career I only knew a handful of guys who would be good enough and
patient enough to go pound that stuff out... of course if I start my own
company someday that's precisely who I'd try to hire, but there aren't so many
of these people to go around.
Matthew.
