We've more or less tried using pluginlib actions to have generalized keymaps. The problem is, it simply doesn't really work. Since the directional keys can be quite different from target to target (on wheel targets up and down are going to be wheel up and down, whereas on joystick targets it will be movements of the stick, and sometimes both can work and sometimes each will need to be handled individually).
Generally speaking, the reason they don't all use actions is because it turns out that in reality you still need to look at every plugin to make sure its keymap works, unless it's a very simply controlled plugin (which are the ones we typically already have using actions). This is one of those cases where the idea has been tried in the past, didn't work out as well as expected, and we moved on from it to what we have now. Going back to it would be a regression without a strong plan to resolve the issues that have already been seen.
