As you say, it's a lot of types. If you would really need to instantiate an exponential number of types then maybe you should reconsider, because the jit compiler has to do quite a lot of work for each type that is used.
But if you're not actually going to instantiate such a humongous number of them, or if you really want to be able to use specialization and dispatch in this way: How about a middle road where you use a parametric type and set the types of all unused fields to Void (the type of nothing)? That should be able to support the cases that you mentioned. Also, in many cases, storage for a value worth known type of eg Void is free, since it is known that there is only one instance. The exception is if the value could be uninitialized as well.
