Your requirements are rather strict. You may add an allowed user from the 'other side' to a group, but if that group exceeds say 20K member attribute values it would become cumbersome. Could you define allowed users on a host or IP basis?
Potentially. Though I think us using groups might be a lot easier.... or client side certificates?
