On 07/09/2013 07:52 AM, Alex Williamson wrote:
> On Sun, 2013-07-07 at 01:07 +1000, Alexey Kardashevskiy wrote:
>> VFIO is designed to be used via ioctls on file descriptors
>> returned by VFIO.
>>
>> However in some situations support for an external user is required.
>> The first user is KVM on PPC64 (SPAPR TCE protocol) which is going to
>> use the existing VFIO groups for exclusive access in real/virtual mode
>> on a host to avoid passing map/unmap requests to the user space which
>> would made things pretty slow.
>>
>> The proposed protocol includes:
>>
>> 1. do normal VFIO init stuff such as opening a new container, attaching
>> group(s) to it, setting an IOMMU driver for a container. When IOMMU is
>> set for a container, all groups in it are considered ready to use by
>> an external user.
>>
>> 2. pass a fd of the group we want to accelerate to KVM. KVM calls
>> vfio_group_get_external_user() to verify if the group is initialized,
>> IOMMU is set for it and increment the container user counter to prevent
>> the VFIO group from disposal prior to KVM exit.
>> The current TCE IOMMU driver marks the whole IOMMU table as busy when
>> IOMMU is set for a container what prevents other DMA users from
>> allocating from it so it is safe to grant user space access to it.
>>
>> 3. KVM calls vfio_external_user_iommu_id() to obtian an IOMMU ID which
>> KVM uses to get an iommu_group struct for later use.
>>
>> 4. When KVM is finished, it calls vfio_group_put_external_user() to
>> release the VFIO group by decrementing the container user counter.
>> Everything gets released.
>>
>> The "vfio: Limit group opens" patch is also required for the consistency.
>>
>> Signed-off-by: Alexey Kardashevskiy <a...@ozlabs.ru>
>> ---
>> diff --git a/drivers/vfio/vfio.c b/drivers/vfio/vfio.c
>> index c488da5..57aa191 100644
>> --- a/drivers/vfio/vfio.c
>> +++ b/drivers/vfio/vfio.c
>> @@ -1370,6 +1370,62 @@ static const struct file_operations vfio_device_fops
>> = {
>> };
>>
>> /**
>> + * External user API, exported by symbols to be linked dynamically.
>> + *
>> + * The protocol includes:
>> + * 1. do normal VFIO init operation:
>> + * - opening a new container;
>> + * - attaching group(s) to it;
>> + * - setting an IOMMU driver for a container.
>> + * When IOMMU is set for a container, all groups in it are
>> + * considered ready to use by an external user.
>> + *
>> + * 2. The user space passed a group fd which we want to accelerate in
>> + * KVM. KVM uses vfio_group_get_external_user() to verify that:
>> + * - the group is initialized;
>> + * - IOMMU is set for it.
>> + * Then vfio_group_get_external_user() increments the container user
>> + * counter to prevent the VFIO group from disposal prior to KVM exit.
>> + *
>> + * 3. KVM calls vfio_external_user_iommu_id() to know an IOMMU ID which
>> + * KVM uses to get an iommu_group struct for later use.
>> + *
>> + * 4. When KVM is finished, it calls vfio_group_put_external_user() to
>> + * release the VFIO group by decrementing the container user counter.
>
> nit, the interface is for any external user, not just kvm.
s/KVM/An external user/ ?
Or add "the description below uses KVM just as an example of an external user"?
>> + */
>> +struct vfio_group *vfio_group_get_external_user(struct file *filep)
>> +{
>> + struct vfio_group *group = filep->private_data;
>> +
>> + if (filep->f_op != &vfio_group_fops)
>> + return NULL;
>
> ERR_PTR(-EINVAL)
>
> There also needs to be a vfio_group_get(group) here and put in error
> cases.
Is that because I do not hold a reference to the file anymore?
>> +
>> + if (!atomic_inc_not_zero(&group->container_users))
>> + return NULL;
>
> ERR_PTR(-EINVAL)
>
>> +
>> + if (!group->container->iommu_driver ||
>> + !vfio_group_viable(group)) {
>> + atomic_dec(&group->container_users);
>> + return NULL;
>
> ERR_PTR(-EINVAL)
>
>> + }
>> +
>> + return group;
>> +}
>> +EXPORT_SYMBOL_GPL(vfio_group_get_external_user);
>> +
>> +void vfio_group_put_external_user(struct vfio_group *group)
>> +{
>> + vfio_group_try_dissolve_container(group);
>
> And a vfio_group_put(group) here
>
>> +}
>> +EXPORT_SYMBOL_GPL(vfio_group_put_external_user);
>> +
>> +int vfio_external_user_iommu_id(struct vfio_group *group)
>> +{
>> + return iommu_group_id(group->iommu_group);
>> +}
>> +EXPORT_SYMBOL_GPL(vfio_external_user_iommu_id);
>> +
>> +/**
>> * Module/class support
>> */
>> static char *vfio_devnode(struct device *dev, umode_t *mode)
>> diff --git a/include/linux/vfio.h b/include/linux/vfio.h
>> index ac8d488..24579a0 100644
>> --- a/include/linux/vfio.h
>> +++ b/include/linux/vfio.h
>> @@ -90,4 +90,11 @@ extern void vfio_unregister_iommu_driver(
>> TYPE tmp; \
>> offsetof(TYPE, MEMBER) + sizeof(tmp.MEMBER); }) \
>>
>> +/*
>> + * External user API
>> + */
>> +extern struct vfio_group *vfio_group_get_external_user(struct file *filep);
>> +extern void vfio_group_put_external_user(struct vfio_group *group);
>> +extern int vfio_external_user_iommu_id(struct vfio_group *group);
>> +
>> #endif /* VFIO_H */
>
>
>
--
Alexey
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
