I had briefly looked at the gres code awhile ago and I could not make sense of how it worked. Perhaps it would be really helpful if Moe or Danny sent a brief explanation to the list.
Almost all of the relevant code is in src/common/gres.c and the key data structures are in src/common/gres.h. There are gres_list fields associated with nodes, jobs and steps. The list contains a key (the gres type) and a pointer to a structure of the appropriate type. If there are no file names or topology defined in the gres.conf file (read by slurmd on the compute node), then the code just needs to keep track of the count of available and allocated gres of each type. Otherwise it needs to manage bitmaps of available gres (available and allocated). The gres plugins are optional. In the case of gres/gpu, it just sets the CUDA_VISIBLE_DEVICES environment variable based upon the position(s) in the bitmap allocated to the job or step.
Probably the simplest way to get the correct environment variable would be to modify node_config_load() to cache device numbers and then use those device numbers rather than the bitmap index to set CUDA_VISIBLE_DEVICES values in job_set_env() and step_set_env()
