Yicong-Huang opened a new pull request, #53707: URL: https://github.com/apache/spark/pull/53707
### What changes were proposed in this pull request? This PR fixes a bug in `python/pyspark/worker.py` where `taskContext._resources` is incorrectly reset inside the for loop, causing only the last resource to be retained when a task has multiple resource types. The bug was introduced in SPARK-28234 (2019) and has been present since then. The existing test only covered the single-resource case, so the bug was not caught. ### Why are the changes needed? When a task has multiple resources (e.g., GPU + FPGA), only the last resource is preserved due to the dictionary being reset on each loop iteration. This breaks the multi-resource functionality that was intended in SPARK-28234. ### Does this PR introduce _any_ user-facing change? Yes. Tasks with multiple resource types will now correctly have all resources available via `TaskContext.get().resources()`. ### How was this patch tested? Modified the existing `TaskContextTestsWithResources` test class to configure and verify multiple resources (GPU + FPGA). ### Was this patch authored or co-authored using generative AI tooling? Yes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
