09/03/2024
Participants:
Alexia Ingerson (Intel)
Jianxin Xiong (Intel)
Ben Lynam (Cornelis)
Bob Cernohous (Cornelis)
Charles Sherada (Cornelis)
Chuck Fossen
Ian Ziemba (HPE)
Jack Morrison (Cornelis)
Jerome Soumagne
Jessie Yang (AWS)
John Byrne (HPE)
Ken Raffenetti (ANL)
Peinan Zhang (Intel)
Rajalaxmi (Intel)
Shi Jin (AWS)
Stephen Oost (Intel)
Steven Welch (HPE)
Zach Dworkin (Intel)
Summary:
Libfabric 2.0.0-alpha was released on 8/30 and includes all deprecations and a
new provider (LPP). Beta release is targeted for 2 months from now (end of
October). Please check new alpha release for any issues, especially
compatibility issues for building. No new 1.x branches are expected though
1.x.y releases may occur as needed.
AWS presented on proposals to expand FI_HMEM interface capabilities. AWS,
Intel, and HPE agree that adding fi_info->hmem_attr would be the best solution.
AWS will draft up a more detailed proposal for this which includes adding
features such as which interfaces support p2p (NIC to GPU) or optimized device
memcopies.
Cornelis proposed adding an opx-specific yaml file to trigger their internal
testing. There were no objections.
Notes:
Libfabric 2.0.0-alpha released 8/30
* Deprecations
* New LPP provider
Please check new alpha release for any issues, especially compatibility issues
for building (added warnings for deprecated features) as well as running.
Beta release planned for 2 months from now (end of October)
* Expected all new features will make it into beta release
Q: What's the plan if need new 1.x release? Can we create new 1.x branch?
A: We can still do 1.x.y releases for bug fixes but shouldn't need new minor
release
NCCL plug in currently cannot build with upstream, may need a fix for build
issue.
Q: Are deprecated features going to be removed for official 2.0 release?
A: Will stick around for about a year before removal. Will take time for
middlewares to officially remove use of deprecated features.
HMEM capability refinement (AWS presentation - Jessie, Shi)
Currently FI_HMEM is on/off capability which represents many interfaces
Proposal to introduce more capabilities to manage more specific FI_HMEM
abilities
Add struct fi_hmem_attr *hmem_attr to info to return specific set of
capabilities
struct fi_hmem_attr {
enum fi_hmem_iface iface;
bool use_p2p;
bool use_dev_reg_copy;
bool api_permitted:
struct fi_hmem_attr *next;
};
iface: memory interface type (CUDA, ROCR, ZE)
use_p2p (accelerator to NIC p2p, not acc-acc p2p): whether peer to peer
transfers should be used, filter out shm, all apps to specific p2p early
use_dev_reg_copy: whether to use optimized memcpy for dev memory (ie GDR)
api_permitted: whether dev specific API call is allowed, prevents unsafe
operations and resource management conflicts
next: pointer to next hmem_attr if using multiple non-system ifaces
Struct will be both input and output - app can set fields to request settings
and used to filter and select certain ifaces, etc
Q: Is it hard for a provider to onboard this new API because it's not using
common code?
A: Could do this either way. Most of the information is in common code. A
provider if they support certain attributes then it should go through common
code. Use_p2p could be the only field not in common code.
Q: use_p2p should be clearer. It means provider should use p2p or allowed to
use p2p?
A: It should follow the current common fi_use_p2p preferred or disabled where
each one already has its defined behavior.
Api_permitted is definitely input value only - weird use case of fi_info attr.
Very unclear - provider has to use API to use accelerator library. Even FI_HMEM
initialization uses API code. Main use case for api_permitted is for MR
registration and data transfer calls. Need to revisit this option.
Alternative: add fields to fi_mr_attr on memory registration path
struct fi_mr_attr {
//existing fields
bool use_p2p;
bool use_dev_reg_copy;
bool api_permitted;
};
Pros: no need for users to set values
Cons: scope is limited to MR registration, users cannot see these fields by
running fi_info
Preference is fi_info path to see and set provider specific settings, wider use
case
HPE, AWS, and Intel all prefer having it at the fi_info level
Q: any comments on input/output p2p settings
A: api_permitted setting is very important to them. Need to think more on
input/output settings
AWS will continue to refine proposal for adding to fi_info and we will revisit
Cornelis Github Action workflow (Jack Morrison):
Go over cn.yaml addition in new PR
github.com/ofiwg/libfabric/pull/10354<https://github.com/ofiwg/libfabric/pull/10354>
Trying to leverage more utilities available as part of Github Actions for opx
testing.
* In cn.yaml - check to see if the PR is targeted for internal Cornelis
libfabric repository. No-op if not targeted for internal repo
Runs on internal Cornelis machines
Will not get triggered for non-Cornelis/upstream PRs
Q: Why not have this only internal? What's the benefit of having it upstream?
A: Makes it easier to handle commits because of rebasing/upstreaming flow
No objections. Fine to move forward
_______________________________________________
ofiwg mailing list
[email protected]
https://lists.openfabrics.org/mailman/listinfo/ofiwg
