cdmikechen commented on PR #989: URL: https://github.com/apache/submarine/pull/989#issuecomment-1254741184
@FatalLin Yes, loading datasets is indeed a method based on the need to adapt for different environments, whether it is in a cloud environment is something that can be determined within the sdk via the k8s api. If it's in the cloud then you can register a prehandler pod resource for doing data synchronisation to minio/s3, or pull the data locally for processing. Users should be able to choose their own method of pulling data that is appropriate for their current environment. I think more details can be added in `submarine.load_datasets`, and I don't have much event confirmation on the huggingface project yet. I think it might also give us a new way of thinking. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
