nivangio commented on issue #39262: URL: https://github.com/apache/airflow/issues/39262#issuecomment-2077814781
Hi @Taragolis . Thx for your reply! TBH, I wasn't aware of the existence of Object Storage. It seems as if many of the things I've implemented were already there. The only thing I cannot find is some sort of `cp` that enables uploading/downloading data from DBFS. At this point I wonder if it wouldn't be better to extend [this](https://filesystem-spec.readthedocs.io/en/stable/_modules/fsspec/implementations/dbfs.html) and then use it within `ObjectStoragePath`. With respect to the SDK, sounds good to me. However, the whole plugin is done pointing directly to the REST Endpoints. I think it may be better in that sense to stick to one strategy (either change everything to point to the SDK or extend it using the REST API) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
