aicam commented on PR #6911: URL: https://github.com/apache/texera/pull/6911#issuecomment-5108565353
The following options were explored: 1- If we can just bump `setuptools`, unfortunately, it has dependency to `fs` which also caused issues in the past and we raised this PR #4199 and `fs` library has not been updated for over 4 years. [Ref](https://pypi.org/project/fs/#history) 2- If we can replace it, the options: - [fsspec](https://pypi.org/project/fsspec/): It directly work with path inside OS but ExecutorManager uses MemoryFileSystem to do tmp directories so its not compatible - [fs2](https://pypi.org/project/fs2/): still deprecated and relies on `fs` - boltons.fileutils: does not support temp path (same as fsspec) The investigation above indicate that the feature of creating temporary temp on the fly is something unique we need which is not core feature of file system libraries 3- If we can remove it, we use `fs` simply to create temporary folders in memory, PR #6946 replace it simply by native file system libraries with identical result @Yicong-Huang Thanks for your help and teaching me, let me know if the above conclusion make sense to you -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
