Yicong-Huang commented on issue #5034: URL: https://github.com/apache/texera/issues/5034#issuecomment-4665188259
> This might answer your question: system-requirements-lock.txt is a superset of requirements.txt that explicitly includes all indirect dependencies. I have several doubts about this approach. My understanding is that dependency resolution is both environment-sensitive and time-sensitive. The same requirements.txt can resolve to different installed versions depending on the Python version, OS, Linux distribution, hardware capabilities such as CUDA support, platform-specific wheels, and the package index state at the time of installation. Because of that, I am not sure whether we should maintain system-requirements-lock.txt as a static checked-in file. A lock file is useful when it captures a specific environment at a specific point in time, but if this file is expected to work across many different environments, it may become misleading or stale. A few questions I have are: 1. What specific environment is system-requirements-lock.txt generated for? For example, is it tied to a specific Python version, OS, Linux distribution, and hardware configuration? 2. Is this lock file expected to be portable across all supported environments, or only reproducible for one specific environment? 3. How often is this file expected to be regenerated, and what process ensures that it stays in sync with the actual target environments? 4. If different environments resolve dependencies differently, should we have separate generated lock files per target environment instead of one shared static file? My suggestion is that requirements.txt should remain the source of truth for declared dependencies, while system-requirements-lock.txt should be treated as an environment-specific generated artifact. It should be generated and resolved on the fly in the target environment where it is used, such as CI, Docker builds, or release jobs. This would make the lock file accurate for that specific environment and point in time, instead of maintaining a static checked-in lock file that may not generalize across different platforms. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
