zero323 commented on pull request #35142: URL: https://github.com/apache/spark/pull/35142#issuecomment-1008298407
@zhongjingxiong > The python compiled by the user in centots' operating system can not be directly migrated The real problem here is that software build using dynamically linked libraries (like Python in your case) is really not intended to be portable. Even if you can make it work in a specific case, it is susceptible to minute changes in the OS configuration (compiling dynamically linked Python in my Debian dev sandbox and putting it in these images, wouldn't work any better than in your CentOS case). That's why we have Linux package managers in the first place. Not to mention, providing large runtime bundle from outside the image for each job is extremely wasteful. If, for whatever reason, you need to compile your own Python version, it should be done as a part of the image building process. This way you can ensure that your images are self-contained. If, for the sake of argument, it is not possible in your case, there are utilities that are designed to create portable Python bundles ([some listed](https://spark.apache.org/docs/latest/api/python/user_guide/python_packaging.html?highlight=packages) in our official docs). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
