gaogaotiantian commented on PR #57313: URL: https://github.com/apache/spark/pull/57313#issuecomment-5039006335
None of the standalone libraries for Python is good enough to ship scripts. I would be against that. I don't really think a single entry would be significantly better than current situation - it would be better probably. What I really want is a util tool that pyspark users can just use when they `pip install pyspark` (I think Python is the majority language spark users are using). So not like `./bin/spark` (this can't be used when you `pip install`), but something installed like `spark-cli ...`, which could be easily implemented in Python. I think you should be clear about whether you want this to be user-interfacing or dev-focusing, because I believe most of the spark users do not have the source code package. Of course, if `pyspark` is not installed with `pip install`, we can always have a fall back with `python -m pyspark.cli` or something like that. I want the cli to be part of pyspark. We should of course also support users that do not use Python. We probably can think of a strategy to organize code. I would not be interested in solutions where we ship an executable (or a shell script) just to be a unified entry. That's just not what python users are used to. However, I'm not against the idea. I won't oppose it - just indifferent. I understand if you want this to be more about spark itself than pyspark - then you'll need to find other committers that are interested in the idea. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
