gaogaotiantian commented on PR #57093: URL: https://github.com/apache/spark/pull/57093#issuecomment-4912701108
Thanks for the suggestion. There are three things here 1. `ruff` vs `black` - this is actually an out-of-date doc. We replaced `black` with `ruff` long time ago. For protobufs too. So if the users follow the current documentation, they will have a wrong result. That's more like a "fix" to the current documentation, instead of a "change". However, if you think it's better to have those in a separate PR, I can do that. 2. `ml_extra` to save some time - I'll do this in a separate PR so we can have proper discussion specific for that. But that can't be done with `requirements.txt` because of the limitation of that approach. 3. `pyproject.toml` vs `requirements.txt` - they should not exist together, otherwise we are introducing another source of dependency. The whole purpose of this refactoring is to have a single source of truth. Notice that the existing CI does not rely on `requirements.txt` - it hard-coded all its dependencies in `Dockerfile`. This has caused us troubles because when we upgrade versions of a dependency, we often miss some in the `Dockerfile`. If we introduce `pyproject.toml` but do not change CI/docs and eventually remove `requirements.txt`, that wouldn't be helpful. I would not touch cut branches because they are stable. We are not supposed to change the dependencies for `branch-4.x`, but I believe that's for the actual spark dependencies. We could potentially upgrade the version of devtools or CI 3rd party libraries. When we do such things, it would be really nice to have a synced version of `branch-4.x` and `master`. We would make less mistakes and backport would be much easier. Do you think we can make this happen in both `branch-4.x` and `master`? We can keep `requirements.txt` for a while just to make sure we do not interrupt developers abruptly. But with this change, CIs can finally refer to a file for Python dependencies, instead of hard-coding. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
