ulysses-you commented on pull request #28778: URL: https://github.com/apache/spark/pull/28778#issuecomment-643002040
Actually, my first thought is the file partitions split and I tried to add another config to control it, like `filesDefaultParallelism`. After that I was thinking is it a special config ？It means that the config just control file parallelism. Parallelism in spark is everywhere, and in sql scene, file is just the first-class citizens. Finally I change to add a common config same as the `spark.default.parallelism`. I also not see the reasonable case without file partition split now. If you think it's not needed, I'm OK. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
