[GitHub] [spark] yangwwei commented on pull request #37622: [SPARK-40187][DOCS] Add `Apache YuniKorn` scheduler docs
yangwwei commented on PR #37622: URL: https://github.com/apache/spark/pull/37622#issuecomment-1234636614 Hi, @dongjoon-hyun thanks a lot for helping on this. This is a great community collaboration between YuniKorn and Spark, thank you so much! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] yangwwei commented on pull request #37622: [SPARK-40187][DOCS] Add `Apache YuniKorn` scheduler docs
yangwwei commented on PR #37622: URL: https://github.com/apache/spark/pull/37622#issuecomment-1232299398 @dongjoon-hyun circle back on this. does the latest version look good to you? Anything else you want me to address in this doc? Please let me know, thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] yangwwei commented on pull request #37622: [SPARK-40187][DOCS] Add `Apache YuniKorn` scheduler docs
yangwwei commented on PR #37622: URL: https://github.com/apache/spark/pull/37622#issuecomment-1227634279 > According to [apache/yunikorn-site#180 (comment)](https://github.com/apache/yunikorn-site/pull/180#issuecomment-1226820265) , I understand the context (including rollback) and let's hold on this until next week (YuniKorn v1.1). Please take a look at the updated version. I do not think we depend on the 1.0.0 doc link anymore. We just make sure we explicitly set the version in the installation example. For general docs, we can point to https://yunikorn.apache.org. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] yangwwei commented on pull request #37622: [SPARK-40187][DOCS] Add `Apache YuniKorn` scheduler docs
yangwwei commented on PR #37622: URL: https://github.com/apache/spark/pull/37622#issuecomment-1227632338 > In that case, why don't you add Apache Spark Support Matrix to Apache YuniKorn page? I don't think that will be necessary. Apache YuniKorn is a scheduler, a replacement to the default scheduler, it isn't so sensitive about Spark versions. Any Spark can run on YuniKorn with some necessary configs. In the recent version with the support of https://issues.apache.org/jira/browse/SPARK-38383, submitting jobs to YuniKorn is even easier (what was introduced in this doc). This is like we do not need a Spark support matrix in YARN, or Kubernetes. Adding such a matrix on the YuniKorn side is cumbersome, we probably will need to list all Spark versions, which isn't useful for the end users. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] yangwwei commented on pull request #37622: [SPARK-40187][DOCS] Add `Apache YuniKorn` scheduler docs
yangwwei commented on PR #37622: URL: https://github.com/apache/spark/pull/37622#issuecomment-1226822958 @dongjoon-hyun could you please take a look at the updated version? Here are the changes: 1. Used 1.0.0 in the installation example to comply with Spark community requirements 2. Removed the "Work with YuniKorn queues" as that adds complexity to the doc, and for Spark users, that's not something that has to be done 3. Addressed other review comments there is one more thing about our version doc. I have implemented some temp workaround in https://issues.apache.org/jira/browse/YUNIKORN-1293, but YuniKorn community folks made a very good point that the workaround wasn't sustainable. Since now there is only the following links in the doc: - https://yunikorn.apache.org/ - https://yunikorn.apache.org/docs/get_started/core_features I think it should be fine that not bind to a specific version, what do you think? YuniKorn folks also pointed out that there are similar docs like the following already: ``` Volcano feature steps help users to create a Volcano PodGroup and set driver/executor pod annotation to link with this [PodGroup](https://volcano.sh/en/docs/podgroup/). ``` and ``` Volcano defines PodGroup spec using [CRD yaml](https://volcano.sh/en/docs/podgroup/#example). ``` please let me know your thought for this, thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] yangwwei commented on pull request #37622: [SPARK-40187][DOCS] Add `Apache YuniKorn` scheduler docs
yangwwei commented on PR #37622: URL: https://github.com/apache/spark/pull/37622#issuecomment-1224867456 > It seems that I wasn't clear enough to you. We need a specific version number, @yangwwei . -1 for adding a doc without version version. > > > https://yunikorn.apache.org/docs/get_started/core_features. You are absolutely clear. Sure, let me find out to get this supported with our documentation framework. Thanks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
