dongjoon-hyun commented on a change in pull request #35504:
URL: https://github.com/apache/spark/pull/35504#discussion_r829285680
##########
File path:
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/BasicExecutorFeatureStep.scala
##########
@@ -59,11 +59,16 @@ private[spark] class BasicExecutorFeatureStep(
private val isDefaultProfile = resourceProfile.id ==
ResourceProfile.DEFAULT_RESOURCE_PROFILE_ID
private val isPythonApp = kubernetesConf.get(APP_RESOURCE_TYPE) ==
Some(APP_RESOURCE_TYPE_PYTHON)
private val disableConfigMap =
kubernetesConf.get(KUBERNETES_EXECUTOR_DISABLE_CONFIGMAP)
+ private val memoryOverheadFactor = if
(kubernetesConf.contains(EXECUTOR_MEMORY_OVERHEAD_FACTOR)) {
+ kubernetesConf.get(EXECUTOR_MEMORY_OVERHEAD_FACTOR)
+ } else {
+ kubernetesConf.get(MEMORY_OVERHEAD_FACTOR)
+ }
Review comment:
Hi, @tgravescs and @mridulm . Yes, we need to fix this, but before that
I'm wondering if we really need this at Apache Spark 3.3. As we noticed here,
it turns out this is not a straight forward contribution. If you don't mind,
I'd like to recommend to keep fixing and stabilizing in `master` branch while
reverting this from `branch-3.3` at least.
##########
File path:
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/BasicExecutorFeatureStep.scala
##########
@@ -59,11 +59,16 @@ private[spark] class BasicExecutorFeatureStep(
private val isDefaultProfile = resourceProfile.id ==
ResourceProfile.DEFAULT_RESOURCE_PROFILE_ID
private val isPythonApp = kubernetesConf.get(APP_RESOURCE_TYPE) ==
Some(APP_RESOURCE_TYPE_PYTHON)
private val disableConfigMap =
kubernetesConf.get(KUBERNETES_EXECUTOR_DISABLE_CONFIGMAP)
+ private val memoryOverheadFactor = if
(kubernetesConf.contains(EXECUTOR_MEMORY_OVERHEAD_FACTOR)) {
+ kubernetesConf.get(EXECUTOR_MEMORY_OVERHEAD_FACTOR)
+ } else {
+ kubernetesConf.get(MEMORY_OVERHEAD_FACTOR)
+ }
Review comment:
> I should also state that if people don't want in 3.3, I'm personally
fine with it just need input from anyone interested in the feature.
Thank you. Then, it's simpler because this PR was backported manually after
feature freeze. :)
We are currently discussing on the whitelisting about late arrivals like
this PR, aren't we, @tgravescs ?
We can discuss there together to get those input you need.
##########
File path:
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/BasicExecutorFeatureStep.scala
##########
@@ -59,11 +59,16 @@ private[spark] class BasicExecutorFeatureStep(
private val isDefaultProfile = resourceProfile.id ==
ResourceProfile.DEFAULT_RESOURCE_PROFILE_ID
private val isPythonApp = kubernetesConf.get(APP_RESOURCE_TYPE) ==
Some(APP_RESOURCE_TYPE_PYTHON)
private val disableConfigMap =
kubernetesConf.get(KUBERNETES_EXECUTOR_DISABLE_CONFIGMAP)
+ private val memoryOverheadFactor = if
(kubernetesConf.contains(EXECUTOR_MEMORY_OVERHEAD_FACTOR)) {
+ kubernetesConf.get(EXECUTOR_MEMORY_OVERHEAD_FACTOR)
+ } else {
+ kubernetesConf.get(MEMORY_OVERHEAD_FACTOR)
+ }
Review comment:
Please unblock K8s module QA period by reverting this. We can land it
back after having a healthy commit.
##########
File path:
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/BasicExecutorFeatureStep.scala
##########
@@ -59,11 +59,16 @@ private[spark] class BasicExecutorFeatureStep(
private val isDefaultProfile = resourceProfile.id ==
ResourceProfile.DEFAULT_RESOURCE_PROFILE_ID
private val isPythonApp = kubernetesConf.get(APP_RESOURCE_TYPE) ==
Some(APP_RESOURCE_TYPE_PYTHON)
private val disableConfigMap =
kubernetesConf.get(KUBERNETES_EXECUTOR_DISABLE_CONFIGMAP)
+ private val memoryOverheadFactor = if
(kubernetesConf.contains(EXECUTOR_MEMORY_OVERHEAD_FACTOR)) {
+ kubernetesConf.get(EXECUTOR_MEMORY_OVERHEAD_FACTOR)
+ } else {
+ kubernetesConf.get(MEMORY_OVERHEAD_FACTOR)
+ }
Review comment:
Please unblock Apache Spark 3.3 K8s module QA period by reverting this.
We can land it back after having a healthy commit.
##########
File path:
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/BasicExecutorFeatureStep.scala
##########
@@ -59,11 +59,16 @@ private[spark] class BasicExecutorFeatureStep(
private val isDefaultProfile = resourceProfile.id ==
ResourceProfile.DEFAULT_RESOURCE_PROFILE_ID
private val isPythonApp = kubernetesConf.get(APP_RESOURCE_TYPE) ==
Some(APP_RESOURCE_TYPE_PYTHON)
private val disableConfigMap =
kubernetesConf.get(KUBERNETES_EXECUTOR_DISABLE_CONFIGMAP)
+ private val memoryOverheadFactor = if
(kubernetesConf.contains(EXECUTOR_MEMORY_OVERHEAD_FACTOR)) {
+ kubernetesConf.get(EXECUTOR_MEMORY_OVERHEAD_FACTOR)
+ } else {
+ kubernetesConf.get(MEMORY_OVERHEAD_FACTOR)
+ }
Review comment:
Thank you for your decision, @tgravescs .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
