[GitHub] spark issue #22392: [SPARK-23200] Reset Kubernetes-specific config on Checkp...

2018-09-13 Thread foxish
Github user foxish commented on the issue: https://github.com/apache/spark/pull/22392 Jenkins, test this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark issue #21067: [SPARK-23980][K8S] Resilient Spark driver on Kubernetes

2018-07-19 Thread foxish
Github user foxish commented on the issue: https://github.com/apache/spark/pull/21067 > ReadWriteOnce storage can only be attached to one node. This is well known. Using the RWO volume for fencing here would work - but this is not representative of all users. This breaks d

[GitHub] spark issue #21067: [SPARK-23980][K8S] Resilient Spark driver on Kubernetes

2018-07-12 Thread foxish
Github user foxish commented on the issue: https://github.com/apache/spark/pull/21067 > After a short/configurable delay the driver pod state changed to Unknown and the Job controller initiated a new spark driver. This is dangerous behavior. The old spark driver can st

[GitHub] spark issue #21583: [SPARK-23984][K8S][Test] Added Integration Tests for PyS...

2018-07-06 Thread foxish
Github user foxish commented on the issue: https://github.com/apache/spark/pull/21583 @ifilonenko, can you elaborate on what the issue is and what version of docker is needed? --- - To unsubscribe, e-mail: reviews

[GitHub] spark issue #21067: [SPARK-23980][K8S] Resilient Spark driver on Kubernetes

2018-07-06 Thread foxish
Github user foxish commented on the issue: https://github.com/apache/spark/pull/21067 I don't think this current approach will suffice. Correctness is important here, especially for folks using spark streaming. I understand that we're proposing the use of backoff limits

[GitHub] spark pull request #21652: [SPARK-24551][K8S] Add integration tests for secr...

2018-07-02 Thread foxish
Github user foxish commented on a diff in the pull request: https://github.com/apache/spark/pull/21652#discussion_r199608693 --- Diff: resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/KubernetesSuite.scala --- @@ -74,10

[GitHub] spark pull request #21652: [SPARK-24551][K8S] Add integration tests for secr...

2018-07-02 Thread foxish
Github user foxish commented on a diff in the pull request: https://github.com/apache/spark/pull/21652#discussion_r199608853 --- Diff: resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/KubernetesSuite.scala --- @@ -150,6

[GitHub] spark issue #21652: [SPARK-24551][K8S] Add integration tests for secrets

2018-07-02 Thread foxish
Github user foxish commented on the issue: https://github.com/apache/spark/pull/21652 jenkins, retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark issue #21697: [SPARK-24711][K8S] Fix tags for integration tests

2018-07-02 Thread foxish
Github user foxish commented on the issue: https://github.com/apache/spark/pull/21697 This is good! Same comment that @srowen had on moving the version up to the parent POM and referencing it here

[GitHub] spark issue #21462: [SPARK-24428][K8S] Fix unused code

2018-07-02 Thread foxish
Github user foxish commented on the issue: https://github.com/apache/spark/pull/21462 Merging to master --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #21462: [SPARK-24428][K8S] Fix unused code

2018-07-02 Thread foxish
Github user foxish commented on the issue: https://github.com/apache/spark/pull/21462 LGTM, we can file a separate task to track what happened to secrets mounting without blocking on removing dead code. Thanks for the fix @skonto

[GitHub] spark issue #21511: [SPARK-24491][Kubernetes] Configuration support for requ...

2018-06-25 Thread foxish
Github user foxish commented on the issue: https://github.com/apache/spark/pull/21511 +1 agreed with @liyinan926. Let's be more careful what we add to the configuration options. --- - To unsubscribe, e-mail

[GitHub] spark issue #21583: [SPARK-23984][K8S][Test] Added Integration Tests for PyS...

2018-06-25 Thread foxish
Github user foxish commented on the issue: https://github.com/apache/spark/pull/21583 @ifilonenko, is this ready to go? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e

[GitHub] spark pull request #21462: [SPARK-24428][K8S] Fix unused code

2018-06-25 Thread foxish
Github user foxish commented on a diff in the pull request: https://github.com/apache/spark/pull/21462#discussion_r197909183 --- Diff: resource-managers/kubernetes/docker/src/main/dockerfiles/spark/entrypoint.sh --- @@ -45,12 +45,10 @@ shift 1 SPARK_CLASSPATH

[GitHub] spark pull request #21462: [SPARK-24428][K8S] Fix unused code

2018-06-25 Thread foxish
Github user foxish commented on a diff in the pull request: https://github.com/apache/spark/pull/21462#discussion_r197909535 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesClusterManager.scala --- @@ -46,8 +46,6

[GitHub] spark pull request #21279: [SPARK-24219][k8s] Improve the docker building sc...

2018-06-25 Thread foxish
Github user foxish commented on a diff in the pull request: https://github.com/apache/spark/pull/21279#discussion_r197906642 --- Diff: bin/docker-image-tool.sh --- @@ -44,15 +44,37 @@ function image_ref { function build { local BUILD_ARGS local IMG_PATH

[GitHub] spark issue #21555: [SPARK-24547][K8S] Allow for building spark on k8s docke...

2018-06-20 Thread foxish
Github user foxish commented on the issue: https://github.com/apache/spark/pull/21555 Will follow-up. Thanks all for the comments! Merging to master. --- - To unsubscribe, e-mail: reviews-unsubscr

[GitHub] spark issue #21555: [SPARK-24547][K8S] Allow for building spark on k8s docke...

2018-06-20 Thread foxish
Github user foxish commented on the issue: https://github.com/apache/spark/pull/21555 @mccheah, that is a good point but I agree with @ifilonenko that we can do it in a subsequent PR. I'm thinking we merge this as-is and I can try to get a follow-up PR here for dockerfile refactor

[GitHub] spark issue #21555: [SPARK-24547][K8S] Allow for building spark on k8s docke...

2018-06-19 Thread foxish
Github user foxish commented on the issue: https://github.com/apache/spark/pull/21555 Lgtm. Will merge when tests pass. Thanks! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21551: [K8S] Fix issue in 'docker-image-tool.sh'

2018-06-19 Thread foxish
Github user foxish commented on the issue: https://github.com/apache/spark/pull/21551 Phew, this was a bad one. Thanks for the fix all. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org

[GitHub] spark issue #21366: [SPARK-24248][K8S] Use level triggering and state reconc...

2018-06-14 Thread foxish
Github user foxish commented on the issue: https://github.com/apache/spark/pull/21366 If last round's comments are addressed, LGTM from me. Important behavior to check is - the snapshot, and creating replacement executors based on captured snapshot

[GitHub] spark issue #20697: [SPARK-23010][k8s] Initial checkin of k8s integration te...

2018-06-08 Thread foxish
Github user foxish commented on the issue: https://github.com/apache/spark/pull/20697 Thanks @mccheah and @ssuchter! Added a deprecation notice to the repo in https://github.com/apache-spark-on-k8s/spark-integration

[GitHub] spark issue #21366: [SPARK-24248][K8S] Use the Kubernetes API to populate an...

2018-06-07 Thread foxish
Github user foxish commented on the issue: https://github.com/apache/spark/pull/21366 If we did miss the watch as you said, when the next poll comes along, that should capture the fact that a pod that we thought existed doesn't exist anymore in the API. So, that would be the level

[GitHub] spark issue #21366: [SPARK-24248][K8S] Use the Kubernetes API to populate an...

2018-06-07 Thread foxish
Github user foxish commented on the issue: https://github.com/apache/spark/pull/21366 Yes, watches + polling things is common in controllers. Shared informers are a good stable abstractions they use so, they can avoid the polling in most cases. However, for now, If we just use

[GitHub] spark issue #21366: [SPARK-24248][K8S] Use the Kubernetes API to populate an...

2018-06-07 Thread foxish
Github user foxish commented on the issue: https://github.com/apache/spark/pull/21366 Sorry, not yet. I explained to @mccheah what I meant about us preferring the level triggered approach and always being able to rebuild state even if some events are lost and resyncing completely

[GitHub] spark issue #21366: [SPARK-24248][K8S] Use the Kubernetes API to populate an...

2018-06-04 Thread foxish
Github user foxish commented on the issue: https://github.com/apache/spark/pull/21366 I was planning to do one last pass over it today/tomorrow. I can merge after that. --- - To unsubscribe, e-mail: reviews

[GitHub] spark issue #21317: [SPARK-24232][k8s] Add support for secret env vars

2018-05-31 Thread foxish
Github user foxish commented on the issue: https://github.com/apache/spark/pull/21317 Merging to master --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark pull request #21317: [SPARK-24232][k8s] Add support for secret env var...

2018-05-31 Thread foxish
Github user foxish commented on a diff in the pull request: https://github.com/apache/spark/pull/21317#discussion_r192206460 --- Diff: docs/running-on-kubernetes.md --- @@ -602,4 +608,20 @@ specific to Spark on Kubernetes. spark.kubernetes.executor.secrets.spark-secret

[GitHub] spark pull request #21317: [SPARK-24232][k8s] Add support for secret env var...

2018-05-31 Thread foxish
Github user foxish commented on a diff in the pull request: https://github.com/apache/spark/pull/21317#discussion_r192168016 --- Diff: docs/running-on-kubernetes.md --- @@ -602,4 +608,20 @@ specific to Spark on Kubernetes. spark.kubernetes.executor.secrets.spark-secret

[GitHub] spark pull request #21462: [SPARK-24428][K8S] Fix unused code

2018-05-30 Thread foxish
Github user foxish commented on a diff in the pull request: https://github.com/apache/spark/pull/21462#discussion_r191936265 --- Diff: docs/running-on-kubernetes.md --- @@ -121,8 +121,8 @@ This URI is the location of the example jar that is already in the Docker image

[GitHub] spark pull request #21462: [SPARK-24428][K8S] Fix unused code

2018-05-30 Thread foxish
Github user foxish commented on a diff in the pull request: https://github.com/apache/spark/pull/21462#discussion_r191930265 --- Diff: docs/running-on-kubernetes.md --- @@ -121,8 +121,8 @@ This URI is the location of the example jar that is already in the Docker image

[GitHub] spark issue #21462: [SPARK-24428][K8S] Fix unused code

2018-05-30 Thread foxish
Github user foxish commented on the issue: https://github.com/apache/spark/pull/21462 jenkins, ok to test --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark pull request #21366: [SPARK-24248][K8S] Use the Kubernetes API to popu...

2018-05-30 Thread foxish
Github user foxish commented on a diff in the pull request: https://github.com/apache/spark/pull/21366#discussion_r191865387 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala --- @@ -154,6 +154,24 @@ private[spark] object Config

[GitHub] spark pull request #21366: [SPARK-24248][K8S] Use the Kubernetes API to popu...

2018-05-30 Thread foxish
Github user foxish commented on a diff in the pull request: https://github.com/apache/spark/pull/21366#discussion_r191868513 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/ExecutorPodsEventQueue.scala --- @@ -0,0 +1,29

[GitHub] spark pull request #21366: [SPARK-24248][K8S] Use the Kubernetes API to popu...

2018-05-30 Thread foxish
Github user foxish commented on a diff in the pull request: https://github.com/apache/spark/pull/21366#discussion_r191866216 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/ExecutorPodsAllocator.scala --- @@ -0,0 +1,120

[GitHub] spark pull request #21366: [SPARK-24248][K8S] Use the Kubernetes API to popu...

2018-05-30 Thread foxish
Github user foxish commented on a diff in the pull request: https://github.com/apache/spark/pull/21366#discussion_r191869618 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/ExecutorPodsLifecycleEventHandler.scala --- @@ -0,0

[GitHub] spark pull request #21366: [SPARK-24248][K8S] Use the Kubernetes API to popu...

2018-05-30 Thread foxish
Github user foxish commented on a diff in the pull request: https://github.com/apache/spark/pull/21366#discussion_r191871593 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/ExecutorPodsAllocator.scala --- @@ -0,0 +1,120

[GitHub] spark pull request #21366: [SPARK-24248][K8S] Use the Kubernetes API to popu...

2018-05-30 Thread foxish
Github user foxish commented on a diff in the pull request: https://github.com/apache/spark/pull/21366#discussion_r191865834 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/ExecutorPodsAllocator.scala --- @@ -0,0 +1,120

[GitHub] spark issue #21317: [SPARK-24232][k8s] Add support for secret env vars

2018-05-17 Thread foxish
Github user foxish commented on the issue: https://github.com/apache/spark/pull/21317 @skonto, looks like a test runner issue here. @ssuchter, can you PTAL? --- - To unsubscribe, e-mail: reviews-unsubscr

[GitHub] spark issue #21279: [SPARK-24219][k8s] Improve the docker building script to...

2018-05-16 Thread foxish
Github user foxish commented on the issue: https://github.com/apache/spark/pull/21279 oops, I didn't notice this. Sorry @jerryshao, will take a look in the next day or two. --- - To unsubscribe, e-mail: reviews

[GitHub] spark issue #21238: [SPARK-24137][K8s] Mount local directories as empty dir ...

2018-05-10 Thread foxish
Github user foxish commented on the issue: https://github.com/apache/spark/pull/21238 SG. @liyinan926, let's revisit this if we hear from 2.3 users. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org

[GitHub] spark issue #21238: [SPARK-24137][K8s] Mount local directories as empty dir ...

2018-05-10 Thread foxish
Github user foxish commented on the issue: https://github.com/apache/spark/pull/21238 @mccheah, wdyt? I just haven't heard from any users here of 2.3 - if you think it's useful for 2.3.1 and low risk, then please feel free to propose a cherrypick

[GitHub] spark issue #21238: [SPARK-24137][K8s] Mount local directories as empty dir ...

2018-05-10 Thread foxish
Github user foxish commented on the issue: https://github.com/apache/spark/pull/21238 Maintenance releases most often have fixes for stability. We could maybe backport this since it's not a new feature but an omission from before. If it is going to be some effort, thanks to all

[GitHub] spark issue #21238: [SPARK-24137][K8s] Mount local directories as empty dir ...

2018-05-10 Thread foxish
Github user foxish commented on the issue: https://github.com/apache/spark/pull/21238 LGTM. Merging to master. Thanks @mccheah --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21092: [SPARK-23984][K8S] Initial Python Bindings for PySpark o...

2018-05-09 Thread foxish
Github user foxish commented on the issue: https://github.com/apache/spark/pull/21092 jenkins, retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark issue #21092: [SPARK-23984][K8S] Initial Python Bindings for PySpark o...

2018-05-09 Thread foxish
Github user foxish commented on the issue: https://github.com/apache/spark/pull/21092 jenkins, ok to test --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark pull request #21238: [SPARK-24137][K8s] Mount local directories as emp...

2018-05-09 Thread foxish
Github user foxish commented on a diff in the pull request: https://github.com/apache/spark/pull/21238#discussion_r187116886 --- Diff: core/src/main/scala/org/apache/spark/SparkConf.scala --- @@ -455,7 +455,8 @@ class SparkConf(loadDefaults: Boolean) extends Cloneable with Logging

[GitHub] spark pull request #21241: [SPARK-24135][K8s] Resilience to init-container e...

2018-05-08 Thread foxish
Github user foxish commented on a diff in the pull request: https://github.com/apache/spark/pull/21241#discussion_r186879387 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesClusterSchedulerBackend.scala --- @@ -320,50

[GitHub] spark pull request #21241: [SPARK-24135][K8s] Resilience to init-container e...

2018-05-08 Thread foxish
Github user foxish commented on a diff in the pull request: https://github.com/apache/spark/pull/21241#discussion_r186869138 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesClusterSchedulerBackend.scala --- @@ -320,50

[GitHub] spark pull request #21241: [SPARK-24135][K8s] Resilience to init-container e...

2018-05-08 Thread foxish
Github user foxish commented on a diff in the pull request: https://github.com/apache/spark/pull/21241#discussion_r186861330 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesClusterSchedulerBackend.scala --- @@ -320,50

[GitHub] spark pull request #21241: [SPARK-24135][K8s] Resilience to init-container e...

2018-05-08 Thread foxish
Github user foxish commented on a diff in the pull request: https://github.com/apache/spark/pull/21241#discussion_r186857969 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesClusterSchedulerBackend.scala --- @@ -320,50

[GitHub] spark issue #21260: [SPARK-23529][K8s] Support mounting hostPath volumes

2018-05-08 Thread foxish
Github user foxish commented on the issue: https://github.com/apache/spark/pull/21260 jenkins, ok to test --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark pull request #21241: [SPARK-24135][K8s] Resilience to init-container e...

2018-05-08 Thread foxish
Github user foxish commented on a diff in the pull request: https://github.com/apache/spark/pull/21241#discussion_r186643210 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesClusterSchedulerBackend.scala --- @@ -320,50

[GitHub] spark pull request #21241: [SPARK-24135][K8s] Resilience to init-container e...

2018-05-08 Thread foxish
Github user foxish commented on a diff in the pull request: https://github.com/apache/spark/pull/21241#discussion_r186642656 --- Diff: docs/running-on-kubernetes.md --- @@ -561,6 +561,13 @@ specific to Spark on Kubernetes. This is distinct from spark.executor.cores

[GitHub] spark pull request #21241: [SPARK-24135][K8s] Resilience to init-container e...

2018-05-08 Thread foxish
Github user foxish commented on a diff in the pull request: https://github.com/apache/spark/pull/21241#discussion_r186641837 --- Diff: docs/running-on-kubernetes.md --- @@ -561,6 +561,13 @@ specific to Spark on Kubernetes. This is distinct from spark.executor.cores

[GitHub] spark pull request #21241: [SPARK-24135][K8s] Resilience to init-container e...

2018-05-08 Thread foxish
Github user foxish commented on a diff in the pull request: https://github.com/apache/spark/pull/21241#discussion_r186642338 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesClusterSchedulerBackend.scala --- @@ -320,50

[GitHub] spark issue #21202: [SPARK-24129] [K8S] Add option to pass --build-arg's to ...

2018-05-02 Thread foxish
Github user foxish commented on the issue: https://github.com/apache/spark/pull/21202 jenkins, ok to test --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #21202: [SPARK-24129] [K8S] Add option to pass --build-arg's to ...

2018-05-02 Thread foxish
Github user foxish commented on the issue: https://github.com/apache/spark/pull/21202 LGTM pending test run --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews

[GitHub] spark issue #21095: [SPARK-23529][K8s] Support mounting hostPath volumes

2018-04-30 Thread foxish
Github user foxish commented on the issue: https://github.com/apache/spark/pull/21095 The testing is not a blocker for the review. When I said "tests for non-hostpath type volumes", I meant to say that we want to cover more than just hostpath mounts with the initial PR -

[GitHub] spark issue #21095: [SPARK-23529][K8s] Support mounting hostPath volumes

2018-04-26 Thread foxish
Github user foxish commented on the issue: https://github.com/apache/spark/pull/21095 @madanadit, sorry about the long turnaround time here. I made a cursory pass over it - and it looks good. Before getting deeper into the code, I think we need more exhaustive unit testing for non

[GitHub] spark issue #21095: [SPARK-23529][K8s] Support mounting hostPath volumes

2018-04-18 Thread foxish
Github user foxish commented on the issue: https://github.com/apache/spark/pull/21095 @madanadit, thanks for following up with the PR and for your [design doc](https://docs.google.com/document/d/15-mk7UnOYNTXoF6EKaVlelWYc9DTrTXrYoodwDuAwY4/edit#heading=h.8jlem461uvwv). I had a few

[GitHub] spark issue #21095: [SPARK-23529][K8s] Support mounting hostPath volumes

2018-04-18 Thread foxish
Github user foxish commented on the issue: https://github.com/apache/spark/pull/21095 jenkins, ok to test --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #21092: [SPARK-23984][K8S][WIP] Initial Python Bindings for PySp...

2018-04-18 Thread foxish
Github user foxish commented on the issue: https://github.com/apache/spark/pull/21092 Thanks for taking this on @ifilonenko. Left some initial comments on the PR without going too much in depth - since as you noted, it's WIP

[GitHub] spark pull request #21092: [SPARK-23984][K8S][WIP] Initial Python Bindings f...

2018-04-18 Thread foxish
Github user foxish commented on a diff in the pull request: https://github.com/apache/spark/pull/21092#discussion_r182523365 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesExecutorBuilder.scala --- @@ -29,9 +30,11

[GitHub] spark pull request #21092: [SPARK-23984][K8S][WIP] Initial Python Bindings f...

2018-04-18 Thread foxish
Github user foxish commented on a diff in the pull request: https://github.com/apache/spark/pull/21092#discussion_r182522479 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/BasicDriverFeatureStep.scala --- @@ -71,7 +77,7 @@ private

[GitHub] spark pull request #21092: [SPARK-23984][K8S][WIP] Initial Python Bindings f...

2018-04-18 Thread foxish
Github user foxish commented on a diff in the pull request: https://github.com/apache/spark/pull/21092#discussion_r182517091 --- Diff: resource-managers/kubernetes/docker/src/main/dockerfiles/spark/entrypoint.sh --- @@ -62,6 +69,14 @@ case "$SPARK_K8

[GitHub] spark pull request #21092: [SPARK-23984][K8S][WIP] Initial Python Bindings f...

2018-04-18 Thread foxish
Github user foxish commented on a diff in the pull request: https://github.com/apache/spark/pull/21092#discussion_r182518301 --- Diff: resource-managers/kubernetes/docker/src/main/dockerfiles/spark/bindings/python/Dockerfile --- @@ -0,0 +1,33 @@ +# +# Licensed

[GitHub] spark pull request #21092: [SPARK-23984][K8S][WIP] Initial Python Bindings f...

2018-04-18 Thread foxish
Github user foxish commented on a diff in the pull request: https://github.com/apache/spark/pull/21092#discussion_r182520090 --- Diff: resource-managers/kubernetes/core/src/test/scala/org/apache/spark/deploy/k8s/submit/KubernetesDriverBuilderSuite.scala --- @@ -89,6 +97,29

[GitHub] spark pull request #21092: [SPARK-23984][K8S][WIP] Initial Python Bindings f...

2018-04-18 Thread foxish
Github user foxish commented on a diff in the pull request: https://github.com/apache/spark/pull/21092#discussion_r182524173 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/BasicDriverFeatureStep.scala --- @@ -88,15 +94,22

[GitHub] spark issue #21092: [SPARK-23984][K8S][WIP] Initial Python Bindings for PySp...

2018-04-18 Thread foxish
Github user foxish commented on the issue: https://github.com/apache/spark/pull/21092 cc @holdenk --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #21032: [SPARK-23529][K8s] Support mounting hostPath volumes for...

2018-04-13 Thread foxish
Github user foxish commented on the issue: https://github.com/apache/spark/pull/21032 @madanadit @liyinan926, can we do a short doc with all the types of volumes and config options and run that through the rest of the community

[GitHub] spark issue #20910: [SPARK-22839] [K8s] Refactor to unify driver and executo...

2018-04-13 Thread foxish
Github user foxish commented on the issue: https://github.com/apache/spark/pull/20910 Merging to master! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #21032: [SPARK-23529][K8s] Support mounting hostPath volumes for...

2018-04-12 Thread foxish
Github user foxish commented on the issue: https://github.com/apache/spark/pull/21032 Yes, that is what I was indicating as well. We should have a design for all different volume types. Extending this PR as the design evolves is a good way to go about that. On Thu, Apr

[GitHub] spark pull request #21032: [SPARK-23529][K8s] Support mounting hostPath volu...

2018-04-12 Thread foxish
Github user foxish commented on a diff in the pull request: https://github.com/apache/spark/pull/21032#discussion_r181176057 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala --- @@ -117,6 +117,13 @@ private[spark] object Config

[GitHub] spark pull request #21032: [SPARK-23529][K8s] Support mounting hostPath volu...

2018-04-12 Thread foxish
Github user foxish commented on a diff in the pull request: https://github.com/apache/spark/pull/21032#discussion_r181172885 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala --- @@ -117,6 +117,13 @@ private[spark] object Config

[GitHub] spark issue #21032: [SPARK-23529][K8s] Support mounting hostPath volumes for...

2018-04-12 Thread foxish
Github user foxish commented on the issue: https://github.com/apache/spark/pull/21032 Jenkins, ok to test --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #20910: [SPARK-22839] [K8s] Refactor to unify driver and executo...

2018-04-12 Thread foxish
Github user foxish commented on the issue: https://github.com/apache/spark/pull/20910 @mccheah I think it's fine as is now. We can take care of moving abstractions between the submission client and the driver in a future PR if necessary. Just the scala style issue needs taking care

[GitHub] spark pull request #20910: [SPARK-22839] [K8s] Refactor to unify driver and ...

2018-04-10 Thread foxish
Github user foxish commented on a diff in the pull request: https://github.com/apache/spark/pull/20910#discussion_r180535990 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/KubernetesSpec.scala --- @@ -14,17 +14,18 @@ * See the License

[GitHub] spark issue #20989: [SPARK-23529][K8s] Support mounting hostPath volumes for...

2018-04-05 Thread foxish
Github user foxish commented on the issue: https://github.com/apache/spark/pull/20989 ok to test --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #20811: [SPARK-23668][K8S] Add config option for passing through...

2018-04-04 Thread foxish
Github user foxish commented on the issue: https://github.com/apache/spark/pull/20811 LGTM. Merging to master --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark issue #20811: [SPARK-23668][K8S] Add config option for passing through...

2018-04-04 Thread foxish
Github user foxish commented on the issue: https://github.com/apache/spark/pull/20811 @mccheah - heads up, this will likely lead to a rebase on https://github.com/apache/spark/pull/20910. --- - To unsubscribe, e

[GitHub] spark issue #20811: [SPARK-23668][K8S] Add config option for passing through...

2018-04-04 Thread foxish
Github user foxish commented on the issue: https://github.com/apache/spark/pull/20811 @andrusha, no worries. I think this is going to be rather important for folks with private registries. Thanks for following up

[GitHub] spark issue #20811: [SPARK-23668][K8S] Add config option for passing through...

2018-04-04 Thread foxish
Github user foxish commented on the issue: https://github.com/apache/spark/pull/20811 Jenkins, test this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark issue #20811: [SPARK-23668][K8S] Add config option for passing through...

2018-04-04 Thread foxish
Github user foxish commented on the issue: https://github.com/apache/spark/pull/20811 Jenkins, test this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark pull request #20811: [SPARK-23668][K8S] Add config option for passing ...

2018-04-03 Thread foxish
Github user foxish commented on a diff in the pull request: https://github.com/apache/spark/pull/20811#discussion_r178982578 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala --- @@ -54,6 +54,12 @@ private[spark] object Config

[GitHub] spark pull request #20811: [SPARK-23668][K8S] Add config option for passing ...

2018-04-03 Thread foxish
Github user foxish commented on a diff in the pull request: https://github.com/apache/spark/pull/20811#discussion_r178982439 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala --- @@ -54,6 +54,12 @@ private[spark] object Config

[GitHub] spark issue #20811: [SPARK-23668][K8S] Add config option for passing through...

2018-04-03 Thread foxish
Github user foxish commented on the issue: https://github.com/apache/spark/pull/20811 LGTM after one comment. Thanks for fixing the style issues @andrusha --- - To unsubscribe, e-mail: reviews-unsubscr

[GitHub] spark pull request #20811: [SPARK-23668][K8S] Add config option for passing ...

2018-04-03 Thread foxish
Github user foxish commented on a diff in the pull request: https://github.com/apache/spark/pull/20811#discussion_r178924651 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala --- @@ -54,6 +54,12 @@ private[spark] object Config

[GitHub] spark issue #20811: [SPARK-23668][K8S] Add config option for passing through...

2018-04-03 Thread foxish
Github user foxish commented on the issue: https://github.com/apache/spark/pull/20811 Jenkins, test this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark issue #20811: [SPARK-23668][K8S] Add config option for passing through...

2018-04-02 Thread foxish
Github user foxish commented on the issue: https://github.com/apache/spark/pull/20811 @andrusha, can you address the style check failure that @felixcheung pointed out. This should be good to merge after

[GitHub] spark issue #20553: [SPARK-23285][K8S] Add a config property for specifying ...

2018-04-02 Thread foxish
Github user foxish commented on the issue: https://github.com/apache/spark/pull/20553 Merging once tests pass. Thanks! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e

[GitHub] spark pull request #20943: [ SPARK-23825] [K8s] Requesting memory + memory o...

2018-04-02 Thread foxish
Github user foxish commented on a diff in the pull request: https://github.com/apache/spark/pull/20943#discussion_r178615678 --- Diff: resource-managers/kubernetes/core/src/test/scala/org/apache/spark/deploy/k8s/submit/steps/BasicDriverConfigurationStepSuite.scala --- @@ -91,7

[GitHub] spark pull request #20553: [SPARK-23285][K8S] Add a config property for spec...

2018-04-02 Thread foxish
Github user foxish commented on a diff in the pull request: https://github.com/apache/spark/pull/20553#discussion_r178614696 --- Diff: docs/running-on-kubernetes.md --- @@ -549,14 +549,22 @@ specific to Spark on Kubernetes. spark.kubernetes.driver.limit.cores (none

[GitHub] spark pull request #20553: [SPARK-23285][K8S] Add a config property for spec...

2018-04-02 Thread foxish
Github user foxish commented on a diff in the pull request: https://github.com/apache/spark/pull/20553#discussion_r178614227 --- Diff: docs/running-on-kubernetes.md --- @@ -549,14 +549,22 @@ specific to Spark on Kubernetes. spark.kubernetes.driver.limit.cores (none

[GitHub] spark pull request #20943: [ SPARK-23825] [K8s] Requesting memory + memory o...

2018-04-02 Thread foxish
Github user foxish commented on a diff in the pull request: https://github.com/apache/spark/pull/20943#discussion_r178609700 --- Diff: resource-managers/kubernetes/core/src/test/scala/org/apache/spark/deploy/k8s/submit/steps/BasicDriverConfigurationStepSuite.scala --- @@ -91,7

[GitHub] spark issue #20669: [SPARK-22839][K8S] Remove the use of init-container for ...

2018-03-19 Thread foxish
Github user foxish commented on the issue: https://github.com/apache/spark/pull/20669 There's a section explaining it at the bottom of https://spark.apache.org/committers.html --- - To unsubscribe, e-mail: reviews

[GitHub] spark issue #20811: [SPARK-23668][K8S] Add config option for passing through...

2018-03-13 Thread foxish
Github user foxish commented on the issue: https://github.com/apache/spark/pull/20811 cc/ @mccheah @liyinan926 @vanzin --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e

[GitHub] spark issue #20811: [SPARK-23668][K8S] Add config option for passing through...

2018-03-13 Thread foxish
Github user foxish commented on the issue: https://github.com/apache/spark/pull/20811 ok to test --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #20791: [SPARK-23618][K8s][BUILD] Initialize BUILD_ARGS in docke...

2018-03-12 Thread foxish
Github user foxish commented on the issue: https://github.com/apache/spark/pull/20791 Merged into master. Thanks! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark issue #20791: [SPARK-23618][K8s][BUILD] Initialize BUILD_ARGS in docke...

2018-03-10 Thread foxish
Github user foxish commented on the issue: https://github.com/apache/spark/pull/20791 LGTM! Thanks @jooseong. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

  1   2   3   4   >