[GitHub] spark pull request #22911: [SPARK-25815][k8s] Support kerberos in client mod...

2018-12-04 Thread ifilonenko
Github user ifilonenko commented on a diff in the pull request: https://github.com/apache/spark/pull/22911#discussion_r238770191 --- Diff: resource-managers/kubernetes/core/src/test/scala/org/apache/spark/deploy/k8s/features/KerberosConfDriverFeatureStepSuite.scala --- @@ -0,0

[GitHub] spark issue #23053: [SPARK-25957][K8S] Make building alternate language bind...

2018-11-20 Thread ifilonenko
Github user ifilonenko commented on the issue: https://github.com/apache/spark/pull/23053 LGTM. I will let @mccheah or @squito sign off on it with any additional comments / reviews --- - To unsubscribe, e-mail

[GitHub] spark issue #23053: [SPARK-25957][K8S] Make building alternate language bind...

2018-11-20 Thread ifilonenko
Github user ifilonenko commented on the issue: https://github.com/apache/spark/pull/23053 You need to make the appropriate change to `setup-integration-test-env.sh` to ensure that the PySpark tests pass in the integration-test suite

[GitHub] spark issue #23053: [SPARK-25957][K8S] Add ability to skip building optional...

2018-11-19 Thread ifilonenko
Github user ifilonenko commented on the issue: https://github.com/apache/spark/pull/23053 I agree with @mccheah, `-p skip` seems odd. It seems cleaner to only include i.e. `-p` flag when we desire python support

[GitHub] spark issue #23017: [SPARK-26015][K8S] Set a default UID for Spark on K8S Im...

2018-11-16 Thread ifilonenko
Github user ifilonenko commented on the issue: https://github.com/apache/spark/pull/23017 https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/5086/ seems to be hanging on the distribution build (45min

[GitHub] spark issue #23017: [SPARK-26015][K8S] Set a default UID for Spark on K8S Im...

2018-11-16 Thread ifilonenko
Github user ifilonenko commented on the issue: https://github.com/apache/spark/pull/23017 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark issue #23037: [SPARK-26083][k8s] Add Copy pyspark into corresponding d...

2018-11-15 Thread ifilonenko
Github user ifilonenko commented on the issue: https://github.com/apache/spark/pull/23037 > This is fine, but please file a bug. Okay, as such, @AzureQ could you add an integration test to `ClientModeTestsSu

[GitHub] spark issue #23026: [SPARK-25960][k8s] Support subpath mounting with Kuberne...

2018-11-15 Thread ifilonenko
Github user ifilonenko commented on the issue: https://github.com/apache/spark/pull/23026 > if such a list exists it should be the same list that triggers regular tests. I differ that to @shanekn

[GitHub] spark issue #23026: [SPARK-25960][k8s] Support subpath mounting with Kuberne...

2018-11-15 Thread ifilonenko
Github user ifilonenko commented on the issue: https://github.com/apache/spark/pull/23026 @vanzin, I added you to the admin list for triggering the `testing-k8s-prb-make-spark-distribution-unified

[GitHub] spark issue #23026: [SPARK-25960][k8s] Support subpath mounting with Kuberne...

2018-11-15 Thread ifilonenko
Github user ifilonenko commented on the issue: https://github.com/apache/spark/pull/23026 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark issue #23037: [MINOR][k8s] Add Copy pyspark into corresponding dir cmd...

2018-11-14 Thread ifilonenko
Github user ifilonenko commented on the issue: https://github.com/apache/spark/pull/23037 I see this customization to be specific towards how you build your custom Docker image. Unless it is relevant towards testing, we are trying to keep the default Docker image as lightweight

[GitHub] spark pull request #22911: [SPARK-25815][k8s] Support kerberos in client mod...

2018-11-14 Thread ifilonenko
Github user ifilonenko commented on a diff in the pull request: https://github.com/apache/spark/pull/22911#discussion_r233660132 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/KerberosConfDriverFeatureStep.scala --- @@ -69,97

[GitHub] spark pull request #22911: [SPARK-25815][k8s] Support kerberos in client mod...

2018-11-14 Thread ifilonenko
Github user ifilonenko commented on a diff in the pull request: https://github.com/apache/spark/pull/22911#discussion_r233657016 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/KerberosConfDriverFeatureStep.scala --- @@ -69,97

[GitHub] spark pull request #22911: [SPARK-25815][k8s] Support kerberos in client mod...

2018-11-14 Thread ifilonenko
Github user ifilonenko commented on a diff in the pull request: https://github.com/apache/spark/pull/22911#discussion_r233652949 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/KerberosConfDriverFeatureStep.scala --- @@ -69,97

[GitHub] spark pull request #22911: [SPARK-25815][k8s] Support kerberos in client mod...

2018-11-14 Thread ifilonenko
Github user ifilonenko commented on a diff in the pull request: https://github.com/apache/spark/pull/22911#discussion_r233650115 --- Diff: resource-managers/kubernetes/core/src/test/scala/org/apache/spark/deploy/k8s/features/KerberosConfDriverFeatureStepSuite.scala --- @@ -0,0

[GitHub] spark pull request #22911: [SPARK-25815][k8s] Support kerberos in client mod...

2018-11-14 Thread ifilonenko
Github user ifilonenko commented on a diff in the pull request: https://github.com/apache/spark/pull/22911#discussion_r233649498 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/submit/KubernetesDriverBuilder.scala --- @@ -81,14 +81,14

[GitHub] spark pull request #22911: [SPARK-25815][k8s] Support kerberos in client mod...

2018-11-14 Thread ifilonenko
Github user ifilonenko commented on a diff in the pull request: https://github.com/apache/spark/pull/22911#discussion_r233649110 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/KerberosConfDriverFeatureStep.scala --- @@ -69,97

[GitHub] spark pull request #22911: [SPARK-25815][k8s] Support kerberos in client mod...

2018-11-14 Thread ifilonenko
Github user ifilonenko commented on a diff in the pull request: https://github.com/apache/spark/pull/22911#discussion_r233648073 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/KerberosConfDriverFeatureStep.scala --- @@ -69,97

[GitHub] spark pull request #22911: [SPARK-25815][k8s] Support kerberos in client mod...

2018-11-14 Thread ifilonenko
Github user ifilonenko commented on a diff in the pull request: https://github.com/apache/spark/pull/22911#discussion_r233642625 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/BasicExecutorFeatureStep.scala --- @@ -144,6 +144,10

[GitHub] spark pull request #22911: [SPARK-25815][k8s] Support kerberos in client mod...

2018-11-14 Thread ifilonenko
Github user ifilonenko commented on a diff in the pull request: https://github.com/apache/spark/pull/22911#discussion_r233588339 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/DelegationTokenFeatureStep.scala --- @@ -0,0 +1,114

[GitHub] spark pull request #22911: [SPARK-25815][k8s] Support kerberos in client mod...

2018-11-14 Thread ifilonenko
Github user ifilonenko commented on a diff in the pull request: https://github.com/apache/spark/pull/22911#discussion_r233542062 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/SparkPod.scala --- @@ -18,7 +18,30 @@ package

[GitHub] spark pull request #23017: [WIP][SPARK-26015][K8S] Set a default UID for Spa...

2018-11-14 Thread ifilonenko
Github user ifilonenko commented on a diff in the pull request: https://github.com/apache/spark/pull/23017#discussion_r233537461 --- Diff: resource-managers/kubernetes/docker/src/main/dockerfiles/spark/entrypoint.sh --- @@ -30,6 +30,10 @@ set -e # If there is no passwd entry

[GitHub] spark issue #22760: [SPARK-25751][K8S][TEST] Unit Testing for Kerberos Suppo...

2018-11-12 Thread ifilonenko
Github user ifilonenko commented on the issue: https://github.com/apache/spark/pull/22760 gentle ping @mccheah :) --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark pull request #22911: [SPARK-25815][k8s] Support kerberos in client mod...

2018-11-06 Thread ifilonenko
Github user ifilonenko commented on a diff in the pull request: https://github.com/apache/spark/pull/22911#discussion_r231359962 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesClusterSchedulerBackend.scala --- @@ -123,7

[GitHub] spark pull request #22911: [SPARK-25815][k8s] Support kerberos in client mod...

2018-11-06 Thread ifilonenko
Github user ifilonenko commented on a diff in the pull request: https://github.com/apache/spark/pull/22911#discussion_r231344398 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesClusterSchedulerBackend.scala --- @@ -123,7

[GitHub] spark pull request #22760: [SPARK-25751][K8S][TEST] Unit Testing for Kerbero...

2018-11-05 Thread ifilonenko
Github user ifilonenko commented on a diff in the pull request: https://github.com/apache/spark/pull/22760#discussion_r230951222 --- Diff: resource-managers/kubernetes/core/src/test/scala/org/apache/spark/deploy/k8s/features/KubernetesFeaturesTestUtils.scala --- @@ -63,4 +63,66

[GitHub] spark issue #22911: [SPARK-25815][k8s] Support kerberos in client mode, keyt...

2018-11-05 Thread ifilonenko
Github user ifilonenko commented on the issue: https://github.com/apache/spark/pull/22911 > But whether to store it in secrets is not a question. You either store it in a secret or you don't support the keytab/principal feature in Spark at all, and we can delete a bunch of code h

[GitHub] spark pull request #22911: [SPARK-25815][k8s] Support kerberos in client mod...

2018-11-05 Thread ifilonenko
Github user ifilonenko commented on a diff in the pull request: https://github.com/apache/spark/pull/22911#discussion_r230860519 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesClusterSchedulerBackend.scala --- @@ -123,7

[GitHub] spark pull request #22915: [SPARK-25825][K8S][WIP] Enable token renewal for ...

2018-11-01 Thread ifilonenko
Github user ifilonenko commented on a diff in the pull request: https://github.com/apache/spark/pull/22915#discussion_r230134430 --- Diff: docs/security.md --- @@ -798,6 +782,50 @@ achieved by setting `spark.kubernetes.hadoop.configMapName` to a pre-existing Co local

[GitHub] spark issue #22911: [SPARK-25815][k8s] Support kerberos in client mode, keyt...

2018-10-31 Thread ifilonenko
Github user ifilonenko commented on the issue: https://github.com/apache/spark/pull/22911 Just a note, this PR has some overlaps with https://github.com/apache/spark/pull/22915. But just after a brief skim, I am a bit hesitant on storing keytabs in secrets, but it might have

[GitHub] spark pull request #22915: [SPARK-25825][K8S][WIP] Enable token renewal for ...

2018-10-31 Thread ifilonenko
GitHub user ifilonenko opened a pull request: https://github.com/apache/spark/pull/22915 [SPARK-25825][K8S][WIP] Enable token renewal for both --keytab and tokenSecret ## What changes were proposed in this pull request? Enabled token renewal when specifying `--keytab

[GitHub] spark issue #22624: [SPARK-23781][CORE] Merge token renewer functionality in...

2018-10-31 Thread ifilonenko
Github user ifilonenko commented on the issue: https://github.com/apache/spark/pull/22624 Thank you for this work @vanzin ! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands

[GitHub] spark issue #22805: [SPARK-25809][K8S][TEST] New K8S integration testing bac...

2018-10-30 Thread ifilonenko
Github user ifilonenko commented on the issue: https://github.com/apache/spark/pull/22805 @mccheah and @liyinan926 for merge --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands

[GitHub] spark issue #22805: [SPARK-25809][K8S][TEST] New K8S integration testing bac...

2018-10-30 Thread ifilonenko
Github user ifilonenko commented on the issue: https://github.com/apache/spark/pull/22805 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark issue #22805: [SPARK-25809][K8S][TEST] New K8S integration testing bac...

2018-10-30 Thread ifilonenko
Github user ifilonenko commented on the issue: https://github.com/apache/spark/pull/22805 test this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews

[GitHub] spark pull request #22608: [SPARK-25750][K8S][TESTS] Kerberos Support Integr...

2018-10-30 Thread ifilonenko
Github user ifilonenko commented on a diff in the pull request: https://github.com/apache/spark/pull/22608#discussion_r229394011 --- Diff: resource-managers/kubernetes/integration-tests/kerberos-yml/data-populator-deployment.yml --- @@ -0,0 +1,49 @@ +# +# Licensed

[GitHub] spark pull request #22608: [SPARK-25750][K8S][TESTS] Kerberos Support Integr...

2018-10-30 Thread ifilonenko
Github user ifilonenko commented on a diff in the pull request: https://github.com/apache/spark/pull/22608#discussion_r229391389 --- Diff: bin/docker-image-tool.sh --- @@ -53,7 +53,7 @@ function build { # contain a lot of duplicated jars with the main Spark directory

[GitHub] spark pull request #22608: [SPARK-25750][K8S][TESTS] Kerberos Support Integr...

2018-10-30 Thread ifilonenko
Github user ifilonenko commented on a diff in the pull request: https://github.com/apache/spark/pull/22608#discussion_r229386227 --- Diff: resource-managers/kubernetes/integration-tests/kerberos-yml/data-populator-deployment.yml --- @@ -0,0 +1,49 @@ +# +# Licensed

[GitHub] spark issue #22760: [SPARK-25751][K8S][TEST] Unit Testing for Kerberos Suppo...

2018-10-29 Thread ifilonenko
Github user ifilonenko commented on the issue: https://github.com/apache/spark/pull/22760 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark issue #22760: [SPARK-25751][K8S][TEST] Unit Testing for Kerberos Suppo...

2018-10-29 Thread ifilonenko
Github user ifilonenko commented on the issue: https://github.com/apache/spark/pull/22760 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark pull request #19045: [WIP][SPARK-20628][CORE][K8S] Keep track of nodes...

2018-10-29 Thread ifilonenko
Github user ifilonenko commented on a diff in the pull request: https://github.com/apache/spark/pull/19045#discussion_r229012644 --- Diff: resource-managers/kubernetes/integration-tests/tests/decomissioning_water.py --- @@ -0,0 +1,38 @@ +# +# Licensed to the Apache

[GitHub] spark pull request #19045: [WIP][SPARK-20628][CORE][K8S] Keep track of nodes...

2018-10-29 Thread ifilonenko
Github user ifilonenko commented on a diff in the pull request: https://github.com/apache/spark/pull/19045#discussion_r229011769 --- Diff: resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/KubernetesSuite.scala

[GitHub] spark pull request #19045: [WIP][SPARK-20628][CORE][K8S] Keep track of nodes...

2018-10-29 Thread ifilonenko
Github user ifilonenko commented on a diff in the pull request: https://github.com/apache/spark/pull/19045#discussion_r229011136 --- Diff: resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/DecommissionSuite.scala --- @@ -0,0

[GitHub] spark issue #22760: [SPARK-25751][K8S][TEST] Unit Testing for Kerberos Suppo...

2018-10-26 Thread ifilonenko
Github user ifilonenko commented on the issue: https://github.com/apache/spark/pull/22760 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark issue #22608: [SPARK-25750][K8S][TESTS] Kerberos Support Integration T...

2018-10-26 Thread ifilonenko
Github user ifilonenko commented on the issue: https://github.com/apache/spark/pull/22608 > Just noticed this, but could you open a separate bug for adding these tests, instead of re-using the one where the main code was added? It's a large enough thing that it should be a separ

[GitHub] spark issue #22820: [SPARK-25828][K8S] Bumping Kubernetes-Client version to ...

2018-10-26 Thread ifilonenko
Github user ifilonenko commented on the issue: https://github.com/apache/spark/pull/22820 @felixcheung and @erikerlandson for merge --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #22624: [SPARK-23781][CORE] Merge token renewer functiona...

2018-10-26 Thread ifilonenko
Github user ifilonenko commented on a diff in the pull request: https://github.com/apache/spark/pull/22624#discussion_r228605725 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/security/KubernetesHadoopDelegationTokenManager.scala

[GitHub] spark issue #22820: [SPARK-25828][K8S] Bumping Kubernetes-Client version to ...

2018-10-25 Thread ifilonenko
Github user ifilonenko commented on the issue: https://github.com/apache/spark/pull/22820 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark issue #22760: [SPARK-25751][K8S][TEST] Unit Testing for Kerberos Suppo...

2018-10-25 Thread ifilonenko
Github user ifilonenko commented on the issue: https://github.com/apache/spark/pull/22760 moved bootstrap as recommended in comments @mccheah --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org

[GitHub] spark issue #22608: [SPARK-25750][K8S][TESTS] Kerberos Support Integration T...

2018-10-25 Thread ifilonenko
Github user ifilonenko commented on the issue: https://github.com/apache/spark/pull/22608 > Kubernetes integration test status success This new build passes and because I resolved the docker image, by building the `hadoop-base` image with each iteration, this PR is now re

[GitHub] spark issue #22608: [SPARK-25750][K8S][TESTS] Kerberos Support Integration T...

2018-10-25 Thread ifilonenko
Github user ifilonenko commented on the issue: https://github.com/apache/spark/pull/22608 @vanzin as per our conversation offline, that is exactly what my current refactor is doing. I will push an update really soon. Thanks for the recommendation

[GitHub] spark issue #22820: [SPARK-25828][K8S][BUILD] Bumping Kubernetes-Client vers...

2018-10-25 Thread ifilonenko
Github user ifilonenko commented on the issue: https://github.com/apache/spark/pull/22820 In reference to manifests that need to be updated in `dev/deps/spark-deps-hadoop-2.7` and `dev/deps/spark-deps-hadoop-3.1`. This most recent commit should fix

[GitHub] spark pull request #22820: [SPARK-25828][K8S][BUILD] Bumping Kubernetes-Clie...

2018-10-25 Thread ifilonenko
Github user ifilonenko commented on a diff in the pull request: https://github.com/apache/spark/pull/22820#discussion_r228269911 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/KubernetesUtils.scala --- @@ -157,7 +157,10 @@ private[spark

[GitHub] spark pull request #22820: [SPARK-25828][K8S] Bumping Kubernetes-Client vers...

2018-10-24 Thread ifilonenko
Github user ifilonenko commented on a diff in the pull request: https://github.com/apache/spark/pull/22820#discussion_r228004310 --- Diff: resource-managers/kubernetes/core/pom.xml --- @@ -29,7 +29,7 @@ Spark Project Kubernetes kubernetes -3.0.0

[GitHub] spark issue #22798: [SPARK-25803][K8S] Fix docker-image-tool.sh -n option

2018-10-24 Thread ifilonenko
Github user ifilonenko commented on the issue: https://github.com/apache/spark/pull/22798 test this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews

[GitHub] spark issue #22820: [SPARK-25828][K8S] Bumping Kubernetes-Client version to ...

2018-10-24 Thread ifilonenko
Github user ifilonenko commented on the issue: https://github.com/apache/spark/pull/22820 @erikerlandson @skonto @liyinan926 for review Do you see additional places where we would want to leverage the newer version for optimality

[GitHub] spark pull request #22820: [SPARK-25828][K8S] Bumping Kubernetes-Client vers...

2018-10-24 Thread ifilonenko
GitHub user ifilonenko opened a pull request: https://github.com/apache/spark/pull/22820 [SPARK-25828][K8S] Bumping Kubernetes-Client version to 4.0.0 ## What changes were proposed in this pull request? Changed the `kubernetes-client` version and refactored code that broke

[GitHub] spark issue #22810: [SPARK-24516][K8S] Change Python default to Python3

2018-10-24 Thread ifilonenko
Github user ifilonenko commented on the issue: https://github.com/apache/spark/pull/22810 or @holdenk for merge :) --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark pull request #22624: [SPARK-23781][CORE] Merge token renewer functiona...

2018-10-24 Thread ifilonenko
Github user ifilonenko commented on a diff in the pull request: https://github.com/apache/spark/pull/22624#discussion_r227984918 --- Diff: core/src/main/scala/org/apache/spark/deploy/security/HadoopDelegationTokenManager.scala --- @@ -17,76 +17,175 @@ package

[GitHub] spark issue #22608: [SPARK-25750][K8S][TESTS] Kerberos Support Integration T...

2018-10-24 Thread ifilonenko
Github user ifilonenko commented on the issue: https://github.com/apache/spark/pull/22608 gentle ping for further reviews :) as I would love to have this merged in the Jenkins :) as well as recommendations on how to include the `hadoop-.tgz` in the distribution, for building

[GitHub] spark issue #22810: [SPARK-24516][K8S] Change Python default to Python3

2018-10-23 Thread ifilonenko
Github user ifilonenko commented on the issue: https://github.com/apache/spark/pull/22810 @felixcheung for merge and @kokes for opinion and raising this JIRA --- - To unsubscribe, e-mail: reviews-unsubscr

[GitHub] spark pull request #22810: [SPARK-24516][K8S] Change Python default to Pytho...

2018-10-23 Thread ifilonenko
GitHub user ifilonenko opened a pull request: https://github.com/apache/spark/pull/22810 [SPARK-24516][K8S] Change Python default to Python3 ## What changes were proposed in this pull request? As this is targeted for 3.0.0 and Python2 will be deprecated by Jan 1st, 2020, I

[GitHub] spark issue #22798: [SPARK-25803] Fix docker-image-tool.sh -n option

2018-10-23 Thread ifilonenko
Github user ifilonenko commented on the issue: https://github.com/apache/spark/pull/22798 Can you add a [K8S] flag to this PR as this is related to the Kubernetes code. Otherwise, this change looks good to me

[GitHub] spark pull request #22805: [WIP][SPARK-25809][K8S][TEST] New K8S integration...

2018-10-23 Thread ifilonenko
Github user ifilonenko commented on a diff in the pull request: https://github.com/apache/spark/pull/22805#discussion_r227527156 --- Diff: resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/backend/IntegrationTestBackend.scala

[GitHub] spark pull request #22805: [WIP][SPARK-25809][K8S][TEST] New K8S integration...

2018-10-23 Thread ifilonenko
Github user ifilonenko commented on a diff in the pull request: https://github.com/apache/spark/pull/22805#discussion_r227499598 --- Diff: resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/backend/docker

[GitHub] spark pull request #22805: [WIP][SPARK-25809][K8S][TEST] New K8S integration...

2018-10-23 Thread ifilonenko
Github user ifilonenko commented on a diff in the pull request: https://github.com/apache/spark/pull/22805#discussion_r227526473 --- Diff: resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/backend/IntegrationTestBackend.scala

[GitHub] spark pull request #22624: [SPARK-23781][CORE] Merge token renewer functiona...

2018-10-22 Thread ifilonenko
Github user ifilonenko commented on a diff in the pull request: https://github.com/apache/spark/pull/22624#discussion_r227075160 --- Diff: core/src/main/scala/org/apache/spark/deploy/security/HadoopDelegationTokenManager.scala --- @@ -17,76 +17,175 @@ package

[GitHub] spark pull request #22624: [SPARK-23781][CORE] Merge token renewer functiona...

2018-10-22 Thread ifilonenko
Github user ifilonenko commented on a diff in the pull request: https://github.com/apache/spark/pull/22624#discussion_r227064934 --- Diff: core/src/main/scala/org/apache/spark/deploy/security/HadoopDelegationTokenManager.scala --- @@ -17,76 +17,175 @@ package

[GitHub] spark pull request #22624: [SPARK-23781][CORE] Merge token renewer functiona...

2018-10-22 Thread ifilonenko
Github user ifilonenko commented on a diff in the pull request: https://github.com/apache/spark/pull/22624#discussion_r227062715 --- Diff: core/src/main/scala/org/apache/spark/deploy/security/HadoopDelegationTokenManager.scala --- @@ -17,76 +17,175 @@ package

[GitHub] spark pull request #22639: [SPARK-25647][k8s] Add spark streaming compatibil...

2018-10-22 Thread ifilonenko
Github user ifilonenko commented on a diff in the pull request: https://github.com/apache/spark/pull/22639#discussion_r227060819 --- Diff: resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/StreamingCompatibilitySuite.scala

[GitHub] spark pull request #22777: [SPARK-25299][K8S][WIP] Enabling Remote Shuffle B...

2018-10-19 Thread ifilonenko
Github user ifilonenko closed the pull request at: https://github.com/apache/spark/pull/22777 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark pull request #22777: [SPARK-25299][K8S][WIP] Enabling Remote Shuffle B...

2018-10-19 Thread ifilonenko
GitHub user ifilonenko opened a pull request: https://github.com/apache/spark/pull/22777 [SPARK-25299][K8S][WIP] Enabling Remote Shuffle Backup ## What changes were proposed in this pull request? TODO ## How was this patch tested? TODO You can merge

[GitHub] spark issue #22748: [SPARK-25745][K8S] Improve docker-image-tool.sh script

2018-10-19 Thread ifilonenko
Github user ifilonenko commented on the issue: https://github.com/apache/spark/pull/22748 @vanzin I think it is related to having access to trigger the k8s builds (I or @shaneknapp ) can add you if you would like

[GitHub] spark issue #22748: [SPARK-25745][K8S] Improve docker-image-tool.sh script

2018-10-19 Thread ifilonenko
Github user ifilonenko commented on the issue: https://github.com/apache/spark/pull/22748 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark pull request #22704: [SPARK-25681][K8S][WIP] Leverage a config to tune...

2018-10-17 Thread ifilonenko
Github user ifilonenko commented on a diff in the pull request: https://github.com/apache/spark/pull/22704#discussion_r226127311 --- Diff: core/src/main/scala/org/apache/spark/deploy/security/HadoopFSDelegationTokenProvider.scala --- @@ -49,8 +49,11 @@ private[deploy] class

[GitHub] spark issue #22760: [SPARK-25751][K8S][TEST] Unit Testing for Kerberos Suppo...

2018-10-17 Thread ifilonenko
Github user ifilonenko commented on the issue: https://github.com/apache/spark/pull/22760 @liyinan926 @mccheah for review on unit testing --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org

[GitHub] spark pull request #22760: [SPARK-25751][K8S][TEST] Unit Testing for Kerbero...

2018-10-17 Thread ifilonenko
GitHub user ifilonenko opened a pull request: https://github.com/apache/spark/pull/22760 [SPARK-25751][K8S][TEST] Unit Testing for Kerberos Support ## What changes were proposed in this pull request? Unit tests for Kerberos support addition ## How was this patch

[GitHub] spark issue #22748: [SPARK-25745][K8S] Improve docker-image-tool.sh script

2018-10-16 Thread ifilonenko
Github user ifilonenko commented on the issue: https://github.com/apache/spark/pull/22748 There seems to be overlapping logic between this PR and https://github.com/apache/spark/pull/22681 --- - To unsubscribe, e

[GitHub] spark issue #22146: [SPARK-24434][K8S] pod template files

2018-10-16 Thread ifilonenko
Github user ifilonenko commented on the issue: https://github.com/apache/spark/pull/22146 Is there any progress on this? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e

[GitHub] spark pull request #22608: [SPARK-25750][K8S][TESTS] Kerberos Support Integr...

2018-10-16 Thread ifilonenko
Github user ifilonenko commented on a diff in the pull request: https://github.com/apache/spark/pull/22608#discussion_r225714964 --- Diff: bin/docker-image-tool.sh --- @@ -71,18 +71,29 @@ function build { --build-arg base_img=$(image_ref spark

[GitHub] spark pull request #22639: [SPARK-25647][k8s] Add spark streaming compatibil...

2018-10-16 Thread ifilonenko
Github user ifilonenko commented on a diff in the pull request: https://github.com/apache/spark/pull/22639#discussion_r225686232 --- Diff: resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/StreamingCompatibilitySuite.scala

[GitHub] spark issue #22722: [SPARK-24432][k8s] Add support for dynamic resource allo...

2018-10-16 Thread ifilonenko
Github user ifilonenko commented on the issue: https://github.com/apache/spark/pull/22722 @huafengw Right now I am testing and further developing the experimental design (Option No. 5) that was proposed in https://issues.apache.org/jira/browse/SPARK-25299, building off of the work

[GitHub] spark pull request #22608: [SPARK-23257][K8S][TESTS] Kerberos Support Integr...

2018-10-16 Thread ifilonenko
Github user ifilonenko commented on a diff in the pull request: https://github.com/apache/spark/pull/22608#discussion_r225631577 --- Diff: bin/docker-image-tool.sh --- @@ -71,18 +71,29 @@ function build { --build-arg base_img=$(image_ref spark

[GitHub] spark pull request #22608: [SPARK-23257][K8S][TESTS] Kerberos Support Integr...

2018-10-16 Thread ifilonenko
Github user ifilonenko commented on a diff in the pull request: https://github.com/apache/spark/pull/22608#discussion_r225630500 --- Diff: bin/docker-image-tool.sh --- @@ -71,18 +71,29 @@ function build { --build-arg base_img=$(image_ref spark

[GitHub] spark issue #22704: [SPARK-25681][K8S][WIP] Leverage a config to tune renewa...

2018-10-15 Thread ifilonenko
Github user ifilonenko commented on the issue: https://github.com/apache/spark/pull/22704 @vanzin and @jerryshao for opinions on approach. The classes are quite difficult to unit-test with mocking libraries, maybe a refactor is necessary, but was wondering if the overall approach

[GitHub] spark issue #22608: [SPARK-23257][K8S][TESTS] Kerberos Support Integration T...

2018-10-15 Thread ifilonenko
Github user ifilonenko commented on the issue: https://github.com/apache/spark/pull/22608 @mccheah and @erikerlandson for review as the `Secure HDFS test with HDFS keytab (Cluster Mode)` passes and should be merged to enable Secure HDFS interaction

[GitHub] spark issue #22608: [SPARK-23257][K8S][TESTS] Kerberos Support Integration T...

2018-10-15 Thread ifilonenko
Github user ifilonenko commented on the issue: https://github.com/apache/spark/pull/22608 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark issue #19045: [WIP][SPARK-20628][CORE] Keep track of nodes (/ spot ins...

2018-10-12 Thread ifilonenko
Github user ifilonenko commented on the issue: https://github.com/apache/spark/pull/19045 please rename PR with [K8S] flag to launch tests. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org

[GitHub] spark issue #19045: [WIP][SPARK-20628][CORE] Keep track of nodes (/ spot ins...

2018-10-12 Thread ifilonenko
Github user ifilonenko commented on the issue: https://github.com/apache/spark/pull/19045 test this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews

[GitHub] spark pull request #22704: [SPARK-25681][K8S][WIP] Leverage a config to tune...

2018-10-11 Thread ifilonenko
GitHub user ifilonenko opened a pull request: https://github.com/apache/spark/pull/22704 [SPARK-25681][K8S][WIP] Leverage a config to tune renewal time retrieval ## What changes were proposed in this pull request? Changes to core allow for a K8S to pass a SparkConf

[GitHub] spark pull request #21669: [SPARK-23257][K8S] Kerberos Support for Spark on ...

2018-10-10 Thread ifilonenko
Github user ifilonenko commented on a diff in the pull request: https://github.com/apache/spark/pull/21669#discussion_r224250496 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/hadooputils/HadoopKerberosLogin.scala --- @@ -0,0 +1,66

[GitHub] spark pull request #21669: [SPARK-23257][K8S] Kerberos Support for Spark on ...

2018-10-09 Thread ifilonenko
Github user ifilonenko commented on a diff in the pull request: https://github.com/apache/spark/pull/21669#discussion_r223885097 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/KerberosConfDriverFeatureStep.scala --- @@ -0,0 +1,179

[GitHub] spark issue #21669: [SPARK-23257][K8S] Kerberos Support for Spark on K8S

2018-10-09 Thread ifilonenko
Github user ifilonenko commented on the issue: https://github.com/apache/spark/pull/21669 thank you @liyinan926 for your review! @vanzin, ill leave it to you to give the final LGTM --- - To unsubscribe, e-mail

[GitHub] spark pull request #21669: [SPARK-23257][K8S] Kerberos Support for Spark on ...

2018-10-08 Thread ifilonenko
Github user ifilonenko commented on a diff in the pull request: https://github.com/apache/spark/pull/21669#discussion_r223535529 --- Diff: docs/running-on-kubernetes.md --- @@ -820,4 +820,37 @@ specific to Spark on Kubernetes. This sets the major Python version

[GitHub] spark pull request #21669: [SPARK-23257][K8S] Kerberos Support for Spark on ...

2018-10-08 Thread ifilonenko
Github user ifilonenko commented on a diff in the pull request: https://github.com/apache/spark/pull/21669#discussion_r223525420 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/hadooputils/HadoopKerberosLogin.scala --- @@ -0,0 +1,75

[GitHub] spark pull request #21669: [SPARK-23257][K8S] Kerberos Support for Spark on ...

2018-10-08 Thread ifilonenko
Github user ifilonenko commented on a diff in the pull request: https://github.com/apache/spark/pull/21669#discussion_r223524110 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/hadooputils/HadoopKerberosLogin.scala --- @@ -0,0 +1,75

[GitHub] spark pull request #21669: [SPARK-23257][K8S] Kerberos Support for Spark on ...

2018-10-08 Thread ifilonenko
Github user ifilonenko commented on a diff in the pull request: https://github.com/apache/spark/pull/21669#discussion_r223523341 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/KerberosConfDriverFeatureStep.scala --- @@ -0,0 +1,156

[GitHub] spark pull request #21669: [SPARK-23257][K8S] Kerberos Support for Spark on ...

2018-10-08 Thread ifilonenko
Github user ifilonenko commented on a diff in the pull request: https://github.com/apache/spark/pull/21669#discussion_r223492379 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/HadoopConfExecutorFeatureStep.scala --- @@ -0,0 +1,51

[GitHub] spark pull request #21669: [SPARK-23257][K8S] Kerberos Support for Spark on ...

2018-10-08 Thread ifilonenko
Github user ifilonenko commented on a diff in the pull request: https://github.com/apache/spark/pull/21669#discussion_r223483718 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/HadoopConfExecutorFeatureStep.scala --- @@ -0,0 +1,51

[GitHub] spark pull request #21669: [SPARK-23257][K8S] Kerberos Support for Spark on ...

2018-10-08 Thread ifilonenko
Github user ifilonenko commented on a diff in the pull request: https://github.com/apache/spark/pull/21669#discussion_r223481427 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/KerberosConfDriverFeatureStep.scala --- @@ -0,0 +1,161

  1   2   3   4   >