[GitHub] [spark] skonto commented on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System

2019-05-22 Thread GitBox
skonto commented on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System URL: https://github.com/apache/spark/pull/23546#issuecomment-494734965 @erikerlandson from what I see there is no more activity, could you merge please?

[GitHub] [spark] skonto commented on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System

2019-05-20 Thread GitBox
skonto commented on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System URL: https://github.com/apache/spark/pull/23546#issuecomment-494104863 @erikerlandson @vanzin are you ok with the current status of things? Should it be merged?

[GitHub] [spark] skonto commented on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System

2019-05-20 Thread GitBox
skonto commented on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System URL: https://github.com/apache/spark/pull/23546#issuecomment-493931364 @srowen @erikerlandson gentle ping

[GitHub] [spark] skonto commented on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System

2019-05-17 Thread GitBox
skonto commented on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System URL: https://github.com/apache/spark/pull/23546#issuecomment-493568264 @srowen I fixed the two pending comments.

[GitHub] [spark] skonto commented on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System

2019-05-16 Thread GitBox
skonto commented on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System URL: https://github.com/apache/spark/pull/23546#issuecomment-492988663 @erikerlandson @srowen I added support for random dirs.

[GitHub] [spark] skonto commented on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System

2019-05-16 Thread GitBox
skonto commented on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System URL: https://github.com/apache/spark/pull/23546#issuecomment-492987168 jenkins test this please This is an

[GitHub] [spark] skonto commented on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System

2019-05-15 Thread GitBox
skonto commented on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System URL: https://github.com/apache/spark/pull/23546#issuecomment-492779301 jenkins test this please This is an

[GitHub] [spark] skonto commented on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System

2019-05-14 Thread GitBox
skonto commented on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System URL: https://github.com/apache/spark/pull/23546#issuecomment-492290588 @srowen ok will do that and if someone can always change the code with a new PR.

[GitHub] [spark] skonto commented on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System

2019-05-14 Thread GitBox
skonto commented on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System URL: https://github.com/apache/spark/pull/23546#issuecomment-492182627 @srowen one reason I dont want to delete the subdir it is because user provides it and may want to

[GitHub] [spark] skonto commented on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System

2019-05-13 Thread GitBox
skonto commented on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System URL: https://github.com/apache/spark/pull/23546#issuecomment-492000853 @srowen thoughts? This is an

[GitHub] [spark] skonto commented on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System

2019-05-11 Thread GitBox
skonto commented on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System URL: https://github.com/apache/spark/pull/23546#issuecomment-491546073 @srowen there a property that can be used by the user to set where he wants to upload the artifacts.

[GitHub] [spark] skonto commented on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System

2019-05-11 Thread GitBox
skonto commented on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System URL: https://github.com/apache/spark/pull/23546#issuecomment-491523244 @srowen I also felt missing the counterargument. @vanzin could you clarify what is the issue here so

[GitHub] [spark] skonto commented on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System

2019-05-09 Thread GitBox
skonto commented on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System URL: https://github.com/apache/spark/pull/23546#issuecomment-491015268 @vanzin can you also focus on my argument I never used such a word btw. It is not the first time

[GitHub] [spark] skonto commented on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System

2019-05-09 Thread GitBox
skonto commented on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System URL: https://github.com/apache/spark/pull/23546#issuecomment-491008961 Btw using strong language to describe a choice and which creates just impressions eg. terrible is

[GitHub] [spark] skonto commented on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System

2019-05-09 Thread GitBox
skonto commented on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System URL: https://github.com/apache/spark/pull/23546#issuecomment-491006649 Should spark manage s3 folders accounts etc? This is optimized for the app dev cycle where I do a

[GitHub] [spark] skonto commented on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System

2019-05-09 Thread GitBox
skonto commented on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System URL: https://github.com/apache/spark/pull/23546#issuecomment-490870511 @vanzin this is not a big its a design choice. If you want separate uploads define a separate path.

[GitHub] [spark] skonto commented on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System

2019-05-07 Thread GitBox
skonto commented on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System URL: https://github.com/apache/spark/pull/23546#issuecomment-490191825 @srowen @vanzin should we merge it?

[GitHub] [spark] skonto commented on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System

2019-04-29 Thread GitBox
skonto commented on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System URL: https://github.com/apache/spark/pull/23546#issuecomment-487606840 @vanzin gentle ping. I think its ready.

[GitHub] [spark] skonto commented on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System

2019-04-28 Thread GitBox
skonto commented on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System URL: https://github.com/apache/spark/pull/23546#issuecomment-487442281 jenkins test this please This is an

[GitHub] [spark] skonto commented on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System

2019-04-28 Thread GitBox
skonto commented on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System URL: https://github.com/apache/spark/pull/23546#issuecomment-487419487 @shaneknapp there is a weird error: ``` 2019-04-28 22:01:02 INFO MicroBatchExecution:54 -

[GitHub] [spark] skonto commented on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System

2019-04-28 Thread GitBox
skonto commented on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System URL: https://github.com/apache/spark/pull/23546#issuecomment-487418537 jenkins test this please This is an

[GitHub] [spark] skonto commented on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System

2019-04-26 Thread GitBox
skonto commented on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System URL: https://github.com/apache/spark/pull/23546#issuecomment-48716 @erikerlandson gentle ping :) This

[GitHub] [spark] skonto commented on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System

2019-04-18 Thread GitBox
skonto commented on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System URL: https://github.com/apache/spark/pull/23546#issuecomment-484733537 @shaneknapp ``` > devtools::install_version('testthat', version = '1.0.2',

[GitHub] [spark] skonto commented on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System

2019-04-18 Thread GitBox
skonto commented on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System URL: https://github.com/apache/spark/pull/23546#issuecomment-484654875 @shaneknapp do you know why checks fail here?

[GitHub] [spark] skonto commented on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System

2019-04-18 Thread GitBox
skonto commented on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System URL: https://github.com/apache/spark/pull/23546#issuecomment-484644274 @srowen I fixed most stuff, for things not modified there is a comment. Please have a look. Yes

[GitHub] [spark] skonto commented on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System

2019-04-15 Thread GitBox
skonto commented on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System URL: https://github.com/apache/spark/pull/23546#issuecomment-483434764 @erikerlandson gentle ping. This is

[GitHub] [spark] skonto commented on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System

2019-04-12 Thread GitBox
skonto commented on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System URL: https://github.com/apache/spark/pull/23546#issuecomment-482549358 @erikerlandson @liyinan926 @vanzin PR has been updated with the addition of integration tests. As

[GitHub] [spark] skonto commented on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System

2019-03-27 Thread GitBox
skonto commented on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System URL: https://github.com/apache/spark/pull/23546#issuecomment-477112560 test this please This is an

[GitHub] [spark] skonto commented on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System

2019-03-20 Thread GitBox
skonto commented on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System URL: https://github.com/apache/spark/pull/23546#issuecomment-474880219 WIP This is an automated message

[GitHub] [spark] skonto commented on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System

2019-03-12 Thread GitBox
skonto commented on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System URL: https://github.com/apache/spark/pull/23546#issuecomment-472035217 @erikerlandson I think rook.io with ceph might be a great option for the tests here instead of minio